Repository: lincc-frameworks/nested-pandas

Issue:
Queries that return empty results can fail
**Bug report**
@AndyTza ran into an issue recently, where he was applying a query on his nested-dask workflow. This query resulting in empty dataframes for a small subset of partitions in the dataset. The nested-pandas implementation of query has issues with this: https://github.com/lincc-frameworks/nested-pandas/blob/main/src/nested_pandas/nestedframe/core.py#L265

These issues relate to the downstream function `query_flat`, having this exception for an empty result: https://github.com/lincc-frameworks/nested-pandas/blob/main/src/nested_pandas/series/accessor.py#L226. The empty series seems to not work correctly in some situations, and can produce this error: `TypeError: incompatible index of inserted column with frame index`

Interestingly, this doesn't come up in a simple toy case:
![Screen Shot 2024-08-02 at 3 42 13 PM](https://github.com/user-attachments/assets/3be035a3-87ce-42d9-8b3f-816124fe1da1)

There's a notebook on epyc (message me for the path) that can reproduce the error. So far, I've seen that simply deleting the `if len(flat) == 0:` section seems effective at removing the error, but we should look deeper at this and make sure we find a robust solution. 

A draft PR with the rough fix is here: https://github.com/lincc-frameworks/nested-pandas/pull/128

**Before submitting**
Please check the following:

- [ ] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [ ] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
- [ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

Discussion:

Patch:
diff --git a/src/nested_pandas/series/accessor.py b/src/nested_pandas/series/accessor.py
index f6253bd..b7936bb 100644
--- a/src/nested_pandas/series/accessor.py
+++ b/src/nested_pandas/series/accessor.py
@@ -223,8 +223,11 @@ class NestSeriesAccessor(Mapping):
             The filtered series.
         """
         flat = self.to_flat().query(query)
+
         if len(flat) == 0:
-            return pd.Series([], dtype=self._series.dtype)
+            return pd.Series(
+                [], dtype=self._series.dtype, index=pd.Index([], dtype=flat.index.dtype, name=flat.index.name)
+            )
         return pack_sorted_df_into_struct(flat)
 
     def get_flat_index(self) -> pd.Index:


Test patch:
diff --git a/tests/nested_pandas/series/test_accessor.py b/tests/nested_pandas/series/test_accessor.py
index a4ac4db..b63c879 100644
--- a/tests/nested_pandas/series/test_accessor.py
+++ b/tests/nested_pandas/series/test_accessor.py
@@ -1,3 +1,4 @@
+import nested_pandas as npd
 import numpy as np
 import pandas as pd
 import pyarrow as pa
@@ -487,6 +488,17 @@ def test_query_flat_empty_rows():
     assert_series_equal(filtered, desired)
 
 
+def test_query_flat_with_empty_result():
+    """Make sure the index is properly set for empty result cases"""
+    base = npd.NestedFrame({"a": []}, index=pd.Index([], dtype=np.float64))
+    nested = npd.NestedFrame({"b": []}, index=pd.Index([], dtype=np.float64))
+
+    ndf = base.add_nested(nested, "nested")
+
+    res = ndf.nested.nest.query_flat("b > 2")
+    assert res.index.dtype == np.float64
+
+
 @pytest.mark.parametrize(
     "df",
     [