Repository: lincc-frameworks/nested-pandas

Issue:
NestedFrame.eval does not recognize new columns in existing nests in multiline strings
**Bug report**


**Before submitting**
Please check the following:

- [X] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
- [X] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a description of what I expected instead.
- [X] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

When `NestedFrame.eval` is used with multiline strings to create new columns in existing nests, it fails to recognize these new columns later within the string:

```python
nf = generate_data(10, 100)                                                                                                                               
nf.eval("""                                                                                                                                               
    c = a + b                                                                                                                                             
    nested.d = nested.flux * 2                                                                                                                            
    n2.gumption = nested.d / (a + b)                                                                                                                      
    """, inplace=False)  # value of inplace does not matter                                                                                               
```

`nested.d` is not recognized.  It will be recognized on the *next* call to `NestedFrame.eval`, but not within this string.

This is because the `_NestedFieldResolver` used by the evaluator caches the flattened nest upon referencing it, so that it doesn't notice changes to it.  Note that the changes really have been made: `nested.d` is created, just not visible to the later lines in the multiline string.

The solution is to invalidate the appropriate `_NestedFieldResolver` instance's cache when an `eval()` assignment is made.  The best place to do this is probably in `_NestResolver.__setitem__`.

Discussion:

Solution patch:
diff --git a/src/nested_pandas/nestedframe/core.py b/src/nested_pandas/nestedframe/core.py
index bf2ea6fb..08907321 100644
--- a/src/nested_pandas/nestedframe/core.py
+++ b/src/nested_pandas/nestedframe/core.py
@@ -94,13 +94,19 @@ def __getitem__(self, item):
             super().__setitem__(top_nest, _NestedFieldResolver(top_nest, self._outer))
         return super().__getitem__(top_nest)
 
-    def __setitem__(self, key, value):
+    def __setitem__(self, item, _):
         # Called to update the resolver with intermediate values.
         # The important point is to intercept the call so that the evaluator
-        # does not create any new resolvers on the fly.  Storing the value
-        # is not important, since that will have been done already in
-        # the NestedFrame.
-        pass
+        # does not create any new resolvers on the fly.  We do NOT want to
+        # store the given value, since the resolver does lazy-loading.
+        # What we DO want to do, however, is to invalidate the cache for
+        # any field resolver for a given nest that is receiving an assignment.
+        # Since the resolvers are created as-needed in __getitem__, all we need
+        # to do is delete them from the local cache when this pattern is detected.
+        if "." in item:
+            top_nest = item.split(".")[0].strip()
+            if top_nest in self._outer.nested_columns and super().__contains__(top_nest):
+                del self[top_nest]  # force re-creation in __setitem__
 
 
 class _NestedFieldResolver:


Test patch:
diff --git a/tests/nested_pandas/nestedframe/test_nestedframe.py b/tests/nested_pandas/nestedframe/test_nestedframe.py
index eb7c3f3d..fa5da349 100644
--- a/tests/nested_pandas/nestedframe/test_nestedframe.py
+++ b/tests/nested_pandas/nestedframe/test_nestedframe.py
@@ -884,14 +884,20 @@ def test_eval_assignment():
     assert (nf_n3["p2.d"] == nf_n2["p2.c2"] + nf["packed.d"] * 2 + nf["b"]).all()
 
     # Now test multiline and inplace=True
+    # Verify the resolution of GH#159, where a nested column created in
+    # an existing nest during a multi-line eval was not being recognized
+    # in a subsequent line.
     nf.eval(
         """
         c = a + b
-        p2.e = packed.d * 2 + c
+        packed.e = packed.d * 2
+        p2.e = packed.e + c
         p2.f = p2.e + b
         """,
         inplace=True,
     )
-    assert len(nf.p2.nest.fields) == 2
+    assert set(nf.nested_columns) == {"packed", "p2"}
+    assert set(nf.packed.nest.fields) == {"c", "d", "e", "time"}
+    assert set(nf.p2.nest.fields) == {"e", "f"}
     assert (nf["p2.e"] == nf["packed.d"] * 2 + nf.c).all()
     assert (nf["p2.f"] == nf["p2.e"] + nf.b).all()