Repository: pandas-dev/pandas

Issue:
Assignment in groupby fails with error (AssertionError: Index length did not match values)
I have a dataset that is inherently hierarchical (from some experiments). For each large category 1, there are several category 2 entries, and in turn for each cat2 entry there are several cat3 entries. For each (cat1, cat2, cat3) tuple I have a measurement value, val below.

The analysis I am doing is per category 1. I first rank all values within cat1, and then I use those ranks to calculate a statistic for each cat2. The code below exemplifies, what I am doing.

When I group by cat1 and apply a function for the calculations, I get an error when I assign to the group. This can be reproduced with the below code.

The error goes away when a copy of the sub-DataFrame is made (in comment below in function f) in the applied function.

I am prepared to think that this should indeed raise an error, as it doesn't strike me as unreasonable that altering the groups over which one is iterating can be bad idea. However, if this is indeed expected behavior, then it's not mentioned in the documentation (as far as I can tell).

Looking forward to any comments on this.
# example of assignment in groupby

mydf = pd.DataFrame({
                     'cat1' : ['a'] \* 8 + ['b'] \* 6,
                     'cat2' : ['c'] \* 2 + ['d'] \* 2 + ['e'] \* 2 + ['f'] \* 2 + ['c'] \* 2 + ['d'] \* 2 + ['e'] \* 2,
                     'cat3' : map(lambda x: 'g{}'.format(x), range(1,15)),
                     'val'  : np.random.randint(100, size=14),
})

def f(x):
    # no error if copy is used:
    # x = x.copy()
    # the following assignment causes an error
    x['rank'] = x.val.rank(method='min')
    return x.groupby('cat2')['rank'].min()

grpby = mydf.groupby('cat1').apply(f)



Discussion:

Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -620,7 +620,9 @@ def apply(self, f, data, axis=0, keep_internal=False):
             try:
                 values, mutated = splitter.fast_apply(f, group_keys)
                 return group_keys, values, mutated
-            except lib.InvalidApply:
+            except (Exception), detail:
+                # we detect a mutatation of some kind
+                # so take slow path
                 pass
 
         result_values = []


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -1491,6 +1491,30 @@ def f(group):
         for key, group in grouped:
             assert_frame_equal(result.ix[key], f(group))
 
+    def test_mutate_groups(self):
+
+        # GH3380
+
+        mydf = DataFrame({
+                'cat1' : ['a'] * 8 + ['b'] * 6,
+                'cat2' : ['c'] * 2 + ['d'] * 2 + ['e'] * 2 + ['f'] * 2 + ['c'] * 2 + ['d'] * 2 + ['e'] * 2,
+                'cat3' : map(lambda x: 'g%s' % x, range(1,15)),
+                'val' : np.random.randint(100, size=14),
+                })
+
+        def f_copy(x):
+            x = x.copy()
+            x['rank'] = x.val.rank(method='min')
+            return x.groupby('cat2')['rank'].min()
+
+        def f_no_copy(x):
+            x['rank'] = x.val.rank(method='min')
+            return x.groupby('cat2')['rank'].min()
+
+        grpby_copy    = mydf.groupby('cat1').apply(f_copy)
+        grpby_no_copy = mydf.groupby('cat1').apply(f_no_copy)
+        assert_series_equal(grpby_copy,grpby_no_copy)
+
     def test_apply_chunk_view(self):
         # Low level tinkering could be unsafe, make sure not
         df = DataFrame({'key': [1, 1, 1, 2, 2, 2, 3, 3, 3],