Repository: pandas-dev/pandas
Pull request: BUG: Break reference from grouping level to MI
Closes https://github.com/pandas-dev/pandas/issues/31068

Problem statement:
REGR: MultiIndex level names RuntimeError in groupby.apply
```
df = pd.DataFrame({
    'A': np.arange(10), 'B': [1, 2] * 5, 
    'C': np.random.rand(10), 'D': np.random.rand(10)}
).set_index(['A', 'B'])  
df.groupby('B').apply(lambda x: x.sum())
```

On master this gives an error:
```
In [40]: df.groupby('B').apply(lambda x: x.sum())
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-40-75bc1ff12251> in <module>
----> 1 df.groupby('B').apply(lambda x: x.sum())

~/scipy/pandas/pandas/core/groupby/groupby.py in apply(self, func, *args, **kwargs)
    733         with option_context("mode.chained_assignment", None):
    734             try:
--> 735                 result = self._python_apply_general(f)
    736             except TypeError:
    737                 # gh-20949

~/scipy/pandas/pandas/core/groupby/groupby.py in _python_apply_general(self, f)
    752 
    753         return self._wrap_applied_output(
--> 754             keys, values, not_indexed_same=mutated or self.mutated
    755         )
    756 

~/scipy/pandas/pandas/core/groupby/generic.py in _wrap_applied_output(self, keys, values, not_indexed_same)
   1200                 if len(keys) == ping.ngroups:
   1201                     key_index = ping.group_index
-> 1202                     key_index.name = key_names[0]
   1203 
   1204                     key_lookup = Index(keys)

~/scipy/pandas/pandas/core/indexes/base.py in name(self, value)
   1171             # Used in MultiIndex.levels to avoid silently ignoring name updates.
   1172             raise RuntimeError(
-> 1173                 "Cannot set name on a level of a MultiIndex. Use "
   1174                 "'MultiIndex.set_names' instead."
   1175             )

RuntimeError: Cannot set name on a level of a MultiIndex. Use 'MultiIndex.set_names' instead.
```

On 0.25.3 this works:

```
In [10]:  df.groupby('B').apply(lambda x: x.sum()) 
Out[10]: 
          C         D
B                    
1  2.761792  3.963817
2  1.040950  3.578762
```

It seems the additional MultiIndex level that is not used to group (['A', 'B'] are index levels, but only grouping by 'B').

Hints from issue discussion:

Commit message: BUG: Break reference from grouping level to MI

Closes https://github.com/pandas-dev/pandas/issues/31068

Patch to review:
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
index 21421a6f6ea62..246697979c66f 100644
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -1235,6 +1235,9 @@ def _set_names(self, names, level=None, validate=True):
     def _get_grouper_for_level(self, mapper, level):
         indexer = self.codes[level]
         level_index = self.levels[level]
+        # break references back to ourself so that setting the name
+        # on the output of a groupby doesn't reflect back here.
+        level_index = level_index.copy()
 
         if mapper is not None:
             # Handle group mapping function and return
diff --git a/pandas/tests/groupby/test_apply.py b/pandas/tests/groupby/test_apply.py
index 2f2f97f2cd993..af8660024c0e3 100644
--- a/pandas/tests/groupby/test_apply.py
+++ b/pandas/tests/groupby/test_apply.py
@@ -752,3 +752,21 @@ def most_common_values(df):
         ["17661101"], index=pd.DatetimeIndex(["2015-02-24"], name="day"), name="userId"
     )
     tm.assert_series_equal(result, expected)
+
+
+def test_apply_multi_level_name():
+    # https://github.com/pandas-dev/pandas/issues/31068
+    df = pd.DataFrame(
+        {
+            "A": np.arange(10),
+            "B": [1, 2] * 5,
+            "C": list(range(10)),
+            "D": list(range(10)),
+        }
+    ).set_index(["A", "B"])
+    result = df.groupby("B").apply(lambda x: x.sum())
+    expected = pd.DataFrame(
+        {"C": [20, 25], "D": [20, 25]}, index=pd.Index([1, 2], name="B")
+    )
+    tm.assert_frame_equal(result, expected)
+    assert df.index.names == ["A", "B"]


Code review comments:
diff_hunk: @@ -1235,6 +1235,9 @@ def _set_names(self, names, level=None, validate=True):
     def _get_grouper_for_level(self, mapper, level):
         indexer = self.codes[level]
         level_index = self.levels[level]
+        # break references back to ourself so that setting the name
+        # on the output of a groupby doesn't reflect back here.
+        level_index = level_index.copy()
path: pandas/core/indexes/multi.py
text: @user1:
you don't need to do this here, rather do it as an else on line 1261 (add it); as take already copies, its just when that condition is not satisfied does it need a copy.

@author:
Fixed, I think. Added a case for Categorical grouper with unobserved categories, so I think all the cases are covered.