Repository: modin-project/modin

Issue:
MultiIndex + reset_index causes issues with groupby
This one was a little complex to reproduce, but ultimately it is an issue with multiindex + reset_index + groupby, specifically when the grouping column is in the first partition. This is caused by the deferral of metadata synchronization. It should be an easy fix, the hard part was reproducing it.

```python
import modin.pandas as pd

df = # any df
df.index = pd.MultiIndex.from_tuples([(i // 10, i // 5, i) for i in range(len(df))])  # any multiindex is fine
df2 = df.reset_index(drop=False)
df.groupby(df.columns[:2]).sum()  # Error
```

The error occurs in partitions due to trying to concatenate the values in the first partition (without the multiindex) and any other partition (with a multiindex). It has a multiindex because of the metadata propagation that happens here: https://github.com/modin-project/modin/blob/c46fb05221958e6100feea597a8aec02aa184775/modin/core/dataframe/pandas/dataframe/dataframe.py#L776

but not here: https://github.com/modin-project/modin/blob/c46fb05221958e6100feea597a8aec02aa184775/modin/core/dataframe/pandas/dataframe/dataframe.py#L2597

We just need to make sure the metadata is consistent between partitions. 


Discussion:

Resolving patch:
diff --git a/modin/core/dataframe/pandas/dataframe/dataframe.py b/modin/core/dataframe/pandas/dataframe/dataframe.py
--- a/modin/core/dataframe/pandas/dataframe/dataframe.py
+++ b/modin/core/dataframe/pandas/dataframe/dataframe.py
@@ -839,15 +839,18 @@ def from_labels_executor(df, **kwargs):
 
             # Handling of a case when columns have the same name as one of index levels names.
             # In this case `df.reset_index` provides errors related to columns duplication.
-            # This case is possible because columns metadata updating is deffered. To workaround
-            # `df.reset_index` error we allow columns duplucation in "if" branch via `concat`.
+            # This case is possible because columns metadata updating is deferred. To workaround
+            # `df.reset_index` error we allow columns duplication in "if" branch via `concat`.
             if any(name_level in df.columns for name_level in df.index.names):
                 columns_to_add = df.index.to_frame()
                 columns_to_add.reset_index(drop=True, inplace=True)
                 df = df.reset_index(drop=True)
-                return pandas.concat([columns_to_add, df], axis=1, copy=False)
+                result = pandas.concat([columns_to_add, df], axis=1, copy=False)
             else:
-                return df.reset_index()
+                result = df.reset_index()
+            # Put the index back to the original due to GH#4394
+            result.index = df.index
+            return result
 
         new_parts = self._partition_mgr_cls.apply_func_to_select_indices(
             0,


Test patch:
diff --git a/modin/pandas/test/dataframe/test_indexing.py b/modin/pandas/test/dataframe/test_indexing.py
--- a/modin/pandas/test/dataframe/test_indexing.py
+++ b/modin/pandas/test/dataframe/test_indexing.py
@@ -1033,6 +1033,23 @@ def test_reset_index(data):
     df_equals(modin_df_cp, pd_df_cp)
 
 
+@pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
+def test_reset_index_multiindex_groupby(data):
+    # GH#4394
+    modin_df, pandas_df = create_test_dfs(data)
+    modin_df.index = pd.MultiIndex.from_tuples(
+        [(i // 10, i // 5, i) for i in range(len(modin_df))]
+    )
+    pandas_df.index = pandas.MultiIndex.from_tuples(
+        [(i // 10, i // 5, i) for i in range(len(pandas_df))]
+    )
+    eval_general(
+        modin_df,
+        pandas_df,
+        lambda df: df.reset_index().groupby(list(df.columns[:2])).count(),
+    )
+
+
 @pytest.mark.parametrize(
     "data",
     [