Repository: modin-project/modin

Issue:
df.groupby('index_column_name') results in a key error. In pandas it doesn't
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: I use arch btw
- **Modin version** (`modin.__version__`): 0.8.0
- **Python version**: 3.8
- **Code we can use to reproduce**: 

### Pandas (works fine):
```
import pandas as pd
from pandas import util
df= util.testing.makeMixedDataFrame()
print(df.head())
df = df.to_numpy()
df = pd.DataFrame(df)
df.columns = ["A", "B", "C", "D"]
df = df.set_index("C")
df.groupby("C")
```
### Modin (gives error):
```
import modin.pandas as pd
from pandas import util
df= util.testing.makeMixedDataFrame()
print(df.head())
df = df.to_numpy()
df = pd.DataFrame(df)
df.columns = ["A", "B", "C", "D"]
df = df.set_index("C")
df.groupby("C") #  <- Key Error! (scroll below)

```
<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->

Using groupby with index column name in Pandas does not give a key error and works fine. However in Modin this results in a key error

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->
```
KeyError                                  Traceback (most recent call last)
 in 
      7 df.columns = ["A", "B", "C", "D"]
      8 df = df.set_index("C")
----> 9 df.groupby("C")

~/anaconda3/envs/recnn/lib/python3.8/site-packages/modin/pandas/dataframe.py in groupby(self, by, axis, level, as_index, sort, group_keys, squeeze, observed)
    434                 pass
    435             else:
--> 436                 by = self.__getitem__(by)._query_compiler
    437         elif isinstance(by, Series):
    438             drop = by._parent is self

~/anaconda3/envs/recnn/lib/python3.8/site-packages/modin/pandas/base.py in __getitem__(self, key)
   3458             return self._getitem_slice(indexer)
   3459         else:
-> 3460             return self._getitem(key)
   3461 
   3462     def _getitem_slice(self, key):

~/anaconda3/envs/recnn/lib/python3.8/site-packages/modin/pandas/dataframe.py in _getitem(self, key)
   2422             # return self._getitem_multilevel(key)
   2423         else:
-> 2424             return self._getitem_column(key)
   2425 
   2426     def _getitem_column(self, key):

~/anaconda3/envs/recnn/lib/python3.8/site-packages/modin/pandas/dataframe.py in _getitem_column(self, key)
   2426     def _getitem_column(self, key):
   2427         if key not in self.keys():
-> 2428             raise KeyError("{}".format(key))
   2429         s = DataFrame(
   2430             query_compiler=self._query_compiler.getitem_column_array([key])

KeyError: 'C'
```




Discussion:
I have a code like this
```
    user_dict = {}

    def app(x):
        userid = int(x.index[0])
        if userid not in user_dict:
            user_dict[userid]= {}
        for dict_name, column_name in [('items', 'movieId'), ('ratings', 'rating')]:
            if column_name in x.columns:
                user_dict[userid][dict_name] = x[column_name].values

    
    ratings.apply(app)
```
and I would like to use userId as an index and group by it
So that when app gets called, I get an index (userId) as well as some other column to process


One stupid workaraound I've sound is to copy the column and group by it:
```
import modin.pandas as pd
from pandas import util
df= util.testing.makeMixedDataFrame()
print(df.head())
df = df.to_numpy()
df = pd.DataFrame(df)
df.columns = ["A", "B", "C", "D"]
df['C_copy'] = df['C']
df = df.set_index("C")
df.groupby("C_copy") #  works fine but requires extra memory 

```
@awarebayes Thanks for posting this issue!

We have this working for the `MultiIndex` case but it is not working for other types of `Index` objects. We will fix this it should be relatively straightforward.

For your workaround, a slightly better way (until we fix the problem) would be as follows:

```python
import modin.pandas as pd
from pandas import util
df= util.testing.makeMixedDataFrame()
print(df.head())
df = df.to_numpy()
df = pd.DataFrame(df)
df.columns = ["A", "B", "C", "D"]
df.index = df["C"]
df.groupby("C")
```
Ran into the same issue today.
Have to say I'm so surprised by trying modin and getting crashes on just a plain and simple .groupby. 
I mean, what else would you use pandas if not for groupbying and making things? :)
>I mean, what else would you use pandas if not for groupbying and making things? :)

@diditforlulz273 It may surprise you but pandas has many other APIs which are common 😄 .

This was an easy fix, treating the index name as a column name for the purpose of groupby. Pandas is not consistent with how index names are treated vs column names.
The issue was only related to non-multiindex names, multiindex was already handled properly.
I reopened the issue because the fix doesn't fully resolve the problem.
```python
import modin.pandas as pd
import pandas
from pandas import util
df = util.testing.makeMixedDataFrame()
df = df.to_numpy()
pdf = pandas.DataFrame(df)
mdf = pd.DataFrame(df)
pdf.columns = ["A", "B", "C", "D"]
mdf.columns = ["A", "B", "C", "D"]
pdf = pdf.set_index("C")
mdf = mdf.set_index("C")
# data frames are equal so far
pdf
      A  B          D
C
foo1  0  0 2009-01-01
foo2  1  1 2009-01-02
foo3  2  0 2009-01-05
foo4  3  1 2009-01-06
foo5  4  0 2009-01-07

mdf
      A  B          D
C
foo1  0  0 2009-01-01
foo2  1  1 2009-01-02
foo3  2  0 2009-01-05
foo4  3  1 2009-01-06
foo5  4  0 2009-01-07

# However, number of groups is not equal
pdf.groupby("C").groups
{'foo1': ['foo1'], 'foo2': ['foo2'], 'foo3': ['foo3'], 'foo4': ['foo4'], 'foo5': ['foo5']

mdf.groupby("C").groups
{'C': ['foo1']}

# `sum` operation is performed incorrectly
pdf.groupby("C").sum()
        A    B
C
foo1  0.0  0.0
foo2  1.0  1.0
foo3  2.0  0.0
foo4  3.0  1.0
foo5  4.0  0.0

mdf.groupby("C").sum()
Empty DataFrame
Columns: []
Index: []
```
In the current [state](dd8b42317a224b1cc139a502b12b5da74087fb15) of master the original problem is observed:
```python
import modin.pandas as pd
from pandas import util
df= util.testing.makeMixedDataFrame()
print(df.head())
df = df.to_numpy()
df = pd.DataFrame(df)
df.columns = ["A", "B", "C", "D"]
df = df.set_index("C")
df.groupby("C") # KeyError
```

Traceback:
```bash
Traceback (most recent call last):
  File "test4.py", line 33, in <module>
    df.groupby("C")
  File "/localdisk/aprutsko/modin/modin/pandas/dataframe.py", line 460, in groupby
    by = self.__getitem__(by)._query_compiler
  File "/localdisk/aprutsko/modin/modin/pandas/base.py", line 2959, in __getitem__
    return self._getitem(key)
  File "/localdisk/aprutsko/modin/modin/pandas/dataframe.py", line 2973, in _getitem
    return self._getitem_column(key)
  File "/localdisk/aprutsko/modin/modin/pandas/dataframe.py", line 2439, in _getitem_column
    raise KeyError("{}".format(key))
KeyError: 'C'
```
Fix for the `KeyError` part of this problem is the following:
```diff
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
index 88f042b6..76d92c6d 100644
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -448,11 +448,7 @@ class DataFrame(BasePandasDataset):
         elif hashable(by) and not isinstance(by, pandas.Grouper):
             drop = by in self.columns
             idx_name = by
-            if (
-                self._query_compiler.has_multiindex(axis=axis)
-                and by in self._query_compiler.get_index_names(axis)
-                and by is not None
-            ):
+            if by in self._query_compiler.get_index_names(axis) and by is not None:
                 # In this case we pass the string value of the name through to the
                 # partitions. This is more efficient than broadcasting the values.
                 level, by = by, None
```
For some reason, this condition says that we can assume that the 'by' is an index level name only when this index is a `MultiIndex`:
https://github.com/modin-project/modin/blob/dd8b42317a224b1cc139a502b12b5da74087fb15/modin/pandas/dataframe.py#L451-L458
This isn't true obviously:
```python
>>> df1
   b  c
a
1  3  7
1  4  8
2  5  9
2  6  1
>>> type(df1.index)
<class 'pandas.core.indexes.numeric.Int64Index'>
>>> type(df)
<class 'pandas.core.frame.DataFrame'>
>>> df.groupby("a").sum()
    b   c
a
1   7  15
2  11  10
```
So, removing the requirement for an index to be a MultiIndex solves the `KeyError` part (all of the existing groupby tests are passing and the reproducer is working). 


Resolving patch:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -2661,7 +2661,7 @@ def is_reduce_fn(fn, deep_level=0):
                 by = [by]
         else:
             if not isinstance(by, list):
-                by = [by]
+                by = [by] if by is not None else []
             internal_by = [o for o in by if hashable(o) and o in self.columns]
             internal_qc = (
                 [self.getitem_column_array(internal_by)] if len(internal_by) else []
@@ -2755,7 +2755,7 @@ def compute_groupby(df, drop=False, partition_idx=0):
                 result_cols = result.columns
                 result.drop(columns=missmatched_cols, inplace=True, errors="ignore")
 
-                if not as_index:
+                if not as_index and len(by) > 0:
                     keep_index_levels = len(by) > 1 and any(
                         isinstance(x, pandas.CategoricalDtype)
                         for x in df[internal_by_cols].dtypes
diff --git a/modin/pandas/dataframe.py b/modin/pandas/dataframe.py
--- a/modin/pandas/dataframe.py
+++ b/modin/pandas/dataframe.py
@@ -448,11 +448,7 @@ def groupby(
         elif hashable(by) and not isinstance(by, pandas.Grouper):
             drop = by in self.columns
             idx_name = by
-            if (
-                self._query_compiler.has_multiindex(axis=axis)
-                and by in self._query_compiler.get_index_names(axis)
-                and by is not None
-            ):
+            if by is not None and by in self._query_compiler.get_index_names(axis):
                 # In this case we pass the string value of the name through to the
                 # partitions. This is more efficient than broadcasting the values.
                 level, by = by, None
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -16,7 +16,7 @@
 import numpy as np
 import pandas
 import pandas.core.groupby
-from pandas.core.dtypes.common import is_list_like
+from pandas.core.dtypes.common import is_list_like, is_numeric_dtype
 from pandas.core.aggregation import reconstruct_func
 from pandas._libs.lib import no_default
 import pandas.core.common as com
@@ -128,7 +128,7 @@ def sem(self, ddof=1):
         return self._default_to_pandas(lambda df: df.sem(ddof=ddof))
 
     def mean(self, *args, **kwargs):
-        return self._apply_agg_function(lambda df: df.mean(*args, **kwargs))
+        return self._apply_agg_function_check_index(lambda df: df.mean(*args, **kwargs))
 
     def any(self, **kwargs):
         return self._wrap_aggregation(
@@ -243,20 +243,18 @@ def _shift(data, periods, freq, axis, fill_value, is_set_nan_rows=True):
             )
             result = result.dropna(subset=self._by.columns).sort_index()
         else:
-            result = self._apply_agg_function(
+            result = self._apply_agg_function_check_index_name(
                 lambda df: df.shift(periods, freq, axis, fill_value)
             )
-            result._query_compiler.set_index_name(None)
         return result
 
     def nth(self, n, dropna=None):
         return self._default_to_pandas(lambda df: df.nth(n, dropna=dropna))
 
     def cumsum(self, axis=0, *args, **kwargs):
-        result = self._apply_agg_function(lambda df: df.cumsum(axis, *args, **kwargs))
-        # pandas does not name the index on cumsum
-        result._query_compiler.set_index_name(None)
-        return result
+        return self._apply_agg_function_check_index_name(
+            lambda df: df.cumsum(axis, *args, **kwargs)
+        )
 
     @property
     def indices(self):
@@ -271,15 +269,17 @@ def filter(self, func, dropna=True, *args, **kwargs):
         )
 
     def cummax(self, axis=0, **kwargs):
-        result = self._apply_agg_function(lambda df: df.cummax(axis, **kwargs))
-        # pandas does not name the index on cummax
-        result._query_compiler.set_index_name(None)
-        return result
+        return self._apply_agg_function_check_index_name(
+            lambda df: df.cummax(axis, **kwargs)
+        )
 
     def apply(self, func, *args, **kwargs):
         if not isinstance(func, BuiltinFunctionType):
             func = wrap_udf_function(func)
-        return self._apply_agg_function(lambda df: df.apply(func, *args, **kwargs))
+
+        return self._apply_agg_function_check_index(
+            lambda df: df.apply(func, *args, **kwargs)
+        )
 
     @property
     def dtypes(self):
@@ -405,10 +405,9 @@ def __getitem__(self, key):
         )
 
     def cummin(self, axis=0, **kwargs):
-        result = self._apply_agg_function(lambda df: df.cummin(axis=axis, **kwargs))
-        # pandas does not name the index on cummin
-        result._query_compiler.set_index_name(None)
-        return result
+        return self._apply_agg_function_check_index_name(
+            lambda df: df.cummin(axis=axis, **kwargs)
+        )
 
     def bfill(self, limit=None):
         return self._default_to_pandas(lambda df: df.bfill(limit=limit))
@@ -466,7 +465,7 @@ def try_get_str_func(fn):
                 **kwargs,
             )
         elif callable(func):
-            return self._apply_agg_function(
+            return self._apply_agg_function_check_index(
                 lambda grp, *args, **kwargs: grp.aggregate(func, *args, **kwargs),
                 *args,
                 **kwargs,
@@ -631,22 +630,21 @@ def ngroup(self, ascending=True):
         return self._default_to_pandas(lambda df: df.ngroup(ascending))
 
     def nunique(self, dropna=True):
-        return self._apply_agg_function(lambda df: df.nunique(dropna))
+        return self._apply_agg_function_check_index(lambda df: df.nunique(dropna))
 
     def resample(self, rule, *args, **kwargs):
         return self._default_to_pandas(lambda df: df.resample(rule, *args, **kwargs))
 
     def median(self, **kwargs):
-        return self._apply_agg_function(lambda df: df.median(**kwargs))
+        return self._apply_agg_function_check_index(lambda df: df.median(**kwargs))
 
     def head(self, n=5):
         return self._default_to_pandas(lambda df: df.head(n))
 
     def cumprod(self, axis=0, *args, **kwargs):
-        result = self._apply_agg_function(lambda df: df.cumprod(axis, *args, **kwargs))
-        # pandas does not name the index on cumprod
-        result._query_compiler.set_index_name(None)
-        return result
+        return self._apply_agg_function_check_index_name(
+            lambda df: df.cumprod(axis, *args, **kwargs)
+        )
 
     def __iter__(self):
         return self._iter.__iter__()
@@ -655,12 +653,9 @@ def cov(self):
         return self._default_to_pandas(lambda df: df.cov())
 
     def transform(self, func, *args, **kwargs):
-        result = self._apply_agg_function(
+        return self._apply_agg_function_check_index_name(
             lambda df: df.transform(func, *args, **kwargs)
         )
-        # pandas does not name the index on transform
-        result._query_compiler.set_index_name(None)
-        return result
 
     def corr(self, **kwargs):
         return self._default_to_pandas(lambda df: df.corr(**kwargs))
@@ -677,10 +672,9 @@ def fillna(self, **kwargs):
             squeeze=self._squeeze,
             **new_groupby_kwargs,
         )
-        result = work_object._apply_agg_function(lambda df: df.fillna(**kwargs))
-        # pandas does not name the index on fillna
-        result._query_compiler.set_index_name(None)
-        return result
+        return work_object._apply_agg_function_check_index_name(
+            lambda df: df.fillna(**kwargs)
+        )
 
     def count(self, **kwargs):
         result = self._wrap_aggregation(
@@ -721,7 +715,7 @@ def quantile(self, q=0.5, **kwargs):
         if is_list_like(q):
             return self._default_to_pandas(lambda df: df.quantile(q=q, **kwargs))
 
-        return self._apply_agg_function(lambda df: df.quantile(q, **kwargs))
+        return self._apply_agg_function_check_index(lambda df: df.quantile(q, **kwargs))
 
     def diff(self):
         return self._default_to_pandas(lambda df: df.diff())
@@ -914,6 +908,9 @@ def _wrap_aggregation(
         else:
             groupby_qc = self._query_compiler
 
+        if all(not is_numeric_dtype(dtype) for dtype in groupby_qc.dtypes):
+            numeric_only = False
+
         result = type(self._df)(
             query_compiler=qc_method(
                 groupby_qc,
@@ -930,6 +927,54 @@ def _wrap_aggregation(
             return result.squeeze()
         return result
 
+    def _apply_agg_function_check_index(self, *args, **kwargs):
+        """
+        Perform `self._apply_agg_function` with additional check.
+
+        Check the result of `self._apply_agg_function` on the need of resetting index.
+
+        Parameters
+        ----------
+        *args : list
+            Positional arguments to pass to `self._apply_agg_function`.
+        **kwargs : dict
+            Keyword arguments to pass to `self._apply_agg_function`.
+
+        Returns
+        -------
+        DataFrame
+        """
+        result = self._apply_agg_function(*args, **kwargs)
+        if self._by is None and not self._as_index:
+            # This is a workaround to align behavior with pandas. In this case pandas
+            # resets index, but Modin doesn't do that. More details are in https://github.com/modin-project/modin/issues/3716.
+            result.reset_index(drop=True, inplace=True)
+
+        return result
+
+    def _apply_agg_function_check_index_name(self, *args, **kwargs):
+        """
+        Perform `self._apply_agg_function` with additional check.
+
+        Check the result of `self._apply_agg_function` on the need of resetting index name.
+
+        Parameters
+        ----------
+        *args : list
+            Positional arguments to pass to `self._apply_agg_function`.
+        **kwargs : dict
+            Keyword arguments to pass to `self._apply_agg_function`.
+
+        Returns
+        -------
+        DataFrame
+        """
+        result = self._apply_agg_function(*args, **kwargs)
+        if self._by is not None:
+            # pandas does not name the index for this case
+            result._query_compiler.set_index_name(None)
+        return result
+
     def _apply_agg_function(self, f, *args, **kwargs):
         """
         Perform aggregation and combine stages based on a given function.


Test patch:
diff --git a/modin/pandas/test/test_groupby.py b/modin/pandas/test/test_groupby.py
--- a/modin/pandas/test/test_groupby.py
+++ b/modin/pandas/test/test_groupby.py
@@ -84,7 +84,7 @@ def test_mixed_dtypes_groupby(as_index):
     ]
 
     for by in by_values:
-        if by_values[0] == "col3":
+        if isinstance(by[0], str) and by[0] == "col3":
             modin_groupby = modin_df.set_index(by[0]).groupby(
                 by=by[0], as_index=as_index
             )
@@ -127,8 +127,6 @@ def test_mixed_dtypes_groupby(as_index):
 
         # TODO Add more apply functions
         apply_functions = [lambda df: df.sum(), min]
-        # Workaround for Pandas bug #34656. Recreate groupby object for Pandas
-        pandas_groupby = pandas_df.groupby(by=by[-1], as_index=as_index)
         for func in apply_functions:
             eval_apply(modin_groupby, pandas_groupby, func)
 
@@ -1216,11 +1214,25 @@ def eval_shift(modin_groupby, pandas_groupby):
         pandas_groupby,
         lambda groupby: groupby.shift(periods=-3),
     )
-    eval_general(
-        modin_groupby,
-        pandas_groupby,
-        lambda groupby: groupby.shift(axis=1, fill_value=777),
-    )
+
+    if isinstance(pandas_groupby, pandas.core.groupby.DataFrameGroupBy):
+        pandas_res = pandas_groupby.shift(axis=1, fill_value=777)
+        modin_res = modin_groupby.shift(axis=1, fill_value=777)
+        # Pandas produces unexpected index order (pandas GH 44269).
+        # Here we align index of Modin result with pandas to make test passed.
+        import pandas.core.algorithms as algorithms
+
+        indexer, _ = modin_res.index.get_indexer_non_unique(modin_res.index._values)
+        indexer = algorithms.unique1d(indexer)
+        modin_res = modin_res.take(indexer)
+
+        df_equals(modin_res, pandas_res)
+    else:
+        eval_general(
+            modin_groupby,
+            pandas_groupby,
+            lambda groupby: groupby.shift(axis=1, fill_value=777),
+        )
 
 
 def test_groupby_on_index_values_with_loop():
@@ -1717,3 +1729,14 @@ def test_not_str_by(by, as_index):
     eval_general(md_grp, pd_grp, lambda grp: grp.agg(lambda df: df.mean()))
     eval_general(md_grp, pd_grp, lambda grp: grp.dtypes)
     eval_general(md_grp, pd_grp, lambda grp: grp.first())
+
+
+def test_sum_with_level():
+    data = {
+        "A": ["0.0", "1.0", "2.0", "3.0", "4.0"],
+        "B": ["0.0", "1.0", "0.0", "1.0", "0.0"],
+        "C": ["foo1", "foo2", "foo3", "foo4", "foo5"],
+        "D": pandas.bdate_range("1/1/2009", periods=5),
+    }
+    modin_df, pandas_df = pd.DataFrame(data), pandas.DataFrame(data)
+    eval_general(modin_df, pandas_df, lambda df: df.set_index("C").groupby("C").sum())