Repository: modin-project/modin

Issue:
`GroupBy.groups/.indices` ignores dropna=False as kwarg
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: MacOS 11.5.2
- **Modin version** (`modin.__version__`): 0.10.0 and 0.10.2
- **Python version**: 3.8.10

### Describe the problem

Using groupby, the modin implementation ignores the parameter dropna.
Ad-hoc, the property "_index_grouped" could be the problem, as it calls `groupby(by)` only with the parameter by, but not with any other kwargs. [Here is the line reference to this property.](https://github.com/modin-project/modin/blob/c3b8d7e4863615f99c8a1c6b7316e680daaa4393/modin/pandas/groupby.py#L767)

The code below can reproduce this issue. As additional information: Using `pandas_grouped.groups` fails with this error: `ValueError: Categorical categories cannot be null`, but iterating over the groups still works.

### Source code / logs

#### Code to reproduce

```python
import modin
import numpy as np
import pandas

def print_name_and_groups(title, grouped):
    print(title)
    for name, group in grouped:
        print("name:\n", name)
        print("group:\n", group)
        print()

data = {"a": [1, 2, 3, 4], "b": [11, np.nan, 13, np.nan]}

df = modin.pandas.DataFrame(data)
modin_grouped = df.groupby("b", dropna=False)
print_name_and_groups("modin groups:", modin_grouped)

df = pandas.DataFrame(data)
pandas_grouped = df.groupby("b", dropna=False)
print_name_and_groups("pandas groups:", pandas_grouped
```

#### Output
```bash
modin groups:
name:
 11.0
group:
    a     b
0  1  11.0

name:
 13.0
group:
    a     b
2  3  13.0

pandas groups:
name:
 11.0
group:
    a     b
0  1  11.0

name:
 13.0
group:
    a     b
2  3  13.0

name:
 nan
group:
    a   b
1  2 NaN
3  4 NaN
```





Discussion:
Hi @ahallermed thanks for the report! I see the problem reproducible with `groups` but not with the builtin groupby functions like `sum`, `count`, `mean`, etc.

>Ad-hoc, the property "_index_grouped" could be the problem, as it calls groupby(by) only with the parameter by, but not with any other kwargs

Looking at the code, I agree and think this is right. Since you've already dove into the code, let me know if you want to contribute the patch to fix this. Thanks again for the detailed report!
Hello @devin-petersohn,
thanks to you for the fast reply.

I had a closer look at the code and especially the tests and the latter kind of overwhelmed me.

I fear the side-effects, the project setup for testing and setting up a test that serves your compliance. But I am willing to learn from your changes and get more involved one after another. Therefore, I want to give you a proposal that may help you do a fix.

## In groupby.py
My suggestion would be, to replace `groupby(by)` with `groupby(by, **self._kwargs)` in the following lines:
[line 821](https://github.com/modin-project/modin/blob/c3b8d7e4863615f99c8a1c6b7316e680daaa4393/modin/pandas/groupby.py#L821)
[line 838](https://github.com/modin-project/modin/blob/c3b8d7e4863615f99c8a1c6b7316e680daaa4393/modin/pandas/groupby.py#L838)
[line 840](https://github.com/modin-project/modin/blob/c3b8d7e4863615f99c8a1c6b7316e680daaa4393/modin/pandas/groupby.py#L840)

I am not deep enough in the code to say, whether one needs to additionally add `self._squeeze` like in [line 316](https://github.com/modin-project/modin/blob/c3b8d7e4863615f99c8a1c6b7316e680daaa4393/modin/pandas/groupby.py#L316)

## In test_groupby.py
For the test, I would suggest to use the reproduction code (see above) as a minimalistic test case, e.g.:

```python
def test_dropna_parameter_is_used:
    data = {"a": [1, 2, 3, 4], "b": [11, np.nan, 13, np.nan]}

    modin_df = modin.pandas.DataFrame(data)
    modin_groupby = modin_df.groupby("b", dropna=False)

    pandas_df = pandas.DataFrame(data)
    pandas_groupby = pandas_df.groupby("b", dropna=False)

    assert 3 == pandas.groupby.ngroups
    assert modin_groupby.ngroups == pandas.groupby.ngroups
    eval_groups(modin_groupby, pandas_groupby)
```

There is already a test for `def test_groupby_multiindex(groupby_kwargs):` on line 1182, but it is not with nan and not with the parameter dropna and only for multiindex.

I am curious how the fix will actually be implemented and tested. Thank you in advance!


Thanks @ahallermed!

Please feel free to put those changed in as a pull request and we can take the changes from there. It would be great to be able to attribute the changes to the one who did the work! We will make any necessary changes on that PR once it is in to ensure that it works with the other components of `groupby`.

If you put in a PR just make sure you take a quick look at the contributing docs on how to format the commit message and sign the commit with `git commit -s` (outlined here: https://modin.readthedocs.io/en/latest/contributing.html)
Hello @devin-petersohn, thanks for helping me about how to do a pull request. I'll do one next week. 

Resolving patch:
diff --git a/modin/pandas/groupby.py b/modin/pandas/groupby.py
--- a/modin/pandas/groupby.py
+++ b/modin/pandas/groupby.py
@@ -846,6 +846,9 @@ def _compute_index_grouped(self, numerical=False):
         --------
         pandas.core.groupby.GroupBy.groups
         """
+        # We end up using pure pandas to compute group indices, so raising a warning
+        ErrorMessage.default_to_pandas("Group indices computation")
+
         # Splitting level-by and column-by since we serialize them in a different ways
         by = None
         level = []
@@ -866,6 +869,8 @@ def _compute_index_grouped(self, numerical=False):
             by = self._by
 
         is_multi_by = self._is_multi_by or (by is not None and len(level) > 0)
+        # `dropna` param is the only one that matters for the group indices result
+        dropna = self._kwargs.get("dropna", True)
 
         if hasattr(self._by, "columns") and is_multi_by:
             by = list(self._by.columns)
@@ -875,7 +880,6 @@ def _compute_index_grouped(self, numerical=False):
             # end up using pandas implementation. Add the warning so the user is
             # aware.
             ErrorMessage.catch_bugs_and_request_email(self._axis == 1)
-            ErrorMessage.default_to_pandas("Groupby with multiple columns")
             if isinstance(by, list) and all(
                 is_label(self._df, o, self._axis) for o in by
             ):
@@ -886,7 +890,7 @@ def _compute_index_grouped(self, numerical=False):
                 by = try_cast_to_pandas(by, squeeze=True)
                 pandas_df = self._df._to_pandas()
             by = wrap_into_list(by, level)
-            groupby_obj = pandas_df.groupby(by=by)
+            groupby_obj = pandas_df.groupby(by=by, dropna=dropna)
             return groupby_obj.indices if numerical else groupby_obj.groups
         else:
             if isinstance(self._by, type(self._query_compiler)):
@@ -908,7 +912,13 @@ def _compute_index_grouped(self, numerical=False):
                 # Since we want positional indices of the groups, we want to group
                 # on a `RangeIndex`, not on the actual index labels
                 axis_labels = pandas.RangeIndex(len(axis_labels))
-            return axis_labels.groupby(by)
+            # `pandas.Index.groupby` doesn't take any parameters except `by`.
+            # Have to convert an Index to a Series to be able to process `dropna=False`:
+            if dropna:
+                return axis_labels.groupby(by)
+            else:
+                groupby_obj = axis_labels.to_series().groupby(by, dropna=dropna)
+                return groupby_obj.indices if numerical else groupby_obj.groups
 
     def _wrap_aggregation(
         self, qc_method, default_func, drop=True, numeric_only=True, **kwargs


Test patch:
diff --git a/modin/pandas/test/test_groupby.py b/modin/pandas/test/test_groupby.py
--- a/modin/pandas/test/test_groupby.py
+++ b/modin/pandas/test/test_groupby.py
@@ -30,6 +30,7 @@
     generate_multiindex,
     test_groupby_data,
     dict_equals,
+    value_equals,
 )
 from modin.config import NPartitions
 
@@ -45,7 +46,7 @@ def modin_groupby_equals_pandas(modin_groupby, pandas_groupby):
     )
 
     for g1, g2 in itertools.zip_longest(modin_groupby, pandas_groupby):
-        assert g1[0] == g2[0]
+        value_equals(g1[0], g2[0])
         df_equals(g1[1], g2[1])
 
 
@@ -1350,6 +1351,74 @@ def test_groupby_multiindex(groupby_kwargs):
     df_equals(md_grp.first(), pd_grp.first())
 
 
+@pytest.mark.parametrize("dropna", [True, False])
+@pytest.mark.parametrize(
+    "groupby_kwargs",
+    [
+        pytest.param({"level": 1, "axis": 1}, id="level_idx_axis=1"),
+        pytest.param({"level": 1}, id="level_idx"),
+        pytest.param({"level": [1, "four"]}, id="level_idx+name"),
+        pytest.param({"by": "four"}, id="level_name"),
+        pytest.param({"by": ["one", "two"]}, id="level_name_multi_by"),
+        pytest.param(
+            {"by": ["item0", "one", "two"]},
+            id="col_name+level_name",
+        ),
+        pytest.param({"by": ["item0"]}, id="col_name"),
+        pytest.param(
+            {"by": ["item0", "item1"]},
+            id="col_name_multi_by",
+        ),
+    ],
+)
+def test_groupby_with_kwarg_dropna(groupby_kwargs, dropna):
+    modin_df = pd.DataFrame(test_data["float_nan_data"])
+    pandas_df = pandas.DataFrame(test_data["float_nan_data"])
+
+    new_index = pandas.Index([f"item{i}" for i in range(len(pandas_df))])
+    new_columns = pandas.MultiIndex.from_tuples(
+        [(i // 4, i // 2, i) for i in range(len(modin_df.columns))],
+        names=["four", "two", "one"],
+    )
+    modin_df.columns = new_columns
+    modin_df.index = new_index
+    pandas_df.columns = new_columns
+    pandas_df.index = new_index
+
+    if groupby_kwargs.get("axis", 0) == 0:
+        modin_df = modin_df.T
+        pandas_df = pandas_df.T
+
+    md_grp, pd_grp = modin_df.groupby(
+        **groupby_kwargs, dropna=dropna
+    ), pandas_df.groupby(**groupby_kwargs, dropna=dropna)
+    modin_groupby_equals_pandas(md_grp, pd_grp)
+
+    by_kwarg = groupby_kwargs.get("by", [])
+    # Disabled because of broken `dropna=False` for MapReduce implemented aggs:
+    # https://github.com/modin-project/modin/issues/3817
+    if not (
+        not dropna
+        and len(by_kwarg) > 1
+        and any(col in modin_df.columns for col in by_kwarg)
+    ):
+        df_equals(md_grp.sum(), pd_grp.sum())
+        df_equals(md_grp.size(), pd_grp.size())
+    # Grouping on level works incorrect in case of aggregation:
+    # https://github.com/modin-project/modin/issues/2912
+    # "BaseOnPython" tests are disabled because of the bug:
+    # https://github.com/modin-project/modin/issues/3827
+    if get_current_execution() != "BaseOnPython" and any(
+        col in modin_df.columns for col in by_kwarg
+    ):
+        df_equals(md_grp.quantile(), pd_grp.quantile())
+    # Default-to-pandas tests are disabled for multi-column 'by' because of the bug:
+    # https://github.com/modin-project/modin/issues/3827
+    if not (not dropna and len(by_kwarg) > 1):
+        df_equals(md_grp.first(), pd_grp.first())
+        df_equals(md_grp._default_to_pandas(lambda df: df.sum()), pd_grp.sum())
+
+
 @pytest.mark.parametrize("groupby_axis", [0, 1])
 @pytest.mark.parametrize("shift_axis", [0, 1])
 def test_shift_freq(groupby_axis, shift_axis):
diff --git a/modin/pandas/test/utils.py b/modin/pandas/test/utils.py
--- a/modin/pandas/test/utils.py
+++ b/modin/pandas/test/utils.py
@@ -1353,12 +1353,16 @@ def _make_default_file(filename=None, nrows=NROWS, ncols=2, force=True, **kwargs
     return _make_default_file, filenames
 
 
+def value_equals(obj1, obj2):
+    """Check wherher two scalar or list-like values are equal and raise an ``AssertionError`` if they aren't."""
+    if is_list_like(obj1):
+        np.testing.assert_array_equal(obj1, obj2)
+    else:
+        assert (obj1 == obj2) or (np.isnan(obj1) and np.isnan(obj2))
+
+
 def dict_equals(dict1, dict2):
     """Check whether two dictionaries are equal and raise an ``AssertionError`` if they aren't."""
     for key1, key2 in itertools.zip_longest(sorted(dict1), sorted(dict2)):
-        assert (key1 == key2) or (np.isnan(key1) and np.isnan(key2))
-        value1, value2 = dict1[key1], dict2[key2]
-        if is_list_like(value1):
-            np.testing.assert_array_equal(value1, value2)
-        else:
-            assert value1 == value2
+        value_equals(key1, key2)
+        value_equals(dict1[key1], dict2[key2])