Repository: modin-project/modin

Issue:
GroupBy with `sort=False` parameter and categorical `by` produces incorrect aggregation results
### System information
- **Ubuntu 20.04**:
- **Modin version** 0.11.1:
- **Python version**: 3.7
- **Code we can use to reproduce**:

Create some random data with a few entries in it

```python
import pandas as pd
import numpy as np

data = ["a"] * 50000 + ["b"] * 10000 + ["c"] * 1000
np.random.seed(42)
data = np.array(data)
np.random.shuffle(data)
df = pd.DataFrame({"some_column": data})
df.to_csv("some_data.csv")
```

Read the resulting csv back with modin

```python
from modin import pandas as md
mdf = md.read_csv("some_data.csv", dtype={"some_column": "category"})
mdf["some_column"].value_counts()
```

This results in an output like:

```
a    40153
b    18726
c     2121
name: some_column, dtype: int64
```

but should return 

```
some_column
a              50000
b              10000
c               1000
dtype: int64
```



Discussion:
I should note: The output is correct, if you **don't** use `dtype: {"some_column": "category"}` so i would guess the problem lies with multiple processes attaching new values to categories during read or something like this
Hi @andreas-kopecky thanks for the report! I modified your comment to format the code with syntax highlighting, I hope that's okay!

This looks like a bug with `value_counts`. When I do a `groupby.count` I get the correct values:

```python
mdf.groupby("some_column").count()
```

This gives:

```
             Unnamed: 0
some_column
a                 50000
b                 10000
c                  1000
```

@dchigarev Will you be able to take a look at the `value_counts` implementation?
Another note: The semantically equivalent (`groupby("some_column")["some_column"].count`) is also returning the correct result.

```python
mdf.groupby("some_column")["some_column"].count()
```

Shows:

```
some_column
a    50000
b    10000
c     1000
Name: some_column, dtype: int64
```
I'll take a look
> This looks like a bug with value_counts. When I do a groupby.count I get the correct values:
`>>> mdf.groupby("some_column").count()`

However, doing the same, but with `sort=False` parameter breaks everything:
```python
>> mdf.groupby("some_column", sort=False).count()
             Unnamed: 0
some_column
a                 39924
b                 20076
c                  1000
>>> mdf.groupby("some_column", sort=True).count()
             Unnamed: 0
some_column
a                 50000
b                 10000
c                  1000
```
`value_counts` actually passes `sort=False` to not do unnecessary sorting: 
https://github.com/modin-project/modin/blob/0f944a19777947277b45db8e779c1566e17f9336/modin/pandas/base.py#L2889

So the essential problem is hidden in our GroupBy implementation, but suddenly, the real source of the problem is pandas. 

Let's emulate Modin's `groupby(sort=False).count` flow with pandas:
```python
>>> part1 = pandas.DataFrame({
...     "by_col": ["a", "a", "b", "c"],
...     "some_col": [1, 2, 3, 4],
... }).astype({"by_col": "category"})
>>> part2 = pandas.DataFrame({
...     "by_col": ["b", "a", "a", "c"],
...     "some_col": [1, 2, 3, 4],
... }).astype({"by_col": "category"})
# Map phase: applying an aggregation function to every partition
>>> grp_res_part1 = part1.groupby("by_col", sort=False).count()
>>> grp_res_part2 = part2.groupby("by_col", sort=False).count()
>>> grp_res_part1
        some_col
by_col
a              2  # 2 elements in 'a' group in the first partition
b              1
c              1
>>> grp_res_part2
        some_col
by_col
b              1
a              2 # 2 elements in 'a' group in the second partition
c              1
# Concatenating partition's results to proceed to the reduce phase:
>>> pandas.concat([grp_res_part1, grp_res_part2])
        some_col
by_col
a              2 # still 2 elements for 'a' in the first partition
b              1
c              1
a              1 # but only 1 element for 'a'?
b              2 # and 2 for 'b'?...
c              1
```
So the data is spoiled by `pandas.concat` when building an axis partition for the reduced phase of groupby. The actual location of this breaking concat is [`PandasFrameAxisPartition.deploy_axis_func`](https://github.com/modin-project/modin/blob/0f944a19777947277b45db8e779c1566e17f9336/modin/core/dataframe/pandas/partitioning/axis_partition.py#L302). The data is spoiled because of a bug in `pandas.concat` that occurs when you're trying to concatenate frames with categorical indices that have the same values but different order. I've filled an issue about this in pandas repo (pandas-dev/pandas#44099).

It seems that there are two possible solutions on Modin side:
1. Wait until pandas fix the bug
2. Freeze `sort=True` parameter at the [map phase of groupby](https://github.com/modin-project/modin/blob/0f944a19777947277b45db8e779c1566e17f9336/modin/core/dataframe/algebra/groupby_function.py#L139-L141) flow in order to avoid concatenating partitions with different index orders.

@devin-petersohn what do you think we should pick?
I do not want to wait on pandas to fix this, how does the `sort=False` order compare to pandas order? If we sort in the map phase, it's going to end up being out of order compared to pandas, correct?
Right, this will break the pandas' unsorted order.

When `sort=False` pandas arrange group names in the result in the same order as they appear in the source frame:
```python
>>> pandas.DataFrame({"col": [10, 0, 5, 0, 10, 4, 4]}).groupby("col", sort=False)["col"].count()
col
10    2
0     2
5     1
4     2
Name: col, dtype: int64
```
I originally thought that the order is random in the case of `sort=False`, so sorted Modin's output would fit for that contract, but now it's clear that it doesn't.  
Using `value_counts` with categorical data seems like a more common use-case rather than the `groupby(sort=False)`. So, in order to fix the common case at first, we can solve the derivative of the source problem by just replacing `sort=False` with `sort=True` at the `value_counts`, this way we will fix the common case sacrificing performance to the correct behavior (`groupby(sort=False)` would still returning incorrect results though, so I'm not sure whether we want such fix).
 @dchigarev thanks a ton for explaining this bug and also provide a workaround for now. 
@dchigarev Do we want to get the `sort=True` fix in place for this release?

Resolving patch:
diff --git a/modin/core/dataframe/algebra/groupby.py b/modin/core/dataframe/algebra/groupby.py
--- a/modin/core/dataframe/algebra/groupby.py
+++ b/modin/core/dataframe/algebra/groupby.py
@@ -18,6 +18,7 @@
 from .map_reduce import MapReduce
 from .default2pandas.groupby import GroupBy
 from modin.utils import try_cast_to_pandas, hashable
+from modin.error_message import ErrorMessage
 
 
 class GroupByReduce(MapReduce):
@@ -290,6 +291,21 @@ def caller(
             )
         assert axis == 0, "Can only groupby reduce with axis=0"
 
+        # The bug only occurs in the case of Categorical 'by', so we might want to check whether any of
+        # the 'by' dtypes is Categorical before going into this branch, however triggering 'dtypes'
+        # computation if they're not computed may take time, so we don't do it
+        if not groupby_args.get("sort", True) and isinstance(by, type(query_compiler)):
+            ErrorMessage.missmatch_with_pandas(
+                operation="df.groupby(categorical_by, sort=False)",
+                message=(
+                    "the groupby keys will be sorted anyway, although the 'sort=False' was passed. "
+                    "See the following issue for more details: "
+                    "https://github.com/modin-project/modin/issues/3571"
+                ),
+            )
+            groupby_args = groupby_args.copy()
+            groupby_args["sort"] = True
+
         if numeric_only:
             qc = query_compiler.getitem_column_array(
                 query_compiler._modin_frame.numeric_columns(True)
diff --git a/modin/pandas/base.py b/modin/pandas/base.py
--- a/modin/pandas/base.py
+++ b/modin/pandas/base.py
@@ -2894,7 +2894,7 @@ def value_counts(
     ):
         if subset is None:
             subset = self._query_compiler.columns
-        counted_values = self.groupby(by=subset, sort=False, dropna=dropna).size()
+        counted_values = self.groupby(by=subset, dropna=dropna, observed=True).size()
         if sort:
             counted_values.sort_values(ascending=ascending, inplace=True)
         if normalize:


Test patch:
diff --git a/modin/pandas/test/dataframe/test_reduction.py b/modin/pandas/test/dataframe/test_reduction.py
--- a/modin/pandas/test/dataframe/test_reduction.py
+++ b/modin/pandas/test/dataframe/test_reduction.py
@@ -408,3 +408,15 @@ def comparator(md_res, pd_res):
         # visit modin-issue#3388 for more info.
         check_exception_type=None if sort and ascending is None else True,
     )
+
+
+def test_value_counts_categorical():
+    # from issue #3571
+    data = np.array(["a"] * 50000 + ["b"] * 10000 + ["c"] * 1000)
+    random_state = np.random.RandomState(seed=42)
+    random_state.shuffle(data)
+
+    eval_general(
+        *create_test_dfs({"col1": data, "col2": data}, dtype="category"),
+        lambda df: df.value_counts(),
+    )
diff --git a/modin/pandas/test/test_groupby.py b/modin/pandas/test/test_groupby.py
--- a/modin/pandas/test/test_groupby.py
+++ b/modin/pandas/test/test_groupby.py
@@ -1731,6 +1731,32 @@ def test_not_str_by(by, as_index):
     eval_general(md_grp, pd_grp, lambda grp: grp.first())
 
 
+@pytest.mark.parametrize("sort", [True, False])
+@pytest.mark.parametrize("is_categorical_by", [True, False])
+def test_groupby_sort(sort, is_categorical_by):
+    # from issue #3571
+    by = np.array(["a"] * 50000 + ["b"] * 10000 + ["c"] * 1000)
+    random_state = np.random.RandomState(seed=42)
+    random_state.shuffle(by)
+
+    data = {"key_col": by, "data_col": np.arange(len(by))}
+    md_df, pd_df = create_test_dfs(data)
+
+    if is_categorical_by:
+        md_df = md_df.astype({"key_col": "category"})
+        pd_df = pd_df.astype({"key_col": "category"})
+
+    md_grp = md_df.groupby("key_col", sort=sort)
+    pd_grp = pd_df.groupby("key_col", sort=sort)
+
+    modin_groupby_equals_pandas(md_grp, pd_grp)
+    eval_general(md_grp, pd_grp, lambda grp: grp.sum())
+    eval_general(md_grp, pd_grp, lambda grp: grp.size())
+    eval_general(md_grp, pd_grp, lambda grp: grp.agg(lambda df: df.mean()))
+    eval_general(md_grp, pd_grp, lambda grp: grp.dtypes)
+    eval_general(md_grp, pd_grp, lambda grp: grp.first())
+
+
 def test_sum_with_level():
     data = {
         "A": ["0.0", "1.0", "2.0", "3.0", "4.0"],
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -3439,6 +3439,18 @@ def sort_sensitive_comparator(df1, df2):
     )
 
 
+def test_value_counts_categorical():
+    # from issue #3571
+    data = np.array(["a"] * 50000 + ["b"] * 10000 + ["c"] * 1000)
+    random_state = np.random.RandomState(seed=42)
+    random_state.shuffle(data)
+
+    eval_general(
+        *create_test_series(data, dtype="category"),
+        lambda df: df.value_counts(),
+    )
+
+
 @pytest.mark.parametrize("data", test_data_values, ids=test_data_keys)
 def test_values(data):
     modin_series, pandas_series = create_test_series(data)