Repository: pandas-dev/pandas

Issue:
groupby aggregation on ordered Categorial with 'observed=True' breaks order
#### Code Sample:

```python
import pandas as pd

# Create a DataFrame with an ordered categorical column, one category not present
df = pd.DataFrame(
            dict(cat = pd.Series([3, 1, 2, 1, 3, 2], 
                                 dtype=pd.CategoricalDtype(
                                              categories=[1, 2, 3, 4], 
                                              ordered=True)
                                ), 
                 val = pd.Series([1.5, 0.5, 1.0, 0.5, 1.5, 1.0])
            )
)
```
**Including unobserved categories gives correct groups:**
```python
# Sum 'val' grouped by 'cat', including unobserved categories
df.groupby('cat', observed=False)['val'].agg('sum')
```
```
cat
1    1.0
2    2.0
3    3.0
4    0.0
Name: val, dtype: float64
```
**Excluding unobserved categories changes the order, groups are wrong:**
```python
# Sum 'val' grouped by 'cat', excluding unobserved categories
df.groupby('cat', observed=True)['val'].agg('sum')
```
```
cat
3    1.0
1    2.0
2    3.0
Name: val, dtype: float64
```

#### Problem description:
The sample code shows that grouping with an ordered factor does not respect the factor's order when ```'observed=True'``` is set. Instead, group labels are in order of first occurrence in the Categorical, _as if it were unordered_. The aggregation results, however, are in the Categorical's order. 
Thus, the result is wrong.

**Related issues:** #25167
There, the Categorical was unordered, and the ```sort=True``` parameter did not work as expected in combination with ```observed=True```. In my case, ```sort``` makes no difference:
```python
df.groupby('cat', observed=True, sort=True)['val'].agg('sum')
df.groupby('cat', observed=True, sort=False)['val'].agg('sum')
```
both give the same, wrong result as shown above.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 3.7.2.final.0
python-bits: 64
OS: Darwin
OS-release: 18.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.24.2
pytest: 4.3.0
pip: 19.0.3
setuptools: 40.8.0
Cython: 0.29.6
numpy: 1.16.2
scipy: 1.2.1
pyarrow: None
xarray: 0.11.3
IPython: 7.1.1
sphinx: 1.8.5
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2018.9
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.9
feather: None
matplotlib: 3.0.3
openpyxl: 2.6.1
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.1.5
lxml.etree: 4.3.2
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: 1.3.1
pymysql: None
psycopg2: 2.7.6.1 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: 0.2.1
pandas_gbq: None
pandas_datareader: None
gcsfs: None
</details>


Discussion:
Unless I am overlooking something this looks to be a duplicate of the issue you've linked to. As such closing this and let's stick to that one
Fair enough.  
The scenario is not identical, though (ordered Categorical vs. ‚sort=True‘).
Looking at the code in grouper.py, it certainly seems to stem from the same place, but the ‚ordered‘ attribute and the ‚sort‘ parameter are separate concepts there, as well.
Maybe make this another unit Test?

A good test should be able to parametrize all of those parameters - if you'd like to take a look at the other issue and post a PR that would certainly be welcome!
Right, I will try. Paid work keeps intruding, though...  
## I have digged further, and found that this issue is **_not_** a duplicate of #25167:

### The fix for #25167 does not fix this one completely
The PR #25173 fixes #25167. So, I applied the two code changes from that PR to my local installation (files **grouper.py** and **categorical.py** in **pandas/core/groupby/**).
**This fixes my test case, but only if the 'sort' parameter is set to True**. If ```sort=False```, the bug persists (same result as shown above):
```python
df.groupby('cat', observed=True, sort=False)['val'].agg('sum')
```
still returns
```
cat
3    1.0
1    2.0
2    3.0
Name: val, dtype: float64
```
after installing PR #25173.

### If I modify it a bit, PR #25173 also fixes my problem
The relevant change from PR #25173 that fixes my ```sort=True``` case are these two new lines in **grouper.py** (inserted after line 302):
```python
                    if sort:
                        codes = np.sort(codes)
```
If I extend the condition and also check for the grouper being ordered, the ```sort=False``` case also works:
```python
                    if sort or self.grouper.ordered:
                        codes = np.sort(codes)
```
(This implies that grouping by an ordered Categorical and setting ```sort=True``` should result in the same ordering of codes. Also, that grouping by an ordered Categorical always sorts.)

**With this modification, my test case seems OK both for ```sort=True```and ```sort=False```**.

## How to proceed
If my modification is OK (as in, breaks no tests and fits into Things), I propose to 

- either reopen #25173 and add my modification
- or reopen this issue, and fix it as shown above
Thanks, @WillAyd. Will you do the change, or is this expected from me?
When I clone the repository, I somehow do not see the changed sources of PR #25173, even though it was closed 7 days ago. It's probably me doing something wrong, but it makes me wary of trying this myself.

(Besides, as mentioned, Paid Work)
If you could push a PR I can take a look on the review side. Be sure to check out the contributing guide if you have trouble and you an ask specific development questions on Gitter:

https://pandas.pydata.org/pandas-docs/stable/development/contributing.html

> (Besides, as mentioned, Paid Work)

Pandas is for all practical purposes maintained entirely by volunteers - any help you can add to that is certainly welcome!

Resolving patch:
diff --git a/pandas/core/groupby/grouper.py b/pandas/core/groupby/grouper.py
--- a/pandas/core/groupby/grouper.py
+++ b/pandas/core/groupby/grouper.py
@@ -301,6 +301,8 @@ def __init__(self, index, grouper=None, obj=None, name=None, level=None,
                 if observed:
                     codes = algorithms.unique1d(self.grouper.codes)
                     codes = codes[codes != -1]
+                    if sort or self.grouper.ordered:
+                        codes = np.sort(codes)
                 else:
                     codes = np.arange(len(categories))
 


Test patch:
diff --git a/pandas/tests/groupby/test_categorical.py b/pandas/tests/groupby/test_categorical.py
--- a/pandas/tests/groupby/test_categorical.py
+++ b/pandas/tests/groupby/test_categorical.py
@@ -451,6 +451,38 @@ def test_dataframe_categorical_with_nan(observed):
     tm.assert_frame_equal(result, expected)
 
 
+@pytest.mark.parametrize("ordered", [True, False])
+@pytest.mark.parametrize("observed", [True, False])
+@pytest.mark.parametrize("sort", [True, False])
+def test_dataframe_categorical_ordered_observed_sort(ordered, observed, sort):
+    # GH 25871: Fix groupby sorting on ordered Categoricals
+    # GH 25167: Groupby with observed=True doesn't sort
+
+    # Build a dataframe with cat having one unobserved category ('missing'),
+    # and a Series with identical values
+    label = pd.Categorical(['d', 'a', 'b', 'a', 'd', 'b'],
+                           categories=['a', 'b', 'missing', 'd'],
+                           ordered=ordered)
+    val = pd.Series(['d', 'a', 'b', 'a', 'd', 'b'])
+    df = pd.DataFrame({'label': label, 'val': val})
+
+    # aggregate on the Categorical
+    result = (df.groupby('label', observed=observed, sort=sort)['val']
+                .aggregate('first'))
+
+    # If ordering works, we expect index labels equal to aggregation results,
+    # except for 'observed=False': label 'missing' has aggregation None
+    label = pd.Series(result.index.array, dtype='object')
+    aggr = pd.Series(result.array)
+    if not observed:
+        aggr[aggr.isna()] = 'missing'
+    if not all(label == aggr):
+        msg = ('Labels and aggregation results not consistently sorted\n' +
+               'for (ordered={}, observed={}, sort={})\n' +
+               'Result:\n{}').format(ordered, observed, sort, result)
+        assert False, msg
+
+
 def test_datetime():
     # GH9049: ensure backward compatibility
     levels = pd.date_range('2014-01-01', periods=4)
diff --git a/pandas/tests/groupby/test_grouping.py b/pandas/tests/groupby/test_grouping.py
--- a/pandas/tests/groupby/test_grouping.py
+++ b/pandas/tests/groupby/test_grouping.py
@@ -253,28 +253,27 @@ def test_groupby_levels_and_columns(self):
         tm.assert_frame_equal(by_levels, by_columns)
 
     def test_groupby_categorical_index_and_columns(self, observed):
-        # GH18432
+        # GH18432, adapted for GH25871
         columns = ['A', 'B', 'A', 'B']
         categories = ['B', 'A']
-        data = np.ones((5, 4), int)
+        data = np.array([[1, 2, 1, 2],
+                         [1, 2, 1, 2],
+                         [1, 2, 1, 2],
+                         [1, 2, 1, 2],
+                         [1, 2, 1, 2]], int)
         cat_columns = CategoricalIndex(columns,
                                        categories=categories,
                                        ordered=True)
         df = DataFrame(data=data, columns=cat_columns)
         result = df.groupby(axis=1, level=0, observed=observed).sum()
-        expected_data = 2 * np.ones((5, 2), int)
-
-        if observed:
-            # if we are not-observed we undergo a reindex
-            # so need to adjust the output as our expected sets us up
-            # to be non-observed
-            expected_columns = CategoricalIndex(['A', 'B'],
-                                                categories=categories,
-                                                ordered=True)
-        else:
-            expected_columns = CategoricalIndex(categories,
-                                                categories=categories,
-                                                ordered=True)
+        expected_data = np.array([[4, 2],
+                                  [4, 2],
+                                  [4, 2],
+                                  [4, 2],
+                                  [4, 2]], int)
+        expected_columns = CategoricalIndex(categories,
+                                            categories=categories,
+                                            ordered=True)
         expected = DataFrame(data=expected_data, columns=expected_columns)
         assert_frame_equal(result, expected)