Repository: pandas-dev/pandas

Issue:
BUG: sort=False ignored when grouping with a categorical column
Hello everyone, 

I stumbled upon the following behavior of groubpy with categorical, which seems at least inconsistent with the way groupby usually operates.

When grouping on a string type column with `sort=False`, the order of the groups is the order in which the keys first appear in the column.

However, when grouping with a categorical column, the groups seem to be always ordered by the categorical, even when `sort=False`.

``` python
import pandas as pd
d = {'foo': [10, 8, 5, 6, 4, 1, 7], 'bar': [10, 20, 30, 40, 50, 60, 70],
     'baz': ['d', 'c', 'e', 'a', 'a', 'd', 'c']}
df = pd.DataFrame(d)
cat = pd.cut(df['foo'], np.linspace(0, 10, 5))
df['range'] = cat
groups = df.groupby('range', sort=True)
# Expected behaviour
result = groups.agg('mean')

# Why are the categorical still sorted in this case ?
groups2 = df.groupby('range', sort=False)
result2 = groups2.agg('mean')

# I would expect an output like this one: keep the order in which the groups
# are first encountered
groups3 = df.groupby('baz', sort=False)
result3 = groups3.agg('mean')
```

``` python
result
```

|  | bar | foo |
| --- | --- | --- |
| range |  |  |
| (0, 2.5] | 60 | 1.0 |
| (2.5, 5] | 40 | 4.0 |
| (5, 7.5] | 55 | 6.5 |
| (7.5, 10] | 15 | CC |

``` python
result2
```

|  | bar | foo |
| --- | --- | --- |
| range |  |  |
| (0, 2.5] | 60 | 1.0 |
| (2.5, 5] | 40 | 4.0 |
| (5, 7.5] | 55 | 6.5 |
| (7.5, 10] | 15 | CC |

``` python
result3
```

|  | bar | foo |
| --- | --- | --- |
| baz |  |  |
| d | 35 | 5.5 |
| c | 45 | 7.5 |
| e | 30 | 5.0 |
| a | 45 | 9.0 |

``` python
pd.__version__
Out[110]: '0.15.1'
```

Setting `as_index=False` does not change the presented bahavior.



Discussion:
Currently this will only work naively, that is the Intervals that are returned are strings
e.g. `(7.5, 10]`. In 0.16 their is work being done on a Interval/IntervalIndex which will allow this to actually be sorted in a certain order. see #8707 

as always pull-requests are welcome to work on these issues. (though reporting them makes good tests cases too!)


Resolving patch:
diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -1925,6 +1925,9 @@ def __init__(self, index, grouper=None, obj=None, name=None, level=None,
 
             # a passed Categorical
             elif isinstance(self.grouper, Categorical):
+                # fix bug #GH8868 sort=False being ignored in categorical groupby
+                if not self.sort:
+                    self.grouper = self.grouper.reorder_categories(self.grouper.unique())
                 self._labels = self.grouper.codes
                 self._group_index = self.grouper.categories
                 if self.name is None:


Test patch:
diff --git a/pandas/tests/test_groupby.py b/pandas/tests/test_groupby.py
--- a/pandas/tests/test_groupby.py
+++ b/pandas/tests/test_groupby.py
@@ -3265,6 +3265,30 @@ def test_no_dummy_key_names(self):
                                   self.df['B'].values]).sum()
         self.assertEqual(result.index.names, (None, None))
 
+    def test_groupby_sort_categorical(self):
+        # dataframe groupby sort was being ignored # GH 8868
+        df = DataFrame([['(7.5, 10]', 10, 10],
+                        ['(7.5, 10]', 8, 20],
+                        ['(2.5, 5]', 5, 30],
+                        ['(5, 7.5]', 6, 40],
+                        ['(2.5, 5]', 4, 50],
+                        ['(0, 2.5]', 1, 60],
+                        ['(5, 7.5]', 7, 70]], columns=['range', 'foo', 'bar'])
+        df['range'] = Categorical(df['range'])
+        index = Index(['(0, 2.5]', '(2.5, 5]', '(5, 7.5]', '(7.5, 10]'], dtype='object')
+        index.name = 'range'
+        result_sort = DataFrame([[1, 60], [5, 30], [6, 40], [10, 10]], columns=['foo', 'bar'])
+        result_sort.index = index
+        index = Index(['(7.5, 10]', '(2.5, 5]', '(5, 7.5]', '(0, 2.5]'], dtype='object')
+        index.name = 'range'
+        result_nosort = DataFrame([[10, 10], [5, 30], [6, 40], [1, 60]], index=index, columns=['foo', 'bar'])
+        result_nosort.index = index
+
+        col = 'range'
+        assert_frame_equal(result_sort, df.groupby(col, sort=True).first())
+        assert_frame_equal(result_nosort, df.groupby(col, sort=False).first())
+
+
     def test_groupby_sort_multiindex_series(self):
         # series multiindex groupby sort argument was not being passed through _compress_group_index
         # GH 9444