BUG: Fix groupby sorting on ordered Categoricals (GH25871)
State: closed

As documented in #25871, groupby() on an ordered Categorical messes up category order when 'observed=True' is specified. 
Specifically, group labels will be ordered by first occurrence (as for an unordered Categorical), but grouped aggregation results will retain the Categorical's order.
The fix is a modified subset of #25173, which fixes a related case, but has not been merged yet.

- [x] closes #25871, also #25167
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Groupby
name: Categorical
description: Categorical Data Type

Comments:
Hello @kpflugshaupt! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-04-03 08:05:15 UTC
Fixed new PEP 8 issues, too
Failing test reproduced here. Will investigate
Adapted the failing test, reopening
@kpflugshaupt you don't need to close and reopen a PR for every commit (causes unnecessary messaging spam) - just push to the PR as you have new commits.

Is this a WIP or ready for review?
@WillAyd: Sorry for the spamming, I did not know. Will leave open in the future, thanks!
As soon as automatic checks come back OK, this is good for review.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=h1) Report
> Merging [#25908](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/4814a2875d078d6e9e9300bd1f77af15ef4f0f0a?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25908/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25908      +/-   ##
==========================================
- Coverage   91.84%   91.84%   -0.01%     
==========================================
  Files         175      175              
  Lines       52550    52552       +2     
==========================================
- Hits        48266    48265       -1     
- Misses       4284     4287       +3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.39% <100%> (ø)` | :arrow_up: |
| #single | `41.89% <0%> (-0.08%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/grouper.py](https://codecov.io/gh/pandas-dev/pandas/pull/25908/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGVyLnB5) | `98.18% <100%> (+0.01%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25908/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-12.5%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/25908/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.79% <0%> (-0.12%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25908/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `100% <0%> (+4.54%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=footer). Last update [4814a28...c9e3883](https://codecov.io/gh/pandas-dev/pandas/pull/25908?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The added test from #25167 fails. Will investigate.
Got the expected values wrong, don't know what I was thinking. Fixed now.
Windows checks were failing because 'int' type apparently means 'int32' on that planet. I learned something today.
> Windows checks were failing because 'int' type apparently means 'int32' on that planet. I learned something today.

This is a 32-bit issue, not Windows as such. This is a bit annoying, but pandas uses ``np.int64`` always in some cases and ``np.intp`` in other cases. In this case you should likely use ``np.intp``.

In general, you should in pandas always use explicit types, so either ``np.int64`` or ``np.intp``, depening on needs. Using just ``int`` causes various issues unless you actually want a python int.
That’s good to know. As I‘m working on Mac and Linux only, I assumed everyone defaulted to int64. 
That should be it -- all checks passing now. I've added #25167 to the solved issues in the whatsnew file.
Changed test as per @jreback 's comments.
thanks @kpflugshaupt nice patch!
You're welcome, nice working with you.