BUG: preserve categorical & sparse types when grouping / pivot
State: closed

closes #18502 
replaces #26550 

Labels:
name: Groupby
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: Categorical
description: Categorical Data Type

Comments:
@WillAyd https://github.com/pandas-dev/pandas/pull/27071#discussion_r297961631

that test is not appropriate for checking the results of this change as most of those ops don't work on ordered categoricals;  i have covered the most common of first/last/min/max above.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=h1) Report
> Merging [#27071](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d94146cf5d5d9f899a31308d0a4d9e5d132f6442?src=pr&el=desc) will **decrease** coverage by `1.38%`.
> The diff coverage is `93.1%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27071/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #27071      +/-   ##
==========================================
- Coverage   92.04%   90.66%   -1.39%     
==========================================
  Files         180      180              
  Lines       50714    50727      +13     
==========================================
- Hits        46680    45991     -689     
- Misses       4034     4736     +702
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.66% <93.1%> (-0.02%)` | :arrow_down: |
| #single | `?` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `88.48% <100%> (-0.86%)` | :arrow_down: |
| [pandas/core/groupby/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9vcHMucHk=) | `96% <100%> (ø)` | :arrow_up: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `94.76% <100%> (ø)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `97.32% <100%> (+0.15%)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `96.21% <100%> (+0.25%)` | :arrow_up: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.95% <71.42%> (-0.19%)` | :arrow_down: |
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `62.5% <0%> (-27.75%)` | :arrow_down: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `64.86% <0%> (-25.44%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/computation/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vY29tbW9uLnB5) | `84.21% <0%> (-5.27%)` | :arrow_down: |
| ... and [10 more](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=footer). Last update [d94146c...424c466](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=h1) Report
> Merging [#27071](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/c1673cf42e2827c06b16c3f9bc2affd9df7c311a?src=pr&el=desc) will **increase** coverage by `50.06%`.
> The diff coverage is `93.1%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/27071/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=tree)

```diff
@@             Coverage Diff             @@
##           master   #27071       +/-   ##
===========================================
+ Coverage   41.96%   92.02%   +50.06%     
===========================================
  Files         180      180               
  Lines       50707    50727       +20     
===========================================
+ Hits        21277    46681    +25404     
+ Misses      29430     4046    -25384
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.66% <93.1%> (?)` | |
| #single | `41.85% <24.13%> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `88.48% <100%> (+73.66%)` | :arrow_up: |
| [pandas/core/groupby/ops.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9vcHMucHk=) | `96% <100%> (+76.23%)` | :arrow_up: |
| [pandas/core/nanops.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvbmFub3BzLnB5) | `94.76% <100%> (+63.17%)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `97.32% <100%> (+73.45%)` | :arrow_up: |
| [pandas/core/internals/construction.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2NvbnN0cnVjdGlvbi5weQ==) | `96.21% <100%> (+31.81%)` | :arrow_up: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.95% <71.42%> (+41.89%)` | :arrow_up: |
| [pandas/core/computation/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vcHl0YWJsZXMucHk=) | `90.24% <0%> (+0.3%)` | :arrow_up: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `90.3% <0%> (+0.96%)` | :arrow_up: |
| [pandas/core/panel.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcGFuZWwucHk=) | `17.8% <0%> (+1.7%)` | :arrow_up: |
| [pandas/util/\_test\_decorators.py](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvX3Rlc3RfZGVjb3JhdG9ycy5weQ==) | `93.84% <0%> (+4.61%)` | :arrow_up: |
| ... and [138 more](https://codecov.io/gh/pandas-dev/pandas/pull/27071/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=footer). Last update [c1673cf...9b8f2b4](https://codecov.io/gh/pandas-dev/pandas/pull/27071?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@WillAyd @jbrockmendel ?
Does this also preserve the dtypes under transpose?
@jbrockmendel 

> Does this also preserve the dtypes under transpose?

no that's a more general issues

> home/vsts/work/1/s/doc/source/whatsnew/v0.25.0.rst:856: WARNING: Unknown interpreted text role "method".

@jorisvandenbossche any idea what this means?