ENH: pivot/groupby index with nan (GH3729)
State: closed

Adds the ability to use NaN as a grouping variable. Required allowing NA as a factorisation variable in `factorize()`.

- [x] closes #3729
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Enhancement
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=h1) Report
> Merging [#21669](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0b63e81dfcd7860a8c30e0837238ff9cc7bbe88d?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21669/graphs/tree.svg?src=pr&width=650&token=eZ4WkYLtcO&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21669      +/-   ##
==========================================
+ Coverage    91.9%    91.9%   +<.01%     
==========================================
  Files         154      154              
  Lines       49656    49663       +7     
==========================================
+ Hits        45637    45644       +7     
  Misses       4019     4019
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.28% <100%> (ø)` | :arrow_up: |
| #single | `42.02% <20%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/21669/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `92.66% <100%> (ø)` | :arrow_up: |
| [pandas/core/algorithms.py](https://codecov.io/gh/pandas-dev/pandas/pull/21669/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYWxnb3JpdGhtcy5weQ==) | `94.89% <100%> (+0.03%)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21669/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.21% <100%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=footer). Last update [0b63e81...7b3cf75](https://codecov.io/gh/pandas-dev/pandas/pull/21669?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Happy to change the name if you like, what don't you like about it? Or do
you have a suggested different keyword?


On Wed., 4 Jul. 2018, 01:01 Jeff Reback, <notifications@github.com> wrote:

> *@jreback* requested changes on this pull request.
>
> needs quite a bit of tests & to parameterize existing tests to accept nan
> groups. i don't like the keyword at all. haven't reviewed fully.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21669#pullrequestreview-134038822>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AD615yfHPDUVnSdwQjtSkJzSMzh3SI7Bks5uC4csgaJpZM4U8YHP>
> .
>

@andrewdalecramer 

this needs to be ``dropna=None`` as a default, with a warning if its *not* passed and a default of ``True`` for back-compat. Eventually we would want to make this False. This should only warn if there are actual nan groups (to avoid gratuitous warnings all over the place) I think.
@andrewdalecramer 
Suggestion for keyword name: allow_na_groups = False
can you rebase this
this is a nice feature, but needs quite a bit of work to fully test this out.

closing as stale. if you'd like to continue, pls ping.