GH24241 make Categorical.map transform nans
State: closed

- [ ] closes #24241
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

Alters `Categorical.map` so that the mapper function is also applied to NaN values. The mentioned bug report brings up the example of calling `apply(pd.isna)` on a categorical series, arguing that the values and dtype of the returned list should be consistent with non-categorical series. This PR makes the values consistent. The discrepancies in dtypes are present in categorical series without NaN's and consistent with documentation.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Categorical
description: Categorical Data Type

Comments:
Hello @JustinZhengBC! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`pandas/core/arrays/categorical.py`](https://github.com/JustinZhengBC/pandas/blob/d765dc33c2326778df30bee1c0046db75a837363/pandas/core/arrays/categorical.py) !

 - There are no PEP8 issues in the file [`pandas/tests/indexes/test_category.py`](https://github.com/JustinZhengBC/pandas/blob/d765dc33c2326778df30bee1c0046db75a837363/pandas/tests/indexes/test_category.py) !


# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=h1) Report
> Merging [#24275](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/31a351258ad36392a8251c195f7e0869b90c7467?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24275/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24275      +/-   ##
==========================================
- Coverage   92.22%   92.22%   -0.01%     
==========================================
  Files         162      162              
  Lines       51787    51798      +11     
==========================================
+ Hits        47761    47771      +10     
- Misses       4026     4027       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <100%> (ø)` | :arrow_up: |
| #single | `42.99% <0%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.37% <100%> (+0.06%)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.61% <0%> (-0.48%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.51% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=footer). Last update [31a3512...d765dc3](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=h1) Report
> Merging [#24275](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/31a351258ad36392a8251c195f7e0869b90c7467?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24275/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24275      +/-   ##
==========================================
- Coverage   92.22%   92.22%   -0.01%     
==========================================
  Files         162      162              
  Lines       51787    51798      +11     
==========================================
+ Hits        47761    47771      +10     
- Misses       4026     4027       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <100%> (ø)` | :arrow_up: |
| #single | `42.99% <0%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.37% <100%> (+0.06%)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.61% <0%> (-0.48%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.51% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=footer). Last update [31a3512...d765dc3](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=h1) Report
> Merging [#24275](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/31a351258ad36392a8251c195f7e0869b90c7467?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24275/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24275      +/-   ##
==========================================
- Coverage   92.22%   92.22%   -0.01%     
==========================================
  Files         162      162              
  Lines       51787    51798      +11     
==========================================
+ Hits        47761    47771      +10     
- Misses       4026     4027       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <100%> (ø)` | :arrow_up: |
| #single | `42.99% <0%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.37% <100%> (+0.06%)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.61% <0%> (-0.48%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.51% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=footer). Last update [31a3512...d765dc3](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=h1) Report
> Merging [#24275](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/31a351258ad36392a8251c195f7e0869b90c7467?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24275/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24275      +/-   ##
==========================================
- Coverage   92.22%   92.22%   -0.01%     
==========================================
  Files         162      162              
  Lines       51787    51798      +11     
==========================================
+ Hits        47761    47771      +10     
- Misses       4026     4027       +1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <100%> (ø)` | :arrow_up: |
| #single | `42.99% <0%> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.37% <100%> (+0.06%)` | :arrow_up: |
| [pandas/io/json/json.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2pzb24vanNvbi5weQ==) | `92.61% <0%> (-0.48%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.51% <0%> (+0.09%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=footer). Last update [31a3512...d765dc3](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=h1) Report
> Merging [#24275](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/6111f645c5adc5bdcd3810b4112392bda3583d59?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/24275/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #24275      +/-   ##
==========================================
+ Coverage   92.29%   92.29%   +<.01%     
==========================================
  Files         162      162              
  Lines       51808    51834      +26     
==========================================
+ Hits        47814    47840      +26     
  Misses       3994     3994
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.7% <100%> (ø)` | :arrow_up: |
| #single | `42.98% <0%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/arrays/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2NhdGVnb3JpY2FsLnB5) | `95.32% <100%> (+0.01%)` | :arrow_up: |
| [pandas/core/groupby/groupby.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9ncm91cGJ5LnB5) | `96.65% <0%> (-0.03%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `87.57% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.66% <0%> (ø)` | :arrow_up: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.91% <0%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.71% <0%> (ø)` | :arrow_up: |
| [pandas/core/reshape/merge.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZXJnZS5weQ==) | `94.3% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYmFzZS5weQ==) | `97.66% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/window.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvd2luZG93LnB5) | `96.41% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `87.15% <0%> (+0.03%)` | :arrow_up: |
| ... and [1 more](https://codecov.io/gh/pandas-dev/pandas/pull/24275/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=footer). Last update [6111f64...82859d9](https://codecov.io/gh/pandas-dev/pandas/pull/24275?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I decided to go with @TomAugspurger's suggestion to just change the documentation. I did leave in one change in the function, because previously calling `np.take` would cause `NaN` values (represented by -1 in `self._codes`) to take the last element of `new_categories`.
@jreback green
Thanks @JustinZhengBC!