ENH: Implement DataFrame.astype('category')
State: closed

- [X] closes #12860
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type

Comments:
One thing I've noticed is that this seems to lead to an inconsistency between `DataFrame(data).astype('category')` and `DataFrame(data, dtype='category')`.

Specifically, per the issue specifications, using `astype` sets all unique labels across all columns as categories, even if they don't appear in a given column:

```python
In [2]: df1 = pd.DataFrame({'A': ['a', 'b', 'c'], 'B': ['c', 'd', 'e']}).astype('category')

In [3]: df1['A'].dtype
Out[3]: CategoricalDtype(categories=['a', 'b', 'c', 'd', 'e'], ordered=False)
```

Whereas instantiating with `dtype='category'` does things column by column:
```python
In [4]: df2 = pd.DataFrame({'A': ['a', 'b', 'c'], 'B': ['c', 'd', 'e']}, dtype='category')

In [5]: df2['A'].dtype
Out[5]: CategoricalDtype(categories=['a', 'b', 'c'], ordered=False)
```

Is this type of inconsistency acceptable?  Or should one method be modified to match the other?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=h1) Report
> Merging [#18099](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/86e9dcc164760c5197438968151b2b852647de84?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18099/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18099      +/-   ##
==========================================
- Coverage   91.25%   91.23%   -0.02%     
==========================================
  Files         163      163              
  Lines       50120    50129       +9     
==========================================
  Hits        45737    45737              
- Misses       4383     4392       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.05% <100%> (ø)` | :arrow_up: |
| #single | `40.31% <9.09%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `92.45% <100%> (+0.03%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=footer). Last update [86e9dcc...081d533](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=h1) Report
> Merging [#18099](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/dfe9d4a7837705ce297186115bd035c3e57a47ec?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18099/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18099      +/-   ##
==========================================
+ Coverage   91.68%   91.69%   +<.01%     
==========================================
  Files         150      150              
  Lines       48976    48978       +2     
==========================================
+ Hits        44906    44910       +4     
+ Misses       4070     4068       -2
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.07% <100%> (ø)` | :arrow_up: |
| #single | `41.86% <25%> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/18099/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `95.89% <100%> (ø)` | :arrow_up: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/18099/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `83.85% <0%> (+0.2%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=footer). Last update [dfe9d4a...293dee5](https://codecov.io/gh/pandas-dev/pandas/pull/18099?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

"More useful" as in more difficult for a user to accomplish, or more people
expect / want?

I agree that table-wise factorization is more difficult for a user. I'm not
sure about whether it's expected.

On Sat, Nov 4, 2017 at 12:05 PM, Jeff Reback <notifications@github.com>
wrote:

> *@jreback* commented on this pull request.
> ------------------------------
>
> In doc/source/categorical.rst
> <https://github.com/pandas-dev/pandas/pull/18099#discussion_r148935998>:
>
> > @@ -143,6 +150,55 @@ constructor to save the factorize step during normal constructor mode:
>      splitter = np.random.choice([0,1], 5, p=[0.5,0.5])
>      s = pd.Series(pd.Categorical.from_codes(splitter, categories=["train", "test"]))
>
> +.. _categorical.objectcreation.frame:
> +
> +Creating categories from a ``DataFrame``
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +.. versionadded:: 0.22.0
> +
> +:meth:`DataFrame.astype` supports simultaneously setting multiple columns as categorical. When setting multiple
>
> column-wise is trivial, uniques across all values then factorize this is
> the more useful factorization (e.g. this one)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/18099#discussion_r148935998>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIv48rOKgvjNkmUgSFVmTNLK5WgbIks5szJlIgaJpZM4QRyEf>
> .
>

> "More useful" as in more difficult for a user to accomplish, or more people
expect / want?

more useful in that this is generally what you want (e.g. conceptually similar to get_dummies). and non-trivial to do. per-column is so easy, e.g. ``df.apply(lambda x: x.astype('category'))`` that having this as the default doesn't make sense. But the most important issues here is that ALL other dtypes coerce when passed to the contructor AND with astype. Diverging here would be a huge problem.

@jschendel pls change the constructor to do essentially this (you will want to move the actual coercing routine to pandas.core.categorical as I mentioned above)
Updated to address the comments:
 - Modified the `DataFrame` constructor to be consistent with `DataFrame.astype`
    - Modified `categorical.rst` to reflect this
    - Added this to the breaking API changes section of the whatsnew
 - Moved logic to the `_get_categorical_dtype_2d` helper function in `categorical.py`
 - Created `unique2d` and `_ensure_arraylike2d` functions in `algorithms.py`
 - Added tests for the three items above

<br />

Regarding  `_ensure_arraylike2d`, the reason I created this was because I was running into an issue with `_ensure_arraylike` where a list of lists gets returned as a numpy array of lists instead of a 2d numpy array:

```python
In [22]: values = [['a', 'b', 'c', 'a'], ['b', np.nan, 'd', 'd']]

In [23]: _ensure_arraylike(values)
Out[23]: array([list(['a', 'b', 'c', 'a']), list(['b', nan, 'd', 'd'])], dtype=object)
```

whereas I'd like to get output along the lines of:
```python
In [24]: np.array([_ensure_arraylike(x) for x in values])
Out[24]:
array([['a', 'b', 'c', 'a'],
       ['b', nan, 'd', 'd']], dtype=object)
```
Should I just try patching `_ensure_arraylike` instead of creating `_ensure_arraylike2d`?  Or is what I did fine?
I am -1 on having `df.astype('category')` work on the full dataframe values instead of column-wise. Yes, working column-wise is also trivial with `apply` (it the the frame-wise which is more difficult out of the box) but IMO that's not a good reason to choose a different default. 
Doing it column-wise a) feels more consistent and b) is IMO what users would expect it to do as the default if they see such a line of code. 
IMO frame-wise is also not 'generally what users want', I think this very much depends on the use case.

To facilitate the more difficult frame-wise case, I would just make sure that we add a clear example on how to achieve this. 
I suppose it is something like:

```
df.astype(pd.api.types.CategoricalDType(categories=np.unique(df.values.ravel())))
```

I would say that is exactly a nice use case for which we made CategoricalDtype publicly available.
> Doing it column-wise a) feels more consistent and b) is IMO what users would expect it to do as the default if they see such a line of code

a) is only true because of the existing behavior.
b) not sure that is the case.

The point here its not clear *what* a user expects. In a 2-d case maybe we should just refuse to guess, IOW raise if ``dtype='category'`` is passed in both construction of a DataFrame and in .astype (meaning ``categories=None`` is passed), rather than picking a default that is unclear.

Of course both examples would be useful in the docs (and even in the error message).



I agree with @jorisvandenbossche and @TomAugspurger here. In my mind, `df.astype` is semantically the same as `df.apply(lambda x: x.astype(...))`, even if in implementation the conversion is done as one operation by the block manager. As a user I find the unification of categories here surprising and non-intuitive. I also did a brief office survey and others found this equally surprising and non-intuitive (cc @mcg1969).

If it's not an easy one-liner I'd prefer a separate function for doing the category unification and conversion instead of using `astype`. Or perhaps a keyword argument to `astype`, with unification as the non-default option.
I appreciate the cc: here. I'm with @jcrist et al regarding what the default behavior should be. The one addition I offered, that I don't think Jim agreed with, was the idea of offering the full-dataframe conversion as a kwarg option.
The "one-liner" currently is 

```
df.astype(pd.api.types.CategoricalDType(categories=np.unique(df.values.ravel())))
```

which is of course not that easy. 
I was thinking we could add a constructor for CategoricalDtype from just values (and categories get inferred), but that is also not much shorter:

```
df.astype(pd.api.types.CategoricalDtype.from_values(df.values.ravel()))
```

(although we could implement it so that the `.ravel()` is not needed to be done by the user)

A keyword argument to `astype` feels a bit strange to me as this seems to be specific for categoricals. Or are there other dtypes were this would give a different result?
I don't have a particularly strong opinion either way; did this PR more out of finding it interesting than an actual want/need for the feature.  Have been in situations where it'd be the behavior I'd want, though not necessarily sure it's what I'd expect.

I can't think of any other dtypes where a keyword argument would yield different results, but if we still want to use one it seems like `axis` could work with behavior similar to how `ndarray.sum` works in numpy, i.e.:
 - `axis=None`: over all elements, i.e. what this PR currently does
 - `axis=0`: column-wise
 - `axis=1`: ??? Not really sure.  Seems like it could just raise a `NotImplementedError` equivalent to what 2d structures currently do, e.g. `pd.Categorical(np.random.random((3,3)))`

Still seems kind of strange to have a keyword argument that's only applicable to categoricals, but at least `axis` seems somewhat logically consistent, since it seems like `axis` would conceptually give the same result regardless of what value is passed for other dtypes, so could minimize any confusion.

Happy to modify the PR however is deemed fit, once a clear path has been established.
Hello @jschendel! Thanks for updating the PR.





Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on February 28, 2018 at 08:13 Hours UTC
ok I will reverse my earlier stance on this, and support this working column-wise, assuring that ``DataFrame.astype('category')`` and ``DataFrame(data, dtype='category')`` work the same. Should add a small section in docs to do whole dataframe coding as an example.

@jschendel if you want to update (and move notes to 0.23.0) could get this in .
Updated to perform columnwise conversion:

- Changed the issue number to reference this PR instead of the original issue
   - Wanted to avoid confusion since the original issue is in regards to tablewise conversion
- Added this behavior to `categorical.rst`
   - Shuffled some examples and remarks around to make things more coherent
   - Nothing should have be removed, just moved to a slightly different location
- Added a small example to the whatsnew
Made the requested doc changes.
Made the doc updates.  Will fix the related issue in a follow-up PR.
very nice @jschendel pls the .astype concat issues in a followup.