Categorical type
State: closed

We extended the CategoricalDtype to accept optional `categories` and `ordered`
argument. CategoricalDtype is now part of the public API. This allows users to
specify the desired categories and orderedness of an operation ahead of time.
The current behavior, which is still possible with `categories=None`, the
default, is to infer the categories from whatever is present.

This change will make it easy to implement support for specifying categories
that are know ahead of time in other places e.g. `.astype`, `.read_csv`, and the
`Series` constructor.

Closes #14711 
Closes #15078
Closes #14676 

Dunno if we would want to do this for 0.20 or not. It ended up not being *too* big a change, so maybe.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
.dtype on a Categorical/CI should construct this

Categorica.is_dtype_equal should just work
can u run asv on this to make sure nothing much changing perf wise
> .dtype on a Categorical/CI should construct this

Ha, that's why it felt like so much less code this time, because I forgot to do the other half :)

I'll work on this a bit more today, but will probably push to 0.21
yeah this feels like it should be in 0.21.0. *maybe* we also expose this in ``pandas.api.types.CategoricalDtype`` rather than at the top-level.
Hello @TomAugspurger! Thanks for updating the PR.





















































Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on September 23, 2017 at 16:34 Hours UTC
@TomAugspurger just to check: this is ready for review?
It's close. This weekend probably. I'll submit a comment with a summary of the changes and let you know when it's really ready.

________________________________
From: Joris Van den Bossche <notifications@github.com>
Sent: Friday, August 18, 2017 5:26:43 PM
To: pandas-dev/pandas
Cc: Tom Augspurger; Mention
Subject: Re: [pandas-dev/pandas] Categorical type (#16015)


@TomAugspurger<https://github.com/tomaugspurger> just to check: this is ready for review?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<https://github.com/pandas-dev/pandas/pull/16015#issuecomment-323474370>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABQHIopwBcbW-elgQJliYxJ-ItR4Idswks5sZg-jgaJpZM4M-hoW>.

Alighty, this should be more or less ready for review. I have the PR split into two parts (well 3 if you count the cherry-picked 1abaecb from a separate PR)

a7eb835 has all the "new" stuff, along with tests for the new things.

f9236b4 has all the "API bending" stuff, the tests I had to update.
@jreback I'm looking into the performance now. Just micro-benchmarks at the moment, I'll follow up with more realistic ones, but on these two:

```python
# 1. single:
CategoricalDtype(range(100))

# 2. cached:
data = range(100)
x = CategoricalDtype(data)  # in the cache
CategoricalDtype(data)
```

I'm comparing two versions: caching (this branch) and no caching (essentially the same, but without the hashing of values / caching in `CategoricalDtype.__new__`).

| Kind   | Caching | No-Caching |
| ------ | ------- | ---------- |
| single | 125     | 35.6       |
| cached | 116     | 35         |

Hashing takes up about 60% of the time in the `CategoricalDtype.__new__`, so it'd be nice to avoid if possible.

---

What kind of restrictions do we have on instances of extension types being singletons (per parameter). I've been basing this off `PeriodDtype`, and we test for it, but how important is it?
I've gone ahead and removed the caching of `{hash: CategoricalDtype}` in https://github.com/pandas-dev/pandas/pull/16015/commits/e8dda56552a2c38b7d80c9b3a5188de815e60008

I like that it's simpler and faster, but can certainly add that back in if required. Clearly there were some issues with the hashing / caching implementation in the [old CI runs](https://travis-ci.org/pandas-dev/pandas/jobs/270193477#L1183)
OK, the only windows failure was on fastparquet [here](https://ci.appveyor.com/project/pandas-dev/pandas/build/1.0.4133/job/7heo6k1pjs3a6up3#L1670) has been fixed and will be in the next release of fastparqeut (next week I think). I can either skip for now or just wait till that's out. The issue was fastparquet using `str(dtype) == 'category'`, which I suppose is a breaking change, but I'm OK with it since `is_categorical_dtype` is the recommended way for checking for a Categorical, and fastparquet was setting some private properties.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=h1) Report
> Merging [#16015](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d4577911c750f2f48f760ce451d413116bed72da?src=pr&el=desc) will **decrease** coverage by `0.17%`.
> The diff coverage is `92.3%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16015/graphs/tree.svg?token=eZ4WkYLtcO&width=650&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16015      +/-   ##
==========================================
- Coverage   91.15%   90.97%   -0.18%     
==========================================
  Files         163      163              
  Lines       49587    49642      +55     
==========================================
- Hits        45203    45164      -39     
- Misses       4384     4478      +94
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.76% <92.3%> (-0.17%)` | :arrow_down: |
| #single | `40.35% <66.66%> (+0.04%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/core/util/hashing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdXRpbC9oYXNoaW5nLnB5) | `98.34% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `96.9% <100%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.95% <100%> (ø)` | :arrow_up: |
| [pandas/core/sorting.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc29ydGluZy5weQ==) | `98.19% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `92.18% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `95.84% <100%> (-0.45%)` | :arrow_down: |
| [pandas/core/indexes/category.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9jYXRlZ29yeS5weQ==) | `98.53% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.28% <100%> (-0.03%)` | :arrow_down: |
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `92.61% <100%> (ø)` | :arrow_up: |
| ... and [15 more](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=footer). Last update [d457791...1ca07b6](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=h1) Report
> Merging [#16015](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/b59f107a529ac9b1e2859217ea58f7e26b6749d0?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `95.04%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16015/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16015      +/-   ##
==========================================
+ Coverage   91.19%    91.2%   +<.01%     
==========================================
  Files         163      163              
  Lines       49626    49763     +137     
==========================================
+ Hits        45258    45384     +126     
- Misses       4368     4379      +11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.99% <95.04%> (+0.02%)` | :arrow_up: |
| #single | `40.27% <60.89%> (+0.01%)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `100% <ø> (ø)` | :arrow_up: |
| [pandas/core/util/hashing.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvdXRpbC9oYXNoaW5nLnB5) | `98.34% <ø> (ø)` | :arrow_up: |
| [pandas/core/indexes/interval.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9pbnRlcnZhbC5weQ==) | `93.57% <0%> (ø)` | :arrow_up: |
| [pandas/core/sorting.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc29ydGluZy5weQ==) | `98.19% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/range.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9yYW5nZS5weQ==) | `92.83% <100%> (+0.24%)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.92% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/multi.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9tdWx0aS5weQ==) | `96.9% <100%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.28% <100%> (ø)` | :arrow_up: |
| [pandas/core/internals.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzLnB5) | `94.28% <100%> (-0.03%)` | :arrow_down: |
| [pandas/core/dtypes/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbW1vbi5weQ==) | `94.45% <66.66%> (-0.41%)` | :arrow_down: |
| ... and [21 more](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=footer). Last update [b59f107...43f90cc](https://codecov.io/gh/pandas-dev/pandas/pull/16015?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

@jreback @jorisvandenbossche gentle ping if you don't mind giving this a look. It'd be nice to have in master for a little while before the release.

I've skipped the fastparquet categorical tests for versions < 0.1.3 (which is all versions but master at the moment). I've tested locally against fastparquet master.
will have a look tomorrow
> how is perf?

Currently there's some issues with our internal fastpaths like `CategoricalIndex._shallow_copy` and `_simple_new`. I think I'm unnecessarily creating another instance of `CategoricalDtype` instead of passing the existing on on. Will fix that tomorrow.
Thanks Joris, I'll try to address those all this afternoon.

> General remark, I find the fact that a CategoricalDtype instance can exist without any categories/ordered specified a bit strange. But that is needed to support the case of dtype='category' where those are inferred from the data?

To be clear, the `ordered` will default to `False`, just like before. Agreed that the `categories` potentially being none is strange. I don't think users should run into it that often though... Anytime you have an actual array of categorical data, the `categories` will not be None. The only case I can imagine is where you know ahead of time that your data is ordered and sorted, but you don't know the categories?

```python
pd.read_csv("foo.csv", dtype={"A": CategoricalDtype(None, ordered=True)})
```

Which seems like a recipe for future bugs to me.
@jorisvandenbossche Agreed about not having a `fastpath` in the public API. see https://github.com/pandas-dev/pandas/pull/16015/commits/91752fe87f6cd5c06b7ee95b7fabb7534631a0df

Initially we used `__new__` for initialization, since we cached the instance (even though it didn't take parameters). That commit cleans things up a bit.
Correct, and I don't plan to deprecate `dtype'category'`.

On Mon, Sep 18, 2017 at 5:55 AM, Joris Van den Bossche <
notifications@github.com> wrote:

> *@jorisvandenbossche* commented on this pull request.
> ------------------------------
>
> In doc/source/categorical.rst
> <https://github.com/pandas-dev/pandas/pull/16015#discussion_r139390284>:
>
> > +should be inferred from whatever is present in the data when the
> +:class:`pandas.Categorical` is created.
> +
> +.. ipython:: python
> +
> +   from pandas.api.types import CategoricalDtype
> +
> +   CategoricalDtype(['a', 'b', 'c'])
> +   CategoricalDtype(['a', 'b', 'c'], ordered=True)
> +   CategoricalDtype()
> +
> +A :class:`~pandas.api.types.CategoricalDtype` can be used in any place pandas
> +expects a `dtype`. For example :func:`pandas.read_csv`,
> +:func:`pandas.DataFrame.astype`, or in the Series constructor.
> +
> +As a convenience, you can use the string ``'category'`` in place of a
>
> Yes, but you didn't do that actual deprecation (or not yet)?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/16015#discussion_r139390284>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHItygdUADe7vz66Heen2kIDzUzdl9ks5sjkwVgaJpZM4M-hoW>
> .
>

> Yes, but you didn't do that actual deprecation (or not yet)?

I don't think this is desirable, 'category' is a nice convenience
https://github.com/pandas-dev/pandas/pull/16015/commits/416d1d7b9ce81287a1b7f2802751dca1fdeb5084 has the changes to `Categorical.__init__` that you requested @jreback. It's a bit messy since there are basically 3 ways for specifying the categories

1. passing `dtype=CategoricalDtype(...)`
2. passing `categories=...`
3. passing a `Categorical` for `values`

I think all three are useful and (hopefully) well-tested now. I'll update the docstring to indicate the priority (basically moving that comment to the docstring)
Thanks. I fixed up the doc comments. I'll merge on green and do the other
followups later today.

On Fri, Sep 22, 2017 at 7:31 PM, Jeff Reback <notifications@github.com>
wrote:

> *@jreback* commented on this pull request.
> ------------------------------
>
> In doc/source/whatsnew/v0.21.0.txt
> <https://github.com/pandas-dev/pandas/pull/16015#discussion_r140620020>:
>
> > @@ -89,6 +91,30 @@ This does not raise any obvious exceptions, but also does not create a new colum
>
>  Setting a list-like data structure into a new attribute now raise a ``UserWarning`` about the potential for unexpected behavior. See :ref:`Attribute Access <indexing.attribute_access>`.
>
> +.. _whatsnew_0210.enhancements.categorical_dtype:
> +
> +``CategoricalDtype`` for specifying categoricals
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +:class:`pandas.api.types.CategoricalDtype` has been added to the public API and
> +expanded to include the ``categories`` and ``ordered`` attributes. A
> +``CategoricalDtype`` can be used to specify the set of categories and
> +orderedness of an array, independent of the data themselves. This can be useful,
> +e.g., when converting string data to a ``Categorical`` (:issue:`14711`, :issue:`15078`):
>
> there is 1 more issue you listed in the top of the PR
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/16015#pullrequestreview-64725698>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIlwjFqIXM4vMTgsVZ20tYOzp9CQSks5slFFogaJpZM4M-hoW>
> .
>

nice patch @TomAugspurger !