[WIP]API: CategoricalType for specifying categoricals
State: closed

closes #14676 

This adds a new top-level class `pd.CategoricalType` for representing the *type* information on a `Categorical`, *without the values*. Users can pass this anywhere they would previously pass `'category'` if they need to specify non-default values for `categories` or `ordered` (you can still use `'category'` of course):

```python
In [1]: from pandas import *

In [2]: paste
    >>> t = CategoricalType(categories=['b', 'a'], ordered=True)
    >>> s = Series(['a', 'a', 'b', 'b', 'a'])
    >>> c = s.astype(t)
    >>> c

## -- End pasted text --
Out[2]:
0    a
1    a
2    b
3    b
4    a
dtype: category
Categories (2, object): [b < a]
```

This is the simplest possible change for now. A bigger change would be to make `c.dtype` return a `CategoricalType` instead of '`category`'. We could probably do that in a way that's backwards compatible, but I'll need to think on it a bit.

Other places this should work

- `select_dtypes` (currently returns all categoricals; should just be ones with same `CategoricalType`?)
- `read_csv`: need to implement, see https://github.com/pandas-dev/pandas/issues/14503
- `Series(..., dtype=...`)
- probably others.

Implementation-wise I need to document, more tests, clean up some things like the repr. But I wanted to get this out there for discussion. @janschulz you might be interested.

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design
name: Categorical
description: Categorical Data Type

Comments:
What would the ideal `__repr__` of `CategoricalDtype` be? Currently it's just `category`. For places like `DataFrame.info` it's probably best to keep that as is, but that's not as useful if you're inpsecing `CategoricalDtype` itself.

Also, for equality semantics, this is what I have right now

``` python
    An instance of ``CategoricalDtype`` compares equal with any other
    instance of ``CategoricalDtype``, regardless of categories or ordered.
    In addition they compare equal to the string ``'category'``.
    To check whether two instances of a ``CategoricalDtype`` match,
    use the ``is`` operator.

    >>> t1 = CategoricalDtype(['a', 'b'], ordered=True)
    >>> t2 = CategoricalDtype(['a', 'c'], ordered=False)
    >>> t1 == t2
    True
    >>> t1 == 'category'
    True
    >>> t1 is t2
    False
    >>> t1 is CategoricalDtype(['a', 'b'], ordered=True)
    True
```

though I don't expect people to be working with these objects that much.

you need to move over the repr (of the categories) and the validation logic on the categories to this class.

Further Categorical should use this internally for storage of categories / ordered. 

This is a fairly invasive change and needs some thought.
I would split off the actual issue change and make that a follow up PR. Just putting in place the correct infrastructure can be the scope of this PR>
> This is a fairly invasive change and needs some thought.

Yep. I had hoped to do the minimal change of just providing a new API, but let's do it right the first time. I'll spend some time on this the next couple weeks and ping when I have something.
Closing for now. Will reopen a new PR on top of a PR fixing https://github.com/pandas-dev/pandas/issues/14711