ENH: allow better categorical dtype strings, e.g. `category[string]`
State: closed

- [X] closes #48515
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This PR is a quality-of-life improvement when working with categorical dtypes:

```python
>>> df = pd.read_csv(file, dtypes={"a": "category[string]", "b": "category[datetime64[ns, UTC]]"} 
```

It can also be visible in reprs and can be used for comparisons:

```python
>>> ser = pd.Series([1,2,3, 4], dtype="category[Int8]")
>>> ser
0    1
1    2
2    3
3    4
dtype: category[Int8]
Categories (4, Int8): [1, 2, 3, 4]
>>> ser.dtype == "category"
True
>>> ser.dtype == "category[Int8]"
True
>>> ser.dtype == "category[string]"
False
```


Still needs to do
- [ ]  update the tests.

EDIT: updated to newest capabilities.

Labels:
name: Enhancement
name: Categorical
description: Categorical Data Type
name: Stale

Comments:
i was expecting this to be about getting a more informative string when doing `df.dtypes`.

I like the added specificity, but dont like adding more statefulness to the Dtype object.  once the categories are known the extra state should be redundant right?
> i was expecting this to be about getting a more informative string when doing `df.dtypes`.

I was actually thinking about doing that as a follow-up, so we'd get e.g.:

```python
>>> cat = pd.Categorical(["a", "f"])
>>> pd.DataFrame(cat).dtypes
0    category[object]
dtype: object
```

> I like the added specificity, but dont like adding more statefulness to the Dtype object. once the categories are known the extra state should be redundant right?

I could not initiate `_categories_dtype` on the class + delete it after the categories have been declared?

The issue is that `CategoricalDtype` can be instantiated without categories, so to make `"category[string]"` work as a string dtype, we have to store the dtype string somewhere, until the categories are declared.
I've updated this, so in addition to being able to use these strings as dtypes in constructors, they will also be visible in reprs and can be used for comparisons:

```python
>>> ser = pd.Series([1,2,3, 4], dtype="category[Int8]")
>>> ser
0    1
1    2
2    3
3    4
dtype: category[Int8]
Categories (4, Int8): [1, 2, 3, 4]
>>> ser.dtype == "category"
True
>>> ser.dtype == "category[Int8]"
True
>>> ser.dtype == "category[string]"
False
```
 I still need to do the tests and before I do them, can you verify you approve of the idea, @jbrockmendel?

> i am surprised this doesn't break a lot of things
> 
> i like the repr idea but i am not sure allowing a dtype that is not inferred is actually useful
> 
> can you show when this is the case?

Hmm, this is backward compatible, so don't see it could break things? E.g. `Series([...], dtype="category")` will still infer.

The use case is generally when we want a more precise categorical dtype, we currently typically do it like this:

```python
>>> df = pd.read_csv(file, dtypes={"a": "string", "b": "datetime64[ns, UTC]"} 
>>> df["a"] = df["a"].astype("category")
>>> df["b"] = df["b"].astype("category")
```

which after this can be: 

```python
df = pd.read_csv(file, dtypes={"a": "category[string]", "b": "category[datetime64[ns, UTC]]"} 
```

More precise type comparisons are currently done like this when comparing to string dtypes:

```python
>>> cat = pd.Categorical(pd.array([1, 2, 3, 4], dtype="string"), dtype="category") 
>>> if cat.dtype = "category" and cat.dtype.categories.dtype == "string":
...     ...
```

which after this PR can be:

```python
>>> cat = pd.Categorical([1, 2, 3, 4], dtype="category[string]") 
>>> if cat.dtype = "category[string]":
...     ...
```

So this makes some common categorical operations more readable and concise IMO.  Note also in the `read_csv` example that readers can after this PR see that the end product will be a categorical of strings, opening up for the readers to possibly be more efficient.

i very much like the `__repr__` and `__eq__` here.  im not sold on it being in the constructor 
I'm not super fond of a categorical dtype repr not being usable as a dtype, i.e. if we allow `"category[string]"` as a dtype repr, IMO not allow it as a dtype could be unexpected.

The `categories` attribute can currently be either None or a sequence. Could an idea be that `categories` could also be a dtype to represent an intermediate state, i.e. no sequence has yet been added, but we've restricted it to be of the given dtype?

For example `Series([1, 2, 3], dtype="category[string]")` would then be equivalent to `Series([1, 2, 3], dtype=CategoryDtype(StringDtype()))`? This would then resolve into `Series([1, 2, 3], dtype=CategoryDtype(Index([1, 2, 3], dtype="string")))` when `CategoricalDtype` has been given the sequence?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Appears the discussion here has stalled and might need a bit more discussion on the design which may good to have in the issue so closing for now. Happy to reopen at a later point