PERF/ENH: add fast astyping for Categorical
State: closed

- [x] closes #8628
- [ ] tests added / passed
- [x] benchmarks added
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

To illustrate the speed-up, the set-up is (from OP):
```python
import numpy as np
import pandas as pd

rng = np.random.default_rng()

df = pd.DataFrame(
    rng.choice(np.array(list("abcde")), 4_000_000).reshape(1_000_000, 4),
    columns=list("ABCD"),
)

for col in df.columns:
    df[col] = df[col].astype("category")
```
On master
``` python
In [5]: %timeit [df[col].astype('unicode') for col in df.columns] 
250 ms ± 601 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
versus on this branch
``` python
In [5]: %timeit [df[col].astype('unicode') for col in df.columns] 
5.38 ms ± 47 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Categorical
description: Categorical Data Type

Comments:
This changes the behaviour unfortunately:

```python
>>> n = 100_000
>>> cat = pd.Categorical(["a", "b"] * n)
>>> ser = pd.Series(cat)
>>> ser.astype(str).dtype
dtype('O')  # master
CategoricalDtype(categories=['a', 'b'], ordered=False)  # this PR, different dtype
```
Can you make it work in some different way?
> This changes the behaviour unfortunately:
> 
> ```python
> >>> n = 100_000
> >>> cat = pd.Categorical(["a", "b"] * n)
> >>> ser = pd.Series(cat)
> >>> ser.astype(str).dtype
> dtype('O')  # master
> CategoricalDtype(categories=['a', 'b'], ordered=False)  # this PR, different dtype
> ```
> 
> Can you make it work in some different way?

Thanks for the catch! Fixed this now
It seems like adding this to astype has a lot of repercussions downstream, casting objects in undesirable ways. The special casing needed to make the tests pass is a code smell. 

I wonder if it might make sense, instead of adding this to `astype`,  to create a dedicated method for astyping a `Categorical` via this method?
```
In []: cat = pd.Categorical(["a", "b", "a", "a"] ) 
   ...: s = pd.Series(cat) 
   ...: s.astype_categories(dtype=str)                                      
Out[]: 
0    a
1    b
2    a
3    a
dtype: object
```

see also #37371; this might be the same path.
Re: review comments:

#### `int` benchmark results
```
In [1]: import numpy as np 
   ...: import pandas as pd 
   ...:  
   ...: rng = np.random.default_rng() 
   ...:  
   ...: df = pd.DataFrame( 
   ...:     rng.choice(np.arange(8), 4_000_000).reshape(1_000_000, 4), 
   ...:     columns=list("ABCD"), 
   ...: ) 
   ...:  
   ...: for col in df.columns: 
   ...:     df[col] = df[col].astype("category") 
   ...:                                                                                                 
```
On master:
```
In [2]: %timeit [df[col].astype('int') for col in df.columns]                                           
20.9 ms ± 181 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
On this branch:
```
In [4]: %timeit [df[col].astype('int') for col in df.columns]                                                                                                                                                      
18.7 ms ± 115 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
this is not showing much of an improvement. can you try someother dtypes as well.
> this is not showing much of an improvement. can you try someother dtypes as well.

There's a decent (~40x) speed up with string->float:
``` python
In [3]: import numpy as np 
   ...: import pandas as pd 
   ...:  
   ...: rng = np.random.default_rng() 
   ...:  
   ...: cats = [str(np.random.random_sample()) for _ in range(10)] 
   ...: df = pd.DataFrame( 
   ...:     rng.choice(np.array(cats), 4_000_000).reshape(1_000_000, 4), 
   ...:     columns=list("ABCD"), 
   ...: ) 
   ...:  
   ...: for col in df.columns: 
   ...:     df[col] = df[col].astype("category") 
   ...:                                                                                                   
```
On master
```
In [4]: %timeit [df[col].astype('float') for col in df.columns]                 
873 ms ± 3.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```
versus on this branch:
```
In [4]: %timeit [df[col].astype('float') for col in df.columns]                                           
23.2 ms ± 1.09 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
> > this is not showing much of an improvement. can you try someother dtypes as well.
> 
> There's a decent (~40x) speed up with string->float:
> 
> ```python
> In [3]: import numpy as np 
>    ...: import pandas as pd 
>    ...:  
>    ...: rng = np.random.default_rng() 
>    ...:  
>    ...: cats = [str(np.random.random_sample()) for _ in range(10)] 
>    ...: df = pd.DataFrame( 
>    ...:     rng.choice(np.array(cats), 4_000_000).reshape(1_000_000, 4), 
>    ...:     columns=list("ABCD"), 
>    ...: ) 
>    ...:  
>    ...: for col in df.columns: 
>    ...:     df[col] = df[col].astype("category") 
>    ...:                                                                                                   
> ```
> 
> On master
> 
> ```
> In [4]: %timeit [df[col].astype('float') for col in df.columns]                 
> 873 ms ± 3.69 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
> ```
> 
> versus on this branch:
> 
> ```
> In [4]: %timeit [df[col].astype('float') for col in df.columns]                                           
> 23.2 ms ± 1.09 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
> ```

right pls show for all dtypes (e.g parameterize the asv)
Addressed comments, waiting on CI (I suspect some test failures are real so may need to address that)
lgtm. merge master and ping on green.
@jreback CI greenish but this probably needs another review because I made a few changes since you last looked (caught edge cases, cleaned up the if/elif/else and consolidated the return)
Hello @arw2019! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:











There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-11-18 16:19:37 UTC
ci / checks failing
> ci / checks failing

Hmmm the mypy complaint looks unrelated (also getting the same thing on my other PRs)

```
mypy --version
mypy 0.782
Performing static analysis using mypy
pandas/core/indexes/datetimelike.py:776: error: Argument 1 to "_simple_new" of "DatetimeIndexOpsMixin" has incompatible type "Union[ExtensionArray, Any]"; expected "Union[DatetimeArray, TimedeltaArray, PeriodArray]"  [arg-type]
Found 1 error in 1 file (checked 1119 source files)
Performing static analysis using mypy DONE
```
Anyway will keep looking and ping when green
> > ci / checks failing
> 
> Hmmm the mypy complaint looks unrelated (also getting the same thing on my other PRs)
> 
> ```
> mypy --version
> mypy 0.782
> Performing static analysis using mypy
> pandas/core/indexes/datetimelike.py:776: error: Argument 1 to "_simple_new" of "DatetimeIndexOpsMixin" has incompatible type "Union[ExtensionArray, Any]"; expected "Union[DatetimeArray, TimedeltaArray, PeriodArray]"  [arg-type]
> Found 1 error in 1 file (checked 1119 source files)
> Performing static analysis using mypy DONE
> ```
> 
> Anyway will keep looking and ping when green

oh i think that's fixed on master
@jreback Green + addressed comment
thanks @arw2019 very nice
thanks @jreback @jbrockmendel for the reviews!