Repository: pandas-dev/pandas
Pull request: BUG: Allow cast from cat to extension dtype
- [x] closes #28668
- [x] tests added / passed
- [x] passes `black pandas`
- [x] whatsnew entry

Modifies `categorical.astype` to allow for casting to extension dtypes.  Also fixes the merge issue identified in #28668.

Problem statement:
Merge error on Categorical Interval columns
Failure on merging on Categorical columns which include intervals.
For instance, the following raises `TypeError: data type not understood`

```python
bins = np.arange(0, 91, 30)
df1 = pd.DataFrame(np.array([[1, 22], [2, 35], [3, 82]]),
                   columns=['Id', 'Dist']).set_index('Id')

df1['DistGroup'] = pd.cut(df1['Dist'], bins)

idx = pd.IntervalIndex.from_breaks(bins)
df2 = pd.DataFrame(np.array(['g1', 'g2', 'g3']), columns=['GroupId'], index=idx)
df2.index.name = 'DistGroup'

res = pd.merge(df1, df2, left_on='DistGroup', right_index=True).reset_index()

```

#### Expected Output

<table border="1" class="dataframe">  <thead>    <tr style="text-align: right;">      <th></th>      <th>Dist</th>      <th>DistGroup</th>      <th>GroupId</th>    </tr>  </thead>  <tbody>    <tr>      <th>0</th>      <td>22</td>      <td>(0, 30]</td>      <td>g1</td>    </tr>    <tr>      <th>1</th>      <td>35</td>      <td>(30, 60]</td>      <td>g2</td>    </tr>    <tr>      <th>2</th>      <td>82</td>      <td>(60, 90]</td>      <td>g3</td>    </tr>  </tbody></table>'

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.9.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.1
numpy            : 1.16.5
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.2
setuptools       : 41.0.1
Cython           : 0.29.13
pytest           : 5.0.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.3.3
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2           : 2.10.1
IPython          : 7.8.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.3
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.7
tables           : 3.5.2
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>

Hints from issue discussion:
Here's another example that shows it's actually the _order_ of the merge that triggers the error (apparently because you can cast from interval to categorical but not vice versa at the moment):

```python
import numpy as np
import pandas as pd

intvl_arr = pd.Series([pd.Interval(0, 1), pd.Interval(1, 2)], dtype="interval")
cat_arr = intvl_arr.astype("category")

df1 = pd.DataFrame({"a": intvl_arr})
df2 = pd.DataFrame({"a": cat_arr})

pd.merge(df1, df2, how="inner", on="a")
# No error
pd.merge(df2, df1, how="inner", on="a")
# TypeError: data type not understood
```

I think a code path would need to be added for casting from categorical to an interval dtype in `categorical.py` around here: https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/categorical.py#L502.  @TomAugspurger Do you have any thoughts as to the best way to do this?
This issue is a bit larger than `Categorical[Interval]` and actually applies to all `Categorial[ExtensionDtype]`, e.g. the same error will occur for periods.

The offending line is the return statement of `Categorical.astype` where we use the numpy constructor, which doesn't understand pandas extension dtypes:
https://github.com/pandas-dev/pandas/blob/c4489cbfa5f93e69301486e1ef867caed9f4eb9e/pandas/core/arrays/categorical.py#L526

The workaround would be to add a check for extension dtypes and return a `pd.array` instead, which should be able to generically handle any pandas extension dtype.  Something like:

```python
from pandas.core.construction import array


if is_extension_array_dtype(dtype):
    return array(self, dtype=dtype, copy=copy)
return np.array(self, dtype=dtype, copy=copy)
```

Commit message: Update test

Patch to review:
diff --git a/doc/source/whatsnew/v1.0.0.rst b/doc/source/whatsnew/v1.0.0.rst
index 2668734031ee1..d9d38dc1c9dc1 100644
--- a/doc/source/whatsnew/v1.0.0.rst
+++ b/doc/source/whatsnew/v1.0.0.rst
@@ -173,6 +173,7 @@ Categorical
 
 - Added test to assert the :func:`fillna` raises the correct ValueError message when the value isn't a value from categories (:issue:`13628`)
 - Bug in :meth:`Categorical.astype` where ``NaN`` values were handled incorrectly when casting to int (:issue:`28406`)
+- Bug in :meth:`Categorical.astype` not allowing for casting to extension dtypes (:issue:`28668`)
 - :meth:`Categorical.searchsorted` and :meth:`CategoricalIndex.searchsorted` now work on unordered categoricals also (:issue:`21667`)
 -
 
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
index 33d1de01fa3db..b7431033dae59 100644
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -57,7 +57,7 @@
 )
 from pandas.core.base import NoNewAttributesMixin, PandasObject, _shared_docs
 import pandas.core.common as com
-from pandas.core.construction import extract_array, sanitize_array
+from pandas.core.construction import array, extract_array, sanitize_array
 from pandas.core.missing import interpolate_2d
 from pandas.core.sorting import nargsort
 
@@ -520,6 +520,8 @@ def astype(self, dtype: Dtype, copy: bool = True) -> ArrayLike:
             if dtype == self.dtype:
                 return self
             return self._set_dtype(dtype)
+        if is_extension_array_dtype(dtype):
+            return array(self, dtype=dtype, copy=copy)  # type: ignore # GH 28770
         if is_integer_dtype(dtype) and self.isna().any():
             msg = "Cannot convert float NaN to integer"
             raise ValueError(msg)
diff --git a/pandas/tests/extension/test_categorical.py b/pandas/tests/extension/test_categorical.py
index c342777b0ebc4..bbd963ceb44e3 100644
--- a/pandas/tests/extension/test_categorical.py
+++ b/pandas/tests/extension/test_categorical.py
@@ -207,6 +207,20 @@ def test_cast_nan_to_int(self, cls, values):
         with pytest.raises((ValueError, TypeError), match=msg):
             s.astype(int)
 
+    @pytest.mark.parametrize(
+        "expected",
+        [
+            pd.Series([pd.Period("2019"), pd.Period("2020")], dtype="period[A-DEC]"),
+            pd.Series([pd.Interval(0, 1), pd.Interval(1, 2)], dtype="interval"),
+            pd.Series([1, np.nan], dtype="Int64"),
+        ],
+    )
+    def test_cast_category_to_extension_dtype(self, expected):
+        # GH 28668
+        result = expected.astype("category").astype(expected.dtype)
+
+        tm.assert_series_equal(result, expected)
+
 
 class TestArithmeticOps(base.BaseArithmeticOpsTests):
     def test_arith_series_with_scalar(self, data, all_arithmetic_operators):


Code review comments:
diff_hunk: @@ -207,6 +207,20 @@ def test_cast_nan_to_int(self, cls, values):
         with pytest.raises((ValueError, TypeError), match=msg):
             s.astype(int)
 
+    @pytest.mark.parametrize(
+        "expected",
+        [
+            pd.Series([pd.Period("2019"), pd.Period("2020")], dtype="period[A-DEC]"),
path: pandas/tests/extension/test_categorical.py
text: @user1:
can you add in a datetime w/tz, and a timedelta case

@author:
Added a datetime test but wasn't sure how to get a timedelta that'd be considered an extension dtype

@user1:
just use dtype='timedelta64'