Repository: pandas-dev/pandas

Issue:
Merge error on Categorical Interval columns
Failure on merging on Categorical columns which include intervals.
For instance, the following raises `TypeError: data type not understood`

```python
bins = np.arange(0, 91, 30)
df1 = pd.DataFrame(np.array([[1, 22], [2, 35], [3, 82]]),
                   columns=['Id', 'Dist']).set_index('Id')

df1['DistGroup'] = pd.cut(df1['Dist'], bins)

idx = pd.IntervalIndex.from_breaks(bins)
df2 = pd.DataFrame(np.array(['g1', 'g2', 'g3']), columns=['GroupId'], index=idx)
df2.index.name = 'DistGroup'

res = pd.merge(df1, df2, left_on='DistGroup', right_index=True).reset_index()

```

#### Expected Output

<table border="1" class="dataframe">  <thead>    <tr style="text-align: right;">      <th></th>      <th>Dist</th>      <th>DistGroup</th>      <th>GroupId</th>    </tr>  </thead>  <tbody>    <tr>      <th>0</th>      <td>22</td>      <td>(0, 30]</td>      <td>g1</td>    </tr>    <tr>      <th>1</th>      <td>35</td>      <td>(30, 60]</td>      <td>g2</td>    </tr>    <tr>      <th>2</th>      <td>82</td>      <td>(60, 90]</td>      <td>g3</td>    </tr>  </tbody></table>'

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.9.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.1
numpy            : 1.16.5
pytz             : 2019.2
dateutil         : 2.8.0
pip              : 19.2.2
setuptools       : 41.0.1
Cython           : 0.29.13
pytest           : 5.0.1
hypothesis       : None
sphinx           : None
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : 4.3.3
html5lib         : 1.0.1
pymysql          : None
psycopg2         : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2           : 2.10.1
IPython          : 7.8.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : 4.3.3
matplotlib       : 3.1.1
numexpr          : 2.7.0
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.1
sqlalchemy       : 1.3.7
tables           : 3.5.2
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None

</details>



Discussion:
Here's another example that shows it's actually the _order_ of the merge that triggers the error (apparently because you can cast from interval to categorical but not vice versa at the moment):

```python
import numpy as np
import pandas as pd

intvl_arr = pd.Series([pd.Interval(0, 1), pd.Interval(1, 2)], dtype="interval")
cat_arr = intvl_arr.astype("category")

df1 = pd.DataFrame({"a": intvl_arr})
df2 = pd.DataFrame({"a": cat_arr})

pd.merge(df1, df2, how="inner", on="a")
# No error
pd.merge(df2, df1, how="inner", on="a")
# TypeError: data type not understood
```

I think a code path would need to be added for casting from categorical to an interval dtype in `categorical.py` around here: https://github.com/pandas-dev/pandas/blob/master/pandas/core/arrays/categorical.py#L502.  @TomAugspurger Do you have any thoughts as to the best way to do this?
This issue is a bit larger than `Categorical[Interval]` and actually applies to all `Categorial[ExtensionDtype]`, e.g. the same error will occur for periods.

The offending line is the return statement of `Categorical.astype` where we use the numpy constructor, which doesn't understand pandas extension dtypes:
https://github.com/pandas-dev/pandas/blob/c4489cbfa5f93e69301486e1ef867caed9f4eb9e/pandas/core/arrays/categorical.py#L526

The workaround would be to add a check for extension dtypes and return a `pd.array` instead, which should be able to generically handle any pandas extension dtype.  Something like:

```python
from pandas.core.construction import array


if is_extension_array_dtype(dtype):
    return array(self, dtype=dtype, copy=copy)
return np.array(self, dtype=dtype, copy=copy)
```

Resolving patch:
diff --git a/pandas/core/arrays/categorical.py b/pandas/core/arrays/categorical.py
--- a/pandas/core/arrays/categorical.py
+++ b/pandas/core/arrays/categorical.py
@@ -57,7 +57,7 @@
 )
 from pandas.core.base import NoNewAttributesMixin, PandasObject, _shared_docs
 import pandas.core.common as com
-from pandas.core.construction import extract_array, sanitize_array
+from pandas.core.construction import array, extract_array, sanitize_array
 from pandas.core.missing import interpolate_2d
 from pandas.core.sorting import nargsort
 
@@ -520,6 +520,8 @@ def astype(self, dtype: Dtype, copy: bool = True) -> ArrayLike:
             if dtype == self.dtype:
                 return self
             return self._set_dtype(dtype)
+        if is_extension_array_dtype(dtype):
+            return array(self, dtype=dtype, copy=copy)  # type: ignore # GH 28770
         if is_integer_dtype(dtype) and self.isna().any():
             msg = "Cannot convert float NaN to integer"
             raise ValueError(msg)


Test patch:
diff --git a/pandas/tests/extension/test_categorical.py b/pandas/tests/extension/test_categorical.py
--- a/pandas/tests/extension/test_categorical.py
+++ b/pandas/tests/extension/test_categorical.py
@@ -207,6 +207,22 @@ def test_cast_nan_to_int(self, cls, values):
         with pytest.raises((ValueError, TypeError), match=msg):
             s.astype(int)
 
+    @pytest.mark.parametrize(
+        "expected",
+        [
+            pd.Series(["2019", "2020"], dtype="datetime64[ns, UTC]"),
+            pd.Series([0, 0], dtype="timedelta64[ns]"),
+            pd.Series([pd.Period("2019"), pd.Period("2020")], dtype="period[A-DEC]"),
+            pd.Series([pd.Interval(0, 1), pd.Interval(1, 2)], dtype="interval"),
+            pd.Series([1, np.nan], dtype="Int64"),
+        ],
+    )
+    def test_cast_category_to_extension_dtype(self, expected):
+        # GH 28668
+        result = expected.astype("category").astype(expected.dtype)
+
+        tm.assert_series_equal(result, expected)
+
 
 class TestArithmeticOps(base.BaseArithmeticOpsTests):
     def test_arith_series_with_scalar(self, data, all_arithmetic_operators):
diff --git a/pandas/tests/reshape/merge/test_merge.py b/pandas/tests/reshape/merge/test_merge.py
--- a/pandas/tests/reshape/merge/test_merge.py
+++ b/pandas/tests/reshape/merge/test_merge.py
@@ -2096,6 +2096,20 @@ def test_merge_equal_cat_dtypes2():
     tm.assert_frame_equal(result, expected, check_categorical=False)
 
 
+def test_merge_on_cat_and_ext_array():
+    # GH 28668
+    right = DataFrame(
+        {"a": Series([pd.Interval(0, 1), pd.Interval(1, 2)], dtype="interval")}
+    )
+    left = right.copy()
+    left["a"] = left["a"].astype("category")
+
+    result = pd.merge(left, right, how="inner", on="a")
+    expected = right.copy()
+
+    assert_frame_equal(result, expected)
+
+
 def test_merge_multiindex_columns():
     # Issue #28518
     # Verify that merging two dataframes give the expected labels