Repository: pandas-dev/pandas
Pull request: REGR: Prevent indexes that aren't directly backed by numpy from entering libreduction code paths
- [X] closes #31223
- [X] closes #31248 
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

No whatsnew since this is a regression.

Problem statement:
GroupBy aggregation fails if DataFrame has CategoricalIndex 
#### Code Sample, a copy-pastable example if possible

```python
ids = pd.Categorical([0, 1, 2])
df = pd.DataFrame({
    'id': ids,
    'groups': [1, 1, 2],
    'value': [0, 1, 0]
}).set_index('id')

df.groupby('groups').agg({'value': pd.Series.nunique})

```
#### Problem description
The above works in v. `0.25.3` but not in `1.0.0rc0`. It fails with: `TypeError: Cannot convert Categorical to numpy.ndarray` 

Full stack trace:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/groupby/generic.py", line 940, in aggregate
    result, how = self._aggregate(func, *args, **kwargs)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/base.py", line 430, in _aggregate
    result = _agg(arg, _agg_1dim)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/base.py", line 397, in _agg
    result[fname] = func(fname, agg_how)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/base.py", line 381, in _agg_1dim
    return colg.aggregate(how)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/groupby/generic.py", line 265, in aggregate
    return self._python_agg_general(func, *args, **kwargs)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/groupby/groupby.py", line 935, in _python_agg_general
    result, counts = self.grouper.agg_series(obj, f)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/groupby/ops.py", line 624, in agg_series
    return self._aggregate_series_fast(obj, func)
  File "~/.virtualenvs/3.6-test-pd-1.0.0/lib/python3.6/site-packages/pandas/core/groupby/ops.py", line 648, in _aggregate_series_fast
    grouper = libreduction.SeriesGrouper(obj, func, group_index, ngroups, dummy)
  File "pandas/_libs/reduction.pyx", line 329, in pandas._libs.reduction.SeriesGrouper.__init__
TypeError: Cannot convert Categorical to numpy.ndarray
```
#### Expected Output
The correct result applying the aggregation function to the DataFrameGroupBy

#### Output of ``pd.show_versions()``


<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.6.9.final.0
python-bits      : 64
OS               : Darwin
OS-release       : 18.7.0
machine          : x86_64
processor        : i386
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.0rc0
numpy            : 1.18.1
pytz             : 2019.3
dateutil         : 2.8.1
pip              : 20.0.1
setuptools       : 45.1.0
Cython           : None
pytest           : 5.2.0
hypothesis       : None
sphinx           : 2.0.1
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.10.3
IPython          : 7.2.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : 0.3.2
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.0.2
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
pytest           : 5.2.0
s3fs             : 0.4.0
scipy            : 1.4.1
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : 0.47.0
</details>


REGR: CategoricalIndex and IntervalIndex are missing _index_data attribute
It looks like the `_index_data` attribute wasn't added to `CategoricalIndex` or `IntervalIndex`.  This leads to some regressions with `GroupBy.apply`.

Not sure what the fix should be, as there isn't a singular array that directly backs the underlying values for `CategoricalIndex` or `IntervalIndex`, so the approach used to add `_index_data` for other types doesn't look like it will work here.

Example of broken behavior on `master`:
```python
In [1]: import pandas as pd; pd.__version__
Out[1]: '1.0.0rc0+146.g6f395ad42'

In [2]: index = pd.CategoricalIndex(list("abc"))  
   ...: df = pd.DataFrame({"group": [0, 0, 1], "value": [1, 2, 3]}, index=index)

In [3]: df.groupby("group").apply(lambda x: x)
---------------------------------------------------------------------------
AttributeError: 'CategoricalIndex' object has no attribute '_index_data'

```

Same behavior worked on 0.25.3:
```python
In [1]: import pandas as pd; pd.__version__
Out[1]: '0.25.3'

In [2]: index = pd.CategoricalIndex(list("abc"))  
   ...: df = pd.DataFrame({"group": [0, 0, 1], "value": [1, 2, 3]}, index=index)

In [3]: df.groupby("group").apply(lambda x: x)
Out[3]: 
   group  value
a      0      1
b      0      2
c      1      3
```
Swapping `CategoricalIndex` with `IntervalIndex` results in the same behavior above.

Hints from issue discussion:
Thanks, I can confirm that this is broken on `master` and was working on 0.25.3.
Looks like this is a little more generic than `Categorical` and occurs for any index that's backed by an extension array, e.g. `ids = pd.interval_range(0, 3)` causes the same error.

The following changes look to fix the issue but I haven't checked that it doesn't end up breaking something else:
```diff
diff --git a/pandas/_libs/reduction.pyx b/pandas/_libs/reduction.pyx
index 8571761f7..635c0e36d 100644
--- a/pandas/_libs/reduction.pyx
+++ b/pandas/_libs/reduction.pyx
@@ -158,7 +158,7 @@ cdef class _BaseGrouper:
         if util.is_array(values) and not values.flags.contiguous:
             # e.g. Categorical has no `flags` attribute
             values = values.copy()
-        index = dummy.index.values
+        index = dummy.index.to_numpy()
         if not index.flags.contiguous:
             index = index.copy()
 
@@ -229,7 +229,7 @@ cdef class SeriesBinGrouper(_BaseGrouper):
         self.arr = values
         self.typ = series._constructor
         self.ityp = series.index._constructor
-        self.index = series.index.values
+        self.index = series.index.to_numpy()
         self.name = series.name
 
         self.dummy_arr, self.dummy_index = self._check_dummy(dummy)
@@ -326,7 +326,7 @@ cdef class SeriesGrouper(_BaseGrouper):
         self.arr = values
         self.typ = series._constructor
         self.ityp = series.index._constructor
-        self.index = series.index.values
+        self.index = series.index.to_numpy()
         self.name = series.name
 
         self.dummy_arr, self.dummy_index = self._check_dummy(dummy)
```

Commit message: REGR: Fix GroupBy aggregation with ExtensionArray backed index

Patch to review:
diff --git a/pandas/_libs/reduction.pyx b/pandas/_libs/reduction.pyx
index 8571761f77265..635c0e36d7457 100644
--- a/pandas/_libs/reduction.pyx
+++ b/pandas/_libs/reduction.pyx
@@ -158,7 +158,7 @@ cdef class _BaseGrouper:
         if util.is_array(values) and not values.flags.contiguous:
             # e.g. Categorical has no `flags` attribute
             values = values.copy()
-        index = dummy.index.values
+        index = dummy.index.to_numpy()
         if not index.flags.contiguous:
             index = index.copy()
 
@@ -229,7 +229,7 @@ cdef class SeriesBinGrouper(_BaseGrouper):
         self.arr = values
         self.typ = series._constructor
         self.ityp = series.index._constructor
-        self.index = series.index.values
+        self.index = series.index.to_numpy()
         self.name = series.name
 
         self.dummy_arr, self.dummy_index = self._check_dummy(dummy)
@@ -326,7 +326,7 @@ cdef class SeriesGrouper(_BaseGrouper):
         self.arr = values
         self.typ = series._constructor
         self.ityp = series.index._constructor
-        self.index = series.index.values
+        self.index = series.index.to_numpy()
         self.name = series.name
 
         self.dummy_arr, self.dummy_index = self._check_dummy(dummy)
diff --git a/pandas/tests/groupby/aggregate/test_aggregate.py b/pandas/tests/groupby/aggregate/test_aggregate.py
index 0a7272bbc131c..3629b0b52b3aa 100644
--- a/pandas/tests/groupby/aggregate/test_aggregate.py
+++ b/pandas/tests/groupby/aggregate/test_aggregate.py
@@ -360,6 +360,17 @@ def test_func_duplicates_raises():
         df.groupby("A").agg(["min", "min"])
 
 
+@pytest.mark.parametrize(
+    "index", [pd.CategoricalIndex(list("abc")), pd.interval_range(0, 3)]
+)
+def test_agg_with_ea_backed_index(index):
+    # GH 31223
+    df = DataFrame({"group": [1, 1, 2], "value": [0, 1, 0]}, index=index)
+    result = df.groupby("group").agg({"value": Series.nunique})
+    expected = DataFrame({"group": [1, 2], "value": [2, 1]}).set_index("group")
+    tm.assert_frame_equal(result, expected)
+
+
 class TestNamedAggregationSeries:
     def test_series_named_agg(self):
         df = pd.Series([1, 2, 3, 4])


Code review comments:
diff_hunk: @@ -158,7 +158,7 @@ cdef class _BaseGrouper:
         if util.is_array(values) and not values.flags.contiguous:
             # e.g. Categorical has no `flags` attribute
             values = values.copy()
-        index = dummy.index.values
+        index = dummy.index.to_numpy()
path: pandas/_libs/reduction.pyx
text: @user1:
for EA cases, we want to avoid going through libreduction altogether

@user2:
What do you mean? We shouldn't be hitting this code path in the first place?

@user3:
hmm, yeah what does this do to the categorical grouping case. This should be a highly optimized path, is this still?

@user1:
> What do you mean? We shouldn't be hitting this code path in the first place?

Correct.  This is called from core.apply.FrameApply.apply_standard after a check:

```
        if (
            self.result_type in ["reduce", None]
            and not self.dtypes.apply(is_extension_array_dtype).any()
            # Disallow complex_internals since libreduction shortcut
            #  cannot handle MultiIndex
            and not isinstance(self.agg_axis, ABCMultiIndex)
        ):
```

The MultiIndex check may need to be expanded to include EAs.

I think this is called from one other place in non-test code, so the missing check may belong there.

@user2:
I vaguely recall a `_has_complex_...` property that indicated with the Index was backed by an ndarray. Was that removed?

@user1:
`self._has_complex_internals` was equivalent to to `isinstance(self, MultiIndex)`

@author:
The issue in question doesn't actually go through `FrameApply.apply_standard`, but rather `_aggregate_series_fast`, which dispatches through `libreduction`. The point still applies that we want to avoid EA backed indexes in `FrameApply.apply_standard`, so I've modified the check to use `_has_complex_internals`.