Repository: pandas-dev/pandas

Issue:
groupby on 2 categorical columns, when one categorical is based on datetimes, incorrectly returns all NaN dataframe
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np

df = pd.DataFrame({
    'label1': list('abcbabcba'),
    'label2': list('xyxyxyxyx'),
    'minute': list(pd.date_range('2018-06-01 00', freq='1T', periods=3)) * 3,
    'n1': np.arange(9, dtype='float'),
    'n2': np.arange(9, dtype='float') ** 2
})

# this is correct
df.groupby(['label1', 'minute'])[['n1', 'n2']].mean()

# convert to categoricals
df['label1'] = df['label1'].astype('category')
df['label2'] = df['label2'].astype('category')
df['minute'] = df['minute'].astype('category')

# this is wrong, returns all NaNs
df.groupby(['label1', 'minute'])[['n1', 'n2']].mean()
```

#### Problem description

When grouping by [str, datetime] columns, results are as expected:

```
>>> df.groupby(['label1', 'minute'])[['n1', 'n2']].mean()
                             n1    n2
label1 minute                        
a      2018-06-01 00:00:00  0.0   0.0
       2018-06-01 00:01:00  4.0  16.0
       2018-06-01 00:02:00  8.0  64.0
b      2018-06-01 00:00:00  3.0   9.0
       2018-06-01 00:01:00  4.0  25.0
       2018-06-01 00:02:00  5.0  25.0
c      2018-06-01 00:00:00  6.0  36.0
       2018-06-01 00:02:00  2.0   4.0
```

After converting `label1`, `label2`, and `minute` to categoricals, that same groupby returns all NaNs:

```
>>> df.groupby(['label1', 'minute'])[['n1', 'n2']].mean()
                            n1  n2
label1 minute                     
a      2018-06-01 00:00:00 NaN NaN
       2018-06-01 00:01:00 NaN NaN
       2018-06-01 00:02:00 NaN NaN
b      2018-06-01 00:00:00 NaN NaN
       2018-06-01 00:01:00 NaN NaN
       2018-06-01 00:02:00 NaN NaN
c      2018-06-01 00:00:00 NaN NaN
       2018-06-01 00:01:00 NaN NaN
       2018-06-01 00:02:00 NaN NaN
```

I only got this bug when grouping on 2 categoricals with one of them being datetime based (order is irrelevant).  Grouping by `['label1', 'label2']` and `'minute'` by itself works as expected.

#### Output of ``pd.show_versions()``

<details>

[paste the output of ``pd.show_versions()`` here below this line]
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-22-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.0
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.14.4
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: 3.4.3
numexpr: 2.6.5
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 1.0.5
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

</details>



Discussion:
Looks very similar to #21334 

cc @jreback 
This seems to boil down to a problem with reindexing with such a categorical index:

```
idx = pd.MultiIndex.from_product([pd.Categorical(['a', 'b', 'c']), pd.Categorical(pd.date_range("2012-01-01", periods=3, freq='H'))])
df = pd.DataFrame({'a': range(len(idx))}, index=idx)
df2 = df.iloc[[0, 1, 2, 3, 4, 5, 6, 8]]
df2.reindex(idx)
```

on 0.22.0 works correctly, but on master gives:

```
In [23]: df2.reindex(idx)
Out[23]: 
                        a
a 2012-01-01 00:00:00 NaN
  2012-01-01 01:00:00 NaN
  2012-01-01 02:00:00 NaN
b 2012-01-01 00:00:00 NaN
  2012-01-01 01:00:00 NaN
  2012-01-01 02:00:00 NaN
c 2012-01-01 00:00:00 NaN
  2012-01-01 01:00:00 NaN
  2012-01-01 02:00:00 NaN
```

cc @toobaz this seems to be related to the new MultiIndexUIntEngine

Using the above example (`idx` is a MultiIndex):

```
In [6]: idx._engine.get_indexer(idx)
Out[6]: array([-1, -1, -1, -1, -1, -1, -1, -1, -1])
```
Sorry Pietro, probably a bit prematurely pointed to that :-), as in the end it is code in the MultiIndexUIntEngine that surfaces another bug. Iterating a MultiIndex (``tolist``) with categorical datetime is broken (but was already broken in 0.22.0, just now surfaces through the use in the multiindex engine):

```
In [21]: list(idx)
Out[21]: 
[('a', 1325376000000000000),
 ('a', 1325379600000000000),
 ('a', 1325383200000000000),
 ('b', 1325376000000000000),
 ('b', 1325379600000000000),
 ('b', 1325383200000000000),
 ('c', 1325376000000000000),
 ('c', 1325379600000000000),
 ('c', 1325383200000000000)]

In [22]: list(idx.get_level_values(1))
Out[22]: 
[Timestamp('2012-01-01 00:00:00'),
 Timestamp('2012-01-01 01:00:00'),
 Timestamp('2012-01-01 02:00:00'),
 Timestamp('2012-01-01 00:00:00'),
 Timestamp('2012-01-01 01:00:00'),
 Timestamp('2012-01-01 02:00:00'),
 Timestamp('2012-01-01 00:00:00'),
 Timestamp('2012-01-01 01:00:00'),
 Timestamp('2012-01-01 02:00:00')]
```
> Sorry Pietro, probably a bit prematurely pointed to that :-)

Good :-) In general, it is unlikely that bugs in the MI engine code are dtype-specific, as it entirely delegates actual lookup to single levels, and only looks for integers (codes).

Resolving patch:
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -11,6 +11,8 @@
 from pandas.compat.numpy import function as nv
 from pandas import compat
 
+from pandas.core.dtypes.dtypes import (
+    ExtensionDtype, PandasExtensionDtype)
 from pandas.core.dtypes.common import (
     _ensure_int64,
     _ensure_platform_int,
@@ -807,20 +809,16 @@ def values(self):
             return self._tuples
 
         values = []
-        for lev, lab in zip(self.levels, self.labels):
-            # Need to box timestamps, etc.
-            box = hasattr(lev, '_box_values')
-            # Try to minimize boxing.
-            if box and len(lev) > len(lab):
-                taken = lev._box_values(algos.take_1d(lev._ndarray_values,
-                                                      lab))
-            elif box:
-                taken = algos.take_1d(lev._box_values(lev._ndarray_values),
-                                      lab,
-                                      fill_value=lev._na_value)
-            else:
-                taken = algos.take_1d(np.asarray(lev._values), lab)
-            values.append(taken)
+
+        for i in range(self.nlevels):
+            vals = self._get_level_values(i)
+            if is_categorical_dtype(vals):
+                vals = vals.get_values()
+            if (isinstance(vals.dtype, (PandasExtensionDtype, ExtensionDtype))
+                    or hasattr(vals, '_box_values')):
+                vals = vals.astype(object)
+            vals = np.array(vals, copy=False)
+            values.append(vals)
 
         self._tuples = lib.fast_zip(values)
         return self._tuples


Test patch:
diff --git a/pandas/tests/frame/test_axis_select_reindex.py b/pandas/tests/frame/test_axis_select_reindex.py
--- a/pandas/tests/frame/test_axis_select_reindex.py
+++ b/pandas/tests/frame/test_axis_select_reindex.py
@@ -10,7 +10,7 @@
 import numpy as np
 
 from pandas.compat import lrange, lzip, u
-from pandas import (compat, DataFrame, Series, Index, MultiIndex,
+from pandas import (compat, DataFrame, Series, Index, MultiIndex, Categorical,
                     date_range, isna)
 import pandas as pd
 
@@ -1129,6 +1129,19 @@ def test_reindex_multi(self):
 
         assert_frame_equal(result, expected)
 
+    def test_reindex_multi_categorical_time(self):
+        # https://github.com/pandas-dev/pandas/issues/21390
+        midx = pd.MultiIndex.from_product(
+            [Categorical(['a', 'b', 'c']),
+             Categorical(date_range("2012-01-01", periods=3, freq='H'))])
+        df = pd.DataFrame({'a': range(len(midx))}, index=midx)
+        df2 = df.iloc[[0, 1, 2, 3, 4, 5, 6, 8]]
+
+        result = df2.reindex(midx)
+        expected = pd.DataFrame(
+            {'a': [0, 1, 2, 3, 4, 5, 6, np.nan, 8]}, index=midx)
+        assert_frame_equal(result, expected)
+
     data = [[1, 2, 3], [1, 2, 3]]
 
     @pytest.mark.parametrize('actual', [
diff --git a/pandas/tests/groupby/test_categorical.py b/pandas/tests/groupby/test_categorical.py
--- a/pandas/tests/groupby/test_categorical.py
+++ b/pandas/tests/groupby/test_categorical.py
@@ -850,3 +850,23 @@ def test_empty_prod():
     result = df.groupby("A", observed=False).B.prod(min_count=1)
     expected = pd.Series([2, 1, np.nan], expected_idx, name='B')
     tm.assert_series_equal(result, expected)
+
+
+def test_groupby_multiindex_categorical_datetime():
+    # https://github.com/pandas-dev/pandas/issues/21390
+
+    df = pd.DataFrame({
+        'key1': pd.Categorical(list('abcbabcba')),
+        'key2': pd.Categorical(
+            list(pd.date_range('2018-06-01 00', freq='1T', periods=3)) * 3),
+        'values': np.arange(9),
+    })
+    result = df.groupby(['key1', 'key2']).mean()
+
+    idx = pd.MultiIndex.from_product(
+        [pd.Categorical(['a', 'b', 'c']),
+         pd.Categorical(pd.date_range('2018-06-01 00', freq='1T', periods=3))],
+        names=['key1', 'key2'])
+    expected = pd.DataFrame(
+        {'values': [0, 4, 8, 3, 4, 5, 6, np.nan, 2]}, index=idx)
+    assert_frame_equal(result, expected)
diff --git a/pandas/tests/indexes/test_multi.py b/pandas/tests/indexes/test_multi.py
--- a/pandas/tests/indexes/test_multi.py
+++ b/pandas/tests/indexes/test_multi.py
@@ -12,8 +12,8 @@
 
 import pandas as pd
 
-from pandas import (CategoricalIndex, DataFrame, Index, MultiIndex,
-                    compat, date_range, period_range)
+from pandas import (CategoricalIndex, Categorical, DataFrame, Index,
+                    MultiIndex, compat, date_range, period_range)
 from pandas.compat import PY3, long, lrange, lzip, range, u, PYPY
 from pandas.errors import PerformanceWarning, UnsortedIndexError
 from pandas.core.dtypes.dtypes import CategoricalDtype
@@ -1596,6 +1596,14 @@ def test_get_indexer_nearest(self):
         with pytest.raises(NotImplementedError):
             midx.get_indexer(['a'], method='pad', tolerance=2)
 
+    def test_get_indexer_categorical_time(self):
+        # https://github.com/pandas-dev/pandas/issues/21390
+        midx = MultiIndex.from_product(
+            [Categorical(['a', 'b', 'c']),
+             Categorical(date_range("2012-01-01", periods=3, freq='H'))])
+        result = midx.get_indexer(midx)
+        tm.assert_numpy_array_equal(result, np.arange(9, dtype=np.intp))
+
     def test_hash_collisions(self):
         # non-smoke test that we don't get hash collisions