BUG: Preserve categorical dtypes in MultiIndex levels (#13743)
State: closed

- [x] closes #13743
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

This commit modifies `MultiIndex.from_array` and `MultiIndex.from_product`.

Example:

``` python
cat = pd.Categorical(['a', 'b'], categories=list("bac"), ordered=True)
mi = pd.MultiIndex.from_arrays([cat, cat])

mi.levels[0]
Out[55]: CategoricalIndex(['b', 'a', 'c'], categories=['b', 'a', 'c'], ordered=True, dtype='category')

mi.get_level_values(0)
Out[56]: CategoricalIndex(['a', 'b'], categories=['b', 'a', 'c'], ordered=True, dtype='category')
```

Previously, the results were:

``` python
mi.levels[0]
Out[345]: Index(['b', 'a', 'c'], dtype='object')

mi.get_level_values(0)
Out[346]: Index(['a', 'b'], dtype='object')
```

This modification makes `groupby`, `pivot`, and `set_index` preserve categorical types in indexes.


Labels:
name: Bug
name: Groupby
name: MultiIndex
name: Categorical
description: Categorical Data Type

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13854?src=pr) is 85.27% (diff: 100%)

> Merging [#13854](https://codecov.io/gh/pydata/pandas/pull/13854?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will decrease coverage by **<.01%**

``` diff
@@             master     #13854   diff @@
==========================================
  Files           139        139          
  Lines         50555      50561     +6   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43111      43116     +5   
- Misses         7444       7445     +1   
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [ccec504...99e4a52](https://codecov.io/gh/pydata/pandas/compare/ccec504e31ce74f8016952ac75add1cc4bec7080...99e4a52737515d1b24473d7c37f7905f808cd7c7?src=pr)

Other places where a `categorical` dtype is lost in similar circumstances.

``` python
cidx = pd.CategoricalIndex(['y', 'x'], categories=list("xyz"), ordered=True)
cidx_nonunique = pd.CategoricalIndex(['y', 'x', 'y'], categories=list("xyz"), ordered=True)
```
1. `concat`
   
   ``` python
   df = pd.DataFrame([[10, 11, 12]])
   
   pd.concat([df, df], keys=cidx).index.levels[0]
   Out[32]: Index(['y', 'x'], dtype='object')
   ```
2. `stack` with a non-unique index/multi-index:
   
   ``` python
   df = pd.DataFrame([[10, 11, 12]], columns=cidx_nonunique)
   
   df.stack().index.levels[1]
   Out[35]: Index(['x', 'y', 'z'], dtype='object')
   ```
3. `get_dummies`
   
   ``` python
   pd.get_dummies(cidx).columns
   Out[36]: Index(['x', 'y', 'z'], dtype='object')
   ```
4. `make_axis_dummies` with `transform`
   
   ``` python
   df = pd.DataFrame([[10, 11]], columns=cidx)
   ldf = pd.Panel({'A': df, 'B': df}).to_frame()
   
   pd.core.reshape.make_axis_dummies(panel.to_frame(), transform=lambda x: x).columns
   Out[53]: Index(['x', 'y', 'z'], dtype='object')
   ```
5. `panel_index`
   
   ``` python
   pi = pd.core.panel.panel_index([0, 1, 2], cidx)
   pi.levels[1]
   Out[57]: Index(['x', 'y'], dtype='object', name='panel')
   ```
6. pytables: `LegacyTable.read()`
   No quick example yet.

@pijucha 5 & 6 you can ignore

@pijucha this should exist as a separate function from the `Categorical` constructor as a private function (but you can put in pandas.core.categorical) is prob not a bad location. maybe `_create_categoricals_from_arrays`? (or similar). E.g. its a 'categorical' function, but returns labels/levels (and not exactly a cat).

can certainly merge this fix and then do a followup with a more reaching name change. lmk.

@jreback OK, Thanks.

Sorry for a bit of delay. I fixed `stack`, `get_dummies`, `make_axis_dummies` (2-4 in the list above) and opened a separate issue for `concat` (and for two other issues I came across).

small changes. looks pretty good.

I moved some tests to files I thought were more appropriate (though, I'm not 100% sure).

update

@pijucha test split looks good.

@sinhrks any comments?

Deprecated `Categorical.from_array`.

update

lgtm. @sinhrks @jorisvandenbossche 

@pijucha can you rebase

@jreback Done (rebase + small update to tests/test_reshape.py).

One build on travis has probably stalled - should I resubmit?

@pijucha i restarted the build. ping when green.

thanks @pijucha really nice PR. touches lots of parts!