ENH: support CategoricalIndex (GH7629)
State: closed

closes #7629
xref #8613 
xref #8074 
- [x] docs / whatsnew
- [x] ~~auto-create a `CategoricalIndex` when grouping by a `Categorical` (this doesn't ATM)~~
- [x] adding a value not in the Index, e.g. `df2.loc['d'] = 5` should do what? (currently will coerce to an `Index`)
- [x] `pd.concat([df2,df])` should STILL have a `CategoricalIndex` (yep)?
- [x] implement `min/max`
- [x] fix groupby on cat column
- [x] add `Categorical` wrapper methods
- [x] make repr evalable / fix
- [x] contains should be on values not categories

A `CategoricalIndex` is essentially a drop-in replacement for `Index`, that works nicely for non-unique values. It uses a `Categorical` to represent itself. The behavior is very similar to using a duplicated Index (for say indexing).

Groupby works naturally (and returns another `CategoricalIndex`). The only real departure is that `.sort_index()` works like you would expected (which is a good thing:). Clearly this will provide idempotency for `set/reset` index w.r.t. Categoricals, and thus memory savings by its representation.

This doesn't change the API at all. IOW, this is not turned on by default, you have to either use `set/reset`, assign an index, or pass a `Categorical` to `Index`.

```
In [1]: df = DataFrame({'A' : np.arange(6,dtype='int64'),
   ...:                         'B' : Series(list('aabbca')).astype('category',categories=list('cab')) })

In [2]: df
Out[2]: 
   A  B
0  0  a
1  1  a
2  2  b
3  3  b
4  4  c
5  5  a

In [3]: df.dtypes
Out[3]: 
A       int64
B    category
dtype: object

In [5]: df.B.cat.categories
Out[5]: Index([u'c', u'a', u'b'], dtype='object')

In [6]: df2 = df.set_index('B')
In [7]: df2
Out[7]: 
   A
B   
a  0
a  1
b  2
b  3
c  4
a  5

In [8]: df2.index
Out[8]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], dtype='category')

In [9]: df2.index.categories
Out[9]: Index([u'c', u'a', u'b'], dtype='object')

In [10]: df2.index.codes     
Out[10]: array([1, 1, 2, 2, 0, 1], dtype=int8)

In [11]: df2.loc['a']
Out[11]: 
   A
B   
a  0
a  1
a  5

In [12]: df2.loc['a'].index 
Out[12]: CategoricalIndex([u'a', u'a', u'a'], dtype='category')

In [13]: df2.loc['a'].index.categories
Out[13]: Index([u'c', u'a', u'b'], dtype='object')

In [14]: df2.sort_index() 
Out[14]: 
   A
B   
c  4
a  0
a  1
a  5
b  2
b  3

In [15]: df2.groupby(level=0).sum()
Out[15]: 
   A
B   
a  6
b  5
c  4

In [16]: df2.groupby(level=0).sum().index
Out[16]: CategoricalIndex([u'a', u'b', u'c'], dtype='category')
```


Labels:
name: Enhancement
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Categorical
description: Categorical Data Type

Comments:
cc @TomAugspurger 
cc @jorisvandenbossche 
cc @JanSchulz
cc @shoyer 
cc @mrocklin

So the main point to note here is that we don't have the concept of `ordered` in a `CategoricalIndex`. (if you pass it its a `ValueError`), because by-definition an `Index` IS ordered.

This is actually a good thing, it makes the entire discussion we had w.r.t. to the groupby sort issue moot (well if you are grouping by a `CategoricalIndex` anyhow)

are there operations that currently return say an `Index` that really should return a `CategoricalIndex`. e.g. similary to how say `pd.cut` should return a `IntervalIndex` (when @shoyer finishes :)

@jreback  What happens if I try to insert new values into a categorical index that aren't already in the categories?

@shoyer 

hmm, I think 

`df2.loc['d'] = 6` should raise (broken now)

```
In [24]: pd.concat([df2,df2])
Out[24]: 
   A
B   
a  0
a  1
b  2
b  3
c  4
a  6
a  0
a  1
b  2
b  3
c  4
a  6

In [25]: pd.concat([df2,df2]).index
Out[25]: Index([u'a', u'a', u'b', u'b', u'c', u'a', u'a', u'a', u'b', u'b', u'c', u'a'], dtype='object')
```

This I could make work, but IIRC we decided to have this merge the categories (though in this case they are the same)....hmmm

```
In [8]: df2 = DataFrame({'A' : np.arange(6,dtype='int64'),
   ...:                         'B' : Series(list('aabbca')).astype('category',categories=list('cab')) }).set_index('B')

In [9]: df2
Out[9]: 
   A
B   
a  0
a  1
b  2
b  3
c  4
a  5

In [10]: df2.index        
Out[10]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], dtype='category')

In [11]: df2.loc['d'] = 10

In [12]: df2
Out[12]: 
    A
a   0
a   1
b   2
b   3
c   4
a   5
d  10

In [13]: df2.index
Out[13]: Index([u'a', u'a', u'b', u'b', u'c', u'a', u'd'], dtype='object')

In [14]: df2.index = pd.CategoricalIndex(df2.index)

In [15]: df2.index
Out[15]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a', u'd'], dtype='category')

In [16]: df2.index.categories
Out[16]: Index([u'a', u'b', u'c', u'd'], dtype='object')
```

so the appending operation works, but converts you to a `Index` (and you lose the ordering by-definition)

@TomAugspurger 

I now dispatch to Categorical for comparisons, providing conversions for Index and Categorical, but they must match in categories/ordered

```
In [4]: ci1 = CategoricalIndex(['a', 'b'], categories=['a', 'b'])

In [5]: ci2 = CategoricalIndex(['a', 'b'], categories=['a', 'b', 'c'])

In [6]: ci1.equals(ci1)
Out[6]: True

# this is always safe (e.g. it returns boolean)
In [7]: ci1.equals(ci2)
Out[7]: False

In [8]: ci1==ci1
Out[8]: array([ True,  True], dtype=bool)

# not the same categories
In [9]: ci1==ci2
TypeError: categorical index comparisions must have the same categories

# scalar comparison
In [10]: ci1=='a'
Out[10]: array([ True, False], dtype=bool)

# ok as all values are in the categories
In [11]: ci1==Index(['a','b'])
Out[11]: array([ True,  True], dtype=bool)

# cannot compare vs unordered
In [13]: ci1 == pd.Categorical(ci1.values, ordered=False)
TypeError: categorical index comparisions must be ordered

# this is not allowed because 'c' is not a category
In [14]: ci1==Index(['a','b','c'])
TypeError: cannot compare versus non-convertible Index type
```

I fixed the concat issues to be the same as concating columns (e.g. non-matching categories make this raise).

```
In [15]:  a = Series(np.arange(6,dtype='int64'))

In [16]:  b = Series(list('aabbca'))

In [17]: df2 = DataFrame({'A' : a, 'B' : b.astype('category',categories=list('cab')) }).set_index('B')

# concat of same-categories is good
In [22]: df2.index
Out[22]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], dtype='category')

In [24]: pd.concat([df2,df2]).index
Out[24]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a', u'a', u'a', u'b', u'b', u'c', u'a'], dtype='category')

# concat of not-same categories is an error
In [25]: df3 = DataFrame({'A' : a, 'B' : b.astype('category',categories=list('abc')) }).set_index('B')

In [26]: df3.index.categories
Out[26]: Index([u'a', u'b', u'c'], dtype='object')

In [27]: df2.index.categories
Out[27]: Index([u'c', u'a', u'b'], dtype='object')

In [28]: pd.concat([df2,df3])
TypeError: categories must match existing categories when appending
```

What about Categorical levels in a MultiIndex?

good repr?
remeniscent of a MultiIndex repr

```
In [1]: pd.Index(pd.Categorical(list('aabca'),categories=list('cab')))
Out[1]: 
CategoricalIndex(categories=[u'c', u'a', u'b'],
                 codes=[1, 1, 2, 0, 1],
                 ordered=False)
```

```
In [9]: df = DataFrame({'A' : np.arange(6), 'B' : Series(['foo']*3 + ['bar'] *3).astype('category'), 'C' : Series([1]*6).astype('category',categories=[1,2,3],ordered=True) })

In [10]: df
Out[10]: 
   A    B  C
0  0  foo  1
1  1  foo  1
2  2  foo  1
3  3  bar  1
4  4  bar  1
5  5  bar  1

In [15]: df.dtypes
Out[15]: 
A       int64
B    category
C    category
dtype: object

In [11]: df.set_index(['B','C'])
Out[11]: 
       A
B   C   
foo 1  0
    1  1
    1  2
bar 1  3
    1  4
    1  5

In [12]: df.set_index(['B','C']).index
Out[12]: 
MultiIndex(levels=[[u'bar', u'foo'], [1, 2, 3]],
           labels=[[1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 0]],
           names=[u'B', u'C'])
```

```
In [13]: df.set_index(['B','C']).index.levels[0]
Out[13]: Index([u'bar', u'foo'], dtype='object')

In [14]: df.set_index(['B','C']).index.levels[1]
Out[14]: Int64Index([1, 2, 3], dtype='int64')
```

So setting a `Categorical` into a level of a `MultiIndex` coerces them ATM. A multi-index is really just a Categorical of the levels (categories) and labels (codes), with an array of these, basically an array of 1-d Categoricals, but they are not represented this way internally (they ARE created this way then separated, I imagine as the Categorical was not fully fleshed out at the time).

What _could_ be interesting is to set the sortorder (of that level?) via a passed in `Categorical/CategoricalIndex`, which would sort in the defined order (rather than lexically).

Would it be a good idea to implement  `slice_locs` for the subclass? I guess that should work for an ordered categorical, but not an unordered one?

What happens when you do `.loc` indexing on a frame with a categorical index...
1. with a list of existing labels?
2. with a list mixing existing and new labels?
3. for assignment to new labels?

Are the results a categorical index with the same categories? New categories? A normal index? Or an exception (for case 2)?

My vote (for what it's worth) is for the strict approach -- either preserve the same categories or raise an error (no reindexing like behavior).  But this is not entirely consistent with what pandas already does.

1) works
2) all labels must be in the categories or it will raise
3) assignment is ok, but must be in the categories

```
In [4]: df = DataFrame({'A' : np.arange(6,dtype='int64'),
   ...:                 'B' : Series(list('aabbca')).astype('category',categories=list('cabe')) }).set_index('B')

In [5]: df
Out[5]: 
   A
B   
a  0
a  1
b  2
b  3
c  4
a  5

In [6]: df.index
Out[6]: 
CategoricalIndex(categories=[u'c', u'a', u'b', u'e'],
                 codes=[1, 1, 2, 2, 0, 1],
                 ordered=False)

In [7]: df.loc[['a','b','e']]
Out[7]: 
    A
B    
a   0
a   1
a   5
b   2
b   3
e NaN

In [8]: df.loc[['a','b','e','g']]
KeyError: 'a list-indexer must only include values that are in the categories'
```

Assignment (we lose the index here, and its not trivial to fix). I'll make an issue about it.

```
In [3]: df.loc['e'] = 20

In [4]: df
Out[4]: 
    A
a   0
a   1
b   2
b   3
c   4
a   5
e  20
```

yes I think we need to allow slice semantics, will put in a follow up issue

@jreback  OK, I think you've covered the bases here. Nice work!

Finally took some time to look at this. Some feedback!

Some things about the repr:
- I am not fully sure about the repr. Now you have repr vs str:
  
  ```
  In [19]: catidx = df2.index
  
  In [20]: catidx
  Out[20]:
  CategoricalIndex(categories=[u'c', u'a', u'b'],
               codes=[1, 1, 2, 2, 0, 1],
               ordered=False)
  
  In [21]: print catidx
  CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], dtype='category')
  ```
  
  And I think I actually like the second more (in one view, it is more clear to me how my index looks like. But some information is missing there (the categories, the ordered)
- Furthermore, the current repr is not eval-able. Which is kind of odd because it looks like this is the case (it is written as a code expression). This is because `CategoricalIndex` does not take a `codes` keyword. Of course the repr does not need to be eval-able, but it would be nice if it is I think (it is the case for eg MultiIndex and Int64Index)

Some other things:
- What with the other methods we have on a Categorical? Do we want to support them for a CategoricalIndex? (the `as_ordered`, 'rename_categories', ...)
- I definitely think contains should check the values and not the categories, which is now not the case:
  
  ```
  In [66]: catidx[0:3]
  Out[66]:
  CategoricalIndex(categories=[u'c', u'a', u'b'],
               codes=[1, 1, 2],
               ordered=False)
  
  In [67]: print catidx[0:3]
  CategoricalIndex([u'a', u'a', u'b'], dtype='category')
  
  In [68]: 'c' in catidx[0:3]
  Out[68]: True
  ```
- grouping on a categorical column, should this give a CategoricalIndex in the result? This is not the case at the moment:
  
  ```
  In [82]: df.dtypes
  Out[82]:
  A       int64
  B    category
  dtype: object
  
  In [83]: df.groupby('B').mean()
  Out[83]:
   A
  B
  c  4.0
  a  2.0
  b  2.5
  
  In [84]: df.groupby('B').mean().index
  Out[84]: Index([u'c', u'a', u'b'], dtype='object')
  ```
  
  But with the categorical already in the index, it is:
  
  ```
  In [88]: df2.groupby(level=0).mean()
  Out[88]:
   A
  B
  a  2.0
  b  2.5
  c  4.0
  
  In [89]: df2.groupby(level=0).mean().index
  Out[89]:
  CategoricalIndex(categories=[u'c', u'a', u'b'],
               codes=[1, 2, 0],
               ordered=False)
  ```
  
  Notice this also gives another order of the result.
- Trying to add an element with a non-existing category raises (which is indeed logical!), but it raises some different errors depending on the case:
  
  ```
  In [114]: df2.loc['e'] = 6
  TypeError: cannot append a non-category item to a CategoricalIndex
  
  In [115]: df2.loc['e', 'A'] = 6
  ValueError: unsafe appending to index of type CategoricalIndex with a key e
  ```

Some more:
- The default for the CategoricalIndex is `ordered=False` (just the same as categorical itself), but min/max seems to work:
  
  ```
  In [165]: catidx = pd.CategoricalIndex(list('aabbca'), categories=['c', 'a', 'b'
  ])
  
  In [166]: catidx
  Out[166]:
  CategoricalIndex(categories=[u'c', u'a', u'b'],
               codes=[1, 1, 2, 2, 0, 1],
               ordered=False)
  
  In [167]: catidx.min()
  Out[167]: 'a'
  
  In [168]: catidx.values.min()
  TypeError: Categorical is not ordered for operation min
  you can use .as_ordered() to change the Categorical to an ordered one
  ```
  
  Should this follow the behaviour of Categorical itself?
- I get that when slicing with a list of which one of the elements is not included in the index, the most logical thing is to raise, as you did now:
  
  ```
  In [198]: df2 = DataFrame({'A' : np.arange(6,dtype='int64'),'B' : Series(list('aabbca')).astype('category',categories=list('cab')) }).set_index('B')
  
  In [200]: df2.loc[['a','e']]
  KeyError: 'a list-indexer must only include values that are in the categories'
  ```
  
  Only, this creates yet another exception in the indexing rules. As for other types the rule as: at least one element of the list should be included in the index values, and the rest are added with NaN values. For string indices with the same frame:
  
  ```
  In [201]: df3 = df2.copy()
  In [202]: df3.index = df2.index.astype(object)
  
  In [203]: df3.loc[['a','e']]
  Out[203]:
  A
  a   0
  a   1
  a   5
  e NaN
  ```
  
  The problem of course is that this would need to alter the categories of the CategoricalIndex. I see this is a problem, but I also do not really like the exception on the indexing rules.
- Should `equals` only check for the containing elements (as the docstring says), or also check if the dtype/categories match? E.g. FloatIndex and IntIndex with equal values will return True. But for a CategoricalIndex, it also checks the categories.

@jorisvandenbossche added your list to the top section.

non-controversial:
- `min/max` are implemented at a lower level and just need some addtl checking
- adding methods of `Categorical`, easy, just need to add a small wrapper
- repr / make evalable, easy - I have to refactor a bit to make MultiIndex / Categorical have a Mixin I think; was avoiding this but more and more things becoming common
- contains - yeh was on my list

questions:
- groupby, I think the groupby as I have it is right, in that the ordering is based on the index values (and not the categories); it should of work with a cat column (have to fix that). This should be in the same order as the index (which may or not be a categorical), irrespective of the grouping column (with `level=0` of course they are the same). I think since we default `sort=True` then it should be by category order here? (as that is the order of the index, e.g. `cab`)
- raising on a non-included list indexer `e.g. df.loc[['a','z']]`. I think it IS correct to raise. The fact that it works differently is correct. I agree this might be a bit odd to a user, but the point is that `Categoricals` are quite strict. Its like trying to index something with another type, which shouldn't work and silently just pass the error. maybe @shoyer has some commentary here.
- `equals` has to be strict, otherwise defeats the purpose of `Categoricals`.

other:
- `is_dtype_equals`, should really be just a comparison between the `CategoricalDtype` object, but haven't really set that up / done that. Everything not in the API is by definition private, so I think its ok.

I'm also not happy with the inconsistent behavior for non-included list indexers.

I'm sure "automatic reindexing" seemed like a convenient feature at the time it was implemented, and it does make some sense, if there is only one index type. The problem is that now we have or want multiple types of indexes, some of which act very differently from a plain Index (like MultiIndex, CategoricalIndex or IntervalIndex), so automatically coercing to a new index type no longer seems like such a good idea.

I do think we should keep the existing behavior in this PR for CategoricalIndex, because it's the only sane approach. Going forward, we might further deprecate and eventually remove all indexing operations that would expand the index type, breaking operations like this:

```
In [6]: s = pd.Series(range(5))

In [7]: s.loc[[0, 2, 2.5]]
Out[7]:
0.0     0
2.0     2
2.5   NaN
dtype: float64
```

repr is now consistent with other Index types.

but, the truncation is really really tricky so not there for now (I see why it hasn't been done).

```
In [1]: df3 = DataFrame({'A' : np.arange(6),
   ...:                                 'B' : Series(list('aabbca')).astype('category',
   ...:                                                                     categories=list('abc'))
   ...:                                }).set_index('B')

In [2]: df3
Out[2]: 
   A
B   
a  0
a  1
b  2
b  3
c  4
a  5

In [3]: df3.index
Out[3]: CategoricalIndex([u'a', u'a', u'b', u'b', u'c', u'a'], categories=[u'a', u'b', u'c'], ordered=False)
```

ok, all points addressed as discussed above, here are the highlites:

```
In [2]: df = DataFrame({'A' : np.arange(6),
   ...:                 'B' : Series(list('aabbca')).astype('category',categories=list('cabe'))})    

In [3]: df
Out[3]: 
   A  B
0  0  a
1  1  a
2  2  b
3  3  b
4  4  c
5  5  a

In [4]: df.dtypes
Out[4]: 
A       int64
B    category
dtype: object

# groupby does the same for a category column / index
In [5]: df.groupby('B').sum()
Out[5]: 
    A
B    
c   4
a   6
b   5
e NaN

# new string output (and repr)
In [6]: df.groupby('B').sum().index
Out[6]: 
CategoricalIndex([u'c', u'a', u'b', u'e'],
                 categories=[u'c', u'a', u'b', u'e'],
                 ordered=False)

In [7]: df.set_index('B').groupby(level=0).sum()
Out[7]: 
    A
B    
c   4
a   6
b   5
e NaN
```

Indexing operations

```
In [15]: df2 = df.set_index('B')

In [16]: df2
Out[16]: 
   A
B   
a  0
a  1
b  2
b  3
c  4
a  5

In [17]: df2.loc[['a','b']]
Out[17]: 
   A
B   
a  0
a  1
a  5
b  2
b  3

In [19]: df2.reindex(['a','b'])
Out[19]: 
   A
B   
a  0
a  1
a  5
b  2
b  3

In [20]: df2.reindex(['a','e'])
Out[20]: 
    A
B    
a   0
a   1
a   5
e NaN
```

Selecting non-categories in indexing operations (The different messages are because they are in different parts of the code); trying to be helpful here. I _could_ fix the single-indexer case to have a more Categorical specific message (its 'generic' atm).

```
In [21]: df2.reindex(['a','d'])
KeyError: 'a reindexing must only include values that are in the categories'

In [22]: df2.loc[['a','d']]
KeyError: 'a list-indexer must only include values that are in the categories'

In [23]: df2.loc[['d']]
KeyError: "None of [['d']] are in the [index]"
```

Big thing in wading thru some indexing code is that `df.loc[listlike]` and `df.reinde(listlike)` are de-factor equivalent, BUT, they are actually somewhat independently implemented (there is a difference though, e.g. `.reindex` would give you all-NaNs if nothing matches, while `.loc` will raise an exception).

So it of course makes sense to combine more of this (don't get me wrong they are not completely independent, but could use more code sharing).

@jreback what is the type of `df2.reindex(['a','b']).index`? My vote is for plain `Index`, unless you're reindexing with another categorical.

no, `CategoricalIndex` types are preserved always (well as much as we have tests coverage for!)

We don't normally coerce to lower? `Index` unless there is a really compelling reason (e.g. concat of a Index and a DatetimeIndex) or something.

From a consistency perspective, it would be nice to ensure the following invariants:
1. `type(df.reindex(labels).index) is type(Index(labels))`
2. `type(df.loc[labels].index) is type(df.index)`

With the current implementation, I believe the type of the index from reindex depends on whether or not all labels were found. That sort of runtime value specific behavior leads to all sorts of hard to catch bugs.

point 1: `.reindex` of a `CategoricalIndex` will be a `CategoricalIndex`, or raise.
point 2: is the same (e.g. as point 1)

```
In [1]: df2 = DataFrame({'A' : np.arange(6,dtype='int64'),
   ...:                               'B' : Series(list('aabbca')).astype('category',categories=list('cabe')) }).set_index('B')

In [3]: df2.reindex(['a','b','e'])
Out[3]: 
    A
B    
a   0
a   1
a   5
b   2
b   3
e NaN

In [4]: df2.reindex(['a','b','e']).index
Out[4]: 
CategoricalIndex([u'a', u'a', u'a', u'b', u'b', u'e'],
                 categories=[u'c', u'a', u'b', u'e'],
                 ordered=False)

In [5]: df2.reindex(['a','b','e','d']).index
KeyError: 'a reindexing must only include values that are in the categories'
```

I think it should be quite explicit that `.reindex(CategoricalIndex)` -> `Index(set_categories_with_new_labels)` and this should not be an accident, that is the point of a `Categorical` after all, very explicity categories.

@shoyer sorry, have a bug, hold on

@shoyer ok, reorged the non-unique with a non-unique keyarr section (its now in Index) where it should really go, so slightly more self-contained. This allows categorical to override it in a trivial way as all indexing in Categorical is non-unique.

lmk if you have additional issues.

Ok I put back to make `.reindex` always work. It will return an `Index` if needed (e.g. in the case where you are reindexing by a value not in the categories).

```
In [2]: df = DataFrame({'A' : [20,20,2,3,4,20],
                              'B' : Series(list('aabbca')).astype('category',categories=list('cab')) }).set_index('B')

In [3]: df
Out[3]: 
    A
B    
a  20
a  20
b   2
b   3
c   4
a  20

In [4]: df.reindex(['a','c']).index
Out[4]: 
CategoricalIndex([u'a', u'a', u'a', u'c'],
                 categories=[u'c', u'a', u'b'],
                 ordered=False)

In [5]: df.reindex(['a','d']).index
Out[5]: Index([u'a', u'a', u'a', u'd'], dtype='object')

In [6]: df.reindex(['a','d'])      
Out[6]: 
    A
B    
a  20
a  20
a  20
d NaN

In [7]: df.loc[['a','b']]
Out[7]: 
    A
B    
a  20
a  20
a  20
b   2
b   3

In [8]: df.loc[['a','d']]
KeyError: 'a list-indexer must only include values that are in the categories'
```

Keep in mind that in general you will be getting a non-unique index, so not very useful, but will work. The `.loc` remains the same (as strict).

Still not really happy with this (see my comment about invariants above):

```
In [4]: df.reindex(['a','c']).index
Out[4]: 
CategoricalIndex([u'a', u'a', u'a', u'c'],
                 categories=[u'c', u'a', u'b'],
                 ordered=False)

In [5]: df.reindex(['a','d']).index
Out[5]: Index([u'a', u'a', u'a', u'd'], dtype='object')
```

NumPy does a really nice job of ensuring that the type/dtype of results only depends on the dtype of the input variables, never runtime values. It would be nice to emulate that in pandas when practical.

any thoughts from @jorisvandenbossche or anyone else reading?

What happens if you reindex with another categorical? `df.reindex(pd.Categorical(['a','c']).index` and `df.reindex(pd.Categorical(['a','d']).index`?

What are the reasons for not raising on a `reindex` with categories outside of the original set? Is it just for consistency with  `.reindex` on other Index flavors?