Repository: pandas-dev/pandas

Issue:
TestCategoricalIndex.test_reindexing failing on Travis / Mac build
https://travis-ci.org/pandas-dev/pandas/jobs/267760269

Master and some PRs (strangely not all) are all failing on the same test, but only on Mac:

```
=================================== FAILURES ===================================
_____________________ TestCategoricalIndex.test_reindexing _____________________
[gw1] darwin -- Python 3.5.4 /Users/travis/miniconda3/envs/pandas/bin/python
self = <pandas.tests.indexes.test_category.TestCategoricalIndex object at 0x111d5e860>
    def test_reindexing(self):
    
        ci = self.create_index()
        oidx = Index(np.array(ci))
    
        for n in [1, 2, 5, len(ci)]:
            finder = oidx[np.random.randint(0, len(ci), size=n)]
            expected = oidx.get_indexer_non_unique(finder)[0]
    
            actual = ci.get_indexer(finder)
>           tm.assert_numpy_array_equal(expected, actual)
pandas/tests/indexes/test_category.py:389: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/util/testing.py:1174: in assert_numpy_array_equal
    _raise(left, right, err_msg)
pandas/util/testing.py:1157: in _raise
    .format(obj=obj), left.shape, right.shape)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
obj = 'numpy array', message = 'numpy array shapes are different', left = (14,)
right = (6,), diff = None
    def raise_assert_detail(obj, message, left, right, diff=None):
        if isinstance(left, np.ndarray):
            left = pprint_thing(left)
        if isinstance(right, np.ndarray):
            right = pprint_thing(right)
    
        msg = """{obj} are different
    
    {message}
    [left]:  {left}
    [right]: {right}""".format(obj=obj, message=message, left=left, right=right)
    
        if diff is not None:
            msg += "\n[diff]: {diff}".format(diff=diff)
    
>       raise AssertionError(msg)
E       AssertionError: numpy array are different
E       
E       numpy array shapes are different
E       [left]:  (14,)
E       [right]: (6,)
pandas/util/testing.py:1105: AssertionError
```



Discussion:
Marking as unreliable given that only *some* PR's are failing on this test.
I think the `np.random.randint` is the problem.
But whathever result of randint that I can think off, the `oidx.get_indexer_non_unique(finder)[0]` and `ci.get_indexer(finder)` should still behave the same ..
I suppose, though I don't know how else you get such a difference of 14 and 6...
FYI, [pytest-repeat](https://pypi.python.org/pypi/pytest-repeat) is helpful for debugging these.

```
pytest pandas/tests/indexes/test_category.py::TestCategoricalIndex::test_reindexing --count=1000 --pdb
=================================================================== test session starts ====================================================================
platform darwin -- Python 3.6.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas, inifile: setup.cfg
plugins: xdist-1.15.0, rerunfailures-2.2, repeat-0.4.1, cov-2.4.0
collected 1000 items

pandas/tests/indexes/test_category.py ...............................................................................................................................F

```
Indeed good tip :-)

So this happens when the random generation results in exactly the same index as the original one:

```
(Pdb) p n
6
(Pdb) finder
Index(['a', 'a', 'b', 'b', 'c', 'a'], dtype='object')
(Pdb) oidx
Index(['a', 'a', 'b', 'b', 'c', 'a'], dtype='object')
(Pdb) expected
array([0, 1, 5, 0, 1, 5, 2, 3, 2, 3, 4, 0, 1, 5])
(Pdb) ci
CategoricalIndex(['a', 'a', 'b', 'b', 'c', 'a'], categories=['c', 'a', 'b'], ordered=False, dtype='category')
(Pdb) ci.get_indexer(finder)
array([0, 1, 2, 3, 4, 5])
```

But I would say this is then rather a bug in `get_indexer` (and not in the test) ?

(still a bit strange that it turned up in so many builds now, and only on mac, as for this failed in 3 out of 1000 runs)
So reproducible example:

```
In [4]: ci = pd.CategoricalIndex( list('aabbca'), categories=list('cab'))

In [5]: ci.get_indexer(['a', 'b', 'c'])
Out[5]: array([0, 1, 5, 2, 3, 4])

In [6]: ci.get_indexer(['a', 'a', 'b', 'b', 'c', 'a'])
Out[6]: array([0, 1, 2, 3, 4, 5])

In [7]: ci.get_indexer(['a', 'a', 'b', 'b', 'c', 'a', 'c'])
Out[7]: array([0, 1, 5, 0, 1, 5, 2, 3, 2, 3, 4, 0, 1, 5, 4])

In [8]: ci.get_indexer_non_unique(['a', 'a', 'b', 'b', 'c', 'a'])[0]
Out[8]: array([0, 1, 5, 0, 1, 5, 2, 3, 2, 3, 4, 0, 1, 5])
```

Compared to normal indices, `get_indexer` is able to handle non-unique index for Categoricals (for example `ci.get_indexer(['a', 'b', 'c'])` in [5]). However, inside `get_indexer` a 'fast path' was added for when the passed index equals the calling one. Therefore in such a case, it does not duplicate the duplicates (as it does in [5] and [8]). This contrast with how `get_indexer_non_unique` works ([7]).

The 'fast path' was added in the IntervalIndex PR, so this also seems a change compared to 0.19:

```
In [1]: pd.__version__
Out[1]: '0.19.2'

In [2]: ci = pd.CategoricalIndex( list('aabbca'), categories=list('cab'))

In [3]: ci.get_indexer(['a', 'a', 'b', 'b', 'c', 'a'])
Out[3]: array([0, 1, 5, 0, 1, 5, 2, 3, 2, 3, 4, 0, 1, 5])
```

(but not sure what actual consequences in user code could be)
A possible fix: add to the fastpath a check for uniqueness of the index here:

https://github.com/pandas-dev/pandas/blob/96f92eb1c696723b6465fdc273dc8406201c606a/pandas/core/indexes/category.py#L486-L491

Resolving patch:
diff --git a/pandas/core/indexes/category.py b/pandas/core/indexes/category.py
--- a/pandas/core/indexes/category.py
+++ b/pandas/core/indexes/category.py
@@ -487,7 +487,7 @@ def get_indexer(self, target, method=None, limit=None, tolerance=None):
         method = missing.clean_reindex_fill_method(method)
         target = ibase._ensure_index(target)
 
-        if self.equals(target):
+        if self.is_unique and self.equals(target):
             return np.arange(len(self), dtype='intp')
 
         if method == 'pad' or method == 'backfill':


Test patch:
diff --git a/pandas/tests/indexes/test_category.py b/pandas/tests/indexes/test_category.py
--- a/pandas/tests/indexes/test_category.py
+++ b/pandas/tests/indexes/test_category.py
@@ -365,18 +365,18 @@ def test_astype(self):
         tm.assert_index_equal(result, expected)
 
     def test_reindex_base(self):
-
-        # determined by cat ordering
-        idx = self.create_index()
+        # Determined by cat ordering.
+        idx = CategoricalIndex(list("cab"), categories=list("cab"))
         expected = np.arange(len(idx), dtype=np.intp)
 
         actual = idx.get_indexer(idx)
         tm.assert_numpy_array_equal(expected, actual)
 
-        with tm.assert_raises_regex(ValueError, 'Invalid fill method'):
-            idx.get_indexer(idx, method='invalid')
+        with tm.assert_raises_regex(ValueError, "Invalid fill method"):
+            idx.get_indexer(idx, method="invalid")
 
     def test_reindexing(self):
+        np.random.seed(123456789)
 
         ci = self.create_index()
         oidx = Index(np.array(ci))
@@ -388,6 +388,18 @@ def test_reindexing(self):
             actual = ci.get_indexer(finder)
             tm.assert_numpy_array_equal(expected, actual)
 
+        # see gh-17323
+        #
+        # Even when indexer is equal to the
+        # members in the index, we should
+        # respect duplicates instead of taking
+        # the fast-track path.
+        for finder in [list("aabbca"), list("aababca")]:
+            expected = oidx.get_indexer_non_unique(finder)[0]
+
+            actual = ci.get_indexer(finder)
+            tm.assert_numpy_array_equal(expected, actual)
+
     def test_reindex_dtype(self):
         c = CategoricalIndex(['a', 'b', 'c', 'a'])
         res, indexer = c.reindex(['a', 'c'])