Repository: pandas-dev/pandas

Issue:
Categorical[idx].codes != Categorical.codes[idx]
I'm unable to isolate this, but maybe this rings a bell for someone. It took me a long time to find the source of error in my code. I don't think this is intentional, and if so, is pretty dangerous.

```
[~/]
[22]: y[idx].codes[:15]
[22]: array([4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1, 4], dtype=int8)

[~/]
[23]: y.codes[idx][:15]
[23]: array([4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4], dtype=int8)

[~/]
[24]: idx[:15]
[24]:
array([ 74172,  53885, 128953,  28125, 136548, 136700,  93633,  61147,
        56535,  90577, 115719,  58038, 111711,  53399,  77475])

[~/]
[25]: type(y)
[25]: pandas.core.categorical.Categorical

[~/]
[27]: pd.version.version
[27]: '0.15.2'
```



Discussion:
It seems that `Categorical._maybe_coerce_indexer` is transforming the int64 index to int8.

The indexer is coerced to the same dtype as the codes (`int8`) in this case. This is necessary to have the indexing work properly (otherwise numpy complains). But I think if the indexer is out of bounds it will wrap around which is prob not what is expected.

can you show a construction for y?

needless to say this is prob not tested well. and prob not used very much as this is almost always used from a Series. (this is somewhat legacy code)

I'm trying to make a MWE.

```
np.random.seed(1)
c = pd.Categorical(np.random.randint(0, 5, size=150000).astype(np.int8))
c.codes[np.array([100000]).astype(np.int64)]
c[np.array([100000]).astype(np.int64)].codes
```

In my case, `y` was created from some real data with 5 categories and the `idx` was created from sklearn cross-validation functions. Given the size of my `y`, int64s. 

Not obvious to me why numpy would care between dtype of indexer and the dtype of the array. AFAIK, it just needs to be any integer type or boolean, no?


Resolving patch:
diff --git a/pandas/core/categorical.py b/pandas/core/categorical.py
--- a/pandas/core/categorical.py
+++ b/pandas/core/categorical.py
@@ -1229,7 +1229,6 @@ def __getitem__(self, key):
             else:
                 return self.categories[i]
         else:
-            key = self._maybe_coerce_indexer(key)
             return Categorical(values=self._codes[key], categories=self.categories,
                                ordered=self.ordered, fastpath=True)
 
@@ -1253,6 +1252,7 @@ def __setitem__(self, key, value):
 
         rvalue = value if is_list_like(value) else [value]
         to_add = Index(rvalue).difference(self.categories)
+
         # no assignments of values not in categories, but it's always ok to set something to np.nan
         if len(to_add) and not isnull(to_add).all():
             raise ValueError("cannot setitem on a Categorical with a new category,"
@@ -1297,7 +1297,6 @@ def __setitem__(self, key, value):
             nan_pos = np.where(isnull(self.categories))[0]
             lindexer[lindexer == -1] = nan_pos
 
-        key = self._maybe_coerce_indexer(key)
         lindexer = self._maybe_coerce_indexer(lindexer)
         self._codes[key] = lindexer
 


Test patch:
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -34,6 +34,50 @@ def test_getitem(self):
         subf = self.factor[np.asarray(self.factor) == 'c']
         tm.assert_almost_equal(subf._codes, [2, 2, 2])
 
+    def test_getitem_listlike(self):
+
+        # GH 9469
+        # properly coerce the input indexers
+        np.random.seed(1)
+        c = Categorical(np.random.randint(0, 5, size=150000).astype(np.int8))
+        result = c.codes[np.array([100000]).astype(np.int64)]
+        expected = c[np.array([100000]).astype(np.int64)].codes
+        self.assert_numpy_array_equal(result, expected)
+
+    def test_setitem(self):
+
+        # int/positional
+        c = self.factor.copy()
+        c[0] = 'b'
+        self.assertEqual(c[0], 'b')
+        c[-1] = 'a'
+        self.assertEqual(c[-1], 'a')
+
+        # boolean
+        c = self.factor.copy()
+        indexer = np.zeros(len(c),dtype='bool')
+        indexer[0] = True
+        indexer[-1] = True
+        c[indexer] = 'c'
+        expected = Categorical.from_array(['c', 'b', 'b', 'a',
+                                           'a', 'c', 'c', 'c'])
+
+        self.assert_categorical_equal(c, expected)
+
+    def test_setitem_listlike(self):
+
+        # GH 9469
+        # properly coerce the input indexers
+        np.random.seed(1)
+        c = Categorical(np.random.randint(0, 5, size=150000).astype(np.int8)).add_categories([-1000])
+        indexer = np.array([100000]).astype(np.int64)
+        c[indexer] = -1000
+
+        # we are asserting the code result here
+        # which maps to the -1000 category
+        result = c.codes[np.array([100000]).astype(np.int64)]
+        self.assertEqual(result, np.array([5], dtype='int8'))
+
     def test_constructor_unsortable(self):
 
         # it works!