Repository: pandas-dev/pandas

Issue:
Series.isin fails (errors) for categoricals 
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np
#%%
print(pd.__version__)
vals = np.array([0, 1,2, 0]);
cats = ['a', 'b', 'c'];

DFtrades = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(vals, cats))});
DFscores = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(np.array([0, 1]), cats))});

print(DFtrades)
print(DFscores)

select_ids = DFtrades['id'].isin(DFscores['id']);

```
#### Problem description

I get an error in 0.20.1

  File "<ipython-input-17-44ce14b18f50>", line 12, in <module>
    select_ids = DFtrades['id'].isin(DFscores['id']);

  File "C:\Users\alexandre\Anaconda3\lib\site-packages\pandas\core\series.py", line 2555, in isin
    result = algorithms.isin(_values_from_object(self), values)

  File "C:\Users\alexandre\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 421, in isin
    return f(comps, values)

  File "C:\Users\alexandre\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 399, in <lambda>
    f = lambda x, y: htable.ismember_object(x, values)

  File "pandas\_libs\hashtable_func_helper.pxi", line 428, in pandas._libs.hashtable.ismember_object (pandas\_libs\hashtable.c:29677)

ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long long'


#### Expected Output
a boolean array (or series?) indicating the third row of DFtrades is not in DFscores but the other three are

for reference, this worked (I did not get an error) in 0.19.(something) 

also this code will work as expected:

`select_ids = DFtrades['id'].isin(DFscores['id'].values);`


#### Output of ``pd.show_versions()``
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.20.1
pytest: 3.1.1
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.19.0
xarray: 0.9.5
IPython: 6.1.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.8.0
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.10
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None


</details>



Discussion:
I'm guessing the fix to this looks something like #16543 - did some refactoring the algorithms file and this is a case that probably got missed
this fixes. Though I think we should add some asv's with categoricals to make sure they are hitting the right path

```
diff --git a/pandas/core/algorithms.py b/pandas/core/algorithms.py
index d74c5e6..a651817 100644
--- a/pandas/core/algorithms.py
+++ b/pandas/core/algorithms.py
@@ -113,7 +113,8 @@ def _ensure_data(values, dtype=None):
 
         return values.asi8, dtype, 'int64'
 
-    elif is_categorical_dtype(values) or is_categorical_dtype(dtype):
+    elif (is_categorical_dtype(values) and
+          (is_categorical_dtype(dtype) or dtype is None)):
         values = getattr(values, 'values', values)
         values = values.codes
         dtype = 'category'
```
@aviolov want to push a PR for the above fix?
@jreback at the risk of sounding ignorant - how would I do that (maybe a link to some documentation / how-to)? 
@aviolov which part, specifically? All the contributing docs are at http://pandas.pydata.org/pandas-docs/stable/contributing.html. If you have any additional questions, just ask them here.
@TomAugspurger , thanks for the link. I guess a 'PR' is a pull request in this case.  Is the idea that I download version 0.20.3 and check that my minimal example above works now or that I branch the current version and implement the fix suggested above and then try to push it back or... ? I haven't made a branch off pandas before, but would be fun to try - the how-to looks quite comprehensive
@aviolov you'll fork the repo as described in http://pandas.pydata.org/pandas-docs/stable/contributing.html#forking

Then create [a new branch](http://pandas.pydata.org/pandas-docs/stable/contributing.html#creating-a-branch)

Then apply your changes:

- add a test in `pandas/tests/test_categorical.py` with your original example,
- [run the tests](http://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-test-suite) something like `pytest pandas/tests/test_categorical.py -k <test name>` to verify that it fails
- Add the fix from @jreback 
- Add a release not in `doc/source/whatsnew/v0.21.0.txt`

Then [push](http://pandas.pydata.org/pandas-docs/stable/contributing.html#pushing-your-changes) and make a pull request (PR)
@TomAugspurger cool, I'll give it a try

Resolving patch:
diff --git a/pandas/core/algorithms.py b/pandas/core/algorithms.py
--- a/pandas/core/algorithms.py
+++ b/pandas/core/algorithms.py
@@ -38,7 +38,6 @@
 # --------------- #
 # dtype access    #
 # --------------- #
-
 def _ensure_data(values, dtype=None):
     """
     routine to ensure that our data is of the correct
@@ -113,7 +112,8 @@ def _ensure_data(values, dtype=None):
 
         return values.asi8, dtype, 'int64'
 
-    elif is_categorical_dtype(values) or is_categorical_dtype(dtype):
+    elif (is_categorical_dtype(values) and
+          (is_categorical_dtype(dtype) or dtype is None)):
         values = getattr(values, 'values', values)
         values = values.codes
         dtype = 'category'


Test patch:
diff --git a/pandas/tests/test_algos.py b/pandas/tests/test_algos.py
--- a/pandas/tests/test_algos.py
+++ b/pandas/tests/test_algos.py
@@ -586,6 +586,16 @@ def test_large(self):
         expected[1] = True
         tm.assert_numpy_array_equal(result, expected)
 
+    def test_categorical_from_codes(self):
+        # GH 16639
+        vals = np.array([0, 1, 2, 0])
+        cats = ['a', 'b', 'c']
+        Sd = pd.Series(pd.Categorical(1).from_codes(vals, cats))
+        St = pd.Series(pd.Categorical(1).from_codes(np.array([0, 1]), cats))
+        expected = np.array([True, True, False, True])
+        result = algos.isin(Sd, St)
+        tm.assert_numpy_array_equal(expected, result)
+
 
 class TestValueCounts(object):