Repository: pandas-dev/pandas
Pull request: BUG: Series.isin fails or categoricals
 - closes #16639 
 - [1 ] tests added / passed
 - ] passes ``git diff upstream/master --name-only -- '*.py' | flake8 --diff`` (On Windows, ``git diff upstream/master -u -- "*.py" | flake8 --diff`` might work as an alternative.)
 - [check ] whatsnew entry


Problem statement:
Series.isin fails (errors) for categoricals 
#### Code Sample, a copy-pastable example if possible

```python
import pandas as pd
import numpy as np
#%%
print(pd.__version__)
vals = np.array([0, 1,2, 0]);
cats = ['a', 'b', 'c'];

DFtrades = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(vals, cats))});
DFscores = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(np.array([0, 1]), cats))});

print(DFtrades)
print(DFscores)

select_ids = DFtrades['id'].isin(DFscores['id']);

```
#### Problem description

I get an error in 0.20.1

  File "<ipython-input-17-44ce14b18f50>", line 12, in <module>
    select_ids = DFtrades['id'].isin(DFscores['id']);

  File "C:\Users\alexandre\Anaconda3\lib\site-packages\pandas\core\series.py", line 2555, in isin
    result = algorithms.isin(_values_from_object(self), values)

  File "C:\Users\alexandre\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 421, in isin
    return f(comps, values)

  File "C:\Users\alexandre\Anaconda3\lib\site-packages\pandas\core\algorithms.py", line 399, in <lambda>
    f = lambda x, y: htable.ismember_object(x, values)

  File "pandas\_libs\hashtable_func_helper.pxi", line 428, in pandas._libs.hashtable.ismember_object (pandas\_libs\hashtable.c:29677)

ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long long'


#### Expected Output
a boolean array (or series?) indicating the third row of DFtrades is not in DFscores but the other three are

for reference, this worked (I did not get an error) in 0.19.(something) 

also this code will work as expected:

`select_ids = DFtrades['id'].isin(DFscores['id'].values);`


#### Output of ``pd.show_versions()``
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.1.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 94 Stepping 3, GenuineIntel
byteorder: little
LC_ALL: None
LANG: en
LOCALE: None.None

pandas: 0.20.1
pytest: 3.1.1
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.11.3
scipy: 0.19.0
xarray: 0.9.5
IPython: 6.1.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.2.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.7
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.6
lxml: 3.8.0
bs4: 4.6.0
html5lib: 0.999
sqlalchemy: 1.1.10
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None


</details>

Hints from issue discussion:
I'm guessing the fix to this looks something like #16543 - did some refactoring the algorithms file and this is a case that probably got missed
this fixes. Though I think we should add some asv's with categoricals to make sure they are hitting the right path

```
diff --git a/pandas/core/algorithms.py b/pandas/core/algorithms.py
index d74c5e6..a651817 100644
--- a/pandas/core/algorithms.py
+++ b/pandas/core/algorithms.py
@@ -113,7 +113,8 @@ def _ensure_data(values, dtype=None):
 
         return values.asi8, dtype, 'int64'
 
-    elif is_categorical_dtype(values) or is_categorical_dtype(dtype):
+    elif (is_categorical_dtype(values) and
+          (is_categorical_dtype(dtype) or dtype is None)):
         values = getattr(values, 'values', values)
         values = values.codes
         dtype = 'category'
```
@aviolov want to push a PR for the above fix?
@jreback at the risk of sounding ignorant - how would I do that (maybe a link to some documentation / how-to)? 
@aviolov which part, specifically? All the contributing docs are at http://pandas.pydata.org/pandas-docs/stable/contributing.html. If you have any additional questions, just ask them here.
@TomAugspurger , thanks for the link. I guess a 'PR' is a pull request in this case.  Is the idea that I download version 0.20.3 and check that my minimal example above works now or that I branch the current version and implement the fix suggested above and then try to push it back or... ? I haven't made a branch off pandas before, but would be fun to try - the how-to looks quite comprehensive
@aviolov you'll fork the repo as described in http://pandas.pydata.org/pandas-docs/stable/contributing.html#forking

Then create [a new branch](http://pandas.pydata.org/pandas-docs/stable/contributing.html#creating-a-branch)

Then apply your changes:

- add a test in `pandas/tests/test_categorical.py` with your original example,
- [run the tests](http://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-test-suite) something like `pytest pandas/tests/test_categorical.py -k <test name>` to verify that it fails
- Add the fix from @jreback 
- Add a release not in `doc/source/whatsnew/v0.21.0.txt`

Then [push](http://pandas.pydata.org/pandas-docs/stable/contributing.html#pushing-your-changes) and make a pull request (PR)
@TomAugspurger cool, I'll give it a try

Commit message: BUG: GH16639 fix Series.isin for Categoricals

Patch to review:
diff --git a/doc/source/whatsnew/v0.21.0.txt b/doc/source/whatsnew/v0.21.0.txt
index d5cc3d6ddca8e..20daa19a7eded 100644
--- a/doc/source/whatsnew/v0.21.0.txt
+++ b/doc/source/whatsnew/v0.21.0.txt
@@ -188,7 +188,7 @@ Numeric
 
 Categorical
 ^^^^^^^^^^^
-
+- Bug in ``Series.isin()`` when called for categoricals (:issue`16639`)
 
 Other
 ^^^^^
diff --git a/pandas/core/algorithms.py b/pandas/core/algorithms.py
index d74c5e66ea1a9..ffb03f52ffbdb 100644
--- a/pandas/core/algorithms.py
+++ b/pandas/core/algorithms.py
@@ -3,14 +3,18 @@
 intended for public consumption
 """
 from __future__ import division
+
 from warnings import warn, catch_warnings
-import numpy as np
 
 from pandas import compat, _np_version_under1p8
+from pandas.compat import string_types
+from pandas.compat.numpy import _np_version_under1p10
+from pandas.core import common as com
+
+import numpy as np
+from pandas._libs import algos, lib, hashtable as htable
+from pandas._libs.tslib import iNaT
 from pandas.core.dtypes.cast import maybe_promote
-from pandas.core.dtypes.generic import (
-    ABCSeries, ABCIndex,
-    ABCIndexClass, ABCCategorical)
 from pandas.core.dtypes.common import (
     is_unsigned_integer_dtype, is_signed_integer_dtype,
     is_integer_dtype, is_complex_dtype,
@@ -26,19 +30,15 @@
     _ensure_platform_int, _ensure_object,
     _ensure_float64, _ensure_uint64,
     _ensure_int64)
-from pandas.compat.numpy import _np_version_under1p10
+from pandas.core.dtypes.generic import (
+    ABCSeries, ABCIndex,
+    ABCIndexClass, ABCCategorical)
 from pandas.core.dtypes.missing import isnull
 
-from pandas.core import common as com
-from pandas.compat import string_types
-from pandas._libs import algos, lib, hashtable as htable
-from pandas._libs.tslib import iNaT
-
 
 # --------------- #
 # dtype access    #
 # --------------- #
-
 def _ensure_data(values, dtype=None):
     """
     routine to ensure that our data is of the correct
@@ -113,7 +113,8 @@ def _ensure_data(values, dtype=None):
 
         return values.asi8, dtype, 'int64'
 
-    elif is_categorical_dtype(values) or is_categorical_dtype(dtype):
+    elif (is_categorical_dtype(values) and
++          (is_categorical_dtype(dtype) or dtype is None)):
         values = getattr(values, 'values', values)
         values = values.codes
         dtype = 'category'
diff --git a/pandas/tests/test_categorical.py b/pandas/tests/test_categorical.py
index 667b26c24c662..24bdae9336d48 100644
--- a/pandas/tests/test_categorical.py
+++ b/pandas/tests/test_categorical.py
@@ -442,6 +442,17 @@ def f():
         if hasattr(np.random, "choice"):
             codes = np.random.choice([0, 1], 5, p=[0.9, 0.1])
             pd.Categorical.from_codes(codes, categories=["train", "test"])
+        
+        # Regression test https://github.com/pandas-dev/pandas/issues/16639
+        vals = np.array([0, 1, 2, 0]);
+        cats = ['a', 'b', 'c'];
+        
+        D = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(vals, cats))});
+        T = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(np.array([0, 1]), cats))});
+        
+        select_ids = D['id'].isin(T['id']);
+                      
+        assert( np.all(select_ids == np.array([True, True, False, True]) ) )
 
     def test_validate_ordered(self):
         # see gh-14058


Code review comments:
diff_hunk: @@ -188,7 +188,7 @@ Numeric
 
 Categorical
 ^^^^^^^^^^^
-
+- Bug in ``Series.isin()`` when called for categoricals (:issue`16639`)
path: doc/source/whatsnew/v0.21.0.txt
text: @user1:
use :func:`Series.isin()`

when called with a categorical

diff_hunk: @@ -442,6 +442,17 @@ def f():
         if hasattr(np.random, "choice"):
             codes = np.random.choice([0, 1], 5, p=[0.9, 0.1])
             pd.Categorical.from_codes(codes, categories=["train", "test"])
+        
+        # Regression test https://github.com/pandas-dev/pandas/issues/16639
+        vals = np.array([0, 1, 2, 0]);
+        cats = ['a', 'b', 'c'];
+        
+        D = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(vals, cats))});
+        T = pd.DataFrame({'id': pd.Series(pd.Categorical(1).from_codes(np.array([0, 1]), cats))});
+        
+        select_ids = D['id'].isin(T['id']);
path: pandas/tests/test_categorical.py
text: @user1:
use result=
the expected= is below and use
``tm.assert_numpy_array_equal``