Repository: pandas-dev/pandas
Pull request: TST: category isin on frame
- [x] closes #34256
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Problem statement:
BUG: DataFrame.isin fails when other is a categorical series
- [ ] I have checked that this issue has not already been reported.

- [ ] I have confirmed this bug exists on the latest version of pandas.

- [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---


#### Code Sample, a copy-pastable example

```python
import pandas as pd
print(pd.__version__)

x = pd.DataFrame.from_dict({'a':[1,2,3], 'b':[4,5,6]})
y = pd.DataFrame({'a':[1,2,3]}, dtype='category')
print(x.isin(y))

y = pd.Series([1,2,3]).astype('category')
print(x.isin(y))

1.0.3
      a      b
0  True  False
1  True  False
2  True  False
Traceback (most recent call last):
  File "/home/brmiller/repro.py", line 9, in <module>
    print(x.isin(y))
  File "/home/brmiller/anaconda3/envs/pandas1/lib/python3.7/site-packages/pandas/core/frame.py", line 8423, in isin
    return self.eq(values.reindex_like(self), axis="index")
  File "/home/brmiller/anaconda3/envs/pandas1/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 814, in f
    self, other, op, fill_value=None, axis=axis, level=level
  File "/home/brmiller/anaconda3/envs/pandas1/lib/python3.7/site-packages/pandas/core/ops/__init__.py", line 618, in _combine_series_frame
    new_data = left._combine_match_index(right, func)
  File "/home/brmiller/anaconda3/envs/pandas1/lib/python3.7/site-packages/pandas/core/frame.py", line 5317, in _combine_match_index
    new_data = func(self.values.T, other.values).T
  File "/home/brmiller/anaconda3/envs/pandas1/lib/python3.7/site-packages/pandas/core/ops/common.py", line 64, in new_method
    return method(self, other)
  File "/home/brmiller/anaconda3/envs/pandas1/lib/python3.7/site-packages/pandas/core/arrays/categorical.py", line 72, in func
    raise ValueError("Lengths must match.")
ValueError: Lengths must match.

```

#### Problem description

This operation previously worked in pandas 0.25.3 and gave the same result as the case when other is a single column DataFrame. 

#### Expected Output
```
      a      b
0  True  False
1  True  False
2  True  False
      a      b
0  True  False
1  True  False
2  True  False
```


#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.6.final.0
python-bits      : 64
OS               : Linux
OS-release       : 4.15.0-76-generic
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : en_US.UTF-8
LOCALE           : en_US.UTF-8

pandas           : 1.0.3
numpy            : 1.18.4
pytz             : 2020.1
dateutil         : 2.8.1
pip              : 20.1
setuptools       : 46.4.0.post20200518
Cython           : 0.29.17
pytest           : 5.4.2
hypothesis       : 5.14.0
sphinx           : 3.0.3
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : None
jinja2           : 2.11.2
IPython          : 7.14.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : None
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : 0.15.0
pytables         : None
pytest           : 5.4.2
pyxlsb           : None
s3fs             : None
scipy            : None
sqlalchemy       : None
tables           : None
tabulate         : None
xarray           : None
xlrd             : None
xlwt             : None
xlsxwriter       : None
numba            : 0.49.1
</details>

Hints from issue discussion:
Seems to be fixed on master. Not sure if we have a test for it though.
Can I work on this?
@saiajay5674 PRs are welcome, see the [contributing guide](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html) for how to get started

Perhaps check first to see if there's already a test for this issue, and if there's not, you could add one
@MarcoGorelli We can create Pytest test cases, right?
> @MarcoGorelli We can create Pytest test cases, right?

Yes, that's the testing framework pandas uses. If you'd like to work on this, please comment 'take' so the issue is assigned to you
Take

Commit message: Added test for Category isin

Patch to review:
diff --git a/pandas/tests/frame/methods/test_isin.py b/pandas/tests/frame/methods/test_isin.py
index 6307738021f68..583647a5efafe 100644
--- a/pandas/tests/frame/methods/test_isin.py
+++ b/pandas/tests/frame/methods/test_isin.py
@@ -189,3 +189,17 @@ def test_isin_empty_datetimelike(self):
         tm.assert_frame_equal(result, expected)
         result = df1_td.isin(df3)
         tm.assert_frame_equal(result, expected)
+
+    def test_isin_against_categoryDataframe(self):
+        # GH#34256
+        df1 = pd.DataFrame.from_dict({"a": [1, 2, 3], "b": [4, 5, 6]})
+        df2 = pd.DataFrame({"a": [1, 2, 3]}, dtype="category")
+        df3 = pd.Series([1, 2, 3]).astype("category")
+
+        expected1 = DataFrame({"a": [True, True, True], "b": [False, False, False]})
+        expected2 = DataFrame({"a": [True, True, True], "b": [False, False, False]})
+        result1 = df1.isin(df2)
+        tm.assert_frame_equal(result1, expected1)
+
+        result2 = df1.isin(df3)
+        tm.assert_frame_equal(result2, expected2)


Code review comments:
diff_hunk: @@ -189,3 +189,17 @@ def test_isin_empty_datetimelike(self):
         tm.assert_frame_equal(result, expected)
         result = df1_td.isin(df3)
         tm.assert_frame_equal(result, expected)
+
+    def test_isin_against_categoryDataframe(self):
path: pandas/tests/frame/methods/test_isin.py
text: @user1:
don't use camelcase, test_isin_category_frame is fine

diff_hunk: @@ -189,3 +189,17 @@ def test_isin_empty_datetimelike(self):
         tm.assert_frame_equal(result, expected)
         result = df1_td.isin(df3)
         tm.assert_frame_equal(result, expected)
+
+    def test_isin_against_categoryDataframe(self):
+        # GH#34256
+        df1 = pd.DataFrame.from_dict({"a": [1, 2, 3], "b": [4, 5, 6]})
+        df2 = pd.DataFrame({"a": [1, 2, 3]}, dtype="category")
+        df3 = pd.Series([1, 2, 3]).astype("category")
+
+        expected1 = DataFrame({"a": [True, True, True], "b": [False, False, False]})
+        expected2 = DataFrame({"a": [True, True, True], "b": [False, False, False]})
+        result1 = df1.isin(df2)
+        tm.assert_frame_equal(result1, expected1)
+
+        result2 = df1.isin(df3)
+        tm.assert_frame_equal(result2, expected2)
path: pandas/tests/frame/methods/test_isin.py
text: @user1:
Would it work to reuse `result`? Something like
```suggestion
        result = df1.isin(df2)
        tm.assert_frame_equal(result, expected)

        result = df1.isin(df3)
        tm.assert_frame_equal(result, expected)
```
(assuming you renamed `expected` as above)

@author:
I have made the necessary changes.

diff_hunk: @@ -189,3 +189,17 @@ def test_isin_empty_datetimelike(self):
         tm.assert_frame_equal(result, expected)
         result = df1_td.isin(df3)
         tm.assert_frame_equal(result, expected)
+
+    def test_isin_against_categoryDataframe(self):
path: pandas/tests/frame/methods/test_isin.py
text: @user1:
pls parametrize this for the cases.

@author:
@user1 I have made the necessary changes. Please review it. Sorry for the inconvenience.

diff_hunk: @@ -189,3 +189,17 @@ def test_isin_empty_datetimelike(self):
         tm.assert_frame_equal(result, expected)
         result = df1_td.isin(df3)
         tm.assert_frame_equal(result, expected)
+
+    def test_isin_against_categoryDataframe(self):
+        # GH#34256
+        df1 = pd.DataFrame.from_dict({"a": [1, 2, 3], "b": [4, 5, 6]})
+        df2 = pd.DataFrame({"a": [1, 2, 3]}, dtype="category")
+        df3 = pd.Series([1, 2, 3]).astype("category")
+
+        expected1 = DataFrame({"a": [True, True, True], "b": [False, False, False]})
+        expected2 = DataFrame({"a": [True, True, True], "b": [False, False, False]})
path: pandas/tests/frame/methods/test_isin.py
text: @user1:
What's the difference between these two? Can you just use one, call it `expected`, and use it for both assertions?

diff_hunk: @@ -189,3 +189,17 @@ def test_isin_empty_datetimelike(self):
         tm.assert_frame_equal(result, expected)
         result = df1_td.isin(df3)
         tm.assert_frame_equal(result, expected)
+
+    def test_isin_against_categoryDataframe(self):
+        # GH#34256
+        df1 = pd.DataFrame.from_dict({"a": [1, 2, 3], "b": [4, 5, 6]})
+        df2 = pd.DataFrame({"a": [1, 2, 3]}, dtype="category")
+        df3 = pd.Series([1, 2, 3]).astype("category")
path: pandas/tests/frame/methods/test_isin.py
text: @user1:
Can you use a different name? `df3` makes it look like it's a DataFrame, but it's a Series