Repository: pandas-dev/pandas

Issue:
ENH/API: DataFrame's isin should accept DataFrames
From [this](http://stackoverflow.com/questions/17979327/drop-rows-with-multiple-keys-in-pandas/17979614?noredirect=1#comment26286280_17979614) SO question.

The user had two dataframes and wanted to use the second as the values argument to `isin`. 

We'd need to think about how to handle the other index.  In this case the user only cared about the columns, not the index labels.

Previous issues/PRs: #4258 and #4211 



Discussion:
So, as you say, at the moment this works by passing in:

```
df1.isin(df2.to_dict(outtype='list'))
```

Is this an efficient operation? Should we just call that if passed a DataFrame?

FYI I believe this will fail with non-unique columns (as to_dict will fail); look at itertuples on a way to deal with this

Should `isin` accept something like a set operator mode? 

maybe `mode=diff` for set differences, or `mode=and` for inclusion? 

http://stackoverflow.com/questions/18180763/set-difference-for-pandas/18187648#18187648

see #4617 from same OP.

I'm not sure I'm 100% with how this was going to work, because the column approach no longer makes sense once we're talking about DataFrames, or at least is more ambiguous (aside from repeated columns):

```
In [2]: df1 = pd.DataFrame([[1, 2], [2, 3]])

In [3]: df2 = pd.DataFrame([[1, 3]])

In [4]: df2.isin(df1.to_dict(outtype='list'))
Out[4]: 
      0     1
0  True  True
```

which probably isn't what the user was expecting.

Which may be something like (possibly user doesn't care about the index):

```
In [11]: pd.Series(map(set(df1.itertuples()).__contains__, df2.itertuples()), df2.index)
Out[11]: 
0    False
dtype: bool
```

Ahem... Not sure what would be an efficient way to do that?

@jreback imo set intersection stuff can relatively easily be done after the fact, e.g. (all/any and boolean indexing, really don't think we need to add loads of kwargs.

@hayd @TomAugspurger for 0.13?

Definitely.

@hayd @TomAugspurger what is left on this?

To actually write this up... I think it's kind of important for 0.13 as otherwise isin is a little confusing (and will break code if we change the API later)....

Thinking about it again, do we actually want to eq **(ag. there a problem with dupes here...)**:

```
In [32]: df1 = pd.DataFrame([[1, 2], [2, 3]], columns=['a', 'b'])

In [33]: df2 = pd.DataFrame([[1, 3]], columns=['a', 'b'])

In [34]: df2.isin(df1.to_dict(outtype='list'))
Out[34]: 
      a     b
0  True  True

In [35]: df2.eq(df1.reindex_like(df2))  # is this isin for DataFrames?
Out[35]: 
      a      b
0  True  False
```

_Worst case we should NotImplement it with a mention of eq..._

@hayd Are you working on this one?  I can give it a shot tonight / tomorrow if you're wanting to focus on the SQLAlchemy stuff.

---

Let's collect some thoughts on what should happen when another DataFrame is passed as values.
1. Identical indicies, no dupes

``` python
In [20]: df1
Out[20]: 
   A   B
0  1   2
1  2 NaN
2  3   4
3  4   4

In [21]: df2
Out[21]: 
    A   B
0   0   2
1   2 NaN
2  12   4
3   4   5

# Expected: df1.isin(df2.to_dict(outtype='list')) does not work
# df1.eq(df2.reindex_like(df1)) does work
Out[23]: 
       A      B
0  False   True
1   True  False
2  False   True
3   True   False
```
1. Dissimilar indices, still no dupes. Should be intersection of indices, check isin on those (optional parameter to ignore index?).

``` python
# df1 same as before
In [35]: df2 = pd.DataFrame({'A': [0, 2, 12, 4], 'C': [2, np.nan, 4, 5]}, index=[0, 1, 3, 4])

In [36]: df2
Out[36]: 
    A   C
0   0   2
1   2 NaN
3  12   4
4   4   5

# Intersection of columns is A, intersection of indices is [0, 1, 3]
# df1.isin(df2.to_dict(outtype='list')) does *NOT* work since it matches (A, 3), it ignores the index.
# Andy's df1.eq(df2.reindex_like(df1)) does work here.
In [39]: df1.isin(df2.to_dict(outtype='list'))
Out[39]: 
       A      B
0  False  False
1   True  False
2  False  False
3  False  False
```
1. Duplicate columns on `values` (the argument). The two options are to `raise` or return `True` wherever there's a match.

``` python
In [74]: df1
Out[74]: 
   A   B
0  1   2
1  2 NaN
2  3   4
3  4   4

In [75]: df2
Out[75]: 
    A   A
0   0   1
1   2   4
3   2 NaN
4   4   5

# Expected if we return True wherever there's a True

       A      B
0   True  False   # from the second A column
1   True  False   # from the first A column
2  False  False
3  False  False
```

~~The `df1.eq(df2.reindex_like(df1))` does not work here, this hits the maximum recursion depth.~~
I had my `df`s backwards; that's actually a `ValueError: cannot reindex from a duplicate axis`.

Oh we definitely need to get this in for .13.  I think DataFrames and Series should behave similarly with respect to labels.  Currently, if `values` is a Series, the index is ignored:

``` python
In [150]: df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [2, np.nan, 4, 4]}, index=['a','b','c','d'])

In [151]: s = pd.Series([1, 3, 11, 12], index=['a','b','c','d'])

In [152]: df.isin(s)
Out[152]: 
       A      B
a   True  False
b  False  False
c   True  False
d  False  False
```

So (A, c) matches on the 3 from the Series, despite the index of that 3 being `b`, not `c`.

What's everyone's thoughts on respecting index/column labels?  I'd say that (A, c) should be False here since the labels don't match.

@TomAugspurger there was a bug in comparing duplicate frames (so your above comparison will work as you had it, or raise if you don't reindex_like), but won't infinity recurse (which generally is a bad thing :)

will merge in a few

```
In [1]: df1 = DataFrame([[1,2],[2,np.nan],[3,4],[4,4]],columns=['A','B'])

In [2]: df2 = DataFrame([[0,1],[2,4],[2,np.nan],[4,5]],columns=['A','A'])

In [3]: df1
Out[3]: 
   A   B
0  1   2
1  2 NaN
2  3   4
3  4   4

In [4]: df2
Out[4]: 
   A   A
0  0   1
1  2   4
2  2 NaN
3  4   5

In [5]: df1.reindex_like(df2) == df2
Out[5]: 
       A      A
0  False   True
1   True  False
2  False  False
3   True  False
```

@TomAugspurger if you need to test if the the columns are duplicated, use `df.columns.is_unique`

@TomAugspurger that bug fix merged in....


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -4239,35 +4239,71 @@ def to_period(self, freq=None, axis=0, copy=True):
 
         return self._constructor(new_data)
 
-
-    def isin(self, values, iloc=False):
+    def isin(self, values):
         """
-        Return boolean DataFrame showing whether each element in the DataFrame is
-        contained in values.
+        Return boolean DataFrame showing whether each element in the
+        DataFrame is contained in values.
 
         Parameters
         ----------
-        values : iterable or dictionary of columns to values
-        iloc : boolean, if passing a dict as values, describe columns using integer
-                        locations (default is to use labels)
+        values : iterable, Series, DataFrame or dictionary
+            The result will only be true at a location if all the
+            labels match. If `values` is a Series, that's the index. If
+            `values` is a dictionary, the keys must be the column names,
+            which must match. If `values` is a DataFrame,
+            then both the index and column labels must match.
 
         Returns
         -------
 
         DataFrame of booleans
+
+        Examples
+        --------
+        When ``values`` is a list:
+
+        >>> df = DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'f']})
+        >>> df.isin([1, 3, 12, 'a'])
+               A      B
+        0   True   True
+        1  False  False
+        2   True  False
+
+        When ``values`` is a dict:
+
+        >>> df = DataFrame({'A': [1, 2, 3], 'B': [1, 4, 7]})
+        >>> df.isin({'A': [1, 3], 'B': [4, 7, 12]})
+               A      B
+        0   True  False  # Note that B didn't match the 1 here.
+        1  False   True
+        2   True   True
+
+        When ``values`` is a Series or DataFrame:
+
+        >>> df = DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'f']})
+        >>> other = DataFrame({'A': [1, 3, 3, 2], 'B': ['e', 'f', 'f', 'e']})
+        >>> df.isin(other)
+               A      B
+        0   True  False
+        1  False  False  # Column A in `other` has a 3, but not at index 1.
+        2   True   True
         """
         if isinstance(values, dict):
             from collections import defaultdict
             from pandas.tools.merge import concat
             values = defaultdict(list, values)
-            if iloc:
-                return concat((self.iloc[:, [i]].isin(values[i])
-                                 for i, col in enumerate(self.columns)), axis=1)
-            else:
-                return concat((self.iloc[:, [i]].isin(values[col])
-                                 for i, col in enumerate(self.columns)), axis=1)
-
-
+            return concat((self.iloc[:, [i]].isin(values[col])
+                           for i, col in enumerate(self.columns)), axis=1)
+        elif isinstance(values, Series):
+            if not values.index.is_unique:
+                raise ValueError("ValueError: cannot compute isin with"
+                                 " a duplicate axis.")
+            return self.eq(values.reindex_like(self), axis='index')
+        elif isinstance(values, DataFrame):
+            if not (values.columns.is_unique and values.index.is_unique):
+                raise ValueError("ValueError: cannot compute isin with"
+                                 " a duplicate axis.")
+            return self.eq(values.reindex_like(self))
         else:
             if not is_list_like(values):
                 raise TypeError("only list-like or dict-like objects are"


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -11431,20 +11431,6 @@ def test_isin_dict(self):
         result = df.isin(d)
         assert_frame_equal(result, expected)
 
-        # iloc
-        df = DataFrame({'A': ['a', 'b', 'c'], 'B': ['a', 'e', 'f']})
-        d = {0: ['a']}
-        expected = DataFrame(False, df.index, df.columns)
-
-        # without using iloc
-        result = df.isin(d)
-        assert_frame_equal(result, expected)
-
-        # using iloc
-        result = df.isin(d, iloc=True)
-        expected.iloc[0, 0] = True
-        assert_frame_equal(result, expected)
-
     def test_isin_with_string_scalar(self):
         #GH4763
         df = DataFrame({'vals': [1, 2, 3, 4], 'ids': ['a', 'b', 'f', 'n'],
@@ -11456,6 +11442,84 @@ def test_isin_with_string_scalar(self):
         with tm.assertRaises(TypeError):
             df.isin('aaa')
 
+    def test_isin_df(self):
+        df1 = DataFrame({'A': [1, 2, 3, 4], 'B': [2, np.nan, 4, 4]})
+        df2 = DataFrame({'A': [0, 2, 12, 4], 'B': [2, np.nan, 4, 5]})
+        expected = DataFrame(False, df1.index, df1.columns)
+        result = df1.isin(df2)
+        expected['A'].loc[[1, 3]] = True
+        expected['B'].loc[[0, 2]] = True
+        assert_frame_equal(result, expected)
+
+        # partial overlapping columns
+        df2.columns = ['A', 'C']
+        result = df1.isin(df2)
+        expected['B'] = False
+        assert_frame_equal(result, expected)
+
+    def test_isin_df_dupe_values(self):
+        df1 = DataFrame({'A': [1, 2, 3, 4], 'B': [2, np.nan, 4, 4]})
+        # just cols duped
+        df2 = DataFrame([[0, 2], [12, 4], [2, np.nan], [4, 5]],
+                        columns=['B', 'B'])
+        with tm.assertRaises(ValueError):
+            df1.isin(df2)
+
+        # just index duped
+        df2 = DataFrame([[0, 2], [12, 4], [2, np.nan], [4, 5]],
+                        columns=['A', 'B'], index=[0, 0, 1, 1])
+        with tm.assertRaises(ValueError):
+            df1.isin(df2)
+
+        # cols and index:
+        df2.columns = ['B', 'B']
+        with tm.assertRaises(ValueError):
+            df1.isin(df2)
+
+    def test_isin_dupe_self(self):
+        other = DataFrame({'A': [1, 0, 1, 0], 'B': [1, 1, 0, 0]})
+        df = DataFrame([[1, 1], [1, 0], [0, 0]], columns=['A','A'])
+        result = df.isin(other)
+        expected = DataFrame(False, index=df.index, columns=df.columns)
+        expected.loc[0] = True
+        expected.iloc[1, 1] = True
+        assert_frame_equal(result, expected)
+
+
+    def test_isin_against_series(self):
+        df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [2, np.nan, 4, 4]},
+                          index=['a', 'b', 'c', 'd'])
+        s = pd.Series([1, 3, 11, 4], index=['a', 'b', 'c', 'd'])
+        expected = DataFrame(False, index=df.index, columns=df.columns)
+        expected['A'].loc['a'] = True
+        expected.loc['d'] = True
+        result = df.isin(s)
+        assert_frame_equal(result, expected)
+
+    def test_isin_multiIndex(self):
+        idx = MultiIndex.from_tuples([(0, 'a', 'foo'), (0, 'a', 'bar'),
+                                      (0, 'b', 'bar'), (0, 'b', 'baz'),
+                                      (2, 'a', 'foo'), (2, 'a', 'bar'),
+                                      (2, 'c', 'bar'), (2, 'c', 'baz'),
+                                      (1, 'b', 'foo'), (1, 'b', 'bar'),
+                                      (1, 'c', 'bar'), (1, 'c', 'baz')])
+        df1 = DataFrame({'A': np.ones(12),
+                         'B': np.zeros(12)}, index=idx)
+        df2 = DataFrame({'A': [1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1],
+                         'B': [1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1]})
+        # against regular index
+        expected = DataFrame(False, index=df1.index, columns=df1.columns)
+        result = df1.isin(df2)
+        assert_frame_equal(result, expected)
+
+        df2.index = idx
+        expected = df2.values.astype(np.bool)
+        expected[:, 1] = ~expected[:, 1]
+        expected = DataFrame(expected, columns=['A', 'B'], index=idx)
+
+        result = df1.isin(df2)
+        assert_frame_equal(result, expected)
+
     def test_to_csv_date_format(self):
         from pandas import to_datetime
         pname = '__tmp_to_csv_date_format__'