Repository: pandas-dev/pandas

Issue:
ENH: Dataframe should have a .isin() method
Any reason not to give `DataFrame` a `.isin()` method like `Series`?

The new wrinkle is that the user needs to specify if they want a logical OR or AND.

e.g.

``` python
In [26]: df = pd.DataFrame({'vals': [1, 2, 3, 4], 'ids': ['a', 'b', 'f', 'f'], 
'ids2': ['e', 'f', 'c', 'f']})

In [27]: df
Out[27]: 
  ids ids2  vals
0   a    e     1
1   b    f     2
2   f    c     3
3   f    f     4

In [1]: other_ids = pd.Series(['a', 'b', 'c', 'c'])

df.isin(other_ids, how='or')

  ids ids2  vals
0   a    e     1
1   b    f     2
2   f    c     3

df.isin(other_ids, how='and')
```

See [this SO post](http://stackoverflow.com/questions/17596733/filtering-rows-from-pandas-dataframe-using-concatenated-strings/17597500#17597500) maybe.

If someone else wants to take this, feel free. Can't promise a PR any time soon, but maybe in the fall :)



Discussion:
i like this idea but it should return a bool not the frame indexed by and-ing or or-ing the `isin`s, because the bool version is consistent with `Series.isin`

Absolutely on returning the bool.  Mistake on my part.

The most obvious uses are how='and' and how='or', which will return a 1-d bool that can be indexed into.

Would it also be useful to have some way to get just the locations where the value is in the array?

``` python
#pseduocode

df.isin(['a', 'b', 'c'], how='any')  # any probably not the right argument name.

     ids   ids2   vals
0   True  False  False
1   True  False  False
2  False   True  False
3  False  False  False
```

This can be achieved by applying `.isin()` to each series and then concatenating together, or even using `df.applymap(lamda x: x in arr)`.  Not sure we'd need a third way.  I'd say just 'and' and 'or' for now.

If no one else minds, I can try to take a crack at this.  I haven't done much with cython so you might want to label it someday.  It looks like the `Series.isin()` uses `ismember` from `lib.pyx`.  I was thinking about either using that for each column of `df` we're checking for, or writing something similar for the `DataFrame` version.

Is there any preference for a default on `and` vs. `or` for the `how` parameter?

Also for the examples I've been giving it should be `df[['ids', 'ids2']].isin()`, not `df.isin()` since the `vals` column won't be in the reference array.

u can prob just do this is python
iterate over columns and use isin

if its not ast enough could change (but also good to get algo right first)

Yep.  Should be much easier than I was thinking.  I'll try to get this done today.  Going to (possibly) be without internet until Sunday, but I should have it done by then.

So right now I'm thinking about going with @jreback's idea and calling isin on each column.

I'm getting one failure locally... (unreleated to what I've change, I think.)

```
FAIL: test_convert_objects (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/tom/pandas/pandas/tests/test_frame.py", line 5486, in test_convert_objects
    assert_frame_equal(result, expected)
  File "/Users/tom/pandas/pandas/util/testing.py", line 238, in assert_frame_equal
    check_less_precise=check_less_precise)
  File "/Users/tom/pandas/pandas/util/testing.py", line 197, in assert_series_equal
    assert_almost_equal(left.values, right.values, check_less_precise)
  File "/Users/tom/pandas/pandas/util/testing.py", line 141, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/Users/tom/pandas/pandas/util/testing.py", line 129, in assert_almost_equal
    assert a == b, "%s != %s" % (a, b)
AssertionError: na != nan
```

I've got it pushed to my branch [here](https://github.com/TomAugspurger/pandas/tree/df_isin).  I'll do a proper PR in a sec, but I forgot to add the changes the release notes.  Can I still make update the release notes and everything into the same commit without screwing up world?  I know you aren't supposed to rebase changes pushed to a remote, but in this case is remote `pandas/pandas` or does `TomAugspurger/pandas` count too?

I get this but only when I use tox. Weird 

I think I'm also going to add an `ordered` argument. If you pass that as True then the function will expect a sequence of tuples. That may be more common than just a flat array. 

Design question for you all.  If we have

``` python

In [43]: df = DataFrame({'A': ['a', 'b', 'c', 'd'], 'B': [1, 2, 3, 4]})

In [44]: df
Out[44]: 
   A  B
0  a  1
1  b  2
2  c  3
3  d  4
```

with something like `other = [('a'), (1, 3)]`.  I'd like to compare column 'A' to 'a' and column 'B' to `(1, 3)`.  Should we insist that each sequence be a tuple or list?  The way I've done it so far requires that each sub-sequence in other be iterable.  So `other =['a', (1, 3)]` would work, but `other = [('a'), 1]` would fail since `int`s are not iterable. Not a great outcome.

FWIW, `Series` will fail on something like `.isin(1)` which makes since because in that case you'd just do `series == 1`.

hm do you mean `other = [('a',), 1]`? `[('a'), 1] == ['a', 1]` since parens with no commas forms an expression.

if it's a scalar then make it a list and it will be fine. OTOH it might be useful to make `s.isin(some_scalar) == s == some_scalar` where `s` is a `Series` object. Then you wouldn't have to worry about forming a `list` since it would be done in `Series`


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -5481,6 +5481,41 @@ def to_period(self, freq=None, axis=0, copy=True):
 
         return self._constructor(new_data)
 
+
+    def isin(self, values, iloc=False):
+        """
+        Return boolean DataFrame showing whether each element in the DataFrame is
+        contained in values.
+
+        Parameters
+        ----------
+        values : iterable or dictionary of columns to values
+        iloc : boolean, if passing a dict as values, describe columns using integer
+                        locations (default is to use labels)
+
+        Returns
+        -------
+
+        DataFrame of booleans
+        """
+        if isinstance(values, dict):
+            from collections import defaultdict
+            from pandas.tools.merge import concat
+            values = defaultdict(list, values)
+            if iloc:
+                return concat((self.iloc[:, [i]].isin(values[i])
+                                 for i, col in enumerate(self.columns)), axis=1)
+            else:
+                return concat((self.iloc[:, [i]].isin(values[col])
+                                 for i, col in enumerate(self.columns)), axis=1)
+
+
+        else:
+            return DataFrame(lib.ismember(self.values.ravel(),
+                                          set(values)).reshape(self.shape),
+                             self.index,
+                             self.columns)
+
     #----------------------------------------------------------------------
     # Deprecated stuff
 


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -10633,6 +10633,55 @@ def _check_f(base, f):
         f = lambda x: x.rename({1: 'foo'}, inplace=True)
         _check_f(data.copy()['c'], f)
 
+    def test_isin(self):
+        # GH #4211
+        df = DataFrame({'vals': [1, 2, 3, 4], 'ids': ['a', 'b', 'f', 'n'],
+                        'ids2': ['a', 'n', 'c', 'n']},
+                        index=['foo', 'bar', 'baz', 'qux'])
+        other = ['a', 'b', 'c']
+
+        result = df.isin(other)
+        expected = DataFrame([df.loc[s].isin(other) for s in df.index])
+        assert_frame_equal(result, expected)
+
+    def test_isin_empty(self):
+        df = DataFrame({'A': ['a', 'b', 'c'], 'B': ['a', 'e', 'f']})
+        result = df.isin([])
+        expected = pd.DataFrame(False, df.index, df.columns)
+        assert_frame_equal(result, expected)
+
+    def test_isin_dict(self):
+        df = DataFrame({'A': ['a', 'b', 'c'], 'B': ['a', 'e', 'f']})
+        d = {'A': ['a']}
+
+        expected = DataFrame(False, df.index, df.columns)
+        expected.loc[0, 'A'] = True
+
+        result = df.isin(d)
+        assert_frame_equal(result, expected)
+
+        # non unique columns
+        df = DataFrame({'A': ['a', 'b', 'c'], 'B': ['a', 'e', 'f']})
+        df.columns = ['A', 'A']
+        expected = DataFrame(False, df.index, df.columns)
+        expected.loc[0, 'A'] = True
+        result = df.isin(d)
+        assert_frame_equal(result, expected)
+
+        # iloc
+        df = DataFrame({'A': ['a', 'b', 'c'], 'B': ['a', 'e', 'f']})
+        d = {0: ['a']}
+        expected = DataFrame(False, df.index, df.columns)
+
+        # without using iloc
+        result = df.isin(d)
+        assert_frame_equal(result, expected)        
+
+        # using iloc
+        result = df.isin(d, iloc=True)
+        expected.iloc[0, 0] = True
+        assert_frame_equal(result, expected)        
+
 
 if __name__ == '__main__':
     # unittest.main()