ENH/API: Accept DataFrame for isin
State: closed

API: Series should respect index labels. This would be incompatible, but there hasn't been any official release since `isin` was merged.

Will close #4421.  Still a WIP for now. Needs more testing.

Thoughts on adding an argument to ignore the index? i.e.

``` python
In [150]: df = pd.DataFrame({'A': [1, 2, 3, 4], 'B': [2, np.nan, 4, 4]}, index=['a','b','c','d'])

In [151]: s = pd.Series([1, 3, 11, 12], index=['a','b','c','d'])

In [152]: df.isin(s)
Out[152]: 
       A      B
a   True  False
b  False  False
c  False  False
d  False  False

In [152]: df.isin(s, ignore_index=True)
Out[152]: 
       A      B
a   True  False
b  False  False
c   True  False
d  False  False
```

I would say there's no need for an `ignore_index` since you could just do `df.isin(s.values)`.


Labels:

Comments:
maybe add/modify the example in v0.13.0 and/or docs? (to show using df), which IMHO will prob be more of a usecase than a dict

looks good....@hayd ?

Aside from that little thing, this looks good!

tbh I think ignore_index may be useful kwarg for some, but we could push it back to 0.14.

@hayd sorry, my last push squashed the discussion about isin with duplicate indicies/columns on `values` here: https://github.com/pydata/pandas/pull/5199#discussion_r6960880

> Sorry, my last comment was wrong.  You're obviously talking about `values` for both.  What do you think about 
> 
> ``` python
> In [7]: df
> Out[7]: 
>    A  B
> 0  1  1
> 1  0  1
> 2  1  0
> 3  0  0
> 
> In [8]: other
> Out[8]: 
>    A  A
> 0  1  1
> 1  1  0
> 2  0  0
> 
> in [9]: df.isin(other)
> ```
> 
> `(A, 0)` is clearly True and `(A, 2)` clearly False, but I don't see any way to make sense of `(A, 1)`.

since the first `A` in other (1) says False, but the second `A` in other (0) says False

Once we get this figured out I'll rebase / squash and it will be good to go.

@TomAugspurger ah ha, you're right that makes no sense, I was thinking the other direction:

```
other.isin(df)
```

which would (in this case) make sense, since df has unique column names... hence the NotImplemented, as potentially be fleshed out for this behaviour in the future/someday (!)

[Failing on 2.6](https://travis-ci.org/TomAugspurger/pandas/jobs/12549634)

```
cannot import name hashtable
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "pandas/__init__.py", line 9, in <module>
    from . import hashtable, tslib, lib
ImportError: cannot import name hash table
```

Is this a known problem or did I break something in the build script?

just have it rebuild again

`git commit -C HEAD --amend` and force push....

travis hiccup

@hayd I think you already had that one sorted out:

``` python
In [8]: df
Out[8]: 
   A  B
0  1  1
1  0  1
2  1  0
3  0  0

In [9]: other
Out[9]: 
   A  A
0  1  1
1  1  0
2  0  0

In [10]: other.isin(df)
Out[10]: 
       A      A
0   True   True
1  False   True
2  False  False
```

Good to go on my end: https://travis-ci.org/TomAugspurger/pandas/builds/12617771

Ah ha, excellent! Then this looks great :), is there/can we add a test for that?

I guess @jorisvandenbossche's point was that the arg description of values is confusing (mainly as there are several cases for it)! To address this one option is to add a few examples into the docstring (copy some of the ones already written in the docs :) ), comparing to list, series/dict, dataframe.

@hayd No, it was actually only from an aesthetical point :-) But after reading it, you are right that is is not that clear. Maybe the options can be listed in bullets. 
And some examples are always good! I am much in favor of more examples in docstrings

@hayd added examples in doctoring and a test for dupes on self.

@TomAugspurger Short, clear but illustrative examples. Nice!

@hayd ok just wanted to be sure (and since the default is the use labels this is sort of innocuous)

OK so we're keeping the API the same? The `iloc`  kwarg stays in?
@jorisvandenbossche I've fixed those typos/formatting/suggestions.  Thanks.

@hayd I reviewed this again
this iloc arg needs to go; don't have any tests for it in any event
and it makes isin incompatible with series.isin
I don't see a compelling case for it at all 

@jreback there is a test for it: `test_isin_dict`

You're right it shouldn't be called iloc, as it's not clear it's to do with **column** location (rather than index). I don't see how this is incompatible with Series isin and thought it _could_ be useful....

@hayd so in the future, I think an ignore index argument would achieve the same thing? Something like `by_loc` Would be useful for dicts, frames, and series. Ignore labels just match on location. I wouldn't be able to get to that this week, so maybe remove the iloc arg for now, to be reimplemented in q.14. 

@hayd @TomAugspurger I like `ignore_index` rather than iloc....

prob not going to cut RC for a few days and you can change even after that...so lmk

i thought that ignore index is slightly different to iloc:

```
In [1]: df = pd.DataFrame([['a', 'b'], ['c', 'a']], columns=['A', 'B'])

In [2]: df.isin(pd.Series(['a']))
Out[2]: 
       A      B
0   True  False
1  False  False

In [3]: df.isin(['a']) # this would be ignore_index=True of [2]
Out[3]: 
       A      B
0   True  False
1  False   True
```

Whereas iloc is the position of column:

```
In [4]: df.isin({0: ['a']})
Out[4]: 
       A      B
0  False  False
1  False  False

In [5]: df.isin({0: ['a']}, iloc=True)  # 0th columns
Out[5]: 
       A      B
0   True  False
1  False  False
```

I think iloc does not work with DataFrames or with Series, so if True and one of those is passed it should raise. Could cut for now I guess... :(

`ignore_index` would need an axis argument for which axis (or axes for DataFrame) you'd like to ignore. This is kinda-sorta like ignoring the keys of the dict and going with the location (but still slightly-different than `iloc` like you say).

``` python
In [14]: df = pd.DataFrame([['a', 'b'], ['c', 'a']], columns=['A', 'B'])

In [17]: df.isin(pd.Series(['a', 'c'], index=[0, 2]))
Out[17]: 
       A      B
0   True  False
1  False  False

In [18]: df.isin(pd.Series(['a', 'c'], index=[0, 2]), ignore_index=True)
Out[18]: 
       A      B
0  True  False
1  True  False  #  ignore that 1 != 2

In [33]: other = pd.DataFrame({'A': ['a', 'c'], 'C': ['b', 'd']}, index=[0, 2])


In [34]: other
Out[34]: 
   A  C
0  a  b
2  c  d

In [35]: df.isin(other)
Out[35]: 
       A      B
0   True  False
1  False  False

In [35]: df.isin(other, ignore_index=0)  # ignore the index
Out[35]: 
       A      B
0   True  False  # b still false since B != C
1   True  False  # now A,2 is true

In [35]: df.isin(other, ignore_index=1)  # ignore cols; like ignoring keys for dict?
Out[35]: 
       A      B
0   True  True
1  False  False

In [35]: df.isin(other, ignore_index='both')  # ignore both; 
Out[35]: 
       A      B
0   True  True
1   True  False
```

I think conceptually the idea of matching based on _location_ rather than _label_ is similar enough between dicts and indexes/columns that they can share the kwarg.

I still don't see a nice use case for `iloc`.  Since its much harder to take out a parameter after its release, and if IIRC `DataFrame.isin` hasn't been released yet (was put in right after 0.13 started out?), let's drop `iloc`.

Unless I am missing something....

I think Andy put it in _seconds_ after .12 was released :)

I just pushed a version without `iloc` and with the `iloc` tests removed. I'd be more comfortable going with this for now, and adding it in later.  (But my opinion may not count for much since I still don't quite understand how the `iloc` code works in the first place :) )

@jorisvandenbossche I went with the double backticks.  Looks better in the documentation.

ok....this looks good....

@hayd ok?

sulk! fine... Thanks for putting this all together, merged! :)

Note: there is still a use of the `iloc` keyword in the docs, which now causes errors. From indexing.rst:

```
You can also describe columns using integer location:

.. ipython:: python

   values = {0: ['a', 'b']}

   df.isin(values, iloc=True)
```