Repository: pandas-dev/pandas

Issue:
Index.isin() always True for nans
This looks like a regression to me, the following code used to print `[ True]`, which I think is the correct behaviour, but prints `[False]` in recent development versions (0.13.1.dev).

```
import numpy as np
import pandas as pd

i = pd.Index([np.nan])
print i.isin({0})  # prints [ True], should be [False]
```

I do not think the following is relevant but just in case:

```
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: es_ES.UTF-8

pandas: 0.13.1.dev
nose: 1.3.1
Cython: None
numpy: 1.8.1
scipy: 0.13.3
statsmodels: 0.6.0.dev-Unknown
IPython: 3.0.0-dev
sphinx: None
patsy: 0.2.1
scikits.timeseries: None
dateutil: 1.5
pytz: 2012c
bottleneck: 0.8.0
tables: 3.1.1
numexpr: 2.2.2
matplotlib: 1.3.1
openpyxl: 1.7.0
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: 0.5.2
lxml: 3.3.3
bs4: 4.2.1
html5lib: 0.999
bq: None
apiclient: None
rpy2: None
sqlalchemy: None
pymysql: None
psycopg2: 2.4.5 (dt dec mx pq3 ext)
```



Discussion:
This is intentional, to allow speedups in `Float64Index`

If you have a `nan` in your index and it can be converted to `Float64Index` it will. Before #6879, the backend was using `object` dtype, which allowed checks for `nan` internally, but `isin` relied on the behavior of `Index` which uses `set`. If you look at the beginning of the issue thread, I was pulling my hair out trying to figure out why `nan` behaves like an object in some cases and like a large integer in others.

Here's why: 

It's because internally `nan` is represented as a large integer, but when you use it as an object, it's a pointer. When it's used as an "object" it's okay to use in a `set` and you'll get `True` if `nan` is in a `set` and `False` otherwise. If you use it like a float value, then each one is a different object (large-ish integers in Python behave like this as well). Of course, you can compare integers and other numbers so e.g., `1 == 1` is `True` and it would be in the `set` object. However, `nan` by definition has the property that `nan == nan` is `False` so comparison of `id` (address) returns `False` and so does equality and thus `set` membership is `False` for these "different" kinds of `nan`s.

Oh wow, I just realized I misread what you wrote.

yeh I think mark as a bug

This is a bug. 


Resolving patch:
diff --git a/pandas/core/index.py b/pandas/core/index.py
--- a/pandas/core/index.py
+++ b/pandas/core/index.py
@@ -2096,7 +2096,8 @@ def isin(self, values):
         """
         value_set = set(values)
         return lib.ismember_nans(self._array_values(), value_set,
-                                 self._hasnans)
+                                 isnull(list(value_set)).any())
+
 
 class MultiIndex(Index):
 


Test patch:
diff --git a/pandas/tests/test_index.py b/pandas/tests/test_index.py
--- a/pandas/tests/test_index.py
+++ b/pandas/tests/test_index.py
@@ -913,6 +913,25 @@ def test_contains_not_nans(self):
         i = Float64Index([1.0, 2.0, np.nan])
         self.assertTrue(1.0 in i)
 
+    def test_doesnt_contain_all_the_things(self):
+        i = Float64Index([np.nan])
+        self.assertFalse(i.isin([0]).item())
+        self.assertFalse(i.isin([1]).item())
+        self.assertTrue(i.isin([np.nan]).item())
+
+    def test_nan_multiple_containment(self):
+        i = Float64Index([1.0, np.nan])
+        np.testing.assert_array_equal(i.isin([1.0]), np.array([True, False]))
+        np.testing.assert_array_equal(i.isin([2.0, np.pi]),
+                                      np.array([False, False]))
+        np.testing.assert_array_equal(i.isin([np.nan]),
+                                      np.array([False, True]))
+        np.testing.assert_array_equal(i.isin([1.0, np.nan]),
+                                      np.array([True, True]))
+        i = Float64Index([1.0, 2.0])
+        np.testing.assert_array_equal(i.isin([np.nan]),
+                                      np.array([False, False]))
+
 
 class TestInt64Index(tm.TestCase):
     _multiprocess_can_split_ = True