Repository: pandas-dev/pandas

Issue:
Non-monotonic-increasing DatetimeIndex claims not to __contain__ duplicate entries
This was fun to debug.

``` python
In [1]: import pandas as pd

In [2]: 0 in pd.Int64Index([0, 0, 1])
Out[2]: True

In [3]: 0 in pd.Int64Index([0, 1, 0])
Out[3]: True

In [4]: 0 in pd.Int64Index([0, 0, -1])
Out[4]: True

In [5]: pd.Timestamp(0) in pd.DatetimeIndex([0, 1, -1])
Out[5]: True

In [6]: pd.Timestamp(0) in pd.DatetimeIndex([0, 1, 0])
Out[6]: False   # BAD

In [7]: pd.Timestamp(0) in pd.DatetimeIndex([0, 0, 1])
Out[7]: True

In [8]: pd.Timestamp(0) in pd.DatetimeIndex([0, 0, -1])
Out[8]: False   # BAD
```

TimedeltaIndex is also broken.

The problem is in [`DatetimeIndexOpsMixin.__contains__`](https://github.com/pydata/pandas/blob/v0.15.2/pandas/tseries/base.py#L68), which checks the type of `idx.get_loc(key)` to determine whether the key was found in the index.  If the index contains duplicate entries and is not monotonic increasing (for some reason, monotonic decreasing doesn't cut it), `get_loc` eventually falls back to [`Int64Engine._maybe_get_bool_indexer`](https://github.com/pydata/pandas/blob/v0.15.2/pandas/index.pyx#L376), which returns an ndarray of bools if the key is duplicated.  Since the original `__contains__` method is looking for scalars or slices, it reports that the duplicated entry is not present.

FIX: Fix some instances where idx[0] not in idx
`DatetimeIndex.__contains__` and `TimedeltaIndex.__contains__` were failing to see duplicated elements in some circumstances.

Fixes #9512



Discussion:
Thanks for the report and the debugging. A PR to fix this would be very welcome!

I'm just afraid that some of the other indexing code accidentally depends on this bug and will blow up...

The fix here might be as simple as just adding `or np.any(res)` to that line... only way to find out is to try!

Truuuue....

The embarrassing truth is that I lost my SSH key for uploading to github, and haven't gotten around to regenerating it yet.  So I won't be able to squash.  /sheepish



Resolving patch:
diff --git a/pandas/tseries/base.py b/pandas/tseries/base.py
--- a/pandas/tseries/base.py
+++ b/pandas/tseries/base.py
@@ -65,7 +65,7 @@ def _format_with_header(self, header, **kwargs):
     def __contains__(self, key):
         try:
             res = self.get_loc(key)
-            return np.isscalar(res) or type(res) == slice
+            return np.isscalar(res) or type(res) == slice or np.any(res)
         except (KeyError, TypeError):
             return False
 


Test patch:
diff --git a/pandas/tseries/tests/test_base.py b/pandas/tseries/tests/test_base.py
--- a/pandas/tseries/tests/test_base.py
+++ b/pandas/tseries/tests/test_base.py
@@ -292,6 +292,12 @@ def test_value_counts_unique(self):
 
             tm.assert_index_equal(idx.unique(), exp_idx)
 
+    def test_nonunique_contains(self):
+        # GH 9512
+        for idx in map(DatetimeIndex, ([0, 1, 0], [0, 0, -1], [0, -1, -1],
+                                       ['2015', '2015', '2016'], ['2015', '2015', '2014'])):
+            tm.assertIn(idx[0], idx)
+
 
 class TestTimedeltaIndexOps(Ops):
 
@@ -684,6 +690,14 @@ def test_value_counts_unique(self):
 
         tm.assert_index_equal(idx.unique(), exp_idx)
 
+    def test_nonunique_contains(self):
+        # GH 9512
+        for idx in map(TimedeltaIndex, ([0, 1, 0], [0, 0, -1], [0, -1, -1],
+                                        ['00:01:00', '00:01:00', '00:02:00'],
+                                        ['00:01:00', '00:01:00', '00:00:01'])):
+            tm.assertIn(idx[0], idx)
+
+
 class TestPeriodIndexOps(Ops):
 
     def setUp(self):