Repository: pandas-dev/pandas
Pull request: [BUG]: Fix bug in MultiIndex.drop dropped nan when non existing key was given
- [x] closes #18853
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Additional question:
The doc says (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.drop.html) that ``drop`` raises when not all labels are found, but
```
pd.DataFrame(index=pd.MultiIndex.from_product([range(3), range(3)])).drop([1, 5], level=0)
```
does not raise, because ``1`` is found. Should we implement, that it raises every time when at least one of the keys is not found?

Problem statement:
DataFrame.drop(Timestamp) on MultiIndex with NaT incorrectly drops the NaT row
#### Code Sample, a copy-pastable example if possible

```python
df = pd.DataFrame(
    index=pd.MultiIndex.from_tuples([('blah', pd.NaT)],
                                    names=['name', 'date']),
)
print df

Empty DataFrame
Columns: []
Index: [(blah, NaT)]

print df.drop(pd.Timestamp('2001'), level='date')

Empty DataFrame
Columns: []
Index: []

```
#### Problem description

Timestamp('2001') isn't actually in the index. The drop() call should raise an error. For example when just operating on a single-level datetime index it throws this error:
```python
df = pd.DataFrame(index=[pd.NaT])
df.drop(pd.Timestamp('2001'))
ValueError: labels [Timestamp('2001-01-01 00:00:00')] not contained in axis
```
I would expect the same thing. Silently dropping the NaT value was causing a hard to find bug in my code.

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-97-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.21.0
pytest: None
pip: None
setuptools: None
Cython: None
numpy: 1.11.1
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 2.0.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.3
blosc: None
bottleneck: None
tables: 3.1.1
numexpr: 2.5
feather: None
matplotlib: 1.5.2
openpyxl: None
xlrd: 0.9.3
xlwt: None
xlsxwriter: None
lxml: 3.4.4
bs4: 4.5.3
html5lib: 1.0b8
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.0
</details>

Hints from issue discussion:

Commit message: Fix if condition again...

Patch to review:
diff --git a/doc/source/whatsnew/v1.2.0.rst b/doc/source/whatsnew/v1.2.0.rst
index f751a91cecf19..650478322068b 100644
--- a/doc/source/whatsnew/v1.2.0.rst
+++ b/doc/source/whatsnew/v1.2.0.rst
@@ -483,6 +483,7 @@ MultiIndex
 - Bug in :meth:`DataFrame.xs` when used with :class:`IndexSlice` raises ``TypeError`` with message ``"Expected label or tuple of labels"`` (:issue:`35301`)
 - Bug in :meth:`DataFrame.reset_index` with ``NaT`` values in index raises ``ValueError`` with message ``"cannot convert float NaN to integer"`` (:issue:`36541`)
 - Bug in :meth:`DataFrame.combine_first` when used with :class:`MultiIndex` containing string and ``NaN`` values raises ``TypeError`` (:issue:`36562`)
+- Bug in :meth:`MultiIndex.drop` dropped ``NaN`` values when non existing key was given as input (:issue:`18853`)
 
 I/O
 ^^^
diff --git a/pandas/core/indexes/multi.py b/pandas/core/indexes/multi.py
index 5a3f2b0853c4f..e18e7d959e3e4 100644
--- a/pandas/core/indexes/multi.py
+++ b/pandas/core/indexes/multi.py
@@ -2156,6 +2156,8 @@ def _drop_from_level(self, codes, level, errors="raise"):
         i = self._get_level_number(level)
         index = self.levels[i]
         values = index.get_indexer(codes)
+        nan_codes = isna(codes)
+        values[(np.equal(nan_codes, False)) & (values == -1)] = -2
 
         mask = ~algos.isin(self.codes[i], values)
         if mask.all() and errors != "ignore":
diff --git a/pandas/tests/indexes/multi/test_drop.py b/pandas/tests/indexes/multi/test_drop.py
index 6ba565f0406ab..06019ed0a8b14 100644
--- a/pandas/tests/indexes/multi/test_drop.py
+++ b/pandas/tests/indexes/multi/test_drop.py
@@ -139,3 +139,11 @@ def test_drop_not_lexsorted():
     tm.assert_index_equal(lexsorted_mi, not_lexsorted_mi)
     with tm.assert_produces_warning(PerformanceWarning):
         tm.assert_index_equal(lexsorted_mi.drop("a"), not_lexsorted_mi.drop("a"))
+
+
+def test_drop_with_nan_in_index(nulls_fixture):
+    # GH#18853
+    mi = MultiIndex.from_tuples([("blah", nulls_fixture)], names=["name", "date"])
+    msg = r"labels \[Timestamp\('2001-01-01 00:00:00'\)\] not found in level"
+    with pytest.raises(KeyError, match=msg):
+        mi.drop(pd.Timestamp("2001"), level="date")


Code review comments:
diff_hunk: @@ -2156,6 +2156,8 @@ def _drop_from_level(self, codes, level, errors="raise"):
         i = self._get_level_number(level)
         index = self.levels[i]
         values = index.get_indexer(codes)
+        nan_codes = isna(codes)
path: pandas/core/indexes/multi.py
text: @user1:
can't you just check `values==-1` ? as these are by definition nan.

@author:
Unfortunately not, values is ``-1`` when the value to drop was not found in the index **and** when the value to drop is ``NaN`` and was found in the index. So we have to check, if it is not ``NaN`` and setting it to something which is definitely not in ``self.codes`` before performing the isin.