Repository: pandas-dev/pandas
Pull request: BUG: drop_duplicates not raising KeyError on missing key
Fix #17879 introduced an error by iterating over the columns in the dataframe, not the columns in the subset. This meant that passing in a column name missing from the dataframe would no longer raise a `KeyError` like it had previously.

This fix checks the subset first before pulling necessary columns from the dataframe, and raises the necessary `KeyError` when a given column doesn't exist.

- [x] closes #19726 
- [x] tests added / passed
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [x] whatsnew entry

Problem statement:
Pandas 0.22.0 does not raise KeyError for misspelled column with .drop_duplicates()
So I have tested two versions of Pandas parallel to each other with exactly the same code. 0.19.2 behaves more as expected, but 0.22.0 does what I am about to describe. Will probably switch to 0.19.2 for now. Am using Python 3.6.4

```python
import pandas as pd
df = pd.DataFrame({"A":[34,12,78,84,26], "B":[54,87,35,81,87], "C":[56,78,0,14,13], "D":[0,87,72,87,14], "E":[78,12,31,0,34]}) 

print(df.drop_duplicates(['b','D']))
print(df.drop_duplicates(['B','D']))
print(df.drop_duplicates(['B']))
print(df.drop_duplicates(['D']))
```
#### Problem description
I became aware of the problem working with a much larger dataframe when it failed to warn me or raise a `KeyError` when I misspelled a column name. 

#### Expected Output

Pandas 0.19.2 gives you the following and but Pandas 22 gives you **no** `KeyError` for the first print statement it just runs.

```
KeyError: 'b'

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0
4  26  87  13  14  34

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
4  26  87  13  14  34
```


#### Output of ``pd.show_versions()``

Below is the output for the Pandas version for where the problem is. 

<details>


INSTALLED VERSIONS
------------------
commit: None

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.1
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None


</details>

Hints from issue discussion:
Thanks for the report. The root cause looks like it's in `DataFrame.duplicated`.

Interested in submitting a fix?
Whoever fixes this may be able to knock out https://github.com/pandas-dev/pandas/issues/12869 at the same time.
@TomAugspurger I might have a crack, see if someone beats me to it. Had a quick look with PyCharm, have not managed to quickly locate that variable / class, it must be in there somewhere..
Great! I think the offending lines are

https://github.com/pandas-dev/pandas/blob/2fdf1e256e5e0b7f1fe909629e2f0b7893c8c7c3/pandas/core/frame.py#L3658-L3660

Making some bad assumptions about `subset`. LMK if you have any troubles.
I've got a working fix. Mind if I submit the pull request? Or did you want to take a shot at it, @aktivkohle?
@NoahTheDuke Yes do it, thanks for asking.. I'm sure I'll get my chance one day, and would probably take hours to work it out on the weekend which is not meant for that kind of thing. Will have a look at how you did it of course..

Commit message: Adds changes to whatsnew

Patch to review:
diff --git a/doc/source/whatsnew/v0.23.0.txt b/doc/source/whatsnew/v0.23.0.txt
index a2198d9103528..a4924417ff2b7 100644
--- a/doc/source/whatsnew/v0.23.0.txt
+++ b/doc/source/whatsnew/v0.23.0.txt
@@ -767,6 +767,8 @@ Indexing
 - Bug in :class:`IntervalIndex` where empty and purely NA data was constructed inconsistently depending on the construction method (:issue:`18421`)
 - Bug in :func:`IntervalIndex.symmetric_difference` where the symmetric difference with a non-``IntervalIndex`` did not raise (:issue:`18475`)
 - Bug in :class:`IntervalIndex` where set operations that returned an empty ``IntervalIndex`` had the wrong dtype (:issue:`19101`)
+- Bug in :meth:`DataFrame.drop_duplicates` where no ``KeyError`` is raised when passing in columns that don't exist on the ``DataFrame`` (issue:`19726`)
+
 
 MultiIndex
 ^^^^^^^^^^
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
index a001037b573d4..d42106c785fed 100644
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3655,6 +3655,10 @@ def f(vals):
               isinstance(subset, tuple) and subset in self.columns):
             subset = subset,
 
+        for name in subset:
+            if name not in self.columns:
+                raise KeyError(name)
+
         vals = (col.values for name, col in self.iteritems()
                 if name in subset)
         labels, shape = map(list, zip(*map(f, vals)))
diff --git a/pandas/tests/frame/test_analytics.py b/pandas/tests/frame/test_analytics.py
index b9275fc69e7ff..5b670eb0e5009 100644
--- a/pandas/tests/frame/test_analytics.py
+++ b/pandas/tests/frame/test_analytics.py
@@ -1492,6 +1492,15 @@ def test_drop_duplicates(self):
         for keep in ['first', 'last', False]:
             assert df.duplicated(keep=keep).sum() == 0
 
+    def test_drop_duplicates_with_misspelled_column_name(self):
+        # GH 18XXX
+        df = pd.DataFrame({'A': [0, 1, 2, 3, 4, 5],
+                           'B': [0, 1, 2, 3, 4, 5],
+                           'C': [0, 1, 2, 3, 4, 6]})
+
+        with pytest.raises(KeyError):
+            df.drop_duplicates(['a', 'B'])
+
     def test_drop_duplicates_with_duplicate_column_names(self):
         # GH17836
         df = DataFrame([


Code review comments:
diff_hunk: @@ -3655,6 +3655,10 @@ def f(vals):
               isinstance(subset, tuple) and subset in self.columns):
             subset = subset,
 
+        for name in subset:
path: pandas/core/frame.py
text: @user1:
can you add a comment here on what you are checking

@user1:
you can do this
```
diff = pd.Index(subset).difference(self.columns)
if len(diff):
     raise KeyError(diff)
```