Repository: pandas-dev/pandas

Issue:
BUG: Fix wrong column selection in drop_duplicates when duplicate column names
- [x] closes #17836
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Pandas 0.22.0 does not raise KeyError for misspelled column with .drop_duplicates()
So I have tested two versions of Pandas parallel to each other with exactly the same code. 0.19.2 behaves more as expected, but 0.22.0 does what I am about to describe. Will probably switch to 0.19.2 for now. Am using Python 3.6.4

```python
import pandas as pd
df = pd.DataFrame({"A":[34,12,78,84,26], "B":[54,87,35,81,87], "C":[56,78,0,14,13], "D":[0,87,72,87,14], "E":[78,12,31,0,34]}) 

print(df.drop_duplicates(['b','D']))
print(df.drop_duplicates(['B','D']))
print(df.drop_duplicates(['B']))
print(df.drop_duplicates(['D']))
```
#### Problem description
I became aware of the problem working with a much larger dataframe when it failed to warn me or raise a `KeyError` when I misspelled a column name. 

#### Expected Output

Pandas 0.19.2 gives you the following and but Pandas 22 gives you **no** `KeyError` for the first print statement it just runs.

```
KeyError: 'b'

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0
4  26  87  13  14  34

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
4  26  87  13  14  34
```


#### Output of ``pd.show_versions()``

Below is the output for the Pandas version for where the problem is. 

<details>


INSTALLED VERSIONS
------------------
commit: None

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.1
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None


</details>

Pandas 0.22.0 does not raise KeyError for misspelled column with .drop_duplicates()
So I have tested two versions of Pandas parallel to each other with exactly the same code. 0.19.2 behaves more as expected, but 0.22.0 does what I am about to describe. Will probably switch to 0.19.2 for now. Am using Python 3.6.4

```python
import pandas as pd
df = pd.DataFrame({"A":[34,12,78,84,26], "B":[54,87,35,81,87], "C":[56,78,0,14,13], "D":[0,87,72,87,14], "E":[78,12,31,0,34]}) 

print(df.drop_duplicates(['b','D']))
print(df.drop_duplicates(['B','D']))
print(df.drop_duplicates(['B']))
print(df.drop_duplicates(['D']))
```
#### Problem description
I became aware of the problem working with a much larger dataframe when it failed to warn me or raise a `KeyError` when I misspelled a column name. 

#### Expected Output

Pandas 0.19.2 gives you the following and but Pandas 22 gives you **no** `KeyError` for the first print statement it just runs.

```
KeyError: 'b'

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0
4  26  87  13  14  34

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
4  26  87  13  14  34
```


#### Output of ``pd.show_versions()``

Below is the output for the Pandas version for where the problem is. 

<details>


INSTALLED VERSIONS
------------------
commit: None

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.1
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None


</details>



Discussion:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=h1) Report
> Merging [#17879](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/aed9b921394efb13ed1b7870508e8638db951101?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17879/graphs/tree.svg?height=150&token=eZ4WkYLtcO&width=650&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17879      +/-   ##
==========================================
- Coverage   91.23%   91.21%   -0.02%     
==========================================
  Files         163      163              
  Lines       50102    50102              
==========================================
- Hits        45712    45703       -9     
- Misses       4390     4399       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.03% <100%> (ø)` | :arrow_up: |
| #single | `40.31% <0%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <100%> (-0.1%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=footer). Last update [aed9b92...41dc08f](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=h1) Report
> Merging [#17879](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/aed9b921394efb13ed1b7870508e8638db951101?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17879/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17879      +/-   ##
==========================================
- Coverage   91.23%   91.21%   -0.02%     
==========================================
  Files         163      163              
  Lines       50102    50102              
==========================================
- Hits        45712    45703       -9     
- Misses       4390     4399       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.03% <100%> (ø)` | :arrow_up: |
| #single | `40.31% <0%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <100%> (-0.1%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=footer). Last update [aed9b92...41dc08f](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=h1) Report
> Merging [#17879](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/aed9b921394efb13ed1b7870508e8638db951101?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17879/graphs/tree.svg?width=650&height=150&src=pr&token=eZ4WkYLtcO)](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17879      +/-   ##
==========================================
- Coverage   91.23%   91.21%   -0.02%     
==========================================
  Files         163      163              
  Lines       50102    50102              
==========================================
- Hits        45712    45703       -9     
- Misses       4390     4399       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.03% <100%> (ø)` | :arrow_up: |
| #single | `40.31% <0%> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <100%> (-0.1%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=footer). Last update [aed9b92...30c7e1b](https://codecov.io/gh/pandas-dev/pandas/pull/17879?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can u add a what’s new note for 0.21.0
@jreback Added what's new note.
thanks @Licht-T keep em coming!
Thanks for the report. The root cause looks like it's in `DataFrame.duplicated`.

Interested in submitting a fix?
Whoever fixes this may be able to knock out https://github.com/pandas-dev/pandas/issues/12869 at the same time.
@TomAugspurger I might have a crack, see if someone beats me to it. Had a quick look with PyCharm, have not managed to quickly locate that variable / class, it must be in there somewhere..
Great! I think the offending lines are

https://github.com/pandas-dev/pandas/blob/2fdf1e256e5e0b7f1fe909629e2f0b7893c8c7c3/pandas/core/frame.py#L3658-L3660

Making some bad assumptions about `subset`. LMK if you have any troubles.
I've got a working fix. Mind if I submit the pull request? Or did you want to take a shot at it, @aktivkohle?
@NoahTheDuke Yes do it, thanks for asking.. I'm sure I'll get my chance one day, and would probably take hours to work it out on the weekend which is not meant for that kind of thing. Will have a look at how you did it of course..
Thanks for the report. The root cause looks like it's in `DataFrame.duplicated`.

Interested in submitting a fix?
Whoever fixes this may be able to knock out https://github.com/pandas-dev/pandas/issues/12869 at the same time.
@TomAugspurger I might have a crack, see if someone beats me to it. Had a quick look with PyCharm, have not managed to quickly locate that variable / class, it must be in there somewhere..
Great! I think the offending lines are

https://github.com/pandas-dev/pandas/blob/2fdf1e256e5e0b7f1fe909629e2f0b7893c8c7c3/pandas/core/frame.py#L3658-L3660

Making some bad assumptions about `subset`. LMK if you have any troubles.
I've got a working fix. Mind if I submit the pull request? Or did you want to take a shot at it, @aktivkohle?
@NoahTheDuke Yes do it, thanks for asking.. I'm sure I'll get my chance one day, and would probably take hours to work it out on the weekend which is not meant for that kind of thing. Will have a look at how you did it of course..

Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3655,6 +3655,13 @@ def f(vals):
               isinstance(subset, tuple) and subset in self.columns):
             subset = subset,
 
+        # Verify all columns in subset exist in the queried dataframe
+        # Otherwise, raise a KeyError, same as if you try to __getitem__ with a
+        # key that doesn't exist.
+        diff = Index(subset).difference(self.columns)
+        if not diff.empty:
+            raise KeyError(diff)
+
         vals = (col.values for name, col in self.iteritems()
                 if name in subset)
         labels, shape = map(list, zip(*map(f, vals)))


Test patch:
diff --git a/pandas/tests/frame/test_analytics.py b/pandas/tests/frame/test_analytics.py
--- a/pandas/tests/frame/test_analytics.py
+++ b/pandas/tests/frame/test_analytics.py
@@ -1492,6 +1492,19 @@ def test_drop_duplicates(self):
         for keep in ['first', 'last', False]:
             assert df.duplicated(keep=keep).sum() == 0
 
+    @pytest.mark.parametrize('subset', ['a', ['a'], ['a', 'B']])
+    def test_duplicated_with_misspelled_column_name(self, subset):
+        # GH 19730
+        df = pd.DataFrame({'A': [0, 0, 1],
+                           'B': [0, 0, 1],
+                           'C': [0, 0, 1]})
+
+        with pytest.raises(KeyError):
+            df.duplicated(subset)
+
+        with pytest.raises(KeyError):
+            df.drop_duplicates(subset)
+
     def test_drop_duplicates_with_duplicate_column_names(self):
         # GH17836
         df = DataFrame([