Repository: pandas-dev/pandas

Issue:
subset keyword argument will include last column if incorrect keys are given
If you give a non-existent key to the subset argument it will default to the last column. For example:

```
In [3]: d = pd.DataFrame({'a':[1],'b':[2],'c':[np.nan]})
In [4]: len(d.dropna(subset=['x']))
Out[4]: 0
```

It seems the problem is in the line where dropna takes the subset:

```
agg_obj = self.take(ax.get_indexer_for(subset),axis=agg_axis)
```

Here, `ax.get_indexer_for(subset)` will return `-1` as a sentinel value for any keys that were not found in `subset`, and `self.take` interprets the `-1` as a request for the last column.



Discussion:
yep would call that a buggy....pull-request welcome!

Would it be best for `dropna` to raise a `KeyError` exception, like `d['x']` would in the example above?  or to return all of the rows, the thinking being that column `'x'` contains no nan's whatsoever (in fact it contains no values whatsoever, since `'x'` doesn't even exist).

no this is very easy to fix
just filter the -1 from the indexer before passing to take

Easy enough, but does this mean we don't support location based indices for the `subset` argument?  I mean, what if someone actually wrote `d.dropna(subset=[-1])` intending to drop the rows where the last column is `nan`?

only support labels - you have the whole complication the of having to have 2 methods to disambiguate 

and get indexer converts labels to locations

use your example (with subset = [-1]) as a test
that will KeyError I think 

Yes, I like just supporting column labels here.  I do note though, that when I do this

```
import pandas as pd
import numpy as np
print(pd.version.version)
d = pd.DataFrame({'a':[1,2],'b':[np.nan,3],'c':[4,np.nan]})
print(d.dropna(subset=[-1]))
```

I get this (i.e. it works as if it was dropping rows where the last were nan)

```
0.14.1-429-g0eb9023
   a   b  c
0  1 NaN  4
```

the indexer returns -1 (meaning it can't find the key) and take returns the last value

so you will need to filter out -1 or maybe if any -1 should raise
(if u have 1 good subset then should it work if another is not found)

we usually do this (eg in loc)

though I think in this case prob should just a raise of any columns are not found
as prob a user error

As far as raising an error goes, I was using subset specifically because I didn't care which of `['Sym', 'Symbol', 'SYMBOL']` it dropped on. 

It's a feature that could (and probably should) be reproduced by a more general function, but would be missed (by me at least) if it was removed without replacement.

It's worth noting `dropna` is currently inconsistent with `duplicated` (and therefore `drop_duplicates`), which raise a `KeyError`.

yes, I think this will increase consistency between `drop_duplicates/duplicated`,`drop` (which will raise a KeyError for an invalid column) and `dropna` (which currently does not)

@arijun 
@mcjcode 
pls see this related issue as well: https://github.com/pydata/pandas/pull/6599

@hayd what do you think?

I wouldn't mind doing this, but I've never developed for a large project and so might have problems with some aspects like
- VS's compiler seems to not work for building the c code in python packages
- I don't know the testing procedures here
- I don't really know the contribution workflow in general.

If you're still interested in having me contribute, can someone please point me to where I can get help with these sorts of issues.

oh, @mcjcode already has a soln (to this immediate problem). love to have contributions, see here for more info (and windows section on how to built for that). All of the above are covered. 

Start and issue and lmk when you need help!

https://github.com/pydata/pandas/wiki

Just to throw in an edge case: column label -1.

Subset arg is clear (we can ignore positions):

> Labels along other axis to consider,


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -2552,7 +2552,11 @@ def dropna(self, axis=0, how='any', thresh=None, subset=None,
             agg_obj = self
             if subset is not None:
                 ax = self._get_axis(agg_axis)
-                agg_obj = self.take(ax.get_indexer_for(subset),axis=agg_axis)
+                indices = ax.get_indexer_for(subset)
+                check = indices == -1
+                if check.any():
+                    raise KeyError(list(np.compress(check,subset)))
+                agg_obj = self.take(indices,axis=agg_axis)
 
             count = agg_obj.count(axis=agg_axis)
 


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -7200,6 +7200,8 @@ def test_dropna_corner(self):
         # bad input
         self.assertRaises(ValueError, self.frame.dropna, how='foo')
         self.assertRaises(TypeError, self.frame.dropna, how=None)
+        # non-existent column - 8303
+        self.assertRaises(KeyError, self.frame.dropna, subset=['A','X'])
 
     def test_dropna_multiple_axes(self):
         df = DataFrame([[1, np.nan, 2, 3],