CLN/API refactor drop and filter and deprecate select
State: closed

closes #4818
- need to unify api for drop and filter
- should they take args for regex (regex, flags, case)
- should we keep select or filter?

closes #6189
xref #8530 
xref #8594


Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
Also, failing tests is to convert to string, basically before doing regex I do index.astype(str), if you have non-ascii this fails.

Also should regex be match or contains... I'm confused about difference between like/regex. (If it's contains can wrap in `^foo$` to make match.

I think if not specified should be like startswith, e.g. `^`,

most often you want give me all the columns which start with `foo`

so `match`

@jreback should there be a way to distinguish between match and contains? Previously regex arg acted as contains and like arg as match (which is startswith).

Was thinking of using regex arg, could hack it to take 'match' or 'contains' (match by default)... :s or maybe regex=None use match, regex=True use contains, regex=False don't use regex ?

yes that makes sense
- default is `match`
- `False` is an exact 
- `contains` uses `contains` rather than `match`

maybe catch this: https://github.com/pydata/pandas/issues/5798 (deal with non-strings, e.g .0)
#5657 (I think this PR will close this - perf of filter)

@jseabold what do you think of this so far? (I think we should put this on the mailng-list when Andy is ready)....

atm I'm just False-ing numbers (rather than string-ifying them.. I tried to do that but got ascii errors, so not sure it's worth it, easy to do though). It doesn't raise now (it use string methods, although unfortunately these NaN for numbers so have to fill in after).

Will check the perf.

I can live with exact string args, gives more flexibility later.

I'm pretty much on board AFAICT. I haven't followed too closely. I'd expect `re.search` rather than `re.match` though just becuse it's the more general one. I don't feel strongly about this though. It's not unheard of to want to match on the end of a variable name. `first_educ`, `second_educ`, etc. Something like that. If user wants match behavior it's easier to add that then to get around it. `^.+` vs `^` (or whatever).

@jseabold you can always provide a regex if you want to (or alternately can have `regex='match|contains|search'` maybe overkill though

Well then I'm a bit confused about what exactly this is doing.

@jreback perf is an issue, for two reasons:
1. str methods are slower than == (although, if regex is False we can just use ==) and isin (esp. for short lists). Not sure how we can speed this up?
2. I'm boolean masking using `_slice`, which is **slow** compared to iloc (can I call loc on a generic axis / or is there a better way completely) ?

@jseabold This PR makes the API the same for filter and drop, and do the same thing under the hood. You can pass drop a regex, a boolean function (to apply to index) or a list of labels to drop. 

hmmmm, unfortunately regex taking string for match/contains screws up backwards compat. Another arg?

alternative soln is to use select over filter (and dep filter instead), but not the best reason for doing so...

@hayd no I think you should always uses a compiled regex (or isin/== if you can)...that's the issue

I think you can just compute the indexer and do an iloc it should work, just construct the tuple

say `indexer` is your axis indexer

```
tuple_indexer = [ slice(None) ] * self.ndim
tuple_indexer[axis] = indexer
return self.iloc[tuple(tuple_indexer]]
```

@hayd what do you mean by:

> > > hmmmm, unfortunately regex taking string for match/contains screws up backwards compat. Another arg?

Previously regex was an kwarg for filter. In my code I was doing:

```
if isinstance(regex, string_types): return self.filter(labels=regex, regex=True, axis=axis)
```

if regex is allowed to be a string (e.g. 'match') I can't do that, and of course this will break old code if they did happen to be filtering by the regex 'match' (unlikely!)

@hayd I believe this should be true

```
indexer = boolean indexer on axis a
df.filter(indexer,axis=a) == df.drop(df._get_axis(a)[~indexer],axis=a)
```

(and I think that that drop should be implemented in terms of filter)

@jreback yup, that's exactly what the `_select` method does.

Was thinking about this, and not sure regex should be name of flag to decide _how_ regex is applied, as you may want to do contains _without_ using regex. If that makes sense...

hmm regex is what replace uses

replace doesn't offer a way to tweak the way regex works (i.e. match vs contains), you either use regex or you don't. IMO regex should be bool (a la everywhere else) and have another argument to distinguish between contains vs match vs whatever ?

```
In [20]: s = pd.Series(['a', 'ab', 'ca'])

In [15]: s.replace(to_replace='a', value='d')
Out[15]:
0     d
1    ab
2    ca
dtype: object

In [16]: s.replace(regex='a', value='d')
Out[16]:
0     d
1    db
2    cd
dtype: object

In [17]: s.replace(to_replace='a', regex=True, value='d')
Out[17]:
0     d
1    db
2    cd
dtype: object
```

I'm still just not sure I understand the issue here. It seems to me (and tell me to shut up, if you get my point already, or this is impossible for backwards compat reasons) that regex should everywhere be search. match is just a way to avoid writing '^'. But since you can just write '^', you shouldn't bend over backwards to provide some weird combinations of keywords that do match and some that do search. AFAICT, you can do everything and more with search (via re.MULTILINE flag) than you can do with match. I always thought it was a mistake (and accident of history) that match was targeted by the old regex functions. Make a one-line note in the docs "if you want to do `match-like` behavior, then prepend '^'". If people know what match vs contains is enough for the keywords to make sense, they probably already know about '^' right?

```
[~/]                                                                            
[5]: s.replace(to_replace='^a', regex=True, value='d')                          
[5]:                                                                            
0     d                                                                         
1    db                                                                         
2    ca                                                                         
dtype: object 
```

*old regex functions are those vectorized .str methods that Wes added in an afternoon.

Yeah, I think contains (aka search?) should be the default,  you can always prepend ^ if you so wish, I'll add it as notes in the docstring (and for legacy like kwarg will still do that). regex being bool makes it easier.

I guess the issue is: that will make it a bigger API change... fixed by making regex=False the default.

I'm never going to be a big fan of any API where what one keyword is affects what another does. I don't think there should be any flag regex True/False. I'm still not understanding the problem that needs to be solved.

the issue is should a passed in labels
be treated as an exact match or a regex

eg

df.filter('A')

if the columns are

['A', 'AB']

I think the default has to be an exact match

Yeah, I think the current behavior is exact match. That's the status quo, and it'll probably have to stay that way right? I always just use the regex keyword anyway, because there's not much sense in an exact match given that I can slice on columns. While unfortunate that it is what it is right now, I'm -1 on having a keyword that controls what labels does. Maybe after a deprecation cycle on `labels`, we can switch to `regex.seach` for labels? I doubt there are too many people that rely on using filter('A') that a deprecation cycle would be too painful. I dunno though.

Ok, think I'm in favour of having regex bool but allowing a regex string if no labels are passed (and regex=False by default). This mirrors behaviour elsewhere in pandas... ?

I need to add some tests to this branch for both filter and drop, and some warnings.

There's a pretty long-standing unwritten rule at least in numpy/scipy stack not to have keywords that change the meaning of other keywords. I'd be interested to see any exceptions to this. I recall only one and it was a new_behavior keyword (for correlation maybe?) that went through a long and loud deprecation cycle and is now gone.

hmm...this sort of goes back tot he orgiinal, but what if we did

`df.filter(labels=None,.....,regex=None)`

labels are function, list-like (or a scalar which gets converted to a list) that exact match
other wise just specify a regex using regex)

then don't really need to change much, this would then align filter/drop I think (still deprecate select)