ENH: Allow where/mask/Indexers to accept callable
State: closed

- [x] closes #12533
- [x] closes #11485 
- [x] tests added / passed
- [x] passes `git diff upstream/master | flake8 --diff`
- [x] whatsnew entry

Current impl is very simple like `assign`. Another idea is applying some restriction to return value from `callable` (for example, only allow aligned `DataFrame` or scalar).


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
One potential problem is when you subclass `DataFrame` and make it callable.

Do you know such a package? The logic is not specific here and I don't think we have to care ATM.

I don't know anyone that uses it that way.  But since the order of consideration is relevant, I think it should be documented that the args are first tested for callability.

let's simultaneously do all of the chained reshaping ops #11485 in 0.18.1, making them consistent.

Sure, once done and changed the title.

## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/12539?src=pr) is **83.91%**

> Merging [#12539](https://codecov.io/gh/pydata/pandas/pull/12539?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will increase coverage by **+<.01%**

``` diff
@@             master     #12539   diff @@
==========================================
  Files           136        136          
  Lines         49908      49933    +25   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          41876      41901    +25   
  Misses         8032       8032          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last updated by 3921b32

@jreback Thanks, fixed the points.

@shoyer @jorisvandenbossche @TomAugspurger 

any comments on the idea/impl?

If we're adding this to indexing methods:
1. Why not add this to `[]` too, for consistency? `df[lambda x: x.A == 'B']` beats `df.loc[lambda x: x.A == 'B']` 
2. What's the point of adding to the `.query` when we can get the exactly same functionality with indexing?

I agree with 1)

2) for consistency.

@jreback this PR overloads indexing with _yet another_ method. That's kind of OK because indexing already does a lot, and this will be quite useful. On the other hand, if `df[lambda x: ...]` works exactly the same as `df.query(lambda x: ...)`, I don't see the point in overloading `query`, too, unless we are going to systematically add this to almost every possible dataframe method.

`.groupby` is another natural method to which to add this support.

Finally, if anyone is wondering whether this is possible without verbosely adding the `df` argument... why yes it is. If you use [this hack](https://github.com/shoyer/pandas-magic) that I wrote last time this came up, you can write `df[lambda: A == 'B']` instead of `df[lambda x: x.A == 'B']`. The [code that does this](https://github.com/shoyer/pandas-magic/blob/master/pandas_magic/__init__.py#L13-L33) is actually remarkably hygienic. We could even restrict it to only working on lambdas by verifying that `f.func_name == '<lambda>'`.

There are probably good reasons why Python discourages this much magic, but it actually seems pretty safe to me, aside from breaking auto-complete/lint. Maybe we could make it optional. On the other hand, if you prefer the magical `X` notation of pandas-ply (even by a third party), that could be straightforwardly hooked into either system once you accept callables.

@shoyer 

`.groupby` already has all of this

`df.groupby(lambda x: .....)`

I don't see why as a user you have to _know_ that a particular indexing method accepts a callable or not. That seems very weird. yes, `.query` is just another method of indexing, BUT, it a) cannot be assigned to (like `.sel` in xarray). and it dispatches to things like `numexpr`.

one could argue that `.loc` should accept a string like `.query` actually (but that introduces some ambiguity on what the user is doing).

Added `__getitem__` impl.

@shoyer I kind of like your 'hack' (note that 'hack' -> elegant if its cool enough :)

want to do a PR to add (after this one)?

I still feel that adding this to `query` breaks the rule for how these callables work. Generally, they work by remapping `first_arg -> first_arg(self)`. But the way this has been added to query breaks the rule: `df.query(lambda x: x > 5)` is not mapped to `df.query(df.x > 5)` (which would error) but rather `df.loc[df.x > 5]`.

@jreback Yeah, I can put together a PR for adding argument-less lambdas together after we add this. Restricting it to lambdas (at least `func.name == '<lambda>'`) actually makes it pretty clean, because you are prohibited from writing larger functions with no-arguments that could be defined anywhere.

+1 on idea and the implementation looks good.

I'm with @shoyer that extending `.query` is a bit strange. It already seems different enough from the other indexing methods, that I'm not sure it would be strangely inconsistent if it couldn't take a callable. I don't feel strongly about this though.

What are the thoughts on setting with this syntax? Right now

``` python

In [13]: df.loc[lambda x: x.b > 0] = 10

In [14]: df
Out[14]:
                                            a          b
0                                    2.252602   0.545462
1                                    1.984207  -2.059620
2                                   -0.392923   1.810029
3                                    0.038127  -2.738341
4                                   -0.299452   1.730878
...                                       ...        ...
6                                    0.164651   1.242235
7                                    0.801972  -0.196810
8                                    0.367329  -0.486618
9                                   -0.928201  -0.120228
<function <lambda> at 0x115266d08>  10.000000  10.000000
```

Should this raise? At the very least we should probably document that users probably don't want to do that.

@TomAugspurger thx, good catch. Fixed and added some tests. Because it can't be used in chainnings, raising an error is alternative idea.Thoughts?

this lgtm. comments

@jorisvandenbossche @TomAugspurger @shoyer 

any comments?

I will take a look in 2-3 hours

@sinhrks Some more doc comments:
- Can you also update the docstrings of `ix/loc/iloc`
- Is there documentation on where/query that needs to be updated (apart from the docstrings, which you did update)?

Further, I have to say I share the reservations about adding this to `query`. Currently, that is a specific function that has a very different way of usage compared to the indexing methods (i.e. a string that is evaluated). I don't really see the advantage of being able to use query with a callable over just using one of the indexer methods with a callable

For the rest, very nice addition!

ok, how about for `.query()` we than raise if the input is NOT a string with a nice mesage (to use `.loc`)? or could show a warning and just do it anyhow.

You already get an error when passing a callable to `query`, but I agree a nicer message can be raised.

I would definitely raise if the input to `.query()` is not a string. Adding warnings and then doing something anyways is a bad idea.

ok @sinhrks if you can make those mods would be great. (you _might_ need to adjust docs/example as I think you use .query, but can just change to `.loc`)

Thanks for all the feedbacks. Updated. Lmk if I missed something.

just minor change otherwise lgtm.

@jorisvandenbossche @shoyer 

@sinhrks Can you also update the docstrings for loc/iloc/ix (eg for loc https://github.com/pydata/pandas/blob/65ed3afd092908130905e7d86ff1b6b9c2002d00/pandas/core/indexing.py#L1314), these are used in the API docs.

For the rest, +1!

@sinhrks some very minor doc-changes. ping when pushed.

thanks @sinhrks !

awesome enhancement!