DataFrameGroupby.boxplot fails when subplots=False
State: closed

- [x] closes #16748
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

![Screen Shot 2020-10-13 at 8 06 48 PM](https://user-images.githubusercontent.com/9269816/95898874-037e6980-0d90-11eb-85b3-e00fe36611b9.png)


Labels:
name: Groupby
name: Visualization
description: plotting

Comments:
is there a flaky test? 🤔 a lot other PRs passed tests though
would appreciate a lot if someone could give some hint of why the failure happens at `test_converter.py`.
Would appreciate a lot if you could give any hint on how `test_converter` could fail the tests? @TomAugspurger  i tried to set it to `xfail` to avoid failures
Just to make sure I understand, is there any API change here for an end-user? Is this code called indirectly via `DataFrame.boxplot`? I want to make sure there are no changes for that, when the user is providing a MultiIndexed dataframe and `columns`.
You are absolutely right, this indeed caused a change on MultiIndexed dataframe and columns when using `DataFrame.boxplot`. But I test again and somehow feel it might be good change to end users. Since right now, for MultiIndexed dataframe, if you have a dataset like:

![Screen Shot 2019-09-01 at 8 18 16 PM](https://user-images.githubusercontent.com/9269816/64080521-a80d2b80-ccf5-11e9-941b-613825d24016.png)

WHen you do `df.boxplot(column='two'), then an error will be raised:

![Screen Shot 2019-09-01 at 8 16 23 PM](https://user-images.githubusercontent.com/9269816/64080524-bc512880-ccf5-11e9-8ce4-aa97acb365de.png)

But with the new change, this can be plotted, and I feel this change is more useful and users might like ti.

In the meanwhile, due to this, i found out actually `DataFrame.boxplot` with multiindex Dataframe are not tested at all at pandas, otherwise, this should raise an error. So I also add another test for this case, the test is exactly the same as I showed in the picture below.

Thanks again for your careful and thorough thought on the potential consequences, I am looking forward to further reviews. @TomAugspurger 

![Screen Shot 2019-08-31 at 10 11 07 PM](https://user-images.githubusercontent.com/9269816/64068751-cc590180-cc3c-11e9-9456-f4c85ef0f1a7.png)

![Screen Shot 2019-08-31 at 10 11 18 PM](https://user-images.githubusercontent.com/9269816/64068753-d11db580-cc3c-11e9-8410-0414c5be9f50.png)

> But with the new change, this can be plotted, and I feel this change is more useful and users might like ti.

The alternative would be for the user to provide something like `columns=pd.IndexSlice[:, 'two']`, correct?

Just to make sure, there's no change to the case where the user-provided `columns` is from the outer level like `bar`?

My main concern is that by automatically doing this, we'd introduce some kind of ambiguity about what `columns` means.
Sorry for my late response, was out sick for a couple days. 
I read your comments again and think I fully get your point. Indeed, this will also change the current behavior from the outer level like `bar`. Right now, if doing `df.boxplot(columns='bar')`, will get:

<img width="498" alt="Screen Shot 2019-09-05 at 8 35 44 PM" src="https://user-images.githubusercontent.com/9269816/64369615-02232f00-d01d-11e9-84d6-2f0e58148c9c.png">

You mean we should keep this behavior, right? @TomAugspurger 

But i might have to say behaviour of `df.boxplot(columns='two')` will change in this case. Before the change of this PR, this would raise a `KeyError` of `['two'] not in index`. And now it plots columns containing `two` because of this `pd.IndexSlice`.

And I also have a second thought, and think we might provide another argument for boxplot, which is something like `include` (could be changed for something else) for multiindex columns, if `True`, then users could define more ambiguous columns to get their columns plotted, so `columns='bar'` will plot all columns as long as `'bar'` is in any multi-level index. And for normal Index, just set to `False` in the function no matter what, since this won't affect normal index at all.
@charlesdong1991 can you merge master; can you summarize the issues here that were raised?
can you merge master
The original issue was, in the case such as:
```python
df = pd.DataFrame({'cat':np.random.choice(list('abcde'), 100), 
                    'v':np.random.rand(100), 
                    'v1':np.random.rand(100)})
df.groupby('cat').boxplot(subplots=False, column='v')
```
Since after groupby, this `v` will not be in the column index, so will raise an error. 

But with `pd.IndexSlice[:, column]`, seems issue could be solved for `groupby.boxplot`. However, @TomAugspurger brought up a very important point that this will however have an endpoint change for normal `df.boxplot` if specified column comes from the first level in column index (for MultiIndex case).

I just updated my previous solution, and now it won't afffect endpoint change for `df.boxplot` on the first level of MultiIndex. However, when it comes to the second level, behaviour will change. Since my description might cause confusion, I will present using several graphs, so pls bear with me, the df is:
```python
arrays = [['bar', 'bar', 'baz', 'baz', 'foo', 'foo', 'qux', 'qux'],
            ['one', 'two', 'one', 'two', 'one', 'two', 'one', 'two']]
tuples = list(zip(*arrays))
index = pd.MultiIndex.from_tuples(tuples, names=['first', 'second'])
df = pd.DataFrame(np.random.randn(3, 8), index=['A', 'B', 'C'], columns=index)
```
Current behaviour in Pandas, it will raise an error on second level:
![Screen Shot 2019-10-19 at 8 21 03 PM](https://user-images.githubusercontent.com/9269816/67149654-1d38be00-f2ae-11e9-8cf0-e547cdcc89a5.png)

With this PR:
![Screen Shot 2019-10-19 at 8 21 43 PM](https://user-images.githubusercontent.com/9269816/67149659-29bd1680-f2ae-11e9-96cb-e8f6f0ede8e5.png)

As you could see above, the second level end behaviour will change, the rest will remain the same. And the original issue for `groupby.boxplot` can still be fixed. @jreback 
can you merge master and we'll look again
Merged! @jreback 

Looks good, except for a doc error raised by `Cython` due to new release of `ipython`.
thanks a lot for picking it up @WillAyd  Yeah, I am active, so is this PR!  xD
I think there is a design issue with this PR, and now the priority is regression and new issues posted for 1.0

I will get back to your comments later next week, since I also came across an issue which requires a feature for plotting based on the Index level, might be relevant to this PR, I will try to get that feature done and then back to address and try to push it through
Can you move the whatsnew?
@charlesdong1991 what's the status here?
yeah, i will give this PR another try later this week @simonjayhawkins 
whats the status here?
@charlesdong1991 gentle ping can you rebase
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
hey @jreback @WillAyd @simonjayhawkins @TomAugspurger 

sorry for very late update here. I just took a look at the original issue and my past commits together with the relevant codebase, I feel it was quite inefficient and confusing. So here, I flipped most of the changes and I _think_ now the new solution looks much better.

So the issue was with a df like below, it will fail doing `df.groupby.boxplot(subplots=False, column='v')`:
```python
df = pd.DataFrame({'cat':np.random.choice(list('abcde'), 100), 
                    'v':np.random.rand(100), 
                    'v1':np.random.rand(100)})
df.groupby('cat').boxplot(subplots=False, column='v')
```
and it is because the data for plotting after `df.groupby('cat')` has changed to MI, so `v` no longer exists in the transformed data.

Therefore, the new solution for this is quite simple, I couple the keys of groupby values (in this case, [a, b, c, d, e]) with the column value sellers assign to `column` argument (in this case, `v`), so we have `[(a, v), (b,v), (c, v), (d, v), (e, v)]` and assign them to `boxplot` function, and then `boxplot` function will look for subset based on this new `column` values, instead of `v` in the original df which is used by sellers.

How does it sound now? Feedbacks and reviews are very welcomed!

~Regarding the error, It seems irrelevant to this PR.~
EDIT:
CI failure is gone now after commiting one more time

@charlesdong1991 this looks good, can you post an updated plot (and your tests look nice); merge master and ping on green-ish..
![Screen Shot 2020-10-13 at 8 06 48 PM](https://user-images.githubusercontent.com/9269816/95898874-037e6980-0d90-11eb-85b3-e00fe36611b9.png)

sure, these are plots with and without specifying `column`, I also update in the PR description on top.
@charlesdong1991 can you merge master
small ping here @jreback sorry for the late and long response
thanks @charlesdong1991 very nice!