API: Return axes from boxplot
State: closed

Closes: https://github.com/pydata/pandas/issues/4264
Will go in place of https://github.com/pydata/pandas/pull/4472

In https://github.com/pydata/pandas/issues/985 we intentionally decided to have boxplot return a dict with the Lines of the boxplot.

I've added a kwarg `return_dict`. By default, `return_dict` is False. In this case only a `matplotlib.axes` is returned. When `return_dict` is True, we should either
- return the `matplotlib.axes` **and** the dict in a namedtuple
- return just the dict

Both of these are API breaking. Right now I've got both in the named tuple, but I may change that to just return just the dict. If we just return the dict, the only change people will have to make to their code is to add `return_dict` to True in their boxplot calls. Thoughts?

cc @jseabold, @fonnesbeck


Labels:
name: Visualization
description: plotting
name: API Design

Comments:
FYI I don't see any uses of `boxplot` in Wes's book.

i think should return the dict by default ... unless there's a reason to break the API more than necessary (i think ok to break by returning the named tuple) but defaulting to not return the boxplot anymore seems too much ... not that big of a deal

@TomAugspurger I think your soln seems reasonable.

@cpcloud I think returning BOTH breaks much more here; with @TomAugspurger PR, you _can_ get both just by adding a kw, otherwise it doesn't break

@jseabold ?

@TomAugspurger ?

Shall we have a vote?
1. Return `dict` only (no change)
2. Default: Return `ax` and `dict` in named tuple (API breaking)
3. Default: return `dict` only, add kwarg `return_ax` (`True` or `False`)
4. Default: return `dict` only, add kwarg `return_ax` (`True`, `False`, or `'both'`)
5. Default: return `ax` only, add kwarg `return_dict` (`True` or `False`) (API breaking)
6. Default: return `ax` only, add kwarg `return_dict` (`True`, `False`, or `'both'`) (API breaking)

2, 5, and 6 will break peoples' code.
3 & 4 are similar, just whether to take True/False or True/False/both; likewise with 5 & 6.

I vote for 5 or 6.

I should note that this only applies when `by` is None. when `by` is not None, there's no change and an array of `axes` will be returned.

@jseabold ?

5 gets my vote also, FWIW

:+1:  on 5 from me too.

@TomAugspurger since current a dict is returned, and proposal is for 5 returning an `ax` will be returned instead (with `return_dict` kw), this _should_ raise in user code if you are attempting to use this, right? IOW the break in the API will make it an obvious error

just thinking that user upgrades and doesn't read the API changes in whatsnew (doesn't everyone??? ) ... lol

5 is ok I suppose. I will only offer as anecdotal evidence that whenever I update my pandas, parts of my large projects inevitably break and it's a huge time-sink to track things down (happened as recently as yesterday). Breaking code is "bad." I've been busy lately, so I've just stopped trying to report these types of things. OTOH, this is a bit of a wart, and it should be pretty clear what's going on. If you decide on 5, then I'd issue a warning for a few releases to help people figure out what's going on.

Another option, that's a pain, but is the way we went with statsmodels in a few cases. Keep returning a dict, so as not to break people's code and add `return_ax` keyword that is `False` by default. Using inspect, if return_ax is not explicitly specified, then issue a deprecation warning that in the next release (or whenever) boxplot will return a tuple (or whatever you decide to move to). If it is explicitly given, then assume the user knows what they're doing. 

Thanks @jseabold. Should we try to deprecate / warn for a release?

I could add a `return_type` keyword, which expects one of `{'dict', 'axes', 'both'}`. In .14 the default is `'dict'` and will behave exactly as .13, but it will warn that in a future version the default will change to `axes`. They can future proof themselves now by setting `return_type='axes'`. Thoughts?

That sounds reasonable to me, but I'll defer to whatever others want to do. API breakage isn't the end of the world in this case. As has been pointed out this isn't exactly a subtle change. IMO deprecation is just a good software habit for such a heavily used package.

I am +1 on (eventually) returning the axis by default. 
I also think you should have the possibility to return both (so not only or axes or dict, in the case you want to use both to customize your plot).

And +1 on having a deprecation cycle, keeping the default for now but raising a warning this will change in the future. `return_type` is good for me. Are there precedents of keywords that determine the return type? (to see what name we used there?)

I am with @jorisvandenbossche on first point `return_type`

but +0 on deprecation cycle, I think this would break loudly and this has been a wart for a long time

so since majority want deprecation @TomAugspurger why don't we use the `return_type` and deprecate for 0.14, chaning in 0.15 then 

OK. So to summarize:
- This PR adds a new kwarg: `return_type` (tentative name; pending investigation if we have any other similar cases).
- `return_type` can be `'dict'` (0.14 and earlier behavior), `'axes'` (0.15+ behavior) or (`'both'` namedtuple)
- In 0.14 the default is `return_type=None` so that I can detect if a warning needs to be raised.
  - If `return_type` is None, a FutureWarning is issued and a dict is returned
  - If `return_type` is `'dict'`, a warning is _not_ raised
- In the Future (0.15?) the default of `return_type` will be changed to `'axes'`

And I should note that all of this goes out the window if `'by'` is not None, in which case we return an array of `axes`. I suspect some documentation is in order.

I pretty much have this done btw. I'm just in the midst of a messy rebase.

@TomAugspurger sounds good to me; create a new issue for DEPR review in 0.15 (the other is for 0.16); just because this has been outstanding for SO long

good summary!

Added to issue for 0.15 deprecations: https://github.com/pydata/pandas/issues/7136

What I just pushed up should implement the keep same default for now, but deprecate and warn strategy.

I also made a shared docstring for `DataFrame.boxplot` and `pandas.tools.plotting.boxplot`.

Finally, I was getting an error on one of the tests that checked a figsize since I change my default figsize in my `.matplotlib.matplotlibrc` file. So I reset the matpltolib paramsRC in the setUp of the plotting tests.

Oh my, the return type gets more complicated. `df.groupby('g').boxplot()` will always return a `dict` of whatever `return_type` is. So either a `{dicts}`, `{axes}`, `{namedtuples}`  I'll add a note to the plotting docs.

I also should look into changing the repr of a named tuple. This isn't great.

```
In [16]: df.boxplot(return_type='both')
Out[16]: Boxplot(ax=<matplotlib.axes._subplots.AxesSubplot object at 0x112e64090>, lines={'boxes': [<matplotlib.lines.Line2D object at 0x112f024d0>, <matplotlib.lines.Line2D object at 0x112f09590>, <matplotlib.lines.Line2D object at 0x112f04f10>, <matplotlib.lines.Line2D object at 0x110af9dd0>, <matplotlib.lines.Line2D object at 0x110b02a10>], 'fliers': [<matplotlib.lines.Line2D object at 0x112f07f10>, <matplotlib.lines.Line2D object at 0x112f04b50>, <matplotlib.lines.Line2D object at 0x110af9790>, <matplotlib.lines.Line2D object at 0x110b023d0>, <matplotlib.lines.Line2D object at 0x110b09fd0>], 'medians': [<matplotlib.lines.Line2D object at 0x112f078d0>, <matplotlib.lines.Line2D object at 0x112f04510>, <matplotlib.lines.Line2D object at 0x110af9150>, <matplotlib.lines.Line2D object at 0x110afdd50>, <matplotlib.lines.Line2D object at 0x110b09990>], 'means': [], 'whiskers': [<matplotlib.lines.Line2D object at 0x112f02750>, <matplotlib.lines.Line2D object at 0x11590a290>, <matplotlib.lines.Line2D object at 0x112f09b90>, <matplotlib.lines.Line2D object at 0x112f0c210>, <matplotlib.lines.Line2D object at 0x112f117d0>, <matplotlib.lines.Line2D object at 0x112f11e10>, <matplotlib.lines.Line2D object at 0x110afb410>, <matplotlib.lines.Line2D object at 0x110afba50>, <matplotlib.lines.Line2D object at 0x110b05050>, <matplotlib.lines.Line2D object at 0x110b05690>], 'caps': [<matplotlib.lines.Line2D object at 0x112f02c10>, <matplotlib.lines.Line2D object at 0x112f07290>, <matplotlib.lines.Line2D object at 0x112f0c850>, <matplotlib.lines.Line2D object at 0x112f0ce90>, <matplotlib.lines.Line2D object at 0x112f15490>, <matplotlib.lines.Line2D object at 0x112f15ad0>, <matplotlib.lines.Line2D object at 0x110afd0d0>, <matplotlib.lines.Line2D object at 0x110afd710>, <matplotlib.lines.Line2D object at 0x110b05cd0>, <matplotlib.lines.Line2D object at 0x110b09350>]})
```

Fixed that wrong kwarg in the docstring and added a note about the return type from a grouped boxplot.

This should be ready

looks good

Added some comments.

And I was thinking: now we have the exception of when `by` is specified, `return_type` is ignored. But why wouldn't you want the customize the boxplot via the dict when you have multiple axes? Would it be difficult to harmonize this? (only that we don't have to change the default behaviour in that case) 
So by default it returns an array of axes, but when `return_type='dict'` an array of dicts? and when `return_type='both'` an array of namedtuples (or something better)?

Thanks for catching those.

re `by` following the return_type kwarg as well, I agree. But I think it should also be consistent with the `groupby().boxplot()` return, which is always a dict of `return_type`. A dict is probably better than an array here, since the order of the `by` is presumably arbitrary.

Let me see what I can do quick

Just threw up those doc changes.

I think it will be possible to fix @jorisvandenbossche concern about the return type being ignored when `by` is not None. I'm hacking on it right now.

@jreback if you want to cut an RC and this is holding things up, this should be mergable as is.

@TomAugspurger prob tomorrow, so see what you can do before then :)

Ok. I'll be out of town this weekend unfortunately, but I'll try to keep up to date.

This is actually messier than I thought. Since

`df.boxplot(by='g')` facets by (draws on separate axes) **column** and has a boxplot per group in `by`.

![boxplot_by](https://cloud.githubusercontent.com/assets/1312546/2997595/5a79cf48-dd01-11e3-98f9-6d3af7c241f8.png)

whereas

`df.groupby('g').boxplot()` facets by **group** and has a boxplot per **column**.

![groupby_boxplot](https://cloud.githubusercontent.com/assets/1312546/2997598/6111f4ac-dd01-11e3-8384-a9080cce1a19.png)

---

I'll keep thinking, but I don't think there's a non breaking to get 

`df.groupby('g').boxplot()` === `df.boxplot(by='g')`, which I think is a reasonable thing to assume

I am able to get every variant `df.groupby('g').boxplot()`, `df.boxplot()`, and `df.boxplot(by='g')` to honor the `return_type` kwarg, so maybe I'll just push that up for people to see.

@TomAugspurger OK, but then `df.groupby('g').boxplot()` &#8800; `df.boxplot(by='g')`, and we have to live with that because I don't think we can make it equal? (as this would be a too big change?)

And as both cases are not equal, it's maybe also not that bad that the basic return type is different?

We could eg have for the `.boxplot(by=..)` case: it returns an array of dicts/axes/tuples (depending on `return_type`), while the `.groupby(..).boxpot()` case returns a dict of dicts/axes/tuples (again, depending on `return_type`)? 
Then for `.boxplot(by=..)` nothing would break. For  `.groupby(..).boxpot()` this would be a similar break as for the normal boxplot now, and we should consider if we want to make the change here too.

But I agree this would become a bit messy in the documentation of boxplot.

3 interfaces:
1. `df.boxplot()`
2. `df.boxplot(by='g')`
3. `df.groupby('g').boxplot()`
- 1 returns a `dict` of Lines (changing to an axes in .15)
- 2 returns an array of axes
- 3 returns a dict of {group: return_type}

I've got a fix so that all 3 honor the `return_type` kwarg to return one of `axes`, a dict of `Lines`, or a named tuple with both. This is backwards compatible.

The two unfortunate things are that
1. `df=groupby('g').boxplot()` will not equal `df.boxplot(by='g')`
2. `df.boxplot(by='g', return_type=None)` (default return_type), will return an array of axes, while `df.boxplot(by='g', return_type=axes)` will return a **dict** of {column: axes}

But hey, this is a constrained optimization. PR coming shortly, after I write tests and document.