ENH: Groupby.plot enhancement
State: closed

Allow to support `DataFrameGroupBy.plot` and `SeriesGroupBy.plot` covering current `DataFrame.hist(by=XXX)` and `DataFrame.boxplot(by=XXX)` usecases. Implementation is incomplete but covering most of usecases / behaviours.
- [x] fix `FIXME`s
- [x] Split to `plot_seriesgroupby` and `plot_dataramegroupby` specifying default values.
- [x] Refactoring `plot_frame` and `plot_series`, modify `docstring`.  (#8037)
- [x] Make line subplot to accept multiple colors (#9894)
- [x] secondary_y
- [x] error bars (raise `NotImplementedError`)
- [x] split some cleanups required in prior to this PR (#10717)
- [x] doc
- [x] tests
## Following gist contains all patterns of outputs.
- https://gist.github.com/sinhrks/ad805b6689140930d99a
## Rendered doc

![untitled 3](https://cloud.githubusercontent.com/assets/1696302/9026114/83da4e80-395d-11e5-86f0-caef498d16d8.jpg)


Labels:
name: Enhancement
name: Groupby
name: Visualization
description: plotting
name: API Design

Comments:
@sinhrks is this stale?

@jreback Let me revisit after #9814 which does some internal changes.

I think this is almost finished. The current output images are attached  to the gist. I'll add tests and docs based on this.
- https://gist.github.com/sinhrks/ad805b6689140930d99a

Added doc and tests, and attached rendered document on the top. The change looks to be clearer once #10717 is applied.

Some legend locates on inappropriate position, and I think it's better to support legend loc keyword.

@sinhrks this looks nice!

@jorisvandenbossche 
@TomAugspurger 

pls have a look

This PR enables some nice capabilities in pandas, but I do have some concerns:
- In general, Seaborn already does this sort of stuff well with FacetGrid -- do we really want/need to be expanding the plotting API in pandas like this?
- It's not immediately obvious how the API here chooses between different ways of plotting grouped values (e.g., colors vs facets). What rules does this PR use? I like Seaborn's explicit `hue`, `row`, `col` and `col_wrap`.
- The repeated legends in the docs don't look very nice. This is something that Seaborn also does better on the FacetGrid.

@shoyer 

**1. Necessity:** Currently, we have separated plotting methods like `hist` `boxplot` and `scatter` which can output grouped plots. Without this functionality, we can't integrate them to provide unified `plot` function (or accessor).

**2. Facets API:** In above case, we already have `DataFrameGroupBy` or `SeriesGroupBy`. Thus, grouping key is already chosen. Users should select whether the key is for facet or column. I've chosen `axis` currently, but it is nice if there is better API.

**3. Legend:** Yes, we need: 
- keyword to specify legend loc (directly passed to mpl)
- hide unnecessary legend in groupby plot case

I'm most concerned about the API here. What is the rule for whether the groups are plotted on top of each other or in different panels? Does the default behavior depend on plot kind?

@shoyer Current impl is as below. 
- Current spec (Option 1): Plot on a single axes if possible. Otherwise subplot by group.
  - SeriesGroupBy:
    - line, bar, area, hist, kde, box: Single plot colorized by group (default). Subplot by group when `subplots=True`.
    - pie: Single plot is not supported (default). Subplot by group / colorized by index when `subplots=True`
    - scatter, hexbin: not supported
  - DataFrameGroupBy:
    - line, bar, area, hist, kde, box: Subplot by group / colorized by columns (default). When axis=1, subplot by column / colorized by group. `subplots=False` is not supported.
    - pie: not supported.
    - scatter, hexbin: Subplot / colorized by group (default). Single plot when `subplot=False`.

It may be consistent to output subplot by group by default:
- Alternative (Option 2): Subplot by group otherwise specified.
  - SeriesGroupBy:
    - line, bar, area, hist, kde, box: Subplot by group / colorized by column (single color) (default). When `subplot=False` is specified, single plot colorized by group.
    - pie: Subplot by group / colorized by index (default). `subplots=False` is not supported.
    - scatter, hexbin: not supported
  - DataFrameGroupBy:
    - line, bar, area, hist, kde, box: Subplot by group / colorized by columns (default). When axis=1, subplot by column / colorized by group. `subplots=False` is not supported.
    - pie: not supported.
    - scatter, hexbin: Subplot / colorized by group (default). When `subplot=False` is specified, single plot colorized by group.

what's the status of this

@sinhrks @shoyer @TomAugspurger 

Now I think above option 2 is consistent and preferable. If no objections, let me fix the PR to meet option 2.

@sinhrks are there any backward incompatibilities with option 2?

@jorisvandenbossche First, Option2 (and option 1 also) will not affect to the existing `Series` and `DataFrame.plot`. 

However, both option1 and option2 change the existing `GroupBy.plot` behavior. Currently, `GroupBy.plot` is not defined by itself and is equivalent to `Groupby.apply(lambda x: x.plot)`. It draws each group on separate mpl figures and it is not very useful. 

Current behaviour is to make separate figures for each group? (maybe we should keep that as an option? and best to mention in whatsnew how to emualate the old behaviour)

Further, I didn't look into detail, but I find the `axis` not very clear to change the subplots between columns/groups (and it should also be documented).

The problem is also that the `subplots` keyword clashes a bit with its use for plain dataframe plotting to separate the columns. While for groupby.plot it would then separate on the groups

@sinhrks what are we doing with this?

As it needs further discussions, let me move the milestone to next major release. I'm going to check `xray`'s spec.

@sinhrks We haven't quite finished the faceting API for xray, but basically the idea is to copy it from Seaborn. Plot methods accept arguments like `row`, `col` and `col_span`  which control which how to create facets. We'll have a `FacetGrid` API for low level control, but the main user facing API will still be convenience methods like `.plot`.

We _could_ copy this API for pandas. But again, Seaborn already does it -- and does it well. In theory, we might even wrap Seaborn's FacetGrid for this stuff, e.g., `df.plot('x', 'y', row='foo', col='bar')` -> `sns.FacetGrid(df, row='foo', col='bar').map(plt.plot, 'x', 'y')`. However there would pretty assuredly be some API breaks.

@shoyer I don't mean to implement full functionality like seaborn. 

I think our goal is to migrate current `DataFrame.hist`, `boxplot` and `scatter_plot` 's grouping functionality to `DataFrame.plot` in a simple / easy way. 

@sinhrks state of this?

@sinhrks 

so a couple of possibilities here:

1) copy `seaborn` API as much as possible from `FacetGrid`. This would be familiar and provides a nice soln.
2) simply defer to `seaborn` and make it an optional dep (to clarify I mean to have a reasonable pandas API that really just does the import of `seaborn` and pipes almost directly, with very little pandas code on the impl side).
3) like 2), but provide this via an `engine='seaborn'` keyword.

I actually like 3) as the start of pandas removing all plotting code and defering to matplotlib/seaborn/bokeh as the default (note that we might need a set of configuration options to ultimately control this), but for now we can simply have an `engine='seaborn'` kw here.

So since this is 'new' functionaility I dont' see a need to have a `engine='pandas'` kw to control the exiting groupby plotting behavior (unless I am missing something here).

cc @tacaswell 
cc @mdboom
cc @mwaskom
cc @pzwang 
cc @bryevdv

I would be in favor of option 1.

@sinhrks status of this?

I am also in favor of 1.

Is there anyway to tweak the API to return a dictionary mapping the groupby index -> the artist that is plotting it?  That will make writing interactive functionality against these plots much easier.

@sinhrks status?

@sinhrks @TomAugspurger @jorisvandenbossche 

status on this?

status?

@sinhrks status on this?
what's status here? 

@sinhrks is this viable
status of this?
closing as stale, but @sinhrks pls reopen if you want / can fix.