API: Reformat output of groupby.describe (#4792)
State: closed

 - [x] closes #4792
 - [x] tests added / passed
 - [x] passes ``git diff upstream/master | flake8 --diff``

Doesn't look like this was address in a PR in 0.20, but the original issue works on master.

Labels:
name: Groupby
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Actually I think the bug is still present. The issue that you want this

```
In [10]: df.groupby('PRICE').VOLUME.describe().unstack(1)
Out[10]: 
       count          mean           std           min           25%           50%           75%           max
PRICE                                                                                                         
24990    1.0  1.500000e+09           NaN  1.500000e+09  1.500000e+09  1.500000e+09  1.500000e+09  1.500000e+09
25499    2.0  2.550000e+09  3.464823e+09  1.000000e+08  1.325000e+09  2.550000e+09  3.775000e+09  5.000000e+09
```

rather than this
```
In [9]: df.groupby('PRICE').VOLUME.describe()
Out[9]: 
PRICE       
24990  count    1.000000e+00
       mean     1.500000e+09
       std               NaN
       min      1.500000e+09
       25%      1.500000e+09
       50%      1.500000e+09
       75%      1.500000e+09
       max      1.500000e+09
25499  count    2.000000e+00
       mean     2.550000e+09
       std      3.464823e+09
       min      1.000000e+08
       25%      1.325000e+09
       50%      2.550000e+09
       75%      3.775000e+09
       max      5.000000e+09
Name: VOLUME, dtype: float64
```

we do a similar unstack already with ``.ohlc``. 
```
In [8]: df.groupby('PRICE').VOLUME.ohlc()
Out[8]: 
             open        high         low       close
PRICE                                                
24990  1500000000  1500000000  1500000000  1500000000
25499  5000000000  5000000000   100000000   100000000
```

So each group gets a single row, while multi-columns are present for multiple aggregations.
multi-index are present for *multiple* groupers.
Ah that makes sense. Thanks for clarifying the expected output!

I've been poking into the code, and since each group goes through `apply()` and `describe()` returns the metrics labeled on the index, it tries to vertically concat the `describe()` results for each group:

```
(Pdb) values
[             VOLUME
count  1.000000e+00
mean   1.500000e+09
std             NaN
min    1.500000e+09
25%    1.500000e+09
50%    1.500000e+09
75%    1.500000e+09
max    1.500000e+09,              
              VOLUME
count  2.000000e+00
mean   2.550000e+09
std    3.464823e+09
min    1.000000e+08
25%    1.325000e+09
50%    2.550000e+09
75%    3.775000e+09
max    5.000000e+09]
```

I could add some logic saying if the indexes for each group are the same, concat on the index and transpose to the columns, but I think that'd be a pretty big change since it will probably affect all `groupby.apply()` functions. Or should we make a special case for `describe()`? Thoughts @jreback?
so you *do* just need to define ``.describe`` as a method on ``DataFrameGroupby`` and ``SeriesGroupby``, then you can do the ``.apply`` (followed by the unstack).

Right now we automagically just do any whitelisted function (including .describe) and they go thru the standard reshaping things. If you can figure out a generic way to do this great, but otherwise defining a method is fine.
Cool, thanks for the insight. 

I defined a new method for `groupby.describe()` and noticed in the process if `groupby(...,axis=1).describe()` is called, the transposing returns the desired results instead of unstacking. I also had to subsequently change a lot of the existing tests since the output has changed. Most notably the test for #14848 changed a lot.
Added previous/current behavior in whatsnew, documentation to describe with `DataFrame.describe.__doc__`, and fixed other failing tests.
Replaced API example using `groupby.agg()` instead of `groupby.ohlc()`, fixed the code block, and simplified `expected` in `test_non_cython_api`
hmm, something is wrong here. Its including the grouper.
```
In [1]: df = pd.DataFrame({'A': [1, 1, 2, 2], 'B': [1, 2, 3, 4]})

In [2]: df.groupby('A').describe()
Out[2]: 
      A                                        B                                          
  count mean  std  min  25%  50%  75%  max count mean       std  min   25%  50%   75%  max
A                                                                                         
1   2.0  1.0  0.0  1.0  1.0  1.0  1.0  1.0   2.0  1.5  0.707107  1.0  1.25  1.5  1.75  2.0
2   2.0  2.0  0.0  2.0  2.0  2.0  2.0  2.0   2.0  3.5  0.707107  3.0  3.25  3.5  3.75  4.0

In [3]: df.groupby('A').agg(['mean', 'std'])
Out[3]: 
     B          
  mean       std
A               
1  1.5  0.707107
2  3.5  0.707107
```
It looks like the grouper is included when using `groupby.apply()`:

```
In [4]: pd.__version__
Out[4]: u'0.19.2' #not current master

In [5]: df.groupby('A').apply(lambda x: x.describe()) 
Out[5]:
           A         B
A
1 count  2.0  2.000000
  mean   1.0  1.500000
  std    0.0  0.707107
  min    1.0  1.000000
  25%    1.0  1.250000
  50%    1.0  1.500000
  75%    1.0  1.750000
  max    1.0  2.000000
2 count  2.0  2.000000
  mean   2.0  3.500000
  std    0.0  0.707107
  min    2.0  3.000000
  25%    2.0  3.250000
  50%    2.0  3.500000
  75%    2.0  3.750000
  max    2.0  4.000000

```

Can be seen with other functions as well:

```
In [11]: df.groupby('A').apply(np.mean) #not idiomatic but should be similar to  df.groupby('A').mean()
Out[11]:
     A    B
A
1  1.0  1.5
2  2.0  3.5
```

Is this a known issue?
@mroeschke so by-definition this is what apply does.

you can use ``._set_group_selection()`` to avoid this problem.
Ah thanks for clarifying that @jreback. Will edit to use `._set_group_selection()` tonight. 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15260?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@c23b1a4`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).


```diff
@@            Coverage Diff            @@
##             master   #15260   +/-   ##
=========================================
  Coverage          ?   86.32%           
=========================================
  Files             ?      141           
  Lines             ?    51177           
  Branches          ?        0           
=========================================
  Hits              ?    44180           
  Misses            ?     6997           
  Partials          ?        0
```


| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15260?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby.py](https://codecov.io/gh/pandas-dev/pandas/compare/c23b1a4c8cb4ac87c9e71703285393e5904e2a8a...184378dd1110f660a41da24c0fb065b9519aa5fb?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS5weQ==) | `95.13% <92.85%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15260?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15260?src=pr&el=footer). Last update [c23b1a4...618bc46](https://codecov.io/gh/pandas-dev/pandas/compare/c23b1a4c8cb4ac87c9e71703285393e5904e2a8a...618bc468d37b9054362bd7ccbf1290ac5bc8baa8?el=footer&src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
Added `_set_group_selection()` to prevent the grouper from being included and resolved conflict with whatsnew
thanks!

keep em coming!