DOC GH17505 Added some links and examples. To little/much/wrong?
State: closed

- [ ] closes #17505 
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Before I spend more time on this, I'd like to know if I am doing to much, to little or just plain all wrong.


Labels:
name: Docs
name: Groupby

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=h1) Report
> Merging [#17908](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/5bf7f9a4ff6968fd444fa7098f8dc95586591994?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17908/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17908      +/-   ##
==========================================
- Coverage   91.23%   91.22%   -0.02%     
==========================================
  Files         163      163              
  Lines       50105    50105              
==========================================
- Hits        45715    45706       -9     
- Misses       4390     4399       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.03% <ø> (ø)` | :arrow_up: |
| #single | `40.31% <ø> (-0.06%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.75% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=footer). Last update [5bf7f9a...ccfa848](https://codecov.io/gh/pandas-dev/pandas/pull/17908?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

can you post a rendered version of this page? since you are doing lots of changes, hard to see what the new version would look like. 
*****************************
Group By: split-apply-combine
*****************************

Split-apply-combine is a common paradigm in data analysis. It involves splitting
the data set into smaller groups, applying some operation to each group
independently and combining the results into a data structure. This strategy is
supported for example in Excels pivot tables, SQLs ``group by`` operator and Rs
``plyr`` package. This section will look at the the Pandas ``groupby`` and
related functions and show you how to do split-apply-combine in Pandas. See the
:ref:`cookbook<cookbook.grouping>` for some advanced strategies

The split step is the most straightforward. See the section on
:ref:`splitting<groupby.split>` below.

In the apply step you may wish to apply one of the following operations:
  - **Aggregate**: Get a single value for each group. This could be a summary
    statistic like sum or mean of some column or counting the number of members
    in the group. See the section on :ref:`aggregating<groupby.aggregate>` below.
  - **Filter**: When you want a subset of your original data. Discard data
    according to some function applied to each group. This can be useful when
    for example you wish to discard groups with low member count. See the section on 
    :ref:`filtering<groupby.filter>` below.
  - **Transform**: A new value for each original row. This can be used to
    normalize/scale data or filling in erroneous or missing values. See the
    section on :ref:`transforming<groupby.transform>` below.
Pandas has direct support for these three operations and will try and return a
sensibly combined result. See here for further help on `when to use
aggregate/filter/transform in Pandas
<https://pythonforbiologists.com/when-to-use-aggregatefiltertransform-in-pandas/>`_.

Pandas also supports iteration over the groups created in the split step; Using
iteration over the groups (rather than the three shortcut functions) renders
more control over the apply and combine parts of the process, but also requires
more work from the programmer. See the section in
:ref:`iterating<groupby.iterating>` below.

<div class="section" id="aggregation"><span id="groupby-aggregate"></span>

## Aggregation[¶](#aggregation "Permalink to this headline")

This section describes how to aggregate data. We will be giving examples using the `<span class="pre">tips.csv</span>` dataset. Each row represents a meal at some restaurant; The columns store the value of the total bill, the size if the tip and some metadata about the customer.

<div class="highlight-ipython">

<div class="highlight">

<pre><span></span><span class="gp">In [54]:</span> <span class="n">tips</span> <span class="o">=</span> <span class="n">pd</span><span class="o">.</span><span class="n">read_csv</span><span class="p">(</span><span class="s1">'./data/tips.csv'</span><span class="p">)</span>

<span class="gp">In [55]:</span> <span class="n">tips</span>
<span class="gh">Out[55]:</span> <span class="go"></span>
 <span class="go">total_bill   tip     sex smoker   day    time  size</span>
<span class="go">0         16.99  1.01  Female     No   Sun  Dinner     2</span>
<span class="go">1         10.34  1.66    Male     No   Sun  Dinner     3</span>
<span class="go">2         21.01  3.50    Male     No   Sun  Dinner     3</span>
<span class="go">3         23.68  3.31    Male     No   Sun  Dinner     2</span>
<span class="go">4         24.59  3.61  Female     No   Sun  Dinner     4</span>
<span class="go">5         25.29  4.71    Male     No   Sun  Dinner     4</span>
<span class="go">6          8.77  2.00    Male     No   Sun  Dinner     2</span>
<span class="go">..          ...   ...     ...    ...   ...     ...   ...</span>
<span class="go">237       32.83  1.17    Male    Yes   Sat  Dinner     2</span>
<span class="go">238       35.83  4.67  Female     No   Sat  Dinner     3</span>
<span class="go">239       29.03  5.92    Male     No   Sat  Dinner     3</span>
<span class="go">240       27.18  2.00  Female    Yes   Sat  Dinner     2</span>
<span class="go">241       22.67  2.00    Male    Yes   Sat  Dinner     2</span>
<span class="go">242       17.82  1.75    Male     No   Sat  Dinner     2</span>
<span class="go">243       18.78  3.00  Female     No  Thur  Dinner     2</span>

<span class="go">[244 rows x 7 columns]</span>
</pre>

</div>

</div>

What if we wanted to know the average total bill on each day? We split the data so that each group consists of all the meals eaten on the same day. We want a single value for each group, so we should use the aggregate function:

<div class="highlight-ipython">

<div class="highlight">

<pre><span></span><span class="gp">In [56]:</span> <span class="n">tips</span><span class="o">.</span><span class="n">groupby</span><span class="p">(</span><span class="s1">'day'</span><span class="p">)</span><span class="o">.</span><span class="n">aggregate</span><span class="p">(</span><span class="s1">'mean'</span><span class="p">)</span>
<span class="gh">Out[56]:</span> <span class="go"></span>
 <span class="go">total_bill       tip      size</span>
<span class="go">day</span> 
<span class="go">Fri    17.151579  2.734737  2.105263</span>
<span class="go">Sat    20.441379  2.993103  2.517241</span>
<span class="go">Sun    21.410000  3.255132  2.842105</span>
<span class="go">Thur   17.682742  2.771452  2.451613</span>
</pre>

</div>

</div>

The result has the group names, in this case the days, as the index along the grouped axis. Along the other axis we have the columns for which Pandas could calculate a mean, i.e. the ones with a numeric data type. We could have selected the `<span class="pre">total_bill</span>` column either before or after aggregating to limit the result to this column, but not before splitting since we need the `<span class="pre">day</span>` column for the splitting.

How about the number of guests for each day and for each time of day? In this case it is not enough to split the data on the day it was eaten, we also need split by the time of day. Instead of calculating the mean, like in the previous example we use the `<span class="pre">sum</span>` function.

<div class="highlight-ipython">

<div class="highlight">

<pre><span></span><span class="gp">In [57]:</span> <span class="n">tips</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">'day'</span><span class="p">,</span> <span class="s1">'time'</span><span class="p">])[</span><span class="s1">'size'</span><span class="p">]</span><span class="o">.</span><span class="n">agg</span><span class="p">(</span><span class="s1">'sum'</span><span class="p">)</span>
<span class="gh">Out[57]:</span> <span class="go"></span>
<span class="go">day   time</span> 
<span class="go">Fri   Dinner     26</span>
 <span class="go">Lunch      14</span>
<span class="go">Sat   Dinner    219</span>
<span class="go">Sun   Dinner    216</span>
<span class="go">Thur  Dinner      2</span>
 <span class="go">Lunch     150</span>
<span class="go">Name: size, dtype: int64</span>
</pre>

</div>

</div>

`<span class="pre">agg</span>` is short for `<span class="pre">aggregate</span>`.

Pandas has support for a number of basic descriptive statistic functions which can be used with aggregate:
    ``count``, Number of non-NA observations
    ``sum``, Sum of values
    ``mean``, Mean of values
    ``mad``, Mean absolute deviation
    ``median``, Arithmetic median of values
    ``min``, Minimum
    ``max``, Maximum
    ``mode``, Mode
    ``abs``, Absolute Value
    ``prod``, Product of values
    ``std``, Bessel-corrected sample standard deviation
    ``var``, Unbiased variance
    ``sem``, Standard error of the mean
    ``skew``, Sample skewness (3rd moment)
    ``kurt``, Sample kurtosis (4th moment)
    ``quantile``, Sample quantile (value at %)

What if we need to know the difference between the smallest and largest total bill for each day? Again we split the data so each group has the meals eaten on the same day. But which function do we use to find the difference? The `<span class="pre">agg</span>` function also accepts a function as argument. The function is called once per group with the current group as argument and should return a single value.

<div class="highlight-ipython">

<div class="highlight">

<pre><span></span><span class="gp">In [58]:</span> <span class="n">tips</span><span class="o">.</span><span class="n">groupby</span><span class="p">([</span><span class="s1">'size'</span><span class="p">])</span><span class="o">.</span><span class="n">agg</span><span class="p">(</span><span class="k">lambda</span> <span class="n">group</span><span class="p">:</span> <span class="nb">max</span><span class="p">(</span><span class="n">group</span><span class="p">)</span> <span class="o">-</span> <span class="nb">min</span><span class="p">(</span><span class="n">group</span><span class="p">))[</span><span class="s1">'total_bill'</span><span class="p">]</span>
<span class="gh">Out[58]:</span> <span class="go"></span>
<span class="go">size</span>
<span class="go">1     7.00</span>
<span class="go">2    34.80</span>
<span class="go">3    40.48</span>
<span class="go">4    31.84</span>
<span class="go">5    20.50</span>
<span class="go">6    21.12</span>
<span class="go">Name: total_bill, dtype: float64</span>
</pre>

</div>

</div>

</div>
The formatting is not super but I hope it gives a better idea of whats been changed and how it would look.
I pasted the 2 sections I rewrote above, it should be obvious where they belong in the original?

Is there a better way to do this? 
@linebp can you post a rendered screenshot of this?
can you rebase and show a rendered screenshot?
@linebp sorry let this get away from us. happy to have some clarifications, but can you do in a more targeted manner, IOW more PR's with smaller changes is usually better.
Ill have a look at again and see if I can do a PR with a smaller change and do a screenschot of the rendered changes?