Allow multiple lambdas in Groupby.aggregate
State: closed

Closes https://github.com/pandas-dev/pandas/issues/26430



Labels:
name: Groupby

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=h1) Report
> Merging [#26905](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/e9555159afd8caabc88b45ca6c06a16efe2f6afc?src=pr&el=desc) will **decrease** coverage by `0.03%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/26905/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #26905      +/-   ##
==========================================
- Coverage   92.04%      92%   -0.04%     
==========================================
  Files         180      180              
  Lines       50714    50783      +69     
==========================================
+ Hits        46679    46724      +45     
- Misses       4035     4059      +24
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.64% <100%> (-0.03%)` | :arrow_down: |
| #single | `41.83% <16.12%> (-0.12%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/groupby/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9nZW5lcmljLnB5) | `89.72% <100%> (+0.38%)` | :arrow_up: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `88.88% <0%> (-11.12%)` | :arrow_down: |
| [pandas/core/internals/managers.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL21hbmFnZXJzLnB5) | `96.09% <0%> (-0.77%)` | :arrow_down: |
| [pandas/core/internals/blocks.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW50ZXJuYWxzL2Jsb2Nrcy5weQ==) | `94.61% <0%> (-0.53%)` | :arrow_down: |
| [pandas/core/panel.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcGFuZWwucHk=) | `17.61% <0%> (-0.2%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `96.89% <0%> (-0.12%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `90.94% <0%> (+0.1%)` | :arrow_up: |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/26905/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `93.48% <0%> (+0.18%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=footer). Last update [e955515...be712d8](https://codecov.io/gh/pandas-dev/pandas/pull/26905?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Am I correct to understand that this also allows using Keyword aggregation with multiple lambdas as a way to avoid the automatic name mangling if one does wish so?

Something like:

```python
df.groupby('A').agg(
    foo=('B', lambda x: x.max() - x.min()),
    bar=('B', lambda x: x.median() - x.mean()),
)
```

Which results in a dataframe with 2 columns `foo` and `bar`.
Right

```python
In [1]: import pandas as pd

In [2]: df = pd.DataFrame({"A": [1, 2]})

In [3]: df.groupby([1, 1]).agg(foo=('A', lambda x: x.max()), bar=("A", lambda x: x.min()))
Out[3]:
   foo  bar
1    2    2
```

On master, that raises with the SpecificationError about multiple lambdas.
Great news! This issue is one of the main reasons I opened issue #18366 at the first place.

Might I suggest including an example in the documentation as a way to skip automatic name mangling and control the output column names?

Then that would be really great, because users can discover it as a replacement of the already working old behaviour which will disappear.
Updated

1. Changed mangling for `[lambda x: 0, lambda x: 1]` to have the names `[<lambda_0>, <lambda_1>]` rather than `[<lambda>, <lambda_1>]`. Note that `.agg([lambda x: 0])` is still just `[<lambda>]`
2. Added a short whatsnew note
3. Added tests for NamedAgg
>     1. Changed mangling for `[lambda x: 0, lambda x: 1]` to have the names `[<lambda_0>, <lambda_1>]` rather than `[<lambda>, <lambda_1>]`. Note that `.agg([lambda x: 0])` is still just `[<lambda>]`

This made me suddenly remember a comment from someone in #18366 saying:

> there is something deeply queer about mixing the Python's function name-space (something to do with the particular implementation) with the data the column names (something that should surely not know about the implementation). The fact that we are seeing columns (potentially multiple columns) named `<lambda>` is causes me severe cognitive dissonance.

which lead me to the idea: _maybe* when using lambda aggfuncs, using named aggregation could be made mandatory?_

This is probably not an acceptable idea now that we are that late in the iteration (I'm fine with the current pull request BTW), but I just wanted to ask your opinion on this (*: maybe for a future deprecation?)
hmm, can you merge master again, have a conflict now.
thanks @TomAugspurger very nice!
> 
> 
> Right
> 
> ```python
> In [1]: import pandas as pd
> 
> In [2]: df = pd.DataFrame({"A": [1, 2]})
> 
> In [3]: df.groupby([1, 1]).agg(foo=('A', lambda x: x.max()), bar=("A", lambda x: x.min()))
> Out[3]:
>    foo  bar
> 1    2    2
> ```
> 
> On master, that raises with the SpecificationError about multiple lambdas.

I am able to run the `SeriesGroupBy` example in the [What’s new in 0.25.0](https://pandas.pydata.org/pandas-docs/version/0.25/whatsnew/v0.25.0.html#groupby-aggregation-with-multiple-lambdas) notes, but Tom's example with `DataFrameGroupBy` here still fails on Pandas 0.25.0 with `KeyError: "None of [MultiIndex([('A', '<lambda>'),\n            ('A', '<lambda>')],\n           )] are in the [columns]"`.

I assume this isn't expected behaviour? Shall I create a new issue for this?

#### Output of ``pd.show_versions()``

<details>

INSTALLED VERSIONS
------------------
commit           : None
python           : 3.7.3.final.0
python-bits      : 64
OS               : Windows
OS-release       : 10
machine          : AMD64
processor        : Intel64 Family 6 Model 142 Stepping 9, GenuineIntel
byteorder        : little
LC_ALL           : None
LANG             : None
LOCALE           : None.None

pandas           : 0.25.0
numpy            : 1.16.4
pytz             : 2019.1
dateutil         : 2.8.0
pip              : 19.1.1
setuptools       : 41.0.1
Cython           : None
pytest           : None
hypothesis       : None
sphinx           : 2.0.1
blosc            : None
feather          : None
xlsxwriter       : None
lxml.etree       : None
html5lib         : None
pymysql          : None
psycopg2         : 2.8.2 (dt dec pq3 ext lo64)
jinja2           : 2.10.1
IPython          : 7.5.0
pandas_datareader: None
bs4              : None
bottleneck       : None
fastparquet      : None
gcsfs            : None
lxml.etree       : None
matplotlib       : 3.1.0
numexpr          : None
odfpy            : None
openpyxl         : None
pandas_gbq       : None
pyarrow          : None
pytables         : None
s3fs             : None
scipy            : 1.3.0
sqlalchemy       : 1.3.3
tables           : None
xarray           : None
xlrd             : 1.2.0
xlwt             : None
xlsxwriter       : None

</details>
@badge a new issue would be good.

Can you note that "in `DataFrameGroupby.aggregate`, `order` needs to be mangled too"?