Repository: pandas-dev/pandas

Issue:
BUG: df.agg, df.transform and df.apply use different methods when axis=1 than when axis=0
- [x] closes #16679
- [x] tests added / passed
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This is a splitoff from #21123, to only fix #16679. #19629 will be fixed in a separate PR afterwards.

Passing functions to ``df.agg``, ``df.transform`` and ``df.apply`` may use different methods when ``axis=1``, than when,``axis=0``, and give different results when NaNs are supplied.

Explanation
-------------
Passing the functions in ``SelectionMixin._cython_table`` to ``df.agg`` should defer to use the relevant cython functions. This currently works as expected when ``axis=0``, but not when ``axis=1``.

The reason for this difference is that ``df.aggregate`` currently defers to ``df._aggregate`` when ``axis=0``, but defers to ``df.apply``, when ``axis=1``, and these may give different result when passed functions and the series/frame contains Nan values. I've solved this by transposing df in ``DataFrame._aggragate`` when ``axis=1``, and passing the possibly transposed on to the super method.

Also, ``df.apply`` delegates back to ``df.agg``, when given lists or dicts as inputs, but only works when axis=0. This PR fixes this, so axis=1 works the as axis=0.

The tests have been heavily parametrized, helping ensure that various ways to call the methods now give correct results for both axes.

@WillAyd @jreback (reviewers of #21123)


Discussion:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=h1) Report
> Merging [#21224](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/7a2fbce899aad302891ff9a95aeb1bd55efe533a?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21224/graphs/tree.svg?token=eZ4WkYLtcO&height=150&src=pr&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #21224      +/-   ##
==========================================
+ Coverage   92.05%   92.05%   +<.01%     
==========================================
  Files         170      170              
  Lines       50708    50716       +8     
==========================================
+ Hits        46677    46685       +8     
  Misses       4031     4031
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.46% <100%> (ø)` | :arrow_up: |
| #single | `42.35% <21.73%> (-0.01%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/apply.py](https://codecov.io/gh/pandas-dev/pandas/pull/21224/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXBwbHkucHk=) | `96.75% <100%> (-0.03%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/21224/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.47% <100%> (-0.01%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/21224/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.21% <100%> (+0.01%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=footer). Last update [7a2fbce...39ced29](https://codecov.io/gh/pandas-dev/pandas/pull/21224?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Hello @topper-123! Thanks for updating the PR.















Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on July 26, 2018 at 21:03 Hours UTC
can you rebase
Rebased and green.
Hold a bit pulling this in. I think I've found an issue that needs fixing.
I discovered a bug in my ``df.aggregate`` implentation. Solving that unearthed several other bugs, as:
* df.transform relies on df.aggregate
* Fixing the bug for df.transform required tests for df.transform(func, axis=1), which unearthed a bug in df.apply(func, axis=1), which also needed fixing

Very much fun!

The most basic bug is that ``df.apply([func], axis=1)`` doesn't work in master. This fixes that, and subsequent bugs in df.transform and df.agg.

```python
>>> from pandas.tests.frame.test_apply import TestDataFrameAggregate
>>> x = TestDataFrameAggregate()
>>> x.frame.apply([np.sqrt], axis=0)  # ok
                   A         B         C         D
                sqrt      sqrt      sqrt      sqrt
ukh1GVv7xX  1.503982       NaN  0.491756  0.787992
wIHkS3BsKA  1.111148       NaN  1.517181       NaN
YWAqLcsqUP       NaN  0.771865  0.760905  0.883760
QhHqzICDVo  0.448304  1.278314  0.517722  0.948910
vEqhEx4vkR       NaN       NaN       NaN  0.913545
4Q1LUyQq9C  0.998117  0.849454       NaN       NaN
XvrwqrxIIK       NaN  0.506467  0.366948  1.183702
xLAqsZsf4n  0.864088       NaN  0.825798  0.865628
Bx8AXfOzTv  1.025291       NaN  0.753349  0.108052
9LoWz0qpSu  0.724288       NaN  1.708485  0.107700
HZI74uGF4k       NaN  0.875412       NaN       NaN
t6Ds6vcRKU       NaN       NaN  0.651367  1.436792
SHSaccg7Wz  0.437198       NaN       NaN       NaN
iZe5ctx7w1  0.127975  1.119973  0.395028       NaN
3aoqjBGybQ       NaN       NaN       NaN  0.772779
saCF7tPAnv  0.719474       NaN  1.045202       NaN
2uo0g5oocb  0.132254  0.563252       NaN       NaN
3duZHu6SDk  0.628383       NaN  0.546263       NaN
p4ug60WPOR       NaN  0.641512  0.406602  0.683798
NbKillWJPL       NaN       NaN  0.699454       NaN
Yc7hoe5odY  0.709012       NaN       NaN       NaN
ZdtVhBzFQZ  0.516489       NaN       NaN  0.351688
SwaSfXJLVm       NaN  0.780858  1.394233  1.076297
LFYhRyDnhZ  0.564244  0.925111       NaN  0.359451
JvfkNGiVkv  1.290368       NaN  0.416229       NaN
iyXPFeeDM2  1.062631  0.532897       NaN       NaN
sy6MqqOmfN  1.443564  0.918141  0.963493       NaN
vkIoXhAka6       NaN       NaN       NaN  0.322189
6MDIrdaygD       NaN       NaN  0.610091       NaN
dhZgEA2cGP  1.505071       NaN       NaN       NaN
>>> x.frame.apply([np.sqrt], axis=1)
TypeError: ("'list' object is not callable", 'occurred at index J5bdGdv0g8')  # master
# this PR below
                        A         B         C         D
ukh1GVv7xX sqrt  1.503982       NaN  0.491756  0.787992
wIHkS3BsKA sqrt  1.111148       NaN  1.517181       NaN
YWAqLcsqUP sqrt       NaN  0.771865  0.760905  0.883760
QhHqzICDVo sqrt  0.448304  1.278314  0.517722  0.948910
vEqhEx4vkR sqrt       NaN       NaN       NaN  0.913545
4Q1LUyQq9C sqrt  0.998117  0.849454       NaN       NaN
XvrwqrxIIK sqrt       NaN  0.506467  0.366948  1.183702
xLAqsZsf4n sqrt  0.864088       NaN  0.825798  0.865628
Bx8AXfOzTv sqrt  1.025291       NaN  0.753349  0.108052
9LoWz0qpSu sqrt  0.724288       NaN  1.708485  0.107700
HZI74uGF4k sqrt       NaN  0.875412       NaN       NaN
t6Ds6vcRKU sqrt       NaN       NaN  0.651367  1.436792
SHSaccg7Wz sqrt  0.437198       NaN       NaN       NaN
iZe5ctx7w1 sqrt  0.127975  1.119973  0.395028       NaN
3aoqjBGybQ sqrt       NaN       NaN       NaN  0.772779
saCF7tPAnv sqrt  0.719474       NaN  1.045202       NaN
2uo0g5oocb sqrt  0.132254  0.563252       NaN       NaN
3duZHu6SDk sqrt  0.628383       NaN  0.546263       NaN
p4ug60WPOR sqrt       NaN  0.641512  0.406602  0.683798
NbKillWJPL sqrt       NaN       NaN  0.699454       NaN
Yc7hoe5odY sqrt  0.709012       NaN       NaN       NaN
ZdtVhBzFQZ sqrt  0.516489       NaN       NaN  0.351688
SwaSfXJLVm sqrt       NaN  0.780858  1.394233  1.076297
LFYhRyDnhZ sqrt  0.564244  0.925111       NaN  0.359451
JvfkNGiVkv sqrt  1.290368       NaN  0.416229       NaN
iyXPFeeDM2 sqrt  1.062631  0.532897       NaN       NaN
sy6MqqOmfN sqrt  1.443564  0.918141  0.963493       NaN
vkIoXhAka6 sqrt       NaN       NaN       NaN  0.322189
6MDIrdaygD sqrt       NaN       NaN  0.610091       NaN
dhZgEA2cGP sqrt  1.505071       NaN       NaN       NaN
```

Similar problem where in master when calling ``df.transform([func], axis=1)`` and ``df.agg([func], axis=1)``.

Likewise with many functions, now ``df.agg(['mean', 'sum'], axis=1)`` is possible, while previously only ``df.agg(['mean', 'sum'], axis=0)`` was possible.
@jreback ?
Comments addressed, except the cython_table one, where I ask for clarification.
comments addressed.
green
thanks @topper-123 nice patch!

Resolving patch:
diff --git a/pandas/conftest.py b/pandas/conftest.py
--- a/pandas/conftest.py
+++ b/pandas/conftest.py
@@ -126,7 +126,7 @@ def all_arithmetic_operators(request):
 # use sorted as dicts in py<3.6 have random order, which xdist doesn't like
 _cython_table = sorted(((key, value) for key, value in
                         pd.core.base.SelectionMixin._cython_table.items()),
-                       key=lambda x: x[0].__class__.__name__)
+                       key=lambda x: x[0].__name__)
 
 
 @pytest.fixture(params=_cython_table)


Test patch: