BUG: fix Series.apply(..., by_row), v2.
State: closed

Fixes https://github.com/pandas-dev/pandas/pull/53400#discussion_r1223790606 by making the `by_row` param take `"compat"` as a parameter and using that internally that when `apply` is given dicts og lists.

The compatability path is now to deprecate `by_rows=(True|"compat")` at some point, so `by_rows=False` will become the default in v3.0.

Supercedes #53584.

Code example:

```
df = pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]})

# On 2.0.x and main
print(df.apply([lambda x: 1]))
#          a        b
#   <lambda> <lambda>
# 0        1        1
# 1        1        1
# 2        1        1
```


Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
CC: @rhshadrach.
I don't see how this addresses https://github.com/pandas-dev/pandas/pull/53584#issuecomment-1585728363. The issue is with DataFrame.apply:

    pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).apply([lambda x: 1])

Users have no argument to specify `by_row`, so they have no way to adopt the future (3.0) behavior of `by_rows=False`. Am I missing it?

Puff, I was more focused on the fact than in main we currently have: 

```python
>>> pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).apply([lambda x: 1])
          a  b
<lambda>  1  1
```

While in v2.0 we had:

```python
>>> pd.DataFrame({'a': [1, 1, 2], 'b': [3, 4, 5]}).apply([lambda x: 1])
         a        b
  <lambda> <lambda>
0        1        1
1        1        1
2        1        1
```

but looks like you're right wrt. deprecation process, if we want to deprecate the current behavior for `DataFrame`, we will need a `by_row` parameter in `DataFrame.apply`.
I've made a new version, where we can deprecate the old behavior by deprecating `by_row=True|"compat"` in `DataFrame.apply`, similarly to how do it in `Series.apply`.  
> I was thinking the removal of by_row=True would also apply in the Series case, is that right? If not, a lot of my requests below are invalid.

Unfortunately not. For `Series.apply` we have:

```python
>>> ser = pd.Series([1, np.nan, 2])
>>> ser.apply(np.sum, by_row=True)
0    1.0
1    NaN
2    2.0
dtype: float64
>>> ser.apply(np.sum, by_row="compat")
3.0
```

for `Series.apply` , `by_row="compat"` ends up calling `SeriesApply.apply_compat`, when given a dict- or listlike.  For example:

```
>>> ser.apply(np.sum)  # by_row=True is implicit here
0    1.0
1    NaN
2    2.0
dtype: float64
>>> ser.apply([np.sum])  # for each func in the list, call ser.apply(np.sum, by_row="compat")
sum    3.0
dtype: float64
```

These two cases can't be combined into one parameter option, so I don't see another way forward here myself.

I'll look into your detailed questions later today. 
In pandas 3.0, we will just have the by_row=False behavior, right? Why does the user need to be able to specify by_row=True?
I've changed the `by_row` parameter value names to better suit your proposals:

* `True` in `Series.apply` has been changed to `"compat"`
* `compat` in `Series.apply` has been changed to `"_compat"`
* `False` is unchanged

This means that for both Series and DataFrame `by_row="compat"` now means  "do it the same way as in v2.0", `by_row=False` means pass the whole Series to func in all cases, while `by_row="_compat"` in `Series.apply` is internal and should not be called by end users.

I think this is better?
I’ve updated. Yeah, it’s needed for backward compatability, unfortunately. After by_row=“compat” has been removed in v.3.0, this will become a whole lot simpler.


> In pandas 3.0, we will just have the by_row=False behavior, right?

Yes, though that parameter will be deprecated and it's default value changed to `lib.no_default` in v3.0 in preparation for its removal in v4.0.
Thanks @topper-123!
Nice. I'll scan to see if there are other changes needed for Series/DataFrame.apply/agg, but if not, this concludes the work for these methods, until the deprecation is activated. Series/DataFrame.transform still need some work to get in line with the others, though.

I have not intended to work on the groupby methods, because you're taking care of those, right?
> I have not intended to work on the groupby methods, because you're taking care of those, right?

Yes - my next step is to refactor groupby methods to by-and-large not share code with `core.apply`. Then after this is done, evaluate what we can reasonable deprecate for 3.0.