PDEP-13: Deprecate the apply method on Series and DataFrame and make the agg and transform methods operate on series data
State: closed

There has been some demand for a PDEP regarding changes to `Series.apply` proposed in #52140, for example https://github.com/pandas-dev/pandas/issues/52140#issuecomment-1684879389.

I've made a proposal in this PR.

@MarcoGorelli , if you want to add/change to the proposal I'm fine with that, especially regarding styles/formats that are used for PDEPs and I've missed.

EDIT 17/09-2023: I've made a version 2 of the PDEP.

Labels:
name: Stale
name: PDEP
description: pandas enhancement proposal

Comments:
Thanks for writing this up, much appreciated

If `Series.apply` and `DataFrame.apply` will both operate on the entire column, then maybe the method could be renamed to something more descriptive?
So:
- `Series.apply` -> `Series.map_column`
- `DataFrame.apply` -> `DataFrame.map_columns`
- `Series.map` -> `Series.map_rows`
- `DataFrame.apply(..., axis=1)` -> `DataFrame.map_rows`
- deprecate `axis` argument in `DataFrame.apply`
?

(for reference, the Polars API is likely to move in the above direction)
One this I don't fully understand is whether `Series.apply` is necessary at all, at least in the case where this PDEP is accepted.  Is there a good reason not to just deprecate it with removal in 3 rather than changing behavior?
Some motivation for that was given [here](https://github.com/pandas-dev/pandas/issues/52140#issuecomment-1493289806)

> Series.apply will do many other things than apply a single callable, for example take strings, or lists and dicts, e.g. do Series.apply([np.sum, "mean"])
> One this I don't fully understand is whether `Series.apply` is necessary at all, at least in the case where this PDEP is accepted.

I addition to the point made by @MarcoGorelli, the way I've seen it so far is that we need to have a method that operates on elements of the series (that will be `Series.map`) and a method that operates on the whole Series (`Series.apply`), because the characteristics of working element-wise are different than working on the whole series (e.g. performance (points 3 in the PDEP) and different results from the operation).

Of course `Series.apply`, `Series.agg` and `Series.transform` will be quite similar, so are you discussing using only `Series.agg` and `Series.transform` and dropping `Series.apply`?
> if Series.apply and DataFrame.apply will both operate on the entire column, then maybe the method could be renamed to something more descriptive?

I have not thought much about this. A few things I would think about here:
* I assume this means we drop `.agg` & `.transform` also?
* we should differentiate between operations that are element-wise and series-wise, how does this do that?

More generally, naming of of this type of methods is of course very fundamental to pandas, and a different, though related, subject than this PDEP. So maybe take that in a separate issue/PDEP, and if that is a serious path forward, I can withdraw this PDEP.

> One this I don't fully understand is whether `Series.apply` is necessary at all, at least in the case where this PDEP is accepted. Is there a good reason not to just deprecate it with removal in 3 rather than changing behavior?

I think this is a good idea.

I also considered going  further and eliminating `apply()` throughout the API, pointing people to the proper methods of `agg()` or  `transform()`  and maybe consider eliminating `map`, since it is the same as `transform()` (I think). But if we were to remove `DataFrame.apply()`, I don't see that we have an alternative that does what `DataFrame.apply(func, axis=1)` does, so maybe the best thing is to just get rid of `Series.apply()`.
> 
> Of course `Series.apply`, `Series.agg` and `Series.transform` will be quite similar, so are you discussing using only `Series.agg` and `Series.transform` and dropping `Series.apply`?

Yes, I think that is what @bashtage  is suggesting, and I think it is worth considering. 


> maybe consider eliminating map, since it is the same as transform() (I think).

`map` will be different than `transform` because `map` will operate on each element, while `transform` will operate on the whole Series. So `transform` will be much faster than `map`, while `map` will be more flexible than `transform`. So we should def keep `map` IMO.

> But if we were to remove DataFrame.apply(), I don't see that we have an alternative that does what DataFrame.apply(func, axis=1) does, so maybe the best thing is to just get rid of Series.apply().

If we keep `DataFrame.apply`, we should IMO have an equivalent `Series.apply`.
First, thanks again to you and @rhshadrach for your work here. There's indeed a tonne of inconsistencies which need resolving, and I think everybody is on board with that.
The question is - in which direction to resolve them?

If I've understood correct, the main issues you want to address are:
- inconsistency between `df[['a']].apply(func)` and `df['a'].apply(func)`
- inconsistency between `df['a'].apply('sum')` and `df['a'].apply(lambda x: sum(x))`
- inconsistency between `df['a'].agg(np.sum)` and `df['a'].apply(np.sum)`

One solution (which you're proposing) would be to change `Series.apply` to work series-wise, as opposed to element-wise.

My issue with this is that, as far as I can tell, users are already only thinking of `apply` as an element-wise operation.

From my Kaggle scrapes*, I only see:
- `DataFrame.apply` with `axis=1` (i.e. row-wise). E.g.:
  - `a-journey-through-titanic`
  - `summary-functions-and-maps.ipynb`
- `Series.apply` intended to be used element-wise. E.g.:
   - `a-statistical-analysis-ml-workflow-of-titanic.ipynb`
   - `a-study-on-regression-applied-to-the-ames-dataset.ipynb`
   - `basic-eda-cleaning-and-glove.ipynb`
   - `bert-for-humans-tutorial-baseline.ipynb`
   - `customer-segmentation.ipynb`
   - ... (too many to list, this is super common)

I couldn't find a single case of `DataFrame.apply` with `axis=0`, nor of any kind of `apply` with a string or a list, nor any case of `Series.agg` or `DataFrame.agg`. TBH, before reading the linked issue, I didn't even know most of these were allowed.

If we're going to make breaking changes, I'd suggest that they be made in the direction of keeping `apply` work element-wise. That's why I'm suggesting:
- `DataFrame.apply(..., axis=0)` -> `DataFrame.map_columns`
- `DataFrame.apply(..., axis=1)` -> `DataFrame.map_rows`
- `Series.apply(callable)` -> `Series.map_rows`
- `Series.apply("sum")` -> `Series.map_column`

Then:
- `df[['a']].map_rows(func)` would be consistent with `df['a'].map_rows(func)`
- `df['a'].map_columns('sum')` and `df['a'].map_column(lambda x: sum(x))` would be consistent
- `df['a'].agg(np.sum)` and `df['a'].map_column(np.sum)` would be consistent

To respond to your questions:

> I assume this means we drop .agg & .transform also?

I don't see why, no. `DataFrame.agg` could stay, but the docs would be changed to say "If a function, must either work when passed a DataFrame or when passed to DataFrame.map_columns", and for `Series.agg` the docs would mention `Series.map_column`

> we should differentiate between operations that are element-wise and series-wise, how does this do that?

The name tells you that. So, `map_rows` means that the callable works on each row, and `map_columns` means it happens on each column.

---

Regarding the deprecation process, the advantage of moving to a new function name is that you tell people to just use that. For example:
```
>>> df['a'].apply(func)
FutureWarning: `Series.apply` is deprecated. Please use either `Series.map_rows`, if you want to operate element-wise, or `Series.map_column`, if you want to operate on the whole column.
```
and users can just change their code once.

Conversely, the PDEP's suggested `by_rows` argument would mean that users need to start using a new argument to avoid a warning, and then remove it because it's deprecated (i.e. change their code twice).

---

Sorry if I've missed anything fundamental, and thanks for your patience pushing this through. I really appreciate it, and agree that these inconsistencies really need addressing
Happy to have a call if you think that'd help reach consensus sooner
@MarcoGorelli i am not sure why you are trying to break with the pandas api

the map_* might be ok if things were all new but they r not

stick with the axis= 

+1 in just deprecating and removing apply in favor of transform, agg and map which can accomplish everything 
> +1 in just deprecating and removing apply in favor of transform, agg and map which can accomplish everything

OK yes, I would also be on board with this, +1. Then there's be no need to break API or behaviour
> > +1 in just deprecating and removing apply in favor of transform, agg and map which can accomplish everything
> 
> OK yes, I would also be on board with this, +1. Then there's be no need to break API or behaviour

Me too!
> 
> > But if we were to remove DataFrame.apply(), I don't see that we have an alternative that does what DataFrame.apply(func, axis=1) does, so maybe the best thing is to just get rid of Series.apply().
> 
> If we keep `DataFrame.apply`, we should IMO have an equivalent `Series.apply`.

But isn't what you are proposing for `Series.apply()` equivalent to what can be done with `Series.agg()` ?

Is there a case that your proposal for `Series.apply()` does something different than what `Series.agg()` does?

If we deprecate/remove `Series.apply()` all together, we just point people to `Series.agg()` and `Series.map()`  as replacements.




> 
> If we deprecate/remove `Series.apply()` all together, we just point people to `Series.agg()` and `Series.map()`  as replacements.

yes exactly - the point is you can just use transform for a same size return shape,
agg for a reduction

and raise if that is not the case 

apply was originally meant to be a one sized fits all but has a lot of warts because of this flexibility 


> +1 in just deprecating and removing apply in favor of transform, agg and map which can accomplish everything

+1. 

I agree that deprecation/removal is the best course here.  `Series.apply` has too many magic behaviors, and the alternatives provide a better API where the intent of the call is unambiguous.

Ok, I can see a consensus is coming along, which is great and I agree apply/agg/transform would be very similar after this PDEP. 

A thought:

Are we talking about deprecating only `Series.apply`  i.e. not `apply` on `DataFrame` and groupby objects? How would we do the below without `Series.apply` (after the changes in this PDEP)?:

```python
>>> ser = pd.Series([1, 2, 3,])
>>> func1 = lambda ser: pd.Series([ser[0] * ser[1]**2, ser[2]])
>>> func2 = lambda ser: pd.Series([ser[0] * ser[1]**3, ser[2]])
>>> ser.apply([func1, func2])
   <lambda>  <lambda>
0         4         8
1         3         3
```

(i.e. the result is not a transformation nor an aggregation).

My guess is it will be be done by converting to a DataFrame and back like `ser.to_frame().apply([func1, func2]).iloc[:, 0]`. so the question is if we're ok with that instead of having a `Series.apply` method. It won't be complex to maintain `Series.apply` after the PDEP is implemented, because we need to have Series functionality internally anyway in order to do columns/row operations in `DataFrame.apply`.

> Are we talking about deprecating only Series.apply i.e. not apply on DataFrame and groupby objects?

I thought we were talking about both `Series.apply` and `DataFrame.apply`, and to point people towards:
- `map` for elementwise
- `transform` and `agg` for operating on the whole column

Easy, simple, predictable. My favourite things

> `func1 = lambda ser: pd.Series([ser[0] * ser[1]**2, ser[2]])`

I'd be OK with it not being a core feature. You can just call the function directly if you really need it:
```python
In [48]: >>> ser = pd.Series([1, 2, 3,])
    ...: >>> func1 = lambda ser: pd.Series([ser[0] * ser[1]**2, ser[2]])
    ...: >>> func2 = lambda ser: pd.Series([ser[0] * ser[1]**3, ser[2]])

In [49]: pd.concat([func1(ser), func2(ser)], axis=1, keys=['func1', 'func2'])
Out[49]:
   func1  func2
0      4      8
1      3      3
```
I'd be fine with deprecating apply in both Series and DataFrame. My issue above was with having an `DataFrame.apply` method that clearly works on Series, but not  having the corresponding series functionality. I agree that using `pd.concat` is fine in the example I showed.

BTW: We should probably deprecate groupby.apply also, if we deprecate `Series|DataFrame.apply`?
groupby.apply is a bit different, I think it operates on the entire subdataframe corresponding to the group? Not sure there's a replacement, so I think it'd be fine to keep it (or in any case, to keep it to a separate discussion - sorting out `Series.apply` and `DataFrame.apply` would already be a win 🏆 )
> I'd be fine with deprecating apply in both Series and DataFrame. My issue above was with having an `DataFrame.apply` method that clearly works on Series, but not having the corresponding series functionality. I agree that using `pd.concat` is fine in the example I showed.

I'm not sure that we can deprecate `DataFrame.apply()`, because of this kind of use case:
```python
In [53]: df = pd.DataFrame({"x":[1,2,3], "y":[10,20,30]})

In [54]: func = lambda s: s["y"] if s["x"]%2 == 1 else s["x"]

In [55]: df.apply(func, axis=1)
Out[55]:
0    10
1     2
2    30
dtype: int64
```
I made up an odd function, but the ability to apply a function to each row of a `DataFrame` is a common usage in code from my team.

can't you do that with DataFrame.agg:
```python
In [8]: In [53]: df = pd.DataFrame({"x":[1,2,3], "y":[10,20,30]})
   ...:
   ...: In [54]: func = lambda s: s["y"] if s["x"]%2 == 1 else s["x"]

In [9]: df.agg(func, axis=1)
Out[9]:
0    10
1     2
2    30
dtype: int64
```
?
> can't you do that with DataFrame.agg:

Wasn't aware of that!  Sounds like `DataFrame.apply()` can disappear as well!
> groupby.apply is a bit different, I think it operates on the entire subdataframe corresponding to the group? Not sure there's a replacement, so I think it'd be fine to keep it (or in any case, to keep it to a separate discussion - sorting out Series.apply and DataFrame.apply would already be a win 🏆 )

@rhshadrach , do you have an opinion, I know you've looked at the groupby apply/agg/transform methods? I've  looked into it a bit and to me it does look quite complex, so I'm not sure if groupby.apply does things differently than groupby.agg and groupby.transform.
Just some info: In order to deprecate apply, we will have to add a keyword parameter to `Series.transform`, because currently we fall back to  use `Series.apply` there (see `Apply.transform_str_or_callable`in `pandas/core/apply.py`).

Example:

```python
>>> df = pd.DataFrame({"x":range(100_000)})
>>> %timeit df.transform(lambda x: x + 1). # operates on the series, fast
142 µs ± 589 ns per loop
>>> %timeit df["x"].transform(lambda x: x + 1). # operates on elements, slow
15.5 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

So the behavioral differences are also present here, like for apply.
thanks @topper-123 

oh, wow - more magic fallbacks. so it only operates on the whole series in some cases, like
```python
In [15]: %timeit df['x'].transform(np.sin)
784 µs ± 19.2 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [16]: %timeit df.transform(np.sin)
769 µs ± 23.1 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [17]: %timeit df['x'].transform(lambda x: np.sin(x))
86.6 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

tbh I'd be fine with removing that fallback (independently of this PDEP)
> tbh I'd be fine with removing that fallback (independently of this PDEP)

👍 
Looked into it a bit more and it's even more random than I thought:

```python
>>> df = pd.DataFrame({"x":range(100_000)})
>>> %timeit df.transform(lambda x: x + 1)  # operates on the series, fast
142 µs ± 589 ns per loop
>>> %timeit df.transform([lambda x: x + 1])  # operates on the elements, slow
16.7 ms ± 165 µs per loop
```

So lists of callables operates on elements, while single callables operate on series. 
From https://github.com/pandas-dev/pandas/pull/54747#pullrequestreview-1596898420

> It seems to me that we should have people no longer use `Series.apply` at all, and have them use `Series.agg()` if the function is aggregation-like, and use `Series.map()` or `Series.transform()` if the function is a true ufunc.

I never have a UDF that doesn't fall into the agg/map/transform buckets, and so for me (as a user) I wouldn't mind apply being removed. But are there _useful_ UDFs that don't fall into one of these buckets _and_ can't be easily broken up into a series of operations that do fall into these buckets? One example has been mentioned above (but I'm having a hard time calling it "useful"); a proposed solution (from https://github.com/pandas-dev/pandas/pull/54747#issuecomment-1697085376) is to use concat. In general I find "just use concat" to be inconvenient, somewhat ugly, and breaks method chaining. I personally think this is only an acceptable replacement when the operation is particularly odd or expected to be very rarely needed. I believe that to be the case here. 

It seems more common to do something like `frame.apply(some_big_ml_op)`, and even more so, `frame.groupby(...).apply(some_big_ml_op)`. I'd guess such a use case could return pandas objects that aren't the same length as the original input - so they are neither a reduction nor a transform. However looking at examples from public sources, we find users again and again using apply for reductions/transforms and not when it is actually needed. So here I'm still okay with saying "use concat".

I am -1 (but very much open to being convinced otherwise) on removing apply from some pandas objects but not all, and +1 on removing apply throughout the entire API - Series, DataFrame, GroupBy, Window, Resample. However, there are consequences of this. Namely, GroupBy.apply will operate on the entire group rather than individual columns. I think that is a nice feature to have, and am in favor of expanding the API of agg/transform (map doens't make sense) to include this via arguments (e.g. `axis=[0, 1, None]`). It would allow a user to control whether the object fed into the passed func are individual columns, rows, or the entire frame for both DataFrame.agg/transform and DataFrameGroupBy.agg/transform.

From https://github.com/pandas-dev/pandas/pull/54747#issuecomment-1697193758:

> (or in any case, to keep it to a separate discussion - sorting out Series.apply and DataFrame.apply would already be a win 🏆 )

When sorting out the API, I think we need to consider implications on all relevant pandas objects - Series, DataFrame, SeriesGroupBy, DataFrameGroupBy, Window, Resample. Having as similar an API as makes sense between the methods on these classes is beneficial to users.

> GroupBy.apply will operate on the entire group rather than individual columns

So it behaves differently that `(Series|DataFrame).apply`, right? So removing `(Series|DataFrame).apply` wouldn't be a problem wrt. having a consistent API, because GroupBy.apply is different? How about `Window.apply`, `Resample.apply`, I assue they work like `Groupby.apply`?

Agree very much about the importance of  API consistency, but if these groupby `apply` methods are different than `(Series|DataFrame).apply` it would IMO be an extra argument for removing `(Series|DataFrame).apply` (because then  users wouldn't have to understand the difference to the groupby apply methods).

> Agree very much about the importance of API consistency, but if these groupby `apply` methods are different than `(Series|DataFrame).apply` it would IMO be an extra argument for removing `(Series|DataFrame).apply` (because then users wouldn't have to understand the difference to the groupby apply methods).

I do think they are different, at least with respect to a single callable passed as an argument.

- `Series.apply(func)` applies `func` element by element or on the whole series, dependent on the argument type
  - I think we believe that users should use `map` or `agg` instead.  Or maybe `transform` instead of `map` ?
- `DataFrame.apply(func, axis=1)` applies `func` to each row of the `DataFrame`
  - I think we believe that users should use `transform` instead.
- `DataFrame.apply(func)` applies `func` to each column of the `DataFrame`
  - I think we believe that users should use `agg` instead.
- `SeriesGroupby.apply(func)` and `DataFrameGroupby.apply(func)` applies `func` to each group.  
  - I think there are cases where  `func` can't be used with `agg` or `transform` to produce the same result.