Difference between apply() and transform() in Pandas | by B. Chen | Towards Data Science

Pandas is an amazing library that contains extensive built-in functions for manipulating data. When looking for applying a custom function, you might be confusing with the following two choices:
apply(func, axis=0): call a function func along an axis of the DataFrame. It returns the result of applying func along the given axis.
transform(func, axis=0): call a function func on self producing a DataFrame with transformed values. It returns a DataFrame that has the same length as self.
They take the same arguments func and axis. Both call the func along the axis of the given DataFrame. So what is the difference? How do you choose one over the other?
In this article, we will cover the following usages and go through their difference:
Manipulating valuesIn conjunction with groupby() results
Manipulating values
In conjunction with groupby() results
Please check out my Github repo for the source code.
Please check out the following articles if you are not familiar with them:
towardsdatascience.com
towardsdatascience.com
Both apply() and transform() can be used to manipulate values. Let’s see how they work with the help of some examples.
df = pd.DataFrame({'A': [1,2,3], 'B': [10,20,30] })def plus_10(x):    return x+10
Both apply() and transform() can be used to manipulate the entire DataFrame.
df.apply(plus_10)
df.transform(plus_10)
Both apply() and transform() support lambda expression and below is the lambda equivalent:
df.apply(lambda x: x+10)df.transform(lambda x: x+10)
Both apply() and transform() can be used for manipulating a single column
df['B_ap'] = df['B'].apply(plus_10)# The lambda equivalentdf['B_ap'] = df['B'].apply(lambda x: x+10)df['B_tr'] = df['B'].transform(plus_10)# The lambda equivalentdf['B_tr'] = df['B'].transform(lambda x: x+10)
Here are the three main differences
(1) transform() works with function, a string function, a list of functions, and a dict. However, apply() is only allowed with function.
(2) transform() cannot produce aggregated results.
(3) apply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
Let’s take a look at them with the help of some examples.
(1) transform() works with function, a string function, a list of functions, and a dict. However, apply() is only allowed a function.
For transform(), we can pass any valid Pandas string function to func
df.transform('sqrt')
func can be a list of functions, for example, sqrt and exp from NumPy:
df.transform([np.sqrt, np.exp])
func can be a dict of axis labels -> function. For example
df.transform({    'A': np.sqrt,    'B': np.exp,})
(2) transform() cannot produce aggregated results.
We can use apply() to produce aggregated results, for example, the sum
df.apply(lambda x:x.sum())A     6B    60dtype: int64
However, we will get a ValueError when trying to do the same with transform() . We are getting this problem because the output of transform() has to be a DataFrame that has the same length as self.
df.transform(lambda x:x.sum())
(3) apply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
To demonstrate this, let’s create a function to work on 2 Series at a time.
def subtract_two(x):    return x['B'] - x['A']
apply() works perfect with subtract_two and axis=1
df.apply(subtract_two, axis=1)0     91    182    27dtype: int64
However, we are getting a ValueError when trying to do the same with transform(). This is because transform() is only allowed to work with a single Series at a time.
# Getting error when trying the same with transformdf.transform(subtract_two, axis=1)
We will get the same result when using a lambda expression
# It is workingdf.apply(lambda x: x['B'] - x['A'], axis=1)# Getting same errordf.transform(lambda x: x['B'] - x['A'], axis=1)
Both apply() and transform() can be used in conjunction with groupby(). And in fact, it is one of the most compelling usages of transform(). For more details, please take a look at “Combining groupby() results” from the following article:
towardsdatascience.com
Here are the 2 differences when using them in conjunction with groupby()
(1) transform() returns a DataFrame that has the same length as the input
(2) apply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
Let’s create a DataFrame and show the difference with some examples
df = pd.DataFrame({    'key': ['a','b','c'] * 4,    'A': np.arange(12),    'B': [1,2,3] * 4,})
In the example above, the data can be split into three groups by key.
(1) transform() returns a Series that has the same length as the input
To demonstrate this, let’s make a function to produce an aggregated result.
# Aggregating the sum of the given Seriesdef group_sum(x):    return x.sum()
For apply() , it returns one value for each group and the output shape is (num_of_groups, 1) .
gr_data_ap = df.groupby('key')['A'].apply(group_sum)gr_data_apkeya     9b    12c    15Name: A, dtype: int64
For transform(), it returns a Series that has the same length as the given DataFrame and the output shape is (len(df), 1)
gr_data_tr = df.groupby('key')['A'].transform(group_sum)gr_data_tr0     91    122    153     94    125    156     97    128    15Name: A, dtype: int64
(2) apply() works with multiple Series at a time. Buttransform() is only allowed to work with a single Series at a time.
This is the same difference as we mentioned in “1. Manipulating values”, and we just don’t need to specify the argument axis on a groupby() result.
To demonstrate this, let’s create a function to work on 2 Series at a time.
def subtract_two(x):    return x['B'] - x['A']
apply() works with multiple Series at a time.
df.groupby('key').apply(subtract_two)key   a    0    1     3   -2     6   -5b    1    1     4   -2     7   -5c    2    1     5   -2     8   -5dtype: int64
However, we are getting a KeyError when trying the same with transform()
df.groupby('key').transform(subtract_two)
Finally, here is a summary
For manipulating values, both apply() and transform() can be used to manipulate an entire DataFrame or any specific column. But there are 3 differences
transform() can take a function, a string function, a list of functions, and a dict. However, apply() is only allowed a function.transform() cannot produce aggregated resultsapply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
transform() can take a function, a string function, a list of functions, and a dict. However, apply() is only allowed a function.
transform() cannot produce aggregated results
apply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
For working in conjunction with groupby()
transform() returns a Series that has the same length as the inputapply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
transform() returns a Series that has the same length as the input
apply() works with multiple Series at a time. But, transform() is only allowed to work with a single Series at a time.
Thanks for reading.
Please checkout the notebook on my Github for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
When to use Pandas transform() function
Using Pandas method chaining to improve code readability
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More can be found from my Github