When to use Pandas transform() function | by B. Chen | Towards Data Science

Pandas is an amazing library that contains extensive built-in functions for manipulating data. Among them, transform() is super useful when you are looking to manipulate rows or columns.
In this article, we will cover the following most frequently used Pandas transform() features:
Transforming valuesCombining groupby() resultsFiltering dataHandling missing value at the group level
Transforming values
Combining groupby() results
Filtering data
Handling missing value at the group level
Please check out my Github repo for the source code
Let’s take a look at pd.transform(func, axis=0)
The first argument func is to specify the function to be used for manipulating data. It can be a function, a string function name, a list of functions, or a dictionary of axis labels -> functions
The 2nd argument axis is to specify which axis the func is applied to. 0 for applying the func to each column and 1 for applying the func to each row.
Let’s see how transform() works with the help of some examples.
We can pass a function to func. For example
df = pd.DataFrame({'A': [1,2,3], 'B': [10,20,30] })def plus_10(x):    return x+10df.transform(plus_10)
You can also use a lambda expression. Below is the lambda equivalent for the plus_10() :
df.transform(lambda x: x+10)
We can pass any valid Pandas string function to func, for example 'sqrt' :
df.transform('sqrt')
func can be a list of functions. for example sqrt and exp from NumPy:
df.transform([np.sqrt, np.exp])
func can be a dict of axis labels -> function. For example
df.transform({    'A': np.sqrt,    'B': np.exp,})
One of the most compelling usages of Pandas transform() is combining grouby() results.
Let’s see how this works with the help of an example. Suppose we have a dataset about a restaurant chain
df = pd.DataFrame({  'restaurant_id': [101,102,103,104,105,106,107],  'address': ['A','B','C','D', 'E', 'F', 'G'],  'city': ['London','London','London','Oxford','Oxford', 'Durham', 'Durham'],  'sales': [10,500,48,12,21,22,14]})
We can see that each city has multiple restaurants with sales. We would like to know “What is the percentage of sales each restaurant represents in the city”. The expected output is:
The tricky part in this calculation is that we need to get a city_total_sales and combine it back into the data in order to get the percentage.
There are 2 solutions:
groupby(), apply(), and merge()groupby() and transform()
groupby(), apply(), and merge()
groupby() and transform()
The first solution is splitting the data with groupby() and using apply() to aggregate each group, then merge the results back into the original DataFrame using merge()
Step 1: Use groupby() and apply() to calculate the city_total_sales
city_sales = df.groupby('city')['sales']             .apply(sum).rename('city_total_sales').reset_index()
groupby('city') split the data by grouping on the city column. For each of these groups, the function sum gets applied to the sales column to calculate the sum for each group. Finally, the new column gets renamed to city_total_sales and the index gets reset (Note: reset_inde() is required to clear the index generated bygroupby('city')).
In addition, Pandas has a built-in sum() function and the following is the Pandas sum() equivalent:
city_sales = df.groupby('city')['sales']             .sum().rename('city_total_sales').reset_index()
Step 2: Use merge() function to combine the results
df_new = pd.merge(df, city_sales, how='left')
The group results get merged back into the original DataFrame using merge() with how='left' for left outer join.
Step 3: Calculate the percentage
Finally, the percentage can be calculated and formatted.
df_new['pct'] = df_new['sales'] / df_new['city_total_sales']df_new['pct'] = df_new['pct'].apply(lambda x: format(x, '.2%'))
This certainly does our work. But it is a multistep process and requires extra code to get the data in the form we require.
We can solve this effectively using the transform() function
This solution is a game-changer. A single line of code can solve the apply and merge.
Step 1: Use groupby() and transform() to calculate the city_total_sales
The transform function retains the same number of items as the original dataset after performing the transformation. Therefore, a one-line step using groupby followed by a transform(sum) returns the same output.
df['city_total_sales'] = df.groupby('city')['sales']                           .transform('sum')
Step 2: Calculate the percentage
Finally, this is the same as the solution one to get the percentage.
df['pct'] = df['sales'] / df['city_total_sales']df['pct'] = df['pct'].apply(lambda x: format(x, '.2%'))
transform() can also be used to filter data. Here we are trying to get records where the city’s total sales is greater than 40
df[df.groupby('city')['sales'].transform('sum') > 40]
Another usage of Pandas transform() is to handle missing values at the group level. Let’s see how this works with an example.
Here is a DataFrame for demonstration
df = pd.DataFrame({    'name': ['A', 'A', 'B', 'B', 'B', 'C', 'C', 'C'],    'value': [1, np.nan, np.nan, 2, 8, 2, np.nan, 3]})
In the example above, the data can be split into three groups by name, and each group has missing values. A common solution to replace missing values is to replace NaN with mean.
Let’s take a look at the average value in each group.
df.groupby('name')['value'].mean()nameA    1.0B    5.0C    2.5Name: value, dtype: float64
Here we can use transform() to replace missing values with the group average value.
df['value'] = df.groupby('name')                .transform(lambda x: x.fillna(x.mean()))
Thanks for reading.
Please checkout the notebook on my Github for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
Difference between apply() and transform() in Pandas
Using Pandas method chaining to improve code readability
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More can be found from my Github