Applying Custom Functions to Groups of Data in Pandas | by Casey Whorton | Towards Data Science

Here, I will share with you two different methods for applying custom functions to groups of data in pandas. There are many out-of-the-box aggregate and filtering functions available for us to use already, but those don’t always do everything we want. Sometimes, when I want to know “what was the second most recent observation per month” or “what is the difference between two weighted averages by product type”, I know I need to define a custom function to do the job.
Usually, I try not to use this toy dataset because it has been done so many times, but it has a nice categorical column from which to create groups of data, so I’m going with it. For each classification, there are 4 features with numerical data in each column that measure aspects of a flowers petals and sepals.
Typically, a first round analysis might include examining descriptive statistics of your dataset. If you have one table that you are looking at in particular, something like the “describe” method can reveal some key information about your columns such as the number of non-NULL values and the distribution of numerical features. A slightly deeper question to ask is “what are some of these descriptive statistics per group?”. Grouping the data together based on the similar values in a single column and taking the maximum value over all rows in each group looks like this:
The maximum sepal length (in centimeters) is 5.8 for the “setosa” group. This is great but if all we cared about were descriptive statistics at group levels then we could probably program a computer to crank through all the possibilities and return some intersesting findings. What I find comes up more often is the need to apply a custom function to each group in a dataframe. Next, I’ll create a custom function that can’t be easily duplicated with a groupby aggregate function.
This could be just about anything. Here, I am interested in the ratio of the range of sepal lengths to petal lengths. Why? Maybe I’m interested in seeing which type of flower has greater sepal variation to petal variation. Who knows. The point is that this is a specific question you want answered, and normally you might do a groupby with the max aggregate, then the mins, save those results, and get the ratios. It’s possible, but why not execute a custom function over the groups? We’re not limited to aggregates, you could find the 2nd value in a dataframe’s column if you wanted.
Notice that the function explicitly names columns from the dataframe using the bracket notation. You can also create custom functions that apply to all columns in the dataframe and use it on groups in the same manner we’ll see in a moment.
Notice that every element in the groupby is a tuple, with the first element containing whatever you grouped the data by (this can be more than one column, in which case another tuple) and the grouped data (a pandas dataframe). This means that whatever function you use on the groupby elements needs to take both the groups and dataframes.
Simply use the apply method to each dataframe in the groupby object. This is the most straightforward way and the easiest to understand. Notice that the function takes a dataframe as its only argument, so any code within the custom function needs to work on a pandas dataframe.
data.groupby([‘target’]).apply(find_ratio)
It is also possible to accomplish this with a lambda function, executed over each dataframe:
data.groupby([‘target’]).apply(lambda x: find_ratio(x))
Save the groupby element and execute the function for every element in the groupby object. This second option takes a little more work, but could be what you are looking for in terms of customization. While we applied our custom function to each dataframe directly before, we will need to rewrite the function slightly to accept the groupby element as the argument. But first, create a groupby object for the column(s) you want to groupby and assign it a variable name. Next, rewrite the function to work on each groupby in the groupby element. Note: a groupby object is iterable (meaning python can loop through it) and contains both the levels of the grouping and the resulting dataframe.
grouping = data.groupby([‘target’])
Execute the function over the groupby element in a loop or using a list comprehension. I’m choosing to update a dictionary in the first example and to append to a list in the second example. Each method has its advantages:
These are two ways to apply custom function to groups of data in pandas. Please, check out the full notebook here.