Creating Custom Aggregations to Use with Pandas groupby | by Rose Day | Towards Data Science

Pandas groupby is a function you can utilize on dataframes to split the object, apply a function, and combine the results. This function is useful when you want to group large amounts of data and compute different operations for each group. If you are using an aggregation function with your groupby, this aggregation will return a single value for each group per function run. After forming your groups, you can run one or many aggregations on the grouped data.
The dataset I am using today is Amazon Top 50 Bestselling Books on Kaggle. This dataset has some nice numeric columns and categories that we can work with. Importing that dataset, we can quickly look at one example of the data using head(1) to grab the first row and .T to transpose the data. Here we can see that Genre is a great category column to groupby, and we can aggregate the user ratings, reviews, price, and year.
df = pd.read_csv("bestsellers_with_categories.csv")print(df.head(1).T)>>> 0>>> Name         10-Day Green Smoothie Cleanse>>> Author                            JJ Smith>>> User Rating                            4.7>>> Reviews                              17350>>> Price                                    8>>> Year                                  2016>>> Genre                          Non Fiction
Now that we have taken a quick look at the columns, we can use groupby to group Genre’s data. Before applying groupby, we can see two Genre categories in this dataset, Non-Fiction, and Fiction, meaning we will have two groups of data to work with. We can play around with the groups if we wanted to consider the author or book title, but we will stick with Genre for now.
df.Genre.unique()>>> array(['Non Fiction', 'Fiction'], dtype=object)group_cols = ['Genre']ex = df.groupby(group_cols)
After setting up our groups, we can begin to create custom aggregations. I have used custom aggregations in functions like this to filter specific values out before performing calculations or aggregations under different conditions. When I am testing out aggregation functions, I like to start with a small series to validate the output, such as the one below.
ser = pd.Series([1,10,4,2,10,10])
Once we have a series of data to test with, we can begin creating our aggregation functions. Below I have created three aggregation functions. The first and second functions are non_nan_mean, and standard_deviation which validate the series is not empty, remove any NA values, and perform a mean or standard deviation calculation. The last aggregation is a mean_lower_rating, which eliminates any upper values greater than five and calculates the mean on the lower values.
def non_nan_mean(x):    if x.empty:        return None    else:         x = x.dropna()        return np.mean(x)def standard_deviation(x):    if x.empty:        return None     else:         x = x.dropna()        return np.nanstd(x)    def mean_lower_rating(x):    upper = []    for val in x:        if val > 5:            continue         else:             upper.append(val)    return np.mean(upper)
Once you have defined your aggregation functions, as many or little as you need, you can apply your series to them to test. I printed my values out to look them over, like below.
print(non_nan_mean(ser))print(standard_deviation(ser))print(mean_lower_rating(ser))>>> 6.166666666666667>>> 3.9334745737353156>>> 2.3333333333333335
After understanding the dataset you are working with and testing out the aggregation functions using a small series of data, you can apply the aggregation functions created using the agg function mentioned earlier. This function works on dataframes, which allows us to aggregate data over a specified axis. A simple way to apply these aggregations is to create a list and pass that list as an argument. This method will apply your aggregations to all numeric columns within your group dataframe, as shown in example one below. In this example, you can see I am calling ex, which is the grouped output from earlier.
aggs = [non_nan_mean, standard_deviation,mean_lower_rating]ex2 = ex.agg(aggs)print(ex2)
As can be seen with the output, the mean_lower_rating aggregation does not perform well on specific columns, caused by the function designed for a particular column in mind, which was User Rating. Considering this, we can look at different ways to pass aggregation arguments into the agg function, which will clean this output up.
Another way to pass arguments to agg is to develop a dictionary. The dictionary maps the column names to aggregation functions to run. An example of this method is seen in example two.
aggs_by_col = {'Reviews': [non_nan_mean],                'Price': [non_nan_mean,standard_deviation],               'User Rating': [mean_lower_rating]}ex1 = ex.agg(aggs_by_col)print(ex1)
This method is preferred if you do not want to apply all aggregations across all columns, as mentioned previously, with the mean_lower_rating aggregation. The process of defining which columns your aggregation applies to can be very beneficial for large datasets as it cleans up the output, providing you just the data you want to see. This method of applying the aggregations allowed me to specify the mean_lower_rating aggregation only for User Rating, and the other aggregations to their respective columns.
Using Pandas groupby with the agg function will allow you to group your data into different categories and aggregate your numeric columns into one value per aggregation function. If you have use cases to create custom aggregation functions, you can write those functions to take in a series of data and then pass them to agg using a list or dictionary. You can pass a list if you want all aggregations applied to all numeric columns, and you can pass a dictionary if you’re going to specify what aggregations apply to what columns. You can also use lambda functions to create your aggregations if you prefer, which I did not cover in this article.
pandas.DataFrame.groupby documentation
pandas.core.groupby.DataFrameGroupBy.aggregate documentation
Group by: split-apply-combine documentation
pandas.DataFrame.agg documentation
If you would like to read more, check out some of my other articles below!