5 tips for data aggregation in pandas | by Zolzaya Luvsandorj | Towards Data Science

When doing data analysis, being able to skillfully aggregate data plays an important role. Pandas provides many useful methods, some of which are perhaps less popular than others. Moreover, even for the well-known methods, we could increase its utility by tweaking its arguments further or complement it with other methods. With these considerations, here are 5 tips on data aggregation in pandas in case you haven’t across these before:
In this post, I will demonstrate how they are useful with examples. I also hope these tips will help you write a clear, concise and readable code.
I assume the reader (👀 yes, you!) has:◼️ access to and is familiar with Python including installing packages, defining functions and other basic tasks◼️ working knowledge using pandas including basic data manipulation.
Make sure you have both pandas and seaborn installed if you haven’t already.
If you are new to Python, this is a good place to get started. If you haven’t used pandas before, this is a good resource to check out.
I have used and tested the scripts in Python 3.7.1 in Jupyter Notebook.
We will use seaborn’s tips dataset to illustrate my tips.
# Import packagesimport pandas as pdfrom seaborn import load_dataset# Import data df = load_dataset('tips').rename(columns={'sex': 'gender'})df
Details about this dataset including data dictionary can be found here (this source is actually for R, but it appears to be referring to the same underlying dataset). I have quoted their data description below for quick access:
“One waiter recorded information about each tip he received over a period of a few months working in one restaurant.”
You are probably already familiar with this series function: value_counts(). Running df['day'].value_counts() will give us the counts of unique values in day variable. If we specify normalize=True inside the method, it will give us percentages instead. This is useful for a single variable, but sometimes we need to see counts by multiple variables. For instance, if we wanted to get counts by day and time, one way to get this is to use groupby() + size() + unstack():
df.groupby(['time', 'day']).size().unstack()
Another more flexible way is to use crosstab():
pd.crosstab(df['time'], df['day'])
Using crosstab() has some advantages. Firstly, it’s easy to get row and column subtotals - we just add margins=True:
pd.crosstab(df['time'], df['day'], margins=True)
Isn’t it awesome? Secondly, we can easily get percentages instead of counts by tweaking the normalize argument:
pd.crosstab(df['time'], df['day'], margins=True, normalize=True)
In this example, we got table percentages by setting normalize=True. This is equivalent to setting it to normalize='all'. For row percentages, we use normalize='index' and normalize='columns' for column percentages. We could further extend the variable sets for columns and rows too:
pd.crosstab([df['time'], df['gender']], [df['day'], df['smoker']],             margins=True)
After discovering crosstab(), I now use it regularly when using pandas.
You may already know about both groupby() and describe(). But have you used them together? By using these two together, we can check the summary statistics of a numeric variable by unique values in a categorical column with just one line like this:
df.groupby('day')['tip'].describe()
We can see how the summary statistics of tip compare across days.
In this post, we will use agg(), the alias of aggregate(). However, both can be used interchangeably.
You may probably know the basic aggregation syntax like this one:
df.groupby('day')[['tip']].mean()
Here are some alternative ways to get the same output with agg():
df.groupby('day')[['tip']].agg('mean')df.groupby('day').agg({'tip': 'mean'})
In this simple example, there isn’t a clear advantage why one should use agg() over the first alternative. However, using agg() gives us more flexibility when we want to look at output of multiple aggregate functions. For instance, we can get both mean and standard deviation at one go by either passing a list or a dictionary to agg().
df.groupby('day')[['tip']].agg(['mean', 'std']) # listdf.groupby(['day']).agg({'tip': ['mean', 'std']}) # dictionary
If we ever had to rename the output columns, instead of doing this:
df.groupby('day')[['tip']].agg(['mean', 'std']).rename(columns={'mean': 'avg', 'std': 'sd'})
we could do either of these more succinctly:
df.groupby(['day'])[['tip']].agg([('avg', 'mean'), ('sd', 'std')])df.groupby(['day']).agg({'tip': [('avg', 'mean'), ('sd', 'std')]})
Using either list or dictionary so far has worked equally. However, using list is more concise if we want to inspect a same set of summary statistics for multiple variables.
df.groupby('day')[['tip', 'size']].agg(['mean', 'std'])
On the other hand, sometimes using dictionary is the way to go. With dictionary, we can specify different sets of aggregate functions for each variable:
df.groupby(['day']).agg({'tip': ['mean', 'std'],                          'size': ['median']})
There are many aggregate functions to use:
◼️ Frequency / Counts: size() , count()◼️ Central tendency: mean() , median()◼️ Variance: std(), var()◼️ Others: min(), max(), ️sum(), prod(), quantile() and many more.
On top of these, we could use any Series or DataFrame method inside agg(). For instance, to see the highest two tips by day, we use:
df.groupby('day')['tip'].nlargest(2)
We can see that some of the biggest tips were on Saturday. 💰
In addition, we could use lambda functions too:
df.groupby(['day']).agg({'tip': [('range', lambda x: x.max() - x.min()),                                  ('IQR', lambda x: x.quantile(.75) - x.quantile(.25))]})
This extends the possibility of what we could do! 😎
Let’s say we needed to get the mean tip by 2 variables. A common approach is to use groupby():
df.groupby(['time', 'day'])['tip'].mean().unstack()
But a slightly better way to do this is to use pivot_table():
df.pivot_table('tip', 'time', 'day')
Here, we didn’t specify the arguments because we have provided the values to the method in the correct order. Here is another version of the same code where all arguments are explicitly specified:
df.pivot_table(values='tip', index='time', columns='day')
This version is probably more readable. I think readability won’t be compromised if we omit the first argument name to be slightly more concise:
df.pivot_table('tip', index='time', columns='day')
We will keep this format moving forward.
By default, pivot_table() gives us mean values. However, we can easily change to our preferred function such as sum() by specifying it to the aggfunc argument:
df.pivot_table('tip', index='time', columns='day', aggfunc='sum')
Similar to crosstab(), it's also easy to get subtotals with pivot_table().
df.pivot_table('tip', index='time', columns='day', aggfunc='sum',                margins=True)
We can also pass dictionary to aggfunc to customise the aggregate functions for each variable passed to values argument. Another useful argument is fill_value where we specify what values we want to see if the output is missing. Let’s see an example exemplifying these points:
df.pivot_table(['tip', 'size'], index=['time', 'smoker'],                columns='day', fill_value=0, margins=True,               aggfunc={'tip': 'sum', 'size': 'max'})
With pivot_table(), you know exactly which variables would be in rows and columns and no reshaping the data is necessary.
This tip is useful when we want to append the group aggregate measures back to the ungrouped data. Here is an example to clarify this:
df['avg_tip_by_gender'] = df.groupby('gender')['tip'].transform('mean')df.head()
In this example, newly created variable avg_tip_by_gender shows average tip by gender. In other words, mean tip by gender from below have been added back to the ungrouped data.
df.groupby('gender')['tip'].agg(['mean', 'std'])
Now, let’s take a slightly more advanced example:
df['n_sd_from_gender_avg_tip'] = df.groupby('gender')['tip'].transform(lambda x: (x-x.mean())/x.std())df.head()
Here, using lambda function, we did 3 things for variable tip:
x.mean(): Find mean by gender.x-x.mean(): Find the distance from mean by gender.(x-x.mean())/x.std(): Find the distance in units of standard deviation.
Let’s take the first record (index=0) as an example and round up the numbers to 2 decimal places for simplicity: x=1.01, x.mean()=2.83, x.std()= 1.16
Then, n_sd_from_gender_avg_tip = (1.01 - 2.83)/ 1.16 = -1.57
That’s what we find in the first row of n_sd_from_gender_avg_tip. For this record, the tip amount was about 1.57 standard deviations lower than the average tip among female customers.
Voila❕ These were my current top data aggregation tips in pandas!
Would you like to access more content like this? Medium members get unlimited access to any articles on Medium. If you become a member using my referral link, a portion of your membership fee will directly go to support me.
Thank you for reading my post. Hope you find my tips useful ✂️. If you are interested to learn more about pandas, here are links to my other posts:◼️️ Writing 5 common SQL queries in pandas◼️️ 5 tips for pandas users◼️️ How to transform variables in a pandas DataFrame
Bye for now 🏃💨