Pandas’ groupby explained in detail | by Fabian Bosler | Towards Data Science

Writing articles about Pandas is the best. The part I love most about it is when I get to search the interwebs for cute panda pictures. Cmon, how can you not love panda bears? And groups of pandas, even better! Anyway, I digress ...
Pandas’ groupby is undoubtedly one of the most powerful functionalities that Pandas brings to the table. However, most users only utilize a fraction of the capabilities of groupby.
Groupby allows adopting a split-apply-combine approach to a data set. This approach is often used to slice and dice data in such a way that a data analyst can answer a specific question.
Split the data based on column(s)/condition(s) into groups;Apply a function/transformation to all the groups and combine the results into an output
Split the data based on column(s)/condition(s) into groups;
Apply a function/transformation to all the groups and combine the results into an output
Loading the DataGroupby — Splitting the DataApply and Combine — apply , agg(regate) , transform and filter
Loading the Data
Groupby — Splitting the Data
Apply and Combine — apply , agg(regate) , transform and filter
You can find the full Jupyter Notebook here. But I urge you to go through the steps yourself. After all, practice makes perfect.
We are going to use data from a hypothetical sales division. The data set consists, among other columns, of fictitious sales reps, order leads, the company the deal might close with, order values, and the date of the lead.
order_leads = pd.read_csv(    'https://raw.githubusercontent.com/FBosler/Medium-Data-Exploration/master/order_leads.csv',    parse_dates = [3])sales_team = pd.read_csv(    'https://raw.githubusercontent.com/FBosler/Medium-Data-Exploration/master/sales_team.csv',    parse_dates = [3])df = pd.merge(  order_leads,  sales_team,  on=['Company Id','Company Name'])df = df.rename(  columns={'Order Value':'Val','Converted':'Sale'})
The default approach of calling groupby is by explicitly providing a column name to split the dataset by. However, and this is less known, you can also pass a Series to groupby. The only restriction is that the series has the same length as the DataFrame.Being able to pass a series means that you can group by a processed version of a column, without having to create a new helper column for that.
First, let’s create a grouped DataFrame, i.e., split the dataset up.
IN:grouped = df.groupby('Sales Rep')groupedOUT:<pandas.core.groupby.generic.DataFrameGroupBy object at 0x12464a160>IN:type(grouped)OUT:pandas.core.groupby.generic.DataFrameGroupBy
We have now created a DataFrameGroupBy object. Let’s further investigate:
Calling groups on the grouped object returns the list of indices for every group (as every row can be uniquely identified via its index)
IN:grouped.groupsOUT:{'Aaron Hendrickson': Int64Index([25612, 25613, 25614, 25615, 25616, 25617, 25618, 25619, 25620, 25621,..., 25894, 25895, 25896, 25897, 25898, 25899, 25900, 25901, 25902, 25903], dtype='int64', length=292),'Adam Sawyer': Int64Index([67140, 67141, 67142, 67143, 67144, 67145, 67146, 67147, 67148, 67149, ..., 67454, 67455, 67456, 67457, 67458, 67459, 67460, 67461, 67462, 67463], dtype='int64', length=324),...'Yvonne Lindsey': Int64Index([20384, 20385, 20386, 20387, 20388, 20389, 20390, 20391, 20392, 20393, 20394, 20395, 20396, 20397, 20398, 20399, 20400, 20401, ... , 20447, 20448, 20449, 20450], dtype='int64', length=67)}
By calling get_group with the name of the group, we can return the respective subset of the data.
grouped.get_group('Aaron Hendrickson')
To demonstrate some advanced grouping functionalities, we will use the simplest version of the apply step (and count the rows in each group) via the size method. We do this so that we can focus on the groupby operations.
We will go into much more detail regarding the apply methods in section 2 of the article.
IN:grouped.size()OUT:Sales RepAaron Hendrickson    292Adam Sawyer          324Adele Kimmel         115Adrian Daugherty     369Adrianna Shelton      37                    ... Willie Lin            44Willie Rau            95Willie Sanchez       309Yvonne Jones          74Yvonne Lindsey        67Length: 499, dtype: int64
The following is the first example where we group by a variation of one of the existing columns. I find this is a vast improvement over creating helper columns all the time. It just keeps the data cleaner.
In this example, we use a string accessor to retrieve the first name. You can read up on accessors here.
IN:df.groupby(  df['Sales Rep'].str.split(' ').str[0]).size()OUT:Sales RepAaron        292Adam         324Adele        115Adrian       369Adrianna      37            ... Wesley       144Wilbert      213William     1393 # Plenty of WilliamsWillie       448Yvonne       141Length: 318, dtype: int64
We saw that there seem to be a lot of Williams, lets group all sales reps who have William in their name together.
IN:df.groupby(  df['Sales Rep'].apply(lambda x: 'William' in x)).size()OUT:Sales RepFalse    97111True      2889dtype: int64
This example is — admittedly — silly, but it illustrates the point that you can group by arbitrary series quite well.
IN:df.groupby(    pd.Series(np.random.choice(list('ABCDG'),len(df)))).size()OUT:A    19895B    20114C    19894D    20108G    19989dtype: int64
In the following example, we apply qcut to a numerical column first. qcut allocates the data equally into a fixed number of bins.
IN:df.groupby(  pd.qcut(    x=df['Val'],    q=3,    labels=['low','mid','high']  )).size()OUT:Vallow     33339mid     33336high    33325dtype: int64
Like in the previous example, we allocate the data to buckets. This time, however, we also specify the bin boundaries.
IN:df.groupby(  pd.cut(    df['Val'],    [0,3000,5000,7000,10000]  )).size()OUT:Val(0, 3000]        29220(3000, 5000]     19892(5000, 7000]     20359(7000, 10000]    30529dtype: int64
pd.Grouper is important! This one took me way too long to learn, as it is incredibly helpful when working with time-series data.
In the following example, we are going to use pd.Grouper(key=<INPUT COLUMN>, freq=<DESIRED FREQUENCY>) to group our data based on the specified frequency for the specified column. In our case, the frequency is 'Y' and the relevant column is 'Date'.
IN:df.groupby(  pd.Grouper(    key='Date',    freq='Y'  )).size()OUT:Date2014-12-31    199562015-12-31    200542016-12-31    201332017-12-31    200792018-12-31    19778Freq: A-DEC, dtype: int64
Instead of 'Y' we can use different standard frequencies like 'D','W','M', or 'Q'. For a list of less common usable frequencies, check out the documentation.I found'SM' for semi-month end frequency (15th and end of the month) to be an interesting one.
IN:df.groupby(pd.Grouper(key='Date',freq='Q')).size()OUT:Date2014-03-31    49492014-06-30    49482014-09-30    49862014-12-31    50732015-03-31    49582015-06-30    49682015-09-30    51092015-12-31    50192016-03-31    50642016-06-30    48922016-09-30    51482016-12-31    50292017-03-31    49592017-06-30    51022017-09-30    50772017-12-31    49412018-03-31    48892018-06-30    49392018-09-30    49752018-12-31    4975Freq: Q-DEC, dtype: int64
So far, we have only grouped by one column or transformation. The same logic applies when we want to group by multiple columns or transformations. All we have to do is to pass a list to groupby.
IN:df.groupby(['Sales Rep','Company Name']).size()OUT:Sales Rep          Company Name               Aaron Hendrickson  6-Foot Homosexuals             20                   63D House'S                    27                   Angular Liberalism             28                   Boon Blish'S                   18                   Business-Like Structures       21                                                  ..Yvonne Jones       Entry-Limiting Westinghouse    20                   Intractable Fairgoers          18                   Smarter Java                   17Yvonne Lindsey     Meretricious Fabrication       28                   Shrill Co-Op                   39Length: 4619, dtype: int64
Random names, I swear!
In the previous section, we discussed how to group the data based on various conditions. This section deals with the available functions that we can apply to the groups before combining them to a final result.
🅐 apply,🅑 agg(regate),🅒 transform, and🅓 filter
If you are anything like me when I started using groupby, you are probably using a combination of 🅐 and 🅑 along the lines of:
grouped = df.groupby('GROUP') and then:- group.apply(mean)- group.agg(mean)- group['INTERSTING COLUMN'].apply(mean)- group.agg({'INTERSTING COLUMN':mean})- group.mean()
Where mean could also be another function.
The good news: All of them work. And most of the time, the result is approximately going to be what you expected it to be.
The bad news: There are nuances to apply and agg that are worthwhile delving into.
Additionally, but much more importantly two lesser-known powerful functions can be used on a grouped object, filter and transform.
Apply is somewhat confusing, as we often talk about applying functions while there also is an apply function. But bear with me. The apply function applies a function along an axis of the DataFrame. The application could be either column-wise or row-wise.apply is not strictly speaking a function that can only be used in the context of groupby. You can also use apply on a full dataframe, like in the following example (where we use the _ as a throw-away variable).
_ = pd.DataFrame(    np.random.random((2,6)),    columns=list('ABCDEF'))_
IN:_.apply(sum, axis=0) # axis=0 is default, so you could drop itOUT:A    0.620289B    0.818850C    0.672706D    1.269064E    1.156606F    0.934941dtype: float64IN:_.apply(sum, axis=1)OUT:0    2.8681451    2.604311dtype: float64
But apply can also be used in a groupby context. Which makes sense, because each group is a smaller DataFrame in its own right. Keep in mind that the function will be applied to the entire DataFrame. Applying the function to the whole DataFrame means typically that you want to select the columns you are applying a function to. We will leave it at the following two examples and instead focus on agg(regation) which is the “intended” way of aggregating groups.
IN:df.groupby(    pd.Grouper(key='Date',freq='Y'))['Sale'].apply(sum)OUT:Date2014-12-31    36812015-12-31    38002016-12-31    38812017-12-31    30682018-12-31    2478Freq: A-DEC, Name: Sale, dtype: int64IN:df.groupby(    pd.Grouper(key='Date',freq='Y'))['Val','Sale'].apply(sum)OUT:Date       Val       Sale2014-12-31 100422394 36812015-12-31 101724648 38002016-12-31 101789642 38812017-12-31 101957784 30682018-12-31 100399962 2478
Please note that agg and aggregate can be used interchangeably. agg is shorter, so this is what I will be using going forward.
Aggregate is by and large the most powerful of the bunch. Let’s dissect above image and primarily focus on the righthand part of the process. The following code snippet creates a larger version of the above image.
df.groupby('Sales Rep').agg({     'Order Id':'size',    'Val':['sum','mean'],    'Sale':['sum','mean']})
We pass a dictionary to the aggregation function, where the keys (i.e. Order Id, Val, Sale) are the columns and the values ('size', ['sum','mean'], ['sum','mean']) are the functions to be applied to the respective columns.
Note that the functions can either be a single function or a list of functions (where then all of them will be applied). Also, note that agg can work with function names (i.e., strings) or actual function (i.e., Python objects). A non-exhaustive list of functions can be found here. The ones I use most frequently are:
'size': Counts the rows
'sum': Sums the column up
'mean'/'median': Mean/Median of the column
'max'/'min': Maximum/Minimum of the column
'idxmax'/'idxmin': Index of the maximum/minimum of the column. Getting the index of the minimal or maximal value is helpful for mapping other columns, i.e., what's the name of the company per sales rep with the biggest deal
pd.Series.nunique: Counts unique values. Note that, unlike the previous functions, this is an actual function and not a string.
Now, One problem, when applying multiple aggregation functions to multiple columns this way, is that the result gets a bit messy, and there is no control over the column names. In the past, I often found myself aggregating a DataFrame only to rename the results directly afterward. I always found that a bit inefficient.
Situations like this are where pd.NamedAgg comes in handy. pd.NamedAgg was introduced in Pandas version 0.25 and allows to specify the name of the target column.
def cr(x):    return round(np.mean(x),2)# Long Form: Explictly specifying the NamedAggaggregation = {    'Potential Sales': pd.NamedAgg(column='Val', aggfunc='size'),    'Sales': pd.NamedAgg(column='Sale', aggfunc='sum'),    'Conversion Rate': pd.NamedAgg(column='Sale', aggfunc=cr)}# Alternative: Since the NamedAgg is just a tuple, we can also pass regular tuplesaggregation = {    'Potential Sales': ('Val','size'),    'Sales': ('Sale','sum'),    'Conversion Rate': ('Sale',cr)}df.groupby('Sales Rep').agg(**aggregation)
running the above snippet results in:
While agg returns a reduced version of the input, transform returns an on a group-level transformed version of the full data. The new output data has the same length as the input data. For users coming from SQL, think of transform as a window function.
A typical example is to get the percentage of the groups total by dividing by the group-wise sum.
IN:df.groupby('Sales Rep')['Val'].transform(lambda x: x/sum(x))OUT:0        0.0049911        0.0056932        0.0039763        0.0007994        0.003300           ...   99995    0.01208899996    0.00071199997    0.01374199998    0.01069599999    0.001533Name: Val, Length: 100000, dtype: float64
Unlike agg, transform is typically used by assigning the results to a new column. In our above example, we could do:
df['%'] = df.groupby('Sales Rep')['Val'].transform(  lambda x: x/sum(x))
Check out this article to learn how to use transform to get rid of missing values for example.
towardsdatascience.com
Filter, as the name suggests, does not change the data in any capacity, but instead selects a subset of the data. For users coming from SQL, think of filter as the HAVING condition.
We could for example filter for all sales reps who have at least made 200k
IN:df.groupby('Sales Rep').filter(  lambda x: (x['Val'] * x['Sale']).sum() > 200000)
Or all sales Reps with a conversion rate of > 30%:
IN:# Let's add this for verificationdf['cr'] = df.groupby('Sales Rep')['Sale'].transform('mean')df.groupby('Sales Rep').filter(lambda x: x['Sale'].mean() > .3)
In this article, you learned how to group DataFrames like a real Pandas pro. You learned a plethora of ways to group your data. You learned to differentiate between apply and agg. You learned and applied the most common aggregation functions. You have seen the less commonly used transform and filter put to good use.
Please connect on LinkedIn if you want to have a chat! I’d love to have a conversation or answer any questions that you might have.
Also, check out the other articles I wrote on Medium
Good job, and thanks for reading!