Learn how to master groupby function in Python now | by WY Fok | Towards Data Science

One popular feature in the Pandas package is groupby function. I believe that almost everyone using Pandas before must use groupby function also. It is so popular because it provides summarized but detailed results efficiently. As described by the doc of Pandas package,
By “group by” we are referring to a process involving one or more of the following steps:
Splitting the data into groups based on some criteria.
Applying a function to each group independently.
Combining the results into a data structure. [1]
There is also a groupby function in SQL. Therefore for someone experienced in SQL, learning groupby function in Python is not a difficult thing. But the thing is groupby in Pandas can perform way more analysis than in SQL and this makes groupby in Pandas a common but essential function.
The reason that groupby in Pandas is more powerful is because of the second step “Applying”. In SQL, most actions in “Applying” steps are statistically related, like min, max, count, etc. However, in Pandas, “Applying” can perform much more.
From the doc of Pandas,
In the apply step, we might wish to one of the following:
Aggregation: compute a summary statistic (or statistics) for each group.
Transformation: perform some group-specific computations and return a like-indexed object.
Filtration: discard some groups, according to a group-wise computation that evaluates True or False. [1]
In this article, I will cover some groupby applications. These applications do not only show me the insights from the data but also help me identify my next move on analyzing the data.
Let’s begin.
The data used in this article is “student-por.csv” in “Student Alcohol Consumption” from Kaggle. You can download the data from this link.
# Inputimport pandas as pd data = pd.read_csv('student-por.csv')data.info()# Output<class 'pandas.core.frame.DataFrame'>RangeIndex: 649 entries, 0 to 648Data columns (total 33 columns):school        649 non-null objectsex           649 non-null objectage           649 non-null int64address       649 non-null objectfamsize       649 non-null objectPstatus       649 non-null objectMedu          649 non-null int64Fedu          649 non-null int64Mjob          649 non-null objectFjob          649 non-null objectreason        649 non-null objectguardian      649 non-null objecttraveltime    649 non-null int64studytime     649 non-null int64failures      649 non-null int64schoolsup     649 non-null objectfamsup        649 non-null objectpaid          649 non-null objectactivities    649 non-null objectnursery       649 non-null objecthigher        649 non-null objectinternet      649 non-null objectromantic      649 non-null objectfamrel        649 non-null int64freetime      649 non-null int64goout         649 non-null int64Dalc          649 non-null int64Walc          649 non-null int64health        649 non-null int64absences      649 non-null int64G1            649 non-null int64G2            649 non-null int64G3            649 non-null int64dtypes: int64(16), object(17)memory usage: 167.4+ KB
The first step is rather simple, grouping data based on the value in one or more columns. You only need to specify how to perform the grouping. Most people already know that a column name or a list of column names can be used for grouping. However, you can also pass a function for the grouping. The function will use the index value as the parameter and perform the grouping. For further info, [2]
The object returned from the groupby function is “DataFrameGroupBy object”. After the grouping, you can use groupto see the groupings.
#Groupby one column# Inputdata.groupby('school')# Output<pandas.core.groupby.generic.DataFrameGroupBy object at 0x0000024D28C9DF98># Inputdata.groupby('school').groups# Output{'GP': Int64Index([  0,   1,   2,   3,   4,   5,   6,   7,   8,   9,...413, 414, 415, 416, 417, 418, 419, 420, 421, 422],dtype='int64', length=423), 'MS': Int64Index([423, 424, 425, 426, 427, 428, 429, 430, 431, 432,...639, 640, 641, 642, 643, 644, 645, 646, 647, 648],dtype='int64', length=226)}# Groupby a function # Inputdef grouping(int):    if int%10==0:        return 'Group1'    if int%5==0:        return 'Group2'    return 'Group3'data.groupby(grouping).groups# Output{'Group1': Int64Index([  0,  10,  20,  30,  40,  50,  60,  70,  80,  90, 100, 110, 120, 130, 140, 150, 160, 170, 180, 190, 200, 210, 220, 230, 240, 250, 260, 270, 280, 290, 300, 310, 320, 330, 340, 350, 360, 370, 380, 390, 400, 410, 420, 430, 440, 450, 460, 470, 480, 490, 500, 510, 520, 530, 540, 550, 560, 570, 580, 590, 600, 610, 620, 630, 640], dtype='int64'), 'Group2': Int64Index([  5,  15,  25,  35,  45,  55,  65,  75,  85,  95, 105, 115, 125,135, 145, 155, 165, 175, 185, 195, 205, 215, 225, 235, 245, 255, 265, 275, 285, 295, 305, 315, 325, 335, 345, 355, 365, 375, 385, 395, 405, 415, 425, 435, 445, 455, 465, 475, 485, 495, 505, 515, 525, 535, 545, 555, 565, 575, 585, 595, 605, 615, 625, 635, 645], dtype='int64'), 'Group3': Int64Index([  1,   2,   3,   4,   6,   7,   8,   9,  11,  12,... 637, 638, 639, 641, 642, 643, 644, 646, 647, 648],dtype='int64', length=519)}
In the above example using a function for grouping, the index in the dataframe is the row number. Therefore, a row is assigned to one of three groups based on the row number. And then groupby function groups these three groups.
To get the data for each group, you can then use the get_group()function and input the group name.
#Inputdata.groupby(grouping).get_group('Group1')
Furthermore, if you want to groupby the data by a non-index column, you can use apply and lambda functions together for the grouping.
# Inputdata.groupby(data['famsize'].apply(lambda x: x.startswith('GT'))).get_group(False)
After grouping, you can already apply a number of computation or statistics functions. For example, size()can give you the numbers of rows for each group; sum()returns the sum for all numeric columns for each group. To know more applicable functions, [3]
#Inputdata.groupby(data['famsize'].apply(lambda x: x.startswith('GT'))).size()#OutputfamsizeFalse    192True     457dtype: int64
It is time to cover the important part of groupby. Groupby function in Pandas is way more powerful than in SQL because you can not only use common statistics functions like min(), max(), mean(),... but also lots of more complex functions. You can also apply your defined functions. In the below section, I will explain the three actions separately.
An easy explanation about aggregation is to perform a computation based on previously grouped data. The aggregation is applied to each group and return its respective result.
Some common computations are like sum, min, max, ... You can perform the computation to all numeric columns, or you can also specify a column to perform the computation.
#Input mjob_gp = data.groupby('Mjob')mjob_gp.agg('mean')
#Inputmjob_gp['age'].agg('mean')#OutputMjobat_home     16.955556health      16.312500other       16.802326services    16.661765teacher     16.583333Name: age, dtype: float64
One less commonly known about aggregation is that you can perform multiple computations on one or more columns together. This one is useful if you want to investigate the statistics of a column.
#Input mjob_gp.agg({'age':['mean','max']})
However, one problem is that the columns are in multi-index which is not so good for further processing(and I don't like multi-index personally). Therefore, we can take one step further and include pd.NamedAgg (New from 0.25.0 version of Pandas) to aggfunction. For further information about pd.NamedAgg, [4]
#Inputmjob_gp.agg(avg_age=pd.NamedAgg(column='age', aggfunc='mean'),            min_age=pd.NamedAgg(column='age', aggfunc='min'),            max_age=pd.NamedAgg(column='age', aggfunc='max'))
There are two advantages using pd.NamedAgg. The first advantage is, of course, no multi-index on the columns. And the second advantage is that you can custom define the column names.
Not surprisingly, you can also use custom-defined functions in the aggregation.
#Input def range_function(x):    return max(x) - min(x)mjob_gp.agg(max_age = pd.NamedAgg(column='age',aggfunc='max'),            min_age = pd.NamedAgg(column='age',aggfunc='min'),            range_age = pd.NamedAgg(column='age',aggfunc=range_function),)
Back to the description of transformation,
Transformation: perform some group-specific computations and return a like-indexed object.
The key part is “group-specific computations”. The computations are different across different groups. For example, maxwill give you the maximum of a column for each group, but not across the whole dataset.
Another characteristic of transformation is that the computation applies to each row and returns back the results with the same length of each group, not like aggregation with reduced lengths. If you are familiar with window function in SQL, transformation in Pandas is similar to that. (PS: You still don't know window function in SQL?!?! Check my previous article that helps you understand why window function in SQL is so important)
towardsdatascience.com
#Input mjob_gp['G1','G2','G3'].transform(lambda x: (x-x.mean())/x.std())
In the above example, for each group, the values of G1 to G3 are normalized based on the mean and standard deviation for the respective group.
From the description in Pandas,
Return a copy of a DataFrame excluding elements from groups that do not satisfy the boolean criterion specified by func. [5]
You can use filterfunction to filter out any groups that do not satisfy the criteria.
#Input mjob_gp.filter(lambda x: x['G1'].mean() > 12)
In the above example, only Mjob under “health” and “teacher” are returned because their means of “G1” are larger than 12.
#Input mjob_gp['G1'].mean()#OutputMjobat_home     10.451852health      12.395833other       11.275194services    11.610294teacher     12.555556Name: G1, dtype: float64
Groupby function in Pandas is a handy tool to perform lots of analyses and data transformations. I hope this article helps you understand the power and usefulness of groupby in Pandas. I am sure you now know more about how to use groupby in your usual data analysis. Have fun with Pandas and see you next time.
Bookmark this if you are new to Python (especially if you self-learn Python)
(Part 2) Bookmark this if you are new to Python (especially if you self-learn Python)
How to use Pandas to analyze numeric data?
Web Scrape Twitter by Python Selenium (Part 1)
Web Scrape Twitter by Python Selenium (Part 2)
[1]: Group By: split-apply-combine
[2]: pandas.DataFrame.groupby
[3]: Computations / descriptive stats
[4]: Named aggregation