Mastering Pandas Groupby. Understanding the Groupby Method | by Sadrach Pierre, Ph.D. | Towards Data Science

Pandas is a python library that provides tools for data transformation and statistical analysis. The ‘groupby’ method in pandas allows us to group large amounts of data and perform operations on these groups. In this post, we will discuss how to use the ‘groupby’ method in Pandas. For our purposes we will be using the WorldWide Corona Virus Dataset which can be found here.
Let’s get started!
First, let’s read the file called “2019-nCoV-cases-JHU.csv” into a Pandas data frame:
import pandas as pd df = pd.read_csv("2019-nCoV-cases-JHU.csv")
Next, let’s print the first five rows of data:
print(df.head())
To make things simple, let’s limit our data to only include records in the US:
df = df[df['Region'] == 'US'] #filter to only include US datadf.reset_index(inplace = True) #reset indexdel df['index'] # remove old indexprint(df.head())
We probably don’t need to consider the ‘Unassigned Location (From Diamond Princess)’ value of ‘Province’, so let’s also take that out as well:
df = df[df['Province'] != 'Unassigned Location (From Diamond Princess)']print(df.head())
Now, let’s also create a ‘State’ column by pulling the last two characters of the string values in the ‘Province’ column:
df['State'] = df['Province'].str[-2:]
We can do something similar to create a ‘County’ column. Let’s extract all characters of the string values in the ‘Province’ column excluding the last 4 characters:
df['County'] = df['Province'].str[:-4]print(df.head())
Next, let’s convert the date column into a Pandas ‘datetime’ :
df['Date'] = pd.to_datetime(df['Date'], format ='%m/%d/%Y %H:%M')
We’re now in a good position to perform some interesting ‘groupby’ operations. Let’s generate some statistics corresponding to the numerical column values, like deaths, confirmed cases, and recoveries, at the state level.
To do this, let’s first perform a ‘groupby’ on the state level:
gb = df.groupby('State')
Here, we’ve created a ‘groupby’’ object. The next thing we can do is pull a specific group. We can do this using the ‘get_group’ method. Let’s get the group corresponding to CA and set the index to be the date:
df_ca = gb.get_group('CA').set_index('Date')print(df_ca.head())
We can also iterate over the ‘groupby’ object and create a data frame containing a column for each state:
deaths = pd.DataFrame()for name, group in df.groupby('State'):    if deaths.empty:        deaths = group.set_index('Date')[["Deaths"]].rename(columns={"Deaths": name})    else:        deaths = deaths.join(group.set_index('Date')[["Deaths"]].rename(columns={"Deaths": name}))
Let’s print the resulting data frame containing the grouped death statistics:
print(deaths.head())
Notice that this data contains a limited number of states. We can also use the describe method to get basic statics about our new data:
death_stats = deaths.describe()death_stats = death_stats.astype(int)print(death_stats)
Here, the count corresponds to the number of rows. We also have the mean, standard deviation, percentile, minimum, and maximum values for the number of deaths. Note that the newest record in this data corresponds to March 3, 2020, which was before the widespread outbreak in cases. We can define a function that automatically generates these statistics for a given numerical column:
def get_statistics(column_name):    column_df = pd.DataFrame()        for name, group in df.groupby('State'):        if column_df.empty:            column_df = group.set_index('Date')[[column_name]].rename(columns={column_name: name})        else:            column_df = column_df.join(group.set_index('Date')[[column_name]].rename(columns={column_name: name}))    column_df.fillna(0, inplace = True)            column_stats = column_df.describe()    column_stats = column_stats.astype(int)    print(column_stats)
Let’s call this function with the ‘Confirmed’ column as input:
get_statistics("Confirmed")
And recovered:
get_statistics("Recovered")
We can also set the index to be the day number. Let’s create a day column and generate grouped statistics for the ‘Confirmed’ column using the day number as the index:
df['Day'] = df['Date'].dt.daydef get_statistics_day(column_name):    column_df = pd.DataFrame()        for name, group in df.groupby('State'):        if column_df.empty:            column_df = group.set_index('Day')[[column_name]].rename(columns={column_name: name})        else:            column_df = column_df.join(group.set_index('Day')[[column_name]].rename(columns={column_name: name}))    column_df.fillna(0, inplace = True)            column_stats = column_df.describe()    column_stats = column_stats.astype(int)    print(column_stats)get_statistics_day("Confirmed")
These are statistics corresponding to daily confirmed cases across states. I’ll stop here but I encourage you to play around with the code and data. For example, you can try generating these statistics at the county level. All you would need to do is change ‘groupby(‘State’)’ to ‘groupby(‘County’)’.
To summarize, in this post we discussed how to use the Pandas ‘groupby’ method. We generated grouped statistics for US confirmed cases, recoveries and deaths caused by the Corona Virus. We generated these statistics using the date time values as well as day number values as indices. I hope you found this post useful/interesting. The code from this post is available on GitHub. Thank you for reading!