Pandas GroupBy — take the most from your data
By: Hana Šturlan
Tags: Pandas, Transformation, Python, Covid 19, Analytics

Taking GroupBy in project

For the demonstration, I will be using COVID-19 dataset (since it is the hottest topic nowadays), you can find it here: https://www.kaggle.com/imdevskp/corona-virus-report.

Take a better look at the data :

import pandas as pd

data=pd.read_csv('covid_19_clean_complete.csv')

As you can see, the dataset contains the following data :

Province/State

Country/Region

Latitude and Longitude of Province/State

Date — data is updated each day, and it is tracking the number of confirmed, Death, and Recovered on daily basis

Confirmed, Deaths and Recovered cases for Province/State by day.

The main problem here — analyzing documented cases by Country. You probably ask why? Looking at this data, you can notice that we have recorded instances daily BUT by Province/State. Countries that don’t have provinces or states, like Croatia, for them, we have one new row each day. But let’s take a look at Australia, there is a daily record for each registered province, on that note, for countries with documented province/state variable, we have multiple rows inserted each day.

I hope you get the gest of it and see how easily someone can make a mistake when analyzing this dataset.

Now, we can see groupby in natural light! The plan is to create a new dataset with the following variables :

Country/Region

Confirmed, Deaths and Recovered

So, let’s do this:

#grouping data

data_by_country=data.groupby(['Country/Region'])[['Confirmed','Deaths','Recovered']].sum() #reset index

data_by_country=data_by_country.reset_index() #print out sorted data from max to min by Confirmed cases

data_by_country.sort_values(by='Confirmed',ascending=False)

Recorded cases by Country

This is great! We can now see the sum of all recorded cases by each Country sorted from a maximum number of cases to a minimum. In a previous example, we only took one variable (Number) and summed it by letter, but now we took three variables and summed it by Country. Please be aware, always use the list of variables that you want to transform!

I know what the following question is.

Can we have multiple variables as indexes in this function? Yes, we can!

Let’s group data by Country and Date variables. Basically, for each Country, we want to have recorded cases by Date. Here is how we can do this :

data_by_country_date=data.groupby(['Country/Region','Date'])[['Confirmed','Deaths','Recovered']].sum() data_by_country_date=data_by_country_date.reset_index() data_by_country_date

Data grouped by Country and Date

This is amazing! We have all recorded cases daily for each Country. With this dataset, we can easily create beautiful visualizations and comparisons between countries.

If we want to filter out only data where confirmed cases exist for each Country, just simply filter with :

data_by_country_date[data_by_country_date.Confirmed>0]

You can see for country Afganistan start date is 24–02–2020, not as above 22–02–2020.

Now you have it, explore this pandas function, use it in your projects when you have to transform your dataset. The following code is a demonstration of how I used it on analysis od COVID-19 data.

I know this isn’t any fancy, but I just want to show you it is used in real-life coding.

The following function flow:

Takes the original dataset and name of the Country you want to present — if Country isn’t entered, then takes data globally — and returns recorded cases from the first case until today.

def get_overall_numbers(df,country='all'): df['Date']=df.Date.astype('datetime64') #set Date to datetime if country=='all': #all - takes data globally

subset=df.groupby('Date') [['Confirmed','Deaths','Recovered']].sum()

subset=subset.reset_index() else:

sub=df[df['Country/Region']==country] #take data for given country

subset=sub.groupby(['Country/Region','Date'])[['Confirmed','Deaths','Recovered']].sum()

subset=subset.reset_index()



return subset[subset.Confirmed>0]

Calling the function :

Globally — Country =’all.’

get_overall_numbers(data)

Globally documented cases by Date

2. Country=’Croatia’

get_overall_numbers(data,'Croatia')

Documented cases by Date for Croatia

If you are not number person, let’s take it to the plot :

#Code is now really simple since we have function #importing libraries

import seaborn as sns

import matplotlib.pyplot as plt sns.lineplot(x=get_overall_numbers(data).Date,y=get_overall_numbers(data).Confirmed/1000000)

Conclusion

I hope this is helpful to all of you out there in the python world, new guys, or someone who just wants to learn more. Please, let me know if you use it in your projects. :)

I wanted to write this article as inspiration from my college colleagues when they mentioned the following: they were looking for simple explanations about how they can group the data and how to use it after transformation.

If someone has similar or the same question, I genuinely hope that you found your answer in this article.

Thank you for reading. Stay tuned.

Bye.