COVID-19 data processing with Pandas DataFrame | by B. Chen | Towards Data Science

I published an article a couple days ago showing a step by step tutorial to building an Interactive Data Visualization for exploring Coronavirus Spreads. Several people left private notes on it asking specifically for the data processing steps.
Today, I am posting the step by step tutorial to download, load, merge, clean and aggregate the COVID-19 time series data. The data was provided by the Johns Hopkins University Center for Systems Science and Engineering (JHU CSSE) who shared their data on a public Github page. The Johns Hopkins CSSE aggregates data from primary sources, such as the World Health Organisation, national, and regional public health institutions. This data is made freely available and is updated every day.
Before we getting into the details, I would like to show you the final output of the COVID-19 data processing. Confirmed, Deaths, Recovered and Active are cumulative data. New cases, New deaths and New Recovered are day wise data. This DataFrames is ordered by Date and Country/Region.
Also, I would like to thank the following projects for their COVID-19 data processing ideas:
https://github.com/imdevskp/covid_19_jhu_data_web_scrap_and_cleaning
https://www.kaggle.com/imdevskp/covid-19-analysis-visualization-comparisons/notebook
https://www.kaggle.com/neelkudu28/covid-19-visualizations-predictions-forecasting
The main reason behind data processing is that data almost never comes in a form that is ready for us. And in my personal experience, a large amount of time spent on a data science project is on manipulating data.
Let’s take a look at the COVID-19 time series data,
Here are some problems for me:
confirmed, deaths and recovered are kept in different CSV files. That makes my life difficult for plotting them in the same data visualization.
Take a look at the above screenshot. Dates are actually displayed as column name and that is very odd data shape.
confirmed, deaths and recovered are not exactly matched because of missing data, and some countries don’t report Province/State-wise stat for recovered.
Other missing values, wrong datatypes and cases reported from three cruise ships.
This COVID-19 data processing tutorial runs the following steps:
Download raw CSV dataset from JHU CSSE public Github pageLoad raw CSV dataset and extract the common date listMerges the raw confirmed, deaths, and recovered CSV data into one DataFrame.Performs data cleanings due to missing values, wrong datatypes and cases from cruise ships.Data Aggregation: Add an active case column Active, which is calculated by active_case = confirmed — deaths — recovered. Aggregate data into Country/Region wise and group them by Date and Country/Region. After that, add day wise New cases, New deaths and New recovered by deducting the corresponding cumulative data on the previous day.
Download raw CSV dataset from JHU CSSE public Github page
Load raw CSV dataset and extract the common date list
Merges the raw confirmed, deaths, and recovered CSV data into one DataFrame.
Performs data cleanings due to missing values, wrong datatypes and cases from cruise ships.
Data Aggregation: Add an active case column Active, which is calculated by active_case = confirmed — deaths — recovered. Aggregate data into Country/Region wise and group them by Date and Country/Region. After that, add day wise New cases, New deaths and New recovered by deducting the corresponding cumulative data on the previous day.
At the end of the data processing, I will show 2 simple data visualizations with Altair for demonstration.
Let’s import the libraries we need
import pandas as pdimport wget
Go to Johns Hopkins CSSE public Github page and navigate to csse_covid_19_data/css_covid_19_time_series/
Click the global CSV dataset and click “Raw” button to get the data url.
Then, keep the urls in a collection and download using wget.download()
# url of the raw csv dataseturls = [    'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv',    'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv',    'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv'][wget.download(url) for url in urls]
Once download complete, it should output like below and the 3 files should appear in your working directory.
['time_series_covid19_confirmed_global.csv', 'time_series_covid19_deaths_global.csv', 'time_series_covid19_recovered_global.csv']
confirmed_df = pd    .read_csv('time_series_covid19_confirmed_global.csv')deaths_df = pd    .read_csv('time_series_covid19_deaths_global.csv')recovered_df = pd    .read_csv('time_series_covid19_recovered_global.csv')
Let’s have a quick look at data, for example, confirmed_df.head() shows 93 columns. It should be the same on deaths_df and recovered_df
By running confirmed_df.columns , deaths_df.columns and recovered_df.columns, all of them should output the same result as below:
Notice that columns are all date from the 4th column onwards and to get the list of dates confirmed_df.columns[4:]
Before merging, we need to use melt() to unpivot DataFrames from current wide format into long format. In other words, we are kinda transposing all date columns into values. Here are the main settings for that:
Use ‘Province/State’, ‘Country/Region’, ‘Lat’, ‘Long’ as identifier variables. We will later use them for merging.
Unpivot date columns (As we saw previously columns[4:]) with variable column ‘Date’ and value column ‘Confirmed’
dates = confirmed_df.columns[4:]confirmed_df_long = confirmed_df.melt(    id_vars=['Province/State', 'Country/Region', 'Lat', 'Long'],     value_vars=dates,     var_name='Date',     value_name='Confirmed')deaths_df_long = deaths_df.melt(    id_vars=['Province/State', 'Country/Region', 'Lat', 'Long'],     value_vars=dates,     var_name='Date',     value_name='Deaths')recovered_df_long = recovered_df.melt(    id_vars=['Province/State', 'Country/Region', 'Lat', 'Long'],     value_vars=dates,     var_name='Date',     value_name='Recovered')
Above should return new long DataFrames. All of them are ordered by Date and Country/Region because raw data was already ordered by Country/Region and the date columns are already in ASC order.
Here is the example of confirmed_df_long
In addition, we have to remove recovered data for Canada due to mismatch issue (🤷‍♂ Canada recovered data is counted by Country-wise rather than Province/State-wise).
recovered_df_long = recovered_df_long[recovered_df_long['Country/Region']!='Canada']
After that, we use merge() to merge the 3 DataFrames one after another
# Merging confirmed_df_long and deaths_df_longfull_table = confirmed_df_long.merge(  right=deaths_df_long,   how='left',  on=['Province/State', 'Country/Region', 'Date', 'Lat', 'Long'])# Merging full_table and recovered_df_longfull_table = full_table.merge(  right=recovered_df_long,   how='left',  on=['Province/State', 'Country/Region', 'Date', 'Lat', 'Long'])
Now, we should get a full table with Confirmed, Deaths and Recovered columns
There are 3 tasks we would like to do
Converting Date from string to datetimeReplacing missing value NaNCoronavirus cases reported from 3 cruise ships should be treated differently
Converting Date from string to datetime
Replacing missing value NaN
Coronavirus cases reported from 3 cruise ships should be treated differently
You probably already notice that the values in the new Date column are all string with m/dd/yy format. To convert Date values from string to datetime, let’s use DataFrame.to_datetime()
full_table['Date'] = pd.to_datetime(full_table['Date'])
Missing values NaN can be detected by running full_table.isna().sum()
full_table.isna().sum()Province/State    16198Country/Region        0Lat                   0Long                  0Date                  0Confirmed             0Deaths                0Recovered          1602dtype: int64
We found a lot NaN in Province/State, and that makes sense as many countries only report the Country-wise data. However, there are 1,602 NaNs in Recovered and let’s replace them with 0.
full_table['Recovered'] = full_table['Recovered'].fillna(0)
Apart from missing values, there are coronavirus cases reported from 3 cruise ships: Grand Princess, Diamond Princess and MS Zaandam. These data need to be extracted and treated differently due to Province/State and Country/Region mismatch over time. Here is what I was talking about:
And here is how we extract the ship data.
ship_rows = full_table['Province/State'].str.contains('Grand Princess') | full_table['Province/State'].str.contains('Diamond Princess') | full_table['Country/Region'].str.contains('Diamond Princess') | full_table['Country/Region'].str.contains('MS Zaandam')full_ship = full_table[ship_rows]
And to get rid of ship data from full_table :
full_table = full_table[~(ship_rows)]
5. Data Aggregation
So far, all the Confirmed, Deaths, Recovered are existing data from raw CSV dataset. Let’s add an active cases column Active, which is calculated by active = confirmed — deaths — recovered .
# Active Case = confirmed - deaths - recoveredfull_table['Active'] = full_table['Confirmed'] - full_table['Deaths'] - full_table['Recovered']
And here is what full_table looks like now.
Next, let’s aggregate data into Country/Region wise and group them by Date and Country/Region.
full_grouped = full_table.groupby(['Date', 'Country/Region'])['Confirmed', 'Deaths', 'Recovered', 'Active'].sum().reset_index()
sum() is to get the total count of ‘Confirmed’, ‘Deaths’, ‘Recovered’, ‘Active’ for the given Date and Country/Region.
reset_index() reset the index and use the default one, which is Date and Country/Region.
And here is what full_grouped looks like now
Now let’s add day wise New cases, New deaths and New recovered by deducting the corresponding accumulative data on the previous day.
# new cases temp = full_grouped.groupby(['Country/Region', 'Date', ])['Confirmed', 'Deaths', 'Recovered']temp = temp.sum().diff().reset_index()mask = temp['Country/Region'] != temp['Country/Region'].shift(1)temp.loc[mask, 'Confirmed'] = np.nantemp.loc[mask, 'Deaths'] = np.nantemp.loc[mask, 'Recovered'] = np.nan# renaming columnstemp.columns = ['Country/Region', 'Date', 'New cases', 'New deaths', 'New recovered']# merging new valuesfull_grouped = pd.merge(full_grouped, temp, on=['Country/Region', 'Date'])# filling na with 0full_grouped = full_grouped.fillna(0)# fixing data typescols = ['New cases', 'New deaths', 'New recovered']full_grouped[cols] = full_grouped[cols].astype('int')# full_grouped['New cases'] = full_grouped['New cases'].apply(lambda x: 0 if x<0 else x)
And finally here is the full_grouped. Be aware of that this final output is Country-wise data with
Confirmed, Deaths, Recovered and Active are cumulative data.
New cases, New deaths and New Recovered are day wise data.
This DataFrames is ordered by Date and Country/Region.
Finally, you can save this final data to a CSV file:
full_grouped.to_csv('COVID-19-time-series-clean-complete.csv')
For simplicity, let’s use Python Data Visualization library Altair to create some simple visualizations. Altair is a declarative statistical visualization library for Python, based on Vega and Vega-Lite. Altair offers a powerful and concise visualization grammar that enables you to build a wide range of statistical visualizations quickly.
For Altair installation, I would strongly recommend creating a new Virtual Environment because Altair has a lot of dependencies. Then, activate your Virtual Environment, and run pip install altair vega_datasets to install Altair along with example dataset in vega_datasets.
For tutorials on Virtual Environment, you can take a look:
towardsdatascience.com
medium.com
First, let’s import the libraries, load data and select a country, for example, United Kingdom:
import pandas as pdimport altair as altfull_grouped = pd.read_csv('COVID-19-time-series-clean-complete.csv', parse_dates=['Date'])uk = full_grouped[full_grouped['Country/Region'] == 'United Kingdom']
Then, let’s create a base chart with the common elements
base = alt.Chart(uk).mark_bar().encode(    x='monthdate(Date):O',).properties(    width=500)
After that, we can concatenate horizontally using the | operator
First, let’s import the libraries, load data and select a list of countries
import pandas as pdimport altair as altfull_grouped = pd.read_csv('COVID-19-time-series-clean-complete.csv', parse_dates=['Date'])countries = ['US', 'Italy', 'China', 'Spain', 'Germany', 'France', 'Iran', 'United Kingdom', 'Switzerland']selected_countries = full_grouped[full_grouped['Country/Region'].isin(countries)]
Let’s create a circle chart to display the day wise New cases,
alt.Chart(selected_countries).mark_circle().encode(    x='monthdate(Date):O',    y='Country/Region',    color='Country/Region',    size=alt.Size('New cases:Q',        scale=alt.Scale(range=[0, 1000]),        legend=alt.Legend(title='Daily new cases')    ) ).properties(    width=800,    height=300)
And here is the output
This article is mainly to show everyone how to process COVID-19 data step by step using Pandas. There are only 2 simple examples in the Data Exploration section for demonstration.
If you are interested in learning Altair, you can take a look at the following two articles.
towardsdatascience.com
towardsdatascience.com
And that’s about it. Thanks for reading