Identifying potential factors that correlate with the trend of COVID-19 cases in Pakistan.
By: Huzaifah Nadeem
Tags: Data Science, Coronavirus, Covid 19, Pandemic, Pakistan

Identifying potential factors that correlate with the trend of COVID-19 cases in Pakistan. Huzaifah Nadeem Mar 22·13 min read

Ever since the first case of COVID-19 in Pakistan, there has been much uncertainty about the government policies. The public criticized the government at the start of the pandemic; however, people praised the government for implementing “smart” policies as the first wave resolved. However, at the same time, the public did not believe that the government’s response was effective because people rarely seem to follow the government directives. Furthermore, as the second and third waves came about, people seemed apprehensive about the government’s response because they were uncertain about their effectiveness during the first wave. Popular culture fantasized the idea that Pakistan was lucky to have its first wave resolved quickly and easily.

Hence, we decided to look directly at the data to figure out “What are the potential factors that correlate with the trend of COVID-19 cases in Pakistan?”. To answer this question, we devised three sub-questions that will help us answer this question better:

How effective are the government policies in slowing down COVID-19?

What was the public response to the government policies?

How representative is the number of confirmed cases of the scale of the outbreak?

To answer these questions, we analyzed the following datasets:

Google’s COVID-19 Community Mobility Reports: Provide insights into how the Pakistani community has reacted in response to policies aimed at combating COVID-19

A cross-country database of COVID-19 testing: To understand how reflective the number of confirmed cases has been of the outbreak in Pakistan

JHU CSSE COVID-19 Data : Detailed information about how the pandemic is progressing in all the countries regarding the number of cases, deaths, etc.

Oxford COVID-19 Government Response Tracker: The dataset tracks 19 metrics to measure world-wide government responses. A subset of this can be used to understand the government response to the outbreak in Pakistan

Data Cleaning and Exploratory Data Analysis

Oxford COVID-19 Government Response Tracker

The dataset contains 19 variables measuring a government’s response to the pandemic. Detailed description of the metrics used is available here.

The dataset contains data for most of the world’s countries; however, since our study is only concerned with Pakistan, we extracted all the rows for Pakistan and discarded the others.

Our data was only limited to Pakistan; so, we discarded the columns that identified the row’s country and region. These columns were “CountryName”, “CountryCode”, “RegionName”, and “RegionCode”. We also dropped the “Jurisdiction” column because Pakistan’s subset only contained the total national data, and that is what our study is concerned with anyway. We also dropped “M1_Wildcard” column, a column for data that does not fit in other columns, and this column was empty for Pakistan. We also dropped the index column, which Pandas labeled as “Unnamed: 0” when the .csv was imported into it.

Lastly, the latest data set (as of March 19, 2021) had incomplete data for the last ten days (last ten rows); therefore, we dropped those as well. Therefore, our data is limited to the period before March 9, 2021.

Next, we dealt with the missing values.

First, we noticed that there is a flag column for all the 19 variables, which describes the extent of the policy by two categories “general” (represented as 1) and “targeted” (defined as 0). However, in some places, the cells were empty, which was important data-wise because they are associated with the corresponding policy variable’s value. If the variable is 0, then it could mean either “no measures” or “no data” if the corresponding flag is non-empty or empty, respectively. Hence, these empty cells are essential and were replaced with –1 to differentiate between these and other missing values.

Secondly, there was a missing cell in ‘H6_Facial Coverings’. By looking at the values before and after it, it was clear that it was supposed to be a zero and was left empty. Therefore, we filled it with zero.

In the columns for confirmed cases and confirmed deaths, the data started after February 26, 2020, when the first case was confirmed, and the columns were left empty before that. However, logically they both should have been filled with zeroes for these rows.

Finally, we converted data types to more appropriate ones.

First, date was stored as integer with “YearMonthDay” format. We converted this to datetime object by first converting this column to string and then passing it to the pandas to_datetime function.

Next, during the import of .csv file, all the columns were interpreted as float values when some of them were integers. All the 19 variables, their flags, confirmed cases and deaths were integers while the rest of the columns were floats. So, we converted the required ones to integers.

The result being:

“EconomicSupportIndex” and “EconomicSupportIndexForDisplay” had a data type of float64 but since they only had values of 25,50 and 75, they were better represented as int32.

The shape of the dataset left was: 434 rows x 43 columns.

The columns left in the data set with their data types were:

Temporal understanding of indexes

We wanted to try and understand how the government stringency index and the number of cases/deaths varied across time. Also, we want to understand whether the government had any measurable impact on the number of covid cases. Before modeling, here are some graphs to get some overall insight.

Government Stringency Index versus date

Here, we can notice in this data set that there is a dip around September — October 2021 in the stringency index. This matches our current experience as students because universities were opened during this time. Also, the data seems trustworthy as the line graph matches the main lockdown stringencies at that time — verified by news articles.

We also saw how Confirmed deaths and Confirmed Cases varied during the whole pandemic.

Confirmed deaths and cases during the covid pandemic

Here we run into a problem, the scale for the Confirmed Deaths and Confirmed Cases don’t match. So, we made another plot for the confirmed cases.

Confirmed Deaths during the covid-19 pandemic

Comparing the two graphs above, we can notice here that the Confirmed Cases are heavily correlated with the Confirmed Deaths. This seems logical.

We also looked at how well government has responded to the COVID-19 Pandemic. We can see other interesting correlations between the stringency index and the number of confirmed cases. Interestingly, we found that there is very little correlation between the two.

Heatmap correlation matrix for some of the attributes for covid-19 dat

Google’s COVID-19 Community Mobility Reports

This dataset provides insights into how the community has reacted in response to policies aimed at combating COVID-19. The dataset contains data for many countries; however, we only need the data for Pakistan since our study is concerned with Pakistan only. Therefore, we downloaded the relevant subset of the data from the website.

The dataset had 8224 rows and 15 columns. The data is continuously updated but we used the data collected till 14th March, 2021.

We also looked at the datatypes of each column to make ourselves more familiar with the data.

Looking at the above results, we came to know that there are 6 columns which show the percentage changes while the rest of the columns are concerned with the sub-regions, codes, etc. of Pakistan. The description of each column is given below:

First, looking at the column names, we realized that there is no need for any changes in them. Neither were there any irregular capitalized letters, nor any spaces. Further, since the data was extracted from a large dataset, containing data of many countries, the columns ‘country_region_code’ and ‘country_region’ had only one unique value in our subset of the data. Therefore, we dropped these columns.

Dealing with Missing Values:

In the next step, we looked at the missing values in each column. The results are shown below

country_region_code : 0

sub_region_1: 5122

sub_region_2: 8224

metro_area: 3496

iso_3166_2_code: 5122

census_fips_code: 8224

place_id : 0

date: 0

retail_and_recreation_percent_change_from_baseline: 871

grocery_and_pharmacy_percent_change_from_baseline: 1022

parks_percent_change_from_baseline: 109

transit_stations_percent_change_from_baseline: 857

workplaces_percent_change_from_baseline: 2

residential_percent_change_from_baseline: 738

Looking at the results we can see that the columns ‘sub_region_2’ and ‘census_fips_code’ have all values missing in them. Therefore, we can also drop these two columns.

After dropping the above mentioned 4 columns, the shape of our dataset was 8224 rows and 11 columns.

In the next step, we looked at the missing values in the last 6 columns, which contained data for the percentage changes in different areas. The missing values in these columns represented 0 percentage change, therefore, we replaced all missing values in these columns with 0.

Removing Unnecessary Columns:

After taking the above steps, we were left with missing values in 3 columns, ‘sub_region_1’, ‘metro area’ and ‘iso_3166_2_code’. We noticed the relation between these columns. All 3 columns represented the sub-regions of Pakistan and whenever ‘sub_region_1’ value was missing, ‘iso_3166_2_code’ value was missing as well and ‘metro area’ had values only when the other 2 columns had no values. Therefore, we merged the data for all these columns and introduced a new column ‘region’ which included the region of Pakistan and dropped the rest of the 3 columns. Moreover, there were rows with missing values in all 3 columns but the ‘place id’ and the rest of the values showed that these rows represented nationwide mean values for the percentage changes. Therefore, ‘Pakistan’ was stored in the ‘region’ column.

Finally, there were no missing values in our dataset and no unnecessary columns. The dataset was cleaned and ready to be used for further analysis. The final shape of our dataset was 8224 rows and 9 columns.

Visualizations:

The plot shows us the rate of percentage changes mobility in different regions of Pakistan. So, we can easily compare where the cases were at the peak and the highest percentage rise was in parks.

Percent Change of some of the different regions with respect to Place IDs.

Similarly, we can look at the mean percentage changes in different regions of Pakistan using the following visualization. As an example, the greatest change was in Gilgit Baltistan parks till March 15, 2021.

Percent Change of some of the different regions

A cross-country database of COVID-19 testing

This dataset describes the extent of testing for COVID-19 globally. The dataset includes only the data on tests that detect the presence of the virus (PCR tests and antigen tests). Besides the number of tests conducted, the dataset also includes the positivity rate and test per case ratio. These values are the inverse of each other and give us an important metric for understanding the spread of the virus and adequacy of testing.

Since our concern is only with data relevant to Pakistan, we extracted the rows where the ISO code was “PAK”.

For these rows, the values in all columns identifying the Country/Region were redundant and hence they were dropped. These columns were: ‘ISO code’, ‘Source URL’, ‘Source label’, ‘Notes’, ‘Entity’. Furthermore, the Notes column in the dataset was completely Null except for one row where it contained information on the data collection from that point onwards. This note was extracted to a separate data frame to use if required, and the column was dropped.

The shape of the dataset after dropping all extra rows and columns was: 374 rows x 9 columns.

Next, we catered the missing values in the dataset. The number of missing values for each column were:

Date: 0

Daily change in cumulative total: 28

Cumulative total: 17

Cumulative total per thousand: 17

Daily change in cumulative total per thousand: 28

7-day smoothed daily change: 7

7-day smoothed daily change per thousand: 7

Short-term positive rate: 8

Short-term tests per case: 8

The missing values in the last 4 columns were filled with 0.00 based on the fact that these columns contain the running average of the last 7 days and for the first 7 days, we do not have enough data to compute them. The last missing value in the last two columns was filled with the average for that month of the respective column.

For the remaining columns, the values were filled using the values from the ‘7-day smoothed daily change’ and ‘7-day smoothed daily change per thousand’ columns.

Once the ‘cumulative total’ and ‘cumulative total per thousand’ columns were filled, the ‘daily change in cumulative total’ and ‘daily change in cumulative total per thousand’ columns were dropped as the information held in them was being captured by other columns. Lastly, the ‘Date’ column was made as index, as it provides a natural and logical index for the data and the column names were changed to shorten them.

The final data frame looked like:

The other dataset for this section describes the number of cases and tests per 1 million people for most countries. From this dataset, we only needed the information on number of cases relevant to Pakistan. Therefore, from the dataset we dropped all rows where the country was not Pakistan, and all columns except ‘Daily new confirmed cases of COVID-19 per million people (rolling 7-day average, right-aligned)’ and ‘Day’.

These columns were renamed to shorter and more appropriate names and the ‘Day’ column was made index as it provides a natural and logical index for the data.

The shape of the dataset after cleaning: 388 rows2 x 1 column. The final dataset looks like:

The dataset was then added as a column to the main data frame by merging based on index

The output being:

Analysis:

We first plotted the trend line for the positivity rate.

Positivity percentage rate as a rolling 7-day average against time

The red line plotted at y = 5 signifies the World Health Organization indicator of epidemic control as published in May 2020. The epidemic in a region is said to be in control if the percentage positivity rate falls below 5%. From the plot, we can see that in Pakistan, the plot dips below this line twice, Jul 2020 — mid Oct 2020 and Jan 2020 — mid Feb 2020. This also matches with our government response dataset, during these two time periods, the government became less stringent in their policies as the pandemic seemed to be under control during these time periods. Other than control on the epidemic, the plot also gives information on the spread of the virus. If at a given time, the positivity rate is rising, it can indicate that the spread of the virus is faster than the growth indicated by the number of confirmed cases.

We also plotted two scatterplots of the number of daily cases per million people against time. In one plot, we added the daily tests per million people as hue to show the variance in number of tests conducted with time and with the variance in number of cases. In the second plot, we added the smoothed average tests per case as hue to show the variance in the ratio of tests to cases with time and the variance in number of cases.

Plot 1:

Daily cases and tests over time

This shows us the extent of testing compared to the scale of the outbreak over time. From the plot, the amount of testing in Pakistan has been high whenever the number of cases were high. This information, however is not enough to say with certainty that the number of cases reported truly depict the scale of the outbreak in Pakistan.

Plot 2:

Daily cases and tests per case over time

This plot shows us the proportion of tests per case compared to the scale of the outbreak over time. From this plot, we can see that in the period of the first spike (May 2020 — June 2020), the number of tests per case is low, this suggests that the outbreak might have been more severe then recorded. In the period following that (Aug 2020 — Oct 2020) when there is a dip in the number of cases, the proportion of tests per case is relatively high, suggesting that the scale of the outbreak recorded in this period is closer to the true scale of the outbreak.