Feature Engineering and Integration of COVID-19 Panel Data
By: Yuting Li
Tags: Feature Engineering, United States, Statistics, Socialdistancing, Covid 19

Feature Engineering and Integration of COVID-19 Panel Data

As COVID-19 sweeps through the globe, we watch intently the climbing number of new cases and its impact on individuals, families, and countries. There is a tremendous influx of information on COVID-19 and significant analytics efforts in understanding these data. Currently, the Kaggle community features two COVID-19 competitions: forecasting analytics on the epidemiological data and NLP studies of key questions in the scientific literature. Both of these approaches make use of data from a single domain.

It is my belief that to gain insights into the COVID-19 situation, we also need to look at the interaction of data from multiple domains. Particularly, I want to answer the following questions:

There are so many factors influencing the spread of the pandemic. Which of them is important? How to integrate COVID19-related data from different domains?

Our team (Yuting Li, Mandy Gu, Han Lu, Iris Wan) takes a shot at this in the COVID-19 Challenge at the UC Davis MSBA. Together, we integrate epidemiological and socioeconomic data in the U.S. and perform feature engineering. Then, we ran a linear regression to investigate which factors are influential in the spread of COVID-19.

Background

We focus on situations in the U.S., as it has the most confirmed cases in the world and the spread has not lost its momentum. To this date, COVID-19 has swept across all states in the country and takes the hardest toll on New York.

Last updated 20 Apr 2:25am EST Source: Johns Hopkins CSSE *Note: The CSSE states that its numbers rely upon publicly available data from multiple sources.

From the above visualization, we also observe the disparate spread of the pandemic across different states. Specifically, let us zoom in on the comparison between the spread in New York, Washington and California.

Thousands of confirmed cases per million in New York and California

We see that the spread of the virus in New York drastically outpaced that in Washington and California. A myriad of factors could have played a role in this difference. The first that jumps into our mind is social distancing policies, which have worked so well in China. The three states have very different timelines of implementing social distancing. While California preemptively rolled out social distancing policies in early March, New York didn’t practice social distancing until mid-March when a big surge in confirmed cases had already occurred. In addition to social distancing, we also thought of other factors such as state-wise medical resources, population aging, and travel activities, which could all have played a role in the different spread of COVID-19 across different states.

Problem Definition

We propose a framework linking relevant factors to the spread of COVID-19.

Framework for linking socioeconomic factors to the COVID-19 spread

In the scope of our analysis, we include factors: social distancing, demographics, medical resources, transportation, and education. We use the daily increment ratio of confirmed COVID-19 cases to define the spread of the pandemic. As proof of concept, we perform a linear regression that maps the factors (X features) to the increment ratio (Y).

Data Source

We collect our data from a wide range of sources. The following is a complete list of state-level data we use:

Feature Engineering

Now, let’s take a deep dive into how we transform raw data into X and Y values. We first separate our X features into time-invariant info and time-variant info within different U.S. states.

Take the state of California for example, time-invariant features include California’s proportion of population older than 65 years old, population density, medical resources, airport density, and education index.

How we transform raw data into X and Y values

We use the given state’s state-wise social distancing policies as the time-variant feature. Importantly, we segment the timeline according to the given state’s policies. In each segmented time interval, the social distancing policies (e.g., gathering bans, closure of public venues) remain the same. Then, for each time interval, we compute the daily increment ratio of confirmed cases and take the average, getting the Average Increment Ratio (AIR), our Y variable.

Through this process, we get our model features about state info and policy configurations and our model target variable, the AIR. Before we dive into modeling, let’s take a look at what the final data frame looks like:

In our data frame, we include 14 states that have the social distancing data available. These states include NY, NJ, CA and WA where COVID-19 has the most impact. For each row, we have state-wise social distancing policies, these policies’ start date, other state information, and the Average Increment Ratio.

For each policy configuration, we record its start date. CA’s data dates back to January and has its first social distancing policy rolled out on March 5.

Note the gathering size limitation is an ordinal variable, ranging from 1 to 7. Each level describes the strictness of the policy. 0 refers to no gathering size limit. 1 indicates the least strict limitation (banning gathering greater than 1000). 7 indicates the most strict limitation (banning gathering greater than 10).

Finally, we have the AIR (Average Increment Ratio), which is calculated using data spanning from the policy start date in the same row until that in the next row, which is the end of the current policy.

Modeling and Insights

With X and Y in place, we perform a linear regression. We regress the Average Increment Ratio on all features. In the process, we also perform variable selection based on variable multicollinearity and model performance. We remove variables that have VIF (Variance Inflation Factor) greater than 10, followed by stepwise forward regression. The final model reports an adjusted R-squared of 0.26. The variable coefficient plot is as follows:

Variable coefficient plot

The red columns are features with positive coefficients so that the greater the value the higher the AIR. The green columns are features with negative coefficients and the higher the value the lower the AIR. Aging has a positive coefficient so it aggravates the AIR. Education index, closure of schools and gathering ban, on the other hand, reduce the AIR. In terms of statistical significance, aging, population density, and education index have p-values < 0.07 and thus are statistically significant.

Summing up both the coefficient value and p-value of the features, aging and education index have a statistically significant impact on the spread of COVID-19. Moreover, population aging has the strongest influence.

While we are not surprised by our findings on population aging, since we know for a fact that seniors and people with preconditions are more susceptible to the virus, we are interested in seeing that the education index plays a significant role. It means that areas with better education tend to see a slower spread of the virus. It could be that educated people are more conscientious in following government policies and good practices such as wearing face coverings. In this light, the government, in fighting the pandemic, should also pay attention to areas with lower education index.

Project Value and Next Steps

As proof of concept, the linear regression model shows that our combined dataset is able to reveal insights about the spread of COVID-19. Our findings show that, among all considered factors, population aging and education index has a stronger influence on the pandemic spread.

Moving forward, we may also add more state-wise data into the model, using our data processing pipeline. More elaborate machine learning methods, such as PCA and clustering can also be applied to the dataset to gain deeper insights.

Please refer to our GitHub page if you are interested and feel free to reach out!