Using Kalman Filter to Predict Coronavirus Spread
By: Ran Kremer
Tags: Machine Learning, Kalman Filter, Time Series Analysis, Health, Data Science

To predict the coronavirus spread, I’ve implemented a Kalman filter algorithm alongside other linear models. The optimization problem was solved with Python, while the script is available in the Google Colab notebook. the process of this project is described below, a full code can be found in Github here.

Pre-processing data:

* Read the data from Github-contain daily total cases of confirmed, death and recovered per location following data of the world health organization.

* Fixing region name and concatenate country and province to one column “region”. Then, clear some wrong characters within the dataset.

Get Population:

I used python ‘qwikidata’ package that extracts population data from Wikipedia based on the region coordinate (longitude, latitude). We will use this data to evaluate the infected rate in each area and test it in the model.

Create Time Series and Plots

The script allows us to generate a visual plot per region to get initial insights about the data. The wide majority of all cases are in Hubei, China (82%) therefore different plots were extracted.

Below are the top areas of daily total cases.

Confirmed:

Total confirmed cases in Hubei and other top regions

We can see a clear eruption trend in Hubei with exceptional day 13.02.20 of a high jump. On that day, there was a change in the counting method and it will later affect the model. We can’t verify the data loyalty but assume data is reliable.

In other top areas, the trend seems different and less sharp. The diamond princess cruise ship seems exceptional with a sharp trend since mid-February. (pink line)

Death:

* Death cases are mostly in Hubei with more than 2,000. After that, Henan with 19.

* Death rate in Hubei is 2.1978% and 1.5055% in Henan. In the rest of the areas, the numbers are too small.

Total death cases in Hubei and other top regions

Recovered:

* Sharp trend of recovered cases in all areas, mostly in Hubei which can imply us for a positive future.

Total recovered cases in Hubei and other top regions

Fitting Prediction Model Phase 1-One Day Prediction Kalman Filter

I implemented a Kalman filter algorithm that fit the problem and generate 1 day ahead prediction for each case -confirmed, death, recovered; for each region.

Kalman filter is a recursive algorithm that uses time-series measurement over time, containing statistical noise and produce estimations of unknown variables.

I built an online-real time algorithm. This algorithm is adaptive, means it doesn’t need a lot of historical/training data. Each day the algorithm is updated with new observation, after the estimation is done it can generate predictions for the next day.

Kalman filter shows very good results predicting the next day observations. It follows the trend of each region separately and rapidly adapts the series and generates the prediction.

It is easy to evaluate the model as we don’t need a training and testing sets. No overfitting or bias — it is an online algorithm.

Implementation:

I wrote an R script to implement an online Kalman algorithm and then used Python ‘ rpy2’ package that allows us to combine R script inside Python notebook and pass the outcome variables.

After a short initialization, the algorithm starts to follow the series and produce a 1-day ahead prediction for each region. It runs on each possible region.

Results and Evaluation:

To evaluate results, I’ve added some basic error estimator parameters for each region: MSE — mean square error, RMSE — root mean square error, MAE — mean absolute error.

I compared them with other methods to optimize the models and find the best one.

The model can also be evaluated with plots and tables as shown below.

The script allows to select regions and get a suitable plot and predictions. It was run iteratively for each region. Below are several examples:

A. Beijing: The orange line is the Kalman prediction, the blue line is the real confirmed cases — as mentioned, this is not bias prediction, it summarizes the previous online predictions.

We can see almost perfect fit:

Kalman predictions vs. actual values — Beijing

The table below shows how close are the predictions to the actual values.

For example, on 19.02.20 Kalman predicts 394 cases which are 7 new confirmed cases while there were actual 6.

For tomorrow (20.02.20) Kalman predicts 5 new confirmed cases in Beijing.

Daily confirmed prediction vs. actual — Beijing

B. Hubei confirmed cases — here we see almost perfect prediction until the 13.02.20, then since the jump in the data that was mentioned above the model had a larger error. But it is adapting fast and, in a few days, will gain better predictions.

Kalman prediction vs actual confirmed cases in Hubei

The plot below shows the great prediction rate of death cases in Hubei:

Kalman prediction vs actual death cases in Hubei

The table below shows the predictions and the actual death in numbers, in Hubei.

For example, on 18.02 Kalman predicted 140 more death where there were eventually 132.

On 19.02 Kalman predict 129 more death where there were eventually 108.

For tomorrow (20.02), Kalman predicts another 132 new death cases. The predictions are very close to the real values.

Daily death prediction vs. actual — Hubei

Recovered daily predictions: We can get some positive insights as to the sharp trend of recovered cases in Hubei.

The Kalman prediction adapts this trend and predicts an increasing number of recovered cases recently.

Kalman prediction vs actual recovered cases in Hubei

C. Shanghai-confirmed prediction. Here also great daily predictions very close to the real values

Kalman predictions vs. actual values — Shanghai

D. Henan — Region with more than 1,200 confirmed cases — again very good predictions that adopt the trend of less confirmed cases in recent days.

Kalman predictions vs. actual values — Henan

Henan recovered prediction:

Kalman predictions vs. actual values — Henan

On the 19.02.20 Kalman predicts 56 new recovered cases where there were eventually 51.

Tomorrow it predicts another 48 new recovered cases.

Similar to Hubei we can see a good trend of recovered cases recently and a good adaptation of the algorithm.

Daily recovered prediction vs. actual — Henan

E. Another interesting region is the well-known diamond princess cruise ship close to Japan.

The behavior of this series is thoroughly different from other regions and still, Kalman’s prediction is doing a good job. There was an eruption of cases in the ship at the beginning of February, then it was sharply increased.

Kalman predicts another 46 new cases for tomorrow (20.02.2020).

Kalman predictions vs. actual values — Diamond princess cruise ship

Fitting Prediction Model Phase 2 — Spread and New Cases Along Time

The Kalman algorithm is very powerful and provides a very good indication of what will be tomorrow.

But to predict a longer period, it is not enough. (reminding this is an online algorithm).

I fit a linear model to predict the spread of COVID-19 along the time.

The idea is to use time series algorithms such as Kalman and send them as the features for another machine learning algorithm, alongside other features that should be considered such as weather, population, infected rate, etc.

The data is not stationary so ML models such as random forest regression didn’t work well.

Eventually, I used a linear model that relay on the Kalman prediction.

Creating Data Sets

The data set for the linear models contain different structure where:

* Each row represents day and region.

* Each column represents feature/variable where the target variable is the number of confirmed/death/recovered cases.

Feature engineering:

Weather data:

I’ve used python ‘pyweatherbit’ package to extract historical and forecast weather data per region coordinate (longitude, latitude). This package requires an API key which is limited to 500 calls per day so I run it for several days and then stored the data in CSV files (w.csv and w_forecast.csv) to merge it with the Github data.

I’ve extracted minimum and maximum temperatures for each day and location. (forecast is limited to 14 days, for a longer period I’ve made a liner temperature forecast).

Hubei daily min and max temperature

The table below shows the top infected areas average temperature (from 22.01.20 until 17.02.20).

We can see that most areas (beside Guangdong) get pretty cold weather with no more than 12–13 maximum average degrees and less than 5 minimum.

Most infected areas average temperature

Infected area data:

I used the population data and the daily total confirmed cases to calculate the infected rate of each area each day in percentages. This feature will be important in the final model.

Features list:

The features that were tested in the model:

1. Min-max temperature.

2. Infected rate per region.

3. Last day total.

4. Last X days change (x=1,3,7).

5. Last X days change rate.

6. The prediction from Kalman.

7. Region.

8. Population.

9. Confirmed/death/recovered — target variable.

Correlation Matrix:

In the plot below we can see the correlation matrix between the variables.

* As expected, X days change/rate are correlated between them. (all the time-depended features)

* Kalman and time-dependent variables are correlated to the target.

* Infected rate also has a relatively high correlation with the target.

* Min-max temperature has a very low correlation to the target. (but it doesn’t mean they aren’t helpful).

Correlation matrix between all variables

Feature Importance

I’ve used python ‘H20’ package and run a separated model for the Hubei region.

Results below show the most important features:

Kalman's prediction is significant, as long as the other time-depended features.

Infected rate is an important feature and will be used in the final model.

Random forest feature importance

Final Model

Usually when training ML models we need to divide our data to training/testing set (~75% to ~25%).

Because we deal with time-depended non-stationary series this method doesn’t fit.

I’ve solved this problem by imitating an iterative “online” system that trains the model every day and predicts the next day.

Then, instead of the getting the feedback of the real value, I updated the next timestamp value with the last prediction plus some calculated noise, and predict again.

Each day all the features are being updated (Kalman, infected rate, weather forecast, and last X days calculation).

After evaluating several models I chose a simple linear regression model using python ‘sklearn’ package.

Results:

The script allows us to choose each region and get the prediction of total confirmed, death and recovered cases. Here I show plots of Hubei and other major regions.

Hubei — Confirmed:

The prediction shows less and less future confirmed issues in Hubei but expects to pass 100,000 confirmed cases next month.

One month confirmed predictions Hubei

Death — following the prediction, 3,100 people will die from COVID-19 next month in Hubei. So far 2,156 people died in Hubei (19.02.20).

One month death predictions Hubei

Recovered- This positive prediction shows the good pace of recovered people in Hubei. Following this prediction, more than 80,000 people will recover from coronavirus next month.

One month recovered predictions Hubei

Major areas:

* We can see that in many regions the prediction for next month shows almost no spread. All the bottom lines and lines where the gradient of the graph is not changing.

* Based on this prediction the massive eruption was passed and the trend showing less and less confirmed cases even in top areas like Zhejiang, Henan, Anhui, Guangdong (top 4 lines in the chart).

* The diamond ship is expected to get more and more confirmed issue — until all the people in the ship will be evacuated. This should be happening soon, then there will be no more predictions. (5th line light blue)

One month confirmed predictions major regions

Death: Following the prediction in most of the regions we won’t see an increase in death cases. The main concerned region is Henan with the prediction of almost 40 death next month (today there are 19)

One month death predictions major regions

Recovered — Prediction shows the recovered rate will increase next month. More and more people will be recovered from COVID-19.

One month recovered predictions major regions

Heatmap:

To present another visualization out of the prediction I’ve used python package ‘ gmplot’ to show the predictions over google maps.

The map shows a hotspot with high infected locations based on the coordinate of the region.

The observations are the prediction of the next month in each region.

The API of google is not free anymore so this version is limited but it does show the infected location.

Infected locations heatmap based on next month prediction

Infected locations heatmap based on next month prediction. Excluding Hubei

Conclusions

The final model predicts the trend of the COVID-19 virus for the next 30 days or more. The one-day Kalman prediction is very accurate and powerful while a longer period prediction is more challenging but provides a future trend.

while a longer period prediction is more challenging but provides a future trend. It seems like the peak of infection is behind us and prediction with this trend showing less confirmed issued and more recovered.

All the graphs were the prediction curve is getting less moderate and the gradient is lower (for confirmed & death) show the trend is positive-optimistic, while in recovered prediction the lines are getting sharper along time means more people will be recovered from the virus.

All the graphs were the prediction curve is getting less moderate and the gradient is lower (for confirmed & death) show the trend is positive-optimistic, while in recovered prediction the lines are getting sharper along time means more people will be recovered from the virus. It’s necessary to divide Hubei and other regions when analyzing the results.

The weather feature can’t be a significant one since the model never saw how COVID-19 spread perform in hot weather, but the temperature did slightly improve the results. Based on the most infected region average weather, we can assume that once these regions will get warm weather (~20 degrees or more) the confirmed cases will reduce. In Hubei, such weather expects to be since mid-April.

Infected rate is one of the main features that affect the spread — the model found it relevant especially in Hubei and The princess diamond ship . The model can’t predict the movement of people among areas that affect this infected rate. Therefore, based on this data it is not relevant to predict which new countries will get affected, but once they did the model will predict the spread in these new regions.

that affect the spread — the model found it relevant especially in and . The model can’t predict the movement of people among areas that affect this infected rate. Therefore, based on this data it is not relevant to predict which new countries will get affected, but once they did the model will predict the spread in these new regions. Following the positive recovered trend prediction, the isolation of the Chinese government on Hubei and the upcoming warmer weather in China I assume a major decrease of COVID-19 spread by the end of March. But till then any movement of infected people to other regions may change the picture…

That’s it, thanks for reading feel free to contact me to share your thoughts questions or feedback.

Updated predictions can be found here