Revisiting Imperial College’s COVID-19 Spread Models
By: Yitaek Hwang
Tags: Data Science, Kubernetes, Machine Learning, Programming

How to run open-source Tensorflow models on Kubernetes and reviewing how effective the COVID-19 spread model was in measuring the effect of interventions.

Photo by Brian McGowan on Unsplash

Earlier this month, the United Kingdom became the first European country to approve and administer the first doses of Pfizer/BioNTech’s COVID-19 vaccine. The United States quickly followed suit with the FDA and CDC recently recommending Moderna’s vaccine as well as Pfizer’s to give the world a glimmer of hope. Other international players, notably China and Russia, are also pushing to approve and produce their own vaccines. Even as COVID-19 continues to rage on, this news of vaccines signals a hopeful end in sight.

To that end, I wanted to revisit a study from the Imperial College COVID-19 Response Team, “Estimating the number of infections and the impact of non-pharmaceutical interventions on COVID-19 in 11 European countries”, published in March. The study used a semi-mechanistic Bayesian hierarchical model to estimate the impact of non-pharmaceutical interventions such as isolation, the closing of public spaces (e.g. schools, churches, sports arenas), as well as widescale social distancing measures.

The Tensorflow implementation used in the paper is open-sourced under the MIT License and available at Tensorflow.org and Github.

Code Setup

While Google provides a free, hosted Jupyter notebook service through Google Colab, I wanted to run the analysis on Kubernetes to practice running data science and machine learning projects on Kubernetes as well as to compare the developer experience for both. To replicate the managed notebook experience of Google Colab, I looked for a similar Kubernetes experience without needing to stand up a cluster myself. At the same time, I wanted some control over my Kubernetes environment and not a fully managed data science platform like the Google AI Platform.

I eventually found puzl.ee, a Kubernetes service provider with GPU support that charges per pod usage. Puzl.ee creates a unique namespace for my workloads and charges for resource usage similar to serverless Kubernetes offerings such as Google’s Cloud Run or AWS Fargate. The number of packaged applications are currently limited (Gitlab CI Runner, SSH Server, and Jupyter Notebook), but support for H2O.ai, PostgreSQL, Determined AI, Redis, Jupyter Hub, Drone CI, MongoDB, and Airflow is on the roadmap. Fortunately, puzl.ee had already published a quick start guide for setting up a Jupyter Notebook with GPU, so I provisioned my Jupyter Notebook after signing up for a free account.

I was given various options for predefined Docker images (as well as an option to use my own). I could also adjust my resource requests, including NVIDIA RTX 2080Ti GPU, before installing my Jupyter Notebook.

Within a minute or so, my Jupyter Notebook with Tensorflow was installed. Afterward, I realized that the code required a Tensorflow 2+ image. Even with the reinstallation, the provisioning process was smooth and painless.

Unfortunately, I ran into several issues trying to run Tensorflow out of the box. I was surprised to see Pandas not included in the standard distribution of the provided Docker image and encountered several errors such as undefined symbol: _ZN10tensorflow8OpKernel11TraceStringEPNS_15OpKernelContextEb . After a few unsuccessful solutions from StackOverflow, I decided to compile my own Docker image with the necessary Tensorflow components installed, which allowed me to move on. However, this highlighted a huge challenge in AIOps where version control and software compatibility are still hard, making these semi-hosted platforms less effective as it requires some DevOps work to untangle and fix dependencies.

Model Setup & Data Preprocessing

After correctly installing all the necessary Python modules, I was able to follow the code to set up the model and load the data. The entire setup is posted on Github, but I’ll summarize the important sections below:

Mechanistic model for infections and deaths

The infection model takes in the timing and the type of interventions, population size, and initial cases per country along with the effectiveness of interventions and the rate of disease transmission as parameters to simulate the number of infections in each European country over time. The model produces two key probabilities conv_serial_interval (convolution of previous daily infections and the distribution over the time of becoming infected and infecting others) and conv_fatality_rate (convolution of previous daily infections and the distribution over the time of infection and death).

Parameters & Probabilities

Parameter values, which are assumed to be independent, include exponential distribution of initial cases per country, negative binomial distribution for the number of deaths, infection rate per each infected person, the effectiveness of each intervention type, and the noise in the infection fatality rate. Given these parameters, a likelihood of observed death is calculated along with the probability of death given infection. Finally, infection transmission is assumed to be Gamma distributed, to turn conv_serial_interval to predict_infections .

Key Assumptions

The study aims to model the effectiveness of intervention measures by looking at the infection and fatality rates. Here the model assumes that the decline in the number of COVID cases is a direct response to interventions rather than gradual changes in behavior. Also, the study assumes that interventions will have the same effect across the European countries selected, not accounting for the size, population density, and perhaps the average age of its citizens, which we now know has a huge impact on fatality rates.

Replicating the Results

The dataset includes interventions enforced and infection/fatality rates from 11 European countries (Austria Belgium, Denmark, France, Germany, Italy, Norway, Spain, Sweden, Switzerland, and the UK). After applying the Tensorflow model as described on the Github post, I was able to replicate the effectiveness of the interventions graph as well as infections/deaths by country:

Effectiveness of interventions (same as Figure 4 on the original paper): shows no effects are significantly different since most measures when in effect around the same time

Infections, deaths, and R_t by country (same as Figure 2 on the original paper)

Comparing the Model with Actual Data

The paper estimated that various intervention measures were successful in curbing the rate of infections in Europe with the caveat that given the long incubation period of COVID-19 and the time between transmission and mortality, the data collected at the time may be premature to conclude effectiveness in certain countries where the pandemic was at its nascent phases.

Looking back, we now know that the initial measures were somewhat successful in slowing down the rate of infection in Europe. It took drastic measures such as country-wide lockdowns and enforcing large-scale social distancing guidelines, but the data shows a downward trend in infection rates until a recent uptick in cases.

Perhaps a more telling graph is comparing the results of Europe as a whole vs. the United States where interventions measures were rolled out in a less coordinated manner:

Now that we have more data, it would be interesting to see a follow-up study to include other parameters to reflect cultural or political factors that made intervention measures more or less successful across the globe.

You can create your own visualization using the link below:

Colab vs. Puzl.ee

Coming back to the data science side of things, this exercise of running the Tensorflow model reminded me of the challenges we still face before making Kubernetes the go-to platform for data science and machine learning tasks. As a free tool, Google Colab makes it easy to clone open-source notebooks and run experiments without any infrastructure setup. With Google Colab Pro, which is priced at $9.99/month, most workloads can be run in a managed manner without too many restrictions.

However, the initial provisioning process on puzl.ee was surprisingly smooth. As the team works on putting together more predefined Docker images, I expect some of the installation and configuration challenges I faced to diminish. I also liked the option of running my own Kubernetes pod to potentially extend the experiment to add other microservices to either fetch/post data or integrate it with other databases within the same Kubernetes cluster. Once puzl.ee adds native support for popular Helm charts via the dashboard similar to how it provides pre-made Docker images, I plan to take a look again for some of my side projects.