COVID-19 vaccine efficacy calculated from data
By: Florin Andrei
Tags: Data Science, Statistics, Vaccines, Covid 19

Pfizer and Moderna — vaccine efficacy calculated from data

As of right now (December 2020), the leading mRNA vaccines for COVID-19 are made by Pfizer-BioNTech and Moderna. Their efficacy is estimated to be most likely around 94% … 95%.

vaccine efficacy likelihood

But how was that number calculated? Turns out, the basic value is pretty easy. I’ll show you how to do that, and then I will make an estimate for how confident we are that the value is right.

Randomized trials

To test a vaccine, you need to do a randomized blind trial. Gather tens of thousands of people. Divide them into two nearly equal groups. One group will receive the vaccine. The other group (the control) will receive an injection that looks exactly like the vaccine, but doesn’t actually do anything.

The control group shows what happens when there is no vaccine, when people are not protected. The other group shows what happens when people are vaccinated. The two groups are identical in every other way.

It’s important that people are chosen at random, and that the test is blind — nobody is sure who got the vaccine and who got the placebo until the trial is over and the numbers are counted. If people knew they got the vaccine (or the placebo), they would behave differently and that would influence the result.

At the end, just count how many people got sick in each group. Was it roughly the same number? Then the vaccine does not work. But if the number of infections in the vaccinated group has decreased when compared proportionally to the control group, that means the vaccine actually works.

Simply compare the ratio of people who got sick in the vaccinated group, with the ratio of people who got sick in the control group, and that tells you how effective the vaccine is.

Pfizer-BioNTech

Pfizer-BioNTech had 21720 people in the vaccinated group and 21728 people in the control group.

https://www.nejm.org/doi/full/10.1056/NEJMoa2034577

n1 = 21720 n2 = 21728

In the vaccinated group, 8 people were infected with the virus. In the control group, 162 people were infected with the virus.

x1 = 8 x2 = 162

Let’s calculate the ratios of infection in each group:

r1 = x1 / n1 = 0.000368 r2 = x2 / n2 = 0.007456

Infection ratios in the two groups for Pfizer

They’re very different — one is about 20x bigger than the other; a lot more people got infected in the control group.

So, if the control group gives us the infection rate in the absence of the vaccine, how many of those infections were eliminated by the vaccine in the other group? The difference (that was eliminated) is r2 — r1, and we have to compare that with the baseline r2:

E = (r2 — r1) / r2 = 95.06%

The Pfizer-BioNTech vaccine can be said to be 95% effective. In other words, it removes about 95% of cases that would otherwise happen.

Moderna

Moderna had about 30000 people divided equally in two groups (vaccine and control). The vaccinated group had 11 cases of infection. The control group had 185 cases of infection.

https://investors.modernatx.com/news-releases/news-release-details/moderna-announces-primary-efficacy-analysis-phase-3-cove-study

n1 = 15000 n2 = 15000 x1 = 11 x2 = 185

The ratios are:

r1 = x1 / n1 = 0.000733 r2 = x2 / n2 = 0.012333

Infection ratios in the two groups for Moderna

Which gives us the efficiency:

E = (r2 — r1) / r2 = 94.05%

So the Moderna shot is about 94.1% effective. Sounds pretty similar to Pfizer, so let’s check them side by side (scaled so the top values match):

Infection ratios in the two groups for Pfizer and Moderna

They look quite similar indeed.

Confidence intervals

All this is the result of random events. In the vaccinated group, 8 people got infected, randomly. In the control group, 162 people got infected, also randomly. But how confident are we that these numbers are meaningful?

If we do the experiment again, perhaps the numbers will be flipped, and we get 162 infections in the vaccinated group, and 8 infections in the control group — how likely is that?

Turns out, it’s very unlikely. This is easy to show, it’s a basic comparison of population proportions.

If you learned Stats 101, you’ll go “oh, okay, let me do a difference of proportions”. That would work, but it’s not the most meaningful thing here. What works here is a lesser used comparison — the risk ratios, or relative risk. Think of it as a ratio of proportions, not a difference of proportions. It’s a common comparison in epidemiology.

Let’s call theta the ratio of infection risks (vaccine group vs control group):

theta = r1 / r2

Assuming the groups are independent (which they are — that’s the point of the randomized trial) and the distribution of infections is binomial (like a flip of a coin — pure chance, which, again, is the whole point of randomized trials), then the variance of the logarithm of theta is:

var(log(theta)) = 1/x1 – 1/n1 + 1/x2 – 1/n2

The standard error is the square root of variance:

SE(log(theta)) = sqrt(var(log(theta)))

Assuming log(theta) is normally distributed, the 95% confidence interval for theta is:

CI = theta * exp(-1.96 * SE(log(theta))) ... theta * exp(+1.96 * SE(log(theta)))

If we plug actual numbers into the formulae, this is what we get — for Pfizer:

CI = 89.96% ... 97.57%

For Moderna:

CI = 89.08% ... 96.76%

In other words, we are 95% confident the real efficiency values are within these intervals. Could the real values be very different? Could they be 1%, or 99.999%? Yes, but it’s very unlikely.

What makes us more confident? The size of the trials. The more people you enroll (tens of thousands), the more narrow those intervals become. You could obtain extremely narrow intervals, but that would require extremely large trials — hundreds of thousands of people, or millions, which is expensive and takes a lot of time.

The more people you use in the trials, the more confident you can be in the results. Very big trials mean very precise numbers. These were pretty big trials, so the numbers are reliable.

Final results

We can plot the likelihood for each vaccine efficiency. We will get two narrow bell curves, centered around 95%. The assumption here is that log(theta) is normally distributed (which should be pretty accurate), and the mean value and standard deviation are calculated above.

vaccine efficiency likelihood

As you can see, it’s extremely likely that the efficiencies are around 95%. It’s quite likely that both are between 90% and 98…99%. Below 90%, the likelihood decreases very quickly until it becomes basically zero.

If we had even more people in the trials (millions), the curves would have been even sharper.

Conclusion: we are quite confident that the mRNA vaccines (Pfizer and Moderna) are around 94% … 95% effective.

This notebook contains all calculations I’ve done, you can use it to check my numbers:

https://github.com/FlorinAndrei/misc/blob/master/pfizer_moderna_vaccine_efficacy.ipynb

Notes

I am the author of all images and code used in this article. Links to data sources are included in the text.

Note from the editors: Towards Data Science is a Medium publication primarily based on the study of data science and machine learning. We are not health professionals or epidemiologists, and the opinions of this article should not be interpreted as professional advice. See our Reader Terms for details. To learn more about the coronavirus pandemic, you can click here.