COVID-19 Testing. What are your chances? | by Rob Harrand | Towards Data Science

Note: I am not a medical professional and this blog is just my take on what I believe are statistical ideas pertinent to diagnostic testing.
Testing for COVID-19You need no introduction to COVID-19, the coronovirus that’s now spread to every country on Earth and is wreaking havoc on economies, health systems and people’s lives.
Beyond social distancing, one of the priorities for governments the world over is to test as many of its citizens as possible, both to see who currently has the virus (the antigen test) and to see who has already had it (the antibody test). Note that in this blog post, I’ll be using ‘positive’ to mean that an antibody is detected and the person has had COVID-19.
So far, so good. You roll out the tests and assess the population, then certain people can return to work.
Evaluating a yes/no diagnostic testThe problem comes when you look down at the test cartridge (typically a point-of-care lateral flow device, similar to how a home pregnancy test works) and see the result. These tests give a ‘positive’ or a ‘negative’ result, and we often hear that such tests are evaluated by their accuracy. We hear it on the news all the time. You run a test that’s 99% accurate, get a positive result, so it’s easy to assume that probability of you having had COVID-19 is 99%, right?
Not quite.
Accuracy is the number of cases, positive or negative, that a test correctly identified during some sort of evaluation. The problem with this way of quoting how good a positive/negative test is, is that it doesn’t distinguish between the positives that it got right and the negatives that it got right.
Imagine you have a test that fundamentally doesn’t work. It just says ‘positive’ every single time you run it, regardless of the patient, the condition, etc. You test it on a random group of medical staff, 98 of whom are known to have had COVID-19 and 2 who haven’t. Your bogus test, diagnosing everyone as positive, would end up with an accuracy of 98%!
Thankfully, there are better ways to summarise the effectiveness of such tests, the most common being sensitivity and specificity. These are defined as follows,
Notice the denominators. For sensitivity, we have ‘true positives’, which are of course positives, and ‘false negatives’, which are also positives. So, sensitivity only deals with positive cases, and the logic is the same with specificity for negative cases. The result is that sensitivity is a measure of the probability of getting a positive result out of all the positive cases, and that specificity is a measure of the probability of getting a negative result out of all the negative cases.
Another way of phrasing this is that sensitivity is the probability of getting a positive result, given that you have had the disease, which is written as P(+ve result|Disease)
However, in practice, we’re interested in the opposite of this, namely the probability of having had the disease, given a positive test result, i.e. P(Disease|+ve result). So, in practice, how do we do this flip?
Enter Reverend BayesThomas Bayes was born in England in 1701. He spent most of his life focused on the Church and only published a single work on mathematics. In later life he developed an interest in probability, possibly motivated by wishing to refute David Hume’s argument against taking personal testimonials as evidence of miracles.
After his death, a friend found and handed in manuscripts on probability to the Royal Society, to which Bayes had been elected a fellow in 1742. One of the essays contained within, titled ‘An Essay towards solving a Problem in the Doctrine of Chances’, led to what is today known as Bayes’ theorem, defined as,
Note the first term on the left and the first on the right. Here we can see the ‘flip’ that we’re after, i.e. obtaining P(Disease|+ve result) when we know P(+ve result|Disease), along with a few other terms which we’ll get to soon. This theorem in turn (after refinement by Pierre-Simon Laplace) led to what is today known as Bayesian statistics.
The prevailing type of statistics and that used in many applications is known as frequentist statistics. This approach assigns probabilities based upon the frequency of events that occur in a sample. In contrast, in the Bayesian mindset, probabilities are related to our current level of knowledge about an event. This latter approach allows the introduction of something called a prior or pretest probability (explained in more detail below), which is a probability assigned before data is collected on an event. This pre-test probability is combined with new evidence to give a posterior or post-test probability.
In the Bayes’ theorem, P(A) is the probability of having had the disease and P(B) is the probability of getting a positive result. What are these values? P(A) is the pre-test probability (mentioned above), for which local disease prevalence is often used. Alternatively, a pre-test probability can also be estimated from a detailed medical examination, a review of a patient’s records, etc. P(B) is the combination of all the ways a positive result could be obtained, i.e. the patient could have had the disease and received a positive result or the patient could not have had the disease and received a positive result. Specifically,
In practice, such calculations are usually done using something called likelihood ratios. These are defined as,
Where LR+ is the positive likelihood ratio and LR- is the negative likelihood ratio. They allow you to go from a pre-test probability to a post-test probability using a graphical tool called a Fagan nomogram. Alternatively, the calculation can be quickly performed using a computer. They also give you an indication of the effectiveness of a test, as they tell you how much your pre-test probability will be affected by either a positive or negative result (see table below).
The exact calculation involves switching from probabilities to odds using,
Taking a look at some of the COVID-19 tests that are being used around the world, values for sensitivity seem to be around 91% and specificity around 99%. That gives us a LR+ of 91 and a LR- of 0.09. Pretty impressive.
We now have all the pieces of the puzzle to work out, given a positive result, what the probability of having had COVID-19 is for a given individual. First, let’s assume you run the test in an area that has a prevalence of 1% and get a positive result. What is the post-test probability?
There we have it. The probability that an individual has had COVID-19 after a positive result from a fantastic test is less than 50%.
Now, imagine if Tom Hanks had suspected COVID-19. No, not the real Tom Hanks, who did actually get COVID-19. I mean Chuck Noland, the character Tom Hanks plays in the film Castaway. If, via a highly convoluted turn of events, Chuck Noland, stranded for years on a desert island, found a COVID-19 rapid point-of-care test washed up on the beach and decided to run it because he had had a bit of a cough, what would it mean if it gave him a positive result? Well, the pre-test probability is zero, which means the post-test probability is also zero. Running the test is a complete waste of time! (not that that is much of an issue when you’re stuck on a desert island).
Now let’s flip it the other way. You run the test on medical staff who have been working endless hours in an emergency COVID-19 shelter, without sufficient personal protective equipment. They’ve all had symptoms but have since recovered. The patients that they’ve been treating all had severe symptoms. They’re were all struggling to breathe. They all had a fever. They’ve all had contact with family members that were themselves confirmed cases. It’s the middle of summer, with very low rates of the common cold and the flu. It’s hard to say what the prevalence is, so you use the method of using (past) clinical symptoms and patient history. You estimate a pre-test probability for each staff member to be 0.95. Here are the numbers,
You’ve gone from a pre-test probability of 0.95 to a post-test probability of, effectively, 1. Not particularly surprising. Again, this test was arguably a waste of time.
Triaging TestsThe conclusion from this is that, given you’re likely to have a shortage of tests, does it make sense to test people who have an extremely low or an extremely high pre-test probability? After all, at the end of the day, what you do after you’ve run the test probably won’t change. If you’re pretty sure someone has had COVID-19 and the plan is to allow them to leave isolation, using up a precious test which comes back positive isn’t going to change things. Even if someone has a pre-test probability of 0.95 and they get a negative result, that doesn’t change things dramatically either (based upon the LR- above),
We get a post-test probability of 0.63. If in the doctor’s mind they’re thinking “I’ll allow a member of the team return to work if they have a post-test probability over 0.5 or above”, this negative result doesn’t change anything. They would have been allowed to return before the test. They’ll be allowed to return after it, too.
This ‘treatment threshold’ is easier to see with a plot. Below is a Python function that works out the post-test probability for a given pre-test probability, with given values for sensitivity and specificity.
def diagnostic_posttest(pretest,sens,spec):    lr_pos = sens / (1 - spec)     lr_neg = (1 - sens) / spec         pretest_odds = pretest / (1-pretest)    posttest_odds_pos = pretest_odds * lr_pos     posttest_odds_neg = pretest_odds * lr_neg     posttest_pos = posttest_odds_pos / (1+posttest_odds_pos)    posttest_neg = posttest_odds_neg / (1+posttest_odds_neg)        return(posttest_pos, posttest_neg)
We can use this function along with sensitivity and specificity values similar to those in the real-world to create a plot of pre-test probability vs post-test probability,
import numpy as npimport matplotlib.pyplot as pltpost = list()prev_options = np.arange(0.0, 1.0, 0.01)for i in prev_options:    post_temp = diagnostic_posttest(i,0.91,0.99)[0]    post.append(post_temp)fig, ax = plt.subplots()ax.plot(prev_options, post)plt.xlim([0.0, 1.0])plt.ylim([0.0, 1.0])plt.rcParams['font.size'] = 12plt.title('Pretest vs Posttest Probability (sens = 0.91, spec = 0.99)')plt.xlabel('Pretest Probability')plt.ylabel('Posttest Probability')
Here is what we get for a positive test result,
And for a negative test result,
As you can see for a positive result, as the pre-test probability increases the post-test probability shoots up rapidly. For a negative result, you need a very high pre-test probability for the post-test probability to also be high.
How Confident?There’s one more aspect to all of this. The numbers typically quoted for COVID-19 tests do not often include confidence intervals.
Confidence intervals are a range of values around some number, such as a sensitivity value, that show you the possible range of values that are compatible with your data. They’re a way to indicate where you may find your result if you were to repeat your experiment. Such information is critical, because if you’re basing key decisions on a particular number which might actually be different in reality, you could end up with serious problems.
Let’s take an example. Let’s say a COVID-19 test is quoted with a sensitivity and specificity of 88% and 90%, respectively. Given the number of cases which fell into each category during testing (true positives, false positive, true negatives and false negatives), we could calculate the confidence intervals.
Let’s say we do this and end up with a sensitivity of 88% (85% to 92%, 95% CI) and a specificity of 90% (84% to 95%, , 95% CI), where the ‘95% CI’ indicates it’s a 95% confidence interval (see more on this in this blog post).
In essence, this COVID–19 could have a sensitivity as low as 85% and a specificity as low as 84%. That gives us an LR+ of 5.3 and an LR- of 0.18. What does this mean in practice?
Let’s say your estimated pre-test probability is 0.1. A positive test result in this case would give you a post-test probability of around 0.38, which is still low (see image below). Do you remain in isolation with a probability of 0.38? What if you have a critical job? Has the test really achieved anything?
Likewise, in a previous example dealing with front-line member of medical staff, their pre-test probability might be estimated to be 0.95. If this test gives you a negative result, the resulting post-test probability is 0.78 (see image below). Still very high.
Closing thoughtsWe’ve seen that the results of rapid, point-of-care yes/no diagnostic tests, which are never perfect, can be placed into a meaningful context using Bayesian statistics and the concept of a pre-test probability. A positive result does not necessarily mean that the person has had COVID-19, especially if they’re in an isolated hamlet in the Scottish Highlights (for the purpose of this illustration, please ignore the fact that people are flocking to the highlands to self-isolate). Likewise, a negative result does not necessary mean that the person has not had COVID-19, especially if they’re just spent 4 weeks on the front-line treating hundreds of patients with the virus.
Context matters, and in these frantic times of testing, such details may be important.
As stated at the start, this article is just an example of some statistical ideas and is in no way meant to offer any sort of guidance with regards to COVID-19.