Why Ontario’s COVID case numbers are BS (and how to fix this)
By: Vladimir Ilinov
Tags: Lockdown, Covid 19 Testing, Covid 19 Crisis, Covid 19, Ontario

Today is the first day of Ontario’s month-long lockdown — mandated by the provincial government after case numbers seemingly spiraled out of control.

While I’m tempted to, I’m not going to make any political commentary here. Much more eloquent people than I have criticized this policy from front-to-back.

Instead, I want to examine how bad stats have led to drastic measures.

Ultimately, it comes down to the scary looking graph above.

Our main “temperature-check” on COVID (case numbers) is telling us that things are out of control. That is, our case numbers are sky-rocketing.

And if you don’t look at the data too deeply, the calls for lockdown almost make sense.

As of late December, we hit 3X more daily cases than we ever saw during the first peak.

To those uninitiated in data analysis, this spells out a seemingly obvious truth: the second wave is way worse than the first. And as such, we need to take whatever measures necessary to curb these numbers.

But does this data actually represent the reality? And if it doesn’t, what data can we use to get a more accurate reading of the situation?

To answer the first question, we need to understand what this data actually represents.

Keep in mind, while some of this stuff may seem elementary to most people — these basic definitions are vital to teasing out the truth.

So let’s get down to it.

What exactly is a “case”?

The average person (or even a medical professional not trained in data analysis) believes that the scary chart above represents the spread of COVID-19 infections.

The higher the number, the more COVID-19 has spread across our province. The bigger the rise, the faster COVID-19 is spreading.

Right?

Not quite.

To understand what the number of “cases” is actually telling us, we have to understand what the number isn’t telling us.

And this number is not telling us the total number of infections.

Here’s why.

An infection is when someone gets COVID-19.

And there are way more infections in our community than there are cases, because we simply can’t test everyone. At least 80% of infections seem to be asymptomatic or mild. Further, symptomatic people don’t like to get tested, and we don’t even have the infrastructure to test everyone.

To truly know the exact number of infections on any given day, we would need to do the following:

Create a 100% accurate COVID-19 test (not possible)

Test the entire population on a daily basis (not possible)

So, if the number above isn’t measuring the number of infections, what does it measure?

A “case” is when a PCR test comes out positive.

Let’s repeat that.

An “infection” is when someone gets COVID-19, and a “case” is when a PCR test comes out positive.

Why is this distinction important?

1. An infection exists regardless of whether or not we test for it

What would happen if we stopped testing for a week and reported 0 new cases?

For one, our graph would look a lot better (for a week, at least). But surely, it wouldn’t mean that COVID infections have stopped dead in their tracks.

In essence, this means that infections may be a prerequisite for positive tests (in theory, a person has to have COVID to get a positive test), but they are not the same thing.

2. Total “cases” are a function of total tests

Assume that there is a community of 100 people with 50 infections.

If we test 1% of the population (1 person), there’s a 50 / 50 chance that we’ll get a positive.

But what if we test 10 people? We’re likely to find about 5 cases.

What if we test 50 people? 25 cases.

What if we test the entire population of 100? We’ll get 50 cases.

And what happens if we conduct 200 tests (2 per person)? We’ll still only have 50 cases.

This may seem a bit elementary, but there are a couple of important truths we can extract from this exercise:

If we have undiscovered infections in the community, the number of cases will increase with the number of tests Cases won’t completely plateau until we discover all infections

The reality is a little bit more complicated — because the number of infections isn’t a static. The number of infections is increasing and decreasing constantly (with people recovering, and more people getting infected). But the two principles above are nevertheless accurate.

3. Our primary testing method has false positives

Without getting too deep into the PCR (our primary COVID testing method) debate, there are a few uncontroversial facts that most medical professionals will agree with.

PCR tests can find and amplify RNA from dead virus (i.e. a person may be immune, and no longer infectious, but will still come out as a positive case)

(i.e. a person may be immune, and no longer infectious, but will still come out as a positive case) PCR tests have an estimated false positive rate of 0.8% to 4% (but we have no way of knowing for sure)

Now, in general, this is to be expected. No test is going to be perfect.

And isn’t 96% — 99% good enough?

Well, that depends on what you are trying to achieve.

If you’re treating a patient (and need to quickly determine whether or not they’re likely to have COVID or something else), a test with 96% accuracy is beyond excellent.

However, if you are trying to determine the absolute number of infections in a population, even 99% accuracy may be troublesome.

Here’s why.

Let’s assume for a second that we have a completely healthy population with no active infections of COVID.

If we used the PCR test with a 1% false positive rate, 1% of all people tested would be COVID-cases — despite the fact that no infection actually exists in the community.

If you test 1,000 healthy people, you would have 10 “cases.”

If you test 10,000 healthy people, you would have 100 “cases.”

If you test 100,000 healthy people, you would have 1,000 “cases.”

While this distortion shrinks in magnitude as real infections increase in a population (i.e. if 10% of the population was infected and you had a false positive rate of 1%, only 1 / 10 cases would be false) — we don’t have an accurate method of determining how many of our “cases” are false positives.

Summing up the issues

In short, using cases as a representation of infections is not a good idea, because:

Cases and infections are not the same thing Even if infection numbers stay the same, case numbers will grow as testing grows Our main testing method can make it seem like a totally healthy population has a 1–4% positivity rate

This is why there is a heavy correlation between the number of tests completed and the number of COVID cases:

Sure, this is not a 1:1 correlation. This is because COVID infections most likely did increase starting in September. But looking at these numbers alone, it’s unclear how much of the spike in cases is caused by increased testing, and how much was caused by increase in infections.

How can we effectively measure the spread of COVID?

With the data we have today, there are two significantly more accurate ways to measure the spread (and subsequent impact) of COVID-19.

1. Using statistical methods

The first way to determine the growth of COVID-19 is to measure the percentage of positive cases — instead of measuring the absolute number of positive cases like we do right now.

To explain why this might work, we need to go back to first-year Stats. Here’s a quick primer:

If you can’t sample the entire population, sample a representative portion (e.g. demographically)

Try to control for variables that would skew your results (i.e. testing populations that are much more likely to be positive than the standard population)

If you are trying to find trends over multiple time periods, ensure that your test conditions remain the same across tests

In theory, we can randomly test a representative portion of the population (based on location, age, social tendencies, etc.) to determine what percentage of the population is likely to have COVID-19. And if we do this multiple times, we can determine if more or less people are infected over a time-scale.

But because our tests are not conducted on representative portions of the population, they cannot be used to accurately determine what percentage of the population is likely to have COVID-19.

Nevertheless, we can still use our current stats to measure the relative growth or shrinkage of infections. This is because our testing conditions have remained the same throughout the pandemic:

We use nasal swabs for PCR testing

We test travellers, at risk populations, self-admitted patients, and those who were in contact with a COVID case

In other words, while we didn’t control for all the variables, we have been using the same methods to test the same types of people.

2. Measuring downstream metrics

There are 3 metrics that we are currently tracking that tell a much more accurate story about the spread of COVID-19. These are:

Hospitalizations

ICU admissions

COVID-related deaths

These metrics are superior for multiple reasons.

Firstly, they are not a function of total tests completed.

Sure, to hospitalize someone for COVID or attribute a death to COVID, a test is conducted. But most people that end up in the hospital are admitting themselves, and hospitals are testing 100% of people that are admitted with respiratory symptoms.

This means that Increasing testing volume will not linearly increase the number of hospitalizations, ICU cases, or deaths.

And secondly, because doctors have additional diagnosis methods, we can be confident that these numbers represent real infections, and not false positives.

While PCR testing remains an important tool for diagnosing COVID in a hospital setting, doctors have additional methods for confirming COVID in patients, like symptom screening, antibody tests, CT scans, biopsies, autopsies, etc.

Here’s what the second wave actually looks like

Percent of positive cases

COVID-related hospitalizations

COVID-related ICU admissions

COVID-related deaths

Putting it all together

These numbers all tell the same story, and it’s a very different story from the one that absolute case numbers tells us.

The story is this:

Ontario is indeed in the middle of a second wave. But it is on par, if not milder than the first wave (especially when looking at death tolls).

What do we do now?

Like I said at the beginning, this is not an article about politics. I’m not going to lambast governments or experts for misleading the public. I believe most of the people involved have good intentions, and are simply trying their best.

However, it is clear that our province, country, and world is being pulled into another round of hysteria — unrightfully so, and as a consequence of bad analysis.

The first thing we have to do is re-calibrate our compasses.

We’ve been paying attention and reacting to the wrong thing. And we’ve been doing this with real consequence to our economy, to our business, to our families, and to the collective psyche of our community.

Would we have done things differently if we based our decisions on more accurate numbers?

I don’t know for certain.

But I do know that if we keep paying attention to the wrong thing, the damage is only going to be worse.

Note: all data came from Ontario’s COVID-19 data-pool.