Can I trust COVID data?
Tags: Data, Data Visualization, Data Science, Covid 19, Covid 19 Crisis

Making sense of data during a pandemic

An Empty White House Press Conference Room | Source Credit: Wall Street Journal (WSJ)

In what many consider to be the global health crisis that defines the century, the White House is scrambling, under the continuous glare of media cameras, to unite a deeply divided country, its wave after wave of false claims backed by misleading data, or in worst cases, none, further eroding the existing fragile trust between them and the members of the public.

On April 17th, President Trump and Vice President Pence unveiled data that they hoped would bolster the confidence of governors and convince them to scale back the social distancing measures that have helped contain the virus’s spread across all the states so far.

“Our experts say the curve has flattened and the peak in new cases is behind us”, Trump said, referring to efforts to quench the rage of infections. “Nationwide, more than 850 counties, or nearly 30 percent of our country, have reported no new cases in the last seven days.” While it is true that more than 850 counties have reported little to no increase in the number of infections in the past week, more than half of these counties didn’t have any cases in the first place. 421 of the remaining counties saw a twofold spike in the number of new cases, and when the data was represented as a function of the population instead of county, only 7% of Americans lived in counties where there was little to no growth of COVID cases.

Change in Confirmed Cases from April 8 to 15 | Source Credit: The Washington Post

It’s not just the White House that is reporting information wrong — public health and data experts have expressed concerns over the credibility of COVID-19 data published by the Georgia Department of Public Health, that excluded cases from the past two weeks that had seen a surge in infections. The missing data is believed to cause the number of new cases to be much higher than the sharp decline that is reported on the state website at present.

Georgia Department of Public Health COVID data

Inaccurate data can be dangerous for policymakers and officials who rely on the numbers to come up with strategic guidelines. The underreporting of COVID cases may brew a storm of new infections in the near future should states open up too quickly. Given the barrage of data that we are faced with daily, how then do we make sense of and discern which piece of data is reliable or not? Here are several observations that I’ve made, and compiled from reputable journals like the Harvard Business Review etc. over months of reading.

1. Check for the source and the purpose of publication.

A lot of people stop at the source part and neglect to check for the intention behind publishing the data. Even if the data came from government websites or state-owned media outlets, data may have been doctored to serve political agenda. Sometimes government officials may also slip up on data, for instance, Chad Wolf, the acting secretary of Homeland Security incorrectly said that the mortality rate of the new virus was similar to that of the seasonal influenza. This highlights the imperative need for us to cross-check for other references before believing them.

2. Learn more about the methods used to collect data.

The statistics presented may be correct, but the methods used to assemble them may not be the most precise or desirable. Data that count the number of people infected based on the number of positive test results obtained daily may be inaccurate as due to geographical or other types of constraints, there may not be enough tests going around to test every civilian in the population. In other words, there may be hidden infections lurking around, or infected but latent individuals silently spreading the virus in the community that we do not know about. Furthermore, double testings are often required to confirm if a person really has COVID-19 or not. Since the number of tests is not equivalent to the number of new infections, data collected based on that parameter, especially in countries with poor testing rates, should not be taken as the gospel.

3. Cross-comparisons of coronavirus outbreaks in different countries can be misleading — and even dangerous.

Different nations have different reporting standards, different approaches to testing and different approaches to contact-tracing. The geographical and demographic differences may have a part to play in the results obtained as well. Large countries, for example, may find it harder to contain a coronavirus outbreak because the virus can spread over a larger region, making contact tracing more difficult, and federal states may enact their own policies that have varying effectiveness in mitigating the outbreak. Population-dense countries or locations may also face problems in containing the spread of the disease as the virus tends to move fast in these countries. One case in point is that of Singapore, who has, due to the rampant spread in cramped migrant worker dorms, seen more than a tenfold spike in daily infections on some days as compared to before. Although the island city has done well in controlling viral spread in both local and imported communities, due to the enclosed space and communal living of the foreign workers, it has seen its coronavirus cases soar in the recent weeks.

On this note, comparing the differences in coronavirus outbreaks between the United States and Singapore would be akin to comparing between that of an apple and pear — their intrinsic differences are just too disparate for a fair comparison to be made.

4. Look at the methods used to represent the data.

I previously talked about the methods used to collect data in point 2, now I am going to talk about the methods used to represent data. Depending on the way in which the data was portrayed, it can give rise to many different interpretations.

· Check for any omission of data

Some data may be excluded from the presentation because of the long and extensive span of the pandemic (thus rendering it inconvenient or irrelevant to be added), or because they do not fit into the intended narrative (ie. The outlook would be worse if presented than if it wasn’t). This would lead to purposeful and selective bias. Such data may include that of coronavirus outbreaks over a certain period of time; which may be high and would shake public and investor confidence and prevent countries from opening up as quickly as they would otherwise want. If presented in a graphical form, do check for the x and y-limits of the graph.

· Check for the units of measurement of the data

Number of confirmed coronavirus cases by days since the 100th day (log scale)| Source credit: Vox

Number of confirmed coronavirus cases by days since the 100th day (log scale) | Source credit: Vox

While both graphs show the number of confirmed cases since the 100th confirmed case, the first graph is presented in that of a logarithmic scale while the second graph in that of a linear scale. While log scales may be used to respond to skewness towards large values (ie. a few points of the data tend towards more extreme values than the bulk of them), based on visual observation of both graphs, while the number of confirmed cases appear to be tapering off for most countries in the log graph, they appear to be increasing exponentially in the linear graph. More needs to be understood about the magnitude and distribution of the cases in these countries before further conclusions can be drawn.

Additionally, as mentioned in the White House example above, different formulas of measurement used can yield different results. The number of new confirmed cases expressed as a function of a state can tell a story vastly different when expressed as a function of a population instead.

· Check for the method of presentation of the data

Let’s use spatial data as an example. Geodata on a basemap that shows the extent of infection across a country based on color-coding may not necessarily capture the actual magnitude and hence severity of the outbreak. The colour codes could have been stacked at wider intervals, thus making it seem like more states / provinces / regions fall into the same bin when their actual values could be significantly different from one another.

Chloropleth Map showing the number of COVID cases in China as of 24th February, 2020 | Source Credit: ArcGis ESRI

Furthermore, GIS maps should not be categorised based on the absolute number of cases in each region, as without knowledge on the size of the population living in these regions, the true extent of the outbreak cannot be known. Hubei may have a larger population size, as compared to a province with similar incidence of outbreaks but of a smaller population size, making the coronavirus situation in the latter province more dire as compared to the former.

5. Beware of the lag effect.

Last but not least, beware of the lag effect in time-sensitive data. Some of the case-fatality rate graphs of the coronavirus for example, are calculated by taking the number of people who have died at a particular time divided by the total population of people infected.

Such a method of measurement is erroneous, because it doesn’t take into account the time in which the person got infected. A data point for the number of people who have died on the 5th of May comprises people who most likely got infected on different dates, some of whom may have been infected weeks earlier as compared to others. Therefore, just because the death rate for one week may be lower doesn't necessarily mean that it cannot increase by the next week, because the next batch of patients may have a higher death rate than the current one.

Case-fatality data can thus be correctly analysed with one of the two ways — firstly, in conjunction with the incidence data, which occur before fatality, and can therefore pre-empt the imminent trend of mortality rates. Secondly, infected individuals must be grouped in batches, and the fatality rate analysed per batch.

Conclusion

With COVID constantly swamping the headlines of virtually every news outlet in the recent days, we are indubitably inundated with a fusilade of data on a daily basis. Although it would be ideal if all the data presented could accurately show a picture of improvement, some data may be prone to distortion. It is thus imperative for us to analyse a data objectively in order to be able to get to the truth of the matter at hand.