Responsibly and Accurately Reporting on COVID-19 Data
By: Berkman Klein Center
Tags: Covid 19, Journalism, Data, Reporting

Responsibly and Accurately Reporting on COVID-19 Data

“Data Overload” emphasizes challenges reporters face covering pandemic and advice for journalists

By Carolyn Schmitt

Journalists around the world are struggling with the global pandemic, from countering misinformation to interpreting data from various sources. Data journalists, in particular, are digging through droves of data to accurately convey information to the masses.

Todd Wallack, a 2019–2020 Berkman Klein-Nieman Fellow and data journalist at the Boston Globe, moderated a conversation with Caroline Chen, who covers health care for ProPublica, and Armand Emamdjomeh, a graphics assignment editor at the Washington Post, about challenges responsibly and accurately reporting on data about the novel coronavirus (COVID-19) for a virtual event co-hosted by the Berkman Klein Center and the Nieman Foundation.

Clockwise from top: Armand Emamdjomeh, Todd Wallack, and Caroline Chen. Screenshot courtesy of Lydia Rosenberg.

Wallack, who previously worked on the Globe’s “Spotlight” investigations team, described one challenge data journalists often face when gathering information for a story: “It’s often hard to get one clean database at a national level or global level. We’re often aggregating it from lots of different places, and each place might have different ways of counting the numbers and reporting the numbers, and the data can be messy.”

In the current pandemic, that’s exactly the case, and it’s no easy feat. “I think the data at a national level is basically nonexistent,” Emamdjomeh said. “Most things are reported at the state level, so that … means you have to either rely on an aggregator or aggregate the data yourself, going to all these different state sites, figuring out where they report it, how they report it, in what format, also noting that…what the states are reporting also changes over time, and what platforms they’re using to report it.”

Further, the qualifications that sources use for reporting a “positive” case vary by state, but also across countries, and are also evolving over time. For example, Chen said, China previously deemed a “positive” case to refer to a person testing positive with PCR test with symptoms, but almost everywhere else, a “positive” case meant a person with just a positive PCR test. This inconsistency is a major challenge in sorting through data.

Adding another layer of complexity, the data is perpetually in flux. “There are certain things that, just by the nature of the pandemic, are going to be constantly changing, so for example, testing capacity,” Chen said. “I’ve done a lot of reporting around testing and testing capacity, and just by the nature of what’s happening, that is changing constantly.”

Watch the full event from the Berkman Klein Center.

How can reporters navigate these challenges and still report responsibly and accurately? Below are a few takeaways gleaned from the conversation.

There are no easy answers. Explain that to readers.

“Everybody is struggling to do their best by patients and to really understand what to do. So I think there are no easy answers in this crisis,” Chen aptly said. “I think this is a failure of communication, both by our officials and also actually by journalists, when we make it sound like there is an obvious or easy answer, and fail to acknowledge that, to some degree, we are all still learning.”

The uncertainty surrounding COVID-19 makes it all the more important to be extra explicit and clear about what data reporters are sharing — and the methods used to get it.

“I’d really encourage other reporters to really try to explain to readers where you got this number, what has gone into this number, and how long of a shelf life the number will have and really try to show your work to your readers more than [you] normally would,” Chen said.

Emamdjomeh agreed, adding, “I think you can only be clear about the caveats of, ‘This is entirely dependent on what’s being reported and how it’s being reported.’”

The suggestion includes providing clear explanations of what the data actually means. “I’ve been encouraging reporters in my newsroom — and trying to explain to the public — just to be aware of what the definitions are of numbers that get thrown around,” Chen said.

Chen gave an example of defining the “fatality rate” and how the number is influenced by other factors, like testing. “The case fatality rate is the number of reported deaths divided by the number of lab-confirmed infections,” she said. “Everybody knows in the U.S. it had been really hard, and continues to be really hard in many places, to get tested in the first place. A lot of places are not testing unless you’re really, really sick. So that denominator is going to be much smaller than the actual number of infections.”

In a similar vein, it’s equally important to be very explicit about the units of measurement discussed in a report. “This has been my soapbox rant for the longest time,” Chen said. “‘Please try to get your units in people.’ Because I think, again, that is what readers care about. They see a million tests, and they think that that is a million people. When you say, ‘We’re rolling out a million tests,’ they will automatically think that is a million people who can get tested.”

“That is sometimes the case, but not always,” Chen clarified after the event. “Some tests need to be divided by two because you run two samples per person.”

Know your audience. Report accordingly.

For some types of information which are estimates, such as forecasts and projections, Chen advised reporters to share numbers in terms of order of magnitude or round numbers for lay audiences, instead of trying to boast specific figures. Otherwise, it may mislead readers to think it’s feasible to determine exact numbers at granular levels, and glazes over the nuances of the data calculations or collection.

Rounding helps to avoid such misunderstanding. “For me, as a writer, I would never give that level of precision because it signals something to a reader. I would round, and use the word ‘around’,” Chen said.

“The question really is, ‘Who is your audience? Who are you writing for?’” she said. “[It’s] one of the things that I try to keep that at the back of my mind because I think there’s a difference here. If you are writing for really a lay public, again, you have to remind them, ‘This is an estimation.’”

From a data visualization standpoint, Emamdjomeh also highlighted that lay readers aren’t used to interpreting the kinds of figures that COVID-19 data requires. “People [are] being exposed to types of scales and visuals that they’re not used to seeing. We’re seeing a lot more logarithmic scales than we’re used to,” he said. “Growth doesn’t look the same way on a log scale than a linear scale. But if you’re looking at it and think you’re on a linear scale, then you might think things are declining or flattening out, when actually that is very much not the case.”

New data quickly becomes old data. Be explicit about timing.

As new data becomes available, previously reported articles — based on old data — subsequently become outdated and can be confusing for readers.

“There is so much we are still learning that we don’t know yet, and we cannot present this as set in stone,” Chen said. This includes using language to signal the temporal and possibly ephemeral nature of the data and information reported.

“One thing I am doing now even more so than I normally do is I am aggressively dating my information,” Chen said. “I’m using a lot of language that indicates, ‘I’m giving you the best understanding at this time.’ Then I’m also linking to studies and putting language in that’s like, ‘As of this interview that I did on this date, this is what I was told.’ I think sort of all of that in combination hopefully, even if a reader comes along later, will know that that was information that was current at the time that I wrote that article, and I think that’s the best you can do.”

In terms of data visualization, “we can either build our pages and apps to plug into live data that updates, so that you are seeing updated data as of the timestamp at the top of the page, or right on the chart. Again, we try to be transparent about when that data is updating,” Emamdjomeh said. “Or, like Caroline says, we can build it statically with Illustrator, or just save it as a static SVG, and have to make very clear that this is data as of X [date]. Otherwise, we’ve been in situations when we’re trying to publish a story, and we just have to keep updating the charts five times because the data keeps changing as we’re writing the story.”

Sometimes, waiting to report is key. Other times, it’s better to not report at all.

In a 24-hour news cycle and amidst a pandemic, it may seem contradictory to slow down. But in some situations, waiting to report on new information can be critically important. Although it can be difficult, Chen suggested reporters “resist the urge to write.”

“I do think that there is a benefit to waiting because I’ve seen, more than I’ve ever seen before in any other outbreak I’ve covered, health departments iterate as they go with the data that they’re releasing,” she said.

Chen also warned of dangers covering non-peer reviewed articles. “I think it’s really sometimes dangerous to write a story off of a pre-print,” she said. “I do think it’s really great that researchers are moving quickly and sharing information on medRxiv and bioRxiv, and not waiting to go through that whole process, but then it’s not peer-reviewed. So this puts you in a really dangerous position as a reporter to have to write a story off of a non-peer-reviewed study.”

Often, she explained, after the first day of reporting, critiques of such studies start to surface. If reporting on a pre-print, “I will run it by a lot more outside experts than I normally would and just go that extra mile, and really ask myself, ‘Do I have to write this now? Can I wait for it to go through that peer review process?’”