Abhishek Dasgupta and Graham Lee: Lessons from building a data platform during the pandemic
Channel: Society of Research Software Engineering
Date: 2023-02-16

Full title: Global.health: Lessons from building a data platform during the pandemic

A talk from the Structuring Data parallel session at RSECon2022.

Details: https://virtual.oxfordabstracts.com/#/event/3101/submission/99

=====

Extra Q&A

Q: Do you have one over-arching data governance policy for all your international datasets? If so what do you do with those data that don’t directly align with that policy?
A: (Graham) Yes we do. This was led by Gal Wachtel on our team, who worked with multiple data governance agencies to craft a policy that aims to balance our value of open science with a respect for privacy and the multiple legislative requirements in place around the world. We try to keep and publish as much original information as we can. Were needed to preserve anonymity, we lower fidelity: for example, reducing precision of geographic data, or bucketing multiple records by age group. If we can't do that, the data doesn't get in.

Q: What approach did you take to gaining understanding how users would use the system and what their access patterns would be? did you communicate “good access patterns” in some way?
A: (Graham) Our partners at Google.org and at Fenris led rounds of user research to find out what people would find useful about our system, and how they would interact with it.

Q: What kind of processing is being done in the ingestion pipeline that takes so long to complete?
A: (Graham) It's mostly a straightforward transformation of an input data format (which might have identifying information that needs removing or anonymizing, or provider-specific codes that must be interpreted) into a common format. Two bottlenecks are geocoding, for which we use an external API, and merging updates, which often means dropping all of the input data for a source and re-ingesting it all again because the data source doesn't contain stable, unique identifiers for each record. But the main reason it takes so long is the number of records, particularly in the larger data sources like the US CDC data.

Q: Do you have a process for identifying data risks and removing them as necessary? Do you accept reports from users?
A: (Abhishek) When we were curating Mpox data, we set up a re-identification risk script to alert if the risk crossed a certain threshold. We have a GitHub repository where data/privacy issues can be raised as well as an email address.

Q: What time pressures did you face in making this platform available during a rapidly changing situation (data updates, additions etc)? How did you overcome them?
A: (Abhishek) Having many people working on the project was helpful in the early days as we had to developing ingestion pipelines for multiple sources. In addition, our infrastructure is mostly automated, and that has helped us iterate faster.

=====

Abstract: Global.health is the first of its kind, easy-to-use global repository and visualization platform that enables open access to real-time de-identified epidemiological line-list case data, that was developed during the COVID-19 pandemic. The project initially started with a group of volunteers recording data on spreadsheets, but that quickly ran into scaling issues. This led to the development of a platform built on MongoDB, Node.js and Python that now contains 100m+ records in a uniform schema, imported daily from official and authoritative sources.

This talk will describe the evolution of our data infrastructure, and in particular, will focus on (i) challenges that we faced in developing and extending the data schema (ii) tackling inefficiencies in large dataset search and filter, and (iii) addressing privacy concerns in importing datasets with potentially identifiable information.

=====

0:00 Chair introduction
0:30 Talk
22:05 Q&A