What are People Asking About COVID-19? A New Question Classification Dataset
By: Jerry Wei
Tags: Machine Learning, Linguistics, Covid 19, Data Science, NLP

What are People Asking About COVID-19? A New Question Classification Dataset

COVID-Q is a new dataset of 1,690 questions about COVID-19 that have been annotated into 15 question categories and 207 question classes.

The Problem. One of the biggest challenges during large pandemics such as the current COVID-19 pandemic is being able to keep people up-to-date with the latest and most relevant information. Even though reputable sources like the CDC and FDA maintain FAQ websites for COVID-19, users might still struggle to find their questions, and many common questions will remain unanswered.

I’ve worked with other researchers to compile COVID-Q [Dataset Link], a dataset of COVID-19 questions, in the hopes that this dataset will be useful to other researchers. Our full paper can be found here.

The Dataset — General Overview. COVID-Q is a dataset of 1,690 questions about COVID-19 from thirteen online sources. The dataset is annotated by classifying questions into 15 question categories and by grouping questions that ask the same thing into 207 question classes.

COVID-Q can be used for several question understanding tasks:

The question categories can be used as a standard text classification task to determine the general category of information that a question is asking about.

The question classes can be used for retrieval question answering. In this task, a system has a database of questions and answers. Given a new question, the system must find the question in the database that asks the same thing as the given question and return the corresponding answer.

Distribution of questions in COVID-Q by source. The reported number of questions excludes unrelated, vague, and nonsensical questions that have been removed. A * denotes sources for which questions came from FAQ pages.

Data Collection and Processing. To collect the data, thirteen sources were scraped to gather questions about COVID-19; seven of these sources were official FAQ websites from reputable organizations like the CDC and FDA and six sources were crowd-based (e.g. Quora, Yahoo Answers).

Distribution of question categories, with number of questions per category shown in parenthesis, in COVID-Q.

Data Annotation. The dataset is annotated in two ways. First, questions that ask the same thing were manually grouped into question classes. Two questions were defined as asking the same thing if they could be answered with the same answer. Next, each question class with at least two questions was given a name that summarized what the question class was asking about. These were then used to place each question class into one of the 15 question categories shown in the figure above. You can also see the distribution of question classes in the figure below.

Number of questions per question class for all question classes with at least two questions. All questions in a question class asked the same thing.

Annotation Quality. Now, you might be wondering: If these questions were manually labeled, how can we trust that you didn’t go crazy on the annotations? Well, that’s a fair point. To validate the quality of annotations, the labels were run through two collaborators in two rounds of validation. In these validations, each collaborator independently examined the category and class label given to each question to verify that the labels were correct. In the first round of validation, 67 labels were changed. In the second round of validation, 15 more labels were modified.

We also validated the annotations by showing 3 random questions from each question class with at least four questions to three Mechanical Turk workers. Given a question, we asked each worker to select the question class that best described the question out of five choices, one of which was our original annotation. For these sampled questions, 93.3% of our labels agreed with the majority vote of the workers.

Examples of frequent questions that were unanswered by official FAQ websites. The middle column represents the number of questions in the question class (i.e. the number of occurrences of questions that asked about this).

Mismatch of FAQs. Something interesting that we found was that the FAQ websites of governmental organizations like the CDC and FDA consisted of a large proportion of questions that did not match questions from any other source. For instance, 44.6% of the CDC’s FAQs were unmatched and 42.1% of the FDA’s FAQs were unmatched. Also, 54.2% of question classes with questions from at least two non-official sources remained unanswered by an official source. These data seem to suggest that there is a mismatch between what organization think people are asking about COVID-19 and what people are actually asking about COVID-19.

Author-Generated Questions. Since the dataset is small when split into training and testing sets, 249 questions were manually generated. These questions were annotated and validated in the same ways as the other questions.