Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper introduces a very interesting and useful question-answering resource about COVID-19. The corpus consists of 2200 frequent question-answer pairs, extracted from 40 trusted online sources plus 24k social media questions semantically-aligned (by experts) with one of the former dataset.

I really like the resource and think it is a good contribution to the workshop. Just have one suggestion regarding the manuscript and one advice about the chatbot that the authors intend to develop based on the corpus:

1. I missed some information on how BM25 was trained. Did you do it taking only the questions into account or you also use the answer linked to the candidate question?
2. Regarding the chatbot to be developed, it is important to notice that the answer to many questions  about COVID may change along the time. For instance, during quarantine, the question “Can I go to a bar?“ would be negative, whereas after the end of the lockdown would be positive. I am not sure how you can control the update of 2200 questions automatically. It is important to think about that in order to do not spread misinformation.

review_title: Interesting question-answering  resource about COVID-19

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes the curation efforts to create a QA dataset on COVID-19. It collected 2,200 question-answer pairs from popular websites. Then it extracted 27, 000 unanswered questions from tweets, identified the most similar questions in the collection and provided top 5 answers. Then the provided answers were further manually annotated (whether they are relevant). Please see my comments below.

For the 27, 000 unanswered questions, it seems that only the questions similar to the existing questions in the collection can be kept. If an unanswered question is a new topic of COVID-19, the top retrieved answers will be irrelevant, and health experts will not provide new answers based on the description. Therefore, many new questions will be potentially missed. And what is the motivation of only annotating the questions that are similar to the existing questions in the collection? Critically, the questions that are significantly different from the existing questions are more valuable and need the power of manual curation. Adding these questions will enrich the datasets. In contrast, if a question is similar to the existing questions, adding it will just have one more similar instance. 

Second, the study needs to explain how to use this dataset in detail. For instance, the statement 'Over 18, 000 examples were judged to be less than 1% relevant, indicating that the majority of the questions extracted from twitter are irrelevant to the answered questions in our dataset' seems that most of the questions do not have a precise answer yet. How to train a QA model in this case?

Third, from the methodology level, while BM25 is effective, it does not capture the semantics of the questions. Did the study also try to use other methods to calculate the semantic similarity between the questions such as using word or sentence embedding?

Also importantly, the dataset is not public for now. More specific comments cannot be made. 
review_title: A COVID-QA dataset is presented but its selection criteria and usage need to be better explained

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a potentially useful dataset of COVID-19-related frequently asked question-answers. The link given in the paper does not yet allow the access to the dataset. (https://covid-19-infobot.org/data/) Presumably this will be made available upon publication?

Introduction: When explaining the three aggregation efforts, what does the 1st mean by "generating high quality information"? 
In this sentence “To aid in this effort, we aggregate factual information in the form of verified questions and answers to help answer frequently asked questions about the pandemic.”
Better to use question-answer pairs instead of “questions and answers”.


Section 2:  This sentence is vague: "abstract away adding this information to our set". When updating a question-answer pair how the update is done. The paper lacks the explanation of how the question-answers are updated. 

Figure 2 is not a clear example. The answer to the question of "What is COVID-19?" is not correct – COVID-19 is an acronym for “Coronavirus Disease 2019”. The answer is a URL, if the correct answer is to be found there, then perhaps an example of the website content should also be included. The example also does not have any indication of the quality of the answer, and the impression gained is that this might not be a very useful resource at all. It might be more accessible to show a few records in the database, or at least to use a very accurate example from the documentation together with a snapshot of the answer that would be made available. The figure's title indicates the inclusion of metadata, but it does not have the "date/ update date" metadata mentioned under section 2.1.

Section 3: Repeated the “We list the the number of question-answer”. 

Section 4: Need correction “We additionally collection”?
Here Qorona and CovidFaq are mentioned, but there is no other reference to them later as if the questions from these sources are used or not? If yes how? If no, why they are mentioned?

Section 4.1 After Twitter data is collected, what technique/tool is used to extract questions? What indicated that a tweet does contain a question?
What techniques is used to group semantically similar questions? Is it topic modelling or something else?
The process of extracting questions from Twitter seems to be the main part of this research but is not explained properly to be useful to the research community.    


 

 

review_title: This paper presents a dataset and an high level description of the techniques used to create it. However since the data is not available the significance of this work cannot be assessed.