Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This work presents a question-answering dataset created on a subset of CORD-19. Obviously, having a large QA dataset is valuable and if done well, can be a useful research both in the bioNLP field. The dataset is created by finding "answers" to potential questions that the text under examination would answer, surely, this effort has taken many hours to create. For what has been done, much less time and effort is put into writing about it. The manuscript is vert fragmented and hard to follow:
- Abstract: Not clear what exactly is done and what contributions are
- Introduction: You could reduce explanation on CORD-19 (removing the name of organisations for example) and refer to it in a citation. Instead it was expected to see what the NLP problem here was and what are the contributions of this work. None of that is there.
- Related work is hardly a literature review. What is the common practice in the NLP community in creating a QA dataset. This is an old area. Surely there would be reputable (not arxiv) papers there to check and compare? Why tools are there and why are they relevant given they are two IR tools mentioned? 
- Section 3 needs a full rewrite. All the important details are missing. What were the annotation instructions and why? Ho w many annotators did you have? Was there any quality control? Any inter-annotator agreements for example?
- Not sure if the  paragraph before 3.1 makes any sense. BioBERT and SciBERT and so on are more representations. What is the relevance here?
- In Section 3, it says annotators are told they are developing a "method". How is this a method? 
- Method section is quite weak but maybe enough for a dataset work. 
- Please check your references, even old ones are arxiv. Maybe they are published somewhere? or maybe you can find a peer-reviewed citation for some of those.

Overall, even with the urgency of publishing COVID-19 related work, the paper should have acceptable quality which I believe this one need revision to get there. 
review_title: Potentially a good resource

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary of Work: This work provides a new resource, a QA dataset built on top of CORD-19 annotated for span prediction task in form of SQuaD dataset. The dataset contains 2K question from 147 articles (I am not clear on this point, the authors say they extracted 445 documents). They also built a model on top of this dataset by finetuning roberta base and find improvement over original roberta base.

While the dataset may be useful, we don't really know what criteria the annotators used to annotate the questions (How do they select what to annotate as answer? , what is considered important? - Do note that the authors also provide online videos of instructions that I have note reviewed which may contain the answer to this.)

Also the authors mention the questions can be reviewed/corrected by other annotators but again I don't know if it was done for every question and by how many other annotators. The annotators are said to be volunteers (I assume that to mean they were not paid) but I am not sure how they were recruited and how their qualification were evaluated (the paper says the authors require them to have a masters in biomedical sciences, although not sure how that is checked).

In general, the paper provides us some info about the new dataset, but the details are left out. As such, I am not confident in its utility for useful (and trustable) research .

The modelling framework is straightforward although their may be obvious improvements that can be made by training on biomed corpora. The top-n-accuracy metric needs more explanation (does the overlap mean overlap in text or position -- the first one is useless since we overlap on stopword easily, the second one might be more useful, but more analysis is needed).

This work needs a spell check (caronavirus -> coronavirus, fee -> see, and more).

I would note that a lot of my criticism maybe resolved by providing more details. But without them, I cannot truly provide a good review of the dataset itself. Therefore, currently I am giving a marginally below threshold score (and not a clear rejection since the dataset *may* be useful)

UPDATE: The authors answered questions presented above. The concern still remains about the quality of annotations, as well as the utility of model released. On the basis of this, I have increased my score.
review_title: QA dataset on CORD-19

final_score: rating: 5: Marginally below acceptance threshold
review_content: Authors have developed a QA dataset based on scientific articles about coronaviruses released as part of the COVID-19 Open Research Dataset (CORD-19). A small set of 450 articles were selected based on keyword matching, random sampling, and expert search, and around 2K Q-A pairs were developed against these articles, where question were generated by annotators and answers were sentences in the articles. Annotators were required to have a degree in the biomedical sciences. Also, the annotated dataset and a fine-tuned BERT-based model have been released. 

Authors investigate an important problem, and the paper is well organized and well written. In addition, such a dataset could be a valuable resource as it enables computer scientists and clinicians to address their information need about COVID-19 based on reliable sources (scientific literature).

However, there are few shortcomings that prevents me from recommending the paper for acceptance at this time:  

1. Data Quality: Each QA pair has been annotated by only one annotator. I suggest having the data annotated by at least two annotators followed by a third one (or a discussion between the first two annotators) to adjudicate the disagreements. It's important to consider that a good model should learn the actual task as apposed to optimally mimicking the behavior of a single annotator. In addition, I wonder if annotators labeled "all" sentences of articles that answer the questions. This is important for correct evaluation (models should not be penalized for correct predictions that are not labeled). In addition, I found many questions in the dataset that:

a): are not necessarily relevant COVID-19:
- "What is the main cause of HIV-1 infection in children?"
- "What plays the crucial role in the Mother to Child Transmission of HIV-1 and what increases the risk"
- "What is the molecular structure of bovine coronavirus?"
- "What causes the outbreak of SARS and MERS?"


b) are general with potentially multiple correct answers:
- "Who plays a role in regulating the immune response to a vital infection?"
- "Which species are more prevalent but less severe?"
- "What is a significant cause of Influenze like illness among healthy adolescents and adults presenting for medical evaluation?"


c) combined several answers together:
- Q: "Which species are more prevalent but less severe?"
  A: "HCoV-HKU1, HCoV-OC43, HCoV-NL63, and HCoV-229E"


In addition, "no answer" questions are probably the most interesting and prevalent questions about COVID-19. Such questions have not been developed in the context of this work.


2. Modeling: Articles were splitted into multiple chunks of text and only two chucks (a positive chunk containing the answer of a given question and a randomly sampled chunk as negative example) were used to test the model. This design choice can oversimplify the task. In addition, given that the "no answer" output of the model was disabled, I'm not sure how the model works on the negative chunks. What should the model predict in cases of negative chunks?


3. Evaluation: Top-n-accuracy measure is defined as follows: "it compares the gold label against n model predictions and looks for any overlap between prediction and answer positions." It is not clear how accuracy is derived from overlaps. But, if "any" overlap between prediction and answer positions is treated as a hit, then Top-n-accuracy is not a good/reliable metric because a small overlap (e.g. a stopword) is just enough to treat a wrong prediction as a correct answer, and an untrustworthy algorithm as a trustworthy one. This is specially problematic for questions with long answers.

I recommend authors to consider and address the above comments and re-submit their papers.
review_title: Development of a QA dataset for COVID-19; a work in progress that requires revision

Meta-review: The reviewers have identified that the authors have created a potentially useful dataset.

However, as a paper primarily about a dataset, the work lacks core details on the annotation guidelines in the paper description, and lacks quality assessment via measurement of agreement. The presented experiments themselves do not give an insight into the quality or value of the dataset beyond the specific scope of the QA data represented by the data set itself (i.e. there is no evidence provided that the dataset supports improvements in extrinsic search tasks, although this is suggested to be a motivation).

Overall, the authors have a potentially useful resource that requires more rigorous presentation. We would like to include an abbreviated (1 page) description of the resource, but encourage the authors to revise the work further.

We hope the feedback has been useful to you.
Decision: Accept (Abstract only)