Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The paper describes a COVID-19 QA system. It uses BM25 for the IR component and BERT models for the QA component. It provides both quantitative evaluations on the COVID-QA dataset and in-depth manual error analysis. The manuscript is well-written and the structure is clear. Two comments are summarized below.

1. It would be more robust to provide the performance of baseline systems and other systems on the COVID-QA dataset. This will make it clear on whether the proposed system is indeed effective.

2. The manual analysis part is interesting. More details are expected: any suggested changes on the evaluation metrics? What are the recommendations from the study? This is critical for QA systems in production. 


review_title: A well-written paper for a QA system for COVID-19 question answering

final_score: rating: 7: Good paper, accept
review_content: In this work the authors introduce their QA system that won one of the tasks in the round one of the Kaggle CORD-19 Challenge, and also make the case of the suitability of automatic metrics. With an ambitious scope, the authors very effectively frame the problem and their QA system in contrast to others in the same competition. The description of their system is quite clear and succinct, making this a very easy and intuitive read. Section 4.1. seems a loose and needs more refinement, as for example Table 1 just cherry-picks some other systems IR module to compare against (the top 5? what number was each? it just picks 3 out of 50 randomly). The evaluation of the QA system in section 4.2 is quite interesting and while brief, it shows the results are unintuitive and lead to the interesting idea of the suitability of automatic metrics after they performed a manual review. This manual review is a good idea but it is lacking in depth and detail to be fully stand-alone, more discussion would be nice to have. Overall, the paper proposes an interesting system and poses a very interesting question with regards the typical evaluation procedures for such systems, while it needs a bit of work and tidying up, it is a nice contribution. 
review_title: Interesting paper on the suitability of automatic metrics in the context of QA systems

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a question answering system that won one of the tasks in the round one of the Kaggle CORD-19 Challenge. The authors describe the two system components – IR (retrieving the most relevant paragraph given a question in natural language) and QA (extracting the answer from the paragraph). The paper further analyses a surprisingly low performance introduced by the system when fine-tuned on SQuAD and QuAC on an annotated dataset, revealing that the inherent preference of short answers by evaluation systems causes the inspected drop in accuracy.

The paper is interesting, insightful, mostly clear and very well-written.

The main contribution of this work is summarized in section 4.2 – the analysis of automatic evaluation of QA systems w.r.t. short vs. long answers, reflecting essentially the same information. My main concern is that the conclusions are drawn based on 50 or so annotations done by a single annotator, who may well have preferences re short vs. long answers for what essentially are factoid questions. That is, the “user preferences” as stated in the title are somewhat misleading. Which leads me to my second concern – the two examples in figure 1 (“how many” and “when” questions) call for a short fact-based answer IMO, or, at least should rank a short and precise fact providing an exact answer to a query not inferior to a long paragraph including the answer plus some bits of potentially unnecessary information. 

Can the entire drop in EM and F1 in the last row in table 2 be attributed to the analysis in section 4.2? Are there any other potential factors that could affect the QA accuracy?

I appreciate the authors’ analysis in figure 2, interesting!

All in all, I think that’s a good work. The fact that the proposed system (when fine-tuned on SquAD only, third row in table 2) outperforms ROBERTA-based solutions highlights the contribution of this work, which is sufficient to recommend for acceptance IMO, despite the two issues above.

review_title: A good QA system and very well-written paper, but questionable evaluation analysis.