Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper describes a question-answering system that is particularly designed for COVID19-related questions. The system is capable of providing two types of answers: potential spans containing the answers to the questions and potential bio-medical entities that the query user might be interested in. 

The system design of the paper is well justified, and a online demonstration of the system is provided. I just have several questions as below.

First, the "related entities" service in the online demonstration seems not working well. I tried several entities including the example shown in the paper but did not get any returned results in terms of the related entities.

Second, I like the idea of retrieving both answers and related entities for query users. Do the authors think that it is possible to use the two systems to enhance the performance of each other in the future? Now the two systems seem disjoint to me.

Finally, the authors claim that one of the highlight of the system is that it's real-time. Of course, I've tried the online demonstration and did not feel significant delay in getting answers. But it is necessary for the authors to provide evalution results on its efficiency. 
review_title: Solid work with potential social impact

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work presents a real-time QA tool covidAsk, which provides both the answer/supporting documents, as well as related entities. The framework mainly uses two previous works (DenSPI and BEST) with BioBERT on CORD-19 and PubMed. Despite some issues mentioned below, the paper contains substantial content that could promote the development of COVID-related QA/IR systems.


**Comments**


- The online demo was not functioning. I have tried on multiple days with multiple ips, but only the cached examples work. For example, https://covidask.korea.ac.kr/search?query=What+can+I+do+to+protect+my+family+against+COVID-19%3F will return an internal server error.

- For a *real-time* system, there should be evaluations on latency, computing resources needed, etc. 

- I didn't see any evaluation on the entity search engine side. How well does it perform?

- Table 5: how should we interpret the result? From my understanding, the performance of this work is much worse compared to some of the systems. In that sense, would it make more sense to use one of the QA systems plus a regular search engine?



**Misc**
- Section 1, Paragraph 4: The development of ...

- Section 1, Paragraph 5: "The first challenge is that there are no QA datasets that are tailored specifically to COVID-19. " there is at least contemporaneous work COVID-QA (https://openreview.net/forum?id=JENSKEEzsoU).

- Section 3.4 NBCI -> NCBI
review_title: Useful tool but needs more details

final_score: rating: 6: Marginally above acceptance threshold
review_content: This study presents a COVID-19 QA system, COVIDASK. It uses two models previously developed (DENSPI and BEST) as the basis and adapted accordingly to the evaluation on a manually crafted COVID-19 QA dataset. The manuscript is easy to follow. I have two main questions.

First, why the indexing is at phrase-level? From Table 5 (the evaluation on the IR part), it seems that the search effectiveness on phrase-level indexing is lower than other systems indexing at passage or paragraph level. 

Second, the evaluation dataset for the QA part was created by the authors themselves. Was the system evaluated on public COVID-QA datasets such as https://github.com/deepset-ai/COVID-QA and https://github.com/castorini/pygaggle/ ? The evaluation results on these datasets will be more representative since different COVID-19 QA systems can be evaluated on the same datasets.

Minor comment: please include more detail on the system such as the update frequency and any APIs supported.
review_title: A COVID-19 QA system from two previously developed models; evaluations need to be expanded on public COVID-19 QA datasets