Venue: ACL 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This work is an introduction to a suite of resources and their corresponding ideas revolving around the COVID data provided by the Allen institute. Authors in particular provide Neural Covidex which allows exploration of information retrieval and NLP techniques over the COVID dataset.

The positives of this manuscript is as it also says gathering all the information and resources one would need to start exploring this dataset and further the techniques without having to spend weeks to put them all together. It is significant effort to get this far and of course authors are relying on their previous work over many years. There are some good pointers for some of the recent work in biomedical IR that is useful to know. I personally learnt from the content and enjoyed reading it.

This work however is written in haste (as it was needed due to the time limit of the current situation) and it is lacking the formal language of a scientific paper and of course evaluation. Obviously evaluation on this particular dataset is not straightforward given for example TREC is only starting on that effort. However though some initial analysis of the dataset could strengthen it. I was half expecting to see the discussion on evaluation metrics be a bit more mature and be given a separate section at least. This work as it stands, while very useful for the IR/NLP community it needs some reworking of the content to get to the state of publication in an ACL workshop. 






review_title: Useful resources collected and introduced

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes the rapid deployment of an IR system for the CORD-19 dataset. The system borrows state-of-the-art pieces, such as T5 and BioBERT, with training on MS MARCO. A search interface is shown that includes potential answer highlighting. Absent hard evaluation data, the authors provide more of a narrative of the system development, from initial conceptualization, early builds, deployment, and social media advertisement.

First and foremost, the authors are to be congratulated for this work. The effort is most laudable.

The style and tenor of the paper should be described as somewhere between a Medium post and a work-in-progress preprint. While normally this would be out of place, it is less out of place in a workshop such as this one. The casual language is fine and almost cute, while the frankness is quite welcome, but the other preprint-like or web post-like aspects of the paper are more problematic.

Let's start with the lack of evaluation, which the authors acknowledge that "It is, of course, expected that papers today have an evaluation section that attempts to empirically quantify the effectiveness of their proposed techniques". Of course, this empirical validation is what makes it science, as opposed to engineering/marketing. I certainly understand that the resource commitment to a TREC-style evaluation is beyond one group's ability under a pandemic situation. But the authors present the solution within their own paper: from footnote 14 it appears that the TREC-style evaluation is underway, and that should have preliminary results available well before this workshop's deadline. So why not simply issue this as a preprint to flagplant their admittedly laudable effort, then wait until some empirical results are available and submit *that* paper to the workshop?  If the authors plan to submit a work with results as a separate publication, this should be made more clear and a more compelling argument needs to be made for the scientific usefulness of the "behind the scenes" story of this search engine.

The authors make the claim that instead of the importance of search result ranking (despite Section 3 being about how great the system is at ranking on other tasks), that instead what is really important are the interface improvements that increase the usability of the system. However, every real-world IR evaluation has come to exactly this conclusion: experts always request various bells and whistles. First, note that users can't see the relevant articles that were missed, so they have little choice but to comment on the interface. But more importantly, good ranking != good usability, this is no novel claim. But usability evaluation is a well-honed science, and the authors do not perform any kind of usability evaluation either. These are less resource-intensive as well, typically requiring just an IRB protocol and some experts willing to provide feedback.

My main stylistic issue is that in many places this paper comes across as a brag-fest for the accomplishments of the group. These accomplishments are beyond doubt, of course, which is all the more reason why the authors can avoid tangents to cite unnecessary prior work (e.g., most of the BERT stuff in the first paragraph of Section 3.3, as this is not even used in the system).  The implications in Section 5 that the authors' group's "culture" and use of good software practices, unlike other poor-coding researchers, is also unnecessary (why is a "faculty advisor" reference even there?). One can extol the virtues of good culture and software engineering without coming across as bragging, so I would recommend a re-write of those parts of this paper to take a more scientific tone.

Minor:
- In Section 3.1 it is unclear what the "1" in "n+1" comes from. Do the authors also include a document that is just the title/abstract without the paragraphs?
- "who are engage in" -> "who are engaged in"


review_title: Laudable effort, but currently more preprint material

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors have quickly stood up a conventional and neural search system based on the CORD-19 dataset, which is a commendable task given the global crisis caused by the Covid-19 pandemic. However, this paper is just a description of their system and design decisions, with some additional discussion of why a conventional TREC system-oriented retrieval evaluation would be limited. I agree with the authors on that point.

However, it would be helpful to have some sort of evaluation of their system to gauge whether their approach offers any novelty beyond the multiple other systems that been stood up and linked to from the Allen Institute site where the data is housed.

While I agree that a system-oriented evaluation approach would be limited, it would also be helpful to, say, compare the baseline and neural approaches. The first round of the TREC-COVID challenge evaluation is being conducted as I write this, and the first set of results will be available by late April or early May. I believe a better approach would be to include these results and then discuss their limitations.

If the authors do not believe the system-oriented results are important, they can explain why. They could provide some usage statistics for their system and describe other real-world use. They can also propose better evaluation studies, including those that involve users as they allude to in their paper.

Overall this is good work, but it could be much better, and we will hopefully learn more as the system is used, the test collection grows, and more complex tasks beyond ad hoc retrieval are evaluated with it.

review_title: Evaluation of a rapidly deployed search system for the CORD-19 data set

Decision: Accept (Abstract only)