Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Core review**

This paper proposes a methodology for curating a topic model from CORD-19 documents to improve downstream NLP tasks in this domain. Authors argue about the importance of curated vs fully automatic models and outline a set of requirements that a high-quality topic model should have, in order to be useful for end-users. The paper also provides some preliminary analysis on the current state of this endeavour, but details are still too superficial to evaluate if the quality of the curated topic model is indeed a significant factor for improving other downstream tasks like QA or IR.

I consider this paper could spark an interesting discussion in the workshop given the inter-disciplinary nature of the work, and this type of discussion is very necessary if the biomedical NLP research community pretends to produce results that are usable in practical scenarios. My primary concern is that the results are still very initial to evaluate the feasibility of the proposal or estimate the value that this resource could provide. The authors appear to have previous experience in this type of task, which is an argument in favour of their success. However, I would have appreciated more details that could help quantify the expected returns of this effort.

**Reasons to accept:** A very interesting and important discussion that can potentially benefit the research community and a very good exposition of the necessity of this resource and its requirements.

**Reason to reject:** Preliminary results are too superficial to allow evaluating the feasibility 
review_title: A potentially important contribution but still too initial to evaluate its merits

final_score: rating: 6: Marginally above acceptance threshold
review_content: 
This paper presents a description of ongoing work involving curated topic modeling for improved exploration of COVID-19 literature. The idea is based on including humans in the loop, so that the initial topic model can be better fit to experts’ needs. The authors reflect on a list of desiderata that would make the use of topic models more acceptable to medical practitioners, such as the use of readable labels and structured search. Then, they discuss their ongoing efforts in this direction. 

The paper is clearly written and a pleasure to read. I especially liked the discussion of the desiderata that would enable easier adoption by medical experts, and the description of the intended use where the topic labels would be used in a way similar to MeSH terms.

The actual work in progress is presented in broad terms only (with a few technical details for the more trivial parts such as preprocessing), which is in my opinion the biggest shortcoming of the paper. I am wondering, for example, how the model is precisely recomputed after the user has provided feedback (incorporation of label information, deletion of words, removal of topics etc.). And, how are the topic labels assigned? More generally, since the document collections are ever growing, how do the authors plan to adapt the topic models when the new data comes in?

I would like to draw the attention of the authors to a related paper at this workshop, which also studies medical multi-word expressions in topic models: https://openreview.net/forum?id=c-TkXmZC-Yk (Improved Topic Representations of Medical Documents to Assist COVID-19 Literature Exploration). The paper also contains other ideas about refining topic models for COVID-19.

Editing remarks:

When introducing the last desideratum, the authors mention that special purpose category systems are needed for COVID-19, but the link to topic models was initially opaque to me. It  became clear only at the end of the section.

The acronyms DSM and RDoC may be unfamiliar to the reader.


Overall: Although I would hesitate to accept the paper for its empirical contributions, I do think the paper could lead to interesting discussion at the workshop due to its contemplating nature about the inclusion of an NLP component into medical literature search.

review_title: A good reflection on bringing topic models to medical experts, but lacking in detail

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents preliminary work towards a topic model for use by Covid-19 researchers, being developed with a human-in-the-loop process. It explains the needs of the subject matter experts (SMEs) who are to use the model, and how fully automatic topic models do not meet their needs. They present the work they have done so far in constructing automatic topic models as a starting point, and describe their plans for the involvement of humans.

Reasons to accept: Their analysis of the desiderata for topic models, and their anticipated uses, is extensive, highly compelling, in line with my own experiences of attempting to deploy topic models and similar systems, and likely to stimulate interesting discussions at the workshop. The initial NLP work looks reasonable, although the details there are vague. The final resource is likely to be of value to NLP workers and SMEs alike.

Reasons to reject: Their description of the plans human involvement is frustratingly brief and vague; the most useful part seems to be a pair of references to other group's work in 2017 and 2018. I get the impression that plans have not been finalised. The vaugeness is also a problem with their descriptions of the NLP work already conducted - the descriptions are a long way from providing reproducibility. Finally, the description of the specific challenges of working with Covid-19 research, rather than biomedical research in general, is minimal.

Overall: This work is highly preliminary, and thus lacking in detail, and as such, in my opinion, publication of this work at this stage would be premature. However, much depends on what the organisers aims for the workshop are; it would make for a stimulating presentation and lively discussion on the day, and if the organisers prioritise this above making a long-term contribution to the literature, then maybe it should be accepted.
review_title: Highly preliminary, but with a compelling analysis of desiderata.