Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors describe the development of a task-informed document retrieval framework that leverages latent factors learned through topic models. The framework is applied to the CORD-19 corpus and compared to a naıve keyword-based document retrieval approach. The authors note that they are not able to quantitatively compare the keyword and topic-modeling based approaches on the basis of precision and recall in the absence of relevance labels. 

The key intent of the CORD-19 dataset was to facilitate automation of knowledge synthesis for specific questions. Given the exponentially increasing number of studies, automation of knowledge synthesis would require a model to, as precisely as possible, identify evidence that is most relevant and contains information which can be analyzed to provide answers to those questions. While the idea of topic modeling is interesting, the authors do not provide any indication that such an approach, as described, would be able to support any relevant evidence synthesis. Therefore, it is impossible to comment on the model and draw conclusions on its usefulness or advantages when compared to other information/document retrieval methods.


review_title: Document Retrieval Using Topic Modeling

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper presents a task-informed document retrieval framework for search over the CORD-19 dataset. The authors suggest that latent variables found in topic distribution of each task and document can contribute to the higher quality retrieval, specifically, they score each document by the JSD between its topic distribution and that of the task at hand. The approach is well described and sufficiently detailed throughout the paper, but lacks better motivation – why the authors think this method is likely to outperform the keyword-based search? Some concrete examples could be helpful. Otherwise, the paper is clear, coherent, and very well-written.

My main concerns about this work lie in two main aspects: (1) the lack of comparison to traditional methods in IR, e.g., vector space, probabilistic and language models, and (2) insufficient evaluation.

Re (1): Why using a somewhat simplistic keyword match model as the baseline while there exist established and successful approaches to IR? I found the proposed methodology creative and potentially novel and would be curios about how it compares to other IR techniques.

Re (2): Considering the two approaches presented in the paper, the evaluation (presented mainly in section 4.2.3) makes is difficult to draw any conclusions: figures 4-5 present highly expected results (“by the virtue of the ranking”); surprisingly little documents are in the overlap of the two approaches, which highlights the need in adequate evaluation; no actual comparison between the two approaches in conducted. The authors could use, e.g., the TREC-COVID labeled dataset for evaluation.

I thought that was an interesting and creative approach, and the paper was a very enjoyable read overall. The two issues above, however, prevent me from recommending it for acceptance. With an adequate evaluation (and proven benefits of the suggested approach) this could be a very good work.

Minor:
Section 4.1.1 could be shortened or omitted; topic model coherence seems a bit off-topic.

Figure 2 could be clearer if presented in negative (i.e., light) colors.

Page 3: “pipeline pipeline” -> “pipeline”

Page 7: “include include” -> “include”

Page 4: why 35 was chosen for k?

review_title: An interesting and creative approach to document retrieval but lacking adequate evaluation.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The study proposed to use NMF to extract hidden topics from the COVID-19 Open Research Dataset  for 17 tasks, which was used for information retrieval for each task.  The method is quite standard and straight-forward. 

Strength of this paper is : i)it showed the effectiveness of NMF over keyword approach.  ii) the examples given in section 5 are inspiring.

However, there are some weaknesses: i) The keyword approach is rather a simple baseline, authors may consider other vector space models or language models for IR. ii) For evaluation, the paper did some effort in assessing topics, it would be better to evaluate the IR results further. e.g., do some extrinsic evaluation on down stream tasks and see if the retrieved results are really beneficial; iii) Not sure whether the two key word sets are good or not. Especially in the second set, it returned some noisy words (e.g. best) using tfidf for key word selection.

Some questions for authors: In algorithm 1, what is ψ exactly ? how did you do the dimension projection (reduction) ?
review_title: Interesting work but not solid enough