Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: We might say, neither NMF nor HNMF is a novel idea in topic inference, regardless the known LDA in history and powerful Transformers variances (If one use BERT or SciBERT to augment semantics in the research) in current days. However, this work is fairly well designed, and I believe it reaches sufficient quality level published in EMNLP workshop.

Strong points:
1. The overall scientific design is reasonable. The manuscript is fairly well-written.
2. In the implementation section, algorithm 2 is carefully designed and it assures the reasonability of topic amount selection. 

Weak points:
1. It seems very doable for this work to provide "raw data/codes" reproducibility, but authors did not mention it. ---This is just a humble suggestion, though.
2. Introducing Word2vec in WMD makes sense, that is true. However, in the meantime, currently there are quite a lot new focuses in semantics representation of phrase or sentence, instead of single lexicon. Conventional method could be augmented with new idea so as to explore more in depth.
3. Somehow, some baseline methods could be introduced, implemented and tested. Before jumping into the analysis of the final 52 (sub-)topics, it is of utmost importance to make the HNMF algorithmic part convincing.  


review_title: Conventional topic inference work with delicate algorithmic design form a nice pipeline to trace Covid-19 literature 

final_score: rating: 7: Good paper, accept
review_content: This paper presents a topic modelling system based on hierarchical nonnegative matrix factorisation. The paper itself is very well written and structured, there is a clear logical progression between sections and ideas. The accompanying website is easy to use, though can be improved in terms of visual representation.
Though the paper does not introduce a novel method, it makes several meaningful contributions including the following:
1. The authors augment Kaggle's COVID-19 dataset by additionally searching 4 databases, removing non-English publications and articles without abstract or full text, which helps to build more meaningful and relevant topic models
2. The resulting model helps to avoid flat structure and discover meaningful subtopics on as many as three levels of nesting. Some subtopics can belong to several major topics and the authors provide examples to justify this. Overall, such hierarchical structure can be useful for researchers to explore the topics of interest in more detail.
3. The topics are thoroughly analysed in terms of major issues discussed in COVID-19 literature, thus presenting a valid use-case scenario.  

However, there are some weak points in terms of reproducibility and implementation:

1. The corpus is not provided, and it is impossible to reproduce based on the information in the article.
2. It is not clear how the texts were preprocessed. The authors state that "words deem to be irrelevant such as “copyright” or “et al” are removes”, but they provide no criteria for this judgment, such as a publicly available stop-words list or, say, 50 most frequent words in the data set. Again, this makes the paper impossible to reproduce and can potentially affect the results of the topic model. For example, if the authors manually checked the texts for “irrelevant” words and then removed them, that would lead to much more relevant and coherent results and thus we’d overestimate what the model can do.
3. More of a suggestion - though the topics look coherent by “eyeballing”, you might consider using a more reliable coherence metric than Mimno et al. 2011, which has the worst correlation with human evaluation (Röder, M., Both, A., & Hinneburg, A. (2015, February). Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining (pp. 399-408).)
review_title: Useful and well-written paper

final_score: rating: 7: Good paper, accept
review_content: The paper describes the creation of a dataset with scientific papers about COVID-19 that have been divided in a three-layer structure based on the topics covered. The authors have used hierarchical nonnegative matrix factorization to organize the topics. A total of 52 major topics were found. The authors evaluated the results trough different measurements. The work is of high interest given the ability of "classify" papers based on specific and concrete topics, allowing to filter the large amount of literature that has been produced in the last months with respect to COVID-19, being an useful tool. My only concern is that I cannot see where is possible to download the dataset used to create the topic structure, or if could be possible to use the tool for new articles classification that will appear in the future.
review_title: Nice paper with practical/useful results