Venue: EMNLP 2020 Workshop NLP-COVID

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The work compares traditional topic models based on word tokens with topic models based on medical concepts and suggests several ways to improve topic coherence and specificity.

Latent Dirichlet allocation is the most commonly used method for topic-based analysis of documents. The authors have pointed out that conventional LDA models work well for topically-diverse document collections but they are less informative in narrow, knowledge-rich domains like medicine and specifically when the corpus consists of documents related to one broad topic, such as coronavirus-related literature. 

The experiments have been conducted on the CORD-19 dataset. Three different models have been considered based on three different input representations of the text: word tokens using NLTK tokenizer, concepts based on the Unified Medical Language Systems concepts and non-generic concepts with more general concepts filtered out.

Human evaluation was performed by two annotators to evaluate the quality of topics by labelling them as incoherent, specific or generic. Although the definitions of these labels are quite vague, the Cohen’s Kappa measure of inter-annotator agreement was 0.87. The necessity of human evaluation has been well explained.

After the models are trained, the dataset is subdivided based on the most prevalent topic in each document and then an LDA model was trained on each subset. Two approached have been proposed for further experimentation: first one based on non-generic concepts and the second one based on non-generic concepts re-weighted based on the log-likelihood.

It has been empirically shown that non-generic concepts based topic modelling makes the identified topics as more specific and less incoherent and less generic.

review_title: Non-generic concept based topic modeling makes the identified topics more specific 

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work the authors present different approaches to improve the topic coherence and specificity of topic representations using LDA with an application to the CORD-19 dataset. The authors do a great job in framing the problem and presenting their approach. Most of the details of their methodology are found in the presented text (document preparation, number of topics), but a few things could be expanded further as other reviewers have pointed out. The evaluation part is very brief and key details about the annotation/evaluation instructions and specifics of how the topics were separated into generic and specific is definitely needed. Did the manual reviewers had access to only the abstract/first 2 paragraphs of paper? or did they had access to the full paper to make their decision on topic relevance? the agreement between reviewers is quite excellent, so a bit more in-depth discussion about this would have been nice to have. The point of using humans instead of automatic metrics for the topic coherence evaluation is essential for this kind of work.  The experimental evaluation nicely characterizes the need of a certain 'guide' (while not being exactly guided-LDA) for the LDA models to be coherent by feeding them concepts of higher relevance in the clinical context. Table 1 is an excellent result. The topic tables on page 4 are nice, but should have been reduced as this didn't leave much space to the authors for a longer discussion and conclusion sections This last section seems to be missing and replaced with an odd "Related Work" section at the end which is not really a satisfactory conclusion to this paper as some of the test should be featured earlier on the paper.  Overall, very solid start of the paper that seems to run out of space near the end, and leaves important details out.
review_title: Interesting approach, paper feels a bit rushed

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper describes how an LDA topic model was improved by using concepts. The evaluation - performed by humans - shows that the more refined topic models (using medical concepts) are indeed "better", meaning here more specific and less incoherent.

The conclusion is not by itself surprising: LDA models are known to be very sensitive to pre-processing (including lemmatization and stop-word removal). This is one of the reasons why embeddings became so powerful as they are less impacted by that. In this sense, comparison with a neural topic model would have been an interesting, and helpful to guide practitioners as well as to have a data-point of the usefulness of old-and-tested LDA vs more novel models.

I found the paper an interesting experience nugget... something which we don't see so much and which practitioners might benefit. The substance however is very limited, as it basically consists of a report of a human evaluation campaign. In addition, I have doubts about the following:

 - what exactly are the documents that are provided as input to the concept-LDA? Are the words enriched with concepts, or are those documents just a sequence of concepts?
 - how is a topic represented to an evaluator? By its 5 most frequent words, or some other way?
 - so what? It is not clear to me why having more "specific" topics would be helpful in the end. Can you relate the 7 specific word-topics to some of the concept-tokens (eg, by comparing the probability distribution over words)? Do they subdivide the space further, or is it a totally different soft clustering? How will this help the final application?

review_title: Interesting analysis, but lacks substance

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes an improvement to LDA-based topic modelling by using only important concept words. It proposes to test whether limiting the vocabulary to UMLS concepts, and a smaller set of non-generic UMLS concepts improve the topics extracted by an LDA method on the CORD-19 dataset. It evaluates the quality of these topics through manual evaluation as incoherent, generic or specific. The paper finds that non-generic UMLS concepts provides improved topics.

This is an interesting and well-written paper that I found easy to understand. While I do find it believable that non-generic UMLS concepts improve over normal LDA methods, I think this paper could do more to evaluate this compared to other methods. Furthermore, it is not clear what this paper adds to the COVID-specific text mining area. I would suggest that it could be fleshed out more and sent to a more general NLP venue. The ideas proposed in Section 4.2. seem intriguing but are too quickly explained and explored.

Comments & Concerns
- The core issue is that this paper cites multiple other extensions to LDA but doesn’t attempt to evaluate them as comparisons.
- Just a note, but why aren’t both evaluation annotators authors on the paper?
- The annotation task didn’t sound pretty vaguely defined so I’m impressed by the high agreement between annotators. That’s excellent.
- Anti-HIV treatments for cancer seems like an odd topic, especially in a coronavirus corpus
- The Data section mentions concepts and non-generic terms before they are defined, which is confusing
- I know it was likely down to the page limit, but it’s weird to have the concluding remarks in the Related Work section.
- I think a stronger case has to be made for fixing the number of topics across all methods (to 25). While it makes the methods comparable using raw counts, it isn’t clear that good methods won’t be unfairly penalised by this.

review_title: A simple good idea that needs better comparisons