Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The author found that the previous methods often suffer topic collapsing issues, which leads to repetitive topics, insufficient topic discovery, and also damages the model interpretability. Motivated by such findings, the paper proposed to add Embedding Clustering Regularization (ECR) to topic modeling models. The ECR helps to keep the topic embedding at the center of word embedding clusters and then alleviates the topic collapsing issue. The proposed method is evaluated on four benchmark datasets and the experimental results show encouraging performance.
strength_and_weaknesses: Strength:
The proposed method regularizes the topic model such that the topic embeddings are located at the center of word embedding clusters. This is demonstrated in the t-sne Figure. Since the clusters are separated, we can observe improvements in terms of TD in both Table 2 and Table 3.

Weakness:
1. As the ECR is proposed as an additional module to further regularize the topic models, it will be great to also apply the ECR to other baseline approaches. 
2. It will be nice to have some proof or analysis to explain the necessity of forcing the topic embeddings to be the center of the word embedding clusters. 
3. More experiments can be conducted to enhance the current findings. Such as running the experiments with different K, ranging from 10 to 100 with a fixed step size.

clarity,_quality,_novelty_and_reproducibility: 1. Why the results of DKM and DKM+entropy are not included in the main result tables? 

2. Compared with DKM, the topic coherence of ECRTM is much lower. Do you think it is meaningless?

3. Since the number of labels of the datasets is small, why do you use a large K?



summary_of_the_review: The paper proposed to add Embedding Clustering Regularization (ECR) to topic modeling models. The ECR helps to keep the topic embedding at the center of word embedding clusters and then alleviates the topic collapsing issue. The proposed method is evaluated on four benchmark datasets and the experimental results show encouraging performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper tackles the well known and widely observed issue of topic collapsing in neural topic models (NTMs). 
- Specifically, it proposes a novel embedding clustering regularization (ECR) objective, in addition to reconstruction error objective, during training variational autoencoder (VAE) based NTM which separately aggregates the word embeddings in clusters and apply constraint on topic embeddings to be within those clusters.
- A new topic model is further proposed which jointly optimizes using reconstruction error objective and the ECR objective to extract non-collapsing, diverse and coherent topics.
- the paper takes inventive steps in addressing topic collapsing in neural topic modeling - however there are limitations, including multi-sense/ambiguous words in topics and comparisons to traditional baselines such as Orthogonal topic modeling.


strength_and_weaknesses: Strengths:
- The motivation is well founded and the claims are sound.
- Paper is clearly presented and easy to follow.
- Mathematical formulation is explanatory and easy to understand.
- Proposed ECR objective is robust and effective.
- Experimental evaluation is extensive and the proposed topic model consistently outperforms multiple NTM baselines on topic quality and topic diversity metrics with significant margins over multiple datasets.

Weaknesses:
- there are limitations to the proposed approach, including multi-sense/ambiguous words in topics and comparisons to traditional baselines such as Orthogonal topic modeling.
- Contributions are incremental and Novelty is limited.
- missing baselines and experimental comparisons such as Orthogonal topic modeling and neural topic models such as DocNADE or iDocNADEe [2]
- topic analysis (qualitative) is unclear and seems biased to initial (3) set of topics - however should be analyzed for all topics extracted (extend the appendix) 

Question:
- how is this work compared to Orthogonal Topic Modeling? 

References:
1. Probabilistic Text Modeling with Orthogonalized Topics. SIGIR 2014.
2. Document Informed Neural Autoregressive Topic Models with Distributional Prior. AAAI 2019.

clarity,_quality,_novelty_and_reproducibility: Quality:
- Topics in Table-1 do not look very informative and coherent. So, is the issue of topic collapsing present in coherent and incoherent topics alike or is it skewed?
- The proposed ECR objective generates sparse soft-assignments of word embeddings to topic embeddings which makes sure that each word embedding only belongs to one topic mainly. So, how this method will deal with the words having multiple semantic meanings like "bank" (river bank, financial bank), "chip" (potato chip, electronic chip), "apple" (fruit, company) etc. belonging to multiple topics?
- How is the performance of ECRTM using contextualized word embeddings like BERT, for initialization of word embeddings matrix, to correctly identify the semantics of ambiguous words compare with non-contextualized word embeddings like GloVe?
- Table-5 (Qualitative analysis): Comparison of 3 unrelated topics of baseline methods and proposed method is uninformative. It would be better to compare all 50 topics (top 5 or 10 words for K = 50) of any baseline method and the proposed method for a better picture regarding topic collapsing and coherency. With such full comparison of all topics it can be checked if good quality and coherent topics which are present in the topics extracted using baseline models are absent or present in the topics extracted using proposed models.

Clarity:
- Typographical errors:
	- [Section 3.2, 2nd paragraph] word distribution x -> word distribution beta
- A figure of the proposed ECRTM model with joint optimization objective would boost understanding of the model architecture and working.
- A table of notations would improve readability of the mathematical formulation.

Novelty:
- Related research has already worked on clustering of word embeddings to extract topics. Although they are not technically topic models, the inspiration for word embedding clustering to extract coherent and diverse topics can be derived. However, the issue of topic collapsing is very critical and the proposed method is promising and should be utilized in future works on neural topic modeling.
- How is this work compared to Orthogonal Topic modeling [1] ? 

Reproducibility:
- Experimental setup and hyperparameter settings are described in detail.

summary_of_the_review: - Although novelty is limited in this paper, but the direction it explores to eliminate topic collapsing issue and generate diverse and coherent topics is interesting and important as diverse topics generate high quality document-topic representations which further boost accuracy in downstream supervised or unsupervised NLP tasks.
- missing comparison to (related) baselines methods (please see weaknesses)
- additionally, please see weaknesses section
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an embedding clustering regularization technique to mitigate the posterior collapse
problem in neural topic models. The regularization technique forces the topic embeddings to be the
centers of separately aggregated clusters of word embeddings. As a result, distinct clusters of topic-
word embeddings are formed wherein each topic comprises distinct words and every topic is distant
from one another in the semantic space. This paper evaluates their technique for different aspects
of topic modeling such as topic quality (using coherence) and diversity and in addition also measures
topic-doc distribution and document classification accuracy. It outperforms several other methods that
it compares with on almost every evaluation metric that they use.
strength_and_weaknesses: # Strengths

- Clear illustration of limitations of existing methods. It is also clearly illustrated that the proposed model
overcomes those limitations.
- A sound theoretical solution to the limitations is presented.
- Overall good results when compared to the selected baselines

#Weaknesses
- The baseline selection is poor. The authors claim that it is difficult to incorporate Dirichlet distribution
in VAEs. This is not entirely true. There has been work by (Burkhardt & Kramer, 2019) on using
Dirichlet as prior for VAEs in topic modeling. This work was again reimplemented by (Hoyle et al.,
2021). This has been overlooked by the authors in the paper. The compared baselines for illustrations of
limitations also do not use Dirichlet prior, which was initially presented to overcome the said limitations.
- The evaluation includes Cv score, which for some experiments is very low. It is recommended to include
NPMI scores as well.
- If I understand the sparse soft-assignment clustering correctly, this technique will assign every word
to one and only one topic cluster but there are several words that belong to multiple contexts for
example, season, which can be related to weather and sports both. So, the proposed model cannot
assign words that occur in different contexts to different topics.
- The most obvious limitation is that this method produces fixed size clusters of all topics. On page
5, section 3.4, they assume that every topic includes same amount of semantic information and this
assumption is supported by practically forcing every topic to have same number of words. This means
having same amount of semantic information is equal to having same number of words. I don’t quite
agree to this assumption because semantic information of a topic cannot be quantified by the number
of words present in it.
- Lastly, posterior collapse is a significant and long standing problem in several other text (and
image) applications such as text generation, summarization etc. The method performs considerably
well for topic modeling but unfortunately does not extend or discuss how it can be generalized to the
other applications.
clarity,_quality,_novelty_and_reproducibility: # Clarity

The paper is written very clearly and the best part is that the author(s) address the limitations or
future research opportunities in this topic wherever necessary. Almost all topics discussed are easy to
follow and understand. This paper sufficiently covers the context of the topic through the mentioned literature which also
seems to be quiet relevant to the work. It also discusses how traditional topic modeling is different
than the recent neural topic models and how the clustering of word embeddings is different than
modeling topic-word distribution. Although these ideas and their differences are covered, not
all of them are presented very well. Some sections can be co-ordinated better and for any section the main ideas should be
discussed to the point

# Quality

- it is mentioned that it is difficult to use the Dirichlet distribution with neural topic models. This is a completely unfounded claim. There are more than five different ways of using Dirichlet priors in VAEs, one is implemented in Pytorch as default where you just have to use the Dirichlet, and the reparameterization is done automatically, and most of them are very easy to use. It has been shown in previous work that the Dirichlet prior gives much better results than the Gaussian one or the Laplace approximation.
- Overall quality of the paper with respect to the posterior collapse problem is fair. Some sections can be written more to the point by focusing on the central
idea of the section at the start of the para. For example, in section ”Fulfilling Preset Cluster Size
constraints” where formation of empty clusters seems to be the main agenda of the paragraph but it
is only discussed towards the end.

# Originality

The novelty in this paper is to produce sparse soft-assignments using Deep KMeans method (Deep
k-means: Jointly clustering with k-means and learning representations) by restricting each word em-
bedding to belong to only single topic embedding by pushing it closer to only ”one” topic embedding
and away from other topic embeddings while also constraining cluster sizes to be equal for all topics. The idea seems to be simple yet significant when tackling posterior collapse problem in topic models.
Although, it is unclear if such separation of topic-word clusters would lead to some other problems
such as latent holes which might produce undesirable results.

# Reproducibility

Code does not seem to be available. Otherwise, datasets and benchmarks are standard and the method is described well.
summary_of_the_review: The work in itself is novel and overcomes topic collapsing limitations. But, the baseline selection is incomplete.
Methodology is clearly explained. There are nice illustrations of limitations and results which suggest that
the proposed model is effective. However, without comparison to SotA baselines, it is difficult to evaluate the results.

Solving a notorious problem of posterior collapse in topic models is the main contribution of this paper.
Although, it is based on the prior clustering techniques such as Deep Kmeans, some small tweaks in
their methodology seems to significantly improve the topic collapsing issue and this paper successfully
demonstrates it via empirical setup. The qualitative results shown also seem to be promising. The
approach outperforms many recent topic models on several evaluation criteria such as topic quality
(coherence), diversity and downstream document classification task but there is still some scope for
rigorous experiments to demonstrate the robustness of the idea against different lengths of text, different
topic sizes and other downstream tasks such as document retrieval and also if it can be generalized for
other NLP tasks.




Open questions:
- What about the robustness of the model against the number of topics? (The results are shown only
for K=50,100)
- Co-occurence information between words is comparatively low for short text such as tweets or news
headlines. How does the method perform on short (short text dataset M10) text sequences?
- Any specific reason for not comparing the results with those from Zhang et al paper (Is Neural Topic
Modelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings
for Topics)?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work presents (yet another) Neural Topic Model (NTM). The main idea is to add a regularization term to force the topic vectors to be away in the space by using a clustering mechanism, which lead to decrease the collapsing problem. The full model, named Embedding Clustering Regularization Topic Model (ECRTM) is based on previous works following an Optimal Transport scheme. It is favorably compared to previous models on a set of classic (and limited) datasets.
strength_and_weaknesses: Strength:
- Adding a regularization term, but also other mechanisms (such as a priori on topic size), improves the results on some datasets

Weaknesses:
- Several mechanisms need to be combined to design the final model (clustering with soft assignment, prior on topic size, reintroduction of the "less sparse beta"). To this extent, stacking various techniques to get the final system introduces some doubt in what really explains the success of the approach. The ablation study of 4.3 doesn't really answer to this concern.
- As in previous approaches (which usually seem to overlook more than 20 years of research in topic model), the experimental framework is questionable. First, the perplexity cannot be discarded as easily: it is clearly not sufficient to assess the quality of topics, but it can be added to check whether the topics can explain/cover the full dataset. It's related to argument (2) p.2 and the way the authors address it is not convincing to me, even though it has already been used in previous ICRL papers. We shouldn't confuse topic modeling and clustering, many papers have been written on that subject. Evaluating the capacity of a topic model to cluster documents into a limited number of quite well-separated clusters (which is the case in most 20 newsgroups classes) is clearly not sufficient to evaluate the doc-topic distribution quality. Another remark: the C_V measure of Röder is not that much used in the literature and a couple of concerns (in term of reproducibility) have been raised online. I suggest to use UCI or UMass instead.
- Recent topic models should be able to deal with more diverse datasets than news and review.
clarity,_quality,_novelty_and_reproducibility: The paper is reasonably well written, even though there is some redundancy while some parts should have been a little bit more explained to make the paper more self-contained (e.g., the quick reference to Canas & Rosasco to explain Eq.2).

The paper is very focused on NTM approaches with a lapidary related work section. Many previous works have tried to leverage word embeddings and some of them may integrate the section. For instance, the work of Das et al. (https://aclanthology.org/P15-1077.pdf), who model the word embedding as a realization of a gaussian centered at the topic vector, seems really related to me.

The new regularization term, based on a clustering approach, looks quite new, even though it's used to address a problem that has been introduced by the neural models. However, the setup seems to need the addition of several "tricks" to make the whole thing works.

I didn't find any code for allowing an easy reproduction of the experiments. The pseudo code is given, so it should be possible to do it by yourself... maybe.

summary_of_the_review: This paper doesn't bring any big improvement in the topic modeling area. The regularization term may be somewhat useful in order to avoid topic collapsing, but the experiments fall short to really prove that the new model will be effective in a large range of dataset profiles.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a novel neural Topic model ECRTM which is claimed to resilient to the problem of Topic Collapse.
It argues that a clustering based regularisation will yield topics that are more well-separated and hence mitigate the problem of Topic collapse.
It sets up an objective which augments the loss by an additive term to enforce the regularisation.

The paper is clearly written and the main claims are well communicated. Experimental evaluation shows that the proposed method outperforms chosen baselines.

Given that Topic modelling is a widely researched area over last two years the proposed novelty is modest.
Concerns are raised about selection of baselines. Measures such as perplexity were not reported.

A more rigorous characterisation of separability of Topics may help the claims.


justification_for_why_not_higher_score: None of the referees raised their scores. Also given that Topic modelling is a well researched area, current results does not seem to make 
too much progress
Decision: Reject