Neural Topic Modeling with Embedding Clustering Regularization
Authors: Xiaobao Wu
Xinshuai Dong
Thong Thanh Nguyen
Anh Tuan Luu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Topic models have been prevalent for decades with various applications like automatic text analysis due to their effectiveness and interpretability. However, existing topic models commonly suffer from the notorious topic collapsing issue: the discovered topics semantically collapse towards each other, leading to highly repetitive topics, insufficient topic discovery, and damaged model interpretability. In this paper, we propose a new neural topic model, Embedding Clustering Regularization Topic Model (ECRTM), to solve the topic collapsing issue. In addition to the reconstruction error of existing work, we propose a novel Embedding Clustering Regularization (ECR), which forces each topic embedding to be the center of a separately aggregated word embedding cluster in the semantic space. Instead of collapsing together, this makes topic embeddings away from each other and cover different semantics of word embeddings. Thus our ECR enables each produced topic to contain distinct word semantics, which alleviates topic collapsing. Through jointly optimizing our ECR objective and the neural topic modeling objective, ECRTM generates diverse and coherent topics together with high-quality topic distributions of documents. Extensive experiments on benchmark datasets demonstrate that ECRTM effectively addresses the topic collapsing issue and consistently surpasses state-of-the-art baselines in terms of topic quality, topic distributions of documents, and downstream classification tasks.


Under review as a conference paper at ICLR 2023
NEURAL TOPIC MODELING WITH EMBEDDING CLUSTERING REGULARIZATION
Anonymous authors Paper under double-bling review
ABSTRACT
Topic models have been prevalent for decades with various applications like au- tomatic text analysis due to their effectiveness and interpretability. However, ex- isting topic models commonly suffer from the notorious topic collapsing issue: the discovered topics semantically collapse towards each other, leading to highly repetitive topies, insufficient topic discovery, and damaged model interpretability. In this paper, we propose a new neural topic model, Embedding Clustering Regu- larization Topic Model (ECRTM), solve the topic collapsing issue. In addition to the reconstruction error of existing work, we propose a novel Embedding Clus- tering Regularization (ECR), which forces each topic embedding to be the center of separately aggregated word embedding cluster in the semantic space, Instead of collapsing together, this makes topic embeddings away from each other and cover different semantics of word embeddings. Thus our ECR enables each pro- duced topic to contain distinct word semantics, which alleviates topic collapsing. Through jointly optimizing our ECR objective and the neural topic modeling ob- jective, ECRTM generates diverse and coherent topics together with high-quality benchmark datasets
topic distributions of documents. Extensive experiments demonstrate that ECRTM effectively addresses the topic collapsing issue and con- sistently surpasses state-of of------trhrr art baselines in terms of topic quality, topic distri- butions of documents, and downstream classification tasks.
(a) ETM
(b) NSTM
(c) WeTe
(d) ECRTM
Figure 1: t-SNE visualization (van der Maaten & Hinton 2008 of word embeddings and topic embeddings (A) under 50 topics. These show that while the topic embeddings mostly collapse together in previous state-of-the-art models (ETM (Dieng al., 2020), NSTM (Zhao et al., b02 and WeTe (Wang et al. 2022)), our ECRTM successfully avoids the collapsing by forcing each topic embedding to be the center separately aggregated word embedding cluster.
INTRODUCTION
Topic models have achieved great success in document analysis via discovering latent semantics. They have various downstream applications (Boyd-Graber et al. (017)) like content recommen- dation (McAuley & Leskovee) 2013), summarization (Ma et al. 2012). and information retrieval (Wang et al.] 2007). Current topic models can be roughly classified as two lines: conventional topic models based on probabilistic graphical models (Blei et al. (003)) or matrix factorization (Kim et al.
2015
Shi et al. 2018) and neural topic models (Miao al. 2016
2017 Srivastava & Sutton 2017).

---

Under review as a conference paper at ICLR 2023
However, despite the current achieve- ments, existing topic models com- monly suffer from topic collapsing: the discovered topics tend to seman- tically collapse towards each other (Srivastaaa & Sutton 2017). as ex- emplified in Table [ We see these collapsed topics include many unin- formative and repetitive words. Topic collapsing brings about several is- sues: (1) Topic collapsing results in highly repetitive topics, which are less useful for downstream applications (Nan et al. 2019 Wu ctt al. 2020b). (2) Topic collapsing incurs insufficient topic discovery. Many latent topics are undisclosed, making the topic discovery insufficient to understand documents (Dieng ct al. 2020). (3) Topic collapsing damages the interpretability of topic models. It becomes difficult to infer the real underlying topics that document contains (Huynh et al. (220). In consequence, topic collaps- ing impedes the utilization and extension of topic models; therefore it is crucial to overcome this challenge for building reliable topic modeling applications.
Topic#l: just show even come time one good really going know
Topic#2: just even really something come going like actually things get
Topicil:: just one even something curne way really like always good
Topic#4: just get going come one know even really something way
Topic#5: just like inside get even look come one everything away
Table Top related words of the discovered topics by NSTM (Zhao et al. 202 IMDB. These topics seman- tically collapse towards each other with many uninformative and repetitive words. Repetitions are underlined.
To address the topic collapsing issue and achieve robust topic modeling, we in this paper propose a novel neural topic model, Embedding Clustering Regularization Topic Model (ECRTM), First, we illustrate the reason for topic collapsing. Figure shows topic embeddings mostly collapse to- gether in the semantic space of previous state-of-the-arr methods. This makes discovered topics contain similar word semantics and thus results in the topic collapsing. Then to avoid the collapsing of topic embeddings, we propose the novel Embedding Clustering Regularization (ECR) in addition to the reconstruction error of previous work. ECR considers topic embedding as cluster centers and word embeddings as cluster samples. To effectively regularize them, ECR models the clustering soft soft-assignments between them by solving specifically defined optimal transport problem on them. As such, ECR forces each topic embedding to be the center a separately aggregated word embed- ding cluster. Instead of collapsing together, this makes topic embeddings away from each other and cover different semantics of word embeddings. Thus our ECR enables each produced topic to con- tain distinct word semanties, which alleviates topic collapsing. Regularized by ECR, our ECRTM achieves robust topic modeling performance by producing diverse and coherent topics together with high-quality topic distributions of documents. Figure [d] shows the effectiveness of ECRTM. We conclude the main contributions of our paper as follows:
We propose a novel embedding clustering regularization that avoids the collapsing of topic embed- dings by forcing each topic embedding be the center of a separately aggregated word embedding cluster, which effectively mitigates topic collapsing.
We further propose a new neural topic model that jointly optimizes the topic modeling objec- tive and the embedding clustering regularization objective. Our model can produce diverse and coherent topics with high-quality topic distributions of documents at the same time.
We conduct extensive experiments on benchmark datasets and demonstrate that our model effee- tively addresses the topic collapsing issue and surpasses state-of-the-art bascline methods with substantially improved topic modeling performance.
2 RELATED WORK
Conventional Topic Models
Conventional topic models (Hofmann 1999 Blei et al. 2003 Blei
& Lafferty| 2006 Mimno al. 2009 Boyd-Graber et al. 2017) commonly employ probabilistic graphical models to model the generation process of documents with topics as latent variables. They infer model parameters with MCMC methods like Gibbs Sampling (Steyvers & Griffiths. 2007) or Variational Inference (Blei et al., 2017). Another line of work uses matrix factorization to model topics Yan et al. 2013] Kim al. 20155] Shi et al. 2018). Conventional topic models usually need model-specific derivations for different modeling assumptions.
Neural Topic Models Due to the success of Variational AutoEncoder (VAE, Kingma & Welling 2014 Rezende al. 2014), several neural topic models have been proposed (Miao et al. 2016
2

---

Under review as a conference paper at ICLR 2023
2017
Srivastava & Sutton 2017: Burkhardt & Kramer 2019: Wu et al. 2020a; 2021 Dieng al.
2019 2020 Meng et al. 2020 Zhao et al. 2021a). Neural topic models commonly optimize the
KL divergence and the reconstruction error between input and output by gradient pack-propagation.
Some studies directly cluster pre-trained word or sentence embeddings to produce topics (Sia et al.)
2020 /Zhan al. 2022), but they are topic models since they cannot infer the topic distributions of documents as required. Recent NSTM (Zhao et al. (2221b) and WeTe (Wang et al. (022) measure the reconstruction error with optimal (conditional) transport distance. However, despite the different reconstruction error measurements, they still suffer from the topic collapsing, producing repetitive and less useful topies (see experiments Sec. 4.2). Different from these, our proposed model aims to address the topic collapsing issue and substantially improve topic modeling performance. In ad- dition to the reconstruction error of these previous work, our model proposes a novel embedding clustering regularization that avoids the collapsing of topic embeddings by forcing each topic em- bedding to be the center separately aggregated word embedding cluster. Then our model learns topics under this effective regularization and particularly addresses the topic collapsing issue.
METHODOLOGGY
3.1 PROBLEM SETTING AND NOTATIONS
We recall the problem setting of topic modeling following LDA (Blei et al. 2003). Consider document collection X with V unique words (vocabulary size), and each document is denoted as x. We require discover K topics from this document collection. The k-th latent topic is defined distribution over all words (topic-word distribution), denoted as Br E RY. We have (= 3x ERRV ERVX xK as the topie-word distribution matrix of all topics. The topic distribution of document (doc-topic distribution) refers to what topics it contains, noted as A E AK. K-
3.2 WHAT CAUSES TOPIC COLLAPSING?
Despite the current achievements, most topic models suffer from topic collapsing: topics semanti- cally collapse towards each other (see Tab We illustrate what causes topic collapsing by analyz- ing a kind of recently proposed tfate-of-the-arr neural topic models (Dieng et al.) [2020) Zhao al., 2021b). These models compute the topic-word distribution matrix with two parameters: B W T. Here Ww wy) E DVV are the embeddings of V words and T= (t,,...., tx) E Dx are the embeddings of K topics, all in the same D-dimensional semantic space. They can facilitate learning by initializing W with pre-trained embeddings like GloVe (Pennington et al. 2014).
However, topic collapsing commonly happens in these models. We believe the reason lies in that their reconstruction error minimization incurs the collapsing of topic embeddings. Specifically, these models learn topic and word embeddings by minimizing the reconstruction error between topic dis- tribution 0 and word distribution x of a document. For example, to measure reconstruction error, ETM (Dieng et al., 2020) uses expected log-likel,hood, and recent NSTM (Zhao et al.. 2021b) and We'le (Wang et al. 2022) use optimal (conditional) transport distance. In fact, words in docu- ment collection commonly are long-tail distributed following Zipf's law (Reed) 2001, Piantadosi speaking, few words are high frequency and most are low firequency. Therefore the reconstruction biased as it mainly reconstructs nigh-frequency words regardless of the recon- struction error measurements. Since topic and word embeddings are learned by minimizing recon- struction error, this biased reconstruction pushes most topic embeddings close to the embeddings of these high-frequency words in the semantic space. As result, topic embeddings collapse together as shown in Figure []] The topic-word distributions become similar to each other, making discovered topics contain similar word semantics. This leads to topic collapsing.
3.3 WHAT IS AN EFFECTIVE CLUSTERING REGULARIZATION ON EMBEDDINGS?
In this section, we explore how to design an effective clustering regularization to address the topic collapsing issue. Our analysis in Sec. 3.2 indicates topic collapsing happens because the recon- struction error minimization incurs the collapsing of topic embeddings in existing work. To solve
Here Ax denotes probability simplex Ax= R ΣK 0c=1
3

---

Under review as a conference paper at ICLR 2023
(a) DKM
(b) DKM
(c) ECR (s=1.0) (d) ECR (s=0.1) 1) (c) ECR (==0.05)(f) Cluster size proportions
Entropy
Figure 2: t-SNE visualization (a-e) of word embeddings (+) and topic embeddings (A) under 50 topics (K=50). (a): DKM cannot form separately aggregated clusters. (b): DKM + Entropy forms clusters but has trivial solution that clusters of most topic embeddings are empty. (c,d,e): Our ECR forms clusters and also avoids the trivial solution of empty clusters with a small e. (f): This quantitatively shows that while most cluster size proportions are zero in DKM Entropy, our ECR successfully avoids this trivial solution with fulfilled preset cluster size constraints, Here all cluster size proportions of topic embeddings are preset equal as 1/K-0.02.
this issue, we propose to design clustering regularization in addition to the reconstruction error of existing work. Considering topic embeddings as cluster centers and word embeddings as cluster samples, we require the regularization to force each topic embedding to be the center of a sepa- rately aggregated word embedding cluster. So instead of collapsing together, topic embeddings will be away from each other and cover different semantics of word embeddings in the space. This will make each discovered topic contain distinct word semantics and thus alleviate topic collapsing- However, it is non-trivial achieve such an effective clustering regularization, and we explore its requirements as follows.
Supporting Joint Optimization As regularize neural topic models, require the cluster- ing regularization to support joint optimization on topic and word embeddings along with a neural topic modeling objective. Some studies (Sia al. 2020) apply classical clustering algorithms, e-g., KMeans and GMM, to produce topics by clustering pre-trained word embeddings. We clarify that they are not topic models as they only produce topics and cannot learn doc-topic distributions as re- quired (but we compare them in experiments). We do not adopt these classical clustering algorithms and some other work (Song et al. [2013] Huang et al. 2014, Xie et al. 2016 Hsu & Lin, 2017]] Yang et al. 2017) as our clustering regularization, because we cannot jointly optimize them along with neural topic modeling objective.
Producing Sparse Soft-assignments We also require the clustering regularization to produce sparse soft-assignments. Even supporting joint optimization, existing clustering methods may still lead to topic collapsing- For example, method, Deep KMeans (DKM, Fard et al. (020)) that supports joint optimization. The clustering objective of DKM minimize the total Euclidean distance between centers and samples weighted by soft-assignments. We use DKM as clustering regularization on topic and word embeddings:
propose to employ the -fate-ofHie af--the-art deep clustering
e -telf/t
ΣΣ [w t2 where
min W.TP
Pjk-
Iw, IP/T
Here Pÿş: denotes the clustering soft-assignment of word embedding w, assigned to topic embedding ts. which is modeled as a softmax function of the Euclidean distance between wj and all topic embeddings (r a temperature hyper-parameter). Unfortunately, DKM still incurs topic collapsing (quantitative results are in Sec. (.3). We see from Figure 2a that DKM cannot form separately aggregated clusters, so the topic embeddings (centers) cannot be separated but collapse together. To solve this issue, we require the clustering regularization to produce sparse soft-assignments-aee each word embedding is mainly assigned to only one topic embedding and rarely to others, which pushes each word embedding only close one topic embedding and away from all others in the semantic space. This way can form separately aggregated word embedding clusters with topic embeddings as centers, which encourages topic embeddings to be away from each other.
Fulfilling Preset Cluster Size Constraints fill preset cluster size constraints. Only producing sparse soft-assignments may still result in topic
We further require the clustering regularization to ful-
4

---

Under review as a conference paper at ICLR 2023
closter center E with size
Algorithm Training algorithm for BCRTM.
Input: document collection X, preset cluster size constraint
number of epochs Repochi Output: model parameters O, W.
1: for Mepoch do
2: for each
(x(1, x(2)
,...,, x()) from X do
3:
// Sinkhorn' al algerithm;
4:
C₁a - ||,, - | V J, k;
5:
M = exp((p-C//e)
6:
Initialize - 1k:
Space
7:
while not converged and not reach max iterations do
Figure 3: Illustration of ECR. ECR clusters word embeddings wj (+) as samples and topic embeddings tx (A) as centers with soft-assinmment Teijk- The cluster size of center tı constrained as Tlk- Here ECR pushes close to t, and away from ty and ts. It is similar for W2-
8:
4 1 Iv
Mh b a
9:
end while
10:
Compute _ t diag( (aoggaaMMiinn (diag
Compute LTM + ^BCR (Eq. (6)); 12: Update θ, W. T with gradient step;
11:
13:
end for
14:
end for
collapsing. To make the soft-assignments sparse, we propose DKM + Entropy that jointly min- imizes the entropy of soft-assignments. Σj= Σ Pys log P₁ı with the clustering objective of DKM (Eq. (T). However, this way still leads to topic collapsing (quantitative results are Sec. (4.3).
k
topic embeddings, but unfortunately the clustering solution trivial-the clusters of most topic embeddings are empty, as quantitatively shown Figure 21 As result, the topic embeddings of these empty clusters can- not be separated to cover distinct semantics but only collapse to others in the space. To avoid such trivial solutions of empty clusters, we propose to preset constraints on the size of each cluster (must not be empty) and require the clustering regularization to fulfill these constraints.
Figure 2b shows this way indeed forms separately aggregated clusters for
3.4 EMBEDDING CLUSTERING REGULARIZATION
To meet the above requirements, we in this section introduce a novel method, Embedding Cluster- ing Regularization (ECR). Figure 3 illustrates ECR, and Figure 2e shows its effectiveness.
Presetting Cluster Size Constraints We first preset the cluster size constraints be fulfilled to avoid trivial solutions of empty clusters. Recall that we have topic embeddings as centers and samples. We denote the cluster size of topic embedding tą ng and the cluster size proportion as st=nv/V. We have s (.,,..., 8K) EA as the vector of all cluster size proportions. In our experiments, we all cluster sizes equal, Ti, -V-,/K, s s=(1K,,... 1/K)T. This is because we often lack prior knowledge about cluster sizes, and is reasonable to assume each cluster (topic) includes the same amount semantic information without loss of generality. Experiments also show this assumption works well (see Sec. 4.2
V word embeddings
is uniform,
Embedding Clustering Regularization (ECR) To meet the above requirements, we propose ECR that models clustering soft-assignments with the transport plan of a specifically defined op- timal transport problem. Specifically, we define two discrete measures of topic (tg) and word em- beddings (w,): 7= y 1. 1ww and Φ= Σ SKDEa, where o denotes the Dirac unit mass on x. We formulate the entropic regularized optimal transport problem between y and 6 as
arg min for, (1, Lor, (7, TERY
[w
11
em (Tgg(()))Tj)) 1),
k=
j=1
s.t. T -
ly and =s..
Note that be determined prior knowledge from experts, and leave this future work.
5

---

Under review as a conference paper at ICLR 2023
Here the first term is the original optimal transport problem, and the second term is the entropic reg- ularization with
E to make this problem tractable (Canas & Rosasco 2012). Eq. (2) is to find the optimal transport plan π. that minimizes the total cost of transporting weight from word embeddings to topie embeddings. We measure the transport cost between word embedding W, and topie embedding t by Euclidean distance: Cソt=||w, - tx and the transport cost matrix CER V The two conditions in Eq. (2) restrict the weight of each word embedding w 4 and each topic embedding t as 8g. where k (Iv) is a K (V) dimensional column vector of ones. Tyk denotes the transport weight from w, to fy: TER VERR is the transport plan that includes the transport weight of each word embedding to fulfill the weight of each topic embedding.
To meet the requirements of an effective clustering regularization, we model clustering soft- assignments with the optimal transport plan Th the soft-assignment of w; to ta is the transport weight between them, Te-jk- Then we formulate our ECR objective by minimizing the total distance between word and topic embeddings weighted by these soft-assignments:
K [w to 1 where क. sinkhorn(7, 0) s arg min Lor, (7, Φ).
LECR
(3)
TER TERVXN
3=1
Here we compute म through Sinkhorn's ' algorithm (Sinkhorn, 1964; Cuturi 2013 Algorithm D fast iterative scheme suited to the execution of GPU (Peyré al.) 2019). By doing so, π. is differentiable variable parameterized by transport cost matrix C' (Salimans et al. 2018 Genevay eet al. 2018). Due to this, minimizing T increases transport cost Cjk. L.e., the distance between W, and tyi otherwise maximizing it decreases the distance (Genevay et al.) 2019). Thus we can exactly model -. as diffcrentiable clustering soft-assignments between topic and word embeddings.
ECR is an Effective Clustering Regularization As shown Figures 2e and 2 ECR is an effec- tive clustering regularization that meets the requirements in Sec. 3.2 First, ECR supports joint optimization since - is differentiable as aforementioned.. Second. ECR produces sparse soft- assignments. is proven that T. converges the optimal solution of the original optimal transport problem when e - 0, which leads to a sparse transport plan (Peyré et al. 2019). Hence ECR (Eq. (3) produces sparse soft-assignments under small E. With sparse soft-assiinments, ECR pushes each word embedding only close to one topic embedding and away from all others, which forms sepa- rately aggregated clusters. We illustrate this property in Figures 2c 2e Last, ECR fulfills preset cluster size constraints. In Eq. (2). the transport plan is restricted by two conditions indicating the weight of each word embedding w, is & and each topic embedding t& is Sk. These ensure the sparse optimal transport plan मे needs to transport 7& word embeddings to topic embedding ty to balance the weight, such that = Accordingly, ECR fulfills the preset cluster size constraints with - as clustering soft-assignments, which avoids trivial clustering solutions of empty clusters as shown in Figure 2i To sum up, our effective ECR forces each topic embedding to be the center of a separately aggregated word embedding cluster. This makes topic embeddings away from each other and cover different semantics word embeddings in the space, which alleviates topic collapsing.
3.5 NEURAL TOPIC MODELING WITH EMBEDDING CLUSTERING REGULARIZATION
In this section, we propose a novel neural topic model, Embedding Clustering Regularization Topic Model (ECRTM) that jointly optimizes the topic modeling objective and our proposed ECR objective. Algorithm shows the training algorithm of ECRTM.
Inferring Topic Distributions We devise the prior and variational distribution following VAE (Kinmma & Welling 2014) to infer topic distributions. Since it is difficult to directly apply the reparameterization trick of VAE on a Dirichlet distribution as in LDA, we employ logistic normal prior to avoid this issue (Srivastava & Sutton 2017). In detail, we draw latent variable, r, from a logistic normal distribution: p(r) = LNN (Mo: Eo). where 14 and Eg are the mean and diagonal covariance matrix. Then we distribution, the mean vector μ= f= (x) and covariance matrix Σ diag(fx) (iiag(f(xx)), (x)). So the variational distribution is qe (f]x))=((u,,) Σ) where @ denotes the parameters of fp and fx. By applying the reparameterization trick (Kingma& & Welling 20144,, we sample r= + //2 where e ~ N/ I). We obtain the topic distribution 0 with softmax function = sof)max(()..
encoder network that outputs parameters of the variational

---

Under review as a conference paper at ICLR 2023
Reconstructing Documents We then reconstruct the input documents with topic-word distribu- tion matrix 3 E x Recall that 33 indicates the weights between all topics and words. Previous methods commonly model B the product of topic and word embeddings. Differently, uses the proposed ECR as a clustering regularization on topic and word embeddings, so ( also needs reflect the learned clustering assignments between them. We do not directly model |3 with the soft- assignments To of our ECR. This is because this way cannot provide sufficient weight information between topics and words for reconstruction since π is very sparse as aforementioned (most values are close to zero). Experiments also show this way is ineffective. Hence we need less sparse / for reconstruction. To this end, we propose model /3 with the less sparse soft-assignment following DKM (Fard et al. 2020):
model
||w, -t/l//
3yk >
-K= --wl
where is temperature hyper-paraee ppper-parameter.. This formulation reflect the learned clustering assign- ments between topic and word embeddings and is less sparse to provide sufficient information for reconstruction. With the topic distribution 0 and the sopic-word distribution matrix B3, we routinely sample the reconstructed document from a Multinomial distribution (3lli())ftmax((33)
mini-batch Overall Objective Function Given a mini batch of N documents (x(")... x N) we formulate
the topic modeling objective function following VAE as
1 CIM
N Σ (+)) log(softmax(380))) KL (r(0 ( {[(((())
(5)
The first term is the reconstruction error, and the second term is the KL divergence between the prior and variational distribution. ECRTM learns topics regularized by our ECR. We define the overall
objective function of ECRTM as combination of LTM (Eq. (5) and LeCr (Eq. 3
LTM + ĐECR LECR e,W,T
min
(6)
where λECR is weight hyper-parameter. This overall objective enables ECRTM to aggregate the embeddings of related words to form separate clusters with topic embeddings as centers and avoids the collapsing of topic embeddings. Thus our ECRTM can alleviate the topic collapsing issue and learn coherent and diverse topics together with high-quality doc-topic distributions at the same time.
4 EXPERIMENT
4.1 EXPERIMENT SETUP
Datasets We adopt the following benchmark document datasets for experiments: (i) 20 News Groups (20NG, Lang 1995) is one of the most popular datasets for evaluating topic models, in- cluding news articles with 20 labels; (ii) IMDB (Maas al. 2011] is the movie reviews containing two labels (positive and negative); (iii) Yahoo Answer (Zhang et al.) 2015) is the question titles, contents, and the best answers from the Yahoo website with 10 labels. (iv) AG News (Zhang al.) 2013) contains news titles and descriptions, divided into categories like Sports and Business. The preprocessing details are described in Appendix A.I
We evaluate topic models concerning topic and doc-topic distribution qual- ity (perplexity is not considered since disagrees with human interpretability (Chang et al.)
Evaluation Metrics
2009
Hoyle et al. 202Th). For topic quality, we consider Topic Coherence and Topic Diversity. Topic
coherence measures whether the top words of discovered topics are coherent (Newman al.
2010
Wang & Blei 201T). We employ the widely-used topic coherence metric, Coherence Value (Cv)
(Röder al. (015). Besides, topic diversity measures the differences between discovered topics to
verify if topic collapsing happens. We use Topic Diversity (TD, D.Dnn al.) (020)) to evaluate topic diversity, which computes the proportion of unique words in the discovered topics. Additionally, we conduct document clustering tasks to evaluate doc-topic distribution quality following ZZhao et al. (2021b): Waan et al.] (2022). We use the most significant topics in the topic distributions of test- ing documents their document clustering assignments and adopt the commonly-used Purity and NMI (Manning et al.] (008) for evaluation. Note that our intention is not to achieve state-of the-art document clustering results but compare doc-topic distribution quality.

---

Under review as a conference paper at ICLR 2023
Table 2: Topic quality of topic coherence (Cv) and topic diversity (TD) under 50 and 100 topics (K 55 and K = 100). The best scores are in bold. I means the gain of ECRTM is statistically significant at 0.05 level.
20NG
IMDB
Model
K=50
K=100
K=50
K=100
Cy
TD
Cv
TD
Cy
TD
TD
LDA
=0035
40.655
10,387
*0.622=0u27
10.347=0004
70.788-4553
10.342=000
70,691
KM
10.251=0c6
10.204
10,294=0013
10.317aam1
10:213=0000
10,219-n01
10.244:am3
10.30220006
WLDA
+0,378=0007
10.375+0mm
f0.369=0004
30,273=0mm
10.311=0007
70.053-0.00
10,320+ama
=0.069=0.00
ETM
10.375
40.704amm
10,36920001
40.573
10.34620005
10.557_n014
10.341
10.37
NSTM
L0.395
40.427
10077
20.473,umms
F0.175 10.334
nome
40.3401am
F0.255
Wale
00.383=we5
10.949:0000
60.352=0306
00.742=0.25
#0.368
=0.931=0021
40.293=a015
60.638:m0004
ECRTM
0.431 1081
0.964
0.405 0.006
0.904 junce
0.395
0.979
0
0.887 (uevv
Yahoo Answer
AG News
Model
K=50
K=100
K=50
K=100
Cv
TD
Cv
TD
Cy
TD
Cy
TD
LDA
$0.359
10.843:10m
$0.359
-0.602
10.364
-0.864_uu22
10m349
=0.696_u.o
KM
=0027
40.242++00s
#0,297=0006
70,345:0000
=0.241=0mm5
70,264-a006
40,289+a007
50.395=0.066
WLDA
10.333_0007
m0
10,33820006
10
10.38420004
10,410_n018
10.378:30
10,323
ETM
F0.354_mees _ucon
40.71910mm2
+0.624 10,0
10.364.com
+0.819 no17
40.371
F0.773_uorv
NSTM
00339D=ace
10.658+om11
0.387=t001
10.659.um0s
10.411=omm
00.8730000
0,421=#0m
10.832:0..00
WeTe
L0.367
40.87811000
t00r
10.544.oomv
40.3831ca0
+0.945_.mo.
40.3631amm
$0.827
ECRTM
0.405=0004
0.985:a003
0.389
0,903
0.466:24412
0.961
0,416
0.981
Table 3: Document clustering of Purity and NMI under 50 and 100 topics (K and K=100). The best scores are in bold. t means the gain of ECRTM is statistically significant at 0.05 level.
20NG
IMDB
K=50
K- 100
K-50
K=100
Purity
NMI
Purity
NMI
Parity
NMI
Purity
NMI
LDA
30.367:00044
f0,364:00s
30.364-3000
F0,346=0006
F0.614=as0
=0.041
F0.600=ncov
#0.037=eu0)
WLDA
10.233:a00s
+0.157
10,292=0.0m
10 =0,00
10.589
10.011
10.602=ncos
10.013+000
EIM
105347 10.319
-0.394-000
+0.339=0000
0066000004
=0.038+0.02
10,6488ncco
40.037=0002
NSTM
10.354.0ms
10.3562n015
203
10 =03363
10.658
1o 10.040amm
10.659.nom
10.03940m
WeTe
10.268=0012
10 0000
=0.338
10. 300000
10.031 10,587=0014
70.589
10.025#c.01
ECRTM
0.560
0.524
0.555 wums
0.494
0.690_am.
0.056:aum
0.694
0.049:amm
Yahoo Answer
AG News
Model
K=50
K=100
K=
K=100
Purity
NM1
Purity
NMI
Purity
NMI
Purity
NMI
LDA
10.288.0m0
20.297 amne
90.148.0000
+0.640__wsox
10.1931omr
4o.654ino12
10.194icu0s
WLDA
10.255 40.084
-0.303_ma00
10.127_.0a
#0.580_nam
10mm
10.653
18.18
EIM
20.405:amm
t0.192ssues
70.428
F0,200 =0000 =cooss
F0.679=u012
40.224:oms
f0.674:ncos
40.204::0.0
NSTM
10.395 40,241 Looce
$0.405
40.242 Lo0ss
F0.719_ues
40.3241omzz
F0,764inew
40.359
Wele
70,389=000 t0.252e0ou
-0.444:a.006
f0.269=0012
00.641-0823
40.268=00m3
10,699
10.271
ECRTM
0.550_am
0.295
0.563
0.311
0.802
0.367
0.812
0.428iams
We consider the following state-of-the-ar models for comparison: (i) LDA Blei et al. 2003), one of the most widely-used probabilistic topic models; (ii) KM (Sia et al. 2020). directly clustering word embeddings to produce topics. Note that we cannot use it for document clustering since it does not infer the doc-topic distributions. (iii) WLDA (Nan et al. 2019). a WAE- based topic model; (iv) ETM (Dieng et al. 2020), neural topic model which models the topic-word distribution matrix with word and topic embeddings; (v) NSTM (Zhao et al., 2021b). using the
Baseline Models
optimal transport distance to measure reconstruction error. (vi) WeTe (Wang al. 2022 following NSTM and using conditional transport distance as reconstruction error.
4.2
MAIN RESULT OF TOPIC AND DOC-TOPIC DISTRIBUTION QUALITY
Table 2 reports the topic quality results concerning Cv and TD, and Table 3 summarizes doc-topic distribution quality results concerning Purity and NMI of document clustering. We have the follow- ing observations: (i) ECRTM effectively addresses the topic collapsing issue and outperforms baselines in topic quality. In Table 2 the much lower TD scores imply baselines generate repet- itive topics and thus suffer from topic collapsing. As aforementioned, these repetitive topics are less useful for downstream tasks and damage the interpretability of topic models. In contrast, we
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation study. While DKM, DKM + Entropy,
w/o ECR all have low TD, our ECRTM achieves much higher TD with better Purity and NMI. The low TD scores
ENSHADA
5.750
Ioe 24
0.000
mean most topics are repetitive and less useful, making
high Cv scores meaningless. $ means the gain of ECRTM is statistically significant at 0.05 level.
REORSTM
2.000
D
ROOFCRTM
20NG
Yahoo Answer
Model
Purity
NMI
Cv
TD
Parity
NMI
Cv
TD
0:000
Anowar News
10.510
10.171
10.577
10.507
10,252
0.403
10.631
Figure 4: Text classification results.
DKM + Eniray
F0.222
f0.148
0.469
10,503
F0.252
10.092
0.433
10.592
10.504
40:14
10.518
+0.498
10.262
0.435
10.608
The improvements of ECRTM are all
ECRTM
0.560
0.524
0.431
0.964
0.550
0.295
0.405
0.985
statistically significant at 0.01 level.
see our ECRTM achieves significantly higher TD scores across all datasets and mostly the best Cy scores meanwhile. We emphasize although the Cv of ECRTM is slightly higher than NSTM (0.389 v.s. 0.387) on Yahoo Answer, ECRTM completely outperforms on TD (0.903 v.s. 0.659). These results demonstrate that ECRTM produces more coherent and diverse topics than state-of-the-art baselines. These improvements are because our ECRTM makes topic embeddings away from each other and cover different semantics of word embeddings in the space instead of collapsing together as some baselines. (ii) ECRTM surpasses baselines in inferring high-quality doc-topic distribu- tions. Table 3 shows our ECRTM consistently outperforms the baseline models by large margin in terms of Purity and NMI. For example, ECRTM reaches 0,560 and 0.524 for Purity and NMI on 20NG, while the runner-up only has 0.354 and 0.356, These manifest that ECRTM not only achieves higher-quality topics but also better doc-topic distributions as document representations. See Appendices A.3 and A.4 for topic examples and more visualizations.
4.3 ABLATION STUDY
We conduct ablation studies and show the necessity of our proposed Embedding Clustering Reg- ularization (ECR). Specifically. we remove the ECR from our ECRTM. denoted as w/o ECR. We also compare with the state-of-the-at deep clustering method, DKM (Fard et al.) 2020 and DKM with minimizing entropy (DKM + Entropy, see Sec. (.3). Table 4 shows DKM, DKM + Entropy, and w/o ECR all suffer from topic collapsing as indicated by their much lower TD scores. Al- though they have high Cv. their terrible TD scores mean most topics are repetitive and less useful for downstream tasks, making the high Cv scores meaningless (see examples in Appendix AA.3 for illustrations). Conversely, document clustering performance with much higher Purity and NMI. This is because our ECR, as an effective regularization, can avoid the collapsing of topic embeddings while DKM, DKM + Entropy, and w/o ECR cannot. These results demonstrate our ECR effective and necessary to address the topic collapsing issue and achieve robust topic modeling performance.
ECRTM improves TD scores by large margin and achieves the best
4.4 TEXT CLASSIFICATION
To evaluate extrinsically, we further conduct text classification experiments as downstream tasks. Specifically, we use the doc-topic distributions learned by topic models as document features and train SVMs to predict the class of each document. As reported in Figure 4 ECRTM significantly outperforms baseline models on all datasets. These results demonstrate that our ECRTM can be better utilized in the classification downstream task.
5 CONCLUSION
In this paper, we propose the novel Embedding Clustering Regularization Topic Model (ECRTM) to address the topic collapsing issue. ECRTM learns topics under the new Embedding Clustering Regularization that forces each topic embedding to be the center of a separately aggregated word embedding cluster. Extensive experiments demonstrate that ECRTM successfully alleviates topic collapsing and consistently achieves state-of-the-an performance in terms of producing high- high-quality topics and topic distributions of documents. We hope our work can contribute to building more reliable topic modeling applications.

---

Under review as a conference paper at ICLR 2023
REFERENCES
David M Blci and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pp. 113-120, 2006.
David Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993-1022 2003.
David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: review for statisti- 2017.
cians. Journal of the American statistical Association,
Jordan L Boyd-Graber, Yuening Hu, David Mimno, et al. Applications of topic models, volume 11. now Publishers Incorporated, 2017.
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the dirichlet varia- tional autoencoder topic model. Mach. Learn. Res., 20(131):1-27, 2019.
Guillermo Canas and Lorenzo Rosasco. Learning probability measures with respect to optimal transport metrics. Advances in Neural Information Processing Systems, 25, 2012.
Dallas Card, Chenhao Tan, and Noah A Smith. Neural Models for Documents with Metadata. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pp. 2031-2040, 2018.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan L Boyd-Graber, and David M Blei. Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems, pp. 288-296, 2009.
Marco Cuturi. Sinkhorn distances: Lightspeed computation optimal transport. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL https: httpss///cccceeeeeeeeeeoo nwwrieuriss.. ccc/papper/213//11ee -ddf
Adji B Dieng, Francisco JR Ruiz, and David M Blei. The dynamic embedded topic model. arXiv preprint urXiv:1907.05544 2019.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Trans- actions of the Association for Computational Linguistics, 8:439 453, 2020.
Maziar Moradi Fard, Thibaut Thonet, and Eric Gaussier. Deep k-means: Jointly clustering with k-means and learning representations. Pattern Recognition Letters, 138:185-192. 2020.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn di- vergences, In International Conference on Artificial Intelligence and Statistics, pp. 1608-1617. PMLR, 2018.
Aude Genevay, Gabriel Dulac-Arnold, and Jean-Philippe Vert. Differentiable deep clustering with cluster size constraints. CoRR, abs/1910.09036 2019. URL
1910 09036
Philipp Hennig, David Stern, Ralf Herbrich, and Thore Graepel. Kernel topic models. In Artificial Intelligence and Statistics, pp. 511-519, 2012.
Thomas Hofmann. Probabilistic latent semantic analysis. In Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence, pp. 289-296. Morgan Kaufmann Publishers Inc., 1999.
Alexander Hoyle, Pranav Goel, Andrew Hian-Cheome Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. Is automated topic model evaluation broken? the incoherence of coherence. Ad- vances in Neural Information Processing Systems, 34:2018-2065 2021.
Chih-Chung Hsu and Chia- Wen Lin. Cnn-based joint clustering and representation learning with feature drift compensation for large-scale image data. IEEE Transactions on Multimedia, 20(2): 421-429, 2017.
10

---

Under review as a conference paper at ICLR 2023
Peihao Huang, Yan Huang, Wei Wang. and Liang Wang. Deep embedding network for clustering. In 2014 22nd International conference on pattern recognition, pp. 1532-1537. IEEE, 2014.
Viet Huynh, He Zhao, and Dinh Phung. Otlda: A geometry-aware optimal transport approach for topic modeling. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 18573-18582. Curran As- sociates, Inc., 2020. URL http:htttsssptceeeeeteeeee 73501444222997744444443446666 wwwpaaoooooopp.p...eeee
neurips.. www.rrreiip....ppprprrreeeeeeeee cctpppaaaae////////1eee////
Hannah Kim, Jaegul Jinou Kim, Chandan Reddy, and Haesun Park. Simultaneous discovery of common and discriminative topics via joint nonnegative matrix factorization. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 567-576, 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. In The International Con- ference on Learning Representations (ICLR), 2014.
Ken Lang. Newsweeder: Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 331-339, 1995.
Zongyang Ma, Aixin Sun, Quan Yuan, and Gao Cong. Topic-driven reader comments summariza- tion. In Proceedings of the 21 ACM international conference on Information and knowledge management, pp. 265-274, ACM, 2012. ISBN 145031 1563..
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011.
Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction to Informa- tion Retrieval. Cambridge University Press, New York, NY, USA, 2008. ISBN 0521865719, 9780521865715.
Julian McAuley and Jure Leskovec. Hidden factors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pp. 165-172, 2013.
Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang, Chao Zhang, Yu Zhang, and Jiawei Han. Discriminative topic mining via category-name guided text embedding. In Proceedings of The Web Conference 2020, pp. 2121-2132, 2020.
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In Interna- tional conference on machine learning, pp. 1727-1736, 2016
Yishu Miao, Edward Grefenstette, and Phil Blunsom. Discovering discrete latent topics with neural variational inference. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2410-2419. JMLR. org, 2017.
David Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. Polylingual topic models. In Proceedings of the 2009 Conference on Empirical Methods in Natu- ral Language Processing, pp. 880-889, Singapore, August 2009. Association for Computational org/ DD9- 10921092
Linguisties. WRLLittps///ccatthoooom
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with Wasserstein au- toencoders. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 6345-6381. Florence, Italy, July 2019. Association for Computational Linguis- tics.
David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics, pp. 100-108. Association for Computational Linguistics, 2010. ISBN 1932432655.

---

Under review as a conference paper at ICLR 2023
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@) in Machine Learning. (1(-6))355-607, 2019.
Steven T' Piantadosi. 7ipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review, 21 2014.
William J Reed. The pareto, zipf and other power laws. Economics lettes, 74(1):15-19, 2001.
Danilo Jimenez Rezende. Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings ofthe 31th International Con- ference on Machine Learning, 2014.
Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference Web search and data mining, pp. 499.408. ACM, 2015.
Tim Salimans, Han Zhang, Alec Radford, and Dimitris Metaxas. Improving gans using optimal transport. arXiv preprint arXiv: 2018.
Tian Shi, Kyeongpil Kang, Jaegul Choo, and Chandan K Reddy. Short-text topic modeling via non- negative matrix factorization enriched with local word-context correlations. In Proceedings of the 2018 World Wide Web Conference, pp. 1105-1114 International World Wide Web Conferences Steering Committee, 2018
Suzanna Sia, Ayush Dalmia, and Sabrina J. Mielke. Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too! In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1728-1736. 1736, Online, Novem- 135.
ber 2020. Association for Computational Linguistics. doi: URL stttps://aclantthoogg.oo www.athhwoooo..o./222oooooo emnlp maiin... 135
Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matri- ces. The annals of mathematical statistics, 35(2):776-879 1964.
Chunfeng Song. Feng Liu, Yongzhen Huang, Liang Wang, and Tieniu Tan. Auto-encoder based data clustering. Proceedings, Part I, of the 18th Iberoamerican Congress on Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications- 8258, pp. 117-124, 2013.
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL.https:/ httphttpss///ppereeiieee net fott/form?? id BybddkyybVKll9
Mark Steyvers and Tom Griffiths. Probabilistic topie models. Handbook of latent semantic analysis, 427(7):424 440, 2007.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 2(N9v):5579-2600 2008
Chong Wang and David M Blei. Collaborative topic modeling for recommending scientific articles. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 448-456 456. ACM, 2011. ISBN 1450308139.
Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zhang, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic em- beddings. International Conference on Learning Representations, 2022. URL hetps: 7/openrevi ew. www.:eet..om.mme. MM
12

---

Under review as a conference paper at ICLR 2023
Xuerui Wang, Andrew McCallum, and Xing Wei. Topical n-grams: Phrase and topic discovery, with an application to information retrieval. In Seventh IEEE international conference on data mining (ICDM 2007), pp. 697-702. IEEE, 2007.
Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. Learning Multilingual Topics with Neural Variational Inference. In International Conference on Natural Language Processing and Chinese Computing. 2020a.
Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. Short text topic modeling with topic distri- bution quantization and negative sampling decoder. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 1772-1782. Online, Novem- ber 2020b.
Xiaobao Wu, Chunping Li, and Yishu Miao. Discovering topics in long-tailed corpora with causal intervention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 175-185, Online, August 2021. Association for Computational Linguisties. doi: 10.15v///v/ 2021 .iidiings-acl.15... ac1.11. URL https: /tttpps//aalaattoooogg.... wwt. wttp://ooooooooo/222..2.... findings-iiiiininiaenaaiaee
Junyuan Xie, Ross Girshick, and Ali Farhadi. Unsupervised deep embedding for clustering analysis. In International conference on machine learning, pp. 478-487. PMLR, 2016
Xiaohui Yan, Jiafeng Guo, Shenghua Liu, Xueqi Cheng, and Yanfeng Wang. Learning topics in short texts by non-negative matrix factorization term correlation matrix. In proceedings of the 2013 SIAM International Conference on Data Mining, pp. 749-757. SIAM, 2013.
Bo Yang, Xiao Fu, Nicholas D Sidiropoulos, and Mingyi Hong. Towards k-means k-means-friendly spaces: Simultaneous deep learning and clustering. international conference on machine learning, pp. 3861-3870 PMLR, 2017.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Zihan Zhang, Meng Fang, Ling Chen, and Mohammad -eaza Namazi- Radd Is neural topic modelling better than clustering? empirical study clustering with contextual embeddings for topies. arXiv preprint arXiv:2204.00877 2022.
He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. Topic modelling meets deep neural networks: A survey. arXiv preprint arXiv:2103.00499 2021a.
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray L. Buntine. Neural topic model via optimal transport. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL URLLtttps///ppeeeviee...ee Corum?id=0os98le
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
Table 5: Top 10 related words of discovered topics from IMDB. Repetitive words are underlined.
like better good especially end look much done way just
ETM
like just one way made much times really even feel one like around sort looking kind good main look just
just show even come time one good rcally going know just even really something come going like actually things get just one something come way really like always good
NSTM
hitchcocks couldve wouldve shouldve familys wifes anyones everyones itil hollywood hitcheocks anyones wouldve everyones wifes couldve familys itll shouldve hollywood
We'ee
hitchcocks couldve wouldve shouldve wifes familys anyones everyones itll hollywood
christmas disney musical songs bill timeless prince art rock holiday christmas santa childrens holiday betty age ann adult children toy fantasy christmas magic effects magical santa special holiday childrens child
DKM
funny day physical semi ever way old due seen zonc funny ever day old seen way physical due semi relationship funny ever day semi seen physical way old due psychological
DKM + Entropy
jackie martial chan kung arts kong hong stunts bruce fight
ECRTM
nominated nancy academy award awards oscar oscars jake nomination dracula vampires vampire freddy zombies zombie nightmare serial halloween killer slasher
(a) ETM
(b) NSTM
(c) WeTe
(d) ECRTM
visualization of word embeddings and topic embeddings (A) under 100 topics. Topic embeddings commonly collapse together in state-of of-the-art models (ETM Dieng al., 2020), NSTM (Zhao et al. 2021b), and WeTe (Wang et al. 2022). ). In contrast, ECRTM can avoid the collapsing by forcing each topic embedding to be the center of separately aggregated word embedding cluster.
Figure 5: t-SNE (van der Maaten & Hinton 2008
A.1 DATASET
We follow the dataset preprocessing steps of Card al. 2018): (1) documents tokenized and converted to lower case; (2) remove punctuation; (3) remove tokens that include numbers; (4) re- move tokens less than characters: (5) remove stop words.
A.2 IMPLEMENTATION DETAIL.
For pre-trained word embeddings, employ 200-dimensional GloVe (Pennington et al. 2014). For the Sinkhom''s algorithm of ECRTM, we set the maximum number of iterations as 1,000, the stop tolerance 0.005 and 0.05 following Cuturi 2013). For ECRTM, the prior distribution is specified
14

---

Under review as a conference paper at ICLR 2023
vampires
disney lion curtoons animation cartoon animated anime animals toy fox
vampire
freddy
zombies
zombie
nightmare
serial
halloween
killer
slasher
*
nominated
jackie
nancy
academy
martial
award
chan
awards
kung
arts
oscar oscars
kong
jake
hong stunts bruce fight
nomination
dracula
album bands concert broadway musicians duncers rup lyries sung musical
Figure 6: Visualization of discovered topics in the semantic space.
with Laplace approximation (Hennig al., 2012) to approximate a symmetric Dirichlet prior as Mo. = 0 and Eo, && = (K- ()7(K)) with hyperparameter The in the prior distribution is 1.0 following Card et al. (0188). Besides, 7 is 0.2, and λEcR is 250, 100, 60 and 5 for 20NG, IMDB, Yahoo Answer and AG News respectively. Our encoder network is the same Srivastava & Sutton (2017): a MLP that has two linear layers with softplus activation function, concatenated with two single layers each for the mean and covariance matrix. We use Adam optimizer (Kingma & Ba 2014) to optimize the model parameters.
A.3 EXAMPLES OF DISCOVERED TOPICS
Table shows randomly selected examples of discovered topics by different models from IMDB. We observe that ETM and NSTM both have highly uninformative and similar topics including common words like "just", "like". "something". WeTe produces some exactly the same topics with words "couldve", "wouldve", and "shouldve". DKM and DKM Entropy also generate repetitive topics with words "chirstmas". "holiday", and "funny". We see the topic collapsing issue commonly exists in these methods. These collapsed topics are uninformative and redundant, which are less useful for downstream applications and damage the interpretability of topic models. In contrast, the topics discovered by ECRTM are more distinct instead of repeating each other. Besides, they are relatively more coherent, such the first topic with relevant words like "jackie", "chan", and "stunts". These examples show that our ECRTM generates higher-qualit topics.
A.4 VISUALIZATION OF EMBEDDINGS
We visualize the learned topic and word embeddings with t-SNE (van der Maaten & Hinton] 2008 under 100 topics (Figure ii under 50 topics). Figure shows while the topic embeddings mostly collapse together in the state-of-the-art baselines. our ECRTM avoids the collapsing of topic embed- dings by forcing each topic embedding to the center separately aggregated word embedding cluster. This illustrates that ECR works effectively under larger number of topies.
15

---

Under review as a conference paper at ICLR 2023
We further annotate the semantic space with the top related words of discovered topies by ECRTM as shown in Figure 6 We see that each word embedding cluster represents a diverse and coherent topic. This verifies that our ECRTM effectively clusters the embeddings of coherent words by jointly optimizing the neural topic modeling objective with the embedding clustering regularization objective.
A.5 RUNTIME ANALYSIS
We train our model with NVIDIA GPU and use PyTorch for implementations. It takes about less than 0.5 GPU hours to train our model on datasets.
16


Reviews:
[{"review_id": "Stg5sPu2ze", "reviewer": "Reviewer_1otX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an embedding clustering regularization technique to mitigate the posterior collapse\nproblem in neural topic models. The regularization technique forces the topic embeddings to be the\ncenters of separately aggregated clusters of word embeddings. As a result, distinct clusters of topic-\nword embeddings are formed wherein each topic comprises distinct words and every topic is distant\nfrom one another in the semantic space. This paper evaluates their technique for different aspects\nof topic modeling such as topic quality (using coherence) and diversity and in addition also measures\ntopic-doc distribution and document classification accuracy. It outperforms several other methods that\nit compares with on almost every evaluation metric that they use.", "summary_of_the_review": "The work in itself is novel and overcomes topic collapsing limitations. But, the baseline selection is incomplete.\nMethodology is clearly explained. There are nice illustrations of limitations and results which suggest that\nthe proposed model is effective. However, without comparison to SotA baselines, it is difficult to evaluate the results.\n\nSolving a notorious problem of posterior collapse in topic models is the main contribution of this paper.\nAlthough, it is based on the prior clustering techniques such as Deep Kmeans, some small tweaks in\ntheir methodology seems to significantly improve the topic collapsing issue and this paper successfully\ndemonstrates it via empirical setup. The qualitative results shown also seem to be promising. The\napproach outperforms many recent topic models on several evaluation criteria such as topic quality\n(coherence), diversity and downstream document classification task but there is still some scope for\nrigorous experiments to demonstrate the robustness of the idea against different lengths of text, different\ntopic sizes and other downstream tasks such as document retrieval and also if it can be generalized for\nother NLP tasks.\n\n\n\n\nOpen questions:\n- What about the robustness of the model against the number of topics? (The results are shown only\nfor K=50,100)\n- Co-occurence information between words is comparatively low for short text such as tweets or news\nheadlines. How does the method perform on short (short text dataset M10) text sequences?\n- Any specific reason for not comparing the results with those from Zhang et al paper (Is Neural Topic\nModelling Better than Clustering? An Empirical Study on Clustering with Contextual Embeddings\nfor Topics)?", "strength_and_weaknesses": "# Strengths\n\n- Clear illustration of limitations of existing methods. It is also clearly illustrated that the proposed model\novercomes those limitations.\n- A sound theoretical solution to the limitations is presented.\n- Overall good results when compared to the selected baselines\n\n#Weaknesses\n- The baseline selection is poor. The authors claim that it is difficult to incorporate Dirichlet distribution\nin VAEs. This is not entirely true. There has been work by (Burkhardt & Kramer, 2019) on using\nDirichlet as prior for VAEs in topic modeling. This work was again reimplemented by (Hoyle et al.,\n2021). This has been overlooked by the authors in the paper. The compared baselines for illustrations of\nlimitations also do not use Dirichlet prior, which was initially presented to overcome the said limitations.\n- The evaluation includes Cv score, which for some experiments is very low. It is recommended to include\nNPMI scores as well.\n- If I understand the sparse soft-assignment clustering correctly, this technique will assign every word\nto one and only one topic cluster but there are several words that belong to multiple contexts for\nexample, season, which can be related to weather and sports both. So, the proposed model cannot\nassign words that occur in different contexts to different topics.\n- The most obvious limitation is that this method produces fixed size clusters of all topics. On page\n5, section 3.4, they assume that every topic includes same amount of semantic information and this\nassumption is supported by practically forcing every topic to have same number of words. This means\nhaving same amount of semantic information is equal to having same number of words. I don’t quite\nagree to this assumption because semantic information of a topic cannot be quantified by the number\nof words present in it.\n- Lastly, posterior collapse is a significant and long standing problem in several other text (and\nimage) applications such as text generation, summarization etc. The method performs considerably\nwell for topic modeling but unfortunately does not extend or discuss how it can be generalized to the\nother applications.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "OrrReSE6oZu", "reviewer": "Reviewer_QM6u", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author found that the previous methods often suffer topic collapsing issues, which leads to repetitive topics, insufficient topic discovery, and also damages the model interpretability. Motivated by such findings, the paper proposed to add Embedding Clustering Regularization (ECR) to topic modeling models. The ECR helps to keep the topic embedding at the center of word embedding clusters and then alleviates the topic collapsing issue. The proposed method is evaluated on four benchmark datasets and the experimental results show encouraging performance.", "summary_of_the_review": "The paper proposed to add Embedding Clustering Regularization (ECR) to topic modeling models. The ECR helps to keep the topic embedding at the center of word embedding clusters and then alleviates the topic collapsing issue. The proposed method is evaluated on four benchmark datasets and the experimental results show encouraging performance.", "strength_and_weaknesses": "Strength:\nThe proposed method regularizes the topic model such that the topic embeddings are located at the center of word embedding clusters. This is demonstrated in the t-sne Figure. Since the clusters are separated, we can observe improvements in terms of TD in both Table 2 and Table 3.\n\nWeakness:\n1. As the ECR is proposed as an additional module to further regularize the topic models, it will be great to also apply the ECR to other baseline approaches. \n2. It will be nice to have some proof or analysis to explain the necessity of forcing the topic embeddings to be the center of the word embedding clusters. \n3. More experiments can be conducted to enhance the current findings. Such as running the experiments with different K, ranging from 10 to 100 with a fixed step size.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "D8I_-i8e_F9", "reviewer": "Reviewer_YKAw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper tackles the well known and widely observed issue of topic collapsing in neural topic models (NTMs). \n- Specifically, it proposes a novel embedding clustering regularization (ECR) objective, in addition to reconstruction error objective, during training variational autoencoder (VAE) based NTM which separately aggregates the word embeddings in clusters and apply constraint on topic embeddings to be within those clusters.\n- A new topic model is further proposed which jointly optimizes using reconstruction error objective and the ECR objective to extract non-collapsing, diverse and coherent topics.\n- the paper takes inventive steps in addressing topic collapsing in neural topic modeling - however there are limitations, including multi-sense/ambiguous words in topics and comparisons to traditional baselines such as Orthogonal topic modeling.\n\n", "summary_of_the_review": "- Although novelty is limited in this paper, but the direction it explores to eliminate topic collapsing issue and generate diverse and coherent topics is interesting and important as diverse topics generate high quality document-topic representations which further boost accuracy in downstream supervised or unsupervised NLP tasks.\n- missing comparison to (related) baselines methods (please see weaknesses)\n- additionally, please see weaknesses section\n\n", "strength_and_weaknesses": "Strengths:\n- The motivation is well founded and the claims are sound.\n- Paper is clearly presented and easy to follow.\n- Mathematical formulation is explanatory and easy to understand.\n- Proposed ECR objective is robust and effective.\n- Experimental evaluation is extensive and the proposed topic model consistently outperforms multiple NTM baselines on topic quality and topic diversity metrics with significant margins over multiple datasets.\n\nWeaknesses:\n- there are limitations to the proposed approach, including multi-sense/ambiguous words in topics and comparisons to traditional baselines such as Orthogonal topic modeling.\n- Contributions are incremental and Novelty is limited.\n- missing baselines and experimental comparisons such as Orthogonal topic modeling and neural topic models such as DocNADE or iDocNADEe [2]\n- topic analysis (qualitative) is unclear and seems biased to initial (3) set of topics - however should be analyzed for all topics extracted (extend the appendix) \n\nQuestion:\n- how is this work compared to Orthogonal Topic Modeling? \n\nReferences:\n1. Probabilistic Text Modeling with Orthogonalized Topics. SIGIR 2014.\n2. Document Informed Neural Autoregressive Topic Models with Distributional Prior. AAAI 2019.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0X-DL8S6waG", "reviewer": "Reviewer_iecT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents (yet another) Neural Topic Model (NTM). The main idea is to add a regularization term to force the topic vectors to be away in the space by using a clustering mechanism, which lead to decrease the collapsing problem. The full model, named Embedding Clustering Regularization Topic Model (ECRTM) is based on previous works following an Optimal Transport scheme. It is favorably compared to previous models on a set of classic (and limited) datasets.", "summary_of_the_review": "This paper doesn't bring any big improvement in the topic modeling area. The regularization term may be somewhat useful in order to avoid topic collapsing, but the experiments fall short to really prove that the new model will be effective in a large range of dataset profiles.", "strength_and_weaknesses": "Strength:\n- Adding a regularization term, but also other mechanisms (such as a priori on topic size), improves the results on some datasets\n\nWeaknesses:\n- Several mechanisms need to be combined to design the final model (clustering with soft assignment, prior on topic size, reintroduction of the \"less sparse beta\"). To this extent, stacking various techniques to get the final system introduces some doubt in what really explains the success of the approach. The ablation study of 4.3 doesn't really answer to this concern.\n- As in previous approaches (which usually seem to overlook more than 20 years of research in topic model), the experimental framework is questionable. First, the perplexity cannot be discarded as easily: it is clearly not sufficient to assess the quality of topics, but it can be added to check whether the topics can explain/cover the full dataset. It's related to argument (2) p.2 and the way the authors address it is not convincing to me, even though it has already been used in previous ICRL papers. We shouldn't confuse topic modeling and clustering, many papers have been written on that subject. Evaluating the capacity of a topic model to cluster documents into a limited number of quite well-separated clusters (which is the case in most 20 newsgroups classes) is clearly not sufficient to evaluate the doc-topic distribution quality. Another remark: the C_V measure of Röder is not that much used in the literature and a couple of concerns (in term of reproducibility) have been raised online. I suggest to use UCI or UMass instead.\n- Recent topic models should be able to deal with more diverse datasets than news and review.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject