Topic Modeling as Multi-Objective Contrastive Optimization
Authors: Thong Thanh Nguyen
Xiaobao Wu
Xinshuai Dong
Cong-Duy T Nguyen
See-Kiong Ng
Anh Tuan Luu
Venue: ICLR 2024 poster (2024)
Keywords: neural topic model; contrastive learning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Recent representation learning approaches enhance neural topic models by optimizing the weighted linear combination of the evidence lower bound (ELBO) of the log-likelihood and the contrastive learning objective that contrasts pairs of input documents. However, document-level contrastive learning might capture low-level mutual information, such as word ratio, which disturbs topic modeling. Moreover, there is a potential conflict between the ELBO loss that memorizes input details for better reconstruction quality, and the contrastive loss which attempts to learn topic representations that generalize among input documents. To address these issues, we first introduce a novel contrastive learning method oriented towards sets of topic vectors to capture useful semantics that are shared among a set of input documents. Secondly, we explicitly cast contrastive topic modeling as a gradient-based multi-objective optimization problem, with the goal of achieving a Pareto stationary solution that balances the trade-off between the ELBO and the contrastive objective. Extensive experiments demonstrate that our framework consistently produces higher-performing neural topic models in terms of topic coherence, topic diversity, and downstream performance.

Published as a conference paper at ICLR 2024
TOPIC MODELING AS MULTI-OBIECTIVE CONTRASTIVE OPTIMIZATION
Thong Nguyen', Xiaobao Wu², Xinshuai Dong², Cong-Duy Nguyen?, See-Kiong Ng', Luu Anh Tuan
'Institute of Data Science (IDS), National University of Singapore (NUS), Singapore, Nanyang Technological University (NTU), Singapore, *Carnegie Mellon University (CMU), USA,
ABSTRACT
Recent representation learning approaches enhance neural topic models by optimiz- ing the weighted linear combination of the evidence lower bound (ELBO) of the log-likelihood and the contrastive learning objective that contrasts pairs of input documents. However, document-level contrastive learning might capture low-level mutual information, such word ratio, which disturbs topic modeling, Moreover, there a potential conflict between the ELBO loss that memorizes input details for better reconstruction quality, and the contrastive loss which attempts to learn topic representations that generalize among input documents. To address these issues, we first introduce novel contrastive learning method oriented towards sets of topic vectors to capture useful semantics that are shared among a set of input documents. Secondly, explicitly cast contrastive topic modeling as gradient-based multi-objective optimization problem, with the goal of achieving Pareto stationary solution that balances the trade-off between the ELBO and the contrastive objective. Extensive experiments demonstrate that our framework consistently produces ligher-performing neural topic models in terms of topic coherence, topic diversity, and downstream performance.
INTRODUCTION
As one of the most prevalent methods for document analysis, topic modeling has been utilized to discover topics of document corpora, with multiple applications such as sentiment analysis (Brody &
Elhadad 2010 Tay et al.) 2017 Wu & Li 2019 Zhao et al., 2020 Nguyen al. 2023), language
generation (Jelodar et al. |019]] Nguyen et al. [2021] Tuan
uaan et al. 2020). and recommendation system (Cao al., 2017 Zhu al.]| 21/7// et al. 2018]Jelodar Jelodar et al. [201)). In recent years, Variational Autoencoder (VAE) Kingmma & Welling 2013) has achieved great success in many fields, and its encoder-decoder architecture has been inherited for topic modeling with neural networks, dubbed as
N019//guyen
Neural Topic Model (NTM) (Miao et al. 2016: Wu et al., 2020b; 2021 2024a 2023c). Exploiting the standard Gaussian as the prior distribution, neural topic models have not only produced expressive
global semanties but also achieved high degree of flexibility and scalability to large-scale document
collections (Wang et al. 2021b 2020 Gupta et al.) 20220).
Based the VAE architecture, neural topic model is trained by optimizing the evidence lower bound (ELBO), which consists of an input reconstruction loss and the KL-divergence between the prior and the topic distribution. Whereas the first component aims optimize the reconstruction quality, the second one pressures the reconstructor by regularizing the topic representations with distributional constraints. Thus, the joint optimization induces a trade-off between the two components (Lin et al.) 2019a, Asperti & Trentin 2020). One solution to reach balance of the trade-off is to search for the optimal weight to combine the two losses, which requires exhaustive and computationally expensive manual hyperparameter tuning (Rybkin et al., 2021). To circumvent the problem, previous works jointly train the NTM with the contrastive learning objective (Le & Akoglu 2019 Nguyen Luu 2021 Li al. 2022),, since contrastive estimation has proven to be an effective regularizer to promote generalizability in input representations (Tang et al. 202T Kim et al. 2021).
"Corresponding Author

---

Published as a conference paper at ICLR 2024
Figure 1: Illustration of the intense low-level feature influence on produced topics. We record the cosine similarity of the input with the document instance / and document instance 2's topic representations generated by NTM+CL (Nguyen & Luu 2021] and our neural topic model. Although the input and instance 2 both describe the electric topic, the similarity for the (impul, instance 7) pair higher, because the impul and instance / have the same number of non-zero entries, i.e. 5, and the same ratio of maximum to minimum frequency. i.e. 10 to 4.
Inpul Number of non-Pamn entries: 8 Max-Min Frequency rati::
Instanoe Number of non-twro entries: 6 Max-Min Frequency ratio:
Instance 2
Number non neroo--wrooo entress Max Frequency
10
10
NTTM-GL:C
10
NTTA+ CL:00422
Model: 0,0460
Our 0.5100
2
2-
2
0
1.50
one ground diCuP coble eloctricl
study WOOK
one
maRD
electfall
Table 1: Illustration of the effects of peculiar words on topic representations. We record the cosine similarity of the input and the document instances' topic representations generated by NTM+CL (Nguyen & Luu 2021) and our neural topic model. As shown with underlined numbers, inserting unusual term, such as zeppelin or scardino, unexpectedly raises the similarity of the input with the document instance, even surpassing the similarity of the semantically close pair.
Cnsine Similarity
Input
Document Instance
NTM+CL
Our Model
job career ask development
0.0093
0.0026
shuttle lands planet
star astronaut planet light moon
0,8895
0,9178
shuttle lands planet zeppelin/scundion
job career development zeppelin/s-
0.9941
0.006470.0080
cardino
star astronaut planet light moon
0.15840.2547
0.8268/0.7188
However, by discriminating different document instances, contrastive learning leads to two significant challenges for neural topic models. First of all, instance discrimination could intensely concentrate on low-level mutual features (Tschannen et al., 2019), e.g. the number of words with non-zero word frequency, which are illustrated in Figure []
frequency and the ratio of to negatively affect topic representations. Second, as the contrastive objective bears a regularizing impact to improve generalization of hidden representations, generalizable but inefficient features to topic modeling could emerge, especially when the amount of mutual information exceeds the minimal sufficient statistics (Tian et al. 2020). For example, the discrimination task could go beyond mutual topic themes to encode common existence of peculiar words or phrases, as depicted in Table [ As a result, there is need to adjust the effect of contrastive learning to better control the influence of mutual information the NTM training.
To address the first issue of low-level mutual information, we propose a novel contrastive learning for sets of documents. We hypothesize that useful signals for neural topic models should be shared by multiple input documents (Hjelm et al.] 2018). For example, the model should be able to detect the sport theme in several documents in order encode the theme into the topic representations. Intuitively, such shared semantics specify sufficient information that forms truly useful features. To implement this idea, given a mini-batch of input documents, we divide the documents into different sets and adopt augmentation to construct two views of a set. Subsequently, we obtain topic representations of document set by aggregating topic vectors of their documents, and pass these representations to the contrastive learning objective. To maximize the training efficiency, we further shuffle the input documents mini-batch before diving them into different sets in order to increase the set quantity in a training iteration.
Regarding the second issue, an apparent solution is to predetermine linear weights for the ELBO and contrastive losses. Nonetheless, in general scenarios, if the tasks conflict with each other, the straightforward linear combination is an ineffective method (Liang et al. 2021: Mahapatra & Rajan 2020). Instead, propose to formulate the training of contrastive NTM oulti-ojjective optimization problem. The optimization takes into account the gradients of the contrastive and the ELBO loss to find the Pareto stationary solution, which optimally balances the trade-off between
2

---

Published as a conference paper at ICLR 2024
the two objectives (Sener & Koltun 2018). Our proposed approach regulates the topic encoder so that it polishes the topic representations of the NTM for better topic quality, as verified by the topic coherence and topic diversity metrics in comprchensive experiments.
The contributions of this paper are summarized as follows:
We propose novel setwise contrastive learning algorithm for neural topic model to resolve the problem of learning low-level mutual information of neural topic models. We reformulate our setwise contrastive topic modeling as a multi-task learning problem and propose to adapt multi-bjjcctvve optimization algorithm to find Pareto solution that moderates the effects of multiple objectives on the update neural topic model parameters. Extensive experiments on four popular topic modeling datasets demonstrate that our ap- proach enhance contrastive neural topic models in terms of topic coherence, topic diversity, and downstream performance.
2 RELATED WORKS
Neural topic models (NTMs). We consider neural topic models of (Miao et al.
Srivastava 2018 Wu al.
2016
Sutton 2017: Dieng et al., 2020; Burkhardt Kramer 2019 Card et al.
2020a Nguyen & Luu 2021 Wu et al. 2023a 2024b) as the closest line of related works to ours.
Recent research has further sought to enrich topic capacity of NTMs through pretrained language
models (Hoyle et al., 2220)) /Zhan et al.
2022
Bahrainian et al. 2021 |Banchi et al.
(021) and word
embeddings (Zhao et al. [2200] Xu et al. inject contextualized representations from language models pretrained on large-scale upstream
2022
Wang et al. al.
Their approaches
datasets into NTMs being fine-tuned on topic modeling downstream ones. Different from such
methods, contrastive neural topic models (Nguyen & Luu 2021 Wu et al. 2022) mainly involve downstream datasets to polish topic representations.
Contrastive representation learning. Contemporary approaches follow the sample-wis approach (Nguyen et al., 2022 2023). Several methods to elegantly generate samples comprise using the momentum encoder to estimate positive views (He al. 2020), modifying original salient/non- salient entries to create negative/positive samples (Nguyen & Luu 2021). etc. A surge of interest presents the cluster concept into CRL, which groups hidden features into clusters and performs discrimination among groups (Guo et al. 2022 Li et al. 2020 202T Caron et al.) [2200] Wang et al., 2021a). Different from these works, our method operates upon the level, directly divides raw documents into random sets, and aggregates features on-the-fly fly during training. (Le & Hua 2022) also allocates raw inputs into different sets. However, their work applies one pooling function to obtain set representation. Such single pooling approach might be ineffective since we would like to maximize the similarity of even unimportant entries of a positive pair, thus using min- min-pooling for positive set representations, and minimize the similarity of important entries of negative pair, thus using max-pont -poolin.. Moreover, while they target image classification tasks computer vision, we concentrate our framework upon neural topic modeling learn discriminative topics from document corpora.
BACKGROUND
3.1 NEURAL TOPIC MODELS
Neural topic model (NTM) inherits the framework of Variational Autoencoder (VAE) where latent variables are construed as topics. Supposing the linguistic corpus possesses V unique words, i.e. vocabulary size, each document is represented as a word count vector E RV and a latent distribution over T topics E R The NTM makes assumption that z is generated from a prior distribution p(z) and x is from the conditional distribution Ps [x (z) parameterized by a decoder şh. The goal of the model is discover the document-topic distribution given the input. The discovery implemented as the estimation of the posterior distribution p(z]x), which is approximated by the variational distribution qo (z)x), modelled by an encoder Inspired by VAEs, the NTM is trained to minimize the following objective based upon the Evidence Lower BOund (ELBO):
min CELBO Eg (x/x)) [log p. (x/z)] + KL (gx(z p(z)]
(1) The first term denotes the expected log- ligellooooodd or the reconstruction error, the second term the Kullback-1 Leibler divergence that regularizes go (z(xx) ) to be close to the prior distribution p(z). By
0.0
3

---

Published as a conference paper at ICLR 2024
enforcing the proximity of latent distribution to the prior, the regularization term puts constraint the latent bottleneck to bound the reconstruction capacity, thus inducing a trade-off between the two terms in the LLBO-based objective.
3.2 CONTRASTIVE REPRESENTATION LEARNING
Contrastive formulation maximizes the agreement by learning similar representations for different views of the same input (called positíve pairs), and ensures the disagreement via generating dissimilar representations of disparate imputs (called negative pairs). Technically, given B-size batch of input documents X = {x1. X2,... XB X, E R² firstly data augmentation t ~ T is applied to each sample x; to acquire the augmented view y;, then form the positive sample Yi), and pairs that involve augmentations of distinct instances (x:, Ny), ) * become negative samples. Subsequently, a prominent approach is to train a projection function f by minimizing the InfoNCE loss (Oord et al. 2018): where defines similarity mapping R. x R + R, estimated as follows:
f(x,y)
||I| Io (y) || /T.
(x)
where 9r denotes the neural network parameterized by ip, 7 the temperature to rescale the score from the feature similarity.
3.3 GRADIENT-BASED MULTI-OBIECTIVE OPTIMIZATION FOR MULTI-TASK LEARNING
Multi-Tast Learning (MTL) problem considers tuple of M tasks with a vector non-negative
losses, such that some parameters @shured are shared while others 01, 0² of.....0"" .M. are task-specific:
LM min Ligsharod 0z o......0" gM (GShardd 0²), L2( Car (Oshared 0 (M)))
where C; denotes the loss of the i-th task. In most cireumstances, no single parameter set can accomplish the minimal values for all loss functions. Instead, the more prevailing solution is to seek the Pareto stationary solution, which sustains optimal trade-offs among objectives (Sener Koltun 2018 Mahapatra & Rajan 2020). Theoretically, a Pareto stationary solution satisfies the Karush- Kuhn- Tucker (KKT) conditions as follows.
Theorem 1. (Sener & Koluun
2018 Let Shored denote Pareto point. Then, there exists non-negative scalars {Arm m= m E
(1. M} such that
M
M
am
and
am V
Crm (8
0¹) 0.
(4)
m==1
In consequence, the conditions lead the following optimization problem:
M am Viaazess Cy (gshured 8m
M
min [Orm momt
Σ
am = 1, am 2 0 Vm
(5)
1
Either the solution the above problem does not exist, i.e. the outcome meets the KKT conditions, or the solution supplies descent direction that decreases all per-task losses (Désidéri, 2012). Hence, the procedure would become gradient descent task-oriented parameters followed by solving Eq. (5) and adapting ΣM=1
Olm gaaeed L. (Oshared (nm) as the gradient update upon shared parameters.
METHODOLOGY
In this section, introduce the details and articulate the analysis of our proposed setwise con- trastive learning approach. Thereupon, we delincate the multi-objective optimization framework for contrastive neural topic models.
4

---

Published as a conference paper at ICLR 2024
4.1 SETWISE CONTRASTIVE LEARNING FOR NEURAL TOPIC MODEL
Algorithm
Setwise Contrasiive Neural Topic Model as Multi Objective Optimizttion.
Input:
Document batch X [x1 x2,.... ...., x ). encoder decoder gis:: Positive augmentation negative augmentation rale Set size K; contrustive weight A: Shufllnn number S;
LLeaming
do
X. =1 (x,),
(x.):
kx ho(x;) x, hx(x7
4: end for
Matrix M mores shuffied rows indices 1, 1,2,,.,
M index matrix of size S x B:;
to do
Group every K documents jK + 0.3K
(K into Difjerent shuffled row leads to different document
sets.
for
R do extraction for document setz. Refer and Appendix choices of pooling functions T and a
10:
respectively.
11:
". =
12:
2, =
(x+, 1mz0
13:
$, =
(2
14:
*h. + (z Fi x
end for
16: end for
17: contrastive learning objective
18: LUINCE
Σ log
expls
Mo (5,00)
19: # Mudtt-objective optimization Refer to Eg For formulation
20: a solver[V Cussees ()), V P]];
21: G. V. Lusonen VoLno:
22: Gg CELBO:
# Updatte topic encoder and topic decoder
24: 8
Go:
6 n
Sample augmentation. Firstly, given batch of B input documents X = {x1. X2. .,,,..,,,)))).. we conduct data augmentation to generate the corresponding 2B documents X'+ = {x] X². ..... ..., x3} and X' = . , where x," and x, are semantically close to and distant from x,, respectively. Previous methods (Yin & Schütze 2016; Nguyen & Luu 202I) utilize pre-trained word embeddings or auxiliary metric such as TF-IDF to generate such positive and negative samples. However, those approaches only modify certain words, which are shown to produce noisy and not true positive/negative documents (Vizcarra & Ochoa-Lunaa 2020). Therefore, with the advance of LLMs, before forming the BoW vector of each input document, we adopt the ChatGPT API to perform the augmentation with the prompt template: A sentence that is <related/unrelted>> this text: <input document>, and use the <related> option for X and the <unrelated> one for X As shown in Appendix our approach produces document augmentations that more natural and less noisy compared with previous methods.
Feature extraction. For a mini- batch, each set is constructed by grouping every K input samples. We proceed extract set representation:
((=(((()))))) spt = ((ha(x)))<==)) s = ((ha((( ) st= + (6)
where he denotes the d-parameterized topic encoder. o and + are pooling functions for positive and negative set, respectively. We experiment with different choices for &p and yot in Section 5.3 and Appendix respectively.
Index shuffling. To create sets of K documents, we can allocate every K document into set. However, this approach results in restricted quantity of document sets, i.e. | sets. Consequently, 1) negative pairs for each mini-batch. This method is also not
this yields K positive and (1 efficient in terms of document input usage, as each document only appears once in a certain set.
Instead, to augment the number of documents and consequently raise the number of positive and negative set pairs, we shuffle the list of indices from to B for times and record the shuffled indices in index matrix S of size S B. We then assemble every K elements corresponding
5

---

BS sets, thus maximizing the capacity of our Published as a conference paper at ICLR 2024
the index matrix into document set. Our proposed approach can increase the number of positive and negative pairs through raising the set number to setwise contrastive neural topic model.
4.2 MULTI-OBJECTIVE OPTIMIZATION FOR NEURAL TOPIC MODEL
Problem statement. For a given neural topic model, we seek to find preference vectors a that produces adept gradient update policy Σ, a, ft. For the contrastive neural topic model, because only the neural topic encoder receives the gradients of both contrastive and topic modeling objectives, we focus on modulating the encoder update while keeping the decoder learning intact. Hence, motivated by formulation our optimization problem becomes:
min
a CintNNCE (0) - (1o
a a220
which is a convex quadratic optimization with linear constraints. Erstwhile research additionally integrates prior information to compose solutions that are preference-specffc among tasks (Mahapatra & Rajan 2020 Lin et al. [2016b] (020). However, for neural topic modeling domain, such information is often unavailable, so we exclude it from our problem notation.
Optimization solution. Our problem formulation involves convex optimization of multi-dimensional quadratic function with linear constraints. Formally, we derive the analytical solution & as:
(VoLintaNCE (0) (oLELBO(O,0)) VoLINIONCE())
á solver (intovcc)))))))) V Vo
VoLintacc VOLELBO
where [.]+ denotes the ReLU operation. After accomplishing á, we plug the vector to control the gradient for the neural topie encoder, while maintaining the update upon the neural topic decoder whose value is determined by the pock-propagation of the reconstruction loss. Our multi-objective framework for setwise contrastive neural topic model is depicted in Algorithm
EXPERIMENTS
In this section, we conduct experiments and empirically demonstrate the effectiveness of the proposed methods for NTMs. We provide the experimental setup and report numerical results along with qualitative studies.
5.1 EXPERIMENTAL SETUP
Benchmark datasets. We adopt popular benchmark datasets spanning various domains, vocabulary sizes, and document lengths for experiments: (i) 20Newsgroups (20NG) (Lang) 1995). one of the most well-known datasets for topic model evaluation, consisting 18000 documents with 20 labels; (ii) IMDb (Maas et al. 2001), the dataset of movie reviews, belonging to two sentiment labels, i.e. positive and negative; (mi) Wikitext-100 (Wiki) (Merity al. 20166, comprising 28500 articles from the Good and Featured section on Wikipedia: (iv) AG News (Zhang et al., 2015), consisting of news titles and articles whose size is 30000 and 1900 for training and testing subsets, respectively.
Evaluation metrics. Following previous mainstream works (Hoyle et al. 2020 Wang et al. 2019
Card et al. 20188] Nguyen & Luu 2021 Xu et al., 2022 Wu al.. 2023h, Zhao et al.] 2020 Wang et al. 2022), we evaluate our topic models concerning topic coherence and topic diversity. In particular, we extract the top 10 words each topic, then utilize the testing split of each dataset as the reference corpus to measure the topics' Normalized Pointwise Mutual Information (NPMI) (Röder et al. 2015) in terms of topic coherence, and the Topic Diversity (TD) metric (Röder et al. 2015) in terms of topic diversity. Furthermore, because topic quality also depends upon the usefulness for downstream applications, we evaluate the document classification performance the produced topics and use F1 score the quality measure.

---

Published as a conference paper at ICLR 2024
Table 2: Topic Coherence and Topic Diversity results on 20NG and IMDb datasets.
20NG
IMDb
Method
Y'= 50
T'
200
7= 50
7 200
NPMI
NPMI
TD
NPMI
TD
NPMI
NTM
0.263=0000
0,731=00.09
0
0.666
0.170=00:0
0.777=0621
0.169==0006
0.690=0.015
ETM
0.305
0.776
0,264:00M
0 0003:
0.174
0.805
0.168:4010
0.687:00M0
DWAE
0.3201.0000
0883
0:2691000
076610001
0.183_00001
0.8%6_nci0
0.173.10006
1.7
BATM
0.31420000
0,786
0.245
0623-0000
0.065=0000
0.619=0016
0.090=0000.
0.652=0.00 00
W-LDA
0.279:00000
0,188 0.719
0.04
0.136=t0007
0.692=0006
0.095
0.666:20109
SCHOLAR
0.7
0.263 Low
0.6341000
0.168 Loacc
0.002,0004
0.140
1
SCHOLAR + BAT
0.324:00006
01
0
0668
0.182=00002
0.825=n0606
0.175
=0
NTM-CL
0.332z0.006
0.853=00005
0,2773mm
0 0.69940004
0.191=0004
0.857=0010
0.186=#000
0.843=0000
HyperMiner
0.305
0.613101ms
0:254
0.646180544
0.182_.0004
0,485_16609
0.177.
1.658
WeTe
0.304:20
0.749gndm
0:254.0ml
0 0.74220005
0.167_00004
0. 0.169 0
TSCTM
0.200
0:668:9.099
0.22640:001
0 0:66:0000
0
0.761
0.145em00C
0.658=00012
Our modiel
0.340::0:66
0.913:4009
0.291:3mm
0.90510004
0.200
0.916:30866
0.197.3003
0.892:0000
Table 3: Topic Coherence and Topic Diversity results on Wiki and AG News datasets.
Wiki
AG News
Method
T
50
T
200
T
50
T
200
NPMI
TD
NPMI
TD
NPMI
TD
NPMI
TD
NTM
0,250_0001
0.817_no0e
20000
0.624_n001
0.197.00m5
0.7291cmm7
0.005
ETM
0.352=0003
0.756=n013
0,317=0000
0.671
0.204
0.736+20.00
0.055=0002
0.63300005
DVAE
0.404=50005
0.815=n000
0.359
0.721
0.271:0016
0.85055m13
0.182:00
0.746
BATM
0.336_0001
0.807
0.319 Lnce
8.732_0002
0:25610mm
0.7541c/0
0.14400003
0.711 10006
W-LDA
0:451 =0012
0 0.836
0.308
0.270aa::53
0.84
0.135:0004
0. 00708
SCHOLAR
0.429=00011
0,821=0000
0.466:n.00
0.860=a..14
0.278
0
0,185=0004
0,747
SCHOLAR
0.824 ruoce
0,455
0.854
0:272
0.8591GIMM
0.189100003
NTM-CL
0.841
0:46220006
0.279 n25
0.890
190g20002
0.700g0006
HyperMiner
0.3M=00005
0.807=00008
0.466
0.861120.012
0.259
0821+
1S00=0001
0,743=000)
0.845:1011
0.453amom
0.827
0:283::00M
0.87818m02
0.18620001
TSCTM
0.327_n0005
0.753_ne00
0,404_nn0s
0.737_.4012
0:252..0m6
0.7831000
166-2000
0.672g0044
Our model
0.4%
0.959
0,491
0.938
0.351
0.934
0.221
0,885
Baseline models. We compare our proposed approaches with the following state-of-theear NTMs: (i) (NTM) (Miao al. 2016), inheriting the encoder-decoder paradigm of the VAE architecture and standard Gaussian as the prior distribution; (ii) (ETM) (Dieng et al. 2020), modeling the topic-word distribution with topic and word embeddings; (iii) (DVAE) (Burkhardt & Kramer (019)), whose Dirichlet is the prior for both topic and word distributions; (iv) BATM (Wang et al. 20200.. which is based on GAN to consist of an encoder, generator, and discriminator; (v) W-LDA (Nan et al.
2019), which utilizes Wasserstein autoencoder as topic model; (vi) SCHOLAR (Card et al. [0188] which incorporates external variables and logistic normal as the prior distribution; (vii) SCHOLAR + BAT (Hoyle et al. 2020). SCHOLAR model applying knowledge distillation in which BERT is the teacher and NTM is the student; (viii) NTM + CL (Nguyen & Luu 202T), which jointly trains evidence lower bound and the individual-base contrastive learning objective; (ix) HyperMiner (Xu
ctt al. 2022). which adopts embeddings in the hyperbolic space to model topics; (x) We'ee (Wang
ct al. 2022), which replaces the reconstruction eттor with the conditional transport distance; (xi) TSCTM (Wu et al., 2022]: A contrastive NTM which uses an auxiliary semantic space to generate positive and negative samples for short text topic modeling.
Implementation details. We evaluate our approach both at T = 50 and T = 200 topics. We experiment with different set cardinality K {1, 2,, 4, 5, 6) and permutation matrix size P (1, 2,2,4,8 4,8, 8, 6,,2). More implementation details can be found in Appendix
5.2
MAIN RESULTS
Overall results. Table and 3 show the topic quality results of our proposed method and the baselines. We run all models five times with different random seeds and record the mean and standard deviation of the results. Our method effectively produces topics that are of high quality in terms of topic diversity and topic coherence. In all four datasets, lower TD scores of baseline methods imply that they generate repetitive and less diverse topics than ours. For example, on 20NG for T = 50 our TD scores all surpass NTM+CL, HyperMiner, and WeTe (0.959 v.s. 0.841, 0.807, and 0.845). With regards to topic coherence, we completely outperform all these methods on NPMI (e.g. on Wiki and T = 200, 0.491 v.s. 0.462, 0.446, and 0.453). We also conduct significance tests and observe that the p-values are all smaller than 0.05, which verifies the significance of our improvements. As
7

---

Published as a conference paper at ICLR 2024
Table 5: Ablation Results on the Objective Control Strategy.
Table 4: Document classification results
with topic representations.
Method
20NG
IMDb
AG News
Method
BATM
30.8
66.0
56.0
T= 50
T= 200
SCHOLAR
UW
0.482=00.009
0.4742:0005
SCHOLAR BAT
52.9
73.1
59.0
GradNorm
0.485:1.0.007
0.469-50.006
NTM+CL
54.4
84.2
76.2
PCGrad
0.489:1:0.011
0,472:40.005
HyperMiner
43.0
RW
0,492-+:0.099
0,476=+0.007
WeTe
48.5
80.3
65.7
Linear-o- 0.25
0.483:10.011
0.487-10.007
Our Model
57.0
86.4
78.6
Linear-o- 0.50
0.491::0.015
0.485+:00004
Model
0.496:10.0110
0.0016000006
Figure 2: (left) Jensen-Shannon for aligned topic pairs of NTM+CL. Nguyen Luu 2021 and
Our Model. (right) The number of aligned topic pairs which Our Model improves upon NTM+CL.
Nguuen Luu 2021
22ng
Cppic Botter thar hed Guntoppt
30
Nocal
Ohengenc: Sooe
H
70
10
0.0
10
25
30
00
60
201g
imdb
wiki
Matched Topic Par (bwd worst match!
Dalssst
such, these results demonstrate the influence of the useful mutual information brought by our setwise contrastive learning method and the balance between the contrastive and ELBO objectives.
Topic-by-topic results. We perform individual comparison upon each of our topics with the aligned one generated by the baseline contrastive NTM. Inspired by (Hoyle al. 2020), we construct bipartite graph connecting our topics with the baseline ones. We adopt competitive linking to greedily estimate the optimal weight for the matching in bipartite graph matching. The weight of every connection is the Jensen-Shannow (JS) divergence between two topics. Each iteration will retrieve two topics whose JS score is the lowest and proceed to remove them from the topic list. The procedure is repeated until the lowest JS score surpasses a definite threshold. Figure (left) depicts the aligned score for three benchmark datasets. Inspecting visually, comparison procedure. As shown in Figure 2 (right), we observe that our approach possesses more high-NPMI topies then the NTM+CL baseline. This demonstrates the setwise contrastive method only improves general topic quality but also manufactures better individual topics.
extract 44 most aligned topic pairs for the
Document classification. We evaluate the effectiveness of our topic representations for the down- stream task. Particularly, we extract latent distributions inferred by NTMs in the T = 50 setting and train Random Forest classifiers with the number of decision trees of 100 to predict the category of each document. Because the Wiki dataset does not include a corresponding label to each input document, we use 20NG, IMDb, and AGNews for this experiment, As shown in Table 4 our model accomplishes the best performance over other NTMs, substantiating the refined usefulness of the topic representations yielded by our model.
5.3 ABLATION STUDY
Objective control strategy. We perform an ablation study and analyze the importance of our proposed multi-objective training framework. In detail, we remove the Parcto stationary solver and then conduct the training with the approaches of linear combination, in which we attach the weight a to the contrastive objective with a E (0.25, 0.5). the Uncertainty Weighting (UW) (Kendall et al. 2018), gradient normalization (GradNorm) (Chen et al.) 2018 Projecting Conflicting Gradients (PCGrad) (Yu et al.) 2020), and Random Weighting (RW) (Lin et al. 2021). Without the multi- objective framework, the contrastive NTM suffers from sub-optimal performance, as shown in Table Hypothetically, this means that in the fixed linear and heuristics weighting systems, the optimization procedure could not efficiently adapt to the phenomenon that the contrastive objective overwhelms the topic modeling task. Hence, contrasting samples might supply excessive mutual information
8

---

Published as a conference paper at ICLR 2024
Table 6: Topic Coherence results with different pooling functions for negative set representation.
Method
50
7- 200
200
200
7
50
T
Minhooling 0. 0.332gn003
0. 0.15000005
01 0.
0.201ana00
0.318aness
02200g00m
MeunPooling
0, 0,334240205
0.283=208
0.194220026
0.195=0000
0.992240395
0.88
0.321 #00544
0.211=500:
SunPooding
0.333 10094
0.191 0.28113000
0.188__cme
0,493100201
0.484 Di0
0.522
0.215
GAP
0.33922004
0.288=300
0. 0.1975093
0 1966::/00
0,49548.005
0.85
0.323
0.221
ESP
0.339
0.286=3000
0.199250054
0.195=5xMm
0.694
0.48
0.324
0.221
PSWE
1.93410006
0.28419001
0.194
0.191
0.494_1930
0.523
0:2181000
MaxPoollig
0.291
0.200
IL197 R200
0,2961mm8
0.35
Table 7: Examples of the topics produced by NTM+CL Nguyen & Luu 2021 and Our Model.
Dataset
Method
NPMI
Topie
20NG
NTM-CI.
0.2766
mouse monitor orange guleway video apple screen card port vgo monitor moniorrs colors video screen eard mhs cards color seagal ninja
Our Model
0.3537
IMDb
NTM-CL.
0.1901
zombnic zombies jet fighter fight helicopter martial arts scagal juckic chan kung bong ninja stunts kong architectural castle archifict buildings grade historic
Our Model
0,3145
STM-CL
0.1070
roller sculpture
Model 0,2513
century huilddnng huilt housse site castle buildings historic listed
which eclipses the useful features for topic learning. Due to the length limit, more experiments upon the objective control strategy can be referred to Appendix B
Pooling functions. We experiment with different pooling operations for negative samples 3 and compare their performance with our choice of MaxPooling. The experiment of pooling functions for positive samples + can be found in Appendix C We adapt MeanPooling, SumPooling, Global Attention Pooling (GAP) (Li et al. 2016), Featurewise Sort Pooling (FSP) (Zhang et al. 2018), and Pooling by Sliced- Wasserstein Embedding (PSWE) (Naderializadeh et al. (021)) As be observed from Table 6 MaxPooling acquires the highest topic coherence quality. We hypothesize that different from other aggregation functions, MaxPooling directly retrieves strong features, thus being able to extract important information and suppress the noise. We also realize that our proposed framework consistently outperforms the individual-basee contrastive NTM, demonstrating the robustness of setwise contrastive learning with a variety of aggregation functions.
5.4 ANALYSIS
Effects of setwise and mustance-based contrastive learning on topic representations. Table [ and Figure iinclude similarity scores of the input with different document samples. Different from NTM+CL, our model pulls together latent representations of semantically close topics. In Figure instance 2 still maintains high equivalence level, despite its noticeable distinction with the input terms of the low-level properties. Furthermore, in Table inserting unfamiliar words does not alter the similarity scale between the input and two document samples. These results indicate that our setwise contrastive learning focuses on encoding topic information and avoids impertinent low-level vector features.
Case study. Table shows randomly selected examples of the discovered topics by the individual- based and our setwise contrastive model. In general, our model produces topics which are more coherent and less repetitive. For example, in the 20NG dataset, our inferred topic focuses on "computer screening" topic with closely correlated terms such as "monitor", "port", and "vga", whereas topie of NTM+CL consists of ""apple" and "orange", which are related to "fruits". For IMDb, our topic mainly involves words concerning "martial art movies". However, NTM+CL mix those words with "zombies", "flight", and "helicopter". In the same vein, there exists irrelevant words within the baseline topic in the Wiki dataset, for example "grade", "roller", and "coaster". while our topic entirely concentrates on "architecture".
CONCLUSION
In this paper, we propose a novel and well-motivated Multi-objective Setwise Contrastive Neural Topic Model. Our contrastive approach incorporates the set concept to encourage the learning of common instance features and potentially more accuracy construction of hard negative samples. We propose to adapt multi-objective optimization to efficiently polish the encoded topic representations. Extensive experiments indicate that our framework accomplishes state-of oftte-of-the-eee the-hhrtt performance in terms of producing both high-quality and useful topics.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENTS
This research/project is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: SGGG3-PDD-2223-8-00555 Thong Nguyen is supported by Google Ph.D. Fellowship in Natural Language Processing.
REFERENCES
Andrea Asperti and Matteo Trentin. Balancing reconstruction enror and bullback-leible divergence in variational autoencoders. IEEE Access, 11199440-199445 2020.
Seyed Ali Bahrainian, Martin Jaggi, and Carsten Eickhoff. Self-supervised neural topic modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 3341-3350, 2021.
Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training
hot topic: contextualized document embeddings improve topic coherence. In 59th Annual Meeting of the Association for Computational Linguisties and the I Ith International Joint Conference on Natural Language Processing. Association for Computational Linguistics, 2021.
Samuel Brody and Noemie Elhadad. An unsupervised aspect-sentiment model for online reviews. In Human language technologies: The 2010 annual conference of the North American chapter of the association for computational linguistics, pp. 804-812, 2010.
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the dirichlet variational autoencoder topic model. Mach. Learn. Res., 20( (0(111):1211 2019.
Da Cao, Xiangnan He, Liqiang Nie, Xiaochi Wei, Xia Hu, Shunxiang Wu, and Tat-Seng Chua. Cross-platform app recommendation by jointly modeling ratings and texts. ACM Transactions on Information Systems (TOIS), 35(4):1-27, 2017.
Dallas Card, Chenhao Tan, and Noah A Smith. Neural models for documents with metadata. 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-9924, 2020.
Zhao Chen, Vijay Badrinarayanan, Chen-Yo Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balaneing in deep multitask networks. In International conferecoonalcooffrenccc on machine learming, pp. 794-803. PMLR, 2018.
Jean-Antoine Désidéri. -uldiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, (550556))313-3111 2012.
Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling embedding spaces. Transac- tions of the Association for Computational Linguisties, 8:439 453, 2020.
Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hesc: Hierarchical contrastive selective coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9706-9715, 2022.
Pankaj Gupta, Yatin Chaudhary, Thomas Runkler, and Hinrich Schuetze. Neural topic modeling with continual lifelong learning. In International Conference on Machine Learning, pp. 3907-3917. PMLR, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Manni Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference Learning Representations, 2018.
10

---

Published as a conference paper at ICLR 2024
Alexander Hoyle, Pranav Goel, Andrew Hian-Cheome Denis Peskov, Jordan Boyd-Grabes and Philip Resnik. Is automated topic model evaluation broken? the incoherence of coherence. Advances in neural information processing systems, 00:2018-2033 2021.
Alexander Miserlis Hoyle, Pranav Goel, and Philip Resnik. Improving neural topic models using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Hamed Jelodar, Yongli Wang, Mahdi Rabbani, and SeyedValyAllah Ayobi. Natural language processing via Ida topic model in recommendation systems. arXiv preprint wrXiv:1909009551 2019.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Sellreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9619-9628, 2021.
Diederik P Kingma and Max Welling. 1rXiv:13/2.611, 2013.
Auto-encoding variational bayes.
arXiv preprint
Ken Lang. Newsweeder: Learning filter netnews. In Machine Learning Proceedings 1995, pp. 331-339. Elsevier, 1995.
Duong Hoang Le and Binh-Son Hua. Set discrimination contrastive learning. 2022.
Tuan Le and Leman Akoglu. Contravis: contrastive and visual topic modeling for comparing document collections. The World Wide Web Conference, pp. 928-938, 2019.
Jiacheng Li, Jingbo Shang. and Julian McAuley. Uctopic: Unsupervised contrastive learning for phrase representations and topic mining. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6159-6169, 2022.
Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi. Prototypical contrastive learning of unsuper- vised representations. In International Conference on Learning Representations, 2020.
Yujia Li, Richard Zemel, Mare Brockschmidt, and Daniel Tarlow. Gated graph sequence neural networks. In Proceedings of ICLR' 16, 2016.
Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, and Xi Peng. Contrastive clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8547-8555, 2021.
Jian Liang, Kaixiong Gong, Shuang Li, Chi Harold Liu, Han Li, Di Liu, Guoren Wang, et al. Pareto domain adaptation. Advances in Neural Information Processing Systems, 34:12917-12929, 2021.
Baijiong Lin, Feiyang Ye, and Yu Zhang. A closer look at loss weighting in multi-task learning. arXiv preprint 1xXiv2211..1660 2021.
Shuyu Lin, Stephen Roberts, Niki Trigoni, and Ronald Clark. Balancing reconstruction quality and regularisation elbo for vaes. arXiv preprint urXiv:1909.03760 2019a.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. Advances in neural information processing systems, 32, 2019b.
Xi Lin, Zhiyuan Yang, Qingfu Zhang, and Sam Kwong. Controllable pareto multi-task learning. arXiv preprint 0rXiv:2010.06313, 2020.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pp. 142-150, 2011. 1.
11

---

Published as a conference paper at ICLR 2024
Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with preferences: Gradient descent with controlled ascent in pareto optimization. In International Conference on Machine Learning. pp. 6597-6607. PMLR, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. 2016.
Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference on machine learning, pp. 1727-1736. PMLR, 2016.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representa- tions in vector space. arXiv preprint arXiv:1301.3781, 2013.
Navid Naderializadch, Joseph F Comer, Reed Andrews, Heiko Hoffmann, and Soheil Kolouri. Pooling by 34:3389-3400, 2021.
embedding. Advances in Neural Information Processing Systems,
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with wasserstein au- toencoders. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 6345-6381, 2019.
Cong-Dup Nguyen, Thong Nguyen, Duc Vu, and Anh Luu. Improving multimodal sentiment analysis: Supervised angular margin-based contrastive learning for enhanced fusion representation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 14714-14724, 2023.
Thong Nguyen and Anh Tuan Luu. Contrastive learning for neural topic model. Advances in Neural Information Processing Systems, 04:111974111 1996, 2021.
Thong Nguyen, Anh Tuan Luu, Truc Lu, and Tho Quan. Enriching and controlling global semantics for text summarization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9443-944 2021.
Thong Nguyen, Xiaobao Wu, Anh-Tuan Luu, Cong-Duy Nguyen, Zhen Hai, and Lidong Bing. Adaptive contrastive learning on multimodal transformer for review helpfulness predictions. arXiv preprint arXiv:2211.035222 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint /807.03788 2018.
Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM international conference on Web search and data mining, pp. 399-408, 2015.
Oleh Rybkin, Kostas Daniliidis, and Sergey Levine. Simple and effective vae training with calibrated decoders. International Conference on Muchine Learning, pp. 9179-9189. PMLR, 2021.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018.
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint ar Xiv: httv:/7733. 703.0/488, 2017.
Shixiang Tang, Peng Su. Dapeng Chen, and Wanli Ouyang. Gradient regularized contrastive learning for continual domain adaptation. Proceedings of the AAAI Conference on Artificial Intelligence. volume 35, pp. 2665-2673, 2021.
Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Dyadic memory networks for aspect-based senti- ment analysis. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 107-116, 2017.
Yi Tay, Luu Tuan, and Siu Hui. Couplenet: Paying attention couples with coupled attention for relationship recommendation. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12, 2018.
12

---

Published as a conference paper at ICLR 2024
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 01166277-839, 2020.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. In International Conference on Learning Representations, 2019.
Luu Anh Tuan, Darsh Shah, and Regina Barzilay. Capturing greater context for question generation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 9065-9072, 2020.
Gerson Vizcarra and José Ochoa-Luna. Paraphrase generation via adversarial penalizations. In Proceedings of the Sixth Workshop on Noisy esser-eeerrated Text (W-NUT 2020). pp. 249-259, 2020.
Dongsheng Wang, Dandan Guo, He Zhao, Huangjie Zheng, Korawat Tanwisuth, Bo Chen, and Mingyuan Zhou. Representing mixtures of word embeddings with mixtures of topic embeddings. arXiv preprint arXiv:2203.01557, 2022.
Rui Wang, Deyu Zhou, and Yulan He. Atm: Adversarial-neural topic model. Information Processing & Management, 56(6):102098, 2019.
Rui Wang, Xuemeng Hu, Deyu Zhou, Yulan He, Yuxuan Xiong, Chenchen Ye, and Haiyang Xu. Neural topic modeling with bidirectional adversarial training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 340-350, 2020.
Xudong Wang, Ziwei Liu, and Stella Yu. Unsupervised feature learning by cross-level instance- group discrimination. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12586-12595, 2021a.
Yiming Wang, Ximing Li, and Jihong Ouyang. Layer-assisted neural topic modeling over document 2021b.
networks. In IJCAI, pp.
Xiaobao Wu and Chunping Li. Short Text Topic Modeling with Flexible Word Patterns. In Interna- lional Joint Conference on Neural Networks, 2019.
Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. Learning Multilingual Topics with Neural Variational Inference. In International Conference on Natural Language Processing and Chinese Computing, 2020a.
Xiaobao Wu, Chunping Li, Yan Zhu, and Yishu Miao. Short text topic modeling with topic distribution quantization and negative sampling decoder. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1772 1782, Online, November 2020b.
Xiaobao Wu, Chumping Li, and Yishu Miao. Discovering topics in long-tailed corpora with causal intervention. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 175-185, Online, August 2021. Association for Computational Linguisties.
Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 2748-2760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Xiaobao WU, Xinshuai Dong, Thong Nguyen, Chaoqun Liu, Liang-Ming Pan, and Anh Tuan Luu. Infoctm: mutual information maximization perspective of cross-lingual topic modeling. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 13763-13771, 2023a.
Xiaobao WU, Xinshuai Dong, Thong Thanh Nguyen, and Anh Tuan Luu. Effective neural topic modeling with embedding clustering regularization. In International Conference Machine Learning, pp. 37355- PMLR, 2023b.
Xiaobao WU, Fengjun Pan, and Anh Tuan Luu. Towards the topmost: A topic modeling system toolkit. arXiv preprint S3iv:2309006900 2023c.
13

---

Published as a conference paper at ICLR 2024
Xiaobao Wu, Thong Nguyen, and Anh Tuan Luu. A survey neural topic models: Methods, applications, and challenges. Artificial Intelligence Review, 2024a.
Xiaobao Wu, Fengjun Pan, Thong Nguyen, Yichao Feng, Chaoqun Liu, Cong-Duy Nguyen, and Anh Tuan Luu, On the affinity, rationality, and diversity of hierarchical topic modeling. arXiv preprint arXiv:2401. 140::24401.1113,,. 2024b.
Yi Xu, Dongsheng Wang, Bo Chen, Ruiying Lu, Zhibin Duan, Mingyuan Zhou, et al. Hyperminer: Topic taxonomy mining with hyperbolic embedding. Advances Neural Information Processing Systems, 35:31557-31155 2022.
Wenpeng Yin and Hinrich Schütze. Diseriminative phrase embedding for paraphrase identification. arXiv preprint arXiv: /r04v 000.0000.000,, 2016.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Linhai Zhang, Xuemeng Hu, Boyu Wang, Deyu Zhou, Qian- Wen Zhang, and Yunbo Cao. Pre-training and fine-tuning neural topic model: simple yet effective approach to incorporating external knowledge. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5980-5989, 2022.
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. Neural topic model via optimal transport. In International Conference Learning Representations, 2020.
Konglin Zhu, Lin Zhang, and Achille Pattavina. Learning geographical and mobility factors for mobile application recommendation. IEEE Intelligent Systems, 32(3):36-44, 2017.
14

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Hyperparameter Settings
15
B Further Study of Objective Control Strategies
15
Further Study of Pooling Function for Positive Set Representation
15
D Additional Evaluation of Topic Quality
E
Examples of data augmentation methods
16
Statistical Significance Tests towards Topic Quality
16
Dynamics Multi-objective optimization
Human Evaluation: Human Ratings and Word Intrusion tasks
17
A HYPERPARAMETER SETTINGS
In 8 we denote hyperparameter details of our neural topic models, i.e. learning rate 7), batch size B, and the temperature 7 for the InfoNCE loss. For training execution, the hyperparameters vary with respect to the dataset.
Table 8: Hyperparameter Settings for Neural Topic Model Training.
20NG
IMDb
Wiki
Hyperparameter
T' = 50
T = 200
T = 50
T = 200
T = 50
T
200
sample set size K
3
permutation matrix size
8
8
8
8
8
8
temperature
0.2
0.2
0.2
0.2
0.2
0.2
learning rate 7)
0.002
0.002
0.002
0.002
0.001
0.002
batch size
200
200
200
200
500
500
B FURTHER STUDY OF OBJECTIVE CONTROL STRATEGIES
We extend the comparison oujti-objecttve optimization (MOO) and other control strategies on 20NG and IMDb datasets in Table 9 In general, find that our adaptation of gradient-based MOO balances the contrastive and ELBO losses more proficiently to obtain better topic quality.
C FURTHER STUDY OF POOLING FUNCTION FOR POSITIVE SET REPRESENTATION
In this appendix, we extend our ablation study of different choices of pooling functions = for negative set in Section(533 to pooling function y for positive set representation. In particular, similar to Section [5.3] also compare choice of pooling function for positive sets, i.e. MinPooling, with MaxPooling, McanPooling, SumPooling, GAP (Li et al. 2016). FSP (Zhang et al. 2018), and
PSWE (Naderializadeh al. 2021). The results denoted Table 10
15

---

Published as a conference paper at ICLR 2024
Table 9: Topic Coherence and Topic Diversity results on 20NG and IMDb datasets with different objective control strategies.
20NG
IMDb
Method
T
50
200
T =
50
T
200
NPMI
TD
NPMI
TD
NPMI
TD
NPMI
TD
UW
0.028
0.837=0001
0.273m000
0.8311200
0.187400044
0.848:2000
0.17520.00
0.81820000
GradNorm
0.038
0.870
0.276220806
0.8391:006
0. 189
0.873
0.187
2.00
PCGrrad
0.327
0.857=0.005
0.272=0.001
0.828
0.189 0.189
0.872=0000
0.186=0011
0.830=a006
+W
0.329:+0.00
0.864=6.009
0.274
0,833+a003
0. 186200003
0.862:00:05
0.181
0.823#000
Linear
0.25
0.327:+0.066
0.862=6.007
0.275=0000
0.835
0. 10.8040
0.853=20001
0.182:000
0.825:00m
Linear a 0.5
0.322:00,09
0.850==0.005
0.272=0.801
0.824::.004
0.187
0.839
0.184=0.006
0.829
Our Model
0.340::9.005
0.913
0.905:10mm
0.200
0.916:2.005
0.197
0.892:10M1
Table 10: Topic Coherence results with different pooling functions for positive set representation.
20NG
AGNews
Method
T 200
T 50
MinPooling
0.333200000
0.284
0.195=0.006
0.191
0,494
0.48440000
0.324:10.008
0.211 10:001
McanPooling
0.335_10000
0.284
0.197-0006
0.194:10.007
0,492:10.00
0.481 1100066
0.324:10.013
0.213-10:022
SumPooling
0.336_10000
0.286110004
0.195-0.007
0.186.10.008
0.493
0.481110:00
0.32010.013
0.212:10001
GAP
0.336:10 006
0.289
0.196=00006
0.186:200005
0.4922:0010
0.482:10007
0.318:10.09
0.210:10002
0.336_10.004
0,285 1000
0.197_00.06
0.19110.010
0.40410..012
0.484
0.323:10.004
0.208
PSWE
0.3
0.285
0.198=000
0.187:00.006
0..94
0.484:20.00
0.322:10.008
0.211
MaxPooling
0.340=0.00s
0,291=
0.200=
0.197=00003
0.496=9..00
0.491=0.06
0.351
0.221=n.000
The table demonstrates that our choice of MinPooling for positive set representation construction provides the most effective topic quality. We hypothesize that learning to capture similarity be- tween strong features does not supply sufficiently hard signal for the model. As such, noo-inoiing will encourage topic model to focus on more difficult mutual information that can polish topic representations.
D ADDITIONAL EVALUATION OF TOPIC QUALITY
We investigate larger scope for our proposed setwise contrastive topic model, where we train prior baselines and our model on Yahoo Answer (Zhang et al. (0155) dataset, which consists of about 1.4 million training and testing documents. The illustrated results in Table TT demonstrate that setwise contrastive topic model does polish the produced topics better than other topic models, proving both of its effectiveness and efficacy.
EXAMPLES OF DATA AUGMENTATION METHODS
In this section, we Ilustrate the examples generated by our augmentation, TFiDF-based, and embedding- based in Table 12 13 and [4 respectively. For embedding- based augmentation, use
Word2Vec (Mikolov et al. 2013) as the pretrained word embedding embedding.
STATISTICAL SIGNIFICANCE TESTS TOWARDS TOPIC QUALITY
this appendix, to more rigorously assess the quality of our generated topics compared with baseline models, we conduct statistical t-test for performance comparison in Section 5 In particular, select the second-be best method in all Tables, í.e. NTM+CL, Linear-a = 0.5, NTM+CL, NTM+CL, GAP, and GAP in Table and The results show that our improvement is statistically significant with all
6 and 10 respectively. We provide our obtained p-values in Table
15 16
valuue < 0.05.
16

---

Published as a conference paper at ICLR 2024
Table 11: Topic Coherence and Topic Diversity results on Yahoo Answers dataset.
Yahoo Answers
Method
T
50
T 200
NPMI
TD
NPMI
TD
NTM
0.290++00003
0.897
0.000
0.718=0.004
ETM
0.207
0.017
0.158:t00002
0.709=0.007
DVAE
0.009
0.932+00014
0.163
0.721=0.004
BATM
0.247:+0.004
0.819
0.177
0.767
W-LDA
0.222++0.004
0.000
0.003
0.000
SCHOLAr
0.007
0,936+10019
0.000
0.784
SCHOLAr+BAT
0.008
0.940=+0020
0.196:10.001
0.703
NTM+CL
0.954 0.004
0.198
0.794=0.003
Our Model
0.348:10.004
0.988
0.882 0.200
Table 12: Examples generated by our augmentation.
Our method
Input document
x
ronaldo poaches the points owen tastes first real.
ronaldo's 's scoring prowess on full display as he secured the
the sunsel the painted the sky with hues of or- ange and pink, casting a peace- ful aura over the coastal town.
victory for real, while owen ex- of suc-
perienced his first cess with the team.
focusing on a compelling story or unique cinematic style often lead to more engaging movie experience, regardless star power the actors volved. finally, way to Google your hard drive You can usually find information more easily on the vast internet.
quite often find that when film doesn't evolve amund
the cat jumped over the fence and landed gracefully in the neighbor's yard.
a famous actor or actress but
rather a story or style.
with this new tool, you'll be to search your hard drive just as efficiently you browse the vast expanse of the internet.
the sound laughter echoed through the empty park.
bioshield effort is inadequate, a study says experts say the bioshield program will not udequate create cines and drugs to protect the nation against either biologi- attack natural epidemic.
the study's 's findings cast doubt on the effectiveness of the bioshield program in developing the necessary vaccines and drugs to safeguard the nation from hi- ological threats, according to ex- perts.
the chef carefully placed the gur- nish on the plute, completing the presentation the dish.
vac-
G DYNAMICS OF a IN MULTI-OBJECTIVE OPTIMIZATION
We inspect the change of alpha during training to better understand the role of our setwise contrastive learning for neural topic model. Figure 3 demonstrates that in the beginning, alpha is fairly small, which indicates that initially focusing on the ELBO objective of neural topic model is more effective. Subsequently, the value of alpha increases, denoting that contrastive learning gradually affects the training. This can be considered as an evidence that validates the effect of our setwise contrastive learning on neural topic model.
H HUMAN EVALUATION: HUMAN RATINGS AND WORD INTRUSION TASKS
Since human judgement towards the generated topics might deviate from the results of automatic metrics (Hoyle et al.) 2021), we decide to conduct human evaluation on the 20NG and IMDb datasets through utilizing two tasks, i.e. human ratings and word intrusion task. Our evaluation is conducted upon Amazon Mechanical Turk (AMT). In particular, for human ratings, we ask annotators to rate the generated topics on the scale from to 3. For word intrusion task, in each topic, we insert an outsider word and ask human annotators select which word is the outsider word. Our human evaluation is performed on our model and the NTM+CL model of (Nguyen & Luu 2021). We record the human As can be observed, human judgements prefer our
evaluation results and plot them in Figure 4
17

---

Published as a conference paper at ICLR 2024
Table 13: Examples generated by TFIDF-based augmentation
(guyen Luu 202
TFaDF-based
Input document x
x
X
ronaldo poaches the points as owen tastes first real.
cristiano cambone this re- bounds one bellamy distinetly fourth gives.
maradona lem following 18 own ritchic mixes 1998 rather.
quite often find that when film duesn't evolve around a famous actor or actress but rather story or style.
never seemed ones way he with screenplay reason alter apart with notable film rather girlfriend only meant with explains rather simple.
get rarely perhaps still actu- ally turn put episode come nothing conversely instead besides scriptwriter there ms. while instance put inspired there quirky.
finally, to Google your hard drive You can usually find information more easily on the vast intemet.
once both with it move online get turning stopping mayhe if instead way account still quickly which this wealth web- sites.
turned instead lets want lot hard lot these less still link longer none last fol- lowing boasts msn.
once rest
bioshield effort is inadequate, a study says experts say the bioshield program will not be adequate to create new vac- cines and drugs protect the nation against either biologi- cal attack or natural epideniic.
nobelity helping part suffi- ciently both with suggests ex- plained researchers know this nobelity planning ready any even providing move rather its doses made treamment move threatened this entire losing however with possessed fire rather unique malaria.
circumvents promises once poorly instead put applied nothing impact n't follow- ing circumvents expanded step thought cannot suitable rest core introduced medication few antiviral rest need fol- lowing strongest twice as put analyzing enemy there hence spreads.
Table 14: Examples generated by embedding- augmentation
Yin & Schüitze 2016
Embedding-
Input document
ronaldo poaches the points as owen tastes first real.
ronaldinho strafaci which
artd newsfcatures yahd nzse40
point also keane taste second kind.
404-526-5456 3,399 leel bised highjump.
quite often find that when a film docsn't 't evolve around famous or actress hut rather story or style.
me very sometimes how but then an movic if know evoly- ing where famed starred in- stead actor though merely an book instead styles.
statistique kunzig 99 deylam suhn micale lacques nzsc40 xinglong myoo filh laeques -5000 basse-normandic zahr debchogne ectacea lacques arabised zahr tih.
finally, a way to google your hard drive you can usually find information more easily on the vast internet.
again an so take yahoo my drives 'Il able typically how souree than either. which surrounding web.
sahk lacques yahd raht 1-732-390-4480 milanka mu- sienotes.com 1-732-390-4480 photoexpress subn deylam tahn 100.00 zhahn romanosp- times.com newsfeatures puh condita.
bioshield effort is inadequate, a study says experts say the bioshield program will not be adequate to create new vac- cines and drugs protect the nation against either biologi- cal attack or natural epidemic.
toshika e efforts this insu fficient an studies thinks specialists be- lieve which toshka pmgrams would be not sufficient take creating for vaccine well drug take protecting which country opponents instead an chemical attacks instcad abundant out- break.
chivenor wmorrisgooee.com nehn europe/africa sahk lacques ortsgemeinden tele- gram.com com gahl e-bangla chivenor sholdersptimes.com ortsge- meinden milanka 47 47-42-80-44 sylviidac yahd troo yanow chron.com bizmags tilove yahd lelegram.con newsfea- tures juh speoch/language tvcolumn lacques fassihi 2378 zahr berris bsc- 100.
newsfeatures
generated topics to the topics of NTM+CL (Nguyen & Luu 202T). The inter-annotator agreement is 0.72, which indicates high agreement among human annotators.
18

---

Published as a conference paper at ICLR 2024
Table 15: Statistical test for document classification results using topic representations of Our model and NTM+CL.
Metric
20NG
IMDb
AGNews
p-value
0.048
0.023
0.019
Table 16: Statistical test for the ablation study on objective control strategy of Our model and Linear-o = 0.5.
Wiki
Metric
T 50
T 200
p-value
0.010
0.039
Table 17: Statistical test for the comparison the topic coherence and topic diversity results between Our model and NTM+CL on 20NG and IMDb datasets.
20NG
IMDb
Metric
=
50
T
200
T 50
T 200
NPMI
TD
NPMI
TD
NPMI
TD
NPMI
TD
p-value
0.034
0.032
0.047
0.047
0.024
0.029
0.015
0.024
Table 18: Statistical for the comparison the topic coherence and topic diversity results between Our model and NTM+CL on Wiki and AG News datasets.
Wiki
AG News
Metric
T 50
T' 200
T 50
T' 200
NPMI
NPMI
NPMI
NPMI
p-value
0.037
0.044
0.005
0.018
0.033
0.047
0.029
0.026
Table 19: Statistical test for the comparison of pooling functions for negative samples between MaxPooling and the GAP method.
20NG
IMDb
Wiki
AGNews
Metric
T 50
T = 200
T = 50
T = 200
T = 50
T = 200
= 50
T = 200
p-value
0.044
0.013
0.013
0.017
0.001
0.005
0.008
0.019
Table 20: Statistical test for the comparison pooling functions for positive samples between our MinPooling and the GAP method.
20NG
IMDb
Wiki
AGNews
Metric
T 50
200
T 50
200
T 50
200
T 50
T 200
p-value
0.017
0.033
0.004
0.047
0.030
0.036
0.011
0.003
19

---

Published as a conference paper at ICLR 2024
Figure 3: Dynamics of a during training on four datasets 20NG, IMDb, Wiki, and AGNews with T = 50.
Dynamies of during training.
0.30
20NG
IMDb
0.25
AG News
0.20
Wiki
3 0.15
0.10
0.05
0.00
100
200
300
400
500
Epoch
Figure 4: Human evaluation 20NG and IMDb datasets with two tasks: human ratings and word intrusion.
Ratings
Intrusion
2.9
2.8
2.7
0.80
2.6
20NG
0.75
2.5
2.4
2.3
2.2
0.65
Our Model
NTM+CI.
Our Model
NTM+CL.
Ratings
Intrusion
0.75
0.65
IMDb
2.3
0.60
2.2
2.1
0.35
2.0
0.50
1.9
Our Model
NTM+CL
Our Model
NTM+CL
20


Reviews:
[{"review_id": "sv7HjXnCkQ", "reviewer": "Reviewer_ioHf", "summary": "This paper aims to address challenges in neural topic models (NTMs) caused by recent approaches that optimize the combination of ELBO and contrastive learning. These challenges include capturing low-level mutual information and a conflict between ELBO's focus on input details for reconstruction quality and contrastive learning's goal of generalizing topic representations. To overcome these issues, the authors propose a novel setwise contrastive learning method for sets of documents, aiming to capture shared semantics among documents. Additionally, they formulate contrastive topic modeling as a multi-objective optimization problem to achieve a balanced solution. Experimental results on 4 benchmark datasets demonstrate that their approach consistently produces higher-performing NTMs in terms of topic coherence, diversity, and document classification performance compared to existing methods.", "questions": "- Could you conduct statistical significance tests to compare your method with the baselines in the experiment tables?\n\n- Could you perform human evaluation (e.g., the intrusion test) to directly examine the quality of extracted topics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "+ The key idea of this paper (i.e., learning low-level mutual information of neural topic models that optimize ELBO and contrastive learning together) is very well-motivated.\n\n+ The usage of setwise contrastive topic modeling is reasonable. Casting it as a multi-task learning problem and adopting multi-objective optimization to find a Pareto solution are technically novel.\n\n+ A comprehensive set of benchmark datasets and baselines are considered. The authors also perform detailed ablation studies, hyperparameter analyses, and case studies.", "weaknesses": "- Statistical significance tests are missing. It is unclear whether the gaps between the proposed model and baselines/ablation versions are statistically significant or not. In particular, some gaps in Tables 3 and 6 are quite subtle, and the variances of classification scores in Table 2 are unknown, therefore p-values should be reported.\n\n- Only automatic metrics (e.g., NPMI and TD) are used to evaluate topic quality. Although the authors also examine document classification as a downstream task, the classification performance is just an indirect measurement of topic quality. Recent work [1] has shown that automatic metrics may deviate from humans' judgment. Therefore, I feel human evaluation (e.g., the intrusion test [2]) is still needed.\n\n[1] Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence. NeurIPS'21.\n\n[2] Topic Intrusion for Automatic Topic Model Evaluation. EMNLP'18.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMzhAWKGmz", "reviewer": "Reviewer_3szg", "summary": "This paper proposes a setwise contrastive learning algorithm for neural topic model to address the problem of learning low-level mutual information of neural topic models. This work explicitly casts contrastive topic modeling as a gradient-based multi-objective optimization problem. Extensive experiments are performed to demonstrate the effectiveness of this method.", "questions": "1. I have a question about the baselines. Table 5 shows topic diversity (TD) score of WeTe is 0.878±0.012 when T = 50 in AG News dataset. However, in [1], this score is 0.966 without data augmentation and 0.991 with data augmentation. There is a large gap between these scores. I wonder how authors implement these baselines. Could you please provide more details or any codes?\n\n[1] Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. arXiv preprint arXiv:2211.12878, 2022.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized and equations are clearly written.\n2. Extensive experimental are performed and results show this method consistently presents high performance.\n3. Codes are provided in supplementary materials to ensure reproducibility.", "weaknesses": "1. Since [1] also uses contrastive learning to capture useful semantics of topic vectors which is similar to the proposed method, this paper does not clearly compare with [1] and explain its novelty. \n2. This paper omits important baselines. For example, [1] also presents great performance in this task but this paper does not compare with it in the experiments. Which contrastive learning method performs better? \n\n[1] Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. arXiv preprint arXiv:2211.12878, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O85BU9IVf5", "reviewer": "Reviewer_QHKf", "summary": "This paper proposes a set-wise contrastive learning algorithm for neural topic models, where the input document batch is divided into sets and positives and negatives are constructed by augmenting and pooling instances in each set. It casts the proposed method as a multi-objective optimization problem to balance the trade-off between the ELBO and the contrastive objectives. The ChaptGPT API is used for data augmentation to generate negative samples. Experiments demonstrate the effectiveness of the proposed method.", "questions": "In Section 4.1, the authors adopt ChatGPT API to perform the augmentation with the specific prompt.\nHowever, at the end of Section 5.1, they use Word2Vec as pretrained embedding for their embedding-based augmentation.\nThis is confusing. Do they perform both ChatGPT-based and embedding-based augmentations for comparisons? \nAnyway, they should show the effectiveness of the ChatGPT-based augmentation. \n\nIn Section 5.5 on page 9, they justify the use of MaxPooling, for extracting set representation,  as MaxPooling directly retrieving strong features. In Table 6, they compare different pooling functions including max and min poolings. \nHowever, in Algorithm 1, both MaxPool and MinPool are used. Here, MinPool is used for positive pair. \nThey should clarify the reason why.\n\nThe purpose of multi-objective optimization is to find a  (possibly diversified) set of Pareto optimal solutions so that it represents the Pareto frontier. Therefore, I would like to know the average and standard deviation of the $\\alpha$  obtained at the end of each run to see how it is diversified. If it is not diversified, then I would like to know the reason.  \n\nJudging from the definition of $\\mathcal{L}_{set}$ in Algorithm 1, both the positive and the negative pairs are generated within the same set and the loss is then accumulated for all the sets. If this is the case, then the first sentence of the Hard Negative Sampling paragraph sounds confusing. Differenciating among sets may be more difficult, but the contrastive pair for differenciating is generated within a set not among sets.  In Algorithm 1,  $\\exp(s_i^{min}, s_i^{+})$ should be $\\exp(f(s_i^{min}, s_i^{+}))$. \n\nHow and what value for the contrastive weight $\\beta$ is determined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The set-wise contrastive learning is new and effective for resolving low-level mutual information of neural topic models. \n- Formulating the contrastive learning as a multi-task learning problem and solving it by a multi-objective optimization algorithm is an interesting idea. \n- Experimental comparisons with state-of-the-art neural topic models that include recent ones such as WeTe demonstrate the effectiveness of the proposed moethod.", "weaknesses": "- The effectiveness of the ChatGPT-based data augmentation is unclear.\n- The formal definition of the contrastive loss is  missing in the main text, while incomplete definition can be found in Algorithm 1. \n- The justification of the use of MaxPooling is unclear. \n- The authors seem to assume as if it is possible to find \"the optimal\" Pareto solution (a Pareto optimal solution with optimal balance), while there is no superiority or inferiority between Pareto optimal solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)