Neural Topic Modeling based on Cycle Adversarial Training and Contrastive Learning
Year: 2023

Abstract:
Neural topic models have been widely used to extract common topics across documents. Recently, contrastive learning has been applied to variational autoencoder-based neural topic models, achieving promising results. However, due to the limitation of the unidirectional structure of the variational autoencoder, the encoder is enhanced with the contrastive loss instead of the decoder, leading to a gap between model training and evaluation. To address the limitation, we propose a novel neural topic modeling framework based on cycle adversarial training and contrastive learning to apply contrastive learning on the generator directly. Specifically, a self-supervised contrastive loss is proposed to make the generator capture similar topic information, which leads to better topicword distributions. Meanwhile, a discriminative contrastive loss is proposed to cooperate with the self-supervised contrastive loss to balance the generation and discrimination. Moreover, based on the reconstruction ability of the cycle generative adversarial network, a novel data augmentation strategy is designed and applied to the topic distribution directly. Experiments have been conducted on four benchmark datasets and results show that the proposed approach outperforms competitive baselines.

heading: 1 Introduction
text: Topic modeling, uncovering the semantic structures within a collection of documents, has been widely used in various natural language processing (NLP) tasks (Zhou et al., 2017; Yang et al., 2018, 2019; Zhou et al., 2021; Wang et al., 2022). Latent Dirichlet Allocation (LDA) (Blei et al., 2003), a probabilistic graphical model, is one of the most popular topic models due to its interpretability and effectiveness. However, the parameter estimation methods for LDA and its variants, such as collapsed Gibbs sampling (Griffiths and Steyvers, 2004), are model-specific and require specialized derivations.
∗Corresponding author.
To tackle such disadvantages, neural topic models have been proposed with a flexible training process, which can be divided into two categories, variational autoencoder (VAE) based and generative adversarial network (GAN) based. VAE-based neural topic models regard the encoded latent vector as the topic distribution of the input document, then employ the decoder to reconstruct the word distribution (Miao et al., 2016; Srivastava and Sutton, 2017; Miao et al., 2017; Card et al., 2018; Wang et al., 2021). To address the limitation that VAE-based neural topic models cannot approximate Dirichlet distribution precisely, Wang et al. (2019) propose an adversarial topic model, in which the topic distribution is sampled from the Dirichlet prior distribution directly and transformed into the word distribution by the generator. In order to uncover topic distribution and infer document topic simultaneously, Bidirectional Adversarial Topic model (Wang et al., 2020) and Topic Modeling with Cycle-consistent Adversarial Training (Hu et al., 2020) have been proposed in turn.
Recently, a neural topic model named CLNTM has been proposed to apply contrastive learning to VAE-based neural topic models (Nguyen and Luu, 2021). A data augmentation strategy is proposed to replace the salient and non-salient parts of the document representation according to word frequency information to construct positive and negative examples. Although achieving promising results, it has such a disadvantage. As shown in Figure 1, due to the limitation of the unidirectional structure of VAE, the encoder is optimized through the contrastive loss, instead of the decoder which generates topic-word distribution, leading to the gap between model training and evaluation.
Therefore, in this paper, we consider discovering topics based on cycle adversarial training and contrastive learning. As illustrated in the lower left part of Figure 1, by incorporating cycle adversarial training, topic distribution θ and word
9720
distribution x will be transformed bidirectionally, breaking the structural limitations of VAE-based neural topic models. However, it is not straightforward to combine contrastive learning and cycle adversarial training. On the one hand, it is crucial to construct applicable positive samples for topic distributions. On the other hand, it is hard to improve the learning of topic-word distribution while maintaining the bidirectional mapping ability of cycle adversarial training.
To overcome the above challenges, we propose a novel Neural Topic Modeling framework based on Adversarial training and Contrastive Learning (NTM-ACL). A self-supervised contrastive loss is employed to make the generator capture similar topic information between positive pairs. The generation of topic-word distribution is improved directly, which mitigates the gap between model training and evaluation. Meanwhile, a discriminative contrastive loss is designed to cooperate with supervised contrastive loss to avoid the adversarial training being undermined by the unbalance between generation and discrimination. Moreover, data augmentation is applied to construct positive samples of topic distribution with the reconstruction ability of cycle generative adversarial network structure. The minimum items in the reconstructed distribution are substituted for corresponding items in the original distribution, which hasn’t been explored before. We conduct extensive experiments to fully exploit the effectiveness of our proposed model.
In a nutshell, the main contributions of our paper can be summarized as follows:
• We propose NTM-ACL, a novel neural topic modeling framework where contrastive learn-
ing is directly applied to the generation of topic-word distribution.
• We propose a novel data augmentation strategy for topic distribution based on the reconstruction ability of cycle adversarial training. To the best knowledge, we are the first to apply data augmentation to construct positive samples of topic distribution.
• We conduct extensive experiments and experimental results show that NTM-ACL outperforms several competitive baselines on four benchmark datasets.

heading: 2 Related Work
text: Our work is mainly related to two lines of research, including neural topic models and contrastive learning.

heading: 2.1 Neural Topic Model
text: Inspired by VAE, Miao et al. (2016) proposed Neural Variational Document Model (NVDM) for text modeling, employing Gaussian as the prior distribution of latent topics. Following that, (Srivastava and Sutton, 2017; Card et al., 2018) proposed to approximate Dirichlet distribution with a logistic normal prior distribution. To break the limitation of VAE, Wang et al. (2019) proposed an Adversarial Topic Model (ATM), which consists of a generator and a discriminator. The generator maps the topic distribution randomly sampled from the Dirichlet prior distribution to the word distribution, and the discriminator judges whether the word distribution comes from real documents or is generated by the generator. The two modules are trained adversarially against each other. In order to realize topic mining and document topic inference simultaneously,
Bidirectional Adversarial Topic (BAT) (Wang et al., 2020) constructs two-way adversarial training on the basis of ATM. Hu et al. (2020) propose Topic Modeling with Cycle-consistent Adversarial Training (ToMCAT) to realize the transformation between a topic distribution and word distribution, inspired by Cycle-GAN (Zhu et al., 2017).

heading: 2.2 Contrastive Learning
text: Contrastive learning, as a self-supervised learning method, improves the transforming ability of models without large-scale labeled data and becomes a popular technique in computer vision domain (Chen et al., 2020a; He et al., 2020; Chen et al., 2020b; Grill et al., 2020; Zhao et al., 2021). Chen et al. (2020a) proposed SimCLR, applied image transformations to generate two positive samples for each image randomly, and used the normalized temperature-scaled cross-entropy loss (NT-Xent) as the training loss to make positive pair close in the representation space.
With the success of contrastive learning in computer vision tasks, recent studies attempt to extend it to other domains. Khosla et al. (2020) extended the self-supervised approach to the fully-supervised setting, allowing models to effectively leverage label information. Jeong and Shin (2021) proposed ContraD to incorporate a contrastive learning scheme into GAN. In natural language processing, contrastive learning is widely applied to various tasks, such as sentence embedding, text classification, information extraction, and stance detection (Gao et al., 2021; Yan et al., 2021; Zhang et al., 2022; Wu et al., 2022; Chuang et al., 2022; Liang et al., 2022). In neural topic modeling, contrastive learning has been used to improve the VAE-based
neural topic model by adding a contrastive objective to the training loss and taking a more principled approach to creating positive and negative samples (Nguyen and Luu, 2021).

heading: 3 Method
text: The overall architecture of the proposed NTMACL is shown in Figure 2, which consists of three parts: 1) Cycle Adversarial Training based Neural Topic Model, which includes the generator, the encoder, and discriminators to transform topic distribution and word distribution bidirectionally; 2) Topic-Augmented Contrastive Learning, which includes the Self-supervised contrastive loss and the Discriminative contrastive loss to enhance the generator without affecting the adversarial training; 3) Reconstruct Min-Term Replacement, which is based on reconstruction ability of cycle generative adversarial network to create positive samples of topic distributions.

heading: 3.1 Problem Setting
text: We denote corpus as D, which consists of M documents {xi}Mi=1. Given the document xi ∈ RV where V is the vocabulary size, the first purpose of topic modeling is topic inference, inferring the corresponding topic distribution θi ∈ RK where K is the number of topics.
To formalize topic modeling, we use X to stand for word distribution set where the document is represented in normalized Term Frequency Inverse Document Frequency (TF-IDF), use Θ to stand for topic distribution set where topic distribution is sampled from a Dirichlet distribution with parameter α ∈ RK .
During the training process, we need to learn two
mapping functions, generator G and encoder E. G transforms samples from Θ into X while E is the reverse function of G. After G is well-trained, the indicator vector of each topic is input to get the topic-word distribution. This is another purpose of topic modeling, referred as topic discovery. The one-hot vector Ik ∈ RK denotes the indicator vector of the k-th topic, where the value at the k-th index is 1.

heading: 3.2 Cycle Adversarial Training
text: Following (Hu et al., 2020), NTM-ACL consists of two mapping functions, generator G: Θ→ X , encoder E: X → Θ and their related discriminators, Dx and Dθ. They are all implemented in the structure of a three-layer multi-layer perceptron (MLP), with a H-dim hidden layer using LeakyReLU as an active function and batch normalization, followed by an output layer using softmax. The cycle adversarial training objective is composed of adversarial loss and cycle consistency loss. We apply Wasserstein GAN (WGAN) (Arjovsky et al., 2017) adversarial losses to train G and corresponding discriminator Dx:
Ladv (G,Dx) =Ex∼pdata (x) [Dx(x)]− Eθ∼pdata (θ) [Dx(G(θ))] , (1)
in which G tries to generate word distributions similar to samples in X , while Dx aims to distinguish generated samples and real samples. G aims to minimize this objective against an adversary Dx that tries to maximize it.
To further constrain the relationship between origin distribution and target distribution, we additionally use cycle-consistency losses, encouraging G and E to reconstruct the origin distribution. Cycleconsistency losses are implemented as follows:
−−→Lcyc (G,E) = Eθ∼p(θ) [∥E(G(θ))− θ∥1] , ←−−Lcyc (G,E) = Ex∼p(x) [∥G(E(x))− x∥1] , (2)
where ∥·∥1 denotes L1 norm. Combining adversarial loss and cycle-consistency loss, the objective of cycle adversarial training is:
LCyc-adv = Ladv (G,Dx) + Ladv (E,Dθ)+ λ1 −−→Lcyc (G,E) + λ2 ←−−Lcyc (G,E), (3)
where λ1 and λ2 control the importance of the losses respectively.

heading: 3.3 Data Augmentation
text: In this subsection, we describe how to apply data augmentation to construct positive samples of topic distribution, which takes advantage of the structural features of cycle adversarial training.
Given distribution θi = {θi1, θi2, · · · , θiK}, the reconstructed distribution θ̂i created by the cycle of G and E is similar to the original distribution: θi → G(θi) → E(G(θi))=θ̂i ≈ θi. Meanwhile, we hypothesize that items with the maximum value in θi indicate the salient topic information, which has a significant effect on generating the word distribution x̃i through G. On the contrary, items with the minimum values in θi have limited effects. After making slight modifications to them, G can still generate a word distribution similar to x̃i.
Based on the above assumptions, we propose a data augmentation strategy for topic distribution named Reconstruct Min-Term Replacement (RMR). For the reconstructed topic distribution θ̂i = {θ̂i1, θ̂i2, · · · , θ̂iK}, we select the minimum p items from it. The indices of these items in θ̂i are denoted as {a1, a2, · · · , ap}. We replace the value at the corresponding index in θi:
θiaj = θ̂iaj (1 ≤ j ≤ p), (4)
For the topic distribution θi, we denote its dataaugmented distribution as θ′i. Correspondingly, the topic distribution set Θ after data augmentation is denoted as Θ′.

heading: 3.4 Topic-Augmented Contrastive Learning
text: In this subsection, we will introduce TopicAugmented Contrastive Learning, which enhances G while keeping the balance of generation and discrimination. This part mainly consists of two training objectives, Self-supervised contrastive loss, and Discriminative contrastive loss.
Self-supervised contrastive loss We follow the setting in SimCLR(Chen et al., 2020a), use Normalized Temperature-Scaled Cross-Entropy Loss (NT-Xent Loss) to calculate the Self-supervised contrastive loss LSelfCon. LSelfCon helps improving the mapping ability of G, capturing similar topic information to generate better topic-word distribution. Self-supervised contrastive loss pulls word distributions of positive topic distribution pairs together while pushing away distance between the word distributions corresponding to the negative sample pairs, which is shown in the upper right part
in Figure 2. Given representation ri, its positive sample is denoted as r+i , and the set of its negative samples is recorded as r−, the NT-Xent Loss between ri, r+i and r − is:
l(ri, r + i , r −)
= −log exp(ri · r + i /τ)
exp(ri · r+i /τ) + ∑|r−| j=1 exp(ri · r−j /τ) ,
(5) where τ is a temperature hyperparameter.
Assuming that topic set Θ of the current training batch contains N samples, we get Θ′ after data augmentation and the number of training samples is expanded to 2N . Two training sets are transformed to X̃ and X̃ ′ respectively. For word distribution x̃i in X̃ , we can find its positive sample x̃′i in X̃
′. The remaining 2N − 2 word distributions form the set of negative samples, denoted as X−i :
X−i = [X̃ \ x̃i; X̃ ′ \ x̃′i] (6)
Based on the above description, we define LSelfCon as follow:
LSelfCon
= 1
2N
N∑
i=1
[ l ( x̃i, x̃ ′ i, X − i ) + l ( x̃′i, x̃i, X − i )] ,
(7)
Discriminative contrastive loss The Selfsupervised contrastive loss LSelfCon can make the generator better perceive the similarity between two topics, then generate topic-word distributions that are more in line with the corresponding topics. However, only improving the mapping ability leads to an imbalance between generation and discrimination, which undermines the performance of cycle adversarial training. Therefore, we additionally design a discriminative contrastive loss LDisCon, leveraging category information of real samples and generated samples to keep the balance of generation and discrimination.
It is obvious that samples in X belong to the real category, while samples in X̃ and X̃ ′ belong to the generated category. For any xi in X , we denote Ui = [X \ xi; X̃; X̃ ′]. The main purpose of discriminative contrastive loss is not to focus on the similarity between the positive sample pair but make samples of the same category closer. We define the discriminative contrastive loss between
xi and Ui as: lDis(xi, Ui)
= − 1|X \ xi| ∑
x+∈X\xi log exp(xi · x+/τ)∑|Ui| j=1 exp(xi · xj/τ) ,
(8) where x+ stands for samples of the same category as xi.
For the whole batch, we define discriminative contrastive loss LDisCon as:
LDisCon = 1
N
N∑
i=1
lSup(xi, Ui) (9)
Overall Training Objective Summing up LCyc-adv, LSelfCon and LDisCon, the overall training objective of our model is:
L = LCyc-adv + λ3LSelfCon + λ4LDisCon, (10)
where λ3 and λ4 control the relative significance of Self-supervised contrastive loss and Discriminative contrastive loss respectively. At each training iteration, the parameters of G and E are updated once after parameters of Dθ and Dx have been updated 5 times.

heading: 4 Experiments

heading: 4.1 Datasets
text: We conduct experiments on four datasets: NYTimes1(NYT), Grolier2(GRL), DBPedia3(DBP) and 20Newsgroups4(20NG). We follow the same processing as (Wang et al., 2019). The statistics of the processed datasets are shown in Table 1.

heading: 4.2 Baselines
text: We compare NTM-ACL with the following baselines:
1http://archive.ics.uci.edu/ml/datasets/Bag+of+Words 2https://cs.nyu.edu/~roweis/data 3http://wikidata.dbpedia.org/develop/datasets 4http://qwone.com/~jason/20Newsgroups
• LDA (Blei et al., 2003), a probabilistic graphical model, which is one of the most popular conventional models, we used the implementation of GibbsLDA++5.
• NVDM (Miao et al., 2016), a VAE-based neural topic model that employs Gaussian prior for topic distributions.
• ProdLDA (Srivastava and Sutton, 2017), a VAE-based neural topic model that employs logistic normal prior to approximate Dirichlet prior.
• Scholar (Card et al., 2018), a VAE-based neural topic model that integrates metadata on the basis of ProdLDA.
• ATM (Wang et al., 2019), the first GAN-based nerual topic model.
• BAT (Wang et al., 2020), an adversarially trained bidirectional neural topic model.
• ToMCAT (Hu et al., 2020), an adversarial neural topic model with cycle-consistent objective.
• CLNTM (Nguyen and Luu, 2021), the first attempt to combine contrastive learning with a VAE-based topic model.
5http://gibbslda.sourceforge.net/

heading: 4.3 Implementation Details and Evaluation
text: We set the Dirichlet parameter α to 1K . The dimension H of the hidden layer is set to 100. The number of replacement items p changes dynamically according to the number of topics K. To be specific, set p = ⌊ K 4 ⌋ . For the training objective, we set λ1, λ2, λ3, and λ4 to be 2, 0.2, 1e-3, and 1e-3 respectively, aligning the magnitudes of different losses. During training, we set the batch size to 256 for NYTimes and Grolier, 1,024 for DBPedia, and 64 for 20Newsgroups. The training epoch is set to 150. We use Adam optimizer to update the model parameters, whose learning rate is 1e-4 and the momentum term is 0.5.
Following the previous work (Wang et al., 2020), we evaluate the performance of NTM-ACL and baselines using topic coherence measures highly correlated with human subjective judgments. For each topic, we select the top 10 topic words based on probability to represent the topic. C_A (Aletras and Stevenson, 2013), C_P (Röder et al., 2015), and NPMI (Aletras and Stevenson, 2013) are three topic coherence measures we use to evaluate models. We apply the Palmetto 6 tool to calculate coherence scores. We refer readers to (Röder et al., 2015) for more details of topic coherence measures.

heading: 4.4 Experiment Results
text: To make a robust comparison of NTM-ACL with baselines, we set topic numbers as 20, 30, 50, 75, and 100 on each dataset. Then we calculate the
6https://github.com/AKSW/Palmetto
average topic coherence score of 5 settings. The experimental results are presented in Table 2.
Compared with baselines of diverse structures, NTM-ACL performs better on most datasets and topic coherence measures, illustrating the effectiveness of our proposed approach. VAE-based neural topic models perform poorly due to the assumption of prior. Compared with GAN-based neural topic models, especially ToMCAT, which is also based on cycle adversarial training, NTM-ACL achieves State-of-Art results on all datasets, demonstrating the effectiveness of Topic-Augmented Contrastive Learning. CLNTM, as the first method to incorporate contrastive learning with topic modeling, performs worse compared with NTM-ACL except for the 20Newsgroups dataset in terms of C_A and C_P score. This result illustrates that combining contrastive learning with cycle adversarial training is more effective to improve the performance of topic discovery by eliminating the gap between model training and evaluation. To calculate coherence scores, Palmetto uses Wikipedia as a reference corpus, while CLNTM uses the training corpus itself as a reference, leading to the result reporting difference between ours and (Nguyen and Luu, 2021).
Based on the C_P score, we select 4 group topics with the best coherence score from 50 topic results of NYTimes and Grolier respectively. Every topic is represented in the form of top-5 topic words. As shown in Table 3, the best topics of the NYTimes are related to sports, politics, and music news, while the topics of Grolier reflect science
and culture.

heading: 5 Analysis and Discussion

heading: 5.1 Ablation Study
text: We conduct an ablation study on the relative contributions of different training objectives to topic modeling performance. We compare our full model with the following ablated variants: 1) Self-supervised only removes LDisCon in contrastive learning objective. 2) Discriminative only removes LSelfCon in contrastive learning objective. 3) w/o Adversarial Loss removes adversarial loss for word distribution and only relies on contrastive learning to distinguish samples. 4) w/o Cycle-Consistency Loss removes Cycleconsistency losses. We perform experiments on the Grolier dataset. The results are shown in Table 4.
From Table 4, we can obtain the following observations: 1) The removal of Adversarial Loss and Cycle-Consistency Loss both lead to performance drops, indicating that reserving the full objective of cycle adversarial training is a necessary condition for the proposed method. 2) Self-supervised only creates an imbalance between generation and discrimination, causing damage to model performance. 3) Although Discriminative only achieves a higher C_P score, the overall performance decreases compared to NTM-ACL, indicating the effectiveness of Self-supervised contrastive loss to improve topic-word generation.

heading: 5.2 Different Data Augmentation Strategies
text: To fully exploit the effectiveness of the proposed Reconstruct Min-Term Replacement strategy, we design two simple data augmentation strategies for comparison: 1) Noise Added (NA), topic distribution θi is added with the noise distribution which is of the same dimension, randomly sampled from a Gaussian distribution with expectation 0 and variance 0.01. 2) Zero Masked (ZM), when getting the indices {a1, a2, · · · , ap}, the value at the corresponding index is set to 0. We apply different data
augmentation strategies and keep other experimental settings unchanged. The results are shown in Table 5.
It can be observed that most coherence scores increase compared to GAN-based neural topic models, indicating the robustness of our contrastive learning approach. On the other hand, the results of NTM-ACL are the highest among the three strategies, which is proved to be a more suitable strategy for topic distribution.

heading: 5.3 Effect of Replacement Number
text: The number of replacement items p is one of the important hyperparameters for the RMR. For different K, the number of topics, it is inappropriate if p is set to a fixed value. In this subsection, we compare the dynamic setting to fixed numbers (1, 5, 15) on four datasets, using C_P coherence measure. The results are shown in Figure 3.
It can be observed that the data augmentation strategy with dynamic replacement numbers achieves the best performance. When the number p is too small, the difference between the positive pair is too slight. When the number p is too large, the similarity between positive samples cannot provide sufficient information for Self-supervised contrastive loss.

heading: 5.4 Training Strategy
text: In this subsection, we explore different training strategies, making contrastive learning and cycle adversarial training work respectively in different stages of 150 epochs. We abbreviate contrastive
learning as CL and cycle adversarial training as CA. We perform experiments on the Grolier dataset with the following strategies: 1) Disentangle, separating CL from CA. The first 50 epochs use CL/CA to update model parameters, and the next 100 epochs only employ CA/CL. Also, we design an alternation strategy in that CL is used to update the parameters after 5 epochs of CA only. 2) Warm up, using CL to warm up in the first 50 epochs, the next 100 epochs adopt the original strategy or are divided into two stages equally, i.e. CL+CA→CA. The results are shown in Table 6.
From the results, we can observe that NTMACL achieves the best performance. The result of CL→CA is second only to the best result. Using CL after CA undermines the stable symmetric structure, instead of further improving mapping ability, which should be avoided in future studies.

heading: 6 Conclusion
text: In this paper, we have proposed NTM-ACL, a novel topic modeling framework based on cycle adversarial training and contrastive learning. Selfsupervised contrastive loss improves the generation of topic-word distribution that is used for the evaluation of topic modeling, while Discriminative con-
trastive loss keeps the balance of generation and discrimination. Moreover, a novel data augmentation strategy is designed to create positive samples of topic distributions based on the reconstruction ability of cycle adversarial training. The experimental results show that the proposed method outperforms competitive baselines of different structures.
Limitations
In this section, we describe the limitation of our proposed method in terms of data augmentation and the way to combine contrastive learning with cycle adversarial training. First, our data augmentation strategy relies on the reconstruction ability of cycle adversarial training. We believe that more data augmentation strategies for topic distribution will be studied. Second, with the symmetrical structure of cycle adversarial training, it is worth exploring how to optimize the encoder E and generator G through contrastive learning simultaneously. We can extend the proposed framework to a conjugated structure in future work. Moreover, although it has been explored that contrastive learning and cycle adversarial training working synchronously performs better, we believe that more sophisticated training strategies will be designed to further improve the performance of topic modeling.

heading: Acknowledgement
text: We would like to thank the anonymous reviewers for their valuable comments and we thank Huawei for supporting this project. This work is funded by the National Natural Science Foundation of China (62176053). This work is supported by the Big Data Computing Center of Southeast University.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
We discuss the limitations of our work after the conclusion section.
7 A2. Did you discuss any potential risks of your work? As a research domain with a lot of practice, the topic model does not show obvious potential risks. So we don’t discuss this aspect specifically.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4.
3 B1. Did you cite the creators of artifacts you used? Section 4.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section 4.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 4.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Previous research extensively validated the datasets we use to ensure data security.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Section 4.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.
C 7 Did you run computational experiments? Our model does not use pre-trained language models and requires few computing resources.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? No response.
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No response.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Our work does not involve human annotators.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We can extend the proposed framework to a conjugated structure in future work. Moreover, although it has been explored that contrastive learning and cycle adversarial training working synchronously performs better, we believe that more sophisticated training strategies will be designed to further improve the performance of topic modeling