Diversity-Aware Coherence Loss for Improving Neural Topic Models
Year: 2023

Abstract:
The standard approach for neural topic modeling uses a variational autoencoder (VAE) framework that jointly minimizes the KL divergence between the estimated posterior and prior, in addition to the reconstruction loss. Since neural topic models are trained by recreating individual input documents, they do not explicitly capture the coherence between topic words on the corpus level. In this work, we propose a novel diversity-aware coherence loss that encourages the model to learn corpus-level coherence scores while maintaining a high diversity between topics. Experimental results on multiple datasets show that our method significantly improves the performance of neural topic models without requiring any pretraining or additional parameters.

heading: 1 Introduction
text: The main goal of topic modeling is to discover latent topics that best explain the observed documents in the corpus. The topics, conceptualized as a multidimensional distribution over the vocabulary, are useful for many downstream applications, including summarization (Wang et al., 2020; Xiao et al., 2022), text generation (Wang et al., 2019; Nevezhin et al., 2020), dialogue modeling (Xu et al., 2021; Zhu et al., 2021), as well as analyzing the data used for pretraining large language models (Chowdhery et al., 2022). When presented to humans, they are often represented as lists of the most probable words to assist the users in exploring and understanding the underlying themes in a large collection of documents. While the extrinsic quality of topics can be quantified by the performance of their downstream tasks, the intrinsic interpretability of topics appears to be strongly correlated with two important factors, namely coherence and diversity (Dieng et al., 2020).
The topic coherence measures to what extent the words within a topic are related to each other in a
meaningful way. Although human studies provide a direct method for evaluation, they can be costly, especially when a large number of models are waiting to be assessed. Therefore, various automatic metrics have been developed to measure topic coherence (Newman et al., 2010; Mimno et al., 2011; Xing et al., 2019; Terragni et al., 2021). For instance, the well-established Normalized Pointwise Mutual Information (NPMI) metric (Lau et al., 2014), based on word co-occurrence within a fixed window, has been found to have a strong correlation with human judgment (Röder et al., 2015). On the other hand, topic diversity measures to what extent the topics are able to capture different aspects of the corpus based on the uniqueness of the topic words (Nan et al., 2019). Importantly, studies have shown that optimizing for coherence can come at the expense of diversity (Burkhardt and Kramer, 2019). Even without accounting for topic diversity, directly optimizing for topic coherence by itself is a non-trivial task, due to the computational overhead and non-differentiability of the score matrix (Ding et al., 2018).
While traditional topic modeling algorithms are in the form of statistical models such as the Latent Dirichlet Allocation (LDA) (Blei et al., 2003), advancements in variational inference methods (Kingma and Welling, 2014; Rezende et al., 2014) have led to the rapid development of neural topic model (NTM) architectures (Miao et al., 2016, 2017; Srivastava and Sutton, 2017). More recently, follow-up works have focused on the integration of additional knowledge to improve the coherence of NTMs. Their attempts include the incorporation of external embeddings (Ding et al., 2018; Card et al., 2018; Dieng et al., 2020; Bianchi et al., 2021a,b), knowledge distillation (Hoyle et al., 2020), and model pretraining (Zhang et al., 2022). However, as the model is designed to operate on a documentlevel input, one significant limitation of NTMs is their inability to explicitly capture the corpus-
1710
level coherence score, which assesses the extent to which words within specific topics tend to occur together in a comparable context within a given corpus. For example, semantically irrelevant words such as “politics” and “sports” might be contextually relevant in a given corpus (e.g., government funding for the national sports body). Recently, one closely related work addresses this gap by reinterpreting topic modeling as a coherence optimization task with diversity as a constraint (Lim and Lauw, 2022).
While traditional topic models tend to directly use corpus-level coherence signals, such as factorizing the document-term matrix (Steyvers and Griffiths, 2007), and topic segment labeling with random walks on co-occurrence graphs (Mihalcea and Radev, 2011; Joty et al., 2013), to the best of our knowledge, no existing work have explicitly integrated corpus-level coherence scores into the training of NTMs without sacrificing topic diversity. To address this gap, we propose a novel coherence-aware diversity loss, which is effective to improve both the coherence and diversity of NTMs by adding as an auxiliary loss during training. Experimental results show that this method can significantly improve baseline models without any pretraining or additional parameters1.

heading: 2 Background
text: Latent Dirichlet Allocation (LDA) (Blei et al., 2003) is a simple yet effective probabilistic generative model trained on a collection of documents. It is based on the assumption that each document w in the corpus is described by a random mixture of latent topics z sampled from a distribution parameterized by θ, where the topics β are represented as a multidimensional distribution over the vocabulary V . The formal algorithm describing the generative process is presented in Appendix A. Under this assumption, the marginal likelihood of the document p(w|α, β) is described as: ∫
θ
( |V |∏
i
K∑
zi
p(wi|zi, β)p(zi|θ) ) p(θ|α)dθ (1)
However, since the posterior distribution p(zi|θ) is intractable for exact inference, a wide variety of approximate inference algorithms have been used for LDA (e.g., Hoffman et al. (2010)).
1The implementation of our work is available at: https://github.com/raymondzmc/Topic-Model-DiversityAware-Coherence-Loss
A common strategy to approximate such posterior is employing the variational auto-encoder (VAE) (Kingma and Welling, 2014). In particular, NTMs use an encoder network to compress the document representation into a continuous latent distribution and pass it to a generative decoder to reconstruct the bag-of-words (BoW) representation of the documents. The model is trained to minimize the evidence lower bound (ELBO) of the marginal log-likelihood described by the LDA generative process:
LELBO =−DKL[q(θ, z|w)||p(θ, z|α)] + Eq(θ,z|w)[log p(w|z, θ, α, β)] (2)
In Equation 2, the first term attempts to match the variational posterior over latent variables to the prior, and the second term ensures that the variational posterior favors values of the latent variables that are good at explaining the data (i.e., reconstruction loss). While standard Gaussian prior has typically been used in VAEs, ProdLDA (Srivastava and Sutton, 2017) showed that using a Laplace approximation of the Dirichlet prior achieved superior performance. To further improve topic coherence, CombinedTM (Bianchi et al., 2021a) concatenated the BoW input with contextualized SBERT embeddings (Reimers and Gurevych, 2019), while ZeroshotTM (Bianchi et al., 2021b) used only contextualized embeddings as input. These are the three baselines included in our experiments.

heading: 3 Proposed Methodology
text: Despite the recent advancements, one significant limitation of the NTM is that since the model is trained on document-level input, it does not have direct access to corpus-level coherence information (i.e., word co-occurrence). Specifically, the topicword distribution β is optimized on the documentlevel reconstruction loss, which may not be an accurate estimate of the true corpus distribution due to the inherent stochasticity of gradient-descent algorithms. We address this problem by explicitly integrating a corpus-level coherence metric into the training process of NTMs using an auxiliary loss.

heading: 3.1 Optimizing Corpus Coherence
text: To improve the topic-word distribution β, we maximize the corpus-level coherence through the wellestablished NPMI metric2 (Bouma, 2009; Lau et al.,
2Detailed definition of NPMI is presented in Appendix B.
2014). After computing the pairwise NPMI matrix N ∈ R|V |×|V | on the corpus, we use the negative β-weighted NPMI scores of the top-n words within each topic as the weight for the coherence penalty of β, where n is a hyperparameter that equals to the number of topic words to use. Specifically, we apply a mask Mc to keep the top-n words of each topic and apply the row-wise softmax operation σ to ensure the value of the penalty is always positive. We define the coherence weight WC in Equation 3.
WC = 1− normalize(σ(β ⊙Mc)N) (3)
Intuitively, each value in σ(β ⊙ Mk)N represents the β-weighted average NPMI score with other words in the topic. Then we use row-wise normalization to scale the values, so WC ∈ [0, 1].

heading: 3.2 Improving Topic Diversity
text: One problem with the coherence weight WC is that it does not consider the diversity across topics. To account for this, we propose an additional method to simultaneously improve topic diversity by encouraging words unused by other topics to have higher probabilities. To achieve this, we bin the words within each topic into two groups, where the words in the first group consist of those that already have a high probability in other topics (i.e., appear within top-n words), while the second group does not. The intuition is that we want to penalize the words in the first group more than the words in the second group. In practice, we use a mask Md ∈ RK×V for selecting β logits in the first group, where hyperparameter λd ∈ [0.5, 1] is a balancing constant between the two groups and n is the number of topic words to use. We then compute the diversity-aware coherence weight WD as the λd-weighted sum of WC :
WD = λdMd⊙WC +(1−λd)(¬Md)⊙WC (4)
From Equation 4, we see that when λd = 0.5, there are no constraints on diversity since the two groups are penalized equally (2WD = WC).

heading: 3.3 Auxiliary Loss
text: From the two definitions of coherence weight (WC ,WD), we propose an auxiliary loss that can be directly combined with the ELBO loss (Equation 2) when training the NTM. Since β are unnormalized logits containing negative values, we
apply the softmax operation σ(β) to avoid unbound optimization.
LAUX = 1
2 [σ(β)]2 ⊙WD (5)
In Equation 5, the topic probabilities are penalized by their negative weighted coherence score with the top-n words. The square operation ensures that words with very high probability are penalized to avoid the global minima, we justify this decision based on its partial derivatives in the next subsection.
The final objective function is the multitask loss consisting of the ELBO and our defined auxiliary loss:
L = LELBO + λaLAUX (6)
During training, we employ a linear warm-up schedule to increase λa gradually, so the model can learn to reconstruct the BoW representation based on the topic distribution α before optimizing for coherence and diversity.

heading: 3.4 Derivatives
text: We justify our auxiliary loss defined in Equation 5 using the derivatives w.r.t. the β parameters. For simplicity, we define pk,i = σ(βk)i as the softmax probability for word i in topic k. Since we detach the gradients when computing W , it can be treated as a constant w in the derivatives.
∂LAUX ∂βk,i = w · pk,i · pk,i(1− pk,i) +
w · ∑
j ̸=i pk,j(−pk,jpk,i)
(7)
In Equation 7, the partial derivatives w.r.t. βk,i can be broken down into two terms. In the first term, the softmax derivative pk,i(1− pk,i) is zero when pk,i is either 0 or 1 (really small or really large). The additional pk,i (from the square operation) penalizes over-confident logits and leads to better topics. Similarly for the second term, since∑
i pk,i = 1, ∑ j ̸=i ( pk,jpk,i ) is zero (global minima) when one logit dominates the others. Therefore, the additional pk,j has the same penalizing effect on the over-confident logits.

heading: 4 Experiments
text: In this section, we describe the experimental settings and present the quantitative results to assess the benefits of our proposed loss.

heading: 4.1 Datasets and Evaluation Metrics
text: To test the generality of our approach, we train and evaluate our models on three publicly available datasets: 20NewsGroups, Wiki20K (Bianchi et al., 2021b), and GoogleNews (Qiang et al., 2022). We provide the statistics of the three datasets in Table 23.
We use automatic evaluation metrics to measure the topic coherence and diversity of the models. For coherence, we use the NPMI and Word Embedding (WE) (Fang et al., 2016) metrics, which measure the pairwise NPMI score and word embedding similarity, respectively, between the top-10 words of each topic. For diversity, we use Topic Uniqueness (TU) (Dieng et al., 2020), which measures the proportion of unique topic words, and Inversed Rank-Biased Overlap (I-RBO) (Terragni et al., 2021; Bianchi et al., 2021a), measuring the rank-aware difference between all combinations of topic pairs.

heading: 4.2 Baselines
text: We plug our proposed auxiliary loss to three baseline NTMs’ training process to demonstrate the benefits of our approach across different settings. Specifically, the three models are (1) ProdLDA
3Detailed description of the three datasets is provided in Appendix C.
(Srivastava and Sutton, 2017), (2) CombinedTM (Bianchi et al., 2021a), and (3) ZeroshotTM (Bianchi et al., 2021b). For comparison, we also include the results of the standard LDA algorithm (Blei et al., 2003).

heading: 4.3 Hyperparemeter Settings
text: We follow the training settings reported by Bianchi et al. (2021a), with 100 epochs and a batch size of 100. The models are optimized using the ADAM optimizer (Kingma and Ba, 2015) with the momentum set to 0.99 and a fixed learning rate of 0.002. We do not modify the architecture of the models, where the inference network is composed of a single hidden layer and 100 dimensions of softplus activation units (Zheng et al., 2015). The priors over the topic and document distributions are learnable parameters. A 20% Dropout (Srivastava et al., 2014) is applied to the document representations. During our evaluation, we follow the same setup and used the top-10 words of each topic for the coherence and diversity metrics.
For the hyperparameters introduced in the diversity-aware coherence loss, both Mc and Md are computed using the top-20 words of each topic. The scaling factor λa is linearly increased for the first 50 epochs and kept constant for the last 50 epochs, we set λa to be 100 in order to balance the loss magnitude of LELBO and LAUX. The λd in the diversity loss is set by taking a mid-range value of 0.7 in the [0.5, 1] range. We do not perform any searches over our defined hyperparameters; we believe that additional experiments will yield better results (i.e., by using a validation set).

heading: 4.4 Results
text: Table 1 shows improvements across all settings. However, with the basic coherence loss (WC), the
significant coherence increase comes at the expense of topic diversity, where a slight decrease can be observed in the I-RBO and TU scores. In contrast, with the diversity-aware coherence loss (WD), we observe that the model improves in coherence while having a significantly higher diversity over the basic loss (WC). The further coherence improvements can be attributed to the regularization effects, where words with a high probability of belonging to another topic are less likely to be related to words in the current topic. Lastly, it is worth noting that due to the gradual increase in λa, our proposed loss has a negligible effect on the original document-topic distribution θ, and only modifies the word distribution within the established topics. We provide some sample model outputs in Appendix D.

heading: 4.5 Coherence and Diversity Trade-off
text: To study the effects of λd on the trade-off between coherence and diversity, we perform experiments with different values of λd with the ZeroshotTM baseline, which has the best overall performance. Note that when λd = 0.5, the objective is equivalent to the basic coherence loss. From results on the 20NewsGroups Dataset (Table 3), we see that coherence peaks at λd = 0.7 before the diversity penalty begins to dominate the loss. Further, while a higher value of λd leads to a lower coherency score, both coherency and diversity are still improved over the baselines for all values of λd, demonstrating the effectiveness of our method without the need for extensive hyperparameter tuning. We observe an identical trend in other datasets.

heading: 4.6 Comparison with Composite Activation
text: The recent work by Lim and Lauw (2022) proposed a model-free technique to refine topics based on the parameters of the trained model. Specifically, they solve an optimization problem (with the NPMI score as the objective) using a pool of candidates while setting the diversity score as a constraint.
Since their goal is similar to ours, we run further evaluations to compare the respective approaches. In particular, we experiment with ZeroshotTM on the 20NewsGroups dataset for K = 25, 50. For comparison, we use their Multi-Dimensional Knapsack Problem (MDKP) formulation, since it achieved the best overall performance. Regrettably, considering larger topic numbers was not possible due to the NP-hard runtime complexity of MDKP. From the results in Table 4, we see that while our methods have similar coherence scores, MDKP archives higher topic diversity due to its selectivity of less-redundant topics. However, when combining MDKP with our proposed loss (+ WD + MDKP), we achieve the highest overall performance across all metrics. This is expected since the pool of potential topic candidates is generated based on the trained model, and better-performing models lead to superior candidates.

heading: K = 25 NPMI WE I-RBO TU

heading: K = 50 NPMI WE I-RBO TU

heading: 5 Conclusion and Future Work
text: In this work, we present a novel diversity-aware coherence loss to simultaneously improve the coherence and diversity of neural topic models. In contrast to previous methods, our approach directly integrates corpus-level coherence scores into the training of Neural Topic Models. The extensive experiments show that our proposal significantly improves the performance across all settings without requiring any pretraining or additional parameters.
For future work, we plan to perform extensive user studies to examine the extent to which improvements in quantitative metrics affect human preference. Further, we would like to extend our approach to other quantitative metrics (e.g., semantic similarity), and perform extrinsic evaluation to study the effects of our approach when the topics are used for downstream tasks (e.g., summarization, dialogue modeling, text generation).
Limitations
We address several limitations with regard to our work. First, the publicly available datasets used in our experiments are limited to English. Documents in different languages (i.e., Chinese) might require different segmentation techniques and may contain unique characteristics in terms of vocabulary size, data sparsity, and ambiguity. Secondly, we only evaluate the quality of the topic models in terms of coherence and diversity. Future work should explore how our method impacts other characteristics, such as document coverage (i.e., how well documents match their assigned topics) and topic model comprehensiveness (i.e., how thoroughly the model covers the topics appearing in the corpus).
Ethics Statement
The datasets used in this work are publicly available and selected from recent literature. There could exist biased views in their content, and should be viewed with discretion.
Our proposed method can be applied to extract topics from a large collection of documents. Researchers wishing to apply our method should ensure that the input corpora are adequately collected and do not violate any copyright infringements.

heading: A LDA Generative Process
text: The formal generative process of a corpus under the LDA assumption can be described by the following algorithm.
Algorithm 1 Generative process of LDA for each document w do
Sample topic distribution θ ∼ Dirichlet(α) for each word wi do
Sample topic zi ∼ Multinomial(θ) Sample word wi ∼ Multinomial(βzi)
end for end for

heading: B Normalized Pointwise Mutual Information
text: Normalized Pointwise Mutual Information (NPMI) (Lau et al., 2014) measures how much more likely the most representative terms of a topic co-occur than if they were independent. The method for computing the NPMI score between word wi and wj is described in Equation 8, where P (wi, wj) is computed using a window size of 10. This metric ranges from −1 to 1.
NPMI(wi, wj) = log
P (wi,wj) P (wi)P (wj)
− logP (wi, wj) (8)
In practice, the pairwise NPMI matrix is computed by first counting the word co-occurrence of all words in the corpus and then calculating the pairwise score following Equation 8. In summary, the NPMI matrix can be computed in O(|W |+ |V |2) for a corpus of |W | words and vocab size |V |. Since the matrix is computed only once for each corpus prior to training, it does not increase the runtime complexity of training time.

heading: C Datasets
text: This section provides details regarding the datasets we used. The 20NewsGroup4 dataset is a collection of email documents partitioned evenly across 20 categories (e.g., electronics, space), we use the same filtered subset provided by Bianchi et al. (2021a). The Wiki20K dataset5 contains randomly sampled subsets from the English Wikipedia abstracts from DBpedia6. GoogleNews7 (Qiang et al., 2022) is downloaded from the Google news site by crawling the titles and snippets. We do not perform any additional pre-processing and directly use the data provided by the sources to create contextualized and BoW representation.

heading: D Sample Output
text: Table 5 provides a qualitative comparison of the topics generated by our proposed method using ZeroshotTM on the 20NewsGroups dataset.
E Implementation Details
We base our implementation using the code provided by the authors of ZeroshotTM and CombinedTM (Bianchi et al., 2021a,b). Their repository8 also provides the evaluation metrics used in our experiments. Our Python code base includes external open-source libraries including NumPy9, SciPy10, PyTorch11, SentenceTransformers12, Pandas13, Gensim14 and scikit-learn15.

heading: F Computing Details
text: All our experiments are run on Linux machines with single 1080Ti GPU (CUDA version 11.4). Each epoch with 100 batch size on the most computationally intensive setting (GoogleNews with K = 150) takes on average 3 seconds to run for the baselines models and 8 and 15 seconds, for WC and WD, respectively. Under this setting, a maximum VRAM usage of 800MB was recorded.
4http://qwone.com/~jason/20Newsgroups 5https://github.com/vinid/data 6https://wiki.dbpedia.org/downloads-2016-10 7https://github.com/qiang2100/STTM/tree/
master/dataset 8https://github.com/MilaNLProc/ contextualized-topic-models 9https://numpy.org/
10https://scipy.org/ 11https://pytorch.org/ 12https://www.sbert.net/ 13https://pandas.pydata.org/ 14https://radimrehurek.com/gensim/ 15https://scikit-learn.org/stable/
Z entry, rules, entries, email, build, info, file, char, program, section Z + WC entry, rules, entries, email, info, build, file, char, section, program Z + WD entry, rules, entries, email, build, info, file, char, program, section
Z army, turkey, muslim, jews, greek, jewish, genocide, professor, ottoman, greece Z + WC army, muslim, turkey, ottoman, jews, greek, genocide, jewish, greece, muslims Z + WD muslim, turkey, ottoman, genocide, army, jews, greek, jewish, greece, muslims
Z board, driver, video, cards, card, monitor, windows, drivers, screen, resolution Z + WC board, video, driver, cards, monitor, card, windows, drivers, screen, printer Z + WD video, board, driver, cards, monitor, card, drivers, printer, screen, windows
Z frequently, previously, suggested, announced, foundation, spent, contain, grant, consistent, authors Z + WC basically, previously, frequently, generally, suggested, primary, authors, appropriate, kinds, greater Z + WD essentially, basically, kinds, consistent, frequently, authors, previously, primary, equivalent, suggested
Z sale, condition, offer, asking, offers, shipping, items, price, email, sell Z + WC sale, condition, offer, shipping, asking, items, offers, sell, email, price Z + WD sale, condition, shipping, offer, asking, items, offers, sell, price, excellent
Z application, window, xterm, motif, font, manager, widget, root, event, server Z + WC xterm, application, window, motif, font, widget, manager, x11r5, server, event Z + WD xterm, motif, font, application, window, widget, manager, x11r5, event, server
Z gun, amendment, constitution, firearms, right, militia, guns, weapon, bear, weapons Z + WC amendment, constitution, firearms, gun, militia, right, guns, weapon, bear, weapons Z + WD amendment, firearms, constitution, gun, militia, guns, right, weapon, bear, weapons
Z suggested, frequently, previously, authors, foundation, consistent, spent, join, et, announced Z + WC suggested, previously, frequently, greater, requirements, consistent, opportunity, authors, particularly, appropriate Z + WD spent, greater, association, appropriate, opportunity, requirements, posts, previously, success, training
Z objective, atheist, atheism, morality, exists, belief, does, exist, atheists, existence Z + WC objective, atheist, atheism, morality, exists, belief, atheists, does, exist, existence Z + WD atheist, objective, atheism, belief, morality, exists, atheists, existence, exist, does
Z think, president, people, Stephanopoulos, dont, jobs, just, know, mr, myers Z + WC think, president, Stephanopoulos, people, dont, jobs, just, know mr, myers Z + WD think, president, Stephanopoulos, people, dont, jobs, just, know, mr, myers
Z board, drive, ide, scsi, bus, isa, mhz, motherboard, internal, pin Z + WC board, drive, ide, scsi, motherboard, bus, isa, mhz, hd, controller Z + WD board, drive, ide, motherboard, scsi, mhz, bus, hd, isa, controller
Z jpeg, images, image, formats, gif, format, software, conversion, quality, color Z + WC jpeg, images, formats, image, gif, format, conversion, software, quality, color Z + WD jpeg, images, formats, gif, image, format, conversion, software, quality, color
Z msg, food, doctor, vitamin, doctors, medicine, diet, insurance, treatment, studies Z + WC msg, food, doctor, medicine, doctors, vitamin, diet, studies, treatment, insurance Z + WD msg, food, doctor, medicine, doctors, vitamin, diet, studies, patients, treatment
Z agencies, encryption, keys, secure, algorithm, chip, enforcement, nsa, clipper, secret Z + WC agencies, encryption, secure, keys, algorithm, nsa, enforcement, encrypted, escrow, chip Z + WD secure, encryption, keys, agencies, algorithm, escrow, encrypted, enforcement, nsa, clipper
Z windows, dos, nt, network, card, disk, pc, software, modem, operating Z + WC windows, dos, nt, card, network, disk, pc, modem, software, operating Z + WD windows, dos, nt, card, network, disk, pc, modem, software, operating
Z address, site, thanks, looking, newsgroup, appreciate, advance, mailing, obtain, domain Z + WC address, thanks, newsgroup, site, appreciate, advance, looking, mailing, thank, reply Z + WD address, appreciate, site, thanks, advance, newsgroup, looking, mailing, thank, obtain
Model Top-10 Topic Keywords
Z launch, nasa, shuttle, mission, satellite, energy, mass, moon, orbit, lunar Z + WC launch, shuttle, nasa, mission, moon, satellite, orbit, energy, mass, lunar Z + WD shuttle, launch, nasa, mission, orbit, moon, satellite, lunar, mass, energy
Z floor, door, said, people, azerbaijani, neighbors, apartment, like, saw, dont Z + WC floor, azerbaijani, door, said, people, apartment, neighbors, like, saw, dont Z + WD azerbaijani, floor, apartment, door, said, people, neighbors, saw, like, building
Z join, grant, foundation, suggested, previously, discussions, frequently, authors, positions, announced Z + WC discussions, topic, suggested, join, mailing, responses, robert, lists, summary, received Z + WD join, discussions, foundation, robert, mailing, lists, topic, grant, received, responses
Z pts, boston, van, pittsburgh, pp, san, vancouver, chicago, la, st Z + WC pts, boston, van, pittsburgh, pp, san, vancouver, chicago, buf, tor Z + WD pts, pittsburgh, van, boston, pp, chicago, buf, tor, san, det
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Ethics

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts?

heading: 4, Appendix C, E

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? In our paper, datasets and software tools used to reproduce our results are open-sourced and available to all developers. All our datasets are available for the general public, the license and terms for use are available using the links provide in Appendix C. For all open-sourced software packages, we include a detailed list of the websites for Python libraries and the baseline code base in Appendix E, the license and terms for use can be easily found on the websites. Since our models are stored in our private repository, and the license will be disclosed when the code base becomes public.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix C, Limitations
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? 4, and Appendix D

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix E
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: For future work, we plan to perform extensive user studies to examine the extent to which improvements in quantitative metrics affect human preference. Further, we would like to extend our approach to other quantitative metrics (e. g. , semantic similarity), and perform extrinsic evaluation to study the effects of our approach when the topics are used for downstream tasks (e. g. , summarization, dialogue modeling, text generation). Limitations
We address several limitations with regard to our work. First, the publicly available datasets used in our experiments are limited to English. Documents in different languages (i. e. , Chinese) might require different segmentation techniques and may contain unique characteristics in terms of vocabulary size, data sparsity, and ambiguity. Secondly, we only evaluate the quality of the topic models in terms of coherence and diversity. Future work should explore how our method impacts other characteristics, such as document coverage (i. e. , how well documents match their assigned topics) and topic model comprehensiveness (i. e. , how thoroughly the model covers the topics appearing in the corpus)