Revisiting Topic-Guided Language Models
Authors: Carolina Zheng
Keyon Vafa
David Blei
Venue: Accepted by TMLR ()

Abstract: A recent line of work in natural language processing has aimed to combine language models and topic models. These \textit{topic-guided language models} augment neural language models with topic models, unsupervised learning methods that can discover document-level patterns of word use. This paper compares the effectiveness of these methods in a standardized setting. We study four topic-guided language models and two baselines, evaluating the held-out predictive performance of each model on four corpora. Surprisingly, we find that \textit{none of these methods outperform a standard LSTM language model baseline}, and most fail to learn good topics. Further, we train a probe of the neural language model that shows that the baseline's hidden states already encode topic information. We make public all code used for this study.

Published in Transactions on Machine Learning Research (11/2023)
Revisiting Topic-Guided Language Models
Carolina Zheng
carozheng@Gss.couum hwwp://wsss.scoooob...... eduu
Deportment of Computer Science Columbia University
Keyon Vafa
edu
Department of Computer Science Columbia University
David M. Blei
david. wew.bbbleiooolumbba. edu
Deportment of Statistics
Department of Computer Science Columbia Universily
Reviewed on OpenReview: https: // openreview. net/ forum? id= LXBEuFfapA
Abstract
A recent line of work in natural language processing has aimed combine language models and topic models. These lopic-guided language models augment neural language models with topic models, unsupervised learning methods that discover document- patterns of word use. This paper compares the effectiveness of these methods in standardized setting. We study four topic-guided language models and two baselines, evaluating the held-out predictive performance each model on four corpora. Surprisingly, we find that none of these methods outperform a standard LSTM language model baseline, and most fail to learn good topics. Further, we train a probe of the neural language model that shows that the baseline's s hidden states already encode topic information. We make public all code used for this study.
1 Introduction
Recurrent neural networks (RNNs) and LSTMs have been an important class of models in the development of methods for many tasks in natural language processing, including machine translation, summarization, and speech recognition. One of the most successful applications of these models in language modeling, where they are effective modeling small text corpora. Even with the advent of transformer-based language models, RNNs and LSTMs outperform non-pretrained transformers various small datascts (Mclis et al., 2020).
While powerful, RNN- and LSTM-based models struggle to capture long-range dependencies in their context history (Bai et al., 2018; Sankar et al., 2019). Additionally, they are not designed to learn interpretable structure in a corpus of documents. To this end, multiple researchers have proposed adapting these models by incorporating topic models (Dieng et al., 2017; Lau et al., 2017; Rezace Ferraro, 2020; Guo et al., 2020). decouple local syntactic structure,
The motivation for combining language models and topie models which can be modeled by a language model, from document-level semantic concepts, which can be captured by a topic model (Khandelwal et al., 2018; O'Connor & Andreas, 2021). The topic model component is also designed to uncover latent structure in documents.
We refer to hese models as lopic-guided language models. Broadly, this body of research has reported good re- sults: topic-guided language models improve next-word predictive performance and learn interpretable topics.
In this work, we re-investigate this class of models by evaluating four representative topic-guided language model (TGLM) papers in unified setting. We train the models from Dieng et al. (2017); Lau et al. (2017);

---

Published in Transactions on Machine Learning Research (11/2023)
Rezace & Ferraro (2020); Guo et al. (2020) on three document-level corpora and evaluate their held-out perplexity. Unlike some prior work, during next-word prediction, we take care to condition the topic model component on only previous words, rather than the entire document. Moreover, we use a baseline language model that is conditioned on all proviously seen document words, rather than being restricted to the current sentence (Lau et al., 2017; Rezaee & Ferraro, 2020; Guo et al., 2020). Additionally, we choose baseline language models with comparable model sizes to ensure valid comparisons. Our finding: no predictive improvement of TGLMs over a standard LSTM-LM baseline (Zaremba et al., 2014).
In order to understand why topic-guided language models offer predictive improvement, we probe the LSTM-LM's hidden representations. A probe is a trained predictor used to mcasure the extent to which fitted "black-box" models, such as neural models, have learned specific linguistic features of the input (Hewitt & Liang, 2019). The probe reveals that the LSTM-LM already encodes topic information, rendering a formal topic model component redundant.
Additionally, topie-guided language models were developed to provide insight into text corpora by uncovering latent topics. This method of exploratory text analysis is commonly used in the social sciences and digital hunanities (Griffiths & Steyvers, 2004; Blei & Lafferty, 2007; Grimmer & Stewart, 2013; Mohr & Bogdanov, 2013). Here, we show that the topics learned by topic-guided language models are not better than a standard topic model and, for some of the models, qualitatively poor.
This paper contributes to a line of reproducibility studies in machine learning that aim to evaluate competing methods in a consistent and equitable manner. These studies have uncovered instances where results are not directly comparable,
reported numbers borrowed from prior works that used different experimental scttings (Maric et al., 2021; Hoyle et al., 2021). Furthermore, they identify cases where baselines either too weak improperly tuned (Dacrema et al., 2019; Nityasya et al., 2023). We observe analogous issues within the topic-guided language modeling literature. To support transparency and reproducibility, we make public all code used in this study.
Finally, we consider how these insights apply to other models. While prior work has incorporated topic model components into RNNs and LSTMs, the topic-guided language model framework agnostic to the class of neural language model used. We conclude by discussing how the results in this paper relevant to researchers considering incorporating topic models into more powerful neural language models, such transformers.
2 Study Design
Let X1:7 (II,.... FTT be a sequence of tokens collectively known as a document, where each indexes one of V words in a vocabulary (words outside the vocabulary are mapped to a special out-of-vocabulary token). Given a corpus of documents, the goal of language modeling is to learn a model p(xur) that approximates the probability of observing a document.
A document can be modeled antoregressively using the chain rule of probability,
p(xi.) II- p(are x<t),
(1)
where Xςi denotes all the words in document before t. A language model parameterizes the predictive distribution of the next word, Pa (x; xxx), with set of parameters μ.. Given set of documents indexed by Dtrain, we compute a paramcter estimate fa by maximizing the log likelihood objective,
Daratn Ete logPu((za Xd,<t),
with respect to μ. Language models are evaluated using perplexity on a held-out set of documents. With Drest as the index set of the test documents, perplexity is defined as
exp {-
El log Pp (F x.
Σ. E
2

---

Published in Transactions on Machine Learning Research (11/2023)
Perplexity is the inverse geometric average of the likelihood of observing each word in the set of test documents under the fitted model; lower perplexity indicates a better model fit.
3 Language Models and Topic Models
Here, we provide an overview of the two components of topic-guided language models: neural language models and topic models. The topie-guided language model literature has focused on models based on RNNs and LSTMs, which are the neural language models we focus on here.
RNN language model. A recurrent neural network (RNN) language model (Mikolov al., 2010) defines each conditional probability in Equation (1) as
h (1)-1, PRNN (Fz xxx) = softmax(W")) (Whh:-1),
(2)
(3)
where W E DRV and h, E D.D. The hidden state hŞ-1 sunmarizes the information in the preceding sequence, while the function f combines h with the word at time to produce new hidden state, h The function f is parameterized by recurrent neural network (RNN).
The parameter W and the RNN model parameters are trained by maximizing the log likelihood of training documents using backpropagation through time (BPTT) (Williams & Peng). (In practice, the backpropaga- tion of gradients is truncated after specified sequence length.) The model directly computes the predictive distribution of the next word, p(x; xx()).
The basclines use the widely used RNN architecture, the LSTM (Hochreiter & Schmidhuber, 1997), as the language model, which we call LSTM-LM (Zaremba et al., 2014). The LSTM architecture is described in Appendix A.
To make full use of the document context, it is natural to condition on all previous words of the document when computing p(#; xxx)- Even when the full document does not fit into memory, this can be done at extra computational cost by storing the previous word's hidden state (h,2 in Equation (2)) (Melis et al., 2017). This is our main baseline.
One can also define X2t to be only the previous words in the current sentence. In this scenario, the model will not condition on all prior words in the document. This the LSTM-LM baseline used in many TGLM papers (Lau et al., 2017; Guo al., 2020; Rezaee & Ferraro, 2020). We call this model the sentence-level LSTM-LM.
Topic model. Another way to model document is with bag-of-words model that represents documents word counts. One such model is probabilistic topic model, which assumes the observed words are conditionally independent given latent variable 0. In topic model, the probability of document is
p(xIr) = = I p(z, ())(A)d
(4)
where p(0) a prior distribution on 0 and p(a: 0) the likelihood of word x conditional on 0.
A widely used probabilistic topic model is Latent Dirichlet Allocation (LDA) (Blei al., 2003). LDA posits that corpus of text is comprised of K latent topics. Each document d contains distribution over topics, Og, and each topic & is associated with a distribution over words, Bg. These two terms combine to form the distribution of each word in a document.
The generative model for LDA is:
1. Draw K topics: 91,...,uu ~ Dirichlety (y).
2. For each document:
(a) Draw topic proportions,
0 Dirichletx
3

---

Published in Transactions on Machine Learning Research (11/2023)
(b) For each word 21,..,,T7
i. Draw topic indicator,
Zaz Categorical(O). ii. Draw word, (ategorical(S...)
Since each word is drawn conditionally independent of the precoding words in the document, LDA is not able to caplure word order or synlax. However, it can caplure document-level patterns since the topic for each word drawn from a -ocument-speccffc distribution. Practitioners typically rely on approximate posterior inference to estimate the LDA posterior. The most common methods are Gibbs sampling (Griffiths & Sleyvers, 2004) or variational inference (Blei et al., 2003).
After approximating the posterior distribution over topics from the training documents, the next-word posterior predictive distribution is
PLDA xxx) = p(1; ()p(O xcr)dO.
(5)
Given words x from document, one use approximate posterior inference to estimate the topic proportions posterior, p(0 f x2t), and then draw Monte Carlo samples of 0 to estimate the predictive distribution.
4 Topic- Language Model
We now discuss topic-guided language models (TGLMss, which are a class of language models that combine topic models and neural language models. TGLMs were initially proposed to combine the fluency of neural language models with the document modeling capabilities of topic models. Dieng et al. (2017) and Lau al. (2017), who propose two of the models that we study here, argue that long-range dependency in language is captured well by topic models. Subsequent TGLM papers build on Dieng et al. (2017) and Lau et al. (2017), but differ from these previous works in evaluation setting (Wang et al., 2018; Rezace & Ferraro, 2020; Guo et al., 2020).
Topic-guided language models can divided into two frameworks, differing whether they model the document's 's bag-of-words counts addition lo the typical next-word prediction objective. In this section, we discuss the two frameworks: topic-biased language model and a joint topic and language model. The graphical structure of these models shown in Figure 1.
4.1 Topic-Biased Language Models
A topic-biased language model defines the next-word probability to be the sum of two terms: linear transformation of the hidden state, in RNN, and the distribution of words according to document's topics, in topic model.
Each document follows the data generating mechanism below:
Draw topic vector, Dirichlet (.).
2. For each word ,y....,yy;
(a) h, RNN h,
(b) Draw l, (ernoull((((Uhh.)
(c) Draw Categorical 0).
(d) Draw I++1 & exp(WT) + (1 Bz.)
Here, a7.) denotes the sigmoid function. The model parameters are the parameters of the RNN, the weights
W E DxV and u E RDD, and the topics BI...,, Br x E R.Y. The latent, variable 0 determines the document's topic proportions.
Of the two additive terms a word's likelihood, the RNN term encourages fluency and syntax while the topic modeling term be understood as bias toward the document's 's global structure. Since topic models

---

Published in Transactions on Machine Learning Research (11/2023)
1
h
14
ht
91
9t
Ct
Ct++
I
241
.M
x
++++
TM
(a)
(b)
Figure 1: Graphical model representations of the two frameworks of topic-guided language models. (a) the topic-biased language model. (b) is the joint topic and language model. Circles denote random variables, while squares denote deterministic variables. Shading indicates that the variable is observed.
struggle with modeling very common words ("stop words") (Wallach al., 2009), a word's likelihood only includes the topic modeling term if it is not predicted to be stop word (e, 0). The realizations of the stop word indicators observed during training (&e if 14+1 belongs to list of stop words, and 0 otherwise). During prediction, the stop word indicators are treated as latent variables and are marginalized out. Hence, the biased language models learn to interpolate between a standard language model's predictions and topics.
TopicRNN. TopicRNN (Diong et al., 2017) approximates Step 2(d) by marginalizing 2; before normaliz- ing:
PTANN(0e41 hr 0) x exp(E (1-())
= e + +11 (1-6))
The topic matrix 9 E kxV contains the topic vectors B Px rows. Additionally, in Step 1, TopicRNN draws 0 from standard Gaussian, rather than Dirichlet distribution.
VRTM. VRTM (Rezace & Ferraro, 2020) (short for Variational Recurrent Topic Model) exactly computes Step 2(d) by marginalizing 2 after normalizing:
P7RTM(2141 h, 0) E|softmax + (x)
-k= 02 * softmax WTh, + (1 - lx)
This makes VRTM a mixture-of-RNNS (Yang et al., 2017), where the mixture proportions are determined by
Inference.
The model parameters for topic-biased language models are learned using variational inference (Wainwright et al., 2008; Blei et al., 2017). We provide high-level overview the method here.
The goal of variational inference is to approximate the posterior of the latent variable 0, p(O x1:)), with a learned distribution (0), called the variational distribution. To fit (0), variational inference minimizes
5

---

Published in Transactions on Machine Learning Research (11/2023)
the KL divergence between the two distributions. This is equivalent to maximizing lower bound of the marginal log likelihood, evidence lower bound (EIBO):
log ((1x) = log f Pa (x1T 0 (0)(0)
2 (0) [Lo (Xi:T 0)] (L(()())ll(()).
The ELBO contains two lerms: reconstruction loss, which is the expected log probability of the data under 20 (0), and the KL divergence between the variational distribution and the prior on 0. By maximizing the ELBO, we can simultancously learn the model parameters and the variational distribution parameters, represented by la and & respectively. order to share the learned variational parameters, the variational distribution defined to be function of the data, i.e. we learn g x1-T), where q parameterized by neural network.
In practice, the ELBO is maximized with respect to parameters and & using backpropagation. The expectation is estimated using samples from qs(0 X1.r) and the KT. can often be computed analytically (e.g., when both distributions are Gaussians) (Kingma & Welling, 2014).
Prediction. For both models, the next-word predictive distribution is
p(æ, xx1) Epro xx))lp(zz xx1, 0)].
(6)
Using learned variational distribution in place of the exact posterior, we approximate the expectation using its mean, ie., let Ô = (0 xxx)]. Then p(r: xxx) s p(z; x 0). For computation reasons, in practice, i is only updated in sliding window (i.e., every N words).
4.2 Joint Topic and Language Model
A joint lopic and language model learns the lopic model and language model simultaneously, essentially fitting two views of the same data. The two models share the document-level latent variable 0.
Each document during training has a pair representations, its bag-of-words X1T LL generated by the topic model and the language model, respectively. For each document, basic version of the data generating mechanism is:
TM and its word sequence
Draw topic vector, Dirichletx<
2. Draw the bag-of-words TM from topic model (Section 3):
(a) XTM (opicModellO))
For each word 2f-M. ..... Z/M:
(a) h, = RNN(.. h.-1).
(b) g: = a(h,, 0).
(c) Draw I x exxp(WTg)
Here, the latent variable 0 determines the document's topic proportions in the topic model. In the language model, the hiddon state h, is combined with in a differentiable function a, usually the Gated Recurrent Unit. (Cho al., 2014). The GRU architecture described in Appendix
The model parameters are the parameters of the topic model, the parameters of the RNN, the parameters of a, and the weights W DDV.
TDLM. TDLM (Lau et al., 2017) (short for Topically Driven Language Model) is variant of the model outlined above. There are two major differences. First, is not considered to be a latent variable. Instead, an encoder function maps a bag-of-words to 0. In the topic model, the bag-of-words used is from the entire document, i.e, TTM = enc(x].M)
Second, the language model component of the data generating process (Step 3) uses a different O than the topic modeling component, which call OLM, To prevent the model from memorizing the current sentence,

---

Published in Transactions on Machine Learning Research (11/2023)
BLM is computed from the document bag-of-words excluding the current sentence. In the language model, if is the index set of the words in the current sentence, OLM = enc(x TM 1.7.
Inference. TDLM is trained by maximizing the log likelihood of the topic model and the language model jointly. The objective, LTDLM, is:
LTM = lxg TM OTM)
LIM E, log p(ot xt LM OLM)
[TDLM LTM + LLM-
Although the original model only conditions previous words in the current sentence when forming LLM, we condition on all prior words in the document because it improves performance.
Prediction. Let - enc(x (1)) The next-word predictive distribution is
p(a
LM)
p(a M
x
LM 0),
(7)
which is defined in Step 3 of the data generating process. In practice, like prediction for the topic-biased LMs, we recompute in a sliding window.
GGBN-NNN. rGBN-RNN (Guo et al., 2020) is an extension of the model outlined in this section. In GBN--NNNss topic model, each sentence has unique bag-of- wordd:: it is the document's bag-of-words with the sentence excluded, denoted xTM TM I.T'J" In Step 1 of the data generating mechanism, different topic vector is drawn soquentially for cach sentence. For sentences j = I,...,J, where J is the total number of sentences,
f ~ Gamma To),
where and To are model parameters. Step 2, for each sentence j, its bag-of-words is drawn:
TM x 1TV
Poisson Φ0,),
where is a model parameter.
For the language modeling component (Step of the data generating mechanism), rGBN-RNN generates individual sentences. In Step 3, each sentence j conditionally independent of the other sentences, given its corresponding topic vector, 0j. In other words,
LM p(a), x <3: 0,) p(= 2x 02).
LM denotes all the words in document before the l'th word of where TrM is the 1'th word of sentence j, x the j'th sentence, and xj.t denotes only the words in the j'th sentence before the f'th word.
:GBN-RNN also introduces multiple stochastic layers to both the topic model and language model, which is simplified to one layer in this exposition. In the experiments, we use the full original model.
Inference. rGBN-RNN is trained using a combination of variational inference and stochastic gradient MCMC (Guo et al., 2018). We refer the reader to Guo et al. (2020) for further mathematical details of the model and inference algorithm.
Prediction. For each sentence j, the nord predictive distribution
LM p MMM = E, TM [p(2).
x 0,)].
The expectation approximated using a sample from the approximate posterior of 0, computed during inference. The topic model parameters, & and II, are similarly marginalized via MCMC sampling (see Guo et al. (2020) for more details).

---

Published in Transactions on Machine Learning Research (11/2023)
Table 1: The LSTM-LM baseline matches or exceeds topic-guided language models in held-out perplexity, In parentheses standard deviations estimated by averaging three runs with different random sceds. The comparable baseline to each topic-guided language model is the LSTM-LM (Zaremba et al., 2014) above it in the table. Parameter counts are listed in Appendix E.
Perplexity
Model
LSTM Size
Topic Size
APNEWS
IMDB
BNC
WT-2
LSTM-LM (sentence-level)
600
65.0 (0.3)
76.8 (0.1)
112.9 (0.3)
115.3 (0.8)
LSTM-LM
600
56.5 (0.3)
73.1 (0.2)
96.4 (0.1)
90.7 (0.7)
TopicRNN (Dieng et al., 2017)
600
100
56.6 (0.3)
73.0 (0.1)
96.8 (0.2)
93.2 (0.6)
VRTM (Rezaee & Ferraro, 2020)
600
50
56.8 (0.3)
73.6 (0.2)
96.3 (0.2)
90.8 (0.6)
LSTM-LM
600 (I))
53.5 (0.2)
68.8 (1.3)
91.2 (0.1)
89.9 (0.8)
TDLM (Lau et al., 2017)
600 (1C)U)
100
53.7 (0.1)
68.8 (0.1)
91.4 (0.2)
90.4 (0.7)
LSTM-LM
600x3
51.9 (0.4)
66.6 (0.8)
88.8 (0.3)
89.5 (0.7)
iGBN-RNN RNN (Guo et al., 2020)
600x3
100-80-50
52.6 (0.3)
64.8 (0.2)
97.7 (1.1)
5
Experiments
In this section, we detail the reproducibility study and results. We also investigate the quality of learned topics and probe the LSTM-LM's hidden representations to find the amount of retained topic information.
5.1 Reproducibility Study
We evaluate the held-out perplexity of four TGLMs and corresponding LSTM-LM basclines on four document-level corpora.
Datasets. We use four publicly available natural language datasets: APNEWS,? IMDB (Maas et al., 2011), BNC (Consortium, 2007), and WikiText-2 (Merity et al., 2017). We follow the training, validation, and test splits from Lau et al. (2017) and Mcrity et al. (2017). Details about the datascts and preprocessing steps are in Appendix C.
Models. The LSTM-LM is described in Section 2. The four topic-guided language models are TDLM (Lau et al., 2017), TopicRNN (Dieng et al., 2017), IGBN-RNN (Guo et al., 2020), and VRTM (Rezace & Ferraro, 2020), and are described in Section
We implement TopicRNN (Dieng et al., 2017), TDLM (Lau et al., 2017), and VRTM (Rezaee & Ferraro, 2020) from scratch. For iGBN-RNN (Guo et al., 2020), we use the publicly available codebase and make minimal adjustments to the codebase to ensure that preprocessing and evaluation are consistent. Some other topic-guided language models do not have public code (Wang et al., 2018; Tang et al., 2019; Wang et al., 2019) and are not straight forward to implement. These models are not part of the study, but their architecture is similar to that of Lau et al. (2017), which we compare to.
For all LSTM-LM basclines, we use a hidden size of 600, word embeddings of size 300 initialized with Google News word2vec embeddings (Mikolov et al., 2013), and dropout of 0.4 between the LSTM input and output layers (and between the hidden layers for the 3-layer models). For the four TGLMs we study, use the same settings as LSTM-LM for the LSTM components. For the additional TGLM-specific components, we use the architectures and set singings from the original papers, except for small details reported in Appendix D to make certain settings consistent across models. 3
To obtain comparable basclines to all TGLMs studied, we train three LSTM-LMS of varying sizes. The default baseline is a 1-layer LSTM-LM. To control for the additional GRU layer in the language model
Phttps: //wp://www//ee ap. org/en/
Thee one additional change is that Guo al. (2020) reports a 600-512-256 size model, but their public code only supports 3- layyerr models with same-size RNN layers, so use 600-600-600.
8

---

Published in Transactions on Machine Learning Research (11/2023)
component of TDLM, we train 1-layer LSTM-LM with GRU layer between the LSTM output and the output embedding layer. To compare to rGBN-RNN, hierarchical model, we train 3-layer LSTM-LM. Finally, we also compare to a baseline considered in prior work, an LSTM-LM conditioned only on previous words in the same sentence. We call this model the sentence-level LSTM-LM.
Training Details. We train the RNN components using truncated backpropagation through time with a sequence length of 30. For sequences within the same document (or sentence for the sentence-level LSTM- LM), if h, is the final hidden state computed by the RNN for the jth sequence, we initialize the initial hidden state for the (i + 1)th sequence with stop_ gro__graienn(b.).. Although the TDLM and iGBN-RNN models assume conditional independence between sentences, we found that in practice, keeping hidden states between sentences improved their performance.
Following Lau et al. (2017), Rezaee & Ferraro (2020), and Guo el al. (2020), we use the Adam optimizer with learning rate of 0.001 on APNEWS, IMDB, and BNC. For WikiText-2, we follow Merity et al. (2017) and use stochastic gradient descent; the initial learning rate is 20 and is divided by 4 when validation perplexity is worse than the previous ileration. The models are trained until validation perplexity does not improve for 5 epochs and we use the best validation checkpoint. The models in our codebase train to convergence within three days on a single Tesla V100 GPU. rGBN-RNN, trained using its public codebase, trains to convergence within one week on the same GPU. We do not include WikiText-2 results for rGBN-RNN because its perplexity did not decrease during training.
Results. Table 1 shows the results. After controlling for language model size, the LSTM-LM baseline consistently matches or oulperforms topic-guided language models with the same number of paramelers. Although most topic-guided language models improve over baseline considered in prior work the sentence level LSTM-LM they are matched by LSTM which, like topic-guided language models, conditions all prior words in a document. As discussed in Section 2, this is standard practice that can be performed at no extra computational cost.
5.2 Probing the RNN hidden states
The motivation for topic-guided language models is to augment language models with document-level in- formation captured by topic models (Dieng et al., 2017). To assess whether topic models are adding useful information, we perform a probe experiment. In NLP, probe tasks are designed to understand the extent to which linguistic structure encoded in the representations produced by black-box models (Alain & Bengio, 2016; Conncan et al., 2018; Hewitt & Liang, 2019; Pimentel et al., 2020). In this case, we probe the bascline LSTM-LM's hidden representations to assess how much topic information it already captures.
Specifically, we evaluate whether an LSTM-LM's hidden representation of the document's 'ss first t words, h is predictive of the topic voctor estimated from the document's first t words, O., of a topic-guided language model. We evaluate TDLM since it is the best performing topic-guided language model and has the highest quality topics (see Section 5.3). We also evaluate TopicRNN, topic-guided language model with lower quality topics.
We use the fitted models from Section 5.1 create training data for the probe experiment. The input is the L.STM-LM's initial hidden state h, for each sequence in a document (in this experiment, we define a sequence to be a 30-word chunk). The output is TDLM's topic proportions at the sequence, transformed with inverse-softmax to ensure it is real-valued: 0, = log(O.) ΣØ= log(Or,), where indexes the dimension of O.
In the experiment, a linear model is trained to predict O, from h, The loss function is mean squared error summed across the topic components. The held-out prediction quality of this model (or "probe") can be viewed as proxy for mutual information between the LSTM-LM's hidden state and the topic proportion vector 0, which is not easily estimable. For each topic-guided language model, we also run bascline experiment where we probe randomly initialized LSTM-LMM that was not fit. to data.
9

---

Published in Transactions on Machine Learning Research (11/2023)
Table 2: The probe experiment reveals that the hidden state of an LSTM-LM is predictive of the topic information of two topic-guided language models, TDLM and TopicRNN, held-out data. We also train baselines where the probe data is from randomly initialized LSTM-LM. Standard deviations are listed in Appendix F.
APNEWS
IMDB
BNC
WT-2
Target
Data
Acc-1
Acc-5
R²
Acc-1
Acc-5
R²
Acc-1
Acc-5
R²²
Acc-1
Acc-5
R²a
TDLM
init.
.036
.155
.134
.028
.132
.023
.098
.294
.073
.128
.379
.018
TDLM
trained
.400
.681
.621
.180
.49
.400
.314
.638
.379
.510
.858
.352
TopicRNN
init.
.075
.224
.022
.086
.259
.014
.077
.195
.017
.197
.422
-000
TopicRNN
trained
.210
.540
.232
.238
.575
.231
.172
.424
.170
.208
.486
.067
Table 2 shows the results of the probe experiment. The linear model can reconstruct TDLM's topic proportion vector to some extent; 62% of the variance in held-out set of APNEWS topic proportions can be explained by the LSTM's hidden state. Moreover, the hidden state predicts TDLM's largest topic for between 18% to 51% of test sequences across datasets, and improves 15% to 30% over the initialization-onlyy baseline. These accuracies indicate that the LSTM-LM has learned capture TDLM's most prevalent topic. Compared TDLM, the TopicRNN probe exhibits smaller improvement over its baseline. Nevertheless, the probe task shows that notable amount of broader topic information is caplured by the baseline LSTM's hidden state.
5.3 Topic quality
Although the predictions from topic-guided language models are matched or exceeded by an LSTM-LM baseline, it is possible that the learned topics will still be useful to practitioners. We compare the topics learned by topic-guided language models to those learned by a classical topic model, LDA (Blei et al., 2003). While LDA's next word predictions worse than those of neural topic models (Dieng et al., 2020), its topics may be more interpretable. To assess the quality of learned topics, we compute automated metric that correlates with human judgements of topic coherence (Aletras & Stevenson, 2013; Lau et al., 2014).
Automated coherence can be estimated using normalized pointwise mutual information (NPMI) scores. The NPMI score of a topic from its top words is defined as
N
N
log p(we.1))) p(ex)p((w))
(9)
log p(w,, w,)"
j=i+1
and ranges from -1 to 1, To compute the word co-occurrence statistics (estimates of p(w) and p(w,, wy) in Equation (9)) we use the corresponding datasct as the reference corpus. To obtain the model-level coherence, we average the scores from the lop 5/10/15/20 topic words for each topic, then average over all topics." 4
The coherence for each model in Table 3. The sopic-biased language models (TopicRNN and VRTM) learn largely incoherent topies, while only TDLM (Lau et al., 2017) achieves comparable coherences to LDA in two out. of the four corpora, APNEWS and BNC. Since the quality of automated topic evaluation metrics has been disputed for neural topic models (Hoyle et al., 2021), Appendix G includes the top words from randomly sampled topics. The joint topic and language models (TDLM and rGBN-RNN) learn qualitatively acceptable topics, while VRTM fails to learn distinct topics entirely.
6 Discussion
This reproducibility study compares topic-guided language models to LSTM baselines. We find that the basclines outperform the topic-guided language models in predictive performance. This finding differs from
We gensim (Rehurek Sojka, 2011) calculate NPMI scores, with window size of 10. In processing the reference corpora, we retain only terms that exist. the topic model vocabulary.
10

---

Published in Transactions on Machine Learning Research (11/2023)
Topic coherences across corpora and models. The topic-guided language models do not learn topics that are more coherent than LDA. In parentheses are standard deviations in coherence from training with three different random seeds per model. Note we do not include VRTM in the table because the model did not learn distinct topics (see Table 7).
Table 3:
Coherence
Model
APNEWS
IMDB
BNC
WT-2
LDA
.125 (.00)
.080 (.01)
.124 (.00)
.093 (.01)
TDLM
.176 (.00)
.011 (.02)
.104 (.01)
-002 (.03)
TopicRNN
-330 (.00)
-.327 (.01)
-2993 (.01)
-3311 (.00)
+GBN-RNN
.017 (.01)
.002 (.01)
-017 (.01)
the results reported in the topic-guided language modeling literature, which shows improvements over base- lines. In general, these differences are due to weaker baselines in the literature and form of evaluation that considers future words.
Baselines. The baseline compared to in most prior work (Lau et al., 2017; Rezaee & Ferraro, 2020; Guo et al., 2020) is the sentence-level LSTM-LM, which we report as the weakest model in Table 1; this baseline does not condition on all words in a document's history. Similarly, the baseline in Dieng et al. (2017) does not condition on the history beyond a fixed-context window. In contrast, the LSTM-LM baseline in this work conditions on all previous words in the document during training and evaluation. Our findings suggest that the predictive advantage of topic-guided language models stems from conditioning on all prior words in document via reprosentation of topic proportions.
Additionally, topic-guided language models typically augment their language model component with addi- tional parameters. We only compare topic-guided language models to baselines with similar number of language model parameters. TDLM (Lau et al., 2017) adds paramcters to its language model via an ad- ditional GRU; however, TDLM was originally compared to a baseline without this module. We find that TDLM's predictive advantage fades when scaling the LSTM-LM bascline match TDLM's language model size.
Different papers have evaluated the performance of topic-guided language models in different ways. In this paper, we slandardize the evaluation of models and their baselines to make sure results are comparable.
Evaluation.
As described in Section 4, topic-guided language models use a representation of the full document to estimate the topic proportions vector 0 during training. During evaluation, 0 must be estimated using only previous document words. Otherwise, a model would be looking ahead when making next-word predictions."
Some methods in the TGLM literature have conditioned future words in their evaluation, making them incomparable to language models that only condition on previous words. For example, TDLM (Tau et al., 2017) is proposed as a sentence-level model, and thus the paper reports results when conditioning on future words. Additionally, VRTM (Rezaee & Ferraro, 2020) and rGBN-RNN (Guo et al., 2020) are proposed as document-level models, but in the respective evaluation scripts of their public codebases, is estimated using future words.
Finally, conditioning during evaluation isn't the only difference among these models. Some prior papers do not use consistent language model vocabulary sizes, which makes reported numbers incomparable. For example, VRTM employs smaller vocabulary size than the baselines and other models it compares to. These discrepancies in evaluation may account for differences in reported results.
We also experiments that corrected this mismatch by estimating using only previous document words both training and evaluation, but found that this did not help performance.
11

---

Published in Transactions on Machine Learning Research (11/2023)
7 Conclusion
We find that compared to a standard LSTM language model, topic-guided language models do not improve language modeling performance. The probe experiment shows that this is due to both the standard LSTM already possessing some level of topic understanding but also that capturing the exact lopic vector is un- necessary for the task. For the two topic-biased language models, the quality of the learned topics poor, This may be due to the choice of latent variable for the topic proportions vector 0, previous work has found that using Gaussian leads to low topic coherence, while using Dirichlet with reparameterization gradients is prone to posterior collapse (Srivastava & Sutton, 2017; Burkhardt & Kramer, 2019).
While this study shows that current topic-guided language models do not improve next-word predictive performance, it possible that incorporating a topic model can provide greater control or diversity in language model generations. Topie-guided language models can generate text conditional on topics (Lau et al., 2017; Guo et al., 2020); one potential direction for future work is systematic investigation of controllability in topic-guided language models.
The topic-guided language modeling literature has focused on LSTMs, but we note that this framework is agnostic to the class of neural language model used. This means the same framework can be used to incorporate topic models into more powerful neural language models, such as transformers (Vaswani et al., 2017). However, incorporating topic models into transformers does not improve predictive performance or provide meaningful latent variables, transformers and LSTMs. Rather, the probing results in this paper indicate that neural language models are sufficiently expressive such that they already retain topic information. Transformers, which more expressive than LSTMs, are likely modeling topics. Novel approaches are needed to enable joint learning of expressive neural language models and interpretable, topic-based latent variables.
not necessarily because of architoctural differences between
more capable of capturing topic information without explicitly
Acknowledgements
We thank Adji Dieng and the reviewers for their thoughtful comments and suggestions, which have greatly improved the paper. This work is supported by NSF grant ITS 21 2127869, ONR grants N00014-17-1-2131 and 100014415-122099 the Simons Foundation, and Open Philanthropy.
References
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprint arXio:1610. 01644, 2016.
Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. In Inter- national Conference on Computational Semantics, 2013.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint 1rXi::180300127 2018.
David M. Blei and John D. Lafferty. A correlated topic model of science. The Annals of Applied Slatislics, 1(1), 2007.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent Dirichlet allocation. Journal of Machine Learning Research, 3(Jan), 2003.
David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518), 2017.
Sophie Burkhardt and Stefan Kramer. Decoupling sparsity and smoothness in the Dirichlet variational autoencoder topic model. Journal of Machine Leurning Research, 20(131), 2019.
Kyunghyun Cho, Bart van Merriënboer, Damitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder approaches. In Workshop on Syntaz, Semantics and Structure in Statistical Translation, 2014.
12

---

Published in Transactions on Machine Learning Research (11/2023)
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you cram into single S&!# vector: Probing sentence embeddings for linguistic properties. In Association for Computational Linguistics, 2018.
BNC Consortium. The British National Corpus, version 3 (BNC XML Edition), 2007.
M. Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. Are we really making much progress? worrying analysis of recent neural recommendation approaches. In Recommender Systems, 2019.
Adji B. Dieng, Chong Wang, Jianfeng Gao, and John Paisley. TopicRNN: A recurrent neural network with
long-range semantic dependency. In International Conference on Learning Representations, 2017.
Adji B. Dieng, Francisco R. Ruiz, and David M. Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 2020.
Thomas L. Griffiths and Mark Steyvers. Finding scientific topics. National Academy of Sciences, 101, 2004.
Justin Grimmer and Brandon M Stewart. Text as data: The promise and pitfalls of automatic content analysis methods for political texts. Political Analysis, 21(3), 2013.
Dandan Guo, Bo Chen, Hao Zhang, and Mingyuan Zhou. Deep Poisson Gamma dynamical systems. In
Newral Information Processing Systems, 2018.
Dandan Guo, Bo Chen, Ruiying Lu, and Mingyuan Zhou. Recurrent hicrarchical topic-guided RNN for language generation. In International Conference on Machine Learning, 2020.
John Hewitt and Percy Liang. Designing and interpreting probes with control tasks. In Empirical Melhods in Natural Language Processing, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neurol Computation, 9(8), 1997.
Alexander Hoyle, Pranav Goel, Andrew Hian-Cheong, Denis Peskov, Jordan Boyd-Graber, and Philip Resnik. Is automated topic model evaluation broken? The incoherence coherence. 2021.
Urvashi Khandelwal, He He, Peng Qi, and Dan Jurafsky. Sharp nearby, fuzzy far away: How neural language models use context. In Association for Computational Linguistics, 2018.
Diederik P. Kingma and Max Welling, Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
Dan Klein and Christopher D. Manning. Accurate unlexicalizod parsing. In Association for Computational Linguistics, 2003.
Jey H. Lau, David Newman, and Timothy Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In European Chapter of the Association for Computational Linguistics, 2014.
Jey H. Lau, Timothy Baldwin, and Trevor Cohn. Topically driven neural language model. In Association for Computational Länguistics, 2017.
Andrew L. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Association for Computational Linguistics: Human Language Technologies, 2011.
Benjamin Maric, Atsushi Fujita, and Raphael Rubino. Scientific credibility of machine translation research:
A meta-evaluation of 769 papers. In Association for Compulational Linguistics, 2021.
Andrew McCallum. MALLET: A machine learning for language toolkit. 2002.
Gábor Mclis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. CoRR, 2017.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Gábor Mclis, Tomás Kočiský, and Phil Blunsom. Mogrifier LSTM. In International Conference on Learning Represenlations, 2020.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. International Conference on Leurning Representations, 2017.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2, 2010.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprinl arXi:: 1311:1301.781,, 2013.
John W. Mohr and Petko Bogdanov. Topic models: What they are and why they matter, 2013.
Made Nindyatama Nityasya, Haryo Wibowo, Alham Fikri Aji, Genta Winata, Radityo Eko Prasojo, Phil Blunsom, and Adhiguna Kuncoro. On "scientific debt" in NLP: A case for more rigour in language model pre-training rescarch. In Association for Computational Linguistics, 2023.
Joe O' O'Connor and Jacob Andreas. What context features can transformer language models use? In Associ- ation for Computational Linguistics, 2021.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. Information-theored probing for linguistic structure. In Association for Computational Linguistics, 2020.
Radim Rehurek and Petr Sojka. Gensim python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic, 3(2), 2011.
Mchdi Rezace and Francis Ferraro. A discrete variational recurrent topic model without the reparametriza- tion trick. Neural Information Processing Systems, 2020.
Chinnadhurai Sankar, Sandeep Subramanian, Chris Pal, Sarath Chandar, and Yoshua Bengio. Do neural dia- log systems use the conversation history effectively? An empirical study. In Association for Computational Linguistics, 2019.
Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. In International Conference on Learning Representations, 2017.
Hongvin Tang, Miao Li, and Beihong Jin. A topic augmented text generation model: Joint learning of semantics and structural features. In Empirical Methods in Natural Language Processing, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Neurul Information Processing Systems, 2017.
Martin J. Wainwright, Michael I. Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends in Machine Learning, 1(1-2), 2008.
Hanna Wallach, David Mimno, and Andrew McCallum. Rethinking LDA: Why priors matter. In Neural Information Processing Systems, 2009.
Wenlin Wang, Zhe Gan, Wenqi Wang. Dinghan Shen, Jiaji Huang, Wei Ping, Sanjeev Satheesh, and Lawrence Carin. Topic compositional neural language model. In International Conference on Artificial Intelligence and Statistics, 2018.
Wenlin Wang, Zhe Gan, Hongteng Xu, Ruiyi Zhang, Guoyin Wang, Dinghan Shen, Changyou Chen, and Lawrence Carin. Topic-guided variational auto-encoder for text generation. In North American Chapter of the Association for Computational Linguistics, 2019.
Ronald J. Williams and Jing Peng. An efficient gradient- based algorithm for on-line training of recurrent network trajectories. Neural Computation, 2(4).
14

---

Published in Transactions on Machine Learning Research (11/2023)
Table Dataset statistics.
Training
Validation
Test
Dataset
Vocab Size
Docs
Tokens
Docs
Tokens
Docs
Tokens
APNEWS
34231
50K
15M
2K
0.6M
2K
0.6M
IMDB
36009
75K
20M
12.5K
0.3M
12.5K
0.3M
BNC
43703
15K
18M
1K
1M
1K
1M
WikiText-2
33280
6182
2M
620
218K
704
246K
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck:
A high-rank RNN language model. CoRR, 2017.
Wojciech Zaremba, 1lya Sutskever, and Oriol Vinyals. Recurrent neural network regularization, 2014.
A LSTM
The LSTM (Hochreiter & Schmidhuber, 1997) components are
i = a (W,v; + U,h;-1 + b;) f. = a (W,v; v Uph,-1 b,)
0: = a (W. ov. U,h; + h;)
é, tanh W.v, Uch:-1 be)
C₁ fi c. +
h, or @ lanh(c.).
The symbol @ denotes element-win product, while it, fi, or the input, forget, and output activations time t. Additionally, Vz, h, c are the input word embedding, hidden state, and cell state at time t. Finally, W, U.b are model parameters.
B
GRU
The GRU (Cho et al., 2014) components are
a (W,, v; + Ugh, + b.)
= oWW, v. + U,h, + b,)
h tanh W n (n (r, 3 he) bn)
h (1 2 3 h; + @ he
Here, 2; and rt are the update and reset gate activations at time t. Meanwhile, v, and h, are the input vector and the hidden state at time t, while W, U, and b are model parameters.
C Datasets
We evaluate on four publicly available corpora. APNEWS is a collection of Associated Press news articles from 2009 to 2016. IMDB is a set of movie reviews collected by Maas el al. (2011). BNC is the written portion of the British National Corpus (Consortium, 2007), which contains excerpts from journals, books, letters, essays, memoranda, news, and other types of text. WikiText-2 is a subset of the verified Good Featured Wikipodia articles (Merity al., 2017). A random subset of APNEWS and BNC is selected for the experiments.
15

---

Published in Transactions on Machine Learning Research (11/2023)
Table 4 shows the dataset statisties. The data is preprocessed as follows. For WikiText-2, we use the standard vocabulary, tokenization, and splits from Merity et al. (2017). We determine documents based on section header lines in the data. The EOS token is prepended to the start of each document and added to the end of cach document.
For APNEWS, TMDB, and BNC, documents are lowercased and tokenized using Stanford CoreNT.P (Klein & Manning, 2003). Tokens that occur less than 10 times are replaced with the UNK token. The SOS token is prepended to the start of each document; the EOS token is appended to the end of cach document. While use the same vocabulary and tokenization Lau et al. (2017), we do not add extra SOS and EOS tokens to the beginning and end of each sentence, so our perplexity numbers are not directly comparable to Lau et al. (2017); Rezaee & Ferraro (2020); Guo et al. (2020), who evaluate on the same datasets. When we redo the reproducibility experiment to use their original preprocessing settings, the trends in model performance are nearly identical to the results in this paper. We use the same splits as Lau et al. (2017).
Each model uses the same vocabulary for next-word prediction, so predictive performance comparable across models. Models have different specifications for the number of words in the topic model component. For rGBNN-NNN and TDLM, we follow the vocabulary preprocessing steps outlined in the respective papers. We exclude the top 0.1% most frequent tokens and tokens that appear in the Mallet stop word list. For TNNicRNN and VRTM, we additionally exclude words that appear in less than 100 documents, following Wang et al. (2018).
D Experiment Settings
We train all models on single GPUs with a language model batch size of 64. The experiments can be replicated on an AWS Tesla V100 GPU with 16GB GPU memory. LSTM-LM, TopicRNN, VRTM, and TDLM are implemented our codebase in Pytorch 1.12. We use the original implementation of rGBN-RNN, which uses Tensorflow 1.9.
We note differences between experiment settings and the original papers here. All other settings the same as in the original papers, and we refer the reader to them for details.
For the topic model components of the topic-guided language models, we keep the settings from the original papers. However, in some cases, the original papers use different language model architectures and settings. In order for a topic-guided language model's performance not to be confounded by use of a stronger or weaker language model component, was necessary to equalize these architectures and settings in the reproducibility study. Specifically, we 600 hidden units for the language model component and truncated BPTT length of 30 for all topic-guided language models.
Additionally, we initialize VRTM with pre-trained word embeddings rather than a random initialization, and we strengthen TopicRNN's stop word prediction component by replacing the linear layer with an MLP. We found these changes to help the performance of the respective models, so we included them in the reproducibility study. As noted the main paper, for rGBN-RNN, we use a model size of 600-600-600 because their public code only supports 3-layer models with same-size RNN layers.
As described in the main text, each model in Table 1 is trained until the validation perplexity does not improve for 5 epochs. After convergence, we use the checkpoint with the best validation perplexity. For each model, we perform three runs with random initializations trained until convergence. We report the mean of these runs along with their standard deviations.
We train LDA via Gibbs sampling using Mallet (McCallum, 2002). The hyperparameters are: a (lopic density) 50, 3 (word density) 0.01, number of iterations 1000.
E
Model Sizes
Table contains parameters counts for the basclines and topic-guided language models. Here, the TGLMs have 100 topics (except rGBN-RNN, which has 100-80-50 hierarchical topics) and the same topic model vocabulary from the APNEWS dataset.
16

---

Published in Transactions on Machine Learning Research (11/2023)
Table 5: Parameter counts for each LSTM-LM baseline and topic-guided language model.
Excluding embeddings
Including embeddings
Model
Total
LM
TM
Total
LM
TM
LSTM-LM (sentence-level)
2.2M
2.2M
33M
33M
LSTM-LM (1 layer)
2.2M
2.2M
33M
33M
TopicRNN (Dieng al., 2017)
2.5M
2.4M
0.1M
45M
33M
12M
VRTM (Rezace & Ferraro, 2020)
3.3M
2.4M
0.9M
37M
33M
4.1M
LSTM-LM (1 layer, with GRU)
3.3M
3.3M
46M
46M
TDLM (Lau al., 2017)
3.3M
3.3M
0.01M
46M
34M
LSTM-LM (3 layers)
7.9M
7.9M
39M
39M
+GBN-RNN (Guo al., 2020)
11.6M
11.6M
0.05M
90M
87M
3.3M
F Full Probing Results
Table shows the full results for the probe experiment. Standard deviations are computed from three runs with different random scods for both the LSTM-LM baseline and the topic-guided language model.
Table 6:
Probing results with standard deviations.
APNEWS
1MDB
Target
Data
Acc-1
Acc-5
R²
Acc-1
Acc-5
R²
TDLM
init.
.036 (.01)
.135 (.02)
.134 (.00)
.028 (.01)
.132 (.02)
.023 (.00)
TDLM
trained
340 (.01)
.681 (.01)
.621 (.01)
.180 (.02)
.449 (.04)
.400 (.02)
TopicRNN
init.
.075 (.00)
.224 (.02)
.022 (.00)
.086 (.01)
.299 (.04)
.014 (.00)
TopicRNN
trained
.110 (.00)
.540 (.01)
.232 (.00)
.238 (.02)
.575 (.00)
.231 (.01)
BNC
WT-2
Target
Data
Acc-1
Acc-5
R2a
Acc-1
Acc-5
TDLM
init.
.098 (.02)
.294 (.05)
.073 (.02)
.128 (.05)
.379 (.06)
.018 (.01)
TDLM
trained
.314 (.07)
.638 (.07)
.379 (.05)
.510 (.07)
.858 (.02)
.352 (.05)
TopicRNN
init.
.077 (.01)
.195 (.05)
.017 (.00)
.197 (.17)
.422 (.10)
-001 (.00)
TopicRNN
trained
.172 (.01)
.424 (.02)
.170 (.00)
(.14)
.486 (.11)
.067 (.01)
G Topic-Guided LM Topics
Table includes randomly sampled topics from each topic-guided language model fit to APNEWS.
17

---

Published in Transactions on Machine Learning Research (11/2023)
Table 7: Randomly selected learned topics from each model on APNEWS.
Model
Topics
stolen robber robbed store robbery stole theft jewelry robbers suspect plane aviation aircraft passengers helicopter pilots airport crashed faa guard crash driver vehicle truck highway car accident died injuries scene assange fifa wikileaks nsa snowden blatter iran ukraine journalists russian emissions nuclear renewable energy congress trade turbines obama reactor china
TDLM
arriving unsuccessful wash. fail audio bases bargaining sunset first-quarter install marion evacuate ceiling skull caliber tend evacuation exist shanghai sank graham turner ellis gordon albany edwards albuquerque davis cia contributions malloy dannel cheyenne buffalo indian hudson paris india carbon broadway follow-up scenario rebound rodham luxury rebel ordinary referring prohibiting insist.
TopicRNN
ground site family left dead kilometers miles residents village members world american disease america military blood days information pentagon top
:GBN-RNN
film art movie actor collection arlists artist studio festival theater recent called lost past washington small big good place today
workers system employees pay cost services agreement authority union contract
counties family high reported billion prosecutors community percentage asked caliber
angeles vegas moines half ap guilty prosecutors paso smith brown counties reported high family billion asked prosecutors gov. percentage earlier high prosecutors gov. family part american recent carlier past long gov. prosecutors high family recent earlier american past part top
VRTM
18


Reviews:
[{"review_id": "Bi48lZIJLR", "reviewer": "Reviewer_dbWw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "__Strengths__\n- Given the incredibly fast pace of the machine learning, NLP, and language modelling literature, __reproducibility and scientific rigor__ --- in particular, by conducting fair comparisons of different models under comparable experimental conditions, and accounting for various potential confounders like model sizes, proper tuning of the baseline models, etc. --- are extremely important in the field. This paper makes an important step in this direction: By comparing standard, well-tuned LSTM language models with their topic-guided counterparts in a level playing field, the paper derives useful insights (primarily around the fact that topic-guided language models do *not* necessarily outperform standard, well-tuned LSTM baselines, and that they do not necessarily learn coherent topics) that will be useful for other researchers in the field and the broader community. This kind of work is important to advance our __scientific understanding__ of what works well today (and what doesn't), above and beyond achieving the next state of the art results.\n- The paper provides a comprehensive review of the important concepts, such as topic models and topic-guided language models, that are necessary for less familiar readers to understand the key ideas, and therefore interpret the paper's key findings. I find the review of background material to be particularly thorough, especially given the page limit.\n- The paper conducts experiments on three different document-level language modelling datasets, and also takes into account the variance resulting from different random seeds, which improves the credibility of the findings and reflects more scientific rigor.\n\n__Weaknesses__\n- The choice of evaluation datasets should be broadened to include more commonly used document-level language modelling benchmarks, such as WikiText-2 and WikiText-103. This would make it easier to benchmark the paper's results against prior work on strong, document-level language models, which is difficult to do at the moment (currently Table 1 in the paper only includes the authors' own LSTM-LM implementation as the baseline). This would enable the readers to assess how the authors' baseline implementation compares against strong LSTM LMs used in prior work (e.g. Mogrifier LSTM, Melis et al., 2019), including those with a similar number of parameters. This would further improve the credibility of the findings.\n- The mathematical notation in the paper can be improved to be clearer and more precise. Following standard notation, I would recommend using bold uppercase letters for matrices, bold lowercase letters for vectors, and non-bold letters for scalars. Similarly, I would recommend using bold lowercase letters to denote a __sequence of words__ (e.g. $\\mathbf{x}_{<t}$ ). In contrast, a single word can be denoted with a lower, non-bold, italic lowercase letter, such as $x_t$. \n- The probing analysis --- while helpful --- can be strengthened by using control task (Hewitt and Liang, 2019) or minimum description length (Voita and Titov, 2020). This would help disentangle how much of the probing performance is due to the presence of the information in the hidden state vector, as opposed to the strength of the classifier itself.\n- Because the primary benefit of topic-guided language models is claimed to be long-range, document-level semantic coherence (maybe human evaluation over the generated text?), it would be nice if there is a targeted metric for this, to assess whether or not such models really have an advantage over standard, well-tuned LSTM LMs in terms of long-range semantic coherence.\n- There are still some open questions, suggestions, and stylistic suggestions (as listed below) that are not resolved yet.\n\n__Questions__\n- Does the topic-biased language model (section 4.1) basically learn to interpolate between the topic model and the standard LSTM LM? If so, it might be worth saying more explicitly.\n- In the \"Inference\" section of the TDLM, the topic model (TM) and the language model (LM) have two different sets of parameters. But in Eq. (7), $\\hat{\\boldsymbol{\\theta}}$ is based on an encoding of the prefix by the topic model (TM) parameters $\\boldsymbol{\\theta}^{\\text{TM}}$ , which is then fed into the LM parameters, even though the equation above it mentioned that $\\ell_{\\text{LM}}$ depends on $\\boldsymbol{\\theta}^{\\text{LM}}$, not $\\boldsymbol{\\theta}^{\\text{TM}}$. Could you please clarify this?\n- The backpropagation through time used a truncated length of 30 (page 8, training details). This seems to be on the shorter end; did you try how the results would change with a longer truncated length?\n- Another potential benefit of topic-guided language models is the __controllability__ angle, as we can ask the model to generate an article for a given topic vector. Having a discussion on whether this advantage is likely to hold (or mentioned as a potential topic of investigation for future work) would be nice.\n\n__Stylistic Suggestions__\n- In Table 1, I recommend putting the best entry model (lowest ppl.) for a given parameter count in bold, which would make the table much easier to read (this might require some rearranging of how the table is structured).\n-  In page 3, \"memoizing\" is a typo.\n- In page 9, \"prevelant\" is a typo.\n- In page 10, \"the the\" is a grammatical error.\n- I find the last sentence in the conclusion (\"Future work can prioritize...\") to be a bit hard to parse.\n\n__Citation__\nThere is a recent paper that discusses a similar issue (lack of fair comparison and scientific rigor when comparing different models, leading the community to have the wrong understanding of what works and what doesn't): Nityasya et al. (2023): \"On \"Scientific Debt\" in NLP: A Case for More Rigour in Language Model Pre-Training Research\".", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper revisits __topic-guided language models__, broadly defined as language models that combine the standard causal, left-to-right language modelling next-word prediction task with __topic models__ (Blei et al., 2003; inter alia). This line of work is motivated by earlier results on the limited ability of RNN and LSTM-based models in capturing long-range dependencies. In theory, such combination of language models and topic models would enhance topic-guided language models' ability to model long-range dependencies, by decoupling the predictions that only require *local* context (e.g. local syntactic and semantic dependencies), which RNN and LSTM-based language models can already do quite well, from those that require long-range, *document-level*, semantic context, which topic models are able to capture well by uncovering latent topics. To that end, prior work has claimed positive results for topic-guided language modelling, demonstrating that such topic-guided language models achieve better perplexity than standard LSTM-based models, whilst also learning interpretable topics as an important byproduct; this stands in contrast to the hidden state activations of standard LSTM-based language models, which are comparatively harder to interpret.\n\nThe paper begins with a review of both language models (in particular, those that are based on RNNs and LSTMs) and topic models, before summarising four major prior work in topic-guided language modelling, covering both __topic-biased language models__ and __joint topic and language model__. The paper then asks three key questions:\n- Under comparable and scientifically rigorous experimental conditions, to what extent --- if at all --- do topic-guided language models really outperform the standard LSTM language model baselines that are properly tuned to the same extent? In this work, the comparable experimental condition is achieved by controlling for three factors: (i) First, the topic model component should depend *only on the preceding words*, in order to preserve the left-to-right causality of the language modelling task; (ii) second, the baseline LSTM models should condition on prior words in the *entire document* history (as is standard in the language modelling literature), as opposed to only the prior words in the current sentence, as done in most prior topic-guided language modelling work; and lastly, (iii) the baseline models and the topic-guided ones should be compared under *similar model sizes*, as model size can be an important determinant of language modelling performance.\n- To what extent --- if at all --- do the learnt topics from topic-guided language models outperform those much simpler, standard topic model approaches?\n- To what extent can a properly tuned, standard LSTM-based language model encode a similar degree of topic information (in its hidden state activations) as the latent topics uncovered by more complicated, topic-guided language modelling approaches?\n\nTo that end, the paper found that, under comparable and scientifically rigorous experimental conditions as outlined above, the baseline LSTM LM approach can actually __outperform__ the four topic-guided language models in terms of perplexity; these results are consistent across three different document-level language modelling benchmarks. Furthermore, the latent topics uncovered by these topic-guided language models are, in fact, __less coherent__ than those found by a much simpler, standard LDA (Blei et al., 2003) approach. Finally, despite the baseline model's lack of an explicit objective function or encouragement to uncover latent topics, probing analysis demonstrates that the hidden state activations of standard LSTM LMs do, in fact, encode a fairly similar extent of latent topic information as that uncovered by a topic-guided language model approach (TDLM, Lau et al., 2017).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- **Strongly Recommended**:  Conducting experiments on a more commonly used document-level language modelling benchmark, such as WikiText 2 or WikiText-103. This would give the readers a better idea of how the numbers compare with strong language models in prior work (weakness 1 above).\n- **Strongly Recommended**: Improving the mathematical notations to be more standard and precise (weakness 2 above).\n- **Strongly Recommended**: Re-running probing analysis with either control task or minimum description length probing, to assess how much of the probe results are really due to the probed information being present in the hidden state activation (as opposed to the classifier's expressivity) (weakness 3 above).\n- **Strongly Recommended**: Resolving the questions (and incorporating the relevant reply into the paper where applicable), stylistic suggestions, and citation as raised above.", "broader_impact_concerns": "I do not foresee any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jD6vWSVdlc", "reviewer": "Reviewer_5ZBx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper represents a nice, if modest, contribution. It is very well written and easy to read. The intuition that the authors support (i.e., that LMs already capture topic information and thus topic modeling is in some ways “redundant”) is an intuition I have personally had when seeing work on TGLMs and so I can’t help but find it satisfying that the authors confirm this intuition. I have some qualms with the experiments (below), in particular the probing studies, and I think augmenting the experiments to address these (or rearranging so that material from appendix can appear in the main text, if needed) would significantly strengthen the work. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper takes a deep dive on evaluating work on topic-guided language modeling (TGLM). TGLM aims to improve language model performance by integrating more traditional topic models with RNN-family language models, under the assumption that topic models will capture global document information better than RNNs do on their own. The authors point to multiple flaws in the experimental design of prior work, and show that when these flaws are fixed and thus evaluation is more apples-to-apples, there is no performance gain to TGLM compared to vanilla LMs. The primary results focus on the language modeling performance, but the authors include two sections of additional results, one in which they use probing to measure whether vanilla LMs capture topic information natively, and one in which they look at the quality of the learned topics using automated metrics. \n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions about Experiments/Content:\n* Since this is a “reproducibility study”, I would like to see more depth and discussion surrounding the differences in your (re)implementation of prior work vs. what was reported in that prior work. In particular, for many models, you say you reimplement from scratch. This is fine—but can you report how your reimplementation compares to the prior work before you make any adjustments to their setup? I.e., can you quantify how much better/worse your implementation’s performance is assuming your goal is just to exactly replicate the prior work? I think it is important to show this, so we can fully understand where the disconnect between your findings and the prior work’s findings stems from. Put differently: a skeptic might say that your lack of difference between models isn’t due to what you say (i.e., isn’t due to a more apples-to-apples comparison) but rather might just mean you did a bad job reimplementing the work! So if you can show that your reimplementation is legit, that strengthens your argument.\n* I am skeptical of the section on probing, I think you need to add more in order to make these results compelling. Specifically:\n—> Can you include a baseline or two in Table 2? E.g., maybe most frequent topic, and performance of a probe trained on a randomly initialized LM? I don’t know how to contextualize these numbers, so I can’t tell if these support the claim that there is significant topic information captured.\n—> Why do you only probe one TGLM? Can you probe the others too, to help contextualize?\n—> The issue you mention about how probe performance is correlated with topic model performance concerns me. Can you dig into this more? I feel like there are some artifacts driving the results which aren’t transparently captured in your analysis. Maybe better baselines will help disentangle, or maybe you can run some other analysis to convince us that the probe is capturing exactly what you want it to capture, and not random “other stuff”.\n* (This is a thought which is harder to address, so not expecting anything, just dropping it here.) I wonder if you can design a baseline or experiment that gets to the heart of what you are trying to claim here. Namely, I feel like you are trying to say that both topic modeling and language modeling are capturing largely the same stuff, e.g., the principle components of the corpus perhaps? And thus a topic-augmented LM is redundant. Can you quantify this intrinsic connection between the models somehow? Either experimentally or mathematically?\n\nComments/Suggestions on Writing and Style:\n* The paper is nicely written, and whoever did the writing is clearly a good teacher/good at explaining things. :) But that said, there was a lot of time spent on background that was not novel to the paper and could be cut. For example, the section on variational inference felt odd. Either the reader already knows VI and will skip this section, or they don’t know about VI in which case this short description will hardly suffice to catch them up to speed. Moreover, understanding VI is not at all necessary for your contribution. I’d recommend cutting sections of this type of background (on VI, on LSTMs, on LDA, etc) and instead use the extra space to deepen your experimental results and analysis.\n* The term “reproducibility study” is a bit of a misnomer here. That implies that you are simply trying to replicate the numbers that others produced, and are not able to do so. But you are actually trying to correct previous problems with experimental design. So it’s not a problem with reproducibility, but rather that when you fix certain problems, you draw an opposite conclusion from the original work. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "31g7dU3ENJ", "reviewer": "Reviewer_5bYd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to read. The problem is clearly defined, the motivation is clear, and the analysis flows smoothly.\n\n2. A comprehensive analysis of topic-guided language models is conducted, reaching insightful conclusions. The conclusion challenges the effectiveness of the methodology in prior work, which would be of interest to the community.\n\n3. This paper would contribute to a line of reproducibility studies that aim to evaluate competing methods in a consistent and equitable manner. The authors promise that the code will be publicly available.\n\nWeaknesses:\n\n1. There are different experimental settings in previous studies. The choice of the unified setting lacks clarity. It is not clear if the conclusion still holds when using different experimental settings, e.g., different or no pre-trained word embeddings, different architecture, etc.\n\n2. Only LSTM approaches are evaluated. As there are transformer-based topic-guided language models proposed in recent years, it would be interesting to see if those transformer approaches also have a similar phenomenon. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper compares the effectiveness of topic-guided language models with the standard LSTM language model baselines in a unified setting. Four topic-guided language models and two baselines are evaluated by the held-out predictive performance of each model on three corpora. The authors find that none of these topic-guided language models outperform a standard LSTM language model baseline, and most fail to learn good topics. Probing analysis shows that the baseline’s hidden states already encode topic information, which explains why the topic-guided language models do not yield better performance. \n\nThe major contribution is that this work provides an insightful analysis showing that topic-guided language models do not perform effectively as reported in prior work due to inconsistent settings with baselines. In contrast, the LSTM language model has already captured effective topic information inside the hidden states.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More comprehensive studies with other model architectures and other tasks are suggested.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, it is a good study and makes good contributions to the community (more details can be found in reviews).

My recommendation is "accept with minor revision". 

The authors need to address the concern about test datasets and revise the paper:
This work focuses on empirical study, but conducted experiments on less well-used datasets, which makes the conclusions less convincing, as there is not much prior work on these datasets. Therefore, the authors need to test on widely used benchmarks like WikiText-103 or other larger datasets and add new results into the final version. 
Decision: Accept (poster)