SlowGuess/NAACL_2021_OCR huggingface dataset:
Aaron Mueller and Mark Dredze
amueller@jhu.edu, mdredze@cs.jhu.edu
Neural topic models can augment or replace bag-of-words inputs with the learned representations of deep pre-trained transformer-based word prediction models. One added benefit when using representations from multilingual models is that they facilitate zero-shot polylingual topic modeling. However, while it has been widely observed that pre-trained embeddings should be fine-tuned to a given task, it is not immediately clear what supervision should look like for an unsupervised task such as topic modeling. Thus, we propose several methods for fine-tuning encoders to improve both monolingual and zero-shot polylingual neural topic modeling. We consider fine-tuning on auxiliary tasks, constructing a new topic classification task, integrating the topic classification objective directly into topic model training, and continued pre-training. We find that fine-tuning encoder representations on topic classification and integrating the topic classification task directly into topic modeling improves topic quality, and that fine-tuning encoder representations on any task is the most important factor for facilitating cross-lingual transfer.
Topic models (Blei et al., 2003) are widely used across numerous disciplines to study large corpora (Boyd-Graber et al., 2017). These data-driven models discover salient themes and semantic clusters without any supervision. Monolingual topic models are language-agnostic but do not align topics across languages, as they have a fixed language-specific vocabulary which cannot be aligned cross-lingually after training. Polylingual topic models (Mimno et al., 2009), however, enable users to consider multilingual corpora, and to discover and align topics across languages.
Recent work has demonstrated the effectiveness of deep transformer-based language models to en
code text documents for a wide variety of applications (Xia et al., 2020). Furthermore, when trained on multilingual corpora, they have been able to discover cross-lingual alignments despite the lack of explicit cross-lingual links (Wu and Dredze, 2019). Models such as multilingual BERT (mBERT; Devlin et al., 2018) or XLM-RoBERTa (XLM-R; Conneau et al., 2019) can produce a representation of text in a shared subspace across multiple input languages, suitable for both monolingual and multilingual settings, including zero-shot language transfer (Pires et al., 2019).
Simultaneously, topic models have increasingly incorporated neural components. This has included inference networks which learn representations of the input document (Miao et al., 2017; Srivastava and Sutton, 2017) that improve over using bags of words directly, as well as replacing bags of words with contextual representations. In particular, the latter allows topic models to benefit from pre-training on large corpora. For example, contextualized topic models (CTMs) (Bianchi et al., 2020a) use autoencoded contextual sentence representations of input documents.
An intriguing advantage of using encoders in topic models is their latent multilinguality. Polylingual topic models (Mimno et al., 2009) are lightweight in their cross-lingual supervision to align topics across languages, but they nonetheless require some form of cross-lingual alignment. While the diversity of resources and approaches for training polylingual topic models enable us to consider many language pairs and domains, there may be cases where existing resources cannot support an intended use case. Can topic models become polylingual models by relying on multilingual encoders even without additional alignments?
Bianchi et al. (2020a) show that CTMs based on contextual sentence representations enable zero-shot cross-lingual topic transfer. While promising, this line of work omits a key step in using con
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3054-3068 June 6-11, 2021. ©2021 Association for Computational Linguistics
textualized embeddings: fine-tuning. It has been widely observed that task specific fine-tuning of pretrained embeddings, even with a small amount of supervised data, can significantly improve performance on many tasks, including in zero- and few-shot settings (Howard and Ruder, 2018; Wu and Dredze, 2019). However, in the case of unsupervised topic modeling, from where are we to obtain task-specific supervised training data?
We propose an investigation of how supervision should be bootstrapped to improve language encoders for monolingual and polylingual topic model learning. We also propose a set of experiments to better understand why certain forms of supervision are effective in this unsupervised task. Our contributions include the following:
1. We fine-tune contextualized sentence embeddings on various established auxiliary tasks, finding that many different tasks can be used to improve downstream topic quality and zero-shot topic model transfer.
2. We construct fine-tuning supervision for sentence embeddings through a proposed topic classification task, showing further improved topic coherence. This task uses only the data on which we perform topic modeling.
3. We integrate a topic classification objective directly into the neural topic model architecture (without fine-tuning the embeddings) to understand whether the embeddings or the topic classification objective is responsible for performance improvements. We find that this approach improves topic quality but has little effect on cross-language topic transfer.
We present results for both monolingual topic models and cross-lingual topic transfer from English to French, German, Portuguese, and Dutch.
Our code, including instructions for replicating our dataset and experimental setup, are publicly available on GitHub.
Neural Topic Models Neural topic models (NTMs) are defined by their parameterization by (deep) neural networks or incorporation of neural elements. This approach has become practical largely due to advances in variational inference—specifically, variational autoencoders (VAEs; Kingma and Welling, 2013). The Neural
Variational Document Model (Miao et al., 2016) and Gaussian Softmax Model (Miao et al., 2017) rely on amortized variational inference to approximate the posterior (Zhao et al., 2017; Krishnan et al., 2018). As these methods employ Gaussian priors, they use softmax transforms to ensure non-negative samples. Another approach has used ReLU transforms (Ding et al., 2018).
Conversely, ProdLDA (Srivastava and Sutton, 2017) uses a Dirichlet prior that produces nonnegative samples which do not need to be transformed. ProdLDA uses an inference network with a VAE to map from an input bag of words to a continuous latent representation. The decoder network samples from this hidden representation to form latent topic representations. Bags of words are reconstructed for each latent space; these constitute the output topics. Others have reported that ProdLDA is the best-performing NTM with respect to topic coherence (Miao et al., 2017).
Contextualized topic models (CTMs; Bianchi et al., 2020a,b) extend ProdLDA by replacing the input bag of words with sentence-BERT (SBERT; Reimers and Gurevych, 2019) embeddings. If the SBERT embeddings are based on a multilingual model such as mBERT (Devlin et al., 2018) or XLM-R (Conneau et al., 2019), then the topic model becomes implicitly polylingual due to the unsupervised alignments induced between languages during pre-training. This is distinct from how polylinguality is induced in approaches based on Latent Dirichlet Allocation (LDA; Blei et al., 2003), which require some form of cross-lingual alignments (Mimno et al., 2009).
Using embeddings in topic models is not new (Das et al., 2015; Liu et al., 2015; Li et al., 2016). While a few recent approaches have leveraged word embeddings for topic modeling (Gupta et al., 2019; Dieng et al., 2020; Sia et al., 2020), none of these have investigated cross-lingual topic transfer.
Polylingual Topic Models Polylingual topic models require some form of cross-lingual alignments, which can come from comparable documents (Mimno et al., 2009), word alignments (Zhao and Xing, 2006), multilingual dictionaries (Jagarlamudi and Daumé, 2010), code-switched documents (Peng et al., 2014), or other distant alignments such as anchors (Yuan et al., 2018). Work on incomparable documents with soft document links (Hao and Paul, 2018) still relies on dictionaries.
While these types of alignments have been com
1https://github.com/aaronmueller/ contextualized-topic-models
mon in multilingual learning (Ruder et al., 2019b), they no longer represent the state-of-the-art. More recent approaches instead tend to employ large pretrained multilingual models (Wu and Dredze, 2019) that induce unsupervised alignments between languages during pre-training.
3 Fine-Tuning Encoders
Fine-tuning is known to improve an encoder's representations for a specific task when data directly related to the task is present (Howard and Ruder, 2018; Wu and Dredze, 2019). Nonetheless, this requires supervised data, which is absent in unsupervised tasks like ours. We consider several approaches to create fine-tuning supervision for topic modeling.
3.1 Fine-tuning on Related Tasks
In the absence of supervised training sets, transfer learning can be used to learn from one supervised task (or many tasks in the case of meta-learning) for improvements on another (Ruder et al., 2019a). While transfer is typically performed from a pretrained masked language model to downstream finetuning tasks, transfer can also be performed from one fine-tuning task to another. The aim is that the auxiliary task should induce representations similar to those needed for the target task.
What task can serve as an effective auxiliary task for topic modeling? We turn to document classification, the task of identifying the primary topic present in a document from a fixed set of (typically human-identified and human-labeled) topics. We may not have a document classification dataset from the same domain as the topic modeling corpus, nor a dataset which uses the same topics as those present in the corpus. However, fine-tuning could teach the encoder to produce topic-level document representations, regardless of the specific topics present in the data. We use MLDoc (Schwenk and Li, 2018), a multilingual news document classification dataset and fine-tune on English.
For comparison, we fine-tune on a natural language inference (NLI) task. While it is not closely related to topic modeling, this task is a popular choice for fine-tuning both word and sentence representations. This allows us to measure how much task relatedness matters for fine-tuning.
3.2 Fine-tuning on Topic Models
The auxiliary tasks use data from a different domain (and task) than the domain of interest for the topic model. Can we bootstrap more direct supervision on our data?
We employ an LDA-based topic model to produce a form of topic supervision. We first run LDA on the target corpus to generate topic distributions for each document. Then, we use the inferred topic distributions as supervision by labeling each document with its most probable topic. We fine-tune on this data as we did for the document classification task; the setup is identical except for how the labels are obtained. The advantage of this method is that LDA topics can be created for any corpus.
3.3 Continued Pre-Training
Gururangan et al. (2020) advocated for adapting an encoder to the domain on which one will later fine-tune. This is done by performing continued pre-training over in-domain data using the masked language modeling (MLM) objective. Because continued pre-training requires no task-specific supervision, and because topic modeling implies a sizeable corpus of in-domain documents, we consider continued pre-training on the target corpus as another approach to adapting an encoder. As continued pre-training can be done before fine-tuning, we also try doing both.
Does topic classification improve performance because fine-tuning itself induces better representations for topic modeling, or because the model has been exposed to in-domain data and/or supervision directly from the target corpus before topic modeling? Continued pre-training on the target corpus may allow us to answer this question, and provides a further approach for adapting encoders to specific domains.
3.4 Modifying the Topic Modeling Objective
Both continued pre-training and fine-tuning provide supervision for our target task, but both create dependence on a pipeline: we must train and/or fine-tune sentence embeddings, then train a neural topic model using the modified embeddings.
However, we can combine the topic classification task and topic modeling into a single end-to-end procedure by modifying the inference network of the CTM. Figure 1 shows our proposed archi
2We note that there are mixed findings in the literature with respect to this method (Du et al., 2020).
Figure 1: Architecture used in the topic classification contextualized topic model (TCCTM) approach (§3.4). This is similar to the architecture of Bianchi et al. (2020a), but with an added fully-connected layer and softmax to produce a topic classification for the input document from its hidden representation.
tecture: a fully-connected layer into a softmax to predict the topic label of the document based on the learned representation of the VAE. Note that we do not necessarily expect this architecture to outperform fine-tuning sentence embeddings: rather, this architecture allows us to ablate over the location of the topic classification objective, which allows us to determine whether improvements in topic quality and/or transfer are due to improved sentence embeddings induced by fine-tuning, or due to the topic classification task itself.
We use the negative log-likelihood loss between the topic predicted by LDA (which we treat as the true label) and the topic predicted by our model, adding this loss term (weighted by a hyperparameter  $\lambda$ ) to the contextualized topic model's loss function. Thus, the new loss becomes
$$
\mathcal {L} _ {\mathrm {T C C T M}} = \mathcal {L} _ {\mathrm {E L B O}} + \lambda \mathcal {L} _ {\mathrm {N L L}}
$$
where  $\mathcal{L}_{\mathrm{ELBO}}$  is the negated evidence lower bound objective of the CTM, and  $\mathcal{L}_{\mathrm{NLL}}$  is the negative log-likelihood loss over topic classifications. We refer to this as the topic classification contextualized topic modeling (TCCTM) loss, denoted  $\mathcal{L}_{\mathrm{TCCTM}}$ .
TCCTM modifies the topic model, but not the embeddings. This approach is therefore orthogonal to fine-tuning, and the two approaches can be combined; thus, we test the performance of TCCTM with and without fine-tuning.
Data We begin by creating a multilingual dataset for topic modeling based on aligned Wikipedia articles extracted from Wikipedia Comparable Corpora³ in English, French, German, Portuguese, and Dutch. We use 100,000 English articles for training the topic models and evaluating monolingual topic coherence. We also extract 100,000 aligned articles for each language to build comparable vocabularies for preprocessing the test data.⁴ For each language, we use a vocabulary of the 5,000 most frequent word types (case-insensitive), excluding stopwords—25,000 types total. We use the English training articles to evaluate monolingual topic quality, and hold out for cross-lingual evaluation a set of 10,000 aligned test articles per-language.
For out-of-domain topic classification, we use a dataset of COVID academic articles (in English).5 To facilitate comparison with the Wikipedia dataset, we extract 100,000 articles and use a vocabulary size of 5,000.
To obtain topic labels for each English document, we run LDA for 400 iterations and choose the number of topics  $\tau$  by performing a search in  $\{10,20,\dots,250\}$ , optimizing over NPMI coherence. We find that  $\tau \in \{100,110,120\}$  is best and use  $\tau = 100$  here. We label each document with its most probable topic by counting the number of tokens in the document in the top-10 token list for each topic, then taking the argmax. We perform the same procedure on the out-of-domain COVID dataset to generate out-of-domain topic classification supervision, finding that  $\tau = 80$  is best on this dataset with respect to NPMI coherence.
For the document classification task, we use MLDoc (Schwenk and Li, 2018), a multilingual news dataset; we fine-tune on the English data. For NLI, we follow Reimers and Gurevych (2020) in using a mixture of SNLI (Bowman et al., 2015) and MultiNLI (Williams et al., 2018), both of which only contain English data.
Training Details We consider embeddings produced by both mBERT (Devlin et al., 2018) and XLM-R (Conneau et al., 2019). For fine-tuning, we append to these models a fully-connected layer fol
3https://linguatools.org/tools/corpora/wikipedia-comparable-corpora/
4We release article IDs and splits with our code.
<sup>5</sup>https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge
lowed by a softmax, using a negative log-likelihood loss for topic/document classification. We perform a search over the number of epochs in the range [1, 8], optimizing over downstream NPMI coherence during topic modeling.
We follow the procedure of Reimers and Gurevych (2019) to create sentence embedding models from contextual word representations: we mean-pool word embeddings for two sentences simultaneously, feeding these as inputs to a softmax classifier. We use batch size 16; other hyperparameters are kept from Reimers and Gurevych (2019).
For NLI fine-tuning, we follow the procedure and use the hyperparameters of Reimers and Gurevych (2020): we first fine-tune monolingual BERT on SNLI and MultiNLI; the embeddings are pooled during fine-tuning to create a sentence embedding model. We then perform a knowledge distillation step from the monolingual SBERT model to XLM-R or mBERT.
Continued pre-training is performed by training with the MLM objective on English Wikipedia. We run for 1 epoch, using gradient accumulation to achieve an effective batch size of 256. We can pool the embeddings from the resulting model directly or perform fine-tuning after continued pre-training.
When topic modeling, we run the CTM for 60 epochs, using an initial learning rate of  $2 \times 10^{-3}$ , dropout 0.2, and batch size 64. The VAE consists of two hidden layers of dimensionality 100 (as in Srivastava and Sutton 2017 and Bianchi et al. 2020b). The ProdLDA baseline is run using the same hyperparameters and the same architecture as a CTM, differing only in using bags of words as input instead of SBERT representations.
For the LDA baseline, we employ MalletLDA (McCallum, 2002) as implemented in thegensim wrapper, running for 400 iterations on the Wikipedia data using  $\tau = 100$ .
We fine-tune  $\lambda$  in the TCCTM objective in  $\{0.1, 0.2, \dots, 3.0\}$ , finding that  $\lambda = 1.0$  yields the best downstream topic coherence for the target Wikipedia data. We try TCCTM based on non-fine-tuned sentence embeddings, as well as models fine-tuned on document classification or NLI. We do not perform this approach on a model fine-tuned on in-domain topic classification to avoid overfitting and confounds from performing the same task in multiple stages of the model.
Evaluation To evaluate topic quality, we measure normalized pointwise mutual information
(NPMI) coherence on the English Wikipedia dataset. NPMI is used because it is comparable across architectures and objectives, and because it tends to correlate better with human judgments of topic quality (Lau et al., 2014). While perplexity has been used to evaluate LDA (Blei et al., 2003) as well as neural topic models in the past (Miao et al., 2017), it is not comparable across different objective functions when using neural approaches (as it depends on the test loss) and tends to correlate poorly with human judgments (Chang et al., 2009). Topic significance ranking (AlSumait et al., 2009) has been used to measure and rank topics by semantic importance/relevance, though we care more about overall topic quality than ranking topics.
As the contextualized topic model is based on a multilingual encoder, it is able to generate  $\theta_{i}$  (a topic distribution over document  $i$ ) given input embeddings from a document  $\mathbf{h}_i$  in any language it has seen. To evaluate multilingual generalization, we measure the proportion of aligned test documents for which the most probable English topic  $\theta_{i_{\text{English}}}$  is the same as the most probable target-language topic  $\theta_{i_{\text{Target}}}$  (the Match metric). We also measure the KL divergence between topic distributions  $\mathrm{D}_{\mathrm{KL}}(\theta_{i_{\text{English}}} \| \theta_{i_{\text{Target}}})$ , taking the mean over all aligned documents (the KL metric). We construct a random baseline by randomly shuffling the English articles and then computing both metrics against the newly unaligned foreign articles.
5.1 Monolingual Topic Modeling
We compare topic coherences on the 100,000 English Wikipedia articles for LDA and ProdLDA baselines, a CTM with no fine-tuning, a CTM with continued pre-training (CPT), and the integrated TCCTM model. We also compare the effect of fine-tuning (FT) on the NLI task, on a document classification task (MLDoc), and on labels from LDA for the out-of-domain COVID dataset and for the in-domain Wikipedia data (Table 1). The baseline LDA and ProdLDA models both achieve the same coherence score of 0.129. Compared to these baselines, models based on contextualized representations always achieve higher topic coherence.
We find that when using a base CTM without modifying its objective, fine-tuning on any auxiliary task improves topic quality for CTMs. Specifically, fine-tuning on in-domain topic clas
Model	Fine-tuning	NPMI	Neural model	Fine-tuned embeddings	Topic classification	In-domain data
LDA	-	0.129
ProdLDA	-	0.129	✓
XLM-R	mBERT
CTM	None	0.144	0.144	✓
NLI	0.153	0.152	✓	✓
Doc. Class.	0.156	0.153	✓	✓
Topic Class. (COVID)	0.156	0.153	✓	✓	✓
Topic Class. (Wiki)	0.160	0.156	✓	✓	✓	✓
CPT+CTM	None	0.147	0.147	✓	✓
NLI	0.150	0.149	✓	✓	✓
Topic Class. (COVID)	0.148	0.147	✓	✓	✓	✓
Topic Class. (Wiki)	0.151	0.149	✓	✓	✓	✓
TCCTM	None	0.157	0.154	✓	✓
NLI	0.152	0.151	✓	✓	✓
Doc. Class.	0.153	0.152	✓	✓	✓
Table 1: NPMI coherences for contextualized topic models (CTM), CTMs with continued pre-training (CPT+CTM), and the TCCTM model on the English Wikipedia dataset. We present results with and without fine-tuning for XLM-R and mBERT-based sentence embeddings. The right side of the table indicates whether each setup is based on a neural architecture, whether the SBERT embeddings are fine-tuned before topic modeling, whether the topic classification task/objective is present, and whether the embeddings have been trained/fine-tuned on the same data later used for topic modeling.
sification data is best for monolingual topic modeling, followed closely by document classification on MLDoc. Topic classification on the out-of-domain COVID data results in the same topic coherence scores as document classification, indicating that topic classification is an effective method for bootstrapping supervision, even compared to established document classification datasets with human-labeled documents. The further gains in topic coherence when fine-tuning on Wikipedia topic classification data may be due to the data being in-domain, rather than due to the topic classification task. Fine-tuning on NLI yields less coherent topics than document or topic classification. For any given approach, XLM-R always outperforms mBERT.
We find that CPT without fine-tuning performs worse than simply fine-tuning, but better than a CTM using embeddings which are not fine-tuned. Fine-tuning after performing continued pretraining (CPT+FT) slightly improves NPMI over CPT alone, but still results in less coherent topics than if we simply fine-tune on the in-domain Wiki data or the out-of-domain COVID data. Thus, the MLM objective seems to induce representations not conducive to topic modeling. Indeed, fine-tuning