SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yishi Xu\*, Jianqiao Sun\*, Yudi Su, Xinyang Liu, Zhibin Duan, Bo Chen
National Key Laboratory of Radar Signal Processing, Xidian University, Xi'an, China, 710071
{xuyishi, jianqiaosun}@stu.xidian.edu.cn, bchen@mail.xidian.edu.cn
Mingyuan Zhou
McCombs School of Business, The University of Texas at Austin, TX 78712, USA
mingyuan.zhou@mccombs.utexas.edu
Embedding-based neural topic models have turned out to be a superior option for low-resourced topic modeling. However, current approaches consider static word embeddings learnt from source tasks as general knowledge that can be transferred directly to the target task, discounting the dynamically changing nature of word meanings in different contexts, thus typically leading to sub-optimal results when adapting to new tasks with unfamiliar contexts. To settle this issue, we provide an effective method that centers on adaptively generating semantically tailored word embeddings for each task by fully exploiting contextual information. Specifically, we first condense the contextual syntactic dependencies of words into a semantic graph for each task, which is then modeled by a Variational Graph Auto-Encoder to produce task-specific word representations. On this basis, we further impose a learnable Gaussian mixture prior on the latent space of words to efficiently learn topic representations from a clustering perspective, which contributes to diverse topic discovery and fast adaptation to novel tasks. We have conducted a wealth of quantitative and qualitative experiments, and the results show that our approach comprehensively outperforms established topic models.
The last two decades have witnessed the enduring popularity of topic models along with their many successful applications in a range of fields [1-7]. And this is predominantly attributed to their ability to reveal the underlying semantic structure from large volumes of textual data. By identifying a group of salient themes, topic models represent each document as a mixture of them, providing an intuitive understanding of the target corpus. Although conventional probabilistic topic models [8-14] have been widely used, new variants continue to spring up in the era dominated by deep neural networks.
Among the proliferation of new methodologies come both the neural topic models (NTMs) [15-18] resulting from the development of variational autoencoders (VAEs) and autoencoding variational Inference (AVI) [19, 20], and the contextualized topic models (CTMs) [21-24] benefiting from the flourishing of pre-trained language models [25, 26]. However, these recently developed approaches essentially maintain the assumption of sufficient resources, i.e., with a plethora of documents being available. Comparatively, little attention has been paid to topic modeling under resource-limited or resource-poor conditions [27], which plays a significant role in real-world applications.
A case in point occurs in personalized recommendation systems where users' preferences are judged based on only a small amount of their historical data, such as past purchases or online behaviors [28]. Another example arises from crisis management. Since being able to quickly identify and monitor emerging topics during a crisis, e.g., a public health emergency, could substantially support a government or organization in responding appropriately to rapidly changing situations [29].
While there have also been some beneficial attempts to study few-shot learning for topic modeling, they more or less exhibit certain limitations. For instance, Iwata [27] first proposed a straightforward method that aims to learn good, i.e., task-specific priors for latent Dirichlet allocation (LDA) [8] by using neural networks. However, this approach suffers from a lack of expressiveness when dealing with difficult novel tasks. Later on, Duan et al. [30] claimed that embedding-based NTMs [31] are naturally superior in generalizing to new and unseen tasks. By considering word embeddings learned from the training task pool as transferable knowledge, they have shown that effective generalization can be achieved by only learning topic embeddings adaptively. Nevertheless, as the word semantics inevitably change with contexts, the learned static word embeddings may not adapt well to the target task with alien contexts. On the other hand, we have experimentally found that the performance of two representative CTMs [21, 22] is also not competitive enough under resource-poor conditions.
To address the above issues, we propose to learn adaptive word embeddings suitable for each task by fully exploiting the contextual grammar information. Concretely, we first construct a task-specific semantic graph between words using well-established dependency parsing tools [32]. This graph helps depict the precise meaning of each word in the given task, which we then model with a created variational graph autoencoder (VGAE) [33], and the resulting latent representations of words merge the contextual information and are thus semantically tailored to the given task. Furthermore, by imposing a Gaussian mixture prior on the latent space of word, we offer a perspective on learning topics through clustering, i.e., each component of the Gaussian mixture can be viewed as the representation of a topic. Consequently, the adaptive word embeddings tend to be reasonably covered by several clusters, facilitating the discovery of more interpretable topics closely related to the given task.
- To solve the problem of few-shot learning for topic modeling, we propose to learn adaptive word embeddings that are semantically matched to the given task.
- To generate adaptive word embeddings for each task, we innovatively introduce a variational graph autoencoder to learn the latent representations of words from the task-specific semantic graph that captures the contextual syntactic dependencies.
- To mine more interpretable topics related to the given task, we offer a perspective on learning topics through clustering by imposing a Gaussian mixture prior on the word latent space. Besides, we also develop an efficient variational inference scheme to approximate the posteriors of latent variables.
2 Meta-CETM
To clarify the problem definition of few-shot learning for topic modeling, we assume that there are  $C$  training corpora  $\{\mathcal{D}_c\}_{c = 1}^C$  from different domains, with the goal of using them to learn a topic model that generalizes to the test corpus  $\mathcal{D}_{test}$  in a new domain, i.e., the resulting topic model is supposed to quickly adapt to a given test task with a few documents and mine topics related to the new domain. Further, we adopt an episodic training strategy as is conventional in most few-shot learning literature. Specifically, we construct a batch of training tasks  $\{\mathcal{T}^{(i)}\}_{i = 1}^{M}$  to mimic the scenario of giving tasks at test time, so that each task contains only several documents from an arbitrary training corpus. And we denote the bag-of-words (BoW) representations of the task documents as  $\mathbf{X}^{(i)}\in \mathbb{R}^{V\times J}$ , where  $V$  is the vocabulary size and  $J$  is the number of documents in each task. In addition, our method also builds a task-specific dependency graph whose adjacency matrix is represented as  $\mathbf{A}^{(i)}\in \mathbb{R}^{V\times V}$ . To understand the concepts of corpus, task, and document more clearly, see the example in Appendix C.
3The semantic graphs are built with the help of spaCy, a public natural language processing tool to analyse the grammatical structure of sentences, https://spacy.io/.
Figure 1: Overview of the proposed method. The top branch establishes a standard neural topic modeling pipeline, and the bottom branch creates a graph VAE to learn contextualized word embeddings, with a Gaussian mixture prior imposed on the latent space to yield task-specific topic representations. Note that the topic-word matrix is derived based on the probability density assigned to the adaptive word embeddings.
2.2 Generative Model
In this section, we present a Contextualized Embedded Topic Model, dubbed as Meta-CETM, to cope with the problem of topic modeling under resource-poor conditions, the essence of which is to make extra use of the contextual information of given documents to learn dynamic word embeddings that are well adapted to the current task. Concretely, for any given task  $\mathcal{T}^{(i)}$ , in addition to obtaining the BoW representations  $\mathbf{X}^{(i)}$  of its documents, which only imply the word co-occurrence patterns, we also build a semantic graph between words based on the contextual dependency grammars. Thus the corresponding adjacency matrix  $\mathbf{A}^{(i)}$  can be viewed as complementary information on the word semantics. With  $\mathbf{X}^{(i)}$  and  $\mathbf{A}^{(i)}$ , then the goal of the generative model is to model these two types of observations jointly, whose key lies in how to establish a bridge connecting the two.
Inspired by the recently developed VGAEs [33] and embedded topic models (ETMs) [18, 31], we create our generative model based on the compact assumption, i.e., there exists a shared latent space of words that can be mapped to the observation spaces of BoW and semantic graph, respectively. Following this conception, we portray a well-structured generative process via a specific task  $\mathcal{T}^{(i)}$ . First, the latent representations of words  $\mathbf{Z}^{(i)} \in \mathbb{R}^{D \times V}$  are sampled from a prior distribution. Acting as the suitable connection, they are responsible for generating both the adjacency matrix  $\mathbf{A}^{(i)}$  and the BoW  $\mathbf{X}^{(i)}$ . To produce  $\mathbf{A}^{(i)}$ , we use a typical inner product graph decoder formulated as
$$
\mathbf {A} ^ {(i)} \sim \operatorname {B e r} \left(\sigma \left(\mathbf {Z} ^ {(i) ^ {\top}} \mathbf {Z} ^ {(i)}\right)\right) \tag {1}
$$
As for the generation of BoW  $\mathbf{X}^{(i)}$ , the latent representations  $\mathbf{Z}^{(i)}$  (or word embeddings) are mainly used to derive the topic-word matrix. Unlike previous ETMs that usually decompose the topic-word matrix into the inner product of learnable word embeddings and topic embeddings, here we provide a Bayesian perspective on interpreting the topic-word matrix by imposing a Gaussian mixture prior to the word latent space. More precisely, we regard each component of the Gaussian mixture prior as the representation of a topic, such that the word embeddings are naturally generated from different clusters (topics). And the prior probability for the  $v^{th}$  word embedding is given by
$$
p \left(\mathbf {z} _ {v} ^ {(i)}\right) = \sum_ {k = 1} ^ {K} \pi_ {k} \cdot \mathcal {N} \left(\mathbf {z} _ {v} ^ {(i)} \mid \boldsymbol {\mu} _ {k}, \boldsymbol {\Sigma} _ {k}\right), \tag {2}
$$
where  $\pi_k$  is the prior mixing coefficient of the  $k^{th}$  topic and  $\sum_{k} \pi_{k} = 1$ . As such, each topic can be represented as a Gaussian distribution in the word latent space, with a probability density assigned to each word embedding. This provides a natural choice to define the topic-word matrix  $\beta^{(i)} \in \mathbb{R}^{V \times K}$ .
Hence, the distribution of topic  $k$  over the vocabulary  $\beta_k^{(i)}\in \mathbb{R}^V$  can be derived by
$$
\boldsymbol {\beta} _ {k} ^ {(i)} = \operatorname {S o f t m a x} (p (\mathbf {Z} ^ {(i)} | \boldsymbol {\mu} _ {k}, \boldsymbol {\Sigma} _ {k})), \tag {3}
$$
In this way, we expect that semantically similar words will be close in the latent space and generated from the same topic. Moreover, we also posit a context variable  $\pmb{c}^{(i)}$  that summarizes the task-level information about the topic proportion and thus serves as a prior for generating the topic proportion  $\pmb{\theta}_{j}^{(i)}$  of each document. Formally, the complete generation process of our model can be formulated as
1. Generate the adjacency matrix of semantic graph using Eq. 1;
2. For the  $v^{th}$  word appeared in the task  $\mathcal{T}^{(i)}$ :
(a) Draw its topic assignment  $y_{v}^{(i)} \sim \mathrm{Cat}(\pi)$ ;
(b) Draw its word embedding  $\mathbf{z}_v^{(i)}\sim \mathcal{N}(\pmb{\mu}_{y_v^{(i)}},\pmb{\Sigma}_{y_v^{(i)}})$
3. Compute task-specific topic-word matrix  $\beta^{(i)}$  based on Eq. 3;
4. Draw task-level context variable  $c^{(i)} \sim \mathcal{N}(0, a^2 I)$ ;
5. For the  $j^{th}$  document in the task  $\mathcal{T}^{(i)}$ :
(a) Draw its topic proportion  $\pmb{\theta}_{j}^{(i)}\sim \mathcal{LN}(\pmb{c}^{(i)},b^{2}\pmb {I})$
(b) For the  $n^{th}$  word in the  $j^{th}$  document:
i. Draw its topic assignment  $e_{jn}^{(i)}\sim \mathrm{Cat}(\pmb{\theta}_j^{(i)})$
ii. Generate the word count  $x_{jn}^{(i)}\sim \mathrm{Cat}(\beta_{e_{jn}^{(i)}}^{(i)})$
where  $\operatorname{Cat}(), \mathcal{N}(), \operatorname{Ber}(),$  and  $\mathcal{LN}()$  denote categorical, Gaussian, Bernoulli, and logistic-normal distributions, respectively.  $\sigma$  is the Sigmoid() operation<sup>4</sup>.  $a$  and  $b$  are both hyper-parameters. Table 4 in Appendix B gives a list of key notations used in this paper.
2.3 Variational Inference Algorithm
Observing a task  $\mathcal{T}^{(i)}$  with data  $\mathbf{X}^{(i)}$ , the goal of inference network is to approximate the posterior distributions over the latent variables,  $\{\pmb{\theta}_j^{(i)},\pmb {c}^{(i)},\mathbf{Z}^{(i)}\}$ , and the parameters of GMM,  $\{\pi_k^{(i)},\pmb {\mu}_k^{(i)},\pmb {\Sigma}_k^{(i)}\}$ .
Document-specific latent variable inference. To model the uncertainty of topic proportions, we define the variational posterior distribution for  $\theta_{j}^{(i)}$ , and a residual multi-layer perception (MLP) is employed to learn distribution parameters. To be specific,
$$
q \left(\boldsymbol {\theta} _ {j} ^ {(i)} \mid \boldsymbol {x} _ {j} ^ {(i)}, \boldsymbol {c} ^ {(i)}\right) = \mathcal {L N} \left(\boldsymbol {\mu} _ {\boldsymbol {\theta} _ {j} ^ {(i)}}, \boldsymbol {\Sigma} _ {\boldsymbol {\theta} _ {j} ^ {(i)}}\right)
$$
$$
\boldsymbol {\mu} _ {\boldsymbol {\theta} _ {j} ^ {(i)}}, \boldsymbol {\Sigma} _ {\boldsymbol {\theta} _ {j} ^ {(i)}} = \operatorname {R e s M L P} \left(\mathbf {h} _ {j} ^ {(i)} + \boldsymbol {c} ^ {(i)}\right) \tag {4}
$$
$$
\mathbf {h} _ {j} ^ {(i)} = \mathrm {M L P} _ {\mathbf {h}} (\mathbf {x} _ {j} ^ {(i)})
$$
where  $\pmb{\mu}_{\pmb{\theta}_j^{(i)}}$  and  $\pmb{\Sigma}_{\pmb{\theta}_j^{(i)}}$  are logistic Gaussian distribution deterministic parameters depending on the latent mean  $\pmb{c}^{(i)}$  and the document latent representation  $\mathbf{h}_j^{(i)}$  of  $\mathbf{x}_j^{(i)}$ .
Task-specific latent variable inference. For topic proportion mean  $c^{(i)}$ , we apply Gaussian distribution to approximate the variational posterior, whose parameters mean vectors and covariance matrices are derived through encoding all documents in task  $\mathcal{T}^{(i)}$  with attention mechanism [34], written as
$$
q \left(\boldsymbol {c} ^ {(i)} \mid \mathbf {X} ^ {(i)}\right) = \mathcal {N} \left(\boldsymbol {\mu} _ {\boldsymbol {c} ^ {(i)}}, \boldsymbol {\Sigma} _ {\boldsymbol {c} ^ {(i)}}\right); \boldsymbol {\mu} _ {\boldsymbol {c} ^ {(i)}}, \boldsymbol {\Sigma} _ {\boldsymbol {c} ^ {(i)}} = \operatorname {A t t n} \left(\mathbf {H} ^ {(i)}\right); \mathbf {H} ^ {(i)} = \mathrm {M L P} \left(\mathbf {X} ^ {(i)}\right), \tag {5}
$$
where  $\mathbf{H}^{(i)}$  is the latent indication of the task, and Attn() is the attention mechanism to capture relationships of  $J$  documents in task  $\mathcal{T}^{(i)}$ .
<sup>4</sup>If a weighted graph  $\mathbf{A}^{(i)}$  is considered, then the Poisson likelihood can be employed with  $\sigma$  set to the Softplus() operation.
Additionally, to ensure the modeling flexibility, we design a simple inference network consisting of a two-layer GCN [35] to infer the latent representations  $\mathbf{Z}^{(i)}$  of words following [33]:
$$
q \left(\mathbf {Z} ^ {(i)} \mid \mathbf {A} ^ {(i)}, \mathbf {E} ^ {(i)}\right) = \mathcal {N} \left(\mathbf {Z} ^ {(i)} \mid \boldsymbol {\mu} _ {\mathbf {Z}} ^ {(i)}, \boldsymbol {\Sigma} _ {\mathbf {Z}} ^ {(i)}\right) \tag {6}
$$
$$
\boldsymbol {\mu} _ {\mathbf {Z}} ^ {(i)}, \boldsymbol {\Sigma} _ {\mathbf {Z}} ^ {(i)} = \operatorname {G C N} (\mathbf {A} ^ {(i)}, \mathbf {E} ^ {(i)})
$$
where  $\mathbf{E}^{(i)}$  is the initialized word features and  $\mathbf{A}^{(i)}$  is derived through a neural parser on  $\mathbf{X}^{(i)}$ . Figure 1 illustrates an overview of our approach, including the variational inference network.
In this paper, for simplicity, we denote  $\Psi$  as network parameters of both the encoder and the decoder.
Expectation Maximization for solving  $\pmb{\mu}_k^{(i)}$  and  $\pmb{\Sigma}_k^{(i)}$ . As discussed before,  $\pmb{\mu}_k^{(i)}$  and  $\pmb{\Sigma}_k^{(i)}$  are task-specific parameters of Gaussian mixture distribution, which are not learnable variables, and we do not have analytic solutions for Maximum Likelihood Estimation (MLE) of GMM containing the non-differentiable sampling process [36]. To approximate the posterior, we resort to Expectation Maximization (EM) [37] algorithm to optimize the parameters, formulated as:
$$
\mathrm {E} - \text {S t e p} Q _ {v} ^ {(i)} := p (y _ {v} ^ {(i)} = k | \mathbf {z} _ {v} ^ {(i)}) = \frac {\pi_ {k} ^ {(i)} \mathcal {N} (\mathbf {z} _ {v} ^ {(i)} ; \boldsymbol {\mu} _ {k} ^ {(i)} , \boldsymbol {\Sigma} _ {k} ^ {(i)})}{\sum_ {k} \pi_ {k} ^ {(i)} \mathcal {N} (\mathbf {z} _ {v} ^ {(i)} ; \boldsymbol {\mu} _ {k} ^ {(i)} , \boldsymbol {\Sigma} _ {k} ^ {(i)})}
$$
$$
\mathrm {M} - \text {S t e p} \quad \boldsymbol {\mu} _ {k} ^ {(i)} := \frac {\sum_ {v} Q _ {v} ^ {(i)} \mathbf {z} _ {v} ^ {(i)}}{\sum_ {v} Q _ {v} ^ {(i)}} \tag {7}
$$
$$
\Sigma_ {k} ^ {(i)} := \frac {\sum_ {v} Q _ {v} ^ {(i)} (\mathbf {z} _ {v} ^ {(i)} - \boldsymbol {\mu} _ {k} ^ {(i)}) (\mathbf {z} _ {v} ^ {(i)} - \boldsymbol {\mu} _ {k} ^ {(i)}) ^ {T}}{\sum_ {v} Q _ {v} ^ {(i)}}
$$
$$
\pi_ {k} ^ {(i)} := \frac {\sum_ {v} Q _ {v} ^ {(i)}}{\sum_ {k} \sum_ {v} Q _ {v} ^ {(i)}}.
$$
Since topic  $k$  is sampled from the Uniform distribution for each task, the mixing coefficients are initialized as  $\frac{1}{K}$ . We initialize  $\pmb{\mu}_k^{(i)}$  and  $\pmb{\Sigma}_k^{(i)}$  as the average of latent variables  $\mathbf{Z}^{(i)}$  and the identity matrix  $I$ , respectively. Here, we only display the final updating formulas. The detailed derivation processes for E-step and M-step are presented in Appendix E.2.
2.4 Training Objective and Optimization
By Jensen's inequality, the evidence lower bound (ELBO) of each task can be derived as
$$
\begin{array}{l} \mathcal {L} _ {E L B O} = \sum_ {j = 1} ^ {J} \mathbb {E} _ {Q} \left[ \log p \left(\mathbf {x} _ {j} ^ {(i)} \mid \boldsymbol {\theta} _ {j} ^ {(i)}, \mathbf {Z} ^ {(i)}\right) \right] + \sum_ {j = 1} ^ {J} \mathbb {E} _ {Q} \left[ \log \frac {p \left(\boldsymbol {\theta} _ {j} ^ {(i)} \mid \boldsymbol {c} ^ {(i)}\right)}{q \left(\boldsymbol {\theta} _ {j} ^ {(i)} \mid \mathbf {x} _ {j} ^ {(i)} , \boldsymbol {c} ^ {(i)}\right)} \right] \tag {8} \\ + \mathbb {E} _ {Q} \left[ \log \frac {p (\boldsymbol {c} ^ {(i)})}{q (\boldsymbol {c} ^ {(i)} \mid \boldsymbol {X} ^ {(i)})} \right] + \mathbb {E} _ {Q} \left[ \log p (\boldsymbol {A} ^ {(i)} \mid \boldsymbol {Z} ^ {(i)}) \right] + \mathbb {E} _ {Q} \left[ \log \frac {p (\boldsymbol {Z} ^ {(i)})}{q (\boldsymbol {Z} ^ {(i)} \mid \boldsymbol {A} ^ {(i)} , \boldsymbol {E} ^ {(i)})} \right] \\ \end{array}
$$
$$
Q = \prod_ {j = 1} ^ {J} q \left(\boldsymbol {\theta} _ {j} ^ {(i)} \mid \mathbf {X} ^ {(i)}, \boldsymbol {c} ^ {(i)}\right) q \left(\boldsymbol {c} ^ {(i)} \mid \mathbf {X} ^ {(i)}\right) q \left(\mathbf {Z} ^ {(i)} \mid \mathbf {A} ^ {(i)}, \mathbf {E} ^ {(i)}\right) \tag {9}
$$
is the variational joint distribution. The first and the fourth terms in Eq. 8 are the reconstruction errors for document BoW and the graph adjacency matrix, respectively. The remaining three terms are all Kullback-Leibler (KL) divergence to constrain the distance between the prior distribution and the variational posterior distribution. Owing to the space limit, we only present the final formulas for  $\mathcal{L}_{ELBO}$  here. The detailed derivations, the training algorithm, and the meta-testing algorithm can be found in Appendix E.1, Alg. 1, and Alg. 2, respectively.
3 Experiments and Analysis
Datasets. We conducted experiments on four widely used textual benchmark datasets, specifically 20Newsgroups (20NG) [38], Yahoo Answers Topics (Yahoo) [39], DBpedia (DB14) [40], and Web of Science (WOS) [41].
Table 1: PPL results on four datasets. "5" and "10" denote the number of documents in each task. *Since ProdLDA and ETM are not designed for few-shot learning, we run their meta versions where parameters are optimized using model-agnostic meta-learning (MAML) strategy [44].
Methods	20NG	Yahoo	DB14	WOS
5	10	5	10	5	10	5	10
LDA [42]	4021±1528	3502±1277	4476±1544	4028±1097	4410±1918	3697±1747	3439±671	3246±461
PFA [12]	3463±1452	3150±1119	3257±1328	3122±1040	3443±1937	3170±1562	3113±819	3431±830
ProLDA [43]	4853±1034	4523±817	5765±1104	5378±826	5477±846	5297±740	4311±469	4220±392
ETM [18]	3192±895	3107±671	2868±909	2817±620	3217±1960	3054±1539	3135±704	3310±455
MAML-ProdLDA*	4292±1123	4355±997	4354±1369	4250±919	4844±1337	4678±1119	4117±462	4068±332
MAML-ETM*	3849±1064	3725±841	3653±1081	3642±776	4448±2737	4279±2301	3483±4044	3277±644
Meta-SawETM [30]	2872±869	2984±740	2365±934	2487±756	2047±1374	1914±1009	2031±445	2253±315
CombinedTM [21]	2660±659	2595±625	2700±590	2674±575	1851±767	1774±731	2562±633	2648±658
ZeroShotTM [22]	2904±851	2569±663	2822±732	2795±721	1938±758	1835±739	2863±704	2775±558
Meta-CETM	954±543	1170±606	1074±442	1219±455	802±571	1084±643	1293±542	1528±218
**Baseline methods.** Our model is compared with exemplary baseline methods, including probabilistic topic models, state-of-the-art NTMs and CTMs. Specifically, we conduct experiments of the following methodologies under the document-limited setting: 1) LDA [42]; 2) PFA [12]; 3) LDA with Products of Experts (ProdLDA) [43], which replaces the mixture model in LDA with products of experts and updates parameters using AVI; 4) Embedded Topic Model (ETM) [18], an NTM incorporating word embeddings and learning with AVI. For a fair comparison, we also consider the variations of ProdLDA and ETM under the few-shot setting, referred to 5) MAML-ProdLDA and 6) MAML-ETM, where the parameters are optimized through MAML algorithm [44]. In addition, we include a model-based hierarchical NTM, 7) Meta-SawETM [30], but we only apply their single-layer model. Moreover, another two CTMs 8) CombinedTM [21] and 9) ZeroShotTM [22] are also compared, both of which contain contextualized Sentence BERT [45] embeddings as the model input.
3.2 Experimental results and analysis
In this section, we evaluate the predictive performance, topic quality, and classification performance of our model through an extensive series of experiments. Note that in all tables, we have highlighted the best and runner-up results in boldface and with an underline, respectively. Our code is available at https://github.com/NoviceStone/Meta-CETM.
3.2.1 Per-holdout-word perplexity
Following the practice in Meta-SawETM [30], we adopt the per-holdout-word perplexity (PPL) [46] to measure the predictive performance of our model. Specifically, for each task composed of several documents,  $80\%$  of the tokens in the BoWs are randomly chosen to form a support set  $D_{test}^{S}$ , which is used to adapt to a task-specific topic-word matrix  $\beta$ , and the remaining  $20\%$  word tokens are held out to form the query set  $D_{test}^{Q}$  with data  $\mathbf{Y}$ . Then, the PPL can be calculated as
$$
\exp \left\{- \frac {1}{y _ {\cdot \cdot}} \sum_ {v = 1} ^ {V} \sum_ {n = 1} ^ {N} y _ {v n} \ln \frac {\sum_ {s = 1} ^ {S} \sum_ {k = 1} ^ {K} \beta_ {v k} ^ {s} \theta_ {k n} ^ {s}}{\sum_ {s = 1} ^ {S} \sum_ {v = 1} ^ {V} \sum_ {k = 1} ^ {K} \beta_ {v k} ^ {s} \theta_ {k n} ^ {s}} \right\}, \tag {10}
$$
where  $S$  is the total number of collected samples and  $y_{\cdot \cdot} = \sum_{v = 1}^{V}\sum_{n = 1}^{N}y_{vn}$