SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Model settings. For all compared methods, we set the number of topics as 10. And for all NTMs, the hidden layers size of the encoder is set to 300. For all embedding-based topic models, i.e., ETM, MAML-ETM, Meta-SawETM and our Meta-CETM, we load pretrained GloVe word embeddings [47] as the initialization for a fair comparison. Finally, We train our model using the Adam optimizer [48] with a learning rate of  $1 \times 10^{-2}$  for 10 epochs on an NVIDIA GeForce RTX 3090 graphics card.
Results. In Table 1, we list the PPL of ten compared methods on four datasets. It can be noticed although LDA and PFA are both traditional probabilistic topic models, PFA presents better results than LDA. By utilizing MAML to learn parameter initializations, MAML-ProdLDA performs better than ProdLDA while MAML-ETM exhibits poorer results than ETM. This can be attributed to that ETM possesses more parameters than ProdLDA, and calculating the gradients in a high-dimensional space with only a few documents is difficult for MAML-ETM. Applying Weibull distribution to model the latent representation for documents and employing task-specific variable designs for both
Figure 2: Topic diversity results (top row) and topic coherence results (bottom row) on four datasets of six compared methods. The number of documents for each task is 10.
topic proportions and topic matrices, Meta-SawETM acts superior to other topic models. Moreover, incorporating contextualized embeddings as input enables CombinedTM and ZeroShotTM to present competitive performance. Furthermore, it is noteworthy that Meta-CETM achieves the lowest PPL among all these methods, indicating the excellent predictive performance of our model.
3.2.2 Topic quality
In this part, we evaluate the topic quality of different methods in terms of the topic diversity (TD) [49], defined as the percentage of unique words in the top 10 words of all topics, and the topic coherence (TC), which measures the average Normalized Pointwise Mutual Information (NPMI) [50, 51] values to count word co-occurrences<sup>5</sup>. The experimental settings are the same as in Sec. 3.2.1. For each task, the number of documents is 10 for all datasets. The results are displayed in Fig. 2 and it can be notably and interestingly observed that MAML-ProdLDA achieves fairly "perfect" TD results, but it shows the worst TC performances among compared methods. Such inconsistency is brought by MAML-ProdLDA's concentration on a large amount of universal and frequently-occurring words when training on the base data, hindering it from extracting informative topics given the meta-test task. Embodying the embedding design for words and topics, MAML-ETM and Meta-SawETM are more likely to mine context-related topics than MAML-ProdLDA. Besides, as the representatives of CTM, CombinedTM and ZeroShotTM equipped with BERT embeddings present more comprehensive performances than previous topic models. From the TC perspective, our Meta-CETM yields the most favorable results among the six methods, indicating the fast adaptability in discovering interpretable topics with limited documents.
3.2.3 Few-shot document classification
To further validate that our model is capable of learning topics which are highly adapted to the task, we undertake experiments on few-shot document classification.
Model settings. As in Meta-SawETM [30], we compare our Meta-CETM with classical meta-learning algorithms under different architectures. Specifically, we design a three-layer feedforward network as MLP structure and three-layer 1-dimensional convolutions followed by batch normalization, ReLU() activation and max-pooling operation as CNN. For meta-learning methods, we apply MAML [44] to learn parameter initializations, and prototypical network (PROTO) [52] to learn an embedding space and minimize the distance between the clustering centroids and the samples. Besides, we adopt two fine-tuning manners, named FT [53] and  $\mathbf{FT}^*$ , to update the classifier parameters and all parameters of the model, respectively. Single-layer HNS-SawETM and single-layer Meta-SawETM in [30] are compared as well. Additionally, we investigate the CombinedTM [21] and ZeroShotTM [22] to evaluate their few-shot classification performance. Different from PPL evaluation in Sec. 3.2.1, the support set and the query set for classification are sourced from two batches of documents.
5The NPMI score is computed through thegensim package in https://radimrehurek.com/gensim/ models/coherencemodel.html.
Table 2: 5-way 5-shot and 5-way 10-shot few-shot document classification results on all four datasets. *denotes all parameters of the model are fine-tuned.
Methods	20NG	DB14	Yahoo	WOS
Rep.	Alg.	5 shot	10 shot	5 shot	10 shot	5 shot	10 shot	5 shot	10 shot
MLP	MAML [44]	32.01	36.20	50.20	60.30	45.42	51.00	37.77	40.43
PROTO [52]	35.20	38.30	54.13	57.16	50.01	56.16	39.61	41.46
FT [53]	29.70	33.04	51.11	53.83	48.59	53.06	36.52	37.22
FT*	38.87	48.52	71.12	77.94	50.73	56.74	45.02	51.20
CNN	MAML [44]	34.08	45.40	66.28	75.96	48.81	56.50	47.28	57.32
PROTO [52]	39.86	49.71	78.58	81.01	53.16	63.66	59.05	67.75
FT [53]	45.70	53.63	74.68	80.75	56.78	66.04	54.68	63.39
FT*	44.53	51.92	72.49	80.07	53.28	52.56	51.42	61.98
HNS-SawETM [30]	39.37	43.78	65.93	71.08	52.35	57.86	42.09	56.91
Meta-SawETM [30]	39.19	45.83	67.20	72.31	52.45	60.58	43.39	57.44
CombinedTM [21]	46.17	52.73	68.42	73.26	57.94	64.75	56.16	65.97
ZeroShotTM [22]	46.65	52.08	71.93	76.09	58.12	66.21	58.50	66.10
Meta-CETM	50.57	58.47	76.85	79.34	63.84	72.67	61.47	67.62
Classification strategy. For typical few-shot learning algorithms, we follow the convention to train parameters of the feature extractor in a supervised manner. At meta-test stage, we use the support set (i.e., a few labeled examples) to adapt to a task-specific classifier and compute the accuracy on the query set. As for topic models, where no dedicated classifiers are available, we first use the support set to learn a group of class-specific topic-word matrices, then for each document in the query set, we calculate its reconstruction error as the basis for classification.
Results. The classification results $^6$  are listed in the Table 2. As we can see, the CNN architecture outperforms MLP under the same algorithms, which can be attributed to CNN's unique inductive bias of the locality. Furthermore, PROTO surpasses MAML by a large margin in most cases, indicating that a good embedding space is more useful for classification than favorable parameter initializations. For the CNN architecture, fine-tuning the classifier only (FT) and fine-tuning all parameters  $(\mathrm{FT}^*)$  make slight differences; but for MLP, we observe a considerable performance boost by updating all parameters of the network over the FT algorithm. We postulate that the classification results of MLP are more susceptible to the variation of feature extractor parameters due to its linear structure. Additionally, CombinedTM and ZeroShotTM, incorporating contextualized representations, outperforms HNS-SawETM and Meta-SawETM by learning class-specific topic-word matrices more effectively. With the design of adaptive word embeddings, our Meta-CETM not only achieves much better results than previous topic models, but also is comparable to few-shot learning algorithms particularly designed for supervised learning.
3.2.4 Embedding space visualization
Figure 3: Visualization of the adapted embedding space for (a) MAML-ETM, (b) Meta-SawETM and (c) Meta-CETM. The small grey points represent word embeddings, and the big blue points denote topic embeddings for MAML-ETM, topic means for Meta-SawETM and our Meta-CETM. For Meta-SawETM and our Meta-CETM, the ellipse coverages represent topic covariances. For MAML-ETM, the ellipse coverages are the areas of top words. The target task is sampled from the sub-topic "rec.sport.hockey" of 20NG dataset.
The classification experiments for all compared methods are conducted ten times.
In addition to quantitative results, we also qualitatively analyzed the effectiveness of our model by visualizing the adapted embedding space, as shown in Figure 3. It can be seen that both MAML-ETM and Meta-SawETM learn topics that are not highly relevant to the target task, as most of their top words (e.g., windows and dos) are inherited from the base training corpora, while those informative words (e.g., players and hockey) associated with the target task are away from the topic embeddings. By contrast, in the embedding space learned by Meta-CETM, the adapted Gaussian distributions, i.e., topics reasonably cover almost all words closely related to the target task, indicating that our model can achieve successful adaptation and discover more interpretable topics.
Furthermore, we investigated whether the adaptive word embeddings generated by our model effectively reflect the contextual information of each given task. As illustrated in Figure 4, the word embedding of "apple" adapted from a task related to company is surrounded by words like "mobile" and "amazon", whereas in another task concerning plant, the embedding of "apple" is closer to the words such as "fruit" and "trees". This phenomenon suggests that Meta-CETM produces word embeddings that are semantically well-matched to the context of the target task.
Figure 4: The adaptive contextual word embeddings learned by our Meta-CETM on DB14 dataset [40]. Left: The local embedding space of a task from the "Company" domain, Right: The local embedding space of a task in the "Plant" domain.
To explore the effectiveness of key designs in our model, we conduct a series of ablation experiments on all four datasets by removing each designed module separately. The numerical results of perplexity, topic diversity, and topic coherence are listed in Table 3, where ETM [18] is chosen as the baseline method. From the results presented below, we have the following observations:
$i)$  Despite achieving the best TD results, ETM obtains the worst PPL and TC scores, which can be attributed to its tendency to extract frequently occurring themes from training, hindering its fast adaptability to meta-test sets with only several documents.  $ii)$  To capture the relations of texts and model the task information, in Section 2.2, we posit a context-specific variable  $c^{(i)}$  as the prior of document-level topic proportion  $\theta_{j}^{(i)}$ . It can be found with  $c^{(i)}$ , our Meta-CETM achieves much lower PPL and higher TC, demonstrating the efficacy of task-specific variables.  $iii)$  For ablation results without the Graph VAE module, we replace it with a Graph AE rather than discarding it completely. Besides, for Meta-CETM without the GMM prior design, we replace it with the standard Gaussian distribution prior rather than imposing no prior completely. From the quantitative results shown
Table 3: Ablation study on four datasets. The number of texts in each task is 10. “√” means we add the corresponding design into ETM [18], which is chosen as the baseline.
context variable	Graph VAE	GMM prior	20NG	DB14
PPL	TD	TC	PPL	TD	TC
ETM	3107	0.8395	-0.8437	3054	0.8106	-0.8719
✓	✓	1964	0.8031	-0.4301	1682	0.7441	-0.4917
✓	✓	1255	0.7983	-0.4169	1131	0.7210	-0.5025
✓	✓	1361	0.6538	-0.4688	1276	0.6562	-0.6677
Meta-CETM	✓	✓	✓	1170	0.8154	-0.3701	1084	0.7475	-0.4783
context variable	Graph VAE	GMM prior	Yahoo	WOS
PPL	TD	TC	PPL	TD	TC
ETM	2817	0.8851	-0.8913	3310	0.9286	-0.9785
✓	✓	1906	0.7243	-0.5097	2023	0.9141	-0.5420
✓	✓	1316	0.7612	-0.4860	1304	0.8387	-0.5262
✓	✓	1271	0.5847	-0.5503	1389	0.7018	-0.5587
Meta-CETM	✓	✓	✓	1219	0.7886	-0.4639	1293	0.8667	-0.5177
Figure 5: Illustration of the advantage of using a GMM prior. The adapted embedding space of our model by using (a) no prior, i.e., Graph VAE is replaced with a vanilla graph AE, and the topic embeddings are learnable point vectors, (b) a standard normal prior, i.e., the topic embeddings are learnable Gaussian distributions which are constrained by KL divergence with the standard normal distribution, (c) a GMM prior. The target task is sampled from the sub-topic "rec.sport.hockey" of the 20NG dataset. Room in for better visual effects.
in the table, it can be found compared with Gaussian prior in vanilla VAE model, GMM priors encourage the semantically coherent words to be allocated to the same topic, leading to higher TC scores and lower PPL. Further, we visualize the embedding space in Figure 5 to demonstrate the benefits of GMM prior.
To discover a group of topics from a batch of documents, probabilistic topic models have been developed in recent years, including hierarchical topic models [54, 14, 46, 55], NTMs [56, 15], embedded topic models [18, 31], and CTMs [22, 21]. Besides, some works are proposed to incorporate knowledge into the modeling process [57-59] and some researchers apply optimal transport to measure the distances between topics and words or documents [60, 61]. From the perspective of modelling word embeddings with Gaussian distribution, Vilnis etc. proposed a novel density-based mapping method [62] and achieved promising performances. Recently, some works [30, 27] aimed at topic modelling under the few-shot setting and proposed to obtain a group of task embeddings to adaptively explore topics within only a few documents in a model-based meta-learning fashion. However, their method cannot address the multiple meanings of one word issue, a prevalent issue in practical document analysis, which is addressed through the introduction of dependency graph and the GMM prior distribution in our work.
In this paper, we propose a novel NTM, Meta-CETM, to address the fast adaptation problem in document analysis under low-resource regimes. Specifically, we construct a task-specific graph to obtain context-related word embeddings. Then we introduce the graph VAE with Gaussian mixture prior to model the word representations and topic embeddings, which are optimized through the EM algorithm. We also propose the task-specific prior for topic proportions. Through extensive experiments and illustrations, we demonstrate the superior performance of our model in solving the adaptation problem in topic modeling.
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006; in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22160; in part by the 111 Project under Grant B18039; in part by the Fundamental Research Funds for the Central Universities; in part by the Innovation Fund of Xidian University under Grant YJSJ23016.
[1] Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences, 101(suppl_1):5228-5235, 2004.
[2] Andrew McCallum, Xuerui Wang, and Andres Corrada-Emmanuel. Topic and role discovery in social networks with experiments on enron and academic email. Journal of artificial intelligence research, 30:249–272, 2007.
[3] David Mimno, Hanna Wallach, Jason Naradowsky, David A Smith, and Andrew McCallum. Polylingual topic models. In Proceedings of the 2009 conference on empirical methods in natural language processing, pages 880-889, 2009.
[4] Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 2, pages 524-531. IEEE, 2005.
[5] Simon Rogers, Mark Girolami, Colin Campbell, and Rainer Breitling. The latent process decomposition of cdna microarray data sets. IEEE/ACM transactions on computational biology and bioinformatics, 2(2):143-156, 2005.
[6] David Mimno. Reconstructing pompeian households. arXiv preprint arXiv:1202.3747, 2012.
[7] Jordan Boyd-Graber, Yuening Hu, David Mimno, et al. Applications of topic models. Foundations and Trends® in Information Retrieval, 11(2-3):143-296, 2017.
[8] David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993-1022, 2003.
[9] David Blei and John Lafferty. Correlated topic models. Advances in neural information processing systems, 18:147, 2006.
[10] David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 113-120, 2006.
[11] Jon Mcauliffe and David Blei. Supervised topic models. Advances in neural information processing systems, 20, 2007.
[12] Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statistics, pages 1462-1471. PMLR, 2012.
[13] Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested Chinese restaurant process. Advances in neural information processing systems, 16, 2003.
[14] John Paisley, Chong Wang, David M Blei, and Michael I Jordan. Nested hierarchical Dirichlet processes. IEEE transactions on pattern analysis and machine intelligence, 37(2):256-270, 2014.
[15] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference on machine learning, pages 1727-1736. PMLR, 2016.
[16] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488, 2017.
[17] Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull hybrid autoencoding inference for deep topic modeling. arXiv preprint arXiv:1803.01328, 2018.
[18] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453, 2020.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
[54] David M Blei, Thomas L Griffiths, and Michael I Jordan. The nested Chinese restaurant process and bayesian nonparametric inference of topic hierarchies. Journal of the ACM (JACM), 57(2): 1-30, 2010.
[55] Mingyuan Zhou, Yulai Cong, and Bo Chen. The Poisson gamma belief network. Advances in Neural Information Processing Systems, 28:3043-3051, 2015.
[56] Dallas Card, Chenhao Tan, and Noah A Smith. A neural framework for generalized topic models. arXiv preprint arXiv:1705.09296, 2017.
[57] Zhibin Duan, Yishi Xu, Bo Chen, Dongsheng Wang, Chaojie Wang, and Mingyuan Zhou. Topic-net: Semantic graph-guided topic discovery. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[58] Dongsheng Wang, Yi.shi Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen, and Mingyuan Zhou. Knowledge-aware bayesian deep topic model. In Advances in Neural Information Processing Systems, 2022.
[59] Yi.shi Xu, Dongsheng Wang, Bo Chen, Ruiying Lu, Zhibin Duan, and Mingyuan Zhou. Hyperminer: Topic taxonomy mining with hyperbolic embedding. In Advances in Neural Information Processing Systems, 2022.
[60] He Zhao, D Phung, V Huynh, T Le, and W Buntine. Neural topic model via optimal transport, 2020.
[61] Viet Huynh, He Zhao, and Dinh Phung. Otlda: A geometry-aware optimal transport approach for topic modeling. Advances in Neural Information Processing Systems, 33:18573-18582, 2020.
[62] Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623, 2014.
A Discussions
A.1 Limitations and Future Work
In this paper, we propose a method to improve existing embedded topic models (ETMs) under the low-resource settings by introducing the task-specific graph with the Gaussian mixture prior. The main limitation of our work could be the reliance on the pre-trained parsing tools. Specifically, we construct the semantic graph according to the syntactic structure of the documents with the public library,  $\mathsf{spaCy}^7$ . However, some other approaches such as the sliding window method can also be used to construct the graph, which is not relied on training. Besides, although strong performances, our framework is still trained and evaluated on the same dataset. While, in the current large models era, pre-training on a large set and then perform few-shot learning or even zero-shot learning attracts researchers' attention and interest, especially such Bayesian models. However, this is beyond the scope of this paper and we will conduct a thorough investigation of this issue in future work.
A.2 Broader impact
Our work builds a novel and advanced topic modeling technique under few-shot settings. For example, at the very beginning of the Covid-19, there were relatively few cases of infection in all regions, and the reports of related medical diagnoses were highly limited. Under such circumstances, few-shot topic models can help us extract key information from limited resources, thus facilitating us to take appropriate and efficient preventive and control measures. On the other hand, the emergence of ChatGPT by OpenAI has attracted a great deal of attention, while in this paper, what we explore is a robust model with generalization capabilities given limited text data, rather than training from a very large corpus. Potential negative societal impact of our work could arise from malicious intent in changing model's behavior by injecting deliberate human prejudice, which may harm the fairness of the community. However, we hope our work is utilized to enable more research and applications primarily from the originality of benefiting the community development.
B Key Notations
In Table 4, we list the key notations, descriptions and corresponding dimensions used in this paper.
Table 4: Notations used in the paper.
Symbol	Dimensionality	Description
M	-	number of total training tasks
J	-	number of documents in each task
K	-	number of topics in each task
V	-	number of vocabulary terms, shared across tasks
D	-	dimensionality of the word latent space
T(i)	-	theithtraining task
X(i)	RVI×J	the BoWs representations for documents in theithtask
H(i)	R300×J	the deterministic hidden features of BoWs X(i)
c(i)	RK	context variable that summarizes the topic proportion information
θi(j)	RK	topic proportion of the jth in theithtask
βi(i)	RV×K	topic-word matrix for theithtask
A(i)	RV×V	the adjacency matrix of dependency graph for theithtask
ev(i)	RD	initialized features of the vthword appeared in theithtask
zvi(i)	RD	adaptive embedding of the vthword appeared in theithtask
πk(i)	-	coefficient of the kthGaussian component for theithtask
μk(i)	RD	mean of the kthGaussian component for theithtask
Σk(i)	RD×D	covariance of the kthGaussian component for theithtask
<sup>7</sup>https://spacy.io/
Figure 6: An illustration of word sense variation caused by different contexts. The task  $i$  is sampled from a corpus about "hardware", and the task  $j$  is sampled from a corpus related to "autos".
C An Illustration of Our Settings
In Section 2.2, we touched on the concepts of corpus, task, and document to describe our problem setting, which is a bit messy to follow. Here, we clarify these concepts through a concrete example so that the reader could understand the problem setting of few-shot learning more easily. Considering the 20NG [38] dataset:
A "corpus" refers to a collection of documents belonging to the same category so that 20NG consists of 20 corpora, each of which contains documents from one of the 20 classes.
A "task" is a smaller unit than a "corpus" that only comprises a few related documents, typically less than 100 documents. Consequently, we could sample a number of tasks from each training corpus.
(we select 12 out of the 20 corpora for training)
Then our goal is to utilize these sampled tasks to train a generalizable topic model that can efficiently adapt to a new task from the test corpus (the remaining 8 corpora are used for testing). In addition, for each task at the testing stage, we split its documents into two parts, one for fine-tuning or retraining the topic model, called the support set, and the other for evaluating the model's performance, called the query set. Note that we do not design different generative processes for the corpus documents versus the task documents. In essence, our proposed Meta-CETM only characterizes the generative process of the task documents by jointly modeling the syntactic graph  $\mathbf{A}$  and the observed BoW  $\mathbf{X}$  in each task. In Fig. 6, we visualize the task and the corresponding unweighted dependency graph  $\mathbf{A}$ .
D Algorithms for training and testing