Context-guided Embedding Adaptation for Effective Topic Modeling in Low-Resource Regimes
Authors: Yishi Xu
Jianqiao Sun
Yudi Su
Xinyang Liu
Zhibin Duan
Bo Chen
Mingyuan Zhou
Venue: NeurIPS 2023 poster (2023)
Keywords: Few-shot generative model; topic modeling;

Abstract: Embedding-based neural topic models have turned out to be a superior option for low-resourced topic modeling. However, current approaches consider static word embeddings learnt from source tasks as general knowledge that can be transferred directly to the target task, discounting the dynamically changing nature of word meanings in different contexts, thus typically leading to sub-optimal results when adapting to new tasks with unfamiliar contexts. To settle this issue, we provide an effective method that centers on adaptively generating semantically tailored word embeddings for each task by fully exploiting contextual information. Specifically, we first condense the contextual syntactic dependencies of words into a semantic graph for each task, which is then modeled by a Variational Graph Auto-Encoder to produce task-specific word representations. On this basis, we further impose a learnable Gaussian mixture prior on the latent space of words to efficiently learn topic representations from a clustering perspective, which contributes to diverse topic discovery and fast adaptation to novel tasks. We have conducted a wealth of quantitative and qualitative experiments, and the results show that our approach comprehensively outperforms established topic models.

Context-guided Embedding Adaptation for Effective Topic Modeling in Low-Resource Regimes
Yishi Xu, Jianqiao Sun; Yudi Su, Xinyang Liu, Zhibin Duan, Bo Chen National Key Laboratory of Radar Signal Processing, Xidian University, Xi an, China, 710071 (xuyishi, jianqiaosun) @stu. xidian edu. en, bchen@mail xidian edu cn
Mingyuan Zhou
McCombs School of Business, The University of Texas Austin, TX 78712, USA mingyuan.. www..nccooossss..seooo utexas.eeoo edu
Abstract
Embedding-based neural topic models have turned out be a superior option for low-resourced topic modeling. However, current approaches consider static word embeddings learnt from source tasks as general knowledge that can be transferred directly to the target task, discounting the dynamically changing nature of word meanings different contexts, thus typically leading to sub-optimal results when adapting to tasks with unfamiliar contexts. To settle this issue, provide effective method that centers on adaptively generating semantically tailored word embeddings for each task by fully exploiting contextual information. Specifically, we first condense the contextual syntactic dependencies of words into a semantic graph for each task, which is then modeled by Variational Graph Auto-Encoder to produce task-specific word representations. On this basis, we further impose a learnable Gaussian mixture prior on the latent space of words efficiently learn topic representations from a clustering perspective, which contributes to diverse topic discovery and fast adaptation novel tasks. We have conducted wealth of quantitative and qualitative experiments, and the results show that our approach comprehensively outperforms established topic models.
Introduction
The last two decades have witnessed the enduring popularity of topic models along with their many successful applications in range of fields [1-7]. And this is predominantly attributed to their ability to reveal the underlying semantic structure from large volumes of textual data. By identifying group of salient themes, topic models represent each document mixture of them, providing intuitive understanding of the target corpus. Although conventional probabilistic topic models [8-14] have been widely used, new variants continue spring up the era dominated by deep neural networks.
Among the proliferation of new methodologies come both the neural topic models (NTMs) [15-18] resulting from the development of variational autoencoders (VAEs) and autoencoding variational Inference (AVI) [19, 20), and the contextualized topic models (CTMss) [21 (21-24) benefiting from the flourishing of pre-trained language models [25, 26]. However, these recently developed approaches essentially maintain the assumption of sufficient resources, i.e., with a plethora of documents being available. Comparatively, little attention has been paid to topic modeling under resource-limited or resource-poor conditions [27], which plays a significant role in real-world applications.
"Equal contribution
i Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

A case in point occurs in personalized recommendation systems where users' preferences are judged based on only small amount of their historical data, such as past purchases or online behaviors [28]. Another example arises from crisis management. Since being able to quickly identify and monitor emerging topics during a crisis, e.g., public health emergency, could substantially support a gover- nment or organization in responding appropriately to rapidly changing situations [29].
While there have also been some beneficial attempts study few-shot learning for topic modeling, they more or less exhibit certain limitations. For instance, Iwata [27] first proposed a straightforward
method that aims to learn good, i.e., task-specific priors for latent Dirichlet allocation (LDA) [8] by using neural networks. However, this approach suffers from a lack of expressiveness when dealing with difficult novel tasks. Later on, Duan et al. [30] claimed that embedding based NTMs [31] are naturally superior in generalizing to new and unseen tasks. By considering word embeddings learned from the training task pool as transferable knowledge, they have shown that effective generalization can be achieved by only learning topic embeddings adaptively. Nevertheless, as the word semantics inevitably change with contexts, the learned static word embeddings may not adapt well to the target task with alien contexts. On the other hand, we have experimentally found that the performance of two representative CTMs [21, 22] is also not competitive enough under resource-poor conditions.
To address the above issues, we propose to learn adaptive word embeddings suitable for each task by fully exploiting the contextual grammar information. Concretely, we first construct task -see-spc semantic graph between words using well-established dependency parsing tools" [32]. This graph helps depict the precise meaning of each word in the given task, which we then model with a created variational graph autoencoder (VGAF) [33], and the resulting latent representations of words merge the contextual information and are thus semantically tailored to the given task. Furthermore, by im- posing Gaussian mixture prior on the latent space of word, we offer perspective on learning topics through clustering, i.e., each component of the Gaussian mixture can be viewed as the representation of topic. Consequently, the adaptive word embeddings tend to be reasonably covered by several clusters, facilitating the discovery of more interpretable topics closely related to the given task.
Our main contributions can be summarized as follows:
To solve the problem of few-shot learning for topic modeling, we propose to learn adaptive word embeddings that are semantically matched to the given task.
To generate adaptive word embeddings for each task, we innovatively introduce a variational graph autoencoder to learn the latent representations of words from the task-specific semantic graph that captures the contextual syntactic dependencies.
To mine more interpretable topics related to the given task, we offer a perspective on learning topics through clustering by imposing Gaussian mixture prior on the word latent space. Besides, we also develop an efficient variational inference scheme to approximate the posteriors of latent variables.
2 Meta-CETM
2.1 Problem Formulation
To clarify the problem definition of few-shot learning for topic modeling, we assume that there are C training corpora [DD)c- from different domains, with the goal of using them to learn a topic model that generalizes to the test corpus Preat in a new domain, i.e., the resulting topic model is supposed to quickly adapt to given test task with a few documents and mine topics related to the new domain. Further, we adopt an episodic training strategy as is conventional in most few-shot learning literature. Specifically, we construct a batch of training tasks (7 1 to mimic the scenario of giving tasks at test time, so that each task contains only several documents from an arbitrary training corpus. And we denote the bag-of-word, words (BoW) representations of the task documents as ( E IR VxJ where V the vocabulary size and J is the number of documents in each task. In addition, our method also builds task-specific dependency graph whose adjacency matrix is represented as A (6) E R RVV. To understand the concepts of corpus, task, and document more clearly, see the example in Appendix C.
'The semantic graphs are built with the help of spaCy, a public natural language processing tool to analyse the grammatical structure of sentences, https: setps://spaae //spas/sppccccc/ 1o/.

---

MIP
Attn
BoWs
Document-level topic proportion
x()
X
H
Reconstructed BoWs
Task /
g(O)
Initialized word embeddings
B(9)
Toplc-word matrix
NCN
DEC
(0)
1
Dependency graph
Adaptive word embeddings
Reconstructed graph
Figure 1: Overview of the proposed method. The top branch establishes standard neural topic modeling
pipeline, and the bottom branch creates graph VAF to learn contextualized word embeddings, with Gaussian mixture prior imposed on the latent space to yield task-specific topic representations. Note that the topic-word matrix is derived based on the probability density assigned to the adaptive word embeddings.
2.2 Generative Model
In this section, we present a Contextualized Embedded Topic Model, dubbed as Meta-CETM, to cope with the problem of topic modeling under resource-poor conditions, the essence of which is to make extra use of the contextual information of given documents to learn dynamic word embeddings that are well adapted to the current task. Concretely, for any given task T in addition to obtaining the BoW representations (f) of its documents, which only imply the word co-occuroo co-occurrence patterns, also build semantic graph between words based on the contextual dependency grammars. Thus the corresponding adjacency matrix A (4) can be viewed as complementary information on the word semantics. With ( and A (8), then the goal of the generative model is to model these two types of observations jointly, whose key lies in how establish bridge connecting the two.
Inspired by the recently developed VGAES [33] and embedded topic models (ETMs) [18, 31], we create our generative model based the compact assumption, i.e., there exists shared latent space of words that can be mapped to the observation spaces of BoW and semantic graph, respectively. Following this conception, we portray a well-structured generative process via a specific task ( First, the latent representations of words (+) E RDV are sampled from a prior distribution. Acting as the suitable connection, they are responsible for generating both the adjacency matrix A (2) and the BoW ()), To produce A (t),
typical inner product graph decoder formulated
A (6) Ber(o (Z ()))
(I)
As for the generation of BoW X the latent representations (t) (or word embeddings) are mainly used to derive the topie-word matrix. Unlike previous ETMs that usually decompose the topic-word matrix into the inner product of learnable word embeddings and topic embeddings, here we provide Bayesian perspective on interpreting the topic-word matrix by imposing a Gaussian mixture prior to the word latent space. More precisely, regard each component of the Gaussian mixture prior as the representation of a topic, such that the word embeddings are naturally generated from different clusters (topics). And the prior probability for the 4tth word embedding is given by
K
p(z
TE NZ Ex),
where Ty is the prior mixing coefficient of the tt topic and Σx Th As such, each topic can be represented as a Gaussian distribution in the word latent space, with a probability density assigned to each word embedding. This provides natural choice to define the topic-word matrix g IR V x

---

In this way, we expect that semantically similar words will be close in the latent space and generated from the same topic. Moreover, we also posit context variable c( that summarizes the task-level information about the topic proportion and thus serves prior for generating the topic proportion Hence, the distribution of topic k over the vocabulary 3 E ER& can be derived by
5 k
Softmax(p(ZZ Mg Σ})),
g of each document, Formally, the complete generation process of our model can be formulated as
1. Generate the adjacency matrix of semantic graph using Eq. 1;
2. For the 4 word appeared in the task (1):
(a) Draw its topic assignment yo Cat(T));
(b) Draw its word embedding Z N Wu (0)):
3. Compute task-specific topie-word matrix ((5) based Eq. 3;
4. Draw task-level context variable c( ~N(00a21))
5. For the jth document in the task (
(a) Draw its topic proportion 8/) LN((()) IPT);
(b) For the th word the jth document:
Draw its topic assignment in
Cat(
ii Generate the word count Zjn
(x) Cat(G
where Cat() .(). Ber(), and LN() denote categorical, Gaussian, Bernoulli, and logistic- normal distributions, respectively. is the Sigmoid() operation*. and b are both hyper-parameters. Table in Appendix B gives a list of key notations used in this paper.
2.3 Variational Inference Algorithm
Observing a task ( with data ((), the goal of inference network approximate the posterior dis-
tributions over the latent variables, {0 cl z()) }. and the parameters of GMM, { M >
(6) (7)
Document-specific latent variable inference. To model the uncertainty of topic proportions,
define the variational posterior distribution for g and a residual multi-layer perception (MLP) is employed to learn distribution parameters. To be specific,
q(0
(1)
())) CN(Mg) E Eg(o)
M Mgin Σ ( = ResMLP(h (0) + +c)))
h (x) (1)
where Maj and Ege0 are logistic Gaussian distribution deterministic parameters depending on the latent mean (()) and the document latent representation h of xj
Task-specific latent variable inference. For topic proportion mean ( we apply Gaussian distribu- tion to approximate the variational posterior, whose parameters mean vectors and covariance matrices are derived through encoding all documents in task (+) with attention mechanism [34], written as
g(c()) (x( NH Eg(h) Mett) Egto AttnnH (0) H MLP(X ()),
(5)
where ( the latent indication of the task, and Attn() the attention mechanism to capture relationships of J documents in task (())
'If weighted graph A (+) is considered, then the Poisson likelihood can be employed with set to the Softplus() operation.

---

Additionally, to ensure the modeling flexibility, we design simple inference network consisting of (6) of words following [33]:
two-layer GCN [35] to infer the latent representations Z
(Z A (0)
E = - ( 12z Σ
(6)
Σイ Z = GCN (0) E(D)
PZ
where E(S) is the initialized word features and ( is derived through a neural parser on X(), Figure illustrates overview of our approach, including the variational inference network.
In this paper, for simplicity, we denote as network parameters of both the encoder and the decoder.
Expectation Maximization for solving ( and Σ As discussed before, f4 and Σ/ are
&
task-specific parameters of Gaussian mixture distribution, which are not learnable variables, and do not have analytic solutions for Maximum Likelihood Estimation (MLE) of GMM containing the non-differentiable sampling process [36]. To approximate the posterior, we resort Expectation Maximization (EM) [37] algorithm to optimize the parameters, formulated as:
" N(
E Step
Q
ply! klZz
(1 Σx "
M(
1
(
Σ. Q00 (9)
20
M Step
E. 0.000
Σn (2)
10
10 )Z)
M 100
M
Σ, Q
E. Q!
Σ. Q!
Since topic l is sampled from the Uniform distribution for each task, the mixing coefficients are
initialized as Φ. We initialize fi matrix I, respectively. Here, we only display the final updating formulas. The detailed derivation
(2) and x as the average of latent variables ( and the identity
processes for E-step and M-step presented in Appendix E.2.
2.4 Training Objective and Optimization
By Jensen's inequality, the evidence lower bound (ELBO) of each task be derived as
p(0
elti)
LELBO
T [ix
Σ
Ee
log
((0()) 1xx² (1) .c(i)
(8)
p(c(it)
p(Z
+ EQ
log g(e(i)
X EQ
| + Eq
log
(Z
A
EG)
where
Q -I j=1 (0) ( ( )g(c()
X (Z(
(9, E())
is the variational joint distribution. The first and the fourth terms in Eq. are the reconstruction errors for document BoW and the graph adjacency matrix, respectively. The remaining three terms are all Kullback-Leible (KL) divergence to constrain the distance between the prior distribution and the variational posterior distribution. Owing to the space limit, we only present the final formulas for LELBO here. The detailed derivations, the training algorithm, and the meta-testing algorithm can be found in Appendix 1, Alg. 1, and Alg. 2, respectively.
3 Experiments and Analysis
3.1 Experimental setup
Datasets. We conducted experiments on four widely used textual benchmark datasets, specifically 20Newsgroups (20NG) [38], Yahoo Answers Topics (Yahoo) [39], DRpedia (DB14) [40], and Web of
Science (WOS) [41].
5

---

Table 1: PPL. results four datasets. "5" and "10" denote the number of documents in each task. *Since ProdLDA and ETM are not designed for few-shot learning, we run their meta versions where parameters are optimized using -agossie meta- meta-nn (MAML) strategy [44].
Methods
20NG
Yahoo
DB14
WOS
5
10
10
10
10
LDA [42]
150211227
4476
4028
4410.119
3697
34791667
3246-1461
PHA [12]
3463:1452
315021119
325741328
3122=1040
3443-1937
3170=1562
3113=819
3431=830
ProdLDA [43]
4853
4523-817
5765
5378.1826
$4771846
5297.1740
4311
4220-1922
ETM [18]
3192:895
3107=671
2868:9099
2817:620
3217=1960
3054=1539
3135=704
3310=455
MAML MAMI-ProdLLDM
4292-11123
4355
435411339
4250.1919
4844
4678
4117.462
4068
MAML-ETN ETM
1849:10044
3725=841
3653+1081
36424776
444812737
4279=2301
34832-4044
3277=644
Meta STwETM [30]
2872:1869
2984-770
2365:1934
24874155
204711374
1914=1009
2031::445
2253=315
CombinedTM [21]
2660+659
2595=625
2700=500
2674::775
1851=767
1774+731
2562=633
2648=658
TeroShorTM [22]
29044-851
663
28224-732
279547221
1938+758
1835+4739
2863-704
2775-558
Meta-CETM
954.1543
1170_1606
107414442
1219.1455
802.1571
1084_1643
1293
1528-1118
Baseline methods. Our model is compared with exemplary baseline methods, including probabilistic topic models, state-of-the-art NTMs and CTMs. Specifically, we conduct experiments of the following methodologies under the document- -imited setting: 1) LDA [42]; 2) PFA [12]; 3) LDA with Products of Experts (DrodLDA) [43], which replaces the mixture model in LDA with products of experts and updates parameters using AVI; 4) Embedded Topic Model (ETM) [18], an NTM incorporating word embeddings and learning with AVI. For a fair comparison, we also consider the variations of ProdLDA and ETM under the few-shot setting, referred to 5) MAML-ProdLDDA and 6) MAML-ETM, where the parameters are optimized through MAMI algorithm [44]. In addition, we include a model-based hierarchical NTM, 7) Meta-SawETTM [30], but we only apply their single-layer model. Moreover, another two CTMs 8) CombinedTTM [21] and 9) ZeroShorTM [22] are also compared, both of which contain contextualized Sentence BERT [45] embeddings as the model input.
3.2 Experimental results and analysis
In this section, we evaluate the predictive performance, topic quality, and classification performance of our model through an extensive series of experiments. Note that in all tables, have highlighted the best and runner-up results in boldface and with an underline, respectively. Our code is available at https::///// wtw.thh...oo/oii.ii.eoeo wwweeeetttteeeeeeeeeeeeeeee
3.2.1 Per-holdout-word perplexity
Following the practice in Meta-SaWETM [30], we adopt the per-holdout perplexity (PPL) [46] measure the predictive performance of our model. Specifically, for each task composed of several documents, 80% of the tokens in the BoWs are randomly chosen to form a support set D which used adapt a task-specific topic-word matrix f,, and the remaining 20% word tokens are held form the query set Pest with data Y. Then, the PPL can be calculated as
E bu 2a=1 20=1 Lk= 1
exp
Von In
(10)
N where S is the total number of collected samples and y. EV Ex= Vvn-
Model settings. For all compared methods, we set the number of topics as 10. And for all NTMs, the hidden layers size of the encoder is set to 300. For all embedding- -basen topic models, i.e., ETM,
MAML-ETM, Meta-SawETM, and our
we load pretrained GloVe word embeddings [47]
as the initialization for fair comparison. Finally, We train our model using the Adam optimizer [48]
with a learning rate x 1 for 10 epochs on an NVIDIA GeForce RTX 3090 graphics card.
Results. In Table 1, we list the PPL of ten compared methods on four datasets. It can be noticed although LDA and PFA are both traditional probabilistic topic models, PFA presents better results than LDA. By utilizing MAML to learn parameter initializations, MAML-ProdLDD performs better than ProdLDA while MAML-ETM exhibits poorer results than ETM. This can be attributed that ETM possesses more parameters than ProdIDA, and calculating the gradients in high-dimensional space with only a few documents is difficult for MAML-ETMM Applying Weibull distribution to model the latent representation for documents and employing task-specific variable designs for both

---

MAML-ProdLDA
MAML-ETM
Mcta-SawETM
CombinedTM
ZeroShorTM
Mcta-CETM
0.8
0.6
0.2
20NG
Yahoo
DB14
WOS
0
-0.2
-0.4
-0.6
-088
-1
Figure 2: Topic diversity results (top row) and topic coherence results (bottom row) on four datasets of six compared methods. The number of documents for cach task is 10.
topic proportions and topic matrices, Meta-SawETh acts superior to other topic models. Moreover, incorporating contextualized embeddings as input enables CombinedTM and ZeroShotTM to present competitive performance. Furthermore, it is noteworthy that Meta-CETM achieves the lowest PPL among all these methods, indicating the excellent predictive performance of our model.
3.2.2 Topic quality
In this part, we evaluate the topic quality of different methods in terms of the topic diversity (TD) [49], defined as the percentage of unique words in the top 10 words all topics, and the topic coherence (TC), which measures the average Normalized Pointwise Mutual Information (NPMI) [50, 51] values to count word co-occurrences' The experimental settings are the same as in See. 3.2.1. For each task, the number of documents is 10 for all datasets. The results are displayed in Fig. and it can be notably and interestingly observed that MAML-ProdL.DA achieves fairly "perfect" TD results, but it shows the worst TC performances among compared methods. Such inconsistency is brought by MAML-ProdLDA'S concentration on a large amount of universal and trequently-occuriin when training on the base data, hindering it from extracting informative topics given the meta-test task. Embodying the embedding design for words and topics, MAML-ETM and Meta-SawETM are more likely to mine context-related topics than MAML-ProdLDa Besides, as the representatives of CTM, CombinedTM and ZeroShotTM equipped with BERT embeddings present more comprehensive performances than previous topic models. From the TC perspective, Meta-CETM, yields the most favorable results among the six methods, indicating the fast adaptability discovering interpretable topics with limited documents.
words
3.2.3 Few-shot document classification
To further validate that our model capable of learning topics which are highly adapted to the task, we undertake experiments on few-shot document classification.
Model settings. As in Meta-SawETM [30], we compare our CEta-CEEM with classical meta-learning algorithms under different architectures. Specifically, we design three-layer feedforward network as MIP structure and three-layer I-dimensional convolutions followed by batch normalization, ReLU() activation and max- max-pooling operation as CNN. For meta-learning methods, we apply MAML [44] to learn parameter initializations, and prototypical network (PROTO) [52] learn an embedding space and minimize the distance between the clustering centroids and the samples. Besides, we adopt two fine-tuning manners, named FT [53] and FT to update the classifier parameters and all parameters of the model, respectively. Single-layer HNS-SaWETM and single-layer Meta-SawETMM in [30] are compared as well. Additionally, we investigate the CombinedT' [21] and ZeroShotTM [22] to evaluate their few-shot classification performance. Different from PPI. evaluation in Sec. 3.2.1, the support set and the query set for classification are sourced from two batches of documents.
'The NPMI score is computed through the gensim package in http:///aceeuuurerooooooosss////
com/ http//gnnimmmm/m/
7

---

Table 2: 5-way 5-shot and 5-way 10-shot few-shot document classification results on all four datasets. *denotes all parameters of the model are fine-tuned.
Methods
20NG
DB14
Yahoo
WOS
Rep.
Alg.
5 shot
10 shot
5 shot
10 shot
shot
10 shot
5 shot
10 shot
MAML [44]
32.01
36.20
50.20
60.30
45.42
51.00
37.77
40.43
PROTO [52]
35.20
38.30
54.13
57.16
50.01
56.16
39.61
41.46
MLP
FT [53]
29.70
33.04
51.11
53.83
48.59
53.06
36.52
37.22
FT#
38.87
48.52
71.12
77.94
50.73
56.74
45.02
51.20
MAML [44]
34.08
45,40
66.28
75.96
48,81
56,50
47.28
$7.32
PROTO [52]
39.86
49.71
78.58
81.01
53.16
63.66
59.05
67.75
CNN
FT [S3]
45.70
53.63
74.68
80.75
56.78
66.04
54.68
63.39
FT
44.53
51.92
72.49
80.07
53.28
52.56
51,42
61.98
INS-SSwETM [30]
39.37
43.78
65.93
71.08
52.35
$7.86
42.09
56,91
Meta-Saw
Saw ETM [30]
39.19
45.83
67.20
72.31
52.45
60.58
43.39
57,44
CombinedT' [21]
46.17
52.73
68.42
73.26
57.94
64.75
56.16
65.97
ZeroSholTM [22]
46.65
52.08
71.93
76.09
58.12
66.21
58.50
66.10
Meta-CETM
50.57
58.47
76.85
79.34
63.84
72.67
61.47
67.62
Classification strategy. For typical few-shot learning algorithms, we follow the convention train parameters of the feature extractor in a supervised manner. At meta-test stage, we use the support set (i.e., few labeled examples) adapt to task-specific classifier and compute the accuracy on the query set. As for topic models, where no dedicated classifiers are available, we first use the support set to learn a group of class-specific topic-word matrices, then for each document in the query set, we calculate its reconstruction error as the basis for classification.
see, the CNN architecture outperforms MLP under the same algorithms, which can be attributed to CNN's unique inductive bias of the locality. Furthermore, PROTO surpasses MAML by large margin in most cases, indicating that a good embedding space is more useful for classification than favorable parameter initializations. For the CNN architecture, fine-tuning the classifier only (FT) and fine-tuning all parameters (FT*) make slight differences; but for MLP, we observe a considerable performance boost by updating all parameters of the network over the FT algorithm. We postulate that the classification results of MLP are more susceptible to the variation of feature extractor parameters due to its linear structure. Additionally, CombinedTM and ZeroShorTM, incorporating contextualized representations, outperforms HNS-SaWETM and Meta-SaWETM by learning class-specific topic-word matrices more effectively. With the design of adaptive word embeddings. our better results than previous topic models, but also is comparable to few-shot learning algorithms particularly designed for supervised learning.
Results. The classification results are listed in the Table 2. As
not only achieves much
3.2.4 Embedding space visualization
Sellanis
who
(4)
(b)
(0)
Figure 3: Visualization of the adapted embedding space for (a) MAML-ETM. (b) Meta- Meta-SawETM and (c) Meta- CETM. The small grey points represent word embeddings, and the big blue points denote topic embeddings for For MTTa-SaWETM and our Meta-CETM,
MAML-ETE -TMMLETM,RR topic means for Meta-SawETM and our the ellipse coverages represent topic covariances. For MAML-ETM, the ellipse coverages are the areas of top words. The target task is sampled from the sub-topic "rec.sport.hockey of 20NG dataset.
'The classification experiments for all compared methods conducted ten times.
8

---

In addition to quantitative results, we also qualitatively analyzed the effectiveness of our model by visualizing the adapted embedding space, as shown in Figure 3. It can be seen that both MAML-ETM and Meta-SawFTT learn topics that are not highly relevant to the target task, as most of their top words (e.g., windows and dos) are inherited from the base training corpora, while those informative words (e.g-, players and hockey) associated with the target task are away from the topic embeddings. By contrast, in the embedding space learned by Meta-CETM, the adapted Gaussian distributions, i.e. topics reasonably cover almost all words closely related to the target task, indicating that our model can achieve successful adaptation and discover more interpretable topics.
Furthermore, we investigated whether the adap- tive word embeddings generated by our model effectively reflect the contextual information of each given task, As illustrated in Figure 4, the word embedding of "apple" adapted from task related to company is surrounded by words like "mobile" and "amazon", whereas in another task concerning plant, the embedding of "apple" is closer to the words such "fruit" and "trees". This phenomenon suggests that Meta-CETM produces word embeddings that are semantically well- matched to the context of the target task.
Jeeckelogies
marsfigturr
accopperies
apple
The adaptive contextual word embeddings learned by our Meta- Meta-CETM DB14 dataset [40]. Left: The local embedding space task from the "Com- pany" domain, Right: The local embedding space of a task in the "Plant" domain.
Figure 4:
3.3 Ablation study
To explore the effectiveness of key designs in model, we conduct series of ablation experiments on all four datasets by removing each designed module separately. The numerical results perplexity, topic diversity, and topic coherence are listed in Table 3, where ETM [18] is chosen the baseline method. From the results presented below, we have the following observations:
Despite achieving the best TD results, ETM obtains the worst PPL and TC scores, which can be attributed to tendency to extract frequently occurring themes from training, hindering its fast adaptability to meta-test sets with only several documents. ii) To capture the relations of texts and model the task information, in Section 2.2, we posit context-specific variable c++) the prior of
document-level topic proportion (/ It can be found with ctt), our Meta-CETM achieves much lower PPL and higher TC, demonstrating the efficacy of task-specific variables. iii) For ablation results without the Graph VAE module, we replace with a Graph AE rather than discarding it completely, Besides, for Meta-CETM without the GMM prior design, we replace it with the standard Gaussian distribution prior rather than imposing no prior completely. From the quantitative results shown
Table 3: Ablation study on four datasets. The number of texts in each task is 10. "V" means we add the corresponding design into ETM [18], which is chosen as the baseline.
context variable
Graph VAE
GMM prior
20NG
DB14
PPL
TD
TC
PPL
TD
TC
ETM
3107
0.8395
-0.8437
3054
0.8106
-0.8719
1964
0.8031
-0.4301
1682
0.7441
-0.4917
1255
0.7983
-0.4169
1131
0.7210
-0.5025
1361
0.6538
-0.4688
1276
0.6562
-0.6677
Meta-CETM
1170
0.8
-0.3701
1084
0.7475
-0.4783
context variable
Graph
GMM
Yahoo
WOS
VAE
prior
PPL
TD
TC
PPL
TD
TC
ETM
2817
0.8851
-0.8913
3310
0.9286
-0.9785
1906
0.7243
-0.5097
2023
0.9141
-0.5420
1316
0.7612
-0.4860
1304
0.8387
-0.5262
1271
0.5847
-0.5503
1389
0.7018
-0.5587
Meta-CETT
V
1219
0.7886
-0.4639
1293
0.8667
-0.5177

---

Hopie
JOHEES
dedli
2000
Sartey
(1)
(6)
(0)
Figure 5: Illustration of the advantage of using GMM prior. The adapted embedding space of our model by using (a) no prior, i.e, Graph VAE is replaced with a vamilla graph AE, and the topic embeddings are lcarnablc point vectors, (b) standard normal prior, i.e. the topic embeddings are learnable Gaussian distributions which are constrained by KL divergence with the standard normal distribution, (c) GMM prior. The target task is sampled from the sub-topic 'poo..oor.hocke"" of the 20NG dataset. Room in for better visual effects.
in the table, it can be found compared with Gaussian prior in vanilla VAE model, GMM priors encourage the semantically coherent words to allocated to the same topic, leading to higher TC scores and lower PPL. Further, we visualize the embedding space in Figure 5 to demonstrate the benefits of GMM prior.
Related Work
To discover group of topics from batch of documents, probabilistic topic models have been devel- oped in recent years, including hierarchical topic models [54, 14, 46, 55]. NTMs [56, 15].embedded topic models (18, 31], and CTMs [22, 21J. Besides, some works are proposed incorporate knowl- edge into the modeling process [57-59] and some researchers apply optimal transport to measure the distances between topies and words or documents [60, 61]. From the perspective of modelling word embeddings with Gaussian distribution, Vilnis etc. proposed novel density-based mapping method [62] and achieved promising performances. Recently, some works [30, 27] aimed at topic modelling under the few-shot setting and proposed to obtain group of task embeddings to adaptively explore topics within only few documents in a model-based meta-leaming fashion. However, their method cannot address the multiple meanings of one word issue, prevalent issue in practical document analysis, which addressed through the introduction of dependency graph and the GMM prior distribution in our work.
5 Conclusion
In this paper, we propose a novel NTM, Mcta-CETM, to address the the fast adaption problem in document analysis under low-resource regimes. Specifically, construct task-specific graph to obtain context-related word embeddings. Then we introduce the graph VAE with Gaussian mixture prior to model the word representations and topic embeddings, which are optimized through the EM algorithm. We also propose the task-specific prior for topic proportions. Through extensive experiments and illustrations, we demonstrate the superior performance of our model in solving the adaptation problem in topic modeling.
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant U21B2006: in part by Shaanxi Youth Innovation Team Project; in part by the Fundamental Research Funds for the Central Universities QTZX23037 and QTZX22 160; part by the 111 Project under Grant B 18039; in part by the Fundamental Research Funds for the Central Universities; in part by the Innovation Fund of Xidian University under Grant YJSI23016.
10

---

References
[1] Thomas Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences, (ttssuppl_):5228-555513 2004.
[2] Andrew McCallum, Xuerui Wang, and Andrés Corrada- Emmandell Topic and role discovery in social networks with experiments on enron and academic email. Journal of artificial intelligence research, 30:249-274 272, 2007.
[3] David Mimno, Hanna Wallach, Jason Naradowsky, David Smith, and Andrew McCallum. Polylingual topic models. In Proceedings of the 2009 conference empirical methods in natural language processing, pages 880-889, 2009.
[4] Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Sociery Conference on Computer Vision and Puttern Recognition (CVPR'03) 105, volume 2, pages 524-531. IEEE, 2005.
[5] Simon Rogers, Mark Girolami, Colin Campbell, and Rainer Breitling. The latent process decomposition of cdna microarray data sets. JEEE/ACM transactions on computational biology and bioinformaties, 2(2):143-156, 2005.
[6] David Mimno. Reconstructing pompeian households. arXiv preprint arXiv: 2012.
[7] Jordan Boyd-Graber, Yuening Hu, David Mimno, et al. Applications of topic models. Founda- tions and Trends@ in Information Retrieval, 1(2-3):143-296, 2017.
[8] David M Blei, Andrew Y Ng, and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):933-1022 2003.
[9] David Blei and John Lafferty. Correlated topic models. Advances in neural information processing systems, 18: 18:147, 2006.
[10] David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning, pages 113-120, 2006.
[11] Jon Mcauliffe and David Blei. Supervised topic models. Advances in neural information processing systems, 20, 2007.
[12] Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statistics, pages 1462-1471. PMLR, 2012.
[13] Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested Chinese restaurant process. Advances neural information processing systems, 16, 2003.
[14] John Paisley, Chong Wang, David M Blei, and Michael Jordan. Nested hierarchical Dirichlet processes. IEEE transactions on pattern analysis and machine intelligence, (7(2)2556-270, 2014.
[15] Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference muchine learning, pages 1727-1736. PMLR, 2016.
[16] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv: 1eei1170.00300 2017.
[17] Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull hybrid autoencoding inference for deep topic modeling. arXiv preprint 1rXiv:1803.01328. 2018.
[18] Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics, 8:439-453, 2020.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

---

[20] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and upproximate inference deep generative models. In International conference on machine learning, pages 1278-1286, PMIR, 2014.
[21] Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training is hot topic: Contextualized document embeddings improve topic coherence. arXiv preprini arXiv:2004 033974 2020.
[22] Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. Cross- lingual contextualized topic models with zero-shot learning. In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1676-1683, Online, April 2021. Association for Computational Linguistics.
[23] Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint urXiv:2203.05794, 2022.
[24] Zihan Zhang, Meng Fang, Ling Chen, and Mohammad-Rere Namazi-Rad. Is neural topic modelling better than clustering? an empirical study on clustering with contextual embeddings for topics. arXiv preprint 0rXiv:220.009874, 2022.
[25] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810.00880 2018.
[26] Nils Reimers and Iryna Gurevych. Sentence-bers Sentence embeddings using siamese bert- networks. arXiv preprint 1908: 2019.
[27] Tomoharu Iwata. Few-shot learning for topic modeling. arXiv preprint arXiv:2104.09011, 2021.
[28] Anand V Bodapati. Recommendation systems with purchase data. Journal of marketing research, 45(1):77-93, 2008
[29] Hao Sha, Mohammad Al Hasan, George Mohler, and P Jeffrey Brantingham. Dynamic topic modeling of the covid-19 twitter narrative among us governors and cabinet executives. arXiv preprint arXiv:2004. 1692, 2020.
[30] Zhibin Duan, Yishi Xu, Jianqiao Sun, Bo Chen, Wenchao Chen, Chaojie Wang, and Mingyuan Zhou. Bayesian deep embedding topic meta-learner. In International Conference Machine Learning, pages 5659-5670. PMLR, 2022.
[31] Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In ICML 2021: International Conference on Machine Learning, July 2021.
[32] Danqi Chen and Christopher D Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 740-750, 2014.
[33] Thomas N Kipf and Max Welling. 1rXiv:1611.07300 2016.
Variational graph auto-encoders. arXiv preprint
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[35] Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 2017). 2016.
[36] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[37] Todd K Moon. The expectation-mmaxiinzziio algorithm. IEFE Signal processing magazine, 13 (6):47-60, 1996.
12

---

[38] Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995, pages 331-339. Elsevier, 1995.
[39] Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28, 2015.
[40] Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In The semantic web, pages 722-735. Springer, 2007.
[41] Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura Barnes. Hdltex: Hierarchical deep learning for text classification. In 20/7 16th IEEE international conference on machine learning and applications (ICMIA), pages 364-371. IEEE, 2017.
[42] David M Blei, Andrew Y Ng. and Michael Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jam):9331102, 2003.
2003.
[43] Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv: 703.001488, 2017.
[44] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In International Conference on Machine Learning, pages 1126-1135. PMLR, 2017.
[45] Nils Reimers and Iryna Gurevych. Centence-BERT Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLPP), pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/D19-1440 URL https:/ pettp://aacaantholoog
Dtrg/Di9-1441 1410.
[46] Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet Allocation with topic-layer-adaptive stochastic gradient riemannian memc. arXiv preprint arXiv: /70.00/77007724 2017.
[47] Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, October 2014. doi:
[48] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[49] Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pages 6345 6381, Florence, Italy, July 2019. Association for Computational Linguistics.
[50] David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguisties, pages 100-108, Los Angeles, California, June 2010. Association for Computational Linguistics.
[51] Michael Röder, Andreas Both, and Alexander Himneburg. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM '15, page 399-408, New York, NY, USA, 2015. Association for Computing Machinery.
[52] Jake Snell, Kevin Swersky, and Richard $ Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv: 1tt:::700.0550 753.05175, 2017.
[53] Wei- Yu Chen, Yen-Cheng Liu, 7solt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification, In International Conference on Learning Representations, 2019.
13

---

[54] David M Blei, Thomas L Griffiths, and Michael I Jordan. The nested Chinese restaurant process and bayesian nonparametric inference of topic hierarchies. Journal of the ACM (JACM). 57(2): 1-30, 2010.
[55] Mingyuan Zhou, Yulai Cong, and Bo Chen. The Poisson gamma belief network. Advances in Neural Information Processing Systems, 28:3043-303 2015.
[56] Dallas Card, Chenhao Tan, and Noah A Smith. A neural framework for generalized topic models. arXiv preprint arXiv: 705.09996. 2017.
[57] Zhibin Duan, Yishi Xu, Bo Chen, Dongsheng Wang, Chaojie Wang, and Mingyuan Zhou. Topic- net: Semantic graph-guided topic discovery. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[58] Dongsheng Wang. Yi.shi Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen, and Mingyuan Zhou. Knowledge-aware bayesian deep topic model. In Advances in Neural Information Processing Systems, 2022.
[59] Yi.shi Xu, Dongsheng Wang, Bo Chen, Ruiying Lu, Zhibin Duan, and Mingyuan Zhou. Hyper- miner:
Topic taxonomy mining with hyperbolic embedding. In Advances in Neural Information Processing Systems, 2022.
[60] He Zhao, D Phung, V Huynh, Le, and W Buntine. Neural topic model via optimal transport, 2020.
[61] Viet Huynh, He Zhao, and Dinh Phung. Otlda: A geometry-awand optimal transport approach for topic modeling. Advances in Neural Information Processing Systems, 33:1857: 117188888188 2020.
[62] Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. arXiv preprint arXiv: 1412.66311. 2014.
14

---

A Discussions
A.1 Limitations and Future Work
In this paper, we propose a method to improve existing embedded topic models (ETMs) under the low-resource settings by introducing the task-specifis graph with the Gaussian mixture prior. The main limitation of our work could be the reliance on the pre-trained parsing tools. Specifically, we construct the semantic graph according the syntactic structure of the documents with the public library, spaCy'. However, some other approaches such as the sliding window method can also be used construct the graph, which is not relied on training. Besides, although strong performances. our framework is still trained and evaluated on the same dataset. While, in the current large models era, pre-training on large set and then perform few-shot learning even zero-shot learning attracts researchers' attention and interest, especially such Bayesian models. However, this is beyond the scope of this paper and we will conduct a thorough investigation of this issue future work.
A.2 Broader impact
Our work builds novel and advanced topic modeling technique under few-shot settings. For example, the very beginning of the Covid-19, 19, there were relatively few cases of infection in all regions, and the reports of related medical diagnoses were highly limited. Under such circumstances, few-shot topic models can help us extract key information from limited resources, thus facilitating us to take appropriate and efficient preventive and control measures. On the other hand, the emergence of ChatGPT by OpenAl has attracted a great deal of attention, while this paper, what we explore is robust model with generalization capabilities given limited text data, rather than training from very large corpus. Potential negative societal impact of our work could arise from malicious intent in changing model's behavior by injecting deliberate human prejudice, which may harm the fairness of the community. However, we hope our work is utilized to enable more research and applications primarily from the originality of benefiting the community development.
B
Key Notations
In Table 4, we list the key notations, descriptions and corresponding dimensions used in this paper.
Table 4: Notations used in the paper.
Symbol
Dimensionality
Description
M
number of total training tasks number of documents in each task
J
K
number of topics in cach task number of vocabulary terms, shared across tasks dimensionality the word latent space the 7th training task
V
D
TO (
RVJ
the BoWs representations for documents in the 5th task the deterministic hidden features of BoWs ( context variable that summarizes the topic proportion information topic proportion of the jth the ęth task topic-word matrix for the 5th task the adjacency matrix of dependency graph for the 5th task initialized features of the with word appeared in the ith task
H
3
e
RK
9
RK
3
RV*K
A
RVV
e
R.D
RD
adaptive embedding of the tt word appeared in the zth task
coefficient of the 4th Gaussian component for the jth task mean of the th Gaussian component the th task covariance the t Gaussian component for the 5th task
RD
Σ
R D
hhtpps:/sspacy. .10/
15

---

.-Whan buzy. This the bus fren nome the dota san be transmitted...
Grnv
deta
The problam
espacially
Secul, that there are to many privatelyrowmed cara, as wall as tace and buses, the nush has become 24 26-hour phenemenen and that there
place
Task
park...
Figure 6: An illustration of word sense variation caused by different contexts. The task is sampled from a corpus about "hardware", and the task j is sampled from a corpus related to "autos".
C An Illustration of Our Settings
In Section 2.2, we touched on the concepts of corpus, task, and document to describe our problem setting, which is bit messy to follow. Here, we clarify these concepts through concrete example so that the reader could understand the problem setting of few-shot learning more easily. Considering the 20NG [38] dataset:
"corpus" refers a collection of documents belonging to the same category so that 20NG consists of 20 corpora, each of which contains documents from one of the 20 classes.
"task" smaller unit than "corpus" that only comprises few related documents, typically less than 100 documents. Consequently, we could sample number of tasks from each training corpus.
(we select 12 out of the 20 corpora for training)
Then our goal is to utilize these sampled tasks train a generalizable topic model that can efficiently adapt to a new task from the test corpus (the remaining 8 corpora are used for testing). In addition, for each task at the testing stage, we split its documents into two parts, one for fine-tuning or retraining the topic model, called the support set, and the other for evaluating the model's performance, called query set. Note we do not design different generative processes for the corpus documents versus the task documents. In essence, our proposed Meta-CETM only characterizes the generative process of the task documents by jointly modeling the syntactic graph and the observed BoW X in each task. In Fig. 6, visualize the task and the corresponding unweighted dependency graph A.
D Algorithms for training and testing
In Alg. 1 and Alg. 2, we present the training and meta-testing procedures of our Meta-CETV
E Derivation of Formulas
In this section, we provide the detailed derivation process of variational evidence lower bound (ELBO) in Eq. 8 and the expectation maximization solver process for multivariate Gaussian distribution in Eq. in Sec. 2.2.
16

---

Algorithm 1: Training process
Input: A set of training corpora (De P2 initialized model parameters Randomly sample tasks from each training corpus Dj to obtain (T for each task T(,,, i 1,2..... .M do
Build semantic graph A (0) with established dependency parsing tools; Infer adaptive word embeddings ( according to Eq. 6;
Initialize parameters of the Gaussian mixture prior: Ths 44g and Σg;
Update to the optimal value T 1k
(i) (8) and > using EM based on Eq.
Compute the topic-word matrix 3 according to Eq. 3;
Infer the latent context varibale c( using Eq. 5;
for each document x5" .j = ,.
Infer topic proportion 0.60 with Eq. 4;
Calculate the log-likelihooo p(x) ( 3()):
Derive the ELBO as Eq. and update vp using SGD;
Algorithm 2: Meta-test for task
Input: A new corpus Dtest, trained model parameters V
Output: Adaptive topic-word matrix B
Randomly sample a task Tnew from the given corpus Diest;
Get the corresponding BoWs Xyew and dependency graph Anew for the current task; Infer the adaptive word embeddings Znew with part of the trained model parameters VD; Initialize parameters of the Gaussian mixture prior: Tk. Mg and Eg;
Compute optimal T H and Σ using EM based on Eq. 7; Derive the adaptive topic-word matrix Bnew by Eq. 3;
E.1
Variational ELBO
log p( A()) log //ʃ A (0, 1000
log /// p(x()) ( z
(pp z )(Z))) (
(1) ( d2(1)
p(((()) (00 z
Pp(c)))((A)) 2((((()))
log EQ
((0 XO),
(())(Z0)
At0, E(J)
p(x Ol0) z (((( (((())))))(( 2 (((())
> EQ
log
( X0,, (g((()) (()) AO) B(O)
p(0()) c())
EQ
log
p(x))) (1) 10(0) )(0)
+EQ log II
| (2) (0(())
j=1
j=1
p(c(1)
((((1)
+ EQQ
log (c(()) x((i)
+ +EQ ((g
[ + R.Q log ((()))
AA0, E(i)
j
j
P(O()) ((())
Σ Ee
[log (1)
1000 2(07)]
+ Σ
EQ
log
a(0 I
,(0)) .(0)
1=1
j=1
p(c(1)
p(Z(1))
+ Eq
log g(c(t) x((i)
+ Eq
log (((()) (0)
[(0)]] + EQ log ((()) 110, E()
LELBO
(LI)
17

---

E.2 Solving topic parameters {Tk Mk Fkmi with Expectation Maximization
(1)
(:(00)
The log likelihood function is given by
K
In p 11) , 10, E(0) = Σ
Σ
N=
1 x
(12)
(i) 1. Deriving P
Setting the derivatives of p( (1() |x(0, 1,,, ,00) w.r.t the means Pi to zero, we have
(6)
Th N2
(
(6) 14 [())
(+)) (=!
1k 0.
(1)
(13)
K (1) N(20)) 1
(2)
( )(
-==1
Define the posterior probabilities as
i Mi P 200)) (
( (
Yok p(u 2
(14)
Σ
(2 p 2!!))
(
Multiplying by Σ/ and rearranging, obtain the updating formula for pi
Ev Yuk 20
(15)
Ev Yok
2. Deriving (()
Similarly, we the derivatives of In (( ,1()) 20) w.r. 2:(h0
to zero, then have
C N
(1))
+ 1 T (=! ((
0. (16)
(
(
Using Yok Eq. and rearranging, we get the updating formula for >x()) as
x Ev Yok (2!
(=))))))
(())))
(17)
E. Yok
3. Deriving = Finally, using Lagrange multiplier algorithm, goal is to maximize the following formula:
K
K
Σ
m
1),
(18)
A=1
where Σ k=
Then setting the derivatives of the above equation w.r.t Tī.
zero, have
TE N(2
(1)
(8)
PR
=
(19)
2e (2200
1
(6) (
(6) Multiplying oi and rearranging, obtain
x N(2 (8) )
(8)
1 N ( 2l
En Yek
T
A
(20)
Considering ΣK = 1, then Σx
Σn Yok
and A E. Tokk Tok.
Hence the updating formula for TTK as
(2)
Ew Pok
m
(4)
Σ. Σp Yok
(21)
18

---

Table 5: results four datasets with 20 documents in each task.
Methods
20NG
Yahoo
DB14
WOS
LDA[42]
2979
3916
3095
2370
PFA[12]
2439
2545
1903
1675
ProdLDA[43]
4807
6093
5819
4617
ETM[18]
3276
2781
2870
3189
MAML-ProdLDDA
4378
4033
4612
3908
MAML-ETM
3287
3439
2819
4189
Meta-SawETM[30
2657
2859
2355
3620
[ombinedTMM21]
2331
2543
1863
2587
[eroShoCTM[22]
2673
2664
1722
2660
Meta-CETM
1216
1369
1109
1482
F More Results
F. Perplexity results
In Table I, we list the PPL results of different compared methods four datasets, where the number of documents or in each task. In this part, we provide more PPL quantitative results with varied numbers of texts task (20, 50, 100) in Table 5, Table 6 and Table 7, respectively.
Table results on four datasets with documents in each task.
Methods
20NG
Yahoo
DB 14
WOS
LDA[42]
2443
3279
2353
2091
PFA[12]
2271
2326
1887
1663
ProdLDA[43]
4489
5784
5794
4386
ETM(18]
3215
3916
3095
2370
MAML-ProdLDA
4372
3951
4463
3863
MAML-ETM
3186
3315
2778
4062
Sata-SawETM[30]
3761
3037
2577
3365
CombinedTM[21]
2267
2481
1765
2473
MeroShotTM[22]
2397
2496
1638
2497
Meta-CETM
1517
1440
1306
1576
F22 Topic quality results
In Sec. 3.2.2, we present the topic interpretability results including topic diversity (TD) and topic coherence (TC) of six compared methods. Except for CombinedTM [21] and ZeroShotTM [22], we carry on experiments applying another contextual topic model (CTM) CETopicTM [24] with SimCSE pretrained word embeddings on four datasets. The results exhibited in Fig. 7. can be notably noticed CETopicTM [24] achieves much competitive results on both TD and TC scores, even compared with CombinedTM [21] and ZeroShorTM [22]. Such superiority is owed to the fact that CETopicTM utilizes word embeddings learned from large-scale BERT data and it performs clustering on sentence embeddings to generate topics. In our settings, the aim is to provide a framework for training a sufficiently generalized topic model in low-resource regimes, while equipped with BERT embeddings, CETopicTM highly likely obtain context-related meanings in advance under most situations. But in some cases where the words or the word meanings have not been encountered or
19

---

Table 7: PPL results four datasets with 100 documents in each task.
Methods
20NG
Yahoo
DB14
WOS
LDA[42]
2118
2833
1858
1896
PFA[12]
2060
2169
1637
1643
ProdLDA[43]
4466
5736
6016
4369
ETM[18]
3199
2811
2837
3296
MAML-ProdLDAA
4359
4202
4381
3845
MAML-ETM
3172
3256
2715
3947
Meta-SawETMi3:
3661
3251
2984
3101
[2mbinedTMM21]
2205
2286
1695
2330
[eroShocTM[22]
2330
2319
1604
2372
Meta-CETM
2138
1743
1468
1786
learned by BERT, such as some specialized occasions, CETopicTM may fail to extract interpretable topics.
MAML-ProdLDA
MAML-ETM
Meta-SaWETM
TombinedTM
ZeroShorTM
CETopicTM
Meta-CETM
0.8
0.6
0.4
0.2
20NG
Yahoo
DB14
WOS
0.8
0.6
0.4
0.2
-02
-0.4
-066
-08
-1
Figure 7: Topic diversity results (top row) and topic coherence results (bottom row) of seven compared methods on four datasets, Compared with Fig. we the results of CETopicTM [24] in this figure.
F33 Topic visualization results
In Fig. 3, we visualize the adapted embedding space of different methods to demonstrate Meta- CETM's successful fast adaption. Further, to better characterize meaningful and coherent topics learned by our model given few number of documents, we display the text and topics extracted by Meta-SawETM [30], CombinedTM [21] and our Ceea in Fig. 8.
20

---

What sports would you say it is easy and difficult to be rookie in? The sports that I would say is difficult to be rookie in basketball, hockey, and soccer. The easiest sport for rookie certainly the NFL. Some the greatest NFL players started out on special teams. They have put much efforts to help the team win the game, especially the super bowl
Meta-SawETM
CombinedTM
Meta-CETM
music like
school help nfl
champion weekend
play players
basketball nfl
soccer
rookie nfi
air
ice team
nfl
ball
teams
perfect
friends fort effort hockey
working
baseball
sport
team
players
time
sports
believe
greatest
rookie
game
team
player
people
breaks
putting
super
Figure 8: A paragraph of text and top five words of three topics from Meta-SawETM,, CombinedTM and our Meta-CETM.. can be clearly found that Meta-CETM learns the most relevant topics among the three models.
21


Reviews:
[{"review_id": "cXtmKi1Ift", "reviewer": "Reviewer_Jyas", "summary": "This paper proposes a method for few-shot topic modeling. Specifically, rather than following traditional wisdoms to learn static word embeddings for all the tasks/domains, the authors allow task-specific word representations such that the knowledge from the source task can be better transferred to a target task. The authors also employ Gaussian mixture prior with EM algorithm to capture the clustering structure of distributed word representations. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "1. In Eq 1, is it $Z^{(i)\\intercal}Z^{(i)}$ or $Z^{(i)}Z^{(i)\\intercal}$? What is the relationship between $\\hat{A}$ and $A$?\n2. In the introduction line 49, the authors claim that \"task-specific semantic graphs between words using well-established dependency parsing tools\". However, I fail to find any information about the semantic graph using parsing tools in the rest of the paper.", "limitations": "The limitations are not discussed in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method for few-shot topic modeling looks novel and intuitive to me.\n2. The experiment looks inclusive and the proposed method achieved leading performance.", "weaknesses": "1. Though the paper is generally well-written, some parts are confusing to me. Please refer to the questions part.\n2. [1] also follows a clustering perspective for topic modeling and the difference between [1] and the proposed method can be discussed.\n\n\n[1] Effective Neural Topic Modeling with Embedding Clustering Regularization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your constructive comments and feedback. The weaknesses have been addressed below.\n\n**Q1:** In Eq 1, is it  ${Z^{(i)}}^{\\top}Z^{(i)}$ or $Z^{(i)}{Z^{(i)}}^{\\top}$? What is the relationship between $\\hat{A}$ and $A$?\n\n**A1:** Since we assume $Z^{(i)} \\in \\mathbb{R}^{D \\times V}$ in the article, it should be ${Z^{(i)}}^{\\top}Z^{(i)}$ in Eq. 1, which corresponds to the generated adjacency matrix $A^{(i)} \\in \\mathbb{R}^{V \\times V}$. Thank you for pointing out our typo here. In addition, we use a variational graph autoencoder to model the adjacency matrix $A$ of the semantic graph, following the standard encoder-decoder architecture shown in Fig. 2, $\\hat{A}$ can be viewed as the reconstruction of $A$, *i.e.*, $A$ is the ground-truth adjacency matrix and $\\hat{A}$ represents the predicted adjacency matrix.\n\n**Q2:** In the introduction line 49, the authors claim that \"task-specific semantic graphs between words using well-established dependency parsing tools\". However, I fail to find any information about the semantic graph using parsing tools in the rest of the paper.\n\n**A2:** We apologize for not covering this part of the information, and we will add the corresponding details in the revision. Actually, we build task-specific semantic graphs with the help of **spaCy**, a library for advanced natural language processing that provides a variety of linguistic annotations to give us insights into texts' grammatical structure. Concretely, for a specific task, we use the built-in syntactic dependency parser to analyze each document, and the resulting dependency labels describe the relations between individual tokens, like a subject or object, which also become the basis for constructing the task-specific semantic graph. For example, if a dependency label is assigned between two vocabulary terms in any document, we add an edge between the corresponding nodes in the semantic graph. Conversely, if two vocabulary terms are not assigned a dependency label in all documents, then there are no edges between the corresponding nodes in the graph. Fig. 1 of our Supplementary Materials illustrates the constructed task-specific semantic graphs.\n\n**W2:** [1] also follows a clustering perspective for topic modeling and the difference between [1] and the proposed method can be discussed.\n\n**Discussion:** While both [1] and our method follow a clustering perspective to learn topics, the focus and target problems they aimed at are different. On the one hand, the starting point of [1] is the phenomenon of \"*topic collapsing*\", a common issue that plagues most existing topic models. And its solution is to regularize topic embeddings as cluster centers and word embeddings as cluster samples on the ground of ETM [2]. It assigns all words properly to each topic by solving a well-defined optimal transportation problem. On the other hand, our method strives to solve the problem of learning topics effectively from only a few documents. And the starting point is to learn adaptive word embeddings whose semantics can be well adapted to the given task by using extra contextual grammar information. Therefore, we posit a graph autoencoder to model the semantic dependency graph. To avoid learning repetitive topics and ensure learned topic distributions cover as many significant words as possible for the given task, we impose a GMM prior on the word latent space such that the adaptive word embeddings can be reasonably encapsulated by the topic distributions.\n\n[1] Effective Neural Topic Modeling with Embedding Clustering Regularization   \n[2] Topic Modeling in Embedding Spaces"}, {"review_id": "4Q85wQGJho", "reviewer": "Reviewer_fyGx", "summary": "This paper addresses the problem of inducing neural topic models in low-resource regimes by learning adaptive word embeddings that exploit contextual grammar information. The adaptive word embeddings are learnt with a variational graph autoencoder and the topics are formed from a gaussian mixture prior of the latent word embedding space that generates the observed words and their semantic relations.  The paper introduces a variational inference algorithm to estimate the document- and topic-specific latent variables using MLP and MLP with attention, respectively. In addition, GCN is employed to learn the latent word representations given initial word embeddings and word relations derived from a neural dependency parser. Finally, the parameters of the GMM are learned independently via EM. Evaluation shows strong empirical results in few-shot topic modeling and document classification.", "questions": "See weaknesses section. ", "limitations": "No limitations were discussed", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The main contribution of the paper is the introduction of contextual semantic information into the neural topic model. Empirical results suggest that this approach is beneficial both in terms of per-holdout-word perplexity and topic coherence while remaining competitive in terms of topic diversity. Strong results are also observed in few-shot classification. The paper is fairly well-written but the baselines could have been better explained either in section 3.1 or the related work.\n", "weaknesses": "I cant parse eq. 10 in the evalution: what is the superscript (1)s?\nIn the experiments in section 3.2.3 (few-shot classification), I could not follow the evaluation scheme described in lines 238-239. How were these experiments conducted? How were the class-specific topic-word matrices calculated, what data was used for this, and how was the reconstruction error measured? Also, I could not find the results from this analysis. The results in Table 2 correspond to datasets that have ground truth topic labels.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the quality of our work! We will take your suggestions to give a more detailed explanation of the baselines in the revision. Here we clarify some of your questions.\n\n**Q1:** I can't parse eq. 10 in the evaluation: what is the superscript (1)s?\n\n**A1:** Eq. 10 is the formula for calculating the per-holdout-word perplexity, where $\\phi \\in \\mathbb{R}^{V \\times K}$ denotes the topic-word matrix and $\\theta \\in \\mathbb{R}^{K \\times N}$ represents the topic proportion matrix, and the superscript $s$ indicates the index of the $i^{th}$ collected sample. Here we use the notation $\\phi^{(1)s}\\theta^{(1)s}$ for the consideration of hierarchical topic models, which have multiple layers of document representations. By using the superscript (1), we mean that we use the representation of the bottom layer to compute the perplexity. Indeed, for most regular topic models with only a single layer of document representation, we can omit the superscript (1) and write the notation as $\\phi^{s}\\theta^{s}$.\n\n**Q2:** In the experiments in section 3.2.3 (few-shot classification), I could not follow the evaluation scheme described in lines 238-239. How were these experiments conducted? How were the class-specific topic-word matrices calculated, what data was used for this, and how was the reconstruction error measured? Also, I could not find the results from this analysis. The results in Table 2 correspond to datasets that have ground truth topic labels.\n\n**A2:** Above all, the few-shot classification experiment requires available ground truth topic labels, which are used to compute the accuracy. Next, we elaborate on the evaluation scheme described in lines 238-239 with an example of a 5-way 5-shot classification task. Specifically, the data of a 5-way 5-shot task consists of a \"*support set*\", which includes 25 documents from 5 different topics (5 for each topic), and a \"*query set*\", which contains 15 documents from each of the 5 topics. Our goal is to train a task-specific classifier using the \"*support set*\" and compute the classification accuracy on the \"*query set*\". So for the well-trained topic models, we use five documents of each topic in the \"*support set*\" to adapt a topic-word matrix, respectively; the resulting five topic-word matrices, denoted as {$\\phi_1$, $\\phi_2$, $\\phi_3$, $\\phi_4$, $\\phi_5$}, are called class-specific topic-word matrices (here a topic refers to a category). Then for each document in the \"*query set*\", we use the trained topic model to derive its topic proportion $\\theta_q$, which is subsequently combined with each of the five class-specific topic-word matrices to calculate the data likelihood $\\\\{ p(x_q | \\theta_q, \\phi_i) \\\\}_{i=1}^5$, $x_q$ is the BoW of the query document. The reconstruction error is defined as the negative data likelihood, so we classify it as the topic with the smallest reconstruction error."}, {"review_id": "gu9pGE7Zhw", "reviewer": "Reviewer_biyw", "summary": "The authors present a new neural topic model which aims to solve the problem of learning task-specific word embeddings in a low resource scenario.  In addition to a somewhat typical neural TM, dependency graphs are collected using parsers, and embedded via GCNs to produce adaptive word embeddings.  A topic-word matrix then models the task-specific distribution over words as a gaussian over their adaptive embeddings.  The topic-specific embedding spaces capture the role a word is likely to play within that context.  The model is evaluated across 4 datasets and a number of metrics, where it shows strong improvements in perplexity, and good performance in document classification.", "questions": "(mixed with weaknesses)", "limitations": "None", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- strong empirical performance in perplexity, and good performance in document classification.\n\n- comparisons to many existing similar models.  The improvements over models using sentence BERT are especially interesting.  Good ablations.", "weaknesses": "- reliance on pre-trained parsing tools.  How well does this approach work in different languages, or styles/domains of text that differ significantly from the parsing training data?\n\n- the problem at the heart of this work is lexical ambiguity.  Here parsing, and then graph-based embedding of the parse, aim to find task-specific meanings of each word, and there is little doubt that adaptive word embeddings are an effective approach to this problem.  Outside of topic modeling, LLMs and other transformer-based models solve this same task, and also refine word-specific embeddings into context-specific ones.  The only aspect in which this is included is via CombinedTM/ZeroShotTM.  Admittedly, I haven't worked on topic modeling since the switch to neural models, but it would seem that BERT would also solve the issues being pursued in this work.  I didn't see a compelling explanation for why this approach based on dependency graphs is an improvement over generic transformer-based sentence embeddings, all other aspects remaining the same.\n\n- even if the work is geared towards low resource settings, I would like to see the performance as a factor of task dataset size.  In its current presentation I think the performance is compelling enough to warrant acceptance, as the performance improvement seems sufficient across a number of metrics to be of great interest to the topic modeling community.  However, if those performance margins decrease significantly (or completely) as dataset size increases to even modest sizes, it would be difficult to find a use-case for this approach.  It would be the story of perhaps offloading some of the learning problem to the pre-trained parsing and priors over their embeddings, which solves a practical problem, but not so much one of academic interest.\n\n- discussion of conceptually related work is sparse.  The related works section is overly brief given the amount of work that solving similar core tasks.  One earlier work that seems spiritually related might be [1].\n\n- surprisingly CNN > MLP is a more important design decision than any architectural change represented across several previous papers.  It raises a little bit of an alarm bell regarding the importance of these architectural designs as evaluated here.  Extending the document classification task to more datasets (than the 2 here) would increase confidence in the importance of the work.\n\nL26: vontextualized\n\n[1] Word Representations via Gaussian Embedding\nhttps://arxiv.org/abs/1412.6623\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your careful consideration and valuable comments. In the following, we are going to try our best to address your concerns.\n\n**W1:** reliance on pre-trained parsing tools. How well does this approach work in different languages, or styles/domains of text that differ significantly from the parsing training data?\n\n**R1:** Thank you very much for raising a very meaningful question. However, we think it is outside the scope of this paper, and we will keep it as a focus of our future work.\n\n**W2:** The problem at the heart of this work is lexical ambiguity ... I didn't see a compelling explanation for why this approach based on dependency graphs is an improvement over generic transformer-based sentence embeddings, all other aspects remaining the same.\n\n**R2:** On the one hand, the remarkable capabilities of pre-trained language models such as BERT rely on the massive training corpus (e.g., BERT was trained on a dataset of over 3.3 billion words). In contrast, we present a data-efficient framework that only relies on a batch of training tasks (about 5 million words for 20NG) to produce well-fitted word embeddings for each task. On the other hand, due to the strong biases introduced by the training corpus (even though it may cover a wide range of content), pre-trained language models may not be so effective in adapting to a completely unfamiliar context. Whereas our proposed framework has been carefully designed to learn how to effectively adapt to novel tasks from an unfamiliar corpus, it should perform better as a few-shot learner.\n\n**W3:** Even if the work is geared towards low resource settings, I would like to see the performance as a factor of task dataset size ... However, if those performance margins decrease significantly (or completely) as dataset size increases to even modest sizes, it would be difficult to find a use-case for this approach. It would be the story of perhaps offloading some of the learning problems to the pre-trained parsing and priors over their embeddings, which solves a practical problem, but not so much one of academic interest.\n\n**R3:** We have conducted additional experiments by varying the number of documents in each task from 20 to 50 to 100. Please refer to our unified response to all reviewers (**Author Rebuttal by Authors**) to see the corresponding results.\n\n**W4:** Discussion of conceptually related work is sparse. The related works section is overly brief given the amount of work that solving similar core tasks. One earlier work that seems spiritually related might be [1].\n\n**R4:** We agree that there has been sparse discussion of conceptually related work, and we will add more abundant discussions about the corresponding works in the revision, including the one [1] you mentioned.\n\n**W5:** surprisingly CNN > MLP is a more important design decision than any architectural change represented across several previous papers. It raises a little bit of an alarm bell regarding the importance of these architectural designs as evaluated here. Extending the document classification task to more datasets (than the 2 here) would increase confidence in the importance of the work.\n\n**R5:** Yeah, we are also surprised by the experimental results showing that CNN > MLP is a more important design decision in our setup, perhaps because our input data is the bag-of-words (BoWs) representations of the documents. Anyway, to increase the credibility of this finding, we conduct document classification experiments on additional datasets, *i.e.,* Yahoo and WOS. The results are listed in the following table. \n\n||Yahoo|||WOS||\n|:--|:--:|:--:|:--:|:--:|:--:|\n|**Methods**|**5way-5shot**|**5way-10shot**||**5way-5shot**|**5way-10shot**\n|MAML (MLP)|45.42|51.00||37.77|40.43|\n|PROTO (MLP)|50.01|56.16||39.61|41.46|\n|FT (MLP)|48.59|53.06||36.52|37.22|\n|FT$^{*}$ (MLP)|50.73|56.74||45.02|51.20|\n|MAML (CNN)|48.81|56.50||47.28|57.32|\n|PROTO (CNN)|53.16|63.66||59.05|**67.75**|\n|FT (CNN)|56.78|66.04||54.68|63.39|\n|FT$^{*}$ (CNN)|53.28|52.56||51.42|61.98|\n|HNS-SawETM|52.35|57.86||42.09|56.91|\n|Meta-SawETM|52.45|60.58||43.39|57.44|\n|CombinedTM|57.94|64.75||56.16|65.97|\n|ZeroShotTM|58.12|66.21||58.50|66.10|\n|Meta-CETM|**63.84**|**72.67**||**61.47**|67.62|\n\n[1] Word Representations via Gaussian Embedding https://arxiv.org/abs/1412.6623"}, {"review_id": "RGc811tZZv", "reviewer": "Reviewer_4MPJ", "summary": "The authors target the problem of multi-meaning words across different tasks for topic models, particularly under low-resource settings. To this end, they propose a variational graph autoencoder with a trainable Gaussian mixture prior to capture the distribution of task-specific word embeddings.", "questions": "- Could you supply more performance comparison of the proposed method with the previous one, especially those utilizing word embeddings for topic modeling?\n\n- Could you in more detail discuss the significance of the Gaussian mixture model as the prior distribution or clarify its benefits over other prior distributions?", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the paper is sound with several strengths:\n\n- The authors address the low-resource regime that has received little attention from the topic modeling research recently.  \n\n- The experiments are encyclopedically covered with detailed discussions.", "weaknesses": "However, the paper exhibits some weaknesses:\n\n- The examples illustrating the applications of few-shot neural topic models are not persuasive. In particular, few-shot topic modeling might not be employed to learn users’ past purchases or online behaviors. In addition, during crises, e.g. Covid-19, the number of documents towards certain topics would rather burgeon than remain limited, hence invalidating the need for few-shot topic models. \n\n- The paper needs more comparison with prior embedding-based topic models [1, 2]. Additional evaluation of the model performance against such works could make the experiments more convincing. \n\n- The choice of prior distribution plays an important role in specifying the latent space. In the paper, the authors have not elaborated on why the Gaussian mixture model is selected as the prior, or provided experiments to demonstrate its advantage over other prior choices. \n\n[1] Neural models for documents with metadata (Card et al., 2018)\n\n[2] Neural topic models via optimal transport (Zhao et al., 2021)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful readings and valuable comments. We believe the constructive feedback will improve the paper and increase its potential impact on the community. Regarding the weaknesses you mentioned, we respond as follows.\n\n**W1:** The examples illustrating the applications of few-shot neural topic models are not persuasive. In particular, few-shot topic modeling might not be employed to learn users’ past purchases or online behaviors. In addition, during crises, e.g. Covid-19, the number of documents towards certain topics would rather burgeon than remain limited, hence invalidating the need for few-shot topic models.\n \n**R1:** We are sorry that you think the examples illustrating the applications of few-shot neural topic models are not persuasive enough. But the few-shot topic models do have their own value in real-world applications. Take the example you listed. To this day, there has indeed been a substantial increase in the number of documents on the topic of \"Covid-19\". However, at the very beginning of this epidemic, there were relatively few cases of infection in all regions, and the reports of related medical diagnoses were very limited. Under such circumstances, few-shot topic models can help us extract key information from limited resources, thus facilitating us to take appropriate preventive and control measures.\n\n**W2:** The paper needs more comparison with prior embedding-based topic models [1, 2]. Additional evaluation of the model performance against such works could make the experiments more convincing.\n\n**R2:**  Based on your suggestion, we compared our model with SCHOLAR [1] and NSTM [2], two prior embedding-based topic models on all four datasets with document size 10. The results are listed in the following. \n\n|20NG|PPL|TD|TC| \n|:--|:--:|:--:|:--:|\n|ETM [a]|3107|0.8395|-0.8437|\n|Meta-SawETM [b]|2984|0.5643|-0.6086|\n|SCHOLAR [1]|4371|0.5779|-0.6792| \n|NSTM [2]|3190|0.7008|-0.6202| \n|Meta-CETM|1170|0.8154|-0.3701|\n\n|Yahoo|PPL|TD|TC| \n|:--|:--:|:--:|:--:|\n|ETM [a]|2817|0.8851|-0.8913|\n|Meta-SawETM [b]|2365|0.5465|-0.6406|\n|SCHOLAR [1]|4697|0.4981|-0.7429| \n|NSTM [2]|3153|0.7505|-0.6351| \n|Meta-CETM|1219|0.7886|-0.4639|\n\n|DB14|PPL|TD|TC| \n|:--|:--:|:--:|:--:|\n|ETM [a]|3054|0.8106|-0.8719|\n|Meta-SawETM [b]|1914|0.7545|-0.9204|\n|SCHOLAR [1]|4913|0.5112|-0.8217| \n|NSTM [2]|3379|0.6195|-0.7626| \n|Meta-CETM|1084|0.7475|-0.4783|\n\n|WOS|PPL|TD|TC| \n|:--|:--:|:--:|:--:|\n|ETM [a]|3310|0.9286|-0.9785|\n|Meta-SawETM [b]|2253|0.7217|-0.7475|\n|SCHOLAR [1]|3884|0.6413|-0.7434| \n|NSTM [2]|3164|0.7659|--0.6472| \n|Meta-CETM|1293|0.8667|-0.5177|\n\n**W3:** The choice of prior distribution plays an important role in specifying the latent space. In the paper, the authors have not elaborated on why the Gaussian mixture model is selected as the prior, or provided experiments to demonstrate its advantage over other prior choices.\n\n**R3:** To explain clearly the motivation for using a GMM prior, we address two key questions. **1)** why do we use a variational graph autoencoder instead of a graph autoencoder? **2)** why do we adopt a Gaussian mixture prior instead of a standard normal prior? Actually, all of these choices serve one purpose, *i.e.,* to alleviate the issue of \"*topic collapsing*\". We found that if the learning of topic embeddings is not constrained, it will lead to highly repetitive topics (the content of all topics tends to be the same). Therefore, we use a variational GAE in the hope of regularizing the word latent space. Then we use Gaussian embeddings to represent topics to model the uncertainty, with a standard normal distribution as the prior. Even so, we found the learned topics are not diverse enough; thus, we rely on the GMM prior to further overcome the problem. **Please refer to our newly added one-page PDF for corresponding visualization and numerical results.**\n\n\n[1] Neural models for documents with metadata (Card et al., 2018)\n\n[2] Neural topic models via optimal transport (Zhao et al., 2021)\n\n[a] Topic modeling in embedding spaces (Dieng et al., 2020)\n\n[b] Bayesian deep embedding topic meta-learner (Duan et al., 2022)"}, {"review_id": "5sXSEugVO7", "reviewer": "Reviewer_VS4z", "summary": "This paper proposed an approach for few-shot topic modeling. The authors first question the limitations of \"static word embeddings\" in previous related work when transferring to new tasks, and then propose to use the \"adaptive word embeddings\" generated by VGAE to address this issue. Although the problem this paper aims to solve is meaningful, I think the description of the motivation is not clear enough. Even though the experimental results are acceptable, the methods used do not reflect the motivation of the paper.", "questions": "Please see the Weaknesses.", "limitations": "Please see the Weaknesses.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Few-shot learning for topic modeling is a meaningful problem.\n\n- By combining VGAE with the neural topic model, fairly good experimental results were achieved.", "weaknesses": "\tThis paper aims to address the limitations of \"static word embeddings\". In line 9, the authors claim that 'we introduce a variational graph autoencoder to learn task-specific word embeddings based on the dependency graph refined from the context of each task'. However, why would the dependency graph and VGAE help in learning task-specific word embeddings? If this claim holds true, simply using the context of each task could achieve the same purpose. In other words, I don't see how VGAE and the dependency graph would aid in learning the so-called 'adaptive word embeddings'. The authors should carefully explain why the dependency graph and VGAE can reflect the characteristics of each task. Would the dependency graph for each task be fundamentally different? In summary, I find this motivation unconvincing.\n\n\tIn line 14, the authors state that the Gaussian mixture prior can \"facilitate the discovery of diverse topics and the quick adaptation to novel tasks.\" However, why using a Gaussian mixture prior can help the fast adaptation to novel tasks? What advantages does it have over the methods used in existing neural topic models? The authors should explain the principles that make the Gaussian mixture prior to work, that is, what the motivation is for using a Gaussian mixture prior, rather than vaguely stating that it can help with adaptation. I agree that the authors provide a perspective on learning topics through clustering by using a Gaussian mixture prior, but the motivation they explained is not convincing. \n\n\tSome sentences are overly long, making this paper hard to follow. For instance: Lines 24-27.\n\n\tIn lines 57-66, the author's description of the contributions of this paper. The first and the second points are repetitive.\n\n\tIn lines 45-46, \"it is experimentally found ...\". This statement is vague and confusing. If it's a finding from the experiments conducted in this paper, then the results should be presented. If it's an experimental finding from previous work, then a reference should be provided.\n\n\tAccording to the introduction and related work, this paper is an important existing work on few-shot topic modeling: \"Few-shot learning for topic modeling\". So why did the authors not compare the methods of this work in their experiments? The authors should give a reason.\n\n\tWhat does ϕ stand for in Equation 10? This paper does not give an explanation.\n\n\tThe author demonstrated the effectiveness of the proposed method in the experiment. However, I believe there are some flaws in the experiment. Firstly, since the author believes that VGAE is the main reason for acquiring \"adaptive word embeddings\", then the visualization results of the model without VGAE should also be provided in Figure 3, in order to prove the role of VGAE in learning \"adaptive word embeddings\", which is a major motivation of this paper.\n\n\tSecondly, the ablation study is not comprehensive enough. On the one hand, the author should provide the results on all four datasets. On the other hand, the author should also provide the results of separately removing the context variable, graph vae, and GMM prior, in order to clearly prove the effectiveness of each module proposed in this work.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your careful consideration and valuable comments. In the following, we are going to try our best to address your concerns.\n\n**W1:** This paper aims to address the limitations ... The authors should carefully explain why the dependency graph and VGAE can reflect the characteristics of each task. Would the dependency graph for each task be fundamentally different? In summary, I find this motivation unconvincing.\n\n**R1:** Above all, we would like to clarify that our method is able to learn \"adaptive word embeddings\" because we utilize contextual information that is closely related to each task. In our problem setup, each task consists of only a handful of documents from a specific domain, which could lead to a world of differences between the contexts of different tasks. For instance, in a task related to \"hardware\", the context of the word **bus** is most likely to cover the words \"data\", \"transmitted\", \"cache\", and so on. Whereas in a task associated with \"autos\", the words that most often co-occur with the word **bus** are probably \"car\", \"taxi\", \"passenger\", etc. An illustrated example can be found in our Supplementary Materials. We believe that this contextual information is very instrumental in capturing the precise meanings of words, so we refine it into a dependency graph and are able to learn \"adaptive word embeddings\" with the help of the VGAE. In other words, the dependency graph incorporating task-specific contextual information can reflect the characteristics of each task.\n\n**W2:** In line 14, the authors state that the Gaussian mixture prior can ... However, why using a Gaussian mixture prior can help the fast adaptation to novel tasks? What advantages does it have over the methods used in existing neural topic models? The authors should explain the principles ... but the motivation they explained is not convincing.\n\n**R2:** To explain clearly the motivation for using a GMM prior, we address two key questions. **1)** why do we use a variational graph autoencoder instead of a graph autoencoder? **2)** why do we adopt a Gaussian mixture prior instead of a standard normal prior? Actually, all of these choices serve one purpose, *i.e.,* to alleviate the issue of \"*topic collapsing*\". We found that if the learning of topic embeddings is not constrained, it will lead to highly repetitive topics (the content of all topics tends to be the same). Therefore, we use a variational GAE in the hope of regularizing the word latent space. Then we use Gaussian embeddings to represent topics to model the uncertainty, with a standard normal distribution as the prior. Even so, we found the learned topics are not diverse enough; thus, we rely on the GMM prior to further overcome the problem. **Please refer to our newly added one-page PDF for corresponding visualization and numerical results.**\n\n**W3:** Some sentences are overly long, making this paper hard to follow. For instance ... If it's an experimental finding from previous work, then a reference should be provided.\n\n**R3**: We agree with you about these flaws in writing and expression, and we will improve our presentation in the revision to make it easier to follow.\n\n**W4:** According to the introduction and related work, this paper is an important existing work on few-shot topic modeling: \"Few-shot learning for topic modeling\". So why did the authors not compare the methods of this work in their experiments? The authors should give a reason.\n\n**R4:** We found that the authors of \"Few-Shot Learning for Topic Modeling\" did not open-source their code. Indeed, we have also tried to contact the authors by email but did not get any response, so we do not compare the method of this work in our experiments. However, we have compared a more recently developed approach with strong performance – \"Bayesian Deep Embedding Topic Meta-Learner\".\n\n**W5:** What does $\\phi$ stand for in Equation 10? This paper does not give an explanation. \n\n**R5:** We apologize for our carelessness. $\\phi$ in Eq. 10 stands for the topic-word matrix, which is actually denoted by $\\boldsymbol{\\beta}$ in our method.\n \n**W6:** The author demonstrated the effectiveness of the proposed method in the experiment. However, I believe there are some flaws in the experiment. Firstly ... to prove the role of VGAE in learning \"adaptive word embeddings\", which is a major motivation of this paper.\n\n**R6:** We think there may be some misunderstanding about the role of VGAE. Indeed, the task-specific dependency graph is the main reason for acquiring \"adaptive word embeddings\" (refer to our response to **W1**). And VGAE is a mapping function that serves as a bridge. If we remove the VGAE module from our model, then contextual information relevant to each task will not be exploited, as the dependency graph is the input to VGAE. However, introducing additional contextual information closely related to each task is precisely the biggest innovation and contribution of our method. If we do not use the dependency graph and only rely on the BoWs of the documents to learn our model, it should perform comparably to the baseline method ETM. Since there is no information increment in this case, the only difference between our model and ETM is the consideration of one more context variable.\n\n**W7:** Secondly, the ablation study is not comprehensive enough. On the one hand, the author should provide the results ... in order to clearly prove the effectiveness of each module proposed in this work.\n\n**R7**: As you mentioned that the ablation study is not comprehensive enough, we have further perfected the ablation study, and the corresponding results are exhibited in our newly added one-page PDF. Please note that the results we report for separately removing the graph vae do not mean that we have completely discarded the VGAE module but have replaced it with a graph autoencoder. The reason for doing so has been explained in our response to **W6**."}, {"review_id": "zYvhwbIr1s", "reviewer": "Reviewer_CvqN", "summary": "This paper proposes a new solution (Meta-CETM) for inferring topics on a dataset with a few available documents only. The main idea is to train the model on various tasks and then use it on a new, small dataset. The are extensive experiments on diverse topic models, in particular in this context of \"few shot\" learning, and the results show that Meta-CETM outperforms its competitors.", "questions": "I have some concerns on this paper.\n\n- The paper is not always well presented. Here I give some examples:\n  * The problem formulation (2.1) is quite confusing, probably because of the limited space. This is much clearer in the supplementary material. Some aspects are not sufficiently described or motivated. For instance, the authors seem to use a self attention mechanism similar to the one in the Transformer (noted Attn). However, this mechanism is normally based on an embedding matrix, which is not the case here for X is a simple BoW representation of the documents.\n  * Why using the Weibull distribution in this context? There is no explanation at all. \n  * What \"latent indication\" means for H(i)?\n  * I guess the step 1.c line 114 is done after having computed all the word embedding of the generated document (we need all the Z(i)). Is it true? If so, this step should be taken outside the for loop.\n\n- Would it be possible to use the adaptive word embedding module (Z(i)) for the bag-of-word encoder?\n\n- The experimental part of the paper looks quite strong. Important solutions for topic modeling (old and new) are considered in the competitors and the authors use several manners to compete with them (perplexity, classification, topic coherence and diversity, qualitative evaluation). However, I would expect more experiments on the few-shot context, for instance by varying the number of available documents (not only 5 or 10) and taking into account the semantic difference between the corpora (i.e., between the training set and the test set). ", "limitations": "There is not a single limitation mentioned in the paper... There is no future work as well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- few-shot learning for topic modeling is an important topic today\n- the model looks sound (even though not really well explained)\n- the experimental framework is quite strong, with good results for the proposed solution\n", "weaknesses": "- The solution is not always well presented\n- It's a quite complicated model with many intertwined modules", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and helpful comments and suggestions. Your concerns have been addressed as follows.\n\n\n**Q1:** The paper is not always well presented. Here I give some examples:\n\n- *Q1.1:* The problem formulation (2.1) is quite confusing ... For instance, the authors seem to use a self-attention mechanism similar to the one in the Transformer (noted Attn). However, this mechanism is normally based on an embedding matrix, which is not the case here for X is a simple BoW representation of the documents.   \n*A1.1:* First, we would like to clarify that some parts, including the notation table, training and testing algorithms, illustration of our settings, and some derivations, are put into the supplementary materials due to the space limit. Secondly, we explain your questioning of the self-attention mechanism. We must apologize for the mistake in Eq. 5 due to our carelessness. Indeed, our self-attention operation does perform on an embedding matrix $\\mathbf{H}^{(i)}\\in \\mathbb{R}^{D \\times J}$, which is obtained by feeding BoW representations $\\mathbf{X}^{(i)}\\in \\mathbb{R}^{V\\times J}$ of the documents into a multi-layer perceptron (MLP), where $V$ is the vocabulary size, $J$ is the number of documents in each task, and $D$ is the dimension of the extracted BoW features. Finally, we apply self-attention mechanism $\\mathrm{Attn}()$ to aggregate the information of $J$ document features to infer the posterior of context variable $\\pmb{c}^{(i)}\\in \\mathbb{R}^{K}$. Hence, the correct version of Eq. (5) should be written as follows (we will fix this error in the revision):\n$$ q(\\pmb{c}^{(i)}|\\mathbf{X}^{(i)}) = \\mathcal{N}(\\pmb{\\mu}\\_{\\pmb{c}^{(i)}}, \\pmb{\\Sigma}\\_{\\pmb{c}^{(i)}}); \\pmb{\\mu}\\_{\\pmb{c}^{(i)}}, \\pmb{\\Sigma}\\_{\\pmb{c}^{(i)}} =\\mathrm{Attn}(\\mathbf{H}^{(i)}); \\mathbf{H}^{(i)}=\\mathrm{MLP}(\\mathbf{X}^{(i)}).$$  \n\n- *Q1.2:* Why using the Weibull distribution in this context? There is no explanation at all.   \n*A1.2:* Sorry, we do not find any part of our model involved in the Weibull distribution. But one of our compared works, Meta-SawETM [1], utilizes the Weibull distribution for posterior approximation.\n\n\n- *Q1.3:* What \"latent indication\" means for H(i)?   \n*A1.3:* Please refer to our answer to *Q1.1*, we call the embedding matrix $\\mathbf{H}^{(i)}\\in \\mathbb{R}^{D \\times J}$ as \"latent indication\", which stands for the extracted deterministic features of BoW representations.\n\n- *Q1.4:* I guess the step 1.c line 114 is done after having computed all the word embedding of the generated document (we need all the Z(i)). Is it true? If so, this step should be taken outside the for loop.   \n*A1.4:* Yeah. You are correct in your understanding, and step 1.c (line 114) should be taken outside the for loop. Thank you for pointing out this error, and we will fix it in the revision.\n\n\n**Q2:** Would it be possible to use the adaptive word embedding module (Z(i)) for the bag-of-word encoder?\n\n**A2:** The question you posed is a bit confusing to us. For most neural topic models, the BoW encoder is typically used to extract document representations (or topic proportions) based on the word frequencies, while the adaptive word embeddings $\\mathbf{Z}^{(i)}$ in our paper are derived by modeling a task-specific semantic graph with a variational GAE. We do not understand exactly what it means to use the adaptive word embedding module for the bag-of-word encoder. Do you mean directly using the variational GAE combined with the semantic graph to learn the document representations (or topic proportions)?\n\n**Q3:** The experimental part of the paper looks quite strong. Important solutions for topic modeling (old and new) are considered ... However, I would expect more experiments on the few-shot context, for instance, by varying the number of available documents (not only 5 or 10) and taking into account the semantic difference between the corpora (i.e., between the training set and the test set).\n\n**A3:** For varying the number of available documents for each task, we conduct additional experiments on all four datasets with \\{20, 50, 100\\} documents in each task and list the perplexity results of different compared methods in the **Author Rebuttal by Authors** part. As for taking the semantic difference between the training set and the test set into account, we have not performed such experiments due to the time limit, but we will leave it as a priority for our future work.\n\n[1] Bayesian deep embedding topic meta-learner. In ICML 2022."}]

Author rebuttal:
We really appreciate all the reviewers for their constructive and helpful comments. And we apologize for typos, grammar mistakes, unclear notations and missing citations. They will be corrected such that the overall writing meet NeurIPS standards. Here we briefly introduce our newly added rebuttal PDF and also provide additional experimental results.

------
1. In our one-page rebuttal PDF, we exhibit the results of:

- visualizations of adapted embedding space with different prior choices for topic-word matrix $\boldsymbol{\beta}$, including **no prior**, **standard Gaussian distribution prior** and **GMM prior**, as suggested by reviewer **4MPJ** and reviewer **VS4z**.

- ablation study on all four datasets (20NG, DB14, Yahoo and WOS) of separately removing our module designs (context variable, Graph VAE and GMM prior) in Table.1, as suggested by reviewer **VS4z**. The ETM is chosen as our baseline. For removing Graph VAE but keeping the GMM prior, we replace the **Gragh VAE** with a **Graph auto-encoder** to model the task-specific dependency graph. For removing GMM prior but maintaining the Graph VAE module, we apply the **standard Gaussian distribution prior** to replace the **GMM prior**.

------
2. Moreover, as mentioned by reviewer **biyw** and reviewer **CvqN**, we vary the number of documents in each task from \{5, 10\} to \{20, 50, 100\} and provide the perplexity (PPL) results on all four datasets as the following. 

|||20NG||||DB14||
|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|**Methods**|**20**|**50**|**100**||**20**|**50**|**100**|
|LDA|2979|2443|2118||3095|2353|1858|
|PFA|2439|2271|**2060**||1903|1887|1637|
|ProdLDA|4807|4489|4466||5819|5794|6016|
|ETM|3276|3215|3199||2870|2834|2837|
|MAML-ProdLDA$^{*}$|4378|4372|4359||4612|4463|4381|
|MAML-ETM$^{*}$|3287|3186|3172||2819|2778|2715|
|Meta-SawETM|2657|3761|3661||2355|2577|2984|
|CombinedTM|2331|2267|2205||1863|1765|1695|
|ZeroShotTM|2673|2397|2330||1722|1638|1604|
|Meta-CETM(ours)|**1216**|**1517**|2138| |**1109**|**1306**|**1468**|

|||Yahoo||||WOS||
|:--|:--:|:--:|:--:|:--:|:--:|:--:|:--:|
|**Methods**|**20**|**50**|**100**||**20**|**50**|**100**|
|LDA|3916|3279|2833||2370|2091|1896|
|PFA|2545|2326|2169||1675|1663|**1643**|
|ProdLDA|6093|5784|5736||4617|4386|4369|
|ETM|2781|2801|2811||3189|3176|3296|
|MAML-ProdLDA$^{*}$|4033|3951|4202||3908|3863|3845|
|MAML-ETM$^{*}$|3439|3315|3256||4189|4062|3947|
|Meta-SawETM|2859|3037|3251||3620|3365|3101
|CombinedTM|2543|2481|2286||2587|2473|2330|
|ZeroShotTM|2664|2496|2319||2660|2497|2372|
|Meta-CETM(ours)|**1369**|**1440**|**1743**||**1482**|**1576**|1786|



Meta-review: The paper proposes a new method for context-guided embedding adaptation for topic modeling in low-resource scenarios. All reviewers are consistent in accepting the paper, so I also recommend acceptance. However, the authors should address the reviewers' concerns in the revision.
Decision: Accept (poster)