Title: Context-guided Embedding Adaptation for Effective Topic Modeling in Low-Resource Regimes
Venue: NeurIPS 2023
Accepted: yes
Keywords: Few-shot generative model; topic modeling;
Abstract: Embedding-based neural topic models have turned out to be a superior option for low-resourced topic modeling. However, current approaches consider static word embeddings learnt from source tasks as general knowledge that can be transferred directly to the target task, discounting the dynamically changing nature of word meanings in different contexts, thus typically leading to sub-optimal results when adapting to new tasks with unfamiliar contexts. To settle this issue, we provide an effective method that centers on adaptively generating semantically tailored word embeddings for each task by fully exploiting contextual information. Specifically, we first condense the contextual syntactic dependencies of words into a semantic graph for each task, which is then modeled by a Variational Graph Auto-Encoder to produce task-specific word representations. On this basis, we further impose a learnable Gaussian mixture prior on the latent space of words to efficiently learn topic representations from a clustering perspective, which contributes to diverse topic discovery and fast adaptation to novel tasks. We have conducted a wealth of quantitative and qualitative experiments, and the results show that our approach comprehensively outperforms established topic models.

Context-guided Embedding Adaptation for Effective Topic Modeling in Low-Resource Regimes

Abstract

Embedding-based neural topic models have turned out to be a superior option for low-resourced topic modeling. However, current approaches consider static word embeddings learnt from source tasks as general knowledge that can be transferred directly to the target task, discounting the dynamically changing nature of word meanings in different contexts, thus typically leading to sub-optimal results when adapting to new tasks with unfamiliar contexts. To settle this issue, we provide an effective method that centers on adaptively generating semantically tailored word embeddings for each task by fully exploiting contextual information. Specifically, we first condense the contextual syntactic dependencies of words into a semantic graph for each task, which is then modeled by a Variational Graph Auto-Encoder to produce task-specific word representations. On this basis, we further impose a learnable Gaussian mixture prior on the latent space of words to efficiently learn topic representations from a clustering perspective, which contributes to diverse topic discovery and fast adaptation to novel tasks. We have conducted a wealth of quantitative and qualitative experiments, and the results show that our approach comprehensively outperforms established topic models.

1 Introduction

The last two decades have witnessed the enduring popularity of topic models along with their many successful applications in a range of fields [1-7]. And this is predominantly attributed to their ability to reveal the underlying semantic structure from large volumes of textual data. By identifying a group of salient themes, topic models represent each document as a mixture of them, providing an intuitive understanding of the target corpus. Although conventional probabilistic topic models [8-14] have been widely used, new variants continue to spring up in the era dominated by deep neural networks.

Among the proliferation of new methodologies come both the neural topic models (NTMs) [15-18] resulting from the development of variational autoencoders (VAEs) and autoencoding variational Inference (AVI) [19, 20], and the contextualized topic models (CTMs) [21-24] benefiting from the flourishing of pre-trained language models [25, 26]. However, these recently developed approaches essentially maintain the assumption of sufficient resources, i.e. , with a plethora of documents being available. Comparatively, little attention has been paid to topic modeling under resource-limited or resource-poor conditions [27], which plays a significant role in real-world applications.

A case in point occurs in personalized recommendation systems where users' preferences are judged based on only a small amount of their historical data, such as past purchases or online behaviors [28]. Another example arises from crisis management. Since being able to quickly identify and monitor emerging topics during a crisis, e.g. , a public health emergency, could substantially support a government or organization in responding appropriately to rapidly changing situations [29].

While there have also been some beneficial attempts to study few-shot learning for topic modeling, they more or less exhibit certain limitations. For instance, Iwata [27] first proposed a straightforward method that aims to learn good, i.e. , task-specific priors for latent Dirichlet allocation (LDA) [8] by using neural networks. However, this approach suffers from a lack of expressiveness when dealing with difficult novel tasks. Later on, Duan et al. [30] claimed that embedding-based NTMs [31] are naturally superior in generalizing to new and unseen tasks. By considering word embeddings learned from the training task pool as transferable knowledge, they have shown that effective generalization can be achieved by only learning topic embeddings adaptively. Nevertheless, as the word semantics inevitably change with contexts, the learned static word embeddings may not adapt well to the target task with alien contexts. On the other hand, we have experimentally found that the performance of two representative CTMs [21, 22] is also not competitive enough under resource-poor conditions.

To address the above issues, we propose to learn adaptive word embeddings suitable for each task by fully exploiting the contextual grammar information. Concretely, we first construct a task-specific semantic graph between words using well-established dependency parsing tools 3 [32]. This graph helps depict the precise meaning of each word in the given task, which we then model with a created variational graph autoencoder (VGAE) [33], and the resulting latent representations of words merge the contextual information and are thus semantically tailored to the given task. Furthermore, by imposing a Gaussian mixture prior on the latent space of word, we offer a perspective on learning topics through clustering, i.e. , each component of the Gaussian mixture can be viewed as the representation of a topic. Consequently, the adaptive word embeddings tend to be reasonably covered by several clusters, facilitating the discovery of more interpretable topics closely related to the given task.

Our main contributions can be summarized as follows:

To solve the problem of few-shot learning for topic modeling, we propose to learn adaptive word embeddings that are semantically matched to the given task.

To generate adaptive word embeddings for each task, we innovatively introduce a variational graph autoencoder to learn the latent representations of words from the task-specific semantic graph that captures the contextual syntactic dependencies.

To mine more interpretable topics related to the given task, we offer a perspective on learning topics through clustering by imposing a Gaussian mixture prior on the word latent space. Besides, we also develop an efficient variational inference scheme to approximate the posteriors of latent variables.

2 Meta-CETM

2.1 Problem Formulation

To clarify the problem definition of few-shot learning for topic modeling, we assume that there are C training corpora {D c } C c =1 from different domains, with the goal of using them to learn a topic model that generalizes to the test corpus D test in a new domain, i.e. , the resulting topic model is supposed to quickly adapt to a given test task with a few documents and mine topics related to the new domain. Further, we adopt an episodic training strategy as is conventional in most few-shot learning literature. Specifically, we construct a batch of training tasks {T ( i ) } i M =1 to mimic the scenario of giving tasks at test time, so that each task contains only several documents from an arbitrary training corpus. And we denote the bag-of-words (BoW) representations of the task documents as X ( i ) ∈ R V × J , where V is the vocabulary size and J is the number of documents in each task. In addition, our method also builds a task-specific dependency graph whose adjacency matrix is represented as A ( i ) ∈ R V × V . To understand the concepts of corpus, task, and document more clearly, see the example in Appendix C.

Task

𝑖

𝒄

(")

𝜽

$

(")

𝑨

(")

𝒁

(")

% 𝑨

(")

GCN

DEC

MLP

Attn

𝑯

(")

+

𝑬

(")

𝝁

%

(")

𝚺

%

(")

𝜷

(")

𝑿

(")

Document-level

topic proportion

Topic-word matrix

Initialized word embeddings

Dependency graph

iPhone

company

trade

Inc.

apple

…

% 𝑿

(")

Reconstructed BoWs

Adaptive word embeddings

Reconstructed graph

2.2 Generative Model

In this section, we present a C ontextualized E mbedded T opic M odel, dubbed as Meta-CETM , to cope with the problem of topic modeling under resource-poor conditions, the essence of which is to make extra use of the contextual information of given documents to learn dynamic word embeddings that are well adapted to the current task. Concretely, for any given task T ( i ) , in addition to obtaining the BoW representations X ( i ) of its documents, which only imply the word co-occurrence patterns, we also build a semantic graph between words based on the contextual dependency grammars. Thus the corresponding adjacency matrix A ( i ) can be viewed as complementary information on the word semantics. With X ( i ) and A ( i ) , then the goal of the generative model is to model these two types of observations jointly, whose key lies in how to establish a bridge connecting the two.

Inspired by the recently developed VGAEs [33] and embedded topic models (ETMs) [18, 31], we create our generative model based on the compact assumption, i.e. , there exists a shared latent space of words that can be mapped to the observation spaces of BoW and semantic graph, respectively. Following this conception, we portray a well-structured generative process via a specific task T ( i ) . First, the latent representations of words Z ( i ) ∈ R D × V are sampled from a prior distribution. Acting as the suitable connection, they are responsible for generating both the adjacency matrix A ( i ) and the BoW X ( i ) . To produce A ( i ) , we use a typical inner product graph decoder formulated as

As for the generation of BoW X ( i ) , the latent representations Z ( i ) (or word embeddings) are mainly used to derive the topic-word matrix. Unlike previous ETMs that usually decompose the topic-word matrix into the inner product of learnable word embeddings and topic embeddings, here we provide a Bayesian perspective on interpreting the topic-word matrix by imposing a Gaussian mixture prior to the word latent space. More precisely, we regard each component of the Gaussian mixture prior as the representation of a topic, such that the word embeddings are naturally generated from different clusters (topics). And the prior probability for the v th word embedding is given by

where π k is the prior mixing coefficient of the k th topic and ∑ k π k = 1 . As such, each topic can be represented as a Gaussian distribution in the word latent space, with a probability density assigned to each word embedding. This provides a natural choice to define the topic-word matrix β ( i ) ∈ R V × K .

Hence, the distribution of topic k over the vocabulary β ( i ) k ∈ R V can be derived by

In this way, we expect that semantically similar words will be close in the latent space and generated from the same topic. Moreover, we also posit a context variable c ( i ) that summarizes the task-level information about the topic proportion and thus serves as a prior for generating the topic proportion θ ( i ) j of each document. Formally, the complete generation process of our model can be formulated as

Generate the adjacency matrix of semantic graph using Eq. 1;

For the v th word appeared in the task T ( i ) :

(a) Draw its topic assignment y ( i ) v ∼ Cat( π ) ;

(b) Draw its word embedding z ( i ) v ∼ N ( µ y ( i ) v , Σ y ( i ) v ) ;

Compute task-specific topic-word matrix β ( i ) based on Eq. 3;

Draw task-level context variable c ( i ) ∼ N (0 , a 2 I ) ;

For the j th document in the task T ( i ) :

(a) Draw its topic proportion θ ( i ) j ∼ LN ( c ( i ) , b 2 I ) ;

(b) For the n th word in the j th document:

Draw its topic assignment e ( i ) jn ∼ Cat( θ ( i ) j ) ;

Generate the word count x ( i ) jn ∼ Cat( β ( i ) e ( i ) jn ) ,

where Cat() , N () , Ber() , and LN () denote categorical, Gaussian, Bernoulli, and logistic-normal distributions, respectively. σ is the Sigmoid() operation 4 . a and b are both hyper-parameters. Table 4 in Appendix B gives a list of key notations used in this paper.

2.3 Variational Inference Algorithm

Observing a task T ( i ) with data X ( i ) , the goal of inference network is to approximate the posterior distributions over the latent variables, { θ ( i ) j , c ( i ) , Z ( i ) } , and the parameters of GMM, { π ( i ) k , µ ( i ) k , Σ ( i ) k } .

Document-specific latent variable inference. To model the uncertainty of topic proportions, we define the variational posterior distribution for θ ( i ) j , and a residual multi-layer perception (MLP) is employed to learn distribution parameters. To be specific,

where µ θ ( i ) j and Σ θ ( i ) j are logistic Gaussian distribution deterministic parameters depending on the latent mean c ( i ) and the document latent representation h ( i ) j of x ( i ) j .

Task-specific latent variable inference. For topic proportion mean c ( i ) , we apply Gaussian distribution to approximate the variational posterior, whose parameters mean vectors and covariance matrices are derived through encoding all documents in task T ( i ) with attention mechanism [34], written as

where H ( i ) is the latent indication of the task, and Attn() is the attention mechanism to capture relationships of J documents in task T ( i ) .

Additionally, to ensure the modeling flexibility, we design a simple inference network consisting of a two-layer GCN [35] to infer the latent representations Z ( i ) of words following [33]:

where E ( i ) is the initialized word features and A ( i ) is derived through a neural parser on X ( i ) . Figure 1 illustrates an overview of our approach, including the variational inference network.

In this paper, for simplicity, we denote Ψ as network parameters of both the encoder and the decoder.

Expectation Maximization for solving µ ( i ) k and Σ ( i ) k . As discussed before, µ ( i ) k and Σ ( i ) k are task-specific parameters of Gaussian mixture distribution, which are not learnable variables, and we do not have analytic solutions for Maximum Likelihood Estimation (MLE) of GMM containing the non-differentiable sampling process [36]. To approximate the posterior, we resort to Expectation Maximization (EM) [37] algorithm to optimize the parameters, formulated as:

Since topic k is sampled from the Uniform distribution for each task, the mixing coefficients are initialized as 1 K . We initialize µ ( i ) k and Σ ( i ) k as the average of latent variables Z ( i ) and the identity matrix I , respectively. Here, we only display the final updating formulas. The detailed derivation processes for E-step and M-step are presented in Appendix E.2.

2.4 Training Objective and Optimization

By Jensen's inequality, the evidence lower bound (ELBO) of each task can be derived as

where

is the variational joint distribution. The first and the fourth terms in Eq. 8 are the reconstruction errors for document BoW and the graph adjacency matrix, respectively. The remaining three terms are all Kullback-Leibler (KL) divergence to constrain the distance between the prior distribution and the variational posterior distribution. Owing to the space limit, we only present the final formulas for L ELBO here. The detailed derivations, the training algorithm, and the meta-testing algorithm can be found in Appendix E.1, Alg. 1, and Alg. 2, respectively.

3 Experiments and Analysis

3.1 Experimental setup

Datasets. We conducted experiments on four widely used textual benchmark datasets, specifically 20Newsgroups ( 20NG ) [38], Yahoo Answers Topics ( Yahoo ) [39], DBpedia ( DB14 ) [40], and Web of Science ( WOS ) [41].

Baseline methods. Our model is compared with exemplary baseline methods, including probabilistic topic models, state-of-the-art NTMs and CTMs. Specifically, we conduct experiments of the following methodologies under the document-limited setting: 1) LDA [42]; 2) PFA [12]; 3) LDA with Products of Experts ( ProdLDA ) [43], which replaces the mixture model in LDA with products of experts and updates parameters using AVI; 4) Embedded Topic Model ( ETM ) [18], an NTM incorporating word embeddings and learning with AVI. For a fair comparison, we also consider the variations of ProdLDA and ETM under the few-shot setting, referred to 5) MAML-ProdLDA and 6) MAML-ETM , where the parameters are optimized through MAML algorithm [44]. In addition, we include a model-based hierarchical NTM, 7) Meta-SawETM [30], but we only apply their single-layer model. Moreover, another two CTMs 8) CombinedTM [21] and 9) ZeroShotTM [22] are also compared, both of which contain contextualized Sentence BERT [45] embeddings as the model input.

3.2 Experimental results and analysis

In this section, we evaluate the predictive performance, topic quality, and classification performance of our model through an extensive series of experiments. Note that in all tables, we have highlighted the best and runner-up results in boldface and with an underline, respectively. Our code is available at https://github.com/NoviceStone/Meta-CETM .

3.2.1 Per-holdout-word perplexity

Following the practice in Meta-SawETM [30], we adopt the per-holdout-word perplexity (PPL) [46] to measure the predictive performance of our model. Specifically, for each task composed of several documents, 80% of the tokens in the BoWs are randomly chosen to form a support set D S test , which is used to adapt to a task-specific topic-word matrix β , and the remaining 20% word tokens are held out to form the query set D Q test with data Y . Then, the PPL can be calculated as

where S is the total number of collected samples and y ·· = ∑ V v =1 ∑ N n =1 y vn .

Model settings. For all compared methods, we set the number of topics as 10. And for all NTMs, the hidden layers size of the encoder is set to 300. For all embedding-based topic models, i.e., ETM, MAML-ETM,Meta-SawETM and our Meta-CETM, we load pretrained GloVe word embeddings [47] as the initialization for a fair comparison. Finally, We train our model using the Adam optimizer [48] with a learning rate of 1 × 10 -2 for 10 epochs on an NVIDIA GeForce RTX 3090 graphics card.

Results. In Table 1, we list the PPL of ten compared methods on four datasets. It can be noticed although LDA and PFA are both traditional probabilistic topic models, PFA presents better results than LDA. By utilizing MAML to learn parameter initializations, MAML-ProdLDA performs better than ProdLDA while MAML-ETM exhibits poorer results than ETM. This can be attributed to that ETM possesses more parameters than ProdLDA, and calculating the gradients in a high-dimensional space with only a few documents is difficult for MAML-ETM. Applying Weibull distribution to model the latent representation for documents and employing task-specific variable designs for both

0

0.2

0.4

0.6

0.8

1

20NG

Yahoo

DB14

WOS

图表标题

MAML-ProdLDA

MAML-ETM

Meta-SawETM

CombinedTM

ZeroShotTM

Meta-CETM

-0.8

-0.6

-0.4

-0.2

0

-1

topic proportions and topic matrices, Meta-SawETM acts superior to other topic models. Moreover, incorporating contextualized embeddings as input enables CombinedTM and ZeroShotTM to present competitive performance. Furthermore, it is noteworthy that Meta-CETM achieves the lowest PPL among all these methods, indicating the excellent predictive performance of our model.

3.2.2 Topic quality

In this part, we evaluate the topic quality of different methods in terms of the topic diversity ( TD ) [49], defined as the percentage of unique words in the top 10 words of all topics, and the topic coherence ( TC ), which measures the average Normalized Pointwise Mutual Information (NPMI) [50, 51] values to count word co-occurrences 5 . The experimental settings are the same as in Sec. 3.2.1. For each task, the number of documents is 10 for all datasets. The results are displayed in Fig. 2 and it can be notably and interestingly observed that MAML-ProdLDA achieves fairly 'perfect' TD results, but it shows the worst TC performances among compared methods. Such inconsistency is brought by MAML-ProdLDA's concentration on a large amount of universal and frequently-occurring words when training on the base data, hindering it from extracting informative topics given the meta-test task. Embodying the embedding design for words and topics, MAML-ETM and Meta-SawETM are more likely to mine context-related topics than MAML-ProdLDA. Besides, as the representatives of CTM, CombinedTM and ZeroShotTM equipped with BERT embeddings present more comprehensive performances than previous topic models. From the TC perspective, our Meta-CETM yields the most favorable results among the six methods, indicating the fast adaptability in discovering interpretable topics with limited documents.

3.2.3 Few-shot document classification

To further validate that our model is capable of learning topics which are highly adapted to the task, we undertake experiments on few-shot document classification.

Model settings. As in Meta-SawETM [30], we compare our Meta-CETM with classical meta-learning algorithms under different architectures. Specifically, we design a three-layer feedforward network as MLP structure and three-layer 1-dimensional convolutions followed by batch normalization, ReLU() activation and max-pooling operation as CNN . For meta-learning methods, we apply MAML [44] to learn parameter initializations, and prototypical network ( PROTO ) [52] to learn an embedding space and minimize the distance between the clustering centroids and the samples. Besides, we adopt two fine-tuning manners, named FT [53] and FT* , to update the classifier parameters and all parameters of the model, respectively. Single-layer HNS-SawETM and single-layer Meta-SawETM in [30] are compared as well. Additionally, we investigate the CombinedTM [21] and ZeroShotTM [22] to evaluate their few-shot classification performance. Different from PPL evaluation in Sec. 3.2.1, the support set and the query set for classification are sourced from two batches of documents.

Classification strategy. For typical few-shot learning algorithms, we follow the convention to train parameters of the feature extractor in a supervised manner. At meta-test stage, we use the support set ( i.e., a few labeled examples) to adapt to a task-specific classifier and compute the accuracy on the query set. As for topic models, where no dedicated classifiers are available, we first use the support set to learn a group of class-specific topic-word matrices, then for each document in the query set, we calculate its reconstruction error as the basis for classification.

Results. The classification results 6 are listed in the Table 2. As we can see, the CNN architecture outperforms MLP under the same algorithms, which can be attributed to CNN's unique inductive bias of the locality. Furthermore, PROTO surpasses MAML by a large margin in most cases, indicating that a good embedding space is more useful for classification than favorable parameter initializations. For the CNN architecture, fine-tuning the classifier only (FT) and fine-tuning all parameters (FT*) make slight differences; but for MLP, we observe a considerable performance boost by updating all parameters of the network over the FT algorithm. We postulate that the classification results of MLP are more susceptible to the variation of feature extractor parameters due to its linear structure. Additionally, CombinedTM and ZeroShotTM, incorporating contextualized representations, outperforms HNS-SawETM and Meta-SawETM by learning class-specific topic-word matrices more effectively. With the design of adaptive word embeddings, our Meta-CETM not only achieves much better results than previous topic models, but also is comparable to few-shot learning algorithms particularly designed for supervised learning.

3.2.4 Embedding space visualization

(a)

(b)

(c)

nanagenent

systen

church

benefitneed

use

factor

better.

Life

Likethinks

said

P06

see

want

general

believe

monday

new

people"

right

dos

1992

nac

year

scsivga

pC

11

server

10

windows

million

graphics

files

wins

fansplayers

gane

season

sport

boston

hockey

nhl

buffalo

hawks

bruins

rangers

american

rangers

wins

state

believe

public

boston

said

people

thingsknow

fact

time want

problem

life

bible

job

systen

software

disk

work

fans

use

card

fax

including

sport

information

windows

sharks

phl,

hawks

dos

seasonPlayoffs

scores

football

players

game

buffalo

hockey

1

1

1

sharks

buffalo

hawksleafs

boston

rangers

nht

fansPlayers

bruins

hockey

playoffs round

goal

showed

awesone

season

show

imagine

art

gane

talking

real

waited

wonderful

talent

pressed

finally

speech

scores

hard

decision

series

vieu

winsfinished

great

obviously

take

upper

impressed

sec

16

17

january

1992

In addition to quantitative results, we also qualitatively analyzed the effectiveness of our model by visualizing the adapted embedding space, as shown in Figure 3. It can be seen that both MAML-ETM and Meta-SawETM learn topics that are not highly relevant to the target task, as most of their top words ( e.g., windows and dos ) are inherited from the base training corpora, while those informative words ( e.g., players and hockey ) associated with the target task are away from the topic embeddings. By contrast, in the embedding space learned by Meta-CETM, the adapted Gaussian distributions, i.e., topics reasonably cover almost all words closely related to the target task, indicating that our model can achieve successful adaptation and discover more interpretable topics.

Furthermore, we investigated whether the adaptive word embeddings generated by our model effectively reflect the contextual information of each given task. As illustrated in Figure 4, the word embedding of 'apple' adapted from a task related to company is surrounded by words like 'mobile' and 'amazon', whereas in another task concerning plant , the embedding of 'apple' is closer to the words such as 'fruit' and 'trees'. This phenomenon suggests that Meta-CETM produces word embeddings that are semantically well-matched to the context of the target task.

3.3 Ablation study

To explore the effectiveness of key designs in our model, we conduct a series of ablation experiments on all four datasets by removing each designed module separately. The numerical results of perplexity, topic diversity, and topic coherence are listed in Table 3, where ETM [18] is chosen as the baseline method. From the results presented below, we have the following observations:

i) Despite achieving the best TD results, ETM obtains the worst PPL and TC scores, which can be attributed to its tendency to extract frequently occurring themes from training, hindering its fast adaptability to meta-test sets with only several documents. ii) To capture the relations of texts and model the task information, in Section 2.2, we posit a context-specific variable c ( i ) as the prior of document-level topic proportion θ ( i ) j . It can be found with c ( i ) , our Meta-CETM achieves much lower PPL and higher TC, demonstrating the efficacy of task-specific variables. iii) For ablation results without the Graph VAE module, we replace it with a Graph AE rather than discarding it completely. Besides, for Meta-CETM without the GMMprior design, we replace it with the standard Gaussian distribution prior rather than imposing no prior completely. From the quantitative results shown

(a)

(b)

(c)

in the table, it can be found compared with Gaussian prior in vanilla VAE model, GMM priors encourage the semantically coherent words to be allocated to the same topic, leading to higher TC scores and lower PPL. Further, we visualize the embedding space in Figure 5 to demonstrate the benefits of GMM prior.

4 Related Work

To discover a group of topics from a batch of documents, probabilistic topic models have been developed in recent years, including hierarchical topic models [54, 14, 46, 55], NTMs [56, 15],embedded topic models [18, 31], and CTMs [22, 21]. Besides, some works are proposed to incorporate knowledge into the modeling process [57-59] and some researchers apply optimal transport to measure the distances between topics and words or documents [60, 61]. From the perspective of modelling word embeddings with Gaussian distribution, Vilnis etc. proposed a novel density-based mapping method [62] and achieved promising performances. Recently, some works [30, 27] aimed at topic modelling under the few-shot setting and proposed to obtain a group of task embeddings to adaptively explore topics within only a few documents in a model-based meta-learning fashion. However, their method cannot address the multiple meanings of one word issue, a prevalent issue in practical document analysis, which is addressed through the introduction of dependency graph and the GMM prior distribution in our work.

5 Conclusion

In this paper, we propose a novel NTM, Meta-CETM, to address the the fast adaption problem in document analysis under low-resource regimes. Specifically, we construct a task-specific graph to obtain context-related word embeddings. Then we introduce the graph VAE with Gaussian mixture prior to model the word representations and topic embeddings, which are optimized through the EM algorithm. We also propose the task-specific prior for topic proportions. Through extensive experiments and illustrations, we demonstrate the superior performance of our model in solving the adaptation problem in topic modeling.

References

Thomas L Griffiths and Mark Steyvers. Finding scientific topics. Proceedings of the National academy of Sciences , 101(suppl_1):5228-5235, 2004.

Andrew McCallum, Xuerui Wang, and Andrés Corrada-Emmanuel. Topic and role discovery in social networks with experiments on enron and academic email. Journal of artificial intelligence research , 30:249-272, 2007.

David Mimno, Hanna Wallach, Jason Naradowsky, David A Smith, and Andrew McCallum. Polylingual topic models. In Proceedings of the 2009 conference on empirical methods in natural language processing , pages 880-889, 2009.

Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) , volume 2, pages 524-531. IEEE, 2005.

Simon Rogers, Mark Girolami, Colin Campbell, and Rainer Breitling. The latent process decomposition of cdna microarray data sets. IEEE/ACM transactions on computational biology and bioinformatics , 2(2):143-156, 2005.

David Mimno. Reconstructing pompeian households. arXiv preprint arXiv:1202.3747 , 2012.

Jordan Boyd-Graber, Yuening Hu, David Mimno, et al. Applications of topic models. Foundations and Trends® in Information Retrieval , 11(2-3):143-296, 2017.

David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research , 3(Jan):993-1022, 2003.

David Blei and John Lafferty. Correlated topic models. Advances in neural information processing systems , 18:147, 2006.

David M Blei and John D Lafferty. Dynamic topic models. In Proceedings of the 23rd international conference on Machine learning , pages 113-120, 2006.

Jon Mcauliffe and David Blei. Supervised topic models. Advances in neural information processing systems , 20, 2007.

Mingyuan Zhou, Lauren Hannah, David Dunson, and Lawrence Carin. Beta-negative binomial process and Poisson factor analysis. In Artificial Intelligence and Statistics , pages 1462-1471. PMLR, 2012.

Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei. Hierarchical topic models and the nested Chinese restaurant process. Advances in neural information processing systems , 16, 2003.

John Paisley, Chong Wang, David M Blei, and Michael I Jordan. Nested hierarchical Dirichlet processes. IEEE transactions on pattern analysis and machine intelligence , 37(2):256-270, 2014.

Yishu Miao, Lei Yu, and Phil Blunsom. Neural variational inference for text processing. In International conference on machine learning , pages 1727-1736. PMLR, 2016.

Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488 , 2017.

Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull hybrid autoencoding inference for deep topic modeling. arXiv preprint arXiv:1803.01328 , 2018.

Adji B Dieng, Francisco JR Ruiz, and David M Blei. Topic modeling in embedding spaces. Transactions of the Association for Computational Linguistics , 8:439-453, 2020.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning , pages 1278-1286. PMLR, 2014.

Federico Bianchi, Silvia Terragni, and Dirk Hovy. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. arXiv preprint arXiv:2004.03974 , 2020.

Federico Bianchi, Silvia Terragni, Dirk Hovy, Debora Nozza, and Elisabetta Fersini. Crosslingual contextualized topic models with zero-shot learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , pages 1676-1683, Online, April 2021. Association for Computational Linguistics.

Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure. arXiv preprint arXiv:2203.05794 , 2022.

Zihan Zhang, Meng Fang, Ling Chen, and Mohammad-Reza Namazi-Rad. Is neural topic modelling better than clustering? an empirical study on clustering with contextual embeddings for topics. arXiv preprint arXiv:2204.09874 , 2022.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bertnetworks. arXiv preprint arXiv:1908.10084 , 2019.

Tomoharu Iwata. Few-shot learning for topic modeling. arXiv preprint arXiv:2104.09011 , 2021.

Anand V Bodapati. Recommendation systems with purchase data. Journal of marketing research , 45(1):77-93, 2008.

Hao Sha, Mohammad Al Hasan, George Mohler, and P Jeffrey Brantingham. Dynamic topic modeling of the covid-19 twitter narrative among us governors and cabinet executives. arXiv preprint arXiv:2004.11692 , 2020.

Zhibin Duan, Yishi Xu, Jianqiao Sun, Bo Chen, Wenchao Chen, Chaojie Wang, and Mingyuan Zhou. Bayesian deep embedding topic meta-learner. In International Conference on Machine Learning , pages 5659-5670. PMLR, 2022.

Zhibin Duan, Dongsheng Wang, Bo Chen, Chaojie Wang, Wenchao Chen, Yewen Li, Jie Ren, and Mingyuan Zhou. Sawtooth factorial topic embeddings guided gamma belief network. In ICML 2021: International Conference on Machine Learning , July 2021.

Danqi Chen and Christopher D Manning. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) , pages 740-750, 2014.

Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308 , 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Max Welling and Thomas N Kipf. Semi-supervised classification with graph convolutional networks. In J. International Conference on Learning Representations (ICLR 2017) , 2016.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Todd K Moon. The expectation-maximization algorithm. IEEE Signal processing magazine , 13 (6):47-60, 1996.

Ken Lang. Newsweeder: Learning to filter netnews. In Machine Learning Proceedings 1995 , pages 331-339. Elsevier, 1995.

Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in neural information processing systems , 28, 2015.

Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. Dbpedia: A nucleus for a web of open data. In The semantic web , pages 722-735. Springer, 2007.

Kamran Kowsari, Donald E Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S Gerber, and Laura E Barnes. Hdltex: Hierarchical deep learning for text classification. In 2017 16th IEEE international conference on machine learning and applications (ICMLA) , pages 364-371. IEEE, 2017.

David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research , 3(Jan):993-1022, 2003.

Akash Srivastava and Charles Sutton. Autoencoding variational inference for topic models. arXiv preprint arXiv:1703.01488 , 2017.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference on Machine Learning , pages 1126-1135. PMLR, 2017.

Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 3982-3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410 .

Yulai Cong, Bo Chen, Hongwei Liu, and Mingyuan Zhou. Deep latent Dirichlet Allocation with topic-layer-adaptive stochastic gradient riemannian mcmc. arXiv preprint arXiv:1706.01724 , 2017.

Jeffrey Pennington, Richard Socher, and Christopher Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1532-1543, October 2014. doi: 10.3115/v1/D14-1162.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015.

Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 6345-6381, Florence, Italy, July 2019. Association for Computational Linguistics.

David Newman, Jey Han Lau, Karl Grieser, and Timothy Baldwin. Automatic evaluation of topic coherence. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , pages 100-108, Los Angeles, California, June 2010. Association for Computational Linguistics.

Michael Röder, Andreas Both, and Alexander Hinneburg. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining , WSDM '15, page 399-408, New York, NY, USA, 2015. Association for Computing Machinery.

Jake Snell, Kevin Swersky, and Richard S Zemel. Prototypical networks for few-shot learning. arXiv preprint arXiv:1703.05175 , 2017.

Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In International Conference on Learning Representations , 2019.

David M Blei, Thomas L Griffiths, and Michael I Jordan. The nested Chinese restaurant process and bayesian nonparametric inference of topic hierarchies. Journal of the ACM (JACM) , 57(2): 1-30, 2010.

Mingyuan Zhou, Yulai Cong, and Bo Chen. The Poisson gamma belief network. Advances in Neural Information Processing Systems , 28:3043-3051, 2015.

Dallas Card, Chenhao Tan, and Noah A Smith. A neural framework for generalized topic models. arXiv preprint arXiv:1705.09296 , 2017.

Zhibin Duan, Yishi Xu, Bo Chen, Dongsheng Wang, Chaojie Wang, and Mingyuan Zhou. Topicnet: Semantic graph-guided topic discovery. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

Dongsheng Wang, Yi.shi Xu, Miaoge Li, Zhibin Duan, Chaojie Wang, Bo Chen, and Mingyuan Zhou. Knowledge-aware bayesian deep topic model. In Advances in Neural Information Processing Systems , 2022.

Yi.shi Xu, Dongsheng Wang, Bo Chen, Ruiying Lu, Zhibin Duan, and Mingyuan Zhou. Hyperminer: Topic taxonomy mining with hyperbolic embedding. In Advances in Neural Information Processing Systems , 2022.

He Zhao, D Phung, V Huynh, T Le, and W Buntine. Neural topic model via optimal transport, 2020.

Viet Huynh, He Zhao, and Dinh Phung. Otlda: A geometry-aware optimal transport approach for topic modeling. Advances in Neural Information Processing Systems , 33:18573-18582, 2020.

Luke Vilnis and Andrew McCallum. Word representations via gaussian embedding. arXiv preprint arXiv:1412.6623 , 2014.

A Discussions

A.1 Limitations and Future Work

In this paper, we propose a method to improve existing embedded topic models (ETMs) under the low-resource settings by introducing the task-specific graph with the Gaussian mixture prior. The main limitation of our work could be the reliance on the pre-trained parsing tools. Specifically, we construct the semantic graph according to the syntactic structure of the documents with the public library, spaCy 7 . However, some other approaches such as the sliding window method can also be used to construct the graph, which is not relied on training. Besides, although strong performances, our framework is still trained and evaluated on the same dataset. While, in the current large models era, pre-training on a large set and then perform few-shot learning or even zero-shot learning attracts researchers' attention and interest, especially such Bayesian models. However, this is beyond the scope of this paper and we will conduct a thorough investigation of this issue in future work.

A.2 Broader impact

Our work builds a novel and advanced topic modeling technique under few-shot settings. For example, at the very beginning of the Covid-19, there were relatively few cases of infection in all regions, and the reports of related medical diagnoses were highly limited. Under such circumstances, few-shot topic models can help us extract key information from limited resources, thus facilitating us to take appropriate and efficient preventive and control measures. On the other hand, the emergence of ChatGPT by OpenAI has attracted a great deal of attention, while in this paper, what we explore is a robust model with generalization capabilities given limited text data, rather than training from a very large corpus. Potential negative societal impact of our work could arise from malicious intent in changing model's behavior by injecting deliberate human prejudice, which may harm the fairness of the community. However, we hope our work is utilized to enable more research and applications primarily from the originality of benefiting the community development.

B Key Notations

In Table 4, we list the key notations, descriptions and corresponding dimensions used in this paper.

⋮

Task

𝑖

Task

𝑗

⋮

busy

school

public

taxis

park

car

bus

Seoul

Korea

place

bus

free

chip

store

card

read

drive

disk

transmit

data

…When the drive has read requested

data from the disk the SCSI bus may

be busy. This data needs to be stored

some where until the bus is free and

the data can be transmitted…

bus

…The problem in Korea, especially

in Seoul, is that there are so many

privately-owned cars, as well as

taxis and buses, the rush-hour has

become a 24-hour phenomenon and

that there is no place to park…

C An Illustration of Our Settings

In Section 2.2, we touched on the concepts of corpus, task, and document to describe our problem setting, which is a bit messy to follow. Here, we clarify these concepts through a concrete example so that the reader could understand the problem setting of few-shot learning more easily. Considering the 20NG [38] dataset:

A ' corpus ' refers to a collection of documents belonging to the same category so that 20NG consists of 20 corpora, each of which contains documents from one of the 20 classes.

A ' task ' is a smaller unit than a 'corpus' that only comprises a few related documents, typically less than 100 documents. Consequently, we could sample a number of tasks from each training corpus.

(we select 12 out of the 20 corpora for training)

Then our goal is to utilize these sampled tasks to train a generalizable topic model that can efficiently adapt to a new task from the test corpus (the remaining 8 corpora are used for testing). In addition, for each task at the testing stage, we split its documents into two parts, one for fine-tuning or retraining the topic model, called the support set , and the other for evaluating the model's performance, called the query set . Note that we do not design different generative processes for the corpus documents versus the task documents. In essence, our proposed Meta-CETM only characterizes the generative process of the task documents by jointly modeling the syntactic graph A and the observed BoW X in each task. In Fig. 6, we visualize the task and the corresponding unweighted dependency graph A .

D Algorithms for training and testing

In Alg. 1 and Alg. 2, we present the training and meta-testing procedures of our Meta-CETM.

E Derivation of Formulas

In this section, we provide the detailed derivation process of variational evidence lower bound (ELBO) in Eq. 8 and the expectation maximization solver process for multivariate Gaussian distribution in Eq. 7 in Sec. 2.2.

Algorithm 1: Training process

Input: A set of training corpora {D c } C c =1 ; initialized model parameters Ψ Randomly sample tasks from each training corpus D c to obtain {T ( i ) } i M =1 ; for each task T ( i ) , i = 1 , 2 , · · · , M do Build semantic graph A ( i ) with established dependency parsing tools; Infer adaptive word embeddings Z ( i ) according to Eq. 6; Initialize parameters of the Gaussian mixture prior: π k , µ k and Σ k ; Update to the optimal value π ( i ) k , µ ( i ) k and Σ ( i ) k using EM based on Eq. 7; Compute the topic-word matrix β ( i ) according to Eq. 3; Infer the latent context varibale c ( i ) using Eq. 5; for each document x ( i ) j , j = 1 , 2 , · · · , J do Infer topic proportion θ ( i ) j with Eq. 4; Calculate the log-likelihood p ( x ( i ) j | θ ( i ) j , β ( i ) ) ;

Derive the ELBO as Eq. 8 and update Ψ using SGD;

Algorithm 2: Meta-test for a new task

Input:

A new corpus D test , trained model parameters Ψ

Output: Adaptive topic-word matrix

β

Randomly sample a task T new from the given corpus D test ;

Get the corresponding BoWs X new and dependency graph A new for the current task;

Infer the adaptive word embeddings Z new with part of the trained model parameters Ψ ;

Initialize parameters of the Gaussian mixture prior: π k , µ k and Σ k ;

Compute optimal π ∗ k , µ ∗ k and Σ ∗ k using EM based on Eq. 7;

Derive the adaptive topic-word matrix β new by Eq. 3;

E.1 Variational ELBO

E.2 Solving topic parameters { π ( i ) k , µ ( i ) k , Σ ( i ) k } K k =1 with Expectation Maximization

The log likelihood function is given by

1. Deriving µ ( i ) k

Setting the derivatives of ln p ( Z ( i ) | π ( i ) , µ ( i ) , Σ ( i ) ) w.r.t the means µ ( i ) k to zero, we have

Define the posterior probabilities as

Multiplying by Σ ( i ) k -1 and rearranging, we can obtain the updating formula for µ ( i ) k as

2. Deriving Σ ( i ) k

Similarly, we set the derivatives of ln p ( Z ( i ) | π ( i ) , µ ( i ) , Σ ( i ) ) w.r.t Σ ( i ) k to zero, then we have

Using γ vk in Eq. 4 and rearranging, we get the updating formula for Σ ( i ) k as

3. Deriving π ( i ) k

Finally, using Lagrange multiplier algorithm, our goal is to maximize the following formula:

where ∑ K k =1 π ( i ) k = 1 .

Then setting the derivatives of the above equation w.r.t π ( i ) k to zero, we have

Multiplying π ( i ) k and rearranging, we obtain

Considering ∑ K k =1 π ( i ) k = 1 , then ∑ k -∑ v γ vk λ = 1 , and λ = ∑ v ∑ k γ vk .

Hence the updating formula for π ( i ) k as

F More Results

F.1 Perplexity results

In Table 1, we list the PPL results of different compared methods on four datasets, where the number of documents is 5 or 10 in each task. In this part, we provide more PPL quantitative results with varied numbers of texts per task {20, 50, 100} in Table 5, Table 6 and Table 7, respectively.

F.2 Topic quality results

In Sec. 3.2.2, we present the topic interpretability results including topic diversity (TD) and topic coherence (TC) of six compared methods. Except for CombinedTM [21] and ZeroShotTM [22], we carry on experiments applying another contextual topic model (CTM) CETopicTM [24] with SimCSE pretrained word embeddings 8 on four datasets. The results are exhibited in Fig. 7. It can be notably noticed CETopicTM [24] achieves much competitive results on both TD and TC scores, even compared with CombinedTM [21] and ZeroShotTM [22]. Such superiority is owed to the fact that CETopicTM utilizes word embeddings learned from large-scale BERT data and it performs clustering on sentence embeddings to generate topics. In our settings, the aim is to provide a framework for training a sufficiently generalized topic model in low-resource regimes, while equipped with BERT embeddings, CETopicTM is highly likely to obtain context-related meanings in advance under most situations. But in some cases where the words or the word meanings have not been encountered or

learned by BERT, such as some specialized occasions, CETopicTM may fail to extract interpretable topics.

0

0.2

0.4

0.6

0.8

1

20NG

Yahoo

DB14

WOS

图表标题

MAML-ProdLDA

MAML-ETM

Meta-SawETM

CombinedTM

ZeroShotTM

CETopicTM

Meta-CETM

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

F.3 Topic visualization results

In Fig. 3, we visualize the adapted embedding space of different methods to demonstrate our MetaCETM's successful fast adaption. Further, to better characterize meaningful and coherent topics learned by our model given a few number of documents, we display the text and topics extracted by Meta-SawETM [30], CombinedTM [21] and our Meta-CETM in Fig. 8.

Meta-SawETM

tv

air

ball

players

team

school

help

nfl

sports

people

music

like

teams

time

player

Meta-CETM

basketball

nfl

baseball

greatest

putting

rookie

nfl

team

game

win

soccer

nfl

sport

rookie

super

… What sports would you say it is easy and difficult to be a rookie in? The

sports that I would say it is difficult to be a rookie in are basketball, hockey,

and soccer. The easiest sport for a rookie is certainly the NFL. Some of the

greatest NFL players started out on special teams. They have put much

efforts to help the team win the game, especially the super bowl …

CombinedTM

champion

weekend

perfect

fort

hockey

players

team

working

believe

breaks

play

ice

friends

effort

nfl

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new solution (Meta-CETM) for inferring topics on a dataset with a few available documents only. The main idea is to train the model on various tasks and then use it on a new, small dataset. The are extensive experiments on diverse topic models, in particular in this context of \"few shot\" learning, and the results show that Meta-CETM outperforms its competitors.\n\nSTRENGTHS:\n- few-shot learning for topic modeling is an important topic today\n- the model looks sound (even though not really well explained)\n- the experimental framework is quite strong, with good results for the proposed solution\n\nWEAKNESSES:\n- The solution is not always well presented\n- It's a quite complicated model with many intertwined modules\n\nQUESTIONS:\nI have some concerns on this paper.\n\n- The paper is not always well presented. Here I give some examples:\n  * The problem formulation (2.1) is quite confusing, probably because of the limited space. This is much clearer in the supplementary material. Some aspects are not sufficiently described or motivated. For instance, the authors seem to use a self attention mechanism similar to the one in the Transformer (noted Attn). However, this mechanism is normally based on an embedding matrix, which is not the case here for X is a simple BoW representation of the documents.\n  * Why using the Weibull distribution in this context? There is no explanation at all. \n  * What \"latent indication\" means for H(i)?\n  * I guess the step 1.c line 114 is done after having computed all the word embedding of the generated document (we need all the Z(i)). Is it true? If so, this step should be taken outside the for loop.\n\n- Would it be possible to use the adaptive word embedding module (Z(i)) for the bag-of-word encoder?\n\n- The experimental part of the paper looks quite strong. Important solutions for topic modeling (old and new) are considered in the competitors and the authors use several manners to compete with them (perplexity, classification, topic coherence and diversity, qualitative evaluation). However, I would expect more experiments on the few-shot context, for instance by varying the number of available documents (not only 5 or 10) and taking into account the semantic difference between the corpora (i.e., between the training set and the test set).\n\nLIMITATIONS:\nThere is not a single limitation mentioned in the paper... There is no future work as well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an approach for few-shot topic modeling. The authors first question the limitations of \"static word embeddings\" in previous related work when transferring to new tasks, and then propose to use the \"adaptive word embeddings\" generated by VGAE to address this issue. Although the problem this paper aims to solve is meaningful, I think the description of the motivation is not clear enough. Even though the experimental results are acceptable, the methods used do not reflect the motivation of the paper.\n\nSTRENGTHS:\n- Few-shot learning for topic modeling is a meaningful problem.\n\n- By combining VGAE with the neural topic model, fairly good experimental results were achieved.\n\nWEAKNESSES:\n\tThis paper aims to address the limitations of \"static word embeddings\". In line 9, the authors claim that 'we introduce a variational graph autoencoder to learn task-specific word embeddings based on the dependency graph refined from the context of each task'. However, why would the dependency graph and VGAE help in learning task-specific word embeddings? If this claim holds true, simply using the context of each task could achieve the same purpose. In other words, I don't see how VGAE and the dependency graph would aid in learning the so-called 'adaptive word embeddings'. The authors should carefully explain why the dependency graph and VGAE can reflect the characteristics of each task. Would the dependency graph for each task be fundamentally different? In summary, I find this motivation unconvincing.\n\n\tIn line 14, the authors state that the Gaussian mixture prior can \"facilitate the discovery of diverse topics and the quick adaptation to novel tasks.\" However, why using a Gaussian mixture prior can help the fast adaptation to novel tasks? What advantages does it have over the methods used in existing neural topic models? The authors should explain the principles that make the Gaussian mixture prior to work, that is, what the motivation is for using a Gaussian mixture prior, rather than vaguely stating that it can help with adaptation. I agree that the authors provide a perspective on learning topics through clustering by using a Gaussian mixture prior, but the motivation they explained is not convincing. \n\n\tSome sentences are overly long, making this paper hard to follow. For instance: Lines 24-27.\n\n\tIn lines 57-66, the author's description of the contributions of this paper. The first and the second points are repetitive.\n\n\tIn lines 45-46, \"it is experimentally found ...\". This statement is vague and confusing. If it's a finding from the experiments conducted in this paper, then the results should be presented. If it's an experimental finding from previous work, then a reference should be provided.\n\n\tAccording to the introduction and related work, this paper is an important existing work on few-shot topic modeling: \"Few-shot learning for topic modeling\". So why did the authors not compare the methods of this work in their experiments? The authors should give a reason.\n\n\tWhat does ϕ stand for in Equation 10? This paper does not give an explanation.\n\n\tThe author demonstrated the effectiveness of the proposed method in the experiment. However, I believe there are some flaws in the experiment. Firstly, since the author believes that VGAE is the main reason for acquiring \"adaptive word embeddings\", then the visualization results of the model without VGAE should also be provided in Figure 3, in order to prove the role of VGAE in learning \"adaptive word embeddings\", which is a major motivation of this paper.\n\n\tSecondly, the ablation study is not comprehensive enough. On the one hand, the author should provide the results on all four datasets. On the other hand, the author should also provide the results of separately removing the context variable, graph vae, and GMM prior, in order to clearly prove the effectiveness of each module proposed in this work.\n\nQUESTIONS:\nPlease see the Weaknesses.\n\nLIMITATIONS:\nPlease see the Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors target the problem of multi-meaning words across different tasks for topic models, particularly under low-resource settings. To this end, they propose a variational graph autoencoder with a trainable Gaussian mixture prior to capture the distribution of task-specific word embeddings.\n\nSTRENGTHS:\nOverall, the paper is sound with several strengths:\n\n- The authors address the low-resource regime that has received little attention from the topic modeling research recently.  \n\n- The experiments are encyclopedically covered with detailed discussions.\n\nWEAKNESSES:\nHowever, the paper exhibits some weaknesses:\n\n- The examples illustrating the applications of few-shot neural topic models are not persuasive. In particular, few-shot topic modeling might not be employed to learn users’ past purchases or online behaviors. In addition, during crises, e.g. Covid-19, the number of documents towards certain topics would rather burgeon than remain limited, hence invalidating the need for few-shot topic models. \n\n- The paper needs more comparison with prior embedding-based topic models [1, 2]. Additional evaluation of the model performance against such works could make the experiments more convincing. \n\n- The choice of prior distribution plays an important role in specifying the latent space. In the paper, the authors have not elaborated on why the Gaussian mixture model is selected as the prior, or provided experiments to demonstrate its advantage over other prior choices. \n\n[1] Neural models for documents with metadata (Card et al., 2018)\n\n[2] Neural topic models via optimal transport (Zhao et al., 2021)\n\nQUESTIONS:\n- Could you supply more performance comparison of the proposed method with the previous one, especially those utilizing word embeddings for topic modeling?\n\n- Could you in more detail discuss the significance of the Gaussian mixture model as the prior distribution or clarify its benefits over other prior distributions?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a new neural topic model which aims to solve the problem of learning task-specific word embeddings in a low resource scenario.  In addition to a somewhat typical neural TM, dependency graphs are collected using parsers, and embedded via GCNs to produce adaptive word embeddings.  A topic-word matrix then models the task-specific distribution over words as a gaussian over their adaptive embeddings.  The topic-specific embedding spaces capture the role a word is likely to play within that context.  The model is evaluated across 4 datasets and a number of metrics, where it shows strong improvements in perplexity, and good performance in document classification.\n\nSTRENGTHS:\n- strong empirical performance in perplexity, and good performance in document classification.\n\n- comparisons to many existing similar models.  The improvements over models using sentence BERT are especially interesting.  Good ablations.\n\nWEAKNESSES:\n- reliance on pre-trained parsing tools.  How well does this approach work in different languages, or styles/domains of text that differ significantly from the parsing training data?\n\n- the problem at the heart of this work is lexical ambiguity.  Here parsing, and then graph-based embedding of the parse, aim to find task-specific meanings of each word, and there is little doubt that adaptive word embeddings are an effective approach to this problem.  Outside of topic modeling, LLMs and other transformer-based models solve this same task, and also refine word-specific embeddings into context-specific ones.  The only aspect in which this is included is via CombinedTM/ZeroShotTM.  Admittedly, I haven't worked on topic modeling since the switch to neural models, but it would seem that BERT would also solve the issues being pursued in this work.  I didn't see a compelling explanation for why this approach based on dependency graphs is an improvement over generic transformer-based sentence embeddings, all other aspects remaining the same.\n\n- even if the work is geared towards low resource settings, I would like to see the performance as a factor of task dataset size.  In its current presentation I think the performance is compelling enough to warrant acceptance, as the performance improvement seems sufficient across a number of metrics to be of great interest to the topic modeling community.  However, if those performance margins decrease significantly (or completely) as dataset size increases to even modest sizes, it would be difficult to find a use-case for this approach.  It would be the story of perhaps offloading some of the learning problem to the pre-trained parsing and priors over their embeddings, which solves a practical problem, but not so much one of academic interest.\n\n- discussion of conceptually related work is sparse.  The related works section is overly brief given the amount of work that solving similar core tasks.  One earlier work that seems spiritually related might be [1].\n\n- surprisingly CNN > MLP is a more important design decision than any architectural change represented across several previous papers.  It raises a little bit of an alarm bell regarding the importance of these architectural designs as evaluated here.  Extending the document classification task to more datasets (than the 2 here) would increase confidence in the importance of the work.\n\nL26: vontextualized\n\n[1] Word Representations via Gaussian Embedding\nhttps://arxiv.org/abs/1412.6623\n\nQUESTIONS:\n(mixed with weaknesses)\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of inducing neural topic models in low-resource regimes by learning adaptive word embeddings that exploit contextual grammar information. The adaptive word embeddings are learnt with a variational graph autoencoder and the topics are formed from a gaussian mixture prior of the latent word embedding space that generates the observed words and their semantic relations.  The paper introduces a variational inference algorithm to estimate the document- and topic-specific latent variables using MLP and MLP with attention, respectively. In addition, GCN is employed to learn the latent word representations given initial word embeddings and word relations derived from a neural dependency parser. Finally, the parameters of the GMM are learned independently via EM. Evaluation shows strong empirical results in few-shot topic modeling and document classification.\n\nSTRENGTHS:\nThe main contribution of the paper is the introduction of contextual semantic information into the neural topic model. Empirical results suggest that this approach is beneficial both in terms of per-holdout-word perplexity and topic coherence while remaining competitive in terms of topic diversity. Strong results are also observed in few-shot classification. The paper is fairly well-written but the baselines could have been better explained either in section 3.1 or the related work.\n\nWEAKNESSES:\nI cant parse eq. 10 in the evalution: what is the superscript (1)s?\nIn the experiments in section 3.2.3 (few-shot classification), I could not follow the evaluation scheme described in lines 238-239. How were these experiments conducted? How were the class-specific topic-word matrices calculated, what data was used for this, and how was the reconstruction error measured? Also, I could not find the results from this analysis. The results in Table 2 correspond to datasets that have ground truth topic labels.\n\nQUESTIONS:\nSee weaknesses section.\n\nLIMITATIONS:\nNo limitations were discussed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-23"}

comments: SUMMARY:
This paper proposes a method for few-shot topic modeling. Specifically, rather than following traditional wisdoms to learn static word embeddings for all the tasks/domains, the authors allow task-specific word representations such that the knowledge from the source task can be better transferred to a target task. The authors also employ Gaussian mixture prior with EM algorithm to capture the clustering structure of distributed word representations. Experimental results demonstrate the effectiveness of the proposed method.

STRENGTHS:
1. The proposed method for few-shot topic modeling looks novel and intuitive to me.
2. The experiment looks inclusive and the proposed method achieved leading performance.

WEAKNESSES:
1. Though the paper is generally well-written, some parts are confusing to me. Please refer to the questions part.
2. [1] also follows a clustering perspective for topic modeling and the difference between [1] and the proposed method can be discussed.


[1] Effective Neural Topic Modeling with Embedding Clustering Regularization

QUESTIONS:
1. In Eq 1, is it $Z^{(i)\intercal}Z^{(i)}$ or $Z^{(i)}Z^{(i)\intercal}$? What is the relationship between $\hat{A}$ and $A$?
2. In the introduction line 49, the authors claim that "task-specific semantic graphs between words using well-established dependency parsing tools". However, I fail to find any information about the semantic graph using parsing tools in the rest of the paper.

LIMITATIONS:
The limitations are not discussed in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27