Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

1 INTRODUCTION . For text analysis , topic models are widely used to extract a set of latent topics from a corpus ( a collection of documents ) . The extracted topics , revealing common word co-occurrence patterns within a document , often correspond to semantically meaningful concepts in the training corpus . Bayesian probabilistic topic models ( BPTMs ) , such as latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ; Griffiths & Steyvers , 2004 ) and its nonparametric Bayesian generalizations ( Teh et al. , 2006 ; Zhou et al. , 2012 ) , have been the most popular ones . A BPTM is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions . Bayesian inference of a BPTM is usually based on Gibbs sampling or variational inference ( VI ) , which can be less scalable for big corpora and need to be customized accordingly . With the recent development in auto-encoding VI , originated from variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) , deep neural networks have been successfully used to develop neural topic models ( NTMs ) ( Miao et al. , 2016 ; Srivastava & Sutton , 2017 ; Burkhardt & Kramer , 2019 ; Zhang et al. , 2018 ; Dieng et al. , 2020 ; Zhao et al. , 2021 ) . The key advantage of NTMs is that approximate posterior inference can be carried out easily via a forward pass of the encoder network , without the need for expensive iterative inference scheme per test observation as in both Gibbs sampling and conventional VI . Hence , NTMs enjoy better flexibility and scalability than BPTMs . However , the reparameterization trick in VAEs can not be directly applied to the Dirichlet ( Burkhardt & Kramer , 2019 ) or gamma distributions ( Zhang et al. , 2018 ) , which are usually used as the prior and conditional posterior of latent topics and topic proportions , so approximations have to be used , potentially putting additional complexity or approximation errors . ∗Equal contribution . To address the above shortcomings , we in this paper propose a novel topic modeling framework in an intuitive and effective manner of enjoying several appealing properties over previously developed BPTMs and NTMs . Like other TMs , we also focus on learning the global topics shared across the corpus and the document-specific topic proportions , which are the two key outputs of a topic model . Without building an explicit generative process , we formulate the learning of topic model ( e.g. , optimizing the likelihood ) as the process of minimizing the distance between each observed document j and its corresponding trainable distribution . More specifically , the former ( document j ) can be regarded as as an empirical discrete distribution Pj , which has an uniform measure over all the words within this document . To construct the latter ( trainable distribution ) , we can represent Pj with K shared topics and its K-dimensional document-specific topic proportion , defined as Qj , where we view shared topics as K elements and topic proportion as the probability measure in Qj . It is very reasonable since the k-th element in topic proportion measures the weight of topic k for a document , and the document can be represented perfectly using the learned topic proportion and topics from a desired TM . Recalling that each topic and word are usually live in the V -dimensional ( vocabulary size ) space in TMs , it might be difficult to directly optimize the distance between Pj and Qj over V -dimensional space . Motivated by Dieng et al . ( 2020 ) , we further assume that both topics and words live in a H-dimensional embedding space , much smaller than vocabulary space . By abuse of notation , we still use Pj over the word embeddings and Qj over the topic embeddings as two representations for document j . Below , we turn towards pushing the document-specific to-be-learned distribution Qj to be as close as possible to the empirical distribution Pj . To this end , we develop a probabilistic bidirectional transport based method to measure the semantic difference between the two discrete distributions in an embedding space . By minimizing the expected difference between two Pj and Qj over all documents , we can learn the topic and word embeddings directly . Importantly , we naturally leverage semantic distances between topics and words in an embedding space to construct the point-to-point cost of moving between them , where the cost becomes a function of topic embeddings . Notably , we consider linking the word embeddings in Pj and topic embeddings in Qj in a bidirectional view . That is , given a word embedding drawn from Pj , it is more likely to be linked to a topic embedding that both is closer to it in the embedding space and exhibits a larger proportion in Qj ; vice versa . Our proposed framework has several key properties : 1 ) By bypassing the generative process , our proposed framework avoids the burden of developing complex sampling schemes or approximations for the posterior of BPTMs or NTMs . 2 ) The design of our proposed model complies with the principles of TMs , whose each learned topic describes an interpretable semantic concept . More interestingly , our model is flexible to learn word embeddings from scratch or use/finetune pretrained word embeddings . When pretrained word embeddings are used , our model naturally alleviates the issue of insufficient word co-occurrence information in short texts as discussed by prior work ( Dieng et al. , 2020 ; Zhao et al. , 2017 ; 2021 ) , which is one of the key drawbacks of many BPTMs and NTMs . 3 ) Conventional TMs usually enforce a simplex constraint on the topics over a fixed vocabulary , which hinders their applications in the case where the vocabulary varies . In our method , we view a document as a mixture of a set of words , which facilitates the deployment of the model when the size of the vocabulary varies . Finally , we have conducted comprehensive experiments on a wide variety of datasets in the comparison with advanced BPTMs and NTMs , which show that our model can achieve the state-of-the-art performance as well as applealing interpretability . 2 BACKGROUND . Topic Models : TMs usually represent each document in a corpus as a bag-of-words ( BoW ) count vector x ∈ RV+ , where xv represents the occurrences of word v in the vocabulary of size V . A TM aims to discover K topics in the corpus , each of which describes a specific semantic concept . A topic is or can be normalized into a distribution over the words in the vocabulary , named word distribution , φk ∈ ΣV , where ΣV is a V − 1 dimensional simplex and φvk indicates the weight or relevance of word v under this topic k. Each document comes from a mixture of topics , associated with a specific mixture proportion , which can be captured by a distribution over K topics , named topic proportion , θ ∈ ΣK , where θk indicates the weight of topic k for a document . As the most fundamental and popular series of TMs , BPTMs ( Blei et al. , 2003 ; Zhou et al. , 2012 ; Hoffman et al. , 2010 ) generate the document x with latent variables ( i.e. , topics { φk } Kk=1 and topic proportion θ ) sampled from pre-specified prior distributions ( e.g. , Gamma or Dirichlet ) . Like other Bayesian models , the learning process of a BPTM relies on Bayesian inference , such as variational inference or Gibbs sampling . Recently , NTMs ( Miao et al. , 2016 ; Srivastava & Sutton , 2017 ; Burkhardt & Kramer , 2019 ; Zhang et al. , 2018 ; Dieng et al. , 2020 ; Zhao et al. , 2021 ) have attracted significant research interests in topic modeling . Most existing NTMs can be regarded as extensions of BPTM like LDA within the VAEs framework ( Zhao et al. , 2021 ) . In general , NTMs consist of an encoder network that maps the ( normalized ) BoW input x to its topic proportion θ , and a decoder network that generates x conditioned on the topics { φk } Kk=1 and proportion θ . Despite their appealing flexibility and scalability , due to the unusable reparameterization trick in original VAEs for the Dirichlet or gamma distributions , NTMs have to develop complex sampling schemes or approximations , leading to potentially large approximation errors or learning complexity . Compare Two Discrete Distributions : This paper aims to quantify the difference between two discrete distributions ( word embeddings and topic embeddings ) , whose supports are points in the same embedding space . Specifically , let p and q be two discrete probability measures on the arbitrary space X ⊆ RH , formulated as p = ∑n i=1 uiδxi and q = ∑m j=1 vjδyj , where u = [ ui ] ∈ Σn and v = [ vj ] ∈ Σm denote two distributions of the discrete states . To measure the distance between p and q , a non-trivial way is to use optimal transport ( OT ) ( Peyré & Cuturi , 2019 ) , which defines the distance as an optimization problem as OT ( p , q ) : = min T∈Π ( u , v ) Tr ( T > C ) , ( 1 ) where C ∈ Rn×m≥0 is the transport cost matrix with Cij = c ( xi , yj ) , T ∈ R n×m > 0 a doubly stochastic transport matrix such that Π ( u , v ) = { T | T1Dv = u , T > 1Du = v } , Tij the transport probability between xi and yj , and Tr ( · ) the matrix trace . Since the transport plan is imposed on the constraint of T ∈ Π ( u , v ) , it has to be computed via constrained optimizations , such as the iterative Sinkhorn algorithm when an additional entropy regularization term is added ( Cuturi , 2013 ) . The recently introduced conditional transport ( CT ) framework ( Zheng & Zhou , 2020 ) can be used to measure the difference between two discrete distributions , which , like OT distance , does not require the distributions to share the same support . CT considers the transport plan in a bidirectional view , which consists of a forward transport plan as Tp→q and backward transport plan Tp←q . Therefore , the transport cost between two empirical distributions in CT can be expressed as CT ( p , q ) : = min Tp→q , Tq→p Tr [ ( Tp→q ) > C + ( Tq→p ) > C ] . ( 2 ) CT specifies Tp→qij =ui vje −dψ ( yj , xi ) ∑m j′=1 vj′e −dψ ( xi , yj′ ) and Tp←qij =vj uie −dψ ( yjxi ) ∑n i′=1 ui′e −dψ ( xi′ , yj ) and hence Tp→qij 1Dv = u and ( T p←q ij ) T1Du = v but in general T p←q ij 1Dv 6= u and ( T p→q ij ) T1Du 6= v. This provides a simpler way to measure the difference between p and q . Here dψ ( x , y ) = dψ ( y , x ) parameterized by ψ is defined to measure the difference between two vectors . This flexibility of CT potentially facilitates an easier integration with deep neural networks with a lower complexity and better scalability . These properties can be helpful to us in the development of a new topic modeling framework based on transportation between distributions , especially for neural topic models .

Summary: This paper presents a new topic modeling framework called WeTe: each document in text corpus is represented as a bag of word embeddings vectors, and each topic is modeled as an embedding vector in the shared word embedding space. WeTe minimizes the expected difference between those two sets over all documents. A bidirectional transport-based method is proposed to learn the topic embeddings as well as topic proportions for documents efficiently. Extensive experiments on news and web pages show that the proposed model outperforms competitive methods for both deriving high-quality latent topics and generating better document representations for clustering tasks.