Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

1 INTRODUCTION . For text analysis , topic models are widely used to extract a set of latent topics from a corpus ( a collection of documents ) . The extracted topics , revealing common word co-occurrence patterns within a document , often correspond to semantically meaningful concepts in the training corpus . Bayesian probabilistic topic models ( BPTMs ) , such as latent Dirichlet allocation ( LDA ) ( Blei et al. , 2003 ; Griffiths & Steyvers , 2004 ) and its nonparametric Bayesian generalizations ( Teh et al. , 2006 ; Zhou et al. , 2012 ) , have been the most popular ones . A BPTM is often formulated as a generative model that explains how each word of a document is generated given a set of topics and document-specific topic proportions . Bayesian inference of a BPTM is usually based on Gibbs sampling or variational inference ( VI ) , which can be less scalable for big corpora and need to be customized accordingly . With the recent development in auto-encoding VI , originated from variational autoencoders ( VAEs ) ( Kingma & Welling , 2014 ; Rezende et al. , 2014 ) , deep neural networks have been successfully used to develop neural topic models ( NTMs ) ( Miao et al. , 2016 ; Srivastava & Sutton , 2017 ; Burkhardt & Kramer , 2019 ; Zhang et al. , 2018 ; Dieng et al. , 2020 ; Zhao et al. , 2021 ) . The key advantage of NTMs is that approximate posterior inference can be carried out easily via a forward pass of the encoder network , without the need for expensive iterative inference scheme per test observation as in both Gibbs sampling and conventional VI . Hence , NTMs enjoy better flexibility and scalability than BPTMs . However , the reparameterization trick in VAEs can not be directly applied to the Dirichlet ( Burkhardt & Kramer , 2019 ) or gamma distributions ( Zhang et al. , 2018 ) , which are usually used as the prior and conditional posterior of latent topics and topic proportions , so approximations have to be used , potentially putting additional complexity or approximation errors . ∗Equal contribution . To address the above shortcomings , we in this paper propose a novel topic modeling framework in an intuitive and effective manner of enjoying several appealing properties over previously developed BPTMs and NTMs . Like other TMs , we also focus on learning the global topics shared across the corpus and the document-specific topic proportions , which are the two key outputs of a topic model . Without building an explicit generative process , we formulate the learning of topic model ( e.g. , optimizing the likelihood ) as the process of minimizing the distance between each observed document j and its corresponding trainable distribution . More specifically , the former ( document j ) can be regarded as as an empirical discrete distribution Pj , which has an uniform measure over all the words within this document . To construct the latter ( trainable distribution ) , we can represent Pj with K shared topics and its K-dimensional document-specific topic proportion , defined as Qj , where we view shared topics as K elements and topic proportion as the probability measure in Qj . It is very reasonable since the k-th element in topic proportion measures the weight of topic k for a document , and the document can be represented perfectly using the learned topic proportion and topics from a desired TM . Recalling that each topic and word are usually live in the V -dimensional ( vocabulary size ) space in TMs , it might be difficult to directly optimize the distance between Pj and Qj over V -dimensional space . Motivated by Dieng et al . ( 2020 ) , we further assume that both topics and words live in a H-dimensional embedding space , much smaller than vocabulary space . By abuse of notation , we still use Pj over the word embeddings and Qj over the topic embeddings as two representations for document j . Below , we turn towards pushing the document-specific to-be-learned distribution Qj to be as close as possible to the empirical distribution Pj . To this end , we develop a probabilistic bidirectional transport based method to measure the semantic difference between the two discrete distributions in an embedding space . By minimizing the expected difference between two Pj and Qj over all documents , we can learn the topic and word embeddings directly . Importantly , we naturally leverage semantic distances between topics and words in an embedding space to construct the point-to-point cost of moving between them , where the cost becomes a function of topic embeddings . Notably , we consider linking the word embeddings in Pj and topic embeddings in Qj in a bidirectional view . That is , given a word embedding drawn from Pj , it is more likely to be linked to a topic embedding that both is closer to it in the embedding space and exhibits a larger proportion in Qj ; vice versa . Our proposed framework has several key properties : 1 ) By bypassing the generative process , our proposed framework avoids the burden of developing complex sampling schemes or approximations for the posterior of BPTMs or NTMs . 2 ) The design of our proposed model complies with the principles of TMs , whose each learned topic describes an interpretable semantic concept . More interestingly , our model is flexible to learn word embeddings from scratch or use/finetune pretrained word embeddings . When pretrained word embeddings are used , our model naturally alleviates the issue of insufficient word co-occurrence information in short texts as discussed by prior work ( Dieng et al. , 2020 ; Zhao et al. , 2017 ; 2021 ) , which is one of the key drawbacks of many BPTMs and NTMs . 3 ) Conventional TMs usually enforce a simplex constraint on the topics over a fixed vocabulary , which hinders their applications in the case where the vocabulary varies . In our method , we view a document as a mixture of a set of words , which facilitates the deployment of the model when the size of the vocabulary varies . Finally , we have conducted comprehensive experiments on a wide variety of datasets in the comparison with advanced BPTMs and NTMs , which show that our model can achieve the state-of-the-art performance as well as applealing interpretability . 2 BACKGROUND . Topic Models : TMs usually represent each document in a corpus as a bag-of-words ( BoW ) count vector x ∈ RV+ , where xv represents the occurrences of word v in the vocabulary of size V . A TM aims to discover K topics in the corpus , each of which describes a specific semantic concept . A topic is or can be normalized into a distribution over the words in the vocabulary , named word distribution , φk ∈ ΣV , where ΣV is a V − 1 dimensional simplex and φvk indicates the weight or relevance of word v under this topic k. Each document comes from a mixture of topics , associated with a specific mixture proportion , which can be captured by a distribution over K topics , named topic proportion , θ ∈ ΣK , where θk indicates the weight of topic k for a document . As the most fundamental and popular series of TMs , BPTMs ( Blei et al. , 2003 ; Zhou et al. , 2012 ; Hoffman et al. , 2010 ) generate the document x with latent variables ( i.e. , topics { φk } Kk=1 and topic proportion θ ) sampled from pre-specified prior distributions ( e.g. , Gamma or Dirichlet ) . Like other Bayesian models , the learning process of a BPTM relies on Bayesian inference , such as variational inference or Gibbs sampling . Recently , NTMs ( Miao et al. , 2016 ; Srivastava & Sutton , 2017 ; Burkhardt & Kramer , 2019 ; Zhang et al. , 2018 ; Dieng et al. , 2020 ; Zhao et al. , 2021 ) have attracted significant research interests in topic modeling . Most existing NTMs can be regarded as extensions of BPTM like LDA within the VAEs framework ( Zhao et al. , 2021 ) . In general , NTMs consist of an encoder network that maps the ( normalized ) BoW input x to its topic proportion θ , and a decoder network that generates x conditioned on the topics { φk } Kk=1 and proportion θ . Despite their appealing flexibility and scalability , due to the unusable reparameterization trick in original VAEs for the Dirichlet or gamma distributions , NTMs have to develop complex sampling schemes or approximations , leading to potentially large approximation errors or learning complexity . Compare Two Discrete Distributions : This paper aims to quantify the difference between two discrete distributions ( word embeddings and topic embeddings ) , whose supports are points in the same embedding space . Specifically , let p and q be two discrete probability measures on the arbitrary space X ⊆ RH , formulated as p = ∑n i=1 uiδxi and q = ∑m j=1 vjδyj , where u = [ ui ] ∈ Σn and v = [ vj ] ∈ Σm denote two distributions of the discrete states . To measure the distance between p and q , a non-trivial way is to use optimal transport ( OT ) ( Peyré & Cuturi , 2019 ) , which defines the distance as an optimization problem as OT ( p , q ) : = min T∈Π ( u , v ) Tr ( T > C ) , ( 1 ) where C ∈ Rn×m≥0 is the transport cost matrix with Cij = c ( xi , yj ) , T ∈ R n×m > 0 a doubly stochastic transport matrix such that Π ( u , v ) = { T | T1Dv = u , T > 1Du = v } , Tij the transport probability between xi and yj , and Tr ( · ) the matrix trace . Since the transport plan is imposed on the constraint of T ∈ Π ( u , v ) , it has to be computed via constrained optimizations , such as the iterative Sinkhorn algorithm when an additional entropy regularization term is added ( Cuturi , 2013 ) . The recently introduced conditional transport ( CT ) framework ( Zheng & Zhou , 2020 ) can be used to measure the difference between two discrete distributions , which , like OT distance , does not require the distributions to share the same support . CT considers the transport plan in a bidirectional view , which consists of a forward transport plan as Tp→q and backward transport plan Tp←q . Therefore , the transport cost between two empirical distributions in CT can be expressed as CT ( p , q ) : = min Tp→q , Tq→p Tr [ ( Tp→q ) > C + ( Tq→p ) > C ] . ( 2 ) CT specifies Tp→qij =ui vje −dψ ( yj , xi ) ∑m j′=1 vj′e −dψ ( xi , yj′ ) and Tp←qij =vj uie −dψ ( yjxi ) ∑n i′=1 ui′e −dψ ( xi′ , yj ) and hence Tp→qij 1Dv = u and ( T p←q ij ) T1Du = v but in general T p←q ij 1Dv 6= u and ( T p→q ij ) T1Du 6= v. This provides a simpler way to measure the difference between p and q . Here dψ ( x , y ) = dψ ( y , x ) parameterized by ψ is defined to measure the difference between two vectors . This flexibility of CT potentially facilitates an easier integration with deep neural networks with a lower complexity and better scalability . These properties can be helpful to us in the development of a new topic modeling framework based on transportation between distributions , especially for neural topic models .

Summary: The manuscript proposes a combined topic modeling and word embedding method.  The idea is to define word and topic embeddings in the same space, represent documents based on both of them, and use a transport-based method to train these two representations to be similar to each other.  The final objective function combines a "conditional transport" objective with a log-likelihood term, and is optimized via Adam.  Experiments find that the method achieves good topic coherence scores and document clustering metrics compared to baselines.