Multi-source Multi-view Transfer Learning in Neural Topic Modeling with Pretrained Topic and Word Embeddings

1 INTRODUCTION . Probabilistic topic models , such as LDA ( Blei et al. , 2003 ) , Replicated Softmax ( RSM ) ( Salakhutdinov & Hinton , 2009 ) and Document Neural Autoregressive Distribution Estimator ( DocNADE ) ( Larochelle & Lauly , 2012 ) are often used to extract topics from text collections and learn latent document representations to perform natural language processing tasks , such as information retrieval ( IR ) . Though they have been shown to be powerful in modeling large text corpora , the topic modeling ( TM ) still remains challenging especially in the sparse-data setting , especially for the cases where word co-occurrence data is insufficient e.g. , on short text or a corpus of few documents . To this end , several works ( Das et al. , 2015 ; Nguyen et al. , 2015 ; Gupta et al. , 2019 ) have introduced external knowledge in traditional topic models via word embeddings Pennington et al . ( 2014 ) . However , no prior work in topic modeling has employed topical embeddings ( obtained from large document collection ( s ) ) , complementary to word embeddings . Local vs Global Views : Though word embeddings ( Pennington et al. , 2014 ) and topics are complementary in how they represent the meaning , they are distinctive in how they learn from word occurrences observed in text corpora . Word embeddings have local context ( view ) in the sense that they are learned based on local collocation pattern in a text corpus , where the representation of each word either depends on a local context window ( Mikolov et al. , 2013 ) or is a function of its sentence ( s ) ( Peters et al. , 2018 ) . Consequently , the word occurrences are modeled in a fine-granularity . On other hand , a topic ( Blei et al. , 2003 ; Gupta et al. , 2019 ) has a global word context ( view ) : TM infers topic distributions across documents in the corpus and assigns a topic to each word occurrence , where the assignment is equally dependent on all other words appearing in the same document . Therefore , it learns from word occurrences across documents and encodes a coarse-granularity description . Unlike topics , the word embeddings do not capture thematic structures ( topical semantics ) underlying in the document collection . Consider the following topics ( Z1-Z4 ) , where ( Z1-Z3 ) are respectively obtained from different ( high-resource ) source ( S1-S3 ) domains whereas Z4 from the ( low-resource ) target domain T in the data-sparsity setting : Z1 ( S1 ) : profit , growth , stocks , apple , fall , consumer , buy , billion , shares→ Trading Z2 ( S2 ) : smartphone , ipad , apple , app , iphone , devices , phone , tablet→ Product Line Z3 ( S3 ) : microsoft , mac , linux , ibm , ios , apple , xp , windows→ Operating System/Company Z4 ( T ) : apple , talk , computers , shares , disease , driver , electronics , profit , ios→ ? Usually , the top words associated with topics learned on a large corpus are semantically coherent and represent meaningful semantics , e.g. , Trading , Product Line , etc . However in sparse-data setting , topics ( e.g. , Z4 ) are incoherent ( noisy ) and therefore , it is difficult to infer meaningful semantics . Additionally , notice that the word apple is topically/thematically contextualized ( topic-word association ) by different semantics in S1-S3 and referring to a Company . Unlike the topics , word embeddings encode syntactic and semantic relatedness in fine-granularity and therefore , do not capture thematic structures . For instance , the top-5 nearest neighbors ( NN ) of apple ( below ) in the embeddings ( Mikolov et al. , 2013 ) space suggest that it refers to a fruit ; however , they do not express anything about its thematic context , e.g. , Health . apple NN==⇒ apples , pear , fruit , berry , pears , strawberry Motivation ( 1 ) Knowledge transfer using pretrained word and topic embeddings : Essentially , the application of TM aims to discover hidden thematic structures ( i.e. , topics ) in text collection ; however , it is challenging in data sparsity settings , e.g , in a short and/or small collection . This leads to suboptimal text representations and incoherent topics ( e.g. , topic Z4 ) . To alleviate the data sparsity issues , recent works ( Das et al. , 2015 ; Nguyen et al. , 2015 ; Gupta et al. , 2019 ) have shown that TM can be improved by introducing external knowledge , where they leverage pretrained word embeddings ( i.e. , local view ) only . However , the word embeddings ignore the thematically contextualized structures ( i.e. , document-level semantics ) , and can not deal with ambiguity . Given that the word and topic representations encode complementary information , no prior work has explored transfer learning in TM using pretrained topics obtained from a large corpus . Motivation ( 2 ) Knowledge transfer from multiple sources of word and topic embeddings : Knowledge transfer via word embeddings is vulnerable to negative transfer ( Cao et al. , 2010 ) on the target domain when domains are shifted and not handled properly . For instance , consider a short-text document v : [ apple gained its US market shares ] in the target domain T . Here , the word apple refers to a company , and hence the word vector of apple ( about fruit ) is an irrelevant source of prior knowledge for both v and the topic Z4 . In contrast , one can better model v and amend the noisy Z4 for coherence , given the meaningful word and topic embeddings . Often , there are several topic-word associations in different domains , e.g. , in topics Z1-Z3 . Given a noisy topic Z4 in T and meaningful topics Z1-Z3 of S1-S3 , we identify multiple relevant ( source ) domains and advantageously transfer their word and topic embeddings in order to facilitate meaningful and positive transfer learning in the sparse corpus , T . Contribution ( 1 ) To our knowledge , it is the first work in unsupervised topic modeling framework that introduces ( external ) knowledge transfer using ( a ) Global-view Transfer : Pretrained topic em- beddings instead of using word embeddings exclusively , and ( b ) Multi-view Transfer : Pretrained word and topic embeddings jointly obtained from a large source corpus in order to deal with polysemy and alleviate data sparsity issues in a small target corpus . Contribution ( 2 ) Multi-source Transfer : Moreover , we first learn word and topic representations on multiple source domains to build WordPool and TopicPool , respectively and then perform multi-view and multi-source transfer learning within neural topic modeling by jointly using the complementary representations . In doing so , we guide the ( unsupervised ) generative process of learning hidden topics of the target domain by embeddings in WordPool and TopicPool such that the hidden topics become more meaningful and representative in explaining the target corpus . We evaluate the effectiveness of our transfer learning approaches in neural topic modeling using 7 ( 5 low-resource and 2 high-resource ) target and 5 ( high-resource ) source corpora from news and medical domains , consisting of short-text , long-text , small and large document collections . Particularly , we quantify the quality of text representations via generalization ( perplexity ) , interpretability ( topic coherence ) and text retrieval . The code is provided with the supplementary . 2 KNOWLEDGE TRANSFER IN NEURAL TOPIC MODELING . Consider a sparse target domain T and a set of |S| source domains S , we first prepare two knowledge bases ( KBs ) of representations ( or embeddings ) from each of the sources : ( 1 ) WordPool : Pretrained word embeddings matrices { E1 , ... , E|S| } , where Ek ∈ RE×K , and ( 2 ) TopicPool : Pretrained latent topic embeddings { Z1 , ... , Z|S| } , where Zk ∈ RH×K encodes a distribution over a vocabulary of K words . E and H are word embedding and latent topic dimensions , respectively . While topic modeling on T , we introduce the two types of knowledge transfers from one or many sources : Local ( LVT ) and Global ( GVT ) View Transfer using the two KBs of pretrained word ( i.e. , WordPool ) and topic ( i.e. , TopicPool ) embeddings , respectively . Specially , we employ a neural autoregressive topic model ( i.e. , DocNADE ( Larochelle & Lauly , 2012 ) ) to build the WordPool and TopicPool . Notice that a superscript indicates a source . See Table 1 for the notations used in this work . 2.1 NEURAL AUTOREGRESSIVE TOPIC MODELS . DocNADE ( Larochelle & Lauly , 2012 ) is an unsupervised neural-network based topic model that is inspired by the benefits of NADE ( Larochelle & Murray , 2011 ) and RSM ( Salakhutdinov & Hinton , 2009 ) architectures . RSM has difficulties due to intractability leading to approximate gradients of the negative log-likelihood , while NADE does not require such approximations . On other hand , RSM is a generative model of word count , while NADE is limited to binary data . Specifically , Doc- NADE factorizes the joint probability distribution of words in a document as a product of conditional distributions and efficiently models each conditional via a feed-forward neural network . Algorithm 1 Computation of log p ( v ) and Loss L ( v ) Input : A target training document v , |S| source domains Input : WordPool : KB of pretrained word embedding matrices { E1 , ... , E|S| } Input : TopicPool : KB of pretrained latent topics { Z1 , ... , Z|S| } Parameters : Θ = { b , c , W , U , A1 , ... , A|S| } Hyper-parameters : θ = { λ1 , ... , λ|S| , γ1 , ... , γ|S| , H } Initialize : a← c and p ( v ) ← 1 for i from 1 to D do hi ( v < i ) ← g ( a ) , where g = { sigmoid , tanh } p ( vi = w|v < i ) ← exp ( bw+Uw , :hi ( v < i ) ) ∑ w′ exp ( bw′+Uw′ , :hi ( v < i ) ) p ( v ) ← p ( v ) p ( vi|v < i ) compute pre-activation at step , i : a← a + W : ,vi if LVT then get word embedding for vi from source domain ( s ) a← a + ∑|S| k=1 λ k Ek : ,vi L ( v ) ← − log p ( v ) if GVT then L ( v ) ← L ( v ) + ∑|S| k=1 γ k ∑H j=1 ||Akj , :W − Zkj , :||22 DocNADE Formulation : For a document v = ( v1 , ... , vD ) of size D , each word index vi takes value in { 1 , ... , K } of vocabulary size K. DocNADE learns topics in a language modeling fashion ( Bengio et al. , 2003 ) and decomposes the joint distribution p ( v ) = ∏D i=1 p ( vi|v < i ) such that each autoregressive conditional p ( vi|v < i ) is modeled by a feed-forward neural network using preceding words v < i in the sequence : hi ( v < i ) = g ( c + ∑ q < i W : ,vq ) and p ( vi = w|v < i ) = exp ( bw + Uw , :hi ( v < i ) ) ∑ w′ exp ( bw′ + Uw′ , :hi ( v < i ) ) for i ∈ { 1 , ... D } , where v < i is the subvector consisting of all vq such that q < i i.e. , v < i ∈ { v1 , ... , vi−1 } , g ( · ) is a non-linear activation function , W ∈ RH×K and U ∈ RK×H are weight matrices , c ∈ RH and b ∈ RK are bias parameter vectors . H is the number of hidden units ( topics ) . Figure 1 ( left ) ( without WordPool ) provides an illustration of the ith autoregressive step of the DocNADE architecture , where the parameter W is shared in the feed-forward networks and hi encodes topic-proportion embedding . Importantly , the topic-word matrix W has a property that the column vector W : ,vi corresponds to embedding of the word vi , whereas the row vector Wj , : encodes the jth topic . We leverage this property to introduce external knowledge via word and topic embeddings . Additionally , DocNADE has shown to outperform traditional models such as LDA ( Blei et al. , 2003 ) and RSM ( Salakhutdinov & Hinton , 2009 ) in terms of both the log-probability on unseen documents and retrieval accuracy . Recently , Gupta et al . ( 2019 ) has improved topic modeling on short texts by introducing word embeddings ( Pennington et al. , 2014 ) in DocNADE architecture . Thus , we adopt DocNADE to perform transfer learning within the neural topic modeling framework . Algorithm 1 ( for DocNADE , set LVT and GVT to False ) demonstrates the computation of log p ( v ) and negative log-likelihood L ( v ) that is minimized using gradient descent . Moreover , computing hi is efficient ( linear complexity ) due to NADE architecture that leverages the pre-activation ai−1 of ( i− 1 ) th step in computing the pre-activation ai . See Larochelle & Lauly ( 2012 ) for further details .

Summary: The paper proposes a multi-source and multi-view transfer learning for neural topic modelling with the pre-trained topic and word embedding. The method is based on NEURAL AUTOREGRESSIVE TOPIC MODELs --- DocNADE (Larochelle&Lauly,2012). DocNADE learns topics using language modelling framework. DocNADEe (Gupta et al., 2019) extended DocNADE by incorporating word embeddings, the approach the authors described as a single source extension of the existing method.