JOINTLY LEARNING TOPIC SPECIFIC WORD AND DOCUMENT EMBEDDING

1 INTRODUCTION . In recent years the neural network-based methods have shown great success in various NLP-related tasks ( Zhao et al. , 2021 ) . Tow popular complementary natural language processing methods are word2vec ( Mikolov et al. , 2013a ; b ) and Latent Dirichlet Allocation ( LDA ) ( Blei et al. , 2003 ) ; but executing a model in two-step lacks mutual interaction between the two paradigms and fails to capture mutual reinforcement between themselves . Doc2vec ( Le & Mikolov , 2014 ) generalizes the idea of vector representation for documents as a stand-alone method . Though it is expensive to infer new document vectors during the test process , and regardless of assigning PV to each word in the document as a single topic , it outperforms BOW ( Zhang et al. , 2010 ) and LDA ( Fan et al. , 2008 ) in various text understanding tasks . Encouraged by word2vec and Doc2vec , Niu et al . ( 2015 ) introduced Topic2Vec . It incorporates the LDA topic model into NPLM ( Bengio et al. , 2003 ) . However , Topic2Vec can not capture the polysomic characteristics of terms as LDA models topic occurrence almost independently without capturing topic correlations ( He et al. , 2017 ) . The purpose of conducting this research is to introduce an efficient multi topical document representation with flowing objectives : • Firstly , learn probabilistic topical word embeddings using document vector as a global context where each word can be part of different underlying topics of the document instead of just a single paragraph vector . Thus , avail to capture polysemous terms . • Secondly , embedding documents by simply averaging the topical word embeddings instead of imposing an expensive inference method to learn the document embedding separately . Unlike clustering-based document modeling with the ability to capture multi-sense of words ( Gupta et al. , 2019 ) , here the challenge is to learn polysemous words in a jointly learning framework for topic-specific word and document embedding . Capturing polysemous terms is vital because the same word around different context words can represent different themes ( Liu et al. , 2020 ) and without noticing them can mislead to an unexpected thematic representation of a document . The Jointly Learning Word Embeddings and Latent Topics ( Shi et al. , 2017 ) comes to the rescue for addressing the polysemous word problem , which learns different topical word embeddings using Skip-gram Topical word Embedding ( STE ) framework . STE gets the term distributions of the topics and the topic distributions of the document using PLSA ( Hofmann , 2013 ) like mixture weighting for each data point . However , STE doesn ’ t deal with document embedding in its jointly learning process . In this research , we propose a novel document embedding framework called TDE : Topical Document Embedding . TDE learns topical word and document embeddings jointly . It incorporates the STE-diff framework ( Shi et al. , 2017 ) and follows the simple averaging of word embedding with a corruption mechanism for document embedding introduced by Chen ( 2017 ) . 2 MOTIVATION AND RELATED WORK . Huang et al . ( 2012 ) proved that combining local and global context performs better for predicting target words in the neural network-based language modeling , but this model doesn ’ t support topic-specific word embeddings . Liu et al . ( 2015 ) proposed topic-specific word embeddings where document embeddings get by aggregating over all topical-word embeddings of each word in the document . Arora et al . ( 2017 ) extend embedding to the sentence level by smooth inverse frequency ( SIF ) weighted averaging of the words in a sentence . However , this model is lack of addressing multiple sense of words . Moody ( 2016 ) introduced a hybrid model called lda2vec , which suffers from uncorrelated topics due to the incorporation of LDA . For word sense disambiguation with correlated topic modeling , Chaplot & Salakhutdinov ( 2018 ) replaced the LDA topic proportions by synset proportions for a document . Other than LDA topic modeling , Arora et al . ( 2018 ) have introduced a Gaussian random walk model for word sense disambiguation . This work can address polysemous word representation but doesn ’ t cover document-level embeddings . Mekala et al . ( 2016 ) introduced the Gaussian mixture model-based soft clustering method called SCDV , which incorporates both the topic and word embeddings . However , SCDV is not suitable for multisentence documents . Hence Luo et al . ( 2019 ) proposed P-SIF , which is fit for large documents with multiple sentences while addressing the underlying topics as well . Though P-SIF overcomes SCDV ’ s shortcomings , it is still a two-step process . From the motivation of the above advantages and drawbacks , we introduce a novel document modeling capable of learning document embedding simultaneously during the jointly learning process of topic-specific word embeddings . This model creates a pivot by concatenating a document vector and center word vector to predict the context word vector . Here the document vector works as a memory backup for missing information . The output document embeddings produced by this model is more expressive as it represents each document into a higher dimensional embedding space similar to P-SIF . 3 MODEL DESCRIPTION . Inspired by the Skip-gram Topical word Embedding ( STE ) ( Shi et al. , 2017 ) and the simplicity of the Document Vector through Corruption ( Chen , 2017 ) , we design a novel document embedding model called TDE : Topical Document Embedding . In contrast to Doc2VecC , TDE constructs global context ( document vector ) as a weighted averaging of the randomly selected topic-specific word embeddings . Unlike STE , TDE uses a generating function , which predicts context words by given the topics and pivot . Pivot constructs by concatenating global context ( document vector ) and center word ( Figure1 ) . The probability of predicting a context word depends on the inner product of the topic-specific embedding of that word ( context word ) and the pivot , which ensures the participation of underlying word , topic , and document levels knowledge . 3.1 TOPICAL DOCUMENT EMBEDDING . A matrix D represents the documents in the corpus , D = d1 , d2 , . . . , dn of size n. Each document d contains a variable-length sequence of wordsW = w1 , w2 , . . . , wTd . Each wordw in the vocabulary V of size v of the corpus is generally associated with an input projection matrix U Rh×v of size h and an output projection matrix V T Rv×h of size h. The input projection matrix represents the input to the hidden space , whereas the output projection matrix represents the hidden to output space . Matrix V T has the same dimension as matrix U but a transpose matrix . The column uw of matrix U is the input vector of word w , and the column vw of matrix V T is the output vector of word w. TDE follows an unbiased mask-out/drop-out corruption mechanism in each update of its stochastic process for randomly select topic-specific word embeddings to create the global context . Under corruption mechanism , each dimension of the original document d overwrites with the probability q and sets the uncorrupted dimensions 1 ( 1−q ) times of their original values to make the corruption unbiased . The corruption mechanism reduces word-level noise in the document by forcing the embeddings of common and non-discriminative ones close to zero . This corruption mechanism leads TDE to speed up the training process by significantly pruning the number of parameters for the backpropagation mechanism , document vectors represented by only embeddings of remaining words . The resultant global context is thus as below : x̃d = 0 , with probability q d 1−q , otherwise ( 1 ) Document vector ( global context ) x̃ = 1 T ∑ x̃d ( 2 ) where T is the length of the document ( number of words ) and the document vector x̃ is the weighted average of the words in a document , weighted by the corruption mechanism mentioned above . The probability of observing the context word wt+i given the center word wt as well as the global context x̃ and the topics zt+i and zt for context and center word respectively as p ( wt+i |wt , zt+i , zt , x̃ ) = exp ( V Twt+i , zt+i . ( Uwt , zt + 1 T Ux̃d ) ) ∑ w′ Λ exp ( V T w′ , zt+i . ( Uwt , zt + 1 T Ux̃d ) ) ( 3 ) where Λ is the vocabulary of the whole corpus and input projection matrix constructs as U = Uwt , zt + 1 T Ux̃d . The denominator in equation ( 3 ) is generally intractable due to the term ∑ w′ Λ . As such , the computational cost is proportional to the size of Λ . To deal with this intractability requires an efficient approximation method . TDE employs the approximation with the negative sampling ( Mikolov et al. , 2013a ; Chen , 2017 ) . In TDE , the probability of predicting a context word wt+i in document d depends on the pivot wtx̃ and the topic distributions of the document . The pivot constructed by concatenating the center word wt and the global context x̃ as- Pivot = wtx̃ = wt + x̃ ( 4 ) So , for every skip-gram 〈wt , wt+i〉 , where the topic assignment zt+i of wt+i is independent of the topic assignment zt of wt we have : p ( wt+i |wtx̃ , d ) = k∑ zt=1 k∑ zt+i=1 p ( wt+i | wtx̃ , zt+i , zt ) p ( zt+i , zt |d ) = k∑ zt=1 k∑ zt+i=1 p ( wt+i |wtx̃ , zt+i , zt ) p ( zt|d ) p ( zt+i |d ) ( 5 ) Following a similar analogy as STE-diff , equation ( 5 ) assumes that the context word and the center word may come from the same topic or may come from two different topics as well without imposing any hard constraint ( Figure 2 ) . Here , multiple meanings of a word depend on their contexts instead of directly k topics disjoint subspace partitioning as commonly does in clustering-based topic modeling ( Gupta et al. , 2016 ) . In TDE , word embeddings from different topics co-exist in a common shared space . Instead of the free flow of scattered words in the embedding space , different topical word embeddings stay close together document-wise rather than topic-wise gathering . Creating pivots by concatenating the center word with the document vector provides extra guidance as a global context with a complete thematic sense of the document . Thus , TDE represents more consistent topical word embeddings over STE-diff . In TDE , the compositionality of words encodes in the learned embeddings ; as a result , the requirement of heuristic weighting for test documents is no more required ; each output document vector is represented simply as an average of the topic-specific word embeddings . Given the learned projection matrix Uw , z , TDE forms document embedding dvec as below : dvec = ⊕kz=1 1 T ∑ w D Uw , z ( 6 ) where T is the length ( number of words ) of the document and k represents the underlying topics of the document . In equation ( 6 ) , the term ⊕kz=1 represents topic-wise concatenation . In this case , words in the same topic are just simple average as they represent the same thematic meaning . However , words in different topics carry different themes . Thus , concatenate over topics increases the expressive power of a document . This strategy represents documents into a higher k × s dimension , where k is the underlying topics , and s is the dimension of the word embeddings .

Summary: Jointly learning word embeddings and document embeddings while taking into account topical information is somewhat interesting. The paper presents an improved skip-gram model where each individual word is associated with two matrices (e.g., the input projection matrix and the output projection matrix) which is designed to capture the topical aspect of words. In other words, each word is associated with K topic embeddings. While computing the predictive probability, the proposed model further considers a global context, i.e, a document embedding computed with the idea of document corruption. Basically, this paper simply combines the work of Shi et al. 2017 and Chen 2017, Both the innovation and experimental improvement are mariginal.