Unsupervised Document Representation using Partition Word-Vectors Averaging

1 INTRODUCTION . Distributed word embeddings such as word2vec ( Mikolov et al. , 2013b ) have shown their success in representing words as latent continuous features in low-dimensional fixed spaces , which can capture their semantic meaning . These embeddings have shown impressive success for improving the performance of machine learning algorithms such as text classification tasks . The success of these word representations is recently formally explained using the random walk-based latent variable model by ( Arora et al. , 2016a ) . Many approaches such as ( Socher et al. , 2013 ; Le & Mikolov , 2014 ; Ling et al. , 2015 ; Liu et al. , 2015a ) are proposed which go beyond words to capture the semantic meaning of sentences . These techniques either used simple composition of the word-vectors or sophisticated neural network architectures for sentence representation . Recently , ( Arora et al. , 2017 ) proposed smooth inverse frequency of word vector averaging model to embed a sentence . They further improved their embedding by removing the first principal component of the weighted average vectors . However , all these approaches are limited to capturing the meaning of a single sentence and represent the sentence in the same space as words , thus reducing their expressive power . Generally , a longer text contain words from multiple different themes ( or topics ) , and creating a single vector from simple averaging of word-vectors will disregard all the thematic structure . Hence , these techniques are largely unable to capture the semantic meaning of larger pieces of text ( phrases , sentences and paragraphs ) . To address this problem , in this paper we present a novel document embedding method called partition SIF weighted averaging ( P-SIF ) , which can efficiently embed documents with multiple sentences . P-SIF learns topic-specific vectors from a document and finally concatenates them all to represent the overall document . Thus , P-SIF retains the simplicity of simple weighted word averaging , while taking a document ’ s thematic structure into account . ( Mekala et al. , 2017 ) recently proposed a clustering based word-vector averaging approach for embedding larger text documents . However , they did not provide any justification or experiments to explain the functionality of their embedding approach . This paper provides an analytical explanation about the success of the mentioned representations and also achieves significant improvements over previous embedding techniques on several natural language tasks . Following are the characteristics of P-SIF . • P-SIF is capable of embedding larger documents with multiple sentences , as it pays attention to the topic/thematic structure of the document . • P-SIF is simple because it is based on weighted word vectors averaging rather than complicated tensor factorization or neural networks methods . • P-SIF is an unsupervised method since it is obtained using pre-trained unsupervised word embeddings without using any label information . • P-SIF outperforms many existing approaches on textual similarity , textual classification and other supervised tasks . The remaining part of the paper is organized as follows . Section 2 discusses related work in document representations . Section 3 provides motivation for the partition averaging through a qualitative example . Section 4 describes our embedding algorithm . Section 5 introduces and explains the background needed . These sections are followed by experiments in Section 6 and the analysis with the discussion in Section 7 . 2 RELATED WORK . Word embeddings . There are mainly two methodologies proposed for unsupervised wordembedding representations , either by internal representations learned through neural network models of text ( Bengio et al. , 2003 ; Collobert & Weston , 2008 ) , and Mikolov et al . ( 2013b ) or by low-rank approximation of co-occurrence statistics by ( Levy & Goldberg , 2014 ; Hashimoto et al. , 2016 ) , and Arora et al . ( 2016a ) . ( Levy & Goldberg , 2014 ) show that both techniques are equivalent . Sentence embeddings . Earlier work has computed sentence embedding by coordinate wise vector and matrix based compositional operation over word vectors , e.g. , ( Levy & Goldberg , 2014 ) use unweighted averaging of word vectors for representing short phrases , ( Singh & Mukerjee , 2015 ) proposed tfidf-weighted averaging of word vectors to form document vectors , ( Socher et al. , 2013 ) proposed a recursive neural network ( RNN ) defined over a parse tree and trained with supervision . Next , ( Le & Mikolov , 2014 ) proposed PV-DM and PV-DBOW model which treats each sentence as a shared global latent vector ( or pseudo word ) . Other approaches use seq2seq models such as Recurrent Neural Networks ( Mikolov et al. , 2010 ) and Long Short Term Memory ( Gers et al. , 2002 ) which can handle long term dependency , hence , capturing the syntax structure . Other neural network models include use of hierarchy and convolution neural networks , such as ( Kim , 2014 ) and ( Kalchbrenner et al. , 2014 ) . ( Wieting et al. , 2015 ) learned paraphrastic sentence embedding by modifying word embeddings based on the supervision from the Paraphrase pairs dataset ( PPDB ) . Recently , a lot of work is harnessing topic modeling ( Blei et al. , 2003 ) along with word vectors to learn better word and sentence representations , e.g. , TWE ( Liu et al. , 2015a ) , NTSG ( Liu et al. , 2015a ) , WTM ( Fu et al. , 2016 ) , w2v-LDA ( Nguyen et al. , 2015 ) , TV+MeanWV ( Li et al. , 2016a ) , LTSG ( Law et al. , 2017 ) , Gaussian-LDA ( Das et al. , 2015 ) , Topic2Vec ( Niu et al. , 2015 ) , Lda2vec ( Moody , 2016 ) , and MvTM ( Li et al. , 2016b ) . Document embeddings . ( Kiros et al. , 2015 ) proposed skip-thought document embedding vectors which transformed the idea of abstracting the distributional hypothesis from word level to sentence level . ( Wieting et al. , 2016a ) proposed a neural network model which optimizes the word embeddings based on the cosine similarity of the sentence embeddings . ( Gupta et al. , 2016 ) proposed a method called Bag of Words Vector ( BoWV ) , which employs a clustering based technique and tf-idf values to form a composite document vector . They represented documents in higher dimensions by using hard clustering over word embeddings . ( Mekala et al. , 2017 ) later extended the model to SCDV by using a fuzzy clustering technique and direct idf weighting of word vectors . The learned representations tried to capture a global context of sentence , similar to an n-gram model . Their method outperformed previous state of art on a variety of NLP tasks . Table 1 : Words with their Topic Proportions word 1 2 3 4 5 data 0.3 0.7 0.0 0.0 0.0 interviewing 0.0 0.0 0.8 0.0 0.2 management 0.0 0.0 0.8 0.0 0.2 public 0.0 0.0 0.0 0.7 0.3 Figure 1 : Word vector space for corpus C. Words in different partitions are represented by different subscripts and separated by hyperplanes . Bold fonts represent words ’ presence in the document 3 MOTIVATION : AVERAGING VS PARTITION AVERAGING . Let ’ s consider a corpus ( C ) with N documents with the corresponding most frequent words vocabulary ( V ) . Figure 1 , represents the word-vectors space of V , where similar meaning words occur closer to each other . We can apply sparse coding to partition the word vectors space to a five ( total topics K = 5 ) topic vector space , as shown in figure 1 . These five topic vector spaces represent the five topics present in corpus ( C ) . Few words are polysemic and belong to multiple topics with some proportion ( see Table 1 ) . In figure 1 we represent the topic number of the word in subscript . Let ’ s consider a document dn : ‘ Data journalists deliver data science news to general public . They often take part in interpreting the data models . In addition , they create graphical designs and interview the directors and CEO ’ s. ‘ One can directly average the word vectors to represent the document ( ~vdn ) , as in ( SIF ) , shown in equation equation 1 . Here , + represents element-wise vector addition . ~vdn = ~vdata2 + ~vjournalist4 + ~vnews4 + ~vdatascience1 + ~vpublic4 + ~vinterpreting1 + ~vmodels2+ ~vgraphical1 + ~vdesign1 + ~vdirector5 + ~vCEO5 + ~vinterviewing2 ( 1 ) In representation equation 1 it can be seen that we are averaging word vectors which have a very different semantic meaning , e.g. , words belonging to partition 1 such as graphical , design , and data are averaged with different semantic meaning words of partition 2 such as datascience , model , and data . In addition , the document is represented in the same d dimensional space as word vectors . Overall , averaging represents the documents as a single point in the vector space and does not consider the 5 different semantic topics or themes . However , we do a weighted ( topic proportion ) average of words within a partition and concatenate the average word vectors across partitions to represent the document ( ~vdn ) , as in ( P-SIF ) , shown in equation equation 2 . Here , + represents a element-wise vector addition , × represents scalar-vector multiplication , and ⊕ represents concatenation . ~vdn = ( ~vinterpreting1 + ~vgraphical1 + ~vdesign1 + 0.3× ~vdata1 ) ⊕ ( 0.7× ~vdata2 + ~vdatascience2 +~vmodels2 ) ⊕ ( ~vjournalist4 + ~vnews4 + 0.7× ~vpublic4 ) ⊕ ( ~vdirector5 + 0.3× ~vpublic5 + ~vCEO5 +0.2× ~vinterviewing5 ) ⊕ ( 0.8× ~vinterviewing3 ) ( 2 ) In representation equation 2 , the final representation takes 5 different semantic topics into account . Words belonging to different semantic topics are separated by concatenation ( ⊕ ) as they represent separate meanings , whereas words coming from same topics are simple averages since they represent the same meaning . E.g. , average of words belonging to partition 1 such as graphical , design , and data are concatenated to average of words of partition 2 such as data − science , model , and data . The final document vector ~vdn is represented in a higher 5 × d dimension vector space , thus having more representational power , where d is the dimension of word vectors . From equation 2 , it is clear that the final representation takes the weight to which each word belongs to various topics into account as well , thus handling the multi-sense nature of words , e.g. , words such as data belong to partition 1 with probability 0.3 and partition 2 with probability 0.7 . Hence , partitioned averaging with topic weighting is essential for representing longer text documents . Thus , P-SIF takes documents ’ thematic structure into account while doing simple weighted word vectors averaging . 4 THE PROPOSED ALGORITHM : P-SIF . In this section , we present the proposed document embedding learning algorithm . We introduce the formal notation needed for the discussion . • C represents text corpus and V represents vocabulary of words in corpus . • ~vw ∈ Rd represents the word vector of word w , where d is dimension of word vector . • p ( w ) represents the unigram probability of word w in the corpus . • c0 and ~vc0 ∈ Rd represent common context and it ’ s corresponding context vector . The details to construct document embeddings are given in algorithm 1 . The feature formation algorithm can be divided into three major steps : Dictionary Learning for Word Vectors : Given word vectors vw ∈ Rd , a sparsity parameter k , and an upper bound m , we find a set of unit norm vectors ~A1 , ~A2 , . . . , ~Am , such that ~vw = ∑m j=1 α ( w , j ) ~Aj + ~ηw , where at most k of the coefficients α ( w,1 ) , . . . , α ( w , m ) are nonzero ( so-called sparsity constraint ) , and ~ηw is a noise vector . Sparse coding is usually solved for given m , k by using alternating minimization ( Arora et al. , 2016b ) and ( Aharon et al. , 2006 ) to find the ~A′is that minimizes the following L2-reconstruction error : ‖~vw − ∑m j=1 α ( w , j ) ~Aj‖ . Here , ~A1 , . . . , ~Am will represents important topic basis in the corpus , which we refer to as the atoms of the topic . Furthermore , restricting m to be much smaller than the number of the words ensures that the same topic needs to be used for multiple words . ~Aj is an interesting or significant topic because the sparse coding ensures that each basis element is softly chosen by many words . Word Topics Vector Formation : For each word ~w , we created K different word-cluster vectors of d dimensions ~cvwk by weighting the word embedding with its learned dictionary coefficient αw , k of the kth context.1 . We then concatenated all theK word-cluster vectors ~cvwk into aK×d dimensional embedding to form a word-topics vector ~tvw ∈ RK×d . ~cvw , k ← ~vw × αw , k ; ~tvw ← K⊕ k=1 ~cvwk Here , ⊕ represents concatenation operation , and × represents vector-scalar multiplication . Smooth Inverse Frequency ( SIF ) re-weighted : Finally , for all words appearing in document Dn , we weighted the word-topics vectors ~tvi by smooth inverse frequency ( a a+p ( w ) ) . Next , we removed the common context from the weighted average document vectors by removing the first principal 1 Empirically , we observed that this weighting generally improves the performance component from the weighted average vectors . 2 . ( Arora et al. , 2017 ) empirically shows that SIF weighting outperforms the tf-idf weighting . They provide a theoretical explanation for their superior performance . ~vdn ← ~vdn − ~u.~uT~vdn Algorithm 1 : P-SIF Embedding Data : Word embeddings { ~vw : w ∈ V } , Documents { dn : dn ∈ D } , a set of sentences D , parameter a and estimated probabilities { p ( w ) : w ∈ V } of the words , a sparsity parameter k , and an upper bound m. Result : Document vectors { ~vdn : dn ∈ D } / * Dictionary learning for word-vectors * / 1 for each word w in V do 2 ~vw = ∑m j=1 αw , j ~Aj + ~ηw ; 3 end / * Word topic-vector formation * / 4 for each word w in V do 5 for each coefficient , αw , k of word w do 6 ~cvw , k ← ~vw × αw , k ; 7 end 8 ~tvw ← ⊕K k=1 ~cvwk ; / * ⊕ is concatenation , × is scalar vector multiplication * / 9 end / * SIF reweighed document vector embedding * / 10 for each document dn in D do 11 ~vdn ← 1|dn| ∑ w∈dn a a+p ( w ) ~tvw ; 12 end 13 Form a matrix X whose columns are { ~vdn : dn ∈ D } , and let ~u be the first singular vector ; 14 for each document dn ∈D do 15 ~vdn ← ~vdn - ~u~uT~vdn ; 16 end

Summary: Paper overview: The paper extends the method proposed by Arora 2017 for sentence embeddings to longer document embeddings. The main idea is that, averaging word embedding vectors mixes all the different topics on the document, and therefore is not expressive enough. Instead they propose to estimate the topic of each word (using dictionary learning) through the $\alpha$ weights (see page 4).These weights give "how much" this word belongs to a certain topic. For every topic we compute the $\alpha$-weighted vector of the word and  concatenate them (see word topic vector formation). Finally, we apply SIF (Arora 2017) using these word embeddings on all the document.