Understanding Composition of Word Embeddings via Tensor Decomposition

1 INTRODUCTION . Word embeddings have become one of the most popular techniques in natural language processing . A word embedding maps each word in the vocabulary to a low dimensional vector . Several algorithms ( e.g. , Mikolov et al . ( 2013 ) ; Pennington et al . ( 2014 ) ) can produce word embedding vectors whose distances or inner-products capture semantic relationships between words . The vector representations are useful for solving many NLP tasks , such as analogy tasks ( Mikolov et al. , 2013 ) or serving as features for supervised learning problems ( Maas et al. , 2011 ) . While word embeddings are good at capturing the semantic information of a single word , a key challenge is the problem of composition : how to combine the embeddings of two co-occurring , syntactically related words to an embedding of the entire phrase . In practice composition is often done by simply adding the embeddings of the two words , but this may not be appropriate when the combined meaning of the two words differ significantly from the meaning of individual words ( e.g. , “ complex number ” should not just be “ complex ” + “ number ” ) . In this paper , we try to learn a model for word embeddings that incorporates syntactic information and naturally leads to better compositions for syntactically related word pairs . Our model is motivated by the principled approach for understanding word embeddings initiated by Arora et al . ( 2015 ) , and models for composition similar to Coecke et al . ( 2010 ) . Arora et al . ( 2015 ) gave a generative model ( RAND-WALK ) for word embeddings , and showed several previous algorithms can be interpreted as finding the hidden parameters of this model . However , the RAND-WALK model does not treat syntactically related word-pairs differently from other word pairs . We give a generative model called syntactic RAND-WALK ( see Section 3 ) that is capable of capturing specific syntactic relations ( e.g. , adjective-noun or verb-object pairs ) . Taking adjective-noun pairs as an example , previous works ( Socher et al. , 2012 ; Baroni & Zamparelli , 2010 ; Maillard & Clark , 2015 ) have tried to model the adjective as a linear operator ( a matrix ) that can act on the embedding of the noun . However , this would require learning a d × d matrix for each adjective while the normal embedding only has dimension d. In our model , we use a core tensor T ∈ Rd×d×d to capture the relations between a pair of words and its context . In particular , using the tensor T and the word embedding for the adjective , it is possible to define a matrix for the adjective that can be used as an operator on the embedding of the noun . Therefore our model allows the same interpretations as many previous models while having much fewer parameters to train . One salient feature of our model is that it makes good use of high order statistics . Standard word embeddings are based on the observation that the semantic information of a word can be captured by words that appear close to it . Hence most algorithms use pairwise co-occurrence between words to learn the embeddings . However , for the composition problem , the phrase of interest already has two words , so it would be natural to consider co-occurrences between at least three words ( the two words in the phrase and their neighbors ) . Based on the model , we can prove an elegant relationship between high order co-occurrences of words and the model parameters . In particular , we show that if we measure the Pointwise Mutual Information ( PMI ) between three words , and form an n × n × n tensor that is indexed by three words a , b , w , then the tensor has a Tucker decomposition that exactly matches our core tensor T and the word embeddings ( see Section 2 , Theorem 1 , and Corollary 1 ) . This suggests a natural way of learning our model using a tensor decomposition algorithm . Our model also allows us to approach the composition problem with more theoretical insights . Based on our model , if words a , b have the particular syntactic relationships we are modeling , their composition will be a vector va + vb + T ( va , vb , · ) . Here va , vb are the embeddings for word a and b , and the tensor gives an additional correction term . By choosing different core tensors it is possible to recover many previous composition methods . We discuss this further in Section 3 . Finally , we train our new model on a large corpus and give experimental evaluations . In the experiments , we show that the model learned satisfies the new assumptions that we need . We also give both qualitative and quantitative results for the new embeddings . Our embeddings and the novel composition method can capture the specific meaning of adjective-noun phrases in a way that is impossible by simply “ adding ” the meaning of the individual words . Quantitative experiment also shows that our composition vector are better correlated with humans on a phrase similarity task . 1.1 RELATED WORK . Syntax and word embeddings Many well-known word embedding methods ( e.g. , Pennington et al . ( 2014 ) ; Mikolov et al . ( 2013 ) ) don ’ t explicitly utilize or model syntactic structure within text . Andreas & Klein ( 2014 ) find that such syntax-blind word embeddings fail to capture syntactic information above and beyond what a statistical parser can obtain , suggesting that more work is required to build syntax into word embeddings . Several syntax-aware embedding algorithms have been proposed to address this . Levy & Goldberg ( 2014a ) propose a syntax-oriented variant of the well-known skip-gram algorithm of Mikolov et al . ( 2013 ) , using contexts generated from syntactic dependency-based contexts obtained with a parser . Cheng & Kartsaklis ( 2015 ) build syntax-awareness into a neural network model for word embeddings by indroducing a negative set of samples in which the order of the context words is shuffled , in hopes that the syntactic elements which are sensitive to word order will be captured . Word embedding composition Several works have addressed the problem of composition for word embeddings . On the theoretical side , Gittens et al . ( 2017 ) give a theoretical justification for additive embedding composition in word models that satisfy certain assumptions , such as the skipgram model , but these assumptions don ’ t address syntax explicitly . Coecke et al . ( 2010 ) present a mathematical framework for reasoning about syntax-aware word embedding composition that motivated our syntactic RAND-WALK model . Our new contribution is a concrete and practical learning algorithm with theoretical guarantees . Mitchell & Lapata ( 2008 ; 2010 ) explore various composition methods that involve both additive and multiplicative interactions between the component embeddings , but some of these are limited by the need to learn additional parameters post-hoc in a supervised fashion . Guevara ( 2010 ) get around this drawback by first training word embeddings for each word and also for tokenized adjective-noun pairs . Then , the composition model is trained by using the constituent adjective and noun embeddings as input and the adjective-noun token embedding as the predictive target . Maillard & Clark ( 2015 ) treat adjectives as matrices and nouns as vectors , so that the composition of an adjective and noun is just matrix-vector multiplication . The matrices and vectors are learned through an extension of the skip-gram model with negative sampling . In contrast to these approaches , our model gives rise to a syntax-aware composition function , which can be learned along with the word embeddings in an unsupervised fashion , and which generalizes many previous composition methods ( see Section 3.3 for more discussion ) . Tensor factorization for word embeddings As Levy & Goldberg ( 2014b ) and Li et al . ( 2015 ) point out , some popular word embedding methods are closely connected matrix factorization problems involving pointwise mutual information ( PMI ) and word-word co-occurrences . It is natural to consider generalizing this basic approach to tensor decomposition . Sharan & Valiant ( 2017 ) demonstrate this technique by performing a CP decomposition on triple word co-occurrence counts . Bailey & Aeron ( 2017 ) explore this idea further by defining a third-order generalization of PMI , and then performing a symmetric CP decomposition on the resulting tensor . In contrast to these recent works , our approach arives naturally at the more general Tucker decomposition due to the syntactic structure in our model . Our model also suggests a different ( yet still common ) definition of third-order PMI . 2 PRELIMINARIES . Notation For a vector v , we use ‖v‖ to denote its Euclidean norm . For vectors u , v we use 〈u , v〉 to denote their inner-product . For a matrix M , we use ‖M‖ to denote its spectral norm , ‖M‖F = √∑ i , jM 2 i , j to denote its Frobenius norm , and Mi , : to denote it ’ s i-th row . In this paper , we will also often deal with 3rd order tensors , which are just three-way indexed arrays . We use ⊗ to denote the tensor product : if u , v , w ∈ Rd are d-dimensional vectors , T = u⊗ v ⊗ w is a d× d× d tensor whose entries are Ti , j , k = uivjwk . Tensor basics Just as matrices are often viewed as bilinear functions , third order tensors can be interpreted as trilinear functions over three vectors . Concretely , let T be a d× d× d tensor , and let x , y , z ∈ Rd . We define the scalar T ( x , y , z ) ∈ R as follows T ( x , y , z ) = d∑ i , j , k=1 Ti , j , kx ( i ) y ( j ) z ( k ) . This operation is linear in x , y and z. Analogous to applying a matrix M to a vector v ( with the result vector Mv ) , we can also apply a tensor T to one or two vectors , resulting in a matrix and a vector , respectively : T ( x , y , · ) ( k ) = d∑ i , j=1 Ti , j , kx ( i ) y ( j ) , T ( x , · , · ) j , k = d∑ i=1 Ti , j , kx ( i ) We will make use of the simple facts that 〈z , T ( x , y , · ) 〉 = T ( x , y , z ) and [ T ( x , · , · ) ] > y = T ( x , y , · ) . Tensor decompositions Unlike matrices , there are several different definitions for the rank of a tensor . In this paper we mostly use the notion of Tucker rank Tucker ( 1966 ) . A tensor T ∈ Rn×n×n has Tucker rank d , if there exists a core tensor S ∈ Rd×d×d and matrices A , B , C ∈ Rn×d such that Ti , j , k = d∑ i′ , j′ , k′=1 Si′ , j′ , k′Ai , i′Bj , j′Ck , k′ = S ( Ai , : , Bj , : , Ck , : ) , The equation above is also called a Tucker decomposition of the tensor T . The Tucker decomposition for a tensor can be computed efficiently . When the core tensor S is restricted to a diagonal tensor ( only nonzero at entries Si , i , i ) , the decomposition is called a CP decomposition Carroll & Chang ( 1970 ) ; Harshman ( 1970 ) which can also be written as T = ∑d i=1 Si , i , iAi , : ⊗Bi , : ⊗Ci , : . In this case , the tensor T is the sum of d rank-1 tensors ( Ai , : ⊗ Bi , : ⊗ Ci , : ) . However , unlike matrix factorizations and the Tucker decomposition , the CP decomposition of a tensor is hard to compute in the general case ( Håstad , 1990 ; Hillar & Lim , 2013 ) . Later in Section 4 we will also see why our model for syntactic word embeddings naturally leads to a Tucker decomposition .

Summary: The authors suggest a method to create combined low-dimensional representations for combinations of pairs of words which have a specific syntactic relationship (e.g. adjective - noun). Building on the generative word embedding model provided by Arora et al. (2015), their solution uses the core tensor from the Tucker decomposition of a 3-way PMI tensor to generate an additive term, used in the composition of two word embedding vectors.