Understanding Composition of Word Embeddings via Tensor Decomposition

1 INTRODUCTION . Word embeddings have become one of the most popular techniques in natural language processing . A word embedding maps each word in the vocabulary to a low dimensional vector . Several algorithms ( e.g. , Mikolov et al . ( 2013 ) ; Pennington et al . ( 2014 ) ) can produce word embedding vectors whose distances or inner-products capture semantic relationships between words . The vector representations are useful for solving many NLP tasks , such as analogy tasks ( Mikolov et al. , 2013 ) or serving as features for supervised learning problems ( Maas et al. , 2011 ) . While word embeddings are good at capturing the semantic information of a single word , a key challenge is the problem of composition : how to combine the embeddings of two co-occurring , syntactically related words to an embedding of the entire phrase . In practice composition is often done by simply adding the embeddings of the two words , but this may not be appropriate when the combined meaning of the two words differ significantly from the meaning of individual words ( e.g. , “ complex number ” should not just be “ complex ” + “ number ” ) . In this paper , we try to learn a model for word embeddings that incorporates syntactic information and naturally leads to better compositions for syntactically related word pairs . Our model is motivated by the principled approach for understanding word embeddings initiated by Arora et al . ( 2015 ) , and models for composition similar to Coecke et al . ( 2010 ) . Arora et al . ( 2015 ) gave a generative model ( RAND-WALK ) for word embeddings , and showed several previous algorithms can be interpreted as finding the hidden parameters of this model . However , the RAND-WALK model does not treat syntactically related word-pairs differently from other word pairs . We give a generative model called syntactic RAND-WALK ( see Section 3 ) that is capable of capturing specific syntactic relations ( e.g. , adjective-noun or verb-object pairs ) . Taking adjective-noun pairs as an example , previous works ( Socher et al. , 2012 ; Baroni & Zamparelli , 2010 ; Maillard & Clark , 2015 ) have tried to model the adjective as a linear operator ( a matrix ) that can act on the embedding of the noun . However , this would require learning a d × d matrix for each adjective while the normal embedding only has dimension d. In our model , we use a core tensor T ∈ Rd×d×d to capture the relations between a pair of words and its context . In particular , using the tensor T and the word embedding for the adjective , it is possible to define a matrix for the adjective that can be used as an operator on the embedding of the noun . Therefore our model allows the same interpretations as many previous models while having much fewer parameters to train . One salient feature of our model is that it makes good use of high order statistics . Standard word embeddings are based on the observation that the semantic information of a word can be captured by words that appear close to it . Hence most algorithms use pairwise co-occurrence between words to learn the embeddings . However , for the composition problem , the phrase of interest already has two words , so it would be natural to consider co-occurrences between at least three words ( the two words in the phrase and their neighbors ) . Based on the model , we can prove an elegant relationship between high order co-occurrences of words and the model parameters . In particular , we show that if we measure the Pointwise Mutual Information ( PMI ) between three words , and form an n × n × n tensor that is indexed by three words a , b , w , then the tensor has a Tucker decomposition that exactly matches our core tensor T and the word embeddings ( see Section 2 , Theorem 1 , and Corollary 1 ) . This suggests a natural way of learning our model using a tensor decomposition algorithm . Our model also allows us to approach the composition problem with more theoretical insights . Based on our model , if words a , b have the particular syntactic relationships we are modeling , their composition will be a vector va + vb + T ( va , vb , · ) . Here va , vb are the embeddings for word a and b , and the tensor gives an additional correction term . By choosing different core tensors it is possible to recover many previous composition methods . We discuss this further in Section 3 . Finally , we train our new model on a large corpus and give experimental evaluations . In the experiments , we show that the model learned satisfies the new assumptions that we need . We also give both qualitative and quantitative results for the new embeddings . Our embeddings and the novel composition method can capture the specific meaning of adjective-noun phrases in a way that is impossible by simply “ adding ” the meaning of the individual words . Quantitative experiment also shows that our composition vector are better correlated with humans on a phrase similarity task . 1.1 RELATED WORK . Syntax and word embeddings Many well-known word embedding methods ( e.g. , Pennington et al . ( 2014 ) ; Mikolov et al . ( 2013 ) ) don ’ t explicitly utilize or model syntactic structure within text . Andreas & Klein ( 2014 ) find that such syntax-blind word embeddings fail to capture syntactic information above and beyond what a statistical parser can obtain , suggesting that more work is required to build syntax into word embeddings . Several syntax-aware embedding algorithms have been proposed to address this . Levy & Goldberg ( 2014a ) propose a syntax-oriented variant of the well-known skip-gram algorithm of Mikolov et al . ( 2013 ) , using contexts generated from syntactic dependency-based contexts obtained with a parser . Cheng & Kartsaklis ( 2015 ) build syntax-awareness into a neural network model for word embeddings by indroducing a negative set of samples in which the order of the context words is shuffled , in hopes that the syntactic elements which are sensitive to word order will be captured . Word embedding composition Several works have addressed the problem of composition for word embeddings . On the theoretical side , Gittens et al . ( 2017 ) give a theoretical justification for additive embedding composition in word models that satisfy certain assumptions , such as the skipgram model , but these assumptions don ’ t address syntax explicitly . Coecke et al . ( 2010 ) present a mathematical framework for reasoning about syntax-aware word embedding composition that motivated our syntactic RAND-WALK model . Our new contribution is a concrete and practical learning algorithm with theoretical guarantees . Mitchell & Lapata ( 2008 ; 2010 ) explore various composition methods that involve both additive and multiplicative interactions between the component embeddings , but some of these are limited by the need to learn additional parameters post-hoc in a supervised fashion . Guevara ( 2010 ) get around this drawback by first training word embeddings for each word and also for tokenized adjective-noun pairs . Then , the composition model is trained by using the constituent adjective and noun embeddings as input and the adjective-noun token embedding as the predictive target . Maillard & Clark ( 2015 ) treat adjectives as matrices and nouns as vectors , so that the composition of an adjective and noun is just matrix-vector multiplication . The matrices and vectors are learned through an extension of the skip-gram model with negative sampling . In contrast to these approaches , our model gives rise to a syntax-aware composition function , which can be learned along with the word embeddings in an unsupervised fashion , and which generalizes many previous composition methods ( see Section 3.3 for more discussion ) . Tensor factorization for word embeddings As Levy & Goldberg ( 2014b ) and Li et al . ( 2015 ) point out , some popular word embedding methods are closely connected matrix factorization problems involving pointwise mutual information ( PMI ) and word-word co-occurrences . It is natural to consider generalizing this basic approach to tensor decomposition . Sharan & Valiant ( 2017 ) demonstrate this technique by performing a CP decomposition on triple word co-occurrence counts . Bailey & Aeron ( 2017 ) explore this idea further by defining a third-order generalization of PMI , and then performing a symmetric CP decomposition on the resulting tensor . In contrast to these recent works , our approach arives naturally at the more general Tucker decomposition due to the syntactic structure in our model . Our model also suggests a different ( yet still common ) definition of third-order PMI . 2 PRELIMINARIES . Notation For a vector v , we use ‖v‖ to denote its Euclidean norm . For vectors u , v we use 〈u , v〉 to denote their inner-product . For a matrix M , we use ‖M‖ to denote its spectral norm , ‖M‖F = √∑ i , jM 2 i , j to denote its Frobenius norm , and Mi , : to denote it ’ s i-th row . In this paper , we will also often deal with 3rd order tensors , which are just three-way indexed arrays . We use ⊗ to denote the tensor product : if u , v , w ∈ Rd are d-dimensional vectors , T = u⊗ v ⊗ w is a d× d× d tensor whose entries are Ti , j , k = uivjwk . Tensor basics Just as matrices are often viewed as bilinear functions , third order tensors can be interpreted as trilinear functions over three vectors . Concretely , let T be a d× d× d tensor , and let x , y , z ∈ Rd . We define the scalar T ( x , y , z ) ∈ R as follows T ( x , y , z ) = d∑ i , j , k=1 Ti , j , kx ( i ) y ( j ) z ( k ) . This operation is linear in x , y and z. Analogous to applying a matrix M to a vector v ( with the result vector Mv ) , we can also apply a tensor T to one or two vectors , resulting in a matrix and a vector , respectively : T ( x , y , · ) ( k ) = d∑ i , j=1 Ti , j , kx ( i ) y ( j ) , T ( x , · , · ) j , k = d∑ i=1 Ti , j , kx ( i ) We will make use of the simple facts that 〈z , T ( x , y , · ) 〉 = T ( x , y , z ) and [ T ( x , · , · ) ] > y = T ( x , y , · ) . Tensor decompositions Unlike matrices , there are several different definitions for the rank of a tensor . In this paper we mostly use the notion of Tucker rank Tucker ( 1966 ) . A tensor T ∈ Rn×n×n has Tucker rank d , if there exists a core tensor S ∈ Rd×d×d and matrices A , B , C ∈ Rn×d such that Ti , j , k = d∑ i′ , j′ , k′=1 Si′ , j′ , k′Ai , i′Bj , j′Ck , k′ = S ( Ai , : , Bj , : , Ck , : ) , The equation above is also called a Tucker decomposition of the tensor T . The Tucker decomposition for a tensor can be computed efficiently . When the core tensor S is restricted to a diagonal tensor ( only nonzero at entries Si , i , i ) , the decomposition is called a CP decomposition Carroll & Chang ( 1970 ) ; Harshman ( 1970 ) which can also be written as T = ∑d i=1 Si , i , iAi , : ⊗Bi , : ⊗Ci , : . In this case , the tensor T is the sum of d rank-1 tensors ( Ai , : ⊗ Bi , : ⊗ Ci , : ) . However , unlike matrix factorizations and the Tucker decomposition , the CP decomposition of a tensor is hard to compute in the general case ( Håstad , 1990 ; Hillar & Lim , 2013 ) . Later in Section 4 we will also see why our model for syntactic word embeddings naturally leads to a Tucker decomposition .

Summary: The authors consider the use of tensor approximations to more accurately capture syntactical aspects of compositionality for word embeddings. Given two words a and b, when your goal is to find a word whose meaning is roughly that of the phrase (a,b), a standard approach to to find the word whose embedding is close to the sum of the embeddings, a + b. The authors point out that others have observed that this form of compositionality does not leverage any information on the syntax of the pair (a,b), and the propose using a tensor contraction to model an additional multiplicative interaction between a and b, so they propose finding the word whose embedding is closest to a + b + T*a*b, where T is a tensor, and T*a*b denotes the vector obtained by contracting a and b with T. They test this idea specifically on the use-case where (a,b) is an adjective,noun pair, and show that their form of compositionality outperforms weighted versions of additive compositionality in terms of spearman and pearson correlation with human judgements. In their model, the word embeddings are learned separately, then the tensor T is learned by minimizing an objective whose goal is to minimize the error in predicting observed trigram statistics. The specific objective comes from a nontrivial tensorial extension of the original matricial RAND-WALK model for learning word embeddings.