Word embedding re-examined: is the symmetrical factorization optimal?

1 INTRODUCTION . Word embedding is a fundamental task in natural language processing . Many different approaches have been proposed , including LSA ( Deerwester et al. , 1990 ) , SGNS ( Mikolov et al. , 2013a ; b ) , GloVe ( Pennington et al. , 2014 ) and others . These methods have achieved huge success in information retrieval ( Salton & Buckley , 1988 ) , entity recognition ( Lample et al. , 2016 ) , sentiment analysis ( Socher et al. , 2013 ) , machine translation ( Sutskever et al. , 2014 ) and so on . Previous studies have demonstrated three well-known facts about the word embedding : ( 1 ) Levy & Goldberg ( 2014b ) showes many different kinds of word embeddings ( e.g . SGNS ) can be understood as the matrix factorization framework . For example , Levy & Goldberg ( 2014b ) proves that SGNS is implicitly factorizing the shifted pointwise mutual information ( PMI ) matrix . ( 2 ) Mikolov et al . ( 2013b ) ; Pennington et al . ( 2014 ) ; Levy et al . ( 2015 ) ; Tian et al . ( 2016 ) and other works observe that word embedding exhibits two nice properties : the word similarity such that words with similar semantic meanings are embedded closely in the embedding space , and the analogy structure such that “ woman is to queen as man is to king ” . ( 3 ) Previous studies , including ( Caron , 2001 ; Turney , 2012 ; Bullinaria & Levy , 2012 ; Levy et al. , 2015 ; Artetxe et al. , 2018 ) , empirically observe that α in word embedding E = Ud ·Σαd 1 has an important influence on the quality of word embedding . But these works did not provide theoretical explanation for this behavior of α or practical guide to set this parameter . In this paper , we show the word embedding can be understood as a low rank transformation process . Furthermore , we investigate the change of the relative distances between words which is the inner mechanism that leads to these two nice properties in the word embedding . Particularly , we find the parameter α has influence on the change of relative distance between words . Based on such analysis , we provide theoretical explanation the behavior of α and derive a method to automatically find its optimal value . To summarize , this research is motivated by two questions : 1 . What is the inner mechanism resulting in these two good properties of word embedding ? 2 . How does the parameter α influence the word embedding ? Can we find a method to determine its optimal value ? 1We will explain the meaning of these variables ( e.g. , Ud ) in the following sections . We make the following contributions in this paper : • We reveal how the relative distances between words change during the embedding process which can be seen as a low rank transformation process . The nice properties of word embeddings are inherited from the original word-context co-occurrence matrix as the result of such low rank transformation . • Compared with previous studies which empirically observed the influence of the parameter α on the word embedding , we theoretically explain the influence of parameter α on the word embedding by revealing it controls the change of relative distances between words . Besides , we derive a method to automatically find its optimal value . We conduct experiments on real datasets to verify our ideas . The following sections are organized as follows : section 2 introduces the background knowledge and necessary preliminaries ; section 3 shows how the relative distances between words change during this linear transformation and it is the inner reason incurring these nice properties of word embedding . Based on such analysis , section 4 shows that symmetric factorization can be suboptimal and how to improve the word embedding model with asymmetric factorization . Section 5 contains the experiments on real datasets and section 6 discusses the related work . Finally , section ? ? is the conclusion and describes a potential work . 2 BACKGROUND AND PRELIMINARIES . The goal of word embedding is to find a good vector representation for every word in a corpus . As summarized in Yin & Shen ( 2018 ) , most existing word embedding methods can be formulated as low rank matrix approximations . So we can analyze the word embedding algorithms ( e.g. , SGNS ) within the matrix factorization framework . Suppose M ∈ Rn×n is the word-context matrix where Mi , j represents some statistics between word wi and its context word wj . These word embedding algorithms are seeking a d-dimension word embedding matrix E ∈ Rn×d accompanied with a context embedding matrix C ∈ Rn×d to approximate the word-context matrix M , such that M ≈ ECT . The matrix M can have different forms due to the design of objective function in every embedding method . For example , M is the shifted pointwise mutual information matrix in the SGNS . Explicitly performing singular value decomposition . This kind of methods can be summarized as follows . Firstly , a word-context matrix M is constructed to capture some co-occurrence statistics of every word and its context . For example , M can be the pointwise mutual information ( PMI ) matrix ( Church & Hanks , 1990 ) , positive PMI ( PPMI ) matrix ( Bullinaria & Levy , 2012 ) and Shifted PPMI ( SPPMI ) matrix ( Levy & Goldberg , 2014b ) . Then , SVD is used to factorize the word-context matrix M as M = U ·Σ ·V T . Finally , only the top d singular values in Σ and the corresponding columns in U and V are kept , producing Σd , Ud and Vd respectively . The word embedding E ∈ Rn×d is obtained by E = Ud or E = Ud ·Σd . Caron ( 2001 ) , Turney ( 2012 ) , Bullinaria & Levy ( 2012 ) , Levy et al . ( 2015 ) and Artetxe et al . ( 2018 ) discussed a more general approach which adds a parameter α to the truncated diagonal matrix Σd and obtain the word embedding E = Ud · Σαd . They observe α has an important influence on the embedding and suggest this parameter should be tuned . However , the reason is not theoretically clear and no clear method has been provided to tune α yet . Implicitly performing matrix factorization . Recently , word embedding methods based on neural networks have been proposed ( e.g. , Bengio et al . ( 2003 ) ) . Particularly , Mikolov et al . ( 2013a ; b ) proposed a popular word embedding approach SGNS based on the skip-gram with negative-sampling and it achieves the state-of-the-art results in different tasks . Besides , GloVe is another widely used word embedding method ( Pennington et al. , 2014 ) . While SGNS and GloVe learn word embeddings by optimizing some objective functions using stochastic gradient methods , it has been shown that these two methods are implicitly performing matrix factorizations . Specifically , Levy & Goldberg ( 2014b ) showed SGNS is implicitly factorizing the shifted Pointwise Mutual Information matrix . Levy et al . ( 2015 ) showed the GloVe is implicitly factorizing the log-count word-context co-occurrence matrix . So these two neural network based embedding methods can also be formu- lated within the matrix factorization framework , and the parameter α equals to 0.5 as they are doing symmetric factorization of some word-context co-occurrence matrix M . To summarize , same as the first assumption in Yin & Shen ( 2018 ) , our analysis assumes the word embeddings can be formulated as low rank matrix approximations , either explicitly or implicitly . Specifically , the embedding matrix E equals to Ud ·Σαd where Ud , Σd and α are as defined before . 3 ANALYSIS : THE REASON WHY THE EMBEDDING EXHIBITS NICE PROPERTIES . Our start point is the relation between word embedding and matrix factorization . As mentioned in section 2 , almost all word embedding methods can be formulated as explicit or implicit matrix factorization ( e.g. , SGNS is implicitly doing a symmetric factorization on the shifted pointwise mutual information matrix ) . We adopt the same notation in previous papers ( Levy et al. , 2015 ; Yin & Shen , 2018 ) to write M = UΣV T as the SVD of some word-context co-occurrence matrix M . For example , M can be the pointwise mutual information matrix . Besides , Σd is the truncated diagonal matrix containing top d singular values . Ud and Vd are the corresponding truncated U and truncated V respectively . The d-dimension word embedding matrix E is obtained by multiplying Ud with a power of Σd , E = Ud ·Σαd ( 1 ) 3.1 WORD EMBEDDING AS LOW RANK TRANSFORMATION . The embedding E in equation 1 is indeed a low rank transformation from the original word-context co-occurrence space M . Let us define another diagonal matrix Σpse = diag ( σ1 , . . . , σd , 0 , . . . , 0 ) whose first d elements are the singular values in Σ and the remaining diagonal elements are zeros . We have the following observation . Observation 1 . E is the first d columns of Epse Epse = M · V ·Σ ( α−1 ) pse ( 2 ) The proof is in Appendix A.1 .. Observation 1 reveals the process to compute E : ( 1 ) first multiplying the matrix V to the right side of M ; ( 2 ) then multiplying the diagonal matrix Σ ( 1−α ) pse on the intermediate result MV ; ( 3 ) finally removing the last n− d columns of M ·V ·Σ ( α−1 ) pse . The first step in this transformation process is a rotation because the matrix V is a unitary matrix . So the relative distances between words are not changed during the first transformation . In the second step , the matrix Σpse is a diagonal matrix and it is multiplied on the right side of MV . So the second step scales every dimension of the rotated matrix . Because we keep the first d dimensions in the final step , we only care about the scaling in the first d dimensions which are determined by the corresponding d singular values with a power α . By explictly expressing the process which transforms the word-context co-occurrence matrix M to the embedding matrix E , we can find a fact that the relative distances between words only changes within the scaling step . Considering vectors of all words are scaled the same at every dimension , it seems that the relative distances among words should be preserved to some extent . In fact , we have the following two theorems : Theorem 1 . For any two words wi and wj , if ‖Mi , : −Mj , :‖2 ≤ δ ( δ ≥ 0 ) , then ‖Ei , : −Ej , :‖2 ≤ σ ( α−1 ) d δ. Theorem 2 . For any two words wi and wj , if ‖Ei , : −Ej , :‖2 ≤ δ ( δ ≥ 0 ) , then ‖Mi , : −Mj , :‖2 ≤ σ ( 1−α ) 1 δ + 2 ( ∑n k=d+1 σ 2 k ) 1/2 . The formal proof of theorem 1 and theorem 2 is in appendix A.2 and A.3 . These two theorems describe how the relative distances ( measured in Euclidean distance ) among words change during the embedding transformation . Particularly , theorem 1 shows that if the distance between two words wi and wj in M space is no more than δ , their distance in the embedding space is also within a constraint . This bound is related with the singular value σd and the parameter α . So theorem 1 implies that the relative distances between words will not change dramatically during this transformation from M to E with suitable α. Theorem 2 demonstrates that if we observe two words are embedded closely in E ( ‖Ei , : −Ej , :‖2 is small ) , we can infer these two words are also close in the original M space . To sum up , the above two theorems indicate the neighborhood structure ( relative distances to other words ) for every word will be preserve to some extent during the transformation from M to E. In other words , matrix E inherits the neighborhood structure that exists in M . As the result , E will inherit the properties in M that are related with the relative distances among words . Now we can answer the question raised in the introduction : What is the inner mechanism resulting in the good properties of word embedding ? In fact , we will show that these properties are the result of inheritance of neighborhood structure .

Summary: This paper provides a closer look at the well-studied problem of learning word embeddings. In particular, it looks at the set of embedding methods that explicitly or implicitly perform a matrix factorization and tries to understand why the word embeddings exhibit analogy structure and why words that are semantically similar get embedded close together. The mechanism it comes up with has to do with the alpha parameter that represents the powers of singular values of the matrix that was factorized to estimate the embeddings. It turns out that alpha controls the distance between the words in the embedding transformation process. Next the paper discusses how to choose/estimate alpha to get better quality embeddings. Results are shown on several word similarity tasks.