An Empirical Study on Post-processing Methods for Word Embeddings

1 INTRODUCTION . Distributed representations of words have been widely dominating the Natural Language Processing research area and other related research areas where discrete tokens in text are part of the learning systems . Fast learning algorithms are being proposed , criticised , and improved . Despite the fact that there exist various learning algorithms ( Mikolov et al. , 2013a ; Pennington et al. , 2014 ; Bojanowski et al. , 2017 ) for producing high-quality distributed representations of words , the main objective is roughly the same , which is drawn from the Distributional Hypothesis ( Harris , 1954 ) . The algorithms assume that there is a smooth transition of meaning at the word-level , thus they learn to assign higher similarity for adjacent word vectors than those that are not adjacent . The overwhelming success of distributed word vectors leads to subsequent questions and analyses on the information encoded in the learnt space . As the learning algorithms directly and only utilise the co-occurrence provided by large corpora , it is easy to hypothesise that the learnt vectors are correlated with the frequency of each word , which may not be relevant to the meaning of a word ( Turney & Pantel , 2010 ) and might hurt the expressiveness of the learnt vectors . One can theorise the frequency-related components in the learnt vector space , and remove them ( Arora et al. , 2017 ; Mu et al. , 2018 ; Ethayarajh , 2018 ) . These post-processing methods , whilst very effective and appealing , are derived from heavy assumptions on the representation geometry and the similarity measure . In this work we re-examine the problem of post-processing word vectors as a shrinkage estimation of the true/underlying oracle gram matrix of words , which is a rank-deficient matrix due to the existence of synonyms and antonyms . Constrained from the semi-Riemannian manifold ( Benn & Tucker , 1987 ; Abraham et al. , 1983 ) where positive semi-definite matrices , including gram matrices , exist , and Centralised Kernel Alignment ( Cortes et al. , 2012 ) , we are able to define an objective to search for the optimal shrinkage parameter , also called mixing parameter , that is used to mix the estimated gram matrix and a predefined target matrix with the maximum similarity with the oracle matrix on the semi-Riemannian manifold . Our contribution can be considered as follows : 1 . We define the post-processing on word vectors as a shrinkage estimation , in which the estimated gram matrix is calculated from the pretrained word embeddings , and the target matrix is a scaled identity matrix for smoothing the spectrum of the estimated one . The goal is to find the optimal mixing parameters to combine the estimated gram matrix with the target in the semi-Riemannian manifold that maximises the similarity between the combined matrix and the oracle one . 2 . We choose to work with the CKA , as it is invariant to isotropic scaling and also rotation , which is a desired property as the angular distance ( cosine similarity ) between two word vectors is commonly used in downstream tasks for evaluating the quality of learnt vectors . Instead of directly deriving formulae from the CKA ( Eq . 1 ) , we start with the logarithm of CKA ( Eq . 2 ) , and then find an informative lower bound ( Eq . 5 ) . The remaining analysis is done on the lower bound . 3 . A useful objective is derived to search for the optimal mixing parameter , and the performance on the evaluation tasks shows the effectiveness of our proposed method . Compared with two other methods , our approach shows that the shrinkage estimation on a sub-Riemannian manifold works better than that on an Euclidean manifold for word embedding matrices , and it provides an alternative way to smooth the spectrum by adjusting the power of each linearly independent component instead of removing top ones . 2 RELATED WORK . Several post-processing methods have been proposed for readjusting the learnt word vectors ( Mu et al. , 2018 ; Arora et al. , 2017 ; Ethayarajh , 2018 ) to make the vector cluster more isotropic w.r.t . the origin such that cosine similarity or dot-product can better express the relationship between word pairs . A common practice is to remove top principal components of the learnt word embedding matrix as it has been shown that those directions are highly correlated with the occurrence . Despite the effectiveness and efficiency of this family of methods , the number of directions to remove and also the reason to remove are rather derived from empirical observations , and they don ’ t provide a principled way to estimate these two factors . In our case , we derive a useful objective that one could take to search for the optimal mixing parameter on their own pretrained word vectors . As described above , we view the post-processing method as a shrinkage estimation on the semiRiemannian manifold . Recent advances in machine learning research have developed various ways to learn word vectors in hyperbolic space ( Nickel & Kiela , 2017 ; Dhingra et al. , 2018 ; Gulcehre et al. , 2019 ) . Although the Riemannian manifold is a specific type of hyperbolic geometry , our data samples on the manifold are gram matrices of words instead of word vectors themselves in previous work , , which makes our work different from previous ones . The word vectors themselves are still considered in Euclidean space for the simple computation of similarity between pairs , and the intensity of the shrinkage estimation is optimised to adjust the word vectors . Our work is also related to previously proposed shrinkage estimation of covariance matrices ( Ledoit & Wolf , 2004 ; Schäfer & Strimmer , 2005 ) . These methods are practically useful when the number of variables to be estimated is much more than that of the samples available . However , the mixing operation between the estimated covariance matrix and the target matrix is a weighted sum , and l2 distance is used to measure the difference between the true/underlying covariance matrix and the mixed covariance matrix . As known , covariance matrices are positive semi-definite , thus they lie on a semi-Riemannian manifold with a manifold-observing distance measure . A reasonable choice is to conduct the shrinkage estimation on the semi-Riemannian manifold , and CKA is applied to measure the distance as rotations and istropic scaling should be ignored with measuring two sets of word vectors . The following sections will discuss our approach in details . 3 SHRINKAGE OF GRAM MATRIX . Learning word vectors can be viewed as estimating the oracle gram matrix where the relationship of words is expressed , and the post-processing method we propose here aims to best recover the oracle gram matrix . Suppose there exists an ideal gram matrix K ∈ Rn×n where each entry represents the similarity of a word pair defined by a kernel function k ( wi , wj ) . Given the assumption of a gram matrix and the existence of polysemies , the oracle gram matrix K is positive semi-definite and its rank is between 0 and n and denoted as k. A set of pretrained word vectors is provided by a previously proposed algorithm , and for simplicity , an embedding matrix E ∈ Rn×d is constructed in which each row is a word vector vi . The estimated gram matrix K′ = EE > ∈ Rn×n has rank d and d ≤ k. It means that overall , the previously proposed algorithms give us a low-rank approximation of the oracle gram matrix K. The goal of shrinkage is to , after the estimated gram matrix K′ is available , find an optimal postprocessing method to maximise the similarity between the oracle gram matrix K and the estimated one K′ given simple assumptions about the ideal one without directly finding it . Therefore , a proper similarity measure that inherits and respects our assumptions is crucial . 3.1 CENTRALISED KERNEL ALIGNMENT ( CKA ) . The similarity measure between two sets of word vectors should be invariant to any rotations and also to isotropic scaling . A reasonable choice is the Centralised Kernel Alignment ( CKA ) ( Cortes et al. , 2012 ) defined below : ρ ( Kc , K ′ c ) = 〈Kc , K′c〉F ||Kc||F ||K′c||F ( 1 ) where Kc = [ I − 1 n 11 > ] K [ I − 1 n 11 > ] , and 〈Kc , K′c〉F = Tr ( KcK′c ) . For simplicity of the derivations below , we assume that the ideal gram matrix K is centralised , and the estimated one K′ can be centralised easily by removing the mean of word vectors . In the following section , K and K′ are used to denote centralised gram matrices . As shown in Eq . 1 , it is clear that ρ ( K , K′ ) is invariant to any rotations and isotropic scaling , and doesn ’ t suffer from the issues of K and K′ being low-rank . The centralised kernel alignment has been recommended recently as a proper measure for comparing the similarity of features learnt by neural networks ( Kornblith et al. , 2019 ) . As learning word vectors can be thought of as a one-layer neural network with linear activation function , ρ ( K , K′ ) is a reasonable similarity measure between the ideal gram matrix and our shrunken one provided by learning algorithms . Given the CauchySchwartz inequality and also the non-negativity of the Frobenius norm , ρ ( K , K′ ) ∈ [ 0 , 1 ] . Since both K and K′ are positive semi-definite , their eigenvalues are denoted as { λσ1 , λσ2 , ... , λσk } and { λν1 , λν2 , ... , λνd } respectively , and the eigenvalues of KK ′ are denoted as { λ1 , λ2 , ... , λd } as the rank is determined by the matrix with lower rank.1 The log-transformation is conducted to simplify the derivation . log ρ ( K , K′ ) = logTr ( KK′ ) − 1 2 log Tr ( KK > ) − 1 2 log Tr ( K′K′ > ) ≥ 1 d log ( det ( K ) det ( K′ ) ) − 1 2 log Tr ( KK > ) − 1 2 log Tr ( K′K′ > ) = 1 d ∑k i=1 log λσi + 1 d ∑d i=1 log λνi − 1 2 log ∑k i=1 λ 2 σi − 1 2 log ∑d i=1 λ 2 νi ( 2 ) where the lower bound is given by the AM−GM inequality , and the equality holds when all eigenvalues of KK ′ are the same λ1 = λ2 = ... = λd . 3.2 SHRINKAGE OF GRAM MATRIX ON SEMI-RIEMANNIAN MANIFOLD . As the goal is to find a post-processing method that maximises the similarity ρ ( K , K′ ) , a widely adopted approach is to shrink the estimated gram matrix K′ towards the target matrix T with a predefined structure . The target matrix T is usually positive semi-definite and has same or higher rank than the estimated gram matrix . In our case , we assume T is full rank as it simplifies equations . Previous methods rely on a linear combination of K′ and T in Euclidean space ( Ledoit & Wolf , 2004 ; Schäfer & Strimmer , 2005 ) . However , as gram matrices are positive semi-definite , they naturally lie on the semi-Riemannian manifold . Therefore , a suitable option for shrinkage is to move the estimated gram matrix K′ towards the target matrix T on a semi-Riemannian manifold ( Brenier , 1987 ) , and the resulting matrix Y is given by Y = T 1/2 ( T−1/2K′T−1/2 ) βT 1/2 ( 3 ) where β ∈ [ 0 , 1 ] is the mixing parameter that indicates the strength of the shrinkage , and T 1/2 is the square root of the matrix T . The objective is to find the optimal β that maximises log ρ ( K , K′ ) . β ? = argmaxβ log ρ ( K , Y ) = argmaxβ log ρ ( K , T 1/2 ( T−1/2K′T−1/2 ) βT 1/2 ) ( 4 ) The objective defined in Eq . 4 is hard to work with , instead , we maximise the lower bound in Eq . 2 to find optimal mixing parameter β ? . By plugging Y into Eq . 2 and denoting λyi as the i-th eigenvalue of Y , we have : log ρ ( K , Y ) ≥ 1 d ∑k i=1 log λσi + 1 d ∑d i=1 log λyi − 1 2 log ∑k i=1 λ 2 σi − 1 2 log ∑d i=1 λ 2 yi ( 5 ) 1λσk = λνd = 0 due to the centralisation .

Summary: The authors propose a novel word embedding post-processing method that maximizes the similarity between the estimated Gram matrix of word vectors and its oracle matrix. To find the optimal Gram matrix, they adopt the shrink method to make Gram matrix K' to target matrix T on semi-Riemannian space. The authors use the shrinkage method to find optimal K' and formulate the maximization on CKA problem as finding the optimal shrinkage parameter that maximizes the lower bound of the CKA between the estimated Gram matrix and oracle. By applying the proposed method to various word embedding methods, the authors show the performance of their post-processing method on word analogy/similarity task, word translation task, and sentence similarity task.