An Empirical Study on Post-processing Methods for Word Embeddings
Authors: Shuai Tang
Mahta Mousavi
Virginia R. de Sa
Venue:  (2020)
Keywords: word vectors
post-processing method
centralised kernel alignment
shrinkage

Abstract: Word embeddings learnt from large corpora have been adopted in various applications in natural language processing and served as the general input representations to learning systems. Recently, a series of post-processing methods have been proposed to boost the performance of word embeddings on similarity comparison and analogy retrieval tasks, and some have been adapted to compose sentence representations. The general hypothesis behind these methods is that by enforcing the embedding space to be more isotropic, the similarity between words can be better expressed. We view these methods as an approach to shrink the covariance/gram matrix, which is estimated by learning word vectors, towards a scaled identity matrix. By optimising an objective in the semi-Riemannian manifold with Centralised Kernel Alignment (CKA), we are able to search for the optimal shrinkage parameter, and provide a post-processing method to smooth the spectrum of learnt word vectors which yields improved performance on downstream tasks.

Under review as a conference paper at ICLR 2020
AN EMPIRICAL STUDY ON POST-PROCESSING METHODS FOR WORD EMBEDDINGS
Anonymous authors Paper under double-blind review
ABSTRACT
Word embeddings learnt from large corpora have been adopted in various appli- cations in natural language processing and served as the general input representa- tions to learning systems. Recently, a series of post-processing methods have been proposed to boost the performance of word embeddings on similarity comparison and analogy retrieval tasks, and some have been adapted to compose sentence rep- resentations. The general hypothesis behind these methods is that by enforcing the embedding space to be more isotropic, the similarity between words can be better expressed. We view these methods an approach to shrink the covariance/gran matrix, which is estimated by learning word vectors, towards a scaled identity matrix. By optimising an objective in the semi-Riemannian manifold with Cen- tralised Kernel Alignment (CKA), we are able to search for the optimal shrinkage parameter, and provide a post-processing method to smooth the spectrum of learnt word vectors which yields improved performance on downstream tasks.
INTRODUCTION
Distributed representations of words have been widely dominating the Natural Language Processing research area and other related research areas where discrete tokens in text are part of the learning systems. Fast learning algorithms are being proposed, criticised, and improved. Despite the fact that there exist various learning algorithms (Mikolov et al., 2013a Pennington et al.) [014]] [Bojanowski al. 2017) for producing high-quality distributed representations of words, the main objective is roughly the same, which is drawn from the Distributional Hypothesis (Harris 1954). The algorithms assume that there is a smooth transition of meaning at the word-level, thus they learn assign higher similarity for adjacent word vectors than those that not adjacent.
The overwhelming success of distributed word vectors leads to subsequent questions and analyses on the information encoded in the learnt space. As the learning algorithms directly and only utilise the co-occurrence provided by large corpora, it is easy to hypothesise that the learnt vectors are correlated with the frequency of each word, which may not be relevant to the meaning of a word (Turney & Pantel 2010)) and might hurt the expressiveness of the learnt vectors. One can theorise the frequency-related components in the learnt vector space, and remove them (Arora et al. 2017 Mu et al. 2018 [Ethayarajh] 2018). These post-processing methods, whilst very effective and appealing, are derived from heavy assumptions on the representation geometry and the similarity measure.
In this work we re-examine the problem of post-processing word vectors as a shrinkage estimation of the true/underlying oracle gram matrix of words, which is rank-deficient matrix due to the existence of synonyms and antonyms. Constrained from the semi- Riemmiiimanninn manifold (Benn & Tucker 1987 Abraham et al. 1983) where positive meni-definite matrices, including gram matrices, exist, and Centralised Kernel Alignment (Cortes et al. 2012), we are able to define an objective to search for the optimal shrinkage parameter, also called mixing parameter, that used to mix the estimated gram matrix and predefined target matrix with the maximum similarity with the oracle matrix on the semi- Riemannian manifold. Our contribution can considered as follows:
1. We define the post-processing on word vectors shrinkage estimation, in which the estimated gram matrix is calculated from the pretrained word embeddings, and the target matrix is a scaled identity matrix for smoothing the spectrum of the estimated one. The goal is to find the optimal mixing parameters to combine the estimated gram matrix with the target in the remi-Riemannian manifold that maximises the similarity between the combined matrix and the oracle one.

---

Under review as a conference paper at ICLR 2020
2. We choose to work with the CKA, as it invariant to isotropic scaling and also rotation, which is a desired property as the angular distance (cosine similarity) between two word vectors is commonly used in downstream tasks for evaluating the quality of learnt vectors. Instead of directly deriving formulae from the CKA (Eq. [. we start with the logarithm of CKA (Eq. 2}. and then find an informative lower bound (Eq. 55. The remaining analysis is done the lower bound.
3. A useful objective is derived to search for the optimal mixing parameter, and the performance on the evaluation tasks shows the effectiveness of our proposed method. Compared with two other methods, our approach shows that the shrinkage estimation on a sub-Riemannian manifold works better than that on an Euclidean manifold for word embedding matrices, and provides an alter- native way to smooth the spectrum by adjusting the power of each linearly independent component instead of removing top ones.
2 RELATED WORK
Several post-processing methods have been proposed for readjusting the learnt word vectors (Mu jt al., 2018 Arora et al. 2017 Ethayarajh] 2018) to make the vector cluster more isotropic w.r.t. origin such that cosine similarity or dot-product can better express the relationship between word pairs. A common practice is to remove top principal components of the learnt word embedding matrix as it has been shown that those directions are highly correlated with the occurrence. Despite the effectiveness and efficiency of this family of methods, the number of directions to remove and also the reason to remove rather derived from empirical observations, and they don't 't provide principled way to estimate these two factors. In our case, we derive a useful objective that one could take to search for the optimal mixing parameter on their own pretrained word vectors.
As described above, we view the post-processing method as a shrinkage estimation on the semi- Riemannian manifold. Recent advances in machine learning research have developed various ways to learn word vectors in hyperbolic space (Nickel & Kiela 2017: Dhingra et al. 2018 Gulcehre specific type of hyperbolic geometry, our
al., 2019). Although the Riemannian manifold data samples on the manifold are gram matrices of words instead of word vectors themselves in previous work, which makes our work different from previous ones. The word vectors themselves are still considered in Euclidean space for the simple computation of similarity between pairs, and the intensity the shrinkage estimation is optimised to adjust the word vectors.
Our work is also related to previously proposed shrinkage estimation of covariance matrices (Ledoit & Wolf 2004; Schäfer & Strimmer 2005). These methods are practically useful when the number of variables to be estimated is much more than that of the samples available. However, the mixing operation between the estimated covariance matrix and the target matrix is a weighted sum, and l2 distance is used measure the difference between the true/underlying covariance matrix and the mixed covariance matrix. As known, covariance matrices are positive semi-definite, thus they lie on semi-Riemannian manifold with manifold-observingg distance measure. A reasonable choice is to conduct the shrinkage estimation on the semi-Riemannian manifold, and CKA is applied to measure the distance as rotations and istropic scaling should be ignored with measuring two sets of word vectors. The following sections will discuss our approach in details.
SHRINKAGE OF GRAM MATRIX
Learning word vectors can be viewed as estimating the oracle gram matrix where the relationship of words is expressed, and the post-processing method we propose here aims to best recover the oracle gram matrix. Suppose there exists an ideal gram matrix E Raxn where each entry represents the similarity of word pair defined by kernel function k[ws, wy). Given the assumption of a gram matrix and the existence of polysemies, the oracle gram matrix K is positive semi-definite and its rank is between 0 and and denoted A:
A set of pretrained word vectors is provided by previously proposed algorithm, and for simplicity, an embedding matrix E E xxd is constructed in which each row is word vector v. The estimated gram matrix K' EE E ==*=" has rank d and < k. It means that overall, the previously proposed algorithms give low-rank approximation of the oracle gram matrix K.
2

---

Under review as a conference paper at ICLR 2020
The goal of shrinkage is to, after the estimated gram matrix K' is available, find optimal post- processing method to maximise the similarity between the oracle gram matrix K and the estimated one K' given simple assumptions about the ideal one without directly finding it. Therefore, proper similarity measure that inherits and respects our assumptions is crucial.
3.1 CENTRALISED KERNEL ALIGNMENT (CKA)
The similarity measure between two sets of word vectors should be invariant to any rotations and also to isotropic scaling. A reasonable choice is the Centralised Kernel Alignment (CKA) (Cortes et al. 2012) defined below:
(Ke, K2)
p c. K.)
where K. = 1 11 and (Ke, K.)) Kr) For simplicity of the deriva- tions below, we assume that the ideal gram matrix is centralised, and the estimated one K' can be centralised easily by removing the mean of word vectors. In the following section, K and K' are used to denote centralised gram matrices.
As shown in Eq. [ it is clear that p(K, is invariant to any rotations and isotropic scaling, and doesn't suffer from the issues K and K' being low-rank. The centralised kernel alignment has been recommended recently neural networks (Korbllith et al.) 2019). As learning word vectors can be thought of as a one-layer neural network with linear activation function, pl K') reasonable similarity measure between the ideal gram matrix and our shrunken one provided by learning algorithms. Given the Cauchy- Schwartz inequality and also the non-negativity of the Frobenius norm, p(K, K') E [0, 1].
proper measure for comparing the similarity of features learnt by
Since both and K' are positive semi-definite, their eigenvalues are denoted as (Aoi Aany A
and (uu , Aug.... } respectively, and the eigenvalues of K K' are denoted as {Ai, An,,2,... .2, Aa} as the rank is determined by the matrix with lower rank [1 The log-transformation conducted to simplify the derivation.
log p( ((KK') K') log Tr(K')) 1/2 log TrKK ) 1/2 log Tr(K'K"))
1 log (det(K) det((')) 1 log Tr KK
log Tr(K'K"T)
log Ao, Et
2 Σi= log Au. log Ei= X?; 1 log Ei= A
where the lower bound is given by the AM-GM inequality, and the equality holds when all eigen- values of K K' are the same a = A2
Ad-
3.2 SHRINKAGE OF GRAM MATRIX ON DEMI-RIEMANNIAN MANIFOLD
As the goal is to find post-processing method that maximises the similarity p(K, K'), a widely adopted approach is to shrink the estimated gram matrix K' towards the target matrix T with predefined structure. The target matrix is usually positive semi-definite and has same or higher rank than the estimated gram matrix. In our case, we assume T is full rank as it simplifies equations.
Previous methods rely linear combination of K' and T in Euclidean space (Ledoit & Wolf, 2004 Schäafer & Strimmer] 2005). However, as gram matrices are positive semi-definite, they nat- urally lie on the Riem-ninannnaan manifold. Therefore, a suitable option for shrinkage is to move the estimated gram matrix K' towards the target matrix T on a semi-Riemannian manifold (Brenier) 1987). and the resulting matrix Y is given by
Y 122'
(3)
where 3 E [0, 1] is the mixing parameter that indicates the strength of the shrinkage, and T/2 is the square root of the matrix T. The objective is to find the optimal / that maximises log pl K, K").
8 arg max, log p p(K, Y) arg max, log e
(4)
The objective defined in Eq. is hard to work with, instead, we maximise the lower bound in Eq. 2 to find optimal mixing parameter Br. By plugging Y into Eq. 2 and denoting Ay as the i-th eigenvalue of Y, we have:
log K, Y) 1 log + + log Ao log Et_ Na, log Ex X
(5)
'No ANg due to the centralisation.
3

---

Under review as a conference paper at ICLR 2020
3.3 SCALED IDENTITY MATRIX AS TARGET FOR SHRINKAGE
Recent progress analysing pretrained word vectors Mu al. 2018 Arora et al. 2017 recom- mended to make them isotropic by removing top principal components as they highly correlate with the frequency information of words, and it results in a more compressed eigenspectrum. In Eu- clidean space-based shrinkage methods (Anderson 2004 [Schäfer& & Strimmer 2005), the spectrum also smoothed by mixing the eigenvalues with ones balance the overestimated large eigenvalues and underestimated small eigenvalues. In both cases, the goal is smooth the spectrum to make more isotropic, thus an easy target matrix to start with is the scaled identity matrix, which is noted as T = aT, where is the scaling parameter and I is the identity matrix. Thus, the resulting matrix Y and the lower bound in Eq. become
Y = (aT) ((o1) (or)) (aT) (aT) a.-88'' KK'
(6)
log p( KY 2 = log Ao; + + E log XE, 1 log Ei= de, log Σ
28
Av;
easy to show that, when B = O, the resulting matrix Y becomes the target matrix aI, and when 8 = 1, no shrinkage is performed as Y K'. Eq. Zindicates that the lower bound is only function of 3 with no involvement from the scaling parameter a brought by the target matrix as the CKA is invariant to isotropic scaling.
3.4 NOISELESS ESTIMATION
Starting from the simplest case, we assume that the estimation process from learning word vectors E to constructing the estimated gram matrix K' is noiseless, thus the first order and the second order derivative of [B3) with respect to B are crucial for finding the optimal 3":
0C(3) 08
Σ E=
log Ave
Σ 1=1
X log Av,
(8)
d
Σ
Aus
5=1
EEE 22 log Aoo
8 CC -2
Ao;
20 log" Au
+
(9)
0
22
Ei= 22
Since L(3) is invariant to isotropic scaling on (2v, /i E 1.2.....d) d], we are able to scale them by the inverse of their sum, then pe; = / Din Au; defines a distribution. To avoid losing information of the estimated gram matrix K", we redefine the plausible range for & as (0, 1]. Eq. interpreted as
8 can be
aC(3) 03
2R Σd= Any
EE=1 9h Pu;
(9: r,r(
2 (H((()) - H(q,r(3))) ( (H(((3)) H(g) D (3)) <0
(10)
where DKz (glpp) is the Kullback-Leibler divergence and it is non-negative, H(g) is the entropy of the uniform distribution and / (r(3),p) p) is the cross-entropy for r and The inequality derives from the fact that the uniform distribution has the maximum entropy. Thus, the first order derivative C') is always less than O. With the same definition of qi and r(3): in Eq. 10 the second order derivative (3(3 can be rewritten as
(²C(8) = - 108). (E r(3), log - Dd log 0,3²
H (r(3),)) (H()( ((()))))(((((()) H(q. (())) <0
(11)
As shown, L'(3) < 0 means that C(3) monotonically decreases in the range of (0, 1) for 3, and the re- sulting matrix Y = aI completely ignores the estimated gram matrix K'. (3) also monotonically decreases in the range of (0, 1] for 3 because ("(B) < 0.
The observation is similar to the finding reported in previous work (Schäfer & Strimmer 2005) on shrinkage estimation of covariance matrices that any target will increase the similarity between the oracle covariance matrix and the mixed covariance matrix. In our case, simply reducing 8 from to 0 increases Eq. and consequently, 3 04 loses all information estimated which is not ideal.
However, one can rewrite Eq. I1 as L" C"(3) = и (r(3). p)² H(g (r(3).p) and q is the uniform distribution which has maximum entropy, the derivative of (B) with respect H (r(3).p) p) gives
4

---

Under review as a conference paper at ICLR 2020
2H (r(3).))-H(C and by setting the derivative to O, it tells us that there exists 3 which results in specific r that gives H p) = (g)). Since H (r(3))p) monotonically increases when 3 moves from 04 to 1, and limg-oo_ H (r(3).p) = H (4. < (q) and H (r(1). > 2 (4), there exists single 8 e (0, 1] that gives the smallest value of (3)). This indicates that there exists a & that leads to the slowest change in the function value [()). Then, we simply set 3 = B. In practice, & should be larger than 0.5 in order not to make the resulting matrix overpowered by the target matrix, so one could run a binary search on 3 [0.5, 1] that gives smallest value of (3)..
Intuitively, this method is similar to using an elbow plot to find the optimal number of components to keep when running Principal Component Analysis (PCA) on observed data. To translate in our case, a larger value of / leads to flatter spectrum and higher noise level as the scaled identity matrix represents isotropic noise. The optimal 3 gives us point where the increase of ((3) is the slowest.
Once optimal mixing parameters 8 are found, the resulting matrix Y = = = E (E*)) and is omitted here. As a post-processing method for word vectors, we propose to transform the word vectors in the following fashion:
1. E = E - > E,
2. Singular Value Decomposition E = USV
3. Optimal Mixing Parameter 3 = arg ming L"
4. Reconstruct the matrix E' = =S) V
4 EXPERIMENTS
Three learning algorithms are applied to derive word vectors, including skipgram (Mikolov al. 2013b), CBOW (Mikolov et al. 2013a) and GloVe (Pennington et al. 2014 EE serves as the estimated gram matrix K". Hyperparameters of each algorithm are set the recommended values. The training corpus is collected from Wikipedia and it contains around 4600 million tokens. The unique tokens include ones that occur at least 100 times in the corpus and the resulting dictionary size is 366990. After learning, the word embedding matrix is centralised and then the optimal mixing parameter /²' directly estimated from the singular values of the centralised word vectors. After- wards, the postprocessed word vectors are evaluated on a series of word-level tasks to demonstrate the effectiveness of our method. Although the range of 3 is set to [0.5, 1.0] and the optimal value is found by minimising (*)8), it seldomly hits 0.5, which means that searching is still necessary.
4.1 COMPARISON PARTNERS
Two comparison partners are chosen demonstrate the effectiveness of our proposed method. One is the method (Mu et al. 2018)) that removes the top 2 or 3 principal components on the zero-centred word vectors, and the number of top principal components to remove depends on the training algo- rithm used for learning word vectors, and possibly the training corpus. Thus, the selection process not automatic, while ours is. The hypothesis behind this method is similar to ours, as the goal is to make the learnt word vectors more isotropic. However, removing top principal components, which are correlated with the frequency of each word, could also remove relevant information that indicates the relationship between words. The observation that our method is able to perform on par with or better than this comparison partner supports our hypothesis.
The other comparison partner is the optimal shrinkage (Ledoit & Wolf, 2004) on the estimated covariance matrices as the estimation process tends to overestimate the directions with larger power and underestimate the ones with smaller power, and the optimal shrinkage aims to best recover the true covariance matrix with minimum assumptions about it, which has the same goal as our method. The resulting matrix Y is defined to be linear combination of the estimated covariance matrix and the target with predefined structure, and it is denoted as Y = (1 P)aI + SΣ', where in our case, '' E E Raxd, The optimal shrinkage parameter 3 is found by minimising the Euclidean distance between the true covariance matrix Σ and the resulting matrix Y. The issue here is that Euclidean distance and linear combination may not be the optimal choice for covariance matrices, and also Euclidean distance is not invariant to isotropic scaling which is desired property when measuring the similarity between sets of word vectors.
5

---

Under review as a conference paper at ICLR 2020
4.2 WORD-LEVEL EVALUATION TASKS
Three categories of word-level evaluation tasks are considered, including 8 tasks for word similarity, 3 for word analogy and for concept categorisation. The macro-averaged results for each category of tasks are presented in Table Details about tasks are included in the appendix.
Table 1: Results of three post-processing methods including ours on our pretrained word vec- tors from three learning algorithms. each cell, the three numbers refer word vectors produced by "Skipgram/ CBOW GloVe", and the dimension of them is 500. Bold indicates the best macro- averaged performance of post- post-processing methods. It shows that overall, our method is effective.
Postprocessing
Similarity (8)
Analogy (3)
Concept (6)
Overall (17)
None
64.66 04.66/57..66 57.65 54.81
46.23 / /8/70/ 46.68
68.70 68.34 / 66.86
62.83 / 15..84//55/ /
Mu 2018
64.24 60.12 61.57
46.87 49. 10 40.09
68.86 70.46 / 65.54
62.81 61.83/1////// 59.18
Ledoit & Wolf 2004
63.39 59.19 50.81
46.64 50.38 45.54
69.45 68.83/ 69.55
62.57 61.03 56.50
Ours
63.79 60.90 53.16
46.51 49.84 47.25
70.83 69.84 65.97
63.23 /2.10/
4.3 WORD TRANSLATION TASKS
Table presents results on supervised and unsupervised word translation tasks (Lample et al. 2018 given pretrained word vectors from FastText (Bojanowski et al. 2017). The supervised word translation is formalised as solving a Procrustes problem directly. The unsupervised task first trains mapping through adversarial learning and then refines the learnt mapping through the Procrustes problem with a dictionary constructed from the mapping learnt in the first step. The evaluation is done by k-NN search and reported in top-1 precision.
word translation. The translation is done through k-NN with two dis- tances, in which one is the cosine similarity (noted as "NN" in the table), and the other one the Cross-domain Similarity Local Scaling (CSLS) (Lample et al. 2018). The Ledoit & Wolf's method didn't converge on unsupervised training so we excluded results from the method in the table.
Table 2: Performance
Post-processing
en-es
es-en
NN
CSLS
NN
CSLS
NN
CSLS
NN
CSLS
NN
CSLS
NN
CSLS
Supervised Procrustes Problem
None
79.0
81.7
79.2
83.3
78.4
82.1
78.5
81.9
71.1
73.4
69.7
72.7
Mu et 2018
80.0
81.7
80.4
83.0
79.3
82.4
78.5
81.7
72.9
75.4
70.7
73.1
Ledod & Woll 2004
80.1
82.3
79.6
82.9
78.7
81.9
78.9
82.0
72.3
74.1
70.6
72.4
Ours
81.1
82.3
80.9
83.0
79.3
82.1
79.2
81.7
72.5
74.5
71.7
72.9
Unsupervised Adversarial Training Refinement
None
79.7
82.0
78.8
83.7
78.4
81.8
77.9
82.1
71.7
74.7
69.7
73.1
Mu et al. 2018
79.7
81.7
80.5
84.3
79.6
82.7
78.9
81.8
72.3
75.0
71.2
72.9
Ours
80.4
82.3
81.1
83.1
79.5
82.0
79.5
82.1
72.1
74.8
71.7
73.7
4.4 SENTENCE-LEVEL EVALUATION TASKS
The evaluation tasks include five tasks from SemEval Semantic Textual Similarity (STS) in 2012- 2016 (Agirre et al. 2012, 2013 2014 2015 2016). sentence vector is constructed by simply averaging word vectors postprocessed word vectors, and the similarity measure is cosine similar- ity as well. The performance on each task is reported as the Pearson's 's correlation score.
We evaluate post-processing at two different levels, in which applics the post-processing methods on word vectors before averaging them to form sentence vectors (Mu et al. [2018), and the other applies methods averaged word vectors, which sentence vectors, directly (Arora

---

Under review as a conference paper at ICLR 2020
Applying the post-pree pest-proeessing to sentence vectors results in better performance and our method leads to the best performance on each dataset. Results are reported in Table
al. 2017 7)...
Table 3: Performance our post- post-proessiin method and the other two comparison partners on SemEval datasets. The task is to make good predictions of sentence vectors composed of averaging word vector. The word-level post-procsssing means that all methods are applied on word vectors before averaging, and the sentence-level one means that all methods are applied sentence vectors. Our method performs better than others at sentence-level and slightly worse than the method that removes top principal components the word level. Overall the sentence-level post-processing results in superior performance. Our method has the best overall performance on each dataset.
STS12
STS13
STS14
STS15
STSI6
STS12
STS13
STS14
STSI5
STS16
Postprocessing
word-Ievel post-processing Mu al.
2018
sentence-level post post-processing (Arora et al
2017
RM Top PCs
59.07
53.30
65.98
69.54
67.22
59.20
57.92
69.31
74.08
72.26
Ledoit & Wolf 2004)
54.75
47.93
60.57
65.36
64.87
54.47
57.92
64.79
70.84
58.53
Ours
57.36
51.80
65.04
68.23
66.31
62.60
62.09
70.99
75.83
74.49
5 DISCUSSION
The results presented in Table [
2 and 3 indicate the effectiveness of our methods. addition, by comparing to two most related post procesrocessing methods, which are removing top principal com- ponents Muu et al. 2018 Arora et al., 2017) and the optimal shrinkage estimation on covariance matrices Euclidean space (Ledoit & Wolf 2004 Schäffer & Strimmer 2005), our method com- bines the best of the both worlds the good performance of the first method and the automaticity property of the second one.
Specifically, our method derives the resulting gram matrix in a semi-Riemannen manifold, and the shrinkage is done by moving the estimated gram matrix K' to the scaled identity matrix af on the geodesic on the semi-Riemannian manifold defined in Eq. 6 Compared to measuring the Euclidean distance between two covariance matrices (Ledoit & Wolf 2004), we chose a distance measure that respects the structure of the space where positive semi-definit matrices exist. Also, since we are working with improving the quality of learnt word vectors, given that mostly the angular distance between a pair of word vectors matters, the similarity measure between two spaces needs to be invariant to rotation and isotropic scaling, and the Centralised Kernel Alignment is good fit in our case. Compared to the post-processing method that removes top principal components, our method defines an objective function where the unique minimum point in the specific range indicates the optimal shrinkage parameter Sr, and which doesn't it require any human supervision finding the parameters (the number of principal components to remove in that method).
The derivations in work mostly done the lower bound in Eq. of the original log of CKA p(K, K'), and the bound is tight when KK' has a flat spectrum. In this sense, the lower bound essentially defines the similarity of the resulting matrix Y and scaled identity matrix where the scaling factor is the average of the eigenvalues of KK'. As the isotropic scaling can be ignored in our case, intuitively, the lower bound (Eq. 2 gives us how to shrink an estimated gram matrix to an identity matrix. Since the target matrix T = al is also a scaled identity matrix, the lower bound keeps increasing as Y travels from K' T. and it explains why the first order derivative is negative and the second order derivative is more informative. Our method is very similar to using an elbow plot find the number of eigenvalues to keep in PCA. Future work should focus on finding tighter bound that considers the interaction between the true gram matrix K and the estimated one K'.
Figure shows that our methods give solid and stable improvement against two comparison part- ners on learnt word vectors provided by skipgram and CBOW with different dimensions, and slightly worse than removing top PCs on GloVe. However, skipgram and CBOW provide much better per- formance thus improvement these algorithms is more meaningful, and our method doesn't it require manual speculations on the word vectors to determine the number of PCs to remove. An interesting observation is that there is a limit the performance improvement from increasing the dimensional- ity of word vectors, which was hypothesised and analysed in prior work (Yin & Shen 2018). Figure 2 shows that our method consistently outperforms two comparison partners when limited amount of

---

Under review as a conference paper at ICLR 2020
skipgram
cbow
glove
0.60
0.55
200
300
400
500
600
200
300
400
500
600
200
300
400
500
600
Figure 1: Performance change of each post-processing method when the dimension of learnt word vectors increases. Our method is comparable to the method that removes top PCs on skip- gram, and better than that on CBOW, and slightly worse than that on GloVe. However, generally Skipgram and CBOW themselves provide better performance, thus boosting performance by post- processing on those two learning algorithms is more important. In addition, our model doesn't 't require manually picking the dimensions to remove instead the optimal 3 is found by optimisation.
training corpus provided, and the observation is consistent across learnt word vectors with varying dimensions. From the perspective of shrinkage estimation of covariance matrices, the main goal is to optimally recover the oracle covariance matrix under a certain assumption about the oracle one when limited data is provided. The results presented in Figure 2 indicate that our method is better at recovering the gram matrix of words, where relationships of word pairs are expressed, as it gives better performance word-level evaluation tasks on average.
ski p50
skippram-400
skippram-300
0.62
0.58
chow-500
chow
cbow-300
0.62
0.60
0.58
10
10a 10
10 10"
10
Figure 2: Averaged score on tasks vs. Percentage of training data. simulate the situa- tion where only limited data is available for shrinkage estimation, two algorithms are trained with subsampled data with different portions, and three postprocessing methods including ours and two comparison partners applied afterwards. The observation here is that method is able to recover the similarity between words better than others when only small amount of data is available.
CONCLUSION
We define a post-processing method in the view of shrinkage estimation of the gram matrix. Armed with CKA and geodesic measured on the remi-Riemannin manifold, a meaningful lower bound is derived and the second order derivative of the lower bound gives the optimal shrinkage parameter 3* to smooth the spectrum of the estimated gram matrix which directly calculated by pretrained word vectors. Experiments on the word similarity and sentence similarity tasks demonstrate the effectiveness of our model. Compared to the two most relevant post-processing methods (Mu et al.) 2018 Ledoit & Wolf 2004), ours is more general and automatic, and gives solid performance im- provement. As the derivation in our paper is based on the general shrinkage estimation of positive semi-definite matrices, it is potentially useful and beneficial for other research fields when Euclidean measures are not suitable. Future work could expand our work into a more general setting for shrink- age estimation currently the target in our case an isotropic covariance matrix.
8

---

Under review as a conference paper at ICLR 2020
ACKNOW
REFERENCES
Ralph Abraham, Jerrold Marsden, Tudor S. Ratiu, and Cécile DeWittMorette. Manifolds, tensor analysis, and applications. 1983.
Eneko Agirre, Enrique Alfonseca, Keith B. Hall, Jana Kravalova, Marius Pasca, and Aitor Soroa. A study on similarity and relatedness using distributional and wordnet-based approaches. HLT- NAACL, 2009.
Eneko Agirre, Daniel M. Cer, Mona Diab, and Aitor Gonzalez-Agiree Semeval-2012 task A pilot on semantic textual similarity. In SemEval@ NAACL-HLT, 2012.
Eneko Agirre, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-ggirre, and Weiwei Guo. *sem 2013 shared task: Semantic textual similarity. #SEM@NAACI-HHIN 2013.
Eneko Agirr,, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. Semeval-2014 task 10: Multi- lingual semantic textual similarity. In SemEval @ COLING, 2014.
Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Weiwei Guo, lñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. SemEval NAACL-HLT, 2015.
Eneko Agirre, Carmen Banea, Daniel M. Cer, Mona T. Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. Semeval-201 task 1: Semantic textual similarity, monolingual and cross- lingual evaluation. In SemEval NAACL-HLT, 2016.
AbduIrahman Almuhareb. Attributes in lexical acquisition. PhD thesis, University of Essex, 2006.
Olivier Ledoit Anderson. A well-conditioned estimator for large-dimensional covariance matrices. 2004.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. simple but tough-to- to-batt baseline for sentence embeddings. In International Conference Learning Representations, 2017.
Marco Baroni, Stefan Evert, and Alessandro Lenci. Esslli workshop on distributional lexical seman- tics bridging the gap between semantic theory and computational simulations. 2008
Marco Baroni, Brian Murphy, Eduard Barbu, and Massimo Poesio. Strudel: A corpus- based seman- tic model based on properties and types. Cognitive science, 34 2:222-54, 2010.
Ian M Benn and Robin W Tucker. An introduction to spinors and geometry with applications in physics. 1987.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. TACL, 5:135-146, 2017.
Yann Brenier. Décomposition polaire et réarrangement monotone des champs de vecteurs. CR Acad. Sci. Paris Sér. 1 Math., 305:805-808, 1987.
Elia Bruni, Yam-KKanh Tran, and Marco Baroni. Multimodal distributional semantics. Artif. Intell. Res., 19:1-47, 2014.
Corinna Cortes, Masanori Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. Journal of Machine Learning Research, 13:795-822 828, 2012.
Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, and George E. Dahl. Embedding text in hyperbolic spaces. In TextGraphs@ CwanGrrpph@NNAACLHHLL. NAACL-HLT, 2018.
Kawin Ethayarajh. Unsupervised random walk sentence embeddings: A strong but simple baseline. In @CC,, 2018

---

Under review as a conference paper at ICLR 2020
Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. Placing search in context: the concept revisited. ACM Trans. Inf. Syst., 26:116- 131. 2002.
Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic attention networks. In International Conference on Learning Representations, 2019.
URL
Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren, Large-scale learning of word relatedness with constraints. In KDD, 2012.
Zellig S Harris. Distributional structure. Word, 10(2-3):146-11) 1954.
Felix Hill, Roi Reichart, and Anna Korhonen. Simlex-999 999: Evaluating semantic models with (gen- uine) similarity estimation. Computational Linguistics, 41:665-695, 2015.
Stanislaw Jastrzebski, Damian Lesniak, and Wojciech Czarnecki. How to evaluate word embed- dings? importance of data efficiency and simple supervised tasks. CoRR, abs/1702.022 2017.
David Jurgens, Saif Mohammad, Peter D. Turney, and Keith J. Holyoak. Semeval-2011 task 2: Measuring degrees of relational similarity. In NAACLLIIL,, 2012.
Simon Korblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Pro- ceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 3519-3529, Long Beach, California, USA, 09-15 Jun 2019. ml press
PMLR. URL. http:ttt/p/ppcceeeeeet
ithl 9a html
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learming Representations, 2018. URL httttttst//ppnnnee
ew www. frrt/foorumiiii d= 96sainb)
Olivier Ledoit and Michael Wolf. Honey, i shrunk the sample covariance matrix. 2004.
Omer Levy, Yoav Goldberg, and Ido Dagan. Improving distributional similarity with lessons learned from word embeddings. TACL, 1-225,,,,, 2015.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. arXiv preprint 13Xiv:1301.3781 2013a.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. In NIPS, 2013b.
Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In HLT-NAACL, 30.
Jiaqi Mu, Suma Bhat, and Pramod Viswanath. All-but-the-top: Simple and effective postprocessing for word representations. In International Conference Learning Representations, 2018.
Maximilian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representa- tions. In NIPS, 2017.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
Herbert Rubenstein and John B. Goodenough. Contextual correlates of synonymy. Commun. ACM, 8:627-633, 1965.
Juliane Schäfer and Korbinian Strimmer. A shrinkage approach to large-scale covariance matrix estimation and implications for functional genomics. Statistical applications in genetics and molecular biology, 4:Article32, 2005.
10

---

Under review as a conference paper at ICLR 2020
Peter D. Tumny and Patrick Pantel. From frequency meaning: Vector space models of semanties.
Artif. Intell. Res., 37:141-188, 2010.
Zi Yin and Yuanyuan Shen. On the dimensionality of word embedding. In Advances in Neural Information Processing Systems, 2018.
APPENDIX
A WORD-LEVEL EVALUATION TASKS
word similarity tasks: MEN Bruni al. 2014), SimLee999 (Hil al. 2015), MTurk MTuukHHalawi
et al. 2012). WordSim353(Finkee stein al. 2002). WordSim-353-RF: (Agirre et al. 2009), BordSim-353SSMM(Aggie
et al., (200)), RG65 Rubenstein Goodenough, 1965). On these, the similarity of pair of words is calculated by the cosine similarity between two word vectors, and the performance reported by computing Spearman's 's correlation between human annotations with the predictions from the learnt vectors.
word analogy tasks: Google Analogy(Mikolov et al. 2013a). 2omEEvll20122uureaal et al. 2012), MMR(Mikolov al., 2013c). The task here is is to a* as b is to b" where a, a" and are given and the goal to find the correct answer b*. The "3CosAdd" method (Levy et al. 2015) is applied retrieve the answer, which is denoted as arg max VVu bbat = cos(b", a a a++ b). Accuracy is reported.
the questions in the form of "a
concept categorisation tasks: BM Baroni et al. 2010), AP(Almuhareb) 2006),
BLESS(Jastizebski et al.] (017)). ESSLI(Baroni al. 2008). The task is to measure whether words with similar meanings are nicely clustered together. An unsupervised clustering algorithm with predefined number of clusters is performed each dataset, and the performance reported
also in accuracy.
B PYTHON CODE
import numpy np
def beta postprocessing (emb):
# embedding size: number of word vectors x dim of vectors
mean np. mean mean(vecs, axis=0, keepdims=True) centred emb emb
u, 8, = np. linalg svd (centred )mb)
def objective(b). **
12
logl mp. (og(1)
13
12b = ** (2+b)
14
first_term np mean logl 12b (olbbllogl>>> ) mean mean (12b)
second term np. mean (log] + +12b ++2. derivative =
17
(np. mean 12b) 2.) (first.
second term
18
return derivative
20
values list (map objective np. arange (0.5, 1.0, 0.001))) optimal beta = np argmin (values) 1000. 0.5
21
22
processed emb @ ap (s (ptimal_beta) @ vh
25
return processed_eme
Listing 1: Python code for searching for the optimal 3*
11


Reviews:
[{"review_id": "rylgvrfAtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel word embedding post-processing method that maximizes the similarity between the estimated Gram matrix of word vectors and its oracle matrix. To find the optimal Gram matrix, they adopt the shrink method to make Gram matrix K' to target matrix T on semi-Riemannian space. The authors use the shrinkage method to find optimal K' and formulate the maximization on CKA problem as finding the optimal shrinkage parameter that maximizes the lower bound of the CKA between the estimated Gram matrix and oracle. By applying the proposed method to various word embedding methods, the authors show the performance of their post-processing method on word analogy/similarity task, word translation task, and sentence similarity task.\n\nStrengths\n* This paper provides a novel post-processing method that can relieve isotropy condition and shows experimental support that solving isotropy condition on word embedding vectors can improve its performance.\n* Large set of experiments on various word embedding benchmark tasks. \n\nWeaknesses\n* It would be nice if the authors show the performance of the post-processed word vectors on other NLP benchmark: text classification, NER, ... etc.\n\nQuestion\n* \"Neural Word Embedding as Implicit Matrix Factorization\" and \"Analogies Explained: Towards Understanding Word Embeddings\" show that PMI is the global optimum point of the previous word embedding model's problem space and prove word analogy can be explained from the PMI characteristics of word embedding models. How can this paper be related to the two papers mentioned above?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1g_kvLgqr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel method for unsupervised post-processing of pretrained word embeddings that enforces the distributional word vector space to be more isotropic, which in turn improves the expressiveness and quality of the space in terms of similarity. The method is based on the shrinkage of the covariance/Gram matrix and its effects on the input space are evaluated across a range of intrinsic evaluation tasks. While I like the idea overall and this line of work in general, there are still some concerns with the current version of the paper:\n\n* Overall, although its design seems more principled, the proposed method does not seem significantly better than the previous (very similar) method of Mu et al. I would like to see more evidence in the favour of the proposed method, pointing that we should use that instead of Mu's method. Also, the gains over non-processed spaces often seem insignificant, and offer only small benefits.\n\n* The derivation of the method seems too verbose, especially in light of the fact that it is directly inspired by previous work on the shrinkage estimation of covariance matrices and CKA. I would suggest the authors to spend more time on linking their high-level hypotheses to the low-level mathematical implementations instead of flooding the paper with equations - for the interested reader a lot of the derivation process can be put into an appendix, the paper should focus on conveying the key principles instead. This would also offer additional space for more experiments.\n\n* One very relevant paper is not mentioned at all: https://arxiv.org/pdf/1809.02094.pdf (Artetxe et al., CoNLL 2018). I would suggest the authors to cite that work and ideally even compare to it on their set of intrinsic tasks (e.g., word similarity, word analogy), and then discuss the difference in results and their approach to unsupervised post-processing. This shouldn't be so difficult to do as the code from that paper is available online: https://github.com/artetxem/uncovec\n\n* The paper mixes true similarity datasets (such as SimLex) with broader semantic relatedness datasets (MEN, WordSim-353), even mixing true semantic similarity of the same dataset (WordSim353-SIM) with its relatedness subset (WordSim353-REL). In light of the known conceptual differences between the relations of similarity versus relatedness, I would suggest to report the results separately for the two tasks. For instance, another true similarity dataset, which is not used in the evaluation is SimVerb-3500. Along the same line, it is also not clear what type of similarity is meant when the authors state that through post-processing 'the similarity between words can be better expressed'. What does it mean to better express the similarity between words in the first place? Do we talk about true similarity or relatedness or both? However, the two relationships between words support different classes of downstream applications, so therefore it is even more problematic 1) not to distinguish between the two and 2) not to report any results in any downstream (extrinsic) tasks where the post-processed embeddings are used as features (STS is still a semi-intrinsic task imho; BLI is considered as an intrinsic task in cross-lingual settings).\n\n* One of the key reasons to apply post-processing is to mitigate the frequency artefacts: however, such an evaluation that goes towards that direction is never executed. For instance, I would like to see a focused experiment that measures how post-processing affects high-frequency versus mid-to-lower frequency or rare words. A recently developed CARD-660 dataset might be used to this end.\n\n* It is not clear how exactly the authors run Mu et al.'s method, that is, how many top principal components are removed for the input vectors? How is this selected? Are always the optimal results reported for the baseline Mu et al.'s method? Why not reporting the results with removing 2 and 3 at the same time to further prove the point that their method is non-automatic? This would also give the reader a hint how much the results with Mu's method actually differ/decrease if one just decides to make the method 'automatic' by just fixing the method to always remove the same number of top principal components.\n\n\nMinor remarks:\n* The title of the paper is a bit imprecise: in the word embedding literature, the term post-processing is often referred to the methods that fine-tune word embeddings using some external knowledge after (i.e., post) the initial distributional training (e.g., the so-called retrofitting methods). However, in the context of the paper post-processing actually refers to some unsupervised post-training steps on the input space without injecting any external information. This should be made clearer in the paper, and perhaps adding a paragraph which outlines the core difference to other work on retrofitting would be helpful as well.\n\n* I might be missing something while reading Section 3, but it is currently not clear to me how the oracle Gram matrix K is obtained in the first place. Perhaps it makes sense to briefly summarize this in a quite direct way to avoid the reader's confusion?\n\n* It is great to see a summary of the key post-processing steps at the very end of Section 3; this is really helpful for everyone who would like to try out the proposed method off-the-shelf. However, the summary is not self-contained as it is not clear what \\mathcal{L}'' refers to (and the reader must search through the derivations again to find its meaning).\n\n* I like the evaluation on word translation, and I believe that the proposed post-processing methods could actually improve word translation through some pre-alignment perturbations. It is a pity that the method is not evaluated on more distant language pairs, as I believe that the method might have much more effect there than on the already-saturated EN-to-ES/FR/IT bilingual lexicon induction tasks.\n\n* For de-en word translation Mu's method actually beats the proposed method (incorrect number in bold)\n\n* It is not clear why MUSE is used for word retrieval experiments, given the fact that it is known to be unstable (Sogaard et al., ACL 2019), and there are more robust and more effective methods available such as VecMap (Artetxe et al., ACL 2018) or RCSLS (Joulin et al., EMNLP 2019)", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygceR5dFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a theoretical framework for post-processing methods of word embeddings. Given the framework, the authors derive their own methods and then a thorough experimental analysis follows.\n\nWhile the paper reflects thorough and substantial work - both in the theoretical framework and in the experimental part, I have serious concerns about its clarity and about the experimental results and also some concerns about comparison with previous work. All these, unfortunately, make me recommend a reject decision. Below, are more details:\n\n1. Clarity: There are multiple aspects of the paper that I needed to read a few times before I understood what the authors mean. \n\nFirst, the problem definition is not clear: Even when I reached the end of the introduction, it was not clear to me what is the problem that the authors are trying to solve and what they are trying to achieve. One example of this problem is the title of the paper, which suggests that the author performs an empirical evaluation, while in practice they also suggest their own method. But this is a much broader issue - the author suggest very little motivation to how the post-processing method should work, what improvements it should provide and why we should expect such improvements. At some points I felt that this is more of a clean mathematical exercise than a discussion of methods that should improve word representations in natural language.\n\nIn addition, the authors assume a strong background in a very specific post-processing  literature and do not provide any details about its fundamentals. Only on the beginning of section 3 I learned the fundamentals of that framework and basic concepts such as the Gram matrix. I believe a scientific paper should be self-contained, the motivations, goals and fundamental concepts form previous work should be clearly stated and explained. This is not done in this paper, unfortunately.\n\nFinally, it was hard for me to determine where the survey of previous work ends and the contribution of this work begins. Particularly, it seems that the authors propose a unified framework for the methods in previous work and it is not clear which parts of that framework were already discussed in previous work and which are original contributions of the authors. This makes it also hard to estimate how different the proposed method is form the previous ones.\n\n2. Experimental analysis:\n\nFirst, the authors describe their evaluation tasks very briefly and only in the appendix. This is just a list of tasks with no insight about natural language (please see a related comment in the clarity section of this review). Then, the results are reported as a macro-average over many tasks: Given the large number of tasks this is a very crude average, and there is no way that any real insight into the change/improvement of the vectors can be derived from this report. Finally, the reported numbers reflect very minor improvements, if at all, compared to previous post-processing methods and to the original vectors. Again, since these are macro-averages over many tasks, the conclusions that can be derived are very limited (e.g. it might be that the proposed method does improve on some of the tasks and harm the performance on others, or that it keeps the vectors very similar to the original ones - we have no way figuring out the actual picture).\n\n\n3. Comparison to previous work:\n\nAs said above, it seems that the authors view their work in a narrow context of a very specific literature. In fact, the NLP literature contains a large number of post-processing word embedding methods (often referred to as \"fitting\" or \"specialization\" methods). While these methods sometimes build on external linguistic knowledge (e.g. from wordNet or from other manually crafted lexicons), they have also shown useful with automatically constructed constraints, that are similar to the structural considerations mentioned in the paper in the sense that they do not require expert knowledge, they only build on common-sensical requirements from a good vector space for word meaning representation. Some relevant papers are:\n\nFaruqui, M., Dodge, J., Jauhar, S. K., Dyer, C., Hovy, E., & Smith, N. A. (2014). Retrofitting word vectors to semantic lexicons. arXiv preprint arXiv:1411.4166.\n\nMrkšić, N., Séaghdha, D. O., Thomson, B., Gašić, M., Rojas-Barahona, L., Su, P. H., ... & Young, S. (2016). Counter-fitting word vectors to linguistic constraints. arXiv preprint arXiv:1603.00892.‏‏\n\nMrkšić, N., Vulić, I., Ó Séaghdha, D., Leviant, I., Reichart, R., Gašić, M., ... & Young, S. (2017). Semantic specialization of distributional word vector spaces using monolingual and cross-lingual constraints. Transactions of the association for Computational Linguistics, 5, 309-324.‏", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores a post-processing method for word vectors to "smooth the spectrum," and show improvements on some downstream tasks. 

Reviewers had some questions about the strength of the results, and the results on words of differing frequency. The reviewers also have comments on the clarity of the paper, as well as the exposition of some of the methods.

Also, for future submissions to ICLR and other such conferences, it is more typical to address the authors comments in a direct response rather than to make changes to the document without summarizing and pointing reviewers to these changes. Without direction about what was changed or where to look, there is a lot of burden being placed on the reviewers to find your responses to their comments.
Decision: Reject