Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper provides theoretical and empirical motivations for removing the top few principle components of commonly-used word embeddings.

The paper is well-written and I enjoyed reading it. However, it does not explain how significant this result is beyond that of (Bullinaria and Levy, 2012), who also removed the top N dimensions when benchmarking SVD-factorized word embeddings. From what I can see, this paper provides a more detailed explanation of the phenomenon ("why" it works), which is supported with both theoretical results and a series of empirical analyses, as well as "updating" the benchmarks and methods from the pre-neural era. Although this contribution is relatively incremental, I find the depth of this work very interesting, and I think future work could perhaps rely on these insights to create better embedding algorithms that directly enforce isotropy.

I have two concerns regarding the empirical section, which may be resolvable fairly quickly:
1) Are the embedding vectors L2 normalized before using them in each task? This is known to significantly affect performance. I am curious whether removing the top PCs is redundant or not given L2 normalization.
2) Most of the benchmarks used in this paper are "toy" tasks. As Schnabel et al (2015) and Tsvetkov et al (2015) showed, there is often little correlation between success on these benchmarks and improvement of downstream NLP tasks. I would like to measure the change in performance on a major NLP task that heavily relies on pre-trained word embeddings such as SQuAD.

Minor Comments:
* The last sentence in the first paragraph ("The success comes from the geometry of the representations...") is not true; the success stems from the ability to capture lexical similarity. Levy and Goldberg (2014) showed that searching for the closest word vector to (king - man + woman) is equivalent to optimizing a linear combination of 3 similarity terms [+(x,king), -(x,man), +(x, woman)]. This explanation was further demonstrated by Linzen (2016) who showed that even when removing the negative term (x, man), many analogies can still be solved, i.e. by looking for a word that is similar both to "king" and to "woman". Add to that the fact that the analogy trick works best when the vectors are L2 normalized; if they are all on the unit sphere, what is the geometric interpretation of (king - man + woman), which is not on the unit sphere? I suggest removing this sentence and other references to linguistic regularities from this paper, since they are controversial at best, and distract from the main findings.
* This is also related to Bullinaria and Levy's (2012) finding that downweighting the eigenvalue matrix in SVD-based methods improves their performance. Levy et al (2015) showed that keeping the original eigenvalues can actually degenerate SVD-based embeddings. Perhaps there is a connection to the findings in this paper?

review_title: In-depth explanation of known phenomenon. Experiments need some strengthening.

final_score: rating: 7: Good paper, accept
review_content: This paper proposes that sets of word embeddings can be improved by subtracting the common mean vector and reducing the effect of dominant components of variation. 

Comments:

Reference to 'energy' and 'isotropic' in the first paragraph come without any explanation. Can plain terms be used instead to express the same ideas? This would make understanding easier (I have a degree in maths, but never studied physics, and I had to look them up). Otherwise, I like the simple explanation of the method given in the intro. 

The experiments conducted in the paper are comprehensive. It is very positive that the improvements appear to be quite consistent across well-known tasks. As well as proposing a simple trick to produce these improvements, the authors aim to provide theoretical insight and (implicitly, at least) pursue a better understanding of semantic word spaces. This has the potential to be a major contribution, as such spaces, and semantic representation in neural nets in general, is poorly understood. However (perhaps because I'm not familiar with Arora et al.) I found the mathematical analysis e.g. S2.2 dense, without any clearly-stated intuitive motivation or conclusions (as per the introduction section) about what is going on semantically. E.g. it is not clear to me why isotropy is something desirable in a word embedding space. I understand that the discarded components tend to encode frequency, and this is very interesting and somewhat indicative f why the method might work. However, Figure 2 is particularly hard to interpret? The correlations, and the distribution of high-frequency words) seems to be quite different for each of the three models?! 

In general, I don't think the authors should rely on readers having read Arora et al. - anything that builds on that work needs to reintroduce their findings in pain terms in the current paper. 

Another concern is the novelty in relation to related work. I have not read Arora et al. but the authors say that they 'null away the first principal component', and Sahlgren et al centre the mean. Taken together, this seems very similar to what the authors propose here (please clarify). More generally, these sorts of tricks have often been applied by deep learning researchers and passed around anecdotally (e.g. initialise transition matrix in RNNs with orthonormal noise) as ways to improve training. It is important to share and verify these things, but such a contribution feels more appropriate for a workshop than the main conference. This makes the work that the authors do in interpreting and understanding why these tricks work particularly important. As is, however, I thing that the conclusions from this analysis are unclear and opaque. Can they be better communicated, or is it the case that the results of the analysis are in fact inconclusive?

The vast amount of work included in the appendix is impressive. What particularly caught my eye was appendix B, where the authors try to understand if their method can simply be 'learned' by any network that uses pre-trained word embeddings. This is a really nice experiment, and I think it could easily be part of the main paper (perhaps swapping with the stuff in section 2.2). 

The conclusion would be a good place for summarising the main findings in plain terms, but that doesn't really happen (unless the finding about frequency is the only finding). Instead, there is a vague connection to population genetics and language evolution. This may be an interesting future direction, but the connection is tenuous, so that this reader, at least, was left a little baffled. 

[REVISED following response]

Thanks for your thorough response which did a good job of addressing most of my concerns. I have changed the score accordingly.
   


review_title: Simple method to tidy up word embedding spaces; novelty unclear, conclusions opaque

final_score: rating: 7: Good paper, accept
review_content: This paper proposes a simple post-processing technique for word representations designed to improve representational quality and performance on downstream tasks. The procedure involves mean subtraction followed by projecting out the first D principle directions and is motivated by improving isotropy of the partition function. Extensive empirical analysis supports the efficacy of the approach.

The idea of post-processing word embeddings to improve their performance is not new, but I believe the specific procedure and its connection to the concept of isotropy has not been investigated previously. Relative to other post-processing techniques, this method has a fair amount of theoretical justification, particularly as described in Appendix A. I think the experiments are reasonably comprehensive. All told, I think this is a good paper, but I do have some comments and questions that I think should be addressed before publication.

1) I think it is useful to analyze the distribution of singular values of the matrix of word vectors. However, I did not find the heuristic analysis based on the visual appearance of these distributions to be convincing. For example, in Fig. 1, it is not clear to me that there exists a separation between regimes of exponential decay and rough constancy. It would be ideal if a more quantitative metric is established that captures the main qualitative behavior alluded to here.

Furthermore, the vocabulary size is likely to have a strong effect on the shape of the distributions. Are the plots in Fig. 4 for the same vocabulary size? Related to this, the dimensionality of the representation will have a strong effect on the shape, and this should be controlled for in Fig. 8. One way to do this would be to instead plot the density of singular values. Finally, for the Gaussian matrix simulations, in the asymptotic limit, the density of singular values depends only on the ratio of dimensions, i.e. the vector dimension to the vocabulary size. Fig. 4/8 might be more revealing if this ratio were controlled for.

2) It would be useful to describe why isotropy of the partition function is the goal, as opposed to isotropy of the vectors themselves. This may be argued in Arora et al. (2016), but summarizing that argument in this paper would be helpful. In fact, an additional experiment that would be very valuable would be to investigate empirically which form of isotropy is more effective in governing performance. One way to do this would be to enforce approximate isotropy of the partition function without also enforcing isotropy of the vectors themselves. Practically speaking, one might imagine doing this by requiring I = 1 to second order without also requiring that the mean vanish. I think this would allow for \sigma_max > \sigma_min while still satisfying I = 1 to second order. (But this is just off the top of my head -- there may be better ways to conduct this experiment).

It is not clear to me why the experiment leading to Table 2 is a good proxy for the exact computation of I. It would be great if there were some mathematical justification for this approximation.

Why does Fig. 3 use D=10, 20 when much smaller D are considered elsewhere? Also I think a log scale on the x-axis might be more informative.

3) It would be good to mention other forms of post-processing, especially in the context of word similarity. For example, in the original paper, GloVe advocates averaging the target and context vector representations, and normalizing across the feature dimension before computing cosine similarity.

4) I think it's likely that there is a strong connection between the optimal value of D and the frequency distribution of words in the evaluation dataset. While the paper does mention that D may depend on specifics of the dataset, etc., I would expect frequency-dependence to be the main factor, and it might be worth exploring this effect explicitly.

review_title: Simple post-processing technique with theoretical motivations

Meta-review: This is a good paper with strong results via a set of simple steps for post processing off the shelf words  embeddings.  Reviewers are enthusiastic about it and the author responses are satisfactory.
Decision: Accept (Poster)