Title: ALL-BUT-THE-TOP: SIMPLE AND EFFECTIVE POST-PROCESSING FOR WORD REPRESENTATIONS

Abstract: Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a very simple, and yet counter-intuitive, postprocessing technique -eliminate the common mean vector and a few top dominating directions from the word vectors -that renders off-the-shelf representations even stronger. The postprocessing is empirically validated on a variety of lexical-level intrinsic tasks (word similarity, concept categorization, word analogy) and sentence-level tasks (semantic textural similarity and text classification) on multiple datasets and with a variety of representation methods and hyperparameter choices in multiple languages; in each case, the processed representations are consistently better than the original ones.

1. INTRODUCTION:

Words and their interactions (as sentences) are the basic units of natural language. Although words are readily modeled as discrete atomic units, this is unable to capture the relation between the words. Recent distributional real-valued representations of words (examples: word2vec, GloVe) have transformed the landscape of NLP applications -for instance, text classification (Socher et al., 2013b; Maas et al., 2011; Kim, 2014) , machine translation (Sutskever et al., 2014; Bahdanau et al., 2014) and knowledge base completion (Bordes et al., 2013; Socher et al., 2013a) . The success comes from the geometry of the representations that efficiently captures linguistic regularities: the semantic similarity of words is well captured by the similarity of the corresponding vector representations.

A variety of approaches have been proposed in recent years to learn the word representations: Collobert et al. (2011) ; Turian et al. (2010) learn the representations via semi-supervised learning by jointly training the language model and downstream applications; Bengio et al. (2003) ; Mikolov et al. (2010) ; Huang et al. (2012) do so by fitting the data into a neural network language model; Mikolov et al. (2013) ; Mnih & Hinton (2007) by log-linear models; and Dhillon et al. (2012) ; Pennington et al. (2014) ; Levy & Goldberg (2014) ; Stratos et al. (2015) ; Arora et al. (2016) by producing a lowdimensional representation of the cooccurrence statistics. Despite the wide disparity of algorithms to induce word representations, the performance of several of the recent methods is roughly similar on a variety of intrinsic and extrinsic evaluation testbeds.

In this paper, we find that a simple processing renders the off-the-shelf existing representations even stronger. The proposed algorithm is motivated by the following observation.

Observation Every representation we tested, in many languages, has the following properties:

• The word representations have non-zero mean -indeed, word vectors share a large common vector (with norm up to a half of the average norm of word vector). • After removing the common mean vector, the representations are far from isotropic -indeed, much of the energy of most word vectors is contained in a very low dimensional subspace (say, 8 dimensions out of 300).

Implication Since all words share the same common vector and have the same dominating directions, and such vector and directions strongly influence the word representations in the same way, we propose to eliminate them by: (a) removing the nonzero mean vector from all word vectors, effectively reducing the energy; (b) projecting the representations away from the dominating D directions, effectively reducing the dimension. Experiments suggest that D depends on the representations (for example, the dimension of the representation, the training methods and their specific hyperparameters, the training corpus) and also depends on the downstream applications. Nevertheless, a rule of thumb of choosing D around d/100, where d is the dimension of the word representations, works uniformly well across multiple languages and multiple representations and multiple test scenarios.

We emphasize that the proposed postprocessing is counter intuitive -typically denoising by dimensionality reduction is done by eliminating the weakest directions (in a singular value decomposition of the stacked word vectors), and not the dominating ones. Yet, such postprocessing yields a "purified" and more "isotropic" word representation as seen in our elaborate experiments.

Experiments By postprocessing the word representation by eliminating the common parts, we find the processed word representations to capture stronger linguistic regularities. We demonstrate this quantitatively, by comparing the performance of both the original word representations and the processed ones on three canonical lexical-level tasks:

• word similarity task tests the extent to which the representations capture the similarity between two words -the processed representations are consistently better on seven different datasets, on average by 1.7%; • concept categorization task tests the extent to which the clusters of word representations capture the word semantics -the processed representations are consistently better on three different datasets, by 2.8%, 4.5% and 4.3%; • word analogy task tests the extent to which the difference of two representations captures a latent linguistic relation -again, the performance is consistently improved (by 0.5% on semantic analogies, 0.2% on syntactic analogies and 0.4% in total). Since part of the dominant components are inherently canceled due to the subtraction operation while solving the analogy, we posit that the performance improvement is not as pronounced as earlier.

Extrinsic evaluations provide a way to test the goodness of representations in specific downstream tasks. We evaluate the effect of postprocessing on a standardized and important extrinsic evaluation task on sentence modeling: semantic textual similarity task -where we represent a sentence by its averaged word vectors and score the similarity between a pair of sentences by the cosine similarity between the corresponding sentence representation. Postprocessing improves the performance consistently and significantly over 21 different datasets (average improvement of 4%).

Word representations have been particularly successful in NLP applications involving supervisedlearning, especially in conjunction with neural network architecture. Indeed, we see the power of postprocessing in an experiment on a standard text classification task using a well established convoluntional neural network (CNN) classifier (Kim, 2014) and three RNN classifiers (with vanilla RNN, GRU (Chung et al., 2015) and LSTM Greff et al. (2016) as recurrent units). Across two different pre-trained word vectors, five datasets and four different architectures, the performance with processing improves on a majority of instances (34 out of 40) by a good margin (2.85% on average), and the two performances with and without processing are comparable in the remaining ones.

Related Work. Our work is directly related to word representation algorithms, most of which have been elaborately cited.

Aspects similar to our postprocessing algorithm have appeared in specific NLP contexts very recently in (Sahlgren et al., 2016) (centering the mean) and (Arora et al., 2017) (nulling away only the first principal component). Although there is a superficial similarity between our work and (Arora et al. 2017) , the nulling directions we take and the one they take are fundamentally different. Specifically, in Arora et al. (2017) , the first dominating vector is *dataset-specific*, i.e., they first compute the sentence representation for the entire semantic textual similarity dataset, then extract the top direction from those sentence representations and finally project the sentence representation away from it. By doing so, the top direction will inherently encode the common information across the entire dataset, the top direction for the "headlines" dataset may encode common information about news articles while the top direction for "Twitter'15" may encode the common information about tweets. In contrast, our dominating vectors are over the entire vocabulary of the language.

More generally, the idea of removing the top principal components has been studied in the context of positive-valued, high-dimensional data matrix analysis (Bullinaria & Levy, 2012; Price et al., 2006) .

Bullinaria & Levy (2012) posits that the highest variance components of the cooccurrence matrix are corrupted by information other than lexical semantics, thus heuristically justifying the removal of the top principal components. A similar idea appears in the context of population matrix analysis (Price et al., 2006) , where the entries are also all positive. Our postprocessing operation is on dense low-dimensional representations (with both positive and negative entries).

We posit that the postprocessing operation makes the representations more "isotropic" with stronger self-normalization properties -discussed in detail in Section 2 and Appendix A. Our main point is that this isotropy condition can be explicitly enforced to come up with new embedding algorithms (of which our proposed post-processing is a simple and practical version).

2. POSTPROCESSING:

We test our observations on various word representations: four publicly available word representations (WORD2VEC1 (Mikolov et al., 2013) trained using Google News, GLOVE2 (Pennington et al., 2014) trained using Common Crawl, RAND-WALK (Arora et al., 2016) trained using Wikipedia and TSCCA3 trained using English Gigaword) and two self-trained word representations using CBOW and Skip-gram (Mikolov et al., 2013) Let v(w) ∈ R d be a word representation for a given word w in the vocabulary V. We observe the following two phenomena in each of the word representations listed above:

• {v(w) : w ∈ V} are not of zero-mean: i.e., all v(w) share a non-zero common vector, v(w) = ṽ(w) + µ, where µ is the average of all v(w)'s, i.e., µ = 1/|V| w∈V v(w). The norm of µ is approximately 1/6 to 1/2 of the average norm of all v(w) (cf. Table 1 ). • {ṽ(w) : w ∈ V} are not isotropic: Let u 1 , ..., u d be the first to the last components recovered by the principal component analysis (PCA) of {ṽ(w) : w ∈ V}, and σ 1 , ..., σ d be the corresponding normalized variance ratio. Each ṽ(w) can be written as a linear combinations of u: ṽ(w) = d i=1 α i (w)u i . As shown in Figure 1 , we observe that σ i decays near exponentially for small values of i and remains roughly constant over the later ones. This suggests there exists D such that α i α j for all i ≤ D and j D; from Figure 1 one observes that D is roughly 10 with dimension d = 300.

Angular Asymmetry of Representations A modern understanding of word representations involves either PMI-based (including word2vec (Mikolov et al., 2010; Levy & Goldberg, 2014) and GloVe (Pennington et al., 2014) ) or CCA-based spectral factorization approaches. While CCA-based spectral factorization methods have long been understood from a probabilistic (i.e., generative model) view point (Browne, 1979; Hotelling, 1936) and recently in the NLP context (Stratos et al., 2015) , a corresponding effort for the PMI-based methods has only recently been conducted in an inspired work (Arora et al., 2016) . Arora et al. (2016) propose a generative model (named RAND-WALK) of sentences, where every word is parameterized by a d-dimensional vector. With a key postulate that the word vectors are angularly uniform ("isotropic"), the family of PMI-based word representations can be explained under the RAND-WALK model in terms of the maximum likelihood rule. Our observation that word vectors learnt through PMI-based approaches are not of zero-mean and are not isotropic (c.f. Section 2) contradicts with this postulate. The isotropy conditions are relaxed in Section 2.2 of (Arora et al., 2016) , but the match with the spectral properties observed in Figure 1 is not immediate. This contradiction is explicitly resloved by relaxing the constraints on the word vectors to directly fit the observed spectral properties. The relaxed conditions are: the word vectors should be isotropic around a point (whose distance to the origin is a small fraction of the average norm of word vectors) lying on a low dimensional subspace. Our main result is to show that even with this enlarged parameter-space, the maximum likelihood rule continues to be close to the PMI-based spectral factorization methods. A brief summary of RAND-WALK, and the mathematical connection between our work and theirs, are explored in detail in Appendix A.

2.1. ALGORITHM:

Since all word representations share the same common vector µ and have the same dominating directions and such vector and directions strongly influence the word representations in the same way, we propose to eliminate them, as formally achieved as Algorithm 1. Significance of Nulled Vectors Consider the representation of the words as viewed in terms of the top D PCA coefficients α (w), for 1 ≤ ≤ D. We find that these few coefficients encode the frequency of the word to a significant degree; Figure 2 illustrates the relation between the (α 1 (w), α 2 (w)) and the unigram probabilty p(w), where the correlation is geometrically visible. Discussion In our proposed processing algorithm, the number of components to be nulled, D, is the only hyperparameter that needs to be tuned. We find that a good rule of thumb is to choose D approximately to be d/100, where d is the dimension of a word representation. This is empirically justified in the experiments of the following section where d = 300 is standard for published word representations. We trained word representations for higher values of d using the WORD2VEC and GLOVE algorithms and repeated these experiments; we see corresponding consistent improvements due to postprocessing in Appendix C.

2.2. POSTPROCESSING AS A "ROUNDING" TOWARDS ISOTROPY:

The idea of isotropy comes from the partition function defined in (Arora et al., 2016) ,

Z(c) = w∈V exp c v(w) ,

where Z(c) should approximately be a constant with any unit vector c (c.f. Lemma 2.1 in (Arora et al., 2016) ). Hence, we mathematically define a measure of isotropy as follows,

EQUATION

where I({v(w)}) ranges from 0 to 1, and I({v(w)}) closer to 1 indicates that {v(w)} is more isotropic. The intuition behind our postprocessing algorithm can also be motivated by letting

I({v(w)}) → 1.

Let V be the matrix stacked by all word vectors, where the rows correspond to word vectors, and 1 |V| be the |V|-dimensional vectors with all entries equal to one, Z(c) can be equivalently defined as follows,

Z(c) = |V| + 1 | V| V c + 1 2 c V V c + ∞ k=3 1 k! w∈V (c v(w)) k .

I({v(w)}) is, therefore, can be very coarsely approximated by,

• A first order approximation:

I({v(w)}) ≈ |V| + min c =1 1 |V| V c |V| + max c =1 1 |V| V c = |V| -1 |V| V |V| + 1 |V| V .

Letting I({v(w)}) = 1 yields 1 |V| V = 0, which is equivalent to w∈V v(w) = 0. The intuition behind the first order approximation matches with the first step of the proposed algorithm, where we enforce v(w) to have a zero mean.

• A second order approximation:

I({v(w)}) ≈ |V| + min c =1 1 |V| V c + min c =1 1 2 c V V c |V| + max c =1 1 |V| V c + max c =1 1 2 c V V c = |V| -1 |V| V + 1 2 σ 2 min |V| + 1 |V| V + 1 2 σ 2 max ,

where σ min and σ max are the smallest and largest singular value of V , respectively. Letting I({v(w)}) = 1 yields 1 |V| V = 0 and σ min = σ max . The fact that σ min = σ max suggests the spectrum of v(w)'s should be flat. The second step of the proposed algorithm removes the highest singular values, and therefore explicitly flatten the spectrum of V .

Empirical Verification Indeed, we empirically validate the effect of postprocessing of on I({v(w)}). Since there is no closed-form solution for arg max c =1 Z(c) or arg min c =1 Z(c), and it is impossible to enumerate all c's, we estimate the measure by,

I({v(w)}) ≈ min c∈C Z(c) max c∈C Z(c) ,

where C is the set of eigenvectors of V V . The value of I({v(w)}) for the original vectors and processed ones are reported in Table 2 , where we can observe that the degree of isotropy vastly increases in terms of this measure.

before after WORD2VEC 0.7 0.95 GLOVE 0.065 0.6

Table 2 : Before-After on the measure of isotropy.

A formal way to verify the isotropy property is to directly check if the "self-normalization" property (i.e., Z(c) is a constant, independent of c (Andreas & Klein, 2015) ) holds more strongly. Such a validation is seen diagrammatically in Figure 3 where we randomly sampled 1,000 c's as (Arora et al., 2016) . 

3. EXPERIMENTS:

Given the popularity and widespread use of WORD2VEC (Mikolov et al., 2013) and GLOVE (Pennington et al., 2014) , we use their publicly available pre-trained reprepsentations in the following experiments. We choose D = 3 for WORD2VEC and D = 2 for GLOVE. The key underlying principle behind word representations is that similar words should have similar representations. Following the tradition of evaluating word representations (Schnabel et al., 2015; Baroni et al., 2014) 

Word Similarity:

The word similarity task is as follows: given a pair of words, the algorithm assigns a "similarity" score -if the pair of words are highly related then the score should also be high and vice versa. The algorithm is evaluated in terms of Spearman's rank correlation compared to (a gold set of) human judgements.

For this experiment, we use seven standard datasets: the first published RG65 dataset (Rubenstein & Goodenough, 1965) ; the widely used WordSim-353 (WS) dataset (Finkelstein et al., 2001) which contains 353 pairs of commonly used verbs and nouns; the rare-words (RW) dataset (Luong et al., 2013) composed of rarely used words; the MEN dataset (Bruni et al., 2014) where the 3000 pairs of words are rated by crowdsourced participants; the MTurk dataset (Radinsky et al., 2011) where the 287 pairs of words are rated in terms of relatedness; the SimLex-999 (SimLex) dataset (Hill et al., 2016) where the score measures "genuine" similarity; and lastly the SimVerb-3500 (SimVerb) dataset (Gerz et al., 2016) , a newly released large dataset focusing on similarity of verbs.

In our experiment, the algorithm scores the similarity between two words by the cosine similarity between the two corresponding word vectors (

CosSim(v 1 , v 2 ) = v 1 v 2 / v 1 v 2 ).

The detailed performance on the seven datasets is reported in Table 3 , where we see a consistent and significant performance improvement due to postprocessing, across all seven datasets. These statistics (average improvement of 2.3%) suggest that by removing the common parts, the remaining word representations are able to capture stronger semantic relatedness/similarity between words. Concept Categorization This task is an indirect evaluation of the similarity principle: given a set of concepts, the algorithm needs to group them into different categories (for example, "bear" and "cat" are both animals and "city" and "country" are both related to districts). The clustering performance is then evaluated in terms of purity (Manning et al., 2008) -the fraction of the total number of the objects that were classified correctly.

We conduct this task on three different datasets: the Almuhareb-Poesio (ap) dataset (Almuhareb, 2006) contains 402 concepts which fall into 21 categories; the ESSLLI 2008 Distributional Semantic Workshop shared-task dataset (Baroni et al., 2008 ) that contains 44 concepts in 6 categories; and the Battig test set (Baroni & Lenci, 2010 ) that contains 83 words in 10 categories.

Here we follow the setting and the proposed algorithm in (Baroni et al., 2014; Schnabel et al., 2015) -we cluster words (via their representations) using the classical k-Means algorithm (with fixed k). Again, the processed vectors perform consistently better on all three datasets (with average improvement of 2.5%); the full details are in Word Analogy The analogy task tests to what extent the word representations can encode latent linguistic relations between a pair of words. Given three words w 1 , w 2 , and w 3 , the analogy task requires the algorithm to find the word w 4 such that w 4 is to w 3 as w 2 is to w 1 .

We use the analogy dataset introduced in (Mikolov et al., 2013) . The dataset can be divided into two parts: (a) the semantic part containing around 9k questions, focusing on the latent semantic relation between pairs of words (for example, what is to Chicago as Texas is to Houston); and (b) the syntatic one containing roughly 10.5k questions, focusing on the latent syntatic relation between pairs of words (for example, what is to "amazing" as "apprently" is to "apparent").

In our setting, we use the original algorithm introduced in (Mikolov et al., 2013) to solve this problem, i.e., w 4 is the word that maximize the cosine similarity between v(w 4 ) and v(w 2 ) -v(w 1 ) + v(w 3 ). The average performance on the analogy task is provided in Table 5 (with a detailed performance provided in Table 19 in Appendix D). It can be noticed that while postprocessing continues to improve the performance, the improvement is not as pronounced as earlier. We hypothesize that this is because the mean and some dominant components get canceled during the subtraction of v(w 2 ) from v(w 1 ), and therefore the effect of postprocessing is less relevant. Semantic Textual Similarity Extrinsic evaluations measure the contribution of a word representation to specific downstream tasks; below, we study the effect of postprocessing on a standard sentence modeling task -semantic textual similarity (STS) which aims at testing the degree to which the algorithm can capture the semantic equivalence between two sentences. For each pair of sentences, the algorithm needs to measure how similar the two sentences are. The degree to which the measure matches with human judgment (in terms of Pearson correlation) is an index of the algorithm's performance. We test the word representations on 20 textual similarity datasets from the 2012-2015 SemEval STS tasks (Agirre et al., 2012; 2013; 2014; 2015) , and the 2012 SemEval Semantic Related task (SICK) (Marelli et al., 2014) .

Representing sentences by the average of their constituent word representations is surprisingly effective in encoding the semantic information of sentences (Wieting et al., 2015; Adi et al., 2016) and close to the state-of-the-art in these datasets. We follow this rubric and represent a sentence s based on its averaged word representation, i.e., v(s) = 1 |s| w∈s v(w), and then compute the similarity between two sentences via the cosine similarity between the two representations. The average performance of the original and processed representations is itemized in Table 6 (with a detailed performance in Table 20 in Appendix E) -we see a consistent and significant improvement in performance because of postprocessing (on average 4% improvement).

4. POSTPROCESSING AND SUPERVISED CLASSIFICATION:

Supervised downstream NLP applications have greatly improved their performances in recent years by combining the discriminative learning powers of neural networks in conjunction with the word representations. We evaluate the performance of a variety of neural network architectures on a standard and important NLP application: text classification, with sentiment analysis being a particularly important and popular example. The task is defined as follows: given a sentence, the algorithm needs to decide which category it falls into. The categories can be either binary (e.g., positive/negative) or can be more fine-grained (e.g. very positive, positive, neutral, negative, and very negative).

We evaluate the word representations (with and without postprocessing) using four different neural network architectures (CNN, vanilla-RNN, GRU-RNN and LSTM-RNN) on five benchmarks: (a) the movie review (MR) dataset (Pang & Lee, 2005) ; (b) the subjectivity (SUBJ) dataset (Pang & Lee, 2004) ; (c) the TREC question dataset (Li & Roth, 2002) ; (d) the IMDb dataset (Maas et al., 2011) ; (e) the stanford sentiment treebank (SST) dataset (Socher et al., 2013a) . A detailed description of these standard datasets, their training/test parameters and the cross validation methods adopted is in Appendix F. Specifically, we allow the parameter D (i.e., the number of nulled components) to vary between 0 and 4, and the best performance of the four neural network architectures with the now-standard CNN-based text classification algorithm (Kim, 2014) (implemented using tensorflow4 ) is itemized in Table 7 . The key observation is that the performance of postprocessing is better in a majority (34 out of 40) of the instances by 2.32% on average, and in the rest the instances the two performances (with and without postprocessing) are comparable. Table 7 : Before-After results (x100) on the text classification task using CNN (Kim, 2014) and vanilla RNN, GRU-RNN and LSTM-RNN.

A further validation of the postprocessing operation in a variety of downstream applications (eg: named entity recognition, syntactic parsers, machine translation) and classification methods (eg: random forests, neural network architectures) is of active research interest. Of particular interest is the impact of the postprocessing on the rate of convergence and generalization capabilities of the classifiers. Such a systematic study would entail a concerted and large-scale effort by the research community and is left to future research.

Discussion All neural network architectures, ranging from feedforward to recurrent (either vanilla or GRU or LSTM), implement at least linear processing of hidden/input state vectors at each of their nodes; thus the postprocessing operation suggested in this paper can in principle be automatically "learnt" by the neural network, if such internal learning is in-line with the end-to-end training examples. Yet, in practice this is complicated due to limitations of optimization procedures (SGD) and sample noise. We conduct a preliminary experiment in Appendix B and show that subtracting the mean (i.e., the first step of postprocessing) is "effectively learnt" by neural networks within their nodes.

5. CONCLUSION:

We present a simple postprocessing operation that renders word representations even stronger, by eliminating the top principal components of all words. Such an simple operation could be used for word embeddings in downstream tasks or as intializations for training task-specific embeddings. Due to their popularity, we have used the published representations of WORD2VEC and GLOVE in English in the main text of this paper; postprocessing continues to be successful for other representations and in multilingual settings -the detailed empirical results are tabulated in Appendix C.

Appendix: All-but-the-Top: Simple and Effective postprocessing for Word Representations

A ANGULAR ASYMMETRY OF REPRESENTATIONS:

A modern understanding of word representations involves either PMI-based (including word2vec (Mikolov et al., 2010; Levy & Goldberg, 2014) and GloVe (Pennington et al., 2014) ) or CCA-based spectral factorization approaches. While CCA-based spectral factorization methods have long been understood from a probabilistic (i.e., generative model) view point (Browne, 1979; Hotelling, 1936) and recently in the NLP context (Stratos et al., 2015) , a corresponding effort for the PMI-based methods has only recently been conducted in an inspired work (Arora et al., 2016) . (Arora et al., 2016) propose a generative model (named RAND-WALK) of sentences, where every word is parameterized by a d-dimensional vector. With a key postulate that the word vectors are angularly uniform ("isotropic"), the family of PMI-based word representations can be explained under the RAND-WALK model in terms of the maximum likelihood rule. Our observation that word vectors learnt through PMI-based approaches are not of zero-mean and are not isotropic (c.f. Section 2) contradicts with this postulate. The isotropy conditions are relaxed in Section 2.2 of (Arora et al., 2016) , but the match with the spectral properties observed in Figure 1 is not immediate.

In this section, we resolve this by explicitly relaxing the constraints on the word vectors to directly fit the observed spectral properties. The relaxed conditions are: the word vectors should be isotropic around a point (whose distance to the origin is a small fraction of the average norm of word vectors) lying on a low dimensional subspace. Our main result is to show that even with this enlarged parameter-space, the maximum likelihood rule continues to be close to the PMI-based spectral factorization methods. Formally, the model, the original constraints of (Arora et al., 2016) and the enlarged constraints on the word vectors are listed below:

• A generative model of sentences: the word at time t, denoted by w t , is generated via a log-linear model with a latent discourse variable c t (Arora et al., 2016) , i.e., p(w

EQUATION

where v(w) ∈ R d is the vector representation for a word w in the vocabulary V , c t is the latent variable which forms a "slowly moving" random walk, and the partition function: Z(c) = w∈V exp c v(w) . • Constraints on the word vectors: (Arora et al., 2016) suppose that there is a Bayesian priori on the word vectors:

The ensemble of word vectors consists of i.i.d. draws generated by v = s • v, where v is from the spherical Gaussian distribution, and s is a scalar random variable. A deterministic version of this prior is discussed in Section 2.2 of (Arora et al., 2016) , but part of these (relaxed) conditions on the word vectors are specifically meant for Theorem 4.1 and not the main theorem (Theorem 2.2). The geometry of the word representations is only evaluated via the ratio of the quadratic mean of the singular values to the smallest one being small enough. This meets the relaxed conditions, but not sufficient to validate the proof approach of the main result (Theorem 2.2); what would be needed is that the ratio of the largest singular value to the smallest one be small. • Revised conditions: We revise the Bayesian prior postulate (and in a deterministic fashion) formally as follows: there is a mean vector µ, D orthonormal vectors u 1 , . . . , u D (that are orthogonal and of unit norm), such that every word vector v(w) can be represented by,

EQUATION

where µ is bounded, α i is bounded by A, D is bounded by DA 2 = o(d), ṽ(w) are statistically isotropic. By statistical isotropy, we mean: for high-dimensional rectangles R,

1 |V|

w∈V 1(ṽ(w) ∈ R) → R f (ṽ)dṽ, as |V| → ∞, where f is an angle-independent pdf, i.e., f (ṽ) is a function of ṽ .

The revised postulate differs from the original one in two ways: (a) it imposes a formal deterministic constraint on the word vectors; (b) the revised postulate allows the word vectors to be angularly asymmetric: as long as the energy in the direction of u 1 ,. . . ,u D is bounded, there is no constraint on the coefficients. Indeed, note that there is no constraint on ṽ(w) to be orthogonal to u 1 , . . . u D .

Empirical Validation:

We can verify that the enlarged conditions are met by the existing word representations. Specifically, the natural choice for µ is the mean of the word representations and u 1 . . . u D are the singular vectors associated with the top D singular values of the matrix of word vectors. We pick D = 20 for WORD2VEC and D = 10 for GLOVE, and the corresponding value of DA 2 for WORD2VEC and GLOVE vectors are both roughly 40, respectively; both values are small compared to d = 300. This leaves us to check the statistical isotropy of the "remaining" vectors ṽ(w) for words w in the vocabulary. We do this by plotting the remaining spectrum (i.e. the (D + 1)-th, ..., 300th singular values) for the published WORD2VEC and GLOVE vectors in Figure 4 . As a comparison, the empirical spectrum of a random Gaussian matrix is also plotted in Figure 4 . We see that both spectra are flat (since the vocabulary size is much larger than the dimension d = 300). Thus the postprocessing operation can also be viewed as a way of making the vectors "more isotropic".

Mathematical Contribution Under the revised postulate, we show that the main theorem in (Arora et al., 2016) (c.f. Theorem 2.2) still holds. Formally: Theorem A.1 Suppose the word vectors satisfy the constraints. Then

EQUATION

where p(w) is the unigram distribution induced from the model (2), and p(w 1 , w 2 ) is the probability that two words w 1 and w 2 occur with each other within distance q.

The proof is in Appendix G. Theorem A.1 suggests that the RAND-WALK generative model and its properties proposed by (Arora et al., 2016) can be generalized to a broader setting (with a relaxed restriction on the geometry of word representations) -relevantly, this relaxation on the geometry of word representations is empirically satisfied by the vectors learnt as part of the maximum likelihood rule.

B NEURAL NETWORKS LEARN TO POSTPROCESS:

Every neural network family posseses the ability to conduct linear processing inside their nodes; this includes feedforward and recurrent and convolutional neural network models. Thus, in principle, the postprocessing operation can be "learnt and implemented" within the parameters of the neural network. On the other hand, due to the large number of parameters within the neural network, it is unclear how to verify such a process, even if it were learnt (only one of the layers might be implementing the postprocessing operation or via a combination of multiple effects).

To address this issue, we have adopted a comparative approach in the rest of this section. The comparative approach involves adding an extra layer interposed in between the inputs (which are word vectors) and the rest of the neural network. This extra layer involves only linear processing.

Next we compare the results of the final parameters of the extra layer (trained jointly with the rest of tne neural network parameters, using the end-to-end training examples) with and without preprocessing of the word vectors. Such a comparative approach allows us to separate the effect of the postprocessing operation on the word vectors from the complicated "semantics" of the neural network parameters. Experiment We construct a modified neural network by explicitly adding a "postprocessing unit" as the first layer of the RNN architecture (as in Figure 5 , where the appended layer is used to test the first step (i.e., remove the mean vector)) of the postprocessing algorithm.

v(w 1 ) v(w 2 ) v(w T ) … … v(w1) b v(w2) b v(wT ) b h1 h2 hT

In the modified neural network, the input word vectors are now v(w) -b instead of v(w). Here b is a bias vector trained jointly with the rest of the neural network parameters. Note that this is only a relabeling of the parameters from the perspective of the RNN architecture: the nonlinear activation function of the node is now operated on A(v(w) -b) + b = Av(w) + (b -Ab) instead of the previous Av(w) + b . Let b proc. and b orig. be the inferred biases when using the processed and original word representations, respectively.

We itemize the cosine similarity between b proc. + µ and b orig. in Table 6 for the 5 different datasets and 3 different neural network architectures. In each case, the cosine similarity is remarkably large (on average 0.66, in 300 dimensions) -in other words, trained neural networks implicitly postprocess the word vectors nearly exactly as we proposed. This agenda is successfully implemented in the context of verifying the removal of the mean vector.

The second step of our postprocessing algorithm (i.e., nulling away from the top principal components) is equivalent to applying a projection matrix P = I -D i=1 u i u i on the word vectors, where u i is the i-th principal component and D is the number of the removed directions. A comparative analysis effort for the second step (nulling the dominant PCA directions) is the following. Instead of applying a bias term b, we multiply by a matrix Q to simulate the projection operation. The input word vectors are now Q orig. v(w) instead of v(w) for the original word vectors, and Q proc. P v(w) instead of P v(w) for the processed vectors. Testing the similarity between Q orig. P and Q proc. , allows us to verify if the neural network learns to conduct the projection operation as proposed.

In our experiment, we found that such a result cannot be inferred. One possibility is that there are too many parameters in both Q proc. and Q orig. , which adds randomness to the experiment. Alternatively, the neural network weights may not be able to learn the second step of the postprocessing operation (indeed, in our experiments postprocessing significantly boosted end-performance of neural network architectures). A more careful experimental setup to test whether the second step of the postprocessing operation is learnt is left as a future research direction. 

C.4 ROLE OF DIMENSIONS:

The main text has focused on the dimension choice of d = 300, due to its popularity. In this section we explore the role of the dimension in terms of both choice of D and the performance of the postprocessing operation -we do this by using skip-gram model on the 2010 snapshot of Wikipedia corpus (Al-Rfou et al., 2013) to train word representations. We first observe that the two phenomena of Section 2 continue to hold:

• From Table 14 we observe that the ratio between the norm of µ and the norm average of all v(w) spans from 1/3 to 1/4; • From Figure 8 we observe that the decay of the variance ratios σ i is near exponential for small values of i and remains roughly constant over the later ones. A rule of thumb choice of D is around d/100. We validate this claim empirically by performing the tasks in Section 3 on word representations of higher dimensions, ranging from 300 to 1000, where we set the parameter D = d/100. In summary, the performance improvement on the four itemized tasks of Section 3 are 2.27%, 3.37%, 0.01 and 1.92% respectively; the detailed results can be found in Table 15, Table 16, Table 17, and Table 18 . Again, note that the improvement for analogy tasks is marginal. These experimental results justify the rule-of-thumb setting of D = d/100, although we emphasize that the improvements can be further accentuated by tuning the choice of D based on the specific setting.

D EXPERIMENTS ON WORD ANALOGY TASK:

The detailed performance on the analogy task is provided in Table 19 .

E EXPERIMENTS ON SEMANTIC TEXTUAL SIMILARITY TASK:

The detailed performance on the semantic textual similarity is provided in 

F STATISTICS OF TEXT CLASSIFICATION DATASETS:

We evaluate the word representations (with and without postprocessing) using four different neural network architectures (CNN, vanilla-RNN, GRU-RNN and LSTM-RNN) on five benchmarks:

• the movie review (MR) dataset (Pang & Lee, 2005) where each review is composed by only one sentence;

• the subjectivity (SUBJ) dataset (Pang & Lee, 2004) where the algorithm needs to decide whether a sentence is subjective or objective; Lemma G.1 Let c be a random variable uniformly distributed over the unit sphere, we prove that with high probability, Z(c)/|V| converges to a constant Z:

p((1 -z )Z ≤ Z(c) ≤ (1 + z )Z) ≥ 1 -δ,

where z = Ω((D + 1)/|V|) and δ = Ω((DA 2 + µ 2 )/d).

Our proof differs from the one in (Arora et al., 2016) in two ways: (a) we treat v(w) as deterministic parameters instead of random variables and prove the Lemma by showing a certain concentration of measure; (b) the asymmetric parts µ and u 1 ,...,u D , (which did not exist in the original proof), need to be carefully addressed to complete the proof.

G.1 PROOF OF LEMMA G.1:

Given the constraints on the word vectors (3), the partition function Z(c) can be rewritten as,

Z(c) = v∈V exp(c v(w)) = v∈V exp c µ + D i=1 α i (w)u i + ṽ(w) = v∈V exp(c µ) D i=1 exp(α i (w)c u i ) exp c ṽ(w) .

The equation above suggests that we can divide the proof into five parts.

Step 1: for every unit vector c, one has,

EQUATION

Proof Let M, N be a positive integer, and let

A M ⊂ R d such that, A M,N = ṽ ∈ R d : M -1 N < exp(c ṽ) ≤ M N .

Since A M,N can be represented by a union of countable disjoint rectangles, we know that for every M, N ∈ N + ,

1 |V| w∈V 1(ṽ(w) ∈ A M,N ) = A M,N

f (ṽ)dṽ. f (ṽ)dṽ = E f exp c ṽ , as |V| → ∞.

Putting ( 6) and ( 7) proves (5).

Step 2: the expected value, E f exp c ṽ is a constant independent of c:

EQUATION

Proof Let Q ∈ R d×d be a orthonormal matrix such that Q c 0 = c where c 0 = (1, 0, ..., 0) and det(Q) = 1, then we have f (ṽ) = f (Qṽ), and, E f exp c 0 ṽ = ṽ f (ṽ) exp c 0 ṽ dṽ = ṽ f (Qṽ) exp c Qṽ det(Q)dṽ = ṽ f (ṽ ) exp c ṽ dṽ = E f exp c ṽ , which proves (8).

Step 3: for any vector µ, one has the following concentration property,

p | exp c µ -1| > k ≤ 2 exp - 1 4 + µ 2 d -1 1 log 2 (1 -k) (9)

Proof Let c 1 ,...,c d be i.i.d. N (0, 1), and let C = C . We divide the proof into the following steps:

• C follows chi-square distribution with the degree of freedom of d, thus C can be bounded by (Laurent & Massart, 2000) , • Since c 1 is a Gaussian random variable with variance 1, by Chebyshev's inequality, one has, p (yc i ≥ k) ≤ y 2 k 2 , p (yc i ≤ -k) ≤ y 2 k 2 , ∀k > 0 and therefore thus, p(exp(yc i ) -1 > k) ≤ y 2 log 2 (1 + k) , p(exp(yc i ) -1 < -k) ≤ y 2 log(1 -k) 2 , ∀k > 0.

• Therefore we can bound exp µ c 1 / √ C by,

p exp µ c 1 √ C -1 > k ≤p (C > d + 1) + p exp µ c 1 √ C -1 > k C < d + 1 p (C < d + 1) ≤ exp - 1 4d + p exp µ c 1 √ d + 1 -1 > k = exp - 1 4d + µ 2 d + 1 1 log(1 -k) 2 . p exp µ c 1 √ C -1 < -k ≤ exp - 1 4d + µ 2 d -1 1 log 2 (1 + k) .

Combining the two inequalities above, one has (9) proved.

Step 4: We are now ready to prove convergence of Z(c). With ( 9 

u i ) -1 < k) -2D ≥ 1 -(2D + 1) exp - 1 4d - 2DA 2 d -1 1 log 2 (1 -k) - µ 2 d -1 1 log 2 (1 -k)

.

Next, we need to show that for every w, the corresponding C(w), i. G.2 PROOF OF THEOREM A.1

Having Lemma G.1 ready, we can follow the same proof as in (Arora et al., 2016) 

Therefore, the inference presented in (Arora et al., 2016) (i.e., (4) ) is obvious by assembling ( 12) and ( 13) together:

PMI(w, w ) → v(w) v(w ) d , as |V| → ∞.

:

https://code.google.com/archive/p/word2vec/

https://github.com/stanfordnlp/GloVe

http://www.pdhillon.com/code.html

https://github.com/dennybritz/cnn-text-classification-tf

Figures and Tables:

Figure 1: The decay of the normalized singular values of word representation.

Postprocessing algorithm on word representations. Input :Word representations {v(w), w ∈ V}, a threshold parameter D, Compute the mean of {v(w), w ∈ V}, µ ← 1 |V| w∈V v(w), ṽ(w) ← v(w) -µ Compute the PCA components: u 1 , ..., u d ← PCA({ṽ(w), w ∈ V}). Preprocess the representations: v (w) ← ṽ(w) -D i=1 u i v(w) u i Output :Processed representations v (w).

Figure 2: The top two PCA directions (i.e, α 1 (w) and α 2 (w)) encode frequency.

Figure 3: The histogram of Z(c) for 1,000 randomly sampled vectors c of unit norm, where x-axis is normalized by the mean of all values and D = 2 for GLOVE and D = 3 for WORD2VEC.

Figure 4: Spectrum of the published WORD2VEC and GLOVE and random Gaussian matrices, ignoring the top D components; D = 10 for GLOVE and D = 20 for WORD2VEC.

Figure 5: Time-expanded RNN architecture with an appended layer involving linear bias.

Figure 6: The cosine similarity (x100) between b proc. + µ and b orig. , where W. and G. stand for WORD2VEC and GLOVE respectively.

Figure 8: The decay of the normalized singular values of word representations.

, since A M,N are disjoint for different M 's and R d = ∪ ∞ M =1 A M,N , one has,The above statement holds for every N . Let N → ∞, by definition of integration, one has, (w) ≤ E f exp c ṽ , as |V| → ∞.

c = (c 1 , ..., c d )/√C is uniform over unit sphere. Since c is uniform, then without loss of generality we can consider µ = ( µ , 0, ..., 0).Thus it suffices to bound exp µ c 1 / √

Therefore for any x > 0, one has,p |C -d| ≥ 2 √ dx ≤ exp(-x) Let x = 1/4d, one has, p(C > d + 1) ≤ exp -

), let C ⊂ R d such that, C = c : exp(c µ) -1 < k, exp(Ac u i ) -1 < k, exp(-Ac u i ) -1 < k ∀i = 1, ..., DThen we can bound the probability on C by,p(C) ≥ p exp(c µ) -1 < k + D i=1 p( exp(Ac

e., C(w) = c : exp(c µ) -1 < k, exp(α i (w)c u i ) -1 < k, ∀i = 1, ..., DWe observe that α i (w) is bounded by A, therefore for any c that,min(exp(-Ac u i ), exp(Ac u i )) ≤ exp(α i c u i ) ≤ max(exp(-Ac u i ), exp(Ac u i )),and thus,min(exp(-Ac u i ), exp(Ac u i )) -1 ≤ exp(α i c u i ) -1 ≤ max(exp(-Ac u i ), exp(Ac u i )) -1, which yields, | exp(α i c u i ) -1| ≤ max(| exp(-Ac u i ) -1|, | exp(Ac u i ) -1|) < k.Therefore we prove C(w) ⊃ C. Assembling everything together, one has,p exp(c µ) D i=1 exp(α i (w)c u i ) -1 > (D + 1)k, ∀i = 1, ..., D,For every c ∈ C, one has,1 |V| |Z(c) -Z 0 | ≤ (D + 1)k |V| Z 0 . Let Z = |V|Z 0 , one can conclude that, p((1z )Z ≤ Z(c) ≤ (1 + z )Z) ≥ 1 -δ,where z = Ω((D + 1)/|V|) and δ = Ω(DA 2 /d).

on the 2010 Wikipedia corpus from(Al-Rfou et al., 2013). The detailed statistics for all representations are listed in Table1. For completeness, we also consider the representations on other languages: a detailed study is provided in Appendix C.2.

Table content: <table><tr><td/><td colspan="2">Language Corpus</td><td colspan="2">dim vocab size avg. v(w) 2</td><td>µ 2</td></tr><tr><td>WORD2VEC</td><td>English</td><td>Google News</td><td colspan="2">300 3,000,000 2.04</td><td>0.69</td></tr><tr><td>GLOVE</td><td>English</td><td colspan="3">Common Crawl 300 2,196,017 8.30</td><td>3.15</td></tr><tr><td colspan="2">RAND-WALK English</td><td>Wikipedia</td><td>300 68, 430</td><td>2.27</td><td>0.70</td></tr><tr><td>CBOW</td><td>English</td><td>Wikipedia</td><td colspan="2">300 1,028,961 1.14</td><td>0.29</td></tr><tr><td>Skip-Gram</td><td>English</td><td>Wikipedia</td><td colspan="2">300 1,028,961 2.32</td><td>1.25</td></tr><tr><td/><td colspan="4">Table 1: A detailed description for the embeddings in this paper.</td></tr></table>

Before-After results (x100) on word similarity task on seven datasets.

Table content: <table><tr><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="4">RG65 76.08 78.34 76.96 74.36</td></tr><tr><td colspan="4">WS 68.29 69.05 73.79 76.79</td></tr><tr><td colspan="4">RW 53.74 54.33 46.41 52.04</td></tr><tr><td colspan="4">MEN 78.20 79.08 80.49 81.78</td></tr><tr><td colspan="4">MTurk 68.23 69.35 69.29 70.85</td></tr><tr><td colspan="4">SimLex 44.20 45.10 40 83 44.97</td></tr><tr><td colspan="4">SimVerb 36.35 36.50 28.33 32.23</td></tr></table>



Table content: <table><tr><td/><td/><td/><td>.</td></tr><tr><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="4">syntax 73.46 73.50 74.95 75.40</td></tr><tr><td colspan="4">semantics 72.28 73.36 79.22 79.25</td></tr><tr><td colspan="4">all 72.93 73.44 76.89 77.15</td></tr><tr><td colspan="4">Table 5: Before-After results (x100) on the word</td></tr><tr><td>analogy task.</td><td/><td/><td/></tr></table>



Table content: <table><tr><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="4">2012 57.22 57.67 48.27 54.06</td></tr><tr><td colspan="4">2013 56.81 57.98 44.83 57.71</td></tr><tr><td colspan="4">2014 62.89 63.30 51.11 59.23</td></tr><tr><td colspan="4">2015 62.74 63.35 47.23 57.29</td></tr><tr><td colspan="4">SICK 70.10 70 20 65.14 67.85</td></tr><tr><td colspan="4">all 60.88 61.45 49.19 56.76</td></tr></table>

.27 71.01 71.11 74.95 74.01 71.14 72.56 77.86 78.26 74.98 75.13 75.69 77.34 72.02 71.84 SUBJ 87.14 87.33 86.98 87.25 82.85 87.60 81.45 87.37 90.96 91.10 91.16 91.85 90.23 90.54 90.74 90.82 TREC 87.80 89.00 87.60 89.00 80.60 89.20 85.20 89.00 91.60 92.40 91.60 93.00 88.00 91.20 85.80 91.20 SST 38.46 38.33 38.82 37.83 42.08 39.91 41.45 41.90 41.86 45.02 36.52 37.69 43.08 42.08 37.51 38.05 IMDb 86.68 87.12 87.27 87.10 50.15 53.14 52.76 76.07 82.96 83.47 81.50 82.44 81.29 82.60 79.10 81.33

Table content: <table><tr><td/><td colspan="2">CNN</td><td/><td/><td colspan="2">vanilla-RNN</td><td/><td/><td colspan="2">GRU-RNN</td><td/><td/><td colspan="2">LSTM-RNN</td><td/></tr><tr><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td><td colspan="2">WORD2VEC</td><td colspan="2">GLOVE</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="2">MR 70.80 71</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

Before-After results (x100) on the word similarity task in multiple languages. 300-dimension representations. In this section, we show that the proposed postprocessing algorithm generalizes to other representation methods. Specifically, we demonstrate this on RAND-WALK (obtained via personal communication) and TSCCA (publicly available) on all the experiments of Section 3. The choice of D is 2 for both RAND-WALK and TSCCA.In summary, the performance improvements on the similarity task, the concept categorization task, the analogy task, and the semantic textual similarity dataset are on average 2.23%, 2.39%, 0.11% and 0.61%, respectively. The detailed statistics are provided in Table10, Table11, Table12 and Table 13, respectively. These results are a testament to the generalization capabilities of the postprocessing algorithm to other representation algorithms.

Table content: <table><tr><td colspan="2">language</td><td>TSCCA orig.</td><td>proc.</td></tr><tr><td colspan="2">RG65 Spanish</td><td colspan="2">60.33 60.37</td></tr><tr><td colspan="4">GUR65 German 61.75 64.39</td></tr><tr><td colspan="2">GUR350 German</td><td colspan="2">44.91 46.59</td></tr><tr><td colspan="2">ZG222 German</td><td colspan="2">30.37 32.92</td></tr><tr><td colspan="2">RAND-WALK</td><td colspan="2">TSCCA</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="2">RG65 80.66 82.96</td><td colspan="2">47.53 47.67</td></tr><tr><td colspan="2">WS 65.89 74.37</td><td colspan="2">54.21 54.35</td></tr><tr><td colspan="2">RW 45.11 51.23</td><td colspan="2">43.96 43.72</td></tr><tr><td colspan="2">MEN 73.56 77.22</td><td colspan="2">65.48 65.62</td></tr><tr><td colspan="2">MTurk 64.35 66.11</td><td colspan="2">59.65 60.03</td></tr><tr><td colspan="2">SimLex 34.05 36.55</td><td colspan="2">34.86 34.91</td></tr><tr><td colspan="2">SimVerb 16.05 21.84</td><td colspan="2">23.79 23.83</td></tr></table>

Before-After results (x100) on the word similarity task on seven datasets.

Table content: <table><tr><td colspan="2">RAND-WALK</td><td colspan="2">TSCCA</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="2">ap 59.83 62.36</td><td colspan="2">60.00 63.42</td></tr><tr><td colspan="2">esslli 72.73 72.73</td><td colspan="2">68.18 70.45</td></tr><tr><td colspan="2">battig 75.73 81.82</td><td colspan="2">70.73 70.73</td></tr><tr><td colspan="4">Table 11: Before-After results (x100) on the categorization task.</td></tr><tr><td colspan="2">RAND-WALK</td><td colspan="2">TSCCA</td></tr><tr><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td colspan="2">syn. 60.39 60.48</td><td colspan="2">37.72 37.80</td></tr><tr><td colspan="2">sem. 83.55 83.82</td><td colspan="2">14.54 14.55</td></tr><tr><td colspan="2">all 70.50 70.67</td><td colspan="2">27.30 27.35</td></tr></table>

Before-After results (x100) on the word analogy task.

Table content: <table/>

Statistics on word representation of dimensions 300, 400, ..., and 1000 using the skip-gram model.

Table content: <table><tr><td/><td colspan="2">RAND-WALK</td><td colspan="2">TSCCA</td></tr><tr><td/><td>orig.</td><td>proc.</td><td>orig.</td><td>proc.</td></tr><tr><td/><td colspan="2">2012 38.03 37.66</td><td colspan="2">44.51 44.63</td></tr><tr><td/><td colspan="2">2013 37.47 36.85</td><td colspan="2">43.21 42.74</td></tr><tr><td/><td colspan="2">2014 46.06 48.32</td><td colspan="2">52.85 52.87</td></tr><tr><td/><td colspan="2">2015 47.82 51.76</td><td colspan="2">56.22 56.14</td></tr><tr><td/><td colspan="2">SICK 51.58 51.76</td><td colspan="2">56.15 56.11</td></tr><tr><td/><td colspan="2">all 43.48 44.67</td><td colspan="2">50.01 50.23</td></tr><tr><td colspan="5">Table 13: Before-After results (x100) on the semantic textual similarity tasks.</td></tr><tr><td>dim</td><td colspan="4">300 400 500 600 700 800 900 1000</td></tr><tr><td colspan="5">avg. v(w) 2 4.51 5.17 5.91 6.22 6.49 6.73 6.95 7.15</td></tr><tr><td>µ 2</td><td colspan="4">1.74 1.76 1.77 1.78 1.79 1.80 1.81 1.83</td></tr></table>

.57 74.72 75.64 79.87 77.72 81.97 77.59 80.7 WS 70.25 71.95 70.8 72.88 70.39 72.73 71.64 74.04 RW 46.25 49.11 45.97 47.63 46.6 48.59 45.7 47.81 MEN 75.66 77.59 76.07 77.89 75.9 78.15 75.88 78.15 Mturk 75.66 77.59 67.68 68.11 66.89 68.25 67.6 67.87 SimLex 34.02 36.19 35.17 37.1 35.73 37.65 35.76 38.04 SimVerb 22.22 24.98 22.91 25.32 23.03 25.82 23.35 25.97 SimLex 35.3 37.59 36.54 37.85 36.62 38.44 36.67 38.58 SimVerb 22.81 25.6 23.48 25.57 23.68 25.76 23.24 26.58

Table content: <table><tr><td>Dim</td><td>orig.</td><td>300 proc.</td><td>orig.</td><td colspan="2">400 proc.</td><td>orig.</td><td>500 proc.</td><td>orig.</td><td>600 proc.</td></tr><tr><td colspan="2">RG65 73Dim orig.</td><td>700 proc.</td><td>orig.</td><td colspan="2">800 proc.</td><td>orig.</td><td>900 proc.</td><td colspan="2">1000 orig. proc.</td></tr><tr><td colspan="10">RG65 77.3 81.07 77.52 81.07 79.75 82.34 78.18 79.07</td></tr><tr><td colspan="10">WS 70.31 73.02 71.52 74.65 71.19 73.06 71.5 74.78</td></tr><tr><td colspan="5">RW 45.86 48.4 44.96</td><td>49</td><td colspan="4">44.44 49.22 44.5 49.03</td></tr><tr><td colspan="10">MEN 75.84 78.21 75.84 77.96 76.16 78.35 76.72 78.1</td></tr><tr><td colspan="5">Mturk 67.47 67.79 67.67</td><td>68</td><td colspan="4">67.98 68.87 68.34 69.44</td></tr></table>

Before-After results (x100) on word similarity task on seven datasets.

Table content: <table><tr><td>Dim</td><td>orig.</td><td>300 proc.</td><td>orig.</td><td colspan="2">400 proc.</td><td>orig.</td><td>500 proc.</td><td>orig.</td><td>600 proc.</td></tr><tr><td colspan="10">ap 46.1 48.61 42.57 45.34 46.85 50.88 40.3 45.84</td></tr><tr><td colspan="10">esslli 68.18 72.73 64.2 82.72 64.2 65.43 65.91 72.73</td></tr><tr><td colspan="5">battig 71.6 77.78 68.18</td><td>75</td><td colspan="4">68.18 70.45 46.91 66.67</td></tr><tr><td>Dim</td><td>orig.</td><td>700 proc.</td><td>orig.</td><td colspan="2">800 proc.</td><td>orig.</td><td>900 proc.</td><td colspan="2">1000 orig. proc.</td></tr><tr><td colspan="10">ap 38.04 41.31 34.76 39.8 34.76 27.46 27.96 28.21</td></tr><tr><td colspan="10">esslli 54.55 54.55 68.18 56.82 72.73 72.73 52.27 52.27</td></tr><tr><td colspan="10">battig 62.96 66.67 67.9 69.14 49.38 59.26 51.85 46.91</td></tr></table>

Before-After results (x100) on the categorization task. .94 61.02 68.38 68.34 60.47 60.30 67.56 67.30 sem. 77.24 77.26 77.24 77.35 76.76 76.90 76.71 76.51 all. 68.36 68.41 68.38 68.50 67.91 67.67 67.56 67.30

Table content: <table><tr><td>Dim</td><td>orig.</td><td>300 proc.</td><td>orig.</td><td>400 proc.</td><td>orig.</td><td>500 proc.</td><td>orig.</td><td>600 proc.</td></tr><tr><td colspan="9">syn. 60.48 60.52 61.61 61.45 60.93 60.84 61.66 61.57</td></tr><tr><td colspan="9">sem. 74.51 74.54 77.11 77.36 76.39 76.89 77.28 77.61</td></tr><tr><td colspan="9">all. 66.86 66.87 68.66 68.69 67.88 68.11 68.77 68.81</td></tr><tr><td>Dim</td><td>orig.</td><td>700 proc.</td><td>orig.</td><td>800 proc.</td><td>orig.</td><td>900 proc.</td><td colspan="2">1000 orig. proc.</td></tr><tr><td colspan="2">syn. 60</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Before-After results (x100) on the word analogy task.

Table content: <table/>

Before-After results (x100) on the semantic textual similarity tasks.

Table content: <table><tr><td>c l</td><td>Train</td><td>Test</td></tr><tr><td>MR 2 20</td><td colspan="2">10,662 10-fold cross validation</td></tr><tr><td>SUBJ 2 23</td><td colspan="2">10,000 10-fold cross validation</td></tr><tr><td>TREC 6 10</td><td>5,952</td><td>500</td></tr><tr><td>SST 5 18</td><td colspan="2">11,855 2,210</td></tr><tr><td colspan="3">IMDb 2 100 25,000 25,000</td></tr></table>

Statistics for the five datasets after tokenization: c represents the number of classes; l represents the average sentence length; Train represents the size of the training set; and Test represent the size of the test set.

Table content: <table/>

that both p(w) and p(w, w ) are correlated with v(w) , formally

Table content: <table><tr><td>log p(w) →</td><td>v(w) 2 2d</td><td colspan="2">-log Z, as |V| → ∞,</td><td>(12)</td></tr><tr><td>log p(w, w ) →</td><td colspan="2">v(w) + v(w ) 2 2d</td><td>-log Z, as |V| → ∞.</td></tr></table>

Peer review (iclr2018):
 This paper provides theoretical and empirical motivations for removing the top few principle components of commonly-used word embeddings. The paper is well-written and I enjoyed reading it. However, it does not explain how significant this result is beyond that of (Bullinaria and Levy, 2012), who also removed the top N dimensions when benchmarking SVD-factorized word embeddings. From what I can see, this paper provides a more detailed explanation of the phenomenon ("why" it works), which is supported with both theoretical results and a series of empirical analyses, as well as "updating" the benchmarks and methods from the pre-neural era. Although this contribution is relatively incremental, I find the depth of this work very interesting, and I think future work could perhaps rely on these insights to create better embedding algorithms that directly enforce isotropy. I have two concerns regarding the empirical section, which may be resolvable fairly quickly: 1) Are the embedding vectors L2 normalized before using them in each task? This is known to significantly affect performance. I am curious whether removing the top PCs is redundant or not given L2 normalization. 2) Most of the benchmarks used in this paper are "toy" tasks. As Schnabel et al (2015) and Tsvetkov et al (2015) showed, there is often little correlation between success on these benchmarks and improvement of downstream NLP tasks. I would like to measure the change in performance on a major NLP task that heavily relies on pre-trained word embeddings such as SQuAD. Minor Comments: * The last sentence in the first paragraph ("The success comes from the geometry of the representations...") is not true; the success stems from the ability to capture lexical similarity. Levy and Goldberg (2014) showed that searching for the closest word vector to (king - man + woman) is equivalent to optimizing a linear combination of 3 similarity terms [+(x,king), -(x,man), +(x, woman)]. This explanation was further demonstrated by Linzen (2016) who showed that even when removing the negative term (x, man), many analogies can still be solved, i.e. by looking for a word that is similar both to "king" and to "woman". Add to that the fact that the analogy trick works best when the vectors are L2 normalized; if they are all on the unit sphere, what is the geometric interpretation of (king - man + woman), which is not on the unit sphere? I suggest removing this sentence and other references to linguistic regularities from this paper, since they are controversial at best, and distract from the main findings. * This is also related to Bullinaria and Levy*s (2012) finding that downweighting the eigenvalue matrix in SVD-based methods improves their performance. Levy et al (2015) showed that keeping the original eigenvalues can actually degenerate SVD-based embeddings. Perhaps there is a connection to the findings in this paper?