Factors Influencing the Surprising Instability of Word Embeddings

Abstract: Despite the recent popularity of word embedding methods, there is only a small body of work exploring the limitations of these representations. In this paper, we consider one aspect of embedding spaces, namely their stability. We show that even relatively high frequency words (100-200 occurrences) are often unstable. We provide empirical evidence for how various factors contribute to the stability of word embeddings, and we analyze the effects of stability on downstream tasks.

Word embeddings are low-dimensional, dense vector representations that capture semantic properties of words. Recently, they have gained tremendous popularity in Natural Language Processing (NLP) and have been used in tasks as diverse as text similarity BIBREF0 , part-of-speech tagging BIBREF1 , sentiment analysis BIBREF2 , and machine translation BIBREF3 . Although word embeddings are widely used across NLP, their stability has not yet been fully evaluated and understood. In this paper, we explore the factors that play a role in the stability of word embeddings, including properties of the data, properties of the algorithm, and properties of the words. We find that word embeddings exhibit substantial instabilities, which can have implications for downstream tasks.
Using the overlap between nearest neighbors in an embedding space as a measure of stability (see sec:definingStability below for more information), we observe that many common embedding spaces have large amounts of instability. For example, Figure FIGREF1 shows the instability of the embeddings obtained by training word2vec on the Penn Treebank (PTB) BIBREF4 . As expected, lower frequency words have lower stability and higher frequency words have higher stability. What is surprising however about this graph is the medium-frequency words, which show huge variance in stability. This cannot be explained by frequency, so there must be other factors contributing to their instability.
In the following experiments, we explore which factors affect stability, as well as how this stability affects downstream tasks that word embeddings are commonly used for. To our knowledge, this is the first study comprehensively examining the factors behind instability.
There has been much recent interest in the applications of word embeddings, as well as a small, but growing, amount of work analyzing the properties of word embeddings.
Here, we explore three different embedding methods: PPMI BIBREF6 , word2vec BIBREF7 , and GloVe BIBREF8 . Various aspects of the embedding spaces produced by these algorithms have been previously studied. Particularly, the effect of parameter choices has a large impact on how all three of these algorithms behave BIBREF9 . Further work shows that the parameters of the embedding algorithm word2vec influence the geometry of word vectors and their context vectors BIBREF10 . These parameters can be optimized; Hellrich and Hahn ( BIBREF11 ) posit optimal parameters for negative sampling and the number of epochs to train for. They also demonstrate that in addition to parameter settings, word properties, such as word ambiguity, affect embedding quality.
In addition to exploring word and algorithmic parameters, concurrent work by Antoniak and Mimno ( BIBREF12 ) evaluates how document properties affect the stability of word embeddings. We also explore the stability of embeddings, but focus on a broader range of factors, and consider the effect of stability on downstream tasks. In contrast, Antoniak and Mimno focus on using word embeddings to analyze language BIBREF13 , rather than to perform tasks.
At a higher level of granularity, Tan et al. ( BIBREF14 ) analyze word embedding spaces by comparing two spaces. They do this by linearly transforming one space into another space, and they show that words have different usage properties in different domains (in their case, Twitter and Wikipedia).
Finally, embeddings can be analyzed using second-order properties of embeddings (e.g., how a word relates to the words around it). Newman-Griffis and Fosler-Lussier ( BIBREF15 ) validate the usefulness of second-order properties, by demonstrating that embeddings based on second-order properties perform as well as the typical first-order embeddings. Here, we use second-order properties of embeddings to quantify stability.
We define stability as the percent overlap between nearest neighbors in an embedding space. Given a word INLINEFORM0 and two embedding spaces INLINEFORM1 and INLINEFORM2 , take the ten nearest neighbors of INLINEFORM3 in both INLINEFORM4 and INLINEFORM5 . Let the stability of INLINEFORM6 be the percent overlap between these two lists of nearest neighbors. 100% stability indicates perfect agreement between the two embedding spaces, while 0% stability indicates complete disagreement. In order to find the ten nearest neighbors of a word INLINEFORM7 in an embedding space INLINEFORM8 , we measure distance between words using cosine similarity. This definition of stability can be generalized to more than two embedding spaces by considering the average overlap between two sets of embedding spaces. Let INLINEFORM12 and INLINEFORM13 be two sets of embedding spaces. Then, for every pair of embedding spaces INLINEFORM14 , where INLINEFORM15 and INLINEFORM16 , take the ten nearest neighbors of INLINEFORM17 in both INLINEFORM18 and INLINEFORM19 and calculate percent overlap. Let the stability be the average percent overlap over every pair of embedding spaces INLINEFORM20 .
Consider an example using this metric. Table TABREF4 shows the top ten nearest neighbors for the word international in three randomly initialized word2vec embedding spaces trained on the NYT Arts domain (see Section SECREF11 for a description of this corpus). These models share some similar words, such as metropolitan and national, but there are also many differences. On average, each pair of models has four out of ten words in common, so the stability of international across these three models is 40%.
The idea of evaluating ten best options is also found in other tasks, like lexical substitution BIBREF16 and word association BIBREF17 , where the top ten results are considered in the final evaluation metric. To give some intuition for how changing the number of nearest neighbors affects our stability metric, consider Figure FIGREF5 . This graph shows how the stability of GloVe changes with the frequency of the word and the number of neighbors used to calculate stability; please see the figure caption for a more detailed explanation of how this graph is structured. Within each frequency bucket, the stability is consistent across varying numbers of neighbors. Ten nearest neighbors performs approximately as well as a higher number of nearest neighbors (e.g., 100). We see this pattern for low frequency words as well as for high frequency words. Because the performance does not change substantially by increasing the number of nearest neighbors, it is computationally less intensive to use a small number of nearest neighbors. We choose ten nearest neighbors as our metric throughout the rest of the paper.
As we saw in Figure FIGREF1 , embeddings are sometimes surprisingly unstable. To understand the factors behind the (in)stability of word embeddings, we build a regression model that aims to predict the stability of a word given: (1) properties related to the word itself; (2) properties of the data used to train the embeddings; and (3) properties of the algorithm used to construct these embeddings. Using this regression model, we draw observations about factors that play a role in the stability of word embeddings.
We use ridge regression to model these various factors BIBREF18 . Ridge regression regularizes the magnitude of the model weights, producing a more interpretable model than non-regularized linear regression. This regularization mitigates the effects of multicollinearity (when two features are highly correlated). Specifically, given INLINEFORM0 ground-truth data points with INLINEFORM1 extracted features per data point, let INLINEFORM2 be the features for sample INLINEFORM3 and let INLINEFORM4 be the set of labels. Then, ridge regression learns a set of weights INLINEFORM5 by minimizing the least squares function with INLINEFORM6 regularization, where INLINEFORM7 is a regularization constant: INLINEFORM8 
We set INLINEFORM0 . In addition to ridge regression, we tried non-regularized linear regression. We obtained comparable results, but many of the weights were very large or very small, making them hard to interpret.
The goodness of fit of a regression model is measured using the coefficient of determination INLINEFORM0 . This measures how much variance in the dependent variable INLINEFORM1 is captured by the independent variables INLINEFORM2 . A model that always predicts the expected value of INLINEFORM3 , regardless of the input features, will receive an INLINEFORM4 score of 0. The highest possible INLINEFORM5 score is 1, and the INLINEFORM6 score can be negative.
Given this model, we create training instances by observing the stability of a large number of words across various combinations of two embedding spaces. Specifically, given a word INLINEFORM0 and two embedding spaces INLINEFORM1 and INLINEFORM2 , we encode properties of the word INLINEFORM3 , as well as properties of the datasets and the algorithms used to train the embedding spaces INLINEFORM4 and INLINEFORM5 . The target value associated with this features is the stability of the word INLINEFORM6 across embedding spaces INLINEFORM7 and INLINEFORM8 . We repeat this process for more than 2,500 words, several datasets, and three embedding algorithms.
Specifically, we consider all the words present in all seven of the data domains we are using (see Section SECREF11 ), 2,521 words in total. Using the feature categories described below, we generate a feature vector for each unique word, dataset, algorithm, and dimension size, resulting in a total of 27,794,025 training instances. To get good average estimates for each embedding algorithm, we train each embedding space five times, randomized differently each time (this does not apply to PPMI, which has no random component). We then train a ridge regression model on these instances. The model is trained to predict the stability of word INLINEFORM0 across embedding spaces INLINEFORM1 and INLINEFORM2 (where INLINEFORM3 and INLINEFORM4 are not necessarily trained using the same algorithm, parameters, or training data). Because we are using this model to learn associations between certain features and stability, no test data is necessary. The emphasis is on the model itself, not on the model's performance on a specific task.
We describe next each of the three main categories of factors examined in the model. An example of these features is given in Table TABREF7 .
We encode several features that capture attributes of the word INLINEFORM0 . First, we use the primary and secondary part-of-speech (POS) of the word. Both of these are represented as bags-of-words of all possible POS, and are determined by looking at the primary (most frequent) and secondary (second most frequent) POS of the word in the Brown corpus BIBREF20 . If the word is not present in the Brown corpus, then all of these POS features are set to zero.
To get a coarse-grained representation of the polysemy of the word, we consider the number of different POS present. For a finer-grained representation, we use the number of different WordNet senses associated with the word BIBREF21 , BIBREF22 .
We also consider the number of syllables in a word, determined using the CMU Pronuncing Dictionary BIBREF23 . If the word is not present in the dictionary, then this is set to zero.
Data features capture properties of the training data (and the word in relation to the training data). For this model, we gather data from two sources: New York Times (NYT) BIBREF24 and Europarl BIBREF25 . Overall, we consider seven domains of data: (1) NYT - U.S., (2) NYT - New York and Region, (3) NYT - Business, (4) NYT - Arts, (5) NYT - Sports, (6) All of the data from domains 1-5 (denoted “All NYT"), and (7) All of English Europarl. Table TABREF10 shows statistics about these datasets. The first five domains are chosen because they are the top five most common categories of news articles present in the NYT corpus. They are smaller than “All NYT" and Europarl, and they have a narrow topical focus. The “All NYT" domain is more diverse topically and larger than the first five domains. Finally, the Europarl domain is the largest domain, and it is focused on a single topic (European Parliamentary politics). These varying datasets allow us to consider how data-dependent properties affect stability.
We use several features related to domain. First, we consider the raw frequency of word INLINEFORM0 in both the domain of data used for embedding space INLINEFORM1 and the domain of data for space INLINEFORM2 . To make our regression model symmetric, we effectively encode three features: the higher raw frequency (between the two), the lower raw frequency, and the absolute difference in raw frequency.
We also consider the vocabulary size of each corpus (again, symmetrically) and the percent overlap between corpora vocabulary, as well as the domain of each of the two corpora, represented as a bag-of-words of domains. Finally, we consider whether the two corpora are from the same domain.
Our final data-level features explore the role of curriculum learning in stability. It has been posited that the order of the training data affects the performance of certain algorithms, and previous work has shown that for some neural network-based tasks, a good training data order (curriculum learning strategy) can improve performance BIBREF26 . Curriculum learning has been previously explored for word2vec, where it has been found that optimizing training data order can lead to small improvements on common NLP tasks BIBREF1 . Of the embedding algorithms we consider, curriculum learning only affects word2vec. Because GloVe and PPMI use the data to learn a complete matrix before building embeddings, the order of the training data will not affect their performance. To measure the effects of training data order, we include as features the first appearance of word INLINEFORM0 in the dataset for embedding space INLINEFORM1 and the first appearance of INLINEFORM2 in the dataset for embedding space INLINEFORM3 (represented as percentages of the total number of training sentences). We further include the absolute difference between these percentages.
In addition to word and data properties, we encode features about the embedding algorithms. These include the different algorithms being used, as well as the different parameter settings of these algorithms. Here, we consider three embedding algorithms, word2vec, GloVe, and PPMI. The choice of algorithm is represented in our feature vector as a bag-of-words.
PPMI creates embeddings by first building a positive pointwise mutual information word-context matrix, and then reducing the dimensionality of this matrix using SVD BIBREF6 . A more recent word embedding algorithm, word2vec (skip-gram model) BIBREF7 uses a shallow neural network to learn word embeddings by predicting context words. Another recent method for creating word embeddings, GloVe, is based on factoring a matrix of ratios of co-occurrence probabilities BIBREF8 .
For each algorithm, we choose common parameter settings. For word2vec, two of the parameters that need to be chosen are window size and minimum count. Window size refers to the maximum distance between the current word and the predicted word (e.g., how many neighboring words to consider for each target word). Any word appearing less than the minimum count number of times in the corpus is discarded and not considered in the word2vec algorithm. For both of these features, we choose standard parameter settings, namely, a window size of 5 and a minimum count of 5. For GloVe, we also choose standard parameters. We use 50 iterations of the algorithm for embedding dimensions less than 300, and 100 iterations for higher dimensions.
We also add a feature reflecting the embedding dimension, namely one of five embedding dimensions: 50, 100, 200, 400, or 800.
Overall, the regression model achieves a coefficient of determination ( INLINEFORM0 ) score of 0.301 on the training data, which indicates that the regression has learned a linear model that reasonably fits the training data given. Using the regression model, we can analyze the weights corresponding to each of the features being considered, shown in Table TABREF14 . These weights are difficult to interpret, because features have different distributions and ranges. However, we make several general observations about the stability of word embeddings.
Observation 1. Curriculum learning is important. This is evident because the top two features (by magnitude) of the regression model capture where the word first appears in the training data. Figure FIGREF15 shows trends between training data position and stability in the PTB. This figure contrasts word2vec with GloVe (which is order invariant).
To further understand the effect of curriculum learning on the model, we train a regression model with all of the features except the curriculum learning features. This model achieves an INLINEFORM0 score of 0.291 (compared to the full model's score of 0.301). This indicates that curriculum learning is a factor in stability.
Observation 2. POS is one of the biggest factors in stability. Table TABREF14 shows that many of the top weights belong to POS-related features (both primary and secondary POS). Table TABREF18 compares average stabilities for each primary POS. Here we see that the most stable POS are numerals, verbs, and determiners, while the least stable POS are punctuation marks, adpositions, and particles.
Observation 3. Stability within domains is greater than stability across domains. Table TABREF14 shows that many of the top factors are domain-related. Figure FIGREF19 shows the results of the regression model broken down by domain. This figure shows the highest stabilities appearing on the diagonal of the matrix, where the two embedding spaces both belong to the same domain. The stabilities are substantially lower off the diagonal.
Figure FIGREF19 also shows that “All NYT" generalizes across the other NYT domains better than Europarl, but not as well as in-domain data (“All NYT" encompasses data from US, NY, Business, Arts, and Sports). This is true even though Europarl is much larger than “All NYT".
Observation 4. Overall, GloVe is the most stable embedding algorithm. This is particularly apparent when only in-domain data is considered, as in Figure FIGREF19 . PPMI achieves similar stability, while word2vec lags considerably behind.
To further compare word2vec and GloVe, we look at how the stability of word2vec changes with the frequency of the word and the number of neighbors used to calculate stability. This is shown in Figure FIGREF20 and is directly comparable to Figure FIGREF5 . Surprisingly, the stability of word2vec varies substantially with the frequency of the word. For lower-frequency words, as the number of nearest neighbors increases, the stability increases approximately exponentially. For high-frequency words, the lowest and highest number of nearest neighbors show the greatest stability. This is different than GloVe, where stability remains reasonably constant across word frequencies, as shown in Figure FIGREF5 . The behavior we see here agrees with the conclusion of BIBREF10 , who find that GloVe exhibits more well-behaved geometry than word2vec.
Observation 5. Frequency is not a major factor in stability. To better understand the role that frequency plays in stability, we run separate ablation experiments comparing regression models with frequency features to regression models without frequency features. Our current model (using raw frequency) achieves an INLINEFORM0 score of 0.301. Comparably, a model using the same features, but with normalized instead of raw frequency, achieves a score of 0.303. Removing frequency from either regression model gives a score of 0.301. This indicates that frequency is not a major factor in stability, though normalized frequency is a larger factor than raw frequency.
Finally, we look at regression models using only frequency features. A model using only raw frequency features has an INLINEFORM0 score of 0.008, while a model with only normalized frequency features has an INLINEFORM1 score of 0.0059. This indicates that while frequency is not a major factor in stability, it is also not negligible. As we pointed out in the introduction, frequency does correlate with stability (Figure FIGREF1 ). However, in the presence of all of these other features, frequency becomes a minor factor.
Word embeddings are used extensively as the first stage of neural networks throughout NLP. Typically, embeddings are initalized based on a vector trained with word2vec or GloVe and then further modified as part of training for the target task. We study two downstream tasks to see whether stability impacts performance.
Since we are interested in seeing the impact of word vector stability, we choose tasks that have an intuitive evaluation at the word level: word similarity and POS tagging.
To model word similarity, we use 300-dimensional word2vec embedding spaces trained on the PTB. For each pair of words, we take the cosine similarity between those words averaged over ten randomly initialized embedding spaces. We consider three datasets for evaluating word similarity: WS353 (353 pairs) BIBREF27 , MTurk287 (287 pairs) BIBREF28 , and MTurk771 (771 pairs) BIBREF29 . For each dataset, we normalize the similarity to be in the range INLINEFORM0 , and we take the absolute difference between our predicted value and the ground-truth value. Figure FIGREF22 shows the results broken down by stability of the two words (we always consider Word 1 to be the more stable word in the pair). Word similarity pairs where one of the words is not present in the PTB are omitted.
We find that these word similarity datasets do not contain a balanced distribution of words with respect to stability; there are substantially more unstable words than there are stable words. However, we still see a slight trend: As the combined stability of the two words increases, the average absolute error decreases, as reflected by the lighter color of the cells in Figure FIGREF22 while moving away from the (0,0) data point.
Part-of-speech (POS) tagging is a substantially more complicated task than word similarity. We use a bidirectional LSTM implemented using DyNet BIBREF30 . We train nine sets of 128-dimensional word embeddings with word2vec using different random seeds. The LSTM has a single layer and 50-dimensional hidden vectors. Outputs are passed through a tanh layer before classification. To train, we use SGD with a learning rate of 0.1, an input noise rate of 0.1, and recurrent dropout of 0.4.
This simple model is not state-of-the-art, scoring 95.5% on the development set, but the word vectors are a central part of the model, providing a clear signal of their impact. For each word, we group tokens based on stability and frequency. Figure FIGREF24 shows the results. Fixing the word vectors provides a clearer pattern in the results, but also leads to much worse performance: 85.0% on the development set. Based on these results, it seems that training appears to compensate for stability. This hypothesis is supported by Figure FIGREF24 , which shows the similarity between the original word vectors and the shifted word vectors produced by the training. In general, lower stability words are shifted more during training.
Understanding how the LSTM is changing the input embeddings is useful information for tasks with limited data, and it could allow us to improve embeddings and LSTM training for these low-resource tasks.
Word embeddings are surprisingly variable, even for relatively high frequency words. Using a regression model, we show that domain and part-of-speech are key factors of instability. Downstream experiments show that stability impacts tasks using embedding-based features, though allowing embeddings to shift during training can reduce this effect. In order to use the most stable embedding spaces for future tasks, we recommend either using GloVe or learning a good curriculum for word2vec training data. We also recommend using in-domain embeddings whenever possible.
The code used in the experiments described in this paper is publicly available from http://lit.eecs.umich.edu/downloads.html.
We would like to thank Ben King and David Jurgens for helpful discussions about this paper, as well as our anonymous reviewers for useful feedback. This material is based in part upon work supported by the National Science Foundation (NSF #1344257) and the Michigan Institute for Data Science (MIDAS). Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or MIDAS.

Figure and table captions:
Figure 1: Stability of word2vec as a property of frequency in the PTB. Stability is measured across ten randomized embedding spaces trained on the training portion of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (x-axis), and each column (frequency bucket) is normalized.
Table 1: Top ten most similar words for the word international in three randomly intialized word2vec models trained on the NYT Arts Domain. Words in all three lists are in bold; words in only two of the lists are italicized.
Figure 2: Stability of GloVe on the PTB. Stability is measured across ten randomized embedding spaces trained on the training data of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (left y-axis) and stability is determined using a varying number of nearest neighbors for each frequency bucket (right yaxis). Each row is normalized, and boxes with more than 0.01 of the row’s mass are outlined.
Table 2: Consider the word international in two embedding spaces. Suppose embedding spaceA is trained using word2vec (embedding dimension 100) on the NYT Arts domain, and embedding space B is trained using PPMI (embedding dimension 100) on Europarl. This table summarizes the resulting features for this word across the two embedding spaces.
Table 3: Dataset statistics.
Table 4: Regression weights with a magnitude greater than 0.1, sorted by magnitude.
Figure 3: Stability of both word2vec and GloVe as properties of the starting word position in the training data of the PTB. Stability is measured across ten randomized embedding spaces trained on the training data of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Boxes with more than 0.02% of the total vocabulary mass are outlined.
Table 5: Percent stability broken down by part-ofspeech, ordered by decreasing stability.
Figure 4: Percent stability broken down by domain.
Figure 5: Percent stability broken down between algorithm (in-domain data only).
Figure 7: Absolute error for word similarity.
Figure 6: Stability of word2vec on the PTB. Stability is measured across ten randomized embedding spaces trained on the training data of the PTB (determined using language modeling splits (Mikolov et al., 2010)). Each word is placed in a frequency bucket (left y-axis) and stability is determined using a varying number of nearest neighbors for each frequency bucket (right yaxis). Each row is normalized, and boxes with more than 0.01 of the row’s mass are outlined.
Figure 8: Results for POS tagging. (a) and (b) show average POS tagging error divided by the number of tokens (darker is more errors) while either keeping word vectors fixed or not during training. (c) shows word vector shift, measured as cosine similarity between initial and final vectors. In all graphs, words are bucketed by frequency and stability.

Questions about this paper:
How is unstability defined?

Answers:
An embedding is unstable if it has a low number of nearest neighbor embeddings of the words within the same frequency bucket.