{
  "name" : "1604.01692.pdf",
  "metadata" : {
    "source" : "CRF",
    "title" : "An Ensemble Method to Produce High-Quality Word Embeddings",
    "authors" : [ "Robert Speer", "Joshua Chin" ],
    "emails" : [ "rspeer@luminoso.com", "joshuarchin@gmail.com" ],
    "sections" : [ {
      "heading" : "1 Introduction",
      "text" : "Vector space models are an effective way to express the meanings of natural-language terms in a computational system.  These models are created using machine-learning techniques that represent words or phrases as vectors in a high-dimensional space, such that the cosine similarity of any two terms corresponds to their semantic similarity.\nThese vectors, referred to as the embeddings of the terms in the vector space, can also be used as an input to further steps of machine learning.  When algorithms expect dense vectors as input, embeddings provide a representation that is both more compact and more informative than the “one-hot” representation in which every term in the vocabulary gets its own dimension.\nThis kind of vector space has been used in applications such as search, topic detection, and text classification, dating back to the introduction of latent semantic analysis (Deerwester et al.,  1990).  In recent years, there has been a surge of interest in natural-language embeddings, as machine-learning techniques such as Mikolov et al. ( 2013)’s word2vec and Pennington et al. ( 2014)’s GloVe have begun to show dramatic improvements.  Word embeddings are often suggested as an initialization for more complex methods, such as the sentence encodings of Kiros et al. ( 2015).\nFaruqui et al. ( 2015) introduced a technique known as “retrofitting”, which combines embeddings learned from the distributional semantics of unstructured text with a source of structured connections between words.  The combined embedding achieves performance on word-similarity evaluations superior to either source individually.\nHere, we build on the retrofitting process to produce a high-quality space of word embeddings.  We extend existing techniques in the following ways:\n• We modify the retrofitting algorithm, making it not depend on the row order of its input matrix, and allowing it to propagate over the union of the vocabularies.  This allows retrofitting to benefit from structured links outside the original vocabulary, such as translations into other languages.  We call this procedure “expanded retrofitting”.\n• We include ConceptNet, a Linked Open Data semantic network that expresses many kinds of relationships between words in many lan-\nar X\niv :1\n60 4.\n01 69\n2v 1\n[ cs\n. C L\n] 6\nA pr\n2 01\nguages, as a source of structured connections between words.\n• We align English terms from different sources using a lemmatizer and a heuristic for merging together multiple term vectors.\n• We fill gaps when aligning the two distributional-semantics sources (GloVe and word2vec) using a locally linear interpolation.\n• We re-scale the distributional-semantics features using L1 normalization.\nWhen we use this process to combine word2vec, GloVe, PPDB, and ConceptNet, this process produces a space of multilingual term embeddings we call the “ConceptNet vector ensemble” that achieves state-of-the-art performance on wordsimilarity evaluations1 over both common and rare words." 
    }, {
      "heading" : "1.1 Related Work",
      "text" : "Agirre et al. ( 2009) observes that distributional similarity and structured knowledge can be combined for a benefit exceeding what each would achieve alone, particularly by extending the vocabulary.  Their system uses a similarity measure over WordNet, and uses distributional similarities to recognize words outside of WordNet’s vocabulary.\nLevy et al. ( 2015) surveys modern methods of distributional similarity and experiments with training them on specific data while varying their parameters.  They compare word2vec and GloVe, tune their hyperparameters in a way that particularly improves word2vec, then proposes a method based on the SVD of the Pointwise Mutual Information matrix that outperforms both.  We use Levy’s results as a point of comparison here.\nAutoExtend (Rothe and Schütze, 2015) is a system with similar methods to ours: it extends word2vec embeddings to cover all the word senses and synsets of WordNet by propagating information\n1Some methods and evaluations (Agirre et al.,  2009) distinguish word similarity from word relatedness. “ Coffee” and “mug”, for example, are quite related, but not actually similar because coffee is not like a mug.  In this paper, however, we conflate similarity and relatedness into the same metric, as most evaluations do.\nover edges, thus combining distributional and structured data after the fact.  The primary goal of AutoExtend is word sense disambiguation, and as such it is optimized for and evaluated on WSD tasks.  Our ensemble aims to extend and improve a vocabulary of undisambiguated words, so there is no direct comparison between AutoExtend’s results and ours." 
    }, {
      "heading" : "2 Knowledge Sources",
      "text" : ""
    }, {
      "heading" : "2.1 ConceptNet and PPDB",
      "text" : "ConceptNet (Speer and Havasi, 2012) is a semantic network of terms connected by labeled relations.  Its terms are words or multiple-word phrases in a variety of natural languages.  For continuity with previous work, these terms are often referred to as concepts.\nConceptNet originated as a machine-parsed version of the early crowd-sourcing project called Open Mind Common Sense (OMCS) (Singh et al.,  2002), and has expanded to include several other data sources, both crowd-sourced and expert-created, by unifying their vocabularies into a single representation.  ConceptNet now includes representations of WordNet (Miller et al.,  1998), Wiktionary (Wiktionary, 2014), and JMDict (Breen, 2004), as well as data from “games with a purpose” in multiple languages (von Ahn et al.,  2006; Kuo et al.,  2009; Nakahara and Yamada, 2011).  We choose not to include ConceptNet’s alignment to DBPedia (Auer et al.,  2007) here, as DBPedia focuses on relations between specific named entities, which do not help with general word similarity.2\nPPDB (Ganitkevitch et al.,  2013) is another resource that is useful for learning about word similarity, providing different information from ConceptNet.  It lists pairs of words that are translated to the same word in parallel corpora, particularly in documents of the European Parliament.  PPDB is used as an external knowledge source by Faruqui et al. ( 2015), so we have evaluated the effect of adding it to our ensemble as well.  As it seems to have a small beneficial effect, we include it as part of the full en-\n2 Given different goals – such as achieving a high score on Mikolov et al. ( 2013)’s analogy evaluation that tests for implicit relations such as “A is the CEO of company B” – including an appropriate representation of DBPedia would of course be helpful.\nsemble." 
    }, {
      "heading" : "2.2 word2vec and GloVe",
      "text" : "word2vec and GloVe are two current systems that learn vector representations of words according to their distributional semantics.  Given a large text corpus, they produce vectors representing similarities in how the words co-occur with other words.\nMikolov et al. ( 2013) described a system of distributional word embeddings called Skip-Grams with Negative Sampling (SGNS), which is more popularly known by the name of its software implementation, word2vec. ( The word2vec software also implements another representation, Continuous Bag-ofWords or CBOW, which is less often used for word similarity.)\nIn SGNS, a neural network with one hidden layer is trained to recognize words that are likely to appear near each other.  Its goal is to output a high value when given examples of co-occurrences that appear in the data, and a low value for negative examples where one word is replaced by a random word.  The loss function is weighted by the frequencies of the words involved and the distance between them in the data.  The word2vec software3 comes with SGNS embeddings of text from Google News.\nGloVe (Pennington et al.,  2014) is an unsupervised learning algorithm that learns a set of word embeddings such that the dot product of two words’ embeddings is approximately equal to the logarithm of their co-occurrence count.  The algorithm operates on a global word-word co-occurrence matrix, and solves an optimization problem to learn a vector for each word, a separate vector for each context (although the contexts are also words), and a bias value for each word and each context.  Only the word vectors are used for computing similarity.\nThe embeddings that GloVe learns from data sources such as the Common Crawl4 are distributed on the GloVe web page5.  Here we evaluate two downloadable sets of GloVe 1.2 embeddings, built from 42 billion and 840 billion tokens of the Common Crawl, respectively.\nThere is some debate about whether GloVe or word2vec is better at representing word meanings\n3https://code.google.com/p/word2vec/ 4http://commoncrawl.org/ 5http://nlp.stanford.edu/projects/glove/\nin general.  GloVe is presented by Pennington et al. ( 2014) as performing better than word2vec on wordsimilarity tasks, but Levy et al. ( 2015) finds that word2vec performs better with an optimized setting of hyperparameters than GloVe does, when retrained with a particular corpus.\nIn this paper, we focus only on the downloadable sets of term embeddings that the GloVe and word2vec projects provide, not on re-running them with tuned hyperparameters.  Using this data makes it possible to reproduce their results and compare directly to them, even when their preferred input data is not available.  We find that we can get very good results derived from the downloadable embeddings, and that GloVe’s downloadable embeddings outperform word2vec’s in this case, but a combination of them can perform even better." 
    }, {
      "heading" : "3 Methods",
      "text" : "Faruqui et al. ( 2015) introduced the “retrofitting” procedure, which adjusts dense matrices of embeddings (such as the GloVe output) to take into account external knowledge from a sparse semantic network.  They tried various sources of external knowledge, and the one that was most helpful to GloVe was PPDB.  We found using ConceptNet to be more effective, and that further marginal improvements could be achieved on some evaluations by combining ConceptNet and PPDB.\nOur goal is to create a 300-dimensional vector space that represents terms based on a combination of GloVe and word2vec’s downloadable embeddings, and structured data from ConceptNet and PPDB.  The resulting vector space allows information to be shared among these various representations, including words that were not in the vocabulary of the original representations.  This includes low-frequency words and even words that are not in English.\nThe complete process of building this vector space, whose steps will be explained throughout this paper, appears in Figure 1.\nAs Levy et al. ( 2015) notes, “[. . . ]  much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves.”  While it is presented as a negative result, this simply emphasizes the importance of these system design choices.\nIndeed, we have found that choices about how to handle terms and their embeddings have a significant impact on evaluation results.  One of these choices involves how to pre-process words and phrases before looking them up, and another involves the scale of the various features in the embeddings." 
    }, {
      "heading" : "3.1 Transforming and Aligning Vocabularies",
      "text" : "Different representations apply different preprocessing steps, placing strings in different equivalence classes.  We can only properly combine these resources if these string representations are comparable to each other.\nPre-processing steps that various resources apply include: tokenizing text to separate words from punctuation (which all inputs except GloVe 840B do), joining multi-word phrases with underscores (ConceptNet and word2vec), removing a small list of stopwords from multi-word phrases (ConceptNet only), folding the text to lowercase (ConceptNet and GloVe 42B), replacing multiple digits with the character # (word2vec only), and lemmatizing English words to their root form using a modification of WordNet’s Morphy algorithm (ConceptNet only).\nWe adapt a text pre-processing function from ConceptNet to apply a combination of all of these processes, yielding a set of standardized, languagetagged labels.  As an example, the text “Giving an example” becomes the standardized form /c/en/give example.  Applying this combined pre-processing function to all labels increases the alignment of the various resources while reducing the size of the combined vocabulary.\nBecause the transformations are many-to-one, this has the effect that a single transformed term can become associated with multiple embeddings in a single vector space.  We considered a few options for dealing with these merged terms, such as keeping only the highest-frequency term, averaging the vectors together, or taking a weighted average based on their word frequency.\nWe found in preliminary evaluations that the weighted average was the best approach.  The multiple rows contain valuable data that should not simply be discarded, but lower-frequency rows tend to have lower-quality data.\nWhen using pre-trained vectors, it is often the case that intermediate computations that produced these vectors (such as word frequencies) are not available.  What we do instead is to infer approximate word frequencies from the fact that both GloVe and word2vec output their vocabularies in descending order of frequency.  We approximate the frequency distribution by assuming that the tokens are\ndistributed according to Zipf’s law (Zipf, 1949): the nth token in rank order has a frequency proportional to 1/n. We use these proportions in the weighted average when combining multiple embeddings.\nThis process alone is a benefit on word-similarity evaluations, even without combining any resources.  For example, the Rare Words (RW) dataset (Luong et al.,  2013) tends to encounter terms that are poorly represented or out-of-vocabulary in most word embeddings.  Lemmatizing them before looking them up, and combining them with more frequently observed representations, improves the evaluation results on these words, even though the process loses the ability to distinguish some word forms.  The raw GloVe 840B data gets a Spearman correlation of ρ = .146 on the RW dataset, which increases to ρ = .494 when its embeddings are standardized and combined in this way.\nFigure 2 shows the size of the vocabularies of ConceptNet, GloVe, and word2vec after this transformation, and the sizes of the overlaps among them, using a proportional-area Venn diagram produced using eulerAPE (Micallef and Rodgers, 2014)." 
    }, {
      "heading" : "3.2 Feature Normalization",
      "text" : "As briefly mentioned by Pennington et al. ( 2014), L2 normalization of the columns (that is, the 300 features) of the GloVe matrix provides a notable increase in performance.  One effect of normalization is to increase the weight of distinguishing features and reduce the impact of noisy features.  Features are more distinguishing for the purpose of cosine similarity when they contain a few large values and many small ones.\nWe find that L1 normalization of GloVe performs even better than L2 normalization.  L1 causes occasional large values to have a smaller impact on the norm than L2 normalization.  When a learning method such as GloVe has provided highly selective features, L1 normalization allows us to use them more effectively in measuring similarity." 
    }, {
      "heading" : "3.3 Retrofitting",
      "text" : "Retrofitting (Faruqui et al.,  2015) is a process of combining existing word vectors with a semantic lexicon.  While the original formulation expresses the problem in terms of updates that propagate over a set of edges, we have found it more convenient to\nexpress it and implement it in terms of an update to a matrix.\nThe inputs to retrofitting are an initialm×n dense matrix of term embeddings, W 0, and a list of known semantic relationships.\nFaruqui et al. ’s retrofitting procedure aims to minimize a sum of a word’s distance from its neighbors in the semantic network and its distance from its original vector.  Its implementation in code takes steps along the gradient toward this minimum by iteratively updating one vector w′i at a time to be a linear combination between its original position w0i and the average of its neighbors in the semantic network.\nThe advantage of this iterative update is that it only requires two copies of W in memory (W 0 and the current state) and converges quickly.  A disadvantage is that the results depend on the order in which the nodes of the graph are iterated, which is arbitrary.\nWe instead choose to update the embeddings all at once by multiplying them by a sparse matrix S of semantic connections.  Letting m′ be the size of the merged vocabulary, S is an m′×m′ matrix containing positive weighted values for terms that are known to be semantically related, and 0 otherwise.  The rows of S are scaled to have a sum of 1.  We then add 1 to its diagonal to help new terms converge on a single vector, as described in more detail below.\nLet W 0 be an m′ × n matrix whose rows come from the original embeddings if available, and are all zeroes for terms outside the vocabulary of the original embeddings.  A is a diagonal matrix of weights in which Aii is 1 if term i is in the original vocabulary, and 0 otherwise (allowing us to keep terms near their original embeddings without also keeping outof-vocabulary terms near the zero vector).  We can now update W iteratively so that the next iteration of W is a combination of its product with S and its weighted original state, followed by L2 normalization of its non-zero rows6:\nW k+1 = normalize [( SW k +AW 0 ) (I +A)−1 ] The diagonal of the S matrix relates each term to itself.  We have found that adding 1 to the diago6 We maintain L2 normalization so that minimizing distance and maximizing cosine similarity are always linked.\nnal – effectively adding “self-loops” to the semantic network – helps the expanded retrofitting process converge.  Without this diagonal, terms that only appear in the semantic network, and not in the original embedding space, would get their value only from their neighbors at every step, because their “original position” is the zero vector.  This causes large oscillations that prevent convergence.  With the diagonal, each term vector is influenced by the vector it had in the previous step.\nAs a practical effect of this, Section 4.5 (Varying the System) will show that expanded retrofitting with self-loops added to the diagonal performs better on word-similarity evaluations than it does without, when allowed to run for 10 steps of retrofitting." 
    }, {
      "heading" : "3.4 ConceptNet as an Association Matrix",
      "text" : "In order to apply the expanded retrofitting method, we need to consider the data in ConceptNet as a sparse, symmetric matrix of associations between terms.  What ConceptNet provides is more complex than that, as it connects terms with a variety of notnecessarily-symmetric, labeled relations.\nHavasi et al. ( 2010) introduced a vector space embedding of ConceptNet, “spectral association”, that disregarded the relation labels for the purpose of measuring the relatedness of terms.  Previous embeddings of ConceptNet, such as that of Speer et al. ( 2008), preserved the relations but were suited mostly for direct similarity and inference, not for relatedness.  Because most evaluation data for word similarity is also evaluating relatedness, unless there has been a specific effort to separate them (Agirre et al.,  2009), we erase the labels as in spectral association.\nEach assertion in ConceptNet corresponds to two entries in a sparse association matrix S. ConceptNet assigns a confidence score, or weight, to each assertion.  These weights are not entirely comparable between the data sources that comprise ConceptNet, so we re-scaled them so that the average weight of each different data source is 1.\nAn assertion that relates term i to term j with adjusted weightw will contributew to the values of Sij and Sji.  If another assertion relates the same terms with a different relation, it will add to that value.  This constructs a symmetric matrix S, but the matrix we actually use in retrofitting is the asymmetric\nS′, whose rows have been L1-normalized to prevent high-frequency concepts from overwhelming the results.\nDue to the structure of ConceptNet, there exists a large fringe of terms that are poorly connected to other nodes.  To make the sparse matrix and the size of the overall vocabulary more manageable, we filter ConceptNet when building its association matrix: we exclude all terms that appear fewer than 3 times, English terms that appear fewer than 4 times, and terms with more than 3 words in them." 
    }, {
      "heading" : "3.5 Locally Linear Alignment",
      "text" : "In order to use both word2vec and GloVe at the same time, we need to align their partially-overlapping vocabularies and merge their features.  This is straightforward to do on the terms that are shared between the two vocabularies, but we would rather not lose the other terms, if we can later benefit from learning more about those terms from ConceptNet.\nBefore merging features, we need to compute GloVe representations for terms represented in word2vec but not GloVe, and vice versa.  The way we do this is inspired by Zhao et al. ( 2015), who infer translations between languages of unknown phrases using a locally-linear projection of known translations of similar phrases.  Instead of known translations, we have the terms that overlap between word2vec and GloVe.  Given a non-overlapping term, we calculate its vector as the average of the vectors of the nearest overlapping terms, weighted by their cosine similarity.\nTo combine the features of word2vec and GloVe, we first concatenate their vectors into 600- dimensional vectors.  We then discount redundancy between its features by transforming these 600- dimensional vectors with a singular value decomposition (SVD).  We factor the matrix M of concatenated vectors as M = UΣV T , then compute the new joint features as UΣ1/2.  UΣ would be an orthogonal rotation of the original features; UΣ1/2 reduces the effect of its largest eigenvalues, making over-represented features relatively smaller.\nAs with many decisions we make in preparing this data, we evaluated the benefit of this step on our development data sets.  Discounting redundancy by replacing Σ by Σ1/2 provides a benefit on two out\nof three data sets for evaluating word similarity, as shown in Section 4.5.\nIt is common to use SVD as a form of dimensionality reduction, by discarding the smallest singular values and truncating the matrix accordingly.  Section 4.5 shows that we can reduce the interpolated matrix to from 600 dimensions to 450 or 300 dimensions without much loss in performance." 
    }, {
      "heading" : "4 Evaluation",
      "text" : ""
    }, {
      "heading" : "4.1 Word-Similarity Datasets",
      "text" : "We evaluate our model’s performance at identifying similar words using a variety of word-similarity gold standards:\n• MEN-3000 (Bruni et al.,  2014), crowd-sourced similarity judgments for 3000 word pairs.\n• The Stanford Rare Words (RW) dataset (Luong et al.,  2013), crowd-sourced similarity judgments for 2034 word pairs, with a bias toward uncommon words.\n• WordSim-353 (Finkelstein et al.,  2001), a widely-used corpus of similarity judgments for 353 word pairs.\n• RG-65 (Rubenstein and Goodenough, 1965), a classic corpus of similarity judgments for 65 word pairs, which has additionally been translated into German (Gurevych, 2005) and French (Joubarne and Inkpen, 2011).\nIn striving to maximize an evaluation metric, it is important to hold out some data, to avoid overfitting to the data by modifying the algorithm and its parameters.  The metrics we focused on improving were our rank correlation with MEN-3000, which emphasizes having high-quality representations of common words, and RW, which emphasizes having a broad vocabulary.\nMEN-3000 comes with a development/test split, where 1000 of the 3000 word pairs are held out for testing.  We applied a similar split to RW, setting aside a sample of 1/3 of its word pairs for testing.  In particular, We set aside every third row, starting from row 3, using the Unix command split -un r/3/3 rw.txt.  Similarly, we split on r/1/3\nand r/2/3 and concatenated the results to get the remaining evaluation data.\nWe did not apply a development/test split to WordSim-353 or RG-65, as they are already much smaller than MEN and RW.\nFor the resources where we applied a development/test split, we evaluated decisions we made in the code – such as those described in Section 4.5 – using only the development set, to preserve the integrity of the test set and avoid “overfitting via code”.  We then evaluated the final ensemble, with various pieces enabled, all at once on the held-out test data to produce the results in this paper." 
    }, {
      "heading" : "4.2 Results",
      "text" : "Table 1 shows the performance of the ensemble as various components of it are enabled.  G and g indicate that the initial embeddings come from GloVe\n(840B or 42B respectively), and W indicates that they are word2vec’s SGNS embeddings built from Google News.  When both W and G are present, the embeddings are combined as in Section 3.5.  L1 indicates that the columns of features were L1normalized; otherwise we used the existing scale of the features.7\nSt indicates that the labels were transformed and rows combined using the method of Section 3.1, which is a prerequisite to combining multiple data sources.  CN and PP indicate adding data from ConceptNet, PPDB, or both using expanded retrofitting.  Note that the row labeled with g alone is simply an evaluation of GloVe 42B that reproduces the evaluation of Pennington et al. ( 2014).  Our results here match the published results to within .001.\nWhile Table 1 shows our correlation with only the test data on these evaluations, Table 2 compares our results on development and test data." 
    }, {
      "heading" : "4.3 Benefits of Lemmatization",
      "text" : "Decisions about how to process the data, even after the fact, make a very large difference in wordsimilarity evaluations.  Comparing the GloVe 42B results to the 840B results (Table 1), we see that GloVe 42B works better “out of the box”, and 840B contains messy data that particularly causes problems on the Rare Words evaluation.  However, our strategy to standardize and lemmatize the term labels of GloVe 840B, combining its rows using the Zipf estimate, makes it perform better than GloVe 42B and other published results, as seen in the St, G, L1 row.  We call this configuration “Modified\n7 When GloVe is not L1-normalized, it is L2-normalized instead, following Pennington et al. ( 2014).\nGloVe”, and similarly, our best configuration of word2vec is “Modified word2vec”.\nThe fact that Modified GloVe performs better than GloVe 42B, GloVe 840B, and many other systems, even before retrofitting any additional data onto it, highlights the unexpectedly large benefit of lemmatization: some of the improvements from this paper can be realized without introducing any additional data.\nIt’s important to note that we are not changing the evaluation data by using a lemmatizer; we are only changing the way we look it up as embeddings in the vector space that we are evaluating.  For example, if an evaluation requires similarities for the words “dry” and “dried” to be ranked differently, or the words “polish” and “Polish”, the lemmatized system will rank them the same, and will be penalized in its Spearman correlation.  However, the benefits of lemmatization when evaluating semantic similarity appear to far outweigh the drawbacks." 
    }, {
      "heading" : "4.4 Comparisons to Other Published Results",
      "text" : "In Table 3, we compare our results on the RW and MEN-3000 datasets to the best published results that we know of.  Levy et al. ( 2015) present results including an SVD-based method that scores ρ = .514 on the RW evaluation, as well as an implementation\nof skip-grams with negative sampling (SGNS), originally introduced by Mikolov et al. ( 2013), with optimized hyperparameters.  We also compare to the original results from GloVe 42B, and the best MEN3000 result from Faruqui et al. ( 2015).  We use the complete RW data, not our test set, so that we can compare directly to previous results.\nLevy’s evaluation uses a version of WordSim353 that is split into separate sets for similarity and relatedness.  We estimate the overall score using a weighted average based on the size of the split datasets.\nThe RW and MEN data is also plotted in Figure 3.  Error bars indicate 95% confidence intervals based on the Fisher transformation of ρ (Fisher, 1915), supposing that each evaluation is randomly sampled from a hypothetical larger data set.\nThe ConceptNet vector ensemble with all six components performs better than the previously published systems: we can reject the null hypothesis that the ensemble performs the same as one of these published systems with p < .01.  It is inconclusive whether it is better to include or exclude PPDB in the ensemble, as the results with and without it are very close.\nTable 4 shows the performance of these systems on gold standards that have been translated to other languages, in comparison to the multilingual results published by Faruqui et al. ( 2015).  Our system performs well in non-English languages even though the vocabularies of word2vec and GloVe are assumed to be English only.  The representa-\ntions of non-English words come from expanded retrofitting, which allows information to propagate over the inter-language links in ConceptNet." 
    }, {
      "heading" : "4.5 Varying the System",
      "text" : "Some of the procedures we implemented in creating the ensemble require some justification.  To show the benefits of certain decisions, such as adding selfloops or the way we choose to merge rows of GloVe, we have evaluated what happens to the system in the absence of each decision.  These evaluations appear in Table 5.  These evaluations were part of how we decided on the best configuration of the system, so they were run on the development sets of RW and MEN-3000, not the held-out test sets.\nIn Table 5, we can see that the choice of how to merge rows of GloVe that get the same standardized label makes a large difference.  Recall that the method we ultimately used was to assign each row a pseudo-frequency based on Zipf’s law, and then take a weighted average based on those frequencies.  The results drop noticeably when we try the other proposed methods, which are taking only the first (most frequent) row that appears, or taking the unweighted average of the rows.\nThe same table shows that we can save some computation time and space by reducing the dimensionality of the feature vectors while building the matrix that combines word2vec and GloVe.  Reducing the dimensionality seems to cause a small degradation in RW score, but the MEN and WordSim scores stay around the same value, even increasing by an inconclusive amount in some cases.\nIf we skip applying the SVD at all in the interpolation step – that is, when a term has features in word2vec and GloVe, we simply concatenate those features with all their redundancy – it lowers the RW score somewhat, but raises the WordSim-353 score.\nIn the retrofitting procedure, we made the decision to add “self-loops” that connect each term to itself, because this helps stabilize the representations of terms that are outside the original vocabulary of GloVe.  Reversing this decision (and running for the same number of steps) causes a noticeable drop in performance on RW, the evaluation that is most likely to involve words that were poorly represented or unrepresented in GloVe.\nIn separate experimentation, we found that when\nwe separate ConceptNet into its component datasets and drop each one in turn, the effects on the evaluation results are mostly quite small.  There is no single dataset that acts as the “keystone” without which the system falls apart, but one dataset — Wiktionary — unsurprisingly has a larger effect than the others, because it is responsible for most of the assertions.  Of the 5,631,250 filtered assertions that come from ConceptNet, 4,244,410 of them are credited to Wiktionary.\nWithout Wiktionary, the score on RW drops from .587 to .541.  However, at the same time, the MEN3000 score increases from .858 to .865.  The system continues to do what it is designed to do without Wiktionary, but there seems to be a tradeoff in performance on rare words and common words involved." 
    }, {
      "heading" : "5 Conclusions",
      "text" : "The work we have presented here involves building on many previous techniques, while adding some new techniques and new sources of knowledge.\nAs Levy et al. ( 2015) found, high-level choices about how to use a system can significantly affect its performance.  While Levy found settings of hyperparameters that made word2vec outperform GloVe, we found that we can make GloVe outperform Levy’s tuned word2vec by pre-processing the words in GloVe with case-folding and lemmatization, and re-weighting the features using L1 normalization.\nWe also showed that it isn’t necessary to choose just one of word2vec or GloVe as a starting point.  Instead, we can benefit from both of them using a locally-linear interpolation between them.\nWe showed that ConceptNet is a useful source of structured knowledge that was not consid-\nered in previous work on retrofitting distributional semantics with structured knowledge, especially when retrofitting is generalized into our expanded retrofitting technique, which can benefit from links that are outside the original vocabulary." 
    }, {
      "heading" : "5.1 Future Work",
      "text" : "One aspect of our method that clearly has room for improvement is the fact that we disregard the labels on relations in ConceptNet.  There is valuable knowledge there that we might be able to take into account with a more sophisticated extension of retrofitting, one that goes beyond simply knowing that particular words should be related, to handling them differently based on how they are related, as in the RESCAL representation (Nickel et al.,  2011).  This seems particularly important for antonyms, which indicate that words are similar overall but different in a key aspect, such as forming two ends of the same scale.\nLemmatization is clearly a useful component of our word-similarity representation, but it loses information.  Representing morphological relationships as operations in the vector space, as in Soricut and Och (2015), could yield a better representation of similarities between forms of words.\nWe believe that the variety of data sources represented in ConceptNet helped to improve evaluation scores by expanding the domain of the system’s knowledge.  There’s no reason the improvement needs to stop here.  It is quite likely that there are more sources of linked open data that could be included, or further standardizations that could be applied to the text to align more data.  An appropriate representation of Universal WordNet (De Melo and Weikum, 2009) could improve the multilingual performance, for example, as could embeddings built\nfrom the distribution of words in non-English text.  Adapting Rothe and Schütze’s AutoExtend representation could provide a representation of word senses and a more specific understanding of words to the system." 
    }, {
      "heading" : "6 Reproducing These Results",
      "text" : "We aim for these results to be reproducible and reusable by others.  The code that we ran to produce the results in this paper is available in the GitHub repository https: //github.com/LuminosoInsight/ conceptnet-vector-ensemble.  We have tagged this revision as submitted-20160406, so that this paper can be reproduced as-is, even if we update the code later.\nThe README of that repository also points to URLs where the computed results can be downloaded, as a 1453348 × 600 matrix of embeddings in NumPy format and a corresponding list of row labels." 
    }, {
      "heading" : "Acknowledgements",
      "text" : "We thank Catherine Havasi for overseeing and reviewing this work, and Avril Kenney, Dennis Clark, and Alice Kaanta for providing feedback on drafts of this paper.\nWe thank the word2vec, GloVe, and PPDB teams for opening their data so that new techniques can be built on top of their results.  We also thank our collaborators who have contributed code and data to ConceptNet over the years, and the tens of thousands of pseudonymous contributors to Wiktionary, Open Mind Common Sense, and related projects, for their frequently uncredited work in providing freelyavailable lexical knowledge." 
    } ],
    "references" : [ {
      "title" : "A study on similarity and relatedness using distributional and WordNet-based approaches",
      "author" : [ "Agirre et al.2009] Eneko Agirre", "Enrique Alfonseca", "Keith Hall", "Jana Kravalova", "Marius Paşca", "Aitor Soroa" ],
      "venue" : "In Proceedings of Human Language Technologies:",
      "citeRegEx" : "Agirre et al\\.,?  \\Q2009\\E",
      "shortCiteRegEx" : "Agirre et al\\.",
      "year" : 2009
    }, {
      "title" : "DBpedia: A nucleus for a web of open data",
      "author" : [ "Auer et al.2007] Sören Auer", "Christian Bizer", "Georgi Kobilarov", "Jens Lehmann", "Richard Cyganiak", "Zachary Ives" ],
      "venue" : null,
      "citeRegEx" : "Auer et al\\.,?  \\Q2007\\E",
      "shortCiteRegEx" : "Auer et al\\.",
      "year" : 2007
    }, {
      "title" : "JMDict: a Japanesemultilingual dictionary",
      "author" : [ "James Breen" ],
      "venue" : "In Proceedings of the Workshop on Multilingual Linguistic Ressources,",
      "citeRegEx" : "Breen.,?  \\Q2004\\E",
      "shortCiteRegEx" : "Breen.", 
      "year" : 2004
    }, {
      "title" : "Multimodal distributional semantics",
      "author" : [ "Bruni et al.2014] Elia Bruni", "Nam-Khanh Tran", "Marco Baroni" ],
      "venue" : "J. Artif.  Intell.  Res. ( JAIR),",
      "citeRegEx" : "Bruni et al\\.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Bruni et al\\.",
      "year" : 2014
    }, {
      "title" : "Towards a universal wordnet by learning from combined evidence",
      "author" : [ "De Melo", "Weikum2009] Gerard De Melo", "Gerhard Weikum" ],
      "venue" : "In Proceedings of the 18th ACM conference on Information and knowledge management,",
      "citeRegEx" : "Melo et al\\.,?  \\Q2009\\E",
      "shortCiteRegEx" : "Melo et al\\.",
      "year" : 2009
    }, {
      "title" : "Indexing by latent semantic analysis",
      "author" : [ "Susan T Dumais", "Thomas K. Landauer", "George W. Furnas", "Richard A. Harshman" ],
      "venue" : "JAsIs,",
      "citeRegEx" : "Deerwester et al\\.,?  \\Q1990\\E",
      "shortCiteRegEx" : "Deerwester et al\\.",
      "year" : 1990
    }, {
      "title" : "Retrofitting word vectors to semantic lexicons",
      "author" : [ "Jesse Dodge", "Sujay K. Jauhar", "Chris Dyer", "Eduard Hovy", "Noah A. Smith" ],
      "venue" : "Proceedings of NAACL",
      "citeRegEx" : "Faruqui et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Faruqui et al\\.",
      "year" : 2015
    }, {
      "title" : "Placing search in context: The concept revisited",
      "author" : [ "Evgeniy Gabrilovich", "Yossi Matias", "Ehud Rivlin", "Zach Solan", "Gadi Wolfman", "Eytan Ruppin" ],
      "venue" : "In Proceedings of the 10th international conference on",
      "citeRegEx" : "Finkelstein et al\\.,?  \\Q2001\\E",
      "shortCiteRegEx" : "Finkelstein et al\\.",
      "year" : 2001
    }, {
      "title" : "Frequency distribution of the values of the correlation coefficient in samples from an indefinitely large population",
      "author" : [ "Ronald A Fisher" ],
      "venue" : null,
      "citeRegEx" : "Fisher.,?  \\Q1915\\E",
      "shortCiteRegEx" : "Fisher.", 
      "year" : 1915
    }, {
      "title" : "PPDB: The paraphrase database",
      "author" : [ "Benjamin Van Durme", "Chris Callison-Burch" ],
      "venue" : "In HLT-NAACL,",
      "citeRegEx" : "Ganitkevitch et al\\.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Ganitkevitch et al\\.",
      "year" : 2013
    }, {
      "title" : "Using the structure of a conceptual network in computing semantic relatedness",
      "author" : [ "Iryna Gurevych" ],
      "venue" : "In Natural Language Processing–",
      "citeRegEx" : "Gurevych.,?  \\Q2005\\E",
      "shortCiteRegEx" : "Gurevych.", 
      "year" : 2005
    }, {
      "title" : "Automated color selection using semantic knowledge",
      "author" : [ "Robert Speer", "Justin Holmgren" ],
      "venue" : "In AAAI Fall Symposium: Commonsense Knowledge",
      "citeRegEx" : "Havasi et al\\.,?  \\Q2010\\E",
      "shortCiteRegEx" : "Havasi et al\\.",
      "year" : 2010
    }, {
      "title" : "Comparison of semantic similarity for different languages using the Google N-gram corpus and second-order co-occurrence measures",
      "author" : [ "Joubarne", "Inkpen2011] Colette Joubarne", "Diana Inkpen" ],
      "venue" : "In Advances in Artificial Intelligence,",
      "citeRegEx" : "Joubarne et al\\.,?  \\Q2011\\E",
      "shortCiteRegEx" : "Joubarne et al\\.",
      "year" : 2011
    }, {
      "title" : "Communitybased game design: experiments on social games for commonsense data collection",
      "author" : [ "Kuo et al.2009] Yen-ling Kuo", "Jong-Chuan Lee", "Kaiyang Chiang", "Rex Wang", "Edward Shen", "Cheng-wei Chan", "Jane Yung-jen Hsu" ],
      "venue" : null,
      "citeRegEx" : "Kuo et al\\.,?  \\Q2009\\E",
      "shortCiteRegEx" : "Kuo et al\\.",
      "year" : 2009
    }, {
      "title" : "Improving distributional similarity with lessons learned from word embeddings.  Transactions of the Association for Computational Linguistics, 3:211–225",
      "author" : [ "Levy et al.2015] Omer Levy", "Yoav Goldberg", "Ido Dagan" ],
      "venue" : null,
      "citeRegEx" : "Levy et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Levy et al\\.",
      "year" : 2015
    }, {
      "title" : "Better word representations with recursive neural networks for morphology",
      "author" : [ "Richard Socher", "Christopher D Manning" ],
      "venue" : null,
      "citeRegEx" : "Luong et al\\.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Luong et al\\.",
      "year" : 2013
    }, {
      "title" : "eulerAPE: Drawing area-proportional 3-venn diagrams using ellipses",
      "author" : [ "Micallef", "Rodgers2014] Luana Micallef", "Peter Rodgers" ],
      "venue" : "PLoS ONE,",
      "citeRegEx" : "Micallef et al\\.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Micallef et al\\.",
      "year" : 2014
    }, {
      "title" : "Efficient estimation of word representations in vector space.  CoRR, abs/1301.3781",
      "author" : [ "Kai Chen", "Greg Corrado", "Jeffrey Dean" ],
      "venue" : null,
      "citeRegEx" : "Mikolov et al\\.,?  \\Q2013\\E",
      "shortCiteRegEx" : "Mikolov et al\\.",
      "year" : 2013
    }, {
      "title" : "A three-way model for collective learning on multi-relational data",
      "author" : [ "Volker Tresp", "Hans-Peter Kriegel" ],
      "venue" : "In Proceedings of the 28th international conference on machine learning",
      "citeRegEx" : "Nickel et al\\.,?  \\Q2011\\E",
      "shortCiteRegEx" : "Nickel et al\\.",
      "year" : 2011
    }, {
      "title" : "GloVe: Global vectors for word representation",
      "author" : [ "Richard Socher", "Christopher D Manning" ],
      "venue" : "Proceedings of the Empiricial Methods in Natural Language Processing",
      "citeRegEx" : "Pennington et al\\.,?  \\Q2014\\E",
      "shortCiteRegEx" : "Pennington et al\\.",
      "year" : 2014
    }, {
      "title" : "AutoExtend: Extending word embeddings to embeddings for synsets and lexemes",
      "author" : [ "Rothe", "Schütze2015] Sascha Rothe", "Hinrich Schütze" ],
      "venue" : "In Proceedings of the 53rd Annual Meeting of the Association",
      "citeRegEx" : "Rothe et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Rothe et al\\.",
      "year" : 2015
    }, {
      "title" : "Contextual correlates of synonymy",
      "author" : [ "Rubenstein", "John B Goodenough" ],
      "venue" : "Communications of the ACM,",
      "citeRegEx" : "Rubenstein et al\\.,?  \\Q1965\\E",
      "shortCiteRegEx" : "Rubenstein et al\\.",
      "year" : 1965
    }, {
      "title" : "Open Mind Common Sense: Knowledge acquisition from the general public.  In On the move to meaningful internet systems 2002: CoopIS",
      "author" : [ "Singh et al.2002] Push Singh", "Thomas Lin", "Erik T Mueller", "Grace Lim", "Travell Perkins", "Wan Li Zhu" ],
      "venue" : null,
      "citeRegEx" : "Singh et al\\.,?  \\Q2002\\E",
      "shortCiteRegEx" : "Singh et al\\.",
      "year" : 2002
    }, {
      "title" : "Unsupervised morphology induction using word embeddings",
      "author" : [ "Soricut", "Och2015] Radu Soricut", "Franz Och" ],
      "venue" : "In Proc.  NAACL",
      "citeRegEx" : "Soricut et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Soricut et al\\.",
      "year" : 2015
    }, {
      "title" : "Representing general relational knowledge in ConceptNet 5",
      "author" : [ "Speer", "Havasi2012] Robert Speer", "Catherine Havasi" ],
      "venue" : "In LREC,",
      "citeRegEx" : "Speer et al\\.,?  \\Q2012\\E",
      "shortCiteRegEx" : "Speer et al\\.",
      "year" : 2012
    }, {
      "title" : "AnalogySpace: Reducing the dimensionality of common sense knowledge",
      "author" : [ "Speer et al.2008] Robert Speer", "Catherine Havasi", "Henry Lieberman" ],
      "venue" : "In AAAI,",
      "citeRegEx" : "Speer et al\\.,?  \\Q2008\\E",
      "shortCiteRegEx" : "Speer et al\\.",
      "year" : 2008
    }, {
      "title" : "Verbosity: a game for collecting common-sense facts",
      "author" : [ "von Ahn et al.2006] Luis von Ahn", "Mihir Kedia", "Manuel Blum" ],
      "venue" : "In Proceedings of the SIGCHI conference on Human Factors in computing systems,",
      "citeRegEx" : "Ahn et al\\.,?  \\Q2006\\E",
      "shortCiteRegEx" : "Ahn et al\\.",
      "year" : 2006
    }, {
      "title" : "Learning translation models from monolingual continuous representations",
      "author" : [ "Zhao et al.2015] Kai Zhao", "Hany Hassan", "Michael Auli" ],
      "venue" : "In Proceedings of NAACL",
      "citeRegEx" : "Zhao et al\\.,?  \\Q2015\\E",
      "shortCiteRegEx" : "Zhao et al\\.",
      "year" : 2015
    }, {
      "title" : "Human behavior and the principle of least effort: an introduction to human ecology",
      "author" : [ "G.K. Zipf" ],
      "venue" : null,
      "citeRegEx" : "Zipf.,?  \\Q1949\\E",
      "shortCiteRegEx" : "Zipf.", 
      "year" : 1949
    } ],
    "referenceMentions" : [ {
      "referenceID" : 19,
      "context" : "We present an ensemble method that combines embeddings produced by GloVe (Pennington et al.,  2014) and word2vec (Mikolov et al.", 
      "startOffset" : 73,
      "endOffset" : 98
    }, {
      "referenceID" : 17,
      "context" : ", 2014) and word2vec (Mikolov et al.,  2013) with structured knowledge from the semantic networks ConceptNet (Speer and Havasi, 2012) and PPDB (Ganitkevitch et al.", 
      "startOffset" : 21,
      "endOffset" : 43
    }, {
      "referenceID" : 9,
      "context" : ", 2013) with structured knowledge from the semantic networks ConceptNet (Speer and Havasi, 2012) and PPDB (Ganitkevitch et al.,  2013), merging their information into a common representation with a large, multilingual vocabulary.", 
      "startOffset" : 106,
      "endOffset" : 133
    }, {
      "referenceID" : 15,
      "context" : "596 on an evaluation of rare words (Luong et al.,  2013) is 16% higher than the previous best known system.", 
      "startOffset" : 35,
      "endOffset" : 55
    }, {
      "referenceID" : 5,
      "context" : "This kind of vector space has been used in applications such as search, topic detection, and text classification, dating back to the introduction of latent semantic analysis (Deerwester et al.,  1990).", 
      "startOffset" : 174,
      "endOffset" : 199
    }, {
      "referenceID" : 5,
      "context" : "This kind of vector space has been used in applications such as search, topic detection, and text classification, dating back to the introduction of latent semantic analysis (Deerwester et al.,  1990).  In recent years, there has been a surge of interest in natural-language embeddings, as machine-learning techniques such as Mikolov et al. ( 2013)’s word2vec and Pennington et al.", 
      "startOffset" : 175,
      "endOffset" : 346
    }, {
      "referenceID" : 5,
      "context" : "This kind of vector space has been used in applications such as search, topic detection, and text classification, dating back to the introduction of latent semantic analysis (Deerwester et al.,  1990).  In recent years, there has been a surge of interest in natural-language embeddings, as machine-learning techniques such as Mikolov et al. ( 2013)’s word2vec and Pennington et al. ( 2014)’s GloVe have begun to show dramatic improvements.", 
      "startOffset" : 175,
      "endOffset" : 386
    }, {
      "referenceID" : 5,
      "context" : "This kind of vector space has been used in applications such as search, topic detection, and text classification, dating back to the introduction of latent semantic analysis (Deerwester et al.,  1990).  In recent years, there has been a surge of interest in natural-language embeddings, as machine-learning techniques such as Mikolov et al. ( 2013)’s word2vec and Pennington et al. ( 2014)’s GloVe have begun to show dramatic improvements.  Word embeddings are often suggested as an initialization for more complex methods, such as the sentence encodings of Kiros et al. ( 2015).", 
      "startOffset" : 175,
      "endOffset" : 573
    }, {
      "referenceID" : 0,
      "context" : "Some methods and evaluations (Agirre et al.,  2009) distinguish word similarity from word relatedness.", 
      "startOffset" : 29,
      "endOffset" : 50
    }, {
      "referenceID" : 22,
      "context" : "ConceptNet originated as a machine-parsed version of the early crowd-sourcing project called Open Mind Common Sense (OMCS) (Singh et al.,  2002), and has expanded to include several other data sources, both crowd-sourced and expert-created, by unifying their vocabularies into a single representation.", 
      "startOffset" : 123,
      "endOffset" : 143
    }, {
      "referenceID" : 2,
      "context" : ", 1998), Wiktionary (Wiktionary, 2014), and JMDict (Breen, 2004), as well as data from “games with a purpose” in multiple languages (von Ahn et al.", 
      "startOffset" : 51,
      "endOffset" : 64
    }, {
      "referenceID" : 13,
      "context" : ", 1998), Wiktionary (Wiktionary, 2014), and JMDict (Breen, 2004), as well as data from “games with a purpose” in multiple languages (von Ahn et al.,  2006; Kuo et al.,  2009; Nakahara and Yamada, 2011).", 
      "startOffset" : 132,
      "endOffset" : 199
    }, {
      "referenceID" : 1,
      "context" : "We choose not to include ConceptNet’s alignment to DBPedia (Auer et al.,  2007) here, as DBPedia focuses on relations between specific named entities, which do not help with general word similarity.", 
      "startOffset" : 59,
      "endOffset" : 78
    }, {
      "referenceID" : 9,
      "context" : "PPDB (Ganitkevitch et al.,  2013) is another resource that is useful for learning about word similarity, providing different information from ConceptNet.", 
      "startOffset" : 5,
      "endOffset" : 32
    }, {
      "referenceID" : 6,
      "context" : "PPDB is used as an external knowledge source by Faruqui et al. ( 2015), so we have evaluated the effect of adding it to our ensemble as well.", 
      "startOffset" : 48,
      "endOffset" : 70
    }, {
      "referenceID" : 17,
      "context" : "2 Given different goals – such as achieving a high score on Mikolov et al. ( 2013)’s analogy evaluation that tests for implicit relations such as “A is the CEO of company B” – including an appropriate representation of DBPedia would of course be helpful.", 
      "startOffset" : 60,
      "endOffset" : 82
    }, {
      "referenceID" : 19,
      "context" : "GloVe (Pennington et al.,  2014) is an unsupervised learning algorithm that learns a set of word embeddings such that the dot product of two words’ embeddings is approximately equal to the logarithm of their co-occurrence count.", 
      "startOffset" : 6,
      "endOffset" : 31
    }, {
      "referenceID" : 18,
      "context" : "GloVe is presented by Pennington et al. ( 2014) as performing better than word2vec on wordsimilarity tasks, but Levy et al.", 
      "startOffset" : 22,
      "endOffset" : 47
    }, {
      "referenceID" : 14,
      "context" : "(2014) as performing better than word2vec on wordsimilarity tasks, but Levy et al. ( 2015) finds that word2vec performs better with an optimized setting of hyperparameters than GloVe does, when retrained with a particular corpus.", 
      "startOffset" : 71,
      "endOffset" : 90
    }, {
      "referenceID" : 14,
      "context" : "As Levy et al. ( 2015) notes, “[.", 
      "startOffset" : 3,
      "endOffset" : 22
    }, {
      "referenceID" : 28,
      "context" : "distributed according to Zipf’s law (Zipf, 1949): the nth token in rank order has a frequency proportional to 1/n.",
      "startOffset" : 36,
      "endOffset" : 48
    }, {
      "referenceID" : 15,
      "context" : "For example, the Rare Words (RW) dataset (Luong et al.,  2013) tends to encounter terms that are poorly represented or out-of-vocabulary in most word embeddings.", 
      "startOffset" : 41,
      "endOffset" : 61
    }, {
      "referenceID" : 19,
      "context" : "As briefly mentioned by Pennington et al. ( 2014), L2 normalization of the columns (that is, the 300 features) of the GloVe matrix provides a notable increase in performance.", 
      "startOffset" : 24,
      "endOffset" : 49
    }, {
      "referenceID" : 6,
      "context" : "Retrofitting (Faruqui et al.,  2015) is a process of combining existing word vectors with a semantic lexicon.", 
      "startOffset" : 13,
      "endOffset" : 35
    }, {
      "referenceID" : 0,
      "context" : "Because most evaluation data for word similarity is also evaluating relatedness, unless there has been a specific effort to separate them (Agirre et al.,  2009), we erase the labels as in spectral association.", 
      "startOffset" : 138,
      "endOffset" : 159
    }, {
      "referenceID" : 27,
      "context" : "The way we do this is inspired by Zhao et al. ( 2015), who infer translations between languages of unknown phrases using a locally-linear projection of known translations of similar phrases.", 
      "startOffset" : 34,
      "endOffset" : 53
    }, {
      "referenceID" : 3,
      "context" : "• MEN-3000 (Bruni et al.,  2014), crowd-sourced similarity judgments for 3000 word pairs.", 
      "startOffset" : 11,
      "endOffset" : 31
    }, {
      "referenceID" : 15,
      "context" : "• The Stanford Rare Words (RW) dataset (Luong et al.,  2013), crowd-sourced similarity judgments for 2034 word pairs, with a bias toward uncommon words.", 
      "startOffset" : 39,
      "endOffset" : 59
    }, {
      "referenceID" : 7,
      "context" : "• WordSim-353 (Finkelstein et al.,  2001), a widely-used corpus of similarity judgments for 353 word pairs.", 
      "startOffset" : 14,
      "endOffset" : 40
    }, {
      "referenceID" : 10,
      "context" : "• RG-65 (Rubenstein and Goodenough, 1965), a classic corpus of similarity judgments for 65 word pairs, which has additionally been translated into German (Gurevych, 2005) and French (Joubarne and Inkpen, 2011).", 
      "startOffset" : 154,
      "endOffset" : 170
    }, {
      "referenceID" : 19,
      "context" : "Note that the row labeled with g alone is simply an evaluation of GloVe 42B that reproduces the evaluation of Pennington et al. ( 2014).  Our results here match the published results to within .", 
      "startOffset" : 110,
      "endOffset" : 135
    }, {
      "referenceID" : 19,
      "context" : "7 When GloVe is not L1-normalized, it is L2-normalized instead, following Pennington et al. ( 2014).  Method RW [all] MEN WS",
      "startOffset" : 74,
      "endOffset" : 99
    }, {
      "referenceID" : 14,
      "context" : "Levy et al. ( 2015) present results including an SVD-based method that scores ρ = .", 
      "startOffset" : 0,
      "endOffset" : 19
    }, {
      "referenceID" : 16,
      "context" : "of skip-grams with negative sampling (SGNS), originally introduced by Mikolov et al. ( 2013), with optimized hyperparameters.", 
      "startOffset" : 70,
      "endOffset" : 92
    }, {
      "referenceID" : 6,
      "context" : "We also compare to the original results from GloVe 42B, and the best MEN3000 result from Faruqui et al. ( 2015).  We use the complete RW data, not our test set, so that we can compare directly to previous results.", 
      "startOffset" : 89,
      "endOffset" : 111
    }, {
      "referenceID" : 8,
      "context" : "Error bars indicate 95% confidence intervals based on the Fisher transformation of ρ (Fisher, 1915), supposing that each evaluation is randomly sampled from a hypothetical larger data set.", 
      "startOffset" : 85,
      "endOffset" : 99
    }, {
      "referenceID" : 6,
      "context" : "Table 4 shows the performance of these systems on gold standards that have been translated to other languages, in comparison to the multilingual results published by Faruqui et al. ( 2015).  Our system performs well in non-English languages even though the vocabularies of word2vec and GloVe are assumed to be English only.", 
      "startOffset" : 166,
      "endOffset" : 188
    }, {
      "referenceID" : 14,
      "context" : "As Levy et al. ( 2015) found, high-level choices about how to use a system can significantly affect its performance.", 
      "startOffset" : 3,
      "endOffset" : 22
    }, {
      "referenceID" : 18,
      "context" : "There is valuable knowledge there that we might be able to take into account with a more sophisticated extension of retrofitting, one that goes beyond simply knowing that particular words should be related, to handling them differently based on how they are related, as in the RESCAL representation (Nickel et al.,  2011).", 
      "startOffset" : 299,
      "endOffset" : 320
    } ],
    "year" : 2016,
    "abstractText" : "A currently successful approach to computational semantics is to represent words as embeddings in a machine-learned vector space.  We present an ensemble method that combines embeddings produced by GloVe (Pennington et al.,  2014) and word2vec (Mikolov et al.,  2013) with structured knowledge from the semantic networks ConceptNet (Speer and Havasi, 2012) and PPDB (Ganitkevitch et al.,  2013), merging their information into a common representation with a large, multilingual vocabulary.  The embeddings it produces achieve state-of-the-art performance on many word-similarity evaluations.  Its score of ρ = .596 on an evaluation of rare words (Luong et al.,  2013) is 16% higher than the previous best known system.", 
    "creator" : "LaTeX with hyperref package"
  }
}