GloVe: Global Vectors for Word Representation
Jeffrey Pennington, Richard Socher, Christopher D. Manning
Computer Science Department, Stanford University, Stanford, CA 94305
jpennin@stanford.edu, richard@socher.org, manning@stanford.edu

Abstract
Recent methods for learning vector space
representations of words have succeededin capturing fine-grained semantic andsyntactic regularities using vector arithmetic, but the origin of these regularities
has remained opaque. We analyze andmake explicit the model properties needed
for such regularities to emerge in word
vectors. The result is a new global logbilinear regression model that combines
the advantages of the two major modelfamilies in the literature: global matrix
factorization and local context window
methods. Our model efficiently leverages
statistical information by training only on
the nonzero elements in a word-word cooccurrence matrix, rather than on the entire sparse matrix or on individual context
windows in a large corpus. The model produces a vector space with meaningful substructure, as evidenced by its performanceof 75% on a recent word analogy task. It
also outperforms related models on similarity tasks and named entity recognition.

1

Introduction

Semantic vector space models of language represent each word with a real-valued vector. These
vectors can be used as features in a variety of applications, such as information retrieval (Manning
et al., 2008), document classification (Sebastiani,
2002), question answering (Tellex et al., 2003),
named entity recognition (Turian et al., 2010), andparsing (Socher et al., 2013).
Most word vector methods rely on the distanceor angle between pairs of word vectors as the primary method for evaluating the intrinsic qualityof such a set of word representations. Recently,
Mikolov et al. (2013c) introduced a new evaluation scheme based on word analogies that probes

the finer structure of the word vector space by examining not the scalar distance between word vectors, but rather their various dimensions of difference. For example, the analogy “king is toqueen as man is to woman” should be encodedin the vector space by the vector equation king −queen = man − woman. This evaluation scheme
favors models that produce dimensions of meaning, thereby capturing the multi-clustering idea of
distributed representations (Bengio, 2009).
The two main model families for learning word
vectors are: 1) global matrix factorization methods, such as latent semantic analysis (LSA) (Deerwester et al., 1990) and 2) local context window
methods, such as the skip-gram model of Mikolov
et al. (2013c). Currently, both families suffer significant drawbacks. While methods like LSA efficiently leverage statistical information, they dorelatively poorly on the word analogy task, indicating a sub-optimal vector space structure. Methods like skip-gram may do better on the analogytask, but they poorly utilize the statistics of the corpus since they train on separate local context windows instead of on global co-occurrence counts.
In this work, we analyze the model properties
necessary to produce linear directions of meaning
and argue that global log-bilinear regression models are appropriate for doing so. We propose a specific weighted least squares model that trains on
global word-word co-occurrence counts and thus
makes efficient use of statistics. The model produces a word vector space with meaningful substructure, as evidenced by its state-of-the-art performance of 75% accuracy on the word analogy
dataset. We also demonstrate that our methods
outperform other current methods on several word
similarity tasks, and also on a common named entity recognition (NER) benchmark.
We provide the source code for the model aswell as trained word vectors at http://nlp.
stanford.edu/projects/glove/.

2

Related Work

Matrix Factorization Methods. Matrix factorization methods for generating low-dimensional
word representations have roots stretching as far
back as LSA. These methods utilize low-rank approximations to decompose large matrices that
capture statistical information about a corpus. The
particular type of information captured by such
matrices varies by application. In LSA, the matrices are of “term-document” type, i.e., the rows
correspond to words or terms, and the columns
correspond to different documents in the corpus.
In contrast, the Hyperspace Analogue to Language(HAL) (Lund and Burgess, 1996), for example,
utilizes matrices of “term-term” type, i.e., the rows
and columns correspond to words and the entries
correspond to the number of times a given word
occurs in the context of another given word.
A main problem with HAL and related methods is that the most frequent words contribute a
disproportionate amount to the similarity measure:the number of times two words co-occur with theor and, for example, will have a large effect on
their similarity despite conveying relatively little
about their semantic relatedness. A number oftechniques exist that addresses this shortcoming of
HAL, such as the COALS method (Rohde et al.,2006), in which the co-occurrence matrix is firsttransformed by an entropy- or correlation-based
normalization. An advantage of this type of transformation is that the raw co-occurrence counts,
which for a reasonably sized corpus might span
8 or 9 orders of magnitude, are compressed so as
to be distributed more evenly in a smaller interval. A variety of newer models also pursue thisapproach, including a study (Bullinaria and Levy,
2007) that indicates that positive pointwise mutual information (PPMI) is a good transformation.
More recently, a square root type transformationin the form of Hellinger PCA (HPCA) (Lebret andCollobert, 2014) has been suggested as an effective way of learning word representations.
Shallow Window-Based Methods. Anotherapproach is to learn word representations that aid
in making predictions within local context windows. For example, Bengio et al. (2003) introduced a model that learns word vector representations as part of a simple neural network architecture for language modeling. Collobert and Weston(2008) decoupled the word vector training fromthe downstream training objectives, which paved

the way for Collobert et al. (2011) to use the fullcontext of a word for learning the word representations, rather than just the preceding context as is
the case with language models.
Recently, the importance of the full neural network structure for learning useful word representations has been called into question. The
skip-gram and continuous bag-of-words (CBOW)
models of Mikolov et al. (2013a) propose a simple single-layer architecture based on the innerproduct between two word vectors. Mnih and
Kavukcuoglu (2013) also proposed closely-related
vector log-bilinear models, vLBL and ivLBL, andLevy et al. (2014) proposed explicit word embeddings based on a PPMI metric.
In the skip-gram and ivLBL models, the objective is to predict a word’s context given the word
itself, whereas the objective in the CBOW andvLBL models is to predict a word given its context. Through evaluation on a word analogy task,these models demonstrated the capacity to learn
linguistic patterns as linear relationships between
the word vectors.
Unlike the matrix factorization methods, theshallow window-based methods suffer from thedisadvantage that they do not operate directly on
the co-occurrence statistics of the corpus. Instead,these models scan context windows across the entire corpus, which fails to take advantage of thevast amount of repetition in the data.

3

The GloVe Model

The statistics of word occurrences in a corpus is
the primary source of information available to all
unsupervised methods for learning word representations, and although many such methods now exist, the question still remains as to how meaningis generated from these statistics, and how the resulting word vectors might represent that meaning.
In this section, we shed some light on this question. We use our insights to construct a new model
for word representation which we call GloVe, for
Global Vectors, because the global corpus statistics are captured directly by the model.
First we establish some notation. Let the matrix
of word-word co-occurrence counts be denoted by
X, whose entries X i j tabulate the number of times
word j occurs in the context of word i. Let X i =P
k X ik be the number of times any word appears
in the context of word i. Finally, let Pi j = P( j |i) =X i j /X i be the probability that word j appear in theTable 1: Co-occurrence probabilities for target words ice and steam with selected context words from a 6
billion token corpus. Only in the ratio does noise from non-discriminative words like water and fashion
cancel out, so that large values (much greater than 1) correlate well with properties specific to ice, andsmall values (much less than 1) correlate well with properties specific of steam.
Probability and Ratiok = solidk = gask = waterk = fashion

P(k |ice)

1.9 × 10 −4

6.6 × 10 −5

3.0 × 10 −3

1.7 × 10 −5

P(k |steam)

2.2 × 10 −5

7.8 × 10 −4

2.2 × 10 −3

1.8 × 10 −5

8.9

8.5 × 10 −2

1.36

0.96

P(k |ice)/P(k |steam)context of word i.
We begin with a simple example that showcases
how certain aspects of meaning can be extracted
directly from co-occurrence probabilities. Consider two words i and j that exhibit a particular aspect of interest; for concreteness, suppose we areinterested in the concept of thermodynamic phase,for which we might take i = ice and j = steam.
The relationship of these words can be examinedby studying the ratio of their co-occurrence probabilities with various probe words, k. For words
k related to ice but not steam, say k = solid, we
expect the ratio Pik /P j k will be large. Similarly,
for words k related to steam but not ice, say k =gas, the ratio should be small. For words k likewater or fashion, that are either related to both ice
and steam, or to neither, the ratio should be close
to one. Table 1 shows these probabilities and theirratios for a large corpus, and the numbers confirm
these expectations. Compared to the raw probabilities, the ratio is better able to distinguish relevant
words (solid and gas) from irrelevant words (waterand fashion) and it is also better able to discriminate between the two relevant words.
The above argument suggests that the appropriate starting point for word vector learning should
be with ratios of co-occurrence probabilities rather
than the probabilities themselves. Noting that theratio Pik /P j k depends on three words i, j, and k,
the most general model takes the form,
F (wi , w j , w̃k ) =

Pik
,
Pj k

(1)

where w ∈ Rd are word vectors and w̃ ∈ Rd
are separate context word vectors whose role will
be discussed in Section 4.2. In this equation, theright-hand side is extracted from the corpus, andF may depend on some as-of-yet unspecified parameters. The number of possibilities for F is vast,
but by enforcing a few desiderata we can select a
unique choice. First, we would like F to encode

the information present the ratio Pik /P j k in theword vector space. Since vector spaces are inherently linear structures, the most natural way to do
this is with vector differences. With this aim, we
can restrict our consideration to those functions Fthat depend only on the difference of the two targetwords, modifying Eqn. (1) to,
F (wi − w j , w̃k ) =

Pik
.
Pj k

(2)

Next, we note that the arguments of F in Eqn. (2)are vectors while the right-hand side is a scalar.
While F could be taken to be a complicated function parameterized by, e.g., a neural network, doing so would obfuscate the linear structure we aretrying to capture. To avoid this issue, we can firsttake the dot product of the arguments, Pik
F (wi − w j )T w̃k =
,
Pj k

(3)

which prevents F from mixing the vector dimensions in undesirable ways. Next, note that for
word-word co-occurrence matrices, the distinction
between a word and a context word is arbitrary andthat we are free to exchange the two roles. To do so
consistently, we must not only exchange w ↔ w̃but also X ↔ X T . Our final model should be invariant under this relabeling, but Eqn. (3) is not.
However, the symmetry can be restored in two
steps. First, we require that F be a homomorphism
between the groups (R, +) and (R>0 , ×), i.e., F (wTi w̃k )F (wi − w j )T w̃k =,F (wTj w̃k )(4)which, by Eqn. (3), is solved by,
F (wTi w̃k ) = Pik =

X ik
.
Xi

(5)

The solution to Eqn. (4) is F = exp, or,wTi w̃k = log(Pik ) = log(X ik ) − log(X i ) .

(6)

1.0

Next, we note that Eqn. (6) would exhibit the exchange symmetry if not for the log(X i ) on theright-hand side. However, this term is independent of k so it can be absorbed into a bias bi for
wi . Finally, adding an additional bias b̃k for w̃krestores the symmetry,wTi w̃k + bi + b̃k = log(X ik ) .

(7)

Eqn. (7) is a drastic simplification over Eqn. (1),but it is actually ill-defined since the logarithm diverges whenever its argument is zero. One resolution to this issue is to include an additive shift
in the logarithm, log(X ik ) → log(1 + X ik ), which
maintains the sparsity of X while avoiding the divergences. The idea of factorizing the log of theco-occurrence matrix is closely related to LSA and
we will use the resulting model as a baseline in
our experiments. A main drawback to this modelis that it weighs all co-occurrences equally, eventhose that happen rarely or never. Such rare cooccurrences are noisy and carry less information
than the more frequent ones — yet even just thezero entries account for 75–95% of the data in X,
depending on the vocabulary size and corpus.
We propose a new weighted least squares regression model that addresses these problems.
Casting Eqn. (7) as a least squares problem andintroducing a weighting function f (X i j ) into thecost function gives us the model
J=

V
X

 
2
f X i j wTi w̃ j + bi + b̃ j − log X i j ,i, j=1(8)
where V is the size of the vocabulary. The weighting function should obey the following properties:
1. f (0) = 0. If f is viewed as a continuousfunction, it should vanish as x → 0 fastenough that the lim x→0 f (x) log2 x is finite.
2. f (x) should be non-decreasing so that rareco-occurrences are not overweighted.
3. f (x) should be relatively small for large values of x, so that frequent co-occurrences are
not overweighted.
Of course a large number of functions satisfy theseproperties, but one class of functions that we foundto work well can be parameterized as,((x/x max ) α if x < x max
f (x) =
(9)1otherwise .

0.8
0.6
0.4
0.2
0.0

Figure 1: Weighting function f with α = 3/4.
The performance of the model depends weakly on
the cutoff, which we fix to x max = 100 for all our
experiments. We found that α = 3/4 gives a modest improvement over a linear version with α = 1.
Although we offer only empirical motivation forchoosing the value 3/4, it is interesting that a similar fractional power scaling was found to give thebest performance in (Mikolov et al., 2013a).
3.1

Relationship to Other Models

Because all unsupervised methods for learning
word vectors are ultimately based on the occurrence statistics of a corpus, there should be commonalities between the models. Nevertheless, certain models remain somewhat opaque in this regard, particularly the recent window-based methods like skip-gram and ivLBL. Therefore, in thissubsection we show how these models are related
to our proposed model, as defined in Eqn. (8).
The starting point for the skip-gram or ivLBL
methods is a model Q i j for the probability thatword j appears in the context of word i. For concreteness, let us assume that Q i j is a softmax,
exp(wTi w̃ j )Q i j = PV.
T w̃ )
exp(w
k
k=1
i

(10)

Most of the details of these models are irrelevant
for our purposes, aside from the the fact that theyattempt to maximize the log probability as a context window scans over the corpus. Training proceeds in an on-line, stochastic fashion, but the implied global objective function can be written as,
XJ=−log Q i j .
(11)
i ∈corpusj ∈context(i)Evaluating the normalization factor of the softmax for each term in this sum is costly. To allow for efficient training, the skip-gram and ivLBL
models introduce approximations to Q i j . However, the sum in Eqn. (11) can be evaluated much

more efficiently if we first group together thoseterms that have the same values for i and j,
J=−

V
V X
X

squared error of the logarithms of P̂ and Q̂ instead,XJˆ =X i log P̂i j − log Q̂ i j 2
i, j

X i j log Q i j ,

(12)

=

i=1 j=1

V
X
i=1

Xi

V
X

Pi j log Q i j =j=1

X i wTi w̃ j − log X i j 2 .

(15)

i, j

where we have used the fact that the number oflike terms is given by the co-occurrence matrix X.
P
Recalling our notation for X i = k X ik andPi j = X i j /X i , we can rewrite J as,
J=−

X

V
X

X i H (Pi ,Q i ) ,

i=1

(13)
where H (Pi ,Q i ) is the cross entropy of the distributions Pi and Q i , which we define in analogy
to X i . As a weighted sum of cross-entropy error,
this objective bears some formal resemblance tothe weighted least squares objective of Eqn. (8).
In fact, it is possible to optimize Eqn. (13) directly
as opposed to the on-line training methods used in
the skip-gram and ivLBL models. One could interpret this objective as a “global skip-gram” model,
and it might be interesting to investigate further.
On the other hand, Eqn. (13) exhibits a number ofundesirable properties that ought to be addressed
before adopting it as a model for learning word
vectors.
To begin, cross entropy error is just one among
many possible distance measures between probability distributions, and it has the unfortunate
property that distributions with long tails are often modeled poorly with too much weight given
to the unlikely events. Furthermore, for the measure to be bounded it requires that the model distribution Q be properly normalized. This presents
a computational bottleneck owing to the sum over
the whole vocabulary in Eqn. (10), and it would bedesirable to consider a different distance measurethat did not require this property of Q. A natural
choice would be a least squares objective in which
normalization factors in Q and P are discarded,XJˆ =X i P̂i j − Q̂ i j 2
(14)
i, j

where P̂i j = X i j and Q̂ i j = exp(wTi w̃ j ) are theunnormalized distributions. At this stage anotherproblem emerges, namely that X i j often takes verylarge values, which can complicate the optimization. An effective remedy is to minimize the

Finally, we observe that while the weighting factor
X i is preordained by the on-line training method
inherent to the skip-gram and ivLBL models, it isby no means guaranteed to be optimal. In fact,
Mikolov et al. (2013a) observe that performancecan be increased by filtering the data so as to reduce the effective value of the weighting factor for
frequent words. With this in mind, we introduce
a more general weighting function, which we arefree to take to depend on the context word as well.
The result is,
X

Jˆ =
(16)
f (X i j ) wTi w̃ j − log X i j 2 ,i, jwhich is equivalent1 to the cost function of
Eqn. (8), which we derived previously.
3.2

Complexity of the model

As can be seen from Eqn. (8) and the explicit formof the weighting function f (X ), the computational
complexity of the model depends on the number ofnonzero elements in the matrix X. As this number is always less than the total number of entries of the matrix, the model scales no worse thanO(|V | 2 ). At first glance this might seem like a substantial improvement over the shallow windowbased approaches, which scale with the corpus
size, |C|. However, typical vocabularies have hundreds of thousands of words, so that |V | 2 can be in
the hundreds of billions, which is actually muchlarger than most corpora. For this reason it is important to determine whether a tighter bound canbe placed on the number of nonzero elements of
X.
In order to make any concrete statements about
the number of nonzero elements in X, it is necessary to make some assumptions about the distribution of word co-occurrences. In particular, we willassume that the number of co-occurrences of word
i with word j, X i j , can be modeled as a power-law
function of the frequency rank of that word pair,ri j :
k
Xi j =.
(17)(r i j ) α
1 We could also include bias terms in Eqn. (16).

The total number of words in the corpus is proportional to the sum over all elements of the cooccurrence matrix X,
|C| ∼

X

Xi j =

|X|
X
k
r =1

ij

rα

= k H | X |,α ,(18)where we have rewritten the last sum in terms ofthe generalized harmonic number Hn, m . The upper limit of the sum, |X |, is the maximum frequency rank, which coincides with the number ofnonzero elements in the matrix X. This number isalso equal to the maximum value of r in Eqn. (17)such that X i j ≥ 1, i.e., |X | = k 1/α . Therefore we
can write Eqn. (18) as,|C| ∼ |X | α H | X |,α .

(19)

We are interested in how |X | is related to |C| whenboth numbers are large; therefore we are free toexpand the right hand side of the equation for large|X |. For this purpose we use the expansion of generalized harmonic numbers (Apostol, 1976),
H x, s =x 1−s+ ζ (s) + O(x −s ) if s > 0, s , 1 ,
1−s
(20)

giving,|C| ∼|X |+ ζ (α) |X | α + O(1) ,
1−α

(21)

where ζ (s) is the Riemann zeta function. In thelimit that X is large, only one of the two terms on
the right hand side of Eqn. (21) will be relevant,and which term that is depends on whether α > 1,(
O(|C|)
if α < 1,|X | =(22)
1/α
O(|C| ) if α > 1.
For the corpora studied in this article, we observethat X i j is well-modeled by Eqn. (17) with α =1.25. In this case we have that |X | = O(|C| 0.8 ).
Therefore we conclude that the complexity of themodel is much better than the worst case O(V 2 ),and in fact it does somewhat better than the on-line
window-based methods which scale like O(|C|).

4

Experiments

4.1

Evaluation methods

We conduct experiments on the word analogy
task of Mikolov et al. (2013a), a variety of word
similarity tasks, as described in (Luong et al.,2013), and on the CoNLL-2003 shared benchmark

Table 2: Results on the word analogy task, given
as percent accuracy. Underlined scores are best
within groups of similarly-sized models; boldscores are best overall. HPCA vectors are publicly
available2 ; (i)vLBL results are from (Mnih et al.,2013); skip-gram (SG) and CBOW results arefrom (Mikolov et al., 2013a,b); we trained SG†
and CBOW† using the word2vec tool3 . See text
for details and a description of the SVD models.
Model
ivLBL
HPCA
GloVe
SG
CBOW
vLBL
ivLBL
GloVe
SVD
SVD-S
SVD-L
CBOW†
SG†
GloVe
CBOW
SG
SVD-L
GloVe

Dim.
100
100
100
300
300
300
300
300
300
300
300
300
300
300
1000
1000
300
300

Size
1.5B
1.6B
1.6B
1B
1.6B
1.5B
1.5B
1.6B
6B
6B
6B
6B
6B
6B
6B
6B
42B
42B

Sem.
55.9
4.2
67.5
61
16.1
54.2
65.2
80.8
6.3
36.7
56.6
63.6
73.0
77.4
57.3
66.1
38.4
81.9

Syn.
50.1
16.4
54.3
61
52.6
64.8
63.0
61.5
8.1
46.6
63.0
67.4
66.0
67.0
68.9
65.1
58.2
69.3

Tot.
53.2
10.8
60.3
61
36.1
60.0
64.0
70.3
7.3
42.1
60.1
65.7
69.1
71.7
63.7
65.6
49.2
75.0

dataset for NER (Tjong Kim Sang and De Meulder, 2003).
Word analogies. The word analogy task consists of questions like, “a is to b as c is to ?”The dataset contains 19,544 such questions, divided into a semantic subset and a syntactic subset. The semantic questions are typically analogies
about people or places, like “Athens is to Greece
as Berlin is to ?”. The syntactic questions are
typically analogies about verb tenses or forms ofadjectives, for example “dance is to dancing as fly
is to ?”. To correctly answer the question, themodel should uniquely identify the missing term,with only an exact correspondence counted as acorrect match. We answer the question “a is to b
as c is to ?” by finding the word d whose representation wd is closest to wb − wa + wc according
to the cosine similarity.4
2 http://lebret.ch/words/
3 http://code.google.com/p/word2vec/
4 Levy et al. (2014) introduce a multiplicative analogy
evaluation, 3C OS M UL, and report an accuracy of 68.24% on

70

70

65

65

60

60

60

Accuracy [%]

55

50
40

100

200
300
400
Vector Dimension

500

55

50

Semantic
Syntactic
Overall30
20
0Accuracy [%]70

Accuracy [%]

80

45
600

(a) Symmetric context

40
2

50

Semantic
Syntactic
Overall
4

6Window Size

8(b) Symmetric context

Semantic
Syntactic
Overall

45
10

40
2

4

6
Window Size

810(c) Asymmetric context

Figure 2: Accuracy on the analogy task as function of vector size and window size/type. All models aretrained on the 6 billion token corpus. In (a), the window size is 10. In (b) and (c), the vector size is 100.
Word similarity. While the analogy task is ourprimary focus since it tests for interesting vector
space substructures, we also evaluate our model ona variety of word similarity tasks in Table 3. These
include WordSim-353 (Finkelstein et al., 2001),
MC (Miller and Charles, 1991), RG (Rubenstein
and Goodenough, 1965), SCWS (Huang et al.,2012), and RW (Luong et al., 2013).
Named entity recognition. The CoNLL-2003
English benchmark dataset for NER is a collection of documents from Reuters newswire articles,
annotated with four entity types: person, location,organization, and miscellaneous. We train models on CoNLL-03 training data on test on threedatasets: 1) ConLL-03 testing data, 2) ACE Phase
2 (2001-02) and ACE-2003 data, and 3) MUC7
Formal Run test set. We adopt the BIO2 annotation standard, as well as all the preprocessing steps
described in (Wang and Manning, 2013). We use acomprehensive set of discrete features that comeswith the standard distribution of the Stanford NERmodel (Finkel et al., 2005). A total of 437,905
discrete features were generated for the CoNLL2003 training dataset. In addition, 50-dimensional
vectors for each word of a five-word context areadded and used as continuous features. With thesefeatures as input, we trained a conditional random
field (CRF) with exactly the same setup as the
CRFjoin model of (Wang and Manning, 2013).
4.2

Corpora and training details

We trained our model on five corpora of varying
sizes: a 2010 Wikipedia dump with 1 billion tokens; a 2014 Wikipedia dump with 1.6 billion tokens; Gigaword 5 which has 4.3 billion tokens; thecombination Gigaword5 + Wikipedia2014, which
the analogy task. This number is evaluated on a subset of thedataset so it is not included in Table 2. 3C OS M UL performed
worse than cosine similarity in almost all of our experiments.

has 6 billion tokens; and on 42 billion tokens of
web data, from Common Crawl5 . We tokenize
and lowercase each corpus with the Stanford tokenizer, build a vocabulary of the 400,000 mostfrequent words6 , and then construct a matrix of cooccurrence counts X. In constructing X, we mustchoose how large the context window should be
and whether to distinguish left context from rightcontext. We explore the effect of these choices below. In all cases we use a decreasing weighting
function, so that word pairs that are d words apartcontribute 1/d to the total count. This is one way
to account for the fact that very distant word pairs
are expected to contain less relevant information
about the words’ relationship to one another.
For all our experiments, we set x max = 100,
α = 3/4, and train the model using AdaGrad(Duchi et al., 2011), stochastically sampling nonzero elements from X, with initial learning rate of
0.05. We run 50 iterations for vectors smaller than
300 dimensions, and 100 iterations otherwise (seeSection 4.6 for more details about the convergencerate). Unless otherwise noted, we use a context often words to the left and ten words to the right.
The model generates two sets of word vectors,
W and W̃ . When X is symmetric, W and W̃ areequivalent and differ only as a result of their random initializations; the two sets of vectors should
perform equivalently. On the other hand, there isevidence that for certain types of neural networks,
training multiple instances of the network and then
combining the results can help reduce overfitting
and noise and generally improve results (Ciresan
et al., 2012). With this in mind, we choose to use
5 To demonstrate the scalability of the model, we also
trained it on a much larger sixth corpus, containing 840 billion tokens of web data, but in this case we did not lowercase
the vocabulary, so the results are not directly comparable.
6 For the model trained on Common Crawl data, we use alarger vocabulary of about 2 million words.

the sum W + W̃ as our word vectors. Doing so typically gives a small boost in performance, with thebiggest increase in the semantic analogy task.
We compare with the published results of a variety of state-of-the-art models, as well as with
our own results produced using the word2vec
tool and with several baselines using SVDs. With
word2vec, we train the skip-gram (SG† ) andcontinuous bag-of-words (CBOW† ) models on the
6 billion token corpus (Wikipedia 2014 + Gigaword 5) with a vocabulary of the top 400,000 mostfrequent words and a context window size of 10.
We used 10 negative samples, which we show in
Section 4.6 to be a good choice for this corpus.
For the SVD baselines, we generate a truncatedmatrix Xtrunc which retains the information of how
frequently each word occurs with only the top
10,000 most frequent words. This step is typical of many matrix-factorization-based methods as
the extra columns can contribute a disproportionate number of zero entries and the methods areotherwise computationally expensive.
The singular vectors of this matrix constitutethe baseline “SVD”. We also evaluate two relatedbaselines:“SVD-S” in which we take the SVD of√
Xtrunc , and “SVD-L” in which we take the SVD
of log(1+ Xtrunc ). Both methods help compress theotherwise large range of values in X.7
4.3

Results

We present results on the word analogy task in Table 2. The GloVe model performs significantly
better than the other baselines, often with smaller
vector sizes and smaller corpora. Our results using the word2vec tool are somewhat better thanmost of the previously published results. This isdue to a number of factors, including our choice touse negative sampling (which typically works better than the hierarchical softmax), the number ofnegative samples, and the choice of the corpus.
We demonstrate that the model can easily betrained on a large 42 billion token corpus, with asubstantial corresponding performance boost. We
note that increasing the corpus size does not guarantee improved results for other models, as can beseen by the decreased performance of the SVD7 We also investigated several other weighting schemes fortransforming X; what we report here performed best. Many
weighting schemes like PPMI destroy the sparsity of X andtherefore cannot feasibly be used with large vocabularies.
With smaller vocabularies, these information-theoretic transformations do indeed work well on word similarity measures,
but they perform very poorly on the word analogy task.

Table 3: Spearman rank correlation on word similarity tasks. All vectors are 300-dimensional. The
CBOW∗ vectors are from the word2vec website
and differ in that they contain phrase vectors.
Model Size WS353 MC
SVD
6B
35.3 35.1
SVD-S 6B
56.5 71.5
SVD-L 6B
65.7 72.7
CBOW† 6B
57.2 65.6
SG†
6B
62.8 65.2
GloVe 6B
65.8 72.7
SVD-L 42B 74.0 76.4
GloVe 42B 75.9 83.6
CBOW∗ 100B 68.4 79.6

RG SCWS RW
42.5 38.3 25.6
71.0 53.6 34.7
75.1 56.5 37.0
68.2 57.0 32.5
69.7 58.1 37.2
77.8 53.9 38.1
74.1 58.3 39.9
82.9 59.6 47.8
75.4 59.4 45.5

L model on this larger corpus. The fact that thisbasic SVD model does not scale well to large corpora lends further evidence to the necessity of thetype of weighting scheme proposed in our model.
Table 3 shows results on five different word
similarity datasets. A similarity score is obtained
from the word vectors by first normalizing eachfeature across the vocabulary and then calculating the cosine similarity. We compute Spearman’srank correlation coefficient between this score andthe human judgments. CBOW∗ denotes the vectors available on the word2vec website that aretrained with word and phrase vectors on 100B
words of news data. GloVe outperforms it while
using a corpus less than half the size.
Table 4 shows results on the NER task with the
CRF-based model. The L-BFGS training terminates when no improvement has been achieved on
the dev set for 25 iterations. Otherwise all configurations are identical to those used by Wang andManning (2013). The model labeled Discrete is
the baseline using a comprehensive set of discretefeatures that comes with the standard distributionof the Stanford NER model, but with no word vector features. In addition to the HPCA and SVD
models discussed previously, we also compare to
the models of Huang et al. (2012) (HSMN) and
Collobert and Weston (2008) (CW). We trained
the CBOW model using the word2vec tool8 .
The GloVe model outperforms all other methodson all evaluation metrics, except for the CoNLL
test set, on which the HPCA method does slightly
better. We conclude that the GloVe vectors areuseful in downstream NLP tasks, as was first
8 We use the same parameters as above, except in this case
we found 5 negative samples to work slightly better than 10.

shown for neural vectors in (Turian et al., 2010).
4.4

Model Analysis: Vector Length andContext Size

In Fig. 2, we show the results of experiments thatvary vector length and context window. A context
window that extends to the left and right of a target word will be called symmetric, and one which
extends only to the left will be called asymmetric. In (a), we observe diminishing returns for vectors larger than about 200 dimensions. In (b) and(c), we examine the effect of varying the window
size for symmetric and asymmetric context windows. Performance is better on the syntactic subtask for small and asymmetric context windows,
which aligns with the intuition that syntactic information is mostly drawn from the immediate context and can depend strongly on word order. Semantic information, on the other hand, is more frequently non-local, and more of it is captured withlarger window sizes.
4.5

Model Analysis: Corpus Size

In Fig. 3, we show performance on the word analogy task for 300-dimensional vectors trained on
different corpora. On the syntactic subtask, there
is a monotonic increase in performance as the corpus size increases. This is to be expected sincelarger corpora typically produce better statistics.
Interestingly, the same trend is not true for the semantic subtask, where the models trained on thesmaller Wikipedia corpora do better than thosetrained on the larger Gigaword corpus. This is
likely due to the large number of city- and countrybased analogies in the analogy dataset and the factthat Wikipedia has fairly comprehensive articles
for most such locations. Moreover, Wikipedia’s

Semantic

Syntactic

Overall85
80

Accuracy [%]Table 4: F1 score on NER task with 50d vectors.
Discrete is the baseline without word vectors. We
use publicly-available vectors for HPCA, HSMN,
and CW. See text for details.
Model
Dev Test ACE MUC7
Discrete 91.0 85.4 77.4
73.4
SVD
90.8 85.7 77.3
73.7
SVD-S 91.0 85.5 77.6
74.3
SVD-L 90.5 84.8 73.6
71.5
HPCA 92.6 88.7 81.7
80.7
HSMN 90.5 85.7 78.7
74.7
CW
92.2 87.4 81.7
80.2
CBOW 93.1 88.2 82.2
81.1
GloVe
93.2 88.3 82.9
82.2

75
70
65
60
55
50
Wiki2010
1B tokensWiki2014
1.6B tokens

Gigaword5
4.3B tokens

Gigaword5 +Wiki2014
6B tokens

Common Crawl
42B tokens

Figure 3: Accuracy on the analogy task for 300dimensional vectors trained on different corpora.
entries are updated to assimilate new knowledge,
whereas Gigaword is a fixed news repository withoutdated and possibly incorrect information.
4.6

Model Analysis: Run-time

The total run-time is split between populating X
and training the model. The former depends on
many factors, including window size, vocabularysize, and corpus size. Though we did not do so,
this step could easily be parallelized across multiple machines (see, e.g., Lebret and Collobert(2014) for some benchmarks). Using a single
thread of a dual 2.1GHz Intel Xeon E5-2658 machine, populating X with a 10 word symmetriccontext window, a 400,000 word vocabulary, anda 6 billion token corpus takes about 85 minutes.
Given X, the time it takes to train the model depends on the vector size and the number of iterations. For 300-dimensional vectors with the above
settings (and using all 32 cores of the above machine), a single iteration takes 14 minutes. See
Fig. 4 for a plot of the learning curve.
4.7

Model Analysis: Comparison withword2vecA rigorous quantitative comparison of GloVe withword2vec is complicated by the existence ofmany parameters that have a strong effect on performance. We control for the main sources of variation that we identified in Sections 4.4 and 4.5 bysetting the vector length, context window size, corpus, and vocabulary size to the configuration mentioned in the previous subsection.
The most important remaining variable to control for is training time. For GloVe, the relevant parameter is the number of training iterations.
For word2vec, the obvious choice would be thenumber of training epochs. Unfortunately, thecode is currently designed for only a single epoch:

72

Training Time (hrs)

1

2

3

4

5

6

72

68

Accuracy [%]Accuracy [%]6

Training Time (hrs)
9

12

15

18

21

24

70

70

GloVe
CBOW

66

68
66

64

64

62

62

60

3

5

10

15

Iterations (GloVe)

1 3 5 7 10 15 20

25 30

20

Negative Samples (CBOW)

25
40

60
50

(a) GloVe vs CBOW

GloVe
Skip-Gram

20

40

60

80

100

10 12

15

20

Iterations (GloVe)

1 2 3 4 5 6

7

Negative Samples (Skip-Gram)(b) GloVe vs Skip-Gram

Figure 4: Overall accuracy on the word analogy task as a function of training time, which is governed by
the number of iterations for GloVe and by the number of negative samples for CBOW (a) and skip-gram(b). In all cases, we train 300-dimensional vectors on the same 6B token corpus (Wikipedia 2014 +Gigaword 5) with the same 400,000 word vocabulary, and use a symmetric context window of size 10.
it specifies a learning schedule specific to a single
pass through the data, making a modification for
multiple passes a non-trivial task. Another choice
is to vary the number of negative samples. Adding
negative samples effectively increases the numberof training words seen by the model, so in someways it is analogous to extra epochs.
We set any unspecified parameters to their default values, assuming that they are close to optimal, though we acknowledge that this simplification should be relaxed in a more thorough analysis.
In Fig. 4, we plot the overall performance on
the analogy task as a function of training time.
The two x-axes at the bottom indicate the corresponding number of training iterations for GloVeand negative samples for word2vec. We notethat word2vec’s performance actually decreases
if the number of negative samples increases beyond about 10. Presumably this is because thenegative sampling method does not approximate
the target probability distribution well.9
For the same corpus, vocabulary, window size,
and training time, GloVe consistently outperforms
word2vec. It achieves better results faster, andalso obtains the best results irrespective of speed.

5Conclusion

Recently, considerable attention has been focusedon the question of whether distributional word
representations are best learned from count-based
9 In contrast, noise-contrastive estimation is an approximation which improves with more negative samples. In Table 1 of (Mnih et al., 2013), accuracy on the analogy task is a
non-decreasing function of the number of negative samples.

methods or from prediction-based methods. Currently, prediction-based models garner substantial
support; for example, Baroni et al. (2014) argue
that these models perform better across a range oftasks. In this work we argue that the two classesof methods are not dramatically different at a fundamental level since they both probe the underlying co-occurrence statistics of the corpus, but
the efficiency with which the count-based methods capture global statistics can be advantageous.
We construct a model that utilizes this main benefit of count data while simultaneously capturing
the meaningful linear substructures prevalent inrecent log-bilinear prediction-based methods likeword2vec. The result, GloVe, is a new global
log-bilinear regression model for the unsupervised
learning of word representations that outperforms
other models on word analogy, word similarity,and named entity recognition tasks.

Acknowledgments
We thank the anonymous reviewers for their valuable comments. Stanford University gratefully
acknowledges the support of the Defense Threat
Reduction Agency (DTRA) under Air Force Research Laboratory (AFRL) contract no. FA865010-C-7020 and the Defense Advanced Research
Projects Agency (DARPA) Deep Exploration andFiltering of Text (DEFT) Program under AFRL
contract no. FA8750-13-2-0040. Any opinions,findings, and conclusion or recommendations expressed in this material are those of the authors anddo not necessarily reflect the view of the DTRA,
AFRL, DEFT, or the US government.

References
Tom M. Apostol. 1976. Introduction to Analytic
Number Theory. Introduction to Analytic Number Theory.
Marco Baroni, Georgiana Dinu, and GermánKruszewski. 2014. Don’t count, predict! A
systematic comparison of context-counting vs.
context-predicting semantic vectors. In ACL.
Yoshua Bengio. 2009. Learning deep architectures
for AI. Foundations and Trends in Machine
Learning.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. JMLR, 3:1137–1155.
John A. Bullinaria and Joseph P. Levy. 2007. Extracting semantic representations from word cooccurrence statistics: A computational study.
Behavior Research Methods, 39(3):510–526.
Dan C. Ciresan, Alessandro Giusti, Luca M. Gambardella, and Jürgen Schmidhuber. 2012. Deepneural networks segment neuronal membranes
in electron microscopy images. In NIPS, pages
2852–2860.

Word Representations via Global Context andMultiple Word Prototypes. In ACL.
Rémi Lebret and Ronan Collobert. 2014. Word
embeddings through Hellinger PCA. In EACL.
Omer Levy, Yoav Goldberg, and Israel RamatGan. 2014. Linguistic regularities in sparse andexplicit word representations. CoNLL-2014.
Kevin Lund and Curt Burgess. 1996. Producing
high-dimensional semantic spaces from lexical
co-occurrence. Behavior Research Methods, Instrumentation, and Computers, 28:203–208.
Minh-Thang Luong, Richard Socher, and Christopher D Manning. 2013. Better word representations with recursive neural networks for morphology. CoNLL-2013.
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient Estimation of Word
Representations in Vector Space. In ICLR Workshop Papers.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg
Corrado, and Jeffrey Dean. 2013b. Distributed
representations of words and phrases and theircompositionality. In NIPS, pages 3111–3119.

Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160–167.

Tomas Mikolov, Wen tau Yih, and Geoffrey
Zweig. 2013c. Linguistic regularities in continuous space word representations. In HLTNAACL.

Ronan Collobert, Jason Weston, Léon Bottou,
Michael Karlen, Koray Kavukcuoglu, and Pavel
Kuksa. 2011. Natural Language Processing (Almost) from Scratch. JMLR, 12:2493–2537.

George A. Miller and Walter G. Charles. 1991.
Contextual correlates of semantic similarity.
Language and cognitive processes, 6(1):1–28.

Scott Deerwester, Susan T. Dumais, George W.
Furnas, Thomas K. Landauer, and Richard
Harshman. 1990. Indexing by latent semantic
analysis. Journal of the American Society for
Information Science, 41.
John Duchi, Elad Hazan, and Yoram Singer. 2011.
Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 12.
Lev Finkelstein, Evgenly Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman,
and Eytan Ruppin. 2001. Placing search in context: The concept revisited. In Proceedingsof the 10th international conference on World
Wide Web, pages 406–414. ACM.
Eric H. Huang, Richard Socher, Christopher D.
Manning, and Andrew Y. Ng. 2012. Improving

Andriy Mnih and Koray Kavukcuoglu. 2013.
Learning word embeddings efficiently withnoise-contrastive estimation. In NIPS.
Douglas L. T. Rohde, Laura M. Gonnerman,
and David C. Plaut. 2006.
An improved
model of semantic similarity based on lexical
co-occurence. Communications of the ACM,
8:627–633.
Herbert Rubenstein and John B. Goodenough.
1965. Contextual correlates of synonymy. Communications of the ACM, 8(10):627–633.
Fabrizio Sebastiani. 2002. Machine learning in automated text categorization. ACM Computing
Surveys, 34:1–47.
Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013. Parsing With
Compositional Vector Grammars. In ACL.

Stefanie Tellex, Boris Katz, Jimmy Lin, Aaron
Fernandes, and Gregory Marton. 2003. Quantitative evaluation of passage retrieval algorithms
for question answering. In Proceedings of the
SIGIR Conference on Research and Development in Informaion Retrieval.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003
shared task: Language-independent named entity recognition. In CoNLL-2003.
Joseph Turian, Lev Ratinov, and Yoshua Bengio.
2010. Word representations: a simple and general method for semi-supervised learning. In
Proceedings of ACL, pages 384–394.
Mengqiu Wang and Christopher D. Manning.
2013. Effect of non-linear deep architecture insequence labeling. In Proceedings of the 6thInternational Joint Conference on Natural Language Processing (IJCNLP).