what are the keypoints:

6.8.2 Learning skip-gram embeddings
The learning algorithm for skip-gram embeddings takes as input a corpus of text, and a chosen vocabulary size N. It begins by assigning a random embedding vector for each of the N vocabulary words, and then proceeds to iteratively shift the em- bedding of each word w to be more like the embeddings of words that occur nearby in texts, and less like the embeddings of words that don’t occur nearby. Let’s start by considering a single piece of training data:
... lemon,  a [tablespoon of apricot jam,      a] pinch ...
               c1         c2    w     c3       c4
This example has a target word w (apricot), and 4 context words in the L = ±2 window, resulting in 4 positive training instances (on the left below):
positive examples +
w cpos apricot tablespoon
apricot of apricot jam apricot a
negative examples -
w cneg apricot aardvark apricot my apricot where apricot coaxial
w cneg apricot seven apricot forever apricot dear apricot if
  For training a binary classifier we also need negative examples. In fact skip- gram with negative sampling (SGNS) uses more negative examples than positive examples (with the ratio between them set by a parameter k). So for each of these (w,cpos) training instances we’ll create k negative samples, each consisting of the target w plus a ‘noise word’ cneg. A noise word is a random word from the lexicon, constrained not to be the target word w. The right above shows the setting where k = 2, so we’ll have 2 negative examples in the negative training set − for each positive example w,cpos.
The noise words are chosen according to their weighted unigram frequency pα (w), where α is a weight. If we were sampling according to unweighted fre- quency p(w), it would mean that with unigram probability p(“the”) we would choose the word the as a noise word, with unigram probability p(“aardvark”) we would choose aardvark, and so on. But in practice it is common to set α = .75, i.e. use the
3 weighting p4 (w):
6.8 • WORD2VEC 21
count (w)α
Pα(w)= 􏰭w′ count(w′)α (6.32)
 Setting α = .75 gives better performance because it gives rare noise words slightly higher probability: for rare words, Pα (w) > P(w). To illustrate this intuition, it might help to work out the probabilities for an example with two events, P(a) = .99 and P(b) = .01:
.99.75
Pα (a) = .99.75 + .01.75 = .97
.01.75
Pα (b) = .99.75 + .01.75 = .03 (6.33)
Given the set of positive and negative training instances, and an initial set of embed- dings, the goal of the learning algorithm is to adjust those embeddings to
• Maximize the similarity of the target word, context word pairs (w,cpos) drawn from the positive examples
• Minimize the similarity of the (w,cneg) pairs from the negative examples.
If we consider one word/context pair (w, cpos ) with its k noise words cneg1 ...cnegk , we can express these two goals as the following loss function L to be minimized (hence the −); here the first term expresses that we want the classifier to assign the real context word cpos a high probability of being a neighbor, and the second term expressesthatwewanttoassigneachofthenoisewordscnegi ahighprobabilityof being a non-neighbor, all multiplied because we assume independence:
  􏰹k􏰺 LCE = − log P(+|w, cpos ) 􏰯 P(−|w, cnegi )
i=1 􏰹k􏰺
= − logP(+|w,cpos)+􏰮logP(−|w,cnegi) i=1
􏰹k􏰺 = − logP(+|w,cpos)+􏰮log􏰵1−P(+|w,cnegi)􏰶
i=1 􏰹k􏰺
= − logσ(cpos ·w)+􏰮logσ(−cnegi ·w) i=1
(6.34)
That is, we want to maximize the dot product of the word with the actual context words, and minimize the dot products of the word with the k negative sampled non- neighbor words.
We minimize this loss function using stochastic gradient descent. Fig. 6.14 shows the intuition of one step of learning.
To get the gradient, we need to take the derivative of Eq. 6.34 with respect to the different embeddings. It turns out the derivatives are the following (we leave the

22
CHAPTER 6 • VECTOR SEMANTICS AND EMBEDDINGS
     aardvark
W
!
C
k=2
zebra
apricot
zebra aardvark
w
 jam
matrix
Tolstoy
cpos
cneg1
cneg2
  move apricot and jam closer, increasing cpos 􏰴 w
“...apricot jam...”
move apricot and matrix apart decreasing cneg1 􏰴 w
move apricot and Tolstoy apart decreasing cneg2 􏰴 w
   Figure 6.14
target embedding context embedding
Just as in logistic regression, then, the learning algorithm starts with randomly ini- tialized W and C matrices, and then walks through the training corpus using gradient descent to move W and C so as to minimize the loss in Eq. 6.34 by making the up- dates in (Eq. 6.38)-(Eq. 6.40).
Recall that the skip-gram model learns two separate embeddings for each word i: the target embedding w and the context embedding c , stored in two matrices, the
Intuition of one step of gradient descent. The skip-gram model tries to shift embeddings so the target embeddings (here for apricot) are closer to (have a higher dot prod- uct with) context embeddings for nearby words (here jam) and further from (lower dot product with) context embeddings for noise words that don’t occur nearby (here Tolstoy and matrix).
proof as an exercise at the end of the chapter):
are thus:
∂LCE = ∂ cpos ∂LCE = ∂cneg
ct+1 = ct −η[σ(ct ·wt)−1]wt pos pos pos
ct+1 = ct −η[σ(ct ·wt)]wt neg neg neg
􏰹k􏰺 wt+1 = wt −η [σ(ctpos ·wt)−1]ctpos +􏰮[σ(ctnegi ·wt)]ctnegi
i=1
∂LCE
∂w =
􏰮k
[σ (cnegi · w)]cnegi
[σ (cpos · w) − 1]w [σ (cneg · w)]w
(6.35) (6.36)
  [σ (cpos · w) − 1]cpos +
The update equations going from time step t to t + 1 in stochastic gradient descent
 ii
target matrix W and the context matrix C. It’s common to just add them together, representing word i with the vector wi +ci. Alternatively we can throw away the C matrix and just represent each word i by the vector wi.
As with the simple count-based methods like tf-idf, the context window size L affects the performance of skip-gram embeddings, and experiments often tune the parameter L on a devset.
i=1
(6.37)
(6.38) (6.39)
(6.40)

      Rohde, Gonnerman, Plaut
WRIST ARAMNKLE
A fasttext open-source library, including pretrained embeddings for 157 languages,
SHOULDER FINGER
also capturing the linear structures used by methods like word2vec.
BRAZIL
EYE FACE
TOE LEG FOOT
TOOTH NOHSEAD
MOUSE
DOG CAT
PUPPY KITTEN
s
EARHAND
OW
ematical relationship with sparse embeddings like PPMI, in which word2vec can be
HAWAII
OKYO
seen as implicitly Toptimizing a shifted version of a PPMI matrix (Levy and Gold-
6.9
Visualizing Embeddings
COW
fasttext
6.9 • VISUALIZING EMBEDDINGS 23 6.8.3 Other kinds of static embeddings
There are many kinds of static embeddings. An extension of word2vec, fasttext (Bojanowski et al., 2017), addresses a problem with word2vec as we have presented it so far: it has no good way to deal with unknown words—words that appear in a test corpus but were unseen in the training corpus. A related problem is word sparsity, such as in languages with rich morphology, where some of the many forms for each noun and verb may only occur rarely. Fasttext deals with these problems by using subword models, representing each word as itself plus a bag of constituent n-grams, with special boundary symbols < and > added to each word. For example, with n = 3 the word where would be represented by the sequence <where> plus the character n-grams:
     <wh, whe, her, ere, re>
Then a skipgram embedding is learned for each constituent n-gram, and the word
where is represented by the sum of all of the embeddings of its constituent n-grams.
Unknown words can then be presented only by the sum of the constituent n-grams.
TURTLE
LION
OYSTER
NASHVILLE
Modeling Word Meaning Using Lexical Co-Occurrence
is available at https://fasttext.cc.
Another very widely used static embedding model is GloVe (Pennington et al.,
2014), short for Global Vectors, because the model is based on capturing global RUSSIA
corpus statistics. GloVe is based on ratios of probabilities from the word-word co-
It turns out that d
e embeddings like word2vec actually have an elegant math-
berg, 2014c).
CH
INA EUROPE
FRANCE
occurrence matrix, combining the intuitions of count-based models like PPMI while
ASIA
AFRICA AMERICA
e
M
O
SC
MONTREAL
CHICAGO ATLANTA
n
“I see well in many dimensions as long as the dimensions are around two.” BULL The late economist Martin Shubik
Figure 8: Multidimensional scaling for three noun classes.
Visualizing embeddings is an important goal in helping understand, apply, and improve these models of word meaning. But how can we visualize a (for example) 100-dimensional vector?
WRIST
ANKLE SHOULDER
ARM LEG
HAND FOOT
HEAD NOSE
FINGER TOE FACE
EAR EYE
TOOTH DOG
CAT
PUPPY
KITTEN COW
MOUSE
TURTLE
OYSTER LION
BULL CHICAGO
The simplest way to visualize the meaning of a word w embedded in a space is to list the most similar words to w by sorting the vectors for all words in the vocabulary by their cosine with the vector for w. For example the 7 closest words to frog using a particular embeddings computed with the GloVe algorithm are: frogs, toad, litoria, leptodactyli- dae, rana, lizard, and eleutherodactylus (Pennington et al., 2014).
Yet another visualization method is to use a clustering algorithm to show a hierarchical representation of which words are similar to others in the embedding space. The uncaptioned figure on the left uses hierarchical clustering of some embedding vectors for nouns as a visualization
       ATLANTA MONTREAL
  CHINA
RUSSIA AFRICA ASIA
EUROPE AMERICA
NASHVILLE TOKYO
  Figure 9: Hierarchical clustering for three noun classes using distances based on vector correlations.
BRAZIL MOSCOW
FRANCE
HAWAII

24 CHAPTER 6
• VECTOR SEMANTICS AND EMBEDDINGS
method (Rohde et al., 2006).
Probably the most common visualization method, how-
ever, is to project the 100 dimensions of a word down into 2 dimensions. Fig. 6.1 showed one such visualization, as does Fig. 6.16, using a projection method called t-SNE (van der
Maaten and Hinton, 2008).
6.10 Semantic properties of embeddings
first-order co-occurrence
second-order co-occurrence
parallelogram model
In this section we briefly summarize some of the semantic properties of embeddings that have been studied.
Different types of similarity or association: One parameter of vector semantic models that is relevant to both sparse tf-idf vectors and dense word2vec vectors is the size of the context window used to collect counts. This is generally between 1 and 10 words on each side of the target word (for a total context of 2-20 words).
The choice depends on the goals of the representation. Shorter context windows tend to lead to representations that are a bit more syntactic, since the information is coming from immediately nearby words. When the vectors are computed from short context windows, the most similar words to a target word w tend to be semantically similar words with the same parts of speech. When vectors are computed from long context windows, the highest cosine words to a target word w tend to be words that are topically related but not similar.
For example Levy and Goldberg (2014a) showed that using skip-gram with a window of ±2, the most similar words to the word Hogwarts (from the Harry Potter series) were names of other fictional schools: Sunnydale (from Buffy the Vampire Slayer) or Evernight (from a vampire series). With a window of ±5, the most similar words to Hogwarts were other words topically related to the Harry Potter series: Dumbledore, Malfoy, and half-blood.
It’s also often useful to distinguish two kinds of similarity or association between words (Schu ̈tze and Pedersen, 1993). Two words have first-order co-occurrence (sometimes called syntagmatic association) if they are typically nearby each other. Thus wrote is a first-order associate of book or poem. Two words have second-order co-occurrence (sometimes called paradigmatic association) if they have similar neighbors. Thus wrote is a second-order associate of words like said or remarked.
Analogy/Relational Similarity: Another semantic property of embeddings is their ability to capture relational meanings. In an important early vector space model of cognition, Rumelhart and Abrahamson (1973) proposed the parallelogram model for solving simple analogy problems of the form a is to b as a* is to what?. In such problems, a system is given a problem like apple:tree::grape:?, i.e., apple is to tree as grape is to   , and must fill in the word vine. In the parallelogram
model, illustrated in Fig. 6.15, the vector from the word apple to the word tree (=
#»
In early work with sparse embeddings, scholars showed that sparse vector mod- els of meaning could solve such analogy problems (Turney and Littman, 2005), but the parallelogram method received more modern attention because of its suc-
#»#»
tree − apple) is added to the vector for grape (grape); the nearest word to that point is returned.
cess with word2vec or GloVe vectors (Mikolov et al. 2013c, Levy and Goldberg #»
2014b, Pennington et al. 2014). For example, the result of the expression king −

6.10 • SEMANTIC PROPERTIES OF EMBEDDINGS 25
    apple
grape
tree
vine
  Figure 6.15
The parallelogram model for analogy problems (Rumelhart and Abrahamson,
#» #»#»
#»#» #»
#»
1973): the location of vine can be found by subtracting apple from tree and adding grape.
man + woman is a vector close to queen. Similarly, Paris − France + Italy results #»
in a vector that is close to Rome. The embedding model thus seems to be extract- ing representations of relations like MALE-FEMALE, or CAPITAL-CITY-OF, or even COMPARATIVE/SUPERLATIVE, as shown in Fig. 6.16 from GloVe.
Relational properties of the GloVe vector space, shown by projecting vectors onto two dimen-
#»
#»#»#»
     (a) (b)
  Figure 6.16
#»#»#»
sions. (a) king − man + woman is close to queen. (b) offsets seem to capture comparative and superlative morphology (Pennington et al., 2014).
For a a : b :: a∗ : b∗ problem, meaning the algorithm is given vectors a, b, and a∗ and must find b∗, the parallelogram method is thus:
bˆ∗ =argmin distance(x,b−a+a∗) (6.41) x
with some distance function, such as Euclidean distance.
There are some caveats. For example, the closest value returned by the paral-
lelogram algorithm in word2vec or GloVe embedding spaces is usually not in fact b* but one of the 3 input words or their morphological variants (i.e., cherry:red :: potato:x returns potato or potatoes instead of brown), so these must be explicitly excluded. Furthermore while embedding spaces perform well if the task involves frequent words, small distances, and certain relations (like relating countries with their capitals or verbs/nouns with their inflected forms), the parallelogram method with embeddings doesn’t work as well for other relations (Linzen 2016, Gladkova et al. 2016, Schluter 2018, Ethayarajh et al. 2019a), and indeed Peterson et al. (2020) argue that the parallelogram method is in general too simple to model the human cognitive process of forming analogies of this kind.

26 CHAPTER 6 • VECTOR SEMANTICS AND EMBEDDINGS
6.10.1 Embeddings and Historical Semantics
Embeddings can also be a useful tool for studying how meaning changes over time,
by computing multiple embedding spaces, each from texts written in a particular time period. For example Fig. 6.17 shows a visualization of changes in meaning in English words over the last two centuries, computed by building separate embedding CspHaAcePsTfoErRea5càh DdeYcNadAeMfrIoCmShOisCtoIrAicLalRcEoPrpRoEraSlEikNeTGAoToIgOleNnS-gOraFmWs O(LRinDeMt aEl.A, 2N0I1N2G) à9 and the Corpus of Historical American English (Davies, 2012).
      Figure 6.17
allocational harm
‘father’ is to ‘doctor’ as ‘mother’ is to ‘nurse’. This could result in what Crawford scale case-studies using temporal topic modelsà 8à construct point-wise mutual
amplification
amplify bias; gendered terms become more gendered in embedding space than they were in the input text statistics (Zhao et al. 2017, Ethayarajh et al. 2019b, Jia et al. 2020), and biases are more exaggerated than in actual labor employment statistics (Garg et al., 2018).
Embeddings also encode the implicit associations that are a property of human reasoning. The Implicit Association Test (Greenwald et al., 1998) measures peo-
A t-SNE visualization of the semantic change of 3 words in English using
Figure 5à: Two-dimensional visualization of semantic change in English using SGNS word2vec vectors. The modern sense of each word, and the grey context words, are com-
vectors see Section 5à8 for the visualization algorithm à A, The word gay shifted puted from the most recent (modern) time-point embedding space. Earlier points are com-
from meaning ÒcheerfulÓ or ÒfrolicsomeÓ to referring to homosexualityà A, In the early puted from earlier historical embedding spaces. The visualizations show the changes in the
2àth century broadcast referred to Òcasting out seedsÓà with the rise of television and word gay from meanings related to “cheerful” or “frolicsome” to referring to homosexuality,
radio its meaning shifted to Òtransmitting signalsÓà C, Awful underwent a process of the development of the modern “transmission” sense of broadcast from its original sense of
pejoration, as it shifted from meaning Òfull of aweÓ to meaning Òterrible or appallingÓ
sowing seeds, and the pejoration of the word awful as it shifted from meaning “full of awe” t2om2eàaning “terrible or appalling” (Hamilton et al., 2016).
that adverbials eàgà, actually have a general tendency to undergo subjectiÞcation where they shift from objective statements about the world eàgà, ÒSorry, the car is
6.11 Bias and Embeddings
actuallybrokenÓ tosubjectivestatements eàgà,ÒIcanÕtbelieveheactuallydidthatÓ,
indicating surprise disbelief à
In addition to their ability to learn word meaning from text, embeddings, alas, also reproduce the implicit biases and stereotypes that were latent in the text. As
5à2à2 Computational linguistic studies
the prior section just showed, embeddings can roughly model relational similar- ity: ‘queen’ as the closest word to ‘king’ - ‘man’ + ‘woman’ implies the analogy
There are also a number of recent works analyzing semantic change using computational
man:woman::king:queen. But these same embedding analogies also exhibit gender methodsà 2àà use latent semantic analysis to analyze how word meanings broaden
stereotypes. For example Bolukbasi et al. (2016) find that the closest occupation atond‘cnoamrprouwteropverorgtriamemàer’3-‘umsaenr’a+w‘cwoo-omcacnu’rriennwceorvde2cvtoercsetmobpeedrdfoinrgmstaraniunemdboenrof
news text is ‘homemaker’, and that the embeddings similarly suggest the analogy
historical case-studies on semantic change, and 252 perform a similar set of small-
(2017) and Blodgett et al. (2020) call an allocational harm, when a system allo- information-based embeddings and found that semantic changes uncovered by their
cates resources (jobs or credit) unfairly to different groups. For example algorithms
mtheathuosdeheamdbreedadsiongasbalesapgareteomfeanstewaritchhfourmhainrinjugdpgomtentisaàl pr2o9granmdmers9oursdeoÒcntoeursralÓ
might thus incorrectly downweight documents with women’s names.
word-embedding methods to detect linguistic change pointsà Finally, 25à analyze
It turns out that embeddings don’t just reflect the statistics of their input, but also
bias historical co-occurrences to test whether synonyms tend to change in similar waysà

representational harm
debiasing
6.12 • EVALUATING VECTOR MODELS 27
ple’s associations between concepts (like ‘flowers’ or ‘insects’) and attributes (like ‘pleasantness’ and ‘unpleasantness’) by measuring differences in the latency with which they label words in the various categories.7 Using such methods, people in the United States have been shown to associate African-American names with unpleasant words (more than European-American names), male names more with mathematics and female names with the arts, and old people’s names with unpleas- ant words (Greenwald et al. 1998, Nosek et al. 2002a, Nosek et al. 2002b). Caliskan et al. (2017) replicated all these findings of implicit associations using GloVe vectors and cosine similarity instead of human latencies. For example African-American names like ‘Leroy’ and ‘Shaniqua’ had a higher GloVe cosine with unpleasant words while European-American names (‘Brad’, ‘Greg’, ‘Courtney’) had a higher cosine with pleasant words. These problems with embeddings are an example of a repre- sentational harm (Crawford 2017, Blodgett et al. 2020), which is a harm caused by a system demeaning or even ignoring some social groups. Any embedding-aware al- gorithm that made use of word sentiment could thus exacerbate bias against African Americans.
Recent research focuses on ways to try to remove these kinds of biases, for example by developing a transformation of the embedding space that removes gen- der stereotypes but preserves definitional gender (Bolukbasi et al. 2016, Zhao et al. 2017) or changing the training procedure (Zhao et al., 2018). However, although these sorts of debiasing may reduce bias in embeddings, they do not eliminate it (Gonen and Goldberg, 2019), and this remains an open problem.
Historical embeddings are also being used to measure biases in the past. Garg et al. (2018) used embeddings from historical texts to measure the association be- tween embeddings for occupations and embeddings for names of various ethnici- ties or genders (for example the relative cosine similarity of women’s names versus men’s to occupation words like ‘librarian’ or ‘carpenter’) across the 20th century. They found that the cosines correlate with the empirical historical percentages of women or ethnic groups in those occupations. Historical embeddings also repli- cated old surveys of ethnic stereotypes; the tendency of experimental participants in 1933 to associate adjectives like ‘industrious’ or ‘superstitious’ with, e.g., Chinese ethnicity, correlates with the cosine between Chinese last names and those adjectives using embeddings trained on 1930s text. They also were able to document historical gender biases, such as the fact that embeddings for adjectives related to competence (‘smart’, ‘wise’, ‘thoughtful’, ‘resourceful’) had a higher cosine with male than fe- male words, and showed that this bias has been slowly decreasing since 1960. We return in later chapters to this question about the role of bias in natural language processing.
6.12 Evaluating Vector Models
The most important evaluation metric for vector models is extrinsic evaluation on tasks, i.e., using vectors in an NLP task and seeing whether this improves perfor- mance over some other model.
7 Roughly speaking, if humans associate ‘flowers’ with ‘pleasantness’ and ‘insects’ with ‘unpleasant- ness’, when they are instructed to push a green button for ‘flowers’ (daisy, iris, lilac) and ‘pleasant words’ (love, laughter, pleasure) and a red button for ‘insects’ (flea, spider, mosquito) and ‘unpleasant words’ (abuse, hatred, ugly) they are faster than in an incongruous condition where they push a red button for ‘flowers’ and ‘unpleasant words’ and a green button for ‘insects’ and ‘pleasant words’.
 
28 CHAPTER 6 • VECTOR SEMANTICS AND EMBEDDINGS
Nonetheless it is useful to have intrinsic evaluations. The most common metric is to test their performance on similarity, computing the correlation between an algorithm’s word similarity scores and word similarity ratings assigned by humans. WordSim-353 (Finkelstein et al., 2002) is a commonly used set of ratings from 0 to 10 for 353 noun pairs; for example (plane, car) had an average score of 5.77. SimLex-999 (Hill et al., 2015) is a more difficult dataset that quantifies similarity (cup, mug) rather than relatedness (cup, coffee), and including both concrete and abstract adjective, noun and verb pairs. The TOEFL dataset is a set of 80 questions, each consisting of a target word with 4 additional word choices; the task is to choose which is the correct synonym, as in the example: Levied is closest in meaning to: imposed, believed, requested, correlated (Landauer and Dumais, 1997). All of these datasets present words without context.
Slightly more realistic are intrinsic similarity tasks that include context. The Stanford Contextual Word Similarity (SCWS) dataset (Huang et al., 2012) and the Word-in-Context (WiC) dataset (Pilehvar and Camacho-Collados, 2019) offer richer evaluation scenarios. SCWS gives human judgments on 2,003 pairs of words in their sentential context, while WiC gives target words in two sentential contexts that are either in the same or different senses; see Chapter 23. The semantic textual similarity task (Agirre et al. 2012, Agirre et al. 2015) evaluates the performance of sentence- level similarity algorithms, consisting of a set of pairs of sentences, each pair with human-labeled similarity scores.
Another task used for evaluation is the analogy task, discussed on page 24, where thesystemhastosolveproblemsoftheformaistobasa*istob*,givena,b,anda*
and having to find b* (Turney and Littman, 2005). A number of sets of tuples have
been created for this task, (Mikolov et al. 2013a, Mikolov et al. 2013c, Gladkova
et al. 2016), covering morphology (city:cities::child:children), lexicographic rela-
tions (leg:table::spout:teapot) and encyclopedia relations (Beijing:China::Dublin:Ireland), some drawing from the SemEval-2012 Task 2 dataset of 79 different relations (Jur-
gens et al., 2012).
All embedding algorithms suffer from inherent variability. For example because
of randomness in the initialization and the random negative sampling, algorithms like word2vec may produce different results even from the same dataset, and in- dividual documents in a collection may strongly impact the resulting embeddings (Tian et al. 2016, Hellrich and Hahn 2016, Antoniak and Mimno 2018). When em- beddings are used to study word associations in particular corpora, therefore, it is best practice to train multiple embeddings with bootstrap sampling over documents and average the results (Antoniak and Mimno, 2018).