Learn NLP the Stanford Way — Lesson 2
By: Thiago Candido
Tags: Programming, NLP, Python, Data Science, Machine Learning

In the previous post, we introduced NLP. To find out word meanings with the Python programming language, we used the NLTK package and worked our way into word embeddings using the gensim package and Word2vec.

Since we only touched the Word2Vec technique from a 10,000-feet overview, we are now going to dive deeper into the training method to create a Word2vec model.

Word2vec family

The Word2vec (Mikolov et al. 2013)[1][2] is not a singular technique or algorithm. It’s actually a family of neural network architectures and optimization techniques that can produce good results learning embeddings for large datasets.

The network architectures are shallow, composed of two layers, and are trained to produce vector representations of words given their context.

The two model variations that can be used are:

Continuous Bag of Words (CBOW), and

Skip-gram.

Training algorithms

Continuous Bag of Words

The CBOW model is based on trying to predict a central word from the context words around it.

We select a few words from a fixed-size window — the authors recommend this technique around a size of 5 — create a dictionary containing the words and their frequencies, and train the model by predicting the central word from the bag of words. The CBOW model doesn’t take into consideration the order of the words inside the “bag.”

Skip-gram

With the Skip-gram model, we predict outside words given a central context word. It works in the opposite way of the CBOW model. With this method, the authors recommend using a window of size 10.

On performance and accuracy: The CBOW model is faster than the Skip-gram, but the Skip-gram architecture works better with infrequent words.

Training Techniques

While the word embeddings created by the network can express the relationships between words, the network itself presents scalability issues. Depending on the vocabulary size, the number of operations needed to calculate the network's output layer is huge. Here are some techniques that are frequently used with Word2vec networks:

Hierarchical Softmax

The hierarchical softmax technique, proposed by Morin and Bengio[1], is applied due to the sheer size of regular vocabularies. In a regular neural network output layer, using the softmax function, the computing power needed to address the probability distribution of a full-sized vocabulary in any given language would be extremely large. We can formalize this by giving a size V vocabulary; we can denote the complexity using O(V).

With hierarchical softmax, the complexity is O(log2(V)) instead of O(V). That is achieved through the use of a multi-layer binary tree to calculate the probability of each word.

Simple exercise

Imagine that we are working with the English vocabulary, which in some libraries is represented by roughly 2 million word embedding — V. That implies a computational cost of O(V) => O(2,000,000). Using the hierarchical softmax, we would work with O(log2(V)) => O(log2(2,000,000) => O(~20).

If you are searching for a more technical, in-depth explanation, I recommend you this blog post.

Negative Sampling

The intuition behind negative sampling, presented by Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean[3] is only to update a subset of weights in the training process, affecting only the target — positive — word and a few of the non-related — negative — words, chosen using a “unigram distribution,” in which more frequent words are preferred to be selected as negative examples.

An illustration of the computations needed in a regular skip-gram model and one with negative sampling — Source

Using negative sampling, the computational cost is dramatically lower than the regular softmax version since it transforms a multi-classification task into a few binary classification tasks.

Implementing different Word2vec models using gensim

You can duplicate my Deepnote notebook here and follow me as I walk through this project for the best experience.

We will use gensim, a Python library, to create different Word2vec models from the same corpus, just passing different parameters to the Word2Vec class constructor.

First, we import the necessary packages and download the corpus:

Then we can create different Word2Vec models using the downloaded corpus and different parameters:

The following parameters are passed to the constructor to define the training algorithm and optimization technique used (source) :

sg ({0, 1}, optional) — Training algorithm: 1 for skip-gram; otherwise CBOW.

({0, 1}, optional) — Training algorithm: 1 for skip-gram; otherwise CBOW. hs ({0, 1}, optional) — If 1, hierarchical softmax will be used for model training. If 0 and negative is non-zero, negative sampling will be used.

({0, 1}, optional) — If 1, hierarchical softmax will be used for model training. If 0 and negative is non-zero, negative sampling will be used. Negative (int, optional) — If > 0, negative sampling will be used; the int for negative specifies how many “noise words” should be drawn (usually between 5–20). If set to 0, no negative sampling is used.

GloVe

GloVe: Global Vectors for Word Representation, presented by Jeffrey Pennington, Richard Socher, and Christopher D. Manning, is another model mainly based on word embeddings.

GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space. — Stanford GloVe

The main intuition is to scan through the whole corpus and compute the co-occurrence statistics for each word given a context. You can picture a matrix, with rows being the words and columns being the different contexts. Then you would reduce the dimensionality of each row to represent a word vector by factoring the matrix.

GloVe combines two model families: the local context window method and the global matrix factorization method.

The main difference between the two is that while the Word2vec model uses local contexts and a shallow neural network, the GloVe model is based on local and global word co-occurrence and uses the matrix factorization method.

Using GloVe with gensim

To use GloVe with gensim is really easy. You can use the api package to download a trained GloVe model.

You can also convert a GloVe model to a Word2Vec model in gensim using the glove2word2vec script.

Word Senses

Now we know how to create and use word embeddings created by the Word2vec and GloVe models. But still — are those vectors enough to represent words accurately in different contexts?

Common or long-lived words can have several meanings. How can we create embeddings that can capture all meanings from a word?

Linear Algebraic Structure of Word Senses, with Applications to Polysemy

Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, Andrej Risteski [4] propose a solution by representing different senses of the same word using a linear superposition — meaning the creation of a word embedding based on a weighted average of each sense embedding and its frequency.

The different senses of the word tie — Taken from the second class slides.

Considering that the vector embedding space is high dimensional and sparse, we can reconstruct the different sense vectors from just the weighted average — or the linear superposition — of the senses.

Conclusion

Next, we will discuss word window classification, neural networks, and PyTorch, topics of the Stanford course’s second lecture. I hope you enjoyed reading this post. If you have any questions, feel free to leave a comment.

Thank you for your time.

Take care, and keep coding!

References

Software and Libraries