Mapping Word Embeddings with Word2vec | by Sam Liebman | Towards Data Science

Natural Language Processing (NLP) is an area of artificial intelligence focused on allowing computers to understand, process, and analyze human language. NLP is widely used in the tech industry, serving as a backbone to search engines, spam filters, language translation and much more. NLP enables computers to transform human language into a form that it can read and understand, such as a vector or discrete symbol. For example, NLP can take in the sentence So hungry, need food and break it down into four arbitrary symbols: so represented as K45, hungry as J83, need as Q67, and food as P21, all of which can then be processed by the computer. Each unique word is represented by a different symbol; however, the downside is that there is no apparent relationship between the symbols designated to hungry and food. This hinders the NLP model from using what it learned about hungry and applying it to food, which are semantically related. Vector Space Models (VSM) help address this issue by embedding the words in a vector space where similarly defined words are mapped near each other. This space is called a Word Embedding.
Word2vec, a brainchild of a team of researchers led by Google’s Tomas Mikolov, is one of the most popular models used to create word embeddings. Word2vec has two primary methods of contextualizing words: the Continuous Bag-of-Words model (CBOW) and the Skip-Gram model, which i will summarize in this post. Both models arrive at a similar conclusion, but take nearly inverse paths to get there.
CBOW, which is the less popular of the two models, uses source words to predict the target words. For example, take the sentence I want to learn python. In this instance, the target word is python, while the source words are I want to learn. CBOW is primarily used in smaller datasets, since it treats the context of the sentence as a single observation towards predicting the target word. In practice, this becomes very inefficient when working with a large set of words.
The Skip-Gram model works in the opposite fashion of the CBOW model, using target words to predict the source, or context, of the surrounding words. Consider the sentence the quick brown fox jumped over the lazy dog and suppose we use a simple definition for the context of a given word as the words immediately preceding and following it. The Skip-Gram model will break the sentence into (context, target) pairs, resulting in a set of pairs in the following format:
([the, brown],quick), ([quick,fox],brown), ([brown,jumped],fox)...
These pairs are winnowed further into (input, output) pairs, representing each word (input) with the word either directly before or after it. This is necessary because the Skip-Gram model works by using the target words (inputs) to predict the context, or output. These pairs are represented as follows:
(quick, the), (quick, brown), (brown, quick), (brown, fox)...
Now that each word is able to be represented in context, the fun begins. I won’t go into the math — this TensorFlow tutorial provides an in depth explanation — but the loss function for predicting each word given context can be optimized using stochastic gradient descent and iterating through every pair in the dataset. From there, the vectors can be reduced to two dimensions using the t-SNE dimensionality reduction technique.
Once the word vectors are reduced to two-dimensions, it is possible to see relationships between certain words. Examples of a semantic relationship are male/female designations and Country/Capital relationships, while an example of syntactic relationship is past vs. present tense. The diagram below does an excellent job of visualizing these relationships:
Words that share semantic or syntactic relationships will be represented by vectors of similar magnitude and be mapped in close proximity to each other in the word embredding. No longer is the case of king being represented with a arbitrary discrete symbol K93 and queen with S83. Instead, the relationship between king and queen is more apparent — in fact, it is exactly the same as the relationship between the vectors for man and woman, respectively. This allows for extremely cool and magically simple arithmetic to be performed on vectors. For example: if you subtract the vector for boy from the vector representation of brother, and then add the vector for girl, you will get sister.
brother - boy + girl = sisterqueen - woman + man = kingbiking - today + yesterday = biked
This opens up an entirely new dimension of possibilities for finding patterns or other insights in the data.
Using a sample code showcased in TensorFlow’s tutorial, I will demonstrate how word2vec works in practice. TensorFlow is a machine learning library developed by the Google Brain team for internal use, open-sourced to the public in 2015 in an effort to accelerate the growth of AI. TensorFlow is a powerful tool for deep learning, logistic regression, and reinforcement learning, and has grown popular due to its ability to optimize computational efficiency when training large datasets.
The example code provided reads a large dataset of 50,000 words, and trains a skip-gram model to vectorize the words in a contextual manner. It then iterates through the data 100,000 times in order to optimize the loss function for a randomized batch of popular words in the dataset. At first, the nearest neighbors for the popular words don’t show any syntactic or semantic relationship with each of the words. After 100,000 steps, a much clearer relationship is visible, and the loss function decreased by over 98%.
As seen above, before the skip-gram model was trained, the eight nearest neighbors for the word three were: bissau, zeal, chong, salting, cooperate, quarterfinals, legislatures, ample. By the time the iterations are complete, the nearest neighbors for three are: five, four, seven, six, two, eight, nine, agouti. While not perfect — last I checked, agouti is a tropical American rodent, not a number — 7 of the 8 nearest neighbors displayed a clear semantic relationship to the word three.
While word2vec can create word embeddings that illustrate semantic and syntactic relationships between words, the model is not without some flaws. A 2016 study entitled Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (Bolukbasi, Chang, Zou, Saligrama, Kalai) demonstrated how word embeddings used by Google reinforced gender stereotypes at an alarming rate, and identified potential fixes to the issue. Data scientist and entrepreneur Emre Şarbak used Google Translate to further emphasize the gender biases exhibited by the word embedding algorithms. Şarbak, who is fluent in Turkish, tested how Google would translate sentences from Turkish, which uses gender neutral pronouns, to English. The results were a mix between fascinating and disturbing. (More examples)
For the most part, when a sentence contained descriptors stereotypically attributed to women (cook, teacher, nurse), the Turkish gender-neutral pronoun o was translated to she. Conversely, sentences with terms such as hard working, lawyer, and engineer saw the pronoun translated to male form. Google is not solely to blame for this — its algorithms are based off a corpus of human words containing billions of data points, so Google is merely reflecting already existing biases. Yet, Google is still determining what millions of people see when they use Translate (or Search, YouTube and any other popular Google platform).
Ultimately, this is most likely an unintended negative consequence of a powerful tool, but it raises an important issue about how easily we let computers and artificial intelligence dictate what we think and see. NLP and word embeddings are essential tools and are arguably the future of artificial intelligence; however, an open dialogue about exactly how machine learning algorithms are making their decisions is important so that marginalized voices don’t get shut out.
TensorFlow Word2vec Tutorial
Word2Vec (skip-gram model): PART 1 — Intuition by Manish Chablani
Word2Vec Tutorial — The Skip-Gram Model by Chris McCormick