How to Solve Analogies with Word2Vec | by Khuyen Tran | Towards Data Science

To make words understood by machine learning algorithms, word embedding is used to map words into vectors of real numbers. There are various word embedding models and word2vec is one of them.
In simple words, word2vec is a group of related models that are used to produce word embeddings. These models are trained to construct the linguistic contexts of words. Word2vec takes a large corpus of text and produces a vector space, with each unique word in the corpus being assigned to a corresponding vector in the space.
But how the word vectors are positioned in the vector space? They are positioned such that words that share common contexts in the corpus are located close to one another in the space.
To understand how word2vec works, let’s explore some methods that we can use with word2vec such as finding similarities between words or solving analogies like this
We will use the pre-trained word vectors from Glove. We are especially interested in Twitter dataset with 2B tweets, 27B tokens, and 200d vectors. The data could be downloaded here. We use Gensim to convert Glove vectors into the word2vec, then use KeyedVectors to load vectors in word2vec format.
Now we use model.most_similar() to find the top-N most similar words. Positive words contribute positively towards the similarity. Negative words contribute negatively towards the similarity, or in another word, contribute to the difference. Let’s test out this model!
Find the words that are most similar to ‘love’
model.most_similar(positive=['love'])
How about ‘girl’?
model.most_similar(positive=['girl'])
.We might as well look up the words that are similar to ‘boy’
model.most_similar(positive=['boy'])
Hmm. So is there a difference in similar words between ‘boy’ and ‘man’
model.most_similar(positive=['man'])
It seems like there are clearly some difference as a ‘boy’ becomes a ‘man’
The word embedding approach is also able to capture multiple different degrees of similarity between words. Semantic patterns can be reproduced using vector arithmetics. Thus, we could also use word2vec to solve analogies. If US has hamburger, what food does Canada have?
The results seem to show a variety of foods that are popular in Canada. Readers who are from Canada can confirm if these top words seem reasonable. Let’s explore a couple more countries with a function called analogy(). This time, we just return the one topmost similar word for each analogy.
Since visualizing is nice, let’s create a map between the countries and their corresponding foods. We use PCA to reduce the dimension of the data to 2.
From this map, the distance between countries indicates which countries are closer to one another in a 2 dimension space. The arrows indicate the relationship between countries and foods.
Congratulations! You have learned what word2vec is and how to find similarities between words with word2Vec. Seeing the power of word2vec, you may wonder: “How this method can work so well?”
We know that word2vec object function causes words that occur in similar contexts to have similar embeddings. But interestingly, what makes word embedding learning in the word2vec framework work so well is poorly understood. Whatever reason makes this embedding successful, I hope you understand its functionality and take advantage of this powerful tool for your text analysis. I encourage you to play with this dataset or the dataset of your choice to create other analogies with word2vec.
Feel free to fork and play with the code for this article in this Github repo.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these:
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Word2vec. Wikipedia. https://en.wikipedia.org/wiki/Word2vec