Word2Vec Explained. Explaining the Intuition of Word2Vec &... | by Vatsal | Towards Data Science

Table of Contents
Introduction
What is a Word Embedding?
Word2Vec Architecture- CBOW (Continuous Bag of Words) Model- Continuous Skip-Gram Model
Implementation- Data- Requirements- Import Data- Preprocess Data- Embed- PCA on Embeddings
Concluding Remarks
Resources
Word2Vec is a recent breakthrough in the world of NLP. Tomas Mikolov a Czech computer scientist and currently a researcher at CIIRC ( Czech Institute of Informatics, Robotics and Cybernetics) was one of the leading contributors towards the research and implementation of word2vec. Word embeddings are an integral part of solving many problems in NLP. They depict how humans understand language to a machine. You can imagine them as a vectorized representation of text. Word2Vec, a common method of generating word embeddings, has a variety of applications such as text similarity, recommendation systems, sentiment analysis, etc.
Before we get into word2vec, let’s establish an understanding of what word embeddings are. This is important to know because the overall result and output of word2vec will be embeddings associated to each unique word passed through the algorithm.
Word embeddings is a technique where individual words are transformed into a numerical representation of the word (a vector). Where each word is mapped to one vector, this vector is then learned in a way which resembles a neural network. The vectors try to capture various characteristics of that word with regard to the overall text. These characteristics can include the semantic relationship of the word, definitions, context, etc. With these numerical representations, you can do many things like identify similarity or dissimilarity between words.
Clearly, these are integral as inputs to various aspects of machine learning. A machine cannot process text in their raw form, thus converting the text into an embedding will allow users to feed the embedding to classic machine learning models. The simplest embedding would be a one hot encoding of text data where each vector would be mapped to a category.
For example: have = [1, 0, 0, 0, 0, 0, ... 0]a    = [0, 1, 0, 0, 0, 0, ... 0]good = [0, 0, 1, 0, 0, 0, ... 0]day  = [0, 0, 0, 1, 0, 0, ... 0] ...
However, there are multiple limitations of simple embeddings such as this, as they do not capture characteristics of the word, and they can be quite large depending on the size of the corpus.
The effectiveness of Word2Vec comes from its ability to group together vectors of similar words. Given a large enough dataset, Word2Vec can make strong estimates about a words meaning based on their occurrences in the text. These estimates yield word associations with other words in the corpus. For example, words like “King” and “Queen” would be very similar with one another. When conducting algebraic operations on word embeddings you can find a close approximation of word similarities. For example, the 2 dimensional embedding vector of "king" - the 2 dimensional embedding vector of "man" + the 2 dimensional embedding vector of "woman" yielded a vector which is very close to the embedding vector of "queen". Note, that the values below were chosen arbitrarily.
King    -    Man    +    Woman    =    Queen[5,3]   -    [2,1]  +    [3, 2]   =    [6,4]  
There are two main architectures which yield the success of word2vec. The skip-gram and CBOW architectures.
This architecture is very similar to a feed forward neural network. This model architecture essentially tries to predict a target word from a list of context words. The intuition behind this model is quite simple, given a phrase "Have a great day" , we will chose our target word to be “a” and our context words to be [“have”, “great”, “day”]. What this model will do is take the distributed representations of the context words to try and predict the target word.
The skip-gram model is a simple neural network with one hidden layer trained in order to predict the probability of a given word being present when an input word is present. Intuitively, you can imagine the skip-gram model being the opposite of the CBOW model. In this architecture, it takes the current word as an input and tries to accurately predict the words before and after this current word. This model essentially tries to learn and predict the context words around the specified input word. Based on experiments assessing the accuracy of this model it was found that the prediction quality improves given a large range of word vectors, however it also increases the computational complexity. The process can be described visually as seen below.
As seen above, given some corpus of text, a target word is selected over some rolling window. The training data consists of pair wise combinations of that target word and all other words in the window. This is the resulting training data for the neural network. Once the model is trained, we can essentially yield a probability of a word being a context word for a given target. The following image below represents the architecture of the neural network for the skip-gram model.
A corpus can be represented as a vector of size N, where each element in N corresponds to a word in the corpus. During the training process, we have a pair of target and context words, the input array will have 0 in all element except for the target word. The target word will be equal to 1. The hidden layer will learn the embedding representation of each word, yielding a d-dimensional embedding space. The output layer is a dense layer with a softmax activation function. The output layer will essentially yield a vector of the same size as the input, each element in the vector will consist of a probability. This probability indicates the similarity between the target word and the associated word in the corpus.
For a more detailed overview of both these models, I highly recommend reading the original paper which outlined these results here.
I’ll be showing how to use word2vec to generate word embeddings and use those embeddings for finding similar words and visualization of embeddings through PCA.
For the purposes of this tutorial we’ll be working with the Shakespeare dataset. You can find the file I used for this tutorial here, it includes all the lines Shakespeare has written for his plays.
nltk==3.6.1node2vec==0.4.3pandas==1.2.4matplotlib==3.3.4gensim==4.0.1scikit-learn=0.24.1
Note: Since we’re working with NLTK you might need to download the following corpus for the rest of the tutorial to work. This can easily be done by the following commands :
import nltknltk.download('stopwords')nltk.download('punkt')
Note: Change the PATH variable to the path of the data you’re working with.
Stopword Filtering Note
Be aware that the stopwords removed from these lines are of modern vocabulary. The application & data has a high importance to the type of preprocessing tactics necessary for cleaning of words.
In our scenario, words like “you” or “yourself” would be present in the stopwords and eliminated from the lines, however since this is Shakespeare text data, these types of words would not be used. Instead “thou” or “thyself” might be useful to remove. Stay keen to these types of miniature changes because they make a drastic difference in the performance of a good model versus a poor one.
For the purposes of this example, I won’t be going into extreme details in identifying stopwords from a different century, but be aware that you should.
Tensorflow has made a very beautiful, intuitive and user friendly representation of the word2vec model. I highly recommend you to explore it as it allows you to interact with the results of word2vec. The link is below.
projector.tensorflow.org
Word embeddings are an essential part of solving many problems in NLP, it depicts how humans understand language to a machine. Given a large corpus of text, word2vec produces an embedding vector associated to each word in the corpus. These embeddings are structured such that words with similar characteristics are in close proximity of one another. CBOW (continuous bag of words) and the skip-gram model are the two main architectures associated to word2vec. Given an input word, skip-gram will try to predict the words in context to the input whereas the CBOW model will take a variety of words and try to predict the missing one.
I’ve also written about node2vec which uses word2vec to generate node embeddings given a network. You can read about it here.
towardsdatascience.com
https://arxiv.org/pdf/1301.3781.pdf
https://www.kdnuggets.com/2019/02/word-embeddings-nlp-applications.html
https://wiki.pathmind.com/word2vec
https://projector.tensorflow.org/
If you enjoyed reading this article, please consider following me for upcoming articles explaining other data science materials and those materials (like word2vec) to solve relevant problems in different areas of data science. Here are some other articles I’ve written which I think you might enjoy.