Word2vec Made Easy. This post is a simplified yet in-depth... | by Munesh Lakhey | Towards Data Science

This post is a simplified yet in-depth guide to word2vec. In this article, we will implement word2vec model from scratch and see how embedding help to find similar/dissimilar words.
Word2Vec is the foundation of NLP( Natural Language Processing). Tomas Mikolov and the team of researchers developed the technique in 2013 at Google. Their approach first published in the paper ‘Efficient Estimation of Word Representations in Vector Space’. They refined their models to improve the quality of representation and speed of computation by using techniques like sub-sampling of frequent words and adopting negative sampling. This work was published in the paper ‘Distributed Representations of Words and Phrases and their Compositionality’.
Word2Vec is an efficient and effective way of representing words as vectors. The whole body of the text is encapsulated in some space of much lower dimension. In this space, all vectors have certain orientation and it is possible to explicitly define their relationship with each other. The distributed representation of words as embedding vectors opens up many possibilities to find the words in ways that suits many applications in NLP.
We will keep the corpus simple as it helps us to understand each step with ease plus we can visualize the relationship clearly to make it more concrete. CBOW (Continuous Bag Of Words) and Skip-Gram are two most popular frames for word embedding. In CBOW the words occurring in context (surrounding words) of a selected word are used as inputs and middle or selected word as the target. Its the other way round in Skip-Gram, here the middle word tries to predict the words coming before and after it.
Consider a text consisting of characters from ‘a’ to ‘z’ sequentially [a, b,c, d, e, f, g, h, i, j, k,l, m, n, o, p, q, r,s, t, u, v, w, x, y, z], further let integers 0 to 25 represent the respective letters. Implementing CBOW keeping the window size of 2, we can see that ‘a’ is related to ‘b’ and ‘c’, ‘b’ to ‘a’, ‘c’ and’ ‘d’ and so forth. It follows that one hot vector representing input words will be of dimension [26, 1]. With the help of the model we will find the dense and distributed vector of size [10, 1]. The size of the embedding vector is selected arbitrarily.
The diagram below illustrates the context and target words.
It is easy to visualize the pattern of relationship. When they occur together within a span of 2 characters they are related or co-occurrence is true. The pattern is depicted in table and map below.
Word2Vec is a probabilistic model. Key components of this model are 2 weight matrices. The rows of the first matrix (w1) and the columns of the second matrix (w2) embed the input words and target words respectively. The product of these two word vectors is then used to get the probabilities for being a target word, given the selected input word. Upon training these embedding vectors are optimized using gradient descent such that probabilities for true targets are maximized. It is obvious that the matrices w1 and w2 are factorization a probability matrix which closely resembles the co-occurrence matrix.
Model is illustrated and explained in the figures below.
As evident from above the network is light and um ambiguous. Here we just need to create one-hot vectors for each letter and pair with all its targets. For example letter ‘a’ is [1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.] (reshaped to size 26, 1) as one-hot vector and it’s targets indices are 1 and 2 (‘b’ and ‘c’).
The network has 2 fully connected layers which compute hidden and output layers. No bias is used as they are redundant for similarity measures. Softmax is the only activation and it converts the output scores to probabilities. Here the loss criterion (nn.CrossEntropyLoss) implements the Softmax and computes the loss. Adam optimizer is selected while optimizing the weights.
#Dependenciesimport torchimport torch.optim as optimimport matplotlib.pyplot as plt#Generating desired pair of inputs and targets#Empty list that collects input and target in pairsinp_target_list = []             for i in range(26):    temp = []    a, b, c, d = i- 2, i - 1, i + 1, i + 2  #targets for input i    temp.extend([a, b, c, d])#keep targets within range of 0 to 25    for j in range(4):        if temp[j] >=0 and temp[j] <=25:            inp_target_list.append([i, temp[j]])print(inp_target_list[:5])#[[0, 1], [0, 2], [1, 0], [1, 2], [1, 3]]#Get one hot vectors for all inputs#Initiate tensor with 0’s that holds all inputs in inp_target pairsinp_tensor= torch.zeros(len(inp_target_list), 26)#Substitute 1 for 0, at position indicated by respective input   for i in range(len(inp_tensor)):    inp_tensor[i, np.array(inp_target_list)[i, 0]] =1#One_hot for 0 or letter 'a'print(inp_tensor[0]) #tensor([1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,0., 0., 0., 0., 0., 0., 0., 0.])#Create Network #2 fully connected layers with NO bias#Embedding dimension is 10#Softmax is implemented using loss criterion (nn.CrossEntropyLoss())fc1 = nn.Linear(26, 10, bias = False)fc2 = nn.Linear(10, 26, bias = False)params = list(fc1.parameters()) + list(fc2.parameters())LR = 0.001  #Learning ratecriterion = nn.CrossEntropyLoss()optimizer = optim.Adam(params, lr = LR)#Train#Define inputs and target tensorsinp_tensor = inp_tensortarget_tensor = torch.tensor(inp_target_list)[:, 1]losses = []for i in range(10000):    out_1 = fc1(torch.Tensor(inp_tensor)) #hidden layer    out_2 = fc2(out_1)                    #Score matrix    optimizer.zero_grad()                 #Flushing gradients         loss = criterion(out_2,               target_tensor.long().view(out_2.shape[0],))#Apply Softmax, get loss         loss.backward()                        #Getting grads    optimizer.step()                       #Correcting parameters    if i % 1000 == 0:        losses.append(loss.item())        print(loss.data)#Loss printedtensor(3.2619) tensor(3.0980) tensor(2.7768) tensor(2.4193) tensor(2.1216) tensor(1.8937) tensor(1.7242)tensor(1.6007) tensor(1.5130) tensor(1.4526) tensor(1.4121) .........
Here we will review a series of matrices. It is much easier to interpret pictures/patterns which give us a clear idea of about the numbers spread throughout the matrix.
The content of each heat map below is discussed and interpreted in the respective captions. Each row in Score matrix is the product of the embedding vector for word as input and word as the target. The products are seen as probabilities in the probability matrix. Neighboring characters within a window of 2 characters have higher values and dense probabilities.
The distance matrix is the cosine distances from each embedding vector for input word to all the vectors embedding vectors for words as input including itself. Hot diagonal values are the product with itself and have distances of 1. Values close to diagonal are warm indicating they are close neighbors to each diagonal element. This pattern is much enhanced when the matrix is plotted as 1, positive and negative distances from each other as in smoothed distances.
#Note: As there is no transposition of weight matrices in forward #propagation columns represent the embedding vectors#Score Matrixplt.matshow((fc1.weight.t() @ fc2.weight.t()).detach().numpy())#Probability Matrixplt.matshow(nn.Softmax(dim = 1)(fc1.weight.t() @ fc2.weight.t()).detach().numpy())#Distance Matrixdist_matrix = torch.zeros(26, 26)for i in range(26):    for j in range(26):        dist = torch.nn.functional.cosine_similarity(fc1.weight.t()[i, :], fc1.weight.t()[j, :], dim = 0, eps = 1e-8)        dist_matrix[i, j] = dist#Smoothed distances if dist_matrix[i, j] ==1 --> 1if dist_matrix[i, j] < 0 --> -0.5if dist_matrix[i, j] 0 to 0.99 --> 0.5
Embedding vectors
First, let’s see the embedding for ‘a’ in w1 and ‘b’ in w2 representing input/surrounding and target/middle words respectively. The vector product between ‘a’ — as input and ‘b’ — as a target is 18.217, a relatively large positive number whereas for all other characters it is small or negative (except for ‘c’). Which implies ‘b’ (and ‘c’) will have a much higher probability. On the other hand, taking both vectors from w1, the cosine similarity between vector ‘a’ -as input and ‘b’ — as input is 0.4496 or about 63 degrees implying some similarity.
#Embedding for ‘a’ and ‘b’vec_a = fc1.weight.t()[0].data        #select 'a' from w1vec_b = fc1.weight.t()[1].data        #select 'b' from w1vec_b_as_target = fc2.weight[1].data  #select 'b' from w2#vec_a [1.7211128,   -1.8706707,   0.15043418,   -1.7761097,   0.25396731,   0.17291844,   0.11687599,   -2.0173464,   1.4879528,   -1.6174731],#vec_b[-1.8015, -0.6789, -1.3880, -1.6618,  1.5363, -1.8415,  1.9647, -1.5331,          2.0684, -1.7526]#vec_b_as_target[1.9113195,   -1.9370987,   1.349203,   -1.5565106,   1.1705781,   -0.78464353,   -1.7225869,   -1.7434344,   1.9383335,   -1.2209699]#Cosine similarity between two characterstorch.nn.functional.cosine_similarity(vec_a_as_input, vec_b_as_input, dim = 0, eps = 1e-8)  #0.4496 or 63 degrees#vector product between input and target vectors --> raw scorevec_a@vec_b_as_target         #18.217
Cosine Similarity
Among different distance metrics, cosine similarity is more intuitive and most used in word2vec. It is normalized dot product of 2 vectors and this ratio defines the angle between them. Two vectors with the same orientation have a cosine similarity of 1, two vectors at 90° have a similarity of 0, and two vectors diametrically opposed have a similarity of -1, independent of their magnitude.
Vector combination and finding it’s neighbors
Once the words are represented by vectors, the task of finding similar or dissimilar words becomes easier. Any combination vectors result in a new vector and the cosine distances or other similarity measures can be used as before. These operations underlie the famous equation ‘king -man + woman = queen’.
t-SNE for dimensionality reduction
t-SNE ( t-distributed Stochastic Neighbor Embedding) is helpful to further reduce the dimensions of the vectors down to 2 or 3 dimensions making it possible to visualize the vectors and their spatial arrangement or distribution. The code below is an example of a t-SNE transformation of the embedding vectors from 10 dimensions to 2.
from sklearn.manifold import TSNEtsne_2d = TSNE(n_components=2).fit_transform(fc1.weight.t().detach().numpy())#First five 2d vectorsprint(tsne_2d[:5])array([[-1.11534159e-04,  1.24468185e-04],        [-8.66758155e-05, -1.63095172e-04],        [-5.46419301e-05,  6.82841112e-05],        [ 1.30452306e-04,  1.16242307e-04],        [-2.39734325e-05,  1.93809960e-04]])
The key to the implementation of word2vec is the construction of 2 complimentary weight matrices to represent words as input and as context or targets. Embedding dimension could be any arbitrary dimension depending upon vocabulary size. The output or prediction of the model is different from the similarity or distances. While the model learns to map from the input to output the embedding capture their relative position with each other. Softmax works well for smaller corpus but it becomes inefficient as the size of vocabulary grows due to the fact that it involves summing scores across the whole length of vocabulary. Skip gram with negative sampling is the preferred algorithm.Finally, Gensim is a popular and freely available library which we can employ readily for word embedding and text analysis. But knowing the underlying principles makes it much clear and easier to use these tools.
Notations and Equations:
x: One hot vector input (size: vocab, 1)
w1: Weight matrix 1 (size: vocab, embedding dim)
h: hidden layer (size: embedding dim, 1)
Vc: Vector embedding for word input(In CBOW input word is context word)
w2: Wight matrix 2 (size: embedding dim, vocab)
Vw_i: Vector embedding for i’th target word (In CBOW the middle word we are trying to predict given the selected surrounding word). Size(embedding dim, 1)
S and S_i : S_i is vector product of h (or Vc or selected row of w1) and Vw_i (or i’th column of w2). S represents scores resulting from Vc with each Vw. (size: S_i is scalar quantity and S has as many scores as vocab)
Y_i: Softmax converts the score S_i to probability Y_i (or probability of i’th target word). Doing so Softmax needs to sum across all scores(S)
Vw_prime: Sum across all target words
P(w|c): Probability of target word given context or input word
L(theta): Likelihood of parameters (theta represents the w1 and w2 which in turn collection of Vc and Vw for all the words in corpus)
l(theta): Log likelihood of theta
rho: Learning rate
Note: Equations below use one character or one word vector at a time (except during Softmax normalization) but a batch or whole of vocabulary processed while implementing in algorithm.
Forward propagation
Likelihood and back-propagation
References:
[1] Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean, Efficient Estimation of Word Representations in Vector Space (2013), arXiv:1301.3781 [cs.CL]
[2] Mikolov, Tomas, Ilya Sutskever, Kai Chen, Gregory S. Corrado and Jeffrey Dean. “Distributed Representations of Words and Phrases and their Compositionality.” NIPS(2013).
Also check out: Ali Ghodsi, Lec 13: Word2Vec Skip-Gram
Do connect via:
mail: mnshonco@gmail.com or