Art of Vector Representation of Words | by ASHISH RANA | Towards Data Science

Expressing power of notations used to represent a vocabulary of a language has been a great deal of interest in the field of linguistics. Languages in practice have semantic ambiguity.
“John kissed his wife, and so did Sam”.
Sam kissed John’s wife or his own? These ambiguities must be handled in order to represent information in true form.
So, how do we develop a machine level understanding for language modelling task. Classical count based or similarity parameter based methods have existed for quite a long time in Computer Science.
But, now with the advent of deep learning models like RNNs this expressing power of language modelling is of great interest for creating efficient system to store information and finding relations between vocabulary terms. This will act as fundamental block in encoder-decoder models like seq-to-seq model.
In this article we will explore different representation system of words and their power of expression, from the classical NLP to modern Neural Network approaches. In the end we will conclude with hands on coding exercises and explanations.
For readers, keep your markers on & mark important points. It is quite a long read with some minor mathematics involved with concept explanation from scratch.
Machines are better at understanding numbers that actual text passed on as tokens. This process of converting text to numbers is called vectorization. Vectors then combine to form vector space which is continuous in nature, an algebraic model where rules of vector addition and similarity measures apply. Different approaches of vectorization exists let’s move from the most primitive ones to most advanced ones.
The representation models that will be discussed in this article are :
1. One-hot representations
2. Distributed Representations
3. Singular Value Decomposition
4. Continuous bag of words model
5. Skip-Gram model
6. Glove Representations
For defining representation power of a system we first will look into workings of different representation systems. These systems represent each and every word of a vocabulary in the form a vector and create a finite vector space.
Let’s see an example of one-hot representation of words. Each word is represented with a large vector of size |V| i.e. vocabulary’s size for the given corpus.
It is a very simple form of representation with very easy implementation. But, many of the faults would have become clear even from such small example. Like, huge memory required for storing and processing such vectors. Along, with sparse nature of these vectors.For example, the size of |V| is very large like 3M for Google 1T corpus. This notation will fail in terms of computation overhead caused by representation power of this system.
Also, no notion of similarity is captured. Cosine similarity b/w unique words is zero and Euclidean distance is always sqrt(2). Meaning, no semantic information is getting expressed with this representation system.
Now, clearly we should move to a representation which saves space and hold some semantic power to express relationships among words. Let’s see a representation based on similarity.
You shall know a word by the company it keeps
— Firth, J.R.
The idea is to quantify co-occurrence of terms in a corpus. This co-occurrence is measured with window size of ‘k’ around the terms which signifies the context being distributed in that window size. With this method in mind, a co-occurrence matrix of terms × terms which captures the number of times a term appears in the context of another term is created. Also, it is a good practice to remove stopwords as these are high frequency words providing least amount of meaningful insight. Also, we can set an upper threshold for t for words.
Now, consider the following corpus:
Human machine interface for computer applications
User opinion of computer system response time
User interface management system
System engineering for improved response time
Here, in above case if stopwords are not handled properly they will create problem with their relative high frequency. So, a new quantity known as Positive Pointwise Mutual Information will be used that takes into account relative and individual occurrences along with size of vocabulary.
With this method we were able to get some idea about context but still this won’t be an ideal approach. Consider example ‘cat’ and ‘dog’ in corpus but they don’t lie within the window gap. Clearly there is relation b/w cat and dog like both being pets, mammal etc. this method won’t be able to give any insight about their relation.
In above case co-occurrence between {system, machine} and {human, user} is not visible. But, they are related to each other in above corpus’s context.
Also, high dimensionality, sparse nature of matrix & redundancy( as symmetric matrix ) still persists along with increase in size along with vocabulary.
High dimensional problem are solved by PCA or by its generalized version SVD. SVD is a generalization of the eigen-decomposition of a positive semi-definite normal matrix to any matrix via an extension of the polar decomposition. It gives the best rank-k approximation of the original data. Let original data X be of dimension m x n. The singular value decomposition will break this into best rank approximation capturing information from most relevant to least relevant ones.
An analogy to this can be seen with with case of colors being represented with 8-bits. These 8-bits will provide more resolution to us. But, now we want to compress this into 4-bits only. Which bits should we retain, lower or higher ones?
When the bits are reduced, most important information is to identify the color not the shades of different colors. Similar resolution will exist for different colors like Red, Blue, Yellow etc.
As by capturing information about different shades only there would be no meaning for colors because all information related to colors was lost. Hence, lower bits are the most important ones. Now, in this case SVD would have captured those lower bits.
With SVD, the latent co-occurrence between {system, machine} and {human, user} will become visible. We take the matrix product of X and Xt, whose ij-th entry is the dot product between the representation of word i (X[i :]) and word j (X[j :]). The ij-th entry of X, Xt roughly captures the cosine similarity between word i , word j.
We would want representations of words(i, j) to be of smaller dimensions but still have the same similarity(dot product) as the corresponding rows of Xhat. Hence, our search for finding more powerful representations must go on.
Also, notice that the dot product between the rows of the the matrix Wword=UΣ is the same as the dot product between the rows of X̂hat.
The methods we have seen are count based models like SVD as it uses co-occurrence count which uses the classical statistic based NLP principles. Now, we will move onto prediction based model which directly learn word representations. Consider a task, predict the nth word given previous (n-1) words. For training data all n-word window in training corpus can be used and corpus can be obtained from scraping any webpage.
Now, How to model this task of predicting nth word? & What’s the connection between this task and learning word representations? For modelling this problem we will use feed-forward neural network as shown below.
Our aim is to predict a probability distribution over these |V| classes as a multi-class classification problem. But this looks very complex and neural networks have very high number parameters. How is this a simpler approach?
For this, we need to look little bit behind the mathematics behind the vector multiplication happening in behind the scenes with this neural network. The product Wcontext*x given that x is a one hot vector.
We can treat the i-th column of Wcontext as the representation of context i. For P(on|sat) is proportional to the dot product between jth column of Wcontext and ith column of Wword. P(word = i|sat) thus depends on the ithcolumn of Wword. We thus treat the ith column of Wword as the representation of word i. This clearly shows the weight parameters are being represented as word vector representation in a neural network architecture.
Having understood the simplicity of interpretation behind the parameters, our aim now is to learn these parameters. For multi-class classification problem use softmax as output activation function and cross-entropy as loss function.
Let’s see what we can interpret from update rule from above loss-function. Put value of yhat in loss-function and differentiate it to derive gradient update rule.
This increases the cosine similarity between vw and uc. As, training objective ensures that the cosine similarity between word (vw) and context word (uc) is maximized. Hence, similarity measure is also captured by this representation. Also, neural network helps in learning much simpler and abstract vector representations of words.
In practice, more than one words are used in the window size, it common to use ‘d’ window size depending on the use-case. That would simply mean we have to stack copies of Wcontext in the bottom layers as two one-hot rod bits will be high.
Here, still computation bottleneck of softmax is present with denominator term involving summation over the entire vocabulary. We must explore some other model mitigating this bottleneck step.
This model predicts context words with respect to given input words. The role of context and word has changed to an almost opposite sense. Now, with given input words as one hot representations our aim is to predict context word related to it. This opposite relation b/w CBOW model & Skip-Gram model will become more clear below.
Given a corpus the model loops on the words of each sentence and either tries to use the current word of to predict its neighbors(its context), in which case the model is called “Skip-Gram”, or it uses each of these contexts to predict the current word, in which case the model is called “Continuous Bag Of Words” (CBOW).
Train a simple neural network with a single hidden layer to perform a certain task, but then we’re not actually going to use that neural network for the task we trained it on every single time for new task of modeling! Instead, the goal is actually just to learn the weights of the hidden layer which will be word vectors as stated by mathematics shown in above section.
In the simple case when there is only one context word, we will arrive atthe same update rule for uc as we did for vw earlier. If we have multiple context words the loss function would just be a summation of many cross-entropy errors.
Again same issue as with CBOW, the problem of softmax being computationally expensive exists for this model also.
Three strategies namely negative sampling, contrastive estimation and hierarchical softmax can be used to eliminate this softmax problem of summation over entire vocabulary.
Negative Sampling: We sample k negative (w, r) pairs with no context for every positive (w, c) context pairs. The size of Ddash is thus k times the size of D. In our neural network we will define loss functions and train on both these sets. These corrupted pairs are drawn from a specially designed distribution, which favours less frequent words to be drawn more often.
Contrastive Estimation: The basic idea is to convert a multinomial classification problem(as it is the problem of predicting the next word) to a binary classification problem. Instead of using softmax to estimate a true probability distribution of the output word, a binary logistic regression (binary classification) is used instead i.e. the optimized classifier simply predicts whether a pair of words is good or bad. For each training sample, the enhanced (optimized) classifier is fed a true pair (a center word and another word that appears in its context) and a number of k randomly corrupted pairs (consisting of the center word and a randomly chosen word from the vocabulary). By learning to distinguish the true pairs from corrupted ones, the classifier will ultimately learn the word vectors.
Hierarchical softmax: In this technique a binary tree is constructed such that there are |V| leaf nodes each corresponding to one word in the vocabulary. There exists a unique path from the root node to a leaf node. In effect, the task gets formulated to the probability of predicting a word is the same as predicting the correct unique path from the root node to that word. The above model ensures that the representation of a context word vc will have a high(low) similarity with the representation of the node ui if ui appears on the path and the path branches to the left(right) at ui. Representations of contexts which appear with the same words will have high similarity.
Count based methods (SVD) rely on global co-occurrence counts from thecorpus for computing word representations. Predict based methods learn word representations using co-occurrence information. Why not combine the two count and learn mechanisms ?
Let’s formulate this idea mathematically and then develop an intuition for it. Let Xij encodes important global information about the co-occurrence between i and j.
Our aim will be to learn word vectors which complies with computed probability on entire corpus. Essentially we are saying that we want wordvectors vi and vj such that vi^T vj is faithful to the globally computed P (j|i).
Add the two equations for Xi and Xj respectively. Also, log(Xi) and log(Xj) depend only on the words i & j and we can think of them as word specific biases which will be learned. Formulate this problem in following way.
Now, the problem is weights of all the co-occurrences are equal. Weight should be defined in such a manner that neither rare or frequent words are over-weighted.
Words are represented by dense vectors where a vector represents the projection of the word into a continuous vector space. It is an improvement over more the traditional bag-of-word model encoding schemes where large sparse vectors were used to represent each word. Those representations were sparse because the vocabularies were vast and a given word or document would be represented by a large vector comprised mostly of zero values.
Considering the recent popular research papers. Boroni et.al [2014] showed that predict models consistently outperform count models in all tasks. Levy et.al [2015] do a much more through analysis (IMO) and show that goodold SVD does better than prediction based models on similarity tasks but noton analogy tasks. Levy showed that word2vec also implicitly does a matrix factorization. It turns out that we can also show that
Good old SVD will just do fine. But, currently working with pre-trained Glove embeddings on huge corpuses results in much better results and once trained, can be reused again. Libraries like Keras, Gensim does provide embedding layers which can be used with ease for modelling tasks. Also, for extensive NLP tasks it is much better to use Standford’s Glove vector representation dataset. Let’s take a look on few examples of these representation models in action.
Let’s see how to encode characters as integers. Which means that each unique character will be assigned a unique integer value and each sequence of characters will be encoded as a sequence of these unique integers. Follow the below script mentioned for doing the above mentioned thing.
As a preprocessing step, first create a file containing all the sequences and store that in a new file. This new file will contain the sequences of characters of equal length. Refer python script below for this preprocessing step. You can also refer to repository link for directly copying complete project and running it onto your machine.
# load doc into memorydef load_docx(filename): # open the file as read only file = open(filename, 'r') # read all text text = file.read() # close the file file.close() return text# save tokens to file, one dialog per linedef save_docx(lines, filename): data = '\n'.join(lines) file = open(filename, 'w') file.write(data) file.close()# load text data in your current directoryraw_text = load_docx('The-Road-Not-Taken.txt')print(raw_text)# clean the texttokens = raw_text.split()raw_text = ' '.join(tokens)# organize into sequences of characterslength = 16sequences = list()for i in range(length, len(raw_text)): # select sequence of tokens seq = raw_text[i-length:i+1] # store sequences.append(seq)print('Total Sequences: %d' % len(sequences))# save sequences to fileout_filename = 'char_sequences.txt'save_docx(sequences, out_filename)
For encoding these sequences we will create the mapping given a sorted set of unique characters in the raw input data. The mapping will be a dictionary of character values to integer values.
char = sorted(list(set(raw_text)))map = dict((c, i) for i, c in enumerate(char))
Next, we process each sequence of characters one at a time and use the dictionary mapping to look up the integer value for each character. The result is a list of integer lists representing sequences of characters in integer format.
sequences = list()for line in lines: # integer encoding line encoded_seq = [mapping[char] for char in line] # store in list of sequences sequences.append(encoded_seq)
Next, step is encoding into one hot vector each of size of vocabulary. Separate input and output columns into different sequence of characters.
sequences = array(sequences)X, Y = sequences[:,:-1], sequences[:,-1] # X containing all chars except at last position, Y containing only the last char# to_categorical() function in the Keras API to one hot encode the input and output sequences.sequences = [to_categorical(x, num_classes=vocab_size) for x in X]X = array(sequences)Y = to_categorical(y, num_classes=vocab_size)
After this preprocessing you can feed your data into your model for learning parameters.
Word embeddings provide a dense representation of words and convey some semantic information along with it. Embeddings can also be learned along with our model specific to our dataset.
The position of a word in the learned vector space is referred to as its embedding.
The Embedding layer provided by keras which requires input to be integer encoded is initialized with random weights and will learn an embedding for all of the words in the training dataset.
e = Embedding(200, 32, input_length=50)200: specifies the vocabulary size32: specifies the output vector dimensionsinput_length=50: length of input sequences in the document
The output of the Embedding layer is a 2D vector with one embedding for each word in the input sequence of words.
Keras Embedding layer can also use a word embedding learned elsewhere like from pretrained Glove vectors from Stanford. Load the Glove vectors into the memory to an embedding array.
# load the whole embedding into memoryembeddings_index = dict()f = open('glove.6B.100d.txt')for line in f: values = line.split() word = values[0] coefs = asarray(values[1:], dtype='float32') embeddings_index[word] = coefsf.close()print('Loaded %s word vectors.' % len(embeddings_index))# for loading again it is better to store this as pickle object.
Then create resultant weight matrix which can be passed onto Keras Embedding layer. Finally, we do not want to update the learned word weights in this model, therefore we will set the trainable attribute for the model to be False.
# Here, we are using 100 dim vectors version from downloaded file# create a weight matrix for words in training docsembedding_matrix = zeros((vocab_size, 100))for word, i in t.word_index.items(): embedding_vector = embeddings_index.get(word) if embedding_vector is not None:  embedding_matrix[i] = embedding_vectore = Embedding(vocab_size, 100, weights=[embedding_matrix], input_length=4, trainable=False)
Above, code snippets do give an idea about on getting started with vectorization task depending on complexity of model that is being used. One hot vectors for least complex to Glove vectors for good performance for complex models.
Thanks to Prof. Mitesh M. Khapra and his TAs for wonderful course on Deep Learning(CS7015). Also, to Jason from machinelearningmastery.com for useful code snippets and few stackoverflow threads.