Word and Text Representations in Natural Language Processing | by Arun Jagota | Towards Data Science

Words are fundamental constructs in many natural languages. Characters in these languages are meaningless until arranged to form a word. When this happens, viola, meaning emerges! When words are arranged sequentially, such as in phrases or sentences, even more meaning emerges.
Not surprisingly, core NLP operations involve processing words or sequences of words appropriately. (In this post a sequence of words, which we will abbreviate to text, will be a phrase, a sentence, a paragraph, a section, a chapter, or an entire document. For notational simplicity, we will treat punctuation symbols also as words.)
Below is an inventory of key operations on text that power many use cases.
Text classification: Classify text into one of a predefined set of categories. Text similarity: Score how similar two pieces of text (sentences, paras, ..., entire documents) are.Text clustering: Cluster several texts — such as documents — into groups of similar ones.Keywords extraction: Extract salient keywords from a corpus of text (e.g. documents).Topics discovery: Discover sets of keywords that go together, i.e. topics, from a corpus of text.
All these involve operating on words. Some, such as text similarity, involves operating on word sequences.
Local Word Representations
By Id
In this, every distinct word is assigned a unique identifier. As an example, say our lexicon only has 10 words: dog, cat, tiger, elephant, the, black, white, is, big, fast. Below depicts one local encoding of this lexicon.
Word dog cat tiger elephant the black white is are andId    0   1    2      3      4    5     6    7  8   9
Now consider the text: the dog and cat are black. This would be represented as the sequence 4 0 9 1 8 5.
This representation retains all the information in the text. This is good. On the other hand texts of differing lengths produce sequences of differing lengths, making them hard to compare.
In a Vector Space
In this representation, each word in the lexicon forms its own dimension in a vector space. So in our example above, our space would have 10 dimensions. A word is represented by the binary vector in which the word’s dimension’s value is 1 and the rest are 0. We will call this vector the word’s indicator vector.
Below are two word indicator vectors from our running example.
dog   → 1000000000black → 0000010000
Okay, now comes the key point. Texts of arbitrary length are also mapped as vectors in this same space. This makes it easy to compare texts of differing lengths. Just compare their vectors using algebraic similarity measures such as cosine similarity or those based on Euclidean distance or others.
Text Vectors
So how is text, i.e. a sequence of words of arbitrary length, mapped to a vector in this space. There are various possibilities. The simplest one is to add up the indicator vectors of the words that appear in the text and binarize (i.e. truncate positive counts to 1). Below is the vector we would get for our example.
the dog and cat are black → 1100110010
This vector just captures which words from the lexicon are in the text and which not.
1    1    0      0      1    1     0    0  1   0dog cat tiger elephant the black white is are and
This representation, also known as bag of words, ignores all word order in the text. Sometimes this is actually beneficial. The vector representations of the cat and dog are black and the dog and cat are black are the same. This is good, as the order in which cat and dog appear here is immaterial.
Enhanced Versions
Term frequency
Consider long texts, such as long documents. The frequencies with which various words appear may also be important. For example, consider two 10,000-word documents, one in which hospital is mentioned 200 times and one in which hospital is mentioned only once. The occurrence of hospital in the first document is more significant than in the second one.
A text encoding that captures word frequency is simple to obtain. Just add up the indicator vectors of all the word occurrences in the text. Just don’t binarize.
TF X IDF
In long documents common words such as the, a, of, and will appear frequently. The frequency-based representation will over-represent their importance. We don’t want two documents to be deemed similar merely because they both have lots of occurrences of the in them. Yes, frequency is important, but only of informative words. In our examples, hospital is an informative word whereas the is not.
The TF X IDF representation first involves assigning an informativeness score to each word in the lexicon. This score is estimated from a diverse corpus of text such as sentences (or even paragraphs) drawn from different types of documents. The fewer texts a word appears in, the more informative it is. This is why this measure is called “inverse document frequency”.
Armed with the IDF scores of the various words, a text’s TF X IDF vector is easily obtained. Run through all the word occurrences in the text. Multiply the indicator vector of each occurrence by the IDF score of its word. Add them all up. This will result in a vector in which a word’s contribution will depend on its frequency in the document adjusted by the word’s informativeness (i.e. IDF) score. Words uncommon in a diverse corpus but frequent in a particular text will be deemed more important in the text.
Let’s see an example. Consider the text the dog and cat are black. We saw its term frequency representation earlier: 1100110010.
1    1    0      0      1    1     0    0  1   0dog cat tiger elephant the black white is are and
The TF X IDF representation might look more like this. (Don’t take the positive numbers too literally.)
1    1    0      0     0.1    1    0    0  0.1   0dog cat tiger elephant the  black white is are   and
The words the and are have a greatly reduced influence.
Local Representations Can’t Capture Word Similarity
Local representations are definitely very useful — indeed widely used. That said, they have a limitation that is a barrier to even more widespread use. They are incapable of accounting for word similarity.
Consider three sentences:
S1: He is a lawyer S2: He is a attorney S3: He is a doctor
Since lawyer and attorney are synonyms we can see that S1 and S2 are semantically identical whereas S3 is unrelated to either. The local representations are incapable of making this distinction because lawyer, attorney, and doctor will be three different dimensions.
We can alleviate this situation in one of two ways (or a combination).
We can augment the local representations with a dictionary of synonyms.We can calculate correlations among dimensions and take these into account in our scoring.
We can augment the local representations with a dictionary of synonyms.
We can calculate correlations among dimensions and take these into account in our scoring.
While dictionary-based approaches are often effective in practice, they cannot deduce more nuanced relationships than those represented explicitly in the dictionaries. As a simple example consider
S5: He teaches computer scienceS6: He teaches data scienceS7: He teaches art
We’d like to be able to deem S5 and S6 as semantically more similar than S6 and S7.
This example is just the tip of the iceberg. Among the millions of words out there, think how many word similarity relationships are buried! Now extend this to proper nouns. As an example, we’d like to capture that canon and cameras are positively related.
Correlation analysis of dimensions can alleviate these issues. From a large corpus of documents, such as Wikipedia, broken down further into smaller chunks such as paragraphs, we might be able to deduce that “computer science” and “data science” are positively correlated, whereas “computer science” and “art” are uncorrelated — perhaps even negatively correlated.
We won’t describe the myriads of ways correlation analysis can be applied to alleviate these issues. Instead we will focus on an approach that does an alternate encoding of a word, called a distributed representation, which does a form of correlation analysis under-the-hood.
Distributed Word Representations
A word’s local representation just has a single 1-valued bit — the one that identifies the word. Consequently, the indicator vectors of any two words differ in exactly two bits. (See example below.) That is, all pairs of indicator vectors are equidistant.
dog      1000000000Elephant 0001000000
What if it were possible, instead, to map words to vectors in such a way that similar words get mapped to nearby vectors. It is possible, albeit more involved. First, we need a key concept.
Context
The context of a word is the words that frequently occur near it, such as in the same sentence.
Why is this concept useful to us? Because two words having similar contextual representations will be positively related. Take “cat” and “whiskers”. They will have similar contextual representations as they will both co-occur in contexts involving cats. On the other hand “cat” and “Jupiter” will not appear in the same contexts. So by comparing contextual representations, we can infer that the first pair is related whereas the second pair is not.
Okay, let’s now formalize this notion. We model a word’s context as a probability distribution over all words in the lexicon. We will call this the word’s context vector. This is a vector in our original local representation space. Unlike an indicator vector, it is distributed. In fact, the indicator vector is a special case, one whose context is the word itself (no more, no less).
Example
Say our lexicon just has four words cat, whiskers, computer, and laptop. Their indicator vectors are 1000, 0100, 0010, and 0001 respectively. No pair overlaps. By contrast, plausible context vectors of these four words might be as below.
word      cat whiskers computer laptopcat       0.7   0.3       0       0whiskers  0.3   0.7       0       0computer  0     0         0.7     0.3laptop    0     0         0.3     0.7
We see that the context vectors of cat and whiskers overlap, as do those of computer and laptop. But not those of whiskers and computer. This makes sense.
Estimating Context Vectors
Now that we understand what a word’s context vector is, how do we estimate it? Having a large corpus of documents helps. Fortunately, such a corpus, for example, Wikipedia, is easily obtained these days.
We simply keep track of co-occurrences of other words with the word of interest in the same proximity, say in the same sentence. Say the word cat occurs in 100 sentences. In 5 of these sentences the word dog also appears. The word dog is in cat’s context with probability 5/100.
Improving on Context Vectors
There is one issue with this approach. Especially common words such as the will appear in most sentences, so will appear in the contexts of unrelated words. For example, cat’s context will include the because in most sentences in which cat appears the also appears.
There is a simple way to alleviate this situation. We divide the word’s contextual probability by its overall probability. Formally, this is as follows. Say context-score(u,v) denotes the score we give to v of being in u’s context. To this point we have defined context-score(u,v) as P(v|u). Instead, we can define it as P(v|u)/P(v). Here P(v|u) is the fraction of sentences containing u that also contain v and P(v) is the fraction of all sentences that contain v.
Here is an example. Let’s say that the occurs in 75% of all sentences and 80% of the sentences that cat occurs in. So the score of the being in cat’s context is 80/75. Consider whiskers. Let’s say it appears in only 1% of all sentences, but in 25% of those in which cat appears. So the score of whiskers being in cat’s context is 25/1.
In short, the score of whiskers being in cat’s context is much higher than that of the’s. As desired.
Context Vectors Will Be Sparse
The vector space might have millions of dimensions. The context vectors, however, will tend to be sparse or can be made sparse without losing much information. A word’s context will normally only contain a small proportion of the millions of words in the lexicon.
Text Vectors
This is just as before. Add the vectors of the word occurrences in the text, after adjusting for the IDF scores of the various words.
It's important to note that adjusting with the IDF scores is even more important here than in local representations. This is because a common word such as the will have its own (especially wide) context, whose undamped contribution will pollute the text vector.
Let’s see an example. Say from a suitable corpus we picked up the following context representations.
Word                contextbark                howl, dog, tree, wood, leash, shrubdog                 bark, howl, leash, ...
Note that bark’s context captures its two word-senses: tree-bark and dog-bark.
Now consider the text the bark of the dog. Assume that IDF will significantly reduce the contribution of the two occurrences of the and the one of of to the text vector. In fact to drop clutter, let’s just set these influences to zero. We are left with: bark dog. Adding up the context vectors of the two words will give us something like
howl, dog, tree, wood, leash, shrub
The words that occur in the contexts of both the words in the text are highlighted in bold. These words will contribute more to the text’s vector.
Now imagine that the text was the bark of the labrador. Assume that our corpus is rich enough to capture the context of labrador. This text’s vector will also be similar. This will let us infer that the bark of the dog and the bark of the labrador are quite similar.
Advanced Word Context Models
In the past decade, a different way of obtaining word context vectors, going by the name word embeddings, has emerged and gained widespread use. Instead of obtaining context vectors via simple statistical analyses, it trains a neural network to learn word embeddings. This is done in an architecture with a hidden layer. Consequently, learning forces the network to learn good latent features of words. Words are embedded in this feature space, not in the original vector space.
We will cover this approach in detail in a separate post, as its description, which involves neural network details, is fairly involved.