An Overview for Text Representations in NLP | by jiawei hu | Towards Data Science

Writing is always a good choice when it comes to clarifying one’s understandings of a given topic. By putting thoughts on papers, ideas will be clarified and confusions exposed. Though it might not be the most comfortable thing to do it’s indeed an efficient way to learn and improve.
If you ever find yourself having a hard time explaining something to a friend, something you’ve been studying for a while but somehow still didn’t manage to portray the subject clearly and intuitively, you should try writing it down.
In this article, I attempt to summarize some of the ideas for text representations in NLP, aiming to build a foundation for future complex concepts to come and hoping to contribute my granito de arena to your learning as well.
The above diagram summarizes the process of transforming a text corpus into different input formats for a Machine Learning model. Starting from the left, the Corpus goes through several steps before obtaining the Tokens, a set of text building blocks i.e. words, subwords, characters, etc. Since ML models are only capable of processing numerical values, the tokens in a sentence are replaced by their corresponding ids, by either looking them up in an associative array(vocabulary) or using a Hashing Trick. Once done, they are then transformed into different input formats shown on the right. Each one of these formats has its own pros and cons and should be chosen strategically according to the characteristics of the task at hand.
We are going to explore each of these input representations shown above, starting directly from the tokens and ignoring the previous steps.
One-Hot Encoding
Count Vectors/Feature Hashing/Tf-idf
Word Embeddings/Training Embeddings/Contextualized Embeddings
One-hot encoding may be one’s go-to move when a categorical feature is encountered in a data set(if you think such a feature is useful for the model, obviously). It’s a simple and straight forward technique and it works by replacing each category with a vector full of zeros, except for the position of its corresponding index value, which has a value of 1.
When applying one-hot encoding to a text document, tokens are replaced by their one-hot vectors and a given sentence is in turn transformed into a 2D-matrix with a shape of (n, m), with n being the number of token in the sentence and m the size of the vocabulary. Depending on the number of tokens a sentence has, its shape will be different.
As an example, say that we were to use one-hot encoding for the following sentences:s1. "This is sentence one."s2. "Now, here is our sentence number two."The vocabulary from the two sentences is:vocabulary = {"here": 0, "is": 1, "now": 2, "number": 3, "one": 4, "our": 5, "sentence": 6, "this": 7, "two": 8}The two sentences represented by one-hot vectors are:indices                             words      0  1  2  3  4  5  6  7  8s1: [[0, 0, 0, 0, 0, 0, 0, 1, 0], - "this"     [0, 1, 0, 0, 0, 0, 0, 0, 0], - "is"     [0, 0, 0, 0, 0, 0, 1, 0, 0], - "sentence"     [0, 0, 0, 0, 1, 0, 0, 0, 0]] - "one"s2: [[0, 0, 1, 0, 0, 0, 0, 0, 0], - "now"     [1, 0, 0, 0, 0, 0, 0, 0, 0], - "here"     [0, 1, 0, 0, 0, 0, 0, 0, 0], - "is"     [0, 0, 0, 0, 0, 1, 0, 0, 0], - "our"     [0, 0, 0, 0, 0, 0, 1, 0, 0], - "sentence"     [0, 0, 0, 1, 0, 0, 0, 0, 0], - "number"     [0, 0, 0, 0, 0, 0, 0, 0, 1]] - "two"
The size of the vocabulary would only grow as the training corpus gets larger and larger, as a result, each token would be represented by vectors with an increasingly larger length, making matrices more sparse. Instead of word-level representations, a more common approach is to use characters as tokens since it’ll limit the length of the vectors.
But either using the word or character-level representations, it is unavoidable that different sentence matrices will have different shapes(different number of rows). This can be an issue for the majority of ML models since their input shapes are meant to be the same. RNN based models on the other hand, if set right, don’t share this concern due to its ‘recurrent’ nature, but all the instances within the same batch are still expected to share a uniformed shape.
To go around this issue, one solution is to fix a length l for all instances — truncating longer ones while shorter ones are padded. For a task such as Sentiment Analysis for short texts(though using one-hot won’t yield good results), it’s typically enough to use say the first 300 characters. In Keras, the padding tokens can be masked so they won’t affect the loss.
The size of the training corpus can get as big as it wants, making the vocabulary richer and richer, but there’s always a possibility of encountering unknown words during inference time. A way to handle this is to reserve some space in the vocabulary during initialization, so when a out-of-vocabulary word pops up, it can be assigned to one of the reserved spots(an oov bucket).
Another way to featurize text is through n-gram count vectorizers, let’s take a look.
The usage of one-hot encoding allows us to achieve a token-level representation of a sentence, it does so by replacing its tokens with vectors while maintaining their original sequential arrangement. A count vectorizer, on the other hand, is based on term frequencies and is capable of squeezing an entire sentence into a single vector. Each position of a count vector is assigned to a particular token as before and its value represents the number of appearances that token has in the sentence.
Tokens are first generated from the corpus and vocabulary is built to map the tokens to their corresponding ids. Instead of building one vector for each token, a count vector simply counts how many times each token appeared in a sentence and places that number in their corresponding position in the vector.
If we were to represent the sentences from the previous example with count vectors, they would look like this:s1: [0, 1, 0, 0, 1, 0, 1, 1, 0]s2: [1, 1, 1, 1, 0, 1, 1, 0, 1]vocabulary = {"here": 0, "is": 1, "now": 2, "number": 3, "one": 4, "our": 5, "sentence": 6, "this": 7, "two": 8}
All sentences now are represented by vectors that share the same length l, which is defined by the number of unique tokens in the vocabulary by default but can also be selected manually. Such a way of representing sentences will fail to provide any token ordering information of the original sentence, and its associated context information is lost — only term frequencies are reflected in count vectors.
For illustration purposes, let's vectorize sentence s3:s3: "this this this is is one one one one"--> feature counts: "this" x 3, "is" x 2, "one" x 4s3: [0, 2, 0, 0, 4, 0, 0, 3, 0]
Instead of using single words/characters as tokens, which could be seen as uni-gram count vectorizers, we can also use two or more consecutive words/characters to form tokens, obtaining what’s known as 2-gram, 3-gram, or more generally n-gram vectorizers.
A use case of count vectors is, for example, Spam Email Detection with Naive Bayes Models.
The main criteria for email filtering are the term frequencies in each type of emails, i.e. how many times a word appeared in spam and non-spam emails.
The model learns through training about what words are more spam-like words, and what words are not. It assumes conditional independence for each email’s content words, meaning they are mutually independent and their appearances are only conditioned by the email’s label(spam, Y=1, or non-spam, Y=0).The probability of a given email being a Spam(Y=1), given its content words(X), is calculated using Bayes rule. According to which, such probability is determined by the probability of seeing these words in those Spam emails, P(X|Y=1), multiplied by the portion of Spam emails the model has seen during training, P(Y=1), and divided by P(X).
With X being the collection of words from an email, {x1, x2, ..., xm}, by assuming conditional independence, P(X|Y=1) is calculated as the product of the probabilities of seeing these words in Spam emails:
One downside of using count vectors here is that the unknown words are thrown away. If you have a sentence that has m words and n of them the model has never seen before, then only m-n words will be taken into account. This is used by sneaky spammers for filter circumvention —modifying the spam keywords making them unknown to the model while still readable to the users. The content the model will see only contains neutral words and the email can be classified as non-spam.
- Sentence from a Spam email:"come make some money, ten thousand dollar a week for doing nothing"- With modified words:"come mayke some m0ney, ten th0usand dollar$ a week for doing nothing"- The sentence the model will see:"come some, ten a week for doing nothing"
Feature Hashing, among its other advantages, can help alleviate the damage.
As models are trained with corpus with larger and larger sizes, it produces vocabularies that take more and more memory space to store. For efficiency purposes, these lookup tables are stored in RAM for quick token-id mapping and can slow down the operations once their size gets too big.
By using a Hashing Trick we can get rid of such memory-consuming vocabularies entirely and a hash function is used for token-id mapping instead. With a given string, a hash function is capable of returning a numerical value, a hash value, that’s unique to that string and use it as token id.
Since there’s no fix-sized vocabulary involved, all tokens can now be assigned to a number, no matter the model has seen it before or not. In the case of Spam Detection, the way of circumventing the filter by making spam words strange to the model is no longer as effective. Any given word, ‘known’ or ‘unknown’, can be fed to the hash function and output a numeric value within the predefined range. Instead of throwing the unknown word away, its corresponding position in the count vector will add 1 so all the words in the email are now taken into account, not only the neutral ones.
Unlike vocabularies, feature hashing is a one-way operation — we can’t find the initial feature using its hash value through the hash function. The same input would always produce the same output, but two distinct features may happen to be mapped to the same hash value(the odds are negligible if the vector size is large, i.e. 224).
As mentioned earlier, count vectors are handicapped when representing sentence context since they don’t reflect any initial token ordering. Things can get worse when terms such as “like”, “a”, or “and” that carry very little meaningful context information appear too frequently, taking the model’s ‘attention’ away from those less frequent yet more interesting terms by shallowing their frequencies.
Tf-idf stands for term-frequency(tf) times inverse document-frequency(idf) and it’s used to address this issue by re-weighting the count features according to how many different sentences each token has appeared in. It ‘penalizes’ the feature count of a term if it’s contained in more sentences, assuming an inverse relationship between a term’s relevance to the context and its number of appearances in different documents.
As an example, let’s take look at Scikit-Learn’s tf-idf implementation, TfidfTransformer, in default settings:
So far we’ve seen two types of representations: One-hot encoding, a token-level representation that allows the preservation of token ordering in the initial sentence, and Count Vectors, a more compact sentence-level representation that relies on term frequencies.
For NLP tasks such as Text Generation or Classification, one-hot representation or count vectors might be capable enough to represent the required information for the model to make wise decisions. However, their usage won’t be as effective for other tasks such as Sentiment Analysis, Neural Machine Translation, and Question Answering where a deeper understanding of the context is required to achieve great results.
Take One-hot encoding as an example, using it will not lead to a well-generalized model for these tasks because no comparison between any two given words can be done. All vectors are orthogonal to one another, the inner product of any two vectors is zero and their similarities cannot be measure by distance nor cosine-similarity.
Say that we were to train a language model for text generation and we've provided the model with the following sentence: "I want to have an orange", expecting it to output "juice".--> "I want to have an orange juice"If done right, the model should learn that "orange juice" is a common thing, or the entire sentence is a common sentence.If we were asked to complete the following sentence with one word: "I want to have an apple". Having seen the first sentence, our common sense will tell us to use "juice", provided that we have a notion on the similarities between "orange" and "apple".But the model we just trained won't be able to mimic this since it doesn't have any clue about how similar these two words are. In fact, it doesn't have any clue on how similar any two given words are, so it won't be able to generalized from "orange juice" to "apple juice".
For this, we turn to Word Embeddings, a featurized word-level representation capable of capturing the semantic meanings of words.
With embeddings, each word is represented by a dense vector of fixed size(generally range from 50 to 300), with values corresponding to a set of features i.e. Masculinity, Femininity, Age, etc. As shown in the figure below, these features are seen as different aspects of a word’s semantic meaning, and their values are obtained by random initialization and are updated during training, just like the parameters of the model do.
When training embeddings, we do not tell the model what these features should be, instead it’s up to the model to decide what are the best ones for the learning task. When setting up an embedding matrix(a set of word embeddings), we only define its shape — the number of words and each vector’s length. What each feature is representing is generally difficult to interpret.
Word Embedding’s ability to capture semantic meanings can be illustrated by projecting these high-dimensional vectors to a 2D space for visualization via t-SNE. If embeddings were obtained successfully, plotting these vectors with t-SNE would demonstrate how words that have similar meaning would end up being closer to one another.
The semantic relationship between different embedding vectors can also be illustrated by the following example.
Say that we have the embedding vectors of the word "king", "queen", "man", and "woman". The resulting vectors of subtracting "queen" from "king" and "woman" from "man" would be very similar direction-wise since they both carry similar values for that Gender feature."king" - "queen" ≃ "man" - "woman"or equivalently,"king" - "queen" + "woman" ≃ "man"
Earlier we mentioned that the embedding vectors can be trained just like another layer(in an NN), they can also be trained separately and used later for different tasks via transfer learning.
There are different ways to train embeddings but the principle remains more and less the same. Here we’ll briefly discuss the two Word2Vec methods, namely Continuous Bag of Words(CBOW) and Skip-gram.
Word2Vec are shallow, two-layer neural networks that are trained to reconstruct linguistic contexts of words. We extract pairs of context/target words from a large corpus for training, with the target word being a randomly chosen word and the context words being those located within a given window around the target word.
There are different varieties of context, they can be 1)all the words within the window, 2) n words before or 3) after the target word, or 4) simply a random word located within the window.Sentence: "Learning Spanish is hard but also fun" - target word: "is"- windows length is 2, for both directions- context: 1. "learning", "Spanish", "hard", "but" 2. "learning", "Spanish" 3. "hard", "but" 4. "learning"
For CBOW, the context words are fed into the model in their embedding form(randomly initialized) and the model is expected to output the target word using a softmax, P(target|context). Skip-gram, on the other hand, does the opposite, it takes in the target word and is expected to output the context words.
Now, these may sound like difficult learning tasks but keep in mind that the goal isn’t to do well on the tasks per se but to learn good embeddings, and these models can do just that.
Polysemy stands for words that written the same but according to the context they can have completely different meanings. Word Embeddings as explained above will fail to handle this.
The word "bank" in the following two sentences has different meanings, but since they are assigned to the same token id, their word embedding vectors are the same.1. "I went to the bank to withdraw some money."2. "I went to the other side of the river bank."
To address the problem, each work embedding must take into account the context in which the word is found and modify its values accordingly. Carrying a generalized embedding matrix around and plug it into the model for the task we’re trying to do is not sufficient, instead, a more complex structure must be included to the lower part of the model to find contextualized embeddings.
The encoder-decoder based structure for Neural Machine Translation(NMT) is an excellent example to introduce the pre-training + fine-tuning methodology that’s responsible for the latest breakthroughs in NLP in recent years.
This diagram from Learning in Translation: Contextualized Word Vectors(CoVe) describes the process of obtaining contextualized embeddings in pre-training and how they are then used for downstream tasks.
On the left, an encoder-decoder based model is trained for NMT, sentences from the original language are first processed y the encoder, and its outputs are then passed to the decoder for the final translation. This pre-training process is supervised, and the goal is for the encoder to learn how to capture syntactic and semantic meanings of words, and output contextualized embeddings. The encoder is based on a two-layer bidirectional LSTM architecture, while the decoder is based on attentional unidirectional LSTMs.
On the right, a pre-trained encoder takes in the inputs, GloVe embeddings, to find contextualized embeddings, they are then combined with the original inputs for a downstream task.
The limitation of CoVe is that 1) the pre-training is supervised so its bounded by the amount of labeled data, and 2) the architecture of the task-specific model remains to be defined and it’s not a trivial task to find one that can achieve outstanding results. By overcoming these two obstacles, we expect to find a model that can be 1)pre-trained with unlimited data — unsupervised training, and 2) fine-tuned on limited labeled data and together with some minor modifications on its architecture to achieve SOTA results for different NLP task.
A brief description of how ELMo, OpenAL GPT, and BERT tackled the above-mentioned limitations is introduced below. For a more detailed explanation check out this amazing post: Generalized Language Models by Lilian Weng.
Embeddings for Language Model(ELMo) obtains contextualized embedding by training a language model in an unsupervised fashion — taking in a sequence of tokens and learns to predict the next one given the history. Its structure is based on bidirectional LSTMs, where L layers of biLSTMs are stacked one on another, with each one outputting a different representation of the sequence.
Different layers are focused on different aspects of the syntactic/semantic meanings of the word. The output of ELMo combines all these hidden states linearly, including the input embeddings, to achieve better results. The higher layers focus more on the semantic meanings while the lower ones can capture better syntactic aspects.
Bidirectional LSTMs are used ensures the model is not only learning to predict a given token’s future but also its past.
ELMo is only used to find contextualized embeddings. For a given task, a specific model architecture still needs to be found.
OpenAI GPT, based on Transformer’s decoder, can be used directly for all end tasks. As shown in the image, different tasks have different preprocessing steps but only slight modifications are to be made on GPT’s Transformer model to complete the task.
Unlike ELMo, GPT is trained only to predict the future, but for a better understanding of the context of a given token, both the items from its left and right should be taken into account.
BERT is based on Transformer’s encoder and is trained to predict the context from both left and right. It’s pre-training consists of two tasks:
Mask Language Model, randomly replacing tokens with “[MASK]” and the model has to output its original word by looking at its context. By doing so, it forces the model to better capture both the syntactic and semantics meanings of the missing word.Next Sentence Prediction. Instead of taking one sentence at the time, BERT takes a pair of them, A and B, as inputs and it has to predict whether B entails A. Having trained on this task, the model will be more capable of understanding the relationships between sentences.
Mask Language Model, randomly replacing tokens with “[MASK]” and the model has to output its original word by looking at its context. By doing so, it forces the model to better capture both the syntactic and semantics meanings of the missing word.
Next Sentence Prediction. Instead of taking one sentence at the time, BERT takes a pair of them, A and B, as inputs and it has to predict whether B entails A. Having trained on this task, the model will be more capable of understanding the relationships between sentences.
BERT uses WordPiece tokenization embeddings for their inputs. Instead of generating normal word tokens, it uses subword tokenization to better address rare and unknown words, as most of them can be reconstructed using subwords. The two input sentences have different Sentence Embeddings and a special character is used for their separation. Positional Embeddings are also used.
When using BERT for a downstream task, just like GPT, only a few new parameters are to be added. Take text classification as an example, what we need to do is to take the “[CLS]” token’s embedding from the last layer and pass it to a softmax.
It wasn’t a short reading if you made it from top to bottom, but if you didn’t, here’s a quick recap:
Tree types of text representation were discussed in this piece, with the first two of them being sparse vectors: One-hot encoding, on one hand, is a type of token-level representation where a sentence before being used as an input is transformed into a matrix that has as many rows as its number of tokens. Count vectorizers, on the other hand, can take a sentence as a whole and squeeze it into one single vector. It relies on counting term frequencies, and it’s done at the expense of losing the information regarding sentence’s token ordering.
The usage of a Hashing Trick can help address the problem of memory consumption of a large vocabulary, and it also mitigates the problem of filter circumvention in the case of Spam Email Detection. Tf-idf is used to re-weight term frequencies for count vectors so less frequent but context-revealing terms won’t be ‘ignored’.
Word Embeddings, a more advanced technique that is used when a better capture of the semantic meanings of words is needed. A more complex structure is used to process the inputs to obtain Contextualized Embeddings, which in turn is used to address the issue of polysemy.
A brief discussion of the latest NLP models is conducted by the end of the article. The aim was to provide some basic ideas, more efforts should be invested for further understandings of these models.