How to turn Text into Features. A comprehensive guide into using NLP... | by Tiago Duque | Towards Data Science

Imagine you’ve been tasked with the activity of building a Sentiment Analysis tool for your company product reviews. As a seasoned Data Scientist, you built many insights about future sale predictions and was even able to classify customers based on their purchase behavior.
But now, you’re intrigued: you have this bunch of text entries and have to turn them into features for a Machine Learning model. How can that be done? That’s a common question when Data Scientists meet text for the first time.
As simple as it may look for experienced NLP Data Scientists, turning text into features is not that trivial for newcomers in the area. The purpose of this article is to provide a guide into turning Text to Features, as a continuation to the NLP Series that I’ve been building for the last months (and its been a while since the last article, I know).
Previously, I’ve talked about several steps in the NLP Preprocessing Pipeline. Now, this is the last step of the Preprocessing Pipeline, when your dearly curated text is finally turned into usable features for Machine Learning Models (if that is your purpose — there can be NLP without Machine Learning, a topic for another time).
As usual, more than just explaining how to do it, I’ll present three of the most commonly used techniques of modeling: the Bag of Words Model, the Model based on the TF-IDF Algorithm and the Word2Vec Models. Let’s get started with the discussion.
For those of you that are not used to this word, let me digress a little over it. Feature is the name given to selected or treated data that is prepared to be used as input to algorithms (usually Machine Learning Algorithms). Features can be things like the price of a house, the RGB value of a pixel or, in our case, the representation of a word.
There’s even a cool skill called “feature engineering” where Data Scientists work over data to get features from data. These features may even not be explicit in data, but can be obtained by modifying existing data or adding new data to make it more complete, hence helping achieve more robust decisions.
So, ultimately, our objective is to get the raw data (text) and turn into features (something computer algorithms can work with).
The techniques used to turn Text into features can be referred to as “Text Vectorization” techniques, since they all aim at one purpose: turning text into vectors (or arrays, if you want it simpler; or tensors, if you want it more complex), that can be then fed to machine learning models in a classical way.
Thinking about our expected outcome as a vector is a good starting point to visualize how we can turn text into features. Let us reason a little more about this. Consider the following phrase:
I want to turn my text into data.
In simple computational terms, vectors are lists with n positions. One natural way to think about how to turn text into vectors would be to create an ordered list of all the words, as such:
However, what happens if you have shorter or larger entries? And also, how could your machine learning algorithm (which is basically a series of matrix and vector computations) compare two words — these symbols invented by mankind that have special meanings?
Since you’re smart, you already thought: let me make a dictionary or some similar structure (in general, a vocabulary map) and use word indexes instead of words!
You’re on the right path, but lets consider some of the problems here: Is the word “my” more important than “want”? That’s what your data ‘tells’ for a Machine Learning Algorithm. Machine Learning Algorithms don’t care if the number is an index, only that it is a value to compute over (of course, you could have a categorical feature, but we’ll see that further down).
“Normalize, normalize, normalize!!!”, one can think. However, remember: these are not values! They are indices.
If you’ve been playing long enough with Data Science and Machine Learning, you’re probably thinking about the one solution: Use “One Hot Encoding”.
One Hot Encoding is a process for encoding categorical features where each possible value for that feature is mapped to a new “column” and, if present, this column is set to 1, 0 otherwise.
Let us consider this using the previously mentioned vocabulary “map” and the proposed phrase (in this case, the vocabulary is the same as the words of the phrase). We get this:
Now, in case we wanted to encode: “I want my data”, we’d get:
Great, we’ve found a way to encode our data into a Machine Learning way! But there are plenty of problems to solve there: let us consider the first one — word frequencies — that’s where Bag of Words models come in!
One hot encoding only treats values as “present” and “not present”. That’s not appropriate for text. In many text applications, the frequency of words play an important role. Consider the two following paragraphs:
The dog is a domesticated carnivore of the family Canidae. It is part of the wolf-like canids, and is the most widely abundant terrestrial carnivore. The dog and the exant gray wolf are sister taxa as modern wolves are not closely related to the wolves that were first domesticated, which implies that the direct ancestor of the dog is extinct. The dog was the first species to be domesticated, and has been selectively bred over millennia for various behaviors, sensory capabilities, and physical attributes.
Today I went out with my dog and I found 100 dollars in the park. I’m sad to think that this money could be the lunch money for a week for some poor elderly lady.
The first is the first paragraph of the Wikipedia article about dogs and the second is a pseudo-blog-entry that I made with the purpose of demonstrating this problem. The question: which web page would you recommend to a user that’s searching a hypothetical engine with the word “dog”?
One hot encoding would give the same value for “dog” in both entries, so that’s not good.
Enters the Bag of Words model!
This model proposes that, instead of a vector of boolean values, you use a vector of word frequencies. In the above example, the column for word ‘dog’ would receive the value of ‘4’ in the first text and only ‘1’ in the second text. Now, it is okay to normalize the values, but not necessary (only for faster processing).
This model is called a “Bag” because it does not keep word “order” (just like our mommy’s bags in the 90’s were always messy).
But before talking about the Bag of Words model shortcomings, let’s see how to implement it using Python and Numpy.
It is very straightforward, but let’s see it step by step. I decided to make it a class, so I can instantiate multiple BoW’s in a model. This is the class constructor:
Basically we have a set of all words and a couple of dicts for storing the word indexes.
Next, we prepare our Bag of Words by fitting it (adding all the documents at once, so it can ‘learn’ which words we can use).
Finally, we can transform new inputs, returning arrays of the size of the entire vocabulary with the count the number of times that the word appears. For that I’m using numpy, which is a math and algebra library that is specialized in vector/matrix algebra. It’s python’s currently default library for this kind of task (and used as input format to most machine learning libraries).
This is an example usage and output:
Now that we’ve seen how it is done, we can talk about the problems in the model:
First: it ignores word order completely. You can understand this by looking at the following image:
Second: It tends to be very high dimensional. If your corpus accounts only the 90% most common English words at least once, according to the Oxford English Corpus, that’ll result in a vector of at least 7000 dimensions (most of which will be of zeroes, but is a lot of dimensions, nonetheless)!
But the BoW is simple and can bring quick results for simple problems. If you’re not sure where to start when building a NLP solution, try the basic: go for a BoW model. It works as a good baseline estimator.
Just to point out, there is one variation that is useful in Big Data cases, in special for situations where you have to compare texts in a character base (not accounting for semantics). This method uses text shingles (shingling). In this case, instead of breaking sentences into words, break at every k characters, or at every stop-word. Read more here:
nlp.stanford.edu
This is not actually a model, but rather an improvement into computing the “pertinence” of a word over a document. For simplicity, I’ll call it a model.
In TF-IDF Model, instead of storing the frequency of words, we store the result of the tf-idf algorithm over the input data. TF stands for Term Frequency and IDF stands for Inverse Document Frequency.
In short, TF-IDF computes the weight of a word in a specific document, taking into account the overall distribution of words. The way the results are presented is the same as Bag of Words: a sparse vector (0’s for words that does not appear, some float otherwise).
TF-IDF models can return better results when used, for example, in Sentiment Analysis tasks, when compared to Bags of Words.
Let us break this algorithm in two parts for better understanding: computing the global Inverse Document Frequency, and then computing the individual TF-IDF Score.
The first part is to compute the global IDF value for every word. This value represents the weight of each word over all documents (in the entire corpus). This means that very common words will have less weight overall, while rare words will weight more (this even removes the need to do stop-word removal, since because they are very common, they will weight less).
There are many distinct ways to do the computation of both TF and IDF (look at the algorithm Wikipedia page, for example). I’ve chosen to use the logscaled IDF, which is computed with the following formula:
Where:
D = total number of documents
freq = number of documents where the term appears.
To start, we define a helper method:
1) A simple method to turn a sentence into a dict of words and frequencies (one can use Python collections “Counter” for best performance, but I’ll use the old dict way for simplicity):
Then we initialize the IDF Class similar to the BoW Class:
For the fitting, we compute the global term frequency, followed by the idf for each word according to the formula we mentioned above.
Great, we have the global IDF for our document. Now, the TF-IDF is computed for each sentence, by finding the Term Frequency Score for each term in each sentence and multiplying by the global term IDF.
In short, we get the TF-IDF score for a sentence by:
Counting the term frequency for the terms in sentence.Multiplying the term frequency for each term by the inverse document frequency.
Counting the term frequency for the terms in sentence.
Multiplying the term frequency for each term by the inverse document frequency.
Here’s the code (notice that I added a few conditionals and private methods to account for batch transformation):
And with that our TF-IDF featurizer is done!
Now, for each sentence, we get an array the size of the entire vocabulary with the pertinence of every word to that sentence (0 if its missing).
Here goes an example output:
Or a little more “readable”:
{‘of’: 0.0, ‘list’: 0.0, ‘sentence’: 0.08664339756999316, ‘a’: 0.07192051811294521, ‘second’: 0.0, ‘sentences’: 0.0, ‘in’: 0.0, ‘complexity’: 0.0, ‘is’: 0.08664339756999316, ‘this’: 0.08664339756999316, ‘word’: 0.08664339756999316, ‘for’: 0.0}
You can find the code used to generate the above ‘explanation’ here.
Here’s the commit with the implementations made up to this point, be aware that there’ll be a few differences in code since I’ll be also using some structures that are built into my Toolset.:
github.com
As we have seen, BoW and TF-IDF methods produce a vector the size of the entire vocabulary for a single sentence. This disregards for word order or position, making these techniques bad for continuity sensitive applications (most NLP applications are).
One possible workaround would be to use a BoW/TFIDF array for each word, stack them and pass them as features, like in the following image (exemplified as the input layer of an Artificial Neural Network):
For the above image, you’d have a 10000*4 sparse matrix for a single sentence of 4 words (even using a bit for every boolean, you’d get around 1kb per word per sentence! Imagine a large corpus?). The compute time and storage used to train a simple sentiment analysis model would make it too expensive or unfeasible (in fact, a few years ago, text was an almost untouched topic when the matter was Machine Learning due to these constraints allied to the lack of enough memory and processing power).
However, this approaches allow us to keep word sequence. What if we could reduce the dimension of this vector?
Enters word embeddings!
I won’t go deep into the explanation of Word Embeddings, since there are several methods of computing them and most involve deep neural networks, which themselves take a time to explain (and this is not the focus of this article).
But I won’t refrain myself to give you the essential and most important information about it.
Let us state it this way:
Word embeddings are vector representation of words learnt from context training. Its not a score for each word, instead, it’s more like the word’s “coordinates”.
So when training a model instead of one-hot encodings the size of the vocabulary, you feed an array of word Embeddings representing the input. This array has a predefined d dimension depth, which is usually much less than the vocabulary size.
One of the most well known techniques to generate Word Embeddings is Word2Vec, which originated the method. Word2Vec itself can be computed using two distinct techniques, but the details are not so important here.
Instead, it is better for you to know that there are variations to the way to train/use embeddings. Here’s a summary of techniques used to create embeddings:
Word2vec (Google) - 2 techniques: Continuous Bag of Words (CBoW) and Skip-Gram;
Global Vectors or GloVe (Stanford);
fastText (Facebook) —interesting fact: accounts for out of vocabulary words.
If you want to learn more, I suggest this amazing course from DeepLearning.AI presented at Coursera: https://www.coursera.org/learn/probabilistic-models-in-nlp
The way word embeddings are generated, usually through a training process, allows the words to be encoded according to their contexts. This causes the interesting effect of embeddings also being able to represent word semantics (to an extent).
Since words are represented by coordinates, they be compared against (to compare similarity). And if the dimension is properly reduced with techniques such as Principal Component Analysis — PCA, words can be plotted and the plotting usually shows words with similar meanings closer together, as like in the following image, taken from IBM Research Blog:
The coordinates are usually given in a big number of dimensions, usually between 8 and 1024. This way, instead of a stack of 10000 dimension arrays filled with 0s, we have a stack of 8 to 1024 dimension arrays that are not sparse (they are densely filled with floats). This is better for computers to work with.
Here’s another cool example taken from an article by David Rozado in Journal Plos One:
When considering how people use Embeddings, it is important to point out that word Embeddings can be generated in two distinct ways:
By being trained together with the ongoing process (like a sentiment analysis task) and capture the relations between words according to that specific task. This is what happens when you define an Embeddings layer using libraries such as Keras, Pytorch or Trax. These layers map word indexes to an Embedding value.By being pretrained on a huge corpus to capture the most common relations in a language. These pretrained Embeddings can then be loaded into the model to help speed up the learning process or achieve better results. This is where fastText, Word2vec or GloVe comes in. Libraries such as spaCy offer you this kind of embeddings usage.
By being trained together with the ongoing process (like a sentiment analysis task) and capture the relations between words according to that specific task. This is what happens when you define an Embeddings layer using libraries such as Keras, Pytorch or Trax. These layers map word indexes to an Embedding value.
By being pretrained on a huge corpus to capture the most common relations in a language. These pretrained Embeddings can then be loaded into the model to help speed up the learning process or achieve better results. This is where fastText, Word2vec or GloVe comes in. Libraries such as spaCy offer you this kind of embeddings usage.
I won’t discuss how one can pretrain an embedding here, but I’m preparing a post exclusively tailored to explain this practice. I suggest you subscribe to my account to get a notification for when it is posted, or keep an eye to this paragraph, for I’ll post a link to the story right underneath it when it’s ready.
Just to add a little more about the topic, one can also do Feature Engineering when turning text into features. In other words, a Data Scientist can apply its own rules (usually through preprocessing) to define what should be extracted before turning text into numerical arrays.
As we’ve seen, turning text into features may seem a simple and trivial thing, but there are many things to consider.
Currently, most state of the art techniques in NLP only use Word Embeddings, since they are more robust and able to be used in a sequential manner.
Of course, each algorithm and problem requires a specific manipulation of input text. As an example, Seq2Seq models usually have a fixed sequence
length. To enforce this length, padding or compression is used. But in the end, the padding and the words are all converted into Embeddings.
Now that we’ve done with the most basic topics in NLP Preprocessing, we can start talking about applications and techniques. Don’t forget to subscribe to stay tuned and receive a notification about the next topic!
Also, always check what’s new on code in my repository. Feel free to reuse and contribute to the code according to the current Licensing. And you can get in touch with me through Linkedin if you want some help, just check my profile! See you in the next article.