How to turn Text into Features
By: Tiago Duque
Tags: NLP, Getting Started, AI, Data Science, Machine Learning

How to turn Text into Features

Simple question: How to turn text into features? Image by author

Imagine you’ve been tasked with the activity of building a Sentiment Analysis tool for your company product reviews. As a seasoned Data Scientist, you built many insights about future sale predictions and was even able to classify customers based on their purchase behavior.

But now, you’re intrigued: you have this bunch of text entries and have to turn them into features for a Machine Learning model. How can that be done? That’s a common question when Data Scientists meet text for the first time.

As simple as it may look for experienced NLP Data Scientists, turning text into features is not that trivial for newcomers in the area. The purpose of this article is to provide a guide into turning Text to Features, as a continuation to the NLP Series that I’ve been building for the last months (and its been a while since the last article, I know).

Previously, I’ve talked about several steps in the NLP Preprocessing Pipeline. Now, this is the last step of the Preprocessing Pipeline, when your dearly curated text is finally turned into usable features for Machine Learning Models (if that is your purpose — there can be NLP without Machine Learning, a topic for another time).

As usual, more than just explaining how to do it, I’ll present three of the most commonly used techniques of modeling: the Bag of Words Model, the Model based on the TF-IDF Algorithm and the Word2Vec Models. Let’s get started with the discussion.

Features?

For those of you that are not used to this word, let me digress a little over it. Feature is the name given to selected or treated data that is prepared to be used as input to algorithms (usually Machine Learning Algorithms). Features can be things like the price of a house, the RGB value of a pixel or, in our case, the representation of a word.

Nice picture taken from Google Developers Machine Learning Crash Course that depicts what are features.

There’s even a cool skill called “feature engineering” where Data Scientists work over data to get features from data. These features may even not be explicit in data, but can be obtained by modifying existing data or adding new data to make it more complete, hence helping achieve more robust decisions.

So, ultimately, our objective is to get the raw data (text) and turn into features (something computer algorithms can work with).

Text Vectorization

The techniques used to turn Text into features can be referred to as “Text Vectorization” techniques, since they all aim at one purpose: turning text into vectors (or arrays, if you want it simpler; or tensors, if you want it more complex), that can be then fed to machine learning models in a classical way.

Text vectorization aims to turn text to integer (or boolean, or floating numbers) vectors. Image by author.

Thinking about our expected outcome as a vector is a good starting point to visualize how we can turn text into features. Let us reason a little more about this. Consider the following phrase:

I want to turn my text into data.

In simple computational terms, vectors are lists with n positions. One natural way to think about how to turn text into vectors would be to create an ordered list of all the words, as such:

Image by author.

However, what happens if you have shorter or larger entries? And also, how could your machine learning algorithm (which is basically a series of matrix and vector computations) compare two words — these symbols invented by mankind that have special meanings?

Since you’re smart, you already thought: let me make a dictionary or some similar structure (in general, a vocabulary map) and use word indexes instead of words!

One supposed solution would be to create a mapping for each word… Image by author.

And then encode a vector with the word indices. Image by author.

You’re on the right path, but lets consider some of the problems here: Is the word “my” more important than “want”? That’s what your data ‘tells’ for a Machine Learning Algorithm. Machine Learning Algorithms don’t care if the number is an index, only that it is a value to compute over (of course, you could have a categorical feature, but we’ll see that further down).

Don’t yell, words are not values! Image from Spongebob tv show edited by author.

“Normalize, normalize, normalize!!!”, one can think. However, remember: these are not values! They are indices.

If you’ve been playing long enough with Data Science and Machine Learning, you’re probably thinking about the one solution: Use “One Hot Encoding”.

One Hot Encoding is a process for encoding categorical features where each possible value for that feature is mapped to a new “column” and, if present, this column is set to 1, 0 otherwise.

Let us consider this using the previously mentioned vocabulary “map” and the proposed phrase (in this case, the vocabulary is the same as the words of the phrase). We get this:

One-hot encoding for “I want to turn my text into data”. Image by author.

Now, in case we wanted to encode: “I want my data”, we’d get:

One-hot encoding for “I want my data”. Image by author.

Great, we’ve found a way to encode our data into a Machine Learning way! But there are plenty of problems to solve there: let us consider the first one — word frequencies — that’s where Bag of Words models come in!