Preprocessing in Text Analytics
By: Nikolaus Herjuno Sapto Dwi Atmojo
Tags: Tokopedia, NLP, Text Analytics, Data Science, Data Analysis

Text Preprocessing

Amazing things we can do with text analytics do not happen like magic. Instead, it takes effort for achieving it. As I mentioned before, machine does not understand text or word explicitly. This is why Preprocessing Data becomes a critical step in transforming data to be well-served for machine to be processed later by algorithm. The process includes five stages as follow.

1. Sentence Segmentation (End of Sentence)

Sentence Segmentation or EoS (End of Sentence) is a process where we take sentences within paragraphs in order to avoid feeding a whole paragraph to algorithm directly.

An example below shows 1 paragraph as the input and several sentences broken down from the paragraph as the output.

In : [‘Monique Corzilius was born and raised in Pine Beach, New Jersey, the youngest of the three children of Fred and Colette Corzilius. While she was a child, her mother began taking her to New York City on child acting auditions. Working under the stage name Monique Cozy, her first job came at age two, modeling in a print advertisement for Lipton soup’] Out : [['Monique Corzilius was born and raised in Pine Beach, New Jersey, the youngest of the three children of Fred and Colette Corzilius'],['While she was a child, her mother began taking her to New York City on child acting auditions.'],['Working under the stage name Monique Cozy, her first job came at age two, modeling in a print advertisement for Lipton soup']]

These arrays of sentence will later be chopped up on the next process: Tokenization.

2. Tokenization / Chunking

Tokenization is a process of breaking sequences of string or character into smaller pieces called tokens. Tokens can be words, keywords, phrases, or even sentences itself. In the process of Tokenization, some characters like punctuation will be discarded.

Let us use the same sample from previous step of EoS.

In : ['Monique Corzilius was born and raised in Pine Beach, New Jersey, the youngest of the three children of Fred and Colette Corzilius'] Out : ['Monique', 'Corzilius', 'was', 'born', 'and', 'raised', 'in', 'Pine', 'Beach', 'New', 'Jersey', 'the', 'youngest', 'of', 'the', 'three', 'childrens', 'of', 'Fred', 'and', 'Collete', 'Corzilius']

As we can see, an array of sentence is chopped up into tokens. Sometimes the output of

Tokenization may not be accurate. Have a look at the result ‘Monique’ and ‘Corzilius’, and ‘New’ and ‘Jersey’. These tokens should be in one phrase, because it can not stand independently. Since we have this kind of problem, we need another process called Chunking. Chunking will take into consideration part of speech in order to group some words into one phrase or chunk. Let’s use previous example for this matter :

In : ['Monique Corzilius was born and raised in Pine Beach, New Jersey, the youngest of the three children of Fred and Colette Corzilius'] Out : [‘Monique Corzilius’, ‘was’, ‘born’, ‘and’, ‘raised’, ‘in’, ‘Pine Beach’, ‘New Jersey’, ‘the’, ‘youngest’, ‘of’, ‘the’, ‘three’, ‘children’, ‘of’, ‘Fred’, ‘and’, ‘Colette Corzilius’]

Instead of breaking ‘Monique Corzilius’ into 2 pieces of tokens, we keep this as 1 chunk.

3. Stop Words Removal

Sometimes, if you find some words that are extremely common on the document or even language, you can call it as stop words. Stop words hold less meaning than other words, so it is safe to discard stop words from documents. The simple ways to detect those words whether they are stop words or not are as follow:

Let us say you have a collection of documents. Group and aggregate the document by its terms. You will get the total count of words in each term. This will serve as Term Frequency. Sort for its term in a descending order.

Some common words excluded from previous step’s example:

In : ['Monique', 'Corzilius', 'was', 'born', 'and', 'raised', 'in', 'Pine', 'Beach', 'New', 'Jersey', 'the','youngest', 'of', 'the', 'three', 'childrens', 'of', 'Fred', 'and', 'Collete', 'Corzilius'] Out : ['Monique', 'Corzilius', 'born', 'raised', 'in', 'Pine', 'Beach', 'New', 'Jersey', 'youngest', 'the', 'three', 'childrens', 'Fred', 'Collete', 'Corzilius']

We remove some common words (‘was’, ‘and’, ‘in’, ‘of’) to keep only tokens which hold more meaning than others. By reducing stop words, we can significantly minimize number of words or tokens that a system has to store. Some libraries such as NLTK in Python can handle this, but sometimes you need to define or at least have your own stop word list if the library does not support your language. Sometimes we can use TF-IDF (Term Frequency — Inverse Document Frequency) instead of TF (Term Frequency) to make our stop words. TF-IDF evaluates the importance of each word or token in sets of corpus or documents.

4. Stemming and lemmatization

For grammatical reasons, we may find that documents need to use different forms of words, such as organize, organization, organizing, or eat, eating, eaten, ate or etc. Both of stemming and lemmatization is normalized at the following examples.

am, are, is => be eat, eating, eaten, ate => eat boy, boys, boy's => boy

Stemming usually refers to a process of removing words from its either prefix or suffix. So, the words or token are transformed into base forms. Different from stemming, lemmatization transforms words or token into base form, taking Part of Speech into account, rather than just derivational affixes.

5. Word Embedding / Text Vector

Word Embedding is a technique and modern way of representing words as vectors. The aim of this stage is to redefine high dimensional words features into low dimensional feature vectors. It represents words into X and Y vector coordinated. Common models that can be used are Word2Vec, FastText.

Text Vector itself is not just word embedding, it has many other techniques which can be used to transform text into vector, such as:

Bag of Words Continuous Bag of Words One Hot Encoding TF-IDF

Preparing good and well-served text dataset is a complicated art which requires choosing and applying correct algorithm. Many pre-built Python libraries can help you a lot in text preprocessing especially NLTK, Scikit-Learn, but sometimes you should define your own function to do some cleaning on your text dataset. If you are interested in Text Analytics or Natural Language Processing, follow our blog!