A Beginner Guide on Preprocessing in Natural Language Processing
By: Limas Jaya Akeh
Tags: Machine Learning, Beginner, Data Preprocessing, Naturallanguageprocessing

Natural Language Processing is used frequently in our daily lives — whether you use Siri for reminding about unfinished tasks, sorting out emails, or even playing games!

All of those are possible thanks to Natural Language Processing, but what is Natural Language Processing? Simply put, It is how computers process human language, whether it is in the form of documents, or voice. The result of Natural Language Processing is a computer that “understands” and recently computers are able to respond back to the user.

Image Retrieved from WordStream

There are few steps involved in (well, at least, traditional) Natural Language Processing, Data Preprocessing, Converting it into features, Training the Model, and evaluating it. For this article, we will take a look at Preprocessing the data.

One of the challenges in Natural Language Processing is pre-processing the data so the computers can learn from it. Usually, Natural conversations have many words that carry almost no meaning, and it is our task to only “highlight” important words or keywords so that our computer knows that this word is carry meaning and informative.

There are several things you could do in Preprocessing the data:

Regex Filter (remove all unnecessary stuff like HTML tags, numbers, etc.) Case Folding (Lowercase Everything) Tokenization (split sentences into words) Stopword Removal (Remove most common words) Stemming / Lemmatization (turning words into their basic form)

Regex Filter

This step is usually used when you collect data, especially from scraping websites and storing it as a string, usually, there should be HTML tags which do not contribute to the meaning or context of the paragraph. So in this step we can use Regex which filters out most HTML tags and unnecessary information such as numbers.

Example Regex in Python language

Case Folding

This step only changes all Uppercase into Lowercase so there is no difference between FiSh, Fish, and fish. You can use the function .lower() in Python to make all words in the data become lowercase.

Tokenization

In this step, we basically just turn the filtered sentence into lexical tokens (or individual words) so that the computer can find importance of each words, instead of a sentence.

Stop word Removal

First of all, what is a Stop word? A stop word usually refers to the most common words in a language, for example, “to”, “you”, “we”, “he”, “a” can be safely ignored while retaining the information contained in the paragraph. You can also add your own stop words while doing the preprocessing.

Stemming / Lemmatization

These two methods have similar purposes — extracting the base form of the current word. Stemming works by removing the affix from each word, for example, “liked” will be changed into “like”. One example of a popular stemming algorithm is using PorterStemmer from NLTK library.

Lemmatization is similar to Stemming, but tends to be more accurate because it tries to refer to the vocabulary and the meaning of the word, while however more slower than Stemming.