Cleaning & Preprocessing Text Data by Building NLP Pipeline | by Kajal Yadav | Towards Data Science

For a while, I was dealing with textual data and I realize in today’s world, It is necessary to know how natural language processing works and what are the main steps to involve in order to get insights out of textual data.As we all know, there are a lot of statistics and maths that we can apply in order to get insights out of numerical data.But when it comes to the tedious form of textual data, we lack it in many places.
What is NLP text preprocessing?
NLP Text preprocessing is a method to clean the text in order to make it ready to feed to models. Noise in the text comes in varied forms like emojis, punctuations, different cases. All these noises are of no use to machines and hence need to clean it.
Why is NLP text preprocessing important?
Importance :
Properly cleaned data will help us to do good text analysis and help us in making accurate decisions for our business problems. Hence text preprocessing for machine learning is an important step.
Before I indulge you guys in the main steps involved in NLP text pre-processing, I would like to say you can add or remove few steps on the basis of the data that you have and its requirement.
In this tutorial, I barely gonna give you a definition for any step as there are plenty on the internet. Instead, I will gonna explain why you should be applying specific steps, how to do text preprocessing using python, what I understood in each step, and what was the outcome of my dataset.
Okay!! enough talking. Let us dive into the code now. 😄
Libraries & text preprocessing tools Required :
# Importing Libraries import unidecode import pandas as pd import re import time import nltk from nltk.corpus import stopwords nltk.download('stopwords') from nltk.tokenize import word_tokenize from nltk.stem import WordNetLemmatizer from autocorrect import Speller from bs4 import BeautifulSoup from nltk.corpus import stopwords from nltk import word_tokenize import string 
Read the Dataset :
# Read Dataset Df = pd.read_csv('New Task.csv', encoding = 'latin-1')# Show DatasetDf.head()
These are some text preprocessing steps that you can add or remove as per the dataset you have:
You may encounter lots of new lines for no reason in your textual dataset and tabs as well. So when you scrape data, those newlines and tabs that are required on the website for structured content are not required in your dataset and also get converted into useless characters like \n, \t. So, I have written a function that will delete all such nonsense.
When you scrape data, you may end up seeing HTML tags in the text of your dataset if you haven’t deal with it already while scraping. So, there is a need to deal with those tags later on. In this function, I am removing everything that matches HTML tags in my text.
This step will remove everything which is similar to hyperlinks of any type. I have added this function here as I have dealt with it on my dataset.
A single line function can be performed to remove extra whitespaces as mentioned below. This step is crucial before performing further NLP tasks.
What are the main NLP text preprocessing steps?
The below list of text preprocessing steps is really important and I have written all these steps in a sequence how they should be.
This is a crucial step to convert all characters like accented characters into machine-understandable language. So that further steps can be implemented easily. Accented characters are characters like â, î, or ô which have diacritics above the characters.
This is the next step in the series which is really important as case, CASE is two different words for the machine. So, you should be converting the case of the text in either lowercase or uppercase to proceed further.
This step is important as there might be scenarios where characters are repeating more than necessary which can’t be detected by a spell checker later on. Therefore, there is a need to deal with such a scenario beforehand before applying the spell checker function. There is another scenario that I encounter while working, there can be repeating punctuations as well. So there is a need to encounter them as well.
When we are very exciting, we do overwrite things that overwhelm readers.
Example:- Cheeeeeerrrrrrss !!!!!!
The explanation for using some symbols in the above regex expression
\1 → is equivalent to re.search(...). group(1). It refers to the first capturing group. \1 matches the exact same text that was matched by the first capturing group.
{1,} → It means we are matching for repetition that occurs more than one time.
DOTALL -> It matches the newline character as well unlike the dot operator which matches everything in the given text except the newline character.
sub() → This function is used to replace occurrences of a particular sub-string with another sub-string. This function takes as input the following: The substring to replace. The sub-string to replace with.
r’\1\1' → It limits all the repetition to two characters.
r’\1' → Limits all the repetition to only one character.
{2,} → It means to match for repetition that occurs more than two times
In order to remove Stop words in the next step, It is crucial that you deal with contractions first. Contractions are nothing but shorthand forms for words like, Do not, would not, It is. Contractions are anything similar to these examples don’t, wouldn’t, It’s.
In this step, We will go to learn how to remove special characters, why to remove them, and what special characters you should be keeping.
So, I have written a function that will remove a set of specified special characters and will gonna keep some important punctuations like (,.?!) excluding brackets. Special characters should be removed as when we gonna tokenized the text, later on, the punctuations don’t occur with more weightage.
To Remove numbers from the text :
All you need is to modify the given regex to
Formatted_Text = re.sub(r"[^a-zA-Z:$-,%.?!]+", ' ', text)
Just exclude the 0–9 range so as to remove all representation of numbers from the text. I didn’t perform this particular step on my dataset as the numbers are really important in my case to consider.
Punctuations that I am considering are Important as per my Dataset as I have to perform text summarization later on.
,.?! → These are some frequent punctuations that occur a lot and needed to be preserved to understand the context of the text.
: → This one is also frequent as per the Dataset. It is important to keep because it is giving sense whenever there is an occurrence of time like 9:05 p.m.
% → This one is also frequently used in many articles and telling more precisely about the data, facts & figures.
$ → This one is used in many articles where prices are considered. So, omitting this symbol will not give much sense about those prices that are left as just some numbers only.
Stopwords should be removed if you are performing tokenization, text summarization, text classification, or any similar task. As without stop-words, you can understand the context of the textual data presented to you. There is a need to remove them so as to reduce their weightage.
You should be considering this step of removing stop words since It didn’t work well with my further analysis. As I found on generating n-Grams, With stopwords files tend to give more reliable results than without stopwords files.
You should be very careful while attempting this one step. As this function may change the true meaning of the word. So you have to be very careful and try to see how things unfold on applying this function. If you are working on some industry-specific dataset, then you may need to consider relating the dictionary which tells this function explicitly to keep those specific words as it is.
In this step, I will gonna talk about two things precisely, Lemmatization and stemming.
People got confused with these two techniques like which one is efficient and what to use. So, Let me tell you about my experience and what did I prefer and why?
So, Both techniques actually trim the word to its root form like Planning will be trimmed to plan but Stemming won’t work in the case of the word “Better” whereas if you apply lemmatization, It will transform the word to its root form i.e., “good”. This is a major difference that Lemmatization works efficiently and I have used it only during my work.
Although I have written this function but on further analysis, I found that it didn’t perform well instead it was creating noise. So, On bit analysis for a few of the tokens, I realized it’s better to not apply lemmatization.
For example:- There was the word “Feed” which is frequently occurring all over the articles and which is really important as well. But on lemmatizing, “Feed” → reduced to → “Fee”. You see how it changed its entire meaning.
The data cleaning step entirely depends on the type of Dataset. Depending on the data, more steps can be include. It is a must to remove extra spaces so as to reduce file size.
These are some important steps that I have considered always while doing pre-processing of the textual data. You may take advantage of the code shared if you have to deal with the textual data as well or also don’t forget to let me know if you have tried something else to clean textual data. These steps are specific to the dataset that I have used, so feel free to add or remove it at your convenience.
You can also visit the complete Github repository which contains all the steps in a sequence of implementation with a brief explanation.
github.com
Cheers !! You have made it to the end. 😉
You can reach me via the following:
Subscribe to my YouTube channel for video content coming soon hereFollow me on MediumConnect and reach me on LinkedInFollow me on my blogging journey:- https://kajalyadav.com/Become a member:- https://techykajal.medium.com/membership
Subscribe to my YouTube channel for video content coming soon here
Follow me on Medium
Connect and reach me on LinkedIn
Follow me on my blogging journey:- https://kajalyadav.com/
Become a member:- https://techykajal.medium.com/membership
Check out my other blogs as well: