A handbook to Text Preprocessing. The first step in Natural Language... | by Ramya Vidiyala | Towards Data Science

Natural Language Processing (NLP) is one of the most complex areas of Artificial Intelligence. The reason behind this is that text data is contextual and needs modification to make it machine-interpretable. It needs to undergo multiple stages of pre-processing.
In this blog, we will get to know what, why, how of text preprocessing with the simplest code to try it out.
Text data can be easily interpreted by humans. But it is a complex task to read and analyze the humongous amount of data.
To accomplish this task, we need to convert this text into machine-understandable data, i.e., transform words into numerical features that work with machine learning algorithms.
However, this transformation is not simple because text data contains redundant and repetitive words. So, we need to Preprocess text data before transforming it into numerical features.
The fundamental steps involved in Text Preprocessing are:
Cleaning raw dataTokenizingNormalizing tokens
Cleaning raw data
Tokenizing
Normalizing tokens
Let us look into each step with a code snippet.
In this article, we will be using Customer Support on Twitter dataset from Kaggle.
www.kaggle.com
The Customer Support on Twitter dataset is a large, modern corpus of tweets and replies to aid innovation in natural language understanding and conversational models, and for the study of modern customer support practices and impact. This dataset offers a large corpus of modern English (mostly) conversations between consumers and customer support agents on Twitter.
Importing data:
import pandas as pddata = pd.read_csv("customer-support-on-twitter.csv")
Using the pandas library to load data from CSV(Comma Separated Values) into the data frame.
data.columns
Let us consider only the ‘text’ column for pre-processing.
This phase involves the deletion of words or characters that do not add any value to the meaning of the text. Some of the common cleaning steps are,
Lowering case
Removal of special characters
Removal of stopwords
Removal of URLs
Removal of HTML tags
Removal of extra spaces
Lowering case of text is essential for the following reasons:
The words, ‘TEXT’, ‘Text’, ‘text’ add the same value to the sentence
For text vectorization, using techniques like TF-IDF counts the frequency of words irrespective of the case
Lowering the case of all words is also super helpful to reduce the dimensions by decreasing the size of the vocabulary.
data["text_lower"] = data["text"].str.lower()
Observation: All words are converted to their lower cases
This is another text preprocessing technique that will help to treat ‘hurray’ and ‘hurray!’ in the same way.
import reno_special_char=[]for sentence in data.text_lower:    no_special_char.append(re.sub('[^A-Za-z0-9]+', ' ', sentence))data["no_special_char"]=no_special_char
The regular expression library provides a function:sub( actual_pattern, replacing_pattern, data ) that helps us replace the first argument of it with the second argument in the data
Observation: Special characters like ‘@’, ‘ ` ’ are removed.
Stopwords are commonly occurring words in a language like ‘the’, ‘a’, and so on. They can be removed from the text most of the time, as they don’t provide valuable information.
from nltk.corpus import stopwordsstopwords = set(", ".join(stopwords.words('english')))
nltk.corpus contains a huge repository of common stop words.
no_stopwords=[]for sentence in data["no_special_char"]:    no_stopwords.append(' '.join(e.lower() for e in sentence.split() if e.lower() not in stopwords))data["no_stopwords"]=no_stopwords
Observation: Words like ‘the’, ‘to’, ‘at’ are removed
The next preprocessing step is to remove any URLs present in the data. There is a good chance that the tweet will have some URLs in it. Probably we might need to remove them for our further analysis. Simple code to eliminate URL is below:
no_url=[]for sentence in data["no_stopwords"]:    no_url.append(re.sub(r"http\S+", "", sentence))data["no_url"]=no_url
Here, re.sub( ), replaces any words starting with “http” with an empty string.
One another common preprocessing technique that will come handy in multiple places is the removal of HTML tags. This is especially useful if we scrap the data from different websites. We might end up having HTML strings as part of our text.
no_html_tags=[]for sentence in data["no_url"]:    no_html_tags.append(re.sub(r"'<.*?>'", "", sentence))
Here, using re.sub( ), we are replacing any pattern which is enclosed in “<>” with an empty string.
Tokenization is the process of splitting the text into smaller chunks, called tokens. Each token is an input to the machine learning algorithm as a feature.
for sentence in data["no_url"]:    sentence.split()
Split( ) converts sentences into words
Normalizing is cleaning based on the semantic of words. Normalizing tokens ensures consistency in the data that is further preprocessed or analyzed. There are two techniques of Normalizing Tokens:
Stemming
Lemmatization
It is the process of removing and replacing suffixes from the token to get the root or base form of the word which is known as a stem. Porter stemmer is a widely used stemming technique.
from nltk.stem.porter import PorterStemmer
nltk.stem provides PorterStemmer
def stem_words(text):    return " ".join([stemmer.stem(word) for word in text.split()])data["text_stemmed"] = data["no_url"].apply(lambda text: stem_words(text))
Observation: All words are transformed into their base form like ‘message’ is transformed into ‘messag’
The goal is the same as with stemming, but stemming a word sometimes loses the actual meaning of the word.
Lemmatization refers to doing things properly using vocabulary and morphological analysis of words.
It returns the base or dictionary form of a word, also known as the lemma.
from nltk.stem import WordNetLemmatizerlemmatizer = WordNetLemmatizer()
nltk.stem provides WordNetLemmatizer which is used widely to perform lemmatization
def lemmatize_words(text):    return " ".join([lemmatizer.lemmatize(word) for word in text.split()])data["text_lemmatized"] =data["text_stemmed"].apply(lambda text: lemmatize_words(text))data["text_lemmatized"]
Observation: Stemming transforms the word into base form by removing the suffix. This may lead to more confusion. Lemmatization transforms the word into the semantic base thus making it interpretable.
These are the different types of text preprocessing steps that are widely performed on text data. However, we need not perform all of these all the time. We need to carefully choose the preprocessing steps based on our use case since that also plays an important role.
I will follow up this article with processing these normalized tokens to vectors which are inputs to machine learning models.
Thanks for the read. I am going to write more beginner-friendly posts in the future. Follow me up on Medium to be informed about them. I welcome feedback and can be reached out on Twitter ramya_vidiyala and LinkedIn RamyaVidiyala. Happy learning!