Text Cleaning Methods for Natural Language Processing | by Rebecca Vickery | Towards Data Science

Natural language processing is defined as “the application of computational techniques to the analysis and synthesis of natural language and speech”. In order to perform these computational tasks, we first need to convert the language of text into a language that the machine can understand.
In the following post, I am going to describe the most common steps involved in preparing text data for natural language processing. I’ll discuss some tools that can be used to perform these steps and provide some example python code to execute them.
Throughout this article, I will be using a data set taken from Kaggle which can be downloaded here. This data set consists of text from tweets and a target variable which categorises them into those that are about a genuine disaster and those that are not. The code below reads the data into a data frame using pandas. For simplicity, I have dropped all columns except the text and target variable.
import pandas as pdpd.set_option('display.max_colwidth', -1)train_data = pd.read_csv('train.csv')cols_to_drop = ['id', 'keyword', 'location']train_data = train_data.drop(cols_to_drop, axis=1)train_data.head()
One of the key steps in processing language data is to remove noise so that the machine can more easily detect the patterns in the data. Text data contains a lot of noise, this takes the form of special characters such as hashtags, punctuation and numbers. All of which are difficult for computers to understand if they are present in the data. We need to, therefore, process the data to remove these elements.
Additionally, it is also important to apply some attention to the casing of words. If we include both upper case and lower case versions of the same words then the computer will see these as different entities, even though they may be the same.
The code below performs these steps. To keep a track of the changes we are making to the text I have put the clean text into a new column. The output is shown below the code.
import redef  clean_text(df, text_field, new_text_field_name):    df[new_text_field_name] = df[text_field].str.lower()    df[new_text_field_name] = df[new_text_field_name].apply(lambda elem: re.sub(r"(@[A-Za-z0-9]+)|([^0-9A-Za-z \t])|(\w+:\/\/\S+)|^rt|http.+?", "", elem))      # remove numbers    df[new_text_field_name] = df[new_text_field_name].apply(lambda elem: re.sub(r"\d+", "", elem))        return dfdata_clean = clean_text(train_data, 'text', 'text_clean')data_clean.head()
Stop words are commonly occurring words that for some computational processes provide little information or in some cases introduce unnecessary noise and therefore need to be removed. This is particularly the case for text classification tasks.
There are other instances where the removal of stop words is either not advised or needs to be more carefully considered. This includes any situation where the meaning of a piece of text may be lost by the removal of a stop word. For example, if we were building a chatbot and removed the word“not” from this phrase “i am not happy” then the reverse meaning may in fact be interpreted by the algorithm. This would be particularly important for use cases such as chatbots or sentiment analysis.
The Natural Language Toolkit (NLTK) python library has built-in methods for removing stop words. The code below uses this to remove stop words from the tweets.
import nltk.corpusnltk.download('stopwords')from nltk.corpus import stopwordsstop = stopwords.words('english')data_clean['text_clean'] = data_clean['text_clean'].apply(lambda x: ' '.join([word for word in x.split() if word not in (stop)]))data_clean.head()
Stemming is the process of reducing words to their root form. For example, the words “rain”, “raining” and “rained” have very similar, and in many cases, the same meaning. The process of stemming will reduce these to the root form of “rain”. This is again a way to reduce noise and the dimensionality of the data.
The NLTK library also has methods to perform the task of stemming. The code below uses the PorterStemmer to stem the words in my example above. As you can see from the output all the words now become “rain”.
from nltk.stem import PorterStemmer from nltk.tokenize import word_tokenizeword_list = ['rains', 'raining', 'rain', 'rained']ps = PorterStemmer()for w in word_list:    print(ps.stem(w))
Before we can perform stemming on our data we need to tokenise the tweets. This is a method used to split the text into its constituent parts usually words. The code below uses NLTK to do this. I have put the output into a new column called “text_tokens”.
import nltk nltk.download('punkt')from nltk.tokenize import sent_tokenize, word_tokenizedata_clean['text_tokens'] = data_clean['text_clean'].apply(lambda x: word_tokenize(x))data_clean.head()
The code below uses the PorterStemmer method from NLTK to apply stemming to the text_tokens and outputs the processed text to a new column.
def word_stemmer(text):    stem_text = [PorterStemmer().stem(i) for i in text]    return stem_textdata_clean['text_tokens_stem'] = data_clean['text_tokens'].apply(lambda x: word_stemmer(x))data_clean.head()
The goal of lemmatization is the same as for stemming, in that it aims to reduce words to their root form. However, stemming is known to be a fairly crude method of doing this. Lemmatization, on the other hand, is a tool that performs full morphological analysis to more accurately find the root, or “lemma” for a word.
Again NLTK can be used to perform this task.
nltk.download('wordnet')from nltk.stem import WordNetLemmatizerdef word_lemmatizer(text):    lem_text = [WordNetLemmatizer().lemmatize(i) for i in text]    return lem_textdata_clean['text_tokens_lemma'] = data_clean['text_tokens'].apply(lambda x: word_lemmatizer(x))data_clean.head()
Part of speech (POS) tagging is a method to categorise words which gives some information relating to the way in which that word is used in speech.
There are eight primary parts of speech and they each have a corresponding tag. These are shown in the table below.
The NLTK libary has a method to perform POS tagging. The below code performs POS tagging on the tweets in our data set and returns a new column.
def word_pos_tagger(text):    pos_tagged_text = nltk.pos_tag(text)    return pos_tagged_textnltk.download('averaged_perceptron_tagger')data_clean['text_tokens_pos_tagged'] = data_clean['text_tokens'].apply(lambda x: word_pos_tagger(x))data_clean.head()
Chunking builds on POS tagging in that it uses the information from the POS tags to extract meaningful phrases from text. In many types of texts, if we reduce everything down to individual words we may lose a lot of meaning. In our tweets, for example, we have a lot of location names and other phrases which are important to keep together.
If we take this sentence “forest fire near la ronge sask canada” the location name “la ronge” and the words “forest fire” will convey an important meaning that we might not want to lose.
The spaCy python library has a method for this. If we apply this method to the above sentence we can see that it separates out the appropriate phrases.
import spacynlp = spacy.load('en')text = nlp("forest fire near la ronge sask canada")for chunk in text.noun_chunks:    print(chunk.text, chunk.label_, chunk.root.text)
This article gives a brief overview of some of the most popular methods for cleaning text data ready for natural language processing. For a deeper dive into these concepts and tools, this blog nlpforhackers.io is one of my favourites.
If you are interested in applying machine learning techniques to this data I wrote an article last week on how to make your first submission to the Kaggle competition where this dataset comes from.
Thanks for reading!