NLP in Python-Data cleaning. Data cleaning steps involved in a... | by Divya Raghunathan | Towards Data Science

Data cleaning is a very crucial step in any machine learning model, but more so for NLP. Without the cleaning process, the dataset is often a cluster of words that the computer doesn’t understand. Here, we will go over steps done in a typical machine learning text pipeline to clean data.
We will work with a dataset that classifies news as fake or real. The dataset is available on Kaggle, the link to the dataset is below,
https://www.kaggle.com/clmentbisaillon/fake-and-real-news-dataset
There are two datasets- Fake and true news in the Kaggle dataset. I have concatenated the fake news and true news dataset. Ten fake and true news is below:
news.head(10)
news.tail(10)
We shall go over several steps to clean the news dataset to remove the unnecessary content and highlight the key attributes suitable for the ML model.
Step 1: Punctuation
The title text has several punctuations. Punctuations are often unnecessary as it doesn’t add value or meaning to the NLP model. The “string” library has 32 punctuations. The punctuations are:
import stringstring.punctuation
To remove the punctuation in our dataset, let’s create a function and apply the function to the dataset:
def remove_punctuation(text):    no_punct=[words for words in text if words not in string.punctation]    words_wo_punct=''.join(no_punct)    return words_wo_punctnews['title_wo_punct']=news['title'].apply(lambda x: remove_punctuation(x))news.head()
The column after step 1 has removed the punctuations from the title texts.
Step 2: Tokenization
Tokenizing is the process of splitting strings into a list of words. We will make use of Regular Expressions or regex to do the splitting. Regex can be used to describe a search pattern.
def tokenize(text):    split=re.split("\W+",text)     return splitnews['title_wo_punct_split']=news['title_wo_punct'].apply(lambda x: tokenize(x.lower()))news.head()
Here, “\W+” splits on one or more non-word character
The column after step 2 has created a list, by splitting all the non-word characters.
Step 3: Stop words
Now, we have a list of words without any punctuation. Let’s go ahead and remove the stop words. Stop words are irrelevant words that won’t help in identifying a text as real or fake. We will use “nltk” library for stop-words and some of the stop words in this library are :
stopword = nltk.corpus.stopwords.words('english')print(stopword[:11])
There are 179 stop words in this library.
def remove_stopwords(text):    text=[word for word in text if word not in stopword]    return textnews['title_wo_punct_split_wo_stopwords'] = news['title_wo_punct_split'].apply(lambda x: remove_stopwords(x))news.head()
The column after step 3 has removed the unnecessary stop words.
Step 4 : Lemmatize/ Stem
Stemming and Lemmatizing is the process of reducing a word to its root form. The main purpose is to reduce variations of the same word, thereby reducing the corpus of words we include in the model. The difference between stemming and lemmatizing is that, stemming chops off the end of the word without taking into consideration the context of the word. Whereas, Lemmatizing considers the context of the word and shortens the word into its root form based on the dictionary definition. Stemming is a faster process compared to Lemmantizing. Hence, it a trade-off between speed and accuracy.
Let’s consider the word “belief” for example. The different variations of believe can be believing, believed, believes, and believe .
print(ps.stem('believe'))print(ps.stem('believing'))print(ps.stem('believed'))print(ps.stem('believes'))
The stem results for all of the above is believ
print(wn.lemmatize(“believe”))print(wn.lemmatize(“believing”))print(wn.lemmatize(“believed”))print(wn.lemmatize(“believes”))
The lemmatize results in the order of print statements are — believe, believing, believed, and belief.
Lemmatize produces the same result if the word is not in the corpus. Believe is lemmatized to belief (the root word)
As seen in the above picture, lemmatize and stem yield different results. We can pick either one for our final model.
Step 5: Other steps
Other cleaning steps can be performed based on the data. I have listed a few of them below,
Remove URLsRemove HTML tagsRemove emojiRemove numbers
Remove URLs
Remove HTML tags
Remove emoji
Remove numbers
...
I’d love to hear your thoughts and feedback on my articles. Please do leave them in the comment section below.