Beginner’s Guide to Data Cleaning and Feature Extraction in NLP | by Enes Gokce | Towards Data Science

This article will explain the steps of data cleaning and future extraction for text analysis done by using Neural Language Processing (NLP).
On the internet, there are many great text cleaning guides. Some of the guides are making feature extraction after text cleaning while some of them are making before the text cleaning. Both of the approaches work fine. However, here is the issue that gets little attention: In the data cleaning process, we are losing some possible features(variables). We need feature extraction before the data cleaning. On the other hand, some features make sense only when they are extracted after the data cleaning. Thus, we also need feature extraction after the data cleaning. This study pays attention to this point, and this is what makes this study unique.
In order to address the stated points above, this study follows three steps in order:
Feature Extraction — Round 1Data CleaningFeature Extraction — Round 2
Feature Extraction — Round 1
Data Cleaning
Feature Extraction — Round 2
This study article is a part of an Amazon Review Analysis with NLP methods. Here is my GitHub repo for the Colab Notebook of the codes for the main study, and codes for this study.
Brief information about the data I use: The data used in this project was downloaded from Kaggle. It was uploaded by the Stanford Network Analysis Project. The original data is coming from the study of ‘From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews’ done by J. McAuley and J. Leskovec (2013). This data set consists of reviews of fine foods from Amazon. The data includes all 568,454 reviews spanning 1999 to 2012. Reviews include product and user information, ratings, and a plain text review.
In this part, the features that are not possible to obtain after data cleaning will be extracted.
Number of stop words: A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query. In Python’s nltk package, there are 127 English stop words default. With applying stopwords, these 127 words were ignored. Before removing the stopwords, let’s have the ‘number of stopwords’ as a variable.
Number of stop words: A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query. In Python’s nltk package, there are 127 English stop words default. With applying stopwords, these 127 words were ignored. Before removing the stopwords, let’s have the ‘number of stopwords’ as a variable.
df['stopwords'] = df['Text'].apply(lambda x: len([x for x in x.split() if x in stop]))df[['Text','stopwords']].head()
2. The number of punctuation: Another feature that can’t be obtained after the data cleaning be because pronunciations will be deleted.
def count_punct(text):    count = sum([1 for char in text if char in string.punctuation])    return count#Apply the defined function on the text datadf['punctuation'] = df['Text'].apply(lambda x: count_punct(x))#Let's check the datasetdf[['Text','punctuation']].head()
3. Number of hashtag characters: One more interesting feature which we can extract from text data is the number of hashtags or mentions present in it. During the data cleaning, hashtags will be deleted, and we won’t have access to this information. Therefore, let’s extract this feature while we can still access it.
df['hastags'] = df['Text'].apply(lambda x: len([x for x in x.split() if x.startswith('#')]))df[['Text','hastags']].head()
4. Number of numerical characters: Having the number of numeric characters that are present in the reviews can be useful.
df['numerics'] = df['Text'].apply(lambda x: len([x for x in x.split() if x.isdigit()]))df[['Text','numerics']].head()
5. Number of Uppercase words: Emotions such as anger, rage are quite often expressed by writing in UPPERCASE words which makes this a necessary operation to identify those words. During the data cleaning, all letters will converted to lowercase.
df['upper'] = df['Text'].apply(lambda x: len([x for x in x.split() if x.isupper()]))df[['Text','upper']].head()
Now, we are done with features that can only be obtained before data cleaning. We are ready to clean the data.
Before applying NLP techniques on the data, firstly the data needs to be cleaned and prepared the data for the analysis. If this process is not done properly, it can ruin the analysis part totally. Here are the steps that were applied to the data:
Make all text lower case: The first pre-processing step is transforming the reviews into lower case. This avoids having multiple copies of the same words. For example, while calculating the word count, ‘Dog’ and ‘dog’ will be taken as different words if we ignore this transformation.
Make all text lower case: The first pre-processing step is transforming the reviews into lower case. This avoids having multiple copies of the same words. For example, while calculating the word count, ‘Dog’ and ‘dog’ will be taken as different words if we ignore this transformation.
df['Text'] = df['Text'].apply(lambda x: " ".join(x.lower() for x in x.split()))df['Text'].head()
2) Removing Punctuation: Punctuations creates noise in the data and, should be cleared. For now, NLP methods do not have a meaningful way to analyze punctuations. Thus, they were removed from the text data. With this step, these characters were removed: [!”#$%&’()*+,-./:;=>?@[\]^_`{|}~]
df['Text'] = df['Text'].apply(lambda x: " ".join(x.lower() for x in df['Text'] = df['Text'].str.replace('[^\w\s]','')df['Text'].head()
3) Removal of Stop Words: With this step, I removed all default English stop words in nltk package.
from nltk.corpus import stopwordsstop = stopwords.words('english')df['Text'] = df['Text'].apply(lambda x: " ".join(x for x in x.split() if x not in stop))df['Text'].sample(10)
Adding your own stopwords: At this point, you may want to add your own stopwords. I do this mostly after checking the most frequent words. We can check the most frequent words in this way:
import pandas as pdfreq = pd.Series(' '.join(df['Text']).split()).value_counts()[:20]freq
From these words, I want to remove ‘br’, ‘get’, and ‘also’ because they don’t make much sense. Let’s add them to the list of the stopwords:
# Adding common words from our document to stop_wordsadd_words = ["br", "get", "also"]stop_words = set(stopwords.words("english"))stop_added = stop_words.union(add_words)df['Text'] = df['Text'].apply(lambda x: " ".join(x for x in x.split() if x not in stop_added))df['Text'].sample(10)
Note: In other guides, you may come across that TF-IDF method. TF-IDF is another way to get rid of words that have no semantic value from the text data. If you are using TF-IDF, you don’t need to apply stopwords (but applying both of them is no harm).
4) Removing URLs: URLs are another noise in the data that were removed.
def remove_url(text):     url = re.compile(r'https?://\S+|www\.\S+')    return url.sub(r'', text)# remove all urls from dfimport reimport stringdf['Text'] = df['Text'].apply(lambda x: remove_url(x))
5) Remove html HTML tags: HTML is used extensively on the Internet. But HTML tags themselves are not helpful when processing text. Thus, all texts beginning with url will be deleted.
def remove_html(text):    html=re.compile(r'<.*?>')    return html.sub(r'',text)# remove all html tags from dfdf['Text'] = df['Text'].apply(lambda x: remove_html(x))
6) Removing Emojis: Emojis can be an indicator of some emotions that can be related to being customer satisfaction. Unfortunately, we need to remove the emojis in our text analysis because for now, it is currently not possible to analyze emojis with NLP.
# Reference : https://gist.github.com/slowkow/7a7f61f495e3dbb7e3d767f97bd7304bdef remove_emoji(text):     emoji_pattern = re.compile("["        u"\U0001F600-\U0001F64F"  # emoticons        u"\U0001F300-\U0001F5FF"  # symbols & pictographs        u"\U0001F680-\U0001F6FF"  # transport & map symbols        u"\U0001F1E0-\U0001F1FF"  # flags        u"\U00002702-\U000027B0"        u"\U000024C2-\U0001F251"        "]+", flags=re.UNICODE)    return emoji_pattern.sub(r'', text)#Exampleremove_emoji("Omg another Earthquake 😔😔")
# remove all emojis from dfdf['Text'] = df['Text'].apply(lambda x: remove_emoji(x))
7) Remove Emoticons: What is the difference between emoji and emoticons?
·:-) is an emoticon
·😜 is an→ emoji.
!pip install emot #This may be required for the Colab notebookfrom emot.emo_unicode import UNICODE_EMO, EMOTICONS# Function for removing emoticonsdef remove_emoticons(text):    emoticon_pattern = re.compile(u'(' + u'|'.join(k for k in   EMOTICONS) + u')')    return emoticon_pattern.sub(r'', text)#Exampleremove_emoticons("Hello :-)")
df['Text'] = df['Text'].apply(lambda x: remove_emoticons(x))
8) Spell Correction: On Amazon reviews, there are a plethora of spelling mistakes. Product reviews are sometimes filled with hastily sent reviews that are barely legible at times.
In that regard, spelling correction is a useful pre-processing step because this also will help us in reducing multiple copies of words. For example, “Analytics” and “analytcs” will be treated as different words even if they are used in the same sense.
from textblob import TextBlobdf['Text'][:5].apply(lambda x: str(TextBlob(x).correct()))
9. Lemmatization: Lemmatization is the process of converting a word to its base form. Lemmatization considers the context and converts the word to its meaningful base form. For example:
‘Caring’ -> Lemmatization -> ‘Care’
Python NLTK provides WordNet Lemmatizer that uses the WordNet Database to lookup lemmas of words.
import nltkfrom nltk.stem import WordNetLemmatizer # Init the Wordnet Lemmatizerlemmatizer = WordNetLemmatizer()df['Text'] = df['Text'].apply(lambda x: lemmatizer(x))
For more detailed background about lemmatization, you can check Datacamp.
Here, I will stop cleaning the data. However, as a researcher, you may need more text cleaning depending on your data. For example, you may want to use:
⚫ stemming the text data
⚫ Alternative ways for spell correction: isolated-term correction and context-sensitive correction methods
⚫ Different packages use different numbers of stopwords. You can try other NLP packages.
Some features will be extracted after text cleaning because they are more meaningful to obtain at this step. For example, the number of characters would be affected badly from URL links if we would extract this feature before the data cleaning. At this point, we must try to extract as many features as possible since extra features have a chance to provide useful information during the text analysis. We don’t have to worry about whether the features will actually be useful in the future or not. In the worst case, we don’t use them.
Number of Words: This feature tells how many words there are in the review
Number of Words: This feature tells how many words there are in the review
df['word_count'] = df['Text'].apply(lambda x: len(str(x).split(" ")))df[['Text','word_count']].head()
2. Number of characters: How many letters are contained in the review.
df['char_count'] = df['Text'].str.len() ## this also includes spacesdf[['Text','char_count']].head()
3. Average Word Length: Average number of letters in the words in a review.
def avg_word(sentence):    words = sentence.split()    return (sum(len(word) for word in words)/(len(words)+0.000001))df['avg_word'] = df['Text'].apply(lambda x: avg_word(x)).round(1)df[['Text','avg_word']].head()
Let’s check how the extracted features look like in the dataframe:
df.sample(5)
This study explains the steps of text cleaning. In addition, this guide is unique in a way that feature extraction is done by two rounds: before the text cleaning and the after text cleaning. We need to remember that for an actual study, text cleaning is a recursive process. Once we see an anomaly, we come back and make more cleaning by addressing the anomaly.
*Special thanks to my friend Tabitha Stickel for proofreading this article.