A Beginner’s Guide to Preprocessing Text Data Using NLP | by Nabil M Abbas | Towards Data Science

Below I’ve outlined the code I used to preprocess my Natural Language Processing projects. This code has mostly been used on tweets obtained from Twitter for classifers. My hope is that this guide assists aspiring data scientists and machine learning engineers by familiarizing them with some of the very useful NLP tools and steps necessary for Natural Language Processing.
from nltk.stem import LancasterStemmer, SnowballStemmer, RegexpStemmer, WordNetLemmatizer #this was part of the NLP notebookimport nltknltk.download('punkt')#import sentence tokenizerfrom nltk import sent_tokenize#import word tokenizerfrom nltk import word_tokenize#list of stopwordsfrom nltk.corpus import stopwordsimport string
import emoji#checking if a character is an emojidef char_is_emoji(character):    return character in emoji.UNICODE_EMOJI#does the text contain an emoji?def text_has_emoji(text):    for character in text:        if character in emoji.UNICODE_EMOJI:            return True    return False#remove the emojidef deEmojify(inputString):    return inputString.encode('ascii', 'ignore').decode('ascii')
punct =[]punct += list(string.punctuation)punct += '’'punct.remove("'")def remove_punctuations(text):    for punctuation in punct:        text = text.replace(punctuation, ' ')    return text
The following function was used to do much of the preprocessing on tweets for a classifier project I was working on. This should be similarly applicable for other NLP projects you may find yourself working on. The above functions will assist below.
def nlp(df):    # lowercase everything    # get rid of '\n' from whitespace    # regex remove hyperlinks    # removing '&gt;'    # check for emojis    # remove emojis    # remove punctuation    # remove ' s ' from removing punctuation      # lowercase everything    df['token'] = df['tweet'].apply(lambda x: x.lower())    # get rid of '\n' from whitespace     df['token'] = df['token'].apply(lambda x: x.replace('\n', ' '))    # regex remove hyperlinks    df['token'] = df['token'].str.replace('http\S+|www.\S+', '', case=False)    # removing '&gt;'    df['token'] = df['token'].apply(lambda x: x.replace('&gt;', ''))    # Checking if emoji in tokens column, use for EDA purposes otherwise not necessary to keep this column    df['emoji'] = df['token'].apply(lambda x: text_has_emoji(x))    # Removing Emojis from tokens    df['token'] = df['token'].apply(lambda x: deEmojify(x))    # remove punctuations    df['token'] = df['token'].apply(remove_punctuations)    # remove ' s ' that was created after removing punctuations    df['token'] = df['token'].apply(lambda x: str(x).replace(" s ", " "))    return df
import spacyfrom spacy.lemmatizer import Lemmatizerfrom spacy.lookups import Lookupssp = spacy.load('en')lookups = Lookups()lemm = Lemmatizer(lookups)
def lemma_function(text):    dummy = []    #this is just a test to see if it works    for word in sp(text):        dummy.append(word.lemma_)    return ' '.join(dummy)df['lemmatized'] = df['token'].apply(lambda x: lemma_function(x))
After lemmatizing the tweets I found ‘-PRON-’ showing up in my text which is the “phrase” that appears after you lemmatize a pronoun using spacy. This is not significant in informing me of the content of a tweet so I remove this ‘-PRON-’ phrase as well.
df['lemmatized'] = df['lemmatized'].apply(lambda x: x.replace('-PRON-', ' '))
df['final_text'] = df['lemmatized'].apply(word_tokenize)
from nltk.corpus import stopwordsmy_stopwords = set(stopwords.words('english'))df['final_text'] = df['final_text'].apply(lambda text_list: [x for x in text_list if x not in stopwords.words('english')])
df['final_text'] = df['final_text'].apply(lambda list_data: [x for x in list_data if x.isalpha()])
Now that much of your tweet data has been preprocessed, your data should be much cleaner now. You have a wider variety of considerations outside of what has been shown above. In my classifier projects, I’ve also used TextBlob to autocorrect any misspellings in my corpus. But be mindful that TextBlob can be computationally expensive. Additionally, once you’ve fully preprocessed your data, and are ready to create your bag of words (count vectorizer) or TF-IDF vectorizer, you can adjust the parameters to fit your requirements for your machine learning problem.
I hope this guide speeds up the preprocessing of your text data for your next NLP project. Feel free to leave any thoughts and insights.