Natural Language Processing Workflow | by Jason Wong | Towards Data Science

Natural Language Processing (NLP) is the study of how computers interact (i.e. understand, interpret, manipulate) with humans through language, (e.g. speech, text). NLP got its start from the field of Linguistics, the study of language, primarily focusing on its semantics, phonetics, and grammar. Before machine learning began to show success with NLP tasks, it was mainly programming algorithms with rule-based methods from linguistics. The machine learning methods provided better accuracy, faster processing times, and dependability, resulting in the rule-based approaches taking the back seat.
I will be going through the NLP process from data preprocessing to model evaluation and selection. The NLP libraries used for this tutorial are:
Scikit-Learn — Documentation
NLTK (Natural Language Toolkit) — Documentation
The concepts that will be covered are as follows:
Exploratory Data Analysis (Frequency Distribution | Parse Trees)
Text Preprocessing (Tokenize, Stem, Lemmatize, Vectorize)
Feature Engineering (Bigrams, POS-Tags, TF-IDF)
Modeling
Model Evaluation
As with all data science projects, I will be following the CRISP-DM workflow. I have labelled steps for NLP to the right side of the CRISP-DM model in the image below.
When working with text data, one of the first steps is to vectorize the text to create a Bag of Words (BoW). This bag will hold information about the individual words, e.g., a count of how many times each word appears in a corpus. The words in the bag are not in any specific order and if we have a large enough corpus, we may begin to notice patterns.
Before creating a BoW, the text data needs to be cleaned and tokenized. This will prevent the words with different punctuation and/or capitalization being counted separately. By removing punctuation and lowercasing everything, we’re creating word tokens.
Regular Expressions, “regex” for short, is a pattern describing a certain amount of text (Goyvaerts). This tool allows us to filter through and pull information from a text document without having to physically read it. Regex is especially helpful when tokenizing text by defining the rules for where strings will be split into tokens. The regular expressions library contains many useful tools:
Ranges — [A-Z] [A-Za-z0-9]
Instead of typing every letter in the alphabet, we can use a range from a-z. The left pattern above, [A-Z], will match on any uppercase letter. The pattern on the right, [A-Za-z0-9], will match on any uppercase letter, lowercase letter, and digit.
Character Classes — \w \W \d
Character classes, in a way, are similar to ranges. Think of these as a shortcut to ranges. The \w class would find any lowercase word and the \d class would match on any digit.
Groups — (A-Z0-9)
Also similar to ranges, but require more specificity to match. The pattern above (A-Z0-9) can only match on that exact sequence.
Quantifiers — {*} {+} {?}
Work hand in hand with groups, giving us the ability to specify the amount of times a group need to appear in order to match. The {*} will match on a group that occurs 0 or more times, {+}, 1 or more times, and {?} 0 or 1 times.
Installing NLTK
!pip install nltkconda install -c anaconda nltk
For this tutorial I will be using a dataset containing satirical and real news articles. The satirical articles were obtained from The Onion and the real news articles were obtained from Reuters. The satirical and real news articles as a whole can be referred to as the corpus.
Loading in the dataset (Corpus)
import pandas as pdcorpus = pd.read_csv('data/satire_nosatire.csv')
Now that the corpus has been loaded in, it’s always a good idea to check how many documents there are, as well as, the class balance of the target variable.
corpus.head()
#Checking to how many documents are in the corpuscorpus.shape#Output:(1000, 2)
This corpus contains 1000 documents in the body column and 2 classes in the target column.
#Checking the class balancecorpus.target.value_counts()#Output:1    5000    500Name: target, dtype: int64
This is a balanced dataset containing 500 documents of each class. Class 1 referring to a satirical article, and class 2, a real news article.
Manually removing capitals
Even though the NLTK library provides tokenizers to perform text preprocessing, manually tokenizing provides some freedom during preprocessing.
first_doc = corpus.iloc[0].body#Removing Capitalsmanual_clean = [x.lower() for x in first_doc.split(' ')]manual_clean[:10]#Output:['noting', 'that', 'the', 'resignation', 'of', 'james', 'mattis', 'as', 'secretary', 'of']
Manually removing punctuation
Next, we need to remove the punctuation. We can manually do this with the String library. First, I am importing the String library and calling the punctuation function. Then I am using list comprehension to join each character together as long as the character is not a punctuation character.
import stringstring.punctuation#Output:'!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'#Removing punctuation#s > Sentence#c > Charactermanual_clean = [''.join(c for c in s if c not in string.punctuation) for s in manual_clean]
Checking the frequency distribution with the FreqDist function from NLTK
#Checking the frequency distribution before stopword removalfrom nltk.probability import FreqDistimport matplotlib.pyplot as pltfdist = FreqDist(manual_clean)plt.figure(figsize=(8,8))fdist.plot(25)
As you can see, the plot above is showing the highest frequency with stopwords. Since these words have low semantic value, they will be removed.
Removing stopwords
Stopwords are basically filler words, e.g., prepositions and conjunctions. These words hold a low semantic value and should be removed so they aren’t counted when we vectorize. We can make use of the list of stopwords provided by NLTK. Something to note, if the stopword list doesn’t contain a stopword in a document, we can just add to the list.
from nltk.corpus import stopwordsmanual_clean = [x for x in manual_clean if x not in stopwords.words('english')]#Checking the frequency distribution with the stopwords removedfdist = FreqDist(manual_clean)plt.figure(figsize=(8,8))fdist.plot(25)
From the frequency distribution plot above, we can see there are still a few filler words we can add to our stopwords list. Let’s go ahead and add could and one to the list.
#Adding to the list of stopwordsstopwords_list = stopwords.words('english')stopwords_list.extend(['could', 'one'])manual_clean = [x for x in manual_clean if x not in stopwords_list]
Removing numbers with regex
Numbers, like stopwords, usually have a low semantic value. This is the case for the text data we’re using so let’s go ahead and remove them. Removing numbers from the first document for example purposes.
Preview of first_doc
‘Noting that the resignation of James Mattis as Secretary of Defense marked the ouster of the third top administration official in less than three weeks, a worried populace told reporters Friday that it was unsure how many former Trump staffers it could safely reabsorb. “Jesus, we can’t just take back these assholes all at once — we need time to process one before we get the next,” said 53-year-old Gregory Birch of Naperville, IL echoing the concerns of 323 million Americans in also noting that the country was only now truly beginning to reintegrate former national security advisor Michael Flynn. “This is just not sustainable. I’d say we can handle maybe one or two more former members of Trump’s inner circle over the remainder of the year, but that’s it. This country has its limits.” The U.S. populace confirmed that they could not handle all of these pieces of shit trying to rejoin society at once.’
(Retrieved from https://www.theonion.com/)
import repattern = '\d+'first_doc = corpus.iloc[0].bodynum = re.findall(pattern, first_doc)num#Output:['53', '323']
Stemming is a method where the end of words are removed if it shows a derivational change to the word. Basically just reducing a word tokens down to its root word. This is helpful because most of the semantic meaning of a word is in the root, and a majority of the time, the beginning of a word is the root.
from nltk.stem import *porter_stemmer = PorterStemmer()snowball_stemmer = SnowballStemmer(language="english")
To see the difference between these two stemmers, let’s stem the first_doc with them both.
#Looping through each word in the document and stem it with both stemmersfor word in first_doc:    porter_word = p_stemmer.stem(word)    snowball_word = s_stemmer.stem(word)        if porter_word != snowball_word:        print(porter_word, snowball_word)#Output:jesu jesuscan’t can'tnapervil napervilli’d i'dtrump’ trumpthat’ that
Lemming is similar to stemming but instead of just chopping off the end, uses part of speech tags when determining how to transform a word. Lemmatization reduces words showing sort of inflection in order to return a root word that belongs to the language. For example, stemming the word mice would not return mouse but lemming would.
from nltk.stem import WordNetLemmatizer   lemmatizer = WordNetLemmatizer()print(f'knives becomes: {lemmatizer.lemmatize("knives")}')print(f'Noting becomes: {lemmatizer.lemmatize(first_doc[0])}')#Output:knives becomes: knifeNoting becomes: noting
If we wanted to PoS tag the text ourselves, we can make use of the pos_tag function from NLTK. This function will tag each word in a document and return the word along with its PoS tag. We can then, transform the NLTK tags to the tags of the WordNetLemmatizer.
from nltk import pos_tagfirst_doc_tagged = pos_tag(first_doc)first_doc_tagged[:10]#Output:[('noting', 'VBG'), ('resignation', 'NN'), ('james', 'NNS'), ('mattis', 'VBP'), ('secretary', 'NN'), ('defense', 'NN'), ('marked', 'VBD'), ('ouster', 'JJ'), ('third', 'JJ'), ('top', 'JJ')]
We can make a function to transform the tags to match the WordNetLemmatizer tags.
from nltk.corpus import wordnetdef wordnet_tags(tag):    if tag.startswith('J'):        return wordnet.ADJ    elif tag.startswith('V'):        return wordnet.VERB    elif tag.startswith('N'):        return wordnet.NOUN    elif tag.startswith('R'):        return wordnet.ADV    else:        return wordnet.NOUN
Transforming part of speech tags
first_doc_tagged = [(x[0], wordnet_tags(x[1])) for x in first_doc_tagged
Lemmatizing first_doc_tagged with the tags we obtained with the wordnet_tags function above
first_doc_lemmed = [lemmatizer.lemmatize(x[0], x[1]) for x in first_doc_tagged]
When performing NLP, the features can be extracted from the text by making a Bag-of-words (BoW). The BoW method will create a Document Term Matrix, a matrix where each column is a unique word and each row is a document. We can use Scikit-Learn’s CountVectorizer to do this on our first_doc_lemmed.
import pandas as pdfrom sklearn.feature_extraction.text import CountVectorizer#Transform text documents to a matrix with token countsvec = CountVectorizer()X = vec.fit_transform([" ".join(first_doc_lemmed)])#Casting results as a Pandas DataFramedf = pd.DataFrame(X.toarray(), columns = vec.get_feature_names())df.head()
Instead of tokenizing the text we could also create Bigrams. A bigram is two adjacent words that are treated as one. Bigrams are helpful when performing sentiment analysis on text data, e.g., upset, barely upset. With bigrams we can apply a frequency filter to remove the bigrams that occur due to random chance. If a bigram occurs multiple times, it must have some meaning. The minimum frequency filter depends on the number of factors in the data, a normal starting value is 5 but should also be experimented with to get the most optimal value.
Bigrams can also be created by calculating their Pointwise Mutual Information Score (PMI). This measures the mutual dependence between a pair of words. The bigram “social media” would most likely return a high PMI score, the pair are more than likely appear together than not.
vec = CountVectorizer(token_pattern=r"([a-zA-Z]+(?:'[a-z]+)?)", stop_words=stopwords_list, ngram_range=[1,2]) #<- Allowing bigramsX = vec.fit_transform(corpus.body[0:5])df = pd.DataFrame(X.toarray(), columns = vec.get_feature_names())df.head()
TF-IDF scales down the frequencies of tokens to represent how important a word is.
TF (Term Frequency) — frequency of the word in a single document divided by the total number of words in the document.
IDF (Inverse Document Frequency) — measure of how much information is provided by a term. IDF can be calculated by taking the total number of documents in the corpus, dividing by the number of documents the term appears in, and taking the logarithm of the result.
TF-IDF — Term Frequency and the Inverse Document Frequency multiplied together
from sklearn.feature_extraction.text import TfidfVectorizertf_vec = TfidfVectorizer(token_pattern=r"([a-zA-Z]+(?:'[a-z]+)?)",    stop_words=stopwords_list)X = tf_vec.fit_transform(corpus.body)df = pd.DataFrame(X.toarray(), columns = tf_vec.get_feature_names())df.head()
Instead of manually going through and performing the preprocessing steps above, let’s just make a function to perform these steps up until count vectorizing.
from nltk.corpus import wordnetfrom nltk import pos_tagfrom nltk.stem import WordNetLemmatizerdef prepare_doc(doc, stopwords=stopwords_list):    regex_tokenizer = RegexpTokenizer(r"([a-zA-Z]+(?:’[a-z]+)?)")    doc = regex_tokenizer.tokenize(doc)    doc = [x.lower() for x in doc]    doc = [x for x in doc if x not in stopwords_list]    doc = pos_tag(doc)    doc = [(x[0], wordnet_pos(x[1])) for x in doc]    #Could stem here instead of lemmatizing    lemma = WordNetLemmatizer()     doc = [lemma.lemmatize(x[0], x[1]) for x in doc]    return ' '.join(doc)
Below, I will import the necessary libraries and use the prepare_doc function above to clean the text. Next, I will perform a train test split and create a pipeline for the remaining preprocessing steps (TF-IDF Vectorizing, TF-IDF Normalisation). I will then fit the pipeline to the train split and predict on the test split.
#Importsimport pandas as pdimport numpy as npimport matplotlib.pyplot as plt%matplotlib inlineimport seaborn as snsimport nltkfrom nltk import pos_tagfrom nltk.corpus import stopwordsfrom nltk.tokenize import regexp_tokenize, word_tokenize, RegexpTokenizerimport sklearn.feature_extraction.text from sklearn.feature_extraction.text import TfidfVectorizer, TfidfTransformer, CountVectorizerfrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.ensemble import RandomForestClassifierfrom sklearn.metrics import confusion_matrix, mean_squared_error, classification_reportfrom sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, roc_auc_score
Loading in the corpus
df = pd.read_csv('../data/satire_nosatire.csv')df.head(10)
#Defining stopwordsstopwords_list = stopwords.words('english')X = df.body#Preprocessing function to remove numbers, stopwords, lowercase the #capitals, generate PoS tags, and lemmatize the textX = [doc_preparer(x, stopwords_list) for x in X]y = df.targetX[:2]
Performing train test split
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
Creating preprocess pipeline and the full pipeline with a Random Forest Classifier
tfidf_vec = Pipeline([('tfidf_vec', TfidfVectorizer())])tfidf = Pipeline([('tfidf', TfidfTransformer())])preprocess = Pipeline([    ('tfidf_vec', tfidf_vec),    ('tfidf', tfidf)])pipe = Pipeline([    ('preprocess', preprocess),    ('clf', RandomForestClassifier())])
Fitting the pipeline to X_train and y_train
pipe.fit(X_train, y_train)
Predicting on X_test
y_hat = pipe.predict(X_test)
Defining evaluation function to plot a confusion matrix and return the accuracy, precision, recall, and f1 scores
def evaluation(y, y_hat, title = 'Confusion Matrix'):    cm = confusion_matrix(y, y_hat)    precision = precision_score(y, y_hat)    recall = recall_score(y, y_hat)    accuracy = accuracy_score(y,y_hat)    f1 = f1_score(y,y_hat)    print('Recall: ', recall)    print('Accuracy: ', accuracy)    print('Precision: ', precision)    print('F1: ', f1)    sns.heatmap(cm,  cmap= 'PuBu', annot=True, fmt='g', annot_kws=    {'size':20})    plt.xlabel('predicted', fontsize=18)    plt.ylabel('actual', fontsize=18)    plt.title(title, fontsize=18)        plt.show();
Evaluating the classifier
evaluation(y_test, y_hat)
With NLP there are many different tools and methods you can use. It is worth taking the time to understand how the NLP libraries preprocess text in different ways, ensuring you choose the best method for your task. Making use of pipelines with the preprocessing and modeling steps help to streamline the workflow while cleaning up the code. I hope the concepts covered in this post can serve as a helpful resource when performing NLP.
Natural Language Toolkit. (n.d.). Retrieved from https://www.nltk.org/
Learn. (n.d.). Retrieved from https://scikit-learn.org/stable/
Goyvaerts, J. (n.d.). Regular Expressions TutorialLearn How to Use and Get The Most out of Regular Expressions. Retrieved November 20, 2020, from https://www.regular-expressions.info/tutorial.html
Brownlee, J. (2019, August 07). What Is Natural Language Processing? Retrieved from https://machinelearningmastery.com/natural-language-processing/
Re — Regular expression operations. (n.d.). Retrieved from https://docs.python.org/3/library/re.html
Saurav Jain Enthusiastic Coder Developer and always a learner, Jain, S., & Enthusiastic Coder Developer and always a learner. (2020, September 17). Introduction to Stemming. Retrieved from https://www.geeksforgeeks.org/introduction-to-stemming/
Stemming and Lemmatization in Python. (n.d.). Retrieved from https://www.datacamp.com/community/tutorials/stemming-lemmatization-python
Document-Term Matrix. (n.d.). Retrieved from https://shorttext.readthedocs.io/en/latest/tutorial_dtm.html
Business & Financial News, U.S & International Breaking News. (n.d.). Retrieved from https://www.Reuters.com/
America’s Finest News Source. (n.d.). Retrieved from https://www.theonion.com/