Build and Compare 3 Models — NLP Prediction | by Rohan Gupta | Towards Data Science

This project was created in an attempt to learn and understand how various classification algorithms work within a Natural Language Processing Model. Natural Language Processing, which I will now refer to as NLP, is a branch of machine learning that focuses on enabling computers to interpret and process human languages in both speech and text forms.
In this pipeline, I go through the following steps:
Import required packages and librariesImport the datasetProcess text in the dataset before it can be analyzed by the computerCreate a Bag of Words modelSplitting the dataset into Train & Test setsNaive Bayes AlgorithmDecision Tree AlgorithmRandom Forest AlgorithmComparing Accuracy, Precision, Recall, and F1 Scores
Import required packages and libraries
Import the dataset
Process text in the dataset before it can be analyzed by the computer
Create a Bag of Words model
Splitting the dataset into Train & Test sets
Naive Bayes Algorithm
Decision Tree Algorithm
Random Forest Algorithm
Comparing Accuracy, Precision, Recall, and F1 Scores
For this project, I will be using a dataset sourced from Kaggle, which contains 1000 reviews for a pizzeria by different users. |Link to the dataset|
Humans can read a review and tell whether it is positive or negative. What if we could create a model to classify them as positive or negative? What’s the best way to do this?
First, let’s talk about the process. We start off by pre-processing the data, removing unnecessary words that don’t help our prediction. Then, we take important words in their stemmed forms (e.g lov is the stem for loved, loving, or lovely). We then train the machine to learn which reviews are positive based on their word stems. After that we test the data using similar information, to see how accurately our machine can predict whether a review is positive or negative (1 or 0).
Here, we import all the libraries required for this model to work. Before you begin, make sure you have all the dependencies installed. We will be working mainly with pandas, numpy, re, nltk, matplotlib, and sci-kit learn.
Make sure to pip installon the command line all the libraries mentioned right above.
import numpy as npimport matplotlib.pyplot as pltimport pandas as pdimport reimport nltkfrom nltk.corpus import stopwordsfrom nltk.stem.porter import PorterStemmerfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.model_selection import train_test_splitfrom sklearn.naive_bayes import GaussianNBfrom sklearn.metrics import confusion_matrixfrom sklearn.tree import DecisionTreeClassifierfrom sklearn.ensemble import RandomForestClassifier
dataset = pd.read_csv(‘Restaurant_Reviews.tsv’, delimiter = ‘\t’, quoting = 3)
In the code above, I am using a .tsv file, rather than a .csv file. The difference is easier to tell when we break down the acronyms. A .tsv (tab-separated values) is separated by spaces in the text file, whereas a .csv (comma separated values) uses commas to separate the different values.
pd.read_csv can be used for both, but to specify a tsv file, I added the ‘\t’ in the delimiter to tell the machine that values are separated by tabs and not commas. Quoting has been set to 3 since our reviews contain some double quotes, and the machine would not be able to interpret them as regular text if we didn’t add this line.Below we can see the first 10 reviews of our dataset and the outcome: positive(1) or negative(0)
dataset.head(10)
nltk.download(‘stopwords’)corpus = []for i in range(0, 1000):review = re.sub(‘[^a-zA-Z]’, ‘ ‘, dataset[‘Review’][i])review = review.lower()review = review.split()ps = PorterStemmer()review = [ps.stem(word) for word in review if not word in set(stopwords.words(‘english’))]review = ‘ ‘.join(review)corpus.append(review)
The code above shows the most important step of this model. I started by importing Regex as re. This class allows us to match and search for string objects. I used the sub-function of re to allow the machine to include the elements of the data we require, which are letters A-Z in upper and lower case.
I also imported nltk which stands for Natural Language Toolkit. From nltk, I imported 2 classes: the stopwords class, and the PorterStemmer class.
stopwords allow us to remove words that would not help our model (such as “the”, “an”, “this” etc). The stopwords class already contains these words, so I didn’t have to manually enter them. I used a for loop to specify to the machine that we need to take out the word if it doesn’t exist in the stopwords class.
PorterStemmer allows us to take the stem of a word and classify it as a common predictor for similar words. For example, in the first review, “Lov” is the stem for words such as “loving” or “loved”, which would both essentially translate to a positive review, thus allowing our model to have fewer words.
Finally, to apply the preprocessing steps to all the 1000 reviews in our dataset, I added another for loop before the text-processing steps.
cv = CountVectorizer(max_features = 1500)X = cv.fit_transform(corpus).toarray()y = dataset.iloc[:, 1].values
The Bag of Words model allows us to extract features from our textual data and in this example, we take out all the words from each of the observations and pool them together in a sort of “bag” to reduce redundancy by not counting duplicates. I did this by importing the CountVectorizer class from sklearn.
Each word forms its own column in a way, and since there are so many words, we could have a huge number of columns. However, I specified the maximum number of columns using the max_features parameter of the CountVectorizer class.
Then all I had to do was fit the column of words to the X (input) variable and specified y (output) variable as our second column in the dataset, which gives you a 1 or 0, depending on whether the review was positive or negative respectively.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 0)
I then had to split the dataset into a training set and a test set, and used a test size of 0.2, so that we have 800 values to train our dataset and 200 values to test it with.
Code below shows us what Train & Test sets look like
X_train[0, 0:10] #First 10 rows of the first column of X_train.Output: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64)X_test[0, 0:10] #First 10 rows of the first column of X_test.Output: array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int64)y_train[:10] #First 10 values of the first column of y_train.Output: array([1, 1, 1, 0, 1, 0, 1, 0, 0, 0])y_test[:10] #First 10 values of the first column of y_test.Output: array([0, 0, 0, 0, 0, 0, 1, 0, 0, 1])
Now that I’m done with all the pre-processing steps, I’m going to start applying some classification algorithms to model to help it predict reviews.
The first model I use is the Naive Bayes Model. In machine learning, naive Bayes classifiers are a family of simple “probabilistic classifiers” based on applying Bayes’ theorem with strong (naive) independence assumptions between the features.
All naive Bayes classifiers assume that the value of a particular feature is independent of the value of any other feature, given the class variable. In our model, the naive Bayes algorithm looks at particular keywords of a review to describe whether it is positive or negative, depending on the output set.
In the code below, I imported the GaussianNB class, which assumes that our data is normally distributed (with a Gaussian Bell Curve).
classifier = GaussianNB()classifier.fit(X_train, y_train)Output: GaussianNB(priors=None, var_smoothing=1e-09)
I created y_pred_NB, which is where our model’s predictions would be stored. The output below shows us what the prediction matrix looks like. It’s a bunch of 1s and 0s, like our y_train dataset. These are the predictions made by our model. y_pred_NB can be compared to y_test and determine how accurate our model is.
y_pred_NB = classifier.predict(X_test)y_pred_NBOutput: array([1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 0, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1])
The confusion matrix for Naive Bayes shows us our True Negatives on the top left, False Negatives on the top right, True Positives on the bottom right, and False Positives on the bottom left.
Below, I’ve coded the numbers for each with acronyms using T for True, F for False, N for negative, P for positive.
cm_NB = confusion_matrix(y_test, y_pred_NB) cm_NBOutput: array([[55, 42],[12, 91]])TP_NB = 91 #True Positives (Naive Bayes)TN_NB = 55 #True Negatives (Naive Bayes)FP_NB = 12 #False Positives (Naive Bayes)FN_NB = 42 #False Negatives (Naive Bayes)
Below, I will use the True/False Positives and Negatives to calculate the Accuracy, Precision, Recall, and F1 scores.
Accuracy is as the name goes. It measures the accuracy by adding True predictions and dividing them by the total number of predictions.
Accuracy_NB = (TP_NB + TN_NB) / (TP_NB + TN_NB + FP_NB + FN_NB) Accuracy_NB  Output: 0.73
Precision refers to the closeness of two or more measurements to each other. It’s calculated by dividing True Positives from Total Positives
Precision_NB = TP_NB / (TP_NB + FP_NB)Precision_NBOutput: 0.883495145631068
Recall is the ratio of correctly predicted positive observations to all observations in the actual class. Divide True positive by the sum of True Positives and False Negatives.
Recall_NB = TP_NB / (TP_NB + FN_NB)Recall_NBOutput: 0.6842105263157895
The F1 Score is the weighted average of Precision and Recall. F1 Score might be a better measure to use if we need to seek a balance between Precision and Recall AND there is an uneven class distribution. It is calculated by multiplying the Precision and Recall, dividing the result by the sum of Precision and Recall, and multiplying the final result by 2.
F1_Score_NB = 2 * Precision_NB * Recall_NB / (Precision_NB + Recall_NB) F1_Score_NBOutput: 0.7711864406779663
The next algorithm I used is the decision tree. Decision trees allow you to develop classification systems that predict or classify future observations based on a set of decision rules.
classifier = DecisionTreeClassifier(criterion = ‘entropy’, random_state = 0)classifier.fit(X_train, y_train)Output: DecisionTreeClassifier(class_weight=None, criterion=’entropy’,max_depth=None,max_features=None,max_leaf_nodes=None,min_impurity_decrease=0.0,min_impurity_split=None,min_samples_leaf=1,min_samples_split=2,min_weight_fraction_leaf=0.0, presort=False, random_state=0,splitter=’best’)
In [59]: y_pred_DT = classifier.predict(X_test)y_pred_DTOutput: array([0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 1, 1, 1, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 0, 0,1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1, 0, 0, 0, 0, 1, 0, 0,0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1,0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 1,0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 0, 1,0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1,0, 0])
cm_DT = confusion_matrix(y_test, y_pred_DT) cm_DTOutput: array([[74, 23],[35, 68]])TP_DT = 68 #True Positives (Decision Tree)TN_DT = 74 #True Negatives (Decision Tree)FP_DT = 35 #False Positives (Decision Tree)FN_DT = 23 #False Negatives (Decision Tree)
Accuracy_DT = (TP_DT + TN_DT) / (TP_DT + TN_DT + FP_DT + FN_DT)Accuracy_DTOutput: 0.71
Precision_DT = TP_DT / (TP_DT + FP_DT)Precision_DTOutput: 0.6601941747572816
Recall_DT = TP_DT / (TP_DT + FN_DT)Recall_DTOutput: 0.7472527472527473
F1_Score_DT = 2 * Precision_DT * Recall_DT / (Precision_DT + Recall_DT)F1_Score_DTOutput: 0.7010309278350515
Finally, I used the Random Forest algorithm, which is just a combination of a number of decision trees. In my example, I chose to use 300 trees, but I could change that number depending on the kind of accuracy I want from the model.
classifier = RandomForestClassifier(n_estimators = 300, criterion = ‘entropy’, random_state = 0)classifier.fit(X_train, y_train)Output: RandomForestClassifier(bootstrap=True, class_weight=None, criterion=’entropy’,max_depth=None,max_features=’auto’,max_leaf_nodes=None,min_impurity_decrease=0.0,min_impurity_split=None,min_samples_leaf=1, min_samples_split=2,min_weight_fraction_leaf=0.0, n_estimators=300, n_jobs=None,oob_score=False, random_state=0, verbose=0, warm_start=False)
y_pred_RF = classifier.predict(X_test)y_pred_RFOutput: array([0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 1, 1, 1, 1, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 1, 0, 1, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 0])
cm_RF = confusion_matrix(y_test, y_pred_RF) cm_RFOutput: array([[87, 10],[47, 56]])# Calculating True/False Positives/NegativesTP_RF = 56 #True Positives (Random Forest)TN_RF = 87 #True Negatives (Random Forest)FP_RF = 47 #False Positives (Random Forest)FN_RF = 10 #False Negatives (Random Forest)
Accuracy_RF = (TP_RF + TN_RF) / (TP_RF + TN_RF + FP_RF + FN_RF)Accuracy_RFOutput: 0.715
Precision_RF = TP_RF / (TP_RF + FP_RF)Precision_RFOutput: 0.5436893203883495
Recall_RF = TP_RF / (TP_RF + FN_RF)Recall_RFOutput: 0.8484848484848485
F1_Score_RF = 2 * Precision_RF * Recall_RF / (Precision_RF + Recall_RF)F1_Score_RFOutput: 0.6627218934911243
In this last section, I will compare the accuracy, precision, recall, and F1 Scores of each algorithm I used. I will plot them on bar charts to give us a graphical representation of how the different models compare to each other.
Accuracy = [Accuracy_RF, Accuracy_DT, Accuracy_NB]Methods = [‘Random_Forest’, ‘Decision_Trees’, ‘Naive_Bayes’]Accuracy_pos = np.arange(len(Methods))plt.bar(Accuracy_pos, Accuracy)plt.xticks(Accuracy_pos, Methods)plt.title(‘comparing the accuracy of each model’)plt.show()
As we can see in the barplot above, Naive Bayes has the highest accuracy of all of the algorithms with 73% correct predictions. Decision Trees and Random Forest algorithms are also close with 71% and 71.5% accuracy respectively.
Precision = [Precision_RF, Precision_DT, Precision_NB]Precision_pos = np.arange(len(Methods))plt.bar(Precision_pos, Precision)plt.xticks(Precision_pos, Methods)plt.title(‘comparing the precision of each model’)plt.show()
Naive Bayes is the most precise model, with a precision of 88.35%, whereas Decision Trees have a precision of 66%. Random Forests have the lowest precision rate of about 54.4%.
Recall = [Recall_RF, Recall_DT, Recall_NB]Recall_pos = np.arange(len(Methods))plt.bar(Recall_pos, Recall)plt.xticks(Recall_pos, Methods)plt.title(‘comparing the recall of each model’)plt.show()
Random Forests have the highest recall of about 84.8%, whereas Decision Trees have a recall of 74.7%. Naive Bayes had the lowest recall with 68.4%
F1_Score = [F1_Score_RF, F1_Score_DT, F1_Score_NB]F1_Score_pos = np.arange(len(Methods))plt.bar(F1_Score_pos, F1_Score)plt.xticks(F1_Score_pos, Methods)plt.title(‘comparing the F1 Score of each model’)plt.show()
Naive Bayes has the highest F1 score of 77.1%
Decision Trees have an F1 score of 70.1%.
Random Forest has the lowest F1 Score of 66.2%
On average our models are about 71.8% accurate. While this may mean that the machine cannot predict every review with accuracy, it also shows us evidence that our models are not overfitting the data. Overfitting is a modeling error that occurs when a function is too closely fit to a limited set of data points. Overfitting the model generally occurs when using an overly complex model to explain idiosyncrasies in the data. However, since our model is being trained to think like a human brain, it is fair to assume that even humans may not be able to predict 100% of the time whether a review is positive or negative. It can really depend on the data and how it has been processed.
So which was the best algorithm for this model?Out of the 3 used in this project, the most accurate and precise was the Naive Bayes algorithm.
However, the recall for our Random Forest algorithm was the highest. This means that the Random Forest algorithm actually calculates how many of the Actual Positives our model capture through labeling it as Positive (True Positive). This would be a good metric we use to select our best model when there is a high cost associated with False Negative.
The truth lies within our F1 scores, which may actually be the best predictor of which model is the best amongst the chosen three. The Naive Bayes algorithm has the highest F1 score, which means that it defines a relationship between Recall and Precision of a particular model. F1 Scores might be a better measure to use if we need to seek a balance between Precision and Recall AND if there is an uneven class distribution (a large number of Actual Negatives).
Thanks for the read! Hope you learned something useful.
Follow Rohan Gupta for more content related Data Science and Machine Learning