Naive Bayes and LSTM Based Classifier Models
By: Ruthu S Sanketh
Tags: Naturallanguageprocessing, Text Processing, Lstm, Naive Bayes, Classification

Naive Bayes and LSTM Based Classifier Models

If you’re not familiar with the NLTK library and data pre-processing, take a look at this article. If you’re interested in language models and how to build them, read this article. If you’re familiar with NLP and language models, continue reading!

What are Language Based Classifier Models?

A statistical language model is a probability distribution over sequences of words which can be used to predict the next word for text generation and many other applications. Classifiers such as Naive Bayes make use of a language model to assign class labels to some instances, based on a set of features which can be numerically represented using statistical techniques. For example, given a set of movie reviews, we can train a model to predict whether the review has a positive or a negative sentiment.

Text generation can now be done by AI — Image by MILKOVÍ on Unsplash

Many machine learning techniques have been applied to automatic text classification apart from the Naive Bayes, such as K-Nearest Neighbors and Support Vector Machines(SVM). The NB classifier is widely used in text classification for its simplicity and efficiency. An LSTM or Long-Short-Term-Memory classifier is an artificial recurrent neural network which has both feedforward and feedback connections, and is usually used for classifying and making predictions on time-series data. In this article, we will be using the NB classifier on an IMDB movie review dataset with contains reviews and their sentiments, and then comparing its accuracy and other metrics with those of an LSTM classifier.

Naive Bayes Classifier

Naive Bayes is a conditional probability model, based on Bayes’ theorem, which states that the conditional probability is given by -

for each of K possible outcomes. Using the chain rule for repeated applications of the definition of conditional probability, this becomes -

Under the naive assumption that all the features x are mutually independent, the probability now becomes -

This formula is the basis behind the Multinomial Naive Bayes classifier which we will be building, which deals with the occurrences of a word in a single document.

Initial Steps

First we import the required libraries and tools.

import pandas as pd

import numpy as np

import nltk, keras, string, re, html, math



from nltk.tokenize import word_tokenize, sent_tokenize

from nltk.corpus import stopwords, wordnet

from nltk.stem import WordNetLemmatizer

from sklearn.preprocessing import LabelEncoder

from sklearn.feature_extraction.text import CountVectorizer

from collections import Counter, defaultdict

from sklearn.model_selection import train_test_split

from keras.preprocessing.text import Tokenizer

from keras.preprocessing.sequence import pad_sequences

from sklearn.metrics import accuracy_score, classification_report

We then import the IMDB movie reviews dataset. The dataset can be found here. We load the data using pandas, as a dataframe. We then look at some important parameters, such as the number of null values in the dataset. If a dataset has many null values, methods such as median or mode value are used to fill up these cells for better results. We also convert the ‘sentiment’ column to lowercase for ease of encoding later on.

#Loads the IMDB dataset. We load it using pandas as dataframe

data = pd.read_csv('/Users/ruthu/Desktop/IMDB Dataset.csv')

print("Data shape - ", data.shape, "

") #prints the number of rows and columns



for col in data.columns:

print("The number of null values - ", col, data[col].isnull().sum())

#prints the number of null values in each column



data["review"]= data["review"].str.lower()

data["sentiment"]= data["sentiment"].str.lower() #converts every value in the column to lowercase

data.head()

A preview of the data — Image by author

Looking at a preview of the data, we see that there are some HTML tags, newlines, and other special characters that need to be removed from the ‘review’ column. We also remove stopwords and perform lemmatization of the words in the column using regular expressions.

def cleaning(data):

clean = re.sub('<.*?>', ' ', str(data))

#removes HTML tags

clean = re.sub('\'.*?\s',' ', clean)

#removes all hanging letters afer apostrophes (s in it's)

clean = re.sub(r'http\S+',' ', clean)

#removes URLs

clean = re.sub('\W+',' ', clean)

#replacing the non alphanumeric characters

return html.unescape(clean)

data['cleaned'] = data['review'].apply(cleaning)





def tokenizing(data):

review = data['cleaned']

#tokenizing is done

tokens = nltk.word_tokenize(review)

return tokens

data['tokens'] = data.apply(tokenizing, axis=1)





stop_words = set(stopwords.words('english'))

def remove_stops(data):

my_list = data['tokens']

meaningful_words = [w for w in my_list if not w in stop_words] #stopwords are removed from the tokenized data

return (meaningful_words)

data['tokens'] = data.apply(remove_stops, axis=1)





lemmatizer = WordNetLemmatizer()

def lemmatizing(data):

my_list = data['tokens']

lemmatized_list = [lemmatizer.lemmatize(word) for word in my_list]

#lemmatizing is performed. It's more efficient than stemming.

return (lemmatized_list)

data['tokens'] = data.apply(lemmatizing, axis=1)



def rejoin_words(data):

my_list = data['tokens']

joined_words = ( " ".join(my_list))

#rejoins all stemmed words

return joined_words

data['cleaned'] = data.apply(rejoin_words, axis=1)



data.head()

Preview of preprocessed data— Image by author

We now have a cleaned set of reviews as well as a list of lemmatized and preprocessed tokens for each review. Now let us print some statistics of the data, such as the number of unique tokens, etc. which can be passed as a parameter while initializing our model.

# Prints statistics of Data like avg length of sentence , proportion of data w.r.t class labels

def sents(data):

clean = re.sub('<.*?>', ' ', str(data))

#removes HTML tags

clean = re.sub('\'.*?\s',' ', clean)

#removes all hanging letters afer apostrophes (s in it's)

clean = re.sub(r'http\S+',' ', clean)

#removes URLs

clean = re.sub('[^a-zA-Z0-9\.]+',' ', clean)

#removes all non-alphanumeric characters except periods.

tokens = nltk.sent_tokenize(clean)

#sentence tokenizing is done

return tokens

sents = data['review'].apply(sents)



length_s = 0

for i in range(data.shape[0]):

length_s+= len(sents[i])

print("The number of sentences is - ", length_s)

#prints the number of sentences



length_t = 0

for i in range(data.shape[0]):

length_t+= len(data['tokens'][i])

print("

The number of tokens is - ", length_t)

#prints the number of tokens



average_tokens = round(length_t/length_s)

print("

The average number of tokens per sentence is - ", average_tokens)

#prints the average number of tokens per sentence



positive = negative = 0

for i in range(data.shape[0]):

if (data['sentiment'][i]=='positive'):

positive += 1

#finds the proprtion of positive and negative sentiments

else:

negative += 1



print("

The number of positive examples are - ", positive)

print("

The number of negative examples are - ", negative)

print("

The proportion of positive to negative sentiments are -", positive/negative)

Statistics of the data — Image by author

Since the ratio of positive to negative samples is 1, we don’t need to do any data normalization. Let us create the NB classifier now. We encode the ‘sentiments’ column by assigning the integer 0 to negative sentiments and 1 to positive sentiments. This makes training as well as data storage easier.

# gets reviews column from df

reviews = data['cleaned'].values



# gets labels column from df

labels = data['sentiment'].values # Uses label encoder to encode labels. Convert to 0/1

encoder = LabelEncoder()

encoded_labels = encoder.fit_transform(labels)

data['encoded']= encoded_labels

print(data['encoded'].head())



# prints(enc.classes_)

encoder_mapping = dict(zip(encoder.classes_, encoder.transform(encoder.classes_)))

print("

The encoded classes are - ", encoder_mapping)



labels = data['encoded']

Encoded sentiments — Image by author

Now that we have the columns ready for the model, we can split the data into train and test sets. We use a 80-20% train-test split and also the stratify parameter so that the ratio of positive to negative sentiments are the same in both the train and test sets.

# Splits the data into train and test (80% - 20%).

# Uses stratify in train_test_split so that both train and test have similar ratio of positive and negative samples. train_sentences, test_sentences, train_labels, test_labels = train_test_split(reviews, labels, test_size=0.2, random_state=42, stratify=labels)

We now have train and test sentences as well as the corresponding train and test labels. There are two approaches possible for building vocabulary for the Naive Bayes classifier -

We can take the whole data (train + test) to build the vocab. In this way while testing no word will be out of vocabulary. We can take only the train data to build the vocab. In this case, some words from the test set may not be in vocab and hence we will need to perform smoothing so that none of the probability terms are zero.

We use the 2nd approach since the 1st one is data intensive with nearly the same results. Also, since building vocab by taking all words in the train set is also memory intensive, we build the vocab by choosing the 3000 most frequent words in the training corpus.

Smoothing to remove 0 probability — Image by author

# Uses Count vectorizer to get frequency of the words

vectorizer = CountVectorizer(max_features = 3000)



sents_encoded = vectorizer.fit_transform(train_sentences) #encodes all training sentences

counts = sents_encoded.sum(axis=0).A1

vocab = list(vectorizer.get_feature_names())

Building the Model

Now that we have the vocabulary, let us build the MNB classifier. NLTK also has an inbuilt Multinomial Bayes Classifier model which we can use, but in this article we will be writing code for the classifier to get a better idea of its working.

# Uses laplace smoothing for words in test set not present in vocab of train set.

class MultinomialNaiveBayes:



def __init__(self, classes, tokenizer):

#self.tokenizer = tokenizer

self.classes = classes



def group_by_class(self, X, y):

data = dict()

for c in self.classes:

#grouping by positive and negative sentiments

data[c] = X[np.where(y == c)]

return data



def fit(self, X, y):

self.n_class_items = {}

self.log_class_priors = {}

self.word_counts = {}

self.vocab = vocab

#using the pre-made vocabulary of 3000 most frequent training words



n = len(X)



grouped_data = self.group_by_class(X, y)



for c, data in grouped_data.items():

self.n_class_items[c] = len(data)

self.log_class_priors[c]=math.log(self.n_class_items[c]/n)

#taking log for easier calculation

self.word_counts[c] = defaultdict(lambda: 0)



for text in data:

counts = Counter(nltk.word_tokenize(text))

for word, count in counts.items():

self.word_counts[c][word] += count



return self def laplace_smoothing(self, word, text_class): #smoothing

num = self.word_counts[text_class][word] + 1

denom = self.n_class_items[text_class] + len(self.vocab)

return math.log(num / denom)



def predict(self, X):

result = []

for text in X:



class_scores = {c: self.log_class_priors[c] for c in self.classes}



words = set(nltk.word_tokenize(text))

for word in words:

if word not in self.vocab: continue



for c in self.classes:



log_w_given_c = self.laplace_smoothing(word, c)

class_scores[c] += log_w_given_c



result.append(max(class_scores, key=class_scores.get))



return result

Now that we have built the MNB classifier, let us pass the train sentences and labels to train the model. After training the model, we obtain its accuracy and other performance metrics on the test set.

MNB = MultinomialNaiveBayes(

classes=np.unique(labels),

tokenizer=Tokenizer()

).fit(train_sentences, train_labels)



# Tests the model on test set and reports the Accuracy

predicted_labels = MNB.predict(test_sentences)

print("The accuracy of the MNB classifier is ", accuracy_score(test_labels, predicted_labels)) print("

The classification report with metrics -

", classification_report(test_labels, predicted_labels))

Performance metrics of the MNB model — Image by author

We see that the accuracy of the MNB model on the test set is around 85%.

LSTM Model

We use the same train and test splits to build and study this model. Keras has an inbuilt LSTM model which we will be using after setting the parameters to our necessity.

Initial Steps

We define some hyperparameters of the model, and perform padding on the train and test sentences.

# Hyperparameters of the model

oov_tok = '<OOK>'

embedding_dim = 100

max_length = 150

padding_type='post'

trunc_type='post'



# tokenizes sentences

tokenizer = Tokenizer()

tokenizer.fit_on_texts(train_sentences)



# vocabulary size

word_index = tokenizer.word_index

vocab_size = len(tokenizer.word_index) + 1



# converts train dataset to sequence and pads sequences

train_sequences = tokenizer.texts_to_sequences(train_sentences)

train_padded = pad_sequences(train_sequences, padding='post', maxlen=max_length)



# converts Test dataset to sequence and pads sequences

test_sequences = tokenizer.texts_to_sequences(test_sentences)

test_padded = pad_sequences(test_sequences, padding='post', maxlen=max_length)

Now we initialize the model, using the Adam optimizer and binary cross entropy loss function. Feel free to try out other optimizers such as SGD. We use word embeddings, which is a technique where words are encoded as real-valued vectors in a high dimensional space, such that the similarity between words in terms of meaning translates to closeness in the vector space, using an Embedded Layer provided by Keras.

# model initialization

model = keras.Sequential([

keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),

keras.layers.Bidirectional(keras.layers.LSTM(64)),

keras.layers.Dense(24, activation='relu'),

keras.layers.Dense(1, activation='sigmoid')

])



# compiles model

model.compile(loss='binary_crossentropy',

optimizer='adam',

metrics=['accuracy'])



# model summary

model.summary()

LSTM model summary — Image by author

Training the Model

Now that we have defined the model and set up the hyperparameters, we train the model. This might take time if you’re running the code on your local machine, but Google Colab with GPU runtime is fast, so we use that.

#training the model

num_epochs = 5

history = model.fit(train_padded, train_labels,

epochs=num_epochs, verbose=1,

validation_split=0.1)

Training the model — Image by author

Now that we have trained the model, we can find its accuracy on the test set. First we find the probabilities, and convert these to predictions using a 0.5 threshold. Then we get a classification report.

# Gets probabilities

prediction = model.predict(test_padded)

print("The probabilities are - ", prediction, sep='

')



# Gets labels based on probability 1 if p>= 0.5 else 0

for each in prediction:

if each[0] >=0.5:

each[0] = 1

else:

each[0] = 0

prediction = prediction.astype('int32')

print("

The labels are - ", prediction, sep='

')



# Calculates accuracy on Test data

print("

The accuracy of the model is ", accuracy_score(test_labels, prediction))

print("

The accuracy and other metrics are

", classification_report(test_labels, prediction, labels=[0, 1]),sep='

')

Performance metrics of the LSTM model — Image by author

We see that the accuracy of the LSTM model on the test set is around 87%.

Predictions on Random Samples

We can also use the model to get predictions for random samples and look at the output.

# reviews on which we need to predict

sentence = ["The movie was very touching and heart whelming",

"I have never seen a terrible movie like this",

"the movie plot is terrible but it had good acting"]



# converts to a sequence

test_sequences = tokenizer.texts_to_sequences(sentence)



# pads the sequence

test_padded = pad_sequences(test_sequences, padding='post', maxlen=max_length)



# Gets probabilities

prediction = model.predict(test_padded)

print("The probabilities are - ", prediction, sep='

')



# Gets labels based on probability 1 if p>= 0.5 else 0

for each in prediction:

if each[0] >=0.5:

each[0] = 1

else:

each[0] = 0

prediction = prediction.astype('int32')

print("

The labels are - ", prediction, sep='

')

LSTM random sample predictions — Image by author

We see that the LSTM model predicts the sentiments with a decent accuracy.

Conclusion

The LSTM model with an accuracy of 87% is slightly better than the MNB model with an accuracy of 85%. Both the models can be used to predict the sentiments of random samples. Their accuracy can be improved by increasing the number of training epochs, or varying the hyperparameters. The entire code used in this article as well as the dataset can be found here.

Further Reading