Sentiment Analysis using LSTM and GloVe Embeddings
By: Ketan Vaidya
Tags: Deep Learning, Sentiment Analysis, Lstm, Word Embeddings, IMDb

Sentiment Analysis has emerged as an important tool for many Organizations and Companies as it gives useful insights into the customer base of a website which in turn helps their decision-making process.

Today I will use the famous IMDb Movie Review Dataset with 50k movie reviews labeled as ‘positive’ or ‘negative’ to implement a simple Sentiment Analysis model using LSTM and GloVe Word Embeddings.

Photo by Markus Winkler on Unsplash

LSTMs

I used LSTM(Long Term Short Memory) for this classification task, it is a type of RNN architecture(Recurrent Neural Network) which are extensively used nowadays for NLP because it handles long sequence dependencies well.

This is a typical LSTM unit that is repeated over the whole length of a sequence.

You can see this video for understanding more about LSTMs

GloVe Word Embeddings

GloVe is an unsupervised learning algorithm to learn vector representation i.e word embedding for various words. GloVe stands for Global Vectors for Word Representations. In this code, I will be using the 50-dimensional GloVe vectors for the task at hand.

With these two things clear, let's start with the code!

1. Importing libraries and loading the dataset

First, we will import all the required libraries and packages

Now let us load the dataset into Google Colab (or any platform you use)

2. Preprocessing the data

This might be among the most important steps because, the data which we have contains a lot of unnecessary things which can be called noise and it only hinders our algorithm’s capability, so we will clean the data a little of this noise.

We will remove stop words from all the movie reviews, now stop words are basically the most commonly occurring words in any language. I have created a list of all the stop words here.

Now let's define some functions to remove the unwanted data

The function remove_stopwords as the name suggests removes stop words from all the movie reviews, another advantage of removing stop words is that the embedding for many of the stop words like ‘they’ll’ or ‘you’re’ isn't available in GloVe Embeddings so this will ease our process.

The second function remove_tags is for removing HTML tags which are common in the text, HTML tags look something like <…>. We apply this function using lambda function.

After this, we get the clean reviews which can be used downstream for training.

3. Tokenizing the words and splitting the data

This can be also included in preprocessing but I’m making a different section

Now we will make our X and Y sets for train and test purposes, first, we will create lists from the data frame columns for movie reviews and their sentiment.

Now let’s convert the sentiment from string to a binary form of 1 and 0, where 1 is for ‘positive’ sentiment and 0 for ‘negative’.

Now we will tokenize all the words in the train set, so the Tokenizer basically indexes all the words in the text and each word gets a unique index. This helps in creating dictionary mapping words to an index which will be very necessary to us later.

words_to_index is the dictionary mapping words to their respective index.

Now we will write a function to read the contents of the GloVe Vector file, which returns us a dictionary that maps the words to their respective word embeddings.

We define maxLen which is the maximum length of one review we will use for our training, I have kept the value to be 150, we will pad this later so all the inputs have the same length

4. Embedding Matrix and setting up the Model

We defined the embedding matrix here, where all the words which are not in the GloVe dictionary being assigned a zero vector.

We also defined the embedding layer using the built-in Keras Embedding layer. The embedding layer maps the words to their embedding vectors from the embedding matrix. We can keep this layer as trainable, which will learn the word embedding itself but as we are using GloVe Embeddings, we won’t keep the layer as trainable.

We will use this layer to define our model architecture.

This function defines our model architecture, first, we use the embedding layer to map the words to their GloVe vectors, and then those vectors are input to the LSTM layers followed by a Dense layer with ‘sigmoid’ activation.

5. Training the data and Testing the model

With our model architecture setup, now we will get our data ready for training.

Basically, we will convert the reviews in the dataset to their index form by using the texts_to_sequences function available with tokenizer. After that, we will pad the sequences so all of them have the same length.

With everything ready, we will move on to training our data!

After training for 15 epochs, I got a reasonable accuracy of 85.29%

After evaluating the model on the test set, I got an accuracy of 85.08% which is pretty good and there is no problem of overfitting.

Now let’s test our model on a random entry in the test set and see if it predicts it correctly.

As you can see the model correctly predicts the sentiment for the randomly selected review from the test set.

In the end, let’s add the sentiment score (basically the probability) and the predicted sentiment to the data frame.

This function returns us a modified data frame which looks like,

So in this article, we learned how to preprocess the given text and implement a binary classifier of movie review text using LSTMs. In the end, we tried a few examples to check the validity of the model.

And we come to the end of this article, this is my first try at writing an article on Medium, hope you like it. Thank You!

You can find my GitHub repo here