Binary Classification of IMDB Movie Reviews | by Rakshit Raj | Towards Data Science

Binary Classification refers to classifying samples in one of two categories.
In this example, we will design a neural network to perform two-class classification, or binary classification, of reviews, from the IMDB movie reviews dataset, to determine whether the reviews are positive or negative. We will use the Python library, Keras.
If you are looking for a more fundamental problem, check out solving the MNIST dataset. The content that follows builds primarily on solving MNIST, the ‘hello world! of deep learning.
towardsdatascience.com
The IMDB dataset is a set of 50,000 highly polarized reviews from the Internet Movie Database. They are split into 25000 reviews each for training and testing. Each set contains an equal number (50%) of positive and negative reviews.
The IMDB dataset comes packaged with Keras. It consists of reviews and their corresponding labels (0 for negative and 1 for positive review). The reviews are a sequence of words. They come preprocessed as a sequence of integers, where each integer stands for a specific word in the dictionary.
The IMDB dataset can be loaded directly from Keras and will usually download about 80 MB on your machine.
Let’s load the prepackaged data from Keras. We will only include 10,000 of the most frequently occurring words.
from keras.datasets import imdb

# Load the data, keeping only 10,000 of the most frequently occuring words
(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words = 10000)

Using TensorFlow backend.

Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz
17465344/17464789 [==============================] - 1s 0us/step

# Since we restricted ourselves to the top 10000 frequent words, no word index should exceed 10000
# we'll verify this below

# Here is a list of maximum indexes in every review --- we search the maximum index in this list of max indexes
print(type([max(sequence) for sequence in train_data]))

# Find the maximum of all max indexes
max([max(sequence) for sequence in train_data])

<class 'list'>

9999
For kicks, let’s decode the first review.
# Let's quickly decode a review

# step 1: load the dictionary mappings from word to integer index
word_index = imdb.get_word_index()

# step 2: reverse word index to map integer indexes to their respective words
reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])

# Step 3: decode the review, mapping integer indices to words
#
# indices are off by 3 because 0, 1, and 2 are reserverd indices for "padding", "Start of sequence" and "unknown"
decoded_review = ' '.join([reverse_word_index.get(i-3, '?') for i in train_data[0]])

decoded_review

Downloading data from https://s3.amazonaws.com/text-datasets/imdb_word_index.json
1646592/1641221 [==============================] - 1s 0us/step

"? this film was just brilliant casting location scenery story direction everyone's really suited the part they played and you could just imagine being there robert ? is an amazing actor and now the same being director ? father came from the same scottish island as myself so i loved the fact there was a real connection with this film the witty remarks throughout the film were great it was just brilliant so much that i bought the film as soon as it was released for ? and would recommend it to everyone to watch and the fly fishing was amazing really cried at the end it was so sad and you know what they say if you cry at a film it must have been good and this definitely was also ? to the two little boy's that played the ? of norman and paul they were just brilliant children are often left out of the ? list i think because the stars that play them all grown up are such a big profile for the whole film but these children are amazing and should be praised for what they have done don't you think the whole story was so lovely because it was true and was someone's life after all that was shared with us all"
We cannot feed a list of integers into our deep neural network. We will need to convert them into tensors.
To prepare our data, we will One-hot Encode our lists and turn them into vectors of 0’s and 1’s. This would blow up all of our sequences into 10,000-dimensional vectors containing 1 at all indices corresponding to integers present in that sequence. This vector will have element 0 at all index, which is not present in the integer sequence.
Simply put, the 10,000-dimensional vector corresponding to each review will have
Every index corresponding to a word
Every index with value 1, is a word that is present in the review and is denoted by its integer counterpart.
Every index containing 0 is a word not present in the review.
We will vectorize our data manually for maximum clarity. This will result in a tensor of shape (25000, 10000).
import numpy as np

def vectorize_sequences(sequences, dimension=10000):
    results = np.zeros((len(sequences), dimension))    # Creates an all zero matrix of shape (len(sequences),10K)
    for i,sequence in enumerate(sequences):
        results[i,sequence] = 1                        # Sets specific indices of results[i] to 1s
    return results

# Vectorize training Data
X_train = vectorize_sequences(train_data)

# Vectorize testing Data
X_test = vectorize_sequences(test_data)

X_train[0]

array([0., 1., 1., ..., 0., 0., 0.])
X_train.shape

(25000, 10000)
y_train = np.asarray(train_labels).astype('float32')
y_test  = np.asarray(test_labels).astype('float32')

Our input data is vectors that need to be mapped to scaler labels (0s and 1s). This is one of the easiest setups, and a simple stack of fully-connected, Dense layers with relu activation perform quite well.
In this network, we will leverage hidden layers. We will define our layers as such.
Dense(16, activation='relu')
The argument being passed to each Dense layer, (16) is the number of hidden units of a layer.
The output from a Dense layer with relu activation is generated after a chain of tensor operations. This chain of operations is implemented as
output = relu(dot(W, input) + b)
Where, W is the Weight matrix and b is the bias (tensor).
Having 16 hidden units means that the matrix W will be of the shape (input_Dimension, 16 ). In this case, where the dimension of the input vector is 10,000, the shape of the Weight matrix will be (10000, 16). If you were to represent this network as a graph, you would see 16 neurons in this hidden layer.
To put it in layman’s terms, there will be 16 balls in this layer.
Each of these balls or hidden units is a dimension in the representation space of the layer. Representation space is the set of all viable representations for the data. Every hidden layer composed of its hidden units aims to learn one specific transformation of the data or one feature/pattern from the data.
DeepAI.org has a very informative write-up on hidden layers.
Hidden layers, simply put, are layers of mathematical functions each designed to produce an output specific to an intended result.
Hidden layers allow for the function of a neural network to be broken down into specific transformations of the data. Each hidden layer function is specialized to produce a defined output.For example, a hidden layer functions that are used to identify human eyes and ears may be used in conjunction by subsequent layers to identify faces in images. While the functions to identify eyes alone are not enough to independently recognize objects, they can function jointly within a neural network.
For our model, we will use
Two intermediate layers with 16 hidden units eachThird layer that will output the scalar sentiment predictionIntermediate layers will use the relu activation function. relu or Rectified linear unit function will zero out the negative values.Sigmoid activation for the final layer or output layer. A sigmoid function “squashes” arbitrary values into the [0,1] range.
Two intermediate layers with 16 hidden units each
Third layer that will output the scalar sentiment prediction
Intermediate layers will use the relu activation function. relu or Rectified linear unit function will zero out the negative values.
Sigmoid activation for the final layer or output layer. A sigmoid function “squashes” arbitrary values into the [0,1] range.
There are formal principles that guide our approach in selecting the architectural attributes of a model. These are not covered in this case study.
In this step, we will choose an optimizer, a loss function, and metrics to observe. We will go forward with
binary_crossentropy loss function, commonly used for Binary Classification
rmsprop optimizer and
accuracy as a measure of performance
We can pass our choices for optimizer, loss function and metrics as strings to the compile function because rmsprop, binary_crossentropy and accuracy come packaged with Keras.
model.complie(optimizer='rmsprop',              loss = 'binary_crossentropy',              metrics = ['accuracy'])
One could use a customized loss function or optimizer by passing a custom class instance as an argument to the loss, optimizer or mertics fields.
In this example, we will implement our default choices, but we will do so by passing class instances. This is precisely how we would do it if we had customized parameters.
We will set aside a part of our training data for validation of the accuracy of the model as it trains. A validation set enables us to monitor the progress of our model on previously unseen data as it goes through epochs during training.
Validation steps help us fine-tune the training parameters of the model.fit function to avoid Overfitting and underfitting of data.
Initially, we will train our models for 20 epochs in mini-batches of 512 samples. We will also pass our validation set to the fit method.
Calling the fit method returns a History object. This object contains a member history which stores all data about the training process, including the values of observable or monitored quantities as the epochs proceed. We will save this object to determine the fine-tuning better to apply to the training step.
At the end of the training, we have attained a training accuracy of 99.85% and validation accuracy of 86.57%
Now that we have trained our network, we will observe its performance metrics stored in the History object.
Calling the fit method returns a History object. This object has an attribute history which is a dictionary containing four entries: one per monitored metric.
history_dict contains values of
Training loss
Training Accuracy
Validation Loss
Validation Accuracy
at the end of each epoch.
Let’s use Matplotlib to plot Training and validation losses and Training and Validation Accuracy side by side.
We observe that minimum validation loss and maximum validation Accuracy is achieved at around 3–5 epochs. After that, we observe two trends:
increase in validation loss and a decrease in training loss
decrease in validation accuracy and an increase in training accuracy
This implies that the model is getting better at classifying the sentiment of the training data, but making consistently worse predictions when it encounters new, previously unseen data. This is the hallmark of Overfitting. After the 5th epoch, the model begins to fit too closely to the training data.
To address Overfitting, we will reduce the number of epochs to somewhere between 3 and 5. These results may vary depending on your machine and due to the very nature of the random assignment of weights that may vary from model to model.
In our case, we will stop training after 3 epochs.
We retrain our neural network based on our findings from studying the history of loss and accuracy variation. This time we run it for 3 epochs so as to avoid Overfitting on training data.
In the end, we achieve a training accuracy of 99% and a validation accuracy of 86%. This is pretty good, considering we are using a very naive approach. A higher degree of accuracy can be achieved by using a better training algorithm.
We will use our trained model to make predictions for the test data. The output is an array of floating integers that denote the probability of a review being positive. As you can see, in some cases, the network is absolutely sure the review is positive. In other cases — not so much!
You could try to find some error metric for the number of sentiments that were wrongly classified by using a metric like mean squared error as I did here. But it would be stupid to do so! The analysis of the result is not something we will cover here. However, I will shed some light on why usingmse is futile in this case.
The result from our model is the measure of how much the model perceives a review to be positive. Rather than telling us the absolute class of the sample, the model tells us by how much it perceives the sentiment to be skewed on one side or the other. MSE is too simple a metric and fails to capture the complexity of the solution.
I did not visualize this neural net. I would, but it is a time-consuming process. I did visualize the neural network I used in solving the MNIST problem. If you want you could check out this GitHub Project for visualizing ANNs
github.com
And thus, we have successfully classified reviews on IMDB. I guess this calls for rewatching The Matrix or whatever IMDB suggests next!
I recommend that you work along with the article. You can solve most binary classification problems using a similar strategy. If you did solve it, try fiddling the design and parameters of the network and its layers. This will help you better understand the integrity of the model architecture you chose.
I discuss a single topic in additional detail in each of my articles. In this one, we delved a little bit into hidden layers. An exhaustive explanation of any particular topic is never in the scope of my article; however, you will find ample quick asides.
I assume that the reader has a working understanding of technicalities like an optimizer, categorical encoding, loss function, and metrics. You can find my practice notes on these concepts here.
For more, please check out the book Deep Learning with Python by Francois Chollet.
Feel free to check out this article’s implementation and more of my work on GitHub.
Thanks for reading!