Sentiment Classifier Using NLP in Python
By: Shivangi Sareen
Tags: Supervised Learning, Classification, NLP, Sentiment Analysis, Python3

The aim is to train a supervised Stochastic Gradient Descent classifier on a training set containing reviews of movies from IMDB, with labels, 0 for a negative review and 1 for a positive review. We’ll use the trained model to predict the polarity of reviews in our testing set. We’ll be training the model using a unigram, bigram, unigram with tf-idf and bigram with tf-idf approach. Read onward to see what it’s all about!

The theory

The dataset

The dataset consists of 50,000 reviews from IMDB. Number of positive and negative reviews are equal, at 25k. Negative reviews have scores ≤ 4 out of 10 while a positive review ≥ 7 out of 10; neutral reviews are not included. The overall distribution of labels is balanced (25k pos and 25k neg). The dataset can be found here.

There is also an additional 50,000 unlabelled documents for unsupervised learning, however, we will be focussing on supervised learning techniques here.

The files

There’s a train/ and test/ folder, each with pos/ and neg/ directories containing text files of reviews named named as [[id]_[rating].txt] where [id] is a unique id and [rating] is the rating corresponding to the review on a 1–10 scale.

Focussing only on supervised learning, the unsup/ directories can be ignored.

The tools and technique

We’ll be using Python’s sci-kit learn library to train a Stochastic Gradient Descent classifier.

In both SGD (Stochastic Gradient Descent) and GD (Gradient Descent), we update parameters iteratively to minimise the loss function. In GD, we run through the whole training data per epoch to update one set of parameters in a given iteration. On the other hand, SGD, as the name suggests (stochastic), randomly chooses only a single training example per epoch to update the parameter in a particular iteration. Mini-batch Gradient Descent lies in between GD and SGD where a small number of training samples (subset) are used per epoch.

Here is a very helpful video and article on SGD.

The goal

The aim is to train our SGD classifier on 25k training samples and then use that to predict the polarity (0 for negative review and 1 for positive review) of the test data.

The process

We’ll do some data preprocessing first — to combine the pos/ and neg/ data to create a training and testing file. Next we’ll do some data cleaning to get rid of any noise in the data.

SGD has to be fitted with two arrays: an array X of shape (n_samples, n_features) holding the training samples, and an array y of shape (n_samples,) holding the target values (class labels) for the training samples.

To convert our text data into such numerical form, we use Bag-of-Words models.

Bag-of-Words (BoW) representation

“The bag-of-words model is commonly used in methods of document classification where the (frequency of) occurrence of each word is used as a feature for training a classifier.” [source]

The most common type of feature extracted from the Bag-of-words model is term frequency, i.e., the number of times a term appears in the text. In our case of sentiment analysis, the more times a word like ‘dislike’, ‘hate’, ‘suck’, etc, the more likely its polarity be 0 (negative). And so this is how we’ll be training our model.

In our case, each review is considered a document Each document is split into words or tokens We then assign a weight to each token as its frequency of appearance in the document Finally, we create a document-term matrix representing exactly the above, of shape (n_samples, n_features) with each samples representing a documents and each features representing a the tokenised word

N-gram representation

In BoW models, only the count of the words matter. As an alternative, we tend to look at n-gram models.

An n-gram model considers the frequency of a word given the previous n-1 words have occurred. So instead of using the whole sequence of words in a sentence, we approximate using the last n-1 words.

Let’s look at a uni-gram model. Here, the frequency of a word is irrespective of any other word. So this model simply corresponds to calculating the number of times each appears in the document. This is the most basic type of representation and does not take into account the context of any word. This unigram model fits into the BoW representation. We can also view this as a special case of the n-gram model with n=1.

In the bi-gram model, we will be looking at n-1 = 1 past words.

E.g. document: I love going swimming.

The bi-gram model will split the whole document into units: (I, love), (love, going), (going, swimming), and store the term frequency of each unit as before.

Sci-kit learn Vectorizers

Sci-kit learn vectorizers are one of the most efficient ways to achieve the above document-term matrix steps, including specifying which n-gram model, all in one. They come in three forms: