Sentiment Classifier Using NLP in Python | by Shivangi Sareen | Towards Data Science

The aim is to train a supervised Stochastic Gradient Descent classifier on a training set containing reviews of movies from IMDB, with labels, 0 for a negative review and 1 for a positive review. We’ll use the trained model to predict the polarity of reviews in our testing set. We’ll be training the model using a unigram, bigram, unigram with tf-idf and bigram with tf-idf approach. Read onward to see what it’s all about!
The dataset consists of 50,000 reviews from IMDB. Number of positive and negative reviews are equal, at 25k. Negative reviews have scores ≤ 4 out of 10 while a positive review ≥ 7 out of 10; neutral reviews are not included. The overall distribution of labels is balanced (25k pos and 25k neg). The dataset can be found here.
There is also an additional 50,000 unlabelled documents for unsupervised learning, however, we will be focussing on supervised learning techniques here.
There’s a train/ and test/ folder, each with pos/ and neg/ directories containing text files of reviews named named as [[id]_[rating].txt] where [id] is a unique id and [rating] is the rating corresponding to the review on a 1–10 scale.
Focussing only on supervised learning, the unsup/ directories can be ignored.
We’ll be using Python’s sci-kit learn library to train a Stochastic Gradient Descent classifier.
In both SGD (Stochastic Gradient Descent) and GD (Gradient Descent), we update parameters iteratively to minimise the loss function. In GD, we run through the whole training data per epoch to update one set of parameters in a given iteration. On the other hand, SGD, as the name suggests (stochastic), randomly chooses only a single training example per epoch to update the parameter in a particular iteration. Mini-batch Gradient Descent lies in between GD and SGD where a small number of training samples (subset) are used per epoch.
Here is a very helpful video and article on SGD.
The aim is to train our SGD classifier on 25k training samples and then use that to predict the polarity (0 for negative review and 1 for positive review) of the test data.
We’ll do some data preprocessing first — to combine the pos/ and neg/ data to create a training and testing file. Next we’ll do some data cleaning to get rid of any noise in the data.
SGD has to be fitted with two arrays: an array X of shape (n_samples, n_features) holding the training samples, and an array y of shape (n_samples,) holding the target values (class labels) for the training samples.
To convert our text data into such numerical form, we use Bag-of-Words models.
“The bag-of-words model is commonly used in methods of document classification where the (frequency of) occurrence of each word is used as a feature for training a classifier.” [source]
The most common type of feature extracted from the Bag-of-words model is term frequency, i.e., the number of times a term appears in the text. In our case of sentiment analysis, the more times a word like ‘dislike’, ‘hate’, ‘suck’, etc, the more likely its polarity be 0 (negative). And so this is how we’ll be training our model.
In our case, each review is considered a documentEach document is split into words or tokensWe then assign a weight to each token as its frequency of appearance in the documentFinally, we create a document-term matrix representing exactly the above, of shape (n_samples, n_features) with each samples representing a documents and each features representing a the tokenised word
In our case, each review is considered a document
Each document is split into words or tokens
We then assign a weight to each token as its frequency of appearance in the document
Finally, we create a document-term matrix representing exactly the above, of shape (n_samples, n_features) with each samples representing a documents and each features representing a the tokenised word
In BoW models, only the count of the words matter. As an alternative, we tend to look at n-gram models.
An n-gram model considers the frequency of a word given the previous n-1 words have occurred. So instead of using the whole sequence of words in a sentence, we approximate using the last n-1 words.
Let’s look at a uni-gram model. Here, the frequency of a word is irrespective of any other word. So this model simply corresponds to calculating the number of times each appears in the document. This is the most basic type of representation and does not take into account the context of any word. This unigram model fits into the BoW representation. We can also view this as a special case of the n-gram model with n=1.
In the bi-gram model, we will be looking at n-1 = 1 past words.
E.g. document: I love going swimming. The bi-gram model will split the whole document into units: (I, love), (love, going), (going, swimming), and store the term frequency of each unit as before.
Sci-kit learn vectorizers are one of the most efficient ways to achieve the above document-term matrix steps, including specifying which n-gram model, all in one. They come in three forms:
CountVectorizer: It counts the number of times a word shows up in the document and uses this value as its weight.
HashingVectorizer: This serves the same purpose as CountVectorizer but is as memory efficient as possible. It uses a hashing trick to find the token string name to feature integer index mapping. Although there are cons with using this vectorizer over an in-memory implemented CountVectorizer.
TfidfVectorizer: This is equivalent to CountVectorizer followed by TfidfTransformer. Tf-idf stands for term frequency-inverse document frequency. The tf-idf score of a word is the product of its tf and idf scores: the number of times a word appears in a document, and the inverse document frequency of the word across a set of documents. The whole idea is to weigh down the frequent terms while scaling up the rare ones. Tf-idf is one of the most popular term-weighting schemes today.
Let’s get right into the code now.
import csvimport reimport osfrom tqdm import tqdmfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.linear_model import SGDClassifierfrom sklearn.metrics import accuracy_score
The csv library in Python is used for CSV file reading and writing. We’ll be writing our data to a csv file and then reading from it
The re library is for regex that is used for data cleaning
The os library provides a portable way of using operating system dependent functionality
tqdm instantly makes loops show a smart progress meter — just wrap any iterable with tqdm(iterable)
CountVectorizer is used to convert a collection of text to a matrix of token counts
TfidfVectorizer is used to convert a collection of text to a matrix of TF-IDF features
SGDClassifier is fitted with two arrays: an array X of shape (n_samples, n_features) holding the training samples, and an array y of shape (n_samples,) holding the target values (class labels) for the training samples
accuracy_score is used to calculate the accuracy of the model
The below function is defined so as to create both test and training csv files. If train = True then it will use the training data path. If False, then it will use the path to the test data.
In both train/ and test/ directories, there are pos/ and neg/ folders with .txt files containing the review.
Each .txt file is read and then written to the respective training or test file. We are also including the polarity header for the test file to get the true values of the reviews that we will later compare with the predicted values.
def create_csv_file(filename, train=True):        header = ['row_number', 'text', 'polarity']        if train == True:        path_to_pos = "aclImdb/train/pos/"        path_to_neg = "aclImdb/train/neg/"    else:        path_to_pos = "aclImdb/test/pos/"        path_to_neg = "aclImdb/test/neg/"            count = 0    with open(filename, "w", newline = '') as f1:         writer = csv.writer(f1, delimiter = ',')         writer.writerow(header)         for f in os.listdir(path_to_pos):            polarity = 1            if f.endswith(".txt"):                open_file = open(path_to_pos+f, "r")                data = open_file.read()                writer.writerow([count, f'"{data}"', polarity])                count += 1                open_file.close()         for f in os.listdir(path_to_neg):            polarity = 0            if f.endswith(".txt"):                open_file = open(path_to_neg+f, "r")                data = open_file.read()                writer.writerow([count, f'"{data}"', polarity])                count += 1                open_file.close()
We use the below function to create X_train (list of all the reviews) and y_train (the polarity label for each of the training reviews) from the training csv file. From the test csv file, we create X_test (list of all the reviews whose sentiment we want to predict) and the true polarities of the test review as y_test_true.
def create_list_of_docs(filename):    docs = []    polarity = []    with open(filename,'r') as csvfile:            reader = csv.reader(csvfile, delimiter = ',')            next(reader, None) #this skips the header row            for row in tqdm(reader):                docs.append(row[1])                polarity.append(row[2])    return docs, polarity
The function remove_special_chars is used to clean up the data. The first regex sub removes html tags and everything in between them. The second regex sub removes anything but characters and a period.
def remove_special_chars (strs):    strs = re.sub(r'<.*?>', '', strs)    strs = re.sub(r'[^a-zA-Z. ]', '', strs)         return strs
The function data_preprocess applies the function remove_special_chars to each row of a list of strings and replaces the original row with the cleaned string
def data_preprocess(docs):    for index, row in tqdm(enumerate(docs)):        docs[index] = remove_special_chars(row)    return docs
Now let’s start using the above functions!
First let’s call create_csv_file to create the training file. Let’s call it imdb_tr.csv.
create_csv_file("imdb_tr.csv", train = True)
Next, let’s create the test file called imdb_te.csv.
create_csv_file("imdb_te.csv", train = False)
Now, we’ll call the create_list_of_docs function on both imdb_tr.csv and imdb_te.csv.
docs_train, y_train = create_list_of_docs("imdb_tr.csv")docs_test, y_test_true = create_list_of_docs("imdb_te.csv")
docs_train is a list of all the reviews for training; docs_test contains the reviews for testing; y_train contains the polarity of the reviews in docs_train and y_test_true contains the true polarity of the reviews in docs_test.
You’ll also notice the progress of the loop with the help of tqdm.
Next, we’re going to clean the string reviews in docs_train and docs_test.
docs_train = data_preprocess(docs_train)docs_test = data_preprocess(docs_test)
We’ll also convert y_train and y_test_true to a list of integers as the polarity got appended as strings.
y_train = list(map(int, y_train))y_test_true = list(map(int, y_test_true))
What about the stop-words you may ask? Read on to find out.
We’ve got our data all ready now! Onto the classifier!
As mentioned before, we’ll be using a stochastic gradient descent classifier, which requires an array X of shape (n_samples, n_features) holding the training samples, and an array y of shape (n_samples,) holding the target values (class labels) for the training samples.
Recall in the above theory, we achieve numerical representation of our text data using Bag-of-Word models or n-grams. And sci-kit learn vectorizers help us do just that.
We’ll be using CountVectorizer and TfidfVectorizer. The steps for both are the same.
Initialise the vectorizer and give it certain parameters that we want-
Initialise the vectorizer and give it certain parameters that we want-
This is where we remove the stop-words — by declaring stop_words = 'english', a built-in stop word list for English is used. We can also create a custom stop-words list that can be used in place of 'english'. This only applies if analyzer == 'word'.
lowercase = True is by default. It converts all characters to lowercase before tokenizing
ngram_range = ngram_range : The lower and upper boundary of the range of n-values for different word n-grams to be extracted. For example an ngram_range of (1, 1) means only unigrams, (1, 2) means unigrams and bigrams, and (2, 2) means only bigrams.
use_idf = True is only for TfidfVectorizer. It is True by default. This is to enable inverse-document-frequency re-weighting.
2. fit_transform on the training data learns the vocabulary dictionary and returns document-term matrix. Stored as X_train.
3. transform transforms the test data to the document-term matrix. Stored as X_test.
Finally, we’ll initialise the SDGClassifier. I’ve given it a loss = "hinge" and penalty = "l1".
The loss function defaults to 'hinge', which gives a linear SVM.
The penalty is the regularization term to be used. It defaults to 'l2' which is the standard regularizer for linear SVM models.
Next, we use the fit method on X_train and y_train to fit linear model with Stochastic Gradient Descent.
And then we use the predict method to predict class labels for samples in X_test.
The below function covers both TfidfVectorizer and CountVectorizer by using tfidf = True and tfidf = False respectively.
def ngram_classifier (docs_train, y_train, docs_test, ngram_range, tfidf):        if tfidf == True:        tfidfvec = TfidfVectorizer(stop_words = "english",                                  analyzer = 'word',                                  lowercase = True,                                  use_idf = True,                                  ngram_range = ngram_range)                X_train = tfidfvec.fit_transform(docs_train)            X_test = tfidfvec.transform(docs_test)            else:        cvec = CountVectorizer(stop_words = "english",                               analyzer = 'word',                               lowercase = True,                               ngram_range = ngram_range)            X_train = cvec.fit_transform(docs_train)                X_test = cvec.transform(docs_test)                    clf = SGDClassifier(loss = "hinge", penalty = "l1")        clf.fit(X_train, y_train)        prediction = clf.predict(X_test)        return prediction
Finally, we’ll predict using different n-gram combinations and get the accuracy of the model.
We get an accuracy of 83.6% for the below unigram without tfidf.
y_pred_unigram = ngram_classifier(docs_train,                                  y_train,                                   docs_test,                                   (1,1),                                   tfidf = False)accuracy_score(y_test_true, y_pred_unigram)
We get an accuracy of 84.04% for the below bigram without tfidf.
y_pred_bigram = ngram_classifier(docs_train,                                  y_train,                                  docs_test,                                  (1,2),                                  tfidf = False)accuracy_score(y_test_true, y_pred_bigram)
We get an accuracy of 86.94% for the below unigram with tfidf.
y_pred_unigram_tfidf = ngram_classifier(docs_train,                                         y_train,                                         docs_test,                                         (1,1),                                         tfidf = True)accuracy_score(y_test_true, y_pred_unigram_tfidf)
We get an accuracy of 85.77% for the below unigram with tfidf.
y_pred_bigram_tfidf = ngram_classifier(docs_train,                                        y_train,                                        docs_test,                                        (1,2),                                        tfidf = True)accuracy_score(y_test_true, y_pred_bigram_tfidf)
Tfidf scales down frequent words and scales up rarer ones. The tdfidf value of stop-words will be very low. So the argument is that do we really need to remove stop-words when using the TfidfVectorizer? The answer is yes. Even though stop-words removal may not help the unigram model (give it a try), in the higher n-gram models like bigrams and trigrams, it surely has an effect. Keeping the stop-words can create noise.
There you have it! I highly recommend if you’re reading this, that you follow along in Jupyter Notebook just to see what the output looks like at each stage, from combining to create training and test data, to cleaning it, and finally getting the predictions.
How about try and see how the above classifier be improved? There are many more parameters to TfidfVectorizer, CountVectorizer and the SGDClassifier. Also add some more steps to data preprocessing — use a lemmatizer and stemmer and see what the results are.
Thank you for taking out the time to read this. Hope this helped!