Sentiment Analysis — A how-to guide with movie reviews | by Shiao-li Green | Towards Data Science

So, what exactly is sentiment? Sentiment relates to the meaning of a word or sequence of words and is usually associated with an opinion or emotion. And analysis? Well, this is the process of looking at data and making inferences; in this case, using machine learning to learn and predict whether a movie review is positive or negative.
Maybe you’re interested in knowing whether movie reviews are positive or negative, companies use sentiment analysis in a variety of settings, particularly for marketing purposes. Uses include social media monitoring, brand monitoring, customer feedback, customer service and market research (“Sentiment Analysis”).
This post will cover:
The competitionPre-processing dataFeaturesModellingEvaluationNext steps
The competition
Pre-processing data
Features
Modelling
Evaluation
Next steps
We’ll be using the IMDB movie dataset which has 25,000 labelled reviews for training and 25,000 reviews for testing. The Kaggle challenge asks for binary classification (“Bag of Words Meets Bags of Popcorn”).
Let’s have a look at some summary statistics of the dataset (Li, 2019). We are told that there is an even split of positive and negative movie reviews. Here are some of the positive and negative reviews:
It’s also interesting to see the distribution of the length of movie reviews (word count) split according to sentiment. The spread is similar in shape for both types of reviews however negative reviews are on average a tad shorter.
Text is messy, people love to throw in attempts at expressing themselves more clearly by adding extravagant punctuation and spelling words incorrectly. However, machine learning models can’t cope with text as input, so we need to map the characters and words to numerical representations.
Basic pre-processing for text consists of removing non-alphabetic characters, stop words (a set of very common words like the, a, and, etc.) and changing all words to lowercase. In this instance, we also need to remove HTML tags from the movie reviews. These steps can be packaged into the following function.
The table below shows how the most frequent words change once stop words are removed from the reviews.
How should we approach transforming word representations to numerical versions that a model can interpret? There are many ways, but for now, let’s try a few of the simple approaches.
We obtain the vocabulary list from the corpus (whole text dataset). The length of the vocabulary list is equal to the length of the vector that will be output when we apply Bag of Words (BOW). For each item (could be an entry, sentence, line of text), we transform the text into a frequency count in the form of a vector. In this case, we limit it to the top 5000 words to restrict the dimensionality of the data. We code this by setting up a count vectorizer from sklearn’s library, fit it on the training data and then transform both the training and test data.
An example of how this works is in the grey box below. We look at all the unique words in the corpus and then count how many times these appear in each of the pieces of texts, resulting in a vector representation of each piece of text.
Text:It was the best of times,it was the worst of times,it was the age of wisdom,it was the age of foolishness.Unique words:"it", "was", "the", "best", "of", "times", "worst", "age", "wisdom" "foolishness""it was the worst of times" = [1, 1, 1, 0, 1, 1, 1, 0, 0, 0]"it was the age of wisdom" = [1, 1, 1, 0, 1, 0, 0, 1, 1, 0]"it was the age of foolishness" = [1, 1, 1, 0, 1, 0, 0, 1, 0, 1]
Word2vec (Word to Vector) is a two-layer neural net that processes text. Its input is a text corpus and its output is a set of vectors: feature vectors for words in that corpus. The algorithm was created by Google in 2013. The 50-D space can be visualised by using classical projection methods (e.g. PCA) to reduce the vectors to two-dimensional data that can be plotted on a graph.
Why would we use Word2vec instead of BOW? It is better at learning words in their surrounding context. The process is a bit more convoluted than implementing BOW so I won’t outline it here, but it can be found in the GitHub folder (Dar, Green, Kurban & Mitchell, 2019). In short, it requires tokenising reviews as sentences rather than words, determining the vector representations and then averaging them appropriately.
This is short for Term-frequency-Inverse-Document-Frequency and gives us a measure of how important a word is in the document.
Term frequency (the same as for bag of words):
Inverse document frequency measures how rare a term is across all documents (the higher the value, the rarer the word)[1]:
We combine these two to get Tf-Idf as follows:
To implement this representation, we use the TfidfTransformer function from sklearn’s library. Fitting occurs on the training set and the values for the same words are determined for the test set.
[1] Where ln is the natural logarithm.
We attempted two different models for this dataset: Naïve Bayes and Random Forest. Naïve Bayes builds on Bayes’ Theorem while a Random Forest is a group of decision trees. Consolidated code for these two implementations can be found in the NLP_IMDb Jupyter notebook in the GitHub Repository (Dar et al., 2019).
This is a probabilistic machine learning model used for classification, based on Bayes Theorem:
B is the evidence while A is the hypothesis (Gandhi, 2018). We must assume that the features are independent (one does not affect the other).
Framing it in terms of this problem we have y representing a positive or negative review, X represents a vector of the features from the reviews. The equation we end up with for finding the class is:
This is implemented using sklearn’s Naïve Bayes package!
A random forest is a series of decision trees in which the leaf nodes indicate the predicted class. We can use the RandomForestClassifier function from sklearn’s ensemble package.
Each decision tree incorporates a selection of features and outputs a decision at the end. These results are then combined from all the decision trees to give the final class prediction. The diagram below from DataCamp visualises the process quite nicely.
Kaggle specifies using the area under the ROC curve as the metric for this competition. ROC is short for Receiving Operator Characteristic and is a probability curve. It plots the true positive rate against the false positive rate.[2] The higher the area under this curve, the better the model is at predicting the output. The graph for Bag of words and a Random Forest is below. The dotted line represents the baseline, which would be expected if the predictions were random.
We can inspect a few different permutations in the table below.
It’s interesting to see that Tf-Idf does marginally better and Naïve Bayes performs slightly better than the Random Forest. However, there is a significant drop in the performance of Naïve Bayes with Word2Vec features. This is likely because some information is lost when the representation is transformed to be positive (since Naïve Bayes only allows for positive values) and Gaussian Naïve Bayes is used instead.
[2] True positive represents when an item that is positive is predicted as positive. False positive represents when an item that is negative is predicted as positive.
This is just the start of sentiment analysis. Further approaches could use bigrams (sequences of two words) to attempt to retain more contextual meaning, using neural networks like LSTMs (Long Short-Term Memory) to extend the distance of relationships among words in the reviews and more.
Bag of Words Meets Bags of Popcorn. (n.d.). Retrieved from https://www.kaggle.com/c/word2vec-nlp-tutorial/
Dar, O., Green, S., Kurban, R., & Mitchell, C. (2019). Sprint 3, Team 1. Retrieved from https://github.com/shiaoligreen/practical-data-science
Gandhi, R. (2018, May 5). Naïve Bayes Classifier. Retrieved from https://towardsdatascience.com/naive-bayes-classifier-81d512f50a7c
Li, S. (2019, March 18). A Complete Exploratory Data Analysis and Visualization for Text Data. Retrieved from https://towardsdatascience.com/a-complete-exploratory-data-analysis-and-visualization-for-text-data-29fb1b96fb6a
Sentiment Analysis. (n.d.). Retrieved from https://monkeylearn.com/sentiment-analysis/