Sentiment Analysis — Comparing 3 Common Approaches: Naive Bayes, LSTM, and VADER
By: Kevin C Lee
Tags: Machine Learning, Sentiment Analysis, Lstm, NLP, Python

Sentiment Analysis — Comparing 3 Common Approaches: Naive Bayes, LSTM, and VADER

Note: The code for this post can be found here

Sentiment Analysis (image by Author)

Sentiment Analysis, or Opinion Mining, is a subfield of NLP (Natural Language Processing) that aims to extract attitudes, appraisals, opinions, and emotions from text. Inspired by the rapid migration of customer interactions to digital formats e.g. emails, chat rooms, social media posts, comments, reviews, and surveys, Sentiment Analysis has become an integral part of analytics organizations must perform to understand how they are positioned in the market. To be clear, Sentiment Analysis isn’t a novel concept. In fact, it has always been an important part of CRM (Customer Relationship Management) and Market Research — companies rely on knowing their customers better to evolve and innovate. The more recent rise is driven largely by the availability/accessibility of customer interaction records and well as improved computing capabilities to process these data. This advancement has really benefited consumers in meaningful ways. More than ever, organizations are listening to their constituents to improve. There are numerous approaches for Sentiment Analysis. In this article, we’ll explore three such approaches: 1) Naive Bayes, 2) Deep Learning LSTM, and 3) Pre-Trained Rule-Based VADER Models. We will focus on comparing simple out-of-the-box version of the models with the recognition that each approach can be tuned to improve performance. The intention is not to go into great details about how each methodology works but rather a conceptual study on how they compare to help determine when one should be preferred over another.

Background on Sentiment Analysis

The objective of Sentiment Analysis ranges on the positive to negative spectrum. As with other NLP efforts, it is generally considered a classification problem, though it can be viewed as a regression problem when precision is important. Sentiment Analysis used to be accomplished by having a large labor force to read through and manually assess texts. This approach is costly and prone to human error. In an effort to automate this process, companies look to advanced analytical methods for solving this problem. The challenge with Sentiment Analysis is that people express and interpret sentiment polarity and intensity differently. Furthermore, words and sentences can have multiple meanings based on the context (known as polysemy). While some of these issues can be mitigated, there is almost always a trade-off between speed and performance like any analytical tasks. We review three general methodologies, each with its own strengths and drawbacks:

Strengths and Weaknesses Analysis

Introducing the Data Set

In addition to highlighting conceptual differences, we use the Sentiment140 data set to benchmark performances. This data set contains 1.6 million tweets and the corresponding sentiment labels (positive and negative).

Original Data

VADER Lexicon and Rule-Based Sentiment Analysis Tool

We begin with an easy to apply approach using the popular VADER (Valence Aware Dictionary and sEntiment Reasoner) tool. At its core, VADER uses a comprehensive, high quality lexicon (~7500 features) and sophisticated linguistic rules to produce sentiment scores. The construction and validation (statistically) of the sentiment dictionary is carefully documented in its published paper — this huge achievement cannot be overstated.

// vader_lexicon.txt

...

good: 1.9

happy 2.7

awesome 3.1

bad -2.5

sad -2.1

catastrophic -2.2

...

As the above sample shows, the VADER lexicon is a dictionary that assigns a predetermined sentiment score between -4 (most extreme negative) and 4 (most extreme positive) to each feature, which can be a word, an acronym, or an emoticon. VADER curated a valence-based lexicon capable of detecting both the intensity and polarity aspects of sentiments. This combined with powerful modifiers such as negations, contractions, conjunctions, booster words, degree adverbs, capitalization, punctuations, and slangs, are used to calculate scores for the input text(s). These modifiers are implemented as a Rule-Based model that changes the initial valence scores. The generous developers of VADER made it open source and easy to use:

# import SentimentIntensityAnalyzer class

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer # polarity_scores method of SentimentIntensityAnalyzer

SentimentIntensityAnalyzer().polarity_scores('Today is a good day.') # output:

# {'neg': 0.0, 'neu': 0.58, 'pos': 0.42, 'compound': 0.4404}

VADER produces a compound score that summarizes the sentiment intensity of the input text. It is computed by summing the valence scores of each feature in the lexicon, adjusted according to the rules, and then normalized to be between -1 (most extreme negative) and +1 (most extreme positive). The documentation describes the compound score as the “normalized, weighted composite score”. Furthermore, the pos , neg , neu scores represent the ratios for proportions of text that fall in each category (ex. Rule-Based enhancements).

VADER Calculations

vader_sentiment_analysis.py

Rule-Based models are easily understood and quick to implement, making it an attractive option for Sentiment Analysis. They are contingent on having a powerful lexicon and comprehensive linguistic rules. The challenge with Rule-Based models is that lexicons are time-consuming to create and validate. They struggle with coverage across endless combination of lexical features that can be important, as well as suitability issues across domains when words have different meanings. Furthermore, the approach only evaluates individual words and ignores the context in which they are used. This often leads to errors, particularly for sarcasm and irony.

Using VADER — we were able to produce an accuracy score of 72% on our test data set. VADER considers emoticons and acronyms/initialisms commonly found in social media text so it can be performant in the domain, but will likely perform worse in others. Important to note that VADER does produces precise polarity scores, which we categorize as positive or negative for comparison purposes.

P.S. Other lexicons include LIWC (Linguistic Inquiry and Word Count) and ANEW (Affective Norms for English Words).

Machine Learning for NLP

While Rule-Based solutions have proved to be performant and reliable, it is rigid and limited by a lexicon that could expire. Manually creating and validating a comprehensive sentiment lexicon is difficult. Instead, we want to move towards a Machine Learning solution that associates input texts to corresponding tags algorithmically. During the training process, pairs of texts and tags (sentiments) are fed into the Machine Learning algorithm to create a model capable of making predictions on new texts. An important caution worth mentioning is that supervised Machine Learning requires data to be labelled, which can mean that any subjectivity and bias in the data would be reflected in the model. Compared to pre-trained models, custom models give more control over the output and are suitable for applications that are specific.

Machine Learning NLP Text Classification Process

Naive Bayes

The Naive Bayes classifier uses probabilities to make predictions based on prior knowledge of conditions that might be related. In other words, it uses conditional probabilities of each lexical feature occurring in either positive or negative text in the training data to arrive at the outcome.

Machine Learning for NLP requires preprocessing to extract features from the original text(s). Specifically, it requires transformation of text data to numerical representations before the algorithms can process them, i.e. vectorization. For Naive Bayes, we will construct a simple DTM (document-term matrix) for model consumption although additional features can be included such as length of text, time/location of publication, named entities, etc. DTM tends to result in a wide feature space as each unique word or phrase in the vocabulary of the entire corpus becomes a feature. We include some data cleaning steps to help with dimensionality reduction and model (classifiers) performance. More about data preprocessing for NLP can be found here. The vector representation is constructed by counting the TF (Term Frequency) and weighting them with IDF (Inverse Document Frequency). We can use N-grams (consecutive terms) to capture some context in the text such as negations. N-gram contexts are a blunt instrument and can’t always capture expressions correctly. They can also negatively impact the model by burdening it with too many features.

We apply the following transformation to generate the training/testing data:

Naive Bayes Data Preprocessing Pipeline (Weighted Frequency Count)

naive_bayes_sentiment_analysis.py

The Naive Bayes model uses features that are reasonably easy to understand. It supports large-scale Sentiment Analysis efforts since the computation required for training is fast. However, it does have some significant drawbacks. As a probabilistic classifier, it is highly reliant on priors so the training data has to be complete and representative. The lack of good training data leads to poor inference on unseen data or out-of-vocabulary texts. It can also suffer if there is bias or imbalance in the data. Furthermore, it assumes that features are independent of one another, which means that the lexical features in the DTM contribute equally throughout all sentences, regardless of the relative position in the text. The Naive Bayes DTM model (with 300K unigram and bigram features) trained quickly (< 7 minutes). It produced a training score of 86% accuracy and testing /validation score of 79% accuracy.

For longer and wider data sets, Spark can be used to train the model quickly using the code here.

Deep Learning

Deep Learning allows for processing data in a much more complex manner. A Long Short-Term Memory model, or LSTM, is a type of Recurrent Neural Network (RNN) used to process temporal data. Since we think that the order of features (words) in a sentence is significant, this is the neural network architecture we use.

Deep Learning is computationally expensive and do not do well (poor performance and slow convergence) with high-dimensional, sparse vectors. When we extract features from the original text for model training, we need to represent them as vectors that are dense. One such technique is to convert each text into a sequence of numbers, where each number is mapped to a word in the vocabulary. Taking it a step further, we need to map words that have similar usage/meaning to similar real number vectors (rather than an index) using Word Embeddings. Without Word Embeddings, the model will misinterpret the index number of the words as having significance. Word Embeddings put all words into a multi-dimensional vector space so their similarities can be measured by distance. Generating Word Embeddings is done using open source pre-trained models (such as Word2vec, GloVe, or fastText) or custom neural network (unsupervised learning) models. When training custom Word Embeddings, it can be done separately or jointly with the neural network model of the task at hand (as an addition layer). This is the approach we take as it tends to lead to embeddings that are targeted to the context of data as well as the objective. In contrast to a document-term matrix that is sparse (hundreds of thousands of dimension), using Word Embeddings produces vectors that have fewer (hundreds) dimensions while capturing semantic similarities. Word Embeddings is considered one of the key breakthroughs of Deep Learning for challenging NLP problems.

We apply the following transformation to generate the training/testing data:

Word Embedding + LSTM Data Preprocessing Pipeline

Word Embedding LSTM Architecture

word_embedding_lstm_sentiment_analysis.py

One big advantage for the Deep Learning neural network approach is that we don’t have to try and engineer features since the neural network will learn the context and important features. However, this comes at the cost of interpretability — making it a significant limitation in some cases. Additionally, it is the most expensive approach in terms of the time to compute, particularly if Word Embeddings are jointly trained since it requires a large data set. For our data set, the Word Embedding LSTM model (with 61MM parameters) took 20 hours to train. It did produce a remarkable training score of 96% accuracy and testing /validation score of 80% accuracy.

Sentiment Analysis Application

When done correctly, Sentiment Analysis can provide tremendous value for any organization. Today, organizations use Sentiment Analysis to understand the public’s feelings towards their products, services, and brand. This can guide marketing strategies, inspire product development, inform political campaigns, and detect potentially damaging events. Below are some proven examples:

Identify potential detractors for prioritization to improve Net Promoter Score (NPS) ( Customer Care/Feedback Analytics )

) Evaluate customer engagements quickly to identify and resolve negative customer experiences, which translates to improved customer service and higher rate of returning customers ( Ticket Triaging )

) Monitor and manage public sentiment about the brand (Brand Intelligence)

Summary and Picking the Right Tool

We reviewed three approaches for Sentiment Analysis, each with its own strengths and drawbacks. When is comes to choosing the appropriate approach, there are some key things to consider:

What is the tolerance for accuracy and/or speed? If speed is prioritized over accuracy, Rule-Based solutions can be the right solution.

can be the right solution. Availability and completeness of data for training? With robust training data, a Naive Bayes model can be the right solution as it is fast to implement.

can be the right solution as it is fast to implement. How important is model interpretability? Deep Learning can be the right solution if accuracy is prioritized over model interpretability.

It is not uncommon to run multiple models in parallel for comparison. Once deployed, the process should include a feedback loop to inform when the models need to be updated.