Fundamentals of Supervised Sentiment Analysis
By: Eunjoo Byeon
Tags: Machine Learning, Sentiment Analysis, NLP, Python, Data Science

Model Evaluation

Evaluation

Typically our data will have a high class-imbalance problem as people are more likely to write about neutral or positive tweets than negative tweets in most cases. But for most business problems, our model must detect these negative tweets. So we will keep an eye out for these by looking at the macro-averaged f1-score as well as the precision-recall curve. Below function will plot the ROC curve and precision-recall curve and print the key evaluation metrics.

Baseline Model

We can use a scikit-learn’s DummyClassifier to first see what our baseline measure would be if we were to just classify it based on how frequently each class occurs.

from sklearn.dummy import DummyClassifier

dummy_classifier = DummyClassifier()

dummy_classifier.fit(tweets_train, labels_train) y_pred_p = dummy_classifier.predict_proba(tweets_validation)

y_pred = dummy_classifier.predict(tweets_validation)

Bag-of-Words Model (Count Vectors)

One of the simplest way to quantify text data is to just count the frequency of each word. The scikit-learn’s CountVectorizer can do that job easily.

from sklearn.feature_extraction.text import CountVectorizer



countvec = CountVectorizer(ngram_range = (1, 2), min_df = 2)

count_vectors = countvec.fit_transform(tweets_train)

This returns the count vectors of single vocabularies and bigrams that occurs at least twice. Then we can use thes count vectors to train different classification algorithms.

TF-IDF Vectors

A problem with the count vector is that it only looks at the frequency of individual word and it does not care about the context in which the word occurs. There’s no way to assess whether how important specific word in a tweet is. This is where the term frequency — inverse document frequency (TF-IDF) score comes in. TF-IDF score weighs words that are more uniquely frequent in one tweet more than words that tend to be frequent across all tweets.

from sklearn.feature_extraction.text import TfidfVectorizer



tfvec = TfidfVectorizer(ngram_range = (1, 2), min_df = 2)

tf_vectors = tfvec.fit_transform(tweets_train)

Now that we have two vectorized texts, we can test different classifiers for each of these vectors.

Naive-Bayes

Naive-Bayes is one of the more popular choices for text classification. It’s a simple application of Bayes Theorem on each class and the predictors, and it assumes that each individual features (words in our case) are independent of each other.

So let’s say, we have a tweet that reads… “I love my new phone. It’s really fast, reliable and well-designed!”. This tweet clearly has a positive sentiment. In this case, the Naive-Bayes model assumes that the individual words like ‘love’, ‘new’, ‘really’, ‘fast’, ‘reliable’, all contribute independently to its positive class. In other words, likelihood of the tweet being positive when it uses the word ‘reliable’ does not change by other words. This does not mean that these words are independent in their appearances. Some words may tend to appear together more often than not, but that does not mean how much each word contributes to its class is dependent.

Naive-Bayes algorithm is simple to use and reliable when the above assumption holds. Since testing on our model requires vectorization, we can get the pipeline built into our model.

from sklearn.naive_bayes import MultinomialNB

from sklearn.pipeline import Pipeline mn_nb = MultinomialNB() # change countvec to tfvec for tf-idf

model = Pipeline([('vectorize', countvec), ('classify', mn_nb)]) # fitting training count vectors (change to tf_vectors for tf-idf)

model['classify'].fit(count_vectors, labels_train) y_pred_p = model.predict_proba(tweets_validation)

y_pred = model.predict(tweets_validation)

evaluating(labels_validation, y_pred, y_pred_p)

Because of its assumption of independence between features, Naive Bayes overestimates confidence of how much each feature contributes to the label, making it a bad estimator. So take the probability of predictions with a grain of salt.

Support Vector Machine (SVM)

Another popular choice of the text classification algorithm is the support vector machine (SVM). Simply put, SVM finds the hyperplane that divides the classes with a maximum margin between them. The main reason SVM is preferred in text classification is that we tend to end up with a lot of features. If we were to work in such a high dimensional space that takes all our features, it would have caused a problem known as the curse of dimensionality. Basically, our space is too big that our observations start to lose their meanings. But SVM is more robust when dealing with a large number of features because it uses the kernel trick. SVM does not actually work in the high dimensions, it just looks at the pairwise distances between observations as if they are in the high dimensions. It does take a long time to do the job, but it is robust.

from sklearn.svm import SVC svm_classifier = SVC(class_weight = 'balanced', probability= True)

# don't forget to adjust the hyperparameters! # change countvec to tfvec for tf-idf

svm_model = Pipeline([('vectorize', countvec), ('classify', svm_classifier)]) # fitting training count vectors (change to tf_vectors for tf-idf)

svm_model['classify'].fit(count_vectors, labels_train) y_pred_p = svm_model.predict_proba(tweets_validation)

y_pred = svm_model.predict(tweets_validation)

evaluating(labels_validation, y_pred, y_pred_p)

SHAP Evaluation

When the SVM uses the kernel trick, things get into a bit of grey area, in terms of interpretability. But we can use the Shapley value to decipher how individual features are contributing to the classification. We’ll use SHAP’s friendly interface to visualize the Shapley values. For a detailed tutorial on this, I recommend reading the documentation on SHAP.

import shap

shap.initjs()

sample = shap.kmeans(count_vectors, 10)

e = shap.KernelExplainer(svm_model.predict_proba, sample, link = 'logit')

shap_vals = e.shap_values(X_val_tf, nsamples = 100)

shap.summary_plot(shap_vals,

feature_names = countvec.get_feature_names(),

class_names = svm_model.classes_)

Phew, that was a lot. Let’s take a breather. Photo by freestocks on Unsplash

LSTM

Let’s dive a little bit deeper (literally). So far we worked with two different frequency measures to quantify our text data. But the frequency of each word tells only little bit of the story. Understanding language and its meaning requires understanding of syntax, or at the very least, the sequence of words. So we will look at a deep learning architecture that cares about the sequence of vocabularies: the long short-term memory (LSTM) architecture.

For the LSTM, we need to feed in texts as a sequence. Below steps outline the steps to run and evaluate the LSTM classifier. I explained each step in the code.

Word Embedding (GloVe)

One downside of our LSTM model is that it only contains information that’s present within our training data, while vocabularies have semantic meanings outside of the tweets. Knowing how each vocabulary relates to each other in terms of semantic similarity may help our model. We can apply weights to our vocabularies based on the pre-trained word embedding algorithm.

For this, we will use GloVe, the vector representations obtained by a team at Stanford. I’m using their 200 dimensions word vectors trained on 2 Billion tweets. You will need to download the vectors from their website.

Then you can add the obtained vector matrix as embedding weights to the Embedding layer of our LSTM architecture.

# adding the bolded part

model.add(Embedding(num_vocab, 200, weights = [vector_matrix],

input_length = max_len))

By using the word embedding and LSTM, my model showed 20% increase in overall accuracy and 16% increase in the macro-averaged F1-score.