Fundamentals of Supervised Sentiment Analysis | by Eunjoo Byeon | Towards Data Science

In this post, I will explain a few basic machine learning approaches in classifying tweet sentiment and how to run them in Python.
Sentiment analysis is used to identify the affect or emotion (positive, negative, or neutral) of the data. For a business, it is a simple way to determine customers’ reactions towards the product or service and to quickly pick up on any change of emotion that may require immediate attention. The most basic approach to this problem is to use supervised learning. We can have actual humans to determine and label the sentiment of our data and treat it like a text classification problem. This is exactly what I will go over in this post, and will revisit the topic in the later post to discuss unsupervised methods.
You may find some human-labeled tweets data on the data.world. Data contains over 8000 tweets that have been labeled to be positive, negative, neutral, or unknown (“I can’t tell’).Sentiment140 provides training data by a team at Stanford. Labels for this dataset were automatically annotated.
You may find some human-labeled tweets data on the data.world. Data contains over 8000 tweets that have been labeled to be positive, negative, neutral, or unknown (“I can’t tell’).
Sentiment140 provides training data by a team at Stanford. Labels for this dataset were automatically annotated.
After cleaning the data (removing observations with missing texts, removing ‘unknown’ class, removing RTs), and splitting our data into train, test, and validation sets, we need to preprocess texts so they are properly quantifiable for our analysis. I will go through them one by one.
Different encodings can result in some strange characters appearing on our data. So first we will make sure we are only keeping ASCII characters. The below code will filter out all characters that are not ASCII.
def ascii_only(str_):    return str_.encode("ascii", "ignore").decode()
This one is straightforward.
def make_lower(str_):    return str_.lower()
Due to mishaps in encodings, some tweets contain HTML symbols such as &nbsp;. Before we remove punctuations, we will get rid of these words first so we are not left with gibberish after we remove punctuations. Other alphabetical words we want to remove are the name of users (e.g. @stereopickle) and hyperlinks (written as {link} in this dataset). We will use regex to do this.
import re def remove_nonwords(str_):    return re.sub("[^A-Za-z0-9 ]\w+[^A-Za-z0-9]*", ' ', str_)
This expression means to substitute all words that do not start with the alphanumeric characters with space. These words may or may not end with a non-alphanumeric character.
This is a step specific to this dataset. Because our dataset contains tweets about products, there’s a lot of references to the actual brand and product names. We don’t want the overall distribution of sentiment per different brands or products to bias our model, so we will remove some of them.
def remove_brandwords(str_):    p = '''#?(iphone|ipad|sxsw|hcsm|google|apple|cisco|austin|    atari|intel|mac|pc|blackberry|android|linux|ubuntu)[a-z0-9]*'''    return re.sub(p, ' ', str_)
Another straightforward cleaning. Please note, that I am removing all punctuation here for the sake of simplicity, but in reality, many tweets may use punctuations to express the sentiment, such as :) and :(.
import stringpunctuations = string.punctuationpunctuations = punctuations + '�' + string.digitsdef remove_punctuations(str_, punctuations):    table_ = str.maketrans('', '', punctuations)    return str_.translate(table_)
If you are curious about the above method, please take a look at my other post on how to effectively removing punctuations.
towardsdatascience.com
Finally, we will lemmatize our vocabularies and remove stopwords using NLTK. In summary, lemmatizer will return the vocabularies into its basic form, lemma.
from nltk.stem import WordNetLemmatizerfrom nltk.corpus import stopwordssw = stopwords.words('english')def lemmatize(str_, sw):    wnl = WordNetLemmatizer()    return ' '.join([wnl.lemmatize(w) for w in x.split() if w not in sw])
Read more about lemmatization or stemming here.
medium.com
After applying the above functions to our text data, we should have a pretty clean set of words to work with. But I like to add a feature selection step here because NLP problems tend to end up with too many features when each vocabulary or bi-grams or tri-grams are tokenized.
Of course, there are many ways to purge features, but I like to combine words that are similar using Spacy. The basic idea is to iterate through words that only occur a few times and to find an existing word with high similarity (closer in the vector space).
This custom function returns a dictionary with low occurring words as keys and their replacements as values. This step often fixes some typos and corrects where lemmatization might have missed. We can use this replacement dictionary to change words to their more frequently occurring counterparts and remove ones that are occurring less than specified times.
Typically our data will have a high class-imbalance problem as people are more likely to write about neutral or positive tweets than negative tweets in most cases. But for most business problems, our model must detect these negative tweets. So we will keep an eye out for these by looking at the macro-averaged f1-score as well as the precision-recall curve. Below function will plot the ROC curve and precision-recall curve and print the key evaluation metrics.
We can use a scikit-learn’s DummyClassifier to first see what our baseline measure would be if we were to just classify it based on how frequently each class occurs.
from sklearn.dummy import DummyClassifierdummy_classifier = DummyClassifier()dummy_classifier.fit(tweets_train, labels_train)y_pred_p = dummy_classifier.predict_proba(tweets_validation)    y_pred = dummy_classifier.predict(tweets_validation)
One of the simplest way to quantify text data is to just count the frequency of each word. The scikit-learn’s CountVectorizer can do that job easily.
from sklearn.feature_extraction.text import CountVectorizercountvec = CountVectorizer(ngram_range = (1, 2), min_df = 2)count_vectors = countvec.fit_transform(tweets_train)
This returns the count vectors of single vocabularies and bigrams that occurs at least twice. Then we can use thes count vectors to train different classification algorithms.
A problem with the count vector is that it only looks at the frequency of individual word and it does not care about the context in which the word occurs. There’s no way to assess whether how important specific word in a tweet is. This is where the term frequency — inverse document frequency (TF-IDF) score comes in. TF-IDF score weighs words that are more uniquely frequent in one tweet more than words that tend to be frequent across all tweets.
from sklearn.feature_extraction.text import TfidfVectorizertfvec = TfidfVectorizer(ngram_range = (1, 2), min_df = 2)tf_vectors = tfvec.fit_transform(tweets_train)
Now that we have two vectorized texts, we can test different classifiers for each of these vectors.
Naive-Bayes is one of the more popular choices for text classification. It’s a simple application of Bayes Theorem on each class and the predictors, and it assumes that each individual features (words in our case) are independent of each other.
So let’s say, we have a tweet that reads... “I love my new phone. It’s really fast, reliable and well-designed!”. This tweet clearly has a positive sentiment. In this case, the Naive-Bayes model assumes that the individual words like ‘love’, ‘new’, ‘really’, ‘fast’, ‘reliable’, all contribute independently to its positive class. In other words, likelihood of the tweet being positive when it uses the word ‘reliable’ does not change by other words. This does not mean that these words are independent in their appearances. Some words may tend to appear together more often than not, but that does not mean how much each word contributes to its class is dependent.
Naive-Bayes algorithm is simple to use and reliable when the above assumption holds. Since testing on our model requires vectorization, we can get the pipeline built into our model.
from sklearn.naive_bayes import MultinomialNBfrom sklearn.pipeline import Pipelinemn_nb = MultinomialNB()# change countvec to tfvec for tf-idf model = Pipeline([('vectorize', countvec), ('classify', mn_nb)])# fitting training count vectors (change to tf_vectors for tf-idf)model['classify'].fit(count_vectors, labels_train)y_pred_p = model.predict_proba(tweets_validation)    y_pred = model.predict(tweets_validation)evaluating(labels_validation, y_pred, y_pred_p)
Because of its assumption of independence between features, Naive Bayes overestimates confidence of how much each feature contributes to the label, making it a bad estimator. So take the probability of predictions with a grain of salt.
Another popular choice of the text classification algorithm is the support vector machine (SVM). Simply put, SVM finds the hyperplane that divides the classes with a maximum margin between them. The main reason SVM is preferred in text classification is that we tend to end up with a lot of features. If we were to work in such a high dimensional space that takes all our features, it would have caused a problem known as the curse of dimensionality. Basically, our space is too big that our observations start to lose their meanings. But SVM is more robust when dealing with a large number of features because it uses the kernel trick. SVM does not actually work in the high dimensions, it just looks at the pairwise distances between observations as if they are in the high dimensions. It does take a long time to do the job, but it is robust.
from sklearn.svm import SVCsvm_classifier = SVC(class_weight = 'balanced', probability= True)# don't forget to adjust the hyperparameters! # change countvec to tfvec for tf-idf svm_model = Pipeline([('vectorize', countvec), ('classify', svm_classifier)])# fitting training count vectors (change to tf_vectors for tf-idf)svm_model['classify'].fit(count_vectors, labels_train)y_pred_p = svm_model.predict_proba(tweets_validation)    y_pred = svm_model.predict(tweets_validation)evaluating(labels_validation, y_pred, y_pred_p)
When the SVM uses the kernel trick, things get into a bit of grey area, in terms of interpretability. But we can use the Shapley value to decipher how individual features are contributing to the classification. We’ll use SHAP’s friendly interface to visualize the Shapley values. For a detailed tutorial on this, I recommend reading the documentation on SHAP.
import shapshap.initjs() sample = shap.kmeans(count_vectors, 10)e = shap.KernelExplainer(svm_model.predict_proba, sample, link = 'logit')shap_vals = e.shap_values(X_val_tf, nsamples = 100)shap.summary_plot(shap_vals,                   feature_names = countvec.get_feature_names(),                  class_names = svm_model.classes_)
Let’s dive a little bit deeper (literally). So far we worked with two different frequency measures to quantify our text data. But the frequency of each word tells only little bit of the story. Understanding language and its meaning requires understanding of syntax, or at the very least, the sequence of words. So we will look at a deep learning architecture that cares about the sequence of vocabularies: the long short-term memory (LSTM) architecture.
For the LSTM, we need to feed in texts as a sequence. Below steps outline the steps to run and evaluate the LSTM classifier. I explained each step in the code.
One downside of our LSTM model is that it only contains information that’s present within our training data, while vocabularies have semantic meanings outside of the tweets. Knowing how each vocabulary relates to each other in terms of semantic similarity may help our model. We can apply weights to our vocabularies based on the pre-trained word embedding algorithm.
For this, we will use GloVe, the vector representations obtained by a team at Stanford. I’m using their 200 dimensions word vectors trained on 2 Billion tweets. You will need to download the vectors from their website.
Then you can add the obtained vector matrix as embedding weights to the Embedding layer of our LSTM architecture.
# adding the bolded partmodel.add(Embedding(num_vocab, 200, weights = [vector_matrix],                     input_length = max_len))
By using the word embedding and LSTM, my model showed 20% increase in overall accuracy and 16% increase in the macro-averaged F1-score.
We reviewed the fundamentals of building a sentiment analysis model using tweets data. I will end the post with some of the questions to think about and to expand upon.
How can we approach the same problem, if we didn’t have labels? (unsupervised learning)What are other ways to reduce dimensions while keeping its interpretability?
How can we approach the same problem, if we didn’t have labels? (unsupervised learning)
What are other ways to reduce dimensions while keeping its interpretability?