Another Twitter sentiment analysis with Python — Part 4 (Count vectorizer, confusion matrix) | by Ricky Kim | Towards Data Science

This is the part 4 of my ongoing Twitter sentiment analysis project. You can find the previous posts from below links.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
In part 3, I mainly focused on EDA and data visualisation, now it’s time to prepare for model building!
Before we can train any model, we first consider how to split the data. Here I chose to split the data into three chunks: train, development, test. I referenced Andrew Ng’s “deeplearning.ai” course on how to split the data.
Train set: The sample of data used for learning
Development set (Hold-out cross validation set): The sample of data used to tune the parameters of a classifier, and provide an unbiased evaluation of a model.
Test set: The sample of data used only to assess the performance of a final model.
The ratio I decided to split my data is 98/1/1, 98% of data as the training set, and 1% for the dev set, and the final 1% for the test set. The rationale behind this ratio comes from the size of my whole data set. The dataset has more than 1.5 million entries. In this case, only 1% of the whole data gives me more than 15,000 entries. This is more than enough to evaluate the model and refine the parameters.
Another approach is splitting the data into only train and test set, and run k-fold cross validation on the training set, so that you can have an unbiased evaluation of a model. But considering the size of the data, I have decided to use the train set only to train a model, and evaluate on the dev set, so that I can quickly test different algorithms and run this process iteratively.
If you want to know more in detail about data split for modelling, I recommend a blog post from “Machine Learning Mastery”. The blog has well-documented posts about a wide range of topics in data science, highly recommended.
Once I load the cleaned tweets dataset, I run data split as below. Note that I have split the data two times, the first to split train and dev+test, and the second to split dev and test.
x = my_df.texty = my_df.targetfrom sklearn.cross_validation import train_test_splitSEED = 2000x_train, x_validation_and_test, y_train, y_validation_and_test = train_test_split(x, y, test_size=.02, random_state=SEED)x_validation, x_test, y_validation, y_test = train_test_split(x_validation_and_test, y_validation_and_test, test_size=.5, random_state=SEED)print "Train set has total {0} entries with {1:.2f}% negative, {2:.2f}% positive".format(len(x_train),                                                                     (len(x_train[y_train == 0]) / (len(x_train)*1.))*100,                                                                           (len(x_train[y_train == 1]) / (len(x_train)*1.))*100)print "Validation set has total {0} entries with {1:.2f}% negative, {2:.2f}% positive".format(len(x_validation),                                                                          (len(x_validation[y_validation == 0]) / (len(x_validation)*1.))*100,                                                                       (len(x_validation[y_validation == 1]) / (len(x_validation)*1.))*100)print "Test set has total {0} entries with {1:.2f}% negative, {2:.2f}% positive".format(len(x_test),                                                          (len(x_test[y_test == 0]) / (len(x_test)*1.))*100,                                                           (len(x_test[y_test == 1]) / (len(x_test)*1.))*100)
When comparing various machine learning algorithms, baseline provides a point of reference to compare. The most popular baseline is the Zero Rule (ZeroR). ZeroR classifier simply predicts the majority category (class). Although there is no predictability power in ZeroR, it is useful for determining a baseline performance as a benchmark for other classification methods. As you can see from the above validation set class division, the majority class is negative with 50.40%, which means if a classifier predicts negative for every validation data, it will get 50.40% accuracy.
Another baseline I wanted to compare the validation results with is TextBlob. TextBlob is a python library for processing textual data. Apart from other useful tools such as POS tagging, n-gram, The package has built-in sentiment classification. This is a so-called out-of-the-box sentiment analysis tool, and in addition to the null accuracy, I will also keep in mind of the accuracy I get from TextBlob sentiment analysis to see how my model is performing.
from textblob import TextBlobfrom sklearn.metrics import accuracy_scorefrom sklearn.metrics import classification_report, confusion_matrixtbresult = [TextBlob(i).sentiment.polarity for i in x_validation]tbpred = [0 if n < 0 else 1 for n in tbresult]conmat = np.array(confusion_matrix(y_validation, tbpred, labels=[1,0]))confusion = pd.DataFrame(conmat, index=['positive', 'negative'],                         columns=['predicted_positive','predicted_negative'])print "Accuracy Score: {0:.2f}%".format(accuracy_score(y_validation, tbpred)*100)print "-"*80print "Confusion Matrix\n"print confusionprint "-"*80print "Classification Report\n"print classification_report(y_validation, tbpred)
TextBlob sentiment analysis yielded 60.65% accuracy on the validation set, which is 10.25% more accurate than null accuracy (50.40%).
If we want to use text in machine learning algorithms, we’ll have to convert them to a numerical representation. One of the methods is called bag-of-words approach. The bag of words model ignores grammar and order of words. Once we have a corpus (text data) then first, a list of vocabulary is created based on the entire corpus. Then each document or data entry is represented as numerical vectors based on the vocabulary built from the corpora.
With count vectorizer, we merely count the appearance of the words in each text. For example, let’s say we have 3 documents in a corpus: “I love dogs”, “I hate dogs and knitting”, “Knitting is my hobby and my passion”. If we build vocabulary from these three sentences and represent each document as count vectors, it will look like below pictures.
But if the size of a corpus gets big, the number of vocabulary gets too big to process. With my 1.5 million tweets, if I build vocabulary without limiting the number of vocabulary, I will have more than 260,000 vocabularies. This means that the shape of training data will be around 1,500,000 x 260,000, this sounds too big to train various different models with. So I decided to limit the number of vocabularies, but I also wanted to see how the performance varies depending on the number of vocabularies.
Another thing I wanted to explore is stop words. Stop Words are words which do not contain important significance, such as “the”, “of”, etc. It is often assumed that removing stop words is a necessary step, and will improve the model performance. But I wanted to see for myself if this is really the case. So I ran the same test with and without stop words and compared the result. In addition, I also defined my custom stop words list, which contains top 10 most frequent words in the corpora: “to”, “the”, “my”, “it”, “and”, “you”, “not”, “is”, “in”, “for”.
A model I chose to evaluate different count vectors is logistic regression. It is one of linear models, so computationally scalable to big data, compared to models like KNN or random forest. And once I have the optimal number of features and make a decision on whether to remove stop words or not, then I will try different models with the chosen number of vocabularies’ count vectors.
from sklearn.feature_extraction.text import CountVectorizerfrom sklearn.linear_model import LogisticRegressionfrom sklearn.pipeline import Pipelinefrom time import time
Below I define two functions to iteratively train on the different number of features, then check the accuracy of logistic regression on the validation set.
def accuracy_summary(pipeline, x_train, y_train, x_test, y_test):    if len(x_test[y_test == 0]) / (len(x_test)*1.) > 0.5:        null_accuracy = len(x_test[y_test == 0]) / (len(x_test)*1.)    else:        null_accuracy = 1. - (len(x_test[y_test == 0]) / (len(x_test)*1.))    t0 = time()    sentiment_fit = pipeline.fit(x_train, y_train)    y_pred = sentiment_fit.predict(x_test)    train_test_time = time() - t0    accuracy = accuracy_score(y_test, y_pred)    print "null accuracy: {0:.2f}%".format(null_accuracy*100)    print "accuracy score: {0:.2f}%".format(accuracy*100)    if accuracy > null_accuracy:        print "model is {0:.2f}% more accurate than null accuracy".format((accuracy-null_accuracy)*100)    elif accuracy == null_accuracy:        print "model has the same accuracy with the null accuracy"    else:        print "model is {0:.2f}% less accurate than null accuracy".format((null_accuracy-accuracy)*100)    print "train and test time: {0:.2f}s".format(train_test_time)    print "-"*80    return accuracy, train_test_timecvec = CountVectorizer()lr = LogisticRegression()n_features = np.arange(10000,100001,10000)def nfeature_accuracy_checker(vectorizer=cvec, n_features=n_features, stop_words=None, ngram_range=(1, 1), classifier=lr):    result = []    print (classifier)    print "\n"    for n in n_features:        vectorizer.set_params(stop_words=stop_words, max_features=n, ngram_range=ngram_range)        checker_pipeline = Pipeline([            ('vectorizer', vectorizer),            ('classifier', classifier)        ])        print "Validation result for {} features".format(n)        nfeature_accuracy,tt_time = accuracy_summary(checker_pipeline, x_train, y_train, x_validation, y_validation)        result.append((n,nfeature_accuracy,tt_time))    return result
Then I checked the accuracy on validation set for the different number of features by calling the “nfeature_accuracy_checker” I defined above. In addition, I have defined the custom stop words of top 10 frequent term to compare the result. Note that ‘term_freq_df.csv’ is the file I created from previous part of the project with the corpus.
csv = 'term_freq_df.csv'term_freq_df = pd.read_csv(csv,index_col=0)term_freq_df.sort_values(by='total', ascending=False).iloc[:10]
Just to double check if these top 10 words are actually included in SKLearn’s stop words list, I run a small code as below and see all the 10 words are also in SKLearn’s stop words list.
from sklearn.feature_extraction import texta = frozenset(list(term_freq_df.sort_values(by='total', ascending=False).iloc[:10].index))b = text.ENGLISH_STOP_WORDSset(a).issubset(set(b))
my_stop_words = frozenset(list(term_freq_df.sort_values(by='total', ascending=False).iloc[:10].index))
Now I can run “nfeature_accuracy_checker” on three different conditions. First with stop words removal, second with custom defined stop words removal, third without stop words removal.
print "RESULT FOR UNIGRAM WITHOUT STOP WORDS\n"feature_result_wosw = nfeature_accuracy_checker(stop_words='english')print "RESULT FOR UNIGRAM WITH STOP WORDS\n"feature_result_ug = nfeature_accuracy_checker()print "RESULT FOR UNIGRAM WITHOUT CUSTOM STOP WORDS (Top 10 frequent words)\n"feature_result_wocsw = nfeature_accuracy_checker(stop_words=my_stop_words)
I will show the result from above accuracy check with a graph.
nfeatures_plot_ug = pd.DataFrame(feature_result_ug,columns=['nfeatures','validation_accuracy','train_test_time'])nfeatures_plot_ug_wocsw = pd.DataFrame(feature_result_wocsw,columns=['nfeatures','validation_accuracy','train_test_time'])nfeatures_plot_ug_wosw = pd.DataFrame(feature_result_wosw,columns=['nfeatures','validation_accuracy','train_test_time'])plt.figure(figsize=(8,6))plt.plot(nfeatures_plot_ug.nfeatures, nfeatures_plot_ug.validation_accuracy, label='with stop words')plt.plot(nfeatures_plot_ug_wocsw.nfeatures, nfeatures_plot_ug_wocsw.validation_accuracy,label='without custom stop words')plt.plot(nfeatures_plot_ug_wosw.nfeatures, nfeatures_plot_ug_wosw.validation_accuracy,label='without stop words')plt.title("Without stop words VS With stop words (Unigram): Accuracy")plt.xlabel("Number of features")plt.ylabel("Validation set accuracy")plt.legend()
By looking at the evaluation result, removing stop words did not improve the model performance, but keeping the stop words yielded better performance. I wouldn’t say that removing stop words are not helping the model performance every time, but as empirical findings, in this particular setting, keeping the stop words improve the model performance.
According to Wikipedia, “n-gram is a continuous sequence of n items from a given sequence of text or speech”. In other words, n-grams are simply all combinations of adjacent words or letters of length n that you can find in your source text. Below picture represents well how n-grams are constructed out of source text.
In this project, I will extend the bag-of-words to trigrams, and see how it affects the performance.
print "RESULT FOR BIGRAM WITH STOP WORDS\n"feature_result_bg = nfeature_accuracy_checker(ngram_range=(1, 2))print "RESULT FOR TRIGRAM WITH STOP WORDS\n"feature_result_tg = nfeature_accuracy_checker(ngram_range=(1, 3))
After I run the above code, I plot the results of unigram, bigram, trigram together on a graph as below.
nfeatures_plot_tg = pd.DataFrame(feature_result_tg,columns=['nfeatures','validation_accuracy','train_test_time'])nfeatures_plot_bg = pd.DataFrame(feature_result_bg,columns=['nfeatures','validation_accuracy','train_test_time'])nfeatures_plot_ug = pd.DataFrame(feature_result_ug,columns=['nfeatures','validation_accuracy','train_test_time'])plt.figure(figsize=(8,6))plt.plot(nfeatures_plot_tg.nfeatures, nfeatures_plot_tg.validation_accuracy,label='trigram')plt.plot(nfeatures_plot_bg.nfeatures, nfeatures_plot_bg.validation_accuracy,label='bigram')plt.plot(nfeatures_plot_ug.nfeatures, nfeatures_plot_ug.validation_accuracy, label='unigram')plt.title("N-gram(1~3) test result : Accuracy")plt.xlabel("Number of features")plt.ylabel("Validation set accuracy")plt.legend()
The best validation set accuracy for each n-gram is as below.
unigram: 80,000 & 90,000 features at validation accuracy 80.28%
bigram: 70,000 features at validation accuracy 82.25%
trigram: 80,000 features at validation accuracy 82.44%
Below I defined another function to take a closer look at best performing number of features with each n-gram. Below function not only reports accuracy but also gives confusion matrix and classification report.
def train_test_and_evaluate(pipeline, x_train, y_train, x_test, y_test):    if len(x_test[y_test == 0]) / (len(x_test)*1.) > 0.5:        null_accuracy = len(x_test[y_test == 0]) / (len(x_test)*1.)    else:        null_accuracy = 1. - (len(x_test[y_test == 0]) / (len(x_test)*1.))    sentiment_fit = pipeline.fit(x_train, y_train)    y_pred = sentiment_fit.predict(x_test)    accuracy = accuracy_score(y_test, y_pred)    conmat = np.array(confusion_matrix(y_test, y_pred, labels=[0,1]))    confusion = pd.DataFrame(conmat, index=['negative', 'positive'],                         columns=['predicted_negative','predicted_positive'])    print "null accuracy: {0:.2f}%".format(null_accuracy*100)    print "accuracy score: {0:.2f}%".format(accuracy*100)    if accuracy > null_accuracy:        print "model is {0:.2f}% more accurate than null accuracy".format((accuracy-null_accuracy)*100)    elif accuracy == null_accuracy:        print "model has the same accuracy with the null accuracy"    else:        print "model is {0:.2f}% less accurate than null accuracy".format((null_accuracy-accuracy)*100)    print "-"*80    print "Confusion Matrix\n"    print confusion    print "-"*80    print "Classification Report\n"    print classification_report(y_test, y_pred, target_names=['negative','positive'])
Before I run the defined function, let me briefly explain about confusion matrix and classification report. In order to evaluate the performance of a model, there are many different metrics that can be used. Below I will talk in case of binary classification, in which the target variable only has two classes to be predicted. In the case of this project, the classes are either “negative” or “positive”.
One obvious measure of performance can be accuracy. It is the number of times the model predicted correctly for the class over the number of the whole data set. But in case of classification, this can be broken down further. Below is a representation of confusion matrix.
In the above matrix, each row represents the instances in an actual class while each column represents the instances in a predicted class, and it can be also presented swapping rows and columns (column for the actual class, row for predicted class). So the accuracy (ACC) I mentioned above can be expressed as below.
When the distribution of the classes in data is well balanced, accuracy can give you a good picture of how the model is performing. But when you have skewed data, for example, one of the class is dominant in your data set, then accuracy might not be enough to evaluate your model. Let’s say you have a dataset which contains 80% positive class, and 20% negative class. This means that by predicting every data into the positive class, the model will get 80% accuracy. In this case, you might want to explore further into the confusion matrix and try different evaluation metrics.
There can be 9 different metrics, just from the combination of numbers from confusion matrix, but I will talk about two of them in particular, and another metric which combines these two.
“Precision” (also called Positive Predictive Value) tells you what proportion of data predicted as positive actually is positive. In other words, the proportion of True Positive in the set of all positive predicted data.
“Recall” (also called Sensitivity, Hit Rate, True Positive Rate) tells you what proportion of data that actually is positive were predicted positive. In other words, the proportion of True Positive in the set of all actual positive data.
Below is the image of confusion matrix of cancer diagnose. If you think of “cancer” as positive class, “no cancer” as a negative class, the image explains well how to think of precision and recall in terms of the confusion matrix.
And finally, F1 score is the harmonic mean of precision and recall. The harmonic mean is a specific type of average, which is used when dealing with averages of units, like rates and ratios. So by calculating the harmonic mean of the two metrics, it will give you a good idea of how the model is performing both in terms of precision and recall. The formula is as below.
%%timetg_cvec = CountVectorizer(max_features=80000,ngram_range=(1, 3))tg_pipeline = Pipeline([        ('vectorizer', tg_cvec),        ('classifier', lr)    ])train_test_and_evaluate(tg_pipeline, x_train, y_train, x_validation, y_validation)
From the above classification reports, we can see that model has slightly higher precision in negative class and higher recall in positive class. But this averages out by calculating the F1 score, and for both classes, we get the almost same F1 score for both positive and negative class. There is also a way to visualise the model performance by plotting ROC curve, but I will explain more in detail later.
In this post, I have looked at how the logistic regression is performing based on the features extracted with count vectorizer. In the next post, I will experiment with Tfidf Vectorizer and see how the results differ.
Thank you for reading this long post! You can find the Jupyter Notebook of above code from below link.