Sentiment Analysis of Tweets using Multinomial Naive Bayes | by Sergey Smetanin | Towards Data Science

The popularization of Web 2.0 significantly increased online communications. As a consequence, it provoked the rapid development research in the field of natural language processing in general and sentiment analysis in particular. Information overload and the growing volume of reviews and messages facilitated the need for high-performance automatic processing methods.
This article is devoted to binary sentiment analysis using the Naive Bayes classifier with multinomial distribution. We go through the brief overview of constructing a classifier from the probability model, then move to data preprocessing, training and hyperparameters optimization stages. We will write our script in Python using Jupyter Notebook.
Multinomial Naive Bayes classification algorithm tends to be a baseline solution for sentiment analysis task. The basic idea of Naive Bayes technique is to find the probabilities of classes assigned to texts by using the joint probabilities of words and classes. Let’s have a brief look at maths.
Given the dependent feature vector (x1, ..., xn) and the class Ck. Bayes’ theorem is stated mathematically as the following relationship:
According to the “naive” conditional independence assumptions, for the given class Ck each feature of vector xi is conditionally independent of every other feature xj for i≠j..
Thus, the relation can be simplified to
Since P(x1, ..., xn) is constant, if the values of the feature variables are known, the following classification rule can be used:
To avoid underflow, log probabilities can be used.
The variety of naive Bayes classifiers primarily differs between each other by the assumptions they make regarding the distribution of P(xi|Ck), while P(Ck) is usually defined as the relative frequency of class Ck in the training dataset.
The multinomial distribution is parametrized by vector θk=(θk1,...,θkn) for each class Ck, where n is the number of features (i.e. the size of the vocabulary) and θki is the probability P(xi|Ck) of feature i appearing in a sample that belongs to the class Ck.
The parameters θk is estimated by a smoothed version of maximum likelihood, i.e. relative frequency counting:
where Nki is the number of times feature i appears in a sample of class k in the training set T, and Ny is the total count of all features for class Ck. The smoothing priors α≥0 accounts for features not present in the learning samples and prevents zero probabilities in further computations. Setting α=1 is called Laplace smoothing, while α<1 is called Lidstone smoothing.
Thus, the final decision rule is defined as follows:
Sentiment dataset of Tweets in Russian [1] is available at http://study.mokoron.com/. The files positive.csv and negative.csv contain positively labelled and negatively labelled tweets, respectively.
import pandas as pdimport numpy as np# Read the data from CSV filesn = ['id', 'date','name','text','typr','rep','rtw','faw','stcount','foll','frien','listcount']data_positive = pd.read_csv('positive.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])data_negative = pd.read_csv('negative.csv', sep=';',error_bad_lines=False, names=n, usecols=['text'])# Create balanced datasetsample_size = min(data_positive.shape[0], data_negative.shape[0])raw_data = np.concatenate((data_positive['text'].values[:sample_size],                            data_negative['text'].values[:sample_size]), axis=0) labels = [1]*sample_size + [0]*sample_size
Texts generated by humans in social media sites contain lots of noise that can significantly affect the results of the sentiment classification process. Moreover, depending on the features generation approach, every new term seems to add at least one new dimension to the feature space. That makes the feature space more sparse and high-dimensional. Consequently, the task of the classifier has become more complex.
To prepare messages, such text preprocessing techniques as replacing URLs and usernames with keywords, removing punctuation marks and converting to lowercase were used in this program.
import redef preprocess_text(text):    text = re.sub('((www\.[^\s]+)|(https?://[^\s]+))','URL', text)    text = re.sub('@[^\s]+','USER', text)    text = text.lower().replace("ё", "е")    text = re.sub('[^a-zA-Zа-яА-Я1-9]+', ' ', text)    text = re.sub(' +',' ', text)    return text.strip()data = [preprocess_text(t) for t in raw_data]cc
A Pipeline class was used to make the vectorizer => transformer => classifier easier to work with. Such hyper-parameters as n-grams range, IDF usage, TF-IDF normalization type and Naive Bayes alpha were tunned using grid search. The performance of the selected hyper-parameters was measured on a test set that was not used during the model training step.
from sklearn.pipeline import Pipelinefrom sklearn.naive_bayes import MultinomialNBfrom sklearn.feature_extraction.text import CountVectorizer, TfidfTransformerfrom sklearn.model_selection import train_test_split, GridSearchCVtext_clf = Pipeline([('vect', CountVectorizer()),                     ('tfidf', TfidfTransformer()),                     ('clf', MultinomialNB())])tuned_parameters = {    'vect__ngram_range': [(1, 1), (1, 2), (2, 2)],    'tfidf__use_idf': (True, False),    'tfidf__norm': ('l1', 'l2'),    'clf__alpha': [1, 1e-1, 1e-2]}
The dataset was splitted into train and test subsets.
x_train, x_test, y_train, y_test = train_test_split(data, labels, test_size=0.33, random_state=42)
Finally, the grid search with 10-fold cross-validation was launched.
from sklearn.metrics import classification_reportclf = GridSearchCV(text_clf, tuned_parameters, cv=10, scoring=score)clf.fit(x_train, y_train)print(classification_report(y_test, clf.predict(x_test), digits=4))
The classification report for test dataset is listed below.
precision    recall  f1-score   support          0     0.7397    0.7941    0.7659     37078          1     0.7759    0.7183    0.7460     36792avg / total     0.7577    0.7564    0.7560     73870
According to the grid search results, best parameters set found on development set is the following: clf__alpha=1, tfidf__norm=l2, tfidf__use_idf=True, vect__ngram_range=(1, 2).
The model, which was trained on the development set, demonstrated F1=0.765 on the evaluation set. Classification metrics probably can be boost even more by using such techniques as stemming, normalization, syntactic and semantic features.
The source code is available at Github.
github.com
Y. Rubtsova, “Constructing a Corpus for Sentiment Classification Training”, Software & Systems, vol. 109, no. 1, pp. 72–78, 2015.“Naive Bayes”, scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/modules/naive_bayes.html. [Accessed: 26- Aug- 2018].“Working With Text Data”, scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html. [Accessed: 26- Aug- 2018].
Y. Rubtsova, “Constructing a Corpus for Sentiment Classification Training”, Software & Systems, vol. 109, no. 1, pp. 72–78, 2015.
“Naive Bayes”, scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/modules/naive_bayes.html. [Accessed: 26- Aug- 2018].
“Working With Text Data”, scikit-learn.org, 2018. [Online]. Available: http://scikit-learn.org/stable/tutorial/text_analytics/working_with_text_data.html. [Accessed: 26- Aug- 2018].