NLP: A Complete Sentiment Classification on Amazon Reviews
By: Ernest Leem
Tags: Text Preprocessing, Machine Learning, Classification, NLP, Sentiment Analysis

Word Vectorization (Embedding)

Word Embeddings is an NLP technique where words or phrases are represented in numerical values in pre-defined vector space. This process is inevitable as in the perspective of machine comprehension, raw strings or text formats cannot be used as input data for machine learning algorithms and deep learning architectures. While there exist numerous approaches such as Bag of Words, TF-IDF, word2vec, and GloVe, we will focus on TF-IDF for this post.

Bag of Words and TF-IDF?

The Bag of Words (BoW) model is the simplest form of representing texts into numerical values. It simply creates a vector space with the count of each word within a sentence or document.

For instance, if there are two reviews saying,

1. Review 1: ‘This TV is nice looking but very heavy’

2. Review 2: ‘This TV is bad looking and heavy’

Hence, the BOW approach would vectorize Review 1 as [1, 1, 1, 1, 0, 1, 1, 0, 1, 1]. The downside of the BOW model is that it may create sparse vectors with numerous 0 values and ultimately expand the dimensionality of the dataset. Also, this occurrence counting approach neglects the fact that longer sentences will tend to have higher counts of certain words. This is when the concept of TF-IDF is introduced.

TF-IDF (Term Frequency-Inverse Document Frequency) is a scoring measure generally used in the field of information retrieval that helps understand how important or relevant a term is within the sentence and a given collection of sentences (= documents).

The equation of TF-IDF is shown below.

While TF measures how frequent a term t shows, IDF measures the importance of term t, and the product of these two terms generates the composite weight of each term. In other words, each word in the preprocessed reviews will be assigned a score/weight that represents its importance among the review corpora. A high value of the TF-IDF score of a word indicates that the certain word is being informative for differentiating the reviews.

Implementing TF-IDF Weighting in Python

For this section, we will be utilizing the Pipeline class from the scikit-learn library which first uses the CountVectorizer to transform the review texts into a matrix of token counts (for TF).

Here, we set the range of n-grams to consider both unigrams (=single word) and bigrams (=combination of two words). Afterward, the TfidfTransformer function is implemented to convert the count matrix into normalized TF-IDF representation. The example code for running the multinomial Naive Bayes Classifier is shown below.

ML Classifiers for Multi-Classification

As we now have the word embeddings of the review data obtained from TF-IDF weighting, let’s try performing some machine learning classifiers for predicting the sentiments of each review. We will be testing these classifiers:

1. Multinomial Naive Bayes Classifier

is one of the simple but powerful algorithms used for classification problems. It is based on the Bayes Theorem with an assumption that all observed features are independent of each other. In our project’s case, it would be assuming that every word in a review is independent of the other ones. 2. Linear Support Vector Machine

is a linear classifier which is optimized by Stochastic Gradient Descent. SGD algorithm computes the minimum of the cost function through iteratively updating the gradient with a randomly selected single sample.

3. Logistic Regression Classifier

is a common supervised machine learning algorithm for categorizing data into binary or multiple groups with discrete prediction values. It is a regression model based on the sigmoid function for computing the probability of each data points belonging to the categories. It follows the assumption that there exists a linear relationship between the logit of independent variables and the response.

Before throwing in our review data to the selected algorithms, let’s first split our dataset into train & test sets using the ‘train_test_split’ function from scikit-learn library. Here, it is important to use the ‘stratify’ parameter, which makes a split with equal proportions of each class (=sentiment labels).

Evaluation Metrics

For evaluating our candidate model for the prediction of sentiment multi-classification, we will be using accuracy on the test data set. Accuracy is a metric simply dividing # of correct predictions by the total # of predictions. We concluded that accuracy would be the most appropriate and concise measure as we previously re-sampled our initial data to become well balanced among all 5 sentiment classes. Just a quick note, in case of dealing with imbalanced data, the f1-score metric (a mixture of recall & precision) is recommended.

Initial ML Classifier’s Results

The following code returns the accuracy score, confusion matrix, and classification report of each method. In the case of classification accuracy on the test set, Naive Bayes showed 0.4582, SGD with 0.4729, and Logistic Regression with 0.5080.

Hyperparameter Tuning

Overall, our initial approach for ML models concluded that the logistic regression classifier shows the best sentiment classification performance with 0.5080 accuracies. Hence, we seek to tune the hyperparameters of this model with the gridsearchCV library. Below are the descriptions of the hyperparameters selected for this process.

> C (Inverse Regularization Parameter):

The inverse regularization parameter C in logistic regression is a control variable that is inversely positioned to the lambda regulator. It can be regarded as a method of controlling the overfitting of the regression model. Hence, a higher value of C indicates a stronger impact on the regularization strength. The grid of this parameter was [0.01, 0.1, 1]. > Optimization Algorithms

– ‘lbfgs’: Algorithm that approximates the estimated Hessian matrix by updating with approximate gradient evaluations.

– ‘sag’: ‘sag’ algorithm is generally faster than other algorithms due to its benefit of using a random sample of previous gradient values for obtaining a faster convergence rate. Thus, it is generally preferred for large datasets.

– ‘saga’: ‘saga’ algorithm is a variant from ‘sag’, as it allows to handle non-smooth L1 penalty regularization. Similar to ‘saga’, it is suitable for large datasets.

Let’s utilize the GridSearchCV function from scikit-learn library as below. The code first outputs the best estimator that showed the highest prediction accuracy on k-fold stratified cross-validation. Then, these estimators are used to get the accuracy score on the test set.

The result of the gridsearchCV showed C=1, and solver=’saga’ to be the best estimator that led to achieving 0.5310 accuracy for predicting the review’s sentiment on the test set.

Discussion

The sentiment classification accuracy achieved throughout this post is not sufficient to be used for an end-user. Therefore, let’s try looking into the confusion matrix for further observations. We utilized the function from the below link, which is built to concisely visualize the confusion matrix.

The confusion matrix is a summary of the prediction results of classification, which can also identify what type of error our model is making. In contrast to our unsatisfactory accuracy result, we can observe how well our model has classified the extreme reviews (=’Very Negative’, ’Very Positive’) into its sentiment class. On the other hand, our model is shown to be performing poorly in distinguishing between ’Somewhat Negative’, ’Neutral’, and ’Somewhat Positive’. Thus, if possible for future work, we conclude to focus on maximizing the prediction accuracy of the vulnerable sentiment classes to enhance the overall accuracy of our selected model.

Any Future Work?