Sentiment Analysis on Customer Review: Understanding NLP models
By: Rachel Zhiqing Zheng
Tags: Sentiment Analysis, Visualization, Review, Ecommerce, NLP

Women using smartphones, @youxventures

Sentiment analysis has many industry use cases, especially for digital economy. It can be used to monitor trends, analyze products and market research etc. The technique used to analyze text is called natural language processing (NLP), which has many fascinating aspects.

This project aims to:

classify customers’ sentiment as accurate as possible;

explore multiple ways of text mining and processing, including TF-IDF and text embedding with Word2Vec;

understand the pros and cons of each process.

Data + EDA

Our data set consists of 23,486 customer reviews from e-commerce clothing website. There are other features other than text, e.g. age of the reviewer, category of the item etc. We chose to ignore all the features as this article focuses on NLP only. Hence review is the only feature that we look at.

Example of the data

The sentiment is determined by “Rating” which is rated from 1 to 5, 5 being most favorable. We have transformed the target variable into binary class, i.e. “positive” for reviews rating 4 and 5 and “not positive” for reviews from 1 to 3.

Rating distribution after binary transformation

Observing the graph, there are more positive reviews than non-positive ones, with 82% positive and 18% not positive. In other words, a do-nothing model can achieve 82% accuracy by giving constant prediction of positive class, which is much better than a random guess model (randomly guess whether it is positive or not positive). Having this in mind helps us to identify the baseline performance of the model.

I will refer to not positive class as negative in the article below though it is actually neutral and negative class combined.

We also want to understand what are the frequently used words by customers.

15 most used words for positive reviews

15 most used words for negative reviews

The top graph shows the most frequently used words in positive reviews and the bottom graph shows the most frequently used words in negative reviews. Both graphs are plotted by removing words that are too frequent and acts like noises to the text, e.g. “that”, “the”, pronouns etc.

The vocabulary extracted by both positive reviews and negative reviews are (surprisingly) similar. Most reviews talk about the size, fit, material, fabric, color etc, which are associated with the inherited property of the clothing items. There are also words representing customers’ emotions like love and like and (surprisingly) found in both categories. Note that words like “no” and “not” are removed as we extracted those words. This may explain why love and like appears in negative reviews. We will understand the phenomenon better in the next sessions.

Methodology Overview

Steps for NLP processing

Like all data science projects, there are feature selection process for text data. The only difference is that we also need to “extract” the feature as text is not suitable for machine learning process without extraction.

We will focus on the difference of different text extraction model including bag of words, term frequency — inverse document frequency (TF-IDF) and text embedding.

For feature selection, we will examine three approaches: lemmatization, stop words removal and latent semantic analysis (LSA).

Logistic regression, multinomial naive bayes (MNB), linear support vector machine (Linear SVC) and random forest models are used in this project.

Bag of Words

The bag of words model is built on counting: it returns a matrix of word counts, each column maps to a unique word appears in the corpus and each row maps to each data point, a review in our case.

It is a simple and easy to understand model for NLP beginners. We have built our baseline model with bag of words model using logistic regression.

Predicted labels v.s. True labels: BOW model

Five short reviews are transformed and predicted with our baseline model. The model got 3 out of 5 correct, which gives an accuracy of 60% base on our input. It is not very ideal as it performs worse than a do-nothing model.

However, the bag of words model has some intrinsic shortcomings. The number of features are numerous and difficult to reduce. In our example, we have over 11,000 features resulting from bag of words transformation. In English particularly, there are multiple forms of the same word. For example “is” “am” “are” are the same word essentially. That is why we need to look into lemmatization and stop words.

Lemmatization is a technique to reduce the words to its original form. In the previous example, “is” “am” “are” can be reduced to “be”. Stop words removal aims to remove words that appears too frequently in the text. You can think of them as “noise” to the text data.

Do note that both ways of treating the text could affect the model performance. Hence instead of treating lemmatization and stop words as part of the pre-processing pipeline, one would want to treat them as hyper parameters in the tuning step.

After applying lemmatization and stop words removal in our model, it does not improve the performance significantly.

TF-IDF

Term frequency inverse document frequency (TF-IDF) model is the next model that we explore. It is built base on bag of words model but with improvement.

Term frequency refers to the frequency that a word appears in one document, while document frequency refers to how many documents that word appears in out of the total number of documents. The transformed matrix contains statistics of term frequency times inverse log document frequency for every word.

In other words, if a word appears too frequent in different documents, the TF-IDF statistic tend to be low in value. While the model gives more weight to moderately “rare” words which appears in not too many documents but still an important word for some documents.

Our best model with TF-IDF is a bigram model. I am not going to elaborate on n-grams here but you can google it for interest. For those that are not familiar with n-grams, you can understand bigram as word fragments of two.

Predicted labels v.s. True labels: TF-IDF bigram model

The new model’s accuracy improves by 20% compare to our first model. It can detect the negative comments more effectively compare to our bag of words model. However, we realized for a bi-gram TF-IDF model, the number of features is numerous. There are 50,000 features used in our example.

This is when latent semantic analysis (LSA) comes to rescue. The idea behind the LSA technique is simple: it compresses the useful information of the TF-IDF matrix into lower dimension at the expense of some information loss (“noise”).

We then apply LSA on the TF-IDF bigram model and wait for magic to happen:

Predicted labels v.s. True labels: TF-IDF bigram + LSA model

Voila! The accuracy is 100% now with simple logistic regression and the features reduce from over 50,000 to only 300, less than 1% of the original number of features.

TF-IDF combined with LSA gives quite a decent prediction but we are feeling greedy to explore another feature extraction model.

Text Embedding

As mentioned, LSA technique keep important information while removing “noise” from the model. Text embedding on the other hand does it differently. This technique transforms each word to a vector. Using SpaCy’s text embedding, it returns document vectors which are average of all the words in a document.

Let’s see if text embedding can improve our model:

Predicted labels v.s. True labels: Text embedding model

The new model is more sensitive to negative comments and the prediction accuracy is not as ideal as TF-IDF model. The finding is a bit surprising as the text embedding model does not work as well as the TF-IDF model. After reading some reference, we understand that text embedding could be too complicated for a simple data set like ours. It would work better for big data applications.

Conclusion

Model results using our test example

TF-IDF with LSA applied appears to be our winner model and our test validation score also confirmed the same. It is effective for our task but the result is difficult to interpret. Text embedding has the same problem which make the model difficult to interpret intuitively. Bag of words model is the easiest to understand among all models but not an effective model to use in our case.

Model performance using customer review data test set

Our best F1 score achieved in this project is 0.931 using TF-IDF. F1 score is a measure to reflect model performance. An ideal model gives an F1 score of 1 and the worst model 0.

I have also made an app demonstrating what the model is capable of with Flask and JavaScript.

Flask app demonstration

The final model chosen is quite accurate detecting consumer sentiment but still with some noise. It can be improved by further feature processing or improved with better machine learning models like deep learning models.

# Discussion

Now we have taught the computer to read human reviews, what’s next?

Hope you enjoyed reading the article.