Practical ML Part 2: NLP on Amazon Alexa Reviews
By: Mariana Almeida
Tags: Machine Learning, Kaggle, Data Science, NLP

A project using NLP techniques to evaluate client’s reviews

This is the second post of a series in which I will implement machine learning models to put into practice the knowledge gained by taking Machine Learning courses. You can find the first post of the series here.

The dataset for the study shown in this post can be found on this link. The full notebook for this analysis can be found here.

1 — Getting to know the data

Loading the dataset:

We have 5 variables to consider on 3150 reviews.

The rating gave by the clients on the reviews, which is an integer number from 1 to 5.

gave by the clients on the reviews, which is an integer number from 1 to 5. The date of the review, an object which will further be transformed into datetime.

of the review, an object which will further be transformed into datetime. Variation of the product, which is a characteristic differing the models of the product.

of the product, which is a characteristic differing the models of the product. The full text of the reviews (verified reviews).

Most of the reviews are positive, being:

Statistics for the whole dataset:

Statistics for the positive reviews dataset:

Statistics for the negative reviews dataset:

From this initial analysis we can gather that:

Ratings from 3 to 5 are classified as positive reviews and from 1 to 2 as negative reviews

Amazon Alexa has very good reviews in general, being classified as positive 91,8% (2893 reviews).

There is a prevalence of ratings equal to 5 in the reviews.

2 — Analyzing the text from the reviews

The more the characters more words we will have for our sentiment analysis. Longer reviews are, in general, more accurate as the customer explains in detail the rating he is giving. It is therefore important to check how long the reviews are in our dataset.

Separating the dataset in two — one for positive reviews (feedback=1) and one for negative reviews (feedback = 0) — we get the following statistics, now with the information for the length:

Statistics for the positive reviews dataset:

Statistics for the negative reviews dataset:

We can conclude that:

Most of the reviews have around 300 characters.

The shortest review has 1 character and the longest review has 2851 characters.

On average, reviews are 132 characters long.

The length of the review differs, on average, if the review is positive or negative. Positive reviews are, on average, 124 characters long and negative reviews are, on average, 215 characters long.

We may conclude that, on average, people who are unhappy with the product tend to write longer reviews.

A positive review is easier to write because you can only say ‘this is a great product’ and give a high rating. When writing a negative review, the person may want to explain the reasons why he or she is not happy with the product and this requires a longer text.

Does variation of the product impact the rating given on the review?

There are 16 variations of the product in this dataset:

Plotting the variations against the ratings in a barplot we can check whether the variation of the product impacts on the reviews:

All variations had average ratings higher than 4 so we can conclude that the variation does not impact on the satisfaction of the clients with the product.

3 — Transforming the data and cleaning

Checking the size of the data to be treated

Joining all sentences in the reviews in a separate object we get a resulting object with 419,105 characters.

From these characters:

363,531 characters are in the positive reviews (86,7%)

55,573 characters are in the negative reviews (13,3%)

Wordcloud before cleaning the data

In a wordcloud, the most important words in the dataset (i.e. the words which appear more frequently) are represented in bigger characters, proportional to their frequency in the data. This is a quick visualization technique which should be used carefully. A wordcloud shouldn’t be the only outcome of an NLP study but one way to visualize the information.

Cleaning and transforming the dataset

Our analysis will take into account the words in the reviews and the feedbacks classification (positive — 1 or negative — 0). The other columns will be deleted (date, rating and length of reviews). The resulting data frame is as follows:

The ‘variation’ column must be treated for the machine learning model. Using one hot encoding we have the following numpy array:

X_cat = reviews[['variation']]

onehotencoder = OneHotEncoder()

X_cat = onehotencoder.fit_transform(X_cat).toarray() X_cat.shape (3150, 16)

Therefore, we have an X matrix containing 16 columns (the different variations of the products) and 3150 rows.

Next steps are transforming the X matrix into a data frame and concatenate it with the original data set containing only the verified reviews:

Working on the verified reviews column

To clean and prepare the verified reviews data for a machine learning model, three steps were taken:

Remove punctuation

Remove stop words such as is, are, what…

Tokenization, which means breaking down text content into its parts, which can be words, characters or sub-words.

Removing punctuation is a clear requirement, as the computer understands characters and punctuation is not useful information to perform sentiment analysis.

Stop words are also unnecessary information as they do not express any sentiment. Removing stop words also improves processing time.

Tokenizing is an important step in NLP in which the data is transformed for processing.

To implement the processes described above, some libraries must be imported:

String, for removal of punctuation.

import string

string.punctuation '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'

Scikit-learn’s Count Vectorizer to tokenize the data.

from sklearn.feature_extraction.text import CountVectorizer

NLTK for removing stopwords.

import nltk

from nltk.corpus import stopwords print(stopwords.words('english')) ['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]

Creating a function to implement the removal of punctuation and stopwords in one batch:

def cleaning(message):

text = [char for char in message if char not in string.punctuation]

text = ''.join(text)

text = [word for word in text.split() if word.lower() not in stopwords.words('english')]

return text reviews_clean = reviews['verified_reviews'].apply(cleaning)

To finish this part of the process, the data will be tokenized:

vectorizer = CountVectorizer(analyzer=cleaning) reviews_countvectorizer = vectorizer.fit_transform(reviews['verified_reviews']) reviews_countvectorizer.shape (3150, 5211)

The reviews_countvectorizer contains the data after tokenization and has the following dimensions: 3150 rows and 5211 columns. Each of these columns is a token extracted from the data.

4 — Preparing the model for Machine Learning

Now, the verified reviews column from the original dataset can be deleted, resulting in a data frame with 3150 rows and 17 columns (the 16 variations of the product and the feedback column):

reviews.drop(['verified_reviews'], axis = 1, inplace=True)

The count_vectorizer object is also transformed into a dataframe, with 3150 rows and 5211 columns

reviews2 = pd.DataFrame(reviews_countvectorizer.toarray())

reviews2

In our model the input (X matrix) will be the tokens (reviews2 dataset) and the output (y matrix , what we want to predict) will be the original dataset containing only the feedback column:

X = reviews2 y = reviews['feedback']

5 — Training and Evaluating the Model

Training the Model

Logistic regression will be used to predict the feedback from the reviews. Before training, we must split the data into test sets and training sets:

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)

The test size considered in the final attempt was of 20%. The first attempt considered a 15% split but the results were better with 20%.

Now, implementing logistic regression on the training data:

logistic = LogisticRegression()

logistic.fit(X_train, y_train)

Evaluating the model

For the evaluation, some scikit-learn modules were imported:

Classification report, displaying the precision, recall , f1-score and support for the model.

Accuracy score, giving the number of the correct predictions as a ratio of all predictions made by the model.

Confusion matrix to illustrate the accuracy of the model showing the wrong and right predictions for each label.

accuracy_score(y_test, y_pred) 0.946031746031746

The model has an accuracy of almost 95%, which is a good overall result. Now, analyzing the classification report:

print(classification_report(y_test, y_pred)) precision recall f1-score support



0 0.89 0.33 0.48 48

1 0.95 1.00 0.97 582



accuracy 0.95 630

macro avg 0.92 0.66 0.73 630

weighted avg 0.94 0.95 0.93 630

Each of the information presented has the following meaning:

Precision — percentage of the results which are relevant.

— percentage of the results which are relevant. Recall — percentage of total relevant results correctly classified by the model.

— percentage of total relevant results correctly classified by the model. f1-score — weighted average of precision and recall.

— weighted average of precision and recall. Support — the number of observations of that class in the training set

The model is having difficulty in identifying the negative reviews — we can see that from the recall result for class 0: only 33% of the registers are correctly estimated. We must remember that we are dealing with a very unbalanced dataset — most of the reviews are positive reviews.

Further analysis should be considered for this model. Some techniques for dealing with unbalanced datasets could be considered. This will be covered in a future post on this project.