Web-Scraping and Sentimental Analysis on Amazon Reviews
By: Rajavi Mehta
Tags: Pre Processing, Scraping Amazon, Classification, NLP, Sentiment Analysis

Scraping reviews of an infamous product of Amazon: The Instapot and performing Sentimental Analysis on the reviews

The tools used for this implementation were:

Python: v3.8

IDE: PyCharm

Website : Amazon.com

Natural Language Processing

Natural Language Processing, also known as NLP, is an interesting branch of Artificial Intelligence.It allows processing of text, unlike other mainstream Machine Learning algorithms, and makes it machine understandable.

Web Scraping

Web scraping is used to extract data from the website and store it either locally or in a database like SQLite, MongoDB, etc.

L et’s dive into the implementation then-

First and foremost, import all the modules:

import numpy as np

import pandas as pd

import nltk

import string

import re

from sklearn.ensemble import RandomForestClassifier

from sklearn.metrics import confusion_matrix, accuracy_score, classification_report

from sklearn.neighbors import KNeighborsClassifier

from xgboost import XGBClassifier

from sklearn.linear_model import LogisticRegression

from sklearn.naive_bayes import MultinomialNB

from sklearn.feature_extraction.text import CountVectorizer

from nltk.tokenize import word_tokenizeThen moving ahead to the first step, i.e., the web scraping part:

Web scraping was done using Scrapy, which is a framework famously used for web-crawling and extracting the data.

Below is the items.py file which is used to declare the field that will be used later

import scrapy class ReviewsItem(scrapy.Item):

stars = scrapy.Field()

comments = scrapy.Field()

pass

Next, we create the spider file that will contain all the details and which will actually perform the scraping of the data.

First we save the name of the spider and set the URL that is to be scraped

import scrapy



class Reviews(scrapy.Spider):



# Spider name

name: str = 'amazon'

myBaseUrl = "https://www.amazon.com/product- reviews/B00FLYWNYQ/ref=cm_cr_arp_d_viewopt_sr?pageNumber="

As you can see, there is no page number defined at the end of the url. That is because there are more than one pages that contain the data that we have to scrape. Therefore, we will write a loop that creates the urls and stores in a list so that all of the pages can be scraped.

urls = []



# Creating list of urls to be scraped by appending page number at the end of base url

for i in range(1, 501):

urls.append(myBaseUrl + str(i))

Next, we write the parse function that defines scrapes the data using the css selectors

def parse(self, response):



# Get the Review List

data = response.css('#cm_cr-review_list')



# Get the Ratings

stars = data.css('.review-rating')



# Get the users Comments

comments = data.css('.review-text')

The data is then returned and stored in a dictionary format

count = 0

# combining the results

for review in stars:

yield {

'Rating': ''.join(review.xpath('.//text()').extract()),

'Comment': ''.join(comments[count].xpath(".//text()").extract()).strip()

}

count = count + 1

This data can be stored in a csv file by using the command:

scrapy crawl amazon -o reviews.csv

This command runs the spider ‘amazon’ and yields the data scraped by the spider and stores the collected data into a ‘reviews.csv’ file. The ‘-o’ is to indicate that we want an ‘output’ in the form of ‘reviews.csv’.

Amazon.com is a very extensive website with millions of pages and the content like reviews is public. But it stores confidential information like the customers personal data etc. So it may not allow to scrape the data if scraping is performed multiple times. To get over that problem, we can write a simple code in the settings.py file which will allow the scraping of the comments over and over again without throwing any error.

DOWNLOADER_MIDDLEWARES = {

'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,

'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,

}

This code let’s one overcome the above problem by creating a false image for amazon.com that there are multiple user agents scraping the data so it does not block the scraping.

By performing the above, we are done with the scraping part and now we have a .csv file containing all the data.

Now, we will analyse the file and evaluate it.

Let’s begin the analysis part!

Load the dataset and check its contents.

# Load the dataset

dataset = pd.read_csv('reviews.csv')



# Checking the contents of the dataset

print('Dataset size:', dataset.shape)

print('Columns are:', dataset.columns)

print(dataset.head())

DATA PREPROCESSING

Now we will start with the data pre processing part that makes it ready to be fed in a model.

The below code splits the Rating column into a column containing the ratings of the customers and excluding the rest of the words.

# Removing extra content from the rating column and keeping only the relevant part - rating

print('Splitting the Ratings column to remove the extra content')

dataset[["Rating", "out", "of", "5", "stars"]] = dataset["Rating"].str.split(" ", expand = True)

print('Columns are:', dataset.columns)

dataset = dataset.drop(dataset.columns[[2, 3, 4, 5, 6, 7]], axis=1)

dataset['Rating'] = pd.to_numeric(dataset['Rating'])

Next, we can check the polarity, subjectivity using the TextBlob module.

Polarity: Polarity is float which lies in the range of [-1,1] where 1 means positive statement and -1 means a negative statement.

Subjectivity: Subjective sentences generally refer to personal opinion, emotion or judgment whereas objective refers to factual information. Subjectivity is also a float which lies in the range of [0,1].

dataset['polarity'] = [TextBlob(sentence).sentiment.polarity for sentence in dataset['Comment']]

dataset['subjectivity'] = [TextBlob(sentence).sentiment.subjectivity for sentence in dataset['Comment']]

dataset['word_list'] = [TextBlob(sentence).words.lemmatize() for sentence in dataset['Comment']]

The most popular techniques for pre processing the textual data are as follows:

· Lowercase: Convert everything to lowercase for uniformity and avoid duplication of the words in capital and lower case form

# Converting all text to lowercase

print('Convert text to lower case')

dataset['Comment'] = dataset['Comment'].str.lower()

print(dataset.head())

· Remove punctuation: Removal of punctuation marks as they are not words

# Removing the punctuation marks

print('Remove Punctuation')

print(string.punctuation)



def remove_punc(text):

text = "".join([char for char in text if char not in string.punctuation])

text = re.sub('[0-9]+', '', text)

return text



dataset['Comment'] = dataset['Comment'].apply(lambda x: remove_punc(x))

· Remove numbers: as the name suggests, it is done to remove the numbers from the text

# Removing the numbers from the text

print('Removing Numbers from the text')

dataset['Comment'] = dataset['Comment'].str.replace('\d+', '')

print(dataset.head())

·Tokenizing: Separating words from a sentence and storing them as separate entities

# Tokenizing the words (Separating words from a sentence)

print('Tokenization of words')

comments = dataset['Comment']

dataset['Comment'] = comments.apply(word_tokenize)

print(dataset.head())

· Remove stopwords: Stopwords are words that do not add any meaning to the sentiments, for eg. A, the, that, there etc.

# Removing unnecessary words - stopwords

print('Removing stopwords')

stopword = nltk.corpus.stopwords.words('english')

print(stopword)





def remove_stopwords(text):

text = [word for word in text if word not in stopword]

return text





dataset['Comment'] = dataset['Comment'].apply(lambda x: remove_stopwords(x))

· Stemming: This is done to convert the words to their root form

# Performing Stemming

print('Performing Stemming')

st = nltk.PorterStemmer()



def stemming(text):

text = [st.stem(word) for word in text]

return text



dataset['Comment'] = dataset['Comment'].apply(lambda x: stemming(x))

·A better form of stemming is called lemmatization. It converts the words to its root form and in such a way that the sentiment can be understood from them

print('Performing Lemmatization')

wn = nltk.WordNetLemmatizer()



def lemmatizer(text):

text = [wn.lemmatize(word) for word in text]

return text



dataset['Comment'] = dataset['Comment'].apply(lambda x: lemmatizer(x))

·Next, we join the remaining words to form a sentence and call it a corpus

# Forming a sentence from the remaining words

print('Joining the words to form a string/sentence')

corpus = []

for i in range(0, 5000):

review = ' '.join(dataset['Comment'][i])

corpus.append(review)



print("Corpus", corpus)

·Next, we classify the reviews as positive/negative/neutral based on ratings

# Classifying Reviews as positive/negative/neutral based on ratings

print("Classifying Reviews as positive/negative/neutral based on ratings")

classify = []

for i in range(0, 5000):

if dataset['Rating'][i] > 3.0: # Posiitve reviews

classify.append(1)

elif dataset['Rating'][i] < 3.0: # Negative reviews

classify.append(-1)

else:

classify.append(0) # Neutral reviews



print(classify)

dataset['Classify'] = classify

·Applying CountVectorizer

# Vectorizing the words and counting the frequency for each

# Extracting the features

cv = CountVectorizer(max_features=1500)

X = cv.fit_transform(corpus).toarray()

y = dataset.iloc[:, 2].values

print(cv.vocabulary_)

Splitting the data into training set, testing set and validation set for feeding them to the models that we are going to build.

X is an array of 0s and 1s indicating the absence and presence of the word in the review from the corpus.

Y is an array that contains [-1, 0, 1] indicating the negative, neutral or positive review.

X_train = X[0:3001, :]

print("X_train: ", X_train)



X_test = X[3001:4501, :]

print("X_test: ", X_test)



y_train = y[0:3001]

print("y_train", y_train)



y_test = y[3001:4501]

print("y_test", y_test)



X_val = X[4501:5001, :]

print("X_val", X_val)



y_val = y[4501:5001]

print("y_val", y_val)

Implementing Logistic Regression algorithm now and calculating the accuracy score and getting the classification report:

lr = LogisticRegression(max_iter=5000) # Keeping it 5000 as there are 5000 rows

lr.fit(X_train, y_train)

y_pred_lr = lr.predict(X_test)

cm = confusion_matrix(y_test, y_pred_lr)

print(cm)

acc_lr = accuracy_score(y_test, y_pred_lr)*100

print('Accuracy from Logistic Regression classifier:', acc_lr)

print(classification_report(y_test, y_pred_lr))

Implementing algorithm now and calculating the accuracy score and getting the classification report:

xgb = XGBClassifier()

xgb.fit(X_train, y_train)

y_pred_xg = xgb.predict(X_test)

cm = confusion_matrix(y_test, y_pred_xg)

print(cm)

acc_xgb = accuracy_score(y_test, y_pred_xg)*100

print('Accuracy from XGBoost:', acc_xgb)

print(classification_report(y_test, y_pred_xg))

Implementing Random Forest algorithm now and calculating the accuracy score:

rf = RandomForestClassifier(n_estimators=50, random_state=42)

rf.fit(X_train, y_train)

y_pred_rf = rf.predict(X_test)

cm = confusion_matrix(y_test, y_pred_rf)

print(cm)

acc_rf = accuracy_score(y_test, y_pred_rf)*100

print('Accuracy from RandomForestClassifier:', acc_rf)

Implementing the KNN Classifier algorithm now and calculating the accuracy score and getting the classification report:

For KNN, we try to get the appropriate number of neighbours so there is least error possible. It is done below as shown:

for i in range(1, 11):

knn = KNeighborsClassifier(n_neighbors=i)

knn.fit(X_train, y_train)

err = knn.score(X_test, y_test)

error.append(err)

print("For " + str(i) + " Neighbours, error is: " + str(err))



i = error.index(min(error))

i += 1

print("Minimum error is for " + str(i) + " Neighbours")

Then developing the model and fitting the training set and predicting for the testing set for the i number of neighbours:

classifier = KNeighborsClassifier(n_neighbors = i)classifier.fit(X_train,y_train)

y_pred_knn = classifier.predict(X_test)

cm = confusion_matrix(y_test, y_pred_knn)

print(cm)

acc_knn = accuracy_score(y_test, y_pred_knn)*100

print('Accuracy from KNN:', acc_knn)

print(classification_report(y_test, y_pred_knn))

Implementing the Multinomial Naive Bayes algorithm now and calculating the accuracy score and getting the classification report:

mnb = MultinomialNB()

mnb.fit(X_train, y_train)

y_pred_mnb = mnb.predict(X_test)

cm = confusion_matrix(y_test, y_pred_mnb)

print(cm)

acc_mnb = accuracy_score(y_test, y_pred_mnb)*100

print('Accuracy from Naive Bayes:', acc_mnb)

print(classification_report(y_test, y_pred_mnb))

Below are the accuracy scores for the above implemented models, for the test set:

Below are the accuracy scores for the above implemented models, for the validation set:

As we can see, the accuracy score is better for validation set for the given models.

Conclusion: After analysing the above models, we come to a conclusion that XGBoost model gives the best accuracy when compared with the other models at 89.00%. Followed by Random Forest at 88.13%, Multinomial NB at 87.93% and Logistic Regression at 86.47%. The least accuracy is given by KNN at 63.53%. But for the validation set, Multinomial NB given the highest accuracy at 89.57% followed by XGBoost at 87.97%. Random Forest Classifier and Logistic Regression is almost same at 86.96% and 86.77% respectively. The lowest accuracy for the validation set is given by KNN at 81.96%. This could be possible as there are only 500 rows in the validation set and there are 1500 rows in the testing set. Hence, the difference in the accuracy. It could be improved by implementing hyperparameter tuning for the algorithms or giving user input to check accuracy of prediction.

For complete code reference: https://github.com/rajavi812/ADS_Assigments/tree/main/ADS_MP2