Clothes reviews analysis with NLP — Part 1 | by Valentina Alto | Towards Data Science

Natural Language Processing (NPL) is a field of Artificial Intelligence whose purpose is finding computational methods to interpret human language as it is spoken or written. The idea of NLP goes beyond a mere classification task that could be carried on by ML algorithms or Deep Learning NNs. Indeed, NLP is about interpretation: you want to train your model not only to detect frequent words, but also to count them or to eliminate some noisy punctuations; you want it to tell you whether the mood of the conversation is positive or negative, whether the content of an e-mail is mere publicity or something important, whether the reviews about thriller books in last years have been good or bad.
The good news is that, for NLP, we are provided with interesting libraries, available in Python, that offer a pre-trained model able to inquire about written text. Among those, I’m gonna be using Spacy and NLTK.
In this article, I’m going to analyze the Women’s Clothing E-Commerce dataset which contains text reviews written by customers (available here).
Here it is the description of the dataset:
This dataset includes 23486 rows and 10 feature variables. Each row corresponds to a customer review, and includes the variables:Clothing ID: Integer Categorical variable that refers to the specific piece being reviewed.Age: Positive Integer variable of the reviewers age.Title: String variable for the title of the review.Review Text: String variable for the review body.Rating: Positive Ordinal Integer variable for the product score granted by the customer from 1 Worst, to 5 Best.Recommended IND: Binary variable stating where the customer recommends the product where 1 is recommended, 0 is not recommended.Positive Feedback Count: Positive Integer documenting the number of other customers who found this review positive.Division Name: Categorical name of the product high level division.Department Name: Categorical name of the product department name.Class Name: Categorical name of the product class name.
So let’s import and inspect our dataset:
import pandas as pddf = pd.read_csv('Womens Clothing E-Commerce Reviews.csv')df.head()
Now let’s clean it, dropping NaN values and removing useless columns, as well as clean the text.
df.dropna(inplace=True)df.reset_index(drop=True, inplace=True)import refor i in range(len(df)):    #print(i)    df['Review Text'][i] = df['Review Text'][i].replace("'s", " is").replace("'ll", " will").replace("'ve", " have").replace("'m", " am").replace("\'", "'")    df['Review Text'][1]df = df.drop('Unnamed: 0', 1)df.head()
The idea is extracting the sentiment of each review and see whether it is consistent with the recommendation indicator (1 if the review recommends the item, 0 otherwise).
To do so, we will use the Vader module available in the NLTK package (you can find the official documentation here). But first, let’s preprocess our text and extract Lemmas:
#importing necessary librariesimport nltkimport en_core_web_smimport spacynlp = spacy.load("en_core_web_sm")from nltk.corpus import stopwordsdef lemmatization(df):    df["Lemmas"] = [" ".join([token.lemma_ if token.lemma_ != "-        PRON-" else token.text.lower()     for sentence in nlp(speech).sents for token in sentence if     token.pos_ in {"NOUN", "VERB", "ADJ", "ADV", "X"} and      token.is_stop == False]) for speech in df.text]    df["Lemmas"] = [" ".join([token.lemma_ if token.lemma_ != "-PRON-"                 else token.text.lower() for sentence in                 nlp(speech).sents for token in sentence if                 token.pos_ in {"NOUN", "VERB", "ADJ", "ADV", "X"}                 and token.is_stop == False]) for speech in                 df['Review Text']]df.head()
Now let’s perform sentiment analysis on the Lemmas column:
from nltk.sentiment.vader import SentimentIntensityAnalyzersid = SentimentIntensityAnalyzer()sentiment=[]for i in range(len(df)):     ss = sid.polarity_scores(df.Lemmas[i])     sentiment.append(ss)    compound=[sentiment[i]['compound'] for i in range(len(sentiment))]df['compound']=compounddf.head()
The compound column contains scores ranging from -1 (very negative) to 1 (very positive). As a general rule, we attribute a ‘positive’ label to a text if its score is greater than 0.05, a ‘negative’ label if it is less than -0.05, and a ‘neutral’ one if it is between -0.05 and 0.05. However, for the purpose of this research question, we will only differentiate between positive and negative, using score=0 as threshold.
sent=[]for i in compound:    if i<0:        sent.append('negative')    else:        sent.append('positive')df['sentiment']=sentdf.head()
Having a first glimpse of the sentiment column, it seems that most of the reviews are positive, but it is consistent with the number of recommended items: out of 19661 reviews, 16087 of them end up with a recommendation.
Now let’s see how many recommended items are consistent with the sentiment of the review. Let me borrow for this task the statistical terminology of false-positive/negative and true positive/negative: I talk about false positive when the item is recommended to be purchased, yet the text review is has a negative sentiment.
#initializing a 2x2 matrix populated with zerosmat = np.zeros(4).reshape(2,2)true_neg=0true_pos=0false_neg=0false_pos=0for i in range(len(df)):    if df['sentiment'][i]=='negative' and df['Recommended IND'][i]==0:        true_neg+=1    elif df['sentiment'][i]=='positive' and df['Recommended IND'][i]==1:        true_pos+=1    elif df['sentiment'][i]=='positive' and df['Recommended IND'][i]==0:        false_neg+=1    elif df['sentiment'][i]=='negative' and df['Recommended IND'][i]==1:        false_pos+=1    mat[0][0]=true_negmat[0][1]=false_negmat[1][0]=false_posmat[1][1]=true_posmat
Let’s make it a bit prettier:
from pandas import DataFrameIndex= ['Not Recommended', 'Recommended']Cols = ['Negative', 'Positive']d = DataFrame(mat, index=Index, columns=Cols)
So out of the 16087 recommended items, only 227 of them are not consistent with the sentiment of the text. On the other side, we are not recommending 3154 items that have actually received a positive review. This kind of results might suggest a different approach towards the recommendation system since more items might be purchased if we look at the sentiment of reviewers.
Sentiment analysis plays a pivotal role in market research and similar fields. Indeed, keeping track of customers’ satisfaction and, most importantly, isolate those drivers which determine their sentiment towards items, could lead to winning marketing campaigns and selling strategies.
In this article I limited my research to the sentiment analysis, however this dataset offers multiple research questions, which will be covered in my next articles, so stay tuned!