Sentiment Analysis on Amazon Reviews | by Enes Gokce | Towards Data Science

Understanding the data better is one of the crucial steps in data analysis. In this study, I will analyze the Amazon reviews. The reviews are unstructured. In other words, the text is unorganized. Sentiment analysis, however, helps us make sense of all this unstructured text by automatically tagging it. Sentiment analysis helps us to process huge amounts of data in an efficient and cost-effective way.
This study in part of the bigger study. In order to check feature extraction and data cleaning part (previous step), you can check my previous posting. You can find all Python codes for this study here. In this study, I will:
Give a brief theoretical background about sentiment analysis
Perform sentiment analysis
Provide python codes for each step.
For performing sentiment analysis, we will use NLTK package of the Python. Christopher Manning says, “NLTK is sort of the Swiss Army Knife of NLP meaning that it’s not terribly good for anything. But it has a lot of basic tools.” For accessing Wordnet, it’s an easy solution.
WordNet is a large lexical database of English developed by the Princeton University. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations (Fellbaum, 1998). In other words, Wordnet can be described as online thesaurus. It tells you about word meanings and relationships between word meanings. Wordnet was first created in 1985, and still in improvement.
Wordnet can be obtained by:
import nltknltk.download('wordnet')
In this study, we will use two main sentiment classifiers:
1. Polarity
2. Subjectivity
The TextBlob package for Python is a convenient way to perform sentiment analysis. When calculating sentiment for a single word, TextBlob takes average for the entire text. For heteronym words, Textblob does not negotiate with different meanings. In the other words, only the most common meaning of a word in entire text is taken into consideration. For making all these modelling, Textblob uses WordNet Database.
Polarity is float which lies in the range of [-1,1] where 1 means positive statement and -1 means a negative statement. Figure 1 shows the distribution of polarity score in reviews. Most of the reviews are on positive side of the plot (Fig. 1).
# Create quick lambda functions to find the polarity of each review# Terminal / Anaconda Navigator: conda install -c conda-forge textblobfrom textblob import TextBlobdf['Text']= df['Text'].astype(str) #Make sure about the correct data typepol = lambda x: TextBlob(x).sentiment.polaritydf['polarity'] = df['Text'].apply(pol) # depending on the size of your data, this step may take some time.import matplotlib.pyplot as pltimport seaborn as snsnum_bins = 50plt.figure(figsize=(10,6))n, bins, patches = plt.hist(df.polarity, num_bins, facecolor='blue', alpha=0.5)plt.xlabel('Polarity')plt.ylabel('Number of Reviews')plt.title('Histogram of Polarity Score')plt.show();
In Figure 2, it can be observed that good reviews (Good reviews =1) have higher polarity compared to bad reviews. On the other hand, good reviews also have higher number of negative polarity reviews. This is an unbalanced data and number of good reviews are higher than bad reviews. Therefore, it is not much surprising to see a greater number of extreme values in this category.
plt.figure(figsize=(10,6))sns.boxenplot(x=’Good_reviews’, y=’polarity’, data=df)plt.show();
As we can see from this box plot,
we have some good reviews that has very low polarity (very negative)
Some bad reviews that has high polarity (positive statement)
Let’s check some of them:
df.loc[(df.polarity == 1 & (df.Good_reviews == 0))].Text.head(10).tolist()
When Table 1 is examined, it can be seen that some of the reviews are actually positive but somehow got bad review scores. Keep in mind that these are extreme case reviews, and it is not surprising to see that their rating doesn’t make much sense.
Punctuation vs Polarity: From Figure 3, we can see that when the value of punctuation is low, polarity is higher. A possible explanation for this is people who are paying more attention to punctuation tend to be more balanced in their product evaluation. Despite outliers, the average polarity score is almost a line, and it is around 0.25. This information is consistent with Figure 1. We can also see that there are extreme cases in both direction of the polarity (Fig. 3)
plt.figure(figsize=(15,8))df3= df.loc[df.upper <= 50]sns.boxenplot(x='upper', y='polarity', data=df3)plt.xlabel('Punctuation', fontsize=13)plt.ylabel('Polarity Score', fontsize=13)plt.title('Punctuation vs Polarity Plot', fontsize=15)plt.show();
Helpfulness vs Polarity: Figure 4 presents the relation between helpfulness and polarity in the Good Reviews category. There are interesting outliers. For example, some reviews have the lowest polarity (most negative) but have a good rating (good review is 1) and helpfulness is more than 3. This combination is a controversial case. When we look at these cases more closely, we can see that those reviews are not using negative words for the purchase (Table 2). Those negative expressions are for comparison with other purchases. For now, NLP methods are not doing great at handling this kind of usage of words.
plt.figure(figsize=(12,6))df_sub= df.loc[df.HelpfulnessNumerator <=30]sns.boxenplot(x='HelpfulnessNumerator', y='polarity', hue='Good_reviews', data=df_sub)plt.xlabel('Helpfulness Numerator', fontsize=13)plt.ylabel('Polarity Score', fontsize=13)plt.title('Helpfulness Numerator vs Polarity', fontsize=15)plt.show();
Subjectivity is used for individual sentences to determine whether a sentence expresses an opinion or not. In terms of subjectivity, textual information in the world can be broadly categorized into two main types: facts and opinions. Subjective sentences generally refer to personal opinion, emotion, or judgment whereas objective refers to factual information. Facts are objective expressions about entities, events, and properties. Opinions are usually subjective expressions that describe people’s sentiments, appraisals, or feelings toward entities, events, and their properties (Liu, 2010).
In sentiment analysis, subjectivity is also a float that lies in the range of [0,1]. When it is close to 0, it is more about facts. When subjectivity increases, it comes close to be an opinion. In the data set, the distribution of subjectivity scores for the reviews are similar to a normal distribution (Fig. 5). When we examined the relation between subjectivity, polarity, and Good Reviews features we can see that subjectivity and polarity shows a funneling pattern (Fig. 6). It can also be observed that low subjectivity score reviews are also neutral reviews in terms of polarity.
For creating subjectivity scores:
sub = lambda x: TextBlob(x).sentiment.subjectivitydf['subjectivity'] = df['Text'].apply(sub)df.sample(10)
Checking the distribution of the subjectivity score:
# Density Plot and Histogram of subjectivityplt.figure(figsize=(10,5))sns.distplot(df['subjectivity'], hist=True, kde=True,bins=int(30), color = 'darkblue',hist_kws={'edgecolor':'black'},kde_kws={'linewidth': 4})plt.xlim([-0.001,1.001])plt.xlabel('Subjectivity', fontsize=13)plt.ylabel('Frequency', fontsize=13)plt.title('Distribution of Subjectivity Score', fontsize=15)
plt.figure(figsize=(10,6))sns.scatterplot(x='polarity', y='subjectivity', hue="Good_reviews", data=df)plt.xlabel('Polarity', fontsize=13)plt.ylabel('Subjectivity', fontsize=13)plt.title('Polarity vs Subjectivity', fontsize=15)plt.show();
Figure 7 is a presentation of how polarity and subjectivity are affected by the rating of the reviews (Good review feature). While reading this plot, we need to keep in mind that the y-axis is in a very small range. We can see that the mean subjectivity score difference between the two groups is negligible. (You can find the codes for this plot on my GitHub repo)
In order to understand our data better, we need to check it from a different perspective. There are some reviews that can be considered as an extreme cases. For example, Table 3 presents 10 reviews that have the highest polarity (most positive sentiment) but the ‘good review’ value is 0, and the most subjective (opinion). These tweets are hard to score for sentiment analysis algorithms. It is not surprising that they have the most positive score (polarity =1).
df.loc[(df["Good_reviews"] == 0) & (df.polarity == 1 ) & (df.subjectivity ==1), "Text"].head(10).tolist()
In order to understand how the data is shaped and how the sentiment analysis works, let’s examine more reviews with different criteria (Table 4 and 5).
df.loc[(df["Good_reviews"] == 1) & (df.polarity == 1 ) & (df.subjectivity ==1), "Text"].sample(5).tolist()
df.loc[(df["Good_reviews"] == 1) & (df.polarity == -1 ) & (df.subjectivity ==1), "Text"].sample(5).tolist()
In conclusion, with this study, I tried to show how sentiment analysis works by applying it on Amazon review data. In the next study, I will show how to perform topic analysis with Latent Dirichlet Allocation (LDA) by explaining it step by step.
Happy analysis!
*Special thanks to my friend Tabitha Stickel for proofreading this article.
My further content recommendations for sentiment analysis:
Alice Zhao: https://youtu.be/xvqsFTUsOmc?t=4125
Calculating Polarity and Subjectivity: https://planspace.org/20150607-textblob_sentiment/
References:
Fellbaum, C. (1998). WordNet: An Electronic Lexical Database. Bradford Books. Guibon, G., Ochs, M., & Bellot, P. (2016, June). From emojis to sentiment analysis.
Liu, B. (2010). Sentiment analysis and subjectivity. Handbook of natural language processing, 2(2010), 627–666.
Stanford CS224N: NLP with Deep Learning