Design your own Sentiment Score. Sentiment Analysis in pandas | by Songhao Wu | Towards Data Science

Sentiment analysis is a commonly used NLP(natural language processing) technique to determine whether the text is positive, negative, or neutral. It has been frequently used to look at customer satisfaction based on review sentiment or serve as an additional perspective when we doing analysis on text data.
There are python packages available to calculate sentiment score directly like NLTK Vadar(The sample code is at the back). However, the main purpose of this article is to help you design and create your own sentiment score based on your logic.
You know what you are getting(because you design the logic of calculation of the sentiment score)It is flexible for you to change the logic based on your needs. (E.g. you can use positive and negative words list related to dining if you doing restaurant reviews sentiment analysis)You have multiple features for analysis when you used multiple logic to calculate the sentiment score
You know what you are getting(because you design the logic of calculation of the sentiment score)
It is flexible for you to change the logic based on your needs. (E.g. you can use positive and negative words list related to dining if you doing restaurant reviews sentiment analysis)
You have multiple features for analysis when you used multiple logic to calculate the sentiment score
Ok now let us get to the example!
Restaurant review data: The sample we are using is the restaurant review data from Yelp. The data can be downloaded from Kaggle. The output of the restaurant review can be used to assess customer satisfaction for different restaurants.
Positive and negative words list: This is the list of positive and negative words we found online related to restaurants. They exist as positive-words.txt and negative-words.txt. You can find a version online and continues to add to that based on the industry you are looking at.
Stop words: list of words that occur frequently but do not carry any sentiment value usually like ‘the’, ‘to’ etc. I am using stop words from nltk package because it is quite standard.
Import all the data to pandas data frame first.
import pandas as pd# Import datasample=pd.read_csv(file_path+'/train.csv')sample.columns=['rating','review']train['review_id']=train.indexpositive=pd.read_csv(file_path+'/positive-words.txt',header=None)positive.columns=['words']negative=pd.read_csv(file_path+'/negative-words.txt',header=None)negative.columns=['words']stop_words=stopwords.words('english')
Data pre-processing: Review usually contains many punctuation, numbers or stop words that are not useful for us to determine the sentiment.
The original review text would be stemmed(including lowercase and word stemming), tokenised (to be broken into words), remove number and stop-words.
Below is an example of how do we get the final bucket of words from each step of pre-processing. (Mobile users may need to zoom in to see the steps clearly)
import reimport nltknltk.download('punkt')nltk.download('stopwords')from nltk.corpus import stopwordsfrom nltk.stem.porter import *stemmer = PorterStemmer()#preprocess function stem_words=lambda x: stemmer.stem(x)break_into_words = lambda x : re.findall("[a-zA-Z0-9]+", x)skip_stop_words = lambda x: [w for w in x if w not in list(stop_words)]skip_numeric = lambda x : [w for w in x if not w.isnumeric()]#apply pre-processing stepsprocessed_text=sample['review'].map(to_lower)\        .map(break_into_words)\        .map(skip_stop_words)\        .map(skip_numeric)sample['processed']=processed_text
Based on the positive and negative word list, you can generate the count of positive and negative words occurrence for each review. Please take note that this step would be slow because it is a multiple to multiple mapping. You can consider a switch to Hive Sparks for faster execution if your data size is very large.
#count number of positive wordspositive_words_count=processed_text.map(lambda x: len([w for w in x if w in list(positive['words'])]))#count number of negative wordsnegative_words_count=processed_text.map(lambda x: len([w for w in x if w in list(negative['words'])]))sample['positive']=positive_words_countsample['negative']=negative_words_count
Method 1: Calculate absolute score by using positive count-negative count
sample['sentiment_score_1']=sample['positive']-sample['negative']
Pros and Cons: This is a direct and simple calculation of how positive/negative the review is. The advantage is that it is straightforward and simple to understand. However, the score may favor longer reviews because they tend to have more counts of positive/negative words even they have the same sentiment.
For example, ‘excellent food and service but the background music can be improved’ and ‘I think the good part of this restaurant is that food is good and the waiter is friendly and I like their recipe design but the background music is bad, etc, etc’ may be of the same sentiment but the score for the second sentence is much bigger because of the longer text.
Method 2: Normalise the score by the length of the review
sample['normalized_sentiment_score']=(sample['positive']-sample['negative'])/(len(sample['processed'])
The logic here is to normalize the score by the length of the review because the problem arises from method 1. You can adjust the logic by dividing the count of positive words+count of negative words(then the score will range between -1 to 1).
Pros and Cons: The score takes into consideration the length of the review text but the number could be very small and hard to differentiate. You can choose to multiply all the scores by a fixed number for easier comparison.
Method 3: Calculate the ratio of positive and negative words count
sample['positive_negative_ratio']=sample['positive']/(sample['negative']+1)
Method 3 takes into consideration of polarity of positive and negative reviews. The ratio also partially normalized the length of the text because the count of positive and negative scores tend to increase together for a longer review. The range of the score ranges between 0–infinity and a score around 1 are considered neutral.
Pros and Cons: This is a more balanced way of calculation between normalization and interpretation. The drawback of this logic is that it would be harder for you to determine the range for neutral sentiments because the denominator has ‘+1’
fastest and easiest method — no text pre-processing needed :)
import nltknltk.download('vader_lexicon')from nltk.sentiment.vader import SentimentIntensityAnalyzeranalyser = SentimentIntensityAnalyzer()sample['review'].map(lambda x: analyser.polarity_scores(x))
We only explored positive and negative single words. However, there could be multiple words combination like ‘Must Try’ that we can put into a positive word list. The calculation would be bigger if we want to incorporate 2-gram or 3-gram phrases.Different positive words are given same score 1 but for example ‘excellent’ has a larger scale than ‘good’ and thus we can give a different score for different levels of words and then add up togetherDouble negative phrases are not captured here like ‘not bad’, this can be partially mitigated by the use of a 2-gram or 3-gram positive or negative words list.
We only explored positive and negative single words. However, there could be multiple words combination like ‘Must Try’ that we can put into a positive word list. The calculation would be bigger if we want to incorporate 2-gram or 3-gram phrases.
Different positive words are given same score 1 but for example ‘excellent’ has a larger scale than ‘good’ and thus we can give a different score for different levels of words and then add up together
Double negative phrases are not captured here like ‘not bad’, this can be partially mitigated by the use of a 2-gram or 3-gram positive or negative words list.
Personally, I prefer method 3 for sentiment score calculation. However, the three methods are just examples and there are unlimited new ways on how you can design & calculate your own sentiment score. You should design your logic based on your project needs. In addition, If you feel that you just want to have a sentiment score quickly, you can choose to use the NLTK Vadar method as shown above.
If you have a well-labeled dataset(with a real sentiment score), you can consider using the classification method instead. Refer to my another article below for more details