SENTIMENTAL ANALYSIS USING VADER. interpretation and classification of... | by Aditya Beri | Towards Data Science

Sentiment analysis is a text analysis method that detects polarity (e.g. a positive or negative opinion) within the text, whether a whole document, paragraph, sentence, or clause.
Sentiment analysis aims to measure the attitude, sentiments, evaluations, attitudes, and emotions of a speaker/writer based on the computational treatment of subjectivity in a text.
Though it may seem easy on paper, Sentiment Analysis is a tricky subject. A text may contain multiple sentiments all at once. For instance,
“The acting was good , but the movie could have been better”
The above sentence consists of two polarities!!!
VADER ( Valence Aware Dictionary for Sentiment Reasoning) is a model used for text sentiment analysis that is sensitive to both polarity (positive/negative) and intensity (strength) of emotion. It is available in the NLTK package and can be applied directly to unlabeled text data.
VADER sentimental analysis relies on a dictionary that maps lexical features to emotion intensities known as sentiment scores. The sentiment score of a text can be obtained by summing up the intensity of each word in the text.
For example- Words like ‘love’, ‘enjoy’, ‘happy’, ‘like’ all convey a positive sentiment. Also VADER is intelligent enough to understand the basic context of these words, such as “did not love” as a negative statement. It also understands the emphasis of capitalization and punctuation, such as “ENJOY”
We won’t try to determine if a sentence is objective or subjective, fact or opinion. Rather, we care only if the text expresses a positive, negative or neutral opinion.
We’ll also try to aggregate all of the sentences in a document or paragraph, to arrive at an overall opinion.
We won’t try to perform a fine-grained analysis that would determine the degree of positivity/negativity. That is, we’re not trying to guess how many stars a reviewer awarded, just whether the review was positive or negative.
First, consider the text being analyzed. A model trained on paragraph-long reviews might not be effective. Make sure to use an appropriate model for the task at hand.
Next, decide the type of analysis to perform. Some rudimentary sentiment analysis models go one step further, and consider two-word combinations, or bigrams. We will be going to work on complete sentences, and for this we’re going to import a trained NLTK lexicon called VADER.
For this model you can use a variety of datasets like amazon reviews, movie reviews, or any other reviews for any product.
import nltknltk.download('vader_lexicon')from nltk.sentiment.vader import SentimentIntensityAnalyzersid = SentimentIntensityAnalyzer()
VADER’s SentimentIntensityAnalyzer() takes in a string and returns a dictionary of scores in each of four categories:
negative
neutral
positive
compound (computed by normalizing the scores above
Let us analyze some random statements through our sentimental analyzer
a = 'This was a good movie.'sid.polarity_scores(a)OUTPUT-{'neg': 0.0, 'neu': 0.508, 'pos': 0.492, 'compound': 0.4404}a = 'This was the best, most awesome movie EVER MADE!!!'sid.polarity_scores(a)OUTPUT-{'neg': 0.0, 'neu': 0.425, 'pos': 0.575, 'compound': 0.8877}
import numpy as npimport pandas as pddf = pd.read_csv('../TextFiles/reviews.tsv', sep='\t')df.head()
df['label'].value_counts()OUTPUT-neg    5097pos    4903Name: label, dtype: int64
This step to clean any blank spaces within the reviews.
# REMOVE NaN VALUES AND EMPTY STRINGS:df.dropna(inplace=True)blanks = []  # start with an empty listfor i,lb,rv in df.itertuples():      if type(rv)==str:                    if rv.isspace():                    blanks.append(i)     df.drop(blanks, inplace=True)
Now we’ll add columns to the original DataFrame to store polarity_score dictionaries, extracted compound scores, and new “pos/neg” labels derived from the compound score. We’ll use this last column to perform an accuracy test. The reviews in this method will be classified into negative, positive and, neutral ratio.
df['sc ores'] = df['review'].apply(lambda review: sid.polarity_scores(review))df.head()
Now will call out compound as a separate column and all values greater than zeroes will be considered a positive review and all values less than zero would be considered as a negative review.
df['compound']  = df['scores'].apply(lambda score_dict: score_dict['compound'])df.head()
df['comp_score'] = df['compound'].apply(lambda c: 'pos' if c >=0 else 'neg')df.head()
So now we have got a complete analysis of every review as either positive or negative.
Now let us pass some new reviews to test how our model performs!
# Write a review as one continuous string (multiple sentences are ok)review = 'The shoes I brought were amazing.'# Obtain the sid scores for your reviewsid.polarity_scores(review)OUTPUT-{'neg': 0.0, 'neu': 0.513, 'pos': 0.487, 'compound': 0.5859}review='The mobile phone I bought was the WORST and very BAD'# Obtain the sid scores for your reviewsid.polarity_scores(review)OUTPUT-{'neg': 0.539, 'neu': 0.461, 'pos': 0.0, 'compound': -0.8849}
The results of VADER analysis don’t seem to be only remarkable but also very encouraging. The results show the advantages which will be attained by the utilization of VADER in cases of web sites wherein the text data could be a complex mixture of a range of text.
There are two of my other articles published in Towards Data Science publication on the related topics for this blog. Do have a read on those for better understanding in Natural Language Processing
Stemming vs Lemmatization — https://link.medium.com/JWpURpQjt6
Word vectors and Semantics — https://link.medium.com/tuVCswhYu6