Performing Sentiment Analysis on Yelp Restaurant Reviews
Tags: Solutions, Machine Learning, Naturallanguageprocessing, Yelp Review

Data Analysis & Visualization

data frame after importing the ‘restaurant.csv’ file

As our goal is to find the sentiment of a given text using previous customer reviews and star ratings associated with them, it is clear that we will only need the two columns of this data frame; text and stars columns.

To better understand the relation between text and stars columns, firstly we examine the stars column; how many unique values does it contain? What is the distribution of each star rating? Which ratings are considered as the positive sentiment, which as the negative sentiment?

df.stars.unique()

#How many unique star values are there >>array([5, 4, 1, 3, 2], dtype=int64) df.stars.value_counts()

#shows the distribution for each star value

#Let's see the distribution of each star rating as a pie chart

#This way we can see that most of our data contains positive comments by costumers plt.figure(figsize=(8,8)) df[‘stars’].value_counts().plot.pie(autopct=’%1.1f%%’, startangle=60) plt.title(‘Yıldızların Dağılım Grafiği’)

x=df['stars'].value_counts()

x=x.sort_index() plt.figure(figsize=(10,6))

ax= sns.barplot(x.index, x.values, alpha=0.8)

plt.title("Star Rating Distribution")

plt.ylabel('count')

plt.xlabel('Star Ratings') rects = ax.patches

labels = x.values for rect, label in zip(rects, labels):

height = rect.get_height()

ax.text(rect.get_x() + rect.get_width()/2, height + 5, label, ha='center', va='bottom') plt.show();

We have used three different methods to showcase the distribution of star ratings. As we can see there are 5 different rating values from 1 to 5.

So we can safely assume that ratings 4 and 5 represent positive sentiment whereas 1 and 2 represent negative, this grouping leaves rating number 3 behind so we will label it as the neutral sentiment.

To form the newly decided column we use loc to select the groups we want and create a column named ‘ sentiment’ after that we assign the sentiment grouping values.

df.loc[df['stars'] == 3, 'sentiment'] = 'neutral'

df.loc[df['stars'] < 3, 'sentiment'] = 'negative'

df.loc[df['stars'] > 3, 'sentiment'] = 'positive'

Language Detection

After labeling the sentiment types, it is time for language detection. Since our data is taken from a global dataset some of the user reviews are not in English, at this point we can handle this process in two ways; one is detecting the non-English languages and translating the texts using TextBlob, the other is simply removing the non-English languages. For this solution, we will follow the second one.

Like we have stated above, we will only need to deal with ‘stars’, ‘sentiment’, and ‘text’ columns so to simplify the data frame we only select those columns.

new_df= df[['stars', 'sentiment','text']]

There are many different libraries and ways to detect languages such as TextBlob, Polyglot, chardet, and langid but what we will use today is langdetect, a python library that supports 55 languages.

$ pip install langdetect

#install the library from langdetect import detect

#import detect function from langdetect nonen=new_df[new_df['text'].apply(detect)!='en']

#nonen is the dataframe of non-english user reviews

Select all texts that are English and create a new data frame called yelp. Yelp is the data frame we will work with from now on.

yelp=new_df[new_df['text'].apply(detect)=='en']

#yelp is the data frame we will continue to work on.

#yelp only contains English reviews

Removing Characters

yelp['text']=yelp['text'].str.lower()

#change all strings to be lower yelp['text']=yelp['text'].str.replace('[^\w\s]','')

#get rid of unwanted characters such as punctuation marks yelp['text']=yelp['text'].str.replace('\d+','')

#removing numerals yelp['text']=yelp['text'].str.replace('

',' ').str.replace('\r','')

Calculating Polarity and Subjectivity

Sentiment analysis is the process of determining the emotion of a given text whether it is positive or negative, or neutral.

The sentiment function of TextBlob helps us understand the underlying emotion of a text by returning two properties, polarity, and subjectivity.

Polarity returns a float value that is in the range of [-1,1] where 1 means a positive statement and -1 means a negative statement. Subjectivity is also a float value that is in the range of [0,1]. 0 means a personal opinion and 1 means factual information.

yelp['polarity'] = yelp['text'].apply(lambda x: TextBlob(x). sentiment)

#applt textblob sentiment to yelp text column

#and assign it to a new column named polarity sentiment_series = yelp['polarity'].tolist()



yelp[['polaarity','subjectivity']]=pd.DataFrame(sentiment_series,

index=yelp.index)

yelp.drop('polarity', inplace=True, axis=1) #split the list into two and create two new columns

#assign the return values of sentiment function;

#polarity and subjectivity to those columns

Creating Word Cloud

Word cloud is a visualization tool used for text-based data. It helps pinpoint the frequency or the importance of each word of the text data. The basic logic of a word cloud is; the more a specific word appears in a text, the bigger and bolder it appears in the word cloud.

There are three steps to construct a word cloud

Extract the review

Create and generate a word cloud image

Display the cloud using matplotlib

from wordcloud import WordCloud wc = WordCloud( background_color="white", colormap="Dark2",

max_font_size=150, random_state=42)

The Word cloud function takes strings as a parameter and since we want to find the word clouds for both the positive and negative reviews, we select text values by the star value and combine all the data in a single string format using pandas ‘join’ method.

text_5_star=yelp[yelp['stars']>3]

text_5_star text_5_star_review = " ".join(review for review in text_5_star.text)

text_5_star_review text_1_star=yelp[yelp['stars']<3]

text_1_star text_1_star_review = " ".join(review for review in text_1_star.text)

text_1_star_review