Sentiment Classification with Logistic Regression — Analyzing Yelp Reviews | by Dehao Zhang | Towards Data Science

You just opened up your own business a couple of weeks ago.
You want to gauge how customers feel about your products and services, so you go on social media platforms (Twitter, Facebook, etc.) to see what people have said.
Whoa! It seems that there are already thousands of posts about your business. You are excited but you soon realize that it is not feasible to read all of them by yourself.
You start to wonder: Is there a way to extract customers’ sentiment from textual information?
Why is sentiment analysis important?What is logistic regression?Which metric(s) should we use to evaluate models?How can a model understand text input?How do we know which text features are important?Can we further improve our model?What can we do next?
Why is sentiment analysis important?
What is logistic regression?
Which metric(s) should we use to evaluate models?
How can a model understand text input?
How do we know which text features are important?
Can we further improve our model?
What can we do next?
Sentiment analysis is a highly effective tool for a business to not only take a look at the overall brand perception, but also evaluate customer attitudes and emotions towards a specific product line or service [1].
This data-driven approach can help the business better understand the customers and detect subtle shifts in their opinions in order to meet changing demand.
This post serves as the second part of my exploration on the Yelp dataset. For more information about the dataset as well as some exploratory data analysis on its business data and tips data, please see my post below:
towardsdatascience.com
This post focuses on the review.json file within the Yelp dataset, which contains reviews written by Yelp users. In addition, each review includes a corresponding “star”, or rating that the user gives to the business, which can be used as a proxy for sentiment. The goal is to build a model that can classify the sentiment of the review (positive or negative) given the text data. Furthermore, we are interested in which text features are the most helpful predictors for this classification task.
Logistic Regression
In general, there are two different types of classification models: generative models (Naive Bayes, Hidden Markov Models, etc.) and discriminative models (Logistic Regression, SVM, etc.). Ultimately, both models try to compute p(class|features), or p(y|x). The key difference is that a generative model tries to model the joint probability distribution p(x,y) first and then compute the conditional probability p(y|x) using Baye’s Theorem, whereas a discriminative one directly models p(y|x). See Andrew Ng’s paper here for detailed discussion on the comparison between the two type of models.
In this post, I will examine a popular discriminative model — logistic regression. See this post for more details on its mathematical foundations (sigmoid function, cost function, decision boundary, etc.).
To see my full Python code, check out my Kaggle kernel or my Github page. Now let’s get started!
Peek at the Reviews
Let’s take out 1 million records from the review dataset for our analysis. The ‘text’ column would be our model inputs. let’s check out a random review with a positive sentiment (rating of 5.0):
"I love Deagan's. I do. I really do. The atmosphere is cozy and festive. The shrimp tacos and house fries are my standbys. The fries are sometimes good and sometimes great, and the spicy dipping sauce they come with is to die for. The beer list is amazing and the cocktails are great. The prices are mid-level, so it's not a cheap dive you can go to every week, but rather a treat when you do. Try it out. You won't be disappointed!"
As we are thinking about feature extraction, a couple of clues here can help us infer that this is positive sentiment, such as ‘love’, ‘cozy’, ‘die for’, ‘amazing’, and ‘won’t be disappointed’.
Let’s see one negative review (rating of 1.0):
"If I could give less than one star, that would have been my choice.  I rent a home and Per my lease agreement it is MY responsibility to pay their Pool Service company.  Within the last year they changed to PoolServ.  I have had  major issues with new techs every week, never checking PH balances, cleaning the filter, and not showing up at all 2 weeks in the past 2 months. I have had 4 different techs in the past 4 weeks.   I have emailed and called them and they never respond back nor even acknowledged my concerns or requests.  I cannot change companies but I'm required to still pay for lousy or no service.  Attached are a couple pictures of my pool recently due to one tech just didn't put any chlorine in it at all according to the tech who came the following week to attempt to clean it up.  Please think twice before working with these people.  No one wants to work with a business that doesn't return phone calls or emails."
Despite its length, we can still see that clues such as ‘less than one star’, ‘lousy’, ‘never respond back’, and ‘no service’ are useful predictors.
Let’s quickly confirm that there is no missing value in the data:
text     0.0stars    0.0dtype: float64
Convert Ratings into Positive and Negative Sentiments
Let’s plot the ratings distribution:
We can see that out of the 1 million reviews, almost half of them contain a rating of 5.0.
In this task, we want to classify all review texts into one of the two categories: positive sentiment and negative sentiment. Therefore, we will first need to transform the ‘stars’ value into the two categories. Here, we can treat ‘4.0’ and ‘5.0’ as positive sentiment and ‘1.0’ and ‘2.0’ as negative sentiment. We can treat ‘3.0’ as neutral or even treat each star as its own sentiment category, which would make it a multi-class classification problem. However, for the sake of a simply binary classification, we can take the ones with ‘3.0’ out.
We then encode positive sentiment as Class 0 and negative sentiment as Class 1. Since we know that we have more samples in Class 0 than Class 1, our baseline model can be one that simply labels every review as Class 0. Let’s check out the baseline accuracy:
0.74
Before we proceed, let’s spend some time on the evaluation metric.
Evaluation Metric
Quoted from Jason Brownlee in one of his posts, “A classifier is only as good as the metric used to evaluate it”[2].
Accuracy might not be the appropriate evaluation metric in every classification problem, especially when the class distribution is imbalanced, and when the business impacts of false positive and false negative are unequal. For example, in a credit card fraud detection problem, a baseline model which predicts every transaction to be non-fraud would have an accuracy of over 99.99%, but it does not mean that it is a proper model since the percentage of false negative would be 100%, and each false negative (fail to detect a fraud transaction) can have a much higher cost to the business and customers compared with a false positive.
In this task, Here are a few appropriate evaluation metrics:
Precision — TP/(TP+FP), meaning the proportion of points that model classify as positives are actually positives.Recall — TP/(TP+FN), meaning the proportion of actual positives that are correctly classified by the model.F1 score —the harmonic mean of precision and recall.
Precision — TP/(TP+FP), meaning the proportion of points that model classify as positives are actually positives.
Recall — TP/(TP+FN), meaning the proportion of actual positives that are correctly classified by the model.
F1 score —the harmonic mean of precision and recall.
Check out this post for more detailed discussion on these metrics.
In this task, we would use F1 score on the test set as the key evaluation metric.
Train-Test Split
Let’s set aside 30% of the data to be the test set, stratified by the class label.
train, test = train_test_split(df_reviews, test_size = 0.3, stratify = df_reviews['labels'], random_state = 42)
Text Preprocessing
In most text mining or NLP related tasks, cleaning text is a crucial step. Let’s first remove all non-letter characters, punctuations, and make sure all letters are in lower-cases. Later we will also evaluate the effect of removing stopwords and stemming/lemmatization.
Vectorization
For a model to be able to process text input, we would need to convert them into vectors. There are a few different ways to represent these text features and here are the most common ones: 1. Binary, e.g. whether the word “good” is present. 2. Count, e.g. how many times does the word “good” appear in this review, similar to the Bag of Word model in Naive Bayes. 3. TF-IDF, which is a weighted importance of each text feature relevant to the document (read more here).
Let’s try using binary representation of all unigrams (single word) first.
cv= CountVectorizer(binary=True, analyzer = text_prep, min_df = 10, max_df = 0.95)cv.fit_transform(train['text'].values)train_feature_set=cv.transform(train['text'].values)test_feature_set=cv.transform(test['text'].values)
I used sklearn’s CountVectorizer object to extract all the word features, and the word would be excluded if it appears in less than 10 reviews or more than 95% of the reviews.
Let’s check how many unique words are in our dictionary:
train_feature_set.shape[1]--------------------------------------------------------------------40245
There are about 40K unique words. As an example, let’s check the index of word ‘tasty’:
cv.vocabulary_['tasty']--------------------------------------------------------------------35283
Fit a LR Model
Now we are ready to fit our first logistic regression model using sklearn:
lr = LogisticRegression(solver = 'liblinear', random_state = 42, max_iter=1000)lr.fit(train_feature_set,y_train)y_pred = lr.predict(test_feature_set)print("Accuracy: ",round(metrics.accuracy_score(y_test,y_pred),3))print("F1: ",round(metrics.f1_score(y_test, y_pred),3))--------------------------------------------------------------------Accuracy:  0.955F1:  0.914
Let’s plot a confusion matrix to visualize the prediction results:
Visualize Feature Importance
One nice thing about logistic regression is that we can find the importance of each feature easily. Let’s visualize the top 10 words that are most correlated to a negative sentiment:
These words are within our expectation. Let’s check all the reviews that contain the word ‘poisoning’ and see how many of them are in the negative sentiment class:
0.904
Similarly, let’s check out the top 10 correlated words for positive sentiment:
Note that a high positive feature importance correlates to high possibility of Class 1 and a low negative (high absolute value) feature importance correlates to high possibility of Class 0.
After examining the positive reviews, I realized that the word ‘docking’ was at the top because most reviews that give a star of ‘4.0’ mention that “I am docking a star because...”.
Improvement Strategies
After we establish the first model, let’s examine couple of ideas to see if our model can be further improved.
Idea 1: Decrease the probability cutoff threshold
To reduce False Negatives, one intuition is to lower the cutoff threshold (default at 0.5). This would increase the recall but also decrease the precision. Therefore, we need to check if this would improve the overall F1 score:
******** For i = 0.3 ******F1: 0.91******** For i = 0.4 ******F1: 0.915******** For i = 0.45 ******F1: 0.915******** For i = 0.5 ******F1: 0.914
We can see that the F1 score is relatively robust to changes in this threshold.
Idea 2: Oversample Class 1 or undersample Class 0
Oversampling minority class and undersampling majority class are common ways to deal with imbalanced classification (Read more here). However, the F1 score does not improve in this case.
Performance for oversampling Class 1:
Accuracy:  0.95F1:  0.908
Performance for undersampling Class 0:
Accuracy:  0.947F1:  0.904
Idea 3: Remove stopwords and stemming
Removing stopwords and stemming can take out noises and thus reduce the vocabulary size. However, both the accuracy and F1 score decreases slightly.
Accuracy:  0.949F1:  0.902
Idea 4: Use TF-IDF instead of binary representation
This time, the F1 score slightly increases but not by much.
Accuracy:  0.958F1:  0.919
Idea 5: Include both unigrams and bigrams as features
The motivation can be shown with this example:
Take the LR model we first developed and then predict on this review — “I did not enjoy the food or the service”:
test_review = cv.transform(["I did not enjoy the food or the service"])lr.predict_proba(test_review)--------------------------------------------------------------------array([[0.50069323, 0.49930677]])
The model considers this review as positive, because the model only takes in unigrams such as ‘enjoy’ which is correlated to a positive sentiment, without considering ‘did not’ which negates the meaning of ‘enjoy’.
After we take both unigrams and bigrams (sequence of two words) into consideration, we first see an increase in vocabulary size:
488683
After fitting this model, we see improvements in both metrics, especially in F1 score.
Accuracy:  0.969F1:  0.942
Let’s check its prediction on the same sentence again:
test_review = cv.transform(["I did not enjoy the food or the service"])lr.predict_proba(test_review)--------------------------------------------------------------------array([[0.2678198, 0.7321802]])
Now it makes the right prediction with relatively high confidence.
We can see the new top 10 features for both positive and negative sentiment again:
We start to see bigrams such as ‘two stars’, ‘not worth’, ‘no thanks’, ‘not recommended’ showing up in the top features.
Note that “be disappointed” might seem like a negative sentiment but that is probably because it is part of “not be disappointed”phrase.
Should we include trigrams (sequence of three words) or even higher oder of N-grams? Note that as we include higher and higher order of N-grams, our feature size grows much larger which consumes more memory space, and there is also a problem of extreme sparsity (read more here).
Here are a few ideas for the next steps:
There are a few hyperparameters within the LR model that we can tune and tuning them can lead to a more optimized model. Also, to avoid over-fitting, try cross-validation to get more accurate metrics for each model (Check out GridSearch here).Treat ‘3.0’ as neutral or treat each category as its own class and re-formulate these models.Build an interactive sentiment analyzer which allows user-inputted reviews and give predictions on its sentiment. Have a built-in functionality of incremental learning where the users can help the model learn when it makes a wrong prediction.
There are a few hyperparameters within the LR model that we can tune and tuning them can lead to a more optimized model. Also, to avoid over-fitting, try cross-validation to get more accurate metrics for each model (Check out GridSearch here).
Treat ‘3.0’ as neutral or treat each category as its own class and re-formulate these models.
Build an interactive sentiment analyzer which allows user-inputted reviews and give predictions on its sentiment. Have a built-in functionality of incremental learning where the users can help the model learn when it makes a wrong prediction.
Let’s recap.
We built a sentiment classification model using logistic regression and tried out different strategies to improve upon the simple model. Among those ideas, including bigrams as features has the most improvement in F1 score. For both the simple model and the improved model, we also analyzed its most important textual features.
I hope that you enjoy this post and please share any thought that you may have :)
DS/ML beginner? Check out my other post on how you can build your first Python classifiers with the classic Iris dataset: