Sentiment Classification with Logistic Regression — Analyzing Yelp Reviews
By: Dehao Zhang
Tags: Machine Learning, Sentiment Analysis, NLP, Python, Data Science

Peek at the Reviews

Let’s take out 1 million records from the review dataset for our analysis. The ‘text’ column would be our model inputs. let’s check out a random review with a positive sentiment (rating of 5.0):

"I love Deagan's. I do. I really do. The atmosphere is cozy and festive. The shrimp tacos and house fries are my standbys. The fries are sometimes good and sometimes great, and the spicy dipping sauce they come with is to die for. The beer list is amazing and the cocktails are great. The prices are mid-level, so it's not a cheap dive you can go to every week, but rather a treat when you do. Try it out. You won't be disappointed!"

As we are thinking about feature extraction, a couple of clues here can help us infer that this is positive sentiment, such as ‘love’, ‘cozy’, ‘die for’, ‘amazing’, and ‘won’t be disappointed’.

Let’s see one negative review (rating of 1.0):

"If I could give less than one star, that would have been my choice. I rent a home and Per my lease agreement it is MY responsibility to pay their Pool Service company. Within the last year they changed to PoolServ. I have had major issues with new techs every week, never checking PH balances, cleaning the filter, and not showing up at all 2 weeks in the past 2 months. I have had 4 different techs in the past 4 weeks. I have emailed and called them and they never respond back nor even acknowledged my concerns or requests. I cannot change companies but I'm required to still pay for lousy or no service. Attached are a couple pictures of my pool recently due to one tech just didn't put any chlorine in it at all according to the tech who came the following week to attempt to clean it up. Please think twice before working with these people. No one wants to work with a business that doesn't return phone calls or emails."

Despite its length, we can still see that clues such as ‘less than one star’, ‘lousy’, ‘never respond back’, and ‘no service’ are useful predictors.

Let’s quickly confirm that there is no missing value in the data:

text 0.0

stars 0.0

dtype: float64

Convert Ratings into Positive and Negative Sentiments

Let’s plot the ratings distribution:

We can see that out of the 1 million reviews, almost half of them contain a rating of 5.0.

In this task, we want to classify all review texts into one of the two categories: positive sentiment and negative sentiment. Therefore, we will first need to transform the ‘stars’ value into the two categories. Here, we can treat ‘4.0’ and ‘5.0’ as positive sentiment and ‘1.0’ and ‘2.0’ as negative sentiment. We can treat ‘3.0’ as neutral or even treat each star as its own sentiment category, which would make it a multi-class classification problem. However, for the sake of a simply binary classification, we can take the ones with ‘3.0’ out.

We then encode positive sentiment as Class 0 and negative sentiment as Class 1. Since we know that we have more samples in Class 0 than Class 1, our baseline model can be one that simply labels every review as Class 0. Let’s check out the baseline accuracy:

0.74

Before we proceed, let’s spend some time on the evaluation metric.

Evaluation Metric

Quoted from Jason Brownlee in one of his posts, “A classifier is only as good as the metric used to evaluate it”[2].

Accuracy might not be the appropriate evaluation metric in every classification problem, especially when the class distribution is imbalanced, and when the business impacts of false positive and false negative are unequal. For example, in a credit card fraud detection problem, a baseline model which predicts every transaction to be non-fraud would have an accuracy of over 99.99%, but it does not mean that it is a proper model since the percentage of false negative would be 100%, and each false negative (fail to detect a fraud transaction) can have a much higher cost to the business and customers compared with a false positive.

In this task, Here are a few appropriate evaluation metrics:

Precision — TP/(TP+FP), meaning the proportion of points that model classify as positives are actually positives. Recall — TP/(TP+FN), meaning the proportion of actual positives that are correctly classified by the model. F1 score —the harmonic mean of precision and recall.

Check out this post for more detailed discussion on these metrics.

In this task, we would use F1 score on the test set as the key evaluation metric.

Train-Test Split

Let’s set aside 30% of the data to be the test set, stratified by the class label.

train, test = train_test_split(df_reviews, test_size = 0.3, stratify = df_reviews['labels'], random_state = 42)

Text Preprocessing

In most text mining or NLP related tasks, cleaning text is a crucial step. Let’s first remove all non-letter characters, punctuations, and make sure all letters are in lower-cases. Later we will also evaluate the effect of removing stopwords and stemming/lemmatization.

Vectorization

For a model to be able to process text input, we would need to convert them into vectors. There are a few different ways to represent these text features and here are the most common ones: 1. Binary, e.g. whether the word “good” is present. 2. Count, e.g. how many times does the word “good” appear in this review, similar to the Bag of Word model in Naive Bayes. 3. TF-IDF, which is a weighted importance of each text feature relevant to the document (read more here).

Let’s try using binary representation of all unigrams (single word) first.

cv= CountVectorizer(binary=True, analyzer = text_prep, min_df = 10, max_df = 0.95)

cv.fit_transform(train['text'].values)

train_feature_set=cv.transform(train['text'].values)

test_feature_set=cv.transform(test['text'].values)

I used sklearn’s CountVectorizer object to extract all the word features, and the word would be excluded if it appears in less than 10 reviews or more than 95% of the reviews.

Let’s check how many unique words are in our dictionary:

train_feature_set.shape[1]

--------------------------------------------------------------------

40245

There are about 40K unique words. As an example, let’s check the index of word ‘tasty’:

cv.vocabulary_['tasty']

--------------------------------------------------------------------

35283

Fit a LR Model

Now we are ready to fit our first logistic regression model using sklearn:

lr = LogisticRegression(solver = 'liblinear', random_state = 42, max_iter=1000)

lr.fit(train_feature_set,y_train)

y_pred = lr.predict(test_feature_set)

print("Accuracy: ",round(metrics.accuracy_score(y_test,y_pred),3))

print("F1: ",round(metrics.f1_score(y_test, y_pred),3))

--------------------------------------------------------------------

Accuracy: 0.955

F1: 0.914

Let’s plot a confusion matrix to visualize the prediction results:

Visualize Feature Importance

One nice thing about logistic regression is that we can find the importance of each feature easily. Let’s visualize the top 10 words that are most correlated to a negative sentiment:

These words are within our expectation. Let’s check all the reviews that contain the word ‘poisoning’ and see how many of them are in the negative sentiment class:

0.904

Similarly, let’s check out the top 10 correlated words for positive sentiment:

Note that a high positive feature importance correlates to high possibility of Class 1 and a low negative (high absolute value) feature importance correlates to high possibility of Class 0.

After examining the positive reviews, I realized that the word ‘docking’ was at the top because most reviews that give a star of ‘4.0’ mention that “I am docking a star because…”.

Improvement Strategies

After we establish the first model, let’s examine couple of ideas to see if our model can be further improved.

Idea 1: Decrease the probability cutoff threshold

To reduce False Negatives, one intuition is to lower the cutoff threshold (default at 0.5). This would increase the recall but also decrease the precision. Therefore, we need to check if this would improve the overall F1 score:

******** For i = 0.3 ******

F1: 0.91



******** For i = 0.4 ******

F1: 0.915



******** For i = 0.45 ******

F1: 0.915



******** For i = 0.5 ******

F1: 0.914

We can see that the F1 score is relatively robust to changes in this threshold.

Idea 2: Oversample Class 1 or undersample Class 0

Oversampling minority class and undersampling majority class are common ways to deal with imbalanced classification (Read more here). However, the F1 score does not improve in this case.

Performance for oversampling Class 1:

Accuracy: 0.95

F1: 0.908

Performance for undersampling Class 0:

Accuracy: 0.947

F1: 0.904

Idea 3: Remove stopwords and stemming

Removing stopwords and stemming can take out noises and thus reduce the vocabulary size. However, both the accuracy and F1 score decreases slightly.

Accuracy: 0.949

F1: 0.902

Idea 4: Use TF-IDF instead of binary representation

This time, the F1 score slightly increases but not by much.

Accuracy: 0.958

F1: 0.919

Idea 5: Include both unigrams and bigrams as features

The motivation can be shown with this example:

Take the LR model we first developed and then predict on this review — “I did not enjoy the food or the service”:

test_review = cv.transform(["I did not enjoy the food or the service"])

lr.predict_proba(test_review)

--------------------------------------------------------------------

array([[0.50069323, 0.49930677]])

The model considers this review as positive, because the model only takes in unigrams such as ‘enjoy’ which is correlated to a positive sentiment, without considering ‘did not’ which negates the meaning of ‘enjoy’.

After we take both unigrams and bigrams (sequence of two words) into consideration, we first see an increase in vocabulary size:

488683

After fitting this model, we see improvements in both metrics, especially in F1 score.

Accuracy: 0.969

F1: 0.942

Let’s check its prediction on the same sentence again:

test_review = cv.transform(["I did not enjoy the food or the service"])

lr.predict_proba(test_review)

--------------------------------------------------------------------

array([[0.2678198, 0.7321802]])

Now it makes the right prediction with relatively high confidence.

We can see the new top 10 features for both positive and negative sentiment again:

We start to see bigrams such as ‘two stars’, ‘not worth’, ‘no thanks’, ‘not recommended’ showing up in the top features.

Note that “be disappointed” might seem like a negative sentiment but that is probably because it is part of “not be disappointed”phrase.

Should we include trigrams (sequence of three words) or even higher oder of N-grams? Note that as we include higher and higher order of N-grams, our feature size grows much larger which consumes more memory space, and there is also a problem of extreme sparsity (read more here).