Classifying Sentiment from Text Reviews
By: Xuankhanh Nguyen
Tags: Logistic Regression, Word Embeddings, Bag Of Words, Random Forest, Machine Learning

Classifying Sentiment from Text Reviews

By Gavin Smith and XuanKhanh Nguyen

This project was the second project for my machine learning class this semester. We were given a dataset of several thousand single-sentence reviews collected from three domains: imdb.com, amazon.com, yelp.com. Each review consists of a sentence and a binary label indicating the sentence's emotional sentiment (1 for positive feelings; 0 for negative feelings). All the provided reviews in the training and test set were scraped from websites whose assumed audience is primarily English speakers. There are 2400 input, output pairs in the training set with 4510 unique words and 600 inputs in the test set with 1921 uniques words. Our main task is to develop a binary classifier that can correctly identify a new sentence's sentiment. Sentiment analysis is performed using three different models.

Problem 1 follows the use of Bag-of-Words feature representation plus binary classifier pipeline. We want to produce the highest balanced accuracy on heldout data.

Problem 2 follows the use of Word Embedding feature representation. We want to produce the highest balanced accuracy on heldout data.

Problem 3 is open-ended. Our task is to find the best possible model to obtain high balanced accuracy on heldout data.

Methodologies are explained in all sections, along with respective figures.

Problem 1: Bag-of-Words Feature Representation

Section 1a: Bag-of-Words Design Decision Description

We used the Natural Language Toolkit (nltk) to preprocess the data. First, we converted all the sentences to lowercase. After that, we handled contractions (words with apostrophes in them) by converting them into a form without apostrophes, as defined within the dictionary named ’appos’ (taken from the comments section in a vidhy blog.) The reason we changed the words like this is that it takes many things written differently (e.g., “I will not”, “I Will Not”, and “I won’t”) and makes them the same, making it easier for the classifier to determine if those are positive or negative words. Then we tokenize the data by converting text to individual words. After this, we removed all the stop words (commonly occurring words that are not relevant in the data context) using nltk’s stopwords dictionary. Then, we removed punctuation from the words. We do not exclude rare words, nor do we exclude common words.

For the vectorization process, we decided to use sklearn’s TfidVectorizer and TfidTransformer classes. Before any preprocessing, the input training data set initially had 4510 unique words, and the test data set had 1921. After preprocessing, which included removing stop words, converting contractions, and removing punctuation, the training and test dataset size stayed the same. The vocabulary size decreased to 3414 words. We used sklearn’s TfidVectorizer and TfidTransformer classes because some words like “a”, “the” may appear many times, and their large counts would not be significant in the encoded vectors. By using the TfidTransformer and TfidVectorizer, common words are penalized, as we use the inverse document frequency (IDF) multiplied by the term frequency (TF) as the “count” for each feature vector.

Section 1b: Cross-Validation and Hyperparameter Selection Design Description

First, we used a random forest classifier. We used RandomForestClassifier from sklearn, using criterion=’gini’. ‘Gini’ calculates the amount of probability of a specific feature that is classified incorrectly when selected randomly, it seemed the best fit for the data being used in this project. The hyperparameters that we used to control the model include the n_estimator=[5,100,500] to determine the number of trees in the forest, and max depth=[3,50,100,500]to determine the number of level in each decision tree. Furthermore, we used extractor_min_df=[1,2,4]to ignore the term that appears in data set less than 1,2 and 4 times.

Second, we used the logistic regression model. We used the LogisticRegression model from sklearn, using its default ’liblinear’ solver, as it seemed the best fit for the size of the data being used in this project and it being relatively computationally inexpensive. The hyperparameters that we used to control the model include the inverse penalty C and the regularization. We determined the inverse penalty C using np.logspace(0, 6, 12), which generated 12 samples of C from 1.0 (the default) to 1000000. We used these ranges of C after experimenting with different ranges — it turned out that smaller and larger values than those defined by the log space used resulted in greater losses inaccuracy. Hyperparameter C applies a penalty to the parameter values to reduce overfitting using the function C=1/λ. Furthermore, we used different values for regularization (L1 and L2) to see any key differences in overfitting. L1 regularization uses ridge regularization, or the absolute value of the coefficient's magnitude as the penalty to the loss function, whereas L2 regularization uses the lasso method or the squared magnitude of the coefficient as its penalty terms. These values for the hyperparameters resulted in 24 different models.

For our problem, we use cross-validation to select the best model by creating models with a range of different hyperparameter and evaluate each one using 5-fold cross-validation; and verbosity of 0 (verbose = 0). We also shuffle the data since the provided data was not shuffled. We want to ensure that each data point creates an independent change on the model by shuffling data without being biased by the same points before them. We choose GridsSearchCV to automate the process of choosing the hyperparameter. The idea is straightforward: rather than using a separate validation set, we split the training set into five subsets, a.k.a folds. We perform a series of train and evaluate cycles where each time we train on 4 of the folds and test on the 5th (the holdout set). We repeat this cycle 5 times, each time using a different fold for evaluation. In the end, we average the scores for each of the folds to determine the overall performance of a given model. The grid search then returns the optimal hyperparameters for the validation set, and from there, we can train the model using all the training data with the given hyperparameter.

To optimize the holdout data, we use balanced_accuracy from gridsearchcv. It is calculated as the average of the proportion corrects of each class individually. Balance_accuracy= (sensitivity_specificity)*2. We used balance_accuracy as it will give us the average of the true positive rate and true negative rate.

Figure 1c: Hyperparameter Selection Figure for Classifier #1 (Random Forest)

From the result, the best hyperparameter is {‘my_bow_feature_extractor__min_df’: 1, ‘my_classifier__max_depth’: 100, ‘my_classifier__n_estimators’: 500} We plot the number of estimator, max_depth and number of word instances to visualize the balance accuracy. As shown in Figure 1c, the model performs worst when the max_depth is 3 with a number of estimators equals to 5. Generally, we want as many trees as will improve our model. More trees also mean more computational cost, and after 150, the improvement is negligible. The depth of the tree's meaning length, so the larger tree helps us convey more info, whereas the smaller tree gives less precise info. So, the depth should be large enough (in our case 100) to split each node to our desired number of observations. The balanced accuracy we got using pipeline on the training set is 0.9962. However, the balanced accuracy on the test set is 0.7962. We can see that Random Forests tend not to overfit because they are an average over many trees, and our graphs reflect that even with a large max depth or number of estimators, the validation prediction is still good.

Figure 1d: Hyperparameter Selection Figure for Classifier #2 (Logistic Regression)

The results show that the model does best using L2 regularization with C values closest to the default value of 1.0. The model that had the highest accuracy, in this case, had a C value of 3.5 and an L2 regularization with an accuracy of 0.833. The models did the worse with inverse C values greater than 10 on L1 and L2 regularizations. We can clearly see overfitting evidence with large C values as the training set balanced accuracy continues to rise while the test set balanced accuracy gets worse. This is because these C values are regularizing the model less and so it can overfit more.

Section 1e: Determination of the Best Classifier Pipeline

The best performing classifier has been the logistic regression classifier, to our surprise. Logistic regression classifier had the highest accuracy had a C value of 3.5 and an L2 regularization with an accuracy of 0.833. We initially thought it would be very unlikely that the logistic regression model would give us the best results. It seemed very straightforward — that seems to work in its favor; however, random forest -a more complicated models, performed the worst. We believe a large part of this success has to do with the feature tuning performed — by so rigorously trying to normalize the input data into feature vectors using TF-IDF, it seems that the Logistic Regression classifier worked the best as it usually performs the best when attributes unrelated to the output variable, as well as closely related attributes are removed from the input set. This was most perceptible within the steps of removing stop words (removing unrelated attributes to the feature set) and the TF-IDF vectorization (penalizing closely related attributes). In combination with the regularization performed on the model, normalizing the data made the output classes rather separable.

Section 1f: Analysis of Predictions for the Best Classifier

The logistic regression model did best on predicting positive values with data sourced from Amazon, with a true positive rate of .9675. The model predicted true negatives, especially from Yelp reviews, with a true negative rate of 0.98. The model also had a higher false-positive rate for Yelp reviews at a false positive rate of 0.0525. Interestingly, it also had the lowest true negative rates for Yelp reviews, at 0.02. Overall, in terms of total accuracy, the model performed best on Amazon reviews, at an accuracy of 0.9725, compared to 0.96375 and 0.95375 for Yelp and IMDb, respectively. Possible reasons for this may be due to the number of sentimental (positive/negative) words within the feature sets for each type of review — Amazon customers may have better use of language in line with how the model calculates an output class for the sentiment. Still, conversely, my model may also overfit for Amazon reviews.

This figure shows examples of true and false positives and true and false negatives of the best logistic regression classifier on the training set. The first thing that can be seen from this figure is that the model has a hard time classifying sentences that contain “positive” words. The model cannot classify “not good” as negative, and it is also not able to classify “will not be disappointed” as positive. The reason for this could be that we tokenize the sentence with unigram, the bag of words cannot capture the fact that “not good” is one entity. The classifier cannot correctly classify sentences that contain a positive or a negative part, but then ultimately become the opposite type of review. For example, “I do not think you will be disappointed” is classified as negative because it starts with a strong negative statement “I do not” but then leads to a positive review. This could be because we tokenize the sentence with 1-gram (unigram), so we get 8 different words, including “not”. And when the model sees the word “not”, it classifies the review as negative.

We noticed that tokenizing the sentence with unigram was not the best method here. If we do with 2-gram, then we could get every combination of two words joined together. This can result in better accuracy since “not good” captures the sentence's negative meaning. We could use it to improve the performance of our problem 3.

Section 1g: Report Performance on Test Set via Leaderboard

Using the leaderboard test set, the logistic regression model gave an error rate of 0.8333 and an AUROC of 0.90727. This matches up with what the training set performance eluded to, given that both the training set and the test set had the best performance compared to the other classifiers. This may suggest that the testing data may be similar in terms of TF-IDF values with the training data. The Logistic Regression model may have overfitted on the training data, leading to increased performance on the testing data (which is similar, in this case). It could also mean that the other models overfit on training data, which is not as similar in comparison to the testing data in this scenario.

Problem 2: Word Embedding Feature Representation

Section 2a: Word Embedding Preprocessing Description

We preprocessed the data similar way as described in part 1a. Since GloVE provides a 50- dimension feature vector for each word, each vocabulary word is represented by a fifty-dimensional vector. Using the average of all the feature vectors, we could standardize the input data based on varying input feature-length. This means that the embedding of all words is averaged for each sentence, which produces a one-dimensional vector of features. After averaging the vector's values, the vector is multiplied by the weights given from the TF-IDF weights, which creates a feature vector that is fifty dimensions. This was done because the dimensionality of the word embedding is limited, so changing it from 3414 to fifty is preferable. We did not handle out of vocabulary words.

Section 2b: Cross-Validation and Hyperparameter Selection Design

First, we tried the Random Forest Classifier. The process of defining hyperparameter is similar to part 1 (as mentioned in 1B).

Second, we tried MLP. The hyperparameters used here control the activation functions, the number of hidden layers, and the number of neurons composing the hidden layers. For the number of hidden layers, the size ranges from 1 to 3, as we learned that for most learning tasks, the number of hidden layers for an MLP model is usually optimized for 1 or 2 hidden layers. For the number of neurons per layer, we used the default value of shape (100,), as well as shape (256,) and shapes (64,64,64), (64,100,64). The values for the number of neurons were derived by experimenting with different shapes and computation times, as well as the rule of thumb that the number of neurons per hidden layer should be between the range of neurons within the input and output layer (the number of features within the data). We also used four different activation functions, ‘identity’, ‘logistic’, ‘relu’, and ‘tanh’. The ’identity’ function uses no-op activation, and it returns f(x) = x. The logistic function uses the sigmoid, defined by an S-shaped curve, but it does have a few problems, including the vanishing gradient problem, slow convergence, and uncentered gradient updates. This function returns f(x) = 1 / (1 + exp(-x)). The rectified-linear unit (reLU) function converges more often than the ’tanh’ activation function. It remedies the vanishing gradient problem — its limitation lies in that it can only be used for hidden layers. It can create weight updates that lead to the non-activation of neurons, leading to neuron death. This function returns f(x)=max(0,x). The ’tanh’ function optimizes faster than the logistic function and has a vanishing gradient problem. This function returns f(x)=tanh(x). We chose these parameters because these were all possible values for the activation function on sklearn’s documentation. We found it interesting to compare different ways of outputting inputs to forward layers. These sets of parameters resulted in 16 different models.

As in part 1, the hyperparameters were selected depending on their performance in predicting the validation set. We chose this because performance on heldout data best predicts how the classifier will perform on totally new data. We can compare the training set performance, and the validation set performance to see how changing parameters impact the classifier’s accuracy and look at hyperparameter values that are causing the classifier to overfit to the data. We create this validation set using 5-fold cross-validation. Each fold is of equal size and is split by the grid searcher. To search for hyperparameters, we used a grid search because it allows us to see how many different configurations of hyperparameters perform. The grid search will return the optimal hyperparameters for the validation set, and from there, we can train a model using all the training data with the given hyperparameters.

Figure 2c: Hyperparameter selection Figure and Paragraph for Random Forest Classifier

We chose to use the Random Forest classifier because this classifier tends to be resistant to overfitting. We could train it, knowing that we would not be overfitting to our training data. The first hyperparameter we looked at was the maximum depth of the trees, which controls how deep the trees can be and makes the model more complex. We chose these values for maximum depth because they ranged from very small values, which only allow for a few numbers of leaves, to very large, which would allow for very complex trees. The second hyperparameter we looked at is the maximum number of features when looking for the best split. We chose this range because it covered small values that would underfit the data because the best split would probably not be found, and large values would tend to find the best split in the data. Lastly, we looked at the total number of estimators in the random forest. Increasing the number of estimators generally increases the model's accuracy because the prediction is averaged between more trees, but the payoff is diminished. From these graphs, we can see that a medium maximum depth, a medium number of features, and many estimators are preferred. It seems that performance diminishes for large max depths and max features because the model is slightly overfitting. Increasing the number of estimators will not overfit the data, so we will not see worse results with more estimators.

Figure 2d: Hyperparameter selection Figure and Paragraph for Classifier #2

The advantage of using an MLP is that MLPs are good at classifying complex data, and since our data text has complex relationships, an MLP seemed appropriate. The first hyperparameter we looked at was the l2 regularization term alpha. A smaller alpha means the model is less regularized, which can lead to overfitting, and a larger value means more regularization and underfitting. The second hyperparameter we looked at is the hidden layer's size (we also looked at multiple layers, but this is not shown as it is difficult to show in a graph). Our range of values covered a small hidden layer that would underfit the data and very large values which overfit the data. Last, we looked at the different types of activation functions. From the graphs, we can see that an alpha value of 0.01 performed the best as it avoided being too large to underfit the data. We can also see that larger sizes of the hidden layer greatly overfit the data as the training accuracy was very high. Still, the validation accuracy diminished, leading to the best value of 256. Then from the four activation functions which were looked at, the relu function performed best.

Section 2e: Determination of the Best Classifier Pipeline

Of the two classifiers we looked at for part 2, the MLP classifier performed better. The reason we know this is because the MLP classifier had a higher balanced accuracy on the validation sets. There are two reasons that we think an MLP classifier would perform better on this set of data. First, as mentioned before, MLPs can model complex relationships. With the word embedding, we are mapping the relationships between all the words in the vocabulary, so we think that an MLP is better suited to quantify those relationships and be able to predict unseen data. Second, neural networks are good at generalizing data, which means that they can make predictions regarding relationships that it has never seen before. Many reviews contain words or combinations of words that are not in the training data, so an MLP can more effectively classify this novel data. Last, MLPs offer many different hyperparameters that we can control, which means that we can tune the final model to be the best fit to classify the test set.

Figure 2f: True and False Positives, and True and False Negatives of Best MLP Classifier

This figure shows examples of true and false positives and true and false negatives of the best MLP classifier on the training set. The first thing that can be seen from this figure is that the model has a hard time classifying sentences that contain “positive” words with a qualifier before them. The model is not able to classify “not good” as negative, and it is also not able to classify “won’t regret” as positive. The reason for this could be that word embedding is not necessarily able to take bigrams into account, so it cannot capture the fact that “not good” is one entity. The classifier also not able to correctly classify sentences that contain a good or a bad part, but then ultimately become the opposite type of review. For example, the sentence that starts “I do love sushi, but” is classified as positive because it starts with a strong positive statement but then leads to a negative review. This could be because our stop words include prepositions like “but”, which usually indicate a change in the type of review.

Section 2g: Report Performance on Test Set via Leaderboard

Our best classifier's ultimate score on the test data was a balanced accuracy of 0.79833 and an AUROC of 0.8752. This is slightly higher than our performance on the validation sets from cross-validation. The reason for this could be because the final model is trained on all the training data rather than just a portion of it. The performance of this classifier was worse than with the bag of words model. We think this is because the bag of words can consider the meanings of two or more words that are close together.

Problem 3: Open-ended challenge

Section 3a: Open-Ended Methods Description

Just as with parts 1 and 2, we cleaned our data by taking out punctuation, converting all words to lower case, tokenizing each word, converting contractions into their expanded forms, and removing stop words from the used dictionary. This produced a vocabulary the same size as in parts 1 and 2. From there, we took the vocabulary and then concatenated on the word embedding vectors from part 2 onto the vocabulary. This created a training set that is still 2400 long (which is the number of examples), but with 4560 features for each example. We did this because it will give the classifier more information to train on and hopefully produce a more accurate result.

The classifier we chose to use for Part 3 is an MLP. We chose this because of the same reasons that we chose it for part 2. The training set that we have created contains many different features that are not necessarily related to each other (with the addition of the word vectors and the vocabulary). Since MLPs are good at learning complex relationships, we thought this was the best option. When training, we had to limit the number of hyperparameters we were testing because with this amount of data, the model took a very long time to train, and with the addition of more hyperparameters, it would have taken an unreasonable amount of time to train for the amount of time we had when starting this problem. The hyperparameters we explored were the type of activation function and the number of hidden layers. For the activation function, we chose the four most common functions, and with the hidden layers, we chose to select from many different numbers of 1-layer models as well as a few models with multiple layers. We did this so that we could see overfitting from too many neurons in the hidden layer, as well as overfitting from having too many layers. We found that the logistic activation function worked best from the grid search, and then a hidden layer of size 256 performed best. We think this hidden layer size performed best because it was not so large that it overfitted to the data, and then it performed better than with multiple layers due to overfitting.

Section 3b: Open-Ended Report Performance

In the end, this model was only able to reach a balanced accuracy of 0.81833 and an AUROC of 0.90559. This is a worse performance than our bag of words classifier but better than the word embedding classifier. We think that this is because the word embedding model that we used was not as effective as predicting these reviews. By combining the bag of words representation with the word embedding representation, we were not improving the initial bag of words representation, so it did not improve the results. We did also use linear regression with the training data we created, and it again did not perform as well as the MLP, so obviously, some classifiers work better with different data representations.

Resources: