1 to 5 Star Ratings — Classification or Regression?
By: Sebastian Poliak
Tags: Supervised Learning, Regression, Machine Learning, Classification, NLP

1 to 5 Star Ratings — Classification or Regression?

Predicting a star rating (or any score rating in general) is a common task, often used in NLP to extract user feedback from open-text reviews without reading them, or in recommender systems to predict what rating would a user give to a movie he has never seen.

The rating itself as a target value maintains the natural ordering, which means that its values can be sorted or compared between each other. This gives us a possibility to model the problem as a regression task.

On the other hand, we could decide to ignore the ordering information and simply represent every value of the rating with a separate class. This would lead us to a classification task.

In this article, I would like to make an experiment using both approaches, and compare their performance on 3 different real world datasets.

Photo by Aperture Vintage on Unsplash

Experiment

For this experiment, I wanted to use at least 3 different datasets, in order to have a higher confidence in the final result. The datasets that have been chosen are Coursera’s Course Review Dataset, Amazon Musical Instruments Reviews and Trip Advisor Hotel Reviews. These all have the target values ranging from 1 star (very negative) to 5 stars (very positive). Additionally, every dataset has a bit different distribution of ratings, although the ratings 4 and 5 are always most represented.

The methods used for the experiment were very similar for every dataset. The dataset was always split into training and validation set in 90:10 ratio. The text of the reviews was not preprocessed in any way, and was represented using Glove: Global Vectors for Word Representation.

For classification I used the following model, which consists of a non-trainable embedding layer on the input, a bidirectional LSTM layer, a dense layer with ReLU, and an output layer with the softmax function over 5 classes. As expected, the model was trained using the cross-entropy loss function.

Classification model

For regression I used exactly the same model, only changing its final layer to have a single output with linear activation function. The model was trained with the mean squared error loss function.

Regression model

Both models were always trained on the same data, with the same batch size, and for the same amount of epochs. In order to compare the scores, I needed to use the same metric for evaluating every model. This was done by simply rounding up the predictions from regression to the nearest possible rating value, and using accuracy. The results are the following:

Accuracy of classification and regression model on three different datasets. Image by author.

What we can see is that classification achieved better result than regression on all three datasets. The difference seems to be quite significant, reaching almost 6% on Amazon Musical Instruments Reviews dataset.

The accuracies overall are not very high, which could possibly be caused by user bias, that is often present in ratings of the online review systems. Additionally, all the models used the same hyperparameters, which were not fine-tuned for every dataset. Still, the results of classification are not very far from the best solutions that are available for the given datasets.