How to Perform Ordinal Regression / Classification in PyTorch | Towards Data Science

When you have a multiclass classification problem and there is an order to the classes, it is known as an “ordinal regression” problem. An example could be the classification of a student’s performance into categories A > B > C > D > E. The issue in solving this kind of problem using a normal classifier is that the model will assume that the error of misclassifying an A as a D is just as bad as misclassifying A as a B — this is obviously not true since the difference between A and D is much bigger than between A and B. Alternative names for ordinal regression are ordinal classification or even ranking learning; as you might have guessed, these methods let the model know the ordinal relationship of the classes, letting the model learn to rank instead of learning to classify.
This post will demonstrate a simple trick for performing ordinal regression in PyTorch using a custom loss function. Although I’ll walk through the trick for a particular problem, you can apply it to any ordinal regression problem and any other framework.
Let us first find a dataset for testing. I recently wrote a post on how to predict the properties of chemical molecules — often for these kinds of problems, it is interesting to predict whether molecular properties lie in given ranges, e.g., maybe we are predicting how efficient different molecules are for a given task, and we wish to classify this efficiency into High > Medium > Low.
towardsdatascience.com
For testing, I fetched a “lipophilicity” regression dataset from GitHub, which contains 2100 molecules and their associated lipophilicity measurements (solubility in oil). I converted the dataset into a multiclass ordinal classification problem, where the goal is to classify the molecules into the 5 classes: Lowest < Low < Medium < High < Highest.
Here is an overview of the data distribution for the ordinal problem:
As demonstrated in my previous post, it is easy to quickly train a standard classification model for our dataset, “standard,” meaning it is trained with a normal cross-entropy loss function given the 5 defined classes. As an initial evaluation, I split the dataset into 50% training and 50% test and ran my evaluations on the test dataset. The results are as follows:
It is not performing very consistently; it’s fairly good at classifying the Low and Highest classes but not so good for the Medium class.
Now let’s try to train an ordinal model instead; the trick we employ was introduced by Cheng et al. in 2008 [1]. The idea is that we will transform each class into the following vectors:
Lowest  -> [1,0,0,0,0]Low     -> [1,1,0,0,0]Medium  -> [1,1,1,0,0]High    -> [1,1,1,1,0]Highest -> [1,1,1,1,1]
The cool thing about this simple encoding (or similar variants) is that the difference between the classes follows an ordinal scale; i.e., the difference between Lowest and Highest is larger than the difference between Lowest and Medium.
The idea is for our network to output a 5-dimensional vector with a binary score for each label (e.g., a sigmoid activation function on the final layer) and train it to predict these one-hot-like embeddings. Once we have trained it, we can convert the predictions back to the target labels by applying a threshold of >0.5 to our predictions and counting from left to right how many consecutive True values are present in the prediction, i.e., in code:
To train the network, we need an appropriate loss function. The 2008 paper states that this can either be a binary cross-entropy loss or a squared error loss. To see how the error increases with increased distance between the classes, consider these cases for a squared error function:
Lowest vs. Lowest: (1-1)2+0+0+0+0 = 0Lowest vs. Low: (1-1)2+(0–1)2+0+0+0 = 1Lowest vs. Medium: (1–1)2+(0–1)2+(0–1)2+0+0 = 2Lowest vs. High: (1–1)2+(0–1)2+(0–1)2+(0–1)2+0 = 3Lowest vs. Highest: (1–1)2+(0–1)2+(0–1)2+(0–1)2+(0–1)2 = 4
The loss function should take two parameters as input, namely the predictions and the targets. In the case of our setup, the input dimensions for the predictions array are [batch_size × 5], and the targets array is simply a list of label ids. To perform ordinal regression, we need to expand the targets list into a [batch_size, num_labels] tensor, according to our previous encoding, and return the mean squared error loss between predictions and the expanded target:
If we re-train our model with this loss function, the result is:
Clearly, the result looks a bit more “diagonal” than the previous classification run, indicating that, indeed, we punish the model more for being off by several labels on the ordinal scale. Can we quantify how well this is working? One option is to look at the root mean square error (RMSE) for the labels indices, i.e., approximately how many categories “away” are our predictions from the targets. In this test case, the RMSE for the classification model was 1.06, while for the ordinal model, it was 0.93 — a 12% improvement!
But! That is just one random train-test split for one dataset — for completeness, I ran this quantification for multiple datasets and different numbers of labels with 10-fold cross-validation (CV); this should give us a more realistic view of how the two methods stack up against each other. The out-of-fold results are as follows, where I have listed the % improvement in RMSE error:
In most cases, we achieve an improvement, but occasionally the result is the opposite — the performance decreases with the ordinal loss. How can this be? Beyond fundamental differences in the datasets and what we are modeling, a few potential factors jump to mind:
Dataset size; if we have a large dataset, we might have a sufficient number of samples for each label to perform normal classification, and the cross-entropy loss may be more well-behaved than the ordinal loss function. To test this, we can run the same CV where we only pick 500 molecules from each dataset (significantly less than the full datasets). The results (shown below) are much better; however, for certain dataset combinations, the performance of the ordinal loss function is still worse than the multiclass model. There is no free lunch, so this ordinal trick is clearly not guaranteed to work.
Rank inconsistencies; what if our ordinal prediction outputs [0.9, 0.9, 0.49, 0.72, 0.1]. Iterating from left to right would result in a prediction label of 2 (since 0.49 < 0.5), even though the right answer might just as well be 4 (since 0.72 > 0.5). I found that next to no rank inconsistencies were present in any of the predictions, so this is not the cause for decreased performance here.
Using an ordinal loss function can quickly get a significant performance boost when working with ordered target classes. However, there is no free lunch, and one should train models with different loss functions to see what works best for a given dataset
In 2020 Cao et al. published a method in [2] for performing ordinal regression in a rank-consistent manner using a shared penultimate layer of the neural network with separate bias units for each binary output. They released their Coral method in this Github repository. Even though there were no rank inconsistencies in the problem sets presented in this post, I did try using Coral in the CV tests performed in this post, but the performance was consistently worse for these datasets. Again, this shows that there is no free lunch, and one should always try multiple things.
[1] Jianlin Cheng; Zheng Wang; Gianluca Pollastri, A neural network approach to ordinal regression (2008), IEEE International Joint Conference on Neural Networks
[2] Wenzhi Cao, Vahid Mirjalili, Sebastian Raschka, Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation (2020), Pattern Recognition Letters.