Calibration of ordinal regression networks

## Abstract

Abstract Recent studies have shown that deep neural networks are not well-calibrated and produce over-confident predictions.
The miscalibration issue primarily stems from the minimization of cross-entropy, which aims to align predicted softmax probabilities with one-hot labels. In ordinal regression tasks, this problem is compounded by an additional challenge: the expectation that softmax probabilities should exhibit unimodal distribution is not met with cross-entropy. Rather, the ordinal regression literature has focused on unimodality and overlooked calibration. To address these issues, we propose a novel loss function that introduces order-aware calibration, ensuring that prediction confidence adheres to ordinal relationships between classes. It incorporates soft ordinal encoding and label-smoothing-based regularization to enforce both calibration and unimodality. Extensive experiments across three popular ordinal regression benchmarks demonstrate that our approach achieves state-of-the-art calibration without compromising accuracy.

## 1 Introduction

Despite significant advances in ordinal regression tasks, such as medical diagnosis and age estimation, one critical aspect has often been overlooked: calibration. While research has predominantly focused on improving accuracy, the importance of well-calibrated predictions in ordinal regression remains underexplored. In high-risk applications, where both accuracy and reliability are crucial, poorly calibrated models can lead to overconfident or underconfident decisions, potentially resulting in harmful outcomes.

Ordinal regression, also called ordinal classification, involves a natural ordering between class labels, setting it apart from nominal tasks. Approaches such as regression (; ; ; ), classification (; ; ), and ranking-based methods (; ; ; ) have been developed to capture this ordinal structure, often outperforming traditional frameworks by better aligning with the data’s inherent order. However, insufficient attention to calibration has led to unreliable confidence estimates, particularly in fields like medical diagnosis, where the consequences of miscalibration can be severe ().

Calibration aims to align a model’s confidence estimates with actual accuracy, ensuring that predicted probabilities reflect the likelihood of correct predictions. Without proper calibration, models risk overconfidence, especially when encountering ambiguous or noisy data, which can lead to unsafe decisions in sensitive domains such as healthcare (; ). For example, in disease severity prediction, a model should adjust its confidence based on input uncertainty, particularly in ambiguous cases.

In addition to calibration, unimodality is crucial in ordinal classification. Unimodal distributions ensure that the model assigns the highest probability to the correct label, with probabilities gradually decreasing as the distance from the true label increases, preventing paradoxical or inconsistent predictions (; ).

To address these dual challenges of calibration and unimodality, we propose Oridnal Regression loss for Calibration and Unimodality (ORCU), a novel loss function that integrates order-aware calibration with a unimodal regularization term. Traditional regularization techniques often neglect the ordered structure of ordinal tasks, but ORCU enforces both calibration and unimodality by explicitly modeling the ordinal relationships between classes. This ensures well-calibrated confidence estimates that reflect the full ordinal structure, enabling the model to produce unimodal probability distributions that smoothly decrease as the distance from the true label increases. As a result, ORCU not only improves prediction accuracy but also enhances the reliability and trustworthiness of predictions, particularly in high-stakes applications like medical diagnosis.

Contributions  We make the following key contributions: (1) We propose ORCU, a novel loss function that unifies calibration and unimodality within a single framework for ordinal regression. ORCU incorporates soft encoding and introduces a regularization term that uniquely applies order-aware conditions, ensuring that predicted probabilities reflect the ordinal relationships between classes while providing reliable confidence estimates. (2) We provide a comprehensive analysis demonstrating how ORCU effectively balances the trade-off between well-calibrated confidence estimates and accurate predictions. By integrating calibration and unimodality constraints into a single loss function, ORCU addresses the limitations of existing methods that focus solely on either calibration or ordinal structure. (3) By unifying calibration and unimodality constraints, we establish a new benchmark for reliable ordinal classification, significantly advancing the state of the art and paving the way for future research on trustworthy models for ordinal tasks.

Figure: Figure 1: Overview of how the ORCU achieves improved calibration and unimodal probability distribution in ordinal classification. It illustrates the role of the calibration regularization term, $\mathcal{L}_{\text{REG}}$, when applied to soft label encoding for the Adience dataset (8 ordinal classes representing age groups) as an example. Soft encoding is applied to hard ordinal labels (a-i). Then, $\mathcal{L}_{\text{REG}}$ is applied differently for $k<y_{n}$ (a-ii) and $k\geq y_{n}$ (a-iii). (b) illustrates how $\mathcal{L}_{\text{ORCU}}$’s gradients are computed with examples shown for $k=0,4$. See section 3.4 and Table 1 for gradient analysis.
Refer to caption: extracted/5941715/Figures/main_1003.png

## 2 Related Works

Ordinal Regression  Ordinal regression addresses the challenge of predicting a target value with an inherent order, unlike nominal classification, where no ordinal relationships exist between labels. Given an input $x$, the goal is to predict a label $y$, which follows an ordinal relationship such that $y_{1}\prec y_{2}\prec\dots\prec y_{C}$, with $\prec$ representing a label order (e.g., $y_{i}$ is less severe or earlier in rank than $y_{i+1}$). Approaches to ordinal classification are broadly divided into three categories: regression methods, classification-based methods, and ranking-based methods. In regression methods (; ; ; ), ordinal labels $y$ are treated as continuous variables, applying losses like L1 or L2 to predict a scalar value reflecting the label ordering. Classification-based methods (; ; ) discretize the continuous target space into bins, treating each bin as a class $y\in\{1,2,\dots,C\}$ and predicting the class directly. Ranking-based methods (; ; ; ) decompose the task into $C-1$ binary classifiers, each determining whether the true label $y$ exceeds a threshold, capturing the ordinal relationships between labels.

Loss functions for ordinal regression  The inherent limitations of Cross-Entropy (CE) loss in handling ordinal relationships have led to several modified approaches (see Section 3.1). One such approach is Soft ORDinal (SORD) encoding method (), which adjusts the label distribution by using soft labels to capture the proximity between classes, thereby ensuring smoother and more order-aware predictions (see Section 3.2). Class Distance Weighted Cross-Entropy (CDW-CE) (), on the other hand, keeps the traditional label structure but incorporates a distance-based penalty into the loss function, encouraging predictions closer to the true class and better aligned with the ordinal structure. CO2 () extends CE with a regularization term enforcing unimodality, ensuring that predicted probabilities decrease smoothly as the distance from the true label increases. In contrast, Probabilistic Ordinal Embeddings (POE) () introduces both a regularization term and architectural changes, representing each label as a probability distribution, thereby modeling both uncertainty and ordinal relationships.

Regularization-based loss functions for calibration  Calibration, which aims to align predicted confidence with actual accuracy, is particularly critical in high-risk tasks involving ordinal categories. Several regularization-based approaches have been proposed to improve calibration during training, as they preemptively address miscalibration without relying on post-hoc adjustments. Label Smoothing (LS) () is one of the foundational techniques, softening the sharp one-hot label distribution to mitigate overconfidence. Sample-dependent Focal Loss (FLSD) () builds upon this by focusing calibration on harder-to-classify examples. Margin-based Label Smoothing (MbLS) () selectively smooths predictions based on the margin between predicted logits and true labels, while Multi-class Difference in Confidence and Accuracy (MDCA) () applies this margin adjustment across the entire predicted distribution. Adaptive and Conditional Label Smoothing (ACLS) () dynamically adjusts the level of smoothing, applying stronger smoothing to miscalibrated predictions while preserving accurate confidence estimates for well-calibrated ones. While regularization-based calibration methods have been extensively studied, their application to ordinal tasks remains underexplored, despite the crucial role of confidence estimation in such settings.

## 3 A unified loss functions for calibration and unimodality

### 3.1 Limitations of Cross-Entropy in Ordinal Regression

In ordinal classification tasks, such as medical diagnosis or rating assessments, the inherent order between classes plays a crucial role. Traditional loss functions, like CE, often fail to capture this ordinal structure, leading to suboptimal performance and poorly calibrated predictions. This limitation is particularly problematic in high-risk applications where both accurate predictions and reliable confidence estimates are essential for informed decision-making.

Let $y_{n}\in\{1,\ldots,C\}$ denote the true class label of the $n$-th sample, where $N$ is the total number of samples and $C$ represents the number of classes. The CE loss is defined as $\mathcal{L}_{\text{CE}}=-\sum_{n=1}^{N}\sum_{k=1}^{C}y_{n,k}\log(\hat{y}_{n,k})$ where $y_{n,k}$ is the one-hot encoded true label and $\hat{y}_{n,k}$ is the predicted probability. The gradient of $\mathcal{L}_{\text{CE}}$ with respect to the logit $z_{n,k}$ is:$\frac{\partial\mathcal{L}_{\text{CE}}}{\partial z_{n,k}}=\hat{y}_{n,k}-y_{n,k}.$
This formulation forces the model to focus sharply on the true label, often resulting in overconfident predictions that ignore relationships between adjacent classes. In ordinal tasks, predictions should reflect the ordered nature of classes, with probability mass distributed smoothly around the true label. However, $\mathcal{L}_{\text{CE}}$ typically produces sharp peaks, disregarding the ordinal structure and potentially leading to unreliable predictions in critical scenarios.

To address these limitations, we propose a novel loss function that explicitly accounts for ordinal relationships between classes while incorporating a regularization term to enforce unimodality and improve calibration. Our approach aims to balance accurate ordinal predictions with well-calibrated confidence estimates, enhancing performance in sensitive applications without compromising overall predictive accuracy.

### 3.2 Soft Ordinal Encoding

One of the primary limitations of the traditional $\mathcal{L}_{\text{CE}}$ loss in ordinal tasks is its reliance on one-hot encoding, which concentrates probability mass entirely on the true label. This results in overconfident predictions, disregarding the relationships between adjacent classes. To address these issues, we employ the SORD encoding method (), which redistributes the probability mass across the ordinal classes in a way that reflects both the inherent ordinal structure and the uncertainty between adjacent labels.

In an ordinal classification problem with $C$ classes, the true label $y_{n}$ for the $n$-th sample is represented as a soft-encoded probability distribution $y^{\prime}_{n,k}$, defined as $y^{\prime}_{n,k}=\frac{e^{-\phi(y_{n},r_{k})}}{\sum_{j=1}^{C}e^{-\phi(y_{n},r_
{j})}}$, for $k=1,\dots,C$, where $\phi(y_{n},r_{k})$ is a distance metric that penalizes deviations from the true class $y_{n}$ to each ordinal class $r_{k}$ during the soft encoding process. The Soft-encoded Cross-Entropy (SCE) loss is then defined as:

$$ $\mathcal{L}_{\text{SCE}}=-\sum_{n=1}^{N}\sum_{k=1}^{C}y^{\prime}_{n,k}\log( \hat{y}_{n,k}),$ (1) $$

The gradient of $\mathcal{L}_{\text{SCE}}$ with respect to the logit $z_{n,k}$ is $\frac{\partial\mathcal{L}_{\text{SCE}}}{\partial z_{n,k}}=\hat{y}_{n,k}-y^{
\prime}_{n,k}$. This gradient encourages the model to distribute probability across adjacent classes in a way that aligns with the soft-encoded true label distribution, mitigating overconfidence and resulting in more calibrated predictions that accurately reflect the ordinal relationships between classes.

### 3.3 Order-Aware Regularization: Enforcing Unimodality and Calibration

While $\mathcal{L}_{\text{SCE}}$ effectively captures the ordinal relationships between labels and mitigates the overconfidence issue in $\mathcal{L}_{\text{CE}}$, it can still lead to underconfident predictions due to the redistribution of probability mass across classes. To address this issue, we introduce a regularization-based method that simultaneously enhances the model’s calibration and reinforces the learning of the ordinal structure (which inherently leads to unimodality in the output distribution).

Unlike traditional regularization methods, which focus solely on the highest probability class and ignore the relationships between adjacent labels (), our approach leverages the ordinal structure of the task. Specifically, our method adjusts the logits based on their ordinal relationship to the true label $y_{n}$, ensuring that the calibration regularization is order-aware. This approach divides the logits into two regions: those where the class index $k$ is smaller than the true label and those where $k$ is greater than or equal to the true label. By doing so, the regularization ensures smooth decreases in probability on both sides of the true label, enforcing a unimodal distribution while simultaneously improving calibration. The specific formulation of the regularization term is as follows:

$$ $\mathcal{L}_{\text{REG}}=\sum_{n=1}^{N}\sum_{k=1}^{C-1}\begin{cases}\hat{I}(z_ {n,k}-z_{n,k+1}),&\text{if }k<y_{n},\\ \hat{I}(z_{n,k+1}-z_{n,k}),&\text{if }k\geq y_{n},\end{cases}$ (2) $$

where $\hat{I}(r)=-\frac{1}{t}\log(-r)\text{ if }r\leq-\frac{1}{t^{2}},\text{ and }
\hat{I}(r)=tr-\frac{1}{t}\log\left(\frac{1}{t^{2}}\right)+\frac{1}{t}\text{ otherwise}$.

This penalty function applies strong corrections when the differences between adjacent logits approach the boundary value $-1/t^{2}$, ensuring unimodality is preserved. The temperature parameter $t$ controls the strength of this regularization. The final loss function ORCU, is defined as:

$$ $\mathcal{L_{\text{ORCU}}}=\mathcal{L}_{\text{SCE}}+\mathcal{L}_{\text{REG}}.$ (3) $$

By integrating both $\mathcal{L}_{\text{SCE}}$ and $\mathcal{L}_{\text{REG}}$, $\mathcal{L_{\text{ORCU}}}$ ensures that the model produces well-calibrated, unimodal predictions that accurately reflect the ordinal relationships between classes.

**Table 1: Gradient analysis of our loss function. The gradient is computed w.r.t. a logit when $k<y_{n}$ and $k\geq y_{n}$. We also show the gradient of each term, $\mathcal{L}_{\text{SCE}}$ and $\mathcal{L}_{\text{REG}}$, to analyze each one’s contribution. $\hat{y}_{n,k}$ denote the predicted probability, $y^{\prime}_{n,k}$ representing the soft-encoded target, and $t$ being a variable controlling the regularization strength. Note that $\frac{\partial\mathcal{L_{\text{ORCU}}}}{\partial z_{k}}=\frac{\partial \mathcal{L_{\text{SCE}}}}{\partial z_{k}}+\frac{\partial\mathcal{L_{\text{REG} }}}{\partial z_{k}}$.**
|  | $k<y_{n}$<br>($r=z_{n,k}-z_{n,k+1}$) | $k\geq y_{n}$<br>($r=z_{n,k+1}-z_{n,k}$) |  |  |
| --- | --- | --- | --- | --- |
|  | $r\leq-\frac{1}{t^{2}}$ | $r>-\frac{1}{t^{2}}$ | $r\leq-\frac{1}{t^{2}}$ | $r>-\frac{1}{t^{2}}$ |
| (a) $\quad\frac{\partial\mathcal{L_{\text{SCE}}}}{\partial z_{n,k}}$ | $\hat{y}_{n,k}-y^{\prime}_{n,k}$ |  |  |  |
| (b) $\quad\frac{\partial\mathcal{L_{\text{REG}}}}{\partial z_{n,k}}$ | $-\frac{1}{tr}$ | $t$ | $\frac{1}{tr}$ | $-t$ |
| (c) $\quad\frac{\partial\mathcal{L_{\text{ORCU}}}}{\partial z_{n,k}}$ | $\hat{y}_{n,k}-(y^{\prime}_{n,k}+\frac{1}{tr})$ | $\hat{y}_{n,k}-(y^{\prime}_{n,k}-t)$ | $\hat{y}_{n,k}-(y^{\prime}_{n,k}-\frac{1}{tr})$ | $\hat{y}_{n,k}-(y^{\prime}_{n,k}+t)$ |

### 3.4 Gradient Analysis: Enforcing Unimodality and Calibration

The gradient behavior of the proposed regularization term $\mathcal{L}_{\text{REG}}$ is crucial for ensuring both unimodality and calibration. As illustrated in Figure 1, the $\mathcal{L}_{\text{ORCU}}$ loss applies soft label encoding and uses regularization differently depending on whether $k<y_{n}$ or $k\geq y_{n}$. To better understand its impact, we divide the gradient into four cases based on the relationship between the class index $k$ and the target label $y_{n}$, as well as the magnitude of the difference between adjacent logits $r$ (Table 1-b). The regularization term adjusts the logits to ensure that the output distribution remains smooth and unimodal, particularly in ordinal classification tasks.

When $k<y_{n}$ (Figure 1-ii), the model adjusts the logits to ensure that the probability distribution decreases smoothly as the distance from the true label increases, maintaining a unimodal structure. This is achieved by keeping the difference between adjacent logits, $r$, negative. If $r$ is negative and has a large absolute value, the gradient of $\mathcal{L}_{\text{REG}}$ becomes smaller, indicating that the model is close to achieving a unimodal distribution. However, as $r$ approaches the boundary $-\frac{1}{t^{2}}$, the gradient increases sharply to prevent any violations of unimodality. On the other hand, when $r>-\frac{1}{t^{2}}$, indicating a deviation from the desired unimodal structure, a constant penalty is applied to restore the distribution.

Dynamic gradient adjustment for simultaneous calibration and unimodality  When the gradients of $\mathcal{L}_{\text{SCE}}$ and $\mathcal{L}_{\text{REG}}$ are combined, the regularization term enables the model to address both calibration and the learning of the ordinal structure (i.e., unimodality) simultaneously.

First, consider the case where $k<y_{n}$ (as shown in Figure 1-(a-ii)), where the predicted class is lower than the true class. In this scenario, for large positive $r$ values (Figure 1-(b-ii)), the regularization term reduces $y^{\prime}_{n,k}$ by a factor of $t$, increasing the difference between $\hat{y}_{n,k}$ and $y^{\prime}_{n,k}-t$. This larger gradient leads to a more substantial update to the logit $z_{k}$, compared to using $\mathcal{L}_{\text{SCE}}$ alone. Consequently, this adjustment not only restores unimodality by driving $r$ towards negative values but also helps correct overconfident predictions for incorrect labels. By increasing the gradient for such incorrect predictions, the model is able to reduce the predicted probability for the incorrect class more effectively, improving overall calibration and ordinal structure learning.

Next, consider the case where $k=y_{n}$, and $r>-\frac{1}{t^{2}}$ (as shown in Figure 1-(b-iii)). In this underconfident scenario, where the predicted probability $\hat{y}_{n,k}$ is lower than the target distribution $y^{\prime}_{n,k}$, the regularization term modifies the gradient to $\hat{y}_{n,k}-(y^{\prime}_{n,k}+t)$. This larger gradient (with a greater absolute value) results in a more substantial update to the logit $z_{k}$, increasing the predicted probability for the true class $y_{n}$. At the same time, this adjustment restores unimodality by ensuring that $r$ remains negative, aligning the probability distribution smoothly around the true label.

By combining these two examples, we demonstrate that the proposed $\mathcal{L}_{\text{REG}}$ term dynamically adjusts the gradients based on the value of $r$, enabling the model to simultaneously achieve well-calibrated predictions and maintain the ordinal structure (unimodality) of the task. Whether the model is overconfident or underconfident, the regularization term ensures appropriate updates to the logits, balancing both calibration and the preservation of the ordinal relationships between classes. This dual mechanism is essential for high-risk ordinal tasks, where both accurate predictions and reliable confidence estimates are critical.

**Table 2: Calibration and accuracy performance for different loss functions on three popular ordinal regression benchmarks. Competing methods in two areas, i.e., ordinal regression and network calibration, are compared with ours. We use ResNet-50 for classifications and use 15 bins for calibration metrics calculation. The measures are presented as the mean and standard deviation over all folds.**
| Loss<br>Evaluation Metrics | SCE($\downarrow$) | ACE($\downarrow$) | ECE($\downarrow$) | Acc($\uparrow$) | QWK($\uparrow$) |  |
| --- | --- | --- | --- | --- | --- | --- |
| Adience ($n=$17,423) |  |  |  |  |  |  |
| Ordinal Loss | Cross Entropy | 0.8495 $\pm$ 0.0033 | 0.8356 $\pm$ 0.0068 | 0.3364 $\pm$ 0.0401 | 0.5639 $\pm$ 0.0486 | 0.8795 $\pm$ 0.0345 |
| SORD () | 0.7823 $\pm$ 0.0105 | 0.7783 $\pm$ 0.0102 | 0.0731 $\pm$ 0.0240 | 0.5910 $\pm$ 0.0439 | 0.8995 $\pm$ 0.0291 |  |
| CDW-CE () | 0.8429 $\pm$ 0.0062 | 0.8372 $\pm$ 0.0071 | 0.2913 $\pm$ 0.0210 | 0.5789 $\pm$ 0.0362 | 0.8988 $\pm$ 0.0257 |  |
| CO2 () | 0.8521 $\pm$ 0.0055 | 0.8368 $\pm$ 0.0080 | 0.3533 $\pm$ 0.0406 | 0.5637 $\pm$ 0.0489 | 0.8728 $\pm$ 0.0418 |  |
| POE () | 0.8340 $\pm$ 0.0042 | 0.8244 $\pm$ 0.0065 | 0.2733 $\pm$ 0.0384 | 0.5652 $\pm$ 0.0522 | 0.8669 $\pm$ 0.0395 |  |
| Calibration Loss | LS () | 0.8195 $\pm$ 0.0071 | 0.8101 $\pm$ 0.0072 | 0.1890 $\pm$ 0.0415 | 0.5792 $\pm$ 0.0495 | 0.8824 $\pm$ 0.0334 |
| FLSD () | 0.8474 $\pm$ 0.0063 | 0.8340 $\pm$ 0.0062 | 0.3193 $\pm$ 0.0407 | 0.5718 $\pm$ 0.0532 | 0.8840 $\pm$ 0.0347 |  |
| MbLS () | 0.8400 $\pm$ 0.0043 | 0.8323 $\pm$ 0.0052 | 0.2815 $\pm$ 0.0344 | 0.5765 $\pm$ 0.0489 | 0.8894 $\pm$ 0.0324 |  |
| MDCA() | 0.8561 $\pm$ 0.0527 | 0.8365 $\pm$ 0.0059 | 0.3372 $\pm$ 0.0411 | 0.5676 $\pm$ 0.0488 | 0.8770 $\pm$ 0.0346 |  |
| ACLS () | 0.8398 $\pm$ 0.0040 | 0.8295 $\pm$ 0.0045 | 0.2847 $\pm$ 0.0378 | 0.5762 $\pm$ 0.0488 | 0.8788 $\pm$ 0.0364 |  |
| ORCU (Ours) | 0.4598 $\pm$ 0.0435 | 0.4565 $\pm$ 0.0437 | 0.0583 $\pm$ 0.0279 | 0.5878 $\pm$ 0.0426 | 0.9036 $\pm$ 0.0281 |  |
| Image Aesthetics ($n=$13,364) |  |  |  |  |  |  |
| Ordinal Loss | Cross Entropy | 0.7637 $\pm$ 0.0037 | 0.7558 $\pm$ 0.0035 | 0.2057 $\pm$ 0.0162 | 0.7030 $\pm$ 0.0080 | 0.4961 $\pm$ 0.0199 |
| SORD () | 0.6844 $\pm$ 0.0018 | 0.6833 $\pm$ 0.0018 | 0.1846 $\pm$ 0.0026 | 0.7092 $\pm$ 0.0044 | 0.5030 $\pm$ 0.0030 |  |
| CDW-CE () | 0.7519 $\pm$ 0.0030 | 0.7480 $\pm$ 0.0026 | 0.1751 $\pm$ 0.0136 | 0.7041 $\pm$ 0.0064 | 0.4837 $\pm$ 0.0148 |  |
| CO2 () | 0.7699 $\pm$ 0.0032 | 0.7614 $\pm$ 0.0036 | 0.2269 $\pm$ 0.0151 | 0.6980 $\pm$ 0.0072 | 0.4919 $\pm$ 0.0158 |  |
| POE () | 0.7559 $\pm$ 0.0020 | 0.7493 $\pm$ 0.0035 | 0.1883 $\pm$ 0.0111 | 0.7010 $\pm$ 0.0097 | 0.4909 $\pm$ 0.0156 |  |
| Calibration Loss | LS () | 0.7222 $\pm$ 0.0010 | 0.7179 $\pm$ 0.0014 | 0.0991 $\pm$ 0.0062 | 0.7063 $\pm$ 0.0054 | 0.4990 $\pm$ 0.0202 |
| FLSD () | 0.7519 $\pm$ 0.0030 | 0.7482 $\pm$ 0.0026 | 0.1751 $\pm$ 0.0136 | 0.7013 $\pm$ 0.0075 | 0.4961 $\pm$ 0.0147 |  |
| MbLS () | 0.7577 $\pm$ 0.0016 | 0.7510 $\pm$ 0.0020 | 0.1895 $\pm$ 0.0062 | 0.7040 $\pm$ 0.0057 | 0.4942 $\pm$ 0.0159 |  |
| MDCA() | 0.7620 $\pm$ 0.0033 | 0.7549 $\pm$ 0.0033 | 0.2022 $\pm$ 0.0124 | 0.7031 $\pm$ 0.0110 | 0.4993 $\pm$ 0.0129 |  |
| ACLS () | 0.7595 $\pm$ 0.0010 | 0.7535 $\pm$ 0.0013 | 0.1957 $\pm$ 0.0097 | 0.7030 $\pm$ 0.0091 | 0.4953 $\pm$ 0.0147 |  |
| ORCU (Ours) | 0.6805 $\pm$ 0.0045 | 0.6794 $\pm$ 0.0046 | 0.1082 $\pm$ 0.0312 | 0.7113 $\pm$ 0.0038 | 0.5188 $\pm$ 0.0139 |  |
| LIMUC ($n=$11,276) |  |  |  |  |  |  |
| Ordinal Loss | Cross Entropy | 0.6997 $\pm$ 0.0076 | 0.6948 $\pm$ 0.0075 | 0.1295 $\pm$ 0.0170 | 0.7702 $\pm$ 0.0066 | 0.8461 $\pm$ 0.0090 |
| SORD () | 0.6382 $\pm$ 0.0031 | 0.6370 $\pm$ 0.0032 | 0.1636 $\pm$ 0.0064 | 0.7749 $\pm$ 0.0060 | 0.8539 $\pm$ 0.0062 |  |
| CDW-CE () | 0.6980 $\pm$ 0.0048 | 0.6927 $\pm$ 0.0042 | 0.1190 $\pm$ 0.0096 | 0.7773 $\pm$ 0.0058 | 0.8551 $\pm$ 0.0069 |  |
| CO2 () | 0.7105 $\pm$ 0.0044 | 0.7042 $\pm$ 0.0042 | 0.1544 $\pm$ 0.0118 | 0.7662 $\pm$ 0.0076 | 0.8411 $\pm$ 0.0081 |  |
| POE () | 0.6933 $\pm$ 0.0043 | 0.6881 $\pm$ 0.0046 | 0.1149 $\pm$ 0.0105 | 0.7724 $\pm$ 0.0040 | 0.8353 $\pm$ 0.0110 |  |
| Calibration Loss | LS () | 0.6647 $\pm$ 0.0010 | 0.6603 $\pm$ 0.0022 | 0.0592 $\pm$ 0.0088 | 0.7633 $\pm$ 0.0053 | 0.8402 $\pm$ 0.0096 |
| FLSD () | 0.6674 $\pm$ 0.0016 | 0.6631 $\pm$ 0.0022 | 0.1069 $\pm$ 0.0167 | 0.7657 $\pm$ 0.0059 | 0.8459 $\pm$ 0.0110 |  |
| MbLS () | 0.6988 $\pm$ 0.0022 | 0.6982 $\pm$ 0.0035 | 0.1301 $\pm$ 0.0095 | 0.7665 $\pm$ 0.0057 | 0.8490 $\pm$ 0.0113 |  |
| MDCA() | 0.6934 $\pm$ 0.0074 | 0.6879 $\pm$ 0.0061 | 0.1197 $\pm$ 0.0129 | 0.7683 $\pm$ 0.0044 | 0.8443 $\pm$ 0.0118 |  |
| ACLS () | 0.6995 $\pm$ 0.0030 | 0.6939 $\pm$ 0.0032 | 0.1299 $\pm$ 0.0140 | 0.7683 $\pm$ 0.0064 | 0.8454 $\pm$ 0.0092 |  |
| ORCU (Ours) | 0.5205 $\pm$ 0.0098 | 0.5182 $\pm$ 0.0107 | 0.0853 $\pm$ 0.0269 | 0.7785 $\pm$ 0.0064 | 0.8578 $\pm$ 0.0048 |  |

## 4 Experiments

### 4.1 Datasets and Implementation Details

We evaluated the proposed $\mathcal{L_{\text{ORCU}}}$ loss function on three public datasets selected for their ordinal nature and sufficient sample sizes, covering diverse tasks such as age estimation, image aesthetics assessment, and medical diagnosis. The Adience dataset (), containing 26,580 images categorized into 8 age groups, was used for age estimation, employing five-fold cross-validation with an 80/20 train-test split(^1^11[https://github.com/GilLevi/AgeGenderDeepLearning](https://github.com/GilLevi/AgeGenderDeepLearning)). For image aesthetics assessment, the Image Aesthetics dataset (), consisting of 13,364 images with five ordinal labels, was used with five-fold cross-validation and an 80/20 train-test split. The LIMUC dataset (), comprising 11,276 images from ulcerative colitis patients with four mayo endoscopic scores (MES), was used for medical diagnosis, employing an 85/15 subject-exclusive () train-test split and a ten-fold cross-validation protocol ().

For all tasks, we used a ResNet-50 architecture () pretrained on ImageNet (). We applied a layer-wise learning rate strategy (0.01 for the fully connected layer, 0.001 for others), using AdamW () for optimization. Training was conducted for 100 epochs with a batch size of 64, applying image augmentations via Albumentations (). The squared distance metric was used for soft encoding, with the temperature parameter $t$ initialized at 10.0 and gradually decreased throughout training. All experiments were conducted on NVIDIA RTX 4090 GPUs.

### 4.2 Performance Metrics

Evaluating calibration is crucial in ordinal classification tasks, where models must provide not only accurate predictions but also reliable confidence estimates. We focus on two specific calibration metrics: static calibration error (SCE) and adaptive calibration error (ACE) (^2^22[https://github.com/Jonathan-Pearce/calibration_library](https://github.com/Jonathan-Pearce/calibration_library)), which are particularly suited to addressing class imbalances and capturing calibration across all predictions. Although expected calibration error (ECE) is commonly used, it can be misleading, especially in imbalanced datasets where confidence is skewed towards high-probability predictions, failing to accurately reflect model performance in all classes (). SCE and ACE provide more robust measures of calibration by evaluating calibration per class or dynamically adjusting bin sizes.

To capture ordinal relationships between classes, we use quadratic weighted kappa (QWK) as the primary metric, as it penalizes larger misclassifications more heavily and better reflects the distance between predicted and true labels compared to accuracy. Although accuracy is reported, QWK serves as the primary metric for assessing classification performance in ordinal tasks.

### 4.3 Results and Analysis

Our proposed loss function $\mathcal{L_{\text{ORCU}}}$ demonstrates superior performance across both calibration and ordinal regression metrics, surpassing baseline methods designed for either objective individually (Table 2). Unlike existing loss functions that primarily focus on a single goal—either calibration or classification—$\mathcal{L_{\text{ORCU}}}$ effectively balances both, achieving significant improvements in calibration metrics while maintaining strong predictive accuracy.

In high-stakes tasks such as medical diagnosis and severity grading, where accurate and well-calibrated predictions are critical, $\mathcal{L_{\text{ORCU}}}$ proves particularly advantageous. Existing ordinal loss functions, such as CE and its variants, tend to overlook calibration error, while traditional calibration losses fail to account for the ordinal structure that is intrinsic to these tasks. To thoroughly evaluate $\mathcal{L_{\text{ORCU}}}$, we compared it against 10 baseline loss functions, categorized into two groups: ordinal loss functions—CE, SORD (), CDW-CE (), CO2 (), and POE ()—and calibration loss functions—LS (), FLSD (), MbLS (), MDCA (), and ACLS (). Each loss function was evaluated under identical experimental conditions to ensure fair benchmarking.

Figure: Figure 2: Reliability diagrams for different ordinal loss functions. The diagrams show model confidence alongside the calibration gap between confidence and accuracy, using the test split of Adience, Image Aesthetics, and LIMUC. Predictions above the diagonal, expected line indicate underconfidence, while those below the line represent overconfidence. ECE is computed using 15 bins.
Refer to caption: extracted/5941715/Figures/Reliability_diagram.png

#### 4.3.1 Comparison with Loss Functions Targeting Individual Objectives

Comparison with Ordinal Loss Functions  Our method, $\mathcal{L_{\text{ORCU}}}$, consistently outperforms all baseline ordinal loss functions, providing superior calibration and classification performance. As detailed in Table 2, $\mathcal{L_{\text{ORCU}}}$ achieves the lowest SCE, ACE, and ECE scores, while maintaining competitive accuracy and QWK scores. This highlights its ability to deliver well-calibrated predictions without compromising on predictive performance.Traditional ordinal loss function, primarily focus on accuracy but tend to produce overconfident predictions by concentrating probability mass on the true label (Figure 2-a, c-e). Although SORD reduces overconfidence by distributing probability mass across adjacent labels, it results in underconfident predictions (Figure 2-c), limiting its effectiveness. In contrast, $\mathcal{L_{\text{ORCU}}}$ effectively balances these extremes. As shown in Figure 2-f, our method mitigates both overconfidence and underconfidence, delivering well-calibrated confidence estimates. Additionally, high QWK scores across all datasets confirm that $\mathcal{L_{\text{ORCU}}}$ captures the inherent ordinal relationships between classes more effectively than baseline ordinal loss functions.

Comparison with Calibration Loss Functions  When compared with calibration-focused loss functions, $\mathcal{L_{\text{ORCU}}}$ also demonstrates superior performance. Calibration losses like LS, FLSD, MbLS, MDCA, and ACLS are primarily designed to reduce calibration error in nominal tasks, without considering the ordinal relationships between classes. However, $\mathcal{L_{\text{ORCU}}}$ incorporates these ordinal relationships into the calibration process, which leads to superior results in tasks that require both accurate predictions and well-calibrated confidence estimates.As seen in Table 2, $\mathcal{L_{\text{ORCU}}}$ consistently achieves the lowest SCE and ACE scores, while also delivering strong accuracy and QWK scores. Although LS slightly outperforms $\mathcal{L_{\text{ORCU}}}$ in ECE, it fails to account for the ordinal structure, as evidenced by its lower QWK scores. This demonstrates the advantage of $\mathcal{L_{\text{ORCU}}}$ in ordinal classification tasks, where preserving the relationships between labels is critical.

In summary, $\mathcal{L_{\text{ORCU}}}$ provides a balanced solution for both calibration and classification, outperforming loss functions designed for either objective individually. By delivering well-calibrated predictions while maintaining high accuracy and QWK scores, $\mathcal{L_{\text{ORCU}}}$ effectively addresses the dual challenges of calibration and ordinal classification. This makes it particularly valuable in such critical tasks where both aspects are essential.

**Table 3: Ablation study on different distance metrics used in the soft encoding. The table analyzes the performance impact of applying various distance metrics (absolute, Huber, exponential, and squared) across multiple datasets. The results are presented as the mean and standard deviation over all folds.**
| Distance Metrics $\phi$<br>Evaluation Metrics | SCE($\downarrow$) | ACE($\downarrow$) | ECE($\downarrow$) | Acc($\uparrow$) | QWK($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| Adience ($n=$17,423) |  |  |  |  |  |
| Absolute | 0.7121 $\pm$ 0.0088 | 0.7167 $\pm$ 0.0088 | 0.0684 $\pm$ 0.0203 | 0.5800 $\pm$ 0.0334 | 0.8961 $\pm$ 0.0299 |
| Huber | 0.7274 $\pm$ 0.0110 | 0.7233 $\pm$ 0.0115 | 0.0553 $\pm$ 0.0084 | 0.5704 $\pm$ 0.0461 | 0.8948 $\pm$ 0.0323 |
| Exponential | 0.7774 $\pm$ 0.0120 | 0.7753 $\pm$ 0.0116 | 0.0868 $\pm$ 0.0262 | 0.5794 $\pm$ 0.0425 | 0.8983 $\pm$ 0.0320 |
| Squared | 0.4598 $\pm$ 0.0435 | 0.4565 $\pm$ 0.0437 | 0.0583 $\pm$ 0.0279 | 0.5878 $\pm$ 0.0426 | 0.9036 $\pm$ 0.0281 |
| Image Aesthetics ($n=$13,364) |  |  |  |  |  |
| Absolute | 0.6849 $\pm$ 0.0020 | 0.6837 $\pm$ 0.0018 | 0.1351 $\pm$ 0.0153 | 0.7101 $\pm$ 0.0044 | 0.5175 $\pm$ 0.0117 |
| Huber | 0.6827 $\pm$ 0.0016 | 0.6839 $\pm$ 0.0034 | 0.1460 $\pm$ 0.0180 | 0.7077 $\pm$ 0.0083 | 0.5127 $\pm$ 0.0167 |
| Exponential | 0.6869 $\pm$ 0.0016 | 0.6859 $\pm$ 0.0014 | 0.1798 $\pm$ 0.0052 | 0.7156 $\pm$ 0.0036 | 0.5265 $\pm$ 0.0125 |
| Squared | 0.6805 $\pm$ 0.0045 | 0.6794 $\pm$ 0.0046 | 0.1082 $\pm$ 0.0312 | 0.7113 $\pm$ 0.0038 | 0.5188 $\pm$ 0.0139 |
| LIMUC ($n=$11,276) |  |  |  |  |  |
| Absolute | 0.6414 $\pm$ 0.0010 | 0.6406 $\pm$ 0.0011 | 0.1552 $\pm$ 0.0113 | 0.7824 $\pm$ 0.0021 | 0.8625 $\pm$ 0.0033 |
| Huber | 0.6405 $\pm$ 0.0037 | 0.6399 $\pm$ 0.0038 | 0.1914 $\pm$ 0.0136 | 0.7809 $\pm$ 0.0075 | 0.8599 $\pm$ 0.0059 |
| Exponential | 0.6397 $\pm$ 0.0036 | 0.6391 $\pm$ 0.0036 | 0.2286 $\pm$ 0.0150 | 0.7794 $\pm$ 0.0072 | 0.8586 $\pm$ 0.0055 |
| Squared | 0.5205 $\pm$ 0.0098 | 0.5182 $\pm$ 0.0107 | 0.0853 $\pm$ 0.0269 | 0.7785 $\pm$ 0.0064 | 0.8578 $\pm$ 0.0048 |

**Table 4: Ablation study showing impacts of different calibration regularization methods in soft label encoding. Our method is compared with $\mathcal{L}_{\text{MbLS}}$, $\mathcal{L}_{\text{MDCA}}$, and $\mathcal{L}_{\text{ACLS}}$. The results demonstrate the importance of incorporating order-aware conditions in the regularization term, which leads to improved calibration.**
| Combination<br>Evaluation Metrics | SCE($\downarrow$) | ACE($\downarrow$) | ECE($\downarrow$) | Acc($\uparrow$) | QWK($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| Adience ($n=$17,423) |  |  |  |  |  |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{MbLS_REG}}$ | 0.7823 $\pm$ 0.0092 | 0.7788 $\pm$ 0.0092 | 0.0711 $\pm$ 0.0217 | 0.5906 $\pm$ 0.0418 | 0.8988 $\pm$ 0.0298 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{MDCA_REG}}$ | 0.7849 $\pm$ 0.0091 | 0.7810 $\pm$ 0.0100 | 0.0677 $\pm$ 0.0315 | 0.5975 $\pm$ 0.0470 | 0.9024 $\pm$ 0.0289 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{ACLS_REG}}$ | 0.7827 $\pm$ 0.0099 | 0.7791 $\pm$ 0.0100 | 0.0746 $\pm$ 0.0239 | 0.5936 $\pm$ 0.0453 | 0.9000 $\pm$ 0.0322 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{ORCU_REG}}$ (Ours) | 0.4598 $\pm$ 0.0435 | 0.4565 $\pm$ 0.0437 | 0.0583 $\pm$ 0.0279 | 0.5878 $\pm$ 0.0426 | 0.9036 $\pm$ 0.0281 |
| Image Aesthetics ($n=$13,364) |  |  |  |  |  |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{MbLS_REG}}$ | 0.6866 $\pm$ 0.0021 | 0.6852 $\pm$ 0.0023 | 0.1921 $\pm$ 0.0057 | 0.7140 $\pm$ 0.0057 | 0.5106 $\pm$ 0.0116 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{MDCA_REG}}$ | 0.6867 $\pm$ 0.0033 | 0.6859 $\pm$ 0.0035 | 0.1501 $\pm$ 0.0080 | 0.7156 $\pm$ 0.0088 | 0.4960 $\pm$ 0.0192 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{ACLS_REG}}$ | 0.6836 $\pm$ 0.0028 | 0.6819 $\pm$ 0.0033 | 0.1829 $\pm$ 0.0081 | 0.7058 $\pm$ 0.0082 | 0.5033 $\pm$ 0.0130 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{ORCU_REG}}$ (Ours) | 0.6805 $\pm$ 0.0045 | 0.6794 $\pm$ 0.0046 | 0.1082 $\pm$ 0.0312 | 0.7113 $\pm$ 0.0038 | 0.5188 $\pm$ 0.0139 |
| LIMUC ($n=$11,276) |  |  |  |  |  |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{MbLS_REG}}$ | 0.6398 $\pm$ 0.0026 | 0.6386 $\pm$ 0.0026 | 0.1662 $\pm$ 0.0056 | 0.7784 $\pm$ 0.0051 | 0.8564 $\pm$ 0.0035 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{MDCA_REG}}$ | 0.6387 $\pm$ 0.0026 | 0.6368 $\pm$ 0.0027 | 0.1373 $\pm$ 0.0065 | 0.7725 $\pm$ 0.0056 | 0.8544 $\pm$ 0.0043 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{ACLS_REG}}$ | 0.6366 $\pm$ 0.0029 | 0.6349 $\pm$ 0.0030 | 0.1590 $\pm$ 0.0067 | 0.7710 $\pm$ 0.0061 | 0.8532 $\pm$ 0.0052 |
| $\mathcal{L}_{\text{SCE}}$ $+$ $\mathcal{L}_{\text{ORCU_REG}}$ (Ours) | 0.5205 $\pm$ 0.0098 | 0.5182 $\pm$ 0.0107 | 0.0853 $\pm$ 0.0269 | 0.7785 $\pm$ 0.0064 | 0.8578 $\pm$ 0.0048 |

#### 4.3.2 Ablation Studies

Impact of distance metrics on calibration and prediction  The choice of distance metric in soft encoding is a key factor affecting both calibration and prediction performance. Given our focus on achieving reliable, well-calibrated predictions in high-risks ordinal regression tasks, we compared four common distance metrics: Absolute, Squared, Huber, and Exponential. As shown in Table 3, the Squared distance metric consistently provided the best calibration results, particularly in terms of ECE, ACE, and SCE. While Absolute and Exponential metrics showed competitive classification performance on some datasets, they delivered less reliable calibration, making them less suitable for high-risk tasks. In contrast, the Squared metric offered a balanced improvement in both calibration and prediction accuracy, making it the most effective choice for our loss function.

Importance of order-aware regularization method  We conducted an ablation study to assess the effect of different calibration regularization methods applied within the $\mathcal{L}_{\text{SCE}}$. Our approach, which integrates both calibration and unimodality constraints, demonstrated consistently superior calibration results, lower SCE, ACE, and ECE scores (see Table 4). Unlike traditional methods that focus calibration only on the highest probability class and overlook ordinal relationships, our method applies calibration across all labels, ensuring the entire ordinal structure is respected. This approach led to improved calibration metrics (SCE, ACE, ECE) and higher QWK scores, which better capture the model’s ability to reflect the inherent order of labels. Although there was a slight decrease in accuracy, this minimal trade-off is offset by the significant gains in calibration and QWK, making it particularly valuable for high-risk, ordinal classification tasks.

## 5 Conclusion

We introduced ORCU, a unified loss function that integrates calibration and unimodality for ordinal regression. Calibration has been largely overlooked in the literature on ordinal tasks, despite its importance in high-risk applications. ORCU addresses the issue by explicitly targeting calibration improvement in ordinal regression, using comprehensive metrics such as SCE, ACE and ECE to evaluate its effectiveness. By leveraging soft ordinal encoding and order-aware regularization, which simultaneously enforces calibration and unimodality, ORCU balances accurate predictions with well-calibrated confidence estiamtes. Without requiring any architectural changes, our method consistently outperformed the latest loss functions in the domains of calibration and ordinal regression. This work sets a new benchmark for reliable ordinal classification and points the way for future research to optimize regularization parameter $t$ and extend the approach to more diverse and larger datasets, fostering the deployment of more robust calibration methods for a wider range of tasks.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. Many things in the paper are not defined properly. Please clearly specify the definitions of calibration and modularity. 

2. Why MAE is not used as a metric in the experiments, as all the loss functions used are surrogates for it. It would be interesting to see comparison results on MAE.

3. Please explain the regularization term in detail as it is still unclear how it promotes unimodularity. A graphical explanation will also help.

4. A theoretical analysis of calibration and unimodularity of the proposed approach is missing. 

5. Is the proposed approach rank consistent? If not, then how frequently it violates ranking of thresholds. An experimental study on it will be helpful.

6. How does the performance of the model vary  with the variation in thevalue of $t$ used in the regularization term.
soundness: 2
strengths: 1. a novel regularization term is used to promote unimodularity.
2. Paper is well written and easy to read.
summary: In this paper, authors propose an approach for calibration of ordinal regression. They propose a loss function that introduces order-aware calibration They use soft ordinal encoding and label-smoothing-based regularization to enforce both calibration and unimodality. To show the efficiency of the proposed approach, authors propose extensive experimental results on benchmark datasets.
weaknesses: 1. Theoretical proofs of calibration and unimodularity are missing.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. In equation 3, we generally adopt a hyperparameter $\lambda$ to balance the loss and the regularization, like $L = L_1 + \lambda L_2$. Could you explain why this method does not require such a hyperparameter? And why $t$ can control the strength of this regularization?

2. Could you explain why SCE and ACE are improved a lot, but ECE is not?
soundness: 3
strengths: 1. The problem of calibration in the context of ordinal regression sounds novel and important. As far as I know, this work should be the first work to solve this issue.

2. The improvement is significant empirically. From Table 2, we can observe a great improvement in the calibration of ordinal regression models and the classification accuracy is preserved.
summary: In this paper, the authors aim to enhance the confidence calibration of ordinal regression in the training stage. The main challenge of this task is to consider the calibration and unimodality together. To address this challenge, they propose a new loss function for ordinal regression, which combines order-aware calibration with a unimodal regularization term (based on the SORD encoding). In particular, their method enforces both calibration and unimodality by explicitly modeling the ordinal relationships between classes. The effectiveness of their method is validated on three public datasets.
weaknesses: 1. The L_{REG} defined in Equation 2 is not clearly explained. In particular, the design of I(r) is hard to understand for readers. It would be better if the authors could elaborate on how the regularization is constructed.

2. The writing of the gradient analysis in Subsection 3.4 is not clear.  The authors may need to improve the writing in this part, or it might be too challenging for readers to follow.

3. The technical novelty of the proposed method is not presented. While the authors claim that the method considers the unimodality compared to current calibration methods and considers the calibration when compared to current Ordinal losses, I am not clear about if this method is newly designed in each aspect. In other words, the authors may need to show the new insight of calibration part compared to calibration methods.

typos:
1. Line 49, Oridnal -> Ordinal.

The major issue of this work is on the writing: readers cannot easily understand why we should design such a regularization and how t works here. I will improve my score if the authors can make it clear in the revised version.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 2
strengths: •	The motivation behind the new method is well-articulated, clearly highlighting the limitations of traditional cross-entropy (CE) loss in ordinal tasks and the miscalibration in modern ordinal regression models.
•	The proposed method is straightforward, and easy to implement.
summary: The proposed method introduces ORCU, a novel loss function designed to ensure calibration and unimodality in ordinal regression tasks. ORCU leverages soft ordinal encoding and order-aware regularization to produce calibrated and unimodal probability distributions, which are particularly valuable in high-stakes applications requiring reliable confidence estimates and accurate predictions.
weaknesses: 1.	It is unclear how the regularization component of ORCU promotes calibration. Lines 240-252 discuss scenarios where the model is under or overconfident, yet this confidence is based on a soft-encoded distribution not directly related to the data, which raises questions about its reflection of "real" confidence. Additionally, I would appreciate a more rigorous explanation of how this regularization approach aligns with the standard mathematical definition of calibration. Could the authors provide a clearer mathematical justification for this relationship?

2.	The paper claims that CE loss leads to overconfident predictions, yet the reliability diagrams presented indicate underconfident outcomes in the experiments, seemingly contradicting this claim. Temperature scaling, a prominent calibration technique, relies on CE, further challenging the assertion that CE is fundamentally flawed for calibration. Could the authors address this discrepancy and clarify why their results show underconfidence in CE where overconfidence might be expected? Additionally, a nuanced discussion of CE’s strengths and limitations for calibration, especially in light of techniques like temperature scaling, would be valuable.

3.	The evaluation is limited to loss function baselines. Including additional non-loss-based methods for ordinal regression, such as the approach presented in https://arxiv.org/pdf/2303.04547 for unimodality could highlight the unique benefits of ORCU more effectively. I recommend incorporating a discussion on why ORCU and loss function-based methods may offer advantages over such approaches.

4.	The experiments were conducted on only three datasets, which limits the scope for evaluating the method’s robustness across a wider range of ordinal regression tasks. Incorporating a more extensive dataset selection would allow for a better assessment of the generalizability of the approach.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: - Please address the points I raised in the weaknesses part
- The methods that are not based on CE loss - like optimal transport loss - how are the limitations applied to them? 
- Missing definition of  z_{n.k} and intermediate step to get f’=y - p, also r is not defined before it presented 
- Can you elaborate please why the gradient encourages the model to distribute probability across adjacent classes ( as you claim in the sentence in Line 176) compared to the statement in Lines 147-149
soundness: 1
strengths: - This work addresses an important overconfidence issue in ordinal regression tasks
- The proposed loss function is assumed to address both accuracy and confidence of the cross entropy loss based model during optimization without additional post-training calibration
- The authors justify the unimodality enforcement of the proposed loss by gradient analysis
summary: The authors proposed an addition to the cross entropy loss term that  corrects overconfident predictions for incorrect labels. The advantage of the proposed loss is that the calibration is done jointly with the accurate prediction learning during the training optimization and doesn’t require additional post-training steps. The method was evaluated on the 3 datasets and compared against CE-based ordinal models and calibration-loss based and showed an improvement in calibration/accuracy metrics.
weaknesses: **Major**:
- The main focus of the work is CE-loss based ordinal regression which is not an optimal loss for this task and  several methods were proposed without CE loss:  [1-4]
- The motivation in Sec 3.1 is unclear, how the calibration is defined and why it is not implied by CE-loss. The discussion seems to be valid for the ordered nature of classes but not for calibration. It is better to discuss the motivation for each problem separately.
 - $\mathcal{L}_{SCE}$ - the explanation in L175-177 is unclear, how the defined loss encourages what the authors claim - maybe it is explained by Diaz et. al but the manuscript should be self contained with additional clarification. It is also not clear how it helps to reflect the ordinal relationships.
- The Sec. 3.3 in unclear, the explanation and derivation of the loss formula should come before presenting the loss term 
  - why the authors choose it
  - how it helps to ensure calibration
  - what is r ?
  - The L181-183 is unclear.

- Weak evaluation with only 3 small-sample datasets - overall the improvement is incremental so presenting results on more datasets could be beneficial.
- Missing unimodality evaluation - the authors claim the model enforces unimodality - please show it in the results as in [5]
- Missing additional deep ordinal regression baselines [1-5].
- Sec. 3.4 - while it is clear why the loss term enforces unimodality, I’m not sure how it enforces calibration. By saying “*by increasing the gradient for such incorrect predictions, the model is able to reduce the predicted probability for the incorrect class more effectively*” you can claim the same for the standard CE loss. 
- while it could be seen from the results that calibration metrics improved, I’m not sure it is clear from the manuscript why it works.

**Minor**:
- Missing additional deep ordinal regression methods in the related work discussion
- It is better to put Figure 1 closer to the gradient analysis section to make it easier to follow

References:
[1] Liu, X., et al. (2019a). Unimodal-uniform constrained wasserstein training for medical diagnosis. In Proceedings of the IEEE International Conference on Computer Vision Workshops

[2] Beckham, C. et al.  (2017). Unimodal probability distributions for deep ordinal classification.

[3] Wenzhi Cao et. al (2020). Rank Consistent Ordinal Regression for Neural Networks with Application to Age Estimation. Pattern Recognition

[4] Xintong Shi et al. (2021). Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities.

[5] Cardoso, J. S. et. al (2023). Unimodal distributions for ordinal regression