Conformal Prediction Sets for Ordinal Classification
Authors: PRASENJIT DEY
Srujana Merugu
Sivaramakrishnan R Kaveri
Venue: NeurIPS 2023 poster (2023)
Keywords: Ordinal Classification
Conformal Predictions
Unimodal modelling
TL;DR: We propose an approach for  minimal contiguous conformal prediction sets for ordinal classification with cardinality guarantees using non-parametric modelling  of unimodal class distributions

Abstract: Ordinal classification (OC), i.e., labeling instances along classes with a natural ordering, is common in multiple  applications such as size or budget based recommendations and disease severity labeling.  Often in practical scenarios, it is desirable to obtain a small set of likely classes with a guaranteed high chance of including the true class. Recent works on conformal prediction (CP) address this problem for the classification setting with non-ordered labels but the resulting prediction sets (PS) are often non-contiguous and unsuitable for ordinal classification. In this work, we propose a framework to adapt existing CP methods to generate contiguous sets with guaranteed coverage and minimal cardinality. Our framework employs a novel non-parametric approach for modeling unimodal distributions. Empirical results on both synthetic and real-world datasets demonstrate our method outperforms SOTA baselines by 4% on Accuracy@K and 8% on PS size.

Conformal Prediction Sets for Ordinal Classification
Prasenjit Dey prasendx@amazon.com com
Srujana Merugu
smerugu@amazon.coo com
Sivaramakrishnan Kaveri
kavers@amazon.com com
Amazon, India
Abstract
Ordinal classification (OC), i.e., labeling instances along classes with a natural ordering, common multiple applications such as disease severity labeling and size or budget based recommendations. Often in practical scenarios, it is desirable to obtain small set of likely classes with a guaranteed high chance of including the true class. Recent works on conformal prediction (CP) address this problem for the classification setting with non-ordered labels but the resulting prediction sets (PS) are often non-contiguous and unsuitable for ordinal classification. In this work, we propose a framework to adapt existing CP methods to generate contiguous sets with guaranteed coverage and minimal cardinality. Our framework employs novel non-parametric approach for modeling unimodal distributions. Empirical results both synthetic and real-world datasets demonstrate that method outperforms SOTA baselines by 4% on Accuracy @ K and 8% on PS size.
Introduction
large number of practical applications involve ordinal classification (OC), i.e., labeling of instances along a of ordered discrete classes. These include applications such as determining severity level of abuse incidents or disease diagnosis for efficient follow up, predicting customer age-range, apparel size, and budget for improving recommendations and reducing customer cognitive effort by offering adaptive search filters. Often, in such cases, it is desirable to identify a small set of classes (called prediction set or PS) for given input instance such that the true class is amongst the identified set with high probability (called coverage level) instead of a single point estimate of the most likely class. This allows more flexibility in choosing operating points that balance precision- recall trade-off. For instance, customers looking for shoes are often recommended products that are not even available in the customer's 's typical size. An OC model that predicts the customer size would be well suited to improve the recommendations offered. However, limiting the recommendations to a single most likely size is too restrictive because the model accuracy often low due to variations in size over time and across brands. Instead, it is preferable to identify small set of product sizes with say > 90% coverage and use that for automatically filtering the recommendations as shown in Fig. I
Conformal prediction (CP) 36 is an emerging area that addresses the problem of predicting of likely outputs along with measure of the confidence or coverage level. While existing CP methods offer case of deployment and provable coverage guarantees, most of these methods deal with unordered classes and output prediction sets that are non-contiguous and unsuitable for OC settings where the underlying distribution is likely unimodal. For example, for any customer, based on their foot size, a good fit is possible only with small set of proximal shoe sizes resulting in unimodal class distribution. Recommending a non-contiguous set of sizes [4, 8, 9] is not desirable customer experience. On the other hand, expanding the PS to a minimal contiguous superset leads to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

NONU Uncmoddd (t)
Below erage shows Gabery
will based
iooge what ssally
Pobabilty Destaede
Syled REG, 8:31
Tyherr
Trudh
2
30 View
Focted
Footbed
Figure 1: A comparison different model fit underlying unimodal distribution. CP on Unconstrained fit results in non- con-contiguuus sets (red). While CP on unimodal fits results in contiguous set (green), better fitted model can result in the ideal minimal set (blue) which we seek.
significant expansion of the set size (e.g.. (4,8,9) 6, diminishing its application
utility and diluting coverage guarantees, similar contiguity requirement arises in applications such as cancer stage diagnosis where predicting non-contiguous labels (stage-1, stage-4) would be discordant compared to contiguous labels. Note that there do exist OC scenarios such as prediction of day of week for event where unimodality might not be valid as we discuss in Section
While ordinal classification [8] has been well-studied in the past, most earlier works [2
do not assumne unimodality. Recent works 4 22 42 12 15 (00 demonstrate strong benefits of assuming the underlying conditional distribution be unimodal for OC tasks However, these techniques primarily focus on point estimates and not conformal predictions. Furthermore, these methods often either impose a restrictive structure on the model 333 or assume parametric models (e.g., Binomial Gaussian 42 ), which limits their representation ability and utility for real-world applications. Given the multitude of practical applications, there strong need for principled approach that combines ideas from ordinal classification and conformal predictions to construct minimal contiguous prediction sets when the underlying true class distribution is unimodal.
In this work, we focus on ordinal classification where the true class distribution is unimodal. We build on existing work, Adaptive Prediction Set (APS) [35], obtain minimal contiguous prediction sets with provable coverage guarantees. Below we summarize our key contributions:
For ordinal classification where the true class distribution is unimodal, show that any model constrained to output a unimodal distribution can be used with APS algorithm to yield contiguous PS with guaranteed coverage level. Further, prove tight upper bound for the cardinality of this set in terms of the optimal set and the divergence between the fitted model and the true distribution.
2. We provide a novel construction to adapt any DNN-based architecture to always yield unimodal distribution over ordinal classes while ensuring that any arbitrary unimodal class distribution can be approximated.
We study the efficacy of our approach for Conformal Predictions for OC (COPOC) via controlled ablation studies on synthetic datasets which demonstrates the utility of unimodal construction in improving learnability relative for small data sizes, and its versatility compared to other methods that make parametric distributional assumptions.
4. We perform extensive evaluation benchmark datasets for facial age estimation, HCI classifica- tion,image aesthetic score estimation and biomedical image classification with results indicating that method outperforms SOTA baselines by 4% Accuracy @ K and 8% PS size.
2 Contiguous Conformal Prediction Set Problem
Notation:
P[E] denotes the probability of event E. [i] denotes conditioning on integer index Vi, i within a range while x = [m denotes the vector or sequence [.a.... ,2%]-
Let X' denote input space and C = k denote an ordered set of K classes. Consider the case where the random variables X, Y are drawn from X' x C following a joint distribution Px,y such that the true underlying conditional distribution Py x unimodal. In other words, for any x E X. denoting P = (Y = ckl X = x),, there exists a class C. E C such that Pt. I Pk+1> [k] and

---

Pk 2 Ph+1, [k]m Given a training dataset D = { with samples drawn exchangeably (e.g., i..d.) from yx, and a desired coverage level (1 a) (0. 1) for any unseen input test instance (Xtest, Ytest) E Drest drawn from same distribution, the objective is to determine the
minimal con-empty contiguous prediction set (PS) Sp. = fq+1,-.. ,Cu,, <
satisfying marginal coverage with finite samples above a specified level:
P[Ytest E SX.a(Xtear]] 2 (1 - a).
(1)
The above probability is taken over all (Xtest, Ylest) E Drest data points. While marginal coverage is achievable, it differs from conditional coverage: P(Xrest E (Xte( Xteat = Xteat M (1 a),
which is stronger notion. In practice, it is desirable for Sp,aa to approximate conditional coverage and achieve it asymptotically in the limit of large sample sizes.
3 Related Work
Conformal Prediction. Originating in 40 41 conformal prediction is a statistical approach for generating predictive sets with marginal coverage guarantees (Eqn [T with finite samples. Earlier methods for calibrating probabilities (e.g.. via Platt scaling [32 or temperature scaling (17) and assembling a prediction set by including the most likely classes till the covered probability mass exceeds specified coverage level do not achieve the desired theoretical guarantees since the cali- bration is often erroneous (2)]. Most recent CP methods use split conformal prediction that enables APS (35) (9]. and (36] introduce techniques aimed at achieving coverage that is similar across regions of feature space whereas [9] propose methods to achieve equal coverage for each class. Of these, Least Ambiguous Set- Classifier (LAC) 36 provides provably smallest average PS size when probability estimates are approximately correct while APS (35 has been shown be superior in practice. For primer on conformal methods please refer 2 RAPS[]] presents a regularized version of APS for Imagenet. Although relatively new, CP methods are being widely applied for regression, vision, NLP and time-series forecasting 34 38 3 [4]. Our work adapts existing CP methods to unimodal ordinal classification setting with an additional contiguity requirement. Ordinal-APS (25) and 43) share the same motivation but propose a greedy CP method that results in larger prediction sets compared to COPOC (Table
easy deployment with any arbitrary predictor 33 21).
Ordinal Classification. Early works ordinal classification [18 7 451 are based regular classification or regression, ignoring ordinal relationship and non non-uniform separation among labels. Some DNN-based methods model cumulative distribution (CDF) up each ordered class using multiple binary classifiers g 39 28 but the output class probabilities are not necessarily unimodal
and are also not guaranteed to form valid CDF. Label smoothing methods [15 16 24
12
20 42
convert hard target labels into unimodal prior distributions to be used as the reference for the training loss but these methods are often sub-optimal since the assumed priors might not reflect the true distribution, classes might not be equi-spaced categories and additionally test predictions might not necessarily be unimodal. [22 5 learns a non-parametric unimodal distribution as a constraint optimization problem in the loss function which is not only difficult to optimize but also does not guarantee unimodality on test data. In 10], unimodality is guaranteed by assuming Binomial or Poisson distribution but has limited representation ability due to a single degree-of of-freedom which does not capture hetereoscedastic noise. Keeping in view the potential benefits of accurate unimodal distribution learning for the ordinal classification setting, our work explores pon-paraettric flexible DNN-based approach that guarantees unimodality of the predicted class distribution even on test data.
4 Solution Approach
To generate conformal predictions for ordinal classification, we consider two questions: (i) what is the primary drawback of the existing CP methods for classification with respect to ordered labels setting? (ii) how do we address this drawback?
Most CP methods for classification
9 are based on a split calibration approach where the training data D is divided into two sets. The first set Dirain is used to learn probabilistic model P-ix(-) using a blackbox learning algorithm while the second set Drat is used to determine the optimal conformity score threshold for a desired coverage level, which is then used for conformal inference. Below we discuss two common CP methods:
35
36
3

---

Least Ambiguous Set Values Classifier (LAC) [36.. Here, the conformal PS for test instance Xtest constructed using a calibration rule of the form,
ax = {o E Clis((ies)) 2 (iDca(a)))
where (x) = Py x = x = x) is the class probability from the model and @D... (a) is the score threshold defined the bias bias-adjusted (a)th quantile of the model score of the true label.
Adaptive Prediction Set (APS) [5). In APS, for every instance (x, y) Deats we compute conformity score defined as the cumulative probability mass required to include the true label corresponding that instance, i.e., s(x) Ex= Pxx (x), where T is chosen such that Cry- y and æ is the permutation of {1...., K} that sorts Py|x (-Ix) in the descending order from most likely to least likely. Given desired coverage (1 a), we compute a suitable score threshold ID.... (a) Quantile({((( E Deat); [u+1]([-a]] where n = -Deall The conformal PS in
APS is constructed using:
(XLett {Co Cma ... ...Cr, where j = sup jr: Σ Paz (Xeeat) EDen (a) (00) +1.
j1
(3)
More details on the coverage guarantees of APS are in Appendix B While the above methods yield minimal sets with provable guarantees on marginal coverage, the primary drawback is that the resulting prediction sets not necessarily contiguous, which is important requirement for ordinal classification when underlying true distribution is unimodal.
A naive solution is consider the minimal contiguous set that covers the PS output by CP method. For example, if CP outputs {2, 4, 6). we can produce a minimal contiguous set by including "in-between" classes i.e., {2, 3, 4, 5, 6J. However, this would end up having much larger cardinality, which invalidates the tight coverage bounds (refer Appendix B (i.e., the new set will have a coverage much higher than the upper bound guaranteed for that method). Hence, we consider alternative approach based on the observation that if the blackbox learning algorithm in the first step is constrained to output a unimodal distribution, then the classes with probability above threshold will cluster around the mode. The conformal calibration rules in Eqn. [ which are designed to choose classes in the order of class probability will thus result in contiguous sets. Using this insight, propose a solution depicted in Fig. 2 that consists of two steps: (a) design learning algorithm that can accurately model training data such that the output class distribution is always unimodal, (b) adapt an appropriate CP algorithm based on Eqn. 2 or Eqn. 3 to identify the PS. While this high level approach can be used with any of the CP method, our current work, we consider the Adaptive Predictive Set (APS) approach [35] since it provides tight guarantees on marginal coverage and has been empirically shown to achieve better conditional coverage than LAC (36) and smaller set size than CQC [9]. We now describe our solution beginning with study of the properties of the resulting followed by construction of the unimodal classifier (Sec. 4.2)
PS (Sec.
4.1 Conformal Prediction Sets from Unimodal Models
We now study the properties of PS corresponding to the COPOC approach outlined in Fig. where for every input instance X, both the true underlying distribution Pyyx and the learned classification model Pvix are unimodal. In other words, there exists class ca E such that Pu S 0 [k] and Pa 2 pk+1, [k] m. For given Xtest and coverage level (1 a)), let S Soracle (Xtest) denote the minimal contiguous PS that can be constructed by an oracle that knows the underlying Pyix, i.e.
goracle (Xtest)
arg min S= S=(ocpi"" ostcusK, P[
Iu 1].
(4)
cx|X Xteat). This is the set with the least cardinality satisfying contiguity and conditional coverage. In case of multiple solutions, the oracle can pick any one at random. In practice, we do not have access to the true Py|x and (Xins)) has to be constructed from the approximate distribution prix using CP methods such as APS for valid coverage guarantees. Lemma esstablishes the contiguity of the prediction sets resulting from LAC and APS. Thm. | bounds the cardinality of the APS PS relative to the oracle sets. Further, when Pr|x x is a consistent estimator of
where Pt: = Prxx =

---

P(P)))
Prediction Set
Sew Daadd
8 P O0 (
Unimedal Training
Conformal Algorithi
Inference
D
True Distribution Pays
Figure 2: Framework of COPOC.
Figure 3: Unimodal DNN Construction.
the true Pyix, the APS conformity score threshold asymptotically converges to a and the cardinality of SD (Xtest) converges to that of the oracle 35
Lemma Given a fitted wnimodal model Py|x- for any Xtest and a E (0, 11, prediction sets Sp (xeest) constructed using 2 or have at least one solution which is contiguous (amongst multiple possibilities). When Py|x strictly unimodal, all the solutions are contiguous.
Proof Sketch: By design, the prediction sets from Eqn have to contain the top k most likely classes for some k which results in contiguity in case of strict unimodality. When the unimodality is not strict, non-contiguous sets may satisfy the required constraints due to classes with equal probabilities the boundaries, but the shortest span solution is contiguous.
Theorem For any x E X, let px((x) PY = ca|X x) and pr(xx) PYIx(Y = cx|X = x) denote the true and fitted model class probabilities that are always unimodal. Let (x(x)) = - (x) and oz (x(x) =k= dw (x) denote the corresponding cumulative distribu- tion functions. (x())(x) - 0x (x(x)) S 6. [Ki] for constant 6, then any a E (0, 1]. VX E Dtest the APS and oracle prediction sets from Eqn 3 and Eqn. 4 satisfy ($D,a(x)) S I
48- (x)
where n is the size of the calibration set.
Proof Sketch:. To establish the result, we prove the following two statements: (a) ($D,a S 1- (6) (æ), and (b) |Steace (a) (x)) S | प.1 --1- From Eqn. 4 and Lemmal we
observe that the unimodality of p(x) and p(x) leads to the oracle prediction sets and at least one of the APS prediction sets being contiguous. Let SD,a fo ca.. 0 00 i < â S K and u² S K. From the definitions and contiguity, we observe that the probability mass of (x) w.r.t. & equals (da(x) (x)) 2 (a) while that of $1-6 (1- (8)-28 (x) w.r.t equals (α,-(()) or- (x)) 2 (a) + 26. Using the divergence bound on the two CDFs and the fact that SD (x) is the minimal contiguous set with probability mass 2 iDD... (a) as per á yields (a). Using the divergence bound, we also observe that the marginal
Soracle (0) (0) (x) {c-++1. ,c-),00
coverage, Ply E SD (x) 2 qD... (a) - 25. Combining this from the APS coverage guarantees
(Theorem in Appendix B yields part (b).
4.2 Non-Parametric Unimodal Distribution Learning
We now consider the problem of learning classification model that is guaranteed to be unimodal and also expressive enough to accurately model any unimodal distribution. Given universal approximation properties of DNNs with respect probability distributions (26). these form natural choice as a multi-class classification learning algorithm. For a given x, let 2(x) = f f(x; f) denote the output of last fully connected layer of a DNN model f(-). where 0 is the model parameters and z(x) K-ddimensional vector where K = C). Softmax operation is applied to z(x) to obtain class probability distribution. Predicted probability of class ea C for x is given by (x(x)
exp(as(x)) Training is usually performed by minimizing cross-entropy (CE) loss between
Σ (xp(z,(x))
'Detailed proofs are in Appendix B
5

---

this predicted distribution and the ground truth. Since CE loss does not assume any constraints on underlying distribution, can. in principle, model any arbitrary distribution asymptotically with large enough data and model size. When the true underlying distribution Pyxx x is unimodal, this standard learning approach results a classifier x that approximates Pr and itself unimodal assuming large enough data and model size. However, in practice with limited high-dimensional data and limited model capacity, the standard training not guaranteed to reach the true "global minima" " [I]] or even adhere to the unimodality constraints that Icad to contiguous prediction sets. Hence, there is a need for introducing the right inductive bias that truncates the hypothesis class to be searched while ensuring it is expressive enough to include the true hypothesis so as to allow for better learnability and generalization [27]. For our scenario, the set unimodal distributions is the most natural hypothesis class. While previous works 44 42 consider set of parameterised unimodal distributions (e.g.. based on Gaussian or Binomial distributions), we propose a construction that ensures the hypothesis class exactly maps to the set of unimodal distributions on the K K-simplex.
Construction. Let 2/ the set of all discrete unimodal distributions on C. For any x with j(x) = Py = x) E U4 the class probabilities [px (x)] would be unimodal, i.e., there exists a mode Gm E C such that the sequence is monotonically non-decreasing before cm and non-increasing post Ca. The corresponding log probabilities denoted by 24 = (og(Px E R [A]] also form a unimodal sequence with the same mode and are all negative valued.
Let vh: R+ + R - be any strictly monotonically decreasing bijective function, Then, for any 2 there a unique E iR+ such that v(r) = z. Let u (zx), [R] and r = - 1/
(zx): 1
then given [= we obtain a unique" non- nec-eccasn sequence [r Let 1h
R be the "even" extension of (²)), i.e., v/ (r) (b(r for E R+ and vi (r) = (x-r) for R then we have vf (rx) = Zg. Possible choices include (xx = lx|' for any real d. The monotonically increasing [ can be generated from the output vector v of a DNN constrained such that vR > 0. [k]2 using a non-negative transformation é R + I followed by a cumulative summation.
Fig. 3 depicts our proposed construction for any DNN f(-) along with equations below.
n)x = f(x, 0);
01 = (x);; ok = ==<(((x)) [B-12,
exp(ze)
(5)
11 = U1; F = r + Uke [ 2か = = DE = ER= exp(2x)
[b]]
This network can be trained with standard CE loss. From Thm. 2 any distribution in I maps to an output vector nxx) E R², which can be approximated using DNNs with appropriate capacity [26]. Fig. 8inn in Appendix C.4 presents some illustrative examples of Unimodal model fit of COPOC
public dataset.
Theorem 2. Letn: X + RR RR and bt. R R. such that () = [+a)))((((-))) (-r),, Vr e R and its restriction to R+ is a strictly monotonically decreasing bijective function. finction.. (a) Then, the
model output constructed as per Eqn. is always unimodal, i.e., p(x) Ud Vx E X. (b) Further, given any p(x) E U for x X, there exists a well defined function n7(x) E R² that satisfies Eqn. if ((-)) is surjective on Rt.
Proof Sketch: Part (a) follows by observing that the non-decreasing sequence [r*] will result in a unimodal [=x]k=n and unimodal p(x) E l with mode corresponding to sign change in rk sequence. Part (b) follows by constructing a unique inverse for (.) by pivoting on the largest indexed mode in of f(x). For & < m, we choose (x))) E R and for k 2 m, we choose (1 (zL) E R.
5 Experiments
We evaluate the utility of our approach (COPOC) for generating minimal contiguous PS for a desired coverage for ordinal classification multiple datasets. We investigate the following questions:
Note for uniqueness, vi has be defined the largest value of mode in of multiple modes.

---

Table 1: Results on HCI: Mean std. error reported for 5 trials, Best results bolded.
MAE
Acc@
Acc@2
Acc@3
CV%
CE
0.68 0.03
54.3 + 2.6
75.3 4: 3.1
88.9 + 1.6
3.28 +: 0.14
24.4 1.2
POE
0.66 + 0.05
56.5 + 1.8
76.5 + 2.5
89.0 + 2.1
3.1 + 0.18
9.8 + 1.2
SORD
0.65 0.06
56.2 + 2.8
77.1 2.9
89.8 + 2.6
2.96 + 0.19
2.7
AVDL
0.64 0.08
56.8 1.5
77.9 +: 2.4
89.8 +: 1.05
2.98 0.11
2.1 # 1.4
HCI
Binomnaa
0.68 0.05
54.3 1.2
75.8 2.6
88.8 + 1.8
3.01 + 0.16
0
Binomial-temp
0.66 0.04
55.5 1.8
78 2.2
90.1 2.1
2.90 0.11
Uni-loss
0.67 0.09
54.5
+ 3.1
74.8 2.5
88.1 + 2.5
3.05 + 0.38
5.1 + 1.9
COPOC
0.65 0.04
56.5 + 2.0
79.8 + 1.6
91.7 + 2.8
2.66 + 0.13
0
R RQ1: How does COPOC perform relative to SOTA methods on OC tasks?
RQ2: What benefits does COPOC offer relative to parametric modeling on synthetic datasets?
RQ3: Under what conditions does COPOC outperform vanilla model trained with CE loss (V-CE)?
Is COPOC an approximately consistent estimator of the underlying distribution?
RQ4: How does COPOC performance vary with a different choice of conformal inference such as LAC (36 instead of APS?
which also outputs contiguous prediction set (PS) over ordinal labels with regular model training fares against COPOC with it's unimodal training?
RQ5: How does Ordinal-APS 25
5.1 Experimental Setup
We compare our proposed unimodal construction (COPOC) against six SOTA methods as well as modeling with vanilla cross-entropy loss V-CE using APS for conformal inference for all methods for fair comparison, For COPOC, we experimented with various choices of Φ and tb and chose ((x; = |x|| and vi (x:) = - |x| based on performance (see Appendix C.3 The SOTA methods include approaches that (a) utilize soft labels generated from linear exponentially decaying distributions SORD [[12, and linear adaptive Gaussians with variance learning AVDI.
Algorithms:
42), (b) specifically model unimodal distribution through parametric assumptions Binomial [4] and Binomial- Temp, and (c) non- parametrii methods based on unimodality aware loss function Uni-Loss
22 or ordinality imposed in embedding space POE In Sec. 5.5 we also evaluate an alternative conformal procedure LAC
36 with our unimodal construction.
Metrics: Similar to [35 we compute PS Size (IPSI) @90% coverage i.e., = 0.1. Since APS does not guarantee contiguity for models that do not output unimodal distribution, we consider minimal contiguous interval that covers the output PS and report the size. We also report the fraction instances which resulted in non-contiguous PS for which we needed to expand the original non-contiguous set (CV%). To evaluate performance on OC tasks, we report MAE mean absolute difference between predicted and ground-truth and Accuracy@ @ K (Acc@ K) that captures if the ground truth was in the top k predictions. This can benefit unimodal constrained models where classes adjacent to the mode will receive the next greatest probability mass. For comparing different size-stratified coverage violation (SSCV) [I] that measures violations of the conditional coverage property and is suited for high dimensional data.
conformal prediction methods in Sec. 5.3
We evaluate COPOC on four public image datasets: regeddetction (Adience [13), historical image dating (HCI (30). image aesthetics estimation (Aesthetic (37) and biomedical classification
Datasets:
(Retina-MNIST (4)). These datasets contain 8, 5, 5, and 5 ordered classes respectively and the number of instances is 26580, 1325, 15680, and 1600. More details are included Appendix C. We also experimented with synthetic data with the generation process and result discussed in Sec 5.3
5.2 RQ1: Performance on ordinal classification on real world datasets
Table presents comparison of COPOC with other methods on the HCI dataset. Results on other datasets included in the Appendix C.T due to space constraints. COPOC performs at par with other baselines in terms MAE & Acc@ and significantly outperforms them on Acc@2, Acc@3 & JPS| ( 8% smaller compared to the next best one). We observe that V-CE and POE have the highest
7

---

Table 2: Results on Synthetic Data: We report KL-Div, PS Size [Ground truth Oracle size], CV%
and MAE. Mean and std. eтror is reported across 10 random trials. Best mean results are bolded.
V-CE
SORD
AVDL
Binomial
Binomial-temp
Uni-Loss
COPOC
KL Div.
0.04 + 0.01
0.03 + 0.01
0.07 + 0.01
0.1 + 0.02
0.09 + 0.01
0.12 + 0.10
0.04 = 0.01
IPSI [3.03]
3.09 0.06
3.09 + 0.04
3.12 + 0.05
3.16 + 0.04
3.16 0.04
3.61 + 0.15
3.09 + 0.05
D1
CVS
0.01
1.7 0.4
MAE
0.02
0.65 0.01
0.67 +:
0.68 0.02
0.69 0.01
0.03
0.65 0.02
KI. Div.
0.06 F: 0.01
0.17 0.02
0.18 + 0.04
0.19 4: 0.08
0.17 0.02
0.26 + 0.11
0.04 + 0.01
IPSI [2.56]
2.65 0.00
2.72 + 0.04
2.78 + 0.04
2.85 + 0.06
2.81 + 0.02
3.12 + 0.09
2.59 + 0.01
CV%
0.05
0.3 0.1
0,6 0.1
0
2.9 0.7
MAE
0.56 0.01
0.59 0.01
0.60 + 0.01
0.61 0.02
0.61 0.01
0.63 + 0.04
0.57 0.02
KI. Div.
0.13++0.00
0.38 0.02
0.17 0.01
0.49
0.08
0.44 0.02
0.4 +: 0.08
0.09 + 0.01
IPSI [1.56]
1.73
1.96 0.0G
1.85 0.04
2.39 + 0.04
2.38 0.02
2.35 = 0.1
1.66 : 0.02
D3
CVS
0.1
1.1
0.8 0.2
0
2.7 + 0.4
MAE
0.24 0.02
0.25 + 0.00
0.23 + 0.03
0.28 + 0.01
0,26 + 0.02
0.27 + 0.04
0.23 + 0.02
KL Div.
0.01
0.00
0.31 + 0.02
0.44 + 0.04
0.35 0.02
0.35 - 0.1
0.08 + 0.01
IPSI [4.40]
4.67
4.73 0.04
4.78 + 0.05
4.83 0.04
4.82
5.04 +: 0.2
4.50 0.02
D4
CVS:
0.7
2.7
2,8
5.6 + 1.1
MAE
1.26 +: 0.02
1.27 0.03
1.27 +: 0.02
1.31 + 0.04
1.29 + 0.02
1.30 + 0.03
1.24 + 0.01
contiguity violations CV%. Though SORD, AVDL, and Uni-loss enforce unimodality in training, it does not necessarily translate to unimodality in test samples (indicated by high CV%) resulting in poor performance on Ace @ 2 and Acc@3 and IPSI, which is critical in OC tasks. Since Binomial-temp enforces unimodality by construction, it performs better than the above methods on Acc@k. However, unimodality constraint on DNN in COPOC results in even better model fit for OC tasks which translates to higher Ace @ K & shorter [PS].
5.3
RQ2: Ablative Study of Unimodal Modeling Methods on Synthetic Data
To generate synthetic data, consider the label set C of size 10, i.e., Y (cI....,,uu) and input space X C R where X₃ Uniform 10) and X2, . X10 follow independent normal distribution. Further, we define hx()) 100 + (sin (0.2+ +0,
To generate the labels, consider conditional distribution function Q that generates Z with h(X) as distributional parameters and a mapping function g : R > C that maps Z to the output label Y E C. Thus, for each z, we generate
2z)x(1) and y g(2). We consider two choices of mapping functions g(-). The first choice Jequi partitions the range of Z into 10 equi- bins which can be mapped to the label set in a linear order while the second choice 9non-equi partitions the range of Z into 10 bins of arbitrary width to assign labels. From the construction, equi-spaced categories while Inon Inon-equs -egui does not. We consider the following four synthetic datasets in increasing order of complexity and train models (details in Appendix C.2
can see that Jegui considers classes to be
DI: Q2jx (I) = Double-Exponential with mean at h(x) and constant variance; g = 9equi-
D2: Qzix (x) Double-Exponential with mean at h(x) and constant variance; 9 = Inon-equi-
D3: Qzix (x) Gaussian with mean and variance varying with h(z); g Inon eegun
D4: Q21x (x) Gaussian.Poisson double Exponential with mean and variance varying with h(x) chosen at random for each x; g Inon-egui-
Since we have the true distribution Py|x, we compute the Oracle PS size and compare it with that of other methods. We report KL Divergence (KL) between the true and predicted distributions with lower KI. indicating better model fit. Table 2 shows that for DI, SORD fits the data well with lowest KL. and MAE as it explicitly models exponential distribution assuming all classes to be equi-spaced. However, on D2 and D3, COPOC outperforms all the other methods on the model fit and also results in shorter IPSI closer to Oracle. On D2 and D3, SORD performs poorly because it assumes both constant variance across input spaces and equi-distant classes which are not valid for these datasets. Since D3 draws samples from Gaussian distribution with heteroskedastic noise, AVDL outperforms SORD but is inferior to COPOC. Since D4 is drawn from more complex distribution, COPOC outperforms the rest with an even larger margin. The Binomial variants underperform throughout due to distributional
8

---

32
Figure 4: Comparison of KL Div. indicates superior model fit of COPOC vs. V-CE for smaller train size with conver- gence as data size increases.
Figure 5:
Comparison of IPSI and CV% indicates that COPOC is superior against V- CE with smaller train size.
Figure LAC vs APS com- parison by varying a. (Top) IPS) and (Bottom) SSCV.
mismatch while Uni-Loss exhibits high variance indicating convergence issues. As with the real benchmarks, enforcing unimodality via loss or soft-labels does not guarantee unimodailty on test data, indicated by high CV%. Interestingly V-CE performs exceedingly well throughout in terms of KL beating more sophisticated methods. The overall observation is that the performance of the methods depends largely on the validity of the assumptions and relatively unconstrained nature of COPOC makes it more versatile.
5.4 RQ3: COPOC vs Vanilla-CE and Consistency Properties
We compare COPOC against V-CE by varying train-data size (= N) on synthetic data 14. Fraction of samples used to train each model is 3/4, and rest is used for calibration and test. All experiments are repeated 10 times with random realization of train set. Comparing the KL loss on train set in Fig. 4 we observe that COPOC has lower KL in early stages with lesser data and performance 80K and KL asymptotically becomes negligible. Fig.
for both converges when N reaches compares IPSI and CV% by varying N on D4. IPSI of COPOC approaches those of Oracle with far lesser samples compared VC-E. Thus, COPOC with its inductive bias fits the data better with lesser samples compared to unbiased CE yielding shorter sets with finite samples. Above results also suggest that COPOC might be approximately consistent. Note that with increased N, V-CE results in a better model fit, lower CV% and shorter sets, indicating that these are correlated.
5.5 RQ4: Choice of Conformal Inference: LAC vs. APS
Table provides comparison of PS size and size-stration strattiadiieed coverage violation(SSCC) of APS against LAC when applied on the output our proposed unimodal model for a = 0.1 Both methods generate contiguous prediction sets. Al- though I.AC consistently produces shorter sets, also has slightly worse SSCV metries across datasets. In Fig. 6 plot IPSI and SSCV across different a for synthetic data D4 and results are similar. We observe that LAC achieves the small- est prediction set size but sacrifices adaptiveness (conditional coverage) in the process.
Table 3: LAC vs APS with COPOC on image datasets. Mean is reported after 100 trials.
LAC
APS IPSI
IPSI
SSCV
SSCV
HCI
2.61
0.09
2.66
0.09
Adience
2.42
0.13
2.52
0.10
Aesthetic
1.58
0.19
1.65
0.07
RetinaMnist
3.08
0.24
3.14
0.09
5.6
RQS: COPOC vs Ordinal-APS
25
Table shows an empirical comparison of COPOC against APS and Ordinal-APS applied over Vanilla DNN trained with Cross-entropy loss (V-CE) on public datasets and synthetic data D4 described in Sec. and For V-CE with APS we consider minimal contiguous interval that covers the output PS and report its size. We observe that Ordinal-APS produces significantly shorter sets compared to V-CE with APS However, COPOC significantly outperforms Ordinal-APS because of better unimodal data fit. Note that Ordinal-APS outputs contiguous PS over ordinal labels

---

Table 4: Ordinal-APS [25 vs COPOC on Synthetic data D4 and public image datasets. Mean and std. error is reported after 10 trials.
APS
Ordinal-APS
COPOC
Synthetic D4
4.67 0.03
4.59 0.03
4.50 + 0.02
HCI
3.28 = 0.14
3.03 4 0.15
2.66
Adience
4.82 = 0.24
2.67 + 0.12
2.26 0.06
Aesthetic
1.96 = 0.2
1.77 = 0.05
1.70 0.06
Retina MNIST
3.6 + 0.08
3.28 = 0.02
3.03 0.01
irrespective of whether the posterior distribution generated by the model is unimodal or not. COPOC with unimodal regularization results in better model fit and hence shorter PS sizes.
6 Validity of the Unimodality Assumption
Unimodality assumption might not be universally applicable for all OC scenarios (eg., prediction of preference ratings, event-hour-of-daa etc.). How- ever, there do exist a large number of OC ap- plications where it is beneficial to assume uni- modality as validated by multiple notable works
Table 5: Comparison of COPOC against V-CE in terms of NLL. Mcan and std. error is reported after 10 trials.
V-CE
COPOC
in computer vision domain [12 20
42 4. Ta-
HCI
1.59 + 0.15
ble [ shows the negative log-likelihood (NL.I)
Adience
2.33 1 0.18
1.66 0.21
of vanilla DNN (V-CE) fitted with CE loss (un- constrained fit) and COPOC (unimodal fit) on four public datasets described in Sec. 5.2 The
Aesthetic
1.49 + 0.01
0.71 0.02
Retina MNIST
+ 0.04
1.23 + 0.04
superior fit of COPOC indicated by lower NLL justifies the unimodality assumption for these datasets. COPOC makes assumption the underlying distribution being unimodal and one could potentially check the validity of the assumption by comparing the likelihood the unimodal and unconstrained fits. Note that even if the unimodality assumption is not true, the theoretical coverage guarantees of the prediction set produced by COPOC would still hold but the cardinality bounds would be weaker since the fitted distribution would deviate significantly from the true underlying distribution.
7 Conclusion, Broader Impact, and Limitations
We proposed an approach to construct minimal contiguous prediction sets for ordinal classification with guarantees on coverage and cardinality along with empirical validation of efficacy on both real and synthetic datasets. Our solution employs novel architecture for -on-paraeetric modelling of arbitrary unimodal class distributions without sacrificing representation ability. With ML-driven systems playing an increasingly important role in our society, it is critical to provide practical guarantees on the reliability and uncertainty of the ML. model output. While existing CP methods B5 address this concern for the general classification setting, there is need for specialized solutions for application scenarios with inherent ordinality and unimodality in class labels, e.g., predicting cancer stage of biopsy signals. Our work makes an important contribution in this direction and also provides a way to effectively communicate the limitations of the model adaptively across the input space with bigger prediction sets for harder instances. The proposed methods have numerous practical applications in improving recommendations based on an ordered attribute (c.g., budget), and reducing human supervision effort for ordinal classification tasks. Future directions include extensions to tasks where classes are partially ordered or hierarchically grouped (e.g., topic hierarchies). Consistency of proposed unimodal construction is also worth investigating.
Limitations:
COPOC makes an assumption on the underlying distribution being unimodal and might lead to sub-optimal fit if the assumption does not hold. Furthermore, it can be paired only with CP methods such as LAC and APS where the prediction sets can be viewed as upper-level sets of the predicted class probabilities, i.e., set of classes with probability higher than threshold. Due to the reliance on the CP methods, the coverage guarantees are also valid only only when the assumption on the exchangeability of the data points holds true.
10

---

References
[1] A. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint 20XY::2009.11993 2020.
[2] A. N. Angelopoulos and S. Bates. A gentle introduction to conformal prediction and distribution- free uncertainty quantification. arXiv preprint arXiv:2107.075// 2021.
[3] A. N. Angelopoulos, A. P. Kohli, S. Bates, M. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano. image-to-imagee regression with distribution-free uncertainty quantification and applications in imaging. In International Conference on Machine Learning, pages 717-730. PMI.R, 2022.
[4] C. Beckham and C. Pal. Unimodal probability distributions for deep ordinal classification. In International Conference on Machine Learning, pages 41 1-419. PMLR, 2017.
[5] S. Belharbi, I. B. Ayed, L. McCaffrey, and E. Granger. Non Non-parametric uni- uni-modality constraints for deep ordinal classification. arXiv preprint arXiv: 2019.
[6] A. Berg, M. Oskarsson, and M. O'Connor. Deep ordinal regression with label diversity. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2740-2747. IEEE, 2021.
[7] R. Can Malli, M. Aygun, and H. Kemal Ekenel. Apparent age estimation using ensemble of deep learning models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 9-16, 2016.
[8] W. Cao, V. Mirjalili, and Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 140:325-331, 2020.
[9] M. Cauchois, S. Gupta, and J. C. Duchi. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. The Journal of Machine Learning Research, 22(1):36813377 2021.
[10] P. da Costa, H. Alonso, and S. Cardoso. The unimodal model for the classification of ordinal data. Neural Networks, 21(1):78-91, 2008.
[11] A. Daniely, N. Linial, and S. Shalev-Shwartz. From average case complexity to improper learning complexity. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing, pages 441-448, 2014.
[12] R. Diaz and A. Marathe. Soft labels for ordinal regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7778-4747,, 2019.
[13] E. Eidinger, R. Enbar, and T. Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions on information forensics and security, (12):2170-2179, 2014.
[14] A. Fisch, T. Schuster, T. Jaakkola, and R. Barzilay. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning, pages 3329-3339, PMLR, 2021.
[15] B.-B. Gao, C. Xing, C.-W. Xie, J. Wu, and X. Geng. Deep label distribution learning with label ambiguity. IEEE Transactions on Image Processing. 26(6): 28388, 2017.
[16] X. Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28(7):1734-1748, 2016.
[17] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks, In International conference on machine learning, pages 1321-1330. PMLR, 2017.
[18] G. Guo, Y. Fu, C. R. Dyer, and T. S. Huang. Image-based human age estimation by manifold learning and locally adjusted robust regression. IEEE Transactions on Image Processing. 17(7):1178-1 188, 2008
[19] Y. Hechtlinger, B. Póczos, and L. Wasserman. Cautious deep learning. arXiv preprint arXiv: 09460, 2018

---

[20] Z. Huo, X. Yang, C. Xing. Y. Zhou, P. Hou, J. Lv. and X. Geng. Deep age distribution learning for apparent age estimation. In Proceedings of the IEEE conference on computer vision and paltern recognition workshops, pages 17-24, 2016.
[21] J. Lei, M. G'Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman. Distribution-free predictive predicci inference for regression. Journal of the American Statistical Association, 113(523):1094-11 1111, 2018.
[22] Q. Li, J. Wang, Z. Yao, Li, P. Yang, J. Yan, C. Wang. and S. Pu. Unimodal loss: Fully adaptive label distribution learning for ordinal regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20513 2022.
[23] W. Li, X. Huang, J. Lu, J. Feng, and J. Zhou. Learning probabilistic ordinal embeddings for uncertainty-awest regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13896 1 13905, 2021.
[24] X. Liu, X. Han, Y. Qiao, Y. Ge, S. Li, and J. Lu. Unimodal-unifomm constrained wasserstein training for medical diagnosis. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pages 0-0, 2019.
[25] C. Lu, A. N. Angelopoulos, and S. Pomerantz. Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2022, Proceedings, Part VIII, page 545-554, Berlin, Heidelberg, 2022. Springer-Verlar Verlag.
[26] Y. Lu and J. Lu. A universal approximation theorem of deep neural networks for expressing probability distributions, Advances in neural information processing systems, 33:3094-3105, 2020.
[27] B. Neyshabur, R. Tomioka, and Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv: 1412,664/ 2014.
[28] Z. Niu, M. Zhou, L. Wang. X. Gao, and G. Hua. Ordinal regression with multiple output cnn for age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1920-4928, 2016.
[29] Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran. Measuring calibration in deep learning. In CVPR workshops, volume 2, 2019.
[30] F. Palermo, J. Hays, and A. A. Efros Dating historical color images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12, pages 499-512. Springer, 2012.
[31] H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13, pages 345-356. Springer, 2002.
[32] J. Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
[33] J. D. Rennie and N. Srebro. Loss functions for preference levels: Regression with discrete or- dered labels. In Proceedings of the IJCAI multidisciplinary workshop on advances in preference handling, volume 1. AAAI Press, Menlo Park, CA. 2005.
[34] Y. Romano, E. Patterson, and E. Candes. Conformalized quantile regression. Advances in neural information processing systems, 32, 2019.
[35] Y. Romano, M. Sesia, and E. Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems, 33:3581-3591, 2020.
[36] M. Sadinle, J. Lei, and L. Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association, 114(525):223 234, 2019.
12

---

[37] R. Schifanella, M. Redi, and L. M. Aiello. An image is worth more than thousand favorites: Surfacing the hidden beauty of flickr pictures. In Proceedings of the international AAAI conference on web and social media, volume 9, pages 397-406, 2015.
[38] K. Stankeviciute, A. M Alaa, and M. van der Schaar. Conformal time-series forecasting. Advances in Neural Information Processing Systems,
2021.
[39] T. Vishnu, P. Malhotra, L. Vig, G. Shroff, et al. Data-driven prognostics with predictive uncertainty estimation using ensemble of deep ordinal regression models. International Journal of Prognostics and Health Management, 10(4), 2019.
[40] V. Vovk, A. Gammerman, and C. Saunders. Machine-lcarning applications of algorithmic randomness. 1999.
[41] V. Vovk, A. Gammerman, and Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
[42] X. Wen, B. Li, H. Guo, Z. Liu, G. Hu, M. Tang, and J. Wang. Adaptive variance based label distribution learning for facial age estimation. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXIII 16, pages 379-395. Springer, 2020.
[43] Y. Xu, W. Guo, and Z. Wei. Conformal risk control for ordinal classification. In R. J. Evans and I. Shpitser, editors, Proceedings of the Thirty-Ninth Conference Uncerlainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 2346-2355 PMLR, 31 Jul-04 Aug 2023.
[44] Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and Ni. Medmnist v2-a sarge-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data, 10(1):41, 2023.
[45] D. Yi, Z. Lei, and Z. Li. Age estimation by multi-scale convolutional network. In Com- puter Vision-ACCV 2014: 12th Asian Conference Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part III 12, pages 144-158. Springer, 2015.
13

---

APPENDIX
A
Broader Impact
Our current work contiguous conformal predictions for ordinal classification is foundational in nature and has multiple real-world applications.
Cancer Diagnosis. Given the huge costs of misprediction for high-stakes applications such cancer diagnosis, instead of a single point prediction it is useful to predict contiguous set. For instance, prediction set of [stage 2, stage 3] gives a better notion of severity than non-contiguous set such as [no cancer, stage 3] which might be discordant point prediction with low accuracy.
Dynamic Product Search Filters. Customers new to any e-commerce platform often experience heavy cognitive load in specifying their requirements via search filters (e.g.. budget, product dimensions). Identifying a small highly likely set of options based on their typical profile or immediate session history would significantly enhance the usability of the search filters and improve the customer experience.
Personalised Fit Recommendations, Shopping apparel and shoes at any e-commerce platform is often tedious due to the limited support for fit-based recommendations Often, customers find that the recommended products do not have options for their size and are forced to use search filters, which need to be repeatedly specified for each query. Addition- ally, customers also tend to order multiple products in the same size (bracketing) that results in a high return rate and excessive shipping costs for the platform. Automatic identification the likely size ranges of a customer would improve the accuracy recommendations and reduce shopping effort as well as return rates.
Personalised Budget Recommendations, Since budget ranges have a natural ordering, automatic personalisation of product and brand recommendations for customers based on their preferred budget ranges is another area that can leverage COPOC to improve customer satisfaction.
Abuse Incident Audits. E-commerce abuse incidents often categorised along severity levels that have a natural ordering. Typically, human auditors are required to audit the abuse incidents, but current models do often distinguish between high chance of low severity incident vs. moderate chance of high severity incident. Conformal predictions can help streamline the audit workflows better focus on the high severity incidents and optimise the overall outcomes both for e-commerce platform and the customers through expedited resolution.
B Theoretical Analysis
Lemma Given a fitted unimodal model Py|x, for any test x and a E (0,1), prediction sets (x) constructed using 6q or has least one solution which is contiguous (amongst multiple possibilities). When Pyix strictly unimodal, all the solutions are contiguous.
Proof: Let Sp (x) be the prediction set with the shortest span (i.e., difference between highest and lowest included labels) as per Eqn 6or
Let 1 and u denote the smallest and largest indices of the labels included in Šp,a (x) so that the span is given by u u.
Assuming (x) is non-contiguous implies that there exists at least one kakip such that (1 [+1) 1) < sskip < u and Ca-kip ŠD,a (x). Let Px (x) (Y ca|X x). Since p(x) is unimodal, there two possible scenarios depending on where kekip relies relative to the mode Gr. of p::
kship < m: In this case, we have PA.KiP 2 P141 since is non-decreasing before the mode kship 2 mi In this case, we have PaukiP 2 Pa. since P is non-increasing after the mode
14

---

Thus, Paokip 2 min(pe+1.
Case I: LAC- PS Construction follows Eqn. 6 For this case, have,
SD (x) - {cr Cliss 2 (a)a:
(6)
where qDe. (a) the bias-adjin adjusted (a)th quantile of the model score of the true label. Since C++1 and Co are included in Sp aD,x(x)) (x), it follows that both Pi+1 2 @De (a) and P. 2 9D (a).
Since PREkuP 2 min(ft+1, Pu), follows that PR (D (a) as well implying that Cgoksp SD,a which leads to contradiction. Hence, the shortest span SD,a(x) has be contiguous. Case 2: APS- PS Construction follows Eqn.[] For this case, we have,
SD.a(x) {(TT CTgy ... Cryy where j = sup > Eim < @Dear (10 1=
where is a permutation of {l...., }} that sorts py in the descending order from most likely to least likely and @D.. (a) is the bias-adjusted (1 aa) quantile of the APS conformity scores as defined for Eqn. 7
Let Paum (S) Enyes Pa denote the (fitted) probability mass within the prediction set S. Due the unimodality of p. it follows that one of the boundary labels Cn and C++1 have the minimum probability among those included in the set SD,0 (x). Without loss of generality, let assume pu one of the minima (since the same argument can be applied for the case where (I + 1) is among the minima).
From the construction, we have, P Peum Peum(Sp.o (Sp.a(x)) 2 QD.... (ar and P Pram ($p, aa (x)) ((w)) < ÂD... (a). Consider the sets S = SD ((cpene)) } {cu) and S2 Si {Cemin } where k-mn is the largest index satisfying g-in arg min [px]. Since Prenii 2 min(pr+1, Pu), follows that Psum (Si) 2
e (a). Further, from the definition of j as the size of largest top is set with probability mass as defined in Eqn. 7 it follows that Pa (S2) < ()D..((0) (a))
Therefore, the set S2 is valid APS prediction set well with span (kmin -)) (u I), which leads contradiction. Thus, the shortest span ŠD,a (x) has be contiguous for this case as well. Thus, in both cases, there exists at least one solution, i.e., shortest span prediction set, which is contiguous for both the constructions.
For the case, where ý is strictly unimodal, from the constructio Eqn 6 or 7 the prediction sets have to contain the top most likely classes for some which results contiguity in case of strict unimodality.
Theorem 1 For any X, let p&(x): Pr =x cx|X x) and px(x) P
Cx|X x) denote the true and fitted model class probabilities that always unimodal. Let Th = Σ (x) and (x(xx) Ek' (x) denote the corresponding cumulative distribu- tion functions. If (x)))xxx= -x (x) S ö, [K] for constant ò, then for any a E (0, Vx E Dtest- the APS and oracle prediction sets from Equ and Eqn. 4 satisfy (x),a(x) S |
(x)l
where n is the size of the calibration set.
Proof. To establish the result, we prove that the following two statements hold true under the assumption on the CDFs of Py and Prix:
(a) (S)aa < / Oraale
1-10 (2) (x)]
(b)
gorade
-pp (a)-25 (x)) < S
Part (a): From Eqn. 4 and Lemma we observe that the unimodality of p(x) and p(x) leads to
the oracle prediction set being contiguous and also the existence of a contiguous APS prediction set.
15

---

Since all the APS solution sets per Equ 7 have the same cardinality, we use SD,a(x) to denote the contiguous solution.
Let SD, 10 , caj. 0 05i< 1" < u² < K. From the definition of the sets and the contiguity, we observe that the probability mass of (x))(x) w.r.t. p equals (ō,(x) - 0x(x)) 2 (D... (a) while that of Sp
K and Spracle
91-40 qut (2) +1 Cu. J. 0 S
oar (2)-28 (x) w.r.t
p equals (aw- (x) or- (x)) 2 (1 - @Deal (a) 26) = @D... (a) 28.
Using the divergence bound the two CDFs, i.e. 0 (x) 8 (x)] S 6, [M:] we have
(o (x) - 51- (x)) 2 (ou- (x) - 6) - (lar-
+ +
Tu (x) 01- (x) 28
@De. (a) + 28
@D.. (a)-
Since Sp (x) is the minimal contiguous set with probability mass greater than or or equal to (Deat (a)) as per p in Eqn 7 we have
(x)a (û ( S (u 11)) | 1040 (a) (0)-
Spraate
Part (b)) Denoting the minimal contiguous APS prediction set by SD (x) before, have
(x)(x) (x(x) 2 @D... (a). Considering the divergence bound on the two CDFs, i.e., (ox(x)
(x(x)] S 6, [K] we have (ōa(x) - -x(x)) - (oa(x) + ((ax(x)- 6) = aa (x) - ox(x) + 28.
Hence, for all x, have
(ā (x) ox(x)) 2 4D... (ax) (x(x) 0x(x) + 25 2 iD cat (a)
(a(x) - ox(x) 2 iD.ci (a)
Since this holds for all x, the marginal coverage Ply E SD)a(X) 2 (a) 25.
From Theorem 3 we also have an upper bound on the marginal coverage for test samples, i.e., P[Y E a. & a + I+I where n is the size of the calibration set.
Hence, have
1
1 a+
PIY (p)a
n n+l 1
1
1 a +
4D... (a) - -5
n n+1 1
1
1 اD (a) 25 48
n +
From the above incquality and the definition of the oracle prediction set, we observe that
gorade
-1-90 11- (a) 26 (x) S
goracte 4-45-
Combining the results in part (a) and (b), we have
garade <
48
As the size of the calibration set increases, the term I+1 vanishes and as the divergence & decreases, then the cardinality the APS set converges to that of the oracle set.
16

---

Theorem Let X + RRR and th. R such that (r) = + Ar E R and its restriction to RJ is strictly monotonically decreasing bijective function. (a) Then, the model output constructed as per Eqn. is always unimodal, i.e., p(x) U, Vx E X. (b) Further,
given any p(x) U for x E X, there exists a function n(x) E R that satisfies Equ. surjective on R*.
if (() is
Proof. We begin by restating the construction:
7xx) f(x, 0); (x); tk <((n((x)) (x)), [k]2
= U1; - Dk. 1 2x (ra); Pv = exp(2x)
Σx=1 k= exp(zx)
P2
Px
SOFTMAX
2K
V R - R
v²
(-r)
F2
Fx
1/2
V2
PK
Ø: R
R
Input
Figure 7: Construction of our DNN
Part a: Following the above construction, for any x X, since é R + R+ the DNN output
2h 2 0, [b] Hence, the cumulative sum sequence is non-decreasing, i.e., T1 I S ... S rg-
There can be possible scenarios:
Scenario 1. ri S r2 I rk S 0 this case, [2x- (rx k is also non-decreasing sequence and is [px k= Here [polk_ is unimodal sequence with mode at сg-
Scenario 2. 0 < S r2- <= TK- In this case, [. (rx))k is also a non-increasing sequence and so is k Here [p is unimodal sequence with mode at c₁.
Scenario 3. 11 < T2 < Tm S 0 < Tm+1 S ... S Tk for some m. In this case, [ = (rx))) nen-eecegg till m and non-increasing from 1 onwards, which makes it unimodal. The mode is either m or m + or both depending on the magnitudes rm and |l The probability distribution p K follows the same pattern and is unimodal as well.
Part b: Let us assume p(x) E 2/ is any arbitrary unimodal distribution conditioned on with class probabilities Pa < Pk+1, [x] and Pa 2 pk+11 [K]K where is the highest indexed (in case of multiple) mode of the unimodal distribution. We can then obtain z = lox( [Ki] and construct
where тą E R for < k S (m - 1) and rı E R+ for m S k S K. Since vb : R + R is a strictly monotonically decreasing bijective function and y is it's even extension, the sequence [r*] is well-defined. Further, since [=<]/[] is unimodal sequence, [rx] is monotonically increasing with mm-1 S 0 S Fm- Then,
the sequence Fą = (
obtain the vector such [ K and ii = r1- When <(-) is surjective function on R+, we can define n (x) - (o) (vk), [k) and 7n (x) = U1- There will always be a valid 77)), which ensures
that U% Fk Tk- 0,
that construction can generate the original ((x).
17

---

B.1 APS Coverage guarantees
Theorem 3. [APS [35]
If samples (I,, vs) 2, X.y: E y are exchangeable V1 S n and all samples from D Dirain: Deat are invariant to permutations. and conformity scores are almost surely distinct, then APS algorithm gives tight marginal coverage given by:
1 - S Plytest E @Spa(XXestt) S 11-ar + Deall +
C Experiment Details
C.1 Benchmark Image Datasets and Implementation Details
We now provide a brief description of the four public datasets and the modeling details. For each of these datasets, we split the data into train, calibration, and test sets. We use calibration set to calibrate APS and report mean and standard deviation (std. error) on the test set across 5 independent splits. Note that for all experiments to avoid ever-fitting, data augmentation, i.e., random horizontal flipping and random cropping for each training image, was applied in our experiments. The predictions was obtained with central crop during testing. COPOC was implemented with ø |a|| and --]]
Age Estimation Adience [3] J: The task associated with this dataset is predict the age for a given facial image. This dataset contains 26580 Flickr photos of 2284 subjects. The age is annotated with eight groups: 0 2, 4 6, 13, 15 20, 25 - 32, 38 43, 48 - 53, and over 60 years old. From the nature of the class labels, it is evident that classes are not equally spaced categories. Hence, previous works which assumed it be equi-spaced (SORD [2] for instance) are suboptimal. For feature extractor backbone, we use ImageNet pre-trained VGG-16 16 network since most competing methods used this model. For our usage we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2 We trained models for 50 epochs with a batch size of 64. For optimization, Adam optimizer was utilized with a learning rate of 0.0001, with decay rate of 0.2.
[23 [2]
Historical Colour Image Dating- HCI [30]: The historical color image dataset is collected for the task of estimating the age of historical color photos. Each image is annotated with its associated decade, where five decades from the 1930s to 1970s are considered. There are 265 images for each category. Following (3) we utilized VGG-16 as the backbone, which was initialized with the ImageNet pre-trained weights for a fair comparison. We trained models for 50 epochs with Adam optimizer with learning rate of 0.0001, with decay rate of 0.2. For COPOC, we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2
Retina-MNIST -NNIST [4]: RetinaMNIST is based on the DeepDRiD24 challenge, which provides dataset of 1600 retina fundus images. The task is ordinal classification for 5-level grading of diabetic retinopathy severity. We use similar feature extractor network used in [44] along with a final unimodality constrained layer at end. The network was trained with same settings as [4].
Image Aesthetic Estimation 377: The Aesthetics dataset consists of 15687 Flickr image belonging to four different nominal categories: animals, urban, people, and nature. All The pictures are anno- tated by 5 different graders in aesthetic categories in an orderly manner: 1) "unacceptable" pictures with extremely low quality, 2) "flawed" low quality images (slightly blurred, suurred.ove/lundeeeppsse))) and with no artistic value; 3) "'oddinay"" images without technical flaws (well framed, in focus), but no artistic value; 4) "professional" images (flawless ramiinglightting appealing images, showing outstanding quality. The ground truth label for each image is set to be the median among all of its gradings. Following [23] [2] we use ImageNet pre- trainadne VGG-16 16 as the backbone for feature extraction. For our usage, append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2 We only report aggregate metric across all the categories for this data.
5) "exceptional", very
On the public benchmark datasets where the official best hyperparameters are available for baseline methods (For instance in Adience, HCI and Aesthetic dataset best settings for POE and SORD, and for Binomial best settings on Adience were available) from the corresponding authors work or code, we directly use those settings. We were able to replicate the results (MAE and Accuracy) on these datasets as reported by them. For all other cases (namely AVDL, Uni-Loss) we optimize for MAE in hyperparameters search since that is the
Additional Implementation Details:
18

---

most common metric used for all ordinal classification tasks across competing benchmarks. We cross-validate over the following grid:
learning rate E {le 2, le -3, 43 with decay rate of 0.2.
weight decay E [0, le - 1e -1e-222 -, -e
dropout rate E {0.1, 0.25, 0.5, 0.75).
Adam optimizer with default settings.
Few additional algorithm specific hyperparameters that needed tuning were:
For POE, there are two extra hyperparameters of a and / in its distance-aware loss function
in embedding space which we search over {1c 3, 1c 4, lc 5, le 6} as suggested by the authors.
SORD describes three type of distance metric losses absolute inter class distance, squared distance and its log variant. We search over these loss functions too.
AVDL requires choosing the initial variance (of the Gaussian) of all images which we search from (0.25, 0.5, 1, 2] similar their work.
Uni-loss has hyperparameter that controls the weightage between unimodality and mean- variance component of its loss function. We search over {10, 100, 500, 1000, 5000}.
Table 6: Results on Image Benchmark Datasets: Mean and std. error is reported for 5 trials. Best mean results bolded.
MAE
Acc@1 1
Acc@2
Acc@3
IPSI
CV%
V-CE
0.68 0.03
54.3
2.6
75.3 + 3.1
88.9 + 1.6
3.28 0.14
24.4 + 1.2
POE
0.05
56.5
76.5
89.0
3.1 0.18
1.2
SORD
0.65 = 0.06
56.2
2 2.8
77.1 2.9
89.8 + 2.6
2.96 0.19
2.7 1.1
AVDL.
0.64 0.08
56.8 + 1.5
77.9 + 2.4
89.8 1.05
2.98 + 0.11
2.1 +
HCT
Binomial
0.05
54.5
1.2
75.8 2.6.
88.8 -
3.01 0.16
Binomial-temp
0.66 = 0.04
55.5 = 1.8
78 13:::2 2.2
90.1 2.1
2.90 = 0.11
Uni-loss
0.67 0.09
54.5 3.1
74.8 2.5
88, 1 + 2.5
3.05 0.38
5.1 +
COPOC
0.04
V-CE
0.57 0.07
38.1 1.6
91.4
4.82 0.24
21.4 + 2.2
POE
0.48 : 0.05
60.3 : 1.5
84.1 2.0
93.9 :: 2.3
4.16 1166100
12.8 1.2
SORD
0.48 0.06
59.9 3.8
85.2 1 2.9
94.3 +: 1.6
2.86 + 0.09
3.7 + 1.1
AVDL
0.49 0.03
3.1
94.0
0.15
4.1 +
Adience
Binomial
0.5 0.5
60.0 1: 1.2
86 :::::11 1.8
95.4 1.9
2.5 0.06
Binomial-tep
0.48 + 0.04
60.5 2.1
86.4 :: 1.2
95.6 1:64110 1.3
0.005
Uni-loss
0.64 0.14
51.5
89.4
3.14
COPOC
0.49 + 0.04
61.0 :1 1.9
86 :::::11 1.5
96.1 06.11:20 2.2
2.26 1: .06
V-CE
0.29 0.01
71.4 1.6
94.6 6
97.8 +
1.96 = 0.2
7.9
POE
0.28 +: 0.05
72.1 1.5
94.1 +:: 1.1
98.0 :1: 0.1
1.85 0.11
7.85 + 0.9
SORD
0.29 0.02
72.0
95.2
98.3
L.78
AVDL
0.28 0.03
72.2 1.5
95,2
98.5
1.75 0.05
0.3
Aesthetic
Binomial
0.31 + 0.01
69.5 0.7
93.1 +. 2.8
96.0 :: 0.9
1.83 1: 0.06
Binomial-temp
0.32 + 0.04
69
93.0
96.2 +
1.89 1. 0.09
Uni-loss
0.37 + 0.14
66.8 5.0
92.0 3.8
97.4
1.94
2.1
COPOC
0.28 1 0.04
72.0 1: 1.3
95.9 1.0
99.0 -: 0.2
1.70 1: .06
0
V-CE
0.73 + 0.02
52.2 - 0.6
72.2 + 0.1
86.0 -: 0.5
3.6 = 0.08
9.8 :: 2.4
POE
0.73 0.02
32.4 + 0.4
72.5 0.6
86.4 0.8
3.4 0.05
6.4 2.8
SORD
0.71 = 0.01
53.5
70.5 +
84.5 -
0.03
3.9
AVDL
0.72 = 0.02
53.0 2:0::00 0.2
71.0 - 0.4
81.6 - 0.9
3.21
3.8 + 1.2
Retina-MNIST
Binomial
0.71 0.01
32.7 0.2
69.7 + 0.6
83.7
3.33 0.02
Binomial-temp
0.70 0.02
53.0 - 0.2
70.5 00.5:40..5
$4.0
3.3 0.02
Uni-loss
0.74 = 0.05
52.0 2:00:00 1.1
72.5:::03
84.55 - 1.5
3.25 - 0.1
4.2 = 1.1
COPOC
0.71 0.01
53.5 0.2
72.5 0.6
87.0 +: 0.3
3.03 +: 0.01
19

---

Result Discussion: : We highlight the key takeaways from Table 6
1. COPOC performs at par with SOTA baselines in terms of MAE and Acc@1.
2. Benefit of COPOC comes with improved gains in Acc&2 and Acca3. Apart from COPOC, there is no single method that performs consistently across the 4 datasets in terms of these metrics. For instance in HCI and Adience, Binomial-temp comes closest to COPOC, but on Aesthetic, both variants of Binomial severely under-perform whereas AVDL and SORD perform quite well and comes the closest to COPOC. In contrast, on -NNIS, non- porammmmttiic models such as V-CE,POE. Uni-loss have Acc@k @k close to COPOC and beat other parametric models significantly. This shows that parametric distribution assumption in any underlying model fits the data well when the data is actually drawn from a similar distribution. Since most methods depend largely on the validity of the assumptions, the relatively unconstrained parameter-free nature of COPOC is more robust and allows it to consistently outperform across datasets.
3. There is a strong correlation between CV% and PS size. This is expected because higher CV% indicates more number of cases for which we had to predict a minimal contiguous super set, thus inflating the size of PS. Better unimodal fit by underlying model is bound to have lesser CV% and and thus, shorter sets. Hence, COPOC again outperforms all other baselines across datasets in term of IPSI consistently, Although Binomial model variants has 0 CV% due to it's construction, it still produces larger sets than COPOC as seen in HCI and Adience. This can be because COPOC results in better unimodal fit which is also idicated by higher Acc@
4. Enforcing unimodality in training scheme in terms of soft-labels (VORD,AVDL) or in loss function (Uni-loss) in embedding space (POF) does not necessarily translate to unimodal distribution in test samples which is indicated by high CV%.
5. Although V - CC in principle should have been able to model any underlying distribution, on high dimensional real- world datasets it fails miserably. This shows the need for injecting prior "bias" into training network like COPOC which aids the model in reaching the optima.
Uni-loss has issues with model convergence as it shows high variance across metrics for all datasets. This could be because its sensitive to X hyperparamter that control the weightage between unimodality and mean-variance component of its loss function which difficult tune.
7. Datasets with higher accuracy results in shorter PS size in general, which is expected. For instance Aesthetic has lower PS size across methods compared to HCI or Retina-MNIST both having same number of class labels.
C.2 Implementation details of experiments on synthetic Data
For all the results synthetic datasets presented in S.c we employ same DNN network across all the methods for fair comparison. To be precise, we use layer DNN architecture having 128 hidden dimensions with a dropout of 0.2. We use the same training paradigm as before Adam optimizer with learning rate of 0.001 and batch size of 512 trained for 500 epochs ensuring convergence. We divide the data into 70% train and 30% test splits. We train our model 10 times for each independent split of the data. For each test set, we again randomly split into calibration for APS and evaluate IPSI on final-test and repeat this 100 times to ensure conver- gence of PS. We use & = |a|] and yp = - |||| for COPOC. We report mean and standard error in Table 2
C.3 Ablation study on the choice of ((-) and v/ (.) for COPOC
Although there can be many possible choices for (d-) and (-) in the COPOC construction Eqn. 5 in practice not all choices leads to good model convergence. In this section, we perform comparison of few common choices and present results in Table 7 We train the model for synthetic data D4 as described in Sec. [5.3 We train the model using CE loss with same model capacity and training paradigm as described in Appendix C.2. We report train CE loss and since we have access to true
underlying distribution for D4 we report KL. Div. to measure goodness of model fit, Below we summarise few observations:
20

---

Table 7: Ablation study on implementation choice of d(-) and (b(-) for COPOC. We report mean results across 10 trials.
Train loss
KL Div.
ReLU, v
3.89
0.24
Φ Softplus, V0-
3.11
0.19
Ø x², 1/
xx
3.48
0.2
rl. 1
1.64
0.04
r), th =x²
2.20
0.13
r & -1x U..
1.89
0.1
1. d -U maps most of [ux] zeroes which results in flat probability distribution for most of the data points while Φ Spllpus instead maps most [vx]2 which again results in almost flat distribution for most points. With x² observed unusually large values for [ux]2 resulting in unstable training. ø = |e| gives a good balance as each [7412 gets linearly mapped [uw]
K to very small values
2. 3/ |x|2 tends over-emphasize higher probability classes in the model fitting while / = - Jr| 0.5 under-emphasizes them, Again since & = |o]] does a linear transformation of Fk either side of the origin it gives good balanced estimate of 2k-
C.4 Illustrative Comparison Between Unimodal and Non-Unimodal Model Fit
Naive (V-CE)
COPOC
Ground Truth
Predicted P
Cq C. Cq Class Label (Y)
G
6
C. Class Label (V)
Class Label (Y)
Figure 8: Predicted probability distributions of (naive) unconstrained DNN trained with vanilla Cross entropy loss (V-CE) (Top) and COPOC (bottom) three datapoints from the public benchmark Adience-Age Estimation task dataset [13,, which is annotated with 8 class labels with age groups as: 0-2, 4-6, 8-13, 15-20, 25-32, 38-43, 48-53, and over 60 years old. Ground truth class is highlighted in red. We present few difficult data instances where both models fail to predict the correct class as the top one. However COPOC with its unimodality bias is able predict the correct class within top 2 or 3 predictions leading to better Accuracy K compared to naive DNN
21


Reviews:
[{"review_id": "zzML1ve0ck", "reviewer": "Reviewer_ieEd", "summary": "The paper addresses the problem of adapting the conformal prediction methods to ordinal classification so that the predictor outputs contiguous prediction sets. Contributions can be split into two parts. The first part deals with adopting the existing conformal prediction methods to ordinal classification. The main observation is that the predictor with unimodal posterior over classes is enough to guarantee that the conformal prediction methods will return contiguous prediction sets. This part is relatively straightforward. The second part involves proposal of a novel non-parametric method for training NN which outputs unimodal posterior over class labels. The proposed method is empirically evaluated and shown to provide competitive results when compared to existing ordinal classification methods used for conformal prediction.", "questions": "In the experiments on real data, tab 1, the proposed method does not perform best in terms of MAE which might be attributed to the fact that true posterior might not be exactly unimodal. It would be instructive to see whether this changes on the synthetic data, which presumably are generated from an unimodal distribution (although whether it is true is not clear to me; the description should be more explicit in this respect). I.e. I would suggest to report also MAE in table 5.3.\n\n---\nThe authors satisfactorily addressed my questions in the rebuttal. I keep my positive ratings.", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is sound and it is very clearly written. \n\nA new non-parametric method for training an NN-based ordinal regression predictor with unimodal posterior over class labels is a simple elegant idea, which is to my knowledge novel and potentially useful in practice; not only in the context of confomal prediction.", "weaknesses": "I have no major objections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments. Please find our response below  \n  \n**Comment:**  \n \nIn the experiments on real data in Table 1, the proposed method does not perform best in terms of MAE which might be attributed to the fact that true posterior might not be exactly unimodal.  \n \n**Response:**\n \nThe reviewer's insight is likely true. For real datasets in Table 4 (in appendix), we do observe that COPOC performs at par with other state-of-the-art baselines in terms of MAE. This is also evident from the fact this model has overlapping error intervals with other state-of-the-art models in terms of MAE. The real benefit of COPOC is in improving the Acc@2 and 3 using its unimodality bias.  \n  \n**Comment:**\n   \nIt would be instructive to see whether this changes on the synthetic data, which presumably are generated from an unimodal distribution (although whether it is true is not clear to me; the description should be more explicit in this respect). I would suggest to report also MAE in table 5.3.  \n   \n**Response:**\n \nWe would like to apologize for the lack of clarity on the synthetic data. All synthetic datasets D1-D4 in Sec 5.3 (Table 2) are indeed drawn from various unimodal distributions. The goal was to do an ablation study of our proposed non-parametric unimodal DNN model against other baseline models for different underlying unimodal data distributions.  Below we present the MAE metrics for the datasets D1-D4 in Table 2 in Sec. 5.3  of our paper. Mean and std. error is reported across 10 random trials. Best mean results are bolded. Labels in the table are as described in Table 2 of Sec 5.3  \n\n|    | V-CE               | SORD               | AVDL               | Binomial       | Binomial-temp  | Uni-loss       | COPOC              |  \n |----|--------------------|--------------------|--------------------|----------------|----------------|----------------|--------------------|  \n | D1 | **0.65** &pm; 0.02 | **0.65** &pm; 0.01 | 0.67 &pm; 0.02     | 0.68 &pm; 0.02 | 0.69 &pm; 0.01 | 0.68 &pm; 0.03 | **0.65** &pm; 0.02 |  \n | D2 | **0.56** &pm; 0.01 |     0.59 &pm; 0.01 | 0.60 &pm; 0.01     | 0.61 &pm; 0.02 | 0.61 &pm; 0.01 | 0.63 &pm; 0.04 | 0.57 &pm; 0.02     |  \n | D3 | 0.24 &pm; 0.02     | 0.25 &pm; 0.02     | **0.23** &pm; 0.03 | 0.28 &pm; 0.01 | 0.26 &pm; 0.02 | 0.27 &pm; 0.04 | **0.23** &pm; 0.02 |  \n | D4 | 1.26 &pm; 0.02     | 1.27 &pm; 0.03     | 1.27 &pm; 0.02     | 1.31 &pm; 0.04 | 1.29 &pm; 0.02 | 1.30 &pm; 0.03 | **1.24** &pm; 0.01 |\n\n  \nOur conclusion from the above table is quite similar to what we have presented in Sec. 5.3. For dataset _D1_, _SORD_ fits the data well and has the lowest MAE as it explicitly models exponential distribution assuming all classes to be equi-spaced. Similarly for D3, AVDL performs well as samples in D3 are drawn from Gaussian distribution which AVDL explicitly models.  COPOC matches the best performance in all the datasets (including the complex dataset D4) except on D2 where it is slightly inferior to Vanilla-cross entropy fit. Thus, we can conclude the performance of the methods depends largely on the validity of the underlying data distribution assumptions and the relatively unconstrained nature of COPOC makes it more versatile. Interestingly, simple vanilla cross-entropy loss model _V-CE_ performs almost at par with other sophisticated baselines in terms of MAE which we did not observe with real data. This could be due to the fact these synthetic datasets lie in a low-dimensional space (10 D), and actual benefits of sophisticated baselines are seen on high dimensional image data.  \n   \nPlease let us know if you have any other questions or if there is anything else that we could add to further improve the submission"}, {"review_id": "V6vL7ERiBK", "reviewer": "Reviewer_vR8Y", "summary": "The authors explained the difference between their method and prior work satisfyingly.\nI had a misunderstanding in my previous reading.\nI'd be happy to support the paper's acceptance.\nIt is a minor contribution from a theoretical perspective, but I agree that practically, it's probably a better way of constructing unimodal prediction sets than Lu et al. \n_______________\nThe paper proposes a modification of conformal prediction for the ordinal classification case.\nThe idea is that, when the distribution is unimodal, you should always be building continuous intervals; so you restrict the family of possible prediction sets to be the ones that are only contiguous.", "questions": "Can the authors clarify my comment about novelty? If I have misunderstood, I would be happy to hear it.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The paper is relatively clear, and the problem is important.", "weaknesses": "Post-rebuttal note: I was wrong about the below.\nThe methods are definitely related, but not identical.\nLu et al. applies the constraint of unimodality during the set construction.\nThe present manuscript does so during the model training.\n_______\nThe method is not novel, and was previously proposed in this reference:\n\nLu, C., Angelopoulos, A. N., & Pomerantz, S. (2022, September). Improving trustworthiness of AI disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 545-554). Cham: Springer Nature Switzerland.\n\n(Equation 2 in https://arxiv.org/abs/2207.02238 is the same as Equation 1 in the present manuscript.)\n\nThere has also been since a followup of that work on risk control: https://openreview.net/forum?id=9R5qObx8WO5 .\n\nFor this reason, I do not believe the paper contains any novel methodological advances, and thus cannot be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for pointing us to Lu et. al. ‘22 and Yunpeng et. al. ’23, both of which we were not aware of earlier. While the motivation for both these papers (namely conformal predictions for ordinal settings through contiguous prediction sets) overlaps with that of ours, our solution approach and contributions are significantly different as we clarify below.\n \n**Comment:** \n   \nThe method is not novel, and was previously proposed in Lu et. al.\n    \n **Response**: \n   \n Lu et. al. propose a new conformal prediction (CP) method to output a contiguous prediction set (PS) over ordinal labels irrespective of whether the posterior distribution generated by the model is unimodal or not.  Similar to typical conformal prediction methods, Lu et. al’s ordinal APS approach involves a calibration step followed by inference. In the calibration step, the conformal threshold $\\lambda (\\alpha)$ for the desired coverage $(1-\\alpha)$ is learned using a calibration dataset following the equation before Theorem 1 in Lu et. al. During inference, given a desired coverage $(1-\\alpha)$, the prediction set for a given instance is constructed by starting at the label with highest predicted score and progressively expanding the intervals on either side till the covered probability mass exceed $\\lambda(\\alpha)$ (Algorithm 1). However, this greedy construction can often lead to large PS sizes when the posterior distributions are far from unimodal (e.g., a non-unimodal output prediction distribution [0.2, 0.1, 0.4, 0.3] over 4 ordinal classes will result in PS size of 4 to include 0.9 probability mass, while a unimodal distribution [0.1, 0.2, 0.4, 0.3] would result in PS size of 3. Figure 2 in the submitted pdf (in global rebuttal) shows the three steps (training, calibration, inference) of COPOC and Lu et. al. \n \n\n**Differences in contributions**\n\n- **Conformal  Calibration and Inference:** Unlike Lu et. al., we do not propose any  new CP method and instead leverage well established works of APS (Romano et. al. '20),  LAC (Sadinle et. al. '19). We show that in case of ordinal classification where  the true class distribution is unimodal, any model  constrained to output a unimodal distribution can be used with APS, LAC or  similar conformal prediction algorithms to yield contiguous PS with a  guaranteed coverage level. \n- **Unimodal  Training:** Our key contribution is a novel non-parametric method for  training DNN which is guaranteed to output a unimodal posterior over class  labels while ensuring that any arbitrary unimodal class distribution can  be approximated (Theorem 2 of our paper). Lu et. al. does not propose any modification to the model  training step.\n- **Theoretical  Results:** In Theorem 1 of our paper, we provide a tight upper bound for the  cardinality of the PS generated by APS on the top of unimodal posterior  predicted distribution in terms of the optimal set.  On the other hand, Theorem 1 in Lu et. al. is a standard coverage guarantee that follows from Vovk et. al. '99 without any bound on PS cardinality size.\n- **Empirical  Results:** We provide empirical results on synthetic and real-world datasets  comparing different approaches to achieve unimodality and SOTA CP  methods to gain insights on different methods and demonstrate the efficacy  of COPOC. Lu et. al.  focuses  primarily on the spine stenosis severity prediction comparing their  proposed Ordinal APS with LAC.\n\n**Relative  Performance of the methods:**  \nTable  below shows an empirical comparison of  COPOC  against  APS and Ordinal APS of Lu et. al. applied  over a Vanilla DNN trained with Cross-entropy loss (V-CE) on synthetic data D4 and public  datasets mentioned in Sec. 5 of our manuscript. For V-CE with APS we consider a minimal contiguous interval that covers the output PS and report its size.\n\n|              | V-CE with APS  | V-CE with Ordinal-APS of Lu et. al. | COPOC          |\n|--------------|----------------|-----------------|----------------|\n| Synthetic-D4 | 4.67 &pm; 0.03 | 4.59 &pm; 0.03  | 4.50 &pm; 0.02 |\n| HCI          | 3.28 &pm; 0.14 | 3.03 &pm; 0.15  | 2.66 &pm; 0.13 |\n| Adience      | 4.82 &pm; 0.24 | 2.67 &pm; 0.12  | 2.26 &pm; 0.06 |\n| Aesthetic    | 1.96 &pm; 0.2  | 1.77 &pm; 0.05  | 1.70 &pm; 0.06 |\n| Retina MNIST | 3.6 &pm; 0.08  | 3.28 &pm; 0.02  | 3.03 &pm; 0.01 |\n \nWe observe that Lu et. al. produces significantly shorter sets compared to V-CE with APS . However, COPOC significantly outperforms Lu et. al. across all datasets because of better unimodal data fit.\n\n**Comment:** \n \nEquation 2 in Lu et. al. is the same as Equation 1 in the present manuscript. \n \n**Response:**  \n \nWe presume the reviewer is referring to Equation 3 in Lu et. al. matching that of Equation 1 in the current paper. This is the definition of marginal coverage. It is  natural that the basic notations and equations defining marginal and conditional coverage are similar as these are common to conformal prediction literature, and based on the seminal work of (Vovk et. al. '99).\n\n**Comment:** \n\nThere has also been since a followup of that work on risk control [Yunpeng et. al. ’23].\n\n**Response:**  \n\nThe cited follow up work was published recently in UAI’23 [Aug 1st-3rd] much after the NeurIPS submission deadline. Furthermore, the primary new contribution of Yunpeng et. al. is extending Lu et. al. to the case where classes have differential weights. It is in fact orthogonal to our work and ideas proposed in Yunpeng et. al. can also be applied along with our COPOC method to accommodate differential weighting of classes.\n \nWe will definitely cite Lu et. al. and Yunpeng et. al. in the revised version and add the experimental comparison with Lu et. al. as additional baseline.\n \nWe respectfully request that the reviewer assess our contributions again and consider revising the score. Please do let us know if there is anything else that we can do to clarify or improve our submission."}, {"review_id": "RGQ9cCWXjN", "reviewer": "Reviewer_Qwu5", "summary": "The authors present a method for ordinal classification, COPOC, which guarantees (by functional form) unimodal prediction distributions over the ordered classes and consequently guarantees contiguous prediction sets for uncertainty estimation via conformal prediction . The authors argue that unimodality is desirable for many ordinal classification tasks, e.g. the prediction set for customer shoe sizes should be something like size 5,6,7 rather than e.g. sizes 4,8,9.  A theoretical result is obtained which bounds the size of the fitted model prediction set in terms of the size of the ground truth prediction set as well as the distance between the fitted prediction distribution and the ground truth prediction distribution for a given desired conformal prediction coverage level. Favorable experimental results in comparison to competing methods are presented on a suite of real-world ordinal image classification datasets.  COPOC is compared to competing techniques on a variety of metrics, including accuracy, size of prediction set and contiguity of prediction set. Experiments on synthetic datasets are used to illustrate the consistency of COPOC as well as to examine the merits of different possible choices of conformal prediction algorithm.", "questions": "Did you optimize the hyperparameters of each competing method on some sort of validation set before doing the final test set evaluation for each method? If so, for which of the various metrics reported was the hyperparameter optimization conducted?", "limitations": "I think the limitations are adequately discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, the work is original to the best of my knowledge.\n\nI find the writing and presentation to be clear for the most part. \n \nThe architecture presented in section 4.2 seems well-designed and does indeed seem to have just the right level of inductive bias, if we take for granted that unimodal predictive distributions are desirable. It guarantees unimodality without making additional parametric assumptions.\n\nAlthough I would have preferred more detail on the real world experiments, the breadth of results and metrics evaluated seems adequate to me. 4 datasets evaluated on 6 metrics is a reasonably thorough set of experiments.\n\nAlthough I would not characterize the topic as being highly significant or extremely important, the significance seems adequate to me for NeurIPs acceptance.  Conformal prediction is a highly useful technique in which interest continues to grow. Ordinal classification, while not a top priority for many modelers, is important enough in certain applications that its intersection with conformal prediction is a worthwhile topic.  ", "weaknesses": "Although unimodal prediction distributions may often be desirable for ordinal classification, I find the case for unimodality to be overstated. Although the authors do acknowledge on line 357 (in the limitations  section) that unimodality might not hold, I would have preferred a bit more discussion of the scenarios where it might not hold. For instance, I have some experience with recommendation systems (Netflix-style 5 star ratings) and there are merits to models in that domain which might detect \"love-it-or-hate-it\" scenarios where both 1-star and 5-star are more likely than 3-star.  Even for cancer stage detection, I think it's possible that non-unimodal predictions could be desirable. One could imagine that there are 2 possible hypotheses explaining an observed symptom. If the observed symptom is due to a preexisting condition unrelated to the cancer, then the overall evidence might point to stage 1. If the observed symptom is due to the cancer itself, stage 4 might be implied. So more of an acknowledgement that unimodality might lead you astray would be good here.\n\nI also find the details of the real-world-dataset experiments to be a bit lacking. There is minimal discussion of hyperparameter optimization. I would have more confidence in the results if I knew a thorough attempt was made to optimize the hyperparameters of each method. Although I don't have a detailed understanding of the competing methods, the nonparametric ones (POE, Uni-Loss) would (at first glance) appear to have some hyperparameters associated with them and it's not clear to me whether those hyperparameters were optimized.  I think more details on the real-world experiments and less detail on the synthetic experiments would be a better use of the 9 pages allowed.\n\nSome suggested typo-ish edits:\n\nLine 137 consider minimal contiguous set -> consider the minimal contiguous set\nLine 169 has->have\nLine 242 of underlying distribution -> of the underlying distribution\nLine 261 Accuracy@k lowercase k vs capital K earlier (line 12 in abstract) is inconsistent. \n\n***** Update post -rebuttal ***\n\nIn light of the additional information provided by the authors (both regarding the appropriateness of the unimodality assumption and the hyperparameter tuning), I have raised my score to a 7. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments. Please find our response below  \n  \n**Comment:**\n  \nI find the case for unimodality to be overstated... So more of an acknowledgement that unimodality might lead you astray would be good here. \n \n**Response:**\n   \nWe completely agree with the reviewer that the unimodality assumption might not be universally applicable for all ordinal classification scenarios. The Netflix ratings scenario mentioned by the reviewer is an apt example. However, there do exist a large number of critical ordinal classification applications where it is beneficial to assume unimodality as validated by multiple notable works in computer vision domain [4, 22, 41, 12, 15, 20]. Examples include medical diagnosis applications such as cancer stage detection. In this case, the symptoms or indicative factors are often real-valued or ordinal variables such as the size of the tumor, number of cells in the tumor, the amount of cancer in mammary, axillary, and sentinal lymph nodes, the number of lymph nodes involved, the number of cancer-afflicted organs, which are monotonically related to the target label i.e the stage of cancer, which makes it reasonable to assume a unimodal class distribution.  \n  \nThe table below shows the negative log-likelihood (NLL) of vanilla DNN fitted with cross-entropy loss (V-CE) and COPOC on four real datasets from our paper. The superior fit of COPOC indicated by lower NLL justifies the unimodality assumption for these datasets.  \n  \n|              | V-CE           | COPOC          |\n|--------------|----------------|----------------|\n| HCI          | 1.73 &pm; 0.13 | 1.59 &pm; 0.15 |\n| Adience      | 2.33 &pm; 0.18 | 1.66 &pm; 0.21 |\n| Aesthetic    | 1.49 &pm; 0.01 | 0.71 &pm; 0.02  |\n| Retina MNIST | 1.24 &pm; 0.04 | 1.23 &pm; 0.04 |\n\nWe do, however, acknowledge the reviewer’s point. In the revised introduction, we will definitely mention ordinal classification examples (e.g., prediction of preference ratings, event-hour-of-day) where unimodality does not hold.  \n  \nWe have also pointed out in the limitation section of our manuscript that COPOC makes an assumption on the underlying distribution being unimodal and might lead to a sub-optimal fit if the assumption does not hold. We will additionally mention that one could potentially check the validity of the assumption by comparing the likelihood of the unimodal and unconstrained fits.  \n  \nPlease do note that even if the unimodality assumption is not true, the theoretical coverage guarantees of the prediction set produced by COPOC would still hold but the bound itself is weaker since the fitted distribution deviates significantly from the true underlying distribution.  \n  \n**Comment:**\n   \nDid you optimize the hyperparameters of each competing method on some sort of validation set before doing the final test set evaluation for each method? If so, for which of the various metrics reported was the hyperparameter optimization conducted? \n \n**Response:**  \n \nWe have presented some of the key implementation details (feature extractor backbone and training procedures) on real-world datasets and experiments in appendix C.1. We do acknowledge that there might be some ambiguity on the hyperparameters. We want to apologize the reviewer for the same and would like to take this opportunity to clarify.  \n  \nOn the public benchmark datasets where the official best hyperparameters are available for baseline methods (For instance in Adience, HCI and Aesthetic dataset best settings for POE and SORD, and for Binomial best settings on Adience were available) from the corresponding authors work or code, we directly use those settings. We were able to replicate the results (MAE and Accuracy) on these datasets as reported by them. For all other cases (namely AVDL, Uni-Loss), we optimize for MAE in hyperparameters search since that is the most common metric used for all ordinal classification tasks across competing benchmarks. We cross-validate over the following grid:  \n \n-   learning rate ∈ {1e−2, 1e−3, 1e−4} with decay rate of 0.2.\n-   weight decay ∈ {0, 1e−3, 1e−2, 1e−1}\n-   dropout rate ∈ {0.1, 0.25, 0.5, 0.75}\n-   Adam optimizer with default settings\n \nFew additional algorithm specific hyperparameters that were missed in the main manuscript and needed tuning were:\n \n-   For POE, there are two extra hyperparameters of $\\alpha$ and $\\beta$ in its distance-aware loss function in embedding space which we search over {1e−3, 1e−4, 1e-5, 1e-6} as suggested by the authors.\n-   SORD describes three type of distance metric losses - absolute inter class distance, squared distance and its log variant. We search over these loss functions too.\n-   AVDL requires choosing the initial variance (of the Gaussian) of all images which we search from {0.25,0.5,1,2} similar to their work.\n-   Uni-loss has $\\lambda$ hyperparameter that controls the weightage between unimodality and mean-variance component of its loss function. We search $\\lambda$ over {10,100,500,1000,5000}.\n  \n \n**Comment:**\n   \nI think more details on the real-world experiments and less detail on the synthetic experiments would be a better use of the 9 pages allowed.\n  \n**Response:**\n   \nThanks for the feedback. We had to perform an ablation study on synthetically generated data drawn from various unimodal distribution to study the efficacy of our proposed non-parametric unimodal DNN model against other state-of-the-art baseline models. In the revised version, we will attempt to better balance the placement of content between the main paper and the appendix.\n   \n**Comment:**\n   \nSome suggested typo-ish edits: … inconsistent\n \n**Response:**\n   \nWe sincerely appreciate the reviewer’s careful reading of our submission and will fix the typos. \n \nWe respectfully request that the reviewer assess our contributions again and consider increasing the score. Please do let us know if there is anything else that we can do to clarify or improve our submission."}, {"review_id": "0kqog35S0m", "reviewer": "Reviewer_w3W6", "summary": "Conformal prediction for classification considers non-ordinal classes, which potentially produces sub-optimal prediction set size for ordinal classification. The proposed approach addresses this problem for a unimodal label distribution. To this end, the proposed approach designs a novel conformity score function that fully utilizes the unimodal assumption, so a trained score function strictly returns a unimodal distribution over labels. The efficacy of the proposed approach is theoretically justified in theorems and empirically demonstrated over one synthetic dataset and four real datasets, showing that the proposed approach consistently achieves smaller set size compared to baselines and always returns contiguous sets (as claimed). ", "questions": "I also only have minor questions. \n\n* What can be potential mitigations if the unimodal assumption is violated?\n* For Table 3, is there a particular reason that LAC and APS are compared via SSCV instead of the coverage rate?\n", "limitations": "The limitations that I expected are discussed in Conclusion, and as mentioned before, it would be more interesting to dicuss possible mitigations on these limitations. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Originality**: To my understanding, this is a novel usage of conformal prediction to ordinal classification. Moreover, the paper proposes a simple yet effective sufficiently-parameterized scoring function (i.e., (5)) that always returns contiguous sets under the unimodal assumption. Even better, this new score function design is theoretically justified in Theorem 2. This theorem is also empirically justified (via CV% = 0). \n\nGiven the unimodal score function, existing conformal prediction algorithms intuitively return a contiguous set. But, this is carefully analyzed in Theorem 1. \n\n**Quality**: I think the paper quality is good. The notations are well-defined, the claims are rigorously analyzed via theorems, and the main claim is also well-justified empirically. \n\n**Clarity**: The paper is clearly written (and thanks for errata in Appendix).\n\n**Significance**: I believe the proposed scoring function (5) is sufficiently novel (and simple) to bring interesting related papers in conformal prediction and ordinal classification. \n\nLast but not least, this paper sufficiently motivates the practical necessity of conformal prediction for ordinal classification in introduction (by using examples, like proximal shoe size). I like this practical connection of conformal prediction in real scenarios, which is usually missing in conformal prediction papers. \n", "weaknesses": "I have minor comments. \n\n* My main concern is the unimodal assumption (which is also mentioned in the limitation section). Based on the empirical results, many distributions for ordinal classification closely satisfy the unimodal assumptions (based on the results that the proposed approach’s CV% is zero). Even though it is violated, I think it does not affect coverage rate. Even though It looks good to discuss potential mitigations if this assumption is violated. \n* When I read Section 4.2, I wanted to see a simple illustration between a label distribution from a naive DNN and that of (5), which contrasts the naive approach fails to achieve a unimodal output, while (5) does. If the left image of Figure 1 is an illustration of the actual data, it’s better to highlight this. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thoughtful comments. Please find our response below  \n  \n**Comment:**\n \nMy main concern is the unimodal assumption … Even though it is violated, I think it does not affect coverage rate.”  \n  \n**Response:**  \n \nYes, it is true that even if the unimodality assumption does not hold, the theoretical coverage guarantees of the prediction set produced by COPOC would still apply. The bound itself, however, would be weaker since the fitted distribution deviates significantly from the true underlying distribution.  \n   \n**Comment:**  \n  \nWhen I read Section 4.2, I wanted to see a simple illustration between a label distribution from a naive DNN and that of (5), which contrasts the naive approach fails to achieve a unimodal output, while (5) does. If the left image of Figure 1 is an illustration of the actual data, it’s better to highlight this.*  \n  \n**Response:**  \n \nThanks for the nice suggestion. Figure 1 in the paper is not from real data but in the Figure 1 of the uploaded 1-pager PDF (in global rebuttal section), we have included an illustration that is based on examples from a real public dataset Adience. In the revised version of the paper, we will consider including this new figure itself.  \n   \n**Comment:**\n \nWhat can be potential mitigations if the unimodal assumption is violated?\n \n**Response:**\n   \nWe acknowledge that this is an important aspect to consider since COPOC will lead to a sub-optimal fit if the unimodality assumption does not hold. One potential mitigation is to evaluate the validity of the assumption by comparing the likelihood of the Vanilla DNN trained with cross-entropy with that based on the COPOC approach. If the COPOC fit is much inferior to that of the unconstrained DNN, it would most likely indicate that the assumption is not valid. In such a case, a direct application of conformal prediction such as APS would be preferable.  \n \nThe table below shows the negative log-likelihood (NLL) of vanilla DNN fitted with cross-entropy loss (V-CE) and COPOC on the four real datasets that we considered. For these datasets, the superior fit of COPOC indicated by lower NLL justifies the validity of the unimodality assumption, which predictably led to smaller prediction set sizes.  \n  \n|              | V-CE           | COPOC          |\n|--------------|----------------|----------------|\n| HCI          | 1.73 &pm; 0.13 | 1.59 &pm; 0.15 |\n| Adience      | 2.33 &pm; 0.18 | 1.66 &pm; 0.21 |\n| Aesthetic    | 1.49 &pm; 0.01 | 0.71 &pm; 0.02  |\n| Retina MNIST | 1.24 &pm; 0.04 | 1.23 &pm; 0.04 |\n  \n**Comment:** \n \nFor Table 3, is there a particular reason that LAC and APS are compared via SSCV instead of the coverage rate?  \n \n**Response:**  \n \nIn Table 3, we compare against two of the most popular conformal prediction (CP) methods- APS and LAC on the output prediction of our proposed unimodal model, at $\\alpha$ = 0.1. Since we fix $\\alpha$ at 0.1 both LAC and APS ought to produce a Prediction set with at least 90% (= $1-\\alpha$) marginal coverage on unseen test points. Equation of marginal coverage is given in Eqn. 1 in our manuscript. Equation of conditional coverage is given in Line 79 which is a stronger notion of coverage is some sense. To compare different CP methods in terms of conditional coverage, Size-Stratified Coverage Violation (SSCV) is a metric that is commonly used to measures violations of the conditional coverage property and is particularly suited for high dimensional data. (A. Angelopoulos et. al. '20). Since LAC produces shorter prediction sets as seen from Table 3, we thought it would be interesting to compare both of them against conditional coverage metric. From Table 3 and Fig. 6, it is evident that LAC achieves the smallest prediction set size but sacrifices adaptiveness (conditional coverage) in the process.  \n  \nPlease let us know if you have any other questions or if there is anything else that we could add to further improve the submission"}]

Author rebuttal:

We thank all the reviewers for their detailed comments and suggestions. We have attempted to address the reviewer concerns and questions to our best including additional experimental results and figures.  
  
Below we summarize the key points of our responses. 
  
**Novelty of our current work and overlap with Lu et. al. '20 and Yunpeng et. al. '23:**
 
In our response to reviewer vR8Y, we have clarified that while the papers cited by the reviewer share the same motivation, the actual methodology and contributions are substantially different. Figure 2 (in the 1-pager PDF in the global rebuttal section) is  meant to further elucidate this point.  We have also included  empirical results on real datasets that demonstrate the superior efficacy of COPOC relative to Lu et. al. We thank reviewer vR8Y for pointing us to these related works and will include them in the revised version.
 
**Validity of unimodality assumption and mitigation approaches:** 
  
We agree with reviewers Qwu5 and w3W6 that this aspect needs more discussion in the paper and will revise it accordingly. In our response to the reviewers, we motivate the validity of the unimodal assumption for ordinal classification applications such as cancer stage detection. The papers cited by Reviewer vR8Y (e.g., Lu et al) that deal with spinal stenosis also make the case for the unimodal formulation. We have also included empirical results comparing the likelihood with unconstrained DNN and unimodal DNN (COPOC) model on four real world public datasets to provide additional justification. Based on our exploration, comparing the likelihoods of the unconstrained and unimodal DNNs to figure out the appropriate conformal prediction approach seems like a good mitigation strategy. We also wish to clarify that the theoretical results (Lemma1 and Theorem 1) continue to hold even if the unimodality assumption is not true.
  
**Details of Hyperparameter Optimization:** 
 
In our response to reviewer Qwu5, we  point to the relevant parts in Appendix C.1 that provide details on the datasets and hyperparameter settings. We also included additional details that should enhance the reproducibility of our work and will add it in the appendix of the revised version.
  
**Questions on metrics:** 
 
We have attempted to address the questions by reviewers ieEd and w3W6 regarding  the choice of SSCV and the behavior of MAE along with the empirical results.
 
**Figure with real world example:** 
 
We have added a new picture (Figure 1 of the uploaded 1-pager PDF in global rebuttal section) with examples from a public age estimation dataset (Adience) to motivate the COPOC approach.  
   
 
Thanks again for the review process and the valuable comments that should aid us in improving the clarity of the paper. We will revise the submission as per the feedback. If there are any comments/questions we overlooked or if there are further ways to improve the paper, please let us know and we will be glad to work on them.


Meta-review: In this paper, the authors propose a method for constructing a prediction set with a coverage guarantee using the framework of conformal prediction for the ordinal classification problem. When applying the conventional conformal prediction method, there was a risk that the prediction set became non-contiguous, which is unnatural for ordinal classification tasks. To resolve the issues, the authors propose a novel method that allows the construction of contiguous prediction sets. All reviewers positively evaluate the proposed method as reasonable and novel. The paper is clearly written,  and the numerical experiment effectively illustrates the advantages of the proposed method.
Decision: Accept (poster)