Title: Conformal Prediction Sets for Ordinal Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: Ordinal Classification

Conformal Predictions

Unimodal modelling
Abstract: Ordinal classification (OC), i.e., labeling instances along classes with a natural ordering, is common in multiple  applications such as size or budget based recommendations and disease severity labeling.  Often in practical scenarios, it is desirable to obtain a small set of likely classes with a guaranteed high chance of including the true class. Recent works on conformal prediction (CP) address this problem for the classification setting with non-ordered labels but the resulting prediction sets (PS) are often non-contiguous and unsuitable for ordinal classification. In this work, we propose a framework to adapt existing CP methods to generate contiguous sets with guaranteed coverage and minimal cardinality. Our framework employs a novel non-parametric approach for modeling unimodal distributions. Empirical results on both synthetic and real-world datasets demonstrate our method outperforms SOTA baselines by 4% on Accuracy@K and 8% on PS size.

Conformal Prediction Sets for Ordinal Classification

Abstract

Ordinal classification (OC), i.e., labeling instances along classes with a natural ordering, is common in multiple applications such as disease severity labeling and size or budget based recommendations. Often in practical scenarios, it is desirable to obtain a small set of likely classes with a guaranteed high chance of including the true class. Recent works on conformal prediction (CP) address this problem for the classification setting with non-ordered labels but the resulting prediction sets (PS) are often non-contiguous and unsuitable for ordinal classification. In this work, we propose a framework to adapt existing CP methods to generate contiguous sets with guaranteed coverage and minimal cardinality. Our framework employs a novel non-parametric approach for modeling unimodal distributions. Empirical results on both synthetic and real-world datasets demonstrate that our method outperforms SOTA baselines by 4% on Accuracy@K and 8% on PS size.

1 Introduction

A large number of practical applications involve ordinal classification (OC), i.e., labeling of instances along a set of ordered discrete classes. These include applications such as determining severity level of abuse incidents or disease diagnosis for efficient follow up, predicting customer age-range, apparel size, and budget for improving recommendations and reducing customer cognitive effort by offering adaptive search filters. Often, in such cases, it is desirable to identify a small set of classes (called prediction set or PS) for a given input instance such that the true class is amongst the identified set with a high probability (called coverage level) instead of a single point estimate of the most likely class. This allows more flexibility in choosing operating points that balance precision-recall trade-off. For instance, customers looking for shoes are often recommended products that are not even available in the customer's typical size. An OC model that predicts the customer size would be well suited to improve the recommendations offered. However, limiting the recommendations to a single most likely size is too restrictive because the model accuracy is often low due to variations in size over time and across brands. Instead, it is preferable to identify a small set of product sizes with say > 90% coverage and use that for automatically filtering the recommendations as shown in Fig. 1.

Conformal prediction (CP)[35, 1, 36, 9] is an emerging area that addresses the problem of predicting a set of likely outputs along with a measure of the confidence or coverage level. While existing CP methods offer ease of deployment and provable coverage guarantees, most of these methods deal with unordered classes and output prediction sets that are non-contiguous and unsuitable for OC settings where the underlying distribution is likely unimodal. For example, for any customer, based on their foot size, a good fit is possible only with a small set of proximal shoe sizes resulting in a unimodal class distribution. Recommending a non-contiguous set of sizes [4 , 8 , 9] is not a desirable customer experience. On the other hand, expanding the PS to a minimal contiguous superset leads to

True P(Y|x)

Unimodal fit (1)

Unconstrained fit (2)

soracle(a)=[6,7]

S2（α)=[4,8,9]

s,（α)=[6,7,8,9]

GroundTruth

*Label

1：

23

45678

10

11

Below image shows how Adidas Galaxy Lite

Below image shows how Adidas Galaxy Lite

Below image shows how Adidas Galaxy Lite

will fit based on what you usually wear

will fit based on what you usually wear

will fit based on what you usually wear

Tighter

Same

Looser

Tighter

Same

Looser

Tighter

Same

Looser

489

6789

6

7

3DView

Footbed

3DView

Footbed

3DView

Footbed

a significant expansion of the set size (e.g., [4 , 8 , 9] → [4 , 5 , 6 , 7 , 8 , 9] ) diminishing its application utility and diluting coverage guarantees. A similar contiguity requirement arises in applications such as cancer stage diagnosis where predicting non-contiguous labels (stage-1, stage-4) would be discordant compared to contiguous labels. Note that there do exist OC scenarios such as prediction of day of week for an event where unimodality might not be valid as we discuss in Section 6.

While ordinal classification [18, 7] has been well-studied in the past, most earlier works [28, 8] do not assume unimodality. Recent works [4, 22, 42, 12, 15, 20] demonstrate strong benefits of assuming the underlying conditional distribution to be unimodal for OC tasks . However, these techniques primarily focus on point estimates and not conformal predictions. Furthermore, these methods often either impose a restrictive structure on the model [33] or assume parametric models (e.g., Binomial[4], Gaussian[42] ), which limits their representation ability and utility for real-world applications. Given the multitude of practical applications, there is a strong need for a principled approach that combines ideas from ordinal classification and conformal predictions to construct minimal contiguous prediction sets when the underlying true class distribution is unimodal.

In this work, we focus on ordinal classification where the true class distribution is unimodal. We build on existing work, Adaptive Prediction Set (APS) [35], to obtain minimal contiguous prediction sets with provable coverage guarantees. Below we summarize our key contributions:

For ordinal classification where the true class distribution is unimodal, we show that any model constrained to output a unimodal distribution can be used with APS algorithm to yield contiguous PS with a guaranteed coverage level. Further, we prove a tight upper bound for the cardinality of this set in terms of the optimal set and the divergence between the fitted model and the true distribution.

We provide a novel construction to adapt any DNN-based architecture to always yield a unimodal distribution over ordinal classes while ensuring that any arbitrary unimodal class distribution can be approximated.

We study the efficacy of our approach for Co nformal P redictions for OC (COPOC) via controlled ablation studies on synthetic datasets which demonstrates the utility of unimodal construction in improving learnability relative for small data sizes, and its versatility compared to other methods that make parametric distributional assumptions.

We perform extensive evaluation on benchmark datasets for facial age estimation, HCI classification,image aesthetic score estimation and biomedical image classification with results indicating that our method outperforms SOTA baselines by 4% on Accuracy@K and 8% on PS size.

2 Contiguous Conformal Prediction Set Problem

Notation: P [ E ] denotes the probability of event E . [ i ] b a denotes conditioning on integer index ∀ i, a ≤ i ≤ b within a range while x = [ x i ] b i = a denotes the vector or sequence [ x a , · · · , x b ] .

Let X denote input space and C = { c k } K k =1 denote an ordered set of K classes. Consider the case where the random variables X,Y are drawn from X × C following a joint distribution P X,Y such that the true underlying conditional distribution P Y | X is unimodal . In other words, for any x ∈ X , denoting p k = P Y | X ( Y = c k | X = x ) , there exists a class c m ∈ C such that p k ≤ p k +1 , [ k ] m -1 1 and

p k ≥ p k +1 , [ k ] K m . Given a training dataset D = { ( x i , y i ) } n i =1 with samples drawn exchangeably (e.g., i.i.d.) from P X,Y and a desired coverage level (1 -α ) ∈ (0 , 1) for any unseen input test instance ( X test , Y test ) ∈ D test drawn from same distribution, the objective is to determine the minimal non-empty contiguous prediction set (PS) ˆ S D ,α ( X test ) = { c l +1 , · · · , c u } , 0 ≤ l < u ≤ K satisfying marginal coverage with finite samples above a specified level:

The above probability is taken over all ( X test , Y test ) ∈ D test data points. While marginal coverage is achievable, it differs from conditional coverage: P [ Y test ∈ ˆ S D ,α ( X test ) | X test = x test ] ≥ (1 -α ) , which is a stronger notion. In practice, it is desirable for ˆ S D ,α to approximate conditional coverage and achieve it asymptotically in the limit of large sample sizes.

3 Related Work

Conformal Prediction. Originating in [40, 41], conformal prediction is a statistical approach for generating predictive sets with marginal coverage guarantees (Eqn.1) with finite samples. Earlier methods for calibrating probabilities (e.g., via Platt scaling [32] or temperature scaling [17]) and assembling a prediction set by including the most likely classes till the covered probability mass exceeds a specified coverage level do not achieve the desired theoretical guarantees since the calibration is often erroneous[29]. Most recent CP methods use split conformal prediction that enables easy deployment with any arbitrary predictor [31, 21]. APS [35], [9], and [36] introduce techniques aimed at achieving coverage that is similar across regions of feature space whereas [19] propose methods to achieve equal coverage for each class. Of these, Least Ambiguous Set-valued Classifier (LAC) [36] provides provably smallest average PS size when probability estimates are approximately correct while APS[35] has been shown to be superior in practice. For a primer on conformal methods please refer [2]. RAPS[1] presents a regularized version of APS for Imagenet. Although relatively new, CP methods are being widely applied for regression, vision, NLP and time-series forecasting [34, 38, 3, 14]. Our work adapts existing CP methods to unimodal ordinal classification setting with an additional contiguity requirement. Ordinal-APS [25] and [43] share the same motivation but propose a greedy CP method that results in larger prediction sets compared to COPOC (Table 4).

Ordinal Classification. Early works on ordinal classification [18, 7, 45] are based on regular classification or regression, ignoring ordinal relationship and non-uniform separation among labels. Some DNN-based methods model cumulative distribution (CDF) up to each ordered class using multiple binary classifiers [8, 39, 6, 28] but the output class probabilities are not necessarily unimodal and are also not guaranteed to form a valid CDF. Label smoothing methods [15, 16, 24, 12, 20, 42] convert hard target labels into unimodal prior distributions to be used as the reference for the training loss but these methods are often sub-optimal since the assumed priors might not reflect the true distribution, classes might not be equi-spaced categories and additionally test predictions might not necessarily be unimodal. [22, 5] learns a non-parametric unimodal distribution as a constraint optimization problem in the loss function which is not only difficult to optimize but also does not guarantee unimodality on test data. In [4, 10], unimodality is guaranteed by assuming a Binomial or Poisson distribution but has limited representation ability due to a single degree-of-freedom which does not capture hetereoscedastic noise. Keeping in view the potential benefits of accurate unimodal distribution learning for the ordinal classification setting, our work explores a non-parametric flexible DNN-based approach that guarantees unimodality of the predicted class distribution even on test data.

4 Solution Approach

To generate conformal predictions for ordinal classification, we consider two questions: (i) what is the primary drawback of the existing CP methods for classification with respect to ordered labels setting? (ii) how do we address this drawback?

Most CP methods for classification [35, 36, 9] are based on a split calibration approach where the training data D is divided into two sets. The first set D train is used to learn a probabilistic model ˆ P Y | X ( · ) using a blackbox learning algorithm while the second set D cal is used to determine the optimal conformity score threshold for a desired coverage level, which is then used for conformal inference. Below we discuss two common CP methods:

Least Ambiguous Set Values Classifier (LAC) [36]. Here, the conformal PS for a test instance x test is constructed using a calibration rule of the form,

where ˆ p k ( x ) = ˆ P Y | X ( Y = c k | X = x ) is the class probability from the model and ˆ q D cal ( α ) is the score threshold defined as the bias-adjusted ( α ) th quantile of the model score of the true label.

Adaptive Prediction Set (APS) [35]. In APS, for every instance ( x , y ) ∈ D cal , we compute a conformity score defined as the cumulative probability mass required to include the true label corresponding to that instance, i.e., s ( x ) = ∑ T k =1 ˆ p π k ( x ) , where T is chosen such that c π T = y and π is the permutation of { 1 , . . . , K } that sorts ˆ P Y | X ( . | x ) in the descending order from most likely to least likely. Given a desired coverage (1 -α ) , we compute a suitable score threshold ˆ q D cal ( α ) = Quantile ( { s ( x ) | x ∈ D cal } , ⌈ ( n +1)(1 -α ) ⌉ n ) , where n = | D cal | . The conformal PS in APS is constructed using:

More details on the coverage guarantees of APS are in Appendix B. While the above methods yield minimal sets with provable guarantees on marginal coverage, the primary drawback is that the resulting prediction sets are not necessarily contiguous, which is an important requirement for ordinal classification when underlying true distribution is unimodal.

Anaive solution is to consider the minimal contiguous set that covers the PS output by CP method. For example, if CP outputs { 2 , 4 , 6 } , we can produce a minimal contiguous set by including "in-between" classes i.e., { 2 , 3 , 4 , 5 , 6 } . However, this would end up having a much larger cardinality, which invalidates the tight coverage bounds (refer Appendix B) (i.e., the new set will have a coverage much higher than the upper bound guaranteed for that method). Hence, we consider an alternative approach based on the observation that if the blackbox learning algorithm in the first step is constrained to output a unimodal distribution, then the classes with probability above a threshold will cluster around the mode. The conformal calibration rules in Eqn. 2 and 3, which are designed to choose classes in the order of class probability will thus result in contiguous sets. Using this insight, we propose a solution depicted in Fig. 2 that consists of two steps: (a) design a learning algorithm that can accurately model training data such that the output class distribution is always unimodal, (b) adapt an appropriate CP algorithm based on Eqn. 2 or Eqn. 3 to identify the PS. While this high level approach can be used with any of the CP method, in our current work, we consider the Adaptive Predictive Set (APS) approach [35] since it provides tight guarantees on marginal coverage and has been empirically shown to achieve better conditional coverage than LAC [36] and smaller set size than CQC [9]. We now describe our solution beginning with a study of the properties of the resulting PS (Sec. 4.1) followed by construction of the unimodal classifier (Sec. 4.2).

4.1 Conformal Prediction Sets from Unimodal Models

We now study the properties of PS corresponding to the COPOC approach outlined in Fig. 2 where for every input instance X , both the true underlying distribution P Y | X and the learned classification model ˆ P Y | X are unimodal. In other words, there exists a class c ˆ m ∈ C such that ˆ p k ≤ ˆ p k +1 , [ k ] ˆ m -1 1 , and ˆ p k ≥ ˆ p k +1 , [ k ] K ˆ m . For a given x test and coverage level (1 -α ) , let S oracle α ( x test ) denote the minimal contiguous PS that can be constructed by an oracle that knows the underlying P Y | X , i.e.,

where p k = P Y | X ( Y = c k | X = x test ) . This is the set with the least cardinality satisfying contiguity and conditional coverage. In case of multiple solutions, the oracle can pick any one at random. In practice, we do not have access to the true P Y | X and ˆ S D,α ( x test ) has to be constructed from the approximate distribution ˆ P Y | X using CP methods such as APS for valid coverage guarantees. Lemma 1 establishes the contiguity of the prediction sets resulting from LAC and APS. Thm. 1 bounds the cardinality of the APS PS relative to the oracle sets. Further, when ˆ P Y | X is a consistent estimator of

ModelP(Y|x)

PredictionSet

SD,α(Xtest)

Unimodal

Training

Model

Conformal

Algorithm

Conformal

Inference

Dtrain

D

cal

Dtest

Uniform Sampling from

TrueDistributionPyx

p1

p2

pK

SOFTMAX

Z1

ZK

→R-;yE（r）=y（-r)

T'2

13

TK

V2

V3

VK

Φ:R→R+

Inputx

the true P Y | X , the APS conformity score threshold asymptotically converges to α and the cardinality of ˆ S D,α ( x test ) converges to that of the oracle [35].

Lemma 1. 1 Given a fitted unimodal model ˆ P Y | X , for any x test and α ∈ (0 , 1] , prediction sets ˆ S D,α ( x test ) constructed using Eqn.2 or 3 have at least one solution which is contiguous (amongst multiple possibilities). When ˆ P Y | X is strictly unimodal, all the solutions are contiguous.

Proof Sketch: By design, the prediction sets from Eqn.2 or 3 have to contain the top k most likely classes for some k which results in contiguity in case of strict unimodality. When the unimodality is not strict, non-contiguous sets may satisfy the required constraints due to classes with equal probabilities at the boundaries, but the shortest span solution is contiguous.

Theorem 1. For any x ∈ X , let p k ( x ) = P Y | X ( Y = c k | X = x ) and ˆ p k ( x ) = ˆ P Y | X ( Y = c k | X = x ) denote the true and fitted model class probabilities that are always unimodal. Let σ k ( x ) = ∑ k k ′ =1 p k ′ ( x ) and ˆ σ k ( x ) = ∑ k k ′ =1 ˆ p k ′ ( x ) denote the corresponding cumulative distribution functions. If | σ k ( x ) -ˆ σ k ( x ) | ≤ δ, [ k ] K 1 for a constant δ , then for any α ∈ (0 , 1] , ∀ x ∈ D test , the APS and oracle prediction sets from Eqn.3 and Eqn. 4 satisfy | ˆ S D,α ( x ) | ≤ | S oracle α -4 δ -1 n +1 ( x ) | where n is the size of the calibration set.

Proof Sketch: . To establish the result, we prove the following two statements: (a) | ˆ S D,α ( x ) | ≤ | S Oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) , and (b) | S oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) | ≤ | S oracle α -4 δ -1 n +1 | . From Eqn. 4 and Lemma 1, we observe that the unimodality of ˆ p ( x ) and p ( x ) leads to the oracle prediction sets and at least one of the APS prediction sets being contiguous. Let ˆ S D,α ( x ) = { c ˆ l +1 , · · · , c ˆ u } , 0 ≤ ˆ l < ˆ u ≤ K and S oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) = { c l ∗ +1 , · · · , c u ∗ } , 0 ≤ l ∗ < u ∗ ≤ K . From the definitions and contiguity, we observe that the probability mass of ˆ S D,α ( x ) w.r.t. ˆ p equals (ˆ σ ˆ u ( x ) -ˆ σ ˆ l ( x )) ≥ ˆ q D cal ( α ) while that of S oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) w.r.t p equals ( σ u ∗ ( x ) -σ l ∗ ( x )) ≥ ˆ q D cal ( α ) + 2 δ . Using the divergence bound on the two CDFs and the fact that ˆ S D,α ( x ) is the minimal contiguous set with probability mass ≥ ˆ q D cal ( α ) as per ˆ p yields (a). Using the divergence bound, we also observe that the marginal coverage, P [ y ∈ ˆ S D,α ( x )] ≥ ˆ q D cal ( α ) -2 δ . Combining this from the APS coverage guarantees (Theorem 3 in Appendix B) yields part (b).

4.2 Non-Parametric Unimodal Distribution Learning

We now consider the problem of learning a classification model that is guaranteed to be unimodal and also expressive enough to accurately model any unimodal distribution. Given universal approximation properties of DNNs with respect to probability distributions [26], these form a natural choice as a multi-class classification learning algorithm. For a given x , let z ( x ) = f ( x ; θ ) denote the output of last fully connected layer of a DNN model f ( · ) , where θ is the model parameters and z ( x ) a K -dimensional vector where K = |C| . Softmax operation is applied to z ( x ) to obtain class probability distribution. Predicted probability of class c k ∈ C for x is given by ˆ p k ( x ) = exp( z k ( x )) ∑ K k =1 exp( z k ( x )) . Training is usually performed by minimizing cross-entropy (CE) loss between

this predicted distribution and the ground truth. Since CE loss does not assume any constraints on underlying distribution, it can, in principle, model any arbitrary distribution asymptotically with large enough data and model size. When the true underlying distribution P Y | X is unimodal, this standard learning approach results in a classifier ˆ P Y | X that approximates P Y | X and is itself unimodal assuming large enough data and model size . However, in practice with limited high-dimensional data and limited model capacity, the standard training is not guaranteed to reach the true "global minima" [11] or even adhere to the unimodality constraints that lead to contiguous prediction sets. Hence, there is a need for introducing the right inductive bias that truncates the hypothesis class to be searched while ensuring it is expressive enough to include the true hypothesis so as to allow for better learnability and generalization [27]. For our scenario, the set of unimodal distributions is the most natural hypothesis class. While previous works [4, 42] consider a set of parameterised unimodal distributions (e.g., based on Gaussian or Binomial distributions), we propose a construction that ensures the hypothesis class exactly maps to the set of unimodal distributions on the K -simplex.

Construction. Let U be the set of all discrete unimodal distributions on C . For any x with ˆ p ( x ) = ˆ P Y | X ( Y | X = x ) ∈ U , the class probabilities [ˆ p k ( x )] K k =1 would be unimodal, i.e., there exists a mode c ˆ m ∈ C such that the sequence is monotonically non-decreasing before c ˆ m and non-increasing post c ˆ m . The corresponding log probabilities denoted by z k = log(ˆ p k ( x )) ∈ R -, [ k ] K 1 also form a unimodal sequence with the same mode and are all negative valued.

Let ψ : R + → R -be any strictly monotonically decreasing bijective function, Then, for any z ∈ R -, there is a unique r ∈ R + such that ψ ( r ) = z . Let r k = ψ -1 ( z k ) , [ k ] K ˆ m and r k = -ψ -1 ( z k ) , [ k ] ˆ m -1 1 then given [ z k ] K k =1 , we obtain a unique 2 non-decreasing sequence [ r k ] K k =1 . Let ψ E : R → R -be the 'even' extension of ψ ( · ) , i.e., ψ E ( r ) = ψ ( r ) for r ∈ R + and ψ E ( r ) = ψ ( -r ) for r ∈ R -, then we have ψ E ( r k ) = z k . Possible choices include ψ E ( x ) = -| x | d for any real d . The monotonically increasing [ r k ] K k =1 can be generated from the output vector v of a DNN constrained such that v k ≥ 0 , [ k ] K 2 using a non-negative transformation ϕ : R ↦→ R + followed by a cumulative summation. Fig. 3 depicts our proposed construction for any DNN f ( · ) along with equations below.

This network can be trained with standard CE loss. From Thm. 2, any distribution in U maps to an output vector η ( x ) ∈ R d , which can be approximated using DNNs with appropriate capacity [26]. Fig. 8 in Appendix C.4 presents some illustrative examples of Unimodal model fit of COPOC on a public dataset.

Theorem 2. Let η : X → R K , ϕ : R → R + and ψ E : R → R -such that ψ E ( r ) = ψ E ( -r ) , ∀ r ∈ R and its restriction to R + is a strictly monotonically decreasing bijective function. (a) Then, the model output constructed as per Eqn. 5 is always unimodal, i.e., ˆ p ( x ) ∈ U , ∀ x ∈ X . (b) Further, given any ˆ p ( x ) ∈ U for x ∈ X , there exists a well defined function η ( x ) ∈ R K that satisfies Eqn. 5 if ϕ ( · ) is surjective on R + .

Proof Sketch: Part (a) follows by observing that the non-decreasing sequence [ r k ] K k =1 will result in a unimodal [ z k ] K k =1 and unimodal ˆ p ( x ) ∈ U with mode corresponding to sign change in r k sequence. Part (b) follows by constructing a unique inverse for ψ E ( · ) by pivoting on the largest indexed mode ˆ m of ˆ p ( x ) . For k < ˆ m , we choose ( ψ E ) -1 ( z k ) ∈ R -and for k ≥ ˆ m , we choose ( ψ E ) -1 ( z k ) ∈ R + .

5 Experiments

We evaluate the utility of our approach (COPOC) for generating minimal contiguous PS for a desired coverage for ordinal classification on multiple datasets. We investigate the following questions:

RQ1: How does COPOC perform relative to SOTA methods on OC tasks?

RQ2: What benefits does COPOC offer relative to parametric modeling on synthetic datasets?

RQ3: Under what conditions does COPOC outperform vanilla model trained with CE loss (V-CE)? Is COPOC an approximately consistent estimator of the underlying distribution?

RQ4: How does COPOC performance vary with a different choice of conformal inference such as LAC [36] instead of APS?

RQ5: How does Ordinal-APS [25] which also outputs a contiguous prediction set (PS) over ordinal labels with regular model training fares against COPOC with it's unimodal training?

5.1 Experimental Setup

Algorithms : We compare our proposed unimodal construction (COPOC) against six SOTA methods as well as modeling with vanilla cross-entropy loss V-CE using APS for conformal inference for all methods for a fair comparison. For COPOC, we experimented with various choices of ϕ and ψ and chose ϕ ( x ) = | x | and ψ E ( x ) = -| x | based on performance (see Appendix C.3). The SOTA methods include approaches that (a) utilize soft labels generated from linear exponentially decaying distributions SORD [[12], and linear adaptive Gaussians with variance learning AVDL [42], (b) specifically model unimodal distribution through parametric assumptions Binomial [4] and Binomial-Temp , and (c) non-parametric methods based on unimodality aware loss function Uni-Loss [22] or ordinality imposed in embedding space POE [23]. In Sec. 5.5, we also evaluate an alternative conformal procedure LAC [36] with our unimodal construction.

Metrics : Similar to [35, 9] we compute PS Size ( |PS| ) @90% coverage i.e., α = 0 . 1 . Since APS does not guarantee contiguity for models that do not output unimodal distribution, we consider a minimal contiguous interval that covers the output PS and report the size. We also report the fraction of instances which resulted in non-contiguous PS for which we needed to expand the original non-contiguous set ( CV% ). To evaluate performance on OC tasks, we report MAE - mean absolute difference between predicted and ground-truth and Accuracy@K ( Acc@K ) that captures if the ground truth was in the top k predictions. This can benefit unimodal constrained models where classes adjacent to the mode will receive the next greatest probability mass. For comparing different conformal prediction methods in Sec. 5.5, we use size-stratified coverage violation (SSCV) [1] that measures violations of the conditional coverage property and is suited for high dimensional data.

Datasets : Weevaluate COPOC on four public image datasets: age-detection (Adience [13]), historical image dating (HCI [30]), image aesthetics estimation (Aesthetic [37]) and biomedical classification (Retina-MNIST [44]). These datasets contain 8 , 5 , 5 , and 5 ordered classes respectively and the number of instances is 26580 , 1325 , 15680 , and 1600 . More details are included in Appendix C.1. We also experimented with synthetic data with the generation process and result discussed in Sec 5.3.

5.2 RQ1: Performance on ordinal classification on real world datasets

Table 1 presents comparison of COPOC with other methods on the HCI dataset. Results on other datasets included in the Appendix C.1 due to space constraints. COPOC performs at par with other baselines in terms of MAE & Acc@1 and significantly outperforms them on Acc@2, Acc@3 & | PS | ( 8% smaller compared to the next best one). We observe that V-CE and POE have the highest

contiguity violations CV% . Though SORD, AVDL , and Uni-loss enforce unimodality in training, it does not necessarily translate to unimodality in test samples (indicated by high CV% ) resulting in poor performance on Acc@2 and Acc@3 and |PS|, which is critical in OC tasks. Since Binomial-temp enforces unimodality by construction, it performs better than the above methods on Acc@k . However, unimodality constraint on DNN in COPOC results in even better model fit for OC tasks which translates to higher Acc@K &shorter | PS | .

5.3 RQ2: Ablative Study of Unimodal Modeling Methods on Synthetic Data

To generate synthetic data, we consider the label set C of size 10, i.e' Y ∈ { c 1 , . . . , c 10 } and input space X ⊆ R 10 , where X 1 ∼ Uniform (1 , 10) and X 2 , . . . , X 10 follow independent normal distribution. Further, we define h ( X ) = 100 ∗ ( sin (0 . 2 ∗ X 1 )+ cos (0 . 4 ∗ X 1 )) . To generate the labels, we consider a conditional distribution function Q Z | X that generates Z with h ( X ) as distributional parameters and a mapping function g : R → C that maps Z to the output label Y ∈ C . Thus, for each x , we generate z ∼ Q Z | X ( x ) and y = g ( z ) . We consider two choices of mapping functions g ( · ) . The first choice g equi partitions the range of Z into 10 equi-bins which can be mapped to the label set in a linear order while the second choice g non -equi partitions the range of Z into 10 bins of arbitrary width to assign labels. From the construction, one can see that g equi considers classes to be equi-spaced categories while g non -equi does not. We consider the following four synthetic datasets in increasing order of complexity and train models (details in Appendix C.2 )

D1 : Q Z | X ( x ) = Double-Exponential with mean at h ( x ) and constant variance; g = g equi .

D2 : Q Z | X ( x ) = Double-Exponential with mean at h ( x ) and constant variance; g = g non -equi .

D3 : Q Z | X ( x ) = Gaussian with mean and variance varying with h ( x ) ; g = g non -equi .

D4 : Q Z | X ( x ) = Gaussian,Poisson or double Exponential with mean and variance varying with h ( x ) chosen at random for each x ; g = g non -equi .

Since we have the true distribution P Y | X , we compute the Oracle PS size and compare it with that of other methods. We report KL Divergence ( KL ) between the true and predicted distributions with lower KL indicating better model fit. Table 2 shows that for D1 , SORD fits the data well with lowest KL and MAE as it explicitly models exponential distribution assuming all classes to be equi-spaced. However, on D2 and D3 , COPOC outperforms all the other methods on the model fit and also results in shorter |PS| closer to Oracle. On D2 and D3 , SORD performs poorly because it assumes both constant variance across input spaces and equi-distant classes which are not valid for these datasets. Since D3 draws samples from Gaussian distribution with heteroskedastic noise, AVDL outperforms SORD but is inferior to COPOC. Since D4 is drawn from a more complex distribution, COPOC outperforms the rest with an even larger margin. The Binomial variants underperform throughout due to distributional

0.07

COPOC

vanilla CE

0.06

0.05

SSO1

0.04

00

0.02

0.01

0

20000

40000

60000

80000

100000

train sample size

-Vanilla CE

peo....

5

4

70

60

50

40

2

30

20

10

0

100

200

500

1000

Train Sample Size

5000100002000030000

40000

50000

4.5

5

LACAPS

4

3.5

3

2.5

2

1.5

0.35

0.3

0.25

2

0.2

0.15

0.1

0.05

0

0.1

0.2

0.3

0.4

0.5

α

mismatch while Uni-Loss exhibits high variance indicating convergence issues. As with the real benchmarks, enforcing unimodality via loss or soft-labels does not guarantee unimodailty on test data, indicated by high CV% . Interestingly V-CE performs exceedingly well throughout in terms of KL beating more sophisticated methods. The overall observation is that the performance of the methods depends largely on the validity of the assumptions and relatively unconstrained nature of COPOC makes it more versatile.

5.4 RQ3: COPOC vs Vanilla-CE and Consistency Properties

We compare COPOC against V-CE by varying train-data size ( = N ) on synthetic data D4 . Fraction of samples used to train each model is 3 / 4 , and rest is used for calibration and test. All experiments are repeated 10 times with random realization of train set. Comparing the KL loss on train set in Fig. 4, we observe that COPOC has lower KL in early stages with lesser data and performance for both converges when N reaches to ∼ 80 K and KL asymptotically becomes negligible. Fig. 5 compares |PS| and CV% by varying N on D4 . |PS| of COPOC approaches those of Oracle with far lesser samples compared to VC-E. Thus, COPOC with its inductive bias fits the data better with lesser samples compared to unbiased CE yielding shorter sets with finite samples. Above results also suggest that COPOC might be approximately consistent. Note that with increased N , V-CE results in a better model fit, lower CV% and shorter sets, indicating that these are correlated.

5.5 RQ4: Choice of Conformal Inference: LAC vs. APS

Table 3 provides a comparison of PS size and size-stratified coverage violation(SSCV) of APS against LAC when applied on the output of our proposed unimodal model for α = 0 . 1 . Both methods generate contiguous prediction sets. Although LAC consistently produces shorter sets, it also has slightly worse SSCV metrics across datasets. In Fig. 6, we plot |PS| and SSCV across different α for synthetic data D4 and results are similar. We observe that LAC achieves the smallest prediction set size but sacrifices adaptiveness (conditional coverage) in the process.

5.6 RQ5: COPOC vs Ordinal-APS [25]

Table 4 shows an empirical comparison of COPOC against APS and Ordinal-APS applied over a Vanilla DNN trained with Cross-entropy loss (V-CE) on public datasets and synthetic data D4 described in Sec. 5.2 and 5.3. For V-CE with APS we consider a minimal contiguous interval that covers the output PS and report its size. We observe that Ordinal-APS produces significantly shorter sets compared to V-CE with APS . However, COPOC significantly outperforms Ordinal-APS because of better unimodal data fit. Note that Ordinal-APS outputs a contiguous PS over ordinal labels

irrespective of whether the posterior distribution generated by the model is unimodal or not. COPOC with unimodal regularization results in better model fit and hence shorter PS sizes.

6 Validity of the Unimodality Assumption

Unimodality assumption might not be universally applicable for all OC scenarios (eg., prediction of preference ratings, event-hour-of-day etc.). However, there do exist a large number of OC applications where it is beneficial to assume unimodality as validated by multiple notable works in computer vision domain [12, 20, 42, 4]. Table 5 shows the negative log-likelihood (NLL) of vanilla DNN (V-CE) fitted with CE loss (unconstrained fit) and COPOC (unimodal fit) on four public datasets described in Sec. 5.2. The

superior fit of COPOC indicated by lower NLL justifies the unimodality assumption for these datasets. COPOCmakes an assumption on the underlying distribution being unimodal and one could potentially check the validity of the assumption by comparing the likelihood of the unimodal and unconstrained fits. Note that even if the unimodality assumption is not true, the theoretical coverage guarantees of the prediction set produced by COPOC would still hold but the cardinality bounds would be weaker since the fitted distribution would deviate significantly from the true underlying distribution.

7 Conclusion, Broader Impact, and Limitations

We proposed an approach to construct minimal contiguous prediction sets for ordinal classification with guarantees on coverage and cardinality along with empirical validation of efficacy on both real and synthetic datasets. Our solution employs a novel architecture for non-parametric modelling of arbitrary unimodal class distributions without sacrificing representation ability. With ML-driven systems playing an increasingly important role in our society, it is critical to provide practical guarantees on the reliability and uncertainty of the ML model output. While existing CP methods [35] address this concern for the general classification setting, there is a need for specialized solutions for application scenarios with inherent ordinality and unimodality in class labels, e.g., predicting cancer stage of biopsy signals. Our work makes an important contribution in this direction and also provides a way to effectively communicate the limitations of the model adaptively across the input space with bigger prediction sets for harder instances. The proposed methods have numerous practical applications in improving recommendations based on an ordered attribute (e.g., budget), and reducing human supervision effort for ordinal classification tasks. Future directions include extensions to tasks where classes are partially ordered or hierarchically grouped (e.g., topic hierarchies). Consistency of our proposed unimodal construction is also worth investigating.

Limitations: COPOC makes an assumption on the underlying distribution being unimodal and might lead to a sub-optimal fit if the assumption does not hold. Furthermore, it can be paired only with CP methods such as LAC and APS where the prediction sets can be viewed as upper-level sets of the predicted class probabilities, i.e., set of classes with probability higher than a threshold. Due to the reliance on the CP methods, the coverage guarantees are also valid only only when the assumption on the exchangeability of the data points holds true.

References

A. Angelopoulos, S. Bates, J. Malik, and M. I. Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193 , 2020.

A. N. Angelopoulos and S. Bates. A gentle introduction to conformal prediction and distributionfree uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

A. N. Angelopoulos, A. P. Kohli, S. Bates, M. Jordan, J. Malik, T. Alshaabi, S. Upadhyayula, and Y. Romano. Image-to-image regression with distribution-free uncertainty quantification and applications in imaging. In International Conference on Machine Learning , pages 717-730. PMLR, 2022.

C. Beckham and C. Pal. Unimodal probability distributions for deep ordinal classification. In International Conference on Machine Learning , pages 411-419. PMLR, 2017.

S. Belharbi, I. B. Ayed, L. McCaffrey, and E. Granger. Non-parametric uni-modality constraints for deep ordinal classification. arXiv preprint arXiv:1911.10720 , 2019.

A. Berg, M. Oskarsson, and M. O'Connor. Deep ordinal regression with label diversity. In 2020 25th International Conference on Pattern Recognition (ICPR) , pages 2740-2747. IEEE, 2021.

R. Can Malli, M. Aygun, and H. Kemal Ekenel. Apparent age estimation using ensemble of deep learning models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 9-16, 2016.

W. Cao, V. Mirjalili, and S. Raschka. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters , 140:325-331, 2020.

M. Cauchois, S. Gupta, and J. C. Duchi. Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. The Journal of Machine Learning Research , 22(1):3681-3722, 2021.

J. F. P. da Costa, H. Alonso, and J. S. Cardoso. The unimodal model for the classification of ordinal data. Neural Networks , 21(1):78-91, 2008.

A. Daniely, N. Linial, and S. Shalev-Shwartz. From average case complexity to improper learning complexity. In Proceedings of the forty-sixth annual ACM symposium on Theory of computing , pages 441-448, 2014.

R. Diaz and A. Marathe. Soft labels for ordinal regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4738-4747, 2019.

E. Eidinger, R. Enbar, and T. Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions on information forensics and security , 9(12):2170-2179, 2014.

A. Fisch, T. Schuster, T. Jaakkola, and R. Barzilay. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning , pages 3329-3339. PMLR, 2021.

B.-B. Gao, C. Xing, C.-W. Xie, J. Wu, and X. Geng. Deep label distribution learning with label ambiguity. IEEE Transactions on Image Processing , 26(6):2825-2838, 2017.

X. Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering , 28(7):1734-1748, 2016.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

G. Guo, Y. Fu, C. R. Dyer, and T. S. Huang. Image-based human age estimation by manifold learning and locally adjusted robust regression. IEEE Transactions on Image Processing , 17(7):1178-1188, 2008.

Y. Hechtlinger, B. Póczos, and L. Wasserman. Cautious deep learning. arXiv preprint arXiv:1805.09460 , 2018.

Z. Huo, X. Yang, C. Xing, Y. Zhou, P. Hou, J. Lv, and X. Geng. Deep age distribution learning for apparent age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 17-24, 2016.

J. Lei, M. G'Sell, A. Rinaldo, R. J. Tibshirani, and L. Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Q. Li, J. Wang, Z. Yao, Y. Li, P. Yang, J. Yan, C. Wang, and S. Pu. Unimodal-concentrated loss: Fully adaptive label distribution learning for ordinal regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 20513-20522, 2022.

W. Li, X. Huang, J. Lu, J. Feng, and J. Zhou. Learning probabilistic ordinal embeddings for uncertainty-aware regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13896-13905, 2021.

X. Liu, X. Han, Y. Qiao, Y. Ge, S. Li, and J. Lu. Unimodal-uniform constrained wasserstein training for medical diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops , pages 0-0, 2019.

C. Lu, A. N. Angelopoulos, and S. Pomerantz. Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2022, Proceedings, Part VIII , page 545-554, Berlin, Heidelberg, 2022. Springer-Verlag.

Y. Lu and J. Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. Advances in neural information processing systems , 33:3094-3105, 2020.

B. Neyshabur, R. Tomioka, and N. Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412.6614 , 2014.

Z. Niu, M. Zhou, L. Wang, X. Gao, and G. Hua. Ordinal regression with multiple output cnn for age estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4920-4928, 2016.

J. Nixon, M. W. Dusenberry, L. Zhang, G. Jerfel, and D. Tran. Measuring calibration in deep learning. In CVPR workshops , volume 2, 2019.

F. Palermo, J. Hays, and A. A. Efros. Dating historical color images. In Computer Vision-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part VI 12 , pages 499-512. Springer, 2012.

H. Papadopoulos, K. Proedrou, V. Vovk, and A. Gammerman. Inductive confidence machines for regression. In Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19-23, 2002 Proceedings 13 , pages 345-356. Springer, 2002.

J. Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers , 10(3):61-74, 1999.

J. D. Rennie and N. Srebro. Loss functions for preference levels: Regression with discrete ordered labels. In Proceedings of the IJCAI multidisciplinary workshop on advances in preference handling , volume 1. AAAI Press, Menlo Park, CA, 2005.

Y. Romano, E. Patterson, and E. Candes. Conformalized quantile regression. Advances in neural information processing systems , 32, 2019.

Y. Romano, M. Sesia, and E. Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 33:3581-3591, 2020.

M. Sadinle, J. Lei, and L. Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association , 114(525):223-234, 2019.

R. Schifanella, M. Redi, and L. M. Aiello. An image is worth more than a thousand favorites: Surfacing the hidden beauty of flickr pictures. In Proceedings of the international AAAI conference on web and social media , volume 9, pages 397-406, 2015.

K. Stankeviciute, A. M Alaa, and M. van der Schaar. Conformal time-series forecasting. Advances in Neural Information Processing Systems , 34:6216-6228, 2021.

T. Vishnu, P. Malhotra, L. Vig, G. Shroff, et al. Data-driven prognostics with predictive uncertainty estimation using ensemble of deep ordinal regression models. International Journal of Prognostics and Health Management , 10(4), 2019.

V. Vovk, A. Gammerman, and C. Saunders. Machine-learning applications of algorithmic randomness. 1999.

V. Vovk, A. Gammerman, and G. Shafer. Algorithmic learning in a random world , volume 29. Springer, 2005.

X. Wen, B. Li, H. Guo, Z. Liu, G. Hu, M. Tang, and J. Wang. Adaptive variance based label distribution learning for facial age estimation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16 , pages 379-395. Springer, 2020.

Y. Xu, W. Guo, and Z. Wei. Conformal risk control for ordinal classification. In R. J. Evans and I. Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence , volume 216 of Proceedings of Machine Learning Research , pages 2346-2355. PMLR, 31 Jul-04 Aug 2023.

J. Yang, R. Shi, D. Wei, Z. Liu, L. Zhao, B. Ke, H. Pfister, and B. Ni. Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data , 10(1):41, 2023.

D. Yi, Z. Lei, and S. Z. Li. Age estimation by multi-scale convolutional network. In Computer Vision-ACCV 2014: 12th Asian Conference on Computer Vision, Singapore, Singapore, November 1-5, 2014, Revised Selected Papers, Part III 12 , pages 144-158. Springer, 2015.

APPENDIX

A Broader Impact

Our current work on contiguous conformal predictions for ordinal classification is foundational in nature and has multiple real-world applications.

Cancer Diagnosis. Given the huge costs of misprediction for high-stakes applications such as cancer diagnosis, instead of a single point prediction it is useful to predict a contiguous set. For instance, prediction set of [ stage 2 , stage 3 ] gives a better notion of severity than a non-contiguous set such as [ no cancer , stage 3 ] which might be discordant or a point prediction with low accuracy.

Dynamic Product Search Filters. Customers new to any e-commerce platform often experience heavy cognitive load in specifying their requirements via search filters (e.g., budget, product dimensions). Identifying a small highly likely set of options based on their typical profile or immediate session history would significantly enhance the usability of the search filters and improve the customer experience.

Personalised Fit Recommendations. Shopping apparel and shoes at any e-commerce platform is often tedious due to the limited support for fit-based recommendations. Often, customers find that the recommended products do not have options for their size and are forced to use search filters, which need to be repeatedly specified for each query. Additionally, customers also tend to order multiple products in the same size (bracketing) that results in a high return rate and excessive shipping costs for the platform. Automatic identification of the likely size ranges of a customer would improve the accuracy of recommendations and reduce shopping effort as well as return rates.

Personalised Budget Recommendations, Since budget ranges have a natural ordering, automatic personalisation of product and brand recommendations for customers based on their preferred budget ranges is another area that can leverage COPOC to improve customer satisfaction.

Abuse Incident Audits. E-commerce abuse incidents are often categorised along severity levels that have a natural ordering. Typically, human auditors are required to audit the abuse incidents, but current models do not often distinguish between a high chance of low severity incident vs. moderate chance of high severity incident. Conformal predictions can help streamline the audit workflows to better focus on the high severity incidents and optimise the overall outcomes both for e-commerce platform and the customers through expedited resolution.

B Theoretical Analysis

Lemma 1. Given a fitted unimodal model ˆ P Y | X , for any test x and α ∈ (0 , 1] , prediction sets ˆ S D,α ( x ) constructed using Eqn.6 or 7 has at least one solution which is contiguous (amongst multiple possibilities). When ˆ P Y | X is strictly unimodal, all the solutions are contiguous.

Proof: Let ˆ S D,α ( x ) be the prediction set with the shortest span (i.e., difference between highest and lowest included labels) as per Eqn.6 or 7.

Let l +1 and u denote the smallest and largest indices of the labels included in ˆ S D,α ( x ) so that the span is given by u -l .

Assuming ˆ S D,α ( x ) is non-contiguous implies that there exists at least one k skip such that ( l +1) < k skip < u and c k skip / ∈ ˆ S D,α ( x ) . Let ˆ p k ( x ) = ˆ P Y | X ( Y = c k | X = x ) . Since ˆ p ( x ) is unimodal, there are two possible scenarios depending on where k skip relies relative to the mode c ˆ m of ˆ p :

k skip < ˆ m : In this case, we have ˆ p k skip ≥ ˆ p l +1 since ˆ p is non-decreasing before the mode

k skip ≥ ˆ m : In this case, we have ˆ p k skip ≥ ˆ p u since ˆ p is non-increasing after the mode

Thus, ˆ p k skip ≥ min(ˆ p l +1 , ˆ p u ) .

Case 1: LAC- PS Construction follows Eqn. 6: For this case, we have,

where ˆ q D cal ( α ) is the bias-adjusted ( α ) th quantile of the model score of the true label. Since c l +1 and c u are included in ˆ S D,α ( x ) , it follows that both ˆ p l +1 ≥ ˆ q D cal ( α ) and ˆ p u ≥ ˆ q D cal ( α ) .

Since ˆ p k skip ≥ min(ˆ p l +1 , ˆ p u ) , it follows that ˆ p k skip ≥ ˆ q D cal ( α ) as well implying that c k skip ∈ ˆ S D,α ( x ) which leads to a contradiction. Hence, the shortest span ˆ S D,α ( x ) has to be contiguous.

Case 2: APS- PS Construction follows Eqn. 7: For this case, we have,

where π is a permutation of { 1 , . . . , K } that sorts ˆ p k in the descending order from most likely to least likely and ˆ q D cal ( α ) is the bias-adjusted (1 -α ) th quantile of the APS conformity scores as defined for Eqn. 7.

Let ˆ P sum ( S ) = ∑ c k ∈ S ˆ p k denote the (fitted) probability mass within the prediction set S . Due to the unimodality of ˆ p , it follows that one of the boundary labels c u and c l +1 have the minimum probability among those included in the set ˆ S D ,α ( x ) . Without loss of generality, let us assume ˆ p u is one of the minima (since the same argument can be applied for the case where ( l +1) is among the minima).

From the construction, we have, ˆ P sum ( ˆ S D ,α ( x )) ≥ ˆ q D cal ( α ) and ˆ P sum ( ˆ S D ,α ( x ) \{ c u } ) < ˆ q D cal ( α ) . Consider the sets S 1 = ˆ S D ,α ( x ) ⋃ { c k skip } \ { c u } and S 2 = S 1 \ { c k min } where k min is the largest index satisfying k min = arg min k | c k ∈ S 1 [ˆ p k ] . Since ˆ p k skip ≥ min(ˆ p l +1 , ˆ p u ) , it follows that ˆ P sum ( S 1 ) ≥

ˆ q D cal ( α ) . Further, from the definition of j as the size of largest top k set with probability mass as defined in Eqn. 7, it follows that ˆ P sum ( S 2 ) < ˆ q D cal ( α ) .

Therefore, the set S 1 is a valid APS prediction set as well with span ( k min -l ) < ( u -l ) , which leads to a contradiction. Thus, the shortest span ˆ S D,α ( x ) has to be contiguous for this case as well. Thus, in both cases, there exists at least one solution, i.e., shortest span prediction set, which is contiguous for both the constructions.

For the case, where ˆ p is strictly unimodal, from the constructio Eqn.6 or 7, the prediction sets have to contain the top k most likely classes for some k which results in contiguity in case of strict unimodality.

Theorem 1. For any x ∈ X , let p k ( x ) = P Y | X ( Y = c k | X = x ) and ˆ p k ( x ) = ˆ P Y | X ( Y = c k | X = x ) denote the true and fitted model class probabilities that are always unimodal. Let σ k ( x ) = ∑ k k ′ =1 p k ′ ( x ) and ˆ σ k ( x ) = ∑ k k ′ =1 ˆ p k ′ ( x ) denote the corresponding cumulative distribution functions. If | σ k ( x ) -ˆ σ k ( x ) | ≤ δ, [ k ] K 1 for a constant δ , then for any α ∈ (0 , 1] , ∀ x ∈ D test , the APS and oracle prediction sets from Eqn.7 and Eqn. 4 satisfy | ˆ S D,α ( x ) | ≤ | S oracle α -4 δ -1 n +1 ( x ) | where n is the size of the calibration set.

Proof. To establish the result, we prove that the following two statements hold true under the assumption on the CDFs of P Y | X and ˆ P Y | X :

(a) | ˆ S D,α ( x ) | ≤ | S Oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) |

Part (a): From Eqn. 4 and Lemma 1, we observe that the unimodality of ˆ p ( x ) and p ( x ) leads to the oracle prediction set being contiguous and also the existence of a contiguous APS prediction set.

Since all the APS solution sets as per Eqn 7 have the same cardinality, we use ˆ S D,α ( x ) to denote the contiguous solution.

Let ˆ S D,α ( x ) = { c ˆ l +1 , · · · , c ˆ u } , 0 ≤ ˆ l < ˆ u ≤ K and S oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) = { c l ∗ +1 , · · · , c u ∗ } , 0 ≤ l ∗ < u ∗ ≤ K . From the definition of the sets and the contiguity, we observe that the probability mass of ˆ S D,α ( x ) w.r.t. ˆ p equals (ˆ σ ˆ u ( x ) -ˆ σ ˆ l ( x )) ≥ ˆ q D cal ( α ) while that of S oracle 1 -ˆ q Dcal ( α ) -2 δ ( x ) w.r.t p equals ( σ u ∗ ( x ) -σ l ∗ ( x )) ≥ 1 -(1 -ˆ q D cal ( α ) -2 δ ) = ˆ q D cal ( α ) + 2 δ.

Using the divergence bound on the two CDFs, i.e., | σ k ( x ) -ˆ σ k ( x ) | ≤ δ, [ k ] K 1 , we have

Since ˆ S D,α ( x ) is the minimal contiguous set with probability mass greater than or or equal to ˆ q D cal ( α ) as per ˆ p in Eqn 7, we have

Part (b): Denoting the minimal contiguous APS prediction set by ˆ S D,α ( x ) as before, we have (ˆ σ ˆ u ( x ) -ˆ σ ˆ l ( x )) ≥ ˆ q D cal ( α ) . Considering the divergence bound on the two CDFs, i.e., | σ k ( x ) -ˆ σ k ( x ) | ≤ δ, [ k ] K 1 , we have (ˆ σ ˆ u ( x ) -ˆ σ ˆ l ( x )) ≤ ( σ ˆ u ( x ) + δ ) -(( σ ˆ l ( x ) -δ ) = σ ˆ u ( x ) -σ ˆ l ( x ) + 2 δ .

Hence, for all x , we have

Since this holds for all x , the marginal coverage P [ Y ∈ ˆ S D,α ( X )] ≥ ˆ q D cal ( α ) -2 δ .

From Theorem 3, we also have an upper bound on the marginal coverage for test samples, i.e., P [ Y ∈ ˆ S D,α ( X )] ≤ 1 -α + 1 n +1 where n is the size of the calibration set.

Hence, we have

From the above inequality and the definition of the oracle prediction set, we observe that

Combining the results in part (a) and (b), we have

As the size of the calibration set increases, the term 1 n +1 vanishes and as the divergence δ decreases, then the cardinality of the APS set converges to that of the oracle set.

Theorem 2. Let η : X → R K , ϕ : R → R + and ψ E : R → R -such that ψ E ( r ) = ψ E ( -r ) , ∀ r ∈ R and its restriction to R + is a strictly monotonically decreasing bijective function. (a) Then, the model output constructed as per Eqn. 5 is always unimodal, i.e., ˆ p ( x ) ∈ U , ∀ x ∈ X . (b) Further, given any ˆ p ( x ) ∈ U for x ∈ X , there exists a function η ( x ) ∈ R K that satisfies Eqn. 5 if ϕ ( · ) is surjective on R + .

Proof. We begin by restating the construction:

P1

p2

pK

SOFTMAX

Z1

ZK

T1

T2

4

r3

rK

V2

V3

VK

Φ:R→R+

V1

天工工工工工菜

Inputx

Part a : Following the above construction, for any x ∈ X , since ϕ : R → R + , the DNN output v k ≥ 0 , [ k ] K 2 . Hence, the cumulative sum sequence r is non-decreasing, i.e., r 1 ≤ r 2 ≤ · · · ≤ r K .

There can be 3 possible scenarios:

Scenario 1. r 1 ≤ r 2 · · · ≤ r K ≤ 0 : In this case, [ z k = ψ E ( r k )] K k =1 is also a non-decreasing sequence and so is [ˆ p k ] K k =1 . Here [ˆ p k ] K k =1 is unimodal sequence with mode at c K .

Scenario 2. 0 ≤ r 1 ≤ r 2 · · · < = r K : In this case, [ z k = ψ E ( r k )] K k =1 is also a non-increasing sequence and so is [ˆ p k ] K k =1 . Here [ˆ p k ] K k =1 is unimodal sequence with mode at c 1 .

Scenario 3. r 1 ≤ r 2 · · · ≤ r m ≤ 0 ≤ r m +1 ≤ ·· · ≤ r K for some m . In this case, [ z k = ψ E ( r k )] K k =1 is non-decreasing till m and non-increasing from m +1 onwards, which makes it unimodal. The mode is either m or m +1 or both depending on the magnitudes | r m | and | r m +1 | . The probability distribution [ˆ p k ] K k =1 follows the same pattern and is unimodal as well.

Part b : Let us assume ˆ p ( x ) ∈ U is any arbitrary unimodal distribution conditioned on x with class probabilities ˆ p k ≤ ˆ p k +1 , [ k ] m -1 1 and ˆ p k ≥ ˆ p k +1 , [ k ] K m , where m is the highest indexed (in case of multiple) mode of the unimodal distribution. We can then obtain z k = log(ˆ p k ) , [ k ] K 1 and construct the sequence r k = ( ψ E ) -1 ( z k ) where r k ∈ R -for 1 ≤ k ≤ ( m -1) and r k ∈ R + for m ≤ k ≤ K . Since ψ : R + → R -is a strictly monotonically decreasing bijective function and ψ E is it's even extension, the sequence [ r k ] K k =1 is well-defined. Further, since [ z k ] K k =1 is a unimodal sequence, [ r k ] K k =1 is monotonically increasing with r m -1 ≤ 0 ≤ r m . Then, we can obtain the vector v such that v k = r k -r k -1 ≥ 0 , [ k ] K 2 and v 1 = r 1 . When ϕ ( · ) is a surjective function on R + , we can define η k ( x ) = ( ϕ ) -1 ( v k ) , [ k ] K 2 and η 1 ( x ) = v 1 . There will always be a valid η ( x ) , which ensures that construction can generate the original ˆ p ( x ) .

B.1 APS Coverage guarantees

Theorem 3. [APS [35]] If samples ( x i , y i ) x i ∈ X , y i ∈ Y are exchangeable ∀ 1 ≤ i ≤ n and all samples from D train , D cal are invariant to permutations, and conformity scores are almost surely distinct, then APS algorithm gives tight marginal coverage given by:

C Experiment Details

C.1 Benchmark Image Datasets and Implementation Details

We now provide a brief description of the four public datasets and the modeling details. For each of these datasets, we split the data into train, calibration, and test sets. We use calibration set to calibrate APS and report mean and standard deviation (std. error) on the test set across 5 independent splits. Note that for all experiments to avoid over-fitting, data augmentation, i.e., random horizontal flipping and random cropping for each training image, was applied in our experiments. The predictions was obtained with a central crop during testing. COPOC was implemented with ϕ = | x | and ψ = -| x | .

Age Estimation - Adience [13] : The task associated with this dataset is to predict the age for a given facial image. This dataset contains 26580 Flickr photos of 2284 subjects. The age is annotated with eight groups: 0 -2 , 4 -6 , 8 -13 , 15 -20 , 25 -32 , 38 -43 , 48 -53 , and over 60 years old. From the nature of the class labels, it is evident that classes are not equally spaced categories. Hence, previous works which assumed it to be equi-spaced (SORD [12] for instance) are suboptimal. For feature extractor backbone, we use ImageNet pre-trained VGG-16 network since most competing methods [23, 12] used this model. For our usage we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2. We trained models for 50 epochs with a batch size of 64 . For optimization, Adam optimizer was utilized with a learning rate of 0 . 0001 , with decay rate of 0 . 2 .

Historical Colour Image Dating - HCI [30] : The historical color image dataset is collected for the task of estimating the age of historical color photos. Each image is annotated with its associated decade, where five decades from the 1930 s to 1970 s are considered. There are 265 images for each category. Following [23] we utilized VGG-16 as the backbone, which was initialized with the ImageNet pre-trained weights for a fair comparison. We trained models for 50 epochs with Adam optimizer with a learning rate of 0 . 0001 , with decay rate of 0 . 2 . For COPOC, we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2.

Retina-MNIST [44] : RetinaMNIST is based on the DeepDRiD24 challenge, which provides a dataset of 1600 retina fundus images. The task is ordinal classification for 5-level grading of diabetic retinopathy severity. We use a similar feature extractor network as used in [44] along with a final unimodality constrained layer at end. The network was trained with same settings as [44].

Image Aesthetic Estimation [37] : The Aesthetics dataset consists of 15687 Flickr image belonging to four different nominal categories: animals, urban, people, and nature. All The pictures are annotated by 5 different graders in 5 aesthetic categories in an orderly manner: 1) 'unacceptable' pictures with extremely low quality, 2) 'flawed' low quality images (slightly blurred,over/underexposed), and with no artistic value; 3) 'ordinary' images without technical flaws (well framed, in focus), but no artistic value; 4) 'professional' images (flawless framing,lightning),and 5) 'exceptional', very appealing images, showing outstanding quality. The ground truth label for each image is set to be the median among all of its gradings. Following [23, 12] we use ImageNet pre-trained VGG-16 as the backbone for feature extraction. For our usage, we append single layer MLP with last layer configured to output unimodal distribution as described in sec. 4.2. We only report aggregate metric across all the categories for this data.

Additional Implementation Details: On the public benchmark datasets where the official best hyperparameters are available for baseline methods (For instance in Adience, HCI and Aesthetic dataset best settings for POE and SORD, and for Binomial best settings on Adience were available) from the corresponding authors work or code, we directly use those settings. We were able to replicate the results (MAE and Accuracy) on these datasets as reported by them. For all other cases (namely AVDL, Uni-Loss), we optimize for MAE in hyperparameters search since that is the

most common metric used for all ordinal classification tasks across competing benchmarks. We cross-validate over the following grid:

learning rate ∈ { 1 e -2 , 1 e -3 , 1 e -4 } with decay rate of 0 . 2 .

weight decay ∈ { 0 , 1 e -3 , 1 e -2 , 1 e -1 } .

dropout rate ∈ { 0 . 1 , 0 . 25 , 0 . 5 , 0 . 75 } .

Adam optimizer with default settings.

Few additional algorithm specific hyperparameters that needed tuning were:

For POE, there are two extra hyperparameters of α and β in its distance-aware loss function in embedding space which we search over { 1 e -3 , 1 e -4 , 1 e -5 , 1 e -6 } as suggested by the authors.

SORD describes three type of distance metric losses - absolute inter class distance, squared distance and its log variant. We search over these loss functions too.

AVDL requires choosing the initial variance (of the Gaussian) of all images which we search from { 0 . 25 , 0 . 5 , 1 , 2 } similar to their work.

Uni-loss has hyperparameter that controls the weightage between unimodality and meanvariance component of its loss function. We search over { 10 , 100 , 500 , 1000 , 5000 } .

Result Discussion : We highlight the key takeaways from Table 6.

1. COPOC performs at par with SOTA baselines in terms of MAE and Acc @1 .

2. Benefit of COPOC comes with improved gains in Acc @2 and Acc @3 . Apart from COPOC, there is no single method that performs consistently across the 4 datasets in terms of these metrics. For instance in HCI and Adience , Binomial-temp comes closest to COPOC , but on Aesthetic , both variants of Binomial severely under-perform whereas AVDL and SORD perform quite well and comes the closest to COPOC . In contrast, on Retina-MNIST , non-parametric models such as V-CE , POE , Uni-loss have Acc@k close to COPOC and beat other parametric models significantly. This shows that parametric distribution assumption in any underlying model fits the data well when the data is actually drawn from a similar distribution. Since most methods depend largely on the validity of the assumptions, the relatively unconstrained parameter-free nature of COPOC is more robust and allows it to consistently outperform across datasets.

3. There is a strong correlation between CV % and PS size. This is expected because higher CV % indicates more number of cases for which we had to predict a minimal contiguous super set, thus inflating the size of PS. Better unimodal fit by underlying model is bound to have lesser CV % and and thus, shorter sets. Hence, COPOC again outperforms all other baselines across datasets in term of |PS| consistently. Although Binomial model variants has 0 CV % due to it's construction, it still produces larger sets than COPOC as seen in HCI and Adience . This can be because COPOC results in better unimodal fit which is also idicated by higher Acc @ K .

4. Enforcing unimodality in training scheme in terms of soft-labels ( SORD , AV DL ) or in loss function ( Uni-loss ) or in embedding space ( POE ) does not necessarily translate to a unimodal distribution in test samples which is indicated by high CV % .

5. Although V -CE in principle should have been able to model any underlying distribution, on high dimensional real-world datasets it fails miserably. This shows the need for injecting prior "bias" into training network like COPOC which aids the model in reaching the optima.

6. Uni-loss has issues with model convergence as it shows high variance across metrics for all datasets. This could be because its sensitive to λ hyperparamter that control the weightage between unimodality and mean-variance component of its loss function which is difficult to tune.

7. Datasets with higher accuracy results in shorter PS size in general, which is expected. For instance Aesthetic has lower PS size across methods compared to HCI or Retina-MNIST both having same number of class labels.

C.2 Implementation details of experiments on synthetic Data

For all the results on synthetic datasets presented in Sec.5.3, we employ same DNN network across all the methods for fair comparison. To be precise, we use 6 layer DNN architecture having 128 hidden dimensions with a dropout of 0 . 2 . We use the same training paradigm as before - Adam optimizer with learning rate of 0 . 001 and batch size of 512 trained for 500 epochs ensuring convergence. We divide the data into 70% train and 30% test splits. We train our model 10 times for each independent split of the data. For each test set, we again randomly split into calibration for APS and evaluate |PS| on final-test and repeat this 100 times to ensure convergence of PS. We use ϕ = | x | and ψ = -| x | for COPOC. We report mean and standard error in Table 2.

C.3 Ablation study on the choice of ϕ ( · ) and ψ E ( · ) for COPOC

Although there can be many possible choices for ϕ ( · ) and ψ E ( · ) in the COPOC construction Eqn. 5, in practice not all choices leads to good model convergence. In this section, we perform a comparison of few common choices and present results in Table 7. We train the model for synthetic data D4 as described in Sec. 5.3. We train the model using CE loss with same model capacity and training paradigm as described in Appendix C.2. We report train CE loss and since we have access to true underlying distribution for D4 we report KL. Div. to measure goodness of model fit. Below we summarise few observations:

ϕ = ReLU maps most of [ v k ] K 2 to zeroes which results in flat probability distribution for most of the data points while ϕ = Softplus instead maps most [ v k ] K 2 to very small values which again results in almost flat distribution for most points. With ϕ = x 2 we observed unusually large values for [ v k ] K 2 resulting in unstable training. ϕ = | x | gives a good balance as each [ η k ] K 2 gets linearly mapped to [ v k ] K 2 .

ψ = -| x | 2 tends to over-emphasize higher probability classes in the model fitting while ψ = -| x | 0 . 5 under-emphasizes them. Again since ψ = -| x | does a linear transformation of r k on either side of the origin it gives a good balanced estimate of z k .

C.4 Illustrative Comparison Between Unimodal and Non-Unimodal Model Fit

Naive DNN (V-CE)

COPOC

Ground Truth

Class

Predicted P(Y|X)

C1

C2

C4C5C6

C7

C8

C1

C2

C4C5C6

C7

C8

C1

C2

C3C4C5C6

G7

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConformal prediction for classification considers non-ordinal classes, which potentially produces sub-optimal prediction set size for ordinal classification. The proposed approach addresses this problem for a unimodal label distribution. To this end, the proposed approach designs a novel conformity score function that fully utilizes the unimodal assumption, so a trained score function strictly returns a unimodal distribution over labels. The efficacy of the proposed approach is theoretically justified in theorems and empirically demonstrated over one synthetic dataset and four real datasets, showing that the proposed approach consistently achieves smaller set size compared to baselines and always returns contiguous sets (as claimed).\n\nSTRENGTHS:\n**Originality**: To my understanding, this is a novel usage of conformal prediction to ordinal classification. Moreover, the paper proposes a simple yet effective sufficiently-parameterized scoring function (i.e., (5)) that always returns contiguous sets under the unimodal assumption. Even better, this new score function design is theoretically justified in Theorem 2. This theorem is also empirically justified (via CV% = 0). \n\nGiven the unimodal score function, existing conformal prediction algorithms intuitively return a contiguous set. But, this is carefully analyzed in Theorem 1. \n\n**Quality**: I think the paper quality is good. The notations are well-defined, the claims are rigorously analyzed via theorems, and the main claim is also well-justified empirically. \n\n**Clarity**: The paper is clearly written (and thanks for errata in Appendix).\n\n**Significance**: I believe the proposed scoring function (5) is sufficiently novel (and simple) to bring interesting related papers in conformal prediction and ordinal classification. \n\nLast but not least, this paper sufficiently motivates the practical necessity of conformal prediction for ordinal classification in introduction (by using examples, like proximal shoe size). I like this practical connection of conformal prediction in real scenarios, which is usually missing in conformal prediction papers.\n\nWEAKNESSES:\nI have minor comments. \n\n* My main concern is the unimodal assumption (which is also mentioned in the limitation section). Based on the empirical results, many distributions for ordinal classification closely satisfy the unimodal assumptions (based on the results that the proposed approach’s CV% is zero). Even though it is violated, I think it does not affect coverage rate. Even though It looks good to discuss potential mitigations if this assumption is violated. \n* When I read Section 4.2, I wanted to see a simple illustration between a label distribution from a naive DNN and that of (5), which contrasts the naive approach fails to achieve a unimodal output, while (5) does. If the left image of Figure 1 is an illustration of the actual data, it’s better to highlight this.\n\nQUESTIONS:\nI also only have minor questions. \n\n* What can be potential mitigations if the unimodal assumption is violated?\n* For Table 3, is there a particular reason that LAC and APS are compared via SSCV instead of the coverage rate?\n\nLIMITATIONS:\nThe limitations that I expected are discussed in Conclusion, and as mentioned before, it would be more interesting to dicuss possible mitigations on these limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method for ordinal classification, COPOC, which guarantees (by functional form) unimodal prediction distributions over the ordered classes and consequently guarantees contiguous prediction sets for uncertainty estimation via conformal prediction . The authors argue that unimodality is desirable for many ordinal classification tasks, e.g. the prediction set for customer shoe sizes should be something like size 5,6,7 rather than e.g. sizes 4,8,9.  A theoretical result is obtained which bounds the size of the fitted model prediction set in terms of the size of the ground truth prediction set as well as the distance between the fitted prediction distribution and the ground truth prediction distribution for a given desired conformal prediction coverage level. Favorable experimental results in comparison to competing methods are presented on a suite of real-world ordinal image classification datasets.  COPOC is compared to competing techniques on a variety of metrics, including accuracy, size of prediction set and contiguity of prediction set. Experiments on synthetic datasets are used to illustrate the consistency of COPOC as well as to examine the merits of different possible choices of conformal prediction algorithm.\n\nSTRENGTHS:\nOverall, the work is original to the best of my knowledge.\n\nI find the writing and presentation to be clear for the most part. \n \nThe architecture presented in section 4.2 seems well-designed and does indeed seem to have just the right level of inductive bias, if we take for granted that unimodal predictive distributions are desirable. It guarantees unimodality without making additional parametric assumptions.\n\nAlthough I would have preferred more detail on the real world experiments, the breadth of results and metrics evaluated seems adequate to me. 4 datasets evaluated on 6 metrics is a reasonably thorough set of experiments.\n\nAlthough I would not characterize the topic as being highly significant or extremely important, the significance seems adequate to me for NeurIPs acceptance.  Conformal prediction is a highly useful technique in which interest continues to grow. Ordinal classification, while not a top priority for many modelers, is important enough in certain applications that its intersection with conformal prediction is a worthwhile topic.\n\nWEAKNESSES:\nAlthough unimodal prediction distributions may often be desirable for ordinal classification, I find the case for unimodality to be overstated. Although the authors do acknowledge on line 357 (in the limitations  section) that unimodality might not hold, I would have preferred a bit more discussion of the scenarios where it might not hold. For instance, I have some experience with recommendation systems (Netflix-style 5 star ratings) and there are merits to models in that domain which might detect \"love-it-or-hate-it\" scenarios where both 1-star and 5-star are more likely than 3-star.  Even for cancer stage detection, I think it's possible that non-unimodal predictions could be desirable. One could imagine that there are 2 possible hypotheses explaining an observed symptom. If the observed symptom is due to a preexisting condition unrelated to the cancer, then the overall evidence might point to stage 1. If the observed symptom is due to the cancer itself, stage 4 might be implied. So more of an acknowledgement that unimodality might lead you astray would be good here.\n\nI also find the details of the real-world-dataset experiments to be a bit lacking. There is minimal discussion of hyperparameter optimization. I would have more confidence in the results if I knew a thorough attempt was made to optimize the hyperparameters of each method. Although I don't have a detailed understanding of the competing methods, the nonparametric ones (POE, Uni-Loss) would (at first glance) appear to have some hyperparameters associated with them and it's not clear to me whether those hyperparameters were optimized.  I think more details on the real-world experiments and less detail on the synthetic experiments would be a better use of the 9 pages allowed.\n\nSome suggested typo-ish edits:\n\nLine 137 consider minimal contiguous set -> consider the minimal contiguous set\nLine 169 has->have\nLine 242 of underlying distribution -> of the underlying distribution\nLine 261 Accuracy@k lowercase k vs capital K earlier (line 12 in abstract) is inconsistent. \n\n***** Update post -rebuttal ***\n\nIn light of the additional information provided by the authors (both regarding the appropriateness of the unimodality assumption and the hyperparameter tuning), I have raised my score to a 7.\n\nQUESTIONS:\nDid you optimize the hyperparameters of each competing method on some sort of validation set before doing the final test set evaluation for each method? If so, for which of the various metrics reported was the hyperparameter optimization conducted?\n\nLIMITATIONS:\nI think the limitations are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explained the difference between their method and prior work satisfyingly.\nI had a misunderstanding in my previous reading.\nI'd be happy to support the paper's acceptance.\nIt is a minor contribution from a theoretical perspective, but I agree that practically, it's probably a better way of constructing unimodal prediction sets than Lu et al. \n_______________\nThe paper proposes a modification of conformal prediction for the ordinal classification case.\nThe idea is that, when the distribution is unimodal, you should always be building continuous intervals; so you restrict the family of possible prediction sets to be the ones that are only contiguous.\n\nSTRENGTHS:\nThe paper is relatively clear, and the problem is important.\n\nWEAKNESSES:\nPost-rebuttal note: I was wrong about the below.\nThe methods are definitely related, but not identical.\nLu et al. applies the constraint of unimodality during the set construction.\nThe present manuscript does so during the model training.\n_______\nThe method is not novel, and was previously proposed in this reference:\n\nLu, C., Angelopoulos, A. N., & Pomerantz, S. (2022, September). Improving trustworthiness of AI disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 545-554). Cham: Springer Nature Switzerland.\n\n(Equation 2 in https://arxiv.org/abs/2207.02238 is the same as Equation 1 in the present manuscript.)\n\nThere has also been since a followup of that work on risk control: https://openreview.net/forum?id=9R5qObx8WO5 .\n\nFor this reason, I do not believe the paper contains any novel methodological advances, and thus cannot be accepted.\n\nQUESTIONS:\nCan the authors clarify my comment about novelty? If I have misunderstood, I would be happy to hear it.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of adapting the conformal prediction methods to ordinal classification so that the predictor outputs contiguous prediction sets. Contributions can be split into two parts. The first part deals with adopting the existing conformal prediction methods to ordinal classification. The main observation is that the predictor with unimodal posterior over classes is enough to guarantee that the conformal prediction methods will return contiguous prediction sets. This part is relatively straightforward. The second part involves proposal of a novel non-parametric method for training NN which outputs unimodal posterior over class labels. The proposed method is empirically evaluated and shown to provide competitive results when compared to existing ordinal classification methods used for conformal prediction.\n\nSTRENGTHS:\nThe paper is sound and it is very clearly written. \n\nA new non-parametric method for training an NN-based ordinal regression predictor with unimodal posterior over class labels is a simple elegant idea, which is to my knowledge novel and potentially useful in practice; not only in the context of confomal prediction.\n\nWEAKNESSES:\nI have no major objections.\n\nQUESTIONS:\nIn the experiments on real data, tab 1, the proposed method does not perform best in terms of MAE which might be attributed to the fact that true posterior might not be exactly unimodal. It would be instructive to see whether this changes on the synthetic data, which presumably are generated from an unimodal distribution (although whether it is true is not clear to me; the description should be more explicit in this respect). I.e. I would suggest to report also MAE in table 5.3.\n\n---\nThe authors satisfactorily addressed my questions in the rebuttal. I keep my positive ratings.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}