Title: Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration
Venue: NeurIPS 2024
Accepted: yes
Keywords: Uncertainty Quantification

Conformal Prediction

Imbalanced Data

Class-conditional Coverage

Deep Models
Abstract: Conformal prediction (CP) is an emerging uncertainty quantification framework that allows us to construct a prediction set to cover the true label with a pre-specified marginal or conditional probability.
Although the valid coverage guarantee has been extensively studied for classification problems, CP often produces large prediction sets which may not be practically useful.
This issue is exacerbated for the setting of class-conditional coverage on imbalanced classification tasks with many and/or imbalanced classes.
This paper proposes the Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce the prediction set sizes to achieve class-conditional coverage, where the valid coverage holds for each class.
In contrast to the standard class-conditional CP (CCP) method that uniformly thresholds the class-wise conformity score for each class, the augmented label rank calibration step allows RC3P to selectively iterate this class-wise thresholding subroutine only for a subset of classes whose class-wise top-$k$ error is small.
We prove that agnostic to the classifier and data distribution, RC3P achieves class-wise coverage. We also show that RC3P reduces the size of prediction sets compared to the CCP method. 
Comprehensive experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and  $26.25\\%$ $\downarrow$ reduction in prediction set sizes on average.

Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration

Abstract

Conformal prediction (CP) is an emerging uncertainty quantification framework that allows us to construct a prediction set to cover the true label with a pre-specified marginal or conditional probability. Although the valid coverage guarantee has been extensively studied for classification problems, CP often produces large prediction sets which may not be practically useful. This issue is exacerbated for the setting of class-conditional coverage on classification tasks with many and/or imbalanced classes. This paper proposes the Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce the prediction set sizes to achieve class-conditional coverage, where the valid coverage holds for each class. In contrast to the standard class-conditional CP (CCP) method that uniformly thresholds the class-wise conformity score for each class, the augmented label rank calibration step allows RC3P to selectively iterate this class-wise thresholding subroutine only for a subset of classes whose class-wise topk error is small. We prove that agnostic to the classifier and data distribution, RC3P achieves class-wise coverage. We also show that RC3P reduces the size of prediction sets compared to the CCP method. Comprehensive experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and 26 . 25% ↓ reduction in prediction set sizes on average.

1 Introduction

Safe deployment of machine learning (ML) models in high stakes applications such as medical diagnosis requires theoretically-sound uncertainty estimates. Conformal prediction (CP) [60] is an emerging uncertainty quantification framework that constructs a prediction set of candidate output values such that the true output is present with a pre-specified level (e.g., ≥ 90% ) of the marginal or conditional probability [65, 19].

A promising property of CP is the model-agnostic and distribution-free coverage validity under certain notions [20]. For example, marginal coverage is the commonly studied validity notion [47, 1, 65], while conditional coverage is a stronger notion. There is a general taxonomy to group data (i.e., input-output pairs) into categories and to study the valid coverage for each group (i.e., the group-wise validity) [61, 60]. This paper focuses on the specific notion of class-conditional coverage that guarantees coverage for each class individually, which is important for classification tasks with many and/or imbalanced classes (e.g., medical applications) [39, 56, 38].

In addition to the coverage validity, predictive efficiency is another important criterion for CP [20, 59], which refers to the size of the prediction sets. Both coverage validity and predictive efficiency are used together to measure the performance of CP methods [1, 45, 47, 15, 22, 18]. Since the two measures are competing [1], our goal is to guarantee the coverage validity with high predictive efficiency, i.e., small prediction sets [20, 47, 18]. Some studies improved the predictive efficiency under the

marginal coverage setting using new conformity score function [1] and new calibration procedures [19, 18, 26, 21]. However, it is not known if these methods will benefit the predictive efficiency for the class-conditional coverage setting. A very recent work [15] proposed the cluster CP method to achieve approximate class-conditional coverage. It empirically improves predictive efficiency over the baseline class-wise CP method (i.e., each class is one cluster) [58], but the approximation guarantee for class-wise coverage is model-dependent (i.e., requires certain assumptions on the model). The main question of this paper is: how can we develop a model-agnostic CP algorithm that guarantees the class-wise coverage with improved predictive efficiency (i.e., small prediction sets)?

To answer this question, we propose a novel approach referred to as Rank Calibrated Class-conditional CP (RC3P) that guarantees the class-wise coverage with small expected prediction sets. The classconditional coverage validity of RC3P is agnostic to the data distribution and the underlying ML model, while the improved predictive efficiency depends on very mild conditions of the given trained classifier. The main ingredient behind the RC3P method is the label rank calibration strategy augmented with the standard conformal score calibration from the class-wise CP (CCP) [58, 2].

The CCP method finds the class-wise quantiles of non-conformity scores on calibration data. To produce the prediction set for a new test input X test , it pairs X test with each candidate class label y and includes the label y if the non-conformity score of the pair ( X test , y ) is less than or equal to the corresponding class-wise quantile associated with y . Thus, CCP constructs the prediction set by uniformly iterating over all candidate labels. In contrast, the label rank calibration allows RC3P to selectively iterate this class-wise thresholding subroutine only if the label y is ranked by the classifier f ( X test ) (e.g., f ( · ) denotes the softmax prediction) in the top k y candidates, where the value of k y is calibrated for each label y individually according to the class-wise topk y error. In other words, given X test, RC3P enables standard class-wise conformal thresholding for the sufficiently certain class labels only (as opposed to all labels). Our theory shows that the class-wise coverage provided by RC3P is agnostic to the data distribution and the underlying ML model. Moreover, under a very mild condition, RC3P guarantees improved predictive efficiency over the baseline CCP method.

Contributions. The main contributions of this paper are:

We design a novel algorithm RC3P that augments the label rank calibration strategy to the standard conformal score calibration step. To produce prediction sets for new inputs, it selectively performs class-wise conformal thresholding only on a subset of classes based on their corresponding calibrated label ranks.

We develop theoretical analysis to show that RC3P guarantees class-wise coverage, which is agnostic to the data distribution and trained classifier. Moreover, it provably produces smaller average prediction sets over the baseline CCP method [58].

We perform extensive experiments on multiple imbalanced classification datasets and show that RC3P achieves the class-wise coverage with significantly improved predictive efficiency over the existing class-conditional CP baselines ( 26 . 25% reduction in the prediction size on average on all four datasets or 35% reduction excluding CIFAR-10). The code is available at https://github.com/YuanjieSh/RC3P.

2 Related Work

Precise uncertainty quantification of machine learning based predictions is necessary in high-stakes decision-making applications. It is especially challenging for imbalanced classification tasks. Although many imbalanced classification learning algorithms [10, 25] are proposed, e.g., re-sampling [11, 42, 33, 54, 63] and re-weighting [28, 40], they do not provide uncertainty quantification with rigorous guarantees over predictions for each class.

Conformal prediction [62, 60] is a model-agnostic and distribution-free framework for uncertainty quantification by producing prediction sets that cover the true output with a pre-specified probability, which means CP could provide valid coverage guarantee with any underlying model and data distribution [32, 52, 16]. Many CP algorithms are proposed for regression [35, 46, 23, 17], classification [45, 1, 64, 37], structured prediction [6, 3, 13, 30], online learning [24, 7], and covariate shift [31, 53, 5] settings. Coverage validity and predictive efficiency are two common and competing desiderata for CP methods [1]. Thus, small prediction sets are favorable whenever the coverage validity is guaranteed [20, 47, 18], e.g., human and machine learning collaborative systems

[39, 56, 38]. Recent work 1 improved the predictive efficiency for marginal coverage by designing new conformity score [1] and calibration procedures [19, 18, 26, 21]. These methods can be combined with class-conditional CP methods including RC3P as we demonstrate in our experiments, but the effect on predictive efficiency is not clear.

In general, the methods designed for a specific coverage validity notion are not necessarily compatible with another notion of coverage, such as object-conditional coverage [58], class-conditional coverage [58], local coverage [36] which are introduced and studied in the prior CP literature [61, 60, 20, 15, 9]. The standard class-conditional CP method in [58, 49] guarantees the class-wise coverage, but does not particularly aim to reduce the size of prediction sets. The cluster CP method [15] which performs CP over clusters of labels achieves a cluster-conditional coverage that approximates the class-conditional guarantee, but requires some assumptions on the underlying clustering model.

Our goal is to develop a provable class-conditional CP algorithm with small prediction sets to guarantee the class-wise coverage that is agnostic to the underlying model.

3 Notations, Background, and Problem Setup

Notations. Suppose ( X,Y ) is a data sample where X ∈ X is an input from the input space X , and Y ∈ Y = { 1 , 2 , · · · , K } is the ground-truth label with K candidate classes. Assume ( X,Y ) is randomly drawn from an underlying distribution P defined on X × Y , where we denote p y = P XY [ Y = y ] . Let f : X → ∆ K + denote a soft classifier (e.g., a soft-max classifier) that produces prediction scores for all candidate classes on any given input X , where ∆ K + denote the K -dimensional probability simplex and f ( X ) y denotes the predicted confidence for class y . We define the class-wise topk error for class y from the trained classifier f as ϵ k y = P { r f ( X,Y ) > k | Y = y } , where r f ( X,Y ) = ∑ K l =1 ✶ [ f ( X ) l ≥ f ( X ) Y ] returns the rank of Y predicted by f ( X ) in a descending order, and ✶ [ · ] is an indicator function. We are provided with a training set D tr for training the classifier f , and a calibration set D cal = { ( X i , Y i ) } n i =1 for CP. Let I y = { i : Y i = y, for all ( X i , Y i ) ∈ D cal } and n y = |I y | denote the number of calibration examples for class y .

Problem Setup of CP. Let V : X × Y → R denote a non-conformity scoring function to measure how different a new example is from old ones [60]. It is employed to compare a given testing sample ( X test , Y test ) with a set of calibration data D cal: if the non-conformity score is large, then ( X test , Y test ) conforms less to calibration samples. Prior work has considered the design of good non-conformity scoring functions, e.g., [2, 50, 47]. In this paper, we focus on the scoring functions of Adaptive Prediction Sets (APS) proposed in [47] and Regularized APS (RAPS) proposed in [1] for classification based on the ordered probabilities of f and true label rank r f ( X,Y ) . For the simplicity of notation, we denote the non-conformity score of the i -th calibration example as V i = V ( X i , Y i ) .

Given a input X , we sort the predicted probability for all classes { 1 , · · · , K } of the classifier f such that 1 ≥ f ( X ) (1) ≥ · · · ≥ f ( X ) ( K ) ≥ 0 are ordered statistics, where f ( X ) ( k ) denotes the k -th largest prediction. The APS [47] score for a sample ( X,Y ) is computed as follows:

where U ∈ [0 , 1] is a uniform random variable to break ties. We also consider its regularized variant RAPS [1], which additionally includes a rank-based regularization λ ( r f ( X,Y ) -k reg ) + to the above equation, where ( · ) + = max { 0 , ·} denotes the hinge loss, λ and k reg are two hyper-parameters.

For a target coverage 1 -α , we find the corresponding empirical quantile on calibration data D cal defined as which can be determined by finding the ⌈ (1 -α )(1+ n ) ⌉ -th smallest value of { V i } n i =1 . The prediction set of a testing input X test can be constructed by thresholding with ̂ Q 1 -α :

Therefore, ̂ C 1 -α gives a marginal coverage guarantee [47, 1]: P ( X,Y ) ∼P { Y ∈ ̂ C 1 -α ( X ) } ≥ 1 -α. To achieve the class-conditional coverage , standard CCP [58] uniformly iterates the class-wise thresholding subroutine with the class-wise quantiles { ̂ Q class 1 -α ( y ) } y ∈Y :

Specifically, CCP pairs X test with each candidate class label y , and includes y in the prediction set ̂ C CCP 1 -α ( X test ) if V ( X test , y ) ≤ ̂ Q class 1 -α ( y ) holds. After going through all candidate class labels y ∈ Y , it achieves the class-wise coverage for any y ∈ Y [58, 2]:

CCP produces large prediction sets which are not useful in practice. Therefore, our goal is to develop a provable CP method that provides class-conditional coverage and constructs smaller prediction sets than those from CCP. We summarize all the notations in Table 3 of Appendix.

4 Rank Calibrated Class-Conditional CP

We first explain the proposed Rank Calibrated Class-conditional Conformal Prediction (RC3P) algorithm and present its model-agnostic coverage guarantee. Next, we provide the theoretical analysis for the provable improvement of predictive efficiency of RC3P over the CCP method.

4.1 Algorithm and Model-Agnostic Coverage Analysis

We start with the motivating discussion about the potential drawback of the standard CCP method in terms of predictive efficiency . Equation (1) shows that, for a given test input X test, CCP likely contains some uncertain labels due to the uniform iteration over each class label y ∈ Y to check if y should be included into the prediction set or not. For example, given a class label y and two test samples X 1 , X 2 , suppose their APS scores are V ( X 1 , y ) = 0 . 9 , V ( X 2 , y ) = 0 . 8 , with ranks r f ( X 1 , y ) = 1 , r f ( X 2 , y ) = 5 . Furthermore, if ̂ Q class 1 -α ( y ) = 0 . 85 , then by (1) for CCP, we know that y / ∈ ̂ C CCP 1 -α ( X 1 ) and y ∈ ̂ C CCP 1 -α ( X 2 ) , even though f ( X 1 ) ranks y at the #1 class label for X 1 with a very high confidence f ( X 1 ) y = 0 . 9 and CCP can still achieve the valid class-conditional coverage. We argue that, the principle of CCP to scans all y ∈ Y uniformly can easily result in large prediction sets, which is detrimental to the effectiveness of human-ML collaborative systems [4, 51].

Consequently, to improve the predictive efficiency of CCP (i.e., reduce prediction set sizes), it is reasonable to include label rank information in the calibration procedure to adjust the distribution of non-conformity scores for predictive efficiency. As mentioned in the previous sections, better scoring functions have been proposed to improve the predictive efficiency for marginal coverage, e.g., RAPS. However, directly applying RAPS for class-wise coverage presents challenges: 1) tuning its hyper-parameters for each class requires extra computational overhead, and 2) fixing its hyper-parameters for all classes overlooks the difference between distributions of different classes. Moreover, for the approximate class-conditional coverage achieved by cluster CP [15], it still requires some assumptions on the underlying model (i.e., it is not fully model-agnostic).

Therefore, the key idea of our proposed RC3P algorithm (outlined in Algorithm 1) is to refine the class-wise calibration procedure using a label rank calibration strategy augmented to the standard conformal score calibration, to enable adaptivity to various classes. Specifically, in contrast to CCP, RC3P selectively activates the class-wise thresholding subroutine in (1) according to their class-wise topk error ϵ k y for class y . RC3P produces the prediction set for a given test input X test with two calibration schemes (one for conformal score and another for label rank) as shown below:

where ̂ Q class 1 -̂ α y ( y ) and ̂ k ( y ) are score and label rank threshold for class y , respectively. In particular, ̂ k ( y ) controls the class-wise uncertainty adaptive to each class y based on the topk error ϵ ̂ k ( y ) y of the

Algorithm 1 RC3P Method for Class-Conditional CP

1: Input : Mis-coverage rate α ∈ (0 , 1) ., topk errors ϵ k y for all classes and ranks y, k ∈ { 1 , · · · , K }

2: Randomly split data into train D tr and calibration D cal and train the classifier f on D tr

3: for y ∈ { 1 , · · · , K } do

4: Compute { V i } n y i =1 for all ( X i , Y i ) ∈ D cal such that Y i = y

5: Configure calibrated label rank ̂ k ( y ) and nominal error ̂ α y :

6: Option I (model-agnostic coverage):

7: Option II (model-agnostic coverage + improved predictive efficiency):

8: ̂ Q class 1 -̂ α y ( y ) ←⌈ (1 -̂ α y )(1 + n y ) ⌉ -th smallest value in { V i } n y i =1 according to Eq (1)

9: end for

10: Construct ̂ C RC3P 1 -α ( X test ) with ̂ Q class 1 -̂ α y ( y ) and ̂ k ( y ) for a test input X test using Eq (3)

classifier. By determining ̂ k ( y ) , the top k predicted class labels of f ( X test ) will more likely cover the true label Y test, making the augmented label rank calibration filter out the class labels y that have a high rank (larger r f ( X,y ) ). As a result, given all test input and label pairs { ( X test , y ) } y ∈Y , RC3P performs score thresholding using class-wise quantiles only on a subset of reliable test pairs.

Determining ̂ k ( y ) and ̂ α y for model-agnostic valid coverage. For class y , intuitively, we would like a value for ̂ k ( y ) such that the corresponding top-̂ k ( y ) error is smaller than α , so that it is possible to guarantee valid coverage (recall P { A,B } = P { A } · P { B | A } ). Since a larger ̂ k ( y ) gives a smaller ϵ ̂ k ( y ) y untill ϵ K y = 0 , it is guaranteed to find a value for ̂ k ( y ) , in which the corresponding ϵ ̂ k ( y ) y < α . As a result, given all test input and label pairs { ( X test , y ) } y ∈Y , RC3P performs score thresholding using class-wise quantiles only on a subset of reliable test pairs and filters out the class labels y that have a high rank (larger r f ( X,y ) ). The following result formally shows the principle to configure ̂ k ( y ) and ̂ α y to guarantee the class-wise coverage that is agnostic to the underlying model.

Theorem 4.1. (Class-conditional coverage of RC3P) Suppose that selecting ̂ k ( y ) values result in the class-wise topk error ϵ ̂ k ( y ) y for each class y ∈ Y . For a target class-conditional coverage 1 -α , if we set ̂ α y and ̂ k ( y ) in RC3P (3) in the following ranges:

then RC3P can achieve the class-conditional coverage for every y ∈ Y :

4.2 Analysis of Predictive Efficiency for RC3P

We further analyze the predictive efficiency of RC3P: under what conditions RC3P can produce a smaller expected prediction set size compared to CCP, when both achieve the same ( 1 -α )-classconditional coverage. We investigate how to choose the value of ̂ α y and ̂ k ( y ) from the feasible ranges in (4) to achieve the best predictive efficiency using RC3P.

Lemma 4.2. (Trade-off condition for improved predictive efficiency of RC3P) Suppose ̂ α y and ̂ k ( y ) satisfy (4) in Theorem 4.1. If the following inequality holds for any y ∈ Y :

then RC3P produces smaller expected prediction sets than CCP, i.e.,

Remark. The above result demonstrates that when both RC3P and CCP achieve the target 1 -α class-conditional coverage, under the condition of (5), RC3P produces smaller prediction sets than CCP. In fact, this condition implies that the combined (conformity score and label rank) calibration

of RC3P tends to include less labels with high rank or low confidence from the classifier. In contrast, the CCP method tends to include relatively more uncertain labels into the prediction set, where their ranks are high and the confidence of the classifier is low. Now we can interpret the condition (5) by defining a condition number, termed as σ y :

In other words, if we can verify that σ y ≤ 1 for all y , then RC3P can improve the predictive efficiency over CCP. Furthermore, if σ y is fairly small, then the efficiency improvement can be even more significant. To verify this condition, our comprehensive experiments (Section 5.2, Figure 3) show that σ y values are much smaller than 1 on real-world data. These results demonstrate the practical utility of our theoretical analysis to produce small prediction sets using RC3P. Note that the reduction in prediction set size of RC3P over CCP is proportional to how small the σ y values are.

̸

̸

Theorem 4.3. (Conditions of improved predictive efficiency for RC3P) Define D = P [ r f ( X,y ) ≤ ̂ k ( y ) | Y = y ] , and ¯ r f ( X,y ) = ⌊ r f ( X,y )+1 2 ⌋ . Denote B = P [ f ( X ) (¯ r f ( X,y )) ≤ ̂ Q class 1 -α ( y ) | Y = y ] if V is APS, or B = P [ f ( X ) (¯ r f ( X,y )) + λ ≤ ̂ Q class 1 -α ( y ) | Y = y ] if V is RAPS. If B -D ≥ p y 1 -p y ( α -ϵ ̂ k ( y ) y ) , then σ y ≤ 1 .

̸

Remark. The above result further analyzes when the condition in Eq (5) of Lemma 4.2 (or equivalently, σ y ≤ 1 ) holds to guarantee the improved predictive efficiency. Specifically, the condition B -D ≥ p y 1 -p y ( α -ϵ ̂ k ( y ) y ) of Theorem 4.3 can be realized in two ways: (i) making LHS B -D as large as possible; (ii) making the RHS p y 1 -p y ( α -ϵ ̂ k ( y ) y ) as small as possible. To this end, we can set Line 7 in Algorithm 1 in the following way:

Therefore, this setting ensures σ y ≤ 1 and as a result improves predictive efficiency.

5 Experiments and Results

We present the empirical evaluation of the RC3P algorithm and demonstrate its effectiveness in achieving class-conditional coverage to produce small prediction sets. We conduct experiments using two baselines ( CCP and Cluster-CP ), four datasets (each with three imbalance types and five imbalance ratios), and two machine learning models (trained for 50 epochs and 200 epochs, with 200 epochs being our main experimental setting). Additionally, we use two scoring functions (APS and RAPS) and set three different α values ( α ∈ 0 . 1 , 0 . 05 , 0 . 01 , with α = 0 . 1 as our main setting).

5.1 Experimental Setup

Classification datasets. We consider four datasets: CIFAR-10, CIFAR-100 [34], mini-ImageNet [57], and Food-101 [8] by using the standard training and validation split. We employ the same methodology as [41, 10, 14] to create an imbalanced long-tail setting for each dataset as a harder challenge: 1) We use the original training split as a training set for training f with training samples ( n tr is defined as the number of training samples), and randomly split the original (balanced) validation set into calibration samples and testing samples. 2) We define an imbalance ratio ρ , the ratio between the sample size of the smallest and largest class: ρ = min i { # samples in class i } max i { # samples in class i } . 3) For each training set, we create three different imbalanced distributions using three decay types over the class indices c ∈ { 1 , · · · , K } : (a) An exponential-based decay (EXP) with n tr K × ρ c K examples in class c , (b) A polynomial-based decay (POLY) with n tr K × 1 √ c 10 ρ +1 examples in class c , and (c) A

majority-based decay (MAJ) with n tr K × ρ examples in classes c > 1 . We keep the calibration and test set balanced and unchanged. We provide an illustrative example of the three decay types in Appendix (Section C.3, Figure 4). Towards a more complete comparison, we also employ balanced datasets. Following Cluster-CP 2 , we employ CIFAR-100, Places365 [66], iNaturalist[55], and ImageNet[48].

Deep neural network models. We consider ResNet-20 [27] as the main architecture to train classifiers for imbalanced classification datasets. To handle imbalanced data, we employ the training algorithm 'LDAM' proposed by [10] that assigns different margins to classes, where larger margins are assigned to minority classes in the loss function. We follow the training strategy in [10] where all models are trained with 200 epochs. The class-wise performance with three imbalance types and imbalance ratios ρ = 0 . 5 and ρ = 0 . 1 on four datasets are evaluated (see Appendix C.1). We also train models with 50 epochs and the corresponding APSS results are reported in Appendix C.8.

For balanced datsets, we follow the same settings from Cluster-CP, which uses IMAGENET1K_V2 as pre-trained weights from PyTorch [44] and then fine-tune models with ResNet-50 for all datasets except ImageNet. For ImageNet, we use SimCLR-v2 [12] as training models.

CP baselines. We consider three CP methods: 1) CCP which estimates class-wise score thresholds and produces prediction set using Equation (1); 2) Cluster-CP [15] that performs calibration over clusters to reduce prediction set sizes; and 3) RC3P that produces prediction set using Equation (3). All CP methods are built on the same classifier and non-conformity scoring function for a fair comparison. We employ the three common scoring functions: APS [47], RAPS [1], and HPS [49]. We set α = 0 . 1 as our main experiment setting and also report other experiment results of different α

values (See Appendix C.7). Meanwhile, the hyper-parameters for each baseline are tuned according to their recommended ranges based on the same criterion (see Appendix C.2). We repeat experiments over 10 different random calibration-testing splits and report the mean and standard deviation.

Evaluation methodology. We use the target coverage 1 -α = 90% class-conditional coverage for CCP , Cluster-CP , and RC3P. We compute three evaluation metrics on the testing set:

Under Coverage Ratio (UCR).

Average Prediction Set Size (APSS).

Note that coverage and predictive efficiency are two competing metrics in CP [1], e.g., achieving better coverage (resp. predictive efficiency) degenerates predictive efficiency (resp. coverage). Therefore, following the same strategy in [20], we choose to control their UCR as the same level that is close to 0 for a fair comparison over three class-conditional CP algorithms in terms of APSS. Meanwhile, to address the gap between population values and empirical ones (e.g., quantiles with ˜ O (1 / √ n y ) error bound, common to all CP methods [58, 22, 2], or class-wise topk error ϵ k y with ˜ O (1 / √ n y ) error bound [43]), we uniformly add g/ √ n y (the same order with the standard concentration gap) to inflate the nominal coverage 1 -α on each baseline and tune g ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 1 } on the calibration dataset in terms of UCR. The detailed g values of each method are displayed in Appendix C.2. In addition, the actual achieved UCR values are shown in the complete results (see Appendix C.4, C.5, and C.6). For a complete evaluation, we add the experiments without controlling coverage on imbalanced datasets under the same setting and use the total under coverage gap (UCG) metric:

Under Coverage Gap (UCG).

Experiments with UCG metric evaluation are shown in the Appendix C.9.

5.2 Results and Discussion

We list empirical results in Table 1 for an overall comparison on four imbalanced datasets with ρ = 0 . 5 , 0 . 1 using all three training distributions (EXP, POLYand MAJ) based on the considered APS, RAPS and HPS scoring functions. Complete experiment results under more values of ρ are in Appendix C). Results with APS, RAPS, and HPS scoring functions on balanced datasets are also summarized in Table 2. We make the following two key observations: (i) CCP , Cluster-CP , and RC3P can guarantee the class-conditional coverage (their UCRs are all close to 0 ) for all settings; (ii) RC3P significantly outperforms CCP and Cluster-CP in APSS on almost all imbalanced settings by reducing APSS with 24 . 47% on all four datasets and 32 . 63% on three datasets excluding CIFAR10 compared with min { CCP , Cluster-CP } on average, while for balanced settings, RC3P still significantly outperforms the best baselines in terms of APSS with 32 . 826% APSS reduction.

To investigate the challenge of imbalanced data and more importantly, how RC3P significantly improves the APSS, we further conduct three careful experiments on imbalanced datasets. First, we report the histograms of class-conditional coverage and the corresponding histograms of prediction set size. This experiment verifies that RC3P derives significantly more class-conditional coverage above 1 -α and thus reduces the prediction set size. Second, we visualize the normalized frequency of label rank included in prediction sets on testing datasets for all class-wise algorithms: CCP , Cluster-CP ,

✶ we empirically verify the trade-off condition number { σ y } K y =1 of Equation 6 on calibration dataset to reveal the underlying reason for RC3P producing smaller prediction sets over CCP with our standard training models (epoch = 200 ). We also evaluate { σ y } K y =1 with less trained models (epoch = 50 ) on imbalanced datasets in Appendix C.10. Additionally, we also repeat all three experiments on balanced datasets (i.e., the histograms of class-conditional coverage and prediction set size, the normalized frequency of label rank included in prediction sets, and { σ y } K y =1 ) in Appendix C.11. Below we discuss our experimental results and findings in detail.

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

0.90

0.92

0.95

0.97

0.99

Class-wise Coverage

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

0.86

0.90

0.93

0.96

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

50.0

Frequency

Method

CCP

0.87

0.90

0.93

0.97

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Frequency

Method

CCP

0.88

0.91

0.94

0.97

1.00

Class-wise Coverage

0.0

5.0

10.0

15.0

20.0

25.0

Frequency

Method

CCP

1.32

1.59

1.86

2.14

2.41

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

19.34 27.89 36.45 45.00 53.56

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

160.0

Frequency

Method

CCP

14.13 17.86 21.60 25.34 29.08

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Frequency

Method

CCP

19.55 23.60 27.64 31.68 35.73

0.0

10.0

20.0

30.0

40.0

Frequency

Method

CCP

RC3P significantly outperforms CCP and Cluster-CP . First, it is clear from Table 6, 8, and 7, and 2 that RC3P , CCP , and Cluster-CP guarantee class-conditional coverage on all settings. This can also be observed by the first row of Fig 1, where the class-wise coverage bars of CCP and RC3P distribute on the right-hand side of the target probability 1 -α (red dashed line). Second, RC3P outperforms CCP and Cluster-CP with 24 . 47% (four datasets) or 32 . 63% (excluding CIFAR-10) on imbalanced datasets and 32 . 63% on balanced datasets decrease in terms of average prediction set size for the same class-wise coverage. We also report the histograms of the corresponding prediction set sizes in the second row of Figure 1, which shows (i) RC3P has more concentrated class-wise coverage distribution than CCP and Cluster-CP ; (ii) the distribution of prediction set sizes produced by RC3P is globally smaller than that produced by CCP and Cluster-CP , which

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1

2

3

4

5

Rank

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Normalized Frequency

Method

CCP

1

24

48

72

95

Rank

0.00

0.01

0.02

0.03

0.04

0.05

Normalized Frequency

Method

CCP

1

17

34

50

66

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Normalized Frequency

Method

CCP

1

26

50

75

100

Rank

0.00

0.01

0.02

0.03

0.04

Normalized Frequency

Method

CCP

is justified by a better trade-off number of { σ y } K y =1 as shown in Figure 3. Note that the class-wise coverage and the corresponding prediction set sizes RC3P overlap with CCP on CIFAR-10 in Figure 1.

Visualization of normalized frequency. Figure 2 illustrates the normalized frequency distribution of label ranks included in the prediction sets across various testing datasets. It is evident that the distribution of label ranks in the prediction set generated by RC3P tends to be lower compared to those produced by CCP and Cluster-CP . Furthermore, the probability density function tail for label ranks in the RC3P prediction set is notably shorter than that of other methods. This indicates that RC3P more effectively incorporates lower-ranked labels into prediction sets, as a result of its augmented rank calibration scheme.

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.06

0.30

0.53

0.77

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.14

0.35

0.57

0.78

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.20

0.40

0.60

0.80

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

Verification of σ y . Figure 3 verifies the validity of Equation (6) on testing datasets and confirms the optimized trade-off between the coverage with inflated quantile and the constraint with calibrated label rank leads to smaller prediction sets. It also confirms that the condition number { σ y } K y =1 could be evaluated on calibration datasets without testing datasets and thus decreases the overall computation cost. We verify that σ y ≤ 1 for all settings and σ y is much smaller than 1 on all datasets with large number of classes.

6 Summary

This paper studies a provable conformal prediction (CP) algorithm that aims to provide classconditional coverage guarantee and to produce small prediction sets for classification tasks with many and/or imbalanced classes. Our proposed RC3P algorithm performs double-calibration, one over conformity score and one over label rank for each class separately, to achieve this goal. Our experiments clearly demonstrate the significant efficacy of RC3P over the baseline class-conditional CP algorithms on both balanced and imbalanced classification data settings.

Acknowledgments. This research was supported in part by United States Department of Agriculture (USDA) NIFA award No. 2021-67021-35344 (AgAID AI Institute) and by NSF CNS-2312125 grant.

References

Anastasios Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I Jordan. Uncertainty sets for image classifiers using conformal prediction. arXiv preprint arXiv:2009.14193 , 2020.

Anastasios N Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. arXiv preprint arXiv:2107.07511 , 2021.

Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. arXiv preprint arXiv:2208.02814 , 2022.

Varun Babbar, Umang Bhatt, and Adrian Weller. On the utility of prediction sets in human-ai teams. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence (IJCAI-22) , 2022.

Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. Conformal prediction beyond exchangeability. The Annals of Statistics , 51(2):816-845, 2023.

Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM) , 68(6):1-34, 2021.

Aadyot Bhatnagar, Huan Wang, Caiming Xiong, and Yu Bai. Improved online conformal prediction via strongly adaptive online learning. In International Conference on Machine Learning , pages 2337-2363. PMLR, 2023.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative components with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13 , pages 446-461. Springer, 2014.

Henrik Boström, Ulf Johansson, and Tuwe Löfström. Mondrian conformal predictive distributions. In Conformal and Probabilistic Prediction and Applications , pages 24-38. PMLR, 2021.

Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution-aware margin loss. Advances in neural information processing systems , 32, 2019.

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research , 16:321-357, 2002.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E Hinton. Big selfsupervised models are strong semi-supervised learners. Advances in neural information processing systems , 33:22243-22255, 2020.

Kfir M Cohen, Sangwoo Park, Osvaldo Simeone, and Shlomo Shamai. Cross-validation conformal risk control. arXiv preprint arXiv:2401.11974 , 2024.

Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9268-9277, 2019.

Tiffany Ding, Anastasios Angelopoulos, Stephen Bates, Michael Jordan, and Ryan J Tibshirani. Classconditional conformal prediction with many classes. Advances in Neural Information Processing Systems , 36, 2024.

Robin Dunn, Larry Wasserman, and Aaditya Ramdas. Distribution-free prediction sets with random effects. arXiv preprint arXiv:1809.07441 , 2018.

Shai Feldman, Stephen Bates, and Yaniv Romano. Calibrated multiple-output quantile regression with representation learning. Journal of Machine Learning Research , 24(24):1-48, 2023.

Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Efficient conformal prediction via cascaded inference with expanded admission. arXiv preprint arXiv:2007.03114 , 2020.

Adam Fisch, Tal Schuster, Tommi Jaakkola, and Regina Barzilay. Few-shot conformal prediction with auxiliary tasks. In International Conference on Machine Learning , pages 3329-3339. PMLR, 2021.

Matteo Fontana, Gianluca Zeni, and Simone Vantini. Conformal prediction: a unified review of theory and new challenges. Bernoulli , 29(1):1-23, 2023.

Subhankar Ghosh, Taha Belkhouja, Yan Yan, and Janardhan Rao Doppa. Improving uncertainty quantification of deep classifiers via neighborhood conformal prediction: Novel algorithm and theoretical analysis. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 7722-7730, 2023.

Subhankar Ghosh, Yuanjie Shi, Taha Belkhouja, Yan Yan, Jana Doppa, and Brian Jones. Probabilistically robust conformal prediction. In Uncertainty in Artificial Intelligence , pages 681-690. PMLR, 2023.

Isaac Gibbs and Emmanuel Candes. Adaptive conformal inference under distribution shift. Advances in Neural Information Processing Systems , 34:1660-1672, 2021.

Isaac Gibbs and Emmanuel J Candès. Conformal inference for online prediction with arbitrary distribution shifts. Journal of Machine Learning Research , 25(162):1-36, 2024.

Lee-Ad Gottlieb, Eran Kaufman, and Aryeh Kontorovich. Apportioned margin approach for cost sensitive large margin classifiers. Annals of Mathematics and Artificial Intelligence , 89(12):1215-1235, 2021.

Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika , 110(1):33-50, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelligence , 42(11):2781-2794, 2019.

Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep classifier via label ranking, 2024.

Kexin Huang, Ying Jin, Emmanuel Candès, and Jure Leskovec. Uncertainty quantification over graph with conformalized graph neural networks. In NeurIPS 2023 .

Ying Jin and Emmanuel J. Candès. Model-free selective inference under covariate shift via weighted conformal p-values.

Lisa Jöckel, Michael Kläs, Janek Groß, and Pascal Gerber. Conformal prediction and uncertainty wrapper: What statistical guarantees can you get for uncertainty quantification in machine learning? In International Conference on Computer Safety, Reliability, and Security , pages 314-327. Springer, 2023.

Bartosz Krawczyk, Michał Koziarski, and Michał Wo´ zniak. Radial-based oversampling for multiclass imbalanced data classification. IEEE transactions on neural networks and learning systems , 31(8):28182831, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J Tibshirani, and Larry Wasserman. Distribution-free predictive inference for regression. Journal of the American Statistical Association , 113(523):1094-1111, 2018.

Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society Series B: Statistical Methodology , 76(1):71-96, 2014.

Lihua Lei and Emmanuel J Candès. Conformal inference of counterfactuals and individual treatment effects. Journal of the Royal Statistical Society Series B: Statistical Methodology , 83(5):911-938, 2021.

Charles Lu, Anastasios N Angelopoulos, and Stuart Pomerantz. Improving trustworthiness of ai disease severity rating in medical imaging with ordinal conformal prediction sets. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 545-554. Springer, 2022.

Charles Lu, Andréanne Lemay, Ken Chang, Katharina Höbel, and Jayashree Kalpathy-Cramer. Fair conformal predictors for applications in medical imaging. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 12008-12016, 2022.

Harish Tayyar Madabushi, Elena Kochkina, and Michael Castelle. Cost-sensitive bert for generalisable sentence classification with imbalanced data. arXiv preprint arXiv:2003.11563 , 2020.

Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 , 2020.

Roweida Mohammed, Jumanah Rawashdeh, and Malak Abdullah. Machine learning with oversampling and undersampling techniques: overview study and experimental results. In 2020 11th international conference on information and communication systems (ICICS) , pages 243-248. IEEE, 2020.

Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of machine learning . MIT press, 2018.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review , 2(2):4, 2020.

Yaniv Romano, Evan Patterson, and Emmanuel Candes. Conformalized quantile regression. Advances in neural information processing systems , 32, 2019.

Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adaptive coverage. Advances in Neural Information Processing Systems , 33:3581-3591, 2020.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115:211-252, 2015.

Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association , 114(525):223-234, 2019.

Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9(3), 2008.

Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with conformal prediction. In International Conference on Machine Learning (ICML) , 2023.

Jiankai Sun, Yiqi Jiang, Jianing Qiu, Parth Nobel, Mykel J Kochenderfer, and Mac Schwager. Conformal prediction for uncertainty-aware planning with diffusion dynamics model. Advances in Neural Information Processing Systems , 36, 2024.

Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. Advances in neural information processing systems , 32, 2019.

Chih-Fong Tsai, Wei-Chao Lin, Ya-Han Hu, and Guan-Ting Yao. Under-sampling class imbalanced datasets by combining clustering analysis and instance selection. Information Sciences , 477:47-54, 2019.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8769-8778, 2018.

Janette Vazquez and Julio C Facelli. Conformal prediction in clinical medical sciences. Journal of Healthcare Informatics Research , 6(3):241-252, 2022.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian conference on machine learning , pages 475-490. PMLR, 2012.

Vladimir Vovk, Valentina Fedorova, Ilia Nouretdinov, and Alexander Gammerman. Criteria of efficiency for conformal prediction. In Conformal and Probabilistic Prediction with Applications: 5th International Symposium, COPA 2016, Madrid, Spain, April 20-22, 2016, Proceedings 5 , pages 23-39. Springer, 2016.

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world . Springer Science & Business Media, 2005.

Vladimir Vovk, David Lindsay, Ilia Nouretdinov, and Alex Gammerman. Mondrian confidence machine. Technical Report , 2003.

Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. 1999.

Pattaramon Vuttipittayamongkol and Eyad Elyan. Neighbourhood-based undersampling approach for handling imbalanced and overlapped data. Information Sciences , 509:47-70, 2020.

Yunpeng Xu, Wenge Guo, and Zhi Wei. Conformal risk control for ordinal classification. In Uncertainty in Artificial Intelligence , pages 2346-2355. PMLR, 2023.

Margaux Zaffran, Aymeric Dieuleveut, Julie Josse, and Yaniv Romano. Conformal prediction with missing values. In International Conference on Machine Learning , pages 40578-40604. PMLR, 2023.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence , 40(6):1452-1464, 2017.

A Mathematical Notations

̂

B Technical Proofs of Theoretical Results

B.1 Proof of Theorem 4.1

Theorem B.1. (Theorem 4.1 restated, class-conditional coverage of RC3P) Suppose that selecting ̂ k ( y ) values result in the class-wise topk error ϵ ̂ k ( y ) y for each class y ∈ Y . For a target classconditional coverage 1 -α , if we set ̂ α y and ̂ k ( y ) in RC3P (3) in the following ranges:

then RC3P can achieve the class-conditional coverage for every y ∈ Y :

Proof. (of Theorem 4.1)

Let y ∈ Y denote any class label. In this proof, we omit the superscript k in the topk error notation ϵ k y for simplicity.

With the lower bound of the coverage on class y (Theorem 1 in [47]), we have

Re-arranging the above inequality, we have

where the last inequality is due to ̂ α y ≤ α -ϵ ̂ k ( y ) y . This implies that RC3P guarantees the classconditional coverage on any class y . This completes the proof for Theorem 4.1.

B.2 Proof of Lemma 4.2

Theorem B.2. (Lemma 4.2 restated, improved predictive efficiency of RC3P) Let ̂ α y and ̂ k ( y ) satisfy Theorem 4.1. If the following inequality holds for any y ∈ Y :

̂ then RC3P produces smaller expected prediction sets than CCP, i.e.,

The proof idea is to reduce the cardinality of the prediction set made by RC3P to that made by CCP in expectation. Let σ y = P X test [ V ( X test ,y ) ≤ ̂ Q class 1 -̂ α ( y ) , r f ( X test ,y ) ≤ ̂ k ( y ) ] P X test [ V ( X test ,y ) ≤ ̂ Q class 1 -α ( y ) ] . According to the assumption in

(9), we know that σ y ≤ 1 , which will be used later.

We start with the expected prediction set size of RC3Pand then derive its upper bound.

where the equality ( a ) is due to the definitions of σ y , and inequality ( b ) is due to the assumption

This shows that RC3P requires smaller prediction sets to guarantee the class-conditional coverage compared to CCP.

B.3 Proof of Theorem 4.3

̸

Theorem B.3. (Theorem 4.3 restated, conditions of improved predictive efficiency for RC3P) Define D = P [ r f ( X,y ) ≤ ̂ k ( y ) | Y = y ] , and ¯ r f ( X,y ) = ⌊ r f ( X,y )+1 2 ⌋ . Denote B = P [ f ( X ) (¯ r f ( X,y )) ≤ ̂ Q class 1 -α ( y ) | Y = y ] if V is APS, or B = P [ f ( X ) (¯ r f ( X,y )) + λ ≤ ̂ Q class 1 -α ( y ) | Y = y ] if V is RAPS. If B -D ≥ p y 1 -p y ( α -ϵ ̂ k ( y ) y ) , then σ y ≤ 1 .

̸

Proof. (of Theorem 4.3)

Based on the different choices of scoring function, we first divide two scenarios:

(i): If V ( X,y ) is the APS scoring function, since the APS score cumulatively sums the ordered prediction of f ( X ) : V ( X,y ) = ∑ r f ( X,y ) l =1 f ( X ) ( l ) , it is easy to verify that V ( X,y ) is concave in

̸

terms of l . As a result, we have

Now we lower bound P X [ V ( X,y ) ≤ ̂ Q class 1 -α ( y )] as follows.

̸

According to the assumption B -D ≥ p y 1 -p y ( α -ϵ ̂ k ( y ) y ) , we have

(ii): If V ( X,y ) is the RAPS scoring function and r f ( X,y ) ≤ k reg , then the RAPS scoring function could be rewritten as: V ( X,y ) = ∑ r f ( X,y ) l =1 f ( X ) ( l ) . As a result, we have:

If r f ( X,y ) > k reg , then the RAPS scoring function could be rewritten as: V ( X,y ) = ∑ r f ( X,y ) l =1 f ( X ) ( l ) + λ ( r f ( X,y ) -k reg ) . As a result, we have

Then, by applying the Inequality 12, we have:

This completes the proof for Theorem 4.3.

C Complete Experimental Results

C.1 Training Details

For CIFAR-10 and CIFAR-100, we train ResNet20 using LDAM loss function given in [10] with standard mini-batch stochastic gradient descent (SGD) using learning rate 0 . 1 , momentum 0 . 9 , and weight decay 2 e -4 for 200 epochs and 50 epochs. The batch size is 128 . For experiments on

̸

mini-ImageNet, we use the same setting. For Food-101, the batch size is 256 and other parameters are kept the same. We reported our main results when models were trained in 200 epochs. Other results are reported in Appendix C.8 and Table 11.

We also evaluate the top-1 accuracy over the majority, medium, and minority groups of classes as the class-wise performance when 200 epochs. To show the variation of class-wise performance, we divide some classes with the largest number of data samples into the majority group, and the number of these classes is a quarter ( 25% ) of the total number of classes. Similarly, we divide the classes with the smallest number of data into the minority group ( 25% ) and the remaining classes as the medium group ( 50% ). In the above table, we show the accuracy of three groups with three imbalance types and two imbalance ratios ρ = 0 . 1 , ρ = 0 . 5 on four datasets.

The results are summarized in Table 4. As can be seen, the group-wise performance can vary significantly from high to very low. The class-imbalance setting is the case where the classifier does not perform very well in some classes.

C.2 Calibration Details

As mentioned in Section 5.1, we balanced split the validation set of CIFAR-10 and CIFAR-100, the number of calibration data is 5000 . For mini-ImageNet, the number of calibration data is 15000 . For Food-101, the total number is 12625 . To compute the mean and standard deviation for the overall performance, we repeat calibration experiments for 10 times. In our main results, We set α = 0 . 1 . We also report other experiment results of different α values, α = 0 . 05 and α = 0 . 01 , in Appendix C.7, and Table 9 and 10.

The regularization parameter for RAPS scoring function is from the set k reg ∈ { 3 , 5 , 7 } and λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } based on the empirical setting in cluster-CP . We select the combination of k reg and λ for each experiment with the same imbalanced type and imbalanced ratio on the same dataset, where most of the APSS values of all methods are minimum.

The hyper-parameter g is selected from the set { 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } to find the minimal g that CCP , Cluster-CP 3 , and RC3P achieve the target class-conditional coverage. We clarify that for each dataset and each class-conditional CP method, we use fixed g values. The detailed g values

are displayed in Table 5. From Table 5, we could observe that the hyperparameter g for RC3P is always smaller than other methods, which means that comparing other class-wise CP algorithms, our algorithm needs the smallest inflation on 1 -̂ α to achieve the target class-conditional coverage. This could also match the result of histograms of class-conditional coverage.

C.3 Illustration of Imbalanced Data

(a) EXP

(b) POLY

(c) MAJ

Class Index

0

100

200

300

400

500

Number of examples

Class Index

Class Index

C.4 Comparison Experiments Using APS Score Function

Based on the results in Table 6, we make the following observations: (i) CCP , Cluster-CP , and RC3P can guarantee the class-conditional coverage; and (ii) RC3P significantly outperforms CCP and Cluster-CCP on three datasets by producing smaller prediction sets.

C.5 Comparison Experiments Using RAPS Score Function

With the same model, evaluation metrics, and RAPS score function [1], we add the comparison experiments with CCP , and Cluster-CP on four datasets with different imbalanced types and imbalance ratio ρ = 0 . 5 and ρ = 0 . 1 . The regularization parameter for RAPS scoring function is from the set k reg ∈ { 3 , 5 , 7 } and λ ∈ { 0 . 001 , 0 . 01 , 0 . 1 } . We select the combination of k reg and λ for each experiment with the same imbalanced type and imbalanced ratio on the same dataset, where most of the APSS values of all methods are minimum. The overall performance is summarized in Table 7. We highlight that we also select the g from the set g ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } to find the minimal g that CCP, Cluster-CP, and RC3P approximately achieves the target class conditional coverage.

Based on the results in Table 7, we make the following observations: (i) CCP , Cluster-CP , and RC3P can guarantee the class-conditional coverage; and (ii) RC3P significantly outperforms CCP and Cluster-CP on three datasets by producing smaller prediction sets.

C.6 Comparison Experiments Using HPS Score Function

With the same model, evaluation metrics, and HPS score function [1], we add the comparison experiments with CCP , and Cluster-CP on four datasets with different imbalanced types and imbalance ratio ρ = 0 . 5 and ρ = 0 . 1 . The overall performance is summarized in Table 8. We highlight that we also select the g from the set g ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } to find the minimal g that CCP, Cluster-CP, and RC3P approximately achieves the target class conditional coverage.

Based on the results in Table 8, we make the following observations: (i) CCP , Cluster-CP , and RC3P can guarantee the class-conditional coverage; and (ii) RC3P significantly outperforms CCP and Cluster-CP on three datasets by producing smaller prediction sets.

C.7 Comparison Experiments with different α values

With the same model, evaluation metrics, and scoring functions, we add the comparison experiments with CCP , and Cluster-CP on four datasets with different imbalanced types and imbalance ratio ρ = 0 . 5 and ρ = 0 . 1 under the different α values. The overall performance is summarized in Table 9 and 10, with α = 0 . 05 and α = 0 . 01 , respectively. We highlight that we also select the g from the set g ∈ [0 . 15 , 0 . 75] with 0 . 05 range to find the minimal g that CCP, Cluster-CP, and RC3P approximately achieves the target class conditional coverage.

Based on the results in Table 7, we make the following observations: (i) CCP , Cluster-CP , and RC3P can guarantee the class-conditional coverage; and (ii) RC3P significantly outperforms CCP and Cluster-CP on three datasets by producing smaller prediction sets.

C.8 Comparison Experiments when models are trained in different epochs

With the same loss function, training criteria, evaluation metrics, and two scoring functions, we add the comparison experiments with CCP , and Cluster-CP on four datasets with different imbalanced types and imbalance ratio ρ = 0 . 5 and ρ = 0 . 1 and α = 0 . 1 when models are trained with 50 epochs. The overall performance is summarized in Table 11. We highlight that we also select the g from the set g ∈ { 0 . 25 , 0 . 5 , 0 . 75 , 1 . 0 } to find the minimal g that CCP, Cluster-CP, and RC3P approximately achieves the target class conditional coverage.

Based on the results in Table 7, we make the following observations: (i) CCP , Cluster-CP , and RC3P can guarantee the class-conditional coverage; and (ii) RC3P significantly outperforms CCP and Cluster-CP on three datasets by producing smaller prediction sets.

Table 11: APSS results comparing CCP , Cluster-CP , and RC3P with ResNet-20 model under different imbalance ratio ρ = 0 . 5 and ρ = 0 . 1 where α = 0 . 1 and models are trained with 50 epochs. For a fair comparison of prediction set size, we set UCR of RC3P the same as or smaller (more restrictive) than that of CCP and Cluster-CP under 0 . 16 on CIFAR-10 and 0 . 03 on other datasets. The APSS results show that RC3P significantly outperforms CCP and Cluster-CP in terms of average prediction set size with 21 . 441% (four datasets) or 28 . 588% (excluding CIFAR-10) reduction in prediction size on average over min { CCP , cluster-CP } .

C.9 Comparison Experiments with UCG metrics

We add the experiments without controlling coverage on imbalanced datasets under the same setting as the main paper. We then use the total under coverage gap (UCG, ↓ better) between class conditional coverage and target coverage 1 -α of all under covered classes. We choose UCG as the fine-grained metric to differentiate the coverage performance in our experiment setting. Conditioned on similar APSS of all methods, RC3P significantly outperforms the best baselines with 35 . 18% (four datasets) or 46 . 91% (excluding CIFAR-10)reduction in UCG on average.

C.10 Complete Experiment Results on Imbalanced Datasets

In this subsection, we report complete experimental results over four imbalanced datasets, three decaying types, and five imbalance ratios when epoch = 200 and α = 0 . 1 . Specifically, Table 13, 14, 15 report results on CIFAR-10 with three decaying types. Table 16, 17, 18 report results on CIFAR-100 with three decaying types. Table 19, 20, 21 report results on mini-ImageNet with three decaying types. Table 22, 23, 24 report results on Food-101 with three decaying types.

Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9 show the class-conditional coverage and the corresponding prediction set sizes on EXP ρ = 0 . 5 , POLY ρ = 0 . 1 , POLY ρ = 0 . 5 , MAJ ρ = 0 . 1 , MAJ ρ = 0 . 5 , respectively. This result on EXP ρ = 0 . 1 is in Figure 1.

Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14 illustrates the normalized frequency distribution of label ranks included in the prediction sets on EXP ρ = 0 . 5 , POLY ρ = 0 . 1 , POLY ρ = 0 . 5 , MAJ ρ = 0 . 1 , MAJ ρ = 0 . 5 , respectively. This result on EXP ρ = 0 . 1 is in Figure 2. It is evident that the distribution of label ranks in the prediction set generated by RC3P tends to be lower compared to those produced by CCP and Cluster-CP . Furthermore, the probability density function tail for label ranks in the RC3P prediction set is notably shorter than that of other methods. This indicates that RC3P more effectively incorporates lower-ranked labels into prediction sets, as a result of its augmented rank calibration scheme.

Figure 15, Figure 16, Figure 17, Figure 18 and Figure 19 verify the condition numbers σ y when models are fully trained (epoch = 200 ) on EXP ρ = 0 . 5 , POLY ρ = 0 . 1 , POLY ρ = 0 . 5 , MAJ ρ = 0 . 1 , MAJ ρ = 0 . 5 , respectively. This result on EXP ρ = 0 . 1 is in Figure Figure 3. We also evaluate the condition numbers σ y when models are lessly trained (epoch = 50 ) and α = 0 . 1 on EXP ρ = 0 . 5 , EXP ρ = 0 . 1 , POLY ρ = 0 . 1 , POLY ρ = 0 . 5 , MAJ ρ = 0 . 1 , MAJ ρ = 0 . 5 , respectively. These results are shown from Figure 21 to Figure 25. These results verify the validity of Lemma 4.2 and Equation 6 and confirm that the optimized trade-off between the coverage with inflated quantile and the constraint with calibrated rank leads to smaller prediction sets. They also show a stronger condition ( σ y ≤ 1 for all y ) than the weighted aggregation condition in (5). They also confirm that the condition number { σ y } C y =1 could be evaluated on calibration datasets without testing datasets and thus decreases the computation cost. We notice that RC3P degenerates to CCP on CIFAR-10, so σ y = 1 for all y and there is no trade-off. On the other three datasets, we observe significant conditions for the optimized trade-off in RC3P .

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

0.92

0.93

0.95

0.96

0.98

Class-wise Coverage

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Frequency

Method

CCP

0.84

0.88

0.92

0.96

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

50.0

Frequency

Method

CCP

0.89

0.92

0.94

0.97

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Method

CCP

0.90

0.92

0.95

0.97

1.00

Class-wise Coverage

0.0

5.0

10.0

15.0

20.0

25.0

Frequency

Method

CCP

1.09

1.30

1.51

1.73

1.94

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

14.12 22.62 31.13 39.63 48.14

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

160.0

Frequency

Method

CCP

14.41 18.08 21.75 25.41 29.08

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Frequency

Method

CCP

15.43 19.01 22.60 26.18 29.76

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

0.88

0.91

0.93

0.96

0.99

Class-wise Coverage

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

0.84

0.88

0.92

0.96

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

50.0

Frequency

Method

CCP

0.86

0.90

0.93

0.96

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Method

CCP

0.90

0.92

0.95

0.97

1.00

Class-wise Coverage

0.0

5.0

10.0

15.0

20.0

Frequency

Method

CCP

1.37

1.55

1.73

1.91

2.09

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

32.10 40.84 49.57 58.31 67.04

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

Frequency

Method

CCP

14.93 18.32 21.72 25.11 28.51

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Frequency

Method

CCP

23.44 27.53 31.62 35.72 39.81

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

0.88

0.91

0.93

0.96

0.98

Class-wise Coverage

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

0.78

0.83

0.89

0.95

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

Frequency

Method

CCP

0.91

0.93

0.95

0.98

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

Frequency

Method

CCP

0.87

0.90

0.94

0.97

1.00

Class-wise Coverage

0.0

5.0

10.0

15.0

20.0

25.0

Frequency

Method

CCP

1.00

1.23

1.45

1.68

1.90

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Frequency

Method

CCP

20.30 28.53 36.76 44.99 53.22

0.0

20.0

40.0

60.0

80.0

100.0

120.0

140.0

Frequency

Method

CCP

14.34 18.15 21.96 25.77 29.58

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Frequency

Method

CCP

19.32 22.80 26.28 29.77 33.25

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

0.88

0.91

0.94

0.97

1.00

Class-wise Coverage

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Frequency

Method

CCP

0.78

0.83

0.89

0.95

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

50.0

Frequency

Method

CCP

0.89

0.92

0.95

0.97

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Method

CCP

0.89

0.92

0.94

0.97

1.00

Class-wise Coverage

0.0

5.0

10.0

15.0

20.0

Frequency

Method

CCP

1.91

2.27

2.64

3.01

3.37

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Frequency

Method

CCP

30.70 40.10 49.51 58.91 68.32

0.0

20.0

40.0

60.0

80.0

100.0

Frequency

Method

CCP

14.18 17.58 20.99 24.39 27.79

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Frequency

Method

CCP

24.77 28.72 32.68 36.63 40.58

0.0

10.0

20.0

30.0

40.0

Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

0.92

0.93

0.95

0.97

0.99

Class-wise Coverage

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Frequency

Method

CCP

0.88

0.91

0.94

0.97

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

50.0

Frequency

Method

CCP

0.88

0.91

0.94

0.97

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Method

CCP

0.88

0.91

0.94

0.97

1.00

Class-wise Coverage

0.0

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

Frequency

Method

CCP

1.36

1.57

1.78

1.98

2.19

0.0

1.0

2.0

3.0

4.0

5.0

Frequency

Method

CCP

16.08 25.07 34.06 43.05 52.04

0.0

25.0

50.0

75.0

100.0

125.0

150.0

175.0

200.0

Frequency

Method

CCP

15.08 18.76 22.44 26.12 29.81

0.0

5.0

10.0

15.0

20.0

25.0

30.0

Frequency

Method

CCP

16.42 20.27 24.13 27.99 31.85

0.0

10.0

20.0

30.0

40.0

Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1

2

3

Rank

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Normalized Frequency

Method

CCP

1

25

49

73

97

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Normalized Frequency

Method

CCP

1

16

32

47

62

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Normalized Frequency

Method

CCP

1

25

48

72

96

Rank

0.00

0.01

0.02

0.03

0.04

0.05

Normalized Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1

2

3

Rank

0.00

0.50

1.00

1.50

2.00

2.50

Normalized Frequency

Method

CCP

1

25

49

73

97

Rank

0.00

0.01

0.01

0.01

0.02

0.03

0.03

Normalized Frequency

Method

CCP

1

20

38

56

75

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Normalized Frequency

Method

CCP

1

20

40

60

79

Rank

0.00

0.01

0.02

0.03

0.04

0.05

Normalized Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1

2

3

Rank

0.00

0.50

1.00

1.50

2.00

2.50

3.00

Normalized Frequency

Method

CCP

1

25

50

74

98

Rank

0.00

0.01

0.02

0.03

0.04

Normalized Frequency

Method

CCP

1

18

36

53

70

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Normalized Frequency

Method

CCP

1

21

41

61

81

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Normalized Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1

2

3

4

5

Rank

0.00

0.20

0.40

0.60

0.80

1.00

Normalized Frequency

Method

CCP

1

26

50

74

99

Rank

0.00

0.01

0.01

0.01

0.02

0.03

0.03

Normalized Frequency

Method

CCP

1

18

34

51

68

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Normalized Frequency

Method

CCP

1

22

43

64

85

Rank

0.00

0.01

0.02

0.03

0.04

Normalized Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1

2

3

4

Rank

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

2.00

Normalized Frequency

Method

CCP

1

24

48

72

95

Rank

0.00

0.01

0.02

0.03

0.04

0.05

Normalized Frequency

Method

CCP

1

20

38

57

76

Rank

0.00

0.01

0.02

0.03

0.04

0.05

Normalized Frequency

Method

CCP

1

21

41

61

81

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

Normalized Frequency

Method

CCP

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.04

0.28

0.52

0.76

1.00

Sigma_y

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Frequency

1.0

0.08

0.31

0.54

0.77

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.24

0.43

0.62

0.81

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.06

0.30

0.53

0.77

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

0.17

0.38

0.59

0.79

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.28

0.46

0.64

0.82

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.05

0.29

0.53

0.76

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.25

0.44

0.63

0.81

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.20

0.40

0.60

0.80

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.03

0.27

0.51

0.76

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.19

0.39

0.59

0.80

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.25

0.44

0.63

0.81

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.02

0.27

0.51

0.76

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.18

0.39

0.59

0.80

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

0.19

0.39

0.60

0.80

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.07

0.30

0.53

0.77

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.23

0.42

0.62

0.81

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

0.38

0.53

0.69

0.84

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.06

0.29

0.53

0.76

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.31

0.48

0.65

0.83

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.29

0.47

0.65

0.82

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.04

0.28

0.52

0.76

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.30

0.48

0.65

0.83

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.41

0.56

0.71

0.85

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.13

0.35

0.56

0.78

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.23

0.42

0.62

0.81

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.39

0.54

0.69

0.85

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.03

0.27

0.51

0.76

1.00

Sigma_y

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

Frequency

1.0

0.38

0.53

0.69

0.84

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.37

0.53

0.68

0.84

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

(a) CIFAR-10

(b) CIFAR-100

(c) mini-ImageNet

(d) Food-101

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.08

0.31

0.54

0.77

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

Frequency

1.0

0.20

0.40

0.60

0.80

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

Frequency

1.0

0.39

0.54

0.70

0.85

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

Frequency

1.0

C.11 Complete Experiment Results on Balanced Classification Datasets

In this subsection, we report complete experimental results over four balanced datasets and α = 0 . 1 . Specifically, Figure 26 shows the class-conditional coverage and the corresponding prediction set sizes. From the first row of Fig 26, the class-wise coverage bars of CCP and RC3P distribute on the right-hand side of the target probability 1 -α (red dashed line). Second, RC3P outperforms CCP and Cluster-CP with 24 . 47% (on four datasets) or 32 . 63% (excluding CIFAR-10) on imbalanced datasets and 32 . 63% on balanced datasets decrease in terms of average prediction set size the same class-wise coverage. The second row of Figure 26 shows (i) RC3P has more concentrated classwise coverage distribution than CCP and Cluster-CP ; (ii) the distribution of prediction set sizes produced by RC3P is globally smaller than that produced by CCP and Cluster-CP , which is justified by a better trade-off number of { σ y } K y =1 as shown in Figure 3.

Figure 27 illustrates the normalized frequency distribution of label ranks included in the prediction sets on balanced datasets. It is evident that the distribution of label ranks in the prediction set generated by RC3P tends to be lower compared to those produced by CCP and Cluster-CP . Furthermore, the probability density function tail for label ranks in the RC3P prediction set is notably shorter than that of other methods. This indicates that RC3P more effectively incorporates lower-ranked labels into prediction sets, as a result of its augmented rank calibration scheme.

Figure 28 verifies the condition numbers σ y on balanced datasets. This result verifies the validity of Lemma 4.2 and Equation 6 and confirm that the optimized trade-off between the coverage with inflated quantile and the constraint with calibrated rank leads to smaller prediction sets.

(a) CIFAR-100

(b) Places365

(c) iNaturalist

(d) ImageNet

0.87

0.90

0.94

0.97

1.00

Class-wise Coverage

0.0

5.0

10.0

15.0

20.0

25.0

Frequency

Method

CCP

0.86

0.89

0.93

0.96

1.00

Class-wise Coverage

0.0

10.0

20.0

30.0

40.0

Frequency

Method

CCP

0.86

0.89

0.93

0.96

1.00

Class-wise Coverage

0.0

20.0

40.0

60.0

80.0

Frequency

Method

CCP

0.82

0.87

0.91

0.96

1.00

Class-wise Coverage

0.0

20.0

40.0

60.0

80.0

100.0

Frequency

Method

CCP

8.61

19.07 29.53 39.99 50.45

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

Frequency

Method

CCP

3.72

12.16 20.60 29.04 37.49

0.0

10.0

20.0

30.0

40.0

50.0

Frequency

Method

CCP

2.33

21.38 40.42 59.47 78.52

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

Frequency

Method

CCP

1.50 76.53151.56 226.59 301.62

0.0

50.0

100.0

150.0

200.0

250.0

300.0

350.0

Frequency

Method

CCP

(a) CIFAR-100

(b) Places365

(c) iNaturalist

(d) ImageNet

1

26

50

75

100

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Normalized Frequency

Method

CCP

1

78

156

233

310

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Normalized Frequency

Method

CCP

1

158

316

473

630

Rank

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

Normalized Frequency

Method

CCP

1

250

498

747

996

Rank

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

Normalized Frequency

Method

CCP

(a) CIFAR-100

(b) Places365

(c) iNaturalist

(d) ImageNet

0.05

0.28

0.52

0.76

1.00

Sigma_y

0.0

2.0

4.0

6.0

8.0

10.0

12.0

Frequency

1.0

0.13

0.35

0.57

0.78

1.00

Sigma_y

0.0

5.0

10.0

15.0

20.0

25.0

Frequency

1.0

0.07

0.30

0.54

0.77

1.00

Sigma_y

0.0

5.0

10.0

15.0

20.0

25.0

Frequency

1.0

0.00

0.25

0.50

0.75

1.00

Sigma_y

0.0

10.0

20.0

30.0

40.0

50.0

60.0

Frequency

1.0

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction has stated the contributions and important assumptions of our paper and match our theoretical and experimental results. We have summarize all claims at the end of introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [No]

Justification: A limitation of our paper is that we assume that ( X i , Y i ) are exchangeable (for example, i.i.d.). This assumption is common and fundamental in CP works, so we do not discuss in our paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: In Section 4.1 and 4.2, we provide the the full set of assumptions for our theoretical result. Corresponding proofs are provided in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided all all the information needed to reproduce the experiments, including experiments setting, evaluation metric and codes.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We have provided the codes in the supplementary material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips.cc/public/guides/CodeSubmissionPolicy) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided the detail about dataset, training and calibration in Section 5.1 and Appendix C.1, C.2, and C.3.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have provide the standard deviation of our main results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We follow the training setting of previous papers, so we choose to not discuss the computer resources.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

Answer: [Yes]

Justification: Our work strictly adheres to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [No]

Justification: The methodological improvements gained in our paper can lead to improvements in safe deployment of classifiers in human-ML collaborative systems. We do not anticipate any negative ethical or societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our experiments are conducted on public and benchmark datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All the assets used in the paper are open-source and have been properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We have provided anonymized zip file in supplementary materiel.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the Rank Calibrated Class-conditional Conformal Prediction (RC3P) algorithm, designed to address the issue of large prediction sets in conformal prediction (CP), especially in imbalanced classification tasks. The RC3P algorithm enhances class-conditional coverage by selectively applying class-wise thresholding based on a label rank calibration strategy. This method reduces the size of prediction sets compared to the standard class-conditional CP (CCP) method while maintaining valid class-wise coverage. The authors demonstrate that RC3P achieves significant reductions in prediction set sizes across multiple real-world datasets.\n\nSTRENGTHS:\n1) The introduction of a rank calibration strategy to the class-conditional CP framework is a novel contribution. It offers a practical solution to the challenge of large prediction sets in CP, especially for imbalanced datasets.\n2) The paper provides comprehensive experimental results across various datasets and imbalance types, showing a significant reduction in prediction set sizes while maintaining valid class-wise coverage.\n3) The authors provide rigorous theoretical guarantees for the class-wise coverage and improved predictive efficiency of the RC3P algorithm, demonstrating its robustness and general applicability.\n4) The paper is well-structured and clearly written, with detailed explanations of the algorithm, theoretical analyses, and experimental setups.\n\nWEAKNESSES:\n1) While the paper provides theoretical guarantees for coverage, the theoretical analysis of the predictive efficiency could be expanded. Specifically, a deeper exploration of the conditions under which RC3P outperforms CCP in terms of predictive efficiency would strengthen the paper.\n\n2) Although the experiments are thorough, additional experiments on more diverse and larger-scale datasets could further validate the generalizability of the RC3P method.\n\n3) One additional comment is to verify the predictive efficiency using approximate conditional coverage metrics suggested in the literature, for example, by \"Cauchois, M., Gupta, S., and Duchi, J. (2021). Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction. Journal of Machine Learning Research, 22(81):1–42\" to ensure that approximate conditional coverage is not negatively impacted.\n\nQUESTIONS:\n1) Can the authors provide more details on the computational complexity of the RC3P algorithm compared to the CCP and Cluster-CP methods?\n2) How does the RC3P algorithm perform on larger and more diverse datasets beyond the ones tested in this paper?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
This paper introduces the Rank Calibrated Class-conditional CP (RC3P) algorithm, which reduces prediction set sizes while ensuring valid class-conditional coverage by selectively applying class-wise thresholding. The RC3P algorithm achieves class-wise coverage regardless of the classifier and data distribution, and show improvement empirically.

STRENGTHS:
The idea is simple but effective, supported by solid theoretical and empirical evidence.

WEAKNESSES:
- Some notions, especially the Y and y, are confusing. Similarly, k is for label class, $\hat{k}(y)$ is for a different notion of label rank, but both uses k.
- More extra illustrations may be helpful for understanding. For example, regarding the tony example in para 1 of section 4, try give a concrete figure example. Also, give a table of important dataset statistics, e.g., total sample size, number of class, imbalance rate.

QUESTIONS:
- For fairness evaluation, especially for many classes, what's the worst under coverage ratio? Only the mean is provided.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to reduce the prediction set sizes of conformal predictors while achieving class-conditional coverage. The authors identify that class-wise conformal prediction (CCP) scans all labels uniformly, resulting in large prediction sets. To address this issue, they propose Rank Calibrated Class-conditional Conformal Prediction (RC3P), which augments the label rank calibration strategy within CCP. Theoretical analysis and experimental results demonstrate that RC3P outperforms CCP.\n\nSTRENGTHS:\n1. The proposed method is well-motivated, and the paper is easy to follow. The authors find that CCP scans all labels uniformly, resulting in large prediction sets.\n    \n2. The proposed method introduces a novel conformal predictor for class-conditional coverage, achieving smaller set sizes than other classical methods on imbalanced datasets.\n\nWEAKNESSES:\n1. The proposed methods and experiments appear logically incoherent. The authors provide an explanation for the large prediction sets of CCP, but the interesting conclusion is not related to imbalanced data. Why do the authors only consider imbalanced datasets? Additionally, could the authors report the performance of different methods on balanced data?\n    \n2. The datasets used in the experiments are small, which is impractical for real-world applications. As shown in Section 5, the number of classes in the four datasets is fewer than 101. Moreover, Cluster-CP was proposed for datasets with many classes, so it may be unfair to compare these methods only on small-scale datasets. Furthermore, the classical method THR[1] should be considered in the experiments.\n    \n3. The contribution of Lemma 4.2 is limited. The assumption in Lemma 4.2 is very strong, which can directly infer the final results.\n\n[1] Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association\n\nQUESTIONS:\n- What is the ratio of training set, calibration set and test set? Does RC3P still outperform other methods when the size of calibration set is small? I think an ablation experiment about the size of calibration set is better.\n    \n- How do you control the UCR of RC3P? Please provide some details.\n\nLIMITATIONS:\nThey are adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

comments: SUMMARY:
The paper proposes a new algorithm called Rank Calibrated Class-conditional CP (RC3P) that augments the label rank calibration to conformal classification calibration step. It theoretically proves it

STRENGTHS:
- Overall, the idea is clearly presented, and the motivation behind the problem—improving efficiency in CCP for imbalanced data—is compelling.
- The idea of only using top-k classes in the conformal calibration is intuitive.
- The experimental results are adequate, covering different datasets/settings.

WEAKNESSES:
- R3CP heavily relies on the ranking of candidate class labels. If the classifier's ranking is not reliable, the result could be more conservative.
- Lemma 4.2 is not very convincing, since the assumption is basically the conclusion itself. Theorem 4.3 tries to give a condition on when Lemma 4.2 holds, but it is still not very informative. It's not clear to me that R3CP is better than CCP.

QUESTIONS:
In the simulation result, you "set the UCR of RC3P the same as or smaller (more restrictive) than that of other methods under 0.16 on CIFAR-10 and 0.03 on other datasets". I though UCR is a metric and why would you be able to do that.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13