Title: Class-Conditional Conformal Prediction with Many Classes
Venue: NeurIPS 2023
Accepted: yes
Keywords: conformal prediction

uncertainty quantification

class imbalance
Abstract: Standard conformal prediction methods provide a marginal coverage guarantee,
which means that for a random test point, the conformal prediction set contains 
the true label with a user-specified probability. In many classification
problems, we would like to obtain a stronger guarantee--that for test points
of a specific class, the prediction set contains the true label with the
same user-chosen probability. For the latter goal, existing conformal prediction
methods do not work well when there is a limited amount of labeled data per
class, as is often the case in real applications where the number of classes is
large. We propose a method called clustered conformal prediction that
clusters together classes having "similar" conformal scores and performs 
conformal prediction at the cluster level. Based on empirical evaluation across
four image data sets with many (up to 1000) classes, we find that clustered
conformal typically outperforms existing methods in terms of class-conditional
coverage and set size metrics.

Class-Conditional Conformal Prediction with Many Classes

Abstract

Standard conformal prediction methods provide a marginal coverage guarantee, which means that for a random test point, the conformal prediction set contains the true label with a user-specified probability. In many classification problems, we would like to obtain a stronger guarantee-that for test points of a specific class , the prediction set contains the true label with the same user-chosen probability. For the latter goal, existing conformal prediction methods do not work well when there is a limited amount of labeled data per class, as is often the case in real applications where the number of classes is large. We propose a method called clustered conformal prediction that clusters together classes having 'similar' conformal scores and performs conformal prediction at the cluster level. Based on empirical evaluation across four image data sets with many (up to 1000) classes, we find that clustered conformal typically outperforms existing methods in terms of classconditional coverage and set size metrics.

1 Introduction

Consider a situation in which a doctor relies on a machine learning system that has been trained to output a set of likely medical diagnoses based on CT images of the head. Suppose that the system performs well on average and is able to produce prediction sets that contain the actual diagnosis with at least 0.9 probability. Upon closer examination, however, the doctor discovers the algorithm only predicts sets containing common and relatively benign conditions, such as { normal , concussion } , and the sets never include less common but potentially fatal diseases, such as intracranial hemorrhage . In this case, despite its high marginal (average) performance, the doctor would not want to use such an algorithm because it may lead to patients missing out on receiving critical care. The core problem is that even though the average performance of the algorithm is good, the performance for some classes is quite poor.

Conformal prediction (Vovk et al., 2005) is a method for producing set-valued predictions that serves as a wrapper around existing prediction systems, such as neural networks. Standard conformal prediction takes a black-box prediction model, a calibration data set, and a new test example X test ∈ X with unknown label Y test ∈ Y and creates a prediction set C ( X test ) ⊆ Y that satisfies marginal coverage :

for a coverage level α ∈ [0 , 1] chosen by the user. However, as the example above shows, the utility of these prediction sets can be limited in some real applications. In classification, which we study in

this paper, the label space Y is discrete, and it is often desirable to have class-conditional coverage :

meaning that every class y has at least 1 -α probability of being included in the prediction set when it is the true label. Note that (2) implies (1). Predictions sets that only satisfy (1) may neglect the coverage of some classes, whereas the predictions sets in (2) are effectively 'fair' with respect to all classes, even the less common ones.

Standard conformal prediction, which we will refer to as STANDARD, does not generally provide class-conditional coverage. We present a brief case study to illustrate.

ImageNet case study. Running STANDARD using a nominal coverage level of 90% on 50,000 examples sampled randomly from ImageNet (Russakovsky et al., 2015), a large-scale image data set described later in Section 3, yields prediction sets that achieve very close to the correct marginal coverage (89.8%). However, this marginal coverage is achieved by substantially undercovering some classes and overcovering others. For example, water jug is severely undercovered: the prediction sets only include it in 50.8% of the cases where it is the true label. On the other hand, flamingo is substantially overcovered: it achieves a class-conditional coverage of 99.2%. This underscores the need for more refined methods in order to achieve the class-conditional coverage defined in (2).

In principle, it is possible to achieve (2) by splitting the calibration data by class and running conformal prediction once for each class (V ovk, 2012). We refer to this as CLASSWISE. However, this procedure fails to be useful in many real applications since data-splitting can result in very few calibration examples for each class-level conformal prediction procedure. This typically happens in problem settings where we have many classes but limited data; in such settings, the classwise procedure tends to be overly conservative and produces prediction sets that are too large to be practically useful. We note that previous papers (Shi et al., 2013; Löfström et al., 2015) on class-conditional conformal have not focused on the many classes regime and have instead studied binary or at most 10-way classification tasks.

In this work, we focus on the challenging limited-data, many-class classification setting and we develop a method targeted at class-conditional coverage. Our method mitigates the problems that arise from data-splitting by clustering together classes that have similar score distributions and combining the calibration data for those classes. Figure 1 illustrates how the method we propose strikes a balance between STANDARD and CLASSWISE. As we will later show in our experiments, this can improve class-conditional coverage in many situations.

Low variance

but insu

ffi

cient granularity

Su

ffi

cient granularity

but high variance

Clustered conformal prediction

88

Our contributions. We make three advances in tackling the problem of class-conditional coverage.

We propose an extension of conformal prediction called clustered conformal prediction that often outperforms standard and classwise conformal, in terms of class-conditional coverage, when there is limited calibration data available per class.

We present a comprehensive empirical evaluation of class-conditional coverage for conformal methods on four large-scale classification data sets, each with many (100 to 1000) classes.

We provide general guidelines to practitioners for how to choose an appropriate conformal prediction method in order to achieve class-conditional coverage in the problem setting at hand.

1.1 Related work

Mondrian conformal prediction. We work in the split-conformal prediction framework (Papadopoulos et al., 2002; Lei et al., 2018) in which the training data set (to train the base classifier) and calibration data set are disjoint. Mondrian conformal prediction (MCP) is a general procedure that encompasses many kinds of conditional conformal prediction (Vovk et al., 2005). For any chosen grouping function G : X × Y → G where G denotes the set of all groups, MCP provides coverage guarantees of the form P ( Y test ∈ C ( X test ) | G ( X test , Y test ) = g ) ≥ 1 -α for all groups g ∈ G . The high-level idea behind MCP is to split the calibration data by group and then run conformal prediction on each group. The CLASSWISE conformal procedure is a special case of MCP where each value of Y defines a group (this is also sometimes referred to as label-conditional conformal prediction).

To the best of our knowledge, previous work has not focused on class-conditional coverage of conformal methods in the many-classes, limited-data regime that is common to many real classification applications. Löfström et al. (2015) present an empirical study of STANDARD and CLASSWISE on binary, 3-way, and 4-way classification data sets (with a focus on the binary setting). For classimbalanced problems, they find that STANDARD tends to overcover the majority class, and undercover the minority class. Shi et al. (2013) consider CLASSWISE on the MNIST and USPS data sets, which are 10-way classification problems. Sun et al. (2017) use a cross-fold variant of CLASSWISE for binary classification on imbalanced bioactivity data. Hechtlinger et al. (2018) run class-conditional experiments on the 3-class Iris data set, and provide preliminary ideas on how to incorporate interactions between classes using density estimation. Guan and Tibshirani (2022) use CLASSWISE in an outlier detection context. Sadinle et al. (2019) consider CLASSWISE with a modification to avoid empty prediction sets and perform experiments on 3-way or 10-way classification tasks. To reiterate, the aforementioned papers all focus on the regime that is generally favorable to CLASSWISE, where data is abundant relative to the number of classes. Our work focuses on the limited-data regime that often arises in practice.

Other types of conditional conformal prediction. 'Conditional coverage' is a term that, within the conformal prediction literature, often refers to coverage at a particular input value X = x . It is known to be impossible to achieve coverage conditional on X = x , for all x , without invoking further distributional assumptions (Lei and Wasserman, 2014; Vovk, 2012; Barber et al., 2021). However, approximate X -conditional coverage can be achieved in practice by designing better score functions (Romano et al., 2019, 2020b; Angelopoulos et al., 2022) or by modifying the conformal procedure itself (Romano et al., 2020a; Guan, 2023; Gibbs et al., 2023). Our work draws inspiration from the latter camp, but our aim is to achieve class-conditional ( Y -conditional) coverage rather than X -conditional coverage. Note that X -conditional coverage is hard to interpret in settings in which Y is not random given X = x (e.g., in image classification, if x is an image of a dog, then the true label is always y = dog , with no intrinsic randomness after conditioning on X = x ). In such settings, it is more natural to consider class-conditional coverage.

1.2 Preliminaries

We work in the classification setting in which each input X i ∈ X has a class label Y i ∈ Y , for some discrete set Y . Let { ( X i , Y i ) } N i =1 denote a calibration data set , where each ( X i , Y i ) iid ∼ F . Given a new independent test point ( X test , Y test ) ∼ F , our goal is to construct (without knowledge of Y test ) a prediction set C ( X test ) that satisfies (2) for some user-specified α ∈ [0 , 1] .

Let s : X × Y → R denote a (conformal) score function , where we take s ( x, y ) to be negatively oriented, which means that lower scores indicate a better agreement between the input x and the proposed class label y . The score function is typically derived from a pre-trained classifier f (a simple example to keep in mind is s ( x, y ) = 1 -f y ( x ) , where f y ( x ) represents the y th entry of the softmax vector output by f for the input x ). For brevity, we denote the score of the i th calibration data point as s i = s ( X i , Y i ) . For τ ∈ [0 , 1] and a finite set A ⊆ R , let Quantile( τ, A ) denote the smallest a ∈ A such that τ fraction of elements in A are less than or equal to a . For τ > 1 , we take Quantile( τ, A ) = ∞ .

With this notation, the STANDARD conformal prediction sets are given by

These prediction sets are guaranteed to satisfy (1) (Vovk et al., 2005). We can interpret ˆ q as a finite-sample adjusted (1 -α ) -quantile of the scores in the calibration data set. Note that all N data points are used for computing a single number, ˆ q .

In contrast, the CLASSWISE procedure computes a separate quantile for every class. Let I y = { i ∈ [ N ] : Y i = y } be the indices of examples in the calibration data set that have label y . The CLASSWISE conformal prediction sets are given by

These prediction sets are guaranteed to satisfy (2) (Vovk, 2012). However, the quantile ˆ q y is only computed using a subset of the data of size |I y | , which might be quite small. Importantly, for any class y for which |I y | < (1 /α ) -1 , we will have ˆ q y = ∞ , hence any prediction set generated by CLASSWISE will include y , no matter the values of the conformal scores.

The advantage of STANDARD is that we do not need to split up the calibration data to estimate ˆ q , so the estimated quantile has little noise even in limited-data settings; however, it does not in general yield class-conditional coverage. Conversely, CLASSWISE is guaranteed to achieve class-conditional coverage, since we estimate a different threshold ˆ q y for every class; however, these estimated quantiles can be very noisy due to the limited data, leading to erratic behavior, such as large sets.

When we assume the scores are almost surely distinct, the exact distribution of the CLASSWISE class-conditional coverage of class y given a fixed calibration set is (V ovk, 2012; Angelopoulos and Bates, 2023):

where k y α = ⌈ ( |I y | +1)(1 -α ) ⌉ . Note that the probability on the left-hand side above is taken with respect to the test point X test as the only source of randomness, as we have conditioned on the calibration set. The beta distribution Beta( k y α , |I y | +1 -k y α ) has mean k y α / ( |I y | +1) (which, as expected, is always at least 1 -α ) and variance

which can be large when |I y | is small. For example, if class y only has 10 calibration examples and we seek 90% coverage, then the class-conditional coverage given a fixed calibration set is distributed as Beta(10 , 1) , so there is probability ≈ 0 . 107 that the coverage of class y will be less than 80%. Somewhat paradoxically, the variance of the class-conditional coverage means that on a given realization of the calibration set, the CLASSWISE method can exhibit poor coverage on a substantial fraction of classes if the number of calibration data points per class is limited.

where where

2 Clustered conformal prediction

With the goal of achieving the class-conditional coverage guarantee in (2), we introduce clustered conformal prediction . Our method strikes a balance between the granularity of CLASSWISE and the data-pooling of STANDARD by grouping together classes according to a clustering function. For each cluster, we calculate a single quantile based on all data in that cluster. We design the clustering algorithm so that clustered classes have similar score distributions, and therefore, the resulting clusterlevel quantile is applicable to all classes in the cluster. Next, in Section 2.1, we formally describe the clustered conformal prediction method; then, in Section 2.2, we describe the clustering step in detail.

2.1 Meta-algorithm

To begin, we randomly split the calibration data set into two parts: the clustering data set D 1 = { ( X i , Y i ) : i ∈ I 1 } , for performing clustering, and a proper calibration data set D 2 = { ( X i , Y i ) : i ∈ I 2 } , for computing the conformal quantiles, where |I 1 | = ⌊ γN ⌋ and |I 2 | = N -|I 1 | for some tuning parameter γ ∈ [0 , 1] . Then, we apply a clustering algorithm to D 1 to obtain a clustering function ˆ h : Y → { 1 , . . . , M } ∪ { null } that maps each class y ∈ Y to one of M clusters or the 'null' cluster (denoted null ). The reason that we include the null cluster is to handle rare classes that do not have sufficient data to be confidently clustered into any of the M clusters. Details on how to create ˆ h are given in the next subsection.

After assigning classes to clusters, we perform the usual conformal calibration procedure within each cluster. Denote by I y 2 = { i ∈ I 2 : Y i = y } the indices of examples in D 2 with label y , and by I 2 ( m ) = ∪ y : ˆ h ( y )= m I y 2 the indices of examples in D 2 with labels in cluster m . The CLUSTERED conformal prediction sets are given by where, for m = 1 , . . . , M

and

In words, for each cluster m , we group together examples for all classes in that cluster and then estimate a cluster-level quantile ˆ q ( m ) . When constructing prediction sets, we include class y in the set if the conformal score for class y is less than or equal to the quantile for the cluster that contains y . For classes assigned to the null cluster, we compare the conformal score against the quantile that would be obtained from running STANDARD on the proper calibration set.

We now consider the properties of the CLUSTERED prediction sets. For all classes that do not belong to the null cluster, we have the following guarantee (the proof of this result, and all other proofs, are deferred to Appendix A).

Proposition 1. The prediction sets C = C CLUSTERED from CLUSTERED achieve cluster-conditional coverage:

This coverage result comes from the exchangeability between a test point drawn from cluster m and all of the calibration points that belong to cluster m . We get this exchangeability for free from the assumption that the calibration point and test points are sampled i.i.d. from the same distribution. Cluster-conditional coverage is a stronger guarantee than marginal coverage, but it is still not as strong as the class-conditional coverage property that we hope to achieve. However, cluster-conditional coverage implies class-conditional in an idealized setting: suppose we have access to an 'oracle' clustering function h ∗ that produces M clusters such that, for each cluster m = 1 , . . . , M , the scores for all classes in this cluster are exchangeable (this would hold, for example, if all classes in the same cluster have the same score distribution). In this case, we have a guarantee on class-conditional coverage.

̸

Proposition 2. If ˆ h = h ∗ , the 'oracle' clustering function as described above, then the prediction sets from CLUSTERED satisfy class-conditional coverage (2) for all classes y such that h ∗ ( y ) = null .

This coverage result arises because the oracle clustering function ensures exchangeability between a test point drawn from class y and all of the calibration points that belong to the cluster to which y belongs. To try to achieve this exchangeability, we need to construct the clusters carefully. Specifically, we want to design ˆ h so that the scores within each cluster m = 1 , . . . , M are as close to identically distributed as possible, an idea we pursue next.

2.2 Quantile-based clustering

We seek to cluster together classes that have similar score distributions. To do so, we first summarize the empirical score distribution for each class via a vector of score quantiles evaluated at a discrete set of levels τ ∈ T ⊆ [0 , 1] . In this embedding space, a larger distance between classes means their score distributions are more different. While there are various options for defining such an embedding, recall that to achieve class-conditional coverage, we want to accurately estimate the (finite-sample adjusted) (1 -α ) -quantile for the score distribution of each class. Thus, we want to group together classes with similar quantiles, which is what our embedding is designed to facilitate. After obtaining these embeddings, we can then simply apply any clustering algorithm, such as k -means.

In more detail, denote by I y 1 = { i ∈ I 1 : Y i = y } the indices of examples in D 1 with label y . We compute quantiles of the scores { s i } i ∈I y 1 from class y at the levels

and collect them into an embedding vector z y ∈ R |T | . If |I y 1 | < n α , where n α = (1 / min { α, 0 . 1 } ) -1 , then the uppermost quantile in z y will not be finite, so we simply assign y to the null cluster. For a pre-chosen number of clusters M , we run k -means clustering with k = M on the data { z y } y ∈Y\Y null , where Y null denotes the set of labels assigned to the null cluster. More specifically, we use a weighted version of k -means where the weight for class y is set to |I y 1 | 1 / 2 ; this allows the class embeddings computed from more data to have more influence on the cluster centroids. We denote the cluster mapping that results from this procedure by ˆ h : Y → { 1 , . . . , M } ∪ { null } .

Of course, we cannot generally recover an oracle clustering function h ∗ with the above (or any practical) procedure, so Proposition 2 does not apply. However, if the score distributions for the classes that ˆ h assigns to the same cluster are similar enough, then we can provide an approximate class-conditional coverage guarantee. We measure similarity in terms of the Kolmogorov-Smirnov (KS) distance, which is defined between random variables X and Y as D KS ( X,Y ) = sup λ ∈ R | P ( X ≤ λ ) -P ( Y ≤ λ ) | .

Proposition 3. Let S y denote a random variable sampled from the score distribution for class y , and assume that the clustering map ˆ h satisfies

̸

̸

Then, for C = C CLUSTERED and for all classes y such that such that ˆ h ( y ) = null ,

To summarize, the two main tuning parameters used in the proposed method are γ ∈ [0 , 1] , the fraction of points to use for the clustering step, and M ≥ 1 , the number of clusters to use in k -means. While there are no universal fixed values of these parameters that serve all applications equally well, we find that simple heuristics for setting these parameters often work well in practice, which we describe in the next section.

3 Experiments

We evaluate the class-conditional coverage of STANDARD, CLASSWISE, and CLUSTERED conformal prediction on four large-scale image data sets using three conformal score functions on each. Code for reproducing our experiments is available at https://github.com/tiffanyding/ class-conditional-conformal .

3.1 Experimental setup

We run experiments on the ImageNet (Russakovsky et al., 2015), CIFAR-100 (Krizhevsky, 2009), Places365 (Zhou et al., 2018), and iNaturalist (Van Horn et al., 2018) image classification data sets, whose characteristics are summarized in Table 1. The first three have roughly balanced classes, the fourth, iNaturalist, has high class imbalance. We consider three conformal score functions: softmax , one minus the softmax output of the base classifier; APS , a score designed to improve X -conditional coverage; and RAPS , a regularized version of APS that often produces smaller sets. Precise definitions of the score functions are provided in Appendix B.1; we refer also to Romano et al. (2020b); Angelopoulos et al. (2021) for the details and motivation behind APS and RAPS . Throughout, we set α = 0 . 1 for a desired coverage level of 90%.

*The number of classes in the iNaturalist data set can be adjusted by selecting which taxonomy level (e.g., species, genus, family) to use as the class labels. We use the species family as our label and then filter out any classes with < 250 examples in order to have sufficient examples to properly perform evaluation.

Our experiments all follow a common template. First, we fine-tune a pre-trained neural network as our base classifier (for details on model architectures, see Appendix B.2) on a small subset D fine of the original data, leaving the rest for calibration and validation purposes. We construct calibration sets of varying size by changing the average number of points in each class, denoted n avg . For each n avg ∈ { 10 , 20 , 30 , 40 , 50 , 75 , 100 , 150 } , we construct a calibration set D cal by sampling n avg ×|Y| examples without replacement from the remaining data D c fine (where c denotes the set complement). We estimate the conformal quantiles for STANDARD, CLASSWISE, and CLUSTERED on D cal . The remaining data ( D fine ∪ D cal ) c is used as the validation set for computing coverage and set size metrics. Finally, this process-splitting D c fine into random calibration and validation sets-is repeated ten times, and the reported metrics are averaged over these repetitions.

Details about clustering. For CLUSTERED, we choose γ ∈ [0 , 1] (the fraction of calibration data points used for clustering) and M ≥ 1 (the number of clusters) in the following way. First, we define n min = min y ∈Y |I y | , the number of examples in the rarest class in the calibration set, and n α = (1 /α ) -1 , the minimum sample size needed so that the finite-sample adjusted (1 -α ) -quantile used in conformal prediction is finite (e.g., n α = 9 when α = 0 . 1 ). Now define ˜ n = max( n min , n α ) and let K be the number of classes with at least ˜ n examples. We then set γ = K/ (75 + K ) and M = ⌊ γ ˜ n/ 2 ⌋ . These choices are motivated by two goals: we want M and γ to grow together (to find more clusters, we need more samples for clustering), and we want the proper calibration set to have at least 150 points per cluster on average; see Appendix B.4 for details on how the latter is achieved. Clustering is carried out by running k -means on the quantile-based embeddings, as described in Section 2.2; we use the implementation in sklearn.cluster.KMeans , with the default settings (Pedregosa et al., 2011).

3.2 Evaluation metrics

Denote the validation data set by { ( X ′ i , Y ′ i ) } N ′ i =1 (recall this is separate from the fine-tuning and calibration data sets) and let J y = { i ∈ [ N ′ ] : Y ′ i = y } be the indices of validation examples with label y . For any given conformal method, we define ˆ c y = 1 |J y | ∑ i ∈J y 1 { Y ′ i ∈ C ( X ′ i ) } as the empirical class-conditional coverage of class y . Our main evaluation metric is the average class coverage gap (CovGap):

This measures how far the class-conditional coverage is from the desired coverage level of 1 -α , in terms of the ℓ 1 distance across all classes (multiplied by 100 to put it on the percentage scale). We also measure the sharpness of the prediction sets by average set size (AvgSize):

For a given class-conditional coverage level (a given CovGap), we want a smaller average set size.

3.3 Results

To begin, we investigate the CovGap of the methods on each data set, and display the results in Figure 2. In brief, CLUSTERED achieves the best or comparable performance across all settings. Restricting our attention to the baseline methods, note that the CovGap of STANDARD does not change much as we vary n avg ; this is as expected, because n avg × |Y| samples are being used to estimate the conformal quantile ˆ q , which will be stable regardless of n avg , provided |Y| is large. Conversely, the CovGap of CLASSWISE decreases significantly as n avg increases, because the classwise conformal quantiles ˆ q y are volatile for small n avg .

For the softmax score (left column of the figure), we see that CLUSTERED clearly outperforms STANDARD, and the gap widens with n avg ; further, CLUSTERED outperforms CLASSWISE for small enough values of n avg (that is, n avg < 75 for ImageNet and Places365, n avg < 150 for CIFAR100, and n avg < 50 for iNaturalist). The comparison between CLUSTERED and CLASSWISE is qualitatively similar for APS score (right column of the figure), with the former outperforming the

.

2

4

6

8

CovGap

ImageNet: softmax

.

ImageNet: APS

Standard

Classwise

Clustered

.

2

4

6

8

CovGap

CIFAR-100: softmax

.

CIFAR-100: APS

.

2

4

6

8

CovGap

Places365: softmax

.

Places365: APS

10

20

30

40

50

75

100

150

navg

2

4

6

8

CovGap

iNaturalist: softmax

10

20

30

40

50

75

100

150

navg

iNaturalist: APS

latter for small enough values of n avg . However, the behavior of STANDARD changes notably as we move from softmax to APS : it becomes comparable to CLUSTERED and only slightly worse for large n avg . Lastly, the results for RAPS (not shown, and deferred to Appendix C.1) are similar to APS but the CovGap is shifted slightly higher.

To examine the potential tradeoffs between class-conditional coverage and average set size, we focus on the n avg ∈ { 10 , 20 , 50 , 75 } settings and report CovGap and AvgSize in Table 2 for all data sets and score functions. We see that CLUSTERED achieves the best or near-best CovGap in any experimental combination, and its improvement over the baselines is particularly notable in the regime where data is limited, but not extremely limited ( n avg = 20 or 50). Meanwhile, AvgSize for STANDARD and CLUSTERED is relatively stable across values of n avg , with the latter being generally slightly larger; AvgSize for CLASSWISE decreases as we increase n avg , but is still quite large relative to STANDARD and CLUSTERED, especially for iNaturalist.

We finish with two more remarks. First, we find CovGap tends to behave quite similarly to the fraction of classes that are drastically undercovered, which we define as having a class-conditional coverage at least 10% below the desired level. Results for this metric are given in Appendix C.2. Second, as is the case in any conformal method, auxiliary randomization can be applied to STANDARD, CLASSWISE, or CLUSTERED in order to achieve a coverage guarantee (marginal, class-conditional, or cluster-conditional, respectively) of exactly 1 -α , rather than at least 1 -α . These results are included in Appendix C.3. In terms of CovGap, we find that randomization generally improves CLASSWISE, and does not change STANDARD or CLUSTERED much at all; however, the set sizes from randomized CLASSWISE are still overall too large to be practically useful (moreover, injecting auxiliary randomness is prediction sets is not always practically palatable).

4 Discussion

We summarize our practical takeaways, in an effort towards creating guidelines for answering the question: for a given problem setting, what is the best way to produce prediction sets that have good class-conditional coverage but are not too large to be useful?

Extremely low-data regime . When most classes have very few calibration examples (say, less than 10), this is not enough data to run CLUSTERED or CLASSWISE unless α is large, so the only option is to run STANDARD. With this method, softmax and RAPS are both good score functions. RAPS tends to yield better class-conditional coverage, while softmax tends to have smaller sets.

Low-data regime . When the average number of examples per class is low but not tiny (say, around 20 to 75), CLUSTERED tends to strike a good balance between variance and granularity and often achieves good class-conditional coverage and reasonably-sized prediction sets with either softmax or RAPS . The STANDARD method with RAPS is also competitive towards the lower-data end of this regime.

High-data regime. When the average number of examples per class is large (say, over 75), CLUSTERED conformal with either softmax or RAPS continues to do well, and CLASSWISE with these same scores can also do well if the classes are balanced. In settings with high class imbalance, CLASSWISE is unstable for rare classes and produces excessively large prediction sets, whereas CLUSTERED is more robust due to the data-sharing it employs.

Extremely high-data regime . When the calibration dataset is so large that the rarest class has at least, say, 100 examples, then CLASSWISE with softmax or RAPS will be a good choice, regardless of any class imbalance.

The boundaries between these regimes are not universal and are dependent on the data characteristics, and the above guidelines are based only on our findings from our experiments. We also note that the precise boundary between the low-data and high-data regimes is dependent on the particular score function that is used: the transition happens around 20-40 examples per class for softmax and 50-100 for RAPS . This serves as further motivation for CLUSTERED, which performs relatively well in all regimes.

As a possible direction for future work, it might be interesting to generalize our approach to the broader problem of group-conditional coverage with many groups. In the present setting, the groups

are defined by class labels, but our clustering methodology could also be applied to other group structures (e.g., defined by the input features or components of a mixture distribution).

Acknowledgments and Disclosure of Funding

We thank Margaux Zaffran and Long Nguyen for helpful suggestions that improved our paper. This work was supported by the National Science Foundation (NSF) Graduate Research Fellowship Program under grant no. 2146752, the European Research Council (ERC) Synergy Grant Program, the Office of Naval Research (ONR) the Mathematical Data Science Program, and the ONR MultiUniversity Research Initiative (MURI) Program under grant no. N00014-20-1-2787.

References

Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification. Foundations and Trends in Machine Learning , 16(4): 494-591, 2023.

Anastasios N. Angelopoulos, Stephen Bates, Jitendra Malik, and Michael I. Jordan. Uncertainty sets for image classifiers using conformal prediction. In Proceedings of the International Conference on Learning Representations , 2021.

Anastasios N. Angelopoulos, Amit Pal Kohli, Stephen Bates, Michael Jordan, Jitendra Malik, Thayer Alshaabi, Srigokul Upadhyayula, and Yaniv Romano. Image-to-image regression with distributionfree uncertainty quantification and applications in imaging. In International Conference on Machine Learning , 2022.

Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution-free conditional predictive inference. Information and Inference , 10(2):455-482, 2021.

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-supervised models are strong semi-supervised learners. In Advances in Neural Information Processing Systems , 2020.

Isaac Gibbs, John J. Cherian, and Emmanuel J. Candès. Conformal prediction with conditional guarantees. arXiv preprint arXiv:2305.12616 , 2023.

Leying Guan. Localized conformal prediction: A generalized inference framework for conformal prediction. Biometrika , 110(1):33-50, 2023.

Leying Guan and Robert Tibshirani. Prediction and outlier detection in classification problems. Journal of the Royal Statistical Society: Series B , 84(2):524, 2022.

Yotam Hechtlinger, Barnabás Póczos, and Larry Wasserman. Cautious deep learning. arXiv preprint arXiv:1805.09460 , 2018.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

Jing Lei and Larry Wasserman. Distribution-free prediction bands for non-parametric regression. Journal of the Royal Statistical Society: Series B , 76(1):71-96, 2014.

Jing Lei, Max G'Sell, Alessandro Rinaldo, Ryan J. Tibshirani, and Larry Wasserman. Distributionfree predictive inference for regression. Journal of the American Statistical Association , 113(523): 1094-1111, 2018.

Tuve Löfström, Henrik Boström, Henrik Linusson, and Ulf Johansson. Bias reduction through conditional conformal prediction. Intelligent Data Analysis , 19(6):1355-1375, 2015.

Harris Papadopoulos, Kostas Proedrou, Volodya Vovk, and Alex Gammerman. Inductive confidence machines for regression. In European Conference on Machine Learning , 2002.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , 2019.

Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research , 12(85):28252830, 2011.

Yaniv Romano, Evan Patterson, and Emmanuel J. Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems , 2019.

Yaniv Romano, Rina Foygel Barber, Chiara Sabatti, and Emmanuel J. Candès. With malice toward none: Assessing uncertainty via equalized coverage. Harvard Data Science Review , 2(2), 2020a.

Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems , 2020b.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet large scale visual recognition challenge. International Journal of Computer Vision , 115:211-252, 2015.

Mauricio Sadinle, Jing Lei, and Larry Wasserman. Least ambiguous set-valued classifiers with bounded error levels. Journal of the American Statistical Association , 114(525):223-234, 2019.

Fan Shi, Cheng Soon Ong, and Christopher Leckie. Applications of class-conditional conformal predictor in multi-class classification. In International Conference on Machine Learning and Applications , 2013.

Jiangming Sun, Lars Carlsson, Ernst Ahlberg, Ulf Norinder, Ola Engkvist, and Hongming Chen. Applying Mondrian cross-conformal prediction to estimate prediction confidence on large imbalanced bioactivity data sets. Journal of Chemical Information and Modeling , 57(7):1591-1598, 2017.

Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The iNaturalist species classification and detection dataset. In IEEE Conference on Computer Vision and Pattern Recognition , 2018.

Vladimir Vovk. Conditional validity of inductive conformal predictors. In Asian Conference on Machine Learning , 2012.

Vladimir Vovk, Alex Gammerman, and Glenn Shafer. Algorithmic Learning in a Random World . Springer, 2005.

Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 40(6):1452-1464, 2018.

A Proofs

Proof of Proposition 1. For each m ∈ { 1 , . . . , M } , denote by G ( m ) the distribution of the score s ( X,Y ) conditioned on Y being in cluster m . Consider a randomly sampled test example ( X test , Y test ) with a label in cluster m ; the test score s test = s ( X test , Y test ) then follows distribution G ( m ) . Next consider { s i } i ∈I 2 ( m ) , the scores for examples in the proper calibration set with labels in cluster m ; these also follow distribution G ( m ) . Furthermore, s test and the elements of { s i } i ∈I 2 ( m ) are all mutually independent, so the result follows by the standard coverage guarantee for conformal prediction with exchangeable scores (e.g., see Vovk et al. (2005) or Lei et al. (2018)).

Proof of Proposition 2. This is a direct result of exchangeability and Proposition 1.

Proof of Proposition 3. Let S = s ( X,Y ) denote the score of a randomly sampled example ( X,Y ) ∼ F . Fix an m ∈ { 1 , . . . , M } . Define Y ( m ) = { y ∈ Y : ˆ h ( y ) = m } as the set of classes that ˆ h assigns to cluster m . Without a loss of generality, we treat both ˆ h and ˆ q ( m ) as fixed for the remainder of this proof. This can be done by conditioning on both the clustering and proper calibration sets, leaving only the randomness in the test point ( X,Y ) ∼ F , and then integrating over the clustering and proper calibration sets in the end.

Let G ( m ) denote the distribution of S conditional on Y ∈ Y ( m ) , and let S ( m ) ∼ G ( m ) . Similarly, let G y denote the distribution of S conditional on Y = y , and let S y ∼ G y . Since we assume that the KS distance between the score distribution for every pair of classes in cluster m is bounded by ϵ , and G ( m ) is a mixture of these distributions (that is, G ( m ) = ∑ y ∈Y ( m ) π y · G y for some fixed probability weights π y , y ∈ Y ( m ) ), it follows by the triangle inequality that

By definition of KS distance, this implies

Since the CLUSTERED procedure includes the true label Y from the prediction set C when S ≤ ˆ q ( m ) , these probabilities can be rewritten in terms of coverage events:

Combining the result from Proposition 1 gives the desired conclusion.

B Experiment details

B.1 Score functions

We perform experiments using three score functions:

softmax : The softmax-based conformal score at an input x and a label y is defined as

where f y ( x ) is entry y of the softmax vector output by the classifier f at input x .

APS : The Adaptive Prediction Sets (APS) score of Romano et al. (2020b) is designed to improve X -conditional coverage as compared to the more traditional softmax score. This score is computed at an input x and label y as follows. Let

denote the sorted values of the base classifier softmax outputs f y ( x ) , y ∈ Y . Let k x ( y ) be the index in the sorted order that corresponds to class y , that is, f ( k x ( y )) = f y ( x ) . The APS score is then defined as

RAPS : The regularized APS (RAPS) score of Angelopoulos et al. (2021) is a modification of the APS score that adds a regularization term designed to reduce the prediction set sizes (which can often be very large with APS). The RAPS score is defined as

where k x ( y ) is as defined above, and λ and k reg are user-chosen parameters. In all of our experiments, we use λ = 0 . 01 and k reg = 5 , which Angelopoulos et al. (2021) found to work well for ImageNet.

B.2 Model training

An important consideration when we fine-tune and calibrate our models is that we must reserve sufficient data to evaluate the class-conditional coverage of the conformal methods. This means we aim to exclude at least 250 examples per class from the fine-tuning and calibration sets so that we can then use this data for validation (applying the conformal methods and computing coverage and set size metrics).

For all data sets except ImageNet, we use a ResNet-50 model as our base classifier. We initialize to the IMAGENET1K_V2 pre-trained weights from PyTorch (Paszke et al., 2019), and then fine-tune all parameters by training on the data set at hand. For ImageNet, we must do something different, as explained below.

ImageNet. Setting up this data set for our experiments is a bit tricky because we need sufficient data for performing validation, but we also need this data to be separate from the fine-tuning and calibration sets. The ImageNet validation set only contains 50 examples per class, which is not enough for our setting. The ImageNet training set is much larger, with roughly 1000 examples per class, but if we want to use this data for validation, then we cannot use the ResNet-50 initialized to the IMAGENET1K_V2 pre-trained weights, as these weights were obtained by training on the whole ImageNet training set. We therefore instead use a SimCLR-v2 model (Chen et al., 2020), which was trained on the ImageNet training set without labels , to extract feature vectors of length 6144 for all images in the ImageNet training set. We then use 10% of these feature vectors for training a linear head (a single fully connected neural network layer). After training for 10 epochs, the model achieves a validation accuracy of 78%. We then apply the linear head to the remaining 90% of the feature vectors to obtain softmax scores for the calibration set.

CIFAR-100. This data set has 600 images per class (500 from the training set and 100 from the validation set). We combine the data and then randomly sample 50% for fine-tuning, and we use the remaining data for calibrating and validating our procedures. After training for 30 epochs, the validation accuracy is 60%.

Places365. This data set contains more than 10 million images of 365 classes, where each class has 5000 to 30000 examples. We randomly sample 90% of the data for fine-tuning, and we use the remaining data for calibrating and validating our procedures. After training for one epoch, the validation accuracy is 52%.

iNaturalist. This data set has class labels of varying specificity. For example, at the species level, there are 6414 classes with 300 examples each (290 training examples and 10 validation examples) and a total of 10000 classes with at least 150 examples. We instead work at the family level, which groups the species into 1103 classes. We randomly sample 50% of the data for fine-tuning, and we use the remaining for calibrating and validating our procedures. After training for one epoch, the validation accuracy is 69%.

However, due to high class imbalance and sampling randomness, some classes have insufficient validation samples, so we filter out classes with fewer than 250 validation examples, which leaves us with 633 classes. The entries of the softmax vectors that correspond to rare classes are removed and the vector is renormalized to sum to one.

B.3 Measuring class balance in Table 1

The class balance metric in Table 1 is defined as the number of examples in the rarest 5% of classes divided by the expected number of examples if the class distribution were perfectly uniform. This metric is bounded between 0 and 1, with lower values denoting more class imbalance. We compute this metric using D c fine .

B.4 Choosing clustering parameters

To choose γ ∈ [0 , 1] and M ≥ 1 for CLUSTERED, as described briefly in Section 3.1, we employ two intuitive heuristics. We restate these heuristics in more detail here.

First, to distinguish between more clusters (or distributions), we need more samples from each distribution. As a rough guess, to distinguish between two distributions, we want at least four samples per distribution; to distinguish between five distributions, we want at least ten samples per distribution. In other words, we want the number of clustering examples per class to be at least twice the number of clusters. This heuristic can be expressed as

where γ ˜ n is the expected number of clustering examples for the rarest class that is not assigned to the null cluster.

Second, we want enough data for computing the conformal quantiles for each cluster. Specifically, we seek at least 150 examples per cluster on average. This heuristic can be expressed as

where K/M is the average number of classes per cluster and (1 -γ )˜ n is the expected number of proper calibration examples for the rarest class not assigned to the null cluster.

Changing the inequalities of (3) and (4) into equalities and solving for γ and M yields

Varying the clustering parameters. As sensitivity analysis, we examine the performance of CLUSTERED across a wide range of values for the tuning parameters γ and M . As the heatmaps in Figure 3 confirm, the performance of CLUSTERED is not particularly sensitive to the values of these parameters. When n avg = 10 , the heuristic chooses γ = 0 . 89 and M = 4 , meanwhile, when n avg = 50 , the heuristic chooses γ ∈ [0 . 88 , 0 . 92] and M ∈ [7 , 12] (since the calibration data set is randomly sampled, and γ and M are chosen based on the calibration data set, there is randomness in the chosen values). However, there are large areas surrounding these values that would yield similar performance. We observe that the heuristics do not always choose the parameter values that yield the lowest CovGap. The heatmaps reveal that the optimal parameter values are dependent not only on data set characteristics, but also on the score function. Future work could be done to extract further performance improvements by determining a better method for choosing γ and M .

C Additional experimental results

We present additional experimental results in this section. As in the main text, the shaded regions in plots denote ± 1 . 96 times the standard errors.

C.1 RAPS CovGap results

Figure 4 shows the CovGap on all data sets when we use RAPS as our score function.

C.2 Additional metrics

Average set size. To supplement Table 2 from the main text, which reports AvgSize for select values of n avg , Figure 5 plots AvgSize for all values of n avg that we use in our experimental setup. Note

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Fraction for clustering (

)

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.3

5.3

5.3

5.3

5.3

5.3

5.3

5.3

5.3

5.3

5.2

5.2

5.2

5.2

5.2

5.3

5.3

5.3

5.3

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.3

5.3

5.3

5.5

5.1

5.1

5.1

5.1

5.1

5.2

5.2

5.2

5.3

5.5

5.1

5

5

5

5.1

5.1

5.1

5.2

5.2

5.6

4.9

4.8

4.8

4.8

4.9

4.9

4.9

5

5.1

5.7

4.9

4.8

4.8

4.9

4.8

5

5

5.1

5.3

6.4

ImageNet: softmax,

navg

=10

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.7

2.7

2.7

2.7

2.7

2.7

2.7

2.7

2.6

2.7

2.7

2.7

2.7

2.7

2.7

2.8

2.8

2.8

3

2.7

2.7

2.7

2.8

2.8

2.8

2.8

2.9

3

3.2

2.8

2.8

2.8

2.8

2.8

2.9

2.9

3

3.2

3.6

2.8

2.7

2.8

2.9

2.9

3

3

3.2

3.3

3.9

2.8

2.9

3

3

3.1

3.2

3.3

3.6

3.8

5

ImageNet: APS,

navg

=10

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3

3.1

3.1

3.1

3.1

3.1

3.1

3.1

3

3

3

3.1

3.1

3.1

3.1

3.1

3.2

3.2

3.4

3.1

3.1

3.1

3.1

3.2

3.2

3.2

3.3

3.3

3.6

3.1

3.1

3.1

3.1

3.1

3.2

3.2

3.3

3.4

3.8

3.1

3.1

3.1

3.1

3.1

3.2

3.3

3.4

3.5

4.2

3.1

3.2

3.2

3.2

3.3

3.5

3.6

3.8

4

5.2

ImageNet: RAPS,

navg

=10

2

3

4

5

6

8

10

15

20

50

# of clusters (

M

)

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Fraction for clustering (

)

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

5.2

4.7

4.5

4.4

4.4

4.4

4.3

4.3

4.3

4.3

4.3

4.3

4

3.9

3.8

3.8

3.8

3.7

3.7

3.7

3.6

4.2

3.8

3.7

3.6

3.5

3.5

3.5

3.4

3.4

3.4

4.1

3.7

3.5

3.4

3.4

3.3

3.3

3.3

3.2

3.2

4.1

3.6

3.4

3.3

3.3

3.2

3.2

3.2

3.1

3.2

4

3.6

3.3

3.3

3.2

3.2

3.1

3.1

3.1

3.1

4

3.5

3.3

3.2

3.2

3.1

3.1

3

3

3.2

4

3.5

3.3

3.2

3.2

3.1

3.1

3.1

3.1

3.6

ImageNet: softmax,

navg

=50

2

3

4

5

6

8

10

15

20

50

# of clusters (

M

)

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.5

2.5

2.5

2.5

2.5

2.5

2.5

2.5

2.5

2.6

2.5

2.4

2.4

2.4

2.4

2.4

2.4

2.4

2.4

2.4

2.5

2.4

2.4

2.4

2.3

2.3

2.3

2.3

2.4

2.4

2.5

2.4

2.3

2.3

2.3

2.3

2.3

2.3

2.3

2.4

2.4

2.3

2.3

2.3

2.3

2.3

2.2

2.3

2.3

2.4

2.4

2.3

2.3

2.2

2.2

2.2

2.2

2.3

2.3

2.5

2.4

2.3

2.2

2.2

2.2

2.2

2.2

2.2

2.3

2.6

2.4

2.3

2.2

2.2

2.2

2.2

2.3

2.3

2.4

3

ImageNet: APS,

navg

=50

2

3

4

5

6

8

10

15

20

50

# of clusters (

M

)

3

3

3

3

3

3

3

3

3

3

2.9

2.9

2.9

2.9

2.9

2.9

2.9

2.9

2.8

2.9

2.9

2.9

2.8

2.8

2.8

2.7

2.7

2.7

2.7

2.7

2.9

2.8

2.7

2.7

2.7

2.7

2.6

2.6

2.6

2.7

2.8

2.7

2.7

2.6

2.6

2.6

2.6

2.6

2.6

2.6

2.8

2.7

2.6

2.6

2.6

2.5

2.5

2.5

2.5

2.6

2.7

2.6

2.6

2.5

2.5

2.5

2.5

2.5

2.5

2.7

2.7

2.6

2.6

2.5

2.5

2.5

2.5

2.5

2.5

2.8

2.7

2.6

2.5

2.5

2.5

2.5

2.5

2.5

2.7

3.2

ImageNet: RAPS,

navg

=50

2

3

4

5

6

7

.

2

4

6

8

CovGap

ImageNet: RAPS

.

CIFAR-100: RAPS

Standard

Classwise

Clustered

10

20

30

40

50

75

100

150

navg

2

4

6

8

CovGap

Places365: RAPS

10

20

30

40

50

75

100

150

navg

iNaturalist: RAPS

that RAPS sharply reduces AvgSize relative to APS on ImageNet and also induces a slight reduction for the other three data sets. This asymmetric reduction is likely due to the fact that the RAPS hyperparameters, which control the strength of the set size regularization, were tuned on ImageNet. The set sizes of RAPS on other data sets could likely be improved by tuning the hyperparameters for each data set.

Fraction undercovered. In various practical applications, we want to limit the number of classes that are severely undercovered, which we define as having a class-conditional coverage more than 10% below the desired coverage level. We define the fraction of undercovered classes (FracUnderCov)

.

0

10

20

30

40

AvgSize

ImageNet: softmax

.

ImageNet: APS

.

ImageNet: RAPS

Standard

Classwise

Clustered

.

0

10

20

30

40

AvgSize

CIFAR-100: softmax

.

CIFAR-100: APS

.

CIFAR-100: RAPS

.

0

10

20

30

40

AvgSize

Places365: softmax

.

Places365: APS

.

Places365: RAPS

10

20 30 4050

75

100

150

navg

0

10

20

30

40

AvgSize

iNaturalist: softmax

10

20 30 4050

75

100

150

navg

iNaturalist: APS

10

20 30 4050

75

100

150

navg

iNaturalist: RAPS

metric as:

recalling that ˆ c y is the empirical class-conditional coverage for class y . Figure 6 plots FracUnderCov for all experimental settings. Comparing to the CovGap plots in Figure 2 and Figure 4, we see that the trends in FracUnderCov generally mirror the trends in CovGap. However, FracUnderCov is a much noisier metric, as evidenced by the large error bars. Another difference is that FracUnderCov as a metric is unable to penalize uninformatively large set sizes. This is best seen in the performance of CLASSWISE on iNaturalist: for every score function, CLASSWISE has very low FracUnderCov, but this is achieved by producing extremely large prediction sets, as shown in the bottom row of Figure 5. Meanwhile, CovGap is somewhat able to penalize this kind of behavior since unnecessarily large set sizes often lead to overcoverage, and CovGap penalizes overcoverage.

C.3 Auxiliary randomization

The conformal methods in the main paper generate deterministic prediction sets, so running the method on the same input will always produce the same prediction set. These prediction sets are designed to achieve at least 1 -α marginal, class-conditional, or cluster-conditional coverage. In

.

0.00

0.02

0.04

0.06

0.08

FracUnderCov

ImageNet: softmax

.

ImageNet: APS

.

ImageNet: RAPS

Standard

Classwise

Clustered

.

0.00

0.02

0.04

0.06

0.08

0.10

FracUnderCov

CIFAR-100: softmax

.

CIFAR-100: APS

.

CIFAR-100: RAPS

.

0.00

0.02

0.04

0.06

0.08

FracUnderCov

Places365: softmax

.

Places365: APS

.

Places365: RAPS

10

20 30 4050

75

100

150

navg

0.00

0.05

0.10

0.15

0.20

FracUnderCov

iNaturalist: softmax

10

20 30 4050

75

100

150

navg

iNaturalist: APS

10

20 30 4050

75

100

150

navg

iNaturalist: RAPS

most practical situations, it is arguably undesirable to use non-deterministic or randomized prediction sets (say, if you are a patient, you would not want your doctor to tell you that your diagnoses change depending on a random seed). However, if one is willing to accept randomized prediction sets, then the conformal methods described in the main text can be modified to achieve exact 1 -α coverage.

Randomizing to achieve exact 1 -α coverage. Recall that the unrandomized STANDARD method used in the main paper uses

which yields a coverage guarantee of

under the assumption that the scores are distinct almost surely. The equality above holds because the event that Y test is included in C ( X test ) is equivalent to the event that s ( X test , Y test ) is one of the ⌈ ( N +1)(1 -α ) ⌉ smallest scores in the set containing the calibration points and the test point, and, by exchangeability, this occurs with probability exactly ⌈ ( N +1)(1 -α ) ⌉ / ( N +1) . By similar

reasoning, if we instead use

as our conformal quantile (note the added -1 in the numerator), then we would have

To summarize, using ˆ q results in at least 1 -α coverage, and using ˜ q results in less than 1 -α coverage. Thus, to achieve exact 1 -α coverage, we can randomize between using ˆ q and ˜ q . Let

be the amount by which the coverage using ˆ q overshoots the desired coverage level and let

be the amount by which the coverage using ˜ q undershoots the desired coverage level. Then, if we define the Bernoulli random variable

independent of everything else that is random, and set

then the prediction sets created using ˆ q rand will have exact 1 -α marginal coverage. The same idea translates to CLASSWISE and CLUSTERED methods (where we randomize ˆ q y and ˆ q ( m ) , respectively).

Figures 7, 8, and 9 display the CovGap, AvgSize, and FracUnderCov for the randomized versions of the conformal methods. Comparing against earlier plots, we observe that the performance of unrandomized and randomized STANDARD and CLUSTERED are essentially identical in terms of all three metrics. However, we cam see that randomized CLASSWISE exhibits a large improvement relative to unrandomized CLASSWISE in terms of CovGap and AvgSize. That said, the previouslygiven qualitative conclusions do not change, and the set sizes of randomized CLASSWISE are still too large to be practically useful.

.

2

4

6

8

CovGap

ImageNet: softmax

.

ImageNet: APS

.

ImageNet: RAPS

Randomized Standard

Randomized Classwise

Randomized Clustered

(Original) Classwise

.

2

4

6

8

CovGap

CIFAR-100: softmax

.

CIFAR-100: APS

.

CIFAR-100: RAPS

.

2

4

6

8

CovGap

Places365: softmax

.

Places365: APS

.

Places365: RAPS

10

20 30 4050

75

100

150

navg

2

4

6

8

CovGap

iNaturalist: softmax

10

20 30 4050

75

100

150

navg

iNaturalist: APS

10

20 30 4050

75

100

150

navg

iNaturalist: RAPS

.

0

10

20

30

40

AvgSize

ImageNet: softmax

.

ImageNet: APS

.

ImageNet: RAPS

Randomized Standard

Randomized Classwise

Randomized Clustered

(Original) Classwise

.

0

10

20

30

40

AvgSize

CIFAR-100: softmax

.

CIFAR-100: APS

.

CIFAR-100: RAPS

.

0

10

20

30

40

AvgSize

Places365: softmax

.

Places365: APS

.

Places365: RAPS

10

20 30 4050

75

100

150

navg

0

10

20

30

40

AvgSize

iNaturalist: softmax

10

20 30 4050

75

100

150

navg

iNaturalist: APS

10

20 30 4050

75

100

150

navg

iNaturalist: RAPS

.

0.00

0.02

0.04

0.06

0.08

FracUnderCov

ImageNet: softmax

.

ImageNet: APS

.

ImageNet: RAPS

Randomized Standard

Randomized Classwise

Randomized Clustered

(Original) Classwise

.

0.00

0.02

0.04

0.06

0.08

0.10

FracUnderCov

CIFAR-100: softmax

.

CIFAR-100: APS

.

CIFAR-100: RAPS

.

0.00

0.02

0.04

0.06

0.08

FracUnderCov

Places365: softmax

.

Places365: APS

.

Places365: RAPS

10

20 30 4050

75

100

150

navg

0.00

0.05

0.10

0.15

0.20

FracUnderCov

iNaturalist: softmax

10

20 30 4050

75

100

150

navg

iNaturalist: APS

10

20 30 4050

75

100

150

navg

iNaturalist: RAPS

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStandard conformal prediction provides a marginal coverage guarantee which is insufficient for many practical applications. Class-conditional conformal prediction is suitable for many applications, especially in $Y \\rightarrow X$ settings, e.g., image data. Achieving class-conditional coverage relies on learning a separate threshold for each label, and hence splitting the calibration set into label-dependent groups. This becomes prohibitive in classification problems with many classes. The current paper proposes a method based on clustering similar classes (which share similar conformal scores) and performing cluster-conditional conformal threshold calibration.\n\nSTRENGTHS:\nThe paper is well-written, and provided figures help the reader to follow the content easily. Relevant works and the limitations of the respective methods are highlighted. The proposed method is validated on a large number of large-scale public datasets, illustrating its general practical relevance.\n\nWEAKNESSES:\nTo avoid tautology, most of my concerns will be outlined in the questions box: those are related to the issues arising empty prediction sets and clustering based on the nonconformity scores. There is also a couple of minor typos:\n1. Line 26: $\\mathcal{C}(X_{test})\\subseteq \\mathcal{Y}$.\n2. Proposition 1: missing full stop in the equation between lines 156 and 157.\n\nFinally, a couple of stylistic suggestions:\n1. In Figure 1, it may be better to make sub-captions consistent (i.e., point out that the left su-plot refers to standard CP and the right one to Classwise CP). \n2. In Figure 2, it may be better to use different markers (in addition to different colors) for depicting the results of using different methods.\n3. I believe that [1] should also be cited and discussed.\n\n[1] \"Least Ambiguous Set-Valued Classifiers with Bounded Error Levels\". Sadinle et al., 2016\n\nQUESTIONS:\nI hope that the authors can provide answers to the following questions/concerns:\n\n1. One of the issues with class-conditional CP which has been pointed out in [1] is that there could be cases when the resulting prediction sets are empty (the authors also proposed several ways to handle this issue). Does this issue also apply to the proposed method?\n\n2. I am concerned regarding clustering based on nonconformity scores. Is it possible to (empirically) demonstrate that (even in the most straightforward cases) such classes form \"meaningful\"/\"interpretable\" clusters if nonconformity scores are used as features? Consider CIFAR-100 dataset as an example: class \"apples\" is \"closer\" to the class \"mushrooms\" than it is to the class \"rocket\". However, suppose those happened to be assigned to the same cluster. In that case, the coverage guarantee reads as \"among images with apples, mushrooms, and rockets, the true label is contained in the prediction set with probability at least $1-\\alpha$\" and may not be that interpretable.\n\n3. For many datasets, there is a known taxonomy/hierarchy of classes, e.g., for CIFAR-100, even though there are 100 classes, these are grouped into 20 superclasses. Such taxonomies can also be used to convert the low-data class-conditional calibration step to a high-data group-conditional calibration step, while resulting in interpretable clusters/groups. Even if such taxonomy is not provided, one can still try to come up with one that guarantees that there are \"enough\" calibration points in each group. What are the advantages of the proposed method over this strategy?\n\n[1] \"Least Ambiguous Set-Valued Classifiers with Bounded Error Levels\". Sadinle et al., 2016\n\nLIMITATIONS:\nThe current work is a part of methodological research and does not have a potential negative societal impact. Overall, I hope that the authors can better highlight the downsides of the proposed method (e.g., grouping based on existing dataset taxonomy).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article studies the clustered conformal prediction, mitigating the issues in the standard conformal and class-wise conformal, by grouping some similar classes together.\n\nSTRENGTHS:\n* The authors proposed clustered conformal prediction to strike the balance between marginal coverage and class-wise coverage for the setting of many classes with limited data per class. \n* When calibration data is limited, the empirical results show that the proposed method achieves a relatively small coverage gap according to the designed metric (CovGap), effectively balancing marginal coverage and class-wise coverage.\n\nWEAKNESSES:\n* The authors need to specify the low-data scenario (at least on the calibration data from the perspective of the experiments) in the title, otherwise, there are other works related to many classes. Moreover, from the main manuscript, it looks more like cluster-conditional conformal instead of class-conditional conformal in the title.\n\n* The setting and the numerical study somehow contradict each other. Now that the authors used deep learning to train the score function, it does not make sense that there is a limited calibration dataset in practice. If that, why don’t allocate some from the training data? \n\n* There is no sensitivity analysis on the clustering procedure. What if different schemes are used for choosing $M$?  How does $M$ (e.g., $M < K$ or $M > K$) affect the results? \n\n* Using CovGap as a metric might obscure the issue of coverage for each class from methods, e.g., the standard and the proposed method. The results are difficult to say without explicitly showing the coverage for each class. For example, class-wise conformal has high coverage than $1-\\alpha$ with the larger gap, but the proposed method could return the coverage below $1-\\alpha$ with a smaller gap.\n\nQUESTIONS:\n* The strikingly large prediction set of class-wise conformal is due to the threshold set as in Line 115, but why don’t we set it as the largest score in the calibration data instead of the infinity? There might be a loss of coverage, but it is unclear how large it is in these empirical results, and whether or not this kind of loss is tolerable, especially when the other two competing methods may also lose the coverage for the individual class.\n\n* Line 149: It seems the idea of finding the threshold for the null cluster is the same as that for the marginal coverage. Now that you criticized the standard conformal’s overall coverage ability, why not merge some clusters until you get the desired sample size instead of merging all clusters in the proposed method?\n\n* Line 176: The authors also notice the issue of class imbalance may affect the clustering, then can we up-sample the minor classes to mitigate the issue mentioned in Line 115 and then follow the class-conditional conformal instead of the proposed clustered conformal? Or what if you do the clustering on the original dataset to obtain the desired sample size, and then use class-conditional conformal?\n\nLIMITATIONS:\nPlease see the above two parts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to achieve class-conditional coverage (in the setting of conformal prediction) for multiclass classification problems, in particular for tasks with large label spaces. Previous techniques either provide no class-wise guarantees or tend to produce conservative prediction sets due to lack of data.\n\nThe key insight here is that non-conformity scores from different classes may be grouped together if their class-conditional score distributions are similar, since the $(1- \\alpha)$-quantile value to achieve desired coverage would be the same across these classes. Essentially, we can extrapolate about a class with low amounts of data by simply using data from other classes with a similar score distribution. \n\nThis work is particularly applicable in settings where there is a large number of classes (and thus higher likelihood of small amount of data for some of these classes) and / or high class imbalance.\n\nSTRENGTHS:\n•\tFirstly, I found this paper particularly well-written and structured; was written to make the discussion quite intuitive and easy to follow. \n\n•\tMethod is a natural solution in settings where there is an underlying structure / similarity in how the base model performs on certain groups of classes. \n\n•\tStrikes a balance between getting meaningful prediction sets (not overly conservative) and getting low variance in class-wise coverage rates.\n\n•\tEmpirically, the proposed method performs quite well, performing on par or better than the baseline methods for the studied metrics.\n\nWEAKNESSES:\n•\tThis performance of CLUSTERED seems to be essentially a data and model-driven problem. If there are different classes with similar score distribution, it improves performance.  Is there a possibility of artificially inducing clusters, if such a similarity doesn’t exist? \n\n•\tThe CovGap metric seems to penalize both overcoverage and undercoverage without distinguishing between the two. It would be nice to see how much each of the methods overcover and undercover (classwise) separately. For example, in Table 2 I would expect that the high CovGap values for CLASSWISE is mostly due to overcoverage, while for STANDARD it is probably undercovering and overcovering in equal measure, but this metric hides that potential distinction. \n\n•\tMinor typo: use of $h$ in line 146, equation after line 146 and in Proposition 1 should be $\\hat{h}$.\n\nQUESTIONS:\n•\tThough equation (2) implies equation (1), in general we may not achieve the guarantee of (2) as you have mentioned in Section 2.2, so the CLUSTERED algorithm may not necessarily achieve marginal coverage. Do you see this method as more of on a continuum with STANDARD and CLASSWISE, with each of them having situations where they work better, especially since CLUSTERED guarantees are not as consistent as the others? \n\n•\tClarification about the usage of Algorithm CLASSWISE: when generating a prediction set for a new input, to choose a conformal predictor to use, we need to know the true label, which we do not. So which of the K predictors is chosen? Do we run all of them and choose the most conservative quantile value?\n\nLIMITATIONS:\n•\tThe theoretical guarantee on class-conditional coverage depends on the value of $\\epsilon$ defined in Proposition 3 which can vary based on the actual score distributions across classes, the value of $m$ used and the clustering algorithm used, so there may be concern on achieving close to $(1 - \\alpha)$-coverage performance in general, although in all experiments, the method seems to work quite well.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a method for conformal prediction specially addressed for the case where there is few avaliable data in some of the classes. To address this problem, the proposed method performs a clustering of classes with similar score distributions with the goal of increasing the data used to select the set of classes for a given sample. Experimental results on standard image datasets with a large number of classes with different settings of data availability are reported comparing the proposed method with other standard approaches for conformal prediction.\n\nSTRENGTHS:\n1. The paper addresses the problem of conformal prediction with a large number of classes and with few data per classe, and proposes a new method specifically designed for this setting. The method is well motivated and formalized.\n2. Extensive experiments are performed on different datasets and different settings of data availability, comparing the proposed method with standard and classwise approaches. Results show that the proposed method achieves a better average coverage. \n3. The paper include useful practical recommendations about when to use each of the conformal prediction approaches.\n\nWEAKNESSES:\n1. One of the motivating points of the paper is that, in some cases average coverage can be good enough, but specific coverage for some underepresented classes can be very low. Results show that average coverage improves with the proposed method, but there is not any discussion or results about the specific coverage for particular classes, specially those with few data. Therefore, we cannot validate the original motivation of the paper. Furthermore, as fas as I understand, classes with few data will most probably assigned to the null cluster and then, those classes will follow the standard approach, and there will be no difference between the proposed method and the standard one for these classes. \n2. Although the average coverage improves, the AvgSize measure is, in general worse than with the standard method. \n3. Some technical details could be better explained or motivated.\n- In section 2.1, what do you exactly mean by \"scores for all classes in a given cluster are exchangeable? That score distributions are equal?\n- The k-means clustering algorithm is very dependent on the selected number of clusters. Although there is a explanation on how the number of cluster is defined, wouldn't it be better to use an adaptive clustering algorithm where the number of clusters can be automatically determined?\n- The representation used to perform the clustering is based on a few discrete number of quantile scores. Wouldn't it be better to use a larger and more dense number of quantile scores that could give a better approximation of the distribution of score values?\n\n3. Related work is very oriented to describe on which types of data conformal prediction has been applied. I miss a more technical discussion describing the different techniques and methods used in the existing works.\n\nQUESTIONS:\nSee above, specially questions regarding performance on specific classes with few data. \n\n-----------------------------\n\nI have read the rebuttal and it has clarified most of the questions I had about the paper, specially those related with classes with few samples and technical issues. After reading all comments and discussion, I increase my rating to weak accept.\n\nLIMITATIONS:\nThe authors discuss on the limitations of the method and no potential negative impacts are foreseen", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}