SlowGuess/NeurIPS_2023_OCR huggingface dataset:
iNaturalist. This data set has class labels of varying specificity. For example, at the species level, there are 6414 classes with 300 examples each (290 training examples and 10 validation examples) and a total of 10000 classes with at least 150 examples. We instead work at the family level, which groups the species into 1103 classes. We randomly sample  $50\%$  of the data for fine-tuning, and we use the remaining for calibrating and validating our procedures. After training for one epoch, the validation accuracy is  $69\%$ .
However, due to high class imbalance and sampling randomness, some classes have insufficient validation samples, so we filter out classes with fewer than 250 validation examples, which leaves us with 633 classes. The entries of the softmax vectors that correspond to rare classes are removed and the vector is renormalized to sum to one.
B.3 Measuring class balance in Table 1
The class balance metric in Table 1 is defined as the number of examples in the rarest  $5\%$  of classes divided by the expected number of examples if the class distribution were perfectly uniform. This metric is bounded between 0 and 1, with lower values denoting more class imbalance. We compute this metric using  $D_{\mathrm{fine}}^c$ .
B.4 Choosing clustering parameters
To choose  $\gamma \in [0,1]$  and  $M \geq 1$  for CLUSTERED, as described briefly in Section 3.1, we employ two intuitive heuristics. We restate these heuristics in more detail here.
- First, to distinguish between more clusters (or distributions), we need more samples from each distribution. As a rough guess, to distinguish between two distributions, we want at least four samples per distribution; to distinguish between five distributions, we want at least ten samples per distribution. In other words, we want the number of clustering examples per class to be at least twice the number of clusters. This heuristic can be expressed as
$$
\gamma \tilde {n} \geq 2 M, \tag {3}
$$
where  $\gamma \tilde{n}$  is the expected number of clustering examples for the rarest class that is not assigned to the null cluster.
- Second, we want enough data for computing the conformal quantiles for each cluster. Specifically, we seek at least 150 examples per cluster on average. This heuristic can be expressed as
$$
(1 - \gamma) \tilde {n} \frac {K}{M} \geq 1 5 0, \tag {4}
$$
where  $K / M$  is the average number of classes per cluster and  $(1 - \gamma)\tilde{n}$  is the expected number of proper calibration examples for the rarest class not assigned to the null cluster.
Changing the inequalities of (3) and (4) into equalities and solving for  $\gamma$  and  $M$  yields
$$
M = \frac {\gamma \tilde {n}}{2} \qquad \text {a n d} \qquad \gamma = \frac {K}{K + 7 5}.
$$
Varying the clustering parameters. As sensitivity analysis, we examine the performance of CLUSTERED across a wide range of values for the tuning parameters  $\gamma$  and  $M$ . As the heatmaps in Figure 3 confirm, the performance of CLUSTERED is not particularly sensitive to the values of these parameters. When  $n_{\mathrm{avg}} = 10$ , the heuristic chooses  $\gamma = 0.89$  and  $M = 4$ , meanwhile, when  $n_{\mathrm{avg}} = 50$ , the heuristic chooses  $\gamma \in [0.88, 0.92]$  and  $M \in [7, 12]$  (since the calibration data set is randomly sampled, and  $\gamma$  and  $M$  are chosen based on the calibration data set, there is randomness in the chosen values). However, there are large areas surrounding these values that would yield similar performance. We observe that the heuristics do not always choose the parameter values that yield the lowest CovGap. The heatmaps reveal that the optimal parameter values are dependent not only on data set characteristics, but also on the score function. Future work could be done to extract further performance improvements by determining a better method for choosing  $\gamma$  and  $M$ .
C Additional experimental results
We present additional experimental results in this section. As in the main text, the shaded regions in plots denote  $\pm 1.96$  times the standard errors.
C.1 RAPS CovGap results
Figure 4 shows the CovGap on all data sets when we use RAPS as our score function.
C.2 Additional metrics
Average set size. To supplement Table 2 from the main text, which reports AvgSize for select values of  $n_{\mathrm{avg}}$ , Figure 5 plots AvgSize for all values of  $n_{\mathrm{avg}}$  that we use in our experimental setup. Note
Figure 3: Average class coverage gap on ImageNet for  $n_{\mathrm{avg}} \in \{10,50\}$ , using the softmax, APS, and RAPS scores, as we vary the clustering parameters. Each entry is averaged across 10 random splits of the data into calibration and validation sets.
Figure 4: Average class coverage gap for ImageNet, CIFAR-100, Places365, and iNaturalist, using the RAPS score, as we vary the average number of calibration examples per class.
that RAPS sharply reduces AvgSize relative to APS on ImageNet and also induces a slight reduction for the other three data sets. This asymmetric reduction is likely due to the fact that the RAPS hyperparameters, which control the strength of the set size regularization, were tuned on ImageNet. The set sizes of RAPS on other data sets could likely be improved by tuning the hyperparameters for each data set.
Fraction uncovered. In various practical applications, we want to limit the number of classes that are severely uncovered, which we define as having a class-conditional coverage more than  $10\%$  below the desired coverage level. We define the fraction of uncovered classes (FracUnderCov)
Figure 5: Average set size for ImageNet, CIFAR-100, Places365, and iNaturalist, for the softmax, APS, and RAPS scores, as we vary the average number of calibration examples per class.
metric as:
$$
\operatorname {F r a c} \text {U n d e r C o v} = \frac {1}{| \mathcal {Y} |} \sum_ {y = 1} ^ {| \mathcal {Y} |} \mathbb {1} \left\{\hat {c} _ {y} \leq 1 - \alpha - 0. 1 \right\},
$$
recalling that  $\hat{c}_y$  is the empirical class-conditional coverage for class  $y$ . Figure 6 plots FracUnderCov for all experimental settings. Comparing to the CovGap plots in Figure 2 and Figure 4, we see that the trends in FracUnderCov generally mirror the trends in CovGap. However, FracUnderCov is a much noisier metric, as evidenced by the large error bars. Another difference is that FracUnderCov as a metric is unable to penalize uninformatively large set sizes. This is best seen in the performance of CLASSWISE on iNaturalist: for every score function, CLASSWISE has very low FracUnderCov, but this is achieved by producing extremely large prediction sets, as shown in the bottom row of Figure 5. Meanwhile, CovGap is somewhat able to penalize this kind of behavior since unnecessarily large set sizes often lead to overcoverage, and CovGap penalizes overcoverage.
C.3 Auxiliary randomization
The conformal methods in the main paper generate deterministic prediction sets, so running the method on the same input will always produce the same prediction set. These prediction sets are designed to achieve at least  $1 - \alpha$  marginal, class-conditional, or cluster-conditional coverage. In
Figure 6: Fraction of severely undercovered classes for ImageNet, CIFAR-100, Places365, and iNaturalist, using the softmax, APS, and RAPS scores, as we vary the average number of calibration examples per class.
most practical situations, it is arguably undesirable to use non-deterministic or randomized prediction sets (say, if you are a patient, you would not want your doctor to tell you that your diagnoses change depending on a random seed). However, if one is willing to accept randomized prediction sets, then the conformal methods described in the main text can be modified to achieve exact  $1 - \alpha$  coverage.
Randomizing to achieve exact  $1 - \alpha$  coverage. Recall that the unrandomized STANDARD method used in the main paper uses
$$
\hat {q} = \mathrm {Q u a n t i l e} \bigg (\frac {\lceil (N + 1) (1 - \alpha) \rceil}{N}, \{s _ {i} \} _ {i = 1} ^ {N} \bigg),
$$
which yields a coverage guarantee of
$$
\mathbb {P} (Y _ {\mathrm {t e s t}} \in \mathcal {C} (X _ {\mathrm {t e s t}}) \mid Y _ {\mathrm {t e s t}} = y) = \frac {\lceil (N + 1) (1 - \alpha) \rceil}{N + 1} \geq 1 - \alpha ,
$$
under the assumption that the scores are distinct almost surely. The equality above holds because the event that  $Y_{\mathrm{test}}$  is included in  $\mathcal{C}(X_{\mathrm{test}})$  is equivalent to the event that  $s(X_{\mathrm{test}},Y_{\mathrm{test}})$  is one of the  $\lceil (N + 1)(1 - \alpha)\rceil$  smallest scores in the set containing the calibration points and the test point, and, by exchangeability, this occurs with probability exactly  $\lceil (N + 1)(1 - \alpha)\rceil /(N + 1)$ . By similar
reasoning, if we instead use
$$
\tilde {q} = \operatorname {Q u a n t i l e} \left(\frac {\lceil (N + 1) (1 - \alpha) \rceil - 1}{N}, \{s _ {i} \} _ {i = 1} ^ {N}\right)
$$
as our conformal quantile (note the added  $-1$  in the numerator), then we would have
$$
\mathbb {P} \left(Y _ {\text {t e s t}} \in \mathcal {C} \left(X _ {\text {t e s t}}\right) \mid Y _ {\text {t e s t}} = y\right) = \frac {\lceil (N + 1) (1 - \alpha) \rceil - 1}{N + 1} <   1 - \alpha .
$$
To summarize, using  $\hat{q}$  results in at least  $1 - \alpha$  coverage, and using  $\tilde{q}$  results in less than  $1 - \alpha$  coverage. Thus, to achieve exact  $1 - \alpha$  coverage, we can randomize between using  $\hat{q}$  and  $\tilde{q}$ . Let
$$
b = \frac {\lceil (N + 1) (1 - \alpha) \rceil}{N + 1} - (1 - \alpha)
$$
be the amount by which the coverage using  $\hat{q}$  overshoots the desired coverage level and let
$$
c = (1 - \alpha) - \frac {\lceil (N + 1) (1 - \alpha) \rceil - 1}{N + 1}
$$
be the amount by which the coverage using  $\tilde{q}$  undershoots the desired coverage level. Then, if we define the Bernoulli random variable
$$
B \sim \mathrm {B e r n} \bigg (\frac {c}{b + c} \bigg),
$$
independent of everything else that is random, and set
$$
\hat {q} _ {\mathrm {r a n d}} = B \hat {q} + (1 - B) \tilde {q}
$$
then the prediction sets created using  $\hat{q}_{\mathrm{rand}}$  will have exact  $1 - \alpha$  marginal coverage. The same idea translates to CLASSWISE and CLUSTERED methods (where we randomize  $\hat{q}^y$  and  $\hat{q}(m)$ , respectively).
Figures 7, 8, and 9 display the CovGap, AvgSize, and FracUnderCov for the randomized versions of the conformal methods. Comparing against earlier plots, we observe that the performance of unrandomized and randomized STANDARD and CLUSTERED are essentially identical in terms of all three metrics. However, we can see that randomized CLASSWISE exhibits a large improvement relative to unrandomized CLASSWISE in terms of CovGap and AvgSize. That said, the previously-given qualitative conclusions do not change, and the set sizes of randomized CLASSWISE are still too large to be practically useful.
Figure 7: Average class coverage gap for randomized methods on ImageNet, CIFAR-100, Places365, and iNaturalist, for the softmax, APS, and RAPS scores, as we vary the average number of calibration examples per class. The unrandomized original CLASSWISE method is also plotted for comparison purposes.
Figure 8: Average set size for randomized methods on ImageNet, CIFAR-100, Places365, and iNaturalist, for the softmax, APS, and RAPS scores, as we vary the average number of calibration examples per class. The unrandomized original CLASSWISE method is also plotted for comparison purposes.
Figure 9: Fraction of severely undercovered classes for randomized methods on ImageNet, CIFAR-100, Pla-ces365, and iNaturalist, for the softmax, APS, and RAPS scores, as we vary the average number of calibration examples per class. The unrandomized original CLASSWISE method is also plotted for comparison purposes.