SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The test results of the Deep Ensemble and SAT ensemble on clean SVHN are shown in Table 8. It is not surprising that the AURCs of the Deep Ensemble and SAT ensemble are significantly lower on the clean SVHN than the original SVHN. Furthermore, on the clean SVHN, when the number of members is 5, the AURC of the Deep Ensemble is lower than that of SAT ensemble. Combined with results on the original SVHN, where the AURC of the Deep Ensemble is higher than that of SAT ensemble, we conclude that label noise in SVHN is why the Deep Ensemble has a higher AURC than SAT ensemble. In other words, label noise is why the Deep Ensemble performs worse in selective classification than SAT ensemble on SVHN.
In summary, by experiments, we show that the Deep Ensemble is not as robust to label noise as SAT ensemble, and label noise in SVHN is why the Deep Ensemble is not as good as SAT ensemble on SVHN. We construct the clean SVHN, which is SVHN without some mislabeled samples. On the clean SVHN, we compare the Deep Ensemble with SAT ensemble and find that the Deep Ensemble is superior to SAT ensemble in selective classification performance. Combined with former experimental results, we conclude that label noise in SVHN is why the Deep Ensemble is inferior to SAT on SVHN.
Considering the experimental results on the clean SVHN and previous experimental results on CIFAR-10 and CIFAR-100 (see Table 8 and Table 1 of the main text), the Deep Ensemble is superior to SAT
Table 8: AURC/10 $^{-4}$  of Deep Ensemble and SAT ensemble on the clean SVHN
Dataset	#Member	Deep Ensemble	SAT Ensemble
clean SVHN	1	12.3	7.8
2	8.2	7.1
3	7.3	6.8
4	6.8	6.8
5	6.4	6.8
ensemble in selective classification on clean image classification datasets, Thus, Deep Ensemble is the state-of-the-art selective classification method on clean image classification datasets, but is not as robust to label noise as SAT ensemble.
E The Lower Bound of Maximum  $\phi_0$  in Theorem 1
This section discusses the lower bound of maximum  $\phi_0$  mentioned in Theorem 1. We aim to calculate the maximum  $\phi_0$ 's lower bound without training an ensemble (otherwise, we can measure it directly on the ensemble).
Optimization Problem. To calculate the lower bound of maximum  $\phi_0$ , we need to solve the following optimization problem
$$
\min  _ {t, t _ {E}} \phi_ {*} (t) \text {s . t .} \phi_ {E} \left(t _ {E}\right) \geq \phi_ {*} (t) \tag {30}
$$
$$
R _ {E} (t _ {E}) <   R _ {*} (t).
$$
Using Lemma 2 and  $\phi_E(t) \geq P(\kappa_E(x) \geq t, a = 0) = P(\kappa_*(x) \geq t, a = 0)$ , we strengthen the constraints of (30) to obtain a looser lower bound of maximum  $\phi_0$ :
$$
\max  _ {t _ {*}, t _ {E}} \phi_ {*} (t _ {*}) \text {s . t .} P (\kappa_ {*} (x) \geq t _ {E}, a = 0) \geq \phi_ {*} (t _ {*})
$$
$$
R _ {*} (t _ {E} | a = 0) + \frac {\gamma \cdot (1 - t _ {E}) ^ {M}}{\gamma \cdot (1 - t _ {E}) ^ {M} + P (\kappa_ {*} (x) \geq t _ {E} , a = 0)} <   R _ {*} (t _ {*}).
$$
For the convenience of solving this optimization problem, we further strengthen the first constraint to obtain a looser lower bound:
$$
\max  _ {t _ {*}, t _ {E}} \phi_ {*} (t _ {*}) \text {s . t .} P (\kappa_ {*} (x) \geq t _ {E}, a = 0) = \phi_ {*} (t _ {*}) \tag {31}
$$
Algorithm. We design Algorithm 1 to search for the solution to (31), where oracle tells whether a sample is definite, and this oracle can be implemented by an ensemble with  $M'$  ( $M' \ll M$ ) members. Since  $t_E$  is determined by  $t_*$  according to the first constraint of (31), (31) can be reduced to a one-dimensional search problem. Our algorithm adopts a binary search for efficiency, although this method might provide a suboptimal solution. Because  $\phi_*(t_*)$  is a non-increasing function of  $t_*$ , Algorithm first search for the minimum  $t_*$  by binary search. The procedure of Algorithm 1 in each iteration of the binary search is as follows.
1. Given current  $t_*$ , Algorithm 1 determines  $t_E$  using SEARCHFORTAUENS (see Algorithm 2), a procedure that searches for  $t_E \in [0,1]$  using binary search s.t.  $P(\kappa_*(x) \geq t_E, a = 0) = \phi_*(t_*)$ . Note that  $t_E$  might not exist if  $t_*$  is so low that  $\phi_*(t_*) > P(a = 0) = \sup_{t_E \in [0,1]} P(\kappa_*(x) \geq t, a = 0)$ . This problem will be addressed shortly.
2. Algorithm 1 exams whether  $t_E$  exists. If  $t_E$  exists, Algorithm 1 then examines whether the second constraint of (31) holds for current  $t_*$  and  $t_E$ , which is implemented by VERIFY-SeCONDCONSTRAINT (see Algorithm 3).
Algorithm 1 A Lower Bound of Maximum  $\phi_0$ .
Input:  $\kappa_{*}(\cdot), B$ , the test set  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ ; the oracle  $\Omega : \mathbb{X} \to \{0,1\}$  that tells whether a sample is definite; the number of member models  $M$ .  
Output: An lower bound of maximum  $\phi_0$  mentioned in Theorem 1  
left = 0  
right = 1  
 $\epsilon = 10^{-9}$   
while right - left >  $\epsilon$  do  
 $t_* = (left + right)/2$ $t_E = \text{SEARCHFORTAUENS}(t_*, \kappa_*, \mathcal{D}, \Omega)$   
if  $t_E$  is not None and  $\text{VERIFYSECONDCONSTRAINT}(\kappa_*, t_*, t_E, \mathcal{D}, \Omega, M, B)$  is True then  
 $\text{right} = t_*$   
else  
 $\text{left} = t_*$   
end if  
end while  
opt = (left + right)/2  
return  $\frac{1}{N} \sum_{i=1}^{N} \mathbf{1}_{\kappa_*(x_i)} \geq opt$
Algorithm 2 SEARCHFORTAUENS
Input:  $\kappa_{*}(\cdot)$ , the confidence threshold  $t_*$ ; the test set  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ ; the oracle  $\Omega : \mathbb{X} \to \{0, 1\}$  that tells whether a sample is definite.  
Output:  $t_E \in [0, 1]$  that satisfies the first constraint of (31) given  $t_*$ .  
 $\phi = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}_{\kappa_{*}(x_i) \geq t_*}$   
if  $\phi > \frac{1}{N} \sum_{i=1}^{N} \Omega(x_i)$  then  
return None  
end if  
left = 0  
right = 1  
 $\epsilon = 10^{-9}$   
while right - left >  $\epsilon$  do  
 $t_E = (\text{left} + \text{right}) / 2$   
if  $\frac{1}{N} \sum_{i=1}^{N} \mathbf{1}_{\kappa_{*}(x_i) \geq t_E} \cdot \Omega(x_i) < \phi$  then  
right =  $t_E$   
else  
left =  $t_E$   
end if  
end while  
return (left + right) / 2
3. If  $t_E$  exists and the second constraint holds, Algorithm 1 searches for a smaller  $t_*$  in the left half feasible area; otherwise, Algorithm 1 searches for a greater  $t_*$  in the right half feasible area.
Once the binary search completes, Algorithm 1 returns the coverage of minimum  $t_*$ .
Experiment. To show that Algorithm 1 works in reality, we run this algorithm in the same setting as Section 6 of the main text. In this experiment,  $M = 5$ , the oracle is implemented by another ensemble with two member models and outputs True if and only if the  $S < 10^{-3}$ . Note that it is difficult to estimate  $B$ , because: 1. we need to train an ensemble with  $M$  models to estimate  $B$ , which is costly; 2. the domain of  $p(\pi_1^k, \dots, \pi_M^k | a = 1)$  is of high dimension, so the observed data points are sparse in this domain, which makes the estimation of  $B$  more difficult. Thus, we do not estimate  $B$  but try several hypothetical values of  $B$  to see at what  $B$  the lower bound of maximum  $\phi_0$  is big.
With different  $B$ s, we obtain different lower bounds of maximum  $\phi_0$  as Table 9 shows. We can see that when the order of magnitude of  $B$  is not too big, the lower bound of maximum  $\phi_0$  is large and stable, which makes it possible for our algorithm to be used in practical applications. This result also
Algorithm 3 VERIFYSECONDCONSTRAINT
Input:  $\kappa_{*}(\cdot)$ ; the confidence threshold  $t_*$ ;  $t_E$ ; the test set  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$ ; the oracle  $\Omega : \mathbb{X} \to \{0, 1\}$  that tells whether a sample is definite; the number of member models  $M$ ; and  $B$ .
Output: True if and only if  $t_*$  and  $t_E$  satisfy the second constraint of (31).
$$
\begin{array}{l} \gamma = B M ^ {M - 1} \frac {1}{N} \sum_ {i = 1} ^ {N} \left[ 1 - \Omega \left(x _ {i}\right) \right] \\ \text{leftHandSide} = \frac{\sum_{i = 1}^{N}\mathbf{1}_{f(x_{i};\theta)}\neq y_{i}\cdot\Omega(x_{i})\cdot\mathbf{1}_{\kappa_{*}(x_{i})\geq t_{E}}}{\sum_{i = 1}^{N}\Omega(x_{i})\cdot\mathbf{1}_{\kappa_{*}(x_{i})\geq t_{E}}} +\frac{\gamma(1 - t_{E})^{M}}{\gamma(1 - t_{E})^{M} + \frac{1}{N}\sum_{i = 1}^{N}\Omega(x_{i})\cdot\mathbf{1}_{\kappa_{*}(x_{i})\geq t_{E}}} \\ r i g h t H a n d S i d e = \frac {\sum_ {i = 1} ^ {N} \mathbf {1} _ {f (x _ {i} ; \theta) \neq y _ {i}} \cdot \mathbf {1} _ {\kappa_ {*} (x _ {i}) \geq t _ {*}}}{\sum_ {i = 1} ^ {N} \mathbf {1} _ {\kappa_ {*} (x _ {i}) \geq t _ {*}}} \\ \text {r e t u r n} 1 _ {\text {l e f t H a n d S i d e}} <   \text {r i g h t H a n d S i d e} \\ \end{array}
$$
Table 9: The lower bound of maximum  ${\phi }_{0}\left( \% \right)$  .
B	1	10	102	103	104	105	106	107	108	109	1010	1011	1012	1013	1014
CIFAR-10	73.72	73.72	73.72	73.70	73.62	73.47	72.90	70.50	59.90	0	0	0	0	0	0
SVHN	77.51	77.51	77.51	77.51	77.44	77.23	76.54	68.86	0	0	0	0	0	0	0
CIFAR-100	32.03	32.03	32.00	32.00	31.93	31.79	31.50	30.45	26.91	17.96	10.36	0	0	0	0
MNLI	58.37	57.62	56.48	53.94	49.62	38.73	0	0	0	0	0	0	0	0	0
QNLI	65.31	65.31	65.31	64.34	62.31	59.20	53.07	37.16	0	0	0	0	0	0	0
MRPC	81.37	81.37	81.37	81.37	81.37	81.13	80.64	79.41	78.43	41.67	41.67	0	0	0	0
indicates the relationship between the ensemble's diversity and its selective classification performance. Since an ensemble with a smaller  $B$  seems to have more diversity over ambiguous samples, the result in Table 9 suggests that as long as the ensemble has enough diversity over ambiguous samples, the ensemble is guaranteed to have a lower selective risk than the member model under a considerable range of coverage.
F Distributions of Ambiguities
See Figure 9 for the results.
(f)
Figure 9: The distribution of ambiguity on different tasks. The pattern of these distributions is clear. For each dataset, the distribution concentrates on a small interval near zero (which we refer to as low-ambiguity samples), and has a long tail (which we refer to as high-ambiguity samples).