SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p \left(\pi_ {E} ^ {k} = t \mid a = 1\right) \leq M \int_ {[ M t - M + 1, 1 ] ^ {M - 1}} \mathrm {d} \pi_ {1} ^ {k} \dots \mathrm {d} \pi_ {M - 1} ^ {k} B _ {k} = M ^ {M} B _ {k} \cdot (1 - t) ^ {M - 1}. \tag {19}
$$
This inequality combined with Lemma 4 derives  $p(\kappa(x) = t | a = 1) \leq \sum_{k=1}^{K} p(\pi_E^k = t | a = 1) \leq M^M(1 - t)^{M-1} \cdot \sum_{k=1}^{K} B_k$ , which is equivalent to the conclusion of this lemma since  $B = \sum_{k=1}^{K} B_k$ .
A.2 Proof of Equation (2) of the Main Text (Preparing for Lemma 2)
According to the definition of selective risk, we have
$$
\begin{array}{l} R (f, \kappa ; t) \\ = \mathbb {E} _ {(x, y) \sim \mathcal {D}} \left[ \mathbf {1} _ {y \neq f (x)} | \kappa (x) \geq t \right] \\ = P (y \neq f (x) | \kappa (x) \geq t) \\ = P (y \neq f (x), a = 0 | \kappa (x) \geq t) + P (y \neq f (x), a = 1 | \kappa (x) \geq t) \\ = P (y \neq f (x) | a = 0, \kappa (x) \geq t) P (a = 0 | \kappa (x) \geq t) + P (y \neq f (x) | a = 1, \kappa (x) \geq t) P (a = 1 | \kappa (x) \geq t). \\ \end{array}
$$
According to the definition of conditional selective risk, the equation above can be rewritten as
$$
\begin{array}{l} R (f, \kappa ; t | a = 0) P (a = 0 | \kappa (x) \geq t) + R (f, \kappa ; t | a = 1) P (a = 1 | \kappa (x) \geq t) \\ = R (f, \kappa ; t | a = 0) [ 1 - P (a = 1 | \kappa (x) ] \geq t) + R (f, \kappa ; t | a = 1) P (a = 1 | \kappa (x) \geq t) \\ = R (f, \kappa ; t | a = 0) + P (a = 1 | \kappa (x) \geq t) \cdot \left[ R (f, \kappa ; t | a = 1) - R (f, \kappa ; t | a = 0) \right] \\ = R (f, \kappa ; t | a = 0) + \lambda (f, \kappa ; t) \cdot P (a = 1 | \kappa (x) \geq t), \tag {20} \\ \end{array}
$$
where  $\lambda(f, \kappa; t) \coloneqq R(f, \kappa; t | a = 1) - R(f, \kappa; t | a = 0)$ . Applying (20) to the ensemble and considering  $\lambda(f_E, \kappa_E; t) \leq 1$ , we derive an upper bound of the ensemble's selective risk
$$
R _ {E} (t) \leq R _ {E} (t | a = 0) + P (a = 1 | \kappa_ {E} (x) \geq t). \tag {21}
$$
Thus, Equation (2) of the main text is proved.
Proof. Step 1. We prove
$$
\lim  _ {t \rightarrow 1 ^ {-}} R _ {E} (t) <   \lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t). \tag {22}
$$
Firstly, we prove  $\lim_{t\to 1^{-}}R_E(t)\leq \lim_{t\to 1^{-}}R_*(t|a = 0)$ . According to Lemma 2, we have
$$
\begin{array}{l} \lim  _ {t \rightarrow 1 ^ {-}} R _ {E} (t) \leq \lim  _ {t \rightarrow 1 ^ {-}} \left[ R _ {*} (t | a = 0) + \frac {\gamma \cdot (1 - t) ^ {M}}{\gamma \cdot (1 - t) ^ {M} + P \left(\kappa_ {*} (x) \geq t , a = 0\right)} \right] \\ = \lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t | a = 0) + \lim  _ {t \rightarrow 1 ^ {-}} \frac {\gamma \cdot (1 - t) ^ {M}}{\gamma \cdot (1 - t) ^ {M} + P (\kappa_ {*} (x) \geq t , a = 0)} \\ = \lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t | a = 0) + \lim  _ {t \rightarrow 1 ^ {-}} \frac {- M \gamma (1 - t) ^ {M - 1}}{- M \gamma (1 - t) ^ {M - 1} - p \left(\kappa_ {*} (x) = t , a = 0\right)} \\ = \lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t | a = 0) \tag {23} \\ \end{array}
$$
where the second-to-last equality is due to L'Hospital's rule, and the last equality is due to  $\lim_{t\to 1^{-}}M\gamma (1 - t)^{M - 1} = 0$  and  $\lim_{t\to 1^{-}}p(\kappa_{*}(x) = t,a = 0) = P(a = 0)\lim_{t\to 1^{-}}p(\kappa_{*}(x) = t|a = 0) > 0$  (Assumption 3).
Secondly, we prove  $\lim_{t\to 1^{-}}R_{*}(t) > \lim_{t\to 1^{-}}R_{*}(t|a = 0)$ . According to (20), we have
$$
\lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t) = \lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t | a = 0) + \lim  _ {t \rightarrow 1 ^ {-}} \lambda \left(f _ {*}, \kappa_ {*}; t\right) \cdot \lim  _ {t \rightarrow 1 ^ {-}} P (a = 1 | \kappa_ {*} (x) \geq t). \tag {24}
$$
Using Bayes' rule and L'Hospital's rule, we have
$$
\begin{array}{l} \lim  _ {t \rightarrow 1 ^ {-}} P (a = 1 | \kappa_ {*} (x) \geq t) = \lim  _ {t \rightarrow 1 ^ {-}} \frac {P (\kappa_ {*} (x) \geq t | a = 1) P (a = 1)}{P (\kappa_ {*} (x) \geq t | a = 1) P (a = 1) + P (\kappa_ {*} (x) \geq t | a = 0) P (a = 0)} \\ = \lim  _ {t \rightarrow 1 ^ {-}} \frac {p \left(\kappa_ {*} (x) = t \mid a = 1\right) P (a = 1)}{p \left(\kappa_ {*} (x) = t \mid a = 1\right) P (a = 1) + p \left(\kappa_ {*} (x) = t \mid a = 0\right) P (a = 0)} \\ > 0, \tag {25} \\ \end{array}
$$
where the last inequality is due to Assumption 3. Applying this inequality along with  $\lim_{t\to 1^{-}}\lambda (f*,\kappa_{*};t) = \lim_{t\to 1^{-}}[R_{*}(t|a = 1) - R_{*}(t|a = 1)] > 0$  (Assumption 2) to (24), we have
$$
\lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t) > \lim  _ {t \rightarrow 1 ^ {-}} R _ {*} (t | a = 0). \tag {26}
$$
Combining (23) and (26), we derive (22).
Step 2. We prove the ensemble's confidence threshold and the member model's confidence threshold approach 1 when their coverage approach 0. Due to  $\phi_{*}(t) = P(\kappa_{*}(x)\geq t) = P(\kappa_{*}(x)\geq t|a = 0)P(a = 0) + P(\kappa_{*}(x)\geq t|a = 1)P(a = 1)$ , we derive
$$
\begin{array}{l} \frac {d \phi_ {*} (t)}{d t} = - p (\kappa_ {*} (x) = t | a = 0) P (a = 0) - p (\kappa_ {*} (x) = t | a = 1) P (a = 1) \\ \leq - p \left(\kappa_ {*} (x) = t \mid a = 0\right) P (a = 0). \tag {27} \\ \end{array}
$$
Because  $\lim_{t\to 1^{-}}p(\kappa_{*}(x) = t|a = 0) > 0$  (Assumption 3), there exists  $\delta_1 > 0$ , such that  $p(\kappa_{*}(x) = t|a = 0) > 0, \forall t\in (1 - \delta_1,1)$ . Combining this with (27), we have  $\frac{d\phi_{*}(t)}{dt} < 0, \forall t\in (1 - \delta_1,1)$  and thus  $\phi_{*}(t)$  is reversible on  $(1 - \delta_1,1)$ . Considering  $p(\kappa_E(x) = t|a = 0) = p(\kappa_{*}(x) = t|a = 0)$ , we derive  $\phi_E(t)$  is reversible on  $(1 - \delta_1,1)$  with the same reasoning as above.
Let  $\phi_{*}^{-1}$  and  $\phi_E^{-1}$  be the reverse functions of  $\phi_{*}$  and  $\phi_E$  on  $(1 - \delta_1, 1)$ , respectively. It is easy to see that  $\phi_{*}^{-1}$  is a continuous function, and  $\phi_{*}^{-1}(0) = 1$ . Therefore,
$$
\lim  _ {\phi \rightarrow 0 ^ {+}} \phi_ {*} ^ {- 1} (\phi) = 1. \tag {28}
$$
For the ensemble, we similarly have
$$
\lim  _ {\phi \rightarrow 0 ^ {+}} \phi_ {E} ^ {- 1} (\phi) = 1. \tag {29}
$$
Step 3. Combining (22), (28) and (29), we have  $R_{E}(\phi_{E}^{-1}(\phi)) < R_{*}(\phi_{*}^{-1}(\phi))$  when  $\phi \to 0^{+}$ , which is equivalent to the result of Theorem 1.
Table 1: Sizes of training sets, development sets, and test sets for each dataset used in experiments
Datasets	Training Set	Development Set	Test Set	Number of Classes
CIFAR-10	50.0k	10.0k	10
CIFAR-100	50.0k	10.0k	100
SVHN	73.3k	26.0k	10
MRPC	3.7k	0.4k	1.7k	2
QNLI	104.7k	5.5k	5.5k	2
MNLI	392.7k	9.8k (m)/ 9.8k(mm)	9.8k(m)/9.8k(mm)	3
Table 2: The value of  $o$  on each dataset
Dataset	CIFAR-10	SVHN	CIFAR-100	MRPC	QNLI	MNLI-(m)
o	2.20	2.60	4.60	1.80	1.60	2.80
B Details of Experiments
The experiments were conducted on multiple data sets of image classification and text classification. The image classification datasets are CIFAR-10, CIFAR-100, [29] and SVHN [30], whose image sizes are all  $32 \times 32 \times 3$  pixels. The datasets of text classification are MRPC [31], MNLI [32] and QNLI [33]. The task of MRPC is to judge whether two paragraphs of text are semantically equivalent. MNLI's task is to judge the inferential relationship between sentences (three categories). The task of QNLI is to determine whether a paragraph has the answer to a given question. The sizes of the training set, development set, and test set of each data set used in experiments are shown in Table 1. MNLI's development set and test set are divided into matched and mismatched parts. In the table, (m) represents matched, and (mm) represents mismatched. The matched parts are sampled from the same source as the training set, while the mismatched parts are sampled from different sources. Current selective classification only considers test samples from the same distribution as the training set, so only the matched parts are used in experiments. In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets. According to [8, 9], since CIFAR-10, CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly divided from corresponding test sets.
B.2 Model Implementations and Training Procedures
For image classification, the backbone model is VGG-16 [34] with Dropout [13], batch normalization [40]. It is trained in the same way as [9]. The model is optimized using SGD with an initial learning rate of 0.1 (the learning rate decays by half in every 25 epochs), the momentum of 0.9, weight decay of 0.0005, batch size of 128, and a total training epoch of 300. Data preprocessing includes data augmentation (random cropping and flip) and normalization. The implementations of the backbone model and data preprocessing are based on the official open-sourced implementation of SAT to ensure a fair comparison.
For text classification, the backbone model of selective classifiers is BERT-base [35]. Pretrained BERT-base is provided by the Huggingface Transformer Library [41]. It is trained/fine-tuned in the same way as [10], except on dataset MRPC. On QNLI and MNLI, the model is trained/fine-tuned using AdamW [42] for 3 epochs, with a learning rate of  $2 \times 10^{-5}$ , batch size of 32, and the maximum input sequence length of 128. On MRPC, the model is trained/fine-tuned for 10 epoch, with other settings the same as those on QNLI and MNLI. This unique setting of training epoch is due to the small number of samples in MRPC, which makes the training require more epochs to reach convergence on MRPC.
Figure 6: Comparison of the risk-coverage curves between the ensembles (including Deep Ensemble and baselines' ensembles) and their corresponding member models on each dataset.
B.3 Hyperparameters of Selective Classifiers
For the hyperparameter  $o$  of Gambler, we tune  $o$  on validation sets in the same way as [8]. The value of  $o$  on each dataset is listed in Table 2. For the hyperparameter  $\alpha$  of SAT, we set  $\alpha = 0.99$ , the same as [9]. For the hyperparameter  $\lambda$  of Reg-curr, we set  $\lambda = 0.05$ .
C Evaluation of Deep Ensemble
To show that Deep Ensemble is still one of the state-of-the-art, it is necessary to evaluate Deep Ensemble again on selective classification. The reasons are as follows. First, the original paper of Deep Ensemble only reports the accuracy given a confidence threshold, which is not a commonly used metric in selective classification. Second, recent work in selective classification excluded Deep Ensemble as a baseline due to its heavy computational cost [10, 43]. Therefore, no comparison between recent methods and Deep Ensemble are available.
To fill this gap, we report the comparison results of Deep Ensemble and other recent work in this section. We compare Deep Ensemble with three non-ensemble methods: Gambler, SAT, Reg-curr, and their corresponding ensembles. The ensemble method for a baseline is a uniformly-weighted soft voting of the classifiers and the confidence estimators:  $\pi_E(x) = \frac{1}{M}\sum_{i=1}^{M}\pi_i(x)$  and  $\kappa_E(x) = \frac{1}{M}\sum_{i=1}^{M}\kappa_i(x)$  if the baseline has separate classifier and confidence estimator (i.e., the case of Gambler and SAT); and otherwise, the ensemble method is the same as Deep Ensemble (i.e., the case of Reg-curr). The details of baselines' hyperparameters are provided in Appendix B. Figure 6 shows the risk-coverage curves of Deep Ensemble, baselines, and their corresponding ensembles on each dataset, where each ensemble consists of five member models. The result shows that the ensemble has consistently lower selective risks than the member model under (almost) any coverage across each baseline on each dataset, so we focus on comparing Deep Ensemble with baselines' ensembles. Table 3 shows the AURCs of the ensembles (including Deep Ensemble and baselines' ensembles) and their corresponding member models. Among all ensembles, Deep Ensemble has the lowest AURCs on CIFAR-10, CIFAR-100, and MRPC; SAT ensemble has the lowest AURC on SVHN; and Reg-curr has the lowest AURCs on QNLI and MNLI. Table 4 and 5 show the selective risks of ensembles under coverage  $10\% - 100\%$  on each dataset. The results show that no ensemble consistently outperforms others under all coverage on all datasets, indicating they have close performances but adopt different trade-offs between coverage and selective risk. In summary, the ensemble largely outperforms more recent methods and is competitive with their ensembles. We hope these results will draw more attention from researchers to the ensemble method.
In addition, we find that the Deep Ensemble can outperform SAT ensemble on SVHN as long as the label noise of SVHN is removed (see Appendix D). These results are surprising considering that Deep Ensemble only uses the MCP as the confidence score, while the baselines adopts more sophisticated techniques to train confidence estimators. Furthermore, we empirically explore further properties of selective classifier ensembles in Appendix D. We show that an ensemble with more member models has a better selective classification performance, and good classification performance does not necessarily imply good selective classification performance.
Table 3: AURC/10 $^{-4}$  of the ensembles (including Deep Ensemble and baselines' ensembles) and their corresponding member models on each dataset, where mem. and ens. represent the member model and the ensemble, respectively. The means and standard deviations are calculated over three trials. The best entries are marked in bold.
Dataset	Deep Ensemble	Gambler Ensemble	SAT Ensemble	Reg-curr Ensemble
CIFAR-10	mem.	66.1±5.6	70.7±2.6	67.1±0.3	67.1±2.0
ens.	51.2±2.0	57.4±1.0	57.6±0.6	53.5±0.5
CIFAR-100	mem.	793.1±9.0	930.6±10.2	807.3±6.0	851.9±12.8
ens.	672.5±1.9	857.1±4.7	715.5±12.2	714.2±6.3
SVHN	mem.	46.7±0.2	44.2±2.5	35.0±0.5	43.8±1.0
ens.	37.1±0.3	37.0±0.8	32.4±0.9	36.6±1.2
MRPC	mem.	654.3±68.3	695.6±65.9	794.6±51.9	696.8±66.0
ens.	561.3±55.6	650.1±24.8	563.0±16.5	640.3±14.1
QNLI	mem.	221.5 ±8.1	235.8 ±35.1	246.6 ±4.2	201.0 ±1.3
ens.	173.3 ±1.8	177.7 ±5.3	192.9 ±1.8	171.9 ±2.4
MNLI	mem.	515.3 ±8.6	607.6 ±19.3	519.8 ±8.0	496.8 ±9.5
ens.	465.2 ±3.8	569.4 ±11.4	454.4 ±3.1	451.9 ±2.1
Figure 7: The AURCs on the test set of CIFAR-10 of the SR ensemble (Deep Ensemble), Gambler ensemble, and SAT ensemble of different numbers of members
D Further Properties of Selective Classifier Ensemble
D.1 The Effect of Number of Members on Selective Classifier Ensemble
We evaluate AURCs of the SR ensemble (i.e., Deep Ensemble), Gambler ensemble, and SAT ensemble of different numbers of members on CIFAR10, and find that an ensemble with more members has a better performance, but is less efficient. The results are shown in Figure 7. In most cases, the AURC on the test set of CIFAR-10 decreases as the number of members in the ensemble increases. In addition, as the number of members in the ensemble grows, the effect of adding one member drops. On the one hand, the result shows that an ensemble with a small number of members has good selective classification performance. On the other hand, it indicates that when the number of member models is large, increasing the number of members to improve the performance of the selective classification ensemble is inefficient.
D.2 Good Classification Performance Does Not Imply Good Selective Classification Performance
It is well known that the ensemble has better classification performance than an individual model, but this does not guarantee a better selective classification performance of the ensemble. To demonstrate
Table 4: The selective risks of ensembles under coverage  $10\% -100\%$  on image classification datasets. The means and standard deviations are calculated over three trials. The best entries are marked in bold.
Dataset	coverage (%)	Deep Ensemble	Gambler ensemble	SAT ensemble	Reg-curr ensemble
CIFAR-10	100	5.31±0.03	5.29±0.03	5.47±0.04	5.74±0.07
90	1.68±0.02	1.99±0.01	2.15±0.06	1.89±0.02
80	0.45±0.05	0.51±0.02	0.63±0.03	0.61±0.09
70	0.17±0.01	0.21±0.01	0.26±0.01	0.18±0.02
60	0.11±0.01	0.18±0.03	0.17±0.01	0.08±0.00
50	0.11±0.01	0.14±0.02	0.11±0.01	0.07±0.01
40	0.12±0.03	0.15±0.02	0.06±0.01	0.07±0.01
30	0.13±0.05	0.13±0.03	0.06±0.02	0.08±0.03
20	0.12±0.02	0.17±0.05	0.00±0.00	0.02±0.02
10	0.10±0.08	0.14±0.05	0.00±0.00	0.00±0.00
SVHN	100	2.44±0.01	2.42±0.02	2.36±0.01	2.41±0.01
90	0.59±0.00	0.60±0.03	0.50±0.01	0.54±0.02
80	0.42±0.03	0.38±0.01	0.34±0.01	0.39±0.02
70	0.34±0.02	0.32±0.01	0.31±0.01	0.35±0.01
60	0.32±0.02	0.30±0.01	0.28±0.01	0.35±0.01
50	0.29±0.02	0.26±0.01	0.26±0.00	0.30±0.01
40	0.25±0.02	0.27±0.02	0.25±0.01	0.28±0.01
30	0.22±0.03	0.26±0.01	0.20±0.01	0.25±0.03
20	0.22±0.01	0.26±0.01	0.18±0.02	0.19±0.03
10	0.21±0.02	0.23±0.00	0.17±0.02	0.18±0.03
CIFAR-100	100	24.66±0.08	25.50±0.05	25.23±0.13	25.70±0.09
90	19.15±0.15	19.88±0.05	19.77±0.28	20.16±0.14
80	14.32±0.22	15.75±0.09	15.00±0.20	15.22±0.07
70	9.78±0.13	12.11±0.18	10.29±0.24	10.41±0.38
60	5.81±0.06	8.89±0.16	6.43±0.20	6.58±0.27
50	2.95±0.04	6.22±0.10	3.41±0.15	3.45±0.05
40	1.40±0.13	4.37±0.06	1.96±0.13	1.74±0.11
30	0.75±0.05	2.67±0.01	1.13±0.02	0.89±0.06
20	0.62±0.06	1.91±0.04	0.72±0.06	0.62±0.04
10	0.33±0.09	1.42±0.16	0.57±0.09	0.13±0.05
this, we design an SR model with a big backbone, and show that it has as good classification performance as an Deep Ensemble with a standard backbone but worse selective classification performance than an SR model with a standard backbone. The big backbone is designed to have twice as many filters in every convolutional layer and neurons in every fully connected hidden layer as those of the standard VGG-16, which is therefore called Big VGG-16. It is easy to see that its number of parameters is approximately  $2^{2} = 4$  times as many as that of standard VGG-16. We train an Deep Ensemble of 4 VGG-16s and an SR model with a backbone of Big VGG-16 on CIFAR-10 and show the evaluation results in Figure 8 and Table 6. Figure 8 shows that when coverage is high, the ensemble and the big individual model have similar selective risks, and especially, the classification error rates (i.e., selective risk of  $100\%$  coverage) of the ensemble and the big individual model are similar. However, when coverage is low, the big individual model has significantly higher selective risk than the ensemble. Table 6 shows that the AURC of Big VGG-16 is much higher than the ensemble of 4 VGG-16s and even higher than SR. In summary, we show that a selective classifier with a good classification performance is not guaranteed to have good selective classification performance, so the good selective classification performance of the ensemble is not a trivial result of its good classification performance.
D.3 The Effects of Label Noise of SVHN on Selective Classifier Ensembles
In this section, we compare the effect of label noise of SVHN on the Deep Ensemble with that on SAT ensemble, whose result might explain the abnormal experimental results (compared to results on other datasets) on SVHN in Section 6 of the main text. SVHN is not a clean dataset, and much more label noise can be detected in SVHN than in CIFAR-10 and CIFAR-100. Using the soft label of SAT
Table 5: The selective risks of ensembles under coverage  $10\% -100\%$  on text classification datasets. The means and standard deviations are calculated over three trials. The best entries are marked in bold.
Dataset	coverage (%)	Deep Ensemble	Gambler ensemble	SAT ensemble	Reg-curr ensemble
MRPC	100	14.13±0.23	14.62±0.23	13.64±0.23	15.28±0.31
90	11.41±1.02	11.50±0.46	10.69±0.13	11.68±0.44
80	7.75±0.29	9.99±0.88	8.46±0.63	8.36±0.29
70	6.41±0.33	8.51±0.72	7.93±0.44	6.64±0.29
60	5.71±0.33	7.35±1.20	6.39±0.19	6.12±0.33
50	4.08±1.29	4.41±0.40	3.59±1.01	4.74±0.23
40	3.25±0.29	3.86±0.76	3.25±0.76	3.66±0.50
30	3.25±0.00	3.52±0.38	3.52±0.38	2.98±0.77
20	2.44±1.72	3.25±0.58	3.25±0.58	3.66±0.00
10	3.25±2.30	4.88±0.00	1.63±1.15	5.69±1.15
QNLI	100	8.16±0.04	8.18±0.20	8.03±0.09	8.17±0.01
90	4.74±0.04	5.04±0.14	5.03±0.09	4.74±0.12
80	2.94±0.11	3.08±0.08	2.97±0.04	2.98±0.01
70	1.84±0.10	1.91±0.04	1.92±0.11	1.88±0.06
60	1.20±0.05	1.27±0.05	1.36±0.01	1.30±0.08
50	1.04±0.05	1.04±0.08	1.13±0.03	0.98±0.03
40	0.72±0.02	0.73±0.04	1.02±0.11	0.70±0.06
30	0.45±0.03	0.51±0.03	0.83±0.08	0.43±0.09
20	0.30±0.11	0.37±0.15	0.67±0.04	0.30±0.09
10	0.30±0.09	0.12±0.09	0.61±0.34	0.12±0.09
MNLI	100	15.04±0.06	14.82±0.15	15.03±0.06	14.89±0.14
90	11.01±0.11	11.78±0.09	11.37±0.06	11.21±0.17
80	7.93±0.08	9.62±0.08	8.26±0.20	8.02±0.07
70	5.81±0.04	8.03±0.23	5.81±0.12	5.85±0.22
60	4.28±0.07	6.41±0.25	4.08±0.19	4.05±0.10
50	3.22±0.04	4.95±0.10	2.99±0.18	3.04±0.10
40	2.69±0.11	3.57±0.21	2.08±0.03	2.22±0.06
30	2.13±0.14	2.35±0.18	1.57±0.03	1.75±0.06
20	1.34±0.10	1.53±0.17	1.39±0.13	1.36±0.06
10	0.98±0.05	1.32±0.14	0.71±0.22	0.71±0.08
Figure 8: Risk-coverage curves of the ensemble of 4 VGG-16s and the Big VGG-16 on CIFAR-10
[9], we detect label noise in SVHN, CIFAR-10, and CIFAR-100, and find that SVHN has significantly more label noise than CIFAR-10 and CIFAR-100. The result is presented in the following. In addition, it is known that SAT is robust to label noise [9], while SR is not so, so we conjecture that the label noise of SVHN is why the Deep Ensemble is inferior to SAT on SVHN.
We detect label noise with the help of the soft label of SAT. For a sample  $x_{i}$ , the soft label of SAT [9],  $t_{i,y_i}$ , is used to measure  $x_{i}$ 's learning difficulty. The soft label of SAT is initialized as 1 and updated
Table 6: The AURCs  $(/10^{-4})$  of Big VGG-16, a vanilla VGG-16, and the ensemble of 4 VGG-16s on CIFAR-10. The best entries are marked in bold.
Dataset	Big VGG-16	VGG-16	Ensemble
CIFAR-10	89.2	69.6	49.3
Table 7: Numbers of mislabeled samples in the top-0.1% difficult training samples of SVHN, CIFAR-10, and CIFAR-100.
Dataset	#Mislabeled	#Top-0.1%	Proportion
SVHN	73	73	100%
CIFAR-10	1	50	2%
CIFAR-100	1	50	2%
at every training epoch as below
$$
t _ {i, y _ {i}} \leftarrow \alpha \times t _ {i, y _ {i}} + (1 - \alpha) \times p _ {\theta} \left(y _ {i} \mid x _ {i}\right),
$$
where  $p_{\theta}(Y|x)$  is the predictive probability distribution of the classifier,  $y_{i}$  is the label of  $x_{i}$ ,  $\alpha$  is a hyperparameter. The smaller the  $t_{i,y_i}$  is, the lower the true class predictive probability of the classifier on  $x_{i}$  during training time, indicating that  $x_{i}$  is more difficult to learn. By selecting a percentage of samples with the lowest  $t_{i,y_i}$ , we get the most difficult samples to learn for the classifier, from which we can easily detect label noise manually.
In training sets of SVHN, CIFAR-10, and CIFAR-100, we detect label noise manually among the top-0.1% difficult (measured by the soft label of SAT) samples. The numbers of mislabeled samples detected in SVHN, CIFAR-10, and CIFAR100 are shown in Table 7. The result shows that SVHN has significantly more unlabeled samples detected than CIFAR-10 and CIFAR-100, indicating much more label noise in SVHN than in CIFAR-10 and CIFAR-100.
To verify the effect of label noise, the following experiments are designed. Firstly, we detect label noise manually among the  $1\%$  of the hardest-to-learn samples of SVHN training set and test set, using the soft label of SAT. Secondly, we remove the detected mislabeled samples from the original dataset. The remaining SVHN dataset is called the clean SVHN. Accordingly, the original dataset is called the original SVHN. Finally, we retrain and test the Deep Ensemble and SAT ensemble and compare their test results. In the second step, the reason for removing mislabeled samples rather than modifying them is that some samples cannot be classified even by humans, and some samples are not in the range of categories of SVHN. Thus, the label noise cannot be eliminated by modifying the labels but by removing mislabeled samples.