Title: Top-Ambiguity Samples Matter: Understanding Why Deep Ensemble Works in Selective Classification
Venue: NeurIPS 2023
Accepted: yes
Keywords: selective classification

uncertainty estimation

ensemble learning
Abstract: Selective classification allows a machine learning model to reject some hard inputs and thus improve the reliability of its predictions. In this area, the ensemble method is powerful in practice, but there has been no solid analysis on why the ensemble method works. Inspired by an interesting empirical result that the improvement of the ensemble largely comes from top-ambiguity samples where its member models diverge, we prove that, based on some assumptions, the ensemble has a lower selective risk than the member model for any coverage within a range. The proof is nontrivial since the selective risk is a non-convex function of the model prediction. The assumptions and the theoretical results are supported by systematic experiments on both computer vision and natural language processing tasks.

1

Abstract

Selective classification allows a machine learning model to reject some hard inputs and thus improve the reliability of its predictions. In this area, the ensemble method is powerful in practice, but there has been no solid analysis on why the ensemble method works. Inspired by an interesting empirical result that the improvement of the ensemble largely comes from top-ambiguity samples where its member models diverge, we prove that, based on some assumptions, the ensemble has a lower selective risk than the member model for any coverage within a range. The proof is nontrivial since the selective risk is a non-convex function of the model prediction. The assumptions and the theoretical results are supported by systematic experiments on both computer vision and natural language processing tasks.

1 Introduction

Although recent years have witnessed the broad applications of deep learning models, their reliability has not been guaranteed, which gives rise to the study of selective classification. For any given deep learning classifier, there might be inputs that the model is not able to classify correctly in practical applications. One approach to overcoming this problem is accurately delimiting the deep learning classifier's application scope so that the classifier rejects to predict over those hard inputs.

To this end, selective classification aims to learn a pair of models consisting of a standard classifier and a selective function that decides whether the system should reject the input. Since the classifier is well studied, the study of selective classification focuses on the design of the selective function. A selective function is usually a confidence estimator with a threshold [1], rejecting the input if and only if the estimated confidence is below the threshold. There are four categories of confidence estimation methods: post-hoc methods [2, 3], Bayesian methods [4, 5, 6], learning-based methods [7, 8, 9, 10], and ensemble methods [11]. Details about these methods is provided in Section 3.

Previous work has shown that Deep Ensemble has a good performance in selective classification [11] 1 , but no solid analysis explains why the ensemble method works. This paper first proposes a theoretical foundation of Deep Ensemble, i.e., with some assumptions, the ensemble has a lower selective risk than the member model for any target coverage within a range. The proof is nontrivial since the selective risk (with the 0/1 loss) are non-convex. We then verify our assumptions and

theoretical result on both computer vision and natural language processing tasks. In summary, the contributions of this paper are summarized as follows.

Weshowan interesting empirical result that the performance improvement of Deep Ensemble in terms of selective classification dominantly comes from top-ambiguity samples.

We are the first to prove that the ensemble has a lower selective risk than the member model for any target coverage within a range based on several reasonable assumptions. Our assumptions and analysis results are verified by systematic experiments on the tasks of image classification and text classification.

2 Preliminary

A selective classifier comprises a standard classifier and a selective function. Considering a standard classification problem, X is a feature space, Y = { 1 , 2 , ..., K } is a finite label set, and a classifier is a function the maps X to Y . A labeled dataset D = { ( x i , y i ) } N i =1 ⊆ X × Y is independently sampled from an identical distribution D . Our goal is to learn a selective classifier ( f, g ) , where f is a standard classifier, and g : X →{ 0 , 1 } is a selective function that estimates the correctness of f 's prediction. Given input x , the output of selective classifier ( f, g ) is

The goal of selective classifiers is to minimize the selective risk subject to a given target coverage . The coverage of ( f, g ) is defined to be the probability of ( f, g ) not rejecting the input, i.e., φ ( f, g ) := E ( x,y ) ∼D [ g ( x )] . The selective risk of ( f, g ) is the risk of f conditioned on g ( x ) = 1 , i.e., R ( f, g ) := E ( x,y ) ∼D [ glyph[lscript] ( f ( x ) , y ) g ( x )] / E ( x,y ) ∼D ( g ( x )) , where glyph[lscript] : Y × Y → R is a given loss function. Typically, glyph[lscript] is the 0/1 loss [1, 7, 8, 9]. Based on these definitions, the objective of selective classifiers is formalized as

where c target is the target coverage. Note that the above objective is evaluated only on in-distribution samples, i.e., samples independently drawn the same distribution of the training data, excluding samples with distribution shifts - also known as OOD samples [12].

The selective function is usually realized by a confidence estimator κ : X → R with a confidence threshold t : g ( x ) = 1 κ ( x ) ≥ t , where 1 is the indicator function [1]. In this case, we denote the selective classifier as ( f, κ ; t ) . As the threshold t controls the coverage, a fixed ( f, κ ) can be fitted to different target coverages (with different confidence thresholds). Thus, ( f, κ ) 's performance can be evaluated under multiple different coverages, raising metrics such as risk-coverage curve (RC curve) [1] and the Area Under the Risk-Coverage curve (AURC) [10]. The RC curve of ( f, κ ) is a plot of ( f, κ ) 's selective risks against its coverages, which depicts the entire performance profile of ( f, κ ) . A lower risk-coverage curve suggests a better selective classifier. The AURC is a more concise and comprehensive metric. Similar to the RC curve, a lower AURC suggests a better selective classifier.

This paper focuses on analyzing the selective risk of Deep Ensemble [11], which is defined as follows. To obtain a Deep Ensemble ( f E , κ E ) , M member models are trained in parallel with random initialization of their parameters and random shuffling of the data points. Assume that each member classifier provides a predictive probability distribution π m = ( π 1 m , ..., π K m ) , 1 ≤ m ≤ M and predicts f m ( x ) = arg max 1 ≤ k ≤ K π k m . Then Deep Ensemble is a uniformly-weighted soft voting of member models: π E ( x ) := 1 M ∑ M m =1 π m ( x ) , and its confidence estimator is κ E ( x ) = max k π k E ( x ) .

3 Related Work

Selective Classification. A selective function usually relies on a confidence estimator. To realize the confidence estimator, four approaches has been investigated by previous work. 1. Post-hoc methods , including Softmax Response (SR) [2] and Trust Score [3]. SR leverages the classifier's maximum class probability (MCP) as the confidence score. Trust Score considers both the distance from the sample to the predicted class and the distance from the sample to another nearest class,

using their ratio as the confidence score. 2. Bayesian methods. MC-Dropout is an easy-to-use Bayesian methods [5], which samples the classifier's parameters through Dropout [13]. Specifically, MC-Droput enables the dropout layer of the classifier at test time, then runs multiple forwards passes, and finally uses the minus variance of the maximum activation of the classifier's softmax layer as the confidence score. A shortcoming of MC-Dropout is its high time cost at test time. 3. Learning-based methods. There are two categories of learning-based methods: 1. using MCP as the confidence score and only modifying the standard classifier's loss function, including OSP [14], and Reg-curr [10]; 2. modifying both the architecture and the loss function of the classifier, including SelectiveNet [7], Gambler [8], ConfidNet [15], and SAT [9]. SAT is the SOTA selective classifier in terms of non-ensemble methods, which adds an extra class to the classifier as a reject option and proposes a loss function to train the classifier and the confidence estimator simultaneously. 4. Ensemble Methods. Deep Ensemble is a vanilla classifiers' ensemble combined by soft voting and equipped with an MCP confidence estimator [11], which has achieved the SOTA in uncertainty estimation. Considering the heavy computational overhead of the ensemble, [16, 17] reduced the ensemble's time consumption by ensemble distillation techniques that distill the ensemble's knowledge into an individual model with the ensemble's diversity retained.

Analyses of Ensemble Methods. Ensemble methods combine multiple individual models to improve machine learning models' predictive performances (see [18] for a review). The error-ambiguity decomposition has been proposed to explain the better performance of the ensemble in regression tasks [19]. However, for classification tasks, there is no such simple and elegant analysis, since the evaluation metrics are non-convex [20]. Thus, the corresponding analysis for classification tasks needs additional assumptions, e.g., unbiased, uncorrelated, and identically distributed estimation errors for the posterior probability distribution [21, 22]. Nevertheless, these assumptions are impractical [22]. In terms of metrics other than 0-1 loss, [23] uses the entropy of the predictive distribution as uncertainty, deriving the decomposition of data uncertainty and knowledge uncertainty of ensembles. Several other works [24, 25, 26] also discuss different forms of ensemble performance decomposition for metrics other than 0-1 loss. Despite these advances in analyzing ensembles, the analysis of the selective risk of the ensemble remains under-explored.

Other related topic of selective classification include Out-Of-Distribution Detection (OOD Detection) and Calibration , which along with selective classification are subdomains of uncertainty estimation [27]. OOD detection (also known as open set recognition ) aims to detect test samples with semantic distribution shift (e.g., occurrence of new classes) without losing the in-distribution classification accuracy [12]. OOD detection and selective classification share the requirement for in-distribution classification, and their difference lies in their mutually exclusive detection targets: the former detects misclassified in-distribution samples; while the latter detects samples with semantic distribution shifts. Calibration tackles the problem of predicting confidence scores that are representative of the true correctness probabilities [28]. Calibration and selective classification focus on different dimensions of uncertain estimation: calibration focuses on the overall confidence level; while selective classification focuses on the relative confidence ranking among the samples [10].

4 The Dominant Source of Performance Improvement of Deep Ensemble

Before we analyze the selective risk of Deep Ensemble, we present an experimental result that helps us to focus on the dominant source of performance improvement of Deep Ensemble. The experiment is to examine whether the performance improvement of the ensemble dominantly comes from high-ambiguity samples, where ambiguity is quantified as S = √ ∑ i M =1 ‖ π i -π E ‖ 2 M -1 . We conduct the experiment on datasets CIFAR-10, CIFAR-100 [29], SVHN [30], MRPC [31], MNLI [32], and QNLI [33] (the details of the datasets and the models are provided in Section 6). The procedure of this experiment is as follows. First, we train a member model and an ensemble, and feed them test data. Second, all test samples are sorted by their ambiguities, and we define samples with the top-50% ambiguities as high-ambiguity samples and the rest 50% samples as low-ambiguity samples 2 . Third, we construct ensembling on high-ambiguity samples , i.e., combining the member models (ensembling) on high-ambiguity samples and using a member model on low-ambiguity samples. Mathematically, denoting the median of ambiguities on the dataset as t , ensembling on

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.01

0.02

0.03

0.04

0.05

0.06

selective risk

CIFAR-10

member

ensemble

top-50%-S ens.

lowest-50%-S ens.

(a)

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.000

0.005

0.010

0.015

0.020

0.025

selective risk

SVHN

member

ensemble

top-50%-S ens.

lowest-50%-S ens.

(b)

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.05

0.10

0.15

0.20

0.25

selective risk

CIFAR-100

member

ensemble

top-50%-S ens.

lowest-50%-S ens.

(c)

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

selective risk

MRPC

member

ensemble

top-50%-S ens.

lowest-50%-S ens.

(d)

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.02

0.04

0.06

0.08

selective risk

QNLI

member

ensemble

top-50%-S ens.

lowest-50%-S ens.

(e)

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

selective risk

MNLI

member

ensemble

top-50%-S ens.

lowest-50%-S ens.

(f)

high-ambiguity samples induces a model ˜ f E that makes predictions as

We equate this model with ensembling on high-ambiguity samples in the following. Similarly, we define ensembling on low-ambiguity samples as combining the member models (ensembling) on low-ambiguity samples and using a member model on high-ambiguity samples. Finally, we plot risk-coverage curves of the member model, the ensemble, ensembling on high-ambiguity samples, and ensembling on low-ambiguity samples.

Figure 1 shows the results of these risk-coverage curves. As we can see, on each dataset, the riskcoverage curve of ensembling on high-ambiguity samples almost coincides with that of the ensemble, and risk-coverage of ensembling on low-ambiguity samples almost coincides with that of the member model. This result suggests that ensembling on high-ambiguity samples provides the dominant performance improvement of the ensemble, while ensembling on low-ambiguity samples provides a minor performance improvement.

5 Analysis

This section aims to prove that the selective risk of Deep Ensemble is lower than its member models for any given coverage within a range. If the selective risk is a convex function of member models' predictive probability distributions, then the definition of the ensemble guarantees that the ensemble's selective risk is less than or equal to that of the member model according to Jensen's inequality. However, the selective risk is not a convex function of member models' predictive probability distributions due to the 0/1 loss. Thus, the definition of the ensemble is not sufficient to prove the superiority of the ensemble. The same problem exists in the analysis of error rate in standard classification tasks. To address this problem in classification, three unrealistic assumptions: unbiased estimation error, identically distributed estimation error, and uncorrelated estimation error have been proposed [22]. However, we develop other assumptions that are more realistic to make our analysis practical.

5.1 Intuition

Our first assumption is motivated by the empirical result in Section 4. In the previous experiment, the samples are divided into two categories: low-ambiguity samples and high-ambiguity samples. The result shows that low-ambiguity samples bring little performance improvement, but high-ambiguity samples bring the dominant performance improvement of the ensemble. For the convenience of theoretical analysis, we approximate these two types of samples as follows: 1. definite samples on which all member models produce the same predictive probability distribution; 2. ambiguous samples on which member models produce statistically-dependent predictive probability distributions (but their joint distribution is unknown).

Based on this assumption, Figure 2 illustrates the intuition of why ensembling works. Given a target coverage, the corresponding confidence thresholds of the member model and the ensemble are shown as the dashed green line and the solid green line, respectively. The solid blue line represents the PDF of the member's (as well as the ensemble's) confidence score over definite samples, and the dashed/solid red line represents the PDF of the member/ensemble model's confidence score over ambiguous samples. Thus, in the upper subfigure, the area under the blue/red line on the right-hand side of the dashed green line is the probability of the member model accepting (i.e., not rejecting) the sample given the sample being definite/ambiguous; and a similar result applies to the ensemble in the lower subfigure. In addition, we assume that the member model is not modest over ambiguous samples and over definite samples, i.e., the right-hand ends of the dashed red line and blue line are above the horizontal axis. We can prove that the ensemble prevents high confidence scores (termed as being modest ) over ambiguous samples, i.e, the right-hand end of the solid red line falls on the horizontal axis. Thus, through ensembling, the PDF over ambiguous samples moves down from the dashed red curve to the solid red curve when the confidence score is near 1 ( as the red arrows show). On the contrary, the PDF over definite samples stays the same. Thus, if the target coverage is small so that the threshold of the member model is near 1, within accepted samples (on the right-hand side of the dashed green line), the proportion of ambiguous samples drops. The threshold of the ensemble model also drops to retain the target coverage, but the net effect of these two changes is a lower proportion of ambiguous samples (or a higher proportion of definite samples) within accepted samples. In addition, we assume that the error rate of classifiers over definite samples is lower than that over ambiguous samples. Thus, the selective risk becomes smaller through ensembling given this coverage.

In the following, we first introduce the assumptions, then we show that the ensemble is modest over ambiguous samples, which leads to an upper bound of ensembles' selective risk, and finally we prove the superiority of the ensemble model.

5.2 Assumptions

This section follows the notations of Section 2. Assume that ( f ∗ , κ ∗ ) is any member model of ( f E , κ E ) , where both κ ∗ and κ E are MCP confidence estimators, whose outputs are in [ 1 K , 1] . For the convenience of notation, let R ∗ ( t ) and φ ∗ ( t ) denote ( f ∗ , κ ∗ ; t ) 's selective risk and coverage, respectively; and R E ( t ) and φ E ( t ) denote ( f E , κ E ; t ) 's selective risk and coverage, respectively, where t is a confidence threshold.

PDF

mem. on ambiguous samples

mem. on definite samples

threshold

confidence score

PDF

1/K

1

ens. on ambiguous samples

ens. on definite samples

threshold

Member

Model

Ensemble

Model

ensembling

According to the motivation in Section 4, we assume that the whole data distribution comprises two types of samples: 1. definite samples for which all member models provide the same predictive probability distribution; 2. ambiguous samples for which member models provide dependent predictive probability distributions (but their dependency is unknown). Formally, we have

Assumption 1. (Correlation and Diversity Assumption) Let a ( x ) ∈ { 0 , 1 } (or a for short) be a hidden variable that denotes whether the sample x is ambiguous ( a = 0 / 1 indicates a definite/ambiguous sample). We assume: first, given a = 0 , π 1 = π 2 = · · · = π M ; second, given

a = 1 , ∀ k ∈ { 1 , . . . , K } , p ( π k 1 , π k 2 , . . . , π k M | a = 1) is bounded, i.e., always finite, where p denotes probability density function (PDF).

Assumption 1 depicts both the correlation and the diversity of the ensemble. The existence of definite samples provides a strong correlation among ensemble members' predictions, and the existence of ambiguous samples provides both the correlation and the diversity of the member models' predictions. Furthermore, because Assumption 1 does not specify the correlation of ensemble members over ambiguous samples, this assumption is a relaxation of the uncorrelated-estimation-error assumption of [21, 22].

In addition, because ambiguous samples seem more difficult to classify than definite samples, we assume the member model's selective risk on ambiguous samples is greater than that on definite samples:

glyph[negationslash]

Assumption 2. (Hardness Assumption) For any member model ( f ∗ , κ ∗ ) , we assume lim t → 1 -R ∗ ( t | a = 1) > lim t → 1 -R ∗ ( t | a = 0) , where R ∗ ( t | a ) is the conditional selective risk of ( f ∗ , κ ∗ ; t ) given a , i.e., E ( x,y ) ∼D [ 1 y = f ∗ ( x ) | a, κ ∗ ( x ) ≥ t ] .

Finally, we assume that the member model is not modest over some ambiguous samples and some definite samples:

Assumption 3. (Confidence Assumption) For any member model ( f ∗ , κ ∗ ) , we have lim t → 1 -p ( κ ∗ ( x ) = t | a = 0) > 0 and lim t → 1 -p ( κ ∗ ( x ) = t | a = 1) > 0 .

5.3 The Ensemble is Modest over Ambiguous Samples

According to Assumption 3, the member model is not modest over ambiguous samples. However, we show that the ensemble is more modest over ambiguous samples (proof is provided in Appendix A):

Lemma 1. Let B := ∑ K k =1 sup [0 , 1] M p ( π k 1 , ..., π k M | a = 1) . Assumption 1 implies ∀ t ∈ [0 , 1] , p ( κ E ( x ) = t | a = 1) ≤ M M B (1 -t ) M -1 . Integrating this inequality, we derive P ( κ E ( x ) ≥ t | a = 1) ≤ M M -1 B (1 -t ) M .

The first inequality in this lemma shows that the ensemble satisfies lim t → 1 -p ( κ E ( x ) = t | a = 1) = 0 , which is the opposite of a member model. In other words, the ensemble is less possible to provide high confidence scores over ambiguous samples than its member models. This result then leads to an upper bound of the ensemble's selective risk.

5.4 An Upper Bound of the Ensemble's Selective Risk

We derive an upper bound of the ensemble's selective risk, demonstrating how members' performance and their diversity affect the ensemble's performance. According to selective risk's definition, the ensemble's selective risk can be bounded as (see Appendix A for proof)

The two terms in the right-hand side of (2) can be further reduced to member models' performance and their diversity as follows. First, given a = 0 , we have f E ( x ) = f ∗ ( x ) and κ E ( x ) = κ ∗ ( x ) (due to Assumption 1), which suggests R E ( t | a = 0) = R ∗ ( t | a = 0) . Second, combining the second inequality in Lemma 1 with Bayes' rule, we can bound the second term as follows:

where γ = BM M -1 · P ( a = 1) . The right-hand side of (3) is negatively related to the diversity of member models. The reason of this statement is as follows. More diverse member models may result in a flatter p ( π k 1 , ..., π k M | a = 1) , leading to a lower B . A lower B further leads to a lower right-hand side of (3). Combining the results above, we have

Lemma 2. (Selective Risk Bound of The Ensemble) The ensemble's selective risk is bounded as

0.990

0.992

0.994

0.996

0.998

confidence threshold

0

1

2

3

4

5

6

selective risk (%)

CIFAR-10

SVHN

CIFAR-100

def.

amb.

(a)

0.90

0.92

0.94

0.96

0.98

confidence threshold

0

5

10

15

20

25

30

35

40

selective risk (%)

MRPC

QNLI

MNLI

def.

amb.

(b)

The first term of this upper bound is the selective risk of a member model on definite samples, and the second one is negatively related to the diversity among member models. When member models' performance on definite samples or the diversity improves, this upper bound decreases, which is a reasonable result.

5.5 Main Result

Based on Lemma 2, we can prove that Deep Ensemble has a lower selective risk than a member model for any target coverage that is in a range (Theorem 1, see Appendix A for the proof).

Theorem 1. If Assumptions 1, 2, and 3 hold, then ∃ φ 0 ∈ (0 , 1) such that ∀ φ ∈ (0 , φ 0 ) , φ E ( t E ) = φ ∗ ( t ∗ ) = φ ⇒ R E ( t E ) < R ∗ ( t ∗ ) .

Furthermore, it is interesting to figure out the maximum φ 0 in Theorem 1. In Appendix E, we use Lemma 2 to estimate the maximum φ 0 (without accessing the ensemble) and find that it can be large, say > 50% , on several datasets. When we instead have the access to the ensemble in experiments, the actual maximums of φ 0 on most datasets are 1, i.e., the largest possible value of φ 0 . These results seem interesting, and we will intuitively explain them in the next section.

6 Experiments

This section verifies the assumptions and analysis results in realistic tasks, and then compares Deep Ensemble with several SOTA methods and their ensembles in image classification and text classification tasks. However, we do not verify Assumption 1 in the following because the empirical result in Section 4 has shown that Assumption 1 is a reasonable approximation that does not compromise the performance of the ensemble.

Datasets. Following [9], we used CIFAR-10, CIFAR-100, and SVHN for image classification tasks. Following [10], we used MRPC, MNLI, and QNLI for text classification tasks. More details of the datasets used in our experiments are described in Appendix B.

Models. Following [9, 10], we use VGG-16 [34] and BERT-base [35] as the backbones of selective classifiers for image classification and text classification, respectively. Each member model is built on basis of a backbone model, and an ensemble consists of five member models by default in experiments. More details of the backbone models and their training procedures are provided in Appendix B.

Verification of Assumptions 2 and 3. We use samples with ambiguity lower than glyph[epsilon1] ( glyph[epsilon1] > 0) (termed as low-ambiguity samples ) to represent definite samples and use samples with ambiguity greater than or equal to glyph[epsilon1] (termed as high-ambiguity samples ) to represent ambiguous samples in experiments as we use definite/ambiguous samples to approximate low/high-ambiguity samples in theory. We choose glyph[epsilon1] = 10 -3 for datasets of image classification and glyph[epsilon1] = 10 -2 for datasets of text classification. Figure 3 shows a member model's conditional selective risks over low-ambiguity samples and high-ambiguity samples against a range of confidence thresholds near 1. The results show that the member model's

0.990

0.992

0.994

0.996

0.998

1.000

confidence

0

1000

2000

3000

4000

5000

number of samples

CIFAR-10

mem. on def.

mem. on amb.

ens. on def.

ens. on amb

(a)

0.990

0.992

0.994

0.996

0.998

1.000

confidence

0

2000

4000

6000

8000

10000

12000

SVHN

mem. on def.

mem. on amb.

ens. on def.

ens. on amb

(b)

0.990

0.992

0.994

0.996

0.998

1.000

confidence

0

200

400

600

800

1000

1200

1400

1600

CIFAR-100

mem. on def.

mem. on amb.

ens. on def.

ens. on amb

(c)

0.90

0.92

0.94

0.96

0.98

1.00

confidence

0

50

100

150

200

250

300

number of samples

MRPC

mem. on def.

mem. on amb.

ens. on def.

ens. on amb.

(d)

0.90

0.92

0.94

0.96

0.98

1.00

confidence

0

500

1000

1500

2000

2500

3000

3500

QNLI

mem. on def.

mem. on amb.

ens. on def.

ens. on amb.

(e)

0.90

0.92

0.94

0.96

0.98

1.00

confidence

0

500

1000

1500

2000

2500

3000

3500

4000

MNLI

mem. on def.

mem. on amb.

ens. on def.

ens. on amb.

(f)

selective risk over low-ambiguity samples is consistently lower than that over high-ambiguity samples given confidence thresholds near 1, which verifies Assumption 2. Figure 4 shows histograms of confidence scores of the member model (as well as the ensemble) on low-ambiguity samples and on high-ambiguity samples. As we can see, the number of low-ambiguity samples and the number of high-ambiguity samples are non-zero in the top bin on each dataset, which verifies Assumption 3. In summary, Assumption 2 and 3 hold on all datasets.

Verification of Lemma 1. Figure 4 shows histograms of the ensemble's confidence scores on lowambiguity samples and on high-ambiguity samples. As we can see, the ensemble's confidence score over high-ambiguity samples has little distribution in the top bin, which is tremendously lower than the corresponding distribution of the member model, while the histogram of confidence score over low-ambiguity samples has almost no change throughout ensembling. This result is consistent with Lemma 1 and our intuition in Figure 2.

Verification of Theorem 1. Figure 1 already shows the risk-coverage curves of Deep Ensemble and a member model on each dataset. As we can see, on each dataset except MRPC, the ensemble has a consistently lower selective risk than the member model under any coverage; on MRPC, the ensemble has a lower selective risk than the member model except for coverage of around 30% and around 90% 3 . These results are consistent with Theorem 1.

Despite of this consistency, the theorem does not explain the lower selective risk of the ensemble given a large coverage, say 70%. This is somehow resolved by the analysis in Appendix E. Here, instead of

diving into theory, we intuitively explain the experimental results as follows. As Lemma 1 claims, P ( the ensemble yielding confidence ≥ t | an ambiguous input example ) is O (1 -t ) M ( t → 1 -) , where M is the number of ensemble members. Therefore, the ensemble hardly provides an ambiguous sample with confidence close to 1. On the contrast, definite samples are always assigned confidence scores close to 1, as the the experimental results show in Figure 4. By this mean, the ensemble stratifies the definite samples and ambiguous sample by their confidence scores, where the definite samples reside on a thin higher layer of confidence than the ambiguous samples (see the rightmost black-edged bars vs. red-edged bars in Figure 4). Combined with the low risk (of both the member model and the ensemble) on definite samples, this stratification lowers the selective risk when the coverage is around the proportion of definite samples in the dataset. Considering the large amount of definite samples (see the heights of black-edged bars in Figure 4), the ensemble model will exhibit lower selective risk than the member model given a considerably large coverage.

In the explanation above, the key factor that leads to the lower selective risk of the ensemble given a large coverage is the experimental fact that the definite samples are large-amounted and always assigned confidence close to 1. This fact is not involved as an assumption in the theory since it seems a very strong assumption (though it holds throughout our experiments). We guess this is attributed to the low bias of DNNs (from a bias-variance perspective), which might be a widespread property of DNNs. Therefore, it would be an interesting direction for future work to strengthen our theory by exploiting this fact.

Extended Experimental Settings. To further validate our analysis, we extended the experimental settings on three dimensions: 1. model architecture (to ResNet [36], AlexNet [37], and DenseNet[38]); 2. dataset (to ImageNet100, a subset of ImageNet [39]); 3. number of member models in the ensemble (to 20).

The results are shown in Figure 5. As the figure shows, Assumptions 1, 2, and 3 are consistently verified across various experiment settings. The results indicate that our assumptions might reflect the general characteristics of DNNs. For example, Assumptions 2 and 3 might be attributed to the low bias of DNNs (from a bias-variance perspective) due to DNNs' large model capacity.

These results were unknown when we established our analysis. However, all of them reproduce similar results to the previous results in new settings, further confirming our analysis in practical settings.

7 Discussion and Conclusion

We prove that under some assumptions, Deep Ensemble has a lower selective risk than the member model for any target coverage within a range. Although the metrics of selective classification are non-convex, we complete the proof with the help of several assumptions motivated by empirical observations, e.g., the performance improvement of the ensemble dominantly comes from highambiguity samples. The assumptions and analysis results are well supported by the experimental results on multiple datasets of image classification tasks and text classification tasks.

Our analysis may benefit the analysis of the Deep Ensemble for standard classification. The previous analyses of the randomization-based ensemble for standard classification require some impractical assumptions [22]. On the contrary, this paper's assumptions are a good approximation of practical settings (see Section 6). Moreover, the standard classification is a subset of selective classification, i.e., selective classification with coverage of 1 (not covered by our analysis result). Thus, our analysis may motivate the analysis of the Deep Ensemble for standard classification in practical settings.

Other possible directions for future work include: 1. extending the (0 , φ 0 ) mentioned in Theorem 1 to (0 , 1] , i.e., proving that the ensemble has a lower selective risk than the member model under any coverage; 2. adapting the analysis to other selective classifier ensembles, e.g., the ensemble of SAT; 3. relaxing the assumptions to generalize the application scope of our analysis.

References

Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems , 2017.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representation , 2017.

Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Advances in neural information processing systems , 2018.

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning , 2015.

Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning , 2016.

Yarin Gal. Uncertainty in deep learning . PhD thesis, University of Cambridge, 2016.

Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning , 2019.

Ziyin Liu, Zhikang Wang, Paul Pu Liang, Russ R Salakhutdinov, Louis-Philippe Morency, and Masahito Ueda. Deep gamblers: Learning to abstain with portfolio theory. In Advances in Neural Information Processing Systems , 2019.

Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: Beyond empirical risk minimization. In Advances in Neural Information Processing Systems , 2020.

Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. The art of abstention: Selective prediction and error regularization for natural language processing. In Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , 2021.

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems , 2017.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 , 2021.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research , 15(1):1929-1958, 2014.

Aditya Gangrade, Anil Kag, and Venkatesh Saligrama. Selective classification via one-sided prediction. In International Conference on Artificial Intelligence and Statistics , 2021.

Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. Addressing failure prediction by learning model confidence. In Advances in Neural Information Processing Systems , 2019.

Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. In International Conference on Learning Representation , 2020.

Giung Nam, Jongmin Yoon, Yoonho Lee, and Juho Lee. Diversity matters when learning from ensembles. In Advances in Neural Information Processing Systems , 2021.

Zhi-Hua Zhou. Ensemble methods: foundations and algorithms . CRC press, 2012.

Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in Neural Information Processing Systems , 1994.

Zhi-Hua Zhou. Machine learning . Springer Nature, 2021.

Kagan Tumer and Joydeep Ghosh. Analysis of decision boundaries in linearly combined neural classifiers. Pattern recognition , 29(2):341-348, 1996.

Giorgio Fumera and Fabio Roli. A theoretical and experimental analysis of linear combiners for multiple classifier systems. IEEE transactions on pattern analysis and machine intelligence , 27(6):942-956, 2005.

Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In Advances in Neural Information Processing Systems , 2018.

Neha Gupta, Jamie Smith, Ben Adlam, and Zelda Mariet. Ensembling over classifiers: a bias-variance perspective. arXiv preprint arXiv:2206.10566 , 2022.

Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural network ensembles. In International Conference on Artificial Intelligence and Statistics , 2022.

Danny Wood, Tingting Mu, Andrew Webb, Henry Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. arXiv preprint arXiv:2301.03962 , 2023.

Dustin Tran, Jeremiah Zhe Liu, Michael W Dusenberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda E Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. In Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML , 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning , 2017.

Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing , 2005.

Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) , 2018.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , 2018.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , 2019.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition , 2016.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems , 2012.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE conference on computer vision and pattern recognition , 2017.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition , 2009.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning , 2015.

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-ofthe-art natural language processing. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2020.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H Abdi. Towards better selective classification. 2023.

A Proofs

A.1 Proof of Lemma 1

The proof of the second inequality in Lemma 1, i.e.,

can be reduced to the proof of the first inequality, i.e.,

The reason is as follows. If (6) holds, then we have

which directly derives (5). Therefore, we only need to show (6) at following.

Firstly, we derive the PDF of the average of multiple continuous random variables expressed by the PDFs of these random variables (Lemma 3), which helps us to analyze the PDF of the ensemble's predictive probabilities.

Lemma 3. Let X 1 , X 2 , . . . , X M be M continuous random variables, and their average is X avg := 1 M ∑ M i =1 X i . Then the PDF of X avg 4 is

where p glyph[vector] X is p X 1 , X 2 ,..., X M for short.

Proof. The distribution function of X avg is

Let x M = u -∑ M -1 i =1 x i , then the integral above is equal to

The PDF of X avg is the derivative of F X avg , which, combined with (,8) derives

which is exactly (7).

Secondly, we show the relationship between the PDF of confidence score and the PDFs of predictive probabilities. Note that the confidence score of an SR model is the maximum predictive probability, so the following lemma bounds the PDF of confidence by PDFs of predictive probabilities.

Lemma 4. Let Π k (1 ≤ k ≤ K ) be K continuous random variables, and C := max k Π k . Then we have

Proof. First of all, we prove ∀ κ 1 , κ 2 , κ 1 < κ 2 ,

It is easy to see that

so the left-hand side of (10) is

where the last equality is due to ( -∞ , κ 1 ] ⊂ ( -∞ , κ 2 ] , and the right-hand side of (10) is

where the last inequality is because R k -1 × [ κ 1 , κ 2 ] × R K -k for different k , 1 ≤ k ≤ K , may have an intersection. To prove (10), we only need to prove that the right-hand side of (11) is less than or equal to the right-hand side of (12), which is equivalent to prove

Now we prove (13). ∀ ( π 1 , . . . , π K ) ∈ ( -∞ , κ 2 ] K \ ( -∞ , κ 1 ] K , we have

where (15) is because if all π k is less than or equal to κ 1 instead, then ( π 1 , . . . , π K ) ∈ ( -∞ , κ 1 ] K , which contradicts with ( π 1 , . . . , π K ) ∈ ( -∞ , κ 2 ] K \ ( -∞ , κ 1 ] K . Thus, π k 0 ∈ [ κ 1 , κ 2 ] , so

which is precisely (13), and therefore (10) is proved.

With (10) and the definition of derivatives, it is easy to see that F ′ C ( κ ) ≤ ∑ K k =1 F ′ Π k ( κ ) , which is equivalent to p C ( κ ) ≤ ∑ K k =1 p Π k ( κ ) . Thus, Lemma 4 is proved.

Based on the lemmas above, we have the following lemma, which is precisely (6). When (6) is proved, the proof of Lemma 1 completes.

Lemma 5. If Assumption 1 holds, then

Proof. Applying Lemma 3 to the ensemble, we have

The integrand in the right-hand side of (17) being non-zero requires

The inequalities above imply Mt ≤ 1 + ∑ M -1 i =1 π k i ≤ M -1 + π k i , ∀ i ∈ { 1 , 2 , ..., M -1 } , which further derives Mt -M +1 ≤ π k i ≤ 1 , ∀ i ∈ { 1 , 2 , ..., M } . Thus, (17) can be rewritten as

Considering that p ( π k 1 , . . . , π k M -1 , π k M | a = 1) is bounded as Assumption 1 claims, let B k be its least upper bound. Then we have

This inequality combined with Lemma 4 derives p ( κ ( x ) = t | a = 1) ≤ ∑ K k =1 p ( π k E = t | a = 1) ≤ M M (1 -t ) M -1 · ∑ K k =1 B k , which is equivalent to the conclusion of this lemma since B = ∑ K k =1 B k .

A.2 Proof of Equation (2) of the Main Text (Preparing for Lemma 2)

According to the definition of selective risk, we have glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

According to the definition of conditional selective risk, the equation above can be rewritten as

where λ ( f, κ ; t ) := R ( f, κ ; t | a = 1) -R ( f, κ ; t | a = 0) . Applying (20) to the ensemble and considering λ ( f E , κ E ; t ) ≤ 1 , we derive an upper bound of the ensemble's selective risk

Thus, Equation (2) of the main text is proved.

glyph[negationslash]

glyph[negationslash]

glyph[negationslash]

A.3 Proof of Theorem 1

Proof. Step 1. We prove

Firstly, we prove lim t → 1 -R E ( t ) ≤ lim t → 1 -R ∗ ( t | a = 0) . According to Lemma 2, we have

where the second-to-last equality is due to L'Hospital's rule, and the last equality is due to lim t → 1 -Mγ (1 -t ) M -1 = 0 and lim t → 1 -p ( κ ∗ ( x ) = t, a = 0) = P ( a = 0) lim t → 1 -p ( κ ∗ ( x ) = t | a = 0) > 0 (Assumption 3).

Secondly, we prove lim t → 1 -R ∗ ( t ) > lim t → 1 -R ∗ ( t | a = 0) . According to (20), we have

Using Bayes' rule and L'Hospital's rule, we have

where the last inequality is due to Assumption 3. Applying this inequality along with lim t → 1 -λ ( f ∗ , κ ∗ ; t ) = lim t → 1 -[ R ∗ ( t | a = 1) -R ∗ ( t | a = 1)] > 0 (Assumption 2) to (24), we have

Combining (23) and (26), we derive (22).

Step 2. We prove the ensemble's confidence threshold and the member model's confidence threshold approach 1 when their coverage approach 0. Due to φ ∗ ( t ) = P ( κ ∗ ( x ) ≥ t ) = P ( κ ∗ ( x ) ≥ t | a = 0) P ( a = 0) + P ( κ ∗ ( x ) ≥ t | a = 1) P ( a = 1) , we derive

Because lim t → 1 -p ( κ ∗ ( x ) = t | a = 0) > 0 (Assumption 3), there exists δ 1 > 0 , such that p ( κ ∗ ( x ) = t | a = 0) > 0 , ∀ t ∈ (1 -δ 1 , 1) . Combining this with (27), we have dφ ∗ ( t ) dt < 0 , ∀ t ∈ (1 -δ 1 , 1) and thus φ ∗ ( t ) is reversible on (1 -δ 1 , 1) . Considering p ( κ E ( x ) = t | a = 0) = p ( κ ∗ ( x ) = t | a = 0) , we derive φ E ( t ) is reversible on (1 -δ 1 , 1) with the same reasoning as above.

Let φ -1 ∗ and φ -1 E be the reverse functions of φ ∗ and φ E on (1 -δ 1 , 1) , respectively. It is easy to see that φ -1 ∗ is a continuous function, and φ -1 ∗ (0) = 1 . Therefore,

Step 3. Combining (22), (28) and (29), we have R E ( φ -1 E ( φ )) < R ∗ ( φ -1 ∗ ( φ )) when φ → 0 + , which is equivalent to the result of Theorem 1.

For the ensemble, we similarly have

B Details of Experiments

B.1 Datasets

The experiments were conducted on multiple data sets of image classification and text classification. The image classification datasets are CIFAR-10, CIFAR-100, [29] and SVHN [30], whose image sizes are all 32 × 32 × 3 pixels. The datasets of text classification are MRPC [31], MNLI [32] and QNLI [33]. The task of MRPC is to judge whether two paragraphs of text are semantically equivalent. MNLI's task is to judge the inferential relationship between sentences (three categories). The task of QNLI is to determine whether a paragraph has the answer to a given question. The sizes of the training set, development set, and test set of each data set used in experiments are shown in Table 1. MNLI's development set and test set are divided into matched and mismatched parts. In the table, (m) represents matched, and (mm) represents mismatched. The matched parts are sampled from the same source as the training set, while the mismatched parts are sampled from different sources. Current selective classification only considers test samples from the same distribution as the training set, so only the matched parts are used in experiments. In addition, test sets of MRPC, QNLI, and MNLI are not accessible, so we use their development sets as test sets. According to [8, 9], since CIFAR-10, CIFAR-100 and SVHN originally had no development set, their development sets were 2000 samples randomly divided from corresponding test sets.

B.2 Model Implementations and Training Procedures

For image classification, the backbone model is VGG-16 [34] with Dropout [13], batch normalization [40]. It is trained in the same way as [9]. The model is optimized using SGD with an initial learning rate of 0.1 (the learning rate decays by half in every 25 epochs), the momentum of 0.9, weight decay of 0.0005, batch size of 128, and a total training epoch of 300. Data preprocessing includes data augmentation (random cropping and flip) and normalization. The implementations of the backbone model and data preprocessing are based on the official open-sourced implementation of SAT to ensure a fair comparison.

For text classification, the backbone model of selective classifiers is BERT-base [35]. Pretrained BERT-base is provided by the Huggingface Transformer Library [41]. It is trained/fine-tuned in the same way as [10], except on dataset MRPC. On QNLI and MNLI, the model is trained/fine-tuned using AdamW [42] for 3 epochs, with a learning rate of 2 × 10 -5 , batch size of 32, and the maximum input sequence length of 128. On MRPC, the model is trained/fine-tuned for 10 epoch, with other settings the same as those on QNLI and MNLI. This unique setting of training epoch is due to the small number of samples in MRPC, which makes the training require more epochs to reach convergence on MRPC.

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.05

0.10

0.15

0.20

0.25

selective risk

Deep Ensemble

CIFAR10

CIFAR100

SVHN

MRPC

QNLI

MNLI

member

ensemble

0.0

0.2

0.4

0.6

0.8

1.0

coverage

Gambler Ensemble

CIFAR10

CIFAR100

SVHN

MRPC

QNLI

MNLI

member

ensemble

0.0

0.2

0.4

0.6

0.8

1.0

coverage

SAT Ensemble

CIFAR10

CIFAR100

SVHN

MRPC

QNLI

MNLI

member

ensemble

0.0

0.2

0.4

0.6

0.8

1.0

coverage

Reg-curr Ensemble

CIFAR10

CIFAR100

SVHN

MRPC

QNLI

MNLI

member

ensemble

B.3 Hyperparameters of Selective Classifiers

For the hyperparameter o of Gambler, we tune o on validation sets in the same way as [8]. The value of o on each dataset is listed in Table 2. For the hyperparameter α of SAT, we set α = 0 . 99 , the same as [9]. For the hyperparameter λ of Reg-curr, we set λ = 0 . 05 .

C Evaluation of Deep Ensemble

To show that Deep Ensemble is still one of the state-of-the-art, it is necessary to evaluate Deep Ensemble again on selective classification. The reasons are as follows. First, the original paper of Deep Ensemble only reports the accuracy given a confidence threshold, which is not a commonly used metric in selective classification. Second, recent work in selective classification excluded Deep Ensemble as a baseline due to its heavy computational cost [10, 43]. Therefore, no comparison between recent methods and Deep Ensemble are available.

To fill this gap, we report the comparison results of Deep Ensemble and other recent work in this section. We compare Deep Ensemble with three non-ensemble methods: Gambler, SAT, Regcurr, and their corresponding ensembles. The ensemble method for a baseline is a uniformlyweighted soft voting of the classifiers and the confidence estimators: π E ( x ) = 1 M ∑ M i =1 π i ( x ) and M across each baseline on each dataset, so we focus on comparing Deep Ensemble with baselines' ensembles. Table 3 shows the AURCs of the ensembles (including Deep Ensemble and baselines'

κ E ( x ) = 1 M ∑ i =1 κ i ( x ) if the baseline has separate classifier and confidence estimator (i.e., the case of Gambler and SAT); and otherwise, the ensemble method is the same as Deep Ensemble (i.e., the case of Reg-curr). The details of baselines' hyperparameters are provided in Appendix B. Figure 6 shows the risk-coverage curves of Deep Ensemble, baselines, and their corresponding ensembles on each dataset, where each ensemble consists of five member models. The result shows that the ensemble has consistently lower selective risks than the member model under (almost) any coverage ensembles) and their corresponding member models. Among all ensembles, Deep Ensemble has the lowest AURCs on CIFAR-10, CIFAR-100, and MRPC; SAT ensemble has the lowest AURC on SVHN; and Reg-curr has the lowest AURCs on QNLI and MNLI. Table 4 and 5 shows the selective risks of ensembles under coverage 10%-100% on each dataset. The results show that no ensemble consistently outperforms others under all coverage on all datasets, indicating they have close performances but adopt different trade-offs between coverage and selective risk. In summary, the ensemble largely outperforms more recent methods and is competitive with their ensembles. We hope these results will draw more attention from researchers to the ensemble method.

In addition, we find that the Deep Ensemble can outperform SAT ensemble on SVHN as long as the label noise of SVHN is removed (see Appendix D). These results are surprising considering that Deep Ensemble only uses the MCP as the confidence score, while the baselines adopts more sophisticated techniques to train confidence estimators. Furthermore, we empirically explore further properties of selective classifier ensembles in Appendix D. We show that an ensemble with more member models has a better selective classification performance, and good classification performance does not necessarily imply good selective classification performance.

5

10

15

20

Number of Members

45

50

55

60

65

70

AURC / 1e-4

AURC on CIFAR-10

SR

Gambler

SAT

D Further Properties of Selective Classifier Ensemble

D.1 The Effect of Number of Members on Selective Classifier Ensemble

Weevaluate AURCs of the SR ensemble (i.e., Deep Ensemble), Gambler ensemble, and SAT ensemble of different numbers of members on CIFAR10, and find that an ensemble with more members has a better performance, but is less efficient. The results are shown in Figure 7. In most cases, the AURC on the test set of CIFAR-10 decreases as the number of members in the ensemble increases. In addition, as the number of members in the ensemble grows, the effect of adding one member drops. On the one hand, the result shows that an ensemble with a small number of members has good selective classification performance. On the other hand, it indicates that when the number of member models is large, increasing the number of members to improve the performance of the selective classification ensemble is inefficient.

D.2 Good Classification Performance Does Not Imply Good Selective Classification Performance

It is well known that the ensemble has better classification performance than an individual model, but this does not guarantee a better selective classification performance of the ensemble. To demonstrate

this, we design an SR model with a big backbone, and show that it has as good classification performance as an Deep Ensemble with a standard backbone but worse selective classification performance than an SR model with a standard backbone. The big backbone is designed to have twice as many filters in every convolutional layer and neurons in every fully connected hidden layer as those of the standard VGG-16, which is therefore called Big VGG-16 . It is easy to see that its number of parameters is approximately 2 2 = 4 times as many as that of standard VGG-16. We train an Deep Ensemble of 4 VGG-16s and an SR model with a backbone of Big VGG-16 on CIFAR-10 and show the evaluation results in Figure 8 and Table 6. Figure 8 shows that when coverage is high, the ensemble and the big individual model have similar selective risks, and especially, the classification error rates (i.e., selective risk of 100% coverage) of the ensemble and the big individual model are similar. However, when coverage is low, the big individual model has significantly higher selective risk than the ensemble. Table 6 shows that the AURC of Big VGG-16 is much higher than the ensemble of 4 VGG-16s and even higher than SR. In summary, we show that a selective classifier with a good classification performance is not guaranteed to have good selective classification performance, so the good selective classification performance of the ensemble is not a trivial result of its good classification performance.

D.3 The Effects of Label Noise of SVHN on Selective Classifier Ensembles

In this section, we compare the effect of label noise of SVHN on the Deep Ensemble with that on SAT ensemble, whose result might explain the abnormal experimental results (compared to results on other datasets) on SVHN in Section 6 of the main text. SVHN is not a clean dataset, and much more label noise can be detected in SVHN than in CIFAR-10 and CIFAR-100. Using the soft label of SAT

0.0

0.2

0.4

0.6

0.8

1.0

coverage

0.00

0.01

0.02

0.03

0.04

0.05

0.06

selective risk

4 VGG-16 Ensemble

Big VGG-16

[9], we detect label noise in SVHN, CIFAR-10, and CIFAR-100, and find that SVHN has significantly more label noise than CIFAR-10 and CIFAR-100. The result is presented in the following. In addition, it is known that SAT is robust to label noise [9], while SR is not so, so we conjecture that the label noise of SVHN is why the Deep Ensemble is inferior to SAT on SVHN.

We detect label noise with the help of the soft label of SAT. For a sample x i , the soft label of SAT [9], t i,y i , is used to measure x i 's learning difficulty. The soft label of SAT is initialized as 1 and updated

at every training epoch as below

where p θ ( Y | x ) is the predictive probability distribution of the classifier, y i is the label of x i , α is a hyperparameter. The smaller the t i,y i is, the lower the true class predictive probability of the classifier on x i during training time, indicating that x i is more difficult to learn. By selecting a percentage of samples with the lowest t i,y i , we get the most difficult samples to learn for the classifier, from which we can easily detect label noise manually.

In training sets of SVHN, CIFAR-10, and CIFAR-100, we detect label noise manually among the top-0.1% difficult (measured by the soft label of SAT) samples. The numbers of mislabeled samples detected in SVHN, CIFAR-10, and CIFAR100 are shown in Table 7. The result shows that SVHN has significantly more mislabeled samples detected than CIFAR-10 and CIFAR-100, indicating much more label noise in SVHN than in CIFAR-10 and CIFAR-100.

To verify the effect of label noise, the following experiments are designed. Firstly, we detect label noise manually among the 1% of the hardest-to-learn samples of SVHN training set and test set, using the soft label of SAT. Secondly, we remove the detected mislabeled samples from the original dataset. The remaining SVHN dataset is called the clean SVHN . Accordingly, the original dataset is called the original SVHN . Finally, we retrain and test the Deep Ensemble and SAT ensemble and compare their test results. In the second step, the reason for removing mislabeled samples rather than modifying them is that some samples cannot be classified even by humans, and some samples are not in the range of categories of SVHN. Thus, the label noise cannot be eliminated by modifying the labels but by removing mislabeled samples.

The test results of the Deep Ensemble and SAT ensemble on clean SVHN are shown in Table 8. It is not surprising that the AURCs of the Deep Ensemble and SAT ensemble are significantly lower on the clean SVHN than the original SVHN. Furthermore, on the clean SVHN, when the number of members is 5, the AURC of the Deep Ensemble is lower than that of SAT ensemble. Combined with results on the original SVHN, where the AURC of the Deep Ensemble is higher than that of SAT ensemble, we conclude that label noise in SVHN is why the Deep Ensemble has a higher AURC than SAT ensemble. In other words, label noise is why the Deep Ensemble performs worse in selective classification than SAT ensemble on SVHN.

In summary, by experiments, we show that the Deep Ensemble is not as robust to label noise as SAT ensemble, and label noise in SVHN is why the Deep Ensemble is not as good as SAT ensemble on SVHN. We construct the clean SVHN , which is SVHN without some mislabeled samples. On the clean SVHN, we compare the Deep Ensemble with SAT ensemble and find that the Deep Ensemble is superior to SAT ensemble in selective classification performance. Combined with former experimental results, we conclude that label noise in SVHN is why the Deep Ensemble is inferior to SAT on SVHN.

Considering the experimental results on the clean SVHN and previous experimental results on CIFAR10 and CIFAR-100 (see Table 8 and Table 1 of the main text), the Deep Ensemble is superior to SAT

ensemble in selective classification on clean image classification datasets, Thus, Deep Ensemble is the state-of-the-art selective classification method on clean image classification datasets, but is not as robust to label noise as SAT ensemble.

E The Lower Bound of Maximum φ 0 in Theorem 1

This section discusses the lower bound of maximum φ 0 mentioned in Theorem 1. We aim to calculate the maximum φ 0 's lower bound without training an ensemble (otherwise, we can measure it directly on the ensemble).

Optimization Problem. To calculate the lower bound of maximum φ 0 , we need to solve the following optimization problem

Using Lemma 2 and φ E ( t ) ≥ P ( κ E ( x ) ≥ t, a = 0) = P ( κ ∗ ( x ) ≥ t, a = 0) , we strengthen the constraints of (30) to obtain a looser lower bound of maximum φ 0 :

For the convenience of solving this optimization problem, we further strengthen the first constraint to obtain a looser lower bound:

Algorithm. We design Algorithm 1 to search for the solution to (31), where oracle tells whether a sample is definite, and this oracle can be implemented by an ensemble with M ′ ( M ′ glyph[lessmuch] M ) members. Since t E is determined by t ∗ according to the first constraint of (31), (31) can be reduced to a one-dimensional search problem. Our algorithm adopts a binary search for efficiency, although this method might provide a suboptimal solution. Because φ ∗ ( t ∗ ) is a non-increasing function of t ∗ , Algorithm first search for the minimum t ∗ by binary search. The procedure of Algorithm 1 in each iteration of the binary search is as follows.

Given current t ∗ , Algorithm 1 determines t E using SEARCHFORTAUENS (see Algorithm 2), a procedure that searches for t E ∈ [0 , 1] using binary search s.t. P ( κ ∗ ( x ) ≥ t E , a = 0) = φ ∗ ( t ∗ ) . Note that t E might not exist if t ∗ is so low that φ ∗ ( t ∗ ) > P ( a = 0) = sup t E ∈ [0 , 1] P ( κ ∗ ( x ) ≥ t, a = 0) . This problem will be addressed shortly.

Algorithm 1 exams whether t E exists. If t E exists, Algorithm 1 then examines whether the second constraint of (31) holds for current t ∗ and t E , which is implemented by VERIFYSECONDCONSTRAINT (see Algorithm 3).

Algorithm 1 A Lower Bound of Maximum φ 0 . N i =1 Algorithm 2 SEARCHFORTAUENS

If t E exists and the second constraint holds, Algorithm 1 searches for a smaller t ∗ in the left half feasible area; otherwise, Algorithm 1 searches for a greater t ∗ in the right half feasible area.

Once the binary search completes, Algorithm 1 returns the coverage of minimum t ∗ .

Experiment. To show that Algorithm 1 works in reality, we run this algorithm in the same setting as Section 6 of the main text. In this experiment, M = 5 , the oracle is implemented by another ensemble with two member models and outputs True if and only if the S < 10 -3 . Note that it is difficult to estimate B , because: 1. we need to train an ensemble with M models to estimate B , which is costly; 2. the domain of p ( π k 1 , ..., π k M | a = 1) is of high dimension, so the observed data points are sparse in this domain, which makes the estimation of B more difficult. Thus, we do not estimate B but try several hypothetical values of B to see at what B the lower bound of maximum φ 0 is big.

With different B s, we obtain different lower bounds of maximum φ 0 as Table 9 shows. We can see that when the order of magnitude of B is not too big, the lower bound of maximum φ 0 is large and stable, which makes it possible for our algorithm to be used in practical applications. This result also

Algorithm 3 VERIFYSECONDCONSTRAINT

Input: κ ∗ ( · ) ; the confidence threshold t ∗ ; t E ; the test set D = { ( x i , y i ) } N i =1 ; the oracle Ω : X → { 0 , 1 } that tells whether a sample is definite; the number of member models M ; and B . Output: True if and only if t ∗ and t E satisfy the second constraint of (31).

glyph[negationslash]

glyph[negationslash]

return 1 leftHandSide<rightHandSide

indicates the relationship between the ensemble's diversity and its selective classification performance. Since an ensemble with a smaller B seems to have more diversity over ambiguous samples, the result in Table 9 suggests that as long as the ensemble has enough diversity over ambiguous samples, the ensemble is guaranteed to have a lower selective risk than the member model under a considerable range of coverage.

F Distributions of Ambiguities

See Figure 9 for the results.

0.0

0.2

0.4

0.6

ambiguity

0

50

100

150

200

250

300

#samples

BERT-base on MRPC

(a)

0.0

0.2

0.4

0.6

ambiguity

0

500

1000

1500

2000

2500

3000

3500

4000

#samples

BERT-base on QNLI

(b)

0.0

0.1

0.2

0.3

0.4

0.5

0.6

ambiguity

0

1000

2000

3000

4000

5000

6000

#samples

BERT-base on MNLI

(c)

0.0

0.2

0.4

0.6

ambiguity

0

1000

2000

3000

4000

5000

6000

7000

8000

#samples

VGG16 on CIFAR-10

(d)

0.0

0.2

0.4

0.6

ambiguity

0

5000

10000

15000

20000

25000

#samples

VGG16 on SVHN

(e)

0.0

0.2

0.4

0.6

ambiguity

0

1000

2000

3000

4000

5000

#samples

VGG16 on CIFAR-100

(f)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on why the ensemble method works.\nAuthors prove that the ensemble has a lower selective risk than the member model for any coverage within a range, based on some assumptions.\nAuthors further conduct experiments on both computer vision and natural language processing tasks to verify proofs and assumptions.\n\nSTRENGTHS:\n1. Selective classification in Related Work is summarized well.\n\nWEAKNESSES:\n1. The ambiguity in Section 4 is measured based on which model, since there exit an ensemble and a member model. What is the meaning of \"ensembling on high-ambiguity samples?\" Do you fine-tune models on these \"high-ambiguity samples?\" If you fine-tune models based on \"high-ambiguity samples\", why the ambiguity is measured by another ensemble?\nPlease clarify the above question.\n\n2. Figure 2 is obtained from which experiments or just a sketch? If figure 2 just a sketch, it cannot be used to verify assumption 1.\nAuthors should conduct massive experiments/proofs to verify this assumption, instead of using a sketch.\nMoreover, what is the definition of \"correlated predictive probability distributions?\" Please clarify it by using mathematic forms.\nWhat is the mathematical definition of \"definite samples\"? How a sample can be considered as definite?\nWhat is the difference between \"definite samples\" and \"low-ambiguity samples\".\nAuthors use too many different terms without clear definition.\n\n3. Authors should conduct more experiments to verify assumption 1 on more dataset and more DNNs. Just one experiment cannot verify the convincingness of assumption 1.\nMoreover, for the verification of Assumptions 2 and 3, authors should use more architectures as the backbone, and construct more ensembles. Just one kind of backbone, and one ensemble cannot sufficiently verify Assumptions 2 and 3.\n\n4. Authors should conduct experiments to explore whether the number of member models affects the performance of the ensemble.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\n1. Some terms are not defined, and authors do not clarify the difference between different terms, which makes this paper hard to follow.\n\n2.Experimental results are not convincing enough, since authors just use one backbone and one ensemble for each task.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to investigate why ensemble models perform superiorly compared to member models. The authors conduct an empirical study to demonstrate their assumptions. They separate the data into two categories: high-ambiguity samples and low-ambiguity samples. Their findings reveal that while ensembling high ambiguity samples improve selective risk, the same process for low ambiguity samples yields similar results to those from the member model. Following these observations, they propose several assumptions about the models and samples, demonstrating that under these conditions, ensemble models can achieve a more favorable selective risk. To verify their assumptions and conclusions, the authors conduct additional experiments.\n\nSTRENGTHS:\n1. This paper offers a theoretical understanding of ensemble models, potentially enlightening future algorithm design.\n2. Compared to other theoretical frameworks,  the assumptions made in this paper are more realistic.\n3. The authors extensively employ empirical experiments to elucidate their motivations and assumptions, enhancing the paper's readability.\n\nWEAKNESSES:\n1. Despite the assumption seeming reasonable, the reviewer does not agree that the proof presented in this paper demonstrates the ensemble model's behavior. The proof heavily relies on the $\\lim_{t\\to 1^-}$ and the main result necessitates a minimal $\\phi_0$ value. Nonetheless, as depicted in Figure 5, even with a significantly large $\\phi_0$, the ensemble model outperforms the member model. Consequently, the theory proposed in this paper does not align with the results of their experiments.\n2. For a theoretical study, the methodologies employed in this paper lack novelty or intrigue.\n\nQUESTIONS:\nWhat is the intuition behind defining low-ambiguity and high-ambiguity samples as in Assumption 1? Given this assumption, it appears that a majority of samples would be classified as high-ambiguity.\n\nLIMITATIONS:\nThe theory provides in this paper can not demonstrate the ensemble model behavior, thus limits the impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper provides a rigorous analysis of the reason for the success of the ensemble method, which includes both empirical evidence and theoretical proof. They found that the power of the ensemble method comes mostly from top-ambiguity samples where the member model diverges, and they provide theoretical evidence as well by proving that the ensemble has a lower selection risk than its member model in certain cases.

STRENGTHS:
The paper is well-motivated and well-written.

The experiments are well-designed and thorough.

The theoretical analysis is well-formulated and sound.

The results are insightful and sensible and provide reassuring evidence of the use of ensemble methods in practice.

WEAKNESSES:
The discussion around Figure 2 is in fact a bit hard to digest, the author might want to think about a better way to convey this explanation.

QUESTIONS:
I don't have any questions.

LIMITATIONS:
Yes, the author has addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present an analysis of deep ensembles in the context of selective classification, where a classifier has an option to abstain from providing a response in situations where it lacks confidence in its predictions. They prove that under reasonable assumptions, the performance of a deep ensemble in selective classification is guaranteed to beat that of its component members under the zero-one loss. They go on to justify their assumptions and provide experimental evidence for their claims, and finally demonstrate that deep ensembles are competitive with currently used methods in selective classification.\n\nSTRENGTHS:\n- Compelling analysis. The authors provide an insightful analysis of ensemble performance that operates under reasonable assumptions that they later justify. Their approach offers theoretical insight into an interesting application of ensembles, namely selective classification. \n- Related work. To my understanding, the authors provide a thorough and easy to understand survey of other methods in the field of selective classification.\n\nWEAKNESSES:\n- I found section 4 of the paper to be weaker than some of the other sections. One instantiation of low vs. high ambiguity samples corresponds to having ensemble members which make identical predictions on 90% of the data, and different predictions only on the remaining 10%, which would also explain the results in Figure 1. While such an effect would still be consistent with the analysis that follows, it would be important to report in its own right. I would like to understand what the distribution of ambiguous samples looks like.\n- Although illustrative, I do not follow how the results in Figure 5 and Table 1 correspond to the rest of the paper. In particular, how do these results relate to the importance of ambiguous samples in the performance of deep ensembles?\n\nQUESTIONS:\n- Figure 2 provides a useful intuition for the analysis that follows. Is it possible to provide a version of Figure 2 with real data as a justification of your assumptions as well? \n- It may be worthwhile to include references to more recent work in ensemble performance decompositions for metrics besides 0-1 loss: \n\t- https://arxiv.org/pdf/2206.10566.pdf\n\t- https://proceedings.mlr.press/v151/ortega22a.html\n\t- https://arxiv.org/abs/2301.03962, \n\t- https://openreview.net/forum?id=6sBiAIpkUiO\n- The choice of confidence estimator seems reasonable, but I could imagine cases where the maximum confidence does not correspond to the chosen output. Out of interest, could the analysis apply to other confidence estimators? \n- In the MRPC dataset, the ensemble has higher selective risk than the member at ~30% coverage, as discussed in the text. Is this a violation of assumptions, or is $\\phi_0$ lower than $30%$ for this dataset? \n- From your analysis, my understanding is that the ensemble should outperform any individual ensemble member (which is better than the average performance, as guaranteed by Jensen for convex losses). Is Figure 5 showing a comparison to the average single model, or the best, or a randomly selected one?\n\nLIMITATIONS:\nAs discussed in the text, the analysis is limited to selective classification within a range of coverages, and extensions to more general settings are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}