SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The first term of this upper bound is the selective risk of a member model on definite samples, and the second one is negatively related to the diversity among member models. When member models' performance on definite samples or the diversity improves, this upper bound decreases, which is a reasonable result.
5.5 Main Result
Based on Lemma 2, we can prove that Deep Ensemble has a lower selective risk than a member model for any target coverage that is in a range (Theorem 1, see Appendix A for the proof).
Theorem 1. If Assumptions 1, 2, and 3 hold, then  $\exists \phi_0 \in (0,1)$  such that  $\forall \phi \in (0,\phi_0)$ ,  $\phi_E(t_E) = \phi_*(t_*) = \phi \Rightarrow R_E(t_E) < R_*(t_*)$ .
Furthermore, it is interesting to figure out the maximum  $\phi_0$  in Theorem 1. In Appendix E, we use Lemma 2 to estimate the maximum  $\phi_0$  (without accessing the ensemble) and find that it can be large, say  $>50\%$ , on several datasets. When we instead have the access to the ensemble in experiments, the actual maximums of  $\phi_0$  on most datasets are 1, i.e., the largest possible value of  $\phi_0$ . These results seem interesting, and we will intuitively explain them in the next section.
This section verifies the assumptions and analysis results in realistic tasks, and then compares Deep Ensemble with several SOTA methods and their ensembles in image classification and text classification tasks. However, we do not verify Assumption 1 in the following because the empirical result in Section 4 has shown that Assumption 1 is a reasonable approximation that does not compromise the performance of the ensemble.
Datasets. Following [9], we used CIFAR-10, CIFAR-100, and SVHN for image classification tasks. Following [10], we used MRPC, MNLI, and QNLI for text classification tasks. More details of the datasets used in our experiments are described in Appendix B.
Models. Following [9, 10], we use VGG-16 [34] and BERT-base [35] as the backbones of selective classifiers for image classification and text classification, respectively. Each member model is built on basis of a backbone model, and an ensemble consists of five member models by default in experiments. More details of the backbone models and their training procedures are provided in Appendix B.
Verification of Assumptions 2 and 3. We use samples with ambiguity lower than  $\epsilon (\epsilon >0)$  (termed as low-ambiguity samples) to represent definite samples and use samples with ambiguity greater than or equal to  $\epsilon$  (termed as high-ambiguity samples) to represent ambiguous samples in experiments as we use definite/ambiguous samples to approximate low/high-ambiguity samples in theory. We choose  $\epsilon = 10^{-3}$  for datasets of image classification and  $\epsilon = 10^{-2}$  for datasets of text classification. Figure 3 shows a member model's conditional selective risks over low-ambiguity samples and high-ambiguity samples against a range of confidence thresholds near 1. The results show that the member model's
(f)
Figure 4: The histograms of the member model's confidence scores and the ensemble's confidence scores on low-ambiguity samples (denoted as def.) and on high-ambiguity samples (denoted as amb.) for each dataset.
selective risk over low-ambiguity samples is consistently lower than that over high-ambiguity samples given confidence thresholds near 1, which verifies Assumption 2. Figure 4 shows histograms of confidence scores of the member model (as well as the ensemble) on low-ambiguity samples and on high-ambiguity samples. As we can see, the number of low-ambiguity samples and the number of high-ambiguity samples are non-zero in the top bin on each dataset, which verifies Assumption 3. In summary, Assumption 2 and 3 hold on all datasets.
Verification of Lemma 1. Figure 4 shows histograms of the ensemble's confidence scores on low-ambiguity samples and on high-ambiguity samples. As we can see, the ensemble's confidence score over high-ambiguity samples has little distribution in the top bin, which is tremendously lower than the corresponding distribution of the member model, while the histogram of confidence score over low-ambiguity samples has almost no change throughout ensembling. This result is consistent with Lemma 1 and our intuition in Figure 2.
Verification of Theorem 1. Figure 1 already shows the risk-coverage curves of Deep Ensemble and a member model on each dataset. As we can see, on each dataset except MRPC, the ensemble has a consistently lower selective risk than the member model under any coverage; on MRPC, the ensemble has a lower selective risk than the member model except for coverage of around  $30\%$  and around  $90\%$ . These results are consistent with Theorem 1.
Despite of this consistency, the theorem does not explain the lower selective risk of the ensemble given a large coverage, say  $70\%$ . This is somehow resolved by the analysis in Appendix E. Here, instead of
3The exception on MRPC might be due to the sparsity of data. The test set of MRPC only contains about 400 examples. Even worse, when the coverage is  $30\%$ , the number of accepted examples is much smaller (about 120). These data are inadequate to accurately estimate the selective risk, leading to its high variance. For example, mis-classifying an example by chance could raise the selective risk by about one percent. This high variance might explain why the risk-coverage curve of the member model shakes violently and goes below that of the ensemble several times when the coverage is low. This problem seems irresolvable since we cannot sample more MRPC data to reduce this variance.
diving into theory, we intuitively explain the experimental results as follows. As Lemma 1 claims,  $P(\text{the ensemble yielding confidence} \geq t|\text{an ambiguous input example})$  is  $O(1 - t)^M (t \to 1^-)$ , where  $M$  is the number of ensemble members. Therefore, the ensemble hardly provides an ambiguous sample with confidence close to 1. On the contrast, definite samples are always assigned confidence scores close to 1, as the the experimental results show in Figure 4. By this mean, the ensemble stratifies the definite samples and ambiguous sample by their confidence scores, where the definite samples reside on a thin higher layer of confidence than the ambiguous samples (see the rightmost black-edged bars vs. red-edged bars in Figure 4). Combined with the low risk (of both the member model and the ensemble) on definite samples, this stratification lowers the selective risk when the coverage is around the proportion of definite samples in the dataset. Considering the large amount of definite samples (see the heights of black-edged bars in Figure 4), the ensemble model will exhibit lower selective risk than the member model given a considerably large coverage.
In the explanation above, the key factor that leads to the lower selective risk of the ensemble given a large coverage is the experimental fact that the definite samples are large-amounted and always assigned confidence close to 1. This fact is not involved as an assumption in the theory since it seems a very strong assumption (though it holds throughout our experiments). We guess this is attributed to the low bias of DNNs (from a bias-variance perspective), which might be a widespread property of DNNs. Therefore, it would be an interesting direction for future work to strengthen our theory by exploiting this fact.
Extended Experimental Settings. To further validate our analysis, we extended the experimental settings on three dimensions: 1. model architecture (to ResNet [36], AlexNet [37], and DenseNet[38]); 2. dataset (to ImageNet100, a subset of ImageNet [39]); 3. number of member models in the ensemble (to 20).
The results are shown in Figure 5. As the figure shows, Assumptions 1, 2, and 3 are consistently verified across various experiment settings. The results indicate that our assumptions might reflect the general characteristics of DNNs. For example, Assumptions 2 and 3 might be attributed to the low bias of DNNs (from a bias-variance perspective) due to DNNs' large model capacity.
These results were unknown when we established our analysis. However, all of them reproduce similar results to the previous results in new settings, further confirming our analysis in practical settings.
We prove that under some assumptions, Deep Ensemble has a lower selective risk than the member model for any target coverage within a range. Although the metrics of selective classification are non-convex, we complete the proof with the help of several assumptions motivated by empirical observations, e.g., the performance improvement of the ensemble dominantly comes from high-ambiguity samples. The assumptions and analysis results are well supported by the experimental results on multiple datasets of image classification tasks and text classification tasks.
Our analysis may benefit the analysis of the Deep Ensemble for standard classification. The previous analyses of the randomization-based ensemble for standard classification require some impractical assumptions [22]. On the contrary, this paper's assumptions are a good approximation of practical settings (see Section 6). Moreover, the standard classification is a subset of selective classification, i.e., selective classification with coverage of 1 (not covered by our analysis result). Thus, our analysis may motivate the analysis of the Deep Ensemble for standard classification in practical settings.
Other possible directions for future work include: 1. extending the  $(0,\phi_0)$  mentioned in Theorem 1 to  $(0,1]$ , i.e., proving that the ensemble has a lower selective risk than the member model under any coverage; 2. adapting the analysis to other selective classifier ensembles, e.g., the ensemble of SAT; 3. relaxing the assumptions to generalize the application scope of our analysis.
This work has been supported by the National Key Research and Development Program of China (No. 2022YFB2702502) and the National Natural Science Foundation of China (No. 62076231,
(j)
(n)
Figure 5: The results of the extended experiments. Here, the 1st/2nd/3rd row reproduces Figure 1/3/4 of the paper in new experiment settings, respectively; each new experiment setting corresponds to a column. Counting from the left, the first three columns show the results of extending model architectures to AlexNet, ResNet50, and DenseNet22 (fixing the dataset to CIFAR10); the 4th column shows the results of extending datasets to ImageNet100 (using the model ResNet50); and the 5th column shows the results of extending the number of member models to 20 (using VGG16 on CIFAR10).
62206265). We thank Keliang Li for his feedback on our theoretical analysis. We also thank anonymous reviewers for their valuable comments.
[1] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. In Advances in Neural Information Processing Systems, 2017.
[2] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representation, 2017.
[3] Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Advances in neural information processing systems, 2018.
[4] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, 2015.
[5] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, 2016.
[6] Yarin Gal. Uncertainty in deep learning. PhD thesis, University of Cambridge, 2016.
[7] Yonatan Geifman and Ran El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In International Conference on Machine Learning, 2019.
[28] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
[29] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[30] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[31] Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop on Paraphrasing, 2005.
[32] Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), 2018.
[33] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
[35] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
[36] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE conference on computer vision and pattern recognition, 2016.
[37] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, 2012.
[38] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In IEEE conference on computer vision and pattern recognition, 2017.
[39] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In IEEE conference on computer vision and pattern recognition, 2009.
[40] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, 2015.
[41] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020.
[42] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[43] Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H Abdi. Towards better selective classification. 2023.
The proof of the second inequality in Lemma 1, i.e.,
$$
P \left(\kappa_ {E} (x) \geq t \mid a = 1\right) \leq M ^ {M - 1} B (1 - t) ^ {M}, \tag {5}
$$
can be reduced to the proof of the first inequality, i.e.,
$$
p \left(\kappa_ {E} (x) = t \mid a = 1\right) \leq M ^ {M} B (1 - t) ^ {M - 1}. \tag {6}
$$
The reason is as follows. If (6) holds, then we have
$$
\begin{array}{l} P (\kappa_ {E} (x) \geq t | a = 1) = \int_ {t} ^ {1} p (\kappa_ {E} (x) = t ^ {\prime} | a = 1) d t ^ {\prime} \\ \leq \int_ {t} ^ {1} M ^ {M} B (1 - t ^ {\prime}) ^ {M - 1} \\ = M ^ {M - 1} B (1 - t) ^ {M}, \\ \end{array}
$$
which directly derives (5). Therefore, we only need to show (6) at following.
Firstly, we derive the PDF of the average of multiple continuous random variables expressed by the PDFs of these random variables (Lemma 3), which helps us to analyze the PDF of the ensemble's predictive probabilities.
Lemma 3. Let  $\mathrm{X}_1, \mathrm{X}_2, \ldots, \mathrm{X}_M$  be  $M$  continuous random variables, and their average is  $\mathrm{X}_{\mathrm{avg}} := \frac{1}{M} \sum_{i=1}^{M} \mathrm{X}_i$ . Then the PDF of  $\mathrm{X}_{\mathrm{avg}}^4$  is
$$
p _ {\mathrm {X} _ {\text {a v g}}} \left(x _ {\text {a v g}}\right) = M \int_ {\mathbb {R} ^ {M - 1}} \mathrm {d} x _ {1} \mathrm {d} x _ {2} \dots \mathrm {d} x _ {M - 1} \cdot p _ {\vec {\mathrm {X}}} \left(x _ {1}, x _ {2}, \dots , x _ {M - 1}, M x _ {\text {a v g}} - \sum_ {i = 1} ^ {M - 1} x _ {i}\right), \tag {7}
$$
where  $p_{\vec{\mathrm{X}}}$  is  $p_{\mathrm{X}_1,\mathrm{X}_2,\dots,\mathrm{X}_M}$  for short.
Proof. The distribution function of  $\mathbf{X}_{\mathrm{avg}}$  is
$$
\begin{array}{l} F _ {\mathrm {X} _ {\text {a v g}}} \left(x _ {\text {a v g}}\right) = \int_ {\sum_ {i} x _ {i} \leq M x _ {\text {a v g}}} \mathrm {d} x _ {1} \dots \mathrm {d} x _ {M - 1} \mathrm {d} x _ {M} \cdot p _ {\vec {\mathrm {X}}} \left(x _ {1}, \dots , x _ {M}\right) \\ = \int_ {\mathbb {R} ^ {M - 1}} \mathrm {d} x _ {1} \dots \mathrm {d} x _ {M - 1} \int_ {- \infty} ^ {M x _ {\text {a v g}} - \sum_ {i = 1} ^ {M - 1} x _ {i}} \mathrm {d} x _ {M} \cdot p _ {\vec {X}} (x _ {1}, \dots , x _ {M}). \\ \end{array}
$$
Let  $x_{M} = u - \sum_{i=1}^{M-1} x_{i}$ , then the integral above is equal to
$$
\begin{array}{l} \int_ {\mathbb {R} ^ {M - 1}} \mathrm {d} x _ {1} \dots \mathrm {d} x _ {M - 1} \int_ {- \infty} ^ {M x _ {\text {a v g}}} \mathrm {d} u \cdot p _ {\vec {X}} \left(x _ {1}, \dots , x _ {M - 1}, u - \sum_ {i = 1} ^ {M - 1} x _ {i}\right) \\ = \int_ {- \infty} ^ {M x _ {\mathrm {a v g}}} \mathrm {d} u \int_ {\mathbb {R} ^ {M - 1}} \mathrm {d} x _ {1} \dots \mathrm {d} x _ {M - 1} \cdot p _ {\vec {\mathbf {X}}} \left(x _ {1}, \dots , x _ {M - 1}, u - \sum_ {i = 1} ^ {M - 1} x _ {i}\right). \tag {8} \\ \end{array}
$$
The PDF of  $\mathbf{X}_{\mathrm{avg}}$  is the derivative of  $F_{\mathrm{X}_{\mathrm{avg}}}$ , which, combined with (8) derives
$$
\begin{array}{l} p _ {\mathrm {X _ {a v g}}} (x _ {\mathrm {a v g}}) = F _ {\mathrm {X _ {a v g}}} ^ {\prime} (x _ {\mathrm {a v g}}) \\ = \frac {\mathrm {d} (M x _ {\mathrm {a v g}})}{\mathrm {d} x _ {\mathrm {a v g}}} \cdot \frac {\mathrm {d} F _ {\mathrm {X} _ {\mathrm {a v g}}}}{\mathrm {d} (M x _ {\mathrm {a v g}})} \\ = M \int_ {\mathbb {R} ^ {M - 1}} \mathrm {d} x _ {1} \dots \mathrm {d} x _ {M - 1} \cdot p _ {\vec {X}} (x _ {1}, \ldots , x _ {M - 1}, M x _ {\mathrm {a v g}} - \sum_ {i = 1} ^ {M - 1} x _ {i}), \\ \end{array}
$$
which is exactly (7).
4A PDF has a subscript to denote which random variable this PDF belongs to.
Secondly, we show the relationship between the PDF of confidence score and the PDFs of predictive probabilities. Note that the confidence score of an SR model is the maximum predictive probability, so the following lemma bounds the PDF of confidence by PDFs of predictive probabilities.
Lemma 4. Let  $\Pi^k$  ( $1 \leq k \leq K$ ) be  $K$  continuous random variables, and  $\mathbf{C} := \max_k \Pi^k$ . Then we have
$$
p _ {C} (\kappa) \leq \sum_ {k = 1} ^ {K} p _ {\Pi^ {k}} (\kappa). \tag {9}
$$
Proof. First of all, we prove  $\forall \kappa_1,\kappa_2,\kappa_1 <   \kappa_2$
$$
F _ {\mathrm {C}} \left(\kappa_ {2}\right) - F _ {\mathrm {C}} \left(\kappa_ {1}\right) \leq \sum_ {k = 1} ^ {K} F _ {\Pi^ {k}} \left(\kappa_ {2}\right) - F _ {\Pi^ {k}} \left(\kappa_ {1}\right) \tag {10}
$$
$$
F _ {\mathrm {C}} (\kappa) = F _ {\Pi^ {1}, \dots , \Pi^ {K}} (\kappa , \dots , \kappa) = \int_ {(- \infty , \kappa ] ^ {K}} \mathrm {d} \pi^ {1} \dots \mathrm {d} \pi^ {K} p _ {\Pi^ {1}, \dots , \Pi^ {K}} (\pi^ {1}, \dots , \pi^ {K}),
$$
so the left-hand side of (10) is
$$
\begin{array}{l} \int_ {(- \infty , \kappa_ {2} ] ^ {K}} \mathrm {d} \pi^ {1} \dots \mathrm {d} \pi^ {K} p _ {\Pi^ {1}, \dots , \Pi^ {K}} \left(\pi^ {1}, \dots , \pi^ {K}\right) \\ - \int_ {(- \infty , \kappa_ {1} ] ^ {K}} \mathrm {d} \pi^ {1} \dots \mathrm {d} \pi^ {K} p _ {\Pi^ {1}, \dots , \Pi^ {K}} (\pi^ {1}, \dots , \pi^ {K}) \\ = \int_ {(- \infty , \kappa_ {2} ] ^ {K} \backslash (- \infty , \kappa_ {1} ] ^ {K}} \mathrm {d} \pi^ {1} \dots \mathrm {d} \pi^ {K} p _ {\Pi^ {1}, \dots , \Pi^ {K}} \left(\pi^ {1}, \dots , \pi^ {K}\right), \tag {11} \\ \end{array}
$$
where the last equality is due to  $(-\infty, \kappa_1] \subset (-\infty, \kappa_2]$ , and the right-hand side of (10) is
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \int_ {[ \kappa_ {1}, \kappa_ {2} ]} \mathrm {d} \pi^ {k} p _ {\Pi^ {k}} (\pi^ {k}) \\ = \sum_ {k = 1} ^ {K} \int_ {\mathbb {R} ^ {k - 1} \times [ \kappa_ {1}, \kappa_ {2} ] \times \mathbb {R} ^ {K - k}} \mathrm {d} \pi^ {1} \dots \mathrm {d} \pi^ {K} \cdot p _ {\Pi^ {1}, \dots , \Pi^ {K}} (\pi^ {1}, \dots , \pi^ {K}) \\ \geq \int_ {\bigcup_ {k = 1} ^ {K} \mathbb {R} ^ {k - 1} \times \left[ \kappa_ {1}, \kappa_ {2} \right] \times \mathbb {R} ^ {K - k}} \mathrm {d} \pi^ {1} \dots \mathrm {d} \pi^ {K} \cdot p _ {\Pi^ {1}, \dots , \Pi^ {K}} \left(\pi^ {1}, \dots , \pi^ {K}\right), \tag {12} \\ \end{array}
$$
where the last inequality is because  $\mathbb{R}^{k - 1}\times [\kappa_1,\kappa_2]\times \mathbb{R}^{K - k}$  for different  $k,1\leq k\leq K$  , may have an intersection. To prove (10), we only need to prove that the right-hand side of (11) is less than or equal to the right-hand side of (12), which is equivalent to prove
$$
\left(- \infty , \kappa_ {2} \right] ^ {K} \backslash \left(- \infty , \kappa_ {1} \right] ^ {K} \subset \bigcup_ {k = 1} ^ {K} \mathbb {R} ^ {k - 1} \times \left[ \kappa_ {1}, \kappa_ {2} \right] \times \mathbb {R} ^ {K - k}. \tag {13}
$$
Now we prove (13).  $\forall (\pi^1,\dots ,\pi^K)\in (-\infty ,\kappa_2]^K\setminus (-\infty ,\kappa_1]^K$  , we have
$$
\forall k, 1 \leq k \leq K, \pi^ {k} \leq \kappa_ {2}, \tag {14}
$$
$$
\exists k _ {0}, 1 \leq k _ {0} \leq K, \pi^ {k _ {0}} > \kappa_ {1}, \tag {15}
$$
where (15) is because if all  $\pi^k$  is less than or equal to  $\kappa_{1}$  instead, then  $(\pi^{1},\ldots ,\pi^{K})\in (-\infty ,\kappa_{1}]^{K}$ , which contradicts with  $(\pi^1,\dots ,\pi^K)\in (-\infty ,\kappa_2]^K\setminus (-\infty ,\kappa_1]^K$ . Thus,  $\pi^{k_0}\in [\kappa_1,\kappa_2]$ , so
$$
\left(\pi^ {1}, \dots , \pi^ {K}\right) \in \mathbb {R} ^ {k _ {0} - 1} \times \left[ \kappa_ {1}, \kappa_ {2} \right] \times \mathbb {R} ^ {K - k _ {0}} \subset \bigcup_ {k = 1} ^ {K} \mathbb {R} ^ {k - 1} \times \left[ \kappa_ {1}, \kappa_ {2} \right] \times \mathbb {R} ^ {K - k},
$$
which is precisely (13), and therefore (10) is proved.
With (10) and the definition of derivatives, it is easy to see that  $F_{\mathbb{C}}^{\prime}(\kappa) \leq \sum_{k=1}^{K} F_{\Pi^{k}}^{\prime}(\kappa)$ , which is equivalent to  $p_{\mathbb{C}}(\kappa) \leq \sum_{k=1}^{K} p_{\Pi^{k}}(\kappa)$ . Thus, Lemma 4 is proved.
Based on the lemmas above, we have the following lemma, which is precisely (6). When (6) is proved, the proof of Lemma 1 completes.
Lemma 5. If Assumption 1 holds, then
$$
p \left(\kappa_ {E} (x) = t \mid a = 1\right) \leq M ^ {M} B (1 - t) ^ {M - 1}. \tag {16}
$$
Proof. Applying Lemma 3 to the ensemble, we have
$$
p \left(\pi_ {E} ^ {k} = t \mid a = 1\right) = M \int_ {\mathbb {R} ^ {M - 1}} \mathrm {d} \pi_ {1} ^ {k} \dots \mathrm {d} \pi_ {M - 1} ^ {k} \cdot p \left(\pi_ {1} ^ {k}, \dots , \pi_ {M - 1} ^ {k}, \pi_ {M} ^ {k} = M t - \sum_ {i = 1} ^ {M - 1} \pi_ {i} ^ {k} \mid a = 1\right), \tag {17}
$$
The integrand in the right-hand side of (17) being non-zero requires
$$
\left\{ \begin{array}{l} 0 \leq \pi_ {i} ^ {k} \leq 1, i = 1, 2, \dots , M - 1 \\ 0 \leq M t - \sum_ {i = 1} ^ {M - 1} \pi_ {i} ^ {k} \leq 1 \end{array} \right. \tag {18}
$$
The inequalities above imply  $M t \leq 1 + \sum_{i=1}^{M-1} \pi_i^k \leq M - 1 + \pi_i^k, \forall i \in \{1,2,\dots,M-1\}$ , which further derives  $M t - M + 1 \leq \pi_i^k \leq 1, \forall i \in \{1,2,\dots,M\}$ . Thus, (17) can be rewritten as
$$
p (\pi_ {E} ^ {k} = t | a = 1) = M \int_ {[ M t - M + 1, 1 ] ^ {M - 1}} \mathrm {d} \pi_ {1} ^ {k} \dots \mathrm {d} \pi_ {M - 1} ^ {k} \cdot p (\pi_ {1} ^ {k}, \ldots , \pi_ {M - 1} ^ {k}, \pi_ {M} ^ {k} = M t - \sum_ {i = 1} ^ {M - 1} \pi_ {i} ^ {k} | a = 1).
$$
Considering that  $p(\pi_1^k, \ldots, \pi_{M-1}^k, \pi_M^k | a = 1)$  is bounded as Assumption 1 claims, let  $B_k$  be its least upper bound. Then we have