SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Pranjal Awasthi, Natalie Frank, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Calibration and consistency of adversarial surrogate losses. CoRR, abs/2104.09658, 2021. URL https://arxiv.org/abs/2104.09658.
[2] Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. A finer calibration analysis for adversarial robustness. CoRR, abs/2105.01550, 2021. URL https://arxiv.org/abs/2105.01550.
[3] Gagan Bansal, Besmira Nushi, Ece Kamar, Eric Horvitz, and Daniel S. Weld. Is the most accurate AI the best teammate? optimizing AI for teamwork. In AAAI, 2021.
[4] Han Bao, Clayton Scott, and Masashi Sugiyama. Calibrated surrogate losses for adversarially robust classification. In  $COLT$ , volume 125 of Proceedings of Machine Learning Research, pages 408-451. PMLR, 2020.
[5] Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. J. Mach. Learn. Res., 9:1823-1840, 2008.
[6] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
[7] Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie GU, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In NeurIPS, 2022.
[8] Nontawat Charoenphakdee, Zhenghang Cui, Yivan Zhang, and Masashi Sugiyama. Classification with rejection based on cost-sensitive classification. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 1507-1517. PMLR, 2021.
[9] Mohammad-Amin Charusaie, Hussein Mozannar, David A. Sontag, and Samira Samadi. Sample efficient learning of predictors that complement humans. In ICML, 2022.
[10] C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970. doi: 10.1109/TIT.1970.1054406.
[11] Scott Clayton. Calibrated asymmetric surrogate losses. Electronic Journal of Stats, 6:238-238, 2012.
[12] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In NeurIPS, pages 1660-1668, 2016.
[13] Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In ALT, volume 9925, pages 67-82, 2016.
[14] Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. Automated hate speech detection and the problem of offensive language. In ICWSM, 2017.
[15] Abir De, Nastaran Okati, Ali Zarezade, and Manuel Gomez Rodriguez. Classification under human assistance. In AAAI, 2021.
[16] Jessica Finocchiaro, Rafael M. Frongillo, and Bo Waggoner. An embedding framework for consistent polyhedral surrogates. In NeurIPS, pages 10780-10790, 2019. URL https://proceedings.neurips.cc/paper/2019/bitnet/9ec51f6eb240fb631a35864e13737bca-Abstract.html.
[17] Wei Gao and Zhi-Hua Zhou. On the consistency of multi-label learning. Artif. Intell., 199-200: 22-44, 2013.
[18] Sorin Mihai Grigorescu, Bogdan Trasnea, Tiberiu T. Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. J. Field Robotics, 37(3):362-386, 2020.
[19] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
[40] Mark D. Reid and Robert C. Williamson. Composite binary losses. J. Mach. Learn. Res., 11: 2387-2422, 2010.
[41] Clayton Scott. Surrogate losses and regret bounds for cost-sensitive classification with example-dependent costs. In Lise Getoor and Tobias Scheffer, editors, ICML, pages 153-160. Omnipress, 2011. URL https://icml.cc/2011/papers/138_icmlpaper.pdf.
[42] Eleni Straitouri, Lequn Wang, Nastaran Okati, and Manuel Gomez Rodriguez. Improving expert predictions with prediction sets, 2022.
[43] Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classification methods. J. Mach. Learn. Res., 8:1007-1025, 2007.
[44] Rajeev Verma and Eric T. Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In ICML, 2022.
[45] Rajeev Verma, Daniel Barrejón, and Eric T. Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In AISTATS, 2023.
[46] Robert C. Williamson, Elodie Vernet, and Mark D. Reid. Composite multiclass losses. J. Mach. Learn. Res., 17:223:1-223:52, 2016.
[47] Guoqiang Wu, Chongxuan Li, Kun Xu, and Jun Zhu. Rethinking and reweighting the univariate losses for multi-label ranking: Consistency and generalization. CoRR, abs/2105.05026, 2021. URL https://arxiv.org/abs/2105.05026.
[48] Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 10727-10735. PMLR, 2020.
[49] Ming Yuan and Marten H. Wegkamp. Classification methods with reject option based on convex risk minimization. J. Mach. Learn. Res., 11:111-130, 2010.
[50] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In BMVC, 2016.
[51] Mingyuan Zhang, Harish Guruprasad Ramaswamy, and Shivani Agarwal. Convex calibrated surrogates for the multi-label f-measure. In ICML, volume 119 of Proceedings of Machine Learning Research, pages 11246-11255. PMLR, 2020.
[52] Tong Zhang. Statistical analysis of some multi-category large margin classification methods. J. Mach. Learn. Res., 5:1225-1251, 2004.
We first prove that if there is an L2D probability estimator  $\hat{\eta}$  for  $L_{\phi}$ , then we can reconstruct a probability estimator  $\tilde{\eta}$  for multiclass classification with  $\phi$  if  $\pmb{\eta}$  is in  $\tilde{\Delta}^{K + 1}$ , which is the collection of all the elements  $\beta \in \Delta^{K + 1}$  with  $\beta_{K + 1} \leq \frac{1}{2}$ . Then we show that according to the symmetry of  $\phi$ , we can extend this estimator to  $\Delta^{K + 1}$ , and then construct an unbounded L2D probability estimator for  $L_{\phi}$ .
Proof. Denote by  $R_{L_{\phi}}(\pmb{u}, \pmb{\eta}(\pmb{x}), \operatorname*{Pr}(M = Y | X = \pmb{x})) = \sum_{y=1}^{K} \eta(\pmb{x})_y \phi(\pmb{u}, y) + \operatorname*{Pr}(M = Y | X = \pmb{x}) \phi(\pmb{u}, K+1)$  the conditional risk for L2D. Denote by  $\hat{\eta}$  a probability estimator for  $L_{\phi}$  that  $\hat{\eta}(\pmb{u}^*) = [\pmb{\eta}(\pmb{x}); \operatorname*{Pr}(M = Y | X = \pmb{x})]$  for all  $\pmb{u}^* \in \operatorname*{argmin}_{\pmb{u}} R_{L_{\phi}}(\pmb{u}, \pmb{\eta}(\pmb{x}), \operatorname*{Pr}(M = Y | X = \pmb{x}))$ , we can learn that  $\Delta^K \times [0,1]$  is in the range of  $\hat{\eta}$ . Then it is easy to verify that  $\tilde{\eta}(\pmb{u}) = \frac{\hat{\eta}(\pmb{u})}{1 + \hat{\eta}_{K+1}(\pmb{u})} \in \tilde{\Delta}^{K+1}$  is a valid probability estimator for  $K+1$ -class multiclass classification with  $\phi$  if the posterior probabilities  $[p(y | \pmb{x})]_{y=1}^{K+1}$  is in  $\tilde{\Delta}^{K+1}$ .
Then we construct a valid estimator for  $\Delta^{K + 1}$  based on  $\tilde{\eta}$ . Denote by  $P$  a permutation matrix that exchanges the value of a vector's first and last dimension, then we have the following estimator  $\tilde{\eta}^{\prime}$ :
$$
\tilde {\boldsymbol {\eta}} ^ {\prime} (\boldsymbol {u}) = \left\{ \begin{array}{l} \tilde {\boldsymbol {\eta}} (\boldsymbol {u}), u _ {K + 1} \neq \max  _ {y} u _ {y}, \\ P \tilde {\boldsymbol {\eta}} (P \boldsymbol {u}), \text {e l s e}. \end{array} \right.
$$
We then prove that it is a valid estimator. Denote by  $\Delta_{+}^{K + 1}$  the set of all the  $\beta \in \Delta^{K + 1}$  that  $\beta_{K + 1} \neq \max_y\beta_y$ , we can learn that  $\Delta_{+}^{K + 1} \in \tilde{\Delta}^{K + 1}$ . Then for any  $\beta \in \Delta_{+}^{K + 1}$ , denote by  $u^{*}$  the minimizer of conditional risk w.r.t.  $\beta$  and  $\phi$ , we can learn that  $u_{K + 1}^{*} \neq \max_y u_y^{*}$  due to the consistency of  $\phi$  and thus  $\tilde{\eta}'(u^*) = \tilde{\eta}(u^*) = \beta$ .
For any  $\beta \notin \Delta_{+}^{K + 1}$ , we can learn that  $P\beta \in \Delta_{+}^{K + 1}$ . Denote by  $\pmb{u}^*$  the minimizer of conditional risk w.r.t.  $\beta$  and  $\phi$ , then we can learn that  $P\pmb{u}^*$  must be the minimizer of  $P\beta$  according to the symmetry of  $\phi$ . Then we have that  $\tilde{\eta}(P\pmb{u}^*) = P\beta$ . Furthermore, notice that  $PP\beta = \beta$ , then we have  $\tilde{\eta}'(\pmb{u}^*) = P\tilde{\eta}(P\pmb{u}^*) = PP\beta = \beta$ .
Combining the two paragraphs above, we can learn that  $\tilde{\eta}'$  is a valid multiclass probability estimator for  $\phi$ . Then we can construct an unbounded L2D estimator as in (3), which indicates the existence of an unbounded probability estimator. Furthermore, since the loss function is unchanged, the collection of all the minimizers of  $R_{L_{\phi}}(\boldsymbol{u}, \boldsymbol{\eta}(\boldsymbol{x}), \operatorname*{Pr}(M = Y | X = \boldsymbol{x}))$  are also unchanged, and thus the all the values should have the same value on  $\mathcal{U}$ .
Proof. The boundedness of the first  $K$  dimensions is straightforward. The  $K + 1$ th dimension's boundedness can also be directly proved by reformulating it as
$$
\frac {\exp (u _ {K + 1})}{\exp (u _ {K + 1}) + \sum_ {y ^ {\prime} = 1} ^ {K} \exp (u _ {y ^ {\prime}}) - \max _ {y ^ {\prime} \in \{1 , \dots , K \}} \exp (u _ {y ^ {\prime}})}.
$$
Then we begin to prove its maxima-preserving. Denote by  $y_{1} = \operatorname{argmax}_{y \in \{1, \dots, K\}} u_{y}$ . It is easy to verify that  $\tilde{\psi}_{y_1}(\boldsymbol{u}) > \tilde{\psi}_y(\boldsymbol{u})$  for any  $y \in \{1, \dots, K\} / \{y_1\}$  due to the property of softmax function. Then we focus on the relation between  $\tilde{\psi}_{y_1}(\boldsymbol{u})$  and  $\tilde{\psi}_{K+1}(\boldsymbol{u})$ . We can learn that:
$$
\frac {\exp (u _ {K + 1})}{\exp (u _ {K + 1}) + \sum_ {y ^ {\prime} = 1} ^ {K} \exp (u _ {y ^ {\prime}}) - \exp (u _ {y _ {1}})} = \frac {1}{1 + \frac {\sum_ {y ^ {\prime} = 1} ^ {K} \exp (u _ {y ^ {\prime}}) - \exp (u _ {y _ {1}})}{\exp (u _ {K + 1})}},
$$
$$
\frac {\exp (u _ {y _ {1}})}{\sum_ {y ^ {\prime} = 1} ^ {K} \exp (u _ {y ^ {\prime}})} = \frac {1}{1 + \frac {\sum_ {y ^ {\prime} = 1} ^ {K} \exp (u _ {y ^ {\prime}}) - \exp (u _ {y _ {1}})}{\exp (u _ {y _ {1}})}}.
$$
When  $u_{K+1} > u_{y_1}$ , we can learn that  $\frac{\sum_{y' = 1}^{K}\exp(u_{y'}) - \exp(u_{y_1})}{\exp(u_{K + 1})} < \frac{\sum_{y' = 1}^{K}\exp(u_{y'}) - \exp(u_{y_1})}{\exp(u_{y_1})}$ , then  $\tilde{\psi}_{K + 1}(\pmb{u}) > \tilde{\psi}_{y_1}(\pmb{u})$ . Based on the formulations above, it is also easy to verify the cases when  $u_{y_1} \geq u_{K + 1}$ . Combining the discussions above and we can conclude the proof.
Proof. The consistency can be directly obtained by recovering the class probability and expert accuracy according to the maxima-preserving, then we first focus on how our proposed estimator recovers the posterior probabilities.
According to the property of log loss, it is easy to learn that  $\tilde{\psi}_y(\pmb{g}^*(\pmb{x})) = p(y|\pmb{x})$  for  $y \in \{1, \dots, K\}$ . When  $y = K + 1$ , we can learn from the range of our estimator and the property of binary log loss that  $\tilde{\psi}_y(\pmb{g}^*(\pmb{x})) = \operatorname*{Pr}(M = Y|X = \pmb{x})$ . Then we can conclude that  $\tilde{\psi}(\pmb{g}^*(\pmb{x})) = \pmb{\eta}(\pmb{x}) \times \operatorname*{Pr}(M = Y|X = \pmb{x})$ .
Then we begin to prove that there is no unbounded probability estimator by contradiction. Suppose there exists an unbounded estimator  $\psi$ . For any  $\pmb{g}$ , it must be the solution of a distribution and expert whose posterior probability is  $\tilde{\psi}(\pmb{g}(\pmb{x}))$  for each point  $\pmb{x}$ . However, for a  $\pmb{g}$  that there exists  $\pmb{x}$  that  $\psi(\pmb{g}(\pmb{x})) \notin \Delta^K \times [0,1]$ , we can learn that it cannot be the solution of any distribution and expert according to the definition of probability estimator. We can learn from this contradiction that  $\psi$  is not a probability estimator as long as its range is not  $\Delta^K \times [0,1]$ .
D Proof of Corollary 1
Proof. We can set the multi-class loss to  $\phi_{\tilde{\psi}}$  to get our proposed loss:
$$
\phi_ {\tilde {\psi}} (\pmb {u}, y) = \left\{ \begin{array}{l l} - \log \left(\frac {\exp (u _ {y})}{\sum_ {y ^ {\prime} = 1} ^ {K ^ {\prime} - 1} \exp (u _ {y ^ {\prime}})}\right) - \log \left(1 - \frac {\exp (u _ {K ^ {\prime}})}{\sum_ {y ^ {\prime} = 1} ^ {K ^ {\prime}} \exp (u _ {y ^ {\prime}}) - \max _ {y ^ {\prime} \in [ 1 , K ^ {\prime} - 1 ]} \exp_ {u _ {y ^ {\prime}}}}\right), y \neq K ^ {\prime}, \\ - \log \left(\frac {\exp (u _ {K ^ {\prime}})}{\sum_ {y ^ {\prime} = 1} ^ {K ^ {\prime}} \exp (u _ {y ^ {\prime}}) - \max _ {y ^ {\prime} \in [ 1 , K ^ {\prime} - 1 ]} \exp_ {u _ {y ^ {\prime}}}}\right) + \log \left(1 - \frac {\exp (u _ {K ^ {\prime}})}{\sum_ {y ^ {\prime} = 1} ^ {K ^ {\prime}} \exp (u _ {y ^ {\prime}}) - \max _ {y ^ {\prime} \in [ 1 , K ^ {\prime} - 1 ]} \exp_ {u _ {y ^ {\prime}}}}\right), \mathrm {e l s e}. \end{array} \right.
$$
A similar result can be deduced for the OvA-based surrogate by considering the following consistent multi-class loss with a strictly proper binary composite loss  $\xi$ :
$$
\phi_ {\mathrm {O v A}} \left(\boldsymbol {u}, y\right) = \left\{ \begin{array}{l} \xi (u _ {y}) + \sum_ {y ^ {\prime} \neq y} \xi (- u _ {y ^ {\prime}}),   y \neq K ^ {\prime}, \\ \xi (u _ {K ^ {\prime}}) - \xi (- u _ {K ^ {\prime}}),   \text {e l s e}. \end{array} \right.
$$
We then begin to prove their consistency. It is easy to verify that  $\phi_{\tilde{\psi}}$  is minimized when  $\tilde{\psi} (\pmb {g}(\pmb {x})) = \frac{p(y|\pmb{x})}{1 - p(K^{\prime}|\pmb{x})}$ , and we can conclude its consistency using the maxima-preserving property of  $\tilde{\psi}$ .
For  $\phi_{\mathrm{OvA}}$ , denote by  $\psi_{\mathrm{OvA}}$  the inverse link of  $\xi$ . We can learn that  $g^{*}(\pmb{x}) = \psi_{\mathrm{OvA}}\left(\frac{p(y|\pmb{x})}{1 - p(K^{\prime}|\pmb{x})}\right)$ , and we can learn the consistency since  $\xi$  is strictly proper and thus  $\psi_{\mathrm{OvA}}$  is increasing.
Proof. We first apply Pinsker's inequality. We can learn that:
$$
R _ {L _ {\tilde {\psi}}} (\boldsymbol {g}) - R _ {L _ {\tilde {\psi}}} ^ {*} \geq \mathbb {E} _ {p (\boldsymbol {x})} \left[ \frac {1}{2} \| \tilde {\psi} _ {1: K} (\boldsymbol {g} (\boldsymbol {x})) - \boldsymbol {\eta} (x) \| _ {1} ^ {2} + 2 (\tilde {\psi} _ {K + 1} (\boldsymbol {g} (\boldsymbol {x}) - \Pr (M = Y | X = \boldsymbol {x})) ^ {2} \right].
$$
We can learn that  $R_{L_{\tilde{\psi}}}(\pmb{g}) - R_{L_{\tilde{\psi}}}^{*} \geq \mathbb{E}_{p(\pmb{x})}\left[\frac{1}{2}\|\tilde{\psi}_{1:K}(\pmb{g}(\pmb{x})) - \pmb{\eta}(x)\|_1^2\right]$  immediately and learn that the bound for  $R_{01}$  following the analysis of cross-entropy loss in ordinary classification. Then we move to analyze the 0-1-deferral risk. We can further learn that:
$$
\begin{array}{l} R _ {L _ {\tilde {\psi}}} (\boldsymbol {g}) - R _ {L _ {\tilde {\psi}}} ^ {*} \geq \mathbb {E} _ {p (\boldsymbol {x})} \left[ \frac {1}{2} \| \tilde {\psi} _ {1: K} (\boldsymbol {g} (\boldsymbol {x})) - \boldsymbol {\eta} (x) \| _ {1} ^ {2} + 2 (\tilde {\psi} _ {K + 1} (\boldsymbol {g} (\boldsymbol {x})) - \Pr (M = Y | X = \boldsymbol {x})) ^ {2} \right] \\ \geq \frac {1}{2} \mathbb {E} _ {p (\boldsymbol {x})} \left[ \| \tilde {\psi} _ {1: K} (\boldsymbol {g} (\boldsymbol {x})) - \boldsymbol {\eta} (x) \| _ {1} ^ {2} + (\tilde {\psi} _ {K + 1} (\boldsymbol {g} (\boldsymbol {x})) - \Pr (M = Y | X = \boldsymbol {x})) ^ {2} \right] \\ \end{array}
$$
For any  $\pmb{x}$ , when  $g(\pmb{x})$  can induce the Bayes optimal solution for it, the excess risk is zero and the bound holds naturally. When it is not optimal, denote by  $\eta_{K + 1}(\pmb{x}) = \operatorname*{Pr}(M = Y | X = \pmb{x})$ .  $y'$  is the
Table 2: Experimental results on CIFAR10H with for 5 trials. We report the mean  $(\%)$  (standard error  $(\%))$  of related statistics. The best and comparable methods based on t-test at significance level  $5\%$  are highlighted in boldface.
Method	Error	Coverage	ECE	Budgeted Error
10%	20%	30%
S-SM	3.82(0.02)	41.28(0.14)	6.64(0.22)	50.25(0.57)	41.00(0.81)	31.47(0.37)
S-OvA	4.30(0.12)	37.08(1.36)	2.46(0.48)	52.75(0.40)	43.31(0.58)	33.23(0.72)
A-OvA	4.00(0.11)	92.75(0.24)	0.95(0.16)	4.00(0.11)	4.00(0.11)	4.00(0.11)
A-SM	3.65(0.09)	94.20(0.37)	0.94(0.17)	3.65(0.09)	3.65(0.09)	3.65(0.09)
Table 3: Experimental results on ImageNet-16H with for 5 trials. We report the mean  $(\%)$  (standard error  $(\%))$  of related statistics. The best and comparable methods based on t-test at significance level  $5\%$  are highlighted in boldface.
Method	Error	Coverage	ECE	Budgeted Error
10%	20%	30%
Image Noise Type = “080”
S-SM	10.63(1.34)	22.59(1.59)	18.31(3.22)	66.58(3.91)	60.41(2.98)	53.33(2.79)
S-OvA	10.48(0.64)	21.88(2.75)	12.79(0.33)	70.50(3.78)	61.58(4.64)	51.22(3.37)
A-OvA	10.36(1.09)	25.86(3.56)	8.69(0.63)	70.83(1.25)	61.32(1.70)	52.26(0.95)
A-SM	7.94(0.85)	37.32(2.72)	7.39(0.23)	54.24(2.89)	45.33(2.91)	34.11(1.65)
dimension with the largest value of  $g(x)$  and that of  $\pmb{\eta}(\pmb{x})$  is  $y''$ . Then we can learn that:
$$
\begin{array}{l} \frac {1}{2} \left(\| \tilde {\psi} _ {1: K} (\boldsymbol {g} (\boldsymbol {x})) - \boldsymbol {\eta} (x) \| _ {1} ^ {2} + \left(\tilde {\psi} _ {K + 1} (\boldsymbol {g} (\boldsymbol {x})) - \Pr (M = Y | X = \boldsymbol {x})\right) ^ {2}\right) \\ \geq \frac {1}{2} \left(\tilde {\psi} _ {y ^ {\prime}} (\boldsymbol {g} (\boldsymbol {x})) - \eta_ {y ^ {\prime}} (\boldsymbol {x}) - \tilde {\psi} _ {y ^ {\prime \prime}} (\boldsymbol {g} (\boldsymbol {x})) + \eta_ {y ^ {\prime \prime}} (\boldsymbol {x})\right) ^ {2} \\ \geq \frac {1}{2} \left(\eta_ {y ^ {\prime}} (\boldsymbol {x}) - \eta_ {y ^ {\prime \prime}} (\boldsymbol {x})\right) ^ {2} \\ \end{array}
$$
The last step is obtained according to the maxima-preserving property. Further generalizing  $y'$  and  $y''$  to be instance-dependent ( $y'(x)$  and  $y''(x)$ ), we can learn the following inequality using Jensen's inequality:
$$
R _ {L _ {\tilde {\psi}}} (\boldsymbol {g}) - R _ {L _ {\tilde {\psi}}} ^ {*} \geq \frac {1}{2} \left(\mathbb {E} _ {p (\boldsymbol {x})} \left[ \left| \eta_ {y ^ {\prime}} (\boldsymbol {x}) - \eta_ {y ^ {\prime \prime}} (\boldsymbol {x}) \right| \right]\right) ^ {2},
$$
which concludes the proof since the second expectation term is  $R_{01}^{\perp}(\pmb{g}) - R_{01}^{\perp *}$
F Details of Experiments
Details of Model and Optimizer: For methods on CIFAR-10, we use the 28-layer WideResNet that is the same as those used in Mozannar and Sontag [30], Charusaie et al. [9]. The optimizer is SGD with cosine annealing, where the learning rate is 1e-1 and weight decay is 5e-4. We conduct the experiments on 8 NVIDIA GeForce 3090 GPUs and the batch size is 1024 (128 on each GPU). The training epoch on CIFAR100 is set to 200.
For methods on CIFAR-10H, we train the same WideResNet on the fully-label CIFAR-10 dataset and then train the linear layer using the CIFAR-10H dataset. The optimizer is AdamW [] and the learning rate, batch size, epoch number are 1e-4, 128, 200. For methods on HateSpeech, we use a 384-dimension embedding and linear model in the experiment. The optimizer is SGD with cosine annealing and the learning rate, batch size, epoch number is 0.1, 128, 50. For methods on ImageNet-16H, we use the embedding generated by the pretrained DenseNet and linear model. The optimizer is Adam and the learning rate, batch size, epoch number is 1e-3, 1e-1, 100.
The extra result on ImageNet with noise type "080" and the exact statistics for CIFAR-10H are shown in the tables in the appendix.
Details of Evaluation Metrics: The reported Error is the sample mean of  $\ell_{01}^{\perp}$ , and Coverage is the ratio of undeferred samples. The ECE of expert accuracy is defined below:
$$
\mathrm {E C E} = \sum_ {i = 1} ^ {N} b _ {i} | p _ {i} - c _ {i} |,
$$
where  $b_{i}$  is the ratio of predictions whose confidence falls into the  $i$ th bin.  $p_{i}$  is the average confidence and  $c_{i}$  is the average accuracy in this bin. We set the bin number to 15. The budgeted error is calculated as below: if the coverage is lower than  $1 - x\%$ , we will use the classifier's prediction instead of the expert's for those samples whose estimated expert accuracy is lower to make the coverage equal to  $1 - x\%$ .
G Limitations and Broader Impact
Limitations: This work is designed for L2D without extra constraints on the number of expert queries. We believe that combining it with selective learning, i.e., adding explicit constraints on the ratio of deferred samples, can be a promising future direction.
Broader Impact: When applied in real-world applications, the frequency of expert queries may be imbalanced due to the performance differences of the expert among samples. This is a common impact shared by all the L2D methods. We believe that introducing fairness targets into L2D can be another promising direction.
H Consistent Multi-Expert Extension
We provide a direct extension of our loss for the multi-expert setting and provide consistency analysis: Denote by the expert advice  $\pmb{m} = [m_1, \dots, m_M] \in \mathcal{Y} \times \dots \times \mathcal{Y}$ . We first define an extended ASM:
Definition 4. (Multi-expert asymmetric softmax parameterization)  $\tilde{\psi}$  is called a multi-expert asymmetric softmax function that for any  $K > 1$  and  $\pmb{u} \in \mathbb{R}^{K + M}$ :
$$
\tilde {\psi} _ {y} ^ {M} (\boldsymbol {u}) = \left\{ \begin{array}{l l} \frac {\exp \left(u _ {y}\right)}{\sum_ {y ^ {\prime} = 1} ^ {K} \exp \left(u _ {y ^ {\prime}}\right)}, & y \notin [ K + 1, K + M ], \\ \frac {\exp \left(u _ {y}\right)}{\sum_ {y ^ {\prime} = 1} ^ {K + 1} \exp \left(u _ {y ^ {\prime}}\right) + \exp (y) - \max  _ {y ^ {\prime} \in \{1, \dots , K \}} \exp \left(u _ {y ^ {\prime}}\right)}, & \text {e l s e .} \end{array} \right. \tag {10}
$$
We can directly extend the property of single-expert ASM to this case:
Proposition 2. (Properties of  $\tilde{\psi}^M$ ) For any  $\pmb{u} \in \mathbb{R}^{K + M}$ :
(i). (Boundedness)  $\tilde{\psi}^M (\pmb {u})\in \Delta^K\times [0,1]^M$
(ii). (Maxima-preserving)  $\operatorname{argmax}_{y\in \{1,\dots ,K + M\}}\tilde{\psi}_y^M (\pmb {u}) = \operatorname{argmax}_{y\in \{1,\dots ,K + M\}}u_y.$
Then we give the following extension:
Definition 5. (Multi-expert extension)
$$
L _ {\tilde {\psi} ^ {M}} (\boldsymbol {u}, y, m) = - \log (\tilde {\psi} _ {y} ^ {M} (\boldsymbol {u})) - \sum_ {i = 1} ^ {M} \left(\llbracket m _ {i} \neq y \rrbracket \log (1 - \tilde {\psi^ {M}} _ {K + i} (\boldsymbol {u})) - \llbracket m _ {i} = y \rrbracket \log (\tilde {\psi_ {K + i} ^ {M}} (\boldsymbol {u}))\right).
$$
According to the position, we can simply give the characterization of  $L_{\tilde{\psi}^M}$ 's optimal solution:
$$
\tilde {\psi} _ {y} ^ {M *} (\boldsymbol {u}) = \left\{ \begin{array}{l l} p (y | \boldsymbol {x}), & y \notin [ K + 1, K + M ], \\ \Pr (M _ {y - K} | \boldsymbol {x}), & \text {e l s e .} \end{array} \right. \tag {11}
$$
According to the maxima-preserving, we can directly learn the consistency of our multiclass extension.
I Estimation Error Bound
In this section, we give a routine analysis of our method's estimation error bound. Our proof of the estimation error bound is based on Rademacher complexity:
Definition 6. (Rademacher complexity) Let  $Z_{1},\ldots ,Z_{n}$  be  $n$  i.i.d. random variables drawn from a probability distribution  $\mu$ ,  $\mathcal{H} = \{h:\mathcal{Z}\to \mathbb{R}\}$  be a class of measurable functions. Then the expected Rademacher complexity of  $\mathcal{H}$  is defined as
$$
\mathfrak {R} _ {n} (\mathcal {H}) = \mathbb {E} _ {Z _ {1}, \dots , Z _ {n} \sim \mu} \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {h \in \mathcal {H}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} h (Z _ {i}) \right]
$$
where  $\sigma = \{\sigma_1, \dots, \sigma_n\}$  are Rademacher variables taking the value from  $\{-1, +1\}$  with even probabilities.
First, let  $\mathcal{G} \subset \mathcal{X} \to \mathbb{R}^{K+1}$  be the model class and each of its dimension is constructed by  $\mathcal{G}_y \subset \mathcal{X} \to \mathbb{R}$ . Then, we define the following scoring function space for L2D task:
$$
\mathcal {L} \circ \mathcal {G} = \{h: (\boldsymbol {x}, y, m) \mapsto L _ {\tilde {\psi}} (\boldsymbol {g} (\boldsymbol {x}), y, m) | \boldsymbol {g} \in \mathcal {G} \}
$$