SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This result shows that the consistent surrogate framework [9] does not directly provide us a solution for solving the unboundedness problem since most multi-class losses we use, e.g., CE loss, OvA loss, Focal loss [26], are symmetric. This motivates the design of surrogates with bounded probability estimators for L2D in our work. Based on Theorem 1, we have the following findings: firstly unboundedness of probability estimators is not necessarily caused by the softmax parameterization, as the induced estimator of any symmetric loss, e.g., standard softmax-free symmetric OvA losses [52], exhibits unboundedness too. Secondly, due to the fact that the OvA strategy for L2D (6) successfully gets rid of the unboundedness issue, we can reasonably deduce that a modified softmax function can also present similar outcomes. In the following section, we propose a novel modification of the softmax parameterization that overcomes these issues and reveals that both the previous OvA-based work [44] and our current work benefit from the use of asymmetric losses.
4 Consistent Softmax-Based Surrogate with Bounded Probability Estimator
In this section, we find a softmax parameterization that is a feasible solution in L2D for the construction of surrogate loss with a bounded probability estimator. We first propose a cross-entropy-like surrogate loss but with an asymmetric transformation that resembles the ordinary softmax function for its first  $K$  elements but processes the  $K + 1$ -th coordinate in a special way. Then we show that such a transformation has useful properties for both classification and probability estimation, and then provide consistency analysis for both the loss function and its induced bounded probability estimator. Finally, we show that our proposed loss and the OvA loss [44] are connected to the consistent surrogate formulation (4) with the use of asymmetric multi-class losses, which indicates the helpfulness of asymmetry and shed light on the future study of consistent surrogates for L2D with bounded estimators.
4.1 Consistent Surrogate Formulation with Asymmetric Bounded Softmax Parameterization
Before searching for a softmax-parameterization that is capable of L2D with probability estimation, we need to better understand the failure of (3) for inducing a bounded probability estimator. By inspecting the proof of consistency in Theorem 1 of Mozannar and Sontag [30], we can learn that the cross-entropy-like surrogate (3) works by fitting a  $K + 1$  class posterior distribution:  $\psi^{\mathrm{sm}}(g^{*}) = \tilde{\eta} (\pmb {x}) = [\frac{\eta_{1}(\pmb{x})}{1 + \operatorname*{Pr}(M = Y|\pmb{x})},\dots ,\frac{\eta_{K}(\pmb{x})}{1 + \operatorname*{Pr}(M = Y|\pmb{x})},\frac{\operatorname*{Pr}(M = Y|\pmb{x})}{1 + \operatorname*{Pr}(M = Y|\pmb{x})} ]$ . Though we can check that the  $\varphi \circ g^{*}$  is exactly a Bayes optimal solution of  $R_{01}^{\perp}(f)$  due to the monotonicity of the softmax function and the form of  $\tilde{\eta}$ , to get  $[\pmb {\eta}(\pmb {x});\operatorname *{Pr}(M = Y|X = \pmb {x})]$  we need to perform an extra inverse transformation should be exerted on  $\psi^{\mathrm{sm}}$ . This is caused by the following dichotomy: the class probabilities and expert accuracy we aim to estimate are in the range of  $\Delta^K\times [0,1]$ , while the standard softmax-parameterization of (3) maps the  $K + 1$  dimensional scoring function  $\pmb{g}$  into  $\Delta^{K + 1}$ .
To solve this issue, a promising approach is to modify the softmax function to make it a transformation that can directly cast the scoring function  $g$  into the target range  $\Delta^K \times [0,1]$ . Since the target range is not symmetric, the modified softmax should also be asymmetric. This idea is given a concrete form in the following asymmetric softmax parameterization:
Definition 2. (Asymmetric softmax parameterization)  $\bar{\psi}$  is called a asymmetric softmax function that for any  $K > 1$  and  $\pmb{u} \in \mathbb{R}^{K + 1}$ :
$$
\tilde {\psi} _ {y} (\boldsymbol {u}) = \left\{ \begin{array}{l l} \frac {\exp \left(u _ {y}\right)}{\sum_ {y ^ {\prime} = 1} ^ {K} \exp \left(u _ {y ^ {\prime}}\right)}, & y \neq K + 1, \\ \frac {\exp \left(u _ {K + 1}\right)}{\sum_ {y ^ {\prime} = 1} ^ {K + 1} \exp \left(u _ {y ^ {\prime}}\right) - \max  _ {y ^ {\prime} \in \{1 , \dots , K \}} \exp \left(u _ {y ^ {\prime}}\right)}, & \text {e l s e .} \end{array} \right. \tag {8}
$$
At first glance, the proposed asymmetric function appears to be the standard softmax function, which excludes the  $K + 1$ -th input of  $\mathbf{u}$  for  $y \neq K + 1$ . However, the last coordinate of the function takes on a quite different form. This special asymmetric structure is designed to satisfy the following properties:
Proposition 1. (Properties of  $\tilde{\psi}$ ) For any  $\boldsymbol{u} \in \mathbb{R}^{K+1}$ :
(i). (Boundedness)  $\tilde{\psi} (\pmb {u})\in \Delta^K\times [0,1],$
(ii). (Maxima-preserving)  $\operatorname{argmax}_{y\in \{1,\dots ,K + 1\}}\tilde{\psi}_y(\boldsymbol {u}) = \operatorname{argmax}_{y\in \{1,\dots ,K + 1\}}u_y.$
It can be seen that the proposed asymmetric softmax  $\tilde{\psi}$  is not only bounded but also successfully maps  $\pmb{g}$  into the desired target range, which indicates that  $\tilde{\psi}$  may directly serve as the probability estimator  $\hat{\eta}^{\tilde{\psi}}(\pmb{g}(\pmb{x})) = \tilde{\psi}(\pmb{g}(\pmb{x}))$ . Furthermore, the maxima-preserving property guarantees that the estimator is also capable of discriminative prediction: if a scoring function  $\pmb{g}'$  can recover the true probability, i.e.,  $\hat{\eta}^{\tilde{\psi}}(\pmb{g}'(\pmb{x})) = [\pmb{\eta}(\pmb{x}); \operatorname*{Pr}(M = Y | X = \pmb{x})]$ , then  $\varphi \circ \pmb{g}'$  must also be the Bayes optimal solution of  $R_{01}^{\perp}(\pmb{g})$ . Based on the asymmetric softmax function, we propose the following surrogate loss for L2D:
Definition 3. (Asymmetric Softmax-Parameterized Loss) The proposed surrogate for L2D with asymmetric softmax parameterization  $L_{\tilde{\psi}}(\boldsymbol{u},y,m):\mathbb{R}^{K + 1}\times \mathcal{V}\times \mathcal{V}$  is formulated as:
$$
L _ {\tilde {\psi}} (\boldsymbol {u}, y, m) = - \log \left(\tilde {\psi} _ {y} (\boldsymbol {u})\right) - \llbracket m \neq y \rrbracket \log \left(1 - \tilde {\psi} _ {K + 1} (\boldsymbol {u})\right) - \llbracket m = y \rrbracket \log \left(\tilde {\psi} _ {K + 1} (\boldsymbol {u})\right). \tag {9}
$$
The proposed loss takes an intuitive form, which can be seen as the combination of the cross-entropy loss (the first term) and a modified version of binary logistic loss (the last two terms). This formulation is inspired by the structure of our problem, where the expert accuracy is not directly related to class probabilities while the class probabilities should fulfill that  $\pmb{\eta}(\pmb{x})\in \Delta^K$ . In fact, the proposed surrogate is not a simple summation of two independent losses and they are indirectly related by the asymmetric softmax function: according to Definition 2, the two counterparts share the same elements  $[g_1,\dots ,g_K]$ . This correlation serves as a normalization for  $\tilde{\psi}_{K + 1}$  that brings the property of maxima-preserving, which is crucial for the following consistency analysis:
Theorem 2. (Consistency of  $L_{\tilde{\psi}}$  and bounded probability estimator  $\hat{\eta}^{\tilde{\psi}}$
The proposed surrogate  $L_{\tilde{\psi}}$  is a consistent surrogate for L2D, i.e.,  $\varphi \circ \pmb{g}^{*}\in \mathrm{argmin}_{f}R_{01}^{\perp}(f)$ ,  $\forall \pmb{g}^{*}\in \mathrm{argmin}_{\pmb{g}}R_{\Phi}^{\perp}(\pmb{g})$ . The bounded probability estimator can also recover the desired class probabilities and expert accuracy:  $\hat{\eta}^{\tilde{\psi}}(\pmb{g}^{*}(\pmb{x})) = [\pmb {\eta}(\pmb {x});\operatorname *{Pr}(M = Y|X = \pmb {x})]$ ,  $\forall \pmb {x}\in \mathcal{X}$ . Furthermore, if there exists other probability estimators for  $L_{\tilde{\psi}}$ , they must also be bounded.
The proof can be found in Appendix C. According to the theorem above, we showed that our proposed asymmetric softmax function induces a consistent surrogate and a bounded probability estimator. We also showed that there does not exist an unbounded probability estimator for  $L_{\tilde{\psi}}$ , which guarantees that our proposed bounded estimator is never the modification of an unbounded one. This result enriches the toolbox for L2D with probability estimation and theoretically justifies the use of softmax parameterization in L2D. In fact, we can further substitute the cross-entropy-like counterpart with any strictly proper loss [46] to get the same consistency result. In the following subsection, we will discuss the relationship between our method and the general consistent surrogate framework [9].
4.2 Connection with Consistent Surrogate Framework [9]: Asymmetry Can Help
In the previous section, we showed that there exists an implementation of softmax  $\tilde{\psi}$  that can induce both a consistent loss and a valid probability estimator for L2D. Given the theoretical successes of our proposed softmax-based method and the previous OvA strategy, we may further expect them to provide more insights into the design of surrogates and probability estimators for L2D. Recalling the consistent surrogate formulation (4) proposed in Charusaie et al. [9] that allows the use of all consistent multi-class losses for constructing L2D surrogates, it is an instinctive idea that our proposed consistent surrogates (9) and the previous work [44] can be included in this framework. However, this idea may not be easily confirmed: Theorem 1 implies that the two surrogates are not the trivial combinations of commonly used symmetric losses and formulation (4) since they can both induce bounded probability estimators. The following corollary shows that the two surrogates are included in
(a) S-SM
(b) S-OvA
(c) A-OvA
Figure 3: Distributions of the true and estimated accuracy of baselines and proposed method on CIFAR10H.
(d) A-SM (Proposed)
the family of consistent surrogate formulation (4), but induced from two novel asymmetric multi-class surrogates.
Corollary 1. We can get the consistent surrogates  $L_{\tilde{\psi}}$  (9) and  $L_{\mathrm{OvA}}$  (6) by setting  $\phi$  in the consistent loss formulation (4) to specific consistent and asymmetric multi-class losses.
Due to page limitations, we present the formulation of the asymmetric multi-class losses and the proof in the Appendix D. Although this conclusion reveals the connection between the two consistent surrogates and the general framework (4), it is important to note that it does not necessarily imply that the results in Verma and Nalisnick [44] and this paper are trivial. Since asymmetric losses are seldom used in ordinary multi-class classification, it is hard to obtain  $L_{\tilde{\phi}}$  and  $L_{\mathrm{OvA}}$  by simply selecting  $\phi$  from known consistent multi-class losses. According to this corollary and theorem 1, we can determine that a multi-class loss  $\phi$  with asymmetric structure is a necessity if we aim to design consistent L2D surrogates with bounded probability estimators based on the general formulation (4). Based on this insight, it is promising to discover more surrogates with bounded probability estimators for L2D by focusing on the design of asymmetric multi-class surrogates.
4.3 Regret Transfer Bounds
In this section, we further study the regret transfer bounds of our proposed surrogate (9) to characterize the effect of minimizing surrogate risk  $R_{L_{\vec{\psi}}}(\pmb{g})$  on both the target risk  $R_{01}^{\perp}(\pmb{g})$  and the misclassification error of the classifier. Denoted by  $\pmb{g}_{\mathrm{class}} = \pmb{g}_{1:K}$  the classifier counterpart of our model and  $R_{01}(\pmb{g})$  is its misclassification error, we can show that:
$$
\text {T h e o r e m 3 .} \max  \left(R _ {0 1} (\boldsymbol {g} _ {\text {c l a s s}}) - R _ {0 1} ^ {*}, R _ {0 1} ^ {\perp} (\boldsymbol {g}) - R _ {0 1} ^ {\perp *}\right) \leq \sqrt {2 \left(R _ {L _ {\tilde {\psi}}} (\boldsymbol {g}) - R _ {L _ {\tilde {\psi}}} ^ {*}\right)}.
$$
The proof can be found in Appendix E. This theorem shows that we can use the excess risk of our proposed surrogate to upper-bound those of our concerned targets, and its proof is conducted by applying Pinsker's inequality and zooming.
This theorem shows that as long as we reach a low risk w.r.t. the proposed surrogate (approximately optimal), we can obtain a model with satisfying performances for both L2D and classification. This conclusion is not trivial: according to the characterization of  $R_{01}^{\perp}$ 's Bayes optimality in Definition 1, the classifier counterpart is not guaranteed to classify accurately on the deferred samples. We will further experimentally demonstrate the efficacy of our surrogate for L2D and classification in the next section.
In this section, we compare our proposed asymmetric softmax estimator and its induced surrogate loss with the estimators and losses in previous works. Detailed setup can be found in Appendix F.
Datasets and Models. We evaluate the proposed method and baselines on widely used benchmarks with both synthetic and real-world experts. For synthetic experts, we conduct experiments on the CIFAR100 [24]. Following the previous works, the expert has a chance of  $p$  to generate correct labels on the first  $k \in \{20, 40, 60\}$  classes and at random otherwise. We set  $p = 94\%$  as in Mozannar and Sontag [30] and  $75\%$  as in Verma and Nalisnick [44] to simulate experts of high and medium accuracy. For real-world experts, we use CIFAR10H, HateSpeech, and ImageNet-16H [36, 14, 21]
Table 1: Experimental results on CIFAR100 with for 5 trials. We report the mean(%) (standard error  $(\%)$ ) of related statistics. The best and comparable methods based on t-test at significance level  $5\%$  are highlighted in boldface.
Method	Expert	Error	Coverage	ECE	Budgeted Error
10%	20%	30%
Expert Accuracy = 94%
S-SM	20	24.58(0.13)	77.72(0.31	4.86(0.11)	31.68(0.39)	25.00(0.24)	24.59(0.13)
40	21.92(0.32)	57.89(0.54)	9.22(0.38)	46.69(0.63)	38.50(0.60)	29.94(0.62)
60	18.69(0.88)	40.34(6.28)	11.10(0.72)	59.44(4.95)	51.36(5.08)	42.77(5.09)
S-OvA	20	27.00(1.91)	85.40(0.91)	5.15(0.31)	28.42(2.26)	27.00(1.91)	27.00(1.91)
40	27.33(1.50)	68.47(1.59)	9.38(0.46)	39.95(2.35)	32.93(2.30)	27.92(2.06)
60	18.44(0.64)	58.18(1.55)	7.70(0.24)	42.52(1.74)	33.67(1.72)	25.74(1.75)
A-OvA	20	25.63(0.97)	90.40(0.89)	4.35(0.29)	25.65(0.99)	25.63(0.97)	25.63(0.97)
40	23.23(0.42)	80.46(0.51)	6.81(0.33)	27.44(0.62)	23.23(0.42)	23.23(0.42)
60	19.64(1.10)	70.44(2.73)	7.34(0.38)	31.90(2.73)	24.75(2.73)	20.05(1.62)
A-SM (Proposed)	20	24.54(0.06)	98.16(0.03)	4.63(0.11)	24.54(0.06)	24.54(0.06)	24.54(0.06)
40	22.17(0.36)	92.20(0.32)	6.58(0.22)	22.17(0.36)	22.17(0.36)	22.17(0.36)
60	19.30(0.58)	84.72(0.30)	5.96(0.44)	22.91(0.42)	19.30(0.58)	19.30(0.58)
Expert Accuracy = 75%
S-SM	20	25.49(0.20)	86.46(0.52)	5.07(0.21)	26.09(0.23)	25.48(0.20)	25.49(0.20)
40	25.13(0.44)	74.65(0.48)	12.82(0.29)	32.82(0.66)	26.99(0.59)	25.13(0.44)
60	24.05(0.43)	60.25(1.21)	18.70(0.55)	42.30(0.98)	36.18(0.99)	29.47(1.10)
S-OvA	20	26.53(0.86)	90.06(2.92)	5.57(0.54)	26.83(1.20)	26.53(0.86)	26.53(0.86)
40	26.89(1.78)	77.69(5.82)	7.14(1.41)	32.30(4.92)	28.33(3.11)	26.90(1.78)
60	25.29(0.69)	69.08(1.35)	7.47(0.12)	36.67(1.45)	30.44(1.33)	25.70(1.15)
A-OvA	20	25.94(0.52)	93.05(0.43)	4.78(0.29)	25.94(0.52)	25.94(0.52)	25.94(0.52)
40	25.73(1.07)	83.29(2.19)	5.75(0.80)	27.75(1.39)	25.73(1.07)	25.73(1.07)
60	22.81(0.52)	79.76(1.96)	6.10(0.32)	27.19(1.64)	23.08(0.87)	22.81(0.52)
A-SM (Proposed)	20	24.58(0.12)	98.90(0.75)	4.34(0.31)	24.58(0.12)	24.58(0.12)	24.58(0.12)
40	24.29(0.35)	96.53(0.21)	5.58(0.13)	24.29(0.35)	24.29(0.35)	24.29(0.35)
60	22.48(0.32)	92.63(0.33)	5.58(0.17)	22.48(0.32)	22.48(0.32)	22.48(0.32)
datasets, where the expert's prediction is generated using the provided auxiliary expert information. More experimental results of real-world experts can be found in Appendix F.
For CIFAR-100, HateSpeech, and ImageNet-16H, we report the misclassification error, coverage, and the ECE of the expert accuracy estimates. We also report the error of our model with the deferral budget as in Figure 4 of Verma and Nalisnick [44] to further evaluate the performance of the obtained classifier. For CIFAR10H, we plot the distribution of the true and estimated expert accuracy in Figure 3 to illustrate the performance of baselines and our method on fitting  $\operatorname{Pr}(M = Y|X = x)$ . The experiments on CIFAR-100 is conducted with 28-layer WideResNet [50] and SGD as in previous works [30, 44], and those on datasets with real-world experts are conducted using pre-trained models/embedding.
Baselines. We compare our Asymmetric SoftMax based method (A-SM) with the previously proposed Symmetric SoftMax-based (S-SM) method [30] and Asymmetric OvA (A-OvA) based method [44]. We also combined the Symmetric OvA logistic loss (S-OvA) with (4) to further evaluate the effect of symmetric loss and its induced unbounded estimator. For unbounded probability estimators in S-SM and S-OvA, we clip their estimates into  $[0,1]$  to make them valid. We directly use the output of baselines and our method without post-hoc techniques [19, 33] to better reflect their own performance.
Experimental Results. By observing Figure 3, we can find that the distributions of bounded method A-OvA and our proposed A-SM have markedly more overlap with the true distribution compared with the symmetric and unbounded ones, which directly shows that the bounded estimators can better estimate of expert accuracy. As can be seen from the experimental results reported in Table 1 and 2, our proposed A-SM is always better than or comparable to all the baselines w.r.t. classification accuracy. We can also see that the coverage of our method is always significantly higher than the baselines, which shows that our surrogate can induce ideal models for L2D. Though S-SM is comparable to A-SM with a high-accuracy expert, it has a significantly lower coverage and is
Table 2: Experimental results on HateSpeech and ImageNet-16H with noise type "095" for 5 trials. We report the mean  $(\%)$  (standard error  $(\%))$  of related statistics. The best and comparable methods based on t-test at significance level  $5\%$  are highlighted in boldface.
Method	Error	Coverage	ECE	Budgeted Error
10%	20%	30%
HateSpeech
S-SM	8.65(0.14)	70.19(1.50)	3.95(0.48)	25.48(0.49)	16.80(0.44)	8.97(0.33)
S-OvA	8.65(0.14)	69.90(0.53)	1.77(0.07)	24.19(0.25)	15.89(0.34)	8.78(0.31)
A-OvA	9.64(0.32)	77.2(0.42)	1.73(0.26)	16.525(0.28)	8.80(0.25)	8.56(0.29)
A-SM	8.06(0.40)	81.98(0.58)	1.53(0.26)	15.07(0.35)	8.06(0.40)	8.06(0.40)
ImageNet-16H
S-SM	15.08(1.19)	26.41(1.78)	22.92(2.59)	68.58(1.30)	60.33(1.19)	51.27(1.94)
S-OvA	15.15(1.50)	23.08(2.39)	11.68(0.53)	68.91(2.51)	60.08(1.82)	51.50(2.05)
A-OvA	14.17(0.58)	21.56(2.13)	11.57(1.50)	71.14(2.10)	62.39(2.10)	53.95(2.27)
A-SM	12.59(0.79)	39.48(2.50)	8.13(1.33)	57.75(1.37)	49.66(1.18)	40.85(1.51)
outperformed by A-SM with an expert of lower accuracy, which indicates that it is suffering from the problem of deferring more samples than necessary. This problem is also observed in learning with rejection [8]. Though S-OvA can mitigate this problem, its coverage is still consistently lower than A-SM. Notice that when there exist deferral budget requirements, all the unbounded baselines are outperformed by A-OvA, and A-OvA is further outperformed by A-SM, which indicates that our method can also efficiently generate a classifier. Meanwhile, the ECE of our estimated expert accuracy is also comparable to or better than A-OvA, while those of the unbounded ones all have significantly higher ECE, which further shows the efficacy of our method in estimating expert accuracy.
In this paper, we provide a novel consistent surrogate loss based on an asymmetric softmax function for learning to defer that can also provide calibrated probability estimates for the classifier and for expert correctness. We reveal that the root cause of the previous unbounded and miscalibrated probability estimators for L2D is not softmax but the intrinsic symmetry of the used loss function. We solve this problem by designing an asymmetric softmax function and using it to induce a consistent surrogate loss and a bounded probability estimator. We further give the regret transfer bounds of our method for both L2D and classification tasks. Finally, we evaluate our method and the baseline surrogate losses and probability estimators on benchmark datasets with both synthetic and real-world experts and show that we outperform prior methods. While we provide a consistent multi-expert extension of our proposed surrogate in Appendix H, it is still a promising future direction to generalize the multi-expert surrogates [45] to all the consistent multiclass losses as in Charusaie et al. [9].
This research/project is supported by the National Research Foundation, Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2-GC-2023-009), and Ministry of Education, Singapore, under its Academic Research Fund Tier 1 (RG13/22). Lei Feng is supported by Chongqing Overseas Chinese Entrepreneurship and Innovation Support Program, CAAI-Huawei MindSpore Open Fund, and Openl Community (https://openi.pcl.ac.cn). Hussein Mozannar is thankful for the support of the MIT-IBM Watson AI Lab.