SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Note that while Sections 3 and 4 both propose new two-stage algorithms based on  $\mathcal{H}$ -consistent surrogate losses, they differ in an important way. Section 3 learns with deferral in a score-based framework, where deferral is associated with extra scores. In contrast, Section 4 learns with deferral in a predictor-rejector setting, where deferral corresponds to a separate function. These represent two distinct learning frameworks that have been studied in the literature. Deriving consistent surrogate losses in the predictor-rejector setting has historically been challenging for traditional single-stage scenarios, leading many to opt for the score-based approach.
We should also highlight that our  $\mathcal{H}$ -consistency bounds in Theorems 1 and 6 can be used to derive finite sample estimation bounds for the minimizer of the surrogate loss over a hypothesis set  $\mathcal{H}$ . This is achieved by upper bounding the estimation error of the minimizer of the surrogate loss using standard Rademacher complexity bounds (see [Mao et al., 2023h]).
In this section, we report the results of our experiments on CIFAR-10 [Krizhevsky, 2009] and SVHN [Netzer et al., 2011] datasets to test the effectiveness of our proposed algorithms for two-stage
Table 4: Accuracy of deferral with multiple experts: mean  $\pm$  standard deviation over three runs.
Dataset	Base cost	Base model	Single expert	Two experts	Three experts
SVHN	✘	91.12	91.85 ± 0.01%	92.77 ± 0.02%	93.30 ± 0.02%
CIFAR-10	✘	70.56	72.63 ± 0.20%	75.84 ± 0.35%	77.68 ± 0.07%
SVHN	✓	91.12	91.66 ± 0.01%	92.05 ± 0.10%	92.19 ± 0.03%
CIFAR-10	✓	70.56	71.73 ± 0.06%	72.31 ± 0.31%	72.42 ± 0.12%
learning to defer with multiple experts. We evaluated the overall accuracy of the learned pairs of predictor and deferral model across different scenarios involving varying the number of experts, where the predictor is pre-learned in the first stage and the deferral is subsequently learned using our proposed surrogate loss. We find that as the number of experts increases, the overall accuracy of the learned pairs also increases, in both scenarios with zero and non-zero base costs. This observation highlights the significance of using a multiple expert framework in our approach and the effectiveness of our surrogate loss within the framework.
We used ResNet architectures [He et al., 2016] for the prediction model, the deferral model and expert models. More precisely, we used ResNet-4 for both the predictor and the deferral. We adopted three expert models: ResNet-10, ResNet-16, ResNet-28 with increasing capacity. For training, we used the Adam optimizer [Kingma and Ba, 2014] with a batch size of 128 and weight decay  $1 \times 10^{-4}$ . Training was run for 15 epochs for SVHN and 50 epochs for CIFAR-10 with the default learning rate. No data augmentation was used in our experiments. We used our two-stage surrogate loss (3) with the logistic loss  $\ell = \ell_{\log}$  to train the deferral model ResNet-4, with a pre-learned predictor ResNet-4 trained using logistic loss. A check mark indicates the presence of a base cost in the cost function, whereas a cross mark signifies its absence. We first set the cost function to be  $\mathbb{1}_{\mathrm{h}_j(x) \neq y}$  without a base cost. Next, for the experimental results shown in the last two rows of Table 4, we chose base costs  $\beta_j$  associated with each expert model as: 0.1, 0.12, 0.14 increasing with model capacity for SVHN and 0.3, 0.32, 0.34 increasing with model capacity for CIFAR-10. A base cost value that is close to the misclassification loss can strike a balance between improving accuracy and maintaining the ratio of deferral. We observed that other neighboring values lead to similar results. Note that the accuracy here refers to the overall accuracy of the learned pairs of predictor and deferral model. It is related to the deferral loss. Specifically, in the absence of the base cost, the accuracy aligns precisely with one minus the expected deferral loss. The results of Table 4 demonstrate the effectiveness of our proposed algorithms for two-stage learning to defer with multiple experts.
To the best of our knowledge, our study pioneers the exploration of a two-stage learning approach for deferral, a framework that is essential in numerous practical applications. Thus, we are unaware of any established baselines within this context.
It is important to underscore the differences between our learning scenario and those presented in [Okati et al., 2021, Narasimhan et al., 2022]. While both of them involve two phases, their methodologies are considerably different from ours. Okati et al. [2021] required conditional probabilities paired with loss estimates from the expert—a component not available in our framework, as emphasized by Mozannar et al. [2023]. On the other hand, Narasimhan et al. [2022] proposed a post-hoc correction for single-stage learning to defer surrogate losses. This approach, however, is not applicable to a pre-trained predictor from the standard multi-class classification. In contrast, our work focuses on enhancing the pre-trained predictor within the standard framework.
A limitation of our study is that the cost function used within the deferral loss is not fixed, and is typically determined through cross-validation in practice. There exists potential to introduce a principled method for selecting the cost function, which we have reserved for future research.
We introduced a novel family of surrogate loss functions and algorithms for a crucial two-stage learning to defer approach with multiple experts. We proved that these surrogate losses are supported by  $\mathcal{H}$ -consistency bounds and established their realizable  $\mathcal{H}$ -consistency properties for a constant cost function. This work paves the way for comparing different surrogate losses and cost functions within our framework. Further exploration, both theoretically and empirically, holds the potential to identify optimal choices for these quantities across diverse tasks.
M. Raghu, K. Blumer, G. Corrado, J. Kleinberg, Z. Obermeyer, and S. Mullainathan. The algorithmic automation problem: Prediction, triage, and human effort. arXiv preprint arXiv:1903.12220, 2019.
N. Raman and M. Yee. Improving learning-to-defer algorithms through fine-tuning. arXiv preprint arXiv:2112.10768, 2021.
H. G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics, 12(1):530-554, 2018.
I. Steinwart. How to compare different loss functions and their risks. Constructive Approximation, 26(2):225-287, 2007.
E. Straitouri, A. Singla, V. B. Meresht, and M. Gomez-Rodriguez. Reinforcement learning under algorithmic triage. arXiv preprint arXiv:2109.11328, 2021.
E. Straitouri, L. Wang, N. Okati, and M. G. Rodriguez. Provably improving expert predictions with conformal prediction. arXiv preprint arXiv:2201.12006, 2022.
S. Tan, J. Adebayo, K. Inkpen, and E. Kamar. Investigating human+ machine complementarity for recidivism predictions. arXiv preprint arXiv:1808.09123, 2018.
R. Verma and E. Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning, pages 22184-22202, 2022.
R. Verma, D. Barrejón, and E. Nalisnick. Learning to defer to multiple experts: Consistent surrogate losses, confidence calibration, and conformal ensembles. In International Conference on Artificial Intelligence and Statistics, pages 11415-11434, 2023.
J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus. Emergent abilities of large language models. CoRR, abs/2206.07682, 2022.
Y. Wiener and R. El-Yaniv. Agnostic selective classification. In Advances in neural information processing systems, 2011.
B. Wilder, E. Horvitz, and E. Kamar. Learning to complement humans. In International Joint Conferences on Artificial Intelligence, pages 1526-1533, 2021.
M. Yuan and M. Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11(1), 2010.
M. Yuan and M. Wegkamp. SVMs with a reject option. In Bernoulli, 2011.
M. Zhang and S. Agarwal. Bayes consistency vs. H-consistency: The interplay between surrogate loss functions and the scoring function class. In Advances in Neural Information Processing Systems, 2020.
T. Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics, 32(1):56-85, 2004.
J. Zhao, M. Agrawal, P. Razavi, and D. Sontag. Directing human attention in event localization for clinical timeline creation. In Machine Learning for Healthcare Conference, pages 80-102, 2021.
C. Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. arXiv preprint arXiv:2302.02334, 2023.
L. Ziyin, Z. Wang, P. P. Liang, R. Salakhutdinov, L.-P. Morency, and M. Ueda. Deep gamblers: Learning to abstain with portfolio theory. arXiv preprint arXiv:1907.00208, 2019.
A Related work 16
B Examples of two-stage score-based surrogate losses 17
C Examples of two-stage predictor-rejector surrogate losses 18
D Proof of  $\mathcal{H}$ -consistency bounds for score-based two-stage surrogate losses (Theorem 1) 19
E Proof of  $\overline{\mathcal{H}}$ -consistency bounds for standard surrogate loss functions (Theorem 3) 21
E.1 Multinomial logistic loss 21
E.2 Sum exponential loss 23
E.3 Generalized cross-entropy loss 24
E.4 Mean absolute error loss 26
Proof of realizable consistency for score-based two-stage surrogate losses (Theorem 5) 27
G Proof of  $(\mathcal{H},\mathcal{R})$  -consistency bounds for predictor-rejector two-stage surrogate losses (Theorem 6) 27
H Proof of realizable consistency for predictor-rejector two-stage surrogate losses (Theorem 7) 29
The scenario of single-stage learning to defer has been extensively explored in previous research. The initial studies focused on the problem of abstention and introduced various approaches such as confidence-based abstention [Chow, 1957, 1970, Herbei and Wegkamp, 2005, Bartlett and Wegkamp, 2008, Grandvalet et al., 2008, Yuan and Wegkamp, 2010, 2011, Ramaswamy et al., 2018, Ni et al., 2019], selective classification [El-Yaniv et al., 2010, Wiener and El-Yaniv, 2011, Kalai et al., 2012, Geifman and El-Yaniv, 2017, 2019, Ziyin et al., 2019, Acar et al., 2020, Gangrade et al., 2021], a predictor-rejection framework for abstention [Cortes et al., 2016a,b, Charoenphakdee et al., 2021, Cortes et al., 2023, Mohri et al., 2023, Mao et al., 2023c], and a score-based setting for abstention [Mozannar and Sontag, 2020, Raman and Yee, 2021, Liu et al., 2022, Verma and Nalisnick, 2022, Charusaie et al., 2022, Cao et al., 2022, Mao et al., 2023f, Verma et al., 2023, Mao et al., 2023b, Mozannar et al., 2023].
Another line of research is centered around the joint learning of prediction and deferral functions. Several publications by Madras et al. [2018], Raghu et al. [2019], Wilder et al. [2021], Pradier et al. [2021], Keswani et al. [2021] delve into this topic, considering single-stage learning to defer and its variants. Additionally, the concept of learning to defer has been explored in different scenarios, including combining human and machine predictions, investigating human preferences, regression problems, reinforcement learning, and more [Kamar et al., 2012, Tan et al., 2018, Kleinberg et al., 2018, Bansal et al., 2021, De et al., 2020, Straitouri et al., 2021, Zhao et al., 2021, Joshi et al., 2021, Gao et al., 2021, Mozannar et al., 2022, Hemmer et al., 2023, Narasimhan et al., 2023]. However, in practice, a predictor such as an LLM is already available and retraining one in conjunction with a deferral function could be prohibitively costly: depending on its size and the amount of data used, retraining could take several weeks or months. Thus, the single-stage learning to defer scenario and its solutions often do not align with the practical challenges encountered in real-world applications.
A key criterion for surrogate losses in the scenario of learning to defer is Bayes-consistency (also known as consistency) [Zhang, 2004, Bartlett et al., 2006, Steinwart, 2007, Mohri et al., 2018]. This property guarantees that minimizing the surrogate loss over the family of measurable functions leads to the minimization of the deferral loss. The surrogate losses proposed in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] have shown to be Bayes-consistent for deferral. However, Bayes-consistency is a property associated with the family of all measurable functions, which of course is considerably broader than the hypothesis sets typically used in learning algorithms, including linear hypothesis sets and the family of neural networks.
Instead, Long and Servedio [2013], Kuznetsov et al. [2014], Zhang and Agarwal [2020] proposed a notion of realizable  $\mathcal{H}$ -consistency, that is consistency associated with a specific hypothesis set in the realizable scenario. Mozannar et al. [2023] recently showed that existing Bayes-consistent surrogate losses in [Mozannar and Sontag, 2020, Verma and Nalisnick, 2022] are not realizable  $\mathcal{H}$ -consistent for learning with deferral, which can pose significant challenges when learning with a restricted hypothesis set  $\mathcal{H}$ , even for simple linear models. Instead, they proposed a new surrogate loss that is realizable  $\mathcal{H}$ -consistent when  $\mathcal{H}$  is closed under scaling. However, they also observed that the loss function of Madras et al. [2018], which is not Bayes-consistent, is actually realizable  $\mathcal{H}$ -consistent. They acknowledged their inability to prove or disprove whether their proposed surrogate loss is Bayes-consistent. Consequently, it has remained an open problem to identify a surrogate loss that is both consistent and realizable-consistent.
In recent work, Verma et al. [2023] proposed the first Bayes-consistent surrogate losses in the scenario of learning to defer with multiple experts [Hemmer et al., 2022, Keswani et al., 2021, Kerrigan et al.,
2021, Straitouri et al., 2022, Benz and Rodriguez, 2022]. This scenario is more attractive and significant in applications such as large language models, where multiple models are often available for deferral. However, the surrogate losses proposed by the authors do not benefit from realizable  $\mathcal{H}$ -consistency, even in the single-expert setting, since they are a straightforward generalization of those of Mozannar and Sontag [2020] and Verma and Nalisnick [2022].
In summary, the problem of learning to defer in a single-stage scenario has been extensively studied, but it is often impractical in real-world applications. Post-hoc methods and surrogate losses have been explored, but the challenge remains to find a surrogate loss that is both consistent and realizable-consistent. Recent research has made progress in the scenario of learning to defer with multiple experts but has not achieved realizable  $\mathcal{H}$ -consistency even in a single-expert setting.
B Examples of two-stage score-based surrogate losses
Example:  $\ell_2 = \ell_{\mathrm{exp}}$ . For  $\ell_2(\overline{h}_d, x, y) = \ell_{\mathrm{exp}}(\overline{h}_d, x, y) = \sum_{y' \neq y} e^{\overline{h}_d(x, y') - \overline{h}_d(x, y)}$ , by (3), we have
$$
\begin{array}{l} \mathsf {L} _ {h _ {p}} (h _ {d}, x, y) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \ell_ {2} (\bar {h} _ {d}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {h} _ {d}, x, j) \\ = \mathbb {1} _ {\mathsf {h} _ {p} (x) = y} \sum_ {y ^ {\prime} \neq 0} e ^ {\overline {{h}} _ {d} (x, y ^ {\prime}) - \overline {{h}} _ {d} (x, 0)} + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \sum_ {y ^ {\prime} \neq j} e ^ {\overline {{h}} _ {d} (x, y ^ {\prime}) - \overline {{h}} _ {d} (x, j)} \\ = \mathbb {1} _ {\mathsf {h} _ {\mathsf {p}} (x) = y} \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x, n + i) - \max _ {y \in \mathfrak {y}} h (x, y)} + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \bigg [ \sum_ {i = 1, i \neq j} ^ {n _ {e}} e ^ {h (x, n + i) - h (x, n + j)} + e ^ {\max _ {y \in \mathfrak {y}} h (x, y) - h (x, n + j)} \bigg ]. \\ \end{array}
$$
Example:  $\ell_{2} = \ell_{\log}$ . For  $\ell_{2}(\overline{h}_{d},x,y) = \ell_{\log}(\overline{h}_{d},x,y) = \log \Bigl (\sum_{y^{\prime}\in \mathcal{Y}\cup \{0\}}e^{\overline{h}_{d}(x,y^{\prime}) - \overline{h}_{d}(x,y)}\Bigr)$ , by (3), we have
$$
\begin{array}{l} \mathsf {L} _ {h _ {p}} (h _ {d}, x, y) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \ell_ {2} (\bar {h} _ {d}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {h} _ {d}, x, j) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \log \left(\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{h}} _ {d} (x, y ^ {\prime}) - \overline {{h}} _ {d} (x, 0)}\right) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \log \left(\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{h}} _ {d} (x, y ^ {\prime}) - \overline {{h}} _ {d} (x, j)}\right) \\ = - \mathbb {1} _ {\mathsf {h} _ {\mathsf {p}} (x) = y} \log \left(\frac {e ^ {\max _ {y \in \mathbb {Y}} h (x , y)}}{e ^ {\max _ {y \in \mathbb {Y}} h (x , y)} + \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x , n + i)}}\right) - \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \log \left(\frac {e ^ {h (x , n + j)}}{e ^ {\max _ {y \in \mathbb {Y}} h (x , y)} + \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x , n + i)}}\right). \\ \end{array}
$$
Example:  $\ell_{2} = \ell_{\mathrm{gce}}$ . For  $\ell_{2}(\overline{h}_{d},x,y) = \ell_{\mathrm{gce}}(\overline{h}_{d},x,y) = \frac{1}{\alpha}\left[1 - \left[\frac{e^{\overline{h}_{d}(x,y)}}{\sum_{y^{\prime}\in\mathcal{Y}\cup\{0\}}e^{\overline{h}_{d}(x,y^{\prime})}}\right]^{\alpha}\right], \alpha \in (0,1)$ , by (3), we have
$$
\begin{array}{l} \mathsf {L} _ {h _ {p}} (h _ {d}, x, y) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \ell_ {2} (\bar {h} _ {d}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {h} _ {d}, x, j) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {\overline {{h}} _ {d} (x , 0)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{h}} _ {d} (x , y ^ {\prime})}} \right] ^ {\alpha} \right] + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {\overline {{h}} _ {d} (x , j)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{h}} _ {d} (x , y ^ {\prime})}} \right] ^ {\alpha} \right] \\ = \mathbb {1} _ {\mathsf {h} _ {\mathsf {p}} (x) = y} \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {\max _ {y \in \mathcal {Y}} h (x , y)}}{e ^ {\max _ {y \in \mathcal {Y}} h (x , y)} + \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x , n + i)}} \right] ^ {\alpha} \right] + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {h (x , n + j)}}{e ^ {\max _ {y \in \mathcal {Y}} h (x , y)} + \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x , n + i)}} \right] ^ {\alpha} \right]. \\ \end{array}
$$
Example:  $\ell_{2} = \ell_{\mathrm{mae}}$ . For  $\ell_{2}(\overline{h}_{d},x,y) = \ell_{\mathrm{mae}}(\overline{h}_{d},x,y) = 1 - \frac{e^{\overline{h}_{d}(x,y)}}{\sum_{y^{\prime}\in y\cup\{0\}}e^{\overline{h}_{d}(x,y^{\prime})}}$ , by (3), we have
$$
\begin{array}{l} \mathsf {L} _ {h _ {p}} (h _ {d}, x, y) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \ell_ {2} (\bar {h} _ {d}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {h} _ {d}, x, j) \\ = \mathbb {1} _ {\mathrm {h} _ {p} (x) = y} \left(1 - \frac {e ^ {\overline {{h}} _ {d} (x , 0)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{h}} _ {d} (x , y ^ {\prime})}}\right) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \left(1 - \frac {e ^ {\overline {{h}} _ {d} (x , j)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{h}} _ {d} (x , y ^ {\prime})}}\right) \\ = \mathbb {1} _ {\mathsf {h} _ {\mathsf {p}} (x) = y} \left[ 1 - \frac {e ^ {\max _ {y \in \mathcal {Y}} h (x , y)}}{e ^ {\max _ {y \in \mathcal {Y}} h (x , y)} + \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x , n + i)}} \right] + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \left[ 1 - \frac {e ^ {h (x , n + j)}}{e ^ {\max _ {y \in \mathcal {Y}} h (x , y)} + \sum_ {i = 1} ^ {n _ {e}} e ^ {h (x , n + i)}} \right]. \\ \end{array}
$$
C Examples of two-stage predictor-rejector surrogate losses
Example:  $\ell_2 = \ell_{\mathrm{exp}}$ . For  $\ell_2(\overline{r},x,y) = \ell_{\mathrm{exp}}(\overline{r},x,y) = \sum_{y'\neq y}e^{\overline{r} (x,y') - \overline{r} (x,y)}$ , by (5), we have
$$
\begin{array}{l} \mathsf {L} _ {h} (r, x, y) \\ = \mathbb {1} _ {\mathrm {h} (x) = y} \ell_ {2} (\bar {r}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {r}, x, j) \\ = \mathbb {1} _ {\mathrm {h} (x) = y} \sum_ {y ^ {\prime} \neq 0} e ^ {\bar {r} \left(x, y ^ {\prime}\right) - \bar {r} (x, 0)} + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \sum_ {y ^ {\prime} \neq j} e ^ {\bar {r} \left(x, y ^ {\prime}\right) - \bar {r} (x, j)} \\ = \mathbb {1} _ {\mathsf {h} (x) = y} \sum_ {i = 1} ^ {n _ {e}} e ^ {- r _ {i} (x)} + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \bigg [ \sum_ {i = 1, i \neq j} ^ {n _ {e}} e ^ {r _ {j} (x) - r _ {i} (x)} + e ^ {r _ {j} (x)} \bigg ]. \\ \end{array}
$$
Example:  $\ell_{2} = \ell_{\log}$ . For  $\ell_{2}(\overline{r},x,y) = \ell_{\log}(\overline{r},x,y) = \log \Bigl (\sum_{y^{\prime}\in \mathcal{Y}\cup \{0\}}e^{\overline{r} (x,y^{\prime}) - \overline{r} (x,y)}\Bigr)$ , by (5), we have
$$
\begin{array}{l} \mathsf {L} _ {h} (r, x, y) \\ = \mathbb {1} _ {\mathrm {h} (x) = y} \ell_ {2} (\bar {r}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {r}, x, j) \\ = \mathbb {1} _ {\mathbf {h} (x) = y} \log \left(\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{r}} (x, y ^ {\prime}) - \overline {{r}} (x, 0)}\right) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \log \left(\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\overline {{r}} (x, y ^ {\prime}) - \overline {{r}} (x, j)}\right) \\ = - \mathbb {1} _ {\mathfrak {h} (x) = y} \log \left(\frac {1}{1 + \sum_ {i = 1} ^ {n _ {e}} e ^ {- r _ {i} (x)}}\right) - \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \log \left(\frac {e ^ {- r _ {j} (x)}}{1 + \sum_ {i = 1} ^ {n _ {e}} e ^ {- r _ {i} (x)}}\right). \\ \end{array}
$$
Example:  $\ell_2 = \ell_{\mathrm{gce}}$ . For  $\ell_2(\overline{r},x,y) = \ell_{\mathrm{gce}}(\overline{r},x,y) = \frac{1}{\alpha}\Bigg[1 - \left[\frac{e^{\overline{r}(x,y)}}{\sum_{y'\in y\cup\{0\}}e^{\overline{r}(x,y')}}\right]^{\alpha}\Bigg],\alpha \in (0,1)$ , by (5), we have
$$
\begin{array}{l} \mathsf {L} _ {h} (r, x, y) \\ = \mathbb {1} _ {\mathrm {h} (x) = y} \ell_ {2} (\bar {r}, x, 0) + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \ell_ {2} (\bar {r}, x, j) \\ = \mathbb {1} _ {\mathsf {h} (x) = y} \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {\bar {r} (x , 0)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\bar {r} (x , y ^ {\prime})}} \right] ^ {\alpha} \right] + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \frac {1}{\alpha} \left[ 1 - \left[ \frac {e ^ {\bar {r} (x , j)}}{\sum_ {y ^ {\prime} \in \mathcal {Y} \cup \{0 \}} e ^ {\bar {r} (x , y ^ {\prime})}} \right] ^ {\alpha} \right] \\ = \mathbb {1} _ {\mathsf {h} (x) = y} \frac {1}{\alpha} \bigg [ 1 - \left[ \frac {1}{1 + \sum_ {i = 1} ^ {n _ {e}} e ^ {- r _ {i} (x)}} \right] ^ {\alpha} \bigg ] + \sum_ {j = 1} ^ {n _ {e}} \bar {c} _ {j} (x, y) \frac {1}{\alpha} \bigg [ 1 - \left[ \frac {e ^ {- r _ {j} (x)}}{1 + \sum_ {i = 1} ^ {n _ {e}} e ^ {- r _ {i} (x)}} \right] ^ {\alpha} \bigg ]. \\ \end{array}
$$
Example:  $\ell_2 = \ell_{\mathrm{mae}}$ . For  $\ell_2(\overline{r},x,y) = \ell_{\mathrm{mae}}(\overline{r},x,y) = 1 - \frac{e^{\overline{r}(x,y)}}{\sum_{y'\in y\cup\{0\}}e^{\overline{r}(x,y')}}$ , by (5), we have