SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\overline {{\mathcal {T}}} ^ {\mathrm {c s t n d}} \left(\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}})\right) \leq \mathcal {R} _ {\ell^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\ell^ {\mathrm {c s t n d}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell^ {\mathrm {c s t n d}}} (\overline {{\mathcal {H}}}). \qquad (6)
$$
with  $\overline{\mathfrak{T}}^{\mathrm{cstnd}}$  the  $\overline{\mathcal{H}}$ -estimation error transformation for constrained losses defined for all  $t \in [0,1]$  by  $\overline{\mathfrak{T}}^{\mathrm{cstnd}}(t) =$
$$
\left\{ \begin{array}{l l} \inf  _ {\tau \geq 0} \sup  _ {\mu \in [ \tau - \Lambda_ {\min }, \tau + \Lambda_ {\min } ]} \left\{\frac {1 - t}{2} \big [ \Phi (\tau) - \Phi (- \tau + \mu) \big ] + \frac {1 + t}{2} \big [ \Phi (- \tau) - \Phi (\tau - \mu) \big ] \right\} & n = 2 \\ \inf  _ {P \in \big [ \frac {1}{n - 1}, 1 \big ]} \inf  _ {\tau_ {1} \geq \max  \{\tau_ {2}, 0 \}} \sup  _ {\mu \in C} \Big \{\frac {2 - P - t}{2} \big [ \Phi (- \tau_ {2}) - \Phi (- \tau_ {1} + \mu) \big ] + \frac {2 - P + t}{2} \big [ \Phi (- \tau_ {1}) - \Phi (- \tau_ {2} - \mu) \big ] \Big \} & n > 2, \end{array} \right.
$$
where  $C = \left[\max \{\tau_1, -\tau_2\} - \Lambda_{\min}, \min \{\tau_1, -\tau_2\} + \Lambda_{\min}\right]$  and  $\Lambda_{\min} = \inf_{x \in \mathcal{X}} \Lambda(x)$ . Furthermore, for any  $t \in [0,1]$ , there exist a distribution  $\mathcal{D}$  and a hypothesis  $h \in \mathcal{H}$  such that  $\mathcal{R}_{\ell_{0-1}}(h) - \mathcal{R}_{\ell_{0-1}}^*(\mathcal{H}) + \mathcal{M}_{\ell_{0-1}}(\mathcal{H}) = t$  and  $\mathcal{R}_{\ell^{\mathrm{cstnd}}}(h) - \mathcal{R}_{\ell^{\mathrm{cstnd}}}^*(\mathcal{H}) + \mathcal{M}_{\ell^{\mathrm{cstnd}}}(\mathcal{H}) = \mathcal{T}^{\mathrm{cstnd}}(t)$ .
The proof is presented in Appendix F.1. Next, we illustrate the application of our theory through an example of constrained exponential losses, that is  $\ell^{\mathrm{cstnd}}$  with  $\Phi(t) = e^{-t}$ . By using the error transformation in Theorem 12, we obtain new  $\overline{\mathcal{H}}$ -consistency bounds in Theorem 13 (see Appendix F.2 for the proof) for bounded hypothesis sets  $\overline{\mathcal{H}}$ .
Theorem 13 ( $\overline{\mathcal{H}}$ -consistency bounds for constrained exponential loss). Let  $\Phi(t) = e^{-t}$ . For any  $h \in \overline{\mathcal{H}}$  and any distribution,
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \Psi^ {- 1} \big (\mathcal {R} _ {\ell^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\ell^ {\mathrm {c s t n d}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell^ {\mathrm {c s t n d}}} (\overline {{\mathcal {H}}}) \big)
$$
$$
w h e r e \Psi (t) = \left\{ \begin{array}{l l} 1 - \sqrt {1 - t ^ {2}} & t \leq \frac {e ^ {2 \Lambda_ {\operatorname* {m i n}}} - 1}{e ^ {2 \Lambda_ {\operatorname* {m i n}}} + 1}. \\ \frac {t}{2} \big (e ^ {\Lambda_ {\operatorname* {m i n}}} - e ^ {- \Lambda_ {\operatorname* {m i n}}} \big) + \frac {2 - e ^ {\Lambda_ {\operatorname* {m i n}}} - e ^ {- \Lambda_ {\operatorname* {m i n}}}}{2} & \text {o t h e r w i s e .} \end{array} \right.
$$
Awasthi et al. [2022b] proves  $\mathcal{H}$ -consistency bounds for constrained exponential losses when  $\mathcal{H}$  is symmetric and complete. Theorem 13 significantly generalizes those results to the non-complete hypothesis sets. Different from the complete case, the functional form of our new bounds has two pieces which corresponds to the linear and the square root convergence respectively, modulo the constants. Furthermore, the coefficient of the linear piece depends on the the magnitude of  $\Lambda_{\mathrm{min}}$ . When  $\Lambda_{\mathrm{min}}$  is small, the coefficient of the linear term in  $\Psi^{-1}$  explodes. On the other hand, making  $\Lambda_{\mathrm{min}}$  large essentially turns  $\Psi^{-1}$  into a square-root function.
Here, we further elaborate on the practical value of our tools and  $\mathcal{H}$ -consistency bounds. Our contributions include a more general and convenient mathematical tool for proving  $\mathcal{H}$ -consistency bounds, along with tighter bounds that enable a better comparison of surrogate loss functions and extensions beyond previous completeness assumptions. As mentioned by [Awasthi et al., 2022b], given a hypothesis set  $\mathcal{H}$ ,  $\mathcal{H}$ -consistency bounds can be used to compare different surrogate loss functions and select the most favorable one, which depends on the functional form of the  $\mathcal{H}$ -consistency bound; the smoothness of the surrogate loss and its optimization properties; approximation properties of the surrogate loss function controlled by minimizability gaps; and the dependency on the number of classes in the multiplicative constant. Consequently, a tighter  $\mathcal{H}$ -consistency bound provides a more accurate comparison, as a loose bound might not adequately capture the full advantage of using one surrogate loss. In contrast, Bayes-consistency does not take into account the hypothesis set and is an asymptotic property, thereby failing to guide the comparison of different surrogate losses.
Another application of our  $\mathcal{H}$ -consistency bounds involves deriving generalization bounds for surrogate loss minimizers [Mao et al., 2023h], expressed in terms of the same quantities previously discussed. Therefore, when dealing with finite samples, a tighter  $\mathcal{H}$ -consistency bound could also result in a corresponding tighter generalization bound. Moreover, our novel results extend beyond previous completeness assumptions, offering guarantees applicable to bounded hypothesis sets commonly used with regularization. This enhancement provides meaningful learning guarantees. Technically, our error transformation function serves as a very general tool for deriving  $\mathcal{H}$ -consistency bounds with tightness guarantees. These functions are defined within each class of loss functions including comp-sum losses and constrained losses, and their formulation depends on the structure of the individual loss function class, the range of the hypothesis set and the number of classes. To derive explicit bounds, all that is needed is to calculate these error transformation functions. Under some broad assumptions on the auxiliary function within a loss function, these error transformation functions can be further distilled into more simplified forms, making them straightforward to compute.
We presented a general characterization and extension of  $\mathcal{H}$ -consistency bounds for multi-class classification. We introduced new tools for deriving such bounds with tightness guarantees and illustrated their benefits through several applications and examples. Our proposed method is a significant advance in the theory of  $\mathcal{H}$ -consistency bounds for multi-class classification. It can provide a general and powerful tool for deriving tight bounds for a wide variety of other loss functions and hypothesis sets. We believe that our work will open up new avenues of research in the field of multi-class classification consistency.
Z. Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, 2018.
C. Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. In International Conference on Machine Learning, 2023.
A Related work 16
B Minimizability gap 17
B.1 Zero minimizability 17
B.2 Relationship with approximation error 17
B.3 Significance of  $\mathcal{H}$  -consistency bounds 18
C Proofs for comp-sum losses 18
C.1 Proof of  $\mathcal{H}$  -consistency bounds with  $\mathcal{T}^{\mathrm{comp}}$  (Theorem 2) 19
C.2 Characterization of  $\mathfrak{T}^{\mathrm{comp}}$  (Theorem 3) 21
C.3 Computation of examples 22
D Proofs for constrained losses 23
D.1 Proof of  $\mathcal{H}$ -consistency bounds with  $\mathfrak{T}^{\mathrm{cstnd}}$  (Theorem 10) 24
D.2 Characterization of  $\mathfrak{F}^{\mathrm{cstnd}}$  (Theorem 11) 26
D.3 Computation of examples 27
E Extensions of comp-sum losses 28
E.1 Proof of  $\overline{\mathcal{H}}$ -consistency bounds with  $\overline{\mathcal{T}}^{\mathrm{comp}}$  (Theorem 5) 28
E.2 Logistic loss 30
E.3 Sum exponential loss 32
E.4 Generalized cross-entropy loss 33
E.5 Mean absolute error loss 34
F Extensions of constrained losses 35
F.1 Proof of  $\overline{\mathcal{H}}$ -consistency bound with  $\overline{\mathcal{T}}^{\mathrm{cstnd}}$  (Theorem 12) 35
F.2 Constrained exponential loss 37
The notions of Bayes-consistency (also known as consistency) and calibration have been well studied not only with respect to the binary zero-one loss [Zhang, 2004a, Bartlett et al., 2006, Steinwart, 2007, Mohri et al., 2018], but also with respect to the multi-class zero-one loss [Zhang, 2004b, Tewari and Bartlett, 2007], the general multi-class losses [Ramaswamy and Agarwal, 2012, Narasimhan et al., 2015, Ramaswamy and Agarwal, 2016], the multi-class SVMs [Chen and Sun, 2006, Chen and Xiang, 2006, Liu, 2007, Dogan et al., 2016, Wang and Scott, 2020], the gamma-phi losses [Wang and Scott, 2023], the multi-label losses [Gao and Zhou, 2011, Dembczynski et al., 2012, Zhang et al., 2020], the losses with a reject option [Ramaswamy et al., 2015, Cortes et al., 2016a,b, 2023], the ranking losses [Ravikumar et al., 2011, Ramaswamy et al., 2013, Gao and Zhou, 2015, Uematsu and Lee, 2017], the cost sensitive losses [Pires et al., 2013, Pires and Szepesvári, 2016], the structured losses [Ciliberto et al., 2016, Osokin et al., 2017, Blondel, 2019], the polyhedral losses [Frongillo and Waggoner, 2021, Finocchiaro et al., 2022], the Top- $k$  classification losses [Thilagar et al., 2022], the proper losses [Agarwal and Agarwal, 2015, Williamson et al., 2016] and the losses of ordinal regression [Pedregosa et al., 2017].
Bayes-consistency only holds for the full family of measurable functions, which of course is distinct from the more restricted hypothesis set used by a learning algorithm. Therefore, a hypothesis set-dependent notion of  $\mathcal{H}$ -consistency has been proposed by Long and Servedio [2013] in the realizable setting, used by Zhang and Agarwal [2020] for linear models, and generalized by Kuznetsov et al. [2014] to the structured prediction case. Long and Servedio [2013] showed that there exists a case where a Bayes-consistent loss is not  $\mathcal{H}$ -consistent while inconsistent losses can be  $\mathcal{H}$ -consistent. Zhang and Agarwal [2020] further investigated the phenomenon in [Long and Servedio, 2013] and showed that the situation of losses that are not  $\mathcal{H}$ -consistent with linear models can be remedied by carefully choosing a larger piecewise linear hypothesis set. Kuznetsov et al. [2014] proved positive results for the  $\mathcal{H}$ -consistency of several multi-class ensemble algorithms, as an extension of  $\mathcal{H}$ -consistency results in [Long and Servedio, 2013].
Recently, Awasthi et al. [2022a,b], Mao et al. [2023h], Zheng et al. [2023] presented a series of results providing  $\mathcal{H}$ -consistency bounds. These are upper bounds on the zero-one estimation error of any predictor in a hypothesis set, expressed in terms of its surrogate loss estimation error. They are more informative guarantees than similar excess error bounds derived in the literature, which correspond to the special case where  $\mathcal{H}$  is the family of all measurable functions [Zhang, 2004a, Bartlett et al., 2006, Mohri et al., 2018]. Awasthi et al. [2022a] studied  $\mathcal{H}$ -consistency bounds in binary classification. They provided a series of tight  $\mathcal{H}$ -consistency bounds for bounded hypothesis set of linear models and one-hidden-layer neural networks. The subsequent study [Awasthi et al., 2022b] further generalized the framework to multi-class classification, where they presented a extensive study of  $\mathcal{H}$ -consistency bounds for diverse multi-class surrogate losses including negative results for max losses [Crammer and Singer, 2001] and positive results for sum losses [Weston and Watkins, 1998], and constrained losses [Lee et al., 2004]. However, the hypothesis sets adopted there were assumed to be complete, which rules out the bounded hypothesis sets typically used in practice. Moreover, the final bounds derived from [Awasthi et al., 2022b] are based on ad hoc methods and may not be tight. [Mao et al., 2023h] complemented the previous work by studying a wide family of comp-sum losses in the multi-class classification, which generalized the sum-losses and included as special cases the logistic loss [Verhulst, 1838, 1845, Berkson, 1944, 1951], the generalized cross-entropy loss [Zhang and Sabuncu, 2018], and the mean absolute error loss [Ghosh et al., 2017]. Here too, the completeness assumption on the hypothesis sets was adopted and their  $\mathcal{H}$ -consistency bounds do not apply to common bounded hypothesis sets in practice. Zheng et al. [2023] proved  $\mathcal{H}$ -consistency bounds for multi-class logistic loss with bounded linear hypothesis sets. However, their bounds require a crucial distributional assumption under which, the minimizability gaps coincide with the approximation errors. Thus, their bounds can be recovered as excess error bounds, which are less significant.
This paper provides both a general characterization and an extension of  $\mathcal{H}$ -consistency bounds for multi-class classification. Our general tools and tight bounds show several remarkable advantages: first, they improve existing bounds for complete hypothesis sets previously proven in [Awasthi et al., 2022b], second, they encompass all previously comp-sum and constrained losses studied thus far as well as many new ones [Awasthi et al., 2022a, Mao et al., 2023h], third, they extend beyond the completeness assumption adopted in previous work, fourth, they give novel guarantees for bounded
hypothesis sets, and finally they help prove a much stronger and more significant guarantee for logistic loss with linear hypothesis set than [Zheng et al., 2023].
Other related work on  $\mathcal{H}$ -consistency bounds includes:  $\mathcal{H}$ -consistency bounds for pairwise ranking [Mao et al., 2023d,e]; theoretically grounded surrogate losses and algorithms for learning with abstention supported by  $\mathcal{H}$ -consistency bounds, including the study of score-based abstention [Mao et al., 2023f], predictor-rejection abstention [Mao et al., 2023c] and learning to abstain with a fixed predictor with application in decontextualization [Mohri et al., 2023]; principled approaches for learning to defer with multiple experts that benefit from strong  $\mathcal{H}$ -consistency bounds, including the single-stage scenario [Mao et al., 2023b] and a two-stage scenario [Mao et al., 2023a];  $\mathcal{H}$ -consistency theory and algorithms for adversarial robustness [Awasthi et al., 2021a,b, 2023a, Mao et al., 2023h, Awasthi et al., 2023b]; and efficient algorithms and loss functions for structured prediction with stronger  $\mathcal{H}$ -consistency guarantees [Mao et al., 2023g].
B Minimizability gap
This is a brief discussion of minimizability gaps and their properties. By definition, for any loss function  $\ell$ , the minimizability gap is defined by
$$
\mathcal{M}_{\ell}(\mathcal{H}) = \inf_{h\in \mathcal{H}}\left\{\underset {(x,y)\sim \mathcal{D}}{\mathbb{E}}\bigl[\ell (h,x,y)\bigr ]\right\} -\underset {x}{\mathbb{E}}\Biggl\[\inf_{h\in \mathcal{H}}\underset {y|x}{\mathbb{E}}\bigl[\ell (h,x,y)\bigr ]\Biggr] = \mathcal{R}_{\ell}^{*}(\mathcal{H}) - \underset {x}{\mathbb{E}}\bigl[\mathcal{C}_{\ell}^{*}(\mathcal{H},x)\bigr].
$$
B.1 Zero minimizability
Lemma 14. Let  $\ell$  be a surrogate loss such that for  $(x,y) \in \mathcal{X} \times \mathcal{Y}$  and any measurable function  $h \in \mathcal{H}_{\mathrm{all}}$ , the loss  $\ell(h,x,y)$  only depends on  $h(x)$  and  $y$  (thus we can write  $\ell(h,x,y) = \overline{\ell}(h(x),y)$  for some function  $\overline{\ell}$ ). Then, the minimizability gap vanishes:  $\mathcal{M}_{\ell}(\mathcal{H}_{\mathrm{all}}) = 0$ .
Proof. Fix  $\epsilon > 0$ . Then, by definition of the infimum, for any  $x \in \mathcal{X}$ , there exists  $h_x \in \mathcal{H}_{\mathrm{all}}$  such that
$$
\mathop{\mathbb{E}}\limits_{y|x}\left[\ell \bigl(h_{x},x,y\bigr)\right]\leq \inf_{h\in \mathcal{H}_{\text{all}}}\mathop{\mathbb{E}}\limits_{y|x}\left[\ell \bigl(h,x,y\bigr)\right] + \epsilon .
$$
Now, define the function  $h$  by  $h(x) = h_x(x)$ , for all  $x \in \mathcal{X}$ .  $h$  can be shown to be measurable, for example, when  $\mathcal{X}$  admits a countable dense subset. Then,
$$
\begin{array}{l} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \ell (h, x, y) ] = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \bar {\ell} (h (x), y) ] = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} [ \bar {\ell} (h _ {x} (x), y) ] \\ = \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \ell \left(h _ {x}, x, y\right) \right] \\ \leq \mathbb{E}_{x}\left[ \inf_{h\in \mathcal{H}_{\text{all}}}\mathbb{E}_{y|x}\left[\ell (h,x,y)\right] + \epsilon \right] \\ = \mathbb {E} _ {x} \left[ \mathcal {C} _ {\ell} ^ {*} \left(\mathcal {H} _ {\text {a l l}}, x\right) \right] + \epsilon . \\ \end{array}
$$
$$
\inf  _ {h \in \mathcal {H} _ {\text {a l l}}} \underset {(x, y) \sim \mathcal {D}} {\mathbb {E}} \left[ \ell (h, x, y) \right] \leq \underset {x} {\mathbb {E}} \left[ \mathcal {C} _ {\ell} ^ {*} \left(\mathcal {H} _ {\text {a l l}}, x\right) \right] + \epsilon .
$$
Since the inequality holds for any  $\epsilon > 0$ , we have  $\mathcal{R}_{\ell}^{*}(\mathcal{H}_{\mathrm{all}}) = \inf_{h \in \mathcal{H}_{\mathrm{all}}} \mathbb{E}_{(x,y) \sim \mathcal{D}}[\ell(h,x,y)] \leq \mathbb{E}_x[\mathcal{C}_{\ell}^{*}(\mathcal{H}_{\mathrm{all}},x)]$ . This implies equality since the inequality  $\mathcal{R}_{\ell}^{*}(\mathcal{H}) \geq \mathbb{E}_x[\mathcal{C}_{\ell}^{*}(\mathcal{H},x)]$  holds for any  $\mathcal{H}$ .
B.2 Relationship with approximation error
Let  $\mathcal{A}_{\ell}$  denote the approximation error of a loss function  $\ell$  and a hypothesis set  $\mathcal{H}$ :  $\mathcal{A}_{\ell}(\mathcal{H}) = \mathcal{R}_{\ell}^{*}(\mathcal{H}) - \mathcal{R}_{\ell}^{*}(\mathcal{H}_{\mathrm{all}})$ . We will denote by  $I_{\ell}(\mathcal{H})$  the difference of pointwise infima  $I_{\ell}(\mathcal{H}) = \mathbb{E}_x[\mathcal{C}_\ell^*(\mathcal{H}, x) - \mathcal{C}_\ell^*(\mathcal{H}_{\mathrm{all}}, x)]$ , which is non-negative. The minimizability gap can be decomposed as
follows in terms of the approximation error and the difference of pointwise infima:
$$
\begin{array}{l} \mathcal {M} _ {\ell} (\mathcal {H}) = \mathcal {R} _ {\ell} ^ {*} (\mathcal {H}) - \mathbb {E} _ {x} \left[ \mathcal {C} _ {\ell} ^ {*} (\mathcal {H}, x) \right] \\ = \mathcal {R} _ {\ell} ^ {*} (\mathcal {H}) - \mathcal {R} _ {\ell} ^ {*} \left(\mathcal {H} _ {\mathrm {a l l}}\right) + \mathcal {R} _ {\ell} ^ {*} \left(\mathcal {H} _ {\mathrm {a l l}}\right) - \mathbb {E} _ {x} \left[ \mathcal {C} _ {\ell} ^ {*} (\mathcal {H}, x) \right] \\ = \mathcal {A} _ {\ell} (\mathcal {H}) + \mathcal {R} _ {\ell} ^ {*} (\mathcal {H} _ {\text {a l l}}) - \mathbb {E} _ {x} \left[ \mathcal {C} _ {\ell} ^ {*} (\mathcal {H}, x) \right] \\ = \mathcal {A} _ {\ell} (\mathcal {H}) + \mathbb {E} _ {x} \left[ \mathcal {C} _ {\ell} ^ {*} \left(\mathcal {H} _ {\text {a l l}}, x\right) - \mathcal {C} _ {\ell} ^ {*} (\mathcal {H}, x) \right] \tag {By Lemma 14} \\ = \mathcal {A} _ {\ell} (\mathcal {H}) - I _ {\ell} (\mathcal {H}). \\ \end{array}
$$
The decomposition immediately implies the following result.
Lemma 15. Let  $\ell$  be a surrogate loss such that for  $(x,y)\in \mathcal{X}\times \mathcal{Y}$  and any measurable function  $h\in \mathcal{H}_{\mathrm{all}}$ , the loss  $\ell (h,x,y)$  only depends on  $h(x)$  and  $y$  (thus we can write  $\ell (h,x,y) = \overline{\ell} (h(x),y)$  for some function  $\overline{\ell}$ ). Then, for any loss function  $\ell$  and hypothesis set  $\mathcal{H}$ , we have:  $\mathcal{M}_{\ell}(\mathcal{H})\leq \mathcal{A}_{\ell}(\mathcal{H})$ .
By Lemma 1, when  $\ell$  is the zero-one loss,  $I_{\ell}(\mathcal{H}) = 0$  when the hypothesis set generates labels that cover all possible outcomes for each input. However, for a surrogate loss function,  $I_{\ell}(\mathcal{H})$  is non-negative, and is generally non-zero.
Take the example of binary classification and denote the conditional distribution as  $\eta(x) = D(Y = 1 | X = x)$ . Let  $\mathcal{H}$  be a family of functions  $h$  with  $|h(x)| \leq \Lambda$  for all  $x \in \mathcal{X}$  and such that all values in  $[- \Lambda, +\Lambda]$  can be reached. Consider for example the exponential-based margin loss:  $\ell(h, x, y) = e^{-yh(x)}$ . Then,  $\mathcal{C}_{\ell}(h, x) = \eta(x)e^{-h(x)} + (1 - \eta(x))e^{h(x)}$ . Upon observing this, it becomes apparent that the infimum over all measurable functions can be expressed in the following way, for all  $x$ :
$$
\mathcal {C} _ {\ell} ^ {*} \left(\mathcal {H} _ {\text {a l l}}, x\right) = 2 \sqrt {\eta (x) \left(1 - \eta (x)\right)},
$$
while the infimum over  $\mathcal{H}$ ,  $\mathcal{C}_{\ell}^{*}(\mathcal{H},x)$ , depends on  $\Lambda$  and can be expressed as
$$
\mathcal {C} _ {\ell} ^ {*} (\mathcal {H}, x) = \left\{ \begin{array}{l l} \max  \{\eta (x), 1 - \eta (x) \} e ^ {- \Lambda} + \min  \{\eta (x), 1 - \eta (x) \} e ^ {\Lambda} & \Lambda <   \frac {1}{2} \Big | \log \frac {\eta (x)}{1 - \eta (x)} \Big | \\ 2 \sqrt {\eta (x) (1 - \eta (x))} & \text {o t h e r w i s e .} \end{array} \right.
$$
Thus, in the deterministic scenario,
$$
I _ {\ell} \big (\mathcal {H} \big) = \mathbb {E} _ {x} \big [ \mathcal {C} _ {\ell} ^ {*} \big (\mathcal {H}, x \big) - \mathcal {C} _ {\ell} ^ {*} \big (\mathcal {H} _ {\mathrm {a l l}}, x \big) \big ] = e ^ {- \Lambda}.
$$
B.3 Significance of  $\mathcal{H}$ -consistency bounds
As shown in the previous section, for target loss  $\ell_{0-1}$ , the minimizability gap coincides with the approximation error  $\mathcal{M}_{\ell_{0-1}}(\mathcal{H}) = \mathcal{A}_{\ell_{0-1}}(\mathcal{H})$  when the hypothesis set generates labels that cover all possible outcomes for each input. However, for a surrogate loss  $\ell$ , the minimizability gap is generally strictly less than the approximation error  $\mathcal{M}_{\ell}(\mathcal{H}) < \mathcal{A}_{\ell}(\mathcal{H})$ . Thus, an  $\mathcal{H}$ -consistency bound, expressed as follows for some increasing function  $\Gamma$ :
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H}) \leq \Gamma \left(\mathcal {R} _ {\ell} (h) - \mathcal {R} _ {\ell} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell} (\mathcal {H})\right).
$$
is more favorable than an excess error bound expressed in terms of approximation errors:
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) + \mathcal {A} _ {\ell_ {0 - 1}} (\mathcal {H}) \leq \Gamma \left(\mathcal {R} _ {\ell} (h) - \mathcal {R} _ {\ell} ^ {*} (\mathcal {H}) + \mathcal {A} _ {\ell} (\mathcal {H})\right).
$$
Here,  $\Gamma$  is typically linear or the square-root function modulo constants. When  $\mathcal{H} = \mathcal{H}_{\mathrm{all}}$ , the family of all measurable functions, by Lemma 14, the  $\mathcal{H}$ -consistency bound coincides with the excess error bound and implies Bayes-consistency by taking the limit. It is therefore a stronger guarantee than an excess error bound and Bayes-consistency.
C Proofs for comp-sum losses
Let  $y_{\max} = \operatorname{argmax}_{y \in \mathcal{Y}} p(x, y)$  and  $h(x) = \operatorname{argmax}_{y \in \mathcal{Y}} h(x, y)$ , where we choose the label with the highest index under the natural ordering of labels as the tie-breaking strategy.
C.1 Proof of  $\mathcal{H}$ -consistency bounds with  $\mathcal{T}^{\mathrm{comp}}$  (Theorem 2)