SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\{ \begin{array}{l l} \inf  _ {\tau \in [ 0, \frac {1}{2} ]} \sup  _ {\mu \in [ - \tau , 1 - \tau ]} \left\{\frac {1 + t}{2} \big [ \Phi (\tau) - \Phi (1 - \tau - \mu) \big ] + \frac {1 - t}{2} \big [ \Phi (1 - \tau) - \Phi (\tau + \mu) \big ] \right\} & n = 2 \\ \inf  _ {P \in [ \frac {1}{n - 1} \vee t, 1 ]} \inf  _ {\substack {\tau_ {1} \geq \max  (\tau_ {2}, 1 / n) \\ \tau_ {1} + \tau_ {2} \leq 1, \tau_ {2} \geq 0}} \sup  _ {\mu \in [ - \tau_ {2}, \tau_ {1} ]} \left\{\frac {P + t}{2} \big [ \Phi (\tau_ {2}) - \Phi (\tau_ {1} - \mu) \big ] + \frac {P - t}{2} \big [ \Phi (\tau_ {1}) - \Phi (\tau_ {2} + \mu) \big ] \right\} & n > 2. \end{array} \right.
$$
Furthermore, for any  $t \in [0,1]$ , there exist a distribution  $\mathcal{D}$  and a hypothesis  $h \in \mathcal{H}$  such that  $\mathcal{R}_{\ell_{0-1}}(h) - \mathcal{R}_{\ell_{0-1}}^*(\mathcal{H}) + \mathcal{M}_{\ell_{0-1}}(\mathcal{H}) = t$  and  $\mathcal{R}_{\ell^{\mathrm{comp}}}(h) - \mathcal{R}_{\ell^{\mathrm{comp}}}^*(\mathcal{H}) + \mathcal{M}_{\ell^{\mathrm{comp}}}(\mathcal{H}) = \mathcal{T}^{\mathrm{comp}}(t)$ .
Thus, Theorem 2 shows that, when  $\mathcal{T}^{\mathrm{comp}}$  is convex, to make these guarantees explicit, all that is needed is to calculate  $\mathcal{T}^{\mathrm{comp}}$ . Moreover, the last statement shows the tightness of the guarantees derived using this function. The constraints in  $\mathcal{T}^{\mathrm{comp}}$  are due to the forms that the conditional probability vector and scoring functions take. These forms become more flexible for  $n > 2$ , leading to intricate constraints. Note that our  $\mathcal{H}$ -consistency bounds are distribution-independent and we cannot claim tightness across all distributions.
The general expression of  $\mathfrak{T}^{\mathrm{comp}}$  in Theorem 2 is complex, but it can be considerably simplified under some broad assumptions, as shown by the following result.
Theorem 3 (characterization of  $\mathfrak{T}^{\mathrm{comp}}$ ). Assume that  $\Phi$  is convex, differentiable at  $\frac{1}{2}$  and  $\Phi' \left( \frac{1}{2} \right) < 0$ . Then,  $\mathfrak{T}^{\mathrm{comp}}$  can be expressed as follows:
$$
\mathfrak {T} ^ {\mathrm {c o m p}} (t) = \left\{ \begin{array}{l l} \Phi \Big (\frac {1}{2} \Big) - \inf  _ {\mu \in [ - \frac {1}{2}, \frac {1}{2} ]} \Big \{\frac {1 - t}{2} \Phi \Big (\frac {1}{2} + \mu \Big) + \frac {1 + t}{2} \Phi \Big (\frac {1}{2} - \mu \Big) \Big \} & n = 2 \\ \inf  _ {\tau \in [ \frac {1}{n}, \frac {1}{2} ]} \Big \{\Phi (\tau) - \inf  _ {\mu \in [ - \tau , \tau ]} \Big \{\frac {1 + t}{2} \Phi (\tau - \mu) + \frac {1 - t}{2} \Phi (\tau + \mu) \Big \} \Big \} & n > 2. \end{array} \right.
$$
The proof of this result as well as that of other theorems in this section are given in Appendix C.
Examples. We now illustrate the application of our theory through several examples. To do so, we compute the  $\mathcal{H}$ -estimation error transformation  $\mathcal{T}^{\mathrm{comp}}$  for comp-sum losses and present the results in
Table 1:  $\mathcal{H}$ -estimation error transformation for common comp-sum losses.
Auxiliary function Φ	-log(t)	1/t-1	1/q(1-t^q), q ∈ (0,1)	1/t	(1-t)^2
Transformation Jcomp	1+t/2 log(1+t) + 1/t/2 log(1-t)	1 - √1 - t^2	1/qn^q((1+t)^1/q + (1-t)^1/q)	1-q/n	t^2/4
Table 1. Remarkably, by applying Theorem 2, we are able to obtain the same  $\mathcal{H}$ -consistency bounds for comp-sum losses with  $\Phi(t) = -\log(t)$ ,  $\frac{1}{t} - 1$ ,  $\frac{1}{q}(1 - t^q)$ ,  $q \in (0,1)$  and  $1 - t$  as those derived using ad hoc methods in [Mao et al., 2023h], and a novel tight  $\mathcal{H}$ -consistency bound for the new comp-sum loss  $\ell_{\mathrm{sq}} = \left[1 - \frac{e^{h(x,y)}}{\sum_{y' \in y} e^{h(x,y')} \right]^2$  with  $\Phi(t) = (1 - t)^2$  in Theorem 4.
The calculation of  $\mathfrak{T}^{\mathrm{comp}}$  for all entries of Table 1 is detailed in Appendix C.3. To illustrate the effectiveness of our general tools, here, we show how the error transformation function can be straightforwardly calculated in the case of the new surrogate loss  $\ell_{\mathrm{sq}}$ .
Theorem 4 (H-consistency bound for a new comp-sum loss). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for all  $h\in \mathcal{H}$  and any distribution, the following tight bound holds.
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) \leq 2 \big (\mathcal {R} _ {\ell_ {\mathrm {s q}}} (h) - \mathcal {R} _ {\ell_ {\mathrm {s q}}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {\mathrm {s q}}} (\mathcal {H}) \big) ^ {\frac {1}{2}} - \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H}).
$$
Proof. For  $n = 2$ , plugging in  $\Phi(t) = (1 - t)^2$  in Theorem 3, gives
$$
\mathfrak {T} ^ {\mathrm {c o m p}} = \frac {1}{4} - \inf _ {\mu \in [ - \frac {1}{2}, \frac {1}{2} ]} \left\{\frac {1 - t}{2} \bigg (\frac {1}{2} - \mu \bigg) ^ {2} + \frac {1 + t}{2} \bigg (\frac {1}{2} + \mu \bigg) ^ {2} \right\} = \frac {1}{4} - \frac {1 - t ^ {2}}{4} = \frac {t ^ {2}}{4}.
$$
Similarly, for  $n > 2$ , plugging in  $\Phi(t) = (1 - t)^2$  in Theorem 3 yields
$$
\begin{array}{l} \mathfrak {T} ^ {\mathrm {c o m p}} = \inf  _ {\tau \in \left[ \frac {1}{n}, \frac {1}{2} \right]} \left\{\left(1 - \tau\right) ^ {2} - \inf  _ {\mu \in [ - \tau , \tau ]} \left\{\frac {1 + t}{2} (1 - \tau + \mu) ^ {2} + \frac {1 - t}{2} (1 - \tau - \mu) ^ {2} \right\} \right\} \\ = \inf  _ {\tau \in \left[ \frac {1}{n}, \frac {1}{2} \right]} \left\{\left(1 - \tau\right) ^ {2} - \left(1 - \tau\right) ^ {2} \left(1 - t ^ {2}\right) \right\} \quad (\text {m i n i m u m a c h i e v e d a t} \mu = t (\tau - 1)) \\ = \frac {t ^ {2}}{4}. \quad (\text {m i n i m u m a c h i e v e d a t} \tau = \frac {1}{2}) \\ \end{array}
$$
By Theorem 2, the bound obtained is tight, which completes the proof.
4.2 Extension to non-complete/bounded hypothesis sets: comp-sum losses
As pointed out earlier, the hypothesis sets typically used in practice are bounded. Let  $\mathcal{F}$  be a family of real-valued functions  $f$  with  $|f(x)| \leq \Lambda(x)$  for all  $x \in \mathcal{X}$  and such that all values in  $[- \Lambda(x), +\Lambda(x)]$  can be reached, where  $\Lambda(x) > 0$  is a fixed function on  $\mathcal{X}$ . We will study hypothesis sets  $\overline{\mathcal{H}}$  in which each scoring function is bounded:
$$
\overline {{\mathcal {H}}} = \{h: \mathcal {X} \times \mathcal {Y} \rightarrow \mathbb {R} \mid h (\cdot , y) \in \mathcal {F}, \forall y \in \mathcal {Y} \}. \tag {4}
$$
This holds for most hypothesis sets used in practice. The symmetric and complete hypothesis sets studied in previous work correspond to the special case of  $\overline{\mathcal{H}}$  where  $\Lambda(x) = +\infty$  for all  $x \in \mathcal{X}$ . The hypothesis set of linear models  $\mathcal{H}_{\mathrm{lin}}$ , defined by
$$
\mathcal {H} _ {\mathrm {l i n}} = \left\{\left(x, y\right) \mapsto w _ {y} \cdot x + b _ {y} \mid \| w _ {y} \| \leq W, | b _ {y} | \leq B, y \in \mathcal {Y} \right\},
$$
is also a special instance of  $\overline{\mathcal{H}}$  where  $\Lambda(x) = W\|x\| + B$ . Let us emphasize that previous studies did not establish any  $\mathcal{H}$ -consistency bound for these general hypothesis sets,  $\overline{\mathcal{H}}$ .
Theorem 5 ( $\overline{\mathcal{H}}$ -consistency bound for comp-sum losses). Assume that  $\overline{\mathcal{T}}^{\mathrm{comp}}$  is convex. Then, the following inequality holds for any hypothesis  $h \in \overline{\mathcal{H}}$  and any distribution:
$$
\overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} \left(\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}})\right) \leq \mathcal {R} _ {\ell^ {\mathrm {c o m p}}} (h) - \mathcal {R} _ {\ell^ {\mathrm {c o m p}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell^ {\mathrm {c o m p}}} (\overline {{\mathcal {H}}})
$$
with  $\overline{\mathcal{T}}^{\mathrm{comp}}$  the  $\overline{\mathcal{H}}$ -estimation error transformation for comp-sum losses defined for all  $t \in [0,1]$  by  $\overline{\mathcal{T}}^{\mathrm{comp}}(t) =$
$$
\left\{ \begin{array}{l l} \inf  _ {\tau \in \left[ 0, \frac {1}{2} \right]} \sup  _ {\mu \in \left[ s _ {\min } - \tau , 1 - \tau - s _ {\min } \right]} \left\{\frac {1 + t}{2} \left[ \Phi (\tau) - \Phi (1 - \tau - \mu) \right] + \frac {1 - t}{2} \left[ \Phi (1 - \tau) - \Phi (\tau + \mu) \right] \right\} & n = 2 \\ \inf  _ {P \in \left[ \frac {1}{n - 1} \vee t, 1 \right]} \inf  _ {S _ {\min } \leq \tau_ {2} \leq \tau_ {1} \leq S _ {\max }} \sup  _ {\mu \in C} \left\{\frac {P + t}{2} \left[ \Phi (\tau_ {2}) - \Phi (\tau_ {1} - \mu) \right] + \frac {P - t}{2} \left[ \Phi (\tau_ {1}) - \Phi (\tau_ {2} + \mu) \right] \right\} & n > 2, \end{array} \right.
$$
where  $C = \left[\max \{s_{\min} - \tau_2, \tau_1 - s_{\max}\}, \min \{s_{\max} - \tau_2, \tau_1 - s_{\min}\}\right]$ ,  $s_{\max} = \frac{1}{1 + (n-1)e^{-2\inf_x\Lambda(x)}}$  and  $s_{\min} = \frac{1}{1 + (n-1)e^{2\inf_x\Lambda(x)}}$ . Furthermore, for any  $t \in [0,1]$ , there exist a distribution  $\mathcal{D}$  and  $h \in \mathcal{H}$  such that  $\mathcal{R}_{\ell_{0-1}}(h) - \mathcal{R}_{\ell_{0-1}}^*(\mathcal{H}) + \mathcal{M}_{\ell_{0-1}}(\mathcal{H}) = t$  and  $\mathcal{R}_{\ell^{\mathrm{comp}}}(h) - \mathcal{R}_{\ell^{\mathrm{comp}}}^*(\mathcal{H}) + \mathcal{M}_{\ell^{\mathrm{comp}}}(\mathcal{H}) = \mathcal{T}^{\mathrm{comp}}(t)$ .
This theorem significantly broadens the applicability of our framework as it encompasses bounded hypothesis sets. The last statement of the theorem further shows the tightness of the  $\mathcal{H}$ -consistency bounds derived using this error transformation function. We now illustrate the application of our theory through several examples.
A. Example: logistic loss. We first consider the multinomial logistic loss, that is  $\ell^{\mathrm{comp}}$  with  $\Phi(u) = -\log(u)$ , for which we give the following guarantee.
Theorem 6 ( $\overline{\mathcal{H}}$ -consistency bounds for logistic loss). For any  $h \in \overline{\mathcal{H}}$  and any distribution, we have
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \Psi^ {- 1} \Big (\mathcal {R} _ {\ell_ {\log}} (h) - \mathcal {R} _ {\ell_ {\log}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\log}} (\overline {{\mathcal {H}}}) \Big),
$$
where  $\ell_{\log} = -\log \left(\frac{e^{h(x,y)}}{\sum_{y'\in\mathcal{Y}}e^{h(x,y')}}\right)$  and  $\Psi (t) = \begin{cases} \frac{1 + t}{2}\log (1 + t) + \frac{1 - t}{2}\log (1 - t) & t\leq \frac{s_{\max} - s_{\min}}{s_{\min} + s_{\max}}\\ \frac{t}{2}\log \left(\frac{s_{\max}}{s_{\min}}\right) + \log \left(\frac{2\sqrt{s_{\max}s_{\min}}}{s_{\max} + s_{\min}}\right) & \text{otherwise.} \end{cases}$
The proof of Theorem 6 is given in Appendix E.2. With the help of some simple calculations, we can derive a simpler upper bound:
$$
\Psi^ {- 1} (t) \leq \Gamma (t) = \left\{ \begin{array}{l l} \sqrt {2 t} & t \leq \frac {(s _ {\max} - s _ {\min}) ^ {2}}{2 (s _ {\min} + s _ {\max}) ^ {2}} \\ \frac {2 (s _ {\min} + s _ {\max})}{s _ {\max} - s _ {\min}} t & \text {o t h e r w i s e .} \end{array} \right.
$$
When the relative difference between  $s_{\mathrm{min}}$  and  $s_{\mathrm{max}}$  is small, the coefficient of the linear term in  $\Gamma$  explodes. On the other hand, making that difference large essentially turns  $\Gamma$  into a square-root function for all values. In general,  $\Lambda$  is not infinite since a regularization is used, which controls both the complexity of the hypothesis set and the magnitude of the scores.
Comparison with [Mao et al., 2023h]. For the symmetric and complete hypothesis sets  $\mathcal{H}$  considered in [Mao et al., 2023h],  $\Lambda(x) = +\infty$ ,  $s_{\max} = 1$ ,  $s_{\min} = 0$ ,  $\Psi(t) = \frac{1+t}{2} \log(1+t) + \frac{1-t}{2} \log(1-t)$  and  $\Gamma(t) = \sqrt{2t}$ . By Theorem 6, this yields an  $\mathcal{H}$ -consistency bound for the logistic loss.
Corollary 7 (H-consistency bounds for logistic loss). Assume that  $\mathcal{H}$  is symmetric and complete. Then, for any  $h\in \mathcal{H}$  and any distribution, we have
$$
\left. \mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) \leq \Psi^ {- 1} \left(\mathcal {R} _ {\ell_ {\log}} (h) - \mathcal {R} _ {\ell_ {\log}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {\log}} (\mathcal {H})\right) - \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H}) \right.
$$
where  $\Psi(t) = \frac{1 + t}{2} \log(1 + t) + \frac{1 - t}{2} \log(1 - t)$  and  $\Psi^{-1}(t) \leq \sqrt{2t}$ .
Corollary 7 recovers the  $\mathcal{H}$ -consistency bounds of Mao et al. [2023h].
Comparison with [Awasthi et al., 2022a] and [Zheng et al., 2023]. For the linear models  $\mathcal{H}_{\mathrm{lin}} = \{(x,y)\mapsto w_y\cdot x + b_y\mid \| w_y\| \leq W,|b_y|\leq B\}$ , we have  $\Lambda (x) = W\| x\| +B$ . By Theorem 6, we obtain  $\mathcal{H}_{\mathrm{lin}}$ -consistency bounds for logistic loss.
Corollary 8 ( $\mathcal{H}_{\mathrm{lin}}$ -consistency bounds for logistic loss). For any  $h \in \mathcal{H}_{\mathrm{lin}}$  and any distribution,
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H} _ {\mathrm {l i n}}) \leq \Psi^ {- 1} \left(\mathcal {R} _ {\ell_ {\log}} (h) - \mathcal {R} _ {\ell_ {\log}} ^ {*} (\mathcal {H} _ {\mathrm {l i n}}) + \mathcal {M} _ {\ell_ {\log}} (\mathcal {H} _ {\mathrm {l i n}})\right) - \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H} _ {\mathrm {l i n}})
$$
where  $\Psi(t) = \begin{cases} \frac{1 + t}{2}\log (1 + t) + \frac{1 - t}{2}\log (1 - t) & t\leq \frac{(n - 1)(e^{2B} - e^{-2B})}{2 + (n - 1)(e^{2B} + e^{-2B})} \\ \frac{t}{2}\log \left(\frac{1 + (n - 1)e^{2B}}{1 + (n - 1)e^{-2B}}\right) + \log \left(\frac{2\sqrt{(1 + (n - 1)e^{2B})(1 + (n - 1)e^{-2B})}}{2 + (n - 1)(e^{2B} + e^{-2B})}\right) & \text{otherwise.} \end{cases}$
For  $n = 2$ , we have  $\Psi(t) = \begin{cases} \frac{t + 1}{2}\log (t + 1) + \frac{1 - t}{2}\log (1 - t) & t \leq \frac{e^{2B} - 1}{e^{2B} + 1} \\ \frac{t}{2}\log \left(\frac{1 + e^{2B}}{1 + e^{-2B}}\right) + \log \left(\frac{2\sqrt{(1 + e^{2B})(1 + e^{-2B})}}{2 + e^{2B} + e^{-2B}}\right) & \text{otherwise,} \end{cases}$  which coincides with the  $\mathcal{H}_{\mathrm{lin}}$ -estimation error transformation in [Awasthi et al., 2022a]. Thus, Corollary 8 includes as a special case the  $\mathcal{H}_{\mathrm{lin}}$ -consistency bounds given by Awasthi et al. [2022a] for binary classification.
Our bounds of Corollary 8 improve upon the multi-class  $\mathcal{H}_{\mathrm{lin}}$ -consistency bounds of recent work [Zheng et al., 2023, Theorem 3.3] in the following ways: i) their bound holds only for restricted distributions while our bound holds for any distribution; ii) their bound holds only for restricted values of the estimation error  $\mathcal{R}_{\ell_{\log}}(h) - \mathcal{R}_{\ell_{\log}}^{*}(\mathcal{H}_{\mathrm{lin}})$  while ours holds for any value in  $\mathbb{R}$  and more precisely admits a piecewise functional form; iii) under their distributional assumption, the minimizability gaps  $\mathcal{M}_{\ell_{0-1}}(\mathcal{H}_{\mathrm{lin}})$  and  $\mathcal{M}_{\ell_{\log}}(\mathcal{H}_{\mathrm{lin}})$  coincide with the approximation errors  $\mathcal{R}_{\ell_{0-1}}(\mathcal{H}_{\mathrm{lin}}) - \mathcal{R}_{\ell_{0-1}}^{*}(\mathcal{H}_{\mathrm{all}})$  and  $\mathcal{R}_{\ell_{\log}}(\mathcal{H}_{\mathrm{lin}}) - \mathcal{R}_{\ell_{\log}}^{*}(\mathcal{H}_{\mathrm{all}})$  respectively (see the note before [Zheng et al., 2023, Appendix F]). Thus, their bounds can be recovered as an excess error bound  $\mathcal{R}_{\ell_{0-1}}(h) - \mathcal{R}_{\ell_{0-1}}^{*}(\mathcal{H}_{\mathrm{all}}) \leq \sqrt{2}\big[\mathcal{R}_{\ell_{\log}}(h) - \mathcal{R}_{\ell_{\log}}^{*}(\mathcal{H}_{\mathrm{all}})\big]^{\frac{1}{2}}$ , which is not specific to the hypothesis set  $\mathcal{H}$  and thus not as significant. In contrast, our  $\mathcal{H}_{\mathrm{lin}}$ -consistency bound is finer and takes into account the role of the parameter  $B$  as well as the number of labels  $n$ ; iv) [Zheng et al., 2023, Theorem 3.3] only offers approximate bounds that are not tight; in contrast, by Theorem 5, our bound is tight.
Note that our  $\mathcal{H}$ -consistency bound in Theorem 6 are not limited to specific hypothesis set forms. They are directly applicable to various types of hypothesis sets including neural networks. For example, the same derivation can be extended to one-hidden-layer neural networks studied in [Awasthi et al., 2022a] and their multi-class generalization by calculating and substituting the corresponding  $\Lambda(x)$ . As a result, we can obtain novel and tight  $\mathcal{H}$ -consistency bounds for bounded neural network hypothesis sets in multi-class classification, which highlights the versatility of our general tools.
B. Example: sum exponential loss. We then consider the sum exponential loss, that is  $\ell^{\mathrm{comp}}$  with  $\Phi(u) = \frac{1 - u}{u}$ . By computing the error transformation in Theorem 5, we obtain the following result.
Theorem 9 ( $\mathcal{H}$ -consistency bounds for sum exponential loss). For any  $h \in \mathcal{H}$  and any distribution,
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \Psi^ {- 1} \big (\mathcal {R} _ {\ell_ {\exp}} (h) - \mathcal {R} _ {\ell_ {\exp}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\exp}} (\overline {{\mathcal {H}}}) \big)
$$
where  $\ell_{\mathrm{exp}} = \sum_{y^{\prime}\neq y}e^{h(x,y^{\prime}) - h(x,y)}$  and  $\Psi (t) = \left\{ \begin{array}{ll}1 - \sqrt{1 - t^2} & t\leq \frac{s_{\mathrm{max}}^2 - s_{\mathrm{min}}^2}{s_{\mathrm{min}}^2 + s_{\mathrm{max}}^2}\\ \frac{s_{\mathrm{max}} - s_{\mathrm{min}}}{2s_{\mathrm{max}}s_{\mathrm{min}}} t - \frac{(s_{\mathrm{max}} - s_{\mathrm{min}})^2}{2s_{\mathrm{max}}s_{\mathrm{min}}(s_{\mathrm{max}} + s_{\mathrm{min}})} & \mathrm{otherwise.} \end{array} \right.$
The proof of Theorem 9 is given in Appendix E.3. Observe that  $1 - \sqrt{1 - t^2} \geq t^2 / 2$ . By Theorem 9, making  $s_{\min}$  close to zero, that is making  $\Lambda$  close to infinite for any  $x \in \mathcal{X}$ , essentially turns  $\Psi$  into a square function for all values. In general,  $\Lambda$  is not infinite since a regularization is used in practice, which controls both the complexity of the hypothesis set and the magnitude of the scores.
C. Example: generalized cross-entropy loss and mean absolute error loss. Due to space limitations, we present the results for these loss functions in Appendix E.
5 Constrained losses
In this section, we present a general characterization of  $\mathcal{H}$ -consistency bounds for constrained loss, that is loss functions defined via a constraint, as in [Lee et al., 2004]:
$$
\ell^ {\text {c s t n d}} (h, x, y) = \sum_ {y ^ {\prime} \neq y} \Phi (- h (x, y ^ {\prime})) \tag {5}
$$
with the constraint that  $\sum_{y\in \mathcal{Y}}h(x,y) = 0$  for a non-negative and non-increasing auxiliary function  $\Phi$ .
5.1  $\mathcal{H}$ -consistency bounds
As in the previous section, we prove a result that supplies a very general tool, an error transformation function for deriving  $\mathcal{H}$ -consistency bounds for constrained losses. When  $\mathfrak{T}^{\mathrm{cstnd}}$  is convex, to make these guarantees explicit, we only need to calculate  $\mathfrak{T}^{\mathrm{cstnd}}$ .
Table 2:  $\mathcal{H}$ -estimation error transformation for common constrained losses.
Auxiliary function Φ	Φexp(t) = e- t	Φhinge(t) = max{0,1-t}	Φsq-hinge(t) = (1-t)2Πt≤1	Φsq = (1-t)2
Transformation Tcstd	Tcstd(n) = 2 - √4 - t2	Tcstd(n) = t	Tcstd(n) = t2/2	Tcstd(n) = t2/2
Theorem 10 ( $\mathcal{H}$ -consistency bound for constrained losses). Assume that  $\mathcal{H}$  is symmetric and complete. Assume that  $\mathfrak{T}^{\mathrm{cstnd}}$  is convex. Then, for any hypothesis  $h\in \mathcal{H}$  and any distribution,
$$
\mathfrak {T} ^ {\mathrm {c s t n d}} \Big (\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell_ {0 - 1}} (\mathcal {H}) \Big) \leq \mathcal {R} _ {\ell^ {\mathrm {c s t n d}}} (h) - \mathcal {R} _ {\ell^ {\mathrm {c s t n d}}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell^ {\mathrm {c s t n d}}} (\mathcal {H})
$$
with  $\mathcal{H}$ -estimation error transformation for constrained losses defined on  $t \in [0,1]$  by  $\mathfrak{T}^{\mathrm{cstnd}}(t) =$
$$
\left\{ \begin{array}{l l} \inf  _ {\tau \geq 0} \sup  _ {\mu \in \mathbb {R}} \Big \{\frac {1 - t}{2} \big [ \Phi (\tau) - \Phi (- \tau + \mu) \big ] + \frac {1 + t}{2} \big [ \Phi (- \tau) - \Phi (\tau - \mu) \big ] \Big \} & n = 2 \\ \inf  _ {P \in \big [ \frac {1}{n - 1}, 1 \big ]} \inf  _ {\tau_ {1} \geq \max  \{\tau_ {2}, 0 \}} \sup  _ {\mu \in \mathbb {R}} \Big \{\frac {2 - P - t}{2} \big [ \Phi (- \tau_ {2}) - \Phi (- \tau_ {1} + \mu) \big ] + \frac {2 - P + t}{2} \big [ \Phi (- \tau_ {1}) - \Phi (- \tau_ {2} - \mu) \big ] \Big \} & n > 2. \end{array} \right.
$$
Furthermore, for any  $t \in [0,1]$ , there exist a distribution  $\mathcal{D}$  and a hypothesis  $h \in \mathcal{H}$  such that  $\mathcal{R}_{\ell_{0-1}}(h) - \mathcal{R}_{\ell_{0-1}}^*(\mathcal{H}) + \mathcal{M}_{\ell_{0-1}}(\mathcal{H}) = t$  and  $\mathcal{R}_{\ell^{\mathrm{cstnd}}}(h) - \mathcal{R}_{\ell^{\mathrm{cstnd}}}^*(\mathcal{H}) + \mathcal{M}_{\ell^{\mathrm{cstnd}}}(\mathcal{H}) = \mathcal{T}^{\mathrm{cstnd}}(t)$ .
Here too, the theorem guarantees the tightness of the bound. This general expression of  $\mathfrak{T}^{\mathrm{cstnd}}$  can be considerably simplified under some broad assumptions, as shown by the following result.
Theorem 11 (characterization of  $\mathfrak{T}^{\mathrm{cstnd}}$ ). Assume that  $\Phi$  is convex, differentiable at zero and  $\Phi'(0) < 0$ . Then,  $\mathfrak{T}^{\mathrm{cstnd}}$  can be expressed as follows:
$$
\begin{array}{l} \mathfrak{T}^{\text{cstnd}}(t) = \left\{ \begin{array}{ll}\Phi (0) - \inf_{\mu \in \mathbb{R}}\Bigl\{\frac{1 - t}{2}\Phi (\mu) + \frac{1 + t}{2}\Phi (-\mu)\Bigr \} & n = 2\\ \inf_{\tau \geq 0}\Bigl\{\bigl(2 - \frac{1}{n - 1}\bigr)\Phi (-\tau) - \inf_{\mu \in \mathbb{R}}\Bigl\{\frac{2 - t - \frac{1}{n - 1}}{2}\Phi (-\tau +\mu) + \frac{2 + t - \frac{1}{n - 1}}{2}\Phi (-\tau -\mu)\Bigr \} \Bigr \} & n > 2 \end{array} \right. \\ \begin{array}{r l} & {\geq \left\{\Phi (0) - \inf _ {\mu \in \mathbb {R}} \Bigl \{\frac {1 - t}{2} \Phi (\mu) + \frac {1 + t}{2} \Phi (- \mu) \Bigr \} \right. n = 2} \\ & {\left. \inf _ {\tau \geq 0} \Bigl \{2 \Phi (- \tau) - \inf _ {\mu \in \mathbb {R}} \Bigl \{\frac {2 - t}{2} \Phi (- \tau + \mu) + \frac {2 + t}{2} \Phi (- \tau - \mu) \Bigr \} \right\} n > 2.} \end{array} \\ \end{array}
$$
The proof of all the results in this section are given in Appendix D.
Examples. We now compute the  $\mathcal{H}$ -estimation error transformation for constrained losses and present the results in Table 2. Here, we present the simplified  $\mathcal{I}^{\mathrm{cstnd}}$  by using the lower bound in Theorem 11. Remarkably, by applying Theorem 10, we are able to obtain tighter  $\mathcal{H}$ -consistency bounds for constrained losses with  $\Phi = \Phi_{\mathrm{hinge}}, \Phi_{\mathrm{sq - hinge}}, \Phi_{\mathrm{exp}}$  than those derived using ad hoc methods in [Awasthi et al., 2022b], and a novel  $\mathcal{H}$ -consistency bound for the new constrained loss  $\ell^{\mathrm{cstnd}}(h,x,y) = \sum_{y' \neq y} (1 + h(x,y'))^2$  with  $\Phi(t) = (1 - t)^2$ .
5.2 Extension to non-complete or bounded hypothesis sets
As in the case of comp-sum losses, we extend our results beyond the completeness assumption adopted in previous work and establish  $\overline{\mathcal{H}}$ -consistency bounds for bounded hypothesis sets. This significantly broadens the applicability of our framework.
Theorem 12 ( $\overline{\mathcal{H}}$ -consistency bound for constrained losses). Assume that  $\overline{\mathfrak{T}}^{\mathrm{cstnd}}$  is convex. Then, the following inequality holds for any hypothesis  $h \in \overline{\mathcal{H}}$  and any distribution: