SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sup _ {\mu \in \left[ s _ {\min } - \tau_ {\epsilon}, 1 - \tau_ {\epsilon} - s _ {\min } \right]} \left\{\frac {1 + t}{2} \left[ \Phi \left(\tau_ {\epsilon}\right) - \Phi \left(1 - \tau_ {\epsilon} - \mu\right) \right] + \frac {1 - t}{2} \left[ \Phi \left(1 - \tau_ {\epsilon}\right) - \Phi \left(\tau_ {\epsilon} + \mu\right) \right] \right\} <   \mathfrak {T} ^ {\operatorname {c o m p}} (t) + \epsilon .
$$
$$
\begin{array}{l} \mathfrak {T} ^ {\operatorname {c o m p}} (t) \leq \mathcal {R} _ {\ell^ {\operatorname {c o m p}}} (h) - \mathcal {R} _ {\ell^ {\operatorname {c o m p}}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell^ {\operatorname {c o m p}}} (\mathcal {H}) \\ = \mathcal {R} _ {\ell \operatorname {c o m p}} (h) - \mathbb {E} _ {X} \left[ \mathcal {C} _ {\ell \operatorname {c o m p}} ^ {*} (\mathcal {H}, x) \right] \\ = \mathcal {C} _ {\ell \text {c o m p}} (h, x) - \mathcal {C} _ {\ell \text {c o m p}} ^ {*} (\mathcal {H}, x) \\ = \sup  _ {\mu \varepsilon \left[ s _ {\min } - \tau_ {\epsilon}, 1 - \tau_ {\epsilon} - s _ {\min } \right]} \left\{\frac {1 + t}{2} \left[ \Phi \left(\tau_ {\epsilon}\right) - \Phi \left(1 - \tau_ {\epsilon} - \mu\right) \right] + \frac {1 - t}{2} \left[ \Phi \left(1 - \tau_ {\epsilon}\right) - \Phi \left(\tau_ {\epsilon} + \mu\right) \right] \right\} \\ <   \mathfrak {T} ^ {\operatorname {c o m p}} (t) + \epsilon . \\ \end{array}
$$
By letting  $\epsilon \to 0$ , we conclude the proof. The proof for  $n > 2$  directly extends from the case when  $n = 2$ . Indeed, For any  $t \in [0,1]$ , we consider the distribution that concentrates on a singleton  $\{x\}$  and satisfies  $p(x,1) = \frac{1 + t}{2}$ ,  $p(x,2) = \frac{1 - t}{2}$ ,  $p(x,y) = 0$ ,  $3 \leq y \leq n$ . For any  $\epsilon > 0$ , by the definition of infimum, we can take  $h \in \mathcal{H}$  such that  $S_h(x,1) = \tau_{1,\epsilon}$ ,  $S_h(x,2) = \tau_{2,\epsilon}$  and  $S_h(x,y) = 0$ ,  $3 \leq y \leq n$  and satisfies  $\tau_{1,\epsilon} + \tau_{2,\epsilon} = 1$ , and
$$
\begin{array}{l} \inf  _ {P \in \left[ \frac {1}{n - 1} \vee t, 1 \right]} \sup  _ {\mu \in C} \left\{\frac {P + t}{2} \left[ \Phi \left(\tau_ {2, \epsilon}\right) - \Phi \left(\tau_ {1, \epsilon} - \mu\right) \right] + \frac {P - t}{2} \left[ \Phi \left(\tau_ {1, \epsilon}\right) - \Phi \left(\tau_ {2, \epsilon} + \mu\right) \right] \right\} \\ = \sup  _ {\mu \in C} \left\{\frac {1 + t}{2} \left[ \Phi \left(\tau_ {2, \epsilon}\right) - \Phi \left(\tau_ {1, \epsilon} - \mu\right) \right] + \frac {1 - t}{2} \left[ \Phi \left(\tau_ {1, \epsilon}\right) - \Phi \left(\tau_ {2, \epsilon} + \mu\right) \right] \right\} \\ <   \mathfrak {T} ^ {\operatorname {c o m p}} (t) + \epsilon . \\ \end{array}
$$
$$
\mathfrak {T} ^ {\operatorname {c o m p}} (t) \leq \mathcal {R} _ {\ell^ {\operatorname {c o m p}}} (h) - \mathcal {R} _ {\ell^ {\operatorname {c o m p}}} ^ {*} (\mathcal {H}) + \mathcal {M} _ {\ell^ {\operatorname {c o m p}}} (\mathcal {H}) <   \mathfrak {T} ^ {\operatorname {c o m p}} (t) + \epsilon .
$$
By letting  $\epsilon \to 0$ , we conclude the proof.
E.2 Logistic loss
$$
\text{where}\ell_{\log} = -\log \Bigg(\frac{e^{h(x,y)}}{\sum_{y^{\prime}\in\mathcal{Y}}e^{h(x,y^{\prime})}}\Bigg)  and  \Psi (t) = \left\{ \begin{array}{ll}\frac{1 + t}{2}\log \bigl(1 + t\bigr) + \frac{1 - t}{2}\log \bigl(1 - t\bigr) & t\leq \frac{s_{\max} - s_{\min}}{s_{\min} + s_{\max}}\\ \frac{t}{2}\log \Big(\frac{s_{\max}}{s_{\min}}\Big) + \log \Big(\frac{2\sqrt{s_{\max}s_{\min}}}{s_{\max} + s_{\min}}\Big) & \text{otherwise.} \end{array} \right.
$$
Proof. For the multinomial logistic loss  $\ell_{\log}$ , plugging in  $\Phi(t) = -\log(t)$  in Theorem 5, gives  $\overline{\mathcal{T}}^{\mathrm{comp}}$
$$
\geq \inf_{P\in \left[\frac{1}{n - 1}\lor t,1\right]}\inf_{\substack{S_{\min}\leq \tau_{2}\leq \tau_{1}\leq S_{\max}\\ \tau_{1} + \tau_{2}\leq 1}}\sup_{\mu \in C}\Bigl\{\frac{P + t}{2} \bigl[ - \log \bigl(\tau_{2}\bigr) + \log \bigl(\tau_{1} - \mu \bigr)\bigr ] + \frac{P - t}{2}\bigl[ - \log \bigl(\tau_{1}\bigr) + \log \bigl(\tau_{2} + \mu \bigr)\bigr ]\Bigr \}
$$
where  $C = \left[\max \{s_{\min} - \tau_2, \tau_1 - s_{\max}\}, \min \{s_{\max} - \tau_2, \tau_1 - s_{\min}\}\right]$ . Here, we only compute the expression for  $n > 2$ . The expression for  $n = 2$  will lead to the same result since it can be viewed as a special case of the expression for  $n > 2$ . By differentiating with respect to  $\tau_2$  and  $P$ , we can see that the infimum is achieved when  $\tau_1 = \tau_2 = \frac{s_{\min} + s_{\max}}{2}$  and  $P = 1$  modulo some elementary analysis. Thus,  $\overline{\mathcal{T}}^{\mathrm{comp}}$  can be reformulated as
$$
\begin{array}{l} \overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} = \sup _ {\mu \in C} \left\{\frac {1 + t}{2} \bigg [ - \log \bigg (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \bigg) + \log \bigg (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} - \mu \bigg) \right] \\ \left. + \frac {1 - t}{2} \left[ - \log \left(\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2}\right) + \log \left(\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2} + \mu\right) \right] \right\} \\ = - \log \left(\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2}\right) + \sup  _ {\mu \in C} g (\mu) \\ \end{array}
$$
where  $C = \left[\frac{s_{\min} - s_{\max}}{2}, \frac{s_{\max} - s_{\min}}{2}\right]$  and  $g(\mu) = \frac{1 + t}{2} \log \left( \frac{s_{\min} + s_{\max}}{2} - \mu \right) + \frac{1 - t}{2} \log \left( \frac{s_{\min} + s_{\max}}{2} + \mu \right)$ . Since  $g$  is continuous, it attains its supremum over a compact set. Note that  $g$  is concave and differentiable. In view of that, the maximum over the open set  $(-\infty, +\infty)$  can be obtained by setting its gradient to zero. Differentiate  $g(\mu)$  to optimize, we obtain
$$
g \left(\mu^ {*}\right) = 0, \quad \mu^ {*} = - \frac {t \left(s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}\right)}{2}.
$$
Moreover, by the concavity,  $g(\mu)$  is non-increasing when  $\mu \geq \mu^{*}$ . Since  $s_{\max} - s_{\min} \geq 0$ , we have
$$
\mu^ {*} \leq 0 \leq \frac {s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}}{2}
$$
In view of the constraint  $C$ , if  $\mu^{*} \geq \frac{s_{\min} - s_{\max}}{2}$ , the maximum is achieved by  $\mu = \mu^{*}$ . Otherwise, if  $\mu^{*} < \frac{s_{\min} - s_{\max}}{2}$ , since  $g(\mu)$  is non-increasing when  $\mu \geq \mu^{*}$ , the maximum is achieved by  $\mu = \frac{s_{\min} - s_{\max}}{2}$ . Since  $\mu^{*} \geq \frac{s_{\min} - s_{\max}}{2}$  is equivalent to  $t \leq \frac{s_{\min} - s_{\min}}{s_{\min} + s_{\max}}$ , the maximum can be expressed as
$$
\max  _ {\mu \in C} g (\mu) = \left\{ \begin{array}{l l} g (\mu^ {*}) & t \leq \frac {s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}}{s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}} \\ g \Big (\frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2} \Big) & \text {o t h e r w i s e} \end{array} \right.
$$
Computing the value of  $g$  at these points yields:
$$
g \left(\mu^ {*}\right) = \frac {1 + t}{2} \log \frac {(1 + t) \left(s _ {\min } + s _ {\max }\right)}{2} + \frac {1 - t}{2} \log \frac {(1 - t) \left(s _ {\min } + s _ {\max }\right)}{2}
$$
$$
g \left(\frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2}\right) = \frac {1 + t}{2} \log \left(s _ {\operatorname* {m a x}}\right) + \frac {1 - t}{2} \log \left(s _ {\operatorname* {m i n}}\right)
$$
Then, if  $t \leq \frac{s_{\max} - s_{\min}}{s_{\min} + s_{\max}}$ , we obtain
$$
\begin{array}{l} \overline {{\mathfrak {T}}} ^ {\text {c o m p}} = - \log \left(\frac {s _ {\min } + s _ {\max }}{2}\right) + \frac {1 + t}{2} \log \frac {(1 + t) (s _ {\min } + s _ {\max })}{2} + \frac {1 - t}{2} \log \frac {(1 - t) (s _ {\min } + s _ {\max })}{2} \\ = \frac {1 + t}{2} \log (1 + t) + \frac {1 - t}{2} \log (1 - t). \\ \end{array}
$$
Otherwise, we obtain
$$
\begin{array}{l} \overline {{\mathfrak {T}}} ^ {\text {c o m p}} = - \log \left(\frac {s _ {\min } + s _ {\max }}{2}\right) + \frac {1 + t}{2} \log \left(s _ {\max }\right) + \frac {1 - t}{2} \log \left(s _ {\min }\right) \\ = \frac {t}{2} \log \left(\frac {s _ {\mathrm {m a x}}}{s _ {\mathrm {m i n}}}\right) + \log \left(\frac {2 \sqrt {s _ {\mathrm {m a x}} s _ {\mathrm {m i n}}}}{s _ {\mathrm {m a x}} + s _ {\mathrm {m i n}}}\right). \\ \end{array}
$$
Since  $\overline{\mathcal{T}}^{\mathrm{comp}}$  is convex, by Theorem 5, for any  $h\in \overline{\mathcal{H}}$  and any distribution,
$$
\Psi \big (t \big) = \left\{ \begin{array}{l l} \frac {1 + t}{2} \log \big (1 + t \big) + \frac {1 - t}{2} \log \big (1 - t \big) & t \leq \frac {s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}}{s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}} \\ \frac {t}{2} \log \Big (\frac {s _ {\operatorname* {m a x}}}{s _ {\operatorname* {m i n}}} \Big) + \log \Big (\frac {2 \sqrt {s _ {\operatorname* {m a x}} s _ {\operatorname* {m i n}}}}{s _ {\operatorname* {m a x}} + s _ {\operatorname* {m i n}}} \Big) & \text {o t h e r w i s e .} \end{array} \right.
$$
E.3 Sum exponential loss
Proof. For the sum exponential loss  $\ell_{\mathrm{exp}}$ , plugging in  $\Phi(t) = \frac{1}{t} - 1$  in Theorem 5, gives  $\overline{\mathcal{T}}^{\mathrm{comp}}$
$$
\geq \inf_{P\in \left[\frac{1}{n - 1}\vee t,1\right]} \inf_{\substack{S_{\min}\leq \tau_{2}\leq \tau_{1}\leq S_{\max}\\ \tau_{1} + \tau_{2}\leq 1}}\sup_{\mu \in C}\Bigl\{\frac{P + t}{2}\biggl\[\frac{1}{\tau_{2}} -\frac{1}{\tau_{1} - \mu}\biggr ] + \frac{P - t}{2}\biggl\[\frac{1}{\tau_{1}} -\frac{1}{\tau_{2} + \mu}\biggr \Bigr \}
$$
$$
\begin{array}{l} \overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} = \sup  _ {\mu \in C} \left\{\frac {1 + t}{2} \left[ \frac {2}{s _ {\min } + s _ {\max }} - \frac {2}{s _ {\min } + s _ {\max } - 2 \mu} \right] \right. \\ + \frac {1 - t}{2} \left[ \frac {2}{s _ {\min } + s _ {\max }} - \frac {2}{s _ {\min } + s _ {\max } + 2 \mu} \right] \Bigg \} \\ = \frac {2}{s _ {\min } + s _ {\max }} + \sup  _ {\mu \in C} g (\mu) \\ \end{array}
$$
where  $C = \left[\frac{s_{\min} - s_{\max}}{2}, \frac{s_{\max} - s_{\min}}{2}\right]$  and  $g(\mu) = -\frac{1 + t}{s_{\min} + s_{\max} - 2\mu} - \frac{1 - t}{s_{\min} + s_{\max} + 2\mu}$ . Since  $g$  is continuous, it attains its supremum over a compact set. Note that  $g$  is concave and differentiable. In view of that, the maximum over the open set  $(-\infty, +\infty)$  can be obtained by setting its gradient to zero. Differentiate  $g(\mu)$  to optimize, we obtain
$$
g \left(\mu^ {*}\right) = 0, \quad \mu^ {*} = \frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \frac {\sqrt {1 - t} - \sqrt {1 + t}}{\sqrt {1 + t} + \sqrt {1 - t}}
$$
$$
\mu^ {*} \leq 0 \leq \frac {s _ {\mathrm {m a x}} - s _ {\mathrm {m i n}}}{2}
$$
In view of the constraint  $C$ , if  $\mu^{*} \geq \frac{s_{\min} - s_{\max}}{2}$ , the maximum is achieved by  $\mu = \mu^{*}$ . Otherwise, if  $\mu^{*} < \frac{s_{\min} - s_{\max}}{2}$ , since  $g(\mu)$  is non-increasing when  $\mu \geq \mu^{*}$ , the maximum is achieved by  $\mu = \frac{s_{\min} - s_{\max}}{2}$ . Since  $\mu^{*} \geq \frac{s_{\min} - s_{\max}}{2}$  is equivalent to  $t \leq \frac{s_{\max}^2 - s_{\min}^2}{s_{\min}^2 + s_{\max}^2}$ , the maximum can be expressed as
$$
\max  _ {\mu \in C} g (\mu) = \left\{ \begin{array}{l l} g (\mu^ {*}) & t \leq \frac {s _ {\operatorname* {m a x}} ^ {2} - s _ {\operatorname* {m i n}} ^ {2}}{s _ {\operatorname* {m i n}} ^ {2} + s _ {\operatorname* {m a x}} ^ {2}} \\ g \big (\frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2} \big) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
g \left(\mu^ {*}\right) = 1 - \sqrt {1 - t ^ {2}} - \frac {2}{s _ {\min} + s _ {\max}}
$$
$$
g \left(\frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2}\right) = - \frac {1 + t}{2 s _ {\operatorname* {m a x}}} - \frac {1 - t}{2 s _ {\operatorname* {m i n}}}
$$
Then, if  $t \leq \frac{s_{\mathrm{max}}^2 - s_{\mathrm{min}}^2}{s_{\mathrm{min}}^2 + s_{\mathrm{max}}^2}$ , we obtain
$$
\begin{array}{l} \overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} = \frac {2}{s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}} + 1 - \sqrt {1 - t ^ {2}} - \frac {2}{s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}} \\ = 1 - \sqrt {1 - t ^ {2}}. \\ \end{array}
$$
$$
\begin{array}{l} \overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} = \frac {2}{s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}} - \frac {1 + t}{2 s _ {\mathrm {m a x}}} - \frac {1 - t}{2 s _ {\mathrm {m i n}}} \\ = \frac {s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}}{2 s _ {\operatorname* {m a x}} s _ {\operatorname* {m i n}}} t - \frac {\left(s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}\right) ^ {2}}{2 s _ {\operatorname* {m a x}} s _ {\operatorname* {m i n}} \left(s _ {\operatorname* {m a x}} + s _ {\operatorname* {m i n}}\right)}. \\ \end{array}
$$
Since  $\overline{\mathcal{J}}^{\mathrm{comp}}$  is convex, by Theorem 5, for any  $h\in \overline{\mathcal{H}}$  and any distribution,
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \Psi^ {- 1} \big (\mathcal {R} _ {\ell_ {\exp}} (h) - \mathcal {R} _ {\ell_ {\exp}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\exp}} (\overline {{\mathcal {H}}}) \big),
$$
$$
\Psi \left(t\right) = \left\{ \begin{array}{l l} 1 - \sqrt {1 - t ^ {2}} & t \leq \frac {s _ {\operatorname* {m a x}} ^ {2} - s _ {\operatorname* {m i n}} ^ {2}}{s _ {\operatorname* {m i n}} ^ {2} + s _ {\operatorname* {m a x}} ^ {2}} \\ \frac {s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}}{2 s _ {\operatorname* {m a x}} s _ {\operatorname* {m i n}}} t - \frac {\left(s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}\right) ^ {2}}{2 s _ {\operatorname* {m a x}} s _ {\operatorname* {m i n}} \left(s _ {\operatorname* {m a x}} + s _ {\operatorname* {m i n}}\right)} & \mathrm {o t h e r w i s e .} \end{array} \right.
$$
E.4 Generalized cross-entropy loss
Theorem 16 ( $\mathcal{H}$ -consistency bounds for generalized cross-entropy loss). For any  $h \in \overline{\mathcal{H}}$  and any distribution, we have
$$
\begin{array}{l} \mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \Psi^ {- 1} \big (\mathcal {R} _ {\ell_ {\mathrm {g c e}}} (h) - \mathcal {R} _ {\ell_ {\mathrm {g c e}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\mathrm {g c e}}} (\overline {{\mathcal {H}}}) \big), \\ \begin{array}{r c l} w h e r e \quad \Psi (t) & = & \left\{ \begin{array}{l l} \frac {1}{q} \Big (\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2} \Big) ^ {q} \Bigg [ \left(\frac {(1 + t) ^ {\frac {1}{1 - q}} + (1 - t) ^ {\frac {1}{1 - q}}}{2}\right) ^ {1 - q} - 1 \Bigg ] \\ & & \frac {t}{2 q} \Big (s _ {\operatorname* {m a x}} ^ {q} - s _ {\operatorname* {m i n}} ^ {q} \Big) + \frac {1}{q} \Big (\frac {s _ {\operatorname* {m i n}} ^ {q} + s _ {\operatorname* {m a x}} ^ {q}}{2} - \Big (\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2} \Big) ^ {q} \Big) \end{array} \right. \quad t \leq \frac {s _ {\operatorname* {m a x}} ^ {1 - q} - s _ {\operatorname* {m i n}} ^ {1 - q}}{s _ {\operatorname* {m i n}} ^ {1 - q} + s _ {\operatorname* {m a x}} ^ {1 - q}} \quad a n d \quad \ell_ {\text {g c e}} = \\ \frac {1}{q} \bigg [ 1 - \left(\frac {e ^ {h (x , y)}}{\sum_ {y ^ {\prime} \in \mathcal {Y}} e ^ {h (x , y ^ {\prime})}}\right) ^ {q} \bigg ]. \\ \end{array}
$$
Proof. For generalized cross-entropy loss  $\ell_{\mathrm{gce}}$ , plugging  $\Phi(t) = \frac{1}{q} (1 - t^q)$  in Theorem 5, gives  $\overline{\mathcal{T}}^{\mathrm{comp}}$
$$
\geq \inf_{P\in \left[\frac{1}{n - 1}\lor t,1\right]}\inf_{\substack{S_{\min}\leq \tau_{2}\leq \tau_{1}\leq S_{\max}\\ \tau_{1} + \tau_{2}\leq 1}}\sup_{\mu \in C}\Bigg\{\frac{P + t}{2}\bigg[-\frac{1}{q} (\tau_{2})^{q} + \frac{1}{q} (\tau_{1} - \mu)^{q}\bigg] + \frac{P - t}{2}\bigg[-\frac{1}{q} (\tau_{1})^{q} + \frac{1}{q} (\tau_{2} + \mu)^{q}\bigg]\Bigg\}
$$
where  $C = \left[\max \left\{s_{\min} - \tau_2, \tau_1 - s_{\max}\right\}, \min \left\{s_{\max} - \tau_2, \tau_1 - s_{\min}\right\}\right]$ . Here, we only compute the expression for  $n > 2$ . The expression for  $n = 2$  will lead to the same result since it can be viewed as a special case of the expression for  $n > 2$ . By differentiating with respect to  $\tau_2$  and  $P$ , we can see that the infimum is achieved when  $\tau_1 = \tau_2 = \frac{s_{\min} + s_{\max}}{2}$  and  $P = 1$  modulo some elementary analysis. Thus,  $\overline{\mathcal{T}}^{\mathrm{comp}}$  can be reformulated as
$$
\begin{array}{l} \overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} = \sup _ {\mu \in C} \left\{\frac {1 + t}{2 q} \bigg [ - \left(\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2}\right) ^ {q} + \left(\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} - \mu\right) ^ {q} \bigg ] \right. \\ + \frac {1 - t}{2 q} \left[ - \left(\frac {s _ {\min } + s _ {\max }}{2}\right) ^ {q} + \left(\frac {s _ {\min } + s _ {\max }}{2} + \mu\right) ^ {q} \right] \Bigg \} \\ = - \frac {1}{q} \left(\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2}\right) ^ {q} + \sup  _ {\mu \in C} g (\mu) \\ \end{array}
$$
where  $C = \left[\frac{s_{\min} - s_{\max}}{2}, \frac{s_{\max} - s_{\min}}{2}\right]$  and  $g(\mu) = \frac{1 + t}{2q}\left(\frac{s_{\min} + s_{\max}}{2} - \mu\right)^q + \frac{1 - t}{2q}\left(\frac{s_{\min} + s_{\max}}{2} + \mu\right)^q$ . Since  $g$  is continuous, it attains its supremum over a compact set. Note that  $g$  is concave and differentiable. In view of that, the maximum over the open set  $(-\infty, +\infty)$  can be obtained by setting its gradient to zero. Differentiate  $g(\mu)$  to optimize, we obtain
$$
g \left(\mu^ {*}\right) = 0, \quad \mu^ {*} = \frac {\left(1 - t\right) ^ {\frac {1}{1 - q}} - \left(1 + t\right) ^ {\frac {1}{1 - q}}}{\left(1 + t\right) ^ {\frac {1}{1 - q}} + \left(1 - t\right) ^ {\frac {1}{1 - q}}} \frac {s _ {\min } + s _ {\max }}{2}.
$$
In view of the constraint  $C$ , if  $\mu^{*} \geq \frac{s_{\min} - s_{\max}}{2}$ , the maximum is achieved by  $\mu = \mu^{*}$ . Otherwise, if  $\mu^{*} < \frac{s_{\min} - s_{\max}}{2}$ , since  $g(\mu)$  is non-increasing when  $\mu \geq \mu^{*}$ , the maximum is achieved by  $\mu = \frac{s_{\min} - s_{\max}}{2}$ . Since  $\mu^{*} \geq \frac{s_{\min} - s_{\max}}{2}$  is equivalent to  $t \leq \frac{s_{\max}^{1 - q} - s_{\min}^{1 - q}}{s_{\min}^{1 - q} + s_{\max}^{1 - q}}$ , the maximum can be expressed as
$$
\max  _ {\mu \in C} g (\mu) = \left\{ \begin{array}{l l} g (\mu^ {*}) & t \leq \frac {s _ {\operatorname* {m a x}} ^ {1 - q} - s _ {\operatorname* {m i n}} ^ {1 - q}}{s _ {\operatorname* {m i n}} ^ {1 - q} + s _ {\operatorname* {m a x}} ^ {1 - q}} \\ g \Big (\frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2} \Big) & \text {o t h e r w i s e} \end{array} \right.
$$
$$
g \left(\mu^ {*}\right) = \frac {1}{q} \left(\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2}\right) ^ {q} \left(\frac {(1 + t) ^ {\frac {1}{1 - q}} + (1 - t) ^ {\frac {1}{1 - q}}}{2}\right) ^ {1 - q}
$$
$$
g \biggl (\frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2} \biggr) = \frac {1 + t}{2 q} \bigl (s _ {\operatorname* {m a x}} \bigr) ^ {q} + \frac {1 - t}{2 q} \bigl (s _ {\operatorname* {m i n}} \bigr) ^ {q}
$$
Then, if  $t \leq \frac{s_{\max}^{1 - q} - s_{\min}^{1 - q}}{s_{\min}^{1 - q} + s_{\max}^{1 - q}}$ , we obtain
$$
\begin{array}{l} \overline {{\mathcal {T}}} ^ {\mathrm {c o m p}} = \frac {1}{q} \bigg (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \bigg) ^ {q} \bigg (\frac {(1 + t) ^ {\frac {1}{1 - q}} + (1 - t) ^ {\frac {1}{1 - q}}}{2} \bigg) ^ {1 - q} - \frac {1}{q} \bigg (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \bigg) ^ {q} \\ = \frac {1}{q} \bigg (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \bigg) ^ {q} \bigg [ \left(\frac {(1 + t) ^ {\frac {1}{1 - q}} + (1 - t) ^ {\frac {1}{1 - q}}}{2}\right) ^ {1 - q} - 1 \bigg ] \\ \end{array}
$$
$$
\begin{array}{l} \overline {{\mathfrak {T}}} ^ {\mathrm {c o m p}} = - \frac {1}{q} \bigg (\frac {s _ {\operatorname* {m i n}} + s _ {\operatorname* {m a x}}}{2} \bigg) ^ {q} + \frac {1 + t}{2 q} (s _ {\operatorname* {m a x}}) ^ {q} + \frac {1 - t}{2 q} (s _ {\operatorname* {m i n}}) ^ {q} \\ = \frac {t}{2 q} \Big (s _ {\mathrm {m a x}} ^ {q} - s _ {\mathrm {m i n}} ^ {q} \Big) + \frac {1}{q} \left(\frac {s _ {\mathrm {m i n}} ^ {q} + s _ {\mathrm {m a x}} ^ {q}}{2} - \left(\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2}\right) ^ {q}\right) \\ \end{array}
$$
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \Psi^ {- 1} \big (\mathcal {R} _ {\ell_ {\mathrm {g c e}}} (h) - \mathcal {R} _ {\ell_ {\mathrm {g c e}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\mathrm {g c e}}} (\overline {{\mathcal {H}}}) \big),
$$
$$
\begin{array}{r} \Psi (t) = \left\{ \begin{array}{l l} \frac {1}{q} \Big (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \Big) ^ {q} \Bigg [ \bigg (\frac {(1 + t) ^ {\frac {1}{1 - q}} + (1 - t) ^ {\frac {1}{1 - q}}}{2} \bigg) ^ {1 - q} - 1 \Bigg ] & t \leq \frac {s _ {\mathrm {m a x}} ^ {1 - q} - s _ {\mathrm {m i n}} ^ {1 - q}}{s _ {\mathrm {m i n}} ^ {1 - q} + s _ {\mathrm {m a x}} ^ {1 - q}} \\ \frac {t}{2 q} \Big (s _ {\mathrm {m a x}} ^ {q} - s _ {\mathrm {m i n}} ^ {q} \Big) + \frac {1}{q} \Big (\frac {s _ {\mathrm {m i n}} ^ {q} + s _ {\mathrm {m a x}} ^ {q}}{2} - \Big (\frac {s _ {\mathrm {m i n}} + s _ {\mathrm {m a x}}}{2} \Big) ^ {q} \Big) & \mathrm {o t h e r w i s e .} \end{array} \right. \end{array}
$$
E.5 Mean absolute error loss
Theorem 17 ( $\overline{\mathcal{H}}$ -consistency bounds for mean absolute error loss). For any  $h \in \overline{\mathcal{H}}$  and any distribution, we have
$$
\mathcal {R} _ {\ell_ {0 - 1}} (h) - \mathcal {R} _ {\ell_ {0 - 1}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {0 - 1}} (\overline {{\mathcal {H}}}) \leq \frac {2 \left(\mathcal {R} _ {\ell_ {\mathrm {m a e}}} (h) - \mathcal {R} _ {\ell_ {\mathrm {m a e}}} ^ {*} (\overline {{\mathcal {H}}}) + \mathcal {M} _ {\ell_ {\mathrm {m a e}}} (\overline {{\mathcal {H}}})\right)}{s _ {\mathrm {m a x}} - s _ {\mathrm {m i n}}}.
$$
Proof. For mean absolute error loss  $\ell_{\mathrm{mae}}$ , plugging  $\Phi(t) = 1 - t$  in Theorem 5, gives  $\overline{\mathcal{J}}^{\mathrm{comp}}$
$$
\geq \inf_{P\in \left[\frac{1}{n - 1}\vee t,1\right]}\inf_{\substack{S_{\min}\leq \tau_{2}\leq \tau_{1}\leq S_{\max}\\ \tau_{1} + \tau_{2}\leq 1}}\sup_{\mu \in C}\Bigg\{\frac{P + t}{2} \big[-(\tau_{2}) + (\tau_{1} - \mu)\big] + \frac{P - t}{2} \big[-(\tau_{1}) + (\tau_{2} + \mu)\big]\Bigg\}
$$
$$
\begin{array}{l} \overline {{\mathfrak {T}}} ^ {\mathrm {c o m p}} = \sup  _ {\mu \in C} \left\{\frac {1 + t}{2} \left[ - \left(\frac {s _ {\min } + s _ {\max }}{2}\right) + \left(\frac {s _ {\min } + s _ {\max }}{2} - \mu\right) \right] \right. \\ \left. + \frac {1 - t}{2} \left[ - \left(\frac {s _ {\min } + s _ {\max }}{2}\right) + \left(\frac {s _ {\min } + s _ {\max }}{2} + \mu\right) \right] \right\} \\ = \sup_{\mu \in C} - t\mu \\ \end{array}
$$
where  $C = \left[\frac{s_{\min} - s_{\max}}{2}, \frac{s_{\max} - s_{\min}}{2}\right]$ . Since  $-t\mu$  is monotonically non-increasing, the maximum over  $C$  can be achieved by
$$
\mu^ {*} = \frac {s _ {\operatorname* {m i n}} - s _ {\operatorname* {m a x}}}{2}, \quad \overline {{\mathfrak {T}}} ^ {\mathrm {c o m p}} = \frac {s _ {\operatorname* {m a x}} - s _ {\operatorname* {m i n}}}{2}   t.
$$
F Extensions of constrained losses
F.1 Proof of  $\overline{\mathcal{H}}$ -consistency bound with  $\overline{\mathcal{T}}^{\mathrm{cstnd}}$  (Theorem 12)
Theorem 12 ( $\overline{\mathcal{H}}$ -consistency bound for constrained losses). Assume that  $\overline{\mathcal{T}}^{\mathrm{cstnd}}$  is convex. Then, the following inequality holds for any hypothesis  $h \in \overline{\mathcal{H}}$  and any distribution: