SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When  $h(\pmb{x}) = \mathbb{E}_{p(y|\pmb{x})}[y]$  makes  $R_{\mathrm{RcR}|\pmb{x},r(\pmb{x}) > 0} = \mathbb{D}_{p(y|\pmb{x})}[y]$  minimum. It is easy to know that  $R_{\mathrm{RcR}|\pmb{x},r(\pmb{x}) > 0} < R_{\mathrm{RcR}|\pmb{x},r(\pmb{x}) < 0}$  when  $c(\pmb{x}) - \mathbb{D}_{p(y|\pmb{x})}[y] > 0$  and  $R_{\mathrm{RcR}|\pmb{x},r(\pmb{x}) > 0} > R_{\mathrm{RcR}|\pmb{x},r(\pmb{x}) < 0}$  when  $c(\pmb{x}) - \mathbb{D}_{p(y|\pmb{x})}[y] < 0$  which means that  $R_{\mathrm{RcR}|\pmb{x}}$  is minimum when the following equation holds.
$$
r ^ {\star} (\boldsymbol {x}) = \mathbb {I} (c (\boldsymbol {x}) - \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ]).
$$
B Proofs of Consistent and Calibration
B.1 Proof of Theorem 5
First, we prove that the optimal regressor  $h^\star$  is also an optimal regressor for  $R_{\mathrm{RcR}}^{\psi}$  as follows.
$$
\begin{array}{l} R _ {\mathrm {B C R}} ^ {\psi} (h ^ {\star}, r) \\ = \mathbb {E} _ {p (\boldsymbol {x}, y)} [ \psi (h ^ {\star}, r, c, \boldsymbol {x}, y) ] \\ = \mathbb {E} _ {p (\boldsymbol {x}, y)} \left[ \left(h ^ {\star} (\boldsymbol {x}) - y\right) ^ {2} \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] \\ = \mathbb {E} _ {p (\boldsymbol {x}, y)} \left[ \left(h ^ {\star} (\boldsymbol {x}) ^ {2} - 2 y h ^ {\star} (\boldsymbol {x}) + y ^ {2}\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] \\ = \int_ {\mathcal {X}} \int_ {\mathcal {Y}} p (\boldsymbol {x}, y) \left[ \left(h ^ {\star} (\boldsymbol {x}) ^ {2} - 2 y h ^ {\star} (\boldsymbol {x}) + y ^ {2}\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] \mathrm {d} y \mathrm {d} \boldsymbol {x} \\ = \int_ {\mathcal {X}} \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) p (\boldsymbol {x}) \left[ \left(h ^ {\star} (\boldsymbol {x}) ^ {2} - 2 y h ^ {\star} (\boldsymbol {x}) + y ^ {2}\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] \mathrm {d} y \mathrm {d} \boldsymbol {x} \\ = \int_ {\mathcal {X}} p (\boldsymbol {x}) \left[ \left(h ^ {\star} (\boldsymbol {x})\right) ^ {2} - \int_ {\mathcal {Y}} 2 y h ^ {\star} (\boldsymbol {x}) p (y | \boldsymbol {x}) d y + \int_ {\mathcal {Y}} y ^ {2} p (y | \boldsymbol {x}) d y\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) ] d \boldsymbol {x} \\ = \int_ {\mathcal {X}} p (\boldsymbol {x}) \left[ \left(h ^ {\star} (\boldsymbol {x}) ^ {2} - 2 h ^ {\star} (\boldsymbol {x}) \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ] + \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ^ {2} ]\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] d \boldsymbol {x} \\ = \int_ {\mathcal {X}} p (\boldsymbol {x}) [ (h ^ {\star} (\boldsymbol {x}) ^ {2} - 2 h ^ {\star} (\boldsymbol {x}) \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ] + \mathbb {E} _ {p (y | \boldsymbol {x})} ^ {2} [ y ] + \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ]) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) ] \mathrm {d} \boldsymbol {x} \\ = \int_ {\mathcal {X}} p (\boldsymbol {x}) \left[ \left(\left(h ^ {\star} (\boldsymbol {x}) ^ {2} - \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ]\right) ^ {2} + \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ]\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] \mathrm {d} \boldsymbol {x}, \tag {12} \\ \end{array}
$$
where  $h^{\star}(\pmb{x}) = \mathbb{E}_{p(y|\pmb{x})}[y]$  makes the above risk minimal when  $\forall \pmb{x} \in \mathcal{X}$ ,  $\ell(r(\pmb{x}), z) \geq 0$ , and thus  $h^{\star}$  is an optimal regressor for  $R_{\mathrm{RcR}}^{\psi}$ , with any rejector  $r$ . On the other hand, we prove that  $h^{\star}$  is the only optimal regressor if the condition:  $\forall \pmb{x} \in \mathcal{X}$ ,  $\ell(r(\pmb{x}), z) > 0$  is achieved.
First we prove that  $h^*$  is not the only optimal regressor when the condition:  $\forall \pmb{x} \in \mathcal{X}, \ell(r(\pmb{x}), z) \geq 0$  is achieved by contradiction. Specifically, we assume that there is at least one other regressor that makes risk minimize. Suppose given an instance  $\pmb{x}_0$  and a rejector  $r'$  such that  $\ell(r'(\pmb{x}_0), -1) = 0$ , hence we have at least one other regressor  $h'$  such that  $R_{\mathrm{RcR}}^{\psi}(h', r') = R_{\mathrm{RcR}}^{\psi}(h^*, r')$  and  $h'(\pmb{x}_0) \neq h^*(\pmb{x}_0)$  due to the following equation holds:
$$
\mathbb {D} _ {p (y | \boldsymbol {x} _ {0})} [ y ] \ell \left(r ^ {\prime} (\boldsymbol {x} _ {0}), - 1\right) = 0. \tag {13}
$$
The above equation implies that there exist multiple regressors that minimize the risk when the condition:  $\forall \pmb{x} \in \mathcal{X}$ ,  $\ell(r(\pmb{x}), z) \geq 0$  is achieved. This is mainly due to the binary classification loss  $\ell(r(\pmb{x}), z) = 0$ . However, we can easily know that  $h^*$  is the only optimal regressor when the condition:  $\forall \pmb{x} \in \mathcal{X}$ ,  $\ell(r(\pmb{x}), z) > 0$  is achieved, since there are no ignored terms in Eq. (12).
B.2 Proof of Theorem 6
Let us go back to the discussion of Eq. (12):
$$
\begin{array}{l} R _ {\mathrm {R c R}} ^ {\psi} (h ^ {\star}, r) = \int_ {\mathcal {X}} [ ((h ^ {\star} (\pmb {x}) ^ {2} - \mathbb {E} _ {p (y | \pmb {x})} [ y ]) ^ {2} + \mathbb {D} _ {p (y | \pmb {x})} [ y ]) \ell (r (\pmb {x}), - 1) + c (\pmb {x}) \ell (r (\pmb {x}), + 1) ] p (\pmb {x}) \mathrm {d} \pmb {x} \\ = \int_ {\mathcal {X}} \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ] \ell (r (\boldsymbol {x}), - 1) p (\boldsymbol {x}) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) p (\boldsymbol {x}) \mathrm {d} \boldsymbol {x}. \\ \end{array}
$$
Similarly to the proof of Theorem 5, the optimal regressor  $h^{\star}$  still minimizes the risk for any rejector. However, it is easy to know that for a rejector  $r_0$  when there exists an instance  $\pmb{x}_0$  such that  $\ell(r'(x_0), -1) = 0$ , there exists at least one other regressor  $h'$  such that  $R_{\mathrm{RcR}}^{\psi}(h', r') = R_{\mathrm{RcR}}^{\psi}(h^{\star}, r')$  and  $h'(x_0) \neq h^{\star}(x_0)$  due to  $((h(\pmb{x})^2 - \mathbb{E}_{p(y|\pmb{x})}[y])^2 + \mathbb{D}_{p(y|\pmb{x})}[y])\ell(r(\pmb{x}), -1) = 0$  always holds. Therefore, the optimal regressor  $h^{\star}$  is not the only optimal solution. Fortunately, we can still show that it is regressor-consistent for some instances in this case.
For a binary classification loss function  $\ell$ , we denote by  $\mathcal{X}_{\ell}^{1}$  the space where  $\forall x \in \mathcal{X}_{\ell}^{1}$ ,  $\ell(r(x), -1) \neq 0$  for any rejector  $r$ . It is worth noting that when  $\ell(r(x), -1) = 0$ , we can infer  $r(x) < 0$ , which means that the instance  $x$  will be rejected. Then we have the following equation:
$$
\begin{array}{l} R _ {\mathrm {R c R}} ^ {\psi} (h, r) = \int_ {\mathcal {X}} \left[ \left(\left(h (\boldsymbol {x}) - \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ]\right) ^ {2} + \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ]\right) \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) \right] p (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} \\ = \int_ {\mathcal {X} _ {\ell} ^ {1}} (h (\boldsymbol {x}) - \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ]) ^ {2} \ell (r (\boldsymbol {x}), - 1) p (\boldsymbol {x}) \mathrm {d} \boldsymbol {x} \\ + \int_ {\mathcal {X} _ {\ell} ^ {1}} \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ] \ell (r (\boldsymbol {x}), - 1) p (\boldsymbol {x}) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1) p (\boldsymbol {x}) \mathrm {d} \boldsymbol {x}. \\ \end{array}
$$
When for all  $\pmb{x} \in \mathcal{X}_{\ell}^{1}$ , we have the above risk is minimised when  $h(\pmb{x}) = \mathbb{E}_{p(y|\pmb{x})}[y]$  for all  $\pmb{x} \in \mathcal{X}_{\ell}^{1}$ . In this case, we obtain a similar form as the proof of Theorem 5. Therefore, we have proved that  $h_{\psi}^{\star}$  is consistent (i.e.,  $h_{\psi}^{\star} = h^{\star}$ ) for accepted instances when the loss  $\ell$  is non-negative. The proof is completed.
B.3 Proof of Theorem 7
By fixing the optimal regressor  $h^*$ , let us discuss the pointwise version of Eq. (12):
$$
\mathbb {E} _ {p (y | \boldsymbol {x})} [ \psi (h ^ {\star}, r, c, \boldsymbol {x}, y) ] = \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ] \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1).
$$
The rejector  $r(\pmb{x})$  that minimizes the expected value is given by
$$
\min  _ {r (\boldsymbol {x})} \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ] \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1)
$$
If the binary classification loss  $\ell(r(\boldsymbol{x}), z)$  is classification-calibrated, then it is easy to see that the optimal rejector  $r_{\psi}^{\star}$  should have the same sign with  $\mathbb{D}_{p(y|\boldsymbol{x})}[y] - c(\boldsymbol{x})$  by the definition of classification calibration. When  $\mathbb{D}_{p(y|\boldsymbol{x})}[y] < c(\boldsymbol{x})$ , the optimal rejector  $r_{\psi}^{*}(\boldsymbol{x})$  must have a positive sign, and when  $\mathbb{D}_{p(y|\boldsymbol{x})}[y] \geq c(\boldsymbol{x})$ , the optimal rejector  $r_{\psi}^{*}(\boldsymbol{x})$  must have a negative sign. The proof is completed.
C Proofs of the Regret Transfer Bound and the Estimation Error Bound
C.1 Proof of Theorem 8
Proof. Without loss of generality, we discuss the regret transfer on a point  $\pmb{x}$ 's conditional risk and it can be generalized to the whole distribution using Jensen's inequality of concave functions. We denote by  $D = \mathbb{D}_{p(y|\pmb{x})}[y]$ ,  $\tilde{D} = \mathbb{E}_{p(\pmb{x},y)}[(h(\pmb{x}) - y)^2]$ ,  $\alpha = r(\pmb{x})$ , and  $c = c(\pmb{x})$ . Notice we are using margin loss  $\ell$ , thus we denote by  $\ell(\alpha, -1) = \phi(\alpha)$  and  $\ell(\alpha, -1) = \phi(\alpha)$ .
Then we can denote our conditional risk as  $R(\tilde{D}, \alpha) = \tilde{D}\phi(-\alpha) + c\phi(\alpha)$ , where  $\mathbb{E}_{p(\boldsymbol{x})}[R(\tilde{D}, \alpha)] = R_{\mathrm{RcR}}(h, r)$ . According to our previous discussion,  $R(\tilde{D}, \alpha)$  achieves its minimum at  $(D, \alpha^*)$ , where  $\alpha^* = \operatorname{argmin}_{\alpha \in \mathbb{R}} D\phi(-\alpha) + c\phi(\alpha)$ . We can learn that  $\tilde{D} \geq D$ . Then we can write our point-wise regret as:
$$
R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) = \tilde {D} \phi (- \alpha) + c \phi (\alpha) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*})
$$
Then we can begin our proof. There are three cases that the point-wise regret of our target loss is non-zero:
- When  $D < c$  but  $\alpha \leq 0$ , the point-wise regret of target loss is  $c - D$ .
In this case:
$$
\begin{array}{l} R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) = \tilde {D} \phi (- \alpha) + c \phi (\alpha) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*}) \\ \geq D \phi (- \alpha) + c \phi (\alpha) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*}) \\ \geq (D + c) \left[ \frac {D}{D + c} \phi (- \alpha) + \frac {c}{D + c} \phi (\alpha) - \frac {D}{D + c} \phi (- \alpha^ {*}) - \frac {c}{D + c} \phi (\alpha^ {*}) \right] \\ \geq (D + c) \xi^ {- 1} \left(\frac {c - D}{D + c}\right) \\ \end{array}
$$
- When  $D > c$  but  $\alpha \geq 0$ , the point-wise regret of target loss is  $D - c$ .
In this case, denote by  $\tilde{\alpha}^{*} = \mathrm{argmin}R(\tilde{D},\alpha)$ , we can learn that  $R(D,\alpha^{*})\leq R(D,\tilde{\alpha}^{*})$
$$
\begin{array}{l} R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) \geq R (\tilde {D}, \alpha) - R (D, \tilde {\alpha} ^ {*}) \\ = \tilde {D} \phi (- \alpha) + c \phi (\alpha) - D \phi \left(- \tilde {\alpha} ^ {*}\right) - c \phi \left(\tilde {\alpha} ^ {*}\right) \\ \geq \tilde {D} \phi (- \alpha) + c \phi (\alpha) - \tilde {D} \phi (- \tilde {\alpha} ^ {*}) - c \phi (\tilde {\alpha} ^ {*}) \\ \geq (\tilde {D} + c) \left[ \frac {\tilde {D}}{\tilde {D} + c} \phi (- \alpha) + \frac {c}{\tilde {D} + c} \phi (\alpha) - \frac {\tilde {D}}{\tilde {D} + c} \phi (- \tilde {\alpha} ^ {*}) - \frac {c}{\tilde {D} + c} \phi (\tilde {\alpha} ^ {*}) \right] \\ \geq (\tilde {D} + c) \xi^ {- 1} \left(\frac {\tilde {D} - c}{\tilde {D} + c}\right) \\ \end{array}
$$
- When  $D < c$  and  $\alpha \geq 0$ , the point-wise regret of target loss is  $\tilde{D} - D$ .
In this case, we should notice that  $\alpha^{*} > 0$ . We further split this case into two cases:
- When  $\tilde{D} > c$ ,  $\tilde{\alpha}^* < 0$ :
$$
\begin{array}{l} R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) = \tilde {D} \phi (- \alpha) + c \phi (\alpha) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*}) \\ = \tilde {D} \phi (- \alpha) + c \phi (\alpha) - \tilde {D} \phi (- \tilde {\alpha} ^ {*}) - c \phi (\tilde {\alpha} ^ {*}) + \tilde {D} \phi (- \tilde {\alpha} ^ {*}) + c \phi (\tilde {\alpha} ^ {*}) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*}) \\ \geq \left(\tilde {D} \phi (- \alpha) + c \phi (\alpha) - \tilde {D} \phi \left(- \tilde {\alpha} ^ {*}\right) - c \phi \left(\tilde {\alpha} ^ {*}\right)\right) + \left(D \phi \left(- \tilde {\alpha} ^ {*}\right) + c \phi \left(\tilde {\alpha} ^ {*}\right) - D \phi \left(- \alpha^ {*}\right) - c \phi \left(\alpha^ {*}\right)\right) \\ \geq (\tilde {D} + c) \xi^ {- 1} \left(\frac {\tilde {D} - c}{\tilde {D} - c}\right) + (D + c) \xi^ {- 1} \left(\frac {c - D}{D + c}\right) \\ \end{array}
$$
- When  $\tilde{D} \leq c$ , we can learn  $\tilde{\alpha}^* \geq 0$  denote by  $\alpha' = \operatorname{argmin}_{\alpha} \tilde{D}\phi(\alpha) + D\phi(-\alpha)$ , and we can learn  $\alpha' \geq 0$ :
$$
\begin{array}{l} R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) = \tilde {D} \phi (- \alpha) + c \phi (\alpha) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*}) \\ \geq \tilde {D} \phi (- \tilde {\alpha} ^ {*}) + c \phi (\tilde {\alpha} ^ {*}) - D \phi (- \alpha^ {*}) - c \phi (\alpha^ {*}) \\ \geq (\tilde {D} + c) H \left(\frac {c}{\tilde {D} + c}\right) + (D + c) H \left(\frac {c}{\tilde {D} + c}\right) \\ \end{array}
$$
Notice that for sigmoid loss and hinge loss,  $\xi^{-1}(\alpha) = |\alpha|$ ,  $H(\alpha) = 2(1 - \alpha)$ . For logistic loss,  $\xi^{-1} = \frac{1}{2}\alpha^2$  and  $H(\alpha) = -\alpha \log(\alpha) - (1 - \alpha)\log(1 - \alpha)$ . Denote by  $\gamma$  the point-wise regret of target loss. Using the linear bound of sigmoid loss and hinge loss, we can learn:
$$
R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) \geq \gamma .
$$
For logistic loss, the derivation is more complicated. In the first two cases:
$$
R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) \geq \frac {\tilde {D} + c}{2} \left(\frac {\gamma}{\tilde {D} + c}\right) ^ {2},
$$
and thus  $\sqrt{2(M + c)(R(\tilde{D},\alpha) - R(D,\alpha^{*}))} \geq \gamma$ . In the third case, we can learn:
$$
\begin{array}{l} R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) \geq \frac {\tilde {D} + c}{2} \left(\frac {\tilde {D} - c}{\tilde {D} + c}\right) ^ {2} + \frac {D + c}{2} \left(\frac {c - D}{D + c}\right) ^ {2} \\ \geq \frac {1}{2 (\tilde {D} + c)} ((\tilde {D} - c) ^ {2} + (c - D) ^ {2}) \\ \geq \frac {1}{4 (\tilde {D} + c)} (\tilde {D} - D) ^ {2} \\ \end{array}
$$
and thus  $2\sqrt{(M + c)(R(\tilde{D},\alpha) - R(D,\alpha^{*}))}\geq \gamma$  . In the last case:
$$
\begin{array}{l} R (\tilde {D}, \alpha) - R (D, \alpha^ {*}) \geq (\tilde {D} + c) H \left(\frac {c}{\tilde {D} + c}\right) + (D + c) H \left(\frac {c}{\tilde {D} + c}\right) \\ \geq - c \log \left(\frac {c}{\tilde {D} + c}\right) - \tilde {D} \log \left(\frac {\tilde {D}}{\tilde {D} + c}\right) + c \log \left(\frac {c}{D + c}\right) + D \log \left(\frac {D}{D + c}\right) \\ = c \log \left(1 + \frac {\tilde {D} - D}{D + c}\right) + \tilde {D} \log \left(1 + \frac {c}{\tilde {D}}\right) - D \log \left(1 + \frac {c}{D}\right) \\ \geq c \log \left(1 + \frac {\tilde {D} - D}{D + c}\right) \\ \geq c * \frac {\frac {\tilde {D} - D}{D + c}}{1 + \frac {\tilde {D} - D}{D + c}} \\ = \frac {c (\tilde {D} - D)}{\tilde {D} + c} \\ \geq \frac {c (\tilde {D} - D)}{c + c} \\ = \frac {\tilde {D} - D}{2}. \\ \end{array}
$$
Combining these cases, we can learn
$$
\gamma \leq \max \{2 (R (\tilde {D}, \alpha) - R (D, \alpha^ {*})), 2 \sqrt {(M + c) (R (\tilde {D} , \alpha) - R (D , \alpha^ {*}))} \}.
$$
C.2 Proof of Theorem 9
Definition 10. (Rademacher complexity) Let  $Z_{1}, \dots, Z_{n}$  be n.i.d. random variables drawn from a probability distribution  $\mu$  and  $\mathcal{F} = \{f: Z \to \mathbb{R}\}$  be a class of measurable functions. Then the expected Rademacher complexity of function class  $\mathcal{F}$  is given as follows:
$$
\mathfrak {R} _ {n} (\mathcal {F}) = \mathbb {E} _ {Z _ {1}, \dots , Z _ {n} \sim \mu} \mathbb {E} _ {\boldsymbol {\sigma}} \left[ \sup  _ {f \in \mathcal {F}} \frac {1}{n} \sum_ {i = 1} ^ {n} \sigma_ {i} f (Z _ {i}) \right], \tag {14}
$$
where  $\sigma_1, \dots, \sigma_n$  are the Rademacher variables that take the value from  $\{-1, +1\}$  uniformly.
Then we can begin proving Theorem 9.
Proof. Suppose that the binary loss is bounded by  $M_1$  and  $\rho$ -Lipschitz continuous,  $|h|, c(\pmb{x})$ ,  $|y|$  is bounded by  $M_2$  and the hypothesis space  $\mathcal{H}$  and  $\mathcal{R}$  is strong enough, then we can learn that the total loss is bounded by  $C_1 = (4M_2^2 + M_2)M_1$ , and is  $L_1$ -Lipschitz continuous w.r.t.  $(h, r)$ , where  $L_1 = \sqrt{(4M_1^2\rho + M_1\rho)^2 + 16M_1^4M_2^2}$ . By applying the McDiarmid's inequality, it is routine to show that the following inequalities hold with probability at least  $1 - \frac{\delta}{2}$ , respectively:
$$
\begin{array}{l} \sup  _ {h, r \in \mathcal {H}, \mathcal {R}} \left(R _ {\mathrm {R c R}} ^ {\psi} (h, r) - \hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r)\right) \leq \underset {\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}} {\mathbb {E}} \left[ \sup  _ {h, r \in \mathcal {H}, \mathcal {R}} \left(R _ {\mathrm {R c R}} ^ {\psi} (h, r) - \hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r)\right) \right] + C _ {1} \sqrt {\frac {\log \frac {2}{\delta}}{2 n}}, \\ \sup  _ {h, r \in \mathcal {H}, \mathcal {R}} \left(\hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r) - R _ {\mathrm {R c R}} ^ {\psi} (h, r)\right) \leq \underset {\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}} {\mathbb {E}} \left[ \sup  _ {h, r \in \mathcal {H}, \mathcal {R}} \left(\hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r) - R _ {\mathrm {R c R}} ^ {\psi} (h, r)\right) \right] + C _ {1} \sqrt {\frac {\log \frac {2}{\delta}}{2 n}}. \\ \end{array}
$$
By applying Talagrand's contraction lemma [34], we can learn that:
$$
\underset {\boldsymbol {x} _ {1}, \dots , \boldsymbol {x} _ {n}} {\mathbb {E}} \left[ \sup  _ {h, r \in \mathcal {H}, \mathcal {R}} \left(\hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r) - R _ {\mathrm {R c R}} ^ {\psi} (h, r)\right) \right] \leq \sqrt {2} L _ {1} \left(\Re_ {n} (\mathcal {H}) + \Re_ {n} (\mathcal {R})\right)
$$
and this conclusion also holds for another direction. Plugging this conclusion into the former inequalities and using the union bound, we can learn this inequality holds with probability at least  $1 - \delta$ :
$$
\sup  _ {h, r \in \mathcal {H}, \mathcal {R}} \left| \hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r) - R _ {\mathrm {R c R}} ^ {\psi} (h, r) \right| \leq \sqrt {2} L _ {1} (\mathfrak {R} _ {n} (\mathcal {H}) + \mathfrak {R} _ {n} (\mathcal {R})) + C _ {1} \sqrt {\frac {\log \frac {2}{\delta}}{2 n}}
$$
According to the definition of empirical risk minimization and identifiable condition, we can get the following conclusion:
$$
\begin{array}{l} R _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r}) - \min _ {h, r \in \mathcal {H}, \mathcal {R}} R _ {\mathrm {R c R}} ^ {\psi} (h, r) = R _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r}) - R _ {\mathrm {R c R}} ^ {\psi *} (h ^ {*}, r ^ {*}) \\ = \left(R _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r}) - \hat {R} _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r})\right) + \left(\hat {R} _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r}) - \hat {R} _ {\mathrm {R c R}} ^ {\psi} (h ^ {*}, r ^ {*})\right) + \left(\hat {R} _ {\mathrm {R c R}} ^ {\psi} (h ^ {*}, r ^ {*}) - R _ {\mathrm {R c R}} ^ {\psi *} (h ^ {*}, r ^ {*})\right) \\ \leq \left(R _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r}) - \hat {R} _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r})\right) + \left(\hat {R} _ {\mathrm {R c R}} ^ {\psi} \left(h ^ {*}, r ^ {*}\right) - R _ {\mathrm {R c R}} ^ {\psi *} \left(h ^ {*}, r ^ {*}\right)\right) \\ \leq 2 \sup _ {h, r \in \mathcal {H}, \mathcal {R}} \left| \hat {R} _ {\mathrm {R c R}} ^ {\psi} (h, r) - R _ {\mathrm {R c R}} ^ {\psi} (h, r) \right|. \\ \end{array}
$$
Combining Theorem 7, we can conclude the proof.
Table 5: Test performance (mean and std) of our surrogate loss equipped hinge loss on BreastPathQ. We repeat the sampling-and-training process 5 times. The metrics Rej, AR, RA are scaled to 0-100 and Sup, RcRLoss, AL, and RL are all magnified by a factor of 1000.
Cost	Sup	Rej	AL	RL	Rej	AR	RA
5	4.74	3.51	53.05	80.86	60.37	4.19
(0.38)	(1.96)	(20.33)	(4.17)	(6.70)	(2.36)
10	8.32	4.58	58.90	68.99	46.63	5.91
(0.21)	(1.74)	(13.15)	(4.71)	(6.45)	(2.54)
15	16.77	11.89	6.44	49.42	62.45	46.86	10.69
(0.31)	(1.82)	(8.12)	(4.76)	(5.12)	(3.84)
20	(1.22)	15.07	9.53	49.58	52.33	38.04	17.88
(0.33)	(1.15)	(8.10)	(3.94)	(3.47)	(5.36)
25	16.54	10.36	58.39	41.23	29.71	25.34
(0.78)	(2.39)	(19.85)	(8.36)	(7.36)	(12.36)
Table 6: Test performance (mean and std) of our surrogate loss equipped hinge loss on AgeDB. We repeat the sampling-and-training process 5 times. The metrics Rej, AR, RA are scaled to 0-100.
Cost	Sup	Rej	AL	RL	Rej	AR	RA
60	59.99	44.80	177.36	97.30	95.84	1.43
(0.10)	(13.97)	(40.19)	(2.16)	(3.19)	(1.17)
70	70.24	71.81	185.68	92.41	88.90	4.32
(0.50)	(4.61)	(26.75)	(1.20)	(1.75)	(0.74)
80	79.67	76.43	185.14	87.23	82.63	7.83
(1.40)	(12.86)	(18.51)	(2.08)	(2.63)	(1.88)
90	100.34	88.71	76.78	166.84	83.43	79.46	11.19
(1.08)	(8.93)	(5.90)	(11.01)	(12.07)	(9.13)
100	96.95	77.02	182.70	84.78	80.39	9.14
(0.67)	(7.46)	(13.20)	(6.77)	(7.29)	(5.49)
110	104.29	85.84	192.05	73.52	67.73	17.41
(0.31)	(6.98)	(26.75)	(10.39)	(10.33)	(9.56)
120	111.54	92.59	186.50	67.31	61.11	21.74
(2.23)	(4.79)	(13.73)	(11.60)	(11.56)	(10.43)