SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xin Cheng<sup>1</sup> Yuzhou Cao<sup>2</sup> Haobo Wang<sup>3</sup> Hongxin Wei<sup>4</sup> Bo An<sup>2</sup> Lei Feng<sup>2*</sup>
<sup>1</sup>College of Computer Science, Chongqing University, China
$^{3}$ School of Software Technology, Zhejiang University, China
$^{4}$ Department of Statistics and Data Science, Southern University of Science and Technology, China Xincheng9215@gmail.com, yuzhou002@e.ntu.edu.sg, wanghaobo@zju.edu.cn weihx@sustech.edu.cn, boan@ntu.edu.sg, lfengqaq@gmail.com
Learning with rejection is an important framework that can refrain from making predictions to avoid critical mispredictions by balancing between prediction and rejection. Previous studies on cost-based rejection only focused on the classification setting, which cannot handle the continuous and infinite target space in the regression setting. In this paper, we investigate a novel regression problem called regression with cost-based rejection, where the model can reject to make predictions on some examples given certain rejection costs. To solve this problem, we first formulate the expected risk for this problem and then derive the Bayes optimal solution, which shows that the optimal model should reject to make predictions on the examples whose variance is larger than the rejection cost when the mean squared error is used as the evaluation metric. Furthermore, we propose to train the model by a surrogate loss function that considers rejection as binary classification and we provide conditions for the model consistency, which implies that the Bayes optimal solution can be recovered by our proposed surrogate loss. Extensive experiments demonstrate the effectiveness of our proposed method.
In machine learning, the learned model from training data is expected to make predictions on unknown test data as accurately as possible. However, it would be unreasonable for the learned model to make predictions on all the test instances, as there may exist some difficult instances that the learned model cannot give an accurate prediction. Incorrect predictions can cause severe consequences and even can be life-threatening, especially in risk-sensitive applications [5, 19, 36, 11], such as healthcare management, autonomous driving, and product inspection. Therefore, the learning with rejection (LwR) framework [10, 11, 36, 27, 31, 17, 8, 6, 38] was extensively investigated, which aims to provide a reject option to not make a prediction in order to prevent critical false predictions. In this case, the LwR model can be learned by balancing the rejection and the prediction.
So far, most of the existing studies on LwR have focused on the classification setting, i.e., classification with rejection [10, 3, 44, 45, 14, 19, 37, 17, 8, 6]. A well-known framework for classification with rejection that has been studied extensively is called the cost-based framework, i.e., classification with cost-based rejection [11, 12, 18, 39, 16, 36, 8, 6]. In the classification with cost-based rejection setting, there is a pre-determined rejection cost  $c$  for each instance, which must be smaller than the classification error 1. A typical approach for classification with cost-based rejection is the confidence-based approach [22, 3, 45, 39, 40, 8]. The main idea is to use the real-valued output of the classifier as the confidence score and decide whether to reject the prediction based on the
confidence score and the given rejection cost  $c$ . Another effective approach is classifier-rejector approach [11, 12, 36], which simultaneously trains a classifier and a rejector, and this approach achieves state-of-the-art performance in binary classification [36].
Despite many previous studies on LwR, they only focused on the classification setting, which cannot handle the continuous and infinite target space in the regression setting. In many real-world scenarios, regression tasks with continuous real-valued targets can be commonly encountered. However, even state-of-the-art regression models may make incorrect predictions, and blindly trusting the model results may lead to critical consequences, especially in risk-sensitive applications [7, 4, 28, 23, 48]. Therefore, it is necessary to consider adding a rejection option for the regression problem to not make predictions in order to avoid critical mispredictions. To this end, many studies have been conducted on regression with rejection. A widely studied framework in regression with rejection is called selective regression [26, 44, 20, 46, 27, 41] that trains a regression model with a reject option given a fixed reject rate of predictions. However, this selective regression setting fails to consider the cost-based rejection scenario where a certain cost could be incurred if the model chooses to refrain from making a prediction for a certain instance.
In this paper, we provide the first attempt to investigate a novel regression setting called regression with cost-based rejection (RcR), where the model could reject to make predictions on some instances at certain costs to avoid critical mispredictions. To solve the RcR problem, we first formulate the expected risk and then derive the Bayes optimal solution, which shows that the optimal model should reject to make predictions on the examples whose variance is larger than the rejection cost when the popular mean squared error is used as the regression loss. However, it is difficult to directly optimize the expected risk to derive the optimal solution, since the variance of the instances cannot be easily accessed. Therefore, we propose a surrogate loss function to train the model that considers the rejection behavior as a binary classification and we provide theoretical analyses to show that the Bayes optimal solution can be recovered by minimizing our surrogate loss under mild conditions. Our main contributions can be summarized as follows:
- We propose a surrogate loss function considering rejection as binary classification and we demonstrate the regressor-consistency and the rejector-calibration when the binary classification loss function is classification-calibrated and is always greater than 0.
- We also provide a relaxed condition that allows the classification-calibrated binary classification loss to be non-negative. In the relaxed condition, the regressor-consistency can still be satisfied for correctly accepted instances.
- We derive a regret transfer bound and an estimation error bound for our proposed method, and extensive experiments demonstrate the effectiveness of our method.
In this section, we introduce preliminary knowledge of ordinary regression, classification with rejection, and selective regression.
2.1 Ordinary Regression
For the ordinary regression problem, let the feature space be  $\mathcal{X} \in \mathbb{R}^d$  and the label space be  $\mathcal{Y} \in \mathbb{R}$ . Let us denote by  $(\boldsymbol{x}, y)$  an example including an instance  $x$  and a real-valued label  $y$ . Each example  $(\boldsymbol{x}, y) \in \mathcal{X} \times \mathcal{Y}$  is assumed to be independently sampled from an unknown data distribution with probability density  $p(\boldsymbol{x}, y)$ . For the regression task, we aim to learn a regression model  $h: \mathcal{X} \mapsto \mathbb{R}$  that minimizes the following expected risk:
$$
R (L) = \mathbb {E} _ {p (\boldsymbol {x}, y)} [ L (h (\boldsymbol {x}), y) ], \tag {1}
$$
where  $\mathbb{E}_{p(\boldsymbol{x},y)}$  denotes the expectation over the data distribution  $p(\boldsymbol{x},y)$  and  $L:\mathbb{R}\times \mathbb{R}\mapsto \mathbb{R}_+$  is a conventional loss function (such as mean squared error and mean absolute error) for regression, which measures how well a model estimates a given real-valued label.
2.2 Classification with Rejection
A widely studied framework in classification with rejection is the cost-based framework [10, 36, 8, 6] that aims to train a classifier  $f: \mathcal{X} \to \mathcal{Z}^{\text{®}}$  that can reject to make a prediction, where  $\text{®}$  denotes the
reject option. The evaluation metric of this task is the zero-one-c loss  $\ell_{01c}$  defined as follows:
$$
\ell_ {0 1 c} (f (\boldsymbol {x}), z) = \left\{ \begin{array}{c l} c (\boldsymbol {x}), & f (\boldsymbol {x}) = @, \\ \ell_ {0 1} (f (\boldsymbol {x}, z), & \text {o t h e r w i s e}, \end{array} \right. \tag {2}
$$
where  $c(\pmb{x})$  is the rejection cost associated with  $\pmb{x}$ . Then, the expected risk with  $\ell_{01c}$  can be represented as follows:
$$
R _ {0 1 c} (f) = \mathbb {E} _ {p (\boldsymbol {x}, y)} [ \ell_ {0 1 c} (f (\boldsymbol {x}), y) ], \tag {3}
$$
The optimal solution for classification with rejection  $f^{\star} = \operatorname*{argmin}_{f\in \mathcal{F}}R_{01c}(f)$  given by Chow's rule [10] can be expressed as follows:
Definition 1. (Chow's Rule [10]) A classifier  $f: \mathcal{X} \to \mathcal{Z}^{\text{®}}$  is the optimal solution of expected risk (3) if and only if the following conditions are almost satisfied:
$$
f (\boldsymbol {x}) = \left\{ \begin{array}{c l} \circledast , & \max  _ {z} \eta_ {z} (\boldsymbol {x}) \leq 1 - c (\boldsymbol {x}), \\ \operatorname {a r g m a x} _ {z} \eta_ {z} (\boldsymbol {x}), & \text {o t h e r w i s e}, \end{array} \right. \tag {4}
$$
where  $\eta_z(\pmb{x}) = p(z|\pmb{x})$ . Chow's rule shows that classification with rejection can be solved when  $\pmb{\eta}(\pmb{x})$  is known. However, the estimation of the probability is difficult especially when using deep neural networks [24].
2.3 Selective Regression
In selective regression, for a given instance  $\pmb{x}$ , the selective model can choose to make a prediction for it or reject to make a prediction without costs. Formally, the selective model is a pair  $(h,r)$  where  $h: \mathcal{X} \to \mathcal{R}$  is a regression prediction model and  $r: \mathcal{X} \to \mathcal{R}$  is a selection model, which serves as the rejection rule as follows,
$$
(h, r) (\boldsymbol {x}) = \left\{ \begin{array}{c l} h (\boldsymbol {x}), & r (\boldsymbol {x}) > 0, \\ \mathbb {R}, & \text {o t h e r w i s e .} \end{array} \right. \tag {5}
$$
Let us denote by  $\epsilon$  the expected rejection rate and denote by  $\phi(r) = \mathbb{E}_{p(\boldsymbol{x},y)}\mathbb{I}[r(\boldsymbol{x}) > 0]$  the coverage of selective regression. The purpose of selective regression is to derive a pair  $(h,r)$  such that the risk  $R(h,r) = \mathbb{E}_{p(\boldsymbol{x},y)}[L(h(\boldsymbol{x}),y)\mathbb{I}[r(\boldsymbol{x}) > 0]]$  is minimized with coverage  $1 - \phi(r) < \epsilon$ , where  $L(h(\boldsymbol{x}),y)$  is a conventional regression loss function. However, the selective regression setting fails to consider the cost-based rejection scenario but fixes the rejection rate  $\epsilon$ . In many real-world scenarios, rejection with costs is more common, and the cost  $c(\boldsymbol{x})$  is easier to provide compared with the rejection rate  $\epsilon$ .
3 Regression with Cost-based Rejection
Let  $\mathcal{X} \in \mathbb{R}^d$  be the  $d$ -dimensional feature space and  $\mathcal{Y} \in \mathbb{R}$  be the label space. Suppose the training set is denoted by  $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^n$ , and each training example  $(x_i, y_i) \in \mathcal{X} \times \mathcal{Y}$  is assumed to be sampled from an unknown data distribution with probability density  $p(x, y)$ . In RcR setting, for a given instance  $x$ , the learner has the option to reject making a prediction or to make a regression prediction. If the learner rejects an instance, the cost is a non-negative loss  $c(x)$ . The goal of RcR is to induce a pair  $(h, r)$  where  $h: \mathcal{X} \mapsto \mathbb{R}$  is a regressor to predict the accepted instance and  $r: \mathcal{X} \mapsto \mathbb{R}$  is a rejector to determine whether to reject an instance or not. The evaluation metric of this task is the following loss function  $\mathcal{L}(h, r, c, x, y)$ :
$$
\mathcal {L} (h, r, c, \boldsymbol {x}, y) = \left\{ \begin{array}{c l} L (h (\boldsymbol {x}), y), & r (\boldsymbol {x}) > 0, \\ c (\boldsymbol {x}), & \text {o t h e r w i s e}, \end{array} \right. \tag {6}
$$
where  $L(h(\pmb{x}), y)$  is a conventional regression loss function (e.g., mean squared error).
In what follows, we will present a Bayes optimal solution to the RcR problem and provide a surrogate loss function to train the regressor-rejector.
3.1 Bayes Optimal Solution
In this paper, we only discuss the case where the loss function  $L(h(\pmb{x}), y)$  is the mean squared error (MSE), which is the most widely used regression loss function. The expected risk of  $\mathcal{L}(h, r, c, \pmb{x}, y)$  over the data distribution can be represented as follows:
$$
R _ {\mathrm {R c R}} (h, r) = \mathbb {E} _ {p (\boldsymbol {x}, y)} [ \mathcal {L} (h, r, c, \boldsymbol {x}, y) ]. \tag {7}
$$
Let us denote by  $(h^{\star}, r^{\star}) = \operatorname*{argmin}_{(h, r)} R_{\mathrm{RcR}}(h, r)$  the optimal pair of the expected risk  $R_{\mathrm{RcR}}$  and we use  $\mathbb{E}_{p(y|\boldsymbol{x})}[y] = \int_{\mathcal{Y}} p(y|\boldsymbol{x}) y \, \mathrm{d}y$  and  $\mathbb{D}_{p(y|\boldsymbol{x})}[y] = \int_{\mathcal{Y}} p(y|\boldsymbol{x})(y - \mathbb{E}_{p(y|\boldsymbol{x})}[y])^{2} \, \mathrm{d}y$  denote the expectation and variance of  $y$  over the distribution  $p(y|\boldsymbol{x})$ . For a given cost function  $c(\boldsymbol{x})$ , we have the following theorem:
Theorem 2. For a given instance  $\pmb{x}$ , a non-negative cost  $c(\pmb{x})$  and the Bayes optimal pair  $(h^{\star}, r^{\star})$  of the risk  $R_{\mathrm{RcR}}$ , the following equality holds:
$$
\left\{ \begin{array}{l} h ^ {\star} (\boldsymbol {x}) = \mathbb {E} _ {p (y \mid \boldsymbol {x})} [ y ], \\ r ^ {\star} (\boldsymbol {x}) = \mathbb {I} \left(c (\boldsymbol {x}) - \mathbb {D} _ {p (y \mid \boldsymbol {x})} [ y ]\right). \end{array} \right. \tag {8}
$$
The proof of Theorem 2 is provided in Appendix A. It is worth noting that our derived Bayes optimal solution in Theorem 2 can be considered as a generalized version of Proposition 2.1 in Zaoui et al. [46], with an instance-dependent cost function. Theorem 2 shows the expected optimal pair  $(h^{\star}, r^{\star})$  of risk  $R_{\mathrm{RcR}}$  where the rejector  $r^{\star}$  should reject making a prediction if the variance of the distribution of labels  $y$  associated with  $x$  is so large that it exceeds a given rejection cost  $c(x)$ . This is intuitive and easy to understand. Unfortunately the probability density function  $p(y|x)$  is usually unknown, meaning that obtaining the variance  $\mathbb{D}_{p(y|x)}[y]$  and expectation  $\mathbb{E}_{p(y|x)}[y]$  is difficult or even impossible. Many previous studies adopted specific assumptions to avoid directly estimating the variance and the expectation (e.g., homoscedasticity [26, 43, 42] and heteroscedasticity [29, 30, 9, 32]), while all of them have certain constraints. Therefore, the key challenge of RcR is how to learn the optimal solution  $(h^{\star}, r^{\star})$  without the expectation and the variance.
3.2 Surrogate Loss Function of Training Regressor-Rejector
From Theorem 2, we know how the optimal pair  $(h^{\star}, r^{\star})$  makes rejection and prediction for an unknown instance, but since the expectation and the variance are difficult to obtain, we cannot directly derive the optimal regressor and rejector. Let us reconsider the RcR loss function  $\mathcal{L}(h, r, c, x, y)$  by the following equation:
$$
\mathcal {L} (h, r, c, \boldsymbol {x}, y) = (h (\boldsymbol {x}) - y) ^ {2} \mathbb {I} [ r (\boldsymbol {x}) > 0 ] + c (\boldsymbol {x}) \mathbb {I} [ r (\boldsymbol {x}) \leq 0 ], \tag {9}
$$
where  $\mathbb{I}[\cdot ]$  denotes the indicator function. We cannot directly derive a regressor  $h$  and a rejector  $r$  by the above loss since the loss function contains non-convex and discontinuous parts  $\mathbb{I}[r(\pmb {x}) > 0]$  and  $\mathbb{I}[r(\pmb {x})\leq 0]$ . In order to efficiently optimize the target loss, using surrogate loss is preferred. It is worth noting that the behavior of the rejector is similar to binary classification due to the only two options, rejection and acceptance. We may consider it directly as a binary classification  $\{+1, - 1\}$ , where  $+1$  means acceptance and  $-1$  means rejection. Then we have the following surrogate loss function:
$$
\psi (h, r, c, \boldsymbol {x}, y) = (h (\boldsymbol {x}) - y) ^ {2} \ell (r (\boldsymbol {x}), - 1) + c (\boldsymbol {x}) \ell (r (\boldsymbol {x}), + 1), \tag {10}
$$
where  $\ell(\cdot)$  is an arbitrary binary classification loss function such as hinge loss. Then the expected risk with our surrogate loss  $\psi$  can be represented as follows:
$$
R _ {\mathrm {R c R}} ^ {\psi} (h, r) = \mathbb {E} _ {p (\boldsymbol {x}, y)} [ \psi (h, r, c, \boldsymbol {x}, y) ]. \tag {11}
$$
The intuition behind this is that when the squared error is less than the given cost, we expect its weight  $\ell(r(\pmb{x}), -1)$  to be larger, i.e.,  $\ell(r(\pmb{x}), +1)$  to be smaller. However, not every binary classification loss is theoretically grounded.
In this section, we first introduce the definitions of regressor-consistency and rejector-calibration. Then, we show the condition that our method can result in the Bayes optimal solution. Furthermore, we provide a relaxed condition that the regressor-consistency is only satisfied for correctly accepted instances. Finally, We derive a regret transfer bound and an estimation error bound for our method.
4.1 Regressor-Consistency and Rejector-Calibration
The rejector-calibration we are talking about here is related to the notion of classification calibration [1, 47, 15]. The notion of classification calibration of surrogate losses is defined as the minimum requirement to ensure that a risk-minimizing classifier becomes the Bayes optimal classifier, which is a pointwise version of consistency. The definition of rejector-calibration is given below.
Definition 3. (Rejector-calibration) We say a rejector  $r: \mathcal{X} \to \mathbb{R}$  is calibrated if  $r$  always makes the same decisions as the Bayes optimal rejector  $r^{\star}$  in Theorem 2, i.e.,  $\operatorname{sign}(r(\boldsymbol{x})) = \operatorname{sign}(r^{\star}(\boldsymbol{x}))$  for all  $\boldsymbol{x} \in \mathcal{X}$  such that  $r^{\star}(\boldsymbol{x}) \neq 0$ .
The definition of rejector-calibration indicates that we do not need to obtain the exact optimal rejector due to the difficulty of obtaining the variance, therefore, we just need to ensure that our rejector makes the same decisions as the optimal rejector in Theorem 2.
On the other hand, we define the regressor-consistency as follows.
Definition 4. (Regressor-consistency) We say a regressor  $h: \mathcal{X} \to \mathbb{R}$  is consistent if  $h$  is equivalent to the Bayes optimal regressor  $h^{\star}$  in Theorem 2, i.e.,  $h = h^{\star}$ .
Then we demonstrate that our method is regressor-consistent by the following theorem:
Theorem 5. Suppose the binary loss  $\ell$  is always larger than 0. For a given non-negative cost function  $c(\pmb{x})$ , for any fixed rejector  $r$ , the optimal regressor  $h_{\psi}^{\star} = \operatorname*{argmin}_{h \in \mathcal{H}} R_{\mathrm{RcR}}^{\psi}(h, r)$  is equivalent to the Bayes optimal regressor  $h^{\star}$ .
The proof of Theorem 5 is provided in Appendix B.1. Theorem 5 shows that the regressor  $h_{\psi}^{\star}$  learned from our method can be equivalent to the Bayes optimal regressor  $h^{\star}$ .
It is worth noting that there is a special case  $\ell(r(\pmb{x}), -1) = 0$ , where the regressor actually ignores the instance  $\pmb{x}$ . Here we show a relaxed condition for regressor-consistency by the following theorem:
Theorem 6. Suppose the binary loss function  $\ell$  is classification-calibrated and is always non-negative. Given a non-negative cost function  $c(\pmb{x})$ , for any fixed rejector  $r$  and for the optimal regressor  $h_{\psi}^{\star} = \operatorname*{argmin}_{h \in \mathcal{H}} R_{\mathrm{RcR}}^{\psi}(h, r)$ , the regressor-consistency can be only satisfied for correctly accepted instances (i.e.,  $\forall \pmb{x} \in \mathcal{X}, r^{\star}(\pmb{x}) > 0$ ).
The proof of Theorem 6 is provided in Appendix B.2. Theorem 6 gives a relaxed condition of consistency, where the regressor-consistency is still satisfied for correctly accepted instances.
Then, we demonstrate that our method is rejector-calibrated by the following theorem:
Theorem 7. Suppose the binary loss  $\ell$  is classification-calibrated and is always larger than 0. For a given non-negative cost function  $c(\pmb{x})$ , the optimal rejector  $r_{\psi}^{\star} = \operatorname*{argmin}_{r \in \mathcal{R}} R_{\mathrm{RcR}}^{\psi}(h_{\psi}^{\star}, r)$  is calibrated (i.e.,  $\mathrm{sign}(r_{\psi}^{\star}(\pmb{x})) = \mathrm{sign}(r^{\star}(\pmb{x})))$ , where  $r^{\star}$  is the Bayes optimal rejector.
The proof of Theorem 7 is provided in Appendix B.3. Theorem 7 shows that our method is rejector-calibrated if the used binary loss  $\ell$  is classification-calibrated.
4.2 Regret Transfer Bound and Estimation Error Bound
In the previous section, we have given the Bayes consistency analysis of our method, i.e., if the minimizer of our proposed risk can be the optimal one in Theorem 2. However, such a result does not guarantee the performance of models that are close to but not the minimizer of the risk  $R_{\mathrm{RcR}}^{\psi}$ , which occurs commonly since we usually minimize the empirical risk in practice. We provide a theoretical guarantee for such cases by showing the following regret transfer bound:
Theorem 8. Suppose that  $\forall \pmb{x} \in \mathcal{X}, \forall y \in \mathcal{Y}, \mathbb{E}_{p(y|\pmb{x})}[(h(\pmb{x}) - y)^2] \leq M$  and  $c(\pmb{x}) \leq C$  hold almost surely:
$$
R _ {\mathrm {R c R}} (h, r) - R _ {\mathrm {R c R}} ^ {*} \leq \xi \left(R _ {\mathrm {R c R}} ^ {\psi} (h, r) - R _ {\mathrm {R c R}} ^ {\psi *}\right)),
$$
This regret transfer bound holds for widely used binary losses, e.g., when  $\ell$  is the sigmoid loss or the hinge loss,  $\xi(u) = |u|$ . When  $\ell$  is the logistic loss or the square loss,  $\xi(u) = \min\{2u, 2\sqrt{(M + C)u}\}$ .
The proof of Theorem 8 is provided in Appendix C.1. This theorem guarantees that even if the obtained  $(h,r)$  is not exactly the minimizer of  $R_{\mathrm{RcR}}^{\psi}$ , we can also expect them to achieve good performance as long as they have a low risk  $R_{\mathrm{RcR}}^{\psi}$ . Then we can further get the following estimation error bound:
Theorem 9. Suppose the binary loss is upper bounded by  $M_1 > 0$  and  $\rho$ -Lipschitz continuous,  $|h|, c(x),$  and  $|y|$  are bounded by  $M_2 > 0$ . Given the empirical risk minimizers  $\hat{h}$  and  $\hat{r}$ , the following bound holds with probability at least  $1 - \delta$ :
$$
R _ {\mathrm {R c R}} ^ {\psi} (\hat {h}, \hat {r}) - R _ {\mathrm {R c R}} ^ {\psi *} \leq 2 \sqrt {2} L _ {1} (\Re_ {n} (\mathcal {H}) + \Re_ {n} (\mathcal {R})) + C _ {1} \sqrt {\frac {2 \log (2 / \delta)}{n}},
$$
where  $C_1 = (4M_2^2 + M_2)M_1$ ,  $L_1 = \sqrt{(4M_1^2\rho + M_1\rho)^2 + 16M_1^4M_2^2}$ ,  $n$  is the i.i.d. sample size, and  $\Re_n$  is the Rademacher complexity [2].
The proof of Theorem 9 is provided in Appendix C.2. Given the fact that the Rademacher complexity usually decays at the rate of  $\mathcal{O}(1 / \sqrt{n})$ , we can finally conclude that the performance of our model can approximate its optimal performance with the increasing size of the training set. In the following sections, we will demonstrate the effectiveness of our approach through experiments.
In this section, we show the experimental results when our method is equipped with various binary classification losses and is compared with selective regression methods. In addition to the evaluation metrics commonly used for LwR, we propose additional evaluation metrics. Details of the experiment and the complete experiment can be found in Appendix D.