Title: Regression with Cost-based Rejection
Venue: NeurIPS 2023
Accepted: yes
Keywords: regression

rejection costs

surrogate loss
Abstract: Learning with rejection is an important framework that can refrain from making predictions to avoid critical mispredictions by balancing between prediction and rejection. Previous studies on cost-based rejection only focused on the classification setting, which cannot handle the continuous and infinite target space in the regression setting. In this paper, we investigate a novel regression problem called regression with cost-based rejection, where the model can reject to make predictions on some examples given certain rejection costs. To solve this problem, we first formulate the expected risk for this problem and then derive the Bayes optimal solution, which shows that the optimal model should reject to make predictions on the examples whose variance is larger than the rejection cost when the mean squared error is used as the evaluation metric. Furthermore, we propose to train the model by a surrogate loss function that considers rejection as binary classification and we provide conditions for the model consistency, which implies that the Bayes optimal solution can be recovered by our proposed surrogate loss. Extensive experiments demonstrate the effectiveness of our proposed method.

Regression with Cost-based Rejection

Abstract

Learning with rejection is an important framework that can refrain from making predictions to avoid critical mispredictions by balancing between prediction and rejection. Previous studies on cost-based rejection only focused on the classification setting, which cannot handle the continuous and infinite target space in the regression setting. In this paper, we investigate a novel regression problem called regression with cost-based rejection, where the model can reject to make predictions on some examples given certain rejection costs. To solve this problem, we first formulate the expected risk for this problem and then derive the Bayes optimal solution, which shows that the optimal model should reject to make predictions on the examples whose variance is larger than the rejection cost when the mean squared error is used as the evaluation metric. Furthermore, we propose to train the model by a surrogate loss function that considers rejection as binary classification and we provide conditions for the model consistency, which implies that the Bayes optimal solution can be recovered by our proposed surrogate loss. Extensive experiments demonstrate the effectiveness of our proposed method.

1 Introduction

In machine learning, the learned model from training data is expected to make predictions on unknown test data as accurately as possible. However, it would be unreasonable for the learned model to make predictions on all the test instances, as there may exist some difficult instances that the learned model cannot give an accurate prediction. Incorrect predictions can cause severe consequences and even can be life-threatening, especially in risk-sensitive applications [5, 19, 36, 11], such as healthcare management, autonomous driving, and product inspection. Therefore, the learning with rejection (LwR) framework [10, 11, 36, 27, 31, 17, 8, 6, 38] was extensively investigated, which aims to provide a reject option to not make a prediction in order to prevent critical false predictions. In this case, the LwR model can be learned by balancing the rejection and the prediction.

So far, most of the existing studies on LwR have focused on the classification setting, i.e., classification with rejection [10, 3, 44, 45, 14, 19, 37, 17, 8, 6]. A well-known framework for classification with rejection that has been studied extensively is called the cost-based framework, i.e., classification with cost-based rejection [11, 12, 18, 39, 16, 36, 8, 6]. In the classification with cost-based rejection setting, there is a pre-determined rejection cost c for each instance, which must be smaller than the classification error 1 . A typical approach for classification with cost-based rejection is the confidence-based approach [22, 3, 45, 39, 40, 8]. The main idea is to use the real-valued output of the classifier as the confidence score and decide whether to reject the prediction based on the

confidence score and the given rejection cost c . Another effective approach is classifier-rejector approach [11, 12, 36], which simultaneously trains a classifier and a rejector, and this approach achieves state-of-the-art performance in binary classification [36].

Despite many previous studies on LwR, they only focused on the classification setting, which cannot handle the continuous and infinite target space in the regression setting. In many real-world scenarios, regression tasks with continuous real-valued targets can be commonly encountered. However, even state-of-the-art regression models may make incorrect predictions, and blindly trusting the model results may lead to critical consequences, especially in risk-sensitive applications [7, 4, 28, 23, 48]. Therefore, it is necessary to consider adding a rejection option for the regression problem to not make predictions in order to avoid critical mispredictions. To this end, many studies have been conducted on regression with rejection . A widely studied framework in regression with rejection is called selective regression [26, 44, 20, 46, 27, 41] that trains a regression model with a reject option given a fixed reject rate of predictions. However, this selective regression setting fails to consider the cost-based rejection scenario where a certain cost could be incurred if the model chooses to refrain from making a prediction for a certain instance.

In this paper, we provide the first attempt to investigate a novel regression setting called regression with cost-based rejection (RcR), where the model could reject to make predictions on some instances at certain costs to avoid critical mispredictions. To solve the RcR problem, we first formulate the expected risk and then derive the Bayes optimal solution, which shows that the optimal model should reject to make predictions on the examples whose variance is larger than the rejection cost when the popular mean squared error is used as the regression loss. However, it is difficult to directly optimize the expected risk to derive the optimal solution, since the variance of the instances cannot be easily accessed. Therefore, we propose a surrogate loss function to train the model that considers the rejection behavior as a binary classification and we provide theoretical analyses to show that the Bayes optimal solution can be recovered by minimizing our surrogate loss under mild conditions. Our main contributions can be summarized as follows:

Wepropose a surrogate loss function considering rejection as binary classification and we demonstrate the regressor-consistency and the rejector-calibration when the binary classification loss function is classification-calibrated and is always greater than 0.

We also provide a relaxed condition that allows the classification-calibrated binary classification loss to be non-negative. In the relaxed condition, the regressor-consistency can still be satisfied for correctly accepted instances.

We derive a regret transfer bound and an estimation error bound for our proposed method, and extensive experiments demonstrate the effectiveness of our method.

2 Preliminaries

In this section, we introduce preliminary knowledge of ordinary regression, classification with rejection, and selective regression.

2.1 Ordinary Regression

For the ordinary regression problem, let the feature space be X ∈ R d and the label space be Y ∈ R . Let us denote by ( x , y ) an example including an instance x and a real-valued label y . Each example ( x , y ) ∈ X × Y is assumed to be independently sampled from an unknown data distribution with probability density p ( x , y ) . For the regression task, we aim to learn a regression model h : X ↦→ R that minimizes the following expected risk:

where E p ( x ,y ) denotes the expectation over the data distribution p ( x , y ) and L : R × R ↦→ R + is a conventional loss function (such as mean squared error and mean absolute error) for regression, which measures how well a model estimates a given real-valued label.

2.2 Classification with Rejection

Awidely studied framework in classification with rejection is the cost-based framework [10, 36, 8, 6] that aims to train a classifier f : X → Z ® that can reject to make a prediction, where ® denotes the

reject option. The evaluation metric of this task is the zero-one-c loss ℓ 01 c defined as follows:

where c ( x ) is the rejection cost associated with x . Then, the expected risk with ℓ 01 c can be represented as follows:

The optimal solution for classification with rejection f ⋆ = argmin f ∈F R 01 c ( f ) given by Chow's rule [10] can be expressed as follows:

Definition 1. (Chow's Rule [10]) A classifier f : X → Z ® is the optimal solution of expected risk (3) if and only if the following conditions are almost satisfied:

where η z ( x ) = p ( z | x ) . Chow's rule shows that classification with rejection can be solved when η ( x ) is known. However, the estimation of the probability is difficult especially when using deep neural networks [24].

2.3 Selective Regression

In selective regression, for a given instance x , the selective model can choose to make a prediction for it or reject to make a prediction without costs. Formally, the selective model is a pair ( h, r ) where h : X → R is a regression prediction model and r : X → R is a selection model, which serves as the rejection rule as follows,

Let us denote by ϵ the expected rejection rate and denote by ϕ ( r ) = E p ( x ,y ) I [ r ( x ) > 0] the coverage of selective regression. The purpose of selective regression is to derive a pair ( h, r ) such that the risk R ( h, r ) = E p ( x ,y ) [ L ( h ( x ) , y ) I [ r ( x ) > 0]] is minimized with coverage 1 -ϕ ( r ) < ϵ , where L ( h ( x ) , y ) is a conventional regression loss function. However, the selective regression setting fails to consider the cost-based rejection scenario but fixes the rejection rate ϵ . In many real-world scenarios, rejection with costs is more common, and the cost c ( x ) is easier to provide compared with the rejection rate ϵ .

3 Regression with Cost-based Rejection

Let X ∈ R d be the d -dimensional feature space and Y ∈ R be the label space. Suppose the training set is denoted by D = { ( x i , y i ) } n i =1 , and each training example ( x i , y i ) ∈ X × Y is assumed to be sampled from an unknown data distribution with probability density p ( x , y ) . In RcR setting, for a given instance x , the learner has the option ® to reject making a prediction or to make a regression prediction. If the learner rejects an instance, the cost is a non-negative loss c ( x ) . The goal of RcR is to induce a pair ( h, r ) where h : X ↦→ R is a regressor to predict the accepted instance and r : X ↦→ R is a rejector to determine whether to reject an instance or not. The evaluation metric of this task is the following loss function L ( h, r, c, x , y ) :

where L ( h ( x ) , y ) is a conventional regression loss function (e.g., mean squared error).

In what follows, we will present a Bayes optimal solution to the RcR problem and provide a surrogate loss function to train the regressor-rejector.

3.1 Bayes Optimal Solution

In this paper, we only discuss the case where the loss function L ( h ( x ) , y ) is the mean squared error (MSE), which is the most widely used regression loss function. The expected risk of L ( h, r, c, x , y ) over the data distribution can be represented as follows:

Let us denote by ( h ⋆ , r ⋆ ) = argmin ( h,r ) R RcR ( h, r ) the optimal pair of the expected risk R RcR and we use E p ( y | x ) [ y ] = ∫ Y p ( y | x ) y d y and D p ( y | x ) [ y ] = ∫ Y p ( y | x )( y -E p ( y | x ) [ y ]) 2 d y denote the expectation and variance of y over the distribution p ( y | x ) . For a given cost function c ( x ) , we have the following theorem:

Theorem 2. For a given instance x , a non-negative cost c ( x ) and the Bayes optimal pair ( h ⋆ , r ⋆ ) of the risk R RcR , the following equality holds:

The proof of Theorem 2 is provided in Appendix A. It is worth noting that our derived Bayes optimal solution in Theorem 2 can be considered as a generalized version of Proposition 2.1 in Zaoui et al. [46], with an instance-dependent cost function. Theorem 2 shows the expected optimal pair ( h ⋆ , r ⋆ ) of risk R RcR where the rejector r ⋆ should reject making a prediction if the variance of the distribution of labels y associated with x is so large that it exceeds a given rejection cost c ( x ) . This is intuitive and easy to understand. Unfortunately the probability density function p ( y | x ) is usually unknown, meaning that obtaining the variance D p ( y | x ) [ y ] and expectation E p ( y | x ) [ y ] is difficult or even impossible. Many previous studies adopted specific assumptions to avoid directly estimating the variance and the expectation (e.g., homoscedasticity [26, 43, 42] and heteroscedasticity [29, 30, 9, 32]), while all of them have certain constraints. Therefore, the key challenge of RcR is how to learn the optimal solution ( h ⋆ , r ⋆ ) without the expectation and the variance.

3.2 Surrogate Loss Function of Training Regressor-Rejector

From Theorem 2, we know how the optimal pair ( h ⋆ , r ⋆ ) makes rejection and prediction for an unknown instance, but since the expectation and the variance are difficult to obtain, we cannot directly derive the optimal regressor and rejector. Let us reconsider the RcR loss function L ( h, r, c, x , y ) by the following equation:

where I [ · ] denotes the indicator function. We cannot directly derive a regressor h and a rejector r by the above loss since the loss function contains non-convex and discontinuous parts I [ r ( x ) > 0] and I [ r ( x ) ≤ 0] . In order to efficiently optimize the target loss, using surrogate loss is preferred. It is worth noting that the behavior of the rejector is similar to binary classification due to the only two options, rejection and acceptance. We may consider it directly as a binary classification { +1 , -1 } , where +1 means acceptance and -1 means rejection. Then we have the following surrogate loss function:

where ℓ ( · ) is an arbitrary binary classification loss function such as hinge loss. Then the expected risk with our surrogate loss ψ can be represented as follows:

The intuition behind this is that when the squared error is less than the given cost, we expect its weight ℓ ( r ( x ) , -1) to be larger, i.e., ℓ ( r ( x ) , +1) to be smaller. However, not every binary classification loss is theoretically grounded.

4 Theoretical Analysis

In this section, we first introduce the definitions of regressor-consistency and rejector-calibration. Then, we show the condition that our method can result in the Bayes optimal solution. Furthermore, we provide a relaxed condition that the regressor-consistency is only satisfied for correctly accepted instances. Finally, We derive a regret transfer bound and an estimation error bound for our method.

4.1 Regressor-Consistency and Rejector-Calibration

The rejector-calibration we are talking about here is related to the notion of classification calibration [1, 47, 15]. The notion of classification calibration of surrogate losses is defined as the minimum requirement to ensure that a risk-minimizing classifier becomes the Bayes optimal classifier, which is a pointwise version of consistency. The definition of rejector-calibration is given below.

̸

Definition 3. (Rejector-calibration) We say a rejector r : X → R is calibrated if r always makes the same decisions as the Bayes optimal rejector r ⋆ in Theorem 2, i.e., sign( r ( x )) = sign( r ⋆ ( x )) for all x ∈ X such that r ⋆ ( x ) = 0 .

The definition of rejector-calibration indicates that we do not need to obtain the exact optimal rejector due to the difficulty of obtaining the variance, therefore, we just need to ensure that our rejector makes the same decisions as the optimal rejector in Theorem 2.

On the other hand, we define the regressor-consistency as follows.

Definition 4. (Regressor-consistency) We say a regressor h : X → R is consistent if h is equivalent to the Bayes optimal regressor h ⋆ in Theorem 2, i.e., h = h ⋆ .

Then we demonstrate that our method is regressor-consistent by the following theorem:

Theorem 5. Suppose the binary loss ℓ is always larger than 0. For a given non-negative cost function c ( x ) , for any fixed rejector r , the optimal regressor h ⋆ ψ = argmin h ∈H R ψ RcR ( h, r ) is equivalent to the Bayes optimal regressor h ⋆ .

The proof of Theorem 5 is provided in Appendix B.1. Theorem 5 shows that the regressor h ⋆ ψ learned from our method can be equivalent to the Bayes optimal regressor h ⋆ .

It is worth noting that there is a special case ℓ ( r ( x ) , -1) = 0 , where the regressor actually ignores the instance x . Here we show a relaxed condition for regressor-consistency by the following theorem:

Theorem 6. Suppose the binary loss function ℓ is classification-calibrated and is always nonnegative. Given a non-negative cost function c ( x ) , for any fixed rejector r and for the optimal regressor h ⋆ ψ = argmin h ∈H R ψ RcR ( h, r ) , the regressor-consistency can be only satisfied for correctly accepted instances (i.e., ∀ x ∈ X , r ⋆ ( x ) > 0 ).

The proof of Theorem 6 is provided in Appendix B.2. Theorem 6 gives a relaxed condition of consistency, where the regressor-consistency is still satisfied for correctly accepted instances.

Then, we demonstrate that our method is rejector-calibrated by the following theorem:

Theorem 7. Suppose the binary loss ℓ is classification-calibrated and is always larger than 0. For a given non-negative cost function c ( x ) , the optimal rejector r ⋆ ψ = argmin r ∈R R ψ RcR ( h ⋆ ψ , r ) is calibrated (i.e., sign( r ⋆ ψ ( x )) = sign( r ⋆ ( x )) ), where r ⋆ is the Bayes optimal rejector.

The proof of Theorem 7 is provided in Appendix B.3. Theorem 7 shows that our method is rejectorcalibrated if the used binary loss ℓ is classification-calibrated.

4.2 Regret Transfer Bound and Estimation Error Bound

In the previous section, we have given the Bayes consistency analysis of our method, i.e., if the minimizer of our proposed risk can be the optimal one in Theorem 2. However, such a result does not guarantee the performance of models that are close to but not the minimizer of the risk R ψ RcR , which occurs commonly since we usually minimize the empirical risk in practice. We provide a theoretical guarantee for such cases by showing the following regret transfer bound:

Theorem 8. Suppose that ∀ x ∈ X , ∀ y ∈ Y , E p ( y | x ) [( h ( x ) -y ) 2 ] ≤ M and c ( x ) ≤ C hold almost surely:

This regret transfer bound holds for widely used binary losses, e.g., when ℓ is the sigmoid loss or the hinge loss, ξ ( u ) = | u | . When ℓ is the logistic loss or the square loss, ξ ( u ) = min { 2 u, 2 √ ( M + C ) u } .

The proof of Theorem 8 is provided in Appendix C.1. This theorem guarantees that even if the obtained ( h, r ) is not exactly the minimizer of R ψ RcR , we can also expect them to achieve good performance as long as they have a low risk R ψ RcR . Then we can further get the following estimation error bound:

Theorem 9. Suppose the binary loss is upper bounded by M 1 > 0 and ρ -Lipschitz continuous, | h | , c ( x ) , and | y | are bounded by M 2 > 0 . Given the empirical risk minimizers ˆ h and ˆ r , the following bound holds with probability at least 1 -δ :

where C 1 = (4 M 2 2 + M 2 ) M 1 , L 1 = √ (4 M 2 1 ρ + M 1 ρ ) 2 +16 M 4 1 M 2 2 , n is the i.i.d. sample size, and R n is the Rademacher complexity [2].

The proof of Theorem 9 is provided in Appendix C.2. Given the fact that the Rademacher complexity usually decays at the rate of O (1 / √ n ) , we can finally conclude that the performance of our model can approximate its optimal performance with the increasing size of the training set. In the following sections, we will demonstrate the effectiveness of our approach through experiments.

5 Experiments

In this section, we show the experimental results when our method is equipped with various binary classification losses and is compared with selective regression methods. In addition to the evaluation metrics commonly used for LwR, we propose additional evaluation metrics. Details of the experiment and the complete experiment can be found in Appendix D.

5.1 Implementation Details

When using deep neural networks as the model and using the gradient descent optimization, we consider a possible scenario where the regressor h predicts any instance x with such a large error that ℓ ( h ( x ) , y ) >> c ( x ) . In this case the rejector r expects to reject all instances to make the empirical risk minimal. However, when the rejector r converges quickly to reject all train instances, i.e., ℓ ( r ( x ) , -1) → 0 for all train instances, the surrogate loss ψ will be constant equal to c ( x ) ℓ ( r ( x ) , +1) . At that point the gradient of the regressor h suffers from gradient vanishing. The main reason for this situation is that the regressor h has not learned the distribution of the label, but the rejector r has converged, which means that the regressor is not ready. Fortunately, we can avoid such a situation by training the rejector after the regressor is ready, and we name such a method Slow-Start. Specifically, Slow-Start prioritizes training the regressor h without training the rejector r , and then co-trains the regressor h and rejector r when the regressor h is capable of making predictions.

5.2 Datasets and Backbone Models

We conduct experiments on seven datasets, including one computer vision dataset (AgeDB [35]), one healthcare dataset (BreastPathQ [33]), and five datasets from the UCI Machine Learning Repository [13] (Abalone, Airfoil, Auto-mpg, Housing and Concrete). For each dataset, we randomly split the original dataset into training, validation, and test sets by the proportions of 60%, 20%, and 20%, respectively. It is worth noting that our approach has no restrictions on the regressor h and rejector r , so h and r can be two separate parts or share parameters.

AgeDB is a regression dataset on age prediction [21] collected by [35]. It contains 16.4K face images with a minimum age of 0 and a maximum age of 101. Age prediction is not an easy task, especially when only a single photo is available. Lighting, clothing, makeup, and facial expressions all tend to affect the intuitive age, and even friends can hardly say they can identify the age in a photo. Rejecting predictions for photos with complex environments can avoid large errors. We employ ResNet-50 [25] as our backbone network for AgeDB, and the regressor h and rejector r share parameters. We use the Adam optimizer to train our method for 100 epochs where the SlowStart is set to 40 epochs, the initial learning rate of 10 -3 and fix the batch size to 256.

BreastPathQ [33] is a healthcare dataset collected at the Sunnybrook Health Sciences Centre, Toronto. The dataset contains 2579 patch images, each patch has been assigned a tumor cellularity score score of 0 to 1 by 1 expert pathologist. Currently, this task is performed manually and relies upon expert interpretation of complex tissue structures. Moreover, cancer cellularity scoring is extremely risky and the use of automated methods could lead to irreversible disasters. Regression with rejection can improve this problem very well by predicting only the accepted samples and leaving the rejected samples back to the experts for evaluation. We use the same network as AgeDB and train 300 epochs using Adam optimizer where the Slow-Start is set to 50 epochs, the initial learning rate of 10 -3 and fix the batch size to 128.

We conducted experiments on five UCI benchmark datasets including Abalone, Airfoil, Auto-mpg, Housing and Concrete. All of these datasets can be downloaded from the UCI Machine Learning [13]. Since our proposed method do not depend on a specific model, and we train two types of base models including the linear model and the multilayer perceptron (MLP) to support the flexibility of our method on choosing a model, where the MLP model is a five-layer ( d -20-30-10-1) neural network with a ReLU activation function. For the rejector r and regressor h , we consider them as two separate parts with the same structure. For both the linear model and the MLP model, we use the Adam optimization method with the batch size set to 1024 and the number of training epochs set to 1000 where the Slow-Start is set to 200 epochs. The learning rate for all UCI benchmark datasets is selected from { 10 -1 , 10 -2 , 10 -3 } .

5.3 Evaluation Metrics

For evaluation metrics, we use the RcR loss (RcRLoss) in Eq. (6) and the rejection rate (Rej). In order to further investigate how the model work, we propose additional metrics. Accepted losses

(AL) and rejected losses (RL) denote losses on accepted instances and rejected instances, and they are defined as ∑ n i =1 I [ r ( x i ) > 0]( h ( x i ) -y i ) 2 ∑ n i =1 I [ r ( x i ) > 0] and ∑ n i =1 I [ r ( x i ) ≤ 0]( h ( x i ) -y i ) 2 ∑ n i =1 I [ r ( x i ) ≤ 0] . We also present the false rejection rate (AR) and false acceptance rate (RA) similar to false negative and false positive, which denote the rate of instances that should be accepted that are rejected and the ratio of instances that should be rejected that are accepted, and they are defined as ∑ n i =1 I [( h ( x i ) -y i ) 2 <c ( x i )] I [ r ( x i ) ≤ 0] ∑ n i =1 I [( h ( x i ) -y i ) 2 <c ( x i )] and n 2

∑ i =1 I [( h ( x i ) -y i ) ≥ c ( x i )] I [ r ( x i ) > 0] ∑ n i =1 I [( h ( x i ) -y i ) 2 ≥ c ( x i )] . It is worth noting that the optimal pair ( h ⋆ , r ⋆ ) is unknown, so AR and RA are for the current regressor and rejector. We also provide the results under supervised regression method (Sup) that directly trains the model with MSE from fully training set.

5.4 Formulation of Surrogates and Setting of Rejection Costs

In our experiments, we consider a variety of binary classification loss functions, such as mean absolute error (MAE), mean square loss, logistic loss, sigmoid and hinge loss. The rejection cost

5

10

15

20

25

Rejection cost

c

4

6

8

10

12

14

16

RcR loss

Breast

Hinge

Logistic

Square

Sigmoid

(a) RcRLoss on Breast

5

10

15

20

25

Rejection cost

c

2

4

6

8

10

12

AL loss

Breast

Hinge

Logistic

Square

Sigmoid

(b) AL on Breast

3

4

5

6

Rejection cost

c

2.4

2.6

2.8

3.0

3.2

3.4

3.6

RcR loss

Abalone

Hinge

Logistic

Square

Sigmoid

(c) RcRLoss on Abalone

3

4

5

6

Rejection cost

c

2.0

2.2

2.4

2.6

2.8

3.0

AL loss

Abalone

Hinge

Logistic

Square

Sigmoid

(d) AL on Abalone

c ( x ) is considered as a constant, which is the most commonly considered scenario in learning with rejection [6, 8, 36, 11]. For each dataset, we set various rejection costs c including extreme cases and unstressed cases depending on the supervised loss. The complete experiments are provided in Appendix D.

5.5 Experimental Performance

Table 1, Table 2 and Table 3 show the experimental results when our surrogate loss function equipped with MAE on the AgeDB, BreastPathQ, and UCI datasets, respectively. From these three tables, we have the following observations: (1) Our proposed method significantly outperforms the supervised regression method in almost all cases, which validates the effectiveness of our method by rejecting difficult test instances. (2) In most cases, the average loss of our method in accepted test instances is always smaller than the average loss of the supervised regression model in all test instances. This further indicates the ability of our method to identify hard-to-predict instances and reject them. (3) As the rejection cost c increases, we can clearly see the following trends in all datasets: RcRLoss decreases; Rejection rate decrease; Accepted loss increases. This is because as the prediction error we can accept increases, i.e., the rejection cost increases, the rejector will accept more instances, leading to a decrease in the rejection rate. However, the regressor capacity remains the same and more instances (containing difficult instances) also face more challenges, so RcRLoss and AL increase but remain smaller than Sup. (4) In setting the rejection cost c , we consider many extreme cases, e.g., the rejection cost is much smaller and much larger than the average loss in the supervised regression. In such extreme cases, our approach is still effective to identify and reject difficult test instances. In addition, we show in Figure 1 a plot of RcRLoss with the increasing rejection cost when different binary classification losses are equipped on the Breast and Abalone datasets. As the rejection cost c increases, both AL loss and RcRLoss also increase.

5.6 Comparison with Selective Regression Methods

Since our paper provides the first attempt to investigate regression with cost-based rejection, there does not exist a baseline that can be directly compared. However, the Bayes optimal solution of regression with cost-based rejection is the same as the Bayes optimal solution of selective regression, so we can compare with selective regression methods. We have conducted additional experiments to compare with SelectiveNet [20] and Knn-Plug [46]. For SelectiveNet, this work proposes a neural network architecture and an optimization goal to control the rejection threshold of the rejector to ensure a specific rejection rate. To ensure a fair comparison, we use the same base model for all datasets. For Knn-Plug, this method proposes a semi-supervised learning process for learning a data-driven predictor with a reject option based on the plug-in principle. Specifically, this method learns a regression function h and a conditional variance function σ from the labeled dataset, while the unlabeled dataset is used to calibrate the conditional variance function σ to ensure the desired rejection rate. For each dataset, we randomly split the original dataset into a labeled training set, an unlabeled training set, and a test set by the proportions of 60% (the same size of labeled training set as ours), 20%, and 20%, respectively. To ensure fairness, we replace the original base model Knn with MLP and named MLP-Plug.

Table 4 shows the results of comparison experiments. Specifically, to be able to establish a connection between our studied RcR and selective regression, we set the expected rejection rate (Rj) of the selective regression method based on the results of RcR_MAE (our surrogate loss equipped MAE). It is important to note that there is no way to perfectly match the rejection rate, as the set Rj is "expected". In addition, we show plots of the variation in AL loss for all methods with different rejection rates in Appendix D.7. As can be seen Table 4, our proposed method outperforms (smaller ALloss with the same rejection rate) almost all compared methods, which validates the effectiveness of our method.

6 Conclusion

In this paper, we investigated a novel regression problem called regression with cost-based rejection, which aims to learn a model that can reject predictions to avoid critical mispredictions at a certain rejection cost. In order to solve this problem, we first formulated the expected risk for regression with cost-based rejection and derived the Bayes optimal solution for the expected risk, which shows that we should reject instances where the variance is greater than the rejection cost. Since the variance is difficult to obtain, we proposed a surrogate loss function that considers rejection as binary classification. Further, we provided conditions for the consistency of our method, implying that the optimal solution can be recovered by our method. Finally, we derived the regret transfer bound and the estimation error bound for our method and conducted extensive experiments on various datasets to demonstrate the effectiveness of our proposed method. We expect that our first study of a simple yet theoretically grounded method for regression with cost-based rejection can inspire more interesting studies on this new problem.

References

P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of the American Statistical Association , 3:463-482, 2002.

P. L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research , 9(8), 2008.

E. Beede, E. Baylor, F. Hersch, A. Iurchenko, L. Wilcox, P. Ruamviboonsuk, and L. M. Vardoulakis. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In CHI , pages 1-12, 2020.

R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovic, et al. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 , 2018.

Y. Cao, T. Cai, L. Feng, L. Gu, J. Gu, B. An, G. Niu, and M. Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In NeurIPS , pages 521-534, 2022.

I. Chalkidis, I. Androutsopoulos, and N. Aletras. Neural legal judgment prediction in english. In ACL , 2019.

N. Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In ICML , pages 1507-1517, 2021.

K. Chaudhuri, P. Jain, and N. Natarajan. Active heteroscedastic regression. In ICML , pages 694-702, 2017.

C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory , 16(1):41-46, 1970.

C. Cortes, G. DeSalvo, and M. Mohri. Boosting with abstention. In NeurIPS , 2016.

C. Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In ALT , pages 67-82, 2016.

D. Dua and C. Graff. UCI machine learning repository, 2017.

R. El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research , 11(5), 2010.

J. Finocchiaro, R. Frongillo, and B. Waggoner. An embedding framework for consistent polyhedral surrogates. In NeurIPS , 2019.

V. Franc and D. Prusa. On discriminative learning of prediction uncertainty. In ICML , pages 1963-1971, 2019.

A. Gangrade, A. Kag, and V. Saligrama. Selective classification via one-sided prediction. In AISTATS , pages 2179-2187, 2021.

W. Gao and Z.-H. Zhou. On the consistency of auc pairwise optimization. arXiv preprint arXiv:1208.0645 , 2012.

Y. Geifman and R. El-Yaniv. Selective classification for deep neural networks. In NeurIPS , 2017.

Y. Geifman and R. El-Yaniv. Selectivenet: A deep neural network with an integrated reject option. In ICML , pages 2151-2159, 2019.

X. Geng, C. Yin, and Z.-H. Zhou. Facial age estimation by learning from label distributions. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(10):2401-2412, 2013.

Y. Grandvalet, A. Rakotomamonjy, J. Keshet, and S. Canu. Support vector machines with a reject option. In NeurIPS , 2008.

S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics , 37(3):362-386, 2020.

C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In ICML , pages 1321-1330, 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

C. M. Jarque and A. K. Bera. Efficient tests for normality, homoscedasticity and serial independence of regression residuals. Economics Letters , 6(3):255-259, 1980.

W. Jiang, Y. Zhao, and Z. Wang. Risk-controlled selective prediction for regression deep neural network models. In IJCNN , pages 1-8, 2020.

M. A. Kadampur and S. Al Riyaee. Skin cancer detection: Applying a deep learning based model driven architecture in the cloud for classifying dermal cell images. Informatics in Medicine Unlocked , 18:100282, 2020.

K. Kersting, C. Plagemann, P. Pfaff, and W. Burgard. Most likely heteroscedastic gaussian process regression. In ICML , pages 393-400, 2007.

M. Lázaro-Gredilla and M. K. Titsias. Variational heteroscedastic gaussian process regression. In ICML , pages 841-848, 2011.

J. K. Lee, Y. Bu, D. Rajan, P. Sattigeri, R. Panda, S. Das, and G. W. Wornell. Fair selective classification via sufficiency. In ICML , 2021.

H. Liu, Y.-S. Ong, and J. Cai. Large-scale heteroscedastic regression via gaussian process. IEEE Transactions on Neural Networks and Learning Systems , 32(2):708-721, 2020.

A. L. Martel, S. Nofech-Mozes, S. Salama, S. Akbar, and M. Peikari. Assessment of residual breast cancer cellularity after neoadjuvant chemotherapy using digital pathology [data], 2019.

A. Maurer. A vector-contraction inequality for rademacher complexities. In ALT , pages 3-17, 2016.

S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, and S. Zafeiriou. Agedb: The first manually collected, in-the-wild age database. In CVPRW , pages 1997-2005, 2017.

C. Ni, N. Charoenphakdee, J. Honda, and M. Sugiyama. On the calibration of multiclass classification with rejection. In NeurIPS , 2019.

T. Pietraszek. Optimizing abstaining classifiers using roc analysis. In ICML , pages 665-672, 2005.

A. Pugnana. Topics in selective classification. In AAAI , pages 16129-16130, 2023.

H. G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with a reject option. arXiv preprint arXiv:1505.04137 , 2015.

H. G. Ramaswamy, A. Tewari, and S. Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics , 12(1):530-554, 2018.

A. Shah, Y. Bu, J. K. Lee, S. Das, R. Panda, P. Sattigeri, and G. W. Wornell. Selective regression under fairness criteria. In ICML , pages 19598-19615, 2022.

B. W. Silverman. Some aspects of the spline smoothing approach to non-parametric regression curve fitting. Journal of the Royal Statistical Society: Series B (Methodological) , 47(1):1-21, 1985.

H. White. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica: Journal of the Econometric Society , pages 817-838, 1980.

Y. Wiener and R. El-Yaniv. Pointwise tracking the optimal regression function. In NeurIPS , 2012.

M. Yuan and M. Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research , 11(1), 2010.

A. Zaoui, C. Denis, and M. Hebiri. Regression with reject option and application to knn. In NeurIPS , pages 20073-20082, 2020.

T. Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004.

Y. Zoabi, S. Deri-Rozov, and N. Shomron. Machine learning-based prediction of covid-19 diagnosis based on symptoms. npj Digital Medicine , 4(1):1-5, 2021.

A Proof of Theorem 2

For an instance x , we have the following expected risk for x :

If we refuse to make a prediction for x , i.e., r ( x ) < 0 , the above expected risk transforms into the following equation:

When we make a prediction for x , i.e., r ( x ) > 0 , the above expected risk transforms into the following equation:

When h ( x ) = E p ( y | x ) [ y ] makes R RcR | x ,r ( x ) > 0 = D p ( y | x ) [ y ] minimum. It is easy to know that R RcR | x ,r ( x ) > 0 < R RcR | x ,r ( x ) < 0 when c ( x ) -D p ( y | x ) [ y ] > 0 and R RcR | x ,r ( x ) > 0 > R RcR | x ,r ( x ) < 0 when c ( x ) -D p ( y | x ) [ y ] < 0 which means that R RcR | x is minimum when the following equation holds.

The proof is completed.

B Proofs of Consistent and Calibration

B.1 Proof of Theorem 5

First, we prove that the optimal regressor h ⋆ is also an optimal regressor for R ψ RcR as follows.

where h ⋆ ( x ) = E p ( y | x ) [ y ] makes the above risk minimal when ∀ x ∈ X , ℓ ( r ( x ) , z ) ≥ 0 , and thus h ⋆ is an optimal regressor for R ψ RcR , with any rejector r . On the other hand, we prove that h ⋆ is the only optimal regressor if the condition: ∀ x ∈ X , ℓ ( r ( x ) , z ) > 0 is achieved.

̸

First we prove that h ∗ is not the only optimal regressor when the condition: ∀ x ∈ X , ℓ ( r ( x ) , z ) ≥ 0 is achieved by contradiction. Specifically, we assume that there is at least one other regressor that makes risk minimize. Suppose given an instance x 0 and a rejector r ′ such that ℓ ( r ′ ( x 0 ) , -1) = 0 , hence we have at least one other regressor h ′ such that R ψ RcR ( h ′ , r ′ ) = R ψ RcR ( h ⋆ , r ′ ) and h ′ ( x 0 ) = h ⋆ ( x 0 ) due to the following equation holds:

The above equation implies that there exist multiple regressors that minimize the risk when the condition: ∀ x ∈ X , ℓ ( r ( x ) , z ) ≥ 0 is achieved. This is mainly due to the binary classification loss ℓ ( r ( x ) , z ) = 0 . However, we can easily know that h ∗ is the only optimal regressor when the condition: ∀ x ∈ X , ℓ ( r ( x ) , z ) > 0 is achieved, since there are no ignored terms in Eq. (12).

B.2 Proof of Theorem 6

Let us go back to the discussion of Eq. (12):

̸

Similarly to the proof of Theorem 5, the optimal regressor h ⋆ still minimizes the risk for any rejector. However, it is easy to know that for a rejector r 0 when there exists an instance x 0 such that ℓ ( r ′ ( x 0 ) , -1) = 0 , there exists at least one other regressor h ′ such that R ψ RcR ( h ′ , r ′ ) = R ψ RcR ( h ⋆ , r ′ ) and h ′ ( x 0 ) = h ⋆ ( x 0 ) due to (( h ( x ) 2 -E p ( y | x ) [ y ]) 2 + D p ( y | x ) [ y ]) ℓ ( r ( x ) , -1) = 0 always holds. Therefore, the optimal regressor h ⋆ is not the only optimal solution. Fortunately, we can still show that it is regressor-consistent for some instances in this case.

̸

For a binary classification loss function ℓ , we denote by X 1 ℓ the space where ∀ x ∈ X 1 ℓ , ℓ ( r ( x ) , -1) = 0 for any rejector r . It is worth noting that when ℓ ( r ( x ) , -1) = 0 , we can infer r ( x ) < 0 , which means that the instance x will be rejected. Then we have the following equation:

When for all x ∈ X 1 ℓ , we have the above risk is minimised when h ( x ) = E p ( y | x ) [ y ] for all x ∈ X 1 ℓ . In this case, we obtain a similar form as the proof of Theorem 5. Therefore, we have proved that h ⋆ ψ is consistent (i.e., h ⋆ ψ = h ⋆ ) for accepted instances when the loss ℓ is non-negative. The proof is completed.

B.3 Proof of Theorem 7

By fixing the optimal regressor h ∗ , let us discuss the pointwise version of Eq. (12):

The rejector r ( x ) that minimizes the expected value is given by

If the binary classification loss ℓ ( r ( x ) , z ) is classification-calibrated, then it is easy to see that the optimal rejector r ⋆ ψ should have the same sign with D p ( y | x ) [ y ] -c ( x ) by the definition of classification calibration. When D p ( y | x ) [ y ] < c ( x ) , the optimal rejector r ∗ ψ ( x ) must have a positive sign, and when D p ( y | x ) [ y ] ≥ c ( x ) , the optimal rejector r ∗ ψ ( x ) must have a negative sign. The proof is completed.

C Proofs of the Regret Transfer Bound and the Estimation Error Bound

C.1 Proof of Theorem 8

Proof. Without loss of generality, we discuss the regret transfer on a point x 's conditional risk and it can be generalized to the whole distribution using Jensen's inequality of concave functions. We denote by D = D p ( y | x ) [ y ] , ˜ D = E p ( x ,y ) [( h ( x ) -y ) 2 ] , α = r ( x ) , and c = c ( x ) . Notice we are using margin loss ℓ , thus we denote by ℓ ( α, -1) = ϕ ( α ) and ℓ ( α, -1) = ϕ ( α ) .

Then we can denote our conditional risk as R ( ˜ D,α ) = ˜ Dϕ ( -α ) + cϕ ( α ) , where E p ( x ) [ R ( ˜ D,α )] = R RcR ( h, r ) . According to our previous discussion, R ( ˜ D,α ) achieves its minimum at ( D,α ∗ ) , where α ∗ = argmin α ∈ R Dϕ ( -α ) + cϕ ( α ) . We can learn that ˜ D ≥ D . Then we can write our point-wise regret as:

Then we can begin our proof. There are three cases that the point-wise regret of our target loss is non-zero:

When D < c but α ≤ 0 , the point-wise regret of target loss is c -D .

In this case:

When D > c but α ≥ 0 , the point-wise regret of target loss is D -c

In this case, denote by ˜ α = argmin R D,α ) , we can learn that R ( D,α ) ≤ R ( D, ˜ α )

. ∗ ( ˜ ∗ ∗ :

When D < c and α ≥ 0 , the point-wise regret of target loss is ˜ D -D .

In this case, we should notice that α ∗ > 0 . We further split this case into two cases:

-When ˜ D ≤ c , we can learn ˜ α ∗ ≥ 0 denote by α ′ = argmin α ˜ Dϕ ( α ) + Dϕ ( -α ) , and we can learn α ′ ≥ 0 :

Notice that for sigmoid loss and hinge loss, ξ -1 ( α ) = | α | , H ( α ) = 2(1 -α ) . For logistic loss, ξ -1 = 1 2 α 2 and H ( α ) = -α log( α ) -(1 -α ) log(1 -α ) . Denote by γ the point-wise regret of target loss. Using the linear bound of sigmoid loss and hinge loss, we can learn:

For logistic loss, the derivation is more complicated. In the first two cases:

and thus √ 2( M + c )( R ( ˜ D,α ) -R ( D,α ∗ )) ≥ γ . In the third case, we can learn:

Combining these cases, we can learn

C.2 Proof of Theorem 9

Definition 10. (Rademacher complexity) Let Z 1 , · · · , Z n be n i.i.d. random variables drawn from a probability distribution µ and F = { f : Z → R } be a class of measurable functions. Then the expected Rademacher complexity of function class F is given as follows:

where σ 1 , · · · , σ n are the Rademacher variables that take the value from {-1 , +1 } uniformly.

Then we can begin proving Theorem 9.

Proof. Suppose that the binary loss is bounded by M 1 and ρ -Lipschitz continuous, | h | , c ( x ) , | y | is bounded by M 2 and the hypothesis space H and R is strong enough, then we can learn that the total loss is bounded by C 1 = (4 M 2 2 + M 2 ) M 1 , and is L 1 -Lipschitz continuous w.r.t. ( h, r ) , where L 1 = √ (4 M 2 1 ρ + M 1 ρ ) 2 +16 M 4 1 M 2 2 . By applying the McDiarmid's inequality, it is routine to show that the following inequalities hold with probability at least 1 -δ 2 , respectively:

By applying Talagrand's contraction lemma [34], we can learn that:

and this conclusion also holds for another direction. Plugging this conclusion into the former inequalities and using the union bound, we can learn this inequality holds with probability at least 1 -δ :

According to the definition of empirical risk minimization and identifiable condition, we can get the following conclusion:

Combining Theorem 7, we can conclude the proof.

D Additional Information of Experiments

D.1 Evaluation Metrics

We describe in detail all the evaluation metrics we used in our experiments.

RcR loss. The RcR loss (RcRLoss) is the main evaluation metric for RcR. For a given example ( x , y ) and the rejection cost c ( x ) , the RcRLoss defined as if r ( x ) > 0 , L ( h, r, c, x , y ) = ( h ( x ) -y ) 2 , otherwise L ( h, r, c, x , y ) = c ( x ) .

Rejection rate. The rejection rate (Rej) is defined as ∑ n i =1 I [ r ( x ) ≤ 0] n . Rej indicates the rejection rate of our model on the test dataset.

Accepted loss. The accepted loss (AL) is defined as ∑ n i =1 I [ r ( x i ) > 0]( h ( x i ) -y i ) 2 ∑ n i =1 I [ r ( x i ) > 0] . AL denotes the average loss of our regressor on the accepted test dataset.

Rejected loss. The rejected loss (RL) is defined as ∑ n i =1 I [ r ( x i ) ≤ 0]( h ( x i ) -y i ) 2 ∑ n i =1 I [ r ( x i ) ≤ 0] . RL denotes the average loss of our regressor on the rejected test dataset.

False rejection rate. The false rejection rate (AR) is defined as ∑ n i =1 I [( h ( x i ) -y i ) 2 <c ( x i )] I [ r ( x i ) ≤ 0] ∑ n i =1 I [( h ( x i ) -y i ) 2 <c ( x i )] . AR denotes the rate of instances that should be accepted that are rejected.

False acceptance rate. The false acceptance rate (RA) denotes the rate of instances that should be rejected that are accepted, and is defined as ∑ n i =1 I [( h ( x i ) -y i ) 2 ≥ c ( x i )] I [ r ( x i ) > 0] ∑ n i =1 I [( h ( x i ) -y i ) 2 ≥ c ( x i )] .

D.2 Some Results for Hinge Loss

In this section, we show some experimental results of the surrogate loss equipped with hinge loss, which can be formulated as follows:

Table 5 and Table 6 show some of the experimental results on the AgeDB adn BreastPathQ when surrogate loss equipped hinge loss, respectively. From this table, we can see that RcRLoss and AL is always lower than Sup in almost all experiments, which means that our method is effective in identifying test instances should be accepted and test instances should be rejected. It is worth noting that in most experiments, there is a low RA, which means that there is a higher tendency to reject hard-to-predict test instances to avoid serious errors when equipped hinge loss.

D.3 Some Results for Logistic Loss

In this section, we show some experimental results of the surrogate loss equipped with logistic loss, which can be formulated as follows:

Table 7, Table 8 and Table 9 show some of the experimental results on the UCI datasets and BreastPathQ when surrogate loss equipped logistic loss, respectively. Our proposed method significantly outperforms the supervised regression method in almost all cases, which verifies the ability of our

method to reject difficult test instances demonstrating the effectiveness of our method. In most cases, the average loss of our method in the accepted test instances (AL) is always smaller than the average loss of the supervised regression model (Sup) in all test instances. This further indicates the ability of our method to identify hard-to-predict samples and reject them. On both MLP and Linear models, our method is effective in avoiding serious errors, which verifies that our method can be adapted to different models.

D.4 Some Results for Square Loss

In this section, we show some experimental results of the surrogate loss equipped with square loss, which can be formulated as follows:

Table 10 and Table 11 show some of the experimental results on the BreastPathQ and UCI datasets with MLP model when surrogate loss equipped square loss, respectively. When the rejection cost c is small, both RcRLoss and AL are significantly smaller than Sup. When the rejection cost c is large, RcRLoss and AL are close to Sup but always smaller, which shows the effectiveness of our method to deal with regression with cost-based rejection.

D.5 Some Results for Sigmoid

In this section, we show some experimental results of the Sigmoid function equipped with sigmoid, which can be formulated as follows:

Unlike other binary classification losses, sigmoid can be viewed as weight balancing prediction loss and the rejection cost due to sigmoid( r ( x )) + sigmoid( -r ( x )) = 1 . Table 12 show some of the experimental results on BreastPathQ when surrogate loss equipped sigmoid, respectively. RcRLoss and AL are always smaller than Sup, verifying the effectiveness of our method.

In our experiments, we used multiple binary classification losses (MAE, hinge loss, logistic loss, square loss and sigmoid) and different datasets including two deep datasets (BreastPathQ and AgeDB) and five uci datasets (Abalone, Airfoil, Auto-mpg, Housing and Concrete), and our method outperformed supervised regression in most cases, which demonstrates the effective of our method.

D.6 Performance of Increasing Training Data

As we showed in Theorem 8 and Theorem 9, the pair ( h, r ) learned by our surrogate loss could converge to the optimal pair ( h ∗ , r ∗ ) when the number of training examples approaches to infinity. Therefore, the performance of the model can be improved as the training samples increase. To empirically validate such a theoretical finding, we further conduct experiments on the Abalone and Auto-mpg datasets by changing the fraction of training examples (100% means that we use all training examples in the training set). As shown in Figure 2, the RcRLoss and the AL generally decreases when more training examples are used for model training. This observation is clearly in accordance with our theoretical analyses in Theorem 8 and Theorem 9, because the learned model would be closer to the optimal model as more training examples are provided.

D.7 Curves of AL and Rej

We show the AL loss for all methods with different rejection rates on Abalone, Autompg and Concrete datasets in Figure 3. In the plot of the relationship between AL loss and rejection rate, a curve at the bottom means better results.

20%

40%

60%

80%

100%

Training set size

3.0

3.5

4.0

4.5

5.0

Test loss

Breast

RcRLoss

AL

(a) Breast with

c

= 5

20%

40%

60%

80%

100%

Training set size

6

7

8

9

10

Test loss

Breast

RcRLoss

AL

(b) Breast with

c

= 10

20%

40%

60%

80%

100%

Training set size

2.4

2.6

2.8

3.0

3.2

Test loss

Abalone

RcRLoss

AL

(c) Abalone with

c

= 4

20%

40%

60%

80%

100%

Training set size

3.0

3.2

3.4

3.6

3.8

4.0

Test loss

Abalone

RcRLoss

AL

(d) Abalone with

c

= 6

20%

40%

60%

80%

100%

Training set size

8

10

12

14

16

18

20

Test loss

Housing

RcRLoss

AL

(e) Housing with

c

= 12

20%

40%

60%

80%

100%

Training set size

8

10

12

14

16

Test loss

Housing

RcRLoss

AL

(f) Housing with

c

= 16

20

25

30

35

40

45

Rejection rate (100%)

2.0

2.2

2.4

2.6

2.8

3.0

AL loss

abalone

MLP-Plug

Selective

Hinge

Logistic

Square

Sigmoid

(a) Abalone

10

20

30

40

50

60

70

Rejection rate (100%)

2

3

4

5

6

7

8

AL loss

Auto-mpg

MLP-Plug

Selective

Hinge

Logistic

Square

Sigmoid

(b) Autompg

0

10 20 30 40 50 60 70

Rejection rate (100%)

15

20

25

30

35

AL loss

Concrete

MLP-Plug

Selective

Hinge

Logistic

Square

Sigmoid

(c) Concrete

E Limitations

In Theorem 4 and Theorem 5, we show that there is a limitation in our proposed method that requires the binary classification loss ℓ ( r ( x ) , z ) to be always greater than 0. This is easily satisfied by the design of the binary classification loss. However, to avoid the modification of the binary classification loss, we further propose Theorem 6, which only requires the binary classification loss to be non-negative, and this is easily satisfied. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method. on the other hand, we use the pointwise cost functions c ( x ) in our theoretical analysis, which shows that our method can be used for pointwise cost functions. For simplicity, in our experiments, we only consider the rejection cost as a constant c , but our method can easily handle the various pointwise cost functions in different application scenarios.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLearning with rejection is an important machine learning problem. Most of the existing papers focus on the classification setting, i.e., classification with rejection and selective classification, and seldom works are targeting at the regression setting. This paper aims to investigate regression with cost-based rejection. Although some papers have studied the selective regression problem, I consider that the problem of regression with cost-based rejection is new. \n\nTo solve this new problem, this paper gives a formulation of the expected risk and derives the Bayes optimal solution. To train an ideal model, this paper also proposes a surrogate loss function that regards rejection as binary classification and provides conditions for the consistency. Experiments are conducted to demonstrate the effectiveness of the proposed.\n\nSTRENGTHS:\n- The problem of regression with cost-based rejection is interesting and new.\n\n- It is quite important to give the formulation of expected risk for cost-based rejection and the Bayes optimal solution is meaningful and significant, which might serve as a pioneer for follow-up works to check whether the derived model and rejector are consistent when the mean squared error is used as the evaluation metric.\n\n- A reasonable approach to training a good regression model with rejection is proposed and theoretical analyses are provided.\n\n- Experimental results are significant, which supports the importance of considering cost-based rejection in the regression setting.\n\nWEAKNESSES:\nIn Theorem 4, I notice that the authors did not introduce the concept \"classification calibrated binary classification loss\". For this concept, I also think that some references are required, e.g., [1] and [2].\n\n[1] P. Bartlett et al. Convexity, classification, and risk bounds. JASA 2006.\n[2] A. Tewari and P. Bartlett. On the consistency of multi-class classification methods. JMLR 2007.\n\n- I also notice that the first two cited references are repeated. I would suggest that the authors should further check the details of the references.\n\n- I also find some typos in this paper, e.g., missing a right parenthesis in Eq. (2).\n\n- It will be interesting to give a general Bayes optimal solution for arbitrary regression losses, instead of limiting to the mean squared error.\n\nQUESTIONS:\nCompared with the selective setting, what is the key challenge of the proposed setting regression with cost-based rejection? The authors are encouraged to further explain this point.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses problem of regression with the reject option. The authors propose the cost-based formulation of an optimal reject-option regression rule and they derive (Bayes) optimal strategy for the case the distribution is known. The authors further propose a surrogate loss to learn the reject-option regression rule from examples. They prove the the consistency and regret bounds for the proposed learning approach.\n\nSTRENGTHS:\nThe paper is sound and it is very clearly written.\n\nThe proposed method based on surrogate loss is simple and potentially effective. \n\nThe authors derive theoretical guarantees for the proposed estimator, namely, they show the consistency and the regret bound.\n\nWEAKNESSES:\nThe first contribution, i.e. formulation of the cost-based reject option regression and the optimal solution, is a known result. E.g. it is given in [41], see equation (1) of the paper. Besides [41], deriving the optimal strategy for a generic reject option predictor (of which the regression with L2-loss is a special case) is straightforward and appears in pattern recognition textbooks, e.g. Schlesinger et al. Ten Lectures on Statistical and Structural Pattern Recognition. Springer 2002.\n\nThe proposed method, based on minimizing the surrogate loss, is not compared against any baseline solution nor any existing methods for learning reject option regression. As a result, when there is no reference, it is difficult to judge about efficiency of the proposed method. The minimal solution would be to use synthetic data with known ground-truth. On real data one could use any regression model which outputs estimate p(y|x), like e.g. Bayesian methods, and plugin Bayes rule.  The author may argue that most existing methods formulate the optimal reject-option regression using the concept of selective risk and coverage [41][20][38]. However, all methods (including the proposed approach) can be compared in terms of the selective risk and the coverage which are reported by the authors anyway in the experiments (section 5) although the authors use different terminology. Namely, the selective risk is denoted as the \"accepted loss\" AL and the coverage equals 1 - rejection rate (RR). Note the cost-based formulation and the selective risk vs. coverage formulation (known as the bounded-improvement or bounded-abstention rejection models) are equivalent in the sense that both lead to the same Bayes-optimal solution, i.e. setting the rejection cost (as in the paper under review) has the same effect as setting threshold on the coverage (or the selective risk), see e.g. Franc et al. Optimal strategies for reject option classifiers. JMLR 2023. \n\nMinor problems:\n\n- Regarding the experiments in sec 5, errors observed on AgeDB dataset are excessively large. The mean error ~100, reported for the standard regression model (sup), makes no sense for age prediction regardless whether the authors report MAE or L2-loss which is not clear from the description.\n\n- The observations derived from the experiments (section 5.5) are questionable or trivial: \"(1) Our proposed method significantly outperforms the supervised reression method\" It is not clear in what sense the propsed method is better as it solves a different problem than the non-reject model. \"(2) In most cases, the average loss of our method in the accepted test instances (AL) is always smaller than the average loss of the supervised regression model\"; note that this holds true for any rejection rule regardless how good the rejector $r(x)$ is. Similarly the obsevation (3) is obvious and it hold for any rejection rule.\n\n- Line 273: \"...RcR loss (RcRLoss) decreases\" -> increases\n\nQUESTIONS:\nPlease explain reasons for not using any baseline method in the experimental evaluation?\n\n---\nThe authors satisfactorily addressed my questions in the rebuttal based on which I increased my ratings.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of regression with rejection, specifically the approach of specifing a cost function and learn the pair of regressor and rejector at the same time. The paper prensents a concrete path to solving the problem. It first properly defines the problem and shows the Bayes optimal solution to it. Since the Bayes optimal solution requires knowing the expectation and the variance of the underlying distribution, it then proposes a learnable risk defined using surrogate loss function. Then, the paper theoretically investigate and show the usefullness of the proposed approach, from the perspective of classification calibration and error bound. Finally it empirically evaluates the proposed method on several typical datsets using various metrics.\n\nSTRENGTHS:\nOriginality:\nThis paper tackles the regression with rejection problem which is of significant importance in the field. The paper solves the problem from a novel perspective and can be seen as a novel combination of several well-known techniques. This paper addresses clearly how it is related and different from related publications.\n\nQuality:\nThe paper is technically sound and self-contained. Its claims are properly supported by theoretical demonstrations.\n\nClarity:\nThe paper is clearly written and easy to follow. The structual is well organized.\n\nSignificance:\nThe proposed method has significance to some extent, as it considers a new approach to an important problem.\n\nWEAKNESSES:\n- Empricail comparison is no sufficiently conducted.\n  - There is no comparison with existing methods. \n  - There is no investigation on varying cost.\n  - There is no investigation on slow-start. This would show the robustness of the proposed method, since slow-start introduces a hyper-parameter to the method.\n  - There is no investigation on varying training data size. Some theoretical results shows how performance would change on different $n$. It would be pursuative to show it empirically to some extent.\n\nQUESTIONS:\n- For the cost funciton $c(x)$, it is used as a function on $x$ instead of a constant in theoretical demonstrations but considered as a constant in experiments. Does theoretical results has some relationship or limitation on the form of the pointwise cost function? Does some results rely on cost being a non-constant function?\n\n- Are there any detailed discussion on the slow-start mechanism, since is part is not covered by any theory but has crucial practical importance? For example, how the slow-start epochs affect the overall performance? Can we completely stop the learning of $h$ after the slow-start epochs?\n\nLIMITATIONS:\nTechinal limitations on loss function are addressed by authors in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
The paper explores the framework of regression with rejection, in which the model can opt to refrain from making predictions on certain instances at specific costs, with the intention of avoiding critical mispredictions. The paper determines the Bayes optimal solution and introduces a theoretically grounded surrogate loss within the framework.

STRENGTHS:
1. The paper is pioneering in studying the regression with rejection setting.
2. The presentation of the paper is clear and concise.

WEAKNESSES:
1. While the regression with rejection setting represents a fresh concept in the literature, the technical approach seems to closely mirror the standard classification with rejection setting. This resemblance potentially limits the novelty of the paper. Could the authors elaborate on the specific technical challenges encountered within this setting?

2. A definition of regressor-consistency that parallels rejector-calibration (Definition 3) is missing. 

3. The use of a supervised regression method may not serve as an appropriate and fair baseline for rejection experiments. It would be more convincing to conduct experiments comparing some straightforward rejection methods against the proposed rejection methods.

4. Additional commentary on the experimental results is required. For instance, the setup considers a range of binary classification loss functions; which among these yields the best results based on the experiments conducted?

QUESTIONS:
See Weakness.

LIMITATIONS:
N/A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07