SlowGuess/NeurIPS_2023_OCR huggingface dataset:
When using deep neural networks as the model and using the gradient descent optimization, we consider a possible scenario where the regressor  $h$  predicts any instance  $\pmb{x}$  with such a large error that  $\ell(h(\pmb{x}), y) >> c(\pmb{x})$ . In this case the rejector  $r$  expects to reject all instances to make the empirical risk minimal. However, when the rejector  $r$  converges quickly to reject all train instances, i.e.,  $\ell(r(\pmb{x}), -1) \to 0$  for all train instances, the surrogate loss  $\psi$  will be constant equal to  $c(\pmb{x})\ell(r(\pmb{x}), +1)$ . At that point the gradient of the regressor  $h$  suffers from gradient vanishing. The main reason for this situation is that the regressor  $h$  has not learned the distribution of the label, but the rejector  $r$  has converged, which means that the regressor is not ready. Fortunately, we can avoid such a situation by training the rejector after the regressor is ready, and we name such a method Slow-Start. Specifically, Slow-Start prioritizes training the regressor  $h$  without training the rejector  $r$ , and then co-trains the regressor  $h$  and rejector  $r$  when the regressor  $h$  is capable of making predictions.
5.2 Datasets and Backbone Models
We conduct experiments on seven datasets, including one computer vision dataset (AgeDB [35]), one healthcare dataset (BreastPathQ [33]), and five datasets from the UCI Machine Learning Repository [13] (Abalone, Airfoil, Auto-mpg, Housing and Concrete). For each dataset, we randomly split the original dataset into training, validation, and test sets by the proportions of  $60\%$ ,  $20\%$ , and  $20\%$  respectively. It is worth noting that our approach has no restrictions on the regressor  $h$  and rejector  $r$ , so  $h$  and  $r$  can be two separate parts or share parameters.
AgeDB is a regression dataset on age prediction [21] collected by [35]. It contains 16.4K face images with a minimum age of 0 and a maximum age of 101. Age prediction is not an easy task, especially when only a single photo is available. Lighting, clothing, makeup, and facial expressions all tend to affect the intuitive age, and even friends can hardly say they can identify the age in a photo. Rejecting predictions for photos with complex environments can avoid large errors. We employ ResNet-50 [25] as our backbone network for AgeDB, and the regressor  $h$  and rejector  $r$  share parameters. We use the Adam optimizer to train our method for 100 epochs where the SlowStart is set to 40 epochs, the initial learning rate of  $10^{-3}$  and fix the batch size to 256.
Table 1: Test performance (mean and std) of our surrogate loss equipped MAE on BreastPathQ. We repeat the sampling-and-training process 5 times. The metrics Rej, AR, RA are scaled to 0-100 and Sup, RcRLoss, AL and RL are all magnified by a factor of 1000.
Cost	Sup	RcRLoss	AL	RL	Rej	AR	RA
5	4.37	2.70	31.51	72.53	52.61	6.53
(0.17)	(1.07)	(2.29)	(4.44)	(5.10)	(2.66)
10	8.22	5.50	37.14	60.08	43.14	11.01
(0.70)	(1.98)	(4.43)	(4.34)	(4.49)	(4.22)
15	16.77	11.11	6.84	40.39	53.49	38.39	15.46
(1.22)	(0.55)	(1.43)	(1.67)	(3.39)	(2.86)	(3.97)
20	13.84	9.53	43.41	40.65	29.98	29.02
(0.62)	(1.69)	(5.34)	(7.28)	(6.58)	(9.81)
25	16.01	12.91	46.62	24.47	17.46	48.97
(1.32)	(2.48)	(9.43)	(4.26)	(4.96)	(8.00)
Table 2: Test performance (mean and std) of our surrogate loss equipped MAE on AgeDB. We repeat the sampling-and-training process 5 times. The metrics Rej, AR and RA are scaled to 0-100.
Cost	Sup	RcRLoss	AL	RL	Rej	AR	RA
60	59.80	54.25	156.81	95.40	93.13	2.51
(0.31)	(4.41)	(23.21)	(2.88)	(4.30)	(1.56)
70	69.00	61.56	151.04	86.22	81.41	8.12
(0.39)	(4.10)	(12.05)	(2.94)	(3.07)	(2.49)
80	77.10	67.32	150.52	76.00	70.63	16.11
(1.72)	(2.21)	(12.36)	(15.71)	(16.36)	(13.20)
90	100.34	85.36	73.07	162.44	73.38	67.33	17.20
(2.23)	(3.21)	(12.45)	(11.50)	(12.07)	(9.08)
100	92.94	82.89	170.04	58.35	52.15	30.56
(3.02)	(7.47)	(20.53)	(12.51)	(11.59)	(12.48)
110	95.08	79.62	166.07	52.15	46.13	34.38
(5.62)	(5.44)	(13.75)	(14.96)	(14.76)	(13.40)
120	96.80	82.44	173.14	37.11	32.54	51.31
(7.45)	(2.40)	(12.58)	(22.64)	(21.42)	(23.96)
BreastPathQ [33] is a healthcare dataset collected at the Sunnybrook Health Sciences Centre, Toronto. The dataset contains 2579 patch images, each patch has been assigned a tumor cellularity score score of 0 to 1 by 1 expert pathologist. Currently, this task is performed manually and relies upon expert interpretation of complex tissue structures. Moreover, cancer cellularity scoring is extremely risky and the use of automated methods could lead to irreversible disasters. Regression with rejection can improve this problem very well by predicting only the accepted samples and leaving the rejected samples back to the experts for evaluation. We use the same network as AgeDB and train 300 epochs using Adam optimizer where the Slow-Start is set to 50 epochs, the initial learning rate of  $10^{-3}$  and fix the batch size to 128.
We conducted experiments on five UCI benchmark datasets including Abalone, Airfoil, Auto-mpg, Housing and Concrete. All of these datasets can be downloaded from the UCI Machine Learning [13]. Since our proposed method do not depend on a specific model, and we train two types of base models including the linear model and the multilayer perceptron (MLP) to support the flexibility of our method on choosing a model, where the MLP model is a five-layer  $(d - 20 - 30 - 10 - 1)$  neural network with a ReLU activation function. For the rejector  $r$  and regressor  $h$ , we consider them as two separate parts with the same structure. For both the linear model and the MLP model, we use the Adam optimization method with the batch size set to 1024 and the number of training epochs set to 1000 where the Slow-Start is set to 200 epochs. The learning rate for all UCI benchmark datasets is selected from  $\{10^{-1}, 10^{-2}, 10^{-3}\}$ .
5.3 Evaluation Metrics
For evaluation metrics, we use the RcR loss (RcRLoss) in Eq. (6) and the rejection rate (Rej). In order to further investigate how the model work, we propose additional metrics. Accepted losses
Table 3: Test performance (mean and std) of our surrogate loss equipped MAE on five UCI datasets trained with the MLP model. We repeat the sampling-and-training process 10 times. The metrics Rej, AR, and RA are scaled to 0-100.
Datasets	Cost	Sup	RcRLoss	AL	RL	Rej	AR	RA
Abalone	3	2.41(0.12)	1.99(0.21)	8.13(1.08)	42.04(3.18)	32.82(3.44)	33.33(3.22)
4	4.44(0.46)	2.88(0.13)	2.30(0.21)	11.37(1.70)	33.70(2.47)	25.56(2.81)	39.27(3.71)
5	3.22(0.23)	2.66(0.35)	10.30(1.25)	23.43(2.94)	16.83(2.41)	48.98(5.90)
6	3.53(0.25)	2.93(0.35)	12.13(1.69)	19.32(3.47)	13.81(3.33)	53.20(5.67)
Airfoil	9	7.20(0.35)	4.23(0.86)	37.80(2.95)	62.23(3.73)	41.49(5.73)	11.60(3.29)
12	8.11(0.36)	5.39(0.86)	51.51(10.51)	40.33(7.95)	23.37(7.20)	25.88(9.04)
16	12.96(2.60)	9.15(0.43)	6.84(0.70)	72.80(20.79)	24.92(5.67)	11.92(6.93)	38.17(5.02)
20	11.32(0.75)	8.83(1.47)	58.28(8.87)	21.53(7.71)	13.70(5.34)	48.66(18.08)
25	11.47(1.54)	9.24(1.35)	74.38(16.07)	14.19(5.11)	8.08(3.60)	52.11(12.32)
30	11.68(3.07)	11.17(3.20)	96.55(16.60)	2.52(3.81)	1.38(1.78)	86.35(20.06)
Auto-mpg	4	3.64(0.29)	2.99(0.83)	13.98(4.16)	56.92(13.00)	46.80(15.49)	28.74(10.51)
6	4.83(0.93)	3.83(1.70)	18.04(5.95)	37.31(14.10)	29.01(12.74)	42.42(19.54)
8	8.34(2.16)	6.75(1.93)	6.14(2.41)	25.59(12.48)	22.95(19.88)	19.26(18.27)	64.99(23.95)
10	7.14(1.64)	6.11(2.24)	23.29(9.54)	24.07(6.58)	17.15(5.12)	48.47(15.98)
13	8.13(2.41)	7.42(2.83)	35.49(23.74)	12.56(6.83)	10.38(6.14)	71.52(14.52)
Housing	9	8.80(0.34)	6.25(3.22)	40.28(17.30)	84.46(11.67)	77.60(15.88)	9.72(5.91)
12	12.57(3.43)	9.52(0.75)	7.40(1.48)	58.94(25.98)	44.65(8.69)	33.30(8.99)	31.25(8.64)
16	10.12(1.84)	8.35(1.58)	88.14(44.53)	22.38(8.90)	14.21(6.81)	51.84(14.41)
20	10.50(3.32)	9.59(3.50)	184.24(109.35)	8.51(6.82)	5.81(5.32)	73.40(13.11)
Concrete	20	18.03(1.32)	13.17(4.91)	82.17(14.58)	69.42(6.92)	54.06(9.37)	12.34(4.47)
30	24.20(1.85)	19.29(3.85)	112.13(30.32)	44.08(8.81)	27.43(8.55)	26.80(7.90)
40	34.44(3.05)	28.63(2.56)	23.12(4.59)	136.51(46.59)	31.50(8.98)	18.32(7.30)	39.49(12.07)
50	32.48(2.76)	27.90(4.31)	168.19(41.73)	19.76(7.54)	10.54(4.51)	53.82(13.74)
60	34.33(3.50)	30.33(4.89)	197.26(49.03)	12.82(6.62)	5.67(3.21)	60.95(14.99)
(AL) and rejected losses (RL) denote losses on accepted instances and rejected instances, and they are defined as  $\frac{\sum_{i=1}^{n}\mathbb{I}[r(\boldsymbol{x}_i)>0](h(\boldsymbol{x}_i)-y_i)^2}{\sum_{i=1}^{n}\mathbb{I}[r(\boldsymbol{x}_i)>0]}$  and  $\frac{\sum_{i=1}^{n}\mathbb{I}[r(\boldsymbol{x}_i)\leq0](h(\boldsymbol{x}_i)-y_i)^2}{\sum_{i=1}^{n}\mathbb{I}[r(\boldsymbol{x}_i)\leq0]}$ . We also present the false rejection rate (AR) and false acceptance rate (RA) similar to false negative and false positive, which denote the rate of instances that should be accepted that are rejected and the ratio of instances that should be rejected that are accepted, and they are defined as  $\frac{\sum_{i=1}^{n}\mathbb{I}[(h(\boldsymbol{x}_i)-y_i)^2 < c(\boldsymbol{x}_i)]\mathbb{I}[r(\boldsymbol{x}_i)\leq0]}{\sum_{i=1}^{n}\mathbb{I}[(h(\boldsymbol{x}_i)-y_i)^2 < c(\boldsymbol{x}_i)]}$  and  $\frac{\sum_{i=1}^{n}\mathbb{I}[(h(\boldsymbol{x}_i)-y_i)^2 \geq c(\boldsymbol{x}_i)]\mathbb{I}[r(\boldsymbol{x}_i)>0]}{\sum_{i=1}^{n}\mathbb{I}[(h(\boldsymbol{x}_i)-y_i)^2 \geq c(\boldsymbol{x}_i)]}$ . It is worth noting that the optimal pair  $(h^{\star}, r^{\star})$  is unknown, so AR and RA are for the current regressor and rejector. We also provide the results under supervised regression method (Sup) that directly trains the model with MSE from fully training set.
5.4 Formulation of Surrogates and Setting of Rejection Costs
In our experiments, we consider a variety of binary classification loss functions, such as mean absolute error (MAE), mean square loss, logistic loss, sigmoid and hinge loss. The rejection cost
(a) RcRLoss on Breast
(b) AL on Breast
Figure 1: Figures (a) and (b) report the RcRLoss and AL with different rejection cost when different binary classification losses are equipped to the Breast dataset, respectively. Figures (c) and (d) report the RcRLoss and AL with different rejection costs when different binary classification losses are equipped to the Abalone dataset, respectively.
(c) RcRLoss on Abalone
(d) AL on Abalone
$c(\pmb{x})$  is considered as a constant, which is the most commonly considered scenario in learning with rejection [6, 8, 36, 11]. For each dataset, we set various rejection costs  $c$  including extreme cases and unstressed cases depending on the supervised loss. The complete experiments are provided in Appendix D.
5.5 Experimental Performance
Table 1, Table 2 and Table 3 show the experimental results when our surrogate loss function equipped with MAE on the AgeDB, BreastPathQ, and UCI datasets, respectively. From these three tables, we have the following observations: (1) Our proposed method significantly outperforms the supervised regression method in almost all cases, which validates the effectiveness of our method by rejecting difficult test instances. (2) In most cases, the average loss of our method in accepted test instances is always smaller than the average loss of the supervised regression model in all test instances. This further indicates the ability of our method to identify hard-to-predict instances and reject them. (3) As the rejection cost  $c$  increases, we can clearly see the following trends in all datasets: RcRLoss decreases; Rejection rate decrease; Accepted loss increases. This is because as the prediction error we can accept increases, i.e., the rejection cost increases, the rejector will accept more instances, leading to a decrease in the rejection rate. However, the regressor capacity remains the same and more instances (containing difficult instances) also face more challenges, so RcRLoss and AL increase but remain smaller than Sup. (4) In setting the rejection cost  $c$ , we consider many extreme cases, e.g., the rejection cost is much smaller and much larger than the average loss in the supervised regression. In such extreme cases, our approach is still effective to identify and reject difficult test instances. In addition, we show in Figure 1 a plot of RcRLoss with the increasing rejection cost when different binary classification losses are equipped on the Breast and Abalone datasets. As the rejection cost  $c$  increases, both AL loss and RcRLoss also increase.
5.6 Comparison with Selective Regression Methods
Since our paper provides the first attempt to investigate regression with cost-based rejection, there does not exist a baseline that can be directly compared. However, the Bayes optimal solution of regression with cost-based rejection is the same as the Bayes optimal solution of selective regression, so we can compare with selective regression methods. We have conducted additional experiments to compare with SelectiveNet [20] and Knn-Plug [46]. For SelectiveNet, this work proposes a neural network architecture and an optimization goal to control the rejection threshold of the rejector to ensure a specific rejection rate. To ensure a fair comparison, we use the same base model for all datasets. For Knn-Plug, this method proposes a semi-supervised learning process for learning a data-driven predictor with a reject option based on the plug-in principle. Specifically, this method learns a regression function  $h$  and a conditional variance function  $\sigma$  from the labeled dataset, while the unlabeled dataset is used to calibrate the conditional variance function  $\sigma$  to ensure the desired rejection rate. For each dataset, we randomly split the original dataset into a labeled training set, an unlabeled training set, and a test set by the proportions of  $60\%$  (the same size of labeled training set as ours),  $20\%$ , and  $20\%$ , respectively. To ensure fairness, we replace the original base model Knn with MLP and named MLP-Plug.
Table 4: Comparison with SelectiveNet and MLP-Plug. The cost is the rejection cost in regression with cost-based rejection and Rj is the expected rejection rate in selective regression.
Datasets	RcR_MA E	MLP Plug	SelectiveNet
Cost	RcRLoss	AL	Rej	Rj	AL	Rej	Rj	AL	Rej
Abalone	3	2.41(0.12)	1.99(0.21)	42.04(3.18)	42.04	2.13(0.22)	40.93(2.70)	42.00	2.10(0.19)	42.80(1.84)
4	2.88(0.13)	2.30(0.21)	33.70(2.47)	33.70	2.36(0.20)	32.30(2.89)	33.00	2.40(0.22)	34.26(1.52)
5	3.22(0.23)	2.66(0.35)	23.43(2.94)	23.43	2.72(0.23)	22.00(3.69)	23.00	2.84(0.42)	25.10(1.71)
6	3.53(0.25)	2.93(0.35)	19.32(3.47)	19.32	2.94(0.31)	17.64(2.71)	19.00	2.92(0.39)	21.20(2.00)
Auto-mpg	4	3.64(0.29)	2.99(0.83)	56.92(13.00)	56.92	4.83(2.28)	56.08(6.53)	56.00	3.58(1.91)	57.44(5.06)
6	4.83(0.93)	3.83(1.70)	37.31(14.10)	37.31	5.64(2.14)	36.46(9.47)	37.00	6.29(2.48)	35.13(6.34)
8	6.75(1.93)	6.14(2.41)	22.95(19.88)	22.95	6.30(2.03)	23.67(9.82)	23.00	6.86(2.08)	23.59(5.91)
13	8.13(2.41)	7.42(2.83)	12.56(6.83)	12.56	6.68(1.93)	10.63(3.54)	13.00	7.79(2.29)	11.15(3.49)
Housing	9	8.80(0.34)	6.25(3.22)	84.46(11.67)	84.46	8.56(4.70)	84.90(8.22)	84.00	7.70(2.22)	86.88(4.56)
12	9.52(0.75)	7.40(1.48)	44.65(8.69)	44.65	9.76(2.82)	44.80(8.62)	45.00	8.20(2.11)	46.83(5.86)
16	10.12(1.84)	8.35(1.58)	22.38(8.90)	22.38	10.52(3.70)	22.55(6.50)	22.00	8.73(1.64)	26.44(4.52)
20	10.50(3.32)	9.59(3.50)	8.51(6.82)	8.51	11.16(3.72)	10.20(4.14)	9.00	8.79(1.67)	11.39(2.44)
Concrete	20	18.03(1.32)	13.17(4.91)	69.42(6.92)	69.00	18.96(5.71)	71.17(4.93)	69.00	14.44(4.42)	71.60(3.16)
30	24.20(1.85)	19.29(3.85)	44.08(8.81)	44.00	25.13(4.57)	48.20(8.23)	44.00	21.48(3.77)	49.27(2.54)
40	28.63(2.56)	23.12(4.59)	31.50(8.98)	32.00	26.69(4.83)	33.74(7.59)	31.00	25.02(1.67)	36.99(2.46)
50	32.48(2.76)	27.90(4.31)	19.76(7.54)	20.00	29.26(4.56)	21.36(5.37)	20.00	28.66(3.23)	28.54(2.91)
60	34.33(3.50)	30.33(4.89)	12.82(6.62)	13.00	31.23(4.65)	14.90(4.81)	12.00	30.51(3.90)	20.00(3.59)
Table 4 shows the results of comparison experiments. Specifically, to be able to establish a connection between our studied RcR and selective regression, we set the expected rejection rate (Rj) of the selective regression method based on the results of RcR_MAe (our surrogate loss equipped MAE). It is important to note that there is no way to perfectly match the rejection rate, as the set Rj is "expected". In addition, we show plots of the variation in AL loss for all methods with different rejection rates in Appendix D.7. As can be seen Table 4, our proposed method outperforms (smaller AL loss with the same rejection rate) almost all compared methods, which validates the effectiveness of our method.
In this paper, we investigated a novel regression problem called regression with cost-based rejection, which aims to learn a model that can reject predictions to avoid critical mispredictions at a certain rejection cost. In order to solve this problem, we first formulated the expected risk for regression with cost-based rejection and derived the Bayes optimal solution for the expected risk, which shows that we should reject instances where the variance is greater than the rejection cost. Since the variance is difficult to obtain, we proposed a surrogate loss function that considers rejection as binary classification. Further, we provided conditions for the consistency of our method, implying that the optimal solution can be recovered by our method. Finally, we derived the regret transfer bound and the estimation error bound for our method and conducted extensive experiments on various datasets to demonstrate the effectiveness of our proposed method. We expect that our first study of a simple yet theoretically grounded method for regression with cost-based rejection can inspire more interesting studies on this new problem.
Lei Feng is supported by Chongqing Overseas Chinese Entrepreneurship and Innovation Support Program, CAAI-Huawei MindSpore Open Fund, and Openl Community (https://openi.pcl.ac.cn). Bo An is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
[1] P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
[2] P. L. Bartlett and S. Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of the American Statistical Association, 3:463-482, 2002.
[3] P. L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9(8), 2008.
[4] E. Beede, E. Baylor, F. Hersch, A. Iurchenko, L. Wilcox, P. Ruamviboonsuk, and L. M. Vardoulakis. A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy. In CHI, pages 1-12, 2020.
[5] R. K. Bellamy, K. Dey, M. Hind, S. C. Hoffman, S. Houde, K. Kannan, P. Lohia, J. Martino, S. Mehta, A. Mojsilovic, et al. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943, 2018.
[6] Y. Cao, T. Cai, L. Feng, L. Gu, J. Gu, B. An, G. Niu, and M. Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. In NeurIPS, pages 521-534, 2022.
[7] I. Chalkidis, I. Androutsopoulos, and N. Aletras. Neural legal judgment prediction in english. In ACL, 2019.
[8] N. Charoenphakdee, Z. Cui, Y. Zhang, and M. Sugiyama. Classification with rejection based on cost-sensitive classification. In ICML, pages 1507-1517, 2021.
[9] K. Chaudhuri, P. Jain, and N. Natarajan. Active heteroscedastic regression. In ICML, pages 694-702, 2017.
[10] C. Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory, 16(1):41-46, 1970.
[11] C. Cortes, G. DeSalvo, and M. Mohri. Boosting with abstention. In NeurIPS, 2016.
[12] C. Cortes, G. DeSalvo, and M. Mohri. Learning with rejection. In ALT, pages 67-82, 2016.
[13] D. Dua and C. Graff. UCI machine learning repository, 2017.
[14] R. El-Yaniv et al. On the foundations of noise-free selective classification. Journal of Machine Learning Research, 11(5), 2010.
[15] J. Finocchiaro, R. Frongillo, and B. Waggoner. An embedding framework for consistent polyhedral surrogates. In NeurIPS, 2019.
[16] V. Franc and D. Prusa. On discriminative learning of prediction uncertainty. In ICML, pages 1963-1971, 2019.
[17] A. Gangrade, A. Kag, and V. Saligrama. Selective classification via one-sided prediction. In AISTATS, pages 2179-2187, 2021.
[18] W. Gao and Z.-H. Zhou. On the consistency of auc pairwise optimization. arXiv preprint arXiv:1208.0645, 2012.
[41] A. Shah, Y. Bu, J. K. Lee, S. Das, R. Panda, P. Sattigeri, and G. W. Wornell. Selective regression under fairness criteria. In ICML, pages 19598-19615, 2022.
[42] B. W. Silverman. Some aspects of the spline smoothing approach to non-parametric regression curve fitting. Journal of the Royal Statistical Society: Series B (Methodological), 47(1):1-21, 1985.
[43] H. White. A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica: Journal of the Econometric Society, pages 817-838, 1980.
[44] Y. Wiener and R. El-Yaniv. Pointwise tracking the optimal regression function. In NeurIPS, 2012.
[45] M. Yuan and M. Wegkamp. Classification methods with reject option based on convex risk minimization. Journal of Machine Learning Research, 11(1), 2010.
[46] A. Zaoui, C. Denis, and M. Hebiri. Regression with reject option and application to knn. In NeurIPS, pages 20073-20082, 2020.
[47] T. Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research, 5(Oct):1225-1251, 2004.
[48] Y. Zoabi, S. Deri-Rozov, and N. Shomron. Machine learning-based prediction of Covid-19 diagnosis based on symptoms. npj Digital Medicine, 4(1):1-5, 2021.
For an instance  $\pmb{x}$ , we have the following expected risk for  $\pmb{x}$ :
$$
\begin{array}{l} R _ {\operatorname {R e R} | \boldsymbol {x}} (h, r) = \mathbb {E} _ {p (y | \boldsymbol {x})} [ \mathcal {L} (h, r, c, \boldsymbol {x}, y) ] \\ = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) \mathcal {L} (h, r, c, \boldsymbol {x}, y) d y \\ \end{array}
$$
If we refuse to make a prediction for  $\pmb{x}$ , i.e.,  $r(\pmb{x}) < 0$ , the above expected risk transforms into the following equation:
$$
\begin{array}{l} R _ {\operatorname {R c R} | \boldsymbol {x}, r (\boldsymbol {x}) <   0} (h, r) = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) \mathcal {L} (h, r, c, \boldsymbol {x}, y) d y \\ = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) c (\boldsymbol {x}) \mathrm {d} y \\ = c (\boldsymbol {x}). \\ \end{array}
$$
When we make a prediction for  $\pmb{x}$ , i.e.,  $r(\pmb{x}) > 0$ , the above expected risk transforms into the following equation:
$$
\begin{array}{l} R _ {\operatorname {R c R} | \boldsymbol {x}, r (\boldsymbol {x}) > 0} (h, r) = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) \mathcal {L} (h, r, c, \boldsymbol {x}, y) \mathrm {d} y \\ = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) \left(h (\boldsymbol {x}) - y\right) ^ {2} \mathrm {d} y \\ = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) \left(h (\boldsymbol {x}) ^ {2} - 2 y h (\boldsymbol {x}) + y ^ {2}\right) d y \\ = \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) h (\boldsymbol {x}) ^ {2} \mathrm {d} y - \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) 2 y h (\boldsymbol {x}) \mathrm {d} y + \int_ {\mathcal {Y}} p (y | \boldsymbol {x}) y ^ {2} \mathrm {d} y \\ = h ^ {2} (\boldsymbol {x}) - 2 h (\boldsymbol {x}) \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ] + \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ^ {2} ] \\ = h ^ {2} (\boldsymbol {x}) - 2 h (\boldsymbol {x}) \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ] + \mathbb {E} _ {p (y | \boldsymbol {x})} ^ {2} [ y ] + \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ] \\ = \left(h (\boldsymbol {x}) - \mathbb {E} _ {p (y | \boldsymbol {x})} [ y ]\right) ^ {2} + \mathbb {D} _ {p (y | \boldsymbol {x})} [ y ] \\ \end{array}
$$