SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: Test performance (mean and std) of our surrogate loss equipped logistic loss on five UCI datasets trained with the Linear model. We repeat the sampling-and-training process 10 times. The metrics Rej, AR, and RA are scaled to 0-100.
Datasets	Cost	Supervised	Rej	AL	RL	Rej	AR	RA
Abalone	3	4.92(0.51)	2.52(0.08)	1.94(0.21)	7.84(1.06)	54.77(2.66)	44.76(3.45)	24.00(1.93)
4	2.99(0.11)	2.39(0.19)	9.83(1.44)	36.93(2.78)	27.94(3.02)	36.55(2.83)
5	3.38(0.18)	2.80(0.25)	11.78(1.86)	25.90(2.27)	18.43(2.08)	46.24(3.20)
6	3.69(0.26)	3.19(0.31)	13.81(2.00)	17.80(1.98)	11.89(1.45)	55.08(4.19)
Airfoil	9	23.32(1.54)	8.83(0.35)	7.55(2.64)	26.44(1.99)	85.58(6.10)	78.07(8.53)	7.24(3.99)
12	11.31(0.49)	8.76(2.18)	27.59(2.06)	79.93(3.65)	71.90(5.57)	9.74(1.97)
16	14.46(0.51)	10.90(1.46)	30.60(2.13)	69.47(6.49)	60.88(7.17)	17.14(5.66)
20	17.10(0.83)	13.01(1.79)	33.51(4.17)	58.54(5.07)	50.38(5.67)	26.19(6.54)
25	19.62(1.26)	15.95(2.52)	35.26(3.40)	39.30(5.76)	34.28(5.41)	47.29(8.82)
30	20.94(1.84)	17.60(3.18)	42.36(7.32)	25.38(8.07)	21.35(6.97)	61.14(12.70)
Auto-mpg	4	11.66(2.26)	3.92(0.26)	2.78(1.68)	15.05(4.12)	79.87(11.10)	73.18(14.13)	15.28(7.25)
6	5.73(0.70)	5.25(1.63)	17.98(6.14)	58.97(8.50)	52.23(9.69)	32.06(10.05)
8	6.73(0.52)	5.72(0.92)	21.58(7.67)	42.56(7.84)	36.08(8.66)	43.16(11.24)
10	7.37(0.95)	5.94(1.61)	26.05(11.45)	31.28(12.77)	25.72(10.98)	53.96(21.76)
13	8.75(1.64)	7.72(1.94)	28.93(12.64)	19.62(4.69)	17.31(4.60)	69.71(13.77)
Housing	9	24.08(5.34)	8.65(0.75)	6.95(3.16)	33.87(12.62)	67.92(11.51)	58.56(14.04)	19.28(7.94)
12	10.27(1.08)	8.19(2.90)	40.93(16.74)	58.32(10.20)	48.25(13.11)	23.32(5.75)
16	12.34(1.14)	9.20(2.03)	50.31(19.14)	45.35(6.04)	36.24(6.00)	31.42(6.77)
20	14.19(1.67)	10.48(2.72)	55.08(23.26)	38.42(6.08)	32.22(6.62)	42.45(12.64)
Concrete	20	111.12(8.01)	19.80(0.29)	10.00(6.44)	204.20(63.34)	97.57(2.04)	95.25(4.29)	1.55(0.86)
30	29.51(0.92)	24.08(12.35)	227.13(99.54)	91.17(4.57)	87.60(6.54)	6.02(3.11)
40	38.09(1.38)	28.95(7.18)	282.98(96.63)	80.34(6.60)	73.83(7.46)	12.05(6.39)
50	46.94(1.82)	34.22(9.57)	242.13(98.34)	75.34(10.15)	68.94(11.79)	15.98(7.51)
60	51.96(2.08)	41.36(5.76)	370.24(113.24)	56.26(2.61)	45.96(2.24)	25.26(4.63)
D Additional Information of Experiments
D.1 Evaluation Metrics
We describe in detail all the evaluation metrics we used in our experiments.
RcR loss. The RcR loss (RcRLoss) is the main evaluation metric for RcR. For a given example  $(\pmb{x},y)$  and the rejection cost  $c(\pmb{x})$ , the RcRLoss defined as if  $r(\pmb{x}) > 0$ ,  $\mathcal{L}(h,r,c,\pmb{x},y) = (h(\pmb{x}) - y)^2$ , otherwise  $\mathcal{L}(h,r,c,\pmb{x},y) = c(\pmb{x})$ .
Rejection rate. The rejection rate (Rej) is defined as  $\frac{\sum_{i=1}^{n} \mathbb{I}[r(\boldsymbol{x}) \leq 0]}{n}$ . Rej indicates the rejection rate of our model on the test dataset.
Table 8: Test performance (mean and std) of our surrogate loss equipped logistic loss on five UCI datasets trained with the MLP model. We repeat the sampling-and-training process 10 times. The metrics Rej, AR, and RA are scaled to 0-100.
Datasets	Cost	Supervised	Rej	AL	RL	Rej	AR	RA
Abalone	3	4.44(0.46)	2.41(0.10)	1.95(0.18)	8.34(0.87)	43.14(2.84)	33.37(3.34)	33.32(2.81)
4	2.87(0.18)	2.33(0.29)	9.68(1.18)	32.29(3.28)	24.28(3.29)	41.96(3.29)
5	3.20(0.20)	2.59(0.29)	10.86(1.34)	25.25(2.15)	17.86(2.02)	45.15(3.61)
6	3.48(0.25)	2.82(0.35)	11.54(1.53)	20.44(1.72)	14.74(1.42)	51.48(4.63)
Airfoil	9	12.96(2.60)	6.78(0.47)	5.05(0.82)	51.45(4.32)	43.68(5.19)	20.35(5.21)	23.75(3.96)
12	7.96(0.64)	5.79(0.82)	59.74(3.53)	35.05(3.02)	12.94(2.79)	26.90(3.03)
16	9.04(0.56)	7.46(0.47)	68.20(10.78)	18.57(4.08)	7.36(3.14)	48.00(7.21)
20	9.64(0.74)	7.81(0.44)	74.27(10.19)	15.05(7.76)	5.83(1.91)	47.91(11.27)
25	10.47(1.08)	8.50(0.52)	80.28(27.67)	12.03(4.43)	3.78(1.89)	49.00(17.65)
30	10.95(1.11)	8.95(0.78)	89.30(31.58)	9.50(3.86)	2.93(1.10)	50.67(18.37)
Auto-mpg	4	8.34(2.16)	3.85(0.56)	3.22(1.51)	11.99(3.81)	62.44(10.72)	54.18(9.53)	25.19(13.29)
6	5.33(0.82)	4.67(1.36)	15.08(3.83)	43.01(16.01)	35.19(16.09)	41.25(15.35)
8	6.53(1.18)	5.86(1.53)	19.34(7.60)	29.49(13.45)	23.19(13.60)	53.57(14.78)
10	7.06(1.60)	6.42(1.91)	21.71(8.88)	17.95(3.63)	14.15(3.48)	65.59(11.17)
13	7.80(1.90)	7.04(2.09)	28.55(14.96)	13.59(5.35)	11.05(5.32)	70.15(14.83)
Housing	9	12.57(3.43)	8.60(2.49)	8.43(3.15)	45.60(27.02)	26.57(5.78)	19.05(5.63)	66.95(10.39)
12	9.50(1.56)	8.63(2.10)	63.52(26.15)	25.44(6.43)	17.38(5.84)	52.68(12.58)
16	9.30(1.37)	8.03(1.70)	90.19(38.08)	15.45(4.30)	10.84(3.87)	62.99(9.05)
20	9.67(1.40)	8.33(1.77)	103.55(54.73)	11.18(2.97)	8.71(2.70)	70.06(8.59)
Concrete	20	34.44(3.05)	18.65(1.41)	14.32(4.80)	58.33(15.88)	68.93(13.47)	57.72(17.08)	16.19(9.45)
30	25.64(2.50)	23.16(4.95)	80.43(14.47)	32.85(12.82)	19.30(10.81)	48.23(15.91)
40	29.79(2.33)	25.25(3.55)	107.54(22.18)	30.24(8.58)	19.97(7.91)	44.38(9.87)
50	31.63(4.29)	25.79(4.22)	120.02(24.12)	24.22(11.22)	15.29(10.47)	47.42(10.35)
60	34.04(4.48)	33.26(4.55)	165.71(62.18)	2.77(5.13)	2.14(2.93)	92.59(12.85)
Accepted loss. The accepted loss (AL) is defined as  $\frac{\sum_{i=1}^{n} \mathbb{I}[r(\boldsymbol{x}_i) > 0](h(\boldsymbol{x}_i) - y_i)^2}{\sum_{i=1}^{n} \mathbb{I}[r(\boldsymbol{x}_i) > 0]}$ . AL denotes the average loss of our regressor on the accepted test dataset.
Rejected loss. The rejected loss (RL) is defined as  $\frac{\sum_{i=1}^{n} \mathbb{I}[r(\pmb{x}_i) \leq 0](h(\pmb{x}_i) - y_i)^2}{\sum_{i=1}^{n} \mathbb{I}[r(\pmb{x}_i) \leq 0]}$ . RL denotes the average loss of our regressor on the rejected test dataset.
False rejection rate. The false rejection rate (AR) is defined as  $\frac{\sum_{i=1}^{n} \mathbb{I}[(h(\boldsymbol{x}_i) - y_i)^2 < c(\boldsymbol{x}_i)] \mathbb{I}[r(\boldsymbol{x}_i) \leq 0]}{\sum_{i=1}^{n} \mathbb{I}[(h(\boldsymbol{x}_i) - y_i)^2 < c(\boldsymbol{x}_i)]}$ . AR denotes the rate of instances that should be accepted that are rejected.
Table 9: Test performance (mean and std) of our surrogate loss equipped logistic loss on BreastPathQ. We repeat the sampling-and-training process 5 times. The metrics Rej, AR, RA are scaled to 0-100 and Sup, RcRLoss, AL and RL are all magnified by a factor of 1000.
Cost	Sup	RcRLoss	AL	RL	Rej	AR	RA
5	4.41(0.35)	2.92(1.51)	35.59(7.36)	71.56(3.20)	47.48(5.95)	5.71(3.02)
10	7.99(0.47)	4.72(1.01)	41.34(9.74)	61.72(9.54)	40.69(4.90)	10.30(1.02)
15	16.77(1.22)	11.52(0.36)	7.98(1.36)	42.67(5.87)	50.48(5.85)	35.50(4.61)	20.18(7.72)
20	13.69(0.81)	8.92(1.05)	63.51(49.02)	43.65(5.24)	31.11(5.00)	22.61(5.72)
25	16.64(0.94)	12.96(2.28)	35.27(2.63)	29.84(5.93)	23.63(5.42)	44.77(10.54)
Table 10: Test performance (mean and std) of our surrogate loss equipped square loss on BreastPathQ. We repeat the sampling-and-training process 5 times. The metrics Rej, AR, RA are scaled to 0-100 and Sup, RcRLoss, AL and RL are all magnified by a factor of 1000.
Cost	Sup	RcRLoss	AL	RL	Rej	AR	RA
5	4.67	3.60	36.70	69.23	44.09	6.26
(0.41)	(1.17)	(4.46)	(4.94)	(4.16)	(3.25)
10	8.13	5.30	43.66	59.69	37.47	10.42
(0.38)	(0.73)	(6.95)	(2.59)	(2.40)	(2.60)
15	16.77	12.02	8.83	39.83	51.70	36.78	17.65
(1.09)	(2.14)	(8.27)	(2.56)	(0.80)	(2.03)
20	(1.22)	14.58	9.66	43.69	44.72	33.20	24.21
(0.57)	(2.55)	(7.58)	(9.54)	(7.89)	(11.09)
25	15.75	11.98	45.65	27.57	19.94	43.73
(0.76)	(2.59)	(7.18)	(8.62)	(7.86)	(12.04)
False acceptance rate. The false acceptance rate (RA) denotes the rate of instances that should be rejected that are accepted, and is defined as  $\frac{\sum_{i=1}^{n} \mathbb{I}[(h(\boldsymbol{x}_i) - y_i)^2 \geq c(\boldsymbol{x}_i)] \mathbb{I}[r(\boldsymbol{x}_i) > 0]}{\sum_{i=1}^{n} \mathbb{I}[(h(\boldsymbol{x}_i) - y_i)^2 \geq c(\boldsymbol{x}_i)]}$ .
D.2 Some Results for Hinge Loss
In this section, we show some experimental results of the surrogate loss equipped with hinge loss, which can be formulated as follows:
$$
\psi (h, r, c, \boldsymbol {x}, y) = (h (\boldsymbol {x}) - y) ^ {2} \max (0, 1 + r (\boldsymbol {x})) + c (\boldsymbol {x}) \max (0, 1 - r (\boldsymbol {x})).
$$
Table 5 and Table 6 show some of the experimental results on the AgeDB adn BreastPathQ when surrogate loss equipped hinge loss, respectively. From this table, we can see that RcRLoss and AL is always lower than Sup in almost all experiments, which means that our method is effective in identifying test instances should be accepted and test instances should be rejected. It is worth noting that in most experiments, there is a low RA, which means that there is a higher tendency to reject hard-to-predict test instances to avoid serious errors when equipped hinge loss.
D.3 Some Results for Logistic Loss
In this section, we show some experimental results of the surrogate loss equipped with logistic loss, which can be formulated as follows:
$$
\psi (h, r, c, \boldsymbol {x}, y) = (h (\boldsymbol {x}) - y) ^ {2} \log (1 + \exp (r (\boldsymbol {x}))) + c (\boldsymbol {x}) \log (1 + \exp (- r (\boldsymbol {x}))).
$$
Table 7, Table 8 and Table 9 show some of the experimental results on the UCI datasets and BreastPathQ when surrogate loss equipped logistic loss, respectively. Our proposed method significantly outperforms the supervised regression method in almost all cases, which verifies the ability of our
Table 11: Test performance (mean and std) of our surrogate loss equipped square loss on five UCI datasets trained with the MLP model. We repeat the sampling-and-training process 10 times. The metrics Rej, AR, and RA are scaled to 0-100.
Datasets	Cost	Supervised	Rej	AL	RL	Rej	AR	RA
Abalone	3	4.44(0.46)	2.39(0.10)	1.96(0.19)	7.82(0.63)	42.54(2.49)	32.79(2.69)	32.58(2.63)
4	2.84(0.16)	2.33(0.25)	8.79(0.97)	31.82(1.87)	23.72(2.14)	40.81(2.77)
5	3.18(0.18)	2.60(0.27)	9.89(1.15)	25.37(2.13)	18.32(2.07)	45.65(4.21)
6	3.50(0.29)	2.89(0.42)	10.40(1.11)	20.38(2.17)	14.37(1.85)	49.83(5.47)
Airfoil	9	12.96(2.60)	6.40(0.25)	4.36(0.36)	51.93(5.13)	43.65(3.26)	22.05(2.93)	22.22(3.71)
12	7.46(0.31)	5.11(0.38)	61.13(5.83)	33.75(2.90)	15.04(3.10)	27.05(2.74)
16	8.57(0.40)	5.81(0.30)	70.20(7.82)	26.98(3.23)	10.54(3.08)	28.83(1.79)
20	9.27(0.42)	6.66(0.43)	76.90(10.02)	19.34(2.34)	7.70(1.54)	35.09(4.65)
25	9.97(0.60)	7.23(0.51)	87.37(9.07)	15.35(2.44)	5.19(1.38)	32.96(5.61)
30	10.33(0.86)	7.82(0.79)	85.67(18.03)	11.23(1.95)	3.92(1.25)	37.40(8.27)
Auto-mpg	4	8.34(2.16)	3.65(0.26)	2.83(0.90)	11.93(3.22)	62.31(11.46)	51.76(12.43)	22.05(10.30)
6	5.19(0.77)	4.31(1.47)	18.00(7.69)	39.62(21.51)	33.51(21.46)	47.55(24.43)
8	6.51(1.35)	5.82(1.74)	22.62(8.96)	29.10(14.57)	22.28(13.86)	52.29(16.21)
10	6.80(1.16)	6.08(1.43)	23.57(9.41)	17.82(2.84)	13.91(1.93)	65.62(9.09)
13	7.28(1.30)	6.45(1.36)	30.51(15.46)	12.69(3.74)	10.05(3.46)	71.16(15.70)
Housing	9	12.57(3.43)	8.41(1.56)	8.22(2.10)	53.44(20.25)	28.22(7.81)	21.42(7.35)	56.77(9.38)
12	9.03(1.26)	8.36(1.64)	76.10(47.37)	17.13(5.71)	12.16(5.11)	66.75(11.99)
16	8.52(1.35)	7.64(1.62)	109.61(60.72)	10.10(3.67)	7.30(2.71)	73.14(13.36)
20	9.40(1.94)	8.56(2.16)	148.19(98.03)	7.03(3.30)	5.09(2.31)	73.76(16.54)
Concrete	20	34.44(3.05)	19.95(2.56)	18.77(5.05)	75.19(11.68)	55.10(13.77)	43.24(14.20)	28.28(11.92)
30	25.22(3.22)	22.44(5.34)	103.99(18.06)	33.45(6.93)	22.25(6.21)	43.75(9.93)
40	31.21(1.50)	28.84(1.84)	127.77(19.65)	21.17(5.11)	12.47(4.12)	56.12(7.44)
50	29.55(2.97)	25.00(3.80)	147.99(36.87)	18.01(4.62)	9.87(3.88)	52.49(9.18)
60	33.07(3.51)	28.81(3.94)	158.65(33.36)	13.64(3.71)	7.28(2.91)	59.55(8.52)
method to reject difficult test instances demonstrating the effectiveness of our method. In most cases, the average loss of our method in the accepted test instances (AL) is always smaller than the average loss of the supervised regression model (Sup) in all test instances. This further indicates the ability of our method to identify hard-to-predict samples and reject them. On both MLP and Linear models, our method is effective in avoiding serious errors, which verifies that our method can be adapted to different models.
D.4 Some Results for Square Loss
In this section, we show some experimental results of the surrogate loss equipped with square loss, which can be formulated as follows:
$$
\psi (h, r, c, \boldsymbol {x}, y) = (h (\boldsymbol {x}) - y) ^ {2} (r (\boldsymbol {x}) + 1) ^ {2} + c (\boldsymbol {x}) (r (\boldsymbol {x}) - 1) ^ {2}.
$$
Table 12: Test performance (mean and std) of our surrogate loss equipped sigmoid on BreastPathQ. We repeat the sampling-and-training process 5 times. The metrics Rej, AR, RA are scaled to 0-100 and Sup, RcRLoss, AL and RL are all magnified by a factor of 1000.
Cost	Sup	Rej	AL	RL	Rej	AR	RA
5	4.42	2.40	43.86	79.22	56.41	2.81
(0.17)	(1.01)	(10.60)	(1.54)	(4.19)	(0.92)
10	8.44	5.09	51.35	69.34	47.12	6.39
(0.46)	(1.64)	(7.30)	(2.56)	(3.83)	(1.83)
15	16.77	11.63	6.30	58.40	60.23	41.23	10.88
(0.38)	(1.83)	(14.79)	(6.75)	(5.42)	(5.57)
20	(1.22)	14.31	8.91	57.83	49.19	33.84	17.31
(0.66)	(1.26)	(9.11)	(4.68)	(5.32)	(3.13)
25	16.61	9.09	95.10	47.38	29.46	17.78
(0.60)	(1.53)	(59.42)	(2.92)	(4.84)	(4.73)
Table 10 and Table 11 show some of the experimental results on the BreastPathQ and UCI datasets with MLP model when surrogate loss equipped square loss, respectively. When the rejection cost  $c$  is small, both RcRLoss and AL are significantly smaller than Sup. When the rejection cost  $c$  is large, RcRLoss and AL are close to Sup but always smaller, which shows the effectiveness of our method to deal with regression with cost-based rejection.
D.5 Some Results for Sigmoid
In this section, we show some experimental results of the Sigmoid function equipped with sigmoid, which can be formulated as follows:
$$
\psi (h, r, c, \boldsymbol {x}, y) = (h (\boldsymbol {x}) - y) ^ {2} \operatorname {s i g m o i d} (r (\boldsymbol {x})) + c (\boldsymbol {x}) \operatorname {s i g m o i d} (- r (\boldsymbol {x})).
$$
Unlike other binary classification losses, sigmoid can be viewed as weight balancing prediction loss and the rejection cost due to  $\mathrm{sigmoid}(r(x)) + \mathrm{sigmoid}(-r(x)) = 1$ . Table 12 show some of the experimental results on BreastPathQ when surrogate loss equipped sigmoid, respectively. RcRLoss and AL are always smaller than Sup, verifying the effectiveness of our method.
In our experiments, we used multiple binary classification losses (MAE, hinge loss, logistic loss, square loss and sigmoid) and different datasets including two deep datasets (BreastPathQ and AgeDB) and five uci datasets (Abalone, Airfoil, Auto-mpg, Housing and Concrete), and our method outperformed supervised regression in most cases, which demonstrates the effective of our method.
D.6 Performance of Increasing Training Data
As we showed in Theorem 8 and Theorem 9, the pair  $(h,r)$  learned by our surrogate loss could converge to the optimal pair  $(h^{*},r^{*})$  when the number of training examples approaches to infinity. Therefore, the performance of the model can be improved as the training samples increase. To empirically validate such a theoretical finding, we further conduct experiments on the Abalone and Auto-mpg datasets by changing the fraction of training examples (100% means that we use all training examples in the training set). As shown in Figure 2, the RcRLoss and the AL generally decreases when more training examples are used for model training. This observation is clearly in accordance with our theoretical analyses in Theorem 8 and Theorem 9, because the learned model would be closer to the optimal model as more training examples are provided.
D.7 Curves of AL and Rej
We show the AL loss for all methods with different rejection rates on Abalone, Autompg and Concrete datasets in Figure 3. In the plot of the relationship between AL loss and rejection rate, a curve at the bottom means better results.
(a) Breast with  $c = 5$
(b) Breast with  $c = 10$
(c) Abalone with  $c = 4$
(d) Abalone with  $c = 6$
(e) Housing with  $c = 12$
(f) Housing with  $c = 16$
Figure 2: The test performance on the Breast, Abalone and Housing datasets for the surrogate loss equipped MAE when the number of training examples increases.
(a) Abalone
Figure 3: Figures (a), (b) and (c) report the accepted loss (AL) for all methods with different rejection rates on abalone, auto-mpg and concrete datasets, respectively.
(b) Autompg
(c) Concrete
In Theorem 4 and Theorem 5, we show that there is a limitation in our proposed method that requires the binary classification loss  $\ell(r(x), z)$  to be always greater than 0. This is easily satisfied by the design of the binary classification loss. However, to avoid the modification of the binary classification loss, we further propose Theorem 6, which only requires the binary classification loss to be non-negative, and this is easily satisfied. Extensive experiments on various datasets demonstrate the effectiveness of our proposed method. On the other hand, we use the pointwise cost functions  $c(x)$  in our theoretical analysis, which shows that our method can be used for pointwise cost functions. For simplicity, in our experiments, we only consider the rejection cost as a constant  $c$ , but our method can easily handle the various pointwise cost functions in different application scenarios.