SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wei Wang $^{1,2}$ , Lei Feng $^{3,2}$ , Yuchen Jiang $^{4}$ , Gang Niu $^{2}$ , Min-Ling Zhang $^{5}$ , Masashi Sugiyama $^{2,1}$
$^{2}$  RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
<sup>3</sup>Nanyang Technological University, Singapore
4 Alibaba Group, Beijing, China
$^{5}$  Southeast University, Nanjing, China
wangw@g.ecc.u-tokyo.ac.jplfengqaq@gmail.com
jiangyuchen.jyc@alibaba-inc.com gang.niu.ml@gmail.com
zhangml@seu.edu.cn sugi@k.u-tokyo.ac.jp
Recently, learning with soft labels has been shown to achieve better performance than learning with hard labels in terms of model generalization, calibration, and robustness. However, collecting pointwise labeling confidence for all training examples can be challenging and time-consuming in real-world scenarios. This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. Instead of pointwise labeling confidence, we are given only unlabeled data pairs with confidence difference that specifies the difference in the probabilities of being positive. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound achieves the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven. Extensive experiments on benchmark data sets and a real-world recommender system data set validate the effectiveness of our proposed approaches in exploiting the supervision information of the confidence difference.
Recent years have witnessed the prevalence of deep learning and its successful applications. However, the success is built on the basis of the collection of large amounts of data with unique and accurate labels. However, in many real-world scenarios, it is often difficult to satisfy such requirements. To circumvent the difficulty, various weakly supervised learning problems have been investigated accordingly, including but not limited to semi-supervised learning [1, 2, 3, 4], label-noise learning [5, 6, 7, 8, 9], positive-unlabeled learning [10, 11, 12], partial-label learning [13, 14, 15, 16, 17], unlabeled-unlabeled learning [18, 19], and similarity-based classification [20, 21, 22].
Learning with soft labels has been shown to achieve better performance than learning with hard labels in the context of supervised learning [23, 24], where each example is equipped with pointwise labeling confidence indicating the degree to which the labels describe the example. The advantages have been validated in many aspects, including model generalization [25, 26], calibration [27, 28], and robustness [29, 30]. For example, with the help of soft labels, knowledge distillation [25, 31] transfers knowledge from a large teacher network to a small student network. The student network can be trained more efficiently and reliably with the soft labels generated by the teacher network [32, 33, 34].
However, collecting a large number of training examples with pointwise labeling confidence may be demanding under many circumstances since it is challenging to describe the labeling confidence for each training example exactly [35, 36, 37]. Different annotators may give different values of pointwise
labeling confidence to the same example due to personal biases, and it has been demonstrated that skewed confidence values can harm classification performance [36]. Besides, giving pointwise labeling information to large-scale data sets is also expensive, laborious, and even unrealistic in many real-world scenarios [38, 39, 9]. On the contrary, leveraging supervision information of pairwise comparisons may ameliorate the biases of skewed pointwise labeling confidence and save labeling costs. Following this idea, we investigate a more practical problem setting for binary classification in this paper, where we are given unlabeled data pairs with confidence difference indicating the difference in the probabilities of being positive. Collecting confidence difference for training examples in pairs is much cheaper and more accessible than collecting pointwise labeling confidence for all the training examples.
Take click-through rate prediction in recommender systems [40, 41] for example. The combinations of users and their favorite/disliked items can be regarded as positive/negative data. Collecting training data takes work to distinguish between positive and negative data. Furthermore, the pointwise labeling confidence of training data may be difficult to be determined due to the extraordinarily sparse and class-imbalance problems [42]. Therefore, the collected confidence values may be biased. However, collecting the difference in the preference between a pair of candidate items for a given user is more accessible and may alleviate the biases. Section 4 will give a real-world case study of this problem. Take the disease risk estimation problem for another example. Given a person's attributes, the goal is to predict the risk of having some disease. When asking doctors to annotate the probabilities of having the disease for patients, it takes work to determine the exact values of the probabilities. Furthermore, the probability values given by different doctors may differ due to their diverse backgrounds. On the other hand, it is much easier and less biased to estimate the relative difference in the probabilities of having the disease between two patients. Therefore, the problem of learning with confidence difference is of practical research value, but has yet to be investigated in the literature.
As a related work, Feng et al. [43] elaborated that a binary classifier can be learned from pairwise comparisons, termed Pcomp classification. For a pair of samples, they used a pairwise label of one is being more likely to be positive than the other. Since knowing the confidence difference implies knowing the pairwise label, our method requires stronger supervision than Pcomp classification.
Nevertheless, we argue that, in many real-world scenarios, we may not only know one example is more likely to be positive than the other, but also know how much the difference in confidence is, as explained in the above examples. Therefore, the setting of the current paper is not so restrictive compared with that of Pcomp classification. Furthermore, our setting is more flexible than that of Pcomp classification from the viewpoint of data generation process. Pcomp classification limits the labels of pairs of training data to be in  $\{(+1, +1), (+1, -1), (-1, -1)\}$ . To cope with collected data with labels  $(-1, +1)$ , they either discard them or reverse them as  $(+1, -1)$ . On the contrary, our setting is more general and we take the examples from  $(-1, +1)$  also into consideration explicitly in the data distribution assumption. Figure 1 shows the results of a pilot experiment. Here,
Figure 1: Experimental results for different strategies of handling examples from  $(-1, +1)$ .
Strategy1 denotes Pcomp-Teacher [43] discarding examples from  $(-1, +1)$  while Strategy2 denotes Pcomp-Teacher reversing examples from  $(-1, +1)$ . We can observe that both strategies perform actually similarly. Since Strategy1 loses many data, it is intuitively expected that Strategy2 works much better, but the improvement is actually marginal. This may be because the training data distribution after reversing differs from the expected one. On the contrary, in this work we take examples from  $(-1, +1)$  into consideration directly, which is a more appropriate way to handle these data.
Learning with pairwise comparisons has been investigated pervasively in the community [44, 45, 46, 47, 48, 49, 50], with applications in information retrieval [51], computer vision [52], regression [53, 54], crowdsourcing [55, 56], and graph learning [57]. It is noteworthy that there exist distinct differences between our work and previous works on learning with pairwise comparisons. Previous works have mainly tried to learn a ranking function that can rank candidate examples according to relevance or preference. In this paper, we try to learn a pointwise binary classifier by conducting empirical risk minimization under the binary classification setting.
- We investigate confidence-difference (ConfDiff) classification, a novel and practical weakly supervised learning problem, which can be solved via empirical risk minimization by constructing an unbiased risk estimator. The proposed approach can be equipped with any model, loss function, and optimizer flexibly.
- An estimation error bound is derived, showing that the proposed approach achieves the optimal parametric convergence rate. The robustness is further demonstrated by probing into the influence of an inaccurate class prior probability and noisy confidence difference.
- To mitigate overfitting issues, a risk correction approach [19] with consistency guarantee is further introduced. Extensive experimental results on benchmark data sets and a real-world recommender system data set validate the effectiveness of the proposed approaches.
In this section, we discuss the background of binary classification, binary classification with soft labels, and Pcomp classification. Then, we elucidate the data generation process of ConfDiff classification.
2.1 Binary Classification
For binary classification, let  $\mathcal{X} = \mathbb{R}^d$  denote the  $d$ -dimensional feature space and  $\mathcal{Y} = \{+1, -1\}$  denote the label space. Let  $p(\boldsymbol{x}, y)$  denote the unknown joint probability density over random variables  $(\boldsymbol{x}, y) \in \mathcal{X} \times \mathcal{Y}$ . The task of binary classification is to learn a binary classifier  $g: \mathcal{X} \to \mathbb{R}$  which minimizes the following classification risk:
$$
R (g) = \mathbb {E} _ {p (\boldsymbol {x}, y)} [ \ell (g (\boldsymbol {x}), y) ], \tag {1}
$$
where  $\ell(\cdot, \cdot)$  is a non-negative binary-class loss function, such as the 0-1 loss and logistic loss. Let  $\pi_{+} = p(y = +1)$  and  $\pi_{-} = p(y = -1)$  denote the class prior probabilities for the positive and negative classes respectively. Furthermore, let  $p_{+}(\boldsymbol{x}) = p(\boldsymbol{x} | y = +1)$  and  $p_{-}(\boldsymbol{x}) = p(\boldsymbol{x} | y = -1)$  denote the class-conditional probability densities of positive and negative data respectively. Then the classification risk in Eq. (1) can be equivalently expressed as
$$
R (g) = \pi_ {+} \mathbb {E} _ {p _ {+} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), + 1) ] + \pi_ {-} \mathbb {E} _ {p _ {-} (\boldsymbol {x})} [ \ell (g (\boldsymbol {x}), - 1) ]. \tag {2}
$$
2.2 Binary Classification with Soft Labels
When the soft labels of training examples are accessible to the learning algorithm, taking advantage of them can often improve the generalization performance [23]. First, the classification risk in Eq. (1) can be equivalently expressed as
$$
R (g) = \mathbb {E} _ {p (\boldsymbol {x})} [ p (y = + 1 | \boldsymbol {x}) \ell (g (\boldsymbol {x}), + 1) + p (y = - 1 | \boldsymbol {x}) \ell (g (\boldsymbol {x}), - 1) ]. \tag {3}
$$
Then, given training data equipped with confidence  $\{(\pmb{x}_i,r_i)\}_{i = 1}^n$  where  $r_i = p(y_i = +1|\pmb{x}_i)$  is the pointwise positive confidence associated with  $\pmb{x}_i$ , we minimize the following unbiased risk estimator to perform empirical risk minimization:
$$
\widehat {R} _ {\text {s o f t}} (g) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(r _ {i} \ell \left(g \left(\boldsymbol {x} _ {i}\right), + 1\right) + \left(1 - r _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i}\right), - 1\right)\right). \tag {4}
$$
However, accurate pointwise positive confidence may be hard to be obtained in reality [36].
2.3 Pairwise-Comparison (Pcomp) Classification
In principle, collecting supervision information of pairwise comparisons is much easier and cheaper than pointwise supervision information [43]. In Pcomp classification [43], we are given pairs of unlabeled data where we know which one is more likely to be positive than the other. It is assumed that Pcomp data are sampled from labeled data pairs whose labels belong to  $\{(+1, -1), (+1, +1), (-1, -1)\}$ . Based on this assumption, the probability density of Pcomp data  $(\pmb{x}, \pmb{x}^{\prime})$  is given as  $q(\pmb{x}, \pmb{x}^{\prime}) / (\pi_{+}^{2} + \pi_{-}^{2} + \pi_{+} \pi_{-})$  where  $q(\pmb{x}, \pmb{x}^{\prime}) = \pi_{+}^{2} p_{+}(\pmb{x}) p_{+}(\pmb{x}^{\prime}) + \pi_{-}^{2} p_{-}(\pmb{x}) p_{-}(\pmb{x}^{\prime}) + \pi_{+} \pi_{-} p_{+}(\pmb{x}) p_{-}(\pmb{x}^{\prime})$ . Then, an unbiased risk estimator for Pcomp classification is derived as follows:
$$
\widehat {R} _ {\mathrm {P c o m p}} (g) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\ell \left(g \left(\boldsymbol {x} _ {i}\right), + 1\right) + \ell \left(g \left(\boldsymbol {x} _ {i} ^ {\prime}\right), - 1\right) - \pi_ {+} \ell \left(g \left(\boldsymbol {x} _ {i}\right), - 1\right) - \pi_ {-} \ell \left(g \left(\boldsymbol {x} _ {i} ^ {\prime}\right), + 1\right)\right). \tag {5}
$$
In real-world scenarios, we may not only know one example is more likely to be positive than the other, but also know how much the difference in confidence is. Next, a novel weakly supervised learning setting named ConfDiff classification is introduced which can utilize such confidence difference.
2.4 Confidence-Difference (ConfDiff) Classification
In this subsection, the formal definition of confidence difference is given firstly. Then, we elaborate the data generation process of ConfDiff data.
Definition 1 (Confidence Difference). The confidence difference  $c(\pmb{x}, \pmb{x}^{\prime})$  between an unlabeled data pair  $(\pmb{x}, \pmb{x}^{\prime})$  is defined as
$$
c \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) = p \left(y ^ {\prime} = + 1 \mid \boldsymbol {x} ^ {\prime}\right) - p (y = + 1 \mid \boldsymbol {x}). \tag {6}
$$
As shown in the definition above, the confidence difference denotes the difference in the class posterior probabilities between the unlabeled data pair, which can measure how confident the pairwise comparison is. In ConfDiff classification, we are only given  $n$  unlabeled data pairs with confidence difference  $\mathcal{D} = \{((\pmb{x}_i,\pmb{x}_i'),c_i)\}_{i=1}^n$ . Here,  $c_i = c(\pmb{x}_i,\pmb{x}_i')$  is the confidence difference for the unlabeled data pair  $(\pmb{x}_i,\pmb{x}_i')$ . Furthermore, the unlabeled data pair  $(\pmb{x}_i,\pmb{x}_i')$  is assumed to be drawn from a probability density  $p(\pmb{x},\pmb{x}') = p(\pmb{x})p(\pmb{x}')$ . This indicates that  $\pmb{x}_i$  and  $\pmb{x}_i'$  are two i.i.d. instances sampled from  $p(\pmb{x})$ . It is worth noting that the confidence difference  $c_i$  will be positive if the second instance  $\pmb{x}_i'$  has a higher probability to be positive than the first instance  $\pmb{x}_i$ , and will be negative otherwise. During the data collection process, the labeler can first sample two unlabeled data independently from the marginal distribution  $p(\pmb{x})$ , then provide the confidence difference for them.
3 The Proposed Approach
In this section, we introduce our proposed approaches with theoretical guarantees. Besides, we show the influence of an inaccurate class prior probability and noisy confidence difference theoretically. Furthermore, we introduce a risk correction approach to improve the generalization performance.
3.1 Unbiased Risk Estimator
In this subsection, we show that the classification risk in Eq. (1) can be expressed with ConfDiff data in an equivalent way.
Theorem 1. The classification risk  $R(g)$  in Eq. (1) can be equivalently expressed as
$$
R _ {\mathrm {C D}} (g) = \mathbb {E} _ {p \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right)} \left[ \frac {1}{2} \left(\mathcal {L} \left(\boldsymbol {x}, \boldsymbol {x} ^ {\prime}\right) + \mathcal {L} \left(\boldsymbol {x} ^ {\prime}, \boldsymbol {x}\right)\right) \right], \tag {7}
$$
$$
\mathcal {L} (\boldsymbol {x}, \boldsymbol {x} ^ {\prime}) = (\pi_ {+} - c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x}), + 1) + (\pi_ {-} - c (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})) \ell (g (\boldsymbol {x} ^ {\prime}), - 1).
$$
Accordingly, we can derive an unbiased risk estimator for ConfDiff classification:
$$
\widehat {R} _ {\mathrm {C D}} (g) = \frac {1}{2 n} \sum_ {i = 1} ^ {n} \left(\mathcal {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}\right) + \mathcal {L} \left(\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i}\right)\right). \tag {8}
$$
Minimum-variance risk estimator. Actually, Eq. (8) is one of the candidates of the unbiased risk estimator. We introduce the following lemma:
Lemma 1. The following expression is also an unbiased risk estimator:
$$
\frac {1}{n} \sum_ {i = 1} ^ {n} \left(\alpha \mathcal {L} \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {i} ^ {\prime}\right) + (1 - \alpha) \mathcal {L} \left(\boldsymbol {x} _ {i} ^ {\prime}, \boldsymbol {x} _ {i}\right)\right), \tag {9}
$$
where  $\alpha \in [0,1]$  is an arbitrary weight.
Then, we introduce the following theorem:
Theorem 2. The unbiased risk estimator in Eq. (8) has the minimum variance among all the candidate unbiased risk estimators in the form of Eq. (9) w.r.t.  $\alpha \in [0,1]$ .
Theorem 2 indicates the variance minimality of the proposed unbiased risk estimator in Eq. (8), and we adopt this risk estimator in the following sections.
3.2 Estimation Error Bound
In this subsection, we elaborate the convergence property of the proposed risk estimator  $\widehat{R}_{\mathrm{CD}}(g)$  by giving an estimation error bound. Let  $\mathcal{G} = \{g:\mathcal{X}\mapsto \mathbb{R}\}$  denote the model class. It is assumed that there exists some constant  $C_g$  such that  $\sup_{g\in \mathcal{G}}\| g\|_{\infty}\leq C_g$  and some constant  $C_\ell$  such that  $\sup_{|z|\leq C_g}\ell (z,y)\leq C_\ell$ . We also assume that the binary loss function  $\ell (z,y)$  is Lipschitz continuous for  $z$  and  $y$  with a Lipschitz constant  $L_{\ell}$ . Let  $g^{*} = \arg \min_{g\in \mathcal{G}}R(g)$  denote the minimizer of the classification risk in Eq. (1) and  $\widehat{g}_{\mathrm{CD}} = \arg \min_{g\in \mathcal{G}}\widehat{R}_{\mathrm{CD}}(g)$  denote the minimizer of the unbiased risk estimator in Eq. (8). The following theorem can be derived:
Theorem 3. For any  $\delta >0$ , the following inequality holds with probability at least  $1 - \delta$ :
$$
R \left(\widehat {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right) \leq 8 L _ {\ell} \Re_ {n} (\mathcal {G}) + 4 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}}, \tag {10}
$$
where  $\Re_n(\mathcal{G})$  denotes the Rademacher complexity of  $\mathcal{G}$  for unlabeled data with size  $n$ .
From Theorem 3, we can observe that as  $n\to \infty$ ,  $R(\widehat{g}_{\mathrm{CD}})\rightarrow R(g^{*})$  because  $\mathcal{R}_n(\mathcal{G})\to 0$  for all parametric models with a bounded norm, such as deep neural networks trained with weight decay [58]. Furthermore, the estimation error bound converges in  $\mathcal{O}_p(1 / \sqrt{n})$ , where  $\mathcal{O}_p$  denotes the order in probability, which is the optimal parametric rate for empirical risk minimization without making additional assumptions [59].
3.3 Robustness of Risk Estimator
In the previous subsections, it was assumed that the class prior probability is known in advance. In addition, it was assumed that the ground-truth confidence difference of each unlabeled data pair is accessible. However, these assumptions can rarely be satisfied in real-world scenarios, since the collection of confidence difference is inevitably injected with noise. In this subsection, we theoretically analyze the influence of an inaccurate class prior probability and noisy confidence difference on the learning procedure. Later in Section 4.5, we will experimentally verify our theoretical findings.
Let  $\bar{D} = \{((\pmb{x}_i, \pmb{x}_i'), \bar{c}_i)\}_{i=1}^n$  denote  $n$  unlabeled data pairs with noisy confidence difference, where  $\bar{c}_i$  is generated by corrupting the ground-truth confidence difference  $c_i$  with noise. Besides, let  $\bar{\pi}_+$  denote the inaccurate class prior probability accessible to the learning algorithm. Furthermore, let  $\bar{R}_{\mathrm{CD}}(g)$  denote the empirical risk calculated based on the inaccurate class prior probability and noisy confidence difference. Let  $\bar{g}_{\mathrm{CD}} = \arg \min_{g \in \mathcal{G}} \bar{R}_{\mathrm{CD}}(g)$  denote the minimizer of  $\bar{R}_{\mathrm{CD}}(g)$ . Then the following theorem gives an estimation error bound:
Theorem 4. Based on the assumptions above, for any  $\delta >0$ , the following inequality holds with probability at least  $1 - \delta$ :
$$
R \left(\bar {g} _ {\mathrm {C D}}\right) - R \left(g ^ {*}\right) \leq 1 6 L _ {\ell} \Re_ {n} (\mathcal {G}) + 8 C _ {\ell} \sqrt {\frac {\ln 2 / \delta}{2 n}} + \frac {4 C _ {\ell} \sum_ {i = 1} ^ {n} \left| \bar {c} _ {i} - c _ {i} \right|}{n} + 4 C _ {\ell} | \bar {\pi} _ {+} - \pi_ {+} |. \tag {11}
$$
Theorem 4 indicates that the estimation error is bounded by twice the original bound in Theorem 3 with the mean absolute error of the noisy confidence difference and the inaccurate class prior probability. Furthermore, if  $\sum_{i=1}^{n} |\bar{c}_i - c_i|$  has a sublinear growth rate with high probability and the class prior probability is estimated consistently, the risk estimator can be even consistent. It elaborates the robustness of the proposed approach.
3.4 Risk Correction Approach
It is worth noting that the empirical risk in Eq. (8) may be negative due to negative terms, which is unreasonable because of the non-negative property of loss functions. This phenomenon will result in severe overfitting problems when complex models are adopted [19, 22, 43]. To circumvent this difficulty, we wrap the individual loss terms in Eq. (8) with risk correction functions proposed in Lu et al. [19], such as the rectified linear unit (ReLU) function  $f(z) = \max(0, z)$  and the absolute value function  $f(z) = |z|$ . In this way, the corrected risk estimator for ConfDiff classification can be
expressed as follows:
$$
\begin{array}{l} \widetilde {R} _ {\mathrm {C D}} (g) = \frac {1}{2 n} (f (\sum_ {i = 1} ^ {n} (\pi_ {+} - c _ {i}) \ell (g (\pmb {x} _ {i}), + 1)) + f (\sum_ {i = 1} ^ {n} (\pi_ {-} - c _ {i}) \ell (g (\pmb {x} _ {i} ^ {\prime}), - 1)) \\ + f \left(\sum_ {i = 1} ^ {n} \left(\pi_ {+} + c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i} ^ {\prime}\right), + 1\right)\right) + f \left(\sum_ {i = 1} ^ {n} \left(\pi_ {-} + c _ {i}\right) \ell \left(g \left(\boldsymbol {x} _ {i}\right), - 1\right)\right). \tag {12} \\ \end{array}
$$
Theoretical analysis. We assume that the risk correction function  $f(z)$  is Lipschitz continuous with Lipschitz constant  $L_{f}$ . For ease of notation, let  $\widehat{A}(g) = \sum_{i=1}^{n} (\pi_{+} - c_{i}) \ell(g(\boldsymbol{x}_{i}), +1)/2n$ ,  $\widehat{B}(g) = \sum_{i=1}^{n} (\pi_{-} - c_{i}) \ell(g(\boldsymbol{x}_{i}', -1)/2n$ ,  $\widehat{C}(g) = \sum_{i=1}^{n} (\pi_{+} + c_{i}) \ell(g(\boldsymbol{x}_{i}', +1)/2n$ , and  $\widehat{D}(g) = \sum_{i=1}^{n} (\pi_{-} + c_{i}) \ell(g(\boldsymbol{x}_{i}), -1)/2n$ . From Lemma 3 in Appendix A, the values of  $\mathbb{E}[\widehat{A}(g)]$ ,  $\mathbb{E}[\widehat{B}(g)]$ ,  $\mathbb{E}[\widehat{C}(g)]$ , and  $\mathbb{E}[\widehat{D}(g)]$  are non-negative. Therefore, we assume that there exist non-negative constants  $a, b, c$ , and  $d$  such that  $\mathbb{E}[\widehat{A}(g)] \geq a$ ,  $\mathbb{E}[\widehat{B}(g)] \geq b$ ,  $\mathbb{E}[\widehat{C}(g)] \geq c$ , and  $\mathbb{E}[\widehat{D}(g)] \geq d$ . Besides, let  $\widetilde{g}_{\mathrm{CD}} = \arg \min_{g \in \mathcal{G}} \widetilde{R}_{\mathrm{CD}}(g)$  denote the minimizer of  $\widetilde{R}_{\mathrm{CD}}(g)$ . Then, Theorem 5 is provided to elaborate the bias and consistency of  $\widetilde{R}_{\mathrm{CD}}(g)$ .
Theorem 5. Based on the assumptions above, the bias of the risk estimator  $\widetilde{R}_{\mathrm{CD}}(g)$  decays exponentially as  $n\to \infty$ :
$$
0 \leq \mathbb {E} [ \widetilde {R} _ {\mathrm {C D}} (g) ] - R (g) \leq 2 \left(L _ {f} + 1\right) C _ {\ell} \Delta , \tag {13}
$$