Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification
Authors: Yuanchao Dai
Ximing Li
Changchun Li
Venue: ICML 2025 poster (2025)
Keywords: weakly supervised classification
error bound
confidence-difference classification
Primary area: general_machine_learning->unsupervised_and_semisupervised_learning
TL;DR: We analyze the noisy problem in  confidence difference classification and propose a novel method with consistency regularization.

Abstract: Training a precise binary classifier with limited supervision in weakly supervised learning scenarios holds considerable research significance in practical settings. Leveraging pairwise unlabeled data with confidence differences has been demonstrated to outperform learning from pointwise unlabeled data. We theoretically analyze the various supervisory signals reflected by confidence differences in confidence difference (ConfDiff) classification and identify challenges arising from noisy signals when confidence differences are small. To address this, we partition the dataset into two subsets with distinct supervisory signals and propose a consistency regularization-based risk estimator to encourage similar outputs for similar instances, mitigating the impact of noisy supervision. We further derive and analyze its estimation error bounds theoretically. Extensive experiments on benchmark and UCI datasets demonstrate the effectiveness of our method. Additionally, to effectively capture the influence of real-world noise on the confidence difference, we artificially perturb the confidence difference distribution and demonstrate the robustness of our method under noisy conditions through comprehensive experiments.

Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification
Yuanchao Dai 12 Ximing Li Changchun Li
Abstract
significant research value for effectively leveraging limited available supervision information. Consequently, the field of weakly supervised learning has increasingly attracted at- tention from experts and scholars in recent years, leading to the emergence of many typical weakly supervised learning methods, such as multi-instance learning (Zhou et al., 2009;
Training a precise binary classifier with limited supervision in weakly supervised learning sce- narios holds considerable research significance in practical settings. Leveraging pairwise unlabeled data with confidence differences has been demon- strated to outperform learning from pointwise un- labeled data. We theoretically analyze the various supervisory signals reflected by confidence differ- ences in confidence difference (ConfDiff) classifi- cation and identify challenges arising from noisy signals when confidence differences are small. To address this, we partition the dataset into two subsets with distinct supervisory signals and pro- pose consistency regularization based risk es- timator to encourage similar outputs for similar instances, mitigating the impact of noisy supervi- sion. We further derive and analyze its estimation eиror bounds theoretically. Extensive experiments benchmark and UCI datasets demonstrate the effectiveness of our method. Additionally, ef- fectively capture the influence of real-world noise on the confidence difference. we artificially per- turb the confidence difference distribution and demonstrate the robustness our method under noisy conditions through comprehensive experi-
Zhang & Zhou, 2009; Wu al., 2018; Shi et al., 2020), positive and unlabeled (PU) learning (Kiryo et al., 2017;
Hammoudeh & Lowd, 2020; Zhao et al., 2022; Luo et al., 2021; Wang et al., 2023), and others.
A prevalent idea in weakly supervised classification involves maximizing the utilization of pointwise weakly supervised information (Feng et al., 2021), thereby prompting the devel- opment of various techniques based on soft labels (Nguyen et al., 2014; Xue & Hauskrecht, 2016), mixup (Zhang et al., 2017; Verma et al., 2019; Yun et al., 2019; Kim et al., 2020; Hendrycks et al., 2019; Li et al., 2021), and others. Never- theless, it is undeniable that annotating pointwise informa- tion in real-world classification problems is a complex and laborious task, further compounded by the personal biases of annotators which frequently exacerbate the probability of inaccuracies. In such scenarios, pairwise comparison information between data points may be more readily ob- tainable in real-world settings than pointwise information, and it often exhibits greater resistance to biases compared to pointwise cum--super information (Bao et al., 2018). For instance, in medical diagnosis, accurately determining whether a patient has a disease solely based on their pre- sented symptoms challenging. However, comparing the symptoms of this patient with those of others provides more accessible information and reduces the probability of misdi- agnosis. Extensive research has been conducted on pairwise analysis in numerous binary classification problems, leading to the development of risk minimization functions capable of inducing binary classifiers across various combinations of pairwise similarities, dissimilarities, and unlabeled data (Bao et al., 2018: 8; Shimada et al., 2021; Lu et al., 2018; 2020; Wang et al., 2024).
ments.
1. Introduction
Weakly supervised learning is an essential research field in machine learning, focusing on training accurate predictive models under conditions of low supervision or imprecise la- beling. Due to the difficulty of obtaining precise supervision in real-world scenarios, weakly supervised learning holds
'College of Computer Science and Technology, Jilin University, China Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, China. Correspondence to: Ximing Li <liximing86 @aaiiing88@gmail.comm gmail.com>.
In recent work, pairwise comparison (Pcomp) classification has shown that in tackling difficult point labeling tasks, peo- ple can more easily gather comparative information between two instances, constituting a form of weakly supervised in- formation (Feng al., 2021). However, in real-world appli-
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Confidence Difference Reflects Various Supervised Signals in Confidence-Differenee Classification
cation scenarios, individuals may not only distinguish which
We theoretically analyze various supervised signals re- flected by different confidence differences in ConfDiff classification. Additionally, we theoretically estimate the error bounds of our proposed method.
of two instances is more likely to be classified positive over the other but also gauge the extent of the disparity in their confidence levels regarding positivity. In light of this framework, Wang et al. introduce a new pairwise weakly su- pervised classification problem called confidence-differenee (ConfDiff) classífication, and propose the corresponding ConfDiff method (Wang et al., 2024). To establish confi- dence difference, the ConfDiff method first utilizes binary- labeled data to train a probability classifier. Subsequently, pairwise instances are fed into the classifier to generate pos- terior probabilities, from which confidence differences are computed based on the differences between these posterior probabilities. However, through analysis of the various su- pervised signals in the ConfDiff method, we identify that the method encourages models to predict opposite classes for pairwise instances, as supported by both experimental and theoretical perspectives. This prediction direction is valid when the confidence difference is large. However, when the confidence difference is small, the instances may belong to either the same or different classes, and such a predictive tendency may lead the model to incorrectly clas- sify instances from the same class as belonging to different classes, thereby introducing less reliable supervised signals.
We validate the effectiveness of our method through experiments by comparing it with existing baselines on datasets of varying scales. In addition, the robustness of our method further validated under the influence of artificially added noise.
2. Preliminaries
In this section, we brietly review the problem definitions of binary classification, binary classification with soft labels, and ConfDiff classification.
Formulation of binary classification Binary classifica- tion is a typical task in the field of supervised learning, where the goal is to induce a classifier that partitions the data space into two categories. Formally, let X' = IRd and y - +-1,+11 be the d-dimensional feature space and la- bel space, respectively. The dataset DBC Dc UDBc for binary classification consists of a positive dataset Dicc and negative dataset Di:
To handle this problem, in this paper, we concentrate on mitigating the impact of these less reliable supervised sig- nals when confidence differences are small. Specifically, we analyze the different supervised signals induced by varying confidence differences in the ConfDiff method ( (Wang et al., 2024). We find that pairwise instances with small confi- dence differences tend to introduce less reliable supervised signals, while those with larger confidence differences pro- vide more reliable supervision. Based on this observation, we propose a ConfDiff classification method that incorpo- rates consistency regularization. By partitioning the dataset based on the reliability of supervised signals, we introduce a consistency regularization term for the subset with less re- liable supervised signals, encouraging the model to produce similar outputs for pairwise instances with small confidence differences. Mcanwhile, for the subset with more reliable supervised signals, we preserve the benefit of these reliable supervised signals. Experimental results demonstrate that our method outperforms existing baselines in most cases and exhibits strong robustness even under artificial noise interference.
Dbc [(x] E ex vº +1)) xf iid.
p(xly +1), fid p(x - -1),
DBC = ((x] E X. = -1)]
xf
where np and n., denote the number of positive and negative instances, respectively. Let ι denote the positive class prior ply = +1) and é: R x y + R, be a binary loss function. Then binary classification induces a classifier g X - R from PBc by minimizing the following classification risk:
R(g) = =nBppp +1)]
+ (1 (1- 110=))))))0p===1((((0000=0 - -1)].
(I)
Formulation of binary classification with soft labels In binary classification, soft labels typically represent the confi- dence of each sample belonging to the positive class. More- over, several studies have shown that using soft labels rather than hard labels can more accurately reflect the data distri- bution (Szegedy et al., 2016), thus enhancing the accuracy of training binary classifiers. Formally, let qi denote the positive confidence of x,, the dataset DBC-soft for binary classification can be defined as follows:
In summary, this paper's key contributions can be outlined as follows:
DBC-soft 9:)7 x, iid. p(x))
We introduce a method for ConfDiff classification which aims to enhance the accuracy of weakly supervised classification by constructing risk esti- mator through Consistency Risk and Consistency Regularization (CRCR).
q p(y = + 1|x;),
where p(x) = Tp(xll = +1) + (1- t)p(x|y = -1).
Subsequently, the risk minimization objective function for
2

---

Confidence Difference Reflects Various Supervised Signals in Confidence--ifffrenee Classification
binary classification with soft labels can be reformulated into the following form:
pricei122
prize0.5 pore--33
ConDDifADD Accurccy
1.0
0.9
RBC-soft (g) - [0)0)) + (1 (){(((x),, 1)].
0.8
0.8
0.7
0.7
0.6
0.6
Formulation of confidence-differrnce (ConfDifff) classifi- cation Given that pairwise supervision is typically more accessible than pointwise supervision and it's feasible not only to determine which sample in an unlabeled data pair is more likely positive but also quantify the confidence dif- ference between them in practical scenarios, ConfDiff clas- sification precisely serves weakly supervised classifi- cation tailored to address this scenario. It specifically deals with weakly supervised classification problems where train- ing data comprises only pairwise unlabeled data and the confidence difference associated with each pair. Formally, let G = c(x,, x = p(y! = +1|x() ply: = +1,x,) be the confidence difference between pairwise unlabeled data (x,, x)) drawn from an independent and identically dis- tributed probability density x(x, ((x)) Let DCD denote pairwise dataset drawn from the pairwise unlabeled data and the confidence differences between them:
0.5
0.2
0.4
0.6
0.8 Scaling Factor
1.0
122
1.4
1.6
1.8
2,0
0.2
20
Scaling Factor
Figure I. The Accuracy for the binary classifier about different pro- portion of pairwise data with |c(x, x')] >> 0.5 two benehmark datasets MNIST (left) and CIFAR-10 10 (right). (The value of the x x-axis values min( s) denotes the proportion of pairwise instances with c(xx x') > 0.5.)
differences |e(x, x') are considered to provide stronger and more reliable supervised signals.
To explain this, we consider the general form of many com- monly used losses for the prediction function g(x) and target y (Zhang et al., 2021):
L= ((((())) ((g(x)))) (()())))))(((()))))))))))))))))) h(g(x)) - yg(())
DCD [((x, xx), x)) x; i.dd p(x), x! i.i.d. p(x).
for some function h)- (5)
Recent studies deal with the ConfDiff classification prob- lem in such challenging scenarios (Wang et al., 2024). They deduce an unbiased risk estimator for confidence-differeecce classification from 1 and train a binary classifier solely utilizing unlabeled data and confidence differences by mini- mizing The classification risk can be expressed as:
Substituting the form of the loss function from Bq.5 into Eq.4, then the classification risk of ConfDiff method can be rewritten as follows and the proof details are presented in the Appendix B:
1 RCD(g) = B p(xxx)
1
[ c(x,x
+1)
2
1
RCD(g) = B x') [[(x, x') + (xxxxx)),
(3)
+( 2
+ c(xxxx)
xxxx,x)))((((xx)) +1]
2
where L(x, x') = (TT - cxx, (g((x) + (1 -1-
+ xx(x,))
1
[
1
+ c(x,x))) -1)
2
c(x, x') (g)))(((x)) -1). Then 3 can be refined as follows:
1
+(
c(x,x')) -1]
RgD(g)
= Fxxxx) | c(x - x()((((())
2
+ 2 E pxxxx))
1
[1 (x(x) -
((x(x)))(x)))] s(x'))
(6)
+ (1 - - ((xxxx)))(((xx)) -1)
+ (IT + c(x x (g(x')++
where the first and second terms denote the pairwise in- stance (x,x') contrastive losses for positive and negative class predictions, respectively; while the third term serves as a regularization. We first analyze the critical compo- nents of the first term, where the weights 1 - c(x, x') and 1/ + c(x, x') determine the contributions of and x' the positive class prediction loss, respectively. These weights demonstrate natural balance as they sum to 1, with acting as a critical threshold that determines prediction di- rectionality. Since the weights 1 c(x, x') and 5 + c(xx x') necessarily fall on opposite sides of this threshold, they cre- ate a contrastive learning dynamic: when one instance is pushed toward stronger positive class prediction, the other
+ ग c(x x') (()(()) 1]]]-
(4)
3. The Proposed Method
In this section, we introduce the proposed ConfDiff method.
3.1. Analysis of the ConfDiff Method
In the ConfDiff method, pairwise instances with small confi- dence differences |e(x, x²) are prone to introducing less re- liable supervised signals, while those with larger confidence
3

---

Confidence Difference Reflects Various Supervised Signals in Confidence-Diffrrence Classification
is simultaneously pushed toward weaker positive class pre- diction (effectively toward the negative class). In other words, the first loss term ensures that and x' adjust their predictions in opposite directions, thereby emphasizing the predictive divergence of pairwise instances in the positive class predictions. Similarly, the second loss term forces the instances to diverge in their predictions for the negative class.
for these pairs, as they likely belong the same class. To achieve this, we introduce consistency regularization term that promotes alignment between the confidence differences and the model's outputs. Meanwhile, for DS, we retain the original strategy to preserve the effectiveness of the predictions driven by this strong guidance.
Our objective is to induce classifier g: R y from D by minimizing the expected risk with respect to the data distribution:
Referring the definition of c(x,x') x'), the magnitude of confidence difference |c(x, x') naturally reflects the relia- bility of the supervised signal between pairwise instances. When |c(x, x') is large, there is a strong indication that x and x' likely belong to different classes, as their posterior probability difference exceeds the classification threshold. Conversely, when Jc(x, x is small, the supervised sig- nal becomes less reliable; x and x' may belong to either the same class or different classes, as the posterior differ- ence is insufficient to provide precise classification signal. Therefore, we consider that pairwise instances with larger le(x, x') tend to provide more reliable supervised signals, while those with smaller lc(x, x'| may contribute less reli- able supervised signals in the existing ConfDiff method.
(C
E p
[TT c(x, (
+ +(1 - x - dx (())x)) -1)
+ +(T + c (()()))))
+(1 - x + cxx ( (g(x). -1)]
+o (xxx) log (c((x, x') + l19(() - g(xxx( l22.
To further validate this perspective. conduct experiments on MNIST and CIFAR-10 10 datasets by varying the propor- tion of the pairwise instances with |c(x,x))= > 0.5, as 0.5 corresponds to the natural classification decision boundary where the posterior probability difference equals the binary classification threshold. The empirical results (shown in Figure 1) illustrate the accuracy under different proportions of the pairwise instances with (e(x, x') > 0.5. We observe positive correlation between classification accuracy and the proportion value. Notably, when the proportion is O, the accuracy is approximately 0.5, indicating that the classifier performs nearly at random. These findings demonstrate that the pairwise instances with Jc(x, x') >> 0.5 provide stronger and more reliable supervised signals and dominate the contribution to RCD-
where a denotes the parameter of the consistency regulariza- tion term, and ē = 1.1 is a smoothing parameter introduced to mitigate numerical issues when |c(x, x')l approaches or equals zero Let |DS| nı and |DO| = n2. Then the empirical risk estimator can be expressed as follows:
RC
Σ = 2m i=1
(() +1)
+(1 -1-13 π -x)e(((()) -1) + (TT + +))((((()))
+ (1 - +a)))(((xx))) -1))
a
1 log (lcal +e) ( - o(x()ll))
(8)
n2
3.2. CRCR Method
3.3. Analysis of Error Bound
Based on the discussion in Section 3.1, it has been demon- strated that pairwise instances with larger |c(x, x') tend to provide more reliable supervised signals, while those with smaller [c(x,x') x') provide less reliable supervised signals. To address this issue, we propose setting threshold 0 to par- tition the dataset into two subsets: one containing pairwise (more reliable supervised sig- nals, denoted as D and the other with |c(x, x') < (less reliable supervised signals, denoted as DC). For DC. we aim to provide additional guidance to direct the predictions of pairwise instances toward more appropriate outcomes. Specifically, for pairwise instances with smaller |c(x, x') x,)l, we encourage the model to produce more similar outputs
Assuming there exists a constant Cg such that supgec |G] < Cg. and another constant Ce such that sup|=/gc, E(2, y) S Cz. Additionally, we presume the binary loss function E(z, y) to be Lipschitz continuous with respect to both 2 and y, and to have a Lipschitz constant denoted by T.g. Rn, (9) and Rng (9) denote the Rademacher complexity of unlabeled data G with size nı and nı2 respectively.
instances with le(x (x, x'| >
Theorem 3.1. Let g" = arg min mingegRiog R(g) be the mini- mizer of the true classification risk in Eq. I and ÒCRCR = arg min RCRCR(9) denotes the minimizer of the risk form in Eq.8. Then for any 8 > O, we believe that the
4

---

Confidence Difference Reflects Various Supervised Signals in Confidence-Differreee Classification
sollowing expression holds with a probability least 6:
Cun et al., 1998), Kuzushiji-MNIST (K-MNIST) (Clanuwat et al., 2018), Fashion-MNIST (F-MNISTT(Xiao et al., 2017)
R(ğeneR) R(g") S (8 + 4,3) L. (g)
and CIFAR-IO (Krizhevsky. Technical report, University of Toronto, 2009). Additionally, experiments are conducted on two UCI datasets ', including Optdigits and Pendigits. Since these datasets contain multiple classes, we categorize the class labels into positive and negative classes, effectively transforming them into binary classification datasets. Fur- thermore, for each dataset, instances to artificially add noise, where the noise ratio m is varied over [0, 50, 75, 100). As result, in our experiments, we generate 24 synthetic datasets in total.
6a
ln(2/0)
+ log(e)
Rng (9) + 2Cr/
2m1
Due to the space limitation, the proof details are presented in Appendix A. As R1, R22 - oc, the Rademacher complex- ities Ri, (g) and Rng (G) decrease to zero, and the third term involving // also diminishes. Furthermore, the convergence rates of (R,(9)) and Rn, are ((1/mm) and (z2) while the third term's 's rate is dominated by 0(1/ /ni). Consequently, as n + xc, R((CRCR) + R(g"), ). and the overall convergence rate is characterized by O (max(11 (T, 1/ /m))).
randomly select m%
Moreover, we choose different models as backbones based on the varying feature dimensions of each dataset. Specif- ically, for MNIST, K-MNIST and F-MNIST, we use 3- layer multilayer perceptron (MLP) with three hidden layers of width 300 equipped with the ReLU (Nair & Hinton, 2010) activation function and batch normalization (loffe & Szegedy, 2015). For CIFAR-I0, we train a ResNet-34 model (He et al., 2016) as the backbone. For all UCI datasets, we use linear model for training. The detailed information for each dataset is presented in Table 1.
3.4. Empirical Risk Correction
can potentially lead to severe overfitting problems when the empirical risk becomes negative due to the application of an unbiased risk estimator. Fortunately, risk correction functions f()) can be utilized to mitigate this issue. Ex- amples include the absolute value function or the rectified linear unit (ReL.)) function. Consequently, the corrected risk estimator can be expressed as follows:
Baseline methods We employ seven state-of-the-art algo- rithms for comparison, including four Pcomp methods (i.e., Pcomp-Teacher, Pcomp-ABS, Pcomp-ReLU and Pcomp- Unbiased) and three ConfDiff methods (i.e., ConfDiff-ABS, ConfDiff- JUU and Unbiisddd. Details of base- lines are described as follows:
Rener (g)
TT- c))(((()))
2n1
+ 2m1
11 ग - ()(()
+
(T + +++))( (g(())
Pointwise Binary Classification with Pairwise Confi- dence Comparisons (Pcomp) (Feng et al., 2021): A weakly supervised learning method that trains a binary classifier using pairwise comparison data, composed of unlabeled data pairs where one is more likely to be positive, instead of using pointwise data. Peomp comprises four versions: Peomp- Teacher, Pcomp-ABS, Pcomp-Rel.U) and Pcomp-Unbiased. We use the code provided by its authors 2.
2m1
+ 2m1
ग + +c)
1
n2
+ a _
f
log (cc]+ + [1(x(x)) - -(x))l2))
122
(10)
Additionally, in our experiments, we report results for two variants of our method that utilizes the absolute value risk correction function (CRCR-ABS) and ReLU risk correction function CCRCR-ReLU
Binary Classification with Confidence Difference (ConfDiff) (Wang et al., 2024): A weakly supervised learning method that trains a binary classifier using pairwise comparison data, which consists of pairwise unlabeled data where the difference in the probabili- ties of being positive (confidence difference) is known. ConfDiff comprises three versions: ConfDiff-ABS. ConiDift-ReLU, and ConiDiff-Unbiasee We utilize the publicly available code online?.
4. Experiments
In this section, we empirically evaluate the proposed CRCR method.
4.1. Experimental Settings
hhtp://archive.iesuuu .es.uci.eduu
Datasets To thoroughly evaluate our method, we employ four popular benchmark datasets, including MNIST (Le-
hhtps://fengg1995 htgp:////gg..SSSStiiiuoooooeeeae http://gghhacccowwwaaaiiilllllln
5

---

Confidence Difference Reflects Various Supervised Signals in Confidence--ifffrenee Classification
Table I. Detailed characteristies of datasets.
Dataset
#Instance
#Trainset
#Testset
#Fea
Pos Class
Neg Class
Backbone
MNIST
70,000
15,000
5,000
28 x 28
0,2,4,6,8
1,3,5,799
3-layer MLP
F-MNIST
70,000
15,000
5,000
x 28
0,2,4,6,8
1,3,5,7,9
3-layer MLP
K-MNIST
70,000
15,000
5,000
28 x 28
0,2,4.6.8
1,3,5,7,9
3-layer MLP
CIFAR-10
60,000
10,000
5,000
3 x 32 x 32
2,3,4,5,6,7
0,1,8,9
ResNet-34
Optdigits
5,620
1,200
1,125
62
0,2,4,6,8
1,3,5,7,9
Linear
Pendigits 10,992
2,500
2,199
16
0,2,4.6,8
1,3,5,7,9
Linear
Implementation details For each comparison method under every experimental configuration, we execute the code five times, employing the logistic loss function and Adam optimizer consistently. Specifically, during the train- ing phase, each run is independently performed for 200 epochs with a batch size of 256. In balanced scenarios (i.e.. т = 0.5), the learning rate is set to 10-1 across all datasets, with weight decay parameters set to 10-5 5 for MNIST, K- MNIST, F-MNIST, and CIFAR-IO, 10-1 for Optdigits, and 10-3 for Pendigits. In imbalanced scenarios (ie.., 0.2), the learning rate is set to 10-4 for MNIST and K-MNIST, and 10-3 for the remaining datasets, with weight decay parameters set to 10-4 for K-MNIST and Optdigits, and 10 5 for the remaining datasets. During the pretraining phase, each run independently executed for 20 epochs with batch size 256. The learning rate and weight decay remain consistent with those in the training phase.
where d(x,) represents the estimated probability density [Xi (x--x,)* is the standard Gaussian kemel function. The parameter h denotes the kernel bandwidth controlling the degree of smoothing and is adaptively set based on the standard deviation of the probability distributions.
function at instance x; and exp
2h
Next, we identify instances with densities below threshold outliers, where is also adaptively determined based on different probability density distributions. In our work, is set at the 2nd percentile of the probability density to avoid excessive filtering. For the remaining non-outlier instances, we rescale their posterior probabilitics to ensure a more uniform distribution within the range [0. 1]:
p(y: = +1(x,)
{ Scaling (p(y: = +1(x)), if d(x;) so p(y: = +1|x;),
otherwise
4.2. Construction of the confidence differences
(12)
In this subsection, we present a method for generating con- fidence differences validate the robustness of our method under noisy conditions.
where Scaling (-) denotes a scaling function as:
log(p: 8), log(1 p: + 0), otherwise (13)
if Pt S 0.5
The confidence differences construction method. The ConfDiff method generates class posterior probabilities us- ing a logistic regression-baset probabilistic classifier trained on labeled data and calculates the confidence difference ac- cording its definition. Although this generation method facilitates comprehensive experimental analysis, it fails to accurately reflect the posterior probability distribution de- rived from manual annotations in real-world scenarios. To address this limitation, we propose an enhanced method that incorporates a posterior probability construction method based on outlier detection. This integration enables us to achieve more uniform and realistic distribution while main- taining the fundamental definition of confidence differences.
Scaling (p:)
where 9 = e-10 10 is smoothing parameter. Then, the confi- dence difference can be calculated according to its definition c(x, x() p(y/ = +1 - p(y: = +1(x,).
Artificially add noise. One straightforward method is to add noise directly to c. However, it overlooks the intrin- sic logic behind the original construction of c. To better simulate real-world conditions, we focus observing how noise impacts the posterior probability distribution, thereby influencing indirectly. Specifically, we introduce noise to the posterior probabilities generated by the probabilistic classifier, which consequently adds noise to e. In the real world, individuals tend to exhibit smaller judgment biases towards more similar pairwise instances, while generating larger biases towards instances with lower similarity. There- fore, White Gaussian Noise (WGN) is introduced into the posterior probabilities p(y: = +1[x,) and p(1/ = +1|x() provided by the probabilistic classifier for the pairwise in- stance (x,, x(). Subsequently, the noisy posterior proba- bilities are used to generate the confidence difference, i.e.,
Our method consists of three main components: probabil- ity density estimation, outlier identification, and posterior probability rescaling.
First, we apply a Gaussian kernel-based probability density estimation method to the discrete posterior probabilities:
1
(xi
(xx=x,)²
xx)) )))
d(xx) = nhy 2T j=1
exp
(11)
2h2
6

---

Confidence Difference Reflects Various Supervised Signals in Confidence--ifferreee Classification
Table 2. Classification accuracy of each comparing method on six datasets (mean=std) when 0.5, where the hest performance
shown in boldface.
m
Method
MNIST
K-MNIST
F-MNIST
CIFAR-I0
Pendigiis
Opudigits
Pcomp-Unbiased
0.815=0.007
0,568+0.087
0.813+0.066
0.752+0.005
0.775+0.018
0.795+0.020
Poomp-ReLU
0.719=0.100 108
0.692+0.012
1.614#0.132
0.794+0.009
0,746+0.014
0.766+00033
Pcomp-ABS
0.830=0.005
0.727=0.015
0.83740.010
0.828+0.006
0.645+0.059
0.722+0.027
Promp-Teacher
0.882=0.024
0.708+0.000 008
0.887+0.012
0.81240.010
0.496+0.016
0. 0,507+0.067
ContDiff-U Unbiased
0.725=0.072
0.376#0.029
0.771=00085
0.84820.014
0.61550.071
0.7
0
ConfDiff-ReLU 0.929=0.003
0.771=0.025
0.912:0.020
0.848:0.014
0.675#0.071
0.79910.023
ConfDiff-ABS
0.944=0.003
0.825#0.011
0.952+0.004
0.848+0.014
0.67540.071
0.799+0.023
CRCR-Unbinsed
0.777=0.034
0.769=0.004
0.921=000099
0.86950.009
0,75680.006
0882360.033
CRCR ReLU
0.919=0.019
0.68500.080
0.925=0.017
0.86920.009
0.753+0.007
0.823=00.033
CRCR-ABS
0.962=0.006
0.84840.013
0,955=0.002
0.869 0.09
0.75340.009
0.823+0.023
Pcomp-Unbiased Unbiased
0.81420.050
0.60620.086
0.85520.061
0.733a0.010
0.760x0.020
0.793x0.022
Poomp-ReLU
0.849=0.008
0.722#0.003
0.833=0.063
0.810=0.000
0.756+0.036
0,772+0.017
Pcomp-ABS
0.853:0.016
0.73020.013
0.87620.015
0.83+#0.005
0.67620.069
0.736+0:017
Teache
0.89820.019
0.723a0.018
0.907
0.81240.007
0.495a0.017
0.503400068
ConfDiffUUnbiasdd 0646
0.602#0.021
0.794#0.054
0,833+0.013
0,675+0.073
0,79240.00 021
50
ConfDiff-ReLU 0.933=0.002
0.766=0.020
0.933=0.012
0.836+0.014
0.675
0.792+0.02 021
ConiDulIAAB 0.937.0.004
0.81920.007
0.95320.007
0.83420.013
0.675a00.07
0.79210.021
CRCR-Unbinsed
0.845=0.043
0.779#0.008
0.928=0.001
0.859=0.000
0.759+0.029
0.821#0.022
CRCR-ReLU
0,923=0.023
0.793=0.019
0.936:0.00 007
0.86020.005
0.757+0.030
0.821=0.022
CRCR-ABS
0.961=0.005
0.851=0.010
0,956=00.005
0.860=00003
0,762+0.033
0.821+0.022
Cunup--Ubbasedd
0.84920.010
0.59610.086
0.832a0.129
0.71620,006
0,754x0.028
0.794g0.021
Peomp ReL.U
0.858:0.006
0.728
D. 8800
0.820#0.008
0,743+0.038
0.783+0.018
Poomp-ABS
0.86520.008
0.73420.017
0.87420.011
836+000003
0.688#0.00 060
0.743+0.020
Pcomp-Teacher Teacher
0.90820.010
0.73520.013
0.92020.018
0.81340.008
0.495
0.50110.064
ConfDiff- Unbiased
0.620=0.084
0.560+0:025
0.650#0.051
0.84420.008
0.674
0,79540:018
75
ConfDiff-ReLU
0.922=0.019
0.77820.008
0.931=0.015
0.843=0.009
0,674+0.073
0.795+0.018
ConDDill-ABS
0,93320.006
0.81720.009
0.954220.004
0,84420,009
0,67410.073
0.795+0.018
CRCR Unbiased
0.797.0.075
0.791
0.926:00DD1
L85820.003
0.723x0.035
0.819100002
CRCR-ReLU
0.938:0.006
0.792=0.010
0.94220.005
0.858 000
0,721
0.819:0.022
CRCR-ABS
0.962:0.003
0.851=0.006
0.95920.001
0.85810.008
0.75620.009
0.81910.022
Pcomp-Unbiased Unbiased
0.83220.051
0.63120.079
0.89720.013
0.708
0.735x0.024
0.79650.015
Peomp- ReLU
0.862#0.015
0.726:0.012
0.883=0.017
0.827+0.00M
0.725+0.035
0.78740.019
Pcomp-ABS
0.865=0.014
0.735#0.009
0.886=00.099
4.066000066
0.688+0.059
0,766+0.022
Peomp-Teacher
0,91420.011
0.73820.020
0.92110.011
0,812+0,010
0,49510.018
0.49910.070
ConIDiff L L'bbiased
0.631=0.056
0.54820.022
0.573
0.835200012
0.66980.070
0.79110.021
100
ConfDiff ReI.U
0.920=0.014
0.769=0.008
0.923=0.032
0.834#0.012
0.66910.070
0.791+0.021
ContDilf-ABS
0.93420.006
0.81220.004
0.95320.005
0.835+0.012
0.66910.070
0,79110.021
CRCR- Unbiased
0.86020.081
0.80420:009
0.91020.030
0.8511000077
0.75120.008
0.815:0.019
CRCR ReLU
0.979=0.006
0.797=0.006
0.941=0.006
000077
0.752+0.008
0.815+0.019
CRCR-ABS
0.96020.002
0,85620.008
0.960=0.002
0.851=0.0077
0.752:0.008
0.815:0.019
2 = ē(x; x1) = ply1- = +1 +1x{) - ply = +1(x)), where
Pcomp with accuracy improvements ranging from 0.02 to 0.341 across different datasets and surpasses ConfDiff with improvements ranging from 0.01 to 0.387, as observed from a baseline perspective. CRCR-ABS outperforms nearly all baselines, with the only observed exception being the results of Pcomp-Unbiased on the Pendigits dataset when artifi- cial noise is added. This exception may be due to the fact that the Poomp method leverages only the information that one instance is more likely to be positive than another, with- out requiring knowledge of the exact difference between them. When no artificial noise is added, Pcomp's posterior probability reconstruction function preserves the monotonic increasing relationship of the posterior probabilities, with- out altering the relative likelihood of positivity between instances. Moreover, compared to Peomp and ConfDiff, our method demonstrates increasingly stable and consistent accuracy as the noise ratio increases, with notable improve- ments in both accuracy and standard deviation, especially when the noise ratio reaches 100%. This results indicates its
P(V- +1 +1xx) = p(u/- = +1 + 41 ([~N(0,07)
p(y: +1 +1/x)) = p(y: = x + Ce. - N(0,00)
(14)
where Si and So represent the noise offsets which follow a standard Gaussian distribution NO0 o²). In our experi- ments, we set = 1/3.
4.3. Result Analysis
Table 2 and Table present the results of all baselines on four benchmark datasets and two UCI datasets for class- balanced (i.e., prior = 0.5) and class-imbalanced scenarios (i.e.. prior 0.2), respectively. Overall, our method per- forms nearly optimally across all scenarios, consistently achieving nearly the best results using the ABS risk correc- tion function.
In scenarios with balanced classes, our method outperforms
7

---

Confidence Difference Reflects Various Supervised Signals in Confidence--ifferenee Classification
Table 3. Classification accuracy of each comparing method on six datasets (mean=std) when 0.2, where the hest performance
shown in boldface.
m
Method
MNIST
K K-MNIST
F-MNIST
CIFAR-10
Pendigiis
Opudigits
Pcomp-Unbiased
0,744=0.037
0,555=0.076
0.748=0.047
0.634+0.021
0.82040.022 025
0.813+0.024
Poomp-ReLU
0.800=0.000
0. 0,800=0.000
0.800#0.000
0,802+0.003
0.819#0.020
0.816+0.007
Pcomp-ABS 0.80420.009
0.800000000
0.80140.001
0.833+0.004
0.797a60023
0.805+0.006
Promp-Teacher
0.788=0.074
0.695+0.044 046
0.833000266
0.813+0.020
0.482+0.212
0.684+0.099
ConDDiffUnbiasse
0.745=0.033
0.622#0.077
0.724
0.81220.004
0.79/30.028
0.830H0.016
0
ConfDiff-ReLU 0.800=0.000
0. 8.000000000
0.84620.00 064
0,80000.000
0.797+0.028
0.830+0:016
ConfDiff-ABS 0.11==0015
0.44+00.014
0.940=0.010
0.800+0.001
0.79740.028
0,830+0.016
CRCR-Unbinsed
0816600043
0.59780.055
O 0.66000009
0.841=0:012
0,823H00005
0.388000177
CRCR ReLU
0.929=0.055
0.814=0.031
D.9 0.930=0.049
0.801 #0.001
0.817+0.012
0,830+0.008
CRCR-ABS
0.916=0.022
0.85620.006
0.922+0.007
0.81240.017
0.784+0.024
0.825+0.005
Pcomp-Unbiased Unbiased
0.742=0.015
0.54720.058
0.76820.070
0.623x0.017
0.81840.025
0.81020.027
Poomp-ReLUU
0,800=0.000
0.801=0.002
0000000000
0.801 003
0.80600025
0.821+0.007
Pcomp-ABS
0.824:0.029
0. 0.80000.000
0.80920.000 006
0.833#0.006
0.801#0.030
0.81110:010
Teacher
0.82220.061
0.70720.062
0.90240.014
0.797+0.033
848340.211
0.68210.096
ConfDiff-Unbiased 0.694=0.030
0.640=0.043
00.11
0,805+0.006
0.79700.229
0.834+0.015
50
ConfDiff-ReLU 0.800=0.000
0,800=0.000
0.046
0.800#0.001
0.797 029
0.834+0.015
ConiDulIAAB 0.89120.025
0.81820.010
0.93820.014
00.012
0.79740.022
0.834g0.015
CRCR-Unbinsed
0.794=0.043
0.623+0.079
0.880=0.016
0.789+0.035
0.795+0.025
0.843+0.023
CRCR-ReLU
0,908=0.063
0.81520.015
0,936 043
0.811
0.808+0.021
0.838+0.014
CRCR-ABS
0.91620.013
0.830=0.029
0.950=0.011
0.850=000117
0.822#0.019
0.835+0.011
Scomp-Unbiasse
0.75320.031
0.53520.048
0.775
0.6162000388
0.81740.017
0.81310.030
Peomp ReL.U
0.804=0.009
0000
D. 0.80000.000
0.805
0.822+0.020
0,827+0.008 008
Pcomp-ABS
0.863=0.014
0.80020.000
0.8 0.82200.016
0.832=0.005
0.803+0.038
0.813+0.000 009
Pcomp-Teacher Teacher
0.840=0.061
0.71420.055
0.90820.019
0.79340.044
0.482x0.211
0.68010.096
ConfDiff- Unbiased
0.70420.058
0.63020:026
0.745
0.80420.000
0.79620.031
0.830+0:016
75
ConfDiff-ReLU
0.800=0.000
0. 0.800000000
0.80220.000 000
0,000000000
O. 0.79640.031
0.830+0.016
ConDDifl-ABS
0,86210.030
0. 8086000000
0.921
0.80010.001
0,79620.031
0.83060.016
CRCR Unbiased
0.792=0.047
0.64010.028
0.86120.033
0,772=0.020
0.811200030
0.83610.002
CRCR-ReLU
0.901=0.055
0.817=0.024
0.80 0.801
0.828=0.024
0.827=0.008
1.836+0.017
CRCR-ABS
0.914=0.008
0.819=0.022
0.947=0.006
0.85340.004
0.81910.011
0.83910.012
Pcomp-Unbiased Unbiased
0.752a0.021
0.54020.069
0.83-120.034
0.64320.053
0.80510.024
0.8171000027
Peomp- ReLU
0.845=0.040
0.80810.010
0.814:0.019
0.80640.005
0.80840.020
0.834+0.008
Pcomp-ABS
0.871=0.006
0.001
0.844:0.015
0.835+0.003
0,80340.029
0.823+0.012
TeacheTe
0.86920.068
0.71120.062
0.92240.011
0,787=0.033
0,48260.211
0.68110.096
ConIDiff L Libbiased
0.772=00056
0.69320.028
0.74820. 101
0.102000077
0,79680.022 028
0.83180.017
100
ConfDiff ReI.U
0.800=0.000
0.800=0.000
0.800=0.000
0.800+0.001
0.796+0.028
0.831+0.016
ContDilf-ABS
0.81420.006
0.80120.002
0.87020.043
0.001
0.79620.028
0,83110.016
CRCR-Unbiased
0.790:0.036
0.63920.059
0.83820.056
0.780200014
0.797a0.018
0,83710:006
CRCR ReLU
0.905=0.059
0. 0.80810.007
0.903=0.039
0.800+0.001
0,800=0.014
0.838+0.020
CRCR-ABS
0.92620.010
0,828=0.025
0.958=0.009
0.841=00000
0.81020.006
0.83950.019
ability to produce more competitive results in the presence of artificial noise interference.
clearly demonstrate that its training challenges on complex and noisy datasets often lead to a notable decline in perfor- mance. This findings further underscore the effectiveness of CRCR-ABS in maintaining robust performance when dealing with complex datasets.
In scenarios with imbalanced classes, Pcomp-ReLU and ConfDiff-ReLL tend to exhibit random outcomes when confronted with imbalanced data augmented with artificial noise. This phenomenon may be attributed to the introduced noise, which significantly increases the likelihood of predic- tions where one instance in a pair is incorrectly predicted to be more likely positive than the other, contrary to the actual scenario. Such contradictions become significantly more pronounced as class imbalance and noise ratio in- crease. Compared to these methods, our method shows advantages in both accuracy mean and variance. From the dataset perspective, CRCR-ABS significantly outper- forms other methods on the MNIST, K-MNIST, F-MNIST, and CIFAR-10 datasets in the presence of artificial noise, while maintaining strong competitiveness on the Pendigits and Optdigits datasets. ERCR-Unbiased shows promising results without artificial noise; however, the experiments
Such contradictions become significantly more pronounced as class imbalance and noise ratio increase. Compared to these methods, our approach shows advantages in both accuracy mean and variance. From the dataset perspective, ERCR_ABS significantly outperforms other methods on the MNIST, K-MNIST, F-MNIST, and CIFAR-10 datasets in the presence of artificial noise, while maintaining strong competitiveness on the Pendigits and Optdigits datasets. CRCR Unbiased shows promising results without artificial noise; however, the experiments elearly demonstrate that its training challenges on complex and noisy datasets often lead to a notable decline in performance. These findings further underscore the effectiveness of CRCR ABS in maintaining robust performance when dealing with complex datasets.
8

---

Confidence Difference Reflects Various Supervised Signals in Confidence-Differenne Classification
Specifically, setting a 0 and 0 represent versions without consistency strategy and without subset segmenta- tion strategy, respectively. The experimental results, pre- sented in Figure 2, demonstrate that our proposed subset segmentation strategy and consistency term contribute to performance improvement to some extent in the context of confidence difference classification under artificially added noise.
Accurncy
0.92
0.54
0.60
MNIST
M-MISST
0.76
MNIST
M-NNNST
0.50
CIFARR11
10a
10
10"
10a
10a 10
10
10
10
10' 10 10'
Accuracy
Accuracy
MNNSST K-MNIST F-MNIST
MNIST K-MNIST F-MNIST
5. Conclusion
0.5
00 0.1 0.2
6.3
0.5 R5
0.1
00 0.3 0.2 0.3
6.8
In this paper, we propose novel ConIDiff classification method based on consistency risk and consistency regular- ization to address the challenge of noisy supervised signals in ConfDiff classification. We conduct a theoretical analy- sis of various supervised signals associated with different confidence differences. Based on this analysis, the ConfDiff' dataset is partitioned into two subsets according to the re- liability of the supervised information. For the subset with more reliable supervision, we employ consistency risk to preserve precise supervised information. Conversely, for the subset with less reliable supervision, we leverage con- sistency regularization to mitigate the impact of erroneous predictions. Extensive experimental results demonstrate that CRCR outperforms state-of-the-ar baselines and exhibits strong robustness, even when artificial noise is introduced.
Figure 2. Sensitivity analysis of parameters a (top) and 0 (bottom) on four benchmark datasets when 0.5 (left) and 0.2 (right).
4.4. Parameter Sensitivity
In this subsection, we conduct experiments with different thresholds 0 for partitioning subsets and the parameter for the consistency term, and the results are shown in Figure 2.
To evaluate the sensitiv- ity of the threshold 0, we vary its value within the range on four distinct benchmark datasets (i.e.,
About different threshold
{0.1, ......................... ......... } MNIST, K-MNIST, F-MNIST and CIFAR- 10). The results reveal that the accuracy peaks for all datasets when 0 = 0.4 with π = 0.5, and when 0 = 0.2 or 0.3 with ग = 0.2. This observation may be attributed to the distribution of confi- dence differences resembling a normal distribution. A low threshold results in numerous inaccurate predictions within the subset D utilized for risk consistency, while high threshold leads to scarcity of samples within DS, thus di- minishing the available supervisory information. Therefore, we empirically recommend setting the threshold at 0 = 0.4 when π 0.5, and (0.2, 0.3) when π = 0.2.
Acknowledgements
We would like acknowledge support for this project from the National Science and Technology Major Project (No.2021ZD01 12500), and the National Natural Science Foundation of China (No.62276113).
Impact Statement
The confidence difference classification proposed in this paper has the potential to significantly improve decision accuracy in real-world applications. It addresses potential noise impacts present in real-world data and holds substan- tial practical significance, especially in weakly supervised domains. This method is applicable to various fields, in- cluding medical diagnosis, rehabilitation assessment, and financial risk management.
About different parameter a To assess the sensitivity of the parameter a, we vary its values across the range {10%)i = -3.... +33 on four benchmark datasets. Our analysis reveals that a shows increased sensitivity on the larger-scale CIFAR-10 10 dataset when æ 0.5, while main- taining relatively stable performance the smaller-scal scale datasets. Moreover, leads to consistent trend in accu- racy variation across the four datasets when π = 0.2. No- tably, it achieves relatively optimal results when a = 1 with π = 0.5, and a = 10 with π = 0.2. Thus, we recommend setting a = 1 or 10 in experimental setups.
However, it is important acknowledge that the confidence differences used in our method may be affected by inherent data biases in real-world scenarios. Furthermore, while we demonstrate the effectiveness of our method in weakly super- vised settings, there remains a risk of excessive dependence on algorithms for decision decision-making. which could potentially overlooking the cultivation individual decision decision-making making capabilities and autonomy.
4.5. Ablation Study
In this subsection, we conduct ablation studies on vari- ous strategies by setting corresponding parameters to zero.

---

Confidence Difference Reflects Various Supervised Signals in Confidence-Difference Classification
References
Li, C., Li, X., Feng, L., and Ouyang, J. Who your right mixup partner in positive and unlabeled learning. In International Conference on Learning Representations, 2021.
Bao, H., Niu, G., and Sugiyama, M. Classification from pairwise similarity and unlabeled data. In International Conference on Machine Learning, pp. 452-461, 2018.
Lu, N., Niu. G., Menon, A. K., and Sugiyama, M. On the minimal supervision for training any binary classifier from only unlabeled data. arXiv preprint arXiv: /808. 10585, 2018.
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A., Yamamoto, K., and Ha, D. Deep learning for classical japanese literature. arXiv preprint arXiv:1812.01718, 2018.
Lu, N., Zhang, T., Niu, G., and Sugiyama, M. Mitigating overfitting in supervised classification from two unla- beled datasets: A consistent risk correction approach. In International Conference on Artificial Intelligence and Statistics, pp. 1115-1125, 2020.
Cortes, C. C., Mansour, Y., and Mohri, M. Learning bounds for importance weighting. In Neural Information Pro- cessing Systems, pp. 442-450, 2010.
Feng, L., Shu, S., Lu, N., Han, B., Xu, M., Niu, G., An, B., and Sugiyama, M. Pointwise binary classification with pairwise confidence comparisons. In International Conference on Machine Learning, pp. 3252-3262. 2021.
Luo, C., Zhao, P., Chen, C., Qiao, B., Du, C., Zhang, H., Wu, W., Cai, S., He, B., Rajmohan, S., and Lin, Q. PULNS: positive-unlabeled learning with effective negative sam- ple selector. In AAAI Conference on Artificial Intelligence, pp. 8784-8792, 2021.
Hammoudeh, Z. and Lowd, D. Learning from positive and unlabeled data with arbitrary positive shift. In Neural Information Processing Systems, pp. 13088-13099, 2020.
Mohri, M., Rostamizadeh, A., and Talwalkar, A. Founda- tions of Machine Learning. Adaptive computation and machine learning. MIT Press, 2012.
He, K., Zhang, X.. Ren, S., and Sun, J. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770- 778, 2016.
Nair, V. and Hinton, Rectified linear units improve re- stricted boltzmann machines. In International Conference on Machine Learning, pp. 807-814, 2010.
Hendryeks, D., MU, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint arXiv: 111::191.000781 2019.
Nguyen, Q. Valizadegan, H., and Hauskrecht, M. Learning classification models with soft-label information. Journal of the American Medical Informatics Association, 21(3): 501-508, 2014.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp. 448-456 2015.
Shi, X., Xing, F., Xie, Y., Zhang, Z., Cui, L., and Yang, L. Loss-based attention for deep multiple instance learning. AAAI Conference on Artificial Intelligence, pp. 5742- 5749, 2020.
Kim, J.-H., Choo, W., and Song. H. O. Puzzle mix: Ex- ploiting saliency and local statistics for optimal mixup. In International Conference on Machine Learning, pp. 5275-5285, 2020.
Shimada, T., Bao, H., Sato, I., and Sugiyama, M. Clas- sification from pairwise simillaities/ddisimilatttte and unlabeled data via empirical risk minimization. Neural Computation, (335)):2344111113 2021.
Kiryo, R., Niu, G., du Plessis, M. C., and Sugiyama, M. Positive-unlabeled learning with non-negative risk esti- mator. In Neural Information Processing Systems, pp. 1675-1685, 2017.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vision. IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Verma, V., Lamb, A., Beckham, C.,, Najafi, A., Mitliagkas, I., Lopez-Paz, D., and Bengio, Y. Manifold mixup: Better representations by interpolating hidden states. In Interna- tional Conference on Machine Learning, pp. 6438 6438-6447, 2019.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1) 11:21127778281 23244, 1998.
10

---

Confidence Difference Reflects Various Supervised Signals in Donfidence-iifferenee Classification
Wang, W., Feng, L.,, Jiang, Y., Niu, G., Zhang, M.-L., and Sugiyama, M. Binary classification with confidence dif- ference. Neural Information Processing Systems, 36, 2024.
Wang, X., Wan, W. Geng, C., Li, S., and Chen, S. Be- yond myopia: Learning from positive and unlabeled data through holistic predictive trends. In Neural Information Processing Systems, 2023.
Wu, J., Pan, S., Zhu, X., Zhang, C., and Wu, X. Multi- instance learning with discriminative bag mapping. IEEE Transactions on Knowledge and Data Engineering. 30 (6):1065-1080, 2018.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist novel image dataset for benchmarking machine learning algorithms. arXiv preprint 1rXi::1008.07747, 2017.
Xue, and Hauskrecht, M. Learning of classification models from noisy soft-labels. European Conference on Artificial Intelligence, pp. 1618-1619, 2016.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. Cutmix: Regularization strategy to train strong classifiers with localizable features. In IEEE/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Zhang. H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411 2017.
Zhang, L.., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. How does mixup help with robustness and gen- cralization? In International Conference on Learning Representations, 2021.
Zhang, M.-L. and Zhou, Z.-H. Multi-instance clustering with applications to multi-instance prediction. Applied Intelligence, 31:47-68, 2009.
Zhao, Y., Xu, Q. Jiang, Y., Wen, P., and Huang, Q. Dist-pu: Positive-unlabeled learning from label distribution per- spective. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14441-14450, 2022.
Zhou, Z.H Sun, Y.-Y,, and Li, Y.-F. Multi-instance leamn- ing by treating instances non-iid samples. In Interna- tional Conference on Machine Learning, pp. 1259-1256, 2009.

---

Confidence Difference Reflects Various Supervised Signals in Confidennee-iffferreee Classification
A. Proof of Theorem 3.1
In this appendix, we provide the proof of the Theorem 3.1 and the corresponding technical lemmas. Lemma A.1. The Rademacher complexity R, g) D for ConfDiff data with noise of size can be defined as follows:
(C o 9) 2Le (9) +
Rn² (9)
(15)
log (e)
The proof of Lemma A.I:
Rr (CCRCR o 9) =Ep,, Eo (sup DEG T1
og C (
Vig
D E sup gEG
a. Le RE
112
1
1
= =Bp,, 1 (sup
0i - (1)-
DET ThI
i=1
+ ( +
(1- (1-
+ E Dng
E, sup
a0
[1()(x;) (x1
gEg 122 1=1
log (1c, +E)
=Ep,, Eg (sup
ai
2 xx
gEg
1=1
112
+ D., E, sup gEg 122
(0: x, の |
x1),, [xx]]]
(16)
where
v (
2 11((--- -1)
(17)
+(T Ho--:/ -1)]|l,
2 (11
-1)
+ + [1 (()(g==)))(()))))) -1)) -1))lla
S | ai Le + 1 1-x - Le + n + cil Le + 2 11 - x + +G Le
2
<2Li
(18)
and,
V
log (Ica) +e) [|(o(x;)) g(x()1l2
g(x;) 9(x))
log (lcs) +)) llg(x;) - g(x2)112
(19)
log (E)
12

---

Confidence Difference Reflects Various Supervised Signals in Confidence-Differenne Classification
Replacing the corresponding term in 16 with 18 and Eq.19, can prove the Lemma A.I:
1 n2
(CRRCR S2,LED, Eg (su gEG 11
o
ogg
gx()x +
ED.. Eg suu DEG 122
Σ
oxg(x))
log (E)
:=1
<2L (9) +
a
Rog (9)
(20)
log (s)
Lemma A.2.
3a log(s)
In(2/0)
sup DEG
R(g) - (c) s (4+ n,,,
(9)
Rno (9) + Ctp
(21)
2n11
The proof of Lemma A.2:
sup gEg
R(g) RCcCR(0) sup gEg
R(g) -E[RCRCR(O)) (+)]
sup
E [[RCRCR()) RERER (g)
(22)
geg
We first discuss the first term (the bias term). Since Rcv(g) is an unbiased risk estimator of R(g), we have:
R(g) E|RRRRR [9)
R(g)) (CD(g) + ReD(g) - E E(ACRCR(s)) (9)l
RR(g) - ReD + Rev(g) E/ RRRRRRR (9) [9)]
RCD(g) (|[RERCR(
Reu(g: Ds) E[ Ds)]
ED
Reg(o)
(23)
where Reg(g)
n2 Li=1
log([c]++e)
(19(x;) ((x))]1 is the regularization term.
Let p(x,''' x') denote the original distribution, and pps (x, x') denote the conditional distribution of DD We define the
density ratio as: w(xxx') Pas (x.x)) Assuming there exists a constant / > O such that w(x, x') Pp5 [x,x(x S | for all
p(x,x))
p(xxx')
(x,x') x') E D Under this assumption, following the theoretical framework proposed by (Cortes et al., 2010), we can derive the following generalization bound for the weighted ConfDiff risk estimation error:
(2/8
sup gEG
Ren(g) - E [e Ds)1 < 2,9R., (9) + Cry
(24)
2m1
For the regularization term ED Reg(o) have:
sup gEg
ED Reg(o) < log(e)
Mi, (9)
(25)
In summary, the bias term satisfies:
R(g) []|RCRC(())] < Reo(g) E[Rc Ds)] + EDe Reg(g)]
ln(2/8) a
S 28L (9) + Cep
Rng (9)
(26)
2n
log(e)
13

---

Confidence Difference Reflects Various Supervised Signals in Confideenee-iifferreee Classification
Then we discuss the second term. Since the quantity suPzeg [[]ccc](0)]] RCRCR (g) is difficult to handle directly, let S,S' are two independent samples of the same size (Mohri et al., 2012), we upper bound it by:
[[R RERCR (0) - Es gEg
Es [ReRcR,s - RCRCR,s(O))
sup
sup gEg
Ss
sup LoEg
[RRRCR,s (g) RCRcR,s(s))
s,
sup gEG n
((oRCR(g, #/) (CRCR(g) 21))
n
Es,s
sup gEg n i=1
Σ
os (CERCR z/) - (CRCR(g) 2i))
< Est
sup 9EG n
o.fCRCR (g. z\}
+ Es,
sup
1
Σ (-o.))(RCR((
gEG Y2 i=1
E.
sup
Σ oy G. 2x)
DEg
22R, (CCRCR o 9)
2a
I RL, (9)
Rns (9)
(27)
log (s)
Then the upper bound of the Lemma A.2 can be expressed as:
sup gEg
R(g) (c)c
S sup gEg
R(g) E[RCRCR())) (0))
+ sup [[ReRc[(g)] RCRCR(9) gEs
In(2/5)
2a Rng (g)
S 2 (9) Cep
+ log(e)
Hn, (9) + 4L, (9) +
2n1
log (E)
3a
ln(2/))
= (4+ 2.2) (9)
Rng (9) Cry
(28)
log(E)
2n1
The proof of Theorem 3.1:
R(ỘCRCK) - R(g") = =(R(OCRCR) - RCACk((CRCC) + (RCRcH((CRcR) RRCCR (g))
+ (RenCR(g") R(g"))
< R(ÓCnCR)) REnucR ((CnCR)) + (Rencn(o") R(g'))
<2 sup R(g) (c) sgg
6a
In(2/8)
<(8 4,,,, (G) + log(e)
(9) + 20pp
(29)
2n
B. Proof of Eq. 6
In this appendix, we provide the proof of the Eq.
14

---

Confidence Difference Reflects Various Supervised Signals in Donfidence-Differenee Classification
Substituting the form of the loss function from Eq.5 into Eq.3, then we can obtain:
RCD(g)= = 2 (TT - (x)
Fxxxx)
(1--- (1--
+(m+cxxxx))(((((x))) + (1 (1==+<(xx))((((x)))
2 Bpxxx>)
- ( g(x)) + (1 - - (hlxxx)))))))(((())) + g(x))
=
(( g(x') + ( - -9(x))
E Ep(x.r))
[b(0(2)) + (1-2 2 -1-
20(xx, x (x((x)
2
x +h(g(x')) + ) [(x'))
E pp(xx")
[M((())) + +2((xx'))(xx) + (o(x)) - 20(xx xxxxxx)((xx)
+
18 pp(xxx))
(1- + [1-2=)((xx++((x))
2
E Fx(xx')
+ [M((())) + 20(x +( - 20(xx x xx)((')
2
+o(x, ()(()))))))))))))))))))))))))( - ((xxx) + 1 (50(x)= - 1 -5xx
+c(xxxx)) ((xxx))o(x))
1 -0(x²) [
1
Bp(xx')
(1 2x) (1(x) +
2
=
E px(xx))
1
( - cxx (x(x))(((x) - 2 29(x) + +xxxxx')
1
2
2
1
[g(x)) + + ()xxx))
_0(x)) - c(xxx x9((x)
2
+ E3 p(xxx')
(1 27) (x(x) +x(x))
1
1
E Bx(xx))
1 (((x)) - c(x (1(xx)
2 50(x) + x(xx)))(xx
2
1
(o(x')) + xxxxxx (Mxxx) xx(x - cxxx, xx)
2
+ E8
xxxxxx)
2 ((((x)) + xxx (M(x)) + *50(x) + c(x, (xx
2
1
(o(x)) )) -xxxxx(M((xx)) + 1 -0(x) c(xx x x((()
2
E Ex(xx)
(1 2x) 2x))(1))))))) (5(x) +
2
1 ())x))))(((()) +1) + 2 + c(x,x (()xx))((((()),,,))))))))))))))) +1]
1
Fxxxx)
=
2
+ 2 Fxxxx))
1/ c(xx xxx, x))))(((((()) -1) + ( cxxxxxx))))) ((xxx)) -1]
1
(1 (1-22 2x) + [
+ E Fxxxx))
(30)
Then Eq. is proven.
15


Reviews:
[{"review_id": "ILZZ2pfN8X", "reviewer": "Reviewer_UWhK", "summary": "The paper studies a special type of weakly-supervised learning known as confidence difference learning. This method leverages confidence differences between unlabeled data pairs to improve classifier training under noisy real-world conditions. By incorporating a noise generation technique and a risk estimation framework that includes consistency risk and regularization, ConfDiff classification demonstrates enhanced robustness and outperforms traditional methods in experiments on benchmark and UCI datasets. Theoretical analyses providing error bounds for the risk estimations further support the method's effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The motivation of this paper is not strong enough, and what real-world scenerios can motivates this problem?**\nThank you for your suggestions. \n\n**About motivation.** The motivation for our method arises from the observation that small confidence differences may lead to imprecise guidance within $R_{CD}$, particularly when the confidence difference equals zero, resulting in a complete lack of predictive guidance. Our experiments in Figure 1 further demonstrate that pairwise instances with larger confidence differences dominate the contribution to $R_{CD}$, while those with smaller confidence differences contribute minimally. To address this issue, we propose a consistency regularization term that encourages $\\mathbf{x}$ and $\\mathbf{x'}$ to produce more similar outputs in the model when $|c(\\mathbf{x}, \\mathbf{x'})|$ is small.\n\n**About real-world scenerios.**\n\n1. Rehabilitation assessment. The assessment of whether a patient meets rehabilitation criteria presents significant challenges. Individual differences in recovery make the evaluation process inherently subjective. In addition, assessments become particularly uncertain when a patient is close to the threshold of rehabilitation. In contrast, clinicians can more reliably generate approximate confidence labels using objective data, such as scores from functional assessment scales (e.g., FIM or Barthel Index), questionnaire responses, and motor performance metrics. Moreover, by considering the continuity of the rehabilitation process and individual differences, changes in a patient's condition over time can be used to construct confidence difference that reflect recovery trends, leading to more robust rehabilitation assessment.\n\n2. Click-through rate prediction. In recommender systems, predicting whether a user will click on a given item is a central task. However, due to the sparsity of click data and the problem of class imbalance, it is often difficult to assign accurate pointwise label to each user-item pair. In contrast, collecting pairwise preference information, which refers to the relative preference between candidate items for a given user, is a more feasible and effective alternative. In practical applications, approximate confidence can be obtained using auxiliary probabilistic classifiers, such as models predicting click probabilities. Moreover, in many recommendation scenarios such as news, short video, and movie recommendations, real-valued feedback like watch ratio or user ratings is often available, which can be leveraged to construct more informative confidence difference.\n\nIn addition, it also holds practical value in various domains such as obstacle detection in autonomous driving, driving behavior analysis, and financial risk assessment.\n\n&nbsp;\n\n**Q2. It seems that this problem is only applicable for binary classification, which further limits its application in real-world scenerios.**\n\nThank you for your insightful comments. The method proposed in this paper is indeed developed within the framework of binary classification. Nevertheless, we respectfully believe that this setting does not substantially limit its applicability to real-world scenarios.\nFirst, a wide range of real-world applications naturally take the form of binary classification problems, where our method can be directly implemented. Typical examples include medical diagnosis, rehabilitation assessment, and financial risk management.\nFurthermore, the proposed method is conceptually general and can be naturally extended to multi-class classification tasks, which we consider a promising direction for future research. Formally，let $ c_i \\in \\mathbb{R}^l $ be the confidence difference between pairwise unlabeled data $(\\mathbf{x}_i,\\mathbf{x}'_i)$ drawn from an independent identically distribution probability density $p(\\mathbf{x},\\mathbf{x}')=p(\\mathbf{x})p(\\mathbf{x}')$:\n\n$$ \\mathbf{c}_i = [c_i^{(1)}, c_i^{(2)},\\dots ,c_i^{(l)}],\\\\:\\\\:c_i^{(k)}=c^{(k)}(\\mathbf{x}_i,\\mathbf{x}'_i)=p(y'_i=k|\\mathbf{x}'_i)-p(y_i=k|\\mathbf{x}_i) $$\n\nwhere $l$ denotes the number of classes. Accordingly, the consistency regularization term over $D^{C}$ in the expected risk can be modified as:\n\n$$\\alpha \\mathbb{E}\\_{p\\_{{\\small \\mathcal{D}^{C}} }(\\mathbf{x},\\mathbf{x}')} [ \\bigl(\\frac{1}{ \\log \\left(\\left| \\mathbf{c(\\mathbf{x},\\mathbf{x}')}\\right|_1 + \\varepsilon  \\right)  } \\bigr)  \\cdot  \\left \\\\| g(\\mathbf{x})-g(\\mathbf{x}') \\right \\\\|_2 ] $$\n\nIn summary, this work proposes a general framework for addressing noisy supervision signals in confidence difference classification. The proposed method is not limited to binary classification tasks and also demonstrates practical applicability in real-world scenarios.", "claims_and_evidence": "yes", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "Claims seems to be correct.", "experimental_designs_or_analyses": "Yes.", "relation_to_broader_scientific_literature": "n.a.", "essential_references_not_discussed": "n.a.", "other_strengths_and_weaknesses": "1. The main theoretical contribution of this paper improved over [1] seems to be the incorporation of consistency regularization and its effect in error bound; based on my understanding, $C_{g}$ bounded the differences between $x$ and $x'$, so that under authors' setup, if the prediction of these data points is close enough, then the perceived generalization error should decrease, which is sensible.\n\n2. After a very coarse examination, the proof of this paper seems to be correct.\n\n3. Encourage instances with smaller confidence differences to produce similar outputs that seem intuitive and sensible, both theoretically and empirically.\n\n[1] Binary classification with confidence difference, NeurIPS 2023.\n\nWeaknesses:\n\n1. I feel the motivation of this paper is not strong enough, and I cannot see much real-world scenerios that motivates this problem.\n\n2. It seems that this problem is only applicable for binary classification, which further limits its application in real-world scenerios.", "other_comments_or_suggestions": "n.a.", "questions_for_authors": "n.a.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bQx5pdeaeq", "reviewer": "Reviewer_aabi", "summary": "In this paper, the authors identify that noise supervision signals emerge in current confidence difference classification methods when the confidence difference is small. Based on this observation, the core focus of this work is to explore a robust solution for confidence difference classification by mitigating the impact of inaccurate supervised signals. This paper proposes a novel robust confidence difference classification method, constructing a risk estimator based on consistency risk and consistency regularization (CRCR), and theoretically derives the error bound of CRCR. The idea presented in this paper is interesting, and extensive experiments on diverse datasets under artificial noise interventions support the generalization capability of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. Which specific loss functions (e.g. logistic) are included in Eq.5?**\n\nThank you for your comments. This function class includes several commonly used loss functions, such as those derived from Generalized Linear Models (GLMs), including the mean squared error (MSE) for linear regression, the logistic loss for logistic regression, the Poisson loss for Poisson regression, the exponential loss for exponential regression, and the cross-entropy loss for neural networks (Zhang et al., 2021).\n\nZhang, L., Deng, Z., Kawaguchi, K., Ghorbani, A., and Zou, J. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021.\n\n&nbsp;\n\n**Q2. Could you provide a clearer explanation of the x-axis in Fig.1?**\n\nThank you for your suggestions. The x-axis values multiplied by $\\pi$ represent the proportion of pairwise instances $(\\mathbf{x}, \\mathbf{x}')$ with confidence differences $c(\\mathbf{x}, \\mathbf{x}') \\in [-1, -0.5) \\cup (0.5, 1]$ relative to all pairwise instances. Thus, the x-axis values effectively serve as a scaling factor used for computing the proportion.\n\n&nbsp;\n\n**Q3. What is the purpose of the design in Section 3.4? Why might negative empirical risk lead to severe overfitting? And how does the risk correction function address this issue?**\n\nMany thanks for your comment. The purpose of Section 3.4 is to address the overfitting problem when using flexible models due to negative empirical risk.\n\nRisk is typically non-negative, reflecting the deviation between model predictions and ground-truth values. The objective of optimization is to minimize risk; if risk could be negative, the model could be inclined to find an optimization direction that continually reduces risk on the training data, leading to overfitting by learning noise and performing poorly on test data. Additionally, (Lu et al., 2020) highlights that negative empirical risk may be a potential cause of overfitting and experimentally demonstrates a strong co-occurrence of negative risk and overfitting across various models and datasets.\n\nThe risk correction function enforces the non-negativity of the risk by using $|\\cdot |$ or $max\\\\{0,\\cdot\\\\}$.\n\nLu, N., Zhang, T., Niu, G., and Sugiyama, M. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In International Conference on Artificial Intelligence and Statistics, pp. 1115–1125. PMLR, 2020.\n\n&nbsp;\n\n**Q4. Would consistency regularization term lead to more instance pairs with smaller confidence differences, thereby increasing the presence of noisy supervised signals?**\n\nThank you for your comments. The core objective of consistency regularization is to enforce consistency in the classifier's predictions for pairwise instances with small confidence differences by constraining the model output. It is important to clarify that our optimization target is the classifier's output $ g(\\cdot)$, not the confidence difference $c$. In our setting, $c$ serves as an attribute used for training, functioning as a form of weak supervision. Our goal is not to optimize $c$, but to use it to guide the classifier toward the desired outputs. Therefore, the concern about \"causing more pairs to have smaller confidence differences\" does not apply here.", "claims_and_evidence": "The claims presented in this paper are supported by clear and convincing evidence. Specifically, this paper introduces a new risk estimator by analyzing the various supervised signals reflected in the ConfDiff method. The proposed risk estimator has been empirically validated; its error bound has been theoretically analyzed; and its robustness has been further verified under artificial noise intervention.", "methods_and_evaluation_criteria": "The proposed method, CRCR, effectively addresses the issue of noisy supervised signals in the ConfDiff method. These noisy signals tend to encourage the classifier to make predictions in the opposite direction, making this a novel and interesting research direction.", "theoretical_claims": "I've carefully checked the correctness of the theoretical claims presented in this paper.\n1) This paper analyzes the various supervised signals reflected by different confidence differences in ConfDiff classification from both experimental and theoretical perspectives, supporting the claim that noisy supervised signals exist in the ConfDiff method.\n2) This paper also derives the error bound of the proposed method and provides detailed proofs and theoretical analysis in the appendix.", "experimental_designs_or_analyses": "I've rigorously reviewed the experimental designs and analyses. In addition to conventional experimental designs, this paper introduces artificial noise at different levels and proposes a method to generate artificial noise to simulate potential real-world noise distributions. This's an interesting perspective.", "relation_to_broader_scientific_literature": "The main motivation of this paper is an observation by analyzing the various supervised signals in ConfDiff method from both experimental and theoretical perspectives. The authors found that the ConfDiff method introduces noisy supervised signals when the confidence difference is small. Consequently, this paper focuses on mitigating the challenges posed by these noisy supervised signals.", "essential_references_not_discussed": "I don't see any essential related works missing from the citations.", "other_strengths_and_weaknesses": "Strengths:\n1) This paper is motivated by the noisy supervised signals introduced in the ConfDiff method. The proposed method, CRCR, effectively addresses this issue, making it a novel and interesting research direction.\n2) The proposed method, which constructs a risk estimator based on consistency risk and consistency regularization, is effective and is supported by both theoretical analysis and experimental validation.\n3) In addition to traditional experimental settings, this paper designs an artificial noise generation method for confidence difference classification. The goal is to test whether the proposed method can maintain robustness under different levels of artificial noise interventions. The experimental results confirm this robustness.\n\nWeaknesses:\n1) Some details are not well explained, as noted in the Questions section.\n2) The reasoning behind the overfitting issue should be better explained.\n3) This paper follows a style similar to the ConfDiff method. However, the introduction of artificial noise at different levels is a noteworthy and distinctive highlight compared to prior work.", "other_comments_or_suggestions": "Please refer to the Weaknesses and Questions.", "questions_for_authors": "1) Eq.5 provides a general form of many commonly used loss functions. Which specific loss functions are included? Is the logistic loss function used in the code also part of this general form?\n2) Figure 1 is somewhat difficult to understand. Could you provide a clearer explanation of the meaning of the x-axis in Figure 1?\n3) What is the purpose of the design in Section 3.4? Why might negative empirical risk lead to severe overfitting? And how does the risk correction function address this issue? This paper seems to lack a reasonable explanation.\n4) The consistency regularization term encourages consistency between confidence differences and model outputs. Would this lead to more instance pairs with smaller confidence differences, thereby increasing the presence of noisy supervised signals?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ozYjPP5pMe", "reviewer": "Reviewer_B2Kr", "summary": "This paper deals with confidence-difference classification, a weakly supervised binary classification problem. To mitigate the noise contained in the confidence differences, a novel risk estimator using consistency regularization is employed to improve performance. Extensive experiments on benchmark datasets validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. If $c>0.5$, the two examples belong to different classes. This is not true.**\n\nThank you for the insightful correction. The statement \"$c>0.5$, the two examples belong to different classes\" is indeed not precise enough. A more accurate expression would be \"$c > 0.5$, the two examples belong to different classes **at a high probability**\". This is also consistent with the data partitioning strategy adopted in our paper, which separates the dataset into a subset $D^S$ with relatively precise predictive information (i.e., $c > 0.5$) and a subset $D^C$ with comparatively imprecise predictive information (i.e., $c \\le 0.5$).\nMoreover, we agree with your point that, there is no direct correspondence between the labels and the posterior probability during training. Our goal is to enable the model to learn and establish this relationship through training. \n\nWe will revise this statement to be more rigorous in the next version.\n\n&nbsp;\n\n**Q2. I am not sure whether equation (7) is biased from the original classification risk in equation (1). This is because the marginal distribution may change after data partitioning. If so, will this affect the theoretical analysis in Theorem 3.1, since the minimizers of the two risks are not the same?**\n\nThank you for your comments. We agree that Eq.7 constitutes a biased estimator of the original classification risk in Eq.1, due to the change in the marginal distribution induced by the subset partitioning strategy. However, this bias does not affect the theoretical analysis in Theorem 3.1.\n\nIn Theorem 3.1, we explicitly model the risks over the two subsets separately, and analyze their contributions through the Rademacher complexities $\\mathfrak{R}\\_{n_1}(\\mathcal{G})$ and $\\mathfrak{R}\\_{n_2}(\\mathcal{G})$, respectively. The first and second terms of the error bound directly involve these complexities. So as $n_1, n_2 \\to \\infty$, both $\\mathfrak{R}\\_{n_1}(\\mathcal{G})$ and $\\mathfrak{R}\\_{n_2}(\\mathcal{G})$ tend to zero. Additionally, the third term, which depends on $\\sqrt{n}/n_1$ and $\\sqrt{n}/n_2$, also diminishes as the sample sizes increase. Consequently, as long as $n_1, n_2 \\to \\infty$, the overall estimation error still converges to the minimum of the original classification risk $R(g^*)$.\n\n&nbsp;\n\n**Q3. Line 183 should read instead $g$ of $\\mathcal{G} $. The condition in line 184 is also incorrect.**\n\nThanks for your correction. We will revise them in the next version.", "claims_and_evidence": "The claim that different examples have different monitoring signals is novel and valid. Empirical analyses show that examples with $c>0.5$ carry more information, while examples with $c<0.5$ should be considered separately.", "methods_and_evaluation_criteria": "The proposed methods are reasonable and the effectiveness is validated by both theoretical analysis and experimental results. The evaluation criteria are reasonable and follow commonly used protocols in the literature.", "theoretical_claims": "The theoretical claims are correct.", "experimental_designs_or_analyses": "The experimental designs are good. First, the experimental data sets are comprehensive and the methods compared are current. Second, the experimental analyses are good. Third, ablation studies and sensitivity analyses are performed.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "All the essential references are discussed.", "other_strengths_and_weaknesses": "### Strengths\n- The problem studied is novel in the literature. It is natural that different confidence differences carry different supervision signals in ConfDiff classification. A simple and effective regularization term is added to the previous unbiased risk estimator, and good performance is achieved.\n\n- The paper is generally well written.\n\n- The effectiveness of the proposed method is supported by solid theoretical analysis and extensive experiments.\n\n### Weakenesses\n- Line 161 says that if $c>0.5$, the two examples belong to different classes. This is not true, because the label is not related to the posterior probability. For example, negative data can have a large true posterior probability ($p=0.8$) and positive data can have a small true posterior probability ($p=0.2$). Therefore, the two examples can still belong to the same class.\n\n- I am not sure whether equation (7) is biased from the original classification risk in equation (1). This is because the marginal distribution may change after data partitioning. If so, will this affect the theoretical analysis in Theorem 3.1, since the minimizers of the two risks are not the same?", "other_comments_or_suggestions": "- There are some notations that need to be revised. For example, line 183 should read $g$ instead of $G$. The condition in line 184 is also incorrect. The details should be checked carefully.", "questions_for_authors": "Please see \"Weaknesses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Three reviewers support acceptance, highlighting the work's novelty, theoretical rigor, and empirical validation. However, Reviewer joRe, who evaluated this paper at ICLR 2025, notes that the current submission does not incorporate updates beyond the ICLR'25 final version (which already addressed their concerns during ICLR'25 rebuttal). So, I recommended Reject in the previous stage.  

After discussion with the SAC, I have compared the current submission with its ICLR'25 versions (both initial and revised version after ICLR'25 rebuttal). The authors did address most of the concerns during the ICLR'25 rebuttal, as the current manuscript differs significantly from the ICLR'25 initial version, with substantive revisions made during the rebuttal process. Therefore, I now lean toward Weak Accept, as the core criticisms were previously mitigated.
Decision: Accept (poster)