Pairwise Confidence Difference on Unlabeled Data is Sufficient for Binary Classification
Authors: Wei Wang
Lei Feng
Gang Niu
Min-Ling Zhang
Masashi Sugiyama
Venue: Submitted to ICLR 2023 (2023)
Keywords: Weakly supervised learning
binary classification
unbiased risk estimator

Abstract: Learning with confidence labels is an emerging weakly supervised learning paradigm, where training data are equipped with confidence labels instead of exact labels. Positive-confidence (Pconf) classification is a typical learning problem in this context, where we are given only positive data equipped with confidence. However, pointwise confidence may not be accessible in real-world scenarios. In this paper, we dive into a novel weakly supervised learning problem called confidence-difference (ConfDiff) classification. Instead of pointwise confidence, we are given only unlabeled data pairs equipped with confidence difference specifying the difference in the probabilities of being positive. An unbiased risk estimator is derived to tackle the problem, and we show that the estimation error bound achieves the optimal convergence rate. Extensive experiments on benchmark data sets validate the effectiveness of our proposed approaches in leveraging the supervision information of the confidence difference.

Under review as a conference paper at ICLR 2023
PAIRWISE CONFIDENCE DIFFERENCE ON UNLABELED DATA IS SUFFICIENT FOR BINARY CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Learning with confidence labels is an emerging weakly supervised learning paradigm, where training data are equipped with confidence labels instead of ex- act labels. Positive-cone contfive-oonfieenc (Pconf) classification is typical learning problem in this context, where we are given only positive data equipped with confidence. However, pointwise confidence may not be accessible in real-world scenarios. In this paper, we dive into a novel weakly supervised learning problem called -iffeeeeedeefeeeceeeeee (ConfDiff) classification. Instead of pointwise confidence, we are given only unlabeled data pairs equipped with confidence difference spec- ifying the difference in the probabilities of being positive. An unbiased risk es- timator is derived to tackle the problem, and we show that the estimation error bound achieves the optimal convergence rate. Extensive experiments bench- mark data sets validate the effectiveness of our proposed approaches in leveraging the supervision information of the confidence difference.
INTRODUCTION
Recent years have witnessed the prevalence of deep learning and its successful applications. How- ever, the success is built on the basis of the collection of large amounts of data with unique and accurate labels. In many real-world scenarios, is often difficult satisfy such requirements.
To circumvent the difficulty, various weakly supervised learning problems have been investigated accordingly, including but not limited to semi-supervised learning (Chapelle et al. [006]] Zhu &
Goldberg 2009 Li & Zhou, 2015 Berthelot et al. 2019). label-noise learning (Patrini et al. 2017
Han et al. 2018
Li al. 2021 Wang et al. 202 Wei al. 2022), positive-unlabeled learning
(du Plessis et al.
2014,
Su et al.
2021 Yao et al. (022). ) partial-label learning (Cour et al. 2011
Wang & Zhang, 2020;
Wen et al. 2021 Wang et al. 2022 Wu et al. 2022), unlabcled-unlabelle
learning (Lu et al.
2019 2020) and similarity-based classification (Bao et al. 2018 Cao et al.
20216
Bao et al. 2022).
Learning with confidence labels (Ishida et al. 2018 Cao et al. 202 b) is another weakly super-
vised learning paradigm, where we are given training examples with confidence labels instead of exact labels. Positive-confidence (Pconf) classification (Ishida al. 2018) is a problem setting within this scope, which is aimed at learning binary classifier from only positive data equipped with confidence (the probability of being positive) without negative data. Pconf classification can alleviate the difficulty when negative data cannot be acquired due to privacy or security issues dur- ing the data annotation process. The need to learn from such inexact supervision widely exists in real-world scenarios, such as purchase prediction (Ishida et al. 2018), user preservation prediction [Ishida al. 2018), drivers' drowsiness prediction (Shinoda et al. 2020), etc.
However, the process of collecting large amounts of training examples with pointwise confidence might be actually demanding under many circumstances, since it tough to describe the probabil- ity of being positive for each training example exactly (Shinoda et al., 2020). |Fen et al. (2021) showed that learning from pairwise comparisons could serve as an alternative strategy given limited pointwise labeling information. Inspired by it, we investigate a more practical problem setting in
this paper, where we are given only unlabeled data pairs with confidence difference indicating the difference in the probabilities of being positive. Compared with pointwise confidence, confidence difference can be collected more easily in many real-world scenarios. Take click-througe through rate pre- diction in recommender systems (Zhang et al. 2019: for example. The combinations of users and

---

Under review as a conference paper at ICLR 2023
their favorite/disliked items can be regarded positive/negative data. When collecting training data, it is not easy to distinguish between positive and negative data. Furthermore, the positive confidence of training data may be difficult to be determined due to the extremely sparse and flss-imbalanne problems Yao et al., 2021). However, it is much easier to obtain the difference in the preference between a pair of candidate items for given user. Take the disease risk estimation problem for another example. The goal is to predict the risk of having some discase given person's 's attributes. When asking doctors to annotate the probabilities of having the disease for people, is not easy to determine the exact values of the probabilities. Furthermore, the probability values given by dif- ferent doctors may be different due personally subjective assumptions and will deviate from the ground-truth values. However, it is much easier and less biased to estimate the relative difference in the probabílities of having the disease between two people.
Our contributions are summarized as follows:
We investigate confidence-differnne (ConfDifff) classification, a novel and practical weakly su- pervised learning problem, which can be solved via empirical risk minimization by constructing an unbiased risk estimator. The proposed approach can be equipped with any model, loss func- tion, and optimizer flexibly.
The estimation error bound is derived, showing that the proposed approach achieves the optimal parametric convergence rate. The robustness is further demonstrated by probing into the influence of an inaccurate class prior probability and noisy confidence difference.
To mitigate overfitting issues, risk correction approach (Lu et al.) (0200) with consistency guar- antee is further introduced. Extensive experimental results on benchmark data sets validate the effectiveness of the proposed approaches.
Related works.
Learning with pairwise comparisons has been investigated pervasively in the community (Burges et al. |2005] et al. 2007 Jamicson & Nowak 2011 Park et al. [013 Kane
et al. 2017 Xu al. [011 Shah et al.] 2019), with applications in information retrieval (Liu 2011
computer vision (Fu et al. 2015), regression (Xu et al.] 2019; 2020). crowdsourcing (Chen et al,
2013 Zeng & Shen (022). graph learning (He al., 2022). etc. is noteworthy that there exist distinct differences between our work and previous works on learning with pairwise comparisons.
Previous works have mainly tried to learn a ranking function which can rank candidate examples according to the relevance or preference. In this paper. we try to learn a pointwise binary classifier by conducting empirical risk minimization under the binary classification setting.
Relationship to Peomp classification. Feng et al. (2021) elaborated that a binary classifier could be learned from pairwise comparisons, which was termed as Pcomp classification. There are distinct differences between our work and Pcomp classification. First, Pcomp classification is not capable of leveraging the fine-grained confidence difference, which can be incidentally obtained when col- lecting pairwise comparison data. We will experimentally elucidate the benefit of exploiting the confidence difference in the later section. Second, the assumptions of the data generation process different. Peomp classification assumes that the unlabeled data pair is ordered, where the first instance is more likely to be positive than the other. In ConfDiff classification, the instances of the unlabeled data pair are independent, which can be easier to collect.
PRELIMINARIES
In this section, introduce the notations used in this paper and discuss the background of binary classification, Pconf classification and Pcomp classification. Then, we elucidate the data generation process of confidence-difffrencce classification.
2.1 BINARY CLASSIFICATION
For binary classification, let X' = Rd denote the d-dimensional feature space and y = {+1, -1} denote the label space. Let p(æ, y) denote the unknown joint probability distribution over random variables (xt,) y) E X' x y. The task of binary classification is to learn a binary classifier g X - IR which minimizes the following classification risk:
R(g) E.p(e..)) (((g(z), v/),
2

---

Under review as a conference paper at ICLR 2023
where ((., -) is non-negative binary-clase loss function, such as the 0-1 loss and logistic loss. Let 4 = p = +1) and π- = ply = -1) denote the class prior probabilities for the positive and negative classes respectively. Furthermore, let p+(x) p(æly +1) and (z) p(æly -1) denote the class-conditional probability densities of positive and negative data respectively. Then the classification risk in Eq. ( can be equivalently expressed as
R(g)> = π. (w)((((())))) +1] + x Pp (2) [e(g(2)) -1]]-
(2)
2.2 POSTTIVE-CONFIDENCE (PCONF) CLASSIFICATION
In many real-world applications, it may be difficult to collect negative data. Pconf classification (Ishida et al., 2018) is aimed at inducing a binary classifier from only positive data. The additional requirement is that the confidence of being positive should be accessible the learning algorithm. Given only positive data equipped with confidence {(x.. rx) Ishida et al. (2018) provided an unbiased risk estimator conduct empirical risk minimization:
Recont (g) = # Σ (C(g(z)), +1)+ 1- 1- ((g(x)) -1))
(3)
where ri = p(y: +1 (ax)) is the positive confidence associated with x,, However, pointwise positive confidence may not be casy to obtain in real-world scenarios (Shinoda et al. 2020))
2.3 PAIRWISE-CONe COMPARISON (PCOMP) CLASSIFICATION
Peomp classification weakly supervised binary classification problem (Feng et al. 2021). In Pcomp classification, we are given pairs of unlabeled data where we know which one is more likely to be positive than the other. It is assumed that Pcomp data are sampled from labeled data pairs whose labels belong to {(+1, 1),, (+1, +1), -1)}. Based on this assumption, the probability density of Pcomp data (ær, æ²) is given as
9(2, x')
5(x, x')
# + +2 + +
where q(æ, x') ( risk estimator for Pcomp classification is derived as follows:
(x') + --p- (x)pp (a') + - (+)ppp (x'). Then, an unbiased
- Σ (l(g(xx), (((((=)) -1) T+((g(x)) -1) - =[((((=))))))) +1)). (5) ni
Recomp (g)
In real-world applications, we may not only know one example is more likely to be positive than the other, but also know how much the difference of confidence is. Next, a novel weakly supervised learning setting named ConfDiff classification is introduced.
2.4 CONFIDENCE- DONFIDENCE-DIFFERRNEE (CONFDIFF) CLASSIHCATION
In this subsection, the formal definition of confidence difference is given firstly. Then, we elaborate the data generation process of ConfDiff data. Definition 1 (Confidence Difference). The confidence difference c(x, x') between the unlabeled data pair (xx, x²) is defined as
x(x =(y' -1 1/z)
(6)
As shown in the definition above, the confidence difference denotes the difference in the class pos- terior probabilities between the unlabeled data pair, which can measure how confident the pairwise comparison is. In ConfDiff classification, we are only given unlabeled data pairs with confidence difference
{((,, az)), (.)), Here, G; = c(x;,,)) is the confidence difference for the unla- beled data pair (æ,, ar)). Furthermore, the unlabeled data pair (æ;; a is assumed to be drawn from a probability density p(x, x') p(x)). This indicates that æ; and x' are two i.i.d. instances sampled from p(æ). is worth noting that the confidence difference will be positive if the second instance x has a higher probability to be positive than the first instance K;, and will be negative otherwise. During the data collection process, the labeler can first sample two unlabeled data from the marginal distribution p(x), then provide the confidence difference for them. This data generation assumption makes the unlabeled data pairs easier to be collected.
3

---

Under review as a conference paper at ICLR 2023
3 THE PROPOSED APPROACH
In this section, an unbiased risk estimator is presented for ConfDiff classification. Then, we give estimation error bound to show the convergence property. Besides, we show the influence of an inaccurate class prior probability and noisy confidence difference on the risk estimator. Furthermore, risk correction approach (Lu al. 2020 elaborated to improve the generalization performance of our proposed approach.
3.1 UNBIASED RISK ESTIMATOR
In this subsection, we show that the classification risk Eq. ①T can be expressed with ConfDiff data in the equivalent way.
Theorem 1. The classification risk R(g) Eq. (7) can be equivalently expressed as RCD(g) = pp( (C(x, x') + ,()(,
where
L(x, x') d(z -x x1 x))) +1) + (a_ - c(x, x' 2')))(()))))))))))))))))))))))))
Accordingly,
derive unbiased risk estimator for ConfDiff classification:
Σ ((++ ((=+-c)e(o(a)) ())(((())))))))))))))))))))))))) +1) + (x (=))((())))) 1) RCD(g) 2n
1
(8) To estimate the class prior probability T+, we can transform ConfDiff data into Peomp data by ranking the two instances in the unlabeled data pair according to the confidence difference. Then, we can adopt the approach proposed in Feng et al. (2021) to estimate नч. It worth noting that the risk estimator in Eq. (3) for Pconf classification very sensitive small confidence values, while our risk estimator will not be influenced by them.
( +1) + ( +1) ())()))))))
Minimum-varriance risk estimator. Actually, Eq. 8 is of the candidates of the unbiased risk estimator. We introduce the following lemma:
Lemma 1. The following expression is also unbiased risk estimalor:
Σ (aL(x, + (L=a)[(z(,, x))),
where a E [0, 1] is an arbitrary weight.
Then, we introduce the following theorem:
Theorem 2. The unbiased risk estimator in Eq. {8) has the minimum variance among all the can- didate unbiased risk estimators in the form of Eq. 19 w.r.t. a E [0,1].
Theorem indicates the variance minimality of the proposed unbiased risk estimator in Eq. { and we adopt this risk estimator in the following sections.
3.2 ESTIMATION ERROR BOUND
In this subsection, elaborate the convergence property of the proposed risk estimator (CD(g) by giving an estimation eтror bound. Let G = {g X' + R denote the model class. It is assumed that there exists some constant Cg such that suppeg ||g]| S Cg and some constant C₃ such that sup]=isc, {(2, 9) I Ct. We also assume that the binary loss function E(z, y) is Lipschitz continuous for z and y with a Lipschitz constant Le. Let g = arg mingeg R(g) denote the minimizer of the classification risk in Eq. and ğCD = arg min gg RCD(g) denote the minimizer of the unbiased risk estimator in Eq. The following theorem can be derived:
'The theoretical analysis in the next subsections is also based these assumptions, For simplicity, we do restate them in the subsections.
4

---

Under review as a conference paper at ICLR 2023
Theorem 3. For any 5 > 0. the following inequality holds with probability least 1
In 2/6
R(GeD) - R(g") S (L) + 4Cr
(10)
2n
where (g) denotes the Rademacher complexity of 9 for unlabeled data with size n.
From Theorem we can observe that as n + 20, R(CcD) ( + R(g") because R, - for all parametric models with bounded norm, such deep neural networks trained with weight decay Golowich et al. 2018). Furthermore, the estimation error bound converges in Op(1/v/ where Op denotes the order probability, which is the optimal parametric rate for empirical risk minimization without making additional assumptions (Mendelson, 2008).
3.3 ROBUSTNESS OF RISK ESTIMATOR
In the previous subsections, was assumed that the class prior probability is known in advance or estimated accurately. In addition, it was assumed that the ground-truth confidence difference of each unlabeled data pair is accessible. However, these assumptions can rarely satisfied in real-world scenarios, since the collection of confidence difference is inevitably injected with noise. In this subsection, we theoretically analyze the influence of an inaccurate class prior probability and noisy confidence difference the learning procedure. Later in subsection 4.4 will experimentally verify our theoretical findings.
Let D {((,,, ((),)))> denote unlabeled data pairs with noisy confidence difference, where C is generated by corrupting the ground-truth confidence difference G; with noise. Besides, let 7. denote the inaccurate class prior probability accessible to the learning algorithm. Furthermore, let (g)(g) denote the empirical risk calculated based on the inaccurate class prior probability and noisy confidence difference. Let ÎCD = arg mingeg (cD denote the minimizer of RCD(g). Then, the theorem demonstrating an estimation error bound is given as follows:
Theorem 4. Based on the assumptions above, for any > O, the following inequality holds with probability least 6:
In 2/6 2n
+ 4C En cil + +C0// T+I- (1I)
R(āCD) - R(gg) S 16L &CC
Theorem 4 indicates that the estimation error is bounded by twice the original bound in Theorem 3 with the mean absolute error of the noisy confidence difference and the inaccurate class prior probability. Furthermore, if Σ ج cal has sublinear growth rate with high probability and the class prior probability estimated consistently, the risk estimator can be even consistent. elaborates the robustness of the proposed approach.
3.4 RISK CORRECTION APPROACH
is worth noting that the empirical risk in Eq. 8 may be negative due to negative terms, which is
unreasonable because of the non negative property of loss functions. This phenomenon will result in severe overfitting problems when complex models are adopted (Lu et al. 2020) Cao et al. 2021h Feng al.) 2021). To circumvent this difficulty, with risk correction functions proposed in Lu et al. (2020), such as the rectified linear unit (ReLU) function fz) max(0, 2) and the absolute value function f(2) |z|. In this way, the corrected risk estimator for ConIDiff classification can be expressed as follows: a_ c))(((a()) -1)
wrap the individual loss terms in Eq.
ReD(g)
n c.) +1))
2n
+ (m + +c)) +1)) x ()((((())))))))))))))))))))))))) - -1))).
(12)
Theoretical analysis. We assume that the risk correction function f2) is Lipschitz continuous with Lipschitz constant Teg. For ease of notation, let A, E (I+ c,))(g(ar))) + +1)/2n, B,
5

---

Under review as a conference paper at ICLR 2023
ET ci ()){((()))) -1) Č Ei= + c))(g(z)), +1)/2n, D,,
- +
)))(g(z;), -1) 2n. From Lemma in Appendix A the values of B[[A], 15 and EID, non-negative. Therefore, we assume that there exist non-negative constants b, c, d such that [[A,] a, [[ 2 b, ElC,) g) c, and [D2p) d. Besides, let JCD arg oig (cD denote
the minimizer of RcD(g). Then, Theorem is provided to elaborate the bias and consistency of RgD(g)
Theorem 5. Based on the assumptions above, the bias of the risk estimator RCD(9) decays expo- nentially as n oo:
s E[RcD(g)] R(g)= 2(1, 1)C,A,
(13)
where A exp (xp(--n'n/C3 + exp 22/2n/C2 + exp (-2e2n/C3) + exp e (-22P//)/)). Fur- thermore, with probability at least 1 6, we have
2/ 2n
(CD(() R(g) - 2C
+ 2( + 1)C2A.
(14)
Theorem demonstrates that (g)(g) + R(g) in (p(1/vn) which means RCD(g) is biased yet consistent. The estimation error bound of ICD is analyzed in Theorem
Theorem 6. Based on the assumptions above, for any 8 > 0, the following inequality holds with probability at least 8:
In 2/6
R(Ten) - R(g") < 8L, ( + 1)V
4(L, 11CCA
(15)
2n
R(g"), ) since Rg, models with a bounded norm (Mohri et al. 2012) and A + O. Furthermore, the estimation error bound converges in O,(1/ which the optimal parametric rate for empirical risk minimization without additional assumptions (Mendelson 2008).
Theorem 6 elucidates that + og, R(OCD)
for all parametric
4 EXPERIMENTS
In this section, we verify the effectiveness of our proposed approaches experimentally.
4.1 EXPERIMENTAL SETUP
We conducted experiments on benchmark data sets, including MNIST (LeCun et al. 1998), Kuzushij-MNNIST (Clanuwat et al., 2018). Fashion-MNIIST (Xiao et al., (017). and CIFAR-10 Krizhevsky & Hinton, 2009). In addition, four UCI data sets (Dua & Graff) 2017) were used, including Optdigits, USPS, Pendigits, and Letter. Since the data sets
originally designed for multi-class classification, we manually partitioned them into binary classes. The detailed descrip- tions of data sets is illustrated in Appendix. For CIFAR-10, 10, we used ResNet-34 (He et al. 2016) as the model architecture. For other data sets, we used a multilayer perceptron (MLP) with three hidden layers of width 300 equipped with the ReLU (Nair & Hinton 2010) activation function and batch normalization (Toffe & Szegedy (015)). The logistic loss is utilized to instantiate the loss function ((.,)). It is worth noting that confidence difference is given by labelers in real-world applications, while it was generated synthetically in this paper to facilitate comprehensive experimental analysis. We firstly trained a probabilistic classifier via logistic regression with ordinarily labeled data and the same neural network architecture. Then, we sampled unlabeled data in pairs at random, and generated the class posterior probabilities by inputting them into the probabilistic classifier. After that, we generated confidence difference for each pair of sampled data according to Definition] !
In the experiments, we adopted the following variants of our proposed approaches: 1) ContDiff- Unbiased, which denotes the method working by minimizing the unbiased risk estimator proposed in Eq. (8: 2) ConfDiff- ReLU, which denotes the method working by minimizing the corrected risk estimator proposed in Eq. (12) with the ReLU function the risk correction function: 3) ConfDiff- ABS, which denotes the method working by minimizing the corrected risk estimator proposed in Eq. (1) with the absolute value function the risk correction function. We compared our proposed

---

Under review as a conference paper at ICLR 2023
Table 1: Classification accuracy (td) of each method on benchmark data with different class priors, where the best performance is shown in bold.
Class Prior
Method
MNIST
Kuzushiji
Fashion
CIFAR-IO
Pcomp-Unie Unbiased
0.761 100017
0.63710.052
0.737=0.050
0.776110.023
Poomp-Rel.U
0.800++.0000
0.800=0.000
0.800=0.000
0.800=+0.000
Poomp-ABS
0.800+0.000
0.800=000000
0.800=0.000
0.800=000000
7 = 0.2
Peomp- Teacher
0.965=000100
0.841=0.046
0.853=0.017
0.836=0.019
ContDiff-Unbiased
0.789=0.041
0.672=0.053
0.855=0.024
0.789=0.025
ConfDiff-Roo
0,968 003
0.860=0.017
0.964=0.004
0.844=0.020
ConIDiff-ABS
0.905
0.090
0.965
0.862
Class Prior
Method
MNIST
Kuzushiji
Fashion
CIFAR-10
Pcomp-Unbiased
0.712+00.00
0.578+0.036
0.723-0.042
0.703
Pcomp-Rel.U
0.502++0.033
0.502++0.04
0.500=0.000
0.602+000322
Pcomp-ABS
0.842:40.012
0.727=0.006
0.851=0.012
0.583=000118
74 = 0.5
Teacher
0,893#0.014
0.782=0.046
0.903=0.016
0.779=0.016
ConIDiff-Unbiased
0.911100046
0.712.10.066
0.896_0.036
0.720
ConIDiff-ReLU 0.94410.011
0..055
0.960-0.003
0.800
ConfDiff-ABS
0.964+0.001
0.867=0.006
0.967=0.001
0.843=0.004
Class Prior
Method
MNIST
Kuzushiji
Fashion
CIFAR-I0
(comp-Unbiasse
0.799100005
0.671
0.813=0.029
0.737=0.022
Peomp-ReLJ
0.910#0.031
0.775=0.022
0.897=0.023
0.851=0.010
Pcomp-ABS 0.85410.027
0.838=10.026
0.921=0.017
0.007
.T. = 0.8
Pcomp- #comp-Teacher 0.943
0.81410.027
0.936
0.000
ConfDiff-Unbiased 0.792#0.017
0.758+0.033
0.810=0.035
0.794=0.012
ConfDiff-ReLU 0.970#
0.886+0.009
0.970=0.002
0.851
ConfDiff-ABS 0.983+10002
0.915=0.001
0.975=0.002
0.874=0.011
approaches with the following approaches: 1) Pcomp-Unbiased, which denotes the method working by minimizing the unbiased risk estimator for Pcomp classification proposed in Feng et al. a (202T): 2)
Ccomp-Rel.U, which denotes the risk correction approach for Pcomp classification with the ReLU function as the risk correction function; 3) Pcomp-ABS, which denotes the risk correction approach for Pcomp classification with the absolute value function as the risk correction function; 4) Pcomp- Teacher, which denotes the state-of-the-an approach improving the label-noise learning approach RankPruning (Northcutt al. (017)) with consistency regularization.
The number of training epoches was set to 200 and we obtained the testing accuracy by averaging the results the last 10 epoches. The detailed hyperparameters can found in Appendix. To verify the effectiveness of our approaches under different class prior settings, we set π E (0.2, 0.5, 0.8) for all the data sets. For ease of implementation, we assumed that the class prior #4 was known for all the compared methods. We repeated the sampling-ande aang---nn--ttinnngg procedure for five times, and the mean accuracy as well as the standard deviation were recorded.
4.2 EXPERIMENTAL RESULTS
Benchmark data sets. Table reports detailed experimental results for all the compared methods on four benchmark data sets. Based Table [ we can draw the following conclusions: a) On all the cases of benchmark data sets, our proposed ConfDiff-ABS method achieves superior perfor- mance against all of the other compared approaches significantly, which validates the effectiveness of our approach in utilizing supervision information from confidence difference; b) Pcomp-Teachen achieves superior performance against all of the other Pcomp approaches by large margin. The excellent performance benefits from the effectiveness of consistency regularization for weakly su- pervised learning problems (Berthelot et al.) 2019 Li al. 2020 Wu et al., 2022 c) The risk cor- rection methods for ConfDiff classification, i.e. Re and ContDiff-ABS. achieve better performance against vantageous; d) It is worth noting that the classification results of ConfDiff-Re ReLU and ConfDiff-ABS have smaller variances than ConfDiff- monDiif-UUnbaseed Unbiased. demonstrates that the risk correction method can enhance the stability and robustness for ConfDiff classification.
Unbiiase, which elaborates that the risk correction technique is ad-

---

Under review as a conference paper at ICLR 2023
Table 2: Classification accuracy (mean-istd) of each method on UCI data sets with different class priors, where the best performance is shown in bold.
Class Prior
Method
Optdigits 0..016
USPS
Pendigits
Letter
Pcomp-Unbiased
0.721
0.743=0.057
0.757=0.022
Poomp-Rel.U
0.800++0000
0.800+0.000
0.800=0.000
0.800=+00000
Pcomp-ABS
0.800++0.001
0.800=000000
0.800=0.000
0.800=0.000
7 = 0.2
Peomp- Teacher
0,901
0.894=0.023
0.928=0.019
0.883=000066
ContDiff-Unbiased
0,831
0.840=0.078
0.865=0.079
0.732=00055
ConfDiff-Rell
0,953+0.014
0.957=0.007
0.987=0.003
0.929=#0.008
ConIDiff-ABS
0.963.10.09
0.960-0.005
0,988-0.002
0.942-1..007
Class Prior
Method
Optdigits
USPS
Pendigits
Letter
Pcomp-Unbiased
0.651+0.112
0.671+0.090
0.038
0.632++0.019
Pcomp-ReI.U
0.630+0.006
0.554+0.048
0.514=0.019
0.525+
Pcomp-ABS
0.787+0.031
0.814=0.018
0.793=0.017
0,748=0.031
44 = 0.5
Poomp-Teacher Teacher
0,890
0.860=0.012
0.883=0.018
0.864=0.024
ConiDiff-Unbiased
0.917100006
0.9361100010
0.945_0.052
0.755
ConIDiff-ReLU 0.92110.011
0.945100.00
0.981
1..06
ConfDiff-ABS
0.962
0.959-0.004
0.988-0.033
0.925=0.003
Class Prior
Method
Optdigits
USPS
Pendigits
Letter
Cobnp-Unbiased
0.765+0.023
0.746=0.012
0.743=0.026
0.694=0.031
Pcomp-ReLL
0,902=100017
0.8014
0.913=0.023
0.827=0.025
Pcomp-ABS 0.89411.001
0.879-10.009
0.911=0.009
0.870
.T = 0.8
Pcomp-Teache 0.918:10.007
0.933-10.023
0.008
0.872.10.011
ConfDiff-Unbiased 0.886#0.037
0.803=0.042
0.892=0.096
0.748=0.015
ConfDiff-ReLUU 0.949=0.007
0.958=00008
0.986=0.003
0.927=0.008
ConfDiff-ABS
0.964+0.005
0.964=0.003
0.987=0.002
0.945=0.007
Tectee
Teacer
(a) Kuzushiji
(b) Fashion
(c) USPS
(d) Letter
Figure I: Classification performance of ConfDiff- ReeL and ConfDiff-ABS given a fraction train- ing data well TeacheTeacher given 100% of training data (π+ 0.2).
UCI data sets. Table 2 reports detailed experimental results on four UCI data sets well. From Table []] we can observe that: a) On all the UCI data sets under different class prior probability settings, our proposed ConfDiff-ABS method achieves the best performance among all the com- pared approaches with significant superiority, which verifies the effectiveness of our proposed ap- proaches again; b) The performance of our proposed approaches is more stable than the compared Peomp approaches under different class prior probability settings, demonstrating the superiority of methods in dealing with various kinds of data distributions; e) ConfDiff-Unniisse has compara- ble performance against its risk correction variants on some data sets while has inferior performance on some other data sets. This is mainly because some data sets have simpler patterns and are thus less affected by overlitting issues.
4.3 PERFORMANCE WITH FEWER TRAINING DATA
To validate the effectiveness of exploiting the confidence difference, we conducted experiments by changing the fraction of training data for ConfDiff- Ren and ConfDiff-ABi (100% indicated that all the ConfDiff data were used for training). For comparison, Team-Teecchrr during the training process. Figure shows the results on four data sets with π = 0.2, and more experimental results can be found in Appendix. We can observe that the classification
used 100% of training data for
8

---

Under review as a conference paper at ICLR 2023
ocqunbcy
56
66
0.1
(a) ContDiff-Unbiaeed
(b) ConIDifi-ReLU
(c) ConfDift-ABS
acquRby
acounacy
11
0.1
(d) ConfDiff-Unbiase
(e) ConfDiff-Reed ReUU
(I) ContDiff-ABS
Figure 2: Classification accuracy on MNIST (the first row) and Pendigits (the second row) with
π 0.5 given inaccurate class prior probability and noisy confidence difference.
performance of our proposed approaches is still advantageous given fraction of training data. Our approaches can achieve superior or comparable performance even when only 10% of training data are used. It validates the benefit and effectiveness of leveraging the supervision information of the confidence difference.
4.4 ANALYSIS ON ROBUSTNESS
In this subsection, we investigate the influence of an inaccurate class prior probability and noisy confidence difference the generalization performance of the proposed approaches. Specifically, let . e denote the corrupted class prior probability with being a real number around 1. Let c/c denote the noisy confidence difference where ef sampled from normal distribution (1, o²). Figure 2 shows the classification performance of our proposed approaches on MNIST and Pendigits ( 0.5) with different e and a. We can observe that Con/Diff-ABS is more robust against corruptions compared with Con with # and é, varying in a reasonable range, the performance generally stable and even still superior against compared approaches. However, the performance degenerates with c 0.8 or e = 1.2 on some data sets, which indicates that is more important to obtain an accurate estimation of the class prior probability to facilitate model training.
and Con!DifT-ReLU It is demonstrated that
CONCLUSION
In this paper, we dived into novel weakly supervised learning setting where only unlabeled data pairs equipped with confidence difference were given. To solve the problem, an unbiased risk es- timator was derived to perform empirical risk minimization. An estimation error bound was estab- lished to show that the optimal parametric convergence rate could be achieved. Furthermore, a risk correction approach was introduced alleviate overfitting issues. Extensive experimental results validated the superiority of our proposed approaches. In future, it would promising to apply our approaches in real-world scenarios.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Han Bao, Gang Niu, and Masashi Sugiyama. Classification from pairwise similarity and unlabeled data. In Proceedings of the 35th International Conference on Machine Learning, pp. 461-470,
2018.
Han Bao, Takuya Shimada, Liyuan Xu, Issei Sato, and Masashi Sugiyama. Pairwise supervision can provably elicit decision boundary. In Proceedings of the 25th International Conference Artificial Intelligence and Statistics, pp. 2618- 2618-2640, 2022.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32, pp. 5050-5060, 2019.
Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. Learning rank using gradient descent. Proceedings of the 22nd International Conference on Machine Learning, pp. 89-96, 2005.
Yuzhou Cao, Lei Feng, Senlin Shu, Yitian Xu, Bo An, Gang Niu, and Masashi Sugiyama. Multi- class classification from single-class data with confidences. CoRR, 0as/2106.08886 2021a.
Yuzhou Cao, Lei Feng, Yitian Xu, Bo An, Gang Niu, and Masashi Sugiyama. Learning from Aimilariy-coonideen data. In Proceedings of the 38th International Conference Machine Learning. pp. 1272-1282, 2021b.
Zhe Cao, Tao Qin, Tie- Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: From pairwise approach to listwise approach. In Proceedings of the 24th International Conference on Machine Learning, pp. 129-136, 2007.
Olivier Chapelle, Bernhard Schölkopf, and Alexander Zien. Semi-Supervised Learning. The MIT Press, 2006.
Xi Chen, Paul N. Bennett, Kevyn Collins- Thommso, and Eric Horvitz. Pairwise ranking aggrega- tion in crowdsourced setting. In Proceedings of the 6th ACM International Conference on Web Search and Data Mining, pp. 193-202, 2013.
Tarin Clanuwat, Mikel Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David
Ha. Deep learning for classical japanese literature. CoRR, abs/1812.01777 12.01718, 2018
Timothee Cour, Ben Sapp, and Ben Taskar. Learning from partial labels. Journal of Machine Learning Research, 12(May):1501-1536 2011.
Marthinus C. du Plessis, Gang Niu, and Masashi Sugiyama. Analysis of learning from positive and unlabeled data. In Advances in Neural Information Processing Systems 27, pp. 70 2014.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL httthttpp//achi
ics. uci. mici.edu/m
Lei Feng, Senlin Shu, Nan Lu, Bo Han, Miao Xu, Gang Niu, Bo An, and Masashi Sugiyama. Pointwise binary classification with pairwise confidence comparisons. In Proceedings of the 38th International Conference on Machine Learning, pp. 3252-32622 2021.
Yanwei Fu, Timothy M. Hospedales, Tao Xiang, Jiechao Xiong, Shaogang Gong, Yizhou Wang, and Yuan Yao. Robust subjective visual property prediction from crowdsourced pairwise labels. IEEE Transactions on Puttern Analysis and Machine Intelligence, 38(3):563-577, 2015.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Proceedings of the 31st Conference On Learning Theory, pp. 297-299, 2018.
Bo Han, Quanming Yao. Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In 2018.
Advances in Neural Information Processing Systems 31, pp.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Yixuan He, Quan Gan, David Wipf, Gesine D. Reinert, Junchi Yan, and Mihai Cucuringu. Gn- nrank: Learning global rankings from pairwise comparisons via directed graph neural networks. In Proceedings of the 39th International Conference on Machine Learning, pp.
2022.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings the 32nd International Conference on Machine Learming, pp. 448 456, 2015.
Takashi Ishida, Gang Niu, and Masashi Sugiyama. Binary classification from positive-confidence data. In Advances in Neural Information Processing Systems 31, pp- 5921-5932, 2018.
Kevin G. Jamieson and Robert D. Nowak. Active ranking using pairwise comparisons. In Advances in Neural Information Processing Systems 24, pp. 2240-2248, 2011.
Daniel M. Kane, Shachar Lovett, Shay Moran, and Jiapeng Zhang. Active classification with com- parison queries. In 2017 IEEE 58th Annual Symposium on Foundations of Computer Science, pp- 355-366, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-25) -2224,, 1998.
Junnan Li, Richard Socher, and Steven C. H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In Proceedings of the 8th International Conference Learning Represen- tations, 2020.
Junnan Li, Caiming Xiong, and Steven C. H. Hoi. Mopro: Webly supervised learning with momen- tum prototypes. In Proceedings of the 9th International Conference on Learning Representations, 2021.
Yu-Feng Li and Zhi-Hua Zhou. Towards making unlabeled data never hurt. IEEE Transactions on Patlern Analysis and Machine Intelligence, 37(1):175-18) 2015.
Tie- Yan Liu. Learning to Rank for Information Retrieval. Springer, 2011.
Nan Lu, Gang Niu, Aditya K. Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. In Proceedings of the 7th International Conference Learning Representations, 2019.
Nan Lu, Tianyi Zhang, Gang Niu, and Masashi Sugiyama. Mitigating overfitting in supervised classification from two unlabeled datasets: A consistent risk correction approach. In Proceedings of the 23rd International Conference Artificial Intelligence and Statistics, pp. 1115-1125, 2020.
Shahar Mendelson. Lower bounds for the empirical minimization algorithm. IEEE Transactions on Information Theory, 54(8)33777-3800, 2008.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning- The MIT Press, 2012.
Vinod Nair and Geoffrey E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference Machine Learning, pp. 807-814, 2010.
Curtis G. Northeutt, Tailin Wu, and Isaac L. Chuang. Learning with confident examples: Rank pruning for robust classification with noisy labels. In Proceedings of the 33rd Conference on Uncertainty in Artificial Intelligence, 2017.
11

---

Under review as a conference paper at ICLR 2023
Dohyung Park, Joe Neeman, Jin Zhang. Sujay Sanghavi, and Inderjit S. Dhillon. Preference comple- tion: Large-scale collaborative ranking from pairwise comparisons. In Proceedings of the 32nd International Conference Machine Learning, pp. 1907-1916, 2015.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8026-8037, 2019.
Giorgio Patrini, Alessandro Rozza, Aditya K. Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Nihar B. Shah, Sivaraman Balakrishnan, and Martin J. Wainwright. Feeling the bern: Adaptive estimators for bernoulli probabilities of pairwise comparisons. IEEE Transactions on Information Theory, 65(8):4854 4874, 2019.
Kazuhiko Shinoda, Hirotaka Kaji, and Masashi Sugiyama. Binary classification from positive data with skewed confidence. In Proceedings of the 29th International Joint Conferences Artificial Intelligence, pp. 3328-3334, 2020.
Guangxin Su, Weitong Chen, and Miao Xu. Positive-unlabeled learning from imbalanced data. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pp. 2995-3001, 2021.
Deng-Bao Bao Wang. Yong Wen, Lujia Pan, and Min-Ling Zhang. Learning from noisy labels with com- plementary loss functions. In Proceedings of the 35th AAAI Conference Artificial Intelligence, pp. 10111-10119 2021.
Haobo Wang. Ruixuan Xiao, Sharon Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disambiguation for partial label learning. In Proceedings of the 10th Interna- tional Conference on Learning Representations, 2022.
Wei Wang and Min-Ling Zhang. Semi-supervised partial label learning via confidence-rated margin maximization. In Advances in Neural Information Processing Systems 33, pp. 6982-6993, 2020.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Hongwei Wen, Jingyi Cui, Hanyuan Hang. Jiabin Liu, Yisen Wang, and Zhouchen Lin. Leveraged weighted loss for partial label learning. Proceedings of the 38th International Conference on Machine Learning, pp. 11091-11100 2021.
Dong-Dong Wu, Deng-Bat Bao Wang, and Min-Ling Zhang. Revisiting consistency regularization for deep partial label learning. In Proceedings of the 39th International Conference Machine Learning, pp. 24212-24225, 2022.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: A novel image dataset for bench- marking machine learning algorithms. CoRR, abs/1708.00747, 2017.
Liyuan Xu, Junya Honda, Gang Niu, and Masashi Sugiyama. Uncoupled regression from pairwise comparison data. In Advances in Neural Information Processing Systems 32, pp. 4002-4002, 2019.
Yichong Xu, Hongyang Zhang, Kyle Miller, Aarti Singh, and Artur Dubrawski. Noise-tolerant interactive learning using pairwise comparisons. In Advances in Neural Information Processing Systems 30, pp. 2428-2437, 2017.
Yichong Xu, Sivaraman Balakrishnan, Aarti Singh, and Artur Dubrawski. Regression with compar- isons: Escaping the curse of dimensionality with ordinal information. The Journal of Machine Learning Research, 21 (1())6480065333 2020.
12

---

Under review as a conference paper at ICLR 2023
Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H. Chi, Steve Tjoa, Jieqi (Jay) Kang, and Evan Ettinger. Self-supervised learning for large-scale item recommendations. In Proceedings of the 30th ACM International Conference Information & Knowledge Management, pp. 4321-433333 2021.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Gang Niu, Masashi Sugiyama, and Dacheng Tao. Rethinking class-pii priss- estimation for positive-unlabeled learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Shiwei Zeng and Jie Shen. Efficient pac learning from the crowd with pairwise comparisons. In
Proceedings of the 39th International Conference on Machine Learning, pp. 25973-25993, 2022.
Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. ACM Computing Surveys, 52(1):1-38, 2019.
Xiaojin Zhu and Andrew B. Goldberg. Introduction cemi-supervised learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 3(1):1-130, 2009.
13

---

Under review as a conference paper at ICLR 2023
A PROOF OF THEOREM 1
Before giving the proof of Theorem begin with the following lemmas:
Lemma 2. The confidence difference c(ar, x!) can be equivalently expressed
dx,xx) T+p(x)p+ (x') T+ (2
(16)
p(x) ((z)p(x2')
(-p-(x)p(a') - (a')
(17)
(()))((x')
Proof. On one hand,
c(ar -ply' = 11(e) ply 1|x))
plo1,y 1// =1 p(æ, y =
p(xx)
p(x)
T+P+ (+P+(xx) (+)+(x)
p(x') p(x)
T+p(x))+(a** (T+P+(x)ppCC
p(x))((x')
On the other hand,
c(2, x') = ply' 1|x²) p④y 1 .r)
(1 p(y' 0 )) (1 ply- 0))))
pạy 0|z) ply 0|z')
p(x, y 0) p(a', y 0)
p(x)
p(x')
r-p- (x)
T- P- (x')
p(x) p(x')
x P- x-p-(x)ple (x)))((()))))))))))))))) (x) x-p(p)))) p(x)pppp1 (x')
p(x))
which concludes the proof.
Lemma 3. The following equations hold:
Ep( [[[ (( ((ox-=(( :::==,,,ppp- ([
(18)
En( + <(x,,")))())) = -()))
(19)
+( ( ((., =x,+)))))++ppp000)) [(0)) (20)
Expce,,r") ((Tr))))(([> c(x,, =()) ((0])
(21)
14

---

Under review as a conference paper at ICLR 2023
Proof. Firstly, the proof of Eq.
18 given:
ple ( - (o,,")))(((()
/ p())) + T-p-)))) ((g()) +1)p(( x') do da'
(x))
-/ / ( - (ran
dx'
= x=p(a)((g(())+t)) dx / p(xx) dx' - T=p(z)((g((), +1) dz / p+(2z)) da'
T=p=(x)((o(m)) + dac p(a')
m=pla))(lgo))+++1))))) T+p(a))(((())) +1) dæ + (=))()))((())) +1)
T+P+
- ((g(()) +1)]
After that, the proof of Eq.
19 is given:
Ep pl ( [[TT dx, (((ll))) -1)
/
+ ( ( -1)(=),-))p(2,00)) x') da dx
(x))p(=)) p(x)p((x')
/
+ + (() - da dæ'
/ T- (=)())((((xx -1) 1)) dar da' (x) - da / ppla' da'
TT pl=))(((()) -1) dæ P- (=(()')
T T=)())(((()) 11 do + (=) -1) dæ ==p())((((x)) -1) dæ
T-P (x) -1) dx
(m)((())))( -1)].
It can be noticed that c(x. x') c(x',_____ xx) and p(x, x') p(æ', x). Therefore, it can be deduced
naturally that e I c(x x' (()()) +1]] = Ep(ar .m)) [ + c(x', x (g)))(())) Be-
cause æ and x' are symmetric, we can swap them and deduce Eq. (20) Eq. 21 can be deduced in
the same manner, which concludes the proof.
Based on Lemma the proof of Theorem is given.
Proof of Theorem 7 To begin with, it can be noticed that Ep (()(((())))+)))
Ep far) [((g(='), +1] and E (a) [((g()) 1]] Bp (wr) (((((())))) - 1)} Then, by summing
(x)
up all the equations from Eq. (18) to Eq. (2T). we can get the following equation:
E Ep(a, (C g(x')) + L- (o(x))g(xx))
2n 2 E, (p) +1)] + 2n 2 (e)) ((g(()))) 1)]
After dividing each side of the equation above by 2, we obtain Theorem
15

---

Under review as a conference paper at ICLR 2023
B ANALYSIS ON VARIANCE OF RISK ESTIMATOR
B.1 PROOF OF LEMMA T
Based on Lemma it can be observed that
E Ep(mr,m)) (1xx, x']] = -pl [(+ -
+t) + (H_ - cx, x (((()))))))))))))))))))))))))))) -1)]
(o) T (=))(((((2'), -1)]
+ E_Ep (m) -1)l
-R(g)
and
Ep(ar,r
c(z, (14 = _T_Epp Pp- (a) [[](()) - 1)] + ग. (e [(g(z)))))
+1)+ (fr
(a)) - 1)] + ग. (() +1)]
= =R(g).
Therefore, for an arbitrary weight E [0,
R(g) =aR(gg) + (1 (1-a) B))
p(lpp(a |[(x,, x² x²)] + (1 - a) ( x)},
which indicates that
1 Σ (aL(x, 21) + (1 - aa) xz))
π 1=1
is also an unbiased risk estimator and concludes the proof.
B.2 PROOF OF THEBRRE 2
In this subsection, we show that Eq.
(8 achieves the minimum variance of
1 Σ (aL a²)) + (1 - ())(
S(g: ax)
n i=1
w.r.t. any a E [0, 1]. To begin with, we introduce the following notations:
1 Σ (()),
11 A Ep
xz)
L(a,, xf)) = Ex(cr,
1 n
12 A E Furthermore, according to Lemma
(
Σ C(x,, a!)) Σ C(xl, xx)].
(22)
we have
Em(m. [S(g: a)] = R(g).
Then, we provide the proof of Theorem as follows.
Proof of Theorem
2
Var(S(g;on
= Ep (S(g)) R(g))
= =Eple.e (g; R(g)²
=a a 2 ) C(x;, x()) + (1 a) Eppe
[(= Σ C C(x,, ax
7
+ +2l E)pe,r, 1/2 Σ C(xi, ar)) Σ
1
L(xr,, ax:)] R(g)²
1=1
=m1a² + 11((1 a)² + +2uzaa a) - R(g)²
=(2m 2a 1 1 + (w) + +22) R(g)²
1
2
2
16

---

Under review as a conference paper at ICLR 2023
Besides, it can be observed that
2μ1 2μ22 = Em Σ
(Ce, a//) Liz), (c)))]] 2 0.
Therefore, Var(S(g:: a)) achieves the minimum value when 1/2, which concludes the proof.
C PROOF OF THEOREM 3
To begin with, we give the definition of Rademacher complexity.
Definition 2 (Rademacher complexity). Let Xn {z),. "........y In denote nt i.i.d. random variables drawn from a probability distribution with density p(z), g وg X + R} denote class of
measurable functions, and a = (01. T2.-.., Tn) denote Rademacher variables taking values from (+1,-1} uniformly. Then, the (expected) Rademacher complexity of G is defined
Rr (G) = Ex, Eg
sup gEG
aig(
(23)
1=1
Let Dn ltd. p(a, x') denote n pairs of ConfDiff data and (c (L(x, x') + L(x', m2)/2, then we introduce the following lemma.
Lemma 4.
RA (CDD 99) S (9),
where LCD o G = {LCD glg E 9) and (.) is the Rademacher complexity over ConfDiff data pairs Dn of size n.
Proof.
1
Ht (CCD 09) =RD,
E. sup - gEg n
o, Lcn
n
-Ep, Eo (sup
[o (G)((=)))(((())))++1))) ()) -1)
-
gEg 2n
+ + + +
(TT_ (
Then, we can induce that
| WVLlllg 2,, 2/)
T - ())(g(Z))) +1) + (T- -c))(()()) -1)
2
(+ + ()) +1) + (T- +i)e((((2)) -1) llle
(T+ c: (l(((()))))))))))))))))))))))(
+1)
( -c))ll(())) -1) )|12
</<V(
)|12 + ||
2
(T4 + tc,
())(((())))))))))))))))))))))))) +1)
/ (T- - (;))((((;))) -1) )/1
|/
)|12 |
2
2
|+ Toe
La
| + cil Te | +ce
(24)
2
Suppose π4 2 T_, the value of RHS of Eq. E 1-1 -T) the value is Lec when
24 can determined as follows: when
-T_), the value is (ΛT c;) when
e E [- ). the value is LG; when e; E ,T,,,,) the value is (T + Lx when e E [ 1],
the value 2c Le. To sum up, when n. .-, the value of RHS of Eq. (24) is less than 2
17

---

Under review as a conference paper at ICLR 2023
When π I --, deduce that the value of RHS of Eq. 24 is less than 2 in the same way. Therefore, RR,CCD gg 22 Eo (sup
og([4)]
Ngg
=2,, E (sup DEG T
(9),
which concludes the proof.
After that, introduce the following lemma.
Lemma 5. The inequality below hold with probability at least
2n2/6
sup R(g) - RRc(())) & 4L, + 2C2/ gEG
2n
Proof. To begin with, we introduce = supgeg (R(g) Ren (g)) and + supgeg (u(g)
RcD(g)), where (g) and (g) denote the empirical risk over two sets of training examples with exactly one different point {(a,, x(), ) and {(ā,, Et), c( a{)) respectively. Then we have
+ S sup(Rec(g) RCD(g))
sEG
LCD (g: o,, xt) LCD (g: Tz. F.))
sup
n
2Ca n
Accordingly, & H can be bounded in the same way. The following inequalities holds with proba- bility at least 1 5/2 by applying McDiarmid's inequality:
2n2
sup ((() - RCD(g)) S ED, [sup(R(g) ( gEG
gEG
2n
Furthermore, we can bound DE Jsupgeg (R(gg) RoD(g))) with Rademacher complexity. is routine work to show by symmetrization Mohri et al. 2012) that
Ep, (sup((((()) - RgD(g))] < 29R, (COD - 9) - (9)
9E9
where the second inequality is from Lemma Accordingly, sup (CD(g)) - R(g)) has the same bound. By using the union bound, the following inequality holds with probability at least 8:
2/2
sup R(g) - ReD(g)) < (L. 2C,
sEG
2n
which concludes the proof.
Finally, the proof of Theorem is provided.
Proof of Theurem
R(ğen) R(g")
(R(ğen) RAD (TCD) + (Tc)) Ren(g")) (Ren(g") - R(g"))
(RGGCD) RoO (CD)) )} + (Rc R(g"))
R(ÍcD) ReD (TCD) + ReD((") R(gg))
s suu RRg) RcD((g)
geg
In2/0
SL + 4CpV
2n
18

---

Under review as a conference paper at ICLR 2023
The first inequality is derived because ÍCD is the minimizer of (cD The last inequality is derived according to Lemma 5
which concludes the proof.
D PROOF OF THEOREM
To begin with, we provide the following inequality:
RCD(g) - RcD(
sup
gEg
(++ - -4 + a
- -1)]
2n
i=1
+
(++ - -T +ā +
1 1
π4 + - -00 -q -1)
2n
+ [
- x4 + 0
1)))
1
(I n + -4 -1)
2n
i=1
+ |
x + ē. - c
-1))
1
- -()
-
2n
+ (| - x4| + . - -a) +1) + -f_- -1))
1 "
(( + c. 2
- (()g((z)) -1)
2n
+
(|
x4| + - -0-c))/2 cl))((((())))))))))))))))))))))) +1) +
- - -1=+== -1)
2C, Σ [a csl
n + 2C, / - *+|
Then, deduce the following inequality:
R(ğcD) R(g") =(R(gcD) RCD(CCD)) (ReD(GeD) RCD(CCD)) + (ReD(GeD) RCD
+ ((CcD(CC)) (ācD)) + (ReD(ŌCD) - R(ŌCD)) + (R(ōen) - R(g"))
susup [R(g) RCD(g)) sup [gD RCD( + (R(ğcd) (((g))
gEG
3E
<usup [R(g) RCD(()) sup RCD(g) RCD(g) gEg
9Eg
In 2/2
4C₁
Di
i cal + 4Cp/=+ - x+1-
$16LA,, + 8C,
+
2n
n
The first inequality is derived because JCD is the minimizer of R(g). The second and third inequality respectively.
derived according the proof of Theorem and Lemma 5
E PROOF OF THEOREM 5
To begin with, let (g) (D, on B(g) 2 (g) 2 on D()) 2 0} and D (g)
(Da) (D))A(g) < U B(g) s (g) < ou D(g) 0}. Before giving the proof of Theorem we give the following lemma based on the assumptions in section 3
Lemma 6. The probability measure of D,, (g) can be bounded as follows:
2nnn
222nn n1
22
22
(() - exp
+ exp
+ exp
(
+ +exp
(25)
C7
C2
C/
C7
19

---

Under review as a conference paper at ICLR 2023
Proof. It can be observed that
p(D.) - -(( p( 2en))
= =p(z1) ... p(== )p(z1) .. p(z2)))
Therefore, the probability measure P(2, (1)) can be defined as follows:
P(D /
p(n,) dD,,
(D,,) dæ ... dx, dxy day
(.)
When exactly one ConfDiff data pair in Sn is replaced, the change of A(g), B(g), Ogg) and D(g)
will be more than Cz/n. By applying McDiarmid's inequality, we can obtain the following inequalities:
-2a2n
P(B[A(g) A(g) 2 a) S exp
C
-262n n
F(E[B(O) B(g) 2 b) < exp
C2
-202n
P(E[(() ((9) 2 c) S exp
C2
22d
P(ED() ( d) s exp
C2
Furthermore,
+ +[E
c) +() D(g) 2 d)
22
266 C2
-2c31
2n
sexp C2
exp(
+ exp
exp
C²
C
which concludes the proof.
Then, the proof of Theorem given.
Proof of Theorem
To begin with, we prove the first inequality in Theorem
E B (g) R(g)
-E (g) - Ren(g))
=/
(Rep(g) (cD(g))p(P.)) dD,
)..eD(()
+ () (Rep(g) - ReD(g)) dD,
(ReD(g) Ren()))((D.) dD, 2 0,
(.E
20

---

Under review as a conference paper at ICLR 2023
where the last inequality is derived because (g)(g) is an upper bound of RCD(g). Furthermore, E[Rcn(g)] - R(gg)
(Ren(g) Ren())))(D.))
(..ED.(0)
< sup (ReD(O) RcD(O)) (2) DrED.(0) sup (ReD(0) (cD())))(27(G)))
dD,
D.,EN(())
([(Ag))++ ((0+0)))+++++
= sup
D.EDT(W)
-A(g) - B(g)) -()) (9))
< sup
(o)
[((g) + [()) + +
L +1 1
sup
T ()) +1)1
c;)((((())) -m)oo()((( -1)1
(T-
2n
(g)
Σ + Ci
+ Σ (()((a)) - -11))(7))11))(777())
I,f +1 1
a
< sup D.EEN(()))
Σ
( - ())(((())) +1)1 + Σ (T - -
2n
n
Σ
(T+ ())(g(<()) +1)1 + Σ (T- + -
1 1
sup
Σ
- - +1) ( c=) 1)
2n
i=1
+[TT a) (g(((())))))))))))))))))))))))))
| -1
(L, + 1)C²
sup D.E07
|T cal |a| |+ +al + パ cs)) (9)).
2n
Similar to the proof of Theorem
3
obtain
|+ cil + | csl + |T+ + cil + m + cal < 4.
Therefore, we have
E[RcD(g)] - R(g) 2(L, + 1)C,A,
which concludes the proof of the first inequality in Theorem Before giving the proof of the second
inequality, we give the upper bound of [R) E E[RCD [g)]). When exactly one ConfDiff data pair D, replaced, the change of ReD(g) no more than 2C /n. By applying McDiarmid's inequality, have the following inequalities with probability at least 1 8/2:
In 2/8
(CD(g) [[(( < 2C2Ls
2n
In 2/8 2n
B[RcD(g)) ReD(g) < 2Cebss
Therefore, with probability at least 1 8, we have
2/2
[g) ([)l < 2C,L,y
2n
21

---

Under review as a conference paper at ICLR 2023
Table 3: Characteristics of experimental data sets.
Data Set
# Train
# Test
Features
# Class Labels
Model
MNIST
60,000
10,000
784
MLP
Kuzushiji
60,000
10,000
784
10
MLP
Fashion
60,000
10,000
784
MLP
CIFAR-10
50,000
10,000
3,072
10
ResNet-34
Optdigits
4,495
1,125
62
MLP
USPS
7,437
1,861
256
10
MLP
Pendigits
8,793
2,199
16
10
MLP
Letter
16,000
4,000
16
26
MLP
Finally, we have
(C) - R(gg) RcD(g) [[ R(g)
< Ren(g) E| E[Rcn + [E(en R(0)
ReD(g) E[ (g)) + R()c R(O)
2n2
- 2C2
+ 2(L, 1)
(26)
2n
with probability at least
6, which concludes the proof.
F
PROOF OF THEOREM 6
With probability least 1 6, we have
R(ğcD) - R((g)) -R((CcD) - (ŌCD)) + (ĀeD( RCD(ŌCD))
+ (RCD((CD) ) R(īcD)) (R(ōcd) R(g"))
S|R(COD)
+ (RCDD@Cu - R(ŌcD)) + (R(Ācc) - R(g"))
In 2/5 4(L + 1)C,A + (9)
<4C,(Lg + 1)
2n
The first inequality derived because JCD the minimizer of RCD(g)- The second inequality is The proof is completed.
derived from Theorem 5 and Theorem
L
G ADDITIONAL INFORMATION ON EXPERIMENTS
In this section, the details of experimental data and hyperparameters provided.
G.1 DETAILS EXPERIMENTAL DATA SETS
The detailed statistics and corresponding model architectures are summarized in Table 3 while the basic information, sources and data split details are elaborated this subsection.
For the four benchmark data sets,
MNIST (LeCun al., 1998): It is grayscale handwritten digits recognition data set. It is com- posed of 60,000 training examples and 10,000 test examples. The original feature dimension is 28*28, and the label space 0-9, The even digits are regarded as the positive class while the odd digits are regarded the negative class. We sampled 15,000 unlabeled data pairs training data. The data set can be downloaded from http: //ap::///an.../ lecun. com/ eexeb/mniisss.
Mizzusiji-MNISS (Clanuwat et al.) 2018): It is grayscale Japanese character recognition data set. composed of 60,000 training examples and 10,000 test examples. The original feature dimension is 28*28, and the label space is {'o", "su", 'na", 'ma', 're', 'ki". "tsu", 'ha', 'ya", 'wo').
22

---

be downloaded from Under review as a conference paper at ICLR 2023
The positive class is composed of 'o', "su','na', 'ma', and 're' while the negative class is com- posed of 'ki', "tsu", 'haa, 'ya', and "wo'. We sampled 13,000 unlabeled data pairs as training data. cww.h/hhhhkaaaa weroddh//niiss kmnist
The data set can be downloaded from https:::////... /ttt:::///ithb.... com/ Pashion-MNIS" (Xiao et al.] 2017): It is a grayscale fashion item recognition data set. It is composed of 60,000 training examples and 10,000 test examples. The original feature dimension is 28*28, and the label space is "T--hhir', 'trouser', 'pullover', 'dress', 'sandal', 'coat', "shirt', "sneaker', 'bag', 'ankle boot'); J. The positive class composed of 'T-shirt', 'pullover'. 'coat', "shirt', and 'bag' while the negative class is composed of 'trouser', 'dress', 'sandal', 'sneaker', and "ankle boot'. We sampled 15,000 unlabeled data pairs as training data. The data set can be downloaded from https: //ss::///ittttt
com za aadd
fashi on mn ist com-
CIFAR-10 (Krizhevsky & Hinton| 2009): It is a colorful object recognition data set. posed of 50,000 training examples and 10,000 test examples. The original feature dimension is 32*32*3, and the label space ("airplane'. 'bird', "automobile', "cat', 'deer'. 'dog'. 'frog', 'horse', 'ship', 'truck'). The positive class is composed of 'bird', 'deer', 'dog'. 'frog'. 'cat', and 'horse" while the negative class is composed of 'airplane', 'automobile', 'ship', and 'truck'. We sampled 10,000 unlabeled data pairs training data. The data https:/ //wt::::/wwww.. www. cs. toronto. edu/ weduuukriiz kriii/iiiii //cifaaarrrrrr html
For the four UCI data sets, they can downloaded from Dua & Graff (2017)
Optdigits, USPS, Pendigits (Dua & Graff (017): They are handwritten digit recognition data
set. The train-test split can be found in Table 3 The feature dimensions are 62, 256, and 16 respectively and the label space is 0-9, The even digits are regarded as the positive class while the odd digits are regarded as the negative class. We sampled 1,200, 2,000, and 2,500 unlabeled data
pairs for training respectively.
Letter (Dua & Graft 2017]: is a letter recognition data set. It is composed of 16,000 training examples and 4,000 test examples. The feature dimension is 16 and the lahel space is the 26 capital letters in the English alphabet. The positive class is composed of the top 13 letters while the negative class is composed of the latter 13 letters. We sampled 4,000 unlabeled data pairs for training.
G.2 DETAILS OF HYPERPARAMETER
All the methods were implemented in Pytorch (Paszke et al. 2019). We used the Adam optimizer (Kingma & Ba 2015). To ensure fair comparisons, We set the same hyperparameter values for all the comparing approaches.
For MNIST, Kuzushiji-MNIS" and Rushion-MNISS the learning rate set le-3 and the weight decay was set to le-5. The batch size was set to 256 data pairs. For training the probabilistic classifier generate confidence, the batch size was set to 256 and the epoch number was set to 10.
For CIFAR1O, 10, the learning rate was set to 5e-4 and the weight decay was set to 1e-5. The batch size was set to 128 data pairs. For training the probabilistic classifier to generate confidence, the batch size was set to 128 and the epoch number was set to 10.
For all the UCI data sets, the learning rate was to le-3 and the weight decay was set to le-5. The batch size was set to 128 data pairs. For training the probabilistic classifier to generate confidence, the batch size was set to 128 and the epoch number was set 10.
The learning rate and weight decay for training the probabilistic classifier were the same as the setting for each data set correspondingly.
H MORE EXPERIMENTAL RESULTS WITH FEWER TRAINING DATA
23

---

Under review as a conference paper at ICLR 2023
Puaey Traner
Teamen
42%
1986
100
30x
13% 60%
(a) Optdigits
(b) Pendigits
(c) MNIST
(d) CIFAR-10
3:00
Contoit cata
1906
Contort cara
20%
Canto duz
43%
Carta daa
Kuzushiji
Fashion
USPS
(h) Letter
ContDit data
ContDitt mara
Can mirtie
(i) Optdigits
Pendigits
(k) MNIST
( CIFAR-10
RAU Sertin
(m) Kuzushiji
(n) Fashion
(0) USPS
(p) Letter
Celonvss Teater
(q) Optdigits
(r) Pendigits
(s) MNIST
(1 CIFAR-10
Figure 3: Classification performance of ConfDiff- ReLU and ConfDiff-ABS given a fraction of training data well as Pcomp-Teaches Teacher given 100% of training data with different prior settings
(T+ = 0.2 for the first row, ग. = 0.5 for the second and the third row, and : = 0.8 for the fourth and the fifth row).
24


Reviews:
[{"review_id": "ewtfnMNoXo", "reviewer": "Reviewer_sacA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to learn from observing probability differences between pairs of labels, in a binary classification setting. The premise is that obtaining such information is sometimes easier than obtaining single probability estimate, and richer than simple qualitative comparisons. \n\nThe paper demonstrates that such information is sufficient to converge towards the optimal risk, deals with the case of corrupted estimates, and also propose a correction to avoid the loss becoming negative through a ReLu correction. \n\nSome experiments are done, in particular to compare with a qualitative feedback. ", "summary_of_the_review": "The paper proposes a new learning framework considering as supervision differences of probabilities. While the theoretical analysis is interesting, it is unclear how practical this framework is, nor how realistic is the assumption that we can have good estimates of probability differences. It is also unclear how the framework could extend beyond binary classification. ", "strength_and_weaknesses": "+: the paper is clearly written, and I think the proofs are correct (I am less familiar with generalisation bounds, so I cannot give strong guarantees for Theorem 3 and the next ones, but the reasoning seems ok to me). \n\n+: As far as I know, the proposed framework is original\n\n-: it is not clear at all to me, despite the two high-level examples given in the introduction, that providing accurate probability differences for pairs of data is much easier/simpler than providing single probability estimates. While I would agree that providing qualitative comparison (Pcomp framework) is easier, it is not so clear for probability differences. And the argument put forward later on that different experts may disagree on probabilistic estimates is equally true for probabilistic differences. Do we have actual evidence (and not illustrative example, that I have a hard time really figuring out) that this is indeed the case in some applications? Maybe the authors could elaborate a bit more on their examples to provide a fuller story, where the estimates are explicitly mentioned?\n\n-: while Pcomp and Pconf can be adapted easily to multi-class setting, it seems intuitively harder to adapt the current proposal to more than two classes, both from a theoretical perspective but also from a practical perspective (it seems very demanding and hard to elicit probability differences over all pair of classes). Could some comments be provided with respect to that? \n\n-: I do not understand why the framework is not compared, in Table 1 (or in appendices tables) to both Pconf and the framework where data are fully observed (rather than using various Pcomp variants).\n\n-(minor): I wonder why in the test non-binary data sets were turned into binary data sets. UCI and other repositories contain plenty of native binary data sets, and it is unclear why those particular data sets were chosen, and why the separation was chosen the way it was? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "c9sCOvl5iV", "reviewer": "Reviewer_dSMd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a novel learning problem for binary classifications, namely Confidence-Difference (ConfDiff) classification, in which the learner is only given unlabeled data pairs $x,x'$ equipped with confidence difference specifying the difference in the probabilities of being positive $P(y'=1|x') - P(y=1|x)$. The authors proposed to solve the problem via ERM by constructing an unbiased risk estimator. Estimation error bound was derived, and the robustness of the risk estimator was analyzed. Experiments demonstrated effectiveness of the proposed method.", "summary_of_the_review": "I vote for a weak accept as I think the paper overall meets the standard, and its contributions outweigh its shortcomings.", "strength_and_weaknesses": "**Strong points.**\n\nThe proposed learning problem is interesting. The proposed method is backed by strong theoretical analyses and empirical results. The paper is well-written and easy to understand.\n\n**Weak points.**\n\nThe concern I have is that whether the proposed learning problem is a valid weakly supervised learning paradigm. Confidence labels (posterior class probabilities $P(Y=1|X)$) certainly contains more information than the usual labels. On the other hand, unlabeled data pairs and differences somewhat constitute a weaker learning problem. Thus, to me, it is not entirely clear whether learning with unlabeled data pairs and confidence differences is a weaker or stronger learning problem.\n\nAlso, in the experiments, Pcomp was compared against. However, as mentioned in Section 2.3, in Pcomp classification, the learner only knows unlabeled pairs, and within each pair $x,x'$, which one is more likely to have positive label than the other (whether $P(y'=1|x') > P(y=1|x)$). Clearly, ConfDiff classification has access to more information. Thus, it is expected that ConfDiff can outperform Pcomp. Nevertheless, it is okay to use Pcomp as a baseline.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4Hgpg9Ltr_", "reviewer": "Reviewer_bMCm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies a weakly supervised learning setting, in which one has limited access to the *confidence labels* of the training examples. Previous work in this line includes the setup with pointwise confidence scores (Pconf), and the setup where pairwise comparisons of the confidence scores are available (Pcomp). This work presumes more fine-grained information than the latter: Unlabeled data pairs with confidence difference (ConfDiff). This learning setup is then formulated as empirical risk minimization and a corresponding unbiased risk estimator is constructed, together with an estimation error bound.\n\n", "summary_of_the_review": "As mentioned earlier, my key issue of this paper is that I don't think the ConfDiff setting is realistic. \n\nIn terms of the level of information required in training, ConfDiff sits in between Pconf and Pcomp, in theory. In practice, however, I don't see a situation where one obtains the exact difference between two confidence scores, **without** first estimating the pointwise confidence scores. Even the experiments in the paper have to first do the point estimation. \n\nIt was mentioned in the paper that *the confidence difference is given by annotators in real-world applications*. Has such annotation procedure ever actually applied in the real world? We have to realize that it is extremely hard for a human annotator to give exact confidence difference between two examples. I would say the annotation settings of Pconf and Pcomp are, comparably speaking, more realistic. In the case of former, each annotation provides more information. (Also I'd say the annotator might have to do pointwise estimates first before giving exact confidence differences). In the case of latter, the annotation is much simpler for the annotator as it is only a qualitative paired comparison.\n\nIn any case, I think the paper should motivate the proposed setting better, ideally with some real-world applications. ", "strength_and_weaknesses": "### Strength  \n\n- The paper is technically well formulated. The proposed learning setting was rigorously set up and an unbiased risk estimator is derived for the empirical risk minimization. \n- The paper is well written and easy to follow. Even people outside this particular field should be able to grasp the general idea proposed therein. \n\n### Weaknesses\n\n- In my opinion, the biggest weakness of this paper is its setting. Is the setting realistic? The paper doesn't provide sufficient motivation in the introduction; it also lacks realistic experimental setup to support the ConfDiff setting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject