Peer Loss Functions: Learning from Noisy Labels without Knowing Noise Rates
Authors: Yang Liu
Hongyi Guo
Venue:  (2020)
Keywords: learning with noisy labels
empirical risk minimization
peer loss

Abstract: Learning with noisy labels is a common problem in supervised learning. Existing approaches require practitioners to specify noise rates, i.e., a set of parameters controlling the severity of label noises in the problem. In this work, we introduce a technique to learn from noisy labels that does not require a priori specification of the noise rates. In particular, we introduce a new family of loss functions that we name as peer loss functions. Our approach then uses a standard empirical risk minimization (ERM) framework with peer loss functions. Peer loss functions associate each training sample with a certain form of "peer" samples, which evaluate a classifier' predictions jointly. We show that, under mild conditions, performing ERM with peer loss functions on the noisy dataset leads to the optimal or a near optimal classifier as if performing ERM over the clean training data, which we do not have access to. To our best knowledge, this is the first result on "learning with noisy labels without knowing noise rates" with theoretical guarantees. We pair our results with an extensive set of experiments, where we compare with state-of-the-art techniques of learning with noisy labels. Our results show that peer loss functions based method consistently outperforms the baseline benchmarks. Peer loss provides a way to simplify model development when facing potentially noisy training labels, and can be promoted as a robust candidate loss function in such situations. 

Under review as a conference paper at ICLR 2020
PEER LOSS FUNCTIONS: LEARNING FROM NOISY LA- BELS WITHOUT KNOWING NOISE RATES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning with noisy labels is a common problem in supervised learning. Existing approaches require practitioners to specify noise rates, i.e., a set of parameters controlling the severity of label noises in the problem. The specifications are ei- ther assumed to be given or estimated using additional approaches. In this work, we introduce technique to learn from noisy labels that does not require a priori specification of the noise rates. In particular, we introduce new family of loss functions that we name as peer loss functions. Our approach then uses standard empirical risk minimization (ERM) framework with peer loss functions. Peer loss functions associate each training sample with certain form of "peer" samples, which evaluate classifier' predictions jointly. We show that, under mild condi- tions, performing ERM with peer loss functions on the noisy dataset leads to the optimal or a near optimal elassifier if performing ERM over the clean training data, which we do not have access to. To our best knowledge, this is the first re- sult on "learning with noisy labels without knowing noise rates" with theoretical guarantees. We pair our results with an extensive set of experiments, where we compare with state-of-the-arr techniques of learning with noisy labels. Our results show that peer loss functions based method consistently outperforms the baseline benchmarks, as well as some recent new results. Peer loss provides a way to sim- plify model development when facing potentially noisy training labels, and can be promoted as a robust candidate loss function such situations.
INTRODUCTION
The quality of supervised learning models depends on the training data ((r,, yn) yn) In practice, label noise can arise due to host of reasons. For instance, the observed labels E.s may represent human observations of a ground truth label. In this case, human annotators may observe the label imperfectly due to differing degrees of expertise measurement error, see e.g., medical examples such as labeling MRI images from patients. Many prior approaches to this problem the machine learning literature aim to develop algorithms to learn models that are robust to label noise (By (Bylander 1994 Sesa-Bianchi et al.) 1999. 2011; Ben--avvdd et al.) Scott et al.] 2013 Natarajan et al. 2013 Scott, 2015). Typical approaches require priori knowledge of noise rates, I.e., a of parameters that control the severity of label noise. Working with unknown noise rates is difficult in practice: Often, one must estimate the noise rates from data, which may require additional data collection (Natarajan et al., 2013); Scott 2015 Van Rooyen et al. 2015) (e.g.. be a redundant set of noisy labels for each sample point, set of ground truth labels for tuning these parameters) and may introduce estimation error that can affect the final model in less predictable ways. Our main goal is to provide an alternative that does not require the specification of the noise rates, nor an additional estimation step for the noises. This target solution might help when the practitioner does not have access to reliable estimates of the noise rates (e.g., when the training data has limited size for the estimation tasks, when the training data is already collected in form that makes the estimation hard to perform).
In this paper, we introduce a new family of loss functions, peer loss functions, to empirical risk minimization (ERM), for a broad class of learning with noisy labels problems. Peer loss functions operate under different noise rates without requiring either priori knowledge of the embedded noise rates, or an estimation procedure. This family of loss functions builds on approaches developed in the peer prediction literature (Miller al. 2005
Dasgupta & Ghosh] 2013 Shnayder et al. 2016).

---

Under review as a conference paper at ICLR 2020
which studies how elicit information from self-interested agents without verification. Typical approaches in the peer prediction literature design scoring functions to score each reported data using another noisy reference answer, without accessing ground truth information. We borrow this idea and the associated scoring functions via making a connection through treating each classifier's agent's private information be elicited and evaluated, and the noisy label as an imperfect reference from a "noisy label agent", The peer loss takes form of evaluating classifiers' prediction using noisy labels on both the targeted samples and a particular form of constructed "peer" samples. The evaluation on the constructed peer sample encodes implicitly the information about the noises as well as the underlying true labels, which helps us offset the effects of label noises. The peer sample evaluation returns us favorable property that expected risk of peer loss turns to be other
prediction
an affine transformation of the true risk of the classifier defined on the clean distribution. words, peer loss is invariant to label noises when optimizing with it. This effect helps us get rid of the estimation of noise rates.
The main contributions of this work are:
1.
We propose new family of loss functions that can easily adapt to existing ERM framework that robust to asymmetric label noises with formal theoretical guarantees and ii) requires no prior knowledge or estimation of the noise rates (no need for specifying noise rates). We believe having the second feature above is non-trivial progress, and it features a promising solution deploy in an unknown noisy training environment.
2. We present formal results showing that performing ERM with a peer loss function can recover an optimal, or near optimal classifier f as if performing ERM on the clean data (Theorem 2 We also provide analysis for peer loss functions' risk guarantees (Theorem and 7
3. We present extensive experimental results to validate the usefulness of peer loss (Section 5 and Appendix). This result is encouraging as it is able remove the long-standing requirement of learning error rates of noises (or estimating transition matrices as used in many relevant papers) before many of the existing methods can be applied. We also provide preliminary results on how peer loss generalizes to multi-class classification problems.
4. We will contribute to the community by publishing our codes and implementations.
1.1 RELATED WORK
Learning from Noisy Labels Our work fits within a stream of research on learning with noisy labels. A large stream of research on this topic works with the random classification noise (RCN) model, where observed labels are flipped independently with probability E [0, 2
11 (Bylander 1994.
Cesa-Bianchi et al., 1999; 2011 Ben-David et al.). Recently, learning with asymmetric noisy data (or also referred as class-conditional random classification noise (CCN)) for binary classification problems has been rigorously studied in (Stempfel & Ralaivola 2009; Scott et al.) 2013 Natarajan ctt al. 2013 Scott 20155] Van Rooyen et al. 2015]] Menon et al. 2015). For a more thorough survey
of classical results learning with noisy data, please refer (Frénay & Verleysen 2014).
Symmetric loss For RCN, where the noise parameters are symmetric, there exists works that show symmetric loss functions (Manwani & Sastry 2013 Ghosh al. 2013 2017] Van Rooyen al., 2013) are robust to the underlying noises, without specifying the noise rates. It was also shown that under certain conditions, the proposed loss functions are able to handle asymmetrie noises. Our focus departs from this line of works, and we will exclusively focus on asymmetric noise setting, and study the possibility of an approach that can ignore the knowledge of noise rates.
Follow-up works DDu Plessis et al. 2013 van Rooyen et al. 2015
Menon et al. 2015 Charoen- phakdee et al. 2019) have looked into leveraging symmetric conditions and 0-1 loss with asymmet- ric noises, and with more evaluation metrics, such as balanced error rate and AUROC. In particular, experimental evidences are reported in (Charoenphakdee et al. 2019) on the importance of sym-
metricity when learning with noisy labels.
More recent works More recent developments include an importance re- re-weighting algorithm (Liu & Tao 2016), a noisy deep neural network learning setting (Sukhbaatar & Fergus) 2014 Han al.
2018
Song et al. 2019). and learning from massive noisy data for image classification (Xiao et al. robust cross entropy loss for neural network (Zhang & Sabuncul 2018), loss correction
2015
PPa-
2

---

Under review as a conference paper at ICLR 2020
trini et al. 2017). , among many others. Loss or sample correction has also been studied in the context of learning with unlabeled data with weak supervisions (Lu et al. 2018). Most of above works either
lacks theoretical guarantee of the proposed method against asymmetric noise rates (Suukhbaatar & Fergus 2014 Zhang & Sabuncu 2018)) ), or requires estimating the noise rate (or transition matrix
between noisy and true labels, (Liu & Tao 2016 [iiao et al.) 2013) Patrini et al.) 2017; Lu al.) 2018). A good number of the recent works can be viewed as derivatives or extention of the unbiased surrogate loss function idea introduced in (Natarajan et al. 2013), therefore they would naturally re- quire the knowledge of the noise rates or transition matrix. We do provide thorough comparisons between peer loss and the unbiased surrogate loss methods.
Mostly relevant to us is a recent work (Xu et al. 2019) that proposes an information theoretical loss (an idea adapted from an earlier theoretical contribution (Kong & Schoeneheck 2018)) that is also robust to asymmetric noises rate. We aimed for simple-to-optimin
loss function that can easily adapt to existing ERM solutions. (Xu et al., 2019) involves estimating joint distribution matrix between classifiers and noisy labels, and then invokes computing certain information theoretical measure based on this matrix. Therefore, its sample complexity requirement and the sensitivity to noises in this estimation are not entirely clear to (not provided in the paper either). We do provide calibration guarantees and generalization bounds. We provide conditions when the loss functions convex. In general, we do think computationally peer loss functions are easy optimize with, in comparing to information theoretical measures. Experiments comparing with (Xu et al. 2019) are also given in Section 5
Peer Prediction Our work also builds on the literature for peer prediction (Prelec 2004 Miller
et al. 2005; Witkowski & Parkes] 2012] Radanovic & Faltings [2033] Witkowski et al. 2013 Das- gupta & Ghosh| [013 Shnayder et al. 2016, Liu & Chen 2017). (Miller et al., 2005) established that strictly proper scoring rule (Gneiting & Raftery] (007) 2007) could be adopted to elicit truthful reports from self-interested agents. Follow-up works that have been done to relax the assumptions imposed
(Witkowski & Parkes 2012, Radanovic & Faltings 2013 Witkowski et al. 2013 Radanovic al.
Liu & Chen (017). Most relevant to us IS (Daguupta & Ghosh| [2013] |Shnayder et al.] (216))
where a correlated agreement (CA) type of mechanism was proposed. CA evaluates a report's cor-
relations with another reference agent- its specific form inspired our peer loss.
2 PRELIMINARIES
Notations and preliminaries: For positive integer n, denote by [n] = (1.2....n). n). Suppose
(X,Y) E X x y are drawn from joint distribution D, with their marginal distributions denoted as Px, Py respectively. We assume X' C Rd, and y = [-1,+1), that is we consider a binary classification problem. Denote by p P(Y +1) E (0. There are N training samples (Ii, v1),, (IN, y drawn i.idd from D.
Instead of observing yns, the learner can only collect a noisy of training labels Sns, generated according to ynS and a certain error rate model, that is we observe a dataset {(In, Dn) } assume uniform error model for all the training samples we collect, in that errors in yr. follow the same error rate model: denoting the random variable for noisy labels as Y and we denote e41 P( = - +1), e-1 := PPY +1|Y -1) such that 0 S 6+1 < 1. e_ + is not unlike the condition imposed in the existing learning literature (Natarajan et al. 2013). and it simply implies that the noisy labels are positively correlating with the true labels (informative about the true labels). Label noises are conditional independent from the features, that is the error rate is uniform across Sn P(Y = y'lY-= = 3) = PPY = X.Y = y), Vy. E (-1.+1). Denote the distribution of the noisy data (X, Y) as D.
===1- We
f : X R is a vellevalued decision function, and its risk w.r.t. the 0-1 loss is defined as Exx,) (xx,))= (IS(X),)) The Bayes optimal classifier f is the one that minimizes the 0-1 risk: f = argmin, B(x,r)-d[1 ((f()) above 0-1 risk, the learner often uses surrogate loss function : IR x {-1,+1 +1} - R4, and find a
Denote this optimal risk as R Instead of minimizing the
F that minimizes the following error: ((x.))) Y)). Denote the following measures:
RD( = (xx.))>((1(((X)))) Re,p(f) ((xx)))
3

---

Under review as a conference paper at ICLR 2020
When there is confusion, we will also short-hand (xxxr)>Dl((f(( Y) as Ep ((((( Using D to denote a dataset collected from distribution D (correspondingly D == {(an Bn)) for D), the empirical risk measure for f' is defined as (e, = [o] E ExnDED {(f(z).))
2.1 LEARNING WITH NOISY LABELS
Typical methods for learning with noisy labels include developing bias removal surrogates loss func- tion methods to learn with noisy data (Natarajan et al., 2013). For instance, Natarajan et al. (2013) tackle this problem by defining an "un-biased" surrogate loss functions over E to help "remove" noise, when e-1 + E+1 < 1: E(t, == (I-e (--_w)) (- ((t,- Vt, y. i is identified such that when 1-6-1-E0 -++
prediction is evaluated against noisy label using this surrogate loss function, the prediction is as if evaluated against the ground-truth label using in expectation. Hence the loss of the prediction is "unbiased", that is V prediction t, (,t(, Y)] = E(t, y) [Lemma 1, (Naaarajan et al. 2013
One important note to make is most, if not all, existing solutions require the knowledge of error rates e-1, 6+1. Previous works either assumed the knowledge of it, or needed additional clean labels or redundant noisy labels to estimate them. This becomes the bottleneck of applying these great techniques practice. Our work is also motivated by the desire to remove this limitation.
2.2 PEER PREDICTION: INFORMATION ELICITATION WITHOUT VERIFICATION
Peer prediction is a technique developed to truthfully elicit information when there is no ground truth verification. Suppose we are interested in eliciting private observations about a binary event y E {-1,+1} generated according to random variable Y. There are K agents indexed by [K] Each of them holds a noisy observation of y. denoted as y(1 E [-1,+1], i [K]. We would like to elicit the y())s, but they are completely private and we won't 't observe to evaluate agents' reports. Denote by r(i) the reported data from each agent i. is completely possible that r(i) * y(i) agents are not compensated properly for their information. Results in peer prediction have proposed scoring or reward functions that evaluate an agent's 's report using the reports of other peer agents. For example, peer prediction mechanism may reward agent i for her report r(i) using S(r(i), r(i)) where r(j) is the report of a randomly selected reference agent j E [K]] The scoring function is designed so that truth-telling is strict Bayesian Nash Equilibrium (implying other agents truthfully report their v())), that is, Vi (S())((((()),,,... (()l(()))())))))))))))) v()))l(()) > [ug)[S(r())
(()),
Vr(1) + y(i).
Correlated Agreement (Shnayder et al.) 2016 Dasgupta & Ghosh 2013) (CA) is recently estab- CA is also the core and the focus of our subsequent sections developing peer prediction based Ioss functions. This mechanism builds on a A matrix that captures the stochastic correlation between the two sources of predictions y(i) and v(). Denote the following mapping function: g(1) 1, 92) = +1, A g2x2 is then defined as a squared matrix with its entries defined as follows:
lished peer prediction mechanism for a multi-task setting [
1(k PP (v) g(k), v() s(0) Pp((() 9(k)) ((()) -()), 1,1= 1,2 1,2
The intuition of above A matrix that each (i entry of A captures the marginal correla- tion between the two predictions. M E ggx² is defined as the sign matrix of A: M : Sgn(A), where Sgn(x) = 1, x > 0; Sgn(x) = 0, o.w. Define the following score matrix
M (-1,t1x
10. M (v.v') =: (v).
where g is the inverse function of CA requires each agent i to perform multiple tasks: denote agent i's observations for the N tasks as (1), .... (i). Ultimately the scoring function S(-) for each task k that is shared between i, is defined as follows: randomly draw two other tasks kp. ke.
(vv( m)) : (1)( un ()) - M (yug (1), Bag ()), kP + k2 +
Note a key difference between the first and second S terms is that the second term is defined for two independent peer tasks k.. Jy (as the reference answers). It was established in (Shnayder al.
We provide other examples of peer prediction functions in the Appendix.
4

---

Under review as a conference paper at ICLR 2020
2016 that CA is truthful and proper (Theorem 5.2, Shnayder et al. 2016))
2 in partieular, if y(())
is categorical w.r.t. y(i): P(v(j)= = y) P(y(7) = v)) v2, J
y # then S(.) is
strictly truthful (Theorem 4.4, Shnayder et al. (0166).
LEARNING WITH NOISY DATA: A PEER PREDICTION APPROACH
In this section, we show that peer prediction scoring functions, when specified properly, will adopt Bayes optimal classifier their maximizers (or minimizers for the corresponding loss form).
3.1 LEARNING WITH NOISY DATA AS AN ELICITATION PROBLEM
We first state our problem of learning with noisy labels as peer prediction problem. The connec- tion made by firstly rephrasing the two data sources, the classifiers and the noisy labels, from agents' perspective. For a task y E [1], say +1 for example, denote the noisy labels Y as r(X), X IP Pxjy=1- In general, r(X) can be interpreted the agent that observes V1, .... UN for a set of randomly drawn feature vectors E1,....,N On ~ r(X)). Suppose the agent's obser- vations are defined follows (similar to the definition of 6_), P(r(X)
-1/Y +1)
e+1, (P(((X)= = +1|Y -1) E-1. Denote another agent whose observations "mimic" the Bayes optimal classifier ʃ* Again denote this optimal classifier agent as r r'())= = f f'()::
Px(r*(X) = - +1) = 41 (x(r*(X) = +1|Y = -1 = e
Suppose we would like to elicit predictions from the optimal classifier agent r", while the reports from the noisy label agent r will serve as the reference reports. Both r and r" are randomly assigned a task x, and each of them observes signal r(x) and r (=(x) re- spectively, Denote the report from agent r* as F*. A scoring function S R x R - is called to induce strictly truthfulness the fol- lowing fact holds: (x(S(r(XX)
Reference report as the noiey label
S(y(),
e poor ② f(x))
Mix reward Min loss
Elicied repoct the classifeer prediction
Figure 1: Illustration of our idea. S is the peer pre- diction function; free is to "evaluate" classi- fier's prediction using a noisy reference.
Ex [x(s(r,, (x))]. VF* * (X). Taking the negative of S() (changing a reward score one aims to maximize to loss to minimize) we also have x S (X), rX))] < Fx [-S"" (F(((((r())) VF (X)( implying when taking -5(-) as the loss function, minimizing -S() w.r.t R will return us the Bayes optimal classifier f*. Our
idea can be summarized easily using Fig. 1
3.2 "PROPER" PEER PREDICTION FUNCTION INDUCED BAYES OPTIMAL CLASSIFIER
When there is no ambiguity, we will shorthand r(X), r (X) as r, r", with keeping in mind that r* encode the randomness in X. Suppose S(.) is able elicit the Bayes optimal classifier f" (agent r") using r, we have the following theorem formally:
Theorem 1. f argmin, E
This proof can be done via showing that any non-optimal Bayes classifier corresponds to a mis- reporting strategy, thus establishing its non-optimality. We emphasize that it is not super restrictive to have strictly truthful peer prediction scoring function S. We provide discussions in Appendix.
Theorem I provides a conceptual connection and can serve as an anchor point when connecting a peer prediction score function to the problem of learning with noisy labels. So far we have not discussed about a specific form of how we construct a loss function using ideas from peer prediction, and have not mentioned the requirement of knowing the noise rates. We will provide the detail about a particular peer loss next section, and explain independence of noise rates.
TTo be precise, it is an informed truthfulness. We refer interested readers to (Shnayder et al. 2016 for the detailed differences.
5

---

Under review as a conference paper at ICLR 2020
4 PEER LOSS FUNCTION
We now present peer loss, family of loss functions inspired by a particular peer prediction mecha- nism, the correlated agreement (CA), as presented in Section 2.2 We going show that peer loss able to induce the minimizer of a concept class F, under a broad set of non-restrictive conditions. In this Section, do not restrict to Bayes optimal classifiers, nor do we impose any restrictions the loss functions' elicitation power.
4.1 PREPARATION: EXPLAINING CA IN OUR CLASSIFICATION PROBLEM
To give gentle start, we repeat the setting of CA for our classification problem.
A and scoring matrix First recall that A E 2 is squared matrix with entries defined between r* (the f*) and (i.c., the noisy labels Y):
Akk () ((f okb), g(()) (())) (z)
-g(0))((()))))))))>))))) k,1= 1,2
Recall g(-) is simply a mapping function: g(1) = -1, g(2) +1. A characterizes the "marginal" correlations between the optimal classifier' prediction and the noisy label Y. Then the following scoring matrix M R=xn sign matrix of A, M :- Sgn( A) computed.
Example 1. Consider binary class label case: IP = -1) = 0.4,P(P = +1) = 0.6,
the noises in the labels are e-1 0.3, e+1 0.4 and e_1 0.2, 0.3. Then we (1 = -0.036.
have (1, 0.036,
(2, = -0.036, A(2 0.036. And:
0.036
-0.036
[
0
= M Sgn(A) = 0
0.036
0.036
1
Peer samples For each sample (x,, ú:), randomly draw another two samples (I.q. Bay): (zgg. Dey) such that i * i2 and &. * We will name (ag. ūey), ), (mag, Day) as i's peer samples. After pairing Z. with Dey (two independent tasks), the scoring function S(.) for each sample point ay is MM (f(z;). 5.) - MM (f (f(gg) ((ngg))0 Veg). Recall M.( is sign
defined follows: S(f 0:)) M
score matrix defined for A (Eqn. 1 Define loss function ε(-) as the negative of S(.):
(2) The first term above evaluates the classifier's 's prediction on using noisy label Ki. and the second "peer" term defined on two independent tasks i.. Ly "punishes" the classifier from overly agreeing with the noisy labels. We will see this effect more clearly. According to Theorem minimizing ((-) is going to find the Bayes optimal classifier, if y and I are categorical, which is easily satisfied: Lemma 1. When e- + e+1 < and e + c < 1, r and r* (Ý and J*) are categorical.
(Generic Peer Loss) {(f(z,), i.) = (1- M (((x))((()),))))))))(()))()()))) 7:)) - (1 -1-M (f(xz))) ((-
M
e 4+ means that the optimal classifier is at least informative (QLiu & Chen 2017 if otherwise, we can flip the classifier's output to obtain one.
4.2 PEER LOss
We need to know Sgn(A) in order to specify S and I, which requires certain information about f and Y. We show that for the cases that the literature is broadly interested in, Sgn(A) simply the identify matrix (same condition as stated in Lemma] D]:
Lemma 2. e- + e+1 < 1, 1, + e+a < 1, then Sgn( I2x2, i.e., the identity matrix.
This basically stating that for A(k, k), k = 1,2, f and y are positively correlating, so the marginal correlation positive; while for off-diagonal entries, they are negatively correlating.
Peer loss When Sgn(A) T2x2, MS (y.
if y y/. and 0 otherwise. ē(.) defined in Eqn. 1 Apcor (f (i) = ((f(x:), (.) - ((f(x.)) Bgg
2 reduces to the following form:
(3)

---

Under review as a conference paper at ICLR 2020
To see this, for instance 1 (f(x)) = +1, +1)==1---- = ( = 1112:2)=1111000 1(f(x)
-1, D: = +1). Replacing 1 1() with any generie loss {(.) we define:
(Peer Loss): (f ū.) ((f(z)), m.) - (f((zz),
(4)
We name above loss as peer loss. This strikingly simple form of (peer (2a)) Dt) implies that knowing e-1 + 1
< 1, 1_ + ++1 <
hold is all we need to specify fpeer-
Why do we not need the knowledge of noise rates explicitly? Both of the terms (z;();) U:) and (f( ), Day) encoded the knowledge of noise rates implicitly. The carefully constructed form as presented in Eqn. alllow peer loss to be invariant against noises (Lemma 3 property we will
explain later). For a preview, for example if we take expectation of 4 (f(x.) = +1, V/ = +1) we will have E perr (xxz) = +1, - +1]] = P(f(X) = +1, = -1)--(((X)) P(f(X) = +1) IP(Y = +1), the marginal correlation between f and Y. which is exactly capturing the entries of A defined between ff and Y! The second term above is a product of marginals because of the independence of peer samples ih, ig. Using the sign of A is all we need to recover this information measure in expectation. In other words, both the joint and marginal distribution terms encode the noise rate information in an implicit way. Later we will show this measure invariant under label noises, which gives us the property of peer loss being invariant to label noises and the ability of dropping the requirement of knowing noise rates. We will instantiate this argument formally with Lemma 3 and establish link between the above measure and the true risk of classifier on the clean distribution. The rest of presentation focuses on feee (Eqn. 4 ) but fpeer recovers peer via replacing with 1.
ERM with peer loss fi arg aig (() arg min N Em (f(n)) Un). Note again that the definition of pear does not require the knowledge of either e-1,e_1 or 1
4.3 PROPERTY OF PEER LOSS
We now present key property of peer loss, which shows that its risk over the noisy labels is simply an affine transformation of true risk on clean data. We denote by Ep [f (f(ee(((())))))) (X),Y) Y)] the expected peer loss of f when (X, Y), as well as its peer samples, are drawn iid. from distribution D.
Lemma 3. E-plpper (X)). Y)] (1 e-1 e+1)
(fX), Y)).
The above Lemma states that peer loss is invariant to label noises in expectation. We have also empirically observed this effect our experiment. Therefore minimizing it over noisy labels is equivalent to minimizing over the true distribution. The Theorems below establish the connection between (f Y),, the expected peer loss over clean data, with the true risk: Denote JEF R2 R1pe (()). B()) With Lemma 3 we can easily prove the following:
Ji = arg min
Theorem 2. [Optimality guarantee with equal prior] When p 0.5, Fi,m. arg aurgmin RD(f).
The above theorem states that for a class-balange balanced dataset with 0.5, peer loss induces the same minimizer as the one that minimizes the 0-1 loss on the clean data. Removing the constraint of
F, i.e., Fi = arg min, R₂ (ff) + f pear f practice we can balance the dataset s.t. + 0.5. When # 0.5, denote A, = P(Y
- -1), we have the following theorem: Theorem 3. [Approximate optimality guarantee with unequal prior] When p + 0.5, suppose the following conditions hold: (1)e-1, < 0.5; (2) (1-e)--iteeeH > e; (3) (1-e) (1-e)
>
where = 2/2 ررك Then [RD(Ji,..) minser Ro(f) = 20(E 2 Ve < | if is bounded with i, denoting its max and min.
Condition (1) is a well-adopted assumption in the literature of learning with noisy labels. When 641, C-1 > we have conditions (2) and (3) hold: (1 - e) --1 E41 (1 e) e e, (1 - e) ee > (1 - e)- e e e = e. When | is small, i.c., p is closer to 0.5, this condition becomes weaker, we will afford to have small but also small e.
Multi-class extension Our results in this section are largely generalizable to multi-class setting. Suppose we have K classes of labels, denoting as 21,2,... K). We denote by Q transition matrix
7

---

Under review as a conference paper at ICLR 2020
that characterizes the relationships between noisy label Y and the true label Y. The (i.j) entry of Q is defined as Qiy IP(Y" =jiY= = i). We write Qi = Oij- For many classes of noise matrices, the (.) matrix is simply a diagonal matrix. Consider the following case: suppose the noisy labels have uniform probability of flipping
wrong class, that is, we pose the following conditions: di qik for all j o k o i. This condition allows us to define K new quantities e; = qij for all i + j. and Gi = 1 Sjtt Cj- We show that M.) is diagonal matrix when Σソ=1 Gj < 1, a similar condition c-1+ 641 < Adapting from our proof for Lemma 3] also have (derivation provided in Appendix) (f()), ()] -1- Eÿ) -ED [1 peer (f (X)), The above again will help us reach the conclusion that minimizing peer loss leads to the same minimizer on the clean data. We
provide experiment results for peer loss with multi-class labels in 3
4.4 Q- a-WEIGHTED PEER LOSS
We take a further look at the case with p o 0.5. Denote by R+1(f) = P(J(X) -1ly +1), R-1(f) P(f(X) = +1yy= -1).
1). It is easy to prove:
Lemma 4. Minimizing peer (f(X), Y)] is equivalent minimizing ( R-((f).
a weighted sum of R41 (f) and (f). The above observation and the failure to reproduce the strong theoretical guarantee when p o 0.5 motivated us to study a a-weighted version of peer loss, to make it robust to the case p * 0.5. We propose the following a-weighted peer loss via adding a weight a 2 0 to the second term, the peer term:
However, minimizing the true risk (f) is equivalent to minimizing p-
(a-Peer Loss): (J = (U(())) - aa.aa... ((((()))) (,g))
(5)
Denote La-peer as a-ppor when replacing l with 1, Ji arg min seF R2 a d (ʃ) as the optimal classifier under 1 peerr and A3 P( +1) - -P(Y - Then we have:
Theorem 4. Leta
(1 c e+1) 2.. Then fia... E arg min (f(
- Denote a := 1 1 (1-11---111 e-1---1-ap c+1)
A Several remarks follow: (1) When = -0.5, we have a" = 1, we recover the earlier definition of fpeer (2) When e e+1, a² = O, we recover & for the clean learning setting. (3) When the signs of P(Y = 1) P(Y 1) and P(
25
P -1) are the same, a= < 1. Otherwise, a* > 1. In other words, when the noise changes the relative quantitative relationship of P(Y 1) and PPYY -1), a* > 1 and vice versa. (4) Knowing a* requires certain knowledge of E+1, c-1 when # 0.5. Though we do not claim this knowledge, this result implies tuning a (using validation data) may improve the performance.
Theorem 2 and imply that performing ERM with peari peari fia..m
arg min, R 1, b(f will
lead to classifer converging to f":
Theorem 5. With probability at least 1 8, RD(fi. .nd ) R S 2(1-a")
log 2/8 2/6
T T-e 1-e--0 1-
2N
4.5 CALIBRATION AND GENERALIZATION
So far our results focused on minimizing 0-1 losses, which hard in practice. We provide evi- dences of épeer's s. and la fa-peer' 's in general, calibration and convexity with a generic and differentiable calibrated loss. We consider a that classification calibrated, convex and L-Liptchitz.
Classification calibration describes the property that the convergence to optimality using loss function would also guarantee the convergence to optimality with 0-1 loss:
Definition 1. l is classification calibrated if there 3 convex, invertible, nondecreasing transfor- mation Ve with (0) = 0 s.t. (f(RD(f) R²) < Re, min, Rt,D(f).
Denote fi E arg min, (f). Below we provide sufficient conditions for la-perr to be calibrated. Theorem 6. Eeepeer is classification calibrated when either of the following two conditions holds: (1) a = 1 (iee E Eapeer = (pear). p = 0.5, and ft satisfies the following: (I((J(X).-Y) YJ. (2) (ax1,,, < 0.5, and t (1, (²)
2
8

---

Under review as a conference paper at ICLR 2020
(1) states that fē not only achieves the smallest risk over (X, Y) but also performs the worst on the "opposite" distribution with flipped labels -Y). (2) E" y) = (t. -y) is satisfied by some common loss function, such square losses and logistic losses, noted in (Natarajan et al. 2013),
Under the calibration condition, and denote the corresponding calibration function for La peer as V Ln Denote by fer arg min Ri (f) == x En= la-poal (In)), In)) We have the following generalization bound:
Theorem 7. The following generalization bound holds for eat peer with probability least 1 - 8:
RD(ft,- R² S 1 1-6-1 E+1
1
Via-
min R2 feT
ol)) min Ri, 5.)
log 4/8
2N (1 (1+ -0))
2(1 + a')L R(F) + 2/
where R(F) is Rademacher complexity F.
Convexity In experiments, we use neural networks which are more robust to non-convex loss functions. We provide sufficient conditions for R, R. (f) to be convex in Appendix (Lemma 8).
5 EXPERIMENTS
We implemented two-layer ReLU Multi-Layer Perceptron (MLP) for classification tasks on 10 UCI Benchmarks and applied our peer loss to update their parameters. We show the robustness of peer loss with increasing rates of label noises on 10 real-world datasets. We compare the perfor- mance of our peer loss based method with surrogate loss method (Natarajan et al. 2013) (unbiased loss correction with known error rates), symmetric loss method (Ghosh et al. 2015 ) DMI (Xu al.) 2019). C-SVM (Liu et al. 2003) and PAM (Khardon& & Wachman 2007). which
ofte--o--theant
methods for dealing with random binary-classificatio noises, as well as a neural network solution with binary cross entropy loss (NN). We use validdsiioitionnn the algorithms. For surrogate loss, we use the true eтror rates c-1 and 6+1 instead of learning them on the validation set. Thus, surrogate loss could be considered favored and advantaged baseline method. Accuracy of a classification algorithm is defined as the fraction of examples in the test set classified correctly with respect the clean and true label. For given noise rates e.1 and e labels of the training data are flipped accordingly.
tune the parameters specific to
Accursyy Duiing Teeting
Accursy Duting Teeting
0.5
0.64
875
6.18
0.5
Sarogite los
040
46
180
323
140
26
30
30
(a) Splice (c-1 = 0.4, 6+1 = 0.4)
(b) Twonorm (c-1 = 0.4, 6+1 = 0.4)
Figure Accuracy on test set during training
subset of the experiment results are shown in Table I full table with all details can be found in Appendix. Equalized Prior means that we pre-sample the dataset to guarantee p = 0.5. For this case we used fpeer without a (or rather a = 1 as perrr.. For p * 0.5, we use validation dataset (using noisy labels) to tune a. Our method is competitive across all datasets and is even able to outperform the surrogate loss method with access to the true error rates in a number of datasets, as well as symmetric loss functions (which does not require the knowledge of noise rates when error rates are symmetric) and the recently proposed information theoretical loss (Xu ct al. 2019). Fig. shows that our method prevent over-fitting when facing noisy labels.

---

Under review as a conference paper at ICLR 2020
Task
With Prior Equalization p = 0.5
Without Prior Equalization p #
Nd, N-)
C-1, 641
Peer
Surr
Symm
DMI
NN
Peer
Surr
Symm
DMI
NN
0.1, 0.3
0.977
0.968
0.969
0.974
0.964
0.977
0.968
0.969
0.974
0.964
Twonorm
0.2, 0.4
0.976
0.919
0.959
0.966
0.911
0.976
0.919
0.959
0.966
0.911
(20,3700,3700)
0.4, 0.4
0.973
0.934
0.958
0.936
0.883
0.973
0.934
0.958
0,936
0.883
0.3
0.919
0.878
0.851
0.875
0.811
0.925
0.885
0.868
0.889
0.809
Splice
0.2, 0.4
0.901
0.832
0.757
0,801
0.714
0.912
0.84
0.782
0.81
0.725
(60.1527.1648)
0.4, 0.4
0.819
0.754
0.657
0.66
0.626
0.822
0.755
0.674
0.647
0.601
0.1, 0.3
0.833
0.78
0.777
0.797
0.756
0.856
0.802
0.803
0.83
0.75
Heart
0.2, 0.4
0.812
0.768
0.717
0.788
0.679
0.856
0.758
0.725
0.797
0.693
(13,165,138)
0.4,
0.75
0.729
0.654
0.69
0.595
0.785
0.728
0.686
0.711
0.554
0.3
0.745
0.707
0.674
0.72
0.667
0.778
0.75
0.738
0.729
0.727
Diabetes
0.2,
0.755
0.681
0.634
0.682
0.596
0.739
0.705
0.695
0.707
0.672
(8,268,500)
0,4, 0.4
0.719
0.645
0.619
0.637
0,551
0.651
0.685
0.68
0.633
0,583
0.1,0.3
0.639
0.563
0.507
0.529
0.519
0.727
0.645
0.709
0.666
0.648
Breast
0.2, 0.4
0.63
0.534
0,482
0.496
0.538
0.73
0.674
0.666
0.58
0.672
(9,852001)
0.4, 0.4
0.596
0.519
0,504
0.526
0.471
0.677
0.628
0.545
0.537
0.529
0.1,
0.928
0.922
0.924
0.934
0.873
0.956
0.949
0.943
0.954
0.92
Breast
0.2, 0.4
0.93
0.885
0.844
0.89
0.844
0.933
0.898
0.898
0.918
0.831
(30,212.357)
0.4, 0.4
0.928
0.867
0.819
0.746
0.824
0.908
0.839
0.817
0.795
0.673
0.1,
0.701
0.624
0.614
0.637
0.581
0.68
0.693
0.603
0.605
0.6
German
0.2,
0.664
0.59
0.6
0.618
0.572
0.676
0.681
0.537
0.573
0.535
(23,300,700)
0.4, 0.4
0.606
0.55
0.573
0.573
0.556
0.654
0.632
0.549
0.611
0.553
0.1, 0.3
0.89
0.895
0.892
0.856
0.868
0.893
0.898
0.883
0.785
0.863
Waveform
0.2, 0.4
0.881
0.89
0.828
0.835
0.81
0.884
0.884
0.745
0.761
0.837
(21,1647,3353)
0.4, 0.4
0.87
0,866
0.867
0.773
0.835
0.853
0.852
0.852
0.672
0.828
0.1, 0.3
0.906
0.9
0.89
0.87
0.909
0.943
0.909
0.897
0.811
0.93
Thyroid
0.2, 0.4
0.863
0.862
0.85
0.784
0.822
0.905
0.898
0.865
0.759
0.881
15665,150)
0.4, 0.4
0.762
0.738
0.859
0.788
0.764
0.769
0.818
0.876
0.738
0.738
0.1, 0.3
0.856
0.875
0.843
0.896
0.866
0.796
0.835
0.903
0.896
0.878
Image (38,1320,990)
0.2, 0.4
0.836
0.862
0.719
0.845
0.832
0.672
0.755
0.722
0.86
0.599
0.4, 0.4
0.741
0.72
0.788
0.763
0.732
0.806
0.803
0.823
0.762
0.8
Table 1: Experiment results 10 UCT Benchmarks (N+, N_ the numbers of positive and negative
ples). Surr: surrogate loss method (Natarajan et al. 2013); DMI: (Xu et al.) 2019); Symm: symmetric loss method (Ghosh et al. 2015). Entries within 2% from the best each row are highlighted in bold. All results are averaged across 8 random seeds. Neural- seetnlnetworr--baa methods (Peer, Surrogate, NN, Symmetric, DMI) the same
Full table with complete of comparisons is in Appendix.
Model
Error Rate 0.2
Error Rate 0.4
Cross Entropy
86.67
82.09
DMI [Xu et al. 2019
85.
81.67
Peer Loss
87.72
83.81
Table 2: Accuracy on CIFAR-10.
Preliminary results multi-class classification We now provide some preliminary results CIFAR-10 in Table We followed the setup (Xu et al. 2019) and used ResNet (He et al.] 2016) the underlying optimization solution. However, different from (Xu et al.) 2019) whose noise only exists between specific class pairs, our noise is universal. For each class, we flip the label to any other label with probability of e/9, where is the eиror rate and 9 is the number of other classes.
We do show peer loss is competitive against Cross Entropy and DMI (Xu et al. 2019 More results and complete details are available in the Appendix.
Conclusion This paper introduces peer loss, family of loss functions that enables training classifier over noisy labels, but without using explicit knowledge of the noise rates of labels. We provide both theoretical justifications and extensive experimental evidences.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds.
Journal of the American Statistical Association, 101(473):138-155 2006.
Shai Ben-David, Dávid Pál, and Shai Shalev-Shwartz. Agnostic online learning. COLT22009
Tom Bylander. Learning linear threshold functions in the presence of classification noise. In Pro- ceedings the seventh annual conference on Compulational learning theory, pp. 340-347. ACM, 1994.
Nicolo Bssa-Bianchi, Eli Dichterman, Paul Fischer, Eli Shamir, and Hans UIrich Simon. Sample- efficient strategies for learning the presence of noise. Journal of the ACM (JACM), 46(5): 684-719, 1999.
Nicolo Cesa-Bianche Shai Shalev-Shwartz, and Ohad Shamir. Online learning of noisy data. IEEE Transactions on Information Theory, 57(12):790-77931, 2011.
Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learn- ing from corrupted labels. In International Conference on Machine Learning, pp. 961-970, 2019.
Anirban Dasgupta and Arpita Ghosh. Crowdsourced judgement elicitation with endogenous profi- ciency. In Proceedings of the 22nd international conference World Wide Web, pp. 319-330. International World Wide Web Conferences Steering Committee, 2013.
Marthinus Christoffel Du Plessis, Gang Niu, and Masashi Sugiyama. Clustering unclustered data: Unsupervised binary labeling of two datasets having different class balances. In 2013 Conference on Technologies and Applications of Artificial Intelligence, pp. 1 1-6. IEEE, 2013.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey, IEEE transactions on neural networks and learning systems, 25(5):845-86) 2014.
Aritra Ghosh, Naresh Manwani, and PS Sastry. Making risk minimization tolerant to label noise. Neurocomputiig, 160:93-107, 2015.
2015.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. In Thirty-First AAAI Conference Artificial Intelligence, 2017.
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):399-378 2007.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-85377 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Roni Khardon and Gabriel Wachman. Noise tolerant variants of the perceptron algorithm. J. Mach. Learn. Res., 8:227-248, May 2007. ISSN 1532-4435. URL http: /dl...l. acm. org/ citation wfm?id= 22486699991299999... 1248667
Yuqing Kong and Grant Schoenebeck. Water from two rocks: Maximizing the mutual informa- tion. In Proceedings of the ACM Conference Economics and Computation, pp. 177-194. ACM, 2018.
Yann LeCun, Y. Bengio, and Geoffrey Hinton. Deep learning. Nature, 521:436 44, 05 2015. doi: 10. 10.8/areee 14539.
Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S. Yu. Building text classifiers using positive and unlabeled examples. In Proceedings of the Third IEEE International Conference Data Mining, ICDM '03, pp. 179-, Washington, DC, USA, 2003. IEEE Computer Society, ISBN 0-7695-1978-4. URL http: //11. acm. www..org//iiaaiioi.. c fm? ? d=951 949. 952139

---

Under review as a conference paper at ICLR 2020
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3)4477 461, 2016.
Yang Liu and Yiling Chen. Machine Learning aided Peer Prediction. ACM EC, June 2017.
Nan Lu, Gang Niu, Aditya K Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. arXiv preprint arXiv: 10585, 2018.
Naresh Manwani and PS Sastry. Noise tolerance under risk minimization. IEEE transactions on cybernetics, 43(3): (333):1146-111 1e11111661111,, 2013.
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from cor- rupted binary labels via class-probability estimation. In International Conference Machine Learning, pp. 125-134, 2015.
Nolan Miller, Paul Resnick, and Richard Zeckhauser. Eliciting informative feedback: The peer- prediction method. Management Science, 51(9):1359-13353 2005.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
D. Prelec. A bayesian truth serum for subjective data. Science, 306(5695) 2004.
G. Radanovic and B. Faltings. A robust bayesian truth serum for non-binary signals, In Proceedings of the 27th AAAI Conference on Artificial Intelligence, AAAI '13, 2013.
Goran Radanovic, Boi Faltings, and Radu Jurea. Incentives for effort in crowdsourcing using the peer truth serum. ACM Transactions on Intelligent Systems and Technology (TIST), 7(4):48, 2016.
Clayton Scott. A rate of convergence for mixture proportion estimation, with application to learning from noisy labels. In AISTATS, 2015.
Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. Classification with asymmetric label noise: Consistency and maximal denoising. In COLT, pp. 489-511, 2013.
V. Shnayder, A. Agarwal, R. Frongillo, and D. C. Parkes. Informed Truthfulness in Multi-Taas Peer Prediction. ACM EC, March 2016.
Victor Shnayder, Arpit Agarwal, Rafael Frongillo, and David C Parkes. Informed truthfulness in multi-task peer prediction. In Proceedings of the 2016 ACM Conference on Economics and Com- putation, pp. 179-196. ACM, 2016.
Hwanjun Song, Minseok Kim, and Jae-Gil Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International Conference on Machine Learning, pp. 5907-5915, 2019.
Guillaume Stempfel and Liva Ralaivola. Learning syms from sloppily labeled data. In International Conference on Artificial Neural Networks, pp. 884-893. Springer, 2009.
Sainbayar Sukhbaatar and Rob Fergus. Learning from noisy labels with deep neural networks. arXiv preprint arXiv:1406 2080. 2(3):4, 2014.
Brendan Van Rooyen, Aditya Menon, and Robert C Williamson. Learning with symmetric label noise: The importance of being unhinged. In Advances in Neural Information Processing Systems, pp. 10-18, 2015.
Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. An average classification algorithm. arXiv preprint arXiv: 1506.0/220 2015.
Witkowski and D. Parkes. A robust bayesian truth serum for small populations. In Proceedings of the 26th AAAI Conference Artificial Intelligence, AAAI '12, 2012.
12

---

Under review as a conference paper at ICLR 2020
Jens Witkowski, Yoram Bachrach, Peter Key, and David C. Parkes. Dwelling on the Negative: Incentivizing Effort Peer Prediction. In Proceedings of the 1st AAAI Conference on Human Computation and Crowdsourcing (HCOMP'13), 2013.
Tong Xiao, Tian Xia, Yi Yang. Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification.. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2691-2699 2015.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L.dmi: An information- noise-robust loss function. NeurIPS, arXiv:1909.0330 2019.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels, 2018.
13

---

Under review as a conference paper at ICLR 2020
ILLUSTRATION OF OUR IMPLEMENTATION OF PEER LOSS
Noisy Training
Neurl Network
Labels
Data
61
(peer = 61 B2
Function
62
Random
Peer Sampling
Forward Propagation
Backward Propagation
Figure 3: Illustration peer loss method.
OTHER PEER PREDICTION FUNCTIONS
Other notable examples include quadratic and logarithmic scoring function, defined as follows:
Example 2. Quadratic scoring function:
= -())))))))
Σ
P(D() slv(1) r()))²
.[(-1,+1]
Example 3. Logarithmic scoring function:
S(r(0),(i) ==
= =
We know the following is true:
Lemma 5 Miilerr et al. (2005)). . S defined in Example & 2 induce strict truthfulness when y(1)
and y(j) are stochastically relevant.
with defining stochastic relevance as follows:
Definition 2. y(i) and v(j) are stochastically relevant if 3 E (-1,+1} s.l.
PP((i) slv(i) = +1) sly)) 11).
Similarly conclude that when and y** stochastic relevant, the correlated agreement scoring rule, quadratic scoring rule and logarithmic scoring rule are strictly truthful. This stochastic rele- vance condition essentially states that the optimal classifier statistically different from the noisy data source r some signals. Stochastic relevance is further satisfied in the binary classification setting when et + ex # under the assumption that e-1 + C+1 < as similarly imposed in learning with noisy labels literature (Scott et al. 2013 Natarajan et al. 1003,,Scor 2015).
Lemma 6. and r" are stochastically relevant if and only if ef"", + + * 1.
Proof. Since r" can be written function of X and Y, due to conditional independence between r and X (conditional on Y), by chain rule
P(r* = -1, +1) = P(Y = +1) - -4)) + P(Y = -)e-- (1 e
Since
P(r = +1) P
P(Y = -1)
P(r" = +1) =11==P(P + P(Y =1)
14

---

Under review as a conference paper at ICLR 2020
We have
P(r* -1) P(r = -10 -
PPY +1) = -1) -41
(6)
For the binary signal case, the condition for stochastic relevance writes as follows:
P(r = +1|-" = +1) + P(r = +1/-" = -1
P(r = +1 +1,--= * Prr = +,, =--1 -1,
P(r* = +11
P(r" = =11
eP(r +1, =+))p(r* -1) 1)= # P(r ====,,,== = +1)
Pr +1,r" =11 + Pr +1) -P(r"=-1)
er +1, r = -1) # P(r +1) P(r = -1)
e-_1 + 441 # 1,
where the last step is a consequence of Eqn.
PROOF FOR THEOREM
Proof. It is equivalent to prove f = argmax, (x)))))(((((X)),.. First S(-) is able to elicit the Bayes optimal classifier f (r*) using implies that:
p [S(r", rr] > ( VF* +
E - .r] > (= VF* * r*
First note that the expected score of a classifier over the data distribution further writes as follows:
[)[s(f(X),r)] ::ep:))p-ppppppaapaeeee EE [S(((x))) (1 p) ([
Denote by f' sub-optimal classifier that disagrees with f on set X = {x|Y = +1: f())) +
(z)). By sub-optimality of f' we know that e := Px (X E xxt) > 0, as a zero measure X does not affect its optimality. Construct the following reporting strategy that
dis
r", w.p. 1 e r", w.p. e
Not hard to check that
ED [S('(X),,)] = [1
Yet have the following fact that
E B [S("",)) (S(""
= E) [
+ (5-
<Bp [S("(())
where the inequality due to strict truthfulness of S and the fact that e 0. We similarly conclude that
E (py==
(8)
Combine Eqn.
and
conclude the proof.
15

---

Under review as a conference paper at ICLR 2020
PROOF FOR LEMMA
1
Proof. Being categorical means
P(r = yyrr = y) < P(r = -y), E f-1,+1}
which further implies
P(r -.. 1r 3) P(r -y)P(r* =y), V +-1,+1}
and
Prr y,r = v) > P(r = w)P(r* = -), E [-1,+1].
Consider the following fact
Pr +1, = +1)
= PY = +r)P((r = +
+ P(Y = - +1, ++|Y = -1)
-P = + +1,
=
IP(r" +1|Y +1)
+P(Y = -1) = - +1
P(r" +1|Y --1
Since f" can be written as a function of X and Y. due conditional independence between r and
X (conditional on Y) we have
P(r + 1|r" = +1, +1) = +1)
Prr = +1 = = -1
Therefore
P(r P(r=+1,7 +1) P(Y -))()=c=)(1- = -1) c-1 e-11.e_
We also have
P(r = +1) = P = +1)(-=4 P(Y = -
P(r" +1) = P = +1)(1- + P(Y .
Then we have
P(r = - =+1
= P(Y +1)PP(Y - 11)1- -1 - E+ - -
>
when 1 > E. + e"
PROOF FOR LEMMA 2
Proof. Again recall that
P(r" +1,-= (or=+11 +1) = P(Y =
= -1)e-11
P(r = +1) PPY +)
P(r* = +1) = P(Y = +1) PPY -1).
=
Then we have
P(r* +1,r +1) - PP(r* = (r
=P(Y = +1)P(YY = 11 6 - e-1) - -+ - e
>0
when
6+1 e-1 > 0. 1 ei eー= > 0. Interestingly this coincides with the condition
imposed NNtaraaaan et al. 2013). Similarly we can prove that
P(r" = +1,r = 1) Pp("" = +I)P(r = =11
IP(Y +1) 1)((1 E+1 - -e-) - e+ -
<0 The other entries for P(r -1, -1 P(r 11)P(r -1) and P(r -1, +1) - P(r* = (r = +1) are symmetric. Therefore the sign matrix above score matrix is [T
exactly the diagonal matrix.
16

---

Under review as a conference paper at ICLR 2020
PROOF FOR LEMMA
3
Proof. We denote by Xip, Yı² the random variable corresponding to the peer samples z,g, ....
First we have
(f(),, = -BC
Consider the two terms on the RHS separately.
B({(f(x),)) = -
=
+ +
[P(Y = + -11-
- -e_1) (1x)=-=(--))))(((())---))))----- -111,)
+ + 641) -1))) -
=Ex,y=11 61-1-1-----1001 e41) [
+ [ - [1 (1
-(1 6-1 -441) Ex,y (|00://0|||||a//////000000aa4/11
And consider the second term:
-Ex((((XX), IP(Ý -1) -x))+ +1)
11 +1)]
+ Exx (1
[(f(x))++]
=11- 1-1--611 c+1) (x(((f(X)), +Ex - ((f(x),+1)
Thus,
- - E(E((()) 8J] = e_1 --1-
-41) ). peer (X(),, Y)
Multi-class extension
Notice the following facts:
E[I(S(X), E Yg)l = P(1(1()) v)) - - -
and
K
P(Y = PP = (1 Σ
Ek) + - P(Y j))e, = (1 Σ ex)Pp(Y = +
P(1(f(X)= R))
K
K
Σ
P(Y = k) Σ P(f(X) = j|1 = =k)gsk
j=1
K
K
P(f(X) -|lY K)P(Y k)ggK
K
K
P(f(X)
= = =]) - -1)
10(1-Eesd
(w) Eee)
ΣΣ P(f(X) = j = k)P(Y = k)e,
j=1
K
K
P(f(X)= = -J = =0 - Σ ek) + Σ
ej (P(f(X) = =) -
k
j=1
K
K
=(1
Σ
= + Σ
exp(((x= =
1=1
17

---

Under review as a conference paper at ICLR 2020
Now consider the following
((((x)= Pa))
K
P(((X)= - j))p( =)
Σ P(f())= = Σ
PP = k)gbk
k=1
Σ P(((X) -( (1-Ee))P(Y- (1-F +e)
==1
Therefore
- =Yg)
-(1 Σ ck) Σ ((((()) j/v j))(( =) -P(((X) = 5)P((= =
For clean labels we have
E)((())))) Σ (((()= -
For the second term we have
K
E[L(S(X;)), Ki)]
Σ (((())=))P() 5)
Therefore
B[I(f(X), -
Σ P(Y = k) (P(((X = //l = -)P = j) - -( = =))
We finish the proof.
PROOF FOR THEOREM 2
Proof. From Lemma
we know
El Elepeer peer (f(X),
= (1 6-1 -41) (XX),)
= (1 -e-1 - -441)-
Xe:])
=(1 - e-1 -41) EE
-1)] ((f(()) 1) 0.5 -xxxxx((( - - 0.5 -00000))))))000---_[(00+++++)
When is the 0-1 loss we have -1) +1) = Vz, and therefore
E(lpeer Sf (f(X), V)] = (1 - e__ --1- [[e(f(x),,
With above we proved fipm arg RD (f)
18

---

Under review as a conference paper at ICLR 2020
PROOF FOR THEOREM 3
Proof. Our proof is inspired by our argument for = 0.5. We ask the following question: if it is possible to show that Y corresponds an error-flipedd distribution of another distribution Y whose marginals py is close or equal 0.5. Observe the following: randomly flipping Y with probability e uniformly, we will have a new distribution of labels Y that satisfies:
PY = P(Y = +1) = P(Y = +1) (1 e) + P(Y = - p(1 -2)) +
2e)
Denote by c the tolerance of py: jpy - 0.5|. When e sets to be: 1- = |
we have
Jpv 0.5| The next question ask: is it possible to find parameters ê-1;
P(Ý = -1) ê-1 P(Ý - = +1) = é+1
Note that
P( 1|1 +1)
=P(Ý = -1 = +1) +1
P =-1l = PIY = +1)
=(1 - e) . te. -e-(tee..) (1-
Similarly P(ý- +1|Y 1) = (1 c) م e (1 è+1). Jointly we need the following
equations to hold:
(1 ê+1 t (1 ê-1) = c+1 - c) ر1 + e (1 (41) C-1
Solving above equations we have
(1 e) c-1 e.
e+1
e
é_
1 2e
1 1 2e
For feasible solution ê-1, ê+1, the conditions need satisfy that (1) é-1, 2 0 and (2) ê_ + ê④1 < 1. First of all, from (2) we have
e (1 - (é-1 + &+1)) = =e_1 ê-1
Then necessary condition for é- + + ê+1 <
+ 2(1 -2e) c-1 - ê_ + e-1
This condition holds as long as c-1,641 < 0.5. From 1_ ê41 2 0 we have
- (--e)-e1+ed e+1 > e, (1 e) e+1 + 6 e_
99)
This above jointly proves that Re,,, pea D (ʃ) equivalent to peer loss defined over the noisy distri- bution of i with error parameters é_1, 641-
Denote by JJ E arg min (.( From the optimality of fipm we have
Rolfi ) - (x), (xI, (1 (1- Exle(ft -x))) (X),++1)
RD(f5) -xx -11 -X)
(10)
Note Vf:
- (x(
(11)
0.5 Ex [x -x)0)))) -1))
-lpy 0.5| -
<c(@ !)
(12)
19

---

Under review as a conference paper at ICLR 2020
Notice that
(D(() - by (000 < [BD(fz) by- - (1-00(((0))))))--------)((((((()
< <BD(ff) - 0.5 Ex (500:(0))+H))00000-EE00000000000
(13)
Combining Eqn.
(10 12 13 we have
RD Rf RD Spr
0.5 -00 +((])((( -
<2e(E-
PROOF FOR LEMMA 4
Proof.
E [ (f
=(1 c-1 - 641)
= -1)P(Y==+ -(()) -1)
((((x)++))P(
= --1)
(11 -c-1 6-11 -6+1) -
(pR+, +(1-
- -p- -PP(((X) -1)
-1))
-(1 c-1 641) ( (p
- --- (pR+1 (1-
-(1-p) (p(1 R41) + R)))
-2(1 111---11 e-1 e41) pp1 - p) (R_1 + R+1 1)
PROOF FOR THEOREM 4
Proof.
Eel (1
=E]]pe((((X),))) (1)))))))+ xx) -1
=E ([( (1-a) (P([(X)=-1)- P(C --1)+P(((X)) = +1) P( = +1)) -1
[F()) [8]+ (1 a)) (1+)) + +(1-+1111 (1-p) -))_))) -1)
+ (pR41 + (1- (1-p)(1- - R)) p(-= = +1) -1
([()) (1-
- a) (P(Ý- = +1) - -P( = =11) (pR+1 (1-1)R +
-2(1 c-1 - (+1) pp(1 p(1-p) (p11-)). (R-1+ R41-1)
+ (1 - a) (P(Ý (P(Ý= +1) - PIý- -1)) (pR+1 - -(- (1- +C
=R41 : 22(1 - -e_1 -
-e-1- e+1) p(1 (p)-)) + (1 - a)p (P(Ý = +1) - P( = -1))
+ R1 (2(1 (21-e1--41) p(1 - p) - (1 a)(1 p)- (P(Ý = +1) - pp -1)) +C",
where C, C' are constants:
C= = (1 - (1- (1 p) PP( +1) P( = -1)) -
+
-'=C 2((1221--) - e_1 2 6+1 p(1 - p)
20

---

Under review as a conference paper at ICLR 2020
Let
P 2(1 -,0---.....0...0000000--0---0--0
1 1-p p
2(1 - c-1 --
that
4 A
a= a= (1 (1- --1 - E+1)
obtain that
a (10
(14)
concluding our proof. The last equation Eqn.
also implies the following proposition:
Proposition 8. For any J'J' we have
(f(_)) - peer
= ( -
PROOF FOR THEOREM 5
Proof. Vf, using Hoeffding's inequality with probability least 1
| (())
D(f)
R 1 (f)
log 2/6
(Ta-peer peer la ap-re))
2N
log 2/5
< (1 + a)
2N
Note also have the following:
R Blamm Rs wor D (ffiaaa (fiam))
R B.f Blfiapal
R axpor MapprrD b(f (alamm)) + (Rampnn R
R (fi
+ (Rl oo Dl5 DlFimppn R2 ren ((inppe))
<0 + max IR (()) R B(one
Now show
RD (f par H²
=RD -m0
RD(Ji per)
(Theorem
-(RR
sour Bl p(fi,-au R ren Ufi ((a--ym)
(Proposition g
C-1 E41
2
max R ((f) R (f))
1 6-1 641
2(1 + +a))
log 2/5
1 c-1 641
2N
We conclude the proof.
PROOF FOR THEOREM 6
Proof. We start with condition (1). From Lemma 3
Eeepeer (f(X), 8)) =(1 e-1 -e+))- (E((((())))) 0.5 -.5.00.0)) 0.5 (((0(()++11)
21

---

Under review as a conference paper at ICLR 2020
The above further derives
pf (f(),, Y)]
(11 - c-1 - -e4)
(() 0.5 () 0.5 -
(E(((f(X))))) -(((())))))))) -,0-)-))
1 c-1 -41
2
Denote by c:=
1-8-1-841 we have
Bl((f(X), = c. (1
Then
E(U -
=c poer (X),, 8)] Blc poer (J7 Y)))
Sc (B[[(Bep (X)) B14 poor (fi (X), Y)])
Further by conditions we know
-() -Y))
E(((f(X), - (X),
Therefore we have proved
Eleperr (f(), Elpgeer (fi (X), (X) ( -(())())))))
Since {(-) is calibrated, and according to Proposition 8 and Theorem
E (e(],,ee(((Xxxx - E. a pear (X)) (X),
=(1 c-1 e+1) (E[LL (1) -(L
S(1 c-1 e+1) v (EI
<(1 e-1 e+1) V (c. (E(fgeer (f (f V) Blepeer (F (X), Vi))..
Therefore 1e (x) VV[ ) ). It's forwwd to verify that Vépee (x) satisfies the
conditions in Definition ! We conclude the proof.
Now we check condition (2). Again, from previously, we know the following holds for a certain Py = P, e,y) + (1 - Py)e e where p+1 = P.P-1 p
Bla-ppor (f
E(((S(X), a. (((())
(1 ex Y) + +ey(((()), -a000(0))-))))----11)0--0- - -x)))))(( -r)
=E
=E (1 (1- er ((())) (ee-)))))))-----aaaaaaaaaaa - -1--
Let ((f(( Y) = ())(),,),, we have
(f
=B (1 ey
Ey apr
apr )o(f(X) -Y) Y) + -a a -1- (1-xx)))(-(()))
=[(e([(X) .Y)
We first introduce Theorem:
Theorem 9 (Theorem 6, (Barllett et al. 2006 Let yo be convex. Then is classification-calibre and only if is differentiable at 0 and up' 0.
22

---

Under review as a conference paper at ICLR 2020
We now show that op is convex:
(3(3) -(1 Ey - apr) (3) (ey a. (-3)
=(1 Cy apr (3) (ey aa. py (B)
=(1 cy apr tey (1 - (B)
-(1 (8) >0
when 1. The last inequality is due the fact that é is convex.
Secondly we show the first derivative of &p is negative at (²((0) < 0:
(0( = (1 ey apv) d' (er a- (1 - py))
= (1 2ey + a(1 2fx))
(15)
Note that
P -ey) + (1 - py)e--
Plug back Eqn.
15
have
(0) (1 By - apr) :d(0)) (ey a. - -1-
(1 2ey + a(1 - -2r ())
(1 appy - 2e,) + a(1 - Py) (1 - e (0
(16)
Since (1 app)(1 - 2e,) + a(1 p1) c-n) 0 and d(()) < 0 (due to calibration property
of f, Theorem of Bartlett et al. (2006)), we proved that (0( < 0. Then based on Theorem 6 of
Bartlett et al. 20066), we know Eep is classification calibrated.
PROOF FOR THEOREM 7
Proof. We first prove the following Rademacher complexity bound
Lemma 7. Let R(F) denote the Rademacher complexity of F. L denote the Lipschitz constant of l.
Then with probability least - 5, maxfer D (f) R R. B (s) (1 +a)L R(F) +
log 4/6 2N (1 la la-peer la la-peer).
V
Note we also have the following Ya:
R4 Riana Blit Rgoom I Ff
Re bll Ri Reogen (Frnp (Be (Ben (Prmpnn Re (P PRoyei)
+ (Beom D D.ftnna Ri oan o (Prnmal
<0 max
Re Re o(F)
f
23

---

Under review as a conference paper at ICLR 2020
Then apply the calibration condition we have
RD (J - perr
R
1
(B1, ( mod R₁ ren, b(f*))
(Proposition
6-1
-
E+4
1
(Rg per
Rr
pper mon alf5a-ma
(Theorem 3
1
6-1
E+1
1
Vi
min EEF
R
min R. D()
((f
(Calibration 1 a ppar)
1--1 -
c+1
porr
+
Re
Bllt
Ri
ner B'Si mom))
nes
1
V
min FEF
R
-minn R pay B(f)
year (f)
e-1 6+1
per
f
+ max fEF
RRg
(lf)
Rgg,
(f)l
<
min fEF
Rs
D(f)
min Re sea B(f)
(Lemma 7
c-1 c+1
pprr
log 4/8
+ 2(1 + I R(F) + +
(1
lala--eeeeee peer la peer))
2N
with probability at least 5.
PROOF FOR LEMMA 7
Proof. Due to the random sampling, via Hoeffding inequality we first have there exists some Pa.. E (0, 1), with probability at least 1 5. 6,
N
N 1
Ca-porr (f(
Σ (((((z)))
N
N n=l
Po Mn) (1 Pin ((f(())) -bn))
log 2/5
(ta-poor la-poer)
2.N
Define the following loss function:
ELnn In) = ((f(xn) Tn) a ee-Pa ((,,)(( 1- 1a-i))a-)- (( -ūn))
Via Rademacher bound on the maximal deviation we have with probability at least 5
log 1/6 2N
max
JEF R( Ri, S2 RiCo F) +
-
(17)
Since & is L-Lipschitz, due to the linear combination, (1 (1+
Based on the Lipschitz
composition of Rademacher averages, we have
RiC F) s (1+ R(F)
24

---

Under review as a conference paper at ICLR 2020
Therefore, via union bound, we know with probability at least 1 28:
la peer (f (f(3))) - Ri Reoppr.D())
N
N
La la-peer - R1, + R1,0(1) - R2 (())
N
La Ea-porr (f(x)) Un) - Rep(( + -|RED(0) - R B(m
N
log 2/6
(La-poor la-poor) + R2 (2) -Rg,(
2N
log 2/0
log 1/6
(La La-ppod ) + (1 L R(FF) +
2N
2N
1og2/0 2/0
<(1 + L R(F) +
(1 La-poor - la-poor)
2N
In above R2 Rg Plug the fact that la peer is linear in l and easy consequence that
pen (ff) (f) because la-peer and l share the same expected risk by construction.
la Pa-peer la-peer S (1 + a)( f),
let = 5/2, we conclude the proof.
PROOF FOR LEMMA g
Nonetheless, despite the fact that la (a-pee(.)) (.) is not convex in general, [Lemma 5, (Naaarajan et al, 2013 informs that long Re (f) is close some convex function, mirror gradient type of algorithms will converge to a small neighborhood the optimal point when performing (J) as
ERM with fa-perr- natural candidate for this convex function is the expectation of R
Eoperr Sasee.DD.)) + R peat
(anee.DD)) (S) when 0c.
Lemma 8. When
max max((et1ee1)) < 0.5, and (t, (t, (t), R2 (...p.r..(.) is convex.
Proof. This was proved the proof for Theorem 6 when proving the classification calibration property of la-poor under condition (2).
EXPERIMENT
IMPLEMENTATION DETAILS
We implemented neural networks (LeCun et al., 2013) for classification on 10 UCI Benchmarks and applied our peer loss update their parameters. For surrogate loss, we the true error rates e and 6+1 instead of learning them on the validation set. Thus, surrogate loss could be considered favored and advantaged baseline method. On each benchmark, use the same for all neural network based methods. For C-SVM, we fix one of the weights to 1, and tune the other. For PAM, we the margin.
RESULTS
The full experiment results are shown in Table.??. ??. Equalized Prior indicates that in the correspond- ing experiments, we resample to make sure P(Y = +1) = P(Y = -1) and we fix a in these experiments. Our method is competitive in all the datasets and even able outperform the surrogate
25

---

Under review as a conference paper at ICLR 2020
Accurasy During lesting
Accurscy During lesting
0.95
0.0
0.85
0.75
0375
0.79
0.75
loss
20
50
10
23
50
ephodes (120)
aplxodes (X20)
(a) Twonorm (e-1 0.2, 6+1 = 0.4)
(b) Splice (c-1 = 0.1, 6+1 0.3)
Accuracy During Testing
Accuracy During Testing
0.7
0.0
0.5
0.6
0.5
cross antrpy
cross untrayy peer
loss
surrogate loss
surrogane loss
(c) Heart (c-1 = 0.2, 6+1 0.4)
(d) Breast (c-1 = 0.4, e+1 = 0.4)
Figure 4: Accuracy on test set during training
loss method with access to the true error rates in most of them. C-SVM is also robust when error rates are symmetric, and is competitive in datasets.
From Figure 4 its achieved high robustness across different datasets and error rates.
peer loss prevent over- wert-fitiing, which is also part of the reason of
26

---

Under review as a conference paper at ICLR 2020
Task
With Prior Equalization p 0.5
Without Prior Equalization p p 0.5
(d, N.,N4.
e- 6-1,441
Peer
Sur
Symm
DMI
NN
C-SVM
Peer
Surr
Symm
DMI
NN
C-SVM
0.3.
0.977
0.968
0.969
0.974
0.964
0.966
0.977
0.968
0.969
0.974
0.964
0.966
0.2, 0.2
0.977
0.969
0.974
0.976
0.972
0.969
0.977
0.969
0.974
0.976
0.972
0.969
Twonorm (20,3700,3700)
0.4
0.976
0.964
0.956
0.974
0.911
0.95
0.976
0.964
0.956
0.974
0.911
0,95
0.2, 0.4
0.976
0.919
0.959
0.966
0.911
0.935
0.976
0.919
0.959
0.966
0.911
0.935
0.4, 0.4
0.973
0.934
0.958
0.936
0.883
0.875
0.973
0.934
0.958
0.936
0.883
0.875
0.3
0.919
0,878
0.851
0.875
0.811
0.928
0.925
0,885
0.868
0.889
0.809
0.933
0.2
0.918
0.874
0.879
0.888
0.819
0.931
0.927
0.876
0.906
0.885
0.812
0,941
Splice
0.4,
0.914
0.86
0.757
0.842
0.743
0.891
0.925
0.862
0.777
0.852
0.754
0.898
164,18
0.4
0.901
0.832
0.757
0.801
0.714
0.807
0.912
0.84
0.782
0.81
0.725
0.824
0.4,
0.819
0.754
0.657
0.66
0.626
0.767
0.822
0.755
0.674
0,647
0.601
0.76
0.3,
0.833
0.78
0.777
0.797
0.756
0.753
0.856
0,802
0.803
0.83
0.75
0.788
0.2,
0.821
0.762
0.795
0.801
0.75
0.717
0.856
0,813
0.793
0.826
0.769
0.796
Heart (13,165,138)
0.4
0.827
0.777
0.714
0.779
0.717
0.744
0.859
0,815
0.725
0.814
0.723
0.677
0.2,
0.812
0.768
0.717
0.788
0.679
0.714
0.856
0.758
0.725
0.797
0.693
0.704
0.4 0.4
0.75
0.729
0.654
0.69
0.595
0.688
0.785
0.728
0.686
0.711
0.554
0.698
0.3,
0.745
0.707
0.674
0.72
0.667
0.67
0.778
0.75
0.738
0.729
0.727
0.726
0.2, 0.2
0.755
0.708
0.72
0.729
0.671
0.745
0.759
0.736
0.753
0.743
0.706
0.759
Diahctes (8,268,500)
0.4 0.745
0.682
0.612
0.701
0.627
0.568
0.777
0.724
0.694
0.713
0.71
0.688
0.2,0.4 0.755
0.681
0.634
0.682
0.596
0.59
0.739
0.705
0.695
0.707
0.672
0.7
0.4.0.4
0.719
0.645
0.619
0.637
0.551
0.654
0.651
0.685
0.68
0,633
0,583
0.702
0.1,
0.639
0.563
0.507
0.529
0.519
0.529
0.727
0.645
0.709
0.666
0.648
0.698
0.2
0.659
0.606
0.537
0.548
0.534
0.615
0.698
0.661
0.655
0.627
0.623
0.695
Breast (9,85,201)
0.4
0.587
0.577
0,504
0.504
0.519
0.553
0.735
0.654
0.685
0.621
0.66
0.698
0.2.0.4
0.63
0.534
0.482
0.496
0.538
0.538
0.73
0,674
0.666
0.58
0.672
0.698
0.4, 0.4
0.596
0.519
0.504
0.526
0.471
0.51
0.677
0.628
0.545
0.537
0.529
0.698
0.3,
0.928
0.922
0.924
0.934
0.873
0.924
0.956
0.949
0.943
0.954
0.92
0.943
0.4
0.932
0.938
0.937
0.944
0.83
0.85
0.951
0.929
0.946
0.941
0.898
0.929
Breast
0.2, 0.2
0.928
0.904
0.835
0.897
0.887
0.961
0.952
0.952
0.897
0.942
0.955
0.946
(30,212,357)
0.2, 0.4
0.93
0.885
0.844
0.89
0.844
0.865
0.933
0.898
0.898
0.918
0.831
0.862
0.4,
0.928
0.867
0.819
0.746
0.824
0.855
0.908
0.839
0.817
0.795
0.673
0.866
0.1
0.701
0.624
0.614
0.637
0.581
0.611
0.68
0.693
0.603
0.605
0.6
0.671
0.2,
0.689
0.65
0.647
0.623
0.611
0.664
0.702
0.693
0.704
0.62
0.6
0.738
German (23,300,700)
0.4
0.696
0.642
0.587
0.63
0.562
0.55
0.667
0.693
0.54
0.594
0.54
0.553
0.4
0.664
0.59
0.6
0.618
0.572
0.469
0.676
0.681
0.537
0.573
0.535
0.581
0.4, 0.4
0.606
0,55
0.573
0.573
0.556
0.572
0.654
0.632
0.549
0.611
0.553
0.696
0.1,
0.89
0.895
0.892
0.856
0.868
0.862
0.893
0.898
0.883
0.785
0.863
0.878
0.2,
0.883
0.899
0.9
0.861
0.894
0.886
0.901
0.899
0.894
0.792
0.898
0.897
Waveform
0.4
0.884
0.893
0.762
0.856
0.771
0.804
0.888
0.894
0.703
0.778
0.821
0.821
(21,1647,3333)
0.2, 0.4
0.881
0.89
0.828
0.835
0.81
0.795
0.884
0.884
0.745
0.761
0.837
0.837
0.4, 0.4
0.87
0.866
0.867
0.773
0.835
0.776
0.853
0.852
0.852
0.672
0.828
0.848
0.3
0.906
0.9
0.89
0.87
0.909
0
0.943
0.909
0.897
0.811
0.93
0.924
0.2
0.913
0.894
0.907
0.897
0.899
0.918
0.905
0.905
0.905
0.91
0.936
0.936
Thyroid (5,65,150)
0.4
0.875
0.862
0.834
0.784
0.88
0.869
0.902
0.924
0.856
0.75
0.919
0.917
0.2,
0.863
0.862
0.85
0.784
0.822
0.781
0.905
0.898
0.865
0.759
0.881
0.92
0.4, 0.4
0.762
0.738
0.859
0.788
0.764
0.781
0.769
0.818
0.876
0.738
0.738
0.837
0.3
0.856
0.875
0.843
0.896
0,866
0.892
0.796
0.835
0.903
0.896
0.878
0.892
0,2, 0.2
0.9
0.835
0.911
0.894
0.908
0.912
0.931
0.896
0,917
0.883
0.934
0,908
Image
0.4,
0.723
0,841
0.705
0.881
0.799
0.785
0.717
0,806
0.679
0.888
0.825
0,808
1320139))
0.2, 0.4
0.836
0.862
0.719
0.845
0.832
0.802
0.672
0.755
0.722
0.86
0.599
0.825
0.4, 0.4
0.741
0.72
0.788
0.763
0.732
0.834
0.806
0.803
0.823
0.762
0.8
0.86
Table 3: Experiment Results on 10 UCI Benchmarks. Entries within 2% from the best in each row are in bold. Surr: surrogate loss method (Natarajan et al.] 2013); DMI: (Xu et al.] 2019); Symm: symmetric loss method Ghosh et al. 2015). All method-speeifie parameters cross-validation. The pro- posed method (Peer) are competitive across all the datasets. Neural- network-based methods (Peer, Surrogate, NN. Symmetric, DMT) use the same hyper-parameters. All the results are averaged across 8 random seeds.
estimated through
27


Reviews:
[{"review_id": "r1ex9g6pKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of learning classifiers from noisy data without specifying the noise rates. Inspired by the literature of peer prediction, the authors propose peer loss. First, a scoring function is introduced, minimizing which we can elicit the Bayes optimal classifier f*. Then the authors use the setting of CA to induces a scoring matrix, and then the peer loss. Moreover, this paper explores the theoretical properties of peer loss when p=0.5. In particular, the authors propose \\alpha weighted peer loss to provide strong theoretical guarantees of the proposed ERM framework. The calibration and generalization abilities are also discussed in section 4.3. Finally, empirical studies show that the propose peer loss indeed remedies the difficulty of determining the noise rates in noisy label learning.\n\nThis paper is well written. The theoretical properties of the proposed peer loss are thoroughly explored. The motivation is rational with a good theoretical guarantee, i.e. Theorem 1. Moreover, the tackled problem, i.e. avoiding specifying the noise rates, is significant to the community.\n\nNevertheless, Some parts of this paper may be confusing:\n- The computation of the scoring matrix delta is not that clear. Can the authors provide the detailed computation steps of the example? \n- In the proof of Lemma 6, can the authors provide a proof sketch of the equivalence of the last two equations?\n- Third, where is the definition of p?\n\nIn the experiments, the authors propose to tuning the hyperparameter alpha. I would be appreciated if the authors provide the sensitivity experiments of alpha to show its fluence for the final prediction.\n\nThough I'm not that familiar with learning from noisy labels, I think it is a good paper and I suggest to accept.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1esAQqrtB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed peer loss function for learning with noisy labels, combining two areas learning with noisy labels and peer prediction together. The novelty and the significance are both borderline (or below). There are 4 major issues I have found so far.\n\nReferences: Looking at section 1.1 the related work, the references are a bit too old. While I am not sure about the area of peer prediction, in the area of learning with noisy labels (in a general sense), there were often 10 to 15 papers from every NeurIPS, ICML, ICLR and CVPR in recent years. The authors didn't survey the literature after 2016 at all... Nowadays most papers focus on sample selection/reweighting and label correction rather than loss correction in this area, but there are still many recent papers on designing more robust losses, see https://arxiv.org/abs/1805.07836 (NeurIPS 2018 spotlight), https://openreview.net/forum?id=rklB76EKPr and references therein. Note also that some label-noise related papers may not have the term label noise or noisy labels in the title, for example, https://openreview.net/forum?id=B1xWcj0qYm (ICLR 2019).\n\nMotivation: The motivating claim \"existing approaches require practitioners to specify noise rates\" is wrong... Many loss correction methods can estimate the transition matrix T (which is indispensable in any loss correction) without knowing the noise rate, when there are anchor points or even no anchor points in the noisy training data. See https://arxiv.org/abs/1906.00189 (NeurIPS 2019) and references therein. See also the public comment posted by Nontawat when a special symmetric condition is assumed on the surrogate loss function.\n\nNovelty: The paper introduced peer prediction, an area in computational economics and algorithmic game theory, to learning with noisy labels. This should be novel (to the best of my knowledge) and I like it! However, the obtained loss is very similar to the general loss correction approach, see https://arxiv.org/abs/1609.03683 (CVPR 2017 oral). This fact undermines the novelty of the paper, significantly. The authors should clarity the connection to and the difference from the loss correction approach.\n\nSignificance: The proposed method focuses on binary classification, otherwise the paper will be much more significant! Note that the backward and forward corrections can both be applied to multi-class classification. Moreover, similar to many theory papers, the experiments are too simple, where single-hidden-layer neural networks were trained on 10 UCI benchmark datasets. I have to say this may not be enough for ICLR that should be a more deep learning conference.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BylUU8YTFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the label noise problem with the motivation of without estimating the flip rate or transition matrix. This is an interesting direction for dealing with label noise. Most of the previous studies need either estimate the transition matrix or put restrictions on it, e.g., to be symmetric. A very related work to this paper: L_{DMI}: A Novel Information-theoretic Loss Function for Training Deep Nets Robust to Label Noise, where no restrictions have been made on the class-dependent transition matrix and the proposed method does not need to estimate the transition matrix. The authors may need to discuss the paper.\n\nThe paper is not well-presented. I tried several times to go through the details but failed. The reasons are, e.g., (1) notation is not clear, e.g., R(X). \\tilde{Y}, and R have been abused. (2) Intuitive explanations are limited. (3) Lots of details can be put on the appendix, keeping the main part to have a strong and clear logic.\n\nIt seems the theories of the paper depends on a very strong assumption, i.e., \"Suppose S(.) is able to elicit the Bayes optimal classifier f^*\".  By quickly go through the proofs in the appendix, it seems there is a strong connection to the paper L_{DMI}.\n\nSome claims are strong. In the literature, with a mild assumption, the class-dependent transition matrix can accurately be estimated just from noisy data with theoretical guarantees. There are also methods proposed for this. Estimating class-dependent transition matrix is not a bottleneck. I think the challenge is about how to learn instance-dependent transition matrix.\n\nOverall, this is an interesting paper but needs to be improved.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you very much for the detailed feedback to the reviewers, which helped us better understand your paper.
Thanks also for revising the manuscript significantly; many parts were indeed revised. 
However, due to the major revision, we find more points to be further discussed, which requires another round of reviews/rebuttals.
For this reason, we decided not to accept this paper.
We hope that the reviewers' comments are useful for improving the paper for potential future publication.

Decision: Reject