Can Less be More? When Increasing-to-Balancing Label Noise Rates Considered Beneficial
Authors: Yang Liu
Jialu Wang
Venue: NeurIPS 2021 Poster (2021)
Keywords: learning with noisy labels
fair classification
data pre-processing
label noise rate balancing

Abstract: In this paper, we answer the question of when inserting label noise (less informative labels) can instead return us more accurate and fair models. We are primarily inspired by three observations: 1) In contrast to reducing label noise rates, increasing the noise rates is easy to implement; 2) Increasing a certain class of instances' label noise to balance the noise rates (increasing-to-balancing) results in an easier learning problem; 3) Increasing-to-balancing improves fairness guarantees against label bias. In this paper, we first quantify the trade-offs introduced by increasing a certain group of instances' label noise rate w.r.t. the loss of label informativeness and the lowered learning difficulties. We analytically demonstrate when such an increase is beneficial, in terms of either improved generalization power or the fairness guarantees. Then we present a method to insert label noise properly for the task of learning with noisy labels, either without or with a fairness constraint. The primary technical challenge we face is due to the fact that we would not know which data instances are suffering from higher noise, and we would not have the ground truth labels to verify any possible hypothesis. We propose a detection method that informs us which group of labels might suffer from higher noise without using ground truth labels. We formally establish the effectiveness of the proposed solution and demonstrate it with extensive experiments. 

Can Less be More? When Increasing-oo-Balanning Label Noise Rates Considered Beneficial
Yang Liu
Jialu Wang
Computer Science and Engineering University of California, Santa Cruz Santa Cruz, CA 95064 yangliuBucsc.edu
Computer Science and Engineering University California, Santa Cruz Santa Cruz, CA 95064 faldict@ucsc.edd
Abstract
In this paper, we answer the question of when inserting label noise (less informative labels) can instead return us more accurate and fair models. We are primarily in- spired by three observations: 1) In contrast to reducing label noise rates, increasing the noise rates is easy to implement; 2) Increasing certain class of instances' label noise to balance the noise rates prrrassingtt-balancinn results an easier learning problem: 3) Increasing- tooiiinggt--beeee to-aaannnn--lllninnggggng improves fairesss guarantees against label bias. In this paper, we first quantify the trade-offs introduced by increasing a certain group of instances' label noise rate w.r.t. the loss of label informativeness and the lowered learning difficulties. We analytically demonstrate when such an increase is beneficial, in terms of either improved generalization power the fairness guarantees. Then we present a method to insert label noise properly for the task of learning with noisy labels, either without with faimess constraint. The primary technical challenge we face due to the fact that we would not know which data instances ground truth labels to verify any possible hypothesis. We propose a detection method that informs us which group of labels might suffer from higher noise without using ground truth labels. We formally establish the effectiveness of the proposed solution and demonstrate with extensive experiments.
suffering from higher noise, and would not have the
Introduction
The presence of training label noise is generally considered harmful. Typically the goal of learning with noisy labels is improve the training by reducing the amount of noise in the training data [4, 1 1]. This paper discusses the feasibility of the opposite strategy and shows cases when adding label noise, leading to a scenario with less informative labels, will result in more accurate and fair models. We are primarily motivated by three observations:
Observation I: Reducing label noise is hard, but increasing it is easy. While the literature has provided us with solutions to perform data cleaning, they often require highly customized training process. Their lack of theoretical rigor has also posed challenges when performing evaluations. On the other hand, as we will see later, increasing label noise is easy - one can always do so by randomly flipping the current noisy labels increase further.
Observation II: Inereasing a certain class of instances' noise rate to balance the noise rates results in easier learning problem. When label noise is class-dependent (label class Y +1 v.s. Y = =-1, -1), popular noise-tolerant learning algorithms typically require the knowledge of noise rates [25]. However, the learner often needs to identify the unknown noise rates by some estimation procedure [26, 27, 36]. We articulate that the mis-specification noise rates will introduce additional learning errors, especially when the label noise is asymmetric (see Theorem 1). On the other
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

hand, although increasing the noise rate of the lower class one to balanced error rates reduces the informativeness of the training labels, is regarded as an easier case handle- when error rates are balanced, invoking loss correction procedures becomes unnecessary (see Lemma 2 & Theorem 3).
Observation III: Increasing-to-balanciii
improves fairness guarantees against label bias. When the label noise rates are group-dependent (data from young group v.s. senior group), it has been reported in the literature that imposing fairness constraints directly on the noisy labels would instead reinforce the unfaimess [30]. Fixing the fairness constraints again requires the knowledge of the label noise rates. We will show that increasing and balancing noise rates allows us to directly impose faimess guarantees on the noisy training data without knowing the noise rates (Theorem 6).
This paper first quantifies the trade-offs introduced by increasing a certain group of instances' label noise rate with respect to the loss of informative labels and the gained benefits for doing so. We analytically demonstrate when such an increase proves beneficial in terms of either improved generalization error or fairness guarantees. Then we present a method to leverage our idea of inserting label noise for the task of learning with noisy labels, either without or with a fairness constraint. The primary technical challenge we face is that we would not know which data instances are suffering from higher noise, and again we would not have the ground truth labels verify any possible hypothesis. In response, we propose a detection method that informs us which group of labels might suffer from higher noise without using ground truth information. The core discovery is a couple of metrics that check the agreements of noisy labels among local neighbors. These two metrics can be easily estimated using noisy labels only and are shown to be sufficient inform us of the class of labels with a higher noise rate. With this knowledge, we propose an algorithm (NoISB+) to gradually insert noise into the lower noise class of labels. Our contributions summarize as follows:
Our paper prototypes the idea of pre-processing noisy training labels for both the constrained and unconstrained learning problems. We show the possibility of improving the training accuracy and fairness guarantees by increasing certain class of instances' noise rates (increasing-to -orreaiinggt--blaaaeiinn)
To enable the deployment of our idea, we propose a detection algorithm to identify the class of instances with higher label noise rate without using any ground truth label information.
Our solution contributes to the learning with noisy labels literature by adding another tool that is robust to noise rate (noise transition matrix) estimation errors.
All omitted proofs can be found in the Appendix. The code for reproducing the experimental results is available at https: / //t//tttttttt wiight..coo.. wtt.com////cc// /UCCc//CCC--CC CEEisSCREAL/CCneee BEALLCanLesssBeoore
1.1 Related work
Our work is contribution to the well-established literature of learning with noisy labels [11, 22, 25, 27-29, 33]. The classical solutions within the literature leveraged the knowledge of noise rates to perform loss correction [25, 27, 28]. label correction [23, 27], loss reweighting [19], among many other treatments. The importance of knowing the correct noise rate parameters is clearly established. Follow-up works have provided tools for estimating the noise rates without accessing the ground truth label. These include the confident learning approach [26]. anchor point [5], and more recently clustering- based ones [36]. Nonetheless, the possibility of unintended harm due mis mis-specified noise rates remains. More recent works have looked into robust loss functions that would not require the knowledge of noise rate, hoping improve the resistance the mis-specifications [14, 21, 32, 34, 35). Our work can be viewed as a contribution to this specific line of literature.
Our work also contributes to the understanding of fairness implications when learning with noisy labels and other imperfect information [3. 7, 10, 17, 20, 24, 30, 31]. For instance, [17] considers amending noisy sensitive attributes by appropriately re-scaling the fairness tolerance but is only restricted to conss-conditional random noise. [30] proposes two noise- approaches by constructing unbiased estimators with group-dependent label noise. While most of the existing works would either require the noise rates to be balanced between groups or the knowledge of the noise rates perform noise correction [30], our work requires neither.
fair classification
2 Preliminaries
We will consider a binary classification problem with d-dimensional feature space X E Rdd and label space Y E (-1,+1), (X, generate according to distribution D. Instead of accessing the clean

---

data, we consider the setting where the learners only have access to noisy labels Y. The generation of
Y follows the noise transition model: e+ (x == PP = - -1-Y = +1, = z), e- := PP =
+1|Y = 1, X =). We further assume Y generates conditionally independently for different X. For now, we do not restrict how e+ (x), e- (4r) depend on x but to enable our analysis, later we will.
Denote the noisy distribution for (X, Y) as D.
In the second constrained learning setting, we assume that for each instance (X, Y), we also observe a sensitive attribute Z E {a, b) that defines two protected groups (e.g., senior v.s. young people) that we will later enforce fairness constraints over.
Unconstrained learning problem. In the unconstrained setting, denote the collected training data as D = {TEn (n)) The goal is to train classifier h E H. that minimizes the empirical risk [p[I(((X) + Y)] with only accessing D, where 1(-) is the 0-1 loss function. We will focus on the standard class-dependent noise rate model where e+ = c.. (x) e Vx. We assume we have informative labels: 6,, < 0.5. Hereby the noise rates depend only on the true label class [25], but not the specific instance when conditional on the true label. We will operate in the challenging asymmetric error rates setting when e. * e..
Fairness constrained learning problem. In the fairnsss constrained setting, denote the collected training data as D [mn, 2n)) kn)]n The goal is to train classifier h E H that maximizes the accuracy while satisfying a certain fairness constraint, measured by a fairness metric (h)::
min (d(1(4(X) + YY] s.t. (Pa(h) - Fo(h)) S 6, NEW
(1)
wherein 5 >> 0 is a tolerance parameter for faimesss violations. Exemplary Fa(h) includes the Equal Opportunity measure F. (h) P(h(X) = +1|Y = +1, a) [12], the demographic parity (h( == XX) = +1|Z = a) [6], among others.
= 6- We assume the error rates are group dependent [30]: e = E2, = (a,b). Again we assume ca, 6b < 0.5. Within each group, we can further assume that the noise rates are class- dependent, but then we will simply apply our solution within each group (hetween +1/- - 1 classes) and across the group. We remove this additional layer of complication to stay concise.
Notations. We define the following quantities that we will repeatedly use. We will denote by (h) : E * Y)] the true generalization error of a classifier h incurred on the clean distribution D. For a classffcatiioncaliiaatte as (h) == Ep (x)),,(X),)))) Y)]. Denote the empirical l-risk of a classifier h on dataset D as (h) == [a] E(.,)ED {(h(z),))
loss function, we will denote its risk
Our solution is generically applicable to multi-class settings, which we will explain later and demon- strate with experiments. For a clear exposition of our core idea, we focus on the binary case.
3 Equalizing error rates can improve model accuracy
In this section, we focus on the unconstrained learning setting. We will show that while increasing one label class' noise rate raises the generalization errors due to less informative labels, at the same time, it helps remove the training's 's dependency on the noise rates (e_.. e_),, and therefore improves robustness to possible mis-specification of them. We will quantify such trade-off
We would mainly demonstrate our idea by comparing to methods that explicitly use the knowledge of noise rates. For one reason, these approaches often have strong theoretical guarantees. The other reason is that the benefit of our approach of not requiring the noise rate would be clearer for this comparison. Particularly, we will use loss correction [25] as the running example. We are aware of the other recent works that do not require the noise rates, First of all, these approaches focus on improving model accuracy, but not the fairness constraints. Secondly, empirically we observe that these approaches can also benefit from our noise rate balancing procedure. We left the discussions to our experiments, where we will demonstrate using one of such losses, peer loss [21].
Loss correction. Once knowing e4,e_, there exist different ways improve training robustness in the presence noisy labels. We consider demonstrating our idea using the classical loss correction framework [25, 27]: denote by ē④, ē_ the estimated version of e4, e_. In practice, this knowledge
3

---

can be obtained by using existing techniques [19, 26, 36]. For a given calibrated loss E, the loss correction approach defines the following loss:
E(h(zn); Bn)) == (1 - é sgn(()))) [(h(Tn), Mn)) Eggn((.) ((h((x)),)))) -bn.)
(2)
where in above sgn is the sign function. A favorable property of is its unbiasedness (expec- tation over Y|H = y) w.r.t. E(h(2), 3) when given the correct noise rate parameters e4, e-. For readers who need more background knowledge of loss correction, Appendix ??. A classifier will be trained following empirical risk minimization (ERM) us- ing E: 1 == arg min Σ ((h((z)). (n). Compared to the standard loss correction, we omit the 1 1- é_ ê+ term from the denominator, which will not affect the minimizer of our ERM problem. Denote hi arg min R2 Re,D(h), the optimal classifier for (k,)(h), and erry == max e+1, 1_ - 1 the maximal mis-specification error. When is L-Lipschitz and bounded by E, let L1 == 4L, L2 == 26 > O, adapting the proof from [25] we show:
reproduce details in
Theorem 1. For any 8 > O, we have with probability at least 1 5:
errи
log 1/5
(c, (e,(hipp)
1-4+-e -1 R(H))
1 e e L2 + 22
(3)
2NV
label informativnness
más mix-speccfiication
where in above R(H) denotes the Rademacher complexity H. H.
Clearly, e4,e_ control the error due to label informativeness: a pair of e4,e_ with higher sum (e+ + e-) induce larger generalization error bound. On the other hand, the potential imperfect estimates of them introduce additional error due to the mis-specification.
3.1 When does equalizing error rate improve performance?
Now we first provide intuitions for why increasing noise rate to balance e4 and e might be considered helpful. Without loss of generality, suppose e+ > e.. When we increase to match e4 such that e+ e- (later will explain how do so), denote by Y the newly generated noisy label with e symmetric error rates for both classes. Note that e < 0.5. Denote by In the generated label for example n, D the distribution of (X, Y), and ò the dataset {(In, }n)) We first show: Lemma 2. When < 0.5, minimizing (x) # Y) is equivalent with minimizing P(h(X) * Y).
The above lemma says that the minimizer of (XX * Y' is equivalent to the optimal classifier for P(N(X) # Y) when the hypothesis space H covers the optimal h! Therefore when l is classification- calibrated on the noisy distribution D, and when the hypothesis space is sufficiently large, we know that the optimal classifier for E, [)(((((X), Y)] will be exactly the same as the one for [E][(((XX)) Y)]. This further implies that when the етror rates are symmetric, the training might not need the error rates information, and performing ERM on the noisy data:
N
h2D = arg min
Σ e(h(an). Un)
(4)
hEH n=1
suffices to find the optimal classifier for the clean distribution, when given a sufficient amount of data. The above argument is slightly trickier when considering finite hypothesis space H. Denote by -h the classifier that always flips the prediction from h. Then we prove that
Theorem 3. Suppose 1) l is symmetric s.t. E(h(x), -y) {(-h(xx), y) and 2) -h, -hi,p E H. Re,0(-hi,0) 2 h- Then for any 8 we have with probability at least 1 8: Re, Re,
1- - L1 R(H) 2/
1/8
2N
The second condition above is simply stating that the opposite classifier of the empirically optimal one incurs high empirical loss, and particularly higher than -hi Intuitively, this condition is satisfied for binary classification - if one classifier performs the best the empirical data, flipping its prediction often results in very wrong one. Now comparing Theorem and 3 we observe that equalizing noise rates increases the error due to loss of informative labels by:
1
1
le+ c-1
La R(H)-
L1 R(H).
1 1- 2c
1
e+
11 11 c+ e- 2c+1

---

On the other hand, equalizing noise rates avoids the error due to mis- mis-speciffing c, by I-c+-e.
R(H)) The L-- Therefore, it is considered helpful increase e- e+ when erry 2 le+ 1-_2__ 12
above condition is likely to hold when the gap between noise rates |e+ - e_ is sufficiently small, and when we do not have high confidence in the estimation of e (therefore high erry).
Equalizing error rates improves fairness guarantee
In this section, we consider the fairness constrained learning problem and show that equalizing noise rates helps improve fairness guarantee when only accessing the noisy labels. There are generally two types fairness constraints: those that do not depend on the label and those that do. For the ones that do not, for example, demographic parity (X) = +1|Z a) PP(h(X) +1 = b), the existence of noisy labels does not impose additional challenges when enforeing such constraints. For those that do, typically they are functions of true positive rates (TPR. also known as the equal opportunity measure) and false positive rates (FPR). was shown [30] that equalizing TPR and FPR the noisy label leads to disparities when * We will reproduce the same observations. We focus on the second type of constraints: equalizing TPR and FPR using the noisy labels.
Having noted that equalizing TPR and FPR on the noisy labels leads to unintended consequences, the literature has observed recent works on performing fairness constraint correction [30]. Similar the loss correction approaches, such correction methods would again require the knowledge of ea eb. which is subject to mis-specification error. Define
(h) = (X) +1|Y +1, 2 2), FPR_ (h) = P(N(X) = +1|Y = -1, 7- = =)
(h) == (X) +1|8 = +1, =). F == P(N(X) = +118- = -1,7
Resampling the noisy data examples such that p(ý = = = 2) =0.5, {a, b), define C2.1 = 0.5 eg: C2,2 0.5 (1 e2), we derive the following relationship:
Lemma 4. (h), FPR, (h) relate to (h), FPR, (h) as follows:
(P) C².1 TPR. (h) C2.2 (h) FPR (h) C².1 FPR - C2,2 TPR. (h)
e, 0.5
6x 0.5
(5)
The above lemma first implies that since groups a, b might have different C2,1 1 C2 equalizing TPR, FPR using the noisy labels naively is insufficient to guarantee equalization of TPR, FPR.
Unfairness due to model error On the other hand, Lemma 4 points out way perform constraint correction using the knowledge of ea, eb. Denote by Ea, éb (both < 0.5) the estimated copies of suffer from the following mis niisspeeiffication::: erry
ea, Gb that have access to. Suppose min(err, jea- eal, erry == jē exl)- Denote the corrected TPR and FPR using TPR and FPR
as well ε Ea as TPR (h). FPR. (h): Define Č2,1 := 0.5 εSS Öz = 0.5 (1 Ez). and:
(h) Č²,1 TPR, (h) Ō2,2 FP)
FPR Ć2,1 FPR, (h) Č222 TPR, (h)
é2 0.5
é - 0.5
Theorem 5 establishes possible fairness violation due to errs, noise rates mis-specification:
Theorem 5. Equalizing TPRR (h) & FPR for a, leads to following possible fairness violation:
ITPR. (h) (h)
(P)
err
TPR, (h)
(2e.. 1)(2ēn 1)
(2es
1)(2e, 1)
FPR.(h)
err
FPRu(h)
(h).(h) FPR((h)) errs
(2e.. 1)(2E. -
err, (2e, 1)(22n 1)
But as a consequence of Lemma 4, we immediately know that
Theorem 6. When ea cb. equalizing TPR and FPR suffices to equalizing the true TPR and FPR.
5

---

This is simply because equalizing ea and es also equalizes both C².1 and C2.2. Theorem 6 helps us establish strong fairness guarantee; as long as we know the training data has balanced label noise rates across different groups, enforcing the constraints directly on the noisy data suffices to guarantee the equality between true TPR and FPR.
Our above observation points out increasing one group's label noise to match another helps establish
the fairness guarantees more easily. Consider the case ea eb. Now suppose we are able to increase e, to match e such that ea ey. Denote by Y the newly generated noisy label with symmetric eπror rates for both groups, and correspondingly On the newly generated training labels. Then performing fairness constrained ERM directly on D == [Irn Un, za))n:
h E
arg min LEH
Σ E(h(xn)) Un) s.t. (h) Fs((h) S
will help equalize TPR and FPR between two groups when the number of training data is sufficiently large. In above, (h), (b) are the empirically computed faimess measures using D.
5 Identifying cleaner class without using ground truth label
With the aforementioned benefits of balancing the noise rates, the remaining technical question is to determine how does e+ e, compare to e;? Of course, when we have access to ground truth labels, we will have a rough estimate and carry on to insert label noise (e.g.. by further randomly flipping the noisy labels, which we will discuss at the end of this section). In this section, describe another approach without the need for ground truth labels. We prove its sufficiency in identifying the noisier class of labels. We demonstrate
idea using the unconstrained learning setting for detecting sgn(e+ e-), but the idea easily generalizes to the fairness setting to detect the order of 6a and Cb-
5.1 Noisy label agreements
We first present the following definition:
Definition 7 (Clusterability). We say the dataset D satisfies 2-NN clusterability if each instance shares the same true label class with its two nearest neighbors measured by || =2'll2
For the rest of the section, we will assume that D satisfies 2-NN clusterability. 2-NN was similarly introduced in a recent work [36] and has been shown to be requirement that is mild to satisfy. Now we define the following two quantities that central to the development of our idea. For arbitrary instance X, with noisy label Yi. denote the noisy labels for two nearest neighbor instances of X, as Y2, Y/.. Define the following agreement measures:
Definition 8 (2-NN Agreements). Let Yı denote the noisy label for randomly selected instance X Y2, Ya are the noisy labels of Xi 's 2-NN instances (measured by ||x x x'112).
Positive Agreements
PA = P(2 = Y, +118 = +1)
(8)
Negative Agreements
NAp = P(2 = Yg = - 111; -1)
(9)
PAD computes the likelihood of the neighbor data points "agreeing" on positive label. NAp computes the one for the negative label. Now we will first sub-sam sampl the noisy distribution and compute PAp, NAp:
Step 1 Sample Y such that IP = -1) IP
F3+ =
-1) 0.5. Denote this resampled distribution as Do.
2AH Agresment for
2-N Aumemmet
Step 2 Compute NApe, PApe.
hepalve Label
Por Labe
Unbrawn chay
Next we prove that knowing NADe, PA suffices to in- form the order between e4, (sgn(e4 -e-))
Figure 1: 2-NN Agreements. "Trans- parent" ones are the solid instance's 2- NN instances.
Theorem 9. When e4,e_ < 0.5 and D satisfies 2-NN
clusterability, PADe, N relate to C4,e- as follows: PA NA = 200.5 - (0.. - --) -.4) 64). Then if PA > NAps, we know that e < e-; otherwise e e_. If PA NApo, then e+ e_.

---

Estimating PA NAps using D Note that both PApe and NAps, P(Y2 = Y = +1)
and P(22 = Y/3- = - = -1) can be counted from the noisy data without accessing the ground truth. For instance for PApe:
Step Find all Fy in D with Un +1 and their 2-NN 11 f2-
#fn: B. Bmg (-) counts the number of samples that satisfy the condition.
Step Count
# b=
We show pseudocode for an implementation of estimating PA in Figure ?? in Appendix ??.
5.2 Our solution: NOISE+
We show that randomly flipping Y = y with the smaller esgn(y) by small probability monoton- ically decreases the gap between noise rates le+ e-l. Without the loss of generality suppose e+ < c-. and we will only flip the y +1 labels (but not flipping the ones with Ý = -1). We show numpy-like pseudocode for the flipping function in Figure 3, and the implementation for exclusively flipping Y -1 labels is symmetric. Denote by Y as the flipped version of Y: = = +1) = c, and è = P( = - +1), é_ == P(Ý +1|Y -1). We have: Proposition 10. e+ (1 e4)
(1- ee.. Further, the new gap between the noise rates of the flipped label Y is monotonic function of e: é_ é+ e e (1 e+ e-)
Since 1- -- e+ + e > 0, when is small, the above derivation shows the effectiveness in reducing the noise rate gap e..- e+ by randomly flipping the noisy labels that correspond the class with lower noise rate. The only remaining question is how to find the optimal s.t. é_ ê = 0.
Calling Theorem we know PAD= NA 200.5 (0.5 é_) ε+). Denote by
f(c) == 0.5 (PAp= NADe).
Easy to derive the three solutions for f()) 0 (setting each of the terms to 0): 61 1 0.5 < 0,
e-
62 I-E++e_ 63 I-e+ 0.5-e+; note that é2 = I-e++e- +e_ (I-e_te_)+(1e__-
(e- (e--a_)-(-e---__)
1-2e+
2(1-+)
E3g
and 63 will lead to an uninformative state where e = 0.5. Therefore 62 is our target root.
The monotonicity of f()) from 0 to e2 motivates us to look for a proper by a binary search procedure. Suppose we have two different flipping parameters
0.015
G,. Initially the synthetic datasets induced by them are Dj and D,,, and the gaps of the counted agreements are C PAD,- -NDD, and C, = PAD, NAD, satisfying C > > Gr- In each iteration, we try a new flip parameter <mid = (er f)/2,, and check whether the new gap Cmid is bounded by a threshold 7. T S Cuid I 7 return the labels flipped by midd- Otherwise, we update the values of eı and e, according to the sign of Cmia: if Cmia < O, set E +- emid, D, t- Dmid (reducing E,); otherwise et < émid, Dj + Dmia (increasing E1). We summarize NOISE+ in Algorithm 1. Here, we initialize 6, 0.3 which empirically succeeds in varied noise settings. In case e, 0.3 fails, we can grid search variety of e values (e.g.. 0.1, 0.2) satisfying C, as the initial to proceed our binary search algorithm. Note that Algorithm 1 assumes 6, < e_, and the implementation symmetric for e+ > c...
0010
0.005
PA
0.5
0.000
0.005
0.00
0.3
Figure 2: Agreement gap PA -NA varies for different c. There are only two positive roots for PA NA = 0. The less one results in é = =__.
Selection of loss function. Our algorithm is particularly suitable for losses that do not require the knowledge of noise rates. Standard cross entropy (CE) would certainly be applicable. Other robust loss functions are great options too. We note recently proposed loss, peer loss function [21], does not require the specification of noise rates:
Epecr (h(o.), (..) = E (M(x.) (n) a - (h(Fp, Dppp)
Here, a > is hyper-parameter to control the balance of the instances for each label, p1 and p2 are uniformly and randomly selected peer samples. Peer loss is also promoted in [30] for the fairness constrained setting. We believe particularly suitable for our NOISE+. We will evaluate CE and peer loss in the experiment section.
7

---

Fairness constrained learning. Our above derivations and observations can be similarly applied to the fairness constrained setting. The only difference lies in that now PA NA are corresponding to the agreements within each of the two groups a, instead of the two label class group Y +1 and Y -1 We provide details in the Appendix ?? without repeating them here.
Training. Once we have the output and nncree-ineeeennte dataset δ == (( Gn)) a from Algorithm 1, we will use it directly in our ERM framework for the unconstrained case as in Eqn. (4) and will use in the fairness constrained ERM in Eqn. (7) directly too.
Extension to multi-class. As explained at the beginning, our algorithm can largely extend to the multi-class/grouy setting. The primary requirement of the extension is extend the definition of PApe, NApe to each label class/group. We defer details to Appendix ??. In the next section, we will demonstrate the effectiveness of our results with the CIFAR-10 dataset.
Algorithm 1 NoISE+: A binary search algorithm for
def flip (dataset, epsilon): unpack the dataset (x, y) = dataset
balancing noise rates.
Require: 7 > 0, 6 = 0, 6, = 0.3
Resample balanced set D from D: Initialize D:- DD
filter positives y-plus yly > flip the positive labels w.p. epsilon is flipped numpy. random binomial epsilon, y-plus size
Flip(D" (r);
Estimate PAc, PAp, NA NA
while PAp, - NAp, > and PA DA < -y do
< (EL + f/2 Dmia Flip(DD,
Estimate PA and NA
PA NADma <-> then 1 emid is at the right of the root %/ E, + Emid, D, + Danei PAD, +- PA NAp, NA
yly 0] - is is_flipped (-y-plus) + (1
else if PAP NArma > then 1s Emid is the
is_flipped) y-plus
the root #/ El 4 Emid, Di 4- Dmid;
pack and return the dataset
PAD, < PAD NAp, <- NADmii
return (X, y)
else
return ò Flip(D, (a c2)/2); end if end while
Figure Pseudocode for Flip. Flip takes the dataset and a small probability e as input, and only flips positive exam- ples with probability
return unsuccessful;
6 Experiments
In order to verify the power of our increasing- to-balancing method, we conduct extensive experiments on both unconstrained learning and constrained learning settings. The datasets include: the UCI Adult Income dataset [9], the Compas recidivism dataset [2], Fairface [15] face attribute dataset, and CIFAR 10 [16] dataset. We defer more dataset details Appendix ??.
6.1 Unconstrained learning
Setup. For the unconstrained learning, we implement a one-layer perceptron for binary classification on Adult, Compas, and Fairface datasets. We flip the true labels on training set according to a set of asymmetric noise rates, train the models the corrupted labels, and evaluate the accuracy test with clean labels. The baseline methods we compare include: surrogate loss [25] with mis-specified noise rates (Mis. SL) and estimated noise rates (Est. SL) respectively, vanilla cross entropy (CE), and peer loss functions [21] (Peer). For Mis. SL, we randomly generate the noise parameters but ensure that the trace is equal that of the true noise transition matrix. For Est. SL. we estimate the noise rates by adopting the confident learning procedure [26]. We note that the training of peer loss functions does require the knowledge of noise rates, and follow their instructions tune the hyper-paramen parameters through cross-validation. For the same corrupted training labels, we deploy our increasing-tor wawtasiingtt--aaaaneeee tolbaaanning procedure, and evaluate the performance functions again. We set the threshold used in Algorithm as 0.1% on Adult and Fairface datasets, and loose to 1% on Compas due its much smaller data size. We repeat all the methods 5 runs with different random seeds and report the mean and standard deviation.
cross entropy and peer loss
8

---

Table 1: Binary classification accuracy of compared methods on datasets across different levels of noise rates. Mis. SL: surrogate loss [25] with misspecified parameters. Est. SL: surrogate loss [25] with estimated parameters. CE: vanilla cross entropy. Peer: peer loss function [21]. All methods are trained with one-layer perceptron with the same hyper-parameters. For each noise setting, we average across runs and report the mean and standard deviation. We find that the increasing- to- bolaaanniing can boost the vanilla cross entropy on all the noise settings. We highlight any boost of macreasing-to-balaeeee for vanilla CE in green, and the best accuracy in blue.
BASELINES (LESS NOISE)
NOISB+ (MORE NOISE)
Dataset
Mis. SL
Est. SL
CE
Peer
CE
Peer
Adult
0.0
0.1
72.79 0.34
72.64 0.38
72.03 + 0.29
72.77 + 0.32
73.62
+ 0.37
73.86 + 0.41
n
48, 842
0.0
0.2
72.27
0.39
71.26 1 0.38
71.95 1 0.34
72.73
73.52 + 0.58
28
0.1
0.2
73.02
+ 0.50
72.68
72.31
0.25
72.88
0.14
71.92 1.98
73.81
0.1
0.3
72.44
0.47
0.23
09.06
2.01
72.26 0.43
09.53 4.90
73.34
1.27
0.2
0.3
72.81
+ 0.51
72.43 0.14
71.44 + 0.93
72.78 + 0.28
73.75 + 0.26
0.2
0.4
72.06
+ 0.19
1.58 63.49
71.97 + 0.37
4.999
71.43 +: 2.26
0.3
0.4
52.65
+
72.67
71.55
0.88
73.49 + 0.18
1.84 72.54
74.27 + 0.20
Compas
0.0
0.1
00.36 + 1.05
66.04 + 1.14
06.16 1.13
68.06 1: 0.70
67.14 0.92
68.22 1 0.68
7, 168
0.0
0.2
66.84 0.69
66.06
1 0.81
65.38
1.40
68.03 1 0.77
66.51 1.90
68.40 1 0.78
= 10
0.1
0.2
66
1 0.43
65.69 + 0.57
65.91 + 0.97
67.49 1 0.40
66.54 +: 0.21
67.80 + 0.44
0.1
0.3
65.91 + 0.42
65.22 = 0.63
61.24 0.70
67.36 + 0.79
65.76 + 2.09
68.05 + 0. 0.56
0.2
0.3
65.00 + 0.72
65.86 - 1.09
65.00 1.48
68.02 - 0.94
06.46 + 1.27
68.04 + 1.11
0.2
0.4
64.82 + 0.52
65.47 0.46
59.68
67.37 + 0.54
3.31 63.85
68.39 0.56
Fairface
0.0
0.1
87
1: 0.03
87.75
0.03
87.41 0.11
87.58 1 0.15
88.23 0.07
88.49 1 0.12
108, 501
0.0
0.2
85.22
1: 0.06
85,83
0.00 0.08
85.08 0.16
85.18 1 0.16
88.55 0.03
88.67 1 0.03
d= 50
0.1
0.2
87.67
+ 0.07
87.56
0.04
87.21 + 0.08
87.28 + 0.05
88.45 + 0.06
88.65 + 0.07
0.1
0.3
72.03
1: 0.13
85.68
1 0.07
83
1: 0.12
84.58 1 0.09
87 0.14
88.50 1 0.12
0.2
0.3
74.18
1: 0.20
+7.3410.11
86.47 0.09
87.00 0.11
48.46-2 0.08
88.58 + 0.10
0.2
0.4
58.30
1: 0.23
85.48
0.09
78.33 0.63
84.05
+ 0.13
81.90 0.58
87.69
0.15
Results. We show the experimental results in Table 1. We observe that our wrresiing-to-balandd method significantly improves the accuracy of the vanilla cross entropy on the majority of noise set- tings. is notable that the cross entropy with prreaaing-tt-balancii with Est. SL, when the noise is relatively small. When the noise rates are large (e.g. 0.2 and 0.4), we observe a significant degradation for cross entropy, but crreasing-to-balancii still boosts the accuracy. Moreover, we find that peer loss after increasing-te torelaannngtt is more robust to noise, and dominantly achieve the highest accuracy.
has a comparable performance
Table 2: Accuracy of compared methods across different lev- els of noise gap for multi-class classification.
Table 3: Accuracy of com- pared methods on Cloth- ingIM dataset.
LESS NOISE
MORE NOISE
Dataset
noise gap
Mis. SL
Est. SL
CE
Peer
CE
Peer
Method
Test Accuracy
0.1
89.59
89.69
86.66
88.12
86.81
89.19
MNIST
88.61
84.53
87.21
85.97
89.12
Loss Correction [27]
CE
69 84%
68.941%
0.3
84.97
86.88
85,24
86.35
81.89
88.75
Co-Teaching [11]
70.13%
0.1
70.90
85.76
88.03
89.66
88.69
89.90
CE + NOISE+
70.37%
CIFAR-10
0.2
80.51
$6.34
88.43
89.36
80.01
90.08
81.30
90.61
89.78
90.24
87.98
89.92
Multi-class extension. We test NOISE+ in the multi-class setting by balancing the noise rates class by class. We evaluate the compared methods on MNIST [18] and CIFAR-10 [16] with more sophisticated noise transition matrices. Considering that both datasets have 10 classes, we adopt the following procedure to generate the 10 x 10 noise transition matrix: (1) manually set the diagonal elements at least 0.4; (2) permute the diagonal elements to increase the randomness;
(3) fill out the non-diagonal elements randomly and ensure the sum of each column is 1. An MLP model is trained from scratch on MNIST dataset, while a pre-trained vision transformer [8] is used to extract visual features on CIFAR-10 10 dataset. As shown in Table 2, we observe that when the noise gap is 0.1 and 0.2, CE and Peer with NOISE+ outperforms the pure CE and Peer, respectively. When the noise gap is 0.3, balancing cannot compensate for the performance drop due increased noise.
IM Experiments on Clothingl M. Clothing M is a large-scale dataset collected from online shopping websites, comprising of million training images of clothes with realistic noisy and 10,000 test data

---

Table 4: Constrained learning results with group-dependent label noise. We only report the results for e, = 0.2 and C₁2 = 0.4. LR: naive logistic regression without noise correction. GPR:
group-weighted peer loss [30]. Peer: peer loss [21]. We highlight numbers of low fairness violation.
LESS NOISE
MORE NOISE
Dataset
Metrics
LR
GPL
LR
Peer
Adult
accuracy
72.73
71.2
71.88
73.02
fairness
6.48
2.95
3.16
1.67
Compas
accuracy
62.60
66.03
66.22
64.15
fairness
2.87
7.55
6.07
3.63
accuracy
86.97
87.47
88.19
87.93
Fairface
fairness 5.87
4.70
1.38
0.25
with clean labels. For a fair comparison, we adopted the same setting as described in [27] and trained a ResNet-50 [13] classifier. Table 3 compares our methods with some other baselines, including loss correction [27] and Co-Teaching [11]. Without a careful tuning of training parameters, vanilla CE with Nosse-reached 70.37% test accuracy. In comparison, standard CE achieves 68. 94., loss correction achieves 69.84% and Co- Teaching is 70.15%.
6.2 Constrained learning
Setup. We conduct experiments with equal odds constraints [12] on Adult, Compas, and Fairface datasets. We add heterogeneous noise, i.e., uneven noise rates for different group memberships but symmetric for different classes, into training labels but keep test labels clean. We stress-test the performance of the naïve logistic regression (I.R) and group-weighted peer loss (GPL.) [30] with and without our increasing-to bobbaancinnn program. We note that GPL degrades to peer loss when noise rates are homogeneous across protected groups [30], and directly apply peer loss after balancing. For fair comparison, we use logistic regression as the base classifier for all the methods. We use the reductions approach [1] to solve the constrained optimization.
Results. We report the accuracy and fairness violation for ea = 0.2 and e = 0.4 in Table 4 and defer more noise settings Appendix ??. We make the following observations: (1) All the methods retain a comparable accuracy. We conjecture that this benefits from the symmetric noise rates between positive and negative classes. (2) After balancing, both naïve LR and Peer significantly mitigate the fairness violations on Adult and Fairface datasets across groups. (3) Peer loss generally has lower fairness violation than naïve LR. This implics its capability of recovering unbiased classifiers. (4) We find out that the performance of LR with is less profitable on Compas dataset because of uninformative features (only 10-dimensional features), and more robust on Fairface dataset with rich image information.
result of equalized noise rates
7 Concluding Remarks
This paper elaborates the possibility of improving the training model accuracy and fairness guarantees by increasing a certain class of instances' noise rates. Our idea is based on several observations that increasing- to---ttbbbaeeee label noise rates can yield an easier learning problem that robust to mis-specified noise rate parameters. The above robustness helps improve generalization power as well as fairness guarantees. To deploy our idea, we propose a detection algorithm to identify the class of instances with higher label noise rate, without using any ground truth label information.
Our noise rate balancing solution primarily data pre-processing procedure and is compatible with most existing learning with noisy label solutions. Our experiment result using peer loss is one such example. Empirically we observe that many other solutions may also enjoy the benefits of balancing the noise rates. We believe this data pre-processing technique has the potential to find applications in other Icarning tasks when balancing label noise or bias is desired.
We did not repeat testing surrogate loss as it is reported in [30] that GPL appears to be better and more robust solution. Also we focus correcting fairness constraints in this section, both SL and GPL would use the constraint corrections.
10

---

Acknowledgments
This work is partially supported by the National Science Foundation (NSF) under grant IIS-200795] and the NSF FAI program in collaboration with Amazon under grant IIS-2040800. The authors would like to thank Jiaheng Wei and Zhaowei Zhu for their prompt discussion and help on the experimental setup for the Clothing M dataset.
References
[1] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudik, John Langford, and Hanna Wallach. A reductions approach to fair classification. In Jennifer Dy and Andreas Krause, editors, Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 60-69. PMLR, 10-15 Jul 2018.
[2] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine Bias, 2016.
[3] Pranjal Awasthi, Alex Beutel, Matthaeus Kleindessner, Jamie H. Morgenstern, and Xuezhi Wang. Evaluating fairness of machine learning models under uncertain and incomplete information. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency. 2021.
[4] Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with instance-dependentt label noise: A sample sieve approach. In International Conference on Learning Representattonss, 2021.
[5] Jiacheng Cheng, Tongliang Liu, Kotagiri Ramamohanarao, and Dacheng Tao. Learning with bounded instance and label-dependent label noise. In Proceedings of the 37th International Conference on Machine Leurning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Reseurch, pages 1789-1799, PMLR, 2020.
[6] A. Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data, 2:153-163, 2017.
[7] Christos Dimitrakakis, Yang Liu, David C Parkes, and Goran Radanovic. Bayesian fairness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 509-516, 2019.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlshy. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
[9] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[10] Riccardo Fogliato, Alexandra Chouldechova, and Max G'Sell. Fairness evaluation in presence of biased noisy labels. In Silvia Chiappa and Roberto Calandra, editors, The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Ital]], volume 108 of Proceedings of Machine Learning Research, pages 2325-2333 PMLR, 2020.
[11] Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pages 8527-8537, 2018.
[12] Moritz. Hardt, Eric Price, and Nathan Srebro. Equality of opportunity in supervised learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 3323-3331 Red Hook, NY, USA, 2016. Curran Associates Inc.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[14] Simon Jenni and Paolo Favaro. Deep bilevel learning. In Proceedings of the European Conference Computer Vision (ECCV), pages 618-633, 2018.

---

[15] Kimmo Karkkainen and Jungseock Joo. Fairface: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1548-1558, 2021.
[16] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[17] Alex Lamy, Ziyuan Zhong, Aditya K Menon, and Nakul Verma. Noise-tolerant fair classification. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[18] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):2278-2324, 1998.
IEEE Transactions on pattern analysis and machine intelligence, 38(3):447 4614441,441- 2016. [19] Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting.
[20] Yang Liu. Understanding instance-level label noise: Disparate impacts and treatments. In International Conference on Machine Learning, pages 6725-6735. PMLR, 2021.
[21] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. ICML, 2020.
[22] Nan Lu, Gang Niu, Aditya K Menon, and Masashi Sugiyama. On the minimal supervision for training any binary classifier from only unlabeled data. arXiv preprint arXiv:/808. /0585, 2018.
[23] Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.02819, 2020.
[24] A. Mchrotra and L. E. Celis. Mitigating bias in set selection with noisy protected attributes. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021.
[25] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar. and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pages 1 1204, 2013.
[26] Curtis G Northcutt, Lu Jiang, and Isaac I. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 2021.
[27] Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1944-1952, 2017.
[28) Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. Classification with asymmetric label noise: Consistency and maximal denoising. COLT, pages 489-511, 2013.
[29] Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization framework for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5552-5560, 2018
[30] Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 2021 ACM Conference on Fairness, Accountability; and Transparency, FAceT '21, page 526-536, New York, NY, USA, 2021. Association for Computing Machinery.
[31] Serena Wang, Wenshuo Guo, Harikrishna Narasimhan, Andrew Cotter, Maya R. Gupta, and Michael I. Jordan. Robust optimization for fairness with noisy protected groups. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee MieaanTienna Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
[32] Jiaheng Wei and Yang Liu. When optimizing f-divergence robust with label noise. In International Conference Learning Representations, 2021.
12

---

[33] Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual Reducing estimation error for transition matrix label-noise learning. arXiv preprint arXiv:2006.07805, 2020.
[34] Kun Yi and Jianxin Wu. Probabilistic end-to-end noise correction for learning with noisy labels. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[35] Zhaowei Zhu, Tongliang Liu, and Yang Liu. A second-order approach to learning with instance- dependent label noise. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 101 13-1022,,, 2021.
[36] Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative to anchor points when learning with noisy labels. Proceedings of the 38th International Conference on Machine Learning, 2021.
13


Reviews:
[{"review_id": "p9F7PbkGAlq", "reviewer": "Reviewer_A8eu", "summary": "This paper explores ways to improve the generalization performance and fairness guarantees, by increasing a certain class of instances' label noise to balance the rates between different classes. Theorem suggests that when the gap between noise rates is small and when we don't have high confidence in estimating the noise rates, it may be beneficial to increase noise rates. The paper further proposes a method to insert label noise without knowing the ground truth training labels. Experiments show that the rate balancing procedure can be beneficial to increase test performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The theorems shown in the paper are interesting, and provides new insights.\n- The paper is well organized, starting from theoretical discussions and then verifying them through experiments. \n\nCons:\n- Although the theoretical contributions are interesting, some practical limitations remain: the noise gap needs to be small in order for the noise equalization to be beneficial in the equation in line 152~153 and also shown in experiments in Table 2. Is it possible to perform similar large gap experiments with the binary classification setup? Currently, the largest gap seems to be 0.2 in Table 1. It would make the paper stronger if we can see consistent results across different datasets. I was also curious if it is still beneficial under a setting with high noise rates but low noise gap.\n\nOther minor comments:\n- It would also be interesting to see experiments without using a pre-trained model for Table 2. Training from scratch with noisy labels may have more significant effects on the trained model.\n- The gamma parameter was fixed in the experiments, but how sensitive are the results w.r.t. gamma?\n- line 126: denonminator --> denominator\n- line 178: FRP --> FPR?\n- Reference [27] and [28] are the same\n\n=======================================\nAfter author response: Thank you for the additional experiments and for answering my questions. Since I do not have any further major concerns, I would like to raise my score to 7. Additional minor comment: typo on agreeing\" on line 215.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses that the method is limited for class-conditional noise. It does not directly discuss societal impacts, but fairness guarantees is discussed throughout the paper.", "review_text": "", "extra_scores": {"ethical_concerns": "No comments.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "2iUdhvxUKq", "reviewer": "Reviewer_K19n", "summary": "In this work, the authors aim to show that adding label noise in some cases could be beneficial to the test performance and fairness of models. Firstly, they find that increasing-to-balancing can result in an easier learning problem and improves fairness guarantees against label bias. Then they propose a method by inserting label noise to the group of labels with lower noise rates. To estimate the noise rates of various groups, the authors propose a detection method by checking the agreements of noisy labels among local neighbors. Experiments are conducted on three tiny datasets and CIFAR-10 dataset across different levels of noise rates to verify the effectiveness of the proposed algorithm.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:  \n\nAs far as I’m aware, this work is the first to consider model fairness in the setting of label noise. In this work, the authors show that increasing-to-balancing can improve the performance under the setting of asymmetric noisy labels, which is somewhat counter-intuitive from the view of noisy rate.\n\nThis paper is well-written and easy-to-follow. The authors also describe the limitation of this work in the aspect of noisy types.\n\n\nCons:\nIn the setting of asymmetric noisy labels, it seems that the gap of noisy rate among classes can be treated as a class-imbalance problem. In the view of class imbalance, the observation would be not surprising because increasing-to-balancing could alleviate the problem of class imbalance. Then the trade-off becomes reducing the noise rate or the class imbalance rate, which reduces the contribution of this work.\n\nThe proposed detection method is similar to the KNN method in [1] because the agreement strategy among local neighbors can be seen as a variant of KNN. \n\nThe three datasets used in experiments are very small. Although CIFAR-10 is also used, they only train with the image features extracted by a pre-trained vision transformer. Therefore, the experiments are not enough to show the conclusion from observations and the proposed method can extend to some large datasets trained from scratch, like MNIST, CIFAR-10/100 and Clothing1M (imbalanced).\n\nThe improvements from the proposed method are weak, especially on the CIFAR10 dataset. Although improving test accuracy is not the only purpose of this work, the weak improvements are still not convincing enough.\n\n\nOther comments:\n\nWhy don’t try reducing label noise of the groups of labels with higher noise rate by sample selection?  Then we don’t need to consider the tradeoff because we reduce both the noise rate and imbalance rate.\n\n[1] Bahri, Dara, Heinrich Jiang, and Maya Gupta. \"Deep k-nn for noisy labels.\" International Conference on Machine Learning. PMLR, 2020.\n\n\n***\n\nMy concerns have been addressed by the response, thus I decide to improve my score to 6. Although the improvements are still somewhat weak in some cases, this work provides new insights about asymmetric noisy labels for the community, which are more important.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations in noise types. It would be better if the authors can add more large datasets to show the applicability of the proposed method.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2DLaenqRETw", "reviewer": "Reviewer_ayxB", "summary": "This paper considers classification in the case of noisy training data, where one subset of points suffers from higher label noise then the complementary subset. This paper claims that by increasing the label noise rate on the less noisy subset, it is possible to obtain more accurate models since the learning problem is easier. \n\nAdditionally, in the context of fairness, increasing the noise rate of the advantaged group improves fairness guarantees in the sense that when we are fair on the noisy data, we are also fair on the clean data. \n\nLastly, this paper contributes a method to detect which subset suffers from higher noise rates (and how much more is suffers) without access to the ground truth.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Main Points\n\n1) Observation 2 states that the learning problem becomes easier. In what way? In fact, later on, we establish that we actually have an increase in learning difficulty while overall we reduce the upper bound on the excess risk. \n\n2) An important point in Theorem 1 seems to be that the noise rates are not known and we use estimated noise rates to correct the loss. This adds some misspecification term in our bound, which is at the core of the tradeoff. This could be stated more upfront in the beginning since the main problem seems to be how to obtain accurate estimates of the noise rates. However, when are the estimated error rates important later on in the experiments? Is this loss correction (2) always applied?  The peer loss seems to specifically remove the limitation of needing the rates. Therefore, the benefit elaborated in Section 3 seems irrelevant here (but is still true for fairness). \n\n3) Furthermore, I dont see why a lower upper bound actually means that a learning problem is 'easier'. Since we are looking only at upper bounds it is well possible that the excess risk of the balanced noise case is still higher than the excess risk in Theorem 1. I would see this observation more as a motivation to experiment with balanced noise which is indeed emperically advantageous as seen in the experiments. \n\n4) Overall, I would suggest to communicate the meaning of Section 3 differently. It seems to depend heavily on the loss corretion approach and estimated noise rates, which later on become irrelevant. I think this theoretical observation is nice to motivate the contributed algorithm. After all, Noise+ still improves the baseline Peer loss, even though the peer loss does not need estimates of the noise rates. \n\n5) Proof of Lemma 1: Why does it hold that P(h(X) \\neq Y^hat | Y^hat \\neq Y) =  P(h(X) = Y| Y^hat \\neq Y) = P(h(X) = Y)? The last step is not trivial, even though we know that the label noise is independent of X.\n\n6) Regarding condition 2 in Theorem 3. How strong is it and what does it imply? We are looking at the optimal classifier on the clean D and the optimal classifier learned with newly generated noise labels. Those two are now flipped and evaluated on the original noisy data. Just intuitively, I would think that h tilde would have a lower loss on the noisy data than h diamond. And if you flip those, you are in the case of condition 2). Is that accurate?\n\n7) It is unclear to me what the competing methods are. What is the difference between MisSL and EstSL? Earlier in the paper, when estimated versions of the noise rates were used, they were called 'model mis-specification error'.\n\nMinor Points: \n\n9) Line 143: The expected value is written with tildes. Is that correct? I think, here we are talking about the noise balanced case notated with hats. \n\n10) Notation is sometimes confusing. h with tilde is defined with y hats and  and h hat is defined with y tildes. If you would rename the loss l^hat, you wouldnt need a hat in h, but could put the loss name in the subscript. \n\n11) Lines 245 to 259 are very condensed and confusing. Is the notation about C_l and C_r established at that point? \n\n12) What does Lemma 5 tell me? Simply that I need to at least estimate the noise rates right? And with Theorem 6, is this lemma even relevant?\n\n13) For more clarity, it would be good to shortly define what a calibrated loss function is. \n\n14) The baseline methods, the noise procedure, and the peer loss need more description. This would be a helpful addition to the supplement. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Regarding limitations, I have addressed this in points 2, 3, and 4, but I would be glad about clarifications from the authors. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10 hours"}, "extra_text": {}}, {"review_id": "24iwDJJgQfZ", "reviewer": "Reviewer_JhxR", "summary": "This submission studies the paradigm of injecting label-noise from the perspective of fairness and robustness. The motivation stems from the fact that balancing the label proportions yields an easier learning problem and improves fairness guarantees. The submission derives a bound between the corrected loss with label noise and the population risk, which through the use of standard concentration inequalities, involves the Rademacher complexity. In order to fix one, they propose a method that detects the right choice of labels that suffer from noise, which then is turned into an algorithm for robust and fair label correction.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main contributions of the paper consist of two major components: (1) elucidating the trade-off and an in-depth analysis of the error rates in classification along with a generalization bound to explicitly outline this and (2) a detection method that allows you to determine the right group to inject noise for further processing.\n\nThe main strength of the submission is indeed (1), which presents an interesting analysis that explains this trade-off and serves an extremely useful purpose to both the practitioner and theoretician working in fairness and robustness. The contribution (2) is more preliminary at this point however will serve useful for future developments that incorporate the results of this paper. \n\nThe weaknesses I can gather include the tightness of the bound presented in (1). Since it involves the Rademacher complexity, it leads me to believe this may be inaccessible to conventional learning settings where we use deep neural networks. I am not entirely familiar with this literature and am not sure whether this as a starting point is substantial enough.\n\nThe submission is very clear with its contribution and makes it easy for the reader to understand and follow the story. The notation is mostly easy to follow and uses standard conventions throughout - small things like Rademacher complexity is not formally defined however this is no big deal as it is not the focus of this result. \n\nBased on my knowledge, it seems the related work is discussed appropriately and the quality of correctness is sufficient throughout the paper as I was not able to find any mistake in the derivation of the results. I believe the topics of dealing at the intersection of robustness, fairness and label-noise is extremely relevant to the NeurIPS community.\n\nOverall, I think this paper meets the bar of inclusion for NeurIPS due to the importance of the problem, elucidation of this trade-off between noise rates and relatively simple method of group-detection for downstream tasks. The results in this work would greatly benefit the machine learning community.\n\nSome questions/comments for the authors:\n\n(1) Why is sgn used in Line 123 in Equation (2)? It seems to me that in the binary classification case it is redundant since Y is either +1 or -1 or perhaps I'm not appreciating some generality here?\n\n(2) It would be useful if you could explain some intuition behind the corrected loss since a non-expert finds it difficult to grasp the weighting parameters.\n\n(2) A minor weakness I pointed out is the Rademacher complexity involvement in the bound - I was wondering if you could have other complexity measures appearing here perhaps if you change the analysis. For example, in the discrete setting, would it be possible to have the VC dimension appearing or even a PAC-Bayesian analysis? I tried to replicate the arguments however it was not so clear to me if the Rademacher complexity can be avoided.\n\n----------------------------------------------------------------------------------------------------------------------------------------\nI have read the other reviews and rebuttals. The authors have addressed my questions and concerned and so I will be increasing my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes they have", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers were unanimously positive about the paper's contribution to a topical problem, namely, learning with label noise and its implications on model fairness. The initial reviews raised some technical questions that were satisfactorily addressed by the authors. The authors are further encouraged to incorporate the other suggestions provided by reviewers for the updated version of the work.
Decision: Accept (Poster)