Can Less be More? When Increasing-to-Balancing Label Noise Rates Considered Beneficial

1 Introduction . The presence of training label noise is generally considered harmful . Typically the goal of learning with noisy labels is to improve the training by reducing the amount of noise in the training data [ 4 , 11 ] . This paper discusses the feasibility of the opposite strategy and shows cases when adding label noise , leading to a scenario with less informative labels , will result in more accurate and fair models . We are primarily motivated by three observations : Observation I : Reducing label noise is hard , but increasing it is easy . While the literature has provided us with solutions to perform data cleaning , they often require a highly customized training process . Their lack of theoretical rigor has also posed challenges when performing evaluations . On the other hand , as we will see later , increasing label noise is easy — one can always do so by randomly flipping the current noisy labels to increase it further . Observation II : Increasing a certain class of instances ’ noise rate to balance the noise rates results in an easier learning problem . When label noise is class-dependent ( label class Y = +1 v.s . Y = −1 ) , popular noise-tolerant learning algorithms typically require the knowledge of noise rates [ 25 ] . However , the learner often needs to identify the unknown noise rates by some estimation procedure [ 26 , 27 , 36 ] . We articulate that the mis-specification of noise rates will introduce additional learning errors , especially when the label noise is asymmetric ( see Theorem 1 ) . On the other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hand , although increasing the noise rate of the lower class one to balanced error rates reduces the informativeness of the training labels , it is regarded as an easier case to handle — when error rates are balanced , invoking loss correction procedures becomes unnecessary ( see Lemma 2 & Theorem 3 ) . Observation III : Increasing-to-balancing improves fairness guarantees against label bias . When the label noise rates are group-dependent ( data from young group v.s . senior group ) , it has been reported in the literature that imposing fairness constraints directly on the noisy labels would instead reinforce the unfairness [ 30 ] . Fixing the fairness constraints again requires the knowledge of the label noise rates . We will show that increasing and balancing noise rates allows us to directly impose fairness guarantees on the noisy training data without knowing the noise rates ( Theorem 6 ) . This paper first quantifies the trade-offs introduced by increasing a certain group of instances ’ label noise rate with respect to the loss of informative labels and the gained benefits for doing so . We analytically demonstrate when such an increase proves beneficial in terms of either improved generalization error or fairness guarantees . Then we present a method to leverage our idea of inserting label noise for the task of learning with noisy labels , either without or with a fairness constraint . The primary technical challenge we face is that we would not know which data instances are suffering from higher noise , and again we would not have the ground truth labels to verify any possible hypothesis . In response , we propose a detection method that informs us which group of labels might suffer from higher noise without using ground truth information . The core discovery is a couple of metrics that check the agreements of noisy labels among local neighbors . These two metrics can be easily estimated using noisy labels only and are shown to be sufficient to inform us of the class of labels with a higher noise rate . With this knowledge , we propose an algorithm ( NOISE+ ) to gradually insert noise into the lower noise class of labels . Our contributions summarize as follows : • Our paper prototypes the idea of pre-processing noisy training labels for both the constrained and unconstrained learning problems . We show the possibility of improving the training accuracy and fairness guarantees by increasing a certain class of instances ’ noise rates ( increasing-to-balancing ) . • To enable the deployment of our idea , we propose a detection algorithm to identify the class of instances with a higher label noise rate without using any ground truth label information . • Our solution contributes to the learning with noisy labels literature by adding another tool that is robust to noise rate ( noise transition matrix ) estimation errors . All omitted proofs can be found in the Appendix . The code for reproducing the experimental results is available at https : //github.com/UCSC-REAL/CanLessBeMore . 1.1 Related work . Our work is a contribution to the well-established literature of learning with noisy labels [ 11 , 22 , 25 , 27–29 , 33 ] . The classical solutions within the literature leveraged the knowledge of noise rates to perform loss correction [ 25 , 27 , 28 ] , label correction [ 23 , 27 ] , loss reweighting [ 19 ] , among many other treatments . The importance of knowing the correct noise rate parameters is clearly established . Follow-up works have provided tools for estimating the noise rates without accessing the ground truth label . These include the confident learning approach [ 26 ] , anchor point [ 5 ] , and more recently clustering-based ones [ 36 ] . Nonetheless , the possibility of unintended harm due to mis-specified noise rates remains . More recent works have looked into robust loss functions that would not require the knowledge of noise rate , hoping to improve the resistance to the mis-specifications [ 14 , 21 , 32 , 34 , 35 ] . Our work can be viewed as a contribution to this specific line of literature . Our work also contributes to the understanding of fairness implications when learning with noisy labels and other imperfect information [ 3 , 7 , 10 , 17 , 20 , 24 , 30 , 31 ] . For instance , [ 17 ] considers amending noisy sensitive attributes by appropriately re-scaling the fairness tolerance but is only restricted to class-conditional random noise . [ 30 ] proposes two noise-resistant fair classification approaches by constructing unbiased estimators with group-dependent label noise . While most of the existing works would either require the noise rates to be balanced between groups or the knowledge of the noise rates to perform noise correction [ 30 ] , our work requires neither . 2 Preliminaries . We will consider a binary classification problem with d-dimensional feature space X ∈ Rd and label space Y ∈ { −1 , +1 } . ( X , Y ) generate according to distribution D. Instead of accessing the clean data , we consider the setting where the learners only have access to noisy labels Ỹ . The generation of Ỹ follows the noise transition model : e+ ( x ) : = P ( Ỹ = −1|Y = +1 , X = x ) , e− ( x ) : = P ( Ỹ = +1|Y = −1 , X = x ) . We further assume Ỹ generates conditionally independently for different X . For now , we do not restrict how e+ ( x ) , e− ( x ) depend on x but to enable our analysis , later we will . Denote the noisy distribution for ( X , Ỹ ) as D̃ . In the second constrained learning setting , we assume that for each instance ( X , Y ) , we also observe a sensitive attribute Z ∈ { a , b } that defines two protected groups ( e.g. , senior v.s . young people ) that we will later enforce fairness constraints over . Unconstrained learning problem . In the unconstrained setting , denote the collected training data as D̃ : = { ( xn , ỹn ) } Nn=1 . The goal is to train a classifier h ∈ H that minimizes the empirical risk ED [ 1 ( h ( X ) 6= Y ) ] with only accessing D̃ , where 1 ( · ) is the 0-1 loss function . We will focus on the standard class-dependent noise rate model where e+ ( x ) ≡ e+ , e− ( x ) ≡ e− , ∀x . We assume we have informative labels : e+ , e− < 0.5 . Hereby the noise rates depend only on the true label class [ 25 ] , but not the specific instance x when conditional on the true label . We will operate in the challenging asymmetric error rates setting when e+ 6= e− . Fairness constrained learning problem . In the fairness constrained setting , denote the collected training data as D̃ : = { ( xn , ỹn , zn ) } Nn=1 . The goal is to train a classifier h ∈ H that maximizes the accuracy while satisfying a certain fairness constraint , measured by a fairness metric Fz ( h ) : min h∈H ED [ 1 ( h ( X ) 6= Y ) ] s.t . |Fa ( h ) − Fb ( h ) | ≤ δ , ( 1 ) wherein δ > 0 is a tolerance parameter for fairness violations . Exemplary Fa ( h ) includes the Equal Opportunity measure Fa ( h ) : = P ( h ( X ) = +1|Y = +1 , Z = a ) [ 12 ] , the demographic parity Fa ( h ) : = P ( h ( X ) = +1|Z = a ) [ 6 ] , among others . We assume the error rates are group dependent [ 30 ] : e+ ( x ) = e− ( x ) = ez , z ∈ { a , b } . Again we assume ea , eb < 0.5 . Within each group , we can further assume that the noise rates are classdependent , but then we will simply apply our solution within each group ( between +1/− 1 classes ) and across the group . We remove this additional layer of complication to stay concise . Notations . We define the following quantities that we will repeatedly use . We will denote by RD ( h ) : = ED [ 1 ( h ( X ) 6= Y ) ] the true generalization error of a classifier h incurred on the clean distribution D. For a classification-calibrated loss function , we will denote its risk as R ` , D ( h ) : = ED [ ` ( h ( X ) , Y ) ] . Denote the empirical ` -risk of a classifier h on dataset D as R̂ ` , D ( h ) : = 1 |D| ∑ ( x , y ) ∈D ` ( h ( x ) , y ) . Our solution is generically applicable to multi-class settings , which we will explain later and demonstrate with experiments . For a clear exposition of our core idea , we focus on the binary case . 3 Equalizing error rates can improve model accuracy . In this section , we focus on the unconstrained learning setting . We will show that while increasing one label class ’ noise rate raises the generalization errors due to less informative labels , at the same time , it helps remove the training ’ s dependency on the noise rates ( e+ , e− ) , and therefore improves robustness to possible mis-specification of them . We will quantify such trade-off . We would mainly demonstrate our idea by comparing to methods that explicitly use the knowledge of noise rates . For one reason , these approaches often have strong theoretical guarantees . The other reason is that the benefit of our approach of not requiring the noise rate would be clearer for this comparison . Particularly , we will use loss correction [ 25 ] as the running example . We are aware of the other recent works that do not require the noise rates . First of all , these approaches focus on improving model accuracy , but not the fairness constraints . Secondly , empirically we observe that these approaches can also benefit from our noise rate balancing procedure . We left the discussions to our experiments , where we will demonstrate using one of such losses , peer loss [ 21 ] . Loss correction . Once knowing e+ , e− , there exist different ways to improve training robustness in the presence of noisy labels . We consider demonstrating our idea using the classical loss correction framework [ 25 , 27 ] : denote by ẽ+ , ẽ− the estimated version of e+ , e− . In practice , this knowledge can be obtained by using existing techniques [ 19 , 26 , 36 ] . For a given calibrated loss ` , the loss correction approach defines the following loss : ˜̀ ( h ( xn ) , ỹn ) : = ( 1− ẽ−sgn ( ỹn ) ) · ` ( h ( xn ) , ỹn ) − ẽsgn ( ỹn ) · ` ( h ( xn ) , −ỹn ) ( 2 ) where in above sgn is the sign function . A favorable property of ˜̀ is its unbiasedness ( expectation over Ỹ |Y = y ) w.r.t . ` ( h ( x ) , y ) when given the correct noise rate parameters e+ , e− . For readers who need more background knowledge of loss correction , we reproduce details in Appendix ? ? . A classifier will be trained following empirical risk minimization ( ERM ) using ˜̀ : h∗˜̀ , D̃ : = arg minh∈H ∑N n=1 ˜̀ ( h ( xn ) , ỹn ) . Compared to the standard loss correction , we omit the 1 − ẽ− − ẽ+ term from the denominator , which will not affect the minimizer of our ERM problem . Denote h∗ ` , D = arg minh∈HR ` , D ( h ) , the optimal classifier for R ` , D ( h ) , and errM : = max { |ẽ+ − e+| , |ẽ− − e−| } the maximal mis-specification error . When ` is L-Lipschitz and bounded by ¯̀ , let L1 : = 4L , L2 : = 2¯̀ > 0 , adapting the proof from [ 25 ] we show : Theorem 1 . For any δ > 0 , we have with probability at least 1− δ : R ` , D ( h ∗ ˜̀ , D̃ ) −R ` , D ( h∗ ` , D ) ≤ 1 1− e+ − e−︸ ︷︷ ︸ label informativeness ·L1 · R ( H ) + errM 1− e+ − e−︸ ︷︷ ︸ mis-specification ·L2 + 2 √ log 1/δ 2N , ( 3 ) where in aboveR ( H ) denotes the Rademacher complexity ofH . Clearly , e+ , e− control the error due to label informativeness : a pair of e+ , e− with higher sum ( e+ + e− ) induce larger generalization error bound . On the other hand , the potential imperfect estimates of them introduce additional error due to the mis-specification .

Summary: In this work, the authors aim to show that adding label noise in some cases could be beneficial to the test performance and fairness of models. Firstly, they find that increasing-to-balancing can result in an easier learning problem and improves fairness guarantees against label bias. Then they propose a method by inserting label noise to the group of labels with lower noise rates. To estimate the noise rates of various groups, the authors propose a detection method by checking the agreements of noisy labels among local neighbors. Experiments are conducted on three tiny datasets and CIFAR-10 dataset across different levels of noise rates to verify the effectiveness of the proposed algorithm.