Can Less be More? When Increasing-to-Balancing Label Noise Rates Considered Beneficial

1 Introduction . The presence of training label noise is generally considered harmful . Typically the goal of learning with noisy labels is to improve the training by reducing the amount of noise in the training data [ 4 , 11 ] . This paper discusses the feasibility of the opposite strategy and shows cases when adding label noise , leading to a scenario with less informative labels , will result in more accurate and fair models . We are primarily motivated by three observations : Observation I : Reducing label noise is hard , but increasing it is easy . While the literature has provided us with solutions to perform data cleaning , they often require a highly customized training process . Their lack of theoretical rigor has also posed challenges when performing evaluations . On the other hand , as we will see later , increasing label noise is easy — one can always do so by randomly flipping the current noisy labels to increase it further . Observation II : Increasing a certain class of instances ’ noise rate to balance the noise rates results in an easier learning problem . When label noise is class-dependent ( label class Y = +1 v.s . Y = −1 ) , popular noise-tolerant learning algorithms typically require the knowledge of noise rates [ 25 ] . However , the learner often needs to identify the unknown noise rates by some estimation procedure [ 26 , 27 , 36 ] . We articulate that the mis-specification of noise rates will introduce additional learning errors , especially when the label noise is asymmetric ( see Theorem 1 ) . On the other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hand , although increasing the noise rate of the lower class one to balanced error rates reduces the informativeness of the training labels , it is regarded as an easier case to handle — when error rates are balanced , invoking loss correction procedures becomes unnecessary ( see Lemma 2 & Theorem 3 ) . Observation III : Increasing-to-balancing improves fairness guarantees against label bias . When the label noise rates are group-dependent ( data from young group v.s . senior group ) , it has been reported in the literature that imposing fairness constraints directly on the noisy labels would instead reinforce the unfairness [ 30 ] . Fixing the fairness constraints again requires the knowledge of the label noise rates . We will show that increasing and balancing noise rates allows us to directly impose fairness guarantees on the noisy training data without knowing the noise rates ( Theorem 6 ) . This paper first quantifies the trade-offs introduced by increasing a certain group of instances ’ label noise rate with respect to the loss of informative labels and the gained benefits for doing so . We analytically demonstrate when such an increase proves beneficial in terms of either improved generalization error or fairness guarantees . Then we present a method to leverage our idea of inserting label noise for the task of learning with noisy labels , either without or with a fairness constraint . The primary technical challenge we face is that we would not know which data instances are suffering from higher noise , and again we would not have the ground truth labels to verify any possible hypothesis . In response , we propose a detection method that informs us which group of labels might suffer from higher noise without using ground truth information . The core discovery is a couple of metrics that check the agreements of noisy labels among local neighbors . These two metrics can be easily estimated using noisy labels only and are shown to be sufficient to inform us of the class of labels with a higher noise rate . With this knowledge , we propose an algorithm ( NOISE+ ) to gradually insert noise into the lower noise class of labels . Our contributions summarize as follows : • Our paper prototypes the idea of pre-processing noisy training labels for both the constrained and unconstrained learning problems . We show the possibility of improving the training accuracy and fairness guarantees by increasing a certain class of instances ’ noise rates ( increasing-to-balancing ) . • To enable the deployment of our idea , we propose a detection algorithm to identify the class of instances with a higher label noise rate without using any ground truth label information . • Our solution contributes to the learning with noisy labels literature by adding another tool that is robust to noise rate ( noise transition matrix ) estimation errors . All omitted proofs can be found in the Appendix . The code for reproducing the experimental results is available at https : //github.com/UCSC-REAL/CanLessBeMore . 1.1 Related work . Our work is a contribution to the well-established literature of learning with noisy labels [ 11 , 22 , 25 , 27–29 , 33 ] . The classical solutions within the literature leveraged the knowledge of noise rates to perform loss correction [ 25 , 27 , 28 ] , label correction [ 23 , 27 ] , loss reweighting [ 19 ] , among many other treatments . The importance of knowing the correct noise rate parameters is clearly established . Follow-up works have provided tools for estimating the noise rates without accessing the ground truth label . These include the confident learning approach [ 26 ] , anchor point [ 5 ] , and more recently clustering-based ones [ 36 ] . Nonetheless , the possibility of unintended harm due to mis-specified noise rates remains . More recent works have looked into robust loss functions that would not require the knowledge of noise rate , hoping to improve the resistance to the mis-specifications [ 14 , 21 , 32 , 34 , 35 ] . Our work can be viewed as a contribution to this specific line of literature . Our work also contributes to the understanding of fairness implications when learning with noisy labels and other imperfect information [ 3 , 7 , 10 , 17 , 20 , 24 , 30 , 31 ] . For instance , [ 17 ] considers amending noisy sensitive attributes by appropriately re-scaling the fairness tolerance but is only restricted to class-conditional random noise . [ 30 ] proposes two noise-resistant fair classification approaches by constructing unbiased estimators with group-dependent label noise . While most of the existing works would either require the noise rates to be balanced between groups or the knowledge of the noise rates to perform noise correction [ 30 ] , our work requires neither . 2 Preliminaries . We will consider a binary classification problem with d-dimensional feature space X ∈ Rd and label space Y ∈ { −1 , +1 } . ( X , Y ) generate according to distribution D. Instead of accessing the clean data , we consider the setting where the learners only have access to noisy labels Ỹ . The generation of Ỹ follows the noise transition model : e+ ( x ) : = P ( Ỹ = −1|Y = +1 , X = x ) , e− ( x ) : = P ( Ỹ = +1|Y = −1 , X = x ) . We further assume Ỹ generates conditionally independently for different X . For now , we do not restrict how e+ ( x ) , e− ( x ) depend on x but to enable our analysis , later we will . Denote the noisy distribution for ( X , Ỹ ) as D̃ . In the second constrained learning setting , we assume that for each instance ( X , Y ) , we also observe a sensitive attribute Z ∈ { a , b } that defines two protected groups ( e.g. , senior v.s . young people ) that we will later enforce fairness constraints over . Unconstrained learning problem . In the unconstrained setting , denote the collected training data as D̃ : = { ( xn , ỹn ) } Nn=1 . The goal is to train a classifier h ∈ H that minimizes the empirical risk ED [ 1 ( h ( X ) 6= Y ) ] with only accessing D̃ , where 1 ( · ) is the 0-1 loss function . We will focus on the standard class-dependent noise rate model where e+ ( x ) ≡ e+ , e− ( x ) ≡ e− , ∀x . We assume we have informative labels : e+ , e− < 0.5 . Hereby the noise rates depend only on the true label class [ 25 ] , but not the specific instance x when conditional on the true label . We will operate in the challenging asymmetric error rates setting when e+ 6= e− . Fairness constrained learning problem . In the fairness constrained setting , denote the collected training data as D̃ : = { ( xn , ỹn , zn ) } Nn=1 . The goal is to train a classifier h ∈ H that maximizes the accuracy while satisfying a certain fairness constraint , measured by a fairness metric Fz ( h ) : min h∈H ED [ 1 ( h ( X ) 6= Y ) ] s.t . |Fa ( h ) − Fb ( h ) | ≤ δ , ( 1 ) wherein δ > 0 is a tolerance parameter for fairness violations . Exemplary Fa ( h ) includes the Equal Opportunity measure Fa ( h ) : = P ( h ( X ) = +1|Y = +1 , Z = a ) [ 12 ] , the demographic parity Fa ( h ) : = P ( h ( X ) = +1|Z = a ) [ 6 ] , among others . We assume the error rates are group dependent [ 30 ] : e+ ( x ) = e− ( x ) = ez , z ∈ { a , b } . Again we assume ea , eb < 0.5 . Within each group , we can further assume that the noise rates are classdependent , but then we will simply apply our solution within each group ( between +1/− 1 classes ) and across the group . We remove this additional layer of complication to stay concise . Notations . We define the following quantities that we will repeatedly use . We will denote by RD ( h ) : = ED [ 1 ( h ( X ) 6= Y ) ] the true generalization error of a classifier h incurred on the clean distribution D. For a classification-calibrated loss function , we will denote its risk as R ` , D ( h ) : = ED [ ` ( h ( X ) , Y ) ] . Denote the empirical ` -risk of a classifier h on dataset D as R̂ ` , D ( h ) : = 1 |D| ∑ ( x , y ) ∈D ` ( h ( x ) , y ) . Our solution is generically applicable to multi-class settings , which we will explain later and demonstrate with experiments . For a clear exposition of our core idea , we focus on the binary case . 3 Equalizing error rates can improve model accuracy . In this section , we focus on the unconstrained learning setting . We will show that while increasing one label class ’ noise rate raises the generalization errors due to less informative labels , at the same time , it helps remove the training ’ s dependency on the noise rates ( e+ , e− ) , and therefore improves robustness to possible mis-specification of them . We will quantify such trade-off . We would mainly demonstrate our idea by comparing to methods that explicitly use the knowledge of noise rates . For one reason , these approaches often have strong theoretical guarantees . The other reason is that the benefit of our approach of not requiring the noise rate would be clearer for this comparison . Particularly , we will use loss correction [ 25 ] as the running example . We are aware of the other recent works that do not require the noise rates . First of all , these approaches focus on improving model accuracy , but not the fairness constraints . Secondly , empirically we observe that these approaches can also benefit from our noise rate balancing procedure . We left the discussions to our experiments , where we will demonstrate using one of such losses , peer loss [ 21 ] . Loss correction . Once knowing e+ , e− , there exist different ways to improve training robustness in the presence of noisy labels . We consider demonstrating our idea using the classical loss correction framework [ 25 , 27 ] : denote by ẽ+ , ẽ− the estimated version of e+ , e− . In practice , this knowledge can be obtained by using existing techniques [ 19 , 26 , 36 ] . For a given calibrated loss ` , the loss correction approach defines the following loss : ˜̀ ( h ( xn ) , ỹn ) : = ( 1− ẽ−sgn ( ỹn ) ) · ` ( h ( xn ) , ỹn ) − ẽsgn ( ỹn ) · ` ( h ( xn ) , −ỹn ) ( 2 ) where in above sgn is the sign function . A favorable property of ˜̀ is its unbiasedness ( expectation over Ỹ |Y = y ) w.r.t . ` ( h ( x ) , y ) when given the correct noise rate parameters e+ , e− . For readers who need more background knowledge of loss correction , we reproduce details in Appendix ? ? . A classifier will be trained following empirical risk minimization ( ERM ) using ˜̀ : h∗˜̀ , D̃ : = arg minh∈H ∑N n=1 ˜̀ ( h ( xn ) , ỹn ) . Compared to the standard loss correction , we omit the 1 − ẽ− − ẽ+ term from the denominator , which will not affect the minimizer of our ERM problem . Denote h∗ ` , D = arg minh∈HR ` , D ( h ) , the optimal classifier for R ` , D ( h ) , and errM : = max { |ẽ+ − e+| , |ẽ− − e−| } the maximal mis-specification error . When ` is L-Lipschitz and bounded by ¯̀ , let L1 : = 4L , L2 : = 2¯̀ > 0 , adapting the proof from [ 25 ] we show : Theorem 1 . For any δ > 0 , we have with probability at least 1− δ : R ` , D ( h ∗ ˜̀ , D̃ ) −R ` , D ( h∗ ` , D ) ≤ 1 1− e+ − e−︸ ︷︷ ︸ label informativeness ·L1 · R ( H ) + errM 1− e+ − e−︸ ︷︷ ︸ mis-specification ·L2 + 2 √ log 1/δ 2N , ( 3 ) where in aboveR ( H ) denotes the Rademacher complexity ofH . Clearly , e+ , e− control the error due to label informativeness : a pair of e+ , e− with higher sum ( e+ + e− ) induce larger generalization error bound . On the other hand , the potential imperfect estimates of them introduce additional error due to the mis-specification .

Summary: This paper explores ways to improve the generalization performance and fairness guarantees, by increasing a certain class of instances' label noise to balance the rates between different classes. Theorem suggests that when the gap between noise rates is small and when we don't have high confidence in estimating the noise rates, it may be beneficial to increase noise rates. The paper further proposes a method to insert label noise without knowing the ground truth training labels. Experiments show that the rate balancing procedure can be beneficial to increase test performance.