Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

1 INTRODUCTION . In real-world machine learning applications , even well-curated training datasets have various types of heterogeneity . Two main types of heterogeneity are : ( 1 ) data imbalance : the input or label distribution often has a long-tailed density , and ( 2 ) heteroskedasticity : the labels given inputs have varying levels of uncertainties across subsets of data stemming from various sources such as the intrinsic ambiguity of the data or annotation errors . Many deep learning algorithms have been proposed for imbalanced datasets ( e.g. , see ( Wang et al. , 2017 ; Cao et al. , 2019 ; Cui et al. , 2019 ; Liu et al. , 2019 ) and the reference therein ) . However , heteroskedasticity , a classical notion studied extensively in the statistical community ( Pintore et al. , 2006 ; Wang et al. , 2013 ; Tibshirani et al. , 2014 ) , has so far been under-explored in deep learning . This paper focuses on addressing heteroskedasticity and its interaction with data imbalance in deep learning . Heteroskedasticity is often studied in regression analysis and refers to the property that the distribution of the error varies across inputs . In this work , we mostly focus on classification , though the developed technique also applies to regression . Here , heteroskedasticity reflects how the uncertainty in the conditional distribution y | x , or the entropy of y | x , varies as a function of x . Real-world datasets are often heteroskedastic . For example , Li et al . ( 2017 ) shows that the WebVision dataset has a varying number of ambiguous or true noisy examples across classes.2 Conversely , we consider a dataset to be homoscedastic if every example is mislabeled with a fixed probably , as assumed by many prior theoretical and empirical works on label corruption ( Ghosh et al. , 2017 ; Han et al. , 2018 ; Jiang et al. , 2018 ; Mirzasoleiman et al. , 2020 ) . We note that varying uncertainty in y | x can come from at least two sources : the intrinsic semantic ambiguity of the input , and the ( data-dependent ) mislabeling introduced by the annotation process . Our approach can handle both types of noisy examples in a unified way , but for the sake of comparisons with past methods , we call them “ ambiguous examples ” and “ mislabeled examples ” respectively , and refer to both of them as “ noisy examples ” . 1Code available at https : //github.com/kaidic/HAR . 2See Figure 4 of ( Li et al. , 2017 ) , the number of votes for each example indicates the level of uncertainty of that example . Overparameterized deep learning models tend to overfit more to the noisy examples ( Arpit et al. , 2017 ; Frénay & Verleysen , 2013 ; Zhang et al. , 2016 ) . To address this issue , a common approach is to detect noisy examples by selecting those with large training losses , and then remove them from the ( re- ) training process . However , an input ’ s training loss can also be big because it is rare or ambiguous ( Hacohen & Weinshall , 2019 ; Wang et al. , 2019 ) , as shown in Figure 1 . Noise-cleaning methods could fail to distinguish mislabeled from rare/ambiguous examples ( see Section 3.1 for empirical proofs ) . Though dropping the former is desirable , dropping the latter loses important information . Another popular approach is reweighting methods that reduce the contribution of noisy examples in optimization . However , failing to distinguish between mislabeled and rare/ambiguous examples makes the decision of the weights tricky – mislabeled examples require small weights , whereas rare / ambiguous examples benefit from larger weights ( Cao et al. , 2019 ; Shu et al. , 2019 ) . We propose a regularization method that deals with noisy and rare examples in a unified way . We observe that mislabeled , ambiguous , and rare examples all benefit from stronger regularization ( Hu et al. , 2020 ; Cao et al. , 2019 ) . We apply a Lipschitz regularizer ( Wei & Ma , 2019a ; b ) with varying regularization strength depending on the particular data point . Through theoretical analysis in the one-dimensional setting , we derive the optimal regularization strength for each training example . The optimal strength is larger for rarer and noisier examples . Our proposed algorithm , heteroskedastic adaptive regularization ( HAR ) , first estimates the noise level and density of each example , and then optimizes a Lipschitz-regularized objective with input-dependent regularization with strength provided by the theoretical formula . In summary , our main contributions are : ( i ) we propose to learn heteroskedastic imbalanced datasets under a unified framework , and theoretically study the optimal regularization strength on onedimensional data . ( ii ) we propose an algorithm , heteroskedastic adaptive regularization ( HAR ) , which applies stronger regularization to data points with high uncertainty and low density . ( iii ) we experimentally show that HAR achieves significant improvements over other noise-robust deep learning methods on simulated vision and language datasets with controllable degrees of data noise and data imbalance , as well as a real-world heteroskedastic and imbalanced dataset , WebVision . 2 ADAPTIVE REGULARIZATION FOR HETEROSKEDASTIC DATASETS . 2.1 BACKGROUNDS . We first introduce general nonparametric tools that we use in our analysis , and review the dependency of optimal regularization strength on the sample size and noise level . Over-parameterized neural networks as nonparametric methods . We use nonparametric method as a surrogate for neural networks because they have been shown to be closely related . Recent work ( Savarese et al. , 2019 ) shows that the minimum norm two-layer ReLU network that fits Ground Truth Weak Unif-reg Strong Unif-reg Adapt-reg ( HAR ) Figure 3 : A one-dimensional example with a three-layer neural network in heteroskedastic and imbalanced regression setting . The curve in blue is the underlying ground truth and the dots are observations with heteroskedastic noise . This example shows that uniformly weak regularization overfits on noisy and rare data ( on the right half ) , whereas uniformly strong regularization causes underfitting on the frequent and oscillating data ( on the left half ) . The adaptive regularization does not underfit the oscillating data but still denoise the noisy data . We note that standard nonparametric methods such as cubic spline do not work here because they also use uniform regularization . the training data is in fact a linear spline interpolation . Parhi & Nowak ( 2019 ) extend this result to a broader family of neural networks with a broader family of activations . Given a training dataset { ( xi , yi ) } ni=1 , nonparametric method with penalty works as follows . Let F : R→ R be a twice-differentiable model family . We aim to fit the data with smoothness penalty minf 1 n n∑ i=1 ` ( f ( xi ) , yi ) + λ ∫ ( f ′ ( x ) ) 2dx ( 1 ) Lipschitz regularization for neural networks . Lipschitz regularization has been shown to be effective for deep neural networks as well . Wei & Ma ( 2019a ) proves a generalization bound of neural networks dependent on the Lipschitzness of each layer with respect to all intermediate layers on the training data , and show that , empirically , regularizing the Lipschitzness improve the generalization . Sokolić et al . ( 2017 ) shows similar results in data-limited settings . In Section 2.3 , we extend the Lipschitz regularization technique to heteroskedastic setting . Regularization strength as a function of noise level and sample size . Finally , we briefly review existing theoretical insights on the optimal choice of regularization strength . Generally , the optimal regularization strength for a given model family increases with the label noise level and decreases in the sample size . As a simple example , consider linear ridge regression minθ 1n ∑n i=1 ( x > i θ − yi ) 2 + λ‖θ‖2 , where xi , θ ∈ Rd and yi ∈ R. We assume yi = x > i θ∗+ ξ for some ground truth parameter θ∗ , and ξ ∼ N ( 0 , σ2 ) . Then the optimal regularization strength λopt = dσ2/n‖θ∗‖22 . Results of similar nature can also be found in nonparametric statistics ( Wang et al. , 2013 ; Tibshirani et al. , 2014 ) . 2.2 HETEROSKEDASTIC NONPARAMETRIC CLASSIFICATION ON ONE-DIMENSIONAL DATA . We consider a one-dimensional binary classification problem whereX = [ 0 , 1 ] ⊂ R andY = { −1 , 1 } . We assume Y given X follows a logistic model with ground-truth function f ? , i.e . Pr [ Y = y|X = x ] = 1 1 + exp ( −yf ? ( x ) ) . ( 2 ) The training objective is cross-entropy loss plus Lipschitz regularization , i.e . f̂ = argminf L̂ ( f ) , 1 n n∑ i=1 ` ( f ( xi ) , yi ) + λ ∫ 1 0 ρ ( x ) ( f ′ ( x ) ) 2dx , ( 3 ) where ` ( a , y ) = − log ( 1 + exp ( −ya ) ) , and ρ ( x ) is a smoothing parameter as a function of the noise level and density of x . Let I ( x ) be the fisher information matrix conditioned on the input , i.e . I ( x ) , E [ ∇2a ` ( a , Y ) |a=f ? ( X ) |X = x ] . When ( X , Y ) follows the logistic model in equation 2 , I ( x ) = 1 ( 1 + exp ( f ? ( x ) ) ( 1 + exp ( −f ? ( x ) ) = Var ( Y |X = x ) . Therefore , I ( x ) captures the aleatoric uncertainty of x . For example , when Y is deterministic conditioned on X = x , we have I ( x ) = 0 , indicating perfect certainty . Define the test metric as the mean-squared-error on the test set { ( xi , yi ) } ni=1 , i.e.,1 MSE ( f̂ ) , E { ( xi , yi ) } ni=1 ∫ 1 0 ( f̂ ( t ) − f ? ( t ) ) 2dt ( 4 ) Our main goal is to derive the optimal choice of ρ ( x ) that minimizes the MSE . We start with an analytical characterization of the test error . LetW 22 = { f ′ is absolute continuous and f ′′ ∈ L2 [ 0 , 1 ] } . We denote the density of X as q ( x ) . The following theorem analytically computes the MSE under the regularization strength ρ ( · ) , building upon ( Wang et al. , 2013 ) for regression problems . The proof of the Theorem is deferred to Appendix A. Theorem 1 . Assume that f ? , q , I ∈W 22 . Let r ( t ) = −1/ ( q ( t ) I ( t ) ) and L0 = ∫∞ −∞ 1 4 exp ( −2|t| ) dt . If we choose λ = C0n−2/5 for some constant C0 > 0 , the asymptotic mean squared error is lim n→∞ MSE ( f̂ ) = Cn ∫ 1 0 λ2r2 ( t ) [ d dt ( ρ ( t ) ( f∗ ) ′ ( t ) ) ] 2 + L0r ( t ) 1/2ρ ( t ) −1/2dt in probability , where Cn is a scalar that only depends on n. Using the analytical formula of the test error above , we want to derive an approximately optimal choice of ρ ( x ) . A precise computation is infeasible , so we restrict ourselves to consider only ρ ( x ) that is constant within groups of examples . We introduce an additional structure – we assume the data can be divided into k groups [ a0 , a1 ) , [ a1 , a2 ) , · · · , [ ak−1 , ak ) . Each group [ aj , aj+1 ) consists of an interval of data with approximately the same aleatoric uncertainty . We approximate ρ ( t ) is constant on each of the group [ ai , ai+1 ) with value ρi . Plugging this piece-wise constant ρ into the asymptotic MSE in Theorem 1 , we obtain lim n→∞ MSE ( f̂ ) = ∑ j [ ρ2j ∫ aj+1 aj r2 ( t ) [ d2 dt2 f ? ( t ) ] 2 dt+ ρ −1/2 j L0 ∫ aj+1 aj r1/2 ( t ) dt ] . Minimizing the above formula over ρ1 , . . . , ρk separately , we derive the optimal weights , ρj = [ L0 ∫ aj+1 aj r ( t ) 1/2dt 4 ∫ aj+1 aj r2 ( t ) [ d2 dt2 f ? ( t ) ] 2 dt ] 2/5 . In practice , we do not know f ? and q ( x ) , so we make the following simplifications . We assume that q ( t ) and I ( t ) are constant on each interval [ aj , aj+1 ] . In other words , we assume that q ( t ) = qj and I ( t ) = Ij for all t ∈ [ aj , aj+1 ] . We further assume that d 2 dt2 f ? ( t ) is close to a constant on the entire space , because estimating the curvature in high dimension is difficult . This simplification yields ρj ∝ [ q −1/2 j I −1/2 j q−2j I −2 j ] 2/5 = q 3/5 j I 3/5 j . We find the simplification works well in practice . Adaptive regularization with importance sampling . It is practically infeasible to implement the integration in equation 3 for high-dimensional data . We use importance sampling to approximate the integral : minimizef L ( f ) , 1 n n∑ i=1 ` ( f ( xi ) , yi ) + λ n∑ i=1 τif ′ ( xi ) 2 ( 5 ) Suppose xi ∈ [ aj , aj+1 ) , we have that τi should satisfy that τiqj = ρj so that the expectation of the regularization term in equation 5 is equal to that in equation 3 . Hence , τi = I 3/5 j q −2/5 j = I ( xi ) 3/5q ( xi ) −2/5 . Adaptive regularization for multi-class classification and regression . In fact , the proof of Theorem 1 is proved for general loss ` ( a , y ) . Therefore , we can directly generalize it to multiclass classification and regression problems . For a regression problem , ` ( a , y ) is the square loss : ` ( y , a ) = 0.5 ( y − a ) 2 , the Fisher information I ( x ) = 1 . Therefore , for a regression problem , we can choose regularization weight τi = q ( xi ) −2/5 . 1Note that we integrate the error without weighting because we are interested in the balanced test performance .

Summary: The authors propose a novel regularization approach aimed at addressing issues of class imbalance and heteroskedasticity. This adaptive approach uses a Lipschitz regularizer with varying strength in different parts of the input space, regularizing harder in cases of rare and noisy examples. The authors derive the optional regularization strength in the one-dimensional setting, to set ground for the proposed approach and its application in higher-dimensional settings. The approach is evaluated on multiple image datasets, and a textual dataset - and compared to a number of baselines, including those involving noise-cleaning, reweigthing-based methods, meta learning, robust loss functions, as well as tuned uniform regularization. The improvements seem quite strong, and clearly demonstrate the utility of the proposed approach. The paper is well structured, clearly written - and was a pleasure to read.