Heteroskedastic and Imbalanced Deep Learning with Adaptive Regularization

1 INTRODUCTION . In real-world machine learning applications , even well-curated training datasets have various types of heterogeneity . Two main types of heterogeneity are : ( 1 ) data imbalance : the input or label distribution often has a long-tailed density , and ( 2 ) heteroskedasticity : the labels given inputs have varying levels of uncertainties across subsets of data stemming from various sources such as the intrinsic ambiguity of the data or annotation errors . Many deep learning algorithms have been proposed for imbalanced datasets ( e.g. , see ( Wang et al. , 2017 ; Cao et al. , 2019 ; Cui et al. , 2019 ; Liu et al. , 2019 ) and the reference therein ) . However , heteroskedasticity , a classical notion studied extensively in the statistical community ( Pintore et al. , 2006 ; Wang et al. , 2013 ; Tibshirani et al. , 2014 ) , has so far been under-explored in deep learning . This paper focuses on addressing heteroskedasticity and its interaction with data imbalance in deep learning . Heteroskedasticity is often studied in regression analysis and refers to the property that the distribution of the error varies across inputs . In this work , we mostly focus on classification , though the developed technique also applies to regression . Here , heteroskedasticity reflects how the uncertainty in the conditional distribution y | x , or the entropy of y | x , varies as a function of x . Real-world datasets are often heteroskedastic . For example , Li et al . ( 2017 ) shows that the WebVision dataset has a varying number of ambiguous or true noisy examples across classes.2 Conversely , we consider a dataset to be homoscedastic if every example is mislabeled with a fixed probably , as assumed by many prior theoretical and empirical works on label corruption ( Ghosh et al. , 2017 ; Han et al. , 2018 ; Jiang et al. , 2018 ; Mirzasoleiman et al. , 2020 ) . We note that varying uncertainty in y | x can come from at least two sources : the intrinsic semantic ambiguity of the input , and the ( data-dependent ) mislabeling introduced by the annotation process . Our approach can handle both types of noisy examples in a unified way , but for the sake of comparisons with past methods , we call them “ ambiguous examples ” and “ mislabeled examples ” respectively , and refer to both of them as “ noisy examples ” . 1Code available at https : //github.com/kaidic/HAR . 2See Figure 4 of ( Li et al. , 2017 ) , the number of votes for each example indicates the level of uncertainty of that example . Overparameterized deep learning models tend to overfit more to the noisy examples ( Arpit et al. , 2017 ; Frénay & Verleysen , 2013 ; Zhang et al. , 2016 ) . To address this issue , a common approach is to detect noisy examples by selecting those with large training losses , and then remove them from the ( re- ) training process . However , an input ’ s training loss can also be big because it is rare or ambiguous ( Hacohen & Weinshall , 2019 ; Wang et al. , 2019 ) , as shown in Figure 1 . Noise-cleaning methods could fail to distinguish mislabeled from rare/ambiguous examples ( see Section 3.1 for empirical proofs ) . Though dropping the former is desirable , dropping the latter loses important information . Another popular approach is reweighting methods that reduce the contribution of noisy examples in optimization . However , failing to distinguish between mislabeled and rare/ambiguous examples makes the decision of the weights tricky – mislabeled examples require small weights , whereas rare / ambiguous examples benefit from larger weights ( Cao et al. , 2019 ; Shu et al. , 2019 ) . We propose a regularization method that deals with noisy and rare examples in a unified way . We observe that mislabeled , ambiguous , and rare examples all benefit from stronger regularization ( Hu et al. , 2020 ; Cao et al. , 2019 ) . We apply a Lipschitz regularizer ( Wei & Ma , 2019a ; b ) with varying regularization strength depending on the particular data point . Through theoretical analysis in the one-dimensional setting , we derive the optimal regularization strength for each training example . The optimal strength is larger for rarer and noisier examples . Our proposed algorithm , heteroskedastic adaptive regularization ( HAR ) , first estimates the noise level and density of each example , and then optimizes a Lipschitz-regularized objective with input-dependent regularization with strength provided by the theoretical formula . In summary , our main contributions are : ( i ) we propose to learn heteroskedastic imbalanced datasets under a unified framework , and theoretically study the optimal regularization strength on onedimensional data . ( ii ) we propose an algorithm , heteroskedastic adaptive regularization ( HAR ) , which applies stronger regularization to data points with high uncertainty and low density . ( iii ) we experimentally show that HAR achieves significant improvements over other noise-robust deep learning methods on simulated vision and language datasets with controllable degrees of data noise and data imbalance , as well as a real-world heteroskedastic and imbalanced dataset , WebVision . 2 ADAPTIVE REGULARIZATION FOR HETEROSKEDASTIC DATASETS . 2.1 BACKGROUNDS . We first introduce general nonparametric tools that we use in our analysis , and review the dependency of optimal regularization strength on the sample size and noise level . Over-parameterized neural networks as nonparametric methods . We use nonparametric method as a surrogate for neural networks because they have been shown to be closely related . Recent work ( Savarese et al. , 2019 ) shows that the minimum norm two-layer ReLU network that fits Ground Truth Weak Unif-reg Strong Unif-reg Adapt-reg ( HAR ) Figure 3 : A one-dimensional example with a three-layer neural network in heteroskedastic and imbalanced regression setting . The curve in blue is the underlying ground truth and the dots are observations with heteroskedastic noise . This example shows that uniformly weak regularization overfits on noisy and rare data ( on the right half ) , whereas uniformly strong regularization causes underfitting on the frequent and oscillating data ( on the left half ) . The adaptive regularization does not underfit the oscillating data but still denoise the noisy data . We note that standard nonparametric methods such as cubic spline do not work here because they also use uniform regularization . the training data is in fact a linear spline interpolation . Parhi & Nowak ( 2019 ) extend this result to a broader family of neural networks with a broader family of activations . Given a training dataset { ( xi , yi ) } ni=1 , nonparametric method with penalty works as follows . Let F : R→ R be a twice-differentiable model family . We aim to fit the data with smoothness penalty minf 1 n n∑ i=1 ` ( f ( xi ) , yi ) + λ ∫ ( f ′ ( x ) ) 2dx ( 1 ) Lipschitz regularization for neural networks . Lipschitz regularization has been shown to be effective for deep neural networks as well . Wei & Ma ( 2019a ) proves a generalization bound of neural networks dependent on the Lipschitzness of each layer with respect to all intermediate layers on the training data , and show that , empirically , regularizing the Lipschitzness improve the generalization . Sokolić et al . ( 2017 ) shows similar results in data-limited settings . In Section 2.3 , we extend the Lipschitz regularization technique to heteroskedastic setting . Regularization strength as a function of noise level and sample size . Finally , we briefly review existing theoretical insights on the optimal choice of regularization strength . Generally , the optimal regularization strength for a given model family increases with the label noise level and decreases in the sample size . As a simple example , consider linear ridge regression minθ 1n ∑n i=1 ( x > i θ − yi ) 2 + λ‖θ‖2 , where xi , θ ∈ Rd and yi ∈ R. We assume yi = x > i θ∗+ ξ for some ground truth parameter θ∗ , and ξ ∼ N ( 0 , σ2 ) . Then the optimal regularization strength λopt = dσ2/n‖θ∗‖22 . Results of similar nature can also be found in nonparametric statistics ( Wang et al. , 2013 ; Tibshirani et al. , 2014 ) . 2.2 HETEROSKEDASTIC NONPARAMETRIC CLASSIFICATION ON ONE-DIMENSIONAL DATA . We consider a one-dimensional binary classification problem whereX = [ 0 , 1 ] ⊂ R andY = { −1 , 1 } . We assume Y given X follows a logistic model with ground-truth function f ? , i.e . Pr [ Y = y|X = x ] = 1 1 + exp ( −yf ? ( x ) ) . ( 2 ) The training objective is cross-entropy loss plus Lipschitz regularization , i.e . f̂ = argminf L̂ ( f ) , 1 n n∑ i=1 ` ( f ( xi ) , yi ) + λ ∫ 1 0 ρ ( x ) ( f ′ ( x ) ) 2dx , ( 3 ) where ` ( a , y ) = − log ( 1 + exp ( −ya ) ) , and ρ ( x ) is a smoothing parameter as a function of the noise level and density of x . Let I ( x ) be the fisher information matrix conditioned on the input , i.e . I ( x ) , E [ ∇2a ` ( a , Y ) |a=f ? ( X ) |X = x ] . When ( X , Y ) follows the logistic model in equation 2 , I ( x ) = 1 ( 1 + exp ( f ? ( x ) ) ( 1 + exp ( −f ? ( x ) ) = Var ( Y |X = x ) . Therefore , I ( x ) captures the aleatoric uncertainty of x . For example , when Y is deterministic conditioned on X = x , we have I ( x ) = 0 , indicating perfect certainty . Define the test metric as the mean-squared-error on the test set { ( xi , yi ) } ni=1 , i.e.,1 MSE ( f̂ ) , E { ( xi , yi ) } ni=1 ∫ 1 0 ( f̂ ( t ) − f ? ( t ) ) 2dt ( 4 ) Our main goal is to derive the optimal choice of ρ ( x ) that minimizes the MSE . We start with an analytical characterization of the test error . LetW 22 = { f ′ is absolute continuous and f ′′ ∈ L2 [ 0 , 1 ] } . We denote the density of X as q ( x ) . The following theorem analytically computes the MSE under the regularization strength ρ ( · ) , building upon ( Wang et al. , 2013 ) for regression problems . The proof of the Theorem is deferred to Appendix A. Theorem 1 . Assume that f ? , q , I ∈W 22 . Let r ( t ) = −1/ ( q ( t ) I ( t ) ) and L0 = ∫∞ −∞ 1 4 exp ( −2|t| ) dt . If we choose λ = C0n−2/5 for some constant C0 > 0 , the asymptotic mean squared error is lim n→∞ MSE ( f̂ ) = Cn ∫ 1 0 λ2r2 ( t ) [ d dt ( ρ ( t ) ( f∗ ) ′ ( t ) ) ] 2 + L0r ( t ) 1/2ρ ( t ) −1/2dt in probability , where Cn is a scalar that only depends on n. Using the analytical formula of the test error above , we want to derive an approximately optimal choice of ρ ( x ) . A precise computation is infeasible , so we restrict ourselves to consider only ρ ( x ) that is constant within groups of examples . We introduce an additional structure – we assume the data can be divided into k groups [ a0 , a1 ) , [ a1 , a2 ) , · · · , [ ak−1 , ak ) . Each group [ aj , aj+1 ) consists of an interval of data with approximately the same aleatoric uncertainty . We approximate ρ ( t ) is constant on each of the group [ ai , ai+1 ) with value ρi . Plugging this piece-wise constant ρ into the asymptotic MSE in Theorem 1 , we obtain lim n→∞ MSE ( f̂ ) = ∑ j [ ρ2j ∫ aj+1 aj r2 ( t ) [ d2 dt2 f ? ( t ) ] 2 dt+ ρ −1/2 j L0 ∫ aj+1 aj r1/2 ( t ) dt ] . Minimizing the above formula over ρ1 , . . . , ρk separately , we derive the optimal weights , ρj = [ L0 ∫ aj+1 aj r ( t ) 1/2dt 4 ∫ aj+1 aj r2 ( t ) [ d2 dt2 f ? ( t ) ] 2 dt ] 2/5 . In practice , we do not know f ? and q ( x ) , so we make the following simplifications . We assume that q ( t ) and I ( t ) are constant on each interval [ aj , aj+1 ] . In other words , we assume that q ( t ) = qj and I ( t ) = Ij for all t ∈ [ aj , aj+1 ] . We further assume that d 2 dt2 f ? ( t ) is close to a constant on the entire space , because estimating the curvature in high dimension is difficult . This simplification yields ρj ∝ [ q −1/2 j I −1/2 j q−2j I −2 j ] 2/5 = q 3/5 j I 3/5 j . We find the simplification works well in practice . Adaptive regularization with importance sampling . It is practically infeasible to implement the integration in equation 3 for high-dimensional data . We use importance sampling to approximate the integral : minimizef L ( f ) , 1 n n∑ i=1 ` ( f ( xi ) , yi ) + λ n∑ i=1 τif ′ ( xi ) 2 ( 5 ) Suppose xi ∈ [ aj , aj+1 ) , we have that τi should satisfy that τiqj = ρj so that the expectation of the regularization term in equation 5 is equal to that in equation 3 . Hence , τi = I 3/5 j q −2/5 j = I ( xi ) 3/5q ( xi ) −2/5 . Adaptive regularization for multi-class classification and regression . In fact , the proof of Theorem 1 is proved for general loss ` ( a , y ) . Therefore , we can directly generalize it to multiclass classification and regression problems . For a regression problem , ` ( a , y ) is the square loss : ` ( y , a ) = 0.5 ( y − a ) 2 , the Fisher information I ( x ) = 1 . Therefore , for a regression problem , we can choose regularization weight τi = q ( xi ) −2/5 . 1Note that we integrate the error without weighting because we are interested in the balanced test performance .

Summary: This paper proposed an adaptive regularization method to handle heteroskedastic and imbalanced datasets, which are closer to real-world large-scale settings. The framework applies a Lipschitz regularizer with varying regularization strength depending on the particular data point. The authors first theoretically study the optimal regularization strength on a one-dimensional binary classification task. By applying some simplification, the result can be extended to high-dimensional multi-class tasks and finally HAR algorithm is proposed. Experiments show that HAR achieves significant improvements over other noise-robust deep learning methods on simulated vision and language datasets with controllable degrees of data noise and data imbalance, as well as a real-world heteroskedastic and imbalanced dataset. The experiments show great improvement. However, since the derivations involve many approximations, the reliability needs to be confirmed by more experiments.