Can gradient clipping mitigate label noise?

1 INTRODUCTION : TWO FACES OF GRADIENT CLIPPING . Gradient clipping is a widely-used technique in the training of deep networks ( Mikolov , 2012 ; Pascanu et al. , 2012 ; 2013 ; Mnih et al. , 2015 ; Gehring et al. , 2017 ; Merity et al. , 2018 ; Peters et al. , 2018 ) , and in its simplest form , involves capping the global parameter gradient norm at a specified threshold . Constraining the gradient norm is paramount in training models such as recurrent neural networks , which may otherwise suffer from an “ exploding gradient ” problem ( Bengio et al. , 1994 ) . Clipping has seen broader value in other neural models , and is abstractly understood as being valuable from an optimisation lens : intuitively , by ensuring that the gradient norm is not too large , the dynamics of iterates are more well-behaved ( Hazan et al. , 2015 ; Levy , 2016 ; Zhang et al. , 2019 ) . A distinct line of work has also explored the value of gradient clipping from a privacy lens ( Abadi et al. , 2016 ; Pichapati et al. , 2019 ) : intuitively , it prevents any single instance from overly influencing parameter updates , which is a core requirement in ensuring differential privacy ( Dwork et al. , 2006 ) . In this paper , we propose a new lens with which to study gradient clipping , namely , robustness : intuitively , clipping the gradient prevents over-confident descent steps , which is plausibly beneficial in the presence of noise . Given this intuition , our interest is whether gradient clipping can mitigate the problem of label noise in classification , which has received significant recent interest ( Scott et al. , 2013 ; Natarajan et al. , 2013 ; Menon et al. , 2015 ; Liu & Tao , 2016 ; Patrini et al. , 2017 ; Ghosh et al. , 2017 ; Han et al. , 2018 ; Zhang & Sabuncu , 2018 ; Song et al. , 2019 ; Thulasidasan et al. , 2019 ; Charoenphakdee et al. , 2019 ) . We study this question , and provide three main contributions : ( a ) we show that gradient clipping alone does not endow label noise robustness to even simple models . Specifically , we show that under stochastic gradient descent with linear models , gradient clipping is related to optimising a “ Huberised ” loss ( Lemma 1 , 2 ) . While such Huberised losses preserve classification calibration ( Lemma 3 ) , they are not robust to label noise ( Proposition 4 ) . ( b ) we propose composite loss-based gradient clipping , a variant that does have label noise robustness . Specifically , for losses comprising a base loss composed with a link function , ( e.g. , softmax crossentropy ) , we only clip the contribution of the base loss . The resulting partially Huberised loss preserves classification calibration ( Lemma 6 ) , while being robust to label noise ( Proposition 7 ) . ( c ) we empirically verify that on both synthetic and real-world datasets , partially Huberised versions of standard losses ( e.g. , softmax cross-entropy ) perform well in the presence of label noise ( §5 ) . To illustrate the difference between standard and composite loss-based gradient clipping in ( a ) and ( b ) , consider the pervasive softmax cross-entropy loss , viz . the logistic loss for binary classification . In ( a ) , we relate gradient clipping to loss “ Huberisation ” , which linearises the logistic loss when it exceeds a fixed threshold . In ( b ) , we introduce composite loss-based gradient clipping , which is equivalent to a “ partial loss Huberisation ” that linearises only the cross-entropy loss but leaves the sigmoid link untouched . Figure 1 illustrates the Huberised and partially Huberised logistic loss . 2 BACKGROUND AND RELATED WORK . We provide background on gradient clipping , loss functions for classification , and label noise . Gradient clipping . Consider a supervised learning task over instances X and labels Y , where we have a family of models indexed by θ ∈ Θ , and the quality of a particular model is measured by a loss function ` θ : X× Y→ R. The gradient for a mini-batch { ( xn , yn ) } Nn=1 is g ( θ ) . = 1 N N∑ n=1 ∇ ` θ ( xn , yn ) . ( 1 ) One may instead compute the clipped gradient , which for user-specified threshold τ > 0 is ḡτ ( θ ) . = clipτ ( g ( θ ) ) clipτ ( w ) . = { τ · w||w||2 if ||w||2 ≥ τ w else . ( 2 ) Employing ḡτ ( θ ) for optimisation corresponds to clipped gradient descent . This is closely related to normalised gradient descent ( NGD ) ( Hazan et al. , 2015 ) , wherein one employs g̃ ( θ ) .= g ( θ ) ||g ( θ ) ||2 . Hazan et al . ( 2015 ) showed that NGD can lead to convergence for a wider class of functions than standard gradient descent . Levy ( 2016 ) showed that NGD can escape saddle points for non-convex problems . Zhang et al . ( 2019 ) showed that gradient clipping can lead to accelerated convergence over gradient descent . Gradient clipping has also been explored for privacy ( Abadi et al. , 2016 ; Pichapati et al. , 2019 ) , motivated by it preventing any single instance from dominating parameter updates . Loss functions . In binary classification , one observes samples from a distribution D over X× { ±1 } , and seeks a predictor f : X→ R with low risk R ( f ) .= E ` ( y , f ( x ) ) according to a loss which , in an overload of notation , we denote ` : { ±1 } × R→ R+ . For the zero-one loss ` 01 ( y , f ) = Jy · f < 0K , R ( f ) is known as the misclassification risk . Rather than directly using ` 01 , for computational convenience one often employs a margin loss ` ( y , f ) = φ ( y · f ) , for convex φ : R→ R+ . We say that φ is classification calibrated ( Zhang , 2004 ; Bartlett et al. , 2006 ) if driving the excess risk over the Bayes-optimal predictor for φ to zero also drives the excess risk for ` 01 to zero ; that is , minimising the φ-risk is statistically consistent for classification . A canonical example is the hinge loss φ ( z ) = [ 1 − z ] + . We call φ admissible if it is “ well-behaved ” in the sense of being bounded from below , strictly convex , continuously differentiable , non-increasing , and classification calibrated . We say that φ is proper composite ( Reid & Williamson , 2010 ; Nock & Nielsen , 2009 ) ( or for brevity composite ) if its minimising scores can be interpreted as probabilities . “ Composite ” here refers to such losses comprising a base loss ϕ composed with an invertible link function F : R→ [ 0 , 1 ] . While φ accepts as input real-valued scores ( e.g. , the final layer logits of a neural network ) , these are internally converted to probabilities via F . A canonical example is the logistic loss φ ( z ) = − logF ( z ) with F : z 7→ σ ( z ) for sigmoid σ ( z ) .= ( 1+e−z ) −1 . The multiclass analogue is the softmax cross-entropy loss , wherein the sigmoid becomes a softmax . See Appendix B for a more technical discussion . Learning under label noise . In classification under label noise , one has samples from distribution D̄ where PD̄ ( y | x ) is a noisy version of PD ( y | x ) , e.g. , all labels are corrupted with a fixed constant probability . The goal remains to ensure low risk with respect to the clean D. This problem has a long history in statistics ( Ekholm & Palmgren , 1982 ; Copas , 1988 ) , and has emerged as a topic of recent interest in machine learning . There has been particular interest in the problem of learning under symmetric label noise , wherein all instances have a constant probability of their labels being flipped uniformly to any of the other classes . Scott et al . ( 2013 ) ; Katz-Samuels et al . ( 2019 ) proposed a framework for analysing the more general setting of class-conditional label noise . Takenouchi & Eguchi ( 2004 ) ; Natarajan et al . ( 2013 ) ; Cid-Sueiro et al . ( 2014 ) ; Patrini et al . ( 2017 ) ; Natarajan et al . ( 2018 ) ; van Rooyen & Williamson ( 2018 ) studied loss-correction for obtaining unbiased risk estimators . Recent works have explored ideas ranging from to model ( dis ) agreement ( Malach & Shalev-Shwartz , 2017 ; Han et al. , 2018 ; Song et al. , 2019 ) to abstention ( Thulasidasan et al. , 2019 ) . 3 GRADIENT CLIPPING AND HUBERISED LOSSES . We first show that gradient clipping in general does not endow robustness to label noise , even in simple settings . Specifically , we establish that stochastic gradient clipping with linear models is equivalent to modifying the underlying loss ( Lemma 1 ) . This modified loss is closely related to a Huberised loss ( Lemma 2 ) , which is equivalent to loss-based gradient clipping ( L-gradient clipping ) . Unfortunately , Huberised losses , and thus L-gradient clipping , are not robust to label noise ( Proposition 4 ) . 3.1 FROM GRADIENT TO L-GRADIENT CLIPPING . Consider a binary classification problem , with Y = { ±1 } . Suppose we use a scoring model sθ ( x ) for θ ∈ Θ with margin mθ ( x , y ) .= y · sθ ( x ) , and margin loss ` θ ( x , y ) .= φ ( mθ ( x , y ) ) for admissible φ : R→ R+ . Then , ∇θ ` θ ( x , y ) = ∇θmθ ( x , y ) · φ′ ( mθ ( x , y ) ) . ( 3 ) Now suppose we perform stochastic gradient descent ( i.e. , we use N = 1 in ( 1 ) ) and use a linear scorer sθ ( x ) = θTx . Here , gradient clipping of ( 3 ) is equivalent to modifying the loss as follows . Lemma 1 . Pick any admissible margin loss φ , and τ > 0 . Then , for loss ` θ ( x , y ) . = φ ( mθ ( x , y ) ) with linear scorer , the clipped loss gradient ( 2 ) is equivalent to the gradient of a modified loss ¯̀ : clipτ ( ∇ ` θ ( x , y ) ) = ∇¯̀θ ( x , y ) , with ¯̀θ ( x , y ) . = { − τ‖∇mθ ( x , y ) ‖2 ·mθ ( x , y ) if |φ ′ ( mθ ( x , y ) ) | ≥ τ‖∇mθ ( x , y ) ‖2 φ ( mθ ( x , y ) ) else . ( 4 ) The loss ¯̀ in ( 4 ) is intuitive : if the derivative of the original loss φ exceeds a certain effective clipping threshold , we replace φ with a linear loss function . This effective threshold τ · ( ‖∇mθ ( x , y ) ‖2 ) −1 takes into account the instance-dependent margin gradient , viz . ‖x‖2 for linear sθ ( x ) . We will comment in §4.4 as to the properties of gradient clipping in more general scenarios . For the moment , we note that Lemma 1 is closely related to the “ Huber loss ” ( Huber , 1964 ) , ` huber ( x , y ; θ ) = { τ · |sθ ( x ) − y| if |sθ ( x ) − y| ≥ τ 1 2 · ( sθ ( x ) − y ) 2 else . ( 5 ) This replaces the extremes of the square loss with the absolute loss . Both the absolute and Huber losses are widely employed in robust regression ( Huber , 1981 , Chapter 3 ) , ( Hampel et al. , 1986 , pg . 100 ) . Note that ( 4 ) slightly differs from ( 5 ) as the effective clipping threshold is instance-dependent . One may nonetheless arrive at Huber-style losses via a variant of gradient clipping ; this connection will prove useful for our subsequent analysis . Per ( 3 ) , gradient clipping involves using clipτ ( ∇ ` θ ( x , y ) ) = clipτ ( ∇mθ ( x , y ) · φ′ ( mθ ( x , y ) ) ) , ( 6 ) wherein the clipping considers the margin gradient . Consider now the following loss-based gradient clipping ( L-gradient clipping ) , wherein we clip only the contribution arising from the loss : l-clipτ ( ∇ ` θ ( x , y ) ) . = ∇mθ ( x , y ) · clipτ ( φ′ ( mθ ( x , y ) ) ) . ( 7 ) Compared to ( 6 ) , we effectively treat the margin gradient norm ‖∇mθ ( x , y ) ‖2 as constant across instances , and focus on bounding the loss derivative . The latter may be a significant component in the gradient norm ; e.g. , for linear models , ‖∇mθ ( x , y ) ‖2 = ‖x‖2 is often bounded . Observe further that for linear models with ‖x‖2 ≡ R across instances , ( 7 ) is a rescaled version of the clipped gradient : clipτ ( ∇ ` θ ( x , y ) ) = τ · y · x · φ′ ( mθ ( x , y ) ) τ ∨ ( R · |φ′ ( mθ ( x , y ) ) | ) = ∇mθ ( x , y ) · clipτ/R ( φ′ ( mθ ( x , y ) ) ) . Interestingly , L-gradient clipping equivalently uses the following “ Huberised ” version of the loss . Lemma 2 . Pick any admissible margin loss φ : R → R+ with Fenchel conjugate φ∗ , and τ > 0 . Then , the clipped gradient in ( 7 ) is equivalent to employing a Huberised loss function φ̄τ such that clipτ ( φ ′ ( z ) ) = φ̄′τ ( z ) with φ̄τ ( z ) . = { −τ · z − φ∗ ( −τ ) if φ′ ( z ) ≤ −τ φ ( z ) else . ( 8 ) Evidently , φ̄τ linearises φ once its derivative is sufficiently large , akin to the Huber loss in ( 5 ) . One may verify that for φ ( z ) = ( 1− z ) 2 , one arrives exactly at ( 5 ) . Example 3.1 : For the logistic loss φ ( z ) = log ( 1 + e−z ) , the Huberised loss for τ ∈ ( 0 , 1 ) is φ̄τ ( z ) = { −τ · z − log ( 1− τ ) − τ · σ−1 ( τ ) if z ≤ −σ−1 ( τ ) log ( 1 + e−z ) else . Per Figure 1a , this linearises φ beyond a fixed threshold . See Appendix C for further illustrations . The use of Huberised losses in classification is not new ( Zhang et al. , 2002 ; Zhang & Johnson , 2003 ; Rosset & Zhu , 2007 ) . However , we now provide a novel study of their label noise robustness .

Summary: This paper studies the relationship between gradient clipping in stochastic gradient descent and robustness to label noise. Theoretical results show that gradient clipping in general is not robust to symmetric label noise. The paper then proposes a variant of gradient clipping (cl-clipping) that induces label noise robustness. Experiments support these claims on synthetic datasets and typical classification benchmarks.