Can gradient clipping mitigate label noise?

1 INTRODUCTION : TWO FACES OF GRADIENT CLIPPING . Gradient clipping is a widely-used technique in the training of deep networks ( Mikolov , 2012 ; Pascanu et al. , 2012 ; 2013 ; Mnih et al. , 2015 ; Gehring et al. , 2017 ; Merity et al. , 2018 ; Peters et al. , 2018 ) , and in its simplest form , involves capping the global parameter gradient norm at a specified threshold . Constraining the gradient norm is paramount in training models such as recurrent neural networks , which may otherwise suffer from an “ exploding gradient ” problem ( Bengio et al. , 1994 ) . Clipping has seen broader value in other neural models , and is abstractly understood as being valuable from an optimisation lens : intuitively , by ensuring that the gradient norm is not too large , the dynamics of iterates are more well-behaved ( Hazan et al. , 2015 ; Levy , 2016 ; Zhang et al. , 2019 ) . A distinct line of work has also explored the value of gradient clipping from a privacy lens ( Abadi et al. , 2016 ; Pichapati et al. , 2019 ) : intuitively , it prevents any single instance from overly influencing parameter updates , which is a core requirement in ensuring differential privacy ( Dwork et al. , 2006 ) . In this paper , we propose a new lens with which to study gradient clipping , namely , robustness : intuitively , clipping the gradient prevents over-confident descent steps , which is plausibly beneficial in the presence of noise . Given this intuition , our interest is whether gradient clipping can mitigate the problem of label noise in classification , which has received significant recent interest ( Scott et al. , 2013 ; Natarajan et al. , 2013 ; Menon et al. , 2015 ; Liu & Tao , 2016 ; Patrini et al. , 2017 ; Ghosh et al. , 2017 ; Han et al. , 2018 ; Zhang & Sabuncu , 2018 ; Song et al. , 2019 ; Thulasidasan et al. , 2019 ; Charoenphakdee et al. , 2019 ) . We study this question , and provide three main contributions : ( a ) we show that gradient clipping alone does not endow label noise robustness to even simple models . Specifically , we show that under stochastic gradient descent with linear models , gradient clipping is related to optimising a “ Huberised ” loss ( Lemma 1 , 2 ) . While such Huberised losses preserve classification calibration ( Lemma 3 ) , they are not robust to label noise ( Proposition 4 ) . ( b ) we propose composite loss-based gradient clipping , a variant that does have label noise robustness . Specifically , for losses comprising a base loss composed with a link function , ( e.g. , softmax crossentropy ) , we only clip the contribution of the base loss . The resulting partially Huberised loss preserves classification calibration ( Lemma 6 ) , while being robust to label noise ( Proposition 7 ) . ( c ) we empirically verify that on both synthetic and real-world datasets , partially Huberised versions of standard losses ( e.g. , softmax cross-entropy ) perform well in the presence of label noise ( §5 ) . To illustrate the difference between standard and composite loss-based gradient clipping in ( a ) and ( b ) , consider the pervasive softmax cross-entropy loss , viz . the logistic loss for binary classification . In ( a ) , we relate gradient clipping to loss “ Huberisation ” , which linearises the logistic loss when it exceeds a fixed threshold . In ( b ) , we introduce composite loss-based gradient clipping , which is equivalent to a “ partial loss Huberisation ” that linearises only the cross-entropy loss but leaves the sigmoid link untouched . Figure 1 illustrates the Huberised and partially Huberised logistic loss . 2 BACKGROUND AND RELATED WORK . We provide background on gradient clipping , loss functions for classification , and label noise . Gradient clipping . Consider a supervised learning task over instances X and labels Y , where we have a family of models indexed by θ ∈ Θ , and the quality of a particular model is measured by a loss function ` θ : X× Y→ R. The gradient for a mini-batch { ( xn , yn ) } Nn=1 is g ( θ ) . = 1 N N∑ n=1 ∇ ` θ ( xn , yn ) . ( 1 ) One may instead compute the clipped gradient , which for user-specified threshold τ > 0 is ḡτ ( θ ) . = clipτ ( g ( θ ) ) clipτ ( w ) . = { τ · w||w||2 if ||w||2 ≥ τ w else . ( 2 ) Employing ḡτ ( θ ) for optimisation corresponds to clipped gradient descent . This is closely related to normalised gradient descent ( NGD ) ( Hazan et al. , 2015 ) , wherein one employs g̃ ( θ ) .= g ( θ ) ||g ( θ ) ||2 . Hazan et al . ( 2015 ) showed that NGD can lead to convergence for a wider class of functions than standard gradient descent . Levy ( 2016 ) showed that NGD can escape saddle points for non-convex problems . Zhang et al . ( 2019 ) showed that gradient clipping can lead to accelerated convergence over gradient descent . Gradient clipping has also been explored for privacy ( Abadi et al. , 2016 ; Pichapati et al. , 2019 ) , motivated by it preventing any single instance from dominating parameter updates . Loss functions . In binary classification , one observes samples from a distribution D over X× { ±1 } , and seeks a predictor f : X→ R with low risk R ( f ) .= E ` ( y , f ( x ) ) according to a loss which , in an overload of notation , we denote ` : { ±1 } × R→ R+ . For the zero-one loss ` 01 ( y , f ) = Jy · f < 0K , R ( f ) is known as the misclassification risk . Rather than directly using ` 01 , for computational convenience one often employs a margin loss ` ( y , f ) = φ ( y · f ) , for convex φ : R→ R+ . We say that φ is classification calibrated ( Zhang , 2004 ; Bartlett et al. , 2006 ) if driving the excess risk over the Bayes-optimal predictor for φ to zero also drives the excess risk for ` 01 to zero ; that is , minimising the φ-risk is statistically consistent for classification . A canonical example is the hinge loss φ ( z ) = [ 1 − z ] + . We call φ admissible if it is “ well-behaved ” in the sense of being bounded from below , strictly convex , continuously differentiable , non-increasing , and classification calibrated . We say that φ is proper composite ( Reid & Williamson , 2010 ; Nock & Nielsen , 2009 ) ( or for brevity composite ) if its minimising scores can be interpreted as probabilities . “ Composite ” here refers to such losses comprising a base loss ϕ composed with an invertible link function F : R→ [ 0 , 1 ] . While φ accepts as input real-valued scores ( e.g. , the final layer logits of a neural network ) , these are internally converted to probabilities via F . A canonical example is the logistic loss φ ( z ) = − logF ( z ) with F : z 7→ σ ( z ) for sigmoid σ ( z ) .= ( 1+e−z ) −1 . The multiclass analogue is the softmax cross-entropy loss , wherein the sigmoid becomes a softmax . See Appendix B for a more technical discussion . Learning under label noise . In classification under label noise , one has samples from distribution D̄ where PD̄ ( y | x ) is a noisy version of PD ( y | x ) , e.g. , all labels are corrupted with a fixed constant probability . The goal remains to ensure low risk with respect to the clean D. This problem has a long history in statistics ( Ekholm & Palmgren , 1982 ; Copas , 1988 ) , and has emerged as a topic of recent interest in machine learning . There has been particular interest in the problem of learning under symmetric label noise , wherein all instances have a constant probability of their labels being flipped uniformly to any of the other classes . Scott et al . ( 2013 ) ; Katz-Samuels et al . ( 2019 ) proposed a framework for analysing the more general setting of class-conditional label noise . Takenouchi & Eguchi ( 2004 ) ; Natarajan et al . ( 2013 ) ; Cid-Sueiro et al . ( 2014 ) ; Patrini et al . ( 2017 ) ; Natarajan et al . ( 2018 ) ; van Rooyen & Williamson ( 2018 ) studied loss-correction for obtaining unbiased risk estimators . Recent works have explored ideas ranging from to model ( dis ) agreement ( Malach & Shalev-Shwartz , 2017 ; Han et al. , 2018 ; Song et al. , 2019 ) to abstention ( Thulasidasan et al. , 2019 ) . 3 GRADIENT CLIPPING AND HUBERISED LOSSES . We first show that gradient clipping in general does not endow robustness to label noise , even in simple settings . Specifically , we establish that stochastic gradient clipping with linear models is equivalent to modifying the underlying loss ( Lemma 1 ) . This modified loss is closely related to a Huberised loss ( Lemma 2 ) , which is equivalent to loss-based gradient clipping ( L-gradient clipping ) . Unfortunately , Huberised losses , and thus L-gradient clipping , are not robust to label noise ( Proposition 4 ) . 3.1 FROM GRADIENT TO L-GRADIENT CLIPPING . Consider a binary classification problem , with Y = { ±1 } . Suppose we use a scoring model sθ ( x ) for θ ∈ Θ with margin mθ ( x , y ) .= y · sθ ( x ) , and margin loss ` θ ( x , y ) .= φ ( mθ ( x , y ) ) for admissible φ : R→ R+ . Then , ∇θ ` θ ( x , y ) = ∇θmθ ( x , y ) · φ′ ( mθ ( x , y ) ) . ( 3 ) Now suppose we perform stochastic gradient descent ( i.e. , we use N = 1 in ( 1 ) ) and use a linear scorer sθ ( x ) = θTx . Here , gradient clipping of ( 3 ) is equivalent to modifying the loss as follows . Lemma 1 . Pick any admissible margin loss φ , and τ > 0 . Then , for loss ` θ ( x , y ) . = φ ( mθ ( x , y ) ) with linear scorer , the clipped loss gradient ( 2 ) is equivalent to the gradient of a modified loss ¯̀ : clipτ ( ∇ ` θ ( x , y ) ) = ∇¯̀θ ( x , y ) , with ¯̀θ ( x , y ) . = { − τ‖∇mθ ( x , y ) ‖2 ·mθ ( x , y ) if |φ ′ ( mθ ( x , y ) ) | ≥ τ‖∇mθ ( x , y ) ‖2 φ ( mθ ( x , y ) ) else . ( 4 ) The loss ¯̀ in ( 4 ) is intuitive : if the derivative of the original loss φ exceeds a certain effective clipping threshold , we replace φ with a linear loss function . This effective threshold τ · ( ‖∇mθ ( x , y ) ‖2 ) −1 takes into account the instance-dependent margin gradient , viz . ‖x‖2 for linear sθ ( x ) . We will comment in §4.4 as to the properties of gradient clipping in more general scenarios . For the moment , we note that Lemma 1 is closely related to the “ Huber loss ” ( Huber , 1964 ) , ` huber ( x , y ; θ ) = { τ · |sθ ( x ) − y| if |sθ ( x ) − y| ≥ τ 1 2 · ( sθ ( x ) − y ) 2 else . ( 5 ) This replaces the extremes of the square loss with the absolute loss . Both the absolute and Huber losses are widely employed in robust regression ( Huber , 1981 , Chapter 3 ) , ( Hampel et al. , 1986 , pg . 100 ) . Note that ( 4 ) slightly differs from ( 5 ) as the effective clipping threshold is instance-dependent . One may nonetheless arrive at Huber-style losses via a variant of gradient clipping ; this connection will prove useful for our subsequent analysis . Per ( 3 ) , gradient clipping involves using clipτ ( ∇ ` θ ( x , y ) ) = clipτ ( ∇mθ ( x , y ) · φ′ ( mθ ( x , y ) ) ) , ( 6 ) wherein the clipping considers the margin gradient . Consider now the following loss-based gradient clipping ( L-gradient clipping ) , wherein we clip only the contribution arising from the loss : l-clipτ ( ∇ ` θ ( x , y ) ) . = ∇mθ ( x , y ) · clipτ ( φ′ ( mθ ( x , y ) ) ) . ( 7 ) Compared to ( 6 ) , we effectively treat the margin gradient norm ‖∇mθ ( x , y ) ‖2 as constant across instances , and focus on bounding the loss derivative . The latter may be a significant component in the gradient norm ; e.g. , for linear models , ‖∇mθ ( x , y ) ‖2 = ‖x‖2 is often bounded . Observe further that for linear models with ‖x‖2 ≡ R across instances , ( 7 ) is a rescaled version of the clipped gradient : clipτ ( ∇ ` θ ( x , y ) ) = τ · y · x · φ′ ( mθ ( x , y ) ) τ ∨ ( R · |φ′ ( mθ ( x , y ) ) | ) = ∇mθ ( x , y ) · clipτ/R ( φ′ ( mθ ( x , y ) ) ) . Interestingly , L-gradient clipping equivalently uses the following “ Huberised ” version of the loss . Lemma 2 . Pick any admissible margin loss φ : R → R+ with Fenchel conjugate φ∗ , and τ > 0 . Then , the clipped gradient in ( 7 ) is equivalent to employing a Huberised loss function φ̄τ such that clipτ ( φ ′ ( z ) ) = φ̄′τ ( z ) with φ̄τ ( z ) . = { −τ · z − φ∗ ( −τ ) if φ′ ( z ) ≤ −τ φ ( z ) else . ( 8 ) Evidently , φ̄τ linearises φ once its derivative is sufficiently large , akin to the Huber loss in ( 5 ) . One may verify that for φ ( z ) = ( 1− z ) 2 , one arrives exactly at ( 5 ) . Example 3.1 : For the logistic loss φ ( z ) = log ( 1 + e−z ) , the Huberised loss for τ ∈ ( 0 , 1 ) is φ̄τ ( z ) = { −τ · z − log ( 1− τ ) − τ · σ−1 ( τ ) if z ≤ −σ−1 ( τ ) log ( 1 + e−z ) else . Per Figure 1a , this linearises φ beyond a fixed threshold . See Appendix C for further illustrations . The use of Huberised losses in classification is not new ( Zhang et al. , 2002 ; Zhang & Johnson , 2003 ; Rosset & Zhu , 2007 ) . However , we now provide a novel study of their label noise robustness .

Summary: Gradient clipping has been studied as an optimization technique and also as a tool for privacy preserving, but in this paper, it studies the robustness properties of gradient clipping.  More specifically, the main question of the paper is: Can gradient clipping mitigate label noise?  The paper reveals that the answer is no, but further proposes a simple variant of gradient clipping is robust and has nice property of classification calibration.  Experiments show that the proposed variant works under label noise.