Can gradient clipping mitigate label noise?
Authors: Aditya Krishna Menon
Ankit Singh Rawat
Sashank J. Reddi
Sanjiv Kumar
Venue:  (2020)

Abstract: Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, it controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum. This intuition has been made precise in a line of recent works, which show that suitable clipping  can yield significantly faster convergence than vanilla gradient descent. In this paper, we propose a new lens for studying gradient clipping, namely, robustness: informally, one expects clipping to provide robustness to noise, since one does not overly trust any single sample. Surprisingly, we prove that  for the common problem of label noise in classification, standard gradient clipping does not in general provide robustness. On the other hand, we show that  a simple variant of gradient clipping is provably robust, and corresponds to suitably modifying the underlying loss function. This yields a simple, noise-robust alternative to the standard cross-entropy loss which performs well empirically.

Published as a conference paper at ICLR 2020
CAN GRADIENT CLIPPING MITIGATE LABEL NOISE?
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J. Reddi, Sanjiv Kumar
Google Research
New York, NY USA
{adi tyakmenon, anwiitttaaaaeee tsrawat, saggannkssaniikk@ggggllee.
ABSTRACT
Gradient clipping is a widely-used technique in the training of deep networks, and is generally motivated from an optimisation lens: informally, clipping controls the dynamics of iterates, thus enhancing the rate of convergence to a local minimum. line of recent works, which show that
This intuition has been made precise suitable clipping can yield significantly faster convergence than vanilla gradient descent. In this paper, we propose a new lens for studying gradient clipping, namely, robusiness: informally, one expects clipping to mitigate the effects of noise, since one does not overly trust any single sample. Surprisingly, we prove that for the common problem of label noise in classification, standard gradient clipping does not in general provide robustness. On the other hand, we show that a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function. As special case, this yields a simple, noise-robust modification of the standard cross-entropy loss which performs well empirically.
INTRODUCTION: TWO FACES OF GRADIENT CLIPPING
Gradient clipping is widely-used technique in the training of deep networks (Mikolov, 2012; Pascanu et al., 2012; 2013; Mnih et al., 2015; Gehring et al., 2017; Merity et al., 2018; Peters et al., 2018 and in its simplest form, involves capping the global parameter gradient norm a specified threshold. Constraining the gradient norm is paramount in training models such as recurrent neural networks, which may otherwise suffer from an "exploding gradient" problem (Bengio et al., 1994). Clipping has seen broader value in other neural models, and is abstractly understood as being valuable from an optimisation lens: intuitively, by ensuring that the gradient norm is not too large, the dynamics of iterates are more well-behaved (Hazan et al., 2015; Levy, 2016; Zhang et al., 2019). A distinct line of work has also explored the value of gradient clipping from a privacy lens (Abadi et al., 2016; Pichapati et al., 2019): intuitively, it prevents any single instance from overly influencing parameter updates, which a core requirement in ensuring differential privacy (Dwork et al., 2006).
In this paper, we propose a new lens with which to study gradient clipping, namely, robustness: intuitively, clipping the gradient prevents over-confident descent steps, which plausibly beneficial in the presence of noise. Given this intuition, our interest is whether gradient clipping can mitigate the problem of label noise in classification, which has received significant recent interest (Scott et al., 2013; Natarajan et al., 2013; Menon et al., 2015; Liu & Tao, 2016; Patrini et al., 2017; Ghosh et al., 2017; Han et al., 2018; 8; Zhang & Sabuncu, 2018; Song et al., 2019; Thulasidasan et al., 2019; Charoenphakdee et al., 2019). We study this question, and provide three main contributions:
(a) we show that gradient clipping alone does not endow label noise robustness to even simple models. Specifically, show that under stochastic gradient descent with linear models, gradient clipping is related to optimising a "Huberised" loss (Lemma 1, 2). While such Huberised losses preserve classification calibration (Lemma 3), they not robust label noise (Proposition 4).
(b) we propose composite loss-based gradient clipping, a variant that does have label noise robustness. Specifically, for losses comprising base loss composed with a link function, (e.g.. softmax cross- entropy), we only clip the contribution of the base loss. The resulting partially Huberised loss preserves classification calibration (Lemma 6), while being robust to label noise (Proposition 7).
(c) we empirically verify that on both synthetic and real-world datasets, partially Huberised versions of standard losses (e.g., softmax cross-entropy) perform well in the presence of label noise ($55).

---

Published as a conference paper at ICLR 2020
Logistic
Logsloss Huberised log-loes
Huherised
Partially Huberised
3-
2
1
10
10
Figure I: Illustration of the Huberised and partially Huberised logistic loss (left pancl). Recall that the logistic loss comprises the log-loss (or cross-entropy) composed with a sigmoid link. The Huberised loss arises from loss-based gradient clipping, and lincarises the entire logistic loss beyond threshold. The partially Huberised loss arises from composite loss-based gradient clipping, and linearises only the base loss (i.e., log-loss) beyond threshold (right panel). Once combined with the sigmoid link, the overall partially Huberised loss asymptotically saturates. Our analysis in $3, $4 establishes that Huberised losses are not robust to label noise, but partially Huberised losses are (Proposition 4, 7).
To illustrate the difference between standard and composite loss based gradient clipping in (a) and (b), consider the pervasive softmax cross-entropy loss, viz. the logistic loss for binary classification. In (a), we relate gradient clipping to loss "Huberisation", which linearises the logistic loss when exceeds fixed threshold. In (b), introduce composite loss-bused gradient clipping, which is equivalent to a "partial loss Huberisation" that lincarises only the cross-cntropy loss but leaves the sigmoid link untouched. Figure illustrates the Huberised and partially Huberised logistic loss.
BACKGROUND AND RELATED WORK
We provide background on gradient clipping, loss functions for classification, and label noise.
Gradient clipping. Consider a supervised learning task over instances X and labels y, where we have a family of models indexed by f O, and the quality of particular model measured by loss function lo: X x y + R. The gradient for mini-batch {(Ins Vn (x)) | is
1 N
s( =
Vlo (Zi, (o Bn).
One may instead compute the clipped gradient, which for user-specified threshold >> 000 is
clip, T Talla
gr (0) = (c(pp)(((())
else.
Employing 3r (0) for optimisation corresponds to clipped gradient descent. This is closely related normalised gradient descent (NGD) (Hazan et al., 2015), wherein employs 5(0) = To Hazan et al. (2015) showed that NGD can lead to convergence for a wider class of functions than standard gradient descent. Levy (2016) showed that NGD can escape saddle points for non-convex problems. Zhang et al. (2019) showed that gradient clipping can lead accelerated convergence over gradient descent. Gradient clipping has also been explored for privacy (Abadi et al., 2016; Pichapati et al., 2019), motivated by it preventing any single instance from dominating parameter updates.
9())
observes samples from a distribution D over X x {+1}. and seeks a predictor f: + I with low risk R(f) overload of notation, R(f) known as the misclassification risk. Rather than directly using loz, for computational convenience often employs margin loss E f) Φ(y f), for convex Φ: R R,,
Loss functions. In binary classification,
E(y, f(x)) according to loss which, in an denote f: {+1} RR R. For the zero-one loss (v. f) [y O].
We say that is classification calibrated (Zhang, 2004; Bartlett et al., 2006) if driving the excess risk over the Bayes-optimal predictor for o to also drives the excess risk for bn to zero; that is,
2

---

Published as a conference paper at ICLR 2020
minimising the ø-risk is statistically consistent for classification. A canonical example is the hinge loss 6(2) = [1 -1-22 z]+. We call & admissible if it is weell-behaved" in the sense of being bounded from below, strictly convex, continuously differentiable, -on-increasing, and classification calibrated.
We say that Φ is proper composite (Reid & Williamson, 2010; Nock & Niclsen, 2009) (or for brevity composite) if its minimising scores can be interpreted as probabilities. "Composite" here refers to such losses comprising a base loss composed with an invertible link function F: IR + [0. 1]. While o accepts as input real-valued scores (e.g., the final layer logits of neural network), these are internally converted to probabilities via F. A canonical example is the logistic loss ((2) = log F()) with The multiclass analogue is the softmax cross-entropy loss, wherein the sigmoid becomes a softmax. See Appendix B for a more technical discussion.
(1+e==) - Learning under label noise. In classification under label noise, one has samples from distribution D where (D æ) is a noisy version of P z), e.g., all labels are corrupted with fixed constant probability. The goal remains to ensure low risk with respect to the clean D. This problem has a long history in statistics (Ekholm & Palmgren, 1982; Copas. 1988), and has emerged as topic of recent interest in machine learning. There has been particular interest in the problem of learning under symmetric label noise, wherein all instances have constant probability of their labels being flipped uniformly to any of the other classes. Scott et al. (2013); Katz- Samuels et al. (2019) proposed framework for analysing the more general setting of caasscoondiional label noise. Takenouchi & Eguchi (2004); Natarajan et al. (2013); Cid-Sueiro et al. (2014); Patrini et al. (2017); Natarajan et al. (2018); van Rooyen & Williamson (2018) studied loss-correction for obtaining unbiased risk estimators. Recent works have explored ideas ranging from to model (dis)agreement (Malach & Shalev-Shward 2017; Han al., 2018; Song al., 2019) to abstention (Thulasidasan al., 2019).
F: 2 or (2) for sigmoid o (2) = (1+e
GRADIENT CLIPPING AND HUBERISED LOSSES
We first show that gradient clipping general does not endow robustness to label noise, even in simple settings. Specifically, we establish that stochastic gradient clipping with linear models is equivalent to modifying the underlying loss (Lemma 1). This modified loss is closely related to a Huberised loss (Lemma 2), which is equivalent to loss-based gradient clipping (L--gradient clipping). Unfortunately, Huberised losses, and thus L-gradient clipping, are not robust to label noise (Proposition 4).
3.1 FROM GRADIENT TO L-GRADIENT CLIPPING
Consider a binary classification problem, with y = {+]} Suppose we use scoring model sp (z) for 0 E θ with margin mo (zr, y) = y so(z), and margin loss lo(x, v) = o(mo(x; v)) for admissible 6: R > R4. Then,
Volo(r, Vamo(x,,) y) &' (m)o(z)
Now suppose we perform stochastic gradient descent (i.e., we use N = in (1)) and use a linear scorer so (z) = 0t x. Here, gradient clipping of (3) is equivalent to modifying the loss as follows.
Lemma 1. Pick any admissible margin loss o, and T >> O. Then, for loss lo (z, y) = Φ(me(a, y)) with linear scorer, the clipped loss gradient (2) is equivalent to the gradient of a modified loss &
clip, (lo(x,))) = V[o(z,)) v),
Mooilllll mo ma( v) ( [m (m
(4)
with Ca v)
else.
The loss in (4) is intuitive: if the derivative of the original loss o exceeds a certain effective clipping threshold, we replace é with linear loss function. This effective threshold 7 (Il [/Vmm (2, 122) takes into account the mnstance-dependee
margin gradient, viz. ||æ||| for linear (z()..
We will comment in $4.4 as to the properties of gradient clipping in more general scenarios. For the moment, we note that Lemma is closely related to the "Huber loss" (Huber, 1964),
T- (=)ss(x)) - vl (s0(x) - y)
if (so(x)) - vl 2 r
Ruber (z, y:
(5)
clse.
3

---

Published as a conference paper at ICLR 2020
This replaces the extremes of the square loss with the absolute loss. Both the absolute and Huber losses are widely employed in robust regression (Huber, 1981, Chapter 3), (Hampel et al., 1986, pg. 100). Note that (4) slightly differs from (5) the effective clipping threshold is instance-dependen One may nonetheless arrive at Huber-style losses via a variant of gradient clipping: this connection will prove useful for subsequent analysis. Per (3), gradient clipping involves using
clip, (Veo v)) clip, (Vmo(a,,) v) o'
wherein the clipping considers the margin gradient. Consider now the following boss-based gradient clipping (L- gradddinnt clipping), wherein we clip only the contribution arising from the loss:
1-clip, (Vlo(z,)) v)) = Vmg(z,y) clip, (ó' (moo(z)))))
Compared to (6), we effectively treat the margin gradient norm (1, v) l as constant across instances, and focus on bounding the loss derivative. The latter may be a significant component the gradient norm; e.g., for linear models, V (2: 8)ll2 = |alll is often bounded. Observe further that for linear models with ||>]| = R across instances, (7) is a rescaled version of the clipped gradient:
y x 6 (m xz v))
clip, (Ve((r =
Vmo(x, y) clip,/n (6' (mo((. y)))
T TV (R 1 (mmo(x v))l
Interestingly, L-gradient clipping equivalently uses the following "Huberised" version of the loss.
Lemma 2. Pick any admissible margin loss o: IR + R4 with Fenchel conjugate Φ², and > 0. Then, the clipped gradient in (7) is equivalent to employing a Huberised loss function o, such that
clip, (0'(z)) = x2) with (z) = - - - (-T))
(f S
(8)
(0(2)
else.
Evidently, Φ- linearises & once its derivative is sufficiently large, akin the Huber loss in (5). One may verify that for ((z) (1 2)², one arrives exactly at (5).
Example 3.1: For the logistic loss (xz) = log + == the Huberised loss for r E (0, 1) is
- -- - T) [og((1 + +
(r) if -=--0-1 (r)
(x)
else.
Per Figure la, this linearises & beyond fixed threshold. See Appendix C for further illustrations.
The use of Huberised losses in classification not new (Zhang al., 2002; Zhang Johnson, 2003; Rosset & Zhu, 2007). However, we now provide novel study of their label noise robustness.
3.2 CONSISTENCY AND LABEL-NOISE ROBUSTNESS OF HUBERISED LOSSES
Before studying the effect of label noise on Huberised losses, it is apposite to consider whether they suitable for even in the absence of noise. One way to formalise this is to ask whether the losses maintain classification calibration, in the sense of $2. This is a minimal requirement on a loss to be useful for classification (Zhang, 2004; Bartlett al., 2006). One may further ask whether the losses preserve class- probabilities, i.c., are proper composite in the sense of $2. It is desirable to preserve this key trait of losses such as the logistic loss. The following clarifies both points.
Lemma 3. Pick any admissible margin loss á and > Then, the Huberised loss Φ in (8) is classification calibrated. If o is proper composite and T 2 - o (0), then Φr is also proper composite.
L-gradient clipping thus benign for classification, generalising Rosset & Zhu (2007, Section 3.4), which was for the square-hinge loss. Interestingly, for composite ò and small the proof reveals that 6, has non-invertible link function. Intuitively, such Φ, are effectively linear, and linear losses are unsuited for estimating probabilities (van Rooyen et al., 2015; Charoenphakdee al., 2019).
We now turn to our central object of inquiry: does gradient clipping endow robustness to label noise? To study this, consider L-gradient clipping, which as noted above is special case of gradient clipping under linear models with constant [/Vma(a, (x. 122 Since L-gradient clipping is in turn equivalent to using Huberised loss, we may study the robustness properties of this loss.
4

---

Published as a conference paper at ICLR 2020
Surprisingly, when our loss is convex (e.g., softmax cross-entropy), Huberised losses robust to even very simple forms of label noise. Essentially, since these losses are still convex, they can still be affected by errant outlier observations. Formally, using result of Long & Servedio (2010) (see Appendix 1, we arrive at the following.
Proposition 4. Pick any admissible margin loss & and which the optimal linear classifier under Φr is equivalent to random guessing under symmetric noise.
Then, 3 separable distribution for
To situate Proposition 4 in broader context, the Huber loss is susceptible to "high leverage" outliers (Hampel et al., 1986, pg. 313), (Rousseeuw & Leroy, 1987, pg. 13), (Maronna al., 2019, pg- 104), i.e., extremal instances which dominate the optimal solution. Proposition 4 complements these for the case of label noise in classification.
that for regression problems, is well known that
Given that gradient clipping does not endow label noise robustness, how else might we proceed? In regression context, the outlier vulnerability of the Huber loss can be addressed by using a trimmed average of the loss (Rousseeuw & Leroy, 1987; Bhatia et al., 2015; Yang et al., 2018). Such ideas have been successfully explored for label noise problems (Shen & Sanghavi, 2019). We will however demonstrate that simple variant of clipping yields a loss that does possess label noise robustness.
COMPOSITE LOSS- BASSE GRADIENT CLIPPING
We now show that noise robustness can be achieved with CL-gradient clipping, variant wherein for composite losses (e.g., softmax cross-entropy), we perform partial Huberisation of the base loss only.
4.1 FROM HUBERISATION TO PARTIAL HUBERISATION
Consider a composite margin loss (o u) = (1ma(1, y)), where y F for some base loss yo and invertible link F: R + [0, 1]; e.g., the logistic loss has spu) = log u and F(z) = (z). We can interpret po (z. y) = F(ma(x, y)) as a probability estimate; e.g., pe p0(z. 1) a (m, 1)) the probability of æ being positive. Now, rewriting (o y) = p(ps (z, v)), we may express (3) as:
(e v) = Vpe v) & (p6 8)).
L-gradient clipping in (7) was defined as only clipping d' (4 o Fy above, which ensures that the resulting Huberised loss is Lipschitz. Typically, however, is already Lipschitz; e.g., this is the case for the commonly used sigmoid or softmax link. This suggests the following composite loss-based gradient clipping (CL-gradient clipping), wherein we only clip the derivative for the base loss yp:
cl-clip, (V 3)) = Vpo(z, y) clip, (pe B)))
As before, CL-gradient clipping corresponds to optimising new, "partially Huberised" loss. Lemma 5. Pick any admissible, composite margin loss & y F, and >> 0. Then, the clipped gradient in (9) is equivalent employing partially Huberised loss o, G, F, where
(+(u)= -T. (-T) (pp(u)
o --
(10)
else.
Compared to the Huberised loss (7), the partially Huberised loss only linearises the base loss i,, while retaining the link F. Consequently, the composite loss 6, behaves like the link beyond a certain threshold, and will thus be bounded: indeed, (+)2) - F(z) (0( if o' (F(()) < -7.
Example 4.1: For the logistic loss, 6(2) = (FF(2)) with ppu) log u and F(2) a(2).
(z) { - - a(2) log + [oo((1 + +
if S o (+)
(11)
else.
Note that partial Huberisation readily generalises to a multi-class setting. Indeed, suppose we have softmax probability estimates po (x, y) & exp(mo(x, y)). Then, whereas the softmax cross-entropy employs (z(2, y) = llg po (x. y). our partially Huberised softmax cross-entropy for T > 1 is
T -opopoo y) log +1 if po v) s + log (x(p))))))
(o( v)
(12)
else.
5

---

Published as a conference paper at ICLR 2020
4.2 CONSISTENCY AND LABEL NOISE ROBUSTNESS OF PARTIALLY HUBERISED LOSSES
Following $3.2, we establish that CL-gradient clipping is always benign from a classification per- spective, and provided is sufficiently large, from probability estimation perspective as well. As before, we do this by exploiting the equivalence of CL-gradient clipping to a partially Huberised loss.
Lemma 6. Pick any admissible composite margin loss 6 = 4 F and T > O. Then, the loss o, in (10) is classification calibrated. If further T -1/2)/), then Gr, is also proper composite.
We now show that partially Huberised losses have an important advantage over Huberised losses: under symmetric label noise, the optimal solution on the clean distribution cannot be too far away from the optimal solution on the noisy distribution. This implies that label noise (such as that considered in Proposition 4) cannot have an excessively deleterious influence on the loss.
Proposition 7. Pick any proper loss & and O. Let I be the risk minimiser of Φr on the clean distribution. For any non-trivial level of symmetric label noise, let reg, (f*) denote the excess risk of f with respect to G, on the noisy distribution. Then, there exists C >> O such that reg, (f*) S C.
Note that by van Rooyen et al. (2015, Proposition 4), is impossible for the above bound to hold with C O without using linear loss. Nonetheless, by virtue of partially Huberised losses being partially linear, we are able to bound the degradation under label corruption. The saturating behaviour of the partially Huberised loss also implies robustness to outliers in feature space; see Appendix D.
4.3 RELATION TO EXISTING WORK
The partially Huberised log loss in (10). (11) can be related to a family of losses studied in several works (Ding& & Vishwanathan, 2010; Hung et al., 2018; Zhang & Sabuncu, 2018; Sypherd et al., 2019; Amid et al., 2019b;a): for E (0, 1). define the generalised cross-entropy Pa: [0, + R. by (o(1)) (u) = (1 /aa/a. See Appendix C.3 for illustration of y.. There are two similarities between (11) and Pa. First, both proposals interpolate between the log and linear losses: when + 0+, Pa approaches the log loss, and when a = 1. fa equals the linear loss. Second, both proposals modify the base loss, allowing the link F to be chosen independently. In particular, one may use the heavy-tailed F of Ding & Vishwanathan (2010); Amid et al. (2019b) in conjunction with our partially Huberised loss.
One difference between (11) and y., is that the partially Huberised loss exactly linear for a suitable region; consequently, it is guaranteed to be Lipschitz, unlike a This can be understood in terms of the loss gradients: for class-protd probabibobbbby estimate pø(x, y), let (z: y) sppp (2,, y)). For the generalised cross-entropy, Vle(I, y) -po(z, y)a-1 Vpo(zz y), while for the partially Huberised loss, Vlg y) = - (p.(x,)) V VT- Vpa (z, y). Both gradients thus take into account whether a sample is "informative", in the sense of being poorly-predictee
(po ~ 0). Further, to guard against such samples being the result of label noise, both ensure this influence is not overwhelming. but in different ways: the generalised cross-entropy softens the influence, while still allowing unbounded as (1 y) - O. On the other hand, the partially Huberised loss enforces a hard cap of --1 on the influence. This is to be contrast with a truncated loss also considered in Zhang & Sabuncu (2018), which enforces a hard cap on the loss, thus completely discarding preeiccpe instances.
be
As shown in Figure la, the partially Huberised loss in (10) exhibits saturating behaviour at the extremes. This by itself is hardly new in the context of robust statistics. For example, Tukey's "biweight" loss replaces the Huber loss' linear behaviour with hard saturation (Gross & Tukey, 1973), and is member of the broader family of redescending M-estimators (Andrews et al., 1972; Shevlyakov et al., 2008), In a similar spirit, Masnadi-Shiran & Vasconcelos (2009) proposed the bounded Savage loss ( = (a (2v) - 1) while Yang et al. (2010); Menon & Williamson (2018) 8) theoretically studied loss clipping. Robust versions of logistic regression derived by variants of the Huber loss have been explored, e.g., Pregibon (1982, Section 4.1), Copas (1988, Section 3.1), (Feng et al., 2014).
The distinction of the above proposals to work is two-fold. First, we explicitly consider the classification setting, wherein we show consistency and robustness to label noise for the partially Huberised loss. Second, proposal explicitly highlights the connection to gradient clipping.
Finally, while our focus has been on label noise, there are other notions of "robustness". Indeed, several works have proposed means of making gradient descent robust heavy-tailed distributions
6

---

Published as a conference paper at ICLR 2020
Clipping type
Clipped gradient form
Equivalent loss
Reference
Label noise robust?
Gradient
clip, (Vm(0). o' (((m(0)))
Equation 4
Equation 6
(Proposition 4)
L--Gradient
Vm(() clip, (6' (m(())))
Huberised
Equation 7 X (Proposition 4)
CL-Gradient
(p(0)- clip, (p(())
Partially Huberised
Equation 9 v (Proposition 7)
Table Summary of types of gradient clipping considered in this paper. We consider binary classification problems involving a labelled example (æ, y) parametrised scoring function s0(z) with margin m m(0) = (x)), and diffcrentiable composite margin loss 6(2). This loss internalyy converts scores to probabilities p(0) F (()( for link function F(-), which is evaluated with some base loss şo; i.c., = ip o F. Gradient clipping applies the full loss, i..., p(m(0)). L-gradient clipping applies only to composite loss, leaving the score untouched; this is equivalent using Huberised loss. CL-gradient clipping applies only to the base loss, leaving the link untouched; this is equivalent to using partially Huberised loss. Only the latter has robustness guarantee under symmetric label noise.
or outliers (Brownlees al., 2015; Hsu & Sabato, 2016; Lugosi & Mendelson, 2016; Lecué et al., 2018; Prasad et al., 2018: Holland & Ikeda, 2019), typically by passing the gradient through a robust operator (Tukey, 1960; Hampel et al., 1986; Rousseeuw Leroy, 1987; Lerasle & Oliveira, 201 Catoni, 2012; Minsker, 2015). By contrast, we analyse the (simpler) gradient clipping approach, showing that while it inadequate to cope with label noise, a simple modification rectifies this. Since proposal only modifies the underlying loss, it may be combined with any of these methods.
4.4 DISCUSSION AND IMPLICATIONS
Table summarises our results, highlighting the perspective of gradient clipping as equivalently modifying the loss. Before proceeding, we make some qualifying comments. First, our analysis has assumed symmetric label noise. Often, one encounters asymmetric or instance-dependee noise (Menon et al., 2018). While corresponding guarantees for the linear loss may be ported over to the partially Huberised loss, they require stronger distributional assumptions (Ghosh et al., 2015).
Second, Proposition 4 exhibits specific distribution which defeats the Huberised loss under linear models. In practice, distributions may be more benign (Patrini et al., 2016), and models are often nonlinear, meaning that Huberised losses (and thus gradient clipping) extremely to label noise as Proposition 4 suggests. The aim of $4 is however to establish that a simple modification of clipping avoids worst-case degradation, without adding significant complexity.
thus unlikely to succumb as
Third, for minibatch size N > 1, the effect of clipping is not a simple loss modification, since the loss gradients for each sample will be modified by the entire minibatch loss gradient norm [1) Since this minibatch is randomly drawn, one cannot mimic gradient clipping by simple modification of the loss function. However, the results for N suffice to establish that gradient clipping cannot in general endow robustness. One may use our partially Huberised loss in conjunction with minibatch gradient clipping, to potentially obtain both robustness and optimisation benefits.
Finally, partially Huberised losses such as (12) require setting hyperparameter (e.g., by cross- validation), similar to the generalised cross-entropy per $4.3. Intuitively, the optimal 7 trades off the noise-robustness of the linear loss, and the gradient informativeness of the base loss (per the discussion in $4.3). Setting 7 to be large tacitly assumes that one's samples are largely noise-free.
5 EXPERIMENTAL ILLUSTRATION
We now present experiments illustrating that: (a) we may exhibit label noise scenarios that defeat a Huberised but not partially Huberised loss, confirming Propositions 4, 7, and (b) partially Huberised versions of existing losses perform well on real-world datasets subject to label noise.
Synthetic datasets. Our first experiments involve two synthetic datasets, which control for con- founding factors. We begin with a setting from Long & Servedio (2010), comprising a 2D linearly separable distribution. (See Appendix E. for illustration.) We draw N 1, 000 random samples from this distribution, and flip each label with = 45% probability. We train a linear classifier to minimise of several losses, and evaluate the classifier's accuracy on 500 clean test samples.
7

---

Published as a conference paper at ICLR 2020
1.0
1.2-
0.9
1.0
Test accuracy
0.8
0.7
R(00
0.6
0.6
0.4
Logistic
0.5
0.2
Huber
Partial Huber
0.4
Logistic
Huberised
Partial Huberised
0.0
-2.0
-1.5 -1.0 -0.5 0.0
0.5
1.0
1.5
2.0
(a) Symmetric label noise.
(b) Outliers feature space.
Figure 2: Results synthetic data. In (b), the solid (dashed) curve denotes empirical risk without (with) outliers.
Dataset
Loss function
p 0.0
p 0.2
0.4
0.6
CE
98.7 + 0.0
97.7 + 0.2
96.4 + 0.3
91.1 + 0.6
CE clipping
98.6 0.1
97.5 0.1
96.8 0.1
88.9 1
Linear
98.3 0.0
0.1
9.6 0.0
MNIST
GCE
98.8 0 0.1
9.1
9.0
9.1
PHuber-CE: 10
98.5 0.0
98.7
98.5 0.11
97.6
PHuber-GCH 7 10
98.6 0.1
98.5
0.14
97.8
CE
92.4 0.1
72.4 + 0.2
6.84
40.6 0.3
CE + clipping
88.3 0.3
68.7 0.2
56.7 0.6
4.1
Linear
91.3 0.1
88.2
83.8 #.2
0.2
CIFAR-10
GCE
91.7 0.0
8.2
80.8
60.0
PHuber- CE T=
91.6
88.6
8.1
7.02
PHuber-GCH
10
92.0 + 0.1
88.5 + 0.1
80.8 0.1
62.6 0.2
66.6 1.4
49.7 0.3
29.9
11.4 0.2
CE + clipping
28.8 1 0.1
1..4
14.7
104:0.4
Linear
12.1 1.6
6.2
5.7
3.6
CIFAR-100
GCE
70.1 0.1
6.1
5.2
0.5
PHuber-CE r 10
66.2 + 1.5
2.2
4.7
18.5
PHuber-GCE 10
69.8 0.2
64.4 0.4
52.4 144100 0.2
31.5 0.8
Table 2: Test set accuracy where the training labels are corrupted with probability p. We report the mean and standard error over 3 trials. The highlighted cells are the best performing loss at given "PHuber" here refers to partial Huberisation from $4, which is equivalent to variant of gradient clipping.
We compare the logistic loss with its Huberised and partially Huberised versions, using T 1 and 2 respectively. Figure 2a presents the results over 500 independent trials. The logistic loss and its Huberised counterpart suffer significantly under noise, while the partially Huberised loss often achieves perfect near- cear-discriminationn This confirms that in the worst case, L-gradient clipping may succumb to noise, while CL-gradient clipping performs well in the same scenario.
We next consider 1D setting based on Ding (2013, Section 3.2.3), comprising 10, 000 linearly separable "inliers" and 50 "outliers". Assuming the use of linear model parameterised by scalar 0 E IR, plot the empirical risk of the samples with and without the outlier observations varied. Figure 2b shows that the logistic loss and its Huberised variant are strongly affected by the outliers: their optimal solution goes from 6* +oc to 0" 0. However, the partially Huberised loss largely immune to the outliers. Appendix contains additional details on this experiment.
is
Real-world datasets. We now demonstrate that partially Huberised losses perform well with deep neural networks trained on MNIST, CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton, 2009). For MNIST, train LeNet (Lecun et al., 1998) using Adam with batch size N 32, and weight decay
8

---

Published as a conference paper at ICLR 2020
of 10-3. For CIFAR-10 and CIFAR- 100, we train a ResNet-50 (He et al., 2016) using SGD with momentum 0.1, weight decay of 5 x 10 10-3, batch normalisation, and N = 64, 128 respectively.
For each dataset, we corrupt the training labels with symmetric noise at flip probability p {0.0, 0.2, 0.4, 0.63. We compare the test set accuracy of various losses combined with a softmax link. Our baseline is the cross-entropy loss (CE). As representative noise-robust losses, we consider the lin- ear or unhinged loss (van Rooyen et al., 2015; Ghosh et al., 2017), and the generalised cross-entropy (GCE) with a 0.7, following Zhang & Sabuncu (2018). We additionally assess global gradient clipping (with T 0.1) of the CE, which per $3 is akin to a Huberised loss.
We apply our partial Huberisation of (10) to the CE (12) ("PHuber-CE"", and the GCE ("PHuber- GCE"). The latter highlights that partial Huberisation is not tied to the cross-entropy, and is applicable even on top of existing noise-robust losses. Recall that partial Huberisation offers a choice of tuning parameter 7, similar to the parameter in GCE, and noise-rate estimates in poss-correction techniques more generally. For each dataset, we pick T E (2, 10) (equivalently corresponding to probability thresholds 0.5 and 0.1 respectively) so as to maximise accuracy validation set of noisy samples with the maximal noise rate p - 0.6; the chosen value separately for each setting of the noise rate / can be expected to help performance, at the expense of increased computational cost. Recall also that as T - 1, partial Huberisation mimics using the base loss. while - toc, partial Huberisation mimics using the linear loss; our hypothesis is that an intermediate r can attain suitable balance between noise robustness, and gradient informativeness.
was then used for each noise level. Tuning
Table 2 shows that in the noise-free case (p 0.0), all methods perform comparably. However, when injecting noise, accuracy for the CE degrades dramatically. Further, gradient clipping sometimes offers improvements under high noise; however, the performance is far inferior to other losses. which is in keeping with their robustness guarantees. Indeed, the linear loss, which is provably robust to symmetric noise, generally performs well even when p 0.6. However, optimisation under this loss is more challenging. since the gradient does not account for instances' importance (per $4.3). This is particularly reflected on the CIFAR-100 100 dataset, where this loss suffers learn even under no noise. The GCE and partially Huberised losses do not suffer from this issue, even at high noise levels.
Generally, the partially Huberised losses are competitive with or improve upon the counterpart losses they build upon. In particular, the partially Huberised CE performs much better than the CE under high noise, while the partially Huberised GCE slightly bumps up the GCE numbers on CIFAR-100 This indicates that partially Huberised may be useful in combining with generic base losses to cope with noise. We reiterate here that our partially Huberised loss may be used in conjunction with other ideas, c.g., pruning (Zhang & Sabuncu, 2018), consensus (Malach & Shalev-Shwarzz, 2017; Han et al., 2018), or abstention (Thulasidasan et al., 2019). We leave such exploration for future work.
CONCLUSION AND FUTURE WORK
We established that gradient clipping by itself does not suffice to endow even simple models with label noise robustness; however, a simple variant resolves this issue. Experiments confirm that our composite loss-based gradient clipping performs well on datasets corrupted with label noise. One interesting direction for future work is to analyse the behaviour of gradient-clippnng inspired losses for the more general problem of distributionally robust learning Shaafiezadeh-Abadeee Namkoong & Duchi, 2016; 2017; Hu et al., 2018; Sinha et al., 2018; Duchi & Namkoong, 2019).
al., 2015;
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '1, pp. 308-318, New York, NY, USA, 2016. ACM. ISBN 978 1-0--------4139-44
Ehsan Amid, Manfred K. Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on Bregman divergences. CoRR, 0//19900003611 2019a.
Ehsan Amid, Manfred K. Warmuth, and Sriram Srinivasan. Two-temperature logistic regression based on the Tsallis divergence. In The 22nd International Conference on Artificial Intelligence and Stafistics, AISTATS 2019, 16-1 18 April 2019, Naha, Okinawa, Japan, pp. 2388-2396, 2019b.

---

Published as a conference paper at ICLR 2020
D. F. Andrews, P. J. Bickel, F. R. Hampel, P. J. Huber, W. H. Rogers, and J. W. Tukey. Robust Estimates of Location. Princeton University Press, Princeton NJ, 1972.
Peter L Bartlett, Michael Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 2006.
Y. Bengio, P. Simard, and Frasconi. Learning long-term dependencies with gradient descent is difficult. Trans. Neur: Netw,, 1(2):157-166, March 1994. ISSN 1045-9227.
Kush Bhatia, Prateek Jain, and Purushottam Kar. Robust regression via hard thresholding. In Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 1, NIPS 15, pp. 721-729, Cambridge, MA, USA, 2015, MIT Press.
Christian Brownlees, Emilien Joly, and Gábor Lugosi. Empirical risk minimization for heavy-tailed losses. Ann. Statist,, 25662250,-2,,,,, 12 2015.
Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Technical report, UPenn, 2005.
Olivier Catoni. Challenging the empirical mean and empirical variance: deviation study. Ann. Inst. H. Poincaré Probab. Statist., 48(4): 148- 185, 11 2012.
Nontawat Charoenphakdee. Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 961-970, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Jesús Cid-Sueiro, Darío Garefa-García, and Raúl Rantos-Rodríguez. Consistency of losses for learning from weak labels. In Toon Calders, Floriana Esposito, Eyke Hüllermeier, and Rosa Meo (eds.), Machine Learning and Knowledge Discovery in Databases, 197-210, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg.
B. Copas. Binary regression models for contaminated data. Journal of the Royal Statistical Society.
Series B (Methodological), 50(2):225 265, 1988. ISSN 00359246.
Nan Ding. Statistical machine learning in the L-exponential family of distributions. PhD thesis,
Purdue University, 2013.
Nan Ding and S.V.N. Vishwanathan. t-logistic regression. In J. D. Lafferty, C. K. I. Williams,
J. Taylor, R. S. Zemel, and A. Culotta (eds.). Advances in Neural Information Processing Systems 23, pp. 514-522, Curran Associates, Inc., 2010.
John Duchi and Hongseok Namkoong. Variance-based regularization with convex objectives. Journal of Machine Learning Research, 20(68): 55, 2019.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference Theory of Cryptography, TCC*06, pp. 265-284, Berlin, Heidelberg, 2006. Springer-Verlag. ISBN 3-540-32731-2, 97833-44--3-3--- 8.
Anders Ekholm and Juni Palmgren. A model for binary response with misclassifications. In Robert Gilchrist (ed.), GLIM 82: Proceedings of the International Conference on Generalised Linear Models, pp. 128-143, New York, NY, 1982. Springer New York. ISBN 978-1-4612-5771-4
Jiashi Feng, Huan Xu, Shie Mannor, and Shuicheng Yan. Robust logistic regression and classification. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 253-261.
Curran Associates, Inc., 2014.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence sequence learning. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML' 17, pp. 1243-1252. JMLR.org, 2017.
Aritra Ghosh, Naresh Manwani, and P. S. Sastry. Making risk minimization tolerant to label noise. Neurocomputing, 160:93-107, 2015.
10

---

Published as a conference paper at ICLR 2020
Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI' 17, pp. 1919-1925. AAAI Press, 2017.
Tilmann Gneiting and Adrian Raftery. Strietly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359-37, 2007.
I. M. Gross and John W. Tukey. The estimators of the Princeton robustness study. Technical report, series 2 38, Princeton, 1973.
F.R. Hampel, E.M. Ronchetti, P.J. Rousseeuw, and W.A. Stahel. Robust Statistics: the Approach Based on Influence Functions, Wiley New York, 1986.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS'18, 18, pp. 8536-8546, USA, 2018. Curran Associates Inc.
Elad Hazan, Kfir Y. Levy, and Shai Chalev-Shwartz Beyond convexity: Stochastic quasi-convex opti- mization. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 1, NIPS' 15, pp. 1594-1602, Cambridge, MA, USA, 2015. MIT Press.
K. He, X. Zhang, Ren, and Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016.
Matthew Holland and Kazushi Ikeda. Better generalization with less data using robust gradient descent. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume Proceedings of Machine Learning Research, pp.
Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Daniel Hsu and Sivan Sabato. Loss minimization and parameter estimation with heavy tails. Journal of Machine Learning Research, 17(18):1-44 (108):44:220000 2016.
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2029-2033 Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Peter J. Huber. Robust estimation of location parameter. Ann. Math. Statist., 35(1):73-100 100,,,111111 03 1964.
Peter J. Huber. Robust Statistics. John Wiley & Sons, 1981.
Hung Hung, Zhi- Yu Jou, and YumYun Huang. Robust mislabel logistic regression without modeling mislabel probabilities. Biometries, 74(1):145-11) 1544,,,54444 2018.
Julian Katz-Samuels, Gilles Blanchard, and Clayton Scott. Decontamination of mutual contamination models. Journal of Machine Learning Research, 20(41):1-57, 2019.
A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Guillaume Lecué, Matthieu Lerasle, and Timothée Mathieu. Robust classification via MOM mini- mization. arXiv e-prints, art. 1xXiv:1808.03106 Aug 2018
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 11:2278:2223 2324, Nov 1998.
M. Lerasle and R. I. Oliveira. Robust empirical mean Estimators. arXiv e-prints, art Dec 2011.
Kfir Y. Levy. The power of normalization: Faster evasion of saddle points. CoRR, abs//blllli. 161 2016.
11

---

Published as a conference paper at ICLR 2020
Tongliang Liu and Dacheng Tao. Transactions on Pattern Analysis and Machine Intelligence, 38(3):447-455 461, 2016. Classification with noisy labels by importance reweighting. IEEE
Philip M. Long and Rocco A. Servedio. Random classification noise defeats all convex potential boosters. Machine Learning, 78(3):287-304, 2010. ISSN 0885-6125.
Gabor Lugosi and Shahar Mendelson. Risk minimization by median-of-means tournaments, arXiv e-prints, art,
Aug 2016.
Eran Malach and Shai Shalev-Shwartz. Decoupling "when to update" from "how to update". In Proceedings of the s1 International Conference on Neural Information Processing Systems, NIPS' 17. pp. 961-971 USA, 2017, Curran Associates Inc. ISBN 978-1 1-5108-6096-44
R. A. Maronna, D. Martin, and V. J. Yohai. Robust Statistics: Theory and Methods. John Wiley & Sons, New York, 2nd edition, 2019.
Hamed Masnadi-Shirazi and Nuno Vasconcelos. On the design of loss functions for classification: theory, robustness outliers, and SavageBoost. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou (eds.). Advances in Neural Information Processing Systems 21, 1049-1056. Curran Associates, Inc., 2009.
Aditya Krishna Menon and Robert C. Williamson. A loss framework for calibrated anomaly detection. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31, pp. 1487-1497, Curran Associates, Inc.. 2018.
Aditya Krishna Menon, Brendan van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via class-probability estimation. In International Conference on Machine Learning (ICML), pp. 134, 2015.
Aditya Krishna Menon, Brendan van Rooyen, and Nagarajan Natarajan. Learning from binary labels with instance-dependent noise. Machine Learning, (07(8-00):161-111 2018.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations, 2018
Tomáš Mikolov. Statistical Language Models Based on Neural Networks. PhD thesis, Brno University of Technology, 2012.
Stanislav Minsker. Geometric median and robust estimation in banach spaces. Bernoulli, 21(4): 23385-2335, 11 2015.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533 02 2015.
H. Namkoong and J. Duchi. Variance regularization with convex objectives. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. In D. Lee, M. Sugiyama, U. V. Luxburg. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2208-2216. Curran Associates, Inc., 2016.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep D Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems (NIPS), pp. 1196-1204, 2013.
Nagarajan Natarajan, Inderjit S. Dhillon, Pradeep Ravikumar, and Ambuj Tewari. Cost-sensitive learning with noisy labels. Journal of Machine Learning Research, 18(155):1-33, 2018
Richard Nock and Frank Nielsen. On the efficient minimization of classification calibrated surrogates. In D. Koller, D. Schuurmans, Bengio, and L. Bottou (eds.), Advances in Neural Information Processing Systems 21, pp. 1201-1208. Curran Associates, Inc., 2009.
12

---

Published as a conference paper at ICLR 2020
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient problem. CoRR, abs/1211.5063, 2012.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on International Conference on Machine Learning- Volume 28, ICML' 13. JMLR.org. 2013.
Giorgio Patrini, Frank Nielsen, Richard Nock, and Marcello Carioni. Loss factorization, weakly supervised learning and label noise robustness. In International Conference on Machine Learning (ICML), pp. 708-77 717, 2016.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: loss correction approach. In Computer Vision and Pattern Recognition (CVPR), pp. 2233-2241, 2017.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
Venkatadheeraj Pichapati, Ananda Theertha Suresh, Felix X. Yu, Sashank Reddi, and Sanjiv Kumar. AdaCliP: Adaptive Clipping for Private SGD. arXiv e-prints, art. arXiv: 000400,4,,, Aug 2019.
Adarsh Prasad, Arun Sai Suggala, Sivaraman Balakrishnan, and Pradeep Ravikumar. Robust Estimation via Robust Gradient Estimation. arXiv e-prints, art. arXiv:1802.06448 Feb 2018.
Daryl Pregibon. Resistant fits for some commonly used logistic models with medical applications. Biometrics, 38(2):485-498, 1982. ISSN 0006341X, 15410420.
Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research, 1 11:2387- -2222, 2010.
Saharon Rosset and Ji Zhu. Piecewise linear regularized solution paths. Ann. Statist., 35(3):1012- 1030, 07 2007.
Peter J. Roussceuw and Annik M. Leroy. Robust Regression and Outlier Detection. John Wiley and Sons, 1987.
Leonard J. Savage. Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):733801, 1971. ISSN 01621459.
Mark J. Schervish. A general method for comparing probability assessors. Ann. Statist., 17(4): 1856-1879, 12 1989.
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: consistency and maximal denoising. In Conference on Learning Theory (COLT), pp. 489-511, 2013.
Soroosh Saaicezzadeh-bbaael. Peyman Mohajerin Esfahani, and Daniel Kuhn. Distributionally robust logistic regression. In Proceedings of the 28th International Conference Neural Information Processing Systems- - Volume 1, NIPS'15, 15. pp. 1576-1584, Cambridge, MA, USA, 2015. MIT Press.
Yanyao Shen and Sujay Sanghavi. Learning with bad training data via iterative trimmed loss minimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5739-57448 5748, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Georgy Shevlyakov, Stephan Morgenthaler, and Alexander Shurygin. Redescending m-estimators. Journal Of Statistical Planning And Inference, 138:2906-2917, 2008
Aman Sinha, Hongseok Namkoong, and John C. Duchi. Certifying some distributional robustness with principled adversarial training. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 2018, Conference Track Proceedings, 2018.
13

---

Published as a conference paper at ICLR 2020
Hwanjun Song, Minseok Kim, and Jac-Gil Lee. SELFIE: Refurbishing unclean samples for robust deep learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5907-5915, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
T. Sypherd, M. Diaz, L. Sankar, and P. Kairouz. A tunable loss function for binary classification. In 2019 IEEE International Symposium on Information Theory (ISIT), pp. 2479- 2483, July 2019.
Takashi Takenouchi and Shinto Eguchi. Robustifying AdaBoost by adding the naive error rate. Neural Computation, 16(4):767-787, 2004.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chennupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6234-6243, Long Beach, California, USA, 09-15 Jun 2019, PMLR.
W. Tukey. A survey of sampling from contaminated distributions. Contributions to probability and statistics, 2:448 485, 1960.
Brendan van Rooyen and Robert C. Williamson. A theory of learning with corrupted labels. Journal of Machine Learning Research, 18(228):1-50, 2018.
Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. Learning with symmetric label noise: the importance of being unhinged. In Advances in Neural Information Processing Systems (NIPS), pp. 10-18, 2015.
Robert C. Williamson, Elodie Vernet, and Mark D. Reid. Composite multiclass losses. Journal of Machine Learning Rexearch, 17(222):1-52, 2016.
Eunho Yang, Aurélie C. Lozano, and Aleksandr Aravkin. A general family trimmed estimators for robust high-ddmeesiionl data analysis. Electron. J. Statist., 12(2):3519-3555 2018
Min Yang, Linli Xu, Martha White, Dale Schuurmans, and Yao liang Yu. Relaxed clipping: A global training method for robust regression and classification. In D. Lafferty, C. I. Williams, Shylor R. S. Zemel, and A. Culotta (eds.), Advances in Neural Information Processing Systems 23, pp. 2532-2540. Curran Associates, Inc., 2010.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Analysis of gradient clipping and adaptive scaling with a relaxed smoothness condition. arXiv e-prints, art. arXiv: 1881, May 2019.
Tong Zhang. Statistical behavior and consistency of classification methods based convex risk minimization. Annals of Statistics, 32(1):56 85, 2004.
Tong Zhang and David Johnson. A robust risk minimization based named entity recognition system. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 Volume 4, CONLL '03, pp. 204-207, Stroudsburg, PA, USA, 2003. Association for Computational Linguisties.
Tong Zhang, Fred Damerau, and David Johnson. Text chunking based on a generalization of winnow. J. Mach. Learn. Res., 2:615-637. March 2002. ISSN 1532-4435.
Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp. 8792-8802, USA, 2018, Curran Associates Inc
14

---

Published as a conference paper at ICLR 2020
A PROOFS OF RESULTS
Proof of Lemma Following Equation
(zo(z.y) Vma (2. y) Ø (m v)). Similarly, for the linear loss function Ein (I, v; ma (r, v). we have
Vinn (z, y; 8) Vmo (xr y).
To compute the normalised gradient, we need (fgg(,,))le = [Vma y)112 10' (m)((,))))
Thus,
(le y) (io(,,
Vms( v) o ('mm((a, u)) [ v))
Vi (ar, w)ll
Vma (x, v) sign( (ign(o(mmoxx; (mo( (x,
Vmm (a, y)1l
Vinn Vin 9),,
Vma v)ll
since (=) < 0 by assumption that ø is admissible, and thus decreasing. Consequently, if N = 1,
[mole.y]la Vena(z, (2, v) if (mm((((r,))) [Vmo(z.9)]l Voo(r,))
(()) = clip, (z =
else.
Assuming a linear scorer s(x; 8) = TTx,
x, the score gradient vs(æ; 0) x, which is independent
The clipped gradient thus corresponds the gradient under "Huberised" loss function
Vmo(nevill (x, if (e' 2 [[mo0 (o(x,))
Eo v)
else.
Proof of Lemma 2. Since & is strictly convex and decreasing, it must be strictly decreasing. We have
d'(x)
if [ø' 27
clip, (d'(z))
[o([=]]
(=(=)
else
T sign( o' (2))
if (f 2T else
('(z)
-T
if d'))) S -T
since Ø is strictly decreasing
('(z)
else
if S (6'))
('(=)
else,
where (d')-1 exists since ø' is strictly inceasing by definition of strict convexity. Now, by definition, the Huberised loss is
x+(2) - (-T))
if
(z(2)
else.
and so
-T
of
(+(2)
()))
else,
which exactly equals clip, (0)( We remark here that o, is continuous, since for any convex conjugate,
0' (- = = (0')- (((00)(-(-1))) Plugging in u = т, at the intersection point 2g = (6)- the two pieces of the function, = d(zo).
(70) = = -(--)
15

---

Published as a conference paper at ICLR 2020
Proof of Lemma 3. For admissible o, the Huberised loss o, of (8) is trivially convex, differentiable everywhere, and decreasing. In particular, we must have (0) < 0, and so o, must be classification calibrated by Bartlett et al. (2006, Theorem 2). As an illustration, Figure shows the minimiser of the conditional risk,
2" (1) argmin n (1 6-(= =ER
+ (1
Note that this quantity must be non-negative if and only if 17 > 1/2 for a loss to be classification calibrated. This easily verified be true for the Huberised logistic loss, regardless of т.
6
4.
2
0
-2-
t= 0.1
T= 0.5
0.0
0.2
0.6
0.8 1.0
Figure 3: Risk minimiser for the Huberised logistic loss.
Now suppose that Φ is a proper composite loss. To show that क, is proper composite, suffices
(x) by Williamson al. (2016, Corollary 12) to show that <[(=)) is strictly monotone and continuous. Continuity is immediate; for monotonicity, observe that by definition,
(+)
-T
if S (0')-
('())
else.
For brevity, let 20 (-T)- where we take 20 = -0 if -T & ran(d'). Then,
(20)
if S 20
J,(2)
d2)
else.
Φ (20)
if 20 else.
8x(-2)
(-2=)
Thus, the quantity of interest is
(30)
if S 20
(2)2)
if < = < 20
(1(22)
if 4 < = S 20
('(+)
if 2 20-
(=0)
Since e is strictly convex, o' is strietly decreasing and thus invertible. Thus, the ratio (x(=)
is invertible, provided 2p < 0, i.e., 2 (00).. Consequently, o. is proper composite when 7 2 (0). To intuit the need for the restriction on r, observe that by (Reid & Williamson, 2010, Corollary 12) the tentative link function for the loss is
F(2) =
((2)
(-
(-2)
if S 20
('
F(2)
if 20 < < 20
20 S 2 S 20 if > 20-
2
d'(o))
(%aa) (z)
16

---

Published as a conference paper at ICLR 2020
When < ~0)(00) the above is be non-invertible. See also Appendix C for illustration of this link function for the logistic loss.
Proof of Proposition 4. In order to apply Long Servedio (2010, Theorem 2), we simply need to check that the loss Ø, is a convex potential in the sense of Long & Servedio (2010, Definition 1). This
requires that o, is convex, non-inereasing, continuously differentiable, and asymptotes to zero (or equally, is bounded from below). Each of these is satisfied by assumption of ø being admissible. D
Proof of Lemma 5. By Lemma 2, we may write clip, (+"(u)) as the derivative of a partially Hu- berised base loss 6r given by
(+) = -T. (²(-)) if (' -T else
(4(u)
This induces composite margin loss Φ, f F. Now let & y: 0) = (mo v)). The gradient under this loss is
F(mo (2, v)), and define
(o,,(x,
Voδ- (mo(()))))) Vop-(po
= Vope(a, u) 5 (pa(z,))))
Vope(r, u) clip, ( (z(())(z)))) cl clip, (Vl((,,)))
Thus, CL-gradient clipping is equivalent to using the loss θ,
Proof of Lemma 6. We proceed in a similar manner to Lemma 3: to show that the loss is proper, we must establish that (.(x) is invertible. By Reid & Williamson (2010, Corollary 14), for the margin loss & to be proper composite with link F', it must be true that F' satisfies the symmetry condition F(-2) = -()). We thus have
(1(2) = F'(x) (L((()))
xx(=2) = F'(-2) of (F(=2))
= =F' 40(1 F()))(
Consequently,
(,(=)
(F(=))
(1(=2)
Since F is invertible by assumption, the above is invertible if and only if (ffa)
is invertible.
Observe that
(+) = -T
u S (-7)
(u)
else.
Let un = (--) for brevity. Then,
if un (1 uo) if un S u S 1 41
(u) / (1 w.
(w)
(I-
1
if uo S 40
('(a)
if 2 uo V (1 uo)
This quantity is invertible provided uo I Le., -2(12). A subtlety, however, is that the above does not necessarily span the entire range of [0, co]: consequently, Φr itself is proper composite, with link function of its own.
Even when 7 is small, one may verify that the loss o, is nonetheless classification calibrated: this is because for any 17 E (0, 1), the minimiser 2" =n))) of the conditional risk
17

---

Published as a conference paper at ICLR 2020
must satisfy the stationarity condition
(1(2²) o (-2") 1
We thus need to find suitable 2" such that the left hand side equates to a given constant. Now, for (x)
C. Thus, provided 7) * 2. there is unique =" (u") such that (((((=))) C. One may verify that this
any C * there unique u such that (.
1. r 2 for
d' (1-F(=)))
example, see Figure 4, which visualises the risk minimiser for various values of r. Thus, the sign of the minimising score conveys whether or not the positive class-probability dominant; thus, the loss is classification calibrated.
10
5
0-
-5.
r= 2.0
1.1
-10
0.0
0.2
0.4
0.6
0.8
7
Figure 4: Risk minimiser for the partially Huberised logistic loss.
Proof of Proposition 7. Let (f) denote the risk on the clean distribution of a predictor f with respect to the partially Huberised loss şr with parameter т. Similarly, Rr denote the risk on the noisy distribution. Following Ghosh et al. (2015); van Rooyen et al. (2015), we have
(f) (1- RR) (E + (1-
That is, the risk on the noisy distribution equals scaled version of the risk on the clean distribution, plus an additional term. This term is a constant independent of and only if Φr satisfies the symmetry condition of Ghosh et al. (2015), namely, (-( + 2-(1 u) C for some constant C.
Even when the symmetry condition does not hold, one may nonetheless aim bound this additional term as follows. For simplicity, we restrict attention here to & being the log cross-entropy loss sp(u) = log u. By definition, for any f E (0, 1).
T f + log
log(1 - f)
if 1 A (1 2) #
f - T (1 - f) - 2- llolog + 2
(+(() + +r(1 -)-
log f - log(1 - -f)
log f r (1 f) + log +
2
- r. f + llog + 1 - log(1 (1 - f)
ifps 1)
+ 2. 2llog +
if ifl- 1 1 2f #
log f log(1 f)
log ʃ - T (1 - f) + log +1
Evidently, all piecewise functions involved are bounded on the respective intervals. For example,
log T + 1
1oll-o log 1 V (1-
if ^ (1 4) if 1 <
-T 2 loo 2
+ f)5
2 log 1
i (1 4) v (1
log [ v (1 - 4)] + +
18

---

Published as a conference paper at ICLR 2020
By taking the maximum of each of the quantities on the right hand side which are constants depending on T - we may thus find constants C1, C2 such that
Rf( (1 - p)
R-(s) +1 2.p)
(+) 2 (1 21 (J) C2.
Now let I denote the minimiser of the clean risk R2 (J), and f the minimiser of the noisy risk (f). Then, using each of the above inequalities,
R-(f*) R+(F*) S 2. p) (R* R-())-(())))- C2
where the last inequality is because (f*) & (f*)) by definition of f*. The claim follows.
19

---

Published as a conference paper at ICLR 2020
B PROPER AND PROPER COMPOSITE LOSSES
Beyond requiring cntafficationccaiibaatio it is often desirable to use classifier outputs as valid probabilities. Proper losses (Savage, 1971; Schervish, 1989; Buja al., 2005) yp: [=1]x(0,1] R+ are the core losses of such class-probability estimation tasks, for which
(Ap E [0, 1]) argmin
(v u)] =
(13)
wE(0.1) y>-Ber((p)
Equation 13 stipulates that when using to distinguish positive and negative labels, is optimal to predict the positive prosabbbbbb....t...
Typically, it is more useful to work with losses that accept real-valued scores, e.g. output by the pre-activation of the final layer of a neural network. To this end, proper composite losses (Reid & Williamson, 2010; Nock & Nielsen, 2009) those f: {+1] - R, such that Fo is proper, for invertible link function F: - [0, 1]. Proper composite losses are inherently classification- calibrated (Reid & Williamson, 2010, Theorem 16). Canonical examples are the logistic loss b(y. log(1 1 -e - with F; - (v)), and exponential loss f(y. v) = e with F;; + a22 v). When é differentiable, we have (Reid & Williamson, 2010, Corollary 12)
(Ae E F()) (1:000111110 (+11,
(14)
(((11,))
Given proper loss yp and "symmetric" link F with F(-t) 1 F(v), the loss Elu, v) = c(v, F(v))
defines a margin loss (Reid Williamson, 2010, Corollary 14).
Proper composite losses may also be extended to multiclass settings in the natural way (Gneiting & Raftery, 2007; Williamson al., 2016): one now defines a proper loss pp: [K] Ax - IR,, where K is the number of classes and Ax denotes the Kimplex. A proper composite loss may be defined using a link F; RK + Ak. such as the softmax operator F&(v) Ex exp(ve.)' Combined with the log-loss p(v.p) p) = log Py, this yields the standard softmax cross-entropy loss.
exp(ve)
20

---

Published as a conference paper at ICLR 2020
C ILLUSTRATION OF LOSSES
We illustrate the Huberised, partially Huberised, and generalised cross-entropy losses as their under- lying tuning parameters are varied. Additionally, we illustrate the link functions that are implicit in each of the losses, which illustrates that they may be non-invertible if T is too large.
C.I HUBERISED LOSS
Figure illustrates the Huberised version the logistic loss, and its derivative. Following the proof - - (T), the Huberised loss o, has implicit link function
of Lemma 3, for E (0, 1) and 2g (see Figure 6)
aiz)
i S 20 z if - 20 5 z S 20
(Vz E (2)
(2)
-20
((=))
if 2 -20-
Compared to the standard sigmoid, the Huberised link saturates more slowly as 7 is decreased. Note that when т S 4. the link function not invertible everywhere: this results in the loss not being proper composite per our definition.
C.2 PARTIALLY HUBERISED LOSS
Figure illustrates the partially Huberised version the logistic loss, as well as the base log-loss. Following the proof of Lemma 6, the partially Huberised loss Φ, has implicit link function
if s ^ (1- 4) if s if if 2 (1 -1-+) v)
I+rial (v)
(+( =
Ta(()) I+T-o(()
Figure 9 illustrates the link function. The logistic loss has (p() - log and so (2(122 = 2. For 1, the link function will non-invertible everywhere. which expected since the loss here is the linear loss, which is not suitable for class-proaability estimation. For r E (1,2), the link function will be invertible for & [[1 - +) ]. Intuitively, the case r (1, 2) corresponds to the linear regions of the losses on positive and negative instances crossing over. For 2 2, the link function will always be invertible.
may be observed that partial Huberisation causes the link function to saturate values [1/(1 T), (1(1+r))) this does not affect classification calibration, but does imply that rescaling is necessary order intepret the output probabilities.
C.3 ILLUSTRATION OF GENERALISED CROSS-ENTROPY
Figure illustrates the base Pa loss, and its composition with sigmoid link function.
21

---

Published as a conference paper at ICLR 2020
6.10
-0
-5
-0.2
0.4
-03
(a) Loss
(b) Derivative.
Figure 5: Illustration of Huberised logistic loss. The effect of Huberisation is to linearise the loss beyond certain point, or equally, to cap the derivative when its magnitude exceeds certain threshold.
F-(w)
0.8
0.6
0.2
2
Figure 6: Illustration of link function for the Huberised logistic loss. The effect of Huberisation is to make the link function saturate slower. Note that when 3. the link function not invertible everywhere.
3
3
9-(0)
2
0.2
0.4
0.6
0.8
(a) Log-l
(b) Log-loss with sigmoid link.
Figure 7: Illustration of partial Huberisation, in terms of the underlying proper loss and its composite form.
22

---

Published as a conference paper at ICLR 2020
Falu)
0
0.1
0.1
0.5
0.5
3
0.9
0.9
0.2
0.4
0.6
0.8
(a) Base loss.
(b) Composite loss with sigmoid link.
Figure 8: Illustration generalised cross-entropy loss various choices of [0, 1).
1x
F(v)
0.2
Figure Ilustration of link function for partially Huberised logistic The effect partial Huberisation make the link function saturate slower. When r < 2, the link function not invertible everywhere.
23

---

Published as a conference paper at ICLR 2020
D ROBUSTNESS TO INSTANCE OUTLIERS
Following Ding (2013, Section 3.2.2), we consider robustness to an arbitrary corruption under a linear scoring model. For margin loss f(x, y; 0) = d(y zTz) with empirical risk minimiser dy sample ( 8n
) 1 suppose that the sample corrupted with an outlier (x',3/) One would like to ensure that 0 is not swayed by making |>>||2 arbitrarily large: that is,
N
Vilzn, UniON ( + Ve(a 0':0N) =
N
n=1
or equivalently, lim Vi(a', v = O. Fortunately, the saturating behaviour of the partially Huberised loss affords this, now show.
Proposition 8. Pick any convex, disserentiable, proper composite margin loss d, whose link F satisfies lim=-+-oc (2) 0. For any > 0, let (x(x, y; 0) = 6-(y (rxr be the r-partially Huberised loss under a linear model. Then, for any (xr, U) and such that [101|2 +00 and 6Tx of
lim I (2, 0)0) = O.
Proof of Proposition 8. By definition,
(1) o xxx).
Thus,
(z, = ||0|| 10'(y 0 x)
1
(y TTx)) 10 (y 0 x)] y. |01l2 cos 2/2.8
[Tx))
where va.0 denotes angle between æ and If 0f 0 x + O, then cos th. + O, and so the third term finite. Thus, i/Vê, (I. y; 0)]l2 - lim limg | (=). depending on the sign of y fy-O== x. By definition of Φr, the derivative of the loss asymptotes to either Fe) or ('v)). Now, lim;---o0 2 (v) = 0
by assumption, and lim,-++e ('( 0 since Φ convex, the claim is shown.
24

---

Published as a conference paper at ICLR 2020
E ADDITIONAL SYNTHETIC EXPERIMENTS
We provide some more details regarding the synthetic data used the body.
E.1 DETAILS OF LONG & SERVEDIO (2010) DATASET
The problem considered Long Servedio (2010) (see Figure 10) comprises distribution con-
centrated on six atoms (+(11 0), + 5r), (n, -7) C RR2 for some 7 > 0; we chose
An instance (I1, x2) is labelled as = [1 2 O]. The instances are weighted so that the first four atoms have probability mass g. and the last two atoms mass 1. We modify this distribution slightly by treating the atoms as means of isotropic Gaussians, and treating the marginal distribution over instances to be mixture of these Gaussians with mixing weights given by the corresponding probability masses of the atoms.
E.2 DETAILS OF OUTLIER DATASET
For the experiment involving outliers in feature space, the data comprises points on the line. Positively labelled samples are drawn from a unit variance Gaussian centered at (1, 1), with positively labeled outliers drawn from (-200, 1). Negatively labelled samples comprise the negation of all points. We
learn an unregularised linear classifier from this data, which corresponds to single scalar 0.
25

---

Published as a conference paper at ICLR 2020
0.4
0.2
X
0.5
0.5
+
-0.2
-0.4
Figure 10: Distribution proposed in Long & Servedio (2010), which defeats any member of broad family of convex losses. The data comprises points lahelled negative. The two "fat" points have twice as much probability mass as their "thin" " counterparts. While the dataset is trivially linearly separable, minimising a broad range of convex losses with linear model under any non-zero amount of symmetric label noise results in predictor that is tantamount to random guessing.
points, with the blue points labelled positive, and the
26


Reviews:
[{"review_id": "r1e6vm9pFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n[Summary]\nThis paper studies the relationship between gradient clipping in stochastic gradient descent and robustness to label noise. Theoretical results show that gradient clipping in general is not robust to symmetric label noise. The paper then proposes a variant of gradient clipping (cl-clipping) that induces label noise robustness. Experiments support these claims on synthetic datasets and typical classification benchmarks.\n\n[Decision]\nThe first contribution, that gradient clipping does not induce robustness to label noise, is an important negative result given the prominence of gradient clipping and datasets with noisy labels. The second contribution, cl-clipping, amounts to minimizing a non-convex loss with saturating regions but, as far as I know, these properties are necessary for robustness to label noise. Theoretical results are limited to SGD with mini-batch size 1 but the insights carry over to larger mini-batches in the experiments. Overall, I recommend acceptance.\n\n[Comments]\nThe parameter tau controls robustness, and a higher noise level requires a higher tau. There is little discussion on how this parameter is chosen in the experiments. On the synthetic dataset, the Huberized loss uses tau=1 and the partially Huberized loss uses tau=2. How are these values chosen? Did the authors observe a U-shaped curve when sweeping over tau? On the real-world datasets, tau is fixed for each method across different noise levels. Does this mean that a single value of tau worked best regardless of the noise level, or was it tuned for a particular noise level?\n\nProposition 4 shows that symmetric noise breaks down the clipping method in Eq (7) which can be seen as a special case of gradient clipping. I might be missing something here, but it is not obvious to me that, when the norm of x is constant across the samples, Eq (7) is equal to gradient clipping.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SkgTDboqtH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studied a fundamental problem in robust machine learning, that is, can gradient clipping mitigate label noise? The paper is well written, clearly motivated, highly novel and significant not only in a theoretical sense but also in a practical sense.\n\nAs argued in the abstract, gradient clipping is a widely-used technique which is generally motivated from the OPTIMIZATION point of view. In this paper, the authors proposed an entirely new motivation of gradient clipping from the ROBUSTNESS point of view, since intuitively it should be able to mitigate label noise. Surprisingly, the authors proved that for some simple binary classification with label noise, standard gradient clipping does not provide robustness; on the other hand, a simple variant of gradient clipping is robust, and is equivalent to suitably modifying the underlying loss function.\n\nThe major contributions were well summarized in the bottom of page 1, the related work was discussed in sec 4.3, and the caveats of this new methodology were also given in sec 4.4. Note that the proposed composite loss-based gradient clipping is applicable even on top of existing noise-robust losses, for example, the generalized cross-entropy loss, and this serves as a convincing demonstration of the great significance of the paper. Actually, the paper is full of insights, and I really enjoyed reviewing it.\n\nI have a few questions on Tables 1 and 2. Table 1 said the standard gradient clipping is not robust according to Proposition 4. However, Proposition 4 is for the loss-based gradient clipping in (7) rather than the standard gradient clipping in (6). Perhaps I have missed something, but why the non-robustness of (7) can imply the non-robustness of (6)?\n\nIn Table 2, Linear loss on MNIST, the test accuracy was 9.6 when rho=0.6. Is this a typo? The accuracy was still 78.8 when rho=0.4. Moreover, the authors explained why the theoretically grounded linear loss performed so badly, that is, the optimization was so difficult. How about CE+clipping loss on CIFAR-100? The standard gradient clipping was also harmful here, even when there was no label noise at all. Was this due to bad optimization or robustness (or both)? \n\nSince distributionally robust supervised learning is a future direction to go, I think the authors may be interested in a thought-provoking paper:\nW. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? ICML 2018.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJggf3N3Kr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nGradient clipping has been studied as an optimization technique and also as a tool for privacy preserving, but in this paper, it studies the robustness properties of gradient clipping.  More specifically, the main question of the paper is: Can gradient clipping mitigate label noise?  The paper reveals that the answer is no, but further proposes a simple variant of gradient clipping is robust and has nice property of classification calibration.  Experiments show that the proposed variant works under label noise.\n\n\nStrength of the paper:\n\nThe motivation and goal of the paper is stated in the title and is very clear, making it easier to follow the story of the paper.  There are sufficient background on the loss functions and gradient clipping in the beginning that helps guide the reader.  The proposed method is robust to label noise and has theoretical guarantees.  The relationship between similar work is summarized.  Experiments have both synthetic and benchmark datasets to demonstrate the behavior of the proposed method.\n\n\nWeakness of the paper:\n\nCurrently, the experiments only include methods studied in the paper.  It would be better to include baseline methods stated in the end of Section 5 or in Section 4.3.\n\nAfter response:\nThank you for the clarification!  I have read the other reviews and author comments.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the effect of clipping on mitigating label noise. The authors demonstrate that standard gradient clipping does not suffice for achieving robustness to label noise. The authors suggest a noise-robust alternative. In the discussion the reviewers raised some interesting questions and technical detailed but mostly agreed that the paper is well-written with nice contributions. I concur with the reviewers that this is a nicely written paper with good contributions. I recommend acceptance but recommend the authors continue to improve their paper based on the reviewers' suggestions.
Decision: Accept (Poster)