Shape Matters: Understanding the Implicit Bias of the Noise Covariance
Authors: Jeff Z. HaoChen
Colin Wei
Jason D. Lee
Tengyu Ma
Venue:  (2021)
Keywords: implicit regularization
implicit bias
algorithmic regularization
over-parameterization
learning theory

Abstract: The noise in stochastic gradient descent (SGD) provides a crucial implicit regularization effect for training overparameterized models. Prior theoretical work largely focuses on spherical Gaussian noise, whereas empirical studies demonstrate the phenomenon that parameter-dependent noise --- induced by mini-batches or label perturbation --- is far more effective than Gaussian noise. 
This paper theoretically characterizes this phenomenon on a quadratically-parameterized model introduced by Vaskevicius et al. and Woodworth et al.  We show that in an over-parameterized setting, SGD with label noise recovers the sparse ground-truth with an arbitrary initialization, whereas SGD with Gaussian noise or gradient descent overfits to dense solutions with large norms. Our analysis reveals that parameter-dependent noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not. 

Under review as a conference paper at ICLR 2021
SHAPE MATTERS: UNDERSTANDING THE IMPLICIT BIAS OF THE NOISE COVARIANCE
Anonymous authors Paper under double-bling review
ABSTRACT
The noise in stochastic gradient descent (SGD) provides a crucial implicit reg- ularization effect for training overparameterized models. Prior theoretical work largely focuses on spherical Gaussian noise, whereas empirical studies demon- strate the phenomenon that parameter-dee dereaneterddeeedennt noise induced by mini- batches or label perturbation - is far more effective than Gaussian noise. This pa- per theoretically characterizes this phenomenon on a aaairaaically--paaaeeetiiie model introduced by Vaskevicius et al. and Woodworth et al. We show that vver-parameterized setting. SGD with label noise recovers the sparse ground- truth with an arbitrary initialization, whereas SGD with Gaussian noise or gradi- ent descent overfits to dense solutions with large norms. Our analysis reveals that noise introduces a bias towards local minima with smaller noise variance, whereas spherical Gaussian noise does not.
INTRODUCTION
One central mystery of deep artificial neural networks is their capability to generalize when having far more learnable parameters than training examples Zhang et all (201G). To add to the mystery, deep nets can also obtain reasonable performance in the absence of any explicit regularization. This has motivated recent work to study the regularization effect due to the optimization (rather than objective function), also known as implicit bias or implicit regularization Gunasekar at (20T7 2018 Soudry et al; (2018); Arora et all (2111)) The implicit bias is induced by and depends on many factors, such as learning rate and batch size Smith et all (2017); Goyal et all (2017)); Keskar et a (2(1 Li et all (20TT)); Hofter et all (2))) initialization and momentum Sutskever et all (201)) adaptive stepsize Kingma and Ba (201 Neyshabur et all (20115); Wilson aa (2017). batch
normalization loffe and Szegedy (2015) and dropout Srivastava et all (2012))
Among these sources of implicit regularization, the SGD noise believed to be a vital one (LeCun eral 2012: Keskar et all 2016). Previous theoretical works (e.g., Lt all (2M1)) have studied the implicit regularization effect from the scale of the noise, which is directly influenced by learning rate and batch size. However, people have empirically observed that the shape of the noise also has a strong (if not stronger) implicit bias. For example, prior works show that mini-batch noise or label noise (label smoothing) noise the parameter updates from the perturbation of labels in training is far more effective than adding spherical Gaussian noise (e.g., see (Shallue et all 2018, Section 4.6) and Szegedy et all (2016); eeal (2OTS)) We also confirm this phenomenon in Figure M (left). Thus, understanding the implicit bias of the noise shape is crucial. Such an understanding may also apply to distributed training because synthetically adding noise may help generalization if parallelism reduces the amount of mini-batch noise (Shallue et all 2018).
In this paper, we theoretically study the effect of the shape of the noise, demonstrating that it can provably determine generalization performance at convergence. Our analysis is based on a nonlinear quadratically- paticcallly-pammtttiieeee model introduced by (Woodworlh et all 2020, Vaskevicmss el aL. 201).. which is rich enough to exhibit similar empirical phenomena as deep networks. Indeed, Figure II (right) empirically shows that SGD with mini-batch noise or label noise can generalize with arbitrary initialization without explicit regularization, whereas GD SGD with spherical Gaussian noise cannot. We aim to analyze the implicit bias label noise and Gaussian noise in the quadratically- parametrized model and explain these empirical observations.

---

Under review as a conference paper at ICLR 2021
:40
Train Test teror
200
trin
last
large
full bacch
small batch
strall pasch
CIAR100 Trein,
I.. bbcl noise
label nose
16 2e-5
oule?
Ib, 40.5
om7eed
emled
20-6
10
Tb. 0022-1
Ib, g 60.4
10
ID. mo.36
Iteration
Iteration
Figure 1: The effect of noise covariance in neural network and models. We demonstrate that label noise induces stronger regularization effect than Gaussian noise. In both real and synthetic data, adding label noise to large batch (or full batch) SGD updates can recover small-batch generalization performance, whereas adding Gaussian noise with optimally- tuned variance o² cannot. Left: Training and validation errors on CIFAR 100 for VGGI9. Adding Gaussian noise to large batch updates gives little improvement (around 2%), whereas adding label noise recovers the small-batch baseline (around 15% improvement). Right: Training and validation error on a 100-dimensional quadrraically-paaantttieee deep models, label noise or mini-batch noise leads to better solutions than optimally-
parameterized model defined in Section Ø Similar to
spherical Gaussian noise. Moreover, Gaussian noise causes the parameter to diverge after sufficient mixing, suggested by our negative result for Langevin dynamics (Theorem (42). More details in Section A..
We choose to study label noise because it can replicate the regularization effects of minibatch noise in both real and synthetic data (Figure W. and has been used regularize large-batch parallel train- ing (Shallne et all 2018). Moreover, label noise is less sensitive to the initialization and the opti- mization history than mini-batch noise, which makes it more amenable to theoretical analysis. For example, in an extreme case, if we happen reach or initialize at solution that overfits the data exactly, then mini- batch SGD will stay there forever because both the gradient and the noise van- ish (Vaswani et all 2UT)). In contrast, label noise will not accidentally vanish, the analysis is more tractable. Understanding label noise may lead to understanding mini-batch noise or replacing it with other more robust choices.
In our setting, we prove that with a proper learning rate schedule, SGD with label noise recovers sparse ground-truth classifier and generalizes well, whereas SGD with spherical Gaussian noise generalizes poorly. Concretely, SGD with label noise biases the parameter towards the low sparsity regime and exactly recovers the sparse ground-truth. even when the initialization arbitrarily large (Theorem I... In this same regime, noise-free gradient descent quickly overfits because it trains in the NTK regime (lacot et all 2018; Chizat and Bacfi, 2018). Adding Gaussian noise is insufficient to fix this, as this algorithm would end up sampling from Gibbs distribution with infinite partition function and fail converge to the ground-truth -tuth (Theorem 242).. In summary, with not too small learning rate noise level, label noise suffices to bias the parameter towards sparse solutions without relying on a small initialization, whereas Gaussian noise cannot.
Our analysis suggests that the fundamental difference between label or mini-batch noise and Gaus- sian noise is that the former is purameter-dependeer and therefore introduces stronger biases than the latter. The conceptual message highlighted by our analysis is that there are two possible implicit biases induced by the noise: 1. prior work (Keskar et all 20G) shows that by escaping sharp local minima, noisy gradient descent biases the parameter towards more robust solutions (i.e, solutions with low curvature, or "flat" minima), and 2. when the noise covariance varies across the param- eter space, there is another (potentially stronger) implicit bias effect toward parameters where the noise covariance is smaller. Label or mini-batch noise benefits from both biases, whereas Gaussian noise is independent of the parameter, it benefits from the first bias but not the second. For the parameterized model, this first bias is not sufficient for finding solutions with good generalization because there is a large set of overfitting global minima of the training loss with reasonable curvature. In contrast, the covariance of label noise is proportional to the scale of the parameter, inducing a much stronger bias towards low norm solutions which generalize well.
1.1 ADDITIONAL RELATED WORKS
Closely related to work, Blanc et all (2019) and Zhu et all (2019) also theoretically studied implicit regularization effects that arise due to shape, rather than scale, of the noise. However, they only considered the local effect of the noise near some local minimum of the loss. In contrast, our
2

---

Under review as a conference paper at ICLR 2021
work analyzes the global effect of noise. For more detailed comparison with (Blanc et all 20190. see Section 2
Woodworth al (2002)); Vaskevicuss ei al (2019) analyze the effect of initialization for the same model that we study, showing that large initialization trains in the NTK regime (shown to generalize poorly (Wer all 2019; Ghorbani et all 2019) whereas small initialization does not. We show that when the initialization is large, adding noise helps avoid the NTK regime (Li and Liang, 2018; Tacot Du all 2018E; Woodworth et all 202 without explicit regularization.
atab 2018:
Several previous works have studied generalization bounds and training dynamics of SGD with state- dependent noises for more general models. Hardt et all (2015) derived stability- -baedd generalization bounds for mini-batch SGD based on training speed. Cheng et al. (2I1)) proved that SGD with state- dependent noises has iterate distribution close the corresponding continuous stochastic differential equation with the same noise covariance. Meng et all (2020; Kie et all (2020) showed that SGD with state-dependent noises escapes local minimum faster than SGD with spherical Gaussian noise.
There has been line of work empirically studying how noise influences generalization. Keskar atal (2UG)) argued that large batch training will converge to "sharp" local minima which do not generalize well. Hoffer et all (20117) argued that large batch size doesn't hurt generalization much if training goes on long enough and additional noise is added with a larger learning rate. [Goyal et al, (2017) and Shallue et al (2IT) showed large batch training with proper Icarning rate and additional label noise can achieve similar generalization as small batch. [We1 and Schwab (2019); Chaudhani and Soatto (2018); Yaıdal (2018) (heuristically) suggested that SGD may encourage solutions with smaller noise covariance. Martin and Mahoney (2018) used random matrix theory to analyze im- plicit regularization effects of noises. The noise induced by dropout has been shown change the expected training objective, hence provides a regularization effect (Mianjy et all 2018; Mianjy and Arorl 20T9; Wer ell all 2020, Arora al (2020). Wet et
(2020) showed that there also exists an
implicit bias induced by dropout noise.
Langevin dynamics or the closely-related stochastic gradient descent with spherical Gaussian noise has been studied in previous works |Welling and Teh (201 Ich et all (2016); Ragınsky et al. (201 70:
Zhang et al. (2017); Mou et all (201 7); Roberts all (199)); Ge (20155); Negrea et al. (2019)
Neelakantan et all (2015); Mou et all (2018). In particular, Raginsky et all (2017) and Li et all
(2019a) provided generalization bounds for SGLD using algorithmic stability.
Several works have theoretically analyzed other types of implicit biases in simplified set- tings (Soudry et all 2018: Gunasekar et all, 2088 Ji and Telgarsky. 2018%).. Gunasekar et all (2111)) and Lietal al (201)) showed that gradient descent finds low rank solutions in matrix com- pletion. Gradient descent has also been shown to maximize the margin in linear and homogeneous models (Soudry al, 2018: Di and Telgarsky, 2018 Nacson et all, 2018 Lyu and Li, 2019; Ghe et 20 Nacson et all 201 9 Poggio et all 2017).. eu (2018)) showed that gradient descent implicitly balances the layers of deep homogeneous models. Other works showed that it may not always be possible to characterize implicit biases in terms of norm (Arora el all 2013 A Razin and
Cohen, (020). Gissin al (201%) showed that gradient descent dynamics exhibit different implicit
biases based on depth. Li al (2019R) studied the implicit regularization effect of a large initial learning rate.
Guo et all (2018) studies the notion of "climination singularities" in RBF networks, where optimiza- tion runs into regime with small weight and small gradient, therefore the training can be slowed down. Our paper also involves training trajectory with small weight norm, but instead focuses on its influnece on the generalization performance.
2 SETUP AND MAIN RESULTS
2.1 SETUP AND BACKGROUNDS
Parameterization. We focus on the nonlinear model parametrization: fz(x) & (o02, .r), where V E Rd is the parameter of the model, x E Rd is the data, and 02 denotes the clement-wise square of v. Prior works (Woodworth all 2020: Vaskevieius et all 2019; Lietall 2017) have studied this model because it is an interesting and informative simplification of nonlinear models. As SGD noise exhibits many of the same empirical behaviors in this simplified model as in deep networks." we use
In contrast, the implicit bias of noise wouldn't show up in a simpler linear regression model.
3

---

Under review as a conference paper at ICLR 2021
this model as testbed to develop mathematical understanding of various sources of implicit biases. As shown in Figure M. both SGD with mini-batch noise and label noise generalize better than GD or SGD with spherical Gaussian noise.
Data distribution assumptions and overparametrization. We assume that there exists a ground- truth parameter v* E R that generates the label y = (u+ 2) given a data point x, which is assumed to he generated from (0, Iaxa). A dataset D = { (x:10,)) (ylyy))))))))))))) )) generated from this distribution. The implicit bias is only needed in weer-paramettrrized regime, and therefore we assume that n < d. To make the ground-trute vector information -heortecitt recoverable, we assume that the ground-truth vector V* is r-sparse. Here r is much smaller than d, and casual readers can treat it as a constant. Because the clement-wise square in the model parameterization is invariant to any sign flip, we assume v" is non- non-negative without loss of generality.
i=1 of i.i.d data points are
For simplicity, we also assume it only takes value in {0, 1). f We use S C [d] with | = r to denote the support of throughout the paper.
We remark that we can recover e" by re-parameterizing u = o0 and applying LASSO (Libshirani. [19К) in the u-space when n 2 Ö(r), which is minimax optimal (Raskufti et al 2012). However, the main goal of the paper, similar to several prior works (Woodworth et all 2020 Vaskevieus atl 20R: Liet all, 2017). is to prove that the implicit biases of non-convex optimization can recover the
ground truth without explicit regularization in the per-parameterized regime when n = poly(r) < d.5 We also assume throughout the paper that n, d are larger than some sufficiently large universal constant.
Loss function. We use the mean-squared loss denoted by (v) & (fo(z(0) - y())²² for the i-th example. The empirical loss is written as L(v) & 1.2 -m (v).
Initialization. We use a large initialization of the form 2.00 = T where denotes the all I's vector, where allow to be arbitrarily large (but polynomial in d).
Algorithm 1 Stochastic Gradient Descent with Label Noise
Require: Number of iterations T, sequence of step sizes ,,(0:T), noise level 5, initialization o.!)
1: for 0 to T 1 do
2: Sample index i, [ni] uniformly and add noise s { to g(t)
3: Let ((%) 1 (f. (x(()) - -(0) (160)ss)) s,)
4:
uit+1] < [t] - ((()
b update with label noise
SGD with label noise. We study SGD with label noise shown in Algorithm MI We sample an example, add label noise sampled from {=6} to the label, and apply the gradient update. Computing the gradient, obtain the update rule written explicitly
[e- + rlt - - (1ol) 2 x(())) (2(6)) C w!th + + spxl)) c ult!
(1)
Langevin dynamics/diffnsion We compare SGD with label noise to Langevin dynamics, which adds spherical Gaussian noise to gradient descent all 20TI):
el++1] + ylth ()(l( + /2n/A
where the noise { ~ (O, Iaxa) and À controls the scale of noise. Langevin dynamics (LD) or its more computationally efficient effffiit variant, stochastic gradient Langevin dynamics (SGLD), is
known to converge to the Gibbs distribution ((v) xc -x) under various settings with sufficiently 1996 Dalalyan, 2017: Bubeck all 2013: Ragınsky et al,, 2017).
small learning rate (Roherls In our negative result about Langevin dynamics/diffusion we directly analyze the Gibbs distribution in order to disentangle the convergence and the generalization. In our negative result about Langevin dynamics/diffusion we directly analyze the Gibbs distribution to disentangle the convergence and the generalization.
Notations. Unless otherwise specified, we use (()) O()) to hide absolute multiplicative fac- tors and ((),, (.), 2(-) )) to hide poly-logarithmic factors in problem parameters such as d and For
"Our analysis can be straightforwardly extended with other non-zero values.
We also remark that it's common to obtain only sub-optimal sample complexity guarantees in the sparsity parameters with non-convex optimization methods (Li et all 2017; Gi et all, 2016; Waskevicnss all 2019: Chietall 201) due to technical limitations.
4

---

Under review as a conference paper at ICLR 2021
example, every occurrence of (zz) is placeholder for a quantity f() that satisfies that for some absolute constants c1, 02 > 0, Vx, [f() I c1l2]
2.2 MAIN RESULTS
Our main result can be summarized by the following theorem, which suggests that stochastic gradi- ent descent with label noise can converge to the ground truth despite a potentially large initialization.
Theorem 2.1. In the setting of Section 27 given target error > O. Suppose we have G(r2)
samples. For any label noise level 6 N dr? we run SGD with label noise (Algorithm ( with the following learning rule schedule:
1. learning rate do = (11/5) for To = ((1) iterations,
2. learning rate 7h (11) for T1 (11/m) iterations,
3. learning rate 722 = -6(2//8 for T- = (1/TT2) iterations.
Then, with probability at least 0.9, the final iterate [!T at time = To Ti + T2 satisfies
| - -0 S e.
(3)
Here ((.) omits poly-logarithmic dependencies on 1/e, and
In other words, with arbitrarily large initialization scale 5, we can choose large label noise level and the learning rate schedule so that SGD with label noise succeeds in recovering the ground truth. In contrast, when is large, gradient flow without noise trains the "kernel" regime as shown by (Woodworth et all, 2020 Chizat and Bach, 2018). The solution in this kernel regime minimizes the RKHS distance initialization, and our setting equates to finding a zero-error solution with minimum || ||0)@2|| 2| Such a solution could be arbitrarily far away when initialization scale T is large and therefore have poor generalization. Figure II (right) confirms GD performs poorly with large initialization whereas SGD with minibatch label noise works. We outline the proof of Theorem 2. in Section E
Blanc et all (2019) also study the implicit bias of the label noise. For our setting, their result implies that when the iterate is near global minimum for sufficient time, the iterates will locally move to the direction that reduces the f2-norm of by a small distance (that is larger than random fluctuation). solution with good generalization with large
However, it does not imply the global convergence (or any) initialization, which is what we prove in Theorem 2 Moreover, our analysis captures the effect of the large noise large learning rate- we require the ratio between the noise and the gradient, which is captured by the value 7782, to be sufficiently large. This is consistent with the empirical observation that good generalization requires sufficiently large learning rate or small batch (Goyal et all 2)17).
On the other hand, the following negative result for Langevin dynamics demonstrates that adding Gaussian noise fails to recover the ground truth even when o = 0. This suggests that spherical Gaussian noise does not induce strong enough implicit bias towards low- now-norm solutions.
Theorem 2.2. Assume in addition to the setting in Section Z. that the ground truth v* 0. When n < d/3, with probability at least 0.9 over the randomness of the data, for any X >> 0, the Gibbs distribution is not well-defined because the partition fanction explodes:
Jr
e
xC(v) do 00.
(4)
As a consequence, Langevin diffusiion does not converge proper stationary distribution.
Theorem 2, helps explain the behavior in Figure II, where adding Gaussian noise generalizes poorly for both synthetic and real data. In particular, in Figure II (right) adding Gaussian noise causes the parameter to diverge for synthetic data, and Theorem ( explains this observation. A priori, the intuition regarding Langevin dynamics is as follows: + +o0, the Gibbs distribution (if it exists) should concentrate on the manifold of global minima with zero loss. The measure on the manifold of
'It also appears difficult to generalize the local analysis directly to global analysis because once the iterate leaves the local minimum, all the local tools do not apply anymore, and it's unclear whether the iterate will converge new local minimum getting stuck some region.
5

---

Under review as a conference paper at ICLR 2021
global minima should be decided by the geometry of [(-). and in particular, the curvature around the global minimum. As À - +oo, the mass should likely concentrate at the flattest global minimum (according to some measure of flatness), which intuitively is o" = 0 in this case.
However, our main intuition is that when n < d, even though the global minimum at e* is the flattest, there are also many bad global minima with only slightly sharper curvatures. The vast volume of bad global minima dominate the flatness of the global minimum at e* = 0 for any X, and hence the partition function blows up and the Gibbs distribution doesn't I exist. The proof of Theorem DAN can be found in Section E
3 ANALYSIS OVERVIEW OF SGD WITH LABEL NOISE (THEOREM 2.
3.1 WARM-UP: UPDATES WITH ONLY GARAMETER-DEPEDDRE NOISE
Towards building intuition and tools for analyzing the parameter-dependee noise, in this subsection we start by studying an extremely simplified random walk in one dimensional space. The random walk is purely driven by mean-zero noisy updates and does not involve any gradient updates:
e - u + 77C v, where ç ~ {1
(5)
Indeed, attentive readers verify that when dimension d = 1, sample size n 1, and er' = 0, equation (I) degenerates to the above random walk if we omit the gradient update term (second to last term in equation (I)). We compare it with the standard Brownian motion (which is the analog of gradient descent with spherical Gaussian noise under this extreme simplification)
<- + n where & ~ (0,
We initialize at v = 1. We observe that both random walks have mean-zero updates, so the mean is preserved: E(u] 1. The variances of the two random walks are also both growing because any mean-zero update increases the variance. Moreover, the Brownian motion diverges because it has Gaussian marginal with variance growing linearly in i, and there is no limiting stationary distribution.
However, the parameter-dependeee random walk (E)) has dramatically different behavior when 1) < 1: the random variable v will eventually converge to v = O with high probability (though the variance grows and the mean remains at 1.). This is because the variance of the noise depends on the scale of v. The smaller v is, the smaller the noise variance is, and so the random walk tends to get "trapped" around This claim has the following informal but simple proof that does not strongly rely on the exact form of the noise and can be extended to more general high-dimensional cases.
Consider an increasing concave potential function & R22 + I220 with d" < 0 (e.g., d(v) = v works). Note that when 7) < 1, the random variable v stays nonnegative. We can show that the expected potential function decreases after any update
E(O(v + m){))] R E[((v) + o' (o))) + o" (0)
(by Taylor expansion)
E [d()]] E(ø" (0""()))"""" (1))7001 B[p(v)]
(by d"(u) 0 and E[g] 0.)
With more detailed analysis, we can formalize the Taylor expansion and control the decrease of the potential function, and conclude that E [ő(v)] converges to zero. Then, by Markov's inequality, with high probability, ((v) is tiny and so is v.
From the 1-D case to the high-dimensional case. In one dimension, the bias is introduced because of the varying scale of noise (i.e., the norm of the covariance). However, in the high dimensional case, the shape of the covariance also matters. For example, if we generalize the random walk (I) to high-dimensions by running d of the random walks in parallel, then we will observe the same phenomenon, but the noise variances in different dimensions are not identical - they depend on the current scales of the coordinates. (Precisely, the noise variance for dimension A is n²²u..) However, suppose we instead add noise of the same variance to all dimensions. Even if this variance depends on the norm of v (say. n/2 (l the implicit bias will be diminished, as the smaller coordinates will have relatively outsized noise and the larger coordinates will have relatively insufficient noise.
'In fact, one can show that if this phenomenon happens for some À > 0, then happens for all other X. TThe same proof strategy fails for the Brownian motion because is not always nonnegative, and there is no concave potential function over the real that can be bounded from below.

---

Under review as a conference paper at ICLR 2021
Outline of the rest of the subsections. We will give a proof sketch of Theorem Mi that consists of three stages. We first show in the initial stage of the training that label noise effectively decreases the parameter on all dimensions, bringing the training from large initialization to small initialization regime, where better generalization is possible (Section (.2)). Then, we show in Section G. that when the parameter is decently small, with label noise and a decayed learning rate, the algorithm
will increase the magnitude of those dimensions in support set of v², while keep decreasing the norm of the rest of the dimensions. Finally, with one more decay, the algorithm can recover the ground truth.
3.2 STAGE O: LABEL NOISE WITH LARGE LEARNING RATE REDUCES THE PARAMETER NORM
We first analyze the initial phase where we use a relatively large learning rate. When the initialization is of decent size, GD quickly overfits bad global minimum nearest the initialization. In contrast, we prove that SGD with label noise biases towards the small norm region, for a similar reason as the random walk example with parameter-dependent noise in Section B
Theorem 3.1. In the setting of Theorem 2. recall that we initialize with o00 = T.T.L 1. Assume n 2 (log d). Suppose we run SGD with label noise with noise level 8 2 O()²d)) and learning
rate 7/0 E (f(+*df/8') ((1/5) for To ((1/(1)62) iterations. Then, with probability at least 0.99 over the randomness of the algorithm,
| Ioc S 11/d.
Moreover, the minimum entry of olToo is bounded below by exp(-ō((76)))))
We remark that our requirement of y being large is consistent with the empirical observation that large initial learning rate helps generalization Goyal et al] (20I7); all (2019)). We provide intuitions and a proof sketch of the theorem in the rest of the subsection and defer the full proof to Section B Our proof is based
the construction of concave potential function 4 similar to Section I. We will show that, at every step, the noise has a second-order effect on the potential function and decrease the potential function by quantity on the order of 1726² (omitting the d dependency) _ On the other hand, the gradient step may increase the potential by quantity at most on the order of 1 (omitting d dependency again). Therefore, when 1728 82 2 17. we expect the algorithm to decrease the potential and the parameter norm.
In particular, we define ((v) A Ei=1 ((v))= = Ede Ven. By the update rule 1. the update for a coordinate & E [d] can be written as
[t+1] + ot 0/
[c]
(5e) [0] (6)) uk n1t1 (1o10002 y 1102)T aol)))
[4]
(8)
where 8t sampled from {-5,8} and in sampled from [n]. Let 9 (te ( 20 be the component coming from the stochastic gradient. Using the fact that ((ab)= = (()))(b) for any a. b 0, we can evaluate the potential function at time L - 1,
A ((u1))2
6000 ) E ( 14
601 )(l
n5sca (4) ng mo) = - [0(
((+) - 16 ((2))) (9) nssa 1
-ngh
Here the expectation is over 8, and i, We perform Taylor-expansion on the term s01 - 1810 (4e)
( to deal with the non-linearity and use the fact that n is mean-zero:
(5e)
T. [61 (a (5) - (
x o(1) Φ (1)
9
2 (1)E
(186T (4)) To0""))
(1)) {
s <1) o' (1)
9
20 (()E)
< o(1) - (1)mg
9 - 20)2)).
(10)
"nn general, any mean-zero noise has a second-ordes effect on any potential function. Therefore, when the noise level is fixed, 7 + 0, the effect of the noise diminishes. This is why lower bound on the learning rate necessary for the noise play role.
7

---

Under review as a conference paper at ICLR 2021
In the second line we used (1) from the concavity and Elns; (5)] O, and the third line uses
the fact that & ~ [=0] and [ ( 2 1 (by the data assumption). The rest of the proof consists of bounding the second term in equation (II)) from above to show the potential function is contracting.
We first note for every 4, it holds that |o - || y 0z (1x000) 1& - <(1o11)3 + + r))l Furthermore, we can bound the &g norm of wltl with the following lemma:
Lemma 3.2. In the setting of Theorem H. for some failure probability > O, let bo & 6Td/P.
Then, with probability at least 1 - p/3. we have that [|ult] 2|2 I for any - To.
Note that v has E norm Tvd, and here we prove that the norm does not exceed rd with high probability. At the first glance, the lemma appears to be mostly auxiliary, but we note that it dis- tinguishes label noise from Gaussian noise, which empirically causes the parameter to blow up as shown in Figure II. The formal proof is deferred to Section B.
By Lemma 6.2 and the bound on lo in terms of ||u|*| we have 1øX S (bg + 1 <
Öbb + r) with by defined in Lemma B.2 (up logarithmic factors). Here we use again that each entry of the data is from (0,, 1). Plugging these into equation (II)) we obtain
E [1 n8 (4+) -not)) - ++ n00g + +r- - -2( < -(1)70
where in the last inequality we use the lower bound on 1) to conclude 1/² Z ojO +r). Therefore, summing equation (3) over all the dimensions shows that the potential function decreases exponen- tially fast: [[ < (1 ( (v)l)) After T = log(d)/(7''') iterations, o!7) will already converge to position such that E E[{(v)T))] & V1/d, which implies || S 1/d with probability at least 1 - P and finishes the proof.
3.3 STAGE 1: GETTING CLOSER TO V" WITH ANNEALED LEARNING RATE
Theorem B shows that the noise decreases the cc-norm of v to 1/d. This means that & or 22-norm of o similar to or smaller than that of v* if constant, and we are in small-norm region where overfitting is less likely to happen. In the next stage, we anneal the learning rate to slightly reduce the bias of the label noise and increase the contribution of the signal. Recall that v" sparse vector with support S C [d]. The following theorem shows that, after annealing the learning rate (from the order of 1822 to 1/5), SGD with label noise increases entries uy and decreases entries in ug simultaneously, provided that the initialization has foe -nom bounded by 1 1/d. (For simplicity and self conttinnaaissssss of the statement, we reset the time step to 0.)
Theorem 3.3. In the setling of Section 21 given targel error bound We run SGD with label noise (Algorithm W with initialization w/0) whose entries are all in [cming 1/d], where Emin 2 exp(--(1))- Let noise level 2 ((log(1/e)) and learning rate n- = e(1/6"), and number of iteratiion T = @(log(1/e\\/m) Then, with probability at least 0.99, after T iterations, we have
O, we assume that
110 - u3lloo I 0.1 and || 0 S 61.
(11)
We remark that even though the initialization is relatively small in this stage, the label noise still helps alleviate the reliance on small initialization. all (2017): Vaskevicius et all (2019) showed that GD converges to the ground truth with sufficiently small initialization, which is required to be smaller than target error E1. In contrast, our result shows that with label noise, the initialization does not need to depend on the target error, but only need to have an -oom bound on the order of 1/d. In other words, gets closer to 8² on both S and S in our case, whereas in ei (2017); Vaskevicuus.co et al (2 the Ug grows slowly.
The proof of this theorem balances the contribution of the gradient against that of the noise S and S. On S, the gradient provides a stronger signal than label noise, whereas on S, the implicit bias of the noise, similar the effect in Section [32 outweighs the gradient and reduces the entries zero. The analysis is more involved than that of Theorem 3.I. and we defer the full proof to Section I
Stage 2: Convergence to the ground-truth u*: The conclusion of Theorem B still allows constant
error in the support, namely, ||us 0i]l I| < 1/10. In Theorem D. we show that further annealing the learning rate will let the algorithm fully converge o² with any target error e.
8

---

Under review as a conference paper at ICLR 2021
Proof of Theorem 7 In Section B of Appendix, combine Theorem 1.11 Theorem Theorem to prove our main Theorem
3
and
4 CONCLUSION
In this work, study the implicit bias effect induced by noise. For quadratically-paaameeeiizz model, we thcoretically show that the parameter-dependent noise has a strong implicit bias, which help recover the sparse ground-truth from limited data. In comparison, shows that such a bias cannot be induced by spherical Gaussian noise. Our result explains the empirical observation that replacing mini-batch noise label noise with Gaussian noise usually leads to degradation in the generalization performance of deep models.
negative result
REFERENCES
D. Amelunxen, M. Lotz, M. B. McCoy, and J. A. Tropp. Living on the edge: Phase transitions in convex programs with random data. Information and Inference: Journal of the IMA, 3(3): 224-294, 2014.
R. Arora, P. Bartlett, P. Mianjy, and N. Srebro. Dropout: Explicit forms and capacity control. arXiv preprint urXiv:2003.0339) 2020.
S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, pages 7411-7422, 2019.
G. Blanc, N. Gupta, G. Valiant, and P. Valiant. Implicit regularization for deep neural networks driven by an orsteein-uhlenbeck like process. arXiv preprint wxXiv:1944000088 2019.
Bubeck, Eldan, and Lehec. Sampling from log-concave distribution with projected langevin monte carlo. Discrete & Computational Geometry, 59(4):757-783, 2018.
Chaudhari and S. Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In 2018 Information Theory and Applications Workshop (ITA), pages 1-10. IEEE, 2018.
X. Cheng, Yin, L. Bartlett, and M. Jordan. Stochastic gradient and langevin processes. arXiv preprint (90T:::9997.32222 2019.
Y. Chi, Y. M. Lu, and Y. Chen. Nonconvex optimization low-rank matrix factorization: An overview. IEEE Transactions on Signal Processing, 67(20):5239-522 2019.
L. Chizat and F. Bach. A note on lazy training in supervised differentiable programming. arXiv preprint ar arXiv:1812.078 8, 2018.
A. S. Dalalyan. Theoretical guarantees for approximate sampling from smooth and log-concave densities. Journal of the Royal Statistical Society: Series (Statistical Methodology). 79(3): 651-676, 2017.
S. S. Du, W. Hu, and J. D. Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Advances in Neural Information Processing Systems, pages 384- 395, 2018a.
Du, D. Lee, H. Li, L. Wang, and X. Zhai. Gradient descent finds global minima of deep neural networks. arXiv preprint arXiv: 181100804000 2018b.
R. Ge, F. Huang, C. Jin, and Y. Yuan. Escaping from saddle pointsonline stochastic gradient for tensor decomposition. In Conference on Learning Theory, pages 797-842, 2015.
R. Ge, J. D. Lee, and T. Ma. Matrix completion has no spurious local minimum. In Advances in Neural Information Processing Systems, pages 2973-2981, 2016.
B. Ghorbani, S. Mei, T. Misiakiewicz, and A. Montanari. Limitations of lazy training of two-layers neural network. In Advances in Neural Information Processing Systems, pages 2108-9118, 2019.

---

Under review as a conference paper at ICLR 2021
D. Gissin, S. drives generalization. arXiv preprint arXiv:1909 12051, 2019. 2019.
and A. Daniely. The implicit bias of depth: How incremental learning
P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet hour. arXiv preprint arXiv:1706.00 100n:/066.7777 2017.
S. Gunasekar, B. E. Woodworth, S. Bhojanapalli, B. Neyshabur, and N. Srebro. Implicit regular- ization in matrix factorization. In Advances in Neural Information Processing Systems, pages 6151 $151-6159, 2017.
Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. arXiv preprint arXiv:/802.08244/ 2018a.
S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in Neural Information Processing Systems, pages 9461- 2018,
W. Guo, H. Wei, Y-S.. Ong, J. R. Hervas, J. Zhao, H. Wang, and K. Zhang, Numerical analysis near singularities in rbf networks. The Journal of Machine Learning Research, 19(1):1-39, 2018.
M. Hardt, B. Recht, and Y. Singer. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint vrXiv:1509.011 2015.
E. Hoffer, I. Hubara, and D. Soudry. Train longer, generalize better: closing the generalization gap in large batch training of neural networks. In Advances in Neural Information Processing Systems, pages 1731-1741, 2017.
S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 2015.
A. Jacot, F. Gabriel, and C. Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, pages 5571-8580, 2018.
Z. Ji and M. Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv:1810.02033 2018a.
Z. and M. Telgarsky. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300 2018b.
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.0448 2016.
D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
Y. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller. Efficient backprop. In Neural networks: Tricks of the trade, pages 9-48. Springer, 2012.
J. Li, X. Luo, and M. Qiao. On generalization error bounds of noisy gradient methods for non-convex learning. arXiv preprint 1rXiv:1902.00621, 2019a.
Y. Li and Y. Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances Neural Information Processing Systems, pages 1157-8166, 2018.
Y. Li, T. Ma, and H. Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. arXiv preprint 1rxiv: 2017.
Y. Li. C. Wei, and T. Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, pages 11669-11688 1160, 2019h.
K. Lyu and J. Li. Gradient descent maximizes the margin homogeneous neural networks. arXiv preprint arXiv:
2019.
10

---

Under review as a conference paper at ICLR 2021
C. H. Martin and M. W. Mahoney. Implicit relf-regularization in deep neural networks: Evidence from random matrix theory and implications for learning. arXiv preprint arXiv:1810.01075. 2018.
Q Meng, S. Gong, W. Chen, Z.-M. Ma, and T.-Y. Liu. Dynamic of stochastic gradient descent with state-dependent noise. arXiv preprint arXiv:2006. 13719, 2020.
P. Mianjy and R. Arora. arXiv: 1887, 2019.
On dropout and nuclear norm regularization. arXiv preprint
P. Mianiy, R. Arora, and R. Vidal. On the implicit bias of dropout. arXiv preprint arXiv:1806.0977 2018.
W. Mou, L. Wang, X. Zhai, and K. Zheng. Generalization bounds of sgld for non-convex learning: Two theoretical viewpoints. arXiv preprint arXiv: 1707.0977, 2017.
W. Mou, L. Wang, X. Zhai, and K. Zheng. Generalization bounds sgld for non-convex learning:
Two theoretical viewpoints. In Conference on Learning Theory, pages 605-638, 2018.
M. Nacson, Lee, S. Gunasekar, P. H. Savarese, Srebro, and D. Soudry. Convergence of gradient descent on separable data, arXiv preprint arXiv:/803.0/9000 2018.
M. S. Nacson, Gunasekar, D. Lee, N. Srebro, and Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. arXiv preprint wrXiv:905.00332. 2019.
R. M. Neal et al. Meme using hamiltonian dynamics. Handbook of markov chain monte carlo, 2 (11):2, 2011.
A. Ncelakantan, L. Vilnis, Q. V. Le, I. Sutskever, L. Kaiser, K. Kurach, and J. Martens. Adding gradient noise improves learning for very deep networks. arXiv preprint 15 2015.
J. Negrea, M. Haghifam, G. K. Dziugaite, A. Khisti, and D. M. Roy. ioormation-theorede gen- eralization bounds for sgid via data-dependent estimates. In Advances Neural Information Processing Systems, pages 11013-11023, 2019.
B. Neyshabur, R. R. Salakhutdinov, and N. Srebro. Path-sgd: Path-normalized optimization in deep neural networks. In Advances in Neural Information Processing Systems, pages 2422-2430, 2015.
T. Poggio, K. Kawaguchi, Q. Liao, B. Miranda, I.. Rosasco, X. Boix, J. Hidary, and H. Mhaskar. The- ory of deep learning ii:: explaining the non-overfitting puzzle. arXiv preprint arXiv:/801 1801.0017, 2017.
M. Raginsky, A. Rakhlin, and M. Telgarsky. Non-convex learning via stochastic gradient langevin dynamics: nonasymptotic analysis. arXiv preprint arXiv: 1702.00/7 2017.
G. Raskutti, M. Wainwright, and B. Yu. Minimax-optimen rates for sparse additive models over kernel classes via convex programming. Journal of Machine Learning Research, 13(Fcb):389- 427, 2012.
N. Razin and N. Cohen. arXiv preprint arXiv:2005.06339, 2020.
Implicit regularization in deep learning may not be explainable by norms.
G. O. Roberts, R. L.. Tweedie, et al. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):34 1996.
C. J. Shallue, J. Lec, J. Antognini. J. Sohl-Dickstein, R. Frostig, and G. E. Dahl. Mcasuring the effects of data parallelism on neural network training. arXiv preprint arXiv:1811.03660 2018.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint XXXx::4409.1555 2014.
L. Smith, P.J. Kindermans, C. Ying, and Q. Le. Don't decay the learning rate, increase the batch size. arXiv preprint 171 2017.

---

Under review as a conference paper at ICLR 2021
D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-288 2878, 2018.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15 (1):1929-1958, 2014.
I. Sutskever, J. Martens, G. Dahl, and G. Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, pages 1 1139-1447 2013.
C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2826,, 2016.
Y. W. Teh, A. H. Thiery, and S. J. Vollmer. Consistency and fluctuations for stochastic gradient langevin dynamics. The Journal of Machine Learning Research, 17(1):193-225, 2016
R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series (Methodological), 58(1):267-288, 1996.
T. Vaskevicius, V. Kanade, and P. Rebeschini. i regularization for optimal sparse recovery. In Advances in Neural Information Processing Systems, pages 2968-2979, 2019.
S. Vaswani, F. Bach, and M. Schmidt. Fast and faster convergence of sgd for wer-parameterized models and an accelerated perceptron. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pages 1195 1204, 2019.
C. Wei, J. D. Lee, Q. Liu, and T. Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, pages 9709-9721, 2019.
C. Wei, S. Kakade, and T. Ma. The implicit and explicit regularization effects of dropout. arXiv preprint arXiv:2002. 12915, 2020.
M. Wei and D. J. Schwab. How noise affects the hessian spectrum in overparameterizzed neural networks. arXiv preprint arXiv: 19.000/99, 2019.
M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Pro- ceedings of the 28th international conference on machine learning (ICML-11), pages 681-688, 2011.
Y. Wen, K. Luk, M. Gazeau, G. Zhang, H. Chan, and J. Ba. Interplay between optimization and gen- eralization of stochastic gradient descent with covariance noise. arXiv preprint 1902.0822..444 2019.
A. C. Wilson, R. Roelofs, M. Stern, N. Srebro, and B. Recht. The marginal value of adaptive gradient methods in machine learning. In Advances in Neural Information Processing Systems, pages 4148 4158, 2017.
B. Woodworth, S. Gunasekar, D. Lee, E. Moroshko, Savarese, I. Golan, D. Soudry, and N. Sre- bro. Kerel and rich regimes in overparametrized models. arXiv preprint 2002:2000,.09277, 2020.
7. Xie, I. Sato, and M. Sugiyama. A diffusion theory for deep learning dynamics: Stochastic gradient descent escapes from sharp minima exponentially fast. arXiv preprint arXiv:2002.034999 2020.
S. Yaida. 10Xiv:1110.000044 2018
Pluctuation-dissiaation relations for stochastic gradient descent.
arXiv preprint
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.0335 2016.
Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynam- ics. arXiv preprint arXiv:1702.cm 055755 2017.
Z. Zhu, J. Wu, B. Yu, L. Wu, and J. Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. 2019.
12

---

Under review as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
A.1 EXPERIMENTAL DETAILS FOR THE
MODEL
In the experiment of our ansdraaically-paaameeeerin model, we use a 100-dimensional model with 40 data randomly sampled from N I100x 100). We set the first 5 dimensions of the ground- truth v" as 1, and the rest dimensions O. We always initialize with e.0) 1. We use a constant learning rate 0.01 for all the experiments except for label noise. For label noise, we start from 0.01 and then decay the learning rate by factor of 10 after 10 and 2 x 10 iterations. For "full batch" experiment, we run full batch gradient descent without noise. For "small batch" experiment, in order to fully disentangle the effect of learning rate and mini- mini-batch sgd noise (i.e., to avoid implicit biases from large Ir rather than noise), we add small batch noise to full gradient via the following sampling method: for each iteration, we randomly sample two data and j from [n]] and add ((Ve()) (v) (v)) to the full gradient (we set = 1.0 in our experiment). For label noise, we randomly sample E [n] and s E -6 (we set 5 = 1.0 in our experiment), and add noise V
(v) to full gradient, where ((0 _ ())(((())) - y(t) - s)². For Gaussian noise experiments, we add noise & (N, o²Iaxa) to full gradient every iteration, where the values of a are shown in Figure M. For experiments except for Gaussian noises, we train a total of x 10 iterations. For more generous comparison,
all the Gaussian noise experiments for 4 times longer (i.e., 1.2 x 10" iterations) while plotting them in the same figure after scaling the x-axis by a factor of 4. The test error measured by the square of le distance between 0 and + which is the same as the expectation of loss on freshly randomly sampled data. The trianing and test error are plotted in Figure II
A.2 EXPERIMENTAL DETAILS FOR DEEP NEURAL NETWORKS ON CIFAR100
We train a VGG19 model (Simonyan and Zisserman 20E on CIFARI00 100, using a small and large batch baseline. We also experiment with adding Gaussian noise to the parameters after every gradi- ent update well as adding label noise in the following manner: with some probability that depends on the current iteration count, we replace the original label with a randomly chosen one.
To add additional mean-zero noise to the gradient which simulates the effect of label noise in the regression setting, we compute a noisy gradient of the cross-entropy loss tee with respect to model output fz) as follows:
V
+ on
(12)
where is 100-dimensional vector (corresponding to each class) distributed according to Z and is the (possibly flipped) label. We backpropagate using this noisy gradi- ent when we compute the gradient of loss w.r.t. parameters for the updates. After tuning, we choose the initial label-flipping probability as 0.1, and reduce it by a factor of 0.5 every time the learning rate is annealed. We choose On such that din / / = 0.1, and also decrease Oin by a factor of 0.5 every time the learning rate annealed.
To add spherical Gaussian noise to the parameter every update, we simply set W + W + a2 after every gradient update, where 2 a mean- zero Gaussian whose coordinates are drawn independently from (0, 1). We tune this over the values shown in Figure II
We turn off weight decay and BatchNorm to isolate the regularization effects of just the noise alone. Standard data augmentation is still present in our runs. Our small batch baseline uses a batch size of 26, and our large batch baseline uses a batch size 256. runs where we add noise, the batch size is always 256. For all runs, we use initial learning rate of 0.004. We train for 410550 iterations (i.e., minibatches). annealing the learning rate by a factor of 0.1 at the 175950-th and 293250-th iteration. Our models take around 20 hours to train on single NVIDIA TitanXp GPU when the batch size is 256. The final performance gap between label noise or small minibatch training v.s. large batch or Gaussian noise is around 13% accuracy.
13

---

Under review as a conference paper at ICLR 2021
A.3 ADDITIONAL PLOTS
Here we show empirical evidence that training with Gaussian noise fails to converge to a stationary distribution. We train VGG19 network CIFARIOR and plot the of model weight along the training trajectory. As shown in Figure A the weigth norm of large batch (LB) and large batch with label noise (LB+LN) both converge some finite value, while the weight norm large batch with Gaussian noise (LB+GN) keeps increasing and fails to converge.
300
VGG19 Weight Norm
250
large batch (LB) lb, label noise (LB+LNN)
200
o= (LB+GN)
150
100
1.0
2.5
Iteration
1e5
Figure 2: The norm of model weight along training trajectory. We demonstrate that the model weight fails converge when training with Gaussian noise. In contrast, the weigth norm converges for training with label noise or without noise.
14

---

Under review as a conference paper at ICLR 2021
B PROOF OF THEOREM 3
In this section, we will first prove several lemmas on which the proof of Theorem is built upon. Then we will provide proof of Theorem [.T.
Definition B.1. (b-bounded coupling) Let v19,, o0I,... .... , be trajectory of label noise gradient descent with initialization ylo), We call the following random sequence Glt a b-bounded coupling of ult starting from e( y101,ft for each time t < T. if || I S b, we let ilt+1) a ult++ll; otherwise if ||600||, b we don't I update, i.e.,
b]++1 A (lt).
Lemma B.2. In the setting of Theorem B. assume ||ا00||
I be for any [n]. Let n S
67 where bo = 6rd Let Glt) be the by- bo-bunde coupling of wlth, If ifall is always positive on each dimension, then with probability at least 1 3. there
bo
(13)
Proof of Lemma B. Recall the update at f-th iteration is:
[[++1] -ld| 7(111102 - v 02) T I (((t)) @ alll 1782((t)) @ ald.
(14)
We first bound the increase of || in expectation. When || || bo, there is:
|]]
= o/ nBI( 11:0))00~~+00 x T ( 2
E 0
(15)
I 0/ nl 1000 11 + 0000.1 1620
(16)
S o + m66 + r
(17)
where the first inequality is because we can separate the last term into u/
part and 00 part and
bound them with [01022| and || respectively, the second inequality is by |o10]] S ||
and sparsity of o* So summing over all dimensions have E. [|]at["]]|] || nybn (bg
r). This bound is obviously also true when || || > by. in which case ū[++1] al9.
We then bound the probability of || being too large:
E
[|| 8 37
Pr
> bu)
(18)
bo
rd + T Tipbo
b3 (bg + +
(19)
0
(20)
3
where the first inequality is Markov Inequality, the second is by the previous equation, and the third by assumption of 17 and the definition of bn-
Proof of Lemma 12
Notice that when | < by. there is w.T) أاसا, Lemma
<
naturally
follows from Lemma B
Definition B.3. (b-bounded funntti)l For vector that is positive on each dimension, we define the b-bounded potential function ((v) as follows: if [|lll I b, we let D(v) & E Vuk: otherwise (vv) 0.
Lemma B.4. In the setting of Theorem
let 6D = l/d. Assume || S br for E [n] with p Assume nYbb + 1 + r)b S To log( Pvea ). Let Ti4 be the bo-bounded coupling of nl and 3d/d//
some by > O, (x (8) ) 2 for all k E [d]. Let bo = and 11822 > 32(t) + 6) and T 5
((.) is the by- by-bounded potential function. If glt) is always positive on each dimension, then with probability least G. there is
(ū(lT) S Ven-
(21)
15

---

Under review as a conference paper at ICLR 2021
Proof of Lemma 4
We first show ((ilt) decreases exponentially in expectation. If
||00 I bu.
we have:
E ((ll+))1)
[++1]
5
(22)
k
km
[Vol 1 1
(1)) 1/4 1n((600 - (
(1))
(23)
k=1
U/ Boeite
[V+ ns
(1:)
+ 176 + r)
(24)
(25)
where the second inequality is because || we notice that by Taylor expansion theorem, there for any general function g(x) vI +1+ I, there is
| 2 S by. Toward bounding the expectation,
M 6
g + S 91) + g')x
2 "1))=²
|=|2.
(26)
where M is upper bound on (1 (1 + 2 for x' in 0 to x, which is less than 3 if 2. So in our
|x|
theorem if A _ 1 ((t)
+ mb + rbb E [- 2 20 we have
1+ 1 (1+) + (6b + 6) $ A 8 A
1
(27)
Also since Es,,, [A] n69 + r)b2. Es,,ü 1
> 17²
B 1< (1)
2 37220, we have when
| s 1 and np02 2 32 + r) we have Exgs,
[V + A
S Es,.
18 A2 <
1 188.
So To
E (aletil) s 32
1
()00))((11))
(28)
Also notice that when
||ااع||| | > by there is (6 ( > 0, so obviously we have (ill+t))) < (1 7 x (old) always true.
Next we prove that (((r)) - Ee with probability more than 1 - f. This is because:
E [+(TT)))
Pr (((l)))
Veo
<
(29)
Vea
(30)
Veu
(31)
where the first inequality by Markov Inequaltiy, the second inequality is by the previous inequality. and the last inequality is because T = [, log( 3d/F
Proof of Theorem 1 Let p 0.01, co 1/d. By Lemma Gal and Lemma e when n
(log d), with probability at least 1 & there is || S b for all [r]] with some by
O( / [lo((dd)). and [e((x()))] Z 2 for all E [d].
Let bo 6rd We try to define 7 and 8 such that when T
32
log( 3d
pueeo 11. the assumptions 17 S
(7bg(63+T) and rit always being positive in Lemma B. and assumptions n6b, + (7663 + r) < 1 and n/8² > 32(63 + r)63 in Lemma B4 are satisfied.
Assume d 2 6x (b2 log 3d Pvro
then we only need 17 E
6x3262 pat
(6 + r) log
3dyT PV
32th
and then all the above assumptions are satisfied.
16

---

Under review as a conference paper at ICLR 2021
Let ūlt) be the by-bounded coupling of ult), According to Lemma least g. (()l)) S Veo. which means that either Σx
84
know with probability
vea or || bo-
According to Lemma 12 know with probability at most
5.
bo- Combining these
two statements, know with probability at least 3 |
bo and Σ k==1
| S by implies y! a while Σk=1 dimension k; so we ve finished the proof for the upper bound.
Notice that ||
[T]
v implies dy S co for all
We then give a lower bound for each dimension g(T), We bound the decrease of any dimension k at time t:
|++1| k
2 (1 n7 + + +))
(32)
2 2nj )ok
(33)
where the first inequality is by update rule and the second is because 8 > (bg + r). Putting in the value of T, we have
171 2 (1 - 27²))=
(34)
64 np (-18
(
3dvT
> exp
log(
(35)
PVT00
17

---

Under review as a conference paper at ICLR 2021
C PROOF OF THEOREM 3
In this section, we will first prove several lemmas on which the proof of Theorem 6 is built upon. Then will provide proof of Theorem 33
Definition C.1. ((b, ---oundddd coupling) Let ul0),,!A,, "..,,,..,,," ,.(T) be a trajectory of label noise gra- dient descent with initialization vl0), Recall S C. [d] is the support set of v², we notate T a r-dimensional vector composed with those dimensions in S of Gl9, and d the other d r dimen- sions. We call the following random sequence o.1)) a (b, c)-bonnded coupling of ult;; starting from g y each time < T, if 210
S Enddd
Us
S b. we let elt+1] & olit!l; otherwise
gie+1] git..
Lemma C.2. In the setting of Theorem 1.
let
100' 11
-
G
12
C2 2
10
maxjØ4 [Egla)
(0) Fk 11. Assume ||a""1| S for i E [ni] for some by > 0, and E
k
for k E [d]. Let b14 be a (1 + bibbundddd coupling of vl41, Assume 3n0
of
2 log GrT
(27 + r) Cb s oo and d 2 bo(e? + r). Then, with probability at least 1 there is
o 6
m s S + c1.
Proof of Lemma M2 For any fixed 1 < < l2 S T and dimension k E S, we consider the event that jilt E [1+ 3 1+ I),, and at time t2 it is the first time in the trajectory such that d(ta > 1+.. We first bound the probability of this event happens, i.e., the following quantity:
(aaall 1 + ok h1l < + Pr
ea 2
sttl
E [1 + 3 1l
(36)
where of 1 + C1.
altistal
E [1 + g, + ei] means that for all such that 4 < l2, there
+ 3 S 7!!l S
Notice that when [ S E and
| ll. S + ci and of! ||(+:1+]]
E [1 + 51 + ci). there is
Blug
[+1]]
1] E.,.. [ [1+ n
- ((60)022 - -00 2 ( ااجع - -1]
(37)
<(((0] -1)
[1]
2
i1 (of t)oo 1) + neac 03
(38)
3
<(1 ()!! - 1).
(39)
where the first inequality is because | 2
s ह and E, [<x 2 1.2 the second inequality is
because (Ef + C2 S 20. Also, we can bound the variance of this martingale as
[+1] 1| 8| 11] = Var a
(1r) [0
(1e) + Var ( 02 02) (b.) )x)
Var 1
U
(40)
< (188 (1 + c))) + ( + 2 + 22)
(41)
< 626.
(42)
where the first inequality is because x (1) (1))
is mean-zero, the second inequality is by ||a(
11 S
k
b. the third inequality is by 5 2 (27 +r).
By Lemma Ga we have
Pr(iltal
1t] 11>6
(43)
of
Se
25
(1-n)
(44)
3
(45)
where the first inequality is by Lemma G the second inequality is by taking the sum of denomina- tor.
18

---

Under review as a conference paper at ICLR 2021
Finally, we finish the proof with a union bound. Since if
> 1+ C1. the event in Equation ktc
has to happen for some l E S and 1 S t < t2 I T. so have
Pr
l+
(46)
Pr
[(tal k
> 1 A of
s 1
C1
A k
cttall
E E[1
3 1 all
(47)
2
BES 1th STaST
rT e
(48)
(49)
where the last incquality is by assumption.
Lemma C.3. In the setting of Lemma assume (2)+))Cg S 12T
Then, with probability at
least 1 &. there is
S E.
Proof of Lemma
0%
We first bound the increase of
(l0 in expectation. When
S + c
and
a10 "
|
$ E1, for any & $ S, there is:
[+1]
kk mE, (( 02
[1]
(1) (1) 02 x 20)
E k
(50)
S oh + nttT + Cr)
(51)
because can bound the dimensions in S and those not in S respectively, So summing over all dimensions not in S we have
[++]]
V1
+ +nle + Crbbb... This bound is obviously
also true when
and
We then bound the probability of T
being too large:
E
[ s
Pr
>
(52)
E1
+ T Tné (ET + C2
(53)
2
(54)
00
where the first inequality is Markov Inequality, the second is by less than 1/d, the third inequality is because El p
< 1 since every dimension
12 and (E + +r
127
Lemma C.4. In the setting of Lemma C.Z. assume & + C2 < 1/2
$
noa
< 4 T' 2 2
10 log
emen
and
8
T
2
20 log Then, for any & E S. with probability at least
6r
either
[T]
mTxisT
2 or
> 1 C₁, or U
Proof of Lemma 44 Fix & E S. Let glt
be the following coupling of alt); starting from a10) = -000
for each time T. [
S and (00
+1+a and a|! < g, we let alt++] A g]++1];
otherwise 6.++]] (1+ + eld Intuitively, whenever ald exceeds the proper range, we only times gld by + n afterwards, otherwise we let it be the same as gld
19

---

Under review as a conference paper at ICLR 2021
We first show that -/ log(111 + n) + log ty is a supermartingale, i.e., Bllog 1/
|++1|
| 2 log(1
2n) + log of. This is obviously true if u Otherwise, there is
> é or [3]]
> 1 + co or o > 2
Eloog of [
6l0] = Ellog
k
|++1|
| |
(55)
= Es log
1 + 178t - (1111/00 - -2 xx 2( + log 0/!1
(56)
3 zr(1 - -( n6? + c222)))) ]
2
2 Esg log
1 + 118 +
+ log o/
(57)
C1 17) - log a!!
2 log(1 (1 +
(58)
4
where the first inequality is by the update rule, the second inequality is because (E) + r)
C o and 4np2 s 2 and 5 2 की +r. So by Azuma inequality, we have
Pr (007)
9 2
(59)
To (1+ n) +lo sman 1ox(1- 4))²
<e
7(2na)"
(60)
7o 1og (1+ (1+ mn 25725
Se
(61)
Tel
Se 2982
(62)
P
(63)
6r
where the first inequality is because Azuma inequality and Var llg a((+1]] | [|1]] I (2n1))2 and the
second inequality is because T [1og(1 + 4n)) 2 2og فف which is true because Tn 2 16
log
Emmn the third inequality is because log(1 (1 + n) z @ 1,. the last inequality is because f
2 ci
log p
O
o
Lemma C.5. In the setting of Lemma C. assume +r S e and
2 log
6r7
Then,
8n6
for any le E S, with probability at least 1 - &. either max 0
[1] < 1 e 7 > 1 c1c
Proof of Lemma e55 For any fixed 1 S < la S T and dimension k S, we consider the event that jitto
0 1- - $1), and at time t2 it is the first time in the trajectory such that elfa] > 1 < ct. We first bound the probability of this event happens, i.c., the following quantity:
[1
61 A o ttt dal
E [1 10--a 11,1- (21).
Pr
< 1 -
2 1
(64)
2
where
dtt k
E
[1-
01,1 1 means that for all t such that t S < 12, there is 1 - S [11 <
1
Notice that when
[
S ē and
S + ei and o|! ][t
E [1 - -i, 1 - 41.
10c
B[1
o
1+
E &o
1
(1
n (7111102 B2 T ((((t) ((t) )m{!)
(65)
<(1 [
2
14 (جt!)
+ 1) - x/-)) + n( + Cr
n
(66)
3
<(1 n1 ((1))
(67)
where the first inequality is because
| 2
< 2 the second inequality is because (2+r)Ccbb3
e1 Also, we can bound the variance of this martingale as
20-
Var 1
10
< (2m1)
(68)
20

---

Under review as a conference paper at ICLR 2021
By Lemma
G
we have
Pr(1
k >
c1)
(69)
250 <e 20 2110
11 25
(70)
(71)
where the first inequality is by Lemma G. the second inequality is by taking the sum of denomina- tor.
Finally, we finish the proof with union bound. Since if m 0 1 2 but oz
[T]
1 C1
the event in Equation + has to happen for some 1 S t < l2 S T. so we have
Pr
max KT
11 & 1-
1
C A 2
<1
<1-a)
(72)
Σ
Pr (iibta)
< 1
A t
[4]
21 -
c1
^ u
E [1
c1, 1 1- (
(73)
2
18h ST
-
<T
e
(74)
P
(75)
6r
Definition C.6. (0, b)-buunded potential function) For vector that is positive on each dimension, we define the (b, b--bundeed potential function ((v) as follows: if [lulll S and ||us|loc S b, we
let ((v) A Ekgs Vuk; otherwise (bv) P O.
Lemma C.7. In the setting of Lemma assume 316
32( r)C,,2 and TT2282 2
16 log
66/2 p/a Then, with probability least
G. there is ((blTl) S Voi.
Proof of Lemma
We first show (ūilt) decreases exponentially in expectation. For any S
+ e1 and [
t t T, if
E1 we have:
E [(b((t]))]
5
IF.
k
1]
(76)
S
- 02) T I Vill + n [!1 1
(1)
Ex
Essit
(77)
[ /1 nsau +ntE? Cr
<
o E Eorit
(78)
S
< (1
1
2 (o(1)))
(79)
16"
where the second inequality is because || < 27, the last inequality is by Taylor expansion and 7102² > 32( + b... Also notice that when ||
> -1 or 10 > εi. there is
(((l++11)= p(tld)) O. obviously we have TB (+lt+1l))
S (1 ( always true.
(ū
21

---

Under review as a conference paper at ICLR 2021
Next we bound the probability of (((lTT) 1 VEI
Pr((((!1) > va)
(80)
v1
(81)
VEI
to 8 Vd
(82)
VEI
14
(83)
where the first inequality if by Markov Inequaltiy, the second inequality is by the previous inequality and initially ((olo) S dd, the third is by 1 - x S e for any æ E I, and the last inequality is by
TTp82 2 16 16log 6v
PVEE)
Proof of Theorem 3.3 Let = 0.01, C1 = 0.1, 61 A 12 C2 A maxf44 [E,la, ( 2 Let ba
V 2 log 30d2 = @(). According to Lemma GaI when n I d, there is with probability at least
1 15 we have ||~ I1 I by for i E [d].
Assume be positive number such that 16 82 log pemeny/ei S 1 and ò 2 (o + r). (since Emiin 2
Gvd
exp(-o(1)) this means 2 Ö(r + log(1/e1)) .) Let P 325 log T Q = 2og t. ?
min ( P. 16) = = T = 16 log 6v7
peminy = 6(log(1/aa/m)) Assume C + r) S
1782
} T(p/ log(1/ei)). (this means C, ((Flo((/5)) ).) 18 127
min
We show the assumptions in the previous lemmas are all satisfied. The assumption 8y,5112 > TT
log in Lemma C is satisfied by
GrT2
2
log
(84)
8188222
p
2
6r
1
log
+
4
log
(85)
8
p
1
1
dt
17 log n
<
P.
(86)
325213 log
where the first is by T < #. the second is by log 4 1 S 4 true because
or log 4. and the last line is
p
P
Q
77 mllgg
<
log
(87)
Q
P
log
log 1/P
P(
(88)
Q
Q
< P.
(89)
The assumption 8 2 (2] + rr Lemma + is satisfied by definition of 8. The assumption
(27 + C. < in Lemma C is satisfied by
C. +r)
n62
<
2
<
9
(90)
48
96
20'
where we use
P <
2
n < 8
(91)
Q
2
22

---

6r Under review as a conference paper at ICLR 2021
The assumption
(r)+r)
1277 in Lemma o% is satisfied by assumption of Cz.
The assumption
f 2
2" 1
log in Lemma is satisfied by
T
16
6r
26
6r
29
log
log
log
(92)
8
P
The other two assumptions (o) + r Cr
<
and no²
Lemma C4 follows from
12
C2b < and no² < The assumption T' 2
n
16 log in Lemma HZ is satisfied by
18
Emin
the definition of
The assumptions (7) + r) <
and
2 log G Lemma
M are satisfied by the same
20
Suyz
reason as that of Lemma en The assumption 2 Z 16 16log
6v
in Lemma ew/ is satisfied
PVei
by the definition of T, the assumption 3716² > 32 C2 Lemma definition of Cs.
satisfied by the
Since data are randomly from No 7), with 2 Ei( ( )2) data, there is with probability at nda 25 1277
P
least 1 1o there is C2 ( (éf +r) S min
Meanwhile, according
Lemma G22 with 2 @11) data with probability least fs there [=((<=])]] 2 for all h E [d]. According to definition of bæ, we know when n S d, with probability at least 1 - is
there is also ||a.00 S b, all i [n]. In summary, with d 2 n 2 Ô(( (" (og(1/e)) )²) data, with rd 48 127 and [< (0)1 2 3 for all
probability at least 1 & there is C,b + +r) I min
[d] and || ||.o s by for all i E [n].
Now we use these lemmas to finish the proof of the theorem. Let rlt) be a (1 + C1 61 (1)-bounded coupling of vlt), we only need to prove with probability at least 1 - p. there is 0
11 S c1 S 61. which follows from union bound of the previous propositions. In particular,
and
Lemma M and Lemma o% tell us that probability of [ Il 1
& is ci is most f. Lemma C tells us the probability of
[T
at most 3. Lemma ME and Lemma
M tell for any k S, probability of 0%
1 + ci and [T]
and
é and
+
cj and
S E is at most &. Combining them together tells
us that probability of
or
> El is at most p.
23

---

Under review as a conference paper at ICLR 2021
D PROOF OF CONVERGENCE TO GROUND TRUTH
The conclusion of Theorem S still allows constant error in the support, namely, 1/10. The following theorem shows that further annealing the learning rate will let the algorithm fully converge to ot with any target error. The end-to-end proof of the convergence to ground truth can be found in the next section (Section (D).
- I 12 S
Theorem D.1. Lets 2 2 be the index of the current round of bootstrapping. Let constant Co 1/10.
In the setting of Theorem R.I. assume v) is an initial parameter satisfying [ - o5lloo S Ca-1
Assume 2 @(r²). Suppose run SGD with label noise with noise level 5 2 0 and learning rate n - (622/(82 +) =log4// log(4//o)) iterations. Then, with probability at least 1 p over the randomness of the algorithm and data, there is llug -F'll
and l/u5 [0]
-vgl11 S Ex-b, where 0 < 6s-1 S Cg-1 S cg- Given a failure rate p
121 -lug"'='lll S Scg - Cs-100 and Ilug rggll S
6s - (4/co) 2e, -e Here ((.) omits poly logarithmic dependency on p.
In the rest of this section, we will first prove several lemmas on which the proof of Theorem DJI is built upon. Then we will provide a proof of Theorem D.I.
Definition D.2. ((b,e)-tov"' coupling) Let e!0), -oll.... (e(T) be trajectory of label noise gradient descent with initialization !00), Recall S C [d] is the support set of v². We call the following random sequence 814 a (b, c)-to-u* coupling of ult); starting from ((9) = =l0), for each time t < T. if
S e eand
u - 1 I b, we let ilt+1] a olt+!l; otherwise e[(+1] _ al.)
Lemma D.3. In the setting of Theorem 1 let Cs max [E (2) (0 Assume | 11 S be
for i E [n] for some by > 0, and E, E. ) 2 3/2 for k E [d]. Ler old be (2c,-1,6,)-10) log 10TT" and (ca 4rcs-1)C.bb 10"
coupling of ult, Assume 27,5 (6 40
2.
c--1 Then,
with probability at least 1 1- G. there is
- Us
-
< 2c,-1
Proof of Lemma D..1 For any fixed 1 S 11 < 12 S T and dimension k S. we consider the event that gl6]l E [L + fc.-1, + c.- and at time 22 it is the first time in the trajectory such that
g(tal > 1 1 20,-1. We first bound the probability of this event happens, i.e., the following quantity:
[tal Pr
kk
- 1 > 2c A %
[66]]
1 S c.-1 ifff
2 dal E [1 + co-11 22.-
(93)
where eflcsta] 1/2
2 GC 1 2c, means that for all /. such that 1 S < 12, there is 1+ 30 S [1+
< 1 + 2c
Notice that when there is
11 S cs and [ng -1
S 20 and o
[1
E [1 + fc--1111 1 + 20
s
1| -Bg (1 + +n - (il11 02 022 T (((
x (14) (((i))a(())
)o -1] 1
E
(94)
<(ul!) 14 - -1)
3
2
non
14 (ol)) 1)(=( 1) + +n(e + Cr,
(95)
<(1 (1!
-1).
(96)
2 |لا5 ||:
where the first incquality is because
e and properties of the data, the second incquality is
because (e3 4rrc, 4rrc-1)633C Cb,CC, V 200 Also, can bound the variance of this martingale
Var (/ 1| إlt| -1] = Var [15
((1 k
+ Var (2((6)))2 w 022) (
x )a (1))
(97)
< (106x + 2c)--1)²² 1))² + T
2) + 2ca-1)²
(98)
s 10 + +b2 +r)
(99)
24

---

Under review as a conference paper at ICLR 2021
By Lemma G we have
Pr(#k t2l > 20s-1 1 A a1li] -1 S cs- A A allicaal E e + z 1 + +20--1)
(100)
-0
2 122 (n) 11- 1)a1
(101)
<e 1 1i 1975
(102)
where the first inequality is by Lemma F the second inequality is by taking the sum of denomina- tor.
Similarly, we bound
Pr 1 - T|(aa -- > 20 A1 ik
I c
A i
a(t1tta E E 2c,-1111 113--11))
k
11 1.1
2 3o.-l).
(103)
Notice that when | || S c, and pog) - 1
I 2ca11 and of s|1:tt11]
E [1 - 2c0-1, 1
2
3
cc--1l.
there is
||
1 - (1 + + - n, ,+ x (5))
)z (8t) 501 )11
B8 k
= E =B
(104)
5(1 o(!))
non S (o 1)(1 - - + nn(e + 4rc, 02
(105)
<(1 (1 )
(106)
where the first inequality is because
ca and the properties of data, the second inequality is
2
because (c. + Dr S 10 So
Pr(1
[tta| > 20 Al 1
S cc-1
2 ot offistal E 2c
(c--1))
(107)
3
Se ( +b? (r3+))
(108)
Se
(109)
where the first inequality is by Lemma GZ the second inequality is by taking the sum of denomina- tor.
Finally, we finish the proof with a union bound. Since |005 - 1/| Equation g3l or in Equation IIE has to happen for some à E S and S
> 2c,-11 either event in
S T, have
Pr 1
1/1
20a-1)
(110)
Σ
Pr
[122l Uk
1 >>>> 2c
c
[tatta) oh
2 E + 3 --- 11 + 2c
keS 18h&tET
(111)
Σ +
Pr Pr(11-1111 (1 > 20 A - aftial
-2fff/ s c ^ of oltiital
E(1 - 20
2
cc--1)
3
KES 15
(112)
<2rT 2nd2 (3
(113)
(114)
5'
where the first inequality is by union bound, the second inequality is by previous results, the third inequality is by assumption of this lemma.
25

---

Under review as a conference paper at ICLR 2021
Lemma D.4. In the setting of Lemma DE assume (c3
4cg-1r) Cs-1, Co > (1
( 1--1 and (0+-0,-11) Fe 6 27 (021111 2 log 5
Then, with probability at least P 5' there
2 8 S Ex.
I 20 and 1 ||, 6.. for any ls $ S, there is:
Proof of Lemma D.4
When
-1
[+++] 1]]
0161 n,, (1011)02 - -b022 x111)) 260
E
(115)
0/ + n 1002
11 + 4c, 1,,
(116)
g
- 0k + n + 4c,- 1c)-1r)C2
(117)
S (1 nc--1)111 )=o²
(118)
where the first inequality is because we can bound the dimensions in S and those not in S with
C2 and 4c,-1rCzb2 respectively, the second inequality is by s W
third is because (e + Cc. I Ca-1- Summing over all k & S we have E[] (1 + 1))
This bound is obviously also true when -10 in which case a(++1) glt)
[
U
1
20
Ex.
is supermartingale. Also notice Therefore we know (1 10--1)- || ||0|||
|++
1++1] ||,11 S sdee, By Azuma Inequality,
E
Pr
c.,)
(119)
((++e---1) Te )2
Se 20 13607 1911
(120)
z P
(121)
4'
here we are using e, (1 + nc.-1) 1) E by assumption and the last step by assumption.
Lemma D.5. In the setting of Lemma
assume (1 - 11) 2c,-1 <
6a 2 2nd
Z
log p and
(e. (4. 1 < 10 Then, for any
E
S, with probability at least 1 - $. either mink [-
1 2 2, or
1 - 2c,-1. or
12
c.
Proof of Lemma D.5
We first consider when [
E [1 + 2, 1,11 + 2c,-1l. For some < Tz if
1
< C.- and
Us
E.. there is
Bluk [++1]
1]
(122)
- -o(f 1)EE, [((T]]] 0. 002) = (a (or( -1
(123)
<(a(i) - 1) not (o( + +1) - 1) + nnc + 4c, 4c.-
(124)
<(1 - (v 1).
(125)
Here the first inequality is by assumption, the second inequality is because (e) + 4c 4c,-1r)C,63, TO C. and C S To
<
26

---

Under review as a conference paper at ICLR 2021
We define the event Eg as
I1 1 11.00 2c or
>
Es. Since (1 - (1-1))=22 720=2
< S by
2
assumption, if 2k
[0]
E
[1+
f. 1 + 20,-1], by Lemma G2
we know:
Pr
min 1ST
[0]
1 +
Co 2
o
-
22 A u5 Sea)
(126)
00
(zc P
(127)
2nd (62 (1)))
(128)
VA
(129)
5r
where the second inequality is because of assumption.
Similarly, when 0
E
2c,-1, 2 ]. there is
Pr
max TST
[4| ke
1
Co 2
is
1/
S
11105 27 $2cg-11
(130)
S
(131)
< 5r
(132)
Since 106 1| < C.-1 by assumption of Theorem DJ by bounding the probability for of
[1+ + 2c, 1] and [1 2c,-1.1 2 repectively we finished the proof.
Lemma D.6. In the setting of Lemma D assume
log 5 and (e%
Crb < 20- Then, for any dimension k S. with probability at most f. there is
mingsT 10 1 s Lec and log [T] 1| > C. and
[2] igg -1 S 2c and [
cs.
Proof of Lemma DIA For any fixed 1 f < 12 S T, we consider the event that R|4al [1 C + jca), and at time l2 it is the first time in the trajectory such that jltal 1+1 C.. We first bound the probability of this event happens, i.e., the following quantity:
tal k
1 1
uk stt
1 E [1 + Co. 1 + col
Pr
1 c.
A b
2 co
3
(133)
[ ||
S c, and |u/5 - -1|
Notice that when
2c and of
E [1 + cc, 1 + ca). there is
1] = (1 + n8aath - 17Ti, [1)10 - 0002) 2((t) jork (2)) 1)o - (134)
<(0 [c] -1)
2
-1t 1 (o 1)(T(() 1) + tn(e? 2rc, Cr
(135)
3
<(1 (o!! 1).
(136)
| 22 < 2. the second inequality because (c3 +
where the first inequality is because
2rc,) C2 < 20. Also, we can bound the variance of this martingale as
[[+1]] -1 1 rlع -1 1 = Var 1 Var 1/
(1) [|4 + Var ((((i))) 62 - 0 02) ( (()))
(137)
k
(nodb (1+ cz)) +r) coa²
(138)
10 + +3,( + }))
(139)
27

---

Under review as a conference paper at ICLR 2021
By Lemma
G we have
Pr(vk [ta
1>> C2 k - - 1
offirtal
1 E E11 3 + csl)
2Cs
(140)
Se 0
01-1a
(141)
Se Se
(142)
where the first inequality is by Lemma the second inequality is by taking the sum of denomina-
tor.
Similarly, we bound
Pr(1 - U/ > Co A - oftal S 20 A o/
1ta
hhuttal E [1 cs
1
(143)
S e 2n3 (1 (113+))))
(144)
Finally, we finish the proof with a union bound:
Pr
min
-1 2 c A
-1
- 1|
es
A
[T
-1
s 2c, A bog || sca)
(145)
Σ
Pr(ut k 1>c A th
-15
1
70
A ih
E [1 + Jo, 1 + c.))
(146)
2
Σ
Pr(1 - p > c ^ -
-
-
altal
- 2
A k
tcttll
E - C, 1 1
3 gcl)
(147)
154
< T And ( (2+2222222))))
(148)
P
(149)
5r
where the first inequality is by union bound, the second inequality is by previous results, the third inequality is by assumption of this lemma.
L
Proof of Theorem DJ Let Cz A maxjœe EB,,, (1) 200 by = 2 log 30d
²(1). According to
Lemma G when < d, there is with probability at least 1 - fo we have || 11.0 S b2 for i E [d].
Set 1) small enough such that sm (.C
2 log 10rT² Obviously only need n s
Ô (24 ), where (..) omits poly logarithmic dependency on d and p. Assume C,, <
e which can be represented as C2 @(t). Recall T 1
log oo c, = 20 ec.- log c
We first show that the additional assumptions in the previous lemmas are satisfied. There is
(1 7)--11" =²= (1 + (]0 log
(150)
< c- log A
(151)
The assumption 6, > (1 (1, T 6-- in Lemma D is therefore satisfied by definition of of The
((1 tn, ) T,,-,, ) log in Lemma D4 is satisfied because:
assumption 27 26
(11 + 7c nc- 1) Es 12 2
2P-11 _ --21-
(152)
2T + 66 +
27 2 (82 (6" 12 (2 +2
(P-1) - 1)2
(153)
27 (82 12 (c +r))
o2_1
(154)
27 62 (2 + r))
28

---

Under review as a conference paper at ICLR 2021
which is larger than log p satisfied because (1-n))²ee S (2) to 2c,-1 = S. All the other assumptions in Lemma Lemma D.4, Lemma and Lemma D.6 naturally follows from the definition of and the require- ment of Cz.
by the definition of 17. The assumption (1 (1-n))2c.-1 S t in Lemma DE
Since data are randomly from M/0, I), with Ō()²) data, there with probability least 15 there is (c. + 4cc-1r) I TO Meanwhile, according to Lemma ST with + data with probability at least 1 - 13 there is [x (6) & ) 2 3/2 for all k E [d]. According to definition of bg. we know when I d, with probability least is there is also || I by for all i [n]. In summary, with d 2 n 2 ()²²) data, with probability at least 1 - there is (e² 4c.- S it and E,((2)
(1) y1 2v 3" for all E [d] and || - by for all E [n]]_
Now finish the proof with the above lemmas. Lemma D3 and Lemma D4 together tell us that 117 5
with probability at least - 2.f there is
20 and
Cs. in which case
there is also اरर ااर" by the definition of न|र Lemma D and Lemma
D
together tell us
the probability of [ 1/
2c,-1 and [7]
I ex and
is no more
than 2g. So together know with probability least 1 p. there is
[T S
I C. and
77
S c.
29

---

Under review as a conference paper at ICLR 2021
E PROOF OF THEOREM 2.
Proof of Thheorem 2. Starting from initialization 7 1, by Theorem B.I. running SGD with label noise with noise level 8 > O(=22) and Y/o x for To = (1) iterations gives us that with
probability least 0.99, Emain
Uk
[bb
where Emin exp( Now ylPol satisfies the
initial condition of Theorem
Recall the final target precision is e, Ej T0... e. By Theorem with Ö(7 log (1/e)) data, after running SGD with label noise with learning rate / = O gz for Ti O log(1/c) iterations, 71
with probability at least 0.99, there is,
1o+ 5
S A 1
U/S
(155)
10'
and
3
[To
03
S c1
(156)
So we have 2 satisfies the initial condition of Theorem
Finally, set 0.01 / 1oo (1/e)), and apply Theorem D.Il for [log1o (1 [/)e] = ((1) rounds. Since cs gets smaller by 1/10 for each round, the final Cn2 satisfies toe S Cn. S e. Since the requirement of for round s is < θ (ج we can set 722 I Ō to satisfy all the rounds at the same time. Set T2 be the total number of iterations all of these rounds, obviously T2 θ (1))
Notice that S - 2n 1 log o0 < 40
we have with probability at least 0.99.
To++++72)
Sc _e,
(157)
and
To To+
v/s
||| S.
(158)
The total failure rate of above three stages is 0.03, with probability at least 0.97, there is Io - w* lloc which finishes the proof.
30

---

Under review as a conference paper at ICLR 2021
F PROOF OF THEOREM 2.2
Lemma F.1. Assume n. S 1 9vd. Let C C TRd be the convex cone where each coordinate is
positive, K be random subspace of dimension - n. Then with probability at least 0.999, there is K k
Proof of Lemma E. By Theorem of Amelunxen at (2014). we only need prove
8(C) + ((K) 2 d 9
(159)
where 6() is the statistical dimension of a set. By equation (2.1) of Amelunxen et (2014), 4), there
8K)) n.
(160)
To calculate 8(C), we use Proposition 2.4 from Amclunxen al (2014).
8(C) = [[]]
(161)
where g is standard random vector, IIc is projection of to C, the expectation is g. Since C the set all points with element- wwse positive coordinate, [g) is simply setting all the negative dimension of g to 0 and keep the positive ones. Therefore,
8(C) E[ (g) d
(162)
2
Therefore we have
3 +(C)+5(K) dd -d d 49d.
(163)
Proof of Theorem 22 Let X be the subspace that is orthogonal the subspace X spanned by
data. Since data is random, with probability 1 the random subspace X is of n dimension. Therefore, according to the previous lemma, with probability at least 0.999, there is X n C y (0). where C is the coordinate-wise positive cone. Let E X be such vector such that μ4 > 0 for Vi E [d]. and we scale such that || 1. We can construct the following orthonormal matrix
A [a a² E Rddd
(164)
such that spanta",... ...a""...""))........ a")
X and a11 μ. Consider the following transformation
Aü = 02
(165)
since only the projection of to the span of data influences L(v), we can write L(v) L(ū as a function the first n dimensions of u.
We can lower bound the partition function with
-A
(166)
v20
(L(ārm) det dr
du det dü
(167)
Ai20
Ou
(1
2d e -Ab((i)))
du dun+i:d Vu:
dug
(168)
i=1
Here the inner loop is integrating over the last d - n dimensions of ū in the set such that Au is coordinate-wise positive. Now we prove that for each ūin such that S {ū 0} is empty set, the inner loop integral is always too.
Fix Uime let u 2 be one possible solution such that u² Au 0. Define constant
a," n+1
min [cd]
max
(169)
j
(d
1)
31

---

Under review as a conference paper at ICLR 2021
can define the following set
S' = 1id #1+1 2 1 A ū, -ū] S c(ūu+1 "i [n + 2, d]}
(170)
In other words, this is convex cone where constraint of ū is linear in in++1 for [n + 2 d.. By definition of c, it is easy to verify that is a subset of Also, for every un+iid E S", u, is upper bounded by
(A)
Tu4tid u1:n 1),
u. a, 10++1
(ūn+1 a.p++((@++
- xx+1) +
Σ
(ū, 8j)
(171)
j=n+2
S un + 2a (un+1 (ix++ (x+1).
(172)
Here the inequality is because of the definition of c.
Let Un +1 have
J
di
(173)
Rn L.EES
1=1
(20z) (202)a-n-1
1
de
(174)
=20
u1 2a
+ 00.
(175)
Here the last step is because < d/2. so the integrand is essentially a polynomial of with degree
larger than -1, so integrating it over all positive z has to be +oo. So we finish the proof that SOER e -- du = +00.
32

---

Under review as a conference paper at ICLR 2021
G EXTRA LEMMAS
Lemma G.1. Suppose (: ~ (O, Iaxd) where i E [m]] are random data. Then with probability at least 1 P. for every i [n] there is
| $ 2 log
2nd
(176)
Proof. By Gaussian tail bound, there is Pr (Ir& > bz)
S 2e 22-
So by union bound we have
Pr & | bz) S 2nde 1 Let by = - 2nd
we finished the proof.
Lemma G.2. Suppose x0 NO (axa) where i E [ri] are random data. Then when > 24 log d with probability at least 1 P. for every k E [d] there
E
(177)
3
Proof. Since [x 1, [x
3, by Hoeffdding inequality we have
Pr
(178)
Therefore, when 24 4, by union bound we finish the proof.
Lemma G.3. Let > 0, 1 y > 0 be real constants. Let A A ... A be series of ran-
dom variables, such that given Alol, ..... ,Al for some t T with Al E ElG.c) c). there is either A
A ||+1] = -... ATT or [[A [++ S (1 >)lA fl with variance Var[A [+]] A A S
Then there is
2 20 (1-333
Pr A441 A A - A [ (
(179)
where A 10:77] E [ means for any 0 < t T, A [j.
We only need to consider when A < 5. Let Åld be the following coupling of All: starting from Aloj = A for each time 1 < T, if Ald = Al++1] ... = A or there is t' < t such that Ald"l $ cj. we let Al++]] A (1 (1- otherwise Ål++]] = A|(+1] Intuitively, whenever Ald stops updating exceeds proper range, only times Áll by 1 7 afterwards, otherwise let it be the same as A Notice that if the event in Equation I7 happens, there has to be A AT) (otherwise A stops updating exceeds range at some time, contradicting the event). So we only need to bound Pr (A[T) > c).
Proof of Lemma
We notice that (1 (1 -tl for . B|A[++I|J < (1 - A). This obviously true when g[t+1| = (1 - A', and also true otherwise by assumption of the lemma. So we have
T' is supermartingale, i.c., given history there is
Pr(A(T) >
(180)
Pr((1 - -TA 1 > (1 - -T
(181)
2(c(1 7)- Alol)²
<e
E1 (1-3) 24.
(182)
-(1-)) <e 25 1 2(1-))
(183)
2a >0>-<00 (1-7)21
(184)
33

---

Under review as a conference paper at ICLR 2021
where the first inequality is because of Azuma Inequality and Var (1 --- [
A
(1 - (1-7) --)))at+a)a a. the second inequality is because A Since the event in Equation 179 only happens when Allll e, we've finished the proof.
Lemma G.4. Let 0 < c1 < C2 be real constants. Let A ..... ,AT,, be a series of random variables. such that given A _10).... Alt for some < T with Alt [c,, 02), there is either event E, happens, [[[
(1 7) All with variance Var[A A Alo] S Then when
A E [er, c2) and - --)"
TT 02 < c1 there is
-(100)))~~110)
Pr min A
Ci A max A
S -BOoT
S
7a
(185)
KT
where -bior means for any St E, doesn't happen.
Proof of Lemma G4 Let Ald be the following coupling of Alti; starting from Á All, for each time t < T, if exists t' < tsuch that Ep happens or Alr"l & [c1, let [ (1 ))+ otherwise A | Intuitively, whenever A exceeds proper range, we only times Åld by 7 afterwards, otherwise we let be the same as Alt), Notice that if the event in Equation happens, there has to A = A (otherwise E; happens sometimes or Al & c2), contradicting the event). So only need bound Pr (ALT) e1).
T is a supermartingale, i.e., given history there is =[A+++I| s (1 - Alt) This is obviously true when o[t+1] = (1 - A) and also true otherwise by assumption of the lemma. So we have
We notice that (1 - -(() for t = .
Pr(A > e))
(186)
Pr((1 - > (1 7T)
(187)
[(1101))---100)
Se
Ei (1 (1-))-1-
(188)
2(1:01=))--A00))
<e
4"
(189)
where the first inequality is because of Azuma Inequality and Var (1 (1-7)- 100--(+1000 A[++1 [+++))+)++++
A
<
(1-7)-2(++1), and A < Se e
2
a-1 2ie
Since the event in Equation IS5 only happens
when AlT] c1, 'e finished the proof.
34


Reviews:
[{"review_id": "klJv4DqW8t7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors study the problem quadratically parameterized (linear) regression and study the behavior of SGD to solve it when the stochasticity added is in terms of label noise. They show that SGD (with this kind of noise) with arbitrarily large initialization converges to the ground truth solution, whereas there exist settings where Langevin dynamics or gradient descent would fail to converge to this solution. The proofs are carried out carefully and are correct as best as I could verify. The authors also provide experiments on synthetic data to support and motivate their theory.\n\n\nSome questions/comments:\n1. Why are three stages with decreasing step-sizes needed for the analysis? Can we expect a similar result to hold if a constant, but, small step-size is chosen instead?\n2. The generative model assumes that the label y has no added noise. Are the theoretical result robust to additive noise?\n3. The authors point out on page 4 that the sample complexity is worse that LASSO (which is fine), but, do not remark why this is the case. It would be insightful if the authors could add a comment about why this is the case, and if this was experimentally observed as well.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MNr1-ET5uw1", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the implicit regularization of stochastic gradient decent (SGD). The authors analyze SGD with label nose in the quadratically-parameterized model and prove that it converges to the sparse ground-truth even if started with large initialization. The authors also prove that SGD with Gaussian noise (Langevin dynamics) does not converge to the ground truth at zero under the overparameterized regime.\n\nThis is a solid contribution with theoretical insights on SGD with label noise. While the theoretical results are deep with long proofs, their outlines and meanings are well explained.\n\n- The difference (affinity) between the deep neural network and the quadratically-parameterized model is mainly discussed numerically (Figure 1). It would be nicer to discuss gaps between them theoretically, if possible. For example, the contraction of SGD in the initial phase (Thm 3.1) is reminiscent of the effect of singularities in neural networks discussed in references such as:\nGuo et al.: Numerical Analysis near Singularities in RBF networks, JMLR, 19(2018), 1-39.\nIt would be nicer to discuss if this analysis of the initial phase has something to do with singularities.\n\n- p.7, footnote 7: What is the second-order effect (of zero-mean noise)?\n\nMinor:\np.4, Langevin dynamics/diffusion: The last sentence is duplicated.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "L9Y_gpxuSFx", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper demonstrates that for a particular model SGD with label noise and proper learning rate schedule recovers the (sparse) data generating model while GD with or without Gaussian noise does not. In the latter case, it fails because a stationary distribution is not achieved. The proofs in the appendix are quite involved, and as a result I did not carefully study them. But the authors provide helpful intuition for the results in the main text. \n\nWhile I think there is value in the work, I am not sure whether the fairly specific setting studied has much to say about neural networks. (Of course, not every paper needs to be about neural networks, but that's certainly the motivation of the paper.) There are a number of aspects to the work that limit its generality: the model, the label noise, the objective of reconstruction of sparse ground truth from the same model class, and the dataset modeling. The authors justify the model by saying that other works that have studied it, but don't otherwise try to justify its relevance. In Figure 1, the authors argue that label noise behaves similar to SGD, but I don't find this thorough or convincing enough that any results on label noise and the mechanism by which it operates should generalize to SGD. Also, whether studying an objective of learning as identifying the sparse ground truth model seems far from standard training of a neural network. As an example of my concern that the specifics by which the results are achieved may not apply in other scenarios, do the authors believe that GD with Gaussian noise fails for neural networks because a stationary distribution is never achieved, whereas it is achieved for SGD?\n\nOverall, I think that the work is a bit too narrow and doesn't change our understanding of what non-spherical noise from SGD does to neural networks beyond what is already known.\n\n\nMinor presentation points:\n\nThe paper became rushed at the end, as if the authors ran out of space. Subsections 3.2 \"Stage 0\" and 3.3 \"Stage 1\" are then followed by a very short paragraph inside 3.3 titled \"Stage 2\". \n\nFigure 1 is fairly difficult to parse with the number of noisy curves overlapping each other. Perhaps the authors could make their point with the minimal amount of experimental data here and relegate the rest to the appendix.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GvpjwkH8wNq", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Problem\n\nThis paper considers the effect of label noise on stochastic gradient descent. The setup is that there is a vector $v \\in R^d$. We observe samples from $v^2\\cdot x$. We only have $n < d$ samples but $v$ is $r$-sparse for $r < n < d$ which makes recovery possible information theoretically. The main result is that stochastic gradient descent with label noise, and without any explicit regularization will recover the ground truth. whereas adding spherical Gaussian noise does not.\n\n### Pros and Cons\n\nThe problem is a clean toy problem with which to illustrate the gap between algorithms. It shows a clean separation between the power of label noise and that of random Gaussian noise. The model appears to be the simplest model where one can hope to see the regularization effects of noise (the simpler linear regression model wouldnt show these effects).  One possible criticism could be to ask if understanding this model is truly getting us closer to understanding what happens in deep nets. At this point it is hard to say, but proving such a result even in this simple model is not trivial, and is definitely a contribution.\n\n\n### Evaluation\nI think this is a solid theoretical contribution on an important problem, and the paper should be accepted.\n\n\n### Further comments\n\nI was a little confused by the comment that the coefficients are assumed to be in ${0,1}$ since they then satisfy $v_i^2 = v_i$ as this seems to linearize the model. The authors should probably clarify that this is actually not what is going on. It might be better to use a different setting of parameters even for exposition. \n\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that for a simple nonlinear (quadratically parametrized linear) model, stochastic gradient descent (SGD) with a certain label noise and learning rate schedule recovers the data generating model. In contrast, gradient descent with or without Gaussian noise fails. While the results are novel and interesting, they hold for a rather specialized model, which may not reveal anything about deep neural networks, which was the original motivation for this work. Given the narrow focus of the work, unfortunately, I cannot recommend that the paper be accepted. 
Decision: Reject