Implicit Regularization Effects of Unbiased Random Label Noises with SGD
Authors: Haoyi Xiong
Xuhong Li
Boyang Yu
Dejing Dou
Dongrui Wu
Zhanxing Zhu
Venue:  (2021)

Abstract: Random label noises (or observational noises) widely exist in practical machinelearning settings. we analyze the learning dynamics of stochastic gradient descent(SGD) over the quadratic loss with unbiased label noises, and investigate a newnoise term of dynamics, which is dynamized and influenced by mini-batch sam-pling and random label noises, as an implicit regularizer. Our theoretical analysisfinds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters. To validate our analy-sis, we use our theorems to estimate the closed-form solution of the implicit reg-ularizer over continuous-time SGD dynamics for Ordinary Least-Square (OLS), where the numerical simulation backups our estimates. We further extend our proposals to interpret the newly-fashioned noisy self-distillation tricks for deep learning,  where the implicit regularizer demonstrates a unique capacity of selecting models with improved output stability through learning from well-trained teach-ers with additive unbiased random label noises

Under review as a conference paper at ICLR 2021
IMPLICIT REGULARIZATION EFFECTS OF UNBIASED RANDOM LABEL NOISES WITH SGD
Anonymous authors Paper under double-blind review
ABSTRACT
Random label noises (or observational noises) widely exist practical machine learning settings. we analyze the learning dynamics of stochastic gradient descent (SGD) over the quadratic loss with unbiased label noises, and investigate a new noise term of dynamics, which is dynamized and influenced by mini-bate sam- pling and random label noises, as an implicit regularizer. Our theoretical analysis finds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters. To validate our anal- ysis, we use our theorems to analyze the implicit regularizer of SGD with unbi- ased random label noises for linear regression via Ordinary Least-Square (OLS), where the numerical simulation backups our theoretical findings. We further ex- tend our proposals to interpret the newly-fashioned noisy self- aif-distillation tricks for deep learning, where the implicit regularizer demonstrates unique capacity of se- lecting models with improved output stability through learning from well-trained teachers with additive unbiased random label noises.
INTRODUCTION
Stochastic Gradient Descent (SGD) has been widely used as an effective way to train deep neural networks with large datasets (Bottou 1991). While the mini-batch sampling strategy was firstly pro- posed to lower the cost of computation per iteration, it has been consider to incorporate an implicit regularizer preventing the learning process from converging to the local minima with poor gener- alization performance (Zhang et al. 2017 Zhu et al. 2019; Jastrzebski et al. 2017 Hoffer et al.) 2017 Keskar al.. 2017). To interpret such implicit regulari eegularization,
model SGD gradient descent (GD) with gradient noises caused by mini-batck batch sampling (Bottou et al. 2018). Studies have demonstrated the potentials of such implicit regularization or gradient noises improve the gener- alization performance of learning from both theoretical (Mandt et al. 2017 Chaudhari & Soatto
2018 Hu al., (2099h) (Sisselii al. 2019) and empirical aspects (Zhu et al.
2019) Hoffer al.,
2017 Keskar et al.) 2017). In summary, gradient noises keep SGD away from converging to the sharp local minima that generalizes poorly (Zhu et al. 2019 Hu al. 2019b Simsekli al.. 2019) and would select flat minima (Hochreiter & Schmidhuber
1997) as the outcome of learning.
In this work, we aim at investigating the influence of random label noises to the implicit regulariza- tion under mini-batch sampling of SGD. To simplify our research, we assume the training dataset as a set of vectors D = {æ1, I22 33, ,NN J. label ÿJ for every vector x; E D is the noisy response of the true neural network (x) such that
/ y. Es, 3: = (x.), and B[e.] = O, var[s.]==
where the label noise is assumed to be an independent zero-mean random variable. In our work, the random label noises can be either (1) drawn from probability distributions before training steps (but dynamized by mini-batch sampling SGD) or (2) realized by the random variables per training iteration Han et al. 2018). Thus learning is to approximate [xx 0) that beats I (Ir). such that
N
N
1 = N
0 argmin V
(.
Σ (f(z;, 0) 32)
(2)
N
Inspired by (Hochreiter & Schmidhuber 1997 Zhu et al., 2019). our work studies how unbiased
label noises E; (1 < < NN would affect the "selection" of 0 from possible solutions, in the viewpoint of learning dynamics (Saxe et al. 2014) of SGD under mini-batch sampling (Li et al. 2017 Wu et al., 2020 Hu et al. (2019b).

---

Under review as a conference paper at ICLR 2021
Contributions. Our analysis shows that under mild conditions, with gradients of label-noisy losses, SGD might incorporate an additional data-dependent noise term, complementing with the stochastic gradient noises (Li et al. 2017 Wu et al., 2020) of label-noiseless losses, through re- sampling the samples with label noises (Li et al. 2018) or dynamically adding noises to labels over iterations (Han et al. 2018). We consider such noises as an implicit regularization caused by unbi- ased label noises, and interpret the effects of such noises as a solution selector of learning procedure. More specifically, this work has made unique contributions follow.
(1) Implicit Regularizer. We reviewed the preliminaries (Li et al. 2017; Ali et al. 2019 Hu et al., 2019b; Wu al., 2020) and extent the analytical framework in (i al. 2017) to interpret the effects of unbiased label noises as an additional implicit regularizer on top of the continuous time dynamics of SGD. Through discretizing the continuous-time dynamics of label-noisy SGD, we write
discrete-time approximation to the learning dynamics, denoted as ULLN for k = 1,2,.... as
QUIN_
? N
VL;(ON EE
.ULN
(BUIN)
R+1
SK
(f(z; f" (arx) refers to the label-noiseless loss function with sample my and where (0)
the true (noiseless) label yı, the noise term Ex (0) refers to the stochastic gradient noise (Li et al. 2017) of label-noiseless loss function Lद (0), then we can obtain the new implicit regularizer caused by the unbiased label noises (ULN) for v0 E Rd, which can be approximated as follow
SK ULN (0)
o²
Vas( OVoos ()) Zh, and zμ NNO,,, Ia),
(4)
V
where = refers to a random noise vector drawn from the standard Gaussian distribution, 02 refers to the parameters of network in the hth iteration, (.)1/2 refers to the Chelosky decomposition of the matrix, Vef (2, 0) (x.,, 0)/00 refers to the gradient of the neural network output for sample I; over the parameter 0z. and B and are defined as the learning rate and the batch size of SGD respectively. Obviously, the strength of such implicit regularizer is controlled by 0², B and 17.
(2) Effects to Linear Regression. To understand behaviors of the implicit regularizer <(UT.N(0)) to the learning dynamics, we studied SGD over Ordinary Least Square (OLS).With With the proposed model, we could easily obtain the implicit regularizer such that
UUN (B) R (n/B((²ENx) 1/2 and z NO Ia) Sk
(5)
where + Z.J. referring to the sample covariance matrix of the training dataset. Our theoretical elaboration suggests that SGD with unbiased random label noises would converge to a distribution of Gaussian-alike centered at the optimal solution of OLS, where the span and shape of the distribution would controlled by 0² and ΣN when and B are constant. We conducted the extensive experiments using SGD with various o² and En. and successfully obtain the results that coincide our theories and directly visualize the effects of the implicit regularizer over the path of learning and converging distribution of SGD for noisy linear regression.
(3) Inference Stabilizer. The regularization effects of unbiased random label noises should be
N
N
Esg || JULN (0w)l2=
no² BN
Vel(z,
0w)1 = BN 20
n102
f(a, 0x)
(6)
1=1
where Vef(2, 8) refers to the gradient of ff over and the effects is controlled by the batch size B and the variance of label noises o², We extend above results to understand the newly-fashioned noisy self-distillation (Zhang et al. 2019a; Kii et al. 2020) paradigms, where a well-trained model supposed to be further improved through learning from its noisy outputs. Our analysis showed that, when the new convergence achieved, the noisy oeff-disiillatin strategy would prefer to re-select
model with a lower neural network gradient norm + ΣN" [Voo( 0)1 where the gradient norm characterizes the variation/instability of neural network inference results (over perturbations) around parameters of the model. We carry out extensive experiments while results backup theories. Note that while the earlier work (Bishop) 1995) found training with input noises can also bring regularization effects, our work focuses the observational noises on labels.
2

---

Under review as a conference paper at ICLR 2021
2 PRELIMINARIES AND RELATED WORK
SGD Dynamics and Implicit Regularization We follow settings in (Li al. 2017 and consider SGD as an algorithm that, in the kdhh iteration with the estimate 8g. it randomly picks up a B-length subset of samples from the training dataset ie.. BR C D. and estimates the mini-batch stochastic gradient & Σ EvzYEB (L,(0,), then updates the estimate for 0x+1 based on lz. as follow
Bx+1 t 02
(0--
E VL,(0N)
8k
7 N
Σ VL,(0k) vTV((0)) +VV(0)))
(7)
VENED
where 1 refers to the step-size of SGD, and V4 (Ox) refers to a stochastic gradient noise term caused by mini-batch sampling. The noise would converge to zero with increasing batch size, as follow
vi(0k) = v ( Σ VL((0)) - 1 E VL((ON) - Od, as B BNN. B Va,EB,
(8)
VE,ED
Let define DN (0x) as the sample covariance matrix of loss gradients (L,(0k) for S i N, where we follow (Li et al. 2017) and do not make low-rank assumptions on ESG (0x). Under mild
conditions (Li al. 2017
Chaudhari & Soatto 2018), one can approximate SGD as 0g such that
O2 t Os
7 N
T + (0,o)), (0x) = V B
SSD (0x))*
Zko 2k ~ NO (9)
VE,ED
The implicit regularizer of SGD could be considered as (0.) = v (2 (0x) zu which is
data-dependent and controlled by the learning rate 17 and batch size B (Smith et al. 2018). (aand ett al. 2017 Chaudhari & Soatto 2018 Hu et al.] (219b) discussed SGD for varational inference and enabled novel applications to samplers (Zhang et al. 2019b Xiong et al. 2019). To understand the effect to generalization performance, (Zhu et al. 2019, Smith et al. 2018) studied the escap- ing behavior from the sharp local minima (Keskar et al., 2017) and convergence to the flat ones. Finally, (Gidel et al.] 2019) studied regularization effects to linear DNNs and (Wu et al. (020) proposed new multiplicative noises to interpret SGD and obtain stronger theoretical properties.
SGD Implicit Regularization for Ordinary Least Square (OLS) The most recent and relevant work in this area is (Ali et al.) 2019 2020), where the same group of authors studied the implicit regularization of gradient descent and stochastic gradient descent for OLS. They investigated an implicit regularizer of f2-norm alike on the parameter, which regularizes OLS Ridge estimator with decaying penalty. Prior to these efforts, F. Bach and his group have studied the convergence of gradient-based solutions for linear regression with OLS and regularized estimators under both noisy and noiseless settings in (Diculeveut et al. 2017 Marteau-Feree et al.
2019 Berthier et al. 2020
Seff-Distillation and Noisy Students Self-distillation (Zhang et al. 2019a Xie et al. 2020 Xu
al. 2020 Kim et al. 2020) has been examined as an effective way to further improve the general- ization performance of well-trained models. Such strategies enable knowledge distillation using the well-trained ones as teacher models and optionally adding noises (e.g., dropout, stochastic depth, and label smoothing or potentially the label noises) onto training procedure of student models.
Discussion the Relevant Work Though tremendous pioneering studies have been done in this area, we still make contributions in above three categories. First of all, this work characterizes the implicit regularization effects of label noises SGD dynamics. Compared
(Ali al.. 2019] 2020) working on linear regression, our model interpreted general learning tasks. Even from lin- ear regression perspectives (Ali et al. [219 2020 Berthier al., 2020), we precisely measured the gaps between SGD dynamics with and without label noises using the continuous-time diffu- sion.Ch CompaCod to (Lopez-Paz et al. 2016, Kim et al. 2020),, our analysis emphasized role of the implicit regularizer caused by label noises for model selection, where models with high inferential stability would be selected. (Li et al.) 2020) is the most relevant work to us, where authors studied the early stopping of gradient descent under label noises via neural tangent kernel (NTK) (Jacot et al.) 2018) approximation. Our work made the analyze for SGD without assumptions for approximation such as NTK. To best of our knowledge, this work is the first to understand the effects of unbiased la- bel noises to SGD dynamics, by addressing technical issues including implicit regularization, OLS, self-distillation model selection, and the stability inference results.
3

---

Under review as a conference paper at ICLR 2021
3 LEARNING DYNAMICS AND IMPLICIT REGULARIZATION OF SGD WITH UNBIASED RANDOM LABEL NOISES
From initialization 0,LN, SGD with Unbiased Random Label Noises uses an iterative algorithm that updates the estimate incrementally. Specifically, in the k iteration, SGD randomly picks up batch of sample By S D to estimate the stochastic gradient, as follow
Ks (5
1
Σ
1 N VL 2
E
EI + <ULN (0 (10)
z.EBr
where (0) for 40 E IRd refers to the loss gradient based on the label-noiseless sample (24, ys) refers to stochastic gradient noises (Li et al. 2017) through mini-batch
and Yr ff (z;), (t) sampling over the gradients of label-noiseled samples, and Se C is an additional noise term caused by the mini-batch sampling and the Unbiased Random Label Noises, such that
0 (f 0) (x;))) (f(zi, 0) (*())) Vf(x, 0),
(0) = 00
N VLJ(0) 11 1100) and EB. [[] Od.
(0) 1
Σ
1
|B
(11)
EyEB
ULN (0) = B& CE
Σ
Ej Vs(z, 0), and BB, 14k UUN (0) = 0,-
BrEB₁
Note that, for every iteration VO E Rd, the random vectors E (0) and & cULN (0) are with zero-mean as E(s,) 0. To characterize the variances of the two random vectors, we define two matrix-value
functions ESGD( (0) and EN (0) over 0 E Rd based the label-noiseless losses, such that NN
N
1 N N
1 N -
N
T
SGD (0)
(L2(0)
Σ 4
LI(0) 1=1
N
N
==1
(12)
d² N
(0)
Vof(z,, (e Vof T)T as varlE]] o².
1=1
Under mild conditions, we have var (5) 1/B (0) and var LLL [0)] (0) 1/B EULN
SGD Learning Dynamics with Unbiased Random Label Noises We consider the SGD gorithm with unbiased random label noises in the form of gradient descent with additive data- dependent noise. When 7 - 0, we assume the noise terms (0,) and ck eUINN (l (0x) are independent, then we can follow the analysis in (Hu et al. 2019ab to derive the diffusion process of SGD with unbiased random label noises, denoted as GUEN (t) over continuous-ting f 2 0, such that
N
d@UIN
VIG dl
T SGD (QUIN)) & dW)
N
B
N
(13)
ULN N
(OULN)) $ dt2(t).
V
where (t) and (2(t) refer to two independent Brownie motions over time and di /n. Again, obtain the discrete-ting approximation (Li et al. 2017: Chaudhari & Soatto 2018) to the SGD dynamics, denoted as ØULN for A = 1,2,... which in the 1 iteration behaves as
JULN JULN_
1 N
(L
7 VV ( (0 (OULN)) zΦ + (> (OULN)) =k) (14)
where Σ4 and = are two independent d-dimensional random vectors drawn from standard d- dimensional Gaussian distribution (O,,, Ia) per iteration independently. and ØLLK = gULN (t 0). Note that the errors from the SGD algorithm to its continuous-time diffusion process and from the continuous-time dynamics to its discretization are bounded under weak convergence (Hu et al.) 2019ab. In this way, we can the trajectory of the discrete-time dynamics 0LLN analyze the behaviors of SGD algorithm PULN over iterations.
4

---

Under review as a conference paper at ICLR 2021
Implicit Regularizer Influenced by Unbiased Random Label Noises Compared the stochastic gradient with unbiased random label noises (1) and the stochastic gradient based on the label- noiseless losses, we find an additional noise term Ek EINNN() as the implicit regularizer.
To interpret 4 (0), we first define the diffusion process of SGD based on Label-NoiseLess losses Σ (OLN))))d V (ENDD())))))) W(t). Through comparing OULN (t) with (l), the effects of E (0) over continuous-time form should (O)) Then, discrete-time, we could get results as follow.
Le., (0) for S N d
be
Proposition 1 (The implicit regularizer (O)) The implicit regularizer of SGD with unbiased random label noises could be approximated (with (v))) approximation ervor due discretiza- tion (Li et al. 2017 as follow,
$
N
ULN Sk
(0)
and 26 ~ OO La). 2ks
(15)
V
B
N
In this way, we can estimate the expected strength of the implicit regularizer UUL((0) as follow
N
E Ex UUN (
170
Σ (2,
(16)
BN
In this way, we can conclude that the effects of implicit regularization caused by unbiased random label noises for SGD is proportional N Σ (V, 00122 the average gradient norm of the neural network (x, 0) over samples. Please refer to appendix for the proof.
Inference Stabilizer Here we extend the existing results on SGD Zhu et al. 2019 2018) to understand Proposition as follows.
Wu et al,
(1) Inference Stability- The gradient norm N Vol 0) & Σ N 2 [x, 0) characterizes the variation of neural network output f(x, 0) based on samples H, (for 1 S i S N) over the parameter interpolation around the point 0. Lower N Σ [e 0) comes higher stability of neural network fxx 0) outputs against the (random) perturbations over parameters.
N
(2) Escape and Converge When the noise 5k eUIN (0) is d-dependent (section 4 would present spe- cial case that C is O-independent with OLS), we follow (Zhu al. 2019) and suggest that the implicit regularizer help SGD escape from the point Ó with high neural network gradient norm [of 0)1 as the scale of noise < EULN (0) is large. Reciprocally, we follow (Wu al, 2018) and suggest that when the SGD with unbiased random label noises converges, the converging point 6* should be with small N EE [o, 0*) |12-
(3) Performance Tuning Considering n B the coefficient balancing the implicit regularizer and vanilla SGD, one can regularize/penalizz the SGD learning procedure with the fixed and B more fiercely using larger o². More specifically, we could expect obtain solutions with lower N EN [V,, 0)1 or higher inference stability of neural networks, as regularization effects become stronger when o² increases.
IMPLICIT REGULARIZATION EFFECTS TO LINEAR REGRESSION
Here, we consider special example of SGD with unbiased random label noises using linear regres- sion, where a simple quadratic loss function is considered for OLS, such that
N L.(3) = Σ (x
BOLS + argmin BER
91 s
(17)
N
where samples are generated through y: = x 3
+ E D[=,] = and var(e,] = 0². Note that in this section, we replace the notation of 0 with to present the parameters of linear regression models.
5

---

Under review as a conference paper at ICLR 2021
Learning Dynamics and Implicit Regularization Effects The continuous time diffusion pro- cesses for SGD algorithms with and without unbiased label noises are as follow
N
d3ULN (t)
1 N
Σ
xa gUIN (t) - x =.T3))d + v 1 2
SGD (B (W))
+ V
B N
UULN D
(18)
N
( = N
Σ nz (o (LNI(() (z) +
SGD (OLND (t) B
where gUIN(()) (t) and (t) refer to the SGD dynamics for OLS under Unbiased Label Noises and Label NoiseLess settings. We then denote the sample covariance matrix of N samples as En = 2i=1 x.a!". Matrices LSG (3(t)) and LUIN (B(t)) in this case are defined as N >i=
1 Σ (2 3 ENB) (a 3- ΣNB) and IN =0)= N
ESGD (3)
(19)
which are both time-homogeneous. Compared (L) the dynamics BULN(1) (t) incorporates an additional noise term / S (
which affects the dynamics.
Proposition 2 (Implicit Regularization OLS) In this way, we could approximate the implicit regularizer of SGD with the random label noises for OLS through discretization such as,
TI SUIN (BUIN (1)) = Sk: CULN B
110?
(IN)
1/2 2k, and 2k N La), (20)
2020), SGD noiseless which is independent with 3 and (the time). According to (Berthier et al.
linear regression would asymplotically converge to the optimal solution 3 With additional noise term & FLN (3) and the single optima 8 (for both noisy and noiseless losses), we could conclude that when k xc, SGD with unbiased random label noises would converge a distribution centered at 3" The distribution would tend to be a Gaussian distribution when o² is significant (we could not ignore the effects of stochastic gradient noise terms of noiseless loss to the overall distribution), as the term Vo²n! BEN (W(L) (L) corresponds to a Gaussian distribution. The span and shape of the distribution are controlled by o² and EN when y and are constant.
Numerical Validation To validate Proposition 2, we carry out numerical evaluation using syn- thesize data to simply visualize the dynamics over iteration of SGD algorithms with label-noisy OLS and label-noiseless OLS. In our experiments, we use 100 random samples realized from 2-dimension Gaussian distribution X; ~ (N, E1.2) for 1 < < 100, where Σ1 is an symmet- ric covariance matrix controlling the random sample generation. To add the noises to the labels, we first drawn 100 copies of random noises from the normal distribution with the given variance (00, then we setup the OLS problem with Y,) pairs using Y; X,T 3" + and 3* = 1]T and various settings of o² and E1. We setup the SGD algorithms with the fixed learning rate 7) 0.01, and bath size B 5, with the total number of iterations K 1, 000, 000 to visualize the complete paths.
presents the results of numerical validations. In Figure (d), we gradually increases the variances of label noises x² from 0.25 to 2.0, where we can observe (1) SGD over -aiel-noiseless OLS converges to the optimal solution 3 [1.0, 1.0] in fast manner, (2) SGD over OLS with unbiased random label noises would asymptotically converge to a distribution centered at the op- timal point, and (3) when o² increases, the span of the converging distribution becomes larger. In Figure [h)), we use four settings of 11,2, where can see (4) matter how Σ1 is set for OLS problems, the SGD with unbiased random label noises would asymptotically converge to a dis- tribution centered at the optimal point. Compared the results in (e) with(f),
Figure Securepresseer
find that, when the trace of Σ1, increases, the span of converging distributions would increases. Furthermore, (5) the shapes of converging distributions depend on Σ1,2 2. In Figure [g),, when we place the principal component of E1,2 onto the vertical axis (i.e., Ver [[10, 0] [0. 100) [), the distribution lays

---

Under review as a conference paper at ICLR 2021
with Label hoces withou Later Ninhor
(a) 0
0.25
(b) 0²
0.5
(c) 0 = 1.0
(d) 0² = 2.0
Label
4.50
2.0
260
400
(g) Figure 1: Trajectories SGD over OLS with and without Unbiased Random Label Noises using various 8² and Σ1, settings for (noisy) random data generation. For Figures (a)-d), the experiments
(e) E1, 10
(1) 11, 100
31 Ver
(h) 31,2 E Hor
are setup with fixed Σ1. = [200 0]T, [0, 20 T] and varying σ². For Figures (e)-(h), the experi- ments are setup with a fixed ō² 0.5 and varying E1, where we set 2 = [10, 0] [0, ] and you = [[100 TJT, [o, 10]T] to shape the converging distributions.
the vertical axis principally. Figure [h) demonstrates the opposite layout of the distribution, when set Horr [1000 " [0, 10] as 11. The scale and shape of the converging distribution backups our theoretical investigation in Eq 20
Note that the unbiased random label noises added the labels prior the learning procedure. In this setting, it is the mini- bani-bbtch sampler of SGD that "dynamizes" the noises and influences the dynamics of SGD through forming the implicit regularizer.
IMPLICIT REGULARIZATION EFFECTS TO DEEP NEURAL NETWORKS
Given well-trained model, Noisy Self-Distillation algorithms (Zhang al. 2019a) Xu et al. 2020; Kim
200
200
-00
et al. 2020: Xie et al. 2020) intend
further improve the performance of model through learning from the "soft label" outputs (i.e., logits) of the model (as the teacher). Further- more, some practices found that the self-distillation could be further im- proved through incorporating certain randomness and stochasticity in the training procedure so as to obtain bet- ter generalization performance (Xie et al. 2020 Kim et al.) 2020). In this work, we study the way that directly adds random label noises to the logit outputs of the pre-trained model so as to improve the self-distillation (Han) ct al. 2018). More specifically, we study two well-known strategies for additive noises as follow.
(a) G.N. (SVHN)
(b) G.N. (CIFARIO) (c) G.N. (CIFAR100)
(d) Val. (SVHN)
(c) Val. (CIFARIO)
(f) Val. (CIFAR100)
(g) Test, (SVHN) (h) Test. (CIFARIO) (i) Test. (CIFARI00) Figure 2: Gradient Norms, Validation Accuracy, and Testing Accuracy in Noisy Self-Distillation using ResNet-56 with varying scale label noises (e.g., and o²))
(1) Gaussian Noises. Given pre- trained model with L-dimensional logit output, for every iteration of self-distillation, this simple method that draw random vectors

---

Under review as a conference paper at ICLR 2021
from L-dimensional Gaussian distribution (O II), adds the vectors to the logit outputs of the model, and makes the student model learn from the noisy outputs. Note that in our analysis, assume the output of the model is single dimension while, in self-distillation, the logit labels are with multiple dimensions. Thus, the diagonal matrix I now refers to the complete form the variances and o² controls the scale.
(2) Symmetric Noises.. Basically, this strategy is derived from (Han et al. 2018) that generates noises through randomly swapping the values of logit output among the dimensions. Specifically, in every iteration of self-distillation, given a swap-probability p. every logit output (denoted as here) from the pre-trained model, and every dimension of logit output denoted as yr. the strategy in probability p swaps the logit value in the dimension that corresponds to y with any other dimension Umsgt in equal prior (i.e., in 1)=11)- probability). In the rest - P probability, the strategy remains the original logit output there. In this way, the new noisy label ÿj is with expectation Elyj as follow,
Exmpt Mm
j = (1 p) y +
(21)
L-1
This strategy introduces explicit bias to the original logit outputs. However, when we consider the expectation 18 [r] as the innovative soft label, then the random noise around the new soft label is still unbiased as E -Eū] = O for all dimensions. Note that this noise is not the symmetric noises studied for robust learning (Wang et al. 2019).
Figure presents the results of above two methods with increasing scales of noises, i.e., increasing o² for Gaussian noises and increasing p for Symmetric noises. In Figure 2 (c), we demonstrate that the gradient norms of neural networks o 0)11 decrease with growing o² and for two strategies. The results backup our theoretical investigation which means the model would be awarded high inferential stability, as the variation of neural network outputs against the potential ran- dom perturbation in parameters has been reduced by the regularization. In Figure dj)) and (g)- ()) we plot the validation and testing accuracy of the models obtained under noisy self-distillatio.. The results show that (1) some of models have been improved through noisy self-distillation com- pared to the teacher model, (2) noisy self-distillation could obtain better performance than noiseless self-distillation, and (3) it is possible to select noisily self-distille models using validation accuracy for better overall generalization performance. All results here are based on 200 epochs of noisy self-distillation.
6 DISCUSSION AND CONCLUSION
While previous studies primarily focus on the performance degradation caused by label noises or corrupted labels (Jiang al., 2018 Li et al. 2020), we investigate the implicit regularization effects of random label noises, under mini-batch sampling settings of stochastic gradient descent (SGD). Specifically, we adopt the dynamical systems interpretation of SGD to analyze the learning proce- dure based on the quadratic loss with unbiased random label noises. We decompose the mini-batch stochastic gradient based on label-noisy losses into three parts in Eq, (11): (i) (O* - the true gra- dient of label-noiseless losses, (ii) 5% (0) the stochastic gradient noise caused through mini-batch sampling over the label-noiseless losses, and (iii) <ULN (0) the noise term influenced by the both random label noises and mini- mini-batch sampling. Our research considers C (0) as an implicit regu- larizer, and finds that effects of such implicit regularizer is to lower the gradient norm of the neural
networks 1 > || [ef 1) over the learning procedure, where the gradient norm of neural N networks here characterizes the variation/stability of the neural network outputs against the random perturbation around the parameters. In summary, the new implicit regularizer ck select point with higher inference stability for convergence.
eULN (0) helps SGD
We carry out extensive experiments to validate our theoretical investigations. The numerical study with linear regression clearly illustrates the trajectories of SGD with and without unbiased random label noises, the observation coincides the SGD dynamics derived from our theories. Evaluation based on deep neural network shows that, in self-distillation settings, one can lower the gradient norm of neural networks, improve the inference stability of networks, and obtain better solutions, through iteratively adding noises to the outputs of teacher models. Note that we do not want to claim that the implicit regularization caused by the label noises would improve the generalization performance in this work. The experiments results well backup our theories.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alnur Ali, J Zico Kolter, and Ryan Tibshirani. A continuous-time view of early stopping for least squares regression. A/STATS, 2019.
Alnur Ali, Edgar Dobriban, and Ryan J Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International Conference on Machine Learning, 2020.
Raphaël Berthier, Francis Bach, and Pierre Gaillard. Tight nonparametric convergence rates for stochastic gradient descent under the noiseless linear model. arXiv preprint arXiv:2006.08212, 2020.
Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computation, 7(1):108-116, 1995.
Léon Bottou. Stochastic gradient learning in neural networks. Proceedings of Neuro-Nmes, 91(8), 1991.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. Siam Review, 60(2):223-33 311, 2018.
Pratik Chaudhari and Stefano Soatto. Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks. In International Conference on Learning Represen- lutions, 2018.
Aymeric Dieuleveut, Nicolas Flammarion, and Francis Bach. Harder, better, faster, stronger con- vergence rates for least-squares regression. The Journal of Machine Learning Research, 18(1): 3520-3570, 2017.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. In Advances in Neural Information Processing Systems, pp- 3202-3211, 2019.
Bo Han, Quanming Yao. Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in neural information processing systems, pp. 8527-8537 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NewrlPS Deep Learning and Representation Learning Workshop, 2015. orq/ abs/ arq/abs// 1503. 02531
S. Hochreiter and J. Schnidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the general- ization gap in large batch training of neural networks. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 1731-1782 1741. Curran Associates, Inc., 2017.
Wenging Hu, Chris Junchi Li, Lei Li, and Tian-Guo Liu. On the diffusion approximation of noncon- vex stochastic gradient descent. Annals of Mathematical Sciences and Applications, 4(1):3-32, 2019a.
Wenqing Hu, Zhanxing Zhu, Haoyi Xiong, and Jun Huan. Quasi-potential as implicit regularizer for the loss function the stochastic gradient descent. arXiv preprint arXiv:1901.060: 2019b.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and gen- cralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.

---

Under review as a conference paper at ICLR 2021
Stanistaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623 2017.
2017.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentorntt: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. pp. 2304-2313, 2018.
N. S. Keskar, D. Mudigere, Nocedal, M. Smelyanskiy, and P. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In In International Conference on Learning Representations (ICLR), 2017.
Kyungyul Kim, BycongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distilla- tion: A simple way for better generalization. arXiv preprint arXiv:2006.12000 12000, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Mingchen Li. Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust label noise for overparameterized neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 2313-4324. PMLR, 2020.
Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms.. In International Conference on Machine Learning, pp. 2101-2110, 2017.
Tianyang Li, Liu Liu, Anastasios Kyrillidis, and Constantine Caramanis. Statistical inference using sgd. In AAAI, 2018
David Lopez-Paz, Léon Bottou, Bernhard Schölkopf, and Vladimir Vapnik. Unifying distillation and privileged information. In In International Conference on Learning Representations (ICLR), 2016.
Stephan Mandt, Matthew Hoffman, and David Blei. Stochastic gradient descent as approximate bayesian inference. The Journal of Machine Learning Research, 18(1))4873 2017.
Ulysse Marteau-Ferey, Dmitrii Ostrovskii, Francis Bach, and Alessandro Rudi. Beyond least- squares: Fast rates for regularized empirical risk minimization through self-concordance. In Conference on Learning Theory, pp. 2294-2340, 2019.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Read- ing digits in natural images with unsupervised feature learning. In NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions the nonlinear dy- namics of learning in deep linear neural networks. In International Conference on Learning Reporesentutions (ICLR), 2014.
Umut Simsekli, Levent Sagun, and Mert Gurbuzbalaban. A tail-index analysis stochastic gradient noise in deep neural networks. arXiv preprint arXiv:1901.0660 2019.
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross en- tropy for robust learning with noisy labels. In Proceedings of the IEEE International Conference on Computer Vision, pp. 322-330, 2019.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning (ICML), 2020.
Lei Wu, Chao Ma, and E Weinan. How sgd selects the global minima in aver-parameterized learning: A dynamical stability perspective. Advances Neural Information Processing Systems, pp- 2018.
10

---

Under review as a conference paper at ICLR 2021
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
Haoyi Xiong, Kafeng Wang, Jiang Bian, Zhanxing Zhu, Cheng-Zhong Xu, Zhishan Guo, and Jun Huan. Sphmc: Spectral hamiltonian monte carlo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 5516-5524, 2019.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference Computer Vision, 2020.
C. Zhang. S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3713-3722, 2019a.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representations, 2019b.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In ICML., pp. 7654-7663, 2019.

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A. SKETCHED PROOFS IN PROPOSITION
To obtain [6] we can use the simple vector- products transform that, for the random vector e and symmetric matrix A there has E,,e"' Av] = (raco(AE((v) "), such that
Ezg L (0)112
[
(0)
T $k
(0)]
1100
Fizh
zk
Vos (Voo 2k
B
N
11022
N
Irace
V Vos(z; VoVo
Ezn [zx=
B
N
1=1
as [=
1
Ia for 2Φ No,,,, Ia)
(22)
no²
trace
B
N
no²
N
Vof(z;
0)1
BN
i=1
no²
a 0g fzz 8)
BN
A.2 TRAINING DETAILS FOR NOISY
WITH DEEP NEURAL NETWORKS
We choose the ResNet-56 (He et al. 2016), one of the most practical deep models, for conducting the experiments on three datasets: SVHN (Netzer et al.] 2011), CIFAR 10 and CIFAR 100 (Krizhevsky al. 2009). We follow the standard training procedure (He al. 2016) for training a teacher model (original model). Specifically we train the model from scratch for 200 epochs and adopt the SGD optimizer with batch size 64 and momentum 0.9. The learning rate is set to 0.1 at the beginning of training and divided by 10 at 100th epoch and 150th epoch. A standard weight decay with a small regularization parameter (10-1 4) applied. As for noiseless self-distillation, we follow the standard procedure (Hinton et al., 2015) for distilling knowledge from the teacher to a student of the same network structure. The training setting is the same as training the teacher model.
For noisy self-distillation, we continue to use the training setting except the labels are noised by the two types of noises as introduced in the main text. We choose the best scale label noises using validation set, where we divide the original training set into new training set (80%) and validation set (20%). set of {0.1. 0.2..... 0.9, 1.0) is tried for the scale of symmetric noises. A set of {0.1, 0.5, 1.0, 2.0, 3.0, .... 9.0, 10.0} is tried for the scale of Gaussian noises. For clarity, also present the results using all the choices of scales of label noises on test set, where the original training set is used for training.
A.3 TRAINING PROCESS OF DEEP NEURAL NETWORKS
We show the evolution of training and test losses during the entire training procedure, and compare the settings of adding no label noises, symmetric and Gaussian noises for self-distillations. Figure) presents the results on the three datasets, i.e., SVHN CIFAR10 and CIFAR 100.
12

---

Under review as a conference paper at ICLR 2021
w/o noise
1.5
w/o noiee
w/o noise Symmettric
Symmellii
Symmelric
Tosining Looss
Traiting
1.0
Gauseinn
Training
1.0
Gaussion:
Gauselan
0.5
0.5
0.0
0.0
100
150
200
Tealning Epoch
Thaining Epoch
Training Epoch
(a) Training Loss (SVHN)
(b) Training Loss (CIFARIO)
Training Loss (CIFAR100)
w/o
w/o
w/o
0.8
Synmmetric
Symmetric
Symnnetric
Gaussin
Tes: Loos
1.0
Gauesian
Test Taoos
Gaussian
0.5
0.2
50
100
150
200
100
150
200
Training Epoch
Training Epoch
Teaining Epoch
(d) Test Loss (SVHN)
(c) Test Loss (CIFAR10)
(f) Test Loss (CIFAR100)
Figure 3
13


Reviews:
[{"review_id": "yUbGoEo1oV7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the implicit regularization effect from random label noise in L2 regression, and argues that it can be captured by the average squared gradient norm over samples, $\\frac1N\\sum_{i=1}^N \\|\\| \\nabla_\\theta f(x_i, \\theta) \\|\\|^2$. This conclusion is achieved via analyzing the additional noise term in the update caused by the label noise, which in the SDE approximation has expected squared L2 norm proportional to $\\frac1N\\sum_{i=1}^N \\|\\| \\nabla_\\theta f(x_i, \\theta) \\|\\|^2$.\n\nThe paper does not cite a very related paper:\n\nImplicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process. Guy Blanc, Neha Gupta, Gregory Valiant, Paul Valiant. COLT 2020\n\nThe above paper studies the same problem and obtains a similar conclusion that the implicit regularization of label noise can be captured by $\\frac1N\\sum_{i=1}^N \\|\\| \\nabla_\\theta f(x_i, \\theta) \\|\\|^2$. It is proved that a solution $\\theta$ which achieves zero training error is an attractive fixed point of the dynamics if and only if it is a local minimizer of the said implicit regularizer.\n\nThe high-level conclusion in the current submission is the same as Blanc et al. In addition, the current submission has a weaker theoretical argument, since it is based on the SDE approximation which may not capture SGD as the noise distribution may not be Gaussian. On the other hand, Blanc et al. rigorously prove that any attractive interpolating solution must also locally minimize the implicit regularizer. Given the overlap, the submission does not meet the bar of publication.", "extra_scores": {}, "extra_text": {}}, {"review_id": "busy-cBhoVj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "My review is going to be short because I truly believe that this paper, in the current form, does not reach the acceptance bar for ICLR.\n\nIndeed, the paper tries to show that SGD carries an implicit regularization along the procedure. The list of the contributions is summarized on page 2: the authors proposed a decomposition of the noise into two terms (i) a noise due to the subsampling of the gradient (ii) a noise due to the noisy labels. Neither this decomposition nor the conclusion from it is new. Worst, there are no proofs or experiments backing up there fuzzy conclusions.\n\nDespite, the fact the story behind SGD seems understood by the authors, I see no contributions in this paper.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Re_R5E0QhC8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the implicit regularization effect that arises from using stochastic gradient descent with label noise and squared loss. They derive the expression for the implicit regularization term and show that it favors solutions which are stable against perturbations of the parameters. This paper validates their empirical findings using SGD on linear regression task with label noise. This paper also uses their results to study the self distillation technique.\n\nThe main concern with this paper is that I think that the authors claim that they are the first ones to characterize the implicit regularization arising from SGD with label noise but this result is already known [1]. I believe that [1] works in the same setting and their implicit regularization for SGD exactly matches as derived in this paper.\n\nI would recommend rejecting this paper because I believe that the main result presented in this paper is already known [1] and this paper has not cited or compared with that paper.\n\nOther questions:\n1) Why do the authors assume that the noise terms in equation 10 are independent when step size is small as mentioned on page 4?\n2) I am not convinced by the self distillation experiments. Why do the authors consider self distillation with label noise added to the logits? Is this form of distillation considered in previous works before? Did the authors consider adding label noise to just the original labels? \n3) Moreover, in the plots in figure 2, why do the curves follow this zig zag pattern with increasing noise?  Moreover, many points in the plots are actually below the red dotted line which is the accuracy without label noise. So, I am not fully convinced that label noise is actually leading to better performance. \n4) It is unclear to me what is the motivation for considering these two different types of noises - gaussian and symmetric and what is the take away for these types of noises?\n\nSome typos\n1. Page 2 - extent -> extended\n2. Page 2 - not clear what new convergence exactly means\n3. Page 4 -  brownie -> brownian\n4. Page 6 - increases -> increase\n\n1) Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process, COLT 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "Anyitg_gxCh", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the implicit regularization effect of unbiased random label noise on the dynamics of stochastic gradient descent(SGD). More precisely, an unbiased random noise is added to the true labels and the paper aims to analyze the regularization effect of it. The paper shows that the unbiased label noise would favor convergence to points which \nstabilize model outputs against perturbation of parameters.\n\nI found the paper very obscure, rigorous justification to the hypothesis and claims are missing. For instance, \na) Below equation (12), it claims that under mild conditions, the variance of the noise vector are the given matrices. What would be the conditions? Please state it properly if it is a mild condition. \nb) In the paragraph \"SGD Learning Dynamics with Unbiased Random Label Noises\", it assumes that terms coming from mini-batch and label noise are independent. This is not true in general, please include some positive/negative examples when the assumption holds or violates. As this is one of the key assumptions which implies the additive Brownian motion, a justification is definitely necessary showing that the assumption is valid most often in practice. \nc) In the paragraph \"Inference Stability/Escape and Converge\", why does Prop 1 implies that the solution has low gradient norm. Intuitively, the unbiased noise increases the variance of the gradient and the algorithm would be less stable compared to noiseless version. \nd) It is claimed in performance tuning that larger $\\sigma$ introduces larger regularization effect with higher inference stability. If this is true, then one should use $\\sigma$ as large as possible, which is clearly wrong. \ne) I don't understand what is the purpose in the first experiment shown in Figure 1. It is clear that SGD ( with or without label noise) converges to a neighborhood of the solution rather than the exact minimum and the radius directly depends on the variance of the gradient. What need to be shown is the claim regarding inference stability. Moreover, one trajectory is clearly not representative as the noise $\\epsilon_i$ is random. Hence experiments should be performed by drawing multiple times $\\epsilon_i$, namely multiple trajectories.  \nOverall, there is a lack of justification regarding the claims in the paper. I won't recommend publication of the paper in the current state. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers pointed out that the claims made in this submission have already appeared (in even stronger forms) before, to which the authors seem to agree. Therefore, this submission is not ready for publication in its current form. 
Decision: Reject