Year: 2020

Abstract:
Random label noises (or observational noises) widely exist in practical machine learning settings. we analyze the learning dynamics of stochastic gradient descent (SGD) over the quadratic loss with unbiased label noises, and investigate a new noise term of dynamics, which is dynamized and influenced by mini-batch sampling and random label noises, as an implicit regularizer. Our theoretical analysis finds such implicit regularizer would favor some convergence points that could stabilize model outputs against perturbation of parameters. To validate our analysis, we use our theorems to analyze the implicit regularizer of SGD with unbiased random label noises for linear regression via Ordinary Least-Square (OLS), where the numerical simulation backups our theoretical findings. We further extend our proposals to interpret the newly-fashioned noisy self-distillation tricks for deep learning, where the implicit regularizer demonstrates a unique capacity of selecting models with improved output stability through learning from well-trained teachers with additive unbiased random label noises.

heading: 1 INTRODUCTION
text: Stochastic Gradient Descent (SGD) has been widely used as an effective way to train deep neural networks with large datasets (Bottou, 1991). While the mini-batch sampling strategy was firstly proposed to lower the cost of computation per iteration, it has been consider to incorporate an implicit regularizer preventing the learning process from converging to the local minima with poor generalization performance (Zhang et al., 2017; Zhu et al., 2019; Jastrzebski et al., 2017; Hoffer et al., 2017; Keskar et al., 2017). To interpret such implicit regularization, one can model SGD as gradient descent (GD) with gradient noises caused by mini-batch sampling (Bottou et al., 2018). Studies have demonstrated the potentials of such implicit regularization or gradient noises to improve the generalization performance of learning from both theoretical (Mandt et al., 2017; Chaudhari & Soatto, 2018; Hu et al., 2019b; Simsekli et al., 2019) and empirical aspects (Zhu et al., 2019; Hoffer et al., 2017; Keskar et al., 2017). In summary, gradient noises keep SGD away from converging to the sharp local minima that generalizes poorly (Zhu et al., 2019; Hu et al., 2019b; Simsekli et al., 2019) and would select a flat minima (Hochreiter & Schmidhuber, 1997) as the outcome of learning.
In this work, we aim at investigating the influence of random label noises to the implicit regularization under mini-batch sampling of SGD. To simplify our research, we assume the training dataset as a set of vectorsD = {x1, x2, x3, . . . , xN}.The label ỹi for every vector xi ∈ D is the noisy response of the true neural network f∗(x) such that
ỹi = yi + εi, yi = f ∗(xi), and E[εi] = 0, var[εi] = σ2 , (1)
where the label noise εi is assumed to be an independent zero-mean random variable. In our work, the random label noises can be either (1) drawn from probability distributions before training steps (but dynamized by mini-batch sampling of SGD) or (2) realized by the random variables per training iteration (Han et al., 2018). Thus learning is to approximate f(x, θ̂) that beats f∗(x), such that
θ̂ ← argmin ∀θ∈Rd
{ 1
N N∑ i=1 L̃i(θ) := 1 N N∑ i=1
(f(xi, θ)− ỹi)2 } . (2)
Inspired by (Hochreiter & Schmidhuber, 1997; Zhu et al., 2019), our work studies how unbiased label noises εi (1 ≤ i ≤ N ) would affect the “selection” of θ̂ from possible solutions, in the viewpoint of learning dynamics (Saxe et al., 2014) of SGD under mini-batch sampling (Li et al., 2017; Wu et al., 2020; Hu et al., 2019b).
Contributions. Our analysis shows that under mild conditions, with gradients of label-noisy losses, SGD might incorporate an additional data-dependent noise term, complementing with the stochastic gradient noises (Li et al., 2017; Wu et al., 2020) of label-noiseless losses, through resampling the samples with label noises (Li et al., 2018) or dynamically adding noises to labels over iterations (Han et al., 2018). We consider such noises as an implicit regularization caused by unbiased label noises, and interpret the effects of such noises as a solution selector of learning procedure. More specifically, this work has made unique contributions as follow.
(1) Implicit Regularizer. We reviewed the preliminaries (Li et al., 2017; Ali et al., 2019; Hu et al., 2019b; Wu et al., 2020) and extent the analytical framework in (Li et al., 2017) to interpret the effects of unbiased label noises as an additional implicit regularizer on top of the continuous-time dynamics of SGD. Through discretizing the continuous-time dynamics of label-noisy SGD, we write discrete-time approximation to the learning dynamics, denoted as θULNk for k = 1, 2, . . . , as
θULNk+1 ← θULNk − η
N N∑ i=1 ∇L∗i (θULNk ) + ξ∗k(θULNk ) + ξULNk (θULNk ), (3)
where L∗i (θ) = (f(xi, θ) − f∗(xi))2 refers to the label-noiseless loss function with sample xi and the true (noiseless) label yi, the noise term ξ∗k(θ) refers to the stochastic gradient noise (Li et al., 2017) of label-noiseless loss function L∗i (θ), then we can obtain the new implicit regularizer caused by the unbiased label noises (ULN) for ∀θ ∈ Rd, which can be approximated as follow
ξULNk (θ) ≈ √ η
B
( σ2
N N∑ i=1
∇θf(xi, θ)∇θf(xi, θ)> ) 1 2 zk, and zk ∼ N (0d, Id) , (4)
where zk refers to a random noise vector drawn from the standard Gaussian distribution, θk refers to the parameters of network in the kth iteration, (·)1/2 refers to the Chelosky decomposition of the matrix, ∇θf(xi, θ) = ∂f(xi, θ)/∂θ refers to the gradient of the neural network output for sample xi over the parameter θk, and B and η are defined as the learning rate and the batch size of SGD respectively. Obviously, the strength of such implicit regularizer is controlled by σ2, B and η.
(2) Effects to Linear Regression. To understand behaviors of the implicit regularizer ξULNt (θt) to the learning dynamics, we studied SGD over Ordinary Least Square (OLS).With the proposed model, we could easily obtain the implicit regularizer such that
ξULNk (β) ≈ √ η/B ( σ2Σ̄N )1/2 zk and zk ∼ N (0d, Id) (5)
where Σ̄N = 1N ∑N i=1 xix > i referring to the sample covariance matrix of the training dataset. Our theoretical elaboration suggests that SGD with unbiased random label noises would converge to a distribution of Gaussian-alike centered at the optimal solution of OLS, where the span and shape of the distribution would controlled by σ2 and Σ̄N when η and B are constant. We conducted the extensive experiments using SGD with various σ2 and Σ̄N , and successfully obtain the results that coincide our theories and directly visualize the effects of the implicit regularizer over the path of learning and converging distribution of SGD for noisy linear regression.
(3) Inference Stabilizer. The regularization effects of unbiased random label noises should be
Ezk ∥∥ξULNk (θk)∥∥22 ≈ ησ2BN N∑ i=1 ‖∇θf(xi, θk)‖22 = ησ2 BN N∑ i=1 ∥∥∥∥ ∂∂θf(xi, θk) ∥∥∥∥2 2 , (6)
where ∇θf(x, θ) refers to the gradient of f over θ and the effects is controlled by the batch size B and the variance of label noises σ2. We extend above results to understand the newly-fashioned noisy self-distillation (Zhang et al., 2019a; Kim et al., 2020) paradigms, where a well-trained model is supposed to be further improved through learning from its noisy outputs. Our analysis showed that, when the new convergence achieved, the noisy self-distillation strategy would prefer to re-select a model with a lower neural network gradient norm 1N ∑N i=1 ‖∇θf(xi, θ)‖22, where the gradient norm characterizes the variation/instability of neural network inference results (over perturbations) around parameters of the model. We carry out extensive experiments while results backup our theories. Note that while the earlier work (Bishop, 1995) found training with input noises can also bring regularization effects, our work focuses on the observational noises on labels.

heading: 2 PRELIMINARIES AND RELATED WORK
text: SGD Dynamics and Implicit Regularization We follow settings in (Li et al., 2017) and consider SGD as an algorithm that, in the kth iteration with the estimate θk, it randomly picks up a B-length subset of samples from the training dataset i.e., Bk ⊂ D, and estimates the mini-batch stochastic gradient 1b ∑ ∀xi∈Bk ∇Li(θk), then updates the estimate for θk+1 based on θk, as follow
θk+1 ← ( θk − η
B ∑ ∀xi∈Bk ∇Li(θk)
) = ( θk − η
N ∑ ∀xi∈D ∇Li(θk) + √ ηVk(θk)
) , (7)
where η refers to the step-size of SGD, and Vk(θk) refers to a stochastic gradient noise term caused by mini-batch sampling. The noise would converge to zero with increasing batch size, as follow
Vk(θk) = √ η
( 1
N ∑ ∀xi∈D ∇Li(θk)− 1 B ∑ ∀xi∈Bk ∇Li(θk)
) → 0d, as B → N . (8)
Let define ΣSGDN (θk) as the sample covariance matrix of loss gradients ∇Li(θk) for 1 ≤ i ≤ N , where we follow (Li et al., 2017) and do not make low-rank assumptions on ΣSGDN (θk). Under mild conditions (Li et al., 2017; Chaudhari & Soatto, 2018), one can approximate SGD as θ̄k such that
θ̄k+1 ← θ̄k− η
N ∑ ∀xi∈D ∇Li(θ̄k) + ξk(θ̄k), ξk(θ̄k) = √ η B ( ΣSGDN (θ̄k) ) 1 2 zk, zk ∼ N (0, Id) . (9)
The implicit regularizer of SGD could be considered as ξk(θ̄k) = √ η/B ( ΣSGDN (θ̄k) ) 1 2 zk which is data-dependent and controlled by the learning rate η and batch size B (Smith et al., 2018). (Mandt et al., 2017; Chaudhari & Soatto, 2018; Hu et al., 2019b) discussed SGD for varational inference and enabled novel applications to samplers (Zhang et al., 2019b; Xiong et al., 2019). To understand the effect to generalization performance, (Zhu et al., 2019; Smith et al., 2018) studied the escaping behavior from the sharp local minima (Keskar et al., 2017) and convergence to the flat ones. Finally, (Gidel et al., 2019) studied regularization effects to linear DNNs and (Wu et al., 2020) proposed new multiplicative noises to interpret SGD and obtain stronger theoretical properties.
SGD Implicit Regularization for Ordinary Least Square (OLS) The most recent and relevant work in this area is (Ali et al., 2019; 2020), where the same group of authors studied the implicit regularization of gradient descent and stochastic gradient descent for OLS. They investigated an implicit regularizer of `2-norm alike on the parameter, which regularizes OLS as a Ridge estimator with decaying penalty. Prior to these efforts, F. Bach and his group have studied the convergence of gradient-based solutions for linear regression with OLS and regularized estimators under both noisy and noiseless settings in (Dieuleveut et al., 2017; Marteau-Ferey et al., 2019; Berthier et al., 2020).
Self-Distillation and Noisy Students Self-distillation (Zhang et al., 2019a; Xie et al., 2020; Xu et al., 2020; Kim et al., 2020) has been examined as an effective way to further improve the generalization performance of well-trained models. Such strategies enable knowledge distillation using the well-trained ones as teacher models and optionally adding noises (e.g., dropout, stochastic depth, and label smoothing or potentially the label noises) onto training procedure of student models.
Discussion on the Relevant Work Though tremendous pioneering studies have been done in this area, we still make contributions in above three categories. First of all, this work characterizes the implicit regularization effects of label noises to SGD dynamics. Compared to (Ali et al., 2019; 2020) working on linear regression, our model interpreted general learning tasks. Even from linear regression perspectives (Ali et al., 2019; 2020; Berthier et al., 2020), we precisely measured the gaps between SGD dynamics with and without label noises using the continuous-time diffusion.Compared to (Lopez-Paz et al., 2016; Kim et al., 2020), our analysis emphasized role of the implicit regularizer caused by label noises for model selection, where models with high inferential stability would be selected. (Li et al., 2020) is the most relevant work to us, where authors studied the early stopping of gradient descent under label noises via neural tangent kernel (NTK) (Jacot et al., 2018) approximation. Our work made the analyze for SGD without assumptions for approximation such as NTK. To best of our knowledge, this work is the first to understand the effects of unbiased label noises to SGD dynamics, by addressing technical issues including implicit regularization, OLS, self-distillation, model selection, and the stability inference results.

heading: 3 LEARNING DYNAMICS AND IMPLICIT REGULARIZATION OF SGD WITH UNBIASED RANDOM LABEL NOISES
text: From initialization θULN0 , SGD with Unbiased Random Label Noises uses an iterative algorithm that updates the estimate incrementally. Specifically, in the kth iteration, SGD randomly picks up a batch of sample Bk ⊆ D to estimate the stochastic gradient, as follow
g̃k(θ ULN k ) =
1 |Bk| ∑ xi∈Bk ∇L̃i(θULNk ) = 1 N N∑ i=1 ∇L∗i (θULNk ) + ξ∗k(θULNk ) + ξULNk (θULNk ), (10)
where ∇L∗i (θ) for ∀θ ∈ Rd refers to the loss gradient based on the label-noiseless sample (xi, yi) and yi = f∗(xi), ξ∗k(θ) refers to stochastic gradient noises (Li et al., 2017) through mini-batch sampling over the gradients of label-noiseless samples, and ξULNk (θ) is an additional noise term caused by the mini-batch sampling and the Unbiased Random Label Noises, such that
∇L∗i (θ) = ∂
∂θ (f(xi, θ)− f∗(xi))2 = (f(xi, θ)− f∗(xi)) · ∇f(xi, θ) ,
ξ∗k(θ) = 1 |Bk| ∑ xj∈Bk ( ∇L∗j (θ)− 1 N N∑ i=1 L∗i (θ) ) , and EBk [ξ∗k(θ)] = 0d,
ξULNk (θ) = − 1 |Bk| ∑ xj∈Bk εj · ∇f(xj , θ), and EBk,εi [ξULNk (θ)] = 0d.
(11)
Note that, for every iteration ∀θ ∈ Rd, the random vectors ξ∗k(θ) and ξULNk (θ) are with zero-mean as E(εj) = 0. To characterize the variances of the two random vectors, we define two matrix-value functions ΣSGDN (θ) and Σ ULN N (θ) over θ ∈ Rd based on the label-noiseless losses, such that
ΣSGDN (θ) = 1
N N∑ j=1 ( ∇L∗j (θ)− 1 N N∑ i=1 L∗i (θ) )( ∇L∗j (θ)− 1 N N∑ i=1 L∗i (θ) )>
ΣULNN (θ) = σ2
N N∑ j=1 ∇θf(xj , θ)∇θf(xj , θ)> as var[εj ] = σ2 .
(12)
Under mild conditions, we have var[ξ∗k(θ)] = 1/B ·ΣSGDN (θ) and var[ξULNk (θ)] = 1/B ·ΣULNN (θ).
SGD Learning Dynamics with Unbiased Random Label Noises We consider the SGD algorithm with unbiased random label noises in the form of gradient descent with additive datadependent noise. When η → 0, we assume the noise terms ξ∗k(θk) and ξULNk (θk) are independent, then we can follow the analysis in (Hu et al., 2019a) to derive the diffusion process of SGD with unbiased random label noises, denoted as θULN(t) over continuous-time t ≥ 0, such that
dθULN = − 1 N N∑ i=1 ∇Li(θULN)dt+ √ η B ( ΣSGDN (θ ULN) ) 1 2 dW1(t)
+
√ η
B
( ΣULNN (θ ULN) ) 1 2 dW2(t) ,
(13)
where W1(t) and W2(t) refer to two independent Brownie motions over time and dt = √ η. Again, we can obtain the discrete-time approximation (Li et al., 2017; Chaudhari & Soatto, 2018) to the SGD dynamics, denoted as θ̄ULNk for k = 1, 2, . . . , which in the k th iteration behaves as
θ̄ULNk+1 ← θ̄ULNk − η
N N∑ i=1 ∇L∗(θ̄ULNk )+ √ η B (( ΣSGDN (θ̄ ULN k ) ) 1 2 zk + ( ΣULNN (θ̄ ULN k ) ) 1 2 z′k ) , (14)
where zk and z′k are two independent d-dimensional random vectors drawn from a standard ddimensional Gaussian distributionN (0d, Id) per iteration independently, and θULN0 = θULN(t = 0). Note that the errors from the SGD algorithm to its continuous-time diffusion process and from the continuous-time dynamics to its discretization are bounded under weak convergence (Hu et al., 2019a). In this way, we can use the trajectory of the discrete-time dynamics θ̄ULNk to analyze the behaviors of SGD algorithm θULNk over iterations.
Implicit Regularizer Influenced by Unbiased Random Label Noises Compared the stochastic gradient with unbiased random label noises g̃k(θ) and the stochastic gradient based on the labelnoiseless losses, we find an additional noise term ξULNk (θ) as the implicit regularizer.
To interpret ξULNk (θ), we first define the diffusion process of SGD based on Label-NoiseLess losses i.e., L∗i (θ) for 1 ≤ i ≤ N as dθLNL = − 1N ∑N i=1∇L∗i (θLNL)dt + √ η B ( ΣSGDN (θ LNL) ) 1
2 W(t). Through comparing θULN(t) with θLNL(t), the effects of ξULNk (θ) over continuous-time form should be √ η/B(ΣULNN (θ)) 1/2dW (t). Then, in discrete-time, we could get results as follow.
Proposition 1 (The implicit regularizer ξULNk (θ)) The implicit regularizer of SGD with unbiased random label noises could be approximated (with O( √ η) approximation error due to discretization (Li et al., 2017)) as follow,
ξULNk (θ) ≈ √ η
B
( σ2
N N∑ i=1
∇θf(xi, θ)∇θf(xi, θ)> ) 1 2 zk, and zk ∼ N (0d, Id) . (15)
In this way, we can estimate the expected strength of the implicit regularizer ξULNk (θ) as follow,
Ezk‖ξULNk (θ)‖22 = ησ2
BN N∑ i=1 ‖∇θf(xi, θ)‖22 . (16)
In this way, we can conclude that the effects of implicit regularization caused by unbiased random label noises for SGD is proportional to 1N ∑N i=1 ‖∇θf(xi, θ)‖ 2 2 – the average gradient norm of the neural network f(x, θ) over samples. Please refer to appendix for the proof.
Inference Stabilizer Here we extend the existing results on SGD (Zhu et al., 2019; Wu et al., 2018) to understand Proposition 1 as follows. (1) Inference Stability - The gradient norm 1N ∑N i=1 ‖∇θf(xi, θ)‖22 = 1 N ∑N i=1 ‖ ∂ ∂θf(xi, θ)‖ 2 2 characterizes the variation of neural network output f(x, θ) based on samples xi (for 1 ≤ i ≤ N ) over the parameter interpolation around the point θ. Lower 1N ∑N i=1 ‖∇θf(xi, θ)‖22 comes higher stability of neural network f(x, θ) outputs against the (random) perturbations over parameters. (2) Escape and Converge - When the noise ξULNk (θ) is θ-dependent (section 4 would present a special case that ξULNk (θ) is θ-independent with OLS), we follow (Zhu et al., 2019) and suggest that the implicit regularizer help SGD escape from the point θ̃ with high neural network gradient norm 1 N ∑N i=1 ‖∇θf(xi, θ̃)‖22, as the scale of noise ξULNk (θ̃) is large. Reciprocally, we follow (Wu et al., 2018) and suggest that when the SGD with unbiased random label noises converges, the converging point θ∗ should be with small 1N ∑N i=1 ‖∇θf(xi, θ∗)‖22. (3) Performance Tuning - Considering ησ2/B as the coefficient balancing the implicit regularizer and vanilla SGD, one can regularize/penalize the SGD learning procedure with the fixed η and B more fiercely using a larger σ2. More specifically, we could expect to obtain solutions with lower 1 N ∑N i=1 ‖∇θf(xi, θ)‖22 or higher inference stability of neural networks, as regularization effects become stronger when σ2 increases.

heading: 4 IMPLICIT REGULARIZATION EFFECTS TO LINEAR REGRESSION
text: Here, we consider a special example of SGD with unbiased random label noises using linear regression, where a simple quadratic loss function is considered for OLS, such that
β̂OLS ← argmin β∈Rd
{ 1
N N∑ i=1 L̃i(β) := 1 N N∑ i=1 ( x>i β − ỹi
)2} , (17)
where samples are generated through ỹi = x>i β ∗+ εi, E[εi] = 0 and var[εi] = σ2. Note that in this section, we replace the notation of θ with β to present the parameters of linear regression models.
Learning Dynamics and Implicit Regularization Effects The continuous-time diffusion processes for SGD algorithms with and without unbiased label noises are as follow
dβULN(t) = − 1 N N∑ i=1 xi(x > i β ULN(t)− x>i β∗)dt+ √ η B ΣSGDN (β ULN(t))1/2dW1(t)
+
√ η
B ΣULNN (β ULN(t))1/2dW2(t)
dβLNL(t) = − 1 N N∑ i=1 xi(x > i β LNL(t)− x>i β∗)dt+ √ η B ΣSGDN (β LNL(t))1/2dW (t)
(18)
where βULN(t) and βLNL(t) refer to the SGD dynamics for OLS under Unbiased Label Noises and Label NoiseLess settings. We then denote the sample covariance matrix of N samples as Σ̄N = 1 N ∑N i=1 xix > i . Matrices Σ SGD(β(t)) and ΣULN(β(t)) in this case are defined as
ΣSGDN (β) = 1
N N∑ i=1 ( xix > i β − Σ̄Nβ ) ( xix > i β − Σ̄Nβ )> and ΣULNN (β) = σ 2Σ̄N , (19)
which are both time-homogeneous. Compared to βLNL(t), the dynamics βULN(t) incorporates an additional noise term √ η BΣ ULN N (β ULN(t))1/2dW2(t) which affects the dynamics.
Proposition 2 (Implicit Regularization on OLS) In this way, we could approximate the implicit regularizer of SGD with the random label noises for OLS through discretization such as,√
η B ΣULNN (β
ULN(t))1/2dW2(t)⇒ ξULNk (β) ≈ √ ησ2
B
( Σ̄N )1/2 zk, and zk ∼ N (0d, Id), (20)
which is independent with β and k (the time). According to (Berthier et al., 2020), SGD for noiseless linear regression would asymptotically converge to the optimal solution β∗. With an additional noise term ξULNk (β) and the single optima β
∗ (for both noisy and noiseless losses), we could conclude that when k →∞, SGD with unbiased random label noises would converge to a distribution centered at β∗. The distribution would tend to be a Gaussian distribution when σ2 is significant (we could not ignore the effects of stochastic gradient noise terms of noiseless loss to the overall distribution), as the term √ σ2η/BΣ̄
1/2 N dW (t) corresponds to a Gaussian distribution. The span and shape of the
distribution are controlled by σ2 and Σ̄N when η and B are constant.
Numerical Validation To validate Proposition 2, we carry out numerical evaluation using synthesize data to simply visualize the dynamics over iteration of SGD algorithms with label-noisy OLS and label-noiseless OLS. In our experiments, we use 100 random samples realized from a 2-dimension Gaussian distribution Xi ∼ N (0,Σ1,2) for 1 ≤ i ≤ 100, where Σ1,2 is an symmetric covariance matrix controlling the random sample generation. To add the noises to the labels, we first drawn 100 copies of random noises from the normal distribution with the given variance εi ∼ N (0, σ2), then we setup the OLS problem with (Xi, Ỹi) pairs using Ỹi = X>i β∗ + εi and β∗ = [1, 1]> and various settings of σ2 and Σ1,2. We setup the SGD algorithms with the fixed learning rate η = 0.01, and bath size B = 5, with the total number of iterations K = 1, 000, 000 to visualize the complete paths.
Figure 1 presents the results of numerical validations. In Figure 1(a)–(d), we gradually increases the variances of label noises σ2 from 0.25 to 2.0, where we can observe (1) SGD over label-noiseless OLS converges to the optimal solution β∗ = [1.0, 1.0]> in a fast manner, (2) SGD over OLS with unbiased random label noises would asymptotically converge to a distribution centered at the optimal point, and (3) when σ2 increases, the span of the converging distribution becomes larger. In Figure 1(e)–(h), we use four settings of Σ1,2, where we can see (4) no matter how Σ1,2 is set for OLS problems, the SGD with unbiased random label noises would asymptotically converge to a distribution centered at the optimal point. Compared the results in (e) with(f), we can find that, when the trace of Σ1,2 increases, the span of converging distributions would increases. Furthermore, (5) the shapes of converging distributions depend on Σ1,2. In Figure 1(g), when we place the principal component of Σ1,2 onto the vertical axis (i.e., ΣVer = [[10, 0]>, [0, 100]>]), the distribution lays on
the vertical axis principally. Figure 1(h) demonstrates the opposite layout of the distribution, when we set ΣHor = [[100, 0]>, [0, 10]>] as Σ1,2. The scale and shape of the converging distribution backups our theoretical investigation in Eq 20.
Note that the unbiased random label noises are added to the labels prior to the learning procedure. In this setting, it is the mini-batch sampler of SGD that “dynamizes” the noises and influences the dynamics of SGD through forming the implicit regularizer.

heading: 5 IMPLICIT REGULARIZATION EFFECTS TO DEEP NEURAL NETWORKS
text: 0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
5000
10000
15000
20000
25000
30000
35000
40000
T ra
ce
Gaussian Symmetric
0.0 0.2 0.4 0.6
(a) G.N. (SVHN)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
10000
20000
30000
40000
50000
60000
T ra
ce
Gaussian Symmetric
0.0 0.2 0.4 0.6
(b) G.N. (CIFAR10)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
200000
300000
400000
500000
600000
T ra
ce
Gaussian Symmetric
0.0 0.2 0.4 0.6
(c) G.N. (CIFAR100)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
95.4
95.5
95.6
95.7
95.8
A cc
u ra
cy
Gaussian Symmetric w/o noise teacher (original model)
0.0 0.2 0.4 0.6 0.8
(d) Val. (SVHN)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
93.25
93.50
93.75
94.00
94.25
94.50
94.75
A cc
u ra
cy
Gaussian Symmetric w/o noise teacher (original model)
0.0 0.2 0.4 0.6
(e) Val. (CIFAR10)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
72.0
72.5
73.0
73.5
74.0
74.5
A cc
u ra
cy
Gaussian Symmetric w/o noise teacher (original model)
0.0 0.2 0.4 0.6
(f) Val. (CIFAR100)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
96.4
96.5
96.6
96.7
96.8
A cc
u ra
cy
Gaussian Symmetric w/o noise teacher (original model)
0.0 0.2 0.4 0.6 0.8
(g) Test. (SVHN)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
93.6
93.8
94.0
94.2
94.4
94.6
A cc
u ra
cy
Gaussian Symmetric w/o noise teacher (original model)
0.0 0.2 0.4 0.6
(h) Test. (CIFAR10)
0 2 4 6 8 10
Scale of label noises (e.g., p or σ2)
74.0
74.5
75.0
75.5
76.0
76.5
77.0
A cc
u ra
cy
Gaussian Symmetric w/o noise teacher (original model)
0.0 0.2 0.4 0.6
(i) Test. (CIFAR100) Figure 2: Gradient Norms, Validation Accuracy, and Testing Accuracy in Noisy Self-Distillation using ResNet-56 with varying scale of label noises (e.g., p and σ2).
Given a well-trained model, Noisy Self-Distillation algorithms (Zhang et al., 2019a; Xu et al., 2020; Kim et al., 2020; Xie et al., 2020) intend to further improve the performance of a model through learning from the “soft label” outputs (i.e., logits) of the model (as the teacher). Furthermore, some practices found that the self-distillation could be further improved through incorporating certain randomness and stochasticity in the training procedure so as to obtain better generalization performance (Xie et al., 2020; Kim et al., 2020). In this work, we study the way that directly adds random label noises to the logit outputs of the pre-trained model so as to improve the self-distillation (Han et al., 2018). More specifically, we study two well-known strategies for additive noises as follow.
(1) Gaussian Noises. Given a pretrained model with L-dimensional
logit output, for every iteration of self-distillation, this simple method that draw random vectors
from a L-dimensional Gaussian distribution N (0L, σ2IL), adds the vectors to the logit outputs of the model, and makes the student model learn from the noisy outputs. Note that in our analysis, we assume the output of the model is single dimension while, in self-distillation, the logit labels are with multiple dimensions. Thus, the diagonal matrix σ2IL now refers to the complete form the variances and σ2 controls the scale.
(2) Symmetric Noises.. Basically, this strategy is derived from (Han et al., 2018) that generates noises through randomly swapping the values of logit output among the L dimensions. Specifically, in every iteration of self-distillation, given a swap-probability p, every logit output (denoted as y here) from the pre-trained model, and every dimension of logit output denoted as yl, the strategy in probability p swaps the logit value in the dimension that corresponds to yl with any other dimension ym 6=l in equal prior (i.e., in (L−1)−1 probability). In the rest 1−p probability, the strategy remains the original logit output there. In this way, the new noisy label ỹ is with expectation E[ỹ] as follow,
E[ỹl] = (1− p) · yl + p · ∑ ∀m6=l ym
L− 1 (21)
This strategy introduces explicit bias to the original logit outputs. However, when we consider the expectation E[ỹ] as the innovative soft label, then the random noise around the new soft label is still unbiased as E[ỹ − E[ỹ]] = 0 for all dimensions. Note that this noise is not the symmetric noises studied for robust learning (Wang et al., 2019).
Figure 2 presents the results of above two methods with increasing scales of noises, i.e., increasing σ2 for Gaussian noises and increasing p for Symmetric noises. In Figure 2(a)–(c), we demonstrate that the gradient norms of neural networks 1N ‖∇θf(xi, θ)‖ 2 2 decrease with growing σ 2 and p for two strategies. The results backup our theoretical investigation , which means the model would be awarded high inferential stability, as the variation of neural network outputs against the potential random perturbation in parameters has been reduced by the regularization. In Figure 2(d)–(f) and (g)– (i), we plot the validation and testing accuracy of the models obtained under noisy self-distillation. The results show that (1) some of models have been improved through noisy self-distillation compared to the teacher model, (2) noisy self-distillation could obtain better performance than noiseless self-distillation, and (3) it is possible to select noisily self-distilled models using validation accuracy for better overall generalization performance. All results here are based on 200 epochs of noisy self-distillation.

heading: 6 DISCUSSION AND CONCLUSION
text: While previous studies primarily focus on the performance degradation caused by label noises or corrupted labels (Jiang et al., 2018; Li et al., 2020), we investigate the implicit regularization effects of random label noises, under mini-batch sampling settings of stochastic gradient descent (SGD). Specifically, we adopt the dynamical systems interpretation of SGD to analyze the learning procedure based on the quadratic loss with unbiased random label noises. We decompose the mini-batch stochastic gradient based on label-noisy losses into three parts in Eq. (11): (i)∇L∗(θ) – the true gradient of label-noiseless losses, (ii) ξ∗k(θ) – the stochastic gradient noise caused through mini-batch sampling over the label-noiseless losses, and (iii) ξULNk (θ) – the noise term influenced by the both random label noises and mini-batch sampling. Our research considers ξULNk (θ) as an implicit regularizer, and finds that effects of such implicit regularizer is to lower the gradient norm of the neural networks 1N ∑N i=1 ‖∇θf(xi, θ)‖22 over the learning procedure, where the gradient norm of neural networks here characterizes the variation/stability of the neural network outputs against the random perturbation around the parameters. In summary, the new implicit regularizer ξULNk (θ) helps SGD select a point with higher inference stability for convergence.
We carry out extensive experiments to validate our theoretical investigations. The numerical study with linear regression clearly illustrates the trajectories of SGD with and without unbiased random label noises, the observation coincides the SGD dynamics derived from our theories. Evaluation based on deep neural network shows that, in self-distillation settings, one can lower the gradient norm of neural networks, improve the inference stability of networks, and obtain better solutions, through iteratively adding noises to the outputs of teacher models. Note that we do not want to claim that the implicit regularization caused by the label noises would improve the generalization performance in this work. The experiments results well backup our theories.

heading: A APPENDIX
text: A.1 SKETCHED PROOFS IN PROPOSITION 1
To obtain Eq 16, we can use the simple vector-matrix-vector products transform that, for the random vector v and symmetric matrix A there has Ev[v>Av] = trace(AE[vv>]), such that
Ezk‖ξULNk (θ)‖22 = Ezk [ ξULNk (θ) >ξULNk (θ) ]
≈ ησ 2
B Ezk
[ z>k ( 1
N N∑ i=1
∇θf(xi, θ)∇θf(xi, θ)> ) zk ]
= ησ2
B trace
(( 1
N N∑ i=1 ∇θf(xi, θ)∇θf(xi, θ)> ) Ezk [zkz>k ] ) as Ezk [zkz>k ] = Id for zk ∼ N (0d, Id)
= ησ2
B trace
( 1
N N∑ i=1
∇θf(xi, θ)∇θf(xi, θ)> )
= ησ2
BN N∑ i=1 ‖∇θf(xi, θ)‖22
= ησ2
BN ∥∥∥∥ ∂∂θf(xi, θ) ∥∥∥∥2 2 .
(22)
A.2 TRAINING DETAILS FOR NOISY SELF-DISTILLATION WITH DEEP NEURAL NETWORKS
We choose the ResNet-56 (He et al., 2016), one of the most practical deep models, for conducting the experiments on three datasets: SVHN (Netzer et al., 2011), CIFAR10 and CIFAR100 (Krizhevsky et al., 2009). We follow the standard training procedure (He et al., 2016) for training a teacher model (original model). Specifically we train the model from scratch for 200 epochs and adopt the SGD optimizer with batch size 64 and momentum 0.9. The learning rate is set to 0.1 at the beginning of training and divided by 10 at 100th epoch and 150th epoch. A standard weight decay with a small regularization parameter (10−4) is applied. As for noiseless self-distillation, we follow the standard procedure (Hinton et al., 2015) for distilling knowledge from the teacher to a student of the same network structure. The training setting is the same as training the teacher model.
For noisy self-distillation, we continue to use the training setting except the labels are noised by the two types of noises as introduced in the main text. We choose the best scale of label noises using a validation set, where we divide the original training set into a new training set (80%) and a validation set (20%). A set of {0.1, 0.2, ..., 0.9, 1.0} is tried for the scale of symmetric noises. A set of {0.1, 0.5, 1.0, 2.0, 3.0, ..., 9.0, 10.0} is tried for the scale of Gaussian noises. For clarity, we also present the results using all the choices of scales of label noises on test set, where the original training set is used for training.
A.3 TRAINING PROCESS OF DEEP NEURAL NETWORKS
We show the evolution of training and test losses during the entire training procedure, and compare the settings of adding no label noises, symmetric and Gaussian noises for self-distillations. Figure 3 presents the results on the three datasets, i.e., SVHN, CIFAR10 and CIFAR100.

Review summaries:
This paper studies the implicit regularization effect that arises from using stochastic gradient descent with label noise and squared loss. They derive the expression for the implicit regularization term and show that it favors solutions which are stable against perturbations of the parameters. This paper validates their empirical findings using SGD on linear regression task with label noise. This paper also uses their results to study the self distillation technique.

The paper studies the implicit regularization effect of unbiased random label noise on the dynamics of stochastic gradient descent(SGD). More precisely, an unbiased random noise is added to the true labels and the paper aims to analyze the regularization effect of it. The paper shows that the unbiased label noise would favor convergence to points which