A Mechanism of Implicit Regularization in Deep Learning

1 INTRODUCTION . Deep Neural Networks ( DNNs ) have demonstrated dominating performance in numerous machine learning tasks , and it shows great generalization performance in the over-parameterized regime . Theoretically , mechanisms of implicit regularization , which is considered as an important factor of such generalization performance in over-parameterized DNNs , still remain unknown ( Zhang et al. , 2016 ; Neyshabur , 2017 ) . In the over-parameterized regime , recent studies report that generalization bounds for DNNs can be obtained by replacing the neural network with its linear approximation model with respect to weight parameters at initialization . Most of these studies rely on the connection between deep learning and neural tangent kernels ( NTKs ) ( Daniely et al. , 2016 ; Jacot et al. , 2018 ; Arora et al. , 2019b ) , which characterizes the dynamics of network outputs throughout gradient descent training in the infinite width limit . However , a source of implicit regularization in over-parameterized DNNs has not been identified . Recent empirical and theoretical results indicate that generalization performance and implicit regularization of over-parameterized DNNs can not be captured by NTK analysis ( Wei et al. , 2018 ; Allen-Zhu & Li , 2019 ; Woodworth et al. , 2019 ; Geiger et al. , 2019 ) . Understanding how implicit regularization properly controls the superfluous expressive power of over-parameterized DNNs gives us new insights into the theoretical analysis in deep learning . This leads to the first question : Question 1 . What kind of low complexity is caused by implicit regularization in deep learning ? Linear networks without activation functions are important subject , and there are a number of theoretical works on the implicit regularization in over-parameterized neural networks mainly focusing on linear models ( Ji & Telgarsky , 2018 ; Gidel et al. , 2019 ; Arora et al. , 2019a ) . In contrast , whole properties of over-parameterized DNNs that may result from nonlinearity of activation functions can not be captured by the approximated linear models , specifically , the kernel regression predictor using the NTK . However , some mechanisms of the implicit regularization can depend on nonlinearity . This leads to the next question : Question 2 . Can we identify a mechanism of implicit regularization that depends on nonlinearity of activation functions ? Now for optimization algorithms , the training dynamics of full batch gradient descent ( GD ) is better understood although GD is too expensive for most applications and one often uses stochastic gra- dient decent ( SGD ) instead . In most cases , the authors used GD to derive their results by the NTK analysis . Recent work showed that without any structural assumptions about the data distribution , two-layer or three-layer over-parameterized networks trained by SGD can learn C∞-class functions ( Allen-Zhu et al. , 2018a ; Arora et al. , 2019c ) . However , it is not clear the relation between generalization and implicit regularization for DNNs optimized by SGD . Towards this end , the following question is also unsolved : Question 3 . Is it possible to obtain provable generalization bounds based on the implicit regularization for DNNs optimized by SGD ? To answer these questions , we introduce a novel analysis for DNNs and characterize a mechanism of implicit regularization that caused by nonlinearity of ReLU activation . Our results indicate that the DNNs ( trained by SGD ) interpolate almost linearly between data points , which leads to the low complexity solutions in the over-parameterized regime . Accordingly , we prove that SGD with random initialization can learn a class of continuously differentiable functions with generalization error bounds of the order of O ( n−2 ) ( n : the number of samples ) , which is independent of the NTK analysis . In order to introduce our analysis for implicit regularization , let us focus on the DNN output on a one-dimensional linear path between training points . We define it as x ( s ) : = ( 1 − s ) x ( p ) + sx ( q ) ( s ∈ [ 0 , 1 ] ) , where x ( p ) and x ( q ) are training points . The corresponding function of the DNN with ReLU ( ReLU DNN ) is continuous piecewise linear on the linear path x ( s ) . In the hidden layer , the corresponding function gl ( x ) of each unit in the l-th layer is also continuous piecewise linear on the linear path x ( s ) , that is , a composite function ( gl ◦x ) ( s ) is continuous piecewise linear . Since DNN nonlinearity is linked to breakpoints , which is caused by ReLU activation , the set of the breakpoints plays a key role in the network behavior . These kind of breakpoints are also known as kinks or knots ( Steinwart , 2019 ) . Since ( gl ◦x ) ( s ) is continuous piecewise linear , the gradient of ( gl ◦x ) ( s ) has gaps at the breakpoints , which we call gradient gaps . We focus on gradient gaps with respect to the parameter s.1 Our key finding is that the gradient gaps are a constant multiple of independent Gaussian random variables according to N ( 0 , 2/m ) , where m is the number of units in the hidden layer . Since the gradient of the unit ( gl ◦ x ) ( s ) is the sum of the gradient gaps , it is a “ Gaussian random walk ” 2 ( see Step 2 in the proof of Lemma 3 ) . The relation between the step size variance and the number of steps determines the Gaussian random walk behavior . In the corresponding function of the DNN , the step size variance is the variance of weights , and the number of steps is the number of breakpoints . In a regime where ( the number of steps × the step size variance ) ≤ O ( 1 ) , the Gaussian random walk strolls little from the origin with high probability ( see Figure 1 ) . Our results show that the ReLU DNN is in the same regime and “ simplicity ” of the gradient depends crucially on the number of breakpoints . Hanin & Rolnick ( 2019 ) proves that the average number of breakpoints 1 It is worth pointing out that in this paper , each gradient is a derivative not with respect to weight parameters but with respect to input parameters . 2 If { Xi } αi=1 are independent Gaussian random variables , then { Yi } αi=1 , where Yi : = X1+X2+ · · ·+Xi , is called a Gaussian random walk . is linear in the number of hidden units at initialization , and we also give a proof of an upper bound on the number of breakpoints even after training ( see §3.1 ) . In this work , we prove a priori generalization estimates , which is independent of the posterior data distribution , by analyzing the behavior of DNNs on the linear path x ( s ) . In other words , we show that in Theorem 2 , the difference between the network output function and its linear interpolation is evaluated only from the amount of the weight change . We note that in Theorem 2 , we dose not use the properties of the trained neural networks . Our key analysis is that the gradient of unit ( gl ◦x ) ( s ) is a “ Gaussian random walk ” and approximately equal to a straight line on the linear path x ( s ) , which we call random walk analysis . Our technique is based on a priori estimates that variation of the gradients between data points is extremely small and depends essentially on the value of the number of breakpoints times the variance of weights . Difference from NTK . Interestingly , our analysis idea is different from other previous work based on NTK . Our findings are some novel aspects as follows : • The NTK is defined using the gradient of the DNN output with respect to weight parameter space . In contrast , the linear approximation ( Lemma 3 in this paper ) is defined using the gradient of the DNN output with respect to input parameter space . In other words , the variables to be differentiated are different . • The random walk analysis indicates that over-parameterized ReLU DNNs interpolate almost linearly between the data points . For ReLU activation , since the NTK kernel mapping is not Lipschitz but 1/2-Hölder , it is difficult to obtain such a result in the NTK analysis without a tradeoff between smoothness and approximation ( Bietti & Mairal , 2019 ) . Our Contributions . In this work , we consider an L-layer over-parameterized ReLU neural network with l2 regression task using SGD from random initialization . We show the constitutive relation between implicit regularization and generalization , which enables us to provide new insights on the role of implicit regularization in deep learning . Our main contributions are as follows : • Our random walk analysis provides a priori estimates of low complexity in overparameterized deep neural networks , and directly indicates that the unit output between data points is properly controlled by weight initialization and SGD to keep connecting the data points almost straight , which is one of the underlying mechanisms of implicit regularization in the over-parameterized regime . • Our result suggests that implicit regularization is attributed to the nonlinearity of ReLU DNN , which is indicated by the fact that variance of weight and the number of breakpoints determine the Gaussian random walk behavior . • We also prove that in one-dimensional input case , SGD with random initialization can learn C1-class functions . Our generalization estimates are based on the implicit regularization . 2 PRELIMINARIES AND NOTATION . Notation . For n ∈ N , we let [ n ] = { 1 , 2 , . . . , n } . We use N ( µ , σ2 ) to denote the Gaussian distribution of mean µ and variance σ2 . We use ‖v‖2 to denote the Euclidean norm of a vector v , use ‖v‖F to denote the Frobenius norm of a vector v. When z is a sub-Gaussian random variable , we let ‖z‖ψ2 = inf { t > 0 | E [ exp ( z2/t2 ) ] ≤ 2 } to denote sub-Gaussian norm ( Vershynin , 2018 ) . For a vector v , we denote by [ v ] i or vi the i-th element of v. For a matrix M , we denote by [ M ] i , j or Mi , j the entry in the i-th row and j-th column of M , and we denote by [ M ] i the i-th row vector of M . We denote by 1 { E } the indicator function for the event E. ReLU activation is given by φ ( x ) = max { 0 , x } , and for a vector a ∈ Rn , we define φ ( a ) = ( φ ( a1 ) , φ ( a2 ) , . . . , φ ( an ) ) . Network structure . In this work , an L-layer fully-connected feed-forward ReLU neural network with m units in each hidden layer is given by f : Rd → Rc . For input x ∈ Rd , unit output gl ( x ) of each layer l ∈ [ L ] and network output f ( x ) are given by the following functions : g0 ( x ) : = Ax , gl ( x ) : = Wlφ ( gl−1 ( x ) ) ( l ∈ [ L ] ) , f ( x ) : = Bφ ( gL ( x ) ) , ( 1 ) where A ∈ Rm×d , Wl ∈ Rm×m ( l ∈ L ) , and B ∈ Rc×m are weight matrices . We assume the following Gaussian initialization : Ai , j ∼ N ( 0 , 2/m ) , Wli , j ∼ N ( 0 , 2/m ) , and Bi , j ∼ N ( 0 , 2/c ) . For input x and weight matrices −→ W : = ( W1 , W2 , . . . , WL ) , the network output f ( x ) is also denoted by f ( −→ W , x ) . In this work , we only update weights in −→ W and leave A and B at the random initialization . For input x ∈ Rd and l ∈ [ L ] , we denote by Gl ( x ) a diagonal matrix , which represents the activation pattern of the l-th layer , which we call an indicator matrix . More precisely , we define the i-th diagonal element as [ Gl ( x ) ] i , i : = 1 { gli ( x ) ≥0 } . Since the ReLU activation is positive homogeneous , we obtain the following equality : gl ( x ) = WlGl−1 ( x ) gl−1 ( x ) . Dataset and loss function . The data are generated from an unknown distribution D over ( x , y ) ∈ Rd × Rc , where x is the input data point and y is the label associated with this data point . We assume without loss of generality that for each input x = ( x1 , . . . , xd ) , using additional coordinates ( xd+1 , xd+2 ) , the replacement input x′′ : = ( x1 , . . . , xd , xd+1 , xd+2 ) is normalized so that ‖x′′‖2 = 1 and its last coordinate xd+2 = 1/ √ 2 . 3 we also use x to denote the replacement input x′′ ∈ Rd+2 . The training data Z : = { ( x ( 1 ) , y ( 1 ) ) , . . . , ( x ( n ) , y ( n ) ) } is given as n i.i.d . samples from D. We define the minimum distance of the training data : δ : = min { ‖x ( i ) −x ( j ) ‖2 : ∀i , j ∈ [ n ] , i 6= j } > 0 . For the l2 regression loss ` ( ŷ , y ) : = 12‖ŷ − y‖ 2 2 and a subset of training data Z ( τ ) ⊂ Z , we define our regression objective as follows : LZ ( τ ) ( −→ W ) : = E ( x , y ) ∼Z ( τ ) [ ` ( f ( −→ W , x ) , y ) ] . Stochastic gradient descent with Gaussian initialization . We use mini-batch SGD to train the network with a constant learning rate η > 0 , a batch size b and iteration number T . Let −→ W ( 0 ) : = ( W1 , ( 0 ) , W2 , ( 0 ) , . . . , WL , ( 0 ) ) , A , B be weight matrices generated from the above Gaussian initialization . Suppose we start at −−−→ W ( 0 ) and for each l ∈ [ L ] and t = 0 , 1 , . . . , T − 1 , Wl , ( t+1 ) = Wl , ( t ) − η · ∇WlLZ ( t ) ( −→ W ( t ) ) , ( 2 ) where Z ( t ) ⊂ Z is a mini-batch of size b . For input x , each layer l ∈ [ L ] and each step t ∈ [ T ] , we denote by gl , ( t ) ( x ) the unit output , f ( t ) ( x ) the network output , Wl , ( t ) the weight matrix , and Gl , ( t ) ( x ) the indicator matrix . In the above setting , recent paper ( Allen-Zhu et al. , 2018b ) shows that SGD can allow an overparameterized multi-layer network to attain arbitrarily low training error as follows : Theorem 1 ( Convergence of SGD ( Allen-Zhu et al. , 2018b ) ) . For any ε ∈ ( 0 , 1 ] , δ ∈ ( 0 , O ( 1/L ) ] and b ∈ [ n ] , let m ≥ Ω̃ ( poly ( n , L , δ−1 ) d b ) , η : = Θ ( bδd poly ( n , L ) m log2m ) , T = Θ ( poly ( n , L ) log2m bδ2 log ( n logm ε ) ) , and −→ W ( 0 ) , A , B are at random initialization . Then , it satisfies with probability at least 1− e−Ω ( log2m ) over randomness of Z ( 1 ) , . . . Z ( T ) : LZ ( −→ W ) ≤ ε , ‖Wl , ( t ) −Wl , ( 0 ) ‖F ≤ O ( λ logm√ m ) , ‖ [ Wl , ( t ) ] i − [ Wl , ( 0 ) ] i‖2 ≤ O ( λ logmm ) ( ∀t ∈ [ T ] ) , where λ : = n 3.5√c δ √ b . Our results on generalization also crucially depend on this analysis .

Summary: This paper suggests a new technique to analyze the implicit regularization caused by ReLU activations. They bound the generalization error by two terms: 1) one term that represents the distance between the trained network output and a piecewise linear function built based on the set of training points and 2) another term that represents the distance between the piecewise linear approximation and the desired target. The first term is bounded using a random walk type of analysis, which to the best of my knowledge is novel.