Kernel and Rich Regimes in Overparametrized Models

1 INTRODUCTION . A string of recent papers study neural networks trained with gradient descent in the “ kernel regime. ” They observe that , in a certain regime , networks trained with gradient descent behave as kernel methods ( Jacot et al. , 2018 ; Daniely et al. , 2016 ; Daniely , 2017 ) . This allows one to prove convergence to zero error solutions in overparametrized settings ( Du et al. , 2018 ; 2019 ; Allen-Zhu et al. , 2018 ) , and also implies gradient descent will converge to the minimum norm solution ( in the corresponding RKHS ) ( Chizat and Bach , 2018 ; Arora et al. , 2019b ; Mei et al. , 2019 ) and more generally that models inherit the inductive bias and generalization behavior of the RKHS . This suggests that , in a certain regime , deep models can be equivalently replaced by kernel methods with the “ right ” kernel , and deep learning boils down to a kernel method with a fixed kernel determined by the architecture and initialization , and thus it can only learn problems learnable by some kernel . This contrasts with other recent results that show how in deep models , including infinitely overparametrized networks , training with gradient descent induces an inductive bias that can not be represented as an RKHS norm . For example , analytic and/or empirical results suggest that gradient descent on deep linear convolutional networks implicitly biases toward minimizing the Lp bridge penalty , for p = 2/depth ≤ 1 , in the frequency domain ( Gunasekar et al. , 2018b ) ; weight decay on an infinite width single input ReLU implicitly biases towards minimizing the second order total variations ∫ |f ′′ ( x ) | dx of the learned function ( Savarese et al. , 2019 ) ; and gradient descent on a overparametrized matrix factorization , which can be thought of as a two layer linear network , induces nuclear norm minimization of the learned matrix ( Gunasekar et al. , 2017 ) and can ensure low rank matrix recovery ( Li et al. , 2018 ) . All these natural inductive biases ( Lp bridge penalty for p < 1 , total variation norm , nuclear norm ) are not Hilbert norms , and therefore can not be captured by a kernel . This suggests that training deep models with gradient descent can behave very differently from kernel methods , and have much richer inductive biases . One might then ask whether the kernel approximation indeed captures the behavior of deep learning in a relevant and interesting regime , or does the success of deep learning come when learning escapes this regime ? In order to understand this , we must first carefully understand when each of these regimes hold , and how the transition between the “ kernel ” regime and the “ rich ” regime happens . Some investigations of the kernel regime emphasized the number of parameters ( “ width ” ) going to infinity as leading to this regime . However Chizat and Bach ( 2018 ) identified the scale of the model as a quantity controlling entry into the kernel regime . Their results suggest that for any number of parameters ( any width ) , a model can be approximated by a kernel when its scale at initialization goes to infinity ( see details in Section 3 ) . Considering models with increasing ( or infinite ) width , the relevant regime ( kernel or rich ) is determined by how the scaling at initialization behaves as the width goes to infinity . In this paper we elaborate and expand of this view , carefully studying how the scale of initialization effects the model behaviour for D-homogeneous models . In Section 4 we provide a complete and detailed study for a simple 2-homogeneous model that can be viewed as linear regression with squared parametrization , or as a “ diagonal ” linear neural network . For this model we can exactly characterize the implicit bias of training with gradient descent , as a function of the scale α of initialization , and see how this implicit bias becomes the ` 2 norm in the α → ∞ kernel regime , but the ` 1 norm in the α → 0 rich regime . We can therefore understand how , e.g . for a high dimensional sparse regression problem , where it is necessary to discover the relevant features , we can get good generalization when the initialization scale α is small , but not when α is large . Deeper networks corresponds to higher orders of homogeneity , and so in Section 5 we extend our study to a D-homogeneous model , studying the effects of D. In Sections 6 and 7 , we demonstrate similar transitions experimentally in matrix factorization and non-linear networks . 2 SETUP AND PRELIMINARIES . We consider models f : Rp × X → R which map parameters w ∈ Rp and examples x ∈ X to predictions f ( w , x ) ∈ R. We denote the predictor implemented by the parameters w as hw = F ( w ) such that hw ( x ) = f ( w , x ) . Much of our focus will on models , such a linear networks , which are linear in x ( but not on the parameters w ! ) , in which case F ( w ) ∈ X ∗ is a linear predictor and can be represented as a vector βw with f ( w , x ) = 〈βw , x〉 . Such models are essentially alternate parametrizations of linear models , but as we shall see that change of parametrization is crucial . In this paper , we consider models that are D-positive homogeneous in the parameters w , for some integerD ≥ 1 , meaning that for any c ∈ R+ , F ( c·w ) = cDF ( w ) and f ( c·w , x ) = cDf ( w , x ) . We refer to such models simply as D-homogeneous . Many interesting model classes have this property , including multi-layer ReLU networks with fully connected and convolutional layers , layered linear neural networks , and matrix factorization where D corresponds to the depth of the network . Consider a training set { ( x ( n ) , y ( n ) ) } Nn=1 consisting of N examples of input label pairs . For a given loss function ` : R × R → R , the loss of the model parametrized by w is L ( w ) = L ( F ( w ) ) = ∑N n=1 ` ( f ( w , x ( n ) ) , y ( n ) ) . We will focus mostly on the squared loss ` sq ( ŷ , y ) = ( ŷ − y ) 2 . We slightly abuse notation and use f ( w , X ) ∈ RN to denote the vector of predictions [ f ( w , x ( 1 ) ) , . . . , f ( w , x ( N ) ) ] and so for the squared loss we can write L ( w ) = ‖f ( w , X ) − y‖22 , where y ∈ RN is the vector of target labels . Minimizing the loss L ( w ) using gradient descent amounts to iteratively updating the parameters w ( k + 1 ) = w ( k ) − η∇L ( w ( k ) ) . ( 1 ) We consider gradient descent with infinitesimally small stepsize η , i.e . gradient flow dynamics ẇ ( t ) = −∇L ( w ( t ) ) . ( 2 ) We are particularly interested in the scale of initialization and we capture it through a scalar parameter α ∈ R+ . For scale α , we will denote by wα ( t ) the gradient flow path ( 2 ) with the initial condition wα ( 0 ) = αw0 for some fixed w0 . We use hα ( t ) = F ( wα ( t ) ) , and for linear predictors βα ( t ) = βwα ( t ) , to denote the dynamics on the predictor F ( w ) induced by the gradient flow on w. In many cases , we expect the dynamics to converge to a minimizer of L ( w ) , though proving this happens will not be our main focus . Rather , we are interested in the underdetermined case , N p , where there are generally many minimizers of L ( w ) , all with f ( w , X ) = y and L ( w ) = 0 . Our main focus is which of the many minimizers does gradient flow converge to . That is , we want to characterize wα ( ∞ ) = limt→∞wα ( t ) or , more importantly , the predictor hα ( ∞ ) = F ( wα ( ∞ ) ) or βα ( ∞ ) = βwα ( ∞ ) we converge to , and how these depend on the scale α . In underdetermined problems , where there are many zero error solutions , simply fitting the data using the model does not provide enough inductive bias to ensure generalization . But in many cases , the specific solution reached by gradient flow ( or some other optimization procedure ) has special structure , or minimizes some implicit regularizer , and this structure or regularizer provides the needed inductive bias ( Gunasekar et al. , 2018b ; a ; Soudry et al. , 2018 ; Ji and Telgarsky , 2018 ) . 3 THE KERNEL REGIME . Gradient descent/flow considers only the first-order approximation of the model w.r.t . w : f ( w , x ) = f ( w ( t ) , x ) + 〈w −w ( t ) , ∇wf ( w ( t ) , x ) 〉+O ( ‖w −w ( t ) ‖2 ) . ( 3 ) That is , locally around any w ( t ) , gradient flow operates on the model as if it were an affine model f ( w , x ) ≈ f0 ( x ) + 〈 w , φw ( t ) ( x ) 〉 with feature map φw ( t ) ( x ) = ∇wf ( w ( t ) , x ) , corresponding to the tangent kernel Kw ( t ) ( x , x′ ) = 〈∇wf ( w ( t ) , x ) , ∇wf ( w ( t ) , x ) 〉 ( Jacot et al. , 2018 ; Zou et al. , 2018 ; Yang , 2019 ; Lee et al. , 2019 ) . Of particular interest is the tangent kernel at initialization , Kwα ( 0 ) = α 2 ( D−1 ) K0 where we denote K0 = Kw0 . The “ kernel regime ” refers to a situation in which the tangent kernel Kw ( t ) does not change over the course of optimization , and less formally to the regime where it does not change significantly , i.e . where ∀tKw ( t ) ≈ Kw ( 0 ) . In this regime , training the model is exactly equivalent to training an affine model f̃ ( w , x ) = αDf ( w0 , x ) + 〈 w , αD−1φ0 ( x ) 〉 with kernelized gradient descent/flow with the kernel α2 ( D−1 ) K0 and a “ bias term ” of αDf ( w0 , x ) . To avoid handling this bias term , and in particular its scaling , Chizat and Bach ( 2018 ) suggest using “ unbiased ” initializations such that F ( w0 ) = 0 , so that the bias term vanishes . This can often be achieved by replicating units or components with opposite signs at initialization , which is the approach we use here ( see Sections 4–6 for examples and details ) . For underdetermined problem with multiple solutions f ( w , X ) = y , unbiased kernel gradient flow ( or gradient descent ) converges to the minimum norm solution ĥK = arg minh ( X ) =y ‖h‖K , where ‖h‖K is the RKHS norm corresponding to the kernel . And so , in the kernel regime , we will have that h ( ∞ ) = ĥK0 , and the implicit bias of training is precisely given by the kernel . When does the “ kernel regime ” happen ? Chizat and Bach ( 2018 ) showed that for any homogeneous1 model satisfying some technical conditions , the kernel regime is reached as α→∞ . That is , as we increase the scale of initialization , the dynamics converge to the kernel gradient flow dynamics with the kernel K0 , and we have limα→∞ hα ( ∞ ) = ĥK . In Sections 4 and 5 we prove this limit directly for our specific models , and we also demonstrate it empirically for matrix factorization and deep networks in Sections 6 and 7 . In contrast , and as we shall see in later sections , the α→ 0 small initialization limit often leads to a very different and rich inductive bias , e.g . inducing sparsity or low-rank structure ( Gunasekar et al. , 2017 ; Li et al. , 2018 ; Gunasekar et al. , 2018b ) , that allows for generalization in many settings where kernel methods would not . We refer to this limit reached as α→ 0 as the “ rich regime. ” This regime is also referred to as the “ active ” or “ adaptive ” regime ( Chizat and Bach , 2018 ) since the tangent kernel Kw ( t ) changes over the course of training , in a sense adapting to the data . We argue that this regime is the one that truly allows us to exploit the power of depth , and thus is the more relevant regime for understanding the success of deep learning .

Summary: This paper analyzes an inductive bias of the gradient flow for diagonal two-or higher-homogeneous models and characterizes a limit point depending on the initialization scale of parameters. Concretely, the paper shows that the gradient flow converges to an interpolator attaining minimum L1- (or L2-norm) when the scale is small (or large). In addition, these analyses are well verified empirically on MNIST and CIFAR-10 datasets.