Kernel and Rich Regimes in Overparametrized Models

1 INTRODUCTION . A string of recent papers study neural networks trained with gradient descent in the “ kernel regime. ” They observe that , in a certain regime , networks trained with gradient descent behave as kernel methods ( Jacot et al. , 2018 ; Daniely et al. , 2016 ; Daniely , 2017 ) . This allows one to prove convergence to zero error solutions in overparametrized settings ( Du et al. , 2018 ; 2019 ; Allen-Zhu et al. , 2018 ) , and also implies gradient descent will converge to the minimum norm solution ( in the corresponding RKHS ) ( Chizat and Bach , 2018 ; Arora et al. , 2019b ; Mei et al. , 2019 ) and more generally that models inherit the inductive bias and generalization behavior of the RKHS . This suggests that , in a certain regime , deep models can be equivalently replaced by kernel methods with the “ right ” kernel , and deep learning boils down to a kernel method with a fixed kernel determined by the architecture and initialization , and thus it can only learn problems learnable by some kernel . This contrasts with other recent results that show how in deep models , including infinitely overparametrized networks , training with gradient descent induces an inductive bias that can not be represented as an RKHS norm . For example , analytic and/or empirical results suggest that gradient descent on deep linear convolutional networks implicitly biases toward minimizing the Lp bridge penalty , for p = 2/depth ≤ 1 , in the frequency domain ( Gunasekar et al. , 2018b ) ; weight decay on an infinite width single input ReLU implicitly biases towards minimizing the second order total variations ∫ |f ′′ ( x ) | dx of the learned function ( Savarese et al. , 2019 ) ; and gradient descent on a overparametrized matrix factorization , which can be thought of as a two layer linear network , induces nuclear norm minimization of the learned matrix ( Gunasekar et al. , 2017 ) and can ensure low rank matrix recovery ( Li et al. , 2018 ) . All these natural inductive biases ( Lp bridge penalty for p < 1 , total variation norm , nuclear norm ) are not Hilbert norms , and therefore can not be captured by a kernel . This suggests that training deep models with gradient descent can behave very differently from kernel methods , and have much richer inductive biases . One might then ask whether the kernel approximation indeed captures the behavior of deep learning in a relevant and interesting regime , or does the success of deep learning come when learning escapes this regime ? In order to understand this , we must first carefully understand when each of these regimes hold , and how the transition between the “ kernel ” regime and the “ rich ” regime happens . Some investigations of the kernel regime emphasized the number of parameters ( “ width ” ) going to infinity as leading to this regime . However Chizat and Bach ( 2018 ) identified the scale of the model as a quantity controlling entry into the kernel regime . Their results suggest that for any number of parameters ( any width ) , a model can be approximated by a kernel when its scale at initialization goes to infinity ( see details in Section 3 ) . Considering models with increasing ( or infinite ) width , the relevant regime ( kernel or rich ) is determined by how the scaling at initialization behaves as the width goes to infinity . In this paper we elaborate and expand of this view , carefully studying how the scale of initialization effects the model behaviour for D-homogeneous models . In Section 4 we provide a complete and detailed study for a simple 2-homogeneous model that can be viewed as linear regression with squared parametrization , or as a “ diagonal ” linear neural network . For this model we can exactly characterize the implicit bias of training with gradient descent , as a function of the scale α of initialization , and see how this implicit bias becomes the ` 2 norm in the α → ∞ kernel regime , but the ` 1 norm in the α → 0 rich regime . We can therefore understand how , e.g . for a high dimensional sparse regression problem , where it is necessary to discover the relevant features , we can get good generalization when the initialization scale α is small , but not when α is large . Deeper networks corresponds to higher orders of homogeneity , and so in Section 5 we extend our study to a D-homogeneous model , studying the effects of D. In Sections 6 and 7 , we demonstrate similar transitions experimentally in matrix factorization and non-linear networks . 2 SETUP AND PRELIMINARIES . We consider models f : Rp × X → R which map parameters w ∈ Rp and examples x ∈ X to predictions f ( w , x ) ∈ R. We denote the predictor implemented by the parameters w as hw = F ( w ) such that hw ( x ) = f ( w , x ) . Much of our focus will on models , such a linear networks , which are linear in x ( but not on the parameters w ! ) , in which case F ( w ) ∈ X ∗ is a linear predictor and can be represented as a vector βw with f ( w , x ) = 〈βw , x〉 . Such models are essentially alternate parametrizations of linear models , but as we shall see that change of parametrization is crucial . In this paper , we consider models that are D-positive homogeneous in the parameters w , for some integerD ≥ 1 , meaning that for any c ∈ R+ , F ( c·w ) = cDF ( w ) and f ( c·w , x ) = cDf ( w , x ) . We refer to such models simply as D-homogeneous . Many interesting model classes have this property , including multi-layer ReLU networks with fully connected and convolutional layers , layered linear neural networks , and matrix factorization where D corresponds to the depth of the network . Consider a training set { ( x ( n ) , y ( n ) ) } Nn=1 consisting of N examples of input label pairs . For a given loss function ` : R × R → R , the loss of the model parametrized by w is L ( w ) = L ( F ( w ) ) = ∑N n=1 ` ( f ( w , x ( n ) ) , y ( n ) ) . We will focus mostly on the squared loss ` sq ( ŷ , y ) = ( ŷ − y ) 2 . We slightly abuse notation and use f ( w , X ) ∈ RN to denote the vector of predictions [ f ( w , x ( 1 ) ) , . . . , f ( w , x ( N ) ) ] and so for the squared loss we can write L ( w ) = ‖f ( w , X ) − y‖22 , where y ∈ RN is the vector of target labels . Minimizing the loss L ( w ) using gradient descent amounts to iteratively updating the parameters w ( k + 1 ) = w ( k ) − η∇L ( w ( k ) ) . ( 1 ) We consider gradient descent with infinitesimally small stepsize η , i.e . gradient flow dynamics ẇ ( t ) = −∇L ( w ( t ) ) . ( 2 ) We are particularly interested in the scale of initialization and we capture it through a scalar parameter α ∈ R+ . For scale α , we will denote by wα ( t ) the gradient flow path ( 2 ) with the initial condition wα ( 0 ) = αw0 for some fixed w0 . We use hα ( t ) = F ( wα ( t ) ) , and for linear predictors βα ( t ) = βwα ( t ) , to denote the dynamics on the predictor F ( w ) induced by the gradient flow on w. In many cases , we expect the dynamics to converge to a minimizer of L ( w ) , though proving this happens will not be our main focus . Rather , we are interested in the underdetermined case , N p , where there are generally many minimizers of L ( w ) , all with f ( w , X ) = y and L ( w ) = 0 . Our main focus is which of the many minimizers does gradient flow converge to . That is , we want to characterize wα ( ∞ ) = limt→∞wα ( t ) or , more importantly , the predictor hα ( ∞ ) = F ( wα ( ∞ ) ) or βα ( ∞ ) = βwα ( ∞ ) we converge to , and how these depend on the scale α . In underdetermined problems , where there are many zero error solutions , simply fitting the data using the model does not provide enough inductive bias to ensure generalization . But in many cases , the specific solution reached by gradient flow ( or some other optimization procedure ) has special structure , or minimizes some implicit regularizer , and this structure or regularizer provides the needed inductive bias ( Gunasekar et al. , 2018b ; a ; Soudry et al. , 2018 ; Ji and Telgarsky , 2018 ) . 3 THE KERNEL REGIME . Gradient descent/flow considers only the first-order approximation of the model w.r.t . w : f ( w , x ) = f ( w ( t ) , x ) + 〈w −w ( t ) , ∇wf ( w ( t ) , x ) 〉+O ( ‖w −w ( t ) ‖2 ) . ( 3 ) That is , locally around any w ( t ) , gradient flow operates on the model as if it were an affine model f ( w , x ) ≈ f0 ( x ) + 〈 w , φw ( t ) ( x ) 〉 with feature map φw ( t ) ( x ) = ∇wf ( w ( t ) , x ) , corresponding to the tangent kernel Kw ( t ) ( x , x′ ) = 〈∇wf ( w ( t ) , x ) , ∇wf ( w ( t ) , x ) 〉 ( Jacot et al. , 2018 ; Zou et al. , 2018 ; Yang , 2019 ; Lee et al. , 2019 ) . Of particular interest is the tangent kernel at initialization , Kwα ( 0 ) = α 2 ( D−1 ) K0 where we denote K0 = Kw0 . The “ kernel regime ” refers to a situation in which the tangent kernel Kw ( t ) does not change over the course of optimization , and less formally to the regime where it does not change significantly , i.e . where ∀tKw ( t ) ≈ Kw ( 0 ) . In this regime , training the model is exactly equivalent to training an affine model f̃ ( w , x ) = αDf ( w0 , x ) + 〈 w , αD−1φ0 ( x ) 〉 with kernelized gradient descent/flow with the kernel α2 ( D−1 ) K0 and a “ bias term ” of αDf ( w0 , x ) . To avoid handling this bias term , and in particular its scaling , Chizat and Bach ( 2018 ) suggest using “ unbiased ” initializations such that F ( w0 ) = 0 , so that the bias term vanishes . This can often be achieved by replicating units or components with opposite signs at initialization , which is the approach we use here ( see Sections 4–6 for examples and details ) . For underdetermined problem with multiple solutions f ( w , X ) = y , unbiased kernel gradient flow ( or gradient descent ) converges to the minimum norm solution ĥK = arg minh ( X ) =y ‖h‖K , where ‖h‖K is the RKHS norm corresponding to the kernel . And so , in the kernel regime , we will have that h ( ∞ ) = ĥK0 , and the implicit bias of training is precisely given by the kernel . When does the “ kernel regime ” happen ? Chizat and Bach ( 2018 ) showed that for any homogeneous1 model satisfying some technical conditions , the kernel regime is reached as α→∞ . That is , as we increase the scale of initialization , the dynamics converge to the kernel gradient flow dynamics with the kernel K0 , and we have limα→∞ hα ( ∞ ) = ĥK . In Sections 4 and 5 we prove this limit directly for our specific models , and we also demonstrate it empirically for matrix factorization and deep networks in Sections 6 and 7 . In contrast , and as we shall see in later sections , the α→ 0 small initialization limit often leads to a very different and rich inductive bias , e.g . inducing sparsity or low-rank structure ( Gunasekar et al. , 2017 ; Li et al. , 2018 ; Gunasekar et al. , 2018b ) , that allows for generalization in many settings where kernel methods would not . We refer to this limit reached as α→ 0 as the “ rich regime. ” This regime is also referred to as the “ active ” or “ adaptive ” regime ( Chizat and Bach , 2018 ) since the tangent kernel Kw ( t ) changes over the course of training , in a sense adapting to the data . We argue that this regime is the one that truly allows us to exploit the power of depth , and thus is the more relevant regime for understanding the success of deep learning .

Summary: I really appreciated this paper. It discusses a very complex question ("Are we learning in a kernel regime, or in a rich regime where features are identified") by looking at perhaps the simplest model the authors could think of, and then study in detail the model. And how simple it turns out to be: just a linear regression with a twist. All in all, the paper is indeed is a clear demonstration that the differences between"Kernel" regime and one where some actual Learning is done can be demonstrated on simple examples. It is also the simplest model where one can observe  a non-trivial inductive bias and 'implicit regularisation'