The Implicit Bias of Depth: How Incremental Learning Drives Generalization

1 INTRODUCTION . Neural networks have led to a breakthrough in modern machine learning , allowing us to efficiently learn highly expressive models that still generalize to unseen data . The theoretical reasons for this success are still unclear , as the generalization capabilities of neural networks defy the classic statistical learning theory bounds . Since these bounds , which depend solely on the capacity of the learned model , are unable to account for the success of neural networks , we must examine additional properties of the learning process . One such property is the optimization algorithm - while neural networks can express a multitude of possible ERM solutions for a given training set , gradient-based methods with the right initialization may be implicitly biased towards certain solutions which generalize . A possible way such an implicit bias may present itself , is if gradient-based methods were to search the hypothesis space for possible solutions of gradually increasing complexity . This would suggest that while the hypothesis space itself is extremely complex , our search strategy favors the simplest solutions and thus generalizes . One of the leading results along these lines has been by Saxe et al . ( 2013 ) , deriving an analytical solution for the gradient flow dynamics of deep linear networks and showing that for such models , the singular values converge at different rates , with larger values converging first . At the limit of infinitesimal initialization of the deep linear network , Gidel et al . ( 2019 ) show these dynamics exhibit a behavior of “ incremental learning ” - the singular values of the model are learned separately , one at a time . Our work generalizes these results to small but finite initialization scales . Incremental learning dynamics have also been explored in gradient descent applied to matrix completion and sensing with a factorized parameterization ( Gunasekar et al . ( 2017 ) , Arora et al . ( 2018 ) , Woodworth et al . ( 2019 ) ) . When initialized with small Gaussian weights and trained with a small learning rate , such a model is able to successfully recover the low-rank matrix which labeled the data , even if the problem is highly over-determined and no additional regularization is applied . In their proof of low-rank recovery for such models , Li et al . ( 2017 ) show that the model remains lowrank throughout the optimization process , leading to the successful generalization . Additionally , Arora et al . ( 2019 ) explore the dynamics of such models , showing the singular values are learned at different rates and that deeper models exhibit stronger incremental learning dynamics . Our work deals with a more simplified setting , allowing us to determine explicitly under which conditions depth leads to this dynamical phenomenon . Finally , the learning dynamics of nonlinear models have been studied as well . Combes et al . ( 2018 ) and Williams et al . ( 2019 ) study the gradient flow dynamics of shallow ReLU networks under restrictive distributional assumptions , Ronen et al . ( 2019 ) show that shallow networks learn functions of gradually increasing frequencies and Nakkiran et al . ( 2019 ) show how deep ReLU networks correlate with linear classifiers in the early stages of training . These findings , along with others , suggest that the generalization ability of deep networks is at least in part due to the incremental learning dynamics of gradient descent . Following this line of work , we begin by explicitly defining the notion of incremental learning for a toy model which exhibits this sort of behavior . Analyzing the dynamics of the model for gradient flow and gradient descent , we characterize the effect of the model ’ s depth and initialization scale on incremental learning , showing how deeper models allow for incremental learning in larger ( realistic ) initialization scales . Specifically , we show that a depth-2 model requires exponentially small initialization for incremental learning to occur , while deeper models only require the initialization to be polynomially small . Once incremental learning has been defined and characterized for the toy model , we generalize our results theoretically and empirically for larger linear and quadratic models . Examples of incremental learning in these models can be seen in figure 1 , which we discuss further in section 4 . 2 DYNAMICAL ANALYSIS OF A TOY MODEL . We begin by analyzing incremental learning for a simple model . This will allow us to gain a clear understanding of the phenomenon and the conditions for it , which we will later be able to apply to a variety of other models in which incremental learning is present . 2.1 PRELIMINARIES . Our simple linear model will be similar to the toy model analyzed by Woodworth et al . ( 2019 ) . Our input space will be X = Rd and the hypothesis space will be linear models with non-negative weights , such that : fσ ( x ) = 〈σ , x〉 σ ∈ Rd≥0 ( 1 ) We will introduce depth into our model , by parameterizing σ using w ∈ Rd≥0 in the following way : ∀i : σi = wNi Where N represents the depth of the model . Since we restrict the model to having non-negative weights , this parameterization doesn ’ t change the expressiveness , but it does radically change it ’ s optimization dynamics . Assuming the data is labeled by some σ∗ ∈ Rd≥0 , we will study the dynamics of this model for general N under a depth-normalized1 squared loss over Gaussian inputs , which will allow us to derive our analytical solution : ` N ( w ) = 1 2N2 Ex [ ( 〈σ∗ , x〉 − 〈wN , x〉 ) 2 ] = 1 2N2 ||wN − σ∗||2 ( 2 ) We will assume that our model is initialized uniformly with a tunable scaling factor , such that : ∀i : wi ( 0 ) = N √ σ0 ( 3 ) 2.2 GRADIENT FLOW ANALYTICAL SOLUTIONS . Analyzing our toy model using gradient flow allows us to obtain an analytical solution for the dynamics of σ ( t ) along with the dynamics of the loss function for a general N . For brevity , the following theorem refers only to N = 1 , 2 and N →∞ , however the solutions for 3 ≤ N < ∞ are similar in structure to N →∞ , but more complicated . We also assume σ∗i > 0 for brevity , however we can derive the solutions for σ∗i = 0 as well . Note that this result is a special case adaptation of the one presented in Saxe et al . ( 2013 ) for deep linear networks : Theorem 1 . Minimizing the toy linear model described in ( 1 ) with gradient flow over the depth normalized squared loss ( 2 ) , with Gaussian inputs and weights initialized as in ( 3 ) and assuming σ∗i > 0 leads to the following analytical solutions for different values of N : N = 1 : σi ( t ) = σ ∗ i + ( σ0 − σ∗i ) e−t N = 2 : σi ( t ) = σ0σ ∗ i e σ∗i t σ0 ( eσ ∗ i t − 1 ) + σ∗i N →∞ : t = 1 ( σ∗i ) 2 log ( σi ( t ) ( σ0 − σ∗i ) σ0 ( σi ( t ) − σ∗i ) ) − 1 σ∗i ( 1 σi ( t ) − 1 σ0 ) Proof . The gradient flow equations for our model are the following : ẇi = −∇wi ` = 1 N wN−1i ( σ ∗ i − wNi ) Given the dynamics of the w parameters , we may use the chain rule to derive the dynamics of the induced model , σ : σ̇i = dσi dwi ẇi = w 2N−2 ( σ∗i − wNi ) = σ 2− 2N i ( σ ∗ i − σi ) ( 4 ) This differential equation is solvable for all N , leading to the solutions in the theorem . Taking N →∞ in ( 4 ) leads to σ̇i = σ2i ( σ∗i − σi ) , which is also solvable . 1This normalization is used for mathematical convenience to have solutions of different depths exhibit similar time scales in their dynamics . Equivalently , we can derive the solutions for the regular square loss and then use different time scalings in the dynamical analysis . Analyzing these solutions , we see how even in such a simple model depth causes different factors of the model to be learned at different rates . Specifically , values corresponding to larger optimal values converge faster , suggesting a form of incremental learning . This is most clear for N = 2 where the solution isn ’ t implicit , but is also the case for N ≥ 3 , as we will see in the next subsection . These dynamics are depicted in figure 2 , where we see the dynamics of the different values of σ ( t ) as learning progresses . When N = 1 , all values are learned at the same rate regardless of the initialization , while the deeper models are clearly biased towards learning the larger singular values first , especially at small initialization scales . Our model has only one optimal solution due to the population loss , but it is clear how this sort of dynamic can induce sparse solutions - if the model is able to fit the data after a small amount of learning phases , then it ’ s obtained result will be sparse . Alternatively , if N = 1 , we know that the dynamics will lead to the minimal ` 2 norm solution which is dense . We explore the sparsity inducing bias of our toy model by comparing it empirically2 to a greedy sparse approximation algorithm in appendix D , and give our theoretical results in the next section . 3 INCREMENTAL LEARNING . Equipped with analytical solutions for the dynamics of our model for every depth , we turn to study how the depth and initialization effect incremental learning . While Gidel et al . ( 2019 ) focuses on incremental learning in depth-2 models at the limit of σ0 → 0 , we will study the phenomenon for a general depth and for σ0 > 0 . First , we will define the notion of incremental learning . Since all values of σ are learned in parallel , we can ’ t expect one value to converge before the other moves at all ( which happens for infinitesimal initialization as shown by Gidel et al . ( 2019 ) ) . We will need a more relaxed definition for incremental learning in finite initialization scales . 2The code for reproducing all of our experiments can be found in https : //github.com/dsgissin/ Incremental-Learning Definition 1 . Given two values σi , σj such that σ∗i > σ∗j > 0 and both are initialized as σi ( 0 ) = σj ( 0 ) = σ0 < σ ∗ j , and given two scalars s ∈ ( 0 , 14 ) and f ∈ ( 3 4 , 1 ) , we call the learning of the values ( s , f ) -incremental if there exists a t for which : σj ( t ) ≤ sσ∗j < fσ∗i ≤ σi ( t ) In words , two values have distinct learning phases if the first almost converges ( f ≈ 1 ) before the second changes by much ( s 1 ) . Note that for any N , σ ( t ) is monotonically increasing and so once σj ( t ) = sσ∗j , it will not decrease to allow further incremental learning . Given this definition of incremental learning , we turn to study the conditions that facilitate incremental learning in our toy model . Our main result is a dynamical depth separation result , showing that incremental learning is dependent on σ ∗ i σ∗j in different ways for different values ofN . The largest difference in dependence happens between N = 2 and N = 3 , where the dependence changes from exponential to polynomial : Theorem 2 . Given two values σi , σj of a toy linear model as in ( 1 ) , where σ∗i σ∗j = r > 1 and the model is initialized as in ( 3 ) , and given two scalars s ∈ ( 0 , 14 ) and f ∈ ( 3 4 , 1 ) , then the largest initialization value for which the learning phases of the values are ( s , f ) -incremental , denoted σth0 , is bounded in the following way : sσ∗j ( s rf ) 1 ( 1−f ) ( r−1 ) ≤σth0 ≤ sσ∗j ( s rf ) 1−s r−1 N = 2 sσ∗j ( ( 1− f ) ( r − 1 ) 1 + ( 1− f ) ( r − 1 ) ) N N−2 ≤σth0 ≤ sσ∗j ( r − 1 r − s ) N N−2 N ≥ 3 Proof sketch ( the full proof is given in appendix A ) . Rewriting the separable differential equation in ( 4 ) to calculate the time until σ ( t ) = ασ∗ , we get the following : tα ( σ ) = ∫ ασ∗ σ0 dσ σ2− 2 N ( σ∗ − σ ) The condition for incremental learning is then the requirement that tf ( σi ) ≤ ts ( σj ) , resulting in : ∫ fσ∗i σ0 dσ σ2− 2 N ( σ∗i − σ ) ≤ ∫ sσ∗j σ0 dσ σ2− 2 N ( σ∗j − σ ) We then relax/restrict the above condition to get a necessary/sufficient condition on σ0 , leading to a lower and upper bound on σth0 . Note that the value determining the condition for incremental learning is σ ∗ i σ∗j - if two values are in the same order of magnitude , then their ratio will be close to 1 and we will need a small initialization to obtain incremental learning . The dependence on the ratio changes with depth , and is exponential for N = 2 . This means that incremental learning , while possible for shallow models , is difficult to see in practice . This result explains why changing the initialization scale in figure 2 changes the dynamics of the N ≥ 3 models , while not changing the dynamics for N = 2 noticeably . The next theorem extends part of our analysis to gradient descent , a more realistic setting than the infinitesimal learning rate of gradient flow : Theorem 3 . Given two values σi , σj of a depth-2 toy linear model as in ( 1 ) , such that σ∗i σ∗j = r > 1 and the model is initialized as in ( 3 ) , and given two scalars s ∈ ( 0 , 14 ) and f ∈ ( 3 4 , 1 ) , and assuming σ∗j ≥ 2σ0 , and assuming we optimize with gradient descent with a learning rate η ≤ cσ∗1 for c < 2 ( √ 2 − 1 ) and σ∗1 the largest value of σ∗ , then the largest initialization value for which the learning phases of the values are ( s , f ) -incremental , denoted σth0 , is lower and upper bounded in the following way : 1 2 s 1− s σ∗j ( 1− f 2rf s 1− s ) 1 A−1 ≤ σth0 ≤ s 1− s σ∗j ( 1− f f s 1− s ) 1 B−1

Summary: This paper deals with the theoretical study of the gradient dynamics in deep neural networks. More precisely, this paper define a notion of incremental learning for a particular learning dynamics and study how the depth of the network influence it. Then, the authors show two cases where it applies: matrix sensing, quadratic neural networks and provide intuitions on how it could also apply to linear convolutional networks.