The Implicit Bias of Depth: How Incremental Learning Drives Generalization

1 INTRODUCTION . Neural networks have led to a breakthrough in modern machine learning , allowing us to efficiently learn highly expressive models that still generalize to unseen data . The theoretical reasons for this success are still unclear , as the generalization capabilities of neural networks defy the classic statistical learning theory bounds . Since these bounds , which depend solely on the capacity of the learned model , are unable to account for the success of neural networks , we must examine additional properties of the learning process . One such property is the optimization algorithm - while neural networks can express a multitude of possible ERM solutions for a given training set , gradient-based methods with the right initialization may be implicitly biased towards certain solutions which generalize . A possible way such an implicit bias may present itself , is if gradient-based methods were to search the hypothesis space for possible solutions of gradually increasing complexity . This would suggest that while the hypothesis space itself is extremely complex , our search strategy favors the simplest solutions and thus generalizes . One of the leading results along these lines has been by Saxe et al . ( 2013 ) , deriving an analytical solution for the gradient flow dynamics of deep linear networks and showing that for such models , the singular values converge at different rates , with larger values converging first . At the limit of infinitesimal initialization of the deep linear network , Gidel et al . ( 2019 ) show these dynamics exhibit a behavior of “ incremental learning ” - the singular values of the model are learned separately , one at a time . Our work generalizes these results to small but finite initialization scales . Incremental learning dynamics have also been explored in gradient descent applied to matrix completion and sensing with a factorized parameterization ( Gunasekar et al . ( 2017 ) , Arora et al . ( 2018 ) , Woodworth et al . ( 2019 ) ) . When initialized with small Gaussian weights and trained with a small learning rate , such a model is able to successfully recover the low-rank matrix which labeled the data , even if the problem is highly over-determined and no additional regularization is applied . In their proof of low-rank recovery for such models , Li et al . ( 2017 ) show that the model remains lowrank throughout the optimization process , leading to the successful generalization . Additionally , Arora et al . ( 2019 ) explore the dynamics of such models , showing the singular values are learned at different rates and that deeper models exhibit stronger incremental learning dynamics . Our work deals with a more simplified setting , allowing us to determine explicitly under which conditions depth leads to this dynamical phenomenon . Finally , the learning dynamics of nonlinear models have been studied as well . Combes et al . ( 2018 ) and Williams et al . ( 2019 ) study the gradient flow dynamics of shallow ReLU networks under restrictive distributional assumptions , Ronen et al . ( 2019 ) show that shallow networks learn functions of gradually increasing frequencies and Nakkiran et al . ( 2019 ) show how deep ReLU networks correlate with linear classifiers in the early stages of training . These findings , along with others , suggest that the generalization ability of deep networks is at least in part due to the incremental learning dynamics of gradient descent . Following this line of work , we begin by explicitly defining the notion of incremental learning for a toy model which exhibits this sort of behavior . Analyzing the dynamics of the model for gradient flow and gradient descent , we characterize the effect of the model ’ s depth and initialization scale on incremental learning , showing how deeper models allow for incremental learning in larger ( realistic ) initialization scales . Specifically , we show that a depth-2 model requires exponentially small initialization for incremental learning to occur , while deeper models only require the initialization to be polynomially small . Once incremental learning has been defined and characterized for the toy model , we generalize our results theoretically and empirically for larger linear and quadratic models . Examples of incremental learning in these models can be seen in figure 1 , which we discuss further in section 4 . 2 DYNAMICAL ANALYSIS OF A TOY MODEL . We begin by analyzing incremental learning for a simple model . This will allow us to gain a clear understanding of the phenomenon and the conditions for it , which we will later be able to apply to a variety of other models in which incremental learning is present . 2.1 PRELIMINARIES . Our simple linear model will be similar to the toy model analyzed by Woodworth et al . ( 2019 ) . Our input space will be X = Rd and the hypothesis space will be linear models with non-negative weights , such that : fσ ( x ) = 〈σ , x〉 σ ∈ Rd≥0 ( 1 ) We will introduce depth into our model , by parameterizing σ using w ∈ Rd≥0 in the following way : ∀i : σi = wNi Where N represents the depth of the model . Since we restrict the model to having non-negative weights , this parameterization doesn ’ t change the expressiveness , but it does radically change it ’ s optimization dynamics . Assuming the data is labeled by some σ∗ ∈ Rd≥0 , we will study the dynamics of this model for general N under a depth-normalized1 squared loss over Gaussian inputs , which will allow us to derive our analytical solution : ` N ( w ) = 1 2N2 Ex [ ( 〈σ∗ , x〉 − 〈wN , x〉 ) 2 ] = 1 2N2 ||wN − σ∗||2 ( 2 ) We will assume that our model is initialized uniformly with a tunable scaling factor , such that : ∀i : wi ( 0 ) = N √ σ0 ( 3 ) 2.2 GRADIENT FLOW ANALYTICAL SOLUTIONS . Analyzing our toy model using gradient flow allows us to obtain an analytical solution for the dynamics of σ ( t ) along with the dynamics of the loss function for a general N . For brevity , the following theorem refers only to N = 1 , 2 and N →∞ , however the solutions for 3 ≤ N < ∞ are similar in structure to N →∞ , but more complicated . We also assume σ∗i > 0 for brevity , however we can derive the solutions for σ∗i = 0 as well . Note that this result is a special case adaptation of the one presented in Saxe et al . ( 2013 ) for deep linear networks : Theorem 1 . Minimizing the toy linear model described in ( 1 ) with gradient flow over the depth normalized squared loss ( 2 ) , with Gaussian inputs and weights initialized as in ( 3 ) and assuming σ∗i > 0 leads to the following analytical solutions for different values of N : N = 1 : σi ( t ) = σ ∗ i + ( σ0 − σ∗i ) e−t N = 2 : σi ( t ) = σ0σ ∗ i e σ∗i t σ0 ( eσ ∗ i t − 1 ) + σ∗i N →∞ : t = 1 ( σ∗i ) 2 log ( σi ( t ) ( σ0 − σ∗i ) σ0 ( σi ( t ) − σ∗i ) ) − 1 σ∗i ( 1 σi ( t ) − 1 σ0 ) Proof . The gradient flow equations for our model are the following : ẇi = −∇wi ` = 1 N wN−1i ( σ ∗ i − wNi ) Given the dynamics of the w parameters , we may use the chain rule to derive the dynamics of the induced model , σ : σ̇i = dσi dwi ẇi = w 2N−2 ( σ∗i − wNi ) = σ 2− 2N i ( σ ∗ i − σi ) ( 4 ) This differential equation is solvable for all N , leading to the solutions in the theorem . Taking N →∞ in ( 4 ) leads to σ̇i = σ2i ( σ∗i − σi ) , which is also solvable . 1This normalization is used for mathematical convenience to have solutions of different depths exhibit similar time scales in their dynamics . Equivalently , we can derive the solutions for the regular square loss and then use different time scalings in the dynamical analysis . Analyzing these solutions , we see how even in such a simple model depth causes different factors of the model to be learned at different rates . Specifically , values corresponding to larger optimal values converge faster , suggesting a form of incremental learning . This is most clear for N = 2 where the solution isn ’ t implicit , but is also the case for N ≥ 3 , as we will see in the next subsection . These dynamics are depicted in figure 2 , where we see the dynamics of the different values of σ ( t ) as learning progresses . When N = 1 , all values are learned at the same rate regardless of the initialization , while the deeper models are clearly biased towards learning the larger singular values first , especially at small initialization scales . Our model has only one optimal solution due to the population loss , but it is clear how this sort of dynamic can induce sparse solutions - if the model is able to fit the data after a small amount of learning phases , then it ’ s obtained result will be sparse . Alternatively , if N = 1 , we know that the dynamics will lead to the minimal ` 2 norm solution which is dense . We explore the sparsity inducing bias of our toy model by comparing it empirically2 to a greedy sparse approximation algorithm in appendix D , and give our theoretical results in the next section . 3 INCREMENTAL LEARNING . Equipped with analytical solutions for the dynamics of our model for every depth , we turn to study how the depth and initialization effect incremental learning . While Gidel et al . ( 2019 ) focuses on incremental learning in depth-2 models at the limit of σ0 → 0 , we will study the phenomenon for a general depth and for σ0 > 0 . First , we will define the notion of incremental learning . Since all values of σ are learned in parallel , we can ’ t expect one value to converge before the other moves at all ( which happens for infinitesimal initialization as shown by Gidel et al . ( 2019 ) ) . We will need a more relaxed definition for incremental learning in finite initialization scales . 2The code for reproducing all of our experiments can be found in https : //github.com/dsgissin/ Incremental-Learning Definition 1 . Given two values σi , σj such that σ∗i > σ∗j > 0 and both are initialized as σi ( 0 ) = σj ( 0 ) = σ0 < σ ∗ j , and given two scalars s ∈ ( 0 , 14 ) and f ∈ ( 3 4 , 1 ) , we call the learning of the values ( s , f ) -incremental if there exists a t for which : σj ( t ) ≤ sσ∗j < fσ∗i ≤ σi ( t ) In words , two values have distinct learning phases if the first almost converges ( f ≈ 1 ) before the second changes by much ( s 1 ) . Note that for any N , σ ( t ) is monotonically increasing and so once σj ( t ) = sσ∗j , it will not decrease to allow further incremental learning . Given this definition of incremental learning , we turn to study the conditions that facilitate incremental learning in our toy model . Our main result is a dynamical depth separation result , showing that incremental learning is dependent on σ ∗ i σ∗j in different ways for different values ofN . The largest difference in dependence happens between N = 2 and N = 3 , where the dependence changes from exponential to polynomial : Theorem 2 . Given two values σi , σj of a toy linear model as in ( 1 ) , where σ∗i σ∗j = r > 1 and the model is initialized as in ( 3 ) , and given two scalars s ∈ ( 0 , 14 ) and f ∈ ( 3 4 , 1 ) , then the largest initialization value for which the learning phases of the values are ( s , f ) -incremental , denoted σth0 , is bounded in the following way : sσ∗j ( s rf ) 1 ( 1−f ) ( r−1 ) ≤σth0 ≤ sσ∗j ( s rf ) 1−s r−1 N = 2 sσ∗j ( ( 1− f ) ( r − 1 ) 1 + ( 1− f ) ( r − 1 ) ) N N−2 ≤σth0 ≤ sσ∗j ( r − 1 r − s ) N N−2 N ≥ 3 Proof sketch ( the full proof is given in appendix A ) . Rewriting the separable differential equation in ( 4 ) to calculate the time until σ ( t ) = ασ∗ , we get the following : tα ( σ ) = ∫ ασ∗ σ0 dσ σ2− 2 N ( σ∗ − σ ) The condition for incremental learning is then the requirement that tf ( σi ) ≤ ts ( σj ) , resulting in : ∫ fσ∗i σ0 dσ σ2− 2 N ( σ∗i − σ ) ≤ ∫ sσ∗j σ0 dσ σ2− 2 N ( σ∗j − σ ) We then relax/restrict the above condition to get a necessary/sufficient condition on σ0 , leading to a lower and upper bound on σth0 . Note that the value determining the condition for incremental learning is σ ∗ i σ∗j - if two values are in the same order of magnitude , then their ratio will be close to 1 and we will need a small initialization to obtain incremental learning . The dependence on the ratio changes with depth , and is exponential for N = 2 . This means that incremental learning , while possible for shallow models , is difficult to see in practice . This result explains why changing the initialization scale in figure 2 changes the dynamics of the N ≥ 3 models , while not changing the dynamics for N = 2 noticeably . The next theorem extends part of our analysis to gradient descent , a more realistic setting than the infinitesimal learning rate of gradient flow : Theorem 3 . Given two values σi , σj of a depth-2 toy linear model as in ( 1 ) , such that σ∗i σ∗j = r > 1 and the model is initialized as in ( 3 ) , and given two scalars s ∈ ( 0 , 14 ) and f ∈ ( 3 4 , 1 ) , and assuming σ∗j ≥ 2σ0 , and assuming we optimize with gradient descent with a learning rate η ≤ cσ∗1 for c < 2 ( √ 2 − 1 ) and σ∗1 the largest value of σ∗ , then the largest initialization value for which the learning phases of the values are ( s , f ) -incremental , denoted σth0 , is lower and upper bounded in the following way : 1 2 s 1− s σ∗j ( 1− f 2rf s 1− s ) 1 A−1 ≤ σth0 ≤ s 1− s σ∗j ( 1− f f s 1− s ) 1 B−1

Summary: This paper studies the phenomenon of incremental learning in several deep models. It starts with analyzing the optimization dynamics of a toy model, and showing that it follows incremental learning, a notion defined clearly in the paper. In particular, it shows that depth affects the strength of incremental learning in the sense that when the depth of the model is increased (especially when going from N=2 to N=3), the maximal initialization value with which incremental learning can occur is increased. In this sense, deeper models experience incremental learning more easily. The paper then moves on to other “deep linear“ models, including matrix sensing, one-hidden-layer quadratic neural networks and diagonal/convolutional linear neural networks, derives ODEs for the evolution of the singular values in the learned models, which is argued to also lead to incremental learning.