Title: Deep linear networks for regression are implicitly regularized towards flat minima
Venue: NeurIPS 2024
Accepted: yes
Keywords: deep learning theory

sharpness

non-convex optimization

implicit regularization

gradient flow
Abstract: The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to understand their optimization dynamics. In this paper, we study the sharpness of deep linear networks for univariate regression. Minimizers can have arbitrarily large sharpness, but not an arbitrarily small one. Indeed, we show a lower bound on the sharpness of minimizers, which grows linearly with depth. We then study the properties of the minimizer found by gradient flow, which is the limit of gradient descent with vanishing learning rate. We show an implicit regularization towards flat minima: the sharpness of the minimizer is no more than a constant times the lower bound. The constant depends on the condition number of the data covariance matrix, but not on width or depth. This result is proven both for a small-scale initialization and a residual initialization. Results of independent interest are shown in both cases. For small-scale initialization, we show that the learned weight matrices are approximately rank-one and that their singular vectors align. For residual initialization, convergence of the gradient flow for a Gaussian initialization of the residual network is proven. Numerical experiments illustrate our results and connect them to gradient descent with non-vanishing learning rate.

Deep linear networks for regression are implicitly regularized towards flat minima

Abstract

The largest eigenvalue of the Hessian, or sharpness, of neural networks is a key quantity to understand their optimization dynamics. In this paper, we study the sharpness of deep linear networks for univariate regression. Minimizers can have arbitrarily large sharpness, but not an arbitrarily small one. Indeed, we show a lower bound on the sharpness of minimizers, which grows linearly with depth. We then study the properties of the minimizer found by gradient flow, which is the limit of gradient descent with vanishing learning rate. We show an implicit regularization towards flat minima : the sharpness of the minimizer is no more than a constant times the lower bound. The constant depends on the condition number of the data covariance matrix, but not on width or depth. This result is proven both for a small-scale initialization and a residual initialization . Results of independent interest are shown in both cases. For small-scale initialization, we show that the learned weight matrices are approximately rank-one and that their singular vectors align. For residual initialization, convergence of the gradient flow for a Gaussian initialization of the residual network is proven. Numerical experiments illustrate our results and connect them to gradient descent with non-vanishing learning rate.

1 Introduction

Neural networks have intricate optimization dynamics due to the non-convexity of their objective. A key quantity to understand these dynamics is the largest eigenvalue of the Hessian or sharpness S ( W ) (see Section 3 for a formal definition), in particular because of its connection with the choice of learning rate η . Classical theory from convex optimization indicates that the sharpness should remain lower than 2 /η to avoid divergence (Nesterov, 2018). The relevance of this point of view for deep learning has recently been questioned since neural networks have been shown to often operate at the edge of stability (Cohen et al., 2021), where the sharpness oscillates around 2 /η , while the loss still steadily decreases, albeit non-monotonically. Damian et al. (2023) explained the stability of gradient descent slightly above the 2 /η threshold to be a general phenomenon for non-quadratic objectives, where the third-order derivatives of the loss induce a self-stabilization effect. They also show that, under appropriate assumptions, gradient descent on a risk R L implicitly solves the constrained minimization problem

Trainability of neural networks initialized with a sharpness larger than 2 /η is also studied by Lewkowycz et al. (2020), which describes a transient catapult regime, which lasts until the sharpness goes below 2 /η . These results beg the question of quantifying the largest learning rate that enables successful training of neural networks. For classification with linearly separable data and logistic loss,

Wu et al. (2024) show that gradient descent converges for any learning rate. In this work, we address the case of deep linear networks for regression. As illustrated by Figure 1a, the picture then differs from the classification case: when the learning rate exceeds some critical value, the network fails to learn . We further remark that this critical value does not seem to be related to the initial scale: when the learning rate is under the critical value, learning is successful for a wide range of initial scales. In Figure 1b, we see that the large initialization scales correspond to initial sharpnesses well over the 2 /η threshold, confirming that training is possible while initializing beyond the 2 /η threshold.

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Scale of initialization

10

13

10

11

10

9

10

7

10

5

10

3

10

1

10

1

Squared distance to optimal regressor

Initialization

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

l. rate: 0.1

l. rate: 0.2

(a) Squared distance of the trained network to the empirical risk minimizer, for various learning rates and initialization scales. Training succeeds when the learning rate is lower than a critical value independent of the initialization scale.

0

100

200

300

400

Training steps

10

12

10

10

10

8

10

6

10

4

10

2

Squared distance to optimal regressor

0

100

200

300

400

Training steps

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

sharpness

2/lr

(c) Evolution during training of the squared distance to the empirical risk minimizer and of sharpness, for η = 0 . 02 and an initialization scale of 0 . 35 . The network does not enter edge of stability.

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Scale of initialization

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

Initialization

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

(b) Sharpness at initialization and after training, for various learning rates and initialization scales. For a given learning rate η , the dashed lines represent the 2 /η threshold. The dotted black lines represent the lower and upper bounds given in Theorem 1 and Corollary 2.

0

100

200

300

400

Training steps

10

3

10

2

10

1

Squared distance to optimal regressor

0

100

200

300

400

Training steps

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

sharpness

2/lr

Figure 1: Training a deep linear network on a univariate regression task with quadratic loss. The weight matrices are initialized as Gaussian random variables, whose standard deviation is the x-axis of plots 1a and 1b. Experimental details are given in Appendix C.

To understand where the critical value for the learning rate comes from, we characterize the sharpness of minimizers of the empirical risk. We show that there exist minimizers with arbitrarily large sharpness, but not with arbitrarily small sharpness. Indeed, the sharpness of any minimizer grows linearly with depth, as made precise next.

Theorem 1. Let X ∈ R n × d be a design matrix and y ∈ R n a target. Then the minimal sharpness S min of any linear network x ↦→ W L . . . W 1 x of depth L that implements the optimal linear regressor w ⋆ ∈ R d satisfies

where Λ is the largest eigenvalue of the empirical covariance matrix ˆ Σ := 1 n X ⊤ X , and a := ( w ⋆ / ∥ w ⋆ ∥ ) ⊤ ˆ Σ( w ⋆ / ∥ w ⋆ ∥ ) .

We note that this bound is similar to a result of Mulayoff and Michaeli (2020), although we alleviate their assumption of data whiteness ( ˆ Σ = I ). In particular, we do not assume that data covariance

matrix is full rank. Furthermore, our proof technique differs, since we do not require tensor algebra, and instead exhibit a direction in which the second derivative of the loss is large.

This result shows that it is not possible to find a minimizer of the empirical risk in regions of low sharpness. Combined with (1), this suggests an interpretation of the critical value for the learning rate: gradient descent should not be able to converge to a global minimizer as soon as

This is confirmed experimentally by Figure 2, which shows that the critical value of the learning rate matches our theoretical prediction. We note that this gives a quantitative answer to the observations of Lewkowycz et al. (2020), which shows the existence of a maximal architecture-dependent learning rate beyond which training fails. The dependence of learning rate on depth (namely, constant over depth) also matches other papers that study scaling of neural networks (Chizat and Netrapalli, 2024; Yang et al., 2024).

10

2

10

1

Learning rate

0.000

0.025

0.050

0.075

0.100

0.125

0.150

0.175

0.200

Squared distance to optimal regressor

depth: 5

depth: 10

depth: 20

depth: 40

To deepen our understanding of the training dynamics, we aim at quantifying the sharpness of the minimizer found by gradient descent (when it succeeds): we know that it has to be larger than S min , but is it close to S min or is it much larger? Inspecting Figure 1b, we see that the answer empirically depends on the interplay between initialization scale and learning rate. For small initialization scale, the sharpness after training is equal to a value relatively close to S min and independent of the learning rate. As the initialization scale increases, the sharpness of the trained network also increases, and plateaus at the value 2 /η . The plateauing for large initialization scales can be explained by the edge of stability analysis (1), which upper bounds the sharpness of the minimizer by 2 /η (see Figure 1d). However, this gives no insight on the value of the sharpness when the learning rate is sufficiently small so that the network does not enter the edge of stability regime (see Figure 1c).

In this paper, we study the limiting case for vanishing η , i.e., training with gradient flow. As our main finding, we bound the sharpness of the minimizer found by gradient flow in the case of overdetermined regression, meaning that the sample size n is larger that the data dimension d and the data empirical covariance matrix is nonsingular. In this case, we prove that the ratio between the sharpness after training and S min is less than a constant depending mainly on the condition number of the empirical covariance matrix ˆ Σ . In particular, the ratio does not depend on the width or depth of the network. This shows an implicit regularization towards flat minima . Note that the phenomenon we exhibit is different from the well-studied implicit regularization towards flat minima caused by stochasticity in SGD (Keskar et al., 2017; Smith and Le, 2018; Blanc et al., 2020; Damian et al., 2021; Li et al., 2022; Liu et al., 2023). In the present study, the dynamics are purely deterministic, and the low sharpness is due to the fact that the weight matrices found by gradient flow have (approximately) the same norm across layers, and that this norm is (approximately) the smallest possible one so that the network can minimize the risk.

Link with generalization. Flatter minima have been found to generalize better (Hochreiter and Schmidhuber, 1997; Jastrz˛ ebski et al., 2017; Keskar et al., 2017; Jiang et al., 2020), although the picture is subtle (Dinh et al., 2017; Neyshabur et al., 2017; Andriushchenko et al., 2023). However, in this paper, we focus on the link of sharpness with (non-convex) optimization dynamics, rather than with generalization abilities. Indeed, our implicit regularization result holds in the overdetermined setting, where all minimizers of the empirical risk implement the same function thus have the same generalization error, although they differ in parameter space and in particular have different sharpnesses. We leave to future work extensions to more complex settings where our approach may link sharpness and generalization, beginning with deep linear networks for underdetermined regression. We refer to Appendix C for more comments on the link with generalization.

We investigate two initialization schemes, quite different in nature: small-scale initialization and residual initialization. Let us explain both settings, by presenting our approach, contributions of independent interest, and related works.

Small-scale initialization. In this setting, we consider an initialization of the weight matrices W k with i.i.d. Gaussian entries of small variance. Initialization scale is known to play a key role in training of neural networks: small-scale initialization corresponds to the 'feature learning' regime where the weights change significantly during training, by opposition to the 'lazy' regime (see, e.g., Chizat et al., 2019). We show convergence of the empirical risk to zero, then characterize the structure of the minimizer found by gradient flow , a novel result of interest independently of its connection with sharpness. At convergence, the weight matrices are close to being rank-one , in the sense that all their singular values but the largest one are small. Furthermore, the first left singular vector of any weight matrix aligns with the first right singular vector of the next weight matrix. From this specific structure, we deduce our bound on the sharpness of the trained network. The bound is illustrated on Figure 1b, where our lower and upper theoretical bounds on the sharpness are plotted as dotted black lines. We observe that the sharpness after training, when starting from a small-scale initialization, is indeed situated between the black lines.

The result and proof extend the study by Ji and Telgarsky (2020) for classification, although the parameters do not diverge to infinity contrarily to the classification case, thus requiring a finer control of the distance to the rank-one aligned solution. In regression, implicit regularization towards lowrank structure in parameter space was also studied by Saxe et al. (2014); Lampinen and Ganguli (2019); Gidel et al. (2019); Saxe et al. (2019); Varre et al. (2023) for two-layer neural networks and in Timor et al. (2023) for deep ReLU networks. This latter paper assumes convergence of the optimization algorithm and show that a solution with minimal ℓ 2 -norm has to be low-rank. In our linear setting, we instead show convergence. As detailed below, we impose mild requirements on the structure on the initialization beyond its scale; they are satisfied for instance by initializing one weight matrix to zero and the others with i.i.d. Gaussian entries. In particular, we do not require the so-called 'zero-balanced initialization' as is common in the literature on deep linear networks (see, e.g., Arora et al., 2018; Advani et al., 2020; Li et al., 2021) or a deficient-margin initialization as in Arora et al. (2019a). Finally, the limit when initialization scale tends to zero has been described for deep linear networks in Jacot et al. (2021) for multivariate regression. It consists in a saddle-to-saddle dynamics, where the rank of the weight matrices increases after each saddle. The present study considers instead a non-asymptotic setting where the initialization scale is small but nonzero, and shows convergence to a rank-one limit because univariate and not multivariate regression is considered.

We note that sharpness at initialization can be made arbitrarily small, since it is controlled by the initialization scale, while sharpness after training scales as Θ( L ) . This therefore showcases an example of sharpening during training (although we make no statement on monotonicity).

Residual initialization. Architectures of deep neural networks used in practice often present residual connections, which stabilize training (He et al., 2015). A simple non-linear residual architecture writes h k +1 = h k + σ ( N k h k ) . Removing the non-linearity, we get h k +1 = ( I + N k ) h k , which prompts us to consider deep linear networks with square weight matrices W k ∈ R d × d that are initialized as

where the N k ∈ R d × d are filled with i.i.d. standard Gaussian entries and s ⩾ 0 is a hyperparameter tuning the initialization scale. The scaling of the residual branch in 1 / √ d is common and corresponds

for instance to the so-called Glorot and He initializations respectively from Glorot and Bengio (2010) and He et al. (2015). It ensures that the variance of the residual branch is independent of the width d . Similarly, as studied in Arpit et al. (2019); Marion et al. (2022), the scaling in 1 / √ L is the right one so that the initialization noise neither blows up nor decays when L →∞ . Note that, in practice, this scaling factor is often replaced by batch normalization (Ioffe and Szegedy, 2015), which has been shown empirically to have a similar effect to 1 / √ L scaling (De and Smith, 2020).

In this setting which we refer to as residual initialization , we show global convergence of the empirical risk. To our knowledge, it is the first time that convergence is proven for a standard Gaussian initialization of the residual network outside the large width d ⩾ n regime . Previous works considered either an identity initialization (Bartlett et al., 2018; Arora et al., 2019a; Zou et al., 2020) or a smooth initialization such that ∥ W k +1 (0) -W k (0) ∥ F = O (1 /L ) (Sander et al., 2022; Marion et al., 2024). The extension to standard Gaussian initialization leverages sharp bounds for the singular values of the product of W k (0) . Our main assumption, in alignment with the literature, is that the risk at initialization should not be larger than a constant (depending on ˆ Σ and s ).

We then show that the weights after training can be written

where the Frobenius norm of the θ k is bounded by a constant (depending only on s ). This structure finally enables us to bound the sharpness of the trained network. Remark that, to connect this analysis with our discussion of sharpness in univariate regression, we add to the residual network a final fixed projection vector p ∈ R d , so that our neural network writes x ↦→ p ⊤ W L . . . W 1 x , but the proof of convergence also holds without this projection.

Experimentally, we give in Appendix C plots in the residual case that are qualitatively similar to Figure 1. The main difference is that the initial sharpness is less sensitive to the initialization scale s .

Organization of the paper. Section 2 details our setting and notations. Section 3 studies the sharpness of minimizers of the empirical risk and proves Theorem 1. Dynamics of gradient flow starting from small-scale initialization and residual initialization are respectively presented in Sections 4 and 5. The Appendix contains proofs, additional plots, experimental details, and related works.

2 Setting

Model. We consider linear networks of depth L from R d to R , which are linear maps

parameterized by weight matrices W 1 , . . . , W L , where W k ∈ R d k × d k -1 , d 0 = d and p ∈ R d L is a fixed vector. This definition includes both fully-connected networks by setting d L = 1 and p = 1 , and residual networks by setting d 1 = · · · = d L = d , the W k close to the identity, and p to some fixed (potentially random) vector in R d . We let W = ( W 1 , . . . , W L ) and w prod = W ⊤ 1 . . . W ⊤ L p . Given X ∈ R n × d a design matrix and y ∈ R n a target, we consider the empirical risk for regression

̸

The notations R L ( W ) and R 1 ( w prod ) may seem redundant, but are actually practical to define gradients of the risk both with respect to a single matrix W k and to the product w prod . Let ˆ Σ := 1 n X ⊤ X the empirical covariance matrix, and λ and Λ respectively its smallest nonzero and largest eigenvalue. For now, we do not assume that ˆ Σ is full rank, so there is more than one solution to the regression problem min w prod ∈ R d R 1 ( w prod ) . We denote by w ⋆ ∈ R d the smallest norm solution, and we let R min = R 1 ( w ⋆ ) be the minimum of R 1 (and R L ). In all the following, we assume that w ⋆ = 0 . Note that, due to the overparameterization induced by the neural network, there exists an infinity of parameterizations of the mapping x ↦→ w ⋆ ⊤ x .

Gradient flow. We consider that the neural network is trained by gradient flow on the empirical risk R L , that is, the parameters evolve according to the ordinary differential equation

An application of the chain rule gives

with

where the second equality is a consequence of ∇ R 1 ( w ⋆ ) = 0 .

Notations. For k ∈ { 1 , . . . L } , we denote respectively by σ k , u k , and v k the first singular value (which equals the ℓ 2 operator norm), the first left singular vector and the first right singular vector of W k . The ℓ 2 operator norm of a matrix M is denoted by ∥ M ∥ 2 and its Frobenius norm by ∥ M ∥ F . Its smallest singular value is denoted by σ min ( M ) . For a vector v , we let ∥ v ∥ 2 its Euclidean norm. Finally, for quantities that depend on the gradient flow time t , we omit for concision their explicit dependence on t when it is dispensable.

3 Estimates of the minimal sharpness of minimizers

To define the sharpness of the model, we let D = ∑ L k =1 d k d k -1 and identify the space of parameters with R D , which amounts to stacking all the entries of the weight matrices in a large D -dimensional vector. Then the norm of the parameters seen as a D -dimensional vector can be related to the Frobenius norm of the matrices by

This allows us to define the Hessian of the risk H : R D → R D × D , and we denote by S ( W ) its largest eigenvalue for some parameters W , or sharpness. We note that there exists alternative definitions of the sharpness, but this one is the most relevant to study optimization dynamics. Our results are specific to this definition. The following result gives estimates on the minimal sharpness of minimizers of the empirical risk (and is a strictly stronger statement than Theorem 1).

Theorem 2. Let S min = inf W∈ arg min R L ( W ) S ( W ) and a := ( w ⋆ / ∥ w ⋆ ∥ ) ⊤ ˆ Σ( w ⋆ / ∥ w ⋆ ∥ ) . We have

and

The proof of the result relies on the following variational characterization of the sharpness as the direction of the highest change of the gradient

Lower bounds are proven by considering well-chosen directions ˜ W k , for instance ˜ W k = (1+ ξβ k ) W k for the first lower bound. The upper bound is proven by constructing a specific minimizer and bounding its sharpness. The first lower bound shows that the sharpness of minimizers can be arbitrarily high if one of the matrices has a low-enough norm. More precisely, take any minimizer W = ( W 1 , . . . W L ) and consider W C = ( CW 1 , W 2 /C, W 3 , . . . , W L ) , for some C > 0 . Then W C is still a minimizer, and

The fact that a reparameterization of the network can lead to arbitrarily high sharpness is consistent with a similar result by Dinh et al. (2017) for two-layer ReLU networks.

Note that the first lower bound is arbitrarily small for minimizers such that the norms ∥ W k ∥ F are large. On the contrary, the second lower bound is uniform and asymptotically matches the upper bound when L →∞ : we have S min ∼ 2 ∥ w ⋆ ∥ 2 2 La .

As already noted by Mulayoff and Michaeli (2020), the intuition behind the linear scaling of the bound with depth can be seen from a one-dimensional example: take f ( x 1 , . . . , x L ) = ∏ L k =1 x i . Then an easy computation shows that the sharpness of f at (1 , . . . , 1) is equal to L -1 . This showcases a simple example where the output of f is constant with L while its sharpness grows linearly with L .

4 Analysis of gradient flow from small initialization

In this section, we characterize the structure of the minimizer found by gradient flow starting from a small-scale initialization. The proof is inspired by the one of Ji and Telgarsky (2020) for linearlyseparable classification, with a finer analysis due to the finiteness of minimizers in our setting.

We consider the model (3) with d L = 1 and p = 1 . Denoting by R 0 the empirical risk when the weight matrices are equal to zero, we can state our assumption on the initialization.

̸

̸

It is satisfied for instance if one of the weight matrices W k is equal to zero at initialization while the others have i.i.d. Gaussian entries, so that R L ( W (0)) = R 0 and ∇ k R L ( W (0)) = 0 (almost surely).

Linear networks trained by gradient flow possess the following remarkable property (Arora et al., 2018) that shall be useful in the remainder.

Lemma 1. For any time t ⩾ 0 and any k ∈ { 1 , . . . , L -1 } ,

Define now

Note that ε only depends on the initialization, and can be made arbitrarily small by scaling down the initialization. The following key lemma connects throughout training three key quantities to ε .

Lemma 2. The parameters following gradient flow satisfy for any t ⩾ 0 that

The first identity of the Lemma bounds the sum of the squared singular values of W k ( t ) , except the largest one. In other words, it quantifies how close W k ( t ) is to the rank-one approximation given by the first term in its singular value decomposition. The second statement bounds the distance between the spectral norms of any two weight matrices. The last bound quantifies the alignment between the first left singular vector of W k ( t ) and the first right singular vector of W k +1 ( t ) . In particular, if ε is small and σ 2 k +1 ( t ) is of order 1 , then v k +1 ( t ) and u k ( t ) are nearly aligned.

Wenext use this Lemma to show that the neural network satisfies a Polyak-Łojasiewicz (PL) condition, which is one of the main tools to study non-convex optimization dynamics (Rebjock and Boumal, 2023). A well-known result, recalled in Appendix A for completeness, shows that this implies exponential convergence of the gradient flow to a minimizer W SI of the empirical risk.

Theorem 3. Under Assumption ( A 1 ) , the network satisfies the PL condition for t ⩾ 1 , in the sense that there exists some µ > 0 such that, for t ⩾ 1 ,

The proof leverages the structure of the gradient of the risk with respect to the first weight matrix, which relies on the linearity of the neural network and the fact that we consider a univariate output. More precisely, recall that, by (5),

Therefore the Frobenius norm of the gradient decomposes as the product of two vector norms

where the lower bound unfolds from a straightforward computation. The delicate step is to lower bound ∥ W L ( t ) . . . W 2 ( t ) ∥ 2 , which we approach by distinguishing depending on the magnitude of σ 1 ( t ) = ∥ W 1 ( t ) ∥ 2 . If σ 1 ( t ) is large, we use Lemma 2 to deduce both that all σ k ( t ) are large and then that the first singular vectors of successive weight matrices are aligned. This implies that the product of weight matrices has a large norm. To analyze the case where σ 1 ( t ) is small, we use Assumption ( A 1 ) to bound away R L ( W ( t )) from R 0 for t ⩾ 1 , and therefore w prod ( t ) from 0 . The fact that w prod ( t ) cannot be too close to 0 while σ 1 ( t ) is small implies that ∥ W L ( t ) . . . W 2 ( t ) ∥ 2 is large. All in all, this allows us to lower bound ∥ W L ( t ) . . . W 2 ( t ) ∥ 2 , and the PL condition follows.

To characterize the weights at the end of the training, we make the following assumption.

( A 2 ) The data covariance matrix ˆ Σ is full rank, and we have ε ⩽ 1 , ∥ w ⋆ ∥ 2 ⩾ 1 , and 32 L √ ε ⩽ 1 .

The first statement ensures unicity of the minimizer w ⋆ of R 1 , and thus, given that the risk goes to 0 , we have w prod → w ⋆ . The last condition means that the initialization has to be scaled down as the depth increases, so that ε = O (1 /L 2 ) . Intermediates conditions are technical. We can then show the following corollary.

Corollary 1. Under Assumptions ( A 1 ) -( A 2 ) , there exists T ⩾ 1 , such that, for all t ⩾ T and k ∈ { 1 , . . . , L } ,

Together with Lemma 2, this result gives a precise description of the structure of the weights at the end of the gradient flow trajectory. Up to the small factor ε , the weights are rank-one matrices, with equal norms and aligned singular vectors. Since the product of weights aligns with w ⋆ , this means that the first right singular vector of W 1 has to align with w ⋆ , and then the weight matrices align with their neighbors in order to propagate the signal in the network.

Combining this specific structure of the weights with the variational characterization (7) of the sharpness and the explicit formulas (5)-(6) for the gradients, we derive the following upper bound on the sharpness of the found minimizer.

Corollary 2. Under Assumptions ( A 1 ) -( A 2 ) , the following bounds on the sharpness of the minimizer W SI hold:

This result shows that the sharpness of the minimizer is close to S min in the sense that their ratio is bounded by a constant times the condition number of ˆ Σ . For example, in the case of white data ( ˆ Σ = I ), we obtain that S ( W SI ) ⩽ 4 S min .

5 Analysis of gradient flow from residual initialization

We now study the case of residual initialization. We consider a linear network of the form (3) with

where each matrix is a d × d matrix and the N k are filled with i.i.d. Gaussian entries N (0 , 1) . We refer to Section 1 for a discussion of the scaling factor in front of N k . Following the standard initialization of residual networks, we assume that the θ k are initialized to zero. Note that the scaling factor 1 /L in front of the θ k has no impact on the dynamics; it is convenient for exposition and computations, since we show that, with this scaling factor, the θ k are of order O (1) after training.

Before stating the main result of this section on the convergence of the gradient flow, recall that p ∈ R d is a fixed vector appended at the end of the residual network to project its output back in R .

Theorem 4. There exist C 1 , . . . , C 5 > 0 depending only on s such that, if L ⩾ C 1 and d ⩾ C 2 , then, with probability at least

the gradient flow dynamics (4) converge to a global minimizer W RI of the risk. Furthermore, the minimizer W RI satisfies

To our knowledge, Theorem 4 is the first result showing convergence of gradient flow for standard Gaussian initialization of residual networks without assuming overparameterization. The main requirement is that the loss at initialization be not too large, as is standard in the literature analyzing gradient flow for deep linear residual networks (Bartlett et al., 2018; Arora et al., 2019a; Zou et al., 2020; Sander et al., 2022). Note that our bound on the loss at initialization does not depend on the width d , depth L , or sample size n . We emphasize that the same proof holds for multivariate regression, in the absence of the projection vector p . We focus here on univariate regression to connect the result with the analysis of sharpness for univariate regression in Section 3. Details on adaptation to multivariate regression are given in Appendix B.7. Finally, the precise dependence of C 1 to C 5 on s can be found in the proof.

The proof is a refinement of the analysis for identity initialization of residual networks (Zou et al., 2020; Sander et al., 2022). From the expression of the gradients (5), we can show that

with Π L : k := W L . . . W k and Π k :1 := W k . . . W 1 . Letting

the crucial step is to lower bound σ 2 min (Π L : k +1 ) and σ 2 min (Π k -1:1 ) uniformly for t ∈ [0 , t ⋆ ] , in order to get a PL condition valid for t ∈ [0 , t ⋆ ] . Then, the condition on the loss at initialization is used to prove that t ⋆ = ∞ , thereby the PL condition holds for all t ⩾ 0 . We deduce both convergence and the bound on the norm of θ RI k . The lower bound on σ 2 min (Π L : k +1 ) and σ 2 min (Π k -1:1 ) is straightforward in the case of an identity initialization. In the case of Gaussian initialization, the proof is more intricate, and leverages the following high probability bounds on the singular values of residual networks.

Lemma 3. There exist C 1 , . . . , C 4 > 0 depending only on s such that, if

then, with probability at least if

it holds for all θ such that max 1 ⩽ k ⩽ L ∥ θ k ∥ 2 ⩽ 1 64 exp( -2 s 2 -4 u ) and all k ∈ { 1 , . . . , L } that

and

The proof of this result goes in three steps. We first study the evolution of the norm of the activations across the layers of the residual network when θ = 0 , and prove a high-probability bound by leveraging concentration inequalities for Gaussian and χ 2 distributions. Then an ε -net argument allows to bound the singular values. Finally, the extension to θ in a ball around 0 is done via a perturbation analysis. The proof technique is related to previous works (Marion et al., 2022; Zhang et al., 2022), but provides a crisper and sounder bound. More precisely, Marion et al. (2022) show a bound on the norm of the activations with a probability of failure that decays polynomially with the width d , which is not sufficient to apply the ε -net argument that requires an exponentially decreasing probability of failure. As for Zhang et al. (2022), they provide a similar bound with the purpose of showing convergence of wide residual networks, however with a less sharp probability of failure that increases polynomially with depth. 1 Finally, as previously, the dependence of C 1 to C 4 on s can be found in the proof.

The characterization (8) of the minimizer in Theorem 4 allows to bound its sharpness, as made precise by the following corollary. It holds under the same assumptions and high-probability bound as the conclusion of Theorem 4.

Corollary 3. Under the assumptions of Theorem 4, and if the data covariance matrix ˆ Σ is full rank, there exists C > 0 depending only on s such that the following bounds on the sharpness of the minimizer W RI hold:

As for Corollary 2, the proof relies on the fact that the norms of the weight matrices are close to each other and to the smallest possible norm to minimize the risk. This result shows again an implicit regularization towards a low-sharpness minimizer. Experimental illustration connecting the result with gradient descent with non-vanishing learning rate is provided in Appendix C.

6 Conclusion

This paper studies dynamics of gradient flow for deep linear networks on a regression task. For small-scale initialization, we prove that the learned weight matrices are approximately rank-one and that their singular vectors align. For residual initialization, convergence of the gradient flow for a Gaussian initialization is proven. In both cases, we obtain that the sharpness of the solution found by gradient flow is close to the smallest sharpness among all minimizers. Interesting next steps include studying the dynamics at any initialization scale, for non-vanishing learning rates, as well as extension to non-linear networks. We refer to Appendix C for additional comments and preliminary experimental results regarding possible extensions.

Acknowledgments

P.M. acknowledges support of Google through a Google PhD Fellowship. Authors thank Matus Telgarsky for insightful discussions, and Eloïse Berthier, Guillaume Dalle, Benjamin Dupuis, as well as anonymous NeurIPS reviewers, for thoughtful proofreading and remarks. An improvement to a proof was also made possible by a tweet of Gabriel Peyré.

References

S. Advani, A. M. Saxe, and H. Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks , 132:428-446, 2020.

Agarwala, F. Pedregosa, and J. Pennington. Second-order regression models exhibit progressive sharpening to the edge of stability. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 169-195. PMLR, 23-29 Jul 2023.

Andriushchenko, F. Croce, M. Müller, M. Hein, and N. Flammarion. A modern look at the relationship between sharpness and generalization. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 840-902. PMLR, 23-29 Jul 2023.

Arora, N. Cohen, and E. Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 244-253. PMLR, 10-15 Jul 2018.

Arora, N. Cohen, N. Golowich, and W. Hu. A convergence analysis of gradient descent for deep linear neural networks. In International Conference on Learning Representations , 2019a.

Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019b.

Arpit, V. Campos, and Y. Bengio. How to initialize your network? Robust initialization for WeightNorm & ResNets. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32, pages 10902-10911. Curran Associates, Inc., 2019.

Bartlett, D. Helmbold, and P. Long. Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 521-530. PMLR, 10-15 Jul 2018.

Blanc, N. Gupta, G. Valiant, and P. Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. In J. Abernethy and S. Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 483-513. PMLR, 09-12 Jul 2020.

Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

Chizat and P. Netrapalli. The feature speed formula: a flexible approach to scale hyper-parameters of deep neural networks. In A. Globerson, L. Mackey, A. Fan, C. Zhang, D. Belgrave, J. Tomczak, and U. Paquet, editors, Advances in Neural Information Processing Systems , volume 37. Curran Associates, Inc., 2024.

Chizat, E. Oyallon, and F. Bach. On lazy training in differentiable programming. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Cohen, S. Kaur, Y. Li, J. Z. Kolter, and A. Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2021.

Damian, T. Ma, and J. D. Lee. Label noise SGD provably prefers flat global minimizers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 27449-27461. Curran Associates, Inc., 2021.

Damian, E. Nichani, and J. D. Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In The Eleventh International Conference on Learning Representations , 2023.

De and S. Smith. Batch normalization biases residual blocks towards the identity function in deep networks. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 19964-19975. Curran Associates, Inc., 2020.

Dinh, R. Pascanu, S. Bengio, and Y. Bengio. Sharp minima can generalize for deep nets. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning , volume 70 of Proceedings of Machine Learning Research , pages 1019-1028. PMLR, 06-11 Aug 2017.

Ghosh. Exponential tail bounds for chisquared random variables. Journal of Statistical Theory and Practice , 15(2):35, 2021.

Gidel, F. Bach, and S. Lacoste-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019.

Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Y. Teh and M. Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics , volume 9 of Proceedings of Machine Learning Research , pages 249-256. PMLR, 2010.

Gunasekar, B. E. Woodworth, S. Bhojanapalli, B. Neyshabur, and N. Srebro. Implicit regularization in matrix factorization. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification. In 2015 IEEE International Conference on Computer Vision (ICCV) , pages 1026-1034. IEEE Computer Society, 2015.

Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

Hochreiter and J. Schmidhuber. Flat minima. Neural Computation , 9(1):1-42, 1997.

Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In F. Bach and D. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 448-456. PMLR, 2015.

Jacot, F. Ged, B. ¸ Sim¸ sek, C. Hongler, and F. Gabriel. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity. arXiv preprint arXiv:2106.15933 , 2021.

Jastrz˛ ebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey. Three factors influencing minima in SGD. arXiv preprint arXiv:1711.04623 , 2017.

Ji and M. Telgarsky. Directional convergence and alignment in deep learning. In Advances in Neural Information Processing Systems , volume 33, pages 17176-17186. Curran Associates, Inc., 2020.

Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations , 2020.

S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, and P. T. P. Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations , 2017.

K. Lampinen and S. Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. In International Conference on Learning Representations , 2019.

Lewkowycz, Y. Bahri, E. Dyer, J. Sohl-Dickstein, and G. Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218 , 2020.

Li, Y. Luo, and K. Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations , 2021.

Li, T. Wang, and S. Arora. What happens after SGD reaches zero loss? -a mathematical framework. In International Conference on Learning Representations , 2022.

Liu, S. M. Xie, Z. Li, and T. Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 22188-22214. PMLR, 23-29 Jul 2023.

E. MacDonald, J. Valmadre, and S. Lucey. On progressive sharpening, flat minima and generalisation. arXiv preprint arXiv:2305.14683 , 2023.

Marion, A. Fermanian, G. Biau, and J.-P. Vert. Scaling ResNets in the large-depth regime. arXiv preprint arXiv:2206.06929 , 2022.

Marion, Y.-H. Wu, M. E. Sander, and G. Biau. Implicit regularization of deep residual networks towards neural ODEs. In The Twelfth International Conference on Learning Representations , 2024.

Michel Petrovitch. Sur une manière d'étendre le théorème de la moyenne aux équations différentielles du premier ordre. Mathematische Annalen , 54(3):417-436, 1901.

Mulayoff and T. Michaeli. Unique properties of flat minima in deep networks. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 7108-7118. PMLR, 13-18 Jul 2020.

Nesterov. Lectures on Convex Optimization . Springer Optimization and Its Applications. Springer Cham, 2nd edition, 2018.

Neyshabur, S. Bhojanapalli, D. Mcallester, and N. Srebro. Exploring generalization in deep learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Rebjock and N. Boumal. Fast convergence to non-isolated minima: four equivalent conditions for C 2 functions. arXiv preprint arXiv:2303.00096 , 2023.

E. Sander, P. Ablin, and G. Peyré. Do residual neural networks discretize neural ordinary differential equations? In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 35, pages 36520-36532. Curran Associates, Inc., 2022.

Saxe, J. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations , 2014.

M. Saxe, J. L. McClelland, and S. Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences , 116(23):11537-11546, 2019.

L. Smith and Q. V. Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations , 2018.

Timor, G. Vardi, and O. Shamir. Implicit regularization towards rank minimization in ReLU networks. In S. Agrawal and F. Orabona, editors, Proceedings of The 34th International Conference on Algorithmic Learning Theory , volume 201 of Proceedings of Machine Learning Research , pages 1429-1459. PMLR, 20 Feb-23 Feb 2023.

V. Varre, M.-L. Vladarean, L. Pillaud-Vivien, and N. Flammarion. On the spectral bias of two-layer linear networks. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems , volume 36, pages 64380-64414. Curran Associates, Inc., 2023.

Vershynin. High-Dimensional Probability: An Introduction with Applications in Data Science . Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018.

Wang, Z. Li, and J. Li. Analyzing sharpness along GD trajectory: Progressive sharpening and edge of stability. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 9983-9994. Curran Associates, Inc., 2022.

Wu, P. L. Bartlett, M. Telgarsky, and B. Yu. Large stepsize gradient descent for logistic loss: Non-monotonicity of the loss improves optimization efficiency. arXiv preprint arXiv:2402.15926 , 2024.

Yang, D. Yu, C. Zhu, and S. Hayou. Tensor programs VI: Feature learning in infinite depth neural networks. In The Twelfth International Conference on Learning Representations , 2024.

Yun, S. Krishnan, and H. Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations , 2021.

Zhang, D. Yu, M. Yi, W. Chen, and T.-Y. Liu. Stabilize deep ResNet with a sharp scaling factor τ . Machine Learning , 111(9):3359-3392, 2022.

Zou, P. M. Long, and Q. Gu. On the global convergence of training deep linear ResNets. In International Conference on Learning Representations , 2020.

Proof. For any x ∈ R ,

Then, for L ⩾ 1 and x ⩾ 1 , we have

For L large enough, L 2 +1 -2 √ L ⩾ 0 . Thus, since x ⩾ 1 ,

where the last inequality holds for L large enough. This concludes the proof.

Lemma 6. Let h ∈ R d , N ∈ R d × d with i.i.d. standard Gaussian entries, and

Then

Appendix

Organization of the Appendix. Appendix A presents some useful preliminary lemmas. The proofs of the results of the main paper are presented in Appendix B, while additional plots, discussion, and experimental details are given in Appendix C. Finally, Appendix D discusses some additional related work.

A Technical lemmas

Lemma 4. For α > 0 and x ∈ [0 , 1 / 2 ] ,

For α > 0 and x > 0 such that αx ⩽ 1 ,

Proof. Regarding the first inequality of the Lemma, we have

where the first inequality holds for x ∈ [0 , 1 / 2 ] . The second inequality of the Lemma is proven by

where the second inequality holds when αx ⩽ 1 .

Lemma 5. There exists an absolute constant C > 0 such that, for L ⩾ C and x ⩾ 1 ,

Proof. We have, for i ∈ { 1 , . . . , d } , which concludes the proof.

The next lemma shows that the PL condition implies exponential convergence of the gradient flow. It is a well-known fact (see, e.g., Rebjock and Boumal, 2023, for an overview of similar conditions), proved here for completeness.

Lemma 7.

Let

f

:

R

D

→

R

the gradient flow dynamics be a differentiable function lower bounded by

If f satisfies the Polyak-Łojasiewicz inequality for t ⩾ 0

then x ( t ) converges to a global minimizer x ∞ , and, for t ⩾ 0 ,

f

min

∈

R

, and consider

Proof. By the chain rule,

Plugging in the Polyak-Łojasiewicz inequality,

Thus

To solve this differential inequality, one can for instance use the comparison theorem (Michel Petrovitch, 1901), which states that f ( x ( t )) -f min is smaller that the solution g of the initial value problem

This shows that

By independence of the ( N ij ) 1 ⩽ j ⩽ d , we deduce that ( Nh ) i follows a N (0 , ∥ h ∥ 2 ) distribution. Furthermore, by independence of the rows of N , the ( Nh ) i are independent. Thus

follows a χ 2 ( d ) distribution. Moving on to Y 2 , we have

Thus Y 2 follows a centered Gaussian distribution, and by independence of the ( N ij ) 1 ⩽ i,j ⩽ d , its variance is equal to

To show convergence of the iterates, let F ( t ) = √ µ ∫ t 0 ∥∇ f ( x ( s )) ∥ ds +2 √ f ( x ( t )) -f min . We have by the Polyak-Łojasiewicz inequality

Then

showing that x ( t ) converges.

Lemma 8. With the notation introduced in Section 2, the following identities holds

and

Proof. We have

where the scalar product is equal to zero because ∇ R 1 ( w ⋆ ) = -2 n X ⊤ ( y -Xw ⋆ ) = 0 . This gives the first identity of the Lemma. Next, denoting π ( w prod ) the projection of w prod on the orthogonal subspace to the kernel of X , we have

By the formula (6) for the gradient of R 1 ,

where the last lower bound holds because both w ⋆ and π ( w prod ) are in the orthogonal to the kernel of X . Plugging in the formula above, we obtain that

Finally, to obtain the upper bound on the gradient, note that

Lemma 9. Take W 1 , . . . W L ∈ R d × d such that, for all k ∈ { 1 , . . . , L } ,

where M ⩾ 1 and m ∈ (0 , 1) . Then, for all θ such that max 1 ⩽ k ⩽ L ∥ θ k ∥ 2 ⩽ m 2 4 M 2 , letting ˜ W k = W k + θ k L , we have and

Proof. First note that the assumptions imply that, for any 1 ⩽ j < k ⩽ L ,

It shall come in handy to extend this formula to the case where j = k , where we define the empty matrix product to be equal to the identity matrix, which has an operator norm of 1 ⩽ M m .

Now, take any θ as in the Lemma and any h 0 ∈ R d . Let, for k ⩾ 0 ,

and

Then and

Thus

Since ˜ h 0 -h 0 = 0 , we get by recurrence that, for k ⩾ 1 ,

From there, let us prove by recurrence that

This equation holds for k = 0 since M ⩾ 1 . Next, assume that it holds up to a certain rank k -1 , and let us prove it at rank k . From (10), we get that

Since M ⩾ 1 and m < 1 , the bound on ∥ θ j ∥ 2 from the assumptions of the Lemma implies in particular that ∥ θ j ∥ 2 ⩽ m 2 M . Utilizing this, as well as (9) and the recurrence hypothesis (11) up to rank k -1 , we get

Then and

This concludes the proof of the recurrence hypothesis at rank k . We therefore get that

To prove the lower bound on the smallest singular value of ˜ W k . . . ˜ W 1 , observe that

by (12). Finally, by (9), the bound on ∥ θ j ∥ 2 from the assumptions, and the upper bound we just proved on ∥ ˜ h j -1 ∥ 2 , we have

This concludes the proof.

B Proofs

B.1 Proof of Theorem 2

For a twice continuously differentiable function f : R D → R , the largest eigenvalue S of its Hessian H ( f ) : R D → R D × D at some x ∈ R D admits the variational characterization

In our case, the parameters are a set of matrices and the formula above translates into

We now take W to be an arbitrary minimizer of R L . To obtain the lower bounds, consider for ξ ⩽ 1

where the M k ∈ R d k × d k -1 are parameters that will be chosen later (depending on the W k but not on ξ ). Then

To alleviate notations, we drop the dependence of ˜ W k on ξ . Recall that, for any parameters W 0 ,

with

For minimizers of the empirical risk, ∇ k R L ( W ) = 0 , so

Minimizers of the empirical risk also satisfy that X ⊤ Xw prod = X ⊤ Xw ⋆ . Thus, by adding and subtracting differences,

Here, and in the remainder of this proof, the notation O is taken with respect to the limit when ξ → 0 , everything else being fixed. In particular, inspecting the expression above, we see that ∇ R ( ˜ w prod ) = O ( ξ ) , and therefore, going back to (15), that

By the inequality of arithmetic and geometric means, and by subadditivity of the operator norm,

By definition of ¯ ∆ k ,

where the last identity comes from the formulas ∇ R 1 ( ˜ w prod ) = 2 n X ⊤ X ( ˜ w prod -w prod ) and X ⊤ Xw prod = X ⊤ Xw ⋆ . Thus

and, by (18),

Coming back to (16), we have

At this point, the computations diverge for the two lower bounds we want to prove. For the first inequality, we take M k = β k W k , where the β k are free parameters to be optimized. We have

Therefore, by (19),

Furthermore,

By (14), we get

The first lower bound unfolds by taking β k = 1 / ∥ W k ∥ F .

Moving on to the second lower-bound, we now take

where again the β k are free parameters to be optimized. We therefore have

where the last line unfolds from subadditivity of the operator norm. We let A k = W ⊤ k +1 . . . W ⊤ L pw ⋆ ⊤ 2 n X ⊤ XW ⊤ 1 . . . W ⊤ k -1 , and choose β k = ∥ A k ∥ 2 . Then

Coming back to (19), we get

Furthermore,

Therefore, we obtain, by (14),

We can lower bound the sum similarly to (18):

and

Thus, using the Cauchy-Schwarz inequality, we get

Therefore

We finally obtain

which gives the second lower bound.

To obtain the upper bound on S min , we construct explicitly a minimizer, and upper bound its sharpness. More precisely, let the W k be rank-one matrices, such that ∥ W k ∥ 2 = ∥ w ⋆ ∥ 1 /L / ∥ p ∥ 1 /L , successive matrices have aligned first singular vectors, the first right singular vector of W 1 is aligned with w ⋆ , and the first left singular vector of W L is aligned with p . We then have

meaning that the network minimizes the loss. We now upper bound its sharpness using (13), where we recall that the matrix appearing in the numerator is denoted by ∆ k and satisfies ∆ k = ¯ ∆ k + O ( ξ 2 ) , where ¯ ∆ k is given by (17). Contrarily to the proof of the lower bounds where we exhibited a specific direction ˜ W , we here seek an upper bound valid for all ˜ W . Recall that, by (16), we have

By subadditivity of the operator norm,

Moving on to bounding the squared Frobenius norm of ∆ k , we observe that ¯ ∆ k decomposes as a rank-one matrix. We split cases for k = 1 and k > 1 . First, for k = 1 , we have, by subadditivity of the operator norm and by (21),

Then

For k > 1 , we have

Let us now bound ∥∇ R 1 ( ˜ w prod ) ⊤ W ⊤ 1 ∥ 2 . By (20), separating the first term, we have

Finally, recall that W 1 is rank-one and its first right singular vector is aligned with w ⋆ . A short computation therefore shows that

Thus

Then, coming back to (22), for k > 1 ,

Thus

where the second inequality holds by the Cauchy-Schwarz inequality. Thus, by (13), putting together the bounds on ∥ ∆ k ∥ 2 F for k = 1 and k > 1 ,

Therefore

which concludes the proof.

B.2 Proof of Lemma 1

This identity can be shown by noting that the identity is trivially true for t = 0 , then differentiating on both sides with respect to time, and using (5). We refer, e.g., to Arora et al. (2018) for a detailed proof.

B.3 Proof of Lemma 2

Before proving the three statements of the lemma in order, we let

Note that ¯ ε ⩽ ε .

First statement. The claim is true for k = L since W L is a (row) vector. For k ∈ { 1 , . . . , L -1 } , taking the 2 -norm in Lemma 1, we have

Thus, using ∥ A ⊤ A ∥ 2 = ∥ AA ⊤ ∥ 2 = ∥ A ∥ 2 2 ,

We now take the trace in Lemma 1 to obtain

Combining with the inequality above, we have that

Summing from k to L -1 and telescoping, we have

The first two terms compensate since W L is a vector, and the remainder of the terms is less than ¯ ε ⩽ ε by definition. This gives the first statement of the Lemma.

Second statement. Assume without loss of generality that j > k . By recurrence over (23), we obtain that

The reverse inequality can be shown similarly: by considering again Lemma 1 and taking the 2 -norm, we get

Thus

As previously, we get by recurrence that

Combined with the reverse bound above, this gives the second statement of the lemma.

Third statement. Let us lower and upper bound u ⊤ k W ⊤ k +1 W k +1 u k . We have on the one hand, by Lemma 1,

On the other hand,

The 2 -norm above is equal to the second largest eigenvalue of W ⊤ k +1 W k +1 , which is the square of the second largest singular value of W k +1 . In particular, it is lower than ∥ W k +1 ∥ 2 F -∥ W k +1 ∥ 2 2 which is the sum of the squared singular values of W k +1 except the largest one. We obtain

by the first statement of the Lemma. Combining the lower and upper bound of u ⊤ k W ⊤ k +1 W k +1 u k , we get

Thus

by the second statement of the Lemma. We finally obtain

by definition of ε and ¯ ε .

B.4 Proof of Theorem 3

We lower bound the first term in the sum of the left-hand side. Recall that, by (5), we have

thus

We show that ∥ W L ( t ) . . . W 2 ( t ) ∥ 2 is large by distinguishing between two cases depending on the magnitude of σ 1 ( t ) = ∥ W 1 ( t ) ∥ 2 . To this aim, let C > √ 2 εL .

Large spectral norm. We first consider the case where

where the second inequality unfolds from (25). Then, again by Lemma 2, for k ∈ { 1 , . . . , L -1 } ,

It is always possible (without loss of generality) to choose the orientation of the u k and v k such that ⟨ v k +1 , u k ⟩ ⩾ 0 for any k ∈ { 1 , . . . L -1 } . Making this choice, the equation above implies that

For x ∈ [0 , 1] , √ 1 -x ⩾ 1 -x , and thus

Let us show that this implies a lower bound on ∥ W L . . . W 2 ∥ 2 . To do so, let us denote recursively

We then have x L = W L . . . W 2 x 1 , thus

Our goal is thus to lower bound ∥ x L ∥ 2 , which entails a lower bound on ∥ W L . . . W 2 ∥ 2 . To this aim, first note that, for any k ∈ { 1 , . . . , L -1 } ,

where the last equation stems from (26). Denote α k = ⟨ x k ∥ x k ∥ 2 , u k ⟩ . Then the previous equation shows that √

Further note that ∥ x k +1 ∥ 2 ⩽ σ k +1 ∥ x k ∥ 2 , thus

By Lemma 2, for k ∈ { 1 , . . . , L -1 } ,

By recurrence,

Coming back to (27), we have

thus by recurrence,

Finally, by definition, α 2 = ⟨ x 2 ∥ x 2 ∥ 2 , u 2 ⟩ . Since x 2 = W 2 x 1 = W 2 v 2 = σ 2 u 2 , we obtain that α 2 = 1 , and thus

We finally get

which is a positive quantity by definition of C .

Small spectral norm. We now inspect the case where (25) is not satisfied, that is, σ 1 ( t ) ⩽ C . First note that, by the formula (6) for the gradient of R 1 ,

Thus, for ∥ w prod ∥ 2 ⩽ ∥ w ⋆ ∥ 2 , w prod ↦→ R 1 ( w prod ) is 4Λ ∥ w ⋆ ∥ 2 -Lipschitz. Let us use this property to lower bound ∥ w prod ( t ) ∥ 2 by a constant independent of t , for t ⩾ 1 . Either we have ∥ w prod ( t ) ∥ 2 ⩾ ∥ w ⋆ ∥ 2 , or ∥ w prod ∥ 2 ⩽ ∥ w ⋆ ∥ 2 , but then, by the Lipschitzness property,

where we recall that R 0 is the risk associated to the null parameters. Furthermore, for t ⩾ 1 ,

̸

Thus, for t ⩾ 1 ,

To summarize, we proved that, for t ⩾ 1 ,

where we recall that ∥ w ⋆ ∥ 2 > 0 by assumption (see Section 2). Furthermore,

Then, for t ⩾ 1 ,

Conclusion. Combining (28) and (29), we obtain that, for t ⩾ 1 ,

where µ 1 > 0 by the proof above. Then, for t ⩾ 1 , by (24),

Thus, taking µ = 4 µ 1 λ > 0 , for t ⩾ 1 ,

B.5 Proof of Corollary 1

We first show that Assumption ( A 2 ) implies a number of estimates that are useful in the following. Since ∥ w ⋆ ∥ 2 ⩾ 1 , we have thus

Moreover, so we also have

Let us now prove the Corollary. We first note that Theorem 3 implies exponential convergence of the empirical risk to its minimum by Lemma 7. This also implies the (exponential) convergence of w prod to w ⋆ , since the covariance matrix X ⊤ X is full rank, so w ⋆ is the unique minimizer of R 1 , and, by Lemma 8,

Furthermore,

Let us show that, for t large enough,

If it were not the case, then

where the last inequality holds by Lemma 4 since

which holds by (31). Then, we obtain

since which also holds by (31). The inequality ∥ w prod ∥ 2 ⩽ 3 ∥ w ⋆ ∥ 2 4 contradicts the fact that w prod converges to w ⋆ , thus proving (33). Then, we have

where the last inequality holds by (30). Furthermore, by Lemma 2, for all k ∈ { 1 , . . . , L } ,

This brings the first inequality of the Corollary. We now show the second inequality of the Corollary. First note that, by Lemma 2,

since both quantities are equal to the sum of the squared singular values of W k except the largest one. Then, adding and substracting,

̸

by the first inequality of the Corollary. Moreover, using again the first inequality of the Corollary and Lemma 2, we have that

Since u L = 1 , we deduce that

where in the last step we used Lemma 4 which is valid since

which holds by (32). By the triangular inequality, we now have

Thus

Using (31) and (32),

Moreover, the product of the singular values can be lower-bounded by the smallest one to the power L , so

Let us show that, for t large enough,

If it were not the case, we would have

where the second inequality holds by Lemma 4 since

by (31), and the third since

also by (31). Since 9 / 8 > 1 , the inequality (34) is a contradiction with the fact that ∥ W ⊤ 1 . . . W ⊤ L ∥ 2 = ∥ w prod ∥ 2 →∥ w ⋆ ∥ 2 , showing that, for t large enough,

By Lemma 2, for all k ∈ { 1 , . . . , L } ,

Thus since

which holds true by Assumption ( A 2 ) since

since ∥ w ⋆ ∥ 2 ⩾ 1 .

B.6 Proof of Corollary 2

The lower bound unfolds by definition of S min since W SI is a minimizer of the empirical risk. To obtain the upper bound, we proceed similarly to the proof of Theorem 2. For simplicity, we denote W = W SI in the remainder of the proof. We have, as in the proof of Theorem 2,

with and

We recall that, as in the proof of Theorem 2, the notation O is taken with respect to the limit when ξ → 0 , everything else being fixed. By subadditivity of the operator norm and by Corollary 1, we have

Moving on to bounding the Frobenius norm of ∆ k , we observe that the dominating term of this matrix decomposes as a rank-one matrix. Thus, again by subadditivity of the operator norm, equation (36) and Corollary 1,

Then

Thus, by (35),

Therefore

which concludes the proof by the second lower bound on S min of Theorem 2, where here ∥ p ∥ = 1 and a ⩾ λ > 0 .

B.7 Proof of Theorem 4

To alleviate notations, we omit in this proof to write the explicit dependence of parameters on time. Starting from (5), since the gradient decomposes as a rank-one matrix, we have

Then

Recall the notation introduced in Section 5

We now let u > 0 (whose value will be specified next), and

Let us lower bound the minimum singular values of Π k : and Π : k uniformly for t ∈ [0 , t ⋆ ] by Lemma 3. By definition of t ⋆ , for t ⩽ t ⋆ and for any k ∈ { 1 , . . . , L } , ∥ θ k ∥ 2 ⩽ 1 64 exp( -2 s 2 -4 u ) . Therefore, renaming ˜ C 1 to ˜ C 4 the constants of Lemma 3, and taking u = ˜ C 3 , we get that, if

then, with probability at least

it holds for t ⩽ t ⋆ and for all k ∈ { 1 , . . . , L } that

and

By symmetry, the same statement holds for Π L : k instead of Π k :1 with the same probability. By the union bound, the event

holds with probability at least

Let now and E 3 the event that the gradient flow dynamics converge to a global minimizer W RI of the risk satisfying the statement (8). We show next that, if E 1 and E 2 hold, then E 3 must hold, which shall conclude the proof of the Theorem with

Under E 1 , we have

thus by (37)

Therefore we get the PL condition, for t ⩽ t ⋆ ,

with

By Lemma 7, this implies that, for t ⩽ t ,

Let us now show that t ⋆ = ∞ . We have, since θ (0) = 0 and by definition of the gradient flow,

We now upper bound the gradient as follows: starting again from (5),

By (39) and (41), for t ⩽ t ⋆ ,

Plugging this into (42), we get, for t ⩽ t ⋆ ,

where the last equality comes from the definition of µ . By E 2 and by definition (40) of C 4 , for t ⩽ t ⋆ ,

If we had t ⋆ < ∞ , we would have, by definition of t ⋆ , ∥ θ k ( t ⋆ ) ∥ F ⩾ 1 64 exp( -2 s 2 -4 ˜ C 3 ) . This contradicts the equation above, showing that t ⋆ = ∞ . By (41), this implies convergence of the risk to its minimum. We also see by (44) that ∇ k R L ( W ) is integrable, so W has a limit as t goes to infinity. This limit W RI is a minimizer of the risk and satisfies the condition (8) by definition of t ⋆ = ∞ .

Extension to multivariate regression. We emphasize that a very similar proof holds in the case of multivariate regression, where the neural network is defined as the linear map from R d to R d

and we now aim at minimizing the mean squared error loss

where X,Y ∈ R n × d . In this case, as shown in Zou et al. (2020); Sander et al. (2022), the following bounds on the gradient hold:

and

Comparing with (37) and (43), the only difference is the absence of ∥ p ∥ 2 here. From there, the same computations as above hold (taking ∥ p ∥ 2 = 1 ), and give the following result.

Theorem 5. There exist C 1 , . . . , C 5 > 0 depending only on s such that, if L ⩾ C 1 and d ⩾ C 2 , then, with probability at least

the gradient flow dynamics (4) converge to a global minimizer W RI of the risk. Furthermore, the minimizer W RI satisfies

if

B.8 Proof of Lemma 3

We begin by proving the result when θ = 0 , then explain how to extend to any θ in a ball around 0 . Recall that Π k :1 denotes the product of weight matrices up to the k -th. Let

and denote by A the event that for all k , ∥ Π k :1 ∥ 2 ⩽ M s,u and B the event that for all k , σ min (Π k :1 ) ⩾ m s,u . We begin by bounding the probability of ¯ A , then bound the probability of ¯ B ∩ A .

Useful identities. To this aim, we first introduce some notations and derive some useful identities. We let F k the filtration generated by the random variables N 1 , . . . , N k . For some h 0 ∈ R d , we let

In particular, h k +1 and h k are related through

Taking the squared norm, we get

Dividing by ∥ h k ∥ 2 2 and taking the logarithm leads to

Let

and Y k = Y k, 1 + Y k, 2 . Then, by Lemma 6,

This shows in particular that Y k, 1 and Y k, 2 are independent of N 1 , . . . , N k , and depend only on N k +1 . Then, letting

both are sums of i.i.d. random variables. In particular,

Bound on P ( ¯ A ) . We first bound the deviation of the norm of h k = Π k :1 h 0 for any fixed h 0 ∈ R d , then conclude on the operator norm of Π k :1 by an ε -net argument. For any (fixed) h 0 ∈ R d and u > 0 , we have

by using ln(1 + x ) ⩽ x . Thus

by the union bound. We now study the two deviation probabilities separately. Beginning by the first one, recall that Ld s 2 S L, 1 follows a χ 2 ( Ld ) distribution. Chi-squared random variables are subexponential, and more precisely satisfy the following property (Ghosh, 2021): if X ∼ χ 2 ( c ) and u > 0 , then

Since the S k, 1 are increasing, we have, for u > 0 ,

Moving on to S k, 2 , we have, for u, λ > 0 ,

Furthermore, exp( λS k, 2 ) is a sub-martingale, since S k, 2 is a martingale and by Jensen's inequality:

Thus, by Doob's martingale inequality,

Furthermore, since S L, 2 ∼ N (0 , 4 s 2 d ) ,

This quantity is minimal for λ = du 8 s 2 . We get, for all u > 0 ,

Therefore,

Therefore, for any u > 0 ,

To conclude on the operator norm of Π k :1 , consider Σ a 1 / 2 -net of the unit sphere of R d . By Vershynin (2018, Corollary 4.2.13), it is possible to take such a net of cardinality 5 d . Let us show that, for any u ∈ R ,

Indeed, assume that there exists k such that ∥ Π k :1 ∥ 2 ⩾ 2 exp( s 2 2 + u ) . By definition of the operator norm, there exists x on the unit sphere of R d such that

But then, by definition of Σ , there exists h 0 ∈ Σ such that ∥ x -h 0 ∥ 2 ⩽ 1 / 2 . Then

Therefore, by the triangular inequality,

Then

which proves (50). By the union bound, we conclude that

where now h 0 denotes any unit-norm fixed vector in R d . Thus, by (49),

This upper bound will be simplified in the conclusion of the proof.

Bound on P ( ¯ B ∩ A ) . We now move on to proving the lower-bound on σ min (Π k :1 ) . We again use an ε -net argument, as follows. Let Σ be an ε -net for ε = m s,u M s,u . Then

Indeed, assume that

A

holds, that is

∥

Π

k

:1

∥

2

⩽

M

s,u

, and that there exists

By definition of the singular value, there exists x on the unit sphere of R d such that

But then, by definition of Σ , there exists h 0 ∈ Σ such that ∥ x -h 0 ∥ 2 ⩽ ε . Then

Under A , the right-hand side is smaller than εM s,u . Therefore, by the triangular inequality,

By the union bound, we conclude that

k

such that

where the cardinality of the net is given by Vershynin (2018, Corollary 4.2.13). We now take a fixed h 0 ∈ R d , and compute

Denote by E the event

We have, by (45),

Using the inequality ln(1 + x ) ⩾ x -x 2 for x ⩾ -1 / 2 , we obtain

Thus, by the union bound,

We handle both terms separately. Beginning by the second term, we have, for k ∈ { 1 , . . . L } ,

where we used (46) and the tail bound P ( N ⩾ u ) ⩽ e -u 2 / 2 if N ∼ N (0 , 1) . Moving on to the first term, we have

where we let S k, 3 = ∑ k -1 j =0 Y 2 j, 2 . By definition of m s,u , we have

Thus we can split the probability into two parts by the union bound:

Let us bound each probability separately. We have, for u > 0 ,

by symmetry of S k, 2 and by (48). Moving on to S k, 3 , we have that

thus

By monotonicity of S k, 3 and by (47),

Putting everything together, we proved that with

and

Bounding the probability of failure. Putting together the two main bounds we showed, we have P ( ¯ A ∪ ¯ B ) = P ( ¯ A ) + P ( ¯ B ∪ A )

This expression is valid for all u, s > 0 and L, d ⩾ 1 . We now simplify the expression of our upper bound by algebraic computations using the assumptions of the Theorem. To somewhat alleviate the technicality of the computations, we stop at this point tracking some of the explicit constants and let C denote a positive absolute constant that might vary from equality to equality. We show next that the conditions imply that

This shall conclude the proof of the Lemma with

First note that the conditions (51) imply that

We study the terms of the bound on P ( ¯ A ∪ ¯ B ) one by one. First, we have by (52) that

Next,

where the first inequality uses u ⩾ s 2 by (51), and the last one uses that Lu s 2 ⩾ C . This is true since L ⩾ C and u ⩾ Cs 2 by (51). Then we can bound this term by

since u ⩽ CL which is implied by the assumption u ⩽ CL 1 / 4 in (51). Next,

where we used that the exponential of a positive term is greater than 1 . Then

since u ⩾ s 2 , d ⩾ 4 by (51). Thus we can bound the three remaining terms appearing in the bound of P ( ¯ A ∪ ¯ B ) , as follows:

since

This is the case by (52), which implies that

Next, by (53),

since du ⩾ u ⩾ Cs 2 by (51). Thus

where the second inequality uses that L ⩾ C by (51), and Lu ⩾ C since L ⩾ C and u ⩾ C by (51) and (52). We can also bound this term by

since L ⩾ Cu/s 2 , using first that L ⩾ Cu 4 then that u ⩾ Cs -2 / 3 , by (51). Regarding the last term of the bound of P ( ¯ A ∪ ¯ B ) , we have by (53) that

Let us show that this implies that

This statement is true because the three terms appearing inside the exponential of the right-hand side of (54) can be bounded by C √ Ldu 2 s 2 . More precisely, we have

which holds by (51);

which is implied by L ⩾ C and u ⩾ Cs 2 by (51); and

which is implied by L ⩾ C and u ⩾ Cs by (51) and (52). Finally, we use Lemma 5 to bound the right-hand side of (55) by

This is possible for L ⩾ C and du 2 32 s 2 ⩾ 1 , which is implied by d ⩾ C and u ⩾ Cs , by (51) and (52). Collecting everything, we bounded P ( ¯ A ∪ ¯ B ) by

which concludes the proof when θ = 0 .

Summary when θ = 0 . In summary, we proved so far the following result: there exist C 1 , . . . , C 4 > 0 depending only on s such that, if

then, with probability at least it holds for all k ∈ { 1 , . . . , L } that

and

Conclusion for arbitrary θ . The conclusion is a direct application of Lemma 9, with W k = I + s √ Ld N k . The size of the admissible ball for θ is

for d ⩾ 4 , which concludes the proof.

Comparison with the bounds of Zhang et al. (2022). Theorem 1 of Zhang et al. (2022) gives an upper bound on the singular values of residual networks at initialization, and their Theorem 2 gives a lower bound on the norm of the activations. Comparing with our results, we note two important differences. First, their probability of failure grows quadratically with the depth, whereas ours is independent of depth. This is achieved by a more precise martingale argument making use of Doob's martingale inequality. Second, their lower bound incorrectly assumes that χ 2 random variables are sub-Gaussian (see equation (21) of their paper), while in fact they are only sub-exponential (Ghosh, 2021). Finally, their upper bound holds for the product

for any 1 ⩽ j ⩽ k ⩽ L , which could seem stronger than our result stated for j = 1 . In fact, both statements are equivalent, because it is possible to deduce the statement for any j by combining the upper bound and the lower bound for j = 1 . The precise argument is given in the beginning of the proof of our Lemma 9.

B.9 Proof of Corollary 3

The beginning of the proof is very similar to the one of Corollary 2. Denoting W := W RI , we have

with and

At this point, the proofs diverge. We have, by subadditivity of the operator norm,

Let us now briefly recall the outline of the proof of Theorem 4, which will be useful in bounding the quantity above. The proof shows the existence of ˜ C 3 depending only on s such that, with high probability (which is exactly the probability in the statement of the Theorem), we have for all t ⩾ 0 and k ∈ { 1 , . . . , L } that

Under this high-probability event, the proof of Theorem 4 shows convergence of the gradient flow to W = W RI . In particular, this means that W also verifies the bounds on the operator norm of the matrix products. We therefore obtain

Moving on to bounding the Frobenius norm of ∆ k , we have

by bounding the three norms by the expressions given above. Then

Thus, by (56),

Therefore

To conclude, we need to upper bound ∥ p ∥ 2 by a constant times ∥ w ⋆ ∥ 2 . To this aim, we leverage the bound from the assumptions of Theorem 4 on the risk at initialization, to show that ∥ p ∥ 2 cannot be too far away from ∥ w ⋆ ∥ 2 . More precisely, by Lemma 8, since the covariance matrix X ⊤ X is full rank,

Thus

Then, by the triangular inequality,

by (57) and by the assumption of Theorem 4 on the risk at initialization. We now note that the value of C 3 is given by (40) as

where ˜ C 2 ⩽ d by (38). Thus

Denoting

we therefore obtain that ∥ w ⋆ ∥ 2 ⩾ C ′ ∥ p ∥ 2 . Therefore, by (58),

Thus, by the second lower bound on S min from Theorem 2, and since a ⩾ λ > 0 ,

which concludes the proof by setting

C Experimental details, additional plots, and additional comments

Our code is available at https://github.com/PierreMarion23/implicit-reg-sharpness . Our framework for experiments is JAX (Bradbury et al., 2018). The experiments take around 3 hours to run on a laptop CPU.

Setup. We take n = 50 , d = 5 , L = 10 . The design matrix X is sampled from an isotropic Gaussian distribution. The target y is computed in two steps. First, we compute y 0 = Xw true + ζ , where w true and ζ are standard Gaussian vectors. Then, we compute w ⋆ 0 as the optimal regressor of y 0 on X . Finally, we let y = y 0 / ∥ w ⋆ 0 ∥ and w ⋆ = w ⋆ 0 / ∥ w ⋆ 0 ∥ . This simplifies the expressions of our bounds by having w ⋆ of unit norm. All Gaussian random variables are independent.

Details of Figure 1. We consider a Gaussian initialization of the weight matrices, where the scale of the initialization (x-axis of some the graphs) is the standard deviation of the entries. All weight matrices are d × d , except the last one which is 1 × d . The square distance to the optimal regressor corresponds to ∥ w prod -w ⋆ ∥ 2 2 . The largest eigenvalue of the Hessian is computed by a power iteration method, stopped after 20 iterations. In Figures 1a and 1b, the 95% confidence intervals are plotted. The number of gradient steps and number of independent repetitions depend on the learning rate, and are given below.

For large values of the initialization scale, it may happen that the gradient descent diverges. Figure 3 shows the probability of divergence depending on the initialization scale and the learning rate.

0.25

0.30

0.35

0.40

0.45

0.50

0.55

0.60

Scale of initialization

0.0

0.2

0.4

0.6

0.8

1.0

Probability of divergence

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

l. rate: 0.1

l. rate: 0.2

When the probability of divergence is equal to one, no point is reported in Figure 1. When it is strictly between 0 and 1 , the confidence intervals are computed over non-diverging runs.

Figures 1c and 1d show one randomly-chosen run each. The plots are subsampled 5 times for readability, due to the oscillations in Figure 1d.

Residual initialization. Wenowconsider the case of a residual initialization as in Section 5. Results are given in Figure 4. The scale of the initialization now corresponds to the hyperparameter s in (2). The projection vector p ∈ R d is a random isotropic vector of unit norm, which does not change during training. For each learning rate, we use 4 , 000 steps of gradient descent, and perform 20 independent repetitions. The plots are similar to the case of Gaussian initialization, apart from the fact that the sharpness at initialization is better conditioned.

As previously, for large values of the initialization scale, it may happen that the gradient descent diverges. Figure 5 shows the probability of divergence depending on the initialization scale and the learning rate.

Details of Figure 2. The setup is the same as for the residual initialization. For each learning rate, we use 1 , 000 steps of gradient descent, and perform 50 independent repetitions. We take s = 0 . 25 .

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Scale of initialization

10

13

10

11

10

9

10

7

10

5

10

3

10

1

10

1

Squared distance to optimal regressor

Initialization

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

l. rate: 0.1

l. rate: 0.2

(a) Squared distance of the trained network to the empirical risk minimizer, for various learning rates and initialization scales. Training succeeds when the learning rate is lower than a critical value independent of the initialization scale.

0

100

200

300

400

Training steps

10

13

10

11

10

9

10

7

10

5

10

3

10

1

Squared distance to optimal regressor

0

100

200

300

400

Training steps

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

sharpness

2/lr

(c) Evolution during training of the squared distance to the empirical risk minimizer and of sharpness, for η = 0 . 02 and an initialization scale of 0 . 5 . The network does not enter edge of stability.

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Scale of initialization

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

Initialization

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

0

100

200

300

400

Training steps

10

4

10

3

10

2

10

1

Squared distance to optimal regressor

0

100

200

300

400

Training steps

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

sharpness

2/lr

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Scale of initialization

0.0

0.2

0.4

0.6

0.8

1.0

Probability of divergence

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

l. rate: 0.1

l. rate: 0.2

Underdetermined regression and link to generalization. Although this is not our original motivation, we note that a simple change to our setting allows to make appear the connection between sharpness and generalization. To this aim, we consider the underdetermined case, where the number of data is lower than the dimension (while keeping the rest of the setup identical). Figure 6 shows in this case a correlation between generalization and sharpness. This suggests that the tools developed in the paper could be used in this case to understand the generalization performance of deep (linear) networks, and we leave this analysis for future work. We also qualitatively observe in Figure 6 a similar connection between learning rate, initialization scale and sharpness as in the case of full-rank data (Figure 1b). We take here n = 15 , d = 20 , L = 5 . The number of gradient steps and number of

independent repetitions depend on the learning rate, and are given below. Other technical details are as Figure 1.

0.10

0.15

0.20

0.25

0.30

0.35

Scale of initialization

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

Initialization

l. rate: 0.001

l. rate: 0.004

l. rate: 0.01

17

31

56

100

177

316

562

1000

1778

Sharpness after training

0.32

1

3.16

10

31.62

Generalization gap

Figure 6: Experiment with a deep linear network and a degenerate data covariance matrix, where the number of data n is less than the dimension d .

Non-linear MLPs. As a first attempt to extend our results to non-linear networks, we consider the case of non-linear MLPs. The non-linearity is GELU (Hendrycks and Gimpel, 2016), a smooth version of ReLU, which we chose because smoothness is necessary in order to compute the sharpness. We qualitatively observe in Figure 7 a similar connection between learning rate, initialization scale and sharpness as for deep linear neural networks (Figure 1b). For large initialization, the sharpness after training plateaus at 2 /η , as in the linear case. For small initialization, the sharpness after training is less that 2 /η , and is close to the bounds in the linear case (dotted black lines). For this experiment, we consider noiseless data, meaning that y 0 = Xw true (see paragraph 'Setup' above for notations). We perform 20 independent repetitions of each experiment. The number of gradient steps depends on the learning rate, and is given below. Other details are as for Figure 1. Finally, we also performed the same experiment in the case of noisy data y 0 = Xw true + ζ (no plot reported). We observed that the sharpness of the network reaches 2 /η , for every learning rate and initialization scale reported in Figure 7. We suspect that this is because the network (over)fits the noise in the data, resulting in a high sharpness.

Comments on the connection between gradient flow and gradient descent. In this paper, we show that gradient flow from a small-scale initialization is driven towards low-sharpness regions. This should imply that gradient flow and gradient descent up to a reasonably large learning rate should follow the same trajectory when starting from small-scale initialization, because they do not

0.45

0.50

0.55

0.60

0.65

0.70

Scale of initialization

10

1

10

0

10

1

10

2

10

3

10

4

Sharpness

Initialization

l. rate: 0.005

l. rate: 0.02

l. rate: 0.07

go in regions of high sharpness where the difference between gradient flow and gradient descent would become significant. This intuition is supported by Figure 1b where we see that, for small initializations, the sharpness after training is independent of the learning rate. We leave further investigation of these questions for future work.

D Additional related work

Progressive sharpening. Cohen et al. (2021) show that the edge of stability phase is typically preceded by a phase of progressive sharpening, where the sharpness steadily increases until reaching the value of 2 /η . Our setting of small-scale initialization presents an example of such a progressive sharpening (although we make no statement on the monotonicity of the increase in sharpness). Other works have proposed analyses of progressive sharpening. Wang et al. (2022) suggest that progressive sharpening is driven by the increase in norm of the output layer. MacDonald et al. (2023) assume from empirical evidence a link between sharpness and the magnitude of the input-output Jacobian, and show that the latter has to be large for the loss to decrease. Finally, Agarwala et al. (2023) propose and analyze a simplified model with quadratic dependence in its parameters, which exhibits a progressive sharpening phenomenon.

Connection with deep matrix factorization. Regression with deep linear networks can be seen as an instance of a matrix factorization problem. There is a well-established literature studying the implicit regularization of gradient descent for this class of problem (see, e.g., Gunasekar et al., 2017; Arora et al., 2019b; Li et al., 2021; Yun et al., 2021). However, this line of work study under-determined settings where there are an infinite number of factorizations reaching zero empirical risk, and study the implicit regularization in function space. On the contrary, we consider an overdetermined setting where there is a single optimal regressor, and study the regularization in parameter space.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims regard the theoretical analysis of sharpness of deep linear networks. The corresponding results are given in Sections 3, 4, and 5, and proven in Appendix B.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper presents work of theoretical nature, and explains clearly the setting and assumptions (e.g., linear activation function) in which the results are applicable.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are stated in the main paper, and all proofs are given in Appendix B.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental details are given in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is given in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental details are given in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All plots have error bars (except the plots showing the trajectory of gradient descent along one run, for illustration).

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The experiments are small-scale and run on a laptop CPU. The runtime is given in Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics, and attest that this paper conforms to the Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a foundational research paper, not tied to particular applications, let alone deployments.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the papers for code we are using in Appendix C.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the behavior of deep linear neural networks in the context of overdetermined univariate regression, which is the setting of a univariate response $y$, more samples than input dimensions, and nonsingular data covariance matrix. \n\nThe first result concerns the empirical risk minimizer (this is not equal to the gradient descent or gradient flow estimator)  and shows a lower bound on the sharpness of the ERM.\n\nThe next set of results show that gradient flow, a limit of gradient descent with a vanishing learning rate, implicitly regularizes the network towards flat minima, with sharpness close to a constant times the lower bound. This is proven for both small-scale and residual initializations.\n\nSTRENGTHS:\n* The exposition is excellent. There are three main results, each given in its own section corresponding to Section 3, 4, and 5. \n* The setting described in Section 2 is mathematically clear\n* The first main result, Theorem 1, reaches a similar conclusion to previous work in Mulayoff and Michaeli (2020) but relaxes identity data covariance assumption and uses a supposedly simpler proof.\n* The next set of results characterize the gradient flow minimizers. This is indeed of independent interest, although the connection to sharpness is also interesting. \n* The sharpness papers I have come across seem to focus on sharpness induced by SGD. It's interesting that here the authors were able to demonstrate a preference for sharpness for deterministic gradient flow, albeit with specific initialization schemes.\n\nWEAKNESSES:\n* I missed a rigorous theoretical connection between the three main results and the behavior of the gradient descent minimizer. If the connection is only empirical in nature, it might be good to highlight this\n* The work employs, to my eyes at least, a rather limited definition of sharpness. Are the results really very particular to sharpness being the largest eigenvalue of the Hessian?\n* I'm missing some motivation for the two initialization schemes studied. Are they studied simply because they've been studied before and so the results here can borrow from existing work? \n* I find the title misleading. An architecture cannot have a preference for flat minima(?). It must be the architecture in conjunction with how it is learned. In this case, the main results concern very specific configurations of gradient flow. \n* The implication for learning rate design from this theoretical analysis seems rather weak to me. I mean that I cannot envision a practitioner implementing a learning rate design based on the results presented here.\n\nQUESTIONS:\n* Apologies for this very naive question, but what does $S(\\mathcal W)$, the largest eigenvalue of $\\mathcal W=(W_1,\\ldots,W_L)$ mean exactly? This $\\mathcal W$ is not itself a matrix. \n* Each of your results in Sections 3, 4, and 5 are based on existing proof techniques. Specifically, Section 3 proof is based on Mulayoff and Michaeli 2020, Section 4 on Ji and Telgarsky (2020), and Section 5 on Zou et al 2020, Sander et al 2022. I am not familiar with these papers. Do they attempt to prove similar things to this paper? Do you reach the same conclusions but with different proof techniques?\n\nLIMITATIONS:\n* There are some obvious limitations that are par for the course in papers attempting to prove theoretical results in deep learning. Namely, the architecture here is extremely oversimplified. The data covariance being full rank is also unrealistic. Finally, the main results are about gradient flow and discrete-time stochastic gradient descent is surely very different in nature.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the implicit bias of gradient flow on deep linear networks for overdetermined univariate regression problems. A lower bound on the sharpness of any minimizer is first derived for the general data covariance matrix, then it is shown that gradient flow with small initialization finds a minimizer with a sharpness no larger than a constant times the lower bound, where the constant depends on the condition number of the data covariance matrix. A similar result is also derived for GF on linear residual networks.\n\nSTRENGTHS:\n1. Well-motivated problem. Good introduction. Clear writings. Theoretical results are carefully explained in the main paper.\n2. New convergence results for deep linear networks and linear residual networks, together with the upper bound on the sharpness.\n\nWEAKNESSES:\n1. **Tightness of Theorem 2**: In Mulayoff and Michaeli (2020), the lower bound on the sharpness is tight for whitened data: there exists a minimizer that achieves the lower bound. However, Theorem 2 in this paper only provides a lower bound. Is this lower bound improvable? For example, if the true lower bound has $\\Lambda$ instead of $\\lambda$, then the following results show that the sharpness minimizes found by GF is no larger than a constant times the lower bound, where the constant now does not depend on condition number.\n2. **Relevance of the problem setting**:  The sharpness is often considered to be affecting the generalization error of the trained network. However, this paper studies overdetermined linear regression, where every minimizer corresponds to the same input-output map $w^*$, thus having the same generalization error. Why does one care about studying the sharpness of the minimizer in this case, if sharpness does not affect generalization at all?\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a toy non-convex optimization problem, namely overdetermined univariate regression with a deep linear network. Their main contributions are:\n\n1) A lower bound on the sharpness of the minimizers of the empirical risk. In particular, if the step size is chosen too big, then gradient descent will fail to converge.\n\n2) They show that the sharpness of the minimizer found by gradient flow is at most a constant multiple of the above lower bound, for both small-scale and residual initialization. This constant does not depend on the width or depth, but only on the conditioning number of the data. This shows an implicit regularization towards flat minima (the empirical minimizers of the risk can have arbitrarily large sharpness).\n\nOn the technical side, they prove convergence of the GF and characterize the solution at convergence in both initialization regimes. Numerical simulations are provided to illustrate and substantiate their claims.\n\nSTRENGTHS:\n- The theoretical analysis is substantial. The results are novel and not straightforward. At the same time, they are easy to understand  and offer clear insights.\n\n- The qualitative picture obtained in this paper, with the interplay between step size, scale at initialization, and sharpness of the gradient descent solution, is convincing and surprisingly clear, given the complex non-convex problem.  In particular, I think it is interesting to have in a single model the behavior of sharpness, edge of stability and GD minimizer (even if only qualitatively).\n\n- The discussions are reasonably clear. The plots help a lot to understand the general message of the paper.\n\nWEAKNESSES:\nThe paper considers a simple setting: deep linear network and underdetermined regime. This allows the authors to characterize interesting and non-trivial behavior. However, it is unclear how much these results can extend beyond this simple setting.\n\nQUESTIONS:\nHere, it is assumed that the data matrix is full rank. Is it not sufficient to assume that there exists a unique minimizer?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
The authors show three new results for deep linear networks (DLN).  First, they show that any DLN that implements the optimal linear regressor must have a certain "sharpness".  This amounts to a lower bound on the largest eigenvalue of the Hessian matrix at that set of weights.  They then argue that the weights found by using gradient flow with two different types of initialization find minima that have a sharpness that is within a constant of this lower bound.  They interpret these results as an implicit regularization of gradient flow towards flat minima.

STRENGTHS:
Results are novel.  They are also of at least abstract interest to the community of researches working on the theory of deep learning.  I also agree with their assertion that some of the intermediate results presented on the structure of weight matrices in DLNs post-training (e.g. approx rank 1 and aligned in small scale initialization case) might be of interest.

WEAKNESSES:
The paper kind of has a mixed message, and doesn't really make the connection to generalization power in a way that is intepretable.  Theorem 1 shows that the sharpness has to grow at a rate that is essentially linear in the number of layers.  But then they mention that prior work indicates that flat minima generalize better, which suggests that making the DLN deeper is going to hurt you in this regard.  The subjective interpretation of gradient flow going to flat minima is hard to digest when the authors just proved that there are no flat minima.

QUESTIONS:
Do these results add to our understanding of the generalization power of deep linear networks trained with gradient flow?

LIMITATIONS:
none
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-19