Title: What is the Inductive Bias of Flatness Regularization? A Study of Deep Matrix Factorization Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Sharpness minimization

Deep learning

Matrix factorization

Deep linear networks

Implicit bias

SGD

Trace of Hessian regularizer
Abstract: Recent works on over-parameterized neural networks have shown that  the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its Hessian) over the family zero-loss solutions. More explicit forms of flatness regularization also empirically improve the generalization performance. However, it remains unclear why and when flatness regularization leads to better generalization. 
This work takes the first step towards understanding the inductive bias of the minimum trace of the Hessian solutions in an important setting: learning deep linear networks from linear measurements, also known as \emph{deep matrix factorization}. We show that with the standard Restricted Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters (i.e., the product of all layer matrices), which in turn leads to better generalization.

The Inductive Bias of Flatness Regularization for Deep Matrix Factorization

Abstract

Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its Hessian) over the family zero-loss solutions. More explicit forms of flatness regularization also empirically improve the generalization performance. However, it remains unclear why and when flatness regularization leads to better generalization. This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in an important setting: learning deep linear networks from linear measurements, also known as deep matrix factorization . We show that for all depth greater than one, with the standard Restricted Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters (i.e., the product of all layer matrices), which in turn leads to better generalization. We empirically verify our theoretical findings on synthetic datasets.

1 Introduction

Modern deep neural networks are typically over-parametrized and equipped with huge model capacity, but surprisingly, they generalize well when trained using stochastic gradient descent (SGD) or its variants [51]. A recent line of research suggested the implicit bias of SGD as a possible explanation to this mysterious ability. In particular, Damian et al. [10], Li et al. [29], Arora et al. [4], Lyu et al. [33], Wen et al. [48], Liu et al. [31] have shown that SGD can implicitly minimize the sharpness of the training loss, in particular, the trace of the Hessian of the training loss, to obtain the final model. However, despite the strong empirical evidence on the correlation between various notions of sharpness and generalization [25, 23, 38, 24] and the effectiveness of using sharpness regularization on improving generalization [16, 49, 53, 39], the connection between penalization of the sharpness of training loss and better generalization still remains majorly unclear [13, 2] and has only been proved in the context of two-layer linear models [29, 37, 12]. To further understand this connection beyond the two layer case, we study the inductive bias of penalizing the trace of the Hessian of training loss and its effect on the generalization in an important theoretical deep learning setting: deep linear networks (or equivalently, deep matrix factorization [3]). We start by briefly describing the problem setup.

Deep Matrix Factorization. Consider an L -layer deep network where L ∈ N + , L ≥ 2 is the depth of the model. Let W i ∈ R d i × d i -1 and d i denote the layer weight matrix and width of the i th ( i ∈ [ L ] )

layer respectively. We use W to denote the concatenation of all the parameters ( W 1 , . . . , W L ) and define the end-to-end matrix of W as

In this paper, we focus on models that are linear in the space of the end-to-end matrix E ( W ) . Suppose M ∗ ∈ R d L × d 0 is the target end-to-end matrix, and we observe n linear measurements (matrices) A i ∈ R d L × d 0 and the corresponding labels b i = 〈 A i , M ∗ 〉 . The training loss of W is the mean-squared error (MSE) between the prediction 〈 A i , W L W L -1 · · · W 1 〉 and the observation b i :

Throughout this paper, we assume that d i ≥ min( d 0 , d L ) for each i ∈ [ L ] and, thus, the image of the function E ( · ) is the entire R d L × d 0 . In particular, this ensures that the deep models are sufficiently expressive in the sense that min W L ( W ) = 0 . For this setting, we aim to understand the structure of the trace of the Hessian minimization, as described below. The trace of Hessian is the sum of the eigenvalues of Hessian, which is an indicator of sharpness and it is known that variants of SGD, such as label noise SGD or 1-SAM, are biased toward models with a smaller trace of Hessian [29, 48].

Min Trace of Hessian Interpolating Solution. Our primary object of study is the interpolating solution with the minimum trace of Hessian, defined as:

As we shall see shortly, the solution to the above optimization problem is not unique. We are interested in understanding the underlying structure of any minimizer W ∗ . This will, in turn, inform us about the generalization nature of these solutions.

1.1 Main Results

Before delving into the technical details, we state our main results in this section. This also serves the purpose of highlighting the primary technical contributions of the paper. First, since the generalization of W only depends on its end-to-end matrix E ( W ) , it is informative to derive the properties of E ( W ∗ ) for any min trace of the Hessian interpolating solution W ∗ defined in (3). Indeed, penalizing the trace of Hessian in the W space induces an equivalent penalization in the space of the end-to-end parameters. More concretely, given an end-to-end parameter M , let the induced regularizer F ( M ) denote the minimum trace of Hessian of the training loss at W among all W 's that instantiate the end-to-end matrix M i.e., E ( W ) = M .

Definition 1 (Induced Regularizer) . Suppose M ∈ R d L × d 0 is an end-to-end parameter that fits the training data perfectly (that is, 〈 A i , M 〉 = b i , ∀ i ∈ [ n ] ). We define the induced regularizer as

Since the image of E ( · ) is the entire R d L × d 0 by our assumption that d i ≥ min( d 0 , d L ) , function F is well-defined for all M ∈ R d L × d 0 . It is easy to see that minimizing the trace of the Hessian in the original parameter space (see (3)) is equivalent to penalizing F ( M ) in the end-to-end parameter. Indeed, the minimizers of the implicit regularizer in the end-to-end space are related to the minimizers of the implicit regularizer in the W space, i.e.,

where for any M ∈ R d L × d 0 , we define L ′ ( M ) /defines 1 n ∑ i =1 ( 〈 A i , M 〉 -b i ) 2 and thus L ( W ) = L ′ ( E ( W )) . This directly follows from the definition of F in (4). Our main result characterizes the induced regularizer F ( M ) when the data satisfies the RIP property.

Theorem 1 (Induced regularizer under RIP) . Suppose the linear measurements { A i } n i =1 satisfy the (1 , δ ) -RIP condition.

Let W ∗ ∈ arg min W : L ( W )=0 tr [ ∇ 2 L ( W )] be an interpolating solution with minimal trace of Hessian . Then E ( W ∗ ) roughly minimizes the nuclear norm among all interpolating solutions of L ′ . That is,

However, for more general cases, it is challenging to compute the closed-form expression of F . In this work, we derive closed-form expressions for F in the following two cases: (1) depth L is equal to 2 and (2) there is only one measurement, i.e. , n = 1 (see Table 1). Leveraging the above characterization of induced regularzier, we obtain the following result on the generalization bounds:

Theorem 2 (Recovery of the ground truth under RIP) . Suppose the linear measurements { ( A i ) } n i =1 satisfy the (2 , δ ( n )) -RIP (Definition 3). Then for any W ∗ ∈ arg min W : L ( W )=0 tr [ ∇ 2 L ( W )] , we have

where δ ( n ) depends on the number of measurements n and the distribution of the measurements.

If we further suppose { A i } n i =1 are independently sampled from some distribution over R d L × d 0 satisfying that E A 〈 A,M 〉 2 = ‖ M ‖ 2 F , e.g. , the standard multivariate Gaussian distribution, denoted by G d L × d 0 , we know δ ( n ) = O ( √ d L + d 0 n ) from Candes and Plan [7] (see Section 5.1 for more examples).

Theorem 3. For n ≥ Ω ( r ( d 0 + d L )) , with probability at least 1 -exp( Ω ( d 0 + d L )) over the randomly sampled { A i } n i =1 from multivariate Gaussian distribution G , for any minimum trace of Hessian interpolating solution W ∗ ∈ arg min W : L ( W )=0 tr [ ∇ 2 L ( W )] , the population loss L ( W ∗ ) /defines

E A ∼ G ( 〈 A,E ( W ∗ ) 〉 - 〈 A,M ∗ 〉 ) 2 satisfies that

Next, we state a lower bound for the conventional estimator for overparameterized models that minimizes the norm. The lower bound states that, to achieve a small error, the number of samples should be as large as the product of the dimensions of the end-to-end matrix d 0 d L as opposed to d 0 + d L in case of the min trace of Hessian minimizer. It is proved in Appendix F.

Theorem 4 (Lower bound for /lscript 2 regression) . Suppose { A i } n i =1 are randomly sampled from multivariate Gaussian distribution G , let ˜ W = arg min W : L ( W )=0 ‖ E ( W ) ‖ F to be the minimum Frobenius norm interpolating solution, then the expected population loss is

The lower bound in Theorem 4 shows in order to obtain an O (1) -relatively accurate estimates of the ground truth in expectation, namely to guarantee E L ( ˜ W ) ≤ O (1) ‖ M ∗ ‖ 2 F , the minimum Frobenius norm interpolating solution needs at least Ω ( d 0 d L ) samples. In contrast, the minimizer of trace of Hessian in the same problem only requires O (( d 0 + d L ) ‖ M ∗ ‖ 2 ∗ / ‖ M ∗ ‖ 2 F ) samples, which is at least ˜ O ( min { d 0 ,d L } r ) times smaller. We further illustrate experimentally the superior generalization ability of sharpness minimization algorithms like label noise SGD [6, 10, 29] compared to vanilla mini-bactch SGD Figure 1. Due to the space limits, we defer the full setting for experiments into Appendix A.

2 Related Work

Connection Between Sharpness and Generalization. Research on the connection between generalization and sharpness dates back to Hochreiter and Schmidhuber [21]. Keskar et al. [25] famously observe that when increasing the batch size of SGD, the test error and the sharpness of the learned solution both increase. Jastrzebski et al. [23] extend this observation and found that there is a positive correlation between sharpness and the ratio between learning rate and batch size. Jiang et al. [24] perform a large-scale empirical study on various notions of generalization measures and show that sharpness-based measures correlate with generalization best. Liu et al. [31] find that among language models with the same validation pretraining loss, those that have smaller sharpness can have better downstream performance. On the other hand, Dinh et al. [13] argue that for networks with scaling invariance, there always exist models with good generalization but with arbitrarily large sharpness. We note this does not contradict our main result here, which only asserts the interpolation solution with a minimal trace of Hessian generalizes well, but not vice versa. Empirically, sharpness minimization is also a popular and effective regularization method for overparametrized models [39, 17, 53, 49, 26, 32, 54, 52, 1].

Implicit Bias of Sharpness Minimization. Recent theoretical works [6, 10, 29, 31] show that SGD with label noise is implicitly biased toward local minimizers with a smaller trace of Hessian under the assumption that the minimizers locally connect as a manifold. Such a manifold setting is empirically verified by Draxler et al. [14], Garipov et al. [18] in the sense that the set of minimizers of the training loss is path-connected. It is the same situation for the deep matrix factorization problem studied in this paper, although we do not study the optimization trajectory. Instead, we directly study properties of the minimum trace of Hessian interpolation solution.

Sharpness-reduction implicit bias can also happen for deterministic GD. Arora et al. [4] show that normalized GD implicitly penalizes the largest eigenvalue of the Hessian. Ma et al. [35] argues that such sharpness reduction phenomena can also be caused by a multi-scale loss landscape. Lyu et al. [33] show that GD with weight decay on a scale-invariant loss function implicitly decreases the spherical sharpness, i.e. , the largest eigenvalue of the Hessian evaluated at the normalized parameter. Another line of work focuses on the sharpness minimization effect of a large learning rate in GD, assuming that it converges at the end of training. This has been studied mainly through linear stability analysis [50, 8, 34, 9]. Recent theoretical analysis [11, 30] showed that the sharpness minimization effect of a large learning rate in GD does not necessarily rely on convergence and linear stability, through a four-phase characterization of the dynamics at the Edge of Stability regime [8].

3 Preliminaries

Notation. We use [ n ] to denote { 1 , 2 , . . . , n } for every n ∈ N . We use ‖ M ‖ F , ‖ M ‖ ∗ , ‖ M ‖ 2 and tr ( M ) to denote the Frobenius norm, nuclear norm, spectral norm and trace of matrix M respectively. For any function f defined over set S such that min x ∈ S f ( x ) exists, we use arg min S f to denote the set { y ∈ S | f ( y ) = min x ∈ S f ( x ) } . Given a matrix M , we use h M to denote the linear map A ↦→〈 A,M 〉 . We use H r to to denote the set H r /defines { h M | ‖ M ‖ ∗ ≤ r } . M i : and M : j are used to denote the i th row and j th column of the matrix M .

The following definitions will be important to the technical discussion in the paper.

Rademacher Complexity. Given n data points { A i } n i =1 , the empirical Rademacher complexity of function class H is defined as

Given a distribution P , the population Rademacher complexity is defined as follows: R n ( H ) = E A i iid ∼ P R n ( H ) . This is mainly used to upper bound the generalization gap of SGD.

Definition 2 (Schatten p -(quasi)norm) . Given any d, d ′ ∈ N + , p ∈ (0 , ∞ ) a matrix M ∈ R d × d ′ with singular values σ 1 ( M ) , . . . , σ min( d,d ′ ) ( M ) , we define the Schattern p -(semi)norm as

Note that in this definition ‖ · ‖ S p is a norm only when p ≥ 1 . When p ∈ (0 , 1) , the triangle inequality does not hold. Note that when p ∈ (0 , 1) , ‖ A + B ‖ S p ≤ 2 1 / p -1 ( ‖ A ‖ S p + ‖ B ‖ S p ) for any matrices A and B , however, 2 1 / p -1 > 1 .

We use L to denote the depth of the linear model and W = ( W 1 , . . . , W L ) to denote the parameters, where W i ∈ R d i × d i -1 . We assume that d i ≥ min( d 0 , d L ) for each i ∈ [ L -1] and, thus, the image of E ( W ) is the entire R d L × d 0 . Following is a simple relationship between nuclear norm and Frobenius norm that is used frequently in the paper.

Lemma 1. For any matrices A and B , it holds that ‖ AB ‖ ∗ ≤ ‖ A ‖ F ‖ B ‖ F .

4 Exact Formulation of Induced Regularizer by Trace of Hessian

In this section, we derive the exact formulation of trace of Hessian for /lscript 2 loss over deep matrix factorization models with linear measurements as a minimization problem over W . We shall later approximate this formula by a different function in Section 5, which allows us to calculate the implicit bias in closed-form in the space of end-to-end matrices.

We first introduce the following simple lemma showing that the trace of the Hessian of the loss is equal to the sum of squares of norms of the gradients of the neural network output.

Lemma 2. For any twice-differentiable function { f i ( W ) } n i =1 , real-valued labels { b i } n i =1 , loss function L ( W ) = 1 n ∑ n i =1 ( f i ( W ) -b i ) 2 , and any W satisfying L ( W ) = 0 , it holds that

Using Lemma 2, we calculate the trace of Hessian for the particular loss defined in (2). To do this, we consider W in Lemma 2 to be the concatenation of matrices ( W 1 , . . . , W L ) and we set f i ( W ) to be the linear measurement 〈 A i , E ( W ) 〉 , where E ( W ) = W L · · · W 1 (see (1)). To calculate the trace of Hessian, according to Lemma 2, we need to calculate the gradient of L ( W ) in (2). To this end, for a fixed i , we compute the gradient of 〈 A i , E ( W ) 〉 with respect to one of the weight matrices W j .

According to Lemma 2, trace of Hessian is given by

As mentioned earlier, our approach is to characterize the minimizer of the trace of Hessian among all interpolating solutions by its induced regularizer in the end-to-end matrix space. The above

calculation provides the following more tractable characterization of induced regularizer F in (12):

In general, we cannot solve F in closed form for general linear measurements { A i } n i =1 ; however, interestingly, we show that it can be solved approximately under reasonable assumption on the measurements. In particular, we show that the induced regularizer, as defined in (7), will be approximately proportional to a power of the nuclear norm of E ( W ) given that the measurements { A i } n i =1 satisfy a natural norm-preserving property known as the Restricted Isometry Property (RIP) [7, 42].

Before diving into the proof of the general result for RIP, we first illustrate the connection between nuclear norm and the induced regularizer for the depth-two case. In this case, fortunately, we can compute the closed form of the induced regularizer. This result was first proved by Ding et al. [12]. For self-completeness, we also provide a short proof.

Theorem 5 (Ding et al. [12]) . For any M ∈ R d L × d 0 , it holds that

Proof of Theorem 5. We first define B 1 = ( ∑ n i =1 A i A i T ) 1 2 and B 2 = ( ∑ n i =1 A i T A i ) 1 2 . Therefore we have that

Further applying Lemma 1, we have that

Next we show this lower bound of F ( M ) can be attained. Let U Λ V T be the SVD of B 1 MB 2 . The equality condition happens for W ∗ 2 = B 1 † U Λ 1 / 2 , W ∗ 1 = Λ 1 / 2 V T B 2 † , where we have that ∑ n i =1 ‖ A i T W ∗ 2 ‖ 2 F + ‖ W ∗ 1 A i T ‖ 2 F = 2 ‖ Λ ‖ 2 F = 2 ‖ B 1 MB 2 ‖ 2 F . This completes the proof.

The right-hand side in (8) will be very close to the nuclear norm of M if the two extra multiplicative terms are close to the identity matrix. It turns out that { A i } n i =1 satisfying the (1 , δ ) -RIP exactly guarantees the two extra terms are O ( δ ) -close to identity. However, the case for deep networks where depth is larger than two is fundamentally different from the two-layer case, where one can obtain a closed form for F . To the best of our knowledge, it is open whether one obtain a closed form for the induced-regularizer for the trace of Hessian when L > 2 . Nonetheless, in Section 5.1, we show that under RIP, we can still approximate it with the nuclear norm.

5 Results for Measurements with Restricted Isometry Property (RIP)

In this section, we present our main results for the generalization benefit of flatness regularization in deep linear networks. We structure the analysis as follows:

In Section 5.1, we first recap some preliminaries on the RIP property.

In Section 5.2, we prove that the induced regularizer by trace of Hessian is approximately the power of nuclear norm for (1 , δ ) -RIP measurements (Theorem 1).

In Section 5.3, we prove that the minimum trace of Hessian interpolating solution with (2 , δ ) -RIP measurements can recover the ground truth M ∗ up to error δ ‖ M ∗ ‖ 2 ∗ . For { A i } n i =1 sampled from Gaussian distributions, we know δ = O ( √ d 0 + d L n ) .

In Section 5.4, we prove a generalization bound with faster rate of d 0 + d L n ‖ M ∗ ‖ 2 ∗ using local Rademacher complexity based techniques from Srebro et al. [44].

Next, we discuss important distributions of measurements for which the RIP property holds.

5.1 Preliminaries for RIP

Definition 3 (Restricted Isometry Property (RIP)) . A family of matrices { A i } n i =1 satisfies the ( r, δ ) -RIP iff for any matrix X with the same dimension and rank at most r :

Next, we give two examples of distributions where Ω ( r ( d 0 + d L )) samples guarantee ( r, O (1)) -RIP. The proofs follow from Theorem 2.3 in [7].

Example 1. Suppose for every i ∈ { 1 , . . . , n } , each entry in the matrix A i is an independent standard Gaussian random variable, i.e. , A i i.i.d. ∼ G d L × d 0 . For every constant δ ∈ (0 , 1) , if n ≥ Ω ( r ( d 0 + d L )) , then with probability 1 -e Ω ( n ) , { A i } n i =1 satisfies ( r, δ ) -RIP.

Example 2. If each entry of A i is from a symmetric Bernoulli random variable with variance 1 , i.e. for all i, k, /lscript , entry [ A i ] k, /lscript is either equal to 1 or -1 with equal probabilities, then for any r and δ , ( r, δ ) -RIP holds with same probability as in Example 1 if the same condition there is satisfied.

5.2 Induced Regularizer of Trace of Hessian is Approximately Nuclear Norm

This section focuses primarily on the proof of Theorem 2. Our proof consists of two steps: (1) we show that the trace of Hessian of training loss at the minimizer W is multiplicatively O ( δ ) -close to the regularizer R ( W ) defined below (Lemma 3) and (2) we show that the induced regularizer of R , F ′ ( M ) , is proportional to ‖ M ‖ 2( L -1) /L ∗ (Lemma 4).

Lemma 3. Suppose the linear measurement { A i } n i =1 satisfy (1 , δ ) -RIP. Then, for any W such that L ( W ) = 0 , it holds that

Since tr ( ∇ 2 L )( W ) closely approximates R ( W ) , we can study R instead of tr [ ∇ 2 L ] to understand the implicit bias up to a multiplicative factor (1 + δ ) . In particular, we want to solve the induced regularizer of R ( W ) on the space of end-to-end matrices, F ′ ( M ) :

Surprisingly, we can solve this problem in closed form.

Lemma 4. For any M ∈ R d L × d 0 , it holds that

Proof of Lemma 4. Applying the L -version of the AM-GM to Equation (10):

Now using Lemma 1, we have for every 1 ≤ j ≤ L -1 :

Multiplying Equation (14) for all 1 ≤ j ≤ L -1 and combining with Equation (13) implies

Now we show that equality can indeed be attained. To construct an example in which the equality happens, consider the singular value decomposition of M : M = U Λ V T , where Λ is a square matrix with dimension rank( M ) .

For 1 ≤ i ≤ L , we pick Q i ∈ R d i × rank( M ) to be any matrix with orthonormal columns. Note that rank( M ) is not larger than d i for all 1 ≤ i ≤ L , hence such orthonormal matrices Q i exist. Then we define the following with α , α ′ > 0 being constants to be determined:

Note that Λ is a square matrix with dimension rank( M ) . First of all, note that the defined matrices satisfy

To gain some intuition, we check that the equality case for all the inequalities that we applied above. We set the value of α in a way that these equality cases can hold simultaneously. Note that for the matrix holder inequality that we applied in Equation (14):

independent of the choice of α . It remains to check the equality case for the AM-GM inequality that we applied in Equation (13). We have for all 2 ≤ j ≤ L -1 :

Hence, equality happens for all of them. Moreover, for cases j = 1 and j = L , we have

Thus it suffices to set α ′ = ( d L d 0 ) 1 / 2 and α = ( ‖ Λ 1 / 2 ‖ F √ d 0 d L ) 2 /L = ( ‖ M ‖ ∗ d 0 d L ) 1 /L so that the left-hand sides of (16), (17), and (18) are equal, which implies that the lower bound in Equation (15) is actually an equality. The proof is complete.

Now we can prove Theorem 1 as an implication of Lemma 4.

Proof of Theorem 1. The first claim is a corollary of Lemma 3. We note that

For the second claim, pick ¯ W that minimizes R ( ¯ W ) over all W 's that satisfy the linear measurements, thus we have that

Now from the definition of E ( W ∗ ) ,

where the last inequality follows from the definition of W . On the other hand

Combining (19), (20) and (21),

The proof is completed by noting that L 1 for all L 2 .

2( L -1) ≤ ≥

Thus combining Example 1 and Theorem 1 with δ = 1 / 2 , we have the following corollary.

Corollary 1. Let { A i } n i =1 be sampled independently from Gaussian distribution G d L × d 0 where n ≥ Ω (( d 0 + d L )) , with probability at least 1 -exp( Ω ( n )) , we have

5.3 Recovering the Ground truth

In this section, we prove Theorem 2. The idea is to show that under RIP, the empirical loss L ( W ) is a good approximation for the Frobenius distance of E ( W ) to the ground truth M ∗ . To this end, we first introduce a very useful Lemma 5 below, whose proof is deferred to Appendix E.

Lemma 5. Suppose the measurements { A i } n i =1 satisfy the (2 , δ ) -RIP condition. Then for any matrix M ∈ R d L × d 0 , we have that

We note that if { A i } n i =1 are i.i.d. random matrices with each coordinate being independent, zero mean, and unit variance (like standard Gaussian distribution), then ‖ W -M ∗ ‖ 2 F is the population squared loss corresponding to W . Thus, Theorem 2 implies a generalization bound for this case. Now we are ready to prove Theorem 2.

Proof of Theorem 2. Note that from Theorem 1,

which implies the following by triangle inequality,

Combining (22) with Lemma 5 (with M = E ( W ∗ ) -M ∗ ):

Since W ∗ satisfies the linear constraints tr ( A i E ( W ∗ )) = b i , 1 n ∑ n i =1 〈 A i , E ( W ∗ ) -M ∗ 〉 2 = 1 n ∑ n i =1 ( 〈 A i , E ( W ∗ ) 〉 -b i ) 2 = 0 , which completes the proof.

5.4 Generalization Bound

In this section, we prove the generalization bound in Theorem 3, which yields a faster rate of O ( d 0 + d L n ‖ M ∗ ‖ 2 ∗ ) compared to O ( √ d 0 + d L n ‖ M ∗ ‖ 2 ∗ ) in Theorem 2. The intuition for this is as follows: By Corollary 1, we know that with very high probability, the learned solution has a bounded nuclear norm for its end-to-end matrix, no larger than 3 ‖ M ∗ ‖ 2 , where M ∗ is the ground truth. The key mathematical tool is Theorem 6, which provides an upper bound on the population error of the learned interpolation solution that is proportional to the square of the Rademacher complexity of the function class H 3 ‖ M ∗ ‖ ∗ = { h M | ‖ M ‖ ∗ ≤ 3 ‖ M ∗ ‖ ∗ } .

Theorem 6 (Theorem 1, Srebro et al. [44]) . Let H be a class of real-valued functions and /lscript : R × R → R be a differentiable non-negative loss function satisfying that (1) for any fixed y ∈ R , the partial derivative /lscript ( · , y ) with respect to its first coordinate is H -Lipschitz and (2) | sup x,y /lscript ( x, y ) | ≤ B , where H,B are some positive constants. Then for any p > 0 , we have that with probability at least 1 -p over a random sample of size n , for any h ∈ H with zero training loss,

One technical difficulty is that Theorem 6 only works for bounded loss functions, but the /lscript 2 loss on Gaussian data is unbounded. To circumvent this issue, we construct a smoothly truncated variant of /lscript 2 loss (41) and apply Theorem 6 on that. Finally, we show that with a carefully chosen threshold, this truncation happens very rarely and, thus, does not change the population loss significantly. The proof can be found in Appendix E.

L=2

L=3

L=5

L=10

SS07

10-4,

10-2

100

102

103

105,

101

10

10-2

10-4

102

100

10-4

102

100

102

10-6,

10-7

106

10-6,

10-8

10-9

10-8

10-8

10-10

10-11

1010

10-10

10-12

10-13

10-12

-01

100

101

102

103104

Iteration

105106

100

101

102

103104

Iteration

105106

100

101

102

103

Iteration

104

105

106

100

101

102

Iteration

103104

105106

103

102

101

10

10

102

10

100

100

100

100

10-1

10-1

10-

10-1

100

101102

103104

Iteration

105

106

100

101102

103104

Iteration

105106

100

101

102

103104

Iteration

105106

100

101

102

103104

Iteration

10106

6 Result for the Single Measurement Case

Quite surprisingly, even though in the general case we cannot compute the closed-form of the induced regularizer in (12), we can find its minimum as a quasinorm function of the E ( W ) which only depends on the singular values of E ( W ) . This yields the following result for multiple layers L (possibly L > 2 ) with a single measurement.

Theorem 7. Suppose there is only a single measurement matrix A , i.e. , n = 1 . For any M ∈ R d L × d 0 , the following holds:

To better illustrate the behavior of this induced regularizer, consider the case where the measurement matrix A is identity and M is symmetric with eigenvalues { σ i } d i =1 . Then, it is easy to see that F ( M ) in (24) is equal to F ( M ) = ∑ i σ 2( L -1) /L i . Interestingly, we see that the value of F ( M ) converges to the Frobenius norm of M and not the nuclear norm as L becomes large, which behaves quite differently (e.g. in the context of sparse recovery). This means that beyond RIP, the induced regularizer can behave very differently, and perhaps the success of training deep networks with SGD is closely tied to the properties of the dataset.

7 Conclusion and Future Directions

In this paper, we study the inductive bias of the minimum trace of the Hessian solutions for learning deep linear networks from linear measurements. We show that trace of Hessian regularization of loss on the end-to-end matrix of deep linear networks roughly corresponds to nuclear norm regularization under restricted isometry property (RIP) and yields a way to recover the ground truth matrix. Furthermore, leveraging this connection with the nuclear norm regularization, we show a generalization bound which yields a faster rate than Frobenius (or /lscript 2 norm) regularizer for Gaussian distributions. Finally, going beyond RIP conditions, we obtain closed-form solutions for the case of a single measurement. Several avenues for future work remain open, e.g., more general characterization of trace of Hessian regularization beyond RIP settings and understanding it for neural networks with non-linear activations.

References

Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware minimization. In International Conference on Machine Learning , pages 639-668. PMLR, 2022.

Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.07011 , 2023.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems , pages 7411-7422, 2019.

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on edge of stability in deep learning. arXiv preprint arXiv:2205.09745 , 2022.

Mohamed Ali Belabbas. On implicit regularization: Morse functions and applications to matrix factorization. arXiv preprint arXiv:2001.04264 , 2020.

Guy Blanc, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an ornstein-uhlenbeck like process. arXiv preprint arXiv:1904.09080 , 2019.

Emmanuel J Candes and Yaniv Plan. Tight oracle inequalities for low-rank matrix recovery from a minimal number of noisy random measurements. IEEE Transactions on Information Theory , 57(4):2342-2359, 2011.

Jeremy M. Cohen, Simran Kaur, Yuanzhi Li, J. Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability, 2021.

Jeremy M Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, David Cardoze, Zachary Nado, George E Dahl, et al. Adaptive gradient methods at the edge of stability. arXiv preprint arXiv:2207.14484 , 2022.

Alex Damian, Tengyu Ma, and Jason Lee. Label noise sgd provably prefers flat global minimizers, 2021.

Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. arXiv preprint arXiv:2209.15594 , 2022.

Lijun Ding, Dmitriy Drusvyatskiy, and Maryam Fazel. Flat minima generalize for low-rank matrix recovery. arXiv preprint arXiv:2203.03756 , 2022.

Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1019-1028. JMLR. org, 2017.

Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning , pages 1309-1318. PMLR, 2018.

Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint arXiv:1703.11008 , 2017.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412 , 2020.

Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations , 2021.

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information processing systems , 31, 2018.

Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely. The implicit bias of depth: How incremental learning drives generalization. arXiv preprint arXiv:1909.12051 , 2019.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems , pages 6151-6159, 2017.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation , 9(1):1-42, 1997.

Arthur Jacot, François Ged, Franck Gabriel, Berfin Simsek, and Clément Hongler. Deep linear networks dynamics: Low-rank biases induced by initialization scale and l2 regularization. arXiv preprint arXiv:2106.15933 , 3, 2021.

Stanis ł aw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623 , 2017.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv:1912.02178 , 2019.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 , 2016.

Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpnessaware minimization for scale-invariant learning of deep neural networks. In International Conference on Machine Learning , pages 5905-5914. PMLR, 2021.

Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. arXiv preprint arXiv:1712.09203 , pages 2-47, 2017.

Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. arXiv preprint arXiv:2012.09839 , 2020.

Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a mathematical framework. In International Conference on Learning Representations , 2021.

Zhouzi Li, Zixuan Wang, and Jian Li. Analyzing sharpness along gd trajectory: Progressive sharpening and edge of stability. arXiv preprint arXiv:2207.12678 , 2022.

Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same pre-training loss, better downstream: Implicit bias matters for language models. 2022.

Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12360-12370, 2022.

Kaifeng Lyu, Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normalization layers: Sharpness reduction. arXiv preprint arXiv:2206.07085 , 2022.

Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems , 34:16805-16817, 2021.

Chao Ma, Lei Wu, and Lexing Ying. The multiscale structure of neural network loss functions: The effect on optimization and origin. arXiv preprint arXiv:2204.11326 , 2022.

Cong Ma, Kaizheng Wang, Yuejie Chi, and Yuxin Chen. Implicit regularization in nonconvex statistical estimation: Gradient descent converges linearly for phase retrieval and matrix completion. In International Conference on Machine Learning , pages 3345-3354. PMLR, 2018.

Mor Shpigel Nacson, Kavya Ravichandran, Nathan Srebro, and Daniel Soudry. Implicit bias of the step size in linear diagonal neural networks. In International Conference on Machine Learning , pages 16270-16295. PMLR, 2022.

Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems , pages 5947-5956, 2017.

Matthew D Norton and Johannes O Royset. Diametrical risk minimization: Theory and computations. Machine Learning , pages 1-19, 2021.

Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. arXiv preprint arXiv:2005.06398 , 2020.

Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In International Conference on Machine Learning , pages 8913-8924. PMLR, 2021.

Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review , 52(3):471-501, 2010.

Mark Rudelson and Roman Vershynin. Non-asymptotic theory of random matrices: extreme singular values. In Proceedings of the International Congress of Mathematicians 2010 (ICM 2010) (In 4 Volumes) Vol. I: Plenary Lectures and Ceremonies Vols. II-IV: Invited Lectures , pages 1576-1602. World Scientific, 2010.

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. Advances in neural information processing systems , 23, 2010.

Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. Advances in Neural Information Processing Systems , 34:23831-23843, 2021.

Colin Wei and Tengyu Ma. Data-dependent sample complexity of deep neural networks via lipschitz augmentation. In Advances in Neural Information Processing Systems , pages 9722-9733, 2019.

Colin Wei and Tengyu Ma. Improved sample complexities for deep networks and robust classification via an all-layer margin. arXiv preprint arXiv:1910.04284 , 2019.

Kaiyue Wen, Tengyu Ma, and Zhiyuan Li. How does sharpness-aware minimization minimize sharpness? arXiv preprint arXiv:2211.05729 , 2022.

Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems , 33:2958-2969, 2020.

Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems , 31, 2018.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR) , 2017.

Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. arXiv preprint arXiv:2202.03599 , 2022.

Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8156-8165, 2021.

Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065 , 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the minimizers of the loss surface of deep matrix factorization that have a minimal trace of the Hessian. The trace of the Hessian is a measure of flatness of the minimum, that is favored by SGD.\n\nThe authors show that for matrix sensing with observations that satisfy RIP (which is in particular true for enough Gaussian observations) the parameters that are the flatest have end-to-end matrix that approximately minimize the nuclear norm amongst fitting matrices. This in turn implies generalization guarantees that are significantly faster than the if one were to minimize the Frobenius norm instead of the nuclear norm.\n\nSTRENGTHS:\nThe description of the flatness bias given by this paper is to my knowledge new, and some of the results are a bit unexpected.\n\nThe writing is clear and thorough and the proofs are easy to understand.\n\nWEAKNESSES:\nThe results hangs on the fact that the trace of the Hessian is the right notion of flatness for SGD, which is to my knowledge not decided yet. For GD the relevant measure of flatness is instead the largest eigenvalue of the Hessian, which will probably lead to a quite different implicit bias. And since SGD can in some cases be quite similar to GD, it seems unlikely that the bias would simply be described by the trace of Hessian, instead this exact bias could change depending on the learning rate and batch size.\n\nThe lack of empirical experiments to test whether the bias described here is observed for SGD, label noise SGD or 1-SAM would help motivate this assumption.\n\nQUESTIONS:\nA shallow linear network with L2 regularization is also known to recover the minimal nuclear norm solution [Dai et al., Representation Costs of Linear Neural Networks: Analysis and Design]. Do you think that there is any advantage to rely on the flatness bias instead of the L2 regularization?\n\nLIMITATIONS:\nThe biggest limitation is clearly the RIP condition, but it is well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the context of matrix sensing with deep matrix factorizations, the paper analyzes the inductive bias of interpolators with minimal Hessian trace, which is a well-known measure of sharpness. Specifically, under a Restricted Isometry Property (RIP) assumption on the linear measurements, it establishes that the minimal Hessian trace with which the deep matrix factorization can express an interpolator is approximately equal to the latter’s nuclear norm. In turn, this yields guarantees on the recovery of the ground truth matrix, as well as a separation result from the recovery obtained by the minimal Frobenius norm interpolator.\n\nAs an additional contribution, develops a closed form expression for the minimal Hessian trace of an interpolator when there is only a single linear measurement.\n\nSTRENGTHS:\n1. Well-written and easy to follow. The motivation, problem at hand, and main results are clearly described.\n\n2. Helpful discussions are provided for each of the technical results, in particular regarding their implications and relation to existing work.\n\n3. The technical contributions establish a new connection between flatness, in terms of minimal Hessian trace, and generalization for deep matrix factorizations (previous work of Ding et al. 2022 studies only depth two factorizations). Despite conventional wisdom and empirical evidence suggesting that flatness may lead to good generalization, formally showing that this is the case has proven challenging. This attests to the significance of the current paper’s results.\nPersonally, I found it interesting that depth does not help in the analyzed setting, in the sense that minimizing the Hessian trace amounts to approximately minimizing the nuclear norm, similar to the depth L = 2 case. This may suggest that deep matrix factorization with RIP measurements is an unsatisfactory setting for uncovering the benefits of depth in deep learning and its (possible) relation with flatness.\n\nWEAKNESSES:\nThe points below pertain to the scope of the technical contributions, suggesting places for improvement. I do not believe that points (2) and (3) significantly harm the quality of the current paper and are perhaps considerations for future work. For (1), it seems that it may be straightforward to incorporate, in which case I believe it can increase the generality of the current work.\n\n1. Only interpolators with exactly minimal Hessian trace are considered. Since in practice a more realistic view is that it is only approximately minimized, if possible, it is worth (even if in an appendix) extending the generalization results to interpolators whose Hessian trace is approximately minimal.\n\n2. The current paper characterizes the regularizer induced by minimizing the Hessian trace only for interpolators. Since using explicit regularization can lead to solutions that don't entirely minimize the loss to zero, I believe it is important to understand the inductive bias of minimizing the Hessian trace for non-interpolating linear mappings as well. While more complex, it may be more realistic.\n\n3. The current paper only treats the question of what interpolating the data with minimal Hessian trace implies, and not the complementary question of whether it, or other measures of sharpness more generally, are implicitly minimized in deep matrix factorization by standard optimizers.\n\nAdditional (minor) comments and typos:\n- Typo in line 75: “regularzier” should be “regularizer”.\n- Typo in line 101: I believe “observe” should have been “observed”.\n- Typo in equation after line 166: I believe there is a missing gradient symbol in the middle expression and a factor of two in both the middle and right expressions.\n\nQUESTIONS:\nHave you considered other notions of sharpness besides the trace of the Hessian, e.g. its the maximal eigenvalue? In particular, can minimizing the maximal eigenvalue of the Hessian ensure generalization in the considered setting, or is it too weak and one must look at the trace of the Hessian?\n\nLIMITATIONS:\nThe authors have adequately addressed limitations of the work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript seeks to understand Hessian trace regularization in the case of deep linear network training with the mean squared error of linear measurements. It obtains a description of the effective regulariser which can be approximated and made more explicit in some cases. The manuscript obtains results on matrix recovery and generalization.\n\nSTRENGTHS:\n* The problem under investigation, to characterize solutions with minimum Hessian trace, is interesting. \n* The work obtains results indicating that the number of measurements needed to obtain a good approximation of a target matrix is smaller when using trace regularization as opposed to l2 regularization of the end to end matrix.\n\nWEAKNESSES:\n* The presentation of the results does not make the assumptions sufficiently clear in a timely manner. \n* The settings are restrictive. The networks are assumed to have no bottlenecks, so that the set of representable end to end matrices have no constraints. The considered objective function is a sum of squared errors of linear measurements, whereby it is assumed that the measurements satisfy an RIP property, or that the network has only two layers, or that there is only one measurement. \n* The theoretical results could have been strengthened by numerical experiments, particularly given the restricted conditions for which the theoretical results are obtained.\n\nQUESTIONS:\n* In Theorem 1 it is indicated that the data should satisfy an RIP assumption, but this condition is not described explicitly. The (1,\\delta)-RIP condition should be described explicitly before or at the time of stating the theorem. \n* After Theorem 1 it is stated that in more general cases it is challenging to compute F, but it is not clearly stated what is meant by more general cases.\n* In Theorem 2, what is the dependence on n? \n* In Theorem 3, what is r? This seems to be introduced 3 pages later in Definition 3. \n* Theorem 4 appears to consider regularization of the end to end matrix. What would be the result if one instead minimises the l2 norm of the parameters (factor matrices)? \n* There are typos in the display equation following line 166. \n* In Theorem 5 the argument of the Hessian of the Loss should be the factor matrices? \n* Example 1, missing negative sign in exponent? \n* Please add an explanation for the second inequality in (13).  \n* In proof of Theorem 1, argument is W_1,\\ldots, W_L instead of M?\n\nLIMITATIONS:\n* The work considers deep linear networks with no bottlenecks, in which case the image function space contains all matrices. \n* It would be interesting to combine and compare the discussion of trace of Hessian minimisation with l2 regularization of the parameters.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work considers the implicit regularization of deep neural networks with linear activations and linear data. While previous works have shown that stochasticity in optimizers has the effect of smoothing the loss function, this work derives how this less sharp loss function can result in better generalization performance under certain assumptions. Specifically, the authors derive the induced regularizer in three cases: (1) two-layer linear networks, (2) networks learned with a single example, and (3) data that satisfies the restricted isometry property. Then, in these settings, generalization bounds are derived.\n\nSTRENGTHS:\n* I found this work to be very well presented and clear. In general, I found the presentation made the content quite accessible for a reader not familiar with the literature. I appreciate the author's effort to ensure the content was as self-contained as possible with all required background knowledge neatly included. \n* I found Sec. 1.1 summarizing the main results of the work to be extremely helpful before proceeding into the details later. \n* Notation was well-considered, clear, and consistent.\n* The related work section was extensive between the main text and the appendix. Although I am not familiar with this literature, this section provided a helpful summary (although I cannot comment on its accuracy). \n* The claims made in the abstract are an accurate reflection of the paper's contributions. \n* Some synthetic numerical experiments are provided to validate the theoretical findings.\n\nWEAKNESSES:\n* My single major issue with this work is with respect to its practical significance. Almost the entire contribution of this work lies in the theoretical results derived within. However, all of these results are derived under extremely restrictive assumptions/conditions. My understanding is that they assume (at a minimum):\n   - A neural network with linear activations. \n   - The regression setting (I believe results would not hold under e.g. cross-entropy loss?) \n   - The relationship between the observations and their ground truth targets is linear. \n   - No hidden layer within the network has fewer hidden neurons than either the input or output layer. This would rule out any problem with a single label per observation. \n\n   Then, the results are derived within three even more restricted settings: (1) the network has exactly two layers (2) the network is learned upon a dataset consisting of a single example or (3) the data satisfies the RIP property. Some of the later theorems require even more assumptions such as $iid$ observations satisfying $\\mathbb{E}_A \\braket{A,M}^2 = ||M||^2_F$. I believe it is uncontroversial to say these results cannot be applied to any practical task. Therefore, I would challenge the authors to make the case as to how this work contributes itself or could have some downstream contribution to real-world problems/tasks. While this conference is a suitable venue for a theory-style paper such as this, I would argue that the choice of whether to accept such a paper should be heavily weighted by its potential practical impact (otherwise we can always invent interesting, but purely fictitious problems). \n\nMinor issues/clarifications:\n* While I realize that the experiment was straightforward, it is always nice to include the code used to generate the results. Could this be added to a camera-ready version? \n* On L22+23 the paper states that \"despite the overwhelming empirical evidence on ... the effectiveness of using sharpness regularization on improving generalization [15, 47, 51, 37], the connection between penalization of the sharpness of training loss and better generalization still remains unclear.\" Could the authors clarify how these are not the same thing? I suspect the point is clear to the authors but could just be worded more clearly. \n* On L169: is this intended to refer to eqn (12)? \n* Should the experiments in Fig 2 not also include a baseline of a model trained without label noise?\n\nQUESTIONS:\n* I found the terminology of \"end-to-end parameters\" (e.g. L52+53) to be slightly non-intuitive for its intended meaning. Possibly because in other parts of the machine learning literature this terminology refers to a different concept (models that are trained jointly in a non-modular way). Is this standard terminology in this literature? If not, it might be helpful to use an alternative or make this more clear.\n\nLIMITATIONS:\nYes. I cannot foresee any potential negative societal impact of this work", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-16"}