Title: Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Generalization bound

minima stability

gradient descent

large learning rate

ReLU neural network
Abstract: We study the generalization of two-layer ReLU neural networks in a univariate nonparametric regression problem with noisy labels. This is a problem where kernels (\emph{e.g.} NTK) are provably sub-optimal and benign overfitting does not happen, thus disqualifying existing theory for interpolating (0-loss, global optimal) solutions. We present a new theory of generalization for local minima that gradient descent with a constant learning rate can \emph{stably} converge to.  We show that gradient descent with a fixed learning rate $\eta$ can only find local minima that represent smooth functions with a certain weighted \emph{first order total variation} bounded by $1/\eta - 1/2 + \widetilde{O}(\sigma + \sqrt{\mathrm{MSE}})$ where $\sigma$ is the label noise level, $\mathrm{MSE}$ is short for mean squared error against the ground truth, and $\widetilde{O}(\cdot)$ hides a logarithmic factor. Under mild assumptions, we also prove a nearly-optimal MSE bound of $\widetilde{O}(n^{-4/5})$  within the strict interior of the support of the $n$ data points. Our theoretical results are validated by extensive simulation that demonstrates large learning rate training induces sparse linear spline fits. To the best of our knowledge, we are the first to obtain generalization bound via minima stability in the non-interpolation case and the first to show ReLU NNs without regularization can achieve near-optimal rates in nonparametric regression.

Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes

Abstract

We study the generalization of two-layer ReLU neural networks in a univariate nonparametric regression problem with noisy labels. This is a problem where kernels ( e.g. NTK) are provably sub-optimal and benign overfitting does not happen, thus disqualifying existing theory for interpolating (0-loss, global optimal) solutions. We present a new theory of generalization for local minima that gradient descent with a constant learning rate can stably converge to. We show that gradient descent with a fixed learning rate η can only find local minima that represent smooth functions with a certain weighted first order total variation bounded by 1 /η -1 / 2 + ˜ O ( σ + √ MSE) where σ is the label noise level, MSE is short for mean squared error against the ground truth, and ˜ O ( · ) hides a logarithmic factor. Under mild assumptions, we also prove a nearly-optimal MSE bound of ˜ O ( n -4 / 5 ) within the strict interior of the support of the n data points. Our theoretical results are validated by extensive simulation that demonstrates large learning rate training induces sparse linear spline fits. To the best of our knowledge, we are the first to obtain generalization bound via minima stability in the non-interpolation case and the first to show ReLU NNs without regularization can achieve near-optimal rates in nonparametric regression.

1 Introduction

How do gradient descent-trained neural networks work? It is an intriguing question that depends on model architecture, data distribution, and optimization algorithms used for training [Zhang et al., 2021]. Specifically, in the overparameterized regime with specific random initialization of the weights, it was shown that gradient descent finds global optimal (0-loss or interpolating) solutions despite the non-convex objective function [Jacot et al., 2018, Du et al., 2018, Liu et al., 2022]. It was also shown that among the (many) global optimal solutions, the particular solutions that are selected by gradient descent often do not overfit despite having 0 training error [Chizat et al., 2019, Arora et al., 2019, Mei et al., 2019], sometimes even if the data is noisy - a phenomenon known as 'benign overfitting' [e.g., Belkin et al., 2019, Bartlett et al., 2020, Frei et al., 2022].

Interpolating

Solutions

(when labels

are  noisy)

All functions representable

by Two-Layer ReLU NN

GD with small

𝜂

GD with

large

𝜂

Benign

overfitting

'Edge of Stability'

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

GD-trained ReLU NN with step size

True Function

=0.4

=0.01

10 1

10 2

1/

10 0

10 1

10 2

Our upper bound on

Hessian

)

1

2

max(Hessian)

1/

1/2 +

+

MSE

1st order Total Variation

10 1

10 2

1/

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Loss and MSE of different

s

Training Loss(

f

) =

1

n

n

i

=1

(

f

(

xi

)

yi

)

2

MSE(

f

) =

1

n

n

i

=1

(

f

(

xi

)

f

0(

xi

))

2

What is less well-known is that interpolating solutions do overfit for ReLU neural networks (ranging from tempered to catastrophic) [Mallinar et al., 2022, Joshi et al., 2023, Haas et al., 2023] and the generalization bounds in the kernel regime are provably suboptimal for certain univariate nonparametric regression problems [Suzuki, 2018, Zhang and Wang, 2022]. These 'exceptions' significantly limit the abilities of the kernel theory or 'benign overfitting' theory in predicting the performance of an overparameterized neural network in practice.

In many learning problems with noisy data, the best solutions are simply not among those that interpolate the data. For example, no interpolating solutions can be consistent in a fixed-design nonparametric regression problem. Even if interpolating solutions that satisfy benign overfitting can be found, they could be undesirable due to their 'spikiness' [Haas et al., 2023] and lack of robustness [Hao and Zhang, 2024]. In addition, it was reported that when the label is noisy, it takes much longer for gradient descent to overfit [Zhang et al., 2021]. Most practical NN training would have entered the Edge-of-Stability regime [Cohen et al., 2020] or stopped before the interpolation regime kicks in.

These observations motivate us to come up with an alternative theory for gradient-descent training of overparameterized neural networks that do not require interpolation.

1.1 Summary of Contributions

In this paper, we present a new theory of generalization for solutions that gradient descent (GD) with a fixed learning rate can stably converge to. Specifically:

We show that for 1D nonparametric regression ( n data points with noisy labels), the solutions that GD can stably converge to must be regular functions with small (weighted) first-order total variation (Theorem 4.1 and Corollary 4.2), thus promoting sparsity in the number of linear pieces. This generalizes the result of Mulayoff et al. [2021] by removing the 'interpolation' assumption. We also show that in the noisy case, there is no 'flat' interpolating solution and gradient descent cannot converge to them unless the learning rate is O (1 /n 2 ) no matter how overparameterized the two-layer ReLU network is (Theorem 3.1).

We show that such solutions (stable local minima that GD converges to) cannot overfit , in the sense that the generalization gap vanishes as n → ∞ inside the strict interior of the data support (Theorem 4.3). Moreover, under a mild additional assumption on gradient descent finding solutions with training loss smaller than σ 2 , we prove that these solutions achieve near-optimal rate for estimating (the strict interior of) first-order bounded variation functions (Theorem 4.4) -provably faster than any kernel ridge regression estimators, including neural networks in the 'kernel' regime.

We conduct extensive numerical experiments to demonstrate our theoretical predictions, validate our technical assumptions, and illustrate the functional form of the ReLU NNs as well as the

learned basis functions that gradient descent finds with different step sizes. These results reveal new insights into how gradient descent training aggressively learns representation and induces implicit sparsity.

To the best of our knowledge, these results are new to this paper. We emphasize that (1) the training objective function is not explicitly regularized; (2) we do not early-stop training in favor of algorithmic-stability; and (3) the solutions that gradient descent converges to are not global optimal (interpolating) solutions unless the label noise is 0 . Our approach is new in that we directly analyze the complexity of a superset of solutions that gradient descent can stably converge to, which enables us to prove end-to-end generalization bounds that are near-optimal in nonparametric regression tasks.

Our analysis for gradient descent-training is categorically different from those in the 'kernel' (a.k.a. 'lazy') regime about interpolating solutions. Instead, we rigorously prove (and empirically demonstrate) that large step-size gradient descent do not behave this way and it does not converge to interpolating solutions. Our results fall into the non-kernel regime of neural network learning known as the 'rich' (a.k.a. the 'feature learning' or 'sparse') regime [Chizat et al., 2019, Woodworth et al., 2020], in which the weights and biases can move arbitrarily far away from their initialization.

Technical novelty. The main technical innovation in our analysis is in handling the 1 n ∑ i ( y i -f θ ( x i )) ∇ 2 θ f θ ( x i ) term that arises in the minima stability analysis when it was previously handled by Mulayoff et al. [2021] using interpolation, i.e., y i = f ( x i ) ∀ i ∈ [ n ] . It turns out in the noisy-label case, we can decompose the term into a certain Gaussian complexity measure and a self-bounding style MSE of that f θ . A non-trivial step is to bound the largest eigenvalue of ∇ 2 f θ ( · ) by a constant which results in a uniform bound of both the Gaussian complexity and the MSE. These bounds themselves are vacuous in terms of the implied generalization error, but plugging them into the function-space constraint imposed by the noisy minima stability bound restricts the learned ReLU NN to be inside a weighted TV1 function class (Details in Theorem 4.1). This, in turn, allows us to amplify a vacuous MSE bound into a new MSE bound that is nearly optimal in the strict interior of the data support. The main technique in the last step involves bounding the metric entropy of the weighted TV1 class and then carefully working out a self-bounding (square loss) version of the Dudley's chaining argument [Wainwright, 2019].

Disclaimers and limitations. It is important to note that while we analyze gradient descent training of overparameterized neural networks, the computational claim is very different from those in the kernel regime. The analysis in the kernel regime ensures that gradient descent finds interpolating solutions efficiently. We do not have a comparable efficiency claim. While computational guarantees on stationary point (and local minima) convergence in non-convex optimization problems are wellunderstood [Ghadimi and Lan, 2013, Jin et al., 2017], we do not have guarantees on whether the solution that gradient descent finds satisfies our assumption on the training loss being 'optimized' (smaller than label noise σ 2 ). Instead, our results provide a generalization gap bound for any stable solutions (Theorem 4.3) and a near-optimal excess risk (MSE vs ground truth) bound (Theorem 4.4) when the solution that GD finds happens to satisfy the assumption (empirically it does!). This is a meaningful middle ground between classical learning theory which does not concern optimization at all and modern theory that is fully optimization-dependent.

While we focus on (full batch) gradient descent for a clean presentation, the minima stability for stochastic gradient descent is immediate under the stochastic definition of minima stability [Mulayoff et al., 2021]. The same reason applies to us focusing on univariate nonparametric regression. Our technique can be used to generalize the multivariate function space interpretation of minima stability from Nacson et al. [2022] to the noisy case, but it will take substantial effort to formalize the corresponding generalization bounds in the multivariate case, which we leave as a future work.

1.2 Related Work and Implications of Our Results

Generalization in Overparameterized NNs, Interpolation, and Benign Overfitting. Most existing theoretical work on understanding the generalization of overparameterized neural networks focuses on the interpolation regime [Cao and Gu, 2019, Frei et al., 2022, Kou et al., 2023, Buzaglo et al., 2024]. Handling label noise either requires explicit regularization [Hu et al., 2020, Zhang and Wang, 2022], algorithmic stability through early stopping [Hardt et al., 2016, Richards and Kuzborskij, 2021], or carefully crafted data distribution that leads to a phenomenon known as benign overfitting [Bartlett et al., 2020, Frei et al., 2022, Kou et al., 2023]. Benign overfitting could happen for nonparametric

regression tasks [Belkin et al., 2019], but there is well-documented empirical and theoretical evidence that benign overfitting does not occur for regression tasks with ReLU activation [Mallinar et al., 2022, Haas et al., 2023, Joshi et al., 2023] and that the excess risk is required to be proportional to the standard deviation of the label noise [Kornowski et al., 2024]. We are the first to go beyond the interpolation regime and show that gradient descent-trained neural networks generalize in noisy regression tasks without explicit regularization.

Implicit bias of gradient descent. The implicit bias of gradient descent training of overparameterized NN is well-studied. It was shown that among the many globally optimal (interpolating) solutions, gradient descent finds the ones with the smallest norm in certain Hilbert spaces [Arora et al., 2019, Mei et al., 2019], classifiers with largest-margin [Chizat and Bach, 2020], or the smoothest cubic spline interpolation [Jin and Montúfar, 2023]. None of these results, however, imply generalization bounds when the labels are noisy. Interestingly, our results show that gradient descent with a large step size induces an implicit bias that resembles sparse L1-regularization rather than the dense L2 regularization from gradient flow [Jin and Montúfar, 2023].

Implicit bias of minima stability. The closest to our work is the line of work on the implicit bias of minima stability [Ma and Ying, 2021, Mulayoff et al., 2021, Wu and Su, 2023]. We build directly on top of the function-space interpretation of minima stability established by Mulayoff et al. [2021]. However, these works critically rely on the minima interpolating the data, which makes their results inapplicable to settings with label noise. Mulayoff et al. [2021] also did not establish formal generalization bounds. Ma and Ying [2021], Wu and Su [2023] do have generalization bounds, but (again) their results require interpolation and thus do not apply to our settings.

Flat/Sharp Minima and generalization. Our work is also connected to the body of work on the hypothesis that 'flat local minima generalize better'. Despite compelling empirical evidence [Hochreiter and Schmidhuber, 1997, Keskar et al., 2017], rigorous theoretical understanding of this hypothesis is still lacking [see, e.g., Wu and Su, 2023, and the references therein]. Our work contributes to this literature by formally proving that the hypothesis is real for two-layer ReLU NNs in a noisy regression task.

Edge-of-Stability and Catapults. Empirical observations on how large learning rate training of NN finds solutions with Hessian's largest eigenvalue dancing around 2 /η , i.e., 'edge of stability' regime [Cohen et al., 2020]; and that the loss may go up first before going down to a good solution ('catapult') [Lewkowycz et al., 2020]. Existing theoretical understanding of these curious behaviors of GD training is still limited to toy-scale settings (e.g., Arora et al. [2022], Ahn et al. [2023], Kreisler et al. [2023]). Our work is complementary in that we provide generalization bound to the final solution GDstabilizes on no matter how GD gets there. Outside the context of GD and neural networks, 'edge of stability' and the implicit bias of large step-size were observed for forward stagewise regression [see, e.g. Tibshirani, 2015, 2014, Section 4.4 and Page 42] albeit only empirically. Our results may provide a theoretical handle in formally analyzing these observations.

Optimal rates of NNs in nonparametric regression. Finally, it was previously known that neural networks can achieve optimal rates for estimating TV1 functions [Suzuki, 2018, Liu et al., 2021, Parhi and Nowak, 2021, Zhang and Wang, 2022]. Specifically, Savarese et al. [2019], Ongie et al. [2020], Parhi and Nowak [2021], Zhang and Wang [2022] show that weight decay in ReLU networks is connected to total variation regularization. However, these works assume one can solve an appropriately constrained or regularized empirical risk minimization problem. Our work is the first to show that the optimal rate is achievable with gradient descent without weight decay. In fact, it was a pleasant surprise to us that both weight decay and large learning rate induce total variation-like implicit regularization in the function space.

2 Notations and Problem Setup

Let us set up the problem formally. Throughout the paper, we use O ( · ) , Ω( · ) to absorb constants while ˜ O ( · ) suppresses logarithmic factors. Meanwhile, [ n ] = { 1 , 2 , · · · , n } .

Two-layer neural network. We consider two-layer ( i.e. one-hidden-layer) univariate ReLU networks,

where the network consists of k hidden neurons and ϕ ( · ) denotes the ReLU activation function.

Training data and loss function. The training dataset is denoted by D = { ( x i , y i ) ∈ R × R , i ∈ [ n ] } . { x i } n i =1 is assumed to be supported by [ -x max , x max ] for some constant x max > 0 . We focus on regression problems with square loss ℓ ( f, ( x, y )) = 1 2 ( f ( x ) -y ) 2 . The training loss is defined as L ( f ) = 1 2 n ∑ n i =1 ( f ( x i ) -y i ) 2 . Notice that f is parameterized by θ := [ w (1) 1: k , b (1) 1: k , w (2) 1: k , b (2) ] ∈ R 3 k +1 . As a short hand, we define ℓ i ( θ ) := ℓ ( f θ , ( x i , y i )) and L ( θ ) := 1 n ∑ i ∈ [ n ] ℓ i ( θ ) .

Gradient descent. We focus on the Gradient descent (GD) learner, which iteratively updates θ :

where η > 0 is the step size (a.k.a. learning rate) and θ 0 is the initial parameter. Detailed calculation of gradient for two-layer ReLU networks is deferred to Appendix E. Below we define stability for local minima and discuss the conditions for a minimum to be stable.

Twice differentiable stable local minima. Similar to Mulayoff et al. [2021], we consider twice differentiable minima. According to Taylor's expansion around a twice differentiable minimum θ ⋆ ,

where ∇ 2 L denotes the Hessian matrix and ∇L ( θ ⋆ ) = 0 . Therefore, as θ t gets close to θ ⋆ , the update rule for GD (2) can be approximated as θ t +1 ≈ θ t -η ( ∇L ( θ ⋆ ) + ∇ 2 L ( θ ⋆ )( θ t -θ ⋆ ) ) . Such approximation motivates the definition of linear stability, which is first stated in Wu et al. [2018].

Definition 2.1 (Linear stability) . With the update rule θ t +1 = θ t -η ( ∇L ( θ ⋆ ) + ∇ 2 L ( θ ⋆ )( θ t -θ ⋆ ) ) , a twice differentiable local minimum θ ⋆ of L is said to be ϵ linearly stable if for any θ 0 in the ϵ -ball B ϵ ( θ ⋆ ) , it holds that lim sup t →∞ ∥ θ t -θ ⋆ ∥ ≤ ϵ .

Note that different from previous works [Wu et al., 2018, Mulayoff et al., 2021], we remove the expectation before ∥ θ t -θ ⋆ ∥ since under GD everything is deterministic. Intuitively speaking, linear stability requires that once we have arrived at a distance of ϵ from θ ⋆ , we end up staying in the ϵ -ball B ϵ ( θ ⋆ ) . It is known that linear stability is connected to the flatness of the local minima.

Lemma 2.2. Consider the update rule in Definition 2.1, for any ϵ > 0 , a local minimum θ ⋆ is an ϵ linearly stable minimum of L if and only if λ max ( ∇ 2 L ( θ ⋆ )) ≤ 2 η .

The implication is that the set of stable minima is equivalent to the set of flat local minima whose largest eigenvalue of Hessian is smaller than 2 /η . The proof is adapted from Mulayoff et al. [2021] and we state the proof in Appendix C for completeness. When the result does not depend on ϵ (as above), we simply say 'linearly stable'. Throughout the paper, we overload the notation by calling a function f = f θ linearly stable function if θ is linearly stable.

'Edge of Stability' regime. Extensive empirical and theoretical evidence (Cohen et al. [2020], Damian et al. [2024], and see Section 1.2) have shown that the threshold of linear stability (from Lemma 2.2) is quite significant in GD dynamics: GD iterations initially tend to exhibit 'progressive sharpening', where λ max ( ∇ 2 L ( θ t )) is increasing, until finally GD reaches the 'Edge of Stability', where λ max ( ∇ 2 L ( θ t )) ≈ 2 /η . We capture this phenomenon with the following definition.

Definition 2.3 (Below Edge of Stability) . Wesay that a sequence of parameters { θ t } t =1 , 2 , ··· generated by gradient descent with step-size η is ϵ -approximately Below-Edge-of-Stability (BEoS) for ϵ > 0 if there exists t ∗ > 0 such that λ max ( ∇ 2 L ( θ t )) ≤ 2 e ϵ η for all t ≥ t ∗ . Any θ t with t ≥ t ∗ is referred to as an ϵ -BEoS solution.

The BEoS regime provides a strong justification for the connection between the step size η and the largest eigenvalue of the Hessian matrix. It holds for all twice-differentiable solutions GD finds along the way - even if the GD does not converge to a (local or global) minimum. Empirically, BEoS is valid for both the 'progressive sharpening' phase and the oscillating EoS phase for a small constant ϵ .

Our goal in this paper is to understand generalization for both (twice differentiable) stable local minima (Definition 2.1) and any other solutions satisfying ϵ -BEoS (Definition 2.3), which are both subsets of

To simplify the presentation, we focus on the case with ϵ = 0 w.l.o.g. 1 and unless otherwise specified, a 'stable solution' refers to an element of F ( η, 0 , D ) in the remainder of the paper.

For the data generation process, we will consider two settings of interest: (1) the fixed design nonparametric regression setting (with noisy labels) (2) the agnostic statistical learning setting. They have different data assumptions and performance metrics to capture 'generalization'.

Nonparametric Regression with Noisy labels. In this setting, we assume fixed input x 1 , · · · , x n and y i = f 0 ( x i ) + ϵ i for i ∈ [ n ] , where f 0 : R → R is the ground-truth (target) function and { ϵ i } n i =1 are independent Gaussian noises N (0 , σ 2 ) . Our goal is find a ReLU NN f using the dataset to minimize the mean squared error (MSE):

It is nonparametric because we do not require f 0 to be described by a smaller number of parameters, but rather satisfy certain regularity conditions. Specifically, we focus on estimating target functions inside the first order bounded variation class

where f ′′ denotes the second-order weak derivative of f and we define a short hand TV (1) ( f ) := ∫ x max -x max | f ′′ ( x ) | dx , which we refer to as the TV (1) (semi)norm of f throughout the paper. We refer readers to a recent paper [Hu et al., 2022, Section 1.2] for the historical importance and the challenges in estimating the BV functions. The complexity of such function class is discussed in Appendix D.

Agnostic statistical learning and generalization gap. In this setting, we assume the n data points { ( x i , y i ) } n i =1 are drawn i.i.d. from an unknown distribution P defined on [ -x max , x max ] × [ -D,D ] . The expected performance on new data points is called 'Risk', R ( f ) = E ( x,y ) ∼P [ ℓ ( f, ( x, y ))] . We define the absolute difference between training loss and the risk:

We say that f generalizes if GeneralizationGap( f ) → 0 as n →∞ with high probability.

3 Stable Solutions Cannot Interpolate Noisy Labels

A large portion of previous works studying minima stability assume the learned function interpolates the data. However, for various optimization problems, it is unclear whether there exists such an interpolating solution that is stable, especially when the number of samples n becomes large.

For the nonparametric regression problem with noisy labels, we design an example where any interpolating function can not be stable. Before presenting the example, we first define the g function, which will be the weight function of the weighted TV (1) norm throughout the paper: for x ∈ [ -x max , x max ] , g ( x ) = min { g -( x ) , g + ( x ) } with

where X is drawn from the empirical distribution of the data (a sample chosen uniformly from { x j } ).

For various distributions of training data ( e.g. Gaussian distribution, uniform distribution), most of g 's mass is located at the center while g decays towards the extreme data points. The same g ( x ) is also applied as the weight function in Mulayoff et al. [2021], where they derived an upper bound ∫ x max -x max | f ′′ ( x ) | g ( x ) dx ≤ 1 η -1 2 assuming f is linearly stable (Definition 2.1) and interpolating. We generalize the same upper bound to all stable solutions in F ( η, 0 , D ) as in Theorem C.2. Below we construct a counter-example, where we can prove a contradicting lower bound of ∫ x max -x max | f ′′ ( x ) | g ( x ) dx for any interpolating f , thus disproving the assumption of interpolation.

Counter-example. We fix x i = 2 x max i n -1 -( n +1) x max n -1 for i ∈ [ n ] and f 0 ( x ) = 0 for any x , which implies that y i 's are independent random variables from N (0 , σ 2 ) .

Theorem 3.1. For the counter-example, with probability 1 -δ , for any interpolating function f ,

where the randomness comes from the noises { ϵ i } . Under this high-probability event, when n ≥ Ω (√ 1 ση log ( 1 δ ) ) , any stable solution f for GD with step size η will not interpolate the data, i.e.

The proof of Theorem 3.1 is deferred to Appendix F due to space limit. This result, together with Mulayoff et al. [2021, Theorem 1], implies that gradient descent cannot converge to interpolating solutions unless η = O (1 /n 2 ) . It also implies (when combined with Theorem 4.1) an intriguing geometric insight that all twice-differentiable interpolating solutions must be very sharp, i.e. , its largest eigenvalue is larger than Ω( n 2 ) (see details in Appendix J ). Moreover, we highlight that the conclusion of Theorem 3.1 is consistent with our observation in Figure 2(a), where the learned function tends to be smoother and would not interpolate the data as η becomes larger. Therefore, in the following discussion, we consider the case without assuming interpolation.

4 Main Results

In this section, we present the main results about stable solutions for GD (functions in F ( η, 0 , D ) ) from three aspects. Section 4.1 describes the implicit bias of stable solutions of gradient descent with large learning rate in the function space. Section 4.2 and 4.3 derive concrete generalization bounds that leverage the implicit biases in the distribution-free statistical learning setting and the non-parametric regression setting respectively. An outline of the proof of our main theorems is given in Appendix B. The full proof details are deferred to the appendix.

4.1 Implicit Bias of Stable Solutions in the Function Space

̸

We begin with characterizing the stable solutions for GD with step size η without the assumption of interpolation (there can be i ∈ [ n ] such that f ( x i ) = y i ). Similar to the interpolating case, the learned stable function f enjoys a (weighted) TV (1) bound as below.

Theorem 4.1. For a function f = f θ where the training (square) loss L is twice differentiable at θ , 2

where g ( x ) is defined as (6) . Moreover, if we assume y i = f 0 ( x i ) + ϵ i for independent noise ϵ i ∼ N (0 , σ 2 ) , then with probability 1 -δ where the randomness is over the noises { ϵ i } ,

In addition, if f = f θ is a stable solution of GD with step size η on dataset D , i.e., f θ ∈ F ( η, 0 , D ) as in (4) , then we can replace λ max ( ∇ 2 θ L ( θ )) 2 with 1 η in (9) and (10) .

Theorem 4.1, which we prove in Appendix G, associates the local curvature of the loss landscape at θ with the smoothness of the function f θ it represents as measured in a weighted TV (1) norm. In short, it says that flat solutions are simple . The result is a strict generalization of Theorem 1 in Mulayoff et al. [2021] which requires interpolation, i.e., L ( θ ) = 0 . Observe that the number of neurons k does not appear in (9) and have no effect in (10) when k > n , thus the result applies to arbitrarily overparameterized two-layer NNs. Under the standard nonparametric regression assumption, (10) is a stronger bound that asymptotically matches the bound under interpolation [Mulayoff et al., 2021, Theorem 1] when k = o ( n ) and MSE( f θ ) = o (1) as the number of data points n →∞ .

Another interesting observation when combining (10) with Theorem 3.1 is that for all interpolating solutions (observe that MSE( f θ ) = ˜ O ( σ 2 ) w.h.p.)

To say it differently, all interpolating solutions are very sharp minima when the labels are noisy . This provides theoretical explanation of the empirical observation that noisy labels are harder to overfit using gradient training [Zhang et al., 2021].

Note that we leave the term L ( θ ) (or MSE( f ) ) in the TV (1) bound. Therefore, we can plug any upper bound for these terms into Theorem 4.1 for a concrete result, and below we instantiate the TV (1) bound with a crude MSE bound under the assumption that f is 'optimized'.

Corollary 4.2. In the nonparametric regression problem with ground-truth function f 0 , for a stable solution f = f θ of GD with step size η where L is twice differentiable at θ , assume that f is optimized, i.e, the empirical loss of f is smaller than f 0 : 1 2 n ∑ n i =1 ( f ( x i ) -y i ) 2 ≤ 1 2 n ∑ n i =1 ( f 0 ( x i ) -y i ) 2 , then with probability 1 -δ , the function f satisfies

where the randomness is over the noises { ϵ i } and ˜ O suppresses logarithmic terms of n, 1 /δ .

The assumption of optimized f is rather mild since, in practice, gradient-based optimizers are commonly quite effective in loss minimization (see also our experiments). With such assumption, we can derive an MSE upper bound (with high probability) of order ˜ O ( σ 2 ) (details in Lemma G.5), and thus the TV (1) bound above.

In some cases, we can decrease the MSE upper bound we assumed in Corollary 4.2, and use this to improve the resulting TV (1) bound (11). For instance, if the neural network is under-parameterized (i.e. k is smaller than n ), in Appendix G.2 we derive a (high-probability) bound for MSE of order ˜ O ( k/n ) , which implies that the last term in (11) becomes ˜ O ( σx max √ k/n ) . The term vanishes if n/k is large enough, where the TV (1) bound reduces to the noiseless and interpolating case.

4.2 GD on ReLU NN Does Not Overfit

Why would anyone care about the function space implications of stable solutions? The next theorem shows that these solutions cannot overfit (in the strict interior of the data support) without making strong assumptions on the shape of the data distribution.

Theorem 4.3. Let P be a joint distribution of ( x, y ) supported on [ -x max , x max ] × [ -D,D ] . Assume the dataset D ∼ P n i.i.d. For any fixed interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that with probability 1 -δ/ 2 , g ( x ) ≥ c for all x ∈ I , if the function f = f θ is a stable solution of GD with step size η such that L is twice differentiable at θ and ∥ f ∥ ∞ ≤ D , with probability 1 -δ (randomness over the dataset), the generalization gap restricted to I satisfies

where E I means that ( x, y ) is a new sample from the data distribution conditioned on x ∈ I and n I is the number of data points in D such that x i ∈ I .

The proof of Theorem 4.3 is deferred to Appendix H. Briefly speaking, we show that in the strict interior of the data support, the generalization gap will vanish as the number of data points n increases. This vanishing generalization gap further implies that the expected performance on new data points is close to the (observable) training loss, i.e., the output stable solutions do not overfit.

Regarding our assumptions, in addition to standard boundedness assumptions, we focus on the strict interior of the domain where g can be lower bounded (i.e. interval I ). This is because for extreme data points, g ( x ) decays and thus imposes little constraint on the output function f . In Appendix H.1, we show that if the marginal distribution of x is the uniform distribution on [ -x max , x max ] , when n is sufficiently large, I can be chosen as [ -2 x max 3 , 2 x max 3 ] . In this case, with high probability, I incorporates a large portion of the data points and n I = Ω( n ) . More illustrations about the choice of I under various data distributions are deferred to Appendix H.2.

Meanwhile, the generalization gap bound has dependence 1 η on the learning rate η . Therefore, as we increase the learning rate (in a reasonable range), the learned stable solution tends to be smoother, which further implies better generalization performances.

4.3 GD on ReLU NN Achieves Optimal Rate for Estimating BV(1) Functions

Finally, we zoom into the nonparametric regression task that we described in Section 2 where x 1 , · · · , x n are fixed and the noisy labels y i = f 0 ( x i ) + N (0 , σ 2 ) independently for i ∈ [ n ] for a ground truth function f 0 in the first order bounded variation class (see Section 2 for details). Similar to Theorem 4.3, we focus on the strict interior I of [ -x max , x max ] , but instead of a generalization gap bound, we prove an MSE bound against f 0 on I that nearly matches the theoretical limit.

Theorem 4.4. Under the same conditions in Corollary 4.2, for any interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that g ( x ) ≥ c for all x ∈ I and f is optimized over I , i.e. ∑ x i ∈I ( f ( x i ) -y i ) 2 ≤ ∑ x i ∈I ( f 0 ( x i ) -y i ) 2 , if the output stable solution θ satisfies ∥ θ ∥ ∞ ≤ ρ (for some constant ρ > 0 ) and the ground truth f 0 ∈ BV (1) ( kρ 2 , 1 c ˜ O ( 1 η + σx max )) , then with probability 1 -δ (over the random noises { ϵ i } ), the function f = f θ satisfies

where n I is the number of data points in D such that x i ∈ I .

The proof of Theorem 4.4 is deferred to Appendix I. Below we discuss some interesting aspects of the result. First of all, Theorem 4.4 focuses on an interval I where g ( x ) can be lower bounded. In this way, we ignore the extreme data points and derive an MSE upper bound (restricted to I ) of order ˜ O ( n -4 / 5 I ) , which matches the minimax optimal rate for estimating BV (1) functions [see, e.g., Donoho and Johnstone, 1998, Theorem 1]. In contrast, it is well-known that neural networks in the kernel regime (and any other 'linear smoothers') must incur a strictly suboptimal worst-case MSE = Ω( n I 2 / 3 ) [Donoho et al., 1990, Suzuki, 2018]. According to the discussions in Appendix H.1, if the data follows a uniform distribution, the interval I can incorporate most of the data points where n I = Ω( n ) .

Meanwhile, the MSE bound has dependence η -2 / 5 on the learning rate η . Such dependence is because with a larger learning rate, the learned function f will have smaller TV (1) bound, and therefore the set of possible output functions will contain fewer non-smooth functions, which implies a tighter MSE bound. However, this does not mean that a larger learning rate is always better. When η is too large, GD may diverge, and even if it does not, the set of stable solutions cannot approximate the ground truth f 0 well if ∫ x max -x max | f ′′ 0 ( x ) | dx ≫ 1 cη + σx max c , thus failing to satisfy the 'optimized' assumption. In our experiments, we verify the 'optimized' assumption numerically for a wide range of η and demonstrate that by tuning learning rate η , we are adapting to the unknown TV (1) ( f 0 ) .

Lastly, we remark that Theorem 4.4 holds for arbitrary k , even if the neural network is heavily over-parameterized ( k ≫ n ). The dependence 1 η + σx max on η results from the TV (1) bound in Corollary 4.2, where the term 1 η will dominate if η ≤ 1 σx max , which is the case if the step size is not large. Furthermore, if n k is sufficiently large, we can improve the TV (1) bound in Corollary 4.2 as in

( (

σ

n

Appendix G.2, and improve the mean squared error

MSE

I

(

f

)

to

˜

O

2

I

η

accordingly.

The assumptions and detailed statements for the improved MSE are deferred to Appendix I.1.

5 Experiments

In this section, we empirically validate our claims by training a two-layer fully connected neural network with ReLU activation using gradient descent (GD) with varying step sizes. We focus on fitting a mildly overparameterized ReLU network to a simple nonparametric regression problem. The input dataset comprises of 30 equally spaced fixed design points { x i } n i =1 , where each x i ∈ [ -0 . 5 , 0 . 5] ( n = 30 ). Label y i = f 0 ( x i )+ N (0 , σ 2 ) with σ = 0 . 5 and f 0 ( x ) = (2 x +1) 1 ( x ≤ 0)+( -2 x +1) 1 ( x > 0) .

)

4

5

(

x

)

2

5

)

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.4

True Function

Fitted function

Fitted Labels

0

50

100

150

200

250

Iterations (in 100s)

10

1

10 0

Loss

Learning Curves:

=0.4

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.0

0.2

0.4

0.6

0.8

Learned basis functions (

=0.4)

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.01

True Function

Fitted function

Fitted Labels

0

2000

4000

6000

8000

10000

Iterations (in 100s)

10

1

Loss

Learning Curves:

=0.01

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Learned basis functions (

=0.01)

The two-layer ReLU network is parameterized by θ (see Section 2) with k = 100 neurons per layer. The network uses standard parameterization (scale factor of 1) and parameters are initialized randomly (see Figure 7 for the initial basis functions).

Figure 2 (in the introduction) illustrates how changing the learning rate affects the learned ReLU NN that GD-training stabilizes on. The main take-aways are (a) large learning rate learns flatter minima which represent more regular functions (in TV (1) ); (b) Our bound from Theorem 4.1 is a very accurate description of the curvature of the Hessian as well as a valid upper bound of the TV (1) -(pseudo) norm; (c) When we tune learning rate η , it is implicitly regularizing the complexity, which provides a satisfying variance tradeoff explanation to how GD-training works.

Figure 3 provides further details on the learning curves and representation learning. We note that the learned representation is very different from the initialization, thus our experiments are clearly describing phenomena not covered by the 'kernel' regime. In addition, it seems that all solutions that GD finds after a small number of iterations satisfy the 'optimized' assumption as required in Theorem 4.4. In the appendix (Figure 8), we provide empirical justification for the other assumption we make about the twice-differentiability of the solutions. More experiments can be found in the appendix with more learning rate choices, as well as a discussion on the catastrophic and tempered overfitting of interpolating solutions when we adjust k .

6 Conclusion

In this paper, we took a new look into how gradient descent-trained two-layer ReLU neural networks generalize from a lens of minima stability (and the closely related Edge-of-Stability phenomena). We focused on univariate inputs with noisy labels and showed GD with typical choice of learning rate cannot interpolate the data. We also established that local smoothness of the training loss functions implies a first order total variation constraint on the function the neural network represents, hence proving that all such solutions have a vanishing generalization gap inside the strict interior of the data support. In addition, under a mild assumption, we prove that these stable solutions achieve near-optimal rate for estimating first-order bounded variation functions. Future work includes generalization beyond 1D input, two-hidden layers, and understanding the choice of optimization algorithms.

Acknowledgments

The research is partially supported by NSF #2134214. DQ, KZ, ES and YW's work was partially completed while they were with the Department of Computer Science at UCSB. The research of DS was Funded by the European Union (ERC, A-B-C-Deep, 101039436). Views and opinions expressed are however those of the author only and do not necessarily reflect those of the European Union or the European Research Council Executive Agency (ERCEA). Neither the European Union nor the granting authority can be held responsible for them. DS also acknowledges the support of the Schmidt Career Advancement Chair in AI.

References

Kwangjun Ahn, Sébastien Bubeck, Sinho Chewi, Yin Tat Lee, Felipe Suarez, and Yi Zhang. Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems , 36, 2023.

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning , pages 322-332. PMLR, 2019.

Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi. Understanding gradient descent on the edge of stability in deep learning. In International Conference on Machine Learning , pages 948-1024. PMLR, 2022.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1611-1619. PMLR, 2019.

Gon Buzaglo, Itamar Harel, Mor Shpigel Nacson, Alon Brutzkus, Nathan Srebro, and Daniel Soudry. How uniform random weights induce non-uniform bias: Typical interpolating neural networks generalize with narrow teachers. In International Conference on Machine Learning , 2024.

Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. Advances in neural information processing systems , 32, 2019.

Herman Chernoff. A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. The Annals of Mathematical Statistics , pages 493-507, 1952.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on learning theory , pages 1305-1338. PMLR, 2020.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems , 32, 2019.

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability. In International Conference on Learning Representations , 2020.

Alex Damian, Eshaan Nichani, and Jason D Lee. Self-stabilization: The implicit bias of gradient descent at the edge of stability. In International Conference on Learning Representations , 2024.

Christoph Dann, Tor Lattimore, and Emma Brunskill. Unifying pac and regret: Uniform pac bounds for episodic reinforcement learning. Advances in Neural Information Processing Systems , 30, 2017.

Ronald A DeVore and George G Lorentz. Constructive approximation , volume 303. Springer Science &Business Media, 1993.

David L Donoho and Iain M Johnstone. Minimax estimation via wavelet shrinkage. The annals of Statistics , 26(3):879-921, 1998.

David L Donoho, Richard C Liu, and Brenda MacGibbon. Minimax risk over hyperrectangles, and implications. The Annals of Statistics , pages 1416-1437, 1990.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations , 2018.

David Eric Edmunds and Hans Triebel. Function spaces, entropy numbers, differential operators. (No Title) , 1996.

Spencer Frei, Niladri S Chatterji, and Peter Bartlett. Benign overfitting without linearity: Neural network classifiers trained by gradient descent for noisy linear data. In Conference on Learning Theory , pages 2668-2703. PMLR, 2022.

Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM journal on optimization , 23(4):2341-2368, 2013.

Moritz Haas, David Holzmüller, Ulrike Luxburg, and Ingo Steinwart. Mind the spikes: Benign overfitting of kernels and neural networks in fixed dimension. Advances in Neural Information Processing Systems , 36, 2023.

Yifan Hao and Tong Zhang. The surprising harmfulness of benign overfitting for adversarial robustness. arXiv preprint arXiv:2401.12236 , 2024.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International conference on machine learning , pages 1225-1234. PMLR, 2016.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation , 9(1):1-42, 1997.

Addison J Hu, Alden Green, and Ryan J Tibshirani. The voronoigram: Minimax estimation of bounded variation functions from scattered data. arXiv preprint arXiv:2212.14514 , 2022.

Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference on Learning Representations , 2020.

Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems , 31, 2018.

Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan. How to escape saddle points efficiently. In International conference on machine learning , pages 1724-1732. PMLR, 2017.

Hui Jin and Guido Montúfar. Implicit bias of gradient descent for mean squared error regression with two-layer wide neural networks. Journal of Machine Learning Research , 24(137):1-97, 2023.

Nirmit Joshi, Gal Vardi, and Nathan Srebro. Noisy interpolation learning with shallow univariate relu networks. In International Conference on Learning Representations , 2023.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations , 2017.

Guy Kornowski, Gilad Yehudai, and Ohad Shamir. From tempered to benign overfitting in relu neural networks. Advances in Neural Information Processing Systems , 36, 2024.

Yiwen Kou, Zixiang Chen, Yuanzhou Chen, and Quanquan Gu. Benign overfitting in two-layer relu convolutional neural networks. In International Conference on Machine Learning , pages 17615-17659. PMLR, 2023.

Itai Kreisler, Mor Shpigel Nacson, Daniel Soudry, and Yair Carmon. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In International Conference on Machine Learning , pages 17684-17744. PMLR, 2023.

Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218 , 2020.

Chaoyue Liu, Libin Zhu, and Mikhail Belkin. Loss landscapes and optimization in over-parameterized non-linear systems and neural networks. Applied and Computational Harmonic Analysis , 59: 85-116, 2022.

Hao Liu, Minshuo Chen, Tuo Zhao, and Wenjing Liao. Besov function approximation and binary classification on low-dimensional manifolds using convolutional residual networks. In International Conference on Machine Learning , pages 6770-6780. PMLR, 2021.

Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems , 34:16805-16817, 2021.

Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems , 35:1182-1195, 2022.

Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory , pages 2388-2464. PMLR, 2019.

Rotem Mulayoff, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability: A view from function space. Advances in Neural Information Processing Systems , 34:17749-17761, 2021.

Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongie, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability in multivariate shallow relu networks. In The Eleventh International Conference on Learning Representations , 2022.

Richard Nickl and Benedikt M Pötscher. Bracketing metric entropy rates and empirical central limit theorems for function classes of besov-and sobolev-type. Journal of Theoretical Probability , 20: 177-199, 2007.

Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. In ICLR , 2020.

Rahul Parhi and Robert D Nowak. Banach space representer theorems for neural networks and ridge splines. Journal of Machine Learning Research , 22(43):1-40, 2021.

Dan Qiao and Yu-Xiang Wang. Near-optimal deployment efficiency in reward-free reinforcement learning with linear function approximation. In The Eleventh International Conference on Learning Representations , 2023a.

Dan Qiao and Yu-Xiang Wang. Near-optimal differentially private reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 9914-9940. PMLR, 2023b.

Dan Qiao and Yu-Xiang Wang. Offline reinforcement learning with differential privacy. Advances in Neural Information Processing Systems , 36, 2023c.

Dan Qiao and Yu-Xiang Wang. Near-optimal reinforcement learning with self-play under adaptivity constraints. In International Conference on Machine Learning , 2024.

Dan Qiao, Ming Yin, Ming Min, and Yu-Xiang Wang. Sample-efficient reinforcement learning with loglog (t) switching cost. In International Conference on Machine Learning , pages 18031-18061. PMLR, 2022.

Dan Qiao, Ming Yin, and Yu-Xiang Wang. Logarithmic switching cost in reinforcement learning beyond linear mdps. arXiv preprint arXiv:2302.12456 , 2023.

Dominic Richards and Ilja Kuzborskij. Stability & generalisation of gradient descent for shallow neural networks without the neural tangent kernel. Advances in neural information processing systems , 34:8609-8621, 2021.

Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory , pages 2667-2690. PMLR, 2019.

Karthik Sridharan. A gentle introduction to concentration inequalities. Dept. Comput. Sci., Cornell Univ., Tech. Rep , 2002.

Taiji Suzuki. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality. arXiv preprint arXiv:1810.08033 , 2018.

Ryan J Tibshirani. Fast stagewise algorithms for approximate regularization paths. https://www. stat.berkeley.edu/~ryantibs/talks/stagewise-2014.pdf , 2014.

Ryan J Tibshirani. A general framework for fast stagewise algorithms. J. Mach. Learn. Res. , 16(1): 2543-2588, 2015.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory , pages 3635-3673. PMLR, 2020.

Lei Wu and Weijie J Su. The implicit regularization of dynamical stability in stochastic gradient descent. In International Conference on Machine Learning , pages 37656-37684. PMLR, 2023.

Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems , 31, 2018.

Jianyu Xu, Dan Qiao, and Yu-Xiang Wang. Doubly fair dynamic pricing. In International Conference on Artificial Intelligence and Statistics , pages 9941-9975. PMLR, 2023.

Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM , 64(3):107-115, 2021.

Kaiqi Zhang and Yu-Xiang Wang. Deep learning meets nonparametric regression: Are weightdecayed dnns locally adaptive? In International Conference on Learning Representations , 2022.

Fuheng Zhao, Dan Qiao, Rachel Redberg, Divyakant Agrawal, Amr El Abbadi, and Yu-Xiang Wang. Differentially private linear sketches: Efficient implementations and applications. Advances in Neural Information Processing Systems , 35:12691-12704, 2022.

A Full Experimental Results

A.1 Stable Minima GD Converges to and Learning Curves

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.5

True Function

Fitted function

Fitted Labels

0

25

50

75

100

125

150

175

200

Iterations (in 100s)

10

1

10 0

Loss

Learning Curves:

=0.5

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.4

True Function

Fitted function

Fitted Labels

0

50

100

150

200

250

Iterations (in 100s)

10

1

10 0

Loss

Learning Curves:

=0.4

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.3

True Function

Fitted function

Fitted Labels

0

50

100

150

200

250

300

Iterations (in 100s)

10

1

10 0

Loss

Learning Curves:

=0.3

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

1.0

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.2

True Function

Fitted function

Fitted Labels

0

100

200

300

400

500

Iterations (in 100s)

10

1

10 0

Loss

Learning Curves:

=0.2

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.1

True Function

Fitted function

Fitted Labels

0

200

400

600

800

1000

Iterations (in 100s)

10

1

Loss

Learning Curves:

=0.1

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.05

True Function

Fitted function

Fitted Labels

0

250

500

750

1000 1250 1500 1750 2000

Iterations (in 100s)

10

1

Loss

Learning Curves:

=0.05

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.02

True Function

Fitted function

Fitted Labels

0

1000

2000

3000

4000

5000

Iterations (in 100s)

10

1

Loss

Learning Curves:

=0.02

train Loss

MSE vs truth

2

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

Trained ReLU NN with

=0.01

True Function

Fitted function

Fitted Labels

0

2000

4000

6000

8000

10000

Iterations (in 100s)

10

1

Loss

Learning Curves:

=0.01

train Loss

MSE vs truth

2

A.2 Interpolating Solutions as the Number of Hidden Neurons Increases

In this section, we illustrate a sequence of interpolating solutions that are the global optimal solutions, which is also the kernel limit in the 'lazy' regime. The results are obtained by randomly initializing

the weights, but solve the minimum norm solution by directly solving the least square problem (optimizing only the second layer weights.)

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

True Function

Fitted function

Fitted Labels

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

True Function

Fitted function

Fitted Labels

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

True Function

Fitted function

Fitted Labels

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

True Function

Fitted function

Fitted Labels

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

True Function

Fitted function

Fitted Labels

0.4

0.2

0.0

0.2

0.4

0.5

0.0

0.5

1.0

1.5

True Function

Fitted function

Fitted Labels

A.3 Representation Learning of Large Learning Rate: Visualizing Learned Basis Functions.

In this section, we visualize the basis functions at initialization and after training with different learning rate.

0.4

0.2

0.0

0.2

0.4

0.0

0.2

0.4

0.6

0.8

1.0

Random basis functions at initialization

0.4

0.2

0.0

0.2

0.4

0.04

0.02

0.00

0.02

0.04

Learned basis functions (

=0.5)

0.4

0.2

0.0

0.2

0.4

0.0

0.2

0.4

0.6

0.8

Learned basis functions (

=0.4)

0.4

0.2

0.0

0.2

0.4

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Learned basis functions (

=0.3)

0.4

0.2

0.0

0.2

0.4

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

Learned basis functions (

=0.2)

0.4

0.2

0.0

0.2

0.4

0.0

0.5

1.0

1.5

2.0

2.5

Learned basis functions (

=0.1)

0.4

0.2

0.0

0.2

0.4

0.0

0.5

1.0

1.5

2.0

2.5

Learned basis functions (

=0.05)

0.4

0.2

0.0

0.2

0.4

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Learned basis functions (

=0.02)

0.4

0.2

0.0

0.2

0.4

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Learned basis functions (

=0.01)

We make several observations about Figure 7. First, the learned basis functions are very different from the initialization, so a lot of representation learning is happening, in comparison to the 'kernel' regime in which nearly no representation learning is happening. Second, as η gets smaller, the number of learned basis functions increases, hence increasing the number of knots in the fitted function. Third, the learned basis function displays a strong 'clustering' effect in the sense that despite overparameterization, many learned basis functions end up being the same on the data support. Interestingly, they are not the same on R , we verified that they are still different outside the data support, e.g., one of the learned basis function has a knot at x < -800 .

A.4 Knots of the Learned ReLU NN (aka Linear Splines) and Their Coefficients

Recall that a linear spline is a continuous piecewise linear function and a two-layer ReLU NN with k neurons span the class of all linear splines with at most k knots. In Figure 8, we visualize the locations of the knots of the linear spline that the learned ReLU neural networks represent.

10 1

10 2

1/

3

2

1

0

1

2

3

location on input axis

Quantiles of the Learned knots

0.05 quantile

0.25 quantile

0.5 quantile

0.75 quantile

0.95 quantile

10 1

10 2

1/

50

100

150

200

250

300

350

400

Sparsity

Sparsity of the learned coefficients

L

1-norm

Lp

norm

p

=0.5

Lp

norm

p

=0.2

10 1

10 2

1/

10

6

10

5

10

4

10

3

10

2

10

1

10 0

Distance to the closest input x

Learned knots to the closest input knot

Minimum

1st percentile

Median

In the first panel of Figure 8, we find for large learning rate, most of the location of the learned knot is actually outside the data support on [ -0 . 5 , 0 . 5] . This is a somewhat surprising finding in that the mechanism of neural network learning may actually be 'pushing the knot outside the data support' so they become inactive on the training data (and only the ReLU truncated 0s are active). This is a new (and very interesting) way to understand how sparsity arrives in gradient descent learning.

The second panel describes the sparsification effect of the implicit biases from large learning rate, which again, indicates that the weighted TV1 constraint is indeed making the learned function sparse (in the coefficient vector). The third panel shows that despite that the learning rate gets as small as 1 e -2 , none of the learned basis function actually have knots coinciding with any of the input data, thus empirically justifying our assumptions on the twice-differentiability of the solutions GD finds.

B Proof Overview

In this section, we outline proof ideas for the main theorems in Section 4.

Proof overview of Theorem 4.1. According to direct calculation, the Hessian is given by

Let v denote the unit eigenvector ( ∥ v ∥ 2 = 1 ) of 1 n ∑ n i =1 ( ∇ θ f θ ( x i ))( ∇ θ f θ ( x i )) T with respect to the largest eigenvalue, then it holds that

For the term ( ⋆ ) , we connect the maximal eigenvalue at θ to the (weighted) TV (1) norm of the corresponding f = f θ . Let g ( x ) be defined as (6), Mulayoff et al. [2021, Lemma 4] shows that (details in Lemma G.1):

For the term (#) , we can bound it by the training loss L ( θ ) using Cauchy-Schwarz inequality and a somewhat surprising uniform upper bound of v T ∇ 2 θ f ( x i ) v in Lemma E.1:

The first conclusion (9) of Theorem 4.1 is derived by combining the inequalities above. For the second conclusion (10), note that the term (#) can be further decomposed as:

Similar to (#) , the term | ( ii ) | can be bounded by 2 x max √ MSE( f θ ) using Cauchy-Schwarz inequality. Under the data-generating assumption y i -f 0 ( x i ) = ϵ i ∼ N (0 , σ 2 ) i.i.d., | ( i ) | can be bounded by a certain empirical Gaussian complexity term sup θ 1 n ∑ i ϵ i h θ ( x i ) with h θ ( x i ) = v T ∇ 2 θ f ( x i ) v . The proof is complete by Lemma G.2 which bounds this Gaussian complexity term (w.h.p.) in two ways: (1) a dimension-free bound of ˜ O ( σx max ) and (2) ˜ O ( σx max √ k/n ) for the under-parameterized case.

Proof overview of Theorem 4.3. Under the boundedness assumption in Theorem 4.3, we can prove a constant upper bound for ∫ x max -x max | f ′′ ( x ) | g ( x ) dx (Lemma H.1), which further implies another constant upper bound for ∫ I | f ′′ ( x ) | dx . Therefore, the metric entropy (logarithmic of ϵ -covering number N ϵ ) of the possible output function class is of order ϵ -1 / 2 (Details in Lemma H.2).

For a fixed ϵ -cover of the possible output function class, according to Hoeffding's inequality and a union bound, the uniform upper bound of Gen I ( f ) can be bounded by ˜ O ( √ log N ϵ /n I ) = ˜ O ( ϵ -1 4 n -1 2 I ) with high probability. Note that the approximation error is of order ˜ O ( ϵ ) , therefore Gen I ( f ) can be uniformly bounded over the possible output function class by

where ϵ is chosen to minimize the bound. More details can be found in the proof of Theorem H.3.

Proof overview of Theorem 4.4. Similar to Theorem 4.3, we can prove a constant upper bound for ∫ I | f ′′ ( x ) | dx , which implies that the metric entropy of the possible output function class (4) is of order ϵ -1 / 2 (Details in Lemma I.2). Therefore, the critical radius r is of order ˜ O ( n -2 5 I ) , which leads to a (high probability) MSE upper bound of order ˜ O ( n -4 5 I ) using a self-bounding technique. More details about handling other parameters can be found in Appendix I.

C Some Optimization Results

Lemma C.1 (Restate Lemma 2.2) . Consider the update rule in Definition 2.1, for any ϵ > 0 , a local minimum θ ⋆ is an ϵ linearly stable minimum of L if and only if λ max ( ∇ 2 L ( θ ⋆ )) ≤ 2 η .

Proof of Lemma C.1. It holds that

where the first equation is from the update rule in Definition 2.1. The second equation holds because θ ⋆ is a local minimum and therefore ∇L ( θ ⋆ ) = 0 . As a result,

On one hand, if λ max ( ∇ 2 L ( θ ⋆ )) ≤ 2 η , it holds that

where the second inequality is because all the eigenvalues of I -η ∇ 2 L ( θ ⋆ ) is bounded between [ -1 , 1] . Therefore, θ ⋆ is ϵ linearly stable for any ϵ .

On the other hand, if θ ⋆ is ϵ linearly stable, we choose θ 0 such that θ 0 -θ ⋆ ∥ θ 0 -θ ⋆ ∥ is the top eigenvector of ∇ 2 L ( θ ⋆ ) and ∥ θ 0 -θ ⋆ ∥ = ϵ . Then we have

which implies that lim sup t →∞ ∣ ∣ 1 -ηλ max ( ∇ 2 L ( θ ⋆ ) )∣ ∣ t ≤ 1 , and therefore λ max ( ∇ 2 L ( θ ⋆ )) ≤ 2 η , which finishes the proof.

The following Theorem C.2 is an extension of the main result in Mulayoff et al. [2021]. Recall that stable solutions refer to the functions in F ( η, 0 , D ) as defined in Section 2.

Theorem C.2 (Extension of Theorem 1 in Mulayoff et al. [2021]) . Let f = f θ be a stable solution for GD with step size η where the training loss L ( f ) = 0 and L is twice differentiable at θ . Then

where g ( x ) = min { g -( x ) , g + ( x ) } for x ∈ [ -x max , x max ] with

Here X is drawn from the empirical distribution of the data (a sample chosen uniformly from { x j } ).

Proof of Theorem C.2. The proof of Theorem 1 in Mulayoff et al. [2021] first proves that λ max ( ∇ 2 L ( θ )) ≤ 2 η according to the assumption that f is linearly stable, and then proves the conclusion above. Therefore, the same conclusion directly follows for the stable solutions f = f θ in F ( η, 0 , D ) satisfying λ max ( ∇ 2 L ( θ )) ≤ 2 η .

Some discussions about the result. Mulayoff et al. [2021] studied the problem assuming that the optimization converges to a global minimum and the learned function interpolates the training data, i.e. L ( f ) = 0 . In this way, they link the Hessian matrix to properties of the learned function f and show that stable solutions of GD correspond to functions whose second order derivative has a bounded weighted norm. Moreover, as the learning rate increases, the set of stable solutions contains less and less non-smooth functions. For a fixed learning rate, according to the curve of g ( x ) , stable solutions tend to be smoother for instances near the center of the data distribution, and less smooth for instances near the edges. More discussions can be found in Mulayoff et al. [2021].

D Bounded Variation Function Class and its Metric entropy

In this section, we first define the Besov function class. Then we recall the definition of bounded variation function class and discuss the connection between these two classes. Finally we bound the metric entropy of bounded variation function class using analysis about Besov class.

D.1 Definition of Besov Class

Let Ω be the domain of the function class (which we omit in the definition) and ∥ · ∥ p denote the ℓ p norm. We first define the modulus of smoothness.

Definition D.1. For a function f ∈ L p (Ω) where 1 ≤ p ≤ ∞ , the r -th modulus of smoothness is defined by

where ∆ is defined as

Then the norm of Besov space is defined as below.

Definition D.2. For 1 ≤ p, q ≤ ∞ , α > 0 , r := ⌈ α ⌉ +1 , define

Then the norm of Besov space is defined as:

Finally, a function f is in the Besov class B α p,q if ∥ f ∥ B α p,q is finite. For more discussions and properties of Besov class, we refer interesting readers to Edmunds and Triebel [1996].

D.2 Definition of Bounded Variation Class and the Connection

For the same domain Ω , recall that the bounded variation function class is defined as

According to DeVore and Lorentz [1993], bounded total variation class is closely connected to the Besov class. Specifically, for any constant B,C n , it holds that

D.3 Metric Entropy of Bounded Total Variation Function Class

Now we bound the complexity of BV (1) (1 , 1) , which will be helpful for bounding the complexity of BV (1) ( B,C n ) in the future. We first define the metric entropy of a metric space ( T , ρ ) .

Definition D.3. For a set T with a corresponding metric ρ ( · , · ) , let N ( ϵ, T , ρ ) denote the ϵ -covering number of T under metric ρ . Then the metric entropy of T with respect to ρ is log N ( ϵ, T , ρ ) .

More details and examples about covering and metric entropy can be found in Chapter 5 of Wainwright [2019]. Next we bound the metric entropy of a bounded subset of BV (1) . Note that the ℓ ∞ metric over domain Ω is ρ ∞ ( f, g ) = sup x ∈ Ω | f ( x ) -g ( x ) | , which we denote by ∥ · ∥ ∞ for short.

Lemma D.4. Assume the set T 1 = { f : [ -1 , 1] → R ∣ ∣ ∣ ∣ ∫ 1 -1 | f ′′ ( x ) | dx ≤ 1 , | f ( x ) | ≤ 1 } and the metric is the ℓ ∞ distance ∥ · ∥ ∞ , then there exists a universal constant C 1 > 0 such that for any ϵ > 0 , the metric entropy of ( T 1 , ∥ · ∥ ∞ ) satisfies

Proof of Lemma D.4. First of all, the domain Ω = [ -1 , 1] is a bounded set in R . Moreover, according to DeVore and Lorentz [1993], we have BV (1) ⊂ B 2 1 , ∞ . Therefore, T 1 is a bounded subset of B 2 1 , ∞ (Ω) with both B 2 1 , ∞ and ℓ ∞ norm bounded by a universal constant.

Therefore, the metric space ( T 1 , ∥ · ∥ ∞ ) satisfies the assumptions in (the second point) of Corollary 2 in Nickl and Pötscher [2007] with r = ∞ , d = 1 , s = 2 . Then combining the conclusion of Corollary 2 in Nickl and Pötscher [2007] and the fact that the metric entropy is upper bounded by bracketing metric entropy (Definition 4 in Nickl and Pötscher [2007]), we finish the proof.

Remark D.5. For our purpose of bounding the metric entropy of BV (1) , we only consider the case where Ω is bounded. For more results regarding the metric entropy of (weighted) Besov space, please refer to Nickl and Pötscher [2007].

E Calculation of Gradient and Hessian Matrix

In this section, we calculate the gradient and Hessian matrix of f θ ( x ) with respect to θ . Recall that f θ ( x ) = ∑ k i =1 w (2) i ϕ ( w (1) i x + b (1) i ) + b (2) where ϕ ( x ) = max { x, 0 } . We denote θ = ( w (1) 1 , · · · , w (1) k , b (1) 1 , · · · , b (1) k , w (2) 1 , · · · , w (2) k , b (2) ) T .

E.1 Calculation of Gradient

According to direct calculation, for a given x ∈ [ -x max , x max ] we have

E.2 Calculation of the Hessian Matrix

In this part, we calculate ∇ 2 θ f θ ( x ) for a given x ∈ [ -x max , x max ] . Below we calculate ∂ 2 f θ ( x ) ∂θ i ∂θ j .

̸

First of all, if θ i = b (2) or θ j = b (2) , ∂ 2 f θ ( x ) ∂θ i ∂θ j = 0 . Then it remains to calculate ∂ 2 f θ ( x ) ∂θ i ∂θ ′ j where i, j ∈ [ k ] and θ, θ ′ ∈ { w (1) , b (1) , w (2) } . It is obvious that if i = j , ∂ 2 f θ ( x ) ∂θ i ∂θ ′ j = 0 . Therefore, we only calculate the case when j = i . Let δ denote the Dirac function, it holds that:

The gradient is generally not well-defined according to the existence of the Dirac function. However, under the assumption that f θ is twice differentiable with respect to θ ( i.e. the knots of f do not coincide with x ), all the Dirac functions take the value 0 . In this case,

E.3 Upper Bound of Operator Norm

In this part, we upper bound the operator norm of the Hessian matrix. Equivalently, we upper bound ∣ ∣ v T ∇ 2 f θ ( x ) v ∣ ∣ under the constraint that ∥ v ∥ 2 = 1 . We have the following lemma.

Lemma E.1. Assume that f θ ( x ) is twice differentiable with respect to θ and x ∈ [ -x max , x max ] , for any v such that ∥ v ∥ 2 = 1 , it holds that

Proof of Lemma E.1. Assume v = ( α 1 , · · · , α k , β 1 , · · · , β k , γ 1 , · · · , γ k , ι ) T ∈ R 3 k +1 such that ∑ k i =1 ( α 2 i + β 2 i + γ 2 i ) + ι 2 = 1 . Note that the Hessian matrix ∇ 2 θ f θ ( x ) follows the structure:

where A w (1) w (1) , A w (1) b (1) , A b (1) w (1) , A b (1) b (1) , A w (2) w (2) ∈ R k × k , A w (1) b (2) , A b (1) b (2) , A w (2) b (2) ∈ R k × 1 , A b (2) w (1) , A b (2) b (1) , A b (2) w (2) ∈ R 1 × k and A b (2) b (2) ∈ R are all zero matrices. Meanwhile, A w (1) w (2) , A b (1) w (2) , A w (2) w (1) , A w (2) b (1) ∈ R k × k are all diagonal matrices whose non-zero elements are between [ -max { x max , 1 } , max { x max , 1 } ] . Therefore, it holds that:

where the second inequality holds because of Cauchy-Schwarz inequality. The last inequality results from x (1 -x ) ≤ 1 4 .

F Proof for the Counter-Example (Theorem 3.1)

Theorem F.1 (Restate Theorem 3.1) . For the counter-example, with probability 1 -δ , for any interpolating function f ,

where the randomness comes from the noises { ϵ i } . Under this high-probability event, when n ≥ Ω (√ 1 ση log ( 1 δ ) ) , any stable solution f for GD with step size η will not interpolate the data, i.e.

Proof of Theorem F.1. Consider any three consecutive points x j , x j +1 , x j +2 where j ∈ [ n -2] , note that their corresponding y 's are y j , y j +1 , y j +2 which are i.i.d. Gaussian random variables N (0 , σ 2 ) . Then according to Mean Value Theorem, there exists a ∈ [ x j , x j +1 ] and b ∈ [ x j +1 , x j +2 ] such that

Therefore, it holds that

where the last equation means that y j +2 -2 y j +1 + y j follows the distribution N (0 , 6 σ 2 ) . We focus on the interval in the middle. For any x ∈ [ x n/ 4 , x 3 n/ 4 ] , we have

Together with the definition of g ( x ) (6), we have for any x ∈ [ x n/ 4 , x 3 n/ 4 ] , g ( x ) ≥ x max 64 . Therefore, for any interpolating solutions f , it holds that

where G i 's are i.i.d samples from N (0 , 6 σ 2 ) .

Assume the median of |N (0 , 1) | is c > 0 , which is a universal constant. For any i ∈ [ n 6 ] , define

Then we have H i = √ 6 cσ with probability 1 2 . In addition, | G i | ≥ H i . According to Lemma K.1, with probability at least 1 -δ ,

for some universal constant c ′ .

Together with the conclusion in Theorem C.2, for any interpolating and stable solution f , with probability 1 -δ ,

which does not hold when n ≥ Ω (√ 1 ση log ( 1 δ ) ) .

G Proof for the TV (1) Bound (Theorem 4.1)

We begin by calculating the gradient of empirical loss L at θ :

where the detailed calculation of ∇ θ f θ ( x ) can be found in Appendix E. Then the Hessian is given by

where ∇ 2 θ f ( x ) is calculated in Appendix E. Let v denote the unit eigenvector ( ∥ v ∥ 2 = 1 ) of 1 n ∑ n i =1 ( ∇ θ f θ ( x i ))( ∇ θ f θ ( x i )) T with respect to the largest eigenvalue, it holds that

For term ( ⋆ ) , we connect the maximal eigenvalue at θ to the (weighted) TV (1) norm of the corresponding f = f θ . Let the weight function g ( x ) be defined as (6), then the lemma below holds.

Lemma G.1. Assume L is twice differentiable at θ and the corresponding function of θ is f , then

where g ( x ) is defined as (6) .

Proof of Lemma G.1. The proof of Lemma 4 in Mulayoff et al. [2021] directly proves the result for λ max ( 1 n ∑ n i =1 ( ∇ θ f θ ( x i ))( ∇ θ f θ ( x i )) T ) , which is the λ max ( ∇ 2 θ L ) in Lemma 4 of Mulayoff et al. [2021] when f is an interpolating solution.

For the first inequality of Theorem 4.1, we directly handle the term (#) as below.

where the first inequality results from Cauchy-Schwarz inequality. The second inequality is because of the uniform upper bound of v T ∇ 2 θ f ( x i ) v (Lemma E.1).

For the second inequality of Theorem 4.1, we bound the two terms (i) and (ii). We begin with | ( i ) | = ∣ ∣ 1 n ∑ n i =1 v T ∇ 2 θ f ( x i ) v · ϵ i ∣ ∣ , which is a weighted sum of noises { ϵ i } .

LemmaG.2. Assume ϵ i 's are independently sampled from N (0 , σ 2 ) for some σ > 0 , with probability at least 1 -δ , uniformly over all θ, v such that L is twice differentiable at θ and ∥ v ∥ 2 = 1 ,

Proof of Lemma G.2. For the first part, according to Lemma E.1, we have

Since ϵ i 's are independently sampled from Gaussian distribution N (0 , σ 2 ) , according to concentration of Gaussian distribution and a union bound, with probability 1 -δ 2 , it holds that:

Under this high-probability event, we have

For the second part, we bound the complexity of { v T ∇ 2 θ f ( x i ) v } n i =1 . Notice that θ is a function of the dataset, thus not independent to ϵ i . Also, v is a function of the dataset and θ . Our strategy is to apply an ϵ -net argument for both v and ∇ 2 θ f ( x i ) for i = 1 , · · · , n .

We begin with considering the possibilities of {∇ 2 θ f ( x i ) } n i =1 . According to the detailed form of ∇ 2 θ f θ ( x ) in Appendix E, we have the set {∇ 2 θ f ( x i ) } n i =1 is fully determined by { 1 ( w (1) j x i + b (1) j > 0 )} n,k i,j =1 , 1 . Therefore it suffices to cover all the possibilities of { 1 ( w (1) j x i + b (1) j > 0 )} n i =1 for all j ∈ [ k ] . Without loss of generality, we can assume that x 1 < x 2 < · · · < x n , and then { w (1) j x i + b (1) j } n i =1 is also monotonic, which implies that there are 2( n +1) possibilities of { 1 ( w (1) j x i + b (1) j > 0 )} n i =1 in total. As a result, the product space { 1 ( w (1) j x i + b (1) j > 0 )} n,k i,j =1 , 1 (and also {∇ 2 θ f ( x i ) } n i =1 ) has N 1 = (2 n +2) k possibilities.

For a fixed matrix M = ∇ 2 θ f ( x ) for some θ, x and v, v ′ such that ∥ v ∥ 2 ≤ 1 , ∥ v ′ ∥ 2 ≤ 1 , ∥ v -v ′ ∥ 2 ≤ ϵ with ϵ ∈ (0 , 1) , it holds that

where the third inequality results from the upper bound of operator norm of M (Lemma E.1). Therefore, the exact covering set of {∇ 2 θ f ( x i ) } n i =1 and an ϵ 6 max { x max , 1 } -covering set of the unit Euclidean Ball with dimension 3 k + 1 (which is exactly the domain of v ) together provides an ϵ -cover of {( v T ∇ 2 θ f ( x i ) v ) n i =1 } with respect to ∥ · ∥ ∞ . Meanwhile, an ϵ 6 max { x max , 1 } -cover with respect to ∥ · ∥ 2 of the unit Euclidean Ball with dimension 3 k + 1 has cardinality bounded by N 2 = ( 1 + 12 max { x max , 1 } ϵ ) 3 k +1 according to Lemma K.2. Combining the two covering arguments, the ϵ -covering set of {( v T ∇ 2 θ f ( x i ) v ) n i =1 } with respect to ∥ · ∥ ∞ has cardinality N ϵ satisfying:

Consider a fixed stream ( a i ) n i =1 in the covering set, we have | a i | ≤ 2 max { x max , 1 } for all i ∈ [ n ] . Then according to the concentration result of Gaussian distribution, with probability 1 -δ ,

With a union bound over the ϵ -covering set of {( v T ∇ 2 θ f ( x i ) v ) n i =1 } and conditioned on the high probability event of (55), with probability 1 -δ , uniformly over all possible ∥ v ∥ 2 = 1 and θ ,

Finally, by choosing ϵ = max { x max , 1 } √ n , we have

Remark G.3. According to the Lemma G.2 above, there are two cases. When the neural network is over-parameterized, we can derive a constant upper bound for term (i). Meanwhile, if the number of neurons is smaller than the sample complexity, we can derive a tighter bound for (i) by covering { v T ∇ 2 θ f ( x i ) v } n i =1 . For k, n such that k is polynomially smaller than n ( e.g. k = n 1 -α for some positive α ), the term (i) will vanish if n converges to infinity.

Meanwhile, the term (ii) can be bounded by the mean squared error MSE( f θ ) using Cauchy-Schwarz inequality and the uniform upper bound of v T ∇ 2 θ f ( x i ) v (Lemma E.1).

Combining the results above, we state the following weighted TV (1) bound of the learned function. We leave the training loss and mean squared error (MSE) in the bound, which will be handled later.

Theorem G.4 (Restate Theorem 4.1) . For a function f = f θ where the training (square) loss L is twice differentiable at θ ,

where g ( x ) is defined as (6) . Moreover, if we assume y i = f 0 ( x i ) + ϵ i for independent noise ϵ i ∼ N (0 , σ 2 ) , then with probability 1 -δ where the randomness is over the noises { ϵ i } ,

In addition, if f = f θ is a stable solution of GD with step size η on dataset D , i.e., f θ ∈ F ( η, 0 , D ) as in (4) , then we can replace λ max ( ∇ 2 θ L ( θ )) 2 with 1 η in (9) and (10) .

Proof of Theorem G.4. The first inequality results from plugging Lemma G.1 and (52) into (50). The second inequality holds by plugging Lemma G.1, Lemma G.2 and inequality (62) into (50). For the instantiation of λ max ( ∇ 2 θ L ( θ )) , the replacement holds due to the definition of F ( η, 0 , D ) in (4).

G.1 A Crude Bound for MSE and the Instantiated TV (1) Bound (Corollary 4.2)

Now we prove a crude upper bound for MSE, which could instantiate a weighted TV (1) bound.

Lemma G.5. Assume that the function f is optimized, then with probability 1 -δ , we have

Proof of Lemma G.5. According to the assumption that f is optimized, it holds that

Then since ( a + b ) 2 ≤ 2 a 2 +2 b 2 always holds (AM-GM inequality), we have

Recall that ϵ i 's are independently sampled from N (0 , σ 2 ) , then according to the concentration of Gaussian distribution and a union bound, with probability 1 -δ ,

Combining the two results, with probability 1 -δ where the randomness is over the noises { ϵ i } ,

Finally, Corollary 4.2 results from directly plugging Lemma G.5 into Theorem 4.1.

G.2 An Improved MSE Bound and the Corresponding TV (1) Bound

In this part, we provide an improved upper bound of the mean squared error MSE( f θ ) and also the term (ii). We first make the assumption below that the parameters are from a bounded space.

Assumption G.6. There exists some constant ρ > 0 such that gradient descent converges to some local minimum θ with ∥ θ ∥ ∞ ≤ ρ . In addition, we assume that ∥ f 0 ∥ ∞ ≤ D where D > 0 is some universal constant satisfying that D ≤ kρ 2 ( x max +1) .

Assumption G.6 ensures that the parameter space is bounded while the ground truth function f 0 can be approximated well by some possible output function. The assumption will surely hold for some large enough constants ρ, D , which is without loss of generality. In the following analysis, we will replace D with its upper bound kρ 2 ( x max +1) to reduce the parameters in the logarithmic terms.

To handle the mean squared error MSE( f θ ) , we begin with an analysis on the complexity of the function class of two-layer ReLU networks with bounded parameters F ρ = { f : [ -x max , x max ] → R ∣ ∣ ∣ ∣ f ( x ) = ∑ k i =1 w (2) i ϕ ( w (1) i x + b (1) i ) + b (2) , ∥ θ ∥ ∞ ≤ ρ } . Note that here we assume that the input is from [ -x max , x max ] and there exists an upper bound ρ > 0 on the parameter θ = ( w (1) 1 , · · · , w (1) k , b (1) 1 , · · · , b (1) k , w (2) 1 , · · · , w (2) k , b (2) ) T ∈ R 3 k +1 .

Lemma G.7. The ϵ -covering number N ϵ of function class F ρ with respect to ∥ · ∥ ∞ satisfies

Proof of Lemma G.7. We consider the discrete function class below

where θ j is the j -th element of ( ¯ w (1) 1 , · · · , ¯ w (1) k , ¯ b (1) 1 , · · · , ¯ b (1) k , ¯ w (2) 1 , · · · , ¯ w (2) k , ¯ b (2) ) T ∈ R 3 k +1 and ¯ ϵ · Z is the set { ¯ ϵ · i, i ∈ Z } . Since for each element, the number of choices is bounded by 2 ρ ¯ ϵ +1 , the total cardinality of ¯ F ¯ ϵ is bounded by ( 2 ρ ¯ ϵ +1 ) 3 k +1 .

For each function f ∈ F ρ with corresponding parameter θ ( ∥ θ ∥ ∞ ≤ ρ ), we choose function ¯ f from ¯ F ¯ ϵ with corresponding parameter ¯ θ such that | ¯ θ j -θ j | is minimized for all j ∈ [3 k +1] . According to our definition of ¯ F ¯ ϵ , we have for all j ∈ [3 k +1] , | ¯ θ j -θ j | ≤ ¯ ϵ . Therefore, it holds that

where the last inequality is by choosing ¯ ϵ = ϵ 5 max { x max , 1 } kρ .

Therefore, the ϵ -covering number N ϵ of function class F ρ with respect to ∥ · ∥ ∞ satisfies

Now we provide an upper bound of MSE( f θ ) under the assumption that θ is optimized, which means that the empirical error of f θ is smaller than that of f 0 , i.e.

Below we state the improved upper bound of mean squared error under such assumptions.

Lemma G.8. Assume ϵ i 's are independently sampled from N (0 , σ 2 ) for some σ > 0 , if Assumption G.6 holds and f θ ∈ F ρ is optimized, then with probability at least 1 -δ , it holds that

Proof of Lemma G.8. Since 1 2 n ∑ n i =1 ( f θ ( x i ) -y i ) 2 ≤ 1 2 n ∑ n i =1 ( f 0 ( x i ) -y i ) 2 , we have

For the optimized function f θ , we choose a function ¯ f from the ϵ -covering set in Lemma G.7 such that ∥ ¯ f -f θ ∥ ∞ ≤ ϵ . Due to identical analysis as (55), with probability 1 -δ 2 , max i {| ϵ i |} ≤ 2 σ √ log ( 4 n δ ) . Under this high-probability event, it holds that

According to Lemma G.7, the ϵ -covering number N ϵ of function class F ρ with respect to ∥ · ∥ ∞ satisfies log N ϵ ≤ 4 k log ( 11 max { x max , 1 } kρ 2 ϵ ) . Due to the concentration of Gaussian distribution and which implies that

a union bound over the covering set, with probability 1 -δ 2 , for all ¯ f in the covering set,

Combining the two high-probability events, with probability 1 -δ , we have

where the forth inequality results from Assumption G.6. Selecting ϵ = 1 k 2 ρ 2 n 2 max { x max , 1 } and solving the second order inequality, it holds that

Plugging in the upper bound for MSE (Lemma G.8) to Theorem 4.1, we have the corollary below.

Corollary G.9 (Improved version of Corollary 4.2) . For a stable solution f = f θ of GD with step size η where L is twice differentiable at θ , assume Assumption G.6 holds and f ∈ F ρ is optimized, with probability 1 -δ , the function f satisfies

where the randomness is over the noises { ϵ i } and g ( x ) is defined as (6) .

Remark G.10. The additional term here ˜ O ( σx max √ k n ) improves over the constant additional term in Corollary 4.2 if k < n . In addition, if n k converges to infinity ( e.g. k = n 1 -α for some α > 0 and n is sufficiently large), the additional term could vanish.

H Proof for the Generalization Gap Bound (Theorem 4.3)

Recall that the generalization gap of function f is defined as

where ( x, y ) is a new sample from the data distribution. The generalization gap measures the difference of the expected testing error and the training loss, and a small generalization gap implies that the model is not overfitting.

For a stable solution f = f θ of GD with step size η where L is twice differentiable at θ , we first derive a corresponding analysis for the weighted TV (1) bound. Recall that the empirical loss is still defined as L ( f ) = 1 2 n ∑ n i =1 ( f ( x i ) -y i ) 2 . Then using the same calculation as (50), we have

where the last inequality results from Cauchy-Schwarz inequality.

According to Lemma G.1, the term ( ⋆ ) satisfies

In addition, the term (#) satisfies (w.l.o.g, we assume x max ≥ 1 )

Under the assumption that the learned function f = f θ satisfies ∥ f ∥ ∞ ≤ D and | y i | ≤ D for all i ∈ [ n ] , it further implies that

Combining the results, the following TV (1) bound holds.

Lemma H.1. Assume the data distribution satisfies that for all possible ( x, y ) from the distribution, | x | ≤ x max and | y | ≤ D , if the function f = f θ is a stable solution of GD with step size η such that L is twice differentiable at θ and ∥ f ∥ ∞ ≤ D , then it holds that

where g ( x ) is defined as (6) .

Note that we assume there exists some interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that with probability 1 -δ/ 2 (randomness over the dataset), g ( x ) ≥ c for all x ∈ I (w.l.o.g. we assume c < 1 ), which further implies that

We base on the high-probability event above in the following discussions. Next we bound the metric entropy of the possible output function class.

LemmaH.2. Define the set T 3 = { f : I → R ∣ ∣ ∥ f ∥ ∞ ≤ D, ∫ I | f ′′ ( x ) | dx ≤ 1 c ( 1 η -1 2 +2 x max D )} , then the metric entropy with respect to ∥ · ∥ ∞ satisfies that

where O also absorbs the constant c .

Proof of Lemma H.2. Define the set T 4 as:

Note that the metric entropy of T 3 is bounded by that of T 4 , therefore we directly prove the upper bound of log N ( ϵ, T 4 , ∥ · ∥ ∞ ) .

Let the set T 1 = { f : [ -1 , 1] → R ∣ ∣ ∣ ∣ ∫ 1 -1 | f ′′ ( x ) | dx ≤ 1 , | f ( x ) | ≤ 1 } (as in Lemma D.4). For a fixed ϵ > 0 , according to Lemma D.4, there exists a ϵ 1 c ( 1 η -1 2 +2 x max D ) x max -covering set of T 1 with respect to ∥ · ∥ ∞ , denoted as { h i ( x ) } i ∈ [ N ] , whose cardinality N satisfies

We define g i ( x ) = 1 c ( 1 η -1 2 +2 x max D ) x max h i ( x x max ) for all i ∈ [ N ] . Then g i 's are all defined on [ -x max , x max ] . Obviously, we have { g i ( x ) } i ∈ [ N ] also has cardinality N .

For any f ( x ) ∈ T 4 , we define g ( x ) = 1 1 c ( 1 η -1 2 +2 x max D ) x max f ( x · x max ) which is defined on [ -1 , 1] . We now show that g ( x ) ∈ T 1 . First of all, for any x ∈ [ -x max , x max ] , we have

Meanwhile, it holds that

Combining the two results, we have g ∈ T 1 . Therefore, there exists some h i such that ∥ g -h i ∥ ∞ ≤ ϵ 1 c ( 1 η -1 2 +2 x max D ) x max . Since f ( x ) = 1 c ( 1 η -1 2 +2 x max D ) x max g ( x x max ) , ∥ g i -f ∥ ∞ = 1 c ( 1 η -1 2 +2 x max D ) x max ∥ h i -g ∥ ∞ ≤ ϵ .

In conclusion, { g i } i ∈ [ N ] is an ϵ -covering of T 4 with respect to ∥ · ∥ ∞ . Moreover, the cardinality of { g i } i ∈ [ N ] is N , which finishes the proof.

Now we provide our main result about the generalization gap (restricted to I ). Below we define the generalization gap restricted to I :

where E I means that ( x, y ) is a new sample from the data distribution conditioned on x ∈ I and n I is the number of data points in D such that x i ∈ I .

Theorem H.3 (Restate Theorem 4.3) . Let P be a joint distribution of ( x, y ) supported on [ -x max , x max ] × [ -D,D ] . Assume the dataset D ∼ P n i.i.d. For any fixed interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that with probability 1 -δ/ 2 , g ( x ) ≥ c for all x ∈ I , if the function f = f θ is a stable solution of GD with step size η such that L is twice differentiable at θ and ∥ f ∥ ∞ ≤ D , with probability 1 -δ (randomness over the dataset), the generalization gap restricted to I satisfies

where E I means that ( x, y ) is a new sample from the data distribution conditioned on x ∈ I and n I is the number of data points in D such that x i ∈ I .

Proof of Theorem H.3. We base on the following event that holds with probability 1 -δ/ 2 :

Then the output function f ∈ T 3 defined in Lemma H.2.

For a fixed ϵ > 0 , according to Lemma H.2, there exists an ϵ -covering set of T 3 (with respect to ∥ · ∥ ∞ ) whose cardinality N satisfies that

For a fixed function ¯ f in the covering set, since the data set { ( x i , y i ) } x i ∈I is still i.i.d. from the data distribution conditioned on x ∈ I , Hoeffding's inequality (Lemma K.3) implies that with probability 1 -δ , it holds that

Together with a union bound over the covering set, we have with probability 1 -δ/ 2 , for all ¯ f in the covering set,

Under such high probability event, for any f ∈ T 3 , let ¯ f be a function in the covering set such that ∥ f -¯ f ∥ ∞ ≤ ϵ . Then it holds that

where the last inequality results from selecting the ϵ that minimizes the objective.

H.1 Choice of the Interval Under Uniform Distribution

In this part, we discuss the choice of the interval I under the case that the marginal distribution of x is the uniform distribution on [ -x , x ] . For simplicity, we assume that x = 1 .

Lemma H.4. Assume that x ∼ Unif([ -1 , 1]) , then we can choose I to be [ -2 3 , 2 3 ] probability 1 -24 e -n 96 , for any x ∈ I , it holds that max max max . In this way, with

As a result, when n ≥ 96 log ( 48 δ ) , we can choose I = [ -2 3 , 2 3 ] and c = 1 4320 .

Proof of Lemma H.4. Let the intervals A i be defined as below: for all i ∈ [12] ,

For a fixed n , let P i denote the number of data points in A i , which follows Binomial distribution with p = 1 12 . Then for a fixed i ∈ [12] , according to Multiplicative Chernoff bound (Lemma K.4), with probability 1 -2 e -n 96 , it holds that

Then according to a union bound, with probability 1 -24 e -n 96 , the above inequality holds for all i ∈ [12] . Under the case above, we prove that g ( x ) = min { g -( x ) , g + ( x ) } ≥ 1 4320 for all x ∈ I .

Recall that g -( x ) = P 2 ( X < x ) E [ x -X | X < x ] √ 1 + ( E [ X | X < x ]) 2 where X is drawn from the empirical distribution of the data (a sample chosen uniformly from { x j } ). Then for any x ≥ -2 3 ,

Combining the inequalities, we have g -( x ) ≥ 1 4320 for all x ∈ I . The result for g + ( x ) can be proven similarly, which implies that with probability 1 -24 e -n 96 , g ( x ) ≥ 1 4320 for all x ∈ I .

The implication can be proven by direct calculation.

Remark H.5. We only consider the case where the data is sampled from uniform distribution, while we remark that for various distributions that are not heavy-tailed ( e.g. Gaussian distribution, Laplace distribution), a similar result can be derived. Some empirical illustrations are shown in Appendix H.2 below.

H.2 More Illustrations of the Choice of the Interval

In this part, we consider the choice of I and c under different data distributions. More specifically, we consider the following four distributions of x .

Uniform distribution : x ∼ Unif([ - 1 , 1])

.

Normal distribution : x ∼ N (0 , 1) .

Laplace distribution

:

x ∼ Laplace(0 , 1) .

Gaussian mixture distribution : x ∼ { N ( - 0 . 5 , 0 . 25) with probability 1 2 , N (0 . 5 , 0 . 25) with probability 1 .

For each distribution, we sample n = 1000 data points from the distribution (conditional on x ∈ [ -1 , 1] ) and construct the g ( x ) function. Then we choose the interval I and the corresponding lower bound c of g ( x ) over I . From Figure 9, we find that for all of the four distributions, with a constant c ≥ 0 . 002 , the interval I can be chosen to incorporate a large portion of the data ( n I ≥ 0 . 65 n ).

1.00

0.60

0.250.00 0.25

0.60

1.00

x

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

g(x)

Uniform distribution, c=0.005,

nI

=679

g

(

x

) = min(

g

(

x

),

g

+ (

x

))

y

=0.005

x

= ±(-0.69)

1.00

0.60

0.250.00 0.25

0.60

1.00

x

0.00

0.02

0.04

0.06

0.08

0.10

0.12

g(x)

Normal distribution, c=0.002,

nI

=780

g

(

x

) = min(

g

(

x

),

g

+ (

x

))

y

=0.002

x

= ±(-0.74)

1.00

0.60

0.250.00 0.25

0.60

1.00

x

0.00

0.02

0.04

0.06

0.08

0.10

g(x)

Laplace distribution, c=0.002,

nI

=797

g

(

x

) = min(

g

(

x

),

g

+ (

x

))

y

=0.002

x

= ±(-0.68)

1.00

0.60

0.250.00 0.25

0.60

1.00

x

0.00

0.02

0.04

0.06

0.08

0.10

0.12

g(x)

Gaussian mixture distribution, c=0.002,

nI

=784

g

(

x

) = min(

g

(

x

),

g

+ (

x

))

y

=0.002

x

= ±(-0.73)

I Proof for the Refined MSE Bound (Theorem 4.4)

In this part, we base on the same conditions in Corollary 4.2, which is the weighted TV (1) upper bound. For an output stable solution f satisfying the conclusion of Corollary 4.2, we have ∫ x max -x max | f ′′ ( x ) | g ( x ) dx ≤ 1 η -1 2 + ˜ O ( σx max ) and we denote the right-hand side by S . In addition, according to the assumption that g ( x ) ≥ c for any x ∈ I , we further have ∫ I | f ′′ ( x ) | dx ≤ S c . Now we bound the complexity of the possible output function class.

According to the definition of two-layer ReLU network and our assumption that ∥ θ ∥ ∞ ≤ ρ , it holds that:

Define the set T = { f : I → R ∣ ∣ | f (0) | ≤ kρ 2 + ρ, | f ′ (0) | ≤ kρ 2 , ∫ I | f ′′ ( x ) | dx ≤ S c } . According to the inequalities above, the possible output function (if restricted to I ) belongs to T . We begin with an analysis of the metric entropy of the intermediate function set T 2 .

LemmaI.1. Assume the set T 2 = { f : [ -x max , x max ] → R ∣ ∣ f (0) = f ′ (0) = 0 , ∫ x max -x max | f ′′ ( x ) | dx ≤ C 2 } for some constant C 2 > 0 , and the metric is ℓ ∞ distance ∥· ∥ ∞ , then there exists a universal constant

C 1 > 0 such that for any ϵ > 0 , the metric entropy of ( T 2 , ∥ · ∥ ∞ ) satisfies

where C 1 can be chosen as the same C 1 in Lemma D.4.

Proof of Lemma I.1. Let the set T 1 = { f : [ -1 , 1] → R ∣ ∣ ∣ ∣ ∫ 1 -1 | f ′′ ( x ) | dx ≤ 1 , | f ( x ) | ≤ 1 } (as in Lemma D.4). For a fixed ϵ > 0 , according to Lemma D.4, there exists a ϵ C 2 x max -covering set of T 1 with respect to ∥ · ∥ ∞ , denoted as { h i ( x ) } i ∈ [ N ] , whose cardinality N satisfies

We define g i ( x ) = C 2 x max h i ( x x max ) for all i ∈ [ N ] . Then g i 's are all defined on [ -x max , x max ] . Obviously, we have { g i ( x ) } i ∈ [ N ] also has cardinality N .

For any f ( x ) ∈ T 2 , we define g ( x ) = 1 C 2 x max f ( x · x max ) which is defined on [ -1 , 1] . We now show that g ( x ) ∈ T 1 . First of all, for any x ∈ [ -x max , x max ] , we have | f ′ ( x ) | ≤ ∫ x max -x max | f ′′ ( x ) | dx ≤ C 2 . Therefore, for any x ∈ [ -x max , x max ] , | f ( x ) | ≤ C 2 x max , which implies that | g ( x ) | ≤ 1 for any x ∈ [ -1 , 1] . Meanwhile, it holds that

Combining the two results, we have g ∈ T 1 . Therefore, there exists some h i such that ∥ g -h i ∥ ∞ ≤ ϵ C 2 x max . Since f ( x ) = C 2 x max g ( x x max ) , ∥ g i -f ∥ ∞ = C 2 x max ∥ h i -g ∥ ∞ ≤ ϵ .

In conclusion, { g i } i ∈ [ N ] is an ϵ -covering of T 2 with respect to ∥ · ∥ ∞ . Moreover, the cardinality of { g i } i ∈ [ N ] is N , which finishes the proof.

With Lemma I.1, we are ready to bound the metric entropy of T .

Lemma I.2. Assume the metric is ℓ ∞ distance ∥ · ∥ ∞ , then the metric entropy of ( T , ∥ · ∥ ∞ ) satisfies

where S is the right-hand side of Corollary 4.2 and O also absorbs the constant c .

Proof of Lemma I.2. For any function f ∈ T , it can be written as below:

where g ( x ) = f ( x ) -f (0) -f ′ (0) x satisfies that g (0) = g ′ (0) = 0 and g ′′ ( x ) = f ′′ ( x ) . Therefore, to cover T to ϵ accuracy, it suffices to cover the three parts to ϵ 3 accuracy with respect to ∥ · ∥ ∞ , respectively.

For f (0) , since | f (0) | ≤ kρ 2 + ρ , the covering number is bounded by

For f ′ (0) x , since | f ′ (0) | ≤ kρ 2 , the covering number is bounded by

Finally, for g ( x ) , since g ( x ) ∈ T 2 with C 2 = S c ( g is extended linearly beyond the interval I ), the covering number is bounded according to Lemma I.1 above.

Combining the three parts, the metric entropy is bounded by

where O also absorbs c , which is the constant lower bound of g ( x ) .

According to the metric entropy above, we are ready to provide a refined (high probability) bound for mean squared error (restricted to I ). Note that we assume that the ground-truth function f 0 ∈ T , which is necessary for the mean squared error to vanish.

Lemma I.3. Under the same conditions in Corollary 4.2, for any interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that g ( x ) ≥ c for all x ∈ I and f is optimized over I , i.e. ∑ x i ∈I ( f ( x i ) -y i ) 2 ≤ ∑ x i ∈I ( f 0 ( x i ) -y i ) 2 , if the output stable solution θ satisfies ∥ θ ∥ ∞ ≤ ρ and the ground truth f 0 ∈ T , then with probability 1 -δ (over the random noises { ϵ i } ), the function f = f θ satisfies

where n I is the number of data points in D such that x i ∈ I .

Proof of Lemma I.3. According to the assumption that f is optimized over I , we have

Similar to the calculation in Lemma G.8, it holds that

It is obvious that the function class T is convex, together with the assumption that f 0 ∈ T , we have the function set T ⋆ := T -{ f 0 } is star-shaped (details in Section 13 of Wainwright [2019]).

Note that the metric entropy of T satisfies that log N ( ϵ, T , ∥ · ∥ ∞ ) ≤ O (√ x max S ϵ ) . According to Corollary 13.7 of Wainwright [2019], the critical radius r satisfies that

Finally, according to Theorem 13.5 of Wainwright [2019], we have with probability 1 -δ ,

which finishes the proof.

Finally, Theorem 4.4 is derived by plugging in the definition of S .

Theorem I.4 (Restate Theorem 4.4) . Under the same conditions in Corollary 4.2, for any interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that g ( x ) ≥ c for all x ∈ I and f is optimized over I , i.e. ∑ x i ∈I ( f ( x i ) -y i ) 2 ≤ ∑ x i ∈I ( f 0 ( x i ) -y i ) 2 , if the output stable solution θ satisfies ∥ θ ∥ ∞ ≤ ρ (for some constant ρ > 0 ) and the ground truth f 0 ∈ BV (1) ( kρ 2 , 1 c ˜ O ( 1 η + σx max )) , then with probability 1 -δ (over the random noises { ϵ i } ), the function f = f θ satisfies

where n I is the number of data points in D such that x i ∈ I .

Proof of Theorem I.4. Note that BV (1) ( kρ 2 , 1 c ˜ O ( 1 η + σx max )) is a subset of T . Then the proof directly results from Lemma I.3 and S = ˜ O ( 1 η + σx max ) .

I.1 The Improved Results for the Under-parameterized Case

We assume that n/k is large enough such that the additional term in the TV (1) bound vanishes.

Assumption I.5. Weassume that n k is large enough such that the last term in (82) ˜ O ( σx max √ k/n ) ≤ 1 2 , which further implies that ∫ x max -x max | f ′′ ( x ) | g ( x ) dx ≤ 1 η , where g ( x ) is defined as (6).

Assumption I.5 requires that n k is larger than some constant, which naturally holds if k = n 1 -α for some α > 0 and n is sufficiently large. Under such assumption, we improve the MSE upper bound.

Theorem I.6. Under the same conditions in Corollary G.9, assume that Assumption I.5 holds. For any interval I ⊂ [ -x max , x max ] and a universal constant c > 0 such that g ( x ) ≥ c for all x ∈ I and f is optimized over I , i.e. ∑ x i ∈I ( f ( x i ) -y i ) 2 ≤ ∑ x i ∈I ( f 0 ( x i ) -y i ) 2 , if the output stable solution θ satisfies ∥ θ ∥ ∞ ≤ ρ (for some constant ρ > 0 ) and the ground truth f 0 ∈ BV (1) ( kρ 2 , 1 cη ) , then with probability 1 -δ (over the random noises { ϵ i } ), the function f = f θ satisfies

where n I is the number of data points in D such that x i ∈ I .

Proof of Theorem I.6. The proof is identical to Theorem I.4, with S replaced by 1 η .

Remark I.7. Compared to Theorem 4.4, Theorem I.6 is better on the dependence of η by removing the additional term σx 2 max . Such improvement results from the improved TV (1) bound (Corollary G.9) and the fact that n/k is sufficiently large.

J Twice-Differentiable Interpolating Solution with Noisy Labels must be 'Sharp'

Recall that in the counter-example, we fix x i = 2 x max i n -1 -( n +1) x max n -1 for i ∈ [ n ] and f 0 ( x ) = 0 for any x , which implies that y i 's are independent random variables from N (0 , σ 2 ) .

Proposition J.1. In the example above, assume f = f θ is an interpolating solution where L is twice differentiable at θ , then with probability 1 -δ , we have

Proof of Proposition J.1. According to Theorem 3.1, with probability 1 -δ , we have

where g ( x ) is defined in (6). Meanwhile, note that f θ is an interpolating solution, and therefore

Finally, combining the results and applying Lemma G.1, it holds that

which finishes the proof.

K Technical Lemmas

Lemma K.1 (Lemma F.4 in Dann et al. [2017]) . Let F i for i = 1 , · · · be a filtration and X 1 , · · · , X n be a sequence of Bernoulli random variables with P ( X i = 1 | F i -1 ) = P i with P i being F i -1 -measurable and X i being F i measurable. It holds that

LemmaK.2 (Covering Number of Euclidean Ball [Wainwright, 2019]) . For any ϵ > 0 , the ϵ -covering number of the Euclidean ball in R d with radius R > 0 is upper bounded by (1 + 2 R ϵ ) d .

Lemma K.3 (Hoeffding's inequality [Sridharan, 2002]) . Suppose X 1 , X 2 , · · · , X n are a sequence of independent, identically distributed (i.i.d.) random variables with mean 0 . Let ¯ X = 1 n ∑ n i =1 X i . Suppose that X i ∈ [ -b, b ] with probability 1, then with probability 1 -δ ,

LemmaK.4 (Multiplicative Chernoff bound [Chernoff, 1952]) . Let X be a Binomial random variable with parameters p, n . Then for any δ ∈ [0 , 1] , it holds that:

The lemmas above are also applied in Qiao et al. [2022], Qiao and Wang [2023a,c], Qiao et al. [2023], Qiao and Wang [2023b, 2024], Zhao et al. [2022], Xu et al. [2023].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the 'Disclaimers and limitations' part in page 3.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The assumptions are stated in the theorems while the proof is stated in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details can be found in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will release the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details can be found in Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Since we use gradient descent, everything is deterministic once the initialization of parameters is finished.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: All experiments are run on a mac book.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a paper regarding theory of neural networks, where we believe there is no possible negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the implicit regularization of large learning rates in gradient descent. The setting is univariate linear regression with two-layer ReLU neural networks. The authors show that, if GD converges to a local minimum, then the function implemented by the neural network at this local minimum has a bounded first-order total variation. This regularity avoids overfitting, as quantified by a generalization bound. Numerical experiments validate the results.\n\nSTRENGTHS:\nUnderstanding the implicit regularization of optimization algorithms for deep learning is a key topic. This paper studies a fairly realistic setting, which does not necessitate interpolation. To my knowledge, the results are novel and go beyond what was already known on the impact of large learning rates (edge of stability, minima stability). Most previous works focus on parameter space and not function space. The interpretation in terms of function space proposed in the present paper is interesting. The paper is very clearly-written. While I have not read the proofs, the mathematical presentation of the results in the main text is very precise.\n\nWEAKNESSES:\nThe paper considers a univariate case, some comments on the extension to the multivariate case are given at the end of Section 1.1.\n\nQUESTIONS:\nDo the authors have any insight on the role of overparameterization in the setting they consider? It does not seem to have a large influence on the generalization bounds, but perhaps it would help the optimization (e.g., via a PL inequality)?\n\nMinor remarks:\n- lines 132: the connection between stepsize and L1/L2 regularization has been thoroughly analysed for diagonal linear networks in [2].\n- lines 147-153: a relevant theoretical paper on the Edge of Stability is [1].\n\n[1] Damin, Nichani, Lee, Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability, ICLR 2023.\n\n[2] Even, Pesme, Gunasekar, Flammarion. (S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability. NeurIPS 2023.\n\nLIMITATIONS:\nLimitations are properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-28"}

comments: SUMMARY:
This paper studies the generalization properties of two-layer ReLU neural networks in a univariate nonparametric regression problem with noisy labels. It proposes a new theory for local minima to which gradient descent (GD) with a fixed learning rate $\eta$ stably converges. The paper shows that GD with a constant learning rate can only find stable local minima whose weighted TV(1) is bounded by $1/\eta-1/2+\tilde{O}(\sigma+\sqrt{\mathrm{MSE}})$. With this property of local minimas, they prove the generalization bound for univariate nonparametric regression. The theoretical results are validated by extensive simulations demonstrating that large learning rate training induces sparse linear spline fits.

STRENGTHS:
This paper gives an end-to-end analysis of the generalization of two layer ReLU networks on learning nonparametric univariate functions. The completeness of the work is significant. The theoretical analysis is solid and the proof is well-organized. The authors also present extensive experiments to support their theoretical findings. In general, it is a good paper.

WEAKNESSES:
1. The analysis in this paper has little focus on optimization. There is no rigorous theoretical evidence that GD will find the solutions that satisfies the assumptions (though I know this is an open problem in the literature).
2. This paper is limited to univariate functions, which is not important in practice. The authors claims that the technique can be generated to multivariate functions but they did not show that.

QUESTIONS:
How large is the family of target functions satisfying the BV(1) condition?

LIMITATIONS:
The authors have addressed some limitations of their work, acknowledging that their analysis is only for full batch gradient descent and univariate nonparamatric regression.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies uni-variate regression with two layer networks and builds on the following observation: If the basin (derived form a quadratic approximation) around a given minimum is too narrow, gradient descent with a fixed step size $\\eta$ will escape it, only sufficiently wide basins can capture the iterates of GD with large step sizes. It is thus reasonable to restrict the function space attainable by GD to the set of functions that can be expressed by a minimum with a wide basin. By extending a previous result to the noisy-label setting, they show that the width of the basin (or the hessian of the loss) relates to a (weighted) measure of the TV norm of the function encoded by the minimum. The function space of interest thus becomes a space functions with low TV norm. By computing covering numbers of this space, uniform generalization bounds are derived for this restricted space of functions.\n\nSTRENGTHS:\nThe paper is very well written. The proofs in the appendix section are also well written and easy to follow.  The minima studied are non-interpolating, unlike the minima studied by a large portion of prior literature (although this work is in the univariate setting). Moreover, although the fact that large stepsizes bias towards solutions with fewer knots appears in prior work, this paper explores this interesting fact further in the noisy-label setting. The paper provides some avenues to understanding generalization when over-fitting is not benign.\n\nWEAKNESSES:\n- *The \"optimized\" assumption*: This assumption which effectively assumes that there exists wide and low minima is somewhat justified in lines 267-270, but I believe it is reasonable to disagree with the statement that it is mild. The experiments provided as evidence for the assumption are conducted with really large noise levels sigmas that are of the same order as $f$. The authors empirical argument could be strengthened by considering noise levels that do not dwarf the signal. It seems difficult to believe that a smooth $f$ can attain a training loss that is lower than $f_0$ when no assumption on the smoothness of $f_0$ is made. The authors should clarify why a condition on $f_0$ is not necessary for this assumption to be mild.\n\n- The refined bounds with underparametrization: The bounds are refined by considering under parametrized settings with the width $k$ being smaller than the number of data points but the authors do not discuss its impact on the \"optimized\" assumption. Why would an underparametrized network be able to attain train losses smaller than a ground truth on which no assumptions are made ? I believe there are trade-offs between $\\eta$, the level of underparametrization and the attained train loss that are not mentioned sufficiently by the authors.\n\n- A final minor weakness: The work extends [Mulayoff et al 2021] in limited ways but does not exploit the added generalization to derive new conclusions. That stepsize affects the number of knots was explored before. There is now a sigma appearing in the smoothness bounds and an MSE term, yet there are no comments on what is added by their extension. The interplay between sigma and eta is not discussed, for instance, does having a large eta  but high noise means that stable functions need no have a low TV norm or is the upper bound loose?  I believe such discussions could strengthen their work as some results, namely Thm 4.3, are straightforward applications of generalization bounds for  spaces with known metric entropy.\n\nQUESTIONS:\nThe weight function $g$ is inherited from [Mulayoff et al 2021]  but would really benefit from some more clarifications, even for completeness. It is clear that the interval $\\mathcal{I}$ must be introduced to remove the weighing and allow the authors to use covering numbers for bounded TV norm spaces. Could the authors explain why it arises in theorem 4.1 ?\n\nLIMITATIONS:\nThe limitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the generalizability of stable local minima in univariate regression with shallow ReLU networks. Along the way, the authors provide a bound on a weighted total variation norm of networks corresponding to stable solutions which in turn provide a tighter generalization bound.\n\nSTRENGTHS:\nThe paper is well written and provides a novel way to obtain generalization bounds using minima stability theory.\n\nWEAKNESSES:\nIt is not clear to me why the experiments (e.g. figure 3) corroborate the theoretical results in the paper. \n\nTo illustrate my point, suppose for a given $\\eta > 0$, a solution $\\theta^*$ is linearly stable. \nNow suppose one trains a ReLU NN with a different learning rate $\\alpha > 0$ and it converges to $\\theta^*$. When considering the generalization gap of the solution $\\theta^*$ (e.g. theorem 4.4), only the constant $\\eta$ is relevant. The point is that the actual learning rate of GD that was used to find stable solutions is irrelevant to the generalizability of the solution (at least in the context of theorem 4.4) . Hence, I feel that more compelling experiments would study generalization errors of stable local minima that are stable for varying $\\eta$. \n\nIn this vein, I feel that the statement, “Meanwhile, the dependence $\\frac 1 {\\eta}$ ...” (line 296 -298) to be inexact as it is only supported by the experiments.\n\nQUESTIONS:\nIn the context of theorem 4.3 and 4.4, should $n_I$ be the number of data points belonging to the interval $I \\subset [-x_{max}, x_{max}|$ instead of the length of the interval $|I|$? Otherwise, the RHS of the bound in theorem 4.4 seems to be non-vanishing as $x_{max} $ is at least on the order of the length of the interval $|I|$.  \n\nCan the authors also provide some intuition on how the weight function $g$ was chosen?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}