Title: Minimum norm interpolation by perceptra: Explicit regularization and implicit bias
Venue: NeurIPS 2023
Accepted: yes
Keywords: Artificial neural network

interpolation

explicit regularization

implicit bias

weight decay

Barron class
Abstract: We investigate how shallow ReLU networks interpolate between known regions. Our analysis shows that empirical risk minimizers converge to a minimum norm interpolant as the number of data points and parameters tends to infinity when a weight decay regularizer is penalized with a coefficient which vanishes at a precise rate as the network width and the number of data points grow. With and without explicit regularization, we numerically study the implicit bias of common optimization algorithms towards known minimum norm interpolants.

Minimum norm interpolation by perceptra: Explicit regularization and implicit bias

Abstract

We investigate how shallow ReLU networks interpolate between known regions. Our analysis shows that empirical risk minimizers converge to a minimum norm interpolant as the number of data points and parameters tends to infinity when a weight decay regularizer is penalized with a coefficient which vanishes at a precise rate as the network width and the number of data points grow. With and without explicit regularization, we numerically study the implicit bias of common optimization algorithms towards known minimum norm interpolants.

1 Introduction

Modern neural networks mostly operate in an overparametrized regime, i.e. they possess more tunable parameters than the number of data points contributing to the loss function. Safran and Shamir [2018a], E et al. [2019b], Du et al. [2018], Chizat and Bach [2018] associate overparametrization with better training properties, and Belkin et al. [2019, 2020] find it to enhance statistical generalization (see also [Loog et al., 2020] for historical context). For many architectures, overparametrization leads to the ability to fit any values y i at a given set of data points { x 1 , . . . , x n } , and Cooper [2018] shows that generically, the set of weights for which the neural network interpolates prescribed values is a submanifold of high dimension and co-dimension in parameter space. Which solution on the manifold is dynamically chosen by an optimization algorithm, and which solutions have favorable generalization properties, is an active area of research in theoretical machine learning.

Current practice is to estimate a model's generalization to previously unseen data by assessing its performance on a hold-out set (a posteriori error estimate) or by using uniform estimates on the generalization of all elements of a function class (a priori estimate if the function class is encoded ahead of time by explicit regularization, a posteriori if membership is determined after optimization). Neither approach yields information on what a neural network does outside the support of the data distribution, a topic of great interest for the study of distributional shift and adversarial stability.

The main contribution of this work is split between two complementary lines of investigation:

We prove that neural network minimizers of regularized empirical risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit (Section 2). Our result improves previous works to more general settings (See Remark 2.2).

Training a neural network generally corresponds to solving a non-convex minimization problem. While we provide convergence guarantees for empirical risk minimizers, in general there is no guarantee that a training algorithm finds a global minimizer of an empirical risk functional. Even if convergence holds, it is unclear which minimizer is selected (in the overparametrized regime, where the set of minimizers is a high-dimensional manifold). In settings where the minimum norm interpolant is known (Section 3), we compare numerical solutions to theoretical predictions to better understand (1) the predictive

power of theoretically studying empirical risk minimizers and (2) the implicit bias of different optimization algorithms (Sections 4 and 5). We believe this to be a useful benchmark problem for a better understanding of explicit regularization and implicit bias in optimization.

Minimum norm interpolants are the regression analogue to maximum margin classifiers. They are associated with favorable generalization properties and relative stability even against adversarial perturbations.

1.1 Previous work

Minimum norm interpolation. In classes of linear functions, minimum norm interpolation has a long history as ridge regression (minimal ℓ 2 -norm) or as the least absolute shrinkage selection and operator (LASSO, minimal ℓ 1 -norm). To the best of our knowledge, minimum norm interpolation by neural networks has only been studied for shallow neural networks in one dimension by Hanin [2021], Debarre et al. [2022], Boursier and Flammarion [2023] and in odd dimension for certain radially symmetric data by Wojtowytsch [2022]. For classification problems, minimum norm/maximum margin classifiers were considered by E and Wojtowytsch [2022]. For finite datasets, the set of minimum norm interpolants was characterized by Parhi and Nowak [2021]. A parametrization of the same function class by neural networks with multiple linear layers and a single ReLU layer induces different concepts of minimum norm interpolation as studied by Ongie and Willett [2022].

Implicit bias. The implicit bias of parameter optimization algorithms has been studied for gradient flows with infinitely wide, but shallow ReLU networks by Chizat et al. [2019], Jin and Montúfar [2020] and for stochastic gradient descent and diagonal linear networks by Pesme et al. [2021]. Chizat and Bach [2020] prove convergence to a maximum margin classifier for infinitely wide ReLU networks with one hidden layer if the parameters follow the gradient flow of an (unregularized) logistic loss risk functional. Many authors, including Damian et al. [2021], Li et al. [2021], Wu et al. [2022] and Wojtowytsch [2020], study the bias of SGD towards solutions at which the loss landscape is 'flat' in the parameter space. Hochreiter and Schmidhuber [1997] conjectured such minimizers to have favorable generalization properties. In many cases, minimizers tend to be flatter if the parameters associated to them are not excessively large. Yang et al. [2021] describe several phase-transitions in parameter space for the relation between flatness in parameter space and generalization. Zhou et al. [2020] compare the implicit bias of SGD and ADAM. Smith et al. [2021], Barrett and Dherin [2020] argue that gradient descent resembles a gradient flow in a modified (regularized) loss landscape more closely than the gradient flow of the original loss function. Hajjar and Chizat [2022] investigate the role of symmetries in optimization.

Barron space. The Barron class is adapted to ReLU networks with a single hidden layer and weights of bounded average magnitude. Slightly different versions of the same function space have been studied by Bach [2017], E et al. [2019a,c], Ongie et al. [2019], E and Wojtowytsch [2020, 2021], Caragea et al. [2020], Parhi and Nowak [2021], Wojtowytsch [2022], Siegel and Xu [2022, 2023] under various names such as F 1 , Radon-BV or the variation space of the ReLU dictionary.

1.2 Preliminaries

Conventions. µ always denotes a σ 2 -sub-Gaussian probability measure on the data domain R d , i.e.

̸

All norms are Frobenius norms ( ℓ 2 -norm for vectors). For n ∈ N , S n = { x n, 1 , . . . , x n,n } is a set of n iid samples from the distribution µ , independent of S n ′ for n ′ = n . When unambiguous, we denote x n,i = x i . We take ℓ ( f, y ) = | f -y | 2 as the mean squared error/ ℓ 2 -loss function, but we remark that the theoretical analysis remains valid if ℓ MSE is replaced by ℓ 1 -loss or a Huber or pseudo-Huber loss

For m ∈ N and ( a, W, b ) ∈ R m × R m × d × R m +1 , let

i.e. f ( a,W,b ) is a ReLU network with a single hidden layer and weights a, W and biases b . The vector w i ∈ R d is the i -th row of the matrix W . For m,n ∈ N and λ ≥ 0 , we denote the regularized empirical risk functional as ̂ R n,m,λ : R m × R m × d × R m +1 → [0 , ∞ ) :

Concepts. We introduce what we dub 'homogeneous Barron space' B heuristically here, and in greater detail in Appendix C. As a measure of magnitude of the function, we consider the weight decay (or Tikhonov) regularizer 1 2 ( ∥ a ∥ 2 2 + ∥ W ∥ 2 Frob ) of the parametrized function f ( a,W,b ) which does not control the magnitude of the bias vector.

Note that the function class f ( a,W,b ) and its complexity do not change when we consider representations of the form f m ( x ) = 1 m ∑ m i =1 a i σ ( w i · x + b i ) with a regularizer 1 2 m ( ∥ a ∥ 2 2 + ∥ W ∥ 2 Frob ) = 1 2 m ∑ m i =1 ( a 2 i + ∥ w i ∥ 2 2 ) . A continuum analogue to these functions represented as an 'empirical average' over individual neurons is a general expectation representation f π ( x ) = E ( a,w,b ) ∼ π [ aσ ( w T x + b ) ] for some probability distribution π on parameter space and a regularizer 1 2 E ( a,w,b ) ∼ π [ a 2 + ∥ w ∥ 2 2 ] . As the parametrization of a function by a neural network is generally non-unique, we define the Barron semi-norm [ f ] B = inf { π : f π ≡ f } E ( a,w,b ) ∼ π [ a 2 + ∥ w ∥ 2 2 ] as the lowest value attained by the regularizer over all possible parametrizations. For a more comprehensive understanding, interested readers may refer to Appendix C.

2 General convergence result

We first state a general convergence result to a minimum norm interpolant of given data generated by functions in the homogeneous Barron class B .

Theorem 2.1. Take µ, S n , ̂ R n,m,λ as in Section 1.2, f ∗ ∈ B , and let y i = f ∗ ( x i ) for i = 1 , . . . , n . Assume that m,λ are parameters which scale with n as m n , λ n such that

Then almost surely over the random selection of data points in S n , the following holds: If ( a, W, b ) n ∈ argmin ̂ R n,m n ,λ n for all n ∈ N , then every subsequence of f n := f ( a,W,b ) n has a further subsequence which converges to some limit ˆ f ∗ ∈ B with ˆ f ∗ = f ∗ µ -almost everywhere and [ ˆ f ∗ ] B ≤ [ f ∗ ] B . Convergence holds in L p ( µ ) for all p < ∞ and uniformly on compact subsets of R d . If E µ ∥ x ∥ + σ 2 ≥ 1 , then for all n ≥ 2 , the following explicit bound holds up to higher order terms in n, m = m n , λ = λ n with probability at least 1 -1 /n 2 :

Remark 2.2. Theorem 2.1 extends previous results of E et al. [2019a] in several ways:

We allow for general sub-Gaussian rather than compactly supported data distributions.

We do not control the magnitude of the bias variables.

Our results apply to ℓ 2 -loss, which is neither globally Lipschitz-continuous nor bounded.

In a limiting regime, we characterize how the empirical risk minimizers interpolate in the region where no data is given by proving uniform convergence to a minimum norm interpolant.

In sum, the first three points necessitate a more careful technical analysis than in the settings of prior work, and the unboundedness of data and loss introduces additional logarithmic terms not present for E et al. [2019a]. To the best of our knowledge, the last point is novel and our work is the first to use the notion of Γ -convergence in this context.

If the data-distribuion µ is supported on the entire space R d (e.g. a non-degenerate Gaussian), then ˆ f ∗ ≡ f ∗ . In many cases, however, µ is supported on a small, potentially compact and generally low-dimensional subset M of the data space. In this case, the function f ∗ is only known on the closed set M ⊊ R d . As a result, there are many f ∈ B such that f ≡ f ∗ on M while f ̸≡ f ∗ on R d in general. The subsequential limit is one of these functions which has a minimal semi-norm [ f ] B .

Thus, beyond knowing that f n asymptotically fits the function f ∗ perfectly at known data, Theorem 2.1 provides information about how it may interpolate at points where µ has no information. Such knowledge is of interest when a population may naturally evolve in time (distributional shift) of if f n is applied to a new problem with similar features but distinct geometry (transfer learning).

The proof of Theorem 2.1 is given in Appendix E. In the proof, we combine a direct approximation theorem to construct risk competitors with Rademacher complexity-based generalization bounds. Concentration inequalities are used to bound tail quantities. The scaling conditions that log n √ n ≪ λ and 1 m ≪ λ are used to ensure that the risk functional has sufficiently strong regularization. They can be weakened if the data measure µ is supported on a finite set of points or the function f ∗ can be represented by a neural network with a finite number of neurons respectively. For instance, an analogous statement holds with a simpler proof for a fixed data-set S of n data points if λ is coupled to m such that 1 mλ m → 0 . In this case, we take the empirical distribution µ = 1 n ∑ x ∈ S δ x as the population and do not need to bound the generalization gap. This model can be considered appropriate when n ≪ m (heavy overparametrization). A precise statement is given in Appendix F.

The proof remains valid if we only assume that

i.e. if ( a, W, b ) n parametrizes a function of low excess risk. In the proof, we obtain a more precise version of (2). Using an a priori Lipschitz-bound, it is possible to obtain a rate of convergence in L p ( µ ) for p < ∞ by interpolation. For uniform convergence on compact sets outside the support of µ , we do not obtain a rate in this work.

The proof of uniform convergence on compact sets utilizes the notion of Γ -convergence from the calculus of variations, a very stable notion of convergence which implies that minimizers of approximating functionals converge to the minimizer of a limiting functional. This notion has recently made inroads into machine learning applications and was used e.g. by Neumayer et al. [2023].

All results can easily be generalized to any more general function class which admits the three key ingredients: A bound on its Rademacher complexity, a compact embedding theorem, and a direct approximation theorem.

3 Minimum norm interpolants

3.1 One-dimensional example

In one dimension, [Wojtowytsch, 2022, Proposition 2.5] shows that [ f ] B = ∫ ∞ -∞ | f ′′ ( x ) | d x for any smooth function f : R → R which satisfies f ′ ( x ) = 0 at some point x ∈ R - see also previous work by Li et al. [2020], E and Wojtowytsch [2020]. This one-dimensional case is, in fact, the simplest case of a general characterization of Barron functions in any dimension by Ongie et al. [2019].

Consider the task of minimizing [ f ] B under the condition that f ( x ) = | x | if | x | ≥ 1 . Then the minimum is attained for any smooth convex function f which satisfies the constraint since

with equality if and only if f ′′ ≥ 0 . The same estimate holds for non-smooth Barron functions if the second derivative is interpreted as a Radon measure. For piecewise linear functions, this corresponds to summing | f ′ ( x + i ) -f ′ ( x -i ) | over the non-smooth points x i . More generally, the set of minimum norm interpolants of one-dimensional convex data was characterized by Savarese et al. [2019].

Proposition 3.1. Let x 0 < · · · < x n and y i = f ∗ ( x i ) for a convex function f ∗ and i = 0 , . . . , n . If y 1 < y 0 and y n > y n -1 , then f is a minimum Barron norm interpolant of the dataset { ( x i , y i ) } n i =0

if and only if f is convex, f ( x i ) = y i for all i = 0 , . . . , n and

The two given slopes are the largest values that are required for derivatives at any point. We give a proof of Proposition 3.1 in Appendix G. In full generality, minimum norm solutions have been characterized by Hanin [2021] using matching convexities to achieve minimal total curvature.

In a recent article, Boursier and Flammarion [2023] show that if the full Barron norm is controlled, i.e. if the magnitude of biases is included in the regularizer, a specific convex function is selected. This corresponds to minimizing a functional

The first functional prefers f ′′ to be large close to the origin, if it has to be large anywhere. All break points occur as close to the origin as possible, in particular: Either at the origin or at data points. Unlike the Barron semi-norm penalty studied in this work, which does not select a specific minimum norm interpolant, the Barron norm penalty thus induces a sparse neural network interpolant.

3.2 Radially symmetric bump function

Another setting where we have explicit minimum norm interpolant is when we fit a bump function for radially symmetric data. Recall a result of Wojtowytsch [2022] on minimum norm fitting of certain radially symmetric data.

Proposition 3.2. [Wojtowytsch, 2022, Theorem 3.1] Let d ≥ 3 be an odd integer and

Then there exists a unique radially symmetric function f ∗ d : R d → R such that f ∗ d ∈ argmin f ∈F [ f ] B . The norm of minimizers grows as lim d →∞ [ f ∗ d ] B /d ≈ 3 . 7 .

We note that the existence of minimum norm interpolants which are not radially symmetric is not excluded, but if ˆ f ∈ argmin f ∈F [ f ] B is any other minimum norm interpolant, then its radial average Av ˆ f coincides with f ∗ d : Av ˆ f ≡ f ∗ d . Here

where H is the uniform distribution (Haar measure) on the group of rotations and π 0 is the uniform distribution on the d -1 -dimensional sphere in R d . In [Wojtowytsch, 2022, Section 6], an algorithm is given to find the minimum norm interpolant f ∗ d by numerically solving a one-dimensional polynomial approximation problem and a linear system of moment conditions.

The uniqueness statement allows us to strengthen the result of Theorem 2.1 in this case. A natural setting is to use a sub-Gaussian data distribution µ which gives positive mass to the origin, but has no mass elsewhere in the unit ball. It should have mass everywhere outside the unit ball. Under this natural setting, we have a stronger result of Theorem 2.1.

Corollary 3.3. Take µ, S n , ̂ R n,m,λ as in Section 1.2, and assume in addition that

µ ( { 0 } ) > 0 (positive mass at the origin).

µ ( B 1 (0) \ { 0 } ) = 0 (no mass elsewhere in the unit ball).

µ ( U ) > 0 for any open set U ⊆ R d \ B 1 (0) (mass everywhere outside the unit ball).

Assume that m,λ scale with n as in (1) . Almost surely over the random selection of data points in S n , the following holds: If ( a, W, b ) n ∈ argmin ̂ R n,m n ,λ n for all n ∈ N , then sequence of radial averages Av f n of f n := f ( a,W,b ) n converges to f ∗ d as in Proposition 3.2. Convergence holds in L 2 ( µ ) for MSE loss (with an explicit rate) and uniformly on compact subsets of R d (without a rate in this work).

In Corollary 3.3, we guarantee convergence to the unique radial minimum norm interpolant f ∗ d (at least for the radial average), while in Theorem 2.1 we may have different subsequences that converge to different minimum norm interpolants. The proof is given in Appendix E.

4 Relating Interpolation, Optimization and Generalization

For a given bounded set K ⊆ R d , Theorem 2.1 states that for a large number of neurons m ∈ N , a large number of data points n ∈ N , and a small penalty λ > 0 , minimizers of the empirical risk functional ̂ R n,m,λ resemble a minimum norm interpolant everywhere in K . As the Barron semi-norm controls the generalization gap (see Appendix D) and a minimum norm interpolant has minimal Barron norm by definition, this suggests that minimum norm interpolants are optimal in terms of generalization, at least when arguing from this upper bound.

Many authors, including Safran and Shamir [2018b], Venturi et al. [2018], demonstrate that neural network training is a non-convex optimization problem. As such, it is not guaranteed that numerical optimizers (1) converge to interpolants at all, and (2) select minimum norm interpolants out of the large set of different neural networks which interpolate given data, even when a regularizer is included in the training loss functional.

On the other hand, there are settings where an optimization algorithm selects a minimum norm solution even without explicit regularization. This is easily proved for gradient descent on the overparametrized least squares regression problem R n ( a ) = 1 n ∑ n i =1 | a T x i -y i | 2 with initial condition a = 0 and n < m = d . Using entirely different methods, Chizat and Bach [2020] prove a similar result for binary classification by shallow neural networks with logistic loss. For regression problems using neural networks, analogous results are not available to the best of our knowledge. This in part motivates the following numerical investigation. Namely, we are interested in exploring the effects of explicit regularization and the implicit bias of optimization algorithms toward minimum norm interpolants. Knowing the analytically optimal solution in between given data provides us the opportunity to compare optimizers on a deeper level than merely testing their performance on unseen data generated from the same distribution.

As seen in Figure 1, the radial profile r ↦→ f ∗ d ( re 1 ) of Wojtowytsch [2022]'s minimum norm interpolant f ∗ d is so close to 0 on [ r d , ∞ ) as to be virtually indistinguishable from zero numerically for some r d < 1 which decreases in d . Indeed, the first ( d -1) / 2 derivatives of vanish at r = 1 due to [Wojtowytsch, 2022, Lemma 4.1] and 0 ≤ f ∗ d ( x ) ≤ Cd 3 / 2 ((1 - ∥ x ∥ 2 ) / ∥ x ∥ ) ( d -3) / 2) due to [Wojtowytsch, 2022, Appendix D.1] for a universal constant C > 0 . In particular, if d is large, ∥ f ∗ d ∥ L ∞ ( R d \ B r d (0)) is negligible compared to the approximation error d 2 /m for any reasonable dimension d and network width m . Consequently, the rescaled function h ∗ d ( x ) := f ∗ d ( r d x ) meets the constraint h ∗ d ≡ 0 outside B 1 (0) almost exactly and has the smaller Barron semi-norm [ h ∗ d ] B = r d [ f ∗ d ] B . For this reason, we compare numerical solutions to the interpolation problem in Corollary 3.3 to rescaled versions of f ∗ d rather than f ∗ d itself, at least in high dimension. For r below the threshold value r d , there is no noticable trade-off between rescaling f ∗ d ( rx ) and data-fitting. For larger values of r , the Barron semi-norm is reduced more significantly, but the data fit becomes appreciably worse.

5 Numerical Experiments

Our main goal in this section is to gain a more precise understanding of different optimization algorithms by comparing numerical solutions to a known minimum norm interpolant. We consider the two settings in which minimum norm interpolation by Barron functions is best understood: Onedimensional and radially symmetric functions. As a benefit, we can easily visualize the numerical results in both settings. We focus on three questions of interest.

Explicit regularization. If λ > 0 is moderately small and m,n are large, then a global minimizer of ̂ R n,m,λ resembles a minimum norm interpolant between known data points due to Theorem 2.1. Is the minimizer which we find numerically close to to a minimum norm interpolant, or does it merely fit the function at known data points?

Implicit bias. If m,n are large, does a training algorithm select a minimum norm interpolant out of potentially many possible solutions without explicit regularization (i.e. for λ = 0 )?

Minimumnorminterpolants

Minimumnorminterpolants

1.0

optimal interpolation, dim 3

1.0

dim3,rescaled withr_d=0.95

optimal interpolation, dim 31

dim 15,rescaled withr_d =0.4

0.8

optimal interpolation, dim 31,rescaled with r_d = 0.4

0.8

dim31,rescaled with r_d=0.28

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

0.00

0.25

0.50

0.75

1.00

1.25

1.50

1.75

Learning symmetries: The optimal minimum norm interpolant f ∗ d described in Proposition 3.2 is radially symmetric and satisfies 0 ≤ f ∗ d ≤ 1 . Proposition 3.2 does not rule out the existence of other minimum norm interpolants which are not radially symmetric. Does an optimization algorithm generally find solutions which are (approximately) radially symmetric and confined to the interval [0 , 1] ? A similar consideration applies in a onedimensional investigation with reflection symmetry.

The third question is of particular interest for algorithms like ADAM, which operate coordinate-bycoordinate and do not respect Euclidean isometries. By comparison, we expect that SGD, initialized at a radially symmetric configuration, preserves Euclidean isometries. More experiments in similar settings can be found in Appendix A.

5.1 One-dimensional experiments

We consider the classical interpolation problem of numerical analysis: Fit values f ∗ ( x i ) ∈ R at points x i ∈ R for i ∈ { 1 , . . . , n } . In contrast to classical numerical analysis, we consider overparametrized ReLU-networks with a single hidden layer as our model class. As in Section 3, we select f ∗ ( x ) = | x | for simplicity.

In Figure 2, a ReLU network with a single hidden layer of width m = 200 was trained to fit f ∗ ( x ) = | x | at a symmetric set containing 15 equi-spaced points in (1 , 2) . Optimizers included SGD (with learning rate η = 5 · 10 -5 and momentum µ = 0 . 99 ), SGD ( η = 10 -2 , µ = 0 ), ADAM ( η = 5 · 10 -5 and default parameters) and the quasi-Newton L-BFGS method. Deterministic gradients based on the n = 30 sample points were used. The final training loss was below 10 -4 on average. The network weights were initialized by a scaled uniform Xavier initialization, i.e. uniformly in a symmetric interval of length 2 α · √ 6 / ( n in + n out ) where n in and n out denote the number of input- and output-units to a layer respectively. The 'gain' factor was selected as α ∈ { 0 . 5 , 1 , 5 } . Without weight decay and for small gain, the optimizers find a solution close to the smallest possible minimum norm interpolant f ( x ) = | x | . The larger the parameters for initialization gain and weight decay penalty, the closer numerical solutions are to the largest possible minimum norm interpolant f ( x ) = max {| x | , 1 } .

We observe that a higher gain factor α corresponds to faster initial training, but a high gain like α = 5 produces interpolants which are non-convex without regularization, while a lower gain factor produces convex interpolants in longer time. This observation agrees with the findings of Chizat et al. [2019], who dub the large α setting the 'lazy training' regime and associate it with worse generalization performance. As Pesme et al. [2021] eloquently put it: 'there is a tension between generalisation and optimisation: a longer training time might improve generalisation but comes at the cost of. . . a longer training time.'

If m is large and α is not too big, the variation of solutions produced by a training algorithm vary less over different stochastic realizations - see Appendix A for experiments for m = 1 , 000 . The

gain =0.5,width=200,lambda=0.0

gain =1.0,width = 200,lambda = 0.0

gain =5.0, width =200,lambda =0.0

2.5

2.5

2.5

2.0

2.0

2.0-

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.0

-2

1

0

1

Z

0.0

-2

-1

0

1

2

0.0

-2

-1

0

2.5

gain =0.5,width =200,lambda =0.002

2.5

gain=1.0,width =200,lambda =0.002

2.5

gain=5.0,width =200,lambda =0.002

2.0

2.0

2.0

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.5

0.0

-2

-1

1

0.0

-2

-1

0

1

2

0.0

2

-1

1

2.5

gain = 0.5,width =200,lambda=0.005

2.5

gain =1.0, width =200,lambda =0.005

2.5

gain =5.0,width = 200,lambda = 0.005

2.0

2.0

2.0

1.5

1.5

1.5

1.0

1.0

1.0

0.5

0.5

0.5

0.0

0.0

0.0

-2

-1

-2

-1

-2

-1

For small gain, the optimizer selects a minimum norm interpolant/convex function. For large gain, the interpolant is often non-convex for any one of the optimization algorithms, unless the weight decay is given a positive value. A specific type of minimum norm interpolant in the large set of possible solutions seems to be selected by specifying optimization algorithm, weight decay and initialization. Evidently, initialization and weight decay have far greater influence than the choice of the optimizer. Small gain seems to preference the sparsest solution f ( x ) = | x | has predicted with bias penalty by Boursier and Flammarion [2023]. With larger weight decay, the solutions become more convex and more symmetric, but the accuracy of data fitting decreases. Visually, the second order L-BFGS method appears to be the least affected by different choices in initialization.

dynamics are close to those of a limiting 'mean field' model studied by Chizat and Bach [2018], Rotskoff and Vanden-Eijnden [2018], Mei et al. [2018], Sirignano and Spiliopoulos [2020a] and Wojtowytsch [2020]. In these works, the limiting model is typically derived with a factor 1 /m outside the function definition, which is implicit in the initialization here since n in + n out ≈ m for both layers and the ReLU activation is positively one-homogeneous. Global convergence to a minimizer (but not necessarily a minimum norm solution) is guaranteed (up to certain technical assumptions) by Chizat and Bach [2018] and Wojtowytsch [2020].

For comparison, we also present the natural cubic spline interpolant, i.e. the function f which minimizes the stronger curvature energy ∫ 2 -2 | f ′′ ( x ) | 2 d x under the condition that f ( x i ) = | x i | for all i = 1 , . . . , n . Unlike the minimum Barron norm interpolants, the natural cubic spline may not be convex (and in fact, it is not if f ∗ is replaced by h ∗ ( x ) = | x -0 . 5 | ).

5.2 Radially symmetric data

We explore the performance of numerical optimization algorithms in the setting of Corollary 3.3 with and without explicit regularization λ ∈ { 0 , 10 -5 } in dimensions d = 3 , d = 15 and d = 31 . The numerical solution is then compared to (a rescaled version of) the analytic minimum norm interpolant f ∗ d described in Proposition 3.2, which we compute by the algorithm described in [Wojtowytsch, 2022, Section 6]. The rescaling factor r d is chosen heuristically for an accurate match.

MomentumSGD.10 epochs.Width=12000.106data.Dim3

MomentumSGD.10epochs.Width=12000.106data.Dim15

1.0

numericalsolution

1.0

numericalsolution

optimalinterpolationindim3

optimalinterpolationindim15

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0.0

0.0

0

1

2

3

4

5

0

1

2

3

4

5

Momentum SGD.1000 epochs.Width = 12000.104 data.Dim 15

Momentum SGD.100 epochs. Width = 12000. 10s data.Dim 15

Momentum SGD. 10 epochs. Width = 12000.10° data. Dim 15

1.0

numerical solution

1.0

numerical solution

1.0

numerical solution

0.8

optimal interpolation in dim 15

0.8

optimal interpolation in dim 15

0.8

optimal interpolation in dim 15

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0

0

3

0

m

Data is generated from a distribution µ = µ 1 + µ 2 + µ 3 where µ 1 is a point mass of magnitude m 1 at the origin, µ 2 is a uniform measure on the unit sphere S d -1 with mass m 2 and µ 3 is the radially symmetric measure of mass 1 -m 1 -m 2 such that ∥ x ∥ 2 is distributed uniformly in [1 , 7] . We numerically explored various values for m 1 ∈ [0 . 1 , 0 . 4] and m 2 ∈ [0 . 0 , 0 . 4] and found simulations to be relatively stable under a number of choices.

Results are presented in Figures 3, 4 and Appendix A. We find that all algorithms find a solution with radial average similar to f ∗ d ( r d x ) , albeit for rescaling factors r d which depend on dimension d and (to a lesser extent) the optimizer. In high dimension, solutions are not perfectly radially symmetric, but the larger amount of variation over a sphere of fixed radius is observed in the domain where f ∗ d ≈ 0 rather than in the transition area (0 , r d ) . Larger datasets improve the compliance with the optimal interface and reduce the radial standard deviation. Solutions do not remain non-negative and drop below zero before leveling off as the radius increases. The drop becomes more noticeable as the dimension increases and less pronounced for wider networks.

The results are essentially identical for normal Xavier initialization and (not radially symmetric) uniform Xavier initialization. In accordance with our expectations, the radial standard deviation is higher for Adam compared to optimizers based in Euclidean geometry. While the neural network function found by Adam resembles a minimum norm interpolant, the weight decay regularizer takes significantly higher values compared to other optimization algorithms.

6 Conclusion

Shallow ReLU networks converge to minimum norm interpolants of given data: Provably if explicit regularization is included and empirically if it is not. We conclude with a summary of our empirical insight into the implicit bias of neural network optimizers.

radial average

radial standard deviation

weight decay regularizer

1.0 -

0.10

1600

1400

0.8

0.08

1200

0.4

0.6

Optimal solution

SGD

Adam

0.06

0.04

Momentum SGO

Adam

SGD

1000

800

SGD

Adam

009

0.2

0.02

400

0.0

0.00

200

2

50000

100000

150000

200000

250000

radial average

radial standard deviation

weight decay regularizer

1.0

0.10

350

0.8

0.08

0.6

0.4

Optimal solution

SGD

Adam

0.06

0.04

SGD

Adam

250

300

SGD

Adam

0.2

0.02

200

0.0

150

0.00

50000

100000

150000

200000

250000

With reasonable (not too large) initialization, all algorithms studied here are biased towards minimum norm interpolant profiles.

At least in the case of Adam, this bias is visible on the function level, but not on the parameter level, as the weight decay regularizer increases rapidly to large magnitude. Despite this, ADAM solutions often appear 'flatter' in high dimension with a lower rescaling factor r d .

Explicit regularization stabilizes towards a minimum norm interpolant shape, but at the cost of a decreased fit with the target values. Its impact is most significant for poorly chosen initial conditions.

When the minimum norm interpolant is non-unique, different types of minimum norm interpolants are found depending on the choice of initialization scheme and optimization algorithm. The impact of initialization scale appears more significant.

Optimization algorithms which are rooted in Euclidean geometry (such as SGD and momentum-SGD) more successfully preserve Euclidean symmetries compared to the 'coordinate-wise' Adam algorithm.

The last observation is not surprising for radially symmetric initialization laws as radially symmetric parameter distributions induce radially symmetric functions. It is, however, observed also for a uniform initialization scheme which only obeys coordinate symmetries.

We believe minimum norm interpolation to be a useful testbed to study the implicit bias of optimizers and the impact of initialization and regularization. While minimum norm interpolation by deeper networks has not been characterized yet, we anticipate no obstructions to implementing a similar program there in the future.

References

Stefan Adams. Lecture notes in high-dimensional probability. https://warwick.ac.uk/ fac/sci/maths/people/staff/stefan_adams/high-dimensional_probability_ ma3k0-notes.pdf , 2022.

Nicholas D. Alikakos, Peter W. Bates, and Xinfu Chen. Convergence of the Cahn-Hilliard equation to the Hele-Shaw model. Arch. Rational Mech. Anal. , 128(2):165-205, 1994. ISSN 0003-9527. doi: 10.1007/BF00375025. URL http://dx.doi.org/10.1007/BF00375025 .

Annika Bach, Roberta Marziani, and Caterina Ida Zeppieri. γ -convergence and stochastic homogenisation of singularly-perturbed elliptic functionals. arXiv preprint arXiv:2102.09872 , 2021.

Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research , 18(1):629-681, 2017.

David GT Barrett and Benoit Dherin. Implicit gradient regularization. arXiv preprint arXiv:2009.11162 , 2020.

Heinz Bauer. Minimalstellen von funktionen und extremalpunkte. Archiv der Mathematik , 9(4): 389-393, 1958.

Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias-variance trade-off. Proceedings of the National Academy of Sciences , 116(32):15849-15854, 2019.

Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science , 2(4):1167-1180, 2020.

Kaushik Bhattacharya, Marta Lewicka, and Mathias Schäffner. Plates with incompatible prestrain. Archive for Rational Mechanics and Analysis , 221(1):143-181, 2016.

Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity. arXiv preprint arXiv:2303.01353 , 2023.

Andrea Braides. Γ -convergence for beginners , volume 22 of Oxford Lecture Series in Mathematics and its Applications . Oxford University Press, Oxford, 2002. ISBN 0-19-850784-4. doi: 10.1093/acprof:oso/9780198507840.001.0001. URL http://dx.doi.org/10.1093/acprof: oso/9780198507840.001.0001 .

Andrea Braides and Lev Truskinovsky. Asymptotic expansions by γ -convergence. Continuum Mechanics and Thermodynamics , 20:21-62, 2008.

Lia Bronsard and Robert V. Kohn. On the slowness of phase boundary motion in one space dimension. Comm. Pure Appl. Math. , 43(8):983-997, 1990. ISSN 0010-3640. doi: 10.1002/cpa.3160430804. URL http://dx.doi.org/10.1002/cpa.3160430804 .

Andrei Caragea, Philipp Petersen, and Felix Voigtlaender. Neural network approximation and estimation of classifiers with classification boundary in a barron class. arXiv preprint arXiv:2011.09363 , 2020.

Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterized models using optimal transport. Advances in neural information processing systems , 31, 2018.

Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory , pages 1305-1338. PMLR, 2020.

Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems , 32, 2019.

Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv:1804.10200 , 2018.

Gianni Dal Maso. An introduction to Γ -convergence , volume 8. Springer Science & Business Media, 2012.

Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems , 34:27449-27461, 2021.

Ennio De Giorgi and Tullio Franzoni. Su un tipo di convergenza variazionale. Atti della Accademia Nazionale dei Lincei. Classe di Scienze Fisiche, Matematiche e Naturali. Rendiconti , 58(6):842850, 1975.

Thomas Debarre, Quentin Denoyelle, Michael Unser, and Julien Fageot. Sparsest piecewise-linear regression of one-dimensional data. Journal of Computational and Applied Mathematics , 406: 114044, 2022.

Manfred Dobrowolski. Angewandte Funktionalanalysis: Funktionalanalysis, Sobolev-Räume und Elliptische Differentialgleichungen . Springer-Verlag, 2010.

Patrick W Dondl, Matthias W Kurzke, and Stephan Wojtowytsch. The effect of forest dislocations on the evolution of a phase-field model for plastic slip. Archive for Rational Mechanics and Analysis , 232(1):65-119, 2019.

Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprint arXiv:1810.02054 , 2018.

Weinan E and Stephan Wojtowytsch. Representation formulas and pointwise properties for Barron functions. Calc. Var. Partial Differential Equations , 61(46), 2020.

Weinan E and Stephan Wojtowytsch. Kolmogorov width decay and poor approximators in machine learning: Shallow neural networks, random feature models and neural tangent kernels. Research in the Mathematical Sciences , 8(1):1-28, 2021.

Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning , pages 270-290. PMLR, 2022.

Weinan E, Chao Ma, and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences , 17(5):1407-1425, 2019a. doi: 10.4310/ cms.2019.v17.n5.a11. URL https://doi.org/10.4310%2Fcms.2019.v17.n5.a11 .

Weinan E, Chao Ma, and Lei Wu. A comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics. Sci. China Math , 2019b.

Weinan E, Chao Ma, and Lei Wu. The barron space and the flow-induced function spaces for neural network models. arXiv:1906.08039 [cs.LG] , 2019c.

Lawrence Craig Evans and Ronald F Gariepy. Measure theory and fine properties of functions . CRC press, 2015.

Gero Friesecke, Richard D James, and Stefan Müller. Rigorous derivation of nonlinear plate theory and geometric rigidity. Comptes Rendus Mathematique , 334(2):173-178, 2002a.

Gero Friesecke, Richard D James, and Stefan Müller. A theorem on geometric rigidity and the derivation of nonlinear plate theory from three-dimensional elasticity. Communications on Pure and Applied Mathematics , 55(11):1461-1506, 2002b.

Gero Friesecke, Richard D James, Maria Giovanna Mora, and Stefan Müller. Derivation of nonlinear bending theory for shells from three-dimensional nonlinear elasticity by gamma-convergence. Comptes Rendus Mathematique , 336(8):697-702, 2003.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Karl Hajjar and Lenaic Chizat. Symmetries in the dynamics of wide two-layer neural networks. arXiv preprint arXiv:2211.08771 , 2022.

Boris Hanin. Ridgeless interpolation with shallow relu networks in 1 d is nearest neighbor curvature extrapolation and provably generalizes on lipschitz functions. arXiv preprint arXiv:2109.12960 , 2021.

Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems , 31, 2018.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026-1034, 2015.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation , 9(1):1-42, 1997.

Jean Honorio and Tommi Jaakkola. Tight Bounds for the Expected Risk of Linear Classifiers and PAC-Bayes Finite-Sample Guarantees. In Samuel Kaski and Jukka Corander, editors, Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics , volume 33 of Proceedings of Machine Learning Research , pages 384-392, Reykjavik, Iceland, 22-25 Apr 2014. PMLR. URL https://proceedings.mlr.press/v33/honorio14.html .

Tom Ilmanen. Convergence of the Allen-Cahn equation to Brakke's motion by mean curvature. J. Differential Geom. , 38(2):417-461, 1993. ISSN 0022-040X. URL http://projecteuclid. org/euclid.jdg/1214454300 .

Hui Jin and Guido Montúfar. Implicit bias of gradient descent for mean squared error regression with two-layer wide neural networks. arXiv preprint arXiv:2006.07356 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Achim Klenke. Wahrscheinlichkeitstheorie , volume 1. Springer, 2006.

Marta Lewicka, Maria Giovanna Mora, and Mohammad Reza Pakzad. Shell theories arising as low energy γ -limit of 3d nonlinear elasticity. Annali della Scuola Normale Superiore di Pisa-Classe di Scienze , 9(2):253-295, 2010.

Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss?-a mathematical framework. arXiv preprint arXiv:2110.06914 , 2021.

Zhong Li, Chao Ma, and Lei Wu. Complexity measures for neural networks with general activation functions using path-based norms. arXiv preprint arXiv:2009.06132 , 2020.

Marco Loog, Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences , 117(20):10625-10626, 2020.

Chao Ma, Lei Wu, et al. Machine learning from a continuous viewpoint, i. Science China Mathematics , 63(11):2233-2266, 2020.

Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of twolayer neural networks. Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018.

Luciano Modica. The gradient theory of phase transitions and the minimal interface criterion. Arch Ration Mech Anal , 98(2):123-142, 1987.

Luciano Modica and Stefano Mortola. Un esempio di Γ -convergenza. Boll. Un. Mat. Ital. B (5) , 14 (1):285-299, 1977.

Luca Mugnai and Matthias Röger. Convergence of perturbed Allen-Cahn equations to forced mean curvature flow. Indiana Univ. Math. J. , 60(1):41-75, 2011. ISSN 0022-2518. doi: 10.1512/iumj. 2011.60.3949. URL http://dx.doi.org/10.1512/iumj.2011.60.3949 .

Sebastian Neumayer, Lénaïc Chizat, and Michael Unser. On the effect of initialization: The scaling path of 2-layer neural networks. arXiv preprint arXiv:2303.17805 , 2023.

Greg Ongie and Rebecca Willett. The role of linear layers in nonlinear interpolating networks. arXiv preprint arXiv:2202.00856 , 2022.

Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. arXiv preprint arXiv:1910.01635 , 2019.

Rahul Parhi and Robert D Nowak. Banach space representer theorems for neural networks and ridge splines. J. Mach. Learn. Res. , 22(43):1-40, 2021.

Rahul Parhi and Robert D Nowak. What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM Journal on Mathematics of Data Science , 4(2):464-489, 2022.

Scott Pesme, Loucas Pillaud-Vivien, and Nicolas Flammarion. Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems , 34:29218-29230, 2021.

Grant M Rotskoff and Eric Vanden-Eijnden. Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error. stat , 1050:22, 2018.

Rudin. Functional Analysis . International series in pure and applied mathematics. McGraw-Hill, 1991. ISBN 9780070542365.

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018 , volume 80 of Proceedings of Machine Learning Research , pages 4430-4438. PMLR, 2018a. URL http://proceedings.mlr.press/ v80/safran18a.html .

Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks. In International conference on machine learning , pages 4433-4441. PMLR, 2018b.

Etienne Sandier and Sylvia Serfaty. Gamma-convergence of gradient flows with applications to Ginzburg-Landau. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences , 57(12):1627-1672, 2004.

Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory , pages 2667-2690. PMLR, 2019.

Sylvia Serfaty. Gamma-convergence of gradient flows on Hilbert and metric spaces and applications. Discrete Contin. Dyn. Syst , 31(4):1427-1451, 2011.

Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Jonathan W Siegel and Jinchao Xu. Approximation rates for neural networks with general activation functions. Neural Networks , 128:313-321, 2020.

Jonathan W Siegel and Jinchao Xu. Characterization of the variation spaces corresponding to shallow neural networks. arXiv preprint arXiv:2106.15002 , 2021.

Jonathan W Siegel and Jinchao Xu. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations of Computational Mathematics , pages 1-57, 2022.

Jonathan W Siegel and Jinchao Xu. Characterization of the variation spaces corresponding to shallow neural networks. Constructive Approximation , pages 1-24, 2023.

Justin Sirignano and Konstantinos Spiliopoulos. Scaling limit of neural networks with the xavier initialization and convergence to a global minimum. arXiv preprint arXiv:1907.04108 , 2019.

Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A law of large numbers. SIAM Journal on Applied Mathematics , 80(2):725-752, 2020a.

Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications , 130(3):1820-1852, 2020b.

Samuel L Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176 , 2021.

Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in two-layer neural network optimization landscapes. arXiv preprint arXiv:1802.06384 , 2018.

Stephan Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime. arXiv preprint arXiv:2005.13530 , 2020.

Stephan Wojtowytsch. Optimal bump functions for shallow relu networks: Weight decay, depth separation and the curse of dimensionality. arXiv preprint arXiv:2209.01173 , 2022.

Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: A stability analysis. Advances in Neural Information Processing Systems , 35:4680-4693, 2022.

Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems , 34:18722-18733, 2021.

Pan Zhou, Jiashi Feng, Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems , 33:21285-21296, 2020.

Appendix

A Numerical Experiments

A.1 Hyperparameter settings and computation effort

In all experiments in Dimensions 3 and 15, the following hyperparameter settings were used unless otherwise indicated:

Normal Xavier initialization with gain α = √ 2

SGD: Learning rate = 10 -2 (Dimension 15), 10 -3 (Dimension 3).

Momentum-SGD: Learning rate = 10 -3 and momentum µ = 0 . 99

ADAM: Learning rate = 10 -3 and PyTorch default hyperparameters for β 1 = 0 . 9 , β 2 = 0 . 999 , ε = 10 -8 .

For experiments in Dimension 31, we drop the learning rate for ADAM after 50 of 150 epochs by a factor of 10 and for Momentum-SGD by a factor of 10 after 100 epochs.

In Dimension 3, a learning rate of 10 -2 was found numerically unstable for SGD without momentum. To compensate for the smaller learning rate and provide a fair comparison, the number of time steps was increased.

All experiments were performed on a free version of google colab or the experimenters' personal computers. One run of the model takes below fifteen minutes on a single graphics processing unit.

A.2 Summary and interpretation of additional simulations

In this Section, we present additional numerical experiments in various situations complementary to those presented in the main body of the text. These include: Wider neural networks (Appendix A.3), experiments with different optimizers (Appendix A.4), experiments with different initialization to explore effects of scale and symmetry and the role of explicit regularization (Appendices A.8 and A.5), experiments with ℓ 1 -loss instead of ℓ 2 -loss (Appendix A.7) and repeated experiments to visualize the stochastic variation between runs (Appendix A.6).

Additionally, we present and investigation into related settings where our theoretical understanding does not apply: In Appendix A.10, we consider linearized (random feature) dynamics to explore how close we are to a (truly non-linear) neural network model. In Appendix A.11, we consider neural networks with a single hidden layer and leaky ReLU activation instead of ReLU activation. In Appendix A.12, we consider ReLU networks with multiple hidden layers. For a detailed list, see the table of contents below.

Our goal is not to explore questions of loss function, initialization, optimization algorithm and the impact of hyperparameters in a systematic fashion, but rather to establish problems in which a minimum norm interpolant can be found in an explicit fashion as instructive benchmarks to numerically study such questions. As a proof of concept, we provide a partial exploration of the parameter space. For the moment, we find ourselves confined to ReLU networks with a single hidden layer, as this is the only case in which explicit minimum norm interpolants are available. Minimum norm interpolation describes the shape of functions between known data clusters and is thus more expressive than a study of generalization error which is naturally confined to data clusters.

The additional experiments corroborate our findings in the main text. Before the detailed presentation, let us briefly summarize the conclusions.

Across a variety of different optimizers, Xavier type (= Glorot type) initialization schemes and loss functions, a minimum Barron norm interpolant-like shape was attained, to varying degrees of accuracy and with different rescaling factors.

Solutions are fairly radially symmetric with standard deviation in radial direction at most 0 . 1 (SGD) and 0 . 2 (Adam).

A geometrically distinct shape is observed for random feature models in the same regime.

Explicit regularization has little effect, even for poorly chosen initializations of Xavier type with high gain. We conjecture that the uniqueness of the radially symmetric minimum norm interpolant induces a higher degree of rigidity and bias, compared to the one-dimensional case where the set of minimum norm solutions was diverse (and infinite).

Functions display larger variation in the radial direction for He initialization. In this regime, explicit regularization has more apparent and beneficial effects on both solution shape and radial symmetry. The solution does not reduce to the random feature model in this case either.

A.3 Wide neural networks in one dimension

In Figure 5, we present the same experiment as in Figure 2 for wider neural networks with m = 1 , 000 neurons in the hidden layer.

A.4 SGD and ADAM: Dimensions 3 and 15

We repeat the experiment on Figure 3 for Stochastic Gradient Descent (SGD) optimizer without momentum and for the Adam optimizer of Kingma and Ba [2014]. The results are displayed in Figure 6 and 7 respectively. The results strongly resemble those obtained for the SGD optimizer with momentum in Figure 3.

A.5 Radial symmetry in Dimension 31

As indicated in Figure 4, we present computational results with the radially symmetric normal Xavier initialization in Figure 8.

101

gain = 0.5, width = 1000, lambda = 0.0

gain = 1.0, width = 1000, lambda = 0.0

gain = 5.0, width = 1000, lambda = 0.0

natural cubic siline

natural cubic spline

natural cubic spline

2.0

ReLU network (Momentum GD)

target function

ReLU network (GD)

2.0

ReLU network (Momentum GD)

ReLU network (GD)

target function

ReLU network (Momentum GD)

ReLU network (GO)

target function

ReLU network (8FGS)

ReLU network (BFGS)

ReLU network (BFGS)

1.5

1.5

S’T

1.0

0.5

gain = 0.5, width = 1000, lambda = 0.002

gain = 1.0, width = 100, lambda = 0.002

gain = 5.0, width = 1000, lambda = 0.002

2.0

natural cubic spline

ReLU network (ADAM)

ReLU network (Momentum GD)

target function

ReLU network (GD)

2.0

一

natural cubic spline

ReLU network (ADAM)

ReLU network (Momentum GD)

ReLU network (GD)

target function

0

natural cubic spline

ReLU network (Momentum GD)

ReLU network (GD)

ReLU network (ADAM)

target function

ReLU network (BFGS)

ReLU network (BFG5)

ReLU network (BFGS)

1.5

1.5

1.0

1.0

0.5

-1

gain = 0.5, width = 1000, lambda = 0.005

gain = 1.0, width = 100, lambda = 0.005

gain = 5.0, width = 1000, lambda = 0.005

natural cubic spline

targetfuntin

natural cubic spline

target function

natural cubic spline

target function

2.0

ReLU nework(ADAM)

ReLU network (Momentum GD)

ReLU network (GD)

2.0

ReLU network (Momentum GO)

ReLU network (GD)

ReLU network (ADAM)

2.0

ReLU network (Momentum GD)

ReLU network (GD)

ReLU network (BFG5)

ReLU network (BFG5)

ReLU network (BFGS)

1.5

1.5

1.5

1.0

1.0

0.5

SGD after 100 epochs. width = 12000. 1000000 data points. Dimension 3. Weight decay = 0.0

1.0

numerical solution

1.0

numerical solution

optimal interpolation in dim 3

optimal interpolation in dim 15

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

00

0°0

0

3

4

0

1

2

4

5

SGD after 1000 epochs. width = 12000. 10000 data points. Dimension 15. Weight decay = 0.0

SGD after 100 epochs. width = 12000. 100000 data points. Dimension 15. Weight decay = 0.0

SGD after 10 epochs. width = 12000. 100000 data points. Dimension 15Weight decay = 0.0

1.0

optimal interpolation in dim 15

1.0

optimal interpolation in dim 15

1.0

numerical solution

optimal interpolation in dim 15

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

Adam after 10 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 3

Adam after 10 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 15

1.0

optimal interpolation in dim 3

1.0

optimal interpolation in dim 15

numerical solution

numerical solution

0.8

80

0.6

0.6

0.4

0

0.2

0.2

0.0

0.0

0

1

2

3

4

5

0

1

3

4

5

Adam after 1000 epochs. Depth = 2, width = 12000, 10000 data points. Dimension 15

Adam after 100 epochs. Depth = 2, width = 12000, 100000 data points. Dimensson 15

Adam after 10 epochs. Depth = 2, wioth = 12000, 1000000 data points. Dimension 15

1.0

St wip u uogelodau (ewgdo

numerical solution

1.0

St wip u uogejodqu jeugo

numerical solution

1.0

St wp u uogejdqu jewgdo

numerical solution

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

radial average

radial standard deviation

weight decay regularizer

1.0

0.10

1600

1400

0.8

0.08

1200

0.6

0.4 -

Adam

Optimal solution

SGD

0.06

0.04

SGD

Adam

1000

800

Adam

SGD

600

0.2

0.02

400

0.0

0.00

50000

100000

150000

200000

250000

A.6 Gradient descent with Momentum

In Figure 9, we present additional runs in the setting of Figure 3. Despite quantitative variation, the geometric shapes of solutions are stable over multiple runs and resemble the minimum norm interpolant f ∗ 15 in all cases.

A.7 ℓ 1 -loss and Huber loss

We repeat the experiment of Figure 3 with the ℓ 1 -loss function in the place of ℓ 2 -loss. To compensate for the lack of smoothness in the loss function, we reduce the learning rate by a factor of 10 to 10 -4 and increase the number of epochs by 50% to compensate. Results are reported in Figure 10.

Momentum SGD after 10 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 15

Momentum SGD after 10 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 15

Momentum SGD after 10 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 15

1.0

optimal interpolation in dim 15

numerical solution

1.0

optimal interpolation in dim 15

numerical solution

1.0

optimal interpolation in dim 15

numerical solution

0.8 -

0.8

0.8

0.6 -

0.6

0.6

0.4 -

0.4

0.4

0.2

0.2

0.2

0.0

- 0′0

0.0

0

2

E

0

Z

E

0

2

E

4

5

Momentum SGD after 15 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 3

Momentum SGD after 15 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 15

optimal interpolation indim 3

1.0

optimal interpolation in dim 15

1.0

numerical solution

numerical solution

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

0°0

0.0

0

1

2

3

4

0

1

2

3

5

Momentum SGD after 1500 epochs. Depth = 2, width = 12000, 10000 data points. Dimension 15

Momentum SGD after 150 epochs. Depth = 2, width = 12000, 100000 data points. Dimension 15

Momentum SGD after 10 epochs. Depth = 2. width = 12000, 1000o00 data points. Dimension 15

1.0

optimal interpolation in dim 15

numerical solution

1.0

ST wp u uogedaqu jeugdo

numerical solution

1.0

SI wp u gejoda jewgdo

numerical solution

0.8 -

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0'0

Similarly, we repeated experiments with the Huber loss function. Since | f ( x i ) -f ∗ ( x i ) | < 1 over the data set during the final stages of training, the loss function coincides with ℓ 2 -loss in the long run and all experiments are identical to ℓ 2 -loss. We therefore do not present additional plots.

In the initial stages of training, Huber loss is more stable numerically than ℓ 2 -loss, especially for large gain or He initialization (compare Section A.9).

A.8 Initialization scaling and explicit regularization: high-dimensional radial data

As noted in Section 5.1 and previously by Chizat et al. [2019], the choice of initialization affects the optimization process of neural networks. Motivated by our observations in the one-dimensional case, we consider the effects of initialization and explicit regularization in the radially symmetric setting (Section 5.2). Our results support the earlier claim that the effects of regularization are advantageous for poorly chosen initialization with high gain.

The experiments were performed in higher dimension 31 and with uniform Xavier initialization for the scenario in which it is most challenging to obtain radially symmetric solutions. As in Appendix A.7, we used ℓ 1 -loss rather than ℓ 2 -loss. To compensate for the non-smoothness of the loss function, we drop the learning rate by a factor of 10 twice during the training process. Similar results were observed for ℓ 2 -loss, but the effects of initialization and regularization were less pronounced compared to the ℓ 1 -case.

Plots for a single representative run are displayed in Figure 11. The explicit regularizer has the clearest effect in the high gain regime, where explicit regularization helps to achieve a better fit with the optimal transition curve and reduces the radial standard variation. Results were less sensitive to poor initialization than the corresponding experiments in one dimension. We conjecture that a higher degree of rigidity is introduced in this setting by the fact that there exists a unique minimum norm interpolant.

A.9 He initialization

All experiments so far were performed with the initialization scaling of Glorot and Bengio [2010]. Especially for deeper neural networks, the initialization scheme of He et al. [2015] is very popular. Hanin and Rolnick [2018] proves in particular that He et al. [2015]'s normalization avoids the vanishing and exploding gradients phenomenon at initialization in expectation. While this consideration does not apply to our shallow networks, we find it informative to compare the two schemes.

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0. gain = 0.25

1.0

numerical solution

optimal interpolation in dim 31

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0. gain = sqrt(2)

1.0

TE wp u uogejoduaqu jewgdo

numerical solution

0t = uje5 oo = Kesap su5jam te uoisuawla squjod epep oooooot ooozt = ypm sypoda ot Jaye 09

1.0 -

numerical solution

optimal interpolation in dim 31

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4 4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0

1

E

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0001. gain = 0.25

1.0

numerical solution

optimal interpolation in dim 31

SGD after 10 epochs. width = 12000. 100000 data points. Dimension 31. Weight decay = 0.0001. gain = sqrt(2)

1.0

numerical solutien

optimal inteepolation in dim 31

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0001. gain = 10

1.0

numerical solution

optimal interpolation in dim 31

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

√

For shallow neural networks

the effect of initialization is as follows:

According to Glorot and Bengio [2010], the parameters a i and w ij , 1 ≤ i ≤ m and 1 ≤ j ≤ d are chosen as random variables with mean 0 and standard deviation √ 2 / ( m +1) for a i and √ 2 / ( m + d ) for w i,j . In particular, if m is much larger than d , then the | a i | ∥ w i ∥ = O (1 /m ) . As we add m terms of magnitude ∼ 1 /m , we consider this the 'law of large numbers' scaling.

According to He et al. [2015], the parameters a i and w ij , 1 ≤ i ≤ m and 1 ≤ j ≤ d are chosen as random variables with mean 0 and standard deviation √ 2 /m for a i and √ 2 /d for w i,j . In particular, if m is much larger than d , then the | a i | ∥ w i ∥ = O (1 / √ m ) . As we add m terms of mean zero and magnitude ∼ 1 / √ m , we consider this the 'central limit theorem' scaling.

Many authors, such as Sirignano and Spiliopoulos [2020a,b, 2019], present the factor 1 /m or 1 / √ m explicitly outside the neural network. As observed above, the effect of initialization can be significant. Unsurprisingly, results in the central limit regime, where all neurons contribute similarly at initialization, are more consistent and predictable. Experimental results are presented in Figure 12 in the one-dimensional setting and in Figures 13 and 14 in the case of radial symmetry. Notably, in high dimension, explicit regularization not only reduced radial variation, but also increased data compliance by reducing the rescaling factor r d . In this setting, we observe the benefits of explicit regularization over relying on implicit bias only.

A.10 Linearized dynamics

Parameter optimization in neural networks depends heavily on the choice of initialization. While the dynamics are truly non-linear in the regime studied by Chizat and Bach [2018], Rotskoff and Vanden-Eijnden [2018], Mei et al. [2018], Sirignano and Spiliopoulos [2020a] and Wojtowytsch [2020], there are scalings for which the directions w i remain close to their initialization for all time - see e.g. [E et al., 2019b] for a derivation. In this case, the solution produced by a neural network is similar to that of a random feature model. In this section, we numerically find the minimum norm interpolant of a random feature model by 'freezing' the inner layer coefficients at their random initialization. We find that the random feature solution differs geometrically from the Barron space solution, e.g. in that it is smooth at the origin, where the Barron space solution has a cone-like singularity of the form f ( x ) = 1 -c d ∥ x ∥ 2 for small x . In particular, we find that our experiments were set appropriately in the non-linear training regime. See Figure 15.

2.5

natural cubic spline

target function

ReLU network (Momentum SGD)

2.0

ReLU network (SGD)

ReLU network (ADAM)

1.5

1.0

0.5

0.0

2

-1

0

1

2

2.5

natural cubic spline

target function

ReLU network (Momentum SGD)

2.0

ReLU network (SGD)

ReLU network (ADAM)

1.5

1.0

0.5

0.0

-2

=1

1

2

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0

SGD after 10 epochs. width = 12000. 100oo00 data points. Dimension 31. Weight decay = 0.0001

1.0

numerical solution

optimal interpolation in dim 31

1.0

numerical solution

optimal interpolation in dim 31

8'0

0.8

0.6

9°0

0.4

0.4

0.2

0.2

0°0

00

E

1

E

Adam after 401 epochs. Depth = 2, width = 12000, 10000 data points.

Adam after 401 epochs. Depth = 2, width = 12000, 10000 data points.

1.0

optimal interpolation in dim 15. Rescaling = 0.4

1.0

optimal interpolation in dim 15. Rescaling = 0.4

optimal interpolation in dim 15. Rescaling = 0.5

optimal interpolation in dim 15. Rescaling = 0.25

numerical solution in Dimension 15

numerical solution in Dimension 15

0.8

0.8

0.6

0.6

0.4

0.4

0.2

0.2

F0°0

00

0

1

E

Bottom row: We repeat the same as above with MSE loss rather than ℓ 1 -loss, using the Adam optimizer with learning rate 10 -5 instead of SGD with momentum, and using an overparametrized rather than underparametrized neural network. Without explicit regularization (left), the radial standard deviation is substantial, while explicit regularization leads to a more radially symmetric function, albeit at the price of a higher rescaling factor. For easy comparison to Figure 3, we present the optimal profile as rescaled in the main document as well. Both functions achieve training loss < 10 -3 , but clearly generalization is poor without regularization: While the radial average is close to the target function f ∗ ≡ 0 for inputs with ∥ x ∥ ≥ 1 , the radial variation is high.

SGD 10 epochs. width = 12000. 1000000 data. Dim 3. Weight decay = 0.0

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 15. Weight decay = 0.0

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0

1.0

optimal interpolation in dim 3

numerical solution

1.0

numerical solution

optimal interpolation in dim 15

1.0

numerical solution

optimal interpolation in dim 31

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

0

2

0

1

2

Q

SGD 10 epochs. width = 12000. 1000000 data. Dim 3. Weight decay = 0.0001

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 15. Weight decay = 0.0001

SGD after 10 epochs. width = 12000. 1000000 data points. Dimension 31. Weight decay = 0.0001

1.0

numerical solution

optimal interpolation in dim 3

1.0

numerical solution

optimal interpolation in dim 15

1.0

numerical solution

optimal interpolation in dim 31

0.8

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.0

0.0

1.0

Momentum SGD after 401 epochs. Depth = 2, width = 12000, 10000 data points.

dimension 3

dimension 15

1.0

Momentum SGD after 401 epochs. Depth = 2, width = 12000, 10000 data points.

dimension 15

dimension 3

1.0

Momentum SGD after 401 epochs. Depth = 2, width = 12000, 10000 data points.

uo(suauip

dimension 15

0.8

0.8

0.8

0.6

0.6

F 9'0

0.4 -

0.4

0.4

0.2 -

0.2

0.2

0.0

0.0

0.0

0.2

0

1

2

E

4

E

Q

2

A.11 Leaky ReLU activation

As noted by Wojtowytsch [2022], minimum norm interpolation is not stable when passing to an equivalent norm. A Barron space theory can be developed in perfect analogy for networks with the leaky ReLU activation function, and it is easy to see that the 'Barron' spaces for both activation functions coincide with equivalent norms, depending on the negative slope of the leaky ReLU function. However, the minimum norm interpolant f ∗ d with respect to the ReLU-based Barron-norm is not guaranteed to coincide with the minimum interpolant for the leaky-ReLU-based Barron norm. Experimentally, however, we observe strong agreement between the geometry of numerical solutions here.

A.12 Deeper neural networks

We train neural networks of depth L > 2 to fit the same radially symmetric data as in Section 5.2. We see in Figure 17 that for depth L ≥ 3 , weight decay-regularized networks strongly resemble the interpolant f Lip ( x ) = max { 1 -∥ x ∥ 2 , 0 } with minimal (Euclidean) Lipschitz constant. This function can be written as a composition of two Barron functions f = f bump ◦ f norm

Momentum SGD after 1000 epochs. Depth = 2. width = 12000, 10000 data points. Dimension 15

Adam after 10 epochs. Depth = 2, width = 12000, 1000000 data points. Dimension 15

Momentum SGD after 10 epochs. Depth = 2, width = 25000, 1000000 data points. Dimension 15

1.0

optimal interpolation in dim 15

numerical solution

1.0

numerical solution

optimal interpolation in dim 15

1.0

optimal interpolation in dim 15

numerical solution

0.8

0.8

0.8

0.6

0.6

9′0

0.4

0.4

0.4

0.2

0.2

Z'0

0.0

0.0

0.0

2

0

3

ADAM after 50 epochs. Width = 250. gain = sqyrt(2), weight decay = 0.01. 10000 data peints. Dimension 15

ADAM after 50 epochs. Width = 250, gain = sqrt(2), weight decay = 0.001. 10000 data peints. Dimension 15

ADAM after 50 epochs. Width = 250, gain = sert(2), weight decay = 0.0001., 10000 data points. Dimension 15

ADAM after 50 epochs. Width = 250, gain = sqrt(2), weigMt decay = 0.0, 10000 data points. Dimension 15

二

二

dge.5

degth 3

legth 10

1

二

二

dtpth 5

depth 10

4.4

二

dhgth 5

degth 3

agth 19

二

二

degth 3

@egh 10

0.4

上京

0.2

4/0

Unlike networks with one hidden layer, deeper networks have positive (or nearly positive) outputs everywhere. While two-layer networks follow the minimum norm interpolation shape closely by the origin and have radial variances which increase outside the unit ball, deeper networks have positive radial variance inside the unit ball, but are essentially radially symmetric outside - compare e.g. Figure 9.

where π 0 denotes the uniform distribution on the unit sphere and c d ∼ √ d is a dimension-dependent constant. We can thus approximate f Lip efficiently by neural networks of depth L ≥ 3 as long as the first layer is sufficiently wide.

Unlike their shallow counterparts, neural networks with multiple hidden layers have no strong geometric prior without weight decay regularization. With weight decay, the observed behavior was relatively stable over a range of dimensions, initializations scalings and optimization algorithms. We are led to conjecture that f Lip is a minimum norm interpolant in this setting. The statement remains imprecise at this point as no function space theory for deeper networks with weight decay regularizer has been developed to the same extent as Barron space theory.

B Γ -convergence

In this appendix, we recall the definition and a few properties of Γ -convergence, a popular notion of the convergence of functionals introduced by De Giorgi and Franzoni [1975] in the calculus of variations to study the convergence of minimization problems. Braides [2002], Dal Maso [2012] provide introductions to the theory and its applications. As the notion is likely not familiar to readers from the machine learning community, we provide some full proofs as well.

Definition B.1. Let ( X,d ) be a metric space and F n , F : X → R ∪{-∞ , ∞} be functions. We say that F n converges to F in the sense of Γ -convergence if two conditions are met:

( lim inf -inquality) If x n is a sequence in X and x n → x , then lim inf n →∞ F n ( x n ) ≥ F ( x ) .

( lim sup -inequality) For every x ∈ X , there exists a sequence x ∗ n ∈ X such that x ∗ n → x and lim sup n →∞ F n ( x ∗ n ) ≤ F ( x ) .

Intuitively, the first condition means that F ( x ) is (almost) a lower bound for F n ( x n ) if n is 'large' and x n is 'close' to x , while the second condition means that there is no larger lower bound that we could choose. The sequence x ∗ n is often referred to as a 'recovery sequence'. Of course,

combining the liminf- and limsup-inequalities, we find that in fact F n ( x ∗ n ) → F ( x ) . We employ Γ -convergence when dealing with minimization problems where uniform convergence fails, but we hope for convergence of minimizers to minimizers.

Often, Γ -convergence is considered as a continuous parameter ε approaches 0 + rather than as the discrete parameter n approaches infinity. The definitions remain largely identical (with obvious substitutions).

To get a feeling for Γ -convergence, we consider a particularly simple situation by looking at two constant sequences of functions. Note that the sequence is constant, not the functions.

Example B.2. Let X = R and consider the constant sequences

̸

We claim that

̸

If x n → x and x = 0 , then F n ( x n ) = 1 and G n ( x n ) = 0 for all but finitely many n ∈ N , meaning that F n ( x n ) → 1 = f ( x ) and G n ( x n ) → 0 . It remains to consider the case x n → 0 .

We see immediately that F n ( x n ) ≥ 0 = f (0) for all n ∈ N . Conversely, if we take x ∗ n = 0 for all n , then x ∗ n → 0 and F n ( x ∗ n ) = f (0) → f (0) . In total, we conclude that Γ -lim F n = f .

For G n , we find that G n ( x n ) ≥ 0 for all n ∈ N . Additionally, we can choose the sequence x n = 1 /n such that G n ( x n ) = 0 for all n ∈ N . Altogether, we find that Γ -lim G n = 0 .

More generally, if F n = F for all n ∈ N and some F : X → R , then Γ -lim n →∞ F n = F is the lower semi-continuous envelope of F . In particular, Γ -lim n →∞ F = F if and only if F is lower semi-continuous. The main useful properties of Γ -convergence are summarized in the following lemma.

Lemma B.3. Assume that F n → F in the sense of Γ -convergence, ε n → 0 + and x n ∈ X is a sequence such that

Assume that x n → x ∗ . Then F ( x ∗ ) = inf x ∈ X F ( x ) . In particular, if x n is a minimizer of F n and the sequence x n converges, then the limit point is a minimizer of F .

Clearly, this is most useful if we can guarantee that the sequence x n converges. For many useful sequences of functionals, the existence of a convergent subsequence can be established by compactness. This is easily sufficient, as we can also pass to a subsequence in F n .

Proof. Due to the liminf-inequality, we have

On the other hand, let x ∈ X be any point. Then, due to the limsup-inequality, there exists some sequence x ′ n such that

In particular inf x ∈ X F ( x ) ≥ lim inf n →∞ inf x ∈ X F n ( x ) . Combining the two estimates, we find that F ( x ∗ ) ≤ inf x ∈ X F ( x ) , which means that x ∗ is a minimizer of F .

For completeness, a few observations are in order.

The notion of Γ -convergence relies on the notion of convergence on the underlying space X , and Γ -limits can change when keeping the set X fixed, but passing to a different topology (e.g. a weak topology in infinite-dimensional spaces).

̸

Γ -convergence is made for minimization problems, and it does not behave well under multiplication by negative real numbers: In general Γ -lim( -F n ) = -(Γ -lim F n ) , even if both limits exist. The reason is the asymmetry between the lim inf - and the lim sup -condition. To see this, consider for instance F n and G n -1 in Example B.2.

̸

If F n → F and G n → G , it is not necessarily true that F n + G n → F + G . While it remains true that lim inf n →∞ ( F n + G n )( x n ) ≥ ( F + G )( x ) if x n → x , it may no longer be possible to find a recovery sequence x n for F n + G n . For example, if F n = 1 Q and G n = 1 R \ Q for all n ∈ N , then F n Γ - → 0 and G n Γ - → 0 , but F n + G n = 1 for all n and F n + G n Γ - → 1 .

However, if F n Γ - → F and G n converges to a continuous limit G uniformly , then ( F n + G n ) Γ - → F + G . In particular, uniform convergence implies Γ -convergence. Namely, if G n → G uniformly, G is continuous and x n → x , then for given ε > 0 , we can choose N ∈ N so large that

(a) | G ( x n ) -G ( x ) | < ε/ 2 for all n ≥ N since G is continuous at x and

(b) | G n ( x n ) -G ( x n ) | < ε/ 2 for all n ≥ N due to uniform convergence.

Then

In particular G n ( x n ) → G ( x ) . Recall that G is guaranteed to be continuous if G n is continuous for all n ∈ N .

Γ -convergence is unrelated to pointwise convergence of functions: Neither does it imply pointwise convergence, nor is it implied by it. Namely, the sequence G n in Example B.2 has the function g as a pointwise limit and the constant function 0 as a Γ -limit.

̸

Γ -convergence is not a notion of convergence derived from a topology. Indeed, even if F n is a constant sequence, i.e. if F n = G for all n , it may happen that Γ -lim n →∞ F n = G (see G n in Example B.2). The Γ -limit is related to G , though: It is the lower semi-continuous envelope of the function G . In fact, every Γ -limit is lower semi-continuous.

Despite its somewhat counterintuitive properties, Γ -convergence has proved invaluable in many areas of the calculus of variations. It has been applied to homogenization by Bach et al. [2021], dimension reduction for thin sheets and shells by Friesecke et al. [2002a, 2003, 2002b], Bhattacharya et al. [2016], Lewicka et al. [2010] and the study of phase boundaries by Modica and Mortola [1977], Modica [1987]. While the Γ -convergence of functionals does not imply the convergence of their gradient flows even in situations of practical significance (see e.g. the example of Dondl et al. [2019]), Serfaty [2011], Sandier and Serfaty [2004], Mugnai and Röger [2011], Ilmanen [1993], Alikakos et al. [1994] provide important examples of situations where this can be established in a suitable sense. Even more, Bronsard and Kohn [1990] use Γ -convergence to gain insight into PDE dynamics.

C Homogeneous Barron spaces

In this section, we introduce the abstract framework which is used to prove our main theoretical result, Corollary 3.3. A neural network with m neurons in a single hidden layer can be represented as

The network weights and biases are ( a, W, b ) ∈ R m × R m × d × R m +1 . The normalization depends on personal preference, with the former being more common in practice and the latter more common in theoretic analyses. We define the weight decay regularizer by

or R WD ( a, W, b ) = 1 2 m ∑ m i =1 ( a 2 i + ∥ w i ∥ 2 ℓ 2 ) respectively. Here ∥·∥ 2 denotes the Euclidean ℓ 2 -norm of a vector and ∥ W ∥ F denotes the Frobenius norm of the matrix W whose rows are the vectors w T i . Note that we do not control the magnitude of the biases b i in the regularizer. This is a common approach, as the bias does not influence the Lipschitz-constant of the function represented by the neural network, which is useful in studying the generalization of the neural network.

We study function classes corresponding to arbitrarily wide neural networks with a single hidden layer, where the norm corresponds to the weight decay regularizer. We dub these function spaces 'homogeneous Barron spaces' in analogy to the more classical Barron spaces studied by E et al. [2019c], Ma et al. [2020], E and Wojtowytsch [2020], which correspond to a weight decay regularizer which also controls the bias. For homogeneous Barron spaces, coordinate transformations by Euclidean motions induce an isometry of the function class, while the origin plays a special role in classical Barron spaces. This justifies our terminology, as the data space is treated as isotropic and homogeneous by this function class. Homogeneous Barron spaces have also been studied by Ongie et al. [2019], Parhi and Nowak [2021, 2022] under the name Radon-BV spaces. A closely related class of spaces has been considered as the 'variation spaces of the ReLU dictionary' by Siegel and Xu [2020, 2022, 2023].

Heuristically, (homogeneous) Barron spaces are a function class tailored to replacing the finite superposition of ReLU ridges in (4) by an arbitrary superposition while keeping the weight decay regularizer finite. Due to the lack of control over the bias term, we will see that a slightly awkward technical definition is needed. Let π be a probability distribution on the parameter space R × R d × R . We would like to define

and

f π,b 0 is an analogue of neural networks with a single hidden layer, but of arbitrary and possibly uncountably infinite width. Every finite neural network can be expressed in this fashion for the empirical measure π m = 1 m ∑ m i =1 δ ( a i ,w i ,b i ) , in which case R WD ( π m ) = R WD ( a, W, b ) .

Unfortunately, even if R WD ( π ) < ∞ , it is not clear that the integral defining f π exists in a meaningful sense. We do however note that formally

Thus, the integral defining f π ( x ) exists for all x if and only if it exists for, say, x = 0 . We exploit this in the following modified definition. Let π be a probability distribution on the parameter space R × R d × R and y ∈ R . We denote

By the same argument as before, we observe that

The class of functions of the form f π,y forms the homogeneous Barron space. Still, every finite neural network f m can be represented in this fashion with b 0 = y -∑ m i =1 a i σ ( b i ) or b 0 = y -1 m ∑ m i =1 a i σ ( b i ) .

Definition C.1 (Homogeneous Barron space) . Let f : R d → R be a function. We define the semi-norms

The homogeneous Barron space is the function class B ( R d ) = { f : R d → R : [ f ] B < ∞} . By the definition of a function [ f ] 0 < ∞ is automatically true.

We note a few important properties. First, we consider two function classes:

Note that F Q ( R ) ⊆ F Q ⊆ { f ∈ B : f (0) = 0 , [ f ] B ≤ Q } . The closure is taken with respect to locally uniform convergence, i.e. pointwise convergence which is uniform on all compact sets. 1 Due to the homogeneity of ReLU activation, we may prove the following.

While the claim is natural, its proof is surprisingly technical and postponed until the end of the section. The class F Q ( R ) will be used below for technical purposes. Of major importance below are the compact embedding theorem and the direct approximation theorem.

Theorem C.3 (Compact embedding) . Let f n ∈ B be a sequence such that lim inf n →∞ [ f n ] 0 + [ f n ] B < + ∞ . Then there exists f in B such that

f n → f in C 0 ( K ) for all compact sets K ⊆ R d .

f n → f in L p ( µ ) for all measures µ with finite p -th moments, p ∈ [1 , ∞ ) .

[ f ] B ≤ lim inf n →∞ [ f n ] B .

Proof. It is sufficient to show that the set ˜ F Q = { a ( σ ( w · x + b ) -σ ( b ) ) | a 2 + | w | 2 ≤ 2 Q } is compact in C 0 ( K ) and L 2 ( µ ) for all K and µ as above, in which case also its closed convex hull is compact [Rudin, 1991, Theorem 3.20.]. To this end, observe that the map

is continuous for any compact set K . Since K is compact, we have K ⊆ B R (0) for some R > 0 and thus | w · x | ≤ √ 2 QR for all x ∈ K . In particular,

Hence ˜ F Q = F ( { ( a, w, b ) : a 2 + | w | 2 ≤ 2 Q,b ≤ √ 2 QR } ) is the continuous image of a compact set, hence compact. We have thus proved a compact embedding into C 0 ( K ) for any compact set K .

Exhausting R d by the sequence of compact sets B m (0) , m ∈ N and using a diagonal sequence argument, we see that under the conditions of Theorem C.3, there exists f ∈ B such that f n → f pointwise everywhere on R d and uniformly on compact subsets. Additionally, we observe that f (0) = 0 and there exists a uniform upper bound on the Lipschitz constants of the sequence f n . We conclude that f n → f in L p ( µ ) from the Dominated Convergence Theorem using Q ∥ x ∥ as a dominating function.

As a consequence, we find the following.

Corollary C.4. 1. B is a Banach space.

C ∞ c ( R d ) ⊆ B ( R d ) .

Proof. The first claim follows as in [Siegel and Xu, 2021, Lemma 1], where it is proved in a more general context for dictionaries which are compact in a Hilbert space - in our case, the dictionary x ↦→ a { σ ( w T x + b ) -σ ( b ) } . The second claim follows from [Ongie et al., 2019, Corollary 1].

We conclude with a theorem which establishes a rate of approximation for Barron functions in a weaker topology.

Theorem C.5 (Direct approximation) . Let f ∈ B and µ a measure on R d with finite second moments. Then for any m ∈ N there exist c ∈ R and ( a i , w i , b i ) ∈ R × R d × R such that

Proof. A proof of this result can be found in [Wojtowytsch, 2022, Appendix C] in the proof of Proposition 2.6.

Proof of Lemma C.2. Step 1. Assume that f ∈ B such that f (0) = 0 and [ f ] B ≤ Q . By the Direct Approximation Theorem (which is proved in [Wojtowytsch, 2022, Appendix C] without using Lemma C.2), we find that for every m ∈ N and every measure µ on R d with finite second moments, there exists

such that ∥ f m -f ∥ L 2 ( µ ) ≤ C µ ∥ f ∥ B m -1 / 2 . In particular, f is in the closed convex hull of { a ( σ ( w T x + b ) -σ ( b ) ) : a 2 + | w | 2 ≤ 2 Q } if the closure is taken with respect to the L 2 ( µ ) topology. Additionally, the sequence f m has a uniformly bounded Lipschitz constant and is therefore compact in C 0 ( K ) for all compact K by a corollary to the Arzela-Ascoli theorem [Dobrowolski, 2010, Satz 2.42]. In particular, f m → f uniformly and thus f ∈ F Q .

Step 2. Denote f ( a,w,b ) ( x ) = a { σ ( w T x + b ) -σ ( b ) } . Since f ( a,w,b ) (0) = a { σ ( b ) -σ ( b ) } = 0 for all a, w, b , we conclude that f (0) = 0 for all f ∈ F Q . If f ∈ F Q , then there exists a sequence

such that f n → f locally uniformly. If the biases remain uniformly bounded, the sequence of empirical distributions

has a convergent subsequence by Prokhorov's Theorem [Klenke, 2006, Satz 13.29]. We denote the limiting distribution as π . The convergence of Radon measures implies the convergence of f n to f π ( x ) = E ( a,w,b ) ∼ π [ a { σ ( w T x + b ) -σ ( b ) } ] by definition. Since f n converges locally uniformly by assumption, f = f π ∈ B and [ f ] B ≤ Q .

If the biases do not remain bounded, we note that for every compact set K ⊆ R d we can extract a convergent subsequence of the measures by the same argument used to prove Theorem C.3, effectively making the sequence of biases bounded. We can extend the argument to the entire space exploiting that

locally uniformly.

D Rademacher complexity of homogeneous Barron space

Following a classical strategy implemented e.g. by E et al. [2019a] in a similar context, we estimate the Rademacher complexity of homogeneous Barron space and use it to bound the generalization gap (i.e. the discrepancy between empirical risk and population risk). In our setting, we face additional technical obstacles:

We deal with general sub-Gaussian data distributions µ rather than data distributions with compact support.

We do not control the magnitude of the bias variables.

We consider ℓ 2 -loss, which is neither globally Lipschitz-continuous nor bounded.

In combination, these complications require a refined technical analysis similar to Appendix C. Let us summarize several notations which will be needed below.

̂ Rad - the empirical Rademacher complexity of a function class over a given dataset.

Rad n - the expected Rademacher complexity of a function class over a data set composed of n iid samples from the data distribution µ .

R - the population risk R ( f ) = ∥ f -f ∗ ∥ 2 L 2 ( µ ) = E x ∼ µ [ | f ( x ) -f ∗ ( x ) | 2 ] . We generally take this to operate on the level of functions, parametrized or not. By an abuse of notation, we identify R ( a, W, b ) := R ( f ( a,W,b ) ) .

̂ R n - the empirical risk ̂ R n ( f ) = ∥ f -f ∗ ∥ 2 L 2 ( µ n ) = 1 n ∑ n i =1 | f ( x i ) -f ∗ ( x i ) | 2 over a data set { x 1 , . . . , x n } where µ n = 1 n ∑ n i =1 δ x i is the empirical measure. Equally, ̂ R n can be considered for functions or parameters with the natural identification.

̂ R n,m,λ . The regularized empirical risk

We only consider this quantity on the parameter level, where it is computable. While the weight decay regularizer is an upper bound for [ f ( a,W,b ) ] B , the two are generally not the same since the parameter-to-function map of a neural network is generally not injective.

R WD - the weight decay regularizer.

F Q - the set of functions for which [ f ] B ≤ Q and f (0) = 0 .

F A,Q - the set of functions for which [ f ] B ≤ Q and | f (0) | ≤ A .

As is common in the mathematics community, C will generally denote a constant which does not depend on quantities (unless specified otherwise) and which may change value from line to line. Some facts about sub-Gaussian distributions, which we believe to be well-known to the experts, are collected in Appendix H.

Definition D.1 (Rademacher Complexity) . Let S = { x 1 , . . . , x n } be a set of points in R d (a data sample) and F a real-valued function class. We define the empirical Rademacher complexity of F on the data sample as

where ε i are iid random variables which take the values ± 1 with equal probability 1 2 . The population Rademacher complexity is defined as

i.e. as the expected empirical Rademacher complexity over a set of n iid data points.

In this section, we will find a upper bound of Rademacher Complexity of B . We will denote by S n the set of n samples, and ̂ Rad( F , S n ) the sample Rademacher Complexity of F given the samples S n . We furthermore denote R := max {∥ x 1 ∥ , . . . , ∥ x n ∥} and consider the function classes F Q and F Q ( R ) as in Appendix C:

Lemma D.2. Let S n = { x 1 , . . . , x n } be a data set in R d . Then

Assume µ is a σ 2 sub-Gaussian distribution in R d . Then

for all n ≥ 2 .

Proof. Initially, we fix a set S = { x 2 , . . . , x n } of n points. We will later take the expectation over S , using the sub-Gaussian property of µ for an explicit norm bound. Define R := max 1 ≤ i ≤ n ∥ x i ∥ . To this end, we first prove the following claim, which enables us to focus on only single neuron functions instead of entire F Q :

Claim: Let ε 1 , . . . , ε n ∈ R . Then

Proof of Claim. Note that F Q is the closed convex hull of single neuron ridge functions, i.e. single neuron ridge functions are the extreme points of the closed convex set F Q .

To verify the claim, first note that f ↦→ ∑ n i =1 ε i f ( x i ) is a continuous linear functional on C 0 ( K ) for any compact K ⊆ R d containing the finite set S . It is well known that C 0 ( K ) is a Banach Space. Therefore, if { a ( σ ( w · x + b ) -σ ( b ) ) | a 2 + | w | 2 ≤ 2 Q } is compact in C 0 ( K ) , then [Rudin, 1991, Theorem 3.20.] implies that F Q , a closed convex hull of the compact set, is also compact. Then, from the compactness of F Q , we can use Bauer [1958]'s maximum principle and see that the supremum is attained at an extreme point. Compactness follows from the compact embedding Theorem, see Theorem C.3 above.

Over the next steps, we will bound ̂ Rad( F Q , S n ) .

Step 1. In this step, we prove that

To show this, we first observe that if | b | ≥ ∥ w ∥ R , then σ ( w · x + b ) -σ ( b ) = σ ( sgn ( b ) ) w · x since | w T x i | ≤ ∥ w ∥ ∥ x i ∥ ≤ | b | R for all 1 ≤ i ≤ n . This means for ∀| b | ≥ ∥ w ∥ R , the precise value of b does not change the value of σ ( w · x + b ) -σ ( b ) .

Now, we compute the ̂ Rad( F Q , S n ) :

For the first line, we used the claim.

Step 2. Using the uniform bound on the magnitude of the bias from the previous step, in this step we bound the Rademacher complexity by

We verify this from the definition of Rademacher Complexity of F Q ( R ) . Note that aσ ( wx + b ) = ( λa ) σ ( ( w/λ ) x + b/λ ) . In particular, we may assume without loss of generality that | a | 2 = ∥ w ∥ 2 ≤ Q for optimal balance which makes a 2 + ∥ w ∥ 2 minimal without changing the neuron output.

In this step, we only consider the first term.:

The first line is again by applying the claim to F Q ( R ) . In the last line, we used two facts:

the observation that

̸

since E [ ε i ] = 0 , ε i and ε j are independent if i = j and ε 2 i ≡ 1 .

Step 3. In this step, we prove that

To this end, we modify the data points as ˜ x i = ( x i , R ) and the parameters as ˜ w = ( w T , b R ) . Then, observe that w · x i + b = ˜ w · ˜ x i , ∥ ˜ x i ∥ = √ ∥ x i ∥ 2 + R 2 ≤ √ 2 R , and a 2 + ∥ ˜ w ∥ 2 = a 2 + ∥ w ∥ 2 +( b R ) 2 ≤ 3 Q . Therefore, we can write the above by the following:

Here, H 2 := { u ∈ R d | ∥ u ∥ 2 ≤ 1 } and ˜ S n = { ˜ x 1 , . . . , ˜ x n } . When removing the absolute value, we used that the sum is always non-negative and that it is symmetric when replacing ε by -ε . When removing σ , we make use of the Contraction Lemma for Rademacher complexity [Shalev-Shwartz and Ben-David, 2014, Lemma 26.9]. Finally, for Rad( H 2 , ˜ S n ) we used the expression for the Rademacher complexity of the class of linear functions on Hilbert space. [Shalev-Shwartz and Ben-David, 2014, Lemma 26.10]. This concludes Step 3.

Step 4. In this step, we finally consider sets S which are sampled from the product measure µ n , i.e. sets where x 1 , . . . , x n are independent data samples with law µ . From steps 1 - 3, we know that

We bound E ( x 1 ,...,x n ) ∼ µ n [ max 1 ≤ i ≤ n ∥ x i ∥ ] by Lemma H.1 to obtain

A similar result follows immediately for the more general function class

Corollary D.3. Under the same conditions as Lemma D.2, we have

Proof. We note that f ∈ F A,Q if and only if f = ˜ f + α with f ∈ F Q and | α | ≤ A . Hence, for any fixed dataset S , we have

by the argument of Step 2 in the proof of Lemma D.2.

A bound on the Rademacher complexity, together with the sub-Gaussian property of the distribution µ , allows us to control the 'generalization gap' in homogeneous Barron spaces.

Corollary D.4. Assume that µ is a σ 2 -sub-Gaussian distribution on R d . Let ( X 1 , . . . , X n ) be iid random variables with law µ and f ∗ a µ -measurable function such that

µ -almost everywhere. Let

Then with probability at least 1 -2 δ over the random draw of X 1 , . . . , X n , the bound

holds for a constant C ∗ > 0 which does not depend on δ, Q, d, µ or n .

Proof. Step 1. From Lemma H.2, with probability at least 1 -δ we have

We denote R n := E x ∼ µ [ ∥ x ∥ ] + σ √ 2 log( n/δ ) for simplicity.

Step 2. Consider the modified loss function

which is bounded by ξ 2 and satisfies | ∂ f ℓ ξ | ≤ 2 R , i.e. ℓ ξ is 2 ξ -Lipschitz continuous. We thus observe that, with probability at least 1 -δ over the choice of random set S = { x 1 , . . . , x n } , we have

by [Shalev-Shwartz and Ben-David, 2014, Theorem 26.5] and the Contraction Lemma for Rademacher complexities, [Shalev-Shwartz and Ben-David, 2014, Lemma 26.9]. In particular

Step 3. By the union bound, with probability at least 1 -2 δ , both the norm bound of Step 1 and the generalization bound of Step 2 hold. In the following, we assume that both bounds hold. Note that

In particular, if ξ ≥ ( A + B 1 ) + ( Q + B 2 ) R , then | f ( x ) -f ∗ ( x ) | ≤ ξ on B R (0) , so ℓ ξ ( f ( x ) -f ∗ ( x )) ≡ ℓ ( f ( x ) -f ∗ ( x )) . Applying the generalization bound with R n = E x ∼ µ ∥ x ∥ + σ √ 2 log( n/δ ) and ξ n = ( Q + B 2 ) R n + ( A + B 1 ) , we find that ℓ ξ n ( f ( x i ) -f ∗ ( x i )) = ℓ ( f ( x i ) -f ∗ ( x i )) for all i by assumption and thus, with probability at least 1 -2 δ , we have

Step 4. Finally, we bound the population risk with the true loss function rather than ℓ ξ . Thus we find that for B R := B R (0) we have

for R ≥ 3 . From Lemma H.3 with R n = E ∥ x ∥ + σ √ 2 log( n/δ ) , we have

Remark D.5. In particular, Corollary D.4 applies if the target function f ∗ is Lipschitz-continuous with B 1 = 0 and B 2 = [ f ∗ ] Lip ≤ [ f ∗ ] B . However, continuity is not necessary, and even noisy labels would be admissible. We do not pursue this generality here.

E Proofs of the convergence theorems

In this appendix, we present the proofs of Theorems 2.1 and 3.3. In these, we combine the upper bound of the Rademacher complexity of the unit ball in homogeneous Barron space in form of the generalization bound of Corollary D.4 with a Γ -convergence argument (to guarantee the convergence of minimizers to minimizers). The main ingredients in the proof of Γ -convergence are

the compact embedding theorem for homogeneous Barron space (to guarantee that every subsequence of f n has a convergent subsequence) and

the direct approximation theorem for homogeneous Barron space (to obtain a bound on the lowest achievable energy using a neural network with m neurons).

We first present convergence proofs in L p ( µ ) in Section E.1, followed by proofs of Γ -convergence in Section E.2. We combine the arguments to prove the statements from the main body of the document in Section E.3.

E.1 Convergence in L p ( µ )

We start by establishing convergence in L 2 ( µ ) at an explicit convergence rate. Then, using this L 2 ( µ ) convergence we will extend this result to general L p ( µ ) .

We introduce one of our main theorem of this section, which gives us an explicit bound of L 2 ( µ ) -loss. For convenience, we denote θ := ( a, W, b ) for the rest of the section.

Theorem E.1 ( L 2 -convergence) . Let ˆ θ ∈ argmin θ ̂ R n,m,λ ( θ ) . If δ ≥ e -n , and f ∗ ∈ F Q ∗ , then with probability at least 1 -4 δ over the choice of random points x 1 , . . . , x n we have

up to higher order terms in the small quantities ( λm ) -1 , m -1 , n -1 / 2 log n .

Proof. Outline. We use Theorem C.5 for L 2 ( µ n ) with f = f ∗ to obtain a function for which ̂ R n,m,λ is low. The empirical risk minimizer (ERM) has even lower risk. The weight decay penalty additionally provides a norm-bound in homogeneous Barron space for the ERM, and we use Corollary D.4 to control the generalization gap.

Step 1. Due to Theorem C.5, there exists a ˜ θ := (˜ a, ˜ w, ˜ b ) ∈ R m × R m × d × R m +1 such that

and

We will always consider δ such that log(1 /δ ) ≤ n . In this regime, plugging-in the bound on the second moments of µ n from Lemma H.4 gives the corresponding bound

with probability 1 -δ . We will assume that this estimate is valid for the remainder of the proof. In particular, since ˆ θ minimizes ̂ R n,m,λ , we find that

Step 2. Next, we bound [ f ˆ θ ] B and | f ˆ θ (0) -f ∗ (0) | ( Q and A in Corollary D.4). We first bound the Barron semi-norm by

Moving on to bounding A , we find from the empirical risk bound

and in particular

With probability at least 1 -δ , we have

by Lemma H.2. Again, we assume that the estimate holds in the following. Hence, the index i for which the minimum is attained in (8) satisfies the bound

Combining the bounds on | f ˆ θ ( x i ) -f ∗ ( x i ) | and the Lipschitz constants of f ˆ θ , f ∗ , we find that

Step 3. Comparing ˆ θ to ˜ θ , we observe that

where we used the fact that ˆ θ is a minimizer of ̂ R n,m,λ ( θ ) and (6). In the following, we use the bound on [ f ˆ θ ] B to control the generalization gap.

Step 4. Recall that [ f ˆ θ ] B ≤ [ f ∗ ] B + 1 λ ̂ R n,m,λ ( f ˜ θ ) = [ f ∗ ] B + O (( λm ) -1 ) . Thus, with probability at least 1 -2 δ , we obtain the bound

from Corollary D.4 for a slightly modified constant C > 0 (with B 1 = 0 ) and up to higher order terms in ( λm ) -1 and n .

Step 5. By the union bound, all probabilistic bounds hold simultaneously with probability at least 1 -4 δ . In this case

up to higher order terms in m , log n/ n, ( λm ) etc.

Since R ( θ ) = ∥ f θ -f ∗ ∥ 2 L 2 ( µ ) , we can interpret Theorem E.1 as a convergence statement in L 2 ( µ ) at a suitable rate. The statement generalizes to L p -convergence at a rate.

Corollary E.2 ( L p -convergence) . Let p ∈ [1 , ∞ ] and ˆ θ as in Theorem E.1. Then there exists a constant ˜ C > 0 depending on E ∥ x ∥ , E [ ∥ x ∥ 2 ] , σ 2 and p such that

Proof. Since µ is sub-Gaussian, we note that all moments of µ are finite: E [(1 + ∥ x ∥ ) q ] < ∞ for all q ∈ [1 , ∞ ) . In particular, if g is a measurable function which satisfies | g ( x ) | ≤ C g (1 + ∥ x ∥ ) for some C ) g > 0 , then

If g = f ˆ θ -f ∗ ∈ B , then by the continuous embedding B ↪ → L q ( µ ) we find that

Recall that | f ˆ -f ∗ | (0) ≤ R n,m,λ ( ˆ θ ) 1 / 2 + C [ f ∗ ] B

We note that Corollary E.2 is generally suboptimal. Indeed, for p ≤ 2 , the stronger bound

holds as L 2 ( µ ) embeds continuously into L p ( µ ) .

E.2 Gamma-expansion of regularized risk functionals

As before, we denote θ n = ( a, W, b ) n ∈ R m n × R m n × d × R m n +1 . Since R ( f θ ) = ∥ f θ -f ∗ ∥ 2 L 2 ( µ ) , Theorem E.1 can be taken as a statement that f ˆ θ n → f ∗ as n →∞ in L 2 ( µ ) . However, this does not tell us about the behavior of f ˆ θ n in a µ -null set, i.e. where the distribution µ provides us no information. This interpolation between known values can be deduced from our next result. We first present a simplified version, in which we assume that we have already taken the limits m,n →∞ before taking λ → 0 . We couple the limits n, m n , λ n below.

We use the notion of Γ -convergence from the calculus of variations. For a brief introduction, see Appendix B. Γ -convergence depends on the underlying topology of the space, and we make the following convention: We say that f λ good - - - → f if f λ → f locally uniformly (uniformly on compact sets) and in L 2 ( µ ) . Other definitions are admissible and lead to the same general theory. Since Barron functions grow at most linearly at ∞ due to Lipschitz-continuity, we note that this is notion of convergence is generated by a metric

at least on bounded subsets of Barron space. This suffices for all applications below and spares us from considering Γ -convergence on more general topological spaces - which is also possible.

Theorem E.3. Let

We denote

Then Γ -lim λ → 0 F λ = F with respect to the notion of convergence good - - - → defined above.

Notably, the Γ -limit of R λ itself would be zero at all points of interest. Rescaling to consider F λ instead has fits into the framework of Γ -expansions considered by Braides and Truskinovsky [2008]. Denote

Proof. Step 1. liminf-inequality. First consider f ∈ F and assume that { f λ } λ> 0 is a family of functions such that f λ good - - - → f . 2 Then by the compactness theorem for Barron functions in coarser topologies (Theorem C.3) we have the following:

Now assume that f / ∈ F and that f λ good - - - → f . We need to show that F λ ( f λ ) → + ∞ . Since f / ∈ F , we see that R ( f ) = ∥ f -f ∗ d ∥ 2 L 2 ( µ ) > 0 . Denote ε = √ R ( f ) and observe that there exists Λ > 0 such that ∥ f λ -f ∥ L 2 ( µ ) < ε/ 2 for all λ < Λ by the definition of the notion of convergence.

In particular, we find that

for all λ < Λ by the inverse triangle inequality and thus

Step 2. limsup-inequality. Again, we first consider the case f ∈ F . Set f λ = f for all λ > 0 and observe that f λ → f as λ → 0 + (trivially). By the same argument F λ ( f λ ) = F ( f ) = [ f ] B for all λ , i.e. the constant sequence is a recovery sequence since F λ ( f λ ) → F ( f ) .

On the other hand, if f / ∈ F , then R ( f ) > 0 and thus F λ ( f ) → + ∞ = F ( f ) . Again, we can use the constant sequence as a recovery sequence, somewhat trivially.

Corollary E.4. Assume that f λ ∈ argmin f ∈B R λ , i.e. f λ minimizes R λ . Then there exists ˆ f ∈ B such that f λ good - - - → ˆ f .

Proof. Clearly f λ minimizes R λ if and only if it minimizes F λ = λ -1 R λ . We note that

In particular, by the compact embedding of Theorem C.3, there exists ˆ f ∈ B such that f λ good - - - → ˆ f up to subsequence. By the properties of Γ -convergence, we conclude that ˆ f is a minimizer of F .

We present a special case of Corollary E.4 in the setting of Proposition 3.2 which exploits the uniqueness of the minimizer. Recall the definition of the radial average in (3) and f ∗ d from Proposition 3.2.

Corollary E.5. Assume that f ∗ (0) = 1 , f ∗ ( x ) = 0 if ∥ x ∥ ≥ 1 and µ satisfies the conditons of Corollary 3.3. Assume additionally that f λ ∈ argmin f ∈B R λ , i.e. f λ minimizes R λ . Then Av f good - - - → f ∗ .

Proof. Step 1. Clearly f λ minimizes R λ if and only if it minimizes F λ = λ -1 R λ . Av f λ is also a minimizer of F λ since the functional is convex and rotationally symmetric, so by averaging in radial direction, we are taking a (continuous) convex combination of minimizers, which is a minimizer again.

Step 2. We find that

By the compactness theorem for Barron functions, Theorem C.3, there exists f ∈ B such that Av f λ good - - - → f (up to a subsequence). Since F λ → F in the sense of Γ -convergence, we find that f is a minimizer of F . Since f is also radially symmetric, we find by Proposition 3.2 that f ≡ f ∗ d .

Step 3. By the exact same logic, we could show that every subsequence of { Av f λ } has a further subsequence which converges to f ∗ d . By a standard argument in topology, the whole sequence converges.

A similar statement can be proved in the more complicated case where f λ is a neural network with finitely many neurons and F λ uses a finite data set rather than a continuous expectation. In this case, the parameter λ must be coupled to the number of parameters m and the number of data points n , such that λ → 0 , but not too quickly. The proof is a more technically challenging variant of those of Theorem E.3 and Corollaries E.4 and E.5, which utilizes the generalization bound of D.4. To this end, we first introduce a new notion of convergence. That is, we define a notion of convergence from the parameter to function. We define a notion of convergence by saying that θ k := ( a k , W k , b k ) good - - - → f iff f θ k good - - - → f as k →∞ .

Theorem E.6. Consider the parameter space Θ m ⊆ R m × R m × d × R m +1 of neural networks with a single hidden layer of width m and the associated functions

Let m n , λ n scale with n according to (1) . We denote

and thus

.

Then almost surely over the choice of data points, we have Γ -lim n →∞ F n = F almost surely with respect to the notion of convergence θ k good - - - → f defined above.

Proof. We use F as in (9) throughout. In the proof we assume that all stochastic quantities in Corollary D.4 and Theorem E.1 are satisfied with probability at least 1 -δ n for δ n = n -2 . The quantity log( δ n ) therefore becomes comparable to log n/n ≪ λ n . Since ∑ ∞ n =1 n -2 < ∞ , we find that all conditions are met for all but finitely many n ∈ N by the Borel-Cantelli Lemma. For questions of asymptotic convergence, we may therefore assume that the statements of both Theorems apply without qualifying for high probability. Note that n -2 ≥ e -n for all n ≥ 2 as needed for Theorem E.1.

Step 1. liminf-inequality. Again, we consider the cases f ∈ F and f / ∈ F separately. First, when f ∈ F , we apply the same method we did in Theorem E.3. For any sequence of parameters θ n good - - - → f , by Theorem C.3 the following holds:

The second inequality comes from [ f θ n ] B being an infimum of weight decay regularizers with any arbitrary probability measure on parameter space.

Second, when f / ∈ F , we need to show lim inf n →∞ F n ( θ n ) = ∞ for any θ n good - - - → f . Wedistinguish two prototypical cases:

[ f θ n ] B → + ∞ as n →∞ . In this case F ( θ n ) ≥ [ f θ n ] B → + ∞ as well by the same logic as above.

lim sup n →∞ [ f θ n ] B < + ∞ . In this case, we take ε := ∥ f -f ∗ ∥ L 2 ( µ ) / 2 > 0 . Then there exists N ∈ N such that for all n ≥ N we have

for all n ≥ N by definition. Additionally

Due to Corollary D.4 and the arguments of Theorem E.1 to control the discrepancy at 0 , we have in this case. Since log n/ √ n ≪ λ n by assumption, we note that

The same holds in the general case by passing to subsequences.

Step 2. limsup-inequality. As in Theorem C.5, the case f / ∈ F follows from the lim inf -ineuality in an essentially trivial fashion. We therefore only consider the case f ∈ F . An approximating sequence in this case is constructed from Theorem C.5 as in Theorem E.1 or Theorem E.3. Namely, we find ˜ θ n such that

Remark E.7. The key ingredients for the proofs of both Theorem E.1 and Theorem E.6 are Theorem C.5 and Corollary D.4, but they are combined differently. While they are paired in Theorem E.1 to obtain a precise rate, they occur separately in Theorem E.6: Theorem C.5 is used for the lim sup -inequality while Corollary D.4 enters in the proof of the lim inf -inequality. Analogously, the condition λ n ≪ log n/ √ n is used in the proof of the lim inf -condition while the fact that 1 m n ≪ λ n is used in the proof of the lim sup -inequality.

E.3 Proofs of the main theorems

The statements of the Theorems in the main body of the text can easily be deduced from the statements proved in this Appendix.

Proof of Theorem 2.1. Convergence in L p holds by Theorem E.1 for 1 ≤ p ≤ 2 and Corollary E.2 (general p ). The proof of uniform convergence follows from Theorem E.6 in the same fashion that Corollary E.4 follows from Theorem E.3. The explicit bound is obtained from Theorem E.1 with δ n = 1 4 n 2 .

Proof of Corollary 3.3. This follows in the same way as the proof of Theorem 2.1 with modifications as in Corollary E.5.

F Theorem 2.1 for finite data sets

Finally, we note that a version of Theorem 2.1 holds if the data set S = { x 1 , . . . , x n } is kept fixed. The proof is a combination of those of Theorems E.3 and E.6, as we deal with a finite approximating neural network, but do not require generalization bounds. The details are left to the reader.

Theorem F.1. We make the following assumptions.

Let S = { ( x 1 , y 1 ) , . . . , ( x n , y n ) } be a fixed dataset of n data points in x i ∈ R d and labels y i ∈ R .

Let the loss function ℓ ( f, y ) be the mean squared error ℓ MSE ( f, y ) = | f -y | 2 .

Assume that λ m is a sequence of parameters such that λ m → 0 , 1 /m ≪ λ m as m →∞ .

Consider the regularized empirical risk functional ̂ R m : R m × R m × d × R m → [0 , ∞ ) ,

Then if ( a, W, b ) m ∈ argmin ̂ R m for all m ∈ N , then every subsequence of f m := f ( a,W,b ) m has a further subsequence which converges to some limit ˆ f ∗ ∈ B uniformly on compact subset of R d . The limiting function satisfies

G Minimum norm interpolation in one dimension

Proof of Proposition 3.1. Any Barron function f is also Lipschitz-continuous, in particular differentiable almost everywhere and f ( b ) -f ( a ) = ∫ b a f ′ ( x ) d x for all a, b ∈ R by Rademacher's Theorem (see e.g. [Evans and Gariepy, 2015, Section 3.1]). In particular, there exist points x + , x -∈ ( a, b ) such that

If f ′ is not constant, both inequalities are satisfied strictly. Under the assumptions, there exists a ∈ ( x 0 , x 1 ) and b ∈ ( x n -1 , x n ) such that

Since the derivative f ′ changes sign, we conclude by [Wojtowytsch, 2022, Proposition 2.5] that [ f ] B = ∫ ∞ -∞ d | µ | where the Radon measure µ is the distributional derivative of f ′ and | µ | denotes the total variation measure of µ . Since f ′ is differentiable at a, b , neither point is an atom of µ and thus

Equality holds if and only if | µ | = µ , i.e. if µ ≥ 0 in the sense of signed measures and if and only if the inequality in (10) can only be satisfied with equality. The first condition means that f must be convex, the second implies that the derivative of f must be constant in the intervals ( x 0 , x 1 ) and ( x n -1 , x n ) . The same can easily be seen for the larger intervals ( -∞ , x 1 ) and ( x n -1 , ∞ ) .

H Sub-Gaussian random variables

In this Appendix we quickly gather some facts about sub-Gaussian random variables. For the reader's convenience, we provide full proofs. Experts are encouraged to skip ahead to Appendix D.

The first property we observe is a bound of expected maximum value of sub-Gaussian.

Lemma H.1. Let µ be σ 2 -sub-Gaussian, and for i = 1 , . . . , n , let x i be a iid random samples from a law µ . Then, the following holds:

Proof. From the sub-Gaussian assumption we have

for some fixed σ > 0 and all λ > 0 . In particular, Jensen's inequality implies the sub-Gaussian maximal inequality

for all λ > 0 . We specifically select λ = √ 2 log n σ 2 , making the bound

Next, we observe the concentration of maximum values among samples of sub-Gaussian distribution.

Lemma H.2. Let µ be σ 2 -sub-Gaussian, and for i = 1 , . . . , n , let x i be a iid random samples from a law µ . Then, with probability at least 1 -δ , the following holds:

Proof. For all t > 0 , we observe that

For fixed t , the bound becomes tightest for λ = t/σ 2 with if

Thus with probability at least 1 -δ , we have

In the following Lemma, we investigate expectation of squared norm of sub-Gaussian near the tail. Lemma H.3. Let µ be σ 2 -sub-Gaussian distribution in R d . For any R > E x ∼ µ ∥ x ∥ , we have the following:

Proof. Recall that

as demonstrated in the proof of Lemma H.1 (consider n = 1 ). Thus

In next Lemma we introduce how the mean of squared norm of sub-Gaussian is concentrated.

2 d

Lemma H.4. Assume µ is a σ -sub-Gaussian distribution in R . Let x 1 , . . . , x n be iid random samples from law µ . Then, with probability at least 1 -δ ,

In particular, if δ ≥ e -n then

with probability at least 1 -δ .

Proof. Firstly since ∥ x i ∥ is σ 2 -sub-Gaussian, from [Honorio and Jaakkola, 2014, Appendix B] we observe that ∥ x i ∥ 2 is (4 √ 2 σ 2 , 4 σ 2 ) -sub-exponential. Next, by independence, for all λ > 0 and for all | t | ≤ 1 4 σ 2 we have the following:

which implies that ∑ n i =1 ∥ x i ∥ 2 is (4 √ 2 nσ 2 , 4 σ 2 ) -sub-exponential. Thus the tail bound of subexponential [Adams, 2022, Proposition 2.38] applied to ∑ n i =1 ∥ x i ∥ 2 yields

Plugging-in s = nt , we have the following:

Lastly, take δ = exp ( -1 2 n min( t 2 32 σ 4 , t 4 σ 2 ) ) . By rearranging the t with respect to δ , we have the following:

This proves the first part of the Lemma. Specifically, we have the following:

The second inequality proves the last part of the Lemma.

Peer reviews:
comments: SUMMARY:
This paper investigates how shallow ReLU networks interpolate between known regions as the number of data points and parameters tends to infinity. The authors prove that minimizers of regularized risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit. The authors study the performance of optimization algorithms by comparing numerical solutions to the minimum norm interpolant in cases where the latter is known explicitly.

STRENGTHS:
The paper has rigorous theoretical proof.

WEAKNESSES:
The paper does not show the effectiveness for practical neural network optimization, such as CNN and Transformer.

QUESTIONS:
The paper does not show the effectiveness for practical neural network optimization, such as CNN and Transformer.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-29

comments: SUMMARY:
This paper investigates the behavior of shallow ReLU networks in terms of convergence to minimum norm interpolants of given data. The authors present provable results when explicit regularization is included and empirical evidence when it is not. The paper concludes with a summary of the authors' empirical insights into the implicit bias of neural network optimizers. . The authors suggest that minimum norm interpolation serves as a useful testbed to study the implicit bias of optimizers, initialization, and regularization, and anticipate the extension of this analysis to deeper networks in the future.

STRENGTHS:
The authors summarize their empirical insights into the implicit bias of neural network optimizers, providing a concise overview of the behaviors exhibited by various algorithms. 

The discussion on the impact of initialization scale, optimization algorithm choice, and explicit regularization is informative and relevant to understanding the behavior of shallow ReLU networks

WEAKNESSES:
Further discussion on the trade-off between fit and regularization, as well as potential methods to balance these factors, would enhance the practical understanding of these observations.
Also, it would be valuable if the authors briefly discussed the potential practical applications or implications of their findings to further motivate researchers in the field.

This work is a bit out of my scope. Please discard my rating for final decision.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper summary:\n* The paper studies the convergence properties of 1-layer Relu nets trained on a regression task with Tikhonov regularisation. The main contribution of the paper is Theorem 2.1, in which the authors prove that under some assumptions on the data distribution and on the optimisation trajectory (or in the limit at they specify in line 110), the model converges to a function f^* that is equal to the optimal unknown data generating function f^* almost everywhere but  f^* has smaller Barron semi-norm. \n * Assumptions of Theorem 2.1: as stated in the theorem and in the main text, the assumption that at each iteration the function f_n minimises the empirical risk given by the current sample is a strong one, but (line 110) the authors specify that the proof holds as long as in the limit of n -> infinity the empirical risk tends to 0.\n * After theorem 2.1, the authors provide two examples in Section 3, first for one-dimensional examples and then for radially symmetric bump functions. In these cases, they provide characterisations of the minimum norm interpolant.\n  * From Section 4 onwards, the authors move on to discuss generalisation, optimisation and implicit bias. \n  * In Line 170 the authors mention the connection between Barron semi-norms and generalisation, following with connections of related work in the space of generalisation and optimisation (though plenty is missing here, see below). \n  * In Section 5, the authors provide a set of experiments in which they test the behaviour of various optimisers on some of the tasks that they discussed previously (optimisers include SGD, SGD with momentum, Adam, BFGS). The experiments assess the effects of data dimensions (Figure 1), optimisers, decay values and gain (Figure 2, where they show the importance of gain in learning smooth convex functions across optimisers), and the gap between the optimisation solution and the optimal interpolation solution at a data dimension (Figure 3).\n\nSTRENGTHS:\nStrengths:\n* The paper tackles an important problem: understanding the convergence properties of neural networks (albeit in a limited setting) and connects it with generalisation. The second half of the paper assesses the impact of optimisation in a series of limited settings. \n* The paper is clearly written, though at times relies too much on previous work and the setting is not clear enough, such as lines 189-200.\n\nWEAKNESSES:\nWeaknesses:\n  * The paper applies to a very limited setting, namely 1 layer Relu nets trained on regression tasks.\n  * The second part of the paper (Section 4 onwards, experiments on optimisation and implicit bias) does not connect strongly with the first. \n  * There is nothing in Thm 2.1 about the implicit bias of specific optimisers. Moreover, this area has been studied extensively in the literature (beyond 1-layer nets), and that literature is not included or mentioned at all in the entire paper. Relevant works include Barrett and Dherin, Smith et al, the momentum recent paper. These works can help explain some of the empirical results shown in the paper, since they show that GD, SGD and SGD with momentum respectively minimise `||\\nabla_{\\vtheta} L||_2^2` where L is the loss function and $\\theta$ are the learned parameters. This quantity is very connected to parameter norms in general networks, though perhaps not in the case of 1-layer Relu nets trained on regression tasks with mean squared error , since there  ``||\\nabla_{W} L||_2^2 = (\\sum_i  f(x_i) - y_i)  \\nabla_{\\vtheta} f(x_i) and \\nabla_{\\vtheta} f(x_i) = Relu’(W x_i + y_i ) x_i``.\n  * The paper title (and parts of the paper) mention “shallow” neural networks. The paper does not work on shallow networks (which are usually referred to networks with a small number of layers) but to 1 layer networks. I suggest the authors replace the wording “shallow” and use the more appropriate language. \n  * The plots are  are lacking relevant information such as x and y axis labels  and hard to read physically (legends are very small, and the lines use the same type of line and similar colours). \n  * Other related works missing: when describing the study of the bias of SGD towards flatness, many recent works are missing, including  Cohen et al,  Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, who show how increasing learning rates leads to SGD traversing flatter areas.\n\nQUESTIONS:\nCan the authors comment on the weaknesses above, address the connection between the theoretical aspect of the work and the optimisation aspect discussed in the second part of the paper, as well as the other aspects highlighted in the weakness section above.\n\nLIMITATIONS:\nThe authors did not discuss limitations in their work (as recommended by the conference). I have highlighted a few limitations above, including the limited applicability inside the 1 layer network family and missing connections with the implicit regularisation literature coming from optimisation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the optimization and generalization for two-layer ReLU neural network models.\nSpecifically, \n* for the generalization part, this paper gives a Barron-norm based generalization bound for the minimize of the $\\ell_2$-regularized squared loss function. The bound involves with three terms: sample size $n$, model width $m$ and the regularization weight $\\lambda$.\n* for the optimization part, this paper conducts numerical experiments on two examples that the minimum Barron-norm solutions can be explicitly obtained, namely, the one-dimensional example and the radially symmetric bump function example. Numerical results reveal that optimization algorithms, initialization scale, and the regularization weight affect the minimum norm interpolant together.\n\nSTRENGTHS:\n* The paper is written clearly and the ideas have been presented in the proper order.\n* The studied topic on the interplay of regularization weight, initialization scaling and the optimization algorithm is important to the deep learning community, which may enhance comprehensive perception for optimization and generalization.\n\nWEAKNESSES:\nIn case I missed something, please correct me if I am wrong.\n\n* The related work description is not comprehensive for me.  Since I am not that familiar with the Barron-norm related fields, I don't know if there exists other works that study the generalization of neural networks in the Barron space. If there indeed exists, it would be better if the authors could make some comparisons on the generalization bound in Theorem 2.1. with others that are related.\n\n* The Baron semi-norm of a general function is usually not easy to assess. In this paper, two examples that can explicitly derive the Baron semi-norm are given: the one-dimensional example and the radially symmetric bump function. \nHowever, these examples are still restrictive in deep learning models. When we look into the definition for the Baron semi-norm, I try to understand this as the expectation version for the $\\ell_2$ norm of the weight parameters, which is easier to calculate.\nIn that sense, I am not sure what are the real advantages for us to consider the Baron semi-norm when compared with the $\\ell_2$ norm?\n\nQUESTIONS:\nIn case I missed something, please correct me if I am wrong.\n\n* I don't understand why Theorem 2.1. shows that *minimizers of regularized risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit.* (Line 26-27).\nAs far as I can understand, the generalization bound is for the gap between the model function and the target function, but why this result can show that the regularized risk minimizer converges to the minimum norm interpolant?\n\n* Also for the generalization bound in Theorem 2.1., it seems that a two-layer ReLU neural network can learn a target function even for $\\frac{m}{n} \\xrightarrow{} 0$. \nTo see this, we let $m = \\sqrt{n}$, $\\lambda =  n^{-1/4}$, which satisfy the condition (1), then the bound (2) goes to zero as $n$ goes to infinity. That means a two-layer ReLU neural network even with the parameter size less than the sample size can learn an arbitrary target function as long as its Baron semi-norm is finite? \nThis is counterintuitive as we usually need to use a large model (with parameter size far greater than the sample size) for learning.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the generalisation behaviour of two-layer neural networks in\nthe over-parameterised regime, where the network can perfectly interpolate its\ndata in many different ways. What can we say about the functions that are learnt\nusing SGD in these cases? The authors address this question using mathematical\nand experimental work.\n\nThey first consider the setting where for a fixed dimension $d$ (I think), a\ntwo-layer ReLU network of width $m$ trained on a data set with $n$ points with\nlabels generated by a target function $f^*$. In the limit where $n$ and $m$ go\nto infinity, they show that minimisers of the mean-square loss with small weight\ndecay penalty, resemble a minimum norm interpolant, which is the equivalent of a\nmaximum margin classifier in classification.\n\nSince there is no guarantee that a neural network trained using gradient descent\nwill converge to an interpolant, or which minimiser of its loss it converges to\nif it converges to zero loss, the authors also conduct a series of numerical\nexperiments to analyse the functional norms of neural networks trained using SGD\non low-dimensional inputs. The experiments confirm the importance of\ninitialisation scale, and small initial weights in particular, and finds some\npeculiarities in the behaviour of Adam.\n\nPS: While I know some of the papers on minimum norm interpolators, I am not\nknowledgeable in the proof techniques used in this paper.\n\nSTRENGTHS:\n- The author provide a rigorous statement about the behaviour of a certain limit\n  of two-layer neural networks.\n- They provide a numerical exploration of neural networks trained using\n  different gradient-based optimisers, including an analysis of the trade-offs\n  between explicit regularistaion and implicit bias.\n\nWEAKNESSES:\n- A minor, but important point: the axes labels and titles of the plots are way\n  too small to be readable.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the global minimizer of the empirical risk with $\\ell_2$ norm regularization of the parameters. It considers a one hidden ReLU layer architecture with square loss. \nThe main result roughly states that for some asymptotic regime on $\\lambda$ (reg parameter), $n$ (number of data points) and $m$ (network width), the minimizers of the considered regularized loss converge towards a minimal norm interpolant. Besides this asymptotic result, it also provides a convergence rate (in $L_2$ functional norm) of the estimated function towards the truth.\n\n-----------\n\nI acknowledge having read the author's rebuttal. The authors answered my main concerns and I thus raised my score. I still think that the writing needs a lot of polishing though and would advice the authors to take into account all the suggested modifications in their revised version.\n\nSTRENGTHS:\nI find the convergence result towards the truth very interesting, ie the second part of Theorem 2.1.\nGetting non-asymptotic generalisation bounds for the considered problem is of particular interest and I really enjoyed the proposed result, which gives nice dependencies in the different parameters $\\lambda,n,m$ and $[f^{*}]_{\\mathcal{B}}$.\n\nI think this result alone deserves a deeper investigation/understanding, as I believe it to be the most significant contribution of this work: I would suggest the authors to put an emphasis on this particular point of their main theorem as being their main contribution.\n\nWEAKNESSES:\nOverall, I find the paper misleading. The writing choices make the message unclear: I am unsure of the conclusions/insights brought by this work. Additionally, I think that the paper lacks contextualizing the results with the literature; numerous works are not cited, while they cleary deserve to be discussed in my opinion.\nLastly, I find the overall contribution rather weak: to me, the only real contribution is the Theorem 2.1, which is not discussed enough and might not be a strong enough result to justify a publication alone. The remaining of the paper is then mostly corrolaries of Theorem 2.1 and numerical experiments, which are of minor interest to me.\nI detail all these points below.\n\nFirst, I find the writing misleading/message unclear. Already in the abstract, the following sentence is problematic: \"Our analysis yields convergence to a minimum norm interpolant [...]\". This paper is actually about the convergence of the global minimum towards the the min norm interpolant when $n,m,\\frac{1}{\\lambda}$ converge to $0$, but I first understood this sentence as \"some optimisation algorithm converges towards the min norm interpolant when $t$ (iterations of opt algorithm) go to $\\infty$\". The abstract does not make it possible to distinguish between both cases, while the latter is actually much stronger.\nThis might seem a small detail, but I think already illustrates how the message is not totally clear and it is sometimes hard to understand where the authors want to go.\nThe other main confusion to me comes from the Section 3. I only understood its purpose once I read Section 5. Actually, I think Section 3 does not deserve so much space, as its goal is only to provide two examples where the min norm interpolant is exactly known (which are not original contributions). I would prefer these results to just be small paragraphs in Section 5.1 and 5.2. \nThe message is not even made clearer in the conclusion: only one sentence summarises the paper contribution and interest, while the remaining of the conclusion are experimental observations that could easily be postponed to the appendix.\nAs another example, implicit bias is mentioned a lot in the literature but is never described, making it hard to follow for a non-expert.\n\nA lot of significant related works are not cited, while they clearly deserve to be discussed into details. \nFor example, there are many works that aim at describing minimal norm interpolants which are not even cited here [1,2,3]. Also, the semi-Baron norm in the unidimensional case has also been characterised by [4].\nThe seminal work of [5] is missing in the implicit bias paragraph. Actually the implicit bias does not exactly consider the same norm on the parameters, as it also counts the bias terms (see [2] for a further discussion on this aspect). Is there a particular reason you are not counting the norm of the bias terms here?\nAbout the generalization bounds for small Baron norms functions, I believe it to be closely related to [7,8]. How are the result from Appendix C different?\nIt has recently been proven by [9] that the implicit bias of GD does not always lead to minimal norm interpolants, but sometimes to different kinds of interpolants: this contradicts some of the claims made in some paragraphs of the paper (eg  in Section 4).\nAbout the conservation of symmetries when training neural networks, I guess you could also mention [10].\n\nThe main result of the paper is Theorem 2.1. I think it deserves a much deeper discussion. The paper gives no idea of how the authors manage to prove it, where lie the difficulties in proving it, what are the reasons for the different needed assumptions/scaling of the parameters? In particular, I would have believed that the Theorem would hold with only assuming $\\lambda_n\\to 0$ and $m_n\\to\\infty$. What is the reason for needing the second limit in equation (1) ? I believe it hides the difficulty of the proof, as the first part of Theorem 2.1 seems easy to me: the result would be a stronger contribution if it indeed is not so easy to prove. \nActually I really like the second part of the Theorem (rate of convergence) and believe it to be a significant result. I would have liked a deeper discussion on it. Why do we need $\\mathbb{E}[\\|x\\|]+\\sigma^2\\geq 1$? I believe this is not really needed as we could otherwise deal with a rescaling argument.\nAbout the fixed $n$ setting, [6] actually proved that having any $m\\geq n+1$ does not change the minimum of the problem (ie having $m=n+1$ or $m=\\infty$ is the same), so I actually think having separately $\\lambda_m\\to 0$ and $m\\geq n+1$ is sufficient in that case.\n\n-----------------\nMinor remarks:\n- in Section 2, why postpone the definition of $\\mathcal{B}$? it can be very shortly defined\n- In Section 3.1, the minimum might be different as you also need to consider non-smooth functions. However, using the characterisation of  [4], it is actually the correct minimum\n- Theorem 3.3 only seems to be a corollary of Theorem 2.1 and Proposition 3.2 (and thus deserves to be called a corrolary in my opinion)\nAlso, $f^*$ is not precised here.\n- Section 5: the implicit bias also depends on the initialisation regime -> it should also be mentioned\n- the sentence line 239 is not clear. Actually, what is happening here seems close to what is described in [2]: for small initialisation, the implicit bias towards min norm (counting bias terms) interpolant kicks in, which leads to the unique min $|x|$. For larger initialisation, there is no such implicit bias and only an explicit regularisation (which does not count the bias terms here) and thus allows for less sparse minima as $f(x)=\\max(|x|,1)$.\n- for $\\lambda=0$ and large $\\alpha$, it seems we are in the NTK regime. In this case, we have another implicit bias towards min spline interpolant (see eg [11])\n- in Section 5.2, what is $n$? \n\n-----------------\nReferences:\n[1] Debarre T, Denoyelle Q, Unser M, Fageot J. \"Sparsest piecewise-linear regression of one-dimensional data\"\n[2] Boursier E, Flammarion N. \"Penalising the biases in norm regularisation enforces sparsity\"\n[3] Stewart L, Bach F, Berthet Q, Vert JP. \"Regression as Classification: Influence of Task Formulation on Neural Network Features\"\n[4] Savarese P, Evron I, Soudry D, Srebro N. \"How do infinite width bounded norm networks look in function space?\"\n[5] Lyu K, and Li J. \"Gradient descent maximizes the margin of homogeneous neural networks\"\n[6] Wang Y, Lacotte J, Pilanci M. \"The hidden convex optimization landscape of regularized two-layer relu networks: an exact characterization of optimal solutions\"\n[7] Kurková V, Sanguineti M. \"Bounds on rates of variable-basis and neural-network approximation\"\n[8] Bach F. \"Breaking the curse of dimensionality with convex neural networks\"\n[9] Chistikov D, Englert M, Lazic R. \"Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs\"\n[10] Hajjar K, Chizat L. \"Symmetries in the dynamics of wide two-layer neural networks\"\n[11] Jin H, Montúfar G. \"Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks\"\n\nQUESTIONS:\nWhat is the main message of this work?\nWhat are the technical difficulties/main novelties of Theorem 2.1?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}