Minimum norm interpolation by perceptra: Explicit regularization and implicit bias
Authors: Jiyoung Park
Ian Pelakh
Stephan Wojtowytsch
Venue: NeurIPS 2023 poster (2023)
Keywords: Artificial neural network
interpolation
explicit regularization
implicit bias
weight decay
Barron class
TL;DR: We show that shallow ReLU networks converge to minimum norm interpolants of given data: Provably if explicit regularization is included and empirically if it is not (at least for suitable initialization).

Abstract: We investigate how shallow ReLU networks interpolate between known regions. Our analysis shows that empirical risk minimizers converge to a minimum norm interpolant as the number of data points and parameters tends to infinity when a weight decay regularizer is penalized with a coefficient which vanishes at a precise rate as the network width and the number of data points grow. With and without explicit regularization, we numerically study the implicit bias of common optimization algorithms towards known minimum norm interpolants.

Minimum norm interpolation by perceptra: Explicit regularization and implicit bias
Jiyoung Park Department of Statistics Texas A&M University #.dydd15510@tamu edu
Ian Pelakh Department of Mathematics Iowa State University
Stephan Wojtowytsch
Department of Mathematics
University of Pittsburgh wo @pitt. .euuu
Aspelakhh@iastteee edu
Abstract
We investigate how shallow ReLU networks interpolate between known regions. Our analysis shows that empirical risk minimizers converge to minimum interpolant as the number data points and parameters tends to infinity when weight decay regularizer is penalized with coefficient which vanishes at precise rate as the network width and the number of data points grow. With and without explicit regularization, we numerically study the implicit bias of common optimization algorithms towards known minimum norm interpolants.
Introduction
Modern neural networks mostly operate in an overparametrrized regime, i.c. they possess more tunable parameters than the number of data points contributing the loss function. [Safran and Shamir
20188a E et al. (209bb), Du et al. 2018), Chizat and Bach 2018] associate overparametrization with better training properties, and Belkin al. 2019 2020 find to enhance statistical generalization (see also [Loog et al., 2020 for historical context). For many architectures, overparametrrizatin
leads to the ability to any values y; given of data points {æ1;-.., Nn and Cooper (2018) shows that generically, the set weights for which the neural network interpolates prescribed values submanifold of high dimension and co-dimension in parameter space. Which solution the manifold is dynamically chosen by an optimization algorithm, and which solutions have favorable generalization properties, is active area of research theoretical machine learning.
Current practice to estimate model's generalization to previously unseen data by assessing its performance hold-out set (a posteriori eиror estimate) by using uniform estimates the generalization of all elements of a function class (a priori estimate if the function class encoded ahead of time by explicit regularization, posteriori if membership is determined after optimization). Neither approach yields information on what neural network does outside the support of the data distribution, topic of great interest for the study of distributional shift and adversarial stability.
The main contribution of this work is split between two complementary lines of investigation:
1. We prove that neural network minimizers of regularized empirical risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit (Section 2 Our result improves previous works to more general settings (See Remark (.2).
2. Training a neural network generally corresponds to solving a non-convex minimization problem. While general there is no guarantee that training algorithm finds a global minimizer of an empirical risk functional. Even if convergence holds, it unclear which minimizer is selected (in the overparametrized regime, where the set of minimizers is a high-dimensional interpolant known (Section 3),
provide convergence guarantees for empirical risk minimizers, in
manifold). In settings where the minimum compare numerical solutions to theoretical predictions to better understand )) the predictive
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

power of theoretically studying empirical risk minimizers and (2) the implicit bias of different optimization algorithms (Sections I 35. We believe this be useful benchmark problem for better understanding of explicit regularization and implicit bias in optimization.
Minimum norm interpolants are the regression analogue to maximum margin classifiers. They are associated with favorable generalization properties and relative stability even against adversarial perturbations.
1.1 Previous work
Minimum norm interpolation. In classes of linear functions, minimum norm interpolation has long history as ridge regression (minimal e (--norm) or as the least absolute shrinkage selection and operator (LASSO, minimal e' nnorm)). To the best of our knowledge, minimum norm interpolation by neural networks has only been studied for shallow neural networks in one dimension by |Hani] [2021] }. Debarre et al. [2022], Boursier and Flammarion [203] and in odd dimension for certain radially symmetric data by Wojtowytsch [022]. For classification problems, minimum norm/maximum margin classifiers were considered by E and Wojtowytschi [002]. For finite datasets, the set of
minimum norm interpolants was characterized by Parhi and Nowak [2021]. parametrization of the same function class by neural networks with multiple linear layers and single ReLU layer induces different concepts of minimum norm interpolation as studied by Ongie and Willett [2022).
Implicit bias. The implicit bias of parameter optimization algorithms has been studied for gradient
flows with infinitely wide, but shallow ReLU networks by Chizat et al. (2019), I Jiin and Montúfar
[2020] and for stochastic gradient descent and diagonal linear networks by Pesme et al. [2021]
Chizat and Bach (020) prove convergence to a maximum margin classifier for infinitely wide ReLU
networks with one hidden layer if the parameters follow the gradient flow of an (unregularized)
logistic loss risk functional. Many authors, including Damian et al. [202]] Li et al. [202]] Wu et al.
[0022] and Wojtowytsch 2020). study the bias of SGD towards solutions at which the loss landscape is 'flat' in the parameter space. Hochreiter and Schmidhuber 1997] conjectured such minimizers to have favorable generalization properties. In many cases, minimizers tend to be flatter if the parameters associated to them are not excessively large. Yang et al. [2021) describe several phase-transitions in parameter space for the relation between flatness parameter space and generalization. Zhou et al. 2020) compare the implicit bias of SGD and ADAM. |mmthh et al. [2021] Barrett and Dherin [2020] argue that gradient descent resembles gradient flow modified (regularized) loss landscape more closely than the gradient flow of the original loss function. Hajjar and Chizat 2022) investigate the
role of symmetries optimization.
Barron space. The Barron class is adapted to ReLU networks with a single hidden layer and weights of bounded average magnitude. Slightly different versions of the same function space have been studied by Bach (017), E et al. [2019a c Ongie et al. [0199] I, E and Wojtowytschi [2020] 2021
Caragea et al. (2020), Parhi and Nowak [2021 Wojtowytsch [2022], Siiegel and Xu [2022 2023
under various names such as F1, RadoRd- BV or the variation space of the ReLU dictionary.
1.2 Preliminaries
Conventions.
f always denotes ---sub-aaussian probability measure on the data domain R i.c.
S.l. Ex- exp (A[I - EXIX))) C exp 12
3 3C >0
All norms are Frobenius norms (o-norm for vectors). For N, S, [In,1.-.... Fn,n) is set of n iid samples from the distribution 14, independent of Sn,' for n' + n. When unambiguous, we denote Fn,i = rú. We take {(f, 3) [f y/2 as the mean squared error/f2-loss function, but we remark that the theoretical analysis remains valid if LMSE is replaced by e'-loss Huber pseudo- Huber loss
[If - y12
if ly - fl < 1 2/y fl if by fl 2121 1
Enabss. y)
hn, v 1 11
For m and (a, W) T mmxd g=+1 let
m
((a, Rd + R, f(a.wor(x)) - bo+ Σ
a; oww x + bi),
a(2) = ReLU(=) = max(2.0).
1=1

---

Le. Fla.w.s) is ReLU network with a single hidden layer and weights W and biases b. The vector wi E R& is the i-th row of the matrix W. For m, n E N and À 2 0, we denote the regularized empirical risk functional as Rn,,, _ Rmxd pm+1 [0, 00):
2n Σ e (F(a, v.) A (1a + [
Rm laa W,b)
Concepts. We introduce what we dub 'homogeneous Barron space' B heuristically here, and in greater detail in Appendix [ As a measure of magnitude of the function, consider the weight decay (or Tikhonov) regularizer 1 [||a]]2 || (lrat) of the parametrized function f(a,W,)) which does not control the magnitude of the bias vector.
Note that the function class f(.,W.5) and its complexity do not change when we consider representa- tions of the form fm (x) = 1
m Ei=1 a; a(w, x + b;) with a regularizer 2m
_ (lall + || ( =
I E (aî + [|==1 A continuum analogue to these functions represented as an 'empirical aver- age' over individual neurons a general expectation representation f=(x) = (a,, (wo [w]]]
2m
b)] for some probability distribution
parameter space and regularizer 1 (..(,=,))==l
As the parametrization of a function by a neural network is generally non-unique, we define the Barron semi-norm [/]s inf (x:f==f) ())= (a + [|| the lowest value attained by the regularizer over all possible parametrizations. For more comprehensive understanding. interested
readers may refer to Appendix C
2 General convergence result
We first state general convergence result to minimum norm interpolant of given data generated by functions in the homogeneous Barron class B.
Theorem 2.1. Take μ. Sn, Rn.m. > as in Section 1.2 f E B, and let y; = (x;) 1,...,n. Assume that m, X are paramelers which scale with as Mn, An such that
1)=2 mn
log
lim Amp +
= 0,
lim
= 0.
An mm
An vn
Then almost surely over the random selection of data points Sa, the following holds: If (a, W,b), b)n E argmin Rum,,i for all n E N, then every subsequence of fn := fla,W.)), has further subsequence which converges some limit f B with f f p-almost everywhere and and uniformly on compact subsets of R... Ef | + + 2 2 1, thenfor all n 2 2, the following explicit bound holds up to higher order terms in
[f*] < [f Convergence holds in LP (1) for all
n, m = mm. A = An with probability at least 1 1/n²:
If(a,, f- I
[
E
+
E |-|| +
log n
+ +A(f's).
(2)
Remark 2.2. Theorem 2.1 extends previous results of et al. [2019a] in several ways:
1. We allow for general sub-Gaussian rather than compactly supported data distributions.
2. We do not control the magnitude of the bias variables.
3. Our results apply to fl-loss, which is neither globally Lipschit-comtinmuus nor bounded.
4. In limiting regime, we characterize how the empirical risk minimizers interpolate in the region where no data given by proving uniform convergence to minimum norm interpolant.
sum, the first three points necessitate a more careful technical analysis than the settings of prior work, and the unboundedness of data and loss introduces additional logarithmic terms not present forrE et al. (2019a). To the best of our knowledge, the last point novel and our work the first to use the notion of L-convergence in this context.

---

If the data-distiin distribuion is supported on the entire space R (e.g. a non-degenerate Gaussian), then a small, potentially compact and generally
f = f In many cases, however, js is supported low-dimensional subset M the data space. In this case, the function f is only known on the closed set M _ Idd As result, there are many f E B such that f = f on . while f # f on R in general. The subsequential limit is one of these functions which has a minimal semi-norm [f]g.
Thus, beyond knowing that fn asymptotically fits the function f* perfectly at known data, Theorem 2.1 provides information about how it may interpolate at points where μ has no information. Such knowledge is of interest when a population may naturally evolve in time (distributional shift) of if fn applied to a new problem with similar features but distinct geometry (transfer learning).
The proof of Theorem 2. is given in Appendix In the proof, we combine a direct approximation theorem construct risk competitors with Rademacher complexity-basee generalization bounds. Concentration inequalities are used bound tail quantities. The scaling conditions that logn < X
v and < À are used to ensure that the risk functional has sufficiently strong regularization. They can be weakened the data measure i is supported on a finite set points or the function f* can be represented by neural network with a finite number of neurons respectively. For instance, an analogous statement holds with a simpler proof for fixed data-set S n data points if A is coupled to such that mmm 0. In this case, take the empirical distribution μ Exes & as the population and do not need to bound the generalization gap. This model can be considered appropriate when n < m (heavy overparametrization)) precise statement is given in Appendix
The proof remains valid if we only assume that
1 lim sup - Rm,,, An (ans Wn, b,.) inf ([fl: f= / almost everywhere),
1=+000 An
Le. if (a, W. b)n, parametrizes a function of low excess risk. In the proof, we obtain a more precise priori Lipschitz-bound, it is possible obtain rate of convergence in (P()) for < oo by interpolation. For uniform convergence on compact sets outside the support of 14, we do not obtain a rate in this work.
version of @ Using
The proof of uniform convergence on compact sets utilizes the notion of r-convergence from the calculus of variations, very stable notion of convergence which implies that minimizers of approximating functionals converge to the minimizer of a limiting functional. This notion has recently made inroads into machine learning applications and was used e.g. by Neumayer al. 2023).
All results can easily be generalized any more general function class which admits the three key ingredients: A bound on its Rademacher complexity, compact embedding theorem, and direct approximation theorem.
Minimum norm interpolants
3.1 One-dimensional example
In one dimension, Wojtowytsch 2022 Proposition 2.5] shows that [f]s 12c [x) da for any smooth function f R + R which satisfies f'(x) = at some point x E IR - see also previous work by Li al. [2020]. E and Wojtowytschl [2020). This mne-iienniooall case is, in fact, the simplest case of a general characterization
Barron functions in any dimension by (Ongie et al. [2019].
Consider the task of minimizing [f]s under the condition that f(x) |x| if [x| 2 1. Then the minimum is attained for any smooth convex function f which satisfies the constraint since
2= r01 (*)) & S
dr = [f]s
with equality if and only if 1" 2 0. The second derivative is interpreted as a Radon measure. For piecewise linear functions, this corresponds fz (<x;)) Il over the non-smooth points x4. More generally, the set of minimum norm interpolants of one- one-dimensional convex data was characterized by (aaaaeee et al. [201]].
estimate holds for non-smooth Barron functions if the
to summing (f
Proposition 3.1. Let Fig < ... < an and y; = (x,) for a convex function f and i = 0,..., n. If 3/ yo and yn Un-1, then is a minimum Barron norm interpolant of the dataset { y1) V

---

if and only if is convex, f(x,) = for all i O...., and
y1 Yo f(x)
and f((x) On Un-1
for x <
for I' > In-1-
I1 20
In
In-1
The two given slopes are the largest values that are required for derivatives at any point. We give a proof of Proposition in Appendix [G] In full generality, minimum norm solutions have been characterized by Hanin [202T] using matching convexities to achieve minimal total curvature.
In recent article, Boursier and Flammarion [0233 show that if the full Barron norm is controlled, i.e. if the magnitude of biases is included in the regularizer, a specific convex function selected. This corresponds minimizing functional
S f" (+)x x2 dæ rather than S (
The first functional prefers f" to be large close to the origin, if has to be large anywhere. All break points occur as close the origin possible, in particular: Either at the origin or data points. Unlike the Barron semi-norm penalty studied in this work, which does not select a specific minimum norm interpolant, the Barron norm penalty thus induces sparse neural network interpolant.
3.2 Radially symmetric bump function
Another setting where we have explicit minimum norm interpolant is when we a bump function for radially symmetric data. Recall result of Wojtowytshh [2022 minimum norm fitting of certain radially symmetric data.
Proposition 3.2. (Wojtowytsch 2022
Theorem 3.1] Let be an odd integer and
F= C.(")) f(0) and f(x) 0 if | 1}.
Then there exists a unique radially symmetricc function fä RRd _ TR. such that fā E frggmnne>ss)) The norm of minimizers grows (ola-solfjjss!! = 3.7.
We note that the existence of minimum norm interpolants which are not radially symmetric is excluded, but f E frgmin is any other minimum norm interpolant, then its radial average Av coincides with fū: Av = fā- Here
[ (so(a)
((Ox) = - [(xx] u) dr
Av f(x)
where is the uniform distribution (Haar measure) on the group of rotations and n. is the uniform distribution the d 1-dimensional sphere in Rdd. In [Wojtowytsch] 20222 Section 6), an algorithm is given to find the minimum norm interpolant f by numerically solving a one-dimensional polynomial approximation problem and linear system moment conditions.
The uniqueness statement allows us to strengthen the result of Theorem 2. in this case. A natural setting is to no mass elsewhere the unit ball. It should have mass everywhere outside the unit ball. Under this natural setting, have stronger result of Theorem 2.1
SabbGausiian data distribution μu which gives positive mass to the origin, but has
Corollary 3.3. Take μ, Sn, Ru,m, X as in Section 1.2 and assume in addition that
7. μ({0)) > 0 (positive mass at the origin).
2. ((B((0) \ {0))= = (no mass elsewhere in the unit ball).
3. μ(U) > 0 for any open set U TRd (0) (mass everywhere outside the unit ball).
Assume that m, X scale with n as in T Almost surely over the random selection of data points in S.. the following holds: If (a, W,b), b), E argmin Rn, for all n E N, then sequence of radial averages Av fn of : (la,W,)). converges to fa as in Proposition 3.2 Convergence holds in T?? (n) for MSE loss (with an explicit rate) and uniformly on compact subsets of RRd (without rate in this work).
5

---

In Corollary 3.3 we guarantee convergence to the unique radial minimum norm interpolant fü (at least for the rachal average), while Theorem2.I we may have different subsequences that converge to different minimum norm interpolants. The proof is given in Appendix
4 Relating Interpolation, Optimization and Generalization
For given bounded set K S Rd Theorem 2.1 states that for large number of neurons m N, a large number of data points n E N, and a small penalty >> O, minimizers of the empirical risk functional R., resemble a minimum norm interpolant everywhere in K. As the Barron semi-norm controls the generalization gap (see Appendix D) and minimum norm interpolant has minimal Barron norm by definition, this suggests that minimum norm interpolants are optimal in terms of generalization, least when arguing from this upper bound.
Many authors, including Safran and Shamir (018b). Venturi et al. 2018 demonstrate that neural network training non-convex optimization problem. As such, is guaranteed that numerical optimizers (1) converge to interpolants at all, and (2) select minimum norm interpolants out of the large set of different neural networks which interpolate given data, even when a regularizer is included in the training loss functional.
On the other hand, there settings where an optimization algorithm selects minimum norm solution even without explicit regularization. This is easily proved for gradient descent on the overparametrized least squares regression problem (a) condition a = O and n < m = d. Using entirely different methods, Chizat and Bach [2020] prove a similar result for binary classification by shallow neural networks with logistic loss. For regression problems using neural networks, analogous results are not available to the best of our knowledge. This in part motivates the following numerical investigation. Namely, we are interested in exploring the effects of explicit regularization and the implicit bias of optimization algorithms toward minimum norm interpolants. Knowing the analytically optimal solution in between given data provides us the opportunity to compare optimizers on a deeper level than merely testing their performance on unseen data generated from the same distribution.
-i= la x; V/ with initial
As seen in Figure [ the radial profile r + fā of Wojtowytsch (2022] 's minimum norm interpolant få is so close to 0 on [ra, oo) to be virtually indistinguishable from numerically for some ry < which decreases in d. Indeed, the first (d - 1)/2 derivatives of vanish at = 1 due to [Wojtowytsch] 2022 Appendix D.1] for universal constant C > O. In partieular, if d is large, ( (0j) is negligible compared to the approximation error d? /m for any reasonable dimension d and network width m. Consequently, the rescaled function hī(x)) = (raar meets the constraint h = outside B, (0) almost exactly and has the smaller Barron semi-norm [Mile- = compare numerical solutions to the interpolation problem in Corollary
due to [Wottowytsch/ 2022] Lemma 4.1] and 0 S (x) S C - ||x]|²) [||
For this reason, 3.3 to rescaled versions of få rather than fü itself, at least in high dimension. For r below the threshold value Td. there is no noticable trade-off between rescaling (ix) and data-fitting. For larger values of r. the Barron semi-norm is reduced more significantly, but the data fit becomes appreciably worse
ra [fill.
5 Numerical Experiments
Our main goal in this section is to gain more precise understanding of different optimization algorithms by comparing numerical solutions to known minimum norm interpolant. We consider the two settings which minimum norm interpolation by Barron functions best understood: One- dimensional and radially symmetric functions. As benefit, we can easily visualize the numerical results in both settings. We focus on three questions of interest.
1. Explicit regularization. If A > 0 is moderately small and m, n are large, then global minimizer of Rm.. resembles a minimum norm interpolant between known data points due to Theorem 2.1 Is the minimizer which we find numerically close to to a minimum norm interpolant, does it merely fit the function at known data points?
2. Implicit bias. If m, n are large, does a training algorithm select a minimum norm interpolant out of potentially many possible solutions without explicit regularization (i.c. for X = 0)?

---

Minimum norm Interpolant nts
Minimum norm interpolants
notimal Interpcltion. din
rescaled with 0.55
ootimal interpoletion.
15 rith
octimal Interpolation. 31. rescaled
0.6
0.-
0.0
0.0
Figure 1: Left: In Dimension d = 31, the minimum Barron norm solution fä satisfies f2 = on R B,, (0) for ra = 0.4 to high precision, albeit not exactly. The rescaled function fā (raa) is a suitable candidate for minimum norm almost-interpolant to high accuracy. Right: For later use, we consider more aggressively rescaled functions fā (raz) for d = 15, 31 with lower semi-norm, but worse data fitting properties. We note that the rescalings of these functions which have essentially the same slope as f3 at r 0 appear to coincide. We conjecture that this statement allows for more rigorous formulation.
3. Learning symmetries:
The optimal minimum norm interpolant f described in Proposition 3.2 is radially symmetric and satisfies fJ S 1. Proposition 3.2 does not rule the existence of other minimum norn interpolants which are not radially symmetric. Does optimization algorithm generally find solutions which are (approximately) radially symmetric and confined to the interval [o, 1j? A similar consideration applies in a one- dimensional investigation with reflection symmetry.
The third question is of particular interest for algorithms like ADAM, which operate coordinate-by- coordinate and do not respect Euclidean isometries. By comparison, we expect that SGD. initialized radially symmetric configuration, preserves Euclidean isometries. More experiments similar settings can be found in Appendix
5.1 One-dimensional experiments
We consider the classical interpolation problem of numerical analysis: Fit values (2,) E R at points
r, TR. for i E (1...., n). In contrast to classical numerical analysis, we consider overparametrized
Rel.U-networks with a single hidden layer as our model class. As in Section 3 we select f (x) = | for simplicity.
In Figure 2 (x)
a ReLU network with a single hidden layer of width m 200 was trained to fit
x at a symmetric set containing 15 equi-spaced points in 21, Optimizers included
SGD (with learning rate 7/ = 5 10-5 and momentum μ = 0.99), SGD (1) = 10-2 0), ADAM (n 5- 10 and default parameters) and the quasi-Newton L-BFGS method. Deterministic gradients based on the n = 30 sample points were used. The final training loss was below 10 10-4 on average. The network weights were initialized by scaled uniform Xavier initialization, i.c. uniformly in symmetric interval of length 2a /6/ (nin Nout) where 7ix and nout denote the number of input- and output-units to a layer respectively. The 'gain' factor was selected as a E {0.5, 1, 5). Without weight decay and for small gain, the optimizers find a solution close to the smallest possible minimum norm interpolant f(x) = xx] The larger the parameters for initialization gain and weight decay penalty, the closer numerical solutions f(x) = max([c]] 1).
the largest possible minimum norm interpolant
We observe that higher gain factor corresponds to faster initial training. but high gain like a = 5 produces interpolants which are non-convex without regularization, while a lower gain factor produces convex interpolants in longer time. This observation agrees with the findings of Chizat et al. [219], who dub the large & setting the 'lazy training' regime and associate it with worse generalization performance. [ssme al. [2021] eloquently put it: "there is tension between generalisation and optimisation: a longer training time might improve generalisation but comes at the cost of... longer training time."
m is large and a is not too big, the variation of solutions produced by training algorithm vary less over different stochastic realizations Appendix A for experiments for 1, 000. The
7

---

ga.n - U.S. witth 200, lambia - D.0
0ain - 1.0, width - 200 lamrods - 0.0
g2in - 5.0. eth - 200. lambes - 0.0
23
20
1.5
032h 200 0.002
0ain - withh 0.002
sain 5.O. Witth 0.002
03
02
200. lambdo 0.005
wiith 200. lambdo 0.005
5.0. Wetth
lambi 0.005
20
13
14
L0-
LO-
03
05
05
02
Figure 2: We compare numerical approximations of a target function for Momentum-Gii (red), GD (magenta), ADAM (purple) and L-BFGS (brown). The target function is drawn in blue and the natural cubic spline in green. For each algorithm, we plot one representative solution to study symmetry selection properties. Vertical grey lines indicate known training data points. The initialization has gain a E {0.5, 1.0, 5.0} in the left, middle and right column. The weight decay penalty is A E (0, 0.002, 0.005) } (top. middle, bottom row). For all optimization algorithms, the final loss is approximately 0, 2. 10 and 10-3 respectively.
For small gain, the optimizer selects minimum norm interpolant/convex function. For large gain, the interpolant is often non-convex for any one of the optimization algorithms, unless the weight decay is given a positive value. A specific type of minimum norm interpolant in the large set of possible solutions seems to be selected by specifying optimization algorithm, weight decay and initialization. Evidently, initialization and weight decay have far greater influence than the choice of the optimizer. Small gain seems to preference the sparsest solution f(x) = |x]] has predicted with bias penalty by Boursier and Flammarion [2023]. With larger weight decay, the solutions become more convex and more symmetric, but the accuracy of data fitting decreases. Visually, the second order L-BFGS method appears to be the least affected by different choices in initialization.
dynamics are close to those of a limiting 'mean field' model studied by Chizat and Bach (2018), Rotskoff and Vanden-Eijnden (2018), Mei al. [2018), (Sirignano and Spiliopoulos [2020a] and Wojtowytsch (2020) In these works, the limiting model is typically derived with a factor 1/m outside the function definition, which is implicit in the initialization here since Thin + Nout m for both layers and the ReLU activation is positively one-homogeneous. Global convergence to minimizer (but not necessarily minimum norm solution) guaranteed (up to certain technical assumptions) by Chizat and Bach (0188) and Wojtowytsch (2020).
For comparison, also present the natural cubic spline interpolant, i.e. the function f which minimizes the stronger curvature energy 1 Iff( (=)l dx under the condition that fax) = |as| for all i L..... n. Unlike the minimum Barron norm interpolants, the natural cubic spline may not be convex (and in fact, it is not if f* is replaced by h" (x) = xx 0.5l).
5.2 Radially symmetric data
We explore the performance of numerical optimization algorithms in the setting of Corollary 3.3 with and without explicit regularization X E (0, 10-5} in dimensions d = 3, = 15 and = 31. The numerical solution is then compared to (a rescaled version of) the analytic minimum norm interpolant f described Proposition 3.2 which we compute by the algorithm described (Wojtowytsch 2022 Section 6]. The rescaling factor ra is chosen heuristically for an accurate match.
8

---

Momentum SGD. eporhs. Witth 12000. 10" data. Dim
Momentum SGD. 10 epochs.
12000. data. Dim
1.0
numerical solution
1.0
numerical solution
optimal interpolation in dim 3
optimal Interpolation in dim 15
0.8
0.6
0.4
0.4
0.2
Vemeettum 900, 1007 spocts, With 12000 10" dat CWn 25
Mementurr 500 300 speets Wi eth 12000. 127-3333 Dim 15
Nomentum spacts. Witth 13000 cate.
13
numorica snlatin
adiltion
08
optimal Intersolaaion
optinal cteppciaiiin in 4in
022
0.3
Figure 3: A neural network with a single hidden layer of width m = 12, 000 was trained by gradient descent with learning rate 17 10 and momentum μ 0.99 in the setting of Section The radial average is sketched by a solid red line. One radial standard deviation around the average, computed over 500 random directions, is shaded.z Top row: Experiment in dimension (left) and dimension 15 (right). The numerical solutions are compared to f (raz) with ra= = 1/1.05 and r1s = 1/2.55. In both cases, the 'minimum norm interpolant' shape attained to high accuracy. Both solutions are approximately symmetric, more so in low dimension. Bottom row: Numerical approximations to (ris.) for neural networks of constant width m = 12, 000, trained data sets of different size (but for an identical number of 200, 000 training steps with stochastic estimates computed over batch of 50 data points). The shape of the radial average is comparable across different dataset sizes, but the fit of the radial average with data is improved and the radial variance reduced for larger datasets. Note that the first two simulations are in the overparametrized regime, whereas the last experiment on the largest dataset is underparametrized.
Data is generated from distribution μ = lai + 142 14 where fai is a point mass of magnitude mi at the origin, μ22 is a uniform measure on the unit sphere 5d-1 with mass m2 and μı is the radially symmetric measure of mass 1 1- m m2 such that ||el| is distributed uniformly in [1, 7]. We numerically explored various values for m2 E [0.1, and m2 E [0.0, 0.4] and found simulations to be relatively stable under a number of choices.
Results are presented in Figures B and Appendix We find that all algorithms find a solution with radial average similar to fü (raz). albeit for rescaling factors ra which depend dimension d and (to a lesser extent) the optimizer. In high dimension, solutions are not perfectly radially symmetric, but the larger amount of variation over sphere of fixed radius is observed in the domain where fā R 0 rather than in the transition area (0, ra). Larger datasets improve the compliance with the optimal interface and reduce the radial standard deviation. Solutions do not remain non-negative and drop below zero before leveling off as the radius increases. The drop becomes more noticeable as the dimension increases and less pronounced for wider networks.
The results are essentially identical for normal Xavier initialization and (not radially symmetric) uniform Xavier initialization. In accordance with expectations, the radial standard deviation is higher for Adam compared to optimizers based in Euclidean geometry. While the neural network function found by Adam resembles a minimum norm interpolant, the weight decay regularizer takes significantly higher values compared to other optimization algorithms.
6 Conclusion
Shallow ReLU networks converge to minimum norm interpolants of given data: Provably if explicit regularization is included and empirically if it is not. We conclude with summary of our empirical insight into the implicit bias of neural network optimizers.

---

14
Figure 4: Top row: We compare different optimizers (SGD = red, Adam green and Momentum- SGD blue) in the setting of Section 5.2 in dimension with 10 data points. All minimizers attain the minimum norm interpolant shape (left column, rescaled optimal solution in pink) curiously for Adam, the correct shape is attained despite the fact that at Rs 1, 600 the weight decay regularizer is an order of magnitude higher than for the other optimizers (right column). The high regularizer value goes hand in hand with higher radial standard deviation (middle column). The initialization is uniform Xavier (in particular, not radially symmetric). Essentially identical results observed for the radially symmetric normal Xavier initialization with the same degree of radial symmetry in Figure Bottom row: If we include an explicit weight-decay regularizer with weight À 10 solutions resemble an optimal interpolant for a smaller rescaling factor F31, = 1/3.8 compared r31 1/3.5 without regularization. This is expected since the norm is weighted more heavily compared to data compliance. Notably, the radial standard deviation does not decrease, even for the (heavily affected) ADAM optimizer. It remains noticeable 0.1 for and Adam.
1. With reasonable (not too large) initialization, all algorithms studied here are biased towards minimum norm interpolant profiles.
2. At least in the case of Adam, this bias is visible on the function level, but not on the parameter level, as the weight decay regularizer increases rapidly large magnitude. Despite this, ADAM solutions often appear 'flatter' in high dimension with lower rescaling factor
3. Explicit regularization stabilizes towards a minimum norm interpolant shape, but at the cost of decreased fit with the target values. Its impact is most significant for poorly chosen initial conditions.
4. When the minimum norm interpolant is non-unique, different types of minimum norm interpolants are found depending the choice of initialization scheme and optimization algorithm. The impact of initialization scale appears more significant.
5. Optimization algorithms which are rooted in Euclidean geometry (such as SGD and momentum-SGD) more successfully preserve Fuclidean symmetries compared to the "coordinate-wise" Adam algorithm.
The last observation is not surprising for radially symmetric initialization laws as radially symmetric parameter distributions induce radially symmetric functions. It is, however, observed also for uniform initialization scheme which only obeys coordinate symmetries.
We believe minimum norm interpolation to be a useful testbed to study the implicit bias of optimizers and the impact of initialization and regularization. While minimum norm interpolation by deeper networks has not been characterized yet, we anticipate no obstructions to implementing similar program there in the future.
Acknowledgements
The research of Jiyoung Park is supported by NSF DMS-2210699
10

---

References
Stefan Adams. Lecture notes high- high-dimensional probability. https://warwick... ac. uk/
fac/sci/maths/ http//////maatsppppppll////////e choopeesstaf/steeeeeeeeeee attpeffannadm////aeaaaaaa www.aaaaaaaaaaiieeeeeeeee httuuu/shhhga---ciimmoossoorarla
ma3k0-notes pdf 2022.
Nicholas D. Alikakos, Peter W. Bates, and Xinfu Chen. Convergence of the Cahn-Hilliard equation
to the Hele-Shaw model. Arch. Rational Mech. Anal., 128(2):165-205, 1994. ISSN 0003-9527. doi: 10. N0.1007/BF00335025..
URL/http: h//p:/ttpt///..../ doi. org/ 1twg/1...oo 10. 1077/BP0337055
Annika Bach, Roberta Marziani, and Caterina Ida Zeppieri. 7-convergence and stochastic homogeni- sation of mingularly-perturber elliptic functionals. arXiv preprint arXiv:2/02.00882 2021.
Francis Bach. Breaking the curse of dimensionality with convex neural networks. The Journal of Machine Learning Research, 18(1):629-66) 681, 2017.
David GT Barrett and Benoit Dherin. Implicit gradient regularization.
arXiv preprint
arXiv:2009.11 2020.
Heinz Bauer. Minimalstellen von funktionen und extremalpunkte. Archiv der Mathematik, 94)) 389-393, 1958.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, (1663)):15849-15844 2019.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 1(4):1167-1180, 2020.
Kaushik Bhattacharya, Marta Lewicka, and Mathias Schäffner. Plates with incompatible prestrain. Archive for Rational Mechanics and Analysis, (21(1):443-18)) 2016.
2016.
Etienne Boursier and Nicolas Flammarion. Penalising the biases in norm regularisation enforces sparsity. arXiv preprint S3d::23000013333 2023.
Andrea Braides. r-convergence for beginners, volume 22 of Oxford Lecture Series in Mathemat- ics and its Applications. Oxford University Press, Oxford, 2002. ISBN 0-19-850784-4. doi: 10. 00.100994app011s0/77700999888810 0e009770119850784000000
URL http: doi 10t..o/../....../.p.r......
Andrea Braides and Lev Truskinovsky. Asymptotic expansions by 7-convergence. Continuum Mechanics and Thermodynumics, 20:21-62, 2008.
Lia Bronsard and Robert V. Kohn. On the slowness of phase boundary motion in space dimension.
Comm. Pure Appl. Math., 43(8):983-997, 1990. ISSN 0010-3640. doi: 10.10022/ca..3164430884.. URL htttp://da http:/xxxxddi.... org/ 10. 0ttpp///.....00///...3.000000000 316043088
Andrei Caragea, Philipp Petersen, and Felix Voigtlaender. Neural network approximation and estima- tion of classifiers with classification boundary in a barron class, arXiv preprint arXiv:20/1.09363,
2020.
Lenaic Chizat and Francis Bach. On the global convergence of gradient descent for over-parameterrized models using optimal transport. Advances in neural information processing systems, 31, 2018.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages 1305-1338. PMLR, 2020.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv:1804. 10200, 2018
Gianni Dal Maso. An introduction to weeeriioottoooooooeeeeeeeee T-convergence, volume 8. Springer Science & Business Media, 2012.

---

Alex Damian, Tengyu Ma, and Jason D Lee. Label noise sgd provably prefers flat global minimizers. Advances in Neural Information Processing Systems. 11-27499-27460 2021.
Ennio De Giorgi and Tullio Franzoni. Su un tipo di convergenza variazionale. Alli della Accademia Nazionale dei Lincei. Classe di Scienze Fisiche, Matemaliche Naturali. Rendiconti, 58(6):842- 850, 1975.
Thomas Debarre, Quentin Denoyelle, Michael Unser, and Julien Fageot. Sparsest piecewise- -ineer regression of one-dimensional data. Journal of Computational and Applied Mathematics, 406: 114044, 2022.
Manfred Dobrowolski. Angewandte Funktionalanalysi:: Funktionalanallsis, Sobolev-Rüume und Elliptische Differentillleechunnen. Springer- Verlag, 2010.
Patrick W Dondl, Matthias W Kurzke, and Stephan Wojtowytsch. The effect of forest dislocations on the evolution of a phase-field model for plastic slip. Archive for Rational Mechanics and Analysis, 232(():65-119, 2019.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. arXiv preprini arXiv:1810.02054, 2018.
Weinan E and Stephan Wojtowytsch. Representation formulas and pointwise properties for Barron functions. Calc. Var. Partial Differential Equations, 6166), 2020.
Weinan E and Stephan Wojtowytsch. Kolmogorov width decay and poor approximators in machine learning: Shallow neural networks, random feature models and neural tangent kernels. Research in the Mathematical Sciences, 8(1):1-28, 2021.
Weinan E and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. Mathematical and Scientific Machine Learning, pages 270-290. PMLR, 2022.
Weinan E, Chao Ma, and Lei Wu. A priori estimates of the population risk for two-layer neural networks. Communications in Mathematical Sciences, (415):1477-125,,,, 2019 doi: 10.4310/ cms.2019.w17.n5..at URL htLpstts::/ paattps://olt org. 40w.0o..31.Fcmommm 2019 v17 7.7...5.111
Weinan E, Chao Ma, and Lei Wu. comparative analysis of optimization and generalization properties of two-layer neural network and random feature models under gradient descent dynamics. Sci. China Math, 2019b.
Weinan E, Chao Ma, and Lei Wu. The barron space and the flow-induced function spaces for neural network models. arXiv: 08039 [cs.LG], 2019c.
Lawrence Craig Evans and Ronald Gariepy. Measure theory and fine properties of functions. CRC press, 2015.
Gero Fricsecke, Richard D James, and Stefan Müller. Rigorous derivation of nonlincar plate theory and geometric rigidity. Comptes Rendus Mathematique, 334(2):173-178 2002a.
Gero Friesecke, Richard D James, and Stefan Müller. theorem on geometric rigidity and the derivation of nonlinear plate theory from three-dimensional elasticity. Communications on Pure and Applied Mathematics, 55(1 1): 166-1155 2002b.
Gero Friesecke, Richard D James, Maria Giovanna Mora, and Stefan Müller. Derivation of nonlinear bending theory for shells from three-dimensional nonlinear elasticity by gamma-convergence Comptes Rendus Mathematique, 336(8):697-702, 2003.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
Karl Hajjar and Lenaic Chizat. Symmetries in the dynamics of wide two-layer neural networks. arXiv preprint arXiv:2211.0077 2022.
12

---

Boris Hanin. Ridgeless interpolation with shallow relu networks in 1d is nearest neighbor curvature extrapolation and provably generalizes on lipschitz functions. arXiv preprint (xXi:2200.119600 2021.
Boris Hanin and David Rolnick. How to start training: The effect of initialization and architecture. Advances in Neural Information Processing Systems, 31, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1): 42, 1997.
Jean Honorio and Tommi Jaakkola. Tight Bounds for the Expected Risk of Linear Classifiers and PAC-Bayes Finite-Sample Guarantees. In Samuel Kaski and Jukka Corander, editors, Proceedings of the Seventeenth International Conference Artificial Intelligence and Statistics, volume 33 of Proceedings of Machine Learning Rexearch, pages 384-392, Reykjavik, Iceland, 22-25 Apr 2014. mlr wwwpppressssse w33//ooooh1111.....111 1atmorio4.hhml1
PMLR. URL BtLhhtts::// httpttt:/ppooceeedgnnge
Tom Ilmanen. Convergence of the Allen-Cahn equation to Brakke's motion by mean curvature. Differential Geom, 38(2):417-611,, 1993. ISSN 0022-040X. URL http://pprjeecteucldd
org/ euclid. jdg/ 1214454300
Hui Jin and Guido Montúfar. Implicit bias of gradient descent for mean squared error regression with two-layer wide neural networks. arXiv preprint arXiv:2006.07:556 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 11Xiv:1412.6988. 2014.
Achim Klenke. Wahrscheiiniccheeisttteeei, volume 1. Springer, 2006.
Marta Lewicka, Maria Giovanna Mora, and Mohammad Reza Pakzad. Shell theories arising as low energy 7-limit of 3d nonlinear elasticity, Annali della Scuola Normale Superiore di Pisa-Classe di Scienze, 9(2):253-295, 2010.
Zhiyuan Li, Tianhao Wang, and Sanjeev Arora. What happens after sgd reaches zero loss? mathematical framework. arXiv preprint arXiv:2110.06914, 2021.
Zhong Li, Chao Ma, and Lei Wu. Complexity measures for neural networks with general activation functions using path-based norms. arXiv preprint arXiv:2009.0611 2020.
Marco Loog, Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences,
2020.
Chao Ma. Lei Wu, et al. Machine learning from continuous viewpoint, Science China Mathematics, 63(11)):233-2225 2020.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two- layer neural networks. Proceedings of the National Academy of Sciences, 115633))77665557777,, 2018.
Luciano Modica. The gradient theory of phase transitions and the minimal interface criterion, Arch Ration Mech Anal, 98(2):123-142, 1987.
Luciano Modica and Stefano Mortola. Un esempio di T-convergenza. Boll. Un. Mat. Ital. B (5), 14 (1):285-299, 1977.
Luca Mugnai and Matthias Röger. Convergence of perturbed Allen-Cahn equations to forced mean curvature flow. Indiana Univ. Math. J., 60(1):41-75, 2011. ISSN 0022-2518. doi: 10.1512/iumj. 2011.60.3949. URLhhtp::// http://xxxxxao doi org/ 11.200...1..iuu //m/////mjjm//mmm 60011.60.33
Sebastian Neumayer, Lénaïc Chizat, and Michael Unser. On the effect of initialization: The scaling path of 2-layer neural networks. arXiv preprint arXiv:2303.17805 2023.
13

---

Greg Ongie and Rebecca Willett. The role of linear layers in nonlinear interpolating networks. arXiv preprint arXiv:2202.00856, 2022.
Greg Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. arXiv preprint arXiv:1910.01633 2019.
Rahul Parhi and Robert D Nowak. Banach space representer theorems for neural networks and ridge splines. Mach. Learn. Res., 22(43):1-44) 2021.
Rahul Parhi and Robert Nowak. What kinds of functions do deep neural networks learn? insights from variational spline theory. SIAM Journal on Mathematics of Data Science, 4(2):464-488) 2022.
Scott Pesme, Loucas Pillaud-Vivee Vivien, and Nicolas Flammarion, Implicit bias of sgd for diagonal linear networks: a provable benefit of stochasticity. Advances in Neural Information Processing Systems, 34:29218 --22200, 2021.
Grant M Rotskoff and Eric Vanden- Eijnden. Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error. stat, 1050:22, 2018.
W. Rudin. Functional Analysis. International series in pure and applied mathematics. McGraw-Hill, 1991. ISBN 9780070542365..
Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural net- works. In Proceedings of the 35th International Conference on Machine Learning, ICMI. 2018, Stockholmsmüsssnn, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4430-4443 PMLR, 2018a. http:httt///poceeee
mlr preess/
#88/safran18a. html
Itay Safran and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks
In International conference on muchine learning, pages 4433-4441 PMLR, 2018b.
Etienne Sandier and Sylvia Serfaty. Gamma-convergence of gradient flows with applications to Ginzburg-Landat Communications on Pure and Applied Mathematics; A Journal Issued by the Courant Institute of Mathematicul Sciences, 57(12):1627-167 2004.
Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Conference on Learning Theory, pages 2667-2690. PMLR, 2019.
Sylvia Serfaty, Gamma--onvergenne of gradient flows on Hilbert and metric spaces and applications. Discrete Contin. Dyn. Syst, 147441444,,,,,,, 2011.
Shai Shalev-Shwartz and Shai Ben-David. Understanding machine learning: From theory to algorithms. Cambridge university press, 2014.
Jonathan W Siegel and Jinchao Xu. Approximation rates for neural networks with general activation functions. Neural Networks, 128:313-321, 2020.
Jonathan W Siegel and Jinchao Xu. Characterization of the variation spaces corresponding to shallow neural networks. arXiv preprint arXiv:2106.15002, 2021.
Jonathan Siegel and Jinchao Xu. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations of Computational Mathematics, pages 1-57, 2022.
Jonathan W Siegel and Jinchao Xu. Characterization of the variation spaces corresponding to shallow neural networks. Constractive Approximation, pages 1-24, 2023.
Justin Sirigmano and Konstantinos Spiliopoulos. Scaling limit of neural networks with the xavier initialization and convergence to a global minimum. arXiv preprint arXiv: 1907:040000 2019.
Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A law of large numbers. SIAM Journal Applied Mathematics, 80(2):225-552 2020a.
14

---

Justin Sirignano and Konstantinos Spiliopoulos. Mean field analysis of neural networks: A central limit theorem. Stochastic Processes and their Applications, 13063):1800-1852 2020b.
Samuel Smith, Benoit Dherin, David GT Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. arXiv preprint arXiv:2101.12176, 2021.
Luca Venturi, Afonso S Bandeira, and Joan Bruna. Spurious valleys in two-layer neural network optimization landscapes. arXiv preprint arXiv: 17Xiv:8802.0634,, 2018.
Stephan Wojtowytsch. On the convergence of gradient descent training for two-layer relu-networks in the mean field regime. arXiv preprint arXiv:2005. 13530. 2020.
Stephan Wojtowytsch. Optimal bump functions for shallow relu networks: Weight decay, depth separation and the curse of dimensionality. arXiv preprint arXiv:2209.0111 2022.
Lei Wu, Mingze Wang, and Weijie Su. The alignment property of sgd noise and how it helps select flat minima: stability analysis. Advances Neural Information Processing Systems, 35:4680-4693, 2022.
Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems, 34:18722-18733, 2021.
Pan Zhou, Jiashi Feng. Chao Ma, Caiming Xiong, Steven Chu Hong Hoi, et Towards theoretically understanding why sgd generalizes better than adam in deep learning. Advances in Neural Information Processing Systems, 33:21285-2111 21296, 2020.
15

---

Appendix
Numerical Experiments
16
B -convergencce
24
Homogeneous Barron spaces
26
D Rademacher complexity of homogeneous Barron space
29
Proofs of the convergence theorems
34
F
Theorem for finite data sets
40
Minimum norm interpolation in one dimension
40
Sub-Gaussian random variables
41
A Numerical Experiments
A.1 Hyperparameter settings and computation effort
In all experiments in Dimensions 3 and 15, the following hyperparameter settings were used unless otherwise indicated:
1. Normal Xavier initialization with gain a = v2
2. SGD: Learning rate 10-2 (Dimension 15), 10-3 (Dimension 3).
3. Momentum-SGD: Learning rate 10-3 and momentum μ 0.99
4. ADAM: Learning rate = 10-3 and PyTorch default hyperparameters for 31 = 0.9, B3 = 0.999, 10-8
For experiments in Dimension 31, we drop the learning rate for ADAM after 50 150 epochs by a factor of 10 and for Momentum-SSOn by factor of 10 after 100 epochs.
In Dimension a learning rate of 10-2 was found numerically unstable for SGD without momentum. To compensate for the smaller learning rate and provide fair comparison, the number of time steps was increased.
All experiments were performed on a free version of google colab or the experimenters' personal computers. One run of the model takes below fifteen minutes single graphics processing unit.
A.2 Summary and interpretation of additional simulations
In this Section, we present additional numerical experiments in various situations complementary to those presented in the main body of the text. These include: Wider neural networks (Appendix A.3), experiments with different optimizers (Appendix (A.4), experiments with different initialization to explore effects of scale and symmetry and the role of explicit regularization (Appendices A.8 and A.5), experiments with e'- -loss instead of fo-loss (Appendix A.7 and repeated experiments to visualize the stochastic variation between runs (Appendix A.6).
Additionally, we present and investigation into related settings where our theoretical understanding does not apply: In Appendix A0 we consider linearized (random feature) dynamics to explore how close we are to a (truly non-linear) neural network model. Appendix A..I we consider neural networks with single hidden layer and leaky ReLU activation instead of ReLU activation. In Appendix 12 we consider ReL.U networks with multiple hidden layers. For detailed list, see the table of contents below.
16

---

Wide neural networks in one dimension
17
A 4
SGD and ADAM: Dimensions 3 and 15
17
Radial symmetry Dimension
17
A.6
Gradient descent with Momentum
19
A.7
llos and Huber loss
19
A.8
Initialization scaling and explicit regularizaaainnn high-dimensions radial data
20
A.9
He initialization)
20
A. 10 Lincarized dynamics
21
Leaky ReLU activation
23
A.12 Deeper neural networks)
23
Our goal is not to explore questions of loss function, initialization, optimization algorithm and the impact of hyperparameters in a systematic fashion, but rather to establish problems in which minimum norm interpolant can be found in an explicit fashion as instructive benchmarks to numerically study such questions.
a proof of concept, we provide a partial exploration of the parameter space. For the moment, find ourselves confined to ReLU networks with a single hidden layer, as this is the only case in which explicit minimum norm interpolants are available. Minimum norm interpolation describes the shape of functions between known data clusters and is thus more expressive than a study of generalization error which is naturally confined to data clusters.
The additional experiments corroborate our findings in the main text. Before the detailed presentation, let us briefly summarize the conclusions.
1. Across variety of different optimizers, Xavier type (= Glorot type) initialization schemes and loss functions, minimum Barron norm interpolant-like shape was attained, to varying degrees of accuracy and with different rescaling factors.
2. Solutions are fairly radially symmetric with standard deviation in radial direction at most 0.1 (SGD) and 0.2 (Adam).
3. A geometrically distinct shape is observed for random feature models the same regime. 4. Explicit regularization little effect, even poorly chosen initializations Xavier type with high gain. We conjecture that the uniqueness of the radially symmetric minimum norm interpolant induces higher degree of rigidity and bias, compared to the one-dimensional case where the set of minimum norm solutions
diverse (and infinite).
5. Functions display larger variation in the radial direction for He initialization. In this regime, explicit regularization has more apparent and beneficial effects on both solution shape and radial symmetry. The solution does not reduce to the random feature model in this case either.
A.3 Wide neural networks in one dimension
In Figure present the same experiment as in Figure for wider neural networks with neurons in the hidden layer.
1, 0,00
A.4 SGD and ADAM: Dimensions 3 and 15
We repeat the experiment on Figure 3 for Stochastic Gradient Descent (SGD) optimizer without momentum and for the Adam optimizer of Kingmma and Ba [2014). The results are displayed in F respectively, The results strongly resemble those obtained for the SGD optimizer with momentum in Figure 3
A.5 Radial symmetry in Dimension 31
As indicated in Figure
we present computational results with the radially symmetric normal Xavier
initialization in Figure
17

---

MED EBC ECO
SC XCO boto 220
AOD TOS Mit RR0
Figure 5: For wider neural networks, we observe less stochastic variation between runs, as the empirical distribution of neurons is closer continuum limit. Solutions are generally more convex and symmetric than their narrow counterparts, The gradient descent optimizer without momentum stands out for its tendency to select solutions with highly localized second derivatives and a preference for piecewise linear functions with few linear regions, while other training algorithms select 'smoother' solutions with curvatures which are dispersed more evenly throughout the domain.
after 200 spocte, with 12035. cata FORth, Dimenecs deeay
S50 opodhs. 12009. data peints. D'mersion LeWsg
1.0
optinall Interpilation
0.
0.
06'
eptta testidapirs ZrewiniS Sipydon
Figure 6: We perform the same experiments as for Figure 3 but with the SGD without momentum optimizer. Learning rate was adjusted to 10-2 for dimension 15, but for dimension 3 we just used 10-3 learning rate and runned 10 times more epochs, due to stability of neural network training in dimension 3 case. The results are comparable, but the rescaling factors were chosen as 1/1.15 in dimension and 1/2.65 dimension 15.
18

---

Idom thter Depprtt Depth (d8t 12600 1000950 data printc Dimercon
Azan the 15 asoms Degth 2 abit 178m0 T200350 rats paint. Dimercnn 15
FurTo velAc1
04-
04-
Figure 7: We repeat the experiments of Figure but with Adam. The numerical solutions resemble those found by Momentum-SGI) but better rescaling factors for numerical solutions are 1/1.2 in dimension and 1/2.75 rather than 1/1.05 and 1 1/2.55, i.e. the functions are 'flatter'.
Figure 8: Results between normal and uniform Xavier initialization are essentially identical in this experiment compare Figure 4 (Approximate) radial symmetry is attained even when parameters initialized in a fashion which is not radially symmetric.
A.6 Gradient descent with Momentum
In Figure we present additional runs in the setting of Figure 3 Despite quantitative variation, the geometric shapes of solutions are stable over multiple runs and resemble the minimum norm interpolant fis in all cases.
A.7 llos and Huber loss
We repeat the experiment of Figure with the -loss function in the place of P-loss. To compensate for the lack of smoothness in the loss function, we reduce the learning rate by factor of 10 to 10 4 and increase the number of epochs by 50% to compensate. Results are reported in Figure 10
Blmeet bute Syltee DABAAO Deeeeder
Metoctiam Shypets Brateed
Figure 9: Three realizations of numerical interpolations in the setting of Section 5.2 computed by SGD with learning rate 77 10 and momentum μ 0.99. In all cases, the minimum norm interpolant shape is attained approximately, but the radial averages briefly dip below zero and exhibit local minimum which is not found in fz- The variation between runs is notable, but not large.
19

---

Mostertum STD aher 15 eports Degih 2. AMHi 17000, 10003000 dets poins Treeeon
Figure 10: -lososs leads to similar numerical results with smaller rescaling factor of 1/2.8 rather than 1/2.5 for fo-loss. Curiously, /(0) > 1 for these algorithms, while /(0) < when optimizing lo-soss. In Dimension d = 3, the non- moo-smoothsess leads to minimization problem that is not well resolved by the numerical optimization algorithm.
Similarly, we repeated experiments with the Huber loss function. Since the data set during the final stages of training, the loss function coincides with losss in the long run and all experiments are identical (2-loss. We therefore do not present additional plots.
1 over
In the initial stages of training, Huber loss is more stable numerically than 2lloss, especially for large gain or He initialization (compare Section[.99
A.8 Initialization scaling and explicit regularization: high-dimensional radial data
As noted in Section 5.1 and previously by Chizat et al. 2019]. the choice of initialization affects the optimization process of neural networks. Motivated by our observations in the one-dimensional case, consider the effects of initialization and explicit regularization in the radially symmetric setting (Section 5.2). Our results support the earlier claim that the effects of regularization are advantageous for poorly chosen initialization with high gain.
The experiments were performed in higher dimension 31 and with uniform Xavier initialization for the scenario in which it is most challenging to obtain radially symmetric solutions. As in Appendix AA.7 we used llos rather than loss. To compensate for the non-smoothness the loss function, we drop the learning rate by factor of 10 twice during the training process. Similar results were observed for f2 loloss, but the effects of initialization and regularization were less pronounced compared to the e! -case.
Plots for single representative run are displayed in Figure II The explicit regularizer has the clearest effect in the high gain regime, where explicit regularization helps to achieve better fit with the optimal transition curve and reduces the radial standard variation. Results were less sensitive to poor initialization than the corresponding experiments in one dimension. We conjecture that higher degree of rigidity is introduced in this setting by the fact that there exists a unique minimum norm interpolant.
A.9 He initialization
All experiments so far were performed with the initialization scaling of Glorot and Bengio 2010).
Especially for deeper neural networks, the initialization scheme of et al. (0155) is very popular. Hanin and Rolnick [2018) proves in particular that He et al. [015)'s normalization avoids the vanish- ing and exploding gradients phenomenon initialization in expectation. While this consideration does not apply to our shallow networks, find it informative compare the two schemes.
20

---

brpert with
We vary initialization scale (a E (0.25, 22, 10) from left to right) and consider training without weight decay (top row) and with weight decay X = 10 (bottom row). The rescaling factors are chosen to be 1/3.9 99 for = 0.25, 1/3.8 for a v2, and 1/4 for a 10.
Figure 11:
For shallow neural networks
m
fm : Rdd - R. (m > a ow x
the effect of initialization is as follows:
1. According to Glorot and Bengio [210)), the parameters Og and waj. 1 S i S m and S S dd are chosen as random variables with mean 0 and standard deviation 22/(m + 1) for a; and / 2/ + d) for Wayy- In particular, if m is much larger than d, then the [a;] || ||sl|| = (1/m). As we add m terms of magnitude ~ 1/m, we consider this the 'law of large numbers' scaling.
2. According to He et al. [2015), the parameters ag and Way. 1 < i < m and S S d are chosen as random variables with mean O and standard deviation /2/m for and 22/d for Way. In particular, if m is much larger than d, then the |a:] ||wl||| = 0(1//m). As we add m terms of mean zero and magnitude ~ 1/ym, we consider this the 'central limit theorem' scaling.
Many authors, such as Sirignano and Spiliopoulos [2020a] b 2019], present the factor 1/m or 1/ /m explicitly outside the neural network. As observed above, the effect of initialization can be significant. Unsurprisingly, results in the central limit regime, where all neurons contribute similarly at initialization, are more consistent and predictable. Experimental results are presented in Figure 12 in the one-dimensional setting and in Figures and the case of radial symmetry. Notably, in high dimension, explicit regularization not only reduced radial variation, but also increased data compliance by reducing the rescaling factor ra- In this setting, we observe the benefits of explicit regularization over relying on implicit bias only.
A.10 Linearized dynamics
Parameter optimization in neural networks depends heavily on the choice of initialization. While the dynamics are truly non-linear the regime studied by Chizat and Bach 20118) Rotskoff and Vanden-Eijnden [2018), Mei et al. [2018), Sirignano and Spiliopoulos [2020a] and WWojtowyscc 2020 there are scalings for which the directions w; remain close to their initialization for all time see e.g. [E et al.) 2019b] for a derivation. In this case, the solution produced by neural network similar to that of a random feature model. In this section, we numerically find the minimum norm interpolant of a random feature model by 'freezing" the inner layer coefficients at their random initialization. We find that the random feature solution differs geometrically from the Barron space solution, e.g. in that it is smooth at the origin, where the Barron space solution has cone-like singularity of the form ((x) - ca||=r||2 for small æ. particular, we find that our experiments were set appropriately in the non-linear training regime. See Figure 15
21

---

Redd odhen
uneen b003
puanes
Figure 12: Experiments for He initialization in one Dimension in the same setting as Figure 2 with Xavier initialization. Left: No explicit regularization, right: Weight decay regularization À = 0.002. Even for Glorot initialization with large gain, this regularizer was sufficient to induce convexity. For He initialization, it has a notable regularizing effect, but it is insufficient to impose convexity. We observe greater deviation from a linear function in the small intervals between known data points on either side of the big "gap'.
550 uPter 20 esschs with 12300 J0c0539 d9s points diicesionn 33, Weight dcoy 20
1.
08-
0--
spedia
30
aprinal rrsperios
cattr cardeii
134
4.5
2.4
Figure 13: Top row: Experiments for He initialization in dimension 31 in the same setting as Appendix A.8 Left: No explicit regularization with a rescaling factor ra = 1/4.9, Right: Weight decay regularizatinn with À 10 and rescaling factor гa = 1/4.2. We observe that under He initialization the effects of the explicit regularizer is even more pronounced. Unlike in other experiments, the presence of regularization increases the rescaling factor and thus improves the fit to training data (for the radial profile).
Bottom row: We repeat the same as above with MSE loss rather than él lloss, using the Adam optimizer with learning rate 10-5 instead of SGD with momentum, and using overparametrized rather than underparametrized neural network. Without explicit regularization (left), the radial standard deviation is substantial, while explicit regularization leads to more radially symmetric function, albeit at the price of a higher rescaling factor. For easy comparison to Figure we present the optimal profile as rescaled in the main document well. Both functions achieve training loss < 10 3 but clearly generalization is poor without regularization: While the radial average is close to the target function f O for inputs with ||>|| 2 1, the radial variation is high.
22

---

5u0 Urspedts with Lommocee the
Figure 14: We examine the effects of the explicit regularizer (weight decay penalty X = 0 for the top row and À = 10-1 for the bottom row) while varying dimensions (d = 3, 15, 31 from left to right) in the He initialization scheme. Optimizer settings were identical to the top row in Figure 13 The rescaling factors were r3 1/1.15, r11 = 1/2.1, and r31 1/4.2. As dimension grows implicit bias may be insufficient find a minimum norm interpolant shape with reasonable scaling factor and may not enforce radial symmetry. In this case explicit regularizer may have an advantage.
A random feature model trained on the same dataset as the neural networks. The solutions produced this way are geometrically distinct from neural network solutions as they 'flat' at the origin. The left two figures correspond to different initializations: Gain v2 (left) and gain (right). Notably, the variation is higher in radial direction in high dimension and higher than for the comparable neural network model. Perhaps surprisingly, higher gain appears to induce better implicit bias in this case. No explicit regularization was used. Here we initialized the random feature by the same law as the neural network rather than initializing the outer layer at zero, since our goal is to study neural network dynamics, not find the optimal random feature solution. For the right plot, the initialization random normal with gain 5 in the inner layer and zero in the outer layer with unsurprisingly better results.
Figure 15:
5
A.11 Leaky ReLU activation
As noted by Wojtowytsch [022]. minimum norm interpolation is not stable when passing to an equivalent norm. A Barron space theory can be developed in perfect analogy for networks with the leaky ReLU activation function, and it is easy to see that the 'Barron' spaces for both activation functions coincide with equivalent norms, depending on the negative slope of the leaky ReLU function. However, the minimum norm interpolant fä with respect to the ReLU-based Barron-norm is not guaranteed to coincide with the minimum interpolant for the Sooky-ReLU-basee Barron norm. Experimentally, however, observe strong agreement between the geometry of numerical solutions here.
A.12 Deeper neural networks
We train neural networks of depth L > 2 to fit the same radially symmetric data as in Section 5.2
We see in Figure 17 that for depth L 2 3, weight decay-regularized networks strongly resemble the interpolant fip (x) = max(1 1 - || ||,, 0} with minimal (Euclidean) Lipschitz constant. This function
can be written as a composition of two Barron functions fbump fnorm.
foump (2) max(1 [1 2, 0) o(1 2),
foom (x) ||ズ||| ca Eu-xo [ove x)]
23

---

dhe Drpatts Busker
Figure Neural networks with single hidden layer and leaky ReL.U activation (() max {2, 0} 0.1 min 0} trained in the setting of Section S5.2 Without theoretical foundation, we observe that the shape of fa is attained to high accuracy also in this setting with the same rescaling factors as in the ReLU setting. Left: Momentum-SGD, Middle: Adam, Right: Momentum-SGD for wider
network with m = 25,000 000.
Figure 17: Neural networks of width 250 and varying depth were trained to data generated as in Section 5.2 with weight decay regularizers 10 10-2, 1003 10- and O (left to right). The initialization gain variable was chosen as a = 2 as required to avoid exploding and vanishing gradients in deeper networks. Evidently, small amount of weight decay regularization provides useful geometric prior without diminishing the quality of data fit.
Unlike networks with one hidden layer, deeper networks have positive (or nearly positive) outputs everywhere. While two-layer networks follow the minimum norm interpolation shape closely by the origin and have radial variances which increase outside the unit ball, deeper networks have positive radial variance inside the unit ball, but are essentially radially symmetric outside compare e.g. Figure
where TO denotes the uniform distribution on the unit sphere and ca vd is dimension constant. We can thus approximate St.p efficiently by neural networks of depth as long as the first layer is sufficiently wide.
Unlike their shallow counterparts, neural networks with multiple hidden layers have no strong geometric prior without weight decay regularization. With weight decay, the observed behavior was relatively stable over range of dimensions, initializations scalings and optimization algorithms. We are led to conjecture that Läp is minimum norm interpolant in this setting. The statement remains imprecise at this point as no function space theory for deeper networks with weight decay regularizer has been developed to the same extent as Barron space theory.
B T-convergence
In this appendix, we recall the definition and a few properties of --convergence, a popular notion of the convergence of functionals introduced byDe De Giorgi and Franzoni [1755] in the calculus of
variations to study the convergence of minimization problems. [Bridees] [0022] Dal Maso 2012) provide introductions the theory and its applications. As the notion likely not familiar to readers from the machine learning community, we provide some full proof' as well.
Definition B.1. Let (X, d) be a metric space and Fay F X - TR (-x, oo} be functions. We say that Fn converges to F in the sense of T -convergence if two conditions met:
1. (lim inf inf-inquality) x,, is sequence in X and xn + then lim inf Fa(xn) 2 F(x).
2. (lim sup-ssppeeeeeee inequality) For every x E X, there exists a sequence 2; E X such that x - and lim sup,,toc Fx(x)) < Fzx)
Intuitively, the first condition means that F(x) is (almost) lower bound for (a(xn) if n is 'large' and Fn is 'close" to x, while the second condition means that there is no larger lower bound that we could choose. The sequence x; is often referred to 'recovery sequence'. Of course,
24

---

combining the liminf- and I I-convergence when dealing with minimization problems where uniform convergence fails, but hope for convergence of minimizers to minimizers.
we find that in fact (2,,,) + F(z). We employ
Often, T-convergence is considered continuous parameter e approaches 0+ rather than as the discrete parameter nu approaches infinity. The definitions remain largely identical (with obvious substitutions).
To get a feeling for I-convergence, we consider particularly simple situation by looking at two constant sequences of functions. Note that the sequence is constant, not the functions.
Example B.2. Let X = and consider the constant sequences
(x(z) f(x) {] x 0 x= 0'
0 x x
Gx(z) g(x)
x=
We claim that
(T- lim (x) [())) f(x); (r- lim Gr)(z)= Vx TR. n-000
n-
If In + x. and x + O, then Fn( = 1 and Gn Gn( for all but finitely many n E N, meaning that (x()))) + = fxx and G (Tr) O. remains to consider the case Vn ->
We see immediately that Fn (z.) 2 0 = f(0) for all n E N. Conversely, if we take x., = 0 for all n, then x + 0 and (x. (0) + (0)). In total, we conclude that - lim F,, f.
For Gn, we find that Gn (Ix) 2 0 for all n E N. Additionally, we can choose the sequence In = 1/n such that (I,) for all N. Altogether, find that r lim GT O.
More generally, if F, = F for all n E N and some F: X then r lim Fn = F is the lower continonn envelope of F. In particular, I lim F F if and only if F is lower semi-continuous. The main useful properties of I-convergence are summarized in the following lemma.
Lemma B.3. Assume that Fn + F in the sense of conovergeeee,, En + 0+ and In E X is sequence such that
Fa(In) S inf Fu(x)) + sEX Assume that In. xx Then F F(x*) inf F(x). In particular, if xn, is minimizer of Fn, and the sequence Vn converges, then the limit point is minimizer of F.
Clearly, this is most useful if we can guarantee that the sequence z, converges. For many useful se- quences of functionals, the existence of convergent subsequence can be established by compactness. This is easily sufficient, as we can also pass to subsequence in F,.
Proof. Due the liminf-inequality, we have F(x")
lim (x,) lim inf (z)... n-000
2-+00 zEX
On the other hand, let x E X be any point. Then, due to the limsup-inequality, there exists some sequence sh such that
A/n x.
F(x)
lim (a( 2 lim inf inf (z)... =2220
n-000 xEX
In particular inf xxx F(x) > lim inf, inf F,(x). Combining the two estimates, find that F(x*) < infaex F(I), which means that x² is a minimizer of F.
For completeness, few observations in order.
1. The notion of [cconvergeece relies on the notion of convergence on the underlying space X, and P-limits can change when keeping the set X fixed, but passing different topology (c.g. a weak topology in infinite dimentioaiai spaces).
2. T-convergence is made for minimization problems, and it does not behave well under multiplication by negative real numbers: In general r lim(-F,): + - -lim F.). even if both limits exist. The reason is the asymmetry between the lim inf and the lim sup- condition. To see this, consider for instance F,, and G, 1 in Example B.2
25

---

3. If + F and Gn + G, it is not necessarily true that Fp + Gn - F G. While it remains true that lim in (F, + (2,) 2 (F + ()) 2n - x, it may no longer be possible to find a recovery sequence Ia for F, + Ga- For example, if Fn = lo and
Gn = lmo for all E N. then Fn I 0 and Gn I O. but Fn + G. = 1 for all and Fn + Gr, 5 1.
However, if F. E F and G2 converges continuous limit G uniformly, then (Fn G.) - F. In particular, uniform convergence implies r-convergence. Namely, if Gn + G uniformly, G continuous and In + I, then for given e > O, we can choose E E N so large that
(a) [G(zn) G(x) < </2 for all n 2 N since G' is continuous at and (b) [Gn(Zn) - G(xn) < < for all 2 N due to uniform convergence.
Then
(Tn) G(x) S (G,(s.) G( (((x)) G(x) <e.
In particular G, (ma) + G(z). Recall that G' is guaranteed to be continuous if G, is continuous for all N.
4. T-convergence unrelated to pointwise convergence of functions:
Neither does it imply pointwise convergence, nor is it implied by it. Namely, the sequence G. in Example B.2 B.2 has the function g as a pointwise limit and the constant function 0 as a I-limit.
5. I-convergence not a notion of convergence derived from a topology. Indeed, even if Fr is constant sequence, i.c. if F2 G for all it may happen that I lim Fn & G (see
Gn in Example The T-limit is related to G, though: It is the lower semi-continuous envelope of the function G. In fact, every P-limit is lower semi-continuous.
Despite its somewhat counterintuitive properties, I Ieconvergeecce has proved invaluable in many areas of the calculus of variations. It has been applied to homogenization by Bach el al. (2021), dimension reduction for thin sheets and shells by |riesecke et al. [2002a 2003 22002)] Bhattacharya al. [2016] Lewicka et al. [2010] and the study of phase boundaries by Modica and Mortola [977].
Modica [987]. While the I'-oovvrgeence of functionals does not imply the convergence of their
gradient flows even in situations of practical significance (see e.g. the example of Dondl ct al. (2019).
Serfaty [201 (Sandier and Serfaty (2004), |Mugnai and Röger [201T]. [[maanen] 1993], Aliikakos
et al. 1994] provide important examples of situations where this can be established in suitable
sense. Even more, Bronsard and Kohn (1990) use I-convergence to gain insight into PDE dynamics.
C Homogeneous Barron spaces
In this section, we introduce the abstract framework which used to prove our main theoretical result, Corollary 3.3 A neural network with neurons in single hidden layer can be represented as
(I) = bo + Σ oww C x bb)
or fm by Σ a ow + b,).
(4)
The network weights and biases are (a, W. b) E R" x x x R The normalization depends on personal preference, with the former being common in practice and the latter more common in theoretic analyses. We define the weight decay regularizer by
||al 22
W
1 2
RwD(a, W.b)
2
or RWDDRD W, -. (af ||w: respectively. Here || 1l2 denotes the Euclidean (--oorm of a vector and ||]] denotes the Frobenius norm of the matrix W whose rows are the vectors wf Note that we do not control the magnitude of the biases b, in the regularizer. This is a common approach, as the bias does not influence the Lipschizzcoontannt of the function represented by the neural network, which is useful in studying the generalization of the neural network.
26

---

We study function classes corresponding to arbitrarily wide neural networks with a single hidden layer, where the norm corresponds to the weight decay regularizer. We dub these function spaces 'homogeneous Barron spaces' in analogy to the more classical Barron spaces studied by / et al. [201c]]. Ma et al. 2020), and Wojtowytschi 2020]. which correspond to a weight decay regularizer which also controls the bias. For homogeneous Barron spaces, coordinate transformations by Euclidean motions induce an isometry of the function class, while the origin plays a special role in classical Barron spaces. This justifies our terminology. as the data space is treated as isotropic and homogencous by this function class. Homogeneous Barron spaces have also been studied by Ongic et al. [019], Parhi and Nowak [202]] 2022] under the name Radon-Bi spaces. A closely related class of spaces has been considered as the "variation spaces of the ReLU dictionary' by Siegel and
Xu
[2020 2022 2023].
Heuristically, (homogeneous) Barron spaces are a function class tailored to replacing the finite superposition of ReLU ridges in 4 by an arbitrary superposition while keeping the weight decay regularizer finite. Due to the lack of control over the bias term, we will see that slightly awkward technical definition is needed. Let x be probability distribution on the parameter space IR IRd x R. We would like to define
- ao(w" fo(wxx= x dT((a,u.)))
f : Rd + R, Sx. = bo + Bla,w,b)~ (ao x (ao("=+++))) = JERRERR
and
RwD( 1
la|2 1|ll| (a..b..))
JR
fx. is an analogue of neural networks with single hidden layer, but of arbitrary and possibly uncountably infinite width. Every finite neural network can be expressed in this fashion for the empirical measure Tm m
E ((a,,m)be)) in which case Rw (TT)) Rw D W.b).
Unfortunately, even if RwD(n < oc, is not clear that the integral defining f= exists in meaningful sense. We do however note that formally
[x)) - fx)( = E [a x + b) oww"y y + b) - [[aa| / x + - Iw y + bl]]
| - yl E [[a [o]|] < ||-
I|a + || = || yll Rw (w))
2
Thus, the integral defining fx (I)) exists for all x if and only if it exists for, say, x 0. We exploit this in the following modified definition. Let æ be probability distribution on the parameter space R x Rd R and y E R. We denote
(N,(()) = y + Bl [a(a(w= x + b) -bb))
[b))]
By the same argument as before, we observe that
1. (0) y and
2. [z,(()) - fr,y(a') S RwD(T) || ='1l-
The class of functions of the form fry forms the homogeneous Barron space. Still, every finite neural network fm can be represented in this fashion with bn y D (b.).
an (bo(b,) or bo y
Definition C.1 (Homogeneous Barron space). Let f Rdd + R be a function. semi-norms
We define the
[fs= = inf BWD(T), f=faN
[f]o = [f(0)).
The homogeneous Barron space is the function class B(R*) = {f Rd + R: [fls < oo}. By the definition of function [f) < oo is automatically true.
We note few important properties. First, we consider two function classes:
Fg = conv {a (a(w x + (b)) a² + || < 2Q}
Fq (R) conv (a(w- + b) - (b)) a² + || < 2Q, s RQ
27

---

Note that RR( S FQ eff B: =0 S Q}. The closure is taken with respect to locally uniform convergence, i.c. pointwise convergence which is uniform on all compact sets Due the homogeneity ReLU activation, we may prove the following.
Lemma C.2. The identity Fq (s e f(0) 0, [A]s s Q) holds.
While the claim is natural, its proof is surprisingly technical and postponed until the end of the section. The class Fo(R) R) will be used below for technical purposes. Of major importance below are the compact embedding theorem and the direct approximation theorem.
Theorem C.3 (Compact embedding). Let fn. E B be sequence such that lim inf (fnl - [fo]8 +oc. Then there exists f in B such that
1. Jn + in (K))
( for all compact sets K C C
2. fn + +f in (P) for all measures j finite p-th moments, p E [1, oc).
3. [f]s in
Proof. It is sufficient to show that the set Fqq fa(o(w- x x - a² + |w|2 - 2Q} is compact in CO and (n) for all K and μ as above, in which case also its closed convex hull is compact [Rudin] 1991 Theorem 3.20.]. To this end, observe that the map
F R
x IR- (K),
(a, w. b) - aaww + (b))
is continuous for any compact set K. Since K is compact, we have K C Bn(0) for some R > 0 and thus |w x S V2QR for x E K. In particular,
o(w + b) o(b) (w-a) if V2QR
if 22QR
(2QR)) is the continuous image of compact Hence Fi F(f(a, w, a² /w|2 2Q,
set, hence compact. We have thus proved a compact embedding into (*(K for any compact set K.
Exhausting I by the sequence of compact sets (0), m E N and using diagonal sequence argument, we see that under the conditions of Theorem C33 there exists f B such that fn + f pointwise everywhere on IR and uniformly
compact subsets. Additionally, we observe that ʃ(0) = 0 and there exists uniform upper bound on the Lipschitz constants the sequence Sn We conclude that fn + f in (P()) from the Dominated Convergence Theorem using Q |>]| dominating function.
As consequence, we find the following. Corollary C.4.
a Banach space.
2. C (TR.) B(R)).
Proof. The first claim follows as in (Siegel and Xu, 2021 Lemma 1]. where proved in a more general context for dictionaries which are compact in a Hilbert space in our case, the dictionary
x + a ( b) (b)). The second claim follows from (Ongie al. 2019 Corollary 1]. Li
conclude with a theorem which establishes a rate of approximation for Barron functions weaker topology.
Theorem C.5 (Direct approximation). Let f E B and measure on Rd with finite second moments. Rdd such that
Then for any m there exist c R and (a,, w,,b,)
2 mm
/
10 12 < [f]e.
f f-c-
Σ a.o f x b.)
I xf2 du
|||||=
(P()))
This notion of convergence is generated by topology, but not metric. Other notions of convergence can be considered and induce the same function class.
28

---

Proof. A proof of this result can be found in (Wojtowyssch 2022 Proposition 2.6.
Appendix CJ in the proof of
Proof of Lemma C.2
Step 1. Assume that f e B such that f(0) = 0 and [f] S Q. By the Direct
Approximation Theorem (which is proved in [Wojtowytsch] 2022 Appendix CJ without using Lemma C.2b, we find that for every E N and every measure on I with finite second moments, there
exists
(m(
m Σ
anfows x b2) o(b))) Fq
such that fl - flles S 1mllfllmm 1/2 In particular, f is in the closed convex hull of {a (o(wFx x b) -(b)) a² |w]2 < 2Q} if the closure is taken with respect to the L (n) topology. Additionally, the sequence fm a uniformly bounded Lipschitz constant and is therefore compact in (K) for all compact K by a corollary to the Arzela-Ascoll theorem [Dobrowolski 2010 Satz 2.42]. In particular, fm + uniformly and thus f FQ.
Step 2. Denote (z) fo(w">x b) [b)}. Since ((a, afa(b) - -bb)) 0 for all a, w. b, conclude that f(0) 0 for all Fq. If FQ, then there exists sequence
No.
fa(z) = Σ Aindin fo (win x + bi,) (bs,))
such that fn - f locally uniformly. If the biases remain uniformly bounded, the sequence of empirical distributions
N Ann Mam.h.
N 1=1
has a convergent subsequence by Prokhorov's Theorem [Klenke, 2006 Satz 13.29]. We denote the limiting distribution as π. The convergence of Radon measures implies the convergence of fn to -f x + b) (b))] by definition. Since fn converges locally uniformly
Sx = E(a, ~i (a (aw by assumption, f = f+ E B and [f] S Q.
If the biases do remain bounded, we note that for every compact K RR we can extract convergent subsequence of the measures by the same argument used to prove Theorem C.3 effectively making the sequence of biases bounded. We can extend the argument the entire space exploiting that
lim (o(w x + o(6)) (b/lbb) b-+00
locally uniformly.
D Rademacher complexity of homogeneous Barron space
in similar context, estimate the Rademacher complexity of homogeneous Barron space and use it to bound the generalization gap (i.e. the discrepancy between empirical risk and population risk). In setting, face additional technical obstacles:
Following classical strategy implemented e.g. by al. [2019]]
We deal with general sub-Gaussian data distributions y rather than data distributions with compact support.
2. We do not control the magnitude of the bias variables.
3. We consider Plloss, which is neither globally Lipschitz-comtinuuus nor bounded.
In combination, these complications require a refined technical analysis similar to Appendix Let summarize several notations which will be needed below.
Rad the empirical Rademacher complexity of a function class over a given dataset.
Rad, the expected Rademacher complexity of a function class over data set composed of iid samples from the data distribution
29

---

R - the population risk R(f) = -If -1ff- -f ff= We generally take this to operate on the level of functions, parametrized or not. By abuse of notation, we identify R(a, W, b) = R(f(a,
Rr - the empirical risk R., (f) = |lf ff = = - (x;)) over data set (.1,..., where Ht, Σi= dr, is the empirical measure. Equally, R., can be considered for functions parameters with the natural identification.
Rnm.d. The regularized empirical risk
Rn, (a, W) Rr (a, W. 2 (Iaal22 + [[]]9).
We only consider this quantity the parameter level, where it is computable. While the weight decay regularizer an upper bound for (fa,,,,)).. the two are generally not the same since the coaameter-to-funciie map of neural network is generally not injective.
Rw D the weight decay regularizer.
Fq the set of functions for which [f]s - Q and f(0) =
FFAQQ the set of functions for which [f] s Q and [f(0) - A.
As is common in the mathematics community, C will generally denote a constant which does not depend quantities (unless specified otherwise) and which may change value from line to line. Some facts about sub-Gaussian distributions, which we believe to be well-known to the experts, are collected in Appendix
Definition D.1 (Rademacher Complexity). Let S (x1,..., In) be set of points in R& (a data sample) and F a real-valued function class. We define the empirical Rademacher complexity of F on the data sample as
Rad (F;() Eg
sup Σ (x)) [ i=1
where E; are iid random variables which take the values 11 with equal probability 1. 2. Rademacher complexity is defined
The population
Rad, Es [R)d(F:)).
i.e. the expected empirical Rademacher complexity over set of n. iid data points.
In this section, will find upper bound of Rademacher Complexity of B. We will denote by S, the set of n samples, and Rad(F, S.) the sample Rademacher Complexity of J given the samples S.. We furthermore denote R = max [|ai]],-.. (,,,,, and consider the function classes FQ and
FR(R) as Appendix
FQ conv (o(w- + (b)) a² | < 2Q}
FQ(R) conv (o(w- a (b)) a² + || < 2Q, S RQ.
Lemma D.2. Let S,, = fai....,, and be data set in Rdd, Then
(1+3v2)Q
Rad(Fo, <
Vn
max 1si5n
||==||
Assume p is a Gaussian distribution in IRd. Then
Exzop
[x]|]
log n
Rad(Fq) < (1+3v2)
og
for all n 2 2.
30

---

Proof. Initially. we fix set = f FnJ of n points. We will later take the expectation over S, using the sub- Gaussian property of μ for an explicit norm bound. Define R == maxi ||2;||- To this end, we first prove the following claim, which enables us to focus on only single neuron functions instead of entire Fq:
Claim: Let , E IR. Then
sup Σ <ff(z))= FQ
Eeaafowwa + - +2+||=1?
sup
Proof of Claim. Note that Fo is the closed convex hull of single neuron ridge functions, i.e. single neuron ridge functions the extreme points of the closed convex set Fq-
To verify the claim, first note that + Σi=1 f is continuous linear functional on (*) for any compact K S Fd containing the finite set S. It is well known that Co (K) is a Banach Space. Therefore, if {a (a(oww x + b) o (b)) Theorem 3.20.] implies that Fo. a closed convex hull of the compact set, is also compact. Then, foom the compactness of Fo. we can use Bauer [958) 's maximum principle and see that the supremum attained at extreme point. Compactness follows from the compact embedding Theorem, see Theorem C.3 above.
(K)
| a² + |wm v 2Q) is compact C [") then [Rudin 1991
Over the next steps, we will bound Rad, Step 1. In this step, prove that
Rad Sw) = (Fq(RF(RR) Sn).
To show this, we first observe that if [6] 2 || R, then ow x- b) - -o(b) = (sgn(bb)) since
|wT':; S ||e]|l ||<]]| S [b] R for all S i S n. This means for b/bl 2 ||m||| R, the precise value of does not change the value of a(w b) (b).
Now, we compute the Rad
Rad,Foo Sm)
E,
sup Σ ( Fi i
E,
sup a² 1 22Q
Σ Euafolw- [eafolw-a; +;+ o(b))
E.
sup
Σeafo(w x, 1 o(6)]
a [ |
n Rad (Fa (1)),
For the first line, we used the claim.
Step 2. Using the uniform bound on the magnitude the bias from the previous step, in this step we bound the Rademacher complexity by
a Σ E ow x, + b)
QR Vn
Rad(Fq: S) Rad (ad((F S E.
sup w|sQ. bison
n
We verify this from the definition of Rademacher Complexity of (R). Note that ad (we + b) (Aa) o ((w (o)t///)+/// A). In particular, we may assume without loss of generality that |a|2 || < Q for optimal balance which makes a² + || minimal without changing the neuron output.
n Rad (A) = E. sup fxxt)
Fo
31

---

E
sup a² || SQ |
ca(lo(w-20 24 + o(6))
c.00(w: Is + +b) + Σ [ccor(1))
E.
sup |||=||=||s MEvQn
In this step, we only consider the first term.:
Eg
sup IISQRR
Σ ((a
S Ee
sup IuISvRIWiSVQR
Σ c(a((b)
||
sup
(blo(b) E.
Σ 6s S QR vn.
/u SVvQR
The first line is again by applying the claim (R). In the last line, we used two facts:
1. a is ReLU, so [b]ar(b= S |abl I VQQR = QR and
2. the observation that
2
F E,
a
E
E2
4
Be Belere =
Σ
EE [e?]
nn
since Elee] = 0, en and E are independent if + and o7 1.
Step 3. In this step, we prove that
1
3V2QR Vn
Eg
sup VERR
s,a (w xz b)
n
To this end, we modify the data points as 5. = (I,, R) and the parameters as w = (w h). Then, observe that tb = W. || 3Q. Therefore, we can write the above by the following:
21 + R < v2R, and a² +| = a² +| R z
Ee
sup
Σ e,ao
I4
b)
-
1
B
sup
9,a (w 2i)
a |an
SQ.
IMSQ
n1
||
S3Q
Eg
sup
|a| |
EGO
160
71
i=1
1 Eg
a. + ||
sup
71
3Q Er
sup
(u F44
2n
||a||a51
3Q Ee
sup
eio (u. xi)
n
||u||a51
3Q Vn
max 1@a.122 S 3/2QR
= 3Q Rad(o o H2, Sw) $ 3Q Rad Rad S.)) <
Vn
Here, H2 = {u E I ||u||2 < }} and S. ( J. When removing the absolute value, we used that the sum always non-negative and that is symmetric when replacing & by -c. When removing a. we make use of the Contraction Lemma for Rademacher complexity (Shalev-Shwart/ and Ben-David 2014 Lemma 26.9]. Finally, for Rad(H2, S,) we used the expression for the Rademacher complexity of the class of linear functions on Hilbert space. [Shalev-Shwane and Ben-David 2014 Lemma 26.10] This concludes Step 3.
32

---

Step 4. In this step, we finally consider sets S which are sampled from the product measure μr", i.c. sets where Ti5...,Ta are independent data samples with law μ. From steps - 3, we know that
(1 + 3v2)
Rad(Fa (F))== Es, mm
Rad(F,SS))
E
max
|
Vn
We bound E (a maxxisisn |1:l|]] by Lemma H.1 obtain
E
[=]]]
log n
Rad(FQ) & (1 + 3v2)
ov
n
n
A similar result follows immediately for the more general function class FaQ:= :- (s B: [s S Q,1((0) S A}.
(5)
Corollary D.3. Under the same conditions Lemma D.2
we have
Ba
E
[|]l]]
log n
A
Rad(FA..) =
+ ov
2
+
Vn
Proof. We note that f E FA.Q if and only if = f + a with f E Fq and fixed dataset S, we have
|a| s A. Hence, for any
n (
= Eg
sup [EF i=1
Σ Exf(
= Eg
sup sEFo
Σ Eila - f(xi))
im1
$ Eg
sup LEF t=1
Σ (xx)
+ +
sup |a|sA
S (Fo))(Fo) +
vn
by the argument Step 2 in the proof of Lemma D.2
bound on the Rademacher complexity, together with the sub-Gaussian property of the distribution 1r, allows to control the ggeeralization gap" in homogeneous Barron spaces.
Corollary D.4. Assume that μ is - Suub-Gaussinn distribution on Id. Let Xn) be iid random variables with law p and f a jumeasurable function such that
lf* f (0) B1 B2llzrl1
J--llmost everywhere. Let
Rn.(f) = " - n
-
R(f) = Exwp (())))))) (()))
Then with probability at least 1 28 over the random draw of X1,..... .i,. the bound
sup (R()) - R()())) s C" ((Q + B2) (Enoy x + 0² + +1) A+ + Bi)² log(n/0)
(0)EFaa
nn.
holds for constant C* 0 which does not depend 6, Q, d, or
Proof. Step 1. From Lemma
H.2
with probability at least 1 - 8 we have
| S Ez [|]=[]]] + a v2 log(n/5).
max Isi5n
We denote R, : Ezap [|||||] + a /2 log(n/5) for simplicity.
Step 2. Consider the modified loss function
4(f) min (f,
33

---

which is bounded by 5² and satisfies 0zke S 22, le is 2 Lipschitz continuous. We thus observe that, with probability least 1 over the choice of random set S (II..... . we have
E Bez [<(((()) - Σ ((f(z))-f(() - 4 (aq((Fq S,)] + + (2/(2/0)
1=1
by Shalev-Shwartz and Ben- Bev-David, 2014 Theorem 26.5] and the Contraction Lemma for Rademacher complexities. (Shalev-Shwartz and Ben- 2014 Lemma 26.9]. In particular
(((()))) - (x:))) + V / 2 log(2/5)
E (
n
Ex~al1=1 ||c|| + a
2 log
AS Vn
4(1 4(1+322)2
v
n
n
Step By the union bound, with probability at least 26, both the norm bound of Step and the generalization bound of Step 2 hold. In the following, we assume that both bounds hold. Note that
[f(z) fx)) $ [xx f* + (f) f (0) S B1) + (Q B2)
In particular, if g > (A + B₁) + (Q + R then [f(x) (x)) < & on Bn(0), so
(<((())= - (x) (f(x)) - (x)). Applying the generalization bound with R. = R | +
ov2 (0)) and En (Q+ Bn + (A + B1), we find that Le. (f - (x;))
((())) 1' (x:)) for all by assumption and thus, with probability at least 26, we have
((Q 1 Σ E(()) - (a.))) ((:)))))a)))) Exzu [e(x)) s
log(2/0)
-
Br + A A:
2 log n
+ 4(1 + (V2)Q((2Q + B2) + A Bi)
+ a V
Vn.
n
Step 4. Finally, we bound the population risk with the true loss function rather than lg. Thus we find that for Bn = B0)( we have
Ex (f(x))
EENN -f (x)) 1Bn + Ezw [(((=) f fxx)) (f((=)
1 120
S Eeery ( Exrn [(A B1 (Q + =x) Ixd
for 2 3. From Lemma H.3 with R. = E||=|| + a /2 log(n/0)) we have
Exzy [||/||2 Ba, (0) (x)] S 2T exp (- log(n/6)
) ((El)=:))² - -000 = =V2T [1( + 20²
2
vn/5
Remark D.5. In particular, Corollary D.4 Dappiiss if the target function f is Lipschitz continuus with B₁ 0 and B2 = [f Lip [f However, continuity is not necessary. and even noisy labels would be admissible. We do not pursue this generality here.
E Proofs of the convergence theorems
In this appendix, we present the proofs of Theorems 2.1 and 3.3 In these, we combine the upper bound of the Rademacher complexity of the unit ball homogeneous Barron space in form of the generalization bound of Corollary D.4 with with a -ecnvvrgeeece argument (to guarantee the convergence of minimizers minimizers). The main ingredients in the proof of T-convergence are
the compact embedding theorem for homogeneous Barron space (to guarantee that every subsequence of f. has convergent subsequence) and
the direct approximation theorem for homogeneous Barron space (to obtain a bound on the lowest achievable energy using neural network with m neurons).
We first present convergence proofs in TP (1)) in Section E.1 followed by proofs of T-convergence in
Section E.2
We combine the arguments to prove the statements from the main body of the document
in Section
3
34

---

E.1 Convergence in L"(p)
We start by establishing convergence in (2) at an explicit convergence rate. Then, using this L2)2)) convergence we will extend this result to general ("))..
We introduce one of main theorem of this section, which gives explicit bound of L2 (2)))-lsss. For convenience, we denote 0 == (a, W,b) for the rest of the section.
Theorem E.1 (Lovevnvergence). Let 0 argming Rm.,, (0). If e and f* FQq, then with probability at least -6 over the choice of random points Ii, In we have
(Q")² (101)
log(n/ö)
R(fo) SC
(E[1l=13))
AQ" + Q" (E||z)) + 0² lf)
m
vn
up higher order terms in the small quantitiies (Am))1,m) mm ,n 1/2 log
Proof. Outline. We use Theorem C.5 for (2())) with f = f to obtain function for which
Rn.mm. is low. The empirical risk minimizer (ERM) has even lower risk. The weight decay penalty additionally provides norm- norm-bound in homogeneous Barron space for the ERM, and we use Corollary D.4 to control the generalization gap.
Step 1. Due to Theorem C.5
there exists a ² == (ā, w; b) E Rm x Rmxd x Rm+1 such that
RwD(() - 1s
(6)
and
4[f*]2 m
4[f
n
(f) -f S
Iw d1dd I
2 12
sup
||||||=1
We will always consider 5 such that (o8(1/6) < In this regime, plugging-in the bound the second moments of pmn from Lemma H.4
gives the corresponding bound
Rfo S 4[5"]
||||=||9 + 802 1og(1/0)
(7)
with probability 1 5. We will assume that this estimate is valid for the remainder of the proof. In particular, since minimizes Rn.m.., we find that
4[f*]²
log (1/8)
Rn, Rm),,
|||+=|1 S0?
+ [[]"]
m
Step Next, we bound [els and [b(0)) - f') (Q and A in Corollary D.4. We first bound Barron semi-norm by
1 Rr. R.. S 1 A [fols S
(0).
X
Moving on to bounding A, we find from the empirical risk bound
min | - f(x)) 15t5
min 1f - ( < 1sisn
- (Ra())
and in particular
min 1s8 [x((s))) V (m.
1si5n
With probability at least 1 - 8, we have
max 18
|
<
Egzn [||>|]] + a /22 log(n/8).
by Lemma H.2 Again, we assume that the estimate holds in the following. Hence, the index for which the minimum is attained in (8) satisfies the bound
|| | Exen [|=|]] 2 log(n/8)..
35

---

Combining the bounds on lfg(x,) - f (x;)) and the Lipschitz constants of fa. f", we find that
1/a (0)0 S 158 (x1 + (Vaae + [[']s) ||x;|| 1 $ v (0) [f + A Rnmaa(0) (Bema [|x=]]]]]] + v log(n/o)
Step 3. Comparing 1 to 0, we observe that
R(fa) (fo + +R(fa) (n((f))
- x (0) + R(fa)
R(Fa)
R R(fa) Rm,(Fo
where used the fact that @ is minimizer of Rn,m,, and (6). In the following, the bound on [Sg] to control the generalization gap.
Step 4. Recall that [als 2 [/*] + x Rn,mas(fa) [54] +(((m)-))) Thus, with probability least 25, we obtain the bound
(R (F)
(Irls + 1 RRma(()) (B||I|| o² + [f*]B)
log(n/6)
Vn
from Corollary D.4 for slightly modified constant > 0 (with B 0) and up to higher order terms in (Am) and n.
Step 5. By the union bound, all probabilistic bounds hold simultaneously with probability at least 1 45. In this case
E | (
Q² m
log( 1/5) o
(ng(n/))
R(La) c
AQ + [
(E | + + +f''s))
n
Vn
up higher order terms m log n/ Vn, (Am)
etc.
Since R(0)- [e [n) can interpret Theorem E. convergence statement in L²( suitable rate. The statement generalizes IP covceeeveeeeeee rate.
Corollary E.2 (LP-convergence). Let p E [o] and Ô as in Theorem
E.1
Then there exists
constant > 0 depending on E |x]]] [[][=]|]) and such that
[l5a - -' S E ( + (0')s+ 1-1/p
1 - 1
Proof. Since y is sub-Gaussian, we note that all moments of μ are finite: EE(1 + ||>]|)]]] < co for
all q E [1, 20). particular, if g is measurable function which satisfies (g(x) S C2 (1 | for some C'gg 0, then
1l9llip6o g. gp-1| s E 1/2 E 2(x 2 1/11/2
1/2 Illllz | 19122-1
If g = fo- f" B, then by the continuous embedding + LT(H) we find that
Ilfa 11 164) S CC - (0) + [fa f*).). Recall that 1ʃ8 (0) R (0) +0
We note that Corollary E.2 is generally suboptimal. Indeed, for S 2, the stronger bound
I1 fr (P S 11fa - = - -0 ((=
1/2
x+ log n
holds (2) embeds continuously into (P))).
36

---

E.2
Gamma-expansion of regularized risk functionals
As before, we denote 8, (a, W. b)n E E R R m Since R(fo) 1lfo - ff Theorem E.1 can be taken as a statement that fā_ + f as + 2o in (p). However, this does not tell us about the behavior of fa_ in a y--ull set, i.e. where the distribution l provides us no information. This interpolation between known values can be deduced from our next result. We first present simplified version, in which assume that we have already taken the limits m, n before taking À -> 0. We couple the limits n, aa An below.
We use the notion of I-convergence from the calculus of variations. For brief introduction, see Appendix --covvereence depends on the underlying topology of the space, and we make the following convention: We say that fx good, ff fx + f locally uniformly (uniformly on compact sets) and in (u). Other definitions are admissible and lead to the same general theory. Since Barron functions grow at most linearly at oc due to Lipschitz-contimuity we note that this is notion of convergence is generated by metric
[f(x) g(x) + ||=||2
d(f.g) = max zER
least on bounded subsets of Barron space. This suffices for all applications below and spares us from considering eeconvereeece more general topological spaces which is also possible.
Theorem E.3. Let
Ra(f) = I 11 - + Af[BB. RA : B -0,000,00))
We denote
FAU) Ra(1)
|lf - Afallal2ata + [A}
Fx B + 10,
A
[ffls if = for p.a.e.
F: B - [0, oo]
F(J) = 100
else
Then limx-0 Fx = with respect to the notion of convergence good defined above.
Notably, the l'-imii of RA itself would be zero at points interest. Rescaling consider Fx, instead has fits into the framework of eexaassions considered by Braides and Truskinovsky [2008]
Denote
F= E f p-a.e.) jt-a..)
Proof. Step
liminf-inequality. First consider ʃ E and assume that {SA}xso family of
functions such that fx topologies (Theorem C.3
good
Then by the compactness theorem for Barron functions in coarser we have the following:
lim inf FX(fx) 2 lim [AAl 2 [A = F())-
X-01
X-001
Now assume that f & F and that fx good f. We need to show that Fx(fx) > +oc. Since f & F, we see that R(f) = /|/ - fä (22 > 0. Denote & = /R()) and observe that there exists > 0 such that |lfx fl (w) < e/2 for all À by the definition of the notion of convergence. In particular, we find that
Ilf fr (ا((ا)) 2 ||f | IIfx fllea()) 2/2
for all À < A by the inverse triangle inequality and thus
lim inf Fx(fx) 2 lim inf (c/2)² t00.
2 It is easy generalize this to continuous limits, but if preferred, then À = Av, can be taken to be a discrete sequence converging zero.
37

---

Step 2. limsup-inequality. Again, we first consider the case f E F. Set fx for all À > 0 and observe that fA f as À 0* (trivially). By the same argument (fA))) F(f) ) = [f]s for all i.e. the constant sequence is recovery sequence since Fx(Sx) - F(J).
On the other hand, if f & F, then R(f) > O and thus Fx( +o0 F(f). Again, can the constant sequence as recovery sequence, somewhat trivially.
Corollary E.4. Assume that fx argmin Rr i.e. fx minimizes Rr. Then there exists f e B such that fx good,
Proof. Clearly JA minimizes R2 if and only if minimizes P2 = x- A-1Rx, Rx. We note that
[Als & A RRA( & X ARR((R(()) + [5 [f"]s.
In particular, by the compact embedding of Theorem C.3 there exists f E B such that fx good
to subsequence. By the properties of --convergence, we conclude that f is a minimizer of F.
We present a special case of Corollary E.4 in the setting of Proposition 3.2 which exploits the uniqueness of the minimizer. Recall the definitio of the radial average in 3 and få from Proposition
3.2
Corollary E.5. Assume that (0) 1, (x) if || > and μ satisfies the conditons Assume additionally that fx E argmin Rx, iee. fx minimizes Rx. Then
of Corollary 3.3
Av fx
good fä-
Proof. Step 1. Clearly fA minimizes R. if and only if it minimizes Fx = X-' Av fx is also minimizer of Fx since the functional convex and rotationally symmetric, so by averaging in radial direction, we are taking (continuous) convex combination of minimizers, which is minimizer again.
Step 2. We find that
[Av fAls S (A( & FX(A) S FX(F2) = [fā]s.
By the compactness theorem for Barron functions, Theorem
C.3 there exists f B such that
Av fx good f (up to subsequence). Since Fx + F in the sense of [-convergence, we find that a minimizer of F. Since is also radially symmetric, we find by (3.posittion3.2]thhaaa
fä- Step 3. By the exact same logic, we could show that every subsequence of (Av ʃx} has a further subsequence which converges to fd. By standard argument in topology, the whole sequence converges.
similar statement can be proved in the more complicated case where is neural network with finitely many neurons and Fx uses a finite data set rather than continuous expectation. In this case, the parameter must be coupled the number of parameters m and the number of data points n. such that À + 0, but not too quickly. The proof is more technically challenging variant of those of Theorem 3 and Corollaries E.4 and E.3] which utilizes the generalization bound of (.4 To this end, we first introduce new notion of convergence. That is, we define notion of convergence from the
parameter to function. We define notion of convergence by saying that 0g := (ax. Wt. bx) good
iff fee good f as k 00.
Theorem E.6. Consider the parameter space Om C Rm gmnd x 1mm+1 of neural networks with a single hidden layer of width m and the associated functions
fa(x) = bo +
a, I x bb)
Let myy An scale with n according to
We denote
R2 mm An (0)
Rn(so)
F, θ + 00)
Fg (()
+ RwD(O)
An
Xn
38

---

F(J) - { if p.a.e.
: B 10, [0,
++0 else
Then almost surely over the choice of data points, we have I lim Fn F almost surely with respect to the notion of convergence O₁. good, f defined above.
Proof. We use F as in ② throughout. In the proof assume that all stochastic quantities in Corollary D.4 and Theorem E.1 are satisfied with probability at least 1 6g for 5n = n The quantity log(ōn) therefore becomes comparable log n/ < Xn Since Σ 2
oc, find that all conditions are met for but finitely many n E N by the Borel-Cantelli Lemma. For questions of asymptotic convergence, may therefore assume that the statements of both Theorems apply without qualifying for high probability. Note that n 2 e for all n 2 2 as needed for Theorem
F.1
Step 1. liminf-inequality. Again, we consider the cases f F. and f $ F separately, First, when E F, apply the same method we did in Theorem E.3 For any sequence of parameters
good f, by Theorem C.3 the following holds:
On
lim inf Fz. (0w) 2 lirm inf (w,) 2 lim inm 2 [/]B P()). n-900
100000
The second inequality comes from [fee.I being an infimum of weight decay regularizers with any arbitrary probability measure on parameter space.
Second, when f & JF, need to show lim inf (r,( oo for any On good f. We distinguish two prototypical cases:
1. [J + toc as as above.
In this case F(O,) [f, too well by the same logic
2. lim sup, So,]s < +00. this case, we take = || f (12) 0. Then there exists N E N such that for all n 2 N we have
llfe. fl 2 |lf - flll - life. - fIlea 2
for all N by definition. Additionally
R., Rr, R(Jo.).
R(Jo..)
Fa Fa())) 2
+ +Joo.s
An Rn(fo,) - -R((o)))R(((o))) An
An
+ Vo,,1s.
Due Corollary D.4 and the arguments of Theorem have
to control the discrepancy
(ro.)..) R(fo..) O log n /n
in this case. Since log n/Vn An by assumption, we note that
Rn([8, -R((e,
lim
R 2-000
An
and thus
lim inf (+
22 +0) +00.
lim (.(O,)) 2 100
An
The same holds in the general by passing to subsequences.
Step limsup-inequallic As Theorem C.5 the case f & F follows from the lim inf-innnnnulii in
an essentially trivial fashion. We therefore only consider the case f E F. An approximating sequence in this case is constructed from Theorem C.5 as in Theorem E.1 or Theorem E.3 Namely, we find é,, such that
C( Fo(On) 5 Anm
(1+ log
+ [f
lim sup F, (0,) ) $ If n-+oc
Vn
39

---

Remark E.7. The key ingredients for the proofs of both Theorem E.1 and Theorem E.6 Theorem C.5 and Corollary |.4 but they are combined differently. While they are paired in Theorem E. obtain precise rate, they occur separately in Theorem E. Theorem C.5 is used for the lim sup- inequality while Corollary D.aaaryyDA4eeea DDAeettes in the proof of the lim int inf-inequality. Analogously, the condition An & logn, n/ n is used in the proof of the lim inf onf-condition while the fact that I & Xn is used in the proof of the lim sup-inequality.
E.3 Proof's of the main theorems
The statements of the Theorems in the main body of the text can easily be deduced from the statements proved in this Appendix.
Proof of Theorem 2.7 Convergence in LP holds by Theorem E.1 for < and Corollary E.2
(general p). The proof of uniform convergence follows from Theorem E.6lin the same fashion that
Corollary E.4 follows from Theorem E.3 The explicit bound is obtained from Theorem E. with 5n 4m
Proof of Corollary 3.3
This follows in the same way as the proof of Theorem 2.1 with modifications
as in Corollary E.5
Theorem 2.1 for finite data sets
Finally, we note that version of Theorem 2.1 holds if the data set S {01,-- ... In is kept fixed. The proof is a combination of those of Theorems E.3 [.6 as we deal with a finite approximating neural network, but do require generalization bounds. The details are left the reader.
Theorem F.1. We make the following assumptions.
1. Let S (1I.81))..... (Zn: yn)} be a fixed dataset of n data points I, Rd and labels Y/ E RR.
2. Let the loss function E(J,y) be the mean squared error MMSE (J (J,y) = =Is y/
3. Assume that Am a sequence of parameters such that Am + 0, 1/m < Am as m - cc.
Consider the regularized empirical risk functional R. IR" x mxd x R" + oc).
R.n (a. W,b)
1 Σ e (fo.w.a) vi) + Am (Iall + || l1
2n 2=1
2
Then if (a, W, 6) E argmin R. for all m E N, then every subsequence of fm := f(a,W,b)... has further subsequence which converges to some limit fo E B uniformly on compact subset of Rdd. The limiting function satisfies
fr
argmin (JEB:f(a)=W Wi
[f]s.
G Minimum norm interpolation in one dimension
Proof of Proposition 3.7 Any Barron function f is also Lipschis in particular differen-
tiable almost everywhere and f(b) ((a =f" fx(x da for all a, TR by Rademacher's Theorem
(see e.g. [Evans and Gariepy
2015 Section 3.).). In particular, there exist points x ,, (a,b)
such that
(( f(b) f(a) = b b 1 ('(x) S ('
1
a
If f' is not constant, both inequalities are satisfied strictly. Under the assumptions, there exists a E (xo, 21) and 1,n-1,44) such that
f((a) < [(x1) - f(x0)
505 [(x,) [(In-1) < {())..
(10)
En pn-1
40

---

Since the derivative f changes sign, we conclude by [Wojtowytsch] 2022 [f]s - d|p where the Radon measure μ the distributional derivative of f and | denotes the total variation measure y. Since f' is differentiable at a, 6, neither point is an atom of p and thus
Proposition 2.5] that
- ((5n-))
[(x1) f(x0) - 11(
[-" du dus
[ dlal S [f]o.
In In-1
I1 T0
Equality holds and only |x| = μu, i.c. if μ 2 in the sense of signed measures and if and only if the inequality in (10) can only be satisfied with equality. The first condition means that f must be convex, the second implies that the derivative of must be constant in the intervals (2o,21) and (2n-1, The same can easily be seen for the larger intervals (-o0, F1) and (Ix-1, 00).
H Sub-Gaussian random variables
In this Appendix we quickly gather some facts about sub-Gauss Gaussian random variables. For the reader's convenience, we provide full proofs. Experts are encouraged to skip ahead Appendix D
The first property we observe is a bound of expected maximum value of sub-Gaussian.
Lemma H.1. Lel yt be o² law μ. Then, the following holds:
and for = 1,..., n, let va be a iid random samples from
max SiSn
[|= S Exzn [|< + 2 logno.
Proof. From the sub-Gaussian assumption we have
log (Ez-al [ [exp (A(IIXII [[[X(]))] σ²x²
for some fixed > and all A 0. In particular, Jensen's inequality implies the sub-Gaussian maximal inequality
E max Isi5a
||>;|]]
E Exwn [|< + E
max 1st5n
(||ææ|) - -B|lz:1)]
(x max Isi5
$ E Exrn
[||+||]
+ X
log
IF
exp
(|æ=|| - 1|| (l)
< Exrn [||=|]]
+ X
log
E
exp (A(1lzi]| - Bl(==:1))
< Evry [||=1]]]
+ x
log
Σ i=1
E [exp (A(1 E|
1
(1802)
Eeou [||=]]]
+ X
log
exp
Egany [||=|]] + log n
o² X
for all À > We specifically select A v 2 2 making the bound
E max
||T|||| < E. Ez
||||||||| /2 no.
Next, we observe the concentration of maximum values among samples of sub-Gaussian distribution.
Lemma H.2. Let μ. be ---uub sub-Gaussian, and for = 1, n, let v; be a iid random samples from law μ. Then, with probability at least - 8, the following holds:
max 18t
||
< Ex-y [||=||] + a /2 log(n/8).
41

---

Proof. For all & >> O, we observe that
14 max Isisn
||aa||
2 Ezup
[||
+
1)
11"
max exp (A(l|z))l - -|]==)))) 2 exp(At) Isish
S ee E
max exp ((\[| - E | D)]
igisn
e n
E exp ||||=
X²0² 2
S exxp
-Xt +
For fixed t, the bound becomes tightest for x = f/o² with
t²
14
max Isisn
|
2 Eewn
[||0||] +1)
S 12 exp
S
8
20
82
S log
120/2 1
20²
Thus with probability at least 5, we have
|| < Rt == Exza | + a /22 log(n/6).
max ISiER
In the following Lemma, we investigate expectation of squared norm of sub- Gaussan near the tail. Lemma H.3. Let y be - Sub-Gaussian distribution in IRd, For any R > Ex>u||zl|, we have the following:
(R==BI==l)* 402
Exzp [1 [| [x)] S exp
V2T
(B]|==1)2 + 200
Proof. Recall that
t²
a (fx: ||,||| 2 (Ee' ||' + +)]) S exp
20
as demonstrated in the proof of Lemma H.I (consider 1). Thus
Ex [1|=0|] Bx((0) (x)) < / 8² >"1(lll=||| 2 8)) ds <
(s -
Elxx11)
8² exp
ds
n
202
R - B||=|)
E|
exp
exp
ds
4g2
2
RR B||
(s
El(=o|))
exp
exp
ds
4a²
4
R B1(x=11)
/2T
(BB|)r))2 + 2022
exp
4
In next Lemma we introduce how the mean of squared norm of sub- sub-Gaussian is concentrated. Lemma H.4 Assume P. is -u--ub- nau--Gassiinn distribution in Rdd. Let 31, ... Fn be iid random samples from law μ. Then, with probability at least 1 - 8.
| 2 < Enzy [|| + 80² max log((/8)
log (1/8)
42

---

In particular, o 2 e- then
| - [B[a]]] + +00² (1/()
1=1
with probability at least
Proof. Firstly since || || is -u- Sau-Saassian, from [Honorio and Jaakkola, 2014 Appendix BJ observe that |a 2|2 is (4v2o², 4o²) -ub-eeexxxpoo Next, by independence, for all and for all [8] I 402 we have the following:
)
E exp
A
1
E
| |1a01(9)
E exp (A(1 I. (1
1=1
320
exp
2
1=1
[32mo 122
exp
which implies that exponential [Adams
12,1 (4v2no² (o?))-sub-eyy sub-exxonenttil. Thus the tail bound of sub-
2022
Proposition 2.38] applied Ei= ||= yields
1
2
E
2 8
S exp (-1 min
s2
s
32no 4
Plugging-in in nt, we have the following:
b 2 Ex 1= 2t s exp 1 min
t2
p
320 40
Lastly, take 5 cxp (- fn min following:
V ) By rearranging the with respect to we have the
80² max (v (og() log(1/8)
))
11th
|as| Ezwu
|
S 6.
n
n
This proves the first part of the Lemma. Specifically, we have the following:
|xi|| Exnn ||]|| 2 80 log(1/6)
1r"
S 8 if em
71
n
log log(1/0) (1/8) Exany || 80²
n m
2 8 i 2
n1
71
The second inequality proves the last part of the Lemma.
43


Reviews:
[{"review_id": "b0rmsKFc7R", "reviewer": "Reviewer_jgQF", "summary": "This work studies the global minimizer of the empirical risk with $\\ell_2$ norm regularization of the parameters. It considers a one hidden ReLU layer architecture with square loss. \nThe main result roughly states that for some asymptotic regime on $\\lambda$ (reg parameter), $n$ (number of data points) and $m$ (network width), the minimizers of the considered regularized loss converge towards a minimal norm interpolant. Besides this asymptotic result, it also provides a convergence rate (in $L_2$ functional norm) of the estimated function towards the truth.\n\n-----------\n\nI acknowledge having read the author's rebuttal. The authors answered my main concerns and I thus raised my score. I still think that the writing needs a lot of polishing though and would advice the authors to take into account all the suggested modifications in their revised version.", "questions": "What is the main message of this work?\nWhat are the technical difficulties/main novelties of Theorem 2.1?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "I find the convergence result towards the truth very interesting, ie the second part of Theorem 2.1.\nGetting non-asymptotic generalisation bounds for the considered problem is of particular interest and I really enjoyed the proposed result, which gives nice dependencies in the different parameters $\\lambda,n,m$ and $[f^{*}]_{\\mathcal{B}}$.\n\nI think this result alone deserves a deeper investigation/understanding, as I believe it to be the most significant contribution of this work: I would suggest the authors to put an emphasis on this particular point of their main theorem as being their main contribution.", "weaknesses": "Overall, I find the paper misleading. The writing choices make the message unclear: I am unsure of the conclusions/insights brought by this work. Additionally, I think that the paper lacks contextualizing the results with the literature; numerous works are not cited, while they cleary deserve to be discussed in my opinion.\nLastly, I find the overall contribution rather weak: to me, the only real contribution is the Theorem 2.1, which is not discussed enough and might not be a strong enough result to justify a publication alone. The remaining of the paper is then mostly corrolaries of Theorem 2.1 and numerical experiments, which are of minor interest to me.\nI detail all these points below.\n\nFirst, I find the writing misleading/message unclear. Already in the abstract, the following sentence is problematic: \"Our analysis yields convergence to a minimum norm interpolant [...]\". This paper is actually about the convergence of the global minimum towards the the min norm interpolant when $n,m,\\frac{1}{\\lambda}$ converge to $0$, but I first understood this sentence as \"some optimisation algorithm converges towards the min norm interpolant when $t$ (iterations of opt algorithm) go to $\\infty$\". The abstract does not make it possible to distinguish between both cases, while the latter is actually much stronger.\nThis might seem a small detail, but I think already illustrates how the message is not totally clear and it is sometimes hard to understand where the authors want to go.\nThe other main confusion to me comes from the Section 3. I only understood its purpose once I read Section 5. Actually, I think Section 3 does not deserve so much space, as its goal is only to provide two examples where the min norm interpolant is exactly known (which are not original contributions). I would prefer these results to just be small paragraphs in Section 5.1 and 5.2. \nThe message is not even made clearer in the conclusion: only one sentence summarises the paper contribution and interest, while the remaining of the conclusion are experimental observations that could easily be postponed to the appendix.\nAs another example, implicit bias is mentioned a lot in the literature but is never described, making it hard to follow for a non-expert.\n\nA lot of significant related works are not cited, while they clearly deserve to be discussed into details. \nFor example, there are many works that aim at describing minimal norm interpolants which are not even cited here [1,2,3]. Also, the semi-Baron norm in the unidimensional case has also been characterised by [4].\nThe seminal work of [5] is missing in the implicit bias paragraph. Actually the implicit bias does not exactly consider the same norm on the parameters, as it also counts the bias terms (see [2] for a further discussion on this aspect). Is there a particular reason you are not counting the norm of the bias terms here?\nAbout the generalization bounds for small Baron norms functions, I believe it to be closely related to [7,8]. How are the result from Appendix C different?\nIt has recently been proven by [9] that the implicit bias of GD does not always lead to minimal norm interpolants, but sometimes to different kinds of interpolants: this contradicts some of the claims made in some paragraphs of the paper (eg  in Section 4).\nAbout the conservation of symmetries when training neural networks, I guess you could also mention [10].\n\nThe main result of the paper is Theorem 2.1. I think it deserves a much deeper discussion. The paper gives no idea of how the authors manage to prove it, where lie the difficulties in proving it, what are the reasons for the different needed assumptions/scaling of the parameters? In particular, I would have believed that the Theorem would hold with only assuming $\\lambda_n\\to 0$ and $m_n\\to\\infty$. What is the reason for needing the second limit in equation (1) ? I believe it hides the difficulty of the proof, as the first part of Theorem 2.1 seems easy to me: the result would be a stronger contribution if it indeed is not so easy to prove. \nActually I really like the second part of the Theorem (rate of convergence) and believe it to be a significant result. I would have liked a deeper discussion on it. Why do we need $\\mathbb{E}[\\|x\\|]+\\sigma^2\\geq 1$? I believe this is not really needed as we could otherwise deal with a rescaling argument.\nAbout the fixed $n$ setting, [6] actually proved that having any $m\\geq n+1$ does not change the minimum of the problem (ie having $m=n+1$ or $m=\\infty$ is the same), so I actually think having separately $\\lambda_m\\to 0$ and $m\\geq n+1$ is sufficient in that case.\n\n-----------------\nMinor remarks:\n- in Section 2, why postpone the definition of $\\mathcal{B}$? it can be very shortly defined\n- In Section 3.1, the minimum might be different as you also need to consider non-smooth functions. However, using the characterisation of  [4], it is actually the correct minimum\n- Theorem 3.3 only seems to be a corollary of Theorem 2.1 and Proposition 3.2 (and thus deserves to be called a corrolary in my opinion)\nAlso, $f^*$ is not precised here.\n- Section 5: the implicit bias also depends on the initialisation regime -> it should also be mentioned\n- the sentence line 239 is not clear. Actually, what is happening here seems close to what is described in [2]: for small initialisation, the implicit bias towards min norm (counting bias terms) interpolant kicks in, which leads to the unique min $|x|$. For larger initialisation, there is no such implicit bias and only an explicit regularisation (which does not count the bias terms here) and thus allows for less sparse minima as $f(x)=\\max(|x|,1)$.\n- for $\\lambda=0$ and large $\\alpha$, it seems we are in the NTK regime. In this case, we have another implicit bias towards min spline interpolant (see eg [11])\n- in Section 5.2, what is $n$? \n\n-----------------\nReferences:\n[1] Debarre T, Denoyelle Q, Unser M, Fageot J. \"Sparsest piecewise-linear regression of one-dimensional data\"\n[2] Boursier E, Flammarion N. \"Penalising the biases in norm regularisation enforces sparsity\"\n[3] Stewart L, Bach F, Berthet Q, Vert JP. \"Regression as Classification: Influence of Task Formulation on Neural Network Features\"\n[4] Savarese P, Evron I, Soudry D, Srebro N. \"How do infinite width bounded norm networks look in function space?\"\n[5] Lyu K, and Li J. \"Gradient descent maximizes the margin of homogeneous neural networks\"\n[6] Wang Y, Lacotte J, Pilanci M. \"The hidden convex optimization landscape of regularized two-layer relu networks: an exact characterization of optimal solutions\"\n[7] Kurková V, Sanguineti M. \"Bounds on rates of variable-basis and neural-network approximation\"\n[8] Bach F. \"Breaking the curse of dimensionality with convex neural networks\"\n[9] Chistikov D, Englert M, Lazic R. \"Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs\"\n[10] Hajjar K, Chizat L. \"Symmetries in the dynamics of wide two-layer neural networks\"\n[11] Jin H, Montúfar G. \"Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their detailed feedback. We are unable to fully address all points within the character limit, but we will take their full feedback into account for a revision.\n\n> What is the main message of this work?\n\nThe key messages of this work are:\n\n1. Neural network minimizers of *regularized* risk functionals converge to minimum norm interpolants (Section 2).\n\n2. Empirically, common gradient-based optimization algorithms often find minimum norm interpolants even without regularization, but there is a notable impact of algorithm and initializations scheme.\n\n3. Especially when minimum norm interpolants are known explicitly, it is possible to use this knowledge productively to study the impact of the choice of optimization algorithm and initialization scheme on what kind of solution is found (Sections 3, 4, 5).\n\nThe program is implemented primarily for neural networks with a single hidden layer, but parts of it generalize to function spaces for more general situations (Lines 115-117).\n\n> What are the technical difficulties/main novelties of Theorem 2.1? \n\nDue to spatial constraints, this discussion was postponed to the beginning of Appendix D, where the generalization bound is proved. We will be sure to include it in the main body in a revised version. The main contributions are:\n\na) We deal with general sub-Gaussian data distributions $\\mu$ rather than data distributions with compact support.\n\nb) We do not control the magnitude of the bias variables.\n\nc) We consider l2-loss, which is neither globally Lipschitz-continuous nor bounded.\n\nd) We use the tools of $\\Gamma$-convergence from the calculus of variations to prove *uniform* convergence on compact subsets, even away from the support of the data distribution (see also the scaling constraints between $m, n, \\lambda$ below).\n\nThe combination of a), b), c) requires a more careful analysis than previous works. To the best of our knowledge, d) is novel and the tools of $\\Gamma$-convergence have not been used in this context before. \n\n> The abstract does not make it possible to distinguish between [convergence of minimizers and convergence of an optimization algorithm]. [...] A lot of significant related works are not cited, while they clearly deserve to be discussed into details. \n\nWe thank the reviewer for bringing this to our attention and will be sure to fix this.\n\n> Is there a particular reason you are not counting the norm of the bias terms here? \n\nWe are not considering the magnitude of the biases in order to utilize known minimum norm interpolants. We are aware that this follows the convention of only part of the research community (such as Hanin (2021) and Ongie et. al. (2019) in our references).\n\n> About the generalization bounds for small Baron norms functions, I believe it to be closely related to [7,8].\n\nWe would like to ask if [7] is the intended article as we were unable to find any mention of generalization there.\n\n> Why do we need $\\mathbb E\\|x\\| + \\sigma^2 \\geq 1$? I believe this is not really needed \n\nThis is true, but the exact presentation of the generalization bound would change. We assumed this lower bound to simplify the presentation and some constants. It is neither crucial to our argument, nor did we find it overly restrictive. \n\n> About the fixed $n$ setting [...] having separately $\\lambda\\to 0$ and $m\\geq n+1$ is sufficient in that case.\n\nUsing $m\\geq n+1$ neurons, perfect interpolation is always possible. In general, there are minimizers which can be represented using $m+1$, but not $m$ neurons (for instance, add break points to a convex one-dimensional function in the setting of Section 3.1). Different versions of this result can be stated under different assumptions.\n\n> In Section 3.1 [...] you also need to consider non-smooth functions.\n\nWe concur: ''The same estimate holds for non-smooth Barron functions if the second derivative is interpreted as a Radon measure.'' (Line 126 - 127) We do not require smoothness in Proposition 3.1 and state that any convex interpolant with the correct derivatives at $\\pm\\infty$ is a minimizer. Our goal was to give intuition without technicality. \n\n> What is the reason for needing the second limit in equation (1)? \n\n$\\lambda > 0$ is used to bound the Barron norm. If $\\lambda$ is too small compared to $\\log n/\\sqrt n$, then an undesirable trade-off between fitting labels at a small number of data points and keeping the norm small may be possible. The network may fail to generalize. If on the other hand $\\lambda$ is too small compared to $1/m$, then it may be possible to perfectly fit all labels (e.g. if $m > n$), but without sufficient regularization, we lose control of the norm and thus the generalization gap. More precisely:\n\n$$\n[f_n ]_ {\\mathcal B} \\leq \\frac1{ \\lambda_n }  \\widehat{ \\mathcal R}_ {n,m_n, \\lambda_n} (a_n, W_n, b_n)\n\\leq  \\frac 1 {\\lambda_n} \\left( \\frac1{m_n}  + \\frac{\\log n} {\\sqrt n} + \\lambda _n [f] _ {B} + l.o.t. \\right)\n$$\n\nWe require that $1/(\\lambda_n m_n), \\log n /(\\lambda_n\\sqrt n) \\to 0$ to prove the $\\Gamma$-convergence of rescaled risk functionals $\\frac1{\\lambda_n} \\widehat{\\mathcal R}_ {n,m_n,\\lambda_n}$ to $[\\cdot]_{B}$ and $+\\infty$ respectively (depending on whether the data is fit). \n\nThe details can be found in Appendix E.2. By proxy, it is required for the uniform convergence to minimum norm interpolants on compact sets, also away from the support of the data distribution $\\mu$. Theorem 2.1 holds more generally without an assumption of exact minimality if\n$$\n\\widehat{\\mathcal R}_ {n,m_n, \\lambda_n} (a_n, W_n, b_n) - \\lambda_n [f^*]_{B} \\ll \\lambda_n.\n$$\nThis is ensured for minimizers by the second limit.\n\n> In Section 5.2, what is $n$?\n\nFor plots in Figure 3, the training set size varied (10^4, 10^5 and 10^6 data points respectively) as described in the caption and the plot titles. We will fix the fact that the plot labels are too small to read. For Figure 4, $n= 10^5$. We will add this information to the caption.\n"}, {"review_id": "Ah6jcivKBu", "reviewer": "Reviewer_etEC", "summary": "The authors study the generalisation behaviour of two-layer neural networks in\nthe over-parameterised regime, where the network can perfectly interpolate its\ndata in many different ways. What can we say about the functions that are learnt\nusing SGD in these cases? The authors address this question using mathematical\nand experimental work.\n\nThey first consider the setting where for a fixed dimension $d$ (I think), a\ntwo-layer ReLU network of width $m$ trained on a data set with $n$ points with\nlabels generated by a target function $f^*$. In the limit where $n$ and $m$ go\nto infinity, they show that minimisers of the mean-square loss with small weight\ndecay penalty, resemble a minimum norm interpolant, which is the equivalent of a\nmaximum margin classifier in classification.\n\nSince there is no guarantee that a neural network trained using gradient descent\nwill converge to an interpolant, or which minimiser of its loss it converges to\nif it converges to zero loss, the authors also conduct a series of numerical\nexperiments to analyse the functional norms of neural networks trained using SGD\non low-dimensional inputs. The experiments confirm the importance of\ninitialisation scale, and small initial weights in particular, and finds some\npeculiarities in the behaviour of Adam.\n\nPS: While I know some of the papers on minimum norm interpolators, I am not\nknowledgeable in the proof techniques used in this paper.", "questions": "n/a", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The author provide a rigorous statement about the behaviour of a certain limit\n  of two-layer neural networks.\n- They provide a numerical exploration of neural networks trained using\n  different gradient-based optimisers, including an analysis of the trade-offs\n  between explicit regularistaion and implicit bias.\n", "weaknesses": "- A minor, but important point: the axes labels and titles of the plots are way\n  too small to be readable.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their kind feedback and insightful comments. We will replot the Figures in the final version with larger labels, if accepted."}, {"review_id": "YI2QSYSYCS", "reviewer": "Reviewer_CjUq", "summary": "This paper studies the optimization and generalization for two-layer ReLU neural network models.\nSpecifically, \n* for the generalization part, this paper gives a Barron-norm based generalization bound for the minimize of the $\\ell_2$-regularized squared loss function. The bound involves with three terms: sample size $n$, model width $m$ and the regularization weight $\\lambda$.\n* for the optimization part, this paper conducts numerical experiments on two examples that the minimum Barron-norm solutions can be explicitly obtained, namely, the one-dimensional example and the radially symmetric bump function example. Numerical results reveal that optimization algorithms, initialization scale, and the regularization weight affect the minimum norm interpolant together.", "questions": "In case I missed something, please correct me if I am wrong.\n\n* I don't understand why Theorem 2.1. shows that *minimizers of regularized risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit.* (Line 26-27).\nAs far as I can understand, the generalization bound is for the gap between the model function and the target function, but why this result can show that the regularized risk minimizer converges to the minimum norm interpolant?\n\n* Also for the generalization bound in Theorem 2.1., it seems that a two-layer ReLU neural network can learn a target function even for $\\frac{m}{n} \\xrightarrow{} 0$. \nTo see this, we let $m = \\sqrt{n}$, $\\lambda =  n^{-1/4}$, which satisfy the condition (1), then the bound (2) goes to zero as $n$ goes to infinity. That means a two-layer ReLU neural network even with the parameter size less than the sample size can learn an arbitrary target function as long as its Baron semi-norm is finite? \nThis is counterintuitive as we usually need to use a large model (with parameter size far greater than the sample size) for learning.", "limitations": "NA", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is written clearly and the ideas have been presented in the proper order.\n* The studied topic on the interplay of regularization weight, initialization scaling and the optimization algorithm is important to the deep learning community, which may enhance comprehensive perception for optimization and generalization.", "weaknesses": "In case I missed something, please correct me if I am wrong.\n\n* The related work description is not comprehensive for me.  Since I am not that familiar with the Barron-norm related fields, I don't know if there exists other works that study the generalization of neural networks in the Barron space. If there indeed exists, it would be better if the authors could make some comparisons on the generalization bound in Theorem 2.1. with others that are related.\n\n* The Baron semi-norm of a general function is usually not easy to assess. In this paper, two examples that can explicitly derive the Baron semi-norm are given: the one-dimensional example and the radially symmetric bump function. \nHowever, these examples are still restrictive in deep learning models. When we look into the definition for the Baron semi-norm, I try to understand this as the expectation version for the $\\ell_2$ norm of the weight parameters, which is easier to calculate.\nIn that sense, I am not sure what are the real advantages for us to consider the Baron semi-norm when compared with the $\\ell_2$ norm?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n> The related work description is not comprehensive for me. Since I am not that familiar with the Barron-norm related fields, I don't know if there exists other works that study the generalization of neural networks in the Barron space. If there indeed exists, it would be better if the authors could make some comparisons on the generalization bound in Theorem 2.1. with others that are related.\n\nThere exist previous works which study generalization in Barron space, most prominently E et. al.: *A Priori Estimates of the Generalization Error for Two-layer Neural Networks*. Due to spatial constraints, the discussion was postponed to the beginning of Appendix D, where the generalization bound is proved. If the referee believes this to be helpful, we will move it to the main article in the final version, if accepted.\n\nThe novelty in our work as it pertains to generalization is the following:\n\n1. We deal with general sub-Gaussian data distributions $\\mu$ rather than data distributions with compact support.\n\n2. We do not control the magnitude of the bias variables.\n\n3. We consider l2-loss, which is neither globally Lipschitz-continuous nor bounded.\n\nThese generalizations require greater technical care and lead to much more involved proofs in some places. They are also the reason behind a logarithmic term which is not present in the calssical setting. \n\nGoing beyond our contribution on generalization, we are unaware of other works that obtain uniform convergence on compact sets in this general setting.\n\n\n\n> The Baron semi-norm of a general function is usually not easy to assess. [...] what are the real advantages for us to consider the Baron semi-norm when compared with the $\\ell^2$-norm?\n\nNorms of general functions are often hard to compute - the $L^\\infty$-norm or Lipschitz constant of a function of many variables is not easy to determine. The same is true of the Barron norm.\n\nFor functions given by a (finite) neural network with a single hidden layer, the Barron norm is the sum of the squares of all weights, except in some pathological situations due to the non-uniqueness in function representation\n$$\n0 \\equiv a\\sigma(w\\cdot x+b) + (-a)\\sigma(w\\cdot x+b).\n$$\nThe sum of squared weights is always an upper bound and easily computed. Thus (an upper bound for) the Barron norm of a given network with one hidden layer is easy to come by. For a general target function $f:\\mathbb R^d\\to \\mathbb R$ which one wishes to approximate on the other hand, it can be very difficult to compute such bounds. \n\nFor general functions, the expectation enters as *infinitely wide neural networks are represented by continuous rather than finite sums (integrals) or normalized sums (expectations)*, and many functions which can be approximated well by neural networks with bounded weights cannot be represented exactly. The expectation could be replaced by an integral with respect to a measure or signed measure as detailed e.g. by E and Wojtowytsch: *Representation formulas and pointwise properties for Barron functions*.\n\n\n> I don't understand why Theorem 2.1. shows that *minimizers of regularized risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit.*\n\nConvergence is stated in the theorem as \"If $(a,W,b)_n$ \n\n$\\in argmin \\widehat{ \\mathcal R}_ {n,m_n, \\lambda_n}$ for all $n \\in \\mathbb N$, then every subsequence of $f_n:= f_{(a,W,b)_ n}$ has a further subsequence which converges to some limit $\\hat f^* \\in \\mathcal B$ with $\\hat f^* =f^*$ $\\mu$-almost everywhere and $[\\hat f^*]_ {\\mathcal B}\\leq [f^*]_ {\\mathcal B}$. Convergence holds in $L^p(\\mu)$ for all $p<\\infty$ and uniformly on compact subsets of $\\mathbb R^d$.\" (Line 92 - 94)\n\nThe fact that $\\hat f^*$ is a minimum norm interpolant follows from this since $[\\hat f^*]_ {\\mathcal B} \\leq [f^*]_ {\\mathcal B}$ for all functions $f^*$ which match the desired values $\\mu$-almost everywhere. In $L^2(\\mu)$, also a convergence rate for $\\|f_n - \\hat f^*\\|$ is given here, while the statement for uniform convergence is purely qualitative. In $L^p(\\mu)$ for $p<\\infty$, a (potentially sub-optimal) rate is obtained in Corollary E.2. \n\nTheorem 2.1 relies in part on a generalization bound (Corollary D.4 in Appendix D), but it additionally uses a Direct Approximation Theorem to construct a function of low error which serves as an energy competitor for the minimizer. The proof of locally uniform convergence uses the notion of $\\Gamma$-convergence from the calculus of variations. It thus goes beyond a mere generalization bound.\n\n\n\n> Also for the generalization bound in Theorem 2.1., it seems that a two-layer ReLU neural network can learn a target function even for $\\frac mn\\to 0$. [...] This is counterintuitive as we usually need to use a large model (with parameter size far greater than the sample size) for learning.\n\nThe observation is correct. We believe that the difference arises since we do not take into account the training process in this part of the article. Commonly, overparametrized models are used as they are less likely to get stuck in local minima, but from the approximation and generalization perspective, underparametization is not a problem. In this case, we are in the first part of the 'double descent curve', and regularization may not be required (although we do not prove this).\n\n\n\n\n\n"}, {"review_id": "4vICHMTY7r", "reviewer": "Reviewer_Gs5Q", "summary": "Paper summary:\n* The paper studies the convergence properties of 1-layer Relu nets trained on a regression task with Tikhonov regularisation. The main contribution of the paper is Theorem 2.1, in which the authors prove that under some assumptions on the data distribution and on the optimisation trajectory (or in the limit at they specify in line 110), the model converges to a function f^* that is equal to the optimal unknown data generating function f^* almost everywhere but  f^* has smaller Barron semi-norm. \n * Assumptions of Theorem 2.1: as stated in the theorem and in the main text, the assumption that at each iteration the function f_n minimises the empirical risk given by the current sample is a strong one, but (line 110) the authors specify that the proof holds as long as in the limit of n -> infinity the empirical risk tends to 0.\n * After theorem 2.1, the authors provide two examples in Section 3, first for one-dimensional examples and then for radially symmetric bump functions. In these cases, they provide characterisations of the minimum norm interpolant.\n  * From Section 4 onwards, the authors move on to discuss generalisation, optimisation and implicit bias. \n  * In Line 170 the authors mention the connection between Barron semi-norms and generalisation, following with connections of related work in the space of generalisation and optimisation (though plenty is missing here, see below). \n  * In Section 5, the authors provide a set of experiments in which they test the behaviour of various optimisers on some of the tasks that they discussed previously (optimisers include SGD, SGD with momentum, Adam, BFGS). The experiments assess the effects of data dimensions (Figure 1), optimisers, decay values and gain (Figure 2, where they show the importance of gain in learning smooth convex functions across optimisers), and the gap between the optimisation solution and the optimal interpolation solution at a data dimension (Figure 3). \n", "questions": "Can the authors comment on the weaknesses above, address the connection between the theoretical aspect of the work and the optimisation aspect discussed in the second part of the paper, as well as the other aspects highlighted in the weakness section above.", "limitations": "The authors did not discuss limitations in their work (as recommended by the conference). I have highlighted a few limitations above, including the limited applicability inside the 1 layer network family and missing connections with the implicit regularisation literature coming from optimisation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n* The paper tackles an important problem: understanding the convergence properties of neural networks (albeit in a limited setting) and connects it with generalisation. The second half of the paper assesses the impact of optimisation in a series of limited settings. \n* The paper is clearly written, though at times relies too much on previous work and the setting is not clear enough, such as lines 189-200.\n", "weaknesses": "Weaknesses:\n  * The paper applies to a very limited setting, namely 1 layer Relu nets trained on regression tasks.\n  * The second part of the paper (Section 4 onwards, experiments on optimisation and implicit bias) does not connect strongly with the first. \n  * There is nothing in Thm 2.1 about the implicit bias of specific optimisers. Moreover, this area has been studied extensively in the literature (beyond 1-layer nets), and that literature is not included or mentioned at all in the entire paper. Relevant works include Barrett and Dherin, Smith et al, the momentum recent paper. These works can help explain some of the empirical results shown in the paper, since they show that GD, SGD and SGD with momentum respectively minimise `||\\nabla_{\\vtheta} L||_2^2` where L is the loss function and $\\theta$ are the learned parameters. This quantity is very connected to parameter norms in general networks, though perhaps not in the case of 1-layer Relu nets trained on regression tasks with mean squared error , since there  ``||\\nabla_{W} L||_2^2 = (\\sum_i  f(x_i) - y_i)  \\nabla_{\\vtheta} f(x_i) and \\nabla_{\\vtheta} f(x_i) = Relu’(W x_i + y_i ) x_i``.\n  * The paper title (and parts of the paper) mention “shallow” neural networks. The paper does not work on shallow networks (which are usually referred to networks with a small number of layers) but to 1 layer networks. I suggest the authors replace the wording “shallow” and use the more appropriate language. \n  * The plots are  are lacking relevant information such as x and y axis labels  and hard to read physically (legends are very small, and the lines use the same type of line and similar colours). \n  * Other related works missing: when describing the study of the bias of SGD towards flatness, many recent works are missing, including  Cohen et al,  Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability, who show how increasing learning rates leads to SGD traversing flatter areas. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper applies to a very limited setting, namely 1 layer Relu nets trained on regression tasks.\n\nAs we note in the final paragraph of Section 2 (Lines 115 - 117), our methods apply to (regression problems in) all function spaces which satisfy a Rademacher bound, a suitable compact embedding and a direct approximation theorem. Spaces with such properties were developed for instance by E and Wojtowytsch in *On the Banach spaces associated with multi-layer ReLU networks* in the context of multi-layer perceptron architectures and non-linear function classes for infinitely deep ResNets (\"neural ODEs\") with these properties are developed by E et. al.: *The Barron Space and the Flow-Induced Function Spaces for Neural Network Models*. However, even in those established settings, minimum norm interpolants are not known, complicating empirical validation. We believe that this may become more accessible within the next years.\n\n\n> The second part of the paper (Section 4 onwards, experiments on optimisation and implicit bias) does not connect strongly with the first.\n\nWe consider the second part of the paper complementary to the first: In the first, we establish the significance of minimum norm interpolation with regularization, while in the second, we empirically study the impact of regularization and use minimum norm interpolants to study the implicit bias of optimization algorithms. \n\n    \n> [...] This area has been studied extensively in the literature (beyond 1-layer nets), and that literature is not included or mentioned at all in the entire paper.\n\nWe would be very grateful to the referee if they could provide a more precise reference for Smith et. al. so that we can incorporate missing works in the final version of the document.\n\n    \n> I suggest the authors replace the wording \"shallow\" and use the more appropriate language.\n\nWe believe that \"perceptra\" in place of \"shallow networks\" would be appropriate.\n\n> The plots are are lacking relevant information.\n\nWe will replot the figures to address their deficiencies. \n\n\n> Other related works missing.\n\nWe are grateful to the referee for pointing us to missing works and will use the feedback when preparing the final version, if accepted."}, {"review_id": "6unH2DAylq", "reviewer": "Reviewer_5DHL", "summary": "This paper investigates the behavior of shallow ReLU networks in terms of convergence to minimum norm interpolants of given data. The authors present provable results when explicit regularization is included and empirical evidence when it is not. The paper concludes with a summary of the authors' empirical insights into the implicit bias of neural network optimizers. . The authors suggest that minimum norm interpolation serves as a useful testbed to study the implicit bias of optimizers, initialization, and regularization, and anticipate the extension of this analysis to deeper networks in the future.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors summarize their empirical insights into the implicit bias of neural network optimizers, providing a concise overview of the behaviors exhibited by various algorithms. \n\nThe discussion on the impact of initialization scale, optimization algorithm choice, and explicit regularization is informative and relevant to understanding the behavior of shallow ReLU networks", "weaknesses": "Further discussion on the trade-off between fit and regularization, as well as potential methods to balance these factors, would enhance the practical understanding of these observations.\nAlso, it would be valuable if the authors briefly discussed the potential practical applications or implications of their findings to further motivate researchers in the field.\n\nThis work is a bit out of my scope. Please discard my rating for final decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and their forthcoming assessment of their level of expertise. In order to make the article accessible and interesting to a broad audience, we will include a more detailed discussion of the fit/regularization trade off and potential implications in the final work, if accepted.\n\nAlso, in the context of potential implications, we have outlined some of potential implications we expect in Section 1 of our paper. We anticipate our work to yield insights into understanding how a neural network works outside the support of data distribution (Section 2), which is one of the main interests in the topics of distributional shift and adversarial stability (Line 23 - 27). In addition, numerical experiments we conducted (Section 4, 5) can provide a useful benchmark to study the implicit bias and explicit regularization in optimization, by enabling comparisons between numerical solutions from optimization algorithms and theoretically known minimum norm interpolant (Line 28 - 31)."}, {"review_id": "GIFZemLlMR", "reviewer": "Reviewer_hUXj", "summary": "This paper investigates how shallow ReLU networks interpolate between known regions as the number of data points and parameters tends to infinity. The authors prove that minimizers of regularized risk functionals converge to minimum norm interpolants of a given target function in an infinite parameter limit. The authors study the performance of optimization algorithms by comparing numerical solutions to the minimum norm interpolant in cases where the latter is known explicitly. ", "questions": "The paper does not show the effectiveness for practical neural network optimization, such as CNN and Transformer. ", "limitations": "NA", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper has rigorous theoretical proof. ", "weaknesses": "The paper does not show the effectiveness for practical neural network optimization, such as CNN and Transformer. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The paper does not show the effectiveness for practical neural network optimization, such as CNN and Transformer.\n\nThis is true, and it is not the article's goal. While more advanced network architectures are of great practical interest, the theoretical instruments to describe them are not sufficiently developed to achieve comparable results in those cases, as the function space theory is not readily available. As we note in the final paragraph of Section 2 (Line 115 - 117), our methods apply to all function spaces which satisfy a Rademacher bound, a suitable compact embedding and a direct approximation theorem. Spaces with such properties were developed for instance by E and Wojtowytsch in *On the Banach spaces associated with multi-layer ReLU networks* in the context of multi-layer perceptron architectures. Non-linear function classes for infinitely deep ResNets (\"neural ODEs\") are developed by E et. al.: *The Barron Space and the Flow-Induced Function Spaces for Neural Network Models*. However, even in those established settings, minimum norm interpolants are not known."}]

Author rebuttal:

Meta-review: The authors investigate the generalization performance of two-layer neural networks in the interpolation regime. They provide both an asymptotic convergence result on the regularized empirical risk minimization (ERM) towards a minimal norm interpolant and a convergence rate of the estimated function towards the ground truth. This paper lies on the borderline, with both reasons for acceptance and rejection. Reviewers acknowledge the paper's valuable theoretical contributions. However, they express concern over the clarity of the presentation, suggesting substantial rewriting for improved understanding. I recommend accepting the paper, but I strongly advise the authors to include the reviewers' suggestions in the final camera-ready version.
Decision: Accept (poster)