Quantitative convergence of trained neural networks to Gaussian processes
Authors: Andrea Agazzi
Eloy Mosig García
Dario Trevisan
Venue: NeurIPS 2025 poster ()
Keywords: Neural networks
NTK
Gaussian process
Wasserstein distance
Gaussian approximation
shallow neural networks
wide limit
infinite-width neural network
Primary area: probabilistic_methods
TL;DR: We prove quantitative convergence estimates for single layer neural networks in the NTK regime to gaussian processes at positive training time

Abstract: In this paper, we study the quantitative convergence of shallow neural networks trained via gradient descent to their associated Gaussian processes in the infinite-width limit. 
    While previous work has established qualitative convergence under broad settings, precise, finite-width estimates remain limited, particularly during training. 
    We provide explicit upper bounds on the quadratic Wasserstein distance between the network output and its Gaussian approximation at any training time $t \ge 0$, demonstrating polynomial decay with network width. 
    Our results quantify how architectural parameters, such as width and input dimension, influence convergence, and how training dynamics affect the approximation error

Quantitative convergence of trained single layer neural networks to Gaussian processes
Eloy Mosig Gareía Department of Mathematics University of Pisa
Andrea Agazzi
Department of Mathematics and Statistics University of Bern Alpeneggstrasse 22, 3012 Bern
Largo Bruno Pontecorvo, 5, 56127 Pisa PI, Italia eloy mosig@phd...mooo unipi.it.
andrea. wwwzziguniiern @un ch
Dario Trevisan
Department of Mathematics University Pisa
Largo Bruno Pontecorvo, 5, 56127 Pisa PI, Italia dario. trevisan@unipi..i
Abstract
In this paper, we study the quantitative convergence of shallow neural networks trained via gradient descent to their associated Gaussian processes in the infinite- width limit. While previous work has established qualitative convergence under broad settings, precise, finite-width estimates remain limited, particularly during training. We provide explicit upper bounds on the quadratic Wasserstein distance between the network output and its Gaussian approximation any training time 0, demonstrating polynomial decay with network width. Our results quantify how architectural parameters, such as width and input dimension, influence convergence, and how training dynamics affect the approximation error.
Introduction
Deep neural networks have achieved remarkable success across a wide range of tasks in computer vision, natural language processing, and scientific computing, often surpassing traditional models by large margins LeCun et al. [2015], Goodfellow et al. [2016]. This empirical progress has sparked substantial interest in understanding the theoretical principles underlying their behavior, particularly in the overparameterized regime, where the number of parameters is larger than the one of training samples.
major avenue of theoretical investigation in this sense focuses on studying the properties of neural networks in the infinite-width limit. For instance, when the network's parameters initialization sampled from a Gaussian distribution, was shown Neal [1996], de G. Matthews et al. [2018] that the network's output converges to Gaussian process width tends infinity, providing tractable framework for theoretical analysis.
This perspective was significantly extended by the introduction of the Neural Tangent Kernel (NTK) framework Jacot al. [2018], allowing to characterize the training dynamics infinitely wide neural networks under gradient descent in function space. In this limit, the network evolves approximately linearly around its initialization, and training can be understood as kernel regression with a fixed kernel which depends the the architecture only, and is evaluated on input data. This lincarization dramatically simplifies the analysis of generalization and convergence, and has led to a large body of theoretical work on the expressivity and limitations of infinitely wide networks.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

However, the practical relevance of NTK-based analyses hinges on the accuracy of their approximation finite width. While existing literature has established qualitative convergence of wide neural networks in the NTK regime to Gaussian processes at positive training time Lee et al. [2020], rigorous quantitative results - providing explicit finite-width error bounds - remain scarce. This gap limits the applicability of NTK theory to realistic settings where network width is large but finite.
Indeed, quantitative convergence guarantees are crucial to bridge theory and practice. They allow one to bound the discrepancy between the predictions of a finite-width network and its infinite-width NTK counterpart, thus enabling quantitative uncertainty quantification estimates and the safe deployment of theoretical insights to real-world architectures. Moreover, such estimates reveal how network width, depth, initialization, and training hyperparameters impact the validity of linear approximations during training. These insights are essential for developing principled training strategies and for diagnosing when the NTK regime offers reliable approximation, or when nonlinear effects beyond NTK must be taken into account.
1.1
Our contributions
This paper provides rigorous quantitative estimates for the convergence of trained shallow neural networks towards their Gaussian process counterparts, measured in terms of quadratic Wasserstein distances. Specifically, we extend previously established convergence bounds at initialization obtained by Basteri and Trevisan [2024], Favaro et al. [2025] and Trevisan [2023] to arbitrary positive training times. Our results deliver explicit convergence rates that decay polynomially with network width, clearly delineating how the approximation error evolves during training.
Concretely, demonstrate that the distance of the distribution of a shallow neural network's output trained via gradient descent to its Gaussian process approximation at any training time / >>> 0 satisfies explicit quantitative bounds dependent on network width. Our main theorem (3.4) shows that for any test point x under mild assumptions on the hidden layer width n and on the regularity of the activation function we have:
log n2
(((f)) =
y1
We also address long-term training dynamics explicitly, characterizing convergence rates training time diverges. Indeed, the above result continues to hold on timescales t growing polynomially the network width n1 as discussed in Remark 3.5.
These results significantly refine prior qualitative statements, providing actionable quantitative guidelines how network parameters and training duration determine the extent to which finite networks emulate their infinite-width limits.
1.2 Related work
The convergence of randomly initialized neural networks to Gaussian processes in the infinite-width limit has been foundational result in the theory of neural networks. This phenomenon was first suggested by Neal [1996] and later formalized for deep architectures by de G. Matthews et al. [2018] 8). The key insight that this correspondence extends beyond initialization was introduced by Jacot et al. [2018], who demonstrated that training dynamics in the infinite-width limit are governed by the so-called Neural Tangent Kernel (NTK), a deterministie kernel that linearizes the training trajectory. This sparked significant interest in the use of kernel methods to analyze deep learning models.
Following these developments, several works studied the convergence of finite-width neural networks to their limiting Gaussian processes. In particular, Lee et al. [2020] established that gradient descent dynamics the NTK regime converge to those of a linearized model. More recently, the works of Basteri and Trevisan [2024] and Trevisan [2023] provided quantitative convergence rates at initialization, measured in Wasserstein distance, which laid the groundwork for a more refined understanding of the finite-width behavior of neural networks. Moreover, in the also recent work by Favaro et al. [2025], additional quantitative results were obtained for total variation and convex distances. Complementary to these works, Bordino et al. [2025] used second-order Poincaré inequalities to derive QCLT's for Gaussian neural networks, obtaining a general but suboptimal convergence rate compared to the optimal n

---

However, these results were largely confined to the initialization regime, To this day, extensions to the full training trajectory remained limited, with few works addressing how approximation errors evolve over time or depend on architectural features such as width and depth. The present work builds on this gap by extending the quantitative convergence discussed above to trained networks, providing explicit bounds on the Wasserstein distance between the network output and the associated Gaussian process for any positive training time.
From a spectral perspective, the NTK's conditioning plays a central role in understanding convergence rates and generalization. Lower bounds on the smallest eigenvalue of the empirical NTK have been derived under various conditions. For instance, Karhadkar et al. [2024] and Bombari et al. [2022] provide sharp bounds in the context of ReLU and smooth activation functions, respectively.
Additionaly, Carvalho et al. [2025] showed that under very mild assumptions on the non-linearity and non-proportionality of the training data, the analytic NTK is not degenerate. These results are essential for establishing the stability of the gradient flow and, hence, for deriving quantitative convergence guarantees.
Our results are closely related to the work of de G. Matthews et al. [2018], who proved weak convergence of fully-connected BNNs at initialization to Gaussian process under the metric fff.f") = Eien 2 min(1, (a.) - f' (a;))). defined on a countable input set. In our setting, the imput set is finite; considering the restriction pF. it follows that convergence in W2 implies convergence in pF. de G. Matthews et al. [2018] also analyzed convergence under the maximum mean discrepancy (MMD). While MMD is not generally controlled by Wasserstein distances, connections have been established via regularized OT divergences (Feydy et al. [2019). Nietert et al. [2021]). Moreover, Vayer and Gribonval [2023] identified conditions the RKHS kernel ky under which MMD S W2. Consequently, our bounds also imply MMD convergence under these conditions. The metric pp which offers notion of pointwise convergence and is oblivious of the tails of the distributions, which helps stablish the results in de G. Matthews et al. [2018]. On the other hand, W2 captures the geometric structure and scaling of the output space. Finally, while de G. Matthews et al. [2018] address the more general setting deep networks, our analysis focuses on the shallow case, yielding new quantitative rates which improve previous ones in our setting.
foundational stream of research has shown that, under sufficient overparameterization, gradient- based training of neural networks converges to a global minimum. Seminal results by Du al. [2019] and Arora et al. [2019] established that for wide two-layer networks with ReLUU activation, the empirical NTK remains well-conditioned, enabling convergence via kernel regression. Subsequent advances generalized these results to deep architectures in different directions, such as Allen-Zhu al. [2019], Zou and Gu [2019], Sankararaman et al. [2020], Wu et al. [2019], Wei al. [2019], Zou et al. [2020]. which provide guarantees that hold with high probability over parameter initalization. These works reinforce that in the NTK regime, the network trajectory stays close its linearization around initialization. Our contributions align with this body of work and further extend this literature by providing novel finite-sample quantitative bounds on the Wasserstein-2 distance between neural network outputs and their Gaussian process approximations.
1.3 Structure of the paper
Section 2 introduces our notation and mathematical preliminaries. In Section 3, we present our primary theoretical contributions, including our main quantitative convergence theorem. The key technical proofs and intermediate results are outlined succinctly, referring the reader to the relevant lemmas in the Supplementary Material. Numerical experiments validating our theoretical predictions appear in Section 4. Section 5 discusses implications and future research directions.
2 Notation
In the following, given a matrix A E RPx4 we will denote by ||A||| its Frobenius norm and by ||AA]] lop
its operator norm. A, E RRy will denote the i-th row of and A_ E IR will denote the j-th column of A, for 1 i < y and 1 < j S q. The symbol denotes the usual matrix product. amin (A), 7 (A) are the smallest and largest singular value of A; and if p = g, Ammn( (A) and Amax (A) denote the smallest and largest cigenvalue of A, respectively. For any vector-valus valued function f. f()), denotes
3

---

the u-th coordinate of fz). For any Polish metric space X, P(X) will denote the space of Borel probability measures on X.
2.1 Shallow neural networks and associated Gaussian process
We consider a fully connected, shallow (i.e. single hidden layer) neural network of width nı and input dimension ni. We assume the output dimension ni to be equal to for simplicity. The output of the neural network as a function of its parameters is given by:
1 -110)=1000 ( TT
f(z; 0)
20 TTD
g(L) R
where ((0) E Mn, and ((1) E R denote the inner and outer (respectively) weights or parameters, (2) is the activation function, which acts entrywise on its input, and x E 1 from now on denotes a test input. Note that our model implicitly covers neural networks with biases 6(0) E R" (2, E R, by substituting the input x with (æ;, 1), using the parameters Ō(O) = (o(0) ((0) E R) ((1) (0(1) B(V) E R and using the activation function ((2) ($(=), 1). We
will denote by N ngnh n1 the total dimension of the parameters. For any ordered set of inputs
X (Zi,..., =aa E R xd we will use the notation f(X;0) = ()) fz 0)) R xd.
In what follows, parameters ( , for < no and J S n1, are drawn independent and standard Gaussian random variables at initialization.
identically distributed (i.i.d.) 1j
We will denote by h2 the preactivation of i-th hidden neuron, for 1 isss ny:
ha(1;; 0)
x0(0)) ) L.R.
no
Now we introduce the Gaussian approximation of the neural network f as the centered Gaussian process associated to the covariance operator XC given by:
x(x,x') 1 -xTxr x',
n10 K(x,x')
(Q)))()))
with
k(xx,x)) ((xl,
T(x,2')=
Explicit convergence rates for the Gaussian approximation at initialization can be found in Basteri and Trevisan [2024], Favaro et al. [2025].
2.2 Training, empirical NTK and limiting kernel
Let D (Z,, 9:) } C "o R be given dataset. Denote by X' (x (2 Znn) X the vector of training inputs, and by (. Bn) ) E R." the vector of outputs. From now on, we will consider the parameters 0 0, evolve training time. Consider the empirical risk for the mean squared error loss:
Rolf
2 (f(x:0) - (f(x;)))-)
Continuous time gradient descent with respect to this objective function yields the following dynamics for the parameters and the network:
का 0: VoRD[f.0,1 - Vaf(x;0)((((;00) -w),
a
0t f(X; 8) Vof(X;;0) (v. -v),
(2.1)
0
0t f 0) vef(x;00) Vof((: -y).
In Lee et al. [2020] the authors showed that the dynamics in (2.1) converge to those of a linearized network, which we now introduce.

---

Definition 2.1. Given neural network f we define its associated linearized network:
fl (æ; 0.) f(x; 0)) + Vof a
with the change of parameters w/ = 0r 0n- the following, we will also consider the linearized gradient flow, given by
0 A: Vof(X( (f (2: δt
The linearized network is known to approximate arbitrarily well the real training dynamics in the wide limit under some stability conditions and positive-definitness of the NTK (see Theorem 5.1 in Bartlett et al. [2021] and Theorem 2.2 in Chizat et al. [2019]). In the Supplementary Material we prove an analogue result (Proposition B.9) adapted to our setting, In particular, our result applies to shallow networks with inner and outer weights and for the MSE loss without scaling over the number of training points, as opposed the cited results.
An alternative formulation of this asymptotic linearzation phenomenon is provided in Jacot et al. [2018], where the neural tangent kernel (NTK) was introduced. The authors showed that under Gaussian initialization, the dynamics (2. (2.1) are governed by this operator which we now define in our setting. Define the empirical kernel, or NTK k: Ro Ruo x EN + R
k(,,,²;; 0) = Vof V (
where 0 R& denotes the matrix of parameters. The empirical kernel at the hidden layer can also be defined as a function of IR x I8 x RN
+ R xni;
Rew (2r, x': :':0) 0) V pron (2;; 0) V h(z;;
1 u, n1
During training, the parameter Og will be omitted when it is clear from the context and we will simply write 2²) = ke(az, x1; 0z) and (z, x²) (,, 0')
The convergence of the NTK to this limiting kernel was first proven by the authors of Jacot al. [2018]. Define the analytical, or limiting kernel k as follows:
kix (x, x') ) = x(x,x')
+ M(x x
with Txx,²)= x(x,x) K(x,x²))
k(x'2z) (((x,x²)) From now on, let l hi (XY, denote the limiting
kernel valued on the training set.
Note that, in the linearized regime, Equation (2.1) can be solved analytically. In effect, letting kxx (XX;; 0o) V 0o) and kzxxx = Vas Vo 0o)T, the gradient flow equations (2.1)
for plin can be rewritten as:
0, vof(X;00) (( (x v),
Jin (x; 0.) - -xxx (in (X: 0.) -y),
(2.2)
Ot
pln (x; 0,) = -kxx (X'; 0,) - v)-
Ot
The inverse of matrix kxx, which is random in the initialization of the network and may not be positive definite for some 0, appears in the solution of Equation (2.2). Thus we introduce the following auxiliary operator:
Definition 2.2. For any symmetric, invertible matrix B E Raxx and for any & > 0, define the x n real matrix
L(B) (1, - B
Note that (B) is invertible and symmetric since the matrix exponential of B is positive definite. The operator I can be extended to general symmetric matrices in the following way: define, for each E IR.
[()a) = f as ds
if a + 0, if a 0.
5

---

Let B E x be symmetric, not necesarily non non-degenerate. Consider the eigenvalue decom- position of B, UDUT with D diag(()..... On) and U orthogonal, then put (B (o)g((())).....((aa))))
Lemma A. shows some properties and well-posedness of the operator 7, defined in Definition
2.2. The matrix (B) can be thought of as the analytic continuation of the matrix function (1, BBBB for any B E _xxx symmetric. With this definition, the solution to (2.2) reads:
("((0; 0.) = (xp(-kxxx)]((x;00) + (In - expp (oo(-xxx()) f (x; (x;0.) 0.) = [xx 0o) ka 1 (x(kkxx
(2.3)
v).
(2.4)
The computation leading to this expression is contained in Supplementary Material A.
In Lee et al. [2020]. the authors showed that when f is linear in its parameters, its output distribution test point R converges weakly, as n1 diverges, to Gaussian process G, with mean and covariance given by:
(e(x) k (2r (2)
Ex(,,x²))= = K(rr,
- x')
K(x,, )io )io (X, kox (2, (
(2.5)
+ h (z, (1x X)I, (o (X),
for every positive training time t. For the sake of completeness we included a proof for the above formula in Supplementary Material A.1.
The solution of (2.2) completely characterizes the dynamics of the Gaussian process G, for any time t t> 0, as consequence of the Central Limit Theorem. In the rest of the paper we will assume that kno (X' X) is positive definite. This is a mild assumption and holds in a very general setting. Indeed, the authors of Carvalho et al. [2025] showed that when the training data is in general position in 1 and & is not a polynomial the smallest eigenvalue of koc (X, X) is strictly greater than zero.
3 Assumptions and main result
In this section we state assumptions and main result.
Assumption 1. The parameters ( o(') for < nn and و S n1, are drawn independent and identically distributed (i.i.d.) from standard Gaussian random variables at initialization. Assumption 2. The limiting kernel koo ()) is positive definite. Assumption 3. & and e' are Lipschitz continuous and bounded. Assumption 4. For some fixed 2 5 the following inequality holds:
||| (1. [!ull)
(10081>< llo + Lip + L /r log n1
min!
Remark 3.1. Note that these are rather mild assumptions. Assumption 2 is mild and holds in very general setting. Indeed, the authors of Carvalho et al. [2025] showed that when the training data is in general position in _0 and & is not polynomial the smallest eigenvalue of (X,, XX is strictly greater than zero. Assumption is standard in literature and is satisfied by most activation functions, such the sigmoid function and other logistic activations, hyperbolic tangent, Gaussian activation or sinusoid, among others. The ReLu family is a notable exception, although we expect our result to also hold in that case. As for Assumption 4, notice that the left hand side tends zero min(n), na} grows. This implies that our hypothesis is satisfied for sufficiently large nin or ni. In particular, it holds for sufficiently overparametrized networks.
Remark 3.2. Assumption 4 may appear somewhat artificial, so we provide an informal intuition on its use. This assumption is needed to control the fluctuations of l[kg kiooll with the (deterministic) smallest eigenvalue of the limiting kernel. This enables the use of Proposition B.9, which provides a quenched estimation of the L2 distance between f and flin, which in turn plays central role in the proof of Theorem 3.4. In particular, the 1,2 norm of this difference is controlled with function of the Lipschitz constant of the Jacobian Voʃ at initialization, and this Lipschitz constant is estimated with an expression in terms of |A - kooll bounded by the left-hand side in Assumption 4. This is the content of Lemmas 15 and B.

---

To measure how well the Gaussian process G, approximates the network f at time t, we will use well--kownn family of metries between probability distributions, the Wasserstein distances:
Definition 3.3. Let p [1, oo] and let v, be two probability measures defined on Polish space dx) with finite p-moment. The p--aaserstein distance between μ and v is given by
inf W)(,u)== TEM(,L)).x'H (E, [dM (X,,)))))
where [(p, v) denotes the set of joint probability measures y defined on M x M with marginal laws and v. With slight abuse of notation, we will often write (X, Y) = W, 2) for any X f and Y v.
Now we can state our main theorem:
Theorem 3.4. Under Assumptions 1, 2, 3 and 4, for each test point x Inn there exist positive
constants 1 and not depending no, nor I such that:
(1)(( r
a1 logn1 (1,00 nn min
ag7o
(1 + +t+))
(A n
Here r is the constant appearing Assumption
Remark 3.5. Note that our result is not limited to fixed training time t. but holds for values of growing polynomially on Indeed, provided that t grows at most polynomially in n2, the constant r can chosen arbitrarily big making the term dependent on time negligible. particular, as long as t grows polynomially on n1. sufficiently big r can be chosen so that the right-hand side in Theorem 3.4 tends to zero as ny diverges.
The term 8 in the right hand side of the inequality is due to Lemma B.12 in Supplementary Material
B. Lemma 1.12 provides upper bounds of the entries o!0) and o(") that account for perturbations that occur on tail events with respect to the initalization distribution (i.e. in the "bad event" SC). A finer control is possible if one is interested in result that holds S only, which has high probability, such as the ones in Bartlett et al. [2021], Chizat et al. [2019]. This finer control corresponds to Theorem B.9.
3.1 Sketch of proof of Theorem 3.4
The proof of main theorem is follows: we bound by triangle inequality
()). & 8),, (z; (";; + W2(f (x; G(()),
and proceed to control the two terms appearing in the right-hand side separately.
To bound the first summand, we partition N into a "good" event S C R.V. in which the assumptions of Proposition B.9 hold, along some other concentration properties of the parameters, and a "bad" event SC in which they do not; so the estimation of the first summand reduces the estimation of integrals over S and S respectively. Moreover, as m diverges, P(S) converges to 1. Proposition B.9 consists on an upper bound of [x; 02) flin 0.) on this "good event"; which is version of Theorem 5.1 in Bartlett et al. [2021] adapted to our setting. This allows to bound the first integral. In 5C the strategy is to show that the measure of SC decreases faster than how the upper bounds in Theorem B. 10 grow. Theorem 1.10 provides estimations for [I; 0z) {ii 0²) that are rougher than the ones in Proposition B.9 in the sense that do not vanish in the wide limit, but hold in SC. We estimate the second integral by partitioning 5C into countably many discs parametrized by y E N, and summing over 7 while exploiting concentration inequalities that hold on each disc. The result of this estimation is summarized in the following Theorem:
Proposition 3.6. On the hypothesis of Theorem 3.4, there exist positive constants and not depending on no, n1 nort such that:
(2( f (x; (x:8.) B.) sr ( 01 log n1
a27g
(1 + ++0)
+
(1000 Ynnno
(A min
'm
7

---

The dependence on time of the right- hand side of statement of the Theorem comes from Lemma B.12. In particular, in the "bad" event S a lower bound of the smallest eigenvalue of the random matrix kn is not available, and hence by Definiton 2.2 the sharpest upper bound for [Tx)
right-hand
|| is l.
The second summand, instead, is estimated with the following result:
Proposition 3.7. Let flin be the linearization of f. and let a E IR be a test point. Then, under assumptions I and 3, there exist positive constants C, C, D not depending nu nor t such that:
1 = C(t 1)e miat
W B.)
11
1 = (C D(t + 1)e Anmnt
n2
The first statement in Proposition 3.7 is proven by bounding Sे plu (x;0,) (X) with Young's Grönwall and Hölder's inequalities to inequality and gradient flow equations. Next, L.2 and L norms of the differences between decompose the problem in some expected values kernels and between f and G. both at initialization. Lastly, the main result in Basteri and Trevisan [2024] and Proposition B.4 providing L estimations for the difference between the empirical NTK and the limiting kernel complete the proof. The proof for the second statement in Proposition 3.7 uses the bound for training points in a triangular system of differential inqualities, inherited by the solution of Equation (2.2).
Theorem 3.4 and Propositions 3.6 and 3.7 are proven in full detail the Supplementary Material C and D.
4 Numerical Experiments
We conduct some numerical experiments in Figure to support our theoretical results. In both experiments, t is taken as the product of the learning rate and the number of iterations or epochs. For simplicity, we considered training and test inputs on the real line. The training set and test set were drawn from uniform distribution on fixed interval, and the labels y of the training points correspond to a sine function with additive noise. The code is available https: //gi thub. com/ emosig/ pooosig/qantt:: i seattive_gaussaeee trainedNN.
4.1 Experiment 1: Gaussian approximation of
The leftmost and center plots in Figure 1 represent 100 trained shallow neural networks with sigmoid activation of width n 700 on the leftmost plot and n1 1000 in the central plot. The networks have been trained for 2 10 epochs with learning rates of 700 leftmost plot and
1 and 1000 (hence == 28. 571 on the
140 the central one) fit two training points, corresponding to different random seeds on each case. Together with the networks, the plot depicts the mean (in black) of Ge and 95% confidence interval (in grey) over 200 equally spaced test points on the interval [-10, 10]. The networks were programmed with PyTorch 2.6.0 Paszke et al. [2019], and the Gaussian process G, was constructed using the library neural tangents 0.6.5 Novak et [2020] for the kernels K. and koc. needed to construct μe and Σ, in (2.5). The operator (B) was programmed by solving the linear system equations BX 1, e Bt with the linalg package of NumPy Harris et al. [2020].
4.2 Experiment 2: (2(f(<;; 0,), G.())) decays with n, for any x
In our second experiment (Figure right), we compute the quadratic Wasserstein distance between the trained shallow network and G, for variety of widths, ranging from 2 to 256. To do this we drew 10* samples of GG, given by the mean 14 and covariance Σ, in (2.5), which were computed with the neural tangents 0.6.5 library Novak et al. [2020]. Then, we trained indepently over a single training point, for 100 epochs and with a learning rate of 0.1 (hence = 10), 10 neural networks for each width and then calculated the empirical Wasserstein distance with the Python Optimal Transport 0.9.5 library Flamary et al. [2021]. As in Experiment 1, PyTorch was used to construct the neural networks, and the activation chosen for ff and G is once again the sigmoid.
8

---

hit GP win Power Law Power Fi:
Neural Predictions
Neurel Network Predictions
0.4
0.0
0.50
-0.2
0.00
0.25
-0.4
200
-0.25
-0.6
-0.8
Input
Input
101
The Gaussian process approximates the neural networks during training (left and center images), and it converges in 2- Wasserstein space to ft (right image). On the rightmost image, the blue points represent the empirical Wasserstein distance between f and G' for increasing widths, and the red plot is the power-law fit between the blue points.
Figure 1:
Discussion on the choice of n1 and the number of samples The authors of Fournier and Guillin [2015] provide an estimation of the error between probability measure l and its empirical counterpart An x Σ i= 6x,, given i..d. sequence (X,), X; μ. More preciscly, Theorem 1 in Fournier and Guillin [2015], for p = 2 and no = 1, provided that μ has finite third moment, reads:
EI AN)] = VN
for some constant C1. This estimation applied to f and G can be combined with our main result to find minimal ratio of samples per width needed for our numerical estimations to be noise-free. There exist constants 02, Cg not depending on nı, N nor such that:
C3 log n1
W O,), Gt(z)) Re (f(;(;;)) N.) G.))))))))))))
VN
T1
Hence our computations make sense when bound for the widths for which our experiments make sense is, approximately, 650.
For N = 10 this means an upper
log
5 Discussion
In this paper we provided quantitative convergence rates in 2- Wasserstein distance between trained shallow neural networks with standard Gaussian initialization and an appropriate Gaussian process, for any positive training time. This was proven for Lipschitz, bounded activations with bounded derivative and for sufficiently large hidden layer width. We now address some limitations of our work and possible future research directions:
1. Our main result is not uniform in time. Although the dependence on time can be minimized at the price of including a sufficiently big multiplicative constant in the right-hand side of our inequality the limit when / tends to infinity exponentially on ni is not available.
discussed in Remark 3.5, a general result holding uniformly in & >> 0, in
This dependence on time could be related to the transition from the NTK regime to a
feature-learning regime, suggested by the work of Huang and Yau [2020]. Their analysis, however, does not address the tails of the distributions, which in our proof correspond to the set 5C and responsible for the t scaling. Moreover, Yang and Hu [2021] show that under standard and NTK parameterizations, wide networks cannot perform feature learning in the infinite-width limit.
This suggests that our observed & scaling might reflect the boundary of the NTK regime: in the "bad event" SC or for sufficiently large times, the training dynamics may drift into feature-learning, where purely kernel-based control breaks down. Our main result remains consistent with works such as Bartlett et al. [2021]. Chizat et al. [2019], which hold with high probability, whereas our analysis explicitly incorporates the contribution of the event Sº. At present, it is unclear whether the t scaling is sharp.
We would like to address this problem in future work.

---

2. The bound in our Theorem 3.4 depends on the test point x. This dependence is explicitly stated on the proof of the auxiliary results Proposition 3.7 and Theorem B.10 in the Supple- mentary Material. Locally uniform bounds on the test point might follow from functional inequalities such as the ones found by Favaro et al. [2025] if extended to the NTK regime.
3. We conjecture that our main result remains valid even without Assumption 3. suggested by our numerical experiments with the ReLUU activation. In this work, we deliberately focused on specialized setting with mild hypotheses to obtain novel and technically precise result while maintaining a clear exposition. Future research will aim to relax the regularity assumptions the activation and extend our analysis to more general setting,
4. When Assumption I and 2 hold, A: (X', X²) is strictly positive definite and & is Lipschitz, the rate of convergence at initialization found in Trevisan [2023] for the squared 2- Wasserstein distance is of n This fact suggests that, in the proof of our main Theorem, a better estimation of W2[[(x;;), (a=; can be found; either by improving Proposition 3.6 or by improving the estimation of the Lipschitz constant and the norm the Jacobian (af Bo) in the "good event" of the proof of Theorem 3.4, possibly by choosing more restrictive "good event" that still makes the infinite sums in the proof of Theorem 3.4 converge. This last possibility calls for finer concentration inequalities for the parameters and the difference between the empirical NTK and its Shinite-widhh limit.
5. Our results could be extended to deep, fully connected neural networks, as done for ini- tialization in Trevisan [2023], Basteri and Trevisan [2024] and Favaro et al. [2025]. We hypothesize that Proposition 3.7 can be easily adapted to the deep setting exploiting recursive characterizations of h; and kise as the ones available in Jacot et al. [2018), Nguyen et al. [2021] or Lee et al. [2020]. The other half of our proof, though, relies in Proposition 3.6, which would need a new proof for the deep setting.
6. Another desirable step could be to study how well our result extends to other architectures, such as convolutional neural networks or the more modern attention- based architectures. This approach is present in Yang and Littwin [2021] but to the best of our knowledge no quantitative results in this direction are available.
Acknowledgements
E.M.G. and D.T. are members of GNAMPA group of the Istituto Nazionale di Alta Matematica (INdAM). All authors acknowledge the support of GNAMPA Project CUP RECC22001930000 E.M.G also gratefully acknowledges the hospitality and support of the Institute of Mathematical Statistics and Actuarial Sciences at University of Bern during part of this work. This research was supported by Ph.D. fellowship funded by the Italian Ministry of University and Research (MUR) under the PNRR program "Transizioni digitali e ambientali (TDA)" (D.M. n. 118/2023, CUP LU11333300040000007 within the project "Limiti di scala di dinamiche stocastiche."
D.T. acknoweddes the MUR Excellence Department Project awarded to the Department of Math- ematics. University of Pisa. CUP 157G22000700001, the HPC Italian National Centre for HPC, Big Data and Quantum Computing Proposal code CN1 CN00000013, CUP 153C22000690001, the PRIN 2022 Italian grant 2022WHZ5XF "understanding the LEarning process of QUantum Neural networks (LeQun)": CUP J53D23003890000 the project G24-202 "Variational methods for geometric and optimal matching problems" funded by Università Italo Francese. Research also partly funded by PNRR- M4C2 Investimento 1.3, Partenariato Esteso PE00000013 "FAIR Future Artificial Intelligence Research" Spoke 1 "Human--cnteree" AI", funded by the European Commission under the NextGeneration EU programme. This research benefitted from the support of the FMJH Program Gaspard Monge for optimization and operations research and their interactions with data seience.
References
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterizzaion.. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, page 242-252. PMLR, 2019. URL https: httpss///oooeodeeeeee mlr. press/v97/ / aallllnlelnllllllllllleeaae chu19a.1e html.
10

---

Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-graing analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 322-332. PMLR, 09-15 Jun 2019. https:/ / htt..tttss/pprrceeddaaeee proceedings.mi. mlr. press/v97 /arora html.
Peter L. Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. Acta Numerica, 30:87-201, 2021. doi: 10. 0011007750062449221000007,
Andrea Basteri and Dario Trevisan. Quantitative gaussian approximation of randomly initialized deep neural networks. Machine Learning, 1 113:1-21, 06 2024. doi: 1..1007//10994-02--0663
Simone Bombari, Mohammad Hossein Amani, and Marco Mondelli. Memorization and optimization in deep neural networks with minimum over-parameterization.. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https: //tp///epeeeeeeee/////ee/eeeeeee .nt seee:ffoourrr1dx281111113
Alberto Bordino, Stefano Favaro, and Sandra Fortini. Non-asymptotic approximations of gaussian neural networks via second-order poincaré inequalities, 2025. URL $304.04010
/raaa///iiorrra/////// /abs/
Luís Carvalho, João L. Costa, José Mourão, and Gonçalo Oliveira. The positivity of the neural tangent kernel. SIAM Journal on Mathematics of Data Science, 7(2):495-515, 2025. doi: 10.1137/ 24M1659534. URL htttt::////oiit org/ 10. 1137 /24M1659534.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Curran Associates Inc., Red Hook, NY, USA, 2019.
Alexander G. de G. Matthews, Mark Rowland, Jiri Hron, Richard E. Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. ArXiv, 15//1804.1120 2018. URL
https: /tpp///app../// http://aap..seaatiieeeemmmm semanticscholla org. CCtrpoIppD:::: 13757156.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes wwe-pparameteriee neural networks, 2019.
S. Favaro, B. Hanin, D. Marinueci, Nourdin, and G. Peccati. Quantitative clts in deep neural networks. Probability Theory and Related Fields, 191(3):933-977, Apr 2025. ISSN 1432-2064. doi: 10.1007::00440--25--11666
1007 URL pattps://doot 10t.///0..110000000000000-0----0 0000000/000000000001000000 01360000000111111000
Jean Feydy, Thibault Séjourné, François- Xavier Vialard, Shun-ichi Amari, Alain Trouvé, and Gabriel Peyré. Interpolating between optimal transport and mmd using sinkhom divergences. In Kamalika Chaudhuri and Masashi Sugiyama, editors, Proceedings of the Twenty-Secone International Conference Artificial Intelligence and Statistics (AISTATS), volume 89 of Proceedings of Machine Learning Research, pages 2681-2690. Proceedings of Machine Learning Research, April 2019. URL https: / /pwoce/ddngc/...e..oo. mlr prees/v89/// h889fffyyyl9g/ html..
Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurā@lie Boisbunon, Stanislas Chambon, Lactitia Chapel, Adrien Corenflos. Kilian Fatras, Nemo Fournier, LÅ@o Gautheron, Nathalic T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78): 1-8, 2021. URL http: / jmlr org/ wwtrg//arrs////eee/ v22/2022222222/ 451 .tmll.
Nicolas Fournier and Arnaud Guillin. On the rate of convergence in wasserstein distance of the empir- ical measure. Probability Theory and Related Fields, 162(3):707-738, Aug 2015. ISSN 1432-2064. doi: 10.1007/4004400-044-5333
URL https: http::::////oooo////////o 000 10070000440001444401110 00004000000000000--0-00000000001
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http: //www. dwwpleermingbook. org.
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Feránndez
11

---

del Río, Mark Wiebe, Pearu Peterson, Pierre Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825)335773362 September 2020. doi: 10. 00.1038/441186-022--00 UtI.ttps: //doi. org/ 10. 038 // -20--6000-249999
Kevin Sheppard, Tyler Reddy,
Jiaoyang Huang and Horng-Tze Tzer Yau. Dynamics of deep neural networks and neural tangent hierarchy. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Leurning Research, pages 4542-4551 PMLR, 13-18 Jul 2020. URL https:://///. Utt.ttps::/ppooeeeeeeeee http///oooeed...c... www.s......eeeeeememeemmmm v1 1 ///huann220.... htmll.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and general- ization in neural networks. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Gamett, editors. Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL httptttt/prrrceeedinngge 1a18/ffle//aabbee http:///s...aaaaaaeeeeeeeaaeaaaa /
paapperfflle/pppprrrr .dff
Kedar Karhadkar, Michael Murray, and Guido Montufar. Bounds for the smallest eigenvalue of the NTK for arbitrary spherical data of arbitrary dimension. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https: http://pppeeeeeeeeeeeeee net/ net/forummid- mee
B. Laurent and P. Massart. Adaptive estimation of a quadratic functional by model selection. The Annals of Statistics, 28(5):1302 1338, 2000. doi: 00.2214/a0//0115553335 URL https:
//doi. //doi.org/// org/o..//// 10. 1214 a0ss1001957333
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553))436-444,
May 2015. ISSN 1476-4687. doi: 10.1038/nature 14539. URL https: /ttp:::///oo...o.....o1.. nature14539.
Jachoon Lee, Lechao Xiao, Samuel S Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent Journal of Statistical Mechanics: Theory and Experiment. 2020(12): 124002, December 2020. ISSN 1742-5468.
doi: 10. 10.1088/1722-55468866622 URL http:
//dx.doi. www.ddii.ogg.//// .rgg 10. 1088/ 1250/1122-AA0 //b6/b..
R. M. Neal. Bayesian Learning for Neural Networks, Vol. 118 of Lecture Notes in Statistics. Springer- Verlag, 1996.
Quynh Nguyen, Marco Mondelli, and Guido F. Montúfar. Tight bounds on the smallest eigenvalue of the neural tangent kernel for deep relu networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8119-8129 PMLR, 2021. URL htthttp://// htttp://proceeddings. mml.eediiinnii.i......mmmm.merdm www.mmrrr..e..e..mm..m..m wwtpp//eeee/eeeeeeeeeeee 113//////1/// html.
Sloan Nietert, Ziv Goldfeld, and Kengo Kato. Smooth p-wasserstein distance: Structure, empirical approximation, and statistical applications. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 172-8183. PMLR, 18 28-24.2 Jul 2021. URL htttpss:/proocediiiin mlr mlr.press/ http///11//////ee/eeeee www.eeeeeeetttttteeeeeeiit
Roman Novak, Lechao Xiao, Jiri Hron, Jaehoon Lee, Alexander A. Alemi, Jascha Sohl-Dickstein, and Samuel Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL https: //ap::://// thub. com/ google/neeurl-ae tangents.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http::/:::tppppe htwp:pppprrr..u.uuppppp cc/paper/ 2015-pytorch soyttorch-an-imereae aaa--iperativ--styleeee htwp://-hhhhhpefffmmaeeeeeeeeeee learning- pdf.
wwwiiingg-bbbry... library.
12

---

Karthik A. Sankararaman, Soham De, Zheng Xu, W. Ronny Huang, and Tom Goldstein. The impact of neural network overparamettrizzaioon on gradient confusion and stochastic gradient descent. Ma- rina Meila and Tong Zhang, editors, Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, page 8469-8479. PMLR, 2020. URL https: / http//poooe/nggee/eeeee mlr press// v1 19/ sankararaman20.. html.
Dario Trevisan. Wide deep neural networks with gaussian weights very close gaussian processes, 2023.
Titouan Vayer and RÁOmi Gribonval. Controlling wasserstein distances by kernel norms with application compressive statistical learning. Journal of Machine Learning Research, 24(149): 1-51, 2023. URL htt: http:///mmmoooo/prrraas/se/ hwtrg//pper//24/211 1516 html.
Cédric Villani. Optimal transport- Old and new, volume 338, pages xxii+973. 01 2008. doi: 10.1007/978- 3-58---050-------
Colin Wei, Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets v.s. their induced kernel. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https::/ neur ips. cc/ pdf.
Xiaoxia Wu, Simon S. Du, and Rachel Ward. Global convergence of adaptive gradient methods for over-parameterized neural network, 2019. URL htt.atggs//rrddd.rr/////////////
Greg Yang and Edward Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 11727-11737. PMLR, 18-24 Jul 2021. URL https: http:::://ppoccedings. mlr. press/ https//vvvv9ag///e/ htmml.....ml......
Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. In International conference on machine learning, pages 11762-11772. PMLR, 2021.
Difan Zou and Quanquan Gu. An improved analysis of training over-parameterizze deep neural networks. Curran Associates Inc., Red Hook, NY, USA, 2019.
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes prer-parameterized deep relu networks. Machine Learning, 109())467-492,, Mar 2020. ISSN 1573-0565. doi: 10.1007/110944411996533--
URL http:///do..00g11.1000000
910994-019-05839-6.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: Our main result Theorem 3.4 4 is precise statement the compressed version included in the introduction. Our abstract accurately reflects this result's content and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: Limitations of our work are presented together with possible future work directions in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes].
Justification: Every new result stated as Theorem, Proposition or Lemma and is accompanied by its proof in the supplementary material. An additional theorem environment was created to present and cross-reference the assumptions in our results.
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulae, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: In Section 4 we provide a detailed exposition of how our numerical experiments were programmed and the Python libraries used on them.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
15

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes].
Justification: The code used to produce the experiments Figure available at an anonymized repository.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips)) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht / /nips. wwtppcccccc//cccccccccccccccclct httegggiiiee/ooeeumbiiiiii /Coo/eeooiiiseeeeeeeeeeeee www.ssoliccccyil for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: detailed description of all the hyperparameters is included in Section 5. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes].
Justification: Our plot of the Gaussian process G2 in 1 includes 95% confidence interval. As for the plot involving the distance (2GG,, ft), a subsection discussing the approximation егror the computation of the empirical Wasserstein distance has been included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiments were run on the author's personal computer and do not require any particular hardware specifications.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes].
Justification:
Our research follows NeurIPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
Justification:
The innovations our paper introduces are of a theoretical nature.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
17

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: No sensitive data is used in our article. The data used in our experiments in Section 4 is artificially generated by our code and consists of points in R.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: The python libraries used in our experiments are adequately referenced in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi cooode..oo cww. com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: Our paper does not involve human subjects in any way.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA].
Justification:
Our paper does not involve human subjects in any way.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA].
Justification: Our results do not regard L.Mss.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tt////errrppe ipe.. for what should or should not be described.
/ttp://///aaaigg.cc//////o/eeete https/////o/oeeeeee///00000 /LLMMMM
20

---

A Gradient flow of the feature function
In this section we provide a closed analytical solution to (2.2) when f ftm.
Fix x E I a test point. For the sake of clearness, we will use the following notation for each 2 0: pسm (x; 0z), fim (x 0z), xxx (o and liz, = X)). Note that ke = ko for each when / flin
We begin by stating a lemma that shows that our definition of 1e consistent and commutes with the wide limit:
Lemma A.1. For any real symmetric matrix B E R we have (B (I, 1, e eB: and for real symmetric matrix sequence (Bo)nEN with B, + we have lim,->o le(B,) = (B)..
The proof of this result follows from properties of the matrix exponential and is left Supplementary Material E.
Consider the system of ODEs in (2.2) given by:
i
δi fthm -xxx(Se y).
(A.1)
Øt 11 - v).
(A.2)
Recall that, in general, the solution to the initial value problem f'(t) A(t)f(t), f(0) fo can
be written as f(t) = exp(S" A(s)ds)Joo fo. where f(t): + R.m.
A(t): R -> IR mm are integrable
functions, and t > 0. Therefore in our case, by letting us = ft y::
ur exp (-// kaxxds) (xxxdd)) un exp(-kxxx)uu
(A.3)
Moreover, by letting D. y and substituting the solution for us we obtain the following expression for vy:
1t K -xxp-xx - y). dp = yo.
which, by integrating and by using Definition 2.2, becomes
00 D. To
(A.4)
-kax L." exp (xp(-kxxx))de ds(fo 3)
(A.5)
= =okxxo(() v).
(A.6)
Note that formulae (A.3) and (A.6) agree with the formulae found by the authors of Lee et al. [2020].
When xxx is not degenerate, taking the limit when t tends to infinity, we get prediction for the output of the linearized network at the end of the training:
f (x) lim Je To kark y)- 1-
A.1 Proof of the characterization of G.
Here we prove the formulae in (2.5).
Define B, - - and C. kax (kxx), that U, -0- B.fo Cty. Also recall that k2 = Ky for 2 0 since of is linear on 6. Note that fo and To are centered Gaussian processes and hence E2yo + B,fol 0. Therefore, taking the expected value of the wide limit yields:
E lim D.l E lim C.vl k (x., X) X )y,
(A.7)
700
where too ko X)). This limit is well defined thanks to Lemma A.I.
21

---

Now let x' E R and put v/ = fi fz') and B; = ( for each t 2 0. Then,
Cov( lim Dr lim 3/) = Ej lim (U. Eu - E(yl)
(A.8)
+30 +00
110000
EJ lim (To + (36 + BiJo)l
(A.9)
m1-000
EJ lim ToVoll + B lim JoJoB
(A.10)
'n1+000
EI lim 36JoB,] El lim B.
(A.11)
ni 300
ni 000
= =K(axx) - K(x, 2 I (1oc (1, (X
(A.12)
- (k)x ((
(A.13)
(o, (koz) Kix, ((,) Te (A: ) (2t,
(A.14)
Again, Lemma A.1 ensures the limit exists.
B Auxiliary and related results
In this Supplementary Material we state intermediate results in the proof of our main theorem and recall some useful results. Throughout this section we will use the following notation for each 2 V/ f(; 0z). f f(X; 0r), he (X, X) and Foc = (X X). All the proofs are deferred to Supplementary Material E.
In the next lemma we collect some well-known properties of the Wasserstein distance:
Lemma B.1. Let p E [1, oo and let X, Y be random variables with values in R." and Z be a random variable with values in R Let Pg denote the law of the random variable for each & E (X,Y Z). Then
1. If X are defined on the same probability space, then W, s E YP
2. If Z is independent from X and Y then W,(X + Z,Y + Z) I W,(x
3. Convexity of Wp: (X, & Je.. Wp Py
4. Let X 1m be constant vector and consider the joint random variables X 2),
(Y, Then
( -W))Z)))))((XX)))) (X, (=_1==--IPeee
5. Let V be random variable with values in mm and consider the joint random variables
X (X, (Y. Then, for 2,
wg(X, Y) < 28
(M)
Moreover, for p WiX, () Wi(ZVV).
The following result provides explicit formulae for the components of the Jacobian of ʃ and the NTK: Lemma B.2 (Gradients and explicit formulae for k and k). The following hold for each v, x' E Ro;
V gios (xx
(0) (0(2) E "o
(B.1)
/ /nino
no
V for,
20)))
E E.
(B.2)
/n1
no
Moreover, k(xxx²)) is a diugonal x matrix with
1
Rys (z, x²)
(B.3)
n2
22

---

and k(æ, x²) is a real function given by:
1AB
n1 Σ (lh,(t)) (B.4)
7h1 70
B.1 Results at initialization
Throughout this subsection, we assume t and omit the subindex unless needed. Our results 3.7 and 3.4 aim to generalize Theorem 4.1 in Trevisan [2023] in different directions. For reference, we reproduce the main result from Trevisan [2023] here in simplified version:
Theorem B.3 (Trevisan). Then, for each p E there exists a constant cp not depending on the network width ny such that:
W,Ls(AA), Go(X)) S Cp Vmn
(B.5)
Furthermore, ip - R is Lipschitz,uuctiinn then
(Lipy + (0)))
Wp
cp
(B.6)
Theorem B.3 provides quantitative bound for the Gaussian approximation of the neural network ft- This can be upgraded to bound for the joint distribution of the empirical kernel and the output of the neural network.
From now to the end of this subsection assume & and o' are bounded and x, E Ro are fixed. Also, adopt the notation introduced in Supplementary Material A. We state helpful estimation: Proposition B.4 (TP bound for the kernel difference). Fix x' in To and let kn = kn (r,,)), k(x, x'), ) = K((r,, x²) and his (x, There exists a constant > 0 independent of n such that:
Blik11 KC1n] 0, Elik koo 1'1
(B.7)
C
(B.8)
ni F.
Remark B.5. Note that since koc is deterministic, we have (k lis)) = E[iAA - 1 constant C in B.4 depends the constants produced by applications of Theorem B.3.
IP]). The
The proof of Proposition B.4 goes by triangle inequality combined with Theorem B.3, and by exploiting the independence between the entries of ((1) and those of ((0), An auxiliary result to prove B.4 is the following:
each Proposition B.6 (TP bounds for the empirical kemne)), Let kij = kiz (xy(x, x'), for each 1 S i, S m. and l a')x, Then, the following inequalities hold:
[[[ 1 2
(B.9)
|| I < 2- (2p 1)!!
2p kn
2D 11012
(B.10)
Now we are ready to show the following result: Proposition B.7 (Joint distribution Masteri-Trevissee There exist positive constant C' such that:
C
Wp ((ko, fo), (Riso, Go)) S Vni
with C not depending on the width nı-
The proof by using Dudley's 's lemma (also referred to as the gluing lemma from optimal transport) as outlined in Villani [2008]. This lemma is used to decompose the Wasserstein distance into two terms. The summand regarding the network and its Gaussian approximation is bounded with Theorem B.3, and the other summand is bounded by Proposition B.4.
Lastly, the following lemma used in the proof of Proposition 3.7. Lemma B.S. The following inequalities hold:
Elll fo blll < 2 + ||vll.
(B.11)
Elll fo y/ $1 < 32m 32nn 1 + 11
(B.12)
23

---

B.2 Approximation by linearization af training time t >> 0
In this subsection we state two results paramount to prove Proposition 3.6, along with all their auxiliary lemmas. The following theorem resembles Theorem 5.4 in Bartlett et al. [2021], or Theorem 2.2 in Chizat al. [2019], but we would like to remark that those result differ from ours since the first applies to neural networks in which the training is restricted to ((0) while keeping ((1) frozen, and in both of them the loss function used for training is different than the one considered in our work.
The proof is similar to the one in Bartlett et al. [2021] and is carried in full detail in Supplementary Material
In this result we prove quenched estimations for the dynamics of the parameters, the linearization error both in the parameters and in the network valued on test points, and the convergence of the network to the labels over the training set.
Assumption 5. The smallest eigenvalue of ko is bounded from below by:
fL yll < >>min (Ko),
where (xx) the Lipschitz constant of Vafo seen as function of 0.
Theorem B.9. Let Amin = Amin (ko); min = min (ko) and max = max and let Assumption 5 hold. Then the following hold for and for any test point x E 1
2
110 Poll x
1150
y/1,
(B.13)
mnin
(8 + 200 2
max (x))
110- Dell 5
Il ull?
(B.14)
o min
Xamin (->aain
llfe y/1? < fl - y/12 exp
(-2
t
(B.15)
2
[f( (1) - (z) BJ)l < 4L(x)
11 ffll?
(B.16)
(8 + 20 max ((x)
115o 166--wli [Vlo\V)
(B.17)
Proposition B.9 relies on strong assumption involving the Lipschitz constant of the Jacobian of the network, the norm of the network at initialization and the positive-definitness of the limiting kernel. The following rougher estimations depend on i, but they do not require Assumption 5 and will be key to prove our main theorem.
Theorem B.10. Assume that & and op are bounded. Let T(X) be the Lipschitz constant of Jo seen as a function of O. and let (ff)) = [l - yll. Then, for each & >> there exist positive constants Ai....,As. Bi...., Bg, Cr..... Cs and Cg not depending on n1, no nor such that, PPalmmsst everywhere:
1lve - fell <
Ao
100 (2)
1
10 (2)))(0)) +
12 (1) 12
(00))
(B.18)
no
nino
Agt
Ast
Ast 1000 112
+
(00))
)6
110
(06))
(B.19)
n nino
7i700
n2n0
|| -1
Bo
10 (0)
(1)
12
B
10
00
1120(00))
(B.20)
nin
ning
Bg
(0)
121100
(0))
B3
112
(B.21)
nino
ning
B4
Bs
1006)1166000022
(B.22)
nino
nino
Bs |1002 12
B7 11000112(0000+)
(B.23)
nino
nino
+ nino
Bs
11000) (0)) (1)
1 1 (0)))
Bg
10 12)(00))))
(B.24)
ning
24

---

L(x)²
(100
C2))
C3r)(66))+
I1fe
fl v
(B.25)
nf
ning
ning
C4
1a 11
CoN(16))
"1 12,0 10
(B.26)
n
Bing
no
073b(00)2100g 1324 Con) + Cgrb(())24
(B.27)
n10
where g(0) (()) E "o denotes the usual product of the matrices ( and ((1)
Remark B.11. The dependence on time of the right- side of the above formulae comes from Lemma B.12. Indeed, by definition of the operator I,, the sharpest upper bound for the matrices (k,))) and (ko) when no lower bound for Amin (ke) and Xmin (Ro) is available is 1 n
The proof of the first two inequalities in this theorem is by exploiting an expression for the gradients of the network, contained in Lemma B.2; together with an integral result describing the behaviour of the parameters at time / with respect to the parameters at initialization. This the content Lemma B.12. The third inequality uses an integral argument together with the cemiposiiieeddefiitensss of k to redirect the problem to studying ||k koll- All the constants in Theorem B.10 are multiples of the norms and Lipschitz constants of & and Φ', and of the norms of x and X.
Now we state Lemma B. 12 along with some concentration inequalities and related auxiliary results.
Lemma B.12 (Inequalities for 0(²)). Fix u and v with S S and 1 S S nn and put N. (w R.. Let Ami and min be the smallest eigenvalues of k2 and ko, respectively.
Then the following inequalities hold:
|0 Il wlll
(0())) b - (0
J0 +
T (Nmin))
(B.28)
(0 S (0 )a + ( | 0o / b1.0ol 11ull
L (e(Anin))
(B.29)
2n1
+ 4 |1 oc. fo 3
3/1l1x..11
(o (Amin)))))
(B.30)
1
0 )r I (08))) )o + ( oo Il - y/ll
I, (2hin))
(B.31)
n2
0 10 & (0 10 + |
/ o| Il 11 I, (2 )2
(B.32)
2n /no
1 |lo fo
(8!)))o (2
(B.33)
Vnnno
Remark B.13. Recall from the definition of L that for t >> 0, [+(a)= > 0, even if a 0. Moreover, (0) t by definition. Hence (> S t and (2 >2 Stt
Recall the well known concentration inequality for disttrrib random variables (see, for example, Laurent and Massart [2000]). For each ? > O:
TP(110()) 2|2 2 2 + 2 + n1) S cxp(->).
(B.34)
The following is a concentration inequality for the sup-norm of 0(4); and as consequence we get an estimation of the norm and Lipschitz constant of the Jacobian of f at initialization.
Lemma B.14. For any 7 > 0:
|106" S / my log ni,
(B.35)
with probability bigger or equal than
This concentration inequality is proven using the fact that 110 is the supremum of n, Gaussian variables in absolute value.
25

---

Lemma B.15 (Norm and Lipschitz constant of the Jacobian at t 0). Fix 2 1. Then for each I
| l
| 1oo
[Vooo(>)]
(B.36)
Tho
V
[ 21 1100 + Lip
|
LLip
LipVofo(r)
rry log n.
(B.37)
Vinno
with probability greater equal than 1- exp(--m1), where fu(z): RN - R understood as a function of 0.
Now we state a concentration inequality controlling the norm of the difference between the NTK and its limit.
Lemma B.16. Let k ko X) and kox Koo (X, (x,,, X) and let y E N. Put Amin Amin and
Am = Amin (. Then, for each N,
7
||k: -ioooll
(B.38)
2
with probability greater or equal than
where a positive constant not depending 5
mi
on nl.
Remark B.17. The previous lemma provides useful bounds for the smallest and largest eigenvalues of the empirical kernel at initialization, with arbitrarily high probability when the width diverges. Recall that the smallest eigenvalue of a matrix a I-Lipschitz function of the operator norm, which is bounded by the Frobenius norm:
Amin (A) Amio Ai))(B)) ||A- Bloop I 11A Bl.
(B.39)
In particular, the previous Lemma implies, for 7 = 1:
00
Amin
A k
10
min
(B.40)
Conversely, an upper bound for the largest eigenvalue of matrix using the operator norm is given by:
Amax ( (A) S Amax (B) + ||A - Bllop I Amux (B) + || - Bl|.
(B.41)
Again, taking in the previous lemma yields:
Amax
Max
min
(B.42)
2
Both inequalities hold with probability greater or equal than
mis
n
C Proof of Theorems 3.4 and 3.6
Here we prove Theorems 3.4 and 3.6, which share some auxiliary lemmas. Throughout this and the remaining appendices will use the following notation for each i. ' 0: 1/2 fe(x), fa f((X),
V. flin 0.), flin (X";
(X; 0.) ki ke(X,)) X) and k kis X'). The gradient and the expectation E will always be taken with respect to the parameters 0, unless otherwise indicated.
Let us begin by Proposition 3.6:
Proof of Proposition 3.6. Fix p, r N. Consider the following subset of I:
| n1
1
S [0]
< 5, || I /r log n1, 1/ke kooll <
}.
2
26

---

By the inequality (B.34) and Lemmas B.14 and B.16, the probability of S is bounded from below by 1 - exp(-n11) Vn (Amiva jas for a positive constant not depending on n4, no nor Then,
(3 ye) inf Boll Ve y=11
0 1/8 11ye - Ve vell2d0 / /sc 11yr Dell2d0
(C.1)
(C.2)
è
sup #E8
11
Bell exp(-ni)
11 aesc
sup
D.ll?
V /np-
(A
/n
(C.3)
Let us begin by estimating the supremum over S. Note that by Lemma B.16, Amin The hypothesis for Proposition B.9 are satisfied when ni, no are large enough. In particular, thanks to Lemma B. 15, Assumption 4
> 0 in S.
||| (v5) + |ox + lIvll)
X Lip vr log
||
+ Lip +
V 7o
TTo
min
implies Assumption for any in Indeed, by Lemmas B.2 and B.15:
1l - yll
(C.4)
4||
||01)|22 + Lip + | m1
(v5ll 0 + |bll
(C.5)
Vnino
/no
in S. These two inequalities together show that Assumption 4, which holds for sufficiently big n, is sufficient condition for Proposition B.9 to hold.
Moreover, Lemma B.16 implies Amin 2 Ame
2
On the other hand, by Lemmas B.16 and B.15, the following inequalities are satisfied in S, for E {3.
||
|
vr log n1
L(Z)
||0'
+
Lip
(C.6)
710
||||||||
1101
Vofoll $
(C.7)
Vni
A
Amax
00
(C.8)
By substituting the estimations of L(x), ((X), V fz;; 00), Amax and Amin above in Proposition B.9, for each E S:
8(V5|12/00 + 11y11)2
X
li Dell z
|| +
(V2 + m
(C.9)
min /ninoo
min
/r log nn
|
| 10o + Lip
(C.10)
n2o
no
v /ni
r log n
$ scV
(C.11)
n1 (Am )31
where the constant e independent of t, no and nı and can be determined explicitly: = 3844 (v5| +| max [|=|||, V2, 15 mxx | Lipo, ||
|1 @|
|||||||| |/oo,
| [[xc]
Hence,
1 11 - - PPS) aES
sup 11 F.ll
(C.12)
< sup PES
11: D.11
(C.13)
2 log n1
< 00 3 n00
(C.14)
27

---

Put 01 = c².
Now it only remains to estimate the second summand in (C.2). For each y Net 9 27 v2 +1 and define the subsets:
1100011
2 =
5, 1100 > Vry n1
(C.15)
12
Also, define the subset
A min
S2 10 || 1oo
(C.16)
2
and let Ω Q2 UsEN
Intuitively, S2, and UyEN Shy are the events in which the lower bound for the smallest eigenvalue of the empirical kernel and the upper bound of the Frobenius and sup-norm of the parameters
initialization, respectively, do not hold. Notice that RN = S L S L UyEN S2. We will use this partition of R&N finish the proof.
By Lemma 1.14 and by (B.34) we have P(22)) S exp(->m)
and P(2) S P(O,) S (n JF Moreover, the family (Sy) is descending filtration of S 12. Let D, = 22 \0 10 for each y N. This allows write:
live - Fel1-d0 do + Σ /
live - d. - 12 80
11 3.11a
(C.17)
TEN D.
<
TP(2) + EP(D))
sup DED
11ye - yell
(C.18)
7EN
S 3P() sup (1ly-- - fell + 11ʃ 1! + 11.! Wz 12) AEN
(C.19)
+3
Σ P(D₂)
sup (llye - f&l12 + IIfe 1l 115 J. v2ll BED,
(C.20)
7EN
3 3
exp( (-m)) sup I1ve - Sxll + Σ
exp(-
pm
sup I1ʃe
DeD,
0E D
7EN
7EN
(C.21)
+3
Σ
exp( m2 sup I1. DED
y.11 +3 Σ
sup BED
1 fell?
7EN
7EN n1
(C.22)
+3 Σ
11ʃe simn 2 +3 Σ
|| lin D. || (C.23)
sup BED,
sup
7EN 11
7EN 11
2
MED.
è
è
+ 3
sup
fell + 3
sup I] s 112
(C.24)
00 (m) )
EEN
(A min v /nn)n )
BEN
+ 3
sup oen
[|!!im
1rl1
(C.25)
00
/11)"
The 6 series in the previous expression are convergent. We compute an upper bound for each of the 6 series in (C.21) with the aid of Theorem 10 Lemma 1. and the inequality (B.34). Let max(A:) (At), B = max } and C = max(C; the maximums among the constants in Theorem B.10.
A = max
1. Let us estimate the first series. Observe that we can bound 7 7 77, Moreover, since A(Vy +I+ [|vl]) < 2A for y large enough, up to adding to the constant A a multiple
28

---

of |yll we can bound:
Σ (xp( sup 11ve Sell?
(C.26)
DED,
TEN
A
exp( mm1 (7 1) + lall 2
(C.27)
MEN
+++ 11ll) (V7+I+ l1
(C.28)
nino
+
(P [1bl))
7+ +++ 11
(C.29)
n2
T2 11
- 74
(77"mqno + 2
(C.30)
YEN
16
64y2t0
2
16y2t4
+
+
+
(C.31)
74
7m,
n1
Since the negative exponential function decracases faster than any polynomial, for each p E N there exists a positive constant N, such that xº I exp(- 2) for each æ 2 Np. Therefore, up multiplicative constant not depending n1, Rg, t nor
Σ exp(->m1) sup 11y. - fell 7-64 61no + (9) PEN
Σ
exp(-
2
(C.32)
BED,
2
7EN
7 64 6Ann + (60e e
(C.33)
1-
2. Now we estimate the second series. Using the bounds from the first series combined with Theorem 1.10:
Σ exp( (xp(-mm1) sup I1
Fell2
(C.34)
AED,
PEN
112^
196 2112
< 7Bn Σ expp(-ynp
(7?211 +
+
(C.35)
n1
no
no
TEN
16
387
222
16y + 28-3122 + 47228)
(C.36)
nin0
no
< 7 196 9.nn (1 + td)
exp(-
711
(C.37)
2
7EN
196 9Bn, (1 + +4)
exp(px9
(C.38)
1-exp(-
3. Now we estimate the third series in the same fashion as we did with the preceding series. For an upper bound of L(X'), recall Lemma B.15, which reads
d
1 Vrl7 + 1) log m
L(X) <
29

---

for A E D, and d' a constant not depending on n1, è nor For ni large enough, we can
suppose 1 x(+++) 1) log 81
2
log No
Then,
na
Σ YEN
exp( (nn1)
sup BED,
fife
ft 11
(C.39)
4Cdr log 11 ning
112 2
647
16
7 Texp(-
+
(C.40)
ning
PEN
49> n
28
Tynit
28 1
26y2t
+
+
47y(
(C.41)
2
n0
n
n10
4 112 9Cdr log ni (1 + (1))
7 2
exp(
(C.42)
ning
7EN
4 112 9Crr + 1
(C.43)
n - e-=
4. Now we estimate the fourth series. Following the reasoning from the first series:
16y²t
Σ Σ sup 11:- - f2ll STA
Σ
77 772n2mn + +2222 +
(C.44)
7EN "1 BED
TEN 11
--1
N1
n
16y
(C.45)
7nfnu
71
Recall that choose r large enough that all the summands have ni on the denomi- nator. In particular, it is enough to choose r 2 5 in this case. Moreover, by reasoning like in the proof of the first series, up to multiplicative constant the terms the form n'n, - are
2%%
bounded from above by n1 Therefore, up to multiplicative constant not depending on ni P nor y::
Texp(-nm)) sup 1191- - < 7 74-4--16 6A (0) Σ
Σ
(C.46)
7EN
BED,
7EN 12
7.64 6Ano + 19)n
(1)))
(C.47)
1- n1
5. Now we estimate the fifth series. Using the bounds from the first series combined with Theorem 1. up to multiplicative constant:
Σ F7 sup BED,
1l - Dell
(C.48)
7EN 12
< 7Bm Σ
1
7 +
112
1967An,t2
28-/n
(C.49)
n1
n0p
nn
TEN
11
16
38
2/2
3 + 28-312 + 472(2)
(C.50)
no
no
As we did for the fourth series, we can choose large enough so that the previous series is convergent. is sufficient. Up to multiplicative constant:
Σ
sup 11
fill < 196 7 7-9Bnu +
(C.51)
7EN 1
DED
PEN 72
196 7- 7-9Bno(11 + +t)
(C.52)
1 - 71
30

---

6. Now we estimate the sixth and last series. Following the reasoning in the third and fourth series, up to a multiplicative constant:
Σ 1EN
1 sup
II BED,
11
(C.53)
dC log n1
Σ
6
16y2t²
(C.54)
nini
7EN 11
TT
ning
720
49 1
28
7yn
28
16
+
+ 4ntt
(C.55)
n
n
Ro
R2 7o
In this case, any r 2 3 makes the series converge.
Σ YEN m
sup BED,
1i - f! 11
(C.56)
4 112 9Crr(1 (8)
Σ
(C.57)
ning
7EN
4- 112 9Cdr(1 + 1 1m
(C.58)
mind (1 n2 )
Now we finish the proof by estimating the last terms in (C.21). Recall that, by definition of Ω the bounds for the Frobenius and the sup-norms of the parameters initialization used in the estimation of Is llve delld0 hold also for A E Ω2 and recall that P(..)= S (A Let R ( Then it suffices to choose p large enough to counterattack the biggest exponent of nt in each of the bounds given by Theorem B.10. In particular, seen while choosing when computing the six series, it is enough to take p = r 2 5. Then, up to multiplicative constants,
R1
sup IIve - fell? N ARino(1 + 19) 3 eEn "
(C.59)
/ni"
R1
1 ViT"
sup 1| < R, (1)
(C.60)
ni
Ri
log 71
sup
1lye fell > CdrRi +8)
(C.61)
n
y
BEN
11
Diyning
S Cdr +
(C.62)
Vuing
Grouping the estimations for the nine summands in (C.21) and taking og R₁ max B, Cd), up to multiplicative constant,
/ 1lve - T. 1. < 022T71
/ (1 + t8).
(C.63)
(A )'m
This concludes the proof.
Then, our main result, Theorem 3.4, is direct application of Propositions 3.6 and 3.7:
Proof of Theorem 3.4. By triangle inequality and the elementary inequality (a+ b)2 < 2n + 212 for a, b 2 decompose:
W G.) S 2W D.) + (W,
(C.64)
Then the thesis follows estimating the first summand with Proposition 3.6 and the second one with Proposition 3.7. For large enough лh we can take the constants a and 02 to be multiple of a and 02 in Proposition 3.6; since the right-hand side in the statement in that result decreases as
n2 n1 + -, which strictly slower than the right-hand side of the statement in Proposition 3.7 for any n₁ 2.
31

---

D Proof of Proposition 3.7
Proof of Proposition 3.7. Fix a E R test point. Let GØ Ge(x) and G G((X).
First we show the result for the training set. With the aid of Equation (2.2), we derive a closed ODE for If Gela. By Cauchy- Schhar''s s inquality and Young's inequality
fl. - G2 2| = (kxxx (y - fe) ki (y -)
(D.1)
2 0t
(lixe (ox) S. Ct) - kz kx G 12
(D.2)
< ||xxxx Ni lll 11y Salllls G Min /1 G2 ||
(D.3)
1Kxx kool 11v fell2 + E llft
G 2 Ama llff
G I|?.
(D.4)
2e
2
Note that, by gradient flow equation have I1fe b|1 S _ x
fl yll for each t 2 0. Choosing
2
E Ami and putting b,
|xxx koo 11 - foll2 yields:
A
a
llft Ill.t
1p S Sb.
Amin llf G
(D.5)
Grönwall's 's inequality applied to (D.5) implies:
[15o Goll? + / Amm Nmm
llf Goll S e Amn
(D.6)
||xxx - 1oo 11 - u112t
c Ami
llfo - (
(D.7)
Recall the definition of Wasserstein distance. Taking the expected value of the previous equation and taking the infimum on all the couplings between f. and G2 we can bound by Hölder's inequality:
W (fr, Go) S Blllf
Gell
(D.8)
t
I ->min
Amint
Bl]lxxx koo 11 y/131
Eillfo Goll71
(D.9)
Ami
S e
Ami
kool - = RillSo Gol19)
(D.10)
min
Now we can take the infimum on the couplings between fo and Go to apply Theorem B.3, Proposition B.4 and Lemma B.8 to estimate the right-hand side of (D.10). There are positive constants ei and C2 not depending on ny such that:
W2 G1) se A
-mm
Cil
V 32 + 8 + C2
(D.11)
Ao0
Ce A
(t 1),
(D.12)
n1
with C max
2el Ape
88n
+
21yl/1 111111,022 o2).
Now we show the result for an arbitrary test point x E Ro Let k-c = hiso (2, X). We can bound, by Equation (2.2),
at (y: - Gz)² 2(kxx - fi) ( G. )) GG
By the formula for the derivative of the product and Cauchy-Schwarz 's inequality, the preceding equation implies:
Ko - - (D.13) Ot (vi Gi) S Sax(y- fi) k (y - G² = ||kxx - 1 fill -
32

---

Put X = minigien Roo (2, xz). The last summand can be further estimated with the first result in this theorem. There exists a positive constants C depending on n2 such that:
A
0 Ot
ACe Vt+1.
G I ||,2x - k2 fill +
(D.14)
Again, we use 11ʃe y|l S e + fo vll- Integrating we obtain:
AC /
A
(ye Gi) (yo- Gi) + ||xx Illl foll e -
Vh Jo
V8
le
2 ds
(D.15)
- (10 - Gb) + | - 2 Illy foll - Vaiit +
JCD (2
vt le
A
vmi
(D.16)
for positive constant D, which explieit computation now show separately. First we compute an antiderivative of /t + le
f Vs le
A3 4m ds
(D.17)
2e 2
2
uu
du
(D.18)
A
2²
ue
ue
2e
du
+C
(D.19)
X0 min
min
3000 2 u2
VT
2e
2e
2
du
+C
(D.20)
2( }3 min
VT
min
Am
V eeff v
2e
+C
(D.21)
00 min
(2 )3
(sM(++1)
Veerl
A
le
(x+1)
v22
2e
+
(D.22)
A min
v 2 00 }3
Aon (s+1) V22
2RE
erf
2 +
+
(D.23)
(100 ()
min
where in the first step we substituted u V + 1, in the third step we substituted the fourth step we used the definition of Gauss error function ex); and C denotes the integration constant. Therefore,
u and in
[ V8 1s+le le Ju ds
2 - 2 Ie
(D.24)
A0o
A V 2TTE
Aie (f+1) v/
V A
erf
erf
v22
(D.25)
(A:00 ) )
Since erf(t) 6] - 1, 11, we can bound:
L/" Vo le
100 ds
2
3000
2π
1 vl
e
(D.26)
20 min
mnn
< D(2 - +t+ 1e
(D.27)
33

---

with D max [1 x
2T mis
C
yuu
Turning back to (D.16), we can finish by applying the elementary inequality (a + + c _ 3a² + +62 + 3c for a, b, 2 and Hölder's inequality. After that, Theorem B.3, Proposition B.4 and Lemma B.8 can be applied in the same fashion as in the proof of the training case, yielding positive constants di, d2 and d3 such that:
W (Ye, G.) Elyy: G
(D.28)
3B Gūl
3E
| &
foll -
(D.29)
3(ADD)²
A
Vt+le"
(D.30)
n1
3d1
6d2
+
8n² | + +
2
(D.31)
11
3(XDD)²
(4 +4+ (t 1)e Amin
(D.32)
71
By putting C max (3d 12( and D max { (6d2 3 +
+11lo] we
obtain the thesis.
Note that C,C and D do depend neither on ni nor
E Proofs of the auxiliary results
We present here all the remaining proofs. For clearness, we will use the following notation: X. l(x)), and X 1(x'), for each
n1, for any 2, R
Proof of Lemmaa A.I. is trivial when B is nonsingular. Let A....., An be the (possible repeated) ordered eigenvalues of and suppose X5 = O. Then, by using the eigenvalue decomposition of and elementary properties of the matrix exponential:
A1t
A1
A
L(B)B U
t
U
UT
(E.1)
- Ant An
An
1 A1e
Apt
U
UT
(E.2)
-nat
A1t
LUT -U
0
UT
(E.3)
e Aut
1n
Bt
(E.4)
The converse equality is proven in analogous way. As for the limit property, it enough to show it for real numbers. Let (a,)nen be a real sequence converging to R.. If a 0, of if an 0 for each the result is trivial. Thus, assume 0
34

---

and, up to taking subsequence, that On # 0. Then
- a
lim I((an) = lim 11-000 n On
=t (0==((0)
Proof of Lemma B.I. For the proof of the first three points we refer to any monograph on the Wasserstein distance such Villani [2008]. In order to show (4), let x XY be optimal transport plan between X and Y, let xx2 be the law of X and let µ² Px be the marginal law of X. Applying the gluing lemma of optimal transport produces a probability measure л on I" x R R" given by
77 XY (a,y) v) 1x (x)
(2,2) x
XY (x, X2 (2,
(2, 2, v)
(x)
xxx(x))
Note that integrating with respect y yields xZ and integrating with respect to 2 yields x XY, Note also that there is a unique coupling between Y and X, which is given by Py x 8x, hence,
Wp {8 (X,8)
[x(x,, x) y (v, " (ctu (dw) (dæ, dz, dy)
(E.5)
Rm
(x(x22) Yv. A) du (dz, d2,
(E.6)
Ro+m+n
| | Y| XY (dx, dy) + / 3m
11 - X11"
(E.7)
R
W (X,Y) + W (Z.
(E.8)
where 2º² is the marginal probability on Z.
On the other hand, to show (5) fix μ x v the product measure on R=+m with the tensor product o-algebra. Then
P(R") and P(R"") two probability measures, and denote
WP Y) < Ex ||| - 111
(E.9)
S E xxu [[| 2 + 1 V
(E.10)
2 [E - 21 + E, 1| V1
(E.
where in the last step we applied the elementary inequality (a b)2² < 2a- + D), for a, 2 0 and p 2 1. For the 1-Wasserstein distance instead, we apply the elementary inequality Va + S
Vu + Vb:
Wi(X, V) Elx Y11]
(E.
E (1xx - YJI + 1| - )
(E.
ExllIx YIIJ + E, Z- VII]-
(E.14)
Lastly, taking the infimum over (p, E P(R") x ((R")) finishes the proof.
Proof of Lemma B.2. The computation of the gradients is by chain rule and definition of f. The claims about the kernels follow from taking the dot product on the gradients we just calculated, for each 1 < i,j S n₁:
(x (V geo) X₂) (V
000 X3)
(E.15)
1
Σ
(x0%) (0)
(xl
(0)
(E.16)
710
00 (0)
2e
1.....723
(E.
70
R.....
1
Ly 2 1,85.
(E.18)
73g
35

---

Lastly,
k(x,20)=
( f(2
v
too
((2) (2)) + (Voco Yoo
fle (x)) 1 m am
(E.19)
+ no
r010
)o(1)
x' (1000))011 (0(1))
(E.20)
11
08
x0
x (0100)0001
(E.21)
2
/mo
00
Vno
n1
Σ
Fy 4
(x.))
+ ((x))
(E.22)
71
n1
E.1 Proof of results at initialization
In this subsection we prove the useful Proposition B.4, which generalises the second part of Theorem B.3. This step is paramount for the proof of the rest of our results. From now to the end of this subsection assume & and C' are bounded and x, x' E no are fixed.
Proof of Proposition B.6. Note that, from Lemma B.2, is can be written as:
k
knn Σ
Σ (Xx))
and recall that & is deterministic, for any fixed inputs x, x'. Hence, by boundedness of d and Φ', and independence of g(1) and g())
B] & (1| B[knn + 110132) = 11
(E.23)
B B||+P]]
El|
(X))
1 (X, (XX))
(B.24)
71
m1
< 2
11 + 20-1
(F.25)
22p 1)!!
20
kurl" 1" 1 x
(E.26)
where in the last inequality we used that the 2p-th moment a standard Gaussian variable is equal to (2p 1)!!
Proof of Proposition B.4. We will use the notation instroduced in Proposition B.6. The first claim is trivial since & coincides with IC. To show the second claim, we split:
-
Σ Σ +(X))(X)))
Bllki - 1 E1 K
(B.27)
n=l
KEG V (G(x))
(G
(E.28)
2 'Ell
Te
KE
n2
(F.29)
+ +2P E
((X))((X)) - Eg (GG())
(E.30)
71
36

---

To bound the first summand in (E.29) we split again by adding and substracting an auxiliary term:
k1
① (Xx
(
(Xx))
- (G)))
(E.31)
11
2 2
E1
1
ki
nn x >>>
(E.32)
111
==1
1
n1
2
E
K.
Φ (Xx))
KEG ( (G(())
(E.33)
71
1
2
EE[
(k11
K)
(E.34)
111
2
BI
(X))) Ea (GG)))
(E.35)
The first summand (E.34) vanishes since equals IC estimate the second summand in (E.34),
once again by adding and substracting an auxiliary term:
E
D ('(x))
(X (X:))(0)) (G))
(E.36)
71
<
2"
E
(X.)) (X))) 1)I
(E.37)
11
+ 2
E
('(X)
Ea (C(
(E.38)
711
The first summand in (E.37) vanishes, by boundedness of and independence of the parameters (
V (X.) (X)(()))
1)P)
(0 1ld]
(E.39)
71
0=1
2p
Σ
II IE
(E.40)
=1y==
0.
(E.41)
As for the second summand in (E.37), by Theorem B.3 there exists constant ei not depending on such that:
Φ
(X,) ('X,)
Eg ("C))
(E.42)
1 R1
E
Φ (X,)) ) (Xx) (C))
(E.43)
71
Lipp' + 4'(0)
<1
(E.44)
/ni
37

---

It remains only to bound the second summand in (E.29). This is done by using again Theorem B.3. There exists constant C2 not depending on n, such that:
WB
$
(F.45)
11
= E
Φb (X, ( -G
(E.46)
R1 p=
Lip + $(0) P
I 02
(E.47)
Vni
Putting together all the preceding estimations obtain:
E K1 1oo P S C 3
(E.48)
ni
with C 2P-1 max (220-222220 20 (Lip&' (0()), C2 (Lip (())).
These results suffice prove Proposition B.7.
Proof of Proposition B.7. Consider the joint random variables X (kxx, f(X)) and Y
(), Then Lemma B.1.4 together with Proposition B.4 and Theorem B.3 yield
W, 8) < (p kin (X, x)) (V)((()) S C+D
(E.49)
VmL
where is the constant in Proposition B.4 and the one in Theorem B.3. Both constants do not depend on n..
Lastly, we prove Lemma B.8.
Proof of Lemma B.8. Let 0g denote the ij-th entry of 000) r and let off) denote the j-th component of 0 R By Jensen's inequality and independence of the parameters and 2 ....... Zni
Billfooll] S Ell 1 (XX0 0 ||21
(E.50)
S Vny (1100 /nllla
(E.51)
(E.52)
As for the fourth moment,
27
foll = E
Σ
2 (x. (z. 10
(E.53)
711
n22
s [ 0 )081)
(E.54)
7
71
+
| n
(3n1 + n²
(E.55)
n
4m
2 |
(E.56)
Triangular incquality finishes the proof.
38

---

E.2 Approximation of the network by linearization
In this subsection we prove the results involved the proof Proposition 3.6.
With a slight abuse of notation, we will denote by || - 2 the positive quantity supisisn | Iz
Also, given any matrix A = (ai)) isisn we will denote by 8^ f the matrix V Laas 15/5m
a
Isisn. 15/sm
We will consider the linearized gradient flow, given by
0 Vofolf (x: y).
ōt
For this subsection introduce the following notations: f!n (2'; and Vr = f (2; 0t).
Proof of Lemma 12. Let Amin be the smallest eigenvalue of kg. By gradient flow equations for the parameters 0g and 0,:
life yll S Amind Ilfo - yll,
(E.57)
! yll S e Amn' Ilfo - y/1.
(E.58)
On the other hand, Lemma B.2 combined with Cauchy-Schward Schwarz's inequality and the gradient flow equations produces the following system of differential inequalities:
01 (0
5
|| fo - y Amit
(E.59)
VTE
(0 ) S 0t
/ || fo - yll I. )
-mint
(E.60)
The previous is triangular system of differential inequalities of the form
à (o Th
)
Bie-Amnt
(0 )
$ Bo ) ) Amn!
with B = 1
Vai | 1//oooll.o yll and Bo Vnino 1''1 - vlll 1
By integration on [0, and substitution we get:,
(0()), S )0 + B. S (0 )o+ + || 00 Ilfo vll
(E.61)
Ie(Mmin);
(E.62)
/mi
(0 ) - (0 )o + B1 [
Ta T. )ds + Boll I (Amin)
(E.63)
| 00 | 11oo Il - 110l-11111200 11/
< (o()) )o
Iel Ie(Nmin)
(E.64)
2m /noi
④ lo llfo vll Ka
Le Le(Amin) (0(1))
(E.65)
VTinoo
Note that in the last inequality, we used 1 (.(D)ss <
1.0)² 2 for any 2 0.
Thanks to (E.57), the linearised parameters 0, also satisfy the preceding inequalities, and hence the thesis holds.
Proof of Lemma 1.1.. Let denote the CDF of a standard Gaussian variable. For each >> 0, since the entries of 8( are nı i.i.d. standard Gaussian variables,
P([10)") ll.0 < = (1 (1- - 2(1 - "
(E.66)
39

---

Bernouili's 's inequality and standard estimations for Gaussian tails yield
10 S a) 2 1 - 2n1 (1 (())
(B.67)
a²
1 ne exp
(F.68)
Let r 2 1 and put a = Vry n1. Then:
P([10 1|20 sa) 2 n1 exp
my log n1
(F.69)
2
exp
(lognn ni
7
(E.70)
1
(E.71)
n1 -1
Proof of Lemma 151 We will write f for short of fo) By Lemma B.2 and Cauchy-Schwarzz
inequality,
a
a f112
|ofll =
fl +
(E.72)
0
00
1 |
||
12
+ 1 10011
+
(E.73)
Then the first claim follows by (B.34) and the elementary incquality Va va+bs S V Vb, for a, 2 0.
Now we prove the second inequality. Let 6, Ø E R.W. then,
a [f) - V (0) = 0 f(0) - a f(0)1 + 1 f(0)
20(1) f(0]11 (E.74)
Let us estimate the first summand in the previous expression. By Lemma B.2, for each < u <
no 1 S S n
f())
f())
(E.75)
0010
nl
12 I
Iu (
x (0) (0(1)
22( 710
(())
(E.76)
710
100 j=1
ju
j=1
Fu
4'(
x (0) (0 0(0)
(E.77)
Vn
Vno
1=1
2()((1) (()
ng
1ig
Fr (8'(
D' n 27,
000000
(E.78)
Too
nO
(8
000)
Fy Lipp'pp 00
no
5
Fu
+
3.
(0(9) 3m j0
0100) ).
(E.79)
Vn1n1
/ninn no
1=1
Hence,
i
10 110)
(E.80)
ōg
[(0
[())
(E.81)
0
00
||>|||
क 11 ||0(1)
11
[I 210001)))))0)0) Ii& (((0) - - 12
(E.82)
n1no
ning
|| x
(10(1)
(1
((0) 1
my logni, n1,
(E.83)
nin
40

---

with probability greater or equal than 1 1 where in the last step we used Lemma B.14.
_
Similarly, the second summand can be estimated as follows. First compute the partial derivatives by using Lemma B.2, for each 1 S v S n21
a
f(0)
f(0)
(E.84)
0 a 1)
(
3 0
200 jo
))
(E.85)
/n1
To
n10
j=1
7=
Lip
I 20 - 0
(E.86)
Therefore,
0
(0)
f(0)112
(E.87)
20(1)
00(1)
0
f(0)
0 () f(0)
(E.88)
00
(Lip
|0000)
11
(E.89)
The preceding estimations, together with 16
for i yield:
110-0112
Vof(0) Vef(@)112
(E.90)
|00 011
| | 111
| (Lip
(Lip
log
(E.91)
72
ning
Taking the square root in the last inequality yields the thisis.
Proof of Lemma 16 Fix y N. The probability of Z ||k li ||
7 can be estimated with
Markov's 's inequality and Proposition B.4. There exists constant C > 0 not depending on n such that:
7
P min
P(Z
min
=
P
7P
>
(F.92)
2
2
2
P
Bi|l B|||2|||]
(E.93)
71.00 2
2
(K:
(E.94)
7
0 min
(E.95)
7000 00
Note that Proposition B.4 holds for every natural p. This concludes the proof.
Now are ready prove Proposition B.9:
Proof of Proposition B.9. For the sake of clearness we introduce the following abbreviations for the
remainder of the proof. Let yt = f(; fr), D. = flin (x) ft = f(X;; 6,) and f! = f (x;; B.).
Also, let hi k(X 0.), V Vo and let ((x) denote the Lipschitz constant of v.. function of 0.
Consider the empirical risk for the quadratic loss Ro(O.) Ei (f(L -
41

---

From gradient flow equations we have:
Ot ft = -ke(f= - y):
(E.96)
0
f y11² - y, kife - y))
(E.97)
0t
Let t, = inf(t 1|0 eigenvalue with respect to the operator norm, and by definition of tw, we obtain an upper bound for Amin (k²):
Poll
Ouie } Then for each S tu, by 1-Lipschitzianity of the smallest 2L(X)
Mmin (Ki)) Aminl S l|ke
kullop I l1ke koll S L(X)11 Ball I min
which implies:
mmin
Aenin
Amia (Kx) 2 Amin
2
This estimation combined with Grönwall's inequality applied to (E.97) yield:
[ y/1² S Ilfo - u112 exp (-22m))
Amin
(E.98)
From (E.97) and Cauchy-Schwarz we deduce:
Vf. 3)12
/ l1 y/l
(E.99)
IIfe yl1
Mmin
V fe x)1l-
(E. 100)
2
Hence,
i [Ife y/l
miin 100, 0oll)
11
a11
min
0.ll O.
(E.101)
01
at.
for all S tx.
Thus, for $ tw:
100
0oll <
Il5o
y|1.
(E.102)
0min
Let us show that this property holds for all Assumption implies
0. By contradiction assume t, co. (E.102) with
2 o o T min (L(X) min
1|0 Ooll
(E.103)
(E.104)
In particular the last inequality holds for t., which contradicts its definition. Hence t. = 00.
Let us now prove the rest of the inequalities in the theorem.
The gradient flow equation for the linearised network reads:
i
0t ft = ku y).
(E.
Define the difference ri = St flin Then
Ft = ke - y) + ko - y)
(E.106)
Øt
--krt - (kg - kJ y).
(E.107)
42

---

Then, by Cauchy-Schwarz and (E.98) combined with (E.105),
||r. = - hrrr - (rx, (K2 (o! w))
(E. 108)
2 01
s - --Aiin (aann(is))l0 1lrel ||relllks koll fl fl yll
(E.109)
Amin ||rl ||rxll Il vll exp (- Amint
(->min²
)
(E.110)
Hence,
3 Ōt
Amin
Amint
ril |1 -
||rlll +
||A,
ko Ilfo
y/l exp
(E.111)
4
4
Now let us bound separately the different factors in the previous equation. The norm of the difference between the kernels can be estimated as:
l|ke koll I IVVVA foV II
(E.112)
= fo V/ V + || - V.oo2?
(E.
1 max L (()) - Bull + L(x) 0: - 0ull
(E.114)
- max (x)) - doll + + 0oll 2
min
(E.115)
max L (X') Bull
(E.116)
where in (E.115) we applied the definition of t,.
Moreover, by Grönwall and -Scwwaz inequalities we have
Amint
=ll / l|ke kollds
||rll s exp
Il
(E.117)
Amint
S exp
11 v/1 sup lik koll e20
(E. (18)
Amint
5
I exp
mmxx [(X) fo yll sup
1|0 - 8oll
(E.119)
4
2
$20
Amint
5
2
I exp
mmax fo Il
sup
Il 311
(E.120)
220 T min
Amint
50 maxax L(X)
S exp
y|
(E.121)
Amin
(E.122)
Moreover, by taking the difference of the gradient flow equations for 6g and Bg we obtain:
0
10, 0.ll < I|V foll vll + V.folll -
(E.123)
01
$ T( - 0oll 1I - -yll + mmax
s!inn
(E.124)
2T((X) I1fo y11 exp
1-Amn'
(E.
Jmin
4
+ 50
Aanina
((X y1l
exp
(E.126)
mmin
4
(2 + 50 max
2
L(x)
(-Amint Amnint
f
y11
exp
(-a4))
(E.127)
min
4
where in (E.125) used 102), (E.98) and E. 121.
43

---

Integrating the previous inequality we obtain:
(2 + 50 max (
115o y11 / exp
Amin
110 0+ll x
ds
(E.128)
min
4(2 5 max [T(2))
|150 y/2 (1 (1- exp (- Axmino ))
(E.129)
(8 + 20 2
((t)
1l5o - y112.
(E.130)
Now we are ready to prove the last inequality in the thesis. Decompose by triangle inequality:
11 - v.ll - 11 - (2; 0e)1l + (x; (c) - yell-
131)
First, let focus the first summand of (E.131). Denote by L(x) the Lipschitz constant of Vyo seen as a function of 0. Then, by Lemma B.15,
11ye - (x;0,)) (VS(;; Vf(a;; 0o)) 2
0,ds|
(E.132)
0t
$ [(xx) sup 100 Poll / || 0
0
111ds
(E.133)
120
2 1y foll
< L(x) sup 0 Poll
(E.134)
(20
Fmin
41yy
foll
S L(x)
(E.135)
Amin
where in the third inequality we used (E. 101) and (E. 102) on the last one.
As for the second summand of (E.131), by (E.130) and Lemma B.15:
1 0t) Tell
V fz; (0) 0.)11
(E.136)
(8 + 20 max )(X)) 3 min
|lfo (b)
(E.137)
Combining the two preceding estimations, we obtain the thesis.
Lastly, we prove Theorem B.10.
Proof of Theorem B.10. We prove the three inequalities separately. Let Amin denote the smallest eigenvalue of kg.
By Lemma B.2 and Cauchy-Schwand inequality,
1yt - fell < Vnino
Lip&l| x211110!0 )(1
(E.138)
44

---

Recall that (miia) S L. Then the norm 11000 0 B.12:
112 can be estimated with the aid of Lemma
no
110000
11
Σ
(0(
(E.139)
no
m2
an (0 Ja (b)))
ao(0²") 10
(0 )) (0(0 Jo +
nivnd (b))))
n1
(E.140)
(00))883 + d' c
((6)) + dha (0!*) mu
2 2x2
4pd1 3
nu mIvnu mo
ni v
(E.141)
ag
ni (0 20 ) (00
(00))
(E.142)
ni
aga
86 20 20
2
(0
)
(0))
(00)
(E.143)
nino
no
717100
0 1|
< m1|100 0 1
ai 11
(00)
(E.144)
n4
agai (0.) 6 86
a
a
11 12
(00)
11
(E. 145)
with ag Hence,
1''00 ab = |0 and @1 = -
(Lip@) 2|1 - XI|
11 - fell <
00!10010011
(E.146)
TOT
Au
11000 1
Art?
110 (06))
A
nino ((6))
1124
(E.147)
n0
non1
Agt
A4t²
1 1|2
(00)
(1'1)1(0000)
(00)
(E.148)
n2n0
nino
with Ao = (Lip - x11 A1 = ai. A2 = ag A3 = af
A4 = 0 and A5 = afpaf.
We follow similar strategy to prove the second inequality in the Theorem. Put = 0, - to- By the triangle inequality and Cauchy-Schwar we decompose:
I| v.112 < 21
yol
Volo
(E.149)
The first summand in (E.149) is bounded exactly as the first summand in (E.138) by setting 0:
Ilfo 30ll2 5
(Lipo)
111
110 0
12
Ao
11000 00 11.
(E.150)
45

---

As for the second summand in (E.149), we decompose by Lemma B.2. Factoring out max; ( ((060))) = 110,"|>> oo permits us write:
Ve - Voyol1 -
(fo
- 80)l + 20(h) (fo - -
(E.151)
20(0)
1
|1x
T
26 - x T (
x (0) 1)00 11 |?
(E.152)
11100
Vno
vno
1
|
1
$ T2o
x61 00)
(E.153)
n1
Vno
2
|1
x
x 40(
1
000
1
(E.154)
71
Vno
2
|1
T Φ
x x T Φ
(0)
(E.155)
11n0
no
Ao 110%0112
(E.156)
2
(Lip&')
X11 012
(E.157)
2
||
10
x113100001 12
Ao n0
1000 ||
(E.158)
71
Moreover we can bound the norm of w. with Lemma 1.
T 2
10,
1 + 10 10
(E.159)
2
2a 20 (000) y
of
(00)*(1
+
(000) +
Σ
(10))
(E.
nino
nio
n1
2a
20 20 (00)) +27
46
(00)
4,4
(E.161)
n
n1
Hence, (E.149) can be written as:
fln
3.12 <
Bu
100
(0)
0 1 1 +
B1
10000 00 0 11)00)0))))
(E.162)
n1n
ning
B2
110
1 1100006 11
10 (00)2
B3
(E.163)
nīn0 3
Bing
B4
Bs
13
+
(E.164)
nimo
ninno
Bs
By 110 (1))((0))''*
B7
(E.165)
nino
Bs
+
Bu
10 (3,5(00))
(E.
nino
nnno
where the constants in the last inequality are, explicitly, Bo = 2Ag. B, = 8(Lip | [|2- X (lla)2² B2 8(Lip& || || X'
(g))) B3 4(Lip
|X| | x((aa)> 2 B4
8( D' 1oo || - X211ay)2 B3 = 8(1| 1|x o x(11())², BB 4(100//oool x²
f By = 4Aoa,, Bs 4Ana" 0- and B, 2
46

---

It remains to estimate the last inequality. Consider A(t) = llfe f! Then by gradient flow equations and Fauchy-Schware
i (A(1)²) (NelSe v) ko y); )(mm)
(E.167)
(kr( v) - -koo (t) v))(f((x.)) (x))) (E.168)
Σ (kr (0i,,2)) 2) - kolos, x²) v)(((z;)) - (z:))
(E.
==1
ko X) - - (x:))
-
(E.170)
11(k, ko) vf 11 1 [(n)({s > 1l (E.171)
By equivalence of the 1-norm and the euclidean norm for E Fd we have ||o|| s ||lll S
Vd|uul.. Then, by Cauchy-Schward inequality,
āt (t) nl|ks - koll life ul V A(t)
(E.172)
<nlk: koll (00)) -Ammt No A(t)
(E.173)
Let bound the norm of k. ko:
| koll Vof((() - -
(E.174)
S (V) + [01662))11((]1100--00 6oll
(E.175)
(E.176)
From Lemmas B.2 and B.12, we have:
( = V ai 1() + Von f(
(E.177)
| 4 11 110 12
| |
(E.178)
7
(108"1 +
||
1500)
11
1
(E.179)
71
n2
Analogously,
Vofo(x>)] = Vo (2) + aan fo(2)11
(E.180)
| 4 113. 1106") 1|2
|
(E.181)
T2p
n1
Moreover, again by Lemma B.
||0g - 0ull2 = -10(0) - 0000 1|2 + 110!00 -000112
(E.182)
20g (0,) 444
200²
<
10 12,2
(E. 183)
n nino
n1
n1
(E. 184)
Inequalities (E.111))) and (E. 183) allow us to estimate:
||,: koll < L(x))
c (1,) (10)))) !
c22)(60)
(3))
(E.185)
n
Vn
ni
/nino
ca call00' 1it
1 11:2
c6 col10( ||t
(E.186)
no
Vnnno
Vno
(00000
(0))22)
+ coo(0601)
(E.
Vno
/n
47

---

with c 2v |+1| llo,
V2aol o co
2v2an 30
2x || ||
V2a' f Iloe, 2v ||1 C7 = 2aq ||
| ||oc Cs
a1| xo and 0g 2ag < lol. Let ((n 2, 0b) be the right hand side of (E. 185). Then, the reight-hand side of (E.173) can be IP-almost surely bounded from above with:
Øt (1) S n - koll (b0)) Amat
A (())
(E.188)
S nC(nn, no. t, 0o).
(E.189)
In the previous inequality we used that the event Amin has null measure. Integrating, and using that (0)) = 0:
(t) S mL(X)
(11(00)91100 l| +
c22(06)
+
(E.190)
Vnnno
2n
Vn
call cal16) 2 11222
css (B)((1)
cs
10
(E.191)
2no
3/11n0
2/no
C7 (0n)
||,2
(x))(00)
(ar)(0n)
(E.192)
2/mo
3/n1no
2
Taking the square, applying the elementary inequality (E a;)² 2 S E of for a; and adjusting the constants yields the desired result.
48


Reviews:
[{"review_id": "ykI1gQyOax", "reviewer": "Reviewer_9rHw", "summary": "The authors addresses a gap in the current literature on infinite width neural networks trained by gradient descent by providing a quantitative characterization of the distance between a finite-width neural network approximation and its infinite counterpart. The authors specifically focus on shallow (single-hidden-layer) networks and demonstrate that the Wasserstein distance decays polynomially with increasing network width (Theorem 3.4).\n\nThis is validated experimentally in Section 4, where the authors empirically compute the Wasserstein distance between a collection of finite-width networks and the samples from the infinite-width network.", "questions": "On Theorem 3.4, the bound's polynomial dependence on $t^6$ is noted as becoming slack for fixed $r$ and $n\\_1$ as $t$ grows unboundedly, even though the authors state that a sufficiently large $r$ can make the time-dependent term negligible for $t$ growing polynomially with $n\\_1$. Could the authors provide a more detailed intuition behind this $t^6$ term? After all, one would expect training to converge as time goes on.\n\nAdditionally, the authors briefly mention the work by de G. Matthews et al. (2018), in that paper, the authors prove the weak convergence of arbitrary width BNNs at initialization under the metric:\n\n$$\n\\rho(f,f') = \\sum\\_{i=1}^\\infty \\min(1, |f(x\\_i)-f'(x\\_i)|),\n$$\n\nwhere $\\\\{x\\_i\\\\}\\subset\\mathcal{X}$ is a countably infinite input set. The same paper also empirically compare the distance of these distributions under the  maximum mean discrepancy (MMD) under a RBF kernel. As the bound of the authors holds for $t=0$, I believe the authors should briefly discuss the relationship, if any, between these two metrics ($\\rho$ and MMD) with the Wasserstein metric. For example, does convergence in one implies convergence in another? Do the bounds relate? Why pick one over the other?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "I believe this paper offers an interesting initial step into the evolution of the convergence of infinite-width neural networks. It is also commendable that the authors address the limitations and future directions of their work in clear terms.\n\nIn terms of clarity, currently, assumptions are stated across two different sections (”Notation” and “Assumptions and Main Result”), I believe keeping all assumptions in one section would improve the flow of the paper. Additionally, the authors use, at the same time, notations where the dependency of $f$ on the parameters is implicit (i.e., $\\dot{f}\\_t(x)$) and where it's explicit (i.e., $\\nabla\\_{\\theta}\\mathcal{R\\_D}[f,\\theta]$). Given that there is still space available, I would suggest the authors keep the notation on the explicit side and write $\\frac{\\partial}{\\partial t}f(x;\\theta\\_t)$ to streamline reading by others.", "was_revised": true, "final_justification": "The authors have addressed all of my issues in a thorough manner, I believe these additions and corrections will increase the connections of this paper with the existing literature and also point to readers where improvements in current results can happen.", "rebuttal": "We thank Reviewer 9rHw for their careful reading of our manuscript and for the constructive comments.\nWe now address each of the weaknesses and questions raised by the reviewer:\n\n1. Intuition on term $t^6$ in the main theorem:\n\nWe added the following remark to the main theorem to try to give an intuition of this term:\n\t\nThe term $t^6$ in the right hand side of the inequality is due to Lemma B.12 and Theorem B.10 in the Supplementary Material. Lemma B.12 provides upper bounds of the entries $\\theta^{(0)}_t$ and $\\theta^{(1)}_t$ that account for perturbations that occur on tail events with respect to the initalization distribution (i.e. in the ``bad event\" $S^C$).\nA finer control is possible if one is interested in a result that holds on $S$ only, which has high probability, such as the ones in [Bartlett, Montanari and Rakhlin '21, Chizat, Oyallon and Bach '19].\nThis finer control corresponds to Theorem B.9.\n\n2. Relation to $\\rho$ and MMD metrics from [Matthews et al. '18]:\n\t\nWe thank the reviewer for raising this important point. Our result connects with the findings of [Matthews et al. '18], which prove the convergence of fully connected neural networks to a Gaussian process under the following metric:\n$$\n\\rho(f,f') = \\sum_{i\\in \\mathbb N}\\frac{1}{2^i} \\min (1,\\vert f(x_i)-f'(x_i)\\vert),\n$$\nwhere the $x_i$ form a countable set of inputs.\nOur setting, as opposed to the setting of [Matthews et al. '18] considers  a finite set of test and training points.\nHowever, it is natural to consider in our setting the version of $\\rho$ restricted to finite input, which we shall denote $\\rho_F$.\nThen the expected value of $\\rho_F$ can be related to $\\mathcal W_2$:\n$$\n\\rho_F(f,f') \\le \\sum_{i=1}^n \\frac{1}{2^i} \\vert f(x_i)-f'(x_i)\\vert < \\| f-f'\\|_2\n$$\nBy taking expected value with respect to a coupling $\\pi$ and then taking the infimum on the couplings $\\pi$ between $f\\sim \\mu$ and $f' \\sim \\nu$ we obtain:\n$$\n\\mathbb E[\\rho_F(f,f')] < \\mathcal W_2(\\mu,\\nu).\n$$\n\nFurthermore, [Matthews et al. '18] also studies convergence under the maximum mean discrepancy (MMD) with respect to a RKHS $(H,k_H)$.\nEven though the connection between MMD and regularized versions of the Wasserstein distance, such as Sinkhorn divergences ([Feydy et al. '19]) or Gaussian-smoothed OT ([Nietert et al. '21]), it is known that in general the MMD cannot be bounded bounded from above by the $p$-Wasserstein distance.\nThe authors of [Vayer,Gribonval '23] found sufficient and necessary conditions on $k_H$ under which, up to a multiplicative constant:\n$$MMD(f,f') \\le \\mathcal W_2(f,f').$$\n\t\nHence convergence in $2$-Wasserstein distance implies convergence in the $\\rho_F$ metric, and convergence with respect to MMD under the hypothesis of Proposition 2 in [Vayer,Gribonval '23].\nNote that, while the metric $\\rho_F$ offers a notion of pointwise convergence, it lacks the geometric interpretability and sensitivity to scaling that Wasserstein-2 captures.\nOn the other hand, also note that the setting of [Matthews et al. '18] is more general since it covers deep networks, while we restrict our study to the shallow case.\n\n\nWe decided to add to the Related Work subsection of the introduction the following paragraph summarizing the above discussion:\n\n\nOur results are closely related to the work of [Matthews et al. '18], who proved weak convergence of fully-connected BNNs at initialization to a Gaussian process under the metric \n$\\rho(f,f') = \\sum_{i \\in \\mathbb{N}} 2^{-i} \\min(1, |f(x_i)-f'(x_i)|)$, defined on a countable input set. \nIn our setting, the input set is finite; considering the restriction $\\rho_F$, it follows that convergence in $\\mathcal{W}_2$ implies convergence in $\\rho_F$. \n[Matthews et al. '18] also analyzed convergence under the maximum mean discrepancy (MMD). \nWhile MMD is not generally controlled by Wasserstein distances, connections have been established via regularized OT divergences [Feydy et al. '19],[Nietert et al. '21].\nMoreover, [Vayer,Gribonval '23] identified conditions on the RKHS kernel $k_H$ under which $MMD \\lesssim \\mathcal{W}_2$. \nConsequently, our bounds also imply MMD convergence under these conditions. \nThe metric $\\rho_F$ which offers a notion of pointwise convergence and, by taking the minimum, is oblivious of the tails of the distributions, which helps stablish the results in [Matthews et al. '18].  \nOn the other hand, $\\mathcal{W}_2$ captures the geometric structure and scaling of the output space. \nFinally, while [Matthews et al. '18] address the more general setting deep networks, our analysis focuses on the shallow case, yielding new quantitative rates which improve previous ones in our setting.\n\n\n3. Weakness: keep all the assumptions (1-4) in the same place?\n\n\nWe agree that the exposition benefits from having all the assumptions collected in one place. Accordingly, we put all the assumptions on a list before Theorem 3.4, which we believe improves the readability of the manuscript.\n\n\n4. Weakness: keep $\\theta$ explicitly in notation.\n\n\nWe agree with the reviewer that it's best to stick with one notation only. This has been now changed to show at all times the dependence on $\\theta$ and $\\bar \\theta$ explicitly."}, {"review_id": "8NOdUV779u", "reviewer": "Reviewer_R9Qg", "summary": "This work investigates the rate of which a wide, two-layer neural network trained under gradient flow converge to a Gaussian process, in Wasserstein 2 distance. In particular, the rates allow to characterize the relative scaling between data dimension, network width and training horizon required for the network to stay close to the corresponding linearized kernel predictor.", "questions": "- In L24-26: \n> \"*In this limit, the network evolves approximately linearly around its initialization, and training can be understood as kernel regression with a fixed data-dependent kernel.*\". \n\nWhat do you mean exactly by \"data dependent kernel\" in this sentence? In the strict infinite width limit, the NTK kernel remains context, and corresponds to the kernel of the network at initialization. \n\n- Although this is something ultimately personal, I find the choice of notation confusing. For instance, from the equation below L114, I understand that since the output is a scalar, $x\\theta^{(0)}\\in\\mathbb{R}^{n_{1}}$ denote the standard (left) matrix multiplication between the matrix $\\theta^{(0)}\\in\\mathbb{R}^{n_{0}\\times n_{1}}$ and the vector $x\\in\\mathbb{R}^{n_{0}}$. Then, to produce a scalar (after applying the non-linearity component-wise), should I understand the product with $\\theta^{(1)}\\in\\mathbb{R}^{n_{1}}$ as a scalar product? But then the eq. below L125 would suggest that $h_{i}\\in\\mathbb{R}$ and not $\\mathbb{R}^{n_{1}}$ since $\\theta^{(0)}_{- i}\\in\\mathbb{R}^{n{0}}$ is a vector? Can you clarify what I am missing or if this is a typo? \n\n- In L182, what is the definition of the mentioned Gaussian process $G_{t}$? The one appearing after in eq. (2.5)?\n\n- What is the main challenging of extending the result for unbounded activation $\\Phi$, e.g. ReLU?\n\n- How tight are these rates? In other words, can one turn this result around and use it as an estimate of the training timescale over which a wide network with NTK scaling start to learn features? \n\n**Small typos**:\n\n- L224: polinomially -> polynomially", "limitations": "The authors discuss the limitations of their result in Section 5.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**: The paper is clearly writen: the motivation and goals are clearly stated, and the result is compared with the relevant related literature. The better control over the rates allowing to consider longer timescales on gradient flow is interesting.\n\n\n**Weaknesses**: The main weakness of this works concerns the significance. Although there is still some interest on the NTK regime in the NeurIPS ML theory community, I believe it is now consensual its limited relevance to the theory of neural networks. Nevertheless, the refined NTK rates could be of interest to a subset of the community.", "was_revised": true, "final_justification": "This is a well-written and technically sound paper with new results concerning the convergence rate of wide neural networks to Gaussian processes. As stated in my review, I believe the main weakness is the significance of the scope of the results to the NeurIPS ML theory community.", "rebuttal": "We thank Reviewer R9Qg for their careful reading of our manuscript and for the constructive comments.\nWe now address each of the weaknesses and questions raised by the reviewer:\n\n1. Clarification on \"In this limit, the network evolves approximately linearly around its initialization, and training can be understood as kernel regression with a fixed data-dependent kernel.\":\n\nWe thank the reviewer for raising this point. With the original phrase we intended to emphasize that the infinite-width kernel is a function of $(x,x')$. \nWe agree that, in the infinite-width limit, the kernel is independent of the network parameters $\\theta^{(0)}, \\theta^{(1)}$. \nTo avoid ambiguity, we have revised the sentence to: \n\"In this limit, the network evolves approximately linearly around its initialization, and training can be understood as kernel regression with a fixed kernel which depends on the the architecture only, and is a function of $(x,x')\".\n\n2. Issue with scalar products:\n\t\t\nWe agree with the reviewer. The i-th preactivation $h_i(x;\\theta)$ belongs to $\\mathbb R$.\nWe corrected this typo and deleted the subsequent definition of $h_{ij}$.\n\n3. In L182, what is the definition of the mentioned Gaussian process ? The one appearing after in eq. (2.5):\n\nWe thank the reviewer for noticing this incoherence in the order of exposition.\nWe moved equation (2.5) before that paragraph in order to fix this.\n\n3. What is the main challenging of extending the result for unbounded activation, e.g. ReLU?\n\nFor unbounded activation, the inequalities in Lemma B.12, which are paramount to showing our main result, become empty.\nAlthough our proof does not hold for unbounded activations for this reason, we believe the result is still true in this setting and that the proof can be improved.\nThis is now acknowledged in a new paragraph in the Discussion section:\n\nWe conjecture that our main result remains valid even without Assumption 3, as suggested by our numerical experiments with the ReLU activation. \nIn this work, we deliberately focused on a specialized setting with mild hypotheses to obtain a novel and technically precise result while maintaining a clear exposition. \nFuture research will aim to relax the regularity assumptions on the activation and extend our analysis to a more general setting.\n\n4. How tight are these rates? In other words, can one turn this result around and use it as an estimate of the training timescale over which a wide network with NTK scaling start to learn features?\n\t\t\nAs far as we know, our bounds are not tight.\nWe rephrased the first item in the discussion session, which discusses limitations of our main result, in order to include a brief discussion on this matter and potential connections to feature learning:\n\t\t\nOur main result is not uniform in time.\nAlthough the dependence on time can be minimized at the price of including a sufficiently big multiplicative constant in the right-hand side of our inequality as discussed in Remark 3.5,\na general result holding uniformly in $t>0$, in the limit when $t$ tends to infinity exponentially on $n_1$ is not available. \n\nThis dependence on time could be related to the transition from the NTK regime to a feature-learning regime, as suggested by the work of Huang and Yau (2020). \nTheir analysis, however, does not address the tails of the distributions, which in our proof correspond to the set $S^C$ and are responsible for the $t^6$ scaling.\nMoreover, Yang and Hu (2021) show that under standard and NTK parameterizations, wide networks cannot perform feature learning in the infinite-width limit. \n\nThis suggests that our observed $t^6$ scaling might reflect the boundary of the NTK regime: \nin the ``bad event'' $S^C$ or for sufficiently large times, the training dynamics may drift into feature-learning, where purely kernel-based control breaks down.\nOur main result remains consistent with works such as [Bartlett, Montanari and Rakhlin '21, Chizat, Oyallon and Bach '19], which hold with high probability, whereas our analysis explicitly incorporates the contribution of the event $S^C$.\nAt present, it is unclear whether the $t^6$ scaling is sharp.\nWe would like to address this problem in future work."}, {"review_id": "qL7Uoj9W6h", "reviewer": "Reviewer_yCLR", "summary": "This paper provides quantitative upper bounds on the Wasserstein-2 distance between the output distribution of large-width shallow neural networks, trained via continuous-time gradient flow, and their associated Gaussian processes at any given test point. Building on prior works that identify the limiting Gaussian processes in the infinite-width regime, the authors derive polynomial convergence rates in network width.", "questions": "- In the experiments, it is unclear how the authors determine the value of $t$ corresponding to the number of gradient steps.\n- Suggestion: Include more experiments aimed at quantifying the quality of the theoretical bound. For instance, in the rightmost plot of Figure 1, the decay rate appears closer to $1/\\sqrt{\\text{width}}$ than the expected $1/\\text{width}$, suggesting the empirical rate may differ from the theoretical one. Clarifying this discrepancy would strengthen the paper.\n- Suggestion: Add a discussion of the related line of work on optimization and generalization dynamics in overparameterized neural networks and clarify how the present contributions differ or connect to those results.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\nThe paper presents novel quantitative results that extend known convergence properties of wide neural networks to the full training trajectory, with explicit polynomial bounds in Wasserstein-2 distance. The writing is clear and well-organized, making the topic accessible. In particular, the proof sketch provides valuable insight into the structure of the argument and highlights the roles of key components, such as linearization and kernel concentration.\n\nWeaknesses:\n- There is a rich, yet unmentioned, line of work studying the convergence properties and training dynamics of shallow neural networks trained with gradient flow or gradient descent, including extensions to deep architectures and various nonlinearities. Notable examples include Arora et al. (Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks) and Du et al. (Gradient descent provably optimizes overparameterized neural networks), among others. A discussion of the similarities and differences with these works would improve the positioning of this paper.\n- The experiments, while appreciated, are quite limited. In particular, there is no empirical assessment of the accuracy of the derived upper bounds, or at least of the convergence rates as a function of the width or input dimension, for example. Such experiments would help evaluate how well the theoretical insights align with observed behavior.", "was_revised": true, "final_justification": "Thank you for the clarifications and the additional discussion of related work. The new paragraph and citations satisfactorily address my concerns about prior literature coverage. I have therefore revised my overall score to reflect these improvements.", "rebuttal": "We thank Reviewer yCLR for their careful reading of our manuscript and for the constructive comments.\nWe now address each of the weaknesses and questions raised by the reviewer:\n\n1. In the experiments, it is unclear how the authors determine the value of t corresponding to the number of gradient steps:\n\nWe thank the reviewer for pointing this out.\nTo clarify, as stated in line 263 of the manuscript, in both experiments we define $t$ as the product of the learning rate and the number of iterations.\nA new sentence to emphasize this has been added to the caption of the plots.\n\n2. About Experiments: \n\nWe thank the reviewer for this observation. \nWe would like to clarify that, while the reviewer is correct that the relation between $n_1$ and $\\mathcal{W}_2^2$ is $\\frac{1}{n_1}$, the plot in question corresponds to $\\mathcal{W}_2$. \nTherefore, a slope of $-\\frac{1}{2}$ in logarithmic scale is indeed expected.\n\n3. Discussion on the works of Arora et al, Du et al:\n\nWe thank the reviewer for this valuable suggestion and fully agree that the line of work initiated by Arora et al. and Du et al. is central to our topic. In response, we have added a new paragraph to the Related Work subsection of the introduction, where we now cite these seminal papers along with several subsequent contributions that build upon them. \nWe hope that this addition provides an adequate overview of the relevant literature. Should we have overlooked any important reference, we would be happy to include it in a revised version.\n\t\t\nHere is the paragraph we propose:\n\nA foundational stream of research has shown that, under sufficient overparameterization, gradient-based training of neural networks converges to a global minimum. \nSeminal results by [Du et al. '19] and [Arora et al. '19] established that for wide two‑layer networks with ReLU activation, the empirical NTK remains well-conditioned, enabling convergence via kernel regression. \nSubsequent advances generalized these results to deep architectures in different directions, such as [Allen‑Zhu et al. '19,\nZou and Gu '19,\nSankararaman et al. '20,\nWu et al. '19, Wei et al. '19, Zou et al. '20],\nwhich provide guarantees that hold with high probability over parameter initalization. \nThese works reinforce that in the NTK regime, the network trajectory stays close to its linearization around initialization. \nOur contributions align with this body of work \nand further extend this literature by providing novel finite-sample quantitative bounds on the Wasserstein‑2 distance between neural network outputs and their Gaussian process approximations."}, {"review_id": "cu8admxIIL", "reviewer": "Reviewer_3AEa", "summary": "This paper studies the Gaussian approximation of shallow neural networks for arbitrary test points and at any time during gradient-descent training. The  approximation is quantified in terms of the quadratic Wasserstein distance between the two random variables $f_t(x)$ and $G_t(x)$ where $G_t$ is the Gaussian approximation. This result can be extended to training times that scale at most polynomially with the network’s width.  This is done by first controlling the distance between the true and the linearized evolutions, and then by bounding the distance between the linearized dynamics driven by the empirical NTK and that driven by the deterministic limiting NTK.", "questions": "- Would it be possible to include a figure showing the best power-law fit to illustrate the bound on the rate of convergence of the Wasserstein distance?\n- Could you add a discussion on the dependence of your bound on the test point? For example, does this Gaussian approximation hold uniformly in $x$?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths: \n- The paper is clearly written, results are in line with expectations and nicely extend previous works, which focused either on the intialization, or provided qualitative results on the training (or results in probability). \n- The strategy of the proof given in Section 3.1. provides a good summary of the key steps, and enables readers to grasp the essential ideas. \n- The experiments address practical consideration. In particular, the authors take into account the fact that the Wasserstein distance is estimated by considering that of the empirical measures. Section 4.2 explicitly discusses the number of samples required to obtain a good approximation. \n\n\nWeaknesses: \n- While the qualitative convergence was previously know, explicit quantitative rates were not derived for the training dynamics. The experiments could have been more illuminating if they explicitly showed the best power-law fits capturing the the decay of the Wasserstein distance. \n- The quantitative bounds shed light the effects of the network width, input dimension, and training time. However, a discussion on the dependence on the test point $x$ would have been useful.", "was_revised": true, "final_justification": "The authors addressed my points (adding a power-law fit and discussion on the x-dependence). After reading the other reviews and answers from the authors, I confirm my score.", "rebuttal": "We thank Reviewer 3AEa for their careful reading of our manuscript and for the constructive comments.\nWe now address each of the weaknesses and questions raised by the reviewer:\n\n1. Would it be possible to include a figure showing the best power-law fit to illustrate the bound on the rate of convergence of the Wasserstein distance?:\n\t\t\nDue to the reviewing policies of Neurips we cannot upload pictures or .pdf files to the responses, but we modified\nExperiment 2 to include the power-law fit between the points in the plot, in a different color.\nThis is also acknowledged in the body of the numerical experiments section.\n\n2. Could you add a discussion on the dependence of your bound on the test point? For example, does this Gaussian approximation hold uniformly in x?\n\nWe thank the reviewer for pointing out an interesting question.\t\t\nThe dependence of our bounds in the choice of the test point comes from Theorem B.10 and Proposition 3.7, and this dependence was previously not discussed in our draft.\nA new paragraph in the discussion section has been added to address this:\n\nThe bound in our Theorem 3.4 depends on the test point $x$.\nThis dependence is explicitly stated on the proof of the auxiliary results Proposition 3.8 and Theorem 3.10 in the Supplementary Material.\nLocally uniform bounds on the test point $x$ might follow from functional inequalities such as the ones found by [Favaro et al. '25] if extended to the NTK regime."}]

Author rebuttal:

Meta-review: This paper focuses on the rate of which a wide and two-layer neural network with gradient descent converge to a Gaussian process in Wasserstein 2 distance. After the discussion phase, almost all responses seem to be fixed well. The novelty and improvements of this work have received praise from reviewers. However, there still are some doubts on the formal expression of the main theorems and the readability of this paper.
Decision: Accept (poster)