Title: Quantitative convergence of trained neural networks to Gaussian processes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural networks

NTK

Gaussian process

Wasserstein distance

Gaussian approximation

shallow neural networks

wide limit

infinite-width neural network
Abstract: In this paper, we study the quantitative convergence of shallow neural networks trained via gradient descent to their associated Gaussian processes in the infinite-width limit. 
    While previous work has established qualitative convergence under broad settings, precise, finite-width estimates remain limited, particularly during training. 
    We provide explicit upper bounds on the quadratic Wasserstein distance between the network output and its Gaussian approximation at any training time $t \ge 0$, demonstrating polynomial decay with network width. 
    Our results quantify how architectural parameters, such as width and input dimension, influence convergence, and how training dynamics affect the approximation error


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the Gaussian approximation of shallow neural networks for arbitrary test points and at any time during gradient-descent training. The  approximation is quantified in terms of the quadratic Wasserstein distance between the two random variables $f_t(x)$ and $G_t(x)$ where $G_t$ is the Gaussian approximation. This result can be extended to training times that scale at most polynomially with the network’s width.  This is done by first controlling the distance between the true and the linearized evolutions, and then by bounding the distance between the linearized dynamics driven by the empirical NTK and that driven by the deterministic limiting NTK.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n- The paper is clearly written, results are in line with expectations and nicely extend previous works, which focused either on the intialization, or provided qualitative results on the training (or results in probability). \n- The strategy of the proof given in Section 3.1. provides a good summary of the key steps, and enables readers to grasp the essential ideas. \n- The experiments address practical consideration. In particular, the authors take into account the fact that the Wasserstein distance is estimated by considering that of the empirical measures. Section 4.2 explicitly discusses the number of samples required to obtain a good approximation. \n\n\nWeaknesses: \n- While the qualitative convergence was previously know, explicit quantitative rates were not derived for the training dynamics. The experiments could have been more illuminating if they explicitly showed the best power-law fits capturing the the decay of the Wasserstein distance. \n- The quantitative bounds shed light the effects of the network width, input dimension, and training time. However, a discussion on the dependence on the test point $x$ would have been useful.\n\nQUESTIONS:\n- Would it be possible to include a figure showing the best power-law fit to illustrate the bound on the rate of convergence of the Wasserstein distance?\n- Could you add a discussion on the dependence of your bound on the test point? For example, does this Gaussian approximation hold uniformly in $x$?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides quantitative upper bounds on the Wasserstein-2 distance between the output distribution of large-width shallow neural networks, trained via continuous-time gradient flow, and their associated Gaussian processes at any given test point. Building on prior works that identify the limiting Gaussian processes in the infinite-width regime, the authors derive polynomial convergence rates in network width.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper presents novel quantitative results that extend known convergence properties of wide neural networks to the full training trajectory, with explicit polynomial bounds in Wasserstein-2 distance. The writing is clear and well-organized, making the topic accessible. In particular, the proof sketch provides valuable insight into the structure of the argument and highlights the roles of key components, such as linearization and kernel concentration.\n\nWeaknesses:\n- There is a rich, yet unmentioned, line of work studying the convergence properties and training dynamics of shallow neural networks trained with gradient flow or gradient descent, including extensions to deep architectures and various nonlinearities. Notable examples include Arora et al. (Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks) and Du et al. (Gradient descent provably optimizes overparameterized neural networks), among others. A discussion of the similarities and differences with these works would improve the positioning of this paper.\n- The experiments, while appreciated, are quite limited. In particular, there is no empirical assessment of the accuracy of the derived upper bounds, or at least of the convergence rates as a function of the width or input dimension, for example. Such experiments would help evaluate how well the theoretical insights align with observed behavior.\n\nQUESTIONS:\n- In the experiments, it is unclear how the authors determine the value of $t$ corresponding to the number of gradient steps.\n- Suggestion: Include more experiments aimed at quantifying the quality of the theoretical bound. For instance, in the rightmost plot of Figure 1, the decay rate appears closer to $1/\\sqrt{\\text{width}}$ than the expected $1/\\text{width}$, suggesting the empirical rate may differ from the theoretical one. Clarifying this discrepancy would strengthen the paper.\n- Suggestion: Add a discussion of the related line of work on optimization and generalization dynamics in overparameterized neural networks and clarify how the present contributions differ or connect to those results.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the rate of which a wide, two-layer neural network trained under gradient flow converge to a Gaussian process, in Wasserstein 2 distance. In particular, the rates allow to characterize the relative scaling between data dimension, network width and training horizon required for the network to stay close to the corresponding linearized kernel predictor.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: The paper is clearly writen: the motivation and goals are clearly stated, and the result is compared with the relevant related literature. The better control over the rates allowing to consider longer timescales on gradient flow is interesting.\n\n\n**Weaknesses**: The main weakness of this works concerns the significance. Although there is still some interest on the NTK regime in the NeurIPS ML theory community, I believe it is now consensual its limited relevance to the theory of neural networks. Nevertheless, the refined NTK rates could be of interest to a subset of the community.\n\nQUESTIONS:\n- In L24-26: \n> \"*In this limit, the network evolves approximately linearly around its initialization, and training can be understood as kernel regression with a fixed data-dependent kernel.*\". \n\nWhat do you mean exactly by \"data dependent kernel\" in this sentence? In the strict infinite width limit, the NTK kernel remains context, and corresponds to the kernel of the network at initialization. \n\n- Although this is something ultimately personal, I find the choice of notation confusing. For instance, from the equation below L114, I understand that since the output is a scalar, $x\\theta^{(0)}\\in\\mathbb{R}^{n_{1}}$ denote the standard (left) matrix multiplication between the matrix $\\theta^{(0)}\\in\\mathbb{R}^{n_{0}\\times n_{1}}$ and the vector $x\\in\\mathbb{R}^{n_{0}}$. Then, to produce a scalar (after applying the non-linearity component-wise), should I understand the product with $\\theta^{(1)}\\in\\mathbb{R}^{n_{1}}$ as a scalar product? But then the eq. below L125 would suggest that $h_{i}\\in\\mathbb{R}$ and not $\\mathbb{R}^{n_{1}}$ since $\\theta^{(0)}_{- i}\\in\\mathbb{R}^{n{0}}$ is a vector? Can you clarify what I am missing or if this is a typo? \n\n- In L182, what is the definition of the mentioned Gaussian process $G_{t}$? The one appearing after in eq. (2.5)?\n\n- What is the main challenging of extending the result for unbounded activation $\\Phi$, e.g. ReLU?\n\n- How tight are these rates? In other words, can one turn this result around and use it as an estimate of the training timescale over which a wide network with NTK scaling start to learn features? \n\n**Small typos**:\n\n- L224: polinomially -> polynomially\n\nLIMITATIONS:\nThe authors discuss the limitations of their result in Section 5.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors addresses a gap in the current literature on infinite width neural networks trained by gradient descent by providing a quantitative characterization of the distance between a finite-width neural network approximation and its infinite counterpart. The authors specifically focus on shallow (single-hidden-layer) networks and demonstrate that the Wasserstein distance decays polynomially with increasing network width (Theorem 3.4).\n\nThis is validated experimentally in Section 4, where the authors empirically compute the Wasserstein distance between a collection of finite-width networks and the samples from the infinite-width network.\n\nSTRENGTHS AND WEAKNESSES:\nI believe this paper offers an interesting initial step into the evolution of the convergence of infinite-width neural networks. It is also commendable that the authors address the limitations and future directions of their work in clear terms.\n\nIn terms of clarity, currently, assumptions are stated across two different sections (”Notation” and “Assumptions and Main Result”), I believe keeping all assumptions in one section would improve the flow of the paper. Additionally, the authors use, at the same time, notations where the dependency of $f$ on the parameters is implicit (i.e., $\\dot{f}\\_t(x)$) and where it's explicit (i.e., $\\nabla\\_{\\theta}\\mathcal{R\\_D}[f,\\theta]$). Given that there is still space available, I would suggest the authors keep the notation on the explicit side and write $\\frac{\\partial}{\\partial t}f(x;\\theta\\_t)$ to streamline reading by others.\n\nQUESTIONS:\nOn Theorem 3.4, the bound's polynomial dependence on $t^6$ is noted as becoming slack for fixed $r$ and $n\\_1$ as $t$ grows unboundedly, even though the authors state that a sufficiently large $r$ can make the time-dependent term negligible for $t$ growing polynomially with $n\\_1$. Could the authors provide a more detailed intuition behind this $t^6$ term? After all, one would expect training to converge as time goes on.\n\nAdditionally, the authors briefly mention the work by de G. Matthews et al. (2018), in that paper, the authors prove the weak convergence of arbitrary width BNNs at initialization under the metric:\n\n$$\n\\rho(f,f') = \\sum\\_{i=1}^\\infty \\min(1, |f(x\\_i)-f'(x\\_i)|),\n$$\n\nwhere $\\\\{x\\_i\\\\}\\subset\\mathcal{X}$ is a countably infinite input set. The same paper also empirically compare the distance of these distributions under the  maximum mean discrepancy (MMD) under a RBF kernel. As the bound of the authors holds for $t=0$, I believe the authors should briefly discuss the relationship, if any, between these two metrics ($\\rho$ and MMD) with the Wasserstein metric. For example, does convergence in one implies convergence in another? Do the bounds relate? Why pick one over the other?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}