Gradient Flow Convergence Guarantee for General Neural Network Architectures

## Abstract

Abstract A key challenge in modern deep learning theory is to explain the remarkable success of gradient-based optimization methods when training large-scale, complex deep neural networks. Though linear convergence of such methods has been proved for a handful of specific architectures, a united theory still evades researchers. This article presents a unified proof for linear convergence of continuous gradient descent, also called gradient flow, while training any neural network with piecewise non-zero polynomial activations or ReLU, sigmoid activations. Our primary contribution is a single, general theorem that not only covers architectures for which this result was previously unknown but also consolidates existing results under weaker assumptions. While our focus is theoretical and our results are only exact in the infinitesimal step size limit, we nevertheless find excellent empirical agreement between the predictions of our result and those of the practical step-size gradient descent method.

## 1 Introduction

Behind training any successful deep learning model is the hope of achieving a low loss. In this article, we turn that hope into certainty, proving that gradient flow based training does not just succeed, but it converges exponentially fast, thus proving that training procedures need not be a gamble.

Understanding why gradient-based optimization methods are so effective at training large, complex deep neural networks is a central question in modern machine learning theory. Despite the non-convex nature of the loss landscape, these methods can find solutions with low training error. This empirical success is well-known and is a reason for the success of neural networks in various real-life tasks.
However, finding theoretical guarantees that training via gradient methods converges, especially at exponential rates(^1^11Linear convergence of loss and exponential decay of loss mean the same thing. This is common terminology used in optimization - linear convergence of loss means that the loss $L$ decreases linearly, ie. $L_{t}\leq\alpha L_{t-1}$ for some $\alpha\geq 0$. Iterating backwards, we get $L_{t}\leq\alpha^{t}L_{0}$.) and across a broad class of architectures, remains an active area of research. Such guarantees are not only of theoretical interest but also provide valuable insights into why neural networks generalize well and how to design better models.

As we discuss below, a number of works have studied linear convergence of gradient methods for specific network architectures, initialization schemes or under strong over-parameterization assumptions. However, a unifying framework that applies across diverse architectures and does not rely on restrictive assumptions has been missing.

In this article, we close that gap by proving that training neural networks using continuous gradient descent will almost always lead to an exponential decay of the training loss, ie. we prove that at time $t$ of applying continuous gradient descent, loss is ${\mathcal{O}}(e^{-t})$. An informal version of our result is as follows.

###### Theorem 1.1 (Informal theorem) .

Let $P$ be the number of trainable parameters of a neural network, $n$ be the training dataset size and $M$ be the output dimension. If the network is over-parametrized, ie. $P\geq nM$ and the initialization, input data distributions are absolutely continuous, then linear convergence of loss holds with probability 1.

Figure: Figure 1: The error curve for a deep neural network trained on random data is bounded by an exponential decay. Thus, the error here is ${\mathcal{O}}(e^{-t})$.
Refer to caption: pics/fig1.png

In other words, as long as the dataset and initialization are not pathological, and the network has sufficient capacity to represent the task, GF (gradient flow) leads inevitably to exponential loss decay. A larger output dimension or larger sample size means the learning is potentially much more complex, and thus needs more parameters to be learnt. This conclusion is aligned with the well known empirical claim that over-paramterized networks perform better.

To do so, we track the training dynamics using the NTK, as done by previous works (Du et al., 2019b, a) and use properties of ODEs (Chatterjee, 2022).

Our novel contribution lies in realizing that the set of initializations for which the NTK has a zero eigenvalue has zero probability with respect to initialization and input data distributions, and that this property is preserved throughout training. Once positive-definiteness of the NTK has been established, routine calculations lead us to an exponential decay of the loss.

Compared to existing work, our proof is both more general and simpler. The argument avoids lengthy technical calculations, requiring only basic probability and ODE theory. Yet it yields a result that appears to be absent from the literature.

One subtelty arises from activation functions. Our proof directly applies to piecewise nonzero polynomial activations, which excludes the widely used ReLU and sigmoid functions. To solve this problem, we use a piecewise non-zero polynomial activation that uniformly lower bounds ReLU/sigmoid and use a limiting argument to show that if this activation converges uniformly to ReLU/sigmoid, then exponential decay holds for ReLU/sigmoid as well.

###### Theorem 1.1 (Informal theorem) .

#### Applicability of our work

Theorem [3.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1) holds for a general function structure composed of polynomial layers and a piecewise nonzero polynomial activation function, which covers many popular architectures like DNNs, ResNets (He et al., 2016), CNNs (Krizhevsky et al., 2012), ConvResNets (Du et al., 2019a), GCNs (Kipf and Welling, 2017), ChebConvNets (Defferrard et al., 2016), GraphConvNets (Morris et al., 2019), U-Nets (Ronneberger et al., 2015), MobileNets (Howard et al., 2017) and SiameseNets (Taigman et al., 2014) with activations like leaky ReLU (Maas et al., 2013) or parametric ReLU (He et al., 2015), just to name a few. Note that corollaries [3.1.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1.Thmcorollary1), [3.1.2](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1.Thmcorollary2) extends our conclusion to ReLU, sigmoid activations as well. Moreover, most common initializations and input data are sampled from an absolutely continuous distribution. Thus, our work holds for most of the practically relevant architectures, input data and initializations.

### 1.1 Comparison with Previous Works

The seminal work by Jacot et al. (2018) proved that the training of DNNs with a smooth activation function in the infinite width limit for a finite time with gradient descent can be characterized by a kernel. Our proof technique relies on the same kernel matrix used by them.

Linear convergence of training error has been known under simple settings. Some works that use a trajectory-based analysis of the algorithm dynamics are compared in table [1](https://arxiv.org/html/2509.23887v1#S1.T1) along important assumptions and conclusions. It is to be noted that despite ease of analysis with GF, most of the previous works do not apply to GF. We have closed this gap with this work.

Chatterjee (2022) is one of the closest to our work. It proves that for DNNs with twice differentiable activations trained on a linearly independent input training dataset with either GF or GD with small step size, convergence is linear. However, their assumptions are yet again much more restrictive in comparison to ours.

**Table 1: Assumptions and conclusions of previous works. ‘w’ stands for architectures without biases, ‘s’ for shallow, ‘L’ for layers, $L$ for width of the network, $n$ for training dataset size, $T$ for training time, $E,V$ for the number of edges and vertices of the graph used for GNN, D for DNN, C for CNN, R for ResNet, CR for CNN-ResNet, G for GNN, pl for polylog, p for poly and e for error. We write etc. whenever there are too many specific conditions (or cases) that need to be satisfied (or can be applied to), and instead just mention the most restrictive (or the most common) one. Specific refers to a condition that is fairly specific and cannot be condensed into one word.**
| Paper | GD or GF | Architectures | Activation | Rate |
| --- | --- | --- | --- | --- |
| Allen-Zhu et al. (2019a) | GD | D (3L,w) | ReLU | $1/\sqrt{T}$ |
| Allen-Zhu et al. (2019b) | GD | D,R,C (w) | ReLU | $e^{-T}$ |
| Du et al. (2019a) | GD | D,R,CR (w) | Analytic, etc. | $e^{-T}$ |
| Nguyen and Mondelli (2020) | GD | D (w) | Smooth, sub-lin. | $e^{-T}$ |
| Zou et al. (2020) | GD | D (w) | ReLU | $1/T$ |
| Awasthi et al. (2021) | GD | G (s) | ReLU | $e^{-T}$ |
| Chen et al. (2021) | GD | D (w) | ReLU | $1/T$ |
| Liu et al. (2022) | GD | D (w) | Twice diffn. | $e^{-T}$ |
| Gopalani and Mukherjee (2025) | GD | D (s,w) | Softplus | $e^{-T}$ |
| Du et al. (2019b) | GF&GD | D (s,w) | ReLU | $e^{-T}$ |
| Chatterjee (2022) | GF&GD | D | $C^{2}$ | $e^{-T}$ |
| Boursier et al. (2022) | GF | D (s,w) | ReLU | $e^{-T}$ |
| This paper | GF | Any | ReLU, $\sigma$, etc. | $e^{-T}$ |
|  | Hidden size | Initialization | Training Data | Out.dim. |
| Allen-Zhu et al. (2019a) | ${\mathcal{O}}(1/\text{e})$ | Normal | Normalized | Multi |
| Allen-Zhu et al. (2019b) | p$(L,n)$ | Normal | Separable | Multi |
| Du et al. (2019a) | ${\mathcal{O}}(n^{4})$ | Arbitrary | Normalized | Single |
| Nguyen and Mondelli (2020) | n | Normal, etc. | Almost any | Multi |
| Zou et al. (2020) | $\Omega(n^{14})$ | Normal | Separability | Single |
| Awasthi et al. (2021) | Arbitrary | Normal | $E=o(\sqrt{V})$ | Multi |
| Chen et al. (2021) | pl$(1/\text{e})$ | Normal | n=${\mathcal{O}}(1/\text{e})$p$(L)$ | Single |
| Liu et al. (2022) | ${\mathcal{O}}(n)$ | Normal | Arbitrary | Multi |
| Gopalani and Mukherjee (2025) | Arbitrary | Arbitrary | Arbitrary | Single |
| Du et al. (2019b) | ${\mathcal{O}}(n^{6})$ | Normal | Non-parallel | Single |
| Chatterjee (2022) | Arbitrary | Arbitrary | Linear ind. | Single |
| Boursier et al. (2022) | Specific | Nonzero | orthogonality | Multi |
| This paper | $P\geq nM$ | Almost any | Almost any | Multi |

The works by Min et al. (2023); Bah et al. (2022); Tarmoun et al. (2021); Xu et al. (2025) proved linear convergence of GF/GD, but for linear networks (ie. networks with linear activations, essentially making it linear regression). Hutzenthaler et al. (2023) provides a detailed analysis of convergence of both GF and GD, but for constant target functions.

Other approaches to finding convergence properties include landscape-based analysis (Arora et al., 2022; Jin et al., 2017), mean-field approximation based analysis (Chen et al., 2022; Ding et al., 2022), using optimal transport theory (Chizat and Bach, 2018; Khamis et al., 2024) and studying models in the richer albeit challenging ‘feature learning regime’ (Yang and Hu, 2020; Kunin et al., 2024).

#### Why Gradient Flow ?

We focus on gradient flow to not have to keep track of discrete step sizes, which complicates the analysis and pulls attention away from the more important insights. Moreover, Elkabetz and Cohen (2021) proves that GF is close to GD with small step sizes for DNNs with homogeneous activations. It is to be noted that even though analysis using GF is usually less cumbersome than GD, the proof in this paper is not anywhere in the literature - despite the simplicity of its arguments. Previous works do talk about the convergence of GF, but they are again limited to either shallow networks (Du et al., 2019b; Boursier et al., 2022; Gopalani and Mukherjee, 2025; Tarmoun et al., 2021) or deep networks with smooth (Chatterjee, 2022) or linear activations (Min et al., 2023; Bah et al., 2022; Xu et al., 2025). As stated before, we nevertheless back up our claims empirically by training with GD.

#### Our Novelty

The major novelty of our result is that it holds for a large variety of network architectures. Previous results have often been found for bias-less architectures (Du et al., 2019b; Allen-Zhu et al., 2019b; Liu et al., 2022), one-dimensional (Gopalani and Mukherjee, 2025; Du et al., 2019a) or constant (Hutzenthaler et al., 2023) targets, very high over-parameterization (Du et al., 2019a; Allen-Zhu et al., 2019a; Chen et al., 2021), a specific initialization of weights (Liu et al., 2022; Zou et al., 2020; Nguyen and Mondelli, 2020) or strong assumptions on input data like orthogonality (Boursier et al., 2022) or linear independence (Chatterjee, 2022). Such convergence guarantees have also been found for other architectures like GNNs (Awasthi et al., 2021), CNNs (Allen-Zhu et al., 2019b) and ResNets (Du et al., 2019a). However, there has not been one work that combines all these results (and more) into a single, easy to follow theorem.

Note that though there have been guarantees for ReLU or sigmoid networks, there has been no work regarding leaky ReLU or parametric ReLU activated networks in the past. Our result holds for, as discussed before, a plethora of networks. We thus keep our experiments in section [4](https://arxiv.org/html/2509.23887v1#S4) limited to the more popular architectures.

To summarise, our major contributions are as follows:

- •
We prove linear convergence of gradient flow for a wide variety of architectures. Previous analyses have worked out linear convergence on a case-by-case basis (Awasthi et al., 2021; Bah et al., 2022). We prove a general theorem that works for any general function structure composed of polynomial layers and any piecewise non-zero polynomial activation, or ReLU/sigmoid activation.
- •
We require initialization to be from any absolutely continuous distribution. Previous analyses usually have only considered normal distributions (Du et al., 2019a; Nguyen and Mondelli, 2020).
- •
We require overparametrization of only at least $nM$. Previous works have required over-parameterization to be of much larger magnitudes (Du et al., 2019b; Chen et al., 2021).

Despite being a rather short and concise proof, our contributions sharpen, generalize, and in some cases correct limitations in prior work by covering a wide range of architectures with weaker assumptions on activation functions, initialization, training data and over-parameterization.

## 2 Setup and Preliminaries

Consider a continuous function $f:\mathbb{R}^{N}\times\mathbb{R}^{P}\to\mathbb{R}^{M}$ defined as follows for some $N,P,M\in\mathbb{N}$:

$$ $f(X,\theta):=g_{L}(\sigma(\ldots g_{2}(\sigma(g_{1}(X,\theta)))\ldots))$ (1) $$

for all inputs $X\in\mathbb{R}^{N}$ and parametrizations $\theta\in\mathbb{R}^{P}$, where $g_{i}:\mathbb{R}^{N_{i-1}}\times\mathbb{R}^{P}\to\mathbb{R}^{N_{i}}$ are polynomials (in each component) for all $i\in[L]$ with $N_{0}=N,N_{L}=M$ and $N_{1},\ldots,N_{L-1}\in\mathbb{N}$ for some $L\in\mathbb{N}$. $\sigma:\mathbb{R}\to\mathbb{R}$ is a continuous, piecewise polynomial (with finitely many closed pieces) which is zero at finitely many values only. Note that $\sigma$ applied to a vector means that it is applied component-wise.

We shall prove our result for this function. We do so because most of the neural network architectures can be represented by this function. Examples will be given in corollary [3.2.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem2.Thmcorollary1).

We intend to find that parametrization of $f$ that minimizes its error over a training dataset $(X_{1},y_{1}),\ldots,(X_{n},y_{n})\in\mathbb{R}^{N}\times\mathbb{R}^{M}$, where $L:\mathbb{R}^{P}\to[0,\infty)$ is the error, defined by $L(\theta):=\frac{1}{2}\sum_{j=1}^{n}||f(X_{j},\theta)-y_{j}||^{2}$
for all $\theta\in\mathbb{R}^{P}$.

To do so, we utilize gradient flow. Start from some initial parameters $\theta_{0}$ and input data $X_{1},\ldots,X_{n}$ sampled from some arbitrary absolutely continuous probability distributions (with respect to the Lebesgue measure). Then, for some $T>0$, solve the following initial-value problem till $t=T$ to get the final parametrization $\theta(T)$.

$$ $\displaystyle\derivative{\theta(t)}{t}$ $\displaystyle=-\partialderivative{L(\theta(t))}{\theta(t)}$ (2) $\displaystyle\theta(0)$ $\displaystyle=\theta_{0}$ $$

We shall often write $\nabla L(\theta(t))$ in place of $\partialderivative{L(\theta(t))}{\theta(t)}$. Note that the $f$ need not be differentiable with respect to $\theta$ everywhere, but it is differentiable almost everywhere, due to the definition of $\sigma$. Note that this is will not be an issue, since the ODE theorems are applied piecewise and the training dynamics are not affected by almost everywhere differentiability.

The following lemma will prove that this ODE has a unique continuous and piecewise differentiable solution that can be extended till any $T>0$.

###### Lemma 2.1 (Existence and invertibility of GF solution curve) .

The GF initial-value problem ([2](https://arxiv.org/html/2509.23887v1#S2.E2)) has a unique, continuous and piecewise differentiable solution that can be extended till any $T>0$, for almost all initializations and input data. Moreover, the solution flow is a diffeomorphism over the parameter space minus a measure zero set.

###### Proof.

Note that we assume $\nabla L(\theta(0))\neq 0$ to exclude the trivial case of constant solution. Since GF cannot reach such a critical point, we need not worry about this case for the inverse solution. Moreover, we assume the initialization does not occur on a boundary, in order to apply Picard’s theorem. Since boundaries and the set $\{\theta\in\mathbb{R}^{P}:\ \nabla L(\theta)=0\}$ have zero measure (since $\nabla L(\theta)$ is a piecewise nonzero polynomial for almost all input datasets), the following arguments hold for almost all intializations amd input data.

Consider $-\nabla L(\theta)$ in its initialization piece. Then, it is a polynomial in $\theta$, and thus is Lipschitz continuous in any closed ball around the initialization $\theta_{0}$. By Picard’s existence and uniqueness theorem (Hartman, 1964) the GF equation has a unique solution in a neighbourhood of $t=0$. Since there is no finite time blowup of the solution (see theorem [A.3](https://arxiv.org/html/2509.23887v1#A1.Thmtheorem3)), it can be extended beyond a neighbourhood of $t=0$ (Hartman, 1964), either to all $t>0$ or till it hits the boundary of its piece.

If this solution hits the boundary of the piece containing the initial condition $\theta(0)$, we can construct solution for the same ODE in the next piece, with initial condition being defined as the end point of the previous solution curve. Note that the curve will reach an intersection of more than two pieces for a zero measure set of initializations (see theorem [A.1](https://arxiv.org/html/2509.23887v1#A1.Thmtheorem1)). Consequently, we do not need to worry about having to choose between two or more pieces for the solution curve to continue into.

Finally, these solutions can be joined together continuously (due to no finite time blowup, see theorem [A.3](https://arxiv.org/html/2509.23887v1#A1.Thmtheorem3)) until we reach $t=T$. Define this piecewise differentiable, but continuous everywhere solution of the GF equation by $\Phi_{t}(\theta_{0}):=\theta(t)$ for all $t\geq 0$.

Now, note that the $\Phi_{t}$ is a piecewise diffeomorphism over the initial parameter space minus a measure zero set (ie. excluding critical points, boundary points and points with image/preimage curve that reaches an intersection of more than two pieces) since $-\nabla L(\theta(t))$ is piecewise continuously differentiable with respect to initial conditions (Hartman, 1964), all piecewise solutions are invertible as they are solutions of an autonomous system of ODEs, which all exist and can be joined uniquely by the above arguments, and the inverse solution $\Phi_{t}^{-1}=\Phi_{-t_{0}}$ is also piecewise differentiable (again by previous arguments).
∎

Figure: (a) Solution curve of the ODE in the initialization (blue) piece till finish ($t=T$). We cut off this solution at the point where it exits the initialization piece.
Refer to caption: pics/init_piece.png

###### Lemma 2.1 (Existence and invertibility of GF solution curve) .

###### Proof.

## 3 Main Result

###### Theorem 3.1 (Linear convergence of GF) .

Appling gradient flow to functions of form ([1](https://arxiv.org/html/2509.23887v1#S2.E1)) with $P\geq nM$, the following holds almost surely with respect to the initialization and input distributions, for all $t\in[0,T]$.

$$ $||y-F_{t}||\leq e^{-\lambda_{0}t}||y-F_{0}||$ (3) $$

which is equivalent to saying $L(\theta(t))\leq e^{-2\lambda_{0}t}L(\theta(0))$, where $\lambda_{0}>0$ and

$$ $\displaystyle F_{t}$ $\displaystyle:=\quantity((f(X_{1},\theta(t)))_{1},\ldots,(f(X_{1},\theta(t)))_{M},\ldots,(f(X_{n},\theta(t)))_{1},\ldots,(f(X_{n},\theta(t)))_{M})^{T}\in\mathbb{R}^{nM}$ $\displaystyle y$ $\displaystyle:=((y_{1})_{1},\ldots,(y_{1})_{M},\ldots,(y_{n})_{1},\ldots,(y_{n})_{M})^{T}\in\mathbb{R}^{nM}$ $$

###### Proof.

The first half of this proof is inspired by previous works (Du et al., 2019a, b), where we bound the error using the least eigenvalue of the ‘neural tangent kernel’.

Fix some $t\geq 0$. Gradient of loss can be written as $\partialderivative{L(\theta(t))}{\theta(t)}=\sum_{j=1}^{n}\partialderivative{f(X_{j},\theta(t))}{\theta(t)}\quantity(f(X_{j},\theta(t))-y_{j})$ by chain rule. Using this fact, we calculate the dynamics of the predictions.

$$ $\displaystyle\derivative{t}f(X_{i},\theta(t))$ $\displaystyle=\quantity(\partialderivative{f(X_{i},\theta(t))}{\theta(t)})^{T}\derivative{\theta(t)}{t}=-\quantity(\partialderivative{f(X_{i},\theta(t))}{\theta(t)})^{T}\partialderivative{L(\theta(t))}{\theta(t)}$ $\displaystyle=-\quantity(\partialderivative{f(X_{i},\theta(t))}{\theta(t)})^{T}\quantity(\sum_{j=1}^{n}\partialderivative{f(X_{j},\theta(t))}{\theta(t)}\quantity(f(X_{j},\theta(t))-y_{j}))$ $\displaystyle=\sum_{j=1}^{n}\quantity(\quantity(\partialderivative{f(X_{i},\theta(t))}{\theta(t)})^{T}\partialderivative{f(X_{j},\theta(t))}{\theta(t)})(y_{j}-f(X_{j},\theta(t)))$ (4) $$

Concatenating output vectors, we get $\derivative{F_{t}}{t}=G(t)(y-F_{t})$, with

$$ $G(t):=\quantity(\quantity(\partialderivative{F_{t}}{\theta(t)})_{p}^{T}\quantity(\partialderivative{F_{t}}{\theta(t)})_{q})_{p,q=1}^{nM}$ (5) $$

where $\partialderivative{F_{t}}{\theta(t)}=\quantity(\partialderivative{(f(X_{1},\theta(t)))_{1}}{\theta(t)},\ldots,\partialderivative{(f(X_{1},\theta(t)))_{M}}{\theta(t)},\ldots,\partialderivative{(f(X_{n},\theta(t)))_{1}}{\theta(t)},\ldots,\partialderivative{(f(X_{n},\theta(t)))_{M}}{\theta(t)})$ is a $P\times nM$ matrix and $P\in\mathbb{N}$ is the number of trainable parameters. $G(t)$ is the gram matrix called the NTK at time $t$ of training.

Now, $G(t)$ is a nonegative definite matrix, since it is a gram matrix. We further claim (from lemma [3.2](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem2)) that it is a positive definite matrix almost surely (with respect to initialization and input data).

Differentiating the loss, we get

$$ $\derivative{t}\quantity(||y-F_{t}||^{2})=-2(y-F_{t})^{T}G(t)(y-F_{t})\leq-2\lambda_{0}||y-F_{t}||^{2}$ (6) $$

where $\lambda_{0}>0$ is the least eigenvalue of $G(t)$ for $t\in[0,T]$. $\lambda_{0}>0$ is true because of the following argument. The minimum eigenvalue of a matrix varies continuously with the matrix entries. $G_{pq}(t)$ is continuous since $\theta(t)$ varies continuously with $t$ and $\sigma$ is continuous. Thus, by extreme value theorem applied on the compact interval $[0,T]$, the minimum eigenvalue is achieved at some $t_{0}\in[0,T]$. Say the minimum eigenvalue of $G(t)$ at $t_{0}$ is $\lambda_{0}$. We know $\lambda_{0}>0$ due to lemma [3.2](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem2).

Rearranging, we get $\derivative{t}\quantity(e^{2\lambda_{0}t}||y-F_{t}||^{2})\leq 0$. Thus, putting in $t=0$ and $t=s\leq T$, we get $||y-F_{s}||\leq e^{-\lambda_{0}s}||y-F_{0}||$.
∎

###### Corollary 3.1.1 (Extension to ReLU activation) .

The above linear convergence result holds for ReLU activation as well.

###### Proof.

Let $F_{\alpha,t},G_{\alpha}(t)$ be the output vector and gram matrix for a leaky ReLU network with hyperparameter $\alpha\leq 1$ at time $t$. Let $F_{t}$ be the output vector for the same network but with ReLU activation instead of leaky ReLU, at time $t$. Now, note that

$$ $(y-F_{\alpha,t})^{T}G_{\alpha}(t)(y-F_{\alpha,t})\geq(y-F_{\alpha,t})^{T}\tilde{G}(t)(y-F_{\alpha,t})$ $$

where $\tilde{G}(t)$ stands for the gram matrix of the linearized network. This holds because $G_{\alpha}(t)_{pq}\geq\tilde{G}(t)_{pq}$ for all $p,q\in[nM]$ since leaky ReLU is bounded below by identity function.
Since equation [4](https://arxiv.org/html/2509.23887v1#S3.E4) holds for any a.e. differentiable function and leaky ReLU converges to ReLU,

$$ $\displaystyle\derivative{t}\quantity(||y-F_{t}||^{2})$ $\displaystyle=-2(y-F_{t})^{T}G(t)(y-F_{t})=-2\limsup_{\alpha\to 0}(y-F_{\alpha,t})^{T}G_{\alpha}(t)(y-F_{\alpha,t})$ $\displaystyle\leq-2\limsup_{\alpha\to 0}(y-F_{\alpha,t})^{T}\tilde{G}(t)(y-F_{\alpha,t})=-2(y-F_{t})^{T}\tilde{G}(t)(y-F_{t})$ $$

Now, following the same arguments as in theorem [3.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1), we reach the conclusion for ReLU function as well.
∎

###### Corollary 3.1.2 (Extension to sigmoid activation) .

The above result holds for sigmoid activation as well.

###### Proof.

Applying Jackson’s inequality (see theorem [A.2](https://arxiv.org/html/2509.23887v1#A1.Thmtheorem2)) to sigmoid, we get that there exists a sequence of polynomials $(p_{n}^{\prime})$ such that $||\sigma-p_{n}^{\prime}||_{\infty}\leq\frac{1}{4}$, since $|\sigma^{\prime}|\leq 1/4$, $\pi/2<1$ and $n+1>1$. Note that Jackson’s inequality holds for differentiable functions over a compact domain, but we can always construct the polynomial such that Jackson’s inequality holds in some $[-M,M]$ for $(p_{n}^{\prime})$.

We then define our new sequence of piecewise polynomials $(p_{n})$ that converges to $\sigma$ as follows : $p_{n}=p_{n}^{\prime}$ over $[-M,M]$, $p_{n}(x)=\sigma(M)$ for $x\geq M$ and $p_{n}(x)=\sigma(-M)$ for $x\leq-M$, where $M>0$ is chosen such that $|\sigma(x)-1|<\varepsilon$ for $x\geq M$ and $|\sigma(x)|<\varepsilon$ for $x\leq-M$, for an appropriate $\varepsilon>0$ so as to ensure uniform convergence. This can be done due to the horizontal asymptotes of sigmoid.

Let $F_{n,t},G_{n}(t)$ be the output vector and gram matrix for a network with activation $p_{n}$ at time $t$. Let $F_{t}$ be the output vector for the same network but with sigmoid activation instead of $p_{n}$, at time $t$. Now, note that

$$ $(y-F_{n,t})^{T}G_{n}(t)(y-F_{n,t})\geq(y-F_{n,t})^{T}\tilde{G}(t)(y-F_{n,t})$ $$

where $\tilde{G}(t)$ stands for the gram matrix of any piecewise nonzero polynomial such that it is always at least $1/4$ below $p_{n}$, since $\sigma>0$ which implies $p_{n}<1/4$. For example the piecewise polynomial $q(x):=x-\frac{1}{4}$ for $x\leq\frac{1}{4}$, and $\frac{1}{4}-x$ for $x\geq\frac{1}{4}$.

Since equation [4](https://arxiv.org/html/2509.23887v1#S3.E4) holds for any a.e. differentiable function and $p_{n}$ converges to $\sigma$,

$$ $\displaystyle\derivative{t}\quantity(||y-F_{t}||^{2})$ $\displaystyle=-2(y-F_{t})^{T}G(t)(y-F_{t})=-2\limsup_{n\to\infty}(y-F_{n,t})^{T}G_{n}(t)(y-F_{n,t})$ $\displaystyle\leq-2\limsup_{n\to\infty}(y-F_{n,t})^{T}\tilde{G}(t)(y-F_{n,t})=-2(y-F_{t})^{T}\tilde{G}(t)(y-F_{t})$ $$

Now, following the same arguments as in theorem [3.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1), we reach the conclusion for sigmoid function as well.
∎

We shall now complete the proof of theorem [3.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1) by proving that $G(t)$ remains positive definite throughout training for almost all initializations and input data.

###### Lemma 3.2 (Positive definiteness of NTK) .

$G(t)$ is a positive definite matrix almost surely (with respect to initialization and input data) for all $t\geq 0$.

###### Proof.

We know that $G(t)$ is positive definite iff $\left\{\quantity(\partialderivative{F_{t}}{\theta(t)})_{1},\ldots,\quantity(\partialderivative{F_{t}}{\theta(t)})_{nM}\right\}\subset\mathbb{R}^{P}$ is linearly independent. Note that this set is always linearly dependent if $nM>P$. Thus, we shall assume that $nM\leq P$. We therefore need the function to be overparametrized with respect to the training sample size.

If this set is linearly dependent at $t=0$, then all sub-square matrices of $\left(\quantity(\partialderivative{F_{0}}{\theta(0)})_{1}\ldots\quantity(\partialderivative{F_{0}}{\theta(0)})_{nM}\right)$ $\in\mathbb{R}^{P\times nM}$ will be singular. Note that the determinants of these matrices are piecewise polynomials in the parameters and the dataset and $\sigma$ is zero only at finitely many points, since it is a piecewise nonzero polynomial. Thus, the determinants are nonzero for Lebesgue-almost all values of the parameters $\theta(0)_{1},\ldots,\theta(0)_{P}$ and the datasets $X_{1},\ldots,X_{n}$ (Caron, 2005). Let $\theta_{\text{degen}}$ denote the set of those parameters for which the determinants are zero, union the set of degenerate parameters from the proof of lemma [2.1](https://arxiv.org/html/2509.23887v1#S2.Thmtheorem1). Due to the continuous distribution of initial parameters and input data, we conclude that ${\mathbb{P}}(\theta(0)\in\theta_{\text{degen}})=0$.

We shall now show that ${\mathbb{P}}(\theta(t)\in\theta_{\text{degen}})=0$ for all $t>0$. From lemma [2.1](https://arxiv.org/html/2509.23887v1#S2.Thmtheorem1), we know that the GF solution flow is a piecewise diffeomorphism over the initial conditions space minus a zero measure set. Thus, it maps zero probability sets to zero probability sets (see theorem [A.4](https://arxiv.org/html/2509.23887v1#A1.Thmtheorem4)). Now, we have shown that with respect to initialization and input data, ${\mathbb{P}}(\theta_{\text{degen}})=0$. Thus, from above arguments, ${\mathbb{P}}(\Phi_{-t}(\theta_{\text{degen}}))=0$, which implies that ${\mathbb{P}}(\theta(0)|\ \Phi_{t}(\theta(0))\in\theta_{\text{degen}})={\mathbb{P}}(\theta(t)\in\theta_{\text{degen}})=0$.
∎

###### Remark .

An analogous proof holds for matrix-valued inputs/outputs by flattening the input/output matrix into a vector. We omit the proof for that case for the sake of notational simplicity.

###### Corollary 3.2.1 (Linear convergence of specific architectures) .

Conclusion of theorem [3.1](https://arxiv.org/html/2509.23887v1#S3.Thmtheorem1) holds for the following forms of $f$ (as defined in equation [1](https://arxiv.org/html/2509.23887v1#S2.E1)):

- 1.
(DNN) $f:\mathbb{R}^{N}\times\mathbb{R}^{P}\to\mathbb{R}^{M}$ with $g_{i}:\mathbb{R}^{N_{i-1}}\times\mathbb{R}^{P}\to\mathbb{R}^{N_{i}}$, defined by $g_{i}(X,\theta)=W_{i}X+B_{i}$ for all $X\in\mathbb{R}^{N_{i-1}}$ with $W_{i}\in\mathbb{R}^{N_{i}\times N_{i-1}},B_{i}\in\mathbb{R}^{N_{i}}$, for all $i\in[L]$, $N_{0}=N,N_{L}=M$ and $\theta\in\mathbb{R}^{P}$ denoting the vector of all weights and biases. Over-parametrization requires $P=\sum_{i=1}^{L}N_{i}(N_{i-1}+1)\geq nM$.
- 2.
(ResNet) $f:\mathbb{R}^{N}\times\mathbb{R}^{P}\to\mathbb{R}^{N}$ with $g_{i}:\mathbb{R}^{N}\times\mathbb{R}^{P}\to\mathbb{R}^{N}$, defined by $g_{i}(X,\theta)=X+W_{i}X+B_{i}$ for all $X\in\mathbb{R}^{N}$ with $W_{i}\in\mathbb{R}^{N\times N},B_{i}\in\mathbb{R}^{N}$, for all $i\in[L]$ and $\theta\in\mathbb{R}^{P}$ denoting the vector of all weights and biases. Over-parametrization requires $P=L(N^{2}+N)\geq nN$.
- 3.
(GCN) $f:\mathbb{R}^{m\times N}\times\mathbb{R}^{P}\to\mathbb{R}^{m\times M}$ with $g_{i}:\mathbb{R}^{m\times N_{i-1}}\times\mathbb{R}^{P}\to\mathbb{R}^{m\times N_{i}}$, defined by $g_{i}(X,\theta)=\hat{D}^{1/2}\hat{A}\hat{D}^{1/2}XW_{i}+B_{i}$ for all $X\in\mathbb{R}^{m\times N_{i-1}}$ with $W_{i}\in\mathbb{R}^{N_{i-1}\times N_{i}},B_{i}\in\mathbb{R}^{m\times N_{i}}$, for all $i\in[L]$, $N_{0}=N,N_{L}=M$, $g_{L}:\mathbb{R}^{m\times N_{L-1}}\to\mathbb{R}^{m}$ and $\theta\in\mathbb{R}^{P}$ denoting the vector of all weights and biases. $\hat{A}=A+\mathbb{I}_{m}$, $\hat{D}=\text{diag}\quantity(\sum_{j=1}^{m}\hat{A}_{1j},\ldots,\sum_{j=1}^{m}\hat{A}_{mj})$ where $A$ is the adjacency matrix of the graph over $m$ vertices. Over-parametrization requires $P=\sum_{i=1}^{L}N_{i}(N_{i-1}+m)\geq nmM$.

In each architecture, $\sigma$ can be any piecewise nonzero polynomial activation like leaky ReLU or parametric ReLU, or it can be the commonly used ReLU or sigmoid.

###### Theorem 3.1 (Linear convergence of GF) .

###### Proof.

###### Corollary 3.1.1 (Extension to ReLU activation) .

###### Proof.

###### Corollary 3.1.2 (Extension to sigmoid activation) .

###### Proof.

###### Lemma 3.2 (Positive definiteness of NTK) .

###### Proof.

###### Remark .

###### Corollary 3.2.1 (Linear convergence of specific architectures) .

## 4 Experiments

We use synthetic data to support our findings. For all architectures, data is generated uniformly from a ball in $\mathbb{R}^{N}$ with the $\mathbb{R}^{M}$-valued labels being generated normally. This is done in concordance to Zhang et al. (2017).
We use random synthetic data instead of real-life data to stress-test our theory. Demonstrating that models are capable of fitting to random noise under our over-parametrization scheme provides strong validation of our result.

Leaky ReLU activation (Maas et al., 2013) is used everywhere, with the initialization being the default Kaiming-He Uniform initialization (He et al., 2015). We perform ‘ablation’ by varying the number of parameters from being less than $nM$ to slowly being overparametrized. The training curves verify the hypothesis that as we increase overparametrization, these architectures converge closer to linearly in training loss.

The experimental setups are as follows. To simulate GF, a very small learning rate is taken (lr$=10^{-5}$) with a large number of epochs (10,000). Each model is trained over a training sample of size 1000. For DNN, the input dimension is 500 with output dimension 50 and the model is trained for varying hidden layer sizes - starting from two hidden layers of 50 neurons each ([50,50]) and ending with two hidden layers of 250 neurons each ([250,250]). For ResNet, the input and output dimensions are 100 and the model is trained for varying number of hidden layers - starting from 8 and ending with 16. We had to remove showing the first 100 epochs of ResNet because the initial error was very large, causing the graph to become incomprehensible - the exponential decay is nevertheless visible. For GCN, the input dimension is 500 with output dimension 50 and the model is trained for varying hidden layer sizes - starting from two hidden layers of 25 neurons each ([25,25]) and ending with two hidden layers of 200 neurons each ([200,200]). The underlying graph used is a $k$-nearest neighbours graph on the input data with $k=10$.

Figure: (a) DNN error curves
Refer to caption: pics/MLP.png

## 5 Conclusion and Future Directions

This paper studies linear convergence of gradient flow to a global minimum almost surely (with respect to initial weights and input distributions) for any function that can be written as a composition of polynomials and piecewise nonzero polynomials, given that the function is sufficiently overparametrized ($P\geq nM$). In particular, we prove linear convergence of GF to a global minimum almost surely for a large variety of neural networks (like DNNs, GCNs, ResNets, etc.) with activations like leaky ReLU, parametric ReLU, ReLU and sigmoid, given that the function is sufficiently overparametrized ($P\geq nM$).

We prove our main result by exploiting properties of the NTK and how its lowest eigenvalue behaves under the operation of the GF ODE. We extend to ReLU and sigmoid by approximating them uniformly with piecewise nonzero polynomials. We shall now present some future directions.

Future works can try to extend this paper’s results more by doing a comparative study of architectures on a case-by-case basis. For example, it has been shown by Shi et al. (2024) that transformers need lesser parameters than DNNs to approximate hierarchial composition models (composed of Korobov/Barron functions), for the same error. Showing that transformers have better convergence properties (in the sense of either weaker assumptions or stronger conclusions) would be an interesting line of work. Moreover, we rely on the loss being the squared loss. This is the most common loss used for regression, but not so much for classification. Future works can try extending this result to other losses like cross-entropy.

## References

- Allen-Zhu et al. [2019a]
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang.
Learning and generalization in overparameterized neural networks,
going beyond two layers.
*Advances in neural information processing systems*, 32,
2019a.
- Allen-Zhu et al. [2019b]
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song.
A convergence theory for deep learning via over-parameterization.
In *International conference on machine learning*, pages
242–252. PMLR, 2019b.
- Arora et al. [2022]
Sanjeev Arora, Zhiyuan Li, and Abhishek Panigrahi.
Understanding gradient descent on the edge of stability in deep
learning.
In *International Conference on Machine Learning*, pages
948–1024. PMLR, 2022.
- Awasthi et al. [2021]
Pranjal Awasthi, Abhimanyu Das, and Sreenivas Gollapudi.
A convergence analysis of gradient descent on graph neural networks.
*Advances in Neural Information Processing Systems*,
34:20385–20397, 2021.
- Bah et al. [2022]
Bubacarr Bah, Holger Rauhut, Ulrich Terstiege, and Michael Westdickenberg.
Learning deep linear neural networks: Riemannian gradient flows and
convergence to global minimizers.
*Information and Inference: A Journal of the IMA*, 11(1):307–353, 2022.
- Boursier et al. [2022]
Etienne Boursier, Loucas Pillaud-Vivien, and Nicolas Flammarion.
Gradient flow dynamics of shallow relu networks for square loss and
orthogonal inputs.
*Advances in Neural Information Processing Systems*,
35:20105–20118, 2022.
- Caron [2005]
Richard Caron.
The zero set of a polynomial.
*10.13140/RG.2.1.4432.8169*, 05 2005.
- Chatterjee [2022]
Sourav Chatterjee.
Convergence of gradient descent for deep neural networks.
*arXiv preprint arXiv:2203.16462*, 2022.
- Chen et al. [2022]
Zhengdao Chen, Eric Vanden-Eijnden, and Joan Bruna.
On feature learning in neural networks with global convergence
guarantees.
*arXiv preprint arXiv:2204.10782*, 2022.
- Chen et al. [2021]
Zixiang Chen, Yuan Cao, Difan Zou, and Quanquan Gu.
How much over-parameterization is sufficient to learn deep relu
networks.
In *International Conference on Learning Representations*, 2021.
- Cheney [1998]
E W Cheney.
*Introduction to Approximation Theory*.
AMS Chelsea Publishing. American Mathematical Society, 2 edition,
1998.
- Chizat and Bach [2018]
Lenaic Chizat and Francis Bach.
On the global convergence of gradient descent for over-parameterized
models using optimal transport.
*Advances in neural information processing systems*, 31, 2018.
- Defferrard et al. [2016]
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst.
Convolutional neural networks on graphs with fast localized spectral
filtering.
*Advances in neural information processing systems*, 29, 2016.
- Ding et al. [2022]
Zhiyan Ding, Shi Chen, Qin Li, and Stephen J Wright.
Overparameterization of deep resnet: zero loss and mean-field
analysis.
*Journal of machine learning research*, 23(48):1–65, 2022.
- Du et al. [2019a]
Simon Du, Jason Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai.
Gradient descent finds global minima of deep neural networks.
In *International conference on machine learning*, pages
1675–1685. PMLR, 2019a.
- Du et al. [2019b]
Simon S Du, Xiyu Zhai, Barnabás Poczós, and Aarti Singh.
Gradient descent provably optimizes over-parameterized neural
networks.
In *International Conference on Learning Representations*,
2019b.
- Elkabetz and Cohen [2021]
Omer Elkabetz and Nadav Cohen.
Continuous vs. discrete optimization of deep neural networks.
*Advances in Neural Information Processing Systems*,
34:4947–4960, 2021.
- Gopalani and Mukherjee [2025]
Pulkit Gopalani and Anirbit Mukherjee.
Global convergence of sgd on two layer neural nets.
*Information and Inference: A Journal of the IMA*, 14(1):iaae035, 2025.
- Gray [2003]
Alfred Gray.
*Tubes*.
Progress in Mathematics. Birkhauser Verlag AG, 2 edition, 2003.
- Hartman [1964]
Philip Hartman.
*Ordinary Differential Equations*.
John Wiley & Sons, Inc., 1964.
- He et al. [2015]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Delving deep into rectifiers: Surpassing human-level performance on
imagenet classification.
In *Proceedings of the IEEE international conference on computer
vision*, pages 1026–1034, 2015.
- He et al. [2016]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition.
In *Proceedings of the IEEE conference on computer vision and
pattern recognition*, pages 770–778, 2016.
- Howard et al. [2017]
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang,
Tobias Weyand, Marco Andreetto, and Hartwig Adam.
Mobilenets: Efficient convolutional neural networks for mobile vision
applications.
*arXiv preprint arXiv:1704.04861*, 2017.
- Hutzenthaler et al. [2023]
Martin Hutzenthaler, Arnulf Jentzen, Katharina Pohl, Adrian Riekert, and Luca
Scarpa.
Convergence proof for stochastic gradient descent in the training of
deep neural networks with relu activation for constant target functions.
*arXiv preprint arXiv:2112.07369*, 2023.
- Jacot et al. [2018]
Arthur Jacot, Franck Gabriel, and Clément Hongler.
Neural tangent kernel: Convergence and generalization in neural
networks.
*Advances in neural information processing systems*, 31, 2018.
- Jin et al. [2017]
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M Kakade, and Michael I Jordan.
How to escape saddle points efficiently.
In *International conference on machine learning*, pages
1724–1732. PMLR, 2017.
- Khamis et al. [2024]
Abdelwahed Khamis, Russell Tsuchida, Mohamed Tarek, Vivien Rolland, and Lars
Petersson.
Scalable optimal transport methods in machine learning: A
contemporary survey.
*IEEE transactions on pattern analysis and machine
intelligence*, 2024.
- Kipf and Welling [2017]
Thomas N Kipf and Max Welling.
Semi-supervised classification with graph convolutional networks.
In *International Conference on Learning Representations*, 2017.
- Krizhevsky et al. [2012]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.
Imagenet classification with deep convolutional neural networks.
*Advances in neural information processing systems*, 25, 2012.
- Kunin et al. [2024]
Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David
Klindt, Andrew Saxe, and Surya Ganguli.
Get rich quick: exact solutions reveal how unbalanced initializations
promote rapid feature learning.
*Advances in Neural Information Processing Systems*,
37:81157–81203, 2024.
- Liu et al. [2022]
Chaoyue Liu, Libin Zhu, and Mikhail Belkin.
Loss landscapes and optimization in over-parameterized non-linear
systems and neural networks.
*Applied and Computational Harmonic Analysis*, 59:85–116, 2022.
- Maas et al. [2013]
Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al.
Rectifier nonlinearities improve neural network acoustic models.
In *Proc. icml*, volume 30, page 3. Atlanta, GA, 2013.
- Min et al. [2023]
Hancheng Min, René Vidal, and Enrique Mallada.
On the convergence of gradient flow on multi-layer linear models.
In *International Conference on Machine Learning*, pages
24850–24887. PMLR, 2023.
- Morris et al. [2019]
Christopher Morris, Martin Ritzert, Matthias Fey, William L Hamilton, Jan Eric
Lenssen, Gaurav Rattan, and Martin Grohe.
Weisfeiler and leman go neural: Higher-order graph neural networks.
In *Proceedings of the AAAI conference on artificial
intelligence*, volume 33, pages 4602–4609, 2019.
- Nguyen and Mondelli [2020]
Quynh N Nguyen and Marco Mondelli.
Global convergence of deep networks with one wide layer followed by
pyramidal topology.
*Advances in Neural Information Processing Systems*,
33:11961–11972, 2020.
- Ronneberger et al. [2015]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In *International Conference on Medical image computing and
computer-assisted intervention*, pages 234–241. Springer, 2015.
- Rudin [2006]
Walter Rudin.
*Real and Complex Analysis*.
McGraw-Hill Education, 3rd edition, 2006.
- Shi et al. [2024]
Zhongjie Shi, Zhiying Fang, and Yuan Cao.
Approximation and estimation capability of vision transformers for
hierarchical compositional models.
*SSRN 4984565*, 2024.
- Taigman et al. [2014]
Yaniv Taigman, Ming Yang, Marc’Aurelio Ranzato, and Lior Wolf.
Deepface: Closing the gap to human-level performance in face
verification.
In *Proceedings of the IEEE conference on computer vision and
pattern recognition*, pages 1701–1708, 2014.
- Tarmoun et al. [2021]
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal.
Understanding the dynamics of gradient flow in overparameterized
linear models.
In *International Conference on Machine Learning*, pages
10153–10161. PMLR, 2021.
- Xu et al. [2025]
Ziqing Xu, Hancheng Min, Salma Tarmoun, Enrique Mallada, and Rene Vidal.
A local polyak-łojasiewicz and descent lemma of gradient descent
for overparametrized linear models.
*Transactions on Machine Learning Research*, 2025.
- Yang and Hu [2020]
Greg Yang and Edward J Hu.
Feature learning in infinite-width neural networks.
*arXiv preprint arXiv:2011.14522*, 2020.
- Zhang et al. [2017]
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.
Understanding deep learning requires rethinking generalization.
In *International Conference on Learning Representations*, 2017.
- Zou et al. [2020]
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu.
Gradient descent optimizes over-parameterized deep relu networks.
*Machine learning*, 109(3):467–492, 2020.

## Appendix A Auxillary Results

###### Theorem A.1 .

The solution curve of the GF equation [2](https://arxiv.org/html/2509.23887v1#S2.E2) reaches a subspace $S$ of codimension $k\geq 2$ for almost no initializations.

###### Proof.

Define the set of bad initializations as

$$ $A:=\{\theta_{0}\in\mathbb{R}^{P}:\ \exists t\in[0,T]\text{ such that }\Phi_{t}(\theta_{0})\in S\}$ (7) $$

We shall show that $\lambda^{P}(A)=0$. Define the earliest time that the curve hits $S$, for $\theta\in A$ by

$$ $t_{0}(\theta):=\inf\{t>0:\ \Phi_{t}(\theta)\in S\}$ (8) $$

Define the $r$-neighbourhood around $S$ for any $r>0$ by

$$ $N_{r}(S):=\{\theta\in\mathbb{R}^{P}:\ ||x-S||<r\}$ (9) $$

Since the flow is Lipschitz continuous in $t$, $||\Phi_{t}(\theta_{0})-\Phi_{q}(\theta_{0})||\leq L|t-q|$ for all $t,q\in[0,T]$. Choose a finite rational grid $Q_{r}$ over $[0,T]$ with spacing $\tau\leq r/L$. Note that we cannot choose a finer spacing as that will allow $\Phi_{q}(\theta)\notin N_{r}(S)$, which will cause a contradiction later on in the proof.

Thus, number of points in $Q_{r}$ is $|Q_{r}|\leq TL/r$. Thus, for all $t\in[0,T]$, we can find $q\in Q_{r}$ such that $||\Phi_{t}(\theta_{0})-\Phi_{q}(\theta_{0})||\leq r$.

Furthermore, since $\Phi_{q}$ is $C^{1}$ almost everywhere, $\lambda^{P}(\Phi_{q}^{-1}(N_{r}(S)))=\int_{N_{r}(S)}|\det D\Phi^{-1}_{q}(y)|dy\leq J\lambda^{P}(N_{r}(S))$ for some constant $J>0$. So, putting all of the above results together,

$$ $\displaystyle\lambda^{P}(A)$ $\displaystyle\leq\sum_{q\in Q_{r}}\lambda^{P}(\Phi_{q}^{-1}(N_{r}(S)))$ (10) $\displaystyle\leq|Q_{r}|J\lambda^{P}(N_{r}(S))$ (11) $\displaystyle={\mathcal{O}}(r^{k-1})$ (12) $$

where last line follows from the fact that volume of $N_{r}(S)$ is ${\mathcal{O}}(r^{k})$ as $r\to 0$, which follows from Weyl’s tube formula (see Gray [2003]). Finally, taking $r\to 0$, we get our conclusion.
∎

###### Theorem A.2 (Jackson’s inequality) .

Let $k\in\mathbb{N}$ and $a<b$. Any $k$-times continuously differentiable function $f:[a,b]\to\mathbb{R}$ can be uniformly approximated by a sequence of polynomials $(p_{n})$ such that

$$ $||f-p_{n}||_{\infty}\leq\frac{\pi}{2}\frac{|f^{(k)}|}{(n+1)^{k}}$ (13) $$

###### Proof.

See Cheney [1998].
∎

###### Theorem A.3 .

Solution of the GF equation [2](https://arxiv.org/html/2509.23887v1#S2.E2) does not blow up in finite time, ie. $||\theta(s)||<\infty$ for all $s\in(0,\infty)$.

###### Proof.

Note that

$$ $\derivative{L(\theta(t))}{t}=\nabla L(\theta(t))^{T}\quantity(\derivative{\theta(t)}{t})=-||\nabla L(\theta(t))||^{2}$ (14) $$

Thus,

$$ $\int_{0}^{s}||\nabla L(\theta(t))||^{2}dt=-\int_{0}^{s}\derivative{L(\theta(t))}{t}dt=L(\theta(0))-L(\theta(s))\leq L(\theta(0))<\infty$ (15) $$

Now,

$$ $||\theta(s)-\theta(0)||=\left|\left|\int_{0}^{s}\derivative{\theta(t)}{t}dt\right|\right|=\left|\left|\int_{0}^{s}\nabla L(\theta(t))dt\right|\right|\leq\int_{0}^{s}||\nabla L(\theta(t))||dt$ (16) $$

Using Cauchy-Schwartz,

$$ $\quantity(\int_{0}^{s}1\cdot||\nabla L(\theta(t))||dt)^{2}\leq s\int_{0}^{s}||\nabla L(\theta(t))||^{2}dt$ (17) $$

Thus,

$$ $||\theta(s)-\theta(0)||^{2}\leq s\int_{0}^{s}||\nabla L(\theta(t))||^{2}dt<\infty$ (18) $$

Thus, $||\theta(s)||<\infty$ for all $s>0$.
∎

###### Theorem A.4 .

Let $f:\mathbb{R}^{P}\to\mathbb{R}^{P}$ be a piecewise differentiable function (with pieces $E_{1},\ldots,E_{p}$ such that $\cup_{i=1}^{p}E_{i}=\mathbb{R}^{P}$). Let $A\subset\mathbb{R}^{P}$ such that $\lambda^{P}(A)=0$. Then, $\lambda^{P}(f(A))=0$.

###### Proof.

We know that $f|_{E_{i}}$ will map zero measure sets to zero measure sets (see Rudin [2006]). Note that $A=A\cap\mathbb{R}^{P}=A\cap(\cup_{i=1}^{p}E_{i})=\cup_{i=1}^{p}(A\cap E_{i})$. Thus, $\lambda^{P}(f(A))\leq\sum_{i=1}^{p}\lambda^{P}(f(A\cap E_{i}))=0$.
∎

###### Theorem A.1 .

###### Proof.

###### Theorem A.2 (Jackson’s inequality) .

###### Proof.

###### Theorem A.3 .

###### Proof.

###### Theorem A.4 .

###### Proof.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: See weaknesses.
soundness: 1
strengths: The paper points to classic results in mathematics that might be of use in giving general and elegant proofs for convergence of optimisation methods in deep learning.
summary: The paper claims to give training error bounds under gradient flow for general deep learning architectures.
weaknesses: The paper's claims are very strong, but the results shown can at best be viewed as uninspiring. A strict reading of the results shows several outright errors. The errors are because of the hand-wavy nature of usage of several terms and several leaps of logic.

This leads to several isses. Details below. I assume output dimesion $M=1$ for simplicity.

1. Theorem 3.1 (excluding the $\lambda_0 > 0$ ) is already well known. e(t) = NTK(t) @ e(t-1) is a well known result from as early as 2017 (Jacot et al.). If the lowest eigen value of NTK(t) can be bounded above zero, a linear convergence is trivial.

2. The lower bound on the smallest eigen value of NTK(t) is the main technical challenge. This is attempted in Lemma 3.2. This does not seem well reasoned however. e.g. Determinant being a piecewise polynomial of the parameters is correct, but why is it non-zero?  $\sigma(t)$ not being zero, does not mean the same as the partial derivative terms in the matrix not being zero. The simplest counter example is that you get derivative terms $\sigma'(t)$ terms in the elements of the Jacobian. So even if $\sigma(t)$ is bounded away from zero, $\sigma'(t) $ could be zero with non-zero probability over init and data. There could be other issues as well. One immediate fix might be to bound the derivative $\sigma'$ also away from zero. But I am not sure even this fix will work. 

3. In Theorem 3.1, what prevents $\lambda_t$ (the minimum eigen value of G(t)) being of the order of 1/t ? In this case, there is no exponential decay  -- $\min_t t \lambda_t $  is a constant. Wiggling around this by saying the upper limit $T$ is fixed, results in a vacuous theorem -- ANY decreasing function $h(t)$ with $h(0)=1$ can be upper bounded  in the domain $[0,T]$ by a decaying exponential $\exp(-\lambda t)$ for some $\lambda > 0$ if $\lambda$ is allowed to depend on $T$. When using constants like $\lambda_0$ in Theorems, either give an explicit formula for it, or make precise what all terms is this constant INDEPENDENT of. Otherwise, most theorems would be vacuous. For this to be a non-vacuous theorem, it should give a lower bound for $\lambda_0$ that is independent of $T, n, M, N, P$ or give an explicit dependence of $\lambda_0$ on these quantities. 

4. Taking Corollary 3.1.1. applied to Theorem 1 means that with probability $1$ training a 2 layer ReLU net, with one hidden neuron in the hidden layer with $N=10, n=10, M=1$  gives exponentially decaying error. This is clearly not true -- all data points that are in the "inactive region" of the initial hidden neuron will NEVER have their error reduced. On average half the data points will be in the inactive region. This example follows all the constraints put in the paper on init and data (absolutely continuous). 

5. Issues with Corollary 3.1.1 proof: What exactly is $\tilde G$ ? Is it simply the pairwise data inner product matrix? That does not depend on training time $t$ however. Anyways, I don't see how both the statements hold: i.e why is $G_\alpha(t) > \tilde G(t) $? ReLU being bounded below by identity does not directly imply this. And how does that implication give the inequality on the Mahalanobis distances? That would require $G_\alpha - \tilde G$ be postive semi-definite -- this was not proved. A more fundamental reason this is wrong is that, the Theorem statement is just not true for ReLU (see counter example in point 3 above).

6. Potentially, you could have a setting where all of these issues vanish. e.g Gaussian data, LeakyReLU, overparameterisation. In this case, I do believe a linear convergence of gradient flow would be reasonable. In this case, however, the analysis is mostly well-known and follows from pre-existing literature. (e.g Arora et al. 2019 On exact computation with infintely wide network, and any classic literature on least squares minimization). No new insight is present. Under this distribution setting and overparameterization, any reasonable algorithm would optimize efficiently, even just linear models with gradient flow. No new insight into architecture, structure or feature learning is available due to this analysis.

7. There also seems to be several other misunderstandings. Resnet is NOT just adding an identity matrix to the initialisation weight matrix. (In Corollary 3.2.1 DNNs become Resnets by replacing W_i with W_i + I ).

8. Minor issue: Equation 5 G(t) is not a concatenation, it is a summation.

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 2
questions: None.
soundness: 1
strengths: 1. The paper aims to tackle an important problem, i.e. the convergence of gradient flow on neural networks. Prior work either require a significant overparameterization, or is restricted to two-layers. The central question in the paper is significant due to its generality and relaxed assumptions.
2. The paper provided results that shows that gradient flow on neural networks with piecewise polynomial activation generates a diffeomorphism over the parameter space.
summary: This paper studies the convergence guarantee of gradient flow when applied to train neural network with general architecture. In particular, the paper considers neural networks that can be written as compositions layer-wise functions and a piecewise polynomial activation, and shows that training the neural network over the MSE loss using gradient flow enjoys a linear convergence with rate depending on the minimum eigenvalue of the neural tangent kernel (NTK). The paper provided concrete examples of the activation functions (ReLU and sigmoid) that falls into the scope of the analysis, and backed up the theory with experimental result that shows the convergence of gradient flow.
weaknesses: 1. The main issue with the paper is its correctness. In particular, in the proof of Lemma 2.1 the paper claims that "Since GF cannot reach such a critical point, we need not worry about this case for the inverse problem". Showing that optimization problems will not be trapped at critical points has been a longstanding difficulty, and in this particular case simply making the claim without a formal proof does not constitutes a valid theoretical result.
2. is positive definite, which is a standard result in NTK literatures. Based on this argument, the paper claims that "(the gradient flow) maps zero probability sets to zero probability sets", thus concluding that the NTK during training with have a zero probability of not being positive definite. There are two issues with this argument. First, the claim that "(the gradient flow) maps zero probability sets to zero probability sets" only shows that $\mathbb{P}(\theta\in \theta_{\text{degen}})$ when $\theta$ follows the same probability distribution as the initialization, which is not the case when $\theta(t)$ is generated by the gradient flow function. Second, the paper only tries to show that with zero probability the eigenvalues of $G(t)$ contains a zero. It does not show that all eigenvalues of $G(t)$ are positive. 
3. The experiment does not given a strong validation for the theory. As in the analysis, the loss convergence should follow a simple rule of $e^{-\lambda_0t}$ where $\lambda_0$ is no smaller than the minimum eigenvalue of the NTK at initialization. The paper should compare the loss convergence curve with the theoretically predicted curve to better support the theoretical analysis.

---

confidence: 5
contribution: 1
initial_rating: 0
presentation: 3
questions: See weaknesses.
soundness: 1
strengths: The paper is well-written and cites a number of references in the literature. I truly appreciate the efforts of authors to abstract proofs in order to extract the core arguments. This is an important contribution to the research community, which is sometimes undervalued.
summary: The paper studies the problem of convergence of gradient flow (GF), which is the small stepsize limit of gradient descent, for general neural network architectures (in fact no argument is specific to neural networks, and the approach applies to any smooth optimization problem). The paper aims at showing exponential convergence of GF for overparameterized problems, i.e., where the number of parameters is no less than the number of data.
weaknesses: The proof of the result is incorrect. The most fundamental issue is that the constant $\lambda_0$ in the exponential decay in the main result depends on the time horizon $T$. This is a critical problem, given that any strictly decreasing function $f(t)$ can be written $e^{-\lambda_0(t) t}$ for some $\lambda_0(t) >0$. This fundamentally undermines the argument of the authors.

This issue is usually solved in the literature by lower-bounding the smallest eigenvalue of the NTK on a ball around initialization, then showing that the trajectory cannot leave this ball. Obtaining this lower-bound is architecture-dependent (and also relies on a so-called lazy initialization), and is the lengthiest part of this kind of proofs. To my knowledge, this step cannot be avoided (although I would be happy to see a simple proof for this step that abstracts away from details of the architecture).

A smaller but still substantial issue is that the non-differentiability of the activation function is not handled appropriately. Even though $\sigma$ is differentiable almost everywhere, its composition with itself might not be (see e.g. Lee et al., 2020), while this property is used by the authors (see e.g. line 226).

Finally, previous works already proved linear convergence with overparameterization that is linear in the sample size, see e.g. Marion et al., 2024, for a proof (which follows the approach outlined above for a class of residual neural networks).

Lee et al. On Correctness of Automatic Differentiation for Non-Differentiable Functions, NeurIPS 2020.

Marion et al. Implicit regularization of deep residual networks towards neural ODEs, ICLR 2024.

---

confidence: 4
contribution: 1
initial_rating: 0
presentation: 2
questions: 1. In eq (1), how does g_i, i >1,  depend on theta? This may become clear later in examples, but this equation seems to be incorrect.
2. In figure 2, what does every piece represent?
soundness: 1
strengths: The topic--to simplify and unify existing gradient flow analysis--is a valid research topic.
summary: This work aims to provide a unified analysis for gradient flow of neural network training. The major goal is to relax the assumptions on parameter size, and model architectures used in known analysis.
weaknesses: My major concern is that this work doesn't prove what it claims to prove. Most tools and arguments are standard in GF analysis of neural nets. The key difference is in line 324-329.

This argument basically explains why the authors can greatly simplify the assumptions by previous works, because it basically just assume the spectral of the grammian always has a positive margin. This assumption 1. makes the analysis trivial, 2. was not explicitly stated in theorems. 

If one does not make the assumption, then the argument in 324-249 itself is incorrect. In asymptotic regime, one needs to first prove that the convergence time is finite, before proving exponential convergence.