Title: Convergence of the Gradient Flow for Shallow ReLU Networks on Weakly Interacting Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: Shallow Neural Networks

Gradient Flow

High-dimension

Convergence rate
Abstract: We analyse the convergence of one-hidden-layer ReLU networks trained by gradient flow on $n$ data points. Our main contribution leverages the high dimensionality of the ambient space, which implies low correlation of the input samples, to demonstrate that a network with width of order $\log(n)$ neurons suffices for global convergence with high probability. Our analysis uses a Polyak–Łojasiewicz viewpoint along the gradient-flow trajectory, which provides an exponential rate of convergence of $\frac{1}{n}$. When the data are exactly orthogonal, we give further refined characterizations of the convergence speed, proving its asymptotic behavior lies between the orders $\frac{1}{n}$ and $\frac{1}{\sqrt{n}}$, and exhibiting a phase-transition phenomenon in the convergence rate, during which it evolves from the lower bound to the upper, and in a relative time of order $\frac{1}{\log(n)}$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper establishes convergence guarantees for one-hidden-layer ReLU Networks, trained on high- dimensional data. Based on gradient flow, a continuous-time version of gradient descent, they analyse\nthe local Polyak-Lojasiewicz (PL) curvature of the training dynamics. Establishing a lover bound of\nthe local PL-curvature, they show that for weakly-correlated inputs, at least 4 log(n/ε) hidden-layer\nneurons and under some initialization constraints (the second layer weights have larger or equal norm\nthan the weights from the first layer), the loss converges to zero with exponential rate of at least 1/n\nwith probability 1−ε. Considering the orthogonal case in particular, they can establish an upper\nbound, ensuring a rate of convergence between 1/n and 1/√n and show empirically that a rate of\n1/√n is met with high probability. Lastly they show that in the case of group initialization, there\nare abrupt phase transitions between an initial convergence rate of 1/n, intermediate rates, and the\nasymptotic convergence rate 1/√n.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors address an important topic of convergence dynamics of shallow ReLU networks in more\nrealistic conditions than in previous works. They explain their results in details. Especially giving a sketch of proof for the main results makes it a lot more understandable and intuitive. Further, by giving an example of a setting in which their assumptions are\nsatisfied, or explaining the implications of their results, they well illustrate their work and it’s relevance.\n\nQUESTIONS:\n1- I am a bit confused with the rates in Theorem 1: in my understanding, the rates slow down, when n increases, and when the norm of data grows fast (role of C);  can the authors elaborate more about these points? \n\n2-In line 64--66, authors mentioned that ``...to understand a simplified version of the optimization dynamics of this neural network, we study the continuous-time limit of gradient descent.'' Could authors please elaborate why and how this setting and the corresponding results can be adapted to discrete time setting which happens in practice? \n\n3-I am confused with the role of p in equation 3? Could authors please elaborate more on that?\n\n\n4. Also, according to results in the literature, local minima exist in shallow ReLU networks. So, what is the main message of the paper? Is it claiming that such suboptimal points do not exist, or that the optimization process escapes them?\n\nLIMITATIONS:\nNo limitation", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence rate of two-layers ReLU networks on \"almost\" orthogonal data. The authors first show a convergence rate of $\\Omega(1/n),$ for the weak-correlated data case, where the correlation of the data points are of order $O(1/\\sqrt{n}).$ Then the authors show the upper bound of convergence rate is of order $\\O(1/\\sqrt{n})$ for completely orthogonal data, and $O(1/n^\\alpha)$ for any $alpha \\in [1/2, 1]$ under extra initialization assumptions. Finally, the authors show the different stages of training under the settings with completely orthogonal data and the initialization assumptions.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n\n1. The paper proved the convergence rate of two-layer ReLU networks on \"almost\" orthogonal data, with $O(\\log(n))$ neurons, I think it's an interesting result.\n\n2. The paper proved the convergence rate by lower-bounding the local-PL curvature, which looks interesting and could technically benefit future works.\n\n3. The paper is in general well-written and is easy to follow the main points. The definition of $r_i(t)$ in Theorem 1 is not clear enough though, I suggest the authors define it properly.\n\n\n### Weaknesses\n\n1. This paper only discussed the convergence rates of the two-layer ReLU networks, while I believe the results are new and interesting, I'm not sure how significant are the results. I wonder if there are further insights, such as the implicit bias of the  weights, can be induced from the results?\n\n2. The results in Proposition 2 which requires the group initialization are a bit restrictive. In particular, the group initialization requires each neuron to be active with only one different group of data points. This seems to mean that: (1) the weights are already learned some non-trivial information of the data, (2)  since all the data are orthogonal, the weights are moving in different subspaces, and the training of the networks are equivalent to training $p_n$ independent $1$-neuron ReLU network on different datasets.\n\n3. There are also a few other assumptions that limit the scope of the results, such as the requirements of large $d,$  special initialization of $|a_j(0)| \\geq \\|w_j(0)\\|_2,$ and completely orthogonal data for faster rates. However, I believe those assumptions may reflect the broader challenge in developing rigorous theoretical descriptions of neural networks, and I do not consider them as major weaknesses.\n\nQUESTIONS:\n1. Please also refer to the Strengths and Weaknesses part.\n\n2. For the results in Proposition 2, what are the technical difficulties that require the completely  orthogonal data? \n\n3. For the group initialization, is it possible to prove it for any reasonable initialization of the weights? For example, random initialization with small enough variance.\n\nLIMITATIONS:\nThere's no potential negative societal impact of this work in my opinion. The authors discuss the limitations in the checklist.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the convergence properties of one-hidden-layer ReLU networks trained via gradient flow on finite datasets. The main contribution shows that when input data have low correlations (which occurs naturally in high dimensions when d ≳ n²), networks with only O(log n) neurons can achieve global convergence with high probability, regardless of initialization scale. The analysis employs a Polyak-Łojasiewicz (PL) framework to characterize convergence rates, proving exponential convergence at rate 1/n in the general case. For orthogonal data, the authors provide refined analysis showing convergence rates between 1/n and 1/√n, and demonstrate phase transition phenomena in the convergence dynamics. The theoretical results are supported by numerical experiments that validate the predicted scaling behaviors.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. **Novel theoretical contribution**: The paper identifies a new regime for neural network convergence that doesn't rely on either the lazy training regime (NTK) or infinite-width assumptions (mean-field). The requirement of only O(log n) neurons is particularly striking and practically relevant.\n\n2. **Rigorous mathematical analysis**: The use of trajectory-wise Polyak-Łojasiewicz analysis is sophisticated and provides sharp convergence rate characterizations. The proofs appear technically sound and the bounds are non-asymptotic.\n\n3. **Natural assumptions**: Unlike previous work requiring carefully tuned initialization scales, the low-correlation assumption (Assumption 2) arises naturally in high dimensions, making the results more broadly applicable.\n\n4. **Comprehensive analysis**: The paper provides both general results and refined analysis for the orthogonal case, including interesting phase transition phenomena that add theoretical depth.\n\n5. **Experimental validation**: The numerical experiments effectively support the theoretical predictions, particularly regarding scaling laws and convergence thresholds.\n\nWeaknesses:\n1. **Restrictive dimensionality requirement**: The condition d ≳ n² is quite strong and may limit practical applicability. While the authors show this leads to √d data points on average, this is still restrictive compared to typical machine learning scenarios where n >> d.\n\n2. **Asymmetric initialization assumption**: Assumption 1 requiring |aj(0)| ≥ ||wj(0)|| is somewhat artificial and doesn't align with standard initialization schemes (as the authors acknowledge). While they provide a counterexample when this fails, the necessity of this assumption is questionable.\n\n3. **Limited scope**: The analysis is restricted to one-hidden-layer networks with square loss. Extension to deeper networks or other loss functions is unclear, limiting the broader impact.\n\n4. **Gap between theory and practice**: The orthogonal data assumption, while mathematically convenient, is unrealistic. The general case requires very high dimensions, creating a disconnect from practical scenarios.\n\n5. **Experimental limitations**: The experiments are conducted on relatively small scales (d=100-2000, n≤3500) and don't fully explore the boundary cases where the theory might break down.\n\nQUESTIONS:\n1. **Necessity of assumptions**: How critical is the asymmetric initialization assumption (Assumption 1)? Can you provide more insight into whether this is merely sufficient or actually necessary? Have you explored alternative initialization schemes that might work?\n\n2. **Scaling improvements**: Your experiments suggest the convergence threshold scales as N(d,p) ≃ C(p)d rather than the theoretical √d. Can you comment on the gap between theory and experiments? Is there potential to improve the theoretical bound to linear scaling in d?\n\n3. **Extension to deeper networks**: How might your analysis extend to networks with more than one hidden layer? Are there fundamental obstacles, or is it primarily a technical challenge?\n\n4. **Practical implications**: Given the restrictive dimensionality requirements, can you discuss scenarios where your results might be practically relevant? Are there applications where d >> n² naturally occurs?\n\n5. **Comparison with existing methods**: How does the performance of networks in your regime compare to other approaches (e.g., kernel methods, linear models) that might be more natural in the high-dimensional, low-sample regime?\n\nLIMITATIONS:\nThe authors adequately acknowledge several key limitations:\n\n1. The restriction to shallow networks and square loss\n2. The artificial nature of the asymmetric initialization\n3. The high-dimensional requirements\n\n\nHowever, they could better address:\n\n1. **Computational considerations**: Training in extremely high dimensions (d ≳ n²) has significant computational costs that aren't discussed\n2. **Statistical considerations**: In the regime where d >> n, overfitting becomes a major concern that the paper doesn't address\n3. **Broader applicability**: More discussion of when the assumptions might be realistic in practice would be valuable\n\nThe limitations discussion is reasonable but could be more comprehensive regarding practical deployment considerations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the training dynamics of a two-layer ReLU neural network in a mean-field setup. When the data matrix is low-correlated (Assumption 2), it is shown that the gradient flow with the regression loss converges to the interpolating solution with order $O(\\exp(-t/n))$, where t is the training time and n is the number of samples.  They confirm that Assumption 2 holds when the data dimensionality d satisfies $d\\gtrsim n^2$ with high probability. Moreover, for orthogonal data, they provide a more detailed analysis of the convergence speed and conclude that a phase transition exists with respect to the number of neurons, where the convergence speed changes from $1/n$ to $1/\\sqrt{n}$.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n- The authors address the challenge of theoretically understanding deep learning training, a significant issue in the field of deep learning. The global convergence analysis using the PL-curvature in the mildly overparameterized mean-field setup is a novel approach, and it achieves a novel global convergence guarantee.\n- This paper is well-written. All theoretical results seem solid.\n\n**Weakness**\n- One significant limitation of the results is that Corollary 1 requires a huge dimensionality of $d\\gtrsim n^2$ (as also mentioned by the authors), which will be violated in practical situations. Can this condition be mitigated by considering a wider network in the same theoretical framework? While Theorem 1 only requires the network width of $O(\\log n)$, I wonder if this narrow width restricts the convergence performance of the network. \n- The group initialization defined in section 4 seems quite restrictive. Can this setting be applied to the case where we employ the Gaussian initialization?\n- The presentation of Theorem 2 seems confusing. For example, the sentence “the transitions happen at each time $t^j=...$”. Moreover, what is the motivation for comparing $t_n^j(\\epsilon)$ and $t_n^j(1-\\epsilon)$? Perhaps I have any misunderstandings about this part, so it would be helpful if these points could be clarified. \n- In Theorem 1, the formal definition of $r_i(t)$ is lacking in the main paper.\n\nQUESTIONS:\nBesides what I raised above, I have the following questions:\n- Can the framework based on PL curvature be applied to other activation functions? It appears that the lower bound on the curvature (7) is crucial for achieving global convergence, which may be extended by replacing the indicator function with the derivative of other activations. \n- What is the primary motivation for employing the mean-field scaling? Since Theorem 1 only requires $p=O(log n)$, the significant difference between the NTK regime and the mean-field scaling may not occur during training. \n- Can the framework be extended to the multi-layer networks (for example, by considering the PL curvature with respect to the output layer weights)?\n\nLIMITATIONS:\nThe authors properly address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}