Title: SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
Venue: NeurIPS 2025
Accepted: yes
Keywords: theory of deep learning

gradient flow

o-minimal structures

GELU function

optimization

loss landscapes

generalized critical values

divergence
Abstract: We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value.	Moreover, we prove the existence of a threshold $\varepsilon>0$ such that the loss value of any gradient flow initialized at most $\varepsilon$ above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn Section 2, the authors recall definitions of o-minimal structures (families of subsets satisfying certain algebraic properties) and functions that are definable with respect to such structures, as well as identify many examples of activation functions which are definable. The authors conclude the section by stating their dichotomy theorem (Theorem 2.8): a network with definable activations, optimized using gradient flow to fit a definable target function using a definable loss on either an empirical or absolutely continuous input measure, must either converge to a critical point of the loss or diverge with loss approaching an asymptotic critical value.\n\nIn Section 3, the authors consider a special case of definable functions, namely sublinear analytic definable (SAD), which are exemplified by typical activations and losses. They state their Theorem 3.4, which says that there are no global minima for the problem of fitting a network with SAD activations to a polynomial of degree at least 2 on a sufficiently large dataset. Theorem 3.4 is counterbalanced by Theorem 3.5, which says that any polynomial can be approximated arbitrarily well by a sufficiently large (with respect to the degree of the polynomial) neural network on compact sets. The results are synthesised into the main result of the paper, Corollary 3.6, which states that in fitting a sufficiently large neural network to a polynomial of degree 2 using gradient flow on a sufficiently large dataset starting from a point with sufficiently small loss, the parameters necessarily diverge.\n\nIn Section 4, the authors provide some numerical support for their divergence theorem.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - The paper is exceptionally well-written and easy to read.\n - The approach taken in this paper of using abstract, formal properties to prove theorems is underutilised in the literature. As a consequence, the theorems are relatively general in their assumptions compared to most other papers on the optimisation of deep networks.\n\nWeaknesses:\n - The paper lacks any \"idea of proof\". It is not clear at all in the paper how the assumptions made are used in the derivation of the theorems. The proofs in the appendix, while complete as far as I can tell, also seem to lack a high level idea of the proof.\n - There is not much specific said about the relation of this work to prior works. In particular, as someone with little knowledge about the employment of o-minimality in optimisation, it is difficult for me to judge the novelty of the work.\n - It is not made clear by the authors why the result is significant.\n\nQUESTIONS:\nI am willing to raise my score to an accept if the authors can answer the following questions.\n1. Why is the result significant, and how is it a significant advance over prior work in this area?\n2. In a couple of paragraphs, can you summarise how the assumptions made imply the result?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper discusses gradient flows (GF) for neural networks with definable activations and loss functions. First, using Kurdyka-Lojasiewicz inequality, the author gives a dichotomy of convergence of GF for such neural networks. This is basically a summary of existing results. Second, the author studies  fitting polynomials (usually with degree $\\ge 2$) with SAD networks. He/she shows that for sufficiently large model and sufficiently large sample size, we have 1) 0 is an asymptotic critical value of the loss, 2) 0 is not a critical value of the loss, and thus 3) GF diverges to infinity. Experiments are presented and limitations of the works are discussed.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1.  The paper has very clear structure and provides sufficient details for readers. \n2.  The paper introduces a useful tool, i.e., theory in o-minimal structure, to the study of machine learning.   I believe that this tool greatly helps us understand the behaviors of neural networks. \n3.  The results shown in the paper, especially the divergence of GF for fitting polynomials, captures, and explains the phenomenon in application in an unambiguous way, meanwhile having sufficient generality in mathematics.\n\nWeakness: \n1.  My major concern is that the experiments are performed using SGD and Adam, which usually differ a lot from GF/GD. \n2.  My second concern is about contribution. The authors mainly discuss the dichotomy of GF (see 1.1.Contribution) in analytic setting. But when the loss is analytic, this behavior of GF has been proven by previous works, such as [1]. And even for general definable functions, it follows quickly from KL inequality. So it might not be appropriate to put it as a contribution of this paper. \n3. I found some typos. See below. \n\n[1] P. Absil, R. Mahony, B. Andrew, “Convergence of the Iterates of Descent Methods for Analytic Cost Functions”, SIAM Journal on Optimization, 2005, 16(2), pp. 531-547.\n\nQUESTIONS:\n1. Is it Okay to mention someone’s real name in the paper?? \n2. Even if GD is not used widely in practice, I suggest performing experiments using GD. In fact, this is necessary based on the authors’ plan for the paper, as experiments “are intended as a proof of concept”, mentioned in 1.3. \n3. Some typos: \n    Typo at line 188-189: We write $d\\mu(x) = p(x) dx$ if $\\mu$ is given by $\\mu(A) = \\int_A p(x) dx$ \n    Typo at line 697: should mention that the $p_0, p_1, …$, etc. are homogeneous polynomials. \n    Typo at line 827: should be $T_n$ instead of $T^n$ in equation (20).\n\nLIMITATIONS:\nYes. Discussed in the article.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submitted manuscript investigates gradient flows in the context of training fully connected neural networks. The first contribution establishes that, under a local Lipschitz continuity assumption on the gradient of the loss function, the trajectory of the gradient flow either converges to a critical point or diverges. In the divergent case, however, the value of the loss function converges to an asymptotic critical value.\n\nThe second main contribution provides a divergence result for gradient flows. Specifically, in the setting of subanalytic definable neural networks, the authors show that there exists no finite parameter configuration at which the loss achieves zero. Nonetheless, there exists a diverging sequence of parameters along which the loss converges to zero. Thus, while the global infimum of the loss is zero, it is only attained in the limit at infinity.\n\nAs a result, the authors argue that in such settings, whenever the loss at a parameter configuration is sufficiently small, the trajectory of the gradient flow necessarily diverges, yet still minimizes the loss. These theoretical findings are supported by two numerical experiments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper extends standard convergence results to stationary points from the setting of global smoothness to one of local smoothness.\n- The convergence analysis presented applies to the training of neural networks with a broad class of activation functions\n- Under the SAD setting the authors, the authors establish a neural network approximation result for polynomials, showing that neural networks can approximate such functions arbitrarily well.\n- By combining these results, the authors demonstrate that while the gradient flow diverges, the loss function along the trajectory converges to zero.\n\n**Weakness:**\n- The description of Theorem 2.8 at certain points slightly overstates the strength of the result. While it is true that most convergence proofs rely on global smoothness assumptions, and it is indeed valuable to see results under local smoothness, the focus here is restricted to gradient flow, i.e., the continuous-time setting. Thus, the resulting statement is expected. The continuous-time limit significantly simplifies the analysis, as it avoids the complexities associated with discrete-time methods—particularly, the need to carefully manage step sizes to ensure descent. Such considerations are inherently bypassed in the continuous-time framework, which limits the practical implications for gradient-based optimization algorithms.\n- I do not fully agree with the interpretation of Figures 1 and 2. Based on the presented plots, it is not clear that the norm of the neural network parameters actually diverges; instead, they appear to approach an upper bound. This observation directly supports the concern raised in my previous point: in the discrete-time setting, there is no guarantee of monotonic decrease in the loss function, and divergence is not automatic. A more convincing evaluation would require running the optimization for more iterations to reveal any long-term trend. Additionally, adjusting the scale of the y-axis could improve the clarity and interpretability of the plots. Moreover, it would be helpful to include a plot of the gradient norm as well.\n\nTo summarize, I believe the true highlights of the paper are Theorems 3.4 and 3.5. That said, approximation theory for neural networks lies outside my primary area of expertise, so I cannot fully assess the novelty or practical impact of these results. In particular, the authors have not situated their findings within the broader literature on neural‐network approximation theory (beyond the standard universal‐approximation theorems).\n\nQUESTIONS:\n- How does the convergence result for the gradient flow in this paper compare to other results in the literature. For example, how does it compare to that of Dello Schiavo and Maas (Local Conditions for Global Convergence of Gradient Flows and Proximal Point Sequences in Metric Spaces, Transactions of the American Mathematical Society, 2024)? \n- In Appendix D, it is mentioned that the proof is partially inspired by [28]. Could you elaborate on this connection? Specifically, what aspects of the argument are derived from [28], and what constitutes the novel contribution in the presented approximation result?\n- How do Theorem 3.4 and Theorem 3.5 connect to other findings of neural network approximation results?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates gradient flows for loss landscapes of fully connected neural networks trained on polynomial targets with commonly used activation functions. The main theoretical result is a dichotomy: gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value, with the latter occurring for polynomial targets under mild assumptions. The key proof relies on the geometry of o-minimal structures from model theory and introduces SAD (Sublinear Analytic Definable) activation functions. Empirically, the authors run simulations on toy datasets and extend to more realistic tasks (e.g., MNIST) to demonstrate this phenomenon / main theoretical results.\n\nSTRENGTHS AND WEAKNESSES:\nNote: I don’t have the necessary background to evaluate the correctness/significance of this work, hence my confidence score is 2. I will communicate with other reviewers to address any misunderstanding.\n\nStrengths:\n- The paper is generally well-written, and the source code is provided for reproducibility.\n- A detailed experimental setup is provided in Appendix E.\n- While I cannot comment on the correctness/significance of the theoretical derivations, the derivations and proofs are well organized (e.g., Figure 4).\n\nWeaknesses:\n- The empirical results don't convincingly demonstrate that the theoretical findings translate into practical settings. For example, it is well known that parameter norms continue to increase during training (motivating techniques like L2 regularization), making it difficult to understand whether the theoretical insights are specifically captured in Figure 1. Does this phenomenon fail to occur with extremely poor initialization or with activation functions such as ReLU (as predicted by theory)? Please see the questions section below.\n- Similar concerns apply to the empirical results presented in Section 4.2. At the moment, the theoretical analysis does not appear to capture any important practical considerations that aren't already well-understood.\n- The theoretical results apply only to gradient flows and smooth activation functions, which significantly reduces the practical relevance.\n\nMinor:\n- It would be helpful to have detailed related work, e.g., other applications of o-minimality to neural networks.\n\nQUESTIONS:\nI am mostly curious if the theoretical analysis suggests any particular insights for the simulation results in Section 4. I'm listing several questions I have below.\n- In line 292, the authors mention that Adam shows better convergence, while the SGD results in Figure 1 seem to indicate that the parameter norm plateaus. Could the authors elaborate on their justification in lines 292-298? I assume that SGD with a small learning rate is closer to the theoretical analysis (gradient flow) than Adam updates.\n- Does this behavior still occur with extremely poor initialization or with other activation functions (e.g., Relu or even identity)? I speculate that they will show the same behavior as in Figure 2. If so, what is the practical significance of the proposed work?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-02"}