SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
Authors: Julian Kranz
Davide Gallon
Steffen Dereich
Arnulf Jentzen
Venue: NeurIPS 2025 poster ()
Keywords: theory of deep learning
gradient flow
o-minimal structures
GELU function
optimization
loss landscapes
generalized critical values
divergence
Primary area: theory
TL;DR: We prove divergence results for gradient flows for deep neural networks with analytic activation and polynomial target functions.

Abstract: We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value.	Moreover, we prove the existence of a threshold $\varepsilon>0$ such that the loss value of any gradient flow initialized at most $\varepsilon$ above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.

SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures
Julian Kranz
Davide Gallon
Steffen Dereich
Arnulf Jentzen
DDepartment Information Systems,
University of Münster, Germany, email: jul ian. kranzoumi monni--uenaa ter .de
Applied Mathematics: Institute for Analysis and Numerics, University Münster, Germany, email: jul ian. kranzduni -menn ter
AApllie Mathematics: Institute for Analysis and Numerics,
University of Münster, Germany, email: davide. grriide.gallln lallgaalllnni onduni muenster de
"RiskLab Switzerland, ETH Zürich, Switzerland, email: dgallon@ethz
"Applied Mathematics: Institute for Mathematical Stochastics.
-menster. de School of Data Science and School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen (CUHK- Shenzhen), China, email: aj entzennecuk. edu, en AApllied Mathematics: Institute for Analysis and Numerics, University of Münster, Germany, email: ajentzenduni pratzzenduiimmensteer
University of Münster, Germany, email: steffen.
Abstract
We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges critical point diverges infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence threshold æ such that the loss value of any gradient flow initialized most above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.
Introduction
The success of deep learning in practice has outpaced our theoretical understanding of neural network training dynamics. While gradient-based optimization methods such as (stochastic) gradient descent and its variants [77, 75, 43, 35, 57] - routinely achieve near-zero training loss on complex tasks, the theoretical principles governing their convergence remain poorly understood, particularly realistic, non-convex settings. Even for the mathematically more tractable gradient flow, which can be viewed as the continuous time limit of gradient descent, a fundamental question persists:
Under what conditions does the gradient flow converge to good local minimum?!
Note that one cannot always expect convergence to global minima [50, 29, 30], but rather local minima with low risk level [46, 40, 41].
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent years have seen significant progress in the theory of deep learning, with convergence analyses
often relying on strong assumptions, such as convexity [11, 22]. overparameeterizaton [34, 2], specific initialization schemes [48], or Eojasiewicz inequalities [25, 24, 11. Moreover, most of these works either assume or imply that the iterates of the optimization scheme or the trajectory of the gradient flow stay within bounded region. At the same time, an increasing body of work shows that this assumption is not always satisfied, with the simplest example being a linear classifier with logistic activation [70, 67, 78, 39].
1.1 Contributions
In this work, we bridge the gap outlined above by observing a general dichotomy: For large class of smooth activation functions (such as the logistic, hyperbolic tangent, softplus or GELU function [42] which has been used in BERT [27] and GPT-3 [12]) and loss functions (such as mean squared error or binary cross entropy), the gradient flow either
converges to a critical point, or
diverges to infinity with the loss converging to a generalized critical value [76].
Using finiteness of the set of generalized critical values [17], we deduce the existence of a threshold
E > 0 such that any gradient flow initialized within e-distance to the optimal level necessarily converges to it. We refer to Theorem 2.8 for a mathematically rigorous statement of our dichotomy result. We emphasize that convergence of gradient methods to critical points under boundedness assumptions is well-understood phenomenon [10, 3, 4, 19, 25, 8, 52, 45J. Our main goal in stating the above dichotomy is to make the results from [17] accessible to the deep learning community as the existence of the threshold
seems to be unnoticed in the deep learning literature.
As a concrete application of this dichotomy, we complement the divergence results obtained in [70, 67, 78, 39] by a general divergence result for polynomial target functions: For polynomial target function p: IR" + IR' of degree at least two, a sufficiently big neural network architecture and sufficiently big training dataset, prove that the loss has global minimum, but infimum zero. In particular, the optimal neural network parameters can only be achieved asymptotically. Thus, by the dichotomy result, the gradient flow diverges whenever the parameters level below s. We refer to Theorem 3.4, Theorem 3.5 and Corollary 3.6 for a mathematically rigorous statement of our divergence result. The "infimum zero" part of our result is essentially well known and a standard ingredient in proofs of the universal approximation theorem [74, 9, 28, 21, 70]. Our novel contribution here the "no global minimum" part and the resulting divergence phenomenon for the gradient flow.
initialized at loss
Our analysis applies to fully connected feedforward deep neural networks with multiple input and output dimensions and a general class of activation functions including the logistic, hyperbolic tangent, softplus and GELU function. Moreover, we can treat both the average loss finite dataset and the true loss with respect to a sufficiently smooth probability density. This level of generality arises because our analysis builds on bootstrapping abstract properties rather than concrete computations:
The functions appearing in our analysis are definable in an o-minimal structure in the sense of mathematical model theory [31], which intuitively means that they can be defined using first order statements involving well-behaved operations such as +, -.. +, exponentials. logarithms, derivatives and anti-derivatives. Definable functions have strong rigidity and finiteness properties such as the uniform finiteness theorem [15], and definable dimension theory [31]. Our proofs use these techniques in a central and novel way.
We isolate the general class of Sublinear Analytic Definable (SAD) functions (see Defini- tion 3.1 and Example 3.2) for which our results hold. SAD functions have good permanence properties ensuring that neural networks with SAD activation function are themselves SAD. The main idea of proving the absence of global minima in our divergence result is that the sublinearity of a SAD neural network prevents it from being globally identical to polynomial of degree at least two, while the analyticity and definability ensure enough rigidity to detect this property on a finite dataset (or compactly supported distribution). On the other hand, analyticity guarantees a sufficient supply of non-trivial Taylor coefficients, allowing polynomials to be approximated arbitrarily well. We believe that our definition of

---

SAD activation functions provides convenient theoretical framework for future research on smooth neural networks.
Building on the theoretical framework established earlier, validate our findings by training neural networks on polynomial target functions using optimization methods such as gradient descent and Adam [57]. The results closely align with the behavior predicted by our gradient flow analysis. We then extend our investigation to more complex and more practically relevant scenarios: numerical solution of PDEs using the Deep Kolmogorov Method [S] and image classification of the MNIST dataset [63]. Across all these experiments, we consistently observe a growth in the norm of the neural network parameters as the loss diminishes. These findings suggest that the divergence phenomenon we analyze actually holds in much broader generality. We include the source code for our numerical simulations at https://github.. http://ggiuuub.co//eeepc.... com//eeplammingmeeooo...o
The paper is structured as follows: We end this introduction by briefly summarizing related work and limitations of ours below. We introduce the necessary mathematical background and formally state the dichotomy result in Section 2. In Section 3, we develop the divergence result for polynomial target functions. Our numerical experiments are reviewed in Section 4. We conclude in Section 5. The proof's of our main results are given in the Appendix.
1.2 Related work
The general theory of o-minimal structures originates from [72, 58, 73] and is introduced accessibly in [31, 15, 81]. We specifically use o-minimality of Pfaffian functions [80], Kurdyka's 's Lojasiewicz inequality [60], definable dimension theory [31], the uniform finiteness theorem [15] and finiteness
of the set of generalized critical values [17]. Applications of o-minimal structures to optimization are surveyed in [47]. There is an increasing body of work on the application of o-minimal structures to convergence results for gradient flows and stochastic gradient descent (SGD) under stability assumptions (assuming that the parameters stay within a bounded region) [10, 3, 4, 19, 25, 8, 52, 45]. Conditions ensuring stability are given in [53, 62]. In the unstable regime, there are divergence results which consider homogeneous networks [67, 78]. very specific regression problems [39], or use convergence the loss and absence of global minimizers as an assumption [70]. Other applications of o-minimality to neural networks given in [55, 56, 54]. Neural network approximation results such as the universal approximation theorem [16, 37, 44, 64] are surveyed in [28, 7]. Our Theorem 3.5 can be extracted from existing literature. The proof in [21] for tanh for instance generalizes to setting. Further neural network approximation results are given in [82, 71, 9, 20, 69, 70].
1.3 Limitations and outlook
While we prove powerful results for gradient flows, our methods are currently not capable of proving analogous results for (stochastic) gradient descent. Although results exist under stability assumptions (see Subsection 1.2 above), the main limitation in the unstable regime is our lack of control over the Hessian. To mitigate this, one could try to prove that the loss function satisfies generalized smoothness condition in the sense of [83, 65] (at least along the gradient trajectory) and use this to employ non-convex convergence proof for SGD. In light of the empirical evidence provided in [83] this approach seems quite promising and will thus be pursued in future work.
Although do provide empirical evidence for similar results for gradient descent and other opti- mizers in Section 4, our numerical results are fairly limited by the variety and size of the considered datasets. Since the main focus of our paper are the mathematical theorems, these experiments intended as a proof of concept rather than an in-depth experimental analysis.
Another limitation of our analysis is that the considered activation functions are required to be at least C² or, in some cases, analytic. In particular, our analysis does not apply to the ReLU activation function. In fact our divergence result for polynomial target functions does not hold for the ReLU function since in this case, global minima exist for shallow networks [49, 26, 23]. Moreover, global minima always exist if one adds an L Ie-reuularization term to the loss function. For the same reason, divergence does not occur if one uses an optimizer with weight decay such as Adam [66].
In several neural network training scenarios with diverging model parameters, one can prove that the normalized parameters converge, see [51, 78, 59]. This phenomenon known as directional convergence is closely related to the gradient conjecture at infinity [18] and can be deduced for
3

---

Cf-functions definable in polynomially bounded o- o-minimal structure as consequence of [61]. However, the case of o-minimal structures containing the exponential function which is relevant to our setting is still open [18] and remains to be explored.
2 A dichotomy for definable gradient flows
In this section, we present our first result (see Theorem 2.8 below) which states that for arbitrary deep neural networks and most of the commonly used loss functions and activation functions (sce Example 2.4 below), any gradient flow associated to the training loss landscape either converges to critical point or diverges
generalized critical value at infinity. The main conceptual reason for this phenomenon is that all the occuring functions are definable in an o-minimal structure in the sense of model theory. We recall the necessary mathematical background below:
Definition 2.1 (O-minimal structure). A structure S (expanding the real closed field R) is collection of subsets Sn C P(R") for all n E N satisfying the following properties:
(i For A, Sn, we have AUB, A n B, R" AA
(ii) For E Sn and B E Sm, we have x Sa+m
(iii) For A S" and m R" RT coordinate projection with m S n, we have a(A) Sm.
(iv) For each polynomial p E [R[ X,,), we have {ar E IR" p(x) > 0} E S and
{x R" ppz)) = 0} E Sa.
structure called o-minimal if all the elements of S₁ are finite unions points and intervals.
If S is o-minimal structure, call the sets A E Sn, the definable sets. A function f: IR" -> R called definable, if its graph f(r) x E [R"} is definable. no o-minimal structure is specified, refer to or function as definable if is definable in some o-minimal structure. An instructive (and in fact universal) example of a structure is the one generated by a set of functions /} which precisely consists of all that can be defined using first order statements involving real numbers and the symbols -,-,, <. (fi)er-
(J:: R.. R
Standard examples of o-minimal structures are the structure of semialgebraic sets (generated by polynomials), the structure subanalytic sets (generated by restricted analytic functions) [38], the structure generated by the exponential function [79], and the combination of the latter two [32]. The popular GELU activation function [42] is definable in none of these structures [33, Theorem 5.11], but it is definable in the structure generated by Pfaffian functions:
Definition 2.2 (Pfaffian functions). A C²-ffuntion f: R + R is called Pfaffian if there exist C'-functions fi...., : R" - IR with f. such that for any i = I,...;y and j = 1,... n, the partial derivative 0fs (z) is polynomial in x,
0zy Theorem 2.3 ([80)). The structure Rptai generated by all Pfaffian functions is o-minimal.
Pfaffian functions include all functions that can be defined iteratively using polynomials, exponentials, logarithms, derivatives and antiderivatives.
Example 2.4. The following activation functions f: + R are Cal and definable.
(i) The logistic function f(x) T-e=x>
(ii) The hyperbolic tangent function f(x) tanh(x),
(iii) The softplus function f(x) log(1 + e²)) [36],
(iv) The swish function f(x) 1te- with 3 > 0.
(v) The Gaussian error linear unit (GELU) function f(x) x ][1 + erf( }),, where crf(x)
2 e dt the Gaussian error function [42].
v
We would like to thank Vincent Grandjean for pointing this out.

---

(vi) The Mish function f(z) = z: tanh + e²)) [68],
(vii) The exponential linear unit (ELU) function f(a) = e" -1,,,
x 1 0
[14],
x > 0
(viii) The softsign function f(x) |
The following loss functions &: R. x R IR are C+ and definable.
(ix) The squared error f(r,y) (x y)².
(x) The binary cross-entropy ((x, y) (x log y (1 x) log(1 - v)), for y E (0,
( - y)² (xi) The Huber loss E(x,y) 6((1c= yl )),
I yl S 8 fr yl > 8
with 8 E (0)
The following functions are definable but not C:
(xiii) The ReLU function f: RR R,
f(x) max 2).
(xiv) The absolute error L: R x R I, ((x v) |r y/.
We proceed with definition of neural network architectures. For function sh: R + I and integer d E N, we denote by v5(4); Rdd + Rd, (p(d) (x1...,, = (W(x1),.... ((())),,,,,(())) its amplification.
Definition 2.5 (Neural network architectures). A neural network architecture? is a tuple A
(do...., .d,, 4) where da...., d4 N and th: R + R a function called the activation function. The dimension of A is given by d(A) = > di(di-1 + 1). Fix an isomorphism Id((A)
I Rd,xdd-, RRd, The response NgA. Rdo Rdx of the architecture and parameters
Wa, by) Rd(A)) with W, E Rd and b, Rdd, is given by NgA(2) (z) = fa o oo..o fI()) where fI...., fu are defined as
=
(Wil) b,), << == k.
fa(x)
W + bb,
Definition 2.6 (Generalized critical values). Let f: IRd + R be a differentiable function.
(i) The set of critical points of f the set (0 E RRd ( = =0]-
(ii) The set of critical values of ff is the set Ko(f) = (ʃ(0) E R VS(0) = =0.-
(iii) The set of asymptotic critical values of f the set
Kso = (c E R 3(0m)nEN Rd s. t. |10,11 - 00, f(0m) + 110,, [I + 0}.
(iv) The set of generalized critical values of f is the set K(f) (,))) Definition 2.7 (Locally Lipschitz derivative), A C -functino f: R R. said have a locally Lipschitz derivative, if for each E R", there is an open neighbourhood U C R" of and constant L such that for all æ1, 12 E U, we have [())( - (x(=))l < L|| x2l1-
Below, characterize the gradient flows of neural network loss landscapes in the C' definable setting. Our result is mostly a well-known consequence of the available literature. We consider both the empirical loss with respect to finite dataset (see item (i) below) and the true loss with respect to a continuous data distribution (see item (ii) below).
For point E _" we denote by 8z the Dirac measure at For measure and measurable function p: R" + [0, oo), we write du(x)) = p(r)dr μ is given by ((A) = (()(dd for all
measurable sets A. The support of function p: R - [0, oo) is the set { E Rn p(()) # 03.
Frr more precisely f-ullyy connected feedfooward neural network
5

---

Theorem 2.8 (Dichotomy for gradient flows). Let A (do...., dR. 3) be a neural network archi- fecture with a Ca definable activation function vo (see Example 2.4 (i) (xi)). Let f: .R + RRd and e: Rd RRd + [0, oo) be Ca definable functions (the target and loss functions). Let p be a proba- bility measure on Rdo with expected loss L: Rd(A) R defined by [(0) ( such that
(i) μ ,. E dz; for some N and 21,... In E Rdo or
(ii) du(x) = p( for a C² function p: Rdd + [0, oo) such that is definable, or
that is a convex combination of cases (i) and (ū). Assume moreover that e, f and up have locally Lipschitz derivatives (see Example 2.4 (i) (xi)). Then the following hold:
(ii)) For every 0o E (Rd(AA,, there exists unique C2 ftunction θ: [0, oc) - TR(( satisfying
( = to. (t) = -VC=))))()()), Vt E (0,
(iv) For every O as in (iii) exactly one of the following holds:
(a) either lim ((t) exists and is a critical point of C. or
(b) lim [tQ = oo and lime- L(e(t)) is an asymptotic critical value of L.
(v) There exists an E > 0 such that for every as in (iil) with ((O(())) infoeRaA) L()) + for some la E [0, oo), it holds that lim (()(() = in ieR (()).
Proof. The proof of Theorem 2.8 is given on page 27 in Appendix B.
Remark 2.9. Note that Theorem 2.8 applies all the activation and loss functions listed in Ex-
ample 2.4 (i) - (xi). In particular, Theorem 2.8 applies to loss functions L: IR + R of the form EN (N, y1 for pairwise distinct Tn E Re and y1.....2 Un E Rdd
[(() +/ Σn Ei=1
3 Divergence of gradient flows for polynomial target functions
In this section, we state the main result of this paper (see Corollary 3.6) saying that for most of the common activation functions and any non-linear polynomial target function, the gradient flow almost always diverges to infinity while the loss converges to zero. Our conceptual starting point to this result is isolate the key properties of common activation functions into the following definition.
Definition 3.1 (SAD activation function). A function f: I=" + Rm is called sublinear analytic definable (SAD) if satisfies the following properties:
(S) lim sup
[[(e]]
oo, for all x R..
(sublinear)
(A) f is analytic,
(analytic)
(D) f is definable in some o-minimal structure (see Definition 2.1).
(definable)
Practically activation functions used in Machine Learning applications satisfy conditions (S) and (D) above. Moreover, many of them satisfy condition (A) too.
Example 3.2 (SAD activation functions). The activation functions (i) (vi) in Example 2.4 are SAD. Definition 3.3 (Loss function). We call function É: R" x g [0, oo) loss function if and only if it satisfies for all x, E IR that x = y < E(x, 3) 0.
The main technical contribution of this work states that on sufficiently big training dataset (or data distribution), given SAD neural network can never fit any non-linear polynomial perfectly.
We call Borel measurable set A C I conull if its complement has Lebesgue measure zero.
Theorem 3.4 (Non-representaabiitty of polynomials). Let A (do...., de, )) be a neural network architecture with SAD activation function vp (see Example 2.4 (i) (vi)). Let f: R - Rdd be polynomial of degree at least 2. Then there exist an integer N E N and a conull dense open subset DN S (Rdo)N such that the following holds: For any measurable loss function L: d x RR [0, oo) and any probability measure μ Rdd such that

---

In E R with (21....., IN) E DN. or (i) μ Ei= oz, for some n 2 N and 21....,
(ii) du(²)) = p( dr for a measurable function p: Rdo - [0, oo), or
such that js is convex combination of (i) and (ii), the expected loss L: Rd((A) - [0, oo) defined by [(0)= == Em ( f [())]] satisfies [(0) > for all 0 E Id((A), Moreover, in the case do = 1, the set Dx can be chosen the set of all tuples (21,...,,, n for which 31, ,1,..., are pairwise distinct.
Proof. The proof of Theorem 3.4 is given page 29 in Appendix C.
While Theorem 3.4 prevents the loss (()) from being exactly zero, we prove that fixed sufficiently big architecture A still achieves arbitrarily good approximations. This is standard result (see
[21, 28, 7D) which we include for convenience of the reader.
Theorem 3.5 (Neural network approximation for polynomials). Let J:R R - Rp" be a polynomial of degree E N. Let A = (do...., du, 1) be neural network architecture with do m input neurons, dy = r oulput neurons and a nol-oolylaiial analytic activation function sp. Assume that A is sufficiently big in the sense that
(i) there is a hidden layer < <i< k of size d, 2 r ((rttr) - -m).
(ii) all previous hidden layers have size min(do...... d;-;))....)..) 2 dn.
(iii) all subsequent hidden layers have size min(d,+1) I............ dj) du.
Then there exists sequence of parameters (02),EN S IR((AA) such that (N jeS converges to uniformly on compact sets. In particular, we have inf L(0) = where L is as in Theorem 3.4.
Proof. The proof of Theorem 3.5 is given page 35 in Appendix D.
We can now phrase our main result. It states that for sufficiently big architecture, sufficiently big dataset (or distribution) and sufficiently good initialization, the gradient flow diverges to infinity while the loss converges zero.
Corollary 3.6 (Divergence for polynomial target functions). Let f: R. + RR" be a polynomial of degree dfg(f) 2 2. Let A = (do.... de, vb) be neural network architecture with do m input neurons, dę output neurons and non non-polynome polynomial SAD activation function vp (see Example 2.4 (i) (vi)). Assume that A is sufficiently big in the sense that
(er((((e ((des) m).
(i) there is a hidden layer < zi< < k of size di 2 r
(it) all previous hidden layers have size min(do...... .... dd-1) do.
(iii) all subsequent hidden layers have size min(d, 1,, dz) du.
Then there is N N and conull dense open subset DN C (RRd)) with the following property: For any C definable loss function É: Rdo x ddo [0, oo) and any probability measure on Rdo
with expected loss L: (d((A) |0, oc) defined by L()) == Exzy ((Ng/ (x), f(x)), such that
(iv) P E oz; for some N and ,i...., In, E Rdd with (21,..., .N) DDN, or
(v) du(z)= = plæ ddx for a C¹ function p: Rd + oo) such that L is definable, or
such that μ is a convex combination of (iv) and (v), there exists an E > 0 such that for every C function θ: [0, 00) + Rd satisfying L(O(t))) < E for some tn [0, oo) and O (t) - - for all t E [0, oo), we have lim (()()) = 0) and lim [ = cc.
Proof. The corollary is direct consequence of Theorem 2.8, Theorem 3.4 and Theorem 3.5.
7

---

4 Numerical experiments
In this section, empirically validate the theoretical results established in the previous sections (see Corollary 3.6), demonstrating the divergence phenomena of the neural network parameters while the loss converges to zero. Interestingly, we observe that despite sufficiently good initialization is required our theorem, in practice it is not required for this phenomenon to occur. Our analysis begins with experiments on polynomial target functions and is then extended to more complex learning tasks, showing that this phenomenon persists beyond our settings and suggesting broader implications for neural network training. Further implementation details are provided in Appendix E.
While our theoretical results hold in the continuous-tin gradient flow regime, practical training is typically performed using discrete-time (stochastic) gradient descent. Since gradient descent can be viewed as an explicit discretization of gradient flow, we expect qualitatively similar behavior, with potential numerical differences arising due discretization.
The gradient flow dynamics of a differentiable loss function L: Rd B. are defined by the ordinary differential equation
(0)) Po, ('()) VV)((() At E [0, oo),
which describes the evolution of parameters under infinitesimal gradient updates. Gradient descent corresponds to an explicit (forward) Euler discretization of this flow:
0 82 + r76' (t)l==kn 8₂ ((xx).
Here O2 E IRd denotes the parameter vector at iteration k;, and >> 0 denotes the learning rate.
4.1
Validation on polynomial target functions
We train fully connected neural networks with three hidden layers to approximate polynomial target functions of varying input dimension. To exhibit the generality of our findings, we consider five different SAD activation functions (see Definition 3.1) )) and for each of them we plot averages of the loss and the parameter norm (using exponential moving averages and Monte Carlo averaging over 20 independent random initializations).
To most closely resemble the gradient flow dynamics, we begin our analysis with gradient descent, see Figure 1. The slow divergence of parameters is theoretically justified by the fact that the growth of parameter norms along the gradient flow is (vT)) (see (8) in Appendix B). In our setting, the growth seems be logarithmic illustrated in the exponentially rescaled Figure 6 in Appendix E. This is consistent with a similar logarithmic behavior proved [67, Theorem 4.3].
31.0
10.5
8.5
6.5
10-
65-
LV
08)
8.-
100000 200000
300000
403000
502050
100000
250300
300103
400202
200000 300000 402000
10-1
2015
10a2
10-1
100000
(al
(8)
(0
Figure 1: Approximation of polynomial target functions using different activation functions and GD algorithm. From left to right: 1-dimensional input, 2-dimensional input, 4 d-dimensional input.

---

Motivated by this limitation, we also employ the Adam optimizer [57], whose adaptive learning rate mitigates the vanishing gradients problem by dynamically rescaling the effective step size. In contrast gradient descent, where the update magnitude decreases as (L1/V)) with time Adam maintains substantially larger updates during training, enabling faster convergence in scenarios where gradient descent or SGD would require more iterations to achieve comparable results.
GLL
Hotenili torgert
150
50
50
100
40
90
100000
10
20-1
10
100000
502020
200300 400302
502000
(al
(b)
(0
Figure 2: Approximation of polynomial target functions using different activation functions and Adam algorithm. From left to right: I-dimensional, 2 2-dimensional, and 4-dimensional input.
In Figure and Figure 2, we consider target functions with 1-, and 4-dimensional input. In the multi-dimeesional cases, we also scale up the network size, increasing the number of hidden neurons from (d1, d2, d3) (10, 20, 10) (20, 40, 20). Despite the simplicity of the task and the relatively small network size, we observe a significant growth in the norm of the neural network parameters when the mean squared error loss converges to zero, confirming our theoretical results.
4.2 Extension more complex learning tasks
We investigate whether the divergence behavior observed in the theoretical settings extends to more complex, real-world learning tasks. Interestingly, we find that the same phenomena persist. In particular, the growth of the norm of neural network parameters becomes even more evident in practical scenarios involving high-dimensional data.
In Figure 3 (a), we train deep neural networks to approximate the solutions of two partial differential PDE. These equations defined follows:
equations (PDEs): the Heat PDE and the
Heat equation:
d, x) = A)(,,,
(1, E [0,T] x Rd, * E Rd.
u(0,z) = ||=||2
Black-Scholes equation:
d 0,((t Σ ((r- c),,,, f On.. utt,
O, (t, E [0, x (0,
u(0,2) x) = -TT max mmxx r - K, 0}.
x E (0, 00)
where the risk-free interest rate, e the cost carry, K the strike price and the volatility. To solve these PDEs employ the deep Kolmogorov method [5] which reformulates the PDEs as

---

stochastic optimization problems using the Feynman (b) we present results from standard supervised learning task: image classification on the MNIST dataset [63]. This allows us to observe a similar blow-up phenomenon in classification setting with discrete labels.
Teynman-Kac Kac representation. Additionally, in Figure
In all cases we observe significant increase in the parameter norm, consistent with the behavior analyzed in the previous section. These results show that our theoretical findings can be generalized to realistic examples and are indicative of general phenomenon. For the approximation of the PDEs
140
HestP3r Mace-Senola 102
MHN
1.00
20
42-
10
40
20
102000
202000
200020
409320
500000
5000
10003
15000
20020
25002
20000
6x13
2x15
4x13-7
3x10
10-1
15000 grapeenssess
25002
(a)
Figure 3: Left: approximation of Heat PDE and Black-Scholes PDE solutions using the deep Kolmogorov method. Right: image classification on the MNIST dataset.
we employ neural network with three hidden layers and the GELU activation function. The loss function displayed is the relative mean squared error between the model prediction and either the true solution or a Monte Carlo approximation of In the MNIST simulations we flatten the input images and train neural network with GELU activation that gradually decrease the dimension to 10, using the cross-entropy loss to predict the correct input class. To generalize our results, every line represents the exponential moving average of the results, averaged over five independent random initializations.
Conclusion
We proved that the gradient flow of definable neural networks either converges to critical point diverges to infinity and that the loss values converge to one of finitely many generalized critical values (see Theorem 2.8). We examined the special case of nonlinear polynomial target functions in detail. More specifically, assuming a sufficiently big training dataset and neural network architecture, showed that if the activation function satisfies mild condition (see Definition 3.1), then the loss function cannot achieve zero loss (see T'eeorem 3.4) but does achieve arbitrarily low values (see Theorem 3.5). From this we deduced our main result stating that in such a situation, the gradient flow diverges to infinity under mild assumptions (see Corollary 3.6). We validated our findings by training neural networks on polynomial target functions using different optimization methods. We then extended our investigation to more realistic scenarios and consistently observed a growth in the norm of the neural network parameters as the loss diminishes, suggesting that the divergence phenomenon we analyzed actually holds in much broader generality (see Section 4).
10

---

Acknowledgements
This work has been supported by the Ministry of Culture and Science NRW as part of the Lamarr Fel- low Network. In addition, this work has been partially funded by the Deutsche Forschungsge- meinschaft (DFG, German Research Foundation) under Germany's s Excellence Strategy EXC 2044- 390685587, Mathematics Münster: via the Al4Forest project, which is funded by the German Federal Ministry of Education and Research (BMBF: grant number 011S23025A), and the French National Research Agency (ANR). We gratefully acknowledge the substantial computational resources that were made available to us by the PALMA Il cluster at the University of Münster (subsidized by the DFG; INST 21 211/667-1). The authors are indebted to Floris Vermeulen and Mariana Vicaria for their help in improving the statement of Theorem 3.4. Helpful conversations with Christopher Deninger, Ksenia Fedosova, Allen Gehret, Fabian Gieseke, Vincent Grandjean, and Dennis Wulle are gratefully acknowledged.
This work is partially supported
References
[1] Pierre-Antoine Absil, Robert Mahony, and Ben Andrews. Convergence of the iterates of descent methods for analytic cost functions. SIAM Journal on Optimization, 16(2):531-547, 2005,
: 1137 111370040005666.
[2] Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-paramettrrizatoon. In International Conference on Machine Learning, pages 142-252,220 2019. www.:mm.s.....pt..mm.mm..
URL:
mlr press /v97 -ellennzzulle .html.
[3] Hédy Attouch, Jérôme Bolte, Patrick Redont, and Antoine Soubeyran. Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-tojasiewice inequality. Mathematics of Operations Research, 35(2):438-45) 457, 2010. doi: 1287 10287/moox 1100. 0449.
[4] Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semi-algebraic and tame problems: proximal algorithms, bard splitting, and
regularized Gauss-Seidel methods. Mathematical Programming, 137(1):91-129 2013. doi:
10. 10.1007/s10107-011-04633
[5] Christian Beck, Sebastian Becker, Philipp Grohs, Nor Jaafari, and Arnulf Jentzen. Solving the Kolmogorov PDE by means of deep learning. Journal of Scientific Computing, 88:1-28, 2021. doi: 10l:10.10011
[6] Alessandro Berarducci and Margarita Otero. An additive measure in o-minimal expansions
of fields. Quarterly Journal of Mathematics, 55(4):41 2004. 4oi:10: 1093 /am//ttahh..... hah010.
[7] Julius Berner, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. Mathematical Aspects of Deep Learning, chapter The Modern Mathematics of Deep Learning, pages 1-111. Cambridge University press, 2022. doi :00 00117/9781000025095 002.
[8] Pascal Bianchi, Walid Hachem, and Sholom Schechtman. Convergence of constant step stochastic gradient descent for non-smooth non-convex functions. Set- Valued and Variational Analysis, 1013):1117-1441 2022. doi: 1007 /1000///1110 swwssa--aaa-- 022-00022332220
[9] Helmut Bolcskei, Philipp Grohs, Gitta Kutyniok, and Philipp Petersen. Optimal approximation with sparsely connected deep neural networks. SIAM Journal on Mathematics of Data Science, 1(1):8-45, 2019. doi: 10. 1 1137 18M1 18709X.
[10] Jérôme Bolte, Aris Daniilidis, and Adrian Lewis. The Lojasiewicz inequality for nonsmooth subanalytic functions with applications to subgradient dynamical systems. SIAM Journal on Optimization,
2007. doi: 10. 1137 /556444444
[11] Léon Bottou. On-line learning and stochastic approximations. In On-Line Learning in Neural Networks, pages 42. Cambridge University Press, 1999. doi: 10.110:::0 110110000/009988866666999 003.
[12] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh,

---

Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemnrr, Sam McCan- dlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877- www.eeddiinnnnuuniuieeeeeee neur ips. c// f wwtppppee.fille//ppppeeeeeeeeeee
1901, 2020. URL:
[13] Marshall W. Buck, Raymond A. Coley, and David P. Robbins. A generalized Vandermonde determinant. Journal of Algebraic Combinatorics, 1(2):105-19, 1992. doi: 1023/A: A: 1022468019197.
[14] Amm Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep net- work learning by exponential linear units (ELUs). International Conference on Learning Representations, 2016. arXiv: 1.11..72889
[15] M. Coste. An Introduction to O-minimal Geometry. Dottorato di ricerca in matematica / Università di Pisa. Dipartimento di Matematica. Istituti Editoriali e Poligrafici Internazionali, 2000. URL: https: //perso wwwvvveennee.... frr frrmmr//iiiee chel. coste/polyess /MMIN. pdf.
[16] George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303-314, 1989. :10:10.00 1007 (B000//F2255124,
[17] Didier d' Acunto. Valeurs critiques asymptotiques d' d'une fonetion définissable dans une structure o-minimale. Annales Polonici Mathematici, 75(1):35-45 455, 2000. doi:10.4060 4064/
ap-75-13 15--455.
[18] Aris Danillidis, Mounir Haddou, and Olivier Ley. A convex function satisfying the tojasiewicz inequality but failing the gradient conjecture both at zero and infinity. Bulletin of the London Mathematical Society, 54(2):590-608, 2022. doi : 10. 11 1112/blms. 12586.
[19] Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D. Lee. Stochastic subgradient method converges on tame functions. Foundations of computational mathematics, 20(1):119 154, 2020. doi :10:10. 1007
[20] Joseph Daws and Clayton Webster. Analysis of deep neural networks with quasi-optimal polynomial approximation rates. 2019. arXiv: 1912. 02300..
[21] Tim De Ryck, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks, 143:732-750, 2021. doi:10. 1oi:1/.1016/] nnunet 2021.
08. 015.
[22] Steffen Dereich and Arnulf Jentzen. Convergence rates for the Adam optimizer. 2024. arXiv: 2407.20 .20778.
[23] Steffen Dereich, Arnulf Jentzen, and Sebastian Kassing. On the existence of minimizers in shallow residual ReLU neural network optimization landscapes. SIAM Journal on Numerical Analysis, 2646)266,, 2024. doi: 10. 1 137 0037/223155624
[24] Steffen Dereich and Sebastian Kassing. Cooling down stochastic differential equations: Almost sure convergence. Stochastic Processes and their Applications, 152:289-311. October 2022.
101:16..116/..... spp. 2022. .6.00...00 .200.
[25] Steffen Dereich and Sebastian Kassing. Convergence of stochastic gradient descent schemes for Setisiewwizz-andseaappee Journal of Machine Learning, 3(3):245-281, 2024. doi: 10. 4208/ jml 240109.
[26] Steffen Dereich and Sebastian Kassing. On the existence of optimal shallow feedforward networks with ReLU activation. Journal of Machine Learning, 3(1):1-22, 2024. do1:10. 4208/jml 2300903.
[27] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, pages 4171-4186, 2019. arXiv: 1810. 04805.
[28) Ronald De Vore, Boris Hanin, and Guergana Petrova. Neural network approximation. Acta Numerica, 30:327-444. 2021. doi: 10. 10.0.011///0000 10177509962299992000005
[29] Thang Do, Sonja Hannibal, and Arnulf Jentzen. Non-convergence to global minimizers in data driven supervised deep learning: Adam and stochastic gradient descent optimization provably fail converge to global minimizers in the training of deep neural networks with
ReLU activation. 2025. ar xixi:24100 10533.
12

---

[30] Thang Do, Aruuff Jentzen, and Adrian Riekert. Non-convergence to the optimal risk for Adam and stochastic gradient descent optimization the training of deep neural networks. 2025.
arXiv: 2503. 250..01660. 01660.
[31] Lou van den Dries. Tame Topology and O-minimal structures, volume 248. Cambridge University Press, 1998. 15t:1000117/B0008// 1525919.
[32] Lou van den Dries, Angus Macintyre, and David Marker. The elementary theory of restricted analytic fields with exponentiation. Annals of Mathematics, 140(1):183-22)) 1994. doi: 10. 233072218845
[33] Lou van den Dries. Angus Macintyre, and David Marker. Logarihmii--epooentii power series. Journal of the London Mathematical Society, 56(3):417-434, 1997. doi: :0. 1112/ S00246101 00024610797005337
[34] Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. International Conference on Machine Learning, 2019.
URL: https::/p///. htttps://proceedinggg mlr. press/v97 /ttp:/ceednnnn..l...peee//////// (tlpu////u19../pp.//ee
[35] John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(61), 2011. URL:
http://jmlr.ote http:attrrrppppp/r///eaaepeeeeee
duchi 1a. html.
[36] Charles Dugas, Yoshua Bengio, François Bélisle, Claude Nadeau, and René Garcia. Incor- porating second-order functional knowledge for better option pricing. Advances in Neural Information Processing Systems, 13, 2000. URL: htppp://p//p neurips...e..............r.. ec/ paper_files/
Paper pdf. [37] Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks Neural networks, 203):183-192, 1989. doi:10 0001660889360888 (89) 90003-8.
[38] Andrei M. Gabrielov. Projections of semi-analytic sets. Functional Analysis and its Applications, 2(4):282 291, 1968. doi: 10. 1007 /BF010766800
[39] Davide Gallon, Arnulf Jentzen, and Felix Lindner. Blow up phenomena for gradient descent optimization methods in the training of artificial neural networks. 2022. a::iv:2211 15641.
[40] Russell Gentile and Gerrit Welper. Approximation results for gradient descent trained shallow neural networks in 1d. 2022. arXiv:2209 08399.
[41] Russell Gentile and Gerrit Welper. Approximation results for gradient flow trained shallow neural networks in 1d. Constructive Approximation, 60(3):547-594, 2024. doi: 101:10.1077
0205--0022-9994---
[42] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). 2023. arXiv: 08415.
[43] G. Hinton. Lecture 6e: RMSprop: Divide the gradient by running average ofits recent magni- tude., 2014. URL: https://twwtt http://wwww.....to.ooooo/ toronto. edu/ (tcij/eeecccc33//eeeeeeeee/////e slides_lec6. pdf.
[44] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward net- works are universal approximators. Neural networks, ((5):39-3666 1989. doi: :0. 0893-6080 90020-8.
[45] Xiaoyin Hu, Nachuan Xiao, Xin Liu, and Kim-Chuar Toh. Learning-ran rate-ffrrre-eeeeee momentum SGD with reshuffling converges in nonsmooth nonconvex optimization. Journal of Scientific Computing, 102(3):85, 2025. doi 10. 0910///119091-00002207009
[46] Shokhrukh Ibragimov, Arnulf Jentzen, and Adrian Riekert. Convergence to good non-optimal critical points in the training of neural networks: Gradient descent optimization with one random initialization overcomes all bad non-global local minima with high probability. 2025.
arXiv: 2212. 13111.
[47] Alexander D loffe. An invitation to tame optimization. SIAM Journal on Optimization, 1944):1894-1917, 2009. doi: 10. 1137/080772209..
[48] Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, vol- ume 31, 2018. URL:
neur neurips. ce ge/paper___ ppaper_fiii / Paper .df.
13

---

[49] Arnulf Jentzen and Adrian Riekert. On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. Journal of Machine Learning, 1(2):141-246, 2022. doi: 10. 1137/133222 04.77223M15660 15555444..
[50] Arnulf Jentzen and Adrian Riekert. Non-convergence to global minimizers for Adam and stochastic gradient descent optimization and constructions of local minimizers in the training of artificial neural networks. SIAM/ASA Journal on Uncertainty Quantification, 13(3).1294-1333,
2025. doi: 10. 1137
[51] Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. 33:17176-17186, 2020. URL: httl:::ttp://////ooto pttpttpppss/ppoooeeeiinggaa nwwrigs...uu........ wwwpuuripp..ppppppp.eeeepppp
[52] Cédric Josz. Global convergence of the gradient method for functions definable in o- minimal structures. Mathematical Programming, 202(():355-383, 2023, 00i:10.1000
s10107-023-01937-
[53] Cédric Josz and Lexiao Lai. Global stability of first-order methods for coercive tame functions. Mathematical Programming, 207(1):551-576, 2024. doi: 10.
[54] Cédric Josz and Xiaopeng Li. Certifying the absence of spurious local minima infinity. SIAM Journal on Optimization, 33(3): 1416-149,, 2023. doi: 10. 1137 2221149951..
[55] Marek Karpinski and Angus Macintyre. Polynomial bounds for VC dimension of sigmoidal neural networks. In Proceedings of the (wenty-seventh amnual ACM symposium on Theory of computing, pages 200-208, 1995. doi :00. 1145/ 0145///2288.../ 225118. 118.
[56] Marek Karpinski and Angus Macintyre. Polynomial bounds for VC dimension of sigmoidal and general pfaffian neural networks. Journal of Computer and System Sciences, 54(1):169-177
1997. doi: 10. 1006/15 jcss. 1997 1477.
[57] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 2014.
arXiv: 1412.6900.
[58] Julia Knight, Anand Pillay, and Charles Steinhorn. Definable sets in ordered structures. II. Transactions of the American Mathematical Society, 295(():593 1986. doi 10. 1090/
10002-9947-19866 0833898-331111
[59] Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in two-homogeneous neural networks, 2024. URL: https: httpe:///oeeeeeeeeeeeee wwt.:/rrttffr////m id= hfrPag75YO.
[60] Krzysziof Kurdyka. On gradients of functions definable in o-minimal structures. Annales de l'institut Fourier, 48(3):669-783 1998. doi: 10. 100:100/000/0 5802/a101111 1638.
[61] Krzysztof Kurdyka and Adam Parusinski. Quasi-convex decomposition in o-minimal structures. application to the gradient conjecture. Advanced Studies in Pure Mathematics., 43:137-177, 2006. doi: 10. 2 hwsp://sp////////aa/eeea 043101377
[62] Lexiao Lai. Stability of first-order methods in tame optimization. PhD thesis, Columbia University, 2024, arXiv:2412. 2.00000.
[63] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998. doi:
10. 1 109/5. 726791.
[64] Moshe Leshno, Vladimir Ya Lin, Allan Pinkus, and Shimon Schocken, Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural networks, 66):861-867, 1993. doi: 1016/10899-6088 (05) 80131-5.
[65] Haochuan Li, Jian Qian, Yi Tian, Alexander Rakhlin, and Ali Jadbabaie. Con- vex and non-convex optimization under generalized smoothness. In Advances in Neural Information Processing Systems, volume 36, pages 40278-40271, 2023. URI::
http:://ppooceeedsn.n..eeeeeeeee re2bbal1111222ee7722888800---pp0 httpc//ppppppeeeeepppppeereeeeee wttp://spaaeee///////////// pdf.
[66] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[67] Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020. ar xiviv: 1906. 05890.
14

---

[68] Diganta Misra. Mish: A self regularized non-monotonic activation function. In British Machine Vision Conference, 2020. arXiv: 1908. 08681.
[69] Joost AA Opschoor, Philipp C Petersen, and Christoph Schwab. Deep ReLU networks and high-order finite element methods. Analysis and Applications, 18(05):715-770, 2020, doi: 10. 0t.114//9921933555100 0136.
[70] Philipp Petersen, Mones Raslan, and Felix Voigtlaender. Topological properties of the set of functions generated by neural networks of fixed size. Found. Comput. Math., 21(2):375
2021. doi: 10. 1007 0000/s11206 -0008-020-9900 0946100
[71] Philipp Petersen and Felix Voigtlaender. Optimal approximation of piecewise smooth functions using deep relu neural networks. Neural Networks, 108:296-330, 2018. doi: :10:10.11 1016/j. neunet. 2018 8. 019.
[72] Anand Pillay and Charles Steinhorn. Definable sets in ordered structures. Trans- actions of the American Mathematical Society, 295(2)5565-52,, 1986. doi:10. 1090/ 80002-9947- 198--77888888883697
[73] Anand Pillay and Charles Steinhorn. Definable sets in ordered structures. III. Transactions of the American Mathematical Society, pages 469-476 1988. : 2307 /2000920.
[74] Allan Pinkus. Approximation theory of the MLP model in neural networks. Acta numerica, 8:143-195, 1999. doi: 10. 1017 10710/00000224/9000000999999
[75] Boris Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1-17, 1964. doi: 1016// 0041-5553 90137-5.
[76] Patrick J. Rabier. Ehresmann fibrations and Palais-Smale conditions for morphisms of Finsler manifolds. Annals of Mathematics, pages 647-691, 1997. do1: 10. 2307 2000/2952222 22524557
[77] Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathematical Statistics, pages 400-407, 1951. doi: 1214/ htt///om//111/ 1177m729511
[78] Gal Vardi, Ohad Shamir, and Nati Srebro. On margin maximization in linear and ReLU networks. In Advances Neural Information Processing Systems, volume 35, pages 37024- 37036, 2022. URL: http://caaaaaaaaacccc.ececeooerp
nwur cc/ wttpp//pppsppepppppeeeeeeeeeeeee _f Mitees/aper//0055
[79] Alex J. Wilkie. Model completeness results for expansions the ordered field of real numbers by restricted pfaffian functions and the exponential function. Jourmal
Journall of the American Mathematical
Society, 1(4):1051-109) 1996. :00:10. 1001000000000007-9--000000000000
[80] Alex J. Wilkie. A theorem of the complement and
o-minimal structures. Selecta
Mathematica, 5:397-421, 1999.
[81] Alex J. Wilkie. o-minimal structures. Astérisque, (326): No. 985, vii, 131-142, 2009. Sémi- naire Bourbaki. Vol. 2007/2008.
URL: https: //ww./w.ww numdam.
_326__1311 .dff.
[82] Dmitry Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Metworks, 94:103-114, 2017. doi: 10. 1e::1...016// 1016/j neunet. 2017 00.002.
[83] Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. International Conference Learning Representations, 2020. arXiv:1905. 11881.
15

---

NeurIPS Paper Checklist
(i) Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both the abstract and the introduction informally state main theorems including their assumtions: The dichotomy for definable gradient flows and the divergence result for polynomial target functions. The numerical experiments confirming theoretical anylsis are outlined as well.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
iii
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss what extent our theoretical analysis resembles real-world learning tasks, and point out our inability of our mathematical tools to tackle the ReLU function or stochastic gradient descent rather than gradient flows. We also acknowledge the limited size of our datasets and emphasized our focus on the mathematical theorems.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specifiiattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
16

---

(iii)
Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The main paper provides mathematically complete and rigorous statements
(theorems and corollaries) of the main claims including all assumptions. All mathematical
claims are either referenced to appropriate sources or proved in full detail in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
(iv) Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes].
Justification: Our models are quite simple, the main details are in Section 4 and Ap- pendix E. Moreover, detailed instruction for reproducing our experiments are provided in the README file included with the supplementary code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
(v) Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide open access code, including configuration files and detailed instructions to faithfully reproduce all main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
herebbiisssoooollloooooooo lonPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (ht
/ //nips.cc/put wttpcccppplccccccc www.c///g//eeeeeeeeoooooooose CodeSubmiss cesttubmmissii... aesonPoooccyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
(vi) Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify the key training and test details in Section 4 and Appendix E. All remaining parameters are included in the configuration files available in the code repository.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
(vii)
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We did not report emror bars because our main focus is to illustrate the growth network parameters as the loss vanishes. reduce variance, we average results over multiple independent runs, which consistently show the same gualitative behavior.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the eπror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
(viii)
Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We specify the resources Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
(ix) Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:
Answer: [Yes]
Justification:
The research conforms the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
(x) Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
19

---

Justification: Our work contributes towards a theoretical mathematical understanding of why gradient based optimization works the way does. Our work does not have an immediate impact on society since it does not develop any novel algorithm and does not use any data apart from the MNIST dataset.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
(xi) Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Being mathematical work focusing on general theory, our paper does not release any relevant models or data. There are no risks for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
(xii) Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited the publicly available MNIST dataset.
Guidelines:
The answer NA means that the paper does not existing assets.
20

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paper swi th code. www.ooo/aaaaeo aseee has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pase re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
(xiii) New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The only asset released the paper is the code for producing the numeri- cal experiments. The code is published along with sufficient documentation and license information in an anonymous GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
(xiv)
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
(xv)
Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
21

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22

---

Appendix
A Locally Lipschitz derivatives
The aim of this section is to establish that the expected loss appearing in Theorem 2.8 has a locally Lipschitz derivative whenever the activation function and the loss function have locally Lipschitz derivative. This is needed to ensure uniqueness of the gradient flow for the ELU function, the softsign function and the Huber loss in Example 2.4. This section can safely be skipped if one is willing to assume that the activation function and the loss function are C².
Lemma A.1. The class fanctionn with locally Lipschitz derivatives (see Definition 2.7) is closed under the following operations.
(i) Every CZ CP-function has locally Lipschitz derivative.
(ii) A function f f= (f1,.............. -fi....., fm): I - 1 has locally Lipschitz derivative if and only if all of its components fI...., fm R - R have locally Lipschitz derivatives.
(ii) if Th - and g: R.aa -> have locally Lipschitz derivatives, then
② g: Rutt _ R, f g0r v) = [xx +(v)
and
f g: R - R, fo f(x)(())) have locally Lipschitz derivatives.
(iv) f:::= R and g: R Th have locally Lipschitz derivatives. then
fog:R" + RR. (og(z)=f((b/)
has a locally Lipschitz derivative.
(v) Let a, b E R with a < and suppose that f: R + has a locally Lipschitz derivative. Then the map
F:R R. F(x) [" s(1,))d
has a locally Lipschitz derivative.
Proof.
This is well-known consequence of the fundamental theorem of calculus.
iii This follows from the inequality
max Iss.
[Df(()) |Df(x)| s m max [Df((x)] Is
for all E R"
(iii) We only prove that f ④ g has a locally Lipschitz derivative since the proof for f o is similar. Fix (zo. vo) R R t Fix Ly, Ly 0 and relatively compact open neighborhoods To E Uy C R" and E Ug C R satisfying
[Df((t) Df(xa) | x2/l, Vrn, Us
and
[Pg(y)) De(u2))l S Lgllen vall. Vyn, U2 E Ug- Since f and g O and thus locally Lipschitz, there are constants L' L2 0 such that
[l((;)) f(x))) L'yla xzll, Vor, E EU,
and
(l g(22)1 1y vell. By continuity and relative compactness, the quantities
Vur, Ug.
Ky= sup [x),,), Ký = sup f(()), Kg = sup (D(()), =EU,
KG = sup Ig(v)) veC,
=eU,
veU.
23

---

are all finite. Defining
1 + K + Kigh + Kglig < <00,
we have for all (21, (12. E UU x Un that
D 2 @ D(J : (2)2
(Df(22)g(12) ( (g(2))
((a)) (lf (o0h:p/u0lllllltt--D00)ooaaaDaad
+ -
, - (//////
<11 1/1 (a:2, 92)ll-
This proves that f@ y has locally Lipschitz derivative.
(iv) Fix 20 IR. By assumption, there are Ly, Lg > and relatively compact open neighbour- hoods g( E U S IR' and zo E Ug S IR" satisfying
[Df((m)) - Df( - 1 - 32ll. Vyl,82 E EU,
and
[Dg(21) - -D(x)i) 1 L |1 xell.
V21, E Ug.
By continuity of g. we may assume that g(U,)) U.. Since g is C² and thus itself locally Lipschitz, we may moreover assume that there is L' 0 such that
l1g(z)) - g(x2))l - Lyll= - x2l1, Vzi, 12 E Ug. By continuity of Df and Dy and relative compactness of U, and Ug. we have
L = و sup Net,
[[gg(>]] + 4g sup [[ < co. yeti
Moreover, for all 1ı, X2 E U, we have
|| o 9)(11) - D(f (x2) = [Df(((=))))(())) - (D
<I]ff(((m)) Df(((())))) Do(z))
+ +[IDf(((=))ll(p((()==- D9(22)]
<b||>> x2ll.
This proves that fo fog has locally Lipschitz derivative.
(v) Fix To R" Since f has a locally Lipschitz derivative, there is for every & [a, b] a constant Lg > 0 and an open neighbourhood (f, E U. S I such that for all (f1, (12, U,, have
[ID[(1,,)) D1(1 < 2ell(),221)= (2))l
find t.....,t E [a, b] such that b] {aat S Ut_ Un. Without loss of generality, we may assume that each U, for = 1,..., is of the form (a,, be) x V for open neighbourhood To E V C R=- and a,, b, R. Now define L = (b- - a) max L,, and let 1 22 E V be arbitrary. Using the Leibniz integral rule,
By compactness,
we get
| (Daf(t, - Da D (2))
[DF(2,)) DF(x2))
Dff(t,z)) Df x2)
/" max Lt, ||||| x2l dt
15iSk
< L| I2ll.
24

---

where (t,(t,z; 24) denotes the differential of the map + f(t, at = 24. This proves that F has a locally Lipschitz derivative.
1b) be a neural network architecture with C' activation function ub: R IR with a locally Lipschitz derivative. Let L: Rd x Rd - 00) be C loss function with locally Lipschitz derivative. Let f: IR + Rd be C' target functionn with locally Lipschitz derivative. Let p be probability measure on Rd such that either
Corollary A.2. Let A
(i) 1 = Ei= Sxg for some E N and ,.,...; E Rdo, or (i) du(x) = p(x))dd for a C² Ca function p: Rddo - [0, 00) of compact support.
Then the function
C: (d(A) + R [()) = - [e (z), f(x)
has locally Lipschitz derivative.
Proof. This immediately follows from Lemma A.1 since & can be constructed iteratively from 1 and linear functions using the operations (i) (v).
L
B Proof of Theorem 2.8
This section gives a proof of Theorem 2.8 which probably well-known to experts and follows standard techniques. The proof is mostly given for lack of precise reference and for convenience of the reader. Before giving the technical details, we outline the main ideas below.
Idea of proof
Existence and uniqueness of the gradient flow for L follow from the fact that L is C'l and has locally Lipschitz derivative. The latter assumption is checked Appendix Now one can distinguish two cases: Either the gradient flow visits some bounded set infinitely often or it does not. In the first case, we use Kurdyka's s definable Lojasiewicz. inequality [60] and follow an argument from [1, Theorem 2.2] to prove convergence to critical point (see Lemma B.1). In the second case, the parameters diverge by assumption. We check that the limit a generalized critical value using the fact that is bounded below and therefore has square-integrabee gradients along any gradient flow trajectory (see Lemma B.2).
For point zo E RR and R > 0, we denote by
BR(Ho) = fa E R" | æoll < R) C R"
the open ball of radius R centered at 20.
Lemma B.1. Let C: IRd + [0, oc) be C²2 definable function and let O: [0, 00) -R Rd a C -ffnctionn satisfying the differential equation (t) -V VV(((()) for all t [0, oo). Suppose that there is
point 02 E Rd satisfying
lim inf [[@(t)= 0-1 = (+000
Then we have li O(1) O..
Proof. Without loss of generality, we may assume that ((0.) = 0. Note that the function Lo θ: [0, oc) + is non-increasing since we have (()(t)) = -IV)(((())] for all t E oo). Since L continuous, (1) and the fact that L is non-increasing imply that
lim L(O(t)) C(0.) = 0. t-+00
Now fix > 0. By [60, Theorem 2 (b)), there a continuous strictly increasing definable function
a: [0, 00) [0, oo) with a00) such that whenever t,, E [0, oc) with f < ta satisfy
0(t,, t2]) C B2g (f.), we have
( c(((t)))
(3)
25

---

Since + L(O(t)) is noninereasing, (3) implies that
for all f1,12 E [0, oo) satisfying ([t tel) S (2.
By continuity of a and (2), we can find a large enough t > such that a ((((((t))) < 5. By (I), we may moreover assume that (tt)) E B (0.). We claim that have O (0,) for all t> h.
Assuming the contrary, there is smallest t2 satisfying [[O(tz) e. (2), S (0.) and by an application of (4), we get
But then have
(ttt)) 0.11 = ((0(t2) O(t,)) ([0(t)) - 0x
(t)ll 8
$ (c(((())))
<,
a contradiction. Thus, we have O([,, o)) _ (O.).. Since > 0 was arbitrary, the lemma is proven.
oo) be C2 function. Then for any Og R.d, there is a C'-function Lemma B.2. Let L: Rd + oo)
:: [0, oo) + Rd satisfying
-(tt)=-V[(0))
@()) 80
(5)
for all t E (0) Moreover, the following statements are true:
(i) If C moreover has a locally Lipschitz derivative, then the solution θ (5) is unique.
(ii) If C is moreover definable, then exactly one of the following two statements holds true.
(a) lim; (tt) exists and is critical point of f.
(b) li [|O(1)] = oo and lime-soo ((O(L) is generulized critical value of C (see Definition 2.6).
Proof. The existence C'Immap θ [0, 0o) -> IRd satisfying (5) follows for instance from [54, Proposition 2.11]. The uniqueness part in item (i) follows from the
theorem.
Now assume that moreover definable. We consider two cases:
Case 1: We have lim in [[0(1)] < oc.
In this case, there is constant R 0 and sequence (fn)nEN in [0, oo) satisfying ((t,) e (R for all n E and lim fn = co. By compactness, a subsequence of (O(tx))nex converges to some point 0, E (0). In particular, we have lim inf,-+oo ||O(t) 0.1l 0. Therefore, Lemma B. implies that
lim ( 0..
(6)
To check that 0, is critical point, note that since L takes only non-negative values, have
S [s)) ds = - S. =
VC(O(s)))
=()))) (()())
L(O()) s (()())
for all E [0, o0). In particular, + ||@((1)] is square-integrable. combination with (6), this proves
|VC().]] iim VV(((()) = inf ||O((t]] = 0,
++000
t+000
so that 0, a critical point of L.
Case 2: We have lim info-4oo [|O(t)]
26

---

In this case, the limit = lim (t)()) exists since the map - (() is decreasing and bounded below. It remains to be shown that lim in [[((t) (t)) [l0] Proposition 2.54], we have
By [39,
||(())| - [[0(0)] + )LC((())),
(8)
and thus
= 1100 ( (10(00) Vic(((o) ( [[0(t))), (9)
[[(t)]|| -
for every E (0,
Assume by contradiction that we have
é = lim inf (10
(6' > 0.
8-000
Then there exists T E (0, oo) such that inf12r (10(0)] VIZ(@(01) |0'(1]] > Thus, we get
0 / (19(0)) + V
-2 di S e² [ [1((
contradicting (7). Thus we have lim inf (10(0)] + (tC(+(0)) ([e (9), this proves that lim ino |[((t = and concludes the proof of the lemma. C
= 0. Together with
Proof of Theorem 2.8. It is either easy to see or stated as an assumption that C is definable. Moreover, is an easy consequence of the Leibniz integral rule that L C Thus, [17] is applicable and establishes that the set K ((C) of generalized critical values of L (see Definition 2.6) is finite. Moreover, has locally Lipschitz derivative by Corollary A.2. Now the theorem follows from Lemma B.2 we choose 0 small enough so that the open interval (inføeR> L(0), infoeR" [()) + s) does not contain any generalized critical value.
C Proof of Theorem 3.4
This section is dedicated to the proof of our main technical contribution Theorem 3.4. This is the main part where o-minimal structures are used in a new way which cannot be found in the neural network literature. We outline the main ideas of the proof below.
Idea of proof
The starting point is that the sublinearity condition (S) in Definition 3.1 also implies that the neural network responses NNA are sublinear (see Lemma C.2). In particular, cannot be equal as a function to polynomial of degree at least (see Lemma C.3). This does not explain why L does not have any zeros yet since N₁4 and ff could equal on the dataset or data distribution for some 0. The main bulk of the proof now exploits the remarkable rigidity properties of definable analytic functions to show that this cannot happen (see Lemma C.4). We treat the one-dimensional and dineiiddimessionll input cases separately. In the one-dimensional case, the combination of the identity theorem for analytic functions and the uniform finiteness theorem from o-minimal geometry imply that for any 6, Ni* and f cannot be equal independent of 0. In the multidimensiionl case, the conclusion can fail for very degenerate datasets (for example if all datapoints lie on line along which the polynomial is linear). But by combining the ideas from the one-dimensional case with dimension theory for definable sets, we show that the space of such "bad datasets" has strictly smaller dimension than the space of all datasets. Therefore its complement contains an open dense subset of full Lebesgue measure. The use of definable dimension theory was kindly suggested to by Floris Vermeulen and Mariana Vicaria.
sufficiently large finite dataset where the required size be made
Lemma C.1. Let f: TR - R be a C¹ definable function. Then f satisfies condition (S) of Defini- tion 3.1 if and only if its derivative is bounded function.
Proof. Assume first that C' == suptER (t) < 30 and fix x E R. Then the fundamental theorem of calculus implies for any / E (0, oo) that
1 [1(0)+ /' [s(ttt))]tx() f(tx)] -
< If(0) + Clxl 1-000 C|x| < oc,
27

---

proving condition (S). We prove the converse and assume f is unbounded. Since f" is definable [15, Lemma 6.1]. it is monotone outside compact interval [-R, R| by the monotonicity theorem
[15, Theorem 2. Without loss of generality, we may assume that f' is increasing and unbounded on RR oo). The cases that f decreasing and unbounded on R, 0o), increasing and unbounded on
(-o0, Rl.. or decreasing and unbounded on (-oo, R] are analogous. Fix constant K > 0. Since
f' is increasing and unbounded on [R, oo), there is a constant R' R such that f'(t) > K for all
t > R.. Thus for all E [Rt', o0), we have
f(t.
S(R))
[ ((s)ds
1
lim sup 1-200
sup 1-000
+ 8 11
-If(RR) + + [ (((s)ds)
limsup 1-000
-IF(R') + +(t (t- R\J
limsup
1-100
= =K
Since K3 O was arbitrary, conclude that lim -0p [(t-1] oc, disproving condition (S).
Lemma C.2. Let A= = (do...., de, +) be a neural network architecture and 0 E Rd(A), If the activation bb: R > I is SAD, then the response NºA: Rddo -> RR SAD, too.
Proof. It is easy see that.W3 satisfies conditions (A) and (D) of Definition 3.1 for all 0 Rd. It remains verify (S). Since condition (S) be checked on each component of .VAA may assume without loss of generality that A = (do...., de, e)) has only dN = 1 output neuron.
We prove that N2AA satisfies condition (S) by induction on k. Assume first that k = 1. Then N₂ is of the form
No. (z) w1 + Td + b
for 0 = (wr,, Wn, b). Then, NV²² affine linear and thus satisfies condition (S).
Assume that we have proven the statement for i 1 and fix 0 E Rd and E Rd We define a smaller architecture A' (da....,ds-.. 1, v)) with only As 1 layers by deleting the (k - 1)-th layer. Note that by rearranging the components of 8 E Fd as 0 = (f1...., Odz- WI..... Wd. b), for appropriate w1,.... Wd.-1; b E R and 0I..... Od._, E (d- 1)/d/ we can write
No - Σ wab(NN (tx)) (tx))()))))))=)))))))))))))))( +
(10)
for all t E R. By the induction hypothesis, the functions + Ng. (tx) satisfiy condition (S) for every = O.... dx-1- In particular, they have bounded derivatives by Lemma C.1. By another application of Lemma C.1 and the chain rule, it follows that the functions + woth (tx)) for
O, i=
satisfy condition (S), too. By linearity and Lemma C.I applied to (10), it follows
that the function / + (tx(tx) satisfies condition (S). Since x E Id was arbitrary, we conclude that NgA satisfies condition (S).
Lemma C.3. Let f: RT + RT be a polynomial. Then f is SAD if and only if f is affine linear.
Proof. Note that by definition, polynomials are analytic and definable. Moreover, every linear function p: Rm - R satisfies condition (S) in Definition 3.1 since we have l|p(tz)) [|p())] for all x E m and t E (0, oo). Conversely, let p: R" + R" be polynomial of degree at least 2 and write p(x) = (pi (Ir): ...)...... . Pr (x)) for R and polynomials pI..... Pr: Rm - IR. To prove that p does not satisfy condition (S), it suffices to prove that one of the components .I,.... Pr does not satisfy condition (S). Thus, we may without loss of generality assume that r = We write p
as p(x) (x) + PL (z) where pz is homogencous polynomial of degree Ao > 2 and p7 is
28

---

polynomial of degree strictly less than k. Fix 20 E R such that | = and pa (mo) o O. Since pr has degree strictly less than k, the limit a lim Ip (tmo))
RR exists. We have
[p(tzo)]
|p(tro))
lim sup
lup t-
tk-1
8+000
|p (Ixo))
Ipn. (1oo))
2 lim sup 8-+00
tk-1
t
th [ (2o))
lup t1
tk-1
OC.
The next lemma helps detect renrreiresentabblity of polynomials a finite dataset. We are indebted to Floris Vermeulen and Mariana Vicaria for explaining to us the proof in the case m 1. Lemma C.4. Let f: Rm x Rd - R. be definable analytic function such that for each 0 E IR, the function x >> f(0, x:) is not identically zero. Then there is an integer N E & such that the interior of the set
DN = [(I1,...,NN) 40 E : :i=:ii.........
f(0, xz) # (R"))
is conull and dense in (R we moreover assume that 1, then N can be chosen such that DN contains all tuples (#1...., IN) for which ø1, ,1....,ZN NN are pairwise distinct.
Proof. Fix N E N such that d + (m 1)N mNN Recall from [31, Chapter 4] that every
definable set X S R for k E N has well defined dimension dim(X) E {-oo, 0. ......,] k} and that dim( X) = l: if and only if has non-empty interior and only if X has non-zero Lebesgue measure [6, Theorem 2.5]. Note that the set
X = {(0,x) x)) Rd x R" [(0,x) = 0} C Rd Rm
as well its slices
(== 1f(0,)) S R
for all 0 Rdd are definable. Since each of the functions f(0. -) for O Rd is analytic and not identically zero, the identity theorem implies that each Xo for 6 E Rdd has empty interior. In particular, we have dim(Xo) I m for each A E Rd Note moreover that we have
(I. N DN = U X C (RM)A.
(11)
BER
The combination of Proposition 1.3 (iii), Proposition 1.5 and Corollary 1.6 (iii) in [31, Chapter 4] applied to (II) thus imply the inequality
dim (RR N Dx) d (m 1)N mN di
Using Theorem 1.8 of [31, Chapter 4], we even get dim
[ N \ DN)
dim ((E" DN) <
dim (IR")) In particular, (Rm)) \ DN S (R is null set with empty interior. In other words:
D2 C (R")& conull and dense.
For the second part of the lemma. assume moreover that m = 1. Then each of the sets Xo for 0 e Ed is finite. The uniform finiteness theorem [15, Theorem 4.9] implies that the sets (X9) (X9)sER4 have uniformly finite cardinality, i.e. supoers Xol < oc. The lemma then follows by picking
N supaer" Xoo + 1.
Proof of Theorem 3.4. By Lemma C.2 and Lemma C.3, for each 0 E TRd(A), the analytic definable function
d R, x No f(x)
(12)
is not identically zero. If is as in item (i) of Theorem 3.4, the theorem now follows from an application of Lemma C.4. Note that the identity theorem implies that (12) is non-zero outside of null set. Since & is loss function, the function
R - R,
+ ((No(z).
29

---

is non-zero outside of null set as well. This proves the theorem in the case that is as in item (ii) of Theorem 3.4. şl is convex combination of the two, the proof follows from the combination the two arguments.
IT
D Proof of Theorem 3.5
This section is devoted to the proof of Theorem 3.5 which follows classical proofs of the unviersal approximation theorem [16, 37, 44, 64]. For the particular case of tanh activations, the proof be found in [21]. We do not claim originality and include the proof mainly for convenience of the reader. Before going into details, we outline the main ideas of the proof below.
Idea of proof
The proof starts from the special case of fitting linear polynomials in one variable with one hidden layer and then successively builds up generality (see Theorem D.4 and Figure The starting point is that if v is an analytic activation function and Zo E R, then one can approximate linear function with slope (z() by neural networks with one hidden neuron using the formula + nz((x/n + zo). Similarly, one can inductively approximate the k-th Taylor polynomial of si using the formula x + n v/r/// + zo)- By adding hidden neurons and compensating the lower order terms of the Taylor polynomial with previous approximations, one can thus approximate polynomials of degree k using neural networks with one hidden layer and k neurons, In order upgrade this the multivariate case, we employ linear algebra argument (see Lemma D.1) that allows us to express arbitrary multivariant polynomials as linear combinations of polynomials of the form p(r,,...,,7; Fn)= = (a1æ1 + +.... d,, To extend our result multiple hidden layers, we let the other layers approximate the identity and apply the Arzela-Ascoli theorem (see Lemma D.3) to achieve compatibility the various approximations. Along the way we carefully keep track of required number of hidden neurons.
Lemma D.1. Fix integers 2 1 and m 2 0. Denote by R[Xo..... Xmhn S c[. Xm the real vector space of homogeneous polynomials of degree in variables. Then, we have dim Xml. ("++m) and moreover
R Xmin span((X, + 01
am XmX a1
am R..
To avoid confusion about the dimensions, we emphasize that this lemma consideres polynomials in m variables and not m variables.
Proof of Lemma D.I. We define the following index set.
S(n, m) {A (A Am) E Nm 10 A; n, ..... Am n) S {o....,,1
For À E S(n, m). we furthermore write An = E Ai. A basis of R[xo....... X Xm), In is given by
( Am
X ..... Xmmm
(13)
XES(n,m)
where (oo....A..) Ag!--AAm denotes the multinomial coefficient. easy to see from a stars and bars argument that dim IR [R [Xo...... [x. [S(n, m)l (*+m).
To prove the second claim of the lemma, we define linear map P: R[Xo. Xmmn +
R[Xo, ... Xm,,, on basis elements by the formula
71
X
Xd m
(Xo AiXi + .... AmXm
Am
Σ
n
Mbm X
X m
(o....., Mm
(ES(r.m)
We introduce a relation < on S(n, m) by defining that x y if and only if A pyy for some E (l...., m] and X; μ4 for all i {I....,m, m) {r}. In this case, we write A(p, X) Hr d
30

---

Using the basis (13) and the generalized Vandermonde determinant formula [13], we can calculate the determinant of P
del = del (eel((X)) ((2f) N
"nr.))/ee)wsnmmn )..ES(mm))
II
A) 0.
A.
In particular, P is surjective and have
R[Xo. ................................ Xmhn S P(R[[Xu..... X.lm,....))))))) span((Xotarx)) att..taammm (m) 1 dl,...,, am RR.
Definition D.2 (Uniform Norm). For subset X C IR" and function f: R - R, we write
Ilflx sup [f())] =EX
For sequence (Jy)seN of functions IR.h -> R. we say that (Jy)JeN converges to uniformly on compact sets, if for every compact set K C RR, we have
Ilfy fllk 3
For approximating the identity with redundant hidden layers, will also need the following elementary application of the Arzelâ-Ascoli theorem.
Lemma D.3. Let n, m, E N and les f Joo: R - TR. and 9j- 960 R - Rhm for all j E N be continuous functions such that f; - foo and - goo uniformly on compact sets, Then we have fj 1-200, foo a 9oe uniformly on compact sets.
Proof. Fix compact subset S R and By the Arzelà-Ascoli theorem, there is satisfying
U jENU(e>}
9₂(K) S BR(
Again by the Arzela-Ascoli theorem, there is 5 > 0 such that for all j E N U {co} and x, E Bn( with Ja yl < we have (x) (v) < 5 Now choose a large enough N N such that for all
j e N with j 2 N, we have
|s, - fool <
and
119; 8xolx <..
Then we have
()(0,(z)) - (g (ff(ي)(z)) - (x (1)(xx())) - Soo (0xo (x)) <e
for all x E K and j E N with j j N.
Theorem D.4. Let ni and vh: - R be a Cfyffunction with v6(a) # and let p: Rmm -> R be polynomial of degree most Let A (do..... de. 10 be neural network architecture with do = m input neurons, dy = r output neurons and activation function tb. Suppose that A is sufficiently big in the sense that
(i) there is a hidden layer < k of size de 2 ((Ttt)) - -m),
(ii) all previous hidden layers have size min(do. ... di-1) 2 dg.
(iii) all subsequent hidden layers have size min(d,+1. dz) de.
Then p can be approximated uniformly compuct sets by realization functions NºA R + Rd for
0 E _dd
31

---

Proof of Theorem D.4
one hidden layer
multiple hidden layers
n
p(x)
general
p(x)
x n+1
general
Figure 4: Structure of the special cases in the proof Theorem D.4.
Proof. We prove the theorem by considering several successively more general cases. The structure of the proof is visualized in Figure 4.
Throughout the proof, we denote for j by
Rm...r
(Nr (mr,))) 1 E R B(m++15+6+ S C"RR",
the set of all realization functions of shallow neural networks with m input neurons, j hidden neurons, r output neurons and activation 1b. Note that
RW m.i.r
D9 m.j.r
(14)
m,itj.r
for all j E We moreover denote by Rm.j.r S C(R", RR) the set all functions f: R + IR", which be approximated uniformly on compact subsets by elements of g mm..rr For a, x R we denote by (x, y) = Σi=1 0;2; their inner product.
Case 1: We have r = and = (m,, (m++m) m, 1) m
We prove Case by induction on n..
We start with the case 1. We first consider the case that is equal to the identity function idg: IR + R, id)(x)= = x. By the assumptions on y there exists 2g E with a p((2) & Observe that the functions (Jz) C R defined by
i (b(x/j + mo) (20)), R
fx)
a
converge along - idg uniformly every compact subset I IR. Indeed, using continuity of v,, we have for every j E N that
to+ (f)ddtt -
sup [x( x =El
sup ae!
i (1) -5) di
sup
zel
2o
(V'(t) al dt
sup
|oc|
wEl
1
sup
|
sup
1 (t) a:
|a||
wel
/E [to
This proves that idg R
32

---

Now let
p: P: B. p(x) (a, x) + ag d + +... am + ag
be an arbitrary polynomial of degree most with E Tm and do R. Choose sequence (JG)JEN S Ri,,, converging to idg uniformly compact subsets. Since p is continuous and fg 2 idg uniformly on compact sets, the sequence (fz))eN E RM,,, defined by
(x)) = fs((a, 2) + on)
converges to p uniformly on compact sets. This proves that p E 1,1. Noting that (*+**) this concludes the case = 1.
We now prove Case 1 for + 1, assuming that we have proved Case for n. We first consider the case that is given by the (n + 1)-th power function id RR + R, idg (x) x²+1 By the assumptions on tb, there exists æ0 IR with
= 4/(+2) (50)) (n 1)!
#0.
(15)
We denote by
Tw th: R R, Tab(z)= = = (20) j=U j!
(x mo)
(16)
the Taylor approximation of Ty of order n. We define functions (g))SEN S C=+1 (R, by j+1 (vbar x/5) Tn Tnv( + xxj)), DER
gj(2)
(17)
a
and claim that
sup [g5(2)) xnt2| 1=00 0
(18)
=e1
for every compact subset C I.
Indeed, by the Lagrange form of the remainder in Taylor's formula, for every j E N there is E, E [ru, To + x/j] satisfying
j (on+D((E)
sup 19, (I) -++ sel
=+
sup #E7
(n 1)! {
(1(n+1) (E))
n+
sup
()(n+1) (Ii)
j-000 0,
where at the last step we have used continuity of (f((+1) and compactness of I.
Now fix an arbitrary polynomial p: RT + E of degree at most n + We write p Po + Pı where Po is a homogeneous polynomial of degree n + and is polynomial of degree at most n. By Lemma D.1 (applied to n <- n + 1 and m + m 1), we can find a E I for i 1...., (a+m) such that
m-11
(m+m))
po(x)=
x))
Vo R.
(19)
Fix an increasing sequence (Ky)jeN of compact subsets of I such that Rm = -,ee Ky. Since vi satisfies the assumptions of the theorem for n + 1, then it does for n. In particular, for every j E N, can find function 4
DR (..(" -m,1
such that
in+1
sup =E
(2) - +
Inbi((at),2))
<
(20)
33

---

where Th is defined in (16).
We now define a sequence of functions (Jz)jeN CC+t(IR IR) by
ju+1 10 Io +
[τ²) - (r(r)) +
= ((aal)
where do E R are defined as in (15).
Note that by definition, the first summand of fy defines an element of R (.("""") -m,1 mm
while the
second summand defines an element of Rd m. (=im),1" Thus, by (14), fy defines an element of
R m.
(++1+m)
) m,,1
Now let K S mh' be compact subset and j E N. Using (17) and (19) at the second step, and the combination of (18) and (20) at the last step, we get
sup =ek [fz(z) p(x)
)+m)
jutl
- ((((),22))
sup
(f(x)
v
2o + j
po()))pi(6 (I))
k
(ntm)
(m+m)
j jn+1
sup =
Qy (z) (2 +
T (a
Σ
()(((((),)) - (fal x)))++))))))))
(u+m)
jn+1
S sup =EK
(1(x) (1
T" ("
+ sup
95((a(), x)))+t)
JEK
1-100 0.
This proves that E R. Rm, P0ttt"tt-m (=+1-m) )))---mmmmm m,
and finishes the proof of Case 1.
Case 2: We have A = (m,r (("tm) m) ,r, This case follows directly from Case 1 by applying Case to each of the components Pi.....Pr Pr of the neural network architectures used the approximation on top of each other.
(pı..... Pr): R -> R and stacking
Case 3: We have a general architecture A = (m, di 1, v) as in the statement of the theorem.
We fix < i < such that
(i) d 2 r (( (n+m) m).
(ii) min (do, dj-1) 2 do.
(iii) min (ditl dx) 2 dr.
By Case 2, there is sequence (1, jjeN C R (m."(("="))m)) converging to p uniformly on compact subsets of R" By componentwise application of Case 1, for any 0 < ( < i, there is a sequence (r(")ser C Re mm converging to idæm uniformly on compact sets. Similarly, for any i << / < k, there is a sequence (1(1)) C Ry rr, converging to ider uniformly compact sets. By Lemma D.3, we have
f c p,0) j
uniformly on compact subsets of Rm Moreover, by the assumptions on do....,dy d2 we have 1100 DR ,.
11 ....o
27
Da
mmm C (N1 10 Rd(()).
(.r((" -m))r
34

---

see Figure for an illustration. This finishes the proof of the theorem.
A
fz
linear
Eneer
linear
linear
layar
Figure 5: An illustration how neural network with hidden layers can realize functions of the form f2 fi where is the realization of a shallow neural network with n neurons and fi are realizations of shallow neural networks with on hidden neuron. Dotted lines correspond to weights that are set to zero.
Proof of Theorem 3.5. Theorem 3.5 follows immediately from Theorem D.4 and the fact that l is probability measure since uniform convergence on compact sets implies L -coovvrreeece with respect to probability measures.
Remark D.5. The bounds on the required number of neurons in Theorem 3.5 are not sharp. many cases, one can approximate polynomials with far fewer neurons. This can be done by following the steps in the proof of Theorem D.4 in detail and depends on several factors such as
(i) the number of nonzero homogeneous parts of the polynomial,
(ii) how many nonzero coefficients needed express the homogenous polynomials in the basis from Lemma C.4,
(iii) the zeros of the Taylor coefficients of the activation function.
Moreover, the number neurons can be reduced drastically if one allows for deep neural networks and rewrites the polynomial as composition of casier polynomials. The polynomial p(x, y) x'y² for example can be rewritten as the composition of the functions (z, y) + (x y and (z. y) + zy and can therefore be approximated by realization functions of the architecture A (2, 16, 4, 1) for any SAD activation function v:
E Experimental Details
In this section, we detail the experimental setup. All experiments were conducted on single Tesla V100 GPU, with each training run requiring between 10 and 20 minutes. Overall, the computational demands of our study remain modest.
Polynomial target function
We train our neural networks to approxomate the polynomial target function p: [a, bjd + where [-,, 1,4, The target functions differ based on the input dimension dg:
[a, 6/dh
(i) for do 1,
p(z1)= = = - -2x + 2 + 3x
2n 5,
(ii) for do = 2,
p(z1, x2) = =z - 40,10 + 3z x rı +2,
and
35

---

(iii) for do =4 4,
p(11 F2, F3, x.a)
5- + x2
2 22
2/2323 + 2 4xza2 + 3x =3z21 + + 3.
In the one-dimensional case, the neural network architecture consists of three hidden layers with widths (d1, d2, d3) (10, 20, 10). For the higher-dimensional settings increase the dimensions to (di, d2, da) = (20, 40, 20). For gradient descent, we use the full training dataset of 10,000 randomly sampled points each step. For Adam, batches of 100 samples are drawn from the same dataset. We use a learning rate of 0.001 for gradient descent and 0.005 for Adam. Note that the plotted loss corresponds the training loss. Our theoretical results, indeed, show that the norm of the neural network parameters diverges while the training loss converges to zero. To reduce noise, we plot the exponential moving average of the training loss with smoothing factor a 0.95, averaged over 20 independent random initializations. We also construct test set as a uniform grid over the domain. The test loss exhibits a similar qualitative behavior. We include the plots of the test losses in the GitHub repository
8.0
9.5
7.2
Sehala
8.5
"gantalc Seigeet
9.0
100003
200000
30000D
400000
500000
130003
200000
330000
400000
500000
100000
200003
300000
400000
500000
10
10-1
10
10-
10.2
10-3
100000 200000 300000
400000
500000
100000
200000 330000 480000
500000
100000 200000 300000 400000 500000 gradiient steos
gradient steps
gradient steps
(9)
(b)
Figure 6: Approximation of polynomial larget functions using different activation functions and GD algorithm. From left to right: 1-dimensional input case, 2-dimensional input case, 4-dimensional input case. The y-axis is rescaled exponentially in order to illustrate the logarithmic growth of the parameter norms.
Real world learning task
To solve the PDE problems we use the Deep Kolmogorov method. This approach approximates the solution using neural network, leveraging the Feynman-Kac representation of the PDE solution. Specifically, the neural network is trained to match the expected terminal value of the underlying stochastic differential equation (SDE), estimated via Monte Carlo sampling. The method provides a generalizable solution across input space, enabling instant evaluations of the PDE once the network is trained. This is particularly advantageous in high-dimensional settings, where traditional solvers suffer the curse dimensionality. We refer to [5] for a comprehensive description of the method.
In the case of the Heat PDE we consider a neural network with architecture (do, d1, d2, d3, dy) = (10, 50, 50, 50, 1) while for the Black-Scholes PDE (do, d2,d2, d3, d4) (10, 200, 300, 200, 1). For the latter, we set the interest rate
0.05, the cost
"The reader might notice that the architecture in the four dimensional input case does not satisfy the crude bounds of (i) from Theorem 3.5. However, as pointed out in Remark D.5, these are only worst case bounds. In specific examples like these, approximate the given polynomials with significantly smaller architectures.
36

---

of carry 0.01, the strike price K 100 and define the volatility vector as (0.1000,0.1444, 0.1889, 0.2333, 0.2778, 0.3222, 0.3667, 0.4111, 0.4536, 0.5000). The loss plots in Figure 3 report the relative mean squared error between the neural network prediction and the reference solutions. For the Heat PDE, the solution is available in closed form, while for the Black-Scholes PDE we use high-precine Sighh-preisionn Monte Carlo approximation by averaging over 1000 independents rounds, each consisting of 1024 sample paths. Importantly, the reference solutions are not used during training. At each training step, single Monte Carlo sample is employed the target solution. We train the neural networks using the Adam optimizer with a learning rate of 0.005.
For the MNIST classification task, we use a neural network with architecture (dg. d1, d2, ds) (784, 256. 256, 10) trained with a standard cross-entropy loss. Optimization is performed with the Adam optimizer and a learning rate of 0.001. As in the previous experiments, we report the training loss the figure. Training is stopped after 30000 steps to prevent overfitting, resulting in a final test accuracy of 97.93%.
In both the MNIST and PDE examples, we smooth the plots by plotting an exponential moving average of the loss with smoothing factor a = 0.95, averaged over 5 independent random initializations.
37


Reviews:
[{"review_id": "vvuifsDwYF", "reviewer": "Reviewer_3GA8", "summary": "The paper investigates gradient flows for loss landscapes of fully connected neural networks trained on polynomial targets with commonly used activation functions. The main theoretical result is a dichotomy: gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value, with the latter occurring for polynomial targets under mild assumptions. The key proof relies on the geometry of o-minimal structures from model theory and introduces SAD (Sublinear Analytic Definable) activation functions. Empirically, the authors run simulations on toy datasets and extend to more realistic tasks (e.g., MNIST) to demonstrate this phenomenon / main theoretical results.", "questions": "I am mostly curious if the theoretical analysis suggests any particular insights for the simulation results in Section 4. I'm listing several questions I have below.\n- In line 292, the authors mention that Adam shows better convergence, while the SGD results in Figure 1 seem to indicate that the parameter norm plateaus. Could the authors elaborate on their justification in lines 292-298? I assume that SGD with a small learning rate is closer to the theoretical analysis (gradient flow) than Adam updates.\n- Does this behavior still occur with extremely poor initialization or with other activation functions (e.g., Relu or even identity)? I speculate that they will show the same behavior as in Figure 2. If so, what is the practical significance of the proposed work?", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Note: I don’t have the necessary background to evaluate the correctness/significance of this work, hence my confidence score is 2. I will communicate with other reviewers to address any misunderstanding.\n\nStrengths:\n- The paper is generally well-written, and the source code is provided for reproducibility.\n- A detailed experimental setup is provided in Appendix E.\n- While I cannot comment on the correctness/significance of the theoretical derivations, the derivations and proofs are well organized (e.g., Figure 4).\n\nWeaknesses:\n- The empirical results don't convincingly demonstrate that the theoretical findings translate into practical settings. For example, it is well known that parameter norms continue to increase during training (motivating techniques like L2 regularization), making it difficult to understand whether the theoretical insights are specifically captured in Figure 1. Does this phenomenon fail to occur with extremely poor initialization or with activation functions such as ReLU (as predicted by theory)? Please see the questions section below.\n- Similar concerns apply to the empirical results presented in Section 4.2. At the moment, the theoretical analysis does not appear to capture any important practical considerations that aren't already well-understood.\n- The theoretical results apply only to gradient flows and smooth activation functions, which significantly reduces the practical relevance.\n\nMinor:\n- It would be helpful to have detailed related work, e.g., other applications of o-minimality to neural networks.", "was_revised": true, "final_justification": "The authors have convincingly addressed my questions (e.g., it makes sense that the parameter growth happens when we have scale invariant models such as with batch norm.)", "rebuttal": "The authors thank the reviewer for the detailed and constructive feedback.\nBelow, we address each of your questions in turn.\n\n- To directly answer the first question: SGD with a sufficiently small learning rate indeed closely follows the continuous gradient flow, while Adam’s adaptive rescaling obscures the slow divergence predicted by our theory.\nThe impression that SGD plateaus is due to the fact that the norms of the gradients along the gradient flow decay like $O(1/\\sqrt{t})$, where $t$ denotes training time, which follows from equation (7). This implies that the parameter norms can grow at most like $O(\\sqrt{t})$ and therefore look like they are plateauing although they are slowly diverging to infinity. \nAdam’s second order momentum‐based normalization effectively re-scales each gradient coordinate to $O(1)$. Therefore, we justify the use of Adam as a more effective variant of SGD, offering faster convergence in scenarios where SGD would require more iterations to achieve similar results.\\\nMoreover, we will include new simulations illustrating gradient descent in the multidimensional input case for approximating polynomial target functions. These experiments will both align closely with our theoretical findings and provide a clearer, more convincing demonstration of parameter norm growth during training.\nIn particular, we will rescale the $x$-axis to demonstrate that for every activation function there exist $a,b$ such that the asymptotic trajectories of the parameters norm behave like $a \\ln(x) + b$.\\\nFinally we wish to emphasize that similar sublinear‐growth of parameter norms has been observed in homogeneous models, e.g. in [49, Theorem 4.3], where gradient descent on an L-layer logistic regression model yields $\\|\\Theta(t)\\|=O(\\ln(t)^{\\frac 1 L})$.\n\n- Concerning the second question: We appreciate the reviewer's interest in how the experiments would look with other activations or extremely poor initialization. \nFirst, regarding poor initialization, we found that in our setting, SGD rarely gets stuck in local minima purely due to initialization, owing to the simplicity of the model and loss landscape. While it is possible to force standard gradient descent into local traps through carefully chosen initializations, we observed that using a sufficiently large dataset effectively prevents such issues by smoothing the loss surface and guiding the optimization away from problematic regions. Of course, in more complex architectures or highly non-convex settings, initialization may have a stronger effect, but in our setting, we did not observe significant sensitivity.\\\nRegarding other activation functions, we verified the same phenomenon with the SiLU, which satisfies our SAD conditions. In contrast, with the identity activation, the network fails to converge to the target function, and we observed a decrease in the parameter norm during training. Note however, that we excluded polynomial activation functions from Theorem 3.5 and Corollary 3.6.\nAlthough ReLU falls outside our SAD framework, prior work on homogeneous ReLU networks [49] proves similar divergence behaviors, and our preliminary experiments confirm divergent trajectories.\\\nFinally, we note that unbounded parameter‐norm growth is a relatively specialized phenomenon. In many practical settings involving non-homogeneous activations or more complex loss landscapes, parameter norms tend to fluctuate during training and stabilize over time. In the literature, pronounced divergence has been documented primarily in scale‑invariant networks, where scaling the weights leaves the model’s predictions unchanged (only their magnitude shifts).\n\nConcerning the weaknesses, we emphasize that our main contributions are the advance in the mathematical tools used in the theoretical analysis of machine learning and the understanding of training dynamics of neural networks.\\\nMoreover, we recognize that results for SGD would be more directly applicable than results for gradient flows. We will pursue these in future work. We do however believe that the gradient flow setting is an important proof of concept and resembles the gradient descent setting with a small step-size. The treated smooth activation functions however are highly relevant in practice. In particular the smooth GELU function which we cover is frequently used in scientific computing problems as well as in Transformer models like BERT and GPT-3.\n\nFurthermore, we agree that it would be helpful to have more detailed work on the applications of o-minimality to neural networks. The upshot is that the *convergence analysis* for definable objective functions, under the assumption that the parameters stay bounded, is very well understood (see for instance references [7, 3, 15, 18, 6, 35, 31] in the paper and [A, B] below). In particular, the convergent part of our Theorem 2.8 is essentially well-known, a fact that we will emphasize in the revision. Other applications of o-minimality include alignment and directional convergence [34, 42, 49] or bounds on the VC-dimension [38, 39]. The significant contribution of our paper is that it is, to the best of our knowledge, the first paper using o-minimal structures to achieve *divergence* results in the training of neural networks. This is the content of our Theorem 3.4 and our main result Corollary 3.6. Another key contribution is that our definition of *SAD* activation functions provides a convenient mathematical framework that simplifies and conceptualizes many arguments while covering most smooth activation functions used in practice.\n\n- [A] Dello Schiavo Lorenzo, Jan Maas, and Francesco Pedrotti. \"Local conditions for global convergence of gradient flows and proximal point sequences in metric spaces.\" Transactions of the American Mathematical Society 377(6):3779-3804, 2024.\n- [B] Hedy Attouch, Jérôme Bolte, Patrick Redont, and Antoine Soubeyran. \"Proximal alternating minimization and projection methods for nonconvex problems: An approach based on the Kurdyka-Łojasiewicz inequality.\" Mathematics of Operations Research, 35(2):438–457, 2010."}, {"review_id": "6C2KXlqX6X", "reviewer": "Reviewer_auYg", "summary": "The submitted manuscript investigates gradient flows in the context of training fully connected neural networks. The first contribution establishes that, under a local Lipschitz continuity assumption on the gradient of the loss function, the trajectory of the gradient flow either converges to a critical point or diverges. In the divergent case, however, the value of the loss function converges to an asymptotic critical value.\n\nThe second main contribution provides a divergence result for gradient flows. Specifically, in the setting of subanalytic definable neural networks, the authors show that there exists no finite parameter configuration at which the loss achieves zero. Nonetheless, there exists a diverging sequence of parameters along which the loss converges to zero. Thus, while the global infimum of the loss is zero, it is only attained in the limit at infinity.\n\nAs a result, the authors argue that in such settings, whenever the loss at a parameter configuration is sufficiently small, the trajectory of the gradient flow necessarily diverges, yet still minimizes the loss. These theoretical findings are supported by two numerical experiments.", "questions": "- How does the convergence result for the gradient flow in this paper compare to other results in the literature. For example, how does it compare to that of Dello Schiavo and Maas (Local Conditions for Global Convergence of Gradient Flows and Proximal Point Sequences in Metric Spaces, Transactions of the American Mathematical Society, 2024)? \n- In Appendix D, it is mentioned that the proof is partially inspired by [28]. Could you elaborate on this connection? Specifically, what aspects of the argument are derived from [28], and what constitutes the novel contribution in the presented approximation result?\n- How do Theorem 3.4 and Theorem 3.5 connect to other findings of neural network approximation results?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper extends standard convergence results to stationary points from the setting of global smoothness to one of local smoothness.\n- The convergence analysis presented applies to the training of neural networks with a broad class of activation functions\n- Under the SAD setting the authors, the authors establish a neural network approximation result for polynomials, showing that neural networks can approximate such functions arbitrarily well.\n- By combining these results, the authors demonstrate that while the gradient flow diverges, the loss function along the trajectory converges to zero.\n\n**Weakness:**\n- The description of Theorem 2.8 at certain points slightly overstates the strength of the result. While it is true that most convergence proofs rely on global smoothness assumptions, and it is indeed valuable to see results under local smoothness, the focus here is restricted to gradient flow, i.e., the continuous-time setting. Thus, the resulting statement is expected. The continuous-time limit significantly simplifies the analysis, as it avoids the complexities associated with discrete-time methods—particularly, the need to carefully manage step sizes to ensure descent. Such considerations are inherently bypassed in the continuous-time framework, which limits the practical implications for gradient-based optimization algorithms.\n- I do not fully agree with the interpretation of Figures 1 and 2. Based on the presented plots, it is not clear that the norm of the neural network parameters actually diverges; instead, they appear to approach an upper bound. This observation directly supports the concern raised in my previous point: in the discrete-time setting, there is no guarantee of monotonic decrease in the loss function, and divergence is not automatic. A more convincing evaluation would require running the optimization for more iterations to reveal any long-term trend. Additionally, adjusting the scale of the y-axis could improve the clarity and interpretability of the plots. Moreover, it would be helpful to include a plot of the gradient norm as well.\n\nTo summarize, I believe the true highlights of the paper are Theorems 3.4 and 3.5. That said, approximation theory for neural networks lies outside my primary area of expertise, so I cannot fully assess the novelty or practical impact of these results. In particular, the authors have not situated their findings within the broader literature on neural‐network approximation theory (beyond the standard universal‐approximation theorems).", "was_revised": true, "final_justification": "The authors have addressed all of my comments satisfactorily. In particular, my primary concern - how the main results of Theorem 3.4 in neural-network approximation theory relate to the broader literature - has been fully resolved. I appreciate the manuscript’s significant contribution and am pleased to raise my score to 5 (accept).", "rebuttal": "We thank the reviewer for the detailed review and suggestions for improvement. We comment on the raised points below.\n\n- Concerning the first weakness raised by the reviewer: We agree that 'the convergent part' of Theorem 2.8 is essentially known and will clarify this in the revision. Our novel contributions are Theorem 3.4 and the resulting Corollary 3.6, which we believe offer new insights into the dynamics of neural network training. While convergence analysis for SGD is generally more difficult than for gradient flows, it is important to note that in the non-convex setting, convergence to critical points is not automatic even in the gradient flow setting. In fact, there exist smooth functions for which gradient flow trajectories exhibit non-convergent behaviors, such as spiraling or limit cycles.\\\nConvergence to critical points is precisely where o-minimality assumptions and Lojasiewicz-type inequalities are commonly used. Another difficulty that arises for both gradient flows and SGD is the possibility of running into one of possibly infinitely many local minima. This possibility is ruled out by our Theorem 2.8 (see item (v)) - a nontrivial implication of the o-minimality assumption.\n\n\n- Concerning the second weakness: \nThe apparent plateauing of SGD in Figure 1 is due to the fact that the norm of the gradients along the gradient flow decays like $O(1/\\sqrt{t})$,  where $t$ denotes training time, (see equation (7) in the paper). This implies that the parameter norms can grow at most like $O(\\sqrt{t})$. Since SGD is an approximation of the gradient flow, this means that the divergence of the parameter norms is necessarily very slow and can easily be confused with a plateau. Adam’s adaptive rescaling effectively normalizes gradients to $O(1)$ - yielding an almost linear increase in the parameter norms. Therefore, we justify the use of Adam as a more effective variant of SGD, offering faster convergence in scenarios where SGD would require more iterations to achieve similar results.\\\nMoreover, we will include additional simulations illustrating gradient descent in the multidimensional input case for approximating polynomial target functions. These experiments will both align closely with our theoretical findings and provide a clearer, more convincing demonstration of parameter norm growth during training.\nIn particular, we will rescale the $x$-axis to demonstrate that for every activation function there exist $a,b$ such that the asymptotic trajectories of the parameters norm behave like $a \\ln(x) + b$.\\\nWe also appreciate the suggestion to include plots of the gradient norm. These will help illustrate how the Adam optimizer mitigates gradient vanishing, producing significant updates.\\\nFinally we wish to emphasize that similar sublinear‐growth of parameter norms has been observed in homogeneous models, e.g. in [49, Theorem 4.3], where gradient descent on an L-layer logistic regression model yields $\\|\\|\\Theta(t)\\|\\|=O(\\ln(t)^{\\frac 1 L})$.\n\n\n- Concerning the first question by the reviewer: The essential distinction between our result and other convergence results lies in our *divergence proof* for gradient flows along with the introduction of a threshold $\\varepsilon$ for the divergence/optimality guarantee. As pointed out above and to be emphasized in the revision, the convergent part of our Theorem 2.8 and its proof summarizes a phenomenon that is very well-understood in the literature. The reference cited by the reviewer is one such instance, offering a general convergence result under broader assumptions (non-smooth functions on arbitrary metric spaces) but significantly differs from our setting by assuming conditions that enforce boundedness of the parameters.\n\n- Concerning the second question of the reviewer: \nIn the reference [28] the authors focus on the approximation of $x^2$ using only shallow neural networks (one hidden layer) and fixed activations. They demonstrate divergence for hand-constructed sequences of network parameters. These constructions are related to our general Taylor expansion argument in the proof of Theorem D.4. Our work generalizes this: we consider arbitrary polynomial target functions, deep networks, and general SAD activation functions, and we prove divergence for gradient flow trajectories, not just for manually chosen parameter sequences converging to the loss infimum.  \n\n\n- Concerning the third question of the reviewer: \nWe emphasize that our main contribution is Theorem 3.4 as this is, to the best of our knowledge, the first *divergence* result for neural network optimization crucially using o-minimal structures and the associated rigidity properties. Other novel contributions include the first application of definable dimension theory to neural network optimization as well as the isolation of a convenient class of activation functions (SAD activation functions) that both allow a unified mathematical analysis and includes most of the commonly used smooth activation functions. \\\nWe will clarify this and the relation of our work within the broader approximation theory literature in the revision.  We list some of the relevant references below. Apart from the standard universal approximation theorems which use increasing network sizes [B, C, D, E], we will also mention the surveys [F, I] and the results in [A, G, H]. The reference [A] in particular approximates polynomials in Sobolev norm with fixed-size shallow neural networks with sigmoid activation - a special case of our Theorem 3.5. In the non-smooth setting, there are moreover many results on approximation rates achieved by ReLU-networks, see [J, K, L, M, N]. \n\n\n- [A] De Ryck, Tim, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks 143:732-750, 2021.\n- [B] Cybenko, George. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4):303–314, 1989.\n- [C] Funahashi, Ken-Ichi. On the approximate realization of continuous mappings by neural networks. Neural Networks, 2(3):183–192, 1989.\n- [D] Hornik, Kurt, Stinchcombe, Maxwell, and White, Halbert. Multilayer feedforward networks are universal approximators. Neural Networks, 2(5):359–366, 1989.\n- [E] Leshno, Moshe, Lin, Vladimir Ya., Pinkus, Allan, and Schocken, Shimon. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks, 6(6):861–867, 1993.\n- [F] DeVore R, Hanin B, Petrova G. Neural network approximation. Acta Numerica 30:327-444, 2021.\n- [G] Petersen, P., Raslan, M. \\& Voigtlaender, F. Topological Properties of the Set of Functions Generated by Neural Networks of Fixed Size. Found Comput Math 21:375–444, 2021.\n- [H] Bölcskei, H., Grohs, P., Kutyniok, G., Petersen, P. Optimal approximation with sparsely connected deep neural networks. SIAM J. Math. Data Sci. 1(1):8–45, 2019.\n- [I] Berner J, Grohs P, Kutyniok G, Petersen P. The Modern Mathematics of Deep Learning. In: Grohs P, Kutyniok G, eds. Mathematical Aspects of Deep Learning. Cambridge University Press:1-111, 2022. \n- [J] Petersen, P., Voigtlaender, F.: Optimal approximation of piecewise smooth functions using deep ReLU neural networks. Neural Netw. 108:296–330, 2018. \n- [K] Herrmann, Lukas, Joost AA Opschoor, and Christoph Schwab. Constructive deep ReLU neural network approximation. Journal of Scientific Computing 90.2(75), 2022.\n- [L] Yarotsky, D.: Error bounds for approximations with deep ReLU networks. Neural Netw. 94:103–114, 2017.\n- [M] Daws, J., Webster, C.: Analysis of deep neural networks with quasi-optimal polynomial approximation rates, 2019. ArXiv: 1912.02302\n- [N] Opschoor, Joost AA, Philipp C. Petersen, and Christoph Schwab. Deep ReLU networks and high-order finite element methods. Analysis and Applications 18.05: 715-770, 2020."}, {"review_id": "3r6KgQFIVw", "reviewer": "Reviewer_7NML", "summary": "The paper discusses gradient flows (GF) for neural networks with definable activations and loss functions. First, using Kurdyka-Lojasiewicz inequality, the author gives a dichotomy of convergence of GF for such neural networks. This is basically a summary of existing results. Second, the author studies  fitting polynomials (usually with degree $\\ge 2$) with SAD networks. He/she shows that for sufficiently large model and sufficiently large sample size, we have 1) 0 is an asymptotic critical value of the loss, 2) 0 is not a critical value of the loss, and thus 3) GF diverges to infinity. Experiments are presented and limitations of the works are discussed.", "questions": "1. Is it Okay to mention someone’s real name in the paper?? \n2. Even if GD is not used widely in practice, I suggest performing experiments using GD. In fact, this is necessary based on the authors’ plan for the paper, as experiments “are intended as a proof of concept”, mentioned in 1.3. \n3. Some typos: \n    Typo at line 188-189: We write $d\\mu(x) = p(x) dx$ if $\\mu$ is given by $\\mu(A) = \\int_A p(x) dx$ \n    Typo at line 697: should mention that the $p_0, p_1, …$, etc. are homogeneous polynomials. \n    Typo at line 827: should be $T_n$ instead of $T^n$ in equation (20).", "limitations": "Yes. Discussed in the article.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: \n1.  The paper has very clear structure and provides sufficient details for readers. \n2.  The paper introduces a useful tool, i.e., theory in o-minimal structure, to the study of machine learning.   I believe that this tool greatly helps us understand the behaviors of neural networks. \n3.  The results shown in the paper, especially the divergence of GF for fitting polynomials, captures, and explains the phenomenon in application in an unambiguous way, meanwhile having sufficient generality in mathematics.\n\nWeakness: \n1.  My major concern is that the experiments are performed using SGD and Adam, which usually differ a lot from GF/GD. \n2.  My second concern is about contribution. The authors mainly discuss the dichotomy of GF (see 1.1.Contribution) in analytic setting. But when the loss is analytic, this behavior of GF has been proven by previous works, such as [1]. And even for general definable functions, it follows quickly from KL inequality. So it might not be appropriate to put it as a contribution of this paper. \n3. I found some typos. See below. \n\n[1] P. Absil, R. Mahony, B. Andrew, “Convergence of the Iterates of Descent Methods for Analytic Cost Functions”, SIAM Journal on Optimization, 2005, 16(2), pp. 531-547.", "was_revised": true, "final_justification": "Final score: 5. The paper is excellent overall and is clearly written.", "rebuttal": "We thank the reviewer for the very thorough reading and the positive and constructive report. Below, we address the concerns and questions raised in detail.\n\n- Concerning the first raised weakness: We agree that SGD and Adam differ from GF and GD in general.  As noted by the reviewer, the main contribution of this work is the development of new mathematical tools for the analysis of training dynamics of neural networks. While we do believe that our results indicate the practical insight that diverging parameters can sometimes be desirable for performance, we acknowledge that our experiments are rather limited and mainly serve as a proof of concept. To motivate the use of Adam in the experiments we point out that the square-integrability of the gradient norms (see equation (7)) implies that the gradient norms decay like $O(1/\\sqrt{t})$ and thus the model parameters can grow at most like $O(\\sqrt{t})$. Adam’s adaptive rescaling effectively normalizes gradients to $O(1)$ per coordinate, making the divergence behavior more visible.\\\nMoreover, we will include additional simulations illustrating gradient descent in the multidimensional input case for approximating polynomial target functions. These experiments will both align closely with our theoretical findings and provide a clearer, more convincing demonstration of parameter norm growth during training.\\\nIn particular, we will rescale the $x$-axis to demonstrate that for every activation function there exist $a,b$ such that the asymptotic trajectories of the parameters norm behave like $a \\ln(x) + b$.\n\n\n- Concerning the second raised weakness: We fully acknowledge that the convergent part of Theorem 2.8 is well-known (see the reviewer's reference and [7, 3, 15, 18, 6, 35, 31] in our paper). We will make this clearer in the revision and emphasize that our main results are Theorem 3.4 and Corollary 3.6 which establish divergence under new conditions and provide novel insights into the asymptotic behavior of gradient flow.\n\n- Concerning the first question: We confirm that all identifying information and acknowledgments have been removed from the submission in accordance with the NeurIPS anonymization policy. \nIn one instance, we cite a mathematical fact attributed to V. Grandjean. We included this currently unpublished fact with his permission and mentioned his name in order to respect his intellectual property. We strongly emphasize that this information cannot be used in order to identify any of the authors and that there is no conflict of interest that could possibly arise. \n\n- Concerning the second question: We agree with the reviewer and appreciate this constructive suggestion. We will include additional experiments with deterministic gradient descent, small learning rates and a large number of steps in order to match the gradient flow behavior as closely as possible.\nThe preliminary simulations are consistent with the expected behavior.\n\n- We warmly thank the reviewer for reading our submission so thoroughly and for for pointing out the typos."}, {"review_id": "hRoBl7pVBj", "reviewer": "Reviewer_hwZX", "summary": "In Section 2, the authors recall definitions of o-minimal structures (families of subsets satisfying certain algebraic properties) and functions that are definable with respect to such structures, as well as identify many examples of activation functions which are definable. The authors conclude the section by stating their dichotomy theorem (Theorem 2.8): a network with definable activations, optimized using gradient flow to fit a definable target function using a definable loss on either an empirical or absolutely continuous input measure, must either converge to a critical point of the loss or diverge with loss approaching an asymptotic critical value.\n\nIn Section 3, the authors consider a special case of definable functions, namely sublinear analytic definable (SAD), which are exemplified by typical activations and losses. They state their Theorem 3.4, which says that there are no global minima for the problem of fitting a network with SAD activations to a polynomial of degree at least 2 on a sufficiently large dataset. Theorem 3.4 is counterbalanced by Theorem 3.5, which says that any polynomial can be approximated arbitrarily well by a sufficiently large (with respect to the degree of the polynomial) neural network on compact sets. The results are synthesised into the main result of the paper, Corollary 3.6, which states that in fitting a sufficiently large neural network to a polynomial of degree 2 using gradient flow on a sufficiently large dataset starting from a point with sufficiently small loss, the parameters necessarily diverge.\n\nIn Section 4, the authors provide some numerical support for their divergence theorem.", "questions": "I am willing to raise my score to an accept if the authors can answer the following questions.\n1. Why is the result significant, and how is it a significant advance over prior work in this area?\n2. In a couple of paragraphs, can you summarise how the assumptions made imply the result?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n - The paper is exceptionally well-written and easy to read.\n - The approach taken in this paper of using abstract, formal properties to prove theorems is underutilised in the literature. As a consequence, the theorems are relatively general in their assumptions compared to most other papers on the optimisation of deep networks.\n\nWeaknesses:\n - The paper lacks any \"idea of proof\". It is not clear at all in the paper how the assumptions made are used in the derivation of the theorems. The proofs in the appendix, while complete as far as I can tell, also seem to lack a high level idea of the proof.\n - There is not much specific said about the relation of this work to prior works. In particular, as someone with little knowledge about the employment of o-minimality in optimisation, it is difficult for me to judge the novelty of the work.\n - It is not made clear by the authors why the result is significant.", "was_revised": true, "final_justification": "I was already very impressed by this paper. My only concern was the significance of the results, however these have been adequately addressed by the authors in their rebuttal.", "rebuttal": "We thank the reviewer for the constructive comments and for the opportunity to clarify the significance and methodology of our work.\n\nRegarding the significance of our results: The core contribution of this work is to introduce new mathematical tools — specifically, o-minimal geometry and definable dimension theory — into the analysis of divergence phenomena in neural network optimization. These tools allow us to study optimization dynamics in a general setting with minimal reliance on the explicit functional form of the neural network.\\\nThe principal novelty lies in Theorem 3.4 and the resulting Corollary 3.6, which together provide, to our knowledge, the first proof of divergence of parameters during training of neural networks using o-minimal techniques. While divergence phenomena are empirically observed and have been studied in special cases (e.g., shallow networks with specific activations), our results generalize and formalize this behavior in a broad setting that includes many commonly used activation functions. In particular:\n   1. Theorem 3.4 establishes the non-existence of global minimizers for networks with SAD activations trained on polynomials of degree $\\geq 2$ over large datasets. Only very special cases of this result were previously known, e.g., [28] for shallow networks fitting $x^2$, and [De Ryck, Tim, Samuel Lanthaler, and Siddhartha Mishra. On the approximation of functions by tanh neural networks. Neural Networks 143:732-750, 2021] for tanh shallow networks.\n   2. Corollary 3.6 then shows that parameter divergence is inevitable in such settings, providing rigorous theoretical support for an empirical phenomenon that had remained poorly understood in generality.\n\nAdditionally, we isolate a flexible and practically relevant class of activation functions — SAD (Sublinear Analytic Definable) — that both allows unified analysis and includes most smooth activations used in practice.\nOur findings suggest an important insight: parameter divergence is not necessarily harmful. On the contrary, it may be required to achieve low loss in certain regimes.\n\nA significant advance over prior work is our abstract, theory-driven approach. While most previous analyses are limited to specific architectures, losses, or activation functions and rely on direct computation, our use of o-minimality allows us to obtain general results in a uniform, mathematically principled way. This opens the door to future applications of definable geometry in deep learning theory and contributes to building a more robust understanding of neural network optimization.\\\nIn the revision, we will expand our discussion of related work, including references to results known to experts (e.g., the convergence part of Theorem 2.8) and clarify which results are new or more general than what has appeared before. More information about other results in the literature can be found in the rebuttals to other reviewers.\n\nWe now answer the second question by giving a summary of the ideas of proof. \n1. **Proof of Theorem 2.8**: The proof of Theorem 2.8 to a large extend follows standard techniques and is only given for convenience of the reader. We will point out in the revision that this theorem was probably well-known to the community and that our main contributions are Theorems 3.4 and 3.5 and our resulting main result Corollary 3.6. Having said this, the proof idea for Theorem 2.8 is the following: Existence and uniqueness of the gradient flow for $\\mathcal L$ follow from the fact that $\\mathcal L$ is $C^1$ and has locally Lipschitz gradients. The latter assumption is checked in Appendix A. Now one can distinguish two cases: Either the gradient flow visits some bounded set infinitely often or it does not. In the first case, we use Kurdyka's definable Lojasiewicz inequality and follow a standard argument to prove convergence to a critical point (see Lemma B.1). In the second case, the parameters diverge by assumption. We check that the limit is a generalized critical value using the fact that $\\mathcal L$ is bounded below and therefore has square-integrable gradients along any gradient flow trajectory (see Lemma B.2). \n2. **Proof of Theorem 3.4**: The proof of Theorem 3.4 is the main part where o-minimal structures are used in a new way which cannot be found in the neural network literature. The starting point is that the sublinearity condition in our SAD activation functions also implies that the neural network responses $\\mathcal N_\\theta^\\mathcal A$ are sublinear (Lemma C.2). In particular, it cannot be equal as a function to a polynomial $f$ of degree at least $2$ (Lemma C.3). This does not explain why $\\mathcal L$ does not have any zeros yet since $\\mathcal N_\\theta^\\mathcal A$ and $f$ could be equal on the dataset or data distribution for some $\\theta$. The main bulk of the proof now exploits the remarkable rigidity properties of definable analytic functions to show that this cannot happen (Lemma C.4). We treat the one-dimensional and multi-dimensional input cases separately. In the one-dimensional case, the combination of the identity theorem for analytic functions and the uniform finiteness theorem from o-minimal geometry imply that for any $\\theta$, $\\mathcal N_\\theta^\\mathcal A$ and $f$ cannot be equal on a sufficiently large finite dataset where the required size can be made independent of $\\theta$. In the multidimensional case, the conclusion can fail for very degenerate datasets (for example if all datapoints lie on a line along which the polynomial is linear). But by combining the ideas from the one-dimensional case with dimension theory for definable sets, we show that the space of such \"bad datasets\" has strictly smaller dimension than the space of all datasets and therefore contains an open dense subset of full Lebesgue measure. \n3. **Proof of Theorem 3.5**: The proof of Theorem 3.5 starts from the special case of fitting linear polynomials in one variable with one hidden layer and then successively builds up generality (see Theorem D.4). The starting point is that if $\\psi$ is the activation function and $x_0\\in \\mathbb{R}$, then one can approximate a linear function with slope $\\psi'(x_0)$ by neural networks with one hidden neuron using the formula $x \\mapsto n \\psi(x/n +x_0)$. Similarly, one can inductively approximate the $k$-th Taylor polynomial of $\\psi$ using the formula $x\\mapsto n^k \\psi(x/n+x_0)$. By adding hidden neurons and compensating the lower order terms of the Taylor polynomial with previous approximations, one can thus approximate polynomials of degree $k$ using neural networks with one hidden layer and $k$ neurons. To upgrade this to the multivariate case, we employ a linear algebra argument (see Lemma D.1) that allows us to express arbitrary multivariate polynomials as linear combinations of polynomials of the form $p(x_1,\\dotsc,x_n)=(a_1 x_1 + \\dotsb + a_n x_n)^k$. To extend our result to multiple hidden layers, we let the other layers approximate the identity and apply the Arzela-Ascoli theorem (Lemma D.3) to achieve compatibility of the various approximations. Along the way we carefully keep track of the required number of hidden neurons. \n4. **Proof of Corollary 3.6**: Corollary 3.6 is an immediate consequence of the previous results."}]

Author rebuttal:

Meta-review: This is a mainly theoretical work, culminating in a proof of divergence for gradient flows for deep neural networks with analytic activation and polynomial target functions.  The techniques used are non-trivial, based on geometry of o-minimal structures.

Reviewers praised the quality of the writing and the generality of the approach.

On the other hand, they had concerns about distinctions from prior work, the restriction to smooth activation functions, relevance of the numerical experiments, and provision of helpful proof sketches.

The authors provided a detailed rebuttal, which was followed by some discussion with the reviewers.  Several reviewers consequently raised their ratings.

All reviewers recommend or lean towards acceptance, which I also support.  The authors should ensure that they address all the comments when revising the paper.
Decision: Accept (poster)