Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations
Authors: Alexandru Crăciun
Debarghya Ghoshdastidar
Venue: NeurIPS 2025 poster ()
Keywords: loss landscape
optimization
gradient descent
neural networks
non-singular
Primary area: deep_learning
TL;DR: We prove that the gradient descent map is non-singular for any neural network using piecewise analytic activation functions.

Abstract: The theory of training deep networks has become a central question of modern machine learning and has inspired many practical advancements. In particular, the gradient descent (GD) optimization algorithm has been extensively studied in recent years. A key assumption about GD has appeared in several recent works: the \emph{GD map is non-singular} --- it preserves sets of measure zero under preimages. Crucially, this assumption has been used to prove that GD avoids saddle points and maxima, and to establish the existence of a computable quantity that determines the convergence to global minima (both for GD and stochastic GD). However, the current literature either assumes the non-singularity of the GD map or imposes restrictive assumptions, such as Lipschitz smoothness of the loss (for example, Lipschitzness does not hold for deep ReLU networks with the cross-entropy loss) and restricts the analysis to GD with small step-sizes. In this paper, we investigate the neural network map as a function on the space of weights and biases. We also prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures (with fully connected, convolutional, or softmax attention layers) and piecewise analytic activations (which includes sigmoid, ReLU, leaky ReLU, etc.) for almost all step-sizes. Our work significantly extends the existing results on the convergence of GD and SGD by guaranteeing that they apply to practical neural network settings and has the potential to unlock further exploration of learning dynamics.

Non-Singularity of the Gradient Descent Map for Neural Networks with Piecewise Analytic Activations
Alexandru Cräciun Technical University of Munich a. craciun@tum.de
Debarghya Ghoshdastidar
Technical University of Munich
Munich Data Science Institute
Munich Center for Machine Learning ghoshdas@cit tum. de
Abstract
The theory of training deep networks has become central question of modern machine learning and has inspired many practical advancements. particular, the gradient descent (GD) optimization algorithm has been extensively studied in recent years. A key assumption about GD has appeared in several recent works: the GD map is non-singular it preserves sets of measure zero under preimages. Crucially, this assumption has been used to prove that GD avoids saddle points and maxima, and to establish the existence of computable quantity that determines the convergence to global minima (both for GD and stochastic GD). However, the current literature either assumes the non-singularity of the GD map imposes restrictive assumptions, such as Lipschitz smoothness of the loss (for example, Lipschitzness does not hold for deep ReLU networks with the cross-entropy loss) and restricts the analysis to GD with small step-sizes. this paper, we investigate the neural network map prove, for the first time, the non-singularity the gradient descent (GD) map on the loss landscape of realistic neural network architectures (with fully connected, convolutional, or softmax attention layers) and piecewise analytic activations (which includes sigmoid, ReLU, leaky ReLU, etc.) for almost all step-sizes. Our work significantly extends the existing results on the convergence of GD and SGD by guaranteeing that they apply to practical neural network settings and has the potential to unlock further exploration of learning dynamics.
function on the space of weights and biases. We also
Introduction
Training deep neural networks involves optimizing highly non-convex loss functions over high- dimensional parameter spaces, a process that poses significant theoretical and practical challenges. Among these are the risks of gradient descent (GD) converging to saddle points or poor local minima, which could hinder the performance trained models, especially when solutions associated with worst-case saddle points are significantly worse than those associated with worst-ease local minima (Dauphin et al. 2014 Pascanu al., 2014]. Recent theoretical advances have shed light on the dynamics of GD in such landscapes. One key result demonstrates that GD avoids converging to saddle points or maxima for almost all initializations, provided certain conditions are satisfied [Lee] al. 2019 This supports empirical observations that GD tends to converge to local minima rather than get trapped in poor solutions, least in the idealized setting in Lee et al. [2019]. Another line of research has explored the stability of minima, introducing Lyapunov exponent-like like quantities that
characterize whether GD converges to a given minimum from nearby initializations [Chemnitz and
Engel| 2024). These findings offer critical insights into why optimizing using GD often succeeds in practice despite the complexity and non-convex nature of the neural network loss.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Gn
Gn
77 = 0.5
17 = 0.25
Figure 1: Illustrating the difference between singular and non-singular GD maps (different step-sizes).
The loss used is [(0) 0.50 302 + 8 if 2 < 0 < 2: 0 otherwise. Left: for 1) 0.5, the GD map is singular. The red interval gets mapped to a single point after one iteration. Right: for n 0.25, the GD map is non-singular. The same interval is mapped not to a point, but to interval.
Both of the analyses rely on the fact that the GD map - defined as -,(0)=0- - where L is the loss function and 1j > 0 is the step-size is non-singular. A map G is non-singular if the preimage of any set of measure zero under G also has measure zero; maps not having this property are called singular. Non-singularity of the GD map ensures that pathological behaviors (e.g.. convergence to undesirable points) occur only on negligible sets, An example of the difference between the two concepts can be seen in Figure [ which illustrates the effect of performing one optimization step. While non-singularity is relevant to many theoretical guarantees (such as the two works mentioned above), its validity in the context of neural networks has not been put on solid footing. If the neural network uses an analytic activation function, it can be shown using only the standard tools of analysis that the gradient descent map is non-singular. However, if one employs a strictly piecewise analytic activation function like ReLU, the standard tools no longer work and novel approach is needed.
=0
In this paper, we prove that the (stochastic) GD map is non-singular for almost all step-sizes n) for neural networks using piecewise analytic activations and number of different architectures, for example using fully connected, convolutional, or softmax attention layers, thereby validating the assumptions underpinning prior optimization theories in the context of practical neural network training. Applying the standard tools no longer works since the composition of two almost everywhere analytic functions need not be almost everywhere analytic itself (cf. Remark [3). However, we can make use of the layered nature of a neural network to prove that an analogue to the chain rule holds for the neural network function (Proposition 6). This result explains why the function is well-behaved for all parameter values except for negligible set. With this insight, we extend the technique used to prove the non-singularity of the GD map in the case of neural networks with strictly analytic activations to networks using piecewise analytic activations (such as ReLU). Thus, our main result is:
Theorem (Stochastic Gradient Descent Map for Neural Networks is Non-Singular).
Consider a deep neural network that consists of fully connected, convolutional, or attention layers and let the non-linear activations the layers be piecewise analytic. Additionally, fix any data and any analytic loss function. Then, for almost all step-sizes n7, any (S)GD map Gv is non-singular.
This contribution bridges a significant gap between theoretical optimization literature and practical neural network training. By establishing the non- -innuuiuul of the GD map, we extend the applicabil- ity of results on saddle-point and maxima avoidance [Lee et al. 2016 Pannageas and Piliouras 2016 Lee et al. [209 [Chriiiito et al. [2022] Panageas et al. 2025 and on the stability of minima Wu et al. 2018 Ma and Ying 2021 Ahnn et al. (0e2m/iZ and Engel 2024) to realistic deep learning settings, offering rigorous explanation for the empirical success of GD. Of particular importance is the fact that the GD map is non-singular irrespective of the number of data points, hence theoretical results on the optimization properties of stochastic GD (SGD), such as those in Chemnitz and Engel 22024] can also be extended to realistic deep learning scenarios using our framework.
1.1 Related Works
Understanding the optimization dynamics of neural networks has been central focus of machine learning research, blending classical mathematical tools with domain -ppecffcc insights. well-known result in this area is Rademacher's 's Theorem [Rockafellar and Wets| 1998), which states that locally Lipschitz functions are differentiable almost everywhere. While this provides a general framework for studying smoothness, it does not directly address the unique structure of neural network losses, which are influenced by architectures and activation functions, nor does it provide more refined knowledge on the GD map (such as singulaug,,,,,, which is usually assumed in studying optimization.
2

---

Recent studies have focused on the piecewise nature of activation functions, such as ReLU, to better to understand how the layered structure of deep neural network tessellates the input space into affine-
linear regions as the parameters vary [Montufar et al.] 2014 Balestriero et al. 2019 Balestriero and Baraniuk] 2020]. Software to efficiently visualize these regions has also been developed [Humayun et [ 2023]. These works have shed light on the geometric properties of the network function from the input space for fixed parameters: they estimate the number of affine linear regions of the resulting function and how this changes as the weights and biases of the network change. However, from an optimization point of view, one is interested in understanding the loss landscape. That is, for fixed data, one wants to know how the piecewise nature of the activation function splits the parameter space into different regions, separated by boundaries where the loss function is not differentiable.
In optimization theory, Panageas and Piliouras (2016), Lee et al. [219 showed that first-order methods like GD almost always avoid saddle points under strict assumptions such as restricting the step-size to small values and requiring the loss function to be Lipschitz smooth. These are used to prove that the GD map non-singular which in turn is used to show that the set of initializations that converge to saddle point has measure zero (if the set of saddle points has measure zero itself). Khromov and Singh (2023) have empirically investigated how the Lipschitzness of neural networks as functions from the input space changes during training and present bounds for number of architectures and datasets. However, it is the Lipschitzness of the map from the parameter space which is needed to prove that GD avoids saddles and maxima. This condition is almost never true in neural networks (for example, it does not hold for deep ReLU networks with cross entropy as loss).
Ideas relating to heuristic notion of stability of minima for GD and SGD have been explored in Wu et al. 2018), Ma and Ying (202T), where they conducted an investigation of how the batch-size and the step-size influence stability. Wu et al. [0018] train model using GD until convergence and then switch to SGD, which, most of the time, converges to a different minimum (that generalizes better). Theoretical works such as Ahn et al. 20022 and Chemnitz and Engel [2024] start by assuming that the GD map is non-singular and use this to show that notion of ityility can be defined for minima. They conclude that stable minima attract nearby initializations, while unstable ones repel almost all starting points. The main contribution of Chemmitz and Engel (2024) is to extend the analysis of stability for GD to the stochastic setting, which is a non-trivial procedure. We mention that all the works referenced in this paragraph treat generic parametric models, not specializing to neural networks. Their assumption, that the gradient descent map is non-singular, while plausible, lacks rigorous justification in the neural network context, especially if the activation function is strictly piecewise analytic (such ReLU, GELU, leaky ReLU, etc.). We also mention that stability has been empirically observed to be related to generalization ability (Hochreiter and Schmidhuber 1997].
Closest to this paper is the work Jentzen and Riekert [2022a b 2023 While their work is more concerned with the convergence properties of gradient descent and gradient flow, they show that the loss function is continuously differentiable almost everywhere for deep neural networks using only ReLU as activation function under the assumption that the underlying function generating the data is polynomial. Our work expands their results by allowing for any piecewise analytic activation function (including, but not limited to, sigmoid or ReLUU,, a broader class of network architectures (including convolutional and networks using attention layers), well as more general learning setting (we do not impose any restrictions on the data-generating process).
Our work builds on and extends these efforts by providing formal proof that the gradient descent map is non-singular for neural networks with piecewise analytic activations, such as sigmoid, tanh, or ReLU. We show that the loss function is analytic almost everywhere and that the GD map preserves the measure zero property under preimages for almost all step-sizes. This result not only validates the assumptions in prior works but also establishes a stronger theoretical foundation for understanding optimization in deep learning, distinguishing our contribution from carlier heuristic analyses.
1.2 Organization of the Paper
In Section 2 we introduce the optimization task, different neural network architectures, and the (stochastic) gradient descent algorithm. In Section 3 we prove that the empirical loss function is almost everywhere analytic for neural networks with piecewise analytic activations. Next, in Section we establish the non-singularity of the gradient descent map, and show why some values of the step-size should be avoided. Finally, in Section 5 we discuss the implications of this work, provide further illustrations and examples, and highlight directions for future research.
3

---

2 Setting
Notation and Conventions. Throughout this paper, we will only use the Lebesgue measure. Addi- tionally, we use the abbreviation a.e. to mean almost everywhere. For a function f: IT + R we denote by Z(f) = far E (zz) 0} the set of points where f is zero and if U C R is a set, we denote by OU its topologiall boundary. For a finite set [ denotes the number of elements in I.
Supervised Learning. In the following, we consider a generic supervised learning problem. Let F: R. x R - TR be parametric model, that is, we consider the first space in the Cartesian product of the domain of F to be parameters. For each parameter 0 E Rhe we get a model that associates an input IR an output vector fo(x)) F(0,z) E RRD
and a loss function I: R" R + R, the goal is Given training data ((x;, )))) C I x "R"
to find set of parameters 0* E RT that minimizes the empirical loss
1 m 77
L(
Σ
Iyr, F(),
Neural Networks. Given a positive integer D called the depth of the neural network, a sequence N (no..... .DD) of positive integers called the widths, and sequence of piecewise analytic functions Σ = (ad : - 1 called the activation functions, we denote by HH.NN the hypothesis class of feed-forward fully connected neural networks (FF-FC) with fixed depth and widths, and with piecewise analytic activation functions. These networks are the simplest and serve as starting point for the more complicated architectures we will deal with.
We now describe how such a network works. An FF-FC network fo is given as
fo(x) (1(D) ((v) (x).
Here O = (W1, bj..... WD. bD) denotes the collection of parameters from each layer, where by e RT and W; qunx=;= are called the bias and weights for that layer, respectively; we let no Mon + ... + nD 1 DnD + mi +... + np denote the total number of parameters.
The d-th network layer f(() takes as input a vector (dd-1), the output of the previous network layer, and acts as follows:
f( (( = 0. (W + ba).
where we apply the activation function de entry-wise.
In the rest of the paper, we will also be referring to convolutional or softmax attention layers. Such layers have been developed solve specific tasks such image recognition and natural language interpretation. A convolutional layer performs a convolution on the output from the previous layer using some kernel. Attention layers use a query, key, value system in order find relevant tokens and better embeddings for sequential data. Deeper network architectures can be obtained by stacking logether different types of layers. This setting is applicable to the transformer architecture (Vaswani et al. 2017 that interleaves convolutional and attention layers, followed by a fully connected network. Please refer to Appendix A for more details on different types of architectures and specialized layers.
Bias is zero. We mention here that for the rest of this paper, we set the bias terms in each layer equal zero. This is done for clarity of exposition as dealing with the bias terms can be done in straightforward way (cf. Remark 15 in Appendix B). but only makes the exposition harder to follow.
Stochastic GD (SGD). Gradient descent (GD) is defined as the map ()) 0 r/ (+) for 7 > 0. For GD, we initialize the parameters 6g according to a probability distribution on RT and update them using the rule 0 (O,) 0, (O,;) where the loss is the one in (I. For stochastic GD (SGD), at each step we use only a sub-sample of < n < m data points to compute the empirical loss. Hence, the point is l (8,) 0; V (f:;,, where randomly choose i,,...,4m in E [m], in + i if j + is and the empirical loss computed on a mini batch is
(.( (Vi, F(O, F(O,zz,)) x4,)).
4

---

3 Analyticity of the Loss Function
We now that if the activation function of the network is analytic and the loss function is analytic as well, the empirical loss function is also analytic. This follows from the usual rules of calculus. is no longer as obvious if that is still the case when the activation function is only piecewise analytic. The goal of this section is to show that the loss function for neural network using piecewise analytic activations is itself analytic on a "big enough" set. To achieve this, start by defining precisely what we mean by piecewise analytic function and extend this definition from univariate to multivariate functions. Next, we highlight why a novel approach is needed and then show how we can leverage the layered structure of the neural network to derive analyticity guarantees. After the network function properly understood, we finish by showing that the empirical loss function is analytic on a "big enough" set as well. Recall that function f I D U + R is analytic if for any point zo in its domain, it can be expressed as a power series converging to f(x) for all in a neighborhood of zo- Definition (Piecewise and almost eveywhere analytic functions). We say a function f R - R piecewise analytic if there exists a strictly increasing sequence of real numbers {si such that I is analytic when restricted to any open interval (Ti, Ii+1). A function f IR _ R" is analytic almost everywhere if there exists an open set C IR" such that flu is analytic and the complement of U has measure zero. By SJ)) we denote the points where f is not analytic and ((f) IRm S(()).
Example 3 (Some almost everywhere analytic functions).
1. The sigmoid function f x + 1+exp(-x) is piecewise analytic with D(f) = R.
2. The ReLU function f x + max (0, 2) is also piecewise analytic. Take x; = i. We have that ((( = if < 0 and ((( = x if 2 O. Here S(f) = {0).
Remark 4 (Non-differentiabll points). In the previous definition, there may be multiple sets {Urhez on which f is analytic and whose complements have measure zero. Denoting by 2 the collection of all such sets, we see that is inductively ordered with respect to the usual inclusion of sets. An application of Zorn's lemma tells us that there exists a maximal such open set. It is not hard to check that also unique. This guarantees that the sets D()) and ((f) are well-defined.
Remark 5 (Why a.e. analytic functions are problematic). Observe that the chain rule does not hold for almost everywhere analytic functions. Whereas it is true that if ʃ is a.e. analytic and g is analytic, we have that o ʃ is a.e. analytic, it is not true that f oo g is a.e. analytic as the next jcounter-)example shows. Let : + IR be a nowhere differentiable function (e.g. the Weierstrass function) and define f: TR22 + R as f(x v) = h(x)) if y O and fx, y) 0 otherwise. Since is not analytic only on the abscissa (y = 0), is a.c. analytic. Taking R - R² to be the function g(x)) (I, 0), we see that the composition f g is equal to h, hence is not a.c. analytic.
As the next proposition shows, it turns out that we can use the structure of a neural network derive an analogue to the chain rule for the function obtained by stacking together layers in a neural network. This is important since it will be used later to show that any gradient descent map obtained from neural network loss function is non-singular.
0 be a positive number, {o, : R. + _2 12 be a collection of a.e. analytic maps and E Re a vector. Then the map defined recursively by
Proposition 6 (Analogue of chain rule for neural networks). Let D >
fD : DRR1XX X... B Xnp-1 Rap
(W,,..., WD) (Wi, WD-1)),
with f1(W)) = 01 (W1a) is analytic almost everywhere and the set (Z() has measure zero.
Proof sketch. We use induction. For the base case, one uses the fact that the map W + Wa is non-singular map (since is a submersion). The points where fi is not differentiable have to lie in m (S((S()))) hence they are of measure zero. A similar argument works for the boundary of (f1))
For the induction step, we split the domain of fD-1: B(fD-1) points; int(2 (Z(fD=1)), the "nice" zeroes; and N ((fD-1) = consisting of all "nice" non-zero points. The points B(fD-1) have measure zero, thus they can be neglected. One can then use the chain rule and properties of matrix multiplication to guarantee the analyticity of fn in a big enough region. A priori, this region might not be equal to D(fp), but is big enough to guarantee that S(fD) has measure zero, showing that fo is a.c. analytic.
the "bad" (in(
L
5

---

Adding one layer.
In red are the points where the network map is not analytic (S(f))). The complement of the green set are the points where the proof guarantees that the network map is analytic (R²2) B(Ja)).
S(Se (2)
0
)( )(
B(
We always have S(Jo) B(Jo).
50 Re)
-el.U((0,ReL(((0))
Figure 2: Illustration of the main idea in the proof of Proposition 6
Proposition 7 (Neural networks are a.c. analytic). Let fo Ro ID be an FF-FC, convolu- tional network or a network using softmax attention with piecewise analytic activation functions. Then for any input I E IR"o, the map 0 + fa( is almost everywhere analytic.
Proof sketch. The proof relies on Proposition Let fo be the network map. Then we can directly apply Proposition to show that the map 0 > fo) is a.e. analytic for any input x E R We explain in Appendix how that can be done for the convolutional and softmax attention layers. D
The previous proposition is key result, but cannot be used directly to conclude that GD map is non-sing non-singular. Theorem will be proven in the next section. We now show that the empirical loss from neural network a.e. analytic. This is important result since need it to rove Theorem We start with a lemma concerning the composition of a.e. analytic maps and analytic maps.
Lemma (Composition a.e. analytic maps). Let f R - R be an a.e. analytic function and R" + Re an analytic function. Then the composition g f R + R! is a.e. analytic.
Proof. We have that fln()) is analytic. Applying the chain rule, we have that gofl = (goo))lm analytic, hence g f is analytic a.c.
Corollary 9. Let fo R + R be any neural network with input dimension na, output dimension n.D and no parameters, using piecewise analytic activation functions. Then, given any dataset {1 y4) y: and any analytic loss function 1: IRD RnD - IR, the empirical loss given by
L(0) = Σ So(x)))
1m
almost everywhere analytic.
Proof. The maps 0 + (yi, fo();;) are a.e. analytic (since, by Proposition 7 each component is), thus we can use Lemma analytic functions is analytic, thus it follows that I. is a.e. analytic.
S to conclude that the compositions o di are a.e. analytic. The sum of a.e.
4 (Stochastic) Gradient Descent Map is Non-Singular
We now use the results of the previous section to show that any gradient descent map is non-singular. Since this true for any number of data points, we can immediately conclude that any SGD map G is also non-singular. Because of this, we will state our results for GD) but keep in mind that they also hold for SGD as well. partieular, observe that we have an explicit description for any gradient descent map for almost all points. We start with helpful lemma.
R" 5 U + 1R be smooth map. If the set of critical points of i.e. points where the Jacobi determinant dot(Df) is zero, has measure zero, then ff is non-singular.
Lemma 10 (Submersions are non-singular). Let :
Proof. Suppose that det (D((()))) # for any x U. This, together with the inverse function theorem [Lee] 2012 Theorem 4.5] tells us that ff is local diffcomorphism.

---

Let B C R" be any measure zero set. Because f is local diffeomorphism, around every point E f f- (B) there exists an open set U, x such that f restricted to it, flu,: U2 fU,), ) is a diffeomorphism. Since (B) C R" and R is second countable, we can extract a countable subcover (Uz,)i=1 of (B). Because B has measure zero, so does each intersection B, = Bn f(Uz). Since / restricted to Uz. is a diffeomorpishm, flu; (B²) has measure zero for all i. Observe that
(B) (f
A countable union of measure zero has measure zero and a subset of measure zero set has measure zero, thus the special case is proved.
Going back to the general case, suppose that = (0) (D(()) 0} has measure zero. The determinant is a continuous map and a singleton is closed, thus V is closed. Then, the restriction of f to the open set U = IR V is a local diffcomorphism.
Let B C R" be any measure zero set. Its preimage under f can be written as
FB) (B)
The first term in the union the intersection of a set with V, which has measure zero by assumption, thus it has measure zero. The second term has measure zero using the special case proved above. It follows that (B) has measure zero.
We use the previous lemma to show that the gradient descent map for analytic loss functions is non-singular. We emphasize that there unique gradient descent map if the loss is analytic.
Proposition 11 (GD map for analytic loss functions is non-singular for almost all step-sizes).
Let L : R 3 U + R. be an analytic function and assume without loss of generality that U is connected. Then, for almost all 1l > 0, the GD map Go x + x - 7/V L(x) non-singular.
Proof. If U has more than one connected component, will show that Gylu, where U2 is connected component of U, is non-singular. Since U has at most countably many connected components, it follows that G. is non-singular well.
If all the eigenvalues A...... Ang of the Hr, the Hessian of L, are constant, then for any n & (1/A1....,1/ 1/\n, we have that det(I nHL ))= = [T na, is non-zero. This means that the Jacobi determinant det DG,, det(I ,,, ) always non-zero, hence G. is non-singular.
If at least one eigenvalue is not constant, there exist two points 0,, 02 E U such that X(O)) * A(02). Take any analytic path 7: R U with ( * l2 IR,
7(4)= 01 and ((22) 0z. This can be done since we can interpolate any finite number of points using analytic functions. Since and the map H2 7tt) R..x,, is analytic as well. In
0 H (0) are both analytic, their composition, this case, the eigenvalues of Hi, y can be globally parameterized by analytic functions, i.c. there exist analytic functions X; o y: RR + IR equal to the eigenvalues of Hl (t) for all t (Katō] 1995].
For a mon-constant analytic function, the set where it is equal to a constant c E R. Z(A; 07 C R has measure zero. For any & RR U Z(A, o 7) C R. it holds that X, (f) #
In particular, for any 17 there exists 0 (() Rue such that det(/ - (0) # O. We have showed that the analytic map 0 + det(/ - n (0)) is not constant, thus its set of zeros has measure zero. We conclude that G. is invertible almost every where, i.c. the set of points where the Jacobi determinant zero has measure zero, whence we can apply the previous lemma to show that G., is non-singular. O
Corollary 12 (GD map is non-singular). Let L R - IR be an a.e. analytic function, 17 > 0, and G, 1n > nh be a function such that for all 0 E D(L) we have that ()) 0 L().. Then, for almost all n > O, any gradient descent map G, is non-singular.
Proof. By the previous proposition, Go is non-singular when restricted to D(L) for almost all values of 17. If B is any measure zero set in _ we have that G7 = Gy (B) ( (B). The first set in the union has measure zero since G. is non-singular on D(L) and the second one as well since S(L) has measure zero
7

---

5 Discussion
In this paper, we have demonstrated that for neural networks with piecewise analytic activation functions, the GD map (,) 0 - L)) and the SGD maps G7,1 are non-singular for almost all step-sizes 17. This result has profound implications for the theoretical understanding of neural network optimization. By proving that the loss function is analytic almost everywhere and that the (S)GD map preserves sets of measure zero under preimages, we confirm a key assumption in prior works, such as those by Lee et al. [019] and Chemnitz and Engel [2024). This bridges critical gap between theoretical guarantees and the practical success of GD in training deep networks.
Our findings imply that, for almost all initializations and step-sizes, the optimization trajectory avoids pathological behaviors - such as convergence to saddle points or unstable minima - aligning with empirical observations of (S)GD's 's effectiveness. The proof relies on two key insights: first, the piecewise analyticity of the activation functions ensures the loss is analytic outside set of measure zero; second, the analytic properties of the loss in the "nice" regions ensure the GI) map is invertible almost everywhere, except for a negligible number of values for the step-size. While our result is robust across almost all step-sizes, we note that non-singularity may fail at specific values for the step-size (where 1j = 1/A; for an eigenvalue A of the Hessian of L), which are negligible in practice.
To illustrate the connection between our result and the stability of minima, we first recall key result from Chemnitz and Engel [2024.. They show that for generic parametric model, if the GD map is non-singular and the set of global minima M forms manifold, then the stability of a minimum 0 e .M is completely determined by a computable quantity. Our Theorem I provides the missing piece to rigorously apply this framework to neural networks. By establishing the non-singularity of the GS) map for networks with piecewise analytic activations, we can now state the following:
Corollary 13. Let the assumptions be as in Theorem / Additionally, suppose the data is generic and let M be the set of global minima of the loss function L. Then, for almost all step-sizes, there exist functions u(0) and X(0) such that a global minimum 0 E M is stable under GD or SGD only if ((0) < or x(0) < 0, respectively. Conversely, if u(0) > 0 or X(0) > 0, then is unstable.
The definitions of μu and A are the same as those in Chemnitz and Engel [2024). For a discussion of "generic data" see Cooper [02T]; note that it reasonable to assume that noisy data is generic.
We now use this corollary to analyze the training dynamics in a few examples. We first compute the functions/ μl and À for the simple setting with loss function (9,.0) + 3.51 ReLU((,, (eLU )) (corresponding to two-layer ReLU neural network as described in Appendix Cj. We then use these functions to: 1) determine the stability of periodic trajectories under GD as the step-size 1) changes (Figure 3. and 2) show that for a fixed step-size, a global minimum can be stable under GD but unstable under SGD, and vice-versa (Figure 4.. The next two paragraphs elaborate on this.
Application:
periodic trajectories. The composition of non-singular maps is non-singular as well, hence the stability analysis of minima can be extended periodic trajectories. The existence of such trajectories can be motivated twofold: (i) the use of large step-sizes (as incentivized by results from Li et al. [2019], Mohtashami et al. 2023] leads minima being stable only if they flat (cf. Chemnitz and Engel| ): (ii) it has been empirically observed that the eigenvalues of the Hessian matrix of the loss oscillate even after numerous training epochs [Cohen al. 202 2023)
which means that the trajectory is not converging. Our results provide a comprehensive framework for dealing with such cases. We illustrate this through experiment described in Appendix C We assume 2-layer ReLU network fe(z)= ReLU ReLU((, (eReLL((
a quadratic loss, and data sampled from a linear function. For k > we are interested in k-periodic points, that points 0* e 122 such that (0)) = 0* The sequence of iterates 0-, (0") Ch (00)) is then a periodic trajectory. A periodic trajectory is stable if there is a region U around 0² such that initializing in U leads to trajectories that stay close to the periodic one. In Figure on the right we plot trajectories with the same initialisation, converging to different periodic trajectories for different step-sizes, In Figure on the left we plot the bifurcation diagram for periodic trajectories lying on the diagonal (#, = 02). As the step-size increases, new periodic trajectories are created and the old ones become unstable.
-
1 'One also has to show that the saddle points and the unstable minima lie in a measure zero set. When ne S m this is true since such points are isolated. In the overparameterized case, when ne > m, proving this requires more care. (ooopr 2020 202l have proven this for neural networks with smooth activations.
8

---

Bifurcation diagram for trajectories on the diagonal (0) 02)
Converging and periodic orbits.
0.9
0.7
Egeeiaiic unaalee
global minima
initialisazion
Aperidi monile
3.8
1-periodic trijectory ( 0.25)
2periodic unjectory (q 0.325)
Figure 3: Left: periodic trajectories for GD on [(0,, 02) 3.53(1 Rel.U(@eReLLU(())) ()) As the siep-size increases, higher order orbits appear and the lower order ones become unstable. Right: for the same initialisation, but different 7), two trajectories, one that converges and one that oscillates.
Stochastic and adaptive variants of GD. Our theoretical framework robustly covers SGD and settings where a learning rate schedule is used. Theorem establishes the non-singularity of the GD map G for any fixed data and any batch size. An SGD step mathematically equivalent GD step performed on a loss function computed over a mini-batch of data. Since our result holds irrespective of the number of data points used compute the loss, it directly applies to each step of an SGD trajectory, where the mini-batch changes at each iteration. The map for each step is therefore non-singular (for almost all 17), and the composition of these maps, which describes the full SGD trajectory, is also non-singular. An optimisation process using GD over I steps with learning rate schedule (7 ................................ (r) can be described by the composite map composition of non-singular maps, hence their composition is also guaranteed to be non-singular. Therefore, our analysis provides rigorous foundation for understanding optimization dynamics not just for a fixed step-size, but for any practical scheme that relies on composing gradient descent steps.
full
G,r
G This is
Application: stable minima of GD SGD. Results from Wu al. (2018) have shown that GD and SGD can have different stable global minima. For the setting described above, we can analytically compute the quantities introduced in Chemnitz and Engel [2024) which determine stability (see Appendix C). The relation between the minima of GD and SGD can be quite complex, so we illustrate two interesting cases. In Figure on the left, we see that there are fewer stable global minima for SGD than for GD. In Figure 4 on the right, we see a setting where the stable minima for GD and SGD do not overlap at all. remains an open problem to determine the stable minima for GD and SGD can be related in a general setting, or if such relation has to be determined on a case-by-cee by-ce basis.
Different architectures. While our results already hold for diverse number of layers (fully connected, convolutional or using softmax attention) this does not cover all cases used in practical settings. An in depth investigation of other architectures may reveal additional structural properties that make it possible to prove that the neural network map is analytic a.e. for those architectures as well. We think that an extension graph and residual neural networks might consist only of routine applications of the ideas already present in our proof'. However, for recurrent neural networks, we see that our techniques are not enough and another approach is needed. One might approach this setting by viewing the recurrent network f : R - RRD as larger fully connected network F N Ra (N > nhp) for which some weights are forced be equal (cf. Appendix A). Proposition 6 guaranttees that S(F), the singularities of F have measure zero in R We have that S()) C nh (FF) (where Th viewed as subset of TR-V). Hence, if R n S(F) has measure zero, so must S(.) too. However, it is not clear that this intersection is negligible and more in depth investigation of S(F) is needed to sure. Even with this current limitation of our proof techniques, conjecture that the GD map is non-singular for the loss landscape of a recurrent neural network.

---

global minima
GD stable minima
SGD stable minima
Stable minima for GD and SGD overlap.
Stable minima do not overlap.
Figure 4: Illustrating the different stable minima for GD and SGD. In blue are the global minima for in green the minima stable for GD; in red those stable for SGD. Left: the stable minima for SGD are a proper subset of the stable minima for GD. Right: the stable minima for SGD and GD do not intersect. This change is due different step-sizes (Tet = 0.15 vs Thight = 0.3) and different probabilities for generating the data (cf. Appendix Cp.
L(0,, 02) 3.53 (eLU((
Other optimization algorithms. The techniques developed this paper could be applied to other optimization methods such as mirror descent proximal point algorithms, potentially broadening the scope of non-singularity guarantees. These alternative optimization methods share with GD the core principle of iteratively updating solution to minimize loss function (commonly using only first-order information like the gradient). [ee et al. 2019) have analysed the convergence properties of such algorithms, albeit in the same restricted setting they have for GD (i.e. requiring the loss function to be Lipschitz smooth). To extend our results to different optimization algorithms, one has modify the proof' in Section of our work. In particular, it suffices investigate the case when the empirical loss function L is analytic, since once it is proven that the optimizer map is non-singular for such losses, the same proof that for Corollary 12 can be used extend a.e. analytic losses.
Acknowledgements
This work has been partially supported by the German Research Foundation (DFG) through the Priority Program SPP 2298 (project GH 257/2-2), We also thank the reviewers for their feedback.
References
Kwangjun Ahn, Jingzhao Zhang, and Suvrit Sra. Understanding the unstable convergence of gradient descent. In International Conference Machine Learning, pages 247-257. PMLR, 2022.
Randall Balestriero and Richard G Baraniuk. Mad max: Affine spline insights into deep learning. Proceedings of the IEEE, 10945):704-777, 2020.
Randall Balestriero, Romain Cosentino, Behnaam Aazhang, and Richard Baraniuk. The geometry of deep networks: Power diagram subdivision. Advances in Neural Information Processing Systems, 32, 2019.
Dennis Chemnitz and Maximilian Engel. Characterizing dynamical stability of stochastic gradient descent in overparameterized learning, 2024. URL wwR.attp:://aaxtv.orrrabbb//// 2000bb 20209
Patrick Cheridito, Arnulf Jentzen, and Florian Rossmannek. Gradient descent provably escapes saddle points the training shallow relu networks. Journal of Optimization Theory and Applications, 203:2617 2648, 2022. og
swmanticschooar....mmmo www.oo/////oo... Drgg/oorpusID:2511200
10

---

Jeremy Cohen, Simran Kaur, Yuanzhi Li, J Zico Kolter, and Ameet Talwalkar. Gradient descent on neural networks typically occurs at the edge of stability, International Conference on Learning Representations, 2021. URL httpttt:///opeeeeeieeee
Jeremy Cohen, Behrooz Ghorbani, Shankar Krishnan, Naman Agarwal, Sourabh Medapati, Michal Badura, Daniel Suo, Zachary Nado, George E. Dahl, and Justin Gilmer. Adaptive gradient methods at the edge of stability. In NewrlPS 2023 Workshop Heavy Tails in Machine Learning, 2023, URL https://openrevve/
Y Cooper. The critical locus of overparameterized neural networks. arXiv preprint arXiv:2005.04210. 2020.
Yaim Cooper. Global minima overparameterized neural networks. SIAM Journal on Mathematics of Data Science, 3(2):676-691, 2021.
Yann N Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. Identifying and attacking the saddle point problem in high- nigh-dimensional non-convex optimization. Advances in neural information processing systems, 27, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning. MIT Press, 2016. http:
//www.deepleec. wwwpplerriingbokk..... org
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computatioon, 9(1):1-42, 01 1997. ISSN 0899-7667. doi: 10. 11.16//ncoo.999....... Uttpttss:://oi org/ 10. 11w..1666/ncco.oo 1997 9.1.1.
Ahmed Imtiaz Humayun, Randall Balestriero, Guha Balakrishnan, and Richard G Baraniuk. Splinecam: Exact visualization and characterization of deep network geometry and decision boundaries. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recog- nition, pages 7799,,798, 2023.
Arnulf Jentzen and Adrian Riekert. On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks. Journal of Machine Learning, 1 (2):141-246, 2022a. ISSN 2790-2048. doi: https://doi.org// 1rtp///oooor/11...0 wtt..///////...//mmmm..mm 4208/jm1.2201 14a.
Arnulf Jentzen and Adrian Rickert. A proof of convergence for the gradient descent optimization method with random initializations in the training of neural networks with relu activation for piecewise linear target functions. Journal of Machine Learning Research, 23(260):1-50 2022b. html
URL httphtttt//mlrrr org wwtg:/apers//2/22100555
Aruuff Jentzen and Adrian Riekert. Convergence analysis for gradient flows in the training of artificial neural networks with relu activation. Journal of Matherattical Analysis and Applications, 517 (2):126601, 2023. ISSN 0022-247X.
doi: https://aai..ee//00000////jaaaa
126601. URL
http:://wwww..iinceediiieee wttp://iteeettcccciieeeeeeeeeeee http:///i..eeeeeraaaaaaa////////
Tosio Katö. Perturbation theory for linear operators. Classics in mathematics. Springer, Berlin, 1995.
ISBN 978-3-540-58661-6
Grigory Khromov and Sidak Pal Singh. Some fundamental aspects about lipschitz continuity of neural networks. arXiv preprint arXiv:2302. 10886, 2023.
Steven G. Krantz and Harold R. Parks, A Primer of Real Analytic Functions. Birkhäuser Boston, Boston, MA. 2002. ISBN 978- 1-6---1--2-66 doi: 10.1007/978-0-8116-881444 URL http
com/ 10. 10w0000000000- 01-7777----------------------0-- 8134-0
Y. Le Cun, L.D. Jackel, B. Boser, J.S. Denker, H.P. Graf. I. Guyon, D. Henderson, R.E. Howard, and W. Hubbard. Handwritten digit recognition: applications of neural network chips and automatic learning. IEEE Communications Magazine, 27(1 1):41-46, 1989. doi: 10. 10.1109/35.41400
Jason D Lee, Max Simchowitz, Michael Jordan, and Benjamin Recht. Gradient descent only converges minimizers. In Conference learning theory, pages 1246-1257. PMLR, 2016.
Jason D Lee, Joannis Panageas, Georgios Piliouras, Max Simchowitz, Michael I Jordan, and Benjamin Recht. First-order methods almost always avoid strict saddle points. Mathematical programming. 176:311-337, 2019.
11

---

John M. Lee. Introduction to Smooth Manifolds, volume 218 of Graduate Texts in Mathemat- ics. Springer New York. New York, NY, 2012. ISBN 978- 1-4419-9981-8. doi: 10.1007/ 978-1-4419-9982-5. htt.ttts:://iik spr inger com/com////ooooo 10. 107711077--14411-99922
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in neural information processing systems, 32, 2019.
Chao Ma and Lexing Ying. On linear stability of sgd and input-smoothness of neural networks. Advances in Neural Information Processing Systems, 34: 24:16805-1681, 2021.
Amirkeivan Mohtashami, Martin Jaggi, and Sebastian U Stich. Special properties of gradient descent with large learning rates. In International Conference on Machine Learning, pages 25082-25104.
PMLR, 2023.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. Advances in neural information processing systems, 27, 2014.
Ioannis Panageas and Georgios Piliouras. Gradient descent only converges to minimizers: Non- isolated critical points and invariant regions. arXiv preprint
2016.
Ioannis Panageas, Georgios Piliouras, and Xiao Wang. First-orde methods almost always avoid saddle points: the case of vanishing step-sizes, 2025. URLhttts:://axxi http:////wwivvoo//./////// http/ab//100 1
Razvan Pascanu, Yann N Dauphin, Surya Ganguli, and Yoshua Bengio. On the saddle point problem for non-convex optimization. arXiv preprint 1 460444 2014.
R. Tyrrell Rockafellar and Roger J. B. Wets. Variational Analysis, volume 317 of Grundlehren der mathematischen Wissenschaften. Springer Berlin Heidelberg, Berlin, Heidelberg, 1998. ISBN 978-3-540-62772-2. doi: 10. 100000997-----6-22233-14 http:////l http:://iiinks..nngee..... springer.comm10 com/10
David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back-propagating errors. Nature, 323(6088):533-558 October 1986. ISSN 1476-4687. doi: 10.1038/323533a0.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 6000-6010, Red Hook, NY. USA, 2017. Curran Associates Inc. ISBN 1781510869944.
Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. C.CaesaBiin Bianchi, and R. Garnett, editors, Advances in Neural Information P Processing Systems. volume 31, Curran Associates, Inc 2018, URL /ttp htwpooceeeiinnnnnnnniiiee..eeeee cc/paper Eftee//arrrr02ff111e100222222220
pdf
12

---

A Different Architectures
In this short appendix, we expand on the discussion from recetionn the different architectures besides the ones made only of fully connected layers.
Convolutional Networks. There exist also other kinds of neural network architectures, more special- ized to certain tasks. The first which we introduce are convolutional neural networks Le Cun et al. [1989]. This kind of networks
different from FF-FC networks because they have convolutional layers. A detailed description of such networks can be found in Goodfellow et al. [2016]. but for our purposes it suffices to know that convolutional layer is such that some entries in the weight matrix are equal to each other. For example, for a univariate convolution, each row of the weight matrix is constrained be equal the row above shifted by one element Goodfellow al. (016))
Recurrent Networks. To better learn time-series, recurrent neural networks Rumelhart et al. 1986 have been introduced. The idea behind them is that if the input to the network is series F.....,,rr.. the network might benefit by using the intermediate activations has com- puted at previous steps when computing the current activation. More explicitly, if denote by f( (z1....,a0) Zz) the output of the d-th layer of the network at time t, then the output of the (d+ 1)-th layer of the network at time + 1 is defined recursively by the following relation
((4) (,,...,,, + Ue+1Se (x.....,, .4. + bat1),
S111 (I,,...., I1-1) (Wa+1
where is applied entry-wise and the base case is p()) (m1) (Wi U1 b1), with fo constant. The new matrices we introduced U/ E R x R represent the recurrent connections.
In this case, the outputs of the network at any time-step can be used when computing the empirical loss function. That is, for every input æ; (we index different inputs with the subscript i), we have a set I [T] of time instances, and we can write the empirical loss
m
((0)
Σ IEL
Myye St 1()) (s..)..... (.,)))
m
T.
Networks using Attention. As alternative recurrent networks, attention layers have been developed Vaswani et al., 2017]. Such a layer has three learnable matrices Wg, Wk, and W., and works as follows: given a sequence I II,.,,,TT Denoting by X the input matrix, i.e. the matrix having the x; 's as columns, we obtain three matrices: Q = W W. WxX, V W,X called the queries, keys, and values, respectively. The output is:
IB. it outputs sequence yi..... Un E 17.
QKT Vn
Attention KQ V) = softmar(
V
where the sof softmmmm t'maa function is independently applied to every row of its argument. The columns of this matrix are the output sequence .1,..., Yn- After applying a number of attention layers, we may use an FF-FC network the end. This is usually done by concatenating the output sequence into vector y = y1 * #... * Yn and using this as the input to an FF-FC network. The important result for our analysis is to observe that for any input sequence Ii,... ... En. the map determined by attention layer on the parameters (Wg. WA, W.) + Attention (Q, K, V) is analytic.
B Proofs of Main Results
In this appendix provide the full proofs of Propositions and There is one difference from the result stated in the main body of the text. Instead of proving the result for a.e. analyticity, we prove a more general result, showing that the smoothness properties of the activation function carry over to the neural network map almost everywhere. What we mean by this is that if the activation function is piecewise C* where k be positive integer (denoting k-times differentiable functions), do (denoting smooth functions), or w (denoting analytic functions), then the neural network function is also almost everywhere C*. The definition of an almost everywhere Ck function is analogous to _
Definition We begin by restating Proposition 6
Proposition (Analogue of chain rule for fully connected layers). Let >> be a positive number and (o, R: - R. J. be a collection of a.e. C* maps with the property that @Z( has
13

---

measure zero and & E R a vector. Then the map fo defined recursively by
La RR Xno
TR xna-1 -> 1R
(W1,..., Wa) + o oa(Wafa-
Wa-1)),
with f((W)) = 01 (WW)) is C* almost everywhere and the set (Z(f) has Lebesgue measure zero.
Proof. We will prove this proposition by induction on D. Df 1, we have that (1(W)) (W))).
If 0, then (Wi) σn (0) is constant function, hence it is CK. Also, we have that ((1i) is either the whole domain of fi or the empty set, hence (Z( has measure zero.
In the case * O, the set of points S(fi) = {Wia E S(o1)} (a priori this set need not be equal to the set of points where f1 is nor Ch; we show in this paragraph that they do coincide) has measure zero and is also closed. This follows since the multiplication map M, : W, + Wia a is non- singula... With this definition, we have that fi is the composition fi = o M1. Taking any W1 & S(fi), we have that M1 is C² in neighborhood of W and that σ is C* in neighborhood of Wja = Mi(W).. hence we can apply the chain rule to conclude that fi is C in neighborhood of W1. We see that fi is Ch in the complement of S(fi). But (ff) is closed set and has measure zero, hence fi is C* almost everywhere.
We now show that (Ti ) has measure zero. Looking at the set 2(í)) = (Wa E ((1)), we see that ((1) MT (Z(r)). We also have that W E aZ(f) implies that M1 (W) E @Z(m). Hence, we see that (2( (0Z(a)), Since Mi is a non-singular map (it is submersion), follows that (Si) has measure zero.
Now proceed to the induction step. Assume that fD-1 is C* a.c. and that 0Z has measure zero. Define the following sets:
1. The set of "bad" points of fD-1:
B(fn-1) OZ((r=I)US((p--))
By assumption, we have that B-) is measure zero set.
2. The set of "nice" points of fD-1 which are not roots:
N(fD-1) = dom(fo-1) (B(fo-) where dom(Jn-1) is the domain of fP-1-
We also the following notation
TD (W....., WD) and fD-1 = WDfD-I WD-))
We now consider two cases:
1. If In-- E N(Jn-i), then take any Wn such that Wpfn-i & ((a Such Wp exists since (D-1(TD-1)) is not zero. fact, the complement of the of all such points has measure zero since it is the preimage of the set S(ap) under the multiplication map MD WD + ( which a non-singular map.
We will write fo as a composition of maps and then show that we can use the chain rule to conclude that fo is Ch in neighborhood of xp. Thus, consider the following maps:
fD-1 x id 1
Xno X ... x R uD xnn- - nn-- R X
(W1.- WD) + (fD-I WD-1).
MD DnD
R RnD
(HD-1, Wp) + WDED-1.
We see that fo can be written as the composition op MD (fD-1 id). By assumption, we have that fD-1 x id is C+ in neighborhood of æp; MD is Ch in its whole domain, in
14

---

particular in an open neighborhood of (fD-1(RD-1), WD); and also op is C on an open neighborhood of yn by assumption. Hence, we can apply the chain rule to conclude that fn is C* in an open neighborhood of dD
2. If æD-1 E in((Z(/D-i)) then we can use a different approach. Since it is an interior point, there exists an open neighborhood U e æp-1 such that U C in((Z(fo=1)). We immediately see that on this neighborhood fn constant, hence it is Ch.
We can now describe the set of points where the previous two cases guarantee that fo differentiable. Start by defining A to be the set
A = ( WD) E ((fD-1) x - (DDD-
It is closed since S(ap) is closed and the map MD : (xD-1, WD) + WDfD-I(ID-L) is continuous. Since this map measurable and is S(rD), it follows that A is also measurable. In the first case above, we proved that for any ID-1 E (ffD-1) the intersection A n {æp-1} x R has measure zero. Use can use a special case of the Fubini's theorem to conclude that A has measure zero as well.
In case I we prove that fo is C* on (N(fD-1) R 1) \ A. In case we showed that fo is C* on int int(Z((n-1)) x R Putting everything together, we see that F is C* on an open set whose complement has measure zero, namely Á U B(fD-1) x R= This shows that I'D is C almost everywhere.
We now prove the second part, namely that @Z(fb) has measure zero We have that Z(fD) = p WD)D (oD). This follows since MD is continuous map. This means that @Z(fD) MD (OZ(a))) We consider three eases:
[( Start by observing that if xn E @Z(fn), then yn
1. If zp-1 E N(fD-1), we can look at the derivative of MD at a point (zp-1, WD) e N(fD-1) R"DXaD 1, For any (#D-1, WD) in the tangent space zp. we have:
M(DD,,, = WDfp-I(aD-)) + - It is surjective since fn- (ap-1) * 0. Hence, M is non-singular on N(fn-i) x Rnxxn 1, Since (Z())) has measure zero, we have that N(F) x RuDxne 1 (002(nn) also has measure zero
2. If T int(Z(fD-i)) then WDfD-1(ZD-1) is constant for any WD. Thus, either all points yD-1 are roots of aD or none. In both cases the set (ff-1)) x R xnox n @Z(fD) has measure zero.
3. If æn-1 E B(fD-1), then the set B(fD-1) RODXAD-1 has measure zero, hence its intersection with @Z(fD) also has measure zero.
Since the domain of fn is a disjoint union of the three sets we have considered in the previous cases and since the intersection of @Z(sp) with each one of them has measure zero, it follows that @Z()p) has measure zero itself. This concludes the proof.
Remark 15 (Why the bias can be dropped). Including bias terms changes the linear transformation in each layer to an affine one: from Wa to Wa + b. The proof of non-singularity relies on this map being a submersion (i.c., its derivative being surjective). The derivative of the affine map is surjective, just as it is for the map without bias. In fact, the presence of the additive bias term makes the surjectivity argument even more direct. The rest of the inductive proof in Proposition 14 proceeds in exactly the same manner.
Proposition 16 (Analogue of chain rule for convolutional layers). Let all the assumptions and definitions in Proposition hold true. However, suppose that some entries in the matrix W are forced to be equal to each other. Then, for generic data, the map FD is a.e. Ck and the set (Z)) has measure zero.
Proof. Let CD be the space spanned by the weight matrices WD. We must consider now the multiplication map as map
Me WD WDa, with domain Cp.
15

---

We now show that Mc is non non-singular. Observe that for any measure zero set B in the codomain, the preimage is M (B) = . -M nB) where M is the multiplication map on the full ambient space R Since M is a submersion (this was shown in the proof of Proposition 14). M- has measure zero. The intersection of measure zero set with a lower-dimensional subspace CD will also have measure zero within CD for generic choice of data. Such choice is justified if the data is assumed to be noisy. While a pathological choice of data could align CD with M MB), such data points form a measure zero set themselves.
From here on, the proof of Proposition 14 applies verbatim.
Proposition 17 (Analogue of chain rule for softmax attention). Let {ff R + ReD 1 be collection of a.e. Ch maps with (f() having measure zero for every i. Let (0) be the matrix with fr(0)'s 's as columns. Then, the map
F: (0, Wk, W, W.) ~ (.
a.e. C³ and the set (F) has measure zero.
In the proposition above, one should think of the Jt's as representing the output of fixed neural network as a function of the networks' parameters for different input values zy in a sequence to be fed into the attention layer. Usually, there is no pre-proce prepprccesiing of the data before it goes into the attention layer, however stating the proposition this way will allow us more flexibility, for example, we can apply it for two attention layers one after another.
If there is are no ft's to be applied to the data before entering the attention layer (as usually the case), the argument becomes simpler, and we also include it as a simplified version of the general proof.
Proof of Proposition 17
In this simplified case, we only have to show that the attention layer A (W&, Wg. W.) +
Attention (W,X, X, W,X) for a fixed input X has the desired properties. Since matrix multipli- cation, the softmax function, and concatenation are analytie, then their composition is definitely a.c. Ck
To show that (Z(F) has measure zero, we need the following lemma.
Lemma 18 (The zero set of analytic functions has measure zero). Let f:: R -> IRm be a non-zero real analytic function. Then the zero set of f. Z()), has measure zero.
Proof. The proof proceeds by induction on n. Let f: IR + 1T be a real analytic function. Suppose that Zf)) is not set having measure zero. Since it is not a null-set, it is an uncountable set, thus it has an accumulation point. From the identity theorem [Krantz and Parks 2002 Corollary 1.2.7] we
get that f = O, contradicting
assumption that ff is non-zero.
Suppose the lemma established for real analytic functions defined on R=-1, Assume that Z(J) is not a null-set. By Fubini's 's Theorem there is set F C IR not having measure zero such that for all F the set ZJ)) n ([x) x R²-1)) does not have measure zero. By the induction hypothesis we conclude that f= 0 on each of those hyperplanes. Since F is uncountable, it has an accumulation point a E R. Thus, there is a sequence of distinct points ag + a such that f = 0 on the hyperplanes with first coordinate in {ai, 02 aa,, ) (a).
Now take any line of the form R x {x²) x .... x {2,), where x; E R are fixed numbers. Since f is real analytic on this line and f = 0 on a set with an accumulation point on that line, we have f = 0 on that line. This was for true any line, hence 0 on R..
With this lemma, we have that Z(F) has measure zero, since attention is analytic and non-constant, thus (F) c (F) also has measure zero.
For the general case, where we apply the fi's to the input of the attention layer, we proceed as follows.
Let S(f) Um S(f.). We have that for any 0 Re and (We, W W.), the function F is Ck since Y is Ch in neighborhood of 0. Hence, we have that S(F) S(f) x Roxao x Roxao x pexeD-1, that is, F is c* on a dense open subset of its domain.
16

---

We write F as the following composition of functions
(0, WR. W W.) & (Y),), WR. Wg. W.) M (WAY, WWW.WWWY,WWW... A F(O, WR. Wa W.).
Using the Lemma 18 we see that ((A) has measure zero, where is the map (K, V) Attention K, Q, V Since M, the matrix multiplication, is submersion, have that M M- (A)) also has measure zero. In fact, since the preimages are hyperplanes in the matrix space, we have that M (Z(-( n (0, R 1, Wg. We) has measure zero and similarly for Wg and W.. This means that
C (I') will also have measure zero.
Remark 19 (Any order for the layers). In Propositions 16 and 17 we have proven the assumptions needed for the induction step in the proof of Proposition 14 This means that our results, namely the almost everywhere k-times differentiability of the network function hold for any combination of convolutional, attention, or fully connected layers.
C Experimental Details
The experiment described in Section 5 is as follows. We have two data points: (0.9.0.9) and (2.5, 2.5) which determine linear function. The model we use the following: fo(x) = ReLU((,ReLU((122) With the quadratic loss, we get the following empirical loss function
02) = 3.53(1 ReLUU((
Then, the global minima for this loss are the points { 02) 02 1 and 0,, 02 > 0j, i.e. the first quadrant part of the hyperbola in the plane. We denote by p the probability of selecting the point (0.9.0.. 0.9) as a mini batch for SGD; by r we denote the step-size.
Chemnitz and Engel 2024 introduce two quamtities, μ and À which determine if global minimum 8" is stable for GD and SGD, respectively. More precisely, we have that 6* is stable for GD u(0*) < 0 and that it is stable for SGD if x(0*) < 0. For the setting we have described above, the analytic form of μ and À is as follows
(()) log( [1 mp0.99 + (1 - p)2.5 65)1),
(0) = p log ,/099² (0 + 03)1 + (1 - p) log( (1 (0)
Figure 4 was obtained by setting the 77 0.15 and p 0.5 for the figure the left and r7 0.3 and p 0.58 for the figure on the right.
and use numerical root finding algorithm find all the real roots. We determine if periodie point period gh, is stable by numerically computing the eigenvalues of the lincarization of the k-times iterated GD map at gk, ((*). If both eigenvalues have absolute value smaller than 1, we know (Chemnitz and Engel [024] that the period is stable; otherwise it is unstable.
To obtain the left part of Figure 3 we look at the polynomial p* (x) = G (z) (z)
(k)
The right part of Figure 3 plots two trajectories of GD initialised at (1.48, 1/1.48 + 0.1) with step- sizes 0.25 for the trajectory converging to minimum (violet) and 17 0.325 for the trajectory converging to periodic orbit of period (brown).
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We took significant effort formulate our claims reflecting our contribution and scope as clear as possible in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss limitation of our approach in Sections and as well as in Section Assumptions are clearly stated in Sections and
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification: We always provide the full set of assumptions and complete proofs for all theoretical results. Some proofs are provided in the main part of the paper, while the remaining proofs are given in the appendices and referenced accordingly in the main paper.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We detail all information needed to reproduce our main experimental results in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: All the details needed to reproduce our experiment (including data) are given in Appendix [
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttpcc/pcccccccccccooooooop httpa//gguiieee/deSuubbbiiiiiiii wwwninsssonnppo swiniiossoopyccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail all relevant details (splits, hyperparameters, type of optimizer, etc.) understand the results in Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: For the illustrative experiment we present, all quantities have been computed analytically, meaning the results we have obtained are exact.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
The experiments is illustrative and small scale, with all plots generated on standard desktop PC less than minutes.
Justification:
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We read the Code of Ethics and our research conforms with in every respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We not aware of any societal impacts our work might have.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We are not releasing any data or models, thus there is no risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not use existing assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We not releasing any assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We neither performed crowdsourcing experiments nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We did not perform a study with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: No LLMs were used in the making of this work.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttpps////euupp..... htt.///rips.cccooofeerrreses//// ccc/Confereencee/2225/LLL for what should or should be described.
24


Reviews:
[{"review_id": "7nlE7eHGZd", "reviewer": "Reviewer_NUng", "summary": "Review: Non-Singularity of the Gradient Descent Map for\nNeural Networks with Piecewise Analytic Activations\n\n1) importance of problem: The paper addresses a fundamental, open theoretical problem—the non-singularity of the gradient descent map—which underpins many theoretical results in optimization, convergence, and stability analyses.\t\n\n2) novelty of methods/approach.The approach was a clear one using proof techniques that offer clarity. ( e.g rather than submresion the paper could have used Morse theory or Sard’s theorem could also have been applied to achieve similar results. Sard’s theorem, in particular, provides another way to argue measure-zero singularities by showing critical values form a measure-zero set). I am not sure about the novelty of the methods, as these are still standard techniques. \n\n3) significance of results: The assumption of non-singularity of gradient descent is a common one, so this proof is validating. However, many works have proven related results, and it was unlikely for the assumption to have been a false one. \n\n4) clarity of presentation: The paper was cleary leaving obscuring technical details to the appendix \n\n5) technical correctness: It’s as technically sound as I can tell given the amount of time I’ve had to read it", "questions": "see above.", "limitations": "see above.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe primary objective of this paper is to generalize existing theoretical results to neural networks using popular activation functions such as ReLU, sigmoid, and leaky ReLU. The authors clearly show  that the gradient descent (GD) map is not singular for almost all step sizes, which extends the range of existing theoretical frameworks. This generalization is backed up by:\n\n1. Analyticity Results: The authors show that neural networks with piecewise analytic activations  are analytic almost everywhere (a.e.). The method  uses structural induction based on the layered structure of neural networks.\n\n2. Non-Singularity Analysis: The authors use submersion theory and measure theory to rigorously show that almost all step-sizes are non-singular, using Kato's theorem to do so.\n\n3. Versatility: The results apply to a wide range of network architectures, such as fully connected, convolutional, and attention layers.\n\n\nCritique \n\n1. Bias Simplification:  The authors set biases to zero as a simplifuing assumption. They claim simplification is easy to generalize, but they don't give a convinceing reason for that claim. This omission could make their findings less useful for a wide range of situations, since biases are used in all practical neural networks. More should be said about this claim in the body or the full proof should just be given in the appendix. \n\n2. Attention Layers: The paper's discussion of attention mechanisms (Proposition 15) is short compared to how well it explains fully connected and convolutional layers. Because attention layers use complicated nonlinear operations, a more detailed and thorough explanation of why non-singularity should apply to attention mechanisms would be good. \n\n3.  Practical Significance of \"Almost Everywhere\" Conditions: The use of measure-zero singular sets, while mathematically sound, lacks a detailed discussion regarding its practical implications. It's not clear if these theoretically unimportant singularities could have a big effect on  optimization problems, especially when there are a lot of parameters. For example what about non analytic points on the boundary of analytic regions? \n\n5. Singular Step-Size Considerations: The authors only briefly talk about singular step-sizes (where η = 1/λᵢ). A more in-depth look at how these unique step sizes might affect adaptive step-size methods that are often used in neural network training would make the results more useful in real life.", "was_revised": true, "final_justification": "The paper addresses a fundamental, open theoretical problem—the non-singularity of the gradient descent map—which underpins many theoretical results in optimization, convergence, and stability analyses.\t Accept the paper.", "rebuttal": "We are grateful for the reviewer's positive assessment, and for recognizing the significance of our work.\n\n### **On the Bias Simplification**\n*Critique: \"The authors set biases to zero as a simplifying assumption. They claim simplification is easy to generalize, but they don't give a convincing reason for that claim.\"*\n\nThis is a fair point. Including bias terms changes the linear transformation in each layer to an affine one: from $W\\alpha$ to $W\\alpha + b$. The proof of non-singularity relies on this map being a submersion (i.e., its derivative being surjective). The derivative of the map $(W, b) \\mapsto W\\alpha + b$ is surjective, just as it is for the map without bias. In fact, the presence of the additive bias term makes the surjectivity argument even more direct. The rest of the inductive proof in Proposition 13 proceeds in exactly the same manner. We will add a paragraph to the appendix to formalize this argument and remove any ambiguity.\n\n### **On the Discussion of Attention Layers**\n*Critique: \"The paper's discussion of attention mechanisms (Proposition 15) is short compared to how well it explains fully connected and convolutional layers.\"*\n\nThe proof for attention layers is indeed concise because the core argument is quite direct. The attention mechanism itself is an analytic function of its parameters $(W_q, W_k, W_v)$ for a fixed input — it consists of matrix multiplication, followed by an application of a softmax function. While attention is complicated, all of the operations involved are analytic, therefore they do not pose any challenges to our work. The almost everywhere analyticity of the full network map is therefore a direct consequence of Lemma 8: composing the analytic attention map with the a.e. analytic output of the preceding layers results in an a.e. analytic map. We will revise the text to better connect the proof in Proposition 15 with the discussion in Appendix A to make the argument more self-contained and clear.\n\n### **On the Practical Significance of \"Almost Everywhere\" Conditions**\n*Critique: \"The use of measure-zero singular sets...lacks a detailed discussion regarding its practical implications.\"*\n\nThe practical significance is profound: if the sets of non-analytic or singular points were \"thick\" (i.e., had positive measure), then standard gradient-based optimizers would frequently fail or become trapped. In such a scenario, one would need to resort to more complex and computationally expensive methods, such as subgradient descent, which are also more difficult to analyze theoretically. Our result, by proving that these pathological sets are of measure zero, provides the rigorous justification for why the simpler, more efficient, and universally used first-order methods like GD and SGD are so effective in practice.\n\n### **On Singular Step-Size Considerations**\n*Critique: \"A more in-depth look at how these unique step sizes might affect adaptive step-size methods...would make the results more useful.\"*\n\nAs we argue, the set of singular step-sizes has measure zero. This provides a strong guarantee of robustness for adaptive methods. Any adaptive algorithm or schedule that selects step-sizes from a continuous range (or even a sufficiently fine-grained discrete set) will have a zero probability of selecting a singular value. Therefore, our result implies that such methods do not need to be specially designed to avoid these specific pathological points, as they are naturally avoided.\n\n### **On the Significance of Results**\nWe thank the reviewer for this comment, which allows us to clarify our work’s novelty. To the best of our knowledge, existing proofs of non-singularity are limited to settings with strong assumptions. For example, the analysis in Lee et al. (2019) relies on Lipschitz smoothnesss, which, as we discuss on lines 97-98, does not hold for many practical deep learning scenarios. Our work is the first to provide a comprehensive proof of non-singularity for the GD map in the more general and practical setting of neural networks with piece-wise analytic activations. If there are specific references the reviewer believes we have overlooked, we would be very grateful for the pointers and would be happy to incorporate them into our discussion. \n\n### **On the Novelty of the Proof Technique**\nFinally, we wish to gently counter the characterization of our proof as not being novel. As we detail in Remark 5, direct application of standard tools like the chain rule fail for the composition of almost everywhere analytic functions, which is the central challenge when moving beyond simple analytic activations. Our key technical innovation, presented in Proposition 6, is a novel \"analogue of the chain rule\" that leverages the specific layered structure of neural networks to overcome this limitation. To our knowledge, this inductive proof technique is a new contribution to the literature."}, {"review_id": "vFqhCo0pKU", "reviewer": "Reviewer_41PX", "summary": "The authors prove that for almost all learning rates the function that applies a gradient descent step to the parameters of a neural network never maps sets of positive measure to null sets, which has been assumed in many theoretical works but without a formal proof so far.", "questions": "Can you justify that the assumption needed for Proposition 14 to work is realistic? In any case, this assumption should ideally be specified in Proposition 7.\n\nWhat exactly does \"different probabilities for sampling mini-batch data\" mean in the caption of Figure 4? Does the expectation of the minibatch gradient equal the full batch gradient?\n\nThe definition of analiticity in lines 185-186 is not correct. The set $N(f_\\theta)$ in Figure 2 is defined only in the appendix. The letter $H$ is used both for the depth of the network and for the Hessian of the loss, which is a bit confusing. It would help the reader if the references were ordered alphabetically.\n\nSome typos: line 197 complemets -> complements, line 340 networks -> network, line 350 of recurrent -> of a recurrent.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "1. Quality: I believe the main results and the corresponding proofs are correct, except the treatment of convolutional neural networks. Proposition 14 in Appendix B assumes that the matrices corresponding to the linear maps that are induced by convolution kernels span the whole space, which seems like an unrealistic assumption and is not mentioned in Proposition 7. The experiment with its results depicted in Figure 4 on the right used \"different probabilities for sampling mini-batch data\", which sounds like a violation of the requirement that the expectation of the minibatch gradient should equal the full batch gradient.\n\n2. Clarity: The paper is well-written, paints a nice picture of relevant works in which the results aim to fill an important whole and the proof ideas are communicated clearly.\n\n3. Significance: The results are already assumed by many works and therefore are expected to be built upon by the community.\n\n4. Originality: While the results were expected to hold, they do fill an important gap in the theory of deep learning. Moreover, the proof techniques can potentially lead to obtaining additional results later on.", "was_revised": true, "final_justification": "The authors properly addressed the issues I raised in the rebuttal. My main concern was the problem with the result concerning convolutional layers, which seems to be solved. Therefore I increase the rating from 4 to 5.", "rebuttal": "We are grateful to the reviewer for their careful and constructive feedback. We are especially thankful for the identification of a subtle error in our proof, which has allowed us to strengthen the paper's technical correctness.\n\n### **On the Assumption for Convolutional Layers (Proposition 14)**\n*Critique: \"Can you justify that the assumption needed for Proposition 14 to work is realistic? In any case, this assumption should ideally be specified in Proposition 7.\"*\n\nWe sincerely thank the reviewer for identifying this issue. Upon reflection, we agree that the assumption in the original Proposition 14—that the matrices corresponding to convolution kernels span the entire ambient space—is not realistic. We have corrected our argument, which we present here and will incorporate into the final version. The claim that convolutions can be represented by matrices with some entries equal to each other can also be found in Chapter 9.1 from Goodfellow et al. (2016).\n\nThe core of the proof remains the same, but we no longer need the unrealistic assumption that the matrices span the ambient space. For a convolutional layer, the weight matrices $W_h$ with equal entries do not span the full matrix space $\\mathbb{R}^{d_h \\times d_{h-1}}$, but rather a smaller linear subspace, let's call it $C_h$. The multiplication map must now be considered as a map $M_c: W_h \\to W_h \\alpha$ with domain $C_h$.\n\nTo show $M_c$ is non-singular, we observe that for any measure-zero set $B$ in the codomain, the preimage is $M_c^{-1}(B) = M^{-1}(B) \\cap C_h$, where $M$ is the multiplication map on the full ambient space. Since $M$ is a submersion for almost all data $\\alpha$, its preimage $M^{-1}(B)$ has measure zero. The intersection of a measure-zero set with a lower-dimensional subspace $C_h$ will also have measure zero within $C_h$ for a generic choice of data. Such a choice is justified if the data is assumed to be noisy. While a pathological choice of data could align $C_h$ within $M^{-1}(B)$, such data points form a measure-zero set themselves (an argument inspired by the genericity results in Cooper (2021)).\n\nWe will therefore revise Proposition 14 to read: \"Let all assumptions and definitions in Proposition 13 hold true. However, suppose that some entries in the matrix $W_h$ are forced to be equal to each other. Then for generic data, the map $F_h$ is a.e. $C^k$ and the set $\\partial Z(F_h)$ has measure zero.\"\n\n### **On the Experimental Details in Figure 4**\n*Critique: \"What exactly does 'different probabilities for sampling mini-batch data' mean in the caption of Figure 4? Does the expectation of the minibatch gradient equal the full batch gradient?\"*\n\nWe apologize for the confusing phrasing. The experiment considers two different data-generating distributions. In each case, the process produces only two points, $x_1$ and $x_2$, with probabilities $p$ and $(1-p)$, respectively. We train with a batch size of $1$. Therefore, $p$ simultaneously defines the underlying data distribution and the probability of sampling a given point. For any fixed $p$, the expectation of the mini-batch gradient is indeed equal to the full-batch gradient for that specific data distribution. The two plots in Figure 4 correspond to two different experiments with different underlying data distributions (i.e., different values of $p$). We will revise the caption in Figure 4 to make this distinction clear.\n\n### **On Minor Corrections**\n*Critique on: definition of analyticity, set definitions in Figure 2, notation for depth ($H$), and typos.*\n* We will correct all of these points in the final version:\n* We will amend the definition of analyticity to explicitly state that the Taylor series must converge to the function.\n* We will add the formal definitions for the sets in Figure 2 to the main body of the text.\n* We will change the notation for network depth to avoid collision with the Hessian.\nAll typos will be corrected, and we will re-order the references alphabetically.\n\nReferences:\n* Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016.\n* Yaim Cooper. Global minima of overparameterized neural networks. SIAM Journal on Mathematics of Data Science, 3(2):676–691, 2021."}, {"review_id": "XpAkZz8ZQk", "reviewer": "Reviewer_bN8y", "summary": "This paper advances the theoretical understanding of gradient descent (GD) optimization in deep neural networks by proving the non-singularity of the GD map, a key assumption for convergence to global minima and avoidance of saddle points. Unlike prior studies that assume non-singularity or rely on restrictive conditions like Lipschitz smoothness or small step-sizes, the authors demonstrate that the GD map is non-singular for almost all step-sizes in realistic neural network architectures, including fully connected, convolutional, and softmax attention layers with piecewise analytic activations (e.g., sigmoid, ReLU, leaky ReLU). The proof first demonstrates that neural network mappings are almost everywhere analytic by leveraging an analogue of the chain rule for neural networks. This property is then used to analyze the measure of preimages under the GD map, establishing its non-singularity.", "questions": "1. Extending Non-Singularity to SGD: The paper claims the non-singularity of the gradient descent (GD) map extends to stochastic GD (SGD), but this is not convincingly supported. Extending GD results to SGD typically requires modeling randomness, introducing significant theoretical complexity. Could the authors provide a rigorous analysis or additional assumptions to justify this extension? \n2. Clarifying the Almost Everywhere Analytic Proof: The proof that neural network mappings are almost everywhere analytic lacks clarity and rigor, particularly for architectures like attention modules, where the appendix provides only brief descriptions without detailed derivations. Additionally, the analysis seems tailored to supervised learning; its applicability to generative model training paradigms is unclear. Could the authors elaborate on the proof for diverse architectures and discuss its relevance to generative models? \n3. Connecting Applications to Theoretical Results: The Discussion section’s applications—periodic trajectories and stable minima of GD versus SGD—feel disconnected from the main non-singularity proof and lack novel insights compared to prior work. Could the authors clarify how these applications leverage the proved results to offer new perspectives or contributions? \n4. Significance of Unrestricted Step-Sizes: The proof emphasizes non-singularity for arbitrary step-sizes, but the practical significance of this claim is unclear. Traditional optimization theory suggests that step-sizes exceeding twice the inverse of the Hessian’s maximum eigenvalue lead to divergence.", "limitations": "1. The use of strong analyticity to prove non-singularity is not justified when weaker smoothness conditions suffice for convergence.\n2. The applicability to stochastic GD lacks analysis of randomness-related complexities.\n3. The proof’s relevance to generative model training is unclear and unaddressed.\n4. The Discussion’s applications (periodic trajectories, stable minima) lack novelty and connection to the main results.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "__Strengths__ \n1. The paper rigorously proves that the gradient descent (GD) map is non-singular for almost all step-sizes in realistic neural network architectures. The proof is clearly presented without apparent theoretical flaws. The core argument begins with establishing the almost everywhere analytic property of neural network mappings. The intuition is that common model components are almost everywhere analytic, with non-analytic points forming measure-zero sets in each layer. Through the composition of these layers, the non-analytic set in the entire parameter space remains measure-zero. Building on this, the authors analyze the measure of preimages in critical and non-critical regions to derive the non-singularity of the GD map. The work appears complete, with a well-structured proof addressing the stated problem. \n2. The significance of the result is tempered by the fact that the strong condition of analyticity—requiring functions to be infinitely differentiable with convergent power series expansions—is used to prove a relatively weaker conclusion of non-singularity. Prior work has directly leveraged analyticity to establish stronger results, such as GD convergence to global minima, while other convergence analyses (e.g., those using the Polyak-Łojasiewicz condition) rely on weaker smoothness assumptions. \n3. Prior work has extensively analyzed the analytic properties of neural networks, particularly for piecewise analytic activations. For instance, studies have shown that neural networks with ReLU activations are almost everywhere analytic. To the best of our knowledge, using the almost everywhere analytic property to prove the non-singularity of the gradient descent (GD) map is novel, offering a new perspective on the theoretical underpinnings of GD optimization in neural networks.\n\n__Weakness__:\n1. The paper is generally well-organized, with a clear logical progression in presenting the proof of the gradient descent (GD) map’s non-singularity, structured around the stepwise demonstration of the core theorem. However, the writing is somewhat difficult to follow due to insufficient explanations of key terms, variables, and the intuitions or practical implications behind the proved lemmas and theorems. \n2. In the section proving that neural network mappings are almost everywhere analytic, the analysis for certain architectures, such as attention modules, lacks rigor in the appendix. The appendix briefly describes the operations and functions of these modules but does not provide a detailed derivation or proof of their analytic properties\n3. The Discussion section presents two applications—periodic trajectories and stable minima of GD versus SGD—that feel disconnected from the main theoretical analysis. These applications address simplistic cases, lack clear connections to the paper’s core results, and do not demonstrate novel contributions compared to existing work on these topics.", "was_revised": true, "final_justification": "Although I believe there are still some points that the paper can optimize as I listed in my comments to authors' rebuttal, I think the paper is ready to be published at NeurIPS. I thus increase my rating to 4.", "rebuttal": "We thank the reviewer for recognizing that our work \"advances the theoretical understanding of gradient descent\" and that our core proof is \"novel, offering a new perspective.\"\n\n### **On the Applicability to Stochastic Gradient Descent (SGD)**\n*Critique: \"The paper claims the non-singularity of the gradient descent (GD) map extends to stochastic GD (SGD), but this is not convincingly supported.\"*\n\nWe apologize if our justification was not sufficiently clear. Our theoretical framework robustly covers the SGD setting. The core of our proof, culminating in Theorem 1, establishes non-singularity for *any fixed data and any batch size*. An SGD step is mathematically equivalent to a full GD step performed on a loss function computed over a mini-batch of data. Since our result holds *irrespective* of the number of data points used to compute the loss (as stated on lines 68-70 and 298-304), it directly applies to each step of an SGD trajectory, where the data changes at each iteration. The map for each step, $G_{\\eta,i}$, is therefore non-singular (for almost all $\\eta$), and the composition of these maps, which describes the full SGD trajectory, is also non-singular. We will revise the text to make this crucial point more explicit and prominent.\n\n### **On the Rigor of the Proof for Different Architectures**\n*Critique: \"The proof that neural network mappings are almost everywhere analytic lacks clarity and rigor, particularly for architectures like attention modules... its applicability to generative model training paradigms is unclear.\"*\n\nWe thank the reviewer for pushing for greater clarity.\n* **Attention Layers**: The proof for attention layers (Proposition 15) relies on a straightforward but powerful observation. The attention mechanism itself, as a map from parameters $(W_q, W_k, W_v)$ and a fixed input matrix $X$, is a fully analytic function. Even though the attention mechanism is rather complicated, *it is made up only of analytic functions*. The almost everywhere analyticity of the full network map arises from the composition of this analytic attention map with the a.e. analytic output of the following layers. As established in Lemma 8, the composition of an analytic function with an a.e. analytic function is itself a.e. analytic. We will expand the appendix to make this argument more self-contained.\n* **Generative Models**: This is an excellent suggestion. While we did not explicitly analyse generative models, our framework is potentially applicable. Many generative training paradigms involve optimizing a loss function. If this loss can be shown to be a.e. analytic with respect to the generator's parameters, our non-singularity results apply. Since we already show that the functions determined by the models are a.e. analytic, all that is left is to show that the specific losses used for generative modelling will preserve this. If the losses are analytic, an argument similar to what we have above, using Lemma 8 for composition of functions, will hold and the GD map will be non-singular. We will add a brief comment on this as a promising direction for future work in the Discussion section.\n\n### **On Connecting the Discussion Section to the Main Result**\n*Critique: \"The Discussion section’s applications—periodic trajectories and stable minima of GD versus SGD—feel disconnected from the main non-singularity proof...\"*\n\nWe appreciate this feedback and will revise the Discussion to make the connection more explicit. The applications presented are direct consequences of our main theorem. The stability analysis of periodic trajectories (Figure 3) and the comparison of stable minima between GD and SGD (Figure 4) both rely on analyzing the behavior of the iterated GD map ($G_\\eta^k$) and the SGD map ($G_{\\eta,i}$), respectively. The non-singularity of these maps, which is guaranteed by our theorem and the property that compositions of non-singular maps are non-singular, is the foundational requirement that makes such stability analyses theoretically sound. Our work provides the missing justification for applying these analytical tools to practical neural networks.\n\n### **On the Significance of Unrestricted Step-Sizes**\n*Critique: \"The proof emphasises non-singularity for arbitrary step-sizes, but the practical significance of this claim is unclear.\"*\n\nWe first emphasise that **the scope of this work was to show that the GD map is non-singular, not to look at other aspects of the optimisation process such as convergence**. This is a crucial point that underscores the significance of our work. Traditional optimization theory (e.g., Lee et al. (2019)) proves saddle-point avoidance under the strong assumption of Lipschitz smoothness, which in turn requires small step-sizes. However, this assumption is violated in many practical, high-performance training regimes. Mohtashami, Jaggi and Stich (2023), for instance, shows that networks are often best trained with large learning rates where the loss is demonstrably not Lipschitz smooth.\n\nOur result is significant precisely because it does not require small step-sizes or Lipschitz smoothness. By proving non-singularity for almost all step-sizes, we provide the first rigorous theoretical foundation that helps explain the success of these empirically validated, aggressive training strategies. Thus, our work allows for the extension of relevant guarantees from the optimisation literature to the practical neural networks setting as well. We will revise the introduction and discussion to more strongly emphasise this important connection.\n\n### **On the Generality of Our Work**\nWe agree that prior works have used analyticity to prove stronger results, such as convergence. However, the analyses *do not hold a priori* for the piecewise analytic setting. This was a key motivation for our work, which aims to bridge this gap. Our contribution is to establish the non-singularity of the GD map in this more general piecewise analytic setting (e.g. for networks with ReLU activation). This result is a foundational prerequisite that enables the rich body of convergence analyses to be extended to these more practical network architectures.\n\n### **On the Scope of Our Claims**\nTo clarify, our paper does not claim to prove convergence. Our focus is specifically on proving the non-singularity of the gradient descent map. We view this as a critical intermediate step that validates a key assumption used in many convergence analyses. By establishing this property, our work provides the necessary tools for future research to carry over convergence results to the piece-wise analytic setting.\n\n### **On the Polyak-Łojasiewicz (PL) Condition**\nRegarding the comparison to the PL condition, we believe there may be a misunderstanding of the relationship between these concepts. Piecewise analyticity is more akin to a structural property, whereas the PL condition is a geometric property. These are distinct and non-hierarchical assumptions; a function can be piecewise analytic without satisfying the PL condition, and vice-versa. Our work focuses on the structural properties stemming from the network’s architecture, which is a different and complementary line of analysis. We will add a brief note to the related works section to clarify this distinction.\n\nReferences:\n* Mohtashami, A., Jaggi, M., and Stich, S. 2023. Special Properties of Gradient Descent with Large Learning Rates.\n* Lee, J. D.; Panageas, I.; Piliouras, G.; Simchowitz, M.; Jordan, M. I.; and Recht, B. 2019. First-order methods almost always avoid strict saddle points."}, {"review_id": "N6Tps9nToS", "reviewer": "Reviewer_9vB6", "summary": "The paper proves that, for neural networks with piece-wise analytic activations (ReLU, sigmoid, etc.), the gradient-descent map (GD map) is non-singular (it does not turn a measure-zero set into a set with positive measure) for neural networks with piece-wise analytic activations and a fixed learning rate $\\eta$. The proof uses standard real-analytic arguments and covers fully connected and convolutional nets. A 2-D toy example is shown.", "questions": "- Can you give a concrete bound or heuristic for choosing $\\eta$ outside the measure-0 “bad’’ set?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Clear mathematics and complete proofs.\n- This paper revises an existing theoretical gap for ReLU-type networks.\n\nWeaknesses \n- Result has little practical impact because, on bounded parameter sets, losses are already Lipschitz and earlier non-singularity arguments apply.\n- Although the experimental results is helpful for showing the theory presented in this paper, in practical setting the learning rate changes (decays) during the learning.  The experiment provides little implication to practical situation in machine learning.", "was_revised": true, "final_justification": "I agree that the paper’s rigorous proofs and the new layered chain-rule lemma provide a somewhat meaningful theoretical advance. Although the experimental validation is limited and the paper does not offer precise guidance on choosing η, these shortcomings are not fatal. Thus, a “4 – Borderline Accept” recommendation is reasonable.", "rebuttal": "We sincerely thank the reviewer for their time and for their assessment that our paper features “clear mathematics and complete proofs\" and \"revises an existing theoretical gap.\" We appreciate the opportunity to clarify the practical impact of our work and answer the questions raised.\n\n### **On the Practical Impact and Boundedness of Parameters**\n*Critique: \"Result has little practical impact because, on bounded parameter sets, losses are already Lipschitz and earlier non-singularity arguments apply.\"*\n\nThanks for raising this important point, which allows us to highlight a key motivation for our work. While it is true that prior non-singularity results apply to Lipschitz-smooth losses, the assumption that GD trajectories remain in a bounded set where the loss is Lipschitz is itself a strong condition that lacks theoretical grounding. To our knowledge, no existing results guarantee that the GD trajectory will remain bounded without a priori assuming Lipschitz smoothness and a sufficiently small step-size. Relying on the empirical observation that successful training runs stay bounded can be a form of survivorship bias, where initializations and hyperparameters leading to divergence are simply discarded.\n\nOur work's practical significance lies in removing this restrictive assumption of Lipschitzness (or, equivalently, of bounded parameter sets). We provide the first non-singularity guarantee that holds even when the trajectory is not confined to a bounded region. This is particularly relevant to modern training paradigms like \"Edge of Stability\" (EoS) training, where networks are intentionally trained with large learning rates that violate the conditions for the Gradient Descent Lemma (used to prove convergence). Our result provides the foundational theoretical justification on top of which other works built in order to explain why optimization in these practical, high-performance regimes does not fail, thus bridging a critical gap between theory and practice.\n\n### **On the Implications for Varying Learning Rates**\n*Critique: \"Although the experimental results are helpful for showing the theory presented in this paper, in practical setting the learning rate changes (decays) during the learning.\"*\n\nThis is an excellent point, and we want to clarify that our framework directly extends to settings with varying learning rates, such as those with schedules or adaptive methods.\n\nOur main result (Theorem 1) establishes that the GD map, $G_\\eta$, is non-singular for almost all step-sizes $\\eta$. An optimization process over $T$ steps with a learning rate schedule $(\\eta_1, \\eta_2, \\dots, \\eta_T)$ can be described by the composite map $G = G_{\\eta_T} \\circ \\dots \\circ G_{\\eta_1}$. A fundamental property of non-singular maps is that their composition is also non-singular. Since each individual map $G_{\\eta_t}$ in the sequence is non-singular (with probability one), their composition $G$ is also guaranteed to be non-singular.\n\nTherefore, our analysis provides a rigorous foundation for understanding optimization dynamics not just for a fixed step-size, but for any practical scheme that relies on composing gradient descent steps. The same reasoning applies if we also use different data for the batches at each step of the optimization, illustrating how Theorem 1 also extends to SGD as well. We will add a sentence to the Discussion to make this important implication more explicit.\n\n### **On Choosing a \"Good\" Learning Rate**\n*Critique: \"Can you give a concrete bound or heuristic for choosing η outside the measure-0 “bad’’ set?\"*\n\nBefore we answer, we have to clarify what it means to have a “good” or a “bad” learning rate. In this work, a “bad” learning rate is one out of a measure zero set for which the GD map is singular; **a “good” learning rate is one that is not “bad”, i.e., one for which the GD map is non-singular**. With this clarification, our theoretical result provides a strong and practical heuristic. The proof of Proposition 11 shows that a step-size $\\eta$ is \"bad\" (i.e., GD map is singular) only if 1/$\\eta$ is precisely an eigenvalue of the loss Hessian. Our work establishes that the set of all such \"bad\" values has Lebesgue measure zero. In practice, this means that the probability of selecting a \"bad\" step-size by chance is zero for any selection method that is absolutely continuous with respect to the Lebesgue measure (this covers common heuristics). Schedules or adaptive methods are also included, since the probability of sampling a finite number of learning rates from the “bad” ones is still zero. Any other criterion for labeling a step size as “good” does not fall within the scope of this work.\n\n### **On the Novelty of the Proof Technique**\nFinally, we wish to gently counter the characterization of our proof as using \"standard real-analytic arguments.\" As we detail in Remark 5, standard tools like the chain rule fail for the composition of almost everywhere analytic functions, which is the central challenge when moving beyond simple analytic activations. Our key technical innovation, presented in Proposition 6, is a novel \"analogue of the chain rule\" that leverages the specific layered structure of neural networks to overcome this limitation. To our knowledge, this inductive proof technique is a new contribution to the literature"}]

Author rebuttal:

Meta-review: The authors prove that for almost all learning rates the function that applies a gradient descent step to the parameters of a neural network never maps sets of positive measure to null sets, which has been assumed in many theoretical works but without a formal proof so far.

The paper is clearly written and features a novel proof technique.  It generalizes existing theoretical results to neural networks using popular activation functions such as ReLU, sigmoid, and leaky ReLU.  The results apply to a wide range of network architectures, such as fully connected, convolutional, and attention layers.

Reviewers expressed concerns about several issues, including setting biases to zero, correctness and completeness of handling convolutional and attention layers, connectedness of applications to the theoretical results, and relevance of the experimental results to practice.

The authors provided a comprehensive rebuttal, and there was some discussion following it.  The authors addressed most of the concerns, and several of the reviewers raised their ratings.  In the end, a consensus was reached among the reviewers, with all either proposing or not opposing acceptance.

I agree that this paper will provide an interesting contribution to the conference, with potentially significant impact on future theoretical works.  The problem is fundamental, and it was addressed successfully and in reasonable generality.
Decision: Accept (poster)