Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding
Authors: Frank Zhengqing Wu
Berfin Simsek
François Gaston Ged
Venue: ICLR 2025 Poster (2025)
Keywords: loss landscape
non-smooth
training dynamics
network embedding
over-parameterization
neural network
Primary area: optimization
TL;DR: We study the stationary points of the loss of shallow ReLU-like networks fully considering the non-smoothness, thereby offering insight into the training dynamics and the reshaping effect of network widening on the stationary points.

Abstract: In this paper, we study the loss landscape of one-hidden-layer neural networks with ReLU-like activation functions trained with the empirical squared loss using gradient descent (GD). We identify the stationary points of such networks, which significantly slow down loss decrease during training. To capture such points while accounting for the non-differentiability of the loss, the stationary points that we study are directional stationary points, rather than other notions like Clarke stationary points. We show that, if a stationary point does not contain "escape neurons", which are defined with first-order conditions, it must be a local minimum. Moreover, for the scalar-output case, the presence of an escape neuron guarantees that the stationary point is not a local minimum. Our results refine the description of the *saddle-to-saddle* training process starting from infinitesimally small (vanishing) initialization for shallow ReLU-like networks: By precluding the saddle escape types that previous works did not rule out, we advance one step closer to a complete picture of the entire dynamics. Moreover, we are also able to fully discuss how network embedding, which is to instantiate a narrower network with a wider network, reshapes the stationary points.

Published as a conference paper at ICLR 2025
Loss LANDSCAPE OF SHALLOW RELU-LIKE NEURAL NETWORKS: STATIONARY POintS, SADDLE ESCAPE, AND NETWORK EMBEDDING
Frank Zhengqing Wu Berfin Şimgek² François Gaston Ged
1 Ácole Polytechnique Fédérale de Lausanne, Lausanne (EPFL.), Switzerland
2 New York University, New York, USA
3 University of Vienna, Vienna, Austria
ABSTRACT
In this paper; we study the loss landscape of one-hidden-layer neural networks with ReLU-like activation functions trained with the empirical squared loss using gradient descent (GD). We identify the stationary points of such networks, which significantly slow down loss decrease during training. To capture such points while accounting for the non-differentiabiliyy of the loss, the stationary points that we study are directional stationary points, rather than other notions like Clarke stationary points. We show that, if stationary point does not contain "escape neurons", which are defined with first-order conditions, it must be local min- imum. Moreover, for the scalar-output case, the presence of an escape neuron guarantees that the stationary point is not local minimum. Our results refine the description of the saddle-to-saddle training process starting from infinitesimally small (vanishing) initialization for shallow ReLU-llkee networks: By precluding the saddle escape types that previous works did not rule out, we advance one step closer to complete picture of the entire dynamics. Moreover, we also able fully discuss how network embedding, which is to instantiate a narrower network with wider network, reshapes the stationary points.
INTRODUCTION
Understanding the training process of neural networks calls for insights into their loss landscapes. Characterization of the stationary points a crucial aspect of these studies. this paper, we investi- gate the stationary points of the loss of a one-nindeee meehiddee--layrr neural network with ReLU-like activation functions. The non-differentiabiliiy of the activation function renders such problems non-trivial. It worth noting that, although the non-differentiable areas only take up zero Lebesgue measure in the parameter space, such areas are often visited by GD and thus should not be neglected.
In particular, we discover that any loss-decreasing path starting from a stationary point must involve changes in the parameters of "escape neurons" (defined in Definition 4. The absence of escape neurons guarantees the stationary point become local minimum. Our results provide insight into the saddle escape process with minimal assumptions. Supplementing Maennel et al. (2018); Boursier et al. (2022); Chistikov et al. (2024); Boursier and Flammarion (2024); Kumar and Haupt (2024), our results lead to a fuller understanding of the saddle-to-sadllt training dynamics, a typical dynamical pattern resulted from vanishing initialization. More specifically, those previous works studied the behavior of gradient flow near specific type of saddle point (Kumar and Haupt, 2024), and this work precludes the existence of other types in the general case.
We are also able to systematically describe how network embedding reshapes the stationary points by examining whether the "escape neurons" are generated from the embedding process. This directly extends the discussion by Fukumizu et al. (2019) oon-differentiable cases.
"Correspondence zhenggi ngg wu@epfl .ch, fged. . matdh@maaahaa com.

---

Published as a conference paper at ICLR 2025
1.1 RELATED WORK
Stationary Points. Stationary points abound on the loss landscape of neural networks. He et al. (2020); Sharifnassab et al. (2020); Liu al. (2021); Arjevani and Field (2021); Şimşek et al. (2023) have shown that saddle points and spurious local minima exist in the landscape of non-linear net- works. Stationary points may significantly affect training dynamics, leading plateaus in the loss curve (Saad and Solla, 1995; Amari, 1998; Jacot et al., 2022; Pesme and Flammarion, 2024). When studying these stationary points, is of both theoretical and practical interest (Fukumizu and Amari, 2000; Şimşek et al., 2021) to discriminate between their different types (Achour et al., 2022), for they affect gradient descent differently (Lee et al., 2017: 2016). Specifically, while local minima and non-strict saddles are not escapable (Achour et al., 2021; Lee et al., 2016; Achour et al., 2022), strict saddles are mostly escapable under mild conditions (Lee et al., 2017: 2016; Jin et al.. 2017; Daneshmand et al., 2018; Ziyin et al., 2023). In our setting, the con-differentiaaility complicates the analysis. Too tame such difficulty, previous works took various simplifications, such as only studying the differentiable areas (Zhou and Liang, 2018; He et al., 2020; Sahs et al., 2022), only studying the non-differentiable stationary points yielded by specific constructions (Liu et al., 2021), and only studying the first two orders of the derivatives (Yun et al., 2019). These theories appeared inconclusive due to such simplifications. So far, a systematic characterization of the potentially non non-differentiable stationary points in our setting has yet to be established.
In this work, we consider stationary points to be where the one-sided directional derivative (ODD) toward any direction is non-negative, which known as directional stationary points (Li et al., 2020). This notion effectively captures the points on the loss landscape that slow down GI) and create loss plateaus, whether differentiable or not (see Section 3.2 and Appendix C.2). Such a property of this notion remained unnoticed by previous works. Earlier literature on ReLU network training dynamics (Boursier et al., 2022; Lee et al., 2022; Kumar and Haupt, 2024) observed such GD- stagnating points for specific cases but did not provide conditions to identify them in broader setup. Notably, our notion of stationarity also contrasts with other methods based on Clarke subdifferential (Wang et al., 2022: Davis et al., 2020a), subgradient, or right-hand derivative (Cheridito et al., 2022), which may be less suitable for characterizing GD stationarity (see Section 3.2.
Training Dynamics. Different initialization scales lead to different training dynamics. In the lazy- training regime (Chizat et al., 2019), which has relatively large initialization scales, the dynamics is well captured by the neural tangent kernel theory (Jacot et al., 2018; Allen-Zhu et al., 2019; Arora et al.. 2019). In the regime of vanishing initialization, the training exhibits a feature learning be- havior (Yang and Hu, 2020), which still awaits a generic theoretical account. However, has been widely observed that the training dynamics in this regime often displays a saddle-to-sadde patter,, meaning the loss will experience intermittent steep declines punctuated by plateaus where it re- mains relatively unchanged. We will describe this process for our setting, furthering the discussions from Maennel et al. (2018); Boursier et al. (2022); Chistikov et al. (2024); Boursier and Flammar- ion (2024); Kumar and Haupt (2024). It is noteworthy that the training dynamics often unveils the implicit biases of the optimization algorithms. In the saddle-to-san tions often prefer lower parameter ranks (Maennel et al., 2018; Luo et al., 2021; Jacot et al., 2022) or smaller parameter norms (Boursier et al., 2022; Pesme and Flammarion, 2024), which could be beneficial for generalization.
regime, the learned func-
Network Embedding. Network embedding provides a perspective for understanding how the optimization landscape transforms given more parameters, shedding light the merit of over- parameterization (Livni et al., 2014; Safran and Shamir, 2015; Soudry and Carmon, 2016; Nguyen and Hein, 2017: Soudry and Hoffer, 2017: Du and Lee, 2018: Venturi et al., 2019; Soltanolkotabi et al., 2022). Previously, network embedding has been studied in various setups (Fukumizu and Amari, 2000; Fukumizu et al., 2019; Safran et al., 2020; Şimşek et al., 2021; Zhang et al., 2021), mostly focusing on how network embedding reshapes stationary points. We extend this line of work to the non-differentiable cases.
1.2 MAIN CONTRIBUTION
In this paper, we study the aforementioned topics for one-hidden-layer networks with ReLU-like activation functions trained by the empirical squared loss. Our contributions are as follows.
Noticing that the non-differentiabiliyy only lies within the hyper-planes orthogonal to training in- puts, and derivatives can be handled easily both sides of those hyperplanes, we develop rou-
2

---

Published as a conference paper at ICLR 2025
tine to fully investigate the ODDs of the loss. With its help, we identify, classify, and characterize the stationary points with con-differentiabiiity fully considered.
We preclude saddle escape types that previous papers (Maennel et al., 2018; Boursier et al., 2022; Chistikov et al., 2024; Boursier and Flammarion, 2024; Kumar and Haupt, 2024) were not able rule out. We show that saddle escape must involve the parameter changes of escape neurons (Definition 4.1) and apply this to describe the saddle-to-saddle dynamics.
We study whether network embedding preserves stationarity or local minimality.
SETUP
Network Architecture.
We study one-hidden- layer networks as illlustrated in Figure
For given input x E R the network output is:
R!ll
(I) In Equation (I), W R and H (hzs) E RMXI are respectively the imput and output weight matrices, and I and are the sets of indices of the hidden neurons and output neurons. The vector P E R² contains all the trainable parameters W and H, that D = J /7] [T] x d. We denote the row of W corresponding to hidden neuron by W; E Rd. The componentwise activation function p(-) reads:
y(H, w = ((P:x) = Hx))))
IRd
R
Ryy
input
layer
hiddlen layen
Figure I: network architecture
p(=) = =a =200 + a 2<0= at a R.
In this paper, the assumptions we make are a * a and d > 1. Both are for brevity in exposition, rather than being indispensable for the validity of the outcomes.
Loss Function. The training input/targets x with & K, where K denotes the set of the
sample indices. The empirical squared loss function takes the form:
C CPP) = 2
Σ [Px - yul 1 ΣΣ JEJ
Σ
(b), yky))²
(2)
where ý1 (P) = xF:),,, and Duj Mkj are the components of ÿf and yı at the output neuron j.
3 STATIONARY POINTS
In this section, we derive the ODDs and identify the stationary points of the loss.
3.1 DERIVING ONE-SIDED DIRECTIONAL DERIVATIVES
Mon-differentiabblity does not completely prohibit the investigation into derivatives. For example, studying the left and right hand derivatives the origin of f(x) = |x| suffices to describe the func- tion locally, despite the non-differentiabiliyy. Generalizing this methodology to higher dimensions, can handle the derivatives of the loss function by deriving them in different directions, Note that the loss of ReLU-like networks is continuous everywhere and piecewise C which enables us to study its ODDs, and the ODDs are informative enough to characterize its first-order properties.
It is not hard to see that our neural network is always differentiable with respect to the output weights hye- con-differentiability only arises from input weights w;. More precisely, the network (thus the loss) is non-differentiable at P if and only if weight w; is orthogonal to some training input x 's (see Figure for an illustration). Nonetheless, when w; moves a sufficiently small distance along a fixed direction, the sign of w; X₁ is fixed for all & E K. Namely, W; X only moves on either the positive leg. the negative leg of the activation function p(-), or stays at the kink, but will not cross the kink during the movement. With w; constrained within such small local region, the loss is polynomial with respect to it. This allows us to study the ODDs. Below, we specify some general directions where the ODDs be written in closed forms.
3

---

Published as a conference paper at ICLR 2025
Figure A diagram demonstrating the non-diferreticabiiiy with re- spect the input weights. Here, we show case where input weights and training inputs are 3-dimensional. There are two training inputs and two input weights. W1 lies in plane orthogonal to X2- The loss is thus locally non-differentiable since contains the term p(w, x2). However,
W1
compute the ODD with respect to W1 since the loss function with w1 constrained on either side of the plane or on the plane polynomial of W1.
Definition 3.1. Consider nonzero W, E R The radial direction of w, is U == T[:: Moreover, any unit vector v, orthogonal u, is called tangential direction. For w; the radial direction does not exist, and a tangential direction is defined to be any unit vector v, E IR..
Remark 3.2. The tangential direction is always definable given our assumption of
Consider a non-zero W, E R and let w = w; + Aw; * w, be in the neighborhood of w,. The vector Aw; admits unique decomposition Aw; Ar,u, As;v; with As; 0 and v, being a specific tangential direction of w, Fixing direction for the derivative is thus equivalent to fix- ing tangential direction. For generic function S. 0f(w:) == limar,-00+ ((w:+Ar.u,)=fiee f(w,)) and dr:
Ar;
0 @f(w)) ds;
limiss -0 f(w; + Aa,
are the rudial derivative and the langential derivative.
As;
For = O, we can likewise define the tangential derivative for every tangential direction v;. and by convention, set the undefinable u, and the radial derivative to zero.
Remark 3.3. The notation of a always implies fixed tangential direction V;-
Ds;
We can then study the ODDs with respect to the output weights and the radial/tangential directions of the input weights.
Lemma 3.4. We first denote (ūkj ykj) Ckj.
Then, we define the following quantities:
d == Σ
at atenax=++ Σ
Kky Xks
d
A:
wi-K
d
Σ
a xxyxxx
Σ
ekyyk
(3)
Asi S
(w;+As2V))xxx0
We have that
OC(P) w, djb EJx
(4)
ahys
0C(P)
Σ Ehndi-u u. VieI.
(5)
Ör:
JEJ
OC(P) 0s.
= hy Vo
ViEl, 1, V tangential direction v,'s w.- W;;
(6)
JEJ
The proof for the above in Appendix A. Such ODD computation can also be adapted for neural networks with more than one hidden layer, which is explained in Appendix B.
is easy to check that the network is linear with respect to the output weights and the norm of the input weights, therefore ØC and Thys dr,
are continuous everywhere. But this is not the case for ac de,
3.2 IDENTIFYING STATIONARY POINTS
To capture the GD-stagnating points on the loss landscape while accounting for the non- differentiability, we invoke the notion of directional stationarity to define stationary points.
Definition 3.5. A set of parameters P is stationary point of the loss in our setting if lima ((+ad)-L(P 2 0, for all d E D.
Notice that the above definition of stationarity also incorporates the smooth stationary points, which where the gradient vanishes and also exist in the loss landscape of ReL.Ulikke networks. By
4

---

Published as a conference paper at ICLR 2025
precluding first-order loss- loss-decreasing paths in all directions, the above definition captures points near which GD infinitely decelerates or effectively comes to a halt.' Capturing these phenomena is the desideratum of the notion of stationary points, which Definition 3.5 intuitively satisfies. We showcase this on a scalar function in Appendix C.1. We also provide examples of non-differentiable stationary points in Appendix C.2.
Exploiting the structure of our problem, we arrive at an equivalent definition of stationary points:
Definition 3.6. A set of parameters P is a stationary point of the loss in our setting if the following
holds: (1) (C(P) = 0, V (1,t) E (2) ör;
82(P) = 0, V E I: (3) ds; 2 0, v E I, V tangential
8C(P)
ahgs direction V 's of W.
For directions along which the loss is continuously differentiable, we require the derivatives to be O, which explains the first two conditions. Along the other directions, our definition permits upward slopes in one direction and the opposite at a stationary point, as indicated in the third condition.
The equivalence between Definition 3.5 and Definition 3.6 is intuitive. For a generic function, there is a maximal subspace in its parameter space where the function is differentiable, which we call the differentiable subspace for convenience. The complementary subspace of the differentiable subspace is where the function is non- dififferentiable, which we call the non- non-differentiable subspace.
To examine all the ODDs surrounding a point in the parameter space, as required by Definition 3.5, it suffices to check the ODDs in the differentiable subspace and the non-differentiable subspace separately. This is because an ODD toward an arbitrary direction d in the parameter space can be written linear combination of the ODD along the projection of d onto the differentiable subspace and the ODD along the project of d onto the non-differentiable subspace. The ODDs along the coordinate axes that span the differentiable subspace differentiable subspace, hence the first two conditions in Definition 3.6. However, the ODDs toward all the directions in the -indeffferentiblee subspace need to be examined in the general case, hence the last condition in Definition 3.6.
sufficient to describe any ODD within the
Notably, Definitions 3.5 and 3.6 are tailored to capture the stationarity of GD it admits positive slopes around stationary points. If we were to define stationarity for, say, gradient ascent, we would admit negative slopes around stationary points instead. One implication of such a design is that, while all local minima are stationary points under Definitions 3.5 and 3.6, this is not the case for non dindiffeernntable local maxima. An interesting side note can be made about the rarity of local maxima on our loss landscape, whether they are diffcrentiable or not, We can prove that necessary condition for P E RR to become a local maximum (P) = for all E K, as shown in Appendix D. Such result extends the arguments regarding the non- non-existence of differentiable local maxima by Liu (2021); Botev et al. (2017), offering a qualitative perspective of the loss landscape.
Definition 3.7. A saddle point is a stationary point that is not local minimum or a local maximum.
3.2.1 COMPARISON WITH OTHER NOTIONS OF STATIONARITY
Previous results on non-smooth non-convex stationarity mostly focus on Clarke stationary points (Davis et al., 2020b; Yun et al., 2019), which is where the Clarke subdifferential (Clarke, 1975) includes zero. But, such notion of stationarity is not the best fit for studying GD. For example, the origins of fxx) = |x| and f(x)= = - |o|| are both Clarke stationary points, but only the former stalls GD and qualifies
stationary point of GD. Notably, directional stationary points are Clarke stationary points with no negative slopes around (see Appendix C.3). As Clarke stationary points are known to be where stochastic subgradient descent converges (Davis et al., 2020b), our notion of stationarity offers a more refindd understanding of such first-order methods. It is also noteworthy that Clarke subdifferential (Clarke, 1975) is a widely adopted tool for characterizing gradient flow of ReLU networks (Boursier et al., 2022; Chistikov et al., 2024: Kumar and Haupt, 2024). Given directional stationarity is more relevant for training dynamics, it might be more advisable to use Fréchet subdifferential, which underlies the definition of directional stationarity Li et al. (2020). ?
We consider GD oscillating around a certain point to be effectively equivalent to GD coming to halt. Pn plain words, if the Fréchet subdifferential at point contains O, it is directional stationary point. For details, please refer Li al. (2020) for review.
5

---

Published as a conference paper at ICLR 2025
Another widely known stationarity criterion is to have the subgradient set to contain O. However, such a notion does not apply to our problem since the subgradient cannot be defined for non-convex functions (see Remark C.4).
Cheridito et al. (2022) defined the stationary points of ReLU networks to be where the right-hand directional derivatives along the canonical coordinate axes are zero. However, in the presence of non-smoothnesss, ODDs derived along the canonical axes might not be informative enough to char- acterize the local structure of the function. We demonstrate this with a toy example in Appendix C.4.
4 MAIN RESULTS
In this section, we classify and characterize stationary points (Section 4.1) and discuss its implication for the training dynamics in the vanishing initialization limit (Section 4.2). We will also describe how network embedding reshapes the stationary points (Section 4.3).
4.1 PROPERTIES OF STATIONARY POINTS
Definition 4.1. At a stationary point, hidden neuron is an escape neuron if and only if there exist j' E J and a tangential direction such that @C(P) ΣjeJ hedd v; = 0. and d * O. 0s,
Theorem 4.2. Let be a stationary point. P does not contain escape neurons, then P is a local minimum. Furthermore, this sufficient condition is also a necessary one when |J| = 1.
Remark 4.3. An intuition of Definition 4.1 and Theorem 4.2 for the scalar-output case is in Ap- pendix E. Theorem 4.2 is proved by Appendix F. and Appendix F.2.
Proving Theorem 4.2 requires performing Taylor expansion methodically toward all directions in the parameter space. Taylor expansion requires differentiability and thus might seem, at first glance, pro- hibited for our problem. We solve this by aligning the coordinate system with the non-differentiable edges in the parameter space, so that the loss is always C within each orthant. Then, we can use Taylor expansion based on the ODDs (of any orders) along the aligned coordinate axes to accurately characterize the function within the orthants, as it would be like studying a differentiable function but limiting the scope an orthant. We provide visualization for such a method in Appendix G.
Since we can also compute the ODDs for deeper nets with an aligned coordinate system, as show- cased in Appendix B, it is conceptually easy to also compose such stationary-point-lassse pllnt--lassiiiggig the- orems for deeper nets. However, deeper nets entail a more cumbersome proof as there will be higher-ordered terms to be controlled. Hence, we limit our current discussion to the shallow struc- ture, which already suffices to solve unresolved problems from previous works of training dynamics and network embedding, discussed in Sections 4.2 and 4.3, respectively.
In the following, we refer to local minima without escape neurons as type-l local minima, and the others as rype-2 local minima, which does not exist when |J] = 1 as per Theorem 4.2. We tend to argue that type-2 local minima are rare and provide our reasoning in Appendix F.2.1.
We also show that the stationary local maxima and saddle points of the loss offer "second-order decrease" along the escape path, which can be formalized as below.
Corollary 4.4. If P is non non-minimum stationary point of L with [J] = 1, then there exist unit vector & E RR. and a constant C < such that 0 > C(P + 86) L(P) C82, as d + 0+.
Corollary 4.4 is explained in Remark F.2. is established by characterizing the loss-decreasing path caused by the variation of escape neuron parameters. It draws similarities between the potentially non-differentiable saddle points in our setting and smooth strict saddles. It is known that saddle points in shallow linear networks are strict (Kawaguchi, 2016).². The above corollary thus serves to extend (Kawaguchi, 2016) to condiifferentialee stationary points.
4.1.1 A NUMERICAL EXPERIMENT FOR THEOREM 4.2
We present a numerical experiment to illustrate Theorem 4.2, also serving as precursor to our discussion on training dynamics in Section 4.2. We train ReLU network that has 50 hidden neurons
(Kawaguchi, 2016) also proved that saddle points in the loss of shallow ReI.U networks strict.

---

Published as a conference paper at ICLR 2025
1203
1000
3.5
grese 1
grup
#20ch
sppc1
1200h
4200h
(a) loss curve
(b) direction of w; (rad)
(c)
will
(d) 3
Figure 3: Evolution of all parameters during the training process from vanishing initialization.
(a) The loss curve encounters three plateaus, the last of which corresponds to local minimum (confirmed by Theorem 4.2). We mark the end of the plateaus with dashed vertical lines. (b) The input weights that are not associated with dead neurons are grouped at several angles. (c) & (d) Grouped neurons have their amplitude increased from near-zero values, which coincides with the saddle escape. The movements of ||wll and f are synchronous (see 4.5).
with GD using 5 training samples (xx, yz). The inputs are Xx: = (mx,, 1) E RR² and ny 's are -1, -0.6. 10.1, 0.3, 0.7. The target yı s, are 0.28, -0.1, 0.03, 0.23, -..22. All the parameters initialized independently with law N (0, (5 x 10 (3). The training lasts 500k epochs with step size of 0.001. We will identify the saddle points and local minima encountered during training.
The training process is visualized in Figure 3. Figure 3a is the loss curve, where the plateaus re- flect the parameters moving near stationary points. Figures 3b 3d show the evolution of input weight orientations, input weight norms, and output weights. The input weight orientations are the counterclockwise angles between wy's and the vector (1,0).
Each curve in Figures 3b to 3d corresponds to one hidden neuron. We see that most of the w,'s group at several angles before all the parameters gain considerable amplitudes. This has been well studied by Maennel et al. (2018); Kumar and Haupt (2024); Boursier and Flammarion (2024). We denote these hidden neurons by red and yellow in the figures and call them group and group neurons. Notice that the w; 's between 3.93 and 5.67 rad (the grey-tinted area in Figure 3b) have p(w; {2) = 0 for all k E K. This means that the gradients of all the parameters associated with these hidden neurons zero (from Lemma 3.4). These hidden neurons the black curves in the figures and are called the dead neurons.
We also note that the group 1 neurons are attracted and stuck direction (<: 2.53 rad) that is orthogonal to one of the training inputs (x; = (0.7, 1)). This implies that the GD traverses a non-differentiable region of the loss landscape throughout the training.
In our example, the parameters escape from 2 saddles, which happens at about epochs 42k and 152k, reflected by the rapid loss drop in Figure 3a. The last plateau in the loss curve is the result of local minimum. To numerically verify this, we perturb the parameters obtained after 500k epochs of training with small noises, and the resulting loss change is always non-negative (see Appendix H).
That the last plateau is a local minimum can also be confirmed by applying Theorem 4.2. In the following, we show that the stationary points that the network parameter is close to at epoch 42k and 152k have escape neurons, while the stationary point stalling the loss decrease in the end does not. For clarity, the index of the only output neuron specified to be jo-
Fact 4.5. Throughout training, (||w,]12- - remains unchanged." 6 Thus, with vanishing (small but non-vanishing, resp.) initialization, we have ||w:]l |/ ([]w:]] Rz |/jool resp.) during training.
Definition 4.1 indicates that the escape neurons have hior = (also explained in Appendix E). By Fact 4.5, the amplitudes of all the parameters associated with escape neurons must be small. On the other hand, dead neurons (which only exist in the ReLU case) are not escape neurons by definition.
Thus, when the network nears a saddle point, it must contain neurons that have small amplitudes and are not dead neurons. We will refer to such neurons as small living neurons in the following. They are the ones corresponding to the escape neurons in the nearby saddle point. In Figure 3, at
"We chose such initialization scale to simulate the vanishing initialization since further diminishing it will not qualitatively change the patterns of the training dynamics.
Note, there are black curves in Figures 3c and 3d. They stay too close to the zero line to show up.
"This conclusion was proved in Section 9.5 of Macnnel al. (2018).
7

---

Published as a conference paper at ICLR 2025
epoch 42k (152k, resp.), neurons belonging to both group 1 and 2 (group 2, resp.) are small living neurons. However, after the network reaches the last plateau, small living neurons are depleted, meaning the stationary point it was approaching must be a local minimum. A formal statement of such observations for vanishing initialization is in Corollary 4.6.
We also conducted two other numerical experiments whose results are presented in Appendix I. One used 3-dimensional input and scalar output, and the other used 2-dimensional input and output. The former shows similar behavior as the experiment in this section, the latter reveals escape neurons that have non-zero output weights, which exist when J| > 1. as predicted by Definition 4.1.
4.2 SADDLE ESCAPE IN THE TOODLE-TO-SADDLE DYNAMICS
Networks trained from vanishing initialization exhibit saddle-to-saddie dynamics (Boursier et al., 2022: Jacot et al., 2022; Pesme and Flammarion, 2024), meaning the loss experiences intermittent decreases punctuated by long plateaus. Multiple previous works have studied this dynamics (Maen- nel et al., 2018; Boursier et al., 2022; Chistikov et al., 2024; Boursier and Flammarion, 2024). How- ever, previous discussions regarding saddle escape were usually premised on strong assumptions, such as orthogonality (Boursier et al., 2022) or correlatedness of training inputs (Chistikov et al., 2024). Without such assumptions, previous works could not go beyond the escape from the origin, which is trivially a saddle point (Maennel al., 2018; Boursier and Flammarion, 2024). These simplifiatiion//arrwiing imposed a specific form on all the saddle points being discussed. Such a form was summarized by Kumar and Haupt (2024), and they further noted that there could be other types of saddle points that remained undiscussed (in their Section 5.2). However, we show that no other types of saddle points need be considered in such dynamics and demonstrate qualitatively how saddle points are escaped from. Aligned with previous works, the discussion in this section is for scalur- ReLU networks.
We can see that, Theorem 4.2 and Fact 4.5 only allow gradient flow from vanishing initialization to approach saddle points that have hidden neurons whose parameters are all zero,' which coincide exactly with saddle points previously studied by Kumar and Haupt (2024); Maennel et al. (201 8); Boursier al. (2022); Chistikov et al. (2024); Boursier and Flammarion (2024), according to Section 5.2 of Kumar and Haupt (2024). To avoid ambiguity, we present our result formally:
Corollary 4.6. Following the setup and notation introduced in Section 2, let P(t) (w,(t), it) E IR' be the parameter trajectory of one-hidden-layer scalar-output ReLU network trained with the empirical squared loss L. where t 2 0 denotes time. Suppose the trajec- tory satisfies dP Ac in which we specify 0ReLU(z)
82 = 11 (x>0) in the chain rule. 8 Let A E TR be an arbitrary vector. We initialize the network with P()) == A Under these conditions, if y (inf, P((t)) - -PI) = O, where P = -(w,, Jeo)) is a non-minimum stationary point, we must have w. = O and J. = 0 for some i E I.
dr oP.
We next discuss how the gradient flow escapes from the saddle points. In the vicinity of the saddle points, the network has hidden neurons with small parameters, which can either be dead neurons or small living neurons. It easy to see that locally perturbing dead neurons does not change the loss (Fukumizu et al., 2019) and they cannot move since they have zero gradients. On the other hand, a direct consequence of Theorem 4.2 (proved in Appendix F.1 and explained in Remark F.1) states:
Fact 4.7 (A partial restatement of Theorem 4.2). A strictly loss-decreasing path from stationary point of the network must involve parameter variations of escape neurons.
As a result, saddle escape must change the parameters of small living neurons, and such parameter change of small living neurons exploits the loss-decreasing path offered by the escape neurons in the nearby saddle point (Fact 4.7). Moreover, when the training stalls near a saddle point, the small living neurons are always attracted where their parameter amplitudes will be increased (Maennel et al., 2018). Combining all these, we know that escaping from saddles always entails amplitude increase of small living neurons.
'I is worth noting that there could be saddle points with Aggi 0 but w: 0 on the entire landscape. However, Fact 4.5 states that such saddle points are not visited if we start from vanishing initialization.
'This subgradient-style differential cannot help us characterize the stationarity of non-conved functions (as discussed in Appendix C.2.2), but suffices to identify unique gradient flow without involving additional concepts. Notably, this is also how PyTorch implements the derivative of ReI.U.
8

---

Published as a conference paper at ICLR 2025
Other forms of saddle escape cannot take place. This contrasted with other different but similar tups. For example, in Fukumizu et al. (2019); Safran et al. (2020); Pesme and Flammarion (2024),' the existence of several input weights with the same direction suffices make a strict saddle, mean- ing saddle escape can take place without involving small living neurons.
In the rest of this section, will combine results with previous works to give full understand- ing of the entire aaddle-tossaddee dynamics, which is summarized in Figure 4.
The initial phase of training will have the w, 's that not associated with dead neurons to group at finitely many attracting directions when all the parameters have negligible amplitudes. Such a phase is termed the alignment phase (Maennel al., 2018; Luo al., 2021; Boursier and Flammarion, 2024; Kumar and Haupt, 2024) and can be observed in Figure 3b. The neurons whose input weights are grouped closely will be called an effective neuron in the following, as they correspond one kink position in the learned function.
After the initial alignment phase, the grouped neu- rons will have their amplitude increased from near- zero values. Notice that these neurons are small liv- ing neurons before the amplitude increase. Those that contribute to one effective neuron would have their amplitudes increased together (Maennel et al., 2018; Boursier et al., 2022; Chistikov et al., 2024), 10 which exploits the loss decreasing path offered by escape neurons in the saddle point nearby (Fact 4.7). To describe such a process, Boursier al. (2022) and Chistikov et al. (2024) rigorously characterized the gradient flow for simplified cases, revealing tra- jectories consistent with Figure 4. The training pro- cess terminates in local minimum after all the small living neurons are depleted.
Training start
(D)
(a) Aiigiment phose
Stagration orationary point
apachs
YES
(c) baddlo-escuping amplitude incrums of small Exing ncurors
statoonay
save sicapa rourent
Training oncis minimum
Figure A flow chart for the training pro- cess. We preclude other schemes of saddle escape, for example, by splitting aligned neu- rons, which is possible in Fukumizu et al. (2019); Safran et al. (2020); Pesme and Flammarion (2024). Also, note that be- sides the indispensable amplitude increase of small living neurons, saddle escape might also be accompanied by amplitude and ori- entation changes of other neurons.
Moving forward, it will be meaningful to investigate the exact dynamics between saddles in the general setup. Boursier et al. (2022); Chistikov et al. (2024) already tackled this for the training process with or- thogonal and correlated training inputs. These sim- plifications averted the simultancous norm change of different groups of neurons during saddle escape, which often takes place in general. An example is the training process after epoch 152k in Figure 3.
Notably, as the loss landscape is continuous, the insight we draw for vanishing initialization might be extrapolated to small but non-vanishing initialization. Coneretely, in the latter case, observe that the accelerations of loss decrease also roughly coincide with the amplitude increase of small living neurons. This means that the acceleration of loss decrease in this regime also exploits the loss-decreasing path given by the escape neurons in the nearby saddle points, similar to the vanishing initialization regime. We clucidate such an observation in Appendix J.
still
4.3 How NETWORK EMBEDDING RESHAPPE STATIONARY POINTS
Network embedding is the process of instantiating network using a wider network without chang- ing the network function, or at least, the output of the network evaluated at all the training input x ss In this section, we demonstrate how network embedding reshapes stationary points, which offers a perspective on the merit of over prrametrirteiztionn.. In particular, we answer the following question: Does network embedding preserve stationarity or local minimality? Such questions were partially addressed by (Fukumizu et al., 2019) for ReLU networks. With more complete characterization of stationary points and local minima in Sections 3.2 and 4.1, we are able to extend their results.
SSe Appendix P for the setting of Fukumizu et al. (2019). Safran et al. (2020) studied teacher-student setup with true loss. Pesme and Flammarion (2024) studied diagonal linear networks. The amplitude increase of one group of small living neurons creates one kink in the learned function (Figure 16).

---

Published as a conference paper at ICLR 2025
Following the naming in (Fukumizu al., 2019), there are three network embedding strategies for ReLU((likee) networks: unit replication, inactive units, and inactive propagation. We will showcase the usability of our theory for unit replication, which is the most technically involved, in the main text. The proof for unit replication and the full discussion for the other two embedding strategies are presented in Appendices M to O.
The process of unit replication is illustrated in Fig- 5: given a of parameters P, replace the hid- den neuron in E associated with Wi, and hiji. with a new set hidden neurons (1): E I.} asso- ciated with weights Wi W and has Thhito for all j E J. To preserve the network function, we specify: : 3i > O, Σ Bm = 1, for all I E L. We can
WI,
LEL
prove the following for unit replication.
Figure 5: unit replication
Proposition 4.8. our setting, unit replication performed on neuron in preserves the stationarity of stationury point if and only if at least one of the following conditions is satisfied: (1) All tangential L.
derivatives of W are (2) n 2 0, Moreover, unit replication performed local minimum if and only if at least one of the following holds: (1) All tangential derivatives of Wi. O. (2) n > 0, 41 L.
neuron in preserves the type-1 local minimality of type-1
Remark 4.9. (Fukumizu et al., 2019) was not able to discuss stationarity preservation for ReLU net- works as the non- differentiability hindered the invocation stationarity conditions for differentiable functions. This problem has been hurdled completely in the current paper. Moreover, (Fukumizu et al., 2019) was only able to discuss local minimality preservation for a highly restricted type of unit replication. Our proposition above nevertheless holds for all unit replication types in general, though the scope is limited to type-1 local minimality preservation.
Remark 10. At first sight, the statement about the preservation of type-1 local minima in the above lemma contradicts Theorem 10 of (Fukumizu et al., 2019), which suggests that embedding a local minimum should result in strict saddle. Nonetheless, the assumptions of that theorem does not apply to our setting. For more details, please refer to Appendix P.
It is noteworthy that unit replication always preserves stationarity for differentiable networks (Zhang et al., 2021), which no longer holds for the non-differentiable ReLU-like networks.
closing remark. Our discussion here extends multiple previous results regarding network em- bedding that did not consider the non-differentiable cases. Particularly, we refine the picture of the embedding principle of neural networks (Zhang et al., 2021), promote a better understanding of overparamettrrizatin (Şimşek et al., 2021; Fukumizu al., 2019; Zhang et al., 2021), and pro- vide theoretical guarantee for training schemes that involves the operations of network embedding (Wu et al., 2019; Wang et al., 2024). A more detailed discussion these topics are deferred to Appendix Q.
5 SUMMARY AND DISCUSSION
In this paper, we identify, classify, and characterize the stationary points for one-hidden-layer neural networks with ReLU-like activation functions. We also study the saddle escape process the train- ing dynamics with vanishing initialization and the effect of network embedding on stationary points. These can lead to several future directions. First, our approach of studying ODDs and aligning coor- dinate axes for Taylor expansion can be extrapolated to other architectures, including convolutional networks, residue networks, transformers, etc. Second, with a better geometric understanding of the potentially non-differentiable stationary points (such Corollary 4.4), we are at a better place to characterize the performance of gradient- based optimization algorithms on such landscapes, which neither smooth convex. Third, with the stationary points identified, of attractions of gradient flow. Lastly, the findings of this paper might help formalize the low-rank. bias formed in generic saddle-to-sades dynamics.
study the basins
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We thank Prof. Martin Jaggi from the Machine Learning and Optimization Laboratory at EPFL and Prof. Clément Hongler from the Chair of Statistical Field Theory at EPFL for their valuable suggestions and feedback. We also appreciate the insightful input from the reviewers, area chairs, and program chairs.
REPRODUCIBILITY STATEMENT
The code for all the numerical experiments can be found at https://githhb. com/ landsscpa.ddapee
REFERENCES
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly, Gradient descent quantizes relu network features, 2018.
Etienne Boursier, Loucas Pillaud-Vivien. and Nicolas Flammarion. Gradient flow dynamics of shal- low relu networks for square loss and orthogonal inputs. Advances in Neural Information Pro- cessing Systems, 35:20105-201 2022.
Dmitry Chistikov, Matthias Englert, and Ranko Lazić. Learning neuron by a shallow relu network: dynamics and implicit bias for correlated inputs. In Proceedings of the 37th International Confer- ence on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is two-edged sword. arXiv preprint arXiv:2401.10799 2024.
Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in two-homogeneous neural networks. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: /ttp///ppeeeeeeeeeeeeeeeee forum? f
Kenji Fukumizu, Shoichiro Yamaguchi, Yoh-ichi Mototake, and Mirai Tanaka. Semi-flat min- ima and saddle points by embedding neural networks to overparameterization. In H. Wal- lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curan Associates, Inc., 2019. URL https: htttp::/pprcceeiingg www.ediiiinnnnnninnieeeeee neurips. ce/ paper/ppr... files/paper/2019) file/a4ee5 68Bb
pdd.
Fengxiang He, Bohan Wang, and Dacheng Tao. Piecewise linear activations substantially shape the loss surfaces of neural networks. In International Conference on Learning Representations, 2020.
URL httph:ttp///cc...tt. /tctpp://ppenrvview..r
forum? dd=B1 6BТЕKwr.
Arsalan Sharifnassab, Saber Salehkaleyhar, and S. Jamaloddin Golestani. Bounds over-
parameterization for guarantced existence of descent paths in shallow relu networks. Inter- national Conference Learning Representations, 2020.
Bo Liu, Zhaoying Liu, Ting Zhang, and Tongtong Yuan. Non-differentiable saddle points and sub-optimal local minima exist for deep relu networks. Neural Networks, 144:75-89, 2021. ISSN 0893-6080. doi: http://wcw.......aeaaaaaaa/////a sciencedirect
URL https://www
Yossi Arjevani and Michael Field. Analytic study of families of spurious minima in two-layer relu neural networks: lale of symmetry ii, 2021.
Berfin Şimşek, Amire Bendjeddou, Wulfram Gerstner, and Johanni Brea. parameterized student networks copy or average teacher weights? arXiv:2311.01644, 2023.
Should under- arXiv preprint
11

---

Published as a conference paper at ICLR 2025
David Saad and Sara A Solla. On-line learning in soft committee machines. Physical Review E, 52 (4):4225, 1995.
Shun-ichi Amari. Natural Gradient Works Efficiently in Learning, Neural Computation, 10(2):251-
276, 02 1998. ISSN 0899-7667. doi: 10.1 10.11022089976693000007777 URL https: 1rg/1000. 11
Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-tossaddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity, 2022.
Scott Pesme and Nicolas Flammarion. Saddle-to-saddee dynamics in diagonal linear networks. Ad- vances Neural Information Processing Systems, 36, 2024.
K. Fukumizu and S. Amari. Local minima and plateaus in hierarchical structures of multi- layer perceptrons. Neural Networks, 1773)23 2000. ISSN 0893-6080. doi: https: 0ttp://000...0000000000000000000 wct.aaec///ttttaaa////////600000
URL
http://ww.ciieeciieeeeee..e. com/
Berfin Şimşek, François Ged, Arthur Jacot, Francesco Spadaro, Clement Hongler, Wulfram Gerst- ner, and Johanni Brea. Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9722-9732. PMLR, 18-24 Jul 2021. URL https: : http:///ooeeeeeeeeee//eeoo mlr. httes//v39/ssmsessee
html.
El Mehdi Achour, François Malgouyres, and Sébastien Gerchinovitz. The loss landscape of deep linear neural networks: second-order analysis, 2022.
Jason D. Lee, Ioannis Panageas, Georgios Piliouras, Max Simchowitz, Michael Jordan, and Ben- jamin Recht. First-order methods almost always avoid saddle points, 2017.
Jason D. Lee, Max Simchowitz, Michael I. Jordan, and Benjamin Recht. Gradient descent converges to minimizers, 2016.
El Mehdi Achour, Franccois Malgouyres, and Sébastien Gerchinovitz. Global minimizers, strict and non-strict saddle points, and implicit regularization for deep linear neural networks. 2021.
Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. How to escape saddle points efficiently. CoRR, 70300703,, 2017. URL http:://txxiiw.ttt/////a//
abs/
1703. 00887.
Hadi Daneshmand, Jonas Moritz Kohler, Aurélien Lucchi, and Thomas Hofmann. Escaping saddles with stochastic gradients. CoRR, abs/1803.05999 2018. URL http:://arxtt orgo / 1803.00 05999.
Liu Ziyin, Botao Li, James B. Simon, and Masahito Ueda. Sgd with constant large learning rate can converge to local maxima, 2023.
Yi Zhou and Yingbin Liang. Critical points of linear neural networks: Analytical forms and landscape properties. In International Conference on Learning Representations, 2018. URL
https: / fttts:///ppereet wpw.revieeeeeeeeeeeeeee/ee.ee.we net forum?i SerrumiiddSSSeeebheeee
Justin Sahs, Ryan Pyle, Aneel Damaraju, Josue Ortega Caro, Onur Tavaslioglu, Andy Lu, Fabio Anselmi, and Ankit B. Patel. Shallow univariate relu networks splines: Initialization, loss sur- face, hessian, and gradient flow dynamics. Frontiers in Artificial Intelligence, 5, 2022. ISSN doi:
2624-8212. articles/10 3389 9 frai ftw..2022......2... 00..0.............
URL https:/ wwwp///wwww..ooo www.fffrniierii....
Chulhee Yun, Suvrit Sra, and Ali Jadbabaie, Efficiently testing local optimality and escaping saddles for relu networks, 2019. URL httpp:://aaxxii./o///////
1809.180000 10858.
Jiajin Li, Anthony Man-Cho So, and Wing-Kin Ma. Understanding notions stationarity in non- smooth optimization, 2020. URL //appp:://///vvv.//// http:///wwxx...ooaa///////////// 1901.
12

---

Published as a conference paper at ICLR 2025
Sangmin Lee, Byeongsu Sim, and Jong Chul Ye. Magnitude and angle dynamics in training single relu neurons, 2022.
Yifei Wang, Jonathan Lacotte, and Mert Pilanci. The hidden convex optimization landscape of regularized two-layer reLU networks: an exact characterization of optimal solutions. Interna- lional Conference Learning Representations, 2022. URL https: // /tte:://pppee wpwnrreveewnneeee net/
Corum?id=270
2
Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D. Lee. Stochastic subgradient method converges on tame functions. Foundations of Compulational Mathematics, 20(1):119- 154, 2020a. ISSN 1615-3383. doi: (0.1007/s12208-01--889 019-0999--.5 URL https:://oi.o..oo 10.1007/ http//10200--0 018 09409-5.
Patrick Cheridito, Amulf Jentzen, and Florian Rossmannek. Landscape analysis for shallow neu- ral networks: Complete classification of critical points for affine target functions. Journal of Nonlinear Science, 32(5), jul 2022. doi: 10. 1..10777000322-02-09822-88
URL https:
/d/odi..om wddpi.org/0.... 10tp:/000.10000=003333303 -20-22-222000 09823-8.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemnl:: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers. Advances neural information processing systems, 32, 2019.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine- line-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pages 322-332. PMLR, 2019.
Greg Yang and J. Edward Hu. Feature learning in infinite-width neural networks. 0as/2011.145525 2020. URL (ctts://arrkee orgg /tbss/211..4522
CoRR,
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at infinite-width limit. The Journal of Machine Learning Research, 22(1):3327-3373, 2021.
Roi Livni, Shai Shalev-Shwart. and Ohad Shamir. On the computational efficiency of training neural networks. Advances in neural information processing systems, 27, 2014.
Itay Safran and Ohad Shamir. On the quality of the initial basin overspecified neural networks. In International Conference on Machine Learning, 2015.
Daniel Soudry and Yair Carmon. No bad local minima: Data independent training error guarantees for multilayer neural networks. ArXiv, hbs//605.088611 2016.
Quynh Nguyen and Matthias Hein. The loss surface of deep and wide neural networks. CoRR, hiss/704.08045, 2017. URL http:/ /ttp///aaxvvvv//. www.riiw....ooo.oooooooooooooere abs/ /1bs/1004 08045.
Daniel Soudry and Elad Hoffer. Exponentially vanishing sub-optimal local minima in multilayer neural networks, 2017.
Simon Du and Jason Lee. On the power of over-parametrization in neural networks with quadratic activation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1329-1338, PMLR, 10-15 Jul 2018. URL https://proceedin: 18C/dul88 aa..lla..... html.
.mlr. press/
Luca Venturi, Afonso S. Bandeira, and Joan Bruna. Spurious valleys in one- ame-hidden-layee neural network optimization landscapes. Journal of Machine Learning Research, 20(133):1 2019. URL http: j/jml r.org www.///pperr////ee.o...o 18- 674 html.
13

---

Published as a conference paper at ICLR 2025
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D. Lee. Theoretical insights into the optimization landscape wver-parameteriee shallow neural networks, 2022.
Itay Safran, Gilad Yehudai, and Ohad Shamir. The effects of mild over-paramecerizatton on the op- timization landscape of shallow relu neural networks. In Annual Conference Learning Theory, 2020. URL https: / /api. semant csschllrr org/ :
Yaoyu Zhang, Zhongwang Zhang, Tao Luo, and Zhiqin J Xu. Embedding principle of loss landscape of deep neural networks. Advances in Neural Information Processing Systems, 34: 11848-448854 2021.
Bo Liu. Understanding the loss landscape of one-hidden-layer relu networks. Knowledge- Based Systems, 220:106923, 2021. ISSN 0950-7051. doi: http://do..org/100100661....ooo 2021.106923. URL https://///... https://wwwwww.e tttp://cceneeeeeeeeeee..cco com http://///cccceeeeaaaelll//////e 50950705121001866
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. International Conference on Machine Learning, pages 557-565. PMLR, 2017.
Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason Lee. Stochastic subgradient method converges tame functions. Foundations of Computational Mathematics, 20(1):119- 154, 2020b. ISSN 1615-3383, doi: 10.10077/10200-005 018-09909-,, URL http::://doo:ot 10. 10.1007/
094009-5.
Frank H. Clarke. Generalized gradients and applications. Transactions of the American Math- ematical Society, 205:247-262, 1975. URL https: http::// C
CorpusID: 1201144228.
Kenji Kawaguchi. Deep learning without poor local minima. D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Informa- tion Processing Systems, volume 29. Curran Associates, Inc., 2016. URL
https:/pproceeings. neurips........................m htt.ee/eeeeeeeeeeeeeee.ooeeeeeee 2at.224444...2122220000000-00010
f2fc990265c712ccm 1dw.aaa22222111a55555535333 2etp
pdf..
Lemeng Wu, Dilin Wang, and Qiang Liu. Splitting steepest descent for growing neural ar- chitectures. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https:/ /
neurips. wewuriss.ccpppprrff.i... Aat.daaaag..iaaaaoooooaaaaaaaaaa
Yite Wang. Jiahao Su, Hanlin Lu, Cong Xie, Tianyi Liu, Jianbo Yuan, Haibin Lin, Ruoyu Sun, and Hongxia Yang. LEMON: Lossless model expansion. In The Twelfth International Confer- ence Learning Representations, 2024. URL https:/ /ttp////ep//eeeee//////eeee net meetfforumidd 3VW7DQqq7U.
14

---

Published as a conference paper at ICLR 2025
Appendix
TABLE OF CONTENTS
Proof of Lemma 3.4
16
A.1 Derivation of Equation (4)
16
A.2 Derivation Equation (5)
16
A.3 Derivation of Equation (6)
17
B Extending Lemma Lemma 3.4 to Networks with Multiple Hidden Layers
17
B.1 Output Weight Derivatives
17
B.2 Hidden Neuron Weight Derivatives
18
Discussions on Stationarity Notions
19
C.I Directional Stationarity Captures the Stagnating Behavior of GD
19
C,2 Examples Non-diffceentiable Stationary Points
20
C.3 Connection between Directional Stationarity and Clarke Stationarity (Wang et al., 2022; Davis et al., 2020b)
C,4 Discussion on the Stationarity Proposed by Cheridito et al. (2022)
22
D Rarity of Local Maxima
23
Intuition of Definition 4.1 and Theorem 4.2
23
Proof of Theorem 4.2
23
F.1 Proof of the Sufficiency
24
F.2 Proof of the Necessity in Theorem 4.2
28
G Taylor Expansion with an Aligned Coordinate System
30
H Verifying Local Minimality in a Real Case
31
Additional Numerical Experiments
32
L1 3-dimensional Input, Scalar Output
32
................................
I.2 2-dimensional Input and Output
32
Applicability of Figure 4 to Small But Non-vanishing Initialization
35
K How Larger Initialization Scale Alleviates the Low-rank Bias
36
L The Evolution of the Learned Function
36
M Proof of Proposition 4.8
37
M.1 Conditions for Unit Replication Preserve Stationarity
37
15

---

Published as a conference paper at ICLR 2025
M.2 Conditions for Unit Replication to Preserve Type- Local Minimality
38
M.3 Putting Things Together.
40
N Results Regarding Inactive Units
40
N.1 Results Regarding Orthogonal Units
40
N.2 Results Regarding Negative Units
41
0 Results Regarding Inactive Propagation
41
Comparison of Theorem 10 of Fukumizu al. (2019) and Proposition 4.8
42
The Applications of the Insight For Network Embedding
42
A PROOF OF LEMMA 3.4
A.1 DERIVATION OF EQUATION (4)
We compute
Oūny (P)
0
Σ
hypplwy xx) 1 (W: xx).
Ohys
This
PEl
Thus, we have
0 Öhys
C(P) ahys 2 JEE
2 Σ Σ (0ky )))
=
ekj' Th.s
Σ
CkiplWw xx)) w. dja-
J'EJ KEK
KEK
A.2 DERIVATION OF EQUATION (5)
For all x, E IR, define
if > 0, p(z) if 0,
a4
py (x)
a=x
if < 0.
The map (-) will be convenient to compute the following derivatives.
Let be such that ||wall + O, let u; || Recall that the derivatives Dr. De; below Definition 3.1. Below, we write w, = r;u, and take the derivative at r; = ||w:ll Note that p(r;w; Xx) = rip(w, xx), that is, along the direction u, the activation is linear. We thus have that
2 8 are defined
0g, (P)
Op(r.u: xx)
h1g
happ(u: xx)..
3r:
Ør:
We then get
OC(P)
Oūjj
ΣΣ
Eki Dr:
ΣΣ
exphiipluu xx)
0r;
JEJ JEJKEK KEK
JEJ KEK
Σ Σ
ekihjill Paix,,
JEJ KEK
Σ
at atexpxt +
Σ a ekyx
JEJ
W.
We
16

---

Published as a conference paper at ICLR 2025
Dyau dj
JEJ
where dys was defined in Lemma 3.4.
A.3 DERIVATION OF EQUATION (6)
Recall that when taking the derivative a Øs." specific direction v, is chosen. One can check that Öp(w W; X&)
1 (w. /0) Pwexx (Vs Xx) + 1 (w: 0) xx)
Os;
Pw. (V: Xxb
This gives us
Pink (P)
Op(w; Xx)
hīi
hjiP (v. xxk)
0s;
08;
We thus have that
OL(P)
Oūkr(()
Ckj
Σ
ckjP (V: Xx))
Os:
Os:
JEJ KEK
JEJ KEK
ΣΣ E
exjVs' (1 #0) (Xx) + (x (xx))
JEJ kEK
Σ
atexyxx+ +
Σ
a Ekixi
(w.+ABlV))
hyyv dy
JEJ where As; > 0 sufficiently small and where d was defined in Lemma 3.4.
B EXTENDING LEMMA LEMMA 3.4 TO NETWORKS WITH MULTIPLE HIDDEN LAYERS
In this section, we demonstrate how we can compute the radial and tangential derivatives with re- spect to weights that are ReLU-like networks with multiple hidden layers.
The notation of this section inherits from the one-hidden-layee case for the most part. Nonetheless, we augment the notation for the hidden layer weights with an index to number the layers.
The training inputs are X 's and we also denote them by x (0) E R vol (for the sake of notation simplicity). The training targets are ys RI The input weight matrix in hidden layer / e {1. L} is w()) E IR 4 x|x|11 I is the set of hidden neurons in layer I. The hidden neuron weights corresponding one row of ( is w!!), where E It The output weight matrix is H [R|)/ each row of which is hy with j E J.
The neural network is defined recursively with the following:
( T
( (1-1)
X (1)
p(())) vLE (1,, ,L, ýx 1xx,
B. OUTPUT WEIGHT DERIVATIVES
We first compute the network outputs' derivatives with respect to the output weights.
Ouky' Oh,
= 1, 1
(L)
Then we have:
aL
Oyki'
Σ Σ
Σ exix!!
Ckj' Th,
Öh
KEE
beK
17

---

Published as a conference paper at ICLR 2025
B.2 HIDDEN NEURON WEIGHT DERIVATIVES
For a specific layer 10 2,, L.), and specific direction of Aw!") Ast ((()) ("),, where u (1") and v((") are the radial direction and tangential direction of (?)) and vv!")
( Art ( "n! (P)
? 40, we compute the radial derivative and tangential derivative.
RADIAL DERIVATIVE
(1)) dz
Dūkj 0r!"
Jūg
(C) dx (10)
Dūki
plu
x
(12 -11)
3z (10
àr (10)
Thus:
0C
dūki
ΣΣ
eki rr? (10)
011100)
(10)
jEJ kEK
Jyki Eki
(1")
[ ((-1)).
uu
X
Dz ko
JEJ REK
which is reminiscent of the one- mae-hidden-laer case.
TANGENTIAL DERIVATIVE
Again, we start by computing the output's derivative.
Dūki
dr ki
Dūn
2) (v.
x 1111
0s
(19)
Ox ki
0s (10)
dr
ki
(10)
w
(10
x
110
Thus:
0C
Tūkj
ΣΣ Σ
0s! (10)
Cki 0s
(1))
jEJ KEK
Σee
Eki
Jyke (10)
00 (v.
x
(0-1))
dIki
iEJ KEK
Σ
Σ
a
Ckj ckj
Ooki
k
(1-11)
JEJ
de
(w!
8yns Ekj
+ Σ
Σ
ir ks
(10)
JEJ
A:
(w("), (w("))
where As; is arbitrarily small. Again, this reminiscent of the one-hidden-layer case.
Oig COMPUTING
802, Bx :o; 0z
shows up in the formula or radial directional derivative and tangential directional derivative.
In this part, compute the vector containing this value, aüg This derivative can be derived from !x!
the routine of back propparopa,,toon adapted to accommodate the condiffferettiability of the activation function.
We first compute:
05k HT 0x (L)
18

---

Published as a conference paper at ICLR 2025
Then, for E {10, + +1, ,L 1}. we have:
Onk (+1)
0xp (111 0ÿn
0ÿk
0x 0x ami (1+1)
0x
(1+1)
= (w:")) diag(a (1)) 0ÿn
0x (1+1)*
(1)
where the vector of a may be componentwise defined as:
(1) (Aw, (1)) 0
if Tiks + Ankki
ki
0
if Thk + Alli (Aw, (()) 0
(!)
if Thk + Amha (Aw;
(9)
Here, Ange !1)
Aw(()) is the change of Ths when an arbitrarily small perturbation of Aw!" (1)) is ap-
(1)
When computing radial derivative, such perturbation can be taken as Aw, (1")
plied w
When computing the tangential derivative, such perturbation be taken Aw! (1))
C DISCUSSIONS ON STATIONARITY NOTIONS
C.I DIRECTIONAL STATIONARITY CAPTURES THE STAGNATING BEHAVIOR OF GD
In this section, we show that the absence of first-order negative slopes at one point on a scalar function prevents GD from stagnating with high probability.
Proposition C.1. Consider a random function f(z) = - anT" ( (x<0) + E Fn (x20). where x E R, N e N. and the random coefficients {a, 33 & {a1, S1, 31, ,,N] are drawn independently from a distribution that is absolutely continuous with respect to Lebesgue measure. Denote left-hand and right-hand derivatives by f-).) and f+('), We study the GD process x++1 = 0. Then,
It ( where 1) 0 is the step size. Suppose that f4( 0 or f0) with probability 1, the following holds. There exists an interval containing the origin x(a, (3) & [u(ox, B), b( 3)), where al u(or, (3) < 0 Dnn b(oc, B) > O, and the time for escaping from this interval can be upper bounded by oo > E(n, a. (3 > O, which means if I₁ E then there exist 0 < t' < such that Zett' $ X-
Remark C.2. The above proposition studies random function to simulate that the loss landscape of neuron networks is usually also (a realization) of a random function, with randomness coming from the dataset. The above proposition shows that if the left-hand right-hand derivative the origin is negative, then the escape time from the origin f only concerns the realization of the function (determined by ax (3) and the learning rate 7), without involving how close GD gets the origin. By contrast, saddle points, no matter whether they are differentiable or not, trap GD or gradient flow for a longer time the trajectory of GD or gradient flow reaches closer to them (Maennel et al., 2018; Boursier et al., 2022; Chistikov et al., 2024). More concretely, if the origin x = 0 is saddle point, then the upper bound for the escape time should be ε(1) a. ,, d). where d infy |æerr| is the shortest distance from the trajectory following I; to the origin O, and it must be part of the upper bound For example, suppose æ = 0 is a conventional smooth saddle point, then, if d y O, we have f / oo; and if = 0, we have = oo (GD is permanently stuck in this case).
Proof. We prove the proposition different situtioon.
Situation 1: (0), (0) < O.
Since (0) 0. based the right continuity of the derivative, we have that there exists
by (ax, 3) > 0 such that f (x) < 0, if E b). Define mf supze(o.b) (1) 0. Then,
19

---

Published as a conference paper at ICLR 2025
have, if 21 E [0,bı] and ti satisfies that 2444 -1 [0. b1], zet4 22 2 -nmith > 0. As result, there exists time step that GD would step beyond 61. Namely, there exists step count tj such that Ft-t; > bi, and ii must be upper-bounded by: 1/ < [a.9) + == li.
Similarly, since f' (0) O. we can have an interval (a, 9), 0) with an a1 B) < such that f" (a) < 0, if E [a1, 0J. Define m = suP=E(e1.0) f=-(x) < O. We have, 2 E [a1 0) and of satisfies that #f
(a1,0), Fitey It S 1pm 14 < O, where we applied ['
- ʃ4 (x),
if re(01,,), 0), to the GD update rule. As a result, there exists time step that GD would step beyond a1. Namely, there exists a step count tf such that setef < a1, and o must be upper-bounded by:
of S [ = P.
Up till now, we have proved that, for situation there exists an interval x(a, by (a, [3] such that, if E X. there exits & (a, B,n) == max(t', such that Z++-' Ja (ax, b)
for some 0- < L' < [ (ax, 1). Notice that this case is Clarke stationary point (Wang et al., 2022; Yun et al., 2019). Nonetheless, we prove here that such a point does not slow down GD.
Situation 2: (0) 0, (0) 00
Since f0) < O, similar to situation 1, we can have the following. There exists an interval [0, (2, [3] where (y B) > 0 such that, if 2, E [0, 62 3)]. then there exists a step count i/2 such that Tite, > b2, and 1/2 must be upper-bounded by: 122 S (a
+ == l22 where my
mm
defined with m = b0 (z) < O.
Moreover, since (0) > 0, we can have an interval [a2(a,, 3). with a2(ox, 33) > and f" (1) < O, if x E [a2, 0]. Define my = inf f- (x) > 0. We have, xz E [a2, 0) and ty satisfies that T1+ [a2, 0). Z - 24 2 my t22 > 0, where we applied fz) = f", (r),
if x E [0z, 0), to the GD update rule. As result, there exists time step that GD would step beyond Namely, there exists a step count t" such that T2tez 0, and to must be upper-bounded by:
[2(a. = P"
Up till now,
have proved that, for situation 2, there exists interval x(a,,3 [a ax, B), ba [3]] such that if 2, X, there exits (2 B, = f + lf such that F+te $ X for some 0 < 1' < (1,,B,1)
Situation (0) > 0, / (0) < 0. We can prove the proposition holds for this situation similar to what we have done for situation 2.
Other situations account for measure and do not concern the validity of the proposition.
L
Remark C.3. Note that there could indeed be cases with probability in which directional stationar- ity fails to capture the GD-stalling behavior of point. For example, if f(:) x²1 (m:0) x: (x>0))> and we start GD from xg with a learning rate 7) < 3. then the GD will be stuck at æ Never- theless, this point is not a directional stationary point. There might exist analogous situations for our setup where directional stationarity fails capture GD-stalling points. However, Such situations have probability 0.
C.2 EXAMPLES OF NON-DIFFERENTIABLE STATIONARY POINTS
Here, we give two examples of non-differentiable (directional) stationary points under GD.
C.2.1 A NON-DIFFERENTIABLE LOCAL MINIMUM
In Figure 6, we show a non-diffeee -in-differentiable local minimum of the function f(x, y) = |a| + ly]. which is the origin and denoted by the red x. It is not hard to see that GD performed the function will approach this point and stop there (or bounce in its vicinity, which can be taken as "GD effectively comes to halt", as phrased in Section 3.2).
20

---

Published as a conference paper at ICLR 2025
10
(Kx))
-
-2
2
x
Figure 6: A non-differentiable local minimum
C.2.2 A NON-DIFFEEETTABLLE SADDLE POINT
In Figure 7a, we illustrate a non-differentiable saddle point and GD's behavior near The function we investigate in this example is the following:
if and > y x
yy
+
if > and S 0
(xx
y²
if S 0 and 0
y2² if S and y
which is showed in Figure 7a. The origin is non non-differentiable saddle point. If we perform GD starting from (z, y) (1e 1), the trajectory GD will be as shown in Figure 7b, and the change of function value during the GD process will be as shown in Figure 7c. The behavior of GD is much like that near a differentiable saddle. However, one can show that, in this case, given step size smaller than 1. the non-differentiable saddle point is non-escapable by GD, even though it has second-order decreasing directions. This in contrast with strict differentiable saddles, which are almost always escapable by GD (Lee et al., 2017; 2016; Jin et al., 2017; Daneshmand al., 2018). Remark C.4. Note, that the subgradient cannot be defined at the saddle point discussed here. We remind the reader that the subgradient is defined as:
Definition C.5. Let f _ R convex function. The subgradient set of f at x is defined as:
0f(x)) {g e RR" f(x) 2 f(xo) + -g"(x xo). Vx E R")
As matter of fact, the definition of subgradient generally fails at saddle points due non-convexity. One can verify that the subgradient cannot be defined for the origin of the loss landscape of our setup in general. Since the ODDs toward all the directions are zero at the origin (by Lemma 3.4), the subgradient g can be defined, it must mean that g O. Taking it to Definition C.5, we find that, if the subgradient at the origin were g = 0, the origin would be a local minimum. However, the origin normally have loss-decreasing path around it.
C.3 CONNECTION BETWEEN DIRECTIONAL STATIONARITY AND CLARKE STATIONARITY (WANG ET AL., 2022; DAVIS ET AL., 2020B)
We show that directional stationary points must also be Clarke stationary points. This insight was also given by et al. (2020). We prove here for convenience, after which we show that directional stationary points are Clarke stationary points with no negative ODD around it.
Proposition C.6. Given a function f R.N R whose ODD
f(x + hv) - f(x)
fxx,v) = lim
0+
h
21

---

Published as a conference paper at ICLR 2025
100
03
0.0
1.0
-1.
0.5
of ao 04 -10
0.0
(b) A GD trajectory
-0.5
0.8
10
0.5
0.0
0.4
-0.4 -0.2
0.0
0.2 0.4
-1.0
0.5
y
0.2
8.0
x
Esocho
(a) The saddle point
(c) Function value during GD
Figure 7: A non-differentiable saddle point
can be defined for all x E R for all v E IR. If satisfies that [xx, v) 2 O, for any E IR,. so that it is directional stationary point, then it must also be a Clarke stationary point (Davis al., 2020b; Wang et al., 2022).
Proof. Clarke stationary points are defined to be where
0 O() = (E E (E.v) ((xV)
(8)
in which (x, v) is the Clarke generalized directional derivative:
fly + hv) fly)
(xxvv) lim sup y- S
limsup
h
Since Equation (9) is equivalent to taking a supremum over all the ODDs computed for neighboring point y, must have
Vv
(10)
in which the second inequality comes from the condition in Proposition C.6. Notice, Equation (10) immediately make Equation (8) hold, which proves the proposition.
Fact C.7. Directional stationary points are Clarke stationary points that do not have negative ODDs
toward any direction around it.
Proof. Following the setup in Proposition C.6. Define O = (xlff v) 2 0, R =
{x|0 E ld (x, v)), L = (xl v) < 0, for some v E RR.), To prove the above fact, we need to prove C L.. By definition O T.. Thus, C T. CNT O. Also, we have that (which proved in Proposition C.6) and I, thus we have C 0 L. As result O C L.
C.4 DISCUSSION
STATIONARITY PROPOSED CHERIDITO
(2022)
Cheridito et al. (2022) defined stationary points to be where the right-hand derivatives along the canonical axes are zero. Please refer to their Equation (2.3) and Definition 2.1 for details. Such a stationarity notion might not suit purpose the ODDs canonical axes might not be informa- tive enough to characterize the local structure function in all directions. We will demonstrate this with toy example.
22

---

Published as a conference paper at ICLR 2025
Consider the origin of the function fz, y, 1r- |x- It has zero right-hand derivative on all the coordinate axes, and thus qualifies as stationary point by the criterion proposed by Cheridito et al. (2022). However, there exist negative slopes around it, for example, in the direction of (1, 0). For such function, the stationarity can be verified by:
(1) checking whether the ODDs toward all directions in IR are non-negative, which is analogous to Definition 3.5; or
(2) checking whether the directional derivative along the axis, where the function is continuously differentiable, zero; then checking whether the ODDs in the directions on the (æs, y) -plane are nonnegative; which is analogous to Definition 3.6.
D RARITY OF LOCAL MAXIMA
In theory, local maxima can exist in our setting Zhou and Liang (2018). Nonetheless, the theorem below suggests that they are extremely rare.
Theorem D.1. If there exists some input weight vector w; in P E RR. satisfying p(w: X + 0 for some inpul Xx, then, cannot a local maximum.
Proof. We need to show that when the parameters P satisfies the sufficient condition in Theo- rem D.I, there exist perturbations leading to strict loss increase. Let us focus on one of the hidden neurons (with the subscript of that has p (W; X&) j for some k K. Let also specify an output neuron indexed with j. We will construct a strictly loss-increasing path only by modifying hye-
Here we could simply use the derivatives that we computed before and show it's positive:
aC(P)
Σ
eajP(W; xx)),
Ohyz
kEK
8 LC) Ep(w: x².
0h3
REK
Hence, we see that either the first derivative is non-zero, in which case there is a loss-increasing direction since the loss is continuously differentiable in hjs, the first derivative is null and the second derivative is strictly positive, in which case we also get a loss-increasing path.
Remark D.2. Liu (2021) proved the absence of differentiable local maxima for || = 1, and Botev al. (2017) precluded the existence of differentiable strict local maxima for networks with piece- wise linear activation functions. Here, we provide more generic conclusion, that if P is local maximum, no matter whether differentiable or not, then all input weight vectors are not "activating" any of the inputs in the dataset, meaning &s (P) = 0 for all k E K.
E INTUITION OF DEFINITION 4. AND THEOREM 4.2
When the output dimension is one, we can understand why the escape neurons, as defined, can lead loss-derressigg path. Let denote the only output neuron with the subscript of jo- For the escape neuron, we have that for some tangential direction fixed, ac(P)
ds. hj v, = 0 and dV: V: + O. This must mean that Toj = 0. we slightly perturb F such that it has a different sign than d V., then the tangential derivative ac(P) after the perturbation is negative, indicating a loss- dess-decreaiing direction.
F PROOF OF THEOREM 4.2
For brevity, we use (at)et. to denote a vector that is composed by stacking together the aj 's with
I E L..
23

---

Published as a conference paper at ICLR 2025
F.1 PROOF OF THE SUFFICIENCY
We start by introducing the ODDs of the loss. Fix a unit vector A E R in the parameter space, along which we investigate the derivative. By doing so, also fix the vectors u, v, for all E I, since the decomposition Aw: = A u4 + A., v, is unique, i.e. there exists a unique unit tangential Wi vector v, orthogonal with u; will and unique Tss S; 2 0. By convention, if w; we set u; = and ri O. See Figure for a visualization u;, Vi.
The ODD of the loss in direction A is then defined by
C(P + + -
daL(P) == lim c-00+
(11)
Note that the limit always exists, since, as we saw in previous sections, the partial derivatives C(P) C(P) da; dr; are always well defined. However, we stress that because the non-differentiability. we can have that & (P) * -PP)
Consider a stationary point P of L with no escape neuron. Also, consider a direction A such that C(P) = Directions with 0₂ C(P) > (which is possible due to the positive tangential slope at imput weights) are guaranteed increase the loss locally. Our strategy is to show that the ODDs of higher orders are non-negative. For example, at order 2, this means that
C&C e - OAC(P) 0.
CLP(
lim
(12)
As a side note, the loss of ReLU-like networks can be understood as numerous patches of linear network loss pieced together. Thus, moving along fixed direction A from given point locally exploits the loss of a linear network, which a polynomial with respect to all the parameters. In other words, the loss landscape is piecewise differentiable (C*). As result, ODDs of any orders along (whose computations are similar to Equations (11) and (12)) are always definable.
Then, we need investigate the directions A such that 03 ((P) 0, since the loss could increase, decrease remain constant in those directions with higher orders. We will see that at order 3, these directions also yield 0% C(P) O, and at order 4, necessarily, 0 CPP) 2 0, and since all higher order derivatives are null" this yields the claim.
These will be achieved by investigating the Taylor expansion of the loss in the direction A, which reads for small e >> 0 as
- C(P + cA) = C(P) -
c0s
2! 20 C(P) + CP + 4
CPP).
3!
F.1.1 SECOND-ORDER TERMS
We organize the Hessian matrix at P correspondingly as:
Tn] columns
T,| columns
[T, columns
CP)
(CP)
(CP(P
[ rows
Thūyú, Ship
Or, Ohizis
0s; Ohisiz
CC)
PC(P)
LL))
Hc =
T rows
(13)
Ohin Orig
0r,, Ori
0sc, Dri
8 (LP)
LLP(
8 LLP)
[I]] rows
0hiji 0sig
Or,, 0s:2
Osi Osi,
Computing the second order partial derivatives. Recall the first order partial derivatives
dC(P)
Σ
exjp(w,
0hg
REK
All ODDs of the fourth order are constants, as shown in Appendix F.1.3.
24

---

Published as a conference paper at ICLR 2025
A
Ar, u,
A
w;
W;
[w,]] >
||z]]=
Figure 8: Breakdown of perturbation applied on input weights. Left: Perturbation on w; with |w:ll > can be decomposed into the radial direction u, and the tangential direction v.. Right:
Perturbation applied on a zero input weight can only be decomposed into one direction v;.
OC(P)
Σ
hy
exjplus X4),
JEJ
KER
OC(P)
Σ
Σ
ekjPwi (v: xx).
0s;
JEJ
KEK
We now compute the second-order derivatives of the loss. Recall that these derivatives are defined for fixed radial and tangential directions, that is, the vectors u, Va, E are fixed. We have
8 LCP)
Dekii p(w; . xx))
Σ
kEK
=
Σ
(i P(W,, xx)p(WE, xk).
kEK
LPP)
Σ enphyy', xx))
Ōri, Shjiz
Ohjiz
YEEREK REK
(eksp(un Xxk xx) kEK
Σ
hump(u, xx)e(wis xx))
8 L(P)
hij Σ ekjpwr, xx. (Vis xs)
0si Ohjiz
alijia
KEK
Σ
eki ekjPw., xx (Vs, X4: (ig= + hyg, E
pur,y xe (Vi, xx)p(w,, xx),
=
KEK
REK
@LL(P)
a
enghyiipuu xx.)
Ōr, Orig
Or2
jEJ kek
Σ
Σ
hyg hissp(u xx)p(uss xx).
jER
LLPP)
0 Or
exghyi Pwr, xx (Vi, xx)
0s, Ör
KEK
Σ
ΣDN
hyg Piss xe (Vi, xx)p(uig xx).
JEJ KEK
CCP)
a
eeghiin xs (Vi,
Xx.
ds; Ush
asta
JEJ KEK
25

---

Published as a conference paper at ICLR 2025
EΣΣ JEJ KEK
hij Pisa x (Vis Xx) xn (Vig Xx)
Second order partial derivatives at the stationary point. When evaluated at the stationary point P. two of these derivatives simplify. Firstly, we note that, if ||w,ll & O, then ΣKEK (E
ac = 0 by stationarity. If ||w]ll = 0, |ull = 0 by convention so the sum is xx) (f1=ia) Tw Thgs
null well. In any case, we have that
CCP 0+
E KER
P(u, xx)p(W(, xx).
0r, Ohjiz
Similarly, since choose the direction A such that OgL(P) = O. must have that 8C(P)
0s; 0.
Since there no escape neuron, necessarily, EBEK ekjPw: (V: (Vi- di; Vi = 0. This implies that
8 CPP) 0 + hash Üs, Dhjiz
Σ KEK
Pw., xee (Vi, Xx)p(Ws, xx).
(14)
The Hessian matrix is positive semidefinite. for E I,j E J, let
Define the following three [] dimeesionnal vectors:
Vay == (p(w: xx)) xx))k=I,....II]
(15)
V = (hysp(u; x=)))=
(16)
V 8: = (hysPw, (v: x==)))==
(17)
Then, we assemble the vectors and for j' E J. we define the matrix
Vj = (11)=19) (wnen, (V% SEL.
E <
SEL,
where 1 ()=زا) is multiplied to each component of V hyu- The vectors are disposed such that each is a column of VJ" We thus see that the directional Hessian Equation (13) can be written as
H = EV JEEJ
(18)
Once again, like the vectors u, V, for i E I, the Hessian matrix implicitly depends on the direc- tion A. Hence, for every fixed unit vector A E IR,, the Hessian matrix H.g is sum of positive semidefinite Gram matrices, and as such, is positive semidefinite.
We deduce that for all unit vector A E IR,, it holds that
(LCP))ATHHAN
as claimed.
F.1 THIRD-ORDER TERMS
In this section, we continue our reasoning and assume that P is stationary point with no escape
neuron and A E IR is fixed unitary vector such that 0 C(P) = TTH = 0. In particular, since Hr Σ jo. VJV, in this case, it holds that V,A E R K is zero vector for all j E J, where
Vj is defined below Equation (15). We show below that this entails C(P) = 0.
From the formulae for the second-order terms, we can see that there can only be six types of non- 82
8 C(P) zero third-order derivatives, namely, Br, Ohing 8h Dr, dri ahyg dr, Usig" 0hge Ua; Ohgng
(CP)
8 L
8 P(P) and Thyt, dss, Osiz" Thzt, ds,, ăr:z"
8 CPC(P)
Recall the vectors V Vi Vi defined in Equation (15), and defined the following vectors
V hr; = (p(u, xx) (e)
26

---

Published as a conference paper at ICLR 2025
Vhs: (pw. (V: xx))
The third order partial derivatives can be conveniently expressed using them. For example, compute the first one
8 P(P)
a
Σ
(espp(us, xx:) + hys p(us, xxx)p(s, ws, xx))
Dhye, dr,, Ohjiz
ahizz KEK
(1 + 1 (1==12))
E
plus; xx)p(wis x1)
keK
(1 1 (11=1)) Var. Vhyyz
The other derivatives follow similar easy calculations that are left to the reader, yielding the follow- ing results:
8 LLP(
= (1 ((1=iz) Vana
This, Dr:, dris
8 ((P))
(1+ 1 (i) (1g= Vini V 8i2
Ōhyi, dr,, 8skz 0r
L(P)
(1+ 1 [(n=is))
Vhsi Vaaa
Ohyt. 0s Ohyjz
0 C(P)
(1 L ((i==ia) Vher
V rig
Ōhji, Osi, Oriz
8 ((P(P
(1 + 1 ((=is))
Vhai
Öhyi, 0s;, 08ig
612"
In the third-order ODD of the loss at P in direction A, each of the above derivatives appear. More specifically, # i2, then any of them counted 3! 6 times (the number of orderings that yield the same derivative) and if i i2, then it is counted 3 times. We thus see that
C(P) = Σ
Σ i1AiZE
Anya Dr., Virii (Anjiy Vhya + Ariz Vri, + Dan, Vorz)
jEJ
+2x3
Alya Ar Vhri (Ahye Vayin + Dri, Vii, + Asn, Van))
eel
6
Absic Asi, Vhsig (Ansi Vhsiz + Drig Vrog + +sig Vsig)
JEJ
$1 i1siREr
+ +2x3
Anya Asy, Visy (Anae VnJa, + Dry Vi, + Asy Vou.)
= 6
3
Aag (Ar, Vir,, + +Aa, Visa, V4
JEJ i1.g
Recall from the discussion the start of the subsection that since P is stationary point with escape neuron and since 0 C(P) 0, it holds that
VAA = 0.
(19)
This shows that (P) = 0, claimed.
F.1.3 FOURTH-ORDER TERMS
This is the last step of our argument, where we show that C(P) 2 0 always holds.
We compute the fourth-order partial derivatives of the loss. Note from the third order partial deriva-
TC((P)
2 (LC(P)
tives that only three of them can be non null, namely Bhey, Ory Gh Oreg" Thyn, 0e,, 0hg
and
27

---

Published as a conference paper at ICLR 2025
(*c(P) 0hzi, dr:, 5h Usi2 following:
straightforward but cumbersome therefore left to the reader to check the
LLP
(1 (in=iz)) Vhhia
Jhii dri, 2hi dri2
0+ ((P
(1 (1=it)) Vas
Vhheg?
Th jo ds,, Jhiji 88
0' ('P(P)
(1 ((=
Jh dr; J 08i;
As for the third order, depending on whether i equals ig, the number of occurrences of these deriva- tives changes between 4! and 4//2, and the fourth-order ODD reads
4!
C(P)= = Σ
=
Σ (iFiEEl
4.An,,, Ar., Ansiz Dr.g Virig
Vhria
Σ
2 A?
A2
Vhri Varaa
2
h1j1
jEJ
+ +Σ
4. Asy Annz Asyg
Vasy
V his
+
Σ
2
Ar Vhsiz
V hsu
2
2
JEJ
EE1
+
+Σ
Σ
4!Ang,n
Dry Angiz Asgg Viri
hsig
+
Σ
2
2
A
h31
Drs Doi Vari
V Vasg
jEJ
61F1ZEl
Σ
Σ
Abya Ar, Angen Ari, Vara Vhre + Aaps Ast, Anyzz
A 5cz
Visi
Vissa
2
JEJ
111tEE
2An, Dr., Angi Daip V Varig -Vhiz
4!
Σ
Σ
(Any Drg Vtri, + Abya Asi, Vhsn ).))))((nn (Ahsi Ari, V Viria
Ahiiz Asig V Vhsig)
JEJ
V
JEJ
where we just introduced the vector V,, defined by
A Ar. Vnn. Dn.. A., Vnn,) jEJ
VE!
We see from the above that C(P(P) 2 as claimed, which concludes the proof the sufficiency in Theorem 4.2.
Remark F.1 (about Fact 4.7). Note, stationary point, if perturbation direction A does not perturb the parameters of escape neurons, then the above proof also effectively shows that such a perturbation can not strictly decrease the loss, which gives rise to Fact 4.7.
F.2 PROOF OF THE NECESSITY IN THEOREM 4.2
The necessity of the condition in Theorem 4.2, which holds in the scalar-output case, will be proved in the following via contradiction. Namely, P is a stationary point on the loss landscape with at least hidden neuron being escape neuron, then this stationary point cannot be local minimum. Let select such hidden neuron i, associated with (h and w.). To construct the loss-decreasing path, perturb Tejos and w, with Ahj and Aw; As; respectively; v, being one of the tangential directions satisfying and aC(P) 0 and dy; * 0. We assert that we design the following perturbation strictly decrease the loss: Ahy. = sgn(d); vz)a, As; = b. with a, b sufficiently small and satisfying certain conditions, which is discussed below.
Be;
28

---

Published as a conference paper at ICLR 2025
To study the loss change after perturbing the weights described. resort to Taylor expansion
again. In this case, we only need to look into terms of no higher than the second order.
For the first-order terms, we have:
Ti Ti AP') = PC(P)
OC(P) As,. Os;
Anjus +
(20)
Ohioi
us. Since P is stationary point, must have (C(P) O. Moreover, we have (C(P) 0, this is
Thyzt
how we choose the tangential perturbation direction vi. Thus we have Ti AP") 0.
Then we calculate the second-order terms. Three types of second-order derivatives are involved, T Notice that, in Equation (14), we could simplify the last derivative given that the neuron is not escape neuron, which we can no longer do here. This causes the Hessian matrix to indefinite.
8 PP) (c(P) and 6 (P)
Notice that, we must also have hi 0 to fulfill the requirements of escape neuron in the scalar output case. Hence, we have V. 0. Thus, the second-orde terms can be derived as:
(P, AP") (() Shin
CP) as?
+2 02 LLP) Ahjor
(A
(Ax;)
(21a)
0s;
Vigg
V Vago (Ahjor)²
+ V.le V (As;)² + 2(d%
joi vi+ +
V
Ah
(=0)
(=0)
(21b)
|V 112 + 2 v.) (- sgu(d) v.)) ab
(21c)
[V 1|2 2/d%;
Joi Valab
(21d)
Note, we have specified that both and are positive. Thus, if we choose these two numbers such that:
> IV Hags
vil a.
(22)
2 Jot
Then we have T2(P, AP') being strictly negative, indicating strictly loss-decreasing path, which completes the proof.
Remark F.2 (about Corollary 4.4). When we perturb the parameters of the escape neurons as above, we are exploiting second-order loss-decreasing paths, which yields the statement of Corollary 4.4.
F.2.1 WHY CANNOT WE PROVE THE NECESSITY FOR NETWORKS WITH MULTIPLE OUTPUT NEURONS?
The construction of the loss-decreasing path in this section is based on the observation that the Hessian matrix is longer positive semi semi-definite matrix when there exist escape neurons for |J| = case. However, in the case where |J] > 1, we cannot draw similar conclusion. When the network has multiple output neurons, there is currently guarantee that escape neurons will introduce negative eigenvalues in the Hessian matrix. More specifically, like the |J] = case, some ('c(P)
Osgōhgs term in the Hessian matrix is no longer simply a dot product of two vectors, V Vayg but is added with a non-zero term of d V;. Owing to this, we can longer write the Hessian matrix as a sum of Gram matrices as in Equation (18). Nonetheless, this do not necessarily mean that the Hessian matrix admits negative cigenvalues, which is why the necessity in Theorem 4.2 does not hold for general cases.
More concretely, the Hessian matrix for multidimennional output network at stationary point with escape neurons can be denoted as:
H = D(
(23)
JEJ defined he H1?
29

---

Published as a conference paper at ICLR 2025
Compared with Equation (18), Equation (23) has extra term (H2) in the summation which con- 22((P) dPL(P) in the Hessian matrix. It tains the terms of dj; v; f 0 at the positions held by 0h or Je,öhy. easy to see that H2 is not positive semi-definite as long as H2 is not zero matrix.
Consequently, if the sufficient condition in Theorem 4.2 is not satisfied, then the Hessian matrix constructed based on the perturbation becomes summation of matrices, some of which are indefi- nite. However, there can be no theoretical guarantee that such a summation cannot lead to a positive semi-definite matrix in the end. Thus, even if the sufficient condition in Theorem 4.2 is violated, the Hessian matrix may still be positive semi- deiineefni However, despite the possibility of this case, the chance of a summation of indefinite matrices and positive semidefinite matrices resulting in positive semidefinite matrix is relatively small. Even if the Hessian matrix is positive semidefinite, the existence of escape neurons seems to hinder us from claiming anything general regarding the loss change incurred by the third-order terms." 12 This is because our current discussion regarding the third-order terms (in Appendix F.1.2) relies on the description of the flat directions in the second- order terms being Equation (19), which is premised
the absence of escape neurons. Hence, we believe that local minima contradicting the sufficient condition of Theorem 4.2 (the so-called type-2 local minima) should be rare if exist. The rigorous characterization regarding the "rarity" will be conducted in the future.
G TAYLOR EXPANSION WITH AN ALIGNED COORDINATE SYSTEM
Taylor expansion requires the function to be differentiable. As a result, the non-differentiabiiity of ReLU-like networks prohibits us from directly invoking Taylor expansion (based on the canonical coordinate system). We circumvent this problem by choosing a coordinate system whose axes cor- respond to the radial/tangential directions of the input weights and the output weights. We clarify this with the following example.
We study a one-hidden-layer layer ReLU network with 2-dimensional input and scalar output, and there is only one hidden neuron. The input weight is W1 (W11, W12)., and the output weight is h There is one training sample, denoted by x = (0.9. 0.3)T and y = 3. We fix h = 0.5 and plot the loss as function of (W11, W12) in both panels of Figure 9, The dotted gray line segments in the plots indicate a non-differentiable edge on the loss surface. We investigate a specific input weight (0.15, 0.445, denoted by the black arrows in Figure 9.
In Figure 9a, we study how the movement of the input weight changes the loss with the canonical axes (the green arrows). Each canonical axis corresponds to an input weight component. More concretely, if we perform Taylor expansion with these axes, we will compute the partial deriva- tives of #C
etc. The "+" sign here means that we need to com- pute the ODD along both the positive and negative directions along the axes to account for non- differentiability. Nonetheless, these partial derivatives do not suffice to describe the function surface since con-differentiability exists within the orthants between the axes. As result, the ODDs com- puted along such canonical axes cannot compose Taylor expansions that accurately characterize the loss surface within the orthants.
8C
82
" " (1
In this paper, we study the axes that are aligned with the non-differentiability on the loss surface, which is indicated by the red arrows in Figure 9b. Notice that these red arrows are exactly the radial and tangential directions of the input weight w1. The loss within any orthant bounded by these aligned axes is always differentiable. Hence, the partial derivatives with respect to these axes suffice to compose Taylor expansion that accurately describes the function anywhere in any orthant. Such method is equivalent to studying differentiable functions but limiting the scope to only one specific orthant. Specifically, we limit the scope of Taylor expansion to one orthant by taking As; > 0+ rather than As; + 0 in Equation (3).
Some back- siit-ffffhheeeveee tnveloee derivation can reveal that the loss is always continuously differentiable along the radial direction of input weights, which can also be observed in Figure 9b. Hence, the partial derivative along the same direction W1 suffices to describe the radial derivative, which means we do not need to study the partial derivative along the direction of Ww. However, as the
12 Note that the fourth-order terms always lead to non-negative loss changes, shown in Appendix E.1.3
30

---

Published as a conference paper at ICLR 2025
2
9/
(a) canonical axes
(b) aligned axes
Figure 9: Comparison between the canonical axes and the aligned axes. The curved surface is the loss plotted as a function of the input weight W1 = (W11, W12) of the only hidden neuron, which is non-differentiable. The black arrow (0.15, -044 denotes direction as the non-differentiable edge (denoted by the gray dotted line segments). The green arrows in (a) represent the canonical axes, while the red arrows in (b) represent the aligned coordinate (composed of the radial directions and tangential directions of w,).
input weight vector that has the same
200
0.0
0.5
1.0
15
2.5
changes
Figure 10: An occurrence histogram showing the loss changes incurred by the 5000 perturbations.
loss is not differentiable along the tangential directions, we need to study the partial derivative along both tangential directions of W1.
H VERIFYING LOCAL MINIMALITY IN A REAL CASE
The fact that the last long plateau in Figure 3a corresponds to local minimum can be verified with numerical experiment. The verification
be conducted by perturbing the parameters the end of the training with small noises and investigating whether such perturbations yield negative loss change. In our experiment, we perturb all the parameters by adding noise all of them. The noise added to each of the parameters is independently generated from zero-centered uniform distribution W(- {). where Ç > perturbation still stays in its neighborhood. Namely, the loss function restricted on the line segment connecting the parameters before and after the perturbation should be C meaning the line segment should not stretch across non-differentiable area in the parameter space. In case, we chose < & 1.28 x 10-4 We conducted the perturbation for 5000 times and recorded the loss change incurred by each perturbation, which is illustrated in frequency histogram in Figure 10. One can see that all the perturbations resulted in positive loss change, agrecing with our theoretical prediction.
small value making sure that the network parameters after
31

---

Published as a conference paper at ICLR 2025
1 ADDITIONAL NUMERICAL EXPERIMENTS
I.1 3-DIMENSIONAL INPUT, SCALAR OUTPUT
In this example, train the network with the following X₁ 's inputs:
00.
-0.
-0.6
-0.4
=
-
0.75
X2 =
0.2
X3
1.0
X4
0.4
0.5
0.4
1.0
0.3
0.6
0.4
0.2
X5
-0.1
X6
-0.9
X7
0.2
0.7
0.3
0.5
The targets yk are as follows: 1/1 87 = 0..1.
0.5, y2
0.1, y/3
0.6, 41
0.3, y/s 0.8, yo -0.3,
We use one-hidden-layer ReLU network with 50 hidden neurons whose parameters are initialized with law N (0, (9.51 x 10-11)²). The training process is visualized in Figure 11, which is quite similar that Figure 3.
15
25
@roup
1.0
0.
group
group
(a) loss curve
(b) direction of w, (rad)
(c)
w.|l
(d) hjos
Figure 11: Evolution of all parameters during the training process from vanishing initialization. The network visualized here has 3-dimensional input and scalar output. The direction of w; in (b) the direction of its first two components. The training dynamics is similar to that in Figure 3. addle escape (marked by the vertical dotted lines) is always accompanied by the amplitude growth of grouped small living neurons.
In short, the existence of small living neurons indicates that the stationary point is saddle point, while their absence means that the stationary point is a local minimum. Moreover, saddle escape always triggered by the amplitude increase of small living neurons. In this example, the training trajectory escaped from four saddle points and eventually ended up at the global minimum with zero loss.
It is worth noting that, in the above example, when escaping from the first saddle, two groups of small living neurons (group and 2) grew together, which is also consistent with Figure
I.2 2- DIMENSIONAL INPUT AND OUTPUT
In this example, we train the network with the following 's inputs:
0.3
(.0)
(-1.0)
0.4
0.5
X2
1.0
X3
x4
-0.4
The targets yı 's
0.6
0.5 -1.0 y3
-0.44 0.6
0.8
Y1
00.5
y2 =
y1 =
0.2
We use a one-hiddennlayer ReLU network with 50 hidden neurons whose parameters are initialized with law V (0, (9.51 10-"))?. The training process is visualized in Figure 12.
This training process escaped from 3 saddles. Notably, the last saddle escape involves the param- eter variation of neurons that small living neurons, which contrasts the scalar-output case,
32

---

Published as a conference paper at ICLR 2025
1.00
0.0
3.5
2.2
0.0
1.5
6.0
0.5
3.5
2.2
25
opech
opoch
a0och
(a) loss curve
(b) direction of w, (rad)
(c) l|w,
group
2
opach
(d) his
(c) h2:
Figure 12: Evolution of all parameters during the training process from vanishing initialization. The network visualized here has 2-dimensional input and output. In this case, the output weights associating with one hidden neuron have 2 components, tracked by (d) and (e). The saddle escape (marked by vertical dotted lines) be caused by the amplitude increasing of small living neurons or the splitting of neurons that already have non-negligible amplitudes. The trajectory of one neuron thickened in (e) for discussion.
350
200
50
2.5
bas chargee
An occurrence histogram showing the loss changes incurred by the 5000 perturbations applied to the end of training in the 2D output case.
Figure 13:
as escape neurons the multidimensional output case might have non-zero output weights (Defini- tion 4.1).
Eventually, the training reached local minimum. We verify that is a local minimum by locally perturbing the network parameter the end of training with independently drawn noise for 5000 times. The resulting changes in loss are all positive. The occurrence histogram demonstrating the loss changes incurred by the 5000 perturbations are shown in Figure 13.
As it is a complicated case, we identify the escape neurons in each saddle encountered dur- ing training. To facilitate the presentation, first present the evolution of errors at both output components,
Úk1 Vel and 6+2 == 022 yı2, which are shown in Figure 14.
The first saddle point the origin. As all the output weights are zero, we know that the 8s; for any E and tangential direction v; (from Equation (6)). We remind the reader that v, the tangentiil direction implied in the tangential derivative 8C De, We will show that there exists a specific tangential direction v; such that y v, * 0, which makes the origin a saddle point by Definition 4.1 and Theorem 4.2.
33

---

Published as a conference paper at ICLR 2025
122"
110
215
0.50
325
0.00
-125-
-10
0.0
0.5
1.5
3,5
2.0
25
33
0.
0.5
3.0
2.5
2.0
2.5
70
epoch
(a)
(b) ex2
Figure 14: Errors at both components of output.
We find v; = 0.51), which corresponds the angle of 1.03 rad in Figure 12b, where the group neuron gathered at epoch 25k. Knowing that ε1 = 0.6 and 621 = -0.5 (which can be roughly read from Figure 14 at epoch 25k), we have:
di; V = (e11x) + x21 (0.85, 0.51) (-0.32, 0.2) (0.85, 0.51) 0.37 0.
We thus identified the escape neurons be the ones that have zero-amplitude parameters this saddle. Escaping from the first saddle was triggered by the movement of a group of small living neurons (group neurons) exploiting the loss-decrg path offered by such escape neurons.
At the second saddle point, group neurons have non-negligible amplitudes while group neurons indicate the existence of sero-amplitude neurons in the nearby saddle. In other words, if the param- eters are placed exactly at the saddle point, the parameters of group 2 neurons should all be zero Consequently, at the saddle point, ac
for any that denotes group 2 neuron and any tangen- tial direction V; (from Equation (6)). In this case, we can find a tangential direction V, of group neurons such that d'; V; * 0. We specify that V; = (-0.65, 0.76), corresponding to 3.22 rad in Figure 12b, which is where group 2 neurons were attracted to before the second saddle point was escaped. Reading from Figure 14 that 631 0.4 and 641 -0.8, we have:
3e;
di; Vi- (c31x3 + +(xx) (-0.65, -0.76) = (-0.48, 0.56) (-0.65, 00.66 0.11 +
This tells us that, the neurons with zero parameters are escape neurons at the second saddle. Escap- ing from the second saddle then triggered by the movement of group of small living neurons (group 2) exploiting the loss-decreasing path offered by such escape neurons.
At the third saddle point, both groups of neurons have non-negligible amplitudes. We will show that one of the group 2 neurons is approximately an escape neuron. It is not exactly an escape neuron since the parameter not exactly the saddle point but in the vicinity. The parameter trajectory of this specific neuron is denoted by thickened curves in Figure 12. In this part, i denotes that single hidden neuron. The input weight this neuron is (-0.12, 0.63) (which can be computed from the readings in Figures 12b and 12c), and we pick one of its tangential derivatives v; = (0.98, 0. 0.19). We will show that d V. j 0 and dc 25, (the tangential direction along v:) equals zero, making this neuron (approximately) an escape neuron.
We can roughly read from Figures 14a and 14b that en = -0.18, 621 = 0.12. 0.12, 622 0.08 at epoch 100k. With this, we can compute that
di V: = (E1IXi + +2122) (0.98, 0.19) = (0.18, 0.03) (0.98, 0.19) = 0.18 + 0.
:x V, = (c12X1 + 622X2) (0.98, 0.19) = (0.12, 0.02) (0.98, 0.19) = 0.12.
We can also read from Figures 12d and 12e get h1 0.37, hg -0.53. These leads
DC
0z, = h v. + h2. v. = 0.00.
With these computations, we show that the neuron we study is (approximately) an escape neuron. We can apply the same computation to any group 2 neurons the third saddle point to find that all of them are (approximately) escape neurons. Eventually, their movement (splitting) caused the parameter escape from this saddle.
34

---

Published as a conference paper at ICLR 2025
With the above, we show that the experiment is consistent with our Fact 4.7. All saddle escapes the results of parameter variation of escape neurons in the nearby saddle point.
We can also analyze that the last stationary point at which the training process ended is a local minimum. At the end of the training, all the input weights of the hidden neurons lic within the angle range of (2 tT, IT), which means that w; X2 O, and {x1, } is the only training sample that has not been perfectly fitted yet, as shown in Figure 14. Thus, according to Equation (3),
dy = 0, for all (1,j) E 7 x J and all tangential directions V,-
(24)
This is due to the following: the first summation Equation (3) is zero because all the error terms involved are zero. The second summation of Equation (3) is also zero because a O. Moreover, dy disqualifies all the hidden neurons from being escape neurons, according to Definition 4.1.
J APPLICABILITY OF FIGURE 4 TO SMALL BUT NON-VANISHING INITIALIZATION
Networks are rarely trained in the vanishing initialization regime. For one thing, the alignment of input weights might excessively reduce the number of kinks in the learned function and prevent the training from reaching global minima Boursier and Flammarion (2024), as is the case in Figure 3. In this section, we train the network initialized with law / (o, (8.75 x 10 10-4)2 (by rescaling the initialization weights in Section 4.1.1 without changing their direction). The process is visualized in Figure 15. The larger initialization scale enables the network to reach zero loss. With this example, show how our insights derived from vanishing initialization be applied to small but non- vanishing initialization.
8388 88
00
po0p3
popp
(a) loss curve
(b) direction of w: (rad)
(c)
w
(d) hoo
Figure 15: Evolution of all parameters during the training process with small but non-vanishing initialization scale. There more groups of neurons formed compared Figure 3. The loss experiences repeated accelerations and decelcrations, influenced by nearby saddle points. The ac- celerations of loss decreasing roughly coincide with the amplitude increase of groups of small living neurons.
The loss curve for this initialization scheme is shown in Figure 15a, where one can observe the de- crease of loss accelerated and decelerated several times. This indicates that the network parameters were affected by several nearby saddle points. We mark the beginnings of the accelerations with dotted vertical lines in Figure 15a. The evolution of all the parameters summarized in Figures 15b to 15d. Just as in Figure 3, we color-code the curves in Figures 15b to 15d based on the w; grouping.
We can observe that the accelerations of loss decreasing also roughly coincide with the amplitude increase of small living neurons. This means that the acceleration of loss decreasing in this regime also exploits the loss-decreasing path given by the escape neurons in the nearby saddle points, similar to the vanishing initialization regime.
We also note two differences between small but non-vanishing and vanishing initialization. First, with the former, the loss curve no longer has extremely flat plateaus, and the small living neurons no longer appear extremely small in amplitude. This because the gradient flow from small but non-vanishing initialization does not get as close to the stationary points as does the trajectory from vanishing initialization Jacot et al. (2022). Second, the former gives rise to four living neuron groups, corresponding to four kinks, 13 which suffices to reach zero loss; while the latter only creates two
1eSee Figure 17 for the emergence of the four kinks.
35

---

Published as a conference paper at ICLR 2025
kinks and gets stuck high loss. The vanishing initialization limit induces low-rank bias the weights (Maennel et al., 2018; Jacot et al., 2022; Chistikov et al., 2024), which is alleviated by larger initialization scales to yield better expressivity with more kinks in the learned function (See Appendix L.). A detailed account of this is in Appendix K. The investigation of the minimal initialization scale that results in global minima meaningful future direction.
K How LARGER INITIALIZATION SCALE ALLEVIATES THE LOW-RANK BIAS
The low-rank bias induced by small initialization is due the distinctive dynamical pattern induced by such initialization. The rotation speed of neuron is, loosely speaking, ((1); while the amplitude increase of neuron slower when its amplitude smaller, as observed by Maennel et al. (2018); Boursier et al. (2022); Chistikov et al. (2024). Thus, given the initialization is sufficiently small, the neurons will have enough time to gather close the attracting angles: The attracting angles are determined by the network function, which remains almost fixed (approximately as a zero function) during the early alignment phase when all the neurons have negligible amplitudes. As result, the attracting angles are also almost fixed during this phase, and the neurons have enough time to approach them. This process causes a lower rank the weight, which is preserved throughout training. With (slightly) larger initialization, the neurons' amplitude increase becomes much faster, and the network function will deviate from the zero function before the neurons rotate to the original attracting angles. The rapid variation of network function gives birth to attracting angles and thus leads to a higher rank the weight matrix, as shown in Figure 15. There, the group neurons and group 3 neurons does not manage to merge with group and group respectively, before being steered away by newly generated attracted angles. However, such merging took place thoroughly with vanishing initialization (Figure 3).
L THE EVOLUTION OF THE LEARNED FUNCTION
In this section, demonstrate the evolution of the learned function for both vanishing initialization and slightly larger initialization. We highlight the output function before and after the amplitude increase of the grouped living neurons.
The learned functions from vanishing initialization are shown in Figure 16. Remember that, to simulate vanishing initialization, we initialized all the parameters in the network independently with law N (o, (5 x 10 6)2). In Figure 16, we only draw the network function (using the blue line) with respect to the first component the input, since the second component always in the training data to simulate the bias. Namely, in Figure 16, we draw the following function:
())) = ((xx 1)), x E IR
(25)
where ((-) is the network function in Equation (1).
2.4
04
04
47
02
02
12
-32
-25
-10
20
22
-15
14
15
22
35
(a) epoch 42k
(b) epoch 152A
(c) epoch 500k
Figure 16: The evolution of the learned function during training with an initialization standard deviation of 10-6
The five red stars are (Ik, ykk ss I& 's being the first component of all the training samples, and yı the scalar target values. The five dots corresponds to (æx, ((x))).
One can observe that, at epoch 42k, the network function is, approximately, a zero function. After the amplitude increase of the group neurons, by epoch 152k, the network function has learned
36

---

Published as a conference paper at ICLR 2025
one kink. The amplitude increase starting from roughly epoch 152k forms the second kink of the network function. Afterward, the network stops evolving, keeping the two kinks before the end of training (epoch 500k).
The evolution of the learned function from a slightly larger initialization scale is shown in Figure 17. In this experiment, the parameters initialized independently with law / (0, (8.75 x 10-*). We can observe that, with the amplitude increase of 4 groups of small living neurons at roughly epochs 14k, 66k, 144k, and 280k; there emerged 4 kinks. Notice, there should be kinks in the learned function at the end. However, only of them are shown, because the last one is located too far away from the input range we show in the image.
64
15
400
-23
-25 +10
-15
-1
(a) Epoch 14k
(b) Epoch 66k
(c) Epoch 144k
29
-35
10
63
39
15
25
20
-12
100
05
38
15
(d) Epoch 280k
(e) Epoch 580k
Figure 17: Evolution of the leamned function during the training starting with a larger initialization scale.
M PROOF OF PROPOSITION 4.8
M.1 CONDITIONS FOR UNIT REPLICATION TO PRESERVE STATIONARITY
Let us denote the parameters of the stationary point before unit replication by P and the parameters
after by P', Then, Definition 3.6 dictates that P must satisfy:
OC(P)
0, Vje
(26a)
Öhys
P=P
(P
0, ViET:
(26b)
Ör;
P
OC(P)
O, tangential direction V; of W,.
(26c)
0s;
P=P
is easy to check that, after unit replication. the above still holds for the hidden neurons within the
set 7 (i which are the hidden neurons untouched by unit replication, since the network function not changed by this process. For the rest of the hidden neurons in P which are indexed by L, also deduce whether they conform to the conditions for stationarity. We have the
following:
AL(P)
aC(P)
=wg d djg. = 3:wi, djio = 3t
= 00
(27)
0h
ahjio P=
P=P
37

---

Published as a conference paper at ICLR 2025
If w Wio > 0, then the newly generated hidden neurons have radial directions:
OC(P)
OC(P)
h jio
d
=
Σ
hyio diio
uig = 71
0.
(28)
3ng.
Ur
P=F'
JEJ
jEJ
P-P
(=0)
If Wa //:/ = 0, then, by convention, the radial derivative
aC(P)
are set to zero without
Dr
loss of rigor.
As for the tangential derivatives of the newly generated hidden neurons, we have:
PC(P)
OC(P)
hyg
Vi = 70 Σ
h3o d
Vio = =1
(29)
0sg
Osi
P-P
JEJ
JEJ
P-P
(20)
where we takev, Vig a tangential direction of Wio- This justified by the fact that B1 > Vl
Namely, we are implying that any tangential direction of wiy must also be
With the above, infer either of the following is true:
CC(P) 8ag
2 0 for all possible tangential direction v.g 's if and only
1. aC(P) 0, tangential direction Vi, Wiy Deip
2. 7/ 2 L
which concludes the proof.
M.2 CONDITIONS FOR UNIT REPLICATION TO PRESERVE TYPE-1 LOCAL MINIMALITY
The necessary and sufficient condition to preserve type-I local minimality after unit replication to avoid generating escape neurons and to preserve conditions for stationarity. Thus, we prove Proposition 4.8 by seeking necessary and sufficient conditions to achieve both these two goals. Let us denote the parameters of the stationary point before unit replication by and the parameters after by P
M.2.1 AVOID GENERATING ESCAPE NEURONS
First, let us consider two unit replication schemes:
1. Replicate a tangentially flat hidden neuron Choose replicate hidden neuron in satisfy- ing 0C(P) = 0. for all tangential direction Vi. 's.
0eso P-P
2. Replicate with active propagation Let & V
Proposition M.I. Unit replication process conforming to either one of the two methods is sufficient and necessary for avoiding escape neurons.
Proof. Sufficiency:
Let us discuss the first way, replicating a tangentially flat hidden neuron. Since we have:
OC(P)
Σ
Vig Tijto 1 V., V tangential direction vi, of W..
(30)
Osi
jEJ
Based on the definition of type-1 local minima, we know that:
Vio
Vio 0, Vj J, tangential direction Via of Wig-
(31)
jio
38

---

Published as a conference paper at ICLR 2025
36 = d y E J, for all tangential direction Vig of Wag (2) wig Biwi, with > 0, which means any tangential direction of wig are also tangential direction of Wig. These, combined with Equation (31), leads to:
After unit replication, we have: For all / E I,, (1)
O, VIELvjE J, tangential direction Vig of w.b
(32)
According to Definition 4.1, such itj cannot be escape neurons.
Next, let us discuss the second way, replicating with active propagation. If the replicated hidden neuron langentially flat hidden neuron, then the discussion above will already guarantee that the unit replication process does not introduce escape neurons. Thus, we only need to focus on the following type neurons:
PC(P)
Jio
Yig Ji0
Vio > 0, for some tangential direction Vio of Wi..
(33)
Usio
jEJ
In this case, for any direction Vo. with unit replication, in the same direction (Vig
8C(P) Osio
=EEE
Vio
Vio > we have that, after
P-F
Vin):
PC(P)
OC(P)
0s
Σ JEJ
Vib 72 Σ
hyto d
Vio
78
Osio
#
(34)
P-P
jEJ
P=P
if n L.
For any direction Vi with aL(P)
EjeJ hgjo d
Vio
Vio = 0, we know that:
Biio
P-F
jio
Vio jip
Vi. 0, VjE
(35)
since the parameter before unit replication type-1 local minimum. After unit replication, we have that, in the same direction, in the same direction (V Vi.)
dC(P)
OC(P)
0sg
Vib
n Σ
hyjo
Vio n
Osio
(36)
P
JEJ
jEJ
P-P
and we also have:
% Vi
v = 0, J.
(37)
jin This shows that replicating with active propagation also avoids introducing escape neurons.
Thus, unit replication strategy conforming at least one of the two methods mentioned above is sufficient to avoid generating escape neurons.
Necessary:
We conduct this proof by contradiction. If we replicate hidden neuron in that is not tangentially flat (described by Equation (33)) with 7 = 0 for some 1 E L, we can prove that there must exist escape neurons in the resulting parameters. For convenience, let us specify that has ?/ 0.
Since Equation (33) holds, we can find a direction, denoted by a unit vector Vio. satisfying vi, Wio 0, this must mean that there exists j with:
and @C(P)
P-P =EEE Tejio jio
Vio
vio
>
25ig
Vio Vio + O. jio
(38)
After unit replication, we have that, for the hidden neuron to. the loss function's derivative with respect to its input weight in the direction of Vig Vi, is:
OC(P)
Ös
d 3
V.
78
Σ
jig
d Vio jio
Via
0.
(39)
P-P
JEJ
JEJ
39

---

Published as a conference paper at ICLR 2025
Additionally, we also have:
d 340
V.6
d fi
vio
# 0.
(40)
Equation (39) and Equation (40) signifies the existence of one escape neuron ig after unit replication.
M.2.2 PRESERVING STATIONARITY CONDITIONS
The first half of Proposition 4.8 is the sufficient and necessary conditions for preserving conditions for stationarity during unit replication.
M.3 PUTTING THINGS TOGETHER
We want to avoid generating escape neurons while preserving conditions for stationarity at the same time. The necessary and sufficient condition for the occurrence of these events should be the intersection the necessary and sufficient conditions Appendix M.2. and Proposition 4.8, which exactly the necessary and sufficient condition of Proposition 4.8.
N RESULTS REGARDING INACTIVE UNITS
There are 2 types of inactive units, depicted in Fig-
18:
(2-1) Orthogonal u4
W
Add more hidden neurons, which are indexed with
1 E 1", where h130 is arbitrary, and Wio X2 = 0, for
all K.
Figure 18: inactive units
(2-2) Negative (positive) units.
Suppose a = 0. Add new hidden neurons i E /-, where w,-- - X < 0 for all E K, and ha- are arbitrary for all j E J. This is what we call negative units. This embedding scheme can likewise be extrapolated for when a = 0, giving positive units.
N.1 RESULTS REGARDING ORTHOGONAL UNITS
This type of network embedding does not generally preserve conditions for stationarity or local minimality. Nonetheless, based on definition of stationary points in Section 3.2 and our analysis of local minima Section 4.1, one can carry out investigations into them on a case-by-case basis when analyzing specific example.
Let us first discuss whether stationarity will be preserved under this network embedding scheme. Let us denote the parameters after adding orthogonal units by P'. Since the newly added hidden neurons
satisfy Wa = 0 for all k E K, we know that @C(P)
0. and 8C(P) equals zero.
ah,,
ār. P-P'
Thus, whether the tangential derivative preserves the condition in Definition 3.6 determines whether stationarity is preserved. For the tangential derivative, have:
OC(P) Es
Σ hood iEJ
j
Vio.
(41)
P-Pr
Stationarity would require the above to be non-negative for all possible tangential direction v which cannot be guaranteed for general cases. Certain trivial cases where definite conclusions can be established are when the network has reached zero loss (d) = 0 for all possible v,o) or 11.0 for all j J. In those cases, stationarity will be preserved.
We are not able to have general conclusions regarding whether the insertion orthogonal units preserves type-1 local minimality well since orthogonal units do not specify anything regarding whether they are escape neurons in general.
14 Orthhggonal units discussed Fukumizu et al. (2019).
40

---

Published as a conference paper at ICLR 2025
N.2 RESULTS REGARDING NEGATIVE UNITS
Proposition N.1. Suppose that a 0. Adding negative units preserves the stationarity station-
ary points.
Proof. Let us denote the parameters after the insertion of negative units by P'. It is easy to check that the parameters associated with the originally existing hidden neurons i / still satisfy the conditions for stationarity after inserting the negative units, since the negative units do not change the network output yı for all inputs X For the negative units i E we observe the following.
dy-
Σ A:
a eRyxx= =0,
(42)
since this summation will be over an empty set of This leads to OC(P) 8h ; P-p'
= and
OC(P)
0 according to their formulae in Equations (4) and (5).
Moreover, have jt dy- for tangential direction V,- of w since w,- is orthogonal to no X 's.. This shows that the tangential derivative of the loss function with respect w,- is also zero, according to Equation (6).
Remark N.2. We remind the readers that Fukumizu al. (2019) has already proved that adding negative units will preserve the local minimality of the network.
O RESULTS REGARDING INACTIVE PROPAGATION
Inactive propagation can be carried out as shown Figure 19: add hidden neurons ix Ix where W,x taken arbitrarily and hyex = O, j E J, as shown in Figure 19.
This type network embedding also does not pre- serve stationarity type-1 local minimality gener- ally.
Ayp
We start by discussing the preservation of stationarity. Let us denote the parameters after adding units with inactive propagation by P'. In this case, the added hidden neurons have hyax 0 for all j E J. As result, according to Equation (5) and Equation (6), the radial derivative (which exists only when W,x * 0) and tangential derivatives (towards all tangential directions) must all be zero. However, the derivative with respect to output weight might not be zero:
Figure 19: inactive propagation
dC(P) Oh
Wsx d djix.
(43)
P-
Thus, the stationarity condition is not guaranteed hold in general after adding inactive propagation units.
However, one may also notice that there are certain ways of enforcing stationarity: we choose W,x O or w,x 3w; with i E 7, B > 0, then equate Equation (43) to zero and preserves stationarity. The former also belongs to the case of orthogonal units, and the latter also belongs to the case of unit replication.
Then, we discuss whether type-1 local minimality is preserved. Preserving type-1 local minimality entails preserving stationarity and avoiding escape neurons, according to Theorem 4.2. We have discussed in the above that stationarity is not necessarily preserved in general. Moreover, regard- ing escape neurons, derivative being zero (since h = 0), without further strong restriction on the d 's, the inactive propagation unit is highly likely an escape neuron.
can find that since inactive propagation unit must have their tangential
41

---

Published as a conference paper at ICLR 2025
P COMPARISON OF THEOREM 10 OF FUKUMIZU ET AL. (2019) AND PROPOSITION 4.8
Fukumizu al. (2019) attempted to address the same question of whether local minimality is pre- served by network embedding for our setup (which is also non-smooth) with its Theorem 10. It managed find specific scheme of unit replication that turns local minima into saddles, poten- tially contradicting Proposition 4.8. However, one can verify that the assumption in their theorem does not conform to our setting, which we explicate below.
Theorem 10 of Fukumizu et al. (2019) requires constructed F matrix not to be zero matrix to construct positive and negative eigenvalues for the Hessian matrix the resulting parameters after unit replication. In this way, it is shown that the parameters after unit replication constitute strict saddle. Please refer to Fukumizu al. (2019) for more detail.
However, it turns out that, for the empirical squared loss, which is widely used loss, the F matrix zero at type-1 local minimum, which we prove in the rest of this section. Hence Theorem of Fukumizu al. (2019) cannot say anything about such situation.
Lemma P.I. Suppose a of parameters P is stationary point in our setting. If an input we is such that Xx. W, & O, for all k & K (an assumption also taken by Theorem 10 of Fukumizu et al. (2019)), must have that d 0 for all E J.
Proof. If w. o 0 for all k E K, then the network is continuously differentiable. Thus must have (CPP) for all v,'s. Otherwise, will not be a stationary point.
Os,
Moreover, the fact that the stationary point we are investigating is a type- I local minimum gives:
d V d v; = 0, for any tangential direction v, of w,.
(44)
Moreover, since the parameter before unit replication P is stationary point, must have:
OC(P)
d O.
(45)
0hgs
PPP
Thus, djio must lies in the tangential space of w,. If djoo 0, then it must be parallel to some unit vector v, satisfying V; w, = 0, which contradicts Equation (44).
Then we investigate the F matrix. It is helpful to recap the setting of Theorem 10 by Fukumizu et al. (2019). First, they studied the ReLU activation function, meaning at 1,a 0. Morcover, they only discussed replicating a hidden neuron in with
X +
(46)
which account for in the above helper lemma. From Fukumizu et al. (2019), know that F E IR dx|x has each of its column being:
Op(x: W
F.J
CkyXx d dyto-
(47)
Ow io
KEK
30
Notice that the above derivative not hindered by the non-differentiabiiity in the activation function thanks to Equation (46).
Remember that we have proved djio = O for all j E J, rendering a zero matrix.
Q THE APPLICATIONS OF THE INSIGHT FOR NETWORK EMBEDDING
Our discussion on network embedding in Section 4.3 extends multiple previous results that did not consider con-differentiable cases.
Wu et al. (2019); Wang et al. (2024) proposed a training scheme where one neuron is split at an underfitting local minimum create escapable saddle point. Such training scheme can lower
42

---

Published as a conference paper at ICLR 2025
the cost of training since the training can start from smaller (pretrained) network. Proposition 4.8 effectively shows how to implement such a scheme for scalar-output networks, where all the local minima are all of type-1: simply replicate newron whose input weight has non-zero tangential derivative and choose "it I 0. Such an insight might also be instrumental for multidimensional- output networks since type-2 local minima should be rare,
discussed in Appendix F.2.1.
Our results regarding stationarity preservation also serve as constructive proof that one can always embed stationary point of narrower network into wider network. In other words, the stationary points of wider networks "contain" those of the narrower networks. This is termed the embedding principle, discussed in Zhang al. (2021) for smooth networks.
Section 4.3 also provides perspective of understanding the merit of overparameterization for opti- mization. Stationary points and spurious local minima might worsen the performance of GD-based optimization. However, to our rescue, network embedding can cause some stationary points (local minima, resp.) to lose stationarity (local minimality, resp.). This might explain why wider networks tend to reach better training loss. It is also easy to check that embedding parameters that are not sta- tionary points (local minima) will not result stationary points (local minima). Similar phenomena are noted in Şimşek et al. (2021); Fukumizu et al. (2019); Zhang et al. (2021) for smooth networks. meaningful next step is to quantify how manifolds of saddle points and local minima scale with network width (Şimşek al., 2021) for ReLU-like networks.
43


Reviews:
[{"review_id": "20lzf4QIiL", "reviewer": "Reviewer_k7xc", "summary": "The paper studies the loss landscape of one hidden layer with ReLU-like activations trained on squared loss. It provides conditions for stationary points that work for both differentiable and non-differentiable parts of the loss landscape. The authors introduce \"escape neurons\" as the indicator of whether stationary points are local minimums. This conclusion is necessary and sufficient for scalar-output nets. With this characterization, they take a step further to understand the saddle-to-saddle dynamics of two-layer networks. They also discuss how network embedding helps reshape the stationary points.", "questions": "Is it possible to come up with a more rigorous example (with gradient dynamics analysis) for Figure 4 to understand the saddle-to-saddle dynamics?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The ReLU networks' stationary points analysis is novel and interesting. The authors are able to characterize the \"escape neurons\" as the indicator of two-layer ReLU networks defined with first-order conditions, which can help to characterize whether a stationary point is a strict saddle to escape. This rigorous analysis can help us further understand the saddle-to-saddle dynamics from a landscape perspective. \n\n- The toy experiments verify the theoretical results above with vanishing initialization, both displaying examples for stationary points without escape direction and saddle-escaping phenomena when \"escape neurons\" exist. The explanation of Figure 4 is intuitive and insightful.\n\n- The escape neuron analysis helps to understand how the interesting \"network embedding\" technique reshapes the landscape.", "weaknesses": "- The analysis for the landscape seems to be a bit limited to the local stationary point analysis as the supplement of [Kumar and Haupt, 2024], precluding other types of saddles. To understand the saddle escaping or saddle-to-saddle dynamics, it would be better to more analyses of gradient dynamics in the neighborhood of saddle points. For example, can we have the escape direction in Corollary 4.4 related to the gradient, implying that GF/GD can help escape the saddle points?\n\n- The discussion of the saddle-to-saddle dynamics with vanishing/finitely small initialization (sec 4.2) looks too heuristic to me. It is better to present those implications or corollaries of the stationary point characterization more formally instead of descriptions like Consequence 4.7.  \n\n- As a minor weakness, the toy examples only have one-dimensional inputs (with bias), which is not comprehensive. More empirical results need to be shown to corroborate the theoretical results' implications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q5fAVLe9dd", "reviewer": "Reviewer_WEjG", "summary": "This paper presents a new definition of stationary point for non-differentiable neural nets caused by ReLU-type activations. It characterizes the relationship between the local minimum and the \"escape neuron\", giving a sufficient and necessary condition for local minima under scalar output. Based on this notion, the authors provide a new perspective on saddle-to-saddle dynamics and show that escape from a saddle must perturb the escape neuron. Furthermore, the authors investigate the effect of unit replication on the stationary point.", "questions": "1. The escape neuron condition requires that the output weight of a specific neuron be zero. Does that imply the landscape of scalar output  2-layer neural network is rather trivial? It seems to be sufficient to just perturb the weight of this specific neuron to escape the saddle. \n\n2. Although the authors argue that the sub-differentiable definition is unsuitable for a non-convex function, does it also fail specifically in two-layer neural networks? In particular, it would be helpful to provide an example in the ReLU network to show that the sub-gradient can not properly capture the saddle point. Since the proposed analysis is restricted to the two-layer neural nets, it would be helpful to understand the difference specifically in this setting.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed definition is novel and interesting, and it provides a more complete characterization of saddle points in two-layer ReLU neural networks. This new definition can help to deepen our understanding of the loss landscape and training dynamics.", "weaknesses": "1. The current studies focus on the two-layer ReLU neural network and it is unclear how the result can be generalized to deep networks, especially since the characteristic of escape neurons can be complicated when the network is deep, it is unclear if we can draw meaningful conclusions there using the proposed notion. \n\n2. The current study focuses on a non-differentiable activation function only, and the conclusion will be trivial if using a differentiable activation function. As more and more smooth activations are used such as GELU and ELU, the result will have less meaningful impact on practice. \n\n3. Fact 4.6 is an important conclusion to discuss the dynamics, but it was provided as an empirical observation. As this paper mainly focuses on theoretical characterization, it would increase the rigor of the analysis by providing a theoretical justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w4RTRDZUiX", "reviewer": "Reviewer_8foJ", "summary": "In this paper, the authors study the dynamics of shallow ReLU-like neural networks. The analysis starts with the definition of stationary points, saddle points, and two types of local minima for ReLU-like networks, whose loss landscape is characterized by non-differentiability confined in hyper-planes. Then, the concept of an escape neuron is defined, whose existence indicates the possibility of escape. Numerical experiment with synthesized data and vanishing initialization motivates the theory that saddle escaping is driven by small living neurons gaining amplitude following alignment. Further experiments show the impact of small but finite initialization. Last but not least, the condition under which embedding preserves stationarity is given in the non-smooth setting.", "questions": "Although I appreciate the experiment in section 4.1.1 very much and agree that it supports the authors’ theory of saddle escaping, I don’t know if the evidence is strong enough to rule out all alternative theories (discovered or not).\n\nIs there any specific reason for choosing $x_k$ and $y_k$ as given in the paper in section 4.1.1?\n\nLine 453/454 should the “does get” be “doesn’t get”?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors developed their own notation based on direction derivatives to tame the non-differentiability in ReLU-like networks, allowing them to study the saddle-to-saddle dynamics in the non-differentiable landscape. With theoretical argument and numerical experiment, the authors manage to explain the whole trajectory when training a shallow ReLU-like network with vanishing initialization. Although not every argument in the paper is original, I am happy to see a paper that collects a lot of existing results and presents them in an integrated manner. The authors put a lot of effort into making their result understandable.", "weaknesses": "There are constraints for the theoretical analysis, and the experiments are performed on synthesized data instead of real data. Also, I find the definition of stationary point a little counter-intuitive. I have the impression that the definition of stationary point in this work is not exactly the extension of stationary point of smooth function, which is where $f’(x)=0$ regardless of trapping GD or not. It appears to me that Clarke subdifferential the authors mentioned could be better in this regard, in the sense that the origin of both $|x|$ and $-|x|$ are considered to be a stationary point. With the authors’ definition of stationary point, I can imagine that local maxima are usually not stationary points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8mP6DTzWJ", "reviewer": "Reviewer_BZd8", "summary": "This paper characterizes the (possibly non-differentiable) stationary points of a one-hidden layer ReLU-like neural networks. By ReLU-like, I mean a piecewise linear activation with a single kink at x = 0. The authors specifically define the notion of a stationary point in their problem of interest, using the notion of directional derivatives. With the definition they are able to show that if a stationary point does not contain \"escape neurons\" - which is a specific type of neuron, the stationary point is a local minimum. This insight enables the authors to understand saddle-to-saddle dynamics, and network embedding principles with their own perspective, and extend existing results to non-differentiable settings.", "questions": "1. Addressing the weaknesses would be good. If I can see that the author's definition of saddle points makes sense, and how they are related to saddle points in other papers, I would raise my score.\n\n2. Some clearer wording is needed: in Lemma 3.2, we have \\delta(s_i) > 0 sufficiently small. Is there a clearer way to state this? In pg. 8, could you be more specific what larger initialization scale 8.75 x 10^-4 means? In pg.20, I am not sure if we can justify the existence of higher order directional derivatives just with continuity, because in general, continuity does not imply the existence of directional derivatives.\n\n3. I got curious about the link between existing work and this paper.\n - It is proven that in [3], first-order methods avoids strict saddle points. The intuition in this paper is that strict saddle points have a descent direction where gradient descent falls down. I think this notion is similar to the notion of escape neuron, where we have a certain direction that the loss decreases. Do the two concepts have something in common, or are they intrinsically different?\n - In pg 6, you mention the notion of dead neurons. [2] also uses the term dead neurons - are they related?\n\n[3] Lee, Jason D., et al. \"First-order methods almost always avoid strict saddle points.\" Mathematical programming 176 (2019): 311-337.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of considering a different coordinate system to exploit the fact that the nonlinearity is only non-differentiable at x = 0 is interesting.", "weaknesses": "1. I am not convinced on the definition of stationary points for this paper (Definition 3.3). Here, the authors define stationary points as points that have zero derivative with respect to second-layer weights and radial direction and nonnegative directional derivative with respect to tangent directions. The point I am not convinced is \"why are the radial terms and tangential terms considered differently?\" - is it because the radial term is differentiable but the tangential term is non-differentiable? \n\nThe definition of stationary points also felt more like a \"local minimum\" to me (this could possibly be wrong - but my understanding is that the defined saddle point has zero differential at differentiable parameters and local minimum for non-differentiable parameters).\n\nOverall, the definition of stationary points in Definition 3.3 seems to be quite restricted. \n\n2. Also, it would be good to explicitly mention that the notion of stationary points this paper discusses may be different from the notion from other papers in the literature, at the introduction and in the abstract. For example, there exists papers that completely characterize Clarke stationary points of a one-hidden layer ReLU network [1], so to make the contributions more concise, describing the notion of stationarity could help. \n\n3. It is unclear to me how their approach is different from using one-sided directional derivatives (e.g. [2]). I think this paper also uses the notion of one-sided differential to mitigate non-differentiability (from the definition of radial and tangential derivative in pg 4)? Probably another major difference is radial/tangential derivatives - but I don't see the necessity of such concepts except when the authors define stationarity.\n\nOverall, it is rather unclear how non-differentiability mitigates an analysis of saddle points, and how the authors actually overcame the difficulty with their idea.\n\n4. Most part of main results are discussions rather than concise theorems or statements. Furthermore, while mentioning saddle points, it is confusing whether the authors are referring to \"their notion of saddle points\" or the notion of saddle points in the literature.\n\n[1] Wang, Yifei, Jonathan Lacotte, and Mert Pilanci. \"The hidden convex optimization landscape of two-layer relu neural networks: an exact characterization of the optimal solutions.\" arXiv preprint arXiv:2006.05900 (2020).\n\n[2] Cheridito, Patrick, Arnulf Jentzen, and Florian Rossmannek. \"Landscape analysis for shallow neural networks: complete classification of critical points for affine target functions.\" Journal of Nonlinear Science 32.5 (2022): 64.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)