Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations
Authors: Akshay Kumar
Jarvis Haupt
Venue: Accepted by TMLR ()

Abstract: This paper studies the gradient flow dynamics that arise when training deep homogeneous neural networks assumed to have locally Lipschitz gradients and an order of homogeneity strictly greater than two. It is shown here that for sufficiently small initializations, during the early stages of training, the weights of the neural network remain small in (Euclidean) norm and approximately converge in direction to the Karush-Kuhn-Tucker (KKT) points of the recently introduced neural correlation function. Additionally, this paper also studies the KKT points of the neural correlation function for feed-forward networks with (Leaky) ReLU and polynomial (Leaky) ReLU activations, deriving necessary and sufficient conditions for rank-one KKT points.

Published in Transactions on Machine Learning Research (03/2025)
Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations
Akshay Kumar
kumar51 @umn. edu
Deportment of Electrical and Computer Engineering University of Minnesola, Minneopolis, MN
Jarvis Haupt
jdhauptshummn edu
Department of Electrical and Computer Engineering University of Minnesota, Minneopolis, MN
Reviewed on OpenReview: https: openreieew. net/ met/ forum? id= VWM6V1gi3k
Abstract
This paper studies the gradient flow dynamics that arise when training deep homogeneous neural networks assumed to have locally Lipschitz gradients and order of homogeneity strictly greater than two. It is shown here that for sulficiently small initializations, during the early stages of training, the weights of the neural network remain small in (Euclidean) norm and approximately converge in direction to the Karush-Kuhn-TTuce the recently introduced newral correlation function. Additionally, this paper also studies the KKT points of the neural correlation function for feed-forward networks with (Teaky) ReLU and polynomial (Leaky) ReLU activations, deriving necessary and sufficient conditions for rank-one KKT points.
(KKT) points of
1 Introduction
Neural networks have achieved remarkable success across various tasks, yet the precise mechanism driving this success remains theoretically elusive. The training of neural networks involves optimizing a non-convex loss function, where the training algorithm typically is a first-order method such as gradient descent or its variants. A particularly puzzling aspect is how these training algorithms succeed in finding a solution with good generalization capabilities despite the non-convexity the loss landscape. In addition to the choice of the training algorithum, the choice of initialization in these algorithms plays crucial role determining the neural network performance. Indeed, recent works have made incrcasingly clear the benefit of small initializations, revealing that neural networks trained using (stochastic) gradient descent with small initializationn exhibit feature learning (Yang & Hu 2021) and also generalize better for various tasks (Chizat et al. 2019 Geiger ct al. 2020 Woodworth et al.) 2020);; see Section 2 for more details into the impact of initialization scale. However, for small initializations, the training dynamics of neural networks is extremely non-linear and not well understood so far. Our focus in this paper is on understanding the effect of small initialization on the training dynamics of neural networks.
In pursuit of deeper understanding of the training mechanism for small initializations, researchers have uncovered the phenomenon of directional convergence in the neural network weights during the early phases of training (Maennel et al. 2018 Luo et al.] 2021). The authors of Maennel et al. (2018) study the gradient flow dynamics of training two-layer Rectified Linear Unit (ReLU) neural norksrks, and demonstrate that in the early stages training, the weights of two-layer ReLU neural networks converge in direction while their norms remain small. This phenomenon is referred to as early directional convergence. In recent work, Kumar& & Haupt 2024 further established early directional convergence for two-homogeneous neural
A neural network defined to I- [=(psiitively) homogeneous, if its output H(x; w) satisfies H(x;ew) cl vo. where the imput and a vector containing all the weights.

---

Published in Transactions on Machine Learning Research (03/2025)
networks, illustrating the importance of homogencity for such types of phenomenon (as opposed to the specific
choice of activation function or architecture).
To describe the phenomenon of early directional convergence for two-homogeneous neural networks, Kunar & Haupt (2024) introduced the notion of the Newral Correlation Function (NCF). For a given neural network and a vector, the NCF quantifies the correlation between the output of neural network and the vector; we provide more precise definition in Section 3 They show that in the early stages of training, the weights of the neural network remain small and converge in direction to a KKT point of the constrained NCF, an optimization problem that maximizes the NCF on unit sphere.
So far, all the works on the early stages of training dynamics of neural networks for small initialization have only considered two-layer neural nelworks (or more broadly, two-homogeneous networks). In contrast, our understanding for deep neural networks is much more limited. In the first part of this work, we establish early directional convergence for deep homogencous neural networks. We particularly consider L-homogencous neural networks, where L > 2, that have locally Lipschilz gradienls, and our main result can be summarized as follows:
In Theorem 1 we describe the gradient flow dynamics of deep homogeneous neural networks, with order of homogeneity strictly greater than two, trained with small initializations. Specifically, for square and logistic losses, we show that if the initialization is sufficiently small, then in the carly stages of training the weights remain small and either approximately converge in direction to a non-negative KKT point of the constrained NCF defined with respect to the labels in the training set, or are approximately zero.
It is worth noting that we assume neural networks to have locally Lipschitx gradients, which excludes deep ReLU networks. However, it does include deep linear networks (Arora et al. 2019a Saxe et al. 2014), and deep neural networks with diffcrentiable and homogencous activation such as polynomial ReLU x(x) == =max(x, 0)P for p 2 2 (Gribouval et al. 2022; Klusowski & Barron, 2018 Zhong el al. 2017 and
monomials (Livni et al. 2014; Soltanolkotabi et al. 2019). We also discuss the challenges in extending our
results for deep ReLU networks in Appendix D
The second part of our paper studies the KKT points of the constrained NCF for forwadd homogeneous neural networks. For such networks, we observe empirically that, along with converging towards KKT point of the constrained NCF during the early stages of training, the hidden weights also exhibit a rank one structure. Motivated by this observation, we mathematically characterize the rank-one KKT points of feed-forward homogeneous neural networks. Our second main result can be summarized as follows:
In Section 5 we derive sufficient conditions for a rank-one KKT point of the constrainod NCF for leed-forward neural network with activation function (<x) = max(æ, ax)),, for some p E N, a E R and arbitrary training data. We also provide matching necessary conditions under additional assumptions and validate these assumptions empirically. Our results also provide method to obtain rank-one KKT points of the constrained NCF using KKT points of smaller optimization problem.
2 Related Works
Understanding the training dynamics of neural networks has been the subject of numerous investigations tionss inchuding (Arora et al., 2019b Chizat & Bach 2020 Chizat et al. 2019 Jacot et al., 2018 Mei et al. 2019 These investigations have revealed two contrasting viewpoints depending on the scale of initialization. For large initialization regimes, the training dynamics of wide neural networks are effectively captured by a kernel referred to as Neural Tangent Kernel (NTK) [Jacot et al. (2018). In this regime, the weights remain close to initialization during training, and neural networks behave like their linearizations around their initializations {Chizat ct al., 2019).. On the other hand, in small initialization regimes, the training dynamics are extremely non-linear, and the change in weights during training is significant. (Chizat & Bach 2018 Geiger et al.) 2020 Mei et al. 2019 Yang & Hu 202I). Moreover, studies like Chizat et al. (2019); Geiger ct al. (2020) have obscrved improved generalization with decreasing initialization scale for various tasks, which makes understanding the dynamics of neural networks in the small initialization regime crucial for better understanding of practical success of deep networks.
2

---

Published in Transactions on Machine Learning Research (03/2025)
While considerable progress has been made on understanding the dynamics of neural networks in the large initialization regime, the theoretical investigations into the small ializatioo regime have been comparatively limited, primarily due to the extremely non-linear dynamics during training. Such investigations have
predominantly focused on linear neural networks (Arora ct al.) 2019a Gunasckar et al.] 2017 Woodworth
et al. 2020) and shallow non-linear neural networks (Chizat & Bach 2018 2020 Mei et al. 2019 Rotskoff &
Vanden-Eijnden, 2018). Even for these networks, a comprehensive understanding of the gradient descent. dynamics is further limited to diagonal linear networks (Woodworth al. 2020), shallow matrix factorization
(Jin et a 2023 Stöger & Soltanolkotabi] 2021), and two-layer (Leaky) ReLU networks under various
simplifying assumptions on the datasets (Boursier et al. 2022 Brutzkus & Globerson, 2019: Lyu et al. 2021
Min et al. 2024 Wang & Ma 2023). Our work could be the first step towards a greater understanding of the
training dynamics of deep neural networks.
The phenomenon of early directional convergence for small initializations was first investigated by Maennel ct al. (2018) for two-layer RcLU neural networks. This was later extended to tro-homogencous neural networks by Kumar& & Haupl (2024), which includes, in addition to (wo-layer ReLU neural networks, single layer squared ReLU neural networks and deep ReLU neural networks with only two trainable layers. Other empirical investigations have observed early dircctional convergence in some deeper networks as well. For example, Zhou et al. (2022) looks at the early training dynamics of three-layer ReLU neural networks and observe that weights converge along some specific orientations. Also, Atanasov et al. (2022) empirically show an alignment effect in the kernel of the neural network during early stages of the training. However, rigorous investigations thus far have been limited to shallow neural networks. Despite focusing solely on the early phases of training, these works demonstrate that interesting behaviors begin to emerge very early in the training dynamics. Morcover, these insights into the initial stages of training have been useful towards a comprehensive understanding of the entire training dynamics in some special cases (Boursier et al. 2022
Min et al. 2024 Lyu et al. 2021 Wang & Ma 2023).
3 Problem Formulation
Notation: For any N E N, we let [N] = 2,,,,,..,,N, denote the set of positive integers less than or equal to N. We let || ll2 denote the lg norm for a vector and the spectral norm for a matrix. For a vector Rh, 2, denotes its ith entry; z| = [|aa] [s2),... |2x and 2 = [>] 22. =g. =g]T, where q E N. We write a(t) =: dx(t), and for the sake of brevity we may remove the independent variable if it is clear from context. For any locally Lipschitz continuous function f: X + R, its Clarke sub-differential is denoted by 0f For a: : R and p E N, a-(-) denotes the function resulting from composing (-) with itself p times, and we assume (xxx)= x. We define a KKT point of an optimization problem to be non-negative (non-zero) KKT point if the objective value at that KKT point is non-negative (non-zero).
We adopt a supervised learning framework for training. where we assume {x,, fx,,0 Bi is the training dataset, and let X = [xi,..., Xn] E RRdx and = [y],..., WM]] E RR. For a neural nelwork H, its output is denoted by H(x; w), where x is the input and w E IR is the vector containing all the weights, and VH(X; w) denotes the gradient of H(x; w) with respect to w. We let H(X; = [H(x;; w)...., ..... H(xn; w)] e R" be the vector containing the output of neural network for all inputs, and J (X:w) R² + R" denotes the Jacobian of H(X; w) with respect to w.
Problem setup:
The following assumptions formalize the properties of neural networks considered in this paper.
Assumption 1. We make the following tripartite assumplion on the neural network function: (1) For any fized x, H(x; w) is locally Lipschitz and is definable under some o-minémal structure that includes polynomials and exponential? (ii) For all e > 0, H(x; ew) = cd ci'H(x; w), for some L> 2 (ie., the network is L-homogenouus. (iii) H(x; w) is differentiable in w and its gradient, VH(x;w), is locally Lipschitz.
We note that definability in some o-minimal structure is mild technical assumption and satisfied by all modern deep neural networks. Tt. allows us ensure convergence certain gradient flow trajectories (Ji Telgarsky 2020).
3

---

Published in Transactions on Machine Learning Research (03/2025)
In this paper, consider the minimization of
C ((w) Σ (H(x;:)),
(1)
where f(D. y) is the loss function, and V Vgl(d. y) is assumed to be locally Lipschitz in ÿ. which is sat- isfied by typical loss functions such as square and logistic losses. For z,y R", we define ('(z,y) Vyl(21,81)... Vgl(zn.Bn)) RR. We minimize eq. (1) using gradient flow, which leads to the following differential equation
(() = V w(0) = Swo.
(2)
where d is positive scalar that controls the scale of initialization, and wo is vector.
Following Kumar & Haupt (0244), for a fixed vector z I² and neural network H. the Neural Correlation Function (NCF) is defined here as
N₂ (w) = = H(::w)
and measures the correlation between the vector and the output of the neural network. The constrained NCF refers to the following constrained optimization problem
max N2, (w).
4
Early Directional Convergence
4.1
Main Result
The following theorem establishes the approximate directional convergence in the early stages of training L-homogeneous neural networks with small initializations.
Theorem 1.
Let be fixed unit vector. Under Assumption 1 for any E (0, n/2), where is positive constant, there erists Te, B, and 8 > 0 such that the following holds: for any 8 E (0,3) and solution w(t) of eq. 22 with initialization w(0) = Swo, we have
1| (t)ll2 S Be5, for all t E [0, [0,
Purther, for T. = Te/8k-2, either
[w(T))l2 2 6n/2, and u, (T..)///((T))]l 21 -1
where u, is a nen-nigative KKT point of
max N (w)
max (0, - HX, ||ul|==1
(3)
or [|w(T.))l2 S 28c.
The above theorem first shows that for given 6, we can choose d small enough such that for & [o, T the norm of the weights increases at most by a multiplicative factor. Thus, for small the weights will remain small for / E [0, T./L--6 Furthermore, since L > 2, for smaller the weights will stay small for longer duration of time.
Next, the thcorem describes what happens at T. = T,/51-2, the duration for which the norm of the weights remain small. There are two possible scenarios. In the first scenario, the weights approximately converge in direction along non-negative KKT point of eq. (3), which the constrained NCF defined with respect to y) and neural network H. Thus, the carly stages of training, the weights of the neural network converge in direction while their norm remains small. In the second scenario, the weights approximately converge to 0, which be observed by noting that ||w(T))]la & 2de, where choose to be arbitrarily
4

---

Published in Transactions on Machine Learning Research (03/2025)
small. This is in contrast with the first scenario, where ||w(Te)l2 2 ón/2 and 17 is constant that does not depend on 8. This shows that in the second scenario, compared to the first scenario, the weights become much smaller. The second scenario arises due to the convergence of the gradient dynamics of the NCF towards O; see the proof for more details. We also note that for a fixed training data and neural network, Wo determines which of the above two scenarios will occur. In addition to this, the positive constant v) in the above theorem is also determined by Wo and independent of
It can be shown that for square loss, ((0,)) t - y)² and (f, y, and for logistic loss, (().)) In(1 + e b-) and -6" (0, y0.
y/2. Therefore, for square and logistic loss, the objective function of the constrained NCF in eq. 33) will be y H). We also study KKT points of the constrained NCF for feed-forward neural networks in Section 5 which some readers may find helpful.
Overall, the phenomenon of early directional convergence in deep homogeneous networks similar to two- homogencous networks (Kumar Haupt, 2024, Theorem 5.1), except for one difference. The amount of time required for directional convergence in deep homogeneous networks has an additional factor of 1/84-2, Thus, directional convergence takes longer as depth increases.
4.1.1 Proof Sketch of Theorem 1
We next provide a brief proof sketch of the above theorem. We begin by describing the (positive) gradient flow dynamics of the NCF.
For given vector z and neural network H. the gradient flow resulting from maximizing the NCF is
u = ((), u(0) up
(4)
where ug is the initialization. Note that the NCF may not be bounded from above. Therefore, gradient flow may potentially diverge to infinity. However, the following lemma shows that the gradient flow either converges in direction to a KKT point of the constrained NCF, or converges to 0.
Lemma 2. Under Assumption 1. for any solution u(t) of eq. one of the following two possibilities is true.
1. There erists some finite T such that lim [u(()]l2 oo. Further, lim,-r- u(t)/|u(()ll exists and is equal to non-negative KKT point of the optimization problem
max (2, = =z =(;:u)
(5)
2. For all t > 0, |lu(O]l2 00, and either lim;-ro u(t)/lu(tlll erists or lim u(() = 0. If lim m(t)/|lu((]]la evists then it is equal to non-negalive KKT point of eq.
To prove the above lemma, require the neural networks to be definable with respect to o-minimal structure. Such a requirement allows us to use the unbounded version of Kurdyka-Lojasiewiez inequality proved in Ji & Telgarsky (2020) for establishing directional convergence. A similar result was demonstrated in Kumar& & Haupt (2024) for two-homogeneous networks. However, in that instance, the solution u(t) remains finite for all finite time (see Lemma 18 whereas in our case, u(t) may become unbounded at some finite time.
Also, for our proof, we had to derive the rate at which u(!) grows, showing that, if utt) becomes unbounded, and for all near T*.
then |u(()]l2 2 x/(T" 1)1/(2-)), for some
We next explain our proof technique for Theorem 1 assuming square loss is used for training. The evolution of w(t) is governed by the differential equation
(u H(x;; w)) VH((I;; w) J(X:)) Ty ((V-H(X;))) w(0) Swo,
(6)
where (as mentioned) 8 >>0 0, and recall J(X; w) R Ra is the Jacobian of the function H(X; w) with respect to w. We define s(t) f (コz=z); then, s()) wo- Further, by using the T--homogeneity of H (X; w)
5

---

Published in Transactions on Machine Learning Research (03/2025)
and (L- hymmogggyyy of J(X: w) (Lemma
can derive the evolution of s(t) as follows
ds d t
dt dt (
(8L-2)
82-1 W
82-2
))
T
t
J
X;w
y y- H
X; w
82-1
86-2
8L-2
J
X;
8
1
w
82-2 (1=2) T
))
y H¹HH1
X: w
8L-2
t
Overall, this implies
s = J (X:s)
- 0 (X:s), s(0) = wo-
(7)
Now, we compare the dynamics for s(t) in eq. with respect to y and neural network H.
7 with the dynamics of gradient flow of the NCF defined
ú == VNy,H(u) J(X:u) u(,)
u(0) Wo.
(8)
Observe that if 5 0, then the two dynamics are exactly the same, and since s(0) = Wo u(0), we have that s(I) and u() will also be same. For ō >> 0, there is an extra term 8-H (X:s) present in eq. 77. For small 6, we show that this extra term is small and while s(t) and u(t) are not exactly the same, the difference between them is small, i.e., ||s(t)- - u(t)ll2 is small. Moreover, we show that the difference can be made sufficiently small for a sufficiently long time by choosing d small enough. We combine this fact with Lemma 2 to prove Theorem 1 Specifically, from Lemma 2 we know that u(() will either converge to 0 or converge in direction to a KKT point of the constrained NCF. Therefore, we first choose a time (say T) such that u(Z) has approximately converged in direction to a KKT point of the constrained NCF or is in the neighborhood of O. Then, we choose ō sufficiently small such that [|s(() - u(O]l2 is small for all E [0, 7). Since s(l) = f (x/z), we have that I (f==x) - u(2 is also small. Finally, since is a positive scalar, and u(T) approximately converges in direction to a KKT point of the coistrained NCF or is approximately 0, it follows that w (xd²) also approximately converges in direction to a KKT point of the constrained NCF or is approximately 0.
As a quick aside, we note that there are two key aspects in the definition s(t) = 1 w (zr=r). First, we have divided by d. We know that å controls the scale of initialization and assumed to be small. Dividing by 5 removes that scaling and makes s(t) constant scale vector. More importantly, dividing by 5 does not alter the direction, i.e., the direction of s(l) and w(l) are always the same. Thus, establishing directional convergence of s(t) will also imply directional convergence of w(t). Second, we rescale the time by 5"-2. To understand the importance of this time rescaling, let us look at the magnitude of the gradient of the loss at initialization in terms of 8. Since 8 is small and Wy has unit norm, [X] w(0)) (y - H(X; w w())) ll2 = [X;; (()) T yl12 = O(84-1), where in the last equality we used (L. - 1)-homogeneity of the Jacobian. We observe that for initialization of scale 8, the gradient scales as 8--11, Thus, in the early stages the gradient will have negligible effect the weights, in terms of both magnitude and direction. Morcover, based on the relative scaling of the gradient and the initialization, the gradient will only start impacting the weights after 0(188-22 time has elapsed. Hence, for smaller initialization, the gradient flow will take longer to make an impact on the weights. Thus, in a way, rescaling time by 8'-2 brings the time of s(t) to "constant scale". We also highlight the importance of homogeneity. It allows the above two changes to interact nicely with the gradient flow and results in dynamics of s(t) that are essentially independent of 6, provided 8 is small.
Finally, in comparison to Kumar& & Haupt (2024), which studies early directional convergence for two- homogeneous neural networks, the main conceptual innovation in this paper is the time rescaling by 81-2, In fact, the proof in that work relies showing that the difference between w(t//8 and u(t) remains small for a sufficiently long time. To see motivation for this, note that we can get eq. 7 from eq. 6 for L= 2 as well. However, for L = 2, the time rescaling factor 84-2 will become unity, yielding s(l) = (1)/6. Now, since the gradient flow dynamics for the NCF of pro-homogeneous neural networks also converges to a KKT point of the constrained NCF, to establish carly directional convergence one must show that the difference between w() and u(t) remains small for a sufficiently long time. Another notable difference with Kumar & Haupt (2024) is in the gradient flow dynamics of the NCF. As mentioned previously, from Lemma we know that.
6

---

Published in Transactions on Machine Learning Research (03/2025)
1.0
1,000
0.9
0.1809
Normaitedd Loss
01.20
0.9889
Normollaad Norm of weighte
TA (411- U.7
00.13
08
0.0844
0.5
0.990
00.00
0.1
10000
20000
30000
10000
5000
0
1000
20000
3000
20000
2000
No. of iterations
No. of iterations
(a)
(b)
Figure 1: Early training dynamics of a squared-ReLU neural network. Panel (a): the evolution of training loss (normalized with respect to loss at initialization) and the fz-norm of all the weights with iterations. Panel (b): the evolution of V.((w()) (L)) (()//|VM((())) (a measure of directional convergence of w(!)). Note that the loss has barcly changod, and the norm of weights has increased by a constant factor but remains small. Also, the weights approximately converge in direction to a KKT point of the constrained NCF.
for L> 2, the gradient flow dynamics of the NCF can become unbounded at some finite time, whereas for L = 2, the dynamics remains finite for finite time. This difference necessitates a more careful analysis of the early stages of training in order to ensure directional convergence for deep homogeneous neural networks.
Challenges for non-differentiable neural networks: Theorem | holds for neural networks with locally Lipschitz gradient, and thus excludes ReLU neural networks. We briefty describe the challenges in extending Theorem D to ReLU networks in Appendix D which also includes an empirical evidence that early directional convergence also occurs in ReLU networks.
4.1.2 A Closer Look at the Weights
In this section, we use a toy example to highlight the emergence of additional structural properties in the weights during the early stage of training that are not explained by Theorem 1
We train the following h-homogeneous neural network H(x; v, U) = v max(Ux, 0)², where 2 and U E 20 by minimizing the square loss with respect to the output of a smaller neural network H" vx, U.) v max(U.x, 0)2, where the entries of v. E 2 U. E 2x are drawn from the standard normal distribution. The training data has 100 points sampled uniformly from unit sphere in R Let w be a vector containing all the entries of and U, and w = w/l|//2. We train for 50360 iterations using gradient descent with step-size 2- 10-2 2 and the initialization w(0) åwo, where d = 0.05 and wo random unit norm vector. Figure Ia| depicts the evolution of training loss and E 22-morm of the weights. Note that the loss almost remains the same, and the norm of the weights has increased but only by a constant factor and is still small. A unit norm vector is a KKT point of the constrained NCF if the vector and the gradient of the NCF at that vector are parallel (see Lemma [1]. Hence, to measure the directional convergence of w to the KKT point of the constrained NCF, in Figure 1b we plot the evolution of the inner product between w(t) and V.V Thcorem the weights approximately converge direction to a KKT point of the constrained NCF.
where N/(:) denotes the NCF for this example. Clearly, accordance with
We next take closer look at the weights of the neural networks at iteration 50360, i.e., when the weights have approximately converged in direction to KKT point of the constrained NCF. In Figure 2 we plot the normalized absolute value of the weights at initialization and at itcration 50360. At initialization the weights appear to be random, as expected, but at iteration 50360, they are more structured. The hidden weights (U) appear to be of rank-one as only one row is non-zero, and the outer weight (v) has a single non-zero entry. This suggests that weights converged towards KKT point of the constrained NCF that have low-rank hidden weights. Similar behavior observed with deeper networks and other activation functions as well (see Section 5.3) and has also been observed in some previous works [hoou et al. 2022 Atanasov et al. 2022
7

---

Published in Transactions on Machine Learning Research (03/2025)
0 //ww]
[0] w
1L12
2.120
033
0.5
10
LOGS
10
10
9.605
0.2
6.1
(a) initialization
(b) At iteration 50360
Figure Normalized absolute value of the weights different stages of training.
Since the constrained NCF is a non-convex problem with potentially multiple KKT points, an important question arising from the above observation why gradient descent converges towards low-rank KKT points. A more basic question, addressed in the next section, is whether low-rank KKT points always exist for the constrained NCF, and can they be characterized? Given the complex nature of the problem, their existence is not immediately clear, especially for deep neural networks. We next study the KKT points of the constrained NCF for feed-forward neural networks with Leaky ReLU and polynomial Leaky ReLU activation functions. For arbitrary training data, necessary and sufficient conditions are derived for a set of weights to a KKT point of the NCF that have rank-one hidden weights.
5 KKT Points of Constrained NCF
This section presents necessary and sufficient conditions for set of weights to be KKT point of the constrained NCF of feed-forward neural network that have rank-one hidden weights. For given input x, the output of an L-layer feed-forward neural network H is defined as
H(x: WI, Wt) W. (W 1-1 (WIx) ...) E R,
(9)
where W, R are the trainable weights, and ko = d and k = 1. The activation function a(-) is applied coordinate-wise to vectors, and in this section is assumed to be of the form axx) = max(ax, z)", where E N and a E R, which includes deep ReLU and polynomial ReLU neural networks. The corresponding constrained NCF is the following optimization problem:
max N(W,,... W.) = y T(X; Wi,... W.), s.L. | + Wi W
| =.
(10)
Note that, we have considered the NCF in the above equation with respect to y for convenience, it may be replaced with any other vector.
5.1 Leaky ReLU
We first consider the activation function of the form o(x) max(æ, ax), for some IR.
Theorem 3. Let H be an L-laage feed-forware neural network as defined in eq. where 2 and
(z) = max(a, ax) for some a E R. Let W LLL defined as
where |ail2 = |bill2, for all I E [L-1]] 1]. W = (W,.... WA W.) (a,bT a aT. (,.,b,,..T))) wT),
(i) Suppose + and the entries of (a (b:) non-negative, then WI. is non-zero KKT point of eq. (10) if and only if the following holds: |aill 1/VL, , for all 1 E [L 1], | = 1/L, , for all [T- 2], and qa T.1/W, for some q (-1,1). Further, by is non-zero
8

---

Published in Transactions on Machine Learning Research (03/2025)
KKT point of
max 1/i0 1 (x,, (x, u), such that | -1/T..
(11)
(i) Nexl, suppose a = then W WI.F. 1.I. a non-zero KKT point of eq. (10) if and only if the following
holds: ||a|| L/VL, for all 1 E [L - 11, w 1/L, qıa: bi+1,, for all l [L 2], and
q1-1ar-1 =L L where qi E 11, Further, bj is non-zero KKT point of
max 1
y:o 1- (x.'1) 1x, u), such that I =1VTT.
(12)
In the above theorem, {a, b, are vectors used to build the rank-one hidden weights. In both cases, we first provide the norm of these vectors. Next, for a * 1 and assuming the entries of are non-negative, we present additional nocessary and sufficient conditions, which can be divided in two parts. First is the alygmment: a = =bbb, for all E [L - 2], and qar,- L,'"w, for some q E (-1,1), which implies that the column-space of W, and row-space of W must be aligned. The second part requires b to be KKT point of eq- [11], a smaller optimization problem, where note that gl-=(()) denotes the function resulting from composing a--) with itself (L-1) - 1) times. The results for require any non-negativity assumption.
{b:)
1 are mostly similar, but do not
5.2 Polynomial Leaky ReLU
We next provide necessary and sufficient conditions for activation function of the form o(x) max(ι, ar)?,
for some E I, 2 2 and N. In the following theorem, for a given p, we define ò = p1-11 + +p'-2 2 +...
Thcorem 4. Let H be an L-layer feed-forward newral network as defined in eq. 9 where L 2 2 and
(x) = max(r, )x)) for some a E R p 2 2 and E N. Let Wv.t. be defined as
W W...: (WI, ..... WI-1, Wt. (a a 1b- wT), where ||il|2 = [b:ll2, for all l [L 1].
(1) Suppose + and the entries of (ai (br)t=² are non-negalive, then W W.L.. is a non-zero KKT
point of eq. (10) if and only if the following holds: ||a [1 p--1/B, for all l E [L 11, | 1/p,
b, = -/1/ 7 1/7 for 2 < 1 S L 1, and w (pp) 1/4 for some q {-1,1}. Further, for
1 < < L - each non-zero entry of a has identical values, and b is non-zero KKT point of
max
y:0 -- (x, u), such that ||u|22 pL--1/p.
(13)
(i) Suppose a 1, then W 1:L. is non-zero KKT point of eq. 10 if and only if the following holds: 1], W 3 1/8, b, q/ w (La)_1/(pp) if is odd,
||a 12 = p1-1/p for all I 1 (/pp) if p is even, where qu E (),,1), for all 2 < 1 < L. and bj = - 1 1/1//4 w = qr/ar.-1l
Further, for < 1 < L - 1, each non-zero entry of a has identical absolute values, and bj is a non-zero KKT point of
max
y:o (x, u), such that ||u|| -l---pee /D--1/p.
(14)
Overall, the results are mostly analogous to Theorem with one key difference: the non-zero entries of {ar) must have identical (absolute) values. This difference is also evident in Figure 3 which depicts KKT points of 3-layer neural network with ReLU and squared- ReLL activation. For ReLU, in contrast with squared-ReL.U, multiple rows and columns of the weights are non-zero. Additionally, the KKT point exhibits sparsity, with ReLU activation having multiple rows and columns with zero norm, while squared-Red ReLL activation has only one non-zero and column. We observe this difference consistently in our experiments.
9

---

Published in Transactions on Machine Learning Research (03/2025)
W
W
12
111
(a) ReLU activation
[W/]
W
0.30
0.25
(b) Squared-Rele ReLU activation
Figure 3: Absolute value of a KKT point WW,, 22, W of eq. 10 for ReLU activation (max(z; 0)) and squared-ReLU activation (max(2, 0)²), where L = 3, 100 and x ~ia N00 I1o), yi PLaa W 1). For ReLU, the non-zero rows/com rows/columns are scalar multiple of each other, implying the hidden weights have rank one. For Squared-ReL.U, it is clear that the hidden weights have rank one.
The results of Theorem g and contain sufficient conditions which imply that KKT points of the constrained NCF with rank-one hidden weights always exists for arbitrary training data or depth of the neural network. The sufficient conditions also provide approach to obtain KKT point of the constrained NCF using KKT points of smaller optimization problem. Additionally, they also contain matching necessary conditions, where, for a + 1, we had to further assume that (a/)/-1 [/r have non-negative entries. This implics that if, for # 1, there is KKT point with rank-one hidden weights such that {a) {b/ have non-negative entries, then it must be of the form described in the above theorems. Interestingly, in our experiments with random training data (Section 5.3 we found that the KKT points found via gradient-based method generally satisfy this non-negativity assumption, along with having rank-one hidden weights.
Remark 1. In the above theorems, me focus on non-zero KKT points, because zero KKT points may not have a rich structure. For instance, using Lemma 11 it can be shown that if the network's s output is always zero, the corresponding weights are zero KKT points. For ReLU activation, the output is zero if the imput is non-posio posipoiiiiee. Thus, for an L-layer ReLU network, the output will be zero if any matria in (W2...., WL-1) has non-positive entries, making those weights zero KKT points.
5.3 Numerical Experiments
In the above theorems, we have made two main assumptions on the KKT points of the NCF (WI, Wr)):
Rank-one hidden weights. W, abb where ||a/||2 |bblll. for all E [L-
2. Non-negative weights. If (x) = max(æ, az) and a + 1, then (au)/11 L-1 (bi)
L-1
are non-negative.
To validate these assumptions, we conduct numerical experiments for two and three layer neural network with activation function (z) = max(ax; where p (1, 2] and a E (0,0.1, For our experiments, we generate random set of training data, and then obtain KKT point of the corresponding constrained NCF by maximizing the NCF via gradient ascent with random initialization more details are provided in the
SSince gradient ascent bocome unbounded, use projected gradient ascent, i.c., after every gradient ascent stop normalize the weights have unit norm. However, also turns that for homogeneous objectives, projected gradient with fixed siep equivalent gradient ascent with adaptive step-size, up to scaling factor (see Lemma 19..
10

---

Published in Transactions on Machine Learning Research (03/2025)
a 0=
a 0.1
a
Q =
a 001 0.1
a
=
5.93
10-9
1.79 10-
2.07 10-3
= 1
2.97-10-14
1.22-10
1.9 10
=
1.76-10-1 10
1.67-10-10
6.86-10-11
= 2
1.48.10
1.12-10
7.93-10
Threo-layer
Two-layer
Table 1: The left table contains the maximum value of <(W 1. W2) across 30 different random instances for three-layer neural network with activation function max (z, ox)), where (W1, W2, Wa) is the KKT point of the constrained NCF obtained via gradient ascent. The right table is similar but for two-layer neural network.
0
0.1
0
0.1
p
4.44 10
3.28 10
p=1 1
1.82 10-7
2.77 10"
p =
5.16 10-6
5 10-6
p =
3.95
10-7
3.58 10-7
Three-layer
Two-layer
Table Suppose (WI: I..... W.. is the KKT point of the constrained NCF obtained via gradient ascent for an L-layer neural network, and let a, be the rank-one approximation of W, such that || lall22 | for all E [L 1]. The left (right) table contains the maximum value of p(a,,8gb) , - across 30 different random instances for three-layer (two-layer) neural network with activation function max ar)". Note that if arb, is non-negative, then we can choose aj and by such that they are also non-negative. Hence, showing a,, ... a....a..b,...................b.. are non-negative is equivalent to showing {ar)=a) {bi LL-1 are non-negative.
The results presented below are for 30 such random instances. Also, discussed earlier, the gradient flow of the training loss converges to the same KKT point as the gradient flow of the NCF. Thus, using gradient ascent. to find the KKT point of the constrained NCF also provides insights into the KKT point towards which gradient flow of the training loss converges in the early stages of training.
Appendix C.4
Rank-one assumption: For any set of matrices (Zi.... Zpt, we use
|Z, Z
R( ZZ, -p) =: max 15i5p
1
measure how close are each of (Z...... -.p. being rank-one. If <zZz, Zp)) = 0, then each of (Z,.... Ip rank one, and thus, for smaller s=Z,, Zp), they approximately rank one. Also, s(.) is related to the notion of numerical rank (Rudelson & Vershynin 2007), stable relaxation of the rank. Now, from the values in Table | we observe that for both two-layer and three-layer neural network, and with different activation functions, each of the hidden weights in the KKT point obtained via gradient ascent are approximately of rank one in all the instances considered (since we are reporting the maximum value of k(-) across all the random instances.)
Non-negativity assumption: For any set (Zi.-.. Zp containing matrices and/or vectors, we use
max (000-.Z2)1 -Z:)
p(Z -p) =: max SStSp
|Z,
measure how close are each of (Z,...., Zp} to being non-negative. Note that, Z. is vector, then |Zel|r F denotes its & norm. If p(Z,.... ... .z,) = 0, then each of (Z,.... Zp} non-negative, and thus, for smaller p(Zi,... Zp), they are approximately non-negative. From the values in Table 2 we observe that for both two-layer and three-layer neural network, and with different activation functions, the appropriate set of weights in the KKT point obtained via gradient ascent are approximately non-negative in all the inslances considered. Note that, we did not consider linear or monomial activation (a(x) = x²) since in those cases we did not make any non-ncgativity assumption.
11

---

Published in Transactions on Machine Learning Research (03/2025)
We conchude this section by again noting that under the rank-one and non-negativity assumption considered here, the sufficient conditions of Theorem B and 4 are also necessary conditions, and thus, the KKT points must be of the form described in those results. Since the rank-one and non-negativity assumption is satisfied for all the instances considered, it implics the KKT points are of form described in Theorem 3 and for all the instances considered. Also, note that the constrained NCF a non-convex problem and could have KKT points that do not satisfy these assumptions, but gradient ascent seems to avoid them. Understanding this phenomenon is an interesting future direction, which would require analyzing the dynamies of gradient ascent.
5.4 Proof Overview
We now provide a brief overview of the proof technique for the above theorems; the details can be found in Appendix [
We start with the following lemma, which is important for our proof.
Lemma 5. Let H be an L-layer leed-forward neural network whose oulpul is as defined in eg. where I, 1 2. Suppose (Wi...", WL) is a non-zero KKT points of the constrained NCF in eq. 10 1f ((x) max(ax, x)P, for some p E N and a E R, then
diag (W..)) - pp- ddiag W (WI1 WHtt), for all l E EL 1],
(15)
and, if (zz) x, then
W.W, = WH4 1+1 W441, for all l E [L- 1].
Furthermore, if [ 1. W L-11 W L))= b, b w where |ayl| ||bull,, for all [L-1] 1],
then lar] p² 2 b141, 11, for all l E [L 2}, ar.-1l = (pp)2/4/w) ||rll = ----1// for all I E [L- 1], and
| 1/p, where == p L
The above lemma implies |ay is parallel to |b This is the first step towards establishing the alignment conditions stated in the above theorems. Now, a and 1 are non-negative, which true in certain cases, we get a is parallel to b We similarly use additional conditions to get alignment results for other cases.
We next describe how to show bi is a KKT point of the smaller optimization problem. For this, we focus deep ReLU networks and assume that the alignment conditions are true, the proof of other activation functions follow similar approach. From the KKT conditions, we know W1 must satisfy
0 DE - Ow: (y H(X: WW)) = daib, + Ow. (y H(X; WI, WW.).,,)..
(16)
Since (= [b)1 have non-negative entries, ReLU is positively homogeneous, ||a:112 1/VL, and a = b141, for all 1 E [L 2], fa I,1/4,, can write
H(x; W1, W.... W.) Wao W 1 W2 (Wix)..)
qa 1 (a _ 20 (a, (az-aaa_aaaaa _a o (aza a
... (W.x))...)/TT1
-llaa-- ||a-- 1 ... |a 12 2000022aaa (TTo(
- (1(V/
L-2 of -- [ o (W
We use the above equality to simplify ow, (y H(X; WI.... WD)) and use it in eq. (66) to prove b is KKT point of the smaller optimization problem. Also, note that since ReLU activation is non-differentiable, we used the Clarke sub-dilferential in the KKT conditions. A challenging aspect of using Clarke sub-differentials is that the chain rule does not hold with equality, instead it yields a set and the true sub-differential belongs to that set (sce Lemma 15). Consoquently, it not feasible to simplify Ow, (y H(X: W,. W,)) using the above equality by simply applying the chain rule. We follow a more direct approach to simplify the Clarke sub-disferential, which uses positive homogeneity of the activation function and non-negativity of a
12

---

Published in Transactions on Machine Learning Research (03/2025)
6 Conclusion and Future Directions
In this paper, we studied the gradient flow dynamics resulting from training deep homogeneous neural networks with small initializations, and established the approximate directional convergence of the weights in the early stages of training. However, these results are not applicable for ReLU networks, so one important future direction would be to rigorously establish similar directional convergence for ReLU networks.
We also derived necessary and sufficient conditions for the KKT points of the constrained NCF that have rank-one hidden weights, for ferd-forward neural networks with Leaky ReLU and polynomial Leaky ReLU activation functions. The constrained NCF is non-convex problem and all KKT points may not be low-rank; however, our experiments suggest that gradient ascent tends to converge towards low-rank KKT points. Understanding this phenomenon is also an important area for future exploration, which will also help us understand why gradient descent on training loss converges towards low-rank KKT points of the NCF in the early stages of training. We hope our characterization of rank-one KKT points will aid in this exploration.
Acknowledgments
The authors graciously acknowledge gift funding from InterDigital, which partially supported this work.
13

---

Published in Transactions on Machine Learning Research (03/2025)
References
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Aduances in Newrll Information Processing Systems, 2019a.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, 2019b.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent aligmment effect. In International Conference on Learning Representations, 2022.
Pascal Bianchi, Walid Hachem, and Sholom Schechtman. Convergence of constant step stochastic gradient descent for non-smooth non-convex functions. Set- Valued and Varialional Analysis, 30(3), 2022.
Jérôme Bolte and Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning. Mathematical Programming, 188, 2021.
Etienne Boursier, Loucas Pillaud- Vivien, and Nicolas Flammarion. Gradient flow dynamics of shallow reLU networks for square loss and orthogonal inputs. In Advances in Newral Information Processing Systems, 2022.
Alon Brutzkus and Amir Globerson. Why do larger models generalize better? A theorctical perspective via the XOR problem. In Proceedings of the 36th International Conference on Machine Learning, 2019.
Lénaïc Chizat and Francis Bach. On the global convergence of gradient descent for wver-parameterized models using optimal transport. In Advances in Newral Information Processing Systems, 2018.
Lénaïc Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Proceedings of Thirty Third Conference on Learning Theory, 2020.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. In Advances in Newral Information Processing Systems, 2019.
F.H. Clarke. Oplimization and Nonsmooth Analysis. Wiley New York, 1983.
Mario Goiger, Stefano Spigler, Arthur Jacot, and Matthicu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Slatisticul Mechanies: Theory and Experiment, 2020, nov 2020.
Rémi Gribonval, Gitta Kutyniok, Morten Nielsen, and Felix Voigtlaender. Approximation spaces of deep
neural networks. Constructive approzimation, 55(1):250 367, 2022.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. In Advances in Neural Information Processing Systems, 2017.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems, 2018.
Ziwei Ji and Matus Telgarsky. Directional convergence and aligument in deep learning, In Advances in Newral Information Processing Systems, 2020.
Jikai Jin, Zhiyuan Li, Kaileng Lyu, Simon Shaolei Du, and Jason D. Lee. Understanding incremental learning of gradient descent: A fine-grained analysis of matrix sensing. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Jason M. Klusowski and Andrew R. Barron. Approximation by combinations of relu and squared relu ridge
functions with and e controls. IEEE Transactions on Information Theory, 64(12):7649 7656, 2018.
Akshay Kumar and Jarvis Haupt. Directional convergence near small initializations and saddles in two- homogeneous neural networks. Transactions Machine Learning Research, 2024.
14

---

Published in Transactions on Machine Learning Research (03/2025)
Roi Livni, Shai Shalev-Shwarte and Ohad Shamir. On the computational efficiency of training neural networks. Advances in neural information processing systems, 27, 2014.
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks at infinite-wied width limit. Journal of Machine Learning Research, 22(71):1-47, 2021.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020.
Kaifeng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. In Advances in Neural Information Processing Systems, 2021.
Hartmut Macnnel, Olivier Bousquet, and Sylvain Gelly. Gradient descent quantizes relu network features. arXiv preprinl arxiv:1800 803. 083668 2018.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Mcan-field thcory of two-layers ncural networks: dimension-free bounds and kernel limit. In Proceedings of the Thirly-Second Conference on Learning Theory, 2019.
Hancheng Min, Enrique Mallada, and Rene Vidal. Early neuron alignment in two-layer reLU networks with small initialization. In The Twelfth International Conference on Learning Representations, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advonces in Neural Information Processing Systems 32. 2019.
Grant M. Rotskoff and Eric Vanden-Eijnden. Trainability and accuracy of neural networks: An interacting particle system approach. ArXiv180500915 Cond-Mal Stal, July 2018.
Mark Rudelson and Roman Vershynin. Sampling from large matrices: An approach through geometric functional analysis. Journal of the ACM (JACM), 2007.
A Saxe, J McClelland, and S Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference on Learning Represenatations 2014, 2014.
Mahdi Soltanolkotabi, Adel Javanmard, and Jason D. Lee. Theoretical insights into the optimization landscape of pver-parameterized shallow neural networks. IEEE Transactions on Information Theory, 65(2):742-769, 2019.
Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin to spectral learning: Opti- mization and generalization guarantees for overparameterized low-rank matrix reconstruction. In Advances in Neural Information Processing Systems, 2021.
Mingze Wang and Chao Ma. Understanding multi-phase optimization dynamics and rich nonlinear behaviors reLU networks. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes overparametrized models. In Proceedings of Thirty Third Conference on Learning Theory, pp. 3635-3673 2020.
Greg Yang and Edward J. Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Kai Zhong, Zhao Song, Prateck Jain, Peter Bartlett, and Inderjit S Dhillon. Recovery guarantees for one-hidden-layer neural networks. In International conference on machine learning, 2017.
Hanxu Zhou, Zhou Qixuan, Zhenyuan Jin, Tao Luo, Yaoyu Zhang, and Zhi-Qin Xu. Empirical phase diagram for throe-layer neural networks with infinite width. In Advances in Neural Information Processing Systems, 2022.
15

---

Published in Transactions on Machine Learning Research (03/2025)
A
Key lemmata
In this section, we state some key lemmata that are important for our results.
A.1
Kurdyka-Lojasiewice inequality for definable functions
The Kurdyka-Looasiewiiz Inequality is useful for demonstrating convergence of bounded gradient flow trajectorics by establishing the existence of desingularizing function, which is formally defined as follows.
Definition A.1. A function V [0, + is called a desingularizing function when continuous on (0,z) with (0) 0, and it continuously differentiable on (0,v) with Y 0.
In establishing the directional convergence of gradient flow trajectories of the neural correlation function, where the trajectories may not be bounded, we will utilize the following unbounded version of the Kurdyka- Lojasiewicz inequality.
Lemma 6. [ji Telgarsky, 2020, Lemma 3.6) Let f be a locally Lipschitz function, definable under on o-minimal structure, and having an open domain D {x|[|x||2 > 1}. For any e, n O, there erists v > and a definable desingularizing function & on (0,v) such that
('(())) (((xx)))x(|lllVV((())) 1, if [(x) E (0, and V f())ll 2
where v.J(x)= = [x((), x
x and V f() = V -x)(x))
A.2 Euler's Theorem for homogeneous functions
The next lemma states two important properties of homogeneous functions.
Lemma 7. (Lyu Li, 2020 Theorem B.2), [Ji & Telgarsky, 2020, Lemma C.1)) Let F R IR. be locally Lipschilz, differentiable, and L posilively homogeneous for some L O. Then,
1. For any w E R and c 2 0,
VF(cw) = vF()..
2. For any E R
VF(w))=FF( = LF(w).
A.3 Gronwall's inequality
In our proofs, we will frequently use Gronwall's inequality. Lemma 8. Let a, 8, u be real-valued functions defined on an interval [a, b), where 8 and u are continuous and min(a,0) is integrable on every closed and bounded sub-interval of [a, 6).
3 is non-negative and if satisfies the integral inequality
VI u(t) S a(1)+ + [ W
then
a(t) < a(t) + / ((s))((s) exp ([((i))a) ds, Vt [J. Tf. in addition, the function is non-decreasing, then
ml() < a(1) exp ( (())ds) vi [a,
We will mainly be using the second part of the above lemma.
16

---

Published in Transactions on Machine Learning Research (03/2025)
B Proofs omitted from Section 4.1
B.1 Proof of Lemma
2
Recall, for a given vector z and a neural network H. the NCF is defined as
(u) = z ((;
(17)
In this scction, for the sake of conciseness, we will use Mu)) in place of (u) Further, u(t) satisfics for all i
du VV(u) = J(:uu) u(0) H1
(18)
dt
where [(X;u) IR + R" the Jacobian of the function u)) Let S denote the unit sphere on R* and define
3 sup(]]H((Xw))) w)]2 E s* and a Sup(ll]((X w)ll2: w Sk-11.
(19)
We will start by establishing some auxiliary lemmata. The following lemma follows from L-homogeneity of ((u) and Lemma 7
Lemma 9. For any u u R and c 2 O, T LN(u) and (N(cu)) = cN-),,( and J(X; u)u u) and -e
=
We next define (u), which plays integral part in the proof, and its gradient.
Lemma 10. For any nonzero u Re we define N(u) = -N(u)/llu ([u)/||ull2 then,
V.(()) V.((u) LN(u)u
(1 uu T VW(u)
u
n 1ul
Proof. The first equality follows by differentiating M()) with respect to un using Lemma we get the second equality.
We next describe the conditions for first-order KKT point of the NCF,
Lemma 11. If vector u. Rx1 is first-order KKT point of
max N(u) = = ((X; u),
(20)
1 ==
then
= z= Xuu,, =
(21)
where X" E IR is the Lagrange multiplier. Also, LN(u,) = X², which implies that X 2 O for non-negative KKT point.
Proof. For eq. 20 the Lagrangian is
L(u, A) = N(u) + X[[
Hence, u, is a first-order KKT point then ||u. 112 1, and for some we have
VA(uu.) + 2āu. 0.
which implies
TT) 2 2Tu, = 0. -2X. Further, by Lemma we have
We get eq. 21 by choosing
=== TVN(u.) LN(u)).
17

---

Published in Transactions on Machine Learning Research (03/2025)
Lemma 12. For any t 2 0, we have
du(u) IVV |((0))
(22)
di
Proof. Using i = (W(((t)), we have
dV(u)
= v.((((()) [V(((())] |
dt
Lemma 13. For any to 2 0, if ((((tt)) 0, then, for all to, we have N(u(t)) 2 NN(((f)) and [|u(t)]22 2 [u((t)]22.
Proof. Using Lemma 12
have
N(u(t)) - (i((())) /" [[ú(s)]Bds.
Therefore, for 1 to: NV((()) 2 (u((t)) 0. The second claim true since for all 2 0
duul
2u T ú
2u VW(u) = 2LN(u).
dt
which implies
[u() - [1x()))] = 21 / NN(u(s))ds 2 O. to
We now turn proving Lemma 2
Proof of Lemma We begin by showing that either the limit of u(t)/||u(t]ll exists or u(t) converges to 0. For this, consider two cases.
Case 1: ((u(0)) > 0.
In this case, we first show that u(t) becomes unbounded at some finite time. Suppose for the sake of contradiction [|n(()]l2 < oo for all finite 2
Let (u((0)) 7 0, thus |u(00 la 0. Using Lemma 13 for all t2 0,
[ 2 ||u(0))22 > 0.
Hence, (u((t) (which recall is equal to /N(u(t))/llu((l]) is defined for all t 0. Now, by Lemma 12 for all t 2 0, we have
dV(u) | 13, and ú V.(u
dt
Therefore, using Lemma
10 for all 2 0,
V.(u)) (u ú (uT
(1 1
uu"
VV(u)) 2 0.
(23)
dt
|
For all l/2 2 4 2 0, integrating the above equality on both sides from b to la we get
N((((22) ((((t))) = fa V
uu?
VW(u)dtt
0.
(24)
|
18

---

Published in Transactions on Machine Learning Research (03/2025)
Thus, (u((t) is an increasing function. Now if we define (uu(O) = then for any t 2 we have ((((()) 2 7>>00 0,
which implies
M(((() 2 251l vI 20.
From the above inequality, we have
d||uul uJu LV(((t))) 2 Li vt 2 0.
(25)
dt
Taking [lu(t) to the LHS and integrating both sides from to f, we get
2L74, VI 20 O.
L/2 |I((0)11
|
Simplifying the above equation gives us
12
2 t- 2) 2)
|u(0) 2-
F. Vt 2 0.
The above inequality indicates that [|u([]]22 is not bounded for all finite 0, which leads a contradiction. Henceforth, choose 7 to denote the time when u(t) becomes infinity. Formally, assume that
lim tT
[|u(t]]22 co, and |u(t)]22 oo, for all (0,7").
We now proceed to prove that lim,-r-- u(t)/||u(t)lla exists by establishing that the length of the curve swept by u(t)/]) given by
dt,
is of finite length. This proof technique is inspired from Ji & Telgarsky 2020 and similar technique also employed in Kunnr & Haupt (2024).
23 and eq. 244. we know (u((t) is an increasing function in [0, (*). Further, since M(u) = x THX: u) S |||z||||||| where the inequality follows from eq. (19). we have that M(u) is bounded from above. Thus, using monotone convergence theorem, we have lim;- (u((t) exists.
From
Now, for all t [0,T*), we know
d
||u||/2 )
(I un?
(I
un? V.(u))
(26)
dt
| |
||u||2
|
||u||2
Therefore,
dt
d
|
11
=
(
V.(())
|ull
(27)
Now, suppose lim (u(()) = if (u(()) converges to 7 before T i.e., Mu((T) = = for some T < T Then, for all t e [TTT",,
dN(u)
0,
di
which implies
(11- (1- u(t) [u(t)
(u((t)))
= 0,
Vt -
[T,T').
19

---

Published in Transactions on Machine Learning Research (03/2025)
Hence, from eq-
27 have
11
O, v (T,T)
dt
||u|2
=
u(t) which implics lim---T- [|u(() la
exists and equal
u(T) [
Thus, we may assume J-N(u(()) > 0, for all E )) We define g(u) - NV)( Then, since
[V,((u))l 0,
have that
V [V)gll)22 2 0 /
Hence, from Lemma 6 there exists 0 and desingularizing function (.) defined on [0, v) such that if | > and g(u) < v, then
1 s (
V - ((l (N))ll
(28)
Since lim (u(t)) J, and lim |u(t)]l2 = 0o, we may choose large enough such that ||u(t)]22 > 1, and g(u(1)) <, for all / [T, T*). Hence, for all [T, 7*); we have
dN dl
[u T
un
Vu(())
|| ||u||
uu?
(u
d
||ull2
dt
2
= ||ulla (u)
2
dt
)
2
V (u)
dt
|
u
In the above chain of equalities and inequalities, we used Lemma 10 in the third equality, and the inequality
follows from eq. (28). Now, since f - (u(t)) E (), for all t E T"), and Y' > 0 on [0, v), we can take V'J - ((u())) the left hand side to get
d
||
dV (V(F-N(N)) vt (T,").
<
dt
|ulp2
dt
Next, integrating the above inequality on both the sides from T to any tı [T,T*), we have
dt ||u||// (111) dt s v ((((T) v f(((t)) s V Muu(T)) oc.
d
From the above inequality, we further have
d
11
d
11
T
u1
12 dt
/
dt
+
dt
dt
|u
dt
|u
2
J
dt
|
d
u
dt + v ((u(T))
<0,
which completes the proof. We next show that there exists k such that
|a(t)]22 (T 1) Vt E T)),
(29)
which will be useful later. Define h(() such that.
h((1)
u(t) (T 1/1-222)
20

---

Published in Transactions on Machine Learning Research (03/2025)
Our goal is to show |lh(t)]l2 for all [0, and for some & 0. Now, since ||u(t)]l2 > 0, for all
E [0, )") therefore, [lh(t]]2 0, for all E (0, We next show lim,-7- [lb())]2 > 0. Since
d |
(u( = LN
1
|
|
2 dt
|u|2
we have
1
d d|jul|2
LN
u
|u|
1-1--11 dt
|
Integrating both sides from 0 to E (0, T*), we get
L0 LN
u(s)
ds.
L-2 2 |u((0)l222 lu(t)l6-2
[|((s]]2
Re-arranging the above equation gives
100) ! L(I- - 2)W
u(s)
ds
(30)
[u(s)
||u(t)l2
Substituting u(t) -tt)/(("=)))ia 2)) we get
1/|lu(0)116-2- L(L ( (())) (T'
(b(t)l2
LHS of the above equality, at = T", the denominator obviously 0, and the numerator is also since, using eq. (30),
{" L(L - 2)W
u(s)) ds (|u(())l2)
|u(0)l22
[u(0)ll J LIL 2)W [Tu(s)]l2
1
u(s)
lim 17
ds lim 177 |lu(t)ll
1-2
Therefore, using L L'Hopital's rule,
lim, - M(L 2)W ((((()/lu(((l)) lim L(L 2))(((()//ll((()))) > t
[b(TT)
Hence, [lh(0)l22 > O, for all LE [0,T", which implies there exists some such that |[h()) K, for all E [0, T*) and proving eq. [29
Case 2: ((u(O)) 5.
In this case, we may further assume that (u((t)) < 0, for all t 0, since if for some F, ((((F)) 0. then we can choose be the new starting time and use the proof for Case 1 to show that u(l) becomes unbounded at some finite time and also converges in direction. Therefore, we assume (N((t)) s 0, for all 2
Now, since
1 u
2 dt u LN(u(t)) 5 Vt 20
we have that [|u((]]22 decreases with time, and lim,-+20 ||u(t]]l2 exists. Now, lim,- ||u(t)]lz = 0, then we have lim u() = and our proof is complete.
Else, suppose lim ||u(t)ll2 = > 0. Then, we have ||(()]la 2 n, for all 0, since ||u(t)]la decreases with
time. We next show that limy->oe /V((t)) = O. From Lemma 12 we know
dW(u)) dt
| Vt 2 0.
21

---

Published in Transactions on Machine Learning Research (03/2025)
Thus, (u((t) increases with time. Since we have assumed (uu(t)) 0 for all 2 0, by monotone convergence we have that lime-soo ((((() exists. Now, to show lime-rooW(u(()) 0, we assume for the sake of contradiction that lim ((((t) = - < O. Then, for all 2 0, we have (u())) 2 -7, since (u(())
is an inereasing function time. Thus,
d d T = (N(((t)) & -by, Vt 2 0.
2 dt
From the above equation we get that ||(()]l2 would become smaller than v after some finite amount of time has elapsed. Since ||u(t)]l2 2 17, for all t 0, this leads to a contradiction. Therefore, lo ((u(t)) = 0. This also implies lim (u())) = 0.
We next prove that lim (tt)/ ([]] exists. We first define û(t) = 2u(l)/n. Now, if û(t) converges in direction, then u(t) also converges in direction. To prove ú(t) converges in direction we can follow the same approach as in Case 1, specifically from eq- (27 onward. This transformation is essential because, recall, to use Lemma 6 in Case we had to choose T large enough such that ||u(t)]l2 > 1, for all t 2 T. Here, ||u(t) 22 may never be greater than but []á()]22 2 2, for all / 2 0.
So far we have established that either [u(()] converges, or u(t) converges to O. We next show that if converges to u. then u, must be a non-negative KKT point of the constrained NCF.
(xt)
n(() [u(e]
From the proof far we already know that N(u.) = N(u)) 2 0. Therefore, from Lemma to show
we only need
LN(u.)u. V.V(u.),
(31)
For the sake of contradiction assume that there exists some 7 0 such that
[VN(u.) LN (u. l2 2 7.
(32)
For any > 0, we define the set u, = {u: : ||u u. S c). Since N(u) has locally Lipschitz gradient, given 7, we can choose sufficiently small 1) such that for all E ue, we have
[V (.(u.) S 7/4.
(33)
Since
( u(0)l2
converges to u, and (u) continuous, we choose T large enough such that for all 2 T,
u(!))
Ta(t)) N u(()) 2 [ (11(6)) -L...N( s 5//4.
Sc u.
(34)
[ converges, then are two scenarios: [|u((]]la either goes infinity or [|u(t)]l2 remains finite, bounded away from 0, for all 2 0 and (u((t) converges 0.
From the two cases discussed above we know that if u(t)
Now, if [2a(()]2a goes to infinity, then we may assume T is large enough such that for some k. > 0
|a(()]la 2 - 111111112 VL (T,T',
(35)
where lim,-r- a(t)/|[[tt]la u. In the second scenario, we may assume that T' is large enough such that for some 7) 0, have
|n(1]]2 2 n, VI T.
(36)
22

---

Published in Transactions on Machine Learning Research (03/2025)
Now, for all non-zero RR we have
uu
(u
(u) |ull
LuN(u (u)
2
V.
u
||u||2 .V .V
Lu
||u||2
|| 2
2
V.(uu.)
Lu
u
V.
u
vVV(u.)
||ul|2
||u||
||
2
VVV(u)) - Lu.N (u.) [u.)ll2
LuuN Lu.N(u.)
(u.)
Lu
-V
u
|ul|a
|
VN
v.V(u.)
|u||
where in the second equality used (L- 1) hhomogeneiyy of V. and L-homogeneity of W()). The inequalities make repeated use of triangle inequality of norms. Hence, using eq. for all t 2 T, we have
32 eq.
33
and eq- 34
(1-
(u(((t) (t) utt
2//2.
[u(t)]2-1
2
Using the above inequality and eq.
23 we have
dt (1(())))
d
(N
u(t
-
-
| (I- a()a(TT
VN(((())
|u(t]
|u(t)l
Now, if [|a((]]aa goes to infinity, then, from eq.
135
we have
dt (N(1((1)))
d (N u(t)
22L-2
Vt E (T.
4(T - t)¹
Integrating the above equation from T to 1 (T,T*), we gel
u(t) ( -
(TT)
2
1 (T 1)
22
2-2
T - -
T
N
In
4
T
As t approaches T*, the LHS remains finite but the RHS goes to infinity, leading to a contradiction. In the second scenario, where [u())]l2 remains finite, bounded away from 0, for all i 2 0, we have, from eq. 36
dt (N (N(( u(() (M(1()))) 2 2/2/-2/4./127
Integrating the above equation from T to 6 T, we gel
N
u(t) [
.V (1u(T))))) 2 T)/4.
Since N (u/llul))) is bounded, the above inequality can not be true for sufficiently large t, leading to contradiction. This completes our proof of Lemma 2
Before procoeding to proof of Theorem we state useful lemma.
Lemma 14. Let us be fized vector and let u(t) be the solution of
du di
VN(u) J(: 2, u(0)
(37)
23

---

Published in Transactions on Machine Learning Research (03/2025)
If -o u(t) # O, then there exists 1j 0 and T 2 O such that ||u(t)]l2 n, for all t T. Further, for ony E 10,1) there emists Te 2 T and B, such that.
u 21 -,, and ||u(()l22 S Be, VI S To:
(38)
[T(T.)l2
where u, is a non-negative KKT point of
max -Vu)) = z (X; || 1==11
(39)
Proof. Similar the proof of Lemma
2 we consider cases.
Case 1: ((u(0)) > 0. Let M(u(O)) > 0, thus [|u(0]]22 > 0. From Lemma 13 can choose = ||u(0)lle and T = 0.
I (1ll2 2 ||(0)]|2, for all 12 0, which implies we
Next, from Lemma and its proof we know that for some finite T", lim,-TT- u
Tutola u, where u, is
non-negative KKT point of the constrained NCF eq. 139 Thus, for any e E (0, 1), we can choose T, 2 = 0, and large enough but less than T such that
u (Te) u [l
T
(40)
Since for all (0, u(t) is finite, there exists B, such that [|u(T)) Be. We note that gets smaller, we may have to choose T, closer to T This implies that, since lim [|u(()]l2 = co, Be will increase. However, for fixed e, both T. and B, will be fixed.
Case 2: M(u(0)) S O.
In this casc, we can also assume that M(u(t)) S 0, for all t 0, since if ((u(t)) > 0, for some T, then from Lemma 13 we know [[[(]]]2 ||u([)]2, [2, for all > T. Moreover, since (u(()) 0 implies [|u [2 O, we choose 17 || and T = f. Also, in this case, from the proof of Lemma 2 we know that for some
finite T*, lim,-7- [[ u., where u, is a non-negative KKT point of the constrained NCF eq. 139
u(t)
Thus, similar to Case 1, for any E (0, 1) we can choose T. 2 t and Bg as desired.
Hence, suppose M(u(t)) < 0, for all 2 0. Then, since
1 1
2 dt u T ú LN(u(t)) 50,tt
we have that [|u())]la is a decreasing function with time, and hence, lim ||u()]ll exists, Also,
| 2 lim []u(()]2,, for all f 2 0.
1-500
Since we have assumed lim u(t) + O, we know lim [u(()]]2 > 0, and we can choose 7) = lim, [|u(t) u(t) [
and T 0. Further, since lim
u., where u. is non-negative KKT point of the constrained NCF eq. (39), for any e E (0, 1), we can choose T. large enough such that
u (TT)) 21-c.
(41)
[TTT)ll2
Also, since |lu(t) ll is a decreasing function with time, we have ||u(t)]l2 e [n. [|u(0))l2), for all 0. Thus, we choose B. [|]]]]]
24

---

Published in Transactions on Machine Learning Research (03/2025)
B.2 Proof of Theorem
Proof. Let wo be a fixed unit norm vector. Suppose u(l) is the solution of
du dt V. (N) - (X: (0, u(0) Wu-
(42)
From Lemma 2 we know that there are two possibilities. Either utt) converges to 0, or u(t) converges in direction to a non-negative KKT point of the constrained NCF.
If (t) converges O, then we define 17 2. Then, for any E E (0,n/2), we define B, and choose large enough such that
[ S B =
(43)
Flse, if (tt) does not converge to then from Lemma 14 there exists ñj > 0 and T such that [|u(t)]l2 7. for all f 2 T. We further define min(f, 1). Then, from Lemma 14 for any E (0, n/2) there exists T. T
and B, such that
(T.)) u. [T(T)ll2
2 1 and |lu())]22 B,, 4I < To,
(44)
where u, is a non-negative KKT point of
max N -o(eoo)). = uX; (() ||
Further, since T, T, we have
[T(T))l2 2 72
(45)
We next define other dependent parameters which are useful in our proof. Let B, = B, +e. Since H(x; w) has locally Lipschitz gradient, there exists K, > 0 such that if [[will2, l|well S Be, then
[X: I (X; w2) ll s Kellwi -wele-
(46)
Also, since Vyl(f.v) is locally Lipschitz in 0 there exists 33 such that if ||2,112, ||22ll2 S SBB, then
|ee (zI.y) E' (zz, y) 1l2 s B1|z1 22l12.
(47)
where recall
8 = sup(l) w)lla w sk-13.
(48)
We further define C. = -B,B,, and
84 = min (1, - ...
(49)
where recall
a (x]]] w sk-1).
(50)
Now, for any 8 E O0, let w(t) be the solution of
((t) = VL((())), w(0) = dwo-
(51)
then, we rewrite the above differential equation as
w oe (H(x,; w), = -IX( -IK w(0) = 6.w-
(52)
25

---

Published in Transactions on Machine Learning Research (03/2025)
We define E(t) := (1H((X:),,)) - y), then the dynamics of w(t) can be written
w WE-J(X:w) el (H((; w) = - J(X; w) (e'()))(e()))) + E(t))
(53)
Now, define s(t) w (अ then s(0) wo, and
ds d ( 1
( )) = 8L-1 w
1
t
di di 8
82-2
(L-Z)) ))==
T
((10.)) + (0i=z)
J
X:w
Xi 1
g (LL-)) (e(o,s)+
0L-2 (x~z))
J
- (X:s(tt (((0,)) +
(54)
82-2
where in third oquality, from Lemma H(X; w), we have
we used (L- ->-momoggcnct of I (X; w). Using L-homogeneity of
= - (0,y)
= -11 - y0
-(0))((x:s(t)))))- (0.,))
Let E(1) = E(1/8--2), then
$ - J (X;s(1)) T (0, ((()) s(0) wo-
Next, we will show that [|s(()- - u(()]l2 S e, for all E [0, Tzl. We first note that, s(0) = (()) wo- Define
TB inf ||s(t) u(t)ll2 = c). 120
Here, Ts indicates the time when [|s(t) - ult) becomes e for the first time, and in the subscript indicates that T3 could change with 8. By definition of To, for all E [o. Tsl, [|s(t) - u(t)lla S c. Now, if we can show T& Te, then we would have proved [|s(() u(t]]l2 S e, for all LE [0, T.).
For the sake of contradiction, suppose Ts < T.. This implies that ||u(t)]l2 s B. and [s((t) l2 B. +e Be, for all t E [0, Ts.. Now, using eq. (46 we have
[I)(X;((t)) ((X;s(()) I K.llu(() s(t) Vt E (0, Tz)
(55)
Next, since 8 < 1 and
|/H(X:s(t)ll= S B|SS < SBB Vt E |0. Tz).
using eq. 47 we have
LE(1)22 |0(8)' ((O,))l < 88-31 s()))la 8'B3BB vi e [0,Ta).
Hence, by the definition of Ce, we have
ll((t)l22 & C.0*, for all te [0, Ta..
(56)
Thus, for any E [g]
1 d s
(s 1) T (s - ii)
2 dt
-ss u) T)T(ce (J[(:]))) [( - (s -s- u) J(X:))
< (0)) + -||s -|stl=llsllllll
I (0 u1l2 -0.5
26

---

Published in Transactions on Machine Learning Research (03/2025)
where the first inequality follows from eq. (55 and eq. (50g. For the second incquality we used eq. (66 and
[|s()]]22 S Be, for all & E Ts). Integrating the above inequality on both sides from 0 to l E [o, To) we have
[|s((t) u(t)) < (aeBL
K. (0.l - d(t)llddt.
Then, using the second Gronwall's inequality in Lomma
we have
[s((T) =l s CacBB-'C.847 se 2T, (.]l I
where the second inequality is true since T, S Te. By definition T we know ||s(To) u(To)ll2 = Thus, using the above inequality, and since 8, where å satisfies eq.
49 we have
= (sTs) u I C.
S 22/2
which leads to contradiction. Hence, Tg > Te. Since |s(() - ( I c, for all t E T2], we have
[Iw(t/6"-3)1 1l2 = (()lu(t)ll +) < B, [O,T.].
We next prove the second part of the theorem. If u(t) converges to 0, then from eq. which implies
43 we know ||u(T.)]l2 S e,
T. 8L-2
| (T)s(T.)ll2 S 20c.
(57)
Else, from eq. 44 and eq. 45
we know
u(T,) ((T.)Tu. u.
2 1 e, and [I([T)]l2 n
(58)
[|u(Te)ll2
Define T. Te/8L-2, Since 3 w (zT=)) (TTT)) 2 S 6, then for some Ç E TR. we have
w(T.) uTT)) + +,
(59)
8
where [K]laa Se From E (0, n/2 and |u(T.)]l2 2 11, we have [lu(T.) <lla 2 n/2, which implies w (T.) 2 5n/2.
Next, from eq.
59 we have
TT.) u(T.) 5
w (T.))ll ||u(T.) + <ll2
Hence,
w (T.)
u(T.)T u, + CTu,
I (2))ll2
||u(Te) + Cll2
(u(T))TT u
[T(T))]2
CTu,
[T(()]]l2 [IT(T)) + Cll
[u(T.) + Clla
|lu
2c
2 (1
| ((lT))>12 + ||cl|2
1
2E
(1
Iclla
u(T.]lla
1 1-6 E
2e
2 (1 (1)
(1-
2
2 1 1
(1+3)
3
1+
1
where in the first inequality we used eq. 458
along with the fact that ||ll2 S and | + <llz n/2. The [|([T]]l2 n.
second inequality follows since ||S||a 5 and
27

---

Published in Transactions on Machine Learning Research (03/2025)
C Proofs omitted from Section 5
We first state some key lemmata which are used to prove Theorem 3 and Theorem 4
C.1 Key Lemmata
Recall that for any locally Lipschitz continuous function f: X - I, V (x) denotes its gradient at x E X, provided f is differentiable at x, and its Clarke Subdifferential is denoted by 0f(x), which defined as
ax(x) == conv lim Vf(x)) lim x; = x.xxen)
where Q any full-measure subset of X such that f is differentiable at each of its points. Note that if f is differentiable everywhere, then 0f(x) (Vf(x)}-
To compute the Clarke subdifferentials of functions which compositions of non-differentiable functions, use Clarke's chain rule of differentiation described in the following lemma
Lemma 15. [Clarke, 1983, Theorem 2.3.9) ) Let hiy...,ha Idd + IR and R - IR be locally Lipschitz functions, and f(x) = (x)1(x).... (,0).....,,,,())) (x)),), then,
0f(x)= S conv Σ
n
a.C Si Oh,(x), (g(h)(x)).... (n(x)) 3
To prove our results, we will frequently require the gradients of the NCF with respect to weights of each layer, which are derived next.
Lemma 16. For L 2 let H be L-layer feed-forward neural network whose output is defined in eq. 9 For all l E [L 1] and i E [n], define
Φ. x; h. = Wi
(60)
6. ob!)) b/, W
(61)
and let A = diag (o' (h(h))
where d.). denotes the Clarke subdifferential of o(-). Also, let
e = 141 A W.T yt and e yss
for all E [L-1] and i E [n], then
Ow, Σ H(x WI.... Wv.) S Σ c((o-11) for all e [L].
Proof. We begin by noting that, for I E [4],
dw. Σ Hx Wi," W..) s Σ aw, (v:H(x; w..)..W.)).
Thus, we need prove
Ow. (y: H(x; W1. ..)) C 1 for all I e [Z].
For / = L, the above equation is true since
dw (xx W1, ... Wt)) ow, (u (uWoof-1) y1 (
For I E [L - 1], we have
Sw, H(x; WI... ... Wz)) = ow.
C c((
where the last inclusion follows from repeatedly applying the chain rule. This completes the proof.
For non-differentiable activation functions such as ReL.U o' could be interval, implying that A! set-valued mapping.
28

---

Published in Transactions on Machine Learning Research (03/2025)
We next restate and prove Lemma
Lemma 17. Let H. be L-layer feed- forward neural network whose output is defined in eq.
19 where
I. 2 2. Suppose (W1,... W.. is non-zero KKT points of the constrained NCF in eq. 10 1 max(aar, for some E N and a E IR, then
=
diag (.W.)) diag
W 141 WHH1, for all E [T. 1}.
(62)
and, of)(x) I, then
W,W, W44 W141, for all l E [L 1].
(63)
Furthermore, if [W 1. W 1-11 W = (a as -1 w ). where |ayl|e = ||billa. for all E [L-
then |ail p for all 1 E [L - 2], | = (pp) 1/4 /// |arll ////
pp-11/p /p. for all I E [I. - 1], and
ww 1/f, where p = p4-1 + p 2-2 + ....
Proof. By KKT condition, for I E [L 1], we have
Σ y. (xx WI, Wo) + +AW
E ow.
(64)
Multiplying the above equation by W, from the right, and using Lemma 16
we have
OE
(
W, #WW.,?
(65)
2=1
Since WiO h/, we have
0 Σ
(b + #WWWW,
(66)
t=1
Since e = A!W 411 A
Li, if we let W 1+1 A
W
1 Ui. then we have
0 7 T +WWWW
(67)
Next, similarly using the KKT condition for W141. we have
O W
O T + AW W W41-
(68)
141
Expanding e|1,, we have
0
W 142" ... WLY, +AW (41 W
(69)
which implies
n 0 E
q ( XW W
(70)
1=1
We next show that
pdiag(q.(0)) (ba)(A)q((h))))
[n] and / E [L 1|,
29

---

Published in Transactions on Machine Learning Research (03/2025)
which will prove the first part of the theorem. To show this, first note that since A is diagonal matrix, and qi hy are vectors, we have
diag( (b)) Afdiag(q
diag A/
diag q, diaggi oo (h/))diag(h))
pdiag((;) (d)))
pdiag(q.)diag(o) pdiag(q:(d))
where in the third equality, we used the definition of A and the fourth equality follows from the fact (x)xx = pa)).)
For (xx) x. note that A will be identity and of h/ for all i [n] and I e [T. 1]. Hence,
(ka) ( Vi [n]] and / [L 1],
which proves eq. 63
We next prove the second part. From eq-
62 have
W 2 =
- W 11
Since
W
| =
if we define p + pb-2 1, then,
(p + pt-- +...+ 1);
which implies W = p'- for all Le [L]. Next, since ||a/|2 = | we further get p²-l/ğ ||ay 11 for
all I E [T.
and [wl3 1/6.
Now, using eq.
62 for all / E [L 2], we have
diag (||b:l faaa) p- .dia / b/41) and diag
p diag (ww))
which implies
ail = p//4|br1l and |a.--|| = (pp) w/
(71)
where the last equality uses pd-1// /p- | Hence, the proof is complete.
C.2 Proof of Theorem 3
From Lemma
we know
w 12 = 1/L and
|aylا
11
|bbll
VT. for all I E [L - 1].
(72)
For the remaining proof, we handle each of the cases individually, and begin by a = 1. Case (a=1) = 1) We begin by proving the "only if" part. From Lemma we have
W.W/
W
M [1-1]] 1},
(73)
which implies, for E [T. 2].
by Baja T
and | 1ar -aa_-1aT_1 ww
30

---

Published in Transactions on Machine Learning Research (03/2025)
Combining the above equality with eq. 72 get
qaay b1 vE [L 2), and 92-1a L
where Qi E (-1,1). Hence,
Wi = a and W, - ---aaaa 2 S 1-
Now, since (W,,.-,, Wr) is a non-zero KKT point of eq. 10 using Lemma 11
we have
b + Vw. (y H(X: WT. ... WE)),
(74)
where À # O. Let 91:L-1 = 9192 9L-1- From Lemma 16
we have
Vw, (y H(x, WI," WE)) =
L
n
91:L-1
a1a₂ a2a3
-aa- -aL-2a,
a1-1 1---21-11--1111-
aa
/L2/4
1
L-2
91:-1
VL
1
a1y:x]
(75)
From eq.
74 and eq.
75
we have
1
L-2
n
=Ab_ + g1:T-1
y
(76)
VL.
L1/4
Now, is KKT point of
max
Σ y1x;
u, such that ||ul|2 = 1/VL,
(77)
n
if there exists X such that
Σ yix; + u² 0, and ||u.1| = 1/VL.
(78)
Hence, from eq.
76 and since |bb113 I/VL, observe that b satisfics the KKT conditions in eq.
78
We now turn towards proving the "iβ" part.
Note that (W,, ...... W,) = (ajb, ggaaa] -L-2arr-aa (0_-aa_-2 11--0----- ( Now, since bj is non-zero KKT point of eq. [2]. from Lemma I1 there exists X" + 0 such that
yı + A = 0.
(79)
We will complete the proof by showing that for = -1: (1/VT.) L-2
112/4
AW, + Vw, (y H(X;; WI.... WE) 0, VI E (L).
(80)
31

---

Published in Transactions on Machine Learning Research (03/2025)
For 1 1, we have
AWI + Vw, (y HX; WI.... Wx))
=babb
W 2
W W
= daab, fh:L-1
a1a2 a2a3
ar-22a 1- a1 136x / IT,
i=1
1-2
= =baib] + Q1:L11
a
VT.
L
i=1
L-2
= = -Q1-L-11
L a =0,
where the penultimate equality follows from eq. 79 For 2 2 < 1, we have
AW, Vw, (y H(X: W
WE))
-19-11;/ T W 41
W
W
W
W
iml
AQ1-1a/a/1 T +
911-29h:11
a:a 8/411/1/// aL aL-2a, 1 bia ajaj --a--2/
L1/4
1=1
L-3
AqI-1a/a/1 +
911-2911111
aja
Σ y,x, b
1,1/4
VL
im1
L-2
91:1-291:-11 X
A
a/a
= 0,
LT/4
VL
where the penultimate equality follows from eq.
79 and since | 112 1/vL. Finally, for L, we have
AWL + Vwz (y H(X; W WE))
-
Σ
yxx
W
W
1=1
1 Q1:L-2 Σ
yix,
blaj
alag
ar. 20
1
L-2
>2 + 91:L-2
Σ
y/ bjaj-1
VT.
L-1
XqL-1aTT -1:L-2X*
aT_1 =0
VL
where the penultimate equality follows from eq.
Case 2 (a 4 1): We begin by proving the "only if" part. Since o(-) is 1-positively homogeneous, have
a (cp) (p), and (dq) (d)).
(81)
where e > 0, is vector, and d E IR, q is a vector with non-negative entries.
Now, from Lemma 5 for E [L 2]. we have
|ai = / and |az-1| = L //4 w/
(82)
32

---

Published in Transactions on Machine Learning Research (03/2025)
Now, since (ail/zi' {br (22
A-1 are non-negative, aL-1 L//4| and
ay = b441, 14 E [L - 2].
We next show that qaz-1 L for some q E (-1,1) Since (W 11
W L is
KKT point of eq.
10
from Lemma 16
we have
0 2E + dw, (y H(X; W1, WL)) Aw
V/ (0! (o!-1)"
(83)
where of-1= = - -. (WIx))
Since | are non-negative, using eq.
81 we have
(...... = =aL-1 aL a ||a -1 r-11 lob
(ar,--aa (ar. (ar. 2a (aaaa (a x:)) ...)
Since ||i1] = = 1/vL,
1 L-2
0= \w
(b) x;)(1i a
From the above equality it clear that w is parallel az-1- However, since ay-1 L I'/ we get
qaL-1 1, for some q E {-1,1].
From Lemma 16 we also have
0 E AW + ow, H(X: WI.... WL))..
(84)
Without loss of generality, we assume ail > O, where an denotes the first entry of aj. Let wi1 denote the first row of W. Then from the above equation we have
E Aan + Own ((X: WW.)).
(85)
Our goal is to simplify Ow., TyTM(X, W WE)) Note that, for any W1, we have
H(x;; W1, W2.
WL) qa T
- (a ( a 20 (as 2a (asa (Wix))( 1)/
= - [2 ...
|a 120 (a (aTTo(
L-2
-ا 2 (w
Next, choose W1 such that its equal to W1 everywhere except for the first row, and the first row is assumod to be equal to Thus, have
g(w) = y H(X; W1, W
Wz)
ki ano xi) +
U:0 4-2
Σ ( xi)
q VL
L i=1
1=2
Now, it is casy to that
Ow.g(anbb))= Own. (y TH(:; W .. Wz)).
(86)
To compute aw.g(an1bi), we first characterize g(w) near ani,, If for some i E [n), bj x; = 0, then
k
db- lanoowTxx xx) + Σ
aayo(ay =0'-1(anw Tx,) x;) -a xx))
1=2
33

---

Published in Transactions on Machine Learning Research (03/2025)
Next, if for some i e [n], x > O, then w' x; > 0, if is in sufficiently small neighborhood of an1b;
w'x,
Further, since > 0 and ( x;) 2 0, for all 2 2, we have
=o--1 w [xx,) tot-2 xt-2 an10 xx1 (u)) Σ aaro((r,
Σ (a,,,, x;))
j=2
jm2
he a1 (arob
2111o-1(( (wT x;) of 2--2
1=
The above equation can also be shown to be true b,,x; an1b1 Therefore, if w in sufficiently small neighborhood of an b1, have
x; < 0 and w in a sufficiently small neighborhood of
L-2 1
n Evott ((w] xx))
Owg(()) = VI
L,14
an y:o 1 (w (wTxx)) xxix - Ow
=qai1
VI
L1/4
Since a11 > 0 and E Vid 1iol-1(w xx;))
x;) is --oositively homogeneous in w, using Lemma
we get
L-2
@wg(an1bl) = qal1
1
Ow
Σ voo -iot-1(an (anb, x:)
VL
L1
qaa1
1
L
Ow
Σ Mio"-1((]]
(b]Tx:) xi)
(87)
From eq. 86 eq. 85
and the above equation, we have
E Xb1 9
L-2
1
Ow
Σ
n
y:o
L-b xii
(88)
VI
L1f4
Now, using Lemma
11, is KKT point of
max
y 1-
ford-1(x (xx u), such that ||u|| 1/VL.
(89)
if there exists X such that
0 on Σ y (x, un) + t,u and - 1/VL.
(90)
Hence, from eq-
88 and since |lb:l| 112 1/VI, we get that b satisfies the KKT conditions in eq- 90
We now turn towards proving the "if" condition.
Note that W..... Wp) = [a aza aL-1a
Since b is non-zero KKT
point of eq. 11 there exists a non-zero X such that
Σ
(x (x.bb) + AAb,.
0 e du
(91)
Multiplying the above inclusion by b, and using | 1/VL, we get x - 110111((x bı). We
will complete the proof by showing that for À = q2- (I/VI) 1-2
/L114
0 E AW, + Ow, H((; WI," WE)) V [L].
(92)
34

---

Published in Transactions on Machine Learning Research (03/2025)
(Wix))... ()), for all E [n] and I [L- 1]. Since (aillz L-1 non-negative, Let Øf o W WW o.o.
using eq. we have
Φ o oW 1-1 -
- (a ay| (a)- (b,ax
||a-111 ... |aj 2a (b, x:).
Now, using Lemma 16 and since ||a: = 1/VT,, we have
AW + ow, H(:; W Wx))
Aqa/-1/1.11
Vi ( 1 T
t=1
-Aqa A
Σ
Vi Villar
aa 2 1 ... 1|la|1/a, 1 10b x;)
i=1
L-2
1
yia 1- [(b,
T
VL
L-1
11//1144
1
A
aL-1
VT.
Next, for any W., where 2515 S L 1, we have
H(g WT,".. W., .. WE) oaa ... (Wid
1.-3 1--- o (
Let wiy denote the j-th row of Wi, where j is arbitrary. Choose Wi such that equal Wi except for the j-th row, and the j-th row is equal to Wiy- Then
gi (Wis) y (X; WT, ... W ... Wt)
L-3
V:ol-14
Σ p=1,13
lally 1120 x,)
=
VL
L1/4
Thus, to prove eq.
92 for 2 < S L 1, we need to prove
0 AW1j + Sw., (91 (Wig).) for all j E [k]]-
Now, if ar O, then Wiy 0 and 91 (Wi;) is independent of Wijs which implics the above equation is satisfied trivially. Now, if ar > 0, then w/ w1,a atyl w > 0, which implies w/Jai-i > 0 and hence, a(wTar-io'-1 x,)) w.Jar-
- o (bj x;), for all wi sufficiently small neighborhood of wij- Therefore,
3
(i(w))) =q
yio L-----1
ai I (bixx) +
Σ
a a 11200 ( xx)
L1/4
p= 1,p.j
L-3
q
yio 1- (b] x;)
any ar,WW,ai-1+
Σ
a? ||a 12
I
p=1,03
35

---

Published in Transactions on Machine Learning Research (03/2025)
for all Wiy in sufficiently small neighborhood of Wty.
The above equation implies
L-3
1
AWlj Sw,,90 91 (Wi)) >ayjaa-11 +
y:o
(bb
(VL)
T,/4
L-2
1
-aaija/-1 92º
VL
L
Now, for any W,, we have
H(x; W1, 2
WL) WLo W (Wo
- 0 1.. aza a.2/ oWW (I)xx)// (a) (
1-2
Let Wij denote the j-th of Wi, where is arbitrary. Choose W such that it equal to W1 except for the j-th row, and the j-th row is equal to Wij- Then
gz(W1,) =: y TT H(:; W1, W2.
WL)
L-2
3ho 1.-2
(1)x(
+ Σ
ob) xi)
LA
p=1,ps
Thus, to prove eq.
92 for 1, we need to prove
0 E XWij + Owi, (g2(W15)), for all j E [ki]-
Now, if aıj O, then Wij 0 and 92(w1,) is independent of wij: which implies the above equation is satisfied trivially. If a1j > then, using the same approach get eq. (87) we get
(w1,))(W1,)) Owge(ar,b;) = qaıj
1
L-2
1
Ow
Ow
Σ
n
yoo o
10, xx:
(93)
VL
L1/4
Hence, using the above equation and eq.
91
we have
XW1 + Owiy (y H(X: W1,". W..., WE)) = λaιدbı + (2(1))
L-2
yoo 1- ((bTxx x:) ))
\bi +
Ow Σ
VL
17/4
1=1
which completes the proof.
C.3 Proof of Theorem
From Lemma we know
p p'-1/p = ||| vl e [L - 1]. and [| = 1/p
where á = p + p +1. 1.
Case (a (a=11 1): We begin by proving the "only if" part. Since o(.) is p- homooeneoos,
(cpp) = d'a(p) =p=-1 and a(ep) p²o(c),
(94)
where e R. p is a vector. Note that, in this case, o(-) is positively and negatively homogeneous. Also, we use (a9)(-) to denote the derivative of o'(-), where g N.
36

---

Published in Transactions on Machine Learning Research (03/2025)
Now, from Lemma 5 for all / [L - 2],
|i p bi41/ and |ay-1] (pp) w
(95)
We next show that, for E [L - 11, each non-zero entry of ai has identical absolute values. Let o oW. ( (Wxx)).... ..),, for / [L 1], then, using eq. [94],
= o W .x.....)) c(ai a(a
(aab) x.)).... (a x.)) ...)= = af
= a (b) o(a--b ... o oa
where cf = ob; o (a ... oayb (a, bj x,) is scalar. Since (W1, WL) is non-zero KKT
point of eq- 10 from Lemma 16 we have, for 2 < < L--1,
AW + Vw, (y H(X: WI.".. WE)) Aaab/ Σ e (0 10 T
Aarb/
e - (al_1)
(96)
Multiplying the above equation by a from the left gives us
n Σ aj ejc, ) (af_i)
b| +
(97)
The above equation implies that b, is parallel to af_1- Now, if p is even, then af_1 non-negative and thus, the non-zero entries of by must have same sign. Therefore, from eq. 195 we get
bi = qi 1//p11 for some @ E {-1,1} and for 2
Else, p is odd, then, a|_ has the same sign as ar-1, and thus, from eq. 95 we gel
bi qar-1/p" for some qi (-1,1} and for all 2 2 1-
The above equations and 95 implies that a is parallel to 2 Now, since 2 2, therefore, for E [L. 2), the non-zero entries of a have identical absolute values.
Choose = in eq. 96 and since ef = yi, we get
0 Xw
Σ yack T
(98)
Hence, is parallel ah.-1- p is even, then a non-negative and thus, the non-zero entries of w must have same sign. Therefore, from eq. 195, we get
92 g for some qL (-1,1).
Else, if is odd, then, af_1, has the same sign ar-1, and thus, from eq. (95), get
w q 1/4 for some Qr. E 11,
The above equations and eq. implics that a parallel to a Since p 2 2, the non-zero entries of a have identical absolute values.
Next, from the KKT conditions, have
0 AW, + Vw, TTT, W..... W.))
(99)
37

---

Published in Transactions on Machine Learning Research (03/2025)
We now aim to simplify Vw: M(X; WI, W2)) For any W1, we have
H(x;; W1, W2, ", W2) = Woo W L- WW ...)
-. w -ab
w aL- (I (1 (b) - 2b, (WWx))
Now,
Vw, - o2 (b (Woo(WWWW)) =
which implies, using eq. 94
Vw, 1 (b2 1x,)) = (o (0" (bo x, di)) (arb) b2)
(o (o x; (o (1o0--1)) ( ybb af) ('bj ( (1))in((
= =00--1y(0 x.) -- (b af diag( (a1))bgx,,
where the last equality follows by using chain rule on (a--1)(+)) Let B1 w ar. [bI and 32 = (yb af',, then
Vw, (y H(X: WI.... WE)) 3132 Σ ui (0o x, ddiag(o (a1))bax} da)dddd((())))
(100)
Multiplying eq.
99 by a from the left and using the above equation, we get
a
[2 \||a
31,32
yi (ob x.)a,
diag(o' diag(d((a)))bx
A||a; [2 BB1 añ
(u(o--)((({xxxx/
(101)
Now, u. is KKT point of
max
y10b- b-
u), such that ||u| = v //p-1/p. 1//
(102)
if |u. /p"-1/p, and there exists X such that
0 vi(a4-1y (,,T Xuu
(103)
From eq. 101 and since |lb p ppp-1111 /D, observe that by is non-zero KKT point of eq- 102
We now turn towards proving the "if" part.
Note that (
WL-1, WL) = (a.b) ... aL- aL-1b w WT). Since by is a non-zero KKT point of
max Σ y u), such that | 2 = / 1
(104)
then there exists X # such that
0 -=
v -- (xx (((x,b,)xx bi)x, + b
(105)
i=
38

---

Published in Transactions on Machine Learning Research (03/2025)
Our aim is to show that for some non-zero X,
Σ H WI... Wo) O, V. E (L].
AW + Vw,
(106)
Let 2 = y H(X; W1
W 1), then, since
Σ ii= yio 1::00~1--1000 (b, x2) + 0, we get
yJ H(X: W2. WL) Σ y:w bf x.)...)
w a
a
Vio 1bb] x;) +0.
1=1
Define Z Vw. (y H(X: W1, W2)) Since H(x: W1,
WL) is p² homogencous with respect to
Wi, from Lemma we have
trace
Zi) = pp - y HX: WI.". WL) 1
(107)
Now, λ + O implies |Z/|| # O, 1I E [L]. Suppose we are able to show
Z |
[Z].
Then we can write Z, = aj W,, for some ay Since Wil = op. from eq.
107 we get
p - trace W, Z,) ag W 11? aip arp' / 1/p
(108)
which implies or = på. Thus, eq. 106 holds for X = - Pd. We next complete the proof by showing Z//lZ//| W./l| for all / [L].
For any two matrices (or vectors) A, B, we say A x BifA= sB, for some non-zero scalar s. We aim show Z Wi, which will imply Z//|Z||| r W Willr, for all I E [Z.]
From eq.
100 eq. 105), and using eq. 94]. know
Z1 ui(o--1((b) (oddiaao((aa))bbag)
(Hig( diag(af diag(af"")bbb) bbb,,
(109)
If p is odd, then p- even. In this case, since b2 = aigqi///p /p1/4 and the entries of a have identical absolute values, diag(a)-1))) }b2 b2 x ai. Hence, Z₁ x a,b W, Now, if p is even, then p p-1 1 is odd. Since by 9//ail/p we have diag(a)- a1, which implies Z₁ x a,bī x W1. Next, as shown in eq.
98 we have
Zr x (ah-1)
(110)
If is odd, then p- 1 is even. Since the entries of as. have identical absolute values, the entries of (af are identical and non-negative. Thus, a²..- =
X az-1- Now, since w qraa-1 (pp) we
have ah-1 w, which implies Z, W.. Next, if p is even, then a²_ |az-IPP. Since the entries of a
have identical absolute values, | P |az-1|- Also, since - 1l)))))/////))//// we have a w
which implics Z & WT.
We next consider Zi, for 2 5 <15 - 1, which requires some additional results. For any Wi, using eq. 94
H(x; WI,'.. Wi.... W.) 1) WyooWW,- 1-1 WWo(W 1-1 o )))
W (.x:)...)))
= wwo((as-bb] ...
= w [ (ي/ 2
T
) i
39

---

Published in Transactions on Machine Learning Research (03/2025)
where
1-1 1
0
W 1-1 ...) = olaba x.))...))
= (bJ
(aib) x1)) ...)
= -a x, (b] ... a(p2)
Hence,
Zi diag(o' (aaggoo'Waaaa.a (Wiaa (af_))) diag(a) (b4
Now, if p is odd, then - 1 is even. Since the entries of a:-1 have identical absolute values, (af_1) have identical entrics, and thus, af-1 " )ai- at-1- Also, since a-11 x bj. Furthermore, since b = ma/pp// and the entries of a have identical absolute values, diag(a) "ag(aa"")b(t x b1+1 O a,. Hence Z a,b; x Wi. Next, if p is even, then af_1 = | Since the entries of at-1 have identical absolute values, |a--1 have identical entries, and thus, a x |a/-//- Also, since by = q-1/at-11/p"/4 we have af_1 b. Furthermore, since b la hulppp/4 /4, we have diag(a?" b)b x a,, which implies Z & a/b, x W,
-9-1a1-1/pp* we have
Case 2 (a 1): We begin by proving the "only if part. Since (+)) is p- positivvll homogeneous, therefore,
(cpp) c'o(p), o' pq"-1 and (dq)) q'o
(111)
where > 0, P is a vector, and d E IR, is vector with non-negative entries. Also, we use (a9)(-) to denote the derivative of ('), where E N.
From Lemma 5, for all Ee [L - 2],
a / and |a6-1| (1p) w/
(112)
Now, since the entrics of {a [b/
1-1 are non-negative, from eq. 112 we have by = -/11/1/44 for < I I L - 1, and |w| = a
We next show that, for I E [L 1], each non-zero entry of a has identical values.
Let of
oWW, (W....o(Wix)) (WIx)))... ..., for I E [L - 1], for <1 L - 1, then, using eq. 111
of = oWW (Wix;) ...) o(aib; o(a:-
xx))
=ffo(b] ( day-1b/ ... o( x.))........ ....) af afd.
where of = ob, oaa-- ... oa2bb b, xx)) ...) is a scalar. Since (W1,'". WL) is a non-zero KKT
point of eq. 10 from Lemma we have, for 2 < L-1,
0- = A W. V (y H(X: WI,". W.)) = Aayb,
of (x!-1) [
dayb,
0 P
(113)
Multiplying the above equation by a, from the left gives us
n Σ aj e/c--1
-=>|aa/113 A|la +
(af=))
(114)
The above equation implics that by parallel to a-_ Hence, from eq. 112 a parallel a-_ Now,
since p 2 2, and a, has non-negative entries, we get that, for 1 < < 2. the non-zero entries of a are identical.
40

---

Published in Transactions on Machine Learning Research (03/2025)
From the KKT condition for I = L, we have
n Aw Σ yic 1-1 -
(115)
Hence, is parallel a Since az-1 has non-negative entries, we get that the entries of have the same sign. Therefore, from |w| a (pp) we have w = qaL-1/(pp) for some g E {-1,1}. Also, ay-1 is parallel to --1- Since 2 2, and az-1 has non-negative entries, the non-zero entries of az-1 are identical.
Next, from the KKT conditions, we have
AW, + Vw, (y H(X: WI,... Wo))
(116)
We now aim to simplify Vw, (y N(X; WE) For any WT, we have
H(x; W1, W.,
Wx) Wio(W, L- o (Wa(o(Wix)) (wx:)) ...)
= w o ... ... o (Wxx)))
=w -w (T
/th 2 (WIx))).
Now,
Vw, (w =
which implies, using eq.
Ow.ot~2(b, (W
=
= = =at-1y(() (o) (baaaaaaaaaaaaa dijagg bb2
where the last equality follows by using chain rule on (a- Lot 6 = w wT a (1I of (b] aa -f)))-))) and f2 (al-2)((b) af), then
Vw, T H(X: WL)) 8182 Σ
(117)
Multiplying eq.
116 by aj from the left and using the above equation, get
0 A||a [2b + +1132
yi 31 x;)a,
diag(o diag(o'(a:))bxs b2x
Al|a Ebb p31 añ
(118)
Now, is non-zero KKT point of
72
max
y10t11(x) u), such that | 1
(119)
if [u.112 = Vpb-1/6, and there exists X* such that
Σ 2x(o + uuu
(120)
From eq.
118 and since |b 12 Vpt-1/6. observe that b non-zero KKT point of eq. 119
41

---

Published in Transactions on Machine Learning Research (03/2025)
We now turn towards proving the "if" condition.
Note that (Wi,''. WL-1, WL) = (a,b) aL -L-1a w w.). Since b, is a non-zero KKT point of
max Σ yio 1x, u), such that | v /pt-1/1 /D-
(121)
then there exists X * 0 such that
Σ
11 b,)x; + b'
8:
(122)
1=1
Our aim is to show that for some non-zero X,
n Σ
EyTH(xe y, H(x:: W1. ". WE) O, 4 E [L].
AW + Vw
(123)
Let 2 x W1. W..... W L), then since E 1/10 1 (x,
b1) + 0, we get
y HX; WT, W 2 WL) Σ yow o(ag-1b/ xa]
p w - III o' (b (a
y:o 1aot--1
1--1 (b]xx,)
x,) # 0.
Define Z = =w (yTH(x W WL)) Since H(x: WI,". WL) is pt homogencous with respect to WT, therefore, from Lemina
trace W, Z₁) - H(X: WI, Wi.... =
(124)
Now, Å * 0 implies |Zil|e + O, V [4]. Suppose we are able show
Z
[4].
Then we write Z og Wi, for some ay Since
W pL-1/D. from eq. 124 we get
p trace W, Z,) asl W appb-1/p /pp
(125)
which implies or = P4. Thus, eq. 123 Z Z//lZ/r W:/[[ll for all I e [L].
holds for X Pd. We next complete the proof by showing
For any two matrices (or vectors) A. B, we say A Af sB, for some non-zero scalar s. We aim to show Z4 x Wt, which will imply Z = W W illE, for all E [Z].
From eq.
117 eq.
122 and using eq.
I11
we know
Z,
y: (ol-1)((b) diaa
(an)( diag(a""
(126)
Since by a1 ht/p// and the entries of a identical and non-negative, diag(a 1)22 by a Hence,
Z, a, b. W
Next, as shown in eq. 115g we have Zr x TT Since w = =(ar-1(ppp)1/ and the entries of a are identical and non-negative, a x a Hence, Zz & WT.
42

---

Published in Transactions on Machine Learning Research (03/2025)
We next consider Z,, for 2 2115 L which requires some additional results. For any Wi, using eq.
111
M(x;; WI.". Wi," ... W.) = WW 1-- (Woo - (WIx))...
1 1 2 (b142a
= w ( ...
1-1 (b/ Wiol-1))
where
8-1 = a W (-1 o (Wix)) ...) = o - oaz,,, b x.))...)
oaab (a)b) x.))
aL_ (b) x; (b)aa) o(b 2(
Hence,
Z x diag(d' (o' Biag(o'(W,af (i x diag(a) (bb
Since b, = a/-1/p//4 and the entries of a are identical and non-negative, af_1 x a x b Also, since b a//p// // and the entries of a are identical and non-negative, diag(af" (b441 x ay. Hence, Z x a/b, WT.
C.4 Experimental details
This section provides more details about the experiments in this paper. They were conducted using PyTrrch Paszke et al. (2019), and the code is submitted along with the paper.
Training data. For our experiments, the training data consists of 100 points with inputs in R drawn uniformly from the unit-norm sphere, and outputs in R, sampled i..d. from (0, 1).
Algorithm. To obtain a KKT point of the constrained NCF, the NCF is maximized using projected gradient ascent with sufficiently small step-size, i.e., after each gradient step, the weights are projected back to the unit sphere. This is primarily done because gradient ascent can diverge to infinity. However, from Lemma [9] it is known that, for homogeneous objectives, projected gradient ascent with fixed learning equivalent gradient ascent with adaptive step-size up scaling factor. Hence, projected gradient ascent is also valid method to discretize gradient flow in this case.
The initial weights are drawn uniformly at random from the unit norm sphere. Since gradient flow can converge to the origin, we only choose those initial weights for which the NCF is positive, which ensures gradient flow diverges to infinity (see proof of Lemma and (Kumar & Haupl. 2024 Lemma 5.3)).
Initialization.
Stopping criterion. From Lemma [I] we know that at a first-order KKT point of the NCF, gradient of the NCF will be aligned with the weights. We use this fact as the stopping criterion for algorithm, that is, run the projected gradient ascent algorithm until the weights are sufficiently aligned with the gradient of the NCF.
Remark 2. We make some changes to the algorithm when the activation function is not continuously differentiable, such as ReLU or Leaky ReLU. Such activation functions require computation of the Clarke subdifferenttil. However, the gradient computed using automatic differentiation provided in PyTorch uses the chain rule of function differentiation, which may not give the true Clarke subdifferential for non-smooth functions. As noted in Lemma 15 the true Clarke subdifferential belongs to a set which is obtained using the chain rule. Therefore, when using such gradients, projected gradient ascent may not converge towards true KKT point which satisfies Lemma 11. To overcome this issue, line of work has focused on the behavior of stochastic gradient descent Bianchi et al. [2022): Bolte & Pauwels) (2021). The authors of Bianchu el al. (2022) show that the conslant slep-size slochastic gradient descent, where the grudient is compuled using automatic differentiation, asymptotically converges to the true KKT point, for almost all initializations. They also derive similar result for stochastic projected gradient descent. Motivated by these results, we run mini-bbtch stochastic projected gradient ascent for sufficiently long time, when the activation function is not continuously differentiable such as ReLU or Leaky ReLU.
43

---

Published in Transactions on Machine Learning Research (03/2025)
1.0
0.35
Normaliedd Lose
0,000
0.30
0..2
WWhre 500
0,9029
Normalized Loes
Norm weights
0.11
0.4
0,0062
-0.10
0.2
0.9050
0.03
0.0
0.00
106100
20000
300-01
110000
110.00
1000
20031
(200
No. of iterations
No. of iterations
(a)
(b)
Figure 4: Early training dynamics of three layer ReLU neural network. Panel (a): the evolution of training loss (normalized with respect to loss at initialization) and the fz-norm of all the weights with iterations. Panel (b): the evolution of (W(((())) (())) (a measure of directional convergence w()). The loss has barely changed, and the norm of weights remains small. Also, the weights approximately converge in direction to KKT point of the constrained NCF.
D Challenges for non-differentiable neural networks
Recall that Theorem holds for neural networks with locally Lipschitz gradient, and thus excludes ReLU neural networks. We first provide an empirical evidence that early directional convergence also occurs ReLU networks, and then briefly describe the challenges in extending Theorem ! ReLU networks.
Let (xx) max(x, We train the following 3 h-momggnnouus ReLU network H(x; W1, W v) v
where W, 222020 W2 E 30 and v E TR30 For training, we minimize the square loss with respect to the output of a smaller neural network H (x; Wi, W = 10 ([Vo(Wa o (Wix), where the entries of Wi E 22220 W; E R2x2,v v. E R2x1 are drawn from standard normal distribution. The training data has 100 points sampled uniformly from unit sphere in 22 We define w to be the vector that contains all the entries of W1, W2 and v, w =
and N(w) denotes the NCF for this case. The network is trained for 64900 iterations using gradient descent with step-size 5 10-3, and the initialization w(0) = dwo, where 8 = 0.01 and Wo is random unit norm vector. From Figure 4a we observe that the training loss does not change much, and the norm of the weights increases by multiplicative factor but remains small. Also, from Figure 4b it is clear that the weights approximately converge in direction to a KKT point of the constrained NCF. In Figure we plot the normalized absolute value of the weights at inilialization and at iteration 64900. Similar to the experiment in Section 4.1.2 low-rank structure emerges among the hidden weights neural networks. In fact, the two largest singular values of w and W2/[w/] are (0.5741, 0.0346) and (0.5772, 0.0067) respectively, indicating that they have approximately rank one.
w/|[w//,, =w//|w]]]
We next describe the difficulty in extending Theorem 1 to ReLU networks. Consider neural network that satisfies Assumption [ except that the gradient is not assumed to be locally Lipschitz (this will include ReLU networks). Then, assuming square loss, we can define the gradient flow dynamics using the Clarke subdifferential as follows
n W
Σ (3: H(x;; w)) 0H((x; w(0) SWa-
(127)
It is also well-known that the Clarke differential of an L-homogencous function is (L (G-1) -)--mooognnoouss Lyu
& Li 2020 Theorem B.2). Thus, if we define s(l) f (--=)) then similar to eq. 77 we can show that s(l)
will evolve according to
$
( 8 (xH s(() Wo-
(128)
1=1
44

---

Published in Transactions on Machine Learning Research (03/2025)
wilew
10
11
0.00
01C
(a) At initialization
Wi / fwlls
W; /:/llis
[ ww
0,000
0.12 10
0.000
10
0,000
25
0.75
(b) iteration 64900
Figure 5: Normalized absolute value of the weights at different stages of training
Further, the gradient flow for the NCF can be defined by the differential inclusion
E U)HH(x u(0) Wo-
(129)
Following the proof sketch of Theorem described in Section converegence of w(l), we would need first show that for small enough 6, the solutions eq. 112 and eq. (128) will be sufficiently small for a sufficiently long time. From comparing eq. (129) and eq. (128). it
to show approximatc directional
is clear that for small 5, the two solutions will be close in the beginning. However, the solutions will remain close for sufficiently long time by choosing small enough 5. In fact, the proof of the analogous result in the context of Theorem [ crucially relied on gradients being locally Lipschitz, which is not the case here. We believe that proving this would be crucial step towards establishing directional convergence for ReLU networks, and defer for future invostigations.
can not claim that
E Additional lemmata
In the following lemma we show that for two-homngeneeous neural networks, the gradient flow dynamics of the NCF remains finite for any finite time.
Lemma 18. Let H(x; w) be bwo-homogeneenu neural nelwork that salisfies first and third properly of Assumption 1 Suppose u(t) is a solution of
du ( J (X:u) 2, u,)( Ug-
(130)
dt
Then, for any finite time T 220, O, [lu(T) is finite.
Proof. Let 3 = sup
H w)ll w Sk-1). Then
1
d u
U
(X: = 2H 2H (X: s 28 2
2 dt
where in the last equality we used Lemma and the inequality follows from Cauchy-Schwartz, The above equation implies [|u[T]]3 juoll
1133+ + 1T 3 2 for T > 0, which proves our claim.
45

---

Published in Transactions on Machine Learning Research (03/2025)
Lemma 19. Let F(w) be g-homogencous function, for some q 2. Consider the following two sequences:
= = = wo. 0,
(131)
and
u(t+ 1) u(() ne VF u(0) Wo, 1/0 7, 77 cm) 1-2 vV 2 0.
(132)
Then, for all T T 1, v(T) = (m u (T). Proof.
We prove this via induction. The claim is true for T = 1, since
v(0) wo u(0), and v(1) co(v(0) + +VF(((0)) cou(1).
Suppose the claim holds for some to 1. Then, we have
v(lo) (1 a) u(to).
(133)
Now,
u(to + 1) u(to) + 17o VF(u(to)).
and
v(to + 1) = ao (v((t) + ++(o((()))
ct,, ( a) u(to) + + (1122- ct) --1 VF(u(to)))
Cto (11 c.) (u(to) + (I12 c)) L-2 VF(u(t)))) (1)) (u(In) + The VF(u(())) (M2 u(lo + 1),
which completes the proof.
If we choose c ( ascent with constant step-size. Therefore, the above lemma shows that, for homogeneous objectives, projected gradient ascent with fixed step-size is equivalent to gradient ascent with adaptive step-size, up to scaling factor. Consequently, for any finite T, the direction of v(T) and u(T) will be the same.
in eq. 132). then eq. 132 becomes the iterates of projected gradient
46


Reviews:
[{"review_id": "gij5BhVKNI", "reviewer": "Reviewer_Aicg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- **Comprehensive examination of the initial training phase**  \n    The paper delves into the dynamics of deep homogeneous network weights during the early stage of training, a period during which weights remain small. This precise focus sheds light on how parameter direction becomes influential even before the network grows to a more conventional operating scale.\n    \n- **Identification of directional convergence at KKT points**  \n    By leveraging the NCF framework, the authors reveal that parameters follow trajectories converging toward the KKT points of this function. This perspective yields a geometrically intuitive explanation for how gradient flow evolves in deep homogeneous networks.\n    \n- **Applicability to saddle points**  \n    Beyond local minima, the analysis extends to examining convergence properties near saddle points of the loss function. Such a broader scope is particularly relevant in deep learning, given the complex critical-point landscape frequently encountered in large-scale networks.\n    \n- **Low-rank parameter observations**  \n    The authors note that, within high-dimensional spaces, certain low-rank behaviors of the parameters can emerge and significantly shape the optimization trajectory. This observation is both intriguing and suggestive of potential reduced-dimensional descriptions of training dynamics.\n    \n\n### Weaknesses\n\n- **Limited quantitative detail**  \n    The paper principally offers qualitative or semi-quantitative insights into convergence. More rigorous quantitative treatments—such as explicit convergence rates or error bounds—would lend greater depth and practical significance to the findings.\n    \n- **Potential oversimplification of the directional aspect**  \n    While the work emphasizes the direction of convergence, some readers may perceive the conclusion as an overarching average-direction argument. A more in-depth exploration of the subtle or local-scale dynamics underlying this macroscopic behavior could strengthen the contribution.\n    \n- **Insufficient discussion of discrete optimization methods**  \n    Although the continuous-time gradient flow approach offers valuable theoretical clarity, it remains one step removed from standard (stochastic) gradient descent approaches used in practice. Establishing stronger connections between the continuous and discrete realms, possibly including empirical validation, would bolster the paper’s relevance to broader applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an analysis of gradient flow dynamics in training deep homogeneous neural networks with small initializations. By assuming locally Lipschitz gradients and focusing on networks whose order of homogeneity exceeds two, the authors establish that, in the early phase of training, the network weights remain small in norm and converge directionally toward Karush-Kuhn-Tucker (KKT) points of a so-called “neural correlation function” (NCF). Furthermore, when employing a square loss and under a separability assumption on the weights, the authors observe a similar directional convergence near certain saddle points of the loss function.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Is it possible to derive some explicit convergence rates for the early convergence? \n- In Assumption 1, it's better to add that you assume $H$ is first order differentiable.\n- Please refer the assumptions in Theorem 1.", "broader_impact_concerns": "I believe there is no concern on the ethical implication.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3m7O7IV95n", "reviewer": "Reviewer_JVoi", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: the paper is well-written and the contribution and limitations are discussed in detail. The proof ideas are easy to follow and the empirical observations align with the theoretical findings.\n\nLimitations: as the authors have mentioned, the gradient flow dynamics is characterized only for networks with locally Lipschitz gradients, which rules out ReLU activation. Also, when assumption 1 is not satisfied, there is no guarantee that the gradient flow will find some low-rank weights.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper analyzes the gradient flow dynamics of training deep homogenous networks with at least 3 layers. The authors show the gradient flow either exhibits early directional convergence or converges to zero when the initialization is small. They establish the connection between gradient flow on deep networks and the gradient flow on maximizing the neural correlation function (NCF). Moreover, the authors empirically observed that the  gradient flow finds low-rank stationary points of the constrained NCF optimization problem. They also derived necessary and sufficient conditions which charaterize these low-rank  stationary points.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "(1).  I wonder if the authors have done experiments on the gradient flow dynamics of the neural network weights when Assumption 1 is violated (e.g., for ReLu activation)? Does the weights empirically exhibits low-rank structure (or properties similar to those the theorem 1, e.g., directional convergence)?\n\n(2). I wonder if the authors could provide more intuition on how the gradient flow dynamics (early directional convergence or convergence to zero) depends on the initialization. For example, what happens if the normalized initialization $w_0$ is a gaussian vector.\n\nMinor issues:\n\n(1) From the statement of Theorem 1, it is unclear what are the  assumptions required for the theorem to hold.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JC735hpXgM", "reviewer": "Reviewer_ZdJ2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- Comparing to the theoretical study regarding ''NTK'' regime, the theoretical understanding of the regime where the initialization is small is very limited. To the best of my knowledge, this paper is the first to provide an implicit bias analysis on deep neural networks with small initializations. The results of this paper are very interesting and novel, which is a good supplement to the existing results in shallow \n- The writing of this paper is clear. Specifically, the proof sketch of Theorem 1 precisely illustrates the intuition of the directional convergence in the early convergence.\n\n**Weaknesses**:\nAlthough I admire the clear writing of this paper, there still exist some minor unclear presentations that make me confused. I'm not sure whether they are my misunderstandings or typos, and I hope the authors could resolve my concerns.\n- First of all, I'm a little confused about the factor $1/\\delta^{L-2}$ in the definition of $\\bar T_\\epsilon$. Notice that according to the $\\epsilon-\\delta$ language in calculus, $\\delta$ can be arbitrarily small compared to $\\epsilon$. Therefore the first claim in Theorem 1 is a little counterintuitive. Since intuitively we usually have ``the weights are close to the initializations (bounded by a small value) at the initial training stage'', however, as the factor $1/\\delta^{L-2}$ could be extremely large, the $\\bar T_\\epsilon$ actually indicates a long training phase. Is this caused by some properties of homogeneity? For a $L$-homogeneous function $f$, I guess it holds that $x^\\top \\nabla f(x) = Lf(x)$. But I'm not clear whether such property can necessarily induce the desired result. \n- The presentation of Theorem 3 and Theorem 4 are confusing to some extent. Take Theorem 3 as an example, the authors first claim that some properties regarding the norms given the assumption $\\bar W_{1:L}$ is a KKT point, and then they provide the necessary and sufficient condition for $\\bar W_{1:L}$ being non-zero KKT point. Is this based on the assumption that $\\bar W_{1:L}$ is already a KKT point?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose a theoretical study of the directional convergence of deep neural networks with small initializations. Within the context of homogeneous neural networks, a common focus in theoretical research, the authors establish that the weights of these networks will theoretically converge in direction towards a KKT point of a constrained optimization problem or remain extremely small. Specifically, the objective function of this optimization problem is the Neural Correlation Function (NCF), which captures the inner product between the output of these neural networks and the gradient of the loss function at zero. Additionally, the authors also propose necessary and sufficient conditions indicating that KKT points of the aforementioned constrained optimization problem consist of rank-one matrices.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Could the author provide more discussions regarding their theoretical findings? Specifically, the interpretation of constrained optimization problem and the NCF function. Since compared to the classic directional convergence results [1, 2], this function and its implication is not straightforward enough.\n- I suggest that the authors could consider revising the representation of their Theorems as I indicated in the weaknesses part.\n \n[1] Lyu, K. and Li, J., Gradient Descent Maximizes the Margin of Homogeneous Neural Networks. In International Conference on Learning Representations.\n\n[2] Soudry, D., Hoffer, E., Nacson, M.S., Gunasekar, S. and Srebro, N., 2018. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70), pp.1-57.", "broader_impact_concerns": "Since this paper focuses on a theoretical phenomenon, there is no need for an ethics review.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is unanimous and strong support from the reviewers following the author’s rebuttal and paper revision.
Decision: Accept (poster)