Embedding Principle of Homogeneous Neural Network for Classification Problem
Authors: Jiahan Zhang
Yaoyu Zhang
Tao Luo
Venue: NeurIPS 2025 poster ()
Keywords: KKT Points
Network Embedding
Homogeneous Neural Networks
Neuron Splitting
Maximum Margin Classification
Primary area: deep_learning
TL;DR: We establish a KKT Point Embedding Principle where max-margin solutions (KKT points) in homogeneous networks cleanly embed into wider networks formed by neuron splitting, a property preserved throughout gradient flow dynamics.

Abstract: In this paper, we study the Karush-Kuhn-Tucker (KKT) points of the associated maximum-margin problem in homogeneous neural networks, including fully-connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths generated. We introduce and formalize the \textbf{KKT point embedding principle}, establishing that KKT points of a homogeneous network's max-margin problem ($P_{\Phi}$) can be embedded into the KKT points of a larger network's problem ($P_{\tilde{\Phi}}$) via specific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, we connect this static embedding to the dynamics of gradient flow training with smooth losses. We demonstrate that trajectories initiated from appropriately mapped points remain mapped throughout training and that the resulting $\omega$-limit sets of directions are correspondingly mapped , thereby preserving the alignment with KKT directions dynamically when directional convergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimization in homogeneous networks of varying sizes.

Embedding principle of homogeneous neural network for classification problem
Jiahan Zhang', Yaoyu Zhang 122g Tao 12,3t
School of Mathematical Sciences, Shanghai Jiao Tong University, 2 Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University, Shanghai, 200240, China CCMA-Shanghai, Shanghai Jiao Tong University, Shanghai, 200240, China
Abstract
In this paper, study the Karush-Kuhn-Tuccke (KKT) points of the associ- ated maximum-marre margin problem in homogeneous neural networks, including fully- connected and convolutional neural networks. In particular, We investigates the relationship between such KKT points across networks of different widths gener- ated. We introduce and formalize the KKT point embedding principle, estab- lishing that KKT points of homogeneous network's max-margin problem (Pa) can be embedded into the KKT points of larger network's s problem (P₃) via spe- cific linear isometric transformations. We rigorously prove this principle holds for neuron splitting in fully-connected networks and channel splitting in convolutional neural networks. Furthermore, connect this static embedding to the dynamics gradient flow training with smooth losses. We demonstrate that trajectories ini- tiated from appropriately mapped points remain mapped throughout training and that the resulting w-limit sets directions are correspondingly mapped, thereby preserving the alignment with KKT directions dynamically when directional con- vergence occurs. We conduct several experiments to justify that trajectories are preserved. Our findings offer insights into the effects of network width, parameter redundancy, and the structural connections between solutions found via optimiza- tion in homogeneous networks of varying sizes.
Introduction
The optimization of neural networks remains central challenge, with significant research dedicated to understanding the training dynamics and the properties of converged solutions. For homoge- neous networks, such as those using ReLU-like activations without biases, a particularly fruitful line of inquiry connects optimization algorithms like gradient descent the concept of margin maxi- mization in classification tasks [Lyu and Li 2019 Ji and Telgarsky 2020). these settings, the Karush-Kuli Kuhn-Tucker (KKT) conditions associated with minimum- norm maximum-manne margin prob- lem provide a theoretical characterization of optimal parameter configurations [Gunasekar et al. 2018 Nacson et al. 2019b].
While much work focuses on the implicit bias and convergence properties within single network architecture, fundamental question arises regarding the relationship between solutions found in networks of different sizes. Specifically, how do the optimal solutions (characterized by KKT points) of smaller homogeneous network relate those of a larger network, particularly one derived by increasing width through operations like neuron splitting? Understanding this relationship is crucial
"Corresponding author: zhyy.sju@@sju.edu.ccr 'Corresponding author: luotao4] @sjtu.edu.cn
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

for insights into the effects of overparamelerrizatio,, the structure of the solution space, and the mechanisms underlying implicit regularization.
To address this question, this paper introduces the KKT Point Embedding Principle. The core idea is that under specific, strueture-preservingg transformations corresponding to neuron splitting, the KKT points of the max-margin problem associated with a smaller homogeneous network (Φ) can be precisely mapped, via linear isometry T', KKT points of the analogous problem for larger network (.). This principle provides concrete link between the optimization landseapes and solution sets of related networks.
Specifically, contributions include:
1. We formalize the KKT Point Embedding Principle (Theorem 4.2 and establishing condi- tions under which a linear transformation preserves KKT points between the max-margin problems Pa and P4 (Theorem 4.5
2. We provide explicit constructions and rigorous proofs demonstrating that this principle holds for neuron splitting in full-connected networks (Theorem (.8) and channel splitting in convolutional neural networks(Theorer 4.11). showing these transformations satisfy the required conditions and are isometric.
3. We connect our static KKT embedding to the dynamics of training by proving strong trajectory preservation principle. Specifically, we show that the entire gradient flow path of the wider network is a linear mapping of its narrower counterpart (17(t) = (t)))) (Theorem 5.2). This dynamic correspondence directly implies that the asymptotic behav- ior is also preserved, ensuring the set of directional limit points is mapped accordingly (T(L(((O)) ((n( and preserving the convergence towards KKT-aligned solutions (Theorem 5.4 Corollary 5.5).
The remainder of this paper is organized as follows. Section 2 discusses related work on implicit bias, network embeddings, and KKT conditions. Section introduces necessary notations and def- initions. Section 4 develops the static KKT Point Embedding Principle and applies it to neuron splitting and channi.
the static principle gradient flow dynamics. Finally, Sec- tion [6] concludes the paper. An overview of our theoretical contributions and their relationships is provided in Figure
2 Related work
Our work builds upon and contributes to several lines of research concerning neural network opti- mization, implicit bias, network structure, and optimality conditions.
Optimization dynamies and implicit bias A significant body of research investigates the im-
plicit bias of gradient-based optimization neural networks. Seminal work by /oudryy et al. (2018) showed that for linear logistic regression separable data, gradient descent (GD) converges in direction to the 7.2-mx--aan max-ma stochastic GD [Nacson et al., 2019c), and generalized to deep linear networks [Ji and Telgarsky 2018). For non-linear homogeneous neural networks (e.g., ReLU networks without biases), studies confirm that GD/GF with exponential-taile maximization [Lyu and Li 2019 Ji and Telgarsky| 20200) Nacson et al. [2019a] Wei et al. 2019]. The resulting solutions frequently characterized KKT points of an associated max- problem
mmagin solution. This finding was extended to various settings, including
losses also typically steer parameters towards margin
Gunasekar et al.) 2018 Nacson et al. 2019c].
More recently. this line of inquiry has been extended to analyze the margin in non-homogenecus models [Kunin et al. 2023, Cai et al.) 2023) and connect KKT conditions with the phenomenon of benign overfitting FFrei et al.) [023]] These analyses, however, primarily focus on the dynamics within a single, fixed-width network, with less exploration of how such KKT-cchrrcttrrzzed solutions
relate across networks of varying widths, such as those generated by neuron splitting.
Network embedding principles The idea that smaller network functionalities can be embedded within larger ones has been explored, notably by Zhang et al. [2021] who proposed an "Embedding Principle." " This concept is related to foundational ideas on hierarchical structures and singularitics

---

Figure 1: Overview of Theoretical Contributions. This figure illustrates the logical flow of paper's ss theoretical framework. The analysis begins with the static principles (top section), which then shown to apply to specific network architectures. These static results form the basis for the dynamic analysis (bottom row), which shows the preservation of training trajectories and their limit sets.
Static Analysis: KKT points of smaller network embed into larger network
KKT Point Embedding Principle (Thm.
Characterizes
Full-conneeted Network neuron Splitting (Thm. 4.8
Special case
Equivalence Conditions (Thm.
Special case
CNN Channel Splitting (Thm. 4.11
Preserves
Preserves
Trajectory Preservation (Thm. 5.2
w-limit Set Mapping (Thm.
Limit Direction Embedding (Cor. 5.5
Implies
5.4
Special
Dynamic Analysis: Train- ing trajectories preserve embedding structure
in the parameter space [Fukumizu and Amari 2000), as well as more recent work exploring sym- metries, such permutation invariance, that connect different global minima [Simsek al.) 202T While general embedding principles focus on preserving the loss landscape. their specific appli- cation the mapping of KKT points within max max-margin settings, to the detailed embedding of entire optimization trajectories, especially where parameter norms diverge, remains a less developed area warranting further investigation.
KKT conditions in optimization and machine learning The KKT conditions (Kuhn and Tucker 1951 are fundamental for constrained optimization, extending non-smooth problems via tools like the Clarke subdifferential (Clarke 1975] (Definition 3.3), and famously characterizing SVM solutions [Cortes and Vapnik 1995] In deep learning, KKT conditions crucial for theoretically characterizing network solutions. As highlighted, they are pivotal in understanding the implicit bias of GD towards max- -argin solutions in homogeneous networks [Gunasekar et al. 2018 Nacson et al. 2019b). These studies analyze KKT points of specific minimum-nore problem (like Po in Definition(3.4) for given network. While this establishes the nature of solutions within fixed architecture, the structural relationship and potential embedding of these KKT points when transitioning between networks of different widths remains complementary and important area of study.
maximum Aaaiimm---agiin margin
3 Preliminaries
Basic notations For n E N, let [n] = {l,... [......n., n.. The Euclidean (L²) norm of a vector v is
denoted by ||>||p- For a function f Rdd + R.
V f(x) is its gradient if f is differentiable, and
8 f(x)) denotes its Clarke subdifferential [Clarke 1975 if is locally Lipschitz (Definition A.
A function f X + Rd is C* sm-sthh if it is k-times continuously differentiable, and f x R locally Lipsehitz if is Lipschitz continuous on neighborhood of every point in X. Vectors
written in bold (e.g., 0). Furthermore, for linear map T R - R and a set S C m we

---

dt denote the image of S under T as TS == (Tx E S}. Consequently, if = 8 f(x), its image is T8 f(x)= == (T' g 0 f(z)).
Definition 3.1 (Homogeneous neural network). A neural network Φ00; with parameters O is (positively) homogeneous of order L >> if: for all > 0: Φ(c0; (0; Common examples include networks composed of linear layers and positive 1-homogeneous activations like ReLU (z(2) = max(0, 2)), potentially excluding bias terms.
Binary classification setup We consider a dataset D {(<kk yk YE= where 2y E R are input features and yk E {=1} are binary labels. The network 4b @ x) maps inputs to scalar output, with parameters E IRM..
Definition 3.2 (Margin). The margin of the network + with parameters on data point is qu (0) ya (0: ax). The margin the dataset is qwin (0) = minkelni 96 (0). For an T--hmooeneeou network 4. the normalized margin is defined as 7(0) = Qmiu (f)/ 1
Definition 3.3 (KKT conditions for non-smooth problems), Consider the optimization problem
mil (() subject to gx(x) S ffr all k: E (n), where J and yn are locally Lipschitz, functions.
feasible point x² is a Karush- Kaatt-Kuhn-Tiaan Tucker (KKT) point if there exist multipliers ds 2 0, k: [n]. such that:
1. Stationarity: 0 E 8 f(x") + Ax gx(").
2. Primal Feasibility: gx(x*) S 0, for all & E [n].
3. Dual Feasibility: X 2 O, for all E [m].
4. Complementary Slackness: ^xg)(x=)) = O. for all k E [n].
Definition 3.4 (Minimum-nome max-margin problem Po). Inspired by the implicit bias literature, we consider the problem of finding minimum-norm parameters that achieve a margin of at least 1;
1
P ((0)
min OE
100
subject to (0)
ya ((0; S O, for all k [n]]
2
(0}) and the constraints involve ga (0). Un- der Assumption A.3 8 gb (0) yI 0g x (0; ).). We denote the analogous problem for a different network & as Pa- KKT points of this problem characterize directions associated with margin maxi- mization.
Here, the objective is s(0) =
101 8 (
Gradient flow We model the training dynamics using gradient flow (GF), which can be as gradient descent with infinitesimal step size. The parameter trajectory (t) is an are satis-
fying the differential inclusion d0(t) -(F(()(t) for almost every t 0. Here L())
Ex=1 ( æx)) is the training loss, where is suitable loss function (e.g., exponential loss). We denote the analogous loss for network T Ć.
4 The KKT point embedding principle: Static setting
We first establish a general principle for mapping KKT points between constrained optimization problems linked by a linear transformation. We then specialize this principle to the context of homogeneous neural network classification via neuron splicting. Throughout this section, we operate under the following assumptions regarding the networks involved.
Assumption 4.1 (Static setting). Let 4 400; x) (parameters O E RT and ( x) (parameters n E R be two neural networks. We assume: (AI) (Regulariy) For any fixed input æ, the functions D(53x) and $(:x)) mapping parameters to outpul are locally Lipschitz. Furthermore, they admit the application of subdifferential chain rules as needed [Assumption]] A.3).
(A2) (Homogeneity) Both t and & are positively homogeneous of the same order L with respect to their parameters (Definition 3.1

---

4.1 The KKT point embedding principle: a general framework
Theorem 4.2 (General KKT mapping via linear transformation). Let f.gk: R + R and f. 9x I. R be locally Lipschitz. Consider problems:
(P) min f(0) (=) _ 0, for all & E [n].
(P) min f(n) s.t. (x(n) S 0, for all ls e [n].
Let T: R" Thhh be linear. Suppose:
1. Constraint Preserving: 9v (TO) = 9x (0), for all 0 I and k [n]]
2. Objective Subgradient Preserving: 8 f(TO) TØ (0), for all TR.
3. Constraint Subgradient Preserving: 3tg 0 d g (TO) T0 go) for fol R and & E [n].
0* is KKT point of (P), then 17" = TTO" is a KKT point of (P).
Proof. Proof deferred to Appendix B.1
We now specialize Theorem 4.2 to the minimum norm max-margin problem Ps (Definition 3.4
Definition 4.3 (KKT point preserving transformation). Let 4 a)) (with parameters O E Rh and (1;; x) (with parameters n E It') be two neural networks, and let Ps and P& be their associated minimum-norm max-margin problems (as defined in Definition 3.4). linear transformation T RT + R is called KKT point preserving from P4 to Pa. if:
For any dataset D {(ER, ((Ex,Y)))< k= 1N
0* is KKT point of Ps. then n' T(0") is KKT point
of Ps.
Proposition 4.4 (Composition of KKT point preserving transformations). Let ④ x)) (parame- ters O E R. 4 ()) (parameters E Rem)), and 42 m) (parameters 722 TR be three neural networks, with inputs x E Rds, Let Ps. Pa., and Pa. be their respective associated minimum-norm max-margin problems (Definition 3.4
Suppose T1 R - R a linear transformation that KKT Point Preserving from Ps to Pa. (as per Definition 4.3y. Suppose T2 _
R is linear transformation that is KKT Point
Preserving from Ps, to Pv, (as per Definition
4.3
Then, the composite linear transformation T T207: Rmm -> R2 is KKT Point Preserving from Ps Pøy. Furthermore, ii and T2 are isometries, then T2 o Ti is also an isometry.
Proof. Proof deferred to Appendix
B.2
Theorem 4.5 (Equivalence conditions for KKT embedding). Let (0: (parameters O E IR" and (n1; x) (parameters E RR) satisfy Assumptions 4.1 (AI, A2). Let P and P1 be their associated min-norm max-margin problems (Definition 3.4 Let T: R Rd be linear transformation. The following conditions are equivalent:
1. Output and subgradient preserving: for all 0 E Rm and a E Rdd,
)(T(0); (00x)), Er(0, > S.I. (T = ((000=)((000000
2. The transformation T is KKT Point Preserving from Ps to Ps (as per Definition 4.3
Proof. Proof deferred to Appendix B.3
5

---

4.2 Applications: KKT embedding via neuron and channel splitting
4.2.1 Neuron splitting in fully-connected networks
The prineiple extends naturally to splitting neurons in hidden layers of fully-connected homoge- neous networks.
Definition 4.6 (Neuron splitting transformation in fully-connected networks). Let & be an layer homogeneous network satisfying Assumptions AI,, A2), defined by weight matrices W()) for E Ja + 1] and positive 1-homogeneous activations oi for hidden layers E [a]] Let = (Vee(W E I.' be the parameter vector.
neuron splitting transformation? T - constructs a new network & (with parameters n E IR*)) from & by splitting single neuron j in hidden layer k (1 S k S a) into maplit new neurons (using Mapist to avoid clash with parameter dimension m)) This splitting is defined by coefficients e; Z for E spt)) such that Ei=1 with the split neuron as follows:
Mipte cf = 1. The transformation T modifies the weights associated
1. The j-th row of W(K) (weights into neuron j. denoted ((l)) is replaced by maplit rows in the corresponding weight matrix W'(k) of t. For each i E [msptic] the i-th of these new rows is c.W
2. The j-th column of W(k-1) (weights out of neuron j. denoted w:lk+1)) is replaced by moplit columns in the corresponding weight matrix WK(++)) of t. For each [msptic], the i-th of these new columns is o; W (k+1)
:j
3. All other weights, rows, and columns in all weight matrices remain unchanged when mapped by T.
The resulting parameter vector for 4 is 1) TO.
two-layer network). As concrete example, the neuron splitting transformation in Definition 4.6 when applied to an = 2 layer network (i.e., a single hidden layer, 1), specializes to the two-layer neuron splitting described in Theorem A.4 Specifically, splitting hidden neuron j involves transforming its inpul weights W(1) (analogous to b to Maplit sets c,W(L) j: and its oulput weights w:(2) (analogous to a) to Maplit sets G; (.W.())
Remark 4.7 (Specialization
:.j This results in effective parameters (c,a, c;b) for each i-th split part of the neuron, consistent with TheoremAA.4
Theorem 4.8 (Properties of deep neuron splitting transformation and KKT embedding). Let T be the deep neuron splitting transformation defined in Defimition(4.) Let Ps and Pa be the min-norm max-margin problems (Definition 3.4 associated with the original nefwork & (with parameters 0 Th and the split network & (with parameters n RT)), respectively. Input data Rdd. The transformation T satisfies the following properties:
1. T is a linear isometry: ||00|, = 11011 for all 0 e R".
2. Output preserving: ((T0; x)) +b(0; x) for all E Te and Rdd.
3. Subgradient preserving: 0 x) = T(0)
0: x) for all 0 E R and a E Rd,,
(This implies T00, x)) = 1 in the context of Theorem 4.5
Consequently, since T satisfies the conditions (specifically, condition I with 1) of Theorem 4.5 this deep neuron splitting transformation T is a KKT point preserving from Ps to Ps-
Proof. The proof proceeds by verifying the three listed properties of T. Isometry (1) follows from comparing the squared Euclidean norms of the parameter vectors. Output Preservation (2) is demon- strated by tracing the signal propagation through the forward pass of both networks. Subgradient Equality (3) is established via a careful analysis of the backward pass using chain rules for Clarke subdifferentials. The preservation of KKT points is then direct consequence of Theorem 4.5 given that 7 fulfills its required conditions. The complete proof of properties (1) - (3) is provided in Appendix B.4

---

Remark 4.9 (Iterative splitting and KKT preservation). Theorem 4.8 establishes that a single deep neuron splitting operation (Definition 4.6 KKT Point Preserving (Definition Since the com- position of KKT Point Preserving transjormations also yields KKT Point Preserving transfor- mation (Proposition 4.4). follows directly that any finite sequence of such deep neuron splitting operations results in composite transformation that is KKT Point Preserving.
4.2.2 Channel splitting in convolutional neural networks
To demonstrate the generality of our framework beyond fully-connected architectures, we now ex- tend the KKT Point Embedding Principle to Convolutional Neural Networks (CNNs). Applying our principle CNNs presents unique challenge: the transformation must respect the architec- tural hallmarks of convolutions, namely weight sharing and locality. We address this by designing novel channel splitting transformation that preserves the network function and subgradient structure, thereby satisfying the conditions of Theorem 4.5
Definition 4.10 (Channel splitting transformation in deep CNNs). Let & be a deep homogeneous CNN satisfying Assumptions AI, A2), defined by a sequence of convolutional filters (weights) {W A channel splitting transformation T 0 + n constructs a new network 4 from & by splitting single output channel of a hidden convolutional layer i into maplit new channels.
This transformation is defined by a set of coefficients C; 2 0 i E [msptit] that satisfy the isometric =1 1. The transformation modifies the filters associated with the split channel condition Exmree c. ==11
as follows:
At Layer k: The filter W3 that produces output channel is replaced by maplit new filters in the corresponding weight tensor W() of &. For each i [msstit), the i-th of these new filters is defined as c.W(h)
At Layer k+1: every filter in the subsequent layer, (k+1),, the input slice corresponding to the original channel j is replaced by Maplit new input slices. For euch i E [Maptic), the i-th of these new input slices is scaled by the corresponding coefficient Ci.
Other Weights: All other filters and weights in the network remain unchanged when mapped by T.
The resulting parameter vector for के is r TO.
This meticulously constructed transformation preserves the functional output of the network while allowing for an isometric embedding of the parameter space. We now state the main result for this section, which shows that this transformation satisfies our core theory.
Theorem 4.11 (Properties of CNN channel splitting and KKT embedding). Let be the deep CNN channel splitting transformation defined in Definition 4.10 Let Ps and Ps be the min-norm max- margin problems associated with the original network & and the split network क respectively. The transformation T satisfies the following properties:
1. T is a linear isometry: ||T0||2 ||2 for all O.
2. Output preserving: (TT;;)) 40; for all 0 and input x.
3. Subgradient preserving: (T; = T(03 (0) for all 0 and input x.
Consequently, since T satisfies the conditions of Theorem 4.5 this channel splitting transformation a KKT point preserving from Ps Po-
Proof. The proof is analogous to the one for fully- fully-connected networks (Theorem 4.8 and pro- ceeds by verifying the three listed properties of T. Isometry (1) confirmed by comparing the squared Frobenius norms of the filter tensors. Output Preservation (2) is demonstrated by tracing the feature map computations through the forward pass, showing that the split signals perfectly re- combine at layer h + 1. Subgradient Preservation (3) is established by applying the chain rule for Clarke subdifferentials to the backward pass. The complete mathematical details are provided in Appendix B.5
7

---

The successful extension of our principle to CNNs validates our framework as flexible blueprint applicable to broad class of homogeneous networks.
5 Connection to training dynamics via gradient flow
Having established the static KKT Point Embedding Principle, we now investigate its implications for the dynamics of training homogeneous networks using gradient flow. We focus on the scenario where gradient flow converges towards max max-margin solutions, a phenomenon linked to specific loss functions. We connect the parameter trajectories and limit directions of the smaller network & and the larger network & related by the neuron splitting transformation T.
We analyze the asymptotic directional behavior using the concept of the w-limit set from dynamical systems theory. Recall that for trajectory z(t) evolving in some space, its w-limit set, denoted (Zo) (where Zu is the initial point), is the set of all points y such that ztxx) + y for some sequence of times 4x + co. Intuitively, it's the set of points the trajectory approaches infinitely often as t + co.
For our dynamic setting, we make the following assumptions, building upon the static ones (A1, A2 from Assumption 4.1):
Assumption 5.1 (Dynamic setting). In addition to Assumptions A.I, A2), we assume: (A3) (Loss Smoothness) The per-sample loss function E R + RR C¹ -moooth and non-increasing.
Building on this, we first demonstrate that the neuron splitting transformation T preserves the gra- dient flow trajectory itself. This result relies only on the smoothness of the loss and the network properties.
Theorem 5.2 (Trajectory preserving for neuron splitting). Let 4 & be homogeneous networks re- lated by a neuron splitting transformation T (as defined in Theorem A.4 or (.8), satisfying As- sumptions Al, A2) and Consider the gradient flow dynamics for the respective losses (()) and (T)) If the initial conditions are related by (n)) = T0()), then the trajectories satisfy (t) = TO (t) for all t 2 0 (assuming solutions exist and norms diverge for normalization where needed, e.g., as per an implicit Assumption (A4) mentioned dependent definitions/thoormms))
Proof. The proof relies on showing that the subdifferential of the loss function transforms according to T( (())) 0 ((TO). which follows from the output Preservation and subgradient equality properties of T (verified in Theorems A.4 4.8) and the chain rule applied with the C' smoooh loss & (Assumption A3 from 5.1) Full details are in Appendix C.I
This trajectory mapping allows us to relate the asymptotic directional behavior. We first define the set of directional limit points.
Definition 5.3 (w-limit set of the normalized trajectory). For given gradient flow trajectory O(t)
starting from 0(0) (with 0 E ITT under Assumptions AI, A2) and (.733))) (and implicitly assuming trajectory properties like norm divergence for normalization, often denoted as (A4) in related theorems), let (t) O(t) [[@(t)]2z be the normalized trajectory. The w w-limit set [see, e.g.. Hirsch et al. 2013 for the general theory and properties] of this normalized trajectory O(t) is defined as
= (x | s.t. tk + 0o and ((tx) + x ++000 -cc.
where 5 is the unit sphere in the parameter space T.Th of This set (0) contains all directional accumulation points of the trajectory (t). Since (t) lies in the compact set Sm- L(((0)) is non-
empty and compact. If the direction (1) converges to a unique limit 0" then w( = (0')
The following theorem shows that the transformation T provides exact mapping between the w-limit sets of the normalized trajectories. This relies on the trajectory mapping (Theorem 5.2) and the properties of T.
Theorem 5.4 (Mapping of w-limit sets of normalized trajectories). Let t. &b be homogeneous net-
works related by a neuron splitting transformation (Theorem A.4 or (.8), satisfying Assump- tions Al, A2), Assumption 5.1 A3),, and further assuming trajectory properties (A4: unique
8

---

solution existence and norm divergence for O(t) when data is classifiable). Let (t)) (in RT)) and ntt) (in R be gradient flow trajectories starting from (0) and n( T respectively. Let
L(O(0)) gm-1 and L(n(0)) S s be the w-limit sets of the respective normalized trajectories (Definition 5.3 Then, these sets are related by T:
T() = L(n(0))
where T(L(O(0))) (Tx æ L(((0))).
Proof. The proof shows inclusions in both directions using the continuity of T, the trajectory map- ping (14)= = 7 (t)) (derived from Theorem 5.2 and isometry of T)), and compactness arguments. relies on Assumption (A4) for the existence and divergence of trajectories needed for normalization. The proof does not require the uniqueness of limits. The full proof is in Appendix C.2
Corollary 5.5 (Embedding of unique limit directions). Ler 4 & be homogeneous networks re- lated by neuron splitting transformation T (Theorem A.4 or 4.8), satisfying Assumptions' 4.7 R.A. A2), Assumption A3). Let O()) and (t) be gradient flow trajectories starting from ( and (0) (0) respectively. the normalized trajectory ( = (tt/ [[((t]] converges a unique
limit direction 0 as + oo, then the corresponding normalized trajectory i(() = nn(1/ [1((2)22
converges the unique limit direction TØ
Proof. Proof deferred to Appendix C,3
Corollary 5.5 which shows the embedding of unique limit directions (TØ"), ), is particularly rele- vant given the optimization dynamics observed in homogeneous neural networks. As established in prior work (e.g., [Lyu and Li] 2019 Ji and Telgarsky 2020)), when training such homogeneous models (including fully-connected and convolutional networks with ReL.U-like activations) with common classification losses such as logistic or cross-entropy loss, methods like gradient descent or gradient flow often steer the parameter direction (tt) to align with solutions that maximize the (normalized) classification margin. These margin-man maximizing directions are typically characterized as KKT points of an associated constrained optimization problem, akin to Ps discussed earlier (Def- inition B.4p. Therefore, our corollary. by demonstrating that the transformation T preserves unique limit directions, implies that this neuron splitting mechanism effectively embeds the structure of these significant, KKT-allgned,, margin-maximizing directions from smaller network into larger one. This preservation of margin-related properties through embedding is also noteworthy due to the well-recognized connection between classification margin and model robustness.
Empirical Validation and Implications A key prediction of our theoretical results is the prin- To justify this claim empirically, we conducted ex- periments verifying this principle using discrete-time gradient descent. We trained pairs of narrow and wide homogencous MLPs on a 2D linearly separable dataset (Exp. 1), with full implementation details deferred to Appendix (D)
ciple of trajectory preservation (Theorem 5
Figure 2 presents the results. As shown in the left panel of Figure the trajectory error, [l((t) ( remains at the level of machine precision (~ 10 13) throughout training. Con- currently, the right panel shows the training loss converging towards zero, indicating successful learning process. These results provide strong empirical validation for our theoretical claim under standard GD. Appendix [D summarizes further experiments demonstrating robustness to SGD (with identical batch sequences) and non-separable data.
A direct practical implication of trajectory preservation is that the function learned by the wider network, Φ, is identical that of its narrower counterpart, 4 every training step (?(n)));--) (((()); )). This suggests that widening a network via our proposed splitting transformation can cre- ate significant parameter redundancy without altering the optimization path in function space. This result empirically supports the idea that solutions from simpler models are structurally embedded within their overparameterized counterparts.
3 Cooee available: https: //githu///tttt/../ www..////weeeoooooooococooo http:////o/oooooooooo//////e//od

---

Trajectory Error (LogY Scale)
Training Loss (Linear Scale)
Euclidean Crro IntU) TotOlls
Expomential Loss
0.
0.3
0.2
0.1
0.0
50020
100030
20000
50000
190000
Training Steps
Trining Steps
Figure 2: Empirical validation using MLP with GD on 2D toy data (Exp. 1). (Left) The trajectory error remains near machine precision throughout training. (Right) The training loss converges to- wards zero, indicating successful training.
Conclusion
This paper introduced the KKT Point Embedding Principle for homogeneous neural networks in max-margin classification. We established that specific neuron splitting transformations T, which linear isometries, map KKT points of the associated min-norm problem Ps for smaller network to those of an augmented network P4 (Theorems 4.5 4.8 4.11)) This static embedding was then connected training dynamics: proved that T preserves gradient flow trajectories and maps the w-limit sets of parameter directions (T(L()(O)) = ((n())) (Theorems 5.2 5.4). Consequently,
this framework allows for the dynamic preservation of KKT directional alignment through the em- bedding when such convergence occurs
Our work establishes foundational KKT point embedding principle and its dynamic implications, leading to several natural further questions:
What are the conditions for transformations T to preserve KKT point structures in other ho- mogeneous network architectures, such as Convolutional Neural Networks (CNNs)? This would likely involve designing transformations T that respect architectural specificities (e.g., locality, weight sharing in CNNs) while satisfying the necessary isometric and out- put/subgradient mapping properties identified in current work.
Can the KKT Point Embedding Principle be generalized to con-homogeneous neural net- works, for instance, those incorporating bias terms or employing activation functions that are not positively l-homogeneous? This presents significant challenge homogeneity is central to the current analysis, and new theoretical approaches might be required to define and analyze analogous embedding phenomena.
Acknowledgments and Disclosure of Funding
This work is sponsored by the National Key R&D Program of China (Grant No. 1008200,,
L., Y. Z.), the Natural Science Foundation of China (No. 12571567, Y. Z.), and the Natural Science Foundation of Shanghai (No. 25ZR 1402280, Y. Z.). We also thank Shanghai Institute for Mathematics and Interdisciplinary Sciences (SIMIS) for their financial support. This research was funded by SIMIS under grant number LIMIS-DD20255. (T. L.). The authors are grateful for the resources and facilities provided by SIMIS, which were essential for the completion of this work.
References
Jérôme Bolte Edouard Pauwels. Conservative set valued fields, automatic differentiation, stochastic gradi- ent methods and deep learning. Mathematical Programming, 188:19-51, 2021.
Yuanhao Cai, Yuanzhi Li, and Zhao Song. Large stepsize gradient descent for non-homogeneous two-layer networks: Margin improvement and fast optimization. arXiv preprint arXiv:2305. 2023.
10

---

Frank H Clarke. Generalized gradients and applications. Trunsactions of the American Mathematical Society, 205:247-262. 1975.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine Learning, 20:273-297, 1995.
Damek Davis, Dmitriy Drusvyatskiy, Sham Kakade, and Jason D Lee. Stochastic subgradient method converges on tame functions. Foundations of Computational Mathematicx, 20(1):1 2020.
Spencer Frei, Gal Vardi, Peter Bartlett, and Nathan Srebro. Benign overfitting in linear classifiers and leaky relu networks from kkt conditions for margin maximization. In The Thirty Sixth Annual Conference Learning Theory, pages 3173-3228. PMLR, 2023.
Kenji Fukumizu and Shun-ichi Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Nenworks, 13(3)317-327, 2000.
Suriya Gunasekar, Jason Lee, Daniel Soudry, and Nathan Srebro. Characterizing implicit bias in terms of optimization geometry. In Interationnal Conference Machine Learning, pages 1832-1841. PMLR, 2018.
Morris W Hirsch, Stephen Smale, and Robert I. Devaney. Differential equations, dynamical systems, and on introduction chaos. Academic Press, 2013.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. arXiv preprint arXiv: 3XIv:1810000032. 2018.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176-17186, 2020.
Harold W Kuhn and Albert W Tucker. Nonlinear programming. In Proceedings of the second Berkeley sympo-
sium on mathematical statistics and probability, volume pages 481-492, Berkeley, Calif., 1951. University of California Press.
Daniel Kunin, Levent Sagun, Roman Novak, Soufiane Jelassi, Surya Ganguli, and Alexander D' Amour. The asymmetric maximum margin bias of quasi-homogeneous neural networks arXiv preprint arXiv:2307.0303 2023.
Kaifeng Lyu and Jian Gradient descent maximizes the margin homogeneous neural networks. arXiv preprint arXiv: /90.009900008900, 2019.
Mor Shpigel Nacson, Suriya Gunasekar, Jason Lee, Nathan Srebro, and Daniel Soudry. Lexicographic and depth-sensitive margins in homogeneous and non-homogeneous deep models. In International Conference on Machine Learning, pages 4683-4692. PMLR, 2019a.
Mor Shpigel Nacson, Jason Lec, Suriya Gunasckar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 3420-3428. PMLR, 2019b.
Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent separable data: Exact convergence with fixed learning rate. In The 22nd International Conference Artificial Intelligence and Statistics, pages 3051-3059. PMLR, 2019c.
Berfin Simsek, François Ged, Arthur Jacot, Francesco Spadaro, Clément Hongler, Wulfram Gerstner, and Johanni Brea. Geometry the loss landscape overparameterized neural networks: Symmetries and invariances. In International Conference on Machine Learning, pages 9722-9733 PMLR, 2021.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018.
Colin Wei, Jason D Lee. Qiang Liu, and Tengyu Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Syxtems, 32, 2019.
Yaoyu Zhang, Zhongwang Zhang, Tao Luo, and Zhiqin Xu. Embedding principle of loss landscape of deep neural networks. Advances in Neural Information Processing Systems, 34:1 1848-14889, 2021.
11

---

A
Mathematical preliminaries and illustrative example
Definition A.1 (Clarke's subdifferential). For a locally Lipschitz,function f X - E. the Clarke subdifferential(Cann 1975 at E X is
8 f(x) = conv
lim k-toc
vf(as) Tk + x is differentiable at 2k
where conv denotes the convex hull. If f is C. near f(x) [Vf(z)).
Definition A.2 (Are and admissible chain rule). We say that function an are if is absolutely continuous for any compact sub-interval of For an are 2, ²(t) for d dt. =t)( stands for the derivative at if exists. Following the terminology in Davis et al. [2020J, we say that a locally Lipschitz function f Rd - I admits a chain rule if for any are (00+ + Rd, for all h 0 f. the equality of (h, E(t)) holds for a.e. t t> O.
TR& the interval
Assumption A.3 (Admissibility of subdifferential chain rule). We assume throughout that the neural networks and loss functions involved are such that standard chain rules for Clarke sub- differentials apply needed for backpropagation. Specifically, for compositions like L(0) Ek-1 1 yy & (0; æx)), we assume the subdifferential d [(0) can be computed by propagating sub- gradients layer-wise. Conditions ensuring this are discussed in, e.g,, Davis al. 2020 Bolte and Pauwels (2027).
Theorem A.4 (Two-layer neuron splitting preserves KKT points). Consider a single hidden neuron network $(0; x) = ao x) with parameters 0 (a.b") E Rl+d, (where a E R. b E RRdz), and is positive I-homogeneous activation function (e.g., Rel.U) satisfying Assumption A.I. Consider a network Φ(1;2²) Σx abo x) with parameters 1 = (R((1+d2), Define the linear transformation T Rl+d, + R+(1+dd) by T(a, b (caa...., cra,e,, ....,,,b"] where 0 are splitting coefficients satisfying Σ of 1. Then satisfies:
(a),... ak. ,..., b
1. Output preserving: ((T0;)) = z). all 0 E R Il+dz and x E R
2. Subgradient preserving: "T, x) T(0g x)). for all Rl+d, and Rd,
3. Isometry: T is a linear isometry (1) [1T0| for all O R Bl
Consequently, by Theorem 4.5 this neuron splitting transformation is a KKT point preserving from Ps and Pū.
Proof. The proof involves directly verifying the three properties using the definitions of 4, 4. T. and subdifferential calculus. Let 0 (a, b b).
12

---

1. Output Preservation: We compute o at r T0:
$TT0; x)
a..(b, x)
(Ga))((())))))
(Substituting transformed parameters)
Σ (c)a) x))
Σ (caa)(caabb x))
(Using 1-homogeneity of
k c
ab x)
1 ao (b x:)
(Since Σ
((0: x))
Thus, the network output preserved.
2. Subgradient Preservation: Let bT xT. The Clarke subdifferential of + w.r.t. 0 is 08 x)) = (0. )) where 0- = [a(2) and 0z (a a | E (=)))) Applying T an element of (0; x) yields vector with components corresponding (2)0())...... (2) for the 'a' parts and (c. (ciage,.... Cxagæ) for the "b" parts. Next, we compute the subdifferential of at = TO. For each component j E [k]:
0 = {o(b] x)} = {o(c==)] = {cjo(z}}
0g = {ag 93 x 93 E O x)} = {eja - x E 0
The equalities follow from the I-homogeneity of a and the O-homogeneity of its subdifferential a. Assembling these partials for chosen 02) shows that any element of 0; @T, x>) matches the structure of an element in T (0; x)). Thus, the sets are identical.
3. Isometry: We compute the squared Euclidean norm of TO:
TO 2
Σ
(Ga) 3
b
| 2
(a² + (b(l))
(Since Σạ=1
|
Since ||00| ||0|2 for all 0, T' is a linear isometry.
B Proofs from section 4
B.1 Proof of theorem 4.2
Let be KKT point of (P). We aim to show that 17" T satisfies the KKT conditions
(Def. 3.3 for problem (P).
Primal Feasibility (for P): By definition, 0" is feasible for (P), meaning ga (0*) < for all k E [n]. Using Condition 1, we have (x""")) = (Te") = (0") < 0 for all l E [n]. Thus, nn is feasible for (P).
13

---

2. Stationarity and Dual Variables (for P): Since 0 a KKT point of (P), there exist dual variables
A 2 0. k E [m], satisfying complementary slackness ( = 0) and the stationarity condition:
O E f(0") + Σk= dx g (0") This means there exist specific subgradient vectors ho E
0 f(0") and ha E 8 (0") for k E [ra] such that ho + Σn=1 Axh2 O.
Applying the linear transformation T to this equation yields: T(ho + Ek=1 Arha)) = Tho + Σx=1 Tr = TO 0.
Now, we relate these transformed subgradients to the subgradients of and ğr at nj =TO.. From
Condition 2, since hg E 0 f(0"), we have The E T8 f(0)) = 8 f(T0") = 0 f(nn") Let h6 = The- From Condition 3, let 17. = L (0*) > 0 for each k E [n]. Since h2 E 0 gu (0"), we have The E T (0"). Using Condition 3, TV gb (0") = (1/tg)00 (1/f))0'x((0"))) Therefore, The (1n")))"'((n")) This implies that hig = The is an element of 0 (n()).
Define new candidate dual variables for (P) as unn = Au/tt for [n]. Since A 2 and t > 0, we have μ 2 0 (Dual Feasibility for P holds). Substitute The = (1/11) /// into the transformed stationarity equation:
Tho
+
AL
h k
= 0
h
h =
h/ +
Hah = O.
k=1
k=
k=1
Since hig E 0 f(()") and h's E (1)) for each n, this demonstrates that 0 E 8e f(n") +
Ek= p2 (.)). The stationarity condition holds for '7" with multipliers p4-
3. Complementary Slackness (for P): We need to verify pag0 (17" 0 for all k E [n]. Case 1: A = 0, then Pi = Ax/Eī = 0, so fs.9k (n)") = O. Case 2: If X2. > 0, then by complementary slackness for (P), we must have ga (0") = O. By Condition 1, (n"" (TO") go (0)) 0. Thus, p4.x (17) p4 0 O. In both cases, complementary slackness holds for (P).
Since '7 is feasible for (P) and satisfies stationarity, dual feasibility, and complementary slackness with multipliers μkg it is a KKT point of (P).
B.2 Proof of proposition
Let Pq, Po., Pa_ be the minimum-norm max-margin problems, and let the linear transformations T : Rm + Rm and T2 Im + R be as defined in Proposition 4.4 We assume Ti is KKT Point preserving from Ps to Pe.. and T3 is KKT Point Preserving from Ps. to Pa, (as per Definition 4.3). Let T = T3 o Ti. Our goal is to show T is KKT Point Preserving from Ps to Pa..
Consider arbitrary dataset D and arbitrary KKT point 0" Th of Pa. Since Ti is KKT Point Preserving. Ti (i0))) ) is a KKT point of Po. Subsequently, since T2 is KKT Point Preserving and T, (0") is KKT point of Pe. ( (0")) is KKT point of P As T (TT(()0)) = (T2 (i)(O") T(0") ((")), it follows that T(0") is KKT point of Paz. This fulfills the condition in
Definition 4.3 for T to be KKT Point Preserving from Ps P4.
Isometry Property: If T; and T2 are linear isometries, then T = T2 o Ti is also a linear isometry. This follows directly: for any 0 E Rm
|2(()) (T (0)( 11T (TT) ||T (0) 12 11012
The third equality holds due to TV being an isometry, and the final equality due to T3 being an isometry.
B.3 Proof of theorem
4.5
We apply Theorem 4.2 with (0) = 1 11012 au (0) = 1 ys (0; æx) for problem (P) = Ps. and f(n) = 1/2 || (n) = 1 017: xx) for problem (P) Pg-
Proof (1) = (2): Assume condition (1) holds. We verify the premises of Theorem 4.2 Constraint peetervinge... (TO) = 1 1 ye (TT; axx). By assumption (7) = $(0): mx), so
0u (T0) bb; ax) 9% (0). This holds.
14

---

2. Objective Subgradient preserving: f - {0] and 0 f(n) = (n}. We require 0º f(TO) =
TO f()). which translates to {TØ} O {TØ}. This holds trivially because T is linear.
3. Constraint Subgradient preserving: We need (T0) = 4. (0)78 gu (0) for some lg > 0. Us-
ing the definition of 9k and properties of subdifferentials (including Assumption A.3 0 gx
0 - y/ (0; xx)) = 0 2000000000000 æz). Similarly, (T0) = (1 - yı 7y a I=
Sub- stituting this into the expression for 0 (TO): "x)) -M: [=00,mm)T((3000:zz:)) Since T linear and r > 0. yk can be moved inside the transformation T: TT= 23 -1
- (T0 æa). From assumption (1), we have 0T; =000,za][(05500::0
= xxx ( ( 9 (0). Thus, Condition 3 of Theorem 4.2 holds with (x) T(0, (0 > 0.
Since all conditions of Theorem 4.2 are satisfied, (2) follows: if 0" is a KKT point for Ps. then TO"
is a KKT point for Po-
Proof (2) = (I): Assume (2) holds: KKT points are preserved for any dataset D.
1.Output Preservation: The preservation of KKT points implies the preservation of primal feasibility and complementary slackness. For any active constraint n at a KKT point 0" (i.e., ga (x())))))0))) 0), we must have Gn. (TO") = O for TO" to be KKT point of Ps. This means yu (0"; xxx) 0 implies 1 - yo (T'0 xxx)
O. This requires (00"; xxx "TT0"; "000=";;, :xx) whenever constraint is active at a KKT point. Assuming KKT points are sufficiently distributed, this suggests the general
identity 400; x)) = (TTO;
2. Subgradient Proportionality: Comparing the stationarity conditions 0 (0') +
Ek=1 )x (0 rx))) and E [TO"} + Ek=1 Mn(- "0 (TT" =x)) for corresponding KKT points 0" and T0" leads to Σx=1 Hn. = T(E h,) where h', E 0g. and hn E 0g For this equality and the relationship between multipliers (Um, = >ς/T,) to hold univer- sally across datasets and active sets, it necessitates structural relationship between the subdifferen- tial sets themselves, namely 0r, (TO": :2)) T (ag æx)) for some Tn 0.
The final statement regarding = and isometry for the specific neuron splitting transformations T is proven directly in Theorems A.4 and 4.8
B.4 Proof of theorem 4.8
The theorem states that the neuron splitting transformation T' for deep networks (as defined Def- inition 4.6, leading to Theorem 4.8 (1) is isometry, (2) preserves the network function, and (3) maps subgradients accordingly, i.c., 02 (TO0 x) ((0 æ)). We prove each claim.
(1) Isometry The squared Euclidean norm of the parameters 0 = (vec(W()).... vec(W((+))) E Σr, (w:!))) The transformation T 0 + 17 only modifies
is |012 M || weights related to the split neuron j in layer k. Specifically, affects the j-th row of (()) (denoted w!!h)) and the j-th column of ((++)) (denoted w: (((+))), All other weight matrix elements unchanged.
The contribution of w:(k) to 10012 is [//b0 Under T, this row is effectively replaced by m rows in W where the i-th such row (corresponding to the i-th split of neuron j) has its weights scaled (i.i).s (W((k) The total contribution of these m new rows to 1l is Em - E. j.a E o (w/ + Em of 1. Thus, the contribution from weights leading into the split neuron (or its parts) is preserved.
by c compared to (-(k) (i.e., woike
j.8
W (2)
[<))) since
1
j.: 12"
(k++) Similarly, the contribution of w:(h+1)) to |101 is |w:(ht)) Under T, this column is effectively replaced by m columns in W1(++1), where the i-th such column (weights from the i-th split of neuron j) has its weights scaled by (i.e., (.(.)) =c.,((+1))) Their total contribution to ||ml|
is Σ E, Eaawwtt)) ((+1)) preserved.
(2 cf) | (1+11) =1 ((+1)) This contribution is also
2
15

---

Since the norms of the modified parts are preserved and all other weights are identical, ||m7122 =
|TO|2 |00| Thus, T' is a linear isometry.
(2) Output preserving We trace the forward signal propagation. Let x()) and z denote the activation and pre-activation vectors at layer / for network ((0; and x 2 for ((TO;)). The relations are z ((0x(111 and x(1) (z) (with x(0) Xinput and 0a+1 being the identity for the output layer).
For layers I < k: w W Since x (0) (0), by induction, z000 z and x10 x()) for I < k.
At layer k: The input is x (()-1), The pre-activation is z((k) W 1), For unsplit
neuron p' in & (corresponding to neuron p * j in Φ), the p'-th row of w W So, (())
Ap'
wi6h x (k-1) zp. For the i-th new neuron (j. i) in & (resulting from splitting neuron j in 2), its
[(k) corresponding row in W(()) is c:W So, (G.)) (ca (1) )x((-1) (W, 5: x(*-1)) 422 (8)
The activation x()) = (z((kk) is then: For p' # (unsplit), z (1)) = =ox(,p)) (()) = ok (cp ) = xp (k)
For split components (j. (1,))
(k)) = 0x (k)
(1.)) = )x(G2)))) Since e 2 and 0g is positive
I-homogeneous, this equals (2 (k) c.jj (k)
At layer k 1: The input x((k,, The pre-activation is z((+1) ( Consider the p-th component 2p ((+1))
1(++1) pp
Σ
wo(k--)) 1) (R)
Σ
m
w((++1) (() p.(j.)) (i.i)
pio'
Eeg
4'' unsplit
m
((++1) (1)
+
((a (atw((++)) (aaag))) PJ
(by definition of T and results from layer k))
P.q
9
97.
i=1
m
W(++1) (k) (x()))
+
Σ
cg
P.9
9f1
Σ
W(++)) (k))
q
+
w((k-1)
p.j
Tjj
Σ
W(++1)
pia
= = (k+1).
{Fj
9
Thus, z (1+1)) = z (k+1), which implies x (k+1) = x (++1)
For layers 1 > h + 1: Since inputs (1-1) = (t-1)) and weights W W(O are identical, all subsequent activations and pre-activations z((1, x(() will be identical to z x() Therefore, the
final output is preserved: (TTO; x) = ((;x))
(3) Subgradient preserving We aim to show that (T,(T0;;)) = (0)))(0:z) This means that any element g' E (T0; x) can be written as Tgg) for some g E 0g x)), and vice versa. We use backpropagation for Clarke subdifferentials (Assumption A.3 Let (()) be an element from 0 ga & (subgradient of final output & w.r.. pre-activations ( e from 2 D, and G
from go Way Φ. Primed versions (8"(1, e e0(), (r(l) are for क. The backpropagation rules are: e(l) (W(++)) g(++1) (for an element choice). ((()) E @Poo(ze))) for each component s. G(L) 80) (x((xx (x() (outer product).
Step 3.1: For layers / A + 1 (above the split output) Starting from the output layer + 1: 1((++)) ((a+1) (e.g., {1} if taking subgradient of scalar & w.r.t. itself, or the initial error signal
from loss). Since z(()) (!) and Wi(+1) W(+1) for I > k + 1, by backward induction,
8n0 =80h 80)) and e!()) = e for all k 1
Step 3.2: For layer & (the layer of the split neuron) The error w.r.t. activations (h) is e(()) =
(W*+++)) T 8((++1), Since 5'(++1)- = 5(++1),
16

---

For an unsplit neuron p' in & (corresponding to * j in 4): The p' th row of (W()+
(i.e., p'-th column of l w So, op'
(k) (W:ltt)) (((tt)) cp (k)
For a split neuron component (i.z) in Φ: The (i.i)-th row of (WIltt1) (i... ((k) (j,i)-th column of W is a. ( So, G.i) - (e. gllet) Tollett))
(((WW(h))) (8((=)))== = =c,
Thus, e(kk has components ep
(k) for split neurons. Now, for errors (k) for unsplit neurons and city w.r.t pre-activations z((k), where o7(h) En (= )e?
For p' j:: 2/
(k) = 2p
(k) (k) and ev
= ep So, oghh E 0'
(2) )ep meaning (()
)
(k) (Assuming consistent choice of subgradient element from d o)).
For (i.t): (G.)
((k) = az2 (R) and (())
(R) (1.)) = aee,)) Since ok is I-homogeneous, 0 0% is 0-
homogeneous (i.e., 0 ('z)(c)) dor(2) for > O: this property extends to q > 0
(k) appropriately for ReLU-like activations). Thus, d'ox(112) ( = 0o )k) So, 0 (30) E
8 )z (H) (eef c:(8 (z)))))) This implies (((]] c,0)t (1.)
So, (*) has components 6/²) for unsplit neurons and c,0y for split neurons. Step 3.3: For layers 1 < k (below the split neuron) The error w.r.t. activations at layer A - 1.
e((k=1), 1), is (Wi(k)) 8*(k), A component of e((k- is:
1k--1)
Σ (wilk)
(W
eg
G.0), (3.8
unsplat
Σ P71
wrlk) g(k)
+ Σ
Wa (crof(o)
(using definitions of w*(k) and results for 5 ((())
1=1
m
Σ W(A)
+
Σ
c
wrihige 8(8)
pss P
w(k) g(k)
+ w(()
(K) g(h)
Σ W( = (1--1)
Thus, (k- = (k-1)) Since W = w()) for l < k. and x (((11) = x(( for 1 S k- 1, by backward induction, 8¹(() = - and e!!)) = ell) for all [ << k.
Step 3.4: Parameter Subgradients ()(l) and (( The subgradient G(l) is 8l (x(1-1)) and (*(()
is 8((() (1)--)))T
For l & k 1): Since 8((1) = 8(1) and (111 (1- it follows that G'(l) GO)) This matches the action of T on these unchanged weight matrices.
For = A (weights ((k) into the split layer): x (1(=1) = (1)1)) For rows p/ + in W((A)
(unsplit neurons), Gp (k()) (x(( oy (x(+-1)) = c/hl). For rows corre- ((())
sponding to split neuron (j. in Wi(),, G((k) = 6.0) (x(+-1)) (xx
(1,));:
GGG. This means the subgradient matrix (T(() has rows ((k) for + j. and m blocks of rows G. (where Gj is the subgradient for original row (K) corresponding to how T transforms G(k)
For 1 k (weights ((++1) out of the split layer): (*++1) = (1++)) 1) For columns p' o
j in W*(++1) (unsplit neurons), C((k++)) = 8 8((+1)) (())) TT = ((x+1) (xx))) T (k++1)
For columns corresponding to split neuron (j. i) in W((++1, (.(k+1) ((i))) = (k(+1) -2(+t))21 (G.)))
186)
17

---

8(h-1) (1)) (caa)) c, This means G'(k-1) has columns ((++1) for + j. and m
blocks of columns e,
1k+ corresponding to how T transforms G(++1)
Step 3.5: Conclusion on Subgradient Sets The above derivations show that for any choice of subgradient path (i.e., selection of elements from 8a at each gate) in calculating an element g {GO) E 0g the corresponding path in & yields an element g' = { 8º such that its components G*() are precisely those obtained by applying the structural transformation T to the components GO)) of g. Specifically, G*0 = G for / $ {k, k + 1]; Gr(k) has its rows transformed as 7 acts on rows of Wi() (scaled by <; for split parts); (*(++1) has its columns transformed as T acts on columns of W7(+1) (scaled by q; for split parts). This structural correspondence for arbitrary elements implies the equality of the entire sets: 0g (TO; x) = T (0; x)). The operator T(-) on the set 0g (0;; æ) is understood as applying the described transformation to each element (collection of subgradient matrices) in the set.
B.5 Proof of Theorem 4.11
The theorem states that the channel splitting transformation T for deep CNNs (as defined in Defini- tion 4.10 leading to Theorem 4.11 (1) is an isometry, (2) preserves the network function, and (3) maps subgradients accordingly, i.e., 0g (T0;2)) =Tag a))) We prove each claim.
Notation for CNNs. We denote feature maps (tensors) with capital letters. Let x(1) and ( be the activation and pre-activation feature maps at layer l. The p-th output channel of the activation map is X (t) The forward pass is defined by Zp Egp wPt) x! and x or (z(2)) ). where * denotes convolution. The parameters 0 are the vectorized collection of all filter tensors (W()). We use primed versions for the split network कb
(1) Isometry The squared Euclidean norm of the parameters 0 = (vec(W())),... vec(Waat)))
is | Di= w00 F. The transformation T only modifies filters related to the split output channel j of layer le and the corresponding input slices of filters at layer k + 1.
The contribution of the filter wi(k) (all filters producing output channel j) to |00 is
3: p
Under T. this is replaced by Maplit new filters (.,.() The total contribution of these new filters
(R)
to || is Emp (
(A)
Emerii ( (k) 1.
(k) J::
since Emante 1. F'
F
This part of the norm is preserved.
Similarly, for any filter l+( at layer k 1, its j-th input slice (K+11)
contributes (w(++1) to
the norm. Under T, this is replaced by meplit new slices e; (k+1)
Their total contribution to
|1m11
across all filters at layer k + is Ep Dmpit s
I (k+1)
2
-EE
Mepiit e))
[ (k+1))
=
[ I|= This contribution is also preserved.
1. 1,E,
Since the norms of the modified parts are preserved and all other weights identical, ||ml1 |TO 2 |00|2 Thus, T' is a linear isometry.
(2) Output preserving We trace the forward signal propagation. For layers I < k, weights and inputs are identical, thus (i) = (1) for 1 < & by induction.
At layer k: The input is (I-1)) x 1), For unsplit output channel p * j. Zz
[k]]
Eg p.
x-"=11 !) - For the i-th new channel (j, the filter is (.W)((). So. z/(B) [, jid
x(k-1) = =. The activation (*k)) then: For p # j. x((k) 0x (z(( (k) For
18

---

split components (i.i),
x (15)
) coal(2 = cx, since og is positive 1-
homogeneous.
At layer & + 1: The input is X
The pre-activation for any output channel p is:
Mapist
71(k+1) 7
E
w p.g"
x
w p.(j.))
x G.))
q6 unsplit
Mapiie
Σ
1(++1)
* x +
(otW Pis
(ciX
(by definition of T)
9#
Maplie
Σ
1i pa
x(() + Σ
(Wi(h+1) x((11)
PJ
9#1
Σ w(++)) x + p.j
x(()
2 1(x+1).
Thus, Z((++) (k+1), which implies x (k+1), For layers 1 > k + 1, all subsequent activations are identical. Therefore, the final output is preserved: (T00; .) +(0;
(3) Subgradient preserving We use backpropagation for Clarke subdifferentials. Let A (t) 0g 2 & and E(') E ()) 4. Primed versions are for 4. The backpropagation rules involve convolu- tions with fppped filters.
Step 3.1: For layers 1 2 k + Since the forward pass is identical for 1 2 k + 1, by backward
induction, the subgradient error signals are also identical: A (() A (1) and E*() EE() for all 1 12 ++
Step 3.2: For layer l The error w.r.t. activations (k) is ( backpropagated from A (1++))
A (k+1) through (*(+1),
For an unsplit channel j, the error is sourced from unchanged filter slices, so Ex (k) E(k)
For split channel (j, the error is sourced from the scaled input slices e, (. By
linearity of the backprop operation, ((..) c,E(%)
The error w.r.t. pre-activations (1) is ((k)
Fo j. (x( z and Ex(0) k thus A7AA) (1)
For (1.i), 2((b) = c, (k) and (..) = c.E. (k) Since Pog is O-homogeneous,
(cx((1) 8a o (2)))).. So, an element of 00 200
+ is given by an element from (3.0
(GF( 00 oz (2) which implies 110) 5,
Step 3.3: For layers k The error E' is backpropagated from A (k) through W*(k), The error contribution from unsplit channels p + j is preserved. The contribution from the split channels
3,2 sum over the new filters (c, (c)w((t) and new errors ( (1)).. The backprop operation results in
a sum over e², which equals 1. Thus, the total error is preserved: (k(k-1) = (k-11), By backward induction, all errors for are identical.
Step 3.4: Parameter Subgradients (*(1) and G!!) The subgradient ( is computed from A (1) and x (1-1)
For l & {k, k + 1}, since errors and activations are identical, (*() G()
For k: The subgradient for the new filter producing channel (j. i) computed from in-
put ())- (k-1) and error A 7(k)
(ja) c; A This yields (()) = c.G. matching
(j,)):
how 7 transforms the filter subgradients.
19

---

For 1 = k + 1: The subgradient for the new input slice (j.i) i) of any filter w is computed from input X x
(i.4) xx (k) and error A ((++1) - (k+1), This yields G((++1)
p.(..)
(G matching how T transforms the subgradients of the filter input slices.
Step 3.5: Conclusion on Subgradient Sets The derivations show that for any choice of subgradient path, the resulting subgradient tensor collection {G"()) is precisely the transformation applied to the original subgradient collection {G This structural correspondence implies the equality of (T0:((000)))
the entire sets: 0g & ( x) = ((00
[T
C Proofs from section 5
C.1 Proof of theorem 5.2
We aim to show that if (()) solves de dt E -(0) then nn()) = =T((() solves dn ar E - ((n()))
We have dn T de. We need to show T T(-0F((0(t)) -(T(((())) which is equivalent to
((8CC(0) = 0 ((T)).
Using the chain rule (valid under Assumption 5.1 A3) A3) as f is -smoooth) and properties of I' (net- work equivalence ((T0: x) = ((0; (0:z) x) and subgradient equality 0r (T0 = T(0 ((; x)) from Theorem 4.8 (which relies on Assumptions [A,, A2)) or Theorem A.4 as appropriate):
8 PiTTT)
El (w
ll (ww. (Using Thm. 4.8 or A.4 properties)
T
(yn
(by linearity of and sum rule)
kk=
T (())).
Thus T(-))L(0)) - L(TO). If de E ((B(()) then dy do E T((O8'
de - ((TO()) - - (t)) With matching initial conditions 17(0) = TØ(O), and assuming
unique solutions exist for the gradient flow (as per Assumption A44), the trajectories coincide: (tt) = TT(() for all 2 0.
C.2 Proof of theorem
We need to show two inclusions: ((()) C L(n(0)) and L(n(0)) C T()() This proof requires Assumptions A.I, A2) and 5.1 A3, A4).
1. Show T(L(O(0))) s I(n(0)): Let x E L(O())). By Definition 5.3 there exists a sequence to such that the normalized trajectory O(t+) (tx))/ (((x)) II converges to æ. Note that [((x)]]l + oo by Assumption 5.1 A4), so normalization is well-defined for large k. From Theorem 5.2 have (t)) (T)).. Since T' is an isometry (proven in Theorems A.4 4.8
[|m())]2 - [TO(()), = [0(()] Therefore, the normalized trajectories are related by:
(tt)
TO(t) T O()) (T(O)TT))
(t) = [n(1122 10(11)22
T(Ō()))
Now consider the sequence (tx)) T(O(t+)). Since T is linear transformation in finite- dimensional spaces, is continuous. As + co, 0(tx) + x implies T(0(t))) - T(x). So, have found sequence t4 such that (7tx) + T(x). By the definition of the w-limit set, this means T(x) E L(n(0)). Since this holds for any arbitrary x e L()(0)), we conclude T(L(O(O)) C L(n(O)).
Show (((((O)) S T(L(((())):: Let y I(n(0).. By Definition 5.3 there exists a sequence t + such that (7(+))) y. We know 7(()) T(O(t)). The sequence (0(t') (f(t())s k lies on
20

---

the unit sphere gm in RT, which is compact set. By the Bolzano- Weierstrass theorem, there must exist convergent subsequence. Let (ty, 1==1 1o0 be the indices of such a subsequence, so that ((12)) + x' for some x' E gm-1 as j + co. Since 122 + 0 as j + co, the limit point x' must belong to the w-limit set of the original trajectory 0(t), i.e., x' E L(((0)). Now consider the corresponding subsequence for n): ( Since T is continuous, as j oc, 0(12,) x' implies T(0 (12, )) + T(x).. So, ((()))) + T(x'). However, the original sequence 77(1)) converges to y. Any subsequence of convergent sequence must converge to the same limit. Therefore, we must have y T(z'). Since y was an arbitrary element of [(m()),, and found element æ' & L()(0)) such that y T(x'), this demonstrates that every element in L(v(O)) is the image under T of some element in (())))). Hence, [(n(0)) C (TL(0))))
Combining both inclusions yields T(T(((O))) L(n(0)).
C.3 Proof of corollary 5.5
If the normalized trajectory O(t) converges to a unique limit 0 then by Definition
5.3 its w-
limit set is (()())) = {Ø'}. Under the specified assumptions in Corollary 5.5 (which include
Assumptions 4.1 A1, A2), Assumption 5..mmpptioooS[A33)33. and trajectory properties (A4)), Theorem 5.4 states
that (()))()) = L(m(0)). Substituting (())))), we have L(n(()) T((0") })- = (TO }. Since
the w-limii set L(n(0)) consists of the single point TØ", this implies that the normalized trajectory (t) converges to TØ as t + co.
D Experimental Details
To validate our thcoretical results on trajectory preservation (Theorem 5.2 we conduct a series of experiments corresponding to the results summarized in Table I and Table 2 We use PyTorch for all implementations. The core methodology involves training a pair of networks narrow network and its wider counterpart & constructed via our transformation and measuring the trajectory error [t) - at each step.
D.1 Experiments on 2D Toy Datasets
Dataset Generation. We use two-dimensional toy dataset consisting of 100 samples from two classes (+1). For the separable case (Exp. 1, 3, 4, 5), the data points are generated from two Gaussian distributions centered at -2) and (-2, 2), ensuring they are separable by a line passing through the origin. For the non-separable case (Exp. 2), the distributions are centered closer to the origin at (1, 11 and (-1, 1) to create overlap.
Model Architectures. We test both fully-connected (MLP) and convolutional (CNN) networks. All models are homogeneous, using Leaky ReLU with a negative slope of a = 0.01 as the activation function. Weights are initialized using Kaiming Normal/Uniform initialization as detailed in the code.
MLP (Exp. 1-4): The narrow network is 2-layer MLP with single hidden neuron (Input(2) + Hidden(1) - Output(1)). (I)) The wide network splits this to two hidden neurons (2nput(2) - H2dden(2) - Output(1)).
CNN (Exp. 5): The narrow network takes 1 x 5 x 5 input and consists of convolutional layer with 2 output channels (3x3 kernel, stride 1, no padding), followed by an average pooling layer (2x2 kernel, stride 2), a flatten operation, and final linear layer to produce single output. The wide network splits the convolutional layer to 4 channels, adjusting the subsequent linear layer input dimension accordingly.
Training Setup. Networks are trained for 100, 000 steps using an exponential loss function. The learning rate is 0.1 for MLP experiments and 0.001 for the CNN experiment. For Gradient Descent (GD, Exp. 1, 2, 5), the full batch is used in each step. For Stochastic Gradient Descent (SGD, Exp. 4),
batch size of 16. Crucially, for the identical batch experiment (Exp. 3), both networks
21

---

are fed the exact same sequence of
whereas for the different batch experiment (Exp- 4), they use independent data loaders. The initial parameters of the wide network are always set to (70) = (0( using splitring coefficients G; = = / Maplit
Results.
The numerical results for maximum trajectory error are summarized in Table] [ Figures 3
and p visual confirmation and further details. The error remains near machine precision 10 13)
cases where the theory predicts preservation (Exp. 1, 2, 3, 5), regardless of data separability or the use of SGD (with identical batches). In contrast, when the SGD batch order differs (Exp. 4), the error grows substantially, confirming necessity of identical training paths. The slightly higher emror floor for the CNN (Exp. 5, 10-19) is consistent with the expected accumulation of floating-Pn point errors due to the higher computational complexity of convolutions.
Table 1: Complete results for maximum trajectory enror on 2D toy datasets.
Exp.
Model
Optimizer
Data Condition
Max Trajectory Error
MLP
GD
Separable
4.46 x 10-13 13
MLP
GD
Non-Separable
5.57 x 10-13
MLP
SGD (Identical batches)
Separable
4.35 x 10-13
4
MLP
SGD (Different batches)
Separable
1.53 x 10-1
CNN
GD
Separable
2.36 x 10-10 10
Trajectory Error Comparison (MLP, GD)
Exp Soparable
Exp Non Separable
10-1 13
10-11
Burliden
10-13
12:44
40000
150002
20002
40000
62000
100300
Steps
Figure 3: Trajectory error comparison for MLP experiments using Gradient Descent (GD). (Left) Exp. separable data shows error machine precision. (Right) Exp. non-separable data also shows error remaining near machine precision, demonstrating robustness to data conditions.
Trajectory Error Comparison (MLP, SGD)
Exp Identical Batches
Exp Different Batches
00-11
0.5
Bucilown Emot
10-11
10
Movino Aueraoe (Rirdis 5001
20000
50003
003c0
100350
6
23600
60003
60260
80200
100300
Training Stops 1D
Training Stops It)
Figure 4: Trajectory error comparison for MLP experiments using Stochastic Gradient Descent (SGD). (Left) Exp. 3, using identical mini-batches for both networks, maintains error near machine precision (moving average shown). (Right) Exp. 4, using different mini-batches, shows substantial error growth, highlighting the necessity of identical data sequences.
22

---

Experiment
Trajectory Error vs. Steps
Training Loss vs. Steps
30
10-11
Expar evetat
10-14
20000
00200
CO030
30003
100303
20020
12003
63000
30300
100003
Training Stops (1)
Traning Staps 13
Figure 5: Results for the CNN experiment using GD on separable data (Exp. 5). (Left) The trajec- tory error remains small (< 10-9) but exhibits a slight upward trend, consistent with accumulated precision errors. (Right) The training loss converges successfully.
D.2 Experiment on the MNIST Dataset
Dataset and Models. To test principle more realistic task, the MNIST dataset, focusing on the binary classification of digits '3' versus '5', We use larger homogeneous models: an MLP splitting from 128 256 hidden units, and CNN splitting from 10 20 channels (detailed architectures follow standard practices, similar to the toy CNN but scaled appropriately for MNIST image size).
Training Setup. Both models are trained for 1000 epochs using SGD with learning rate of 0.001 and batch size of 64. Data is normalized. Identical mini-batches are used for the narrow and wide networks at each step.
The trajectory preservation principle continues hold with remarkable precision this practical task. The maximum trajectory errors, summarized in Table 2 remain close to machine precision for both architectures. Figures/[andd providd a visual representation of the dynamics. The left panels show the trajectory eтrоr remaining consistently low throughout the 1000 epochs. The right panels display the corresponding training loss (evaluated on the full training set), confirming that both models learned effectively, achieving high final classification accuracies (MLP: 99.99% CNN: 99.82%). These results demonstrate the principle's 's validity on a standard dataset with larger models.
Results.
Table 2: Maximum trajectory error on the MNIST dataset.
Model Configuration
Max Trajectory Error
MLP (128 > 256 units)
2.19 x 10-13 13
CNN (10 20 channels)
1.28 10-13
23

---

Experiment (MLP)
Trajectory Erior vs. Epochs
Training Loss vs. Epochs
Diponombbll UOSS (Pui battoh)))))
10-44
202
100
620
803
1000
230
600
65.
805
1305
Epoch
Epoch
Figure 6: MNIST results for MIP (128 256 units). (Lefft) Trajectory emorr per epoch remains
near machine precision. (Right) Training loss (full-batch evaluation) converges successfully.
MNIST Experiment Results (CNN)
Trjectory Epochs
Exparronta conornnnaall Locs F-lll botchh
Busitdeen
10:
200
400
600
800
1000
270
600
510
300
1302
Epoch
Figure 7: MNIST results for CNN (10 + 20 channels). (Left) Trajectory error per epoch remains near machine precision. (Right) Training loss (full-batch evaluation) converges successfully.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately summarize the paper's main contri-
butions: formalizing the KKT Point Embedding Principle (Theorems 4.2 and 4.5), proving its validity for neuron splitting in fully-connected networks and channel splitting in CNNs (Theorems 4.8) and 4.11), connecting it to gradient flow dynamics ( (Theorems 5.2 and (.4). and providing empirical validation for the dynamic preservation principle.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are primarily addressed through the stated assumptions (Assump- tions 4.T and [.T e.g., network homogeneity, smooth losses). The Conclusion (Section 6) also discusses future work, implying current scope limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might be used reliably provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairnsss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
25

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Each theoretical result is presented with its necessary assumptions. Proofs are provided either as sketches in the main text or with full versions deferred to the appendices
(Appendices B C)
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple-
mented by formal proofs provided in appendix or supplemental material. Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes. The main text describes the experimental sctup and key findings. A dedi- cated appendix (Appendix provides comprehensive details on dataset generation, model architectures, training hyperparameters, and the exact procedure for measuring trajectory error, which are sufficient to reproduce all results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurlPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to re- produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
26

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case au- thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Yes. The paper provides open access to the implementation code. A footnote provide link to public GitHub repository containing the code to reproduce all experi- ments and figures.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nips.c. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
//nips. cttpss/cccpbccccuctcceeee wwtpdiiccggui///e/eeeeeoo http://gguuieee/eeeeuuuutttta Cowe///eeeeeeeeeeeeeeeeeeeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Ye]]
Justification: Yes. The main text provides high-level overview of the experimental setting. The appendix (Appendix[D) contains a detailed breakdown of all settings, including model architectures, dataset generation procedures, optimizers, learning rates, batch sizes, and total training steps.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Question:
Answer: [NA]
27

---

Justification:
The experiments presented are deterministic numerical simulations designed verify theoretical principle (i.e., showing an error close zero). They are not stochastic experiments comparing performance between methods, so statistical significance reporting such as error bars is not applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors verified.
For asymmetric distributions, the authors should careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory. time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes. The experiments are conducted on small-scale 2D toy datasets and the MNIST dataset. They are computationally inexpensive and designed to be reproducible in minutes on a standard consumer laptop or desktop computer. The appendix specifies the software framework (PyTorch) used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't t make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttttt://turiiptttt wwtpe//pppwpccccictiiiiiiiee
Answer: [Yes]
Justification: The research is theoretical and involves numerical simulations public datasets. does not involve human subjects, PII, collection of sensitive data, or high-risk applications that would raise concerns under the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code Ethics.
28

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational theoretical research into the mathematical proper- ties of neural network optimization. Direct or immediate societal impacts are not primary focus, and the paper does not propose deployable systems.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not release any new data models, particularly those with high risk for misuse. The code provided for reproducing theoretical experiments.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by re- quiring that users adhere usage guidelines or restrictions access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
29

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: The paper's experiments use the public MNIST dataset and a custom- generated toy dataset. The use of standard libraries like PyTorch is implicit. No other external assets are used.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the li- cense of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not introduce or release new assets such as datasets or models for public use, other than the code for reprodueibility.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodll part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: The research does not involve crowdsourcing or experiments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
30

---

According to the NeurIPS Code of Ethics, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects. and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does involve human subjects, IRB approval equivalent is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equiva-
lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Question:
Answer: [NA]
Justification: Large Language Models were not used as component of the core research methodology. Any LLM usage was limited ancillary tasks such as assisting with writing and editing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy httpp:://eerrips.cc//ooferrrrrse for what should or should not be described.
31


Reviews:
[{"review_id": "8NdjjrUoyU", "reviewer": "Reviewer_ZZqy", "summary": "A mathematical paper shows that a particular neuron splitting operation in homogeneous neural networks (i.e. neural networks f(x, \\theta) with input x and parameters \\theta, such that f(x, c \\theta) = c^L f(x, \\theta) for some L; e.g. networks without biases, and with ReLU or linear activation functions), preserves Karush-Kuhn-Tucker (KKT) conditions and gradient flow dynamics. The non-differentiability of ReLU at 0 is addressed using Clarke's subdifferential.", "questions": "- line 197: What is \"(Theorem 4.1 in your original numbering)\" referring to?", "limitations": "See significance and originality.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Quality**\n\n- The paper is carefully written.\n\n**Clarity**\n\n- The paper is very clearly written. I read it linearly and could always follow.\n\n**Significance**\n\n- The scope (a specific kind of neuron splitting in homogeneous neural networks) is very limited.\n\n- While I appreciate the clarity of the formal details, the result that this neuron splitting operation leaves KKT conditions and gradient flow invariant is intuitive and could be explained much more succinctly. Many formal details feel to me more like good exercises for students getting familiar with subdifferentials and KKT conditions.\n\n**Originality**\n\n- Neuron splitting has been studied in many other works. I haven't come across the specific splitting operation T(a, b) = (c_1 a, ..., c_k a, c_1 b, ..., c_k b) needed in this paper to assure subdifferential preservation.\n\n- The propositions and theorems look original to me, but the proof techniques are straightforward.", "was_revised": true, "final_justification": "Thank you for your detailed and thoughtful response. I apologize for calling the formal details a good exercise; I realize that may come across as rude. I am a reader who is comfortable with mathematical formalism but prefers learning about new ideas succinctly and intuitively, leaving technical details to a nicely written appendix.\n\nFrom this perspective, I disagree that the level of formalization yields significant insights into why overparameterized models can generalize well. For me, the key takeaway remains point 2 of Theorem 4.6: that transformation T preserves the gradient for homogeneous networks. This is an interesting result, but of limited scope due to the homogeneity assumption. This result can be presented straightforwardly without using Clarke's subdifferential. Of course, it is worth mentioning the consequences of this insight for deeper networks, convolutions, KKT conditions, gradient dynamics, and generalization; however, these insights can be conveyed intuitively and much more succinctly.\n\nI think your revision goes in this direction, but without having read a substantially revised version, I prefer not to vote for acceptance.", "rebuttal": "We sincerely thank the reviewer for their positive feedback on the clarity and careful writing of our manuscript. We appreciate the thoughtful critique and would like to address the primary concerns regarding the paper's significance and originality, which we believe are more substantial than perceived.\n\n---\n### On the Significance and Originality of Our Work\n\nWe respectfully argue that the significance of our findings is substantial, as it provides a foundational, structural explanation for key questions in deep learning generalization. Our contribution is best understood in two interconnected parts:\n1.  **A Static KKT Point Embedding Principle:** This proves that the solutions (KKT points) of a smaller network's max-margin problem are isometrically embedded within the solution set of any wider network derived from it.\n2.  **A Dynamic Trajectory Embedding Principle:** This shows that the entire gradient flow training trajectory is similarly preserved under our transformation. The static principle is the essential foundation that guarantees this dynamic preservation.\n\nWhile the invariance of the network function under neuron splitting may seem intuitive, the core of our contribution lies in rigorously proving that this intuition extends to the optimization landscape's critical points (the KKT points) and, crucially, to the training dynamics (the gradient flow trajectories). This formalization is not merely a \"good exercise\"; it yields significant insights into why overparameterized models can generalize well.\n\nA fundamental question in deep learning is how a highly complex neural network can effectively learn a low-complexity classification boundary. Our work provides a direct theoretical answer:\n* Our KKT embedding principle formally **guarantees the existence of low-complexity solutions** (those embedded from smaller, simpler models) within the vast solution space of overparameterized networks.  Prior work has shown that gradient flow converges in direction to a KKT point of the max-margin problem (**Lyu & Li, 2019**), but a significant gap remained: large networks can possess many KKT points with varying complexities. Our work bridges this gap by proving that simple, generalizable solutions are structurally preserved as network width increases.\n* This has immediate practical implications. The co-existence of both low-complexity (embedded) and high-complexity KKT points suggests that **initialization and hyperparameter tuning play a critical role in *selecting* a solution**. Our framework provides a theoretical basis for understanding this selection process and for future work on designing training procedures that explicitly steer optimizers toward these desirable, low-complexity solutions to improve generalization.\n\n**Regarding Originality:** While neuron splitting has been studied, our work's originality lies not in the operation itself, but in the **development of a formal KKT Point Embedding Principle that uses such operations to create a rigorous link between the solutions and training dynamics of networks of different sizes**. We are the first to establish this comprehensive static and dynamic correspondence under a unified theoretical framework.\n\nRegarding the proof techniques, we agree that they are direct applications of established tools like the Clarke subdifferential. We see this as a strength, lending clarity and rigor to our results. The novelty lies not in the techniques themselves, but in their application to establish a new and significant principle that was previously unformalized.\n\n---\n### Revisions to Enhance Contribution and Clarity\n\nWe appreciate your perspective on the paper's limitations of our core results. To better demonstrate the significance of our framework and the depth of its contributions beyond the initial intuition, we have made the following substantial revisions\n\n**1. Broadened Scope with Extension to CNNs:** We acknowledge that focusing solely on fully-connected homogeneous networks could be perceived as a limitation. To demonstrate the generality and power of our approach, **we have introduced a new main result extending our principle to Convolutional Neural Networks (CNNs)**. This required designing a novel **'channel-splitting'** transformation, meticulously constructed to respect the architectural hallmarks of CNNs, such as weight sharing and locality. The transformation isotropically splits a single feature channel into $m_{split}$ new channels via coefficients $\\{c_i\\}$ that satisfy $\\sum_{i=1}^{m_{split}} c_i^2 = 1$. The specific construction is as follows:\n* **At Layer k**: The filter $F_j^{(k)}$ that produces channel $j$ is replaced by $m_{split}$ new filters, where the i-th new filter is defined as $c_i F_j^{(k)}$.\n* **At Layer k+1**: In turn, every filter in the next layer that takes channel $j$ as input has its j-th input slice replaced by $m_{split}$ new slices, where the i-th new slice is scaled by the corresponding coefficient $c_i$.\n\nSuccessfully proving that this transformation satisfies our core theory (**Theorem 4.5**) validates our framework as a flexible blueprint applicable to a broad class of homogeneous networks, with future work poised to explore architectures like Transformers.\n\n**2. Improved Narrative and Contribution Density:** To address the impression that the paper was overly detailed or like an \"exercise,\" we have significantly restructured the manuscript for a more focused and impactful narrative. This includes:\n* **Consolidating Theorems:** We have consolidated the application theorems. The two-layer network case, being a special case of the deep network one, has been moved to the appendix, allowing the main text to focus on the more general and powerful deep network result.\n* **Streamlining Presentation:** We have deferred highly technical definitions (formerly Def. 3.1, 3.2, and Asm. 3.3) to the appendix to improve readability and focus on the core contributions.\n\n---\n### Response to Minor Issue\n\n* **Comment on line 197:** We sincerely apologize for this oversight. This was a remnant from an internal draft and has been removed in the revised version.\n\n---\n\nOnce again, we thank you for your time and valuable feedback. We hope these clarifications and the significant revisions to the manuscript, including the new CNN result, will help you re-evaluate the significance and originality of our contribution more favorably.\n### References\n\n* **Lyu, K., & Li, J. (2019).** Gradient descent maximizes the margin of homogeneous neural networks. *arXiv preprint arXiv:1906.05890*."}, {"review_id": "iSa9PgH2tG", "reviewer": "Reviewer_jDVY", "summary": "I am very much not an expert in the mathematical side of things here. My understanding is that the paper proves a neat theoretical fact about bias-free ReLU networks. If you take a trained narrow network and simply clone each neuron (splitting one neuron into two identical ones and rescaling weights), every optimal solution and even the whole gradient-flow training path of the small net (under very strict assumptions on the path and the net) sits perfectly inside the bigger net. => widening in this very specific way does not create new, better optima. The authors first show a general optimization theorem and then apply it to two-layer and deep ReLU models.", "questions": "- Bias terms: how hard would it be to make this work for NNs with bias?\n- Weight sharing: the same question, this would make it much more actionable as a result\n\nMy main ask = empirical demo\nA tiny experiment on a 2-layer ReLU network would greatly strengthen the paper and I'm primarily interested to see how the theory and the experiment deviate as the conditions in the experiments are not matching exactly the idealized conditions in the theory.\n\nMy score would increase if you include convincing experiments.", "limitations": "Yes, pretty well addressed modulo the experiment I would like to see.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Solid proofs: the math seems self-contained and careful (as far as i can tell)\n- Clear structure: the paper moves from a general theorem to concrete neural net corollaries, which is easy to follow\n- new angle: It is the first work (as far as i can tell) to show that neuron-splitting preserves both KKT optima and whole gradient-flow paths. previous works focused primarily on optima alone\n\nWeaknesses:\n- Very strong assumptions: No biases, full batch gradient flow, and perfectly separable data = unrealistic for modern training, and it is therefore hard to say how much of this generalizes to anything real\n- No experiments: even a toy example would help me trust the theory more and I would love to see how much the theory deviates from the simple experiments as I make the conditions imperfect\n- small presentation issues: several “Theorem ??” placeholders\n- Limited scope: results do not cover CNNs, attention layers, or mini-batch SGD (no weight tying permitted in the theory)", "was_revised": true, "final_justification": "I am very happy with the additional experiments the authors ran and the way they addressed my concerns. If these are incorporated in the paper, I believe it would definitely be stronger than the original. I am increasing my rating from 3 = borderline reject to 4 = borderline accept. Not being an expert in the sub-field, I'm not confident to do 5 = accept, but I think the paper would be a valuable addition to the field. Thank you for addressing my concerns and running additional experiments!", "rebuttal": "We are very grateful for your insightful review and constructive feedback. We especially appreciate that you recognized the novelty of our work in showing the preservation of the **entire gradient-flow path**, which is a core part of our contribution. Your feedback has been instrumental in helping us bridge the gap between our theory and its practical implications.\n\n---\n### Our Main Response: A Comprehensive Empirical Demonstration\n\nIn direct response to your excellent suggestion to bridge the gap between our idealized theory and practice, we have conducted a comprehensive suite of experiments to systematically test the robustness and generality of our Trajectory Embedding Principle. We present these findings as a three-step narrative, moving from ideal conditions to more realistic scenarios. Our theory predicts that the maximum deviation between the two trajectories, $\\max_{k \\in \\{0, 1, \\dots, N\\}} ||\\eta_k - T\\theta_k||_2$, will be negligible.\n\n**Step 1 & 2: Verification and Robustness on Toy Models**\n\nWe began by verifying our theory under ideal conditions and then tested its robustness when key assumptions were relaxed.\n\n* **Setup:** We constructed 2-layer homogeneous MLPs (with a hidden layer of 1 neuron vs. 2 neurons) and simple 2-layer CNNs (consisting of one convolutional of 2 channels vs. 4 channels and one fully-connected layer). We initialized them such that the initial parameters of the larger network were a direct mapping of the smaller one via our transformation $T$ (i.e., $\\eta_0 = T\\theta_0$). This transformation T implements a neuron splitting mechanism for MLPs and a channel splitting mechanism for CNNs. The latter is described in detail in our response to Question 2.The latter is described in detail in our response to Question 2. Both networks were then trained under various conditions for 1 million steps on a two-dimensional linearly separable dataset.\n\n* **Conclusion:** The results, summarized below, confirm that the trajectory error remains extremely small across all setups.\n\n| Exp. | Model & Optimizer | Data Condition | Max Trajectory Error  |\n| :--- | :--- | :--- | :--- | \n| 1    | MLP with GD       | Separable      | `4.46e-13`             | \n| 2    | MLP with SGD      | Separable      | `4.35e-13`             | \n| 3    | MLP with GD       | Non-Separable  | `5.57e-13`             | \n| 4    | CNN with GD       | Separable      | `2.36e-10`             | \n\nThe small, non-zero errors (on the order of `1e-13` to `1e-10`) are an expected consequence of numerical precision. Over a million steps, minuscule floating-point rounding errors inevitably accumulate.\n\nIt is crucial to note that for the SGD experiment (Exp. 2), both networks were trained using an **identical sequence of mini-batches**. We also ran a control experiment where the two networks were trained with **different, randomized batch orders**. In this scenario, the maximum trajectory error increased dramatically to **`1.5316e-01`**. This result is expected, as a network's training path is highly dependent on its specific data sequence.\n\n**Step 3: Demonstrating Generality on MNIST**\n\nFinally, to address the scope and demonstrate the framework's applicability to more realistic architectures, we extended our tests to the MNIST dataset.\n\n* **Setup:** We scaled up to the MNIST dataset (classifying 3s vs 5s), training both **two-layer** homogeneous MLPs (128 vs 256 hidden units) and **two-layer** CNNs (10 vs 20 channels). We used SGD with a **learning rate of 0.001 and a batch size of 64** for **1000 epochs**. The models achieved high classification accuracy **(e.g., 99.99% for MLP, 99.79% for CNN)**, ensuring we observed the trajectory error during a meaningful learning phase.\n\n* **Conclusion:** The results, summarized below, confirm that the trajectory preservation principle holds with remarkable precision even on this more complex task. Both the MLP and CNN models maintained a near-zero trajectory error throughout training, demonstrating that the training dynamics of the wider network are highly predictable from its narrower counterpart.\n\n| Model Configuration  | Max Trajectory Error | \n| :---                | :---                 | \n| MLP (Homogeneous) | ` 2.1868e-13`         | \n| CNN (Homogeneous) | `1.2839e-13`         | \n\nThis comprehensive suite of experiments confirms that our theoretical principle is not only correct in its ideal setting but is also a robust and generalizable framework for understanding the relationship between homogeneous networks of different sizes.\n\nWe will add a dedicated section for these results to the main paper. A detailed description of all experimental setup will be provided in a dedicated appendix section in our revised manuscript. Furthermore, we will make the code publicly available on GitHub to ensure full reproducibility.\n\n---\n### On Assumptions, Scope, and Specific Questions\n\n**On Strong Assumptions and Limited Scope:** We agree that our theoretical setup involves strong assumptions. However, we wish to clarify a critical distinction: the assumptions of full-batch gradient flow and perfectly separable data are not required for our KKT and Trajectory Embedding Principles.\n\nThese conditions are primarily needed for the well-known result from prior work that guarantees the training direction converges to a max-margin KKT point. In contrast, our embedding principles (Theorems 4.5 and 5.2) describe a more fundamental structural relationship between networks. This relationship holds regardless of whether the training converges to that specific max-margin solution.\n\nOur new experiments explicitly confirm this. The trajectory embedding holds with near-perfect precision even when using Stochastic Gradient Descent (Exp. 2 & MNIST) and on non-separable data (Exp. 3). This demonstrates that our framework is robust and not confined to those idealized convergence scenarios. Furthermore, to broaden the scope, we are happy to clarify that our revised manuscript introduces a new main result extending our framework to Convolutional Neural Networks (CNNs).\n\n**Question 1: How hard would it be to include bias terms?**\nThis is a challenging but important question. Extending our framework to include bias terms is non-trivial because biases break the network's strict homogeneity, which is a cornerstone of our current analysis and the associated max-margin dynamics. Specifically, this extension (with bias) would require a fundamental redesign of the corresponding linear transformation T, which currently relies on this homogeneous property.\n\nHowever, very recent work has begun to bridge this gap. For instance, Cai et al. (2025) extend the implicit bias analysis to a broad class of near-homogeneous networks. Their framework is powerful because it formally includes architectures with components that break strict homogeneity, such as bias terms and residual connections. They show that for these networks, gradient descent still converges in direction to a KKT point of a corresponding margin maximization problem. This development provides a promising theoretical pathway for our own work. A natural future direction is to investigate whether our KKT embedding principle can be generalized to this near-homogeneous setting, thereby extending our results to practical networks that include biases.\n\n**Question 2: How hard would it be to handle weight sharing?**\n\nTo demonstrate the generality and power of our approach, **we have introduced a new main result extending our principle to Convolutional Neural Networks (CNNs)**.This required designing a novel **'channel-splitting'** transformation, meticulously constructed to respect the architectural hallmarks of CNNs, such as weight sharing and locality. The transformation isotropically splits a single feature channel into $m_{split}$ new channels via coefficients $\\{c_i\\}$ that satisfy $\\sum_{i=1}^{m_{split}} c_i^2 = 1$. The specific construction is as follows:\n\n* **At Layer k (Outgoing Weights):** The filter $F_j^{(k)}$ responsible for generating channel $j$ is replaced by $m_{split}$ new filters. The i-th new filter is simply a scaled version of the original: $c_i F_j^{(k)}$.\n* **At Layer k+1 (Incoming Weights):** Consequently, every filter in the next layer that receives channel $j$` as input has its j-th input slice replaced by $m_{split}$ new slices, with each new slice being scaled by the corresponding coefficient $c_i$.\n\nSuccessfully proving that this transformation satisfies our core theory (**Theorem 4.5**) validates our framework as a flexible blueprint applicable to a broad class of homogeneous networks, with future work poised to explore architectures like Transformers.\n\n---\n### Minor Issues\n\nThe \"Theorem ??\" placeholders were indeed artifacts from our drafting process and have all been corrected in the final version. We apologize for the oversight.\n\n---\n\nThank you again for your valuable and pragmatic feedback. We believe the new empirical demonstrations, a broader scope that now includes CNNs, and our detailed clarifications directly address your main concerns. We hope these additions will convince you of the paper's value and merit your support.\n\n### References\n\n* Cai, Y., et al. (2025). Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. *Proceedings of the 42nd International Conference on Machine Learning*."}, {"review_id": "QdWZxxXkvC", "reviewer": "Reviewer_rUJh", "summary": "This paper studies how the KKT points of the max-margin problem for homogeneous networks relate to those of wider homogeneous networks. It first establishes the KKT point embedding principle, showing that KKT points of a homogeneous network’s max-margin problem can be embedded into the KKT points of a larger network’s problem via specific linear isometric transformations. Second, it provides explicit constructions of such transformations for two-layer and deep homogeneous networks using a neuron splitting technique. Lastly, it demonstrates that this neuron splitting transformation preserves the entire gradient flow trajectory when initialized appropriately.", "questions": "# Major Questions\n\n- Based on your Remark 4.8, is the neuron splitting transformation for two-layer networks a simple special case of the transformation for deep networks? Based on this, I don’t see why you need to introduce Theorem 4.6. In some sense, this just makes the writing more verbose if this theorem is fully captured by Theorem 4.9. Maybe this is not the case; in that case, can you explain why having both theorems is better (i.e., that Theorem 4.6 covers a setting not covered by Theorem 4.9)?\n\n- Can you elaborate more on “designing transformations T that respect architectural specificities (e.g., locality, weight sharing in CNNs) while satisfying the necessary isometric and output/subgradient mapping properties identified in our current work.” I think spending some time expanding on this would be a way to improve the significance of your work. It would provide a roadmap for others to build upon your work to generalize the ideas to more complex architectures while continuing to use your main theorems.\n\n- Can your analysis also be used to map KKT points of the max-margin problem associated with a larger homogeneous network to KKT points of the analogous problem for a smaller network? Rather than neuron splitting, can you construct operations that involve neuron merging? What limitations on this operation would exist? I think this would be an interesting direction to discuss as it relates to network pruning/compression.\n\n# Minor Comments\n\n- I like Figure 1, but I would recommend making it smaller or more horizontal and adding a caption to discuss it in more detail. It should also be referenced somewhere in the main text.\n\n- Definitions 3.1 and 3.2, and Assumption 3.3 are all quite technical; you could consider placing these in the appendix. Many prior works in this literature tend to do this.\n\n- I would split Section 4 into two sections: one section up to Theorem 4.5 and one section for Theorem 4.6 and beyond. This would match your main contributions list as well.\n\n- I was not familiar with ω-limit sets. I would provide a reference for this in lines 279–283 or perhaps formally define it.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "# Strengths\n\n- Clearly defines notation, definitions, assumptions, and theorems.\n- Extends an important line of theoretical literature on neural networks in an original way.\n\n# Weaknesses\n\n- **Insufficient references to important related work.**  \n  In your related work subsection *“Optimization dynamics and implicit bias in homogeneous networks,”* you cite many classic works connecting maximum-margin biases and homogeneous networks (i.e., works before 2020). However, many extensions to this line of analysis have appeared since then. For example:\n  - *The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks*\n  - *Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization*\n  - *Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization*\n  - *Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks*\n\n  In your related work subsection *“Network embedding principles,”* you only cite a single reference (Zhang et al., 2021). While this is certainly a relevant work, many other studies have considered this classical idea. For instance, you might look at the pruning and lottery ticket hypothesis literature, such as *“Proving the Lottery Ticket Hypothesis: Pruning is All You Need.”*\n\n- **Sections 3 and 4 lack narrative flow.**  \n  Much of these sections read as a list of definitions, assumptions, theorems, and proofs, rather than a coherent story. There should be a stronger narrative connecting the material back to the main contributions stated in the introduction. Consider deferring some technical details to the appendix to improve readability. See my comments below.\n\n- **The significance of the results is unclear.**  \n  While there is some attempt to explain their importance in the introduction and conclusion, much more effort is needed to contextualize why these results matter. See my suggestions below for directions to expand on the significance. For example, in the introduction you state:  \n  > “A fundamental question arises regarding the relationship between solutions found in networks of different sizes.”  \n  \nBut why is this a fundamental question? Does it relate to scaling laws, robustness, generalization, or practical considerations when training larger models?", "was_revised": true, "final_justification": "My two main concerns with this paper were (1) insufficient discussion of related work  and (2) the significance of the theoretical results seem to be limited. Based on the author's responses, my concern for (1) has been largely alleviated. The authors discussed (2), but I am not convinced there is much of an argument here and it would require empirical results. So I will be keeping my original evaluation, leaning towards acceptance.", "rebuttal": "We are grateful to the reviewer for the thorough and insightful feedback on our manuscript. Your comments on the related work, narrative structure, and significance of our results have been instrumental in strengthening the paper and have also highlighted promising future research directions. We have undertaken a substantial revision based on your detailed guidance.\n\n---\n### On Related Work and Significance\n\n**Broadened Context of Related Work**\n\nFollowing your guidance, we have substantially revised our discussion of the related literature in two key areas:\n\n1.  **“Optimization dynamics and implicit bias”**: We have updated this section's title. We expanded our discussion to cover recent developments, including the analysis of margin in non-homogeneous models  **(Kunin et al., 2023)**、**(Cai et al. 2025)**, and the connection between KKT conditions and benign overfitting **(Frei et al., 2023)**.\n\n2.  **“Network embedding principles”**:we have broadened the discussion to better situate our work, citing foundational research on hierarchical structures that is highly relevant to our embedding principles **(Fukumizu & Amari, 2000)**, as well as recent studies on permutation symmetries that connect different global minima **(Şimşek et al., 2021)**. we carefully considered the suggested literature on network pruning and the Lottery Ticket Hypothesis. We agree this is an important field, but we concluded that its core principle is conceptually distinct from our work. Specifically, pruning focuses on **identifying and removing** less important neurons to create a sparse sub-network. In contrast, our method focuses on merging neurons that have a specific structural relationship to create a smaller, equivalent network.\n\n**Articulating the Significance of Our Work**\n\nOur contribution is best understood in two interconnected parts:\n1.  **A Static KKT Point Embedding Principle:** This proves that the solutions (KKT points) of a smaller network's max-margin problem are isometrically embedded within the solution set of any wider network derived from it.\n2.  **A Dynamic Trajectory Embedding Principle:** This shows that the entire gradient flow training trajectory is similarly preserved under our transformation. The static principle is the essential foundation that guarantees this dynamic preservation.\n\nWhile the invariance of the network function under neuron splitting may seem intuitive, the core of our contribution lies in rigorously proving that this intuition extends to the optimization landscape's critical points (the KKT points) and, crucially, to the training dynamics (the gradient flow trajectories). This formalization yields significant insights into why overparameterized models can generalize well.\n\nA fundamental question in deep learning is how a highly complex neural network can effectively learn a low-complexity classification boundary. Our work provides a direct theoretical answer:\n* Our KKT embedding principle formally **guarantees the existence of low-complexity solutions** (those embedded from smaller, simpler models) within the vast solution space of overparameterized networks. Prior work has shown that gradient flow converges in direction to a KKT point of the max-margin problem (**Lyu & Li, 2019**), but a significant gap remained: large networks can possess many KKT points with varying complexities. Our work bridges this gap by proving that simple, generalizable solutions are structurally preserved as network width increases.\n* This has immediate practical implications. The co-existence of both low-complexity (embedded) and high-complexity KKT points suggests that **initialization and hyperparameter tuning play a critical role in selecting a solution**. Our framework provides a theoretical basis for understanding this selection process and for future work on designing training procedures that explicitly steer optimizers toward these desirable, low-complexity solutions to improve generalization.\n\n---\n### On Narrative Flow and Major Questions\n\n**On the Narrative Flow of Sections 3 and 4**\n\nFollowing your valuable suggestions, we have improved the narrative flow by:\n* Deferring technical definitions (formerly Def. 3.1, 3.2, Asm. 3.3) to the appendix.\n* Splitting Section 4 into two separate sections: one establishing the general KKT embedding framework and another dedicated to its application to neuron splitting and channel splitting.\n\n### On the Major Questions\n\n**1. On the relation between Theorem 4.6 and 4.9:**\n We agree that the two-layer network transformation is a special case of the deep network one. To streamline the paper, we have removed the separate theorem for two-layer networks and now present only the general deep network result as our main theorem. The two-layer construction is now included as a pedagogical example in the appendix.\n\n\n**2. On extending the framework to CNNs:** We have formally extended our framework to homogeneous Convolutional Neural Networks (CNNs) by generalizing \"neuron splitting\" to \"channel splitting\" ans included this as a new main result. The transformation splits a channel into $m_{split}$ new channels using coefficients $\\{c_i\\}$ that satisfy $\\sum_{i=1}^{m_{split}} c_i^2 = 1$. It is defined as follows:\n * **Layer k**: Replacing the filter $F_j^{(k)}$ that produces channel $j$ with $m_{split}$ new filters, where the $i$-th new filter is defined as $c_i F_j^{(k)}$. \n * **Layer k+1**: For every filter $F_p^{(k+1)}$, replacing its $j$-th input channel slice with $m_{split}$ new slices, where the $i$-th new slice is scaled by the corresponding coefficient $c_i$.\n \n  Successfully proving that this general transformation satisfies the conditions of our core theory (**Theorem 4.5**) is a powerful validation of our framework. It shows that the principles of isometric and output/subgradient preservation are not confined to simple network topologies but provide a robust blueprint for analyzing a wider class of homogeneous networks. This strongly suggests that our framework can be adapted to other architectures with specific structural invariances.\n\n**3. On mapping KKT points from larger to smaller networks:**\nApplying our framework to the larger-to-smaller network mapping via neuron merging is a valuable direction that connects our work to network pruning.\n\nTo map Karush-Kuhn-Tucker (KKT) points from a larger network (parameters $\\eta$) to a smaller one ($\\theta$), we can define a \"neuron merging\" operator $M$ where $\\theta = M(\\eta)$. For this operator to be KKT-preserving , it must satisfy conditions analogous to our **Theorem 4.5**, namely:\n1.  **Output Preservation**: $\\Phi(M(\\eta); x) = \\tilde{\\Phi}(\\eta; x)$. This is analogous to the output-preserving condition for splitting.\n2.  **Subgradient Preservation**: $\\partial_{\\theta}^{\\circ}\\Phi(M(\\eta); x)$ must maintain a specific relationship with $M(\\partial_{\\eta}^{\\circ}\\tilde{\\Phi}(\\eta; x))$, analogous to the subgradient condition for splitting.\n\nThe output preservation condition imposes a strict structural constraint. For instance, merging several neurons into one ($\\sum_{i}a_i\\sigma(b_i^T x) \\to a\\sigma(b^T x)$) requires the weight vectors $\\{b_i\\}$ of the neurons to be **collinear**.\n\nWhile this requirement seems restrictive, it connects to the practical phenomenon of neural condensation, where neuron weights are observed to align and become collinear during training (Zhou, et al.,2022). This suggests that valid opportunities for such a theoretically-grounded merge may arise naturally.\n\nOur future work will explore the precise KKT point mapping from larger to smaller networks under valid merging operations.\n\n\n### Response to Minor Comments\n\nWe have incorporated all of your minor comments:\n\n* Figure 1 has been resized and given a more detailed caption, and it is now referenced at the end of the introduction to serve as a roadmap.\n\n* The technical definitions have been moved to the appendix.\n\n* Section 4 has been split into two separate sections.\n\n* We have clarified the definition of ω-limit sets and added a formal citation.\n\n---\n\nOnce again, we would like to express our gratitude for your thorough and constructive review. Your comments have been instrumental in helping us strengthen the manuscript.\n### References\n\n* **Kunin, D., et al. (2023).** The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks. *International Conference on Learning Representations (ICLR)*.\n* **Cai, Y., et al. (2025).** Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks. *Proceedings of the 42nd International Conference on Machine Learning*.\n* **Frei, S., et al. (2023).** Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization. *arXiv preprint arXiv:2303.01462*.\n* **Fukumizu, K., & Amari, S. (2000).** Local minima and plateaus in hierarchical structures of multilayer perceptrons. *Neural Networks*.\n* **Şimşek, B., et al. (2021).** Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. *Proceedings of the 38th International Conference on Machine Learning (ICML)*.\n* **Lyu, K., & Li, J. (2019).** Gradient descent maximizes the margin of homogeneous neural networks. *arXiv preprint arXiv:1906.05890*.\n* **Zhou, H., et al. (2022).** Towards understanding the condensation of neural networks at initial training. *Advances in Neural Information Processing Systems, 35*.\n* **Hirsch, M. W., Smale, S., & Devaney, R. L. (2013).** *Differential equations, dynamical systems, and an introduction to chaos*. Academic Press."}, {"review_id": "Ej8dsPEJto", "reviewer": "Reviewer_dAiL", "summary": "This paper shows theoretically that, when mapping a narrow neural network to a wider network by neuron splitting, all the solutions and the gradient flow trajectory of the narrow network can be also mapped to solutions and trajectory of the wider network.\nThe result is obtained for classification problems and homogeneous neural networks. Previous work obtained similar results in different settings. In particular, the main novel contribution of this work is the equivalence of the gradient flow trajectory.", "questions": "NA", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nTheoretical results are correct, novel and will be useful for future research.\n\nWeaknesses:\n- The density of theoretical contribution does not quite justify 9 pages of a NeurIPS paper. Most theorems are just variant of each other with small differences in technical assumptions, and most derivations consist in just applying a chain rule. It would have been much nicer to reduce the space given by technical details of the theory and provide some empirical contribution to complement the theory, as it was done in many related papers.\n\nMinor issues:\n- Most mathematical symbols are introduced without any explanation or definition, both in the abstract and in the introduction.\n- Some important references are missing, about “Network embedding principles”, that include:\nFukumizu and Amari. Local minima and plateaus in hierarchical structures of multilayer perceptrons. Neural Networks (2000)\nSimsek et al, Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. ICML (2021)\n- There is a comment on line 197 that does not seem to belong to the paper: \"(Theorem 4.1 in your original numbering)\"", "was_revised": true, "final_justification": "I am keeping the initial score because my main issue with the paper, the redundancy of theoretical results and lack of empirical testing, cannot be fixed in the rebuttal. The authors promised that they will significantly change the paper but I obviously cannot evaluate those eventual changes now.", "rebuttal": "We are sincerely grateful to the reviewer for their careful assessment and for recognizing that our theoretical results are correct, novel, and useful for future research. Your feedback on the paper's structure and clarity has been invaluable, and we have undertaken a significant revision to address your concerns and enhance the manuscript's impact.\n\n---\n### On Contribution Density and Paper Structure\n\nWe understand and appreciate your central concern regarding the density of the theoretical contribution and the overall structure. To address this, we have substantially restructured the paper to improve both its clarity and theoretical density.\n\n**1. Streamlined Theoretical Presentation:** We have consolidated our main theoretical results to create a more focused narrative. This includes deferring highly technical definitions (formerly Def. 3.1, 3.2, and Asm. 3.3) to the appendix and splitting Section 4 into two more focused parts: one establishing the general KKT mapping framework, and the other detailing its application to neuron splitting. Furthermore, to resolve the feeling of redundancy you noted, we have consolidated the application theorems. The two-layer network transformation, being a special case of the deep network one, has been moved to the appendix as a pedagogical example. \n\n**2. Enhanced Contribution Density with a New Major Result:** To further address your point on contribution density and to showcase the power of our framework, we will include a **new major result** in the revised manuscript: the formal extension of our framework to homogeneous **Convolutional Neural Networks (CNNs)**. This is a non-trivial extension that required designing a new \"channel splitting\" transformation that respects the architectural constraints of CNNs, such as weight sharing and locality. The transformation splits a channel into $m_{split}$ new channels using coefficients $\\{c_i\\}$ that satisfy $\\sum_{i=1}^{m_{split}} c_i^2 = 1$. It is defined as follows:\n* **Layer k**: Replacing the filter $F_j^{(k)}$ that produces channel $j$ with $m_{split}$ new filters, where the $i$-th new filter is defined as $c_i F_j^{(k)}$. \n* **Layer k+1**: For every filter $F_p^{(k+1)}$, replacing its $j$-th input channel slice with $m_{split}$ new slices, where the $i$-th new slice is scaled by the corresponding coefficient $c_i$.\n\nSuccessfully proving that this general transformation satisfies the conditions of our core theory (**Theorem 4.5**) is a powerful validation of our framework. It shows that our principles are not confined to simple network topologies but provide a robust blueprint for analyzing a wider class of homogeneous networks.\n\n**3. On Providing Empirical Contribution:**  we have conducted a comprehensive suite of experiments to empirically validate our theory and will add a dedicated section for these results to the main paper. Our experiments are presented as a three-step narrative, moving from ideal conditions to realistic scenarios, to systematically test the robustness and generality of our Trajectory Embedding Principle.\n\n1. Verification under Ideal Conditions:\nFirst, we verified our theory in an idealized setting (homogeneous MLP, GD, separable data). As predicted, the trajectories of the smaller and larger networks remained perfectly aligned, with a minuscule error of 4e-13 attributable solely to numerical precision drift over one million steps.\n\n2. Robustness to Relaxed Assumptions:\nNext, we tested the principle's robustness by relaxing these ideal conditions, using SGD, non-separable data, and even applying the theory to homogeneous CNNs via our \"channel splitting\" transformation. In all cases, the trajectory error remained vanishingly small, confirming that the principle holds even without continuous-time optimizers or linearly separable data.\n\n3. Generality on a Practical Task (MNIST):\nFinally, to demonstrate practical relevance, we scaled our experiments to the MNIST dataset, training both homogeneous MLPs and CNNs that achieved high accuracy (>99%). The trajectory preservation principle held with remarkable precision (1e-14 error), confirming its applicability to modern architectures during meaningful learning on a real-world task.\n\nConclusion:\nCrucially, the tiny, non-zero errors observed are an expected consequence of floating-point arithmetic. This comprehensive validation provides strong evidence that our theoretical framework is not just mathematically sound but also a robust and generalizable tool for understanding practical neural networks.\n\nFull experimental details will be added in the appendix, and our code will be made publicly available to ensure full reproducibility.\n\n---\n### Response to Minor Issues\n\nWe thank the reviewer for catching these important details. We have addressed all of them in the revised manuscript.\n\n* **On Undefined Mathematical Symbols**: We apologize for the lack of clarity. We have carefully revised the abstract and introduction to ensure that all key mathematical notations (such as $P_{\\Phi}$, $L(\\theta(0))$, and $T$) are properly introduced and briefly defined upon their first appearance, improving the paper's accessibility.\n\n* **On Missing References**: We are grateful for the excellent suggestions on related work. We have substantially revised our literature review to provide a more comprehensive context. This includes broadening our discussion on **‘Network embedding principles’** to include the foundational work on hierarchical structures and symmetries (**Fukumizu & Amari, 2000; Şimşek et al., 2021**) and the lottery ticket hypothesis (**Malach et al., 2020**). Additionally, we updated our discussion on **‘Implicit Bias’** to cover recent advancements in non-homogeneous models and connections to benign overfitting (**Kunin et al., 2023; Frei et al., 2023**). We believe these additions now situate our work much more clearly within the landscape of contemporary research.\n\n* **On the Stray Comment**: We sincerely apologize for this oversight. The comment on line 197 was indeed a leftover note from an internal draft and has been removed from the final version.\n\n---\n\nOnce again, we would like to express our gratitude for your thorough and constructive review. Your comments have been instrumental in helping us strengthen the manuscript.\n\n### References\n\n* **Fukumizu, K., & Amari, S. (2000).** Local minima and plateaus in hierarchical structures of multilayer perceptrons. *Neural Networks*.\n* **Şimşek, B., et al. (2021).** Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances. *Proceedings of the 38th International Conference on Machine Learning (ICML)*.\n* **Malach, E., et al. (2020).** Proving the Lottery Ticket Hypothesis: Pruning is All You Need. *Proceedings of the 37th International Conference on Machine Learning (ICML)*.\n* **Kunin, D., et al. (2023).** The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks. *International Conference on Learning Representations (ICLR)*.\n* **Frei, S., et al. (2023).** Benign Overfitting in Linear Classifiers and Leaky ReLU Networks from KKT Conditions for Margin Maximization. *arXiv preprint arXiv:2303.01462*."}, {"review_id": "6nuNeI8uDG", "reviewer": "Reviewer_iDLM", "summary": "This paper inreoduces a technique, KKT Point Embedding, to establish the relationship of the learning dynamics between two homogeneous neural networks. The authors then use this technique to analyze the learning dynamics of neural networks with splitted neurons.", "questions": "See weaknesses.", "limitations": "The authors didn't explicitly discuss the limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is clearly written and uses rigorous mathematical notation. This makes this paper easy to follow.\n- This paper provides a novel result of the learning dynamics of neuron splitted NNs. \n- The proposed technique, KKT Point Embedding, might be useful to the community.\n\nWeaknesses:\n- I don't understand why this problem is important. What is the point of studying the learning dynamics of neuron splitted NNs?\n- Also, I don't understand why we have to invent techniques such as KKT Point Embedding to study the dynamics of neuron splitted NNs. Can't we prove it by simple techniques such as, say, showing the neurons remain splitted all the time? I hope the authors can briefly explain what is the technical difficulty of this problem and why the proposed technique is non-trivial and necessary.\n- The condition for KKT Point Embedding (Theorem 4.5, 1.) seems to be very strong. I wonder if it has an application in other scenarios? For example, is it possible to show KKT Point Embedding from **any** small model to large model, since there must exsits a linear transformation that maps a low-dim vector to a high-dim vector?\n\nI do like the overall writing of this paper. I will consider increasing my rating if the authors provide satisfactory responses to my questions.\n\nMinor writing issues:\n- Line 66, $L_2$ should be $L^2$ (to be consistent with notations used later, e.g. line 98).\n- In Definition 3.2, do you mean $h \\in \\partial^\\circ f$? It's kind of confusing what does it mean by $h \\in \\partial^\\circ f(z(t))$.\n- The symbol $L$ is overloaded. In Definitin 3.4 it denotes the order of homogeneity, whereas in Definition 5.3 it refers tothe $\\omega$-limit set (why don't you just write $\\omega(\\bar \\theta)$?).\n- Lines 197-198 contain a parenthetical remark: \"(Theorem 4.1 in your original numbering)\", which appears to be an internal note that was unintentionally left in the manuscript.\n- Line 300: \"$C^1$-smooth\" should be $\\mathcal C^1$-smooth.", "was_revised": true, "final_justification": "This paper is well written and clearly structured. It presents a series of non-trivial theoretical results. During the rebuttal, the authors proposed a channel-splitting model that demonstrates the practical relevance of their theory. While I still have some reservations about the overall significance of the contribution, I believe it has the potential to benefit the community.", "rebuttal": "We sincerely thank you for your detailed review and for the positive feedback on the clarity and rigor of our paper. Your questions are insightful and get to the heart of our work's motivation and technical contribution. We are confident that we can provide satisfactory answers and are grateful for the opportunity to clarify these points.\n\n---\n### On the Motivation: Why Study Neuron Splitting?\n\nThis is a fundamental question. The importance of studying neuron splitting is not about the operation itself, but about using it as a **precise, mathematically tractable model for understanding overparameterization and generalization**.It provides insights into why overparameterized models can generalize well detailed as follow:\nA fundamental question in deep learning theory is how a highly complex neural network can generalize well. The embedding principle uncovered in our work provides a key mechanism:\n* Our KKT embedding principle formally **guarantees the existence of low-complexity solutions** (those embedded from smaller, simpler models) within the vast solution space of overparameterized networks. Prior work has shown that gradient flow converges in direction to a KKT point of the max-margin problem (**Lyu & Li, 2019**), but a significant gap remained: large networks can possess many KKT points with varying complexities. Our work bridges this gap by proving that simple, generalizable solutions are structurally preserved as network width increases.\n* This has immediate practical implications. The co-existence of both low-complexity (embedded) and high-complexity KKT points suggests that **initialization and hyperparameter tuning play a critical role in *selecting* a solution**. Our framework provides a theoretical basis for future studying this selection process and designing training procedures that explicitly steer optimizers toward these desirable, low-complexity solutions to improve generalization.\n\n---\n### On Technical Necessity: Why is KKT Point Embedding Needed?\n\nThis is an excellent point that clarifies the core technical challenge. Your intuition that one could prove this by \"showing the neurons remain splitted all the time\" is correct, and our dynamic trajectory preservation result (Theorem 5.2) can be proved by this way.\n\nHowever, Our contribution is best understood in two interconnected parts **for homogeneous networks**:\n1.  **A Static KKT Point Embedding Principle:** This proves that the solutions (KKT points) of a smaller network's max-margin problem are isometrically embedded within the solution set of any wider network derived from it. It uncovers the hierarchical nature of the KKT points of the max-margin problem. Moreover, it opens doors to further studies that identifies all KKT points from low to high complexity, thus obtaining all candidate solutions of the neural network training.\n2.  **A Dynamic Trajectory Embedding Principle:** This part is a further extension to the previous results, in which we realize that the embeddings we use to characterize KKT points can be further employed to characterize the trajectory of training dynamics: the entire gradient flow training trajectory is similarly preserved under our transformation.This result provides us a very powerful tool to charaterize (i) the embedding of non-global solutions (training data not perfectly separated) of neural networks and (ii) embedding of solutions for nonseparable training data. Note that the trajectory embedding principle in it self doesnot imply KKT embedding principle because there is no guarantee that all KKT points are neural network solutions . \n\nA central concern in neural network theory is the connection between the final parameters of a trained network and the solutions to a corresponding KKT problem. To address this, our Static KKT Point Embedding Principle fundamentally relies on techniques related to the KKT conditions. This analytical approach is non-trivial and necessary to establish our results. Subsequently, the Dynamic Trajectory Embedding Principle, which is an extension of the static problem, naturally leverages the same KKT-based framework for its analysis.\n\n---\n\n### On the Generality of the Conditions in Theorem 4.5\n\nThis is a very sharp observation. The conditions in Theorem 4.5—that the transformation $T$ must preserve the network's output and subgradient structure—are indeed strong, and this is by design.\n\nA generic linear transformation would completely change the function computed by the network, making a comparison of the dynamics meaningless. Instead, our conditions define precisely what it means for a transformation to be **structure-preserving**, ensuring that the function and its optimization landscape are isometrically embedded.\n\nTo directly address your question about applications in other scenarios, we have formally extended our framework to homogeneous **Convolutional Neural Networks (CNNs)**.  This required designing a novel **'channel-splitting'** transformation, meticulously constructed to respect the architectural hallmarks of CNNs, such as weight sharing and locality. The transformation isotropically splits a single feature channel into $m_{split}$ new channels via coefficients $\\{c_i\\}$ that satisfy $\\sum_{i=1}^{m_{split}} c_i^2 = 1$. The specific construction is as follows:\n\n* **At Layer k**: The filter $F_j^{(k)}$ that produces channel $j$ is replaced by $m_{split}$ new filters, where the i-th new filter is defined as $c_i F_j^{(k)}$.\n* **At Layer k+1**: In turn, every filter in the next layer that takes channel $j$ as input has its j-th input slice replaced by $m_{split}$ new slices, where the i-th new slice is scaled by the corresponding coefficient $c_i$.\n\nCrucially, we prove that this transformation satisfies the strong conditions of our core theory in **Theorem 4.5**. This is a powerful validation of our framework, showing it provides the rules for designing such embeddings in other complex, homogeneous architectures. Given its significance, **we will integrate this CNN extension as a new main result in Sections 4 and 5 of our revised manuscript**, with an eye toward future applications in architectures like Transformers.\n\n---\n### Minor Writing Issues\n\nThank you for your careful reading and for spotting these issues. We appreciate the detailed feedback and have corrected all of the issues in the revised manuscript:\n\n* The notation for norms and vectors has been made consistent throughout the paper, as you suggested (e.g., using $L^2$ notation).\n* The chain rule expression in **Definition 3.2** has been clarified to resolve the ambiguity, reflecting the structure you proposed.\n* The overloaded symbol $L$ has been resolved. We now use $L$ only for the order of homogeneity and have adopted your excellent suggestion to use $\\omega(\\cdot)$ for the ω-limit set.\n* The stray parenthetical note, which was an internal remnant from a previous draft, has been removed.\n* The typo has been corrected to the proper mathematical notation in Line 300.\n\n---\n\nWe wish to express our sincere gratitude for your exceptionally thorough and insightful review. Your constructive comments were invaluable; they helped us clarify the core contributions of our work and have significantly strengthened the manuscript. We hope these detailed responses address your questions, and that our revised paper will merit your support.\n\n\n\n### References\n\n* **Lyu, K., & Li, J. (2019).** Gradient descent maximizes the margin of homogeneous neural networks. *arXiv preprint arXiv:1906.05890*."}]

Author rebuttal:

Meta-review: The paper enjoyed a fruitful discussion with some reviewers increasing their score. 
Based, on the reviews, the discussion, and my own reading here are the pros and cons:

Strength:
- Interesting approach and, compared to many other NeurIPS papers, a solid mathematical approach
- Theorem 5.2: It which shows that training trajectories are preserved under the considered neuron splitting 

Weakness:
- Writing, narrative, and text organization require a substantial overhaul: Even for a mathematical text there is lot room for improvements
- Strong assumptions on architecture: positive homogeneous networks exclude biases and many activation functions

If the writing was better, I would recommend acceptance. In its current form, however, the paper should be compared to
competing papers in the gray zone. As a result of this comparison, the paper got accepted.
Decision: Accept (poster)