When Are Bias-Free ReLU Networks Effectively Linear Networks?
Authors: Yedi Zhang
Andrew M Saxe
Peter E. Latham
Venue: Accepted by TMLR ()

Abstract: We investigate the implications of removing bias in ReLU networks regarding their expressivity and learning dynamics. We first show that two-layer bias-free ReLU networks have limited expressivity: the only odd function two-layer bias-free ReLU networks can express is a linear one. We then show that, under symmetry conditions on the data, these networks have the same learning dynamics as linear networks. This enables us to give analytical time-course solutions to certain two-layer bias-free (leaky) ReLU networks outside the lazy learning regime. While deep bias-free ReLU networks are more expressive than their two-layer counterparts, they still share a number of similarities with deep linear networks. These similarities enable us to leverage insights from linear networks to understand certain ReLU networks. Overall, our results show that some properties previously established for bias-free ReLU networks arise due to equivalence to linear networks.

Published in Transactions on Machine Learning Research (04/2025)
When Are Bias-Frea ReLU Networks Effectively Linear Networks?
Yedi Zhang
yediiggatsy..uclccco www.bby..cl..cc. uk
Gotsby Computational Neuroscience Unit
University College London
Andrew Saxe
a.sare www...a...c..ooooooo
Gotsby Compulational Neuroscience Unit & Sainsbury Wellcome Centre University College London
Peter E. Latham
pel@gatsby, cecpootsss...c..ccm ucl. uk
Gotsby Computational Neuroscience Unit University College London
Reviewed on OpenReview: https: oppere/e/perreveeu. forum? id- Ucp fdn6kk2
Abstract
We investigate the implications of removing bias in ReLU networks regarding their expres- sivity and learning dynamics. We first show that two-layer bias-free ReLU networks have limited expressivity: the only odd function two-layer bias-free ReLU networks can express a linear onc. We then show that, under symmotry conditions on the data, these networks have the same learning dynamics as linear networks. This enables us to give analytical time-course solutions certain two-layer bias-free (leaky) ReLU networks outside the lazy learning regime. While deep bias-free ReLU networks more expressive than their two- layer counterparts, they still share number of similarities with deep linear networks. These similarities enable us to leverage insights from linear net to understand certain ReL.U networks. Overall, our results show that some properties previously established for bias-free ReLU networks arise due to equivalence to linear networks.
1 Introduction
Theorists make simplifications real-world models because simplified models are mathematically more tractable, yet discoveries made in them may hold in general. For instance, linear models have illuminated benign overfitting (Bartlett et al. 20200) and double descent (Advani et al. 2020 in practical neural networks. In this paradigm, understanding the consequences of simplification is critical, since it informs us which discoveries in simple models extend to complex ones. Here we inspect specific simplification that is common in theoretical work on ReLU networks (Zhang et al., 2019 Du et al. 2019 Arora et al. 2019 Lyu & Li 2020: Vardi & Shamir 2021h: the removal of the bias terms. The removal bias not only appears in theoretical work but also has practical applications. Some real-world models adopt bias removal to introduce scale invariance, which can be a beneficial property for image denoising (Mohan et al. 2020 Zhang al., 2022). image classification (Zarka et al., 2021), and diffusion models (Kadkhodaiee al. 2024). This paper seeks to illuminate the implications of bias removal in ReLU networks, and provide insight for theorists when bias removal is desirable.
We investigate how removing bias affects the expressivily and the learning dynamics of ReLU networks and identify scenarios where bias-free ReLU networks are effectively linear networks. For expressivity, we show that two-layer bias-frce (leaky) ReLU networks cannot express odd functions except linear functions. This was proven for input uniformly distributed on sphere (Basri et al., 2019 Theorem 2 and 4), but we prove it for arbitrary input with simpler approach. We then consider deep bias-free (leaky) ReLU networks and

---

Published in Transactions on Machine Learning Research (04/2025)
show depth separation result, i.c., deep bias-free ReLU networks can express homogencous nonlinear odd functions while two-layer ones cannot. For learning dynamics, we show that two-layer bias-free (leaky) ReLU networks have the same learning dynamics as a linear network when trained with square loss or logistic loss on symmetric datascts, whose target function is odd and imput distribution even. Our symmetry Condition 3 on the dataset incorporates the datasets studied in several prior works (Sarussi el al. 2021: Lyu et al. 2021 Zhang ct al. 2024]. We also present two cases where two-layer bias-free ReLU networks evolve like multiple independent linear networks. Finally, we empirically find that when the target function is linear, deep bias-free ReLU networks form low-rank weights similar to those in deep linear networks.
By revealing regimes where bias-free ReLU networks behave like linear networks, we provide an accessible way of understanding ReLU networks within these regimes, as well as a cautionary note that studying nonlinear behaviors generally requires stepping beyond these regimes. This understanding leverages insights from linear networks, which are simpler and thus enjoy much richer theoretical results than ReLU networks
[aldi & Hornik 1989; Fukumizu 1998 Saxc ct al. 2014 2019 Arora et al., 2018 Ji Telgarsky 2019:
Lampinen& & Ganguli, 2019 Gidel et al. 2019 Tarmoun et al. 2021 Braun et al.) 2022 Ziyin al. 2022).
For example, we are able to give closed-form time-couree solutions to certain two-layer ReLU networks
outside the lazy learning regime in Corollary 9 Our findings suggest that the bias terms in a ReLU network
play an important role in learning nonlinear tasks. Our contributions are the following:
Section proves the limited expressivity of bias-free (leaky) ReLU networks, and shows a depth separation result between two-layer and deep bias-free ReLU networks;
proves that under symmetry Condition 3 on the dataset, two-layer bias-free (leaky) ReLU networks trained with square loss or logistic loss evolve the same as linear networks, and gives analytical time-course solutions for ReLU networks in this regime.
Section 4.1
shows that bias-free ReLU networks behave similarly to multiple independent linear networks on orthogonal and XOR. datasets;
Section 4.2
Section 5 shows the similarities between deep bias-free ReLU networks and deep linear networks, and finds specific rank-one and rank-two structure in the weights.
1.1
Related Work
Basri et al. 2019) proved, using harmonic analysis, that two-layer bins-free ReLU networks can neither learn nor express odd nonlinear functions when input is uniformly distributed a sphere (Basri et al. 2019 Theorem 2 and 4). We make a similar argument with simpler proof. Our Theorem handles arbitrary input, includes both ReLU and leaky ReLU networks, and the proof only involves rewriting the (Ieaky) ReLU activation function as the sum of a lincar function and an absolute value function.
Lyu et al. 2 proved two-layer bias-free leaky ReLU networks trained with logistic loss converge to a linear, max-marge classifier on linearly separable tasks with a data augmentation procedure. Our Theorem g shows that the learning dynamics of leaky ReLU networks in their setup is equivalent to that of a linear network. In light of this equivalence, their result is guaranteed given that linear networks trained with logistic loss converge to the max-margin classifier on linearly separable tasks (Soudry et al. 2018). In addition, we relax the assumption on the task from being linearly separable to being odd, and thus identify a practical challenge: the data angmentation procedure of et al. (2021) can cause the ReLU network to fail to learn a linearly non-scparable task- - a task the network might have succeeded to learn without data augmentation.
Zhang et al. (2024) found that two-layer bias-free ReLU networks have similar loss and weight norm curves as linear networks when trained on datasets with zero mean Gaussian input and linear target. They reported that training the ReLU networks is about twice as slow as their linear counterpart. Our Theorem S explains their observation: we prove that the dynamics of two-layer bias-free ReLU networks is exactly twice as slow as their linear counterpart for a general class of datasets, including theirs.
A few other works have alluded to the connections between two-layer ReLU and linear networks. Sarussi et al. (2021) discovered that two-layer bias-free leaky ReLU networks converge to decision boundary that is very close to linear when the teacher model is linear. Their theoretical results assume that the second
2

---

Published in Transactions on Machine Learning Research (04/2025)
layer is fixed while we train all layers of the network. Saxe et al. 2022 studied gated deep linear networks and found they closely approximate a two-layer bias-free ReLU network trained on an XOR task. But they did not generalize the connection between gated linear networks and ReLU networks beyond the XOR case. Boursier & Flammarion (0244) gave an example dataset with three scalar input data points, in which two- layer bias-free (leaky) ReL.U networks converge to the linear, ordinary least square estimator. Holzmüler & Steinwart 2022) studied two-layer leaky ReLU networks with bias and found that they perform lincar regression on certain data distributions with scalar input, because the bias fails move far away from their initialization at zero.
While prior works have studied cases where bias-frec ReLU networks behave like linear networks, their connections have not been explicitly highlighted or systematically sunmarized. This paper aims to bring the connections between bias- free ReLU and linear networks into focus, offering a comparative perspective on ReLU networks.
2 Preliminaries
Non-bold symbols denote scalars. Bold symbols denole vectors and matrices. Double-pipe brackets ||- denote the L2 norm of a vector or the Frobenius norm of matrix. Angle brackets (-) denote the average over the dataset. The circled dot @ denotes the element-wise product.
Notations:
2.1 Two-Layer Bias-Free (Leaky) ReLU and Linear Networks
A two-layer bias-free (Leaky) ReLU network with H hidden neurons defined as
H
f(;; W) = W20 = Σ h=1
w2nd (w],), where a(2) = max(z, E 1].
(1)
Here x E R.D the input. W1 E HRD is the first-layer weight, W2 E RIxH the second-layer weight, and W denotes all weights collectively. This ReLU network when = 0 and leaky ReLU network when (0,1) When 1, the network is linear network, and can be written f(x) W, W,.. We also denote the linear network flin (x; Wllm) wiii when we need to distinguish it from ReLU networks.
We consider the rich regime (Woodworth et al. 2020 in which the network is initialized with small random weights. The network is trained with gradient descent on a dataset {xxu Fu } consisting of P samples. We study square loss f = (vv- [(=))²) /2 and logistic loss LLG = (In (1 + eu(()))). We focus square loss in the main text and provide derivations with logistic loss in the appendix. The learning rate is and the inverse of the lcarning rate the time constant 5 = 1/n. In the limit small learning rate, the gradient descent dynamics are well approximated by the gradient flow differential equations
+ TW = =[o((Wzz) O W. (y - W20 (Wix) x T)),
(2a)
W ((y W20 (Wix))
(2b)
where is the derivative of or, @ is the element-wit wise product, and the angle brackets (-) denote taking the average over the dataset.
For linear networks, a(z) z, the gradient flow dynamics can be written as
r Wilin Wili T (B will - W = (IT w
(3a)
lin lin wilin" T
(3b)
where Σ denotes the input covariance and B denotes the input-output correlation,
Σ (arar )
(ya).
(4)
3

---

Published in Transactions on Machine Learning Research (04/2025)
2.2
Deep Networks
A deep neural network of depth L is f((x) hy. where hr is recursively defined as
h: (io(hh 25154
(5)
h Wi
Here h1. h are vectors and h is the scalar output. The gradient flow dynamics trained with square loss is
W = ( |ahz
(y (1)o
(6)
Oh,
For deep linear networks, the gradient flow dynamics can be written
)(1
>
L
[
Wi
II wi!h)
B
(7)
M=|+1
where II, W; represents the ordered product of matrices with the largest index on the left and smallest on the right.
3 Network Expressivity
We first examine the expressivity of bias-free ReLU networks. It is well known that standard ReLU networks with bias are universal approximators (Hornik et al. 1989 Pinkus 1999 while bias-free ReLU networks are not since they can only express positively homogeneous functions, i.e., ((xx) = ag(x) Va >> O. Moreover, Section 3.1 shows that two-layer bias-free ReLU networks cannot express any odd function except linear functions. Scction 3.2 shows that deep bias-frec ReLU networks are more expressive than two-layer ones, but are still limited to positively homogeneous fuucction..
3.1 Two-Layer Bias-Free (Leaky) ReLU Networks
The set of functions that can be expressed by two-layer bias-free (leaky) ReLU networks is a subset of the set of functions of the form: f(x) = hu(x:) + g(=), where h((x)) is linear and g(ar) posilively homogeneous even function, meaning g(æ) g( x)) and g(ax) ag(2) Va.
Theorem 1.
two-layer
three-layer
two-layer
three-layer
without bias
without bias
X
with
with bias
bias
(a) Fan dataset.
(b) Circle dataset.
The expressivity of two-layer and deep ReLU networks with and without bias. The networks are trained with logistic loss until the loss stops decreasing. The empty circles are data points with +1 labels; short lines are data points with -1 labels. The network output is plotted in color. (a) The fan dataset is odd, homogeneous, and satisfies Condition Two-layer bias-free ReLU networks cannot express it. (b) The circle dataset is not homogeneous. Two-layer and deep bias-free ReLU networks cannot express it. Experimental details are provided in Appendix
Figure 1:
4

---

Published in Transactions on Machine Learning Research (04/2025)
Proof. Notice that the (leaky) ReLU activation function admits decomposition o(2)
1+a + 1
Thus, any two-layer (leaky) ReLU network can be written as
[1ta
1 1 [1
[z; W)=
Σ
1 (1
W2h
2 +
(8)
2
which is a linear function plus a positively homogeneous even function.
Corollary 2. The only odd function that bias-free two-layer (leaky) ReLU networks can express is the linear function.
Duc to this restricted expressivity, two-layer bias-free ReLU networks fail to classify the fan datasct, as shown in Figure la
3.2 Deep Bias-Free (Leaky) ReLU Networks
Similarly to two-layer bias-free ReLU networks, deep bias-free ReLU networks can express only positively homogencous functions. Thus, as shown in Fig- ure [b both two-layer and deep bias-free ReLU networks fail to classify the circle dataset. However, contrast to two-layer bias-free ReLU networks, deep bias-free ReLU networks can express some odd nonlinear functions. For instance, for two-dimensional input z [2], the function below is odd, nonlinear, and can be implemented by three-layer bias-free ReLU nelwork,
1
R 0
0
-1
((x) = o - (x2)) (a((->)) o(-r2)), where (()) = max(z, (9)
X]
Thus, we have depth separation result for bias-free ReLU networks: there exist odd nonlinear functions, such g(x) defined in Equation (9) and visualized in Figure 2 that two-layer bias-free ReLU networks cannot express but deep bias- free ReLU networks can.
Figure 2: Function g(x;)
defined in Equation (9)
plotted with color.
4
Learning Dynamics in Two-Layer Bias-Frea ReLU Networks
4.1 Symmetric Datasets
3.1 has proven that two-layer bias-free ReLU networks cannot express odd functions except linear functions. We now show that under Condition on the dataset, two-layer bias-free ReLU networks not only find a linear solution but also have the same learning dynamics as a two-layer linear network.
Section
Condition 3. The datasct satisfics the following two symmctry conditions:
The empirical input data distributtion even: p(x:) = =p(=a))
2. The target function is odd: y(z) =-y(=x)
Remark 4. For infinite data, the first part of Condition includes common distributions, such any Gaussian distribution with zero mean. For finite data, the first part of Condition 3 means that if is present in the dataset, x is also present. Condition 3 includes the dataset studied in Lyu et al. (2021). They considered linearly separable binary classification tasks with a data augmentation procedure in which (-x--y) is added to the dataset if (x. y) is in the dataset. We have the same assumption on the input data distribution but relax the assumption on the target function from being linearly separable being odd.
The key implication of Condition g is that the input covariance matrix and the input-output correlation averagod over any half space are equal to those averaged over the entire space. We state this in Lemma and prove it in Appendix B.2
We that decompasing the ReLU activation function into linear and an function is technically incomplicated and has appeared in prior literature before, e.g., Ghorbani all (2021 Section 1.3) and Martinelli 2024 page 4).

---

Published in Transactions on Machine Learning Research (04/2025)
Lemma 5. Let set S+ be on arbitrary half space divided by hyperplane with normal vector r, namely s+ = ( E R r T x 0}. Under Condition 3 we have Vr
(xx =I, E, (zy(œz))s B.
(10)
Recall that and 3 are averages over the entire space as desined in Equation
Under Condition two-layer fies-free (leaky) ReLU networks initialized with small random weights evolve approximately according to a linear differential equation in the early phase of learning. We can solve the approximate linear diffcrential equation and bound the errors of the approximation, leading the following lemma.
Lemma We define the initialization scale as Wint max (()(WW(())) II (O)l)) For time T In -., tillization erhibits exponential growth along one direction with small errors
wints the solution to the dynamics of two-layer (leaky) ReLU networks starting from small ini-
a+1 st rr +Winit)) W2(1)= = - a+1 (i( =e 25
+ ((wiit))
(11)
where || 8 3/s, and rı determined by random initialization T (W)) W₂ [()) /2.
The proof for Lemma can be found Appendix C.1 Lemma G indicates that the weights of the ReLU network and the linear network form the same rank-one structure in the early phase, differing only in the speed of exponential growth. At the end of the early phase, the weights bounded errors, (wwii)) For simplicity, we will assume in Assumption 7 that the weights are exactly aligned and rank-one, which is justified when the initialization is infinitesimal, Whni + 0. Assumption 7 further assumes that W2 has equal L2 norms for their positive and negalive elements. This assumption is supported by the fact that W2 is proportional to Tı, which follows a zero-mean Gaussian distribution because the initial weights W1(0), W2(0) are sampled from a zero-mean Gaussian distribution. As the network width approaches infinity, W2 consists of infinitely many zero-mean Gaussian random samples, which have equal L2 norms for their positive and negative elements.
aligned and rank-one with
Assumption At initialization, there exists an unit vector r such that W W2 T and the 1,2 norms of the positive and negative elements in W2 are equal, that is max 0)ll max 22, 0)]l where max(-) is applied clement-wise.
Theorem 8. A two-layer (leaky) ReLU network and a linear network are trained with square or logistic loss starting from weights which differ by scale factor, (()) 22/(a + 1) Wiln (0). Under Condition the dataset and Assumption 7 the initial weights, the learning dynamics of the two-layer (leaky) ReLU nelwork reduces to
a
(a+1>
W
W₂ TT
W2 W2W,E,
(12a)
2
W = a a
- W.W, WW,
2
2 3 W.T
(12b)
For all t 2 O, Assumption 7 remains valid and the following hold:
1. The (leaky) ReLU network implements the same linear function the lânear network with scaled time
[a;; W()) = plim ar; Wiin
α aatl1) 11))
(13)
2. The weights in the (leaky) ReLU network are the same scaled weights in the linear network
a + 1
W(1) = v V
Win lin
(14)
2

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
Simulation
1.0
Loss
1%
a=
Theory
Enor
== 0.8
Loss
Enror
a 0.6
0.5
0.5%
== 0.4
== 0.2
a 0.0
0.0
0.0
5000
10000
5000
10000
(a) Loss
(b) Loss & error with rescaled time.
Figure 3: Two-layer bias-free (leaky) ReLU networks can evolve like a linear network. (a) Loss curves with different leaky ReLU parameters a (note a = is a linear network). The simulations match the theoretical so- lutions in Equation (15). The loss converges global minimum, which is not zero due to the restricted expres- sivity of two-layer bias- ReLU networks. (b) The simulated loss curves are plotted against a rescaled time axis; they collapse to curve, demonstrating the (leaky) ReLU and linear networks implementing the
same linear function as in Equation (13). The error, defined as |[V W
2 w (t) (/l [|
is less than 0.3%, demonstrating that the weights in the (leaky) ReLU network are close to the weights in the linear network as in Equation (14). The errors are not exactly zero because the initial weights are sampled from zero-mean Gaussian distribution, which does not satisfy Assumption 7 but better reffects practical initialization schemes. Experimental details are provided in Appendix H
The proof for Theorem 8 can be found in Appendix C.2 The key step is that the reduced dynamics of the ReLU network given in Equation (12) is the same as that for the linear network in Equation (3) modulo the constant coefficients. Hence, apart from the fact that learning is + 11/2 times slower and the weights are /2/(a 1) times larger, the ReLU network has the same learning dynamics as its linear counterpart.
We validate Theorem and the plausibility of Assumption i with numerical simulations in Figure B In Figure 3b the initialization is small random Gaussian weights and thus does not satisfy Assumption 7 yet. Theorem holds with small errors (less than 0.3%). Furthermore,
provide theoretical proof that Theorem holds with L2 regularization and empirical evidence that some of Theorem g hold with large
initialization and modcrately large learning rate in Appendices C.4 to C.6
If the input covariance is white, we can further write down the exact time-course solution in closed form for two-layer bias-free (leaky) ReLU networks by adopting the solutions from linear networks (Braun et al. 2022. Theorem 3.1). This gives us the following corollary.
Corollary 9. For learning with square loss, if the input covariance is white, E = I, the solution to Equa- tion 13 is fa: W) = w(t) Ta x with
(t)
+
2xF
91
2sF
+
2
(I
1-3
re -s
Q2
92
Q2
Wiit
2
(1- (r (rT
2z
1
q 2s
(1-0-201)
(15)
where 8 is shorthand for rescoled time i = atit and the constant quantities are || 8 B/s,q1
-rrT8.90 11 B, Wiit = (0)]].
The solution given in Equation
15 matches simulations, as shown in Figure 3a
Since the time evolution of two-layer bias-free (leaky) ReLU networks is the same as that of linear networks (modulo scale factors), their converged weights will also be the same. For learning with square loss, linear networks converge to the ordinary least squares solution (Saxe al.] (2044). For linearly separable binary classification with logistic loss, linear networks converge to the max-margin (hard margin SVM) solution (Soudry et al. 2018). Thus two-layer bias-free (leaky) ReLU networks also converge to these solutions when they behave like linear networks; see Appendix C.3
7

---

Published in Transactions on Machine Learning Research (04/2025)
21
1,0
1.00
R
RelU
Lin +1
2
0.75
+1
Ln -1
0.5
0-
0.50
Lin
-1
-2
0.25
-2
0.0
0.00
10000
20000
10000
20000
(a) Orthogonal data
(b) Loss
(c) XOR data
(d) Loss
Figure 4: Two-layer bias-free ReLU networks evolve like multiple independent linear networks. (a) An orthogonal input dataset used in (Boursier et al. 2022, Figure 3). The + and - signs represent data points with +1 and -1 labels respectively. Their different colors are used only to distinguish the loss curves. The black arrows are the first-layer weights at convergence. (b) The loss curve of the ReLU network overlaps with two linear networks trained on each of the two data points respectively.
(c) An XOR-like dataset. (d) The loss curve of the RcLU network overlaps with four lincar networks trained on each of the four data points separately. Details: We use summed (instead of averaged) square loss for this figure. The initial losses are vertically aligned to help illustrate the overlap. More details are in Appendix H
Corollary 10. Under the same conditions as Theorem 8 the two-layer bias-free (leaky) ReLU network converges to a linear solution f(x; (Oo)) ="" ". For square loss, w" the ordinary least squares solulion, w* E-13, which is the global minimum. For linearly separable binary classification with logistic loss, w" aligns with the maz-margin solution, w* 00/[|="'|| Wavm! [| where
Wgym = argmin||wl| s.t. ynw xu 2 1, 1,
(16)
4.2 Orthogonal and XOR Datasets
In Section
4.1 we showed that for symmetric datasets satisfying Condition 3 a two-layer bias-free (leaky) ReLU network evolves like a linear network. For more general datasets, the equivalence no longer holds, but comparing ReLU with linear networks remains useful for understanding the learning dynamics of ReLU networks. Specifically, we highlight two cases where two-layer bias-free ReLU network evolves like multiple independent linear networks. These cases, i..,, orthogonal input datasct and an XOR-like dataset, are commonly considered in theoretical literature on ReLU net work learning dynamics, while their connection to linear network dynamics has not been previously highlighted.
We first look into datascts with orthogonal input, that is Vp + D. Fr = 0, a common setting in the literature (Boursier et al. 2022 Telgarsky 2023 Frei et al.] 2023b c
Kou al. 2023 Dana el al., 2025). We illustrate with a dataset with two orthogonal data points from Boursier et al. (2022). and handle an arbitrary number of data points in Proposition [8] We train two-layer bias-free ReLU network this dataset (Figure [aa) and reproduce the loss curve in (Boursier two-layer linear networks each data point separately. We find that the timing and the amount of the loss drop overlap with the loss curves of the two linear networks as shown in Figure 4b) To explain this overlap, we plot the first-layer weights of the ReLU network in black arrows in Figure 4a and find that the weights align with either one of the two data points. Since the two directions are orthogonal, the learning dynamics of the two groups of neurons decouple, as derived in Proposition 18 Each group of neurons evolves like a linear network trained on that single data point. Hence, weights in the ReLU network evolve like a linear network trained on either one of the two data points separately. For a dataset with an arbitrary number of orthogonal inputs, the dynamics of the two-layer bias-free ReLU network evolves like two linear networks trained separately on data points with positive labels and data points with negative labels, as validated in Figure The same applies to learning with logistic loss, as shown in Figure
al. 2022 Figure 3). We then train two
We observe similar behaviors the XOR-like task shown Figure 4c XOR-like datasots are also common
setting in the literature (Refnnetii al. 2021 Saxe al. 2022: Frei al.) 2023a Meng et al.) 2024 Xu al.
2024
Glasgow 2024). As shown in Figure 4d we find that the loss curves of two-layer bias- free ReLC
8

---

Published in Transactions on Machine Learning Research (04/2025)
network trained the XOR task overlap with four linear networks trained on each data point separately. In this case, the dynamics of multiple linear networks well approximate that of ReLU network, even though the ReLU network learns a nonlinear function.
In Figures 4b and 4d the loss curves go through multiple drops, each corresponding to learning a data point.
Similar behaviors were examined by Boursier et al. (2022); Xu al. (2024) and characterized as saddle-to- saddle dynamics. The connections we find between ReLU and linear networks may help understand these
behaviors in ReLU networks because saddle-to-aaddle dynamics has been well studied for linear networks
(Saxe et al.
2014
2019
Gissin et al. 2020 Jacot et al. 2022 Berthier
2023
Pesme & Flammarion 2023
5 Learning Dynamics in Deep Bias-Free ReLU Networks
In Section we showed that two-layer bias-fres ReLU networks behave like linear networks under symmetry Condition 3 and small initialization. This does not extend to deep bias-free notworks. When trained on a dataset satisfying Condition 3 deep bias-free ReLU networks can learn nonlinear solutions if the target function is nonlinear, as shown Figure Ia (upper right). Nonetheless, we find deep fiee-free ReLU networks form low-rank weights that are similar to those in deep linear networks. We give an example where the empirical input distribution is even and the target function is lincar.
In deep linear network, weights form an approximately rank one structure and adjacent layers are approx- imately aligned when trained from small initialization (Ji & Telgarsky 2019 Advani et al. 2020 Atanasov et al. 2022: Marion & Chizat 2024]. The rank-one weight matrices can be written approximately as outer- products two vectors
Wiii urir
u
r
(17a)
r1
Wiiin wr
u
ri 1-
1=2, 1=2...,,-11 --
(17b)
ri T
Wiin wrI rrt r
(17c)
where 0 represents the norm of each layer, and T, T1, ... TL are unit norm column vectors. The
vectors rī ry denote the positive and negative elements in rı. The equal norm of all layers is consequence of small initialization (Du et al. 2018). Note that the weights can be written in blocks, Equation (17) only after permuting the positive and negative elements. We use this permuted notation for the sake of exposition; no additional assumptions are required.
In deep ReLU network, we empirically find that when the weights are trained from small initialization and the target function is linear, the weights exhibit a particular rank-one and rank-two structure, as shown Figures and I0. For the first and last layers, the weights in the deep ReLU network have the same rank-one structure as their linear counterpart. For the intermediate layers, weights in the deep ReLU network are rank-two. Specifically, positive weights in the ReLU network correspond to positive weights in the linear network and zero weights in the ReLU network correspond to negative weights in the lincar network. Basod on the empirical observation, we propose the following conjecture on the weights of the deep ReLU network.
Conjecture 11. deep bias-free ReLU network is trained from small initial weights on a dataset where the empirical input distribution is even, p(x)= = p(-x), and the target function linear, that is the target output generated as y = w" x. We conjecture that the weights at a certain time to during training take the following form:
rrt
Wi = uri
(18a)
V2ri r
0
W
1= 2,
,L- 1,
(18b)
22r Ti-1
T
Wz ur 1-1
r
T
rī-1
(18c)
9

---

Published in Transactions on Machine Learning Research (04/2025)
W
W5
W:
(a) Weights in 3-layer linear network Equation 17
(b) Weights in 3-layer ReLU network as Equation
18
Figure 5: Low-rank weights in deep linear and ReLU bias-free networks. A three-layer linear network and a throe-layer ReLU network are trained on the same dataset starting from the same small random weights. The dataset has a linear target function and an even empirical input data distribution. We plot the weights when the loss has approached zero. W1, W3, and positive elements in W2 have approximately the same structure in the linear and ReLU networks. Elements of W2 that negative in the linear network are approximately zero in the ReLU network. The neurons are permuted for visualization. Simulations with deeper networks are presented in Figure 10 Experimental details are provided in Appendix H
where the notations are consistent with Equation 1,2, 1.
We also conjecture that |rt|
1r
Proposition 12. If Conjecture 11 is true, then for all 2 to, the weights of the deep bias-free ReLU network will maintain the form in Equation 18 and the network implements a linear function given by
f(x: = WWWL-I (21(W WL W2W13.
(19)
2
Moreover, its learning dynamics is described by
L
B )
1 2
L I --=-1
W = II Wr
II
W
Wr
(20)
which is the same as that of deep linear network in Equation
modulo the constant coefficients.
We provide the proof of Proposition 12 in Appendix E and offer a brief explanation for Equation 19 here. In the first equality of Equation 19 we dropped the activation functions except for the one between the first and second layers. This is because the second layer weights, W2, is non-negative as shown in Figure 5b. and so is the output of ReLU activation function, oW, Hence, their product, W21(W, is also non-negative. We thus have (Wzzo(W1z)) second equality obtained by substituting the weights defined in Equation (18) into the expression.
(21(W The same applies all subsequent layers. The
While the proof of Conjecture 11 remains open question, provide some intuition to interpret it. Specifically, we notice that in deep linear networks and certain deep ReLU networks, the weights of an intermediate layer align with the inputs to that layer. For example, the second-layer weight in a deep linear network is W = ur2ri as given in Equation (17). Every row of W aligns with rí which is parallel to any input to the second layer, Will = urir æ. The socond-layer weight the deep ReLU network is
given in Equation 18). Some rows of W₂ align with |rt o]. which is parallel to some inputs (r > 0)
rrt T Other rows of W2 align with O ri T. which is parallel with
to the second layer, a (Wix) u
0 T This alignment phenomenon has been proven
inputs (r T x < 0) to the second layer, (Wix)= =
for deep linear networks [Ji & Telgarsky 2019 empirically in deep ReL.U networks.
Marion & Chizat 2024), and we here find similar phenomena
10

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
0 (syen)
0d
ReLU
6-0.10
Uiaas
0.5-
0.14
0.1016
0.0
-1
-++
10000
100000
0.0
0.1
0.2
0
(a) Loss
(b) Plateau duration
(c) == 10000
(d) == 100000
Figure 6: Two-layer bias-frec Chear/ReLAr network trained on datasct that slightly violates the symmetry Condition 8 The + and signs represent data points with +1 and - labels respectively. The right middle data point is slight asymmetric with input coordinates (1,6). (a) Loss curves of the ReLU network with different and the linear network with 0.1. (b) The duration of the plateau, during which the ReLU network implements a nearly lincar solution, scales approximately with 1/6. (c,d) The ReLU network output during and the end of training. This ReLU network is trained on the dataset with 8 0.1. Experimental
details are provided in Appendix H
6 Discussion
6.1 Implication of Bias Removal
We studied the implications of removing bias in ReLU networks in terms of the expressivity and learning dynamics. Theorem Sicoormm[shows shows that two-layer bias-free (leaky) ReLU networks cannot express any odd functions except for linear functions. Theorem X shows that for datasets with even input distribution and an odd target function, two-layer bias-frec (leaky) ReLU networks have the same time evolution as a lincar network (modulo scale factors) under initialization Assumption 7 We also presented examples in which the bias-free ReLU network evolves like multiple independent linear networks, in Section 4.2] In these cases, comparing a bias-free ReLU network with its linear counterpart provides an intuitive understanding of the behavior of ReLU networks. On the flip side, the simplicity of bias-free ReLU networks suggests that ReLU networks with bias may exhibit more complicated behaviors, which not fully addressed by studies on bias-free networks, and remain open questions.
One common argument in studies of bias-free ReLU networks is that we can stack the input with an additional one, i.e., à = [ 1]. The behaviors of a biased ReLU network with input x are the same as those of a bias-free ReLU network with input æ, suggesting that the implication of bias removal might be trivial. While this argument valid in certain scttings (Allen Zhu et al. 2019 Zou al. 2020). it comes with important caveats in others. For example, [Soudry al. (2018) showed that two-layer bias-free ReLU networks trained with logistic loss converge to the max-margin classifier lincarly separable datasets. As clarified by Soudry et al. (2018), this technical result holds when the inputs are stacked with an additional one. However, the max- margin solution of the dataset {mp.. y is not the max-margin solution of the original dataset {æ ya 1- Hence, a ReLU network with bias converges to solution different from the max-margin solution obtained by the bras-free ReLU network. This distinction highlights that the presence of bias terms can change the inductive bias of the ReLU network, leading to convergence to different solutions.
6.2 Perturbed Symmetric Dataset
We have shown an exact equivalence between two-layer bias-free (leaky) ReLU networks and linear networks under symmetry Condition g on the dataset in Theorem g In practice, datasets satisfies Condition 3 precisely. However, two-layer bias-frec ReLU networks may still struggle to fit a datasct that approximately satisfies Condition 3 We present simple example with six data points in Figure G The ReLU network loss curve closely follows the linear network loss curve in the early phase, when it first learns a nearly linear solution, as shown Figures (aa] and [c] After a plateau, the ReLU network diverges from the lincar network dynamics and converges to a nonlinear solution. The more symmetric the dataset, the longer the plateau two-layer bias-free ReLU network undergoes before learning a nonlinear solution. When the dataset is
11

---

Published in Transactions on Machine Learning Research (04/2025)
exactly symmetric, the ReLU network never learns a nonlinear solution. As shown in Figure 6b the inverse of the plateau duration scales approximately linearly with the deviation of the asymmetric data point, 6. The scaling becomes less accurate for larger ó because the corresponding dataset more severely violates symmetric Condition 3 where the ReLU network no longer behaves like a linear network. Furthermore, as shown in Figure [d the decision boundaries at convergence are close to the data points, and thus probably not robust.
We note concurrent work (Boursier & Flammarion, 2024bb that considers two-layer bias-free ReLU networks trained on symmetric datasets with different form of perturbation. They assume the empirical input distribution is even and the target output is generated as y w' x + &. where is the noise sampled independently from & zero-mean Gaussian distribution. In their setup, the ReLU network converges to the ordinary least square linear estimator when the number of training samples exceeds a certain threshold. In their case, the mean of the noise approaches zero as the number of training samples increases, which reduces the asymmotric perturbation and leads convergence to a linear solution. By analogy, in our case, a smaller ó means smaller asymmelric perturbation, which leads to longer plateau during which the ReLU network is stuck at linear solution.
Acknowledgement
The authors are grateful to Peter Orbanz, Ingo Steinwart, Spencer Frei, Rodrigo Carrasco-Davis, Lukas Braun, Clémentine Dominé, Zheng He, and William Dorrell for helpful discussions. The authors thank the following funding sources: Gatsby Charitable Foundation (GAT3850) to YZ, AS, and PFI, Sainsbury Wellcome Centre Core Grant from Wellcome (219627/Z/19/Z) to AS; Schmidt Science Polymath Award to AS; Wellcome Trust (110114/2/15/2) to PEL.
References
Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generaliza- tion error in neural networks. Neural Networks, 132:428 446, 2020. ISSN 0893-6080. doi: https:// http://0.010000..mmmt20000000000
doi.org/10.1001 nwwnct.2020.08..22.... URL http:::/ww..cceeeeeeeree.. $0893608020303117
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedliggs of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 242-252 PMLR, 09-15 Jun 2019. URL httpp::/pproceediiiiee
mlr. www.errressse/al/eee allenn wwwllennnnuliaaa.m zhu19a. html
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 244-253. PMLR, 10 15 Jul 2018. URL http:::/prroceeedggg.. .mlr. wtt.mm/.........e.....emmmmmeeee
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparametteizee two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97
of Proceedings of Machine Learning Research, pp. 322-332. PMLR, 09-15 Jun 2019. URL https://
proceedings. mlr. #rees//S7//aoora996 html
Alexander Atanasov, Blake Bordclon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022. URL https:
//openreview.netee
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from ex- amples without local minima. Neural Networks, 2(1):53-55 58, 1989. ISSN 0893-6080. doi: https:
//dol.rrg/10.101//0088 (000:/080000000000999000000-0000 pi1/008936000899000122
URL https://www.sciencee/ www.://cweeeiieee..eoeeeeeeeeeee
12

---

Published in Transactions on Machine Learning Research (04/2025)
Peter L. Bartlett, Philipp M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in lin- ear regression. Proceedings of the National Academy of Sciences, 117(48)330033-30077 2020. doi: (0.1078/pnas.190777887 URL
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neural networks for learned functions of different frequencies. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, F.. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL Ittpss//pproced htt.:anppps.ccpppppp..eeeee////e Paper par
Raphaäl Berthier. Incremental learning in diagonal linear networks. Journal of Machine Learning Research, 24(171):1-22 26, 2023. URL / wtt/paaper//////2222222222 hmmm
Etienne Boursier and Nicolas Flammarion. Early alignment in two-layer networks training is a two-edged sword, 2024a. URL https://arxiv.org// wwwpx//worggas////oeee
abs/ wttp://///bb///1..100999// 2471.
Etienne Boursier and Nicolas Flammarion. Simplicity bias and optimization threshold in two-layer relu networks, 2024b. URL sttp:://aaaxxvoor///b///111
Etienne Boursier, Loucas Pillaud- and Nicolas Flammarion. Gradient flow dynamics of shallow relu networks for square loss and orthogonal inputs. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volune 35, pp. 20105 20118. Curran Associates, Inc., 2022. URL Paper--Paaarrrrrrreaeee
ctppppp.cppppppppppppppp.pp.p hwwpri/ss.c/apppe__lll///paappp 2022/ ht00://2277eeebteeoreeo222226260 ohgpapaoeeeeaaaaea22ooo-eee--eee
pdf
Lukas Braun, Clémentine Dominé, James Fitzzerald, and Andrew Saxe. Exact learning dynamics of deep linear networks with prior knowledge. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 6615 6629. Cur- ran Associates, Inc., 2022. URL https://proceeding..nt wwwp::/innw....r...ee.o.... ctcpspsrcppppprrrpee http:pssccccppprrrreeeeee/eeeeee wtt.e///pppr///2//////////
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev Shwaar-S SGD learns ver-parameeerizzd networks that provably generalize on linearly separable data. In International Conference on Learning Representations, 2018. URL lttpps://openrevien http:s//ppparrrreeeeeteeefooorr/ httt///orummddrr333warettt
Dmitry Chistikov, Matthias Englert, and Ranko Lazic. Learning a neuron by a shallow relu network: Dynamics and implicit bias for correlated inputs. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 23748 23760. Curran Associates, Inc., 2023. URL httppp::/prroeeedingg neuriss.e.pppp.ppp ceppppp.c.p........pp...r.. wttppppr..peeeeeeeeeeeppppppppee files/ pdf
Léo Dana, Francis Bach, and Loucas Piliend-Vivien. Convergence of shallow relu networks on weakly interacting data, 2025. URL /tttps///arxi, asspabb/.///2/1//1
Simon Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Informalion Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https::/prooeeddingg www.iiinnnn..uu....mmmmm cwturips.cc/pple paper http://pppp..lleepppp/////////// fe131d
pdf.
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. In International Conference on Learning Representations, 2019. URL
Spencer Frei, Niladri S. Chattorji, and Peter L. Bartlett. Random feature amplification: Feature learning and generalization in neural networks. Journal of Machine Learning Research, 24(303):1-44 2023a. URL http:////ilr..orr/pppps////22221 html
13

---

Published in Transactions on Machine Learning Research (04/2025)
Spencer Frei, Gal Vardi, Peter Bartlett, and Nathan Srebro. Benign overfitting in linear classifiers and leaky relu networks from kkt conditions for margin maxinization. In Gergely Neu and Lorenzo Rosasco (eds.), Proceedings of Thirty Sirth Conference on Learning Theory, volume 195 of Proceedings of Machine Learning Research, pp. 3173-3228. PMLR, 12-15 Jul 2023b. URL v195/frei23a.. htmll
mlr press/
Spencer Frei, Gal Vardi, Peter Bartlett, Nathan Srebro, and Wei Hu. Implicit bias in leaky reLU networks trained on high-dimensional data. In The Fleventh International Conference on Learning Representations, 2023c. URL htttps://operreiiee net/forrumidd--pooyEEEE..
Spencer Frei, Gal Vardi, Peter Bartlett, and Nati Srebro. The double-edged sword of implicit bias: Gener- alization vs. robustness in relu networks. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Aduances in Neural Information Processing Systems, volume 36, pp. 8887-8897. Cur- ran Associates, Inc., 2023d. URL https: httpp::://pocccciiinneeee//e//ee itwprooeeiiinnneeerrioo.o....... (C763386601190044446666666666 ottp://ewwaeeeeiieeeeeoorrrrrrna Paper-Coo wtt.rrpp.cc/ppppr...f.eeea/ppppp Filee//ppeee///22////eeeee
Kenji Fukumizu. Effect of batch learning in multilayer neural networks. Gen, 1(04):1E-033 1998.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two- two-layers neural networks in high dimension. The Annals of Statistics, 49(2):1029 - 1054, 2021. doi: 10.1214/20-AOSS
URL https://doi. org/10. 2214/20-A0S1990
Gauthier Gidel, Francis Bach, and Simon Lacosto-Julien. Implicit regularization of discrete gradient dynamics in linear neural networks. In H. Wallach, Н. Larochelle, A. Beygekzimer F. d' Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32, Cur- ran Associates, Inc., 2019. URL stttp::/prrocedingg neurips. neurips.cc/paper wwtppaper.illls/ppppeeeeeeeeeeee filess/papeee//11///ell 13000fffaa1111223042266011111111 Pappooe Paper.ddf
Daniel Gissin, Shai Shalev-Shwartz, and Amit Danicly. The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations, 2020. URL https://
openreview..et/foruutff www.://ooun.dditi.ioo...........
Margalit Glasgow. SGD finds then tunes features in two-layer neural networks with near-optimal sample complexity: A case study in the XOR problem. In The Twelfth International Conference on Learning Representations, 2024. URL
T. H. Grönwall. Note on the derivatives with respect to parameter of the solutions of system of differential equations. Annals of Mathematics, 20(4):292 296, 1919. ISSN 0003486X. URL http:://ww.jstttt/ org/
stable/1967124
David Holzmüller and Ingo Steinwart. Training two-layer relu networks with gradient descent is inconsistent. Journal of Machine Learning Research, 23(181):1 82, 2022. URL http:/pp///al..oog/ppp/s//////// html
Kurt Hornik, Maxwell Stincheombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks, 2(5):359-366, 1989. ISSN 0893-6080. doi: httpss//doi.
org/10.1016//0890 (00000//000000000089990090990000 08936080899002088
URL
Arthur Jacot, François Ged, Berfin Şimşek, Clément Hongler, and Franck Gabriel. Saddle-to-suudll dynamics in deep linear networks: Small initialization training, symmetry, and sparsity, 2022. URL httts://aarxi
org/ 2106. 15933
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference Learning Representations, 2019. URL http:://pppaneeess..ee//rrrrrrep
Zahra Kadkhodaic, Florentin Guth, Ecro P Simoncelli, and Stéphanc Mallat. Generalization diffusion models arises from geometry-adaptive harmonic representations. In The Twelfth International Conference on Learning Representations, 2024. URL. http::///gaaaaarieeeeeee/////ear
14

---

Published in Transactions on Machine Learning Research (04/2025)
Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks on nearly-orthogonal data. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Newral Information Processing Systems, volume 36, pp. 30167 30221. Cur- ran Associates, Inc., 2023. URL https::/ppocceed/eeee
neurips weurips.cc.paperr.... httpriissccppppprf_eeeeeeee filess/paper/222//11o
Andrew K. Lampinen and Surya Ganguli. An analytic theory of generalization dynamics and transfer learning in deep linear networks. In International Conference on Learning Representations, 2019. URL
Thien Le and Stefanie Jegelka. Training invariances and the low-rank phenomenon: beyond linear networks. In International Conference on Learning Representations, 2022. URL https:///penreviev..net.t id=XEW8COgArno
Tao Luo, Zhi-Qin John Xu, Zheng Ma, and Yaoyu Zhang. Phase diagram for two-layer relu neural networks infinite-winn width limit. Journal of Machine Learning Research, 22(71):1 47, 2021. URL http://jmlr;
hwg/papprssv22220000
html
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks, In International Conference on Learning Representations, 2020. URL https:://opperreeiiett net//forummdd ddd. SJeLIgBKPS
Kaiseng Lyu, Zhiyuan Li, Runzhe Wang, and Sanjeev Arora. Gradient descent on two-layer nets: Margin maximization and simplicity bias. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, volume 34, pp. 12978-12991. Cur-
ran Associales, Inc., 2021. URL htttt:://proccednnng
ceourips.cc/paaee httpp////c.ppppee..eee//p/pp////
Hartmut Maennel, Olivier Bousquet, and Sylvain Gelly, Gradient descent quantizes relu network features, 2018. URL htttps://axxii....... org/ 1wtp://11...08800
Pierre Marion and Lénaïc Chizat. Deep linear networks for regression are implicitly regularized to- wards flat minima. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 76848 76900. Cur- ran Associates, Inc., 2024. URL https:// newrriinnnnnnnnnnsnoooeeo neurips cttpsscccpppppreeeeeeee wttppppr_fillespppprr////e/e//ee fille//pppo/2222///1l///
Flavio Martinelli, Berfin Simsck, Wulfram Gerstner, and Johanni Brea. Expaadd-nnd-custerr Parameter recovery of neural networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 34895 34919. PMLR, 21-27 Jul 2024. URL https: htttss:/prrceedeee mlr.rnggsrrr.....mm.. wttppppsss/s//pppppppppp 22235/martinel12445 html
Xuran Meng, Dilan Zou, and Yuan Cao. Benign overfitting in two-layer relu convolutional neural networks for XOR data. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. -54449.54. PMLR, 21-27 Jul 2024. URL hattps:/pproceednng: mlr wwwps//v2//mmeeeeeeeeeeeeee html
Hancheng Min, Enrique Mallada, and Rene Vidal. Early neuron alignment in two-layer relu networks with small initialization. In The Twelfth International Conference on Learning Represenlations, 2024. URL http://ppwweeeeeeeeeeeerrrrraaaa
Sreyas Mohan, Zahra Kadkhodaic, Ecro P. Simoncelli, and Carlos Fernandez- Granda. Robust and inter- pretable blind image denoising via bias-free convolutional neural networks. In International Conference on Learning Representations, 2020. URL. http://e/.gaaaaaeeee...ea////aed
15

---

Published in Transactions on Machine Learning Research (04/2025)
Mor Shpigel Nacson, Rotem Mulayoff, Greg Ongic, Tomer Michaeli, and Daniel Soudry. The implicit bias of minima stability in multivariate shallow reL.U networks. In The Eleventh International Conference Learning Representations, 2023. URL https://openreviex http://ppeeeeeeeeeeeeeet/ftrrree netffoormm?ddxxxbordff
Grog Ongie, Rebecca Willett, Daniel Soudry, and Nathan Srebro. A function space view of bounded norm infinite width relu nets: The multivariate case. In International Conference on Learning Representations, 2020. URL https://openrevieer httpps//eppaaaooeeeeeeaeoorrataa
Scott Pesme and Nicolas Flammarion. Saddle-to-saddle dynamics in diagonal linear networks. In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Ad- vances in Neural Information Processing Systems, volume 36, pp. 7475-7505. Curran As- sociates, Inc., 2023. URL https://proceeddngss neureedipdnngsn.unn... neurips. wwwurip..ccpppppeeeeeeeeee cc/paper 1tlle//paper/222//E1lee Trab:10099eee000BBBadaaaaappppp0
Leonardo Petrini, Francesco Cagnetta, Eric Vanden-Fijnden, and Matthieu Wyart. Learning sparse features can lead to overfitting in neural networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Newral Information Processing Systems, volume 35, pp. 9403-9416. Cur- ran Associates, Inc., 2022. URL httpp::/ www.ips.eripcccm Giaas3885000caaeee11111100000000
newrips.cc/ppppprrrrrr cc/paper files//paperr2222//ibel
Leonardo Petrini, Francesco Cagnetta, Eric Vanden-Eijndeen, and Matthieu Wyart. Learning sparse fea- tures can lead to overfitting in neural networks. Journal of Statistical Mechanics: Theory and Experi- ment, 2023 (1))::14000333 nov 2023. doi: 11088/1742-5405/aadee 1142-5468/ad0111
URL https: //dx:d//xi.oiiioo doi wtw.bxooo..rgg/..11111 org/10.1088/
Mary Phuong and Christoph H. Lampert. The inductive bias of relu networks on orthogonally separable data. In International Conference on Learning Representations, 2021. URL https:://penreevie net/
forum?id=kr2/7oxxez)
Allan Pinkus. Approximation theory of the mlp model in neural networks. Acta Numerica, 8:143-195, 1999.
doi: 10. 10.1017/5002244990000009
Maria Refinetti, Sebastian Goldt, Florent Krzakala, and Lenka Zdeborova. Classifying high-dimensional gaussian mixtures: Where kernel methods fail and neural networks succeed. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 8936-8947. PMIR, 18 Jul 2021. URL https:
//proceedings.co mwr. /ttp:///ig.......ppre/////////// http://vv111reffeetttoe
Roei Sarussi, Alon Brutzkus, and Amir Globerson. Towards understanding learning in neural networks with linear teachers. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 9313 9322. PMLR, 18-24 Jul 2021. URL httpss:/proceeedee mlr. mlr httpss//111///aassss.a... html
Andrew Saxe, Shagun Sodhani, and Sam Jay Lewallen. The neural race reduction: Dynamics of abstraction in gated networks. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 Proceedings of Machine Learning Research, pp. 19287-19309. PMLR, 17-23 Jul 2022. URL https: //proceedings.. mml. pttps//v12/saaee22/222 html
Androw M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014.
URL https://arxiv..rgg/ /ttp///aiaaaaraaaa///////////.11 htt////b111/11.111111
Andrew M. Saxe, James L. McClelland, and Surya Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116((3) 2019. doi: 10.1073, ppaa..
htt///maa.11222200000 18202261 16. URL httts::/pp//ttt http://wwaapaarrrroo//////////lp
16

---

Published in Transactions on Machine Learning Research (04/2025)
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-50 57, 2018. URL
Salma Tarmoun, Guilherme Franca, Benjamin D Haeffele, and Rene Vidal. Understanding the dynamics of gradient flow in overparameterized lincar models. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learn- ing Research, pp. 10153-10161. PMLR, 18 18-24 Jul 2021. URL tarmoun21a.
mmlr mlr, Saaar.press//ee press/v139/
Matus Telgarsky. Feature selection and low test error in shallow low-rotation relu networks. In The Fleventh International Conference on Learning Representations, 2023. URL httpg:://operreviie.ntt//
swEsxiem99
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu net- works. Shipra Agrawal and Francesco Orabona (eds.), Proceedings of The 34th International Conference on Algorithmic Learning Theory, volume 201 of Proceedings of Machine Learning Research, pp. 1459. PMLR, 20 Feb 23 Feb 2023. URL httpss:/pproeeedigg..m mlr press/v201/timmr222 ing.aeeeeaaaaaattpppooeoooooooaa html
Gal Vardi and Ohad Shamir. Implicit regularization relu networks with the square loss. In Mikhail Belkin and Samory Kpotufe (eds.), Proceedings of Thirty Fourth Conference on Learning Theory, volume 134 of Proceedings of Machine Learning Research, pp. 4224 4258. PMLR, 15 19 Aug 2021. URL https:
//procediiggs.o mlr www.mll.......... wwt.pp/ess////eeee
haaess/vvv4// www.//1/...221bbboo html
Gal Vardi, Gilad Ychudai, and Ohad Shamir. Gradient methods provably converge to non- robust networks. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 20921 20932. Curran Associates, Inc., 2022. URL https::/prooceednng/ neurips. couuips.cc/ppperrrr / Rae913572bbbbbab44400eeeeee
Gang Wang, Georgios B. Giannakis, and Jie Chen. Learning relu networks on linearly separable data: Algorithm, optimality, and generalization. IEEE Transactions on Signal Processing, 67()))2357-2370. 2019. doi: 21109/TSP:0199299444
Mingze Wang and Chao Ma. Understanding multi-phase optimization dynamics and rich nonlinear behaviors of reLU networks. In seirth-seventh Conference on Neural Information Processing Systems, 2023. URL
https://openreviee
Yifci Wang and Mert Pilanci. The convex geometry of backpropagation: Neural network gradient flows converge to extreme points of the dual convex program. In International Conference on Learning Repre- sentations, 2022. URL httpp:://penreevier http://www.....e....o.oooaaaa/aa
Yifei Wang, Yixuan Hua, Emmanuel J. Candès, and Mert Pilanci. Overparameterized relu neural networks learn the simplest model: Neural isometry and phase transitions. IEEF Transactions on Information Theory, 71(3):1926 1977, 2025. doi: 10.1109 202...00...33335555.
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirly Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 3635 3673. PMLR, 09-12 12 Jul 2020. URL https: //proceedings...
.ml...g.........mmmm.mm.....m..m mlr, grres//v155/ooodorrtt
html
Zhiwei Xu, Yutong Wang, Spencer Frei, Gal Vardi, and Wei Hu. Benign overfitting and grokking in relu networks for xor cluster data. In The Twelfth International Conference on Learning Representations, 2024.
URL httpps://penrreiiee..ee wwtp://eeettttttttteeettaao wwt.:e////ooooiiddddedeeer..
John Zarka, Florentin Guth, and Stéphane Mallat. Scparation and concentration deep networks. In International Conference on Learning Representations, 2021. URL BHhkbjrWLdE
17

---

Published in Transactions on Machine Learning Research (04/2025)
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Luc Van Gool, and Radu Timofte. Plug-and-ple image restoration with deep denoiser prior. TEEE Transactions on Patlern Analysis and Machine Intelligence, 44(10):6360 6376, 2022. doi: 2TL.0//TAAAAM.222.00880008.
Xiao Zhang, Yaodong Yu, Lingxiao Wang, and Quanquan Gu. Learning one-hidden-layer relu networks via gradient descent. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Arlificial Intelligncce and Slalislics, volume 89 of Proceedings of Machine Learning Research, 1524-1534 PMLR, 16 18 Apr 2019. URL https://proceeinnss. mlr wwt.ggsss......ss////// v89/ ztt///aannng.... htmmgl99g.tmm1111
Yedi Zhang, Peter E. Latham, and Andrew M Saxe. Understanding unimodal bias in multimodal deep linear networks. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 59100-59125. PMLR, 27 Jul 2024. URL
mlr press/ /225///v23//// zhang24aa html
Liu Ziyin, Botao Li, and Xiangming Meng. Exact solutions of deep linear network. In S. Kayejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Nearal Information Processing Systems, volume 35, pp. 24446-24458. 24458. Curran As- sociates, Inc., 2022. URL https://proceeding..ee wwwpinnw.nnrrri...ooooooooo httpipp.cc/ppppfffleeeeeeeeeeeee ftlles/papeer/222//ii
Difan Zou, Yuan Cao, Dongruo Zhou, and Quanquan Gu. Gradient descent optimizes over-parameterizel deep relu nelworks. Machine learning, 109:467 492, 2020.
18

---

Published in Transactions on Machine Learning Research (04/2025)
A Additional Related Work
Implicit/ / Simplicity Bias. Many works have studied the implicit bias or simplicity bias of two-layer bias-
free ReLU networks under various assumptions the dataset. [Butzkuu et al. (2018) : Wann et al. (2019)
Lyu et al. (2021) Sarussi et al. (2021] :
Wang & Ma (2023) considered linearly separable binary classification
tasks. Phuong & Lampert (2021) ; Wang & Pilanci 2022]; Min et al.
(2024) studied orthogonally separable
dlassification (i.e., where for every pair of labeled examples (at, Vi). 2
we have x 2;, > 0 if y/
and x; 2/ - 0 otherwise). Boursier et al. (2022); ); Frei et al. (2023b ct; Kou et al. (2023 : Dana et al.
(2025) studied binary classification with exactly or nearly orthogonal input (i.c., where 2; =; 0 if + j).
Orthogonal input is sufficient condition for linear separability for binary classification tasks. Relinetti
et al. [021]; Frei et al. (2023a); Meng et al. (2024); Xu et al. (2024) studied XOR-like datasets. Vardi
et al. 2022]; Frei et al. (2023d) studied datasets with adversarial noise. We add to this line of research by studying a case with extreme simplicity bias, i.c., behaving like linear networks.
Low-Rank Weights, Maennel et al. (2018) the seminal work on the low-rank weights in two-layer ReLU networks trained from small initialization. They described the phenomenon as "quantizing', where the first layer weight vectors align with small number of directions in the early phase of training. [uo et al.) (2021) identified when two-layer bias-free ReLU networks form low-rank weights in terms of the initialization and the network width. Timor et al. (023)) provided cases where gradient flow two-layer and deep ReLU networks provably minimize not minimize the ranks of weight matrices. Frei et. al. (2023c); Kou et al. (2023) computed the numerical rank of the converged weights of two-layer bias-free ReLU networks for nearly orthogonal datasets, and found that weights in leaky ReLU networks have rank at most two and weights in ReLU networks have & numerical rank upper bounded by a constant. Chistikov showed two-layer bras-free ReLU networks are implicitly biased to learn the network of minimal rank under the assumption that training points are correlated with the teacher neuron. Min al.) (2024): Boursier & Flammarion (0024a) studied the carly phase learning dynamics to understand how the low-rank weights
al. (2023)
form Petrini et al. 2022 2023) conducted experiments practical datasets to show that two-layer bias-Tree
ReLU networks learn sparse features, which can be detrimental and lead to overfitting. Le & Jegelka (2022)
generalize the low-rank phenomenon in linear and ReLU networks to arbitrary non-homogeneous networks whose last few layers contain linear fully-connected and linear ResNet blocks.
B
Useful Lemmas
B.1 Grönwall's Inequality
Grönwall's Inequality /Grönwall 1919 is a common tool to obtain error bounds when considering approxi- mate differential equations.
Lemma 13 (Grönwall's Inequality). Let denote an interval of the real line of the form [a, oc) or [a. bj or ja, b) with a < b. Let B and u be real-valued functions defined on I. Assume that 3 and u are continuous and that the negative part of a is integrable on every closed and bounded subinterval of I. If B is non-negative and u satisfies the integral inequality
m(1) s a(()) + f B(s)u(s)ds, VL
then
u(t) <a()) I' a(s))(s)el s()rr ds, tel.
(21)
B.2 Data Statistics
We here prove Lemma which states the input covariance matrix and the input-output correlation averaged any half space are equal to those averaged over the entire space, under Condition 3
19

---

Published in Transactions on Machine Learning Research (04/2025)
Proof of Lemma Define
S- {I R < 0}. Because Condition states that p(z) is even, have
/ plat)da = / p(ar)de 2
By the definition of the conditional expectation, we have that
(xx > Jst axx TurTples p(æ:)dæ
= Jst ax TeTple p(x)ddx
Ssl p(
Jst xy(())p())dd Sg p( /f Jst
(xy(œ))s
xy((x)p(x)dd
Because a.w p(x) is an even function about æ, the integral of a:." p(æ)) over St or S is the same
[ ax pla)ddx = xx T p(
Thus, the average of aa T over S- is equal to the average in the entire space,
Σ xx p(ar)dax / xw plac)da =2 / wx p(a))dx (xxx )
(22)
Si
Because xy(x) is also an even function about æ under Condition 3 the same argument holds
/
3
zy(z))(())dz + my(x))p(x))d= = zy(x))p(z)dz = (xy(()))
(23)
s+
Note that Equation
23
needs both conditions in Condition that are p(æ) = p(-x) and v(x) = v2) only needs the first condition, that p(x) = p(=)))
However, Equation
22
Lemma 14. Under Condition
the first terms in the differential Equation
reduce to
(o'(W WT yæ
a a+
W a a+ 9 W.T. 2
ST.
(24a)
(Wa))
(24b)
Proof- Let consider the h-th of the matrix (o'(War) W₂ yæu which is
a a 2
(oownny
(w2nyx)
where the first equality is the law of total expectation and the second equality uses Lemma same holds for all rows, Equation (4a)) is true.
5
Because the
Let us consider the h-th element of the row vector (yo(W )) which
(yo(w1)))) = (a wne
+ ( (y )
++ 2 WinB,
where the first equality is the law of total expectation and the second equality uses Lemma same holds for all elements, Equation (4b is true.
5 Because the
Lemma 15.
The second terms in the differential Equation can be bounded by the norm of the weights
and the trace of the input covariance matria.
[[<((in)o(W11))]]
T
[[(o((Wam) @ WT (2
| T.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Proof. For the first inequality,
(W) >> [[l[(W))))))))
$ <[] (w
[W]]l TF
For the second inequality,
(W o W2 W20 I1 ([ W (1 T>
(1
[W2ll (I) [|
s [|2]l (IIW1)))))))
[
C Two-Layer Networks on Symmetric Datasets
C.1 Learning Dynamics: Early Phase
In the early phase of learning, the network output small compared to the target output because the initialization is small. Lemma [6] specifies how small the norm of the weights is.
Lemma 16. Denote the larger L2 norm of the weights in a two-layer network as u(f) max (O)l
The inilial weights are small, that is Winit = u(0) & 1. For two-layer lineur, ReLU, or leaky ReLU U networks trained with square loss from small initialization, u(t) is bounded by
u(t) & u(O)e(++))))/
(25)
for time e+Ir E
Winit
Proof. For two-layer linear, ReLU, leaky ReLU networks, the learning dynamics are given general Equation (2) Using the equality in Lemmna 14 and the inequality in Lemma 15 we can bound the dynamics of 2
u1 T d dt dt
12l2 = (a T W. W. - 2w, (wW())))))( W.
< I(a 1)B W, w (WW])((( w.l
< 2||l|||| ||| W2ll + W T
< 2su2 + 2u 4 T.E.
where s= ||S||. For a < 1, we have
d 2 < 2su² + 2n T 2 11s+TTEjp u.. dt
Via Lemma
13
Grönwall's Inequality, we obtain
2 < 2 e 2(+ D11/
u(t) < Wint (a+ Tr E)t/T
This holds for
1
t
8 Tr Σ
Winit
21

---

Published in Transactions on Machine Learning Research (04/2025)
Since the weights small in the early phase, we approximate the early phase dynamics with only the that is
first terms in Equation 2
W1 ~ (Wa
@ W2 ya ))
W B
(26)
2
W2 R (wo
a + 1 B B...
(27)
2
where the equalities hold under Condition proven by Lemma 14 For Lemma we solve the approximate early phase dynamics and show that the errors introduced by the approximation are bounded. We presented
Lemma 6 in the main text and now prove it.
Proof of Lemma 0 We first consider the approximate learning dynamics:
W
a + 2
W BT,
WW2 a+
W...
3
W
(28)
This is a linear dynamical system with an analytical solution available. We re-write it as:
a a
0 18
B
W
W
MW, where M
W
(29)
dt
2
0
W2
Since matrix M only has two nonzero cigenvalues is, the solution to Equation
29 is
1 a+1 2F
W
(B W, (0) + W2(0))
2
[
(30)
B W (0) W2(0)) + [I (I- 3,3 W. (0)
2r
-1
Note that only the first term in Equation 30 is growing.
We then consider the exact learning dynamics given by Equation and prove its solution is close to Wtt). The dynamics of the difference between the exact and approximate dynamics are
d
a+1 W2 W2) W:)" TT + (d'(W @ W₂ W20(W1 W2 W:)
W Wi)
(31a)
dt
d
+ BT (Wi -wi) (WW
(31b)
dt
We we-write Equation
31 as
d ow a-1 MdW +
(32)
dt
2
The norm of the two components of can be bounded via Lemma 15
(Wiz)) WT (W | < Tr
|| W (W(W )) || < n E.
We can then substitute in Equation
25 and obtain
|e|| < 22 Tr < E/1/ E/1/r Tr E.
22

---

Published in Transactions on Machine Learning Research (04/2025)
We now bound the norm of W W
W-
W
a +
M W W) edt
2
[M|
W w/ + ||||cl
/ (V28 W w| + Vunge T dt
22 TE
(1 - -1) + 228
S [/' W W dl.
3(s TrE)
Via Lemma 13
Grönwall's Inequality, obtain
w w $
V2TH Eng
Me+TTYE)/ - -1 | /
-1) 2se dt'
3(s TTE
V2s (00(0tT)))+ -1) e
V2 Eng
3(++T) E//T +
TT
3(s + 2) + 22
When
we have
W W- w/|
< Gnū for some constant C₁.
34 ug
We now ready to bound the difference between the exact solution and exponential function along direction
B
W
ri
W
W)
+
W
e
a+1 st 2r
B
P7
W
W
+
1l st
3 -1 (3
W, W, (0) W2(0) (I- (T-33) W. (0)
2
The first term arises from approximation of dropping the cubic terms in the dynamics. Tts norm is bounded by Crwin. The second term arises from initialization, which is O(wnit). Via triangle inequality, the norm of the total error is of order O(wise)
w
a+1
r1
Ciwiit + C2Winit Cwiiit
Lemma ① iimples two messages. Firstly, the (leaky) ReLU network has the same time-course solution as its lincar counterpart except a scale factor determined by a, which is consistent with Theorem [ Secondly, the (leaky) ReLU and linear networks form rank-one weights with small errors in the early phase. We exploit the rank-one weights to reduce the learning dynamics to Equation 12
C.2 Learning Dynamics: Late Phase
Proof of Theorem (square loss). Theorem g relies on Condition 3 and Assumption 7 and arrives at three statements: implementing the same function as in Equation 13) having the same weights as in Equa-
tion 14 and retaining rank-one weights as Assumption We prove them one by one.
Part We first prove that the (leaky) ReLU network and the linear network implement the same linear function except scaling when their weights satisfy Assumption Denote Wa [W, W where WP+ are
23

---

Published in Transactions on Machine Learning Research (04/2025)
the positive elements in W2 and W2- are the negative elements in W2. For (leaky) ReLU network with
rank-one weights sstiing Assumption 7 we have
f(: W) = W21(W = W20 (W₂ (W rTTr x).
Notate the positive and negative half-space
s+ { E R
I x 0}
s E <0}.
(33)
For St, have
W T aW2
fæ; W) = r xW2o = 1
= x (1 11 + a:l 11))
T
For E S we have
a + 2
f(x: w) = -- xa = = W
W2]
=
x (a W 21 + 1 11²)
W
Under Assumption
we have I = W ||. Hence, the (leaky) ReLU network implements
Vr, f(;; W) W2o(Wix) = 2 r x W2ll
(34)
Under Assumption the lincar network implements
elin x: W)
W.W z-= =- -
(35)
Comparing Equations (34p and 135 we find that when the weights satisfy Assumption 7 the (leaky) ReLU network implements the same function as the linear network except a scale factor
W20 (1a) = a+1 2
W12.
(36)
Part 2: We then look into the learning dynamics to prove that the weights in the (leaky) ReLU and the linear network are the same except scaling. Substituting Equation (36 into the dynamics, we get
WW a W 2
TT (o'(Wiz) O W. (Wi(W)))
a+
a+1 (o((Wz)) @ Wº W2W1axT))
W T
(37a)
2
W a+l 1
B W.
-W2 (
a 1
B W. a+l WaW, (oo (Wix)
(37b)
We compute the second terms in the dynamics under Condition 3 and Assumption (d'(Wiz) @ W. We Waxx
(o'(Wiz) C WT W, Wa
181
(o'(Wiz) & W. W, W/ T)
1
1
a1
W W2W, Wi (rx
3 W W.W, (wx T
al
8+
2
W
1
aW
T
W2W Σ
W.WWY
aW
W
a +
W. W.W,E,
(38)
24

---

Published in Transactions on Machine Learning Research (04/2025)
and
( 1 ([wx))
(zo(Wii) )/-- + (zo(()x)))
(x /81 r ?W aW,] +
x (x>)) ) r [a ]
Er W
+ EWT
(39)
Substituting Equations 38
and
39
into Equation
37
we reduce the dynamics
a+
+ 1
W1
W2 T
W W&WEE,
2
2
a a+1
a + 1
W
3
W
W WWW.
2
2
This is the same expression as Equation
12 in the main text. If we scale the weights
W
W1,
W =
2 W2.
(40)
the scaled (leaky) ReLU network dynamics W(I) is the same as that of a linear network given in Equation (3 except for a different time constant
2r W
2r W (B W
W2 (B - W.-WE
a
a +
Because Theorem defines the initial condition W (
W (0), the weights in the linear network and the scaled weights in the (leaky) ReLU network start from the same initialization, obey the same dynamics, and consequently stay the same Vt 2 0
W(1) lin (at11)
2
W +
W(I)
V
This proves Equation 14
Substituting Equation into Equation
36 proves Equation
f(x: (I)) 2 W(I))(() wiln
aa
Win
a +
x
2
= lin
x; Wii lin
a +
2
Part 3: We show that Assumption made at time t remains valid for 0, mcaning that weights which start with rank-one structure remain rank-one. With Assumption 7 time = 0, the dynamics of
the (leaky) ReLU network is described by Equation 12 This dynamics is the same scaled dynamics in
linear network and thus satisfies the balancing property of linear networks (Ji & Telgarsky 2018
2019
Du et al.
(W.TT - W. WT) = 0. de
(41)
Under Assumption 7 at time t = 0, this quantity is zero at time = 0 and will stay zero
0: 0: (t) - Wa( W2(t) (O) W2(0) Tw2(0) 0.
25

---

Published in Transactions on Machine Learning Research (04/2025)
Because rank(W; ) rank(Wi), the balancing property enforces that W1 and W2 have equal rank. Since W2 is vector, W1 must also have rank one. We can write a rank-one matrix as the outer-produch of two vectors W1 = or We can assume ||r|| = for convenience and get
W.W.
re =v W W
==
Because Assumption specifies Wi WT. then W To summarize, Assumption at time 0 reduces the learning dynamies of the ReLU network to be similar to that of a linear network. The reduced dynamics satisfies the balancing properly which enforces that the weights remain rank-one, thus satisfying Assumption for all tltz 0.
Proof of Theorem (logistic loss). We prove Theorem 8 for logistic loss LLG = (In(1 + v-)))
Part Same as the square loss case because proving Equation
36 does not involve the loss function.
Part 2: The gradient flow dynamics of two-layer linear network trained with logistic loss are
Wl W
wiln W
(42a)
Wh Wing
W li W w lin
T.
W
(42b)
W W
The gradient flow dynamics two-layer (leaky) ReLU network trained with logistic loss are
W (
(o
W W20 (W1z)x²
(43a)
e ( +1
(11))
7 W2 = W2
(43b)
eVW2o +
Substituting Equation 366
into Equation
43
we gel
" W = a
(W
O W W2 Wiax"
(44a)
2
3 W W +1
a +
wo
W (
W2
W2W1
(44b)
2
W.W
Under Condition and Assumption Equation 44 can be simplified
0 (W + WT W.W,as cæ:
W +1
a' dWW))) W. W₂ W 2000
d d'
(W
2 W W, W ax
+
W
81
2
c
W +1
8
aW,
C.
W
EC
W
W2W1
2 W
W W. 2
8-
2
aW,
wW, W
8-
a +
W2
W
2
0+1
W W a
+
and
xo W1x)
wW, W
+1
xo (W1
20 (Wxz
WWW Wi
+1 8+
WiW +
8-
TT
(aW
W₂
r
aW,
2
0:1
W W in
2
W, W
81
6
3
( +
Wi
y., Wi
2
26

---

Published in Transactions on Machine Learning Research (04/2025)
Thus, the reduced dynamics of the two-layer (leaky) ReLU network are
a + 1
T
W2
W2W,
2
w
W
a + 1
TEC
w2
W2W1
W1
2
W.W
If scale the weights Equation
40), the (leaky) ReLU network dynamics the same that of linear except for different time constant
network given in Equation 42
2r
W.
W W.W,
TEC
a + 1
2r
T.
T
a+ W₂
W
W
e
Through the same reasoning as the square loss casc, Equations
13
and
14 are proven.
Part Same as the square loss case.
C.3 Global Minimum
Proof of Corollary I0 The converged solution w" is direct consequence of the equivalence we showed in Theorem & and prior results of linear networks (Saxe et al.] 2014 Soudry et al., 2018).
We now show that for symmetric datasets satisfying Condition 3 the global minimum solution of two-layer bias-free (leaky) ReLU network trained with square loss is linear.
Based on Theorem we can write a two-layer bias-free (leaky) ReLU network as a linear function plus an
even function (xx x T +=* fe(z)) where fe()) denotes an even function. For datasets satisfying Condition the square loss is
1
C
2 (y x w" fc(x))
p(n)
2 ((v x Tw=))² "²²)² 2(yy Aa)S((w)) + Sa(xz)²)
1
pla))
1 (Iv - x uu))² p(x)
+ 2 (fa(z)))) p(a)
2
The square loss attains its minimum when both (w (w--xTuwe xT --====")))) )²)
and (f)))) p(x) are minimized. The former is minimized when w" = -1 The latter is minimized when fe(x) = 0. Hence, for symmetric
p(x)
datasets satisfying Condition 3 the two-layer bias-free (leaky) ReLU network achieves globally minimum
square loss with the linear, ordinary least squares solution f(x) x "w" =x 1-13..
C.4 Effect of Regularization
Theorem S still holds when L2 regularization is applied. Specifically, Theorem g holds if L2 regularization is added with hyperparameter X, = = X, i.e., the loss is freg L + a+1 \| l With similar calculations, we find that the regularized dynamics of the two-layer bias-free (leaky) ReLU network is
(a+1) 2 W₂ W,W,E 2
a W. TT
a+1 AW
7 W
(45a)
2
W a+1 1
a (a+1)
a a+1
B W
W.W, ΣW,
XW.
(45b)
2
2
27

---

Published in Transactions on Machine Learning Research (04/2025)
If we scale the weights as Equation [10), the regularized (leaky) ReLU network dynamics is again the same as that of a regularized linear network except for different. time constant.
2T W W2 (8 W W 1) AW
2T W2 (B - W AW
a a+1
a+1
We validate this with simulations in Figure 7a As in the unregularized case, we find that the loss curves with different leaky ReLU slopes collapse to one curve after rescaling time and the differences between weight matrices are small.
We note concurrent work (Wang et al. 2025 that considers two-layer bias-free ReLU networks with L2 regularization.
C.5 Effect of Learning Rate
We empirically find that with moderately large learning rate, the behaviors of two-layer bias-free (leaky) ReLU networks are consistent with Theorem 8 For sinulations in Figure 7b we use learning rate of 0.6, which is 150 times larger than the learning rate used in Figure 3 0.004. Due to the larger learning rate, the loss curves in Figure 7b is less smooth than those Figures 7a. and 7c Nonctheless, the loss curves with
different leaky ReLU slopes collapse to one curve after rescaling time and the differences between weight matrices are small.
If the learning rate is further increased, oscillations in the loss curves occur, suggesting unstable training. In such cases, the equivalence described in Theorem no longer holds. However, the learning rate is typically chosen to avoid such oscillations in training.
C.6 Effect of Initialization
We empirically find that under large initialization, two-layer bias-free (leaky) ReLU networks still have similar learning dynamics as its linear counterpart. As in the small initialization case, the loss curves in Figure /egure/c with different leaky ReLU slopes collapse to one curve after rescaling time. The differences between weight matrices are larger than those in the case of small initialization but are still less than 3%. With large initialization and a moderately large learning rate, the behavior remains consistent, as shown in Figure 7
This is related the limited expressivity of bias-free ReLU networks. Within the expressivity of two-layer bias-free ReLU networks, the linear solution is the global minimum for symmetric datasets. The two-layer bias-frec ReLU network learns the linear solution starting from either small or large initialization.
28

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
1.0
Loss Error
1%
1.0
0.8
Loss
0.6
0.5
0.5
0.5%
0.4
0.2
0.0
0.0
7500
15000
7500
15000
.21
(a) Loss error curves with L2 regularization with hyperparameter Aa 0.2(c + 1).
1.0
Loss
11
a= 1.0
0.6
- 0.4
0.5-
-0.5%
0.0
0.0
36
70
(b) Loss and with moderately large learning rate, 0.6.
1.0
1.0
Loss Error
10%
a'
0.8
Loss
Error
0.5
0.5
5%
a
a 0.4
0.0
0.0
a 0.0
500
1000
500
t
.+I
(c) Loss and error curves with large initialization, Wini 0.5.
1.0
1,0
Loss
10%
Error
0 0.8
Loss
Loss
Error
0.5
5%
0.6
0.2
0.0
0.0
0.0
5
10
10
a+1
(d) Loss and error curves with large initialization, Winit = 0.5, and a moderately large learning rate, 0.6.
Figure 7: Two-layer bias-free (leaky) ReLU networks evolve like linear network even when some of the assumptions in Theorem Saammmaareee are lifted. The setup is the same Figure exxcept for the condition(s) specified in each individual subcaption. In (b,c,d), the time rescaling is implemented by inversely rescaling the learning rate. This avoids the inaccuracy induced by rounding the rescaled time to an integer number of epoch, which becomes non-negligible the case of a small total number of epochs. In (c,d), with large initialization, the errors between weight matrices are larger than those in the case of small initialization but are still less than 3%.
29

---

(a) Mean square error loss trajectory. (b) Logistic Published in Transactions on Machine Learning Research (04/2025)
21
0.8
R
RaLU
Lin +1
2
Un -1
0
Lin
0.2
0.2
-1
-2
-2
0.0
0.0
10000
20000
1000
2000
(a) Orthogonal data
(b) Loss
(c) XOR data
(d) Loss
Figure 8: The same as Figure but with logistic loss.
1.0
ReLU
RaLU
Ln-1
Lin-1
0.4
0.0
0.0
10000
20000
10000
20000
(a) MSE loss
(b) Logistic loss
Figure 9: Two-layer bias-free ReLU networks evolve like two linear networks when trained on a dataset with pairwise orthogonal input, that is Vμ # v, F, = 0. The dataset has 20 samples, with samples labeled +1 and 10 samples labeled -1. The input points with +1 label have a larger L2 norm than those with -1 label. The black curve is the loss of two-layer bias-free ReLU network trained on all 20 samples. The blue dashed curve is the loss of two-layer linear network trained the ten data points with +1 label; and the red dashed curve is that trained the ten data points with -1 label. The loss curve of the ReLU network overlaps with those of the two linear networks. loss trajectory.
D Two-Layer Networks Learning Dynamics on Orthogonal Datasets
Assumption 17. A two-layer fres ReLU network is trained from small initiatii on dataset with orthonormal input, that is -_ = 0 if μ # v, and x Tt = 1. We assume that the weights at time to during training have the following form:
W1
W WB
A UBTBSp
W₂ [
W
[MAT]
-upriT]
(46)
where u > 0, ||TA| || = |ra] || = 1 and all entries in TA, Tn are non-negative. The vectors BA, BB are defined as
Epesa DK
AA
SA = {ulya > 0}.
(47a)
Eyesa B.
Epesn D1
BB
SB {uly < 0).
(47b)
Expes B1
Here the set SA denotes the indices of the data points with positive target output, and SB denotes those with negative target output.
Proposition 18. Under Assumption 17 for all 2 to, the weights of the two-layer bias-free ReL U network will maintain the form in Equation 46 and its learning dynamics is described by
W w." (BT W WFEC),
W (BT w&w, W
(48)
30

---

Published in Transactions on Machine Learning Research (04/2025)
where C represents either A or B, and
Bc P
Σ
1 Ec = P
E x
(49)
peSc
PESo
The dynamics of W W is the same as that of a twoo-aaer linear network trained with data points in Sc, as given by Equation
Proof. Because the weights satisfy Equation [6 and the data points are orthonormal, we have
Au E SA.
(W xx) a (W B xx) W wu o (W," x,,) O.
W Fu a (W x.,) 0;
Au SB, For data points in the datasel, the network computes
Ap SA. f(zz: W)
1 a (W + W (W xx) W.W W/ Zy Vp E SB [ax W) = W20 Wi xpp) = w (WAx,,) w o (WFx,) W.
The learning dynamics of WA, W can be calculated as
W. (ya- (2 TWA Σ o' (Wi'x,))
P
Σ W
(1μ- -W WT' x
P
HESA
WAT T
(B) W. WyAWAEE)
(Wi^ T
TW 2 =
(Up
μ=1
Σ (Up
(W,ix,)) T
MESA
-BI
W WAWAA WAT.
This proves Equation 48
We now prove that the weights will maintain the form in Equation 46 for 2 to. We first calculate term that we will need:
Epesa M
Epess a
Σ =
Epesa yn
P T
EnweSS yut w 10
P EpESA B1
Euesa JT
Epesa Uu P
(50)
By substituting Equation
146
into Equation
(88
and using Equation 50 we obtain
d uA UAFA (oX
1 -
TWA
MATIuArASTEE UATABT [|BAal
(51a)
dt
W
d uAr = - UABARI AA (118All
p _112 rT.
1
(51b)
dt
31

---

Published in Transactions on Machine Learning Research (04/2025)
Equating the coefficients in Equation 51 yields one-dimensional ordinary differential equation about MA!
TuA UA (1sAall
(52)
Hence, for the WA, W blocks in Equation (46), only the scalar variable vn evolves while the vectors YA, A stay unchanged.
With the same approach, we can calculate the learning dynamics of WP. 2
W W ( - W w. W B (B] - BBB w
Substituting in Equation 46
we obtain
7 W = r dt
"Br Bg - ugTB (Bg + "Brg "Br EB) = ugrg [|8B||
p "B
(53a)
7 B = - di
d
"BTB = (BB + u uBFB EB) uBBrrB = uB (-1[sel +
p rB.
(53b)
Equating the coefficients in Equation 53
yields one-dimensional ordinary differential equation about uB:
1 --"p "B
TúB WB |18x||
(54)
Hence, for the W W blocks Equation (46). only the scalar variable up evolves while the vectors TB. BB stay unchangod. Consoquently, if the weights satisfy Equation (46) at time to, Equation (46) will hold for all l to-
E Deep ReLU Network Learning Dynamics
Proof of Proposition 12 According to Equation 18 all weight elements in the intermediate layers (WL-1," W3. W₂) are non-negative numbers. According to the definition of the ReLU activation func- tion, o (Wiw) yields a vector with non-negative numbers. Thus (20(Wiz) yields a vector with non-negative numbers and we have (W2z)(W1x) = (2,(W Similarly, all subsequent ReLU activation functions can be ignored? 2 With weights satisfying Equation 18 a deep bias-free ReLU network implements
f(x; W) = WLo (Woo (Wa=)) = WT ... (21(W
We stick to the notation for the positive and negative half-space defined in Equation have
33
For x E St, we
wh-1
f(x; (l: = wW We
Tx= = (V2)L-2 r x
WT
T-1
r
x =
u
r x.
/2
For ET,, we have
= uW W2
0
1-
WL
0
T
u
r
x.
L22
LL
/
I. TTx.
Hence, the deep free ReLU network implements a linear function f(x; W) deep linear network with such weights implement
T
Notice that
/
uL
Wz -. W = uW W2 P -
-
W WITL-IT (V2)
(V2)L-2 TT. x
Thhe activation functions can be ignored when caleulating the network output but cannot be ignored when calculating the gradients. This is because for ReLU function (=) max(2,0) linear function ø(z) 2, the function values are equal at (0)) = ((0) but the derivatives are not equal at a' d0()) # (0).
32

---

Published in Transactions on Machine Learning Research (04/2025)
W
W
W4
0.0
-0.1
W3
W
W
0.1
W4
0.0
0.1
(a) Weights in a 4-layer linear (top) and ReLU (bottom) network as Equations
17
and
18
W4
W
W
W
0.1
W3
0.0
-0.1
W4
W
W
W
0.1
Ws
0.0
0.1
(b) Weights in 5-layer linear (top) and ReLU (bottom) network as Equations
17 and
18
Figure 10: Same Figure but with deeper networks.
33

---

Published in Transactions on Machine Learning Research (04/2025)
Equation 19
is thus proven.
We now prove that under Equation 18 on the weights at time t = to, we have that V 2 for Equation 18
remains valid. We assume that o0(0) O. We substitute the low-rank weights defined in Equation (18) into
the learning dynamics of deep bias-free ReLU networks and make simplifications. For the first layer,
W T (o'(W
WE (y f(x)))= )
1
(V2)
d'
rr
y
V2
T
T
r+
L
u 1ー1
{(v
(V2)2
y
v2
r
x
T
(v2)²
-
v2
x
x
T
gt
s-
x6-1
(v2)² ri
gT
r I
(55)
v2
For intermediate layers 1 <<< 1 << L,
W (o'(h))
W 411
W., (v f(z)))((h))) )T)
L
21
T
o|
y
r
8
x
r
r--1
22
S
u
7
+
y
T
x
r
0
ri
V2
2
8-
L-1
L
u
3
u
r
T
Σ
r
2
0
0
2
L
+
u
T9T
r
Σ
r
22
0
ri
T--1
v2
V2r+T
B
u
r
r.
(56)
(v2)²
0
V2r ri
22
For the last layer,
7 W = ((y
f(x))o (hI-1)) 1)
wt-1
(V2)L
y -
u
2
r
x
x
r
Th-1
+
0]
S+
4--1
L
11
T
7]
r
r
0
ri
(V2)z
2
8-
wt-1
13
u
r
T
Σ
PT 1-1
(57)
(v2)z
2
Equations 555 to 57 can be rewritten as Equation (20 if we substitute the weights back in. The dynamics of the deep ReLU network as in Equation (20 is the same as a deep linear network as in Equation 7 for constant coefficients.
except
34

---

Published in Transactions on Machine Learning Research (04/2025)
We now prove that the low-rank weights remain low-rank once formed. We substitute the low-rank weights defined in Equation 18 into the left-hand side of Equations 55
to
57 and get
4h-1
dt
urrr
(v2)r
m
B9
22
11
E
Σ
d dt
u 22r vi
0
4b-1
V2r ri
1
BT
BT
u1
I E
v 2r,
r 1-1
T
(v2)L
V2r r--1
T
v2
r
r.
w
L
dt
ur T
L-1
(V2)*
BT
/2
u
r
rT
LT--1
We cancel out the nonzero common terms on both sides and reduce the dynamics to two differential equations. The first one about the norm of layer u. The second one is about the rank-one direction in the first layer ur.
u6-1
L
dt (V2)L
11
TT
12
u
T
Σ
r,
ur 46-1
u L
gT
Σ
(v2)L
After the weights have formed the low-rank structure specified in Equation
18 the norm of cach layer
and the rank-one direction of the first layer evolve while Vı, r2,
Pt-1 stay fixed. Hence, under
Equation 18 on the weights at time t= to, we have that Vt 2 to, Equation 18 remains valid.
We complement Figure 5 in the main text with Figure 10 which shows the converged weights in deeper bias-free linear and ReLU networks. Figures and 10 are the empirical results that motivate us to make Conjecture
F Depth Separation
A clarification Section 3.2 is that deep bias-free ReLU networks are not more expressive than their two- layer counterparts if the input is scalar. For scalar input functions, the only positively homogeneous odd function the linear function. Neither two-layer nor deep bias-frec RcLU networks can express nonlincar odd functions with scalar input.
Another relevant fact is that there is also depth separation between two-layer and deep ReLU networks with bias. One example the pyramid function, a(1 - -| which was studied in Ongie al. 2020 Example Proposition 2).
and Nacson et al. 2023
35

---

Published in Transactions on Machine Learning Research (04/2025)
G
Additional Figure
0.6
0.4
200
400
600
800
1000
(a)
Loss
(b) Dataset and solution
Figure 11: Two-layer bias-free ReLU network trained on a lincarly separable binary classification task with label flipping noise. Since the dataset satisfies symmetric Condition g the network follows linear network dynamics and converges to linear decision boundary, which is presumably robust solution here as it avoids overfitting the two noisy labels. (a) Loss curve. Logistic loss is used here.
(b) The dataset is plotted with empty circles and short lines, representing data points with +1 labels and -1 labels respectively. The network output at the end training is plotted in color.
H Implementation Details
All networks are initialized with small random weights. Specifically, the initial weights in the I-th layer are sampled i.i.d. from normal distribution (00,, winst /Ni) where N₁ is the number of weight parameters in the (-th layer. The initialization scale Wini is specified below.
Figure The networks have width 100. The initialization scale Winit = 10-2, The learning rate 0.2. The two-layer networks are trained 10000 epochs. The three-layer networks are trained 80000 epochs. The datasot is plotted in the figure. The size of the datasots 120.
Figure 3 The networks have width 500. The initialization scale is Winit 10- 8, The learning rate is 0.004. The imput is 20-dimensional, x 20 We sample 1000 ii.d. vectors 2n (00 I) and include both æn and =2, in the dataset, resulting in 2000 data points. The output is generated as w x + sin (4w x)) where elements of w are randomly sampled from a uniform distribution 41[0.5, 0.5). This dataset satisfy Condition since the empirical input distribution is even and the output is generated by an odd function.
Figures and We use the same hyperparameters Boursier al. (2022). The network width 60. The initialization scale Winit 10-6 The learning rate is 0.001 for square loss and 0.004 for logistic loss. The orthogonal input dataset contains two data points, i.e., [-0.5, 1), [2, 1]. The XOR input dataset contains four data points, ie., [0, 1], [2,
0). [0, -3), -3],1-4, [], OJ.
Figure The networks have width 100. The initialization scale Winit = 10-2 The learning rate is 0.1. The networks are trained 20000 epochs. The dataset is generated in the same way as Figure 3 except that the output is generated as w æ..
Figure The network width is 100. The initialization scale Wint 10-3 The learning rate 0.025. The dataset contains six data points: [1, 1], |-,, -1], -1]. |-1 1), [-11-110 1,0 0j, [1,5].
36


Reviews:
[{"review_id": "qmczhKF2Og", "reviewer": "Reviewer_gZN1", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: The study of the training dynamics of two-layer and deep ReLU networks beyond the lazy regime is challenging, with many aspects that remain unresolved. This paper provides a clean analysis of the optimization dynamics under particular assumptions in a bias-free setting, and is able to highlight the importance of the bias term to obtain highly expressive prediction functions.\n\n**Weaknesses**:\n* A key concern for me is Assumption 5, a core assumption in the theoretical results, which seems unrealistic. Specifically, it assumes a rank-1 initialization which is not used in practice. The authors argue that this is not an issue since the rich training regime uses small initialization, but then this assumption can be rewritten to better match the rich training regime and practical initializations. This is further discussed in the section below.\n\n* There are not sufficient experiments in the paper to support the conjecture of Equation (15b). To my understanding the only empirical evidence is Figure 4 which considers a 3-layer ReLU network. More experiments are needed to understand the effect of depth and data distribution on this observation. This is the reason I answered \"No\" to Claims and Evidence, and I would be happy to change to \"Yes\" if the authors could point at additional experiments and justifications, or if they limit this section to networks of depth 3.\n\n* The experiments in Section 4.2 seem to cover simple toy data distributions and two-layer ReLU networks. There is some theoretical justification of the observations in Appendix D. I think extending this theory to more than two samples and presenting precise theoretical results in the main text should be within reach, and would make this section much stronger.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the expressivity and training dynamics of two-layer and deep bias-free ReLU networks, and identifies conditions under which they are equivalent to linear networks. In particular, under the assumption that the input distribution is even and the label is an odd function of the input, the authors prove that when using a particular rank-one initialization for the first layer, the gradient flow dynamics remains identical to a time-rescaled dynamics for linear networks. The authors also demonstrate that on orthogonal datasets, the dynamics of two-layer bias-free ReLU networks can be identified with multiple linear networks. Finally, the authors observe certain sutrcutres in deep bias-free ReLU networks that are similar to those of deep linear networks, leading to equivalent training dynamics in specific settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* I believe to achieve the rich training regime, one can initialize the coordinates of $W_2$ to be of order $1/H$, while the rows of $W_1$ can be unit-norm (therefore not necessarily) and are typically initialized isotropically in the space, see e.g. Table 1 of Yang and Hu, 2020. This would not correspond to ``small initialization'' for $W_1$. Can the authors show that with this type of initialization instead of Assumption 5, a statement similar to Theorem 7 holds, perhaps as $H \\to \\infty$? \n\n* If the above scaling does not work, can the authors provide another initialization scaling which achieves the rich regime, is isotropic in the first layer, and for which a variant of Theorem 7 can be proved?\n\nG. Yang and E. J. Hu. \"Feature Learning in Infinite-Width Neural Networks.\" ICML 2021.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LdecouRNav", "reviewer": "Reviewer_xhwY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper should be of interest to the TMLR audience, as it collects a number of extensions to previous linearization results. The manuscript is relatively clearly written, but (as noted below under Requested Changes) there are places where I think its organization could be improved. The analysis is in general straightforward and the results appear mathematically correct, though I have not exhaustively checked the proofs.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript studies conditions under which bias-free ReLU neural networks display identical gradient flow training dynamics to linear networks. Its main result (Theorem 7) shows an equivalence for networks with a single hidden layer under symmetry assumptions on the data and an alignment assumption on the weights. The remainder of the paper consideres extensions to deeper networks and other datasets through a combination of partial analytical results and numerics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please note that I've ordered these requested changes by appearance in the text, not importance. \n\n1. Though the proof of Theorem 1 is immediate, I think it would be clearer to first note that the leaky ReLU function admits a decomposition as $\\max(x,\\alpha x) = \\frac{1+\\alpha}{2} x + \\frac{1-\\alpha}{2} |x|$ and then that this leads to a decomposition of the network as a whole. This decomposition is implied by the drawing below equation (8), but I think this schematic is less helpful than simply stating the decomposition of the activation function. \n\n2. I think the clarity of Section 4 could be improved by deferring less material to the appendices, and better referencing what is shown there. First, Remark 6 should be made more mathematically precise. I would suggest that the authors state Theorem 15 from Appendix C in the main text, and also give a precise statement of the conditions under which the assumption that \"the second-layer weights have equal L2 norm for their positive and negative elements [...] is justified by the width approaching infinity\". Next, the authors should state below Theorem 7 that its proof can be found in Appendix C.2, rather than only referencing that appendix in a parenthetical above. Also, is there a reason why Equations (10a-b) are stated separately from Theorem 7, rather than as part of the Theorem? \n\n3. It would be useful to state Lemma 11 in the main text, as this is fundamentally why the authors arrive at Condition 3. Indeed, the proof of Theorem 7 depends on Condition 3 only through Lemma 11 (up to replacement of the factors of 1/2 in the derivation of equation 36).\n\n4. Is Corollary 9 correct as written? As I recall, Soudry et al. (2018) show that the norm of the weight vector continues to grow through training, and the convergence to the max-margin solution is only in direction. Please correct me if I've misremembered. \n\n5. The desired take-away from Section 4.2 could be made more clear - is the idea here just to illustrate that with other datasets the equivalence with a single linear network breaks down but there can be more general equivalences? \n\n6. It could be nice to highlight the main empirical results of Section 5 (i.e. equation 15) as a conjecture. Then, you could state equations (16) and (17) as a result conditional on that conjecture. Again, it would be useful to give a more nuanced statement of when the balancing of positive and negative parts of the weights hold at large width, rather than simply asserting that this is true.", "broader_impact_concerns": "I do not see any broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "99WfLrOweH", "reviewer": "Reviewer_TNKe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this paper suits well the line of TMLR. The proofs are simple to follow and I can thus assess their soundness. Additionally, the contribution brings nuance to previous literature on ReLU networks by showing that under some conditions, the studied dynamics is no different from a linear network ones.\n\nMy main concerns are about the clarity of the paper. First, some theoretical statements are given without any math definition, making the claimed results unclear at times. For example, I do not understand what does the second part of Assumption 5 means \"W2 has equal L2 norms for its positive and negative elements\". Similarly, Theorem 1 is only written in words. As a consequence, it kind of hides that the statement is only an inclusion (i.e., the set of functions that can be represented by neural networks is included in ...).\nOn a similar note, I did not understand what the sentence \"To simplify the analysis, Assumption 5 assumes ...\" means. Does this mean that $W_1$ is rank one at initalization? If so, this seems like a very strong assumption.\nAs a last example of such lack of definition, it is mentioned several times that \"the target function is linear\". What does this mean? I understand that as\n$$y_i=x_i^{\\top}w^{\\star} +\\varepsilon_i,$$\nfor some random noise $\\varepsilon_i$. But here, it is not clear whether such a noise is included or not in the statement.\n\nMaybe related to this point, I had a hard time to understand Section 5. I did not get all the notations, but also, it is here hard to parse what properties are just \"empirical observations\" (for what kind of init, what setup?) and mathematical derivations from the observations/assumptions. This gives a \"speculative vibe\", from which I have difficulties to extract interesting insights.\nAlso in this section, I did not understand why \"which is true when the network width approaches infinity\" is actually true. Some additional explanations might be helpful.\n\n## Minor remarks\n\nAbout the literature related work:\n- I am unsure about the discussion of Soudry et al (2018) in Section 6. To me, convergence towards max-margin holds, whether or not we count any bias term. It's just that the definition of max-margin solution differs whether or not we count the bias norm in the norm penalization.\n- *The convergence rate of neural networks for learned functions of different frequencies* by Ronen et al (2019) already studied the functions that can be represented by no-bias ReLU networks. Your proof and result is much simpler and clearer, but I think it would still be great to mention this paper somewhere.\n- To complete the discussion on the perturbed symmetric dataset, the paper *Simplicity bias and optimization threshold in two-layer ReLU networks* by Boursier and Flammarion (2024) even seems to suggest that for finite values of $\\Delta y$, the plateau you mention might even be infinite, i.e., the optimization scheme is unable to leave this plateau even for an infinite time (while you here suggest it might leave it after a duration that grows larger as $\\Delta y\\to 0$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the (theoretical) implication of omitting bias terms in the parametrization of (two layer) ReLU networks. It provides a short proof of the fact it can only represent the sum of a linear function and an even function in the case of 2 layer, as well as providing a depth separation example, where more complex odd functions can be represented.\n\nIt is also shown that under a specific set of assumptions on both the dataset and initialization, the training dynamics of two layer ReLU networks is equivalent to the training dynamics of a linear network. The authors then complement these theoretical results with experiments on orthogonal and XOR datasets, as well as deeper architectures.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Following my concerns mentioned above, I wish the authors could provide the following changes to the current paper:\n\n- add rigorous, mathematical definitions in the paper when needed\n- clarify assumption 5: I am not sure yet of how restrictive/common it currently is\n- add a figure illustrating the function g(x) (equation 9)\n\nLastly for for Theorem 1, I believe a stronger theorem with an equality of sets could be possible. I think for example following the same kind of analytical tools as in *How do infinite width bounded norm networks look in function space?* by Savarese et al (2019) might lead to the reverse inclusion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Based on the reviews, the manuscript provides an interesting contribution to the understanding of neural network dynamics, particularly highlighting the conditions under which bias-free ReLU networks exhibit linear behaviour. All three reviewers recognised the technical soundness of the main theoretical results and relevance to ongoing research beyond the lazy training regime. 

The main concern raised by the reviewers was that the initialization assumption (Assumption 5) is restrictive and not standard in practice. The reviewers also made some constructive suggestions that were implemented by the authors, helping improving the original version of the manuscript.
Decision: Accept (poster)