Implicit Bias and Loss of Plasticity in Matrix Completion: Depth Promotes Low-Rankness
Authors: Baekrok Shin
Chulhee Yun
Venue: ICLR 2026 Poster (2026)
Keywords: Implicit Bias
Implicit Regularization
Loss of Plasticity
Matrix Completion
Depth
Low-Rank
Primary area: optimization
TL;DR: This reseacrh contribute to a theoretical understanding of the implicit bias of depth and loss of plasticity in matrix completion.

Abstract: We study matrix completion via deep matrix factorization (a.k.a. deep linear neural networks) as a simplified testbed to examine how network depth influences training dynamics. Despite the simplicity and importance of the problem, prior theory largely focuses on shallow (depth-2) models and does not fully explain the implicit low-rank bias observed in deeper networks. We identify coupled dynamics as a key mechanism behind this bias and show that it intensifies with increasing depth. Focusing on gradient flow under block-diagonal observations, we prove: (a) networks of depth $\geq 3$ exhibit coupling unless initialized diagonally, and (b) convergence to rank-1 occurs if and only if the dynamics is coupled—resolving an open question by Menon (2024) for a family of initializations. We also revisit the loss of plasticity phenomenon in matrix completion (Kleinman et al., 2024), where pre-training on few observations and resuming with more degrades performance. We show that deep models avoid plasticity loss due to their low-rank bias, whereas depth-2 networks pre-trained under decoupled dynamics fail to converge to low-rank, even when resumed training (with additional data) satisfies the coupling condition—shedding light on the mechanism behind this phenomenon.

Published as a conference paper at ICLR 2026
IMPLICIT BIAS AND Loss OF PLASTICITY IN MATRIX COMPLETION: DEPTH PROMOTES LOW-RANKNESS
Baekrok Shin, Chulhee Yun Kim Jaechul Graduate School of AI, KAIST (br. shin, n, chulhee, yun) @kai st. ac kr
ABSTRACT
We study matrix completion via deep matrix factorization (a.k.a. deep linear neural networks) as simplified testbed to examine how network depth influences training dynamics. Despite the simplicity and importance of the problem, prior theory largely focuses on shallow (depth-2) models and does not fully explain the implicit low-rank bias observed deeper networks. We identify coupled dynamics as a key mechanism behind this bias and show that it intensifies with increasing depth. Focusing on gradient flow under block-diagonal observations, we prove: (a) networks of depth 2 3 exhibit coupling unless initialized diagonally, and (b) convergence to rank-1 occurs if and only if the dynamics is coupled-reson open question by Menon (2024) for family of initializations. We also revisit the loss of plasticity phenomenon in matrix completion (Kleimman et al., 2024), where pre-training few observations and resuming with more degrades performance. We show that deep models avoid plasticity loss due to their low-rank bias, whereas depth-2 networks pre-trained under decoupled dynamics fail converge low- rank, even when resumed training (with additional data) satisfies the coupling condition-sheddine light on the mechanism behind this phenomenon.
INTRODUCTION
Overparameterized neural networks have the capacity to perfectly memorize the training data, even when they are given random labels (Zhang et al., 2017). Despite their large capacity, neural networks often generalize well to unseen data without any explicit regularization techniques, which challenges conventional statistical wisdom. Recent studies attribute this phenomenon to the implicit bias of neural networks, arguing that among the many possible global minima, first-order algorithms such as (stochastic) gradient descent favor solutions that generalize well (Neyshabur et al., 2014; 2017; Huh et al., 2021; Timor al., 2023; Frei et al., 2023; Kou et al., 2023; Galanti et al., 2024; Jacot, 2022).
Matrix completion, a task with practical applications areas like recommender systems and image restoration, provides key framework for investigating these implicit biases, particularly the tendency towards low-rank solutions. While matrix completion can be viewed as a special case of the broader matrix sensing framework (Jin et al., 2023; Soltanolkotabi et al., 2023; Ma & Fattahi, 2023; Stöger Soltanolkotabi, 2021; Li et al., 2018), which offers general tools for understanding recovery from limited data, specific challenges can emerge when applying these general theories directly. Notably, common theoretical assumptions prevalent in matrix sensing analyses, such as the Restricted Isometry Property (RIP) (Candes & Tao, 2005), often prove too stringent or may not adequately capture the nuances of many practical matrix completion tasks. For instance, even when completing the 2 x 2 matrix Mc (introduced Figure la), which can successfully converge to a low-rank solution, the RIP condition cannot be satisfied. Therefore, researchers have investigated implicit bias phenomena specifically within matrix completion, without assuming the RIP condition (Menon, 2024; Bai et al., 2024; Razin & Cohen, 2020; Ma & Fattahi, 2024; Kim & Chung, 2023).
The goal of the matrix completion task is to recover a low-rank ground truth matrix W using only a subset of its entries. A common strategy for matrix completion involves matrix factorization, which can also be viewed as linear neural networks. These networks reparameterize the target matrix X a product of factor matrices, X W,, W ... Wi, and optimize the factors {Wiheet by

---

Published as a conference paper at ICLR 2026
Disconnected Observation
Connected Observation
My
?
1.35
1:35
Effective Rank
1.30
1.30
L-3
1.25
L-4
1.20
3
Effective
1.20
L= 5
2.15
L= 4
1:15
5
1.19
1:30
1.00
1e-1
1c:3
1e:5
le-7
1e:1
1c:3
1e:5
1
Initial Scale of
Initial Scale af
(a) Bipartite graph of MD & Mc (b) Effective rank trained w/ MD
(c) Effective rank trained w/ Me
Figure I: (a) Examples of bipartite graphs corresponding to observation patterns of MD (discon- nected) and Mc (connected). (b-c) Training results showing effective rank (ef. Roy & Vetterli (2007)) for completing rank-1 matrices Mn and Mc, respectively. The rank-l ground truth matrices were generated via ue where u, u R²² with entries sampled i.i.d. from standard normal dis- tribution. We initialized each layer's 's entries by sampling from a Gaussian distribution with mean zero and standard deviation a, chosen ensure the initial scale of the product matrix WL.1 (0) is approximately invariant to depth L. Each result shows an average of 300 independent random trials.
minimizing the mean squared error over the observed entries using gradient descent. The observed entries constitute the training set, while the unobserved entries act as the test set.
The problem of predicting W* is underdetermined, as infinitely many completions are possible. Nevertheless, both theory and experiments indicate that training even a simple two-layer factorization (L = 2) with gradient descent, without explicit rank constraints, typically yields low-rank solution under reasonable assumptions (Razin & Cohen, 2020; Bai et al., 2024; Ma & Fattahi, 2024).
A recent work by Bai et al. (2024) formalizes this phenomenon using the concept of data connectivity. They demonstrate that if the observed entries form a connected bipartite graph (meaning any observed entry can be reached from any other via shared rows or columns), depth-2 factorization initialized at an infinitesimally small scale converges to a low-rank solution. Conversely, the network may converge to higher-rank matrix if the observations are disconnected (see Definition and Figure la).
However, the situation changes significantly for deeper (I. 2 3) networks, as empirically demonstrated in Figure 1. Consider the task of completing the 2 matrix
w' ?
MD
(I)
W22
where only the diagonal entries are observed. This observation pattern forms a disconnected graph as illustrated in Figure la. Consistent with the theory for disconnected graphs, L = 2 models fail to find a low-rank solution, empirically converging to rank-2 regardless of initialization scale. In contrast, deeper models (I 3) with small initialization tend to converge to a rank-1 solution, shown in Figure 1b. This specific example highlights that the implicit low-rank bias appears to be strengthened by depth, in a way that cannot be explained solely by the data connectivity framework developed for L = 2 models. Furthermore, considering connected cases as well, Figure Ic demonstrates that this strong low-rank bias is generally robust, tending to strengthen further as depth increases.
bias remains elusive, largely due to the complex, coupled dynamics during training. While Arora al. (2019) offer insights, their claim that the gap between two arbitrary singular values widens with depth is not fully formal. It stems largely from their analysis assuming stabilized singular vectors, which limits its scope. Indeed, Menon (2024) notes that even for a simple case like (1) with win wis 1, proving that gradient descent with deep factorization converges to a low-rank solution is still an open problem. Motivated by this gap in understanding, we theoretically analyze such settings, including the example (1).
However, theoretical understanding of this
Investigating the implicit low-rank bias in matrix completion can also shed light on the phenomenon of "loss of plasticity"" a challenge widely observed in general neural network training (Shin et al., 2024; Ash & Adams, 2020; Achille et al., 2018; Berariu et al., 2021). The term loss of plasticity describes the tendency of neural networks, particularly after initial training, to lose their adaptability
2

---

Published as a conference paper at ICLR 2026
to information, hindering their generalization capabilities. A recent work by Kleinman et al. (2024) empirically reports this phenomenon even in matrix completion. They observe that models trained with insufficient data often yield high-rank solutions. If these models then warm-start using augmented data, they frequently struggle to achieve low-rank solutions. To provide a theoretical explanation for why this loss of plasticity occurs, this paper elucidates the phenomenon.
To summarize, here are the main research questions that we address throughout the paper:
What is the fundamental difference between deep (L 2 3) and shallow (L 2) factorizations regurding their implicil low-rank bias, particularly for disconnected observations?
Can we theoretically establish that deeper models (i.e., with larger + 3) exhibit stronger implicit bias toward low-rank solutions?
What is the undertying cause of the loss of plasticity phenomenon, and how does depth interplay with it?
In Section 3.1, we begin by examining the depth-2 case to elucidate the key mechanism of connectivity. We find that coupled training dynamics induces low-rank bias, phenomenon generalizable to deeper networks. Section 3.2 further investigates this for all 2 2 using block-diagonal observation case. Our analysis reveals that, for deep models, this bias distinctively promotes low-rank solutions compared to 2 models, strengthening with depth. Finally, Section explores the loss of plasticity phenomenon in matrix completion. We observe that deep models typically avoid this phenomenon due to their low-rank bias. In contrast, we empirically observe and prove that depth-2 networks pre-trained with limited observations (yielding decoupled dynamics) and subsequently trained with augmented observations (yielding coupled dynamics) fail to find low-rank solution. Please refer Appendix A for further discussion of related work.
PROBLEM SETTING
We consider the problem of estimating ground truth matrix W E [Rdxd based on observations of its entries (1.5) where Q [d] x [d] is the of observed indices. We model the estimate as
linear network WT.1 W, Wr.1 W,, where Wi E Rdxxdr= with dg d, = d. We denote the (it entry the matrix L:1 wiy- The factor matrices (Wi) are trained by minimizing an objective function é defined as the mean squared error l over the observed entries in Q:
@W,..... 22 A ((W L: 92)
Σ (i.)E
(wis - -w
We study the overparameterized regime where the intermediate dimensions satisfy di for all I E [L - imposing no explicit rank constraints on the product model WT.1- Consistent with prior works, our analysis focuses gradient flow dynamies (gradient descent with infinitesimal step size) for given objective function Φi. The dynamics for each layer (() evolve according to:
wi(()) & d di
Wi(t)= = ow, (W) W2(t)..... (1),,, LE [L]. t 0.
(3)
For depth-2 networks (L = 2), the product of factor matrices E Ddxd, (representing Wa) and B RR. xd (representing WT), we denote WA.B A AB. We define the stable rank of matrix W as srank( () A |
W W.2. We also denote by Ia the d x d identity matrix and by Ja the d x
all-ones matrix.
Bai et al. (2024) introduce the concept of data connectivity for an incomplete matrix M. Connectivity characterized by its set of observed indices Ω [d] x [d] and the corresponding observation matrix (i,j) E Ω, and O otherwise). The formal definition is as follows:
P (where Psp
Definition 1 1 (Connectivity from Bai et al. (2024)), An incomplete matrix M is connected if the bi-
partite graph Gm, constructed from its observation matrix P using the adjacency matrix 0 PT
0
connected after removing isolated vertices. Otherwise, M disconnected.
3

---

Published as a conference paper at ICLR 2026
3 IMPLICIT BIAS OF DEPTH INDUCED BY COUPLED TRAINING DYNAMICS
In this section, we extend the connectivity argument of Bai et al. (2024) to general depth factorizations. We first demonstrate how the coupling of training dynamics serves as the key mechanism explaining data connectivity's 's role in depth-2 models, through the completion of two previously introduced 2 x 2 matrices, MD and Mc, as illustrative examples. Building on the insights derived from these 2epth-2 model analyses, we hypothesize that deep networks exhibit an intrinsic low-rank bias because they maintain a high degree of coupled training dynamics, irrespective of observation patterns. This hypothesis is further corroborated by the block-diagonal observation results presented in Section 3.2.
3.1 WARM-UP: COUPLED DYNAMICS vs. DECOUPLED DYNAMICS IN DEPTH-2 NETWORKS
We focus on the simple 2 x matrix completion of MD and Mc. using depth-2 models W (t, A(t (t).. For brevity, let a, (t) Id be the transpose of the i-th row of At), and let (t) E Rd be the j-th column of B (t). Our aim is to see how training dynamics affect the alignment of the rows of (l) or the columns of (1), as such alignment leads to a rank-1 product matrix WA,((I)
Decoupled Dynamics. In the MD case (disconnected observations Wi,, Wy2), the gradient flow using the objective defined in (2), results in independent dynamics for the pairs (a1, b1) and (az, b2):
(t) (wh (t) ((())b;;l) (t)) bt()), 6,(t) = (w], - (t) T (t)) (t()) for 1,
Note that while the dynamics couple a (t) with b (t) and (t))) with (t) within each pair, the two pairs (a1, b,) and (a2, b2) are decoupled from each other. This decoupling means the overall system's 's dynamics separate into two independent systems. Consequently, there no compelling reason to align vectors from different pairs, typically leading to high-rank solutions with generic initializations (Figure 1b). Indeed, we can obtain closed- form solutions solely dependent on initialization (see Proposition 4) For instance, with (() = (0) aJ2, we have WA, (. (00) diag(wTi: w22).
Coupled Dynamics. In contrast, for the Mc case (connected observations w w²1). ). the gradient flow on the objective (2) yields coupled dynamics that do not decompose into independent pairs:
dj (t) = (wi1 - a1 (t) T by (t)) b1(t), (z(()) = (wi1 - az()) To (t)) (t),
(t) (wi1 a (t) b, (t) a (1) + (wh) at))) bi (t) az()).
An important observation from (4) is that (0) 0 ensures rank-1 WA,B due to persistent alignment of a (f), a2 (4) and 6, (t). Although non-zero initialization leads to more complex behavior arising from coupled training dynamics, the following theorem demonstrates that sufficiently small initial norms in (0) also result in the alignment of (1) and az(() with (1).
Theorem 3.1. For the product model W (t, = ( E R2x2 we consider the gradient flow dynamics (4), where the observations are wi (# 0) and w (+ 0). We assume convergence to the zero-loss solution (i.e., w11(0c)= = win 221 (oc) = w(r). Defining u bifx)) and the orthogonal component a (00) = a;(o))- (oo) T uu )us,, we have:
[ob
|(0)))
(IIb + 4w 4w + [Ib,(0)13
|a. (oo) (
for i = 1,2.
2w
The theorem shows that small initial norms for A()) lead to the alignment of a (co) and a2 (oo) with bi (oo), implying a near rank- I product matrix VAA,(Cc) This suggests that for depth-2 networks, coupled training dynamics (resulting from connected observations) facilitate the emergence of low- rank solutions under such small initialization, in contrast to the decoupled dynamics of disconnected observations, where no such bias exists regardless of initialization scale. This connection between observation connectivity and the coupling of training dynamics in depth-2 models motivates our investigation into how coupled dynamics manifest and induce low-rank bias in deeper networks, irrespective of connectivity patterns, as explored in the subsequent sections.
Remark. Analyzing these dynamics is challenging because the time evolutions of a1, a2 and b mutually dependent. We note that Theorem 3.1 is not direct corollary of Theorem 3 in Bai et al. (2024). We explicitly characterize the degree of misalignment as a function of the initialization scale, unlike their assumption of infinitesimal initialization scale with additional conditions.
4

---

Published as a conference paper at ICLR 2026
3.2 COUPLED DYNAMICS IN DEEP NETWORKS INDUCE IMPLICIT BIAS TOWARDS Low RANK
Section 3.1 illustrated the importance of coupled training dynamics, driven by data connectivity, for achieving low-rank solutions in simple two-layer factorizations (L = 2). Building on this understanding, we now extend our analysis to deep networks (L 2 3). For illustrative purposes, consider a depth-3 network 331 An arbitrary observed entry Way from this matrix is given by:
Wij Σ k=
d2
E
1=1 (W3). (W
di
Crucially, because all elements of the intermediate matrix W2 contribute the computation of Waj regardless of ji, gradients of different observed entries will propagate through and update these shared elements in W2. This inherently couples their training dynamics, a structural feature distinct from the depth-2 case, where coupling is primarily determined by the observation pattern. Such inherent coupling, in turn, implies a potential intrinsic bias towards low-rank solutions for deep models. To formalize this notion, we introduce the following definition of coupled dynamics.
Definition (Coupled/Decoupled Dynamics), Consider the malrix completion setup with the model (t) (t) (t) gald, Let O(t) be the vector of all trainable parameters evolving according to the gradient flow dynamics (defined in (3)). The gradient flow dynamics are decoupled
if there exists a partition of Q into non-empty, disjoint subsets Di...., .gr (K 2 2) such that
U 94 = Ω and the following condition holds for any (i, j) E Q2 and (p, q) E 94 with k + l:
(oo (t), Vo (B) = 2.
The gradient flow dynamics are coupled if they are not decoupled.
While Bai al. (2024) introduce similar terminology in Definition A.5, their definition is restricted to depth-2 networks. We extend this notion to networks of arbitrary depth. For depth-2 matrices, it is straightforward to verify that coupled and decoupled dynamics typically correspond to connected and disconnected graphs, respectively, based on Definitions and 2. For depth 2 3 matrices, any initialization with an absolutely continuous distribution (e.g., Gaussian, uniform) yields gradient flow dynamics that are coupled with probability one, irrespective of the observation pattern (see Proposition B. in Appendix B). However, special cases exist where training dynamics are decoupled even for L 2 3. Refer to Appendix B for further discussion.
3.2.1 IMPLICIT BIAS OF DEPTH UNDER
OBSERVATIONS
To gain deeper theoretical insight into how coupled dynamics induce low-rank bias depth increases, we study the block dock-diagonaa observation setting. Specifically, we consider a ground-truth matrix W' E pdxd with the block-diagonal observation set
block
A
U ((i.)) i.j E {(b 1)s 1, 1,....,bs) bs),. bE(x]
where s, N satisfy d= sn. Here, denotes the block size and the number of blocks. We consider W with positive and identical observations w* A w > for (i.j) E Dlock In this setting, the observed entries are confined to disjoint square blocks along the diagonal, forming disconnected observation pattern. Note that this formulation recovers the diagonal observation setting as the special case 1, and therefore strictly generalizes the diagonal case. As highlighted in the 2 x 2 example with s = 1 (cf. Figure 1b), this setting reveals a stark difference between shallow and deep networks despite the lack of connectivity.
We consider a depth-L factorization of the model, (L.(t) = (t- Wi(t) where Wi dx for all [L]. To investigate how dynamic coupling affects the low-rank bias, consider a family of initializations where, for parameters a > and m > 1, each factor matrix (0) is initialized follows:
a a/m
a/m a/m
a/m
Wi(0)=
E Radd
VI E (Z).
(7)
a m a/m m
5

---

Published as a conference paper at ICLR 2026
Remark. Random Gaussian initialization allows coupling but introduces Ld degrees of freedom, making it almost impossible to analyze individual training trajectories. For this reason, prior work often adopts deterministic initializations such as aLd (Gunasekar al., 2017; Arora al., 2019). We follow this approach but adopt a more general deterministic family that is adequate for establishing theoretical claims. Our initialization interpolates between oJa (as and the parameter m allows direct control over the initial numerical rank.
1) and oda (as o0),
Using this initialization scheme with diagonal observations, the following proposition specifies how parameters m and network depth L determine if training dynamics are coupled or decoupled:
Proposition 3.2. Consider a depth- L model, where each factor Wi(0) R.xdd initialized with (7)
trained with blookk Then, by Definition 2, the following hold for any 2 1 and n 2 2:
For depth I. 2, the training dynamics are decoupled for all m > 1.
* For depth I. 2 3:
-The training dynamics are coupled if m < oc. The training dynamics are decoupled if m = oo (i.e., inifialization with ala).
By Proposition D.1 1 in Appendix D.4, the loss converges to zero under the gradient flow dynamics (3). Building this result, our objective is to determine the rank of solutions found by gradient flow depending on the coupling of dynamics. The theorem below presents an equation of each singular value of the converged matrix WL. (oo), for all I, 2 2.
Theorem 3.3. Consider the product matrix WLa Fal whose factor matrices W E Raxd are initialized
according to (7). Under the gradient flow dynamics (3), we have ((W L. (oo); S (laon
0 (Proposi-
tion D.I, Appendix D.4). For all parameters a > 0. > 1, 2 2, 8 2 I, and L 2 2, the singular
values 01 2 02 2 .... aa of (oo) satisfy o, 0 for all n. The principal singular value T1 and the secondary singular values og (for E (2...., n)} are determined as follows:
If L 2 (decoupled dynamics): The singular values given in closed form by w* d(m + d 1)
w² wmm(mm- - 1)²
01 = (m + d 1) 1)2 + (n () 1)2²
0: (m (m + (n (m 1) 1)2
If 3 and 1 < < co (coupled dynamics): The singular values satisfy the implicit equations:
01 2 - /w"d= -5-0p 01
= =Cam,,
(8)
11 1-
(w" d - (n - 1)0r;) 2-6
2-1
6.
0,
La.m,L,ds
(9)
where Lo A (=)2-1
2-L (m 1)2 (m 1mm-1)2-4 1)
Lf 2 3 and m = oo (decoupled dynamics): The singular values converge to:
0. sw"
The proof of the theorem is provided in Appendix D.3. The theorem details the converged singular values of WtAT (Do) for our initialization scheme (7). Crucially, it reveals distinct outcomes based on the nature of the training dynamics. For decoupled camaamis--specfficeeee
when T. 2 (for sufficiently large m > 1), or when L 2 3 and m o--iingular values from to a, approach sw" and are independent of the scale a. This implies convergence to a rank-n solution. contrast, for coupled dynamics (L > 3 with finite m), the outcome becomes a- a-dependent. To illustrate the implications of these implicit equations, we present the following corollary.
Corollary 3.4. Let 1 < m
> 2, 2 1, w" O, and L 2 3 be fixed. Then, 0, the stable rank of the limit product matrix WL.1 (20) converges to one; that is,
srank ( (oo)) 1.
The proof of the corollary is provided in Appendix D.6. Note that, according to Theorem 3.3, the stable rank of the depth-2 network satisfies srank (W21(0))) (m+d-1)* (m-)) which is (m+d-11

---

Published as a conference paper at ICLR 2026
m77 -
m 100
m 1000
m 11
10
Singular Values
0.1
0.002
1c
10
1c:10
0.2
0.001
15
10:07
1e
0.1
0.001
10:05
1c:07
1c:00
0.1
0.001
10-05
1c:77
1c:10
at.
at
3,0
7.0
1-2.0
1 3,
L-5,
7.0
Figure 2: Singular values of W (oe) (numerically obtained from Theorem 3.3) against initialization scale a=_ for the diagonal observation task where 8 = 1. Solid lines represent the largest singular value σi: dashed lines denote the other (identical) singular values o, for 2 2. For finite m. these results illustrate that both greater depth L and a smaller initial scale a strengthen the low-rank bias, in contrast to the L = 2 case. Conversely, very large m (m 10² approximating initialization, leads to decoupled dynamics and full-rank solution, independent of both L and a.
ald (rank-d)
independent of the initialization scale a, and is approximately when is large. In contrast, for any depth L 2 with finite m, Corollary 3.4 implies that as a - O. then srank (L.((oo) + 1, so the depth-T. network converges nearly rank-1 solution.
Remark. Readers may wonder why, even under decoupled dynamics, the solution converges to rank-n rather than a rank-d matrix. Although the dynamics are decoupled at the level of the full matrix (Definition 2), applying the same notion coupling to each diagonal block separately reveals that the dynamics are coupled within each block. Since all rows (or columns) inside a block share the same observation pattern, the row (or column) space is spanned by at most block-wise patterns, yielding rank at most In particular, when solution becomes full rank (see Figure 2). This block-diagonal example thus highlights how coupled versus decoupled dynamics govern the strength of the low-rank bias.
(diagonal observations) and hence d, the
We consider diagonal observations (s = 1) with w and d = 10, and numerically solve the im- plicit equations (8) and (9) to examine how the depth L and initialization parameters (a, m)) influence the singular value distribution. Note that both equations admit unique solutions (Proposition D.2 in Appendix D.5). To ensure fair comparison across depths, set the initialization scale so that the scale of the W L.:1(0) (0) is comparable across depths; concretely, we match the scale of a across different values of L. The results in Figure 2 confirm that these coupled dynamics in models with L 2 3 and finite indeed induce low-rank bias, contrasting with the full-rank outcomes of the decoupled cases. Moreover, this bias becomes more pronounced L increases, evidenced by a wider gap between i and o, for 2 2.
Additional numerical evidences are provided Figures 5-8 (Appendix C.I). Moreover, Figure in Appendix C.I shows that these numerical results agree with the outcomes of gradient descent with sufficiently small learning rate. Moreover, although we do not provide theoretical proof, we conduct experiments under noisy diagonal observations (Figure 11), non-equal diagonal observations (Figure 12), and with various optimizers including SGD, GD with momentum, Adam, RMSProp, and Adagrad (Figures 13-17). Across all settings, consistently observe the emergence of depth- induced low-rank bias. We further train practical neural networks to examine whether increased depth indeed leads low-rank bias. The results shown in Figures 18-21 (SGD with momentum), 22-25 (Adam), and 26-29 (RMSProp) in Appendix C.I.1 indicate that as depth increases (e.g.. ResNet- to 101 and VGG-11 to 19), the average effecctie rank decreases. highlighting the emergence of low-rank bias in practical neural networks across these optimizers.
Remark. Our analysis of low-rank bias for specific family of deterministic initializations resolves the challenging open problem (1) highlighted Section 14.1 of Menon (2024). Figure 10 in Appendix C.1 further demonstrates that our proposed deterministic initialization exhibits qualitative trends similar to Gaussian initialization. We therefore argue that our results provide foundational insights into low-rank bias applicable more general random initializations.
7

---

Published as a conference paper at ICLR 2026
Pre-training
Warm-start Training
Cold-start Training
17.5
17.5
17.5
15.0
13.2
10.0
10.0
7.3
7.5
7.5
1.0
1.0
1.0
Reconstruction
6.-
0.8
0.6
0.-
0.2
0.0
Initial Scale d
Initial Scale a'
Initial Scale d'
Figure 3: Experiments use 100 x 100 rank-5 ground-truth matrix. Pre-training utilizes 2000 randomly sampled entries (pere 12prel 2000), while post-training adds 1000 more, forming Ω p (Pepe Aposts [S2pert = 3000). The top row of panels displays effective rank, and the bottom row shows reconstruction error, both measured at convergence. The leftmost panels depict training on Opre. and the rightmost on Spostt both starting from random Gaussian initialization. The middle panels show warm-stan training on Spost: initialized from converged pre-trained models with Spre-
4 UNDERSTANDING LOss OF PLASTICITY IN DEPTH-2 MATRIX COMPLETION
Studying the inherent tendency towards low-rank solutions in matrix completion can offer further insights into the loss of plasticity phenomenon. Kleinman et al. (2024) report the emergence of this phenomenon in matrix completion: models pre-trained on limited observations struggle to adapt when training continues on augmented observations.
We conduct an experiment using a rank-5 ground-truth matrix W 10000100 where pre-training is performed on a sparse observation set and post-training continues with additional observations. In Figure we compare two post-training strategies: warm-stort training, initialized from the pre- trained model, and cold-start training, initialized from scratch on the augmented observations, across different depths in terms effective rank and reconstruction error. We observe that, even when pre-trained sparse observations, deeper models increasingly favor low-rank solutions as depth increases. This supports our argument (Section 3.2) that deeper networks inherently converge toward low-rank solutions from limited and disconnected initial data. Consequently, further training on augmented data does not substantially increase the rank compared to training from scratch on the augmented observations. Based our observations, models helps them mitigate the loss of plasticity, while the phenomenon more pronounced in depth-2 models. To theoretically understand the underlying cause of this phenomenon itself, henceforth focus our analysis on depth-2 models.
conclude that the low-rank bias of deep
In Section 4.1, we study pre-training on diagonal-only observations. We then consider post-training on 2 x 2 (Section 4.2) and d x d (Section 4.3) matrices. For the 2 2 case, we set 9 & {(1, (2, and obtain the post-training set ( by adding single off-diagonal entry ensure connectivity, ie. 22 & 1), (1, (2, 2)). Likewise, for the d d case, 2ide {(i, i) he(d) and (2 is formed by adding additional (off (off-diagonal) observations; see Section 4.3 for details.
Remark. Kleinman et al. (2024) observe that loss of plasticity is further intensified with increasing network depth, conclusion they reached by measuring "relative reconstruction loss" when compared to models trained from scratch on the augmented dataset. In their setup, training is run for a fixed number of iterations without waiting for convergence. whereas in our experiments terminate each training phase once the loss falls below a fixed threshold.
4.1 PRE-TRAINING WITH DIAGONAL OBSERVATIONS
To clearly observe loss of plasticity in a setting consistent with Section 3.2, we pre-train using only diagonal entries, yielding a disconnected pattern. We consider decoupled-to-couplled scenarios,
8

---

Published as a conference paper at ICLR 2026
where additional data is introduced to induce coupled training dynamics. For depth-2 models, they correspond to a disconnected to- connected observation pattern. For the pre-training closed-fone solutions that depend solefy on the network's 's initialization be found in the following proposition:
Proposition 4.1. Consider a ground truth matrix W E Rddd with diagonal observations D The model is factorized as (t) (t)) where A(t), B(t) Rdxd, For each observation (i,i) i) E Spre. define the constants P: and Q; based on the initial values:
d
P.
O.kk (0) and Qi (oxx(0)2 baa
1=
1=1
Furthermore, for each diagonal observation, let the parameter F3 be determined from the ground truth entry wit and the constants defined above,
P.+ 2
log
Then, assuming
w1 P. 22
convergence to zero-loss solution of the loss <(W A.W.;; Pppre) any entry apg apg (0o) of the converged
matrix (po) and any entry (00) of the converged matrix B (Do) (for any p. E [d] are given by:
(pc = Opq (0) cosh (Fp) dap (0) sinh (Fp),
Upg (00) = bpg (0) cosh (Fy) aep (0) sinh (Fg).
Remark. The proposition covers arbitrary initializations with distinct wag which goes beyond L = 2 setting. While the above analysis focuses on (block-)diagonal
Theorem 3.3 in the observation cases, it can be generalized to any fully disconnected case (i.e., single observation per row and column). This yields distinct solutions for various types of observation sets, detailed in Appendix E.
We analyze the scenario where training resumes from a state obtained through pre-training. Let the pre-training phase conclude at sufficiently large timestep Ti. For simplicity, we assume that the solution W A, (Ti) has perfectly converged with respect to the pre-training objective, neglecting any residual error due to the finite duration of this phase. Our subsequent analysis demonstrates that, starting from W W. (Ti), the model WA A.B (t) cannot converge to a low-rank solution.
4.2 POST-TRAINING: 2 MATRIX EXAMPLE
We aim to analyze scenarios where training is resumed under coupled dynamics, building upon solutions obtained from an initial decoupled pre-training phase (Proposition 4. To this end, we first define the specific pre- pre-training setup for an illustrative 2 2 case: We observe diagonal entries (2) which are identical and positive, i.e., w² - wii = w > 0. To make loss of plasticity particularly pronounced during the pre-training, we initialize the model with al2 (for > 0), which is the m = oo setting of our initialization scheme in (7). Then, from Proposition 4.1, it follows that:
V V
A(Ti) = B(T))
(10)
For the subsequent post-training phase, an additional off-diagonal observation is introduced to establish connectivity. Without loss of generality, we assume wi2
revealed, while the diagonal entries wi, and w/2 from the pre-training phase remain observed. Thus, the updated set of observed entries becomes () = {(1, 1), (1,2), (2, 2)). The ground-truth matrix is assumed to be rank-1, ensuring the setting is non-trivial, and the task is thus to predict the remaining entry win = 2² 2/ > O. The following theorem, however, reveals contrasting outcome for this entry. Theorem 4.2. Let A(Ti), B(T,) be the factor matrices obtained from the pre-training phase, as specified by (10). Then, running gradient flow during the subsequent post-training phase (for t 2 Ti). starting from A(Ti) and (Ti)), results in exponential decay of the loss:
((W.B((); post S
2 2 -20""
2w" (1-T.).
Consequently, a lower bound for the stable rank of the converged matrix WA.B(oo) is given by:
(-8 Wi2
srank (W ( 2 1 exp
w
Furthermore, for all 1. Ty, w21 (1) of the evolving matrix W (l, satisfies w21 (1) < 0.

---

Published as a conference paper at ICLR 2026
The theorem indicates that the loss decreases exponentially fast, particularly when starting from large-norm solutions (at a rate governed by w*). Therefore, since the model converged to high-rank solutions during pre-training, its singular values remain largely unchanged from this initial state, as long as w12 has a small magnitude compared to w". Furthermore, the unobserved entry w21 (1) converges to negative value, which contradicts the positive 2 expected for the true rank-1 solution.
4.3 POST-TRAINING: d X d MATRIX UNDER LAZY TRAINING REGIME
We attribute Theorem 4.2 primarily the model's "lazy training" (Chizat et al., 2019) as large-norm initializations lead to faster loss decay, causing the model to converge to nearby global minimum that may not be a low-rank solution. Drawing on this concept, we extend the preceding analysis of loss of plasticity to the more general case of d x d ground-trune truth matrices. The following theorem states that when the model is initialized with a sufficiently small loss, resulting from warm-starting that perfectly fits all previously observed data, the model exhibits lazy training. This, in turn, prevents further learning that would reduce the rank and instead steers the model towards a nearby minimum.
Theorem 4.3. For factor matrices A, B E Rdxd, suppose A and B are balanced at t = 0, i.e., A(O) ( ((O) T. Let ['AA, B) be the function that maps (A, B) to the vector of model predictions for given set of observed entries D We then define max and min as the maximum and minimum singular values, respectively, of the Jacobian of the function I evaluated at the pre- trained state (at t Tı). the loss time T1 satisfies e (WA.B(T)): results in exponential decay of the loss:
omiin 1152do
ttis
8 Oposs) ) (W (Ti); D
a (- minn -TTi
exp
Consequently, the stable rank of A(t) (which is equal to that of B())) remains bounded below by
|AATT))lE
Omin
srank (A(t))
4v/2d
[A(TT))l2 +
Amii
1v2d
The theorem states that if model has little remaining learn (achieved via pre-training), it undergoes lazy training regime. In this regime, the loss converges rapidly, while its stable rank remains largely unchanged from the initial state. Thus, once model has converged a high-rank state, it struggles to recover a low-rank structure even when new observations are introduced to form connectivity. The proof of Theorem 4.3 is provided in Appendix E.3.
Example. As illustrative example, consider a rank-1 ground-truth matrix W' E Radd
w
cw"
w
c c--
w"
w
w"
0 1
W
c=
5
2-= w
2-dw
w
We pre-train only the identical diagonal observations w" using Dio with initialization A(0) (0) = aL up to time T, (see Proposition 4.1 for the pre-training solution). We then reveal the full upper-triangular set Dod {i S i < d) to form connectivity and continue training. By Theorem 4.3, for every 2 T1, the stable rank of A(t) is uniformly lower-bounded by 2(d):
4d 1
srank (A(1)) 2
4vd+11
CONCLUSION
We demonstrate that in matrix completion, deeper networks (I. 3) inherently exhibit a stronger low-rank bias than shallow networks, primarily due to their coupled training dynamics that manifest regardless of observation patterns. For tractable analysis, we consider gradient flow starting at a family of deterministic initializations, showing in the block-diagonal observation setting that depth amplifies the low-rank bias. Furthermore, our theoretical analysis of warm- starm-startngg scenarios details the loss of plasticity phenomenon, revealing how large- high-rank initial states hinder convergence to low-rank solutions. We believe the theoretical results from matrix completion provide broader insight into how depth shapes implicit bias and explains the loss of plasticity in practical deep networks.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work is purely theoretical and involves no human subjects, personal data, or new dataset collection. We foresee no safety, fairness, or privacy risks and confirm that we are in accordance with the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
The proof' of all theorems and propositions in the main text appear in the corresponding appendices: Theorem 3.1 in Appendix D.I, Proposition 3.2 in Appendix D.2. Theorem 3.3 in Appendix D.3, Proposition 4. in Appendix 1, and Theorems 4.2 and 4.3 in Appendices E.2 and E.3, respectively.
ACKNOWLEDDGMMENT
This work was supported by two Institute of Information & communications Technology Planning Evaluation (IITP) grants (No. RS-2022-I1220184 Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics; No. 15-20019-II19000 Artificial Intelligence Graduate School Program (KAIST)) funded by the Korean government (MSIT). The work was also partly supported by National Research Foundation of Korea (NRF) grant (No. RS-2024-00421 2032 funded by the Korean government (MSIT).
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2018
Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36: 17032-47051 2023.
Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparamettrization. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 244-253. PMLR, 10-15 Jul 2018. URL https: htteps://proceeee /ptpc//ecoccee....... mlr. wal.spres/// wttp:///a///rroa.aaaaaaaaa html.
Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems, 32, 2019.
Jordan Ash and Ryan P Adams, On warm-starting neural network training. Advances neural information processing systems, 33:3884-3899 2020.
Zhiwei Bai, Jiajie Zhao, and Yaoyu Zhang. Connectivity shapes implicit regularization matrix factorization models for matrix completion. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024.
Tudor Berariu, Wojeiech Czarnecki, Soham De, Jorg Bornschein, Samuel Smith, Razvan Pascanu, and Claudia Clopath. A study on the plasticity of neural networks. arXiv preprint 106.v:2006.00442 2021.
E.J. Candes and T. Tao. Decoding by linear programming. IEEE Transactions on Information Theory, 51(12))4203-421 2005.
Matias D. Cattaneo, Jason Matthew Klusowski, and Boris Shigida. On the implicit bias of Adam. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, Machine
and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Learning. volume 235 of Proceedings of Machine Learning Research, pp. 2862-5906. PMLR, 21- 27 Jul 2024. URL https: proceed i ngs ml pprsss v235/ http://////////a////////// wat.aneo2aaaaoccc 4a .tml..

---

Published as a conference paper at ICLR 2026
Yihong Chen, Kelly Marchisio, Roberta Raileanu, David Adelani, Pontus Lars Erik Saito Stenetorp, Sebastian Riedel, and Mikel Artetxe. Improving language plasticity via pretraining with active forgetting. Advances in Neural Information Processing Systems, 36:31543-31557, 2023.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. Advances in neural information processing systems, 32, 2019.
On lazy training in differentiable programming.
Shibhansh Dohare, Richard S Sutton, and A Rupam Mahmood. Continual backprop: Stochastic gradient descent with persistent randomness. arXiv preprint arXiv:2108.0665 2021.
Spencer Frei, Gal Vardi, Peter Bartlett, Nathan Srebro, and Wei Hu. Implicit bias in leaky reLU networks trained on high-dimensional data. In The Eleventh International Conference on Learning Representutions, 2023. URL https: /ttp://///////eeeeee/ee/e net forum ? hitp:ammmiippppp 1d-Jppo..EEIEEW
Tomer Galanti, Zachary S Siegel, Apama Gupte, and Tomaso Poggio. SGD and weight decay provably induce a low-rank bias neural networks, 2023. URI.
net/forum? (otorrummmdnn//aaaaaaaa d=N77v4474Cyy.
Tomer Galanti, Zachary S Siegel, Aparna Gupte, and Tomaso A Poggio. SGD and weight decay secretly minimize the rank your neural network. In NeurlPS 2024 Workshop on Mathemat- ics of Modern Machine Learning, 2024. URL https: http:////pppeeeeeeeeeeee net forum m Pid= xhW2WyPhRP.
Daniel Gissin, Shai Shalev-Shwartz, and Amit Daniely. The implicit bias of depth: How incremental learning drives generalization. In International Conference on Learning Representations, 2020.
URL https:ttp//.. htt.ttssss//ppeeeeieeeeeeeeeeeee /wwpe//wwweeeweeeeeeeeeeeee.e..p net www.nnrrtffftr/to forum? 1 1d==11jonNFw.. JOnNFwB.
Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in neural information processing systems, 30, 2017.
Isaac Han, Sangyeon Park, Seungwon Oh, Donghu Kim, Hojoon Lee, and KyungJoong Kim. FIRE: Frobenius-isoment reinitialization for balancing the stability plasticityy tradeoff. In The Fourteenth International Conference on Learning Representations, 2026. URL httpt::// /RRphttpp/:/opeeeeeieee net/
J Fernando Grrnandez-Gareiaa Shibhansh Dohare, Jun Luo, and Rich S Sutton. Reinitializing weights vs units for maintaining plasticity in neural networks. arXiv preprint arXiv:2508. 00000000022220 /2, 2025.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint 2rXiv:2103..0427,
Xiangyun Hui, Xiaoxuan Ma. Yixuan Yang, and Song Li. The implicit regularization of gradient flow on separable datasets in relu networks. Neurocomputing, pp. 131367, 2025. ISSN 0925-2312. doi: http:///b...gggloooooo..e.eem.m. 1033 URL https: www. wttp///wwwwneceieeeet...... com/ http://ccaeeeearrrreea/a//////22
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient non-stationarity and generalisation in deep reinforcement learning. arXiv preprint
arXiv:2006.05826, 2020.
Arthur Jacot. Implicit bias of large depth networks: notion of rank for nonlinear functions, arXiv preprint arXiv:2209, 15055, 2022.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference Learning Representations, 2019a. URL https:://penrrevieww net
Ziwei Ji and Matus Telgarsky. The implicit bias of gradient descent on nonseparable data. In Conference learning theory, pp. 1772-1798. PMLR, 2019b.
12

---

Published as a conference paper at ICLR 2026
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 17176-17186. Curran Associates, Inc., 2020. URL https: http::///ococeeddeeeeee neur cc pacpppppr_ff wtt.fe///pppp///222222 4ccee00dee2eebbeeeapapeeeee 4c2eb9-Paper
Saaee4440000119950dccooo 9a5c0de1
Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon Shaolei Du, and Jason D Lee. Understanding incremental learning of gradient descent: A fine aee-ggained analysis of matrix sensing. In International Conference on Machine Learning, pp. 15200-15238. PMLR, 2023.
Hyunji Jung, Hanseul Cho, and Chulhee Yun. Convergence and implicit bias of gradient descent on continual linear classification. The Thirteenth International Conference on Learning Represen- tations, 2025. URL https::t///.tc http:////ppppeeeeeee/eeeeeeeeee. net/ forum Poorummid-DII Poruumiid--Tqx331ijkk
Daesung Kim and Hye Won Chung. Rank-1 matrix completion with gradient descent and small random initialization. Advances in Neural Information Processing Systems, 36:10530-10566, 2023.
Jiyeon Kim, Hyunji Lee, Hyowon Cho, Joel Jang, Hyeonbin Hwang, Seungpil Won, Youbin Ahn, Dohaeng Lee, and Minjoon Seo. Knowledge entropy decay during language model pretraining hinders new knowledge acquisition. In The Thirteenth International Conference Learning Representations, 2025. URL https: / www.oeeeieii.eee net/
Michael Kleinman, Alessandro Achille, and Stefano Soatto. Critical learning periods emerge even in deep linear networks. In The Twelfth International Conference on Learning Representations, 2024.
URL https: / opwnrevieweeeeeeeeeeeeeeee net forum? id 5g12ckkk
Yiwen Kou, Zixiang Chen, and Quanquan Gu. Implicit bias of gradient descent for two-layer relu and leaky relu networks on -othhoontll data. Advances in Neural Information Processing Systems, 3630067-30211, 2023.
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learn- ing via regenerative regularization. In Vincenzo Lomonaco, Stefano Melacci, Tinne Tuytelaars, Sarath Chandar, and Razvan Pascanu (eds.), Proceedings The 3rd Conference on Lifelong Learn- ing Agents, volume 274 of Proceedings of Machine Learning Research, pp. 410-430. PMLR, 29 Jul-01 Aug 2025. URL https: /ttp//ppooednnss////ee/ee ml wwr.ppres////. v2 4 kumar255 html.
Hojoon Lee, Hyeonseo Cho, Hyunseung Kim, Donghu Kim, Dugki Min, Jaegul Choo, and Clare Lyle. Slow and steady wins the race: Maintaining plasticity with hare and tortoise networks. In ICML, 2024. URL https://///....... / hete:://ppeeeeeeee opwnreview...eeeo net/ forum ? 1id--1id111 777...
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. In The Thirteenth International Conference on Learn- ing Representations, 2025. URL https: /ppereview net/ forum i d= jxLi DKsuDo.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over parameterizee matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory, pp. 2-47. PMLR, 2018
Zhiyuan Li, Yuping Luo, and Kaifeng Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2021. URL https: htpp///eeeeeeeeeeeeeeeeeeeeeeeee net forum AHO 5M577R.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney, Understanding plasticity in neural networks. In International Conference on Machine Learning, pp. 23190-23211 PMLR, 2023.
Clare Lyle, Gharda Sokar, Razvan Pascanu, and Andras Gyorgy. What can grokking teach us about learning under nonstationarity? arXiv preprint arXiv:2507.20057 2025.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In International Conference on Learning Representations, 2020. URL https: httthtt//ppeeeeeeeeeeee /pppnreviee. net. forum?
13

---

Published as a conference paper at ICLR 2026
Jianhao Ma and Salar Fattahi. Global convergence of sub-gradiee gradiend method for robust matrix recovery. Small initialization, noisy measurements, and weer-paraeeterization Journal of Machine Learning Research, 24(96): 84, 2023.
Jianhao Ma and Salar Fattahi. Convergence of gradient descent with small initialization for unreg- ularized matrix completion. In The Thirty Seventh Annual Conference Learning Theory, pp. 3683-3742. PMLR, 2024.
Govind Menon. The geometry of the deep linear network. arXiv preprint arXiv:2411.00000 2024.
Chaewon Moon, Dongkuk Si, and Chulhee Yun. Minor first, major last: A depth-induced implicit bias of sharpness-awann minimization. In The Fourteenth International Conference on Learning Representations, 2026. URL https: http//pppeeeeeeeeeeeeeeeeeeeeeee www.eeeieiieeeeeeeeeeeeeeeeeeeee
Mor Shpigel Nacson, Jason Lee, Suriya Gunasekar, Pedro Henrique Pamplona Savarese, Nathan Srebro, and Daniel Soudry. Convergence of gradient descent on separable data. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty- -econn International Con- ference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 3420-3428. PMLR, 16-18 Apr 2019a. URL https::/p//......... proceeooceeieeeec ngs. mll. press/v88 / nacsonl 9b. html.
Mor Shpigel Nacson, Nathan Srebro, and Daniel Soudry. Stochastic gradient descent separable data: Exact convergence with a fixed learning rate. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learming Research, pp. 3051 3051-3059, PMLR, 16-18 Apr 2019b. URL httph:tp////m http:///ooceeeeneeee/ee mlr. pwwpsrrsssseeeeee / vv nacsonn 9a html.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint 1rXiv:1412.66111 2014.
Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of optimization and implicit regularization in deep learning. CoRR, abs/1705.0307/ 2017. URL
hittp://arxe wwt//bs//////o/ooo.ooo...o.....r abs 700.705000000
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Sangyeon Park, Isaac Han, Seungwon Oh, and Kyung-Joong Kim. Activation by interval- wise dropout: A simple way to prevent neural networks from plasticity loss. arXiv preprini arXiv:2502.01342, 2025.
Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in neural information processing systems, 33:21174-21 2020.
Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In International Conference on Machine Learning, pp. 8913-8924. PMLR, 2021.
Seyed Roozbeh Razavi Rohani, Khashayar Khajavi, Wesley Chung, Mo Chen, and Sharan Vaswani. Preserving plasticity in continual learning with adaptive linearity injection. arXiv preprint arXiv:2505.094480 09446, 2025.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEEE, 2007.
Backrok Shin, Junsoo Oh, Hanseul Cho, and Chulhee Yun. Dash: Warm-starting neural network training in stationary settings without loss of plasticity. Advances in Neural Information Processing Systems, 37:43300-43340 2024.
Mahdi Soltanolkotabi, Dominik Stöger, and Changzhi Xie. Implicit balancing and regularization: Generalization and convergence guarantees for overparameterizee asymmetric matrix sensing. In The Thirty Sixth Annual Conference Learning Theory, pp. 5140-5142. PMLR, 2023.
14

---

Published as a conference paper at ICLR 2026
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57, 2018.
Jacob Mitchell Springer, Sachin Goyal, Kaiyue Wen, Tanishq Kumar, Xiang Yue, Sadhika Mal- ladi, Graham Neubig, and Aditi Raghunathan. Overtrained language models are harder to fine-tune, In Forty-second International Conference on Machine Learning, 2025. URL
https://openreview httpeeewweee//eeeee wtt.::/tottoounodddeeeeeeeee WdWWWeeSSft.t..ttt
Dominik Stöger and Mahdi Soltanolkotabi. Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction.
Advances in Neural Information Processing Systems, 34:23831-238433 2021.
Matus Telgarsky. Deep learning theory lecture notes. Lecture Notes vO. 0-e7150f2d (alpha), Univ. Illinois Urbana-Champaise Champaign, II, USA, 2021.
Nadav Timor, Gal Vardi, and Ohad Shamir. Implicit regularization towards rank minimization in relu networks. In International Conference on Algorithmic Learning Theory, pp. 1429-1459. PMLR, 2023.
Simon Vock and Christian Meisel. Critical dynamics governs deep learning. arXiv preprint arXiv:2507.08527, 2025.
Bohan Wang, Qi Meng, Wei Chen, and Tie-Yan Liu. The implicit bias for adaptive optimization algorithms homogeneous neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10849-10858. PMLR, 18-24 Jul 2021. URL https: http///poceeee.....//////
m.rrspres/ / v/1vvv333. wang21g; html.
Bohan Wang, Qi Meng, Huishuai Zhang, Ruoyu Sun, Wei Chen, Zhi-Ming Ma, and Tie- Yan Liu. Does momentum change the implicit regularization on separable data? In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances Advances in Neural Information Processing Systems, 2022. URL https:://///. wpwpre/eeeeeeeeeeeeooooom. net/ fareetfoormmiiiiie www.ummiiiiiiaiii -uuglurji..
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3635-3673, PMLR, 2020.
Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations, 2021, URL 1MMQH.
https: /ttts::/cpoee /tt.nreveeeeeeeeeeeeeeeeeeeeeeee .ne// /tto//oooddddsii/e
Chenyang Zhang, Difan Zou, and Yuan Cao. The implicit bias of adam separable data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https: /ttp::///ppeeeveeeeeeeeee net/ forum? fote:/oooooodxxxeeaeannn d=xpxxxaaxx.x...
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017. URL https: http///pppeeeieeeeeeeeeeeeeeeeep 2tte://foormmm1118yyyg3323.
Dan Zhao. Combining implicit and explicit regularization for efficient learning in deep networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https:/ /pwp//vepneeeeeeeooe/ee/e net forum?
1d=sADLR12ST1c
15

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
Problem Setting
3 Implicit Bias of Depth Induced by Coupled Training Dynamics
3.1 Warm-up: Coupled Dynamics vs. Decoupled Dynamics Depth-2 Networks
3.2 Coupled Dynamics in Deep Networks Induce Implicit Bias Towards Low Rank
4
Understanding Loss of Plasticity in Depth-2 Matrix Completion
4.1
Pre-training with Diagonal Observations
4.2
Post-training: 2 x Matrix Example
4.3 Post-training: d x d Matrix under Lazy Training Regime
10
5
Conclusion
10
Further Related Works
17
A.I Implicit Regularization in Neural Networks
17
A.2 Loss of Plasticity
18
Coupled and Decoupled Training Dynamics
19
B.1 Coupled Dynamics Example
20
B.2 Decoupled Dynamics Example
21
C Additional Experiments
23
C.1 Implicit Bias Experiments
23
C.2 Loss of Plasticity Experiments
32
Proof for Section 3
D.I Proof for Theorem 3.1
34
................................
D.2 Proof for Proposition 3.2
D.3 Proof for Theorem 3.3
39
D.4 Loss Convergence
47
D.5 Uniqueness of The Limiting Singular Values
49
D.6 Proof for Corollary 3.4
51
E Proof for Section 4
52
E.1 General Form and Proof of Proposition 4.1
52
E.2 Proof of Theorem 4.2
56
E.3 Formal Statement and Proof of Theorem 4.3
67
F
Useful Lemmas
73
16

---

Published as a conference paper at ICLR 2026
A FURTHER RELATED WORKS
A.1 IMPLICIT REGULARIZATION IN NEURAL NETWORKS
A substantial body of work investigates the implicit regularization of gradient-based training in overparameterized models (Gunasekar et al., 2017: Ji & Telgarsky, 2019a;b; Woodworth et al., 2020; Yun et al., 2021; Razin et al., 2021; Andriushchenko et al., 2023; Frei et al., 2023; Jung et al., 2025; Hui et al., 2025; Moon et al., 2026). For linearly separable classification trained with (S)GD, Soudry et al. (2018) show that gradient descent the logistic loss converges in direction to the lg max- max-margin classifier. Building on this result, Nacson et al. (2019b) establish analogous directional convergence guarantees for SGID, and Nacson et al. (2019a) extend the theory to broader family of loss functions. For homogeneous neural networks, gradient descent likewise exhibits directional convergence, and problem (Ji &
the limit direction coincides with a KKT point of an appropriate margin-maximian Telgarsky, 2020; Lyu & Li, 2020).
For adaptive methods in linearly separable classification, Wang et al. (2022) analyze (S)GD with momentum and deterministic Adam and show that these methods also converge in direction to the max- max-margin solution. This analysis is further extended to homogeneous models by Wang et al. (2021). More recently. Zhang et al. (2024) demonstrate that when the stability constant is negligible, Adam exhibits a qualitatively different implicit bias and converges to the maximum for margin rather than the lg max- max-margin direction selected by (S)GD. Along related line, Cattanco et al. (2024) use backward error analysis to study RMSProp and Adam and show that their implicit regularization depends sensitively on hyperparameters and the training stage. Closely related to our setting, Zhao (2022) examine matrix completion and show that Adam, when combined with an explicit spectral ratio penalty, induces a strong low-rank bias even in depth-1 linear networks. However, their analysis focuses on deriving the flow of Adam and does not characterize the limiting solution.
Several works investigate how depth promotes low-rank solutions (Arora al., 2019; Gissin et al., 2020; Li et al., 2021; Huh et al., 2021; Jacot, 2022; Timor et al., 2023). Huh et al. (2021) provide empirical evidence that deeper networks (both linear and nonlinear) tend to find solutions with lower effective-rank embeddings. Complementing this, Timor et al. (2023) show theoretically that ReLU networks trained with squared loss exhibit bias toward low-rank solutions under the assumption that gradient flow converges to the solution minimizing the &g norm.
Turning to deep linear networks, Gissin et al. (2020) and Li et al. (2021) study depth-induced bias as a function of initialization scale. They report that, as depth increases, the dependence on initialization can become weaker, and incremental learning can emerge. However, their analyses consider a matrix factorization task, which they frame as matrix completion with full observations. Therefore, in their setting, convergence to a low-rank solution is guaranteed if the model converges to zero-loss, which does not hold in our matrix completion task settings.
While Arora et al. (2019) investigate the matrix completion task in deep linear networks, offering insights from derived singular value dynamics, they cannot fully track these dynamics to prove low-rank convergence as network depth increases. Their analysis is primarily restricted to the regime where & 2 ln. after which singular vectors are assumed have stabilized. For t 2 la- they find that one singular value can be expressed as a function of another, involving a constant term that emerges from the state to (which can be the dominant component). Based on this derivation, they demonstrate that the gap between these singular values widens with increasing depth. In contrast, our Theorem 3.3, by precisely tracking the converged values of singular values, rigorously establishes their ultimate behavior and the resulting low-rank bias.
Closely related our setting, Razin & Cohen (2020) study a depth L 2 matrix completion problem in a 2 x 2 example with three observations (one diagonal and two off diagonal entries). Their Theorems and 2 show that, as the loss converges, the effective rank converges to its infimum. However, their analysis does not distinguish between the depth I. = 2 and I. 2 3 regimes, and therefore does not identify a depth dependent low-rank bias or an underlying mechanism that explains it. In addition, their guarantees are independent of the initialization scale, so they do not capture the empirically observed phenomenon that low-rank bias becomes stronger as the initialization scale decreases. In contrast, our results explicitly separate the L 2 and L 2 3 cases, characterize the limiting singular values, and show how depth and initialization scale jointly control the emergence of low rank solutions in matrix completion.
17

---

Published as a conference paper at ICLR 2026
For depth-2 matrix completion tasks, Bai et al. (2024) introduce the connectivity argument. They prove that if the observations construct a connected bipartite graph, the model can converge to a low-rank solution when the initialization scale is infinitesimally small, subject to certain technical assumptions. Conversely, if the observations form a disconnected graph, the model generally cannot converge a low-rank solution. However, special case occurs if this disconnected graph is composed of complete bipartite components: here, the model converges to the minimum nuclear norm solution, again under specific technical assumptions. This characterization of implicit bias does not readily generalize to matrices with deeper matrices, as depicted in Figure 1.
A.2 LOsS OF PLASTICITY
Loss of plasticity describes a widely observed phenomenon where a model's ability to adapt to new information diminishes over time (Achille et al., 2018; Ash & Adams, 2020; Dohare et al., 2021; Nikishin et al., 2022; Lee et al., 2024; Shin et al., 2024; Lee et al., 2025; Lyle et al., 2025; Springer al., 2025; Kim et al., 2025; Han et al., 2026). The phenomenon is frequently observed in scenarios with gradually changing datasets, such as those encountered in reinforcement learning (Igl et al., 2020; Nikishin al., 2022; Lyle al., 2023) continual learning (Dohare al., 2021; Kumar et al., 2025; Chen et al., 2023; Park et al., 2025; Hernandez-Gaun Garcia et al., 2025; Rohani et al., 2025), where the model may struggle to adapt to new environments.
Although loss of plasticity is typically studied in non-stationary settings, a similar effect arises in stationary regimes where the dataset grows incrementally while the underlying distribution remains fixed (Ash & Adams, 2020; Berariu et al., 2021: Shin et al., 2024). In such cases, a model is first trained to convergence on an initial i.i.d. subset (e.g., a subset of CIFAR-10/100) and then warm-started for continued training on an expanded sample from the same distribution (e.g., the full CIFAR-10/100) Perhaps counterintuitively, these warm-started models often generalize worse, yielding lower test accuracy than models trained from scratch on the combined dataset.
While this phenomenon is problematic in many real-world applications where new data continuously added, theoretical studies on it remain scarce. Shin et al. (2024), for instance, offer theoretical explanation using an artificial framework. Within this framework, they demonstrate that such behavior occurs because warm-started models often complete training by memorizing data-dependent noise, which is not useful for generalization. However, the analytical framework they employ is considered artificial and limited in its ability to accurately characterize the optimization processes of typical deep learning models.
Recently, Kleinman et al. (2024) observed loss of plasticity in deep linear networks, identifying "critical learning periods": an initial phase of effective learning followed by a significantly reduced capacity to learn later (Achille et al., 2018: Vock & Meisel, 2025). They employ matrix completion framework to further observe this behavior. When observations from matrix completion tasks are treated as training samples in neural network training, they observed that model initially trained on sparse set of observations and subsequently retrained (i.e., warm-started) on expanded dataset typically exhibits larger performance gap (in terms of reconstruction error) compared to a model trained from scratch on the entire expanded dataset. However, their work does not offer theoretical guarantees to account for these observations. Motivated by this, in Section 4, we attempt to explain this behavior within the specific context of depth-2 matrix completion settings.
18

---

Published as a conference paper at ICLR 2026
B COUPLED AND DECOUPLED TRAINING DYNAMICS
This section introduces coupled and decoupled training dynamics (Definition 2) and illustrates them with concrete examples. Before that, we present Proposition B.I, which shows that for deep models (L 2 3), generic (absolutely continuous) initialization yields coupled dynamics almost surely.
Lemma B.1. Define Wha W. W- Wa and W ash La where 2 a. For Wey (1)
e, (L),
(w = (WLa+1(t) e.) (Wi-ia(t)e,)' E Bddd
Hence, for any (i, and (p,9).
(Vowsy(t), Vowpy(t)) = Σ (e, T((t)ep) [e] Si(t),,) 1-1
where Ti(t) & W1a4 (t) (rrei(t)))) (t) and Si(t) & wia (t) WTW (i are symmetric positive
semidefinite matrix.
Proof. Define af (1) A (1a++1(() e, and (1) Wi-1a())ey. By
wiy (1) e, W (1) (1)e; - - (1) (1)
we have V ww (t) at (t)b() () (t) Furthermore.
(oowy(t), Vewpy (t))
Ww, (t), Vw (p)
(1)6 (1) aa (1))()))))))
Σ ((.000 (1) a. (())) (89)()) b/0(0)
Σ (eTT(())pp)))))))))))))))))))))) (c] ((())e))
which concludes the proof.
with i.i.d. entries from any absolutely continuous distribution. For any observation set Ω C d [d] where [ 2 with probability I,
Proposition B.1. Let 2 3 and initialize (W())
WVo (0), V Vown (0) #
holds for all distinct (i.j), (p.g) E Ω. Consequently, K can satisfy the decoupling condition (6) at 0. Hence, by Definition 2, the gradient flow dynamics coupled with probability irrespective of the observation pattern.
nontrivial partition Ω = UK Pg with
Proof. By Lemma B.1, /. 0 we have
Pij. Wi...., Wr, A (Vowiy, Vowp) Σ (e Tiep) (e) Sie,),
which is polynomial in the entries of (Wi).. For any (1.0) # (p,9), we now show that Pij.pg is not the zero polynomial.
If i p, the L term reduces to e) SLey. By choosing WILL that Sz has nonzero (j. q) entry, this term evaluates to nonzero value; hence Pij.m is not identically zero. By symmetry, the same argument applies when j q-
19

---

Published as a conference paper at ICLR 2026
If p and + 9, consider | 2. Setting all other layers La, choose W3 that (e, Tzep) + and choose W1 so that (e) S2e,) * O. Then Pij.pp (e, (e, S2e,) o 0. Consequently, in all cases Pit.pg is not identically zero.
Since Pij.po is a nonzero polynomial in the entries of (WilfE) its zero set Zij.m {(W1..... WL) Pij.pg (W (Wi..... ..... WL) 0} is proper algebraic set in R Ld and hence has Lebesgue measure zero.
Let the initialization distribution of (W) (L())) be absolutely continuous with respect to Lebesgue measure. Then
Pr (W((), (Z))) E Zij.zy = 0,
so for this fixed pair (i,j) & (p, ②) we have Pijj.no (0)...... (z) * O almost surely. There are only finitely many distinct pairs in Ω. A finite union of measure-zero sets still has measure zero; hence, with probability one,
Pil.a # for all distinct (f,j), (p. e
(11)
By Definition 2, a decomposition Ω = U Qg (K 2 2) yields decoupled dynamics only if (V),;y(t), vowm pq (t)) 0 for all (i.j) S4 (p.g) E 24 with * and for all 2 0.
However, this already fails at = 0, since every cross-pair inner product is nonzero by (11). Thus, no such partition exists. Consequently, for L 2 3 and any observation set 92, the gradient flow dynamies are coupled almost surely under any absolutely continuous initialization.
B.1 COUPLED DYNAMICS EXAMPLE
B.1.1 DEPTII-2 MODEL
For shallow (L = 2) matrices, coupled dynamics typically correspond to connected observations under generic initialization, in accordance with Definitions I and 2 (the specific case of initialization, such as zero matrices, which leads to decoupled dynamics, will be further detailed a later subsection). We illustrate this principle with example where the observed entries form the first column of 2 x 2 matrix.
Consider 2 matrix, denoted Mc. which is to be completed using its first column as observations:
?
Mc
221 ?
The corresponding observation pattern matrix Pc
is:
Pc
The associated adjacency matrix Ac for the bipartite graph constructed as:
OO P P
0
0
Ac =
02.
0
0
0
which forms a connected graph as illustrated in Figure 1a. This setup leads coupled training dynamics under non-zero initialization. The coupling arises because parameters used to construct W11 and W2n overlap. Specifically, elements from the first column of matrix (i.e., d1 d21) are common to the computation of both Wn1 and W21 This shared dependency links the dynamics. The below illustration highlights these shared (teal) and distinct (red/blue) parameters involved in forming the observed entries wi1 and W21:
WiL
W12
Tai
012
b
b12
W22
a21
022
byz
anbii
+ a12b21
W21
dgوو1 + a22b21
The shared use of by and bạ in reconstructing both observed entries is what couples their learning dynamics.
20

---

Published as a conference paper at ICLR 2026
B.1.2 DEPTH2 3 MODEL
For deeper matrices (L 2 3), training dynamics are typically coupled, irrespective of the observation pattern (See Proposition B. Consider, for instance, predicting entries from the disconnected matrix MD where only diagonal elements are observed:
MD
W
11
? W22
Even with such observations, for L coupling arises because parameters intermediate layers are involved in computing multiple observed entries. This is illustrated in the following depth-3 example (W3:1 3:1 W1 W. W) Elements of the intermediate matrix W2 (colored teal) contribute to both the computation of W11 and u22:
W11
W12
(wi)n
(w) 12
(w2)11 (w1 (w2)21
(w2) 12
(ws) (wa)21
(wss 12
W21
W22
(w1)a
(w2) 22
(wa)
Specifically, the observed entries are formed as:
(wI)) (w) 12 12 (12)21 (wa)
W11
(w)) (wy (wi 12 (w2
(w3 21
U22
(w1)22 (W2) w )2z( (w3)12
((11))2 (12)) (wh 22 (2) )2 (w3)22.
The shared involvement of all elements from W2 (the teal matrix) in forming both and W22 leads to coupled dynamics, provided these elements are non-zero. (Conversely, if some elements were to become zero, this could potentially lead to decoupled dynamics, as illustrated in the subsequent subsection.)
B.2 DECOUPLED DYNAMICS EXAMPLE
B.2.1 DEPTH-2 MODEL
For depth-2 models, decoupled dynamics coincide with disconnected observation patterns. Indeed, by Lemma B.I,
(VeWij. v
Σ c Tee, (e) Ste,)
1=1
= (e, ex) 6jg + dip (e, W, Wie,),
where 8,s 1f and 0 otherwise. Hence, if i and j # p. the inner product is identically zero for all weights, which explains the decoupling for the depth-2 matrix when the observations are disconnected.
To illustrate the disconnected case, consider the x incomplete matrix example Mp, to be completed from diagonal-only observations.
wil
?
MD
W22
Then the observation matrix Pb can be constructed as:
Pb
and the adjacency matrix Ap can be constructed as:
02.2
PD
1
Ap
Pb
02.2
0
0
0
0
21

---

Published as a conference paper at ICLR 2026
which forms the disconnected graph as illustrated in Figure la. This setup inherently leads to decoupled training dynamics. The decoupling can visually understood by examining how distinct sets of clements in the factor matrices A and B contribute to the observed entries Wi1 and w22-- Specifically, as illustrated helow, red-colored entries are exclusively involved in predicting W11, while blue-colored entries exclusively involved in predicting w22- These sets of entries disjoint, confirming the decoupled nature of the dynamics:
W1
W12
an
012
bi
b12
W21
W/2
a21
02
b21
b22
W11
a
b1
012b21,
W22
021
b12
a22b22-
B.2.2 DEPTH 3 MODEL
For deep (L 2 3) matrices, decoupled training dynamics are observed in at least two key scenarios. First, as detailed in Appendix D.2.3, o.Ia initialization combined with block-diagonal observations leads to decoupled dynamics for any depth-factorized matrix.
To illustrate this for a deeper case, we revisit the MD observation pattern in depth-3 context. Appendix D.2.3 states that with such an initialization and observing only diagonal entries (which corresponds s case), all off off-diagonal elements of the factor matrices Wi(t) remain zero throughout training. Consequently, the factor matrices W1. W2, W3 are diagonal. The product matrix W 1: (4) thus formed as:
Wi1
W12
w 1
(w2)
(wa)11
W21
W22
)23
0
(w2)22
(wr3) 22
The observed entries are therefore computed as products of the respective diagonal elements:
Wii = (wi) (w2) (wg)
W/2 (w1)22 (22))) )22 (wz)22-
Since Wi1 depends only on the set of parameters {(W) ([W&)11 -ull and w22 depends only on the entirely disjoint set of parameters {(W))22)ks) k= their training dynamics are decoupled.
Second, the training dynamics are also decoupled when all factor matrices are initialized as d zero matrices, Oáxd- To see this, note that by the chain rule, we have
@wig (1) = (W-(() ((u((t))
(1)) (W (W) Wi))) jg
(12)
where we define the (it entry of the factor matrix (t( as (w)(t)) some time t all factor matrices satisfy W 0, then the right-hand side of (12) is the zero matrix, and thus
@wpg(t) (t) 0 for all p,
a (wo
Therefore,
0Φ a(w(()))
po 0. a(Wi(t))ss
Σ (P.))
(wpg(t) wpm)
which implies
Oib
(wi (wi(t)) (w)(()))
0.
Since the initial condition is (w)(0)); 0, uniqueness of ODE solutions guarantees that (wr(t)) = 0 for all & 2 O. As this holds for arbitrary I, j. we conclude that Wi(() = for all and all / 2 0.
Finally, because Wm (1) for all p, q and / 0, the inner product condition
(Vor) Why (t), Wpp))
is satisfied for all (i.g), (p, q) Ω and for all l 0. Hence, the dynamics (trivially) decoupled.
22

---

Published as a conference paper at ICLR 2026
C ADDITIONAL EXPERIMENTS
This section provides additional experiments omitted from the main text.
C.I IMPLICIT BIAS EXPERIMENTS
Connected Disconnected Observation Patterns. In Figure 1, present experiments with specific choices Mc and MD. which are x rank-1 ground-truth matrices illustrating connected and disconnected examples, respectively. To generalize these observations, we extended our experi- ments to a 3 X 3 rank-l ground truth matrix, considering all possible connected and disconnected observation patterns. After accounting for symmetries eliminate duplicates, this results in a total of 23 unique observation patterns, which are categorized into 17 connected and 6 disconnected cases.
For each of these 23 observation patterns, the 3 x rank-1 ground truth matrix was generated using constituent vectors whose entries were sampled from standard normal distribution. Each factor matrix was then initialized by sampling entries from a Gaussian distribution with a mean of zero and standard deviation of o.. We performed 10 independent trials for each pattern.
Figure 4 illustrates that, consistent with the findings in Figure 1, a significant discrepancy exists between the behavior of depth-2 matrices and that of deeper matrices. This discrepancy becomes notably more pronounced for the disconnected observation patterns.
Connected Observation
Disconnected Observation
Depth 2
1.-
Depth 3
Depth
Effective Rank
14-
1.2
16-1
16-1
Initial Scale
Initial Scale
Figure 4: The left panel shows the averaged effective rank of all possible connected patterns as a function of the initial scale a The right panel displays the averaged effective rank of all possible disconnected patterns.
Numerical Solutions of the Implicit Equations. We next provide a theoretical validation of our main claim: coupled dynamics induce a low-rank bias, whereas decoupled dynamics do not. This validation builds on Theorem 3.3, under various conditions, by numerically solving the equations while varying the ground truth value w", the number of blocks n, and the block size s. The results
shown in Figure 5 (for w* = 1, n = 5, s 2), Figure (for w² = 10, n = 10, 1), Figure 7 (for w² 0.1, n 10, 1), and Figure 8 (for w" 1, 3, evidence for the claim.
provide strong supporting
Gradient Descent Validation. Furthermore, we ran gradient descent with a sufficiently small step size to validate our derived equations. For the results shown in Figure 9, we replicated the setup of Figure (w" = 1, n 3, = 1), excluding the = 10-10 case due to prohibitive computation time. The observed values closely match the theoretical predictions from Theorem 3.3, as illustrated in Figure 8.
Comparison with Gaussian Initialization. To validate that our initialization scheme (7) can achieve comparable outcomes to Gaussian initialization while offering more control, we conducted experiments 3 3 x 3 matrix completion task with diagonal observations (i.e., w w wag 1). While our scheme allows initial rank properties to be adjusted via the parameter m, Gaussian initialization's 's inherent randomness precludes such direct control. Therefore, for comparison with
23

---

case. Conversely, when Published as a conference paper at ICLR 2026
m
m 100
m 1000
m 11
Singula Values
0.1
0.002
15:05
1o:07
1c:10
0.2
0.001
15
1c:07
1e
0.1
0.001
10:05
10:07
1c:00
0.1
0.001
1e:05
10:7
1e:10
at.
at
1-2.0
1- 3,0
L-5,
7.0
Figure 5: Singular values of W (oe) (numerically obtained from Theorem 3.3) against initialization
scale a for the block-diagonal observation task. Solid lines represent the largest singular value 01: dashed lines denote the identical singular values d; for j {n + 1,...,d df are all zero. For finite m, these results show that both greater depth L and smaller initial scale a strengthen the low-rank bias, in contrast the m is extremely large (e.g., m 10""), approximating an a a rank d initialization, the dynamics decouple and cannot achieve the minimal low-rank solution, regardless of L a.
n... Note that σ; for
1.0
Singular Values
0.1
0.001
16:05
16:07
16:10
0.002
16:55
20.07
1s:10
0.001
16:05
16/07
16:10
0.1
0.001
1e:05
1c:07
1610
a²
a'
a²
(=2.0
(=
L=5.0,
-
L
1.7700
(=
Figure 6: Numerical conditions identical to those in Figure 2, except with ground truth value w 10 and dimension d = 10 where the block size is 1.
100
1000
1e
Singular Values
0.6
0.2
0.1
0.
0.001
16:05
1c:07
16:30
0.1
0.001
06:05
16.07
I6:0
0.1
0.001
15:05
16:07
16-10
0.1
0.001
16:05
1c:07
16.10
a
L=2,
(=3,
L=5. O
L 7,0
1.2,0
(.3,
Lw5,
7.7.0
Figure Numerical conditions identical those in Figure 2, except with ground truth value w" 0.1 and dimension d = 10 where the block size is s = 1.
Gaussian initialization, we ran 1000 independent seeds and sorted the converged solutions by their rank. A comparison of the results in Figure 10 suggests that the behavioral trends may appear similar. In the depth-2 case, both initializations tend to converge to high-rank solutions. Morcover, for both initializations, clear gap emerges between L = 2 and stronger low-rank bias. For deeper networks (L 3), the tendency converge toward lower-rank solutions becomes increasingly pronounced as depth increases.
3, with the depth-3 model exhibiting a
Noisy Diagonal Experiments. We also experimented with observing noisy diagonal entries using gradient descent. In particular, instead of fixing all ground truth diagonal entries to be equal, we perturbed them (W*), w² + eg, where 6; (0,0²)) We set (w" 1), dimension (d 5),
24

---

Published as a conference paper at ICLR 2026
m 10000
m 1000
m -ee+
Singuiar Values
0.9
1c:10
0.001 1e:05
at
a+
at
a²
1 (m3.0
Lm5.0
7,7,0
L=3,0
L=7.0,
Figure 8: Numerical conditions identical to those in Figure 2, except with ground truth value w" = and dimension d 3 where the block size
10
100
m = 1000
m = le 10
Singular Values
0.1
0.001
1e-05
16-07
0.1
0.001
1e-05
1e-07
0.1
0.001
1e-05
1e:07
0.1
0.001
1e-05
1e-07
a²
a²
1-2.0,
2-3,0
1.7,0
1=2,0
2-3,
C- 5,
7,
Figure 9: Gradient descent experiments conducted under conditions identical those in Figure 8.
3.00
2.75
2.75
L
Effecitve Rank
Effecitve Rank
2.50
2.25
7
2.00
1.75
1.75
1.50
1.25
1:25
1.00
1.00
10 100
1000
200
400
600
800
1000
Sorted Index
(a) Results from Initialization using (7).
(b) Results from Gaussian initialization.
Figure 10: (a) Effective rank for the initialization scheme in (7). The x-axis denotes the parameter m, which controls the initial rank characteristics of the model, while the y-axis represents the corre- sponding effective rank after convergence. (b) Effective rank distributions for Gaussian initialization. The results are from 1000 independent trials, sorted by their converged effective rank. The x-axis denotes the sorted trial index (from lowest to highest converged rank), and the y-axis represents the corresponding effective rank after convergence.
and used the initialization scheme (7) with 100. For each configuration, we independently sampled 10 noise realizations and report the average behavior along with the standard deviations.
As shown in Figure 11, the qualitative trends are consistent with our theory. When = 2, the model converges to high-rank solution largely independently of the initialization scale, whereas for deeper networks the stable rank decreases as depth increases, indicating a stronger low-rank bias. We also observe that larger noise levels lead to more pronounced low-rank behavior. This is natural, since increasing the noise drives the ground truth further away from the identity. Moreover, the dependence on the noise magnitude appears continuous: in the small noise regime (leftmost panel), the change in stable rank is relatively mild, while in the larger noise regime (rightmost panel), the gap
25

---

Published as a conference paper at ICLR 2026
becomes more substantial. These experiments suggest that our depth-induced induced low-rank phenomenon empirically robust to moderate perturbations of the diagonal entries.
noley
Stable Rank
3.5
2 noiseless
noiseless
2.5
6 noisy
noisy
6-7 noiseless
1.
0"
Figure 11: Limiting stable rank (y-axis) as function of (x-axis) under noisy diagonal observations. Dashed lines indicate the noiseless baseline, and solid lines indicate the noisy case. The noise standard deviation set to a 0.01 (leftmost), a = 0.05 (middle), and a = 0.1 (rightmost). The depth dependent low-rank bias persists and follows trend similar to the noiseless setting.
Non-equal Diagonal Experiments. We also experimented with observing non-equal diagonal entries using gradient descent. In particular, instead of fixing all ground truth diagonal entries to be equal, we assigned different values each diagonal entry. We the dimension to d 5 and take the diagonal entries of W to be 0, 0.5, 1, 1.5, 2, respectively, and used the initialization scheme (7).
As shown in Figure 12, the qualitative trends are consistent with our theory. When model converges to high rank solution independently of the initialization scale, whereas for deeper networks the stable rank decreases as depth increases. For the case m = oc (rightmost plot), all models converge to high rank solutions regardless of depth, which is consistent with Theorem 3.3.
2, the
m 10
- 100
m 1000
0.3
0.01
3.003 0.1
1m-25
10-07
2.03
0.201
1e 05
16:77
0.2
3.03
5.001
16-05
10-07
0.3
0.01
0.005
10:55
10-07
0
0²
g²
Figure 12: Limiting stable rank (y-axis) as function of the initialization scale (x-axis) under non- equal diagonal observations. The low-rank bias induced by coupled training dynamics persists and closely matches the behavior in the equal- -iul-diagonal setting described in Theorem 3.3 and Figure 2.
Additional Optimizer Ablations. We also experimented with other optimizers, including adaptive methods, such as stochastic gradient descent (SGD), gradient descent with momentum, Adam, RM- SProp, and Adagrad. In this experiment, we fix the dimension to d = 5, use Gaussian initialization with diagonal observations (s 1) with w* 1, and run gradient based optimization with suffi- ciently small step size over 10 random seeds. For each optimizer, we use the default hyperparameters from the PyTorch implementation, and for SGD we update the model using one observed entry per iteration.
The results Figures 13-17 align well with our theory: for depth-2 (which induces decoupled dynamics), the model converges to high-rank solutions across initialization scales, whereas for depth L 2 3 (which induces coupled dynamics) the solutions become increasingly low-rank as the initialization scale decreases and as depth increases.
26

---

Published as a conference paper at ICLR 2026
0t 0.001
af 1e-05
1
1.4
12
Depth
Depth
Depth
Figure 13: Final stable rank as a function of depth. Each panel corresponds to different initialization scale. Results are obtained using SGD.
144
Depth (4)
Dopth (1)
Depth
Figure 14: Final stable rank function of depth. Each panel corresponds to different initialization scale. Results are obtained using GD with momentum.
0.001
1e 05
2.50
Stable Bank
2.35
2.00
1.25
1.0
Depth
Figure 15: Final stable rank as a function of depth. Each panel corresponds to different initialization scale. Results are obtained using Adam.
0.0 001
2e
2.75
2.57
2.25
Stadee
1.5-
Depth (C)
Depth (C)
Depth 011
Figure 16: Final stable rank function of depth. Each panel corresponds to different initialization scale. Results are obtained using RMSProp.
C.1.1 EXPERIMENTS NEURAL NETWORKS
To study how depth influences low rank bias in practice, we train ResNet and VGG models across varying depths. While Huh et al. (2021) show that deeper networks yield lower rank embeddings,
27

---

Published as a conference paper at ICLR 2026
0" 0.001
1e-d5
1e:07
2.90
2.23
Sab
125-
Depth
Figure 17: Final stable rank as a function of depth. Each panel corresponds to different initialization scale. Results are obtained using Adagrad.
their analysis does not address the weight matrices. Following Galanti et al. (2023), we measure the effective rank of the weight matrices directly and find that deeper networks are biased toward low-rank solutions.
To be more specific, we train ResNet-18, 34, 50, and 101, as well as VGG-I1, 13, 16, and 19, on CIFAR-10 and CIFAR-100 for 200 epochs with batch size of 1 128. Training uses SGD with momentum, Adam, and RMSProp. The initial learning rates are 0.1 for SGD with momentum, and 0.001 for Adam and RMSProp. We apply weight decay of 0.0005 for SGD with momentum and le-05 for Adam and RMSProp. A cosine annealing scheduler is used together with standard data augmentation (horizontal flipping and random cropping).
We measure the effective rank across all layers except the final one and average them to obtain single scalar. Following Galanti et al. (2023), each weight tensor Z R& X x x of convolutional layer, where Gn and Cout. denote the numbers of input and output channels and (kı, k2) is the kernel size, is reshaped into a matrix W E RR k k2) to measure the layer's effective rank. We report averages over five runs with 95% confidence intervals.
The results in Figures 18 to 21 for SGD with momentum, Figures 22 to 25 for Adam, and Figures 26 29 for RMSProp consistently show that the average effective rank decreases as depth increases. This trend is consistent with Theorem 3.3, which establishes the depth induced low-rank bias in matrix completion settings.
28

---

Published as a conference paper at ICLR 2026
100.0
Average Effective Rark.
2.00
ResNet: 18
94
Train Accuracy
97.5
Test Accuracy
1.50
ResNett-50
ResNet- 101
95.0
90
90.0
50
86
0
50
100 Epochs
150
200
0
50
100
150
200
50
100
150
200
Figure 18: We train CIFAR- 10 with ResNet models ranging from 18 to 101 layers using SGD with momentum, averaging results over five independent runs with 95% confidence intervals. The leftmost plot reports the training accuracy, the middle plot the test accuracy, and the rightmost plot the average effective rank. As depth increases, the average effective rank decreases.
100
80
Average Effective Rank
ResNict
2.0
ResNet:34
Test Accuracy
ResNet-55
1.5
ResNet: 101
75
60
Epochs
Epochs
Epochs
Figure 19:
The results for CIFAR-100 with ResNet-18 to 101, under the same conditions as in
Figure 18.
100
94
Average Effective Rank
Accuracy
98
120
VGG-13
VGG-16
90
1.00
VGG-19
Test
88
86
60
84
100 Epochs
200
150
150
200
Epochs
Epochs
Figure 20: The results for CIFAR-10 with VGG-11 to 19, under the same conditions as in Figure
100
Average Effective Rank
200
Test Accuracy
70
65
VGG-11
60
1.0
VGG-13
75
VGG-16
70
55
50
VGG-19
100
200
150
150
200
Epochs
Epochs
Epochs
Figure The results for CIFAR-100 with VGG-11 19, under the same conditions as Figure
29

---

Published as a conference paper at ICLR 2026
100
94
Average Effective Rank
200
ResNet-11
ResNet-34
Yest Accuracy
02
1.50
ResNet-50
ResNet-100 101
100
88
50
86
50
100 Epochs
150
200
0
50
100
150
200
50
100 Epochs
150
200
Figure 22: We train CIFAR-10 with ResNet models ranging from 18 to 101 layers using Adam, averaging results over five independent runs with 95% confidence intervals. The leftmost plot reports the training accuracy, the middle plot the test accuracy, and the rightmost plot the average effective rank. As depth increases, the average effective rank decreases.
100
Average Effective Rank
ResNet: 18
Accuracy
RcsNet-50 50
1.50
ResNet- 101
70
Test
85
68
66
50
64
Epochs
Epochs
Epochs
Figure 23:
The results for CIFAR-100 with ResNet-18 to 101, under the same conditions as in
Figure 22.
100.0
VGG-11
97.5
VGG-13
Accuracy
95.0
Test Accuracy
90-
Effective
150
VGG-19
VGG-16
92.5
88
90.0
100
87.5
86
50
200
150
200
150
200
Epechs
Epochs
Epochs
Figure 24: The results for CIFAR-10 with VGG-11 to 19, under same conditions as in Figure 22.
100
250
Average Effective Rank
VGG-11
Accuracy
95
lest Accuracy
68
200
VGG-13
VGG-16
66
64
100
60
50
58
100
200
50
150
150
200
Epochs
Epochs
Epochs
Figure The results for CIFAR-100 with VGG-11 19, under the same conditions as Figure 22.
30

---

Published as a conference paper at ICLR 2026
100
fwerage Lifective Rank
ResNet-11
80
ResNet-34
Test Accuracy
90
ResNet-50
88
60
ResNet-100 101
40
85
20
82
50
100
150
200
50
100
150
200
0
50
100
150
200
Epachs
Figure 26: We train CIFAR- 10 with ResNet models ranging from 18 to 101 layers using RMSProp. averaging results over five independent runs with 95% confidence intervals. The leftmost plot reports the training accuracy, the middle plot the test accuracy, and the rightmost plot the average effective rank. As depth increases, the average effective rank decreases.
100
67.5
Effective Rank
ResNet-16
ResNet-34
Accuracy
Tet Accuracy
ResNet-50
62.5
60
ResNet: 101
60.0
40
57.5
70
20
150
Epochs
Epochs
Epochs
Figure 27:
The results for CIFAR-100 with ResNet-18 to 101, under the same conditions as in
Figure 26.
100.0
200
VGG-11
97.5
VGG-13
Accuracy
95.0
Test Accuracy
90-
Average Effective
15.-
VGG-19
VGG-16
92.5
88
100
90.0
87.5
86
50
200
150
200
150
200
Epechs
Epochs
Epochs
Figure 28: The results for CIFAR-10 with VGG-11 to 19, under same conditions as in Figure 26.
70
Average Effective Rank
VGG-11
200
VGG-13
Accuracy
Accuracy
65
90
VGG-19
85
100
55
100
150
50
150
200
100
150
200
Epochs
Epochs
Epochs
Figure The results for CIFAR-100 with VGG-11 19, under the same conditions as Figure 26.
31

---

Published as a conference paper at ICLR 2026
Coupled vs. Decoupled Dynamics in NN. To examine whether coupled and decoupled training dynamics intensify low-rank bias in practical neural networks, we conducted an additional experiment with fully connected networks with ReLU activations, under both Gaussian and identity- based initializations, using the CIFAR-10 dataset. For the Gaussian initialization, all layers are initialized with i.i.d. Gaussian weights. For the identity-based initialization, all hidden layers are initialized as scaled identity matrices, while the first and last layers are initialized with Gaussian weights, since these layers are not square.
We train networks of depth L E (2,3,5) with a fixed hidden width of 512 for 100 epochs, using SGD with momentum and constant learning rate of 0.01. The results show that, even when both initializations successfully achieve low training loss, the low-rank bias is substantially stronger under Gaussian initialization compared to identity initialization, which indicates that low-rank bias intensified under coupled training dynamics in way that is consistent with our theoretical findings.
Furthermore, as depth increases, the stable rank of the weight matrices decreases under Gaussian initialization. In contrast, with identity-based initialization, deeper networks tend to converge to higher rank solutions. plausible explanation is that, as depth grows, a larger fraction of the layers are initialized using identity (recall that the first and last layers are initialized under Gaussian), which makes the overall dynamics closer to decoupled regime and therefore less biased toward low-rank solutions.
10a
175
10-4
125
Gaussian
10-2
-X- [- 2-22 Identity
100
+ L 3. Gaussian
10-3
-X 1=3, 3, Identity
5, Goussian
50
10-1
-x- L-5, Identity
25-
20
40
60
60
100
20
40
60
80
100
Epoch
Eporh
Figure 30: Left: training loss (log scale). Right: average stable rank across all layers except the last. Solid lines correspond to Gaussian initialization and dashed lines to identity- initialization. Gaussian initialization (corresponding to coupled training dynamics) converges to noticeably lower rank than identity-based initialization (corresponding to more decoupled training dynamics).
C.2 LOSS OF PLASTICITY EXPERIMENTS
Section 4.2 discusses a scenario where pre-training employs diagonal entries, after which an off- diagonal term (specifically, w(p) is introduced to restore connectivity, leading coupled dynamics. Theorem 4.2 establishes that, in this situation, the model indeed does not converge to a low-rank solution. To empirically validate this theoretical finding, we conducted experiments using the family of initializations (7) tailored to this specific scenario, with results detailed in Figures 31 and 32. These experiments utilized depth-2 model reconstruct the ground-truth matrix, with initialization scale set to a 10 35 Notably, if the initialization scale is set significantly lower, as the dynamics are coupled, cold-started model can converge to solutions exhibiting a more pronounced low-rank structure.
For the case presented Figure 31, where w" wi2 0.1, following Theorem 4.2, the theoretical lower bound on the stable rank for warm-started model initialized diagonally (m oc) approximately 1.45, while the empirically observed stable rank is approximately 1.8. Even in scenarios where substantial new information must be learned (e.g., by setting wi, a large value), loss of plasticity empirically observed, primarily manifesting as high test error (i.e., a significant gap between the target w and the converged W21). While Theorem 4.2's analysis via stable rank does not fully explain an accompanying low-rank bias (a point consistent with Figure 32), the theorem does predict that W21 converges to negative value, which implies large test loss.
32

---

Published as a conference paper at ICLR 2026
Furthermore, performed additional experiments with different diagonal entry values to investigate whether this argument extends to other scenarios (results shown in Figure 33), although specific theoretical guarantees have not been established for these broader cases. We observe that even in these varied settings, both the effective rank and the stable rank of a warm-started model substantially exceed one, whereas cold-started models can converge lower-rank solutions.
2.0
2.0
Effective Rank
1.8
Stable Rank
1.8
1.6
Pre-train
Warm-starting Cold starting
1.4
1.2
1-
1.0
10
100
1000
10000
100
1000
10000
10 100
1000
10000
m
m
Figure 31: Experimental results for 2 rank-1 ground-truth matrix W with wi1 WWy and w12 0.5 (implying Wai = 2 for rank- I structure). Models, initialized according to (7), are first pre-trained on diagonal entries. After achieving zero-loss convergence pre-training, the off-diagonal element wi2 is introduced, and models are subsequently trained combined diagonal and off-diagonal observations. The plots display: (Left and Middle) effective rank under different settings; (Right) converged value of W21 (oc). Key observations: (I) Warm-starting with model that converged to a high-rank solution during pre-training tends to maintain this high rank, even when presented with the same subsequent observations cold-started model. (2) In the theoretically analyzed m oc case, W21 (20) < 0 observed, which correlates with the highest effective rank.
0.1
Effective Rank
Stable Rank
1.6
1.6
0.0
Pre-train
-0.1
Warm-starte
1.4
1.4
Cold-starting
-02
1.2
1.2
1.0
1.0
-044
100
1000
10000
10
100
1000
10000
10
100
1000
10000
Figure 32: Experimental conditions identical to those in Figure 31, except with ground truth value
W = 10. The model have to predict wa1 as 0.1
18
Effective Rank
Stable Rank
1.6
Pre-train
Warm-starting Cold-starting
1.2
1.05
1
1.00
10
100
1000
20000
100
2000
10000
100
1000
10000
Figure 33: Experimental conditions identical to those in Figure 31, except with ground truth value Wi1 1, why = 2, and wi, = 0.5. The model have to predict wy as 4.
33

---

Published as a conference paper at ICLR 2026
D PROOF FOR SECTION 3
In this and the following sections, we prove the Propositions and Theorems presented in the main text. We begin with the proof of Theorem 3.1.
D.1 PROOF FOR THEOREM 3.1
When convergence is guaranteed, we can define the reference vector u' is entirely determined by their initial values and the targets. Note that u² does not change with time, since it is defined at = oc. We decompose aı (t), az(t). and (t( into two components: one parallel to u" and one perpendicular to u²:
(o())) [
R4, which
a (t) = 01j (t) + at)(t),
(t = (t) + 02 (t), b1(t) = (t) + bi_ (t).
For any vector Rd the parallel component is defined fn (u" u)) wz,, and the perpendicular component as 4y = u - U|-.
We introduce notation to quantify the alignment of each vector with u=:
Oaa u a1 (t), Gaz (1) u² az(t), ob, (1) u by (t).
(13)
Additionally, we define notation to measure the magnitude of the perpendicular components:
Bas (f) |a (t)1 Then, using equation (4), time evolution of each component equation (13) can be written as:
(t) ||a (t)li2, 8e (t) | (t)
(14)
aaa (t) = u di(t)
(wi ai (t)bi(t) u² b bi(t)
(n(()
(t(t)ab, (t).
(15)
Likewise, for Cas (t), we derive:
das (t) u² di(() TTatt(t)
= (wi) (t)br(tt) n² bi
(r,(t)
(t)aab, (t).
(16)
Finally, for Obb, (!), have:
àb, (t) u b()()
(wii aj (1)b. (1))²² ai(1) + (w a (1)bi as(() Ta2()) = =ri(t)aa (1) + r2(t)aa, (t).
(17)
Also, for the perpendicular components, their time evolution can be derived as:
Ba (t) 20, (t) á1 it)(
= 2a (t) (a1(t) (u- (utaai aa(((t)))) ui dt
d
20 (1)) (1)())b1()) -ra(1) (u:bb(() ur))
Noting that ai (f) is perpendicular to u", the second term in the parenthesis is zero. Thus, we have
Ba. (t) = (t)(t)a1 (t) b (t).
Likewise, for Bag (t) and (b, (t), derive their time derivative as:
Bag (t) 2rz(t) b (t), 3o, (t) Ba, (t) Bag (t).
34

---

Published as a conference paper at ICLR 2026
Note that by the definition of u", have Bb, (oo) 0. Integrating the identity Bo, (t) Ba. (t)
Bag (t) from 0 to gives:
Sa, (Oc) + Baz (00) = Bas (0) + Sa, (0) - 3o, (0)
P020
This equation shows that the initial value 30 is small, it constrains the total perpendicular magnitude at convergence. However, since we do not know u² in advance, one natural way ensure small perpendicular components is to initialize the entire norms of a1 (0), (0 to be sufficiently small.
To develop more rigorous understanding, analyze the parallel components. Under the assumption of convergence, we have:
a) bi wii.
(oo) = w21-
Decomposing a1 (oo) and (2(()) leads to:
a1 (00) b1()))= = (00) + u² (((()))) (i())) = Oaj (0o) (oo) =iil
(18)
a2(00))bi((0) 61(0) = (221 + u az(=))))) T 6i()
aa (00) (00) wÿ1.
(19)
Using equations (15)-(17), and noting that
d o (1)
d (a (1) aā (1))
d
dt
we can integrate both sides of the equation over time from 0 to oo to obtain:
a (0o) adg (oo) og (oo)
a a (0) ag (0)
(20)
By solving equations (18), (19), and (20), we can obtain closed-form solutions of caj (xc), aaz (OC), and ab, (oo) as follows:
2wi1 2
2w 2
a² (00)
a² (00)
(21)
Vag + 4w + Aw²i
V vag + 4w 2 + Aw 2
000
a (0c) Vaß + 4w0 2 + 4w 2 co
(22)
Thus, we can upper bound the proportion of the perpendicular component a (00) and az( relative to its total magnitude as follows:
Bo (Vaă + 4ww + 4ws ax
| (a11(00)) (20)
Ban (00)
$
||a (20) a. (00) + Ba, (00)
2w 2
Bo (Vag + 4w + 4w
aa | (10)
(20) 2 (20) Bag (00)
000
aa2 (2))
2w2 2
To further refine these bounds, analyze the terms Bo and ((ao) ع
/
|a 4w 2 4w 2 ag-
By the definition of Su. it is upper bounded by [|a] + (0)]2 = ||A())) Also, by the definition of ogg, we have:
- (0)11 < on V A(0))
35

---

Published as a conference paper at ICLR 2026
Noting that the function f(x) V=² + - x (where C > 0) is non-negative and monotonically decreasing for all E R, we can upper bound S(av) using the lower bound of on:
S(ao) - S ((-1)b,0))13)
V ( [b. 4(11 + = 2) ())
1 (0) + 4("i + w 9) + (0)
Substituting these bounds for Sn and S(oo) into the inequality ||a (oo)) (,5(o final upper bound for the proportion of the perpendicular component of (oo):
[|a, (oc)ll S 2w
we obtain the
(VIb + 4(wf) + wb 2) + |6 (0)
|A(O))
|| 1 (1o) |a (2o)
2wii
A similar bound applies to
1003 (00))
[0)]
(IIb( + 4(wii + wh1 2) + [ (0)
|a21 (00)112
la (02()0
2w2
36

---

Published as a conference paper at ICLR 2026
D.2 PROOF FOR PROPOSITION 3.2
According to the definition of coupled/decoupled dynamics presented in Definition 2, for the family of initializations defined in (7) along with the block-diagonal observations
block
U b=1
S4 - [(i,)) :.j E {(b- 1)s + 1, bs,...,b))).
so that sn and each 94 corresponds to the index set of the b-th s s diagonal block. We divide the cases to ensure that all possible scenarios for this family of initializations are covered.
The derivative of block-diagonal observed entry (ty E D with respect to (Wi(t)) is:
@Wpg (t) (WL(t)/L-(()) Wi+1(t)) (Wi-1(t) (t) Wi(t)) a(Wi(t))s
(23)
where the first term is (p, i)-th element of the product W ( term is (j, q)-th element of the product W[-1(t)Wiize) gradient of (ty(t) with respect ((t), which is the concatenation of all trainable parameters as follows:
L-) (t) and the second (t( Then, we can express the
@wp(() (1)
ppp
V Vowp (1) = ((W2(t)) ((W2(1))22
(24)
(WW)()))
D.2.1
CASE FOR =
For L = 2, each observed entry can be written as Why (t) (W (Wa.B(t)), = ax(t) (t).
where (t) is the i-th row of A(t) and (t) is the j-th column of Bt)).
Let (t) be the vector obtained by stacking all entries of A(t) and B(t). The gradient of wty(t) with respect to (()) is given by
( = ((V wig(() (v,, uny (1){))))))
where Va. and Vb, denote derivatives with respect to the row vector (t) and column vector by (t). respectively. Since wiy(t) (t) by we have
Va. (t) = { by (1), r= 0 r
Vo,w = {o,
(t)...
8 =-j. j.
8 *
Thus V Vow; (t) has nonzero components only in the coordinates corresponding to at and (t), and all other coordinates are identically zero.
Now fix two observed indices (i.j) Og and (p. 4) Gy with b b b'. By the definition of 2g we have
j.j E {(b - 1)s + 1, 1...., bs}, P.q E ((b' - 1)s + 1....,
and these index sets are disjoint. Therefore the supports of the two gradient vectors are disjoint. Hence, for all 2
(Vewi (t), Vew (1) O.
Therefore, the gradient flow dynamics are decoupled with respect to the partition {9.) in the sense of Definition 2.
D.2.2 CASE FOR I. 2 3 AND 1 < < 00
For the deeper matrix (L 3) with
oo, every entry of each weight matrix Wi(0) (for
1= 1,.,, L) is initialized to be positive. Then, for any (i.j) E (=.n) the entry wiy (0) a sum of products of these positive entries, (0) O.
Evaluating (23) and (24) at t = 0, the derivative of wiy with respect to any parameter in is given by the product of entries derived in (23). Since T. > there exists least one intermediate
37

---

Published as a conference paper at ICLR 2026
layer .2....,L- L
1}. For these intermediate layers, both products in the derivative formula products of matrices with strictly positive entries. Consequently, for < L, every coordinate of W,, (0) is strictly positive. For the boundary layers (1 = 1 and I L), the derivatives non-negative.
Because the gradient contains strictly positive sub-vector (corresponding the intermediate layers) and is non-negative everywhere else, for any two distinct observed indices (i,j) and (p, q), their inner product satisfies:
(V (Vow (0), Ve Wy (0) 0.
This shows that there no partition of 2 (a,n) for which the cross-block inner products in (6) vanish = 0, by Definition the gradient flow dynamics are coupled for L 2 3 and 1 < Co.
D.2.3 CASE FOR L 2 3 AND = 00
For a, b E R, define (a,, b) A (a b)I, + bJ,. Lemma D.2 shows that the family
M (I, & ((a + (Jn I,) cJ..
closed under scalar multiplication, addition, and matrix multiplication, and that any matrices in M commute. As a consequence, Lemma D.3 implies that the factor matrices (0) are initialized to lie in M, then under the gradient flow dynamics in (3), each Wi(() remains in the family M for all t 2 0 and all E [L].
The case 0 follows immediately as a special case. When c 0, the family reduces
Mo _ [ @ b)),
and all closure, commutativity. and invariance properties follow directly from Lemma D.I together with the Kronecker product identity.
We now focus on the setting m oc. In this case, Wi(t) lies in Mo for all 2 and all e [L]. Since every matrix in Mo is block-diagonal with identical s x blocks, it follows that both products WL (1) ... W (t) and Wt) .... W1 inherit the same alock-diagonal structure. As a result, for a prediction Wpg (t) where (p,g) ②) S4 the partial derivative in (23) is nonzero only if the parameter indices and j also belong to the same block (i.e., (i,j) E Qq).
Now fix two observed indices (i.j) E 24 and (p, 9) Gar with + b'. Since the corresponding index sets are disjoint, the supports of Vowng (1) and Vowp(()) (4) are disjoint as well. Consequently, for all 2 0,
Vown (t), We (t)) =
This verifies that the gradient flow dynamics are decoupled with respect to the partition {24) in the sense of Definition
38

---

Published as a conference paper at ICLR 2026
D.3 PROOF FOR THEOREM 3.3
For a, c E R, define
D b) = (a - b) + bJs,
O cJ,:
where I, is the x 8 identity matrix and Jg is the x all-ones matrix. Consider the d x block matrix
(a, b) In D(a (J, I,,)
(cc)
((a.b)
O(c)
0(c)
O(c) D(a
O
E RRdxd
(
(
((a
which is block matrix with blocks. Define
M (M(a, b) a, bR).
We now state a lemma that captures the key algebraic features of this family. Lemma D.I. Let In denote the n x n identity malrix and Jn 2 1, 1 denote the n x n matrix with all entries equal Then the set
S = (aI, + bJn a. E R) is closed under scalar multiplication, addition, and matrix multiplication. Also, any two matrices A, B E commute.
Proof. Let
A = aIn + bJ, and B= dJn.
with a, b, d R. and let X R be an arbitrary scalar. Scalar Multiplication.
X4 A(aI, bJ,) (Aa) + (Ab)J
Since Aa, Ab R, follows that AAES.
Addition.
A+ + bJ,)) + (cI, + dJ,))
Since a + c, b- --de IR, we have A + B S.
Matrix Multiplication.
AB (aIn + bJJ, + dJn).
Using the distributive property and the facts that
Jn Jn Jn and J² = nJnn
we expand:
AB = ac I, In + ad InJn + bc Jn In + bd J2
ac + ad Jn + bc bd (nJn) ac In + (ad be + nbdl) Thus, AB is of the form al, + JJ, with ac and ad + be nbd, and hence AB S. Commutativity. By the same procedure above,
AB (aIn + bJn (cI, + dJn) acI, + (ad be nbd)J
cal, + (eb + da + ndb)J
BA,
which completes the proof.
39

---

Published as a conference paper at ICLR 2026
Lemma D.2. The set M is closed under scalar multiplication and addition, and it is also closed under matrix multiplication. Moreover, for any (a1, b1, 01) and (az, b2, 02), the matrices M (a1, b1, 01) and M (a2, C2) commute.
Proof. Note that by Lemma D.I, ( b) is closed under scalar multiplication, addition, and matrix multiplication. Since J, also closed under these operations, the same holds for (c)).
Scalar multiplication. For any scalar À E R,
(a, b, c) > I @ b) I,) O
I, 6 (aD b)) + (Jn - I,) @ (xO(c)
In D Ab) (J, ((Ac)
= -Maa, Ab, de) E M.
Addition. For any (a1, (1) and (az, b2, 02),
M(a (1) M ( 02) = In 3 D(a b1) + (Jn I,) s 0(a)]
+ [In & D(2,, b2) (Jn I,) 8 0 ((2)
In @ (D(a), bb) + D(a2, b2)) + (J, - I.) @ (0(c))
J, 3 D(1 + 02 b + b2) + (Jn I,,) @
((, 02 bi + b + +2) M.
Matrix multiplication. First observe that
b2) = D(aia2 + (s 1)b1d2, aib + azbi + (s 2)b1b2),
(cc))((e)) = O(sci02),
D(a,b)O(e = Occ)D(a,b) = O + (s 1)be).
Multiplying M ((a,, ci) and M (a2, b2, 02) gives
T T3
T3
T2 T
T2
M(a, (a),) =
T3
T2
T.
where
T bi b2) (n (1))))(()0(()),
Tb = (c) (a2, ((22, b2) + D(a,, b, (1-))
+
Using the identities above, we can rewrite Ti and T2
T D(a102 + (s 1)b2b2, aidı + azhi + (s 2)b,b2) + 0 1)sc1e2)
( + (s - 1)b2b2 + (n - 1)sccc, b1bb + azج, + (s- 2)b, by + (n 1)sc122),
T2 ( + +s 1)62c) + (s ( 2))
= (a100 a2 (s 1) 12 (s 1)b2c1 + (n 2) 22)
Hence Ma1, b1, e1 (a2, b2, 02) again has the same block structure as M (....), so M is closed under matrix multiplication.
Commutativity. The expressions for Ti and T2 above are symmetric in (a1, 61, e1) and (a2, b2, c2). In partieular, if we interchange (a1, bi, (1) and (az, by, 02) in the formulas for T; and T2. we obtain the same matrices. Therefore
(a by, e1 M (az, b2, = M b2, e2 M br, b1,
and the matrices in M commute pairwise.
Using the above lemma, we show that if all factor matrices W, are initialized according to (7), then (t( stays in M for every > 0.
40

---

Published as a conference paper at ICLR 2026
Lemma D.3. Let s,n E N and set d Consider a ground truth matrix W E B with observation set block with all observed entries sharing the same positive value w > O. Consider the product matrix Waa, where the factor matrices W1 E & are initialized according to (7). Under the gradient flow dynumics (3), Wi(l) remains in the family M for all i 2 0 and all I E [L].
Proof. First note that the initialization in (7) belongs to the family M, since each factor is of the form
Wi( = (a, a/m, a/m), LE [L].
We will show that M is invariant under the gradient flow.
Fix any time t 2 and assume that Wi(t) E .M for all E [L]. By Lemma D.2, M is closed under matrix multiplication and every matrix in .M is symmetric by construction, it is also closed under transpose. Hence the product matrix Wr.1 = (t.()).. (..t)..Wi(())) (I))) lies M. In particular, there exist scalars A, B, C E R such that W (L) = (A, (,, C).
By the definition of the observation set and the assumption that all observed entries share the same ground-truth value w", the loss has the form
(( 2
Σ (ineen
((Waa) - w²)²
Since S2 contains exactly the entries inside each diagonal block, and (z) = ( A, B, C) has diagonal blocks with diagonal entries A and off-diagonal entries B, a direct computation gives
(e() = M(A - w", B w²,0) E M.
The gradient flow dynamics for each factor matrix are
II wi(t) (e(WLa(t) 1i=++
[I vill
Wi(t)
EE [L].
Each factor in the products the right-hand side belongs to M, and by Lemma D.2 the product of matrices in M remains in M. Since (e(Wi.(tt) E .M as well, it follows that
Wi(t) M for all [ [Z].
Since the initial condition satisfies (i(0) E M for all / E [L], we conclude that
Wi(() E M for all t 2 0,/ E [4].
Beyond showing that every factor matrix remains in the family M, further establish that all layers evolve identically with below lemma:
Lemma D.4. Under the setting of Lemma D.3,
WL(t)
W1((t)
holds for all 2 O.
Proof. By Lemma D.3 and Lemma D.2, we know that for all t 2 0 and all e: [L] we have Wi(t) M,, and that matrices in M are closed under matrix multiplication, transpose, and commute pairwise. Moreover, as shown the proof of Lemma D.3, the loss gradient Ve(WL.(tt) also lies in M.
Fix any time t and suppose that Wr(1)= Wr.1(()) .... =(()=
Then the product matrix satisfies WLa(!) = (11) and the gradient flow dynamics for each layer can be written as
wi(t) - I wi(0) Ve(WL.(() (I
v=++1
-- U(N-
41

---

Published as a conference paper at ICLR 2026
Since (t) and (t)((t))) both lie in M and matrices in M commute pairwise, reorder the factors to obtain
(t( - - (U(t)* U(t)--1 for all / E [L].
Thus, whenever (t) = (z(t) holds at some time t, the time derivatives of all layers coincide that time:
(1.()) = W (() = =.. Wi(().
By the initialization scheme (7) we have
(0 W =... Wi(0).
Since the gradient flow admits unique solution for this initial condition, it follows that the equalities between the layers are preserved for all times 2 0, that is,
(I.()) WI-1 (1)
(i()) for all / 0.
Using the lemma above, we can parameterize every factor matrix (t (a(t), b(t), e(t)) for all / E [L]. where (a(t), b(t), c(t)) are shared coefficients. Likewise, we write the product matrix as WL.1 (t) = M (A(t,, B(t), C(t)). . We now derive the eigenvalues each factor matrix.
Lemma D.5. Let 8, E N and d = sn. For b, R. let M b, E Rdxd be the block matrix defined by
M b, In ((a,)) b) (Jn I.) 2 Oc)),
where ba, (a- b)I, bJ, and (cc) cJ.. The eigenvalues of M M b. and their corresponding multiplicitiies are:
\1 a (s 1)b snn 1)e with multiplicity 1,
A = a + (s 1)6 sc with multiplicity n - 1,
with multiplicity n(s 1).
Proof. First, we express M (a, b, c) in terms of Kronecker products of identity matrices I and all-ones
matrices J. Substituting the definitions of D and
M I, & ((a b) bJ.) (J, In) 2 (cJa)
= (a (I, @ I,) + b(I, @ J.) + c(J,, @ J.) c(I, @ J.) (a (I, (b- (I J.) ccJ, J.).
The matrix Jn has two distinct eigenvalues: n (corresponding to eigenvector 1,,) and 0 (corresponding the orthogonal complement 1, We construct the eigenbasis M using tensor products of the eigenvectors of Jn and J..
Case 1. Consider the eigenvector i1 1, 1., Since J,H, ni,, and J,H, sh,, have: Mv ((a b) + (b c)s + c(ns)) i1
= (a (s 1)b + s(n 1)c) 01.
This subspace has dimension x 1 = 1.
Case 2. Consider eigenvectors V2 = u 2 1,, where u E 1 C R". Here J. = 0 and J.1, = sl,,
Thus:
M = ((a (b- c(0 8)) U2
(a (s 1)6 sc) U2-
The dimension of 1 is 1, so the multiplicity is (n 1) x 1
w @ where E Rh is arbitrary and E 1 C Rt. Here J,z = 0. Consequently, any term containing J. the Kronecker product sends this vector to zero: (A (w & x) Aw 2 J.2 Aw 0==0. 0. Therefore, only the identity term remains:
Case 3. Consider eigenvectors v3
Mvs (a Ins U3 + 0 +
(a bi)33-
The dimension of Rd is and the dimension of 1 is 1. Thus, the multiplicity is nn(s 1). 1
42

---

Published as a conference paper at ICLR 2026
Lemma D.6. Let (x(t) for i {1,2,3) denote the eigenvalues of the factor matrix Wi(t) from Lemma D.5. Under gradient flow (3). the evolution of these eigenvalues governed by the following system of ODE:
(1) XF(1) + (n- (1)
sw") (
n
(t) (F(1)) + (n 1)2)(1) sw" (t))
n
Aa(1 -
Proof. Given that the factor matrices Wt(t) share the same form (Lemma D.4), let ), denote their eigenvalues. Consequently, the eigenvalues of the product matrix (t)(t)) are (t). Using Lemma D.5 to invert the eigenvalue relations, we can express the parameters of WL. ( B)C) as follows:
X + (n 1)13 n(s 1)
A
88
B Af + (n - 1)14 T
C X X1
sn
Recall from the proof of Lemma D.3 that the gradient takes the form (((WL.)) L.1)
M wt B
w", 0). Let % denote the eigenvalue of (((W),)) corresponding to the i-th eigenvalue class in
Lemma D.5. Note that for the gradient matrix, the off-diagonal block parameter zero (c = 0).
Consequently, the eigenvalues for 7/ and 72 coincide. Specifically:
71 (AA w²) + (s B) - -w) + - =))
(A w (s 1)(B -w"),
1/2 (A w²) + (s 13-11 1)(B - w²) - -0(
71: 33 (A w²) (B w*).
Substituting the expressions for A and B into the equations above yields %4 in terms of X/:
n1 72 A1 + (n - 1)21 + n(s 1)14 w0 + (s - 11) 11(())
[ + (n (1 1)2F
ni w
87
A + (n - 11)4/
sw.
A + (n 1)14 + n(s 1)14 w0) -(At At + (n 1) n1 w²
73
Sn
sn
A/-
Finally, recall that the gradient flow dynamics for each layer are governed by
I. II WiO!T Ve(WLa(()) (1)) j=1+1
1-1 II
Wi(1)
Since the weight matrices Wi(t) and the gradient matrix Ve(WL.n(t) belong to M, they are commutative and simultaneously diagonalizable. Let P E Bdxd be the sime-independent common orthogonal matrix such that Wi(t) = P and (t))aa)t) = PT(t) where A(t) and [(t) are diagonal matrices containing the eigenvalues (t) and (t). respectively.
Projecting the gradient flow dynamics onto the eigenspace spanned by the i-th eigenvector, we obtain the evolution of the eigenvalues. Using the fact that (i( (!( due to symmetry, the dynamics
43

---

Published as a conference paper at ICLR 2026
for the 1-th layer become:
Wi(t) PA(()P (PA(t)PT) '-1 (PI())PT) (PA(t)PT)) '-1 =-P (OTAAA))()) PT.
Multiplying by PT on the left and P on the right yields the diagonal evolution:
Á(t) ((t))
For each distinct eigenvalue index i E 33) the scalar dynamics simplify to:
(1) (1)
Substituting the values of 7/ derived previously, we obtain the specific evolution equations for each cigenvalue:
(t) (n10 (t) + (n--1)5(t)
sw"
n )e(t) (2( (t) (tf(t) + (n ()) )t( = (1) (11())11(() (1) = -133-1
=w" (t),
7
Building the lemma above, can identify conserved quantity that depends the depth.
Lemma D.7. Under the gradient flow dynamics defined in Lemma D.6, the eigenvalues x (t) and (t) satisfy the following conservation laws for all 2 0:
1. If L = 2, the ratio of the eigenvalues is conserved:
(t) A1
(2 A2
2. If 2 3, the difference of the negated powers is conserved:
X (t) X2 (t) X2 -() X2- (0).
Proof. From Lemma D.6, the scalar dynamics for the first two eigenvalues given by:
(,(1) (1) for {1,2],
where 7(t) = (x=0)=+(=-1)2a(111 sw". We consider the two cases based on the depth
Case 1: (L = 2). In this case, the dynamics simplify to ii(t) (t)) Rearranging the terms to separate variables, we have:
(t
>²(t)
(t),
(t))
(t)
A2(t)
Subtracting the second equation from the first eliminates (t):
log A (t)
log >x2(t) = O (t)
dt
dt d
log
0.
dt
Ax(()
This implies that the ratio (t)/A22) is constant in time.
Case 2: (I 2 3). Consider the time derivative of the quantity Q(1) -0) Applying
the chain rule:
d (2 (t)) = -2 - (--))x---(1 ii(t)
dt
-(2 (0)
=- 183-110 --t)-L)(t).
44

---

Published as a conference paper at ICLR 2026
Similarly, for the second term:
(x (0)
= (0))
(->(())))))
-2 L)(().
Subtracting the two derivatives yields:
_
d
()((t) (- L))(t) =
Since the time derivative is zero, the quantity is conserved throughout the training, proving the statement.
We are now ready prove Theorem 3.3.
Proof. Using the inverse relations from Lemma D.5, express the parameters A(t) and (t) in terms of the eigenvalues:
A(t) = (t) + (n (t) + n(s (t)
87
B(I) ( + (n- -
- -
Consider the difference between the parameters:
A(t) - B()) ms>a(() xz(
8n
Since loss converges to zero (Proposition D.I), implies global optimality, which requires A(xo) = B(xc) = Taking the limit + co, the difference vanishes, yielding:
\a(xc) = 0.
Next, substituting \o() 0 and A(x0) w² into the expression for A(t), obtain:
Xf( + (n (o0)
w²
sn
Multiplying by sn d, we arrive at the first constraint:
( + (n (o)) dw."
(25)
Let 01 2 02 ... aa denote the singular values of the limiting product matrix WL.1 (oo). Under our initialization scheme and Lemma F.3, the factor matrices remain positive definite, implying that the singular values of the product matrix coincide with the L-th power of the eigenvalues. Based on the multiplicities derived in Lemma D.5, we identify:
σ1 = X (Do), o, = X (po) for E (2...., n... o, = X/ = 0 for > n.
We now solve for the non-zero singular values by considering two cases based on the depth L. Case 1: (L = 2). For L = 2, Lemma D.7 states that the ratio of cigenvalues is preserved. Using the initialization values from (7) and Lemma D.5, this ratio is given by:
(o0) A1 m + d
(26)
A2( (0
Substituting {f(0o) = into (25) and combining it with the squared ratio from (26), we can solve for oj and og
w 2(mmmm + d - 1)
01 (m d 1)2 + (n () - 1)2
wº d(mmm - 1)2
0; (m d 1)2 + (n 1) 10--1 1)2
for all i (2.....n).
45

---

Published as a conference paper at ICLR 2026
Case 2: (L 2 3 and finite m). For L 2 3 with 1 < < co, Lemma D.7 ensures the conservation of the difference of negated powers:
x (00) -
= -
Substituting the initial eigenvalues from (7), the right-hand side becomes:
2-L ( -- (m 1001] -)
(oo) - - (00)
(m-
(27)
Finally, expressing the eigenvalues terms of singular values via X, = 0./² (implying X?- and combining (25) with (27), we obtain the system of implicit equations:
2-1.
2-k
2-L w= d 01 2
L,d
11
7-6 - 0;
2
(w' d (n
1)0,)
La,m,Ldd for all E (2....,
where Co,m, L.d A a 2-
(m + d 1)2-L (m -1)2-1
Case 3: (L 2 3 and m 0o). In this case, the initial eigenvalues of the factor matrices become:
1 d -
(0) lim a
a a.
m0000
m
(0)
lim m-+00
1
Since the initial eigenvalues are identical, i.e., A (0) (0), the conserved quantities derived in Lemma D.7 dictate that the limiting values must also be identical. When L 2 3, the conservation law states:
(o0) - (00) (0) 10) In the case where L = 2, the conservation law states:
a- a-=-_ 0.
This implies ( = )2(2o). Consequently, the singular values of the product matrix satisfy 01 = og for all i n
\ (oc) A (0) =
>2(x)) A2 a
This also implies (oc) )2(00) and thus 0 = o. for all i (2....,n)) n). Consequently, by (25), we have σi = = sw"
46

---

Published as a conference paper at ICLR 2026
D.4 LOss CONVERGENCE
We further establish loss convergence in the following proposition.
Proposition D.1. Under the setting of Theorem 3.3, suppose the factor matrices are initialized according to (7). Then, under the gradient flow dynamics (3), the loss converges to zero:
lim ((WL.1 400
2 = 0.
Proof. According to Lemma D.6, the dynamics of the third eigenvalue are governed by Aa(() --/ (t). For L 2 2, this is separable ODE whose explicit solution is given by
(1 = (22 - 2)t + A3 (2L-2) 21
(0))
As oc, the term inside the parenthesis grows without bound, which directly implies λs(oc) 0.
Next, we consider the global error term
7(1) _ x1(1) (n 1)1 sw"
(28)
n
By substituting the dynamics from Lemma D.6, we obtain its time derivative:
+(t) ( + (n - 1)1 1 (t)a(((()
(221-2 + (n 1)12(~2(1 7(1)
K(()))
The solution to this linear ODE is (t) = 70)) exp (- K(F)dT)) which implies that 7(t) pre-
serves its initial sign for all 2 Since L 2. the map f(x) 21-2 is convex on E.,, By Jensen's 's inequality, we can lower bound K for any fixed t:
K(t) = (AF(1)) (n (t) 222
n
2 L x1(1) + 1en-mmm11-00 1)) 24
(29)
Case (7) S 0). Suppose the initialization scale a satisfies
w² dm L
0 < a
(m m 1 +
1)(m 1)4²
This ensures 70) < 0, and consequently 7(1) < 0 for all t 0. From (i(() --(((>)-1())
(1)
it follows that A (t) 2 Aı (0) > 0. Similarly, A2 2 >² > 0. Therefore, K K(t) is strictly lower
bounded by its initial value:
L (x22-2 (0) + (n 1)x32-2 (0)) >
K(L) 2 ((0) n
Case 2 (7)(0) 0). Suppose the initialization scale satisfies
w= dm
at (m + d 1)t (n 1) 1)-
so that (()) > 0, which implies (t)) > for all 0. Using the lower bound from (29) and the fact that (100(111))) -(t) + sw", have: "
21-2 =
K(t)= 2 L ( sw")
47

---

Published as a conference paper at ICLR 2026
Since ytt) > 0 for all t 0, can further lower bound (t) by constant:
K(t) > L(sw") _ > 0.
In both cases, there exists a uniform lower bound Kuin Substituting this into (28), we have
such that [(t) 2 Kmin for all 2
hn( hx)) exp ((T)))r)) - lh0ole 1 Kmint
Taking the limit as - oc, we obtain lim 7(1) O, which implies
xf(1) (n (1) 8w".
lim t1000
(30)
Recall from the inverse relations in Lemma D.5 and Lemma D.6 that the parameters of W WL.(t)) (A(I), ((), C(1) are given by
A(I) = x((1 + (n - (t) + n(s - (1)
B(t) (F + (n- - (t) n2f(t)
sn
Combining the results As((0) 0 and (30), the limits of these parameters are
lim A(I) nsw n(s 1) w",
t1000
871
nsw" -n-
lim B(t)
w²
sn
The loss function is defined by the error on the observed (2 which consists of the diagonal blocks. Specifically, for each block, the diagonal entries converge to A(oc) w" and the off- diagonal entries converge to ((x) = w". Since all observed entries W share the same value wt, we conclude
lim L (t: (15.11 = 8-000
Σ (L.))
(wiy((0) - w²²² = 0,
which completes the proof.
48

---

Published as a conference paper at ICLR 2026
D.5 UNIQUENESS OF THE LIMITING SINGULAR VALUES
Proposition D.2. Under the setting of Theorem 3.3,
2-h.
fi(0) 20 2T
w d 0
strictly decreases in (0, w² d). Also,
f2 & (w' (n- 1)0) 7T. 2-1 strictly increases in (o, w'd n-1) Therefore, Equations (8) and (9) admit unique solution (01, os).
Proof. Under Theorem 3.3 with I, 2 3 and 1 < xc, the limiting nonzero singular values satisfy
(n 1 10;; w
(31)
where E nn) Since these are the nonzero singular values of the limiting product matrix, we have
0,
o
Combining this with (31) yiclds
0 0c < w² d
0 < w'd
1
Hence the implicit equations are naturally defined on the intervals (0, wnd) and (0, w'd For brevity,
P.n-1)
2
0.
L
Uniqueness of σ₁. Define
( (w d - σ
S1(0) - o=
E (0, wdd). d).
n-1
Then
ww'd
f((0)= =0
n
Since < 0 and both terms inside the powers are positive on (0, w d), we obtain
fo) < 0 for all E (0, w w²d).
Therefore fi is strictly decreasing on (0, w d).
Moreover,
lim fa +00,
lim f1(0) = -00
< d)
Hence Si is continuous bijection from (0, w onto R. Therefore, for any constant Ca,m, R. there exists unique di E (0, w d) such that
f((0))
Cam,Ldd
Uniqueness o, Deñin
OE 0, w=d
fz(0) e (w"d (n 1)0)a 00,
n1
Differentiating gives
f100 -a(n 1)( d - (n 1) a-1
49

---

Published as a conference paper at ICLR 2026
Since -a >> and both bases are positive on (o, wn
we have
for all o E (0, w'd
f2(0)> >0 0
nn-1
Thus f2 is strictly increasing on that interval.
Also,
lim f2(0) 0-0+ -0,
lim a a (m=1)
f2(0)
+00.
Hence fa is continuous bijection from
0, snd
onto R. Therefore, for any constant Ca,m, R.
there exists unique a;
10 (0
such that
7
f2(0,) L.
This proves that (8) and (9) admit unique solutions.
50

---

Published as a conference paper at ICLR 2026
D.6 PROOF FOR COROLLARY 3.4
Corollary 3.4. Let 1 < m < oc, 2 2, 8 2 1, w² > O, and L2 3 be fixed. Then, as a - O, the stable rank of the limil product mairix Wt.1 (00) converges to one; that is,
srank W (oo))
Proof. Fix > 1, 2, 1,
0, and L 2 3. Let
2 -
a _
0. L
First, we analyze the behavior of
2-L
L.d
((m + d 1)2-1- (m (m-1)2-11 1)2-1)
m
as a + O. Since L 3, we have 2 - L < 0. The map a + 22-1. is strictly decreasing on (0, oc).
and because m d 1 >
0,
(m + d 1)2- (m 1)2-L < 0.
Moreover, (o) 2-2 + 0. Hence
C La L.d -00 as a a- 0.
Next, consider the function from (9):
f2(0) (n 1)0) 0", oe (0(0, w'd
n1
By Proposition D.2, we know that f2 is a continuous, strictly increasing bijection from
(0,
und
onto R, and for each C e IR there is a unique oCO) such that f2(o(C)) = C.
Now apply this C. L, Since Co.m.L.d 00 as a 0 and is strictly increasing with lim fa(a) -c, it follows that a++
a,(a) as a 0.
Using the linear constraint (25), then obtain
0 (a) w² d - (n 1)o;(a)) - w'd a 0.
The stable rank of W W.. (co)
Frank(WL.1(00) 01(a)² + (n- ());(()² + n(s 1)0?
[oi(a) 2 01 (1(()
11 (n 1)
(a)²
Since (a) and の (a) + w pdd have
(:(a) [1(a)
0,
and therefore
srank (oo) +
as a - 0.
51

---

Published as a conference paper at ICLR 2026
E PROOF FOR SECTION 4
In this section, we provide the proofs for the propositions and theorems presented in Section 4. First, Subsection E.1 presents the general form of Proposition 4.1 along with its proof. Next, Subsection E.2 details the proof of Theorem 4.2, focusing on the 2 2 matrix case, Lastly, Subsection E.3 generalizes the core ideas of Theorem 4.2 to d x d matrices and provides the formal statement and the proof of Theorem 4.3.
E.1 GENERAL FORM AND PROOF OF PROPOSITION 4.1
We first present the general form of Proposition 4.1. This proposition applies to any "fully discon- nected case", a scenario that involves the diagonal entries introduced within this same proposition.
For d x d ground truth matrix W the observed entries are given by Q = {(ins jn)]=== Since we consider the fully disconnected case, in * im * I' for all n + m [d.. We factorize the solution model at time as W (t) (t) where WA, (f), A(t), B(t) E Rdxd We consider the gradient flow dynamics with the loss function defined as (2).
For given row index k, since there exists a unique entry (k,j) E I2 we denote this unique column index by j(k), Thus, (k,00 and (k (t) refer to the ground truth weight wt, and the time-varying weight wx.s (t) respectively, where j = ((8), Similarly, for given column index I, since there exists a unique entry (i,4) E Ω we denote this unique row index by z00, Thus wao, and W;m, refer to the ground truth weight i and the vire-ngrying weight (t. respectively, where = j Defining the residuals as riy (4) == wii - ())((), we adopt this compact notation for residuals as well. Then, can derive closed-form solution for arbitrary initialization with below proposition.
Proposition E.1. Consider a ground truth matrix W E IRd and a set of d fully disconnected observations Ω {(i, [n)) ) The model is factorized as W (t. = (()) where the
factors A(1), B(I) E IRddd For euch observed pair (Inyjn) E Ω define the constants P Pi.cin and Qin.ja based on the initial values A(0) and B(0):
d oi., (O)DE.,) and Qincin
Pt
(0)) be.J.. (0)3) 1=1
1=1
Furthermore, for each such observed pair (ins jn), let the parameter Pinda be determined from the ground truth entry w and the constants defined above, as follows:
Panin + Pinsin
A 1 log
F Findo
2
2
Winn + V Winn
2 pd tn
+
Qiguia
Then, assuming convergence to zero-loss solution (i.e., Win (00) = w in for all (Ins E Q). any entry ap (20) of the converged matrix A(xo and any entry by.g( of the converged matrix ((0) (for arbitrary indices p, q [d) are explicilly given by:
(2.9(20) ap.9 (0) cosh (Fp.tr)) by sinh (Fp.(t)) (00) (0) cosh (Fat).g) 0,to), sinh (Frm).)
Proof. We express their evolution in the following vector form using the vectorized parameter ((t) = vvec( A()))
vee (1)) E R22"
0...0
R(I)
0(1) =
O())
(32)
R(t) 0
52

---

Published as a conference paper at ICLR 2026
where R(t) Rdxxdd
xd is defined as:
(4) F1 (t) 3o4dd
(t) -id-11
R(()
(33)
(2 (1)e e 22 (1) (2)
(ra)(e)) (t)e
+d- 1)d
for e, E od form the standard basis. Since
02e 22 R(()
R(t)
commutes with any other t values,
0,
the solution is given
R(()
((t) = exp
0(0) dr 0(0)
dr
(34)
R(t)
Og d2
O.e.a R(t)
R(t))
exp
(35)
0,
where
(1)
F1 (t) (ardd
R(I) = / R(T)dr= =
F1,j (t) Ju)
F2, (1) j00
1 (1) j
Fa (1) T +(d- 1)d
for for (t) = (T.((T)dT. If we assume convergence, get:
((0o) = exp (-
0 R(
R(o0)
dr
0(0)
(36)
L
Og
R(())
(())R((1
0.
+
(37)
L2z
R(t)
0,
2
Oct
R(t) R(O))
0g R(I)T (())
R(t) R(t) R(t)
1
(()))
O
+
(38)
6
0
24
0,2
((())R(()))
0(0).
(39)
which can be simplified as:
C
D
((00)
0(0),
(40)
E
F
53

---

Published as a conference paper at ICLR 2026
with C, D, E and F are defined following:
diag (1.00........ .... 11500 P2,,), ..... ............ P..00....... ,............................. Fa...)))))))
C cosh
F cosh
diag (201,, ( F000 a ................................ P20, F0 ................................ Fr000
aa Pa (d- 1)d )")
D sinh
E sinh [1.00e,......,.00ee....
Fay Paji Paj
Here, for any matrix the operations cosh(P) and sinh( are performed elementwise. For set of d observed indices Ω there exists corresponding unknown variables, Fae.js If convergence guaranteed, the model yields d equations relating these variables the ground truth values. This implies that the variables Ji can be characterized as a closed-form. To characterize more rigorously, we substitute C, D, E, and F into (40):
(0) cosh(Fi,, 01,2(0) cosh(Fi.ga) b2 (0) sinh (1.ja)
b1,500 (inh(T 1,50))
11,1
(1))
(1.a(0) cosh(Fi,,a))
bago (0) sinh (f (0,)
01
02.1 co (2,
a (0) sinh (inn((2,,(2)) (2,)(2)
(221
(2) cosh (T2,(())
ba (0) sinh (72,(()
02,2 00)
(00)(00)
(0) (o (.)
b ba (0) sinh 2,)
(00)
(0) cosh(Fa,te))
b (0) sinh (Fa)(()))))))
(00)
(os)0)((sFFF b (0) sinh (dn)))))))))
0(oo)
(41)
(00)
a.00,1 sinh F + b1,1 cosh (1)0).1) (000,, sinh + 2 cosh (F)()
(00)
(20)
-aya sinh a + b1.a(0) cosh (a, (r(c) a)
(20)
a,00,2 sinh ( + b2.1 cosh (1))) -,,02, sinh (,02))) + b2,2(0) cosh ())
(20)
(20)
(0) sinh ( a) + b2.a cosh (f(e) a)
(20)
4,00 (0) + b (0) (osh(F,,),))
(oo)(
4,0 (0) sinh + (a,a(0) cosh(F a))
Then, assuming convergence, for each observation (In) Ω (for 1,..., d), obtain the equation:
Wi = Win.,to (00) = 0 (20,1,000), + .... at, (200) (oo)
d
(oin (0) cosh ((n) (0) sinh
(bk.jn (0) cosh (Fr.. &in sinh (Fr (r...
54

---

Published as a conference paper at ICLR 2026
Let Cn cosh(fin...o) and Sn sinh(FFr..Ja)) Then rewrite the above equation
ai x (0)bk.n. (0)C ai ai,, (0) be.. (0), + a (0)ba,)
k=1
aig (0))EE.. (0)
(0 + S
Σ
(0 (o... SnSn
k=1
Vinci sinh(2Fin...l))
Pennin cosh(2fin.ii)
(42)
2
where Phncch Ex= Ci (0)bb. (0) and Qinon = Ek=n (a)), + bx (0)
By solving (42) with respect Fin can get:
P ia (e (e + e -2inn)) Qintra (e2tnndn 22 -m
2w 2w
2
Qinci
Qinin
2f Piavia 2
Pinnin
Multiply by Minnn leads to:
Qiyia
Pin.inn + Qinin
2w in e Penii e APtnin
Pinnin
2
Rearrange into a quadratic equation by setting u = e2ftninm:
Qia.in u² 2w, in in + Pinit
Qinwa 0.
+
2
2
By solving the above equation while noting that Pin.in Qinda explicit solutions for Fin
< by the definition, we can get
Pen + Dinuia
1 2
F Pin
log
2
+
W
2 P? in
+
(Qig
Note that each Fi...., is solely determined by the initial points ((0). With J.., determined for each observed entry, we have closed-form expressions characterizing the model's 's learned relationship for these observations. Consequently, by (41), we have:
(o.g(00)= ap.g (0) cosh (Fp.(to) by (0) sinh (Fp.sto)
bp.g (20) = bp.g (0) cosh (Frto.g) ag00) (0) sinh (Frto.g)
55

---

Published as a conference paper at ICLR 2026
E.2 PROOF OF THEOREM 4.2
In this section, we will provide the analysis of 2 x 2 matrix that starts from pre-trained weights with
diagonal observations w" _ wii w5g. WA,B(1) cannot converge to a low-rank solution. Let
Ti 4 be the timestep that concludes the pre-train phase. For the sake of simplicity, we omit the term introduced in the pre-training phase. Then, we know from Proposition E.I, we have:
(VW 0
A(T)) = B(T)) =
(43)
In the post-train phase, we introduce an additional observation in the off-diagonal entries, specifically "i2 w21- Without loss of generality, assume wi2 > 0 is revealed while other observations remain the same, i.c. Sppctt = [[1, 1), (1, (2, 2) Note that the gradient of the post-train loss is:
W11 w 112 W
vE(WA.B)
0
R222 w²
O1 011d12 + 012b22 wi2 01ib11 + 012b21 w"
0
021b12 + a22b22 w
For simplicity, we again omit the Q term in the loss specification. We define the residuals for the relevant matrix elements as r11 = W11 w", r12 == W12 Wi2. and r22 == W/22 w=.
We begin by demonstrating a pairwise symmetry between the entries of A(t) and Bt),, which simplifies subsequent analysis. To this end, we first provide the time derivatives for the elements of (t) and (t). Given the general gradient flow dynamies A(t) V((WA, A. (t)) T (8) and (t) = - AT (t) (t)), the component wise updates are as follows. For A()):
ăn (t) = (t) wn (t)) + (1 w12(t)),
(12(() bui (1)(w" wnn (1)) (22(() w12(())
(44)
à21 (4) b12(1)(w" m22(()),
á22 (t) b22(t)(wx m22(t)),
and for (t)):
by (1) = (1)(t)(w w wI)(
(12(t) = (11(t)(wiz w12(t) + a21(t) ww2(()),
(45)
bai (1) a12(1)(w² win (I))
(t) = (12(t)(wi2 - w1)(t)) + a22(t)(w* -222
Using the equations above, we first present a result showing that the k-th derivative each element in A(t) and (t) at initialization exhibits pairwise symmetry:
Lemma E.1. Let W (Ti) = A(Ti)B((T)))
E 2 be product matrix, where A(Ti) and B are matrices that are obtained at the end of the pre-training phase. Suppose the ground truth matrix satisfies wi w22. Then for every k E N U (0), the following identities hold:
4
(k) (Ti) 6/2 (Ti),
d (Ti) -b/2) (Ti),
(46)
(b) (Ti) = 111)) (Ti),
(2) (Ti) (((1) (Ti),
and consequently.
(1) wir (Ti) = W2z (Ti).
(k)
(47)
Proof. We prove the statement by induction on k. When = 0, by the initialization assumption, we have
(Ti) (Ti), (12(TT) (Ti) aan (TT) but (T1), az2 (Ti) b (Ti),
56

---

Published as a conference paper at ICLR 2026
and therefore W11 (Ti) (T)))
Assume that for all orders m < (with & 2 1) the identities
(mm) "11 (Ti) = 1 (Ti),
d (Ti) = (/))) (Ti);
(m)
(vm) (Ti) (T)) =
( (Ti),
(m) 022 (Ti) ((T)) (Ti),
22
(m) (Zi). By the Leibniz rule, each element of the k-th derivative can be written as finite sum involving derivatives of orders strictly less than k. For A()):
hold, and hence also wi (Ti) = W22
19)
k: k-1)
k:
(8) (t)
22 (1)
k (k-1-1 1-11-1) (1)r (1),
b
k 1) blh-1 1- (1) (1)
222 (1)
and for Bt):
k-
(1) =
Σ
(k-1-)) (1)
d11
j
(1)
(1)
1
d11
(k-1-5)
a (1) (1).
(1)
(k-1-)) (1))
12
(1)
k k-1) j
(k-1-1) (1)
222 (1)
01 (1)-
012
By the inductive hypothesis, all derivatives of order less than 1 satisfy the symmetric relations at t t= Ti. Inserting these equalities into the expressions with Ti above shows that the symmetry is maintained at the k-th order:
o (T)) = (k) (Ti),
1 (Ti) ((2((T)) (Ti), (Ti)
(Ti) b (Ti),
d22 (Ti) (Ti) b1b (Ti),
proving equations (46) and (47).
Lemma E.2. Under the setting of Lemma E.1, below relationships hold all t 2 Tı: a12(t) bi2(t), azz(t) b11(t),
an (t) b22(1), az (1) = ba (t),
(48)
which further leads to wni(t) We2 (t).
Proof. By Lemmas F.6 and 1. we may conclude that for all t 2 Ti, equation (48) holds, and therefore wnn(1) (1).
By Lemma E.2. all entries of Bt) can be expressed in terms of the entries of A(t) for all t Ti. From this point onward, we will represent WA (t. solely using the elements of A(t). We begin by simplifying the time derivative of (() as follows:
57

---

Published as a conference paper at ICLR 2026
â11 (t) a22(t)(w* -wn((t a1(())( (12
(12(t) a21 (t) wn wn (t)) + an wl2(t))
(49)
d2 (t) a12(t)(w* -222(t).
ó22 (t) an((t((w -w22(t).
Rewriting W. (t) () in terms of the elements of A(t) yields:
W (A, (())
(n)(t) a12(t)
(22(() a12(1)
a2i(t) (22(t)
(21
an1(t)
a22 (t) (12(t)aa2() (t)
2011 (t)a12()))
an (t)a2222 + (12())aa(()))
(50)
2021 (t) (t)
We can also simplify the time derivative of WA,B(t) as follows:
wn (t) (w" (t)) (tf afz(t) a (
+ (wi2 w12(()) ((n())) (1) + a02(((222(
122 (t) 2(w w12(t)) (t) + a
(51)
+ 2(w" wn (()) (an()az1 (12(()2()))))
Ú2 (1) 2(w" - -wi (()
+ (12))))22
222 (t) (t).
Using (50), we state the basic conservation law from Arora et al. (2018): the matrices are initialized in balanced manner, this balancedness is preserved throughout the training process. That is,
A(Ti) A(T) ((T))
holds at initialization, this leads to
(t) + (t) a 2 Ti.
(52)
Now, going to examine the time derivative of the loss: d dt (WA.B())) = Ve(WA.B(t)) W())
Tr (VE (WAA())(((((()))))) (A())()+ A(1)B()))
Tr (VE (VE
+Th (V (TA (W(((A.B(()A(())))
Tr (W)(((W (+)(()
-Tr
- Tr (V((WA.(()) B (I)
- -Tr (V (
(53)
The third equality follows from the fact that for any two matrices A and B of the same size, (A, B) Tr(A B). The last equation holds due the cyclic property of the trace. Combining (53) with Lemma F.7, we can ensure (t(t) and (t( are both positive semidefinite, which implies the loss is monotonically non-increasing for all Ti.
With Lemma E.2 and the monotonicity of the loss, we can guarantee positiveness of a11 022, W11+ and W22 after the pre-train phase:
58

---

Published as a conference paper at ICLR 2026
Lemma E.3, For product matrix WA.B(t) ((t) R2x af (Ti), a22 (T1) wn (Ti), and W22 (Ti) have all positive values, following inequalities hold for all t 2 T1:
@11(t), a22 (1) 0, a12(t) 2 0.
Furthermore,
W11(),, w22 > 0
holds for all 2 Ti.
Proof. We will prove the inequalities step by step.
Positiveness of a1 (t). For the sake of contradiction, assume that there exists timestep 7i > Ti where a1 (TI) = 0 holds. From (50) and Lemma F.3, we must have det(A((r)) > O, which implies that (12 00 (T)) < O. Given the monotonicity of L, W (t) must satisfy:
((WA.B(t) & ((WA,B(Ti).
(54)
for all / Ti. However, WA, (T, cannot satisfy (54) because wn wI(()), (22(()) and W12(T1) = for any 71 2 0. This contradiction implies that such cannot exist.
Positiveness of (t). Similarly, let's assume there exists time 72 Ti such that (22(T2) 0 for the first time, We can express WA. WA.B(T2) as:
WA,B(72) = (112(T))22112)) 2011 (Tz)
@12(72) 0021 (T2)
where the diagonal entries are negative due to the condition (el(A(22) > Therefore, the time derivative of d22 at timestep Ta is positive:
áz2 (72) = an (12) - wn1 (T2)) > 0.
Since (22 is increasing point 72, there exists time '' < 72 such that (22 0 (since a22(1) is continuous and differentiable). which is contradictory. Consequently, there cannot exist a 72 such that 0:22 (2)) O.
Positiveness of a12 a1)(t). Given that E non-decreasing, we can state:
1 ((WA.B [[ wi1 (t))² (wf2 w12(t))² + (w w22 (t))²
S <((WA.B(Ti) 1 2 1122"
for all t 2> Tí. Since (w* - (t) and (w* W22 (t))² are non-negative, (12(() must be non- negative for all t 2 Tı. From (50), we know (1) = 2011 (t) (t)aiz(t), which implies 012(t) 2 0 for all t 2 Ti with the above conclusion which states 011 (t) > 0.
Positiveness of (t) (22 Likewise, assume for the sake of contradiction that there exists time 73 2 Ti when (T3) 0 is first satisfied. This directly implies that @n (73)022(73) -012 (Ts) (22)))1((33))) Squaring both sides of the equation yields:
af (Ts (T2) = (Ta)
Subtracting (T (T3))az from both sides:
a (T8) (Ta2(Ta))22 = (f
Factoring:
(T)) (T)) - (Ta( = (T)) (a (T:) o (Ta))
By the conservation law (52). we have af (T:) + a (Tx) = (T)(T)) + az which leads to
a (Ts) (Ts) = az2 (Ts) - an (Ts). Replacing (7s) (T2(Ta) with - - (Ts) ad (Ts)):
- (Ta) (1a) (Ta( dT) (T)) af)(
59

---

Published as a conference paper at ICLR 2026
This gives us:
(a12(Ts) + (Ts)))) (a (Ts)
(Ts)) 0.
Since 022 (22((T)) > 0 from the previous result, we can conclude that d21 (T3) +t(22((T)) the sign of 021 (7a), recall that W WA.B(1) (Ta( is written
To determine
2a (T3))122((T)
(Ta)(Ta) = 2021 (25
0
Since a11 (71) > 0, (Ta 2 0 from the previous result, 2011 (T3))12(T3) 2 0 holds. Also, given that det WAWWWAW (a,,,WWATTTT (T3) O, we can determine that (21(TT) negative, which implies a21 (Ta) -22 (Ts) Additionally, by the conservation law, have af (Ta) af2(T)), which leads to (T) = 012(73) 0.
Finally, consider the time derivative of win at timestep 73, substituting 0n (73) and a2i (73) with (12(T3) and (T2) respectively:
in (T3) (w* wnn (13 (af) (T3) aiz(7s) + a (Ta) + a32()s))
+ (wi2 - (an) + (1((5)))227Ts)
2w (af (Ta))
0,
which contradicts our initial assumption.
Given that the time derivative the (51) includes the term an (1) (1) + (12(())22(4), we need to verify the sign of 011 012022 in order to proceed with the analysis. Below lemma shows that as long as W12(t) (t) S wi2 holds, a11 (t) + a12(t) is always lower bounded by zero. Lemma E.4. For product matrix W (t) A(t) R if at any point E TT, we have (1) S w12. then the following inequality holds throughout the entire interval [Ti,T]]: Tz]:
011 (11(1)2210) + (12(()22((1) 2 O.
Proof. We first define g(t) - au (t))21(1) + (1))2222))) Recall that at Ti, we have (22 = az (Ti) 0, which implies ((Ti) = 0 as well. Note that by (49), at timestep Tı, we have
(T2(Ti) a11 (oi(TT))) (Ti) (w w12 + az (T)) (w" wn1 (Ti)) O,
while other elements remain unchanged. This indicates that g(() > 0 immediately after Ti. We now show that g(T) > 0 for any r E (T1, 72), then there is no 7' E [r, Te] which satisfies both
g(r') = and d [t)( t=r' 0. This implies that g(t) never becomes negative under the assumption of (12 & wi2-
Suppose, for the sake of contradiction, that there exists r' E [T, where g(T') and L
< 0. Given g(r') 0 and the conservation law in (52), and the inequalities from Lemma E.3, we can determine that there exist two combinations the solution:
1. an (n1(7) a22(7'), a12(7') a02)(T)) a11(7') a12(7').
2. an1(7) (2),,)), (12(7') az (T') 0.
We take the time derivative of g(t) at timestep 7' and substitute the values from (49) as follows:
dt (() g
d
(=-' "án(F")a (7') + a1 (7))221(())))))))) (T') + à12((') (12(F')àzz(7')
2(w" wn (T') (")
azi (r')2222)()"))
+ (wi2 (mn( (71))222) + (ar())a)2(7)))
(55)
60

---

Published as a conference paper at ICLR 2026
For the first case, substituting equations d11(7') d22(T') and 012(7') -021 (T') to (55) leads to:
d
g(t)
(wiz (112(T)))w1((
dt 1=T'
Since (t) 0 for all t Tú, if w12(1') wf2 holds, then g(t) cannot take negative values at time r'.
For the second case, substituting equations d11(T)) a22(7') and a12(r') a21(T') 0 to (55) leads to:
dt (()) = (ws2 (f
which is again non-negative value if w12(r') S wi2, leading contradiction.
Lemma E.5. For product matrix W (t. ((t) E R2 the following inequalities holds for all timestep 2 Tí:
W12(t) S wi2 Wn (t), W222 (t) 2 w"
22 (t) < 0.
Proof. We will prove this lemma in several steps:
Step 1: (1) S 2 for all t 2 Tı.
We know (12 = 0 < wiz- Assume, for the sake contradiction, that there exists a time t' > Ti where t' is the first timestep such that W1a(t') > wiz- If this were true, there must exist time where 7i S < such that:
w12(s) = wia wi2()) >
For these conditions be met, (1)(s) must satisfy: 2 (8) 2(w" wi (s)) (1)))))))))())))))))))))) 0.
(12())222(5))
(56)
To satisfy (56), there are two possibilities:
(w wi (+)) > 0 and (an (s)a21 (s) + (12()))22(5)) > 0,
(57)
(w" wwi (s)) < 0 and (s)(2) (s) + (12(s))22(s)) 0.
(58)
However, neither of these can be true:
1. Equation (58) contradicts Lemma E.4, given that t'.
2. Equation (57) cannot be satistied because there is no s where w² wii(s). there were. there would be a time s' where Ti I s' < both satisfying wii(s') w", and (s')ss) < 0. But we find:
ا (s') (wi2 W12(s
(12(s')azo( (s' (an (2)) a12 (azz(s)) (s')) 2 0.
This is because W12(s') < wi2, and thus an (s'))zn(( (s') + a12(s' (12(s"))zz(f (s') 2 by Lemma E.4. Therefore, our initial assumption must be false, implying that (1) I wia for all 1> Ti
Step Prove w11(t) 2 win and (22(t) 2 wāz for 2 Ti.
Given (1z(t) < wis for all 2 Ti, Lemma E.4 implies an (t) (t) (12(t)aze(t) 2 for all t t T. The evolution of win is given by:
vhn (1) (w-- - (o) (0 (0)))))))))))
61

---

Published as a conference paper at ICLR 2026
By above equation, if there exists time t' > Ii where (t') w", conclude 1111 2 0,
and thus W11 (t) 2 w" for all t 2 Ti. By Lemma E.2, W22 has the same value as W11, so Wa2 (t) 2 w for all 2 Ti.
Step 3: Prove W21 (t) & 0 for all t 2 Ti.
The evolution of W21 is given by:
ر2n (1) 2(w" wn (1)())((n(()aa1)) (12(()az2()).
Since w11 (1) 2 w" and o1 (1)221 (1) + (12(())22(() 0 for 2 Ti, we can conclude W22 (4) S for all / 2 71
E.2.1 PROOF OF Loss CONVERGENCE
Recall that the time derivative of the loss function is written as:
di (WA.B()) = Ir(L, - Tr(,2(1))
where L1 and (t( are defined (53). To further our analysis, we can expand the time derivative of the loss by calculating the trace of L, (t) and (t).) We omit the time index when clear from context.
L1
ri1
r12
a
22
a 011021 a12022
T11
0
0
T22
0111221
012022
"11
4i
2
F12
T22
(a2
+ 222))
21
(111 021
012022)
r
(a
aîz)
C1 +
C1
r22
(a
a42))
for some time-dependent value Cj. Following similar process, we calculate Lz:
Fi1
at + a
011021 + aj2022
("o
T12
L
r12
122/
Q11 + 012022
a
az
T22
(ri (a + afz)
(a a2z)) + 221122 + 012022) + (22 + )
C2
C2
again for the time-dependent value C2.. With these expressions for L1 and L2, we now rewrite equation (53) in a more explicit form:
(WA.B = = - dt
l
--Tr (L)(())) - Tr (L2(t))
= =-) (df afz(t) + (t) (t)
2r (of,(t) + diz(t))
(1) (1) (12 + 2 (1) + (1)
(f) (an1(t)a (t) (12(t)az2(t)
2r (f)r1a()))))) (an (t)an (t) + (12(t) (t))
(59)
Note that the (59) is the non- non- positive term. Given that L, and L2 positive sefiidef, analyze each diagonal entry separately. This leads us to the following inequalities:
(a + 2²22) + 2 + (1242)) + 2 + b(2) 2 0,
(a + ) + 2/12r2((m121 + (12(22) + +22 + a 2 0.
By rearranging the above inequalities, obtain:
2r1ii2(0n12)) d12022) I (a + 4²2) + (0 + az2);
-2r a11121 + 012(2)) 5 (a + a2z) + (1 + +2z))
Substituting these inequalities into equation (59), we derive:
d (Wa,(()))) < (1) (df +z) (1) (4) + (1)
(60)
62

---

Published as a conference paper at ICLR 2026
This provides tighter upper bound the time derivative of the loss. However, it is still insufficient to guarantee convergence, as the bound does not depend on the term (12 As a result, even though the right-hand side converges to zero, this alone does not imply that the loss itself converges.
To further tighten the bound, we leverage the positive semidefinienness of L1 and L2- Specifically, note that for both QKQ and Q KQ to be positive semi-definite, the only necessary condition is K 0. Therefore, we modify L1 to L1) - VI(WA, A, (t)) (B
e2e,) (WA, where jut) is chosen ensure that the matrix B (L) p(()- e2ez remains positive semidefinite. This guarantees that ()) > 0. To ensure this condition, p(t) must satisfy:
0 (t) + (2)(1) an an1()2210 (1) +(2(()a)))))) (tt) B(t) p(t) e2e2 (n1(t) (t) + (22()) (t) + ((2(() - m(t)
(1)) + (1) m() (an(()a22()) - (12(1)a21 (1))
2
Rearranging this inequality with respect to (tt),, we get:
m(t)= < (an (1))(222221 - (12(())21(1)
(61)
22 + (t) det (t)))) 2 (1) ()2
Therefore, p(t) satisfy the above inequality, we can guarantee L1 to be a positive semidefinite matrix. Now, (t) can be calculated as:
L i'1 ri2 d + o
a11021 a12(122 af + di2 1
Ti1
0 T22 a11 012022
r22
122
Ri (a3) + a22) + 201 + 012022) + ri2 + afz
p1)
(afa + az - v
Č
for some Č. Since the matrix B B- is positive semi-tefi we can ensure +22 This leads to the following inequality from [
12 ( (011021 + a12022) S r (421 + 2²2) + r2 + d -1).
Finally, substituting this inequality into (59), we arrive
( (WA.B(1) S - (r (1) + (1) (1) afz(()) - (t))))))())
(62)
dt
To prove the convergence of the loss, our main remaining goal is to establish time-invariant lower bound for
min {a (t) + af2 m(t))
to apply Grönwall's 's inequality.
Lemma E.6. For solution matrix W. A. the matrix after pre-training up to time Tı, the inequality
W,,((t) (t) initialized as W (Ti), which represents the state of
det (WA,B(t)) 2 2
holds for all 2 Tí.
Proof. Since W12(t) must satisfy (W12(t) w121 < (e,.(t)()t S wia by the monotonic- ity of the loss, we can ensure that (12(t) 2 for all 2 TІ Also, by Lemma E.5, we have wii (t)(t), W22(1) > w", and w21 (t) < 0 for all f 2 T.. Under these conditions, det(WA,n(( can be lower bounded as:
det det(WW.B(t)) wn (t) - wis (f) (t) 2 2²
for all timesteps > Ti
63

---

Published as a conference paper at ICLR 2026
Lemma E.7. For u(t) defined to satisfy (61) and the entries in A(t), the following inequality holds for all timesteps 2 Ti:
min {af1(() + (1) u()) 2 w".
Proof. To prove the lower bound of (t) (t)( Our goal is to demonstrate that af (t) at)+ > w" for all timesteps & after Ti. By Lemma E.7, we have (| 2 V2w", which leads to:
V2w" S [ [a..(tt)l
Vo (WA.((t)) +2 (Wa.B(t).
By applying Lemma F.4, we have:
(oR(WA.B(O) + + (A(()))) + +
-o (( of (A())) - 207 (A()))3 (A())
VIIA(t)) - 2 det( ((t))
(63)
Rewriting (63) while applying Lemmas F.4 and E.6 leads
[(())) 2 2w"² + 2 (ee((A()))
2w + det (t).(())
2 4w
Thus, A(t) have to satisfy [[((t) > 2w for all timesteps t > Ti. Now, assume that there exists time e' Ti such that aj (t)) + (t') < w², To satisfy inequality [|A(t))) > 2w", would
need at least a31 (t') + (t) > w to hold. To verify the value of 0z (t') + 422 (t'), we take its
time derivative using (49):
d
dt (a2 (t) + (t)) 2021 (t) (t) 20 (t)a(()))
2212 (f)a21 (4) (1) 20 (4)222(()))))
2r (t) (a) (t) + a12(t)
2wn (t)).
Since (11(t) 2 w" holds by Lemma E.5 for all > Tí. we conclude af (t) + a22 monotonically non-increasing from time 2 Ti. Since (Ti) + (Ti) is initialized as w". this implies that a2 (t") + 232 (f)) & w". Consequently, there cannot exist 1' > Ti such that of (t') + (f) w holds, which leads to contradiction.
Next, we are now showing that the term
az (3) is lower bounded by w". Therefore, we set
det 2
((t) as w", we can guarantee the positive semidefiniteness of (t)..
By applying Lemma F.4 and the lower bound of det (AWWAA A. (()) by Lemma E.6, we have
det (B(t)) det (W (t)
2=
2 (1) (t)
+ az (t)
a
(1) (1)
Also, from the previous result, we have an upper bound on a (t)+ (t), which is (t) < w". Combining these results, the following inequality holds:
det (t))B(t)
a (t) + 22 2 w".
Therefore, if we set jutt) to be w", put) can satisfy the positive semidefiniteness condition. By combining the results, we can finally guarantee:
min (t) + afz( m(t) 2 w".
64

---

Published as a conference paper at ICLR 2026
Using the results of Lemma E.7, we rewrite (62) as follows:
d
(( (ri1(t) (af)(t) + af2(t)) (t)
dt
f (ff + (t) + +t)2(t) w
S 22w ((W),(tt).
Applying Grönwall's inequality to previous result, where 2 Ti:
demonstrate loss convergence
(WA.B((t) S (T))) ( (t-Tt
=2 12 e-=2=" (t-T:.
(64)
This inequality allows us to conclude that ((WW((WWW (.A(B)()) converges to zero exponentially.
E.2.2 PROOF OF STABLE RANK BOUND
From (64), we know that convergence, W11 = (22(00) w- and W12 (2o) wi2- Although a closed-form expression for W21 (oo) is unavailable, Lemma E.5 shows that W21 (1) S 0 for I. 2 Ti, which implies W21 (oo) S This indicates that the test loss remains strictly positive, as the ground- truth value w = w" is assumed to be strictly positive. wia
In this section, leverage the fast convergence rate detailed in (64) to establish bounds the singular values of the converged matrix WA,B ((o)). Subsequently, these singular value bounds are used further bound the stable rank of W A, (co).
Lemma E.S. The singular values of W A.B WA.B WA,B(o0) (0) fulfill:
(W (WA,B(o)) S w0 exp
2 W12
w
2 W12
05 (WA,BB(())) w exp
w³
Proof. We denote the singular values WA, (t) as (t) for simplicity. By Lemma F.I, we can get general solution of each singular value (t) by solving linear differential equation:
(-2 (VE(WA,D(t)) u, (t')o, (t)) ox(t)= o,(s) exp (-2 Jt's
1,2,
(65)
where u, (t) and (t) denotes left and right singular vector of corresponding r-th singular value, respectively. Since u. (t) and (y)(t) are both unit vectors, applying Gauchy-Schward inequality, we can bound (V.(WA..(t)) (t)o (t)) by:
( (1)) S (77((A,B(())) (1),(()), (1)ls
= =\V(WW.lt)))l
22((WA,B(().
we can get bound (t)() as following:
(xx-epxxp (2v2 / Ve(Wa.(()))ee) (o, (s) 2V2 I' ( K=s
(66)
With the setting above, in the pre-train section, after Ti timesteps, we prove that o (T)) = (Ti w". Starting from T; with pre-trained weights, can lower bound (aWaA,L (()) with equations (64)
65

---

Published as a conference paper at ICLR 2026
and (66) when 2 Ti as follows:
(2 2 (2() exp
-2v2 Ve(Wa.((t)arr
JI=
2 w² exp
220
e (1
v=
2w12 w*
(1 11-e (1---1())21))
wn exp
and when + oo, (2((0)) can be lower bounded by:
x2((0) 2 w² e -2. 2j3
In the same way, we can upper bound o1 (Do) by:
(10(c) S w² 6 Vit
By Lemma E.8, we can now lower bound the stable rank of matrix W (oo)
(
+
[A.B oo)
(WA,(()))
(ox)
(A..((O))
Vi2
2 21+ exp
w
which concludes the proof of Theorem 4.2.
66

---

Published as a conference paper at ICLR 2026
E.3 FORMAL STATEMENT AND PROOF OF THEOREM 4.3
We now extend the preceding analysis the general case involving a ground truth matrix W E R The solution matrix WA.B E [2d is again factorized as WA.B AAB where both A, B E Rddd In this section, our detailed presentation and proof of Theorem 4.3 (from the main text) are structured follows: first introduce and prove Theorem E.2, which is then followed by its direct consequence, Corollary E.3.
We the slightly modified loss function:
1 N C(A, B)
Σ 2
(AB, Xn)) ynn)²
(67)
n=1
where the measurement matrix X, Gi, e, represents masking matrix, with the n-th observed
entry set to one and all other entries set to zero, and Yn E IR denotes the ground truth value of the
A
1 0 X,
n-th observation. Then, by defining c can rewrite the (67) as:
2ddd and X, E
R 2dx2d we
B
2
x
N
C(A, B) = -(())
Σ (00 T .x,) - vnn)
2 n=1
2 [F - v11..
(68)
Here, F(O) and y represent vectors defined as:
N X1) (00 X2)
1/1
V2
F(O)=
ERN, D
E ERN.
(69)
(ee XN)
LyN
By reparameterizing A, B to ④, and X,, to X 2. we can reduce the parameter matrices into single We train the model @ via gradient flow, where the
matrix θ while ensuring the symmetry of θ loss evolution is given by:
((((1)) = (F(((tt) y) F(O(t))
10000)0) X,)
= (F(((()) T
4 (000)) (N))
(X)((),(()))
(x,0(),0(1)
-(F(((()) 3)
X (XvO (1), (1))
x()(t)) T
vec
vec (X20(1)
=(((((()) T)
vec (é(t)
(70)
X (
vec
=(F(Ott) b) (()(t)) vee (
(71)
67

---

Published as a conference paper at ICLR 2026
Here, the Jacobian matrix (()(t)) is defined as:
vec (e (e(t)
vec (x))(t) T
(t)) (()(t)) Ovec (t))
vec (Ve(O(t)@(t) (x2))
vec (X²O(t))
E RR22d
=2
vec (Ve(O(t)0(1),, x
vec (x))(
(72)
With the notations defined above, we state the following theorem: Theorem E.2. Let the combined weight matrix be
e -
R 2dxd
B
and consider the loss function C defined in (67). Denote
o miin 2 min (O)),
O max
o max (J()
If the initialization satisfies:
((O())) S
min
1152do max
then for every t 2 0 the following hold:
((O(t)) S ((0())) exp
2 mint
6v 20 ((((0)).
(e(()) - (0)] S
o min
The above theorem tells that, if the model is initialized with sufficiently small loss, the model's loss will converge to zero quickly, and the parameters will not move significantly from the initializa- tion. With the above theorem, can state the following corollary:
Corollary E.3. Suppose and B are initialized as balanced, i.e.:
A(O) A(0) = B(O)
Under the conditions of Theorem E.2, for every singular index i [d] and all t 2 0: (i(A(()) (B()(()) and (o.(A(t) oz (A(A)))) S
Gmin
4v2d
Consequently, the stable rank of A(t) remains bounded below by
A
||A(l))lr
Gmain
4v22
A(t
[A(0)
4/2d
E.3.1 PROOF OF THEOREM E.2
We begin the proof of the thcorem by noting that the Jacobian J() is a Lipschitz function, as stated in the following lemma:
Lemma E.9. The Jacobian matrix J(W), as defined in (72), is Vd-Lipschitz. Specifically, for any matrices W. V E R22xdd the following inequality holds:
[[(WW) - J(VII vallvec(W) vec(V)]
(73)
68

---

Published as a conference paper at ICLR 2026
Proof. Note that for each n-th observation,
Jn (e) 2vec
(X
0
X,
A
vec
X
B T
(
X x A
T
R
vec
E
Let Mi denote the (-th row of matrix M, and let M, denote its I-th column. We have
[[)( = =||X, A | + X,
=Ilej, e,, A lle e, B |F
A| 112 B 1l.
Now, suppose we observe all entries, i.e., N = d? Then for any fixed n, i, = im can be satisfied for all m [d]. meaning each element of A is observed times. Similarly, each element of B is also observed d times.
Therefore, we can upper bound the Frobenius norm of the Jacobian matrix by the Frobenius norm of the Jacobian under full observation:
|J(O)
Σ (I Al
+ |X l1²)
d (1) p
B
d e
By upper-bounding the spectral norm of the difference between two Jacobian matrices and applying the inequality above, we obtain:
[J(W) 2 aJ[( V
[[[W V)Il
$ d W V
which concludes the proof.
Next, we borrow lemma from Telgarsky (2021), which states that for Lipschitz function J, if we consider sufficiently small neighborhood around the initialization (0), then the singular values of the Jacobian J (O) remain close to those at initialization:
Lemma E.10 (Lemma 8.3 in Telgarsky (2021)). If we suppose || (+) vec (O)) I
Emin
2v
have the following:
miin (J(9)) 2 Fmin
mmax (J(O)) S
30 max
2
2
where we denote min amin (J))( and max maxx (J(O())).
For simplicity, we denote 0 as the vectorized version of θ i.e., 0 - vec(O). We define the time step T, which the first time step when the trajectory (t) touches the boundary:
inf
[[((t) e(0)l 2
Tmin
120
2
We now demonstrate the convergence of the loss when t [0, 7] using the following lemma.
Lemma E.11. For all E [), the loss defined in (67) converges as follows:
1
Z(O(1)) < ((((()) exp - min'
where define min Jmin (J())
69

---

Published as a conference paper at ICLR 2026
Proof. Recall that the time derivative of the loss be written as follows, according to (71):
((0())) = (F( - y) J(O(I))
(F( y) ((0())) (F(O(t)) y)
noting that
(() -oen -J((())) y).
By Lemma 10,0, for any [0, r], we can upper bound the above term follows:
(())))) S Ami (J(O(t))((((t))) [IF(@(t)) yll
1 2
2 min ((O()))
Applying Grönwall's inequality gives:
1
[(O(t)) - (() exp (-2omm)) fort [0,
The above lemma shows that the loss decays rapidly to zero if (()) stays within a small neighborhood around the initialization. We now show that the loss converges quickly near initialization, then ((t) does not move far from its initial value:
Lemma E.12. Let min & mmin (min(((((O))) and max - max (J()) For all distance between the weight vector at time and the initial weight vector is bounded by:
[0, r). the
[0(()) 0(0) < 6/20max V
min
Proof. We start by evaluating the distance between O(t) and (()) using Lemma E.10:
l/ (ss) ds
|(((1) 0(0)1l
- =)|l ds
m (J(())))) - y ds
3
1 [IF(((()) yl ds.
$ 2 mmxx
By Lemma 1.11, we know that the objective function ž(e) satisfies:
1 a
[F()(()) y/1² < [[((())) y exp
min'
2
Taking the square root of both sides, we obtain:
(
(- 1
(F((((t) yl1 < [F(((()) yll exp
mintt
Substituting this into the previous inequality:
(ie(t) 0(0) < max [(((0)) yl / /'' exp -
-
1 ds
min
s max
[F((()) y|!
min
where we used the fact that:
[( exp( (xp(-C's) ds S C
for C > 0.
70

---

Published as a conference paper at ICLR 2026
By combining Lemmas E.11 and E.12, we obtain the following results:
[()(t)) = (() exp
(-2omni)) minn
0
(74)
6v maxx max (()())).
[0(1) - ( S
(75)
min
which hold for & E [0, r]. If we can demonstrate that 7 = co, the proof is complete.
Actually, if initialize (0)) satisfy the condition:
o min
(() I
1152do max
and substitute this condition into (75), we obtaín an upper bound for 10(1) - 0(0)]l:
[[@(t) - (()) I
6v20
Omnn
min
V1152do
4vd
Recall the definition of which is the first time when O(t) touches the boundary of the small ball around the initialization:
2 inf $ (1(()) Gwin
0(0)1 Z
min
120
2vd
O()) cannot ever touch the boundary. This is because []O(t) ((0) bounded above by 4vd' which is strictly less than Zwin. Therefore, the 2 parameter will remain inside the ball indefinitely, meaning T = oo. This completes the proof of theorem.
However, with the condition (() <
T152do max Emin.
E.3.2 PROOF OF COROLLARY E.3
First, we establish the equality o, (A(t)) and (0) are initialized as "balanced". satisfying A(0) T A()) B() T. By Lemma F.4, this balanced condition ensures that the singular values of (t) and Btt remain identical for all t 2 0:
= oB(((t) for all i [d]. Corollary E.3 assumes that A())
0. (A()) - o. (B()))
Second, we address the change in the singular values of combined parameter matrix @ (related to A(t) and (t)). Theorem E.2 states that under a specified condition on the initial loss, (() s 1152do the deviation of (t) from its initialization (() is bounded for all t 2 by:
[I()) - (0))] < 4
Let K Zmin By Weyl's inequality, (o(X))o,(Y)) < X Y l12. and noting that ||-112 S 11-1|x, Avd'
we have for all i E [d]:
(o,(O(t)) - (O(0)) < [0(() (
< | (0)] < K.
This inequality allows us to establish bounds for [[)[]]] (using reverse triangle inequality) and its largest singular value o (O)) (0)]l
1 (t)l 2 [0(0]] r - K,
σ 2 (t)) の (0) K.
This yields the following lower bound on the stable rank of (tt):
(
(10001l- -K)
[0(0]]
Amin
4vd
(
o. (O(O)) +K
[
Mmin 4vd
71

---

Published as a conference paper at ICLR 2026
Furthermore, the balancedness condition implies A(t) A(t) = B(t) By the definition of (t), ( (t)) A ( + B(t) this leads to ( @(t) = 2A(t) A().. This
relationship implies (())))) = (A(t) for all i. Substituting this into the bounds for (t), have
A 2 ||A())] K/v2, A - ||A())]2 + K/v2.
This leads to the final lower bound on the stable rank of A(t) (which, by balancedness, is equal to that of (()):
(
||A(O)]le K/V2 2
(I|A(0))lE
Omin
4v
A(tl
+ [(0))2+KK/V)
[A(0))l22 +
Amia
4v/dd
72

---

Published as a conference paper at ICLR 2026
F USEFUL LEMMAS
Lemma F.1 (Adaptation of Lemma 1 and Theorem 3 in Arora et al. (2019)). For any time t, the product matrix (t)) E Rd.d can be decomposed into its singular value decomposition:
( = E or (t))))))))
where (1) are the singular values of (!), and ue (1), Vr (1) are the corresponding left and right singular vectors, respectively. Moreover, A, B are balanced at inifialization, i.e.,
A (0) B(O) T (0),
the time evolution of the singular values o, (t) represented as:
(1 -2 (V:()).)((W()))
I.....,,d
(76)
Lemma F.2. For any real-valued square matrix A Rddd, the absolute value of its determinant equals the product of its singular values:
dot [A) = or
r=1
where oy are the singular values of A.
Proof. We express A using SVD: = UEVT Applying the determinant to both sides, we get:
det( (A) = det deEVV
= (U' det (et(())el(())
Here, U and V have orthonormal columns, and Σ is diagonal with singular values along its main diagonal. Since the determinant of an orthonormal matrix either +1,
d
[det(A)] det
II dr- r=1
Lemma F.3 (Determinant of A(t)). Consider matrix A(t) E Rdd initialized as det(A(0)) > Then, det(A(() 0 0fror all ( 0.
Proof. This follows directly from Lemma F.1 and F.2. Since the singular values are initialized as positive, and their evolution continuous according the given differential equation, they cannot become zero or negative. Therefore, A(t) maintains its sign of the determinant at initialization throughout the optimization process.
Lemma F.4 (Adaptation of Lemma 8 in Razin & Cohen (2020)). Consider product matrix (t) A(t) E Idxdd where A(t) and (t)) are of equal size and balanced at initialization. Under these conditions, the following equality holds for all t and all singular values:
Tr (W()) o, (A(1)) a, (B())²
where o-(-) denotes the r-th singular value of the respective matrix where T E [d]. Moreover, if det (A(0)) and det (B))) both positive, then by Lemma F.3, we guarantee that for all t 2 0:
det (W (t)) det (A(t))² = det (B(t))
Lemma F.5 (Adaptation of Theorem in Arora al. (2019)). Consider product matrix W(t) A(t) B() Idx We can guarantee A(t) and (t) are analytic functions of t. As a result. W(t)
also function t.
73

---

Published as a conference paper at ICLR 2026
Lemma F.6 (Lemma 10 in Razin & Cohen (2020)). Let f.g [0, - R be real analytic functions
such that f(k) 9 (0) for all k N U {0}. Then, f(t) g(t) for all 2 0.
Lemma F.7 (Positive Semidefiniteness of )BA"), For matrices A, E IR..d, if B is positive semi-definite, then both ABA" and A' BA are positive semi-definite.
Proof. For any vector R
ABA = 2()
since B is positive semi-definite matrix. In the same way, for any vector R we have:
A BAx (Ax) B(Ax) 20
which concludes the proof.
74


Reviews:
[{"review_id": "pTqjiKicyu", "reviewer": "Reviewer_Qs2f", "summary": "Matrix completion by deep linear networks is an important model problem for understanding a variety of phenomena in the training of deep neural networks. It was previously observed empirically that the such models have a tendency to discover low-rank solutions. In the case of two-layer networks, it was observed that this is the case if and only if the observations are \"connected\" in a suitable sense.\n\nThe reviewed work attempts to extend this understanding to the case of deeper networks (which are known to provide low-rank solutions even for disconnected observations). As a starting point, it uses the fact that the characterization of low-rankness through connected observations proceeds by observing that disconnected observations lead to decoupled gradient dynamics of different degrees of freedom. \n\nTo this end, the reviewed work definines a notion of \"coupled gradient flow dynamics\" and shows that under diagonal (and thus disconnected) observations and a structured initialization scheme, the training dynamics are coupled if and only if the initialization is diagonal. It implicitly characterizes the singular values of the converged matrices in this case.\n\nFinally, the authors aim to illucidate the loss of plasticity phenomenon by showing that fast convergence of a pretrained network prevents significant decay of the stable rank.", "questions": "Typos:\nLine 306 \"of of\"\n\n1. You solve the equations (8) (9) numerically. Can you show that this system has a unique solution? If not, how do you know that the solutions of this system of equations characterizes the training behavior?\n\n2. On a related note, what is the benefit of solving (8) (9) over directly observing the training trajectories under the (fairly restrictive) conditions where Theorem 3.3 holds?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- This work addresses the important and thorny question of implicit bias in deep learning \n\n- Generalizes the independent observations mechanism beyond the 2 layer case\n\n- Provides an potentially useful perspective on the plasticity phenomenon", "weaknesses": "Maybe due to the challenging nature of the questions studied, the connection between the highly restricted theoretical setting and the claims about the behavior of the real algorithm (deep learning or even just deep matrix completion) is somewhat tenuous at times. I am especially doubtful about the additional insight of the implicit characterization (8), (9). If it is possible to derive additional insight from it rather than just solving it numerically, this would significantly strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DhbGsOKxhJ", "reviewer": "Reviewer_TLLi", "summary": "The paper uses deep linear matrix factorization as a controlled setting to explain why depth strengthens the implicit bias toward low rank. It identifies coupled vs. decoupled training dynamics as the underlying mechanism: in depth-2, coupling depends on observation-graph connectivity; in depth $\\ge3$, coupling arises generically and drives stronger rank-1 attraction. Using diagonal observations and a family of initializations, the authors derive limiting singular-value formulas and connect the mechanism to loss of plasticity (LoP) under warm starts.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A clear, mechanism-first account that unifies several scattered observations about low-rank bias and depth.\n\n2. Nontrivial analytical traction via diagonal observations and limiting SVD characterizations, with numerics that mirror the theory.\n\n3. Bridges the mechanism to LoP with formal statements (stable-rank lower bounds) rather than just empirical anecdotes.", "weaknesses": "1. Scope of formalism: The most rigorous theorems rely on diagonal or highly structured observation patterns and gradient-flow analysis. This leaves a substantial gap to practical regimes, for example finite-step SGD with noise/momentum and unstructured sparsity. Without non-asymptotic bounds in these regimes, it’s unclear how predictive the theory is for typical training runs.\n\n2. Initialization dependence: Several results hinge on a specific initialization family that tunes coupling. While the intuition is compelling, robustness across standard random inits is argued more heuristically than proved. This creates ambiguity about when the rank-1 bias reliably manifests.\n\n3. Transfer to nonlinear/realistic setups: The LoP story is precise for depth-2 linear models and toy expansions of the observation set. Generalization to deep nonlinear networks, noisy completion, and rectangular cases is suggestive but not sealed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "67wV05FHIO", "reviewer": "Reviewer_U2hd", "summary": "This paper explores the use of deep matrix factorization for matrix completion tasks, with a focus on how network depth influences training dynamics and the implicit bias of neural networks. Building on the prior work of Bai et al. (2024), the paper emphasizes that coupled dynamics are the core mechanism behind this low rank bias. For deep networks, the dynamics are coupled, meaning that even if the observed entries are decoupled, the training process strongly tends toward a low-rank solution. This bias arises from the inherent interdependence between layers in deep factorizations, which naturally favors low-rank solutions, regardless of the observation pattern. The paper also discusses the phenomenon of plasticity loss in matrix completion tasks, where models struggle to adapt to new information after initial training, a problem particularly evident in shallow networks. The paper shows that deep models mitigate plasticity loss through their low-rank bias, enabling them to maintain good adaptability even when trained on limited data.", "questions": "1. It seems to be focused on a very specific case (2x2 matrix), and while it provides some detailed characterization, it is weak overall. Additionally, I believe Bai et al. (2024)'s Theorem 2 offers a more general result that already characterizes the explicit relationship between initialization scale and alignment. Could the authors clarify how Theorem 3.1 adds value over this more general result?\n2. The definitions of couple and decouple seem to have similar characterizations in Bai et al. (2024) (Definition A.5). Would it be possible to cite this source to avoid redundancy and clarify the connection?\n3. Why does Eq. (7) consider such a special initialization? In practice, we usually consider random initializations rather than the specific initialization used here. Could the authors provide a justification for using this specific initialization in their analysis?\n4. What is the relationship between the implicit equation and low-rank? Why does satisfying the implicit equation imply low-rank? Could the authors analyze how this implicit equation precisely corresponds to low rank, especially for the case where $L = 3$? I think this is the core argument of the paper (that coupling leads to low-rank solutions), but Theorem 3.3 reduces this to an implicit equation without further analysis. The low-rank results are instead verified through numerical experiments (e.g., Figure 2), which weakens the theoretical contribution. Could the authors provide a more detailed theoretical explanation of this?\n5. Regarding Line 306: I think the argument that increasing depth promotes low rank might have a trivial reason behind it. For example, in the depth-2 case, the origin is a strict saddle point, and a very small initialization is required for the escape time to be long, which enables alignment. However, in deeper networks, the origin is no longer a strict saddle point, and the escape time significantly increases. This suggests that stacking more layers inherently leads to a smaller initialization scale. Comparing the rank under the same initialization scale might not be fair. \n6. How do the authors argue the differences between matrix completion tasks and real-world neural network tasks in terms of data characteristics? Real neural networks typically sample continuous data, which is almost surely connected. In contrast, matrix completion tasks involve discrete sampling. Is this discreteness unique to matrix completion tasks?\n7. In the Loss of Plasticity paper (Kleinman et al. 2024), the authors compare networks of different depths, finding that deeper networks show more pronounced phenomena at critical learning stages (Fig. 1). This seems contradictory to the claim that 3-layer networks exhibit more coupling than 2-layer networks, leading to low rank early on and thus avoiding plasticity loss. Could the authors discuss this discrepancy further?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written and has a clear overall structure, making it relatively easy to read.\n- The paper addresses a key issue in matrix completion, specifically the role of network depth in shaping training dynamics and the implicit bias toward low-rank solutions. This focus provides valuable insights into how deeper networks perform better in this context.\n- The paper builds on and extends the work of Bai et al. (2024), with a emphasis on the coupling dynamics and their relation to low-rank solutions. The paper does well to highlight the phenomenon of low-rank bias in deep networks, showing how deeper networks (L ≥ 3) exhibit stronger tendencies to converge to low-rank solutions, even when trained with limited data. The use of the theoretical framework surrounding coupled and decoupled dynamics is a solid foundation for exploring the behavior of deep networks in matrix completion tasks.\n- The analysis of plasticity loss, especially the argument that deeper networks avoid this issue better than shallower networks, provides new insights to this phenomenon.", "weaknesses": "- While the paper’s theoretical results are solid, some claims feel weak or underexplored. For instance, Theorem 3.1 is focused on a very specific case (2x2 matrices), and while it does provide some insights, it lacks generality. Additionally, Theorem 3.3 is reduced to an implicit equation that is not deeply analyzed.\n- Although numerical experiments (e.g., Figure 2) are used to validate the low-rank results, the theoretical argumentation feels limited. The paper would benefit from a more detailed analysis of how the implicit equation correlates with low-rank solutions, particularly for deeper networks (L ≥ 3). \n- The definitions of \"couple\" and \"decouple\" (in the context of training dynamics) are important for understanding the paper’s core argument, but they have been discussed in similar terms in Bai et al. (2024) (Definition A.5). The paper would benefit from clearer citations to this previous work to avoid redundancy and ensure clarity.\n- The discussion of special initializations, especially in Eq. (7), is quite restrictive and doesn't align well with practical network initialization strategies, which are typically random. This narrow focus on a specific initialization limits the applicability of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xi3gzTJvTd", "reviewer": "Reviewer_DsYE", "summary": "The paper studies implicit low-rank bias in deep linear matrix factorization for matrix completion and connects it to loss of plasticity when warm-starting from partially trained solutions. The core thesis is that while L=2 models exhibit coupling only when the observation pattern connects entries, depth >= 3 architectures induce structural coupling regardless of observation connectivity, yielding a stronger implicit bias toward low rank. The paper provides theoretical results on coupled/decoupled regime and on plasticity loss under warm starts supported by toy experiments.", "questions": "* Would it be possible to go beyond diagonals? Can the authors extend analysis/conclusions to non-diagonal, sparse patterns (even banded or block-diagonal) to demonstrate generality of the depth-induced bias? The main theorem(s) only seems to cover an extremely stylized regime which is much narrower than generic matrix completion settings. Right now, I believe this setup is quite limiting to the generalizability of said findings as this makes the conclusions hinge on a stylized regime.\n\n* How robust are the results to generic random initialization and to noisy/unequal diagonal entries? Can the authors provide some characterization of formal continuity/stability arguments in these cases?\n\n* Given that many works nowadays, both in theoretical analysis and empirical methods, focus on adaptive optimizers (e.g., Adam and its variants etc.), I wonder how these findings change under these optimizers? Many works ([1][2] to just name a few with [1] also focusing on matrix completion) have explored the implicit bias of these types of optimizers with some focusing on the role of how adaptive optimizers provide pre-conditoning in certain tasks/testbeds similar to that of this paper. \n  * Can the related works section be made a bit clearer in terms of distinguishing implicit reg/bias between 1st order and adaptive, etc. 2nd order optimizers (see below)? I think mentioning these works that focus on the implicit bias of adaptive optimizers (beyond just GD) like [1] and [2], among others, are worth contextualizing (I see the paper already cites the \"The implicit bias of adam on separable data\" but it's clumped together with several other works under a generic sense of implicit regularization for GD etc.); this would better help frame this paper's relevance and contribution while improving the flow. Or one could better structure related works according to the testbed/problem (e.g., compressed sensing, matrix factorization, etc.). I'm also not sure why the related works is in the appendix.\n  * Adding optimizer ablations (Adam/Adagrad/SGD) would clarify whether depth-induced coupling is robust across optimizers; for instance, core results (Prop. 3.2, Thm. 3.3; Thm. 4.2/4.3) are proved under gradient flow but practical claims and CNN experiments implicitly target regimes where training with Adam etc. is typically more standard. It's also unclear whether the depth-induced coupling and ensuing low-rank spectra hold beyond the GF/SGD limit. I view this optimizer-dependence gap as a key reason to temper claims of practical significance (despite the mention of practical tests using CNNs etc.).\n\n* For figs. 10-13 in Appendix, for the CNN/resnet experiments, could the authors please detail how convolutional kernels are reshaped, which layers are measured, provide seeds/variance/CI to support trends, etc.\n\n* The plasticity theorems are insightful but tend to be confined to toy regimes.\n\n[1] \"Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks\" Zhao et al. (2022)\n\n[2] \"On the Implicit Bias of Adam\" Cattaneo et al. (2024)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Provides an intuitive coupled vs. decoupled framework explaining how depth alone can produce low-rank bias.\n* Theoretical results are mathematically careful, with complete proofs and well-presented figures.\n* The paper connects depth-induced bias to plasticity loss, an emerging topic in optimization and continual learning.\n* Writing and presentation are clear and well-formatted.", "weaknesses": "The main theoretical contributions rely heavily on diagonal observation structures and specific initializations which limits generalizablity of said findings; it is also unclear how the results extend to generic or noisy data patterns. Much of the clean spectral characterization is within this stylized regime. Additionally, the finding that deeper (linear) networks inherently exhibit a stronger\nlow-rank bias than shallow networks is not an entirely novel finding though I appreciate that the novelty here is the coupling mechanism + solvable diagonal case + plasticity proofs. For other weaknesses and more details, see questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)