SGD vs GD: Rank Deficiency in Linear Networks

Authors: aditya  varre
margarita  sagitova
nicolas  flammarion
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:32:43

Abstract: In this article, we study the behaviour of continuous-time gradient methods on a two-layer linear network with square loss. A dichotomy between SGD and GD is revealed: GD preserves the rank at initialization while (label noise) SGD diminishes the rank regardless of the initialization. We demonstrate this rank deficiency by studying the time evolution of the *determinant* of a matrix of parameters. To further understand this phenomenon, we derive the stochastic differential equation (SDE) governing the eigenvalues of the parameter matrix. This SDE unveils a *replusive force* between the eigenvalues: a key regularization mechanism which induces rank deficiency. Our results are well supported by experiments illustrating the phenomenon beyond linear networks and regression tasks.

Main paper: 
 introduction: 
 Deep neural networks have significantly advanced machine learning in recent decades. A key attribute of these models is their ability, despite being heavily overparameterized, to learn effective representations which generalizes well across different tasks. This capability has sparked substantial interest in understanding how neural networks learn internal representations for specific tasks [Bengio et al., 2013] . Gaining deeper insights into these mechanisms is crucial for enhancing model interpretability and refining training and application methodologies in real-world scenarios.The success in learning these representations is often attributed to the gradient methods used in training. These methods navigate complex non-convex landscapes, finding solutions that not only minimize the training objective but also yield effective representations. They achieve this generalization while avoiding the spurious features that could potentially arise from the models' large number of parameters. Empirical studies have shown that the stochastic noise in gradient algorithms enhances generalization [Keskar et al., 2017 ] by favoring solutions with simpler structures that mitigate spurious features [Andriushchenko et al., 2022] . This paper address the overarching question:How does stochasticity facilitate the discovery of solutions with simplified structures?We explore this question using a simplified model: a single hidden-layer linear network. Despite lacking non-linearity, such networks capture some intricate phenomena of real-world deep networks and have been extensively studied to understand convergence [Arora et al., 2019a , Min et al., 2021] , learning dynamics [Saxe et al., 2014] , and the implicit bias of optimization algorithms [Gunasekar et al., 2017 , Soudry et al., 2018] . Our work builds on this foundation by comparing stochastic algorithms with their deterministic counterparts, focusing on how these differences influence the learning of simpler structures. Specifically, we analyze vector regression on two-layer linear networks trained with both gradient flow and stochastic gradient flow methods. Our contributions include:• In Section 4, we track the evolution of the determinant of the parameter matrix under gradient flow and stochastic gradient flow. We show that stochastic gradient flow drives the determinant towards zero, effectively removing irrelevant direction(s). • In Section 5, we derive a stochastic differential equation that describes the behavior of the eigenvalues of the parameter matrix. This analysis reveals a repulsive force between eigenvalues that pushes them apart and a geometric Brownian motion that pulls them toward zero. • In Section 6, we discuss the generalizability of our approach beyond square loss and various noise models, including discrete step sizes. Finally, we present experimental results in Section 7 that support our theoretical findings. 
related work: 
 Our work lies at the convergence of distinct research topics:Effect of SGD on generalization. The relationship between the stochasticity of SGD and its generalization capabilities has been extensively examined [Mandt et al., 2016 , Jastrzebski et al., 2018 , He et al., 2019 , Hoffer et al., 2017 , Kleinberg et al., 2018] . Notably, SGD tends to yield models with superior generalization compared to gradient descent [Keskar et al., 2017 , Jastrzebski et al., 2018 , He et al., 2019] . Various explorations into this phenomenon have been conducted through various approaches: hypothesizing that SGD favors flatter minima linked to better generalization, as opposed to sharp minima associated with poor generalization [Hochreiter and Schmidhuber, 1997 , Keskar et al., 2017 , Andriushchenko et al., 2023] , using a random walk on a random landscape model to understand the impact of stochasticity [Hoffer et al., 2017] , proposing that the inherent noise in SGD smooths the loss landscape [Kleinberg et al., 2018] , and exploring the implications of dynamical stability [Wu et al., 2018] .Stochastic dynamics and Label Noise. Recent literature has explored label noise-driven Gradient Descent as an effective method to probe the beneficial impact of stochasticity on generalization, with two distinct perspectives emerging. Firstly, an asymptotic view on general model parametrization is considered, where Blanc et al. [2020] , Damian et al. [2021] suggest that stochastic dynamics preferentially optimize a hidden objective linked to the curvature of the loss. In a related vein, Li et al. [2021] demonstrates appropriate limiting dynamics on the manifold of interpolators through time rescaling. Secondly, specifically for diagonal linear networks, HaoChen et al. [2021] , Pillaud-Vivien et al. [2022] observe a similar collapsing effect due to label noise but with a finer characterization of the limiting process. Finally, in the absence of label noise, Pesme et al. [2021] , Even et al. [2023] have characterized the outcomes of stochastic GF and GD for diagonal linear networks as the solutions to an implicit regularization problem that results in sparser solutions than without stochasticity. Recently, Ghosh et al. [2023] further exhibit a similar sparser features effect for single-neuron autoencoder. Chen et al. [2023] provides a condition under which an invariant set is attractive for SGD -characterizing the local behavior around these sets. The paper also studies linear networks in a teacher-student setup, however due to structured label-noise [Chen et al., 2023, A2 in p.30 ], the analysis falls short of capturing the repulsive force in the singular values.Linear Networks. The study of two-layer linear networks has been explored extensively, particularly when optimized using gradient flow on the square loss, across various settings including zero-balance initialization and whitened data Fukumizu [1998] , Saxe et al. [2014 Saxe et al. [ , 2019]] , Braun et al. [2022] .Early work by Saxe et al. [2014 Saxe et al. [ , 2019] ] elucidates the temporal changes in the singular values of the predictor, assuming decoupled dynamics and a specific data-dependent weight initialization. This condition is broadened by the analyses of Fukumizu [1998] and Braun et al. [2022] , Tarmoun et al. [2021] , who apply solutions from a matrix Riccati equation to characterize the weights dynamics under full-rank network initialization. Furthermore, Gidel et al. [2019] extends the existing framework by relaxing the whitened data assumption, conducting a perturbation analysis, and discussing the temporal evolution of the weight matrices' singular values. Additionally, Varre et al. [2024] eliminates the need for zero-balanced and full-rank initializations. Their study provides detailed formulas for weight evolution as a function of the initial scale , also studies a simple version of a stochastic flow without the drift. Wang and Jacot [2023] studied the implicit bias of SGD with ℓ 2 -regularization.Matrix valued stochastic process and their eigenvalues. Stochastic process on the space of symmetric (or Hermitian) matrices and the evolution of their eigenvalues are well studied since Dyson [1962] . These techniques were further developed by Bru [1989 Bru [ , 1991] ] to study perturbations of principal component analysis and the eigenvalues of Wishart processes. Norris et al. [1986] , Graczyk and Małecki [2013] applied SDE-based techniques to study the eigenvalues and eigenvectors of Brownian motion on ellipsoids.3 Linear networks and continuous-time gradient method Notation We use ⟨., .⟩ to denote the inner product, i.e., ⟨u, v⟩ = u ⊤ v for vectors, and ⟨A, B⟩ = Tr (AB ⊤ ) for matrices. I d denotes the identity matrix of dimension d and 0 p×k denote the matrix with all zero entries of dimension p × k.Vector regression. We study the vector regression problems with inputs x 1 , . . . , x n in (R p ) n and outputs y 1 , . . . , y n in (R k ) n . We consider the minimization of the square loss over a class of parametric modelsH = {f θ (•) : R p → R k | θ ∈ R d } specified in the next paragraph.The train loss therefore can be written asL (θ) = 1 2n n i=1 ∥y i -f θ (x i )∥ 2 .Parameterization with a linear network. We focus on two-layer linear neural networks of width l ∈ N * . The model is described by the parameterization θ = (W 1 , W 2 ), where W 1 ∈ R p×l and W 2 ∈ R l×k , and the functionf θ (x) = W ⊤ 2 W ⊤ 1 x.This model is linear with respect to the input x. In terms of expressivity, it is comparable to the linear class of predictors, represented as f β (x) = β ⊤ x, where β equals W 1 W 2 . Throughout our analysis, we denote the equivalent linear predictor of the network as β. A key aspect of this parametrization is that the prediction function f θ is positive homogeneous of degree 2 with respect to θ: specifically, for any λ ∈ R, f λθ = λ 2 f θ . This property mirrors that of two-layer ReLU networks and significantly influences the loss landscape navigated by the parameters θ. It is important to note that this parameterization introduces some redundancy, a single linear predictor β can have multiple representations W 1 , W 2 such that W 1 W 2 = β. Some representations have a rich structure whereas other resemble random features. For example, consider the case of scalar regression (k = 1), for a vector β there exists rich parameterizations where all the neurons, i.e., columns of W 1 align with β and also some lazy structures where W 1 resembles a random matrix [Chizat et al., 2019 , Varre et al., 2023] .Train loss. By defining X ⊤ = [x 1 , . . . , x n ] and Y ⊤ = [y 1 , . . . , y n ], the loss function is given by:L (W 1 , W 2 ) = 1 2n ∥XW 1 W 2 -Y ∥ 2 . (3.1)For simplicity, we adjust for the normalization factor n by rescaling the data to (X, Y ) ←( X / √ n, Y / √ n), thereby implicitly considering it in the loss function without directly mentioning n in the formula. Note that the loss is non-convex in W 1 , W 2 .Gradient flow. The dynamics induced in parameter space by running GF on Equation (3.1) is given by p+k) and denoting the residual matrix by R = X ⊤ (Y -XW 1 W 2 ), the evolution of Θ can be written asdW 1 = -∇ W1 L (W 1 , W 2 ) dt = X ⊤ (Y -XW 1 W 2 )W ⊤ 2 dt, (3.2) dW 2 = -∇ W2 L (W 1 , W 2 ) dt = W ⊤ 1 X ⊤ (Y -XW 1 W 2 )dt. (3.3) Introducing the block matrix, Θ = W ⊤ 1 | W 2 ∈ R l×(dΘ = dW 1 ⊤ | dW 2 = W 2 R ⊤ dt | W ⊤ 1 Rdt = W ⊤ 1 | W 2 ï 0 p×p R R ⊤ 0 k×k ò dt.The gradient flow can therefore be compactly written asdΘ = ΘJdt, where J = ï 0 p×p R R ⊤ 0 k×k ò . (3.4)The gradient flow (GF), when expressed in this form, reveals an inherent multiplicative structure with respect to Θ in the gradient of the loss. As we see in subsequent sections, this representation of the gradient flow with block matrices proves to be very convenient.Label noise gradient descent. Label noise gradient descent (LNGD) is a theoretically studied alternative to SGD that mirrors its practical behavior by sharing the geometric properties of the noise Blanc et al. [2020] , Damian et al. [2021] . Let ε t ∈ R n×k , where each entry of ε t is an independent Gaussian random variable. At iteration t, the labels are perturbed with this Gaussian noise at an intensity δ, i.e., Y = Y + √ δε t . The LNGD algorithm updates the iterates with a step size η in the direction of the gradient computed after the labels have been perturbed, as follows:W t+1 1 = W t 1 -η∇ W1 L Ä Y , X, W t 1 , W t 2 ä ; W t+1 2 = W t 2 -η∇ W2 L Ä Y , X, W t 1 , W t 2 ä ,where, by an abuse of notation, L (Y, X,W 1 , W 2 ) = 1 /2∥XW 1 W 2 -Y ∥ 2 .The iterates can then be restructured into a block matrix:Θ t+1 = Θ t -ηΘ t J t -η √ δΘ t ξ t , where ξ t = ï 0 p×p X ⊤ ε t ε ⊤ t X 0 k×k ò , (3.5)and J t is defined as in Equation (3.4).. We aim to model the aforementioned LNGD in continuous time using an appropriate SDE. Stochastic continuous-time counterparts of discrete stochastic gradient algorithms are favored for their enhanced amenability to theoretical analysis. We propose the following stochastic differential equation (SDE) to model LNGD in continuous time:dΘ = Θ î Jdt + ηδdξ ó , where dξ = ï 0 p×p X ⊤ dB t dB t ⊤ X 0 k×k , ò (3.6)where B t denotes a matrix Brownian motion in R n×k . LNGD as defined in Equation (3.5), can be interpreted as the the Euler-Maryama discretization of the above SGF with a stepsize η. Although the inclusion of step size in the continuous-time modeling of an SDE may seem counter-intuitive, it is a necessary component [Li et al., 2019b] . As all the terms of the SDE in Equation (3.6) are polynomial in Θ, both the drift and diffusion terms are locally Lipschitz continuous. Hence, the solution of the SDE is uniquely defined up to the explosion time τ ∞ [see, e.g., Khasminskii, 2012] . Furthermore, the explosion time can be proven to be infinite (τ ∞ = ∞ almost surely), by using that the GF does not diverge and applying the techniques outlined by Pillaud-Vivien et al. [2022, Proposition 10] .Initialization. The dynamics of gradient methods on homogeneous models are significantly influenced by initialization, which determines the regime they operate in-specifically, the lazy regime for large initializations and the rich regime for small ones [Chizat et al., 2019 , Woodworth et al., 2020] . Thus, the scale of initialization has garnered significant interest, particularly its impact on the training of linear and non-linear networks with GD [Woodworth et al., 2020 , Boursier et al., 2022] . It is observed that stochastic methods eliminate the dependence on initialization [Pesme et al., 2021] .Conserved quantities and balanceness. Gradient flows follow specific conservation laws along their trajectory [Marcotte et al., 2023] , maintaining characteristics of the initial conditions. For linear networks, this conservation manifests as the balanceness property [Du et al., 2018] , described by: Saxe et al. [2014] , Arora et al. [2018 Arora et al. [ , 2019b] ] have adopted balanced initialization, where ∆(0) = 0, to ensure that weight matrices remain low rank throughout the trajectory. However, unbalanced initialization do not preserve these simple low-rank structures, as aspects of the initial conditions persist.∆ = W ⊤ 1 W 1 -W 2 W ⊤ 2 = W ⊤ 1 (0)W 1 (0) -W 2 (0)W ⊤ 2 (0). As a result,In contrast, stochastic methods do not adhere to these conservation laws [Ziyin et al., 2023] and the evolution of the imbalance ∆ for SGF isd∆ = d W ⊤ 1 W 1 -W 2 W ⊤ 2 = tr (XX ⊤ ) W 2 W ⊤ 2 dt -k W ⊤ 1 X ⊤ XW 1 dt.While there is no diffusion term in the derivative, the matrices remain stochastic and no definitive conclusions can be drawn from this. However, in the case where k = p and X ⊤ X = I p , it can be shown that W ⊤ 1 W 1 -W 2 W ⊤ 2 → 0, indicating that the stochastic noise eliminates initial imbalance. Conclusion. Understanding how stochastic methods mitigate dependency on initialization requires exploring beyond the evolution of the imbalance ∆. To this end, we identify and discuss other conserved quantities, such as the determinant of the block matrix Θ ⊤ Θ in the following sections. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental Setting/Details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: The setup is discussed in the section 7.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
separation between gradient flow through determinant: 
 Here, we present our first separation result between GF and SGF. While the determinant of the parameters is preserved in GF, it is driven to zero by the stochasticity of SGF, leading to a simplistic low-rank structure.The theorem below demonstrates that the determinant of the parameters is preserved in gradient flow. Theorem 4.1. For the gradient flow defined in Equation (3.4), the following property holds,d det Θ ⊤ Θ = 0. Hence, det Θ(t) ⊤ Θ(t) = det Θ ⊤ 0 Θ 0 , where Θ 0 = Θ(0) is the initialisation at time t = 0.The proof presented in the App. B.1, is based on straightforward computations of the derivative of the determinant and the fact that the matrix J has zero trace. We note that the simplicity of the proof arises from the strategically chosen block structure of Θ. This result would have been less straightforward with different parametrizations, which likely explains why such a simple finding appears to be novel. The theorem implies that the determinant of M along the trajectory remains equal to the determinant at initialization. If Θ ⊤ 0 Θ 0 is full-rank initially, meaning the determinant is non-zero, the theorem ensures that the determinant of M remains non-zero. Consequently, the rank of Θ does not diminish along the trajectory. When l ≥ p + k, i.e., the hidden layer has a large width and W 1 , W 2 are initialized randomly from a Gaussian distribution, Θ ⊤ 0 Θ 0 has full rank almost surely. The theorem also reveals some implications regarding the impact of initialization scale. Note that λ min (A) ≤ n √ det A, indicating that when the scale of initialization is very small, at least one singular value of Θ is small.In contrast, the theorem presented below demonstrates that the determinant of the parameters converges to zero in stochastic gradient flow. Theorem 4.2. For the SDE, defined in the Equation (3.6), for t ≤ τ ∞ , the following property holds for the evolution of determinantd det Θ ⊤ Θ = -2ηδktr X ⊤ X det Θ ⊤ Θ dt. Hence, det Ä Θ(t) ⊤ Θ(t) ä = det Θ ⊤ 0 Θ 0 exp -2ηδktr X ⊤ X t, where Θ 0 is the initialization.Although the evolution of the parameters in SGF is random, the evolution of the determinant is deterministic. The theorem highlights a striking phenomenon: the noise in SGF diminishes the determinant along the trajectory, leading to a simplification of the network over time. The larger the noise and the stepsize, the faster the determinant vanishes. The vanishing of the determinant suggests that the rank of the parameters decreases by at least one, effectively eliminating some components. It holds for any initialization of Θ 0 and indicates how the SGF overrides some aspects of initialization.The proof uses the fact that stochastic Brownian term in the SDE, through Itô's calculus, introduces a negative drift, ultimately driving the determinant to zero (refer to B.3 for the proof).Limitations. Given the large width of the hidden layer, the determinant converging to zero does not fully reveal the complexity of the situation. It merely indicates that at least one singular value is approaching zero. Furthermore, the theorem provides limited insights when the determinant is already 0 at initialization, det Θ 0 = 0 which happens whenever l < p + k. Next, we explore the mechanisms behind this low-rank phenomenon, suggesting that the repulsive forces induced by stochasticity drive the spurious singular values to zero as seen in the right plot of Figure 1 .we consider the case where k = 1 and for notational convenience, we letW 1 = W, W 2 = a.Additionally, we assume that l ≤ p, however the results can be extended to any l.Warm-up: Comparison with diagonal networks. Let W = UΣV ⊤ be the singular value decomposition (assuming l ≤ p). The predictor β can be expressed asWa = UΣV ⊤ a = U [σ ⊙ c] , where c = V ⊤ a.This expression reveals a Hadamard product between σ and c, reminiscent of diagonal networks which are widely studied to understand the nonconvex dynamics of gradient algorithms [Woodworth et al., 2020 , Pesme et al., 2021 , Pillaud-Vivien et al., 2022] . In the context of diagonal networks, SGD is known to provably induce sparsity in predictions. Similarly, for linear networks, SGF may induce sparsity in terms of the singular value σ. We next derive the SDE governing the evolution of the singular values Σ of the weight matrix to gain a clearer understanding of the low-rank phenomenon.Scalar Regression. We assume that the data is isotropic, i.e., X = I p . Under these conditions, the loss function for scalar regression can be written asL (W, a) = 1 2 ∥y -Wa∥ 2 .(5.1)We train the above objective with SGF, formulated as follows,dW = (y -Wa)a ⊤ dt + ηδ dB t a ⊤ ; da = W ⊤ (y -Wa)dt + ηδ W ⊤ dB t . (5.2)where B t is the standard Brownian motion in R p . For analytical convenience, we rescale the time t → t /ηδ and use the process dX = 1 /ηδ(y -Wa)dt + dB t . The SGF can then be rewritten as,dW = dXa ⊤ ; da = W ⊤ dX. (5.3)Our focus is on understanding the evolution of the singular values of the matrix W. This aim is facilitated by considering the symmetric matrix M = W ⊤ W, whose eigenvalues are the squares of the singular values of W. Taking the derivative of M, we finddM = dW ⊤ W + W ⊤ dW + dW ⊤ dW = adX ⊤ W + W ⊤ dXa ⊤ + paa ⊤ dt.(5.4)Note that dxdy represents d[x, y] for any continuous semi-martingales x, y [see, e.g., Ikeda and Watanabe, 1981 , chapter 3 for reference].Eigenvalues of a matrix-valued stochastic process. We leverage tools from the study of eigenvalues of matrix-valued stochastic processes [Bru, 1989, Graczyk and Małecki, 2013] to derive the evolution of the eigenvalues of M in the theorem that follows.Theorem 5.1. Let s 1 > . . . > s l be the order of the eigenvalues of the matrix M defined by Equation (5.4). Let the collision time for the eigenvalues be defined asτ = {inf t : s i (t) = s j (t) for 1 ≤ i ̸ = j ≤ l}.(5.5)For t ≤ τ , the eigenvalues are semi-martingales given by the solution of the following SDEd(s i ) = pc 2 i dt + l j=1, j̸ =i s i c 2 j + s j c 2 i s i -s j dt + 2 » s i c 2 i Ä d Xä i (5.6) where c = V ⊤ a and Ä d Xä i = 1 /ηδ Ä ⟨u i , y⟩ -s i c 2 i ädt + dε i with u i being the i th column of U and (ε 0 , . . . , ε l-1 ) is the standard Brownian motion in R l . The evolution of c i and U are presented in the appendix B.5. This theorem can be interpreted as the stochastic counterpart to the evolution of eigenvalues previously described for linear networks by Arora et al. [2019c] , Varre et al. [2023] . The derivation of the eigenvalues is inspired by the work of Bru [1989] .The evolution of the eigenvalues features a key term highlighted in Equation (5.6) consisting of the sum of skew-symmetric elements sic 2 j +sj c 2 i/s i-sj . For a pair of indices (i 0 , j 0 ) with i 0 < j 0 and thus s i0 > s j0 , the term si 0 c 2 j 0 +sj 0 c 2 i 0 /si 0 -sj 0 positively influences the evolution of the larger eigenvalue ds i0 and negatively affects the smaller eigenvalue ds j0 . Therefore, this force is repulsive, driving the eigenvalues apart and increasing their gap. Another factor influencing the dynamics is the presence of Geometric Brownian motion, where the singular value σ i multiplicatively influences the Brownian motion ass i c 2 i Ä d Xä i, similar to what is observed in diagonal linear networks (refer to the previous discussion for similarities). This effect tends to pull the singular values toward zero. Together with the fact that (s i , c i ) = (0, 0) represents a fixed point of the dynamics, these two forces collectively push redundant singular values toward zero.To further understand the interplay of repulsive forces and geometric Brownian motion, we consider the evolution of the smaller singular value s p for l = p. Using the Ito chain rule, we analyze the evolution of log s p , expressed as,d(log s p ) = p c 2 p s p dt + 1 s p p j=1, j̸ =p s p c 2 j + s j c 2 p s p -s j dt -2 c 2 p s p + 2 c 2 p s p Ä d Xä p .Using that spc 2 j +sj c 2 p/s p -sj < -c 2 p , for all indices j, the repulsive force accumulates to -(p -1)(c 2 p /s p ) and the Ito correction term from the logarithm contributes an additional -2(c 2 p /s p ) (the GBM component) thus offsetting the positive drift of p(c 2 p /s p ). In the case of l ̸ = p, considering a polynomial x α with an appropriate α would demonstrate similar behaviour. This discussion outlines the forces at play, yet a complete characterization of the solution of the SDE Equation (5.6) remains missing. Moreover, we have not established that the eigenvalues avoid a.s. collision, i.e., the explosion time τ ∞ = ∞ which is in itself a significant challenge [Bru, 1989, Graczyk and Małecki, 2014] .A simplified two-vector problem. To enhance our understanding of the SDE governing the evolution of the eigenvalues detailed in Equation (5.6), we consider the large noise limit. In this scenario, the process described in Equation ( 5.3) simplifies to a purely noise-driven process without drift: dW = dB t a ⊤ ; da = W ⊤ dB t . This SDE exhibits notable symmetry; allowing for an analysis using a matrix with sub-sampled columns. Let S be any subset of 1, . . . , l, with (w i ) l i=1 representing the columns of W. We define W S ∈ R p×|S| as the subsampled matrix obtained by selecting columns w i where i ∈ S, and similarly, we define a subsampled vector a S by selecting the corresponding coordinates. The SDE restricted to the set S is structured as follows:dW S = dB t a ⊤ S ; da S = W ⊤ S dB t . To demonstrate that the columns of W align, we leverage the symmetry of the SDE by examining the restricted problem on every pair of rows S = {i, j}, and proving alignment within this subset. This approach leads us to consider the two vector problem (l = 2), where W = [w 1 |w 2 ] and w 1 , w 2 ∈ R p , a ∈ R 2 . We describe the behavior of the eigenvalues for this two-vector problem in the theorem below. Theorem 5.2. In the large noise limit, let s 0 > s 1 be the eigenvalues of W, the following properties hold, for t ≤ τ defined by τ = {inf t : s 0 (t) = s 1 (t)}, (a) s 0 , s 1 are greater than zero almost surely,(b) for α = (p -3)/2, s -α 0 is a super-martingale while s -α 1 is a sub-martingale.This model for l = 2 mirrors the dynamics of the Wishart process studied by Bru [1991] , motivating the exploration of the evolution of an appropriately chosen exponent of s 0 , s 1 . The first part of the theorem arises from the fact that s -α 1 s -α 2 is a local continuous martingale that cannot explode to infinity in finite time. The second part highlights a clear separation between the eigenvalues: one is a sub-martingale that consistently increases in expectation, while the other is a super-martingale that diminishes (note that the eigenvalues are raised to a negative power). This dynamic, coupled with the symmetry argument, suggests that for every pair of columns, there is a component that strengthens the alignment through its increases in expectation. Refer to App. B.6 for the proof.In this section, we derive the SDE of eigenvalues for the matrix of parameters evolving under SGF. This derivation provides deeper insights into the mechanisms contributing to low-rank behavior. Specifically, repulsive forces drive the eigenvalues apart, while the geometric Brownian motion pulls them towards zero. These forces, unique to training with SGF, highlight the regularization effects of stochastic methods compared to gradient flow. However, fully characterizing the solution of this SDE remains a challenging open problem we let as future work. 
generalization to other settings: 
 In this section, we generalize our results beyond the square loss and the label noise gradient flow. We consider the general framework of a loss function over the weight product W 1 W 2 defined asL (W 1 , W 2 ) = L(W 1 W 2 ) = E (x,y)∼D [ℓ(W 1 W 2 ; x, y)] ,In this framework, the loss function ℓ combines the prediction loss directly with the parametrized model f θ . This approach applies, for example, to classification problems using linear networks where ℓ might represent any classification loss and f θ = W 1 W 2 . It also directly extends to more complex architectures where f θ = σ(W 1 W 2 ) for an activation function σ, including settings like a self-attention layer with frozen value vectors. We denote the product by β = W 1 W 2 noting it solely controls the loss. We investigate the evolution of the weight matrix determinant for a general loss across various algorithms, from gradient flow to gradient descent, and demonstrate that a similar separation occurs due to stochasticity.Warm-up: Gradient flow. The gradient flow on the loss L can be written as the following, dΘ = ΘJdt,whereJ = ñ 0 p×p -∇ L(β) -∇ L(β) ⊤ 0 k×k ô . (6.1)Following a similar proof as in Theorem 4.1, we obtain that d det Θ ⊤ Θ = 0. For separable classification problem, the gradient flow converges to infinity [Soudry et al., 2018, Ji and Telgarsky, 2019] , hence, after appropriate rescaling, the layers are aligned, as shown by Ji and Telgarsky [2019] .Next, we contrast this result with the outcomes observed in stochastic and discrete algorithms.Continuous modelling of SGD. We consider the SGD algorithm with a batch size B. We denote the mini-batch version of the loss functions L and L as L B and L B , respectively. The SGD update with stepsize η can be represented with the following block structure,Θ t+1 = Θ t -ηΘ t J t -ηΘ t ξ t , where ξ t = 0 p×p - Ä ∇ L(β) -∇ L B (β) ä - Ä ∇ L(β) -∇ L B (β) ä ⊤ 0 k×k .We denote the SGD noise as g t = Ä ∇ L(β) -∇ L B (β) ä and the noise covariance asΣ t = E î g t (g t ) ⊤ ówhere the expectation is over all the minibatches. Following Li et al. [2019a] , the SGD update can be modelled with the following SDE, dΘ = -ΘJdt -√ ηdξ, where dξ = 0 p×p -Σ1/2 t dB t - Ä Σ 1/2 t dB t ä ⊤ 0 k×k . (6.2)The main difference with SGF is that, in overparameterized problems, the noise covariance is timevarying and decreases to zero upon convergence. Using Theorem B.3, the evolution of the determinant of M = Θ ⊤ Θ is given by d(det (M)) = -ηdet (M)Tr (Σ(t))dt and can be explicitly solved asd(det (M)(t)) = det (M(0))exp{-η t 0Tr (Σ(s))ds}.Hence, the decay in the determinant is governed by the integral ∞ 0 Tr (Σ(t))dt which is a stochastic quantity. Tr (Σ(t)) represents the strength of the stochastic noise, which, in over-parameterized regression, is proportional to the loss, i.e., Tr (Σ(t)) ∝ L (Θ) [Pesme et al., 2021] . Therefore, the rate of decay in the determinant depends on ∞ 0 L ((Θ(t))) dt, with slower convergence leading to a simpler model at convergence, as observed in the case of diagonal networks by Pesme et al. [2021] . The result above also holds for non-separable classification tasks where the noise of SGD drives the determinant to 0, a scenario not covered by the previous analysis of Ji and Telgarsky [2019] .Discrete gradient algorithms. We can extend the previous results to discrete (possibly stochastic) gradient algorithm. Both algorithms can be written asΘ t+1 = Θ t (I p+k + ηJ t ) ,for stepsize η and J t the possibly stochastic block gradient matrix defined in Equation (6.1). In the context of discrete algorithms, the determinant is controlled by the following lemma (refer to B.4 for the proof). |det Θ t+1 | ≤ exp Å - η 2 2 J t 2 F ã |det Θ t |.If the factor η 2 J tF ≤ 1 at every iteration t, the determinant is reduced by the discrete step size. However, there is a tradeoff: the sum S := ∞ t=0 η 2 J t 2 F can be finite, indicating that it does not completely drive the determinant to zero. Increasing η to increase S might lead to instability and divergence. Furthermore, since J t 2 F ∝ L (Θ t ), there is an additional tradeoff between convergence and the simplicity of the parameters. This illustrates how step sizes that produce non-convergent training loss patterns, such as the catapult effect [Lewkowycz et al., 2020] or the edge of stability mechanisms [Cohen et al., 2020] , can simplify the network's parameters. 
experimental evidence: 
 We consider a regression problem on synthetic data with n = 1000 samples of Gaussian data in R 5 (p = 5) with labels in R 2 (k = 2) generated by some ground truth β ∈ R 5×2 , the width of the network is l = 10. We use Gaussian initialization of the network parameters with entries from N (0, 1). Experiments details can be found in the appendix C. In the left plot of Figure 1 , we show the time evolution of the determinant of matrix M. As suggested by theorems 4.1 and 4.2, in the case without label noise, det Θ ⊤ Θ stays constant, while with the Label Noise of intensity δ = 2 it goes to zero with time. In the right plot of Figure 1 , we demonstrate the time evolution of the top-5 singular values of the matrix W 1 . Note that in the case of Gradient Flow all except the first k singular values (σ 0 and σ 1 ) stay at the same scale, while adding Label Noise forces smallest d + l -k singular values (σ 2 , σ 3 , and σ 4 ) to tend toward zero. Further experiments illustrate in Figure 2 the evolution of singular values of parameter matrix W 1 when optimized with SGD, for classification tasks and with ReLU network. These results also confirm that the beneficial effects of stochasticity hold in these contexts.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] Justification: All the graphs of the parameters evolution are accompanied with the 95% confidence interval calculated on the 20 runs with different random seeds. Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we demonstrate a distinct separation between GF and SGF when trained on linear networks. This separation is obtained by tracking the evolution of the determinant of the parameter matrix. However, while the determinant is a significant factor, it does not fully capture the implicit regularization effects. Notably, the determinant mirrors the imbalance u 2 -v 2 in diagonal networks represented by u ⊙ v, whose dynamics play a crucial role in attuning the implicit regularization across various algorithms [Woodworth et al., 2020 , Pesme et al., 2021 , Papazov et al., 2024] . Our analysis presents the initial step in deciphering implicit regularization for stochastic methods in linear networks, yet achieving a complete characterization remains a promising direction for future research. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The work doesn't entail models or datasets releases. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All relevant information is stated in the appendix section C. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Notation S d , S + d , S ++ d denote the set of symmetric, positive semi-definite and positive definite matrices in R d×d . We use ⊙ to denote the Hadamard product.Theorem B.1. For the gradient flow defined in Equation (3.4), the following property holds,d det Θ ⊤ Θ = 0. Hence, det Θ(t) ⊤ Θ(t) = det Θ ⊤ 0 Θ 0 , where Θ 0 = Θ(0) is the initialisation at time t = 0.First, we present a proof of this theorem, based on straightforward computations of the derivative of the determinant and the fact that the matrix J has zero trace.Proof. Let M = Θ ⊤ Θ. The dynamics of M are governed by the ODE,dM = dΘ ⊤ Θ + Θ ⊤ dΘ = Θ ⊤ ΘJdt + JΘ ⊤ Θdt = (MJ + JM)dt.Using the gradient of the determinant given in Proposition B.2, the determinant of M evolves as follows,d(det (M)) = ⟨∇det (M), dM⟩ = det (M) M -1 , MJ + JM dt, = det (M) M -1 , MJ + M -1 , JM = 2det (M) ⟨I p+k , J⟩ = 2det (M)Tr (J).Given that Tr (J) = 0, it follows that d(det (M)) = 0.Proposition B.2. For any matrix M in S ++ d , the first two derivatives of the determinant of M, denoted by det (M ) are the following (i) ∇det (M ) = det (M )M -1 (ii) For 1 ≤ a, b, k, l ≤ d, the second order partial derivative is given by∂ 2 det (M ) ∂M ab ∂M kl = det (M ) (M -1 ) ba (M -1 ) lk -(M -1 ) bk (M -1 ) la (B.1)Theorem B.3. For a stochastic process given by the SDE,dΘ = Θ [Jdt + dξ] (B.2)with Tr J = Tr ξ = 0, the determinant of the M = Θ ⊤ Θ evolves asd(det (M)) = -det (M)Tr [dξdξ]. (B.3)Proof. First, we compute the evolution of M = Θ ⊤ Θ using the Ito's product rule,dM = d Θ ⊤ Θ = dΘ ⊤ Θ + Θ ⊤ dΘ + dΘ ⊤ dΘThe last term is interpreted as a derivative of the finite variation and it should be computed using dt. (dB t ) ij = 0 and (dB t ) ij . (dB t ) kl = δ i=k∧j=l dt. Using Eq. (3.6),dM = [Jdt + dξ]Θ ⊤ Θ + Θ ⊤ Θ[Jdt + dξ] + dξΘ ⊤ Θdξ, = JMdt + MJdt + dξMdξ + dξM + Mdξ.Using the Ito chain rule, we can compute the evolution of determinant as following,d(det (M)) = ⟨∇det (M), dM⟩ + 1 2 a,b,k,l ∂ 2 det (M) ∂M ab ∂M kl dM ab dM kl ,The first term is⟨∇det (M), dM⟩ = det (M) M -1 , JMdt + MJdt + dξMdξ + dξM + Mdξ , = 2det(M) ⟨I p+k , J⟩ dt + 2det (M) ⟨I p+k , dξ⟩ + det (M) M -1 , dξMξUsing the property that Tr (J) = Tr (dξ) = 0. We get that ⟨∇det (M), dM⟩ = M -1 , dξMξ . For the second term1 2 a,b,k,l ∂ 2 det (M) ∂M ab ∂M kl dM ab dM kl = 1 2 a,b,k,l det M (M -1 ) ba (M -1 ) lk -(M -1 ) bk (M -1 ) la dM ab dM kl , = det(M) 2 a,b,k,l (M -1 ) ba (M -1 ) lk dM ab dM kl - a,b,k,l (M -1 ) bk (M -1 ) la dM ab dM kl ,Rearranging the terms in the summation, we get,a,b,k,l (M -1 ) ba (M -1 ) lk dM ab dM kl = a,b,k,l (M -1 ) ba dM ab (M -1 ) lk dM kl , = b,l a (M -1 ) ba dM ab k (M -1 ) lk dM kl , = b,l M -1 dM bb M -1 dM ll = b M -1 dM bb l M -1 dM ll , = Tr M -1 dM Tr M -1 dM .Similarly for the other term, we get, M -1 dM bl M -1 dM lb = b M -1 dMM -1 dM bb , = Tr M -1 dM M -1 dM .Note that the diffusion part of M -1 dM is dξ + M -1 dξM. Using thisTr M -1 dM Tr M -1 dM = Tr dξ + M -1 dξM Tr dξ + M -1 dξM = 0,as Tr dξ = 0. For the other term,Tr M -1 dM M -1 dM = Tr dξ + M -1 dξM dξ + M -1 dξM , = 2Tr [dξdξ] + 2Tr M -1 dξMdξ .Putting everything together, we get, |det Θ t+1 | ≤ exp Å - η 2 2 J t 2 F ã |det Θ t |.Proof. Note that because of the block structure of the matrix J t , its nonzero eigenvalues come in ±-pairs: ±σ 1 , . . . , ±σ m , moreover, since J t is symmetric, singular values of J t are the absolute values of eigenvalues, i.e. σ 1 , . . . , σ m . Then, the determinant of Θ t+1 can be written as the following,det Θ t+1 = det Θ t det (I p+k + ηJ t ) = det Θ t m i=1 (1 -η 2 σ 2 i ).Using that 1 -x 2 ≤ e -x 2 for all x, we can estimatem i=1 (1 -η 2 σ 2 i ) ≤ exp -η 2 m i=1 σ 2 i = exp Å - η 2 2 J t 2 F ã .We obtain the required inequality by observing thatm i=1 (1 -η 2 σ 2 i ) = m i=1 (1 -η 2 σ 2 i ) since each term 1 -η 2 σ 2 i ≥ 0 when η 2 J t 2 F < 1. Theorem B.5.Let s 1 > . . . s l be the order of the eigenvalues of the matrix M defined by Equation (5.4). Let the collision time for the eigenvalues be defined asτ = {inf t : s i (t) = s j (t) for 1 ≤ i ̸ = j ≤ l}. (B.4)For t ≤ τ , the eigenvalues are semi-martingales given by the solution of the following SDEd(s i ) = pc 2 i dt + l j=1, j̸ =i s i c 2 j + s j c 2 i s i -s j dt + 2 » s i c 2 i Ä d Xä i (B.5) where Ä d Xä i = 1 /ηδ Ä ⟨u i , y⟩ -s i c 2 i ädt + dε i with u i being the i th column of U and (ε 0 , . . . , ε l-1 ) is the standard Brownian motion in R l . The evolution of c i and U are presented in the appendix.Proof. The proof follows the approach of Bru [1989] . Let W = UΣV ⊤ be the singularvalue decomposition (see Def.D.1 involved with r = l and l < p and it will be the rank). Our focus is on understanding the evolution of the singular values and singular vectors of the matrix W. To derive the evolution of Σ, V we can consider the eigenvalues and eigenvectors of the PSD matrix processM. Note that M = VΣ 2 V ⊤ , let D = Σ 2 .Evolution of D and V Taking the derivative of M, we finddM = dW ⊤ W + W ⊤ dW + dW ⊤ dW = adX ⊤ W + W ⊤ dXa ⊤ + paa ⊤ dt. (B.6)We invoke the theorem D.2 we derived to give the eigenvalues of any matrix valued stochastic process. Note that VV ⊤ = I l , so some terms of the computation are not required.dD = I ⊙ N dt + I ⊙ d M dt + I ⊙ Ä d M Ä S ⊙ d M ää .and the evolution of the eigenvectors,dV = V Ä Q ∥ dt + S ⊙ ( Ndt + d M) äwhere you define,Q ∥ = I ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M äó 2 -S ⊙ îÄ S ⊙ d M ä î d M ⊙ I óó + S ⊙ Ä d M Ä S ⊙ d M ääwhere the matrix S is given byS ij = ® 0 if i = j, (s j -s i ) -1 o.w. N = V ⊤ (paa ⊤ )V = pcc ⊤ . d M = V ⊤ adX ⊤ W + W ⊤ dXa ⊤ V, = cdX ⊤ UΣ + ΣU ⊤ dXc ⊤ .Note that Σ = diag ((σ 0 , . . . , σ l-1 )) where σ 0 > σ 1 . . . > σ l-1 . Let D = Σ 2 and denote the entires of D as following, D = diag ((s 0 , . . . , s p-1 )). Note thatU ⊤ dX = U ⊤ ( 1 ηδ (y -Wa)dt + dB t ), = 1 ηδ U ⊤ y -Σc dt + U ⊤ dB t . Using Levy's characterization U ⊤ dB t is a Brownian motion in R l , lets call that d Bt . The diffusion part of d M (say dF) dF = ΣV ⊤ dB t c ⊤ + cdB t ⊤ VΣ, = Ä σ ⊙ d Bt ä c ⊤ + c Ä σ ⊙ d Bt ä ⊤ = dm t c ⊤ + cdm t ⊤ where dm t def = (σ ⊙ d Bt ).We are required to compute dF(S ⊙ dF) to compute the evolution of eigenvalues. Using the lemma D.4, we getdF(S ⊙ dF) = cs ⊤ Sdiag (c) dt -Ddiag (Sdiag (c) c) dt + Ddiag (c) Sdiag (c) dt, I ⊙ [dF(S ⊙ dF)] = I ⊙ cs ⊤ Sdiag (c) dt -Ddiag (Sdiag (c) c) dtThe element wise computation of this term gives the required result for evolution of eigenvalues.Evolution of c. Note that c = V ⊤ a. Computing the derivative using the Ito's product rule, we get,dV ⊤ a = V ⊤ da + dV ⊤ a + dV ⊤ da, = V ⊤ da + dV ⊤ VV ⊤ a + dV ⊤ VV ⊤ da, dV ⊤ V = îÄ Q ⊤ ∥ dt -S ⊙ dX äó , V ⊤ da = V ⊤ W ⊤ dB t + 1 ηδ U ⊤ y -Σc dt = Σd Bt = dm t + 1 ηδ U ⊤ y -Σc dt, dV ⊤ VV ⊤ da = -(S ⊙ dF)dm t . dV ⊤ VV ⊤ da = îÄ Q ⊤ ∥ dt -S ⊙ Ä Ndt + d M ääó cUsing the lemma D.6, D.5, D.4 and computing the element wise summation, we get the following evolution for dcdc i = - 1 2 l j=1 S ij (s i c 2 j + s j c 2 i )dt -c i l j=1 (S ij c 2 j ) Ñ k̸ =i,j s k S ki é -(p -2)c i l j=1 S ij c 2 i dt - l j=1S ij s j dt,+ σ i (U ⊤ dX) i (1 - l j=1 S ij c 2 j ) -c i j S ij σ j c j (U ⊤ dX) jEvolution of U. To compute the evolution of U, we invoke the theorem D.2 on the evolution of WW ⊤ = UDU ⊤ . We ignore it here as it does not have much consequence on our results.Theorem B.6. In the large noise limit, when l = 2, the following properties hold, for t ≤ τ , (a) s 0 , s 1 are greater than zero almost surely.(b) for α = (p -3)/2, s -α 0 is a super-martingale while s -α 1 is a sub-martingale.Proof. First, note that in the large noise limit with l = 2, the evolution of the eigenvalues is expressed asd(s 0 ) = pc 2 0 dt + s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 0 c 2 0 Ä d Bt ä 0 , (B.7) d(s 1 ) = pc 2 1 dt - s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 1 c 2 1 Ä d Bt ä 1 . (B.8)Using the Ito chain rule, for the evolution of s -α 0 we can writed s -α 0 = ∂(s -α 0 ) ∂s 0 Å pc 2 0 dt + s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 0 c 2 0 Ä d Bt ä 0 ã + 1 2 ∂ 2 (s -α 0 ) ∂ 2 s 0 2 » s 0 c 2 0 2 dt = -αs -α-1 0 Å pc 2 0 dt + s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt -2(α + 1)c 2 0 dt + 2 » s 0 c 2 0 Ä d Bt ä 0 ã = -αs -α-1 0 Å c 2 0 dt + s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 0 c 2 0 Ä d Bt ä 0 ã , analogously d s -α 1 = -αs -α-1 1 Å c 2 1 dt - s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 1 c 2 1 Ä d Bt ä 1 ã ,and finally for s -α 0 s -α1 d s -α 0 s -α 1 = d s -α 0 s -α 1 + s -α 0 d s -α 1 + d s -α 0 d s -α 1 = -αs -α-1 0 s -α 1 Å c 2 0 dt + s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 0 c 2 0 Ä d Bt ä 0 ã -αs -α 0 s -α-1 1 Å c 2 1 dt - s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 dt + 2 » s 1 c 2 1 Ä d Bt ä 1 ã .Now, we can show that the drift term in the SDE that describes the dynamics of s -α 0 s -α 1 is zero, which gives us the first part of the result by Mckean's argument [Mayerhofer et al., 2011] ,-αs -α-1 0 s -α-1 1 Å s 1 c 2 0 + s 1 s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 + s 0 c 2 1 + s 0 s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 ã = -αs -α-1 0 s -α-1 1 Å s 1 c 2 0 + s 0 c 2 1 + s 0 s 1 c 2 1 + s 2 1 c 2 0 -s 2 0 c 2 1 + s 0 s 1 c 2 0 s 0 -s 1 ã = -αs -α-1 0 s -α-1 1 Ç s 1 c 2 0 + s 0 c 2 1 + (s 1 -s 0 ) s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 å = 0.The second part is obtained by noticing thatc 2 0 + s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 = s 0 c 2 1 + c 2 0 s 0 -s 1 ≥ 0, c 2 1 - s 0 c 2 1 + s 1 c 2 0 s 0 -s 1 = - s 1 c 2 1 + c 2 0 s 0 -s 1 ≤ 0,and hence the drift term of d s -α 0 is not positive, while the drift term of d s -α 1 is not negative.In all the graphs we plot the values averaged on 20 runs with different random seeds as well as the 95% confidence interval (lightly colored). To numerically emulate GF (Figure 1 ), we set a stepsize of 1e -6 in numerical simulation.In the further experiments, we study the behaviour of the linear network for regression with the same synthetic data and same network initialization as in previous experiment. As seen in the left plot of the Figure 2 , when the stepsize is large (η = 0.1), singular values exhibit behavior similar to the case of LNGF, while with the small stepsize (η = 0.005) the evolution of singular values is closer to GF case. Next, we examine the effect of SGD in the case of classification task with logistic loss, as illustrated in the middle plot of the Figure 2 . We consider synthetic data with n = 1000 samples of Gaussian data in R 5 (d = 5) constituting two clusters corresponding to two classes (k = 1). Note that larger stepsize (η = 0.5) in this case also forces the smallest singular value to tend to zero, however the effect is not so dramatic for the rest of singular values. Additionally, we study the 2-layer ReLu network optimized with SGD on the same regression task as before. As seen in the right plot of the Figure 2 , the decrease of the last singular value σ 4 is much slower than in the case of the linear network, however, the larger stepsize still facilitates divergence of k largest (σ 0 and σ 1 ) and p -k smallest (σ 2 , σ 3 and σ 4 ) singular values.All experiments are implemented with Python 3 [Van Rossum and Drake, 2009] under PSF license, NumPy [Harris et al., 2020] under BSD license, and PyTorch [Paszke et al., 2019] under BSD-3-Clause license.The experiments were run on a Intel i5-8250U, 8-GB RAM, with OS Ubuntu 20.04.6.D.1 Notations and preliminary definitions Definition D.1 (Eigen decomposition and Singular Value decomposition). We discuss the eigen value decomposition for a symmetric square matrix, and the singular value decompostion for any matrix is defined as the following (a) Eigen decomposition. For any rank r matrix R ∈ S p , R = VDV ⊤ is the eigen decomposition, where V ∈ R p×r , D ∈ R r×r , D is a diagonal matrix and V ⊤ V = I r , however, VV ⊤ is not necessarily an identity matrix unless r = p.(b) Singular Value Decomposition. For any rank r matrix W ∈ R p×l , W = UΣV ⊤ , whereU ∈ R p×r , V ∈ R l×r , Σ ∈ R r×r , Σ is a diagonal matrix and U ⊤ U = V ⊤ V = I r , howeverthe UU ⊤ and VV ⊤ are not necessarily identity unless r = p or r = l respectively.Theorem D.2. For a matrix-valued stochastic process on S ++ p+k ,dR = Ndt + dMwhere dM is a local martingale process. Let R = VDV ⊤ is the eigenvalue decomposition of the process, the evolution of eigenvalues satisfy the SDE for time t less than the collision time,dD = I ⊙ N dt + I ⊙ d M dt + I ⊙ Ä d M Ä S ⊙ d M ää + D -1 ⊙ V ⊤ dR I -VV ⊤ dRV .where S is defined as per Eq. D.1 and d M = V ⊤ dMV, N = V ⊤ NV. The evolution of the eigenvectors,dV = V Q ∥ dt + S ⊙ dF + (I -VV ⊤ ) Q ⊥ dt + dR VD -1 .where you define,Q ∥ = I ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M äó 2 - I ⊙ D -1 V ⊤ dR I -VV ⊤ dRVD -1 2 -S ⊙ îÄ S ⊙ d M ä î d M ⊙ I óó + S ⊙ Ä d M Ä S ⊙ d M ää + S ⊙ V ⊤ dR I -VV ⊤ dRVD -1 , Q ⊥ = dRVD -1 îî S ⊙ d M ó D -d M ó D -1 .Evolution of eigenvalues for general matrix SDE Proof. Using the eigen decomposition, we haveR = VDV ⊤ , D = V ⊤ RV, dD = V ⊤ dRV + V ⊤ RdV + dV ⊤ RV + V ⊤ dRdV + dV ⊤ dRV + dV ⊤ RdV, = V ⊤ dRV + DV ⊤ dV + dV ⊤ VD + V ⊤ dRdV + dV ⊤ dRV + dV ⊤ V D V ⊤ dV .The approach we follow is use the jacobian of the evolution of V (see [Townsend, 2016] ) and solve the constrains equations to obtain the Ito correction term as done in Bru [1989] . Let (s 1 , s 2 , . . . , s r ) denote the diagonal entries of D. Furthermore, we define the matrix S, which plays a notable role in Jacobian w.r.t V, as the following,S ij = ® 0 if i = j, (s j -s i ) -1 o.w. (D.1)For the sake of brevity, we denote the evolutiondF def = V ⊤ dRV = V ⊤ NV dt + V ⊤ dMV, def = N dt + d MThe evolution of the eigenvectors,dV = VdΩ V + (I -VV ⊤ )dΞ V .Using the Jacobian of the eigen vectors, we write,dΩ V = Q ∥ dt + S ⊙ dF, dΞ V = Q ⊥ dt + dR VD -1 .Note that V ⊤ V = I r , using this we have,0 = d V ⊤ V = dV ⊤ V + V ⊤ dV + dV ⊤ dV, = dΩ V ⊤ + dΩ V + dV ⊤ VV ⊤ dV + dV ⊤ I -VV ⊤ dV, = dΩ V ⊤ + dΩ V + dΩ V ⊤ dΩ V + dΞ V ⊤ I -VV ⊤ dΞ V , = dΩ V ⊤ + dΩ V -(S ⊙ dF) (S ⊙ dF) + D -1 V ⊤ dR I -VV ⊤ dRVD -1 . Using dΩ ⊤ V = Q ⊤ ∥ dt -S ⊙ dF, we have dΩ V ⊤ + dΩ V = Ä Q ⊤ ∥ + Q ∥ ä dt. Ä Q ∥ + Q ⊤ ∥ ä dt = Ä S ⊙ d M ä Ä S ⊙ d M ä -D -1 V ⊤ dR I -VV ⊤ dRVD -1 . (D.2)Coming back to the evolution of singular values,dD = V ⊤ dRV + DV ⊤ dV + dV ⊤ VD + V ⊤ dRdV + dV ⊤ dRV + dV ⊤ V D V ⊤ dV . = dF + Ä DQ ∥ + Q ⊤ ∥ D ä dt + D (S ⊙ dF) -(S ⊙ dF) D + dΩ ⊤ V DdΩ V + V ⊤ dR VdΩ V + I -VV ⊤ dΞ V + î dΩ V ⊤ V ⊤ + dΞ V ⊤ I -VV ⊤ ó dRV, dD = I ⊙ dF + Ä DQ ∥ + Q ⊤ ∥ D ä dt - Ä S ⊙ d M ä D Ä S ⊙ d M ä + d M Ä S ⊙ d M ä - Ä S ⊙ d M ä d M + V ⊤ dR I -VV ⊤ dRVD -1 + D -1 V ⊤ dR I -VV ⊤ dR. (D.3)Note that dD is diagonal, hence, I ⊙ dD = dD.I ⊙ dD = I ⊙ dF + I ⊙ Ä DQ ∥ + Q ⊤ ∥ D ä dt -I ⊙ îÄ S ⊙ d M ä D Ä S ⊙ d M äó + 2I ⊙ Ä d M Ä S ⊙ d M ää + 2I ⊙ D -1 V ⊤ dR I -VV ⊤ dRNote that I ⊙ (DM ) = I ⊙ (M D) = D ⊙ M for any matrix M and diagonal matrix D, using this property, we can simplify the above expression as,dD = I ⊙ dF + D ⊙ Ä Q ∥ + Q ⊤ ∥ ä dt -I ⊙ îÄ S ⊙ d M ä D Ä S ⊙ d M äó + 2I ⊙ Ä d M Ä S ⊙ d M ää + 2D -1 ⊙ V ⊤ dR I -VV ⊤ dRUsing Eq. D.2, we have,D ⊙ Ä Q ∥ + Q ⊤ ∥ ä dt = D ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M ä -D -1 V ⊤ dR I -VV ⊤ dRVD -1 ó , = I ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M ä D ó -D -1 ⊙ V ⊤ dR I -VV ⊤ dRV .Using this,dD = I ⊙ dF + I ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M ä D ó -I ⊙ îÄ S ⊙ d M ä D Ä S ⊙ d M äó + 2I ⊙ Ä d M Ä S ⊙ d M ää + D -1 ⊙ V ⊤ dR I -VV ⊤ dRV , = I ⊙ dF + I ⊙ îÄ S ⊙ d M ä îÄ S ⊙ d M ä D -D Ä S ⊙ d M äóó + 2I ⊙ Ä d M Ä S ⊙ d M ää + D -1 ⊙ V ⊤ dR I -VV ⊤ dRV , = I ⊙ dF + I ⊙ îÄ S ⊙ d M ä d M ó + 2I ⊙ Ä d M Ä S ⊙ d M ää + D -1 ⊙ V ⊤ dR I -VV ⊤ dRV , = I ⊙ dF + I ⊙ Ä d M Ä S ⊙ d M ää + D -1 ⊙ V ⊤ dR I -VV ⊤ dRV .Evolution of eigenvectors for general matrix SDE. Here, we derive the evolution of eigenvectors, Using Eq. D.2, we have,Ä Q ∥ D + Q ⊤ ∥ D ä dt = Ä S ⊙ d M ä Ä S ⊙ d M ä D -D -1 V ⊤ dR I -VV ⊤ dRVNow further using the constrain that dD needs to be diagonal we get,Ä DQ ∥ + Q ⊤ ∥ D ä dt = dD -I ⊙ dF + Ä S ⊙ d M ä D Ä S ⊙ d M ä -d M Ä S ⊙ d M ä + Ä S ⊙ d M ä d M -V ⊤ dR I -VV ⊤ dRVD -1 -D -1 V ⊤ dR I -VV ⊤ dR. DQ ∥ -Q ∥ D dt = dD -I ⊙ dF - Ä S ⊙ d M ä îÄ S ⊙ d M ä D -D Ä S ⊙ d M äó -d M Ä S ⊙ d M ä + Ä S ⊙ d M ä d M -V ⊤ dR I -VV ⊤ dRVD -1 , = dD -I ⊙ dF - Ä S ⊙ d M ä î d M ⊙ Īó -d M Ä S ⊙ d M ä + Ä S ⊙ d M ä d M -V ⊤ dR I -VV ⊤ dRVD -1 , = dD -I ⊙ dF + Ä S ⊙ d M ä î d M ⊙ I ó -d M Ä S ⊙ d M ä -V ⊤ dR I -VV ⊤ dRVD -1 . Ī ⊙ DQ ∥ -Q ∥ D dt = Ī ⊙ (dD -I ⊙ dF) + Ī ⊙ îÄ S ⊙ d M ä î d M ⊙ I óó -Ī ⊙ Ä d M Ä S ⊙ d M ää -Ī ⊙ V ⊤ dR I -VV ⊤ dRVD -1 . Ī ⊙ Q ∥ dt = S ⊙ î - Ä S ⊙ d M ä î d M ⊙ I ó + d M Ä S ⊙ d M ä + V ⊤ dR I -VV ⊤ dRVD -1 ó .Combing these, we get the diagonal and off diagonal terms ofQ ∥ I ⊙ Q ∥ dt = 1 2 I ⊙ Ä Q ∥ + Q ⊤ ∥ ä dt, = I ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M äó 2 - I ⊙ D -1 V ⊤ dR I -VV ⊤ dRVD -1 2 . Q ∥ = I ⊙ îÄ S ⊙ d M ä Ä S ⊙ d M äó 2 - I ⊙ D -1 V ⊤ dR I -VV ⊤ dRVD -1 2 -S ⊙ îÄ S ⊙ d M ä î d M ⊙ I óó + S ⊙ Ä d M Ä S ⊙ d M ää + S ⊙ V ⊤ dR I -VV ⊤ dRVD -1 .Computing of Q ⊥ . Recalling the evolution of the eigenvectors,dV = VdΩ V + (I -VV ⊤ )dΞ V .Using the Jacobian of the eigen vectors, we write,dΩ V = Q ∥ dt + S ⊙ dF, dΞ V = Q ⊥ dt + dR VD -1 , dV = V Q ∥ dt + S ⊙ dF + (I -VV ⊤ ) Q ⊥ dt + dR VD -1 , dV ⊤ = î Q ⊤ ∥ dt -S ⊙ dF ó V ⊤ + Q ⊤ ⊥ dt + D -1 V ⊤ dR (I -VV ⊤ ).Using the fact that I -VV ⊤ R = 0 and deriving it,0 = I -VV ⊤ R, 0 = d I -VV ⊤ R , dR = d VV ⊤ R , = dVV ⊤ R + VdV ⊤ R + VV ⊤ dR + dVdV ⊤ R + dVV ⊤ dR + VdV ⊤ dR, dRV = dVD + VdV ⊤ VD + VV ⊤ dRV + dVdV ⊤ VD + dVV ⊤ dRV + VdV ⊤ dRV, dVD = V Q ∥ D dt + (S ⊙ dF) D + (I -VV ⊤ ) [Q ⊥ D dt + dR V] , VdV ⊤ VD = V î Q ⊤ ∥ dt -S ⊙ dF ó D, dVdV ⊤ VD = -V [S ⊙ dF] [S ⊙ dF] D -I -VV ⊤ dRVD -1 [S ⊙ dF] D, dVV ⊤ dRV = V [S ⊙ dF] dF + I -VV ⊤ dRVD -1 dF, VdV ⊤ dRV = -V [S ⊙ dF] dF + VD -1 V ⊤ dR(I -VV ⊤ )dRV.Adding the terms up we get,V î Q ∥ + Q ⊤ ∥ ó Ddt + I -VV ⊤ Q ⊥ Ddt -V [S ⊙ dF] [S ⊙ dF] D -I -VV ⊤ dRVD -1 [S ⊙ dF] D + I -VV ⊤ dRVD -1 dF + VD -1 V ⊤ dR(I -VV ⊤ )dRV = 0. I -VV ⊤ Q ⊥ Ddt -I -VV ⊤ dRVD -1 [S ⊙ dF] D + I -VV ⊤ dRVD -1 dF = 0. I -VV ⊤ Q ⊥ Ddt = I -VV ⊤ dRVD -1 [S ⊙ dF] D -I -VV ⊤ dRVD -1 dF, I -VV ⊤ Q ⊥ = I -VV ⊤ dRVD -1 [[S ⊙ dF] D -dF] D -1 Q ⊥ = dRVD -1 [[S ⊙ dF] D -dF] D -1This gives the expression for Q ⊥ and this ends our computation. Proof.S ⊙ dF = [diag (c) Sdiag (dm t ) + diag (dm t ) Sdiag (c)] , dF(S ⊙ dF) = cdm t ⊤ + dm t c ⊤ [diag (c) Sdiag (dm t ) + diag (dm t ) Sdiag (c)] , = cs ⊤ Sdiag (c) dt -Ddiag (Sdiag (c) c) dt + Ddiag (c) Sdiag (c) dt. Lemma D.5. With S defined in Equation (D.1), dF = dF = ΣV ⊤ dB t c ⊤ + cdB t ⊤ VΣ and dm t def = (σ ⊙ d Bt ). (S ⊙ dF)(S ⊙ dF) = Ddiag Ä Sdiag (c) 2 S ä dt + diag (c) SDSdiag (c) dt. (D.7) Proof. (S ⊙ dF) = S ⊙ dm t c ⊤ + cdm t ⊤ , = diag (c) Sdiag (dm t ) + diag (dm t ) Sdiag (c) .Now, computing the product,(S ⊙ dF)(S ⊙ dF) = [diag (c) Sdiag (dm t ) + diag (dm t ) Sdiag (c)] [diag (c) Sdiag (dm t ) + diag (dm t ) Sdiag (c)] , = Ddiag Ä Sdiag (c) 2 S ä dt + diag (c) SDSdiag (c) dt.Lemma D.6.(S ⊙ dF)dm t c ⊤ dF = Proof. (S ⊙ dF)dm t = [diag (c) Sdiag (dm t ) + diag (dm t ) Sdiag (c)] dm t = diag (c) S(σ ⊙ σ)NeurIPS Paper Checklist Justification: The limitations of the work are discussed in section 4 in a designated paragraph as well as in section 5 in the discussion of theorem 5.1. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Mechanism behind the low-rank phenomenonIn this section, we investigate the evolution of singular values under stochastic training to gain deeper insights into the low-rank phenomenon. To simplify the discussion, throughout the section 


Decision: The paper investigates the differences between Stochastic Gradient Descent (SGD) and Gradient Descent (GD) in the context of a two-layer linear network with square loss. It reveals a dichotomy: GD maintains the rank of the parameter matrix from initialization, whereas SGD, especially in the presence of label noise, reduces the rank over time. This phenomenon is analyzed by examining the evolution of the determinant of the parameter matrix, and the stochastic differential equation (SDE) governing the eigenvalues is derived. The SDE indicates a repulsive force between eigenvalues, leading to rank deficiency. The findings are supported by experiments extending beyond linear networks and regression tasks.

The referees highlighted that the analysis was comprehensive: the derivation of the SDE governing eigenvalues adds depth to the understanding of rank deficiency, providing a strong mathematical foundation for the observed phenomena. I recommend to accept it to NeurIPS 2025.

Nevertheless, I believe the authors should take into account the issues raised by the referees for the camera-ready version, but it seems the authors acknowledged them during the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper proposes an analysis of the gradient flows of two layer linear neural networks with a squared loss borrowing tools from differential equations. The paper's result establishes that the stochastic gradient method generates solutions (limit of the flow as time goes to infinity) with determinant of the parameters x parameters transpose tending to zero. This implies that the parameters of a model optimized with stochastic gradient converge to a low-rank (simple) solution. While the full gradient method's solutions for the same problem do not exhibit the same simple structure. Authors conclude that this finding gives an explanation to the implicit bias of SGD.
strength_weakness: strength_weakness: The paper studies a very simple model. Numerical experiments are also on the same and in very small data. It would benefit the paper to run experiments on larger / more complex models to discuss the limits and validity of this theory. The generalization to other settings section 6 is only changing the loss function. The section's title is perhaps overpromising.
questions: questions: Is it possible to verify the breadth of validity of the result (parameters of SGD based solution live in a low-rank manifold, not the FGD) on different neural network architectures and see whether similar conclusions hold? Or which other property (if low rank is too restrictive) would you test for?

Does this theory imply that for the simpler vector problem the solution of SGD is a sparse vector where FGD finds dense solutions?


If we add a ReLU non-linearity between W1 and W2, then in what form the simplicity of the solution by SGD will get modified? locally low-rank?
limitations: limitations: The papers does not discuss how broadly their findings are valid. Numerical experiments could be used for testing whether findings are still valid in slightly more complex models
paper_summary: paper_summary: This paper studies the implicit bias of SGD for two-layer linear networks. The authors study this primarily using (stochastic) gradient flow (S)GF, the continuous time version of (S)GD. To model the stochasticity, they approximate the SGD noise with independent label noise. Under these assumptions, they prove that the determinant of the Gram matrix of the parameters is invariant under GF, whereas it exponentially decays under SGF. Hence, if such a network is initialized with a full-rank parameter matrix, the rank asymptotically drops by at least one under label noise SGF, which lends some evidence to a low-rank bias for SGD not present in GD. The stochastic eigenvalue dynamics are investigated further, and while the exact dynamics are not rigorously pinned down, the paper calls out the repulsive forces that could explain why SGD converges to lower rank solutions. These results are empirically verified using some synthetic data experiments for regression and classification.
strength_weakness: strength_weakness: It was nice to see some experiments validating the theory, but I felt that the dimensionality of the problems should be increased to be more convincing. Right now the experiments are done with $(p, l, k) = (5, 10, 2)$. Even increasing these to being in the hundreds would be interesting, and correspondingly it might be appropriate to use other metrics to measure the rank of $W_1$, such as the effective rank  $\tr(M)/||M||$.
questions: questions: 1. In line 95, should $d$ and $p$ be swapped? Since the input is $p$-dimensional, whereas the parameter $\theta$ is $d$-dimensional (for example, the two-layer setting $d = p \cdot k$).
2. In line 96, there should be an $\ell_2$ norm outside of $y_i - f_\theta(x_i)$. 
3. In line 108, $k=1$ should be in math mode. 
4. There is a minor typo on the equation after line 208: it should read $Wa$ rather than $W^\top a$.
5. In the statement of Theorem 5.1 on Line 228, there is a missing comma in the definition of the ordered eigenvalues. 
6. Could the authors add a reference to the proofs in the appendix when the result is stated in the main text (e.g. Lemma 6.1)?
7. In Section 7, I think I am missing something obvious: why are the $d + \ell - k$ smallest singular values equal $\sigma_2, \sigma_3, \sigma_4$, since to my understanding $d + \ell - k = 10 + 10 - 2 = 18$?
8. What can one hope to say about deeper linear networks? If I understand correctly, if one naively generalizes the construction of $\Theta$ then the special block structure of the label noise portion will no longer hold (in particular, it seems that $Theta$ would need to contain all the prefix and suffix products of the linear layers)
limitations: limitations: Yes
paper_summary: paper_summary: This paper studies and analyzes how the rank of the parameter matrices evolves for two-layer linear networks when using GD and label noise SGD. The paper basically shows that while GD preserves the rank at initialization throughout its trajectory, SGD reduces the rank, thereby removing spurious directions. A stochastic differential equation (SDE) is derived to characterize the evolution of the eigenvalues of the parameter matrix to formalize the rank-reducing behavior of SGD. Small-scale experiments validate the derived theory.
strength_weakness: strength_weakness: **1.** While the rank-reducing property of SGD is interesting, it does not imply that the prediction error (which is the generalization metric most people would care about) of SGD is lower than that of GD. What if SGD converges to a completely erroneous low-rank solution? I understand that the paper is not directly about the generalization abilities of SGD and GD, but I feel this point should be discussed somewhere. 

**2.** Since the results here are only for linear networks (which is completely fine), is there any intuition for why the results and claims of this paper should extend to non-linear networks?
questions: questions: Please see the Weaknesses. Also, I have the following questions:

**1.** In line 174, is $M = \Theta^\top \Theta$?

**2.** In line 207 and the equation thereafter, if $W = U \Sigma V^\top$, then $W^\top a$ should be $V \Sigma U^\top a$.

**3.** I don't understand the jump from eq. (5.2) to (5.3). There seems to be an issue with the $\sqrt{\eta \delta}$ term (with the definition of $d X$).
limitations: limitations: Discussed somewhat throughout the paper.
paper_summary: paper_summary: This paper presents an interesting dichotomy between SGD vs. GD, and how stochasticity in the gradients have the ability to implicitly regularize towards low-rank solutions. I believe this point is best highlighted by Theorem 5.1, where the show that the highlighted term in Equation (5.6) has a repulsive force, increasing the gap between larger and smaller eigenvalues. The "punchline" of the paper is technically given in Theorems 4.1 and 4.2, where they show that the solutions given by GD cannot have its rank diminished (rather stays at initialization), whereas we can see a decay in SGD (or SGF rather).
strength_weakness: strength_weakness: - I think the presentation of the paper can be slightly improved. For example, the discussion on initialization on Page 4 was a bit confusing. The authors discuss different initialization tactics and the balancing effect, but they do not clarify which initialization they ultimately chose. I believe the intention was to show how initialization plays a role in Theorems 4.1 and 4.2, but this could be made clearer if that was indeed the intention. Furthermore, the authors could also change the notation to $\Delta(t)$ between lines 149-150. The authors are also missing a > in line 228.
- The experiments section is a bit lacking. Are there ways we can leverage the results from this paper to more practical scenarios / networks?
- My main concern (which is also the weakness) is the similarity between this paper and [1], which I don't believe the authors cited. From what I read, the main takeaway in [1] is that the noise in SGD has the ability to "push" or "attract" networks towards simpler networks, which they describe to be invariant sets. If you look at Sections 5 and 6 in [1], I believe what they are trying to say is that the noise has the ability to collapse the network to an invariant set that is low-rank, and one in which some singular modes are actually never learned. At a high-level, it seems to me that this might be a stronger result than to say that there exists a repulsive force between the eigenvalues, which is the current result. If I am missing something, please feel free to correct me. Though, at the very least, I think the authors should cite this paper and delineate the similarities / differences.

[1] Feng Chen, Daniel Kunin, Atsushi Yamamura, Surya Ganguli. "Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworks". NeurIPS 2023.
questions: questions: See Weaknesses. I would be more than happy to raise my score if my concerns can be addressed.
limitations: limitations: I have stated the limitations throughout my review.

Comments: title: Reviewer Response
comment: Thanks for the response!
Thank you for the clarifications, particularly with the missing citation. I now see clearly the difference in contribution. I think the proof techniques used in the paper, especially in tracking the determinant, is useful for future work and will raise my score.
Thank you for the additional experiments and clarifications; I am definitely convinced by the plots you included in the supplemental figures. 
Also, I appreciate the thoughtful response to my question about deeper linear layers, and would love to see any future work in this direction.
Cheers!