Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Authors: Tom Jacobs
Chao Zhou
Rebekka Burkholz
Venue: ICML 2025 poster (2025)
Keywords: Implicit bias
explicit regularization
weight decay
matrix sensing
LoRA
attention
mirror flow
time-dependent Legendre function
Primary area: theory->deep_learning
TL;DR: This work studies how explicit regularization influences implicit bias in overparameterized models, with applications spanning various problem domains, including sparse coding, matrix sensing, single-layer attention and LoRA.

Abstract: Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understanding their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incorporate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encompasses a broad class of problems, including sparse coding, matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regularization and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Tom Jacobs
Chao Zhou
Rebekka Burkholz'
Abstract
ing the learning algorithm from overfitting (Goodfellow et al., 2016), as follows:
Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is of- ten employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understand- ing their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incor- porate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encom- passes a broad class of problems, including sparse coding. matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regulariza- tion and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.
min f(z) + aa(z),
(2)
ZER
where controls the trade-off between the objective and the penalty. This approach regulates the model capacity (Dai et al., 2021) and encourages simpler solutions that are more likely to generalize well to unseen data (Tian & Zhang, 2022). Common explicit regularization methods include L1 (LASSO) and Lg (weight decay) (Bishop & Nasrabadi, 2006). The effectiveness of explicit regularization tech- niques has been demonstrated across various machine learn- ing paradigms (Arpit et al., 2016), including supervised learning, unsupervised learning and reinforcement learning,
Implicit bias (Gunasekar et al., 2017; Woodworth et al., 2020; Li et al., 2022; Sheen et al., 2024; Vasudeva et al., 2024; Tarzanagh et al., 2023; Jacobs & Burkholz, 2025) can be considered an inherent aspect of the model design and optimizer that does not require explicit modifications of the objective function. The goal of characterizing the implicit bias is to understand how overparameterization impacts the training dynamics and, consequently, model selection. For example, in the presence of many global minima, op- timization algorithms like gradient descent inherently con- verge towards low-norm solutions (Woodworth et al., 2020; Pesme al., 2021), which impacts model properties such as generalization (Belkin et al., 2019) and memorization (Radhakrishnan et al., 2020).
1. Introduction
Regularization is fundamental technique in machine learn- ing that helps control model complexity, prevent overfitting and improve generalization (Kukačka al., 2017). We focus on the interplay between two major categories of reg- ularization: explicit regularization and implicit bias. We introduce both concepts within a general minimization prob- lem. Consider the objective function f: R" + R to be minimized with respect to x:
Implicit bias is often associated with a mirror flow (Karimi et al., 2024; Li et al., 2022), which results from a reparam- eterization of f by setting x g(w), where w E . and M is a smooth manifold. It is important to highlight a fun- damental distinction between the explicit regularization in the original space and the mirror flow with the objective function, formulated as follows:
min fx)
(1)
#ER*
In the context of explicit regularization, a penalty term h(z) is incorporated into the objective function, directly prevent-
min f(g(w)) + ah(w)) WEM
(3)
The explicit regularizer h now acts on the parameters w instead of g(w). Our main goal is to understand how the explicit regularization l((w) affects the implicit bias, thereby shaping the effective regularization in the original parameter space x. To achieve this, we analyze their inter- play by integrating explicit regularization into the mirror flow framework.
'CISPA Helmholtz Center for Information Security, Saar- hawajacooossssp.........
brücken, Germany. Correspondence to: Tom Jacobs
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Typically, the nature and strength of implicit bias remain constant throughout training they are inherently deter- mined by the model parameterization. For instance, it has been shown that specific forms of overparametrization lead to low-rank or sparse solutions (Arora et al., 2019; Pesme et al., 2021; Sheen et al., 2024; Woodworth et al., 2020; Gunasekar et al., 2017), revealing a bias towards spansity in particular settings. Nevertheless, factors such as small initialization, large learning rates and noise needed to obtain this sparsity bias, without guarantees. However, the inherent bias can degrade performance if it does not fit to the learning task. Our key insight to overcome this issue is that implicit bias can be adapted and controlled by explicit and potentially dynamic regularization, which induces a time-dependent mirror flow. To analyze the resulting opti- mization problem within the extended mirror flow frame- work (Li et al., 2022) and obtain convergence and optimality results, we provide sufficient conditions for the reparame- lerization and explicit regularization h. Additionally, we characterize the regularization in terms of to understand their interplay and impact on the Legendre function, which associated with the implicit bias.
Type of bias
La
L
Positional bias
Range shrinking
Figure I. Illustration of three established effects of explicit regu- larization (-) on implicit bias.
This offers a deeper theoretical insight into the interplay between implicit bias and explicit regularization.
Our theoretical framework has multiple application-relevant implications. As examples, we explore sparse coding, ma- trix sensing, attention mechanisms in transformers, and Low-Rank Adaptation (LoRA) through experiments. By switching off weight decay during training, we demonstrate the positive impact of dynamic regularization on generaliza- tion performance and analyze its effect on implicit bias.
Concretely, we identify three distinct effects:
Type of bias: the explicit regularization changes the shape of the Legendre function and thus the implicit bias. For example, the shape changes from an L2 norm to L1 norm.
Contributions:
We establish sufficient conditions for incorporating dif- ferent types of explicit regularization into the mirror flow framework and characterize their effects, focusing on three key impacts on implicit bias: positional bias shift, type of bias, and range shrinking.
Positional bias: in the standard case without explicit regularization, the global minimum of the Legendre function corresponds to the parameter initialization (Li et al., 2022). Explicit regularization shifts this global minimum, gradually moving it closer zero during training.
We propose systematic procedure for identifying appro- priate regularizations and establish general convergence and optimality results. These results provide guidance on how to manage the above effects by adjusting the explicit regularization.
Range shrinking: the explicit regularization shrinks the range of the attainable values for the Legendre function. For example,the the L1 norm of the parameters becomes stationary during training.
We gain the insight that explicit regularization controls the strength of implicit sparsification and has a lasting effect by changing the geometry of the training dynamics.
The three effects are illustrated in Figure. 1. They all have lasting impact on implicit bias, as they change the geometry of the training dynamics.
We highlight the positive impact of dynamic regulariza- tion and the resulting implicit bias through experiments such as sparse coding, matrix sensing, attention mecha- nism, and LoRA fine-tuning large language models.
Weight decay provides an illustrative example of an explicit regularization that has a desirable impact on the training dynamics (D' Angelo al., 2023). While (Dai et al., 2021) studied the effect of constant regularization pacity, and (Khodak et al., 2021; Kobayashi et al., 2024) empirically observed that weight decay leads to sparsity bias for quadratic reparameterizations, our focus lies on under- standing the effects of both constant and dynamic explicit regularization on the training dynamics and implicit bias.
model ca-
2. Related Work
Regularization There are multiple ways to regularize in machine learning. Some of the most widely used tech- niques include weight decay (D' Angelo al., 2023; Krogh
2

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
& Hertz, 1991), data augmentation (Cubuk et al., 2020; Orvieto al., 2023), dropout (Srivastava al., 2014), and batch normalization (Ioffe & Szegedy, 2015). Weight decay, or L2 regularization, discourages large weights to mitigate overfitting and induces a desirable change in training dy- namics. This change can be effectively captured using the time-dependent mirror flow framework that we extend. As another example, dynamic weight decay has been proposed for ADAM to keep the gradient norms in check (Xie et al., 2023). In comparison, we analyze the effect of more general dynamic regularization on the implicit bias of gradient flow.
2019). Accordingly, the flow tends be implicitly biased towards solutions with low rank or nuclear norm. We show that dynamic explicit La regularization can further enhance this effect in the context of quadratic overparameterization. This is illustrated through experiments on transformer net- works. Moreover, we identify the inherent bias of LoRA (Hu et al., 2022; Wan et al., 2024) and delve into the associated challenges. This is especially of interest, as LoRA has gained significant popularity in the field of large language models (LLMs), as it allows for cost-effective finetuning. Another application is sparse coding (SC) which is similar to diagnonal linear networks with explicit regularization. This representation technique is widely employed in signal processing and pattern recognition (Zhang et al., 2015). The core principle of SC is to find a sparse representation by im- posing constraints, typically using the Lo-norm. However, this formulation leads to an NP-hard problem (Tropp, 2004). An alternative strategy relaxes the constraint, transforming the original problem into a convex, albeit non-smooth opti- mization task. Proximal algorithms have proven effective to solve these non non-smooth problems (Daubechies et al., 2004). Similarly, reparameterization with explicit regularization can be used to solve this.
Implicit Bias The implicit bias is a well-studied phe- nomenon (Woodworth al., 2020; Gunasekar et al., 2017; 2018; Li et al., 2022), which has primarily been character- ized within the mirror flow framework, a well-established concept in convex optimization (Alvarez et al., 2004; Beck & Teboulle, 2003; Rockafellar& & Fenchel, 1970; Boyd & Vandenberghe, 2009), which we extend by introducing ex- plicit regularization that can induce a time-dependent Leg- endre function. Moreover, for convergence guarantees the time-dependent Legendre function needs to satisfy addi- tional assumptions, i.e., it needs to be time-dependent Bregman function. For this class of functions we show con- vergence with decaying regularization. A mirror flow can be interpreted as a gradient flow on a Riemannian manifold (Li et al., 2022; Alvarez et al., 2004), which has also been studied in stochastic gradient descent (SGD) (Pesme al., 2021; Even et al., 2023; Lyu & Zhu, 2023) context. The main observation is that large learning rates and stochastic noise from SGD have a generalization benefit by inducing sparsity, although uncontrollable. We derive a similar but controllable benefit of explicit regularization. Still, it is pos- sible to also combine stochastic noise and a large learning rate with our framework. Discrete versions of mirror flow (Sun et al., 2022) have led to novel algorithmic designs (Raj & Bach, 2021: Gonzalez et al., 2024; Azizan et al., 2022). Time- Time-dependent mirror descent, in comparison, is under- explored, except for an analysis of its intrinsic properties and an application to continuous sparsification (Radhakrish- nan et al., 2021; Jacobs & Burkholz, 2025). Our framework covers multiple application relevant architectures and more general cases.
3. Theory: Integrating Explicit Regularization in the Extended Mirror Flow Framework
We begin by reviewing the theoretical background on repa- rameterizations and when they induce mirror flows. Our main result, Theorem 3.6, integrates explicit regularization into the mirror flow framework. Building on this, we ex- plore key implications, including a geometric interpretation of the interaction between implicit bias and explicit regular- ization. Using this interpretation and assuming the Legendre function R is Bregman function, we extend convergence results to the time-dependent setting by introducing the con- tracting property (Definition 3.8, Theorem 3.10). We also prove optimality in underdetermined linear regression (The- orem 3. 11). To apply our theory in practice, we show how to choose an explicit regularizer h for a given reparameter- ization g, often determined by neural network design. We characterize h for known reparameterizations (Woodworth et al., 2020; Pesme et al., 2021; Gunasekar et al., 2017) and examine their practical effects- -type of bias, positional bias, and range shrinking in Section 4.
Applications of the Mirror Flow Framework The mir- ror flow framework has been applied to various architectures, including attention mechanisms in transformers (Vaswani, 2017: Vasudeva et al., 2024; Sheen et al., 2024; Tarzanagh et al., 2023; Julistiono et al., 2024; Pesme et al., 2024), ma- trix factorization (Li et al., 2021; Gunasekar et al., 2017; 2018) and diagonal linear networks (Li et al., 2022; Pesme et al., 2021; Woodworth et al., 2020). The implicit bias of deep matrix factorization has also been analyzed with gradient flow methods (Marion& & Chizat, 2024; Arora et al.,
3.1. Preliminaries
To analyze the impact of regularization the training dy- namies of deep neural networks, we start from the gradient flow for our general optimization problem in Eq. (1). We assume f E C (IR", R) to be a continuously differentiable objective function. The corresponding gradient flow is:
dw - V (gw((((
Mp = Winit;
3

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
where V is the gradient with respect to w and g E C (M, R"). For specific choice of g. reparameterizing the loss function leads to mirror flow with related implicit bias. For this we recall two definitions that characterize the parameterization. Moreover, we define the Legendre function needed to define the mirror flow.
Wu & Rebeschini, 2021). Depending on the initialization of the reparameterization, the hyperbolic entropy is equiva- lent to either L2 or L1 implicit regularization. A Legendre function R that resembles an L1 regularization is associated with the so-called feature learning regime, which has been argued to improve generalization performance. Accordingly, it presents a positive impact of overparameterization on deep learning.
Definition 3.1. (Regular Parameterization, Definition 3.4 (Li et al., 2022)) Let M be a smooth submanifold of RDD. regular parameterization g : M -> RR" C'l parame- terization such that the Jacobian (G(w) is of rank for all E M.
Notably, in the presence of explicit regularization, the Leg- endre function R can change over time, which has been recognized by Jacobs & Burkholz (2025) with the specific goal to exploit the implicit bias for gradual sparsification. (Lyu et al., 2024) has analyzed how small constant weight decay impacts implicit bias to study its effect on Grokking, but has done so outside of the mirror flow framework. We allow for dynamic and possibly large regularization of rel- atively general form. While the implicit bias can change dynamically also for different reasons like large learning rate and stochastic noise as in (Pesme et al., 2021; Lyu & Zhu, 2023), we focus on dynamic explicit regularization to control this change.
This ensures that the gradient flow for 24 g(we) does not have an additional null space i.e. the gradient flow can not get stuck due to reparameterization. For the second definition, we first need to define the Lie bracket operator where 0 is the Jacobian operator.
Definition 3.2. (Lie Bracket, Definition 3.4 eLi al., 2022)) Let MM be a smooth submanifold of IR. Given two C Y.Y vector fields X, Y on M, we define the Lie Bracket of X
and Y as XX, (]() == OY (1x (w) ax (w))
Definition 3.3. (Commuting Parameterization, Definition 4.1 (Li et al., 2022)) Let M be smooth submanifold of RP. A C² parameterization g M - Rd is commuting in subset S C M iff for any j.j E [n]. the Lie bracket [Vg,, Vgyi (w) = 0 for all w E S. Moreover, we call g commuting parameterization it is commuting in M.
3.2. Main Result
We characterize the interplay between explicit regulariza- tion and implicit bias by a time-dependent Legendre func- tion. In the setting of Eq. (3) with reparameterizatioo g (M,,R") and explicit regularization h C³ (M, RR, allow the regularization strength a vary over time during the gradient flow, as indicated by an index az. This induces the following gradient flow:
Definition 3.3 ensures appropriate eigen basis alignment. We now introduce the Legendre function which governs the mirror flow dynamics.
Definition 3.4. (Legendre Function, Definition 3.8 ((Li al., 2022))) Let R : Fd R U {oc) be a differentiable convex function. We say R is a Legendre function when the following holds:
dw,- = V + a vwh(w:)) dt, wo Winst.
To rigorously define the corresponding time-dependent mir- ror flow, we define parameterized Legendre function based on Definition 3.4.
Definition 3.5. Let A be subset of TR. A parameterized Legendre function is R, : I + R" such that for all a E A. R, is a Legendre function (Definition 3.4).
R is strictly convex on int(dom/t).
For any sequence {- going to the boundary of dom/i, lim [V( xc.
The next theorem is our main result and builds on Defini- tion 3.5 and Theorem A.7 in the appendix.
Appendix A summarizes the main aspects of the mirror flow framework (Li et al., 2022), which explains their rela- tionship. Formally, let the reparameterization g regular (Definition 3.1), commuting (Definition 3.3) and satisfy As- sumption A.6. Then, by Theorem A.7, there is an Legendre function R : IR" + R (Definition 3.4) that follows the dynamics:
Theorem 3.6. Let hg. M + R be regular and commuting reparameterization satisfying Assumption A.6. Then there exists a time dine-dependent Legendre function Ra such that
dV Ra, (2+) = - V (xx)dtt
(5)
where at - agdss. Moreover, Ra, only depends on the initialization Wiit and the reparameterizatiin and regularization h. and is independent of the loss function f.
dV dv.R(I)) = - -af((x))
x = g (..nt)
(4)
The Legendre function is associated with the implicit bias of the optimization. For example, R can be the hyperbolic en- tropy studied in (Pesme et al., 2021; Woodworth et al., 2020;
Proof. See Theorem I. in the appendix. The main steps of the proof are: 1) We apply Theorem 4.9 (Li et al., 2022)
4

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
to the time-dependent loss function L, (x) (x) + oty with the reparameterization x g(w) and explicit regularization y = h(w) to get the resulting mirror flow with Legendre function R(z.v) v). 2) R is strietly convex. We utilize this to show that y + By R(x,y) y) is invertible. 3) We use the fact that the mirror flow for Yt is defined by 0g R(a,, Wr) = ar, where at- = - So a,,ds. Next, we plug in the inverse Yt = Q( 0z) into VzR(z Me) to get an expression for the gradient of the time-dependent Legendre function R. This leads to an equation for the time- dependent mirror flow A R(x,, ((x,; = Mr,, where N4 - -0 (x )ds. 4) In the final step, we show that V. (z. ((z, a)), where V is the derivative with respect to the first entry, is the gradient of a Legendre function for a fixed.
et al., 2004)) function R is called Bregman function if it satisfies the following properties:
domA is closed. R is strictly convex and continuous on domR. R is C' on int(domR)).
For any I E domft and 7 E R.
{u E dom/i)Dx(z, v) S 5 is bounded.
For any x E domR and sequence { C int(domR) such that l Ty = x, it holds that lim;- Dn(x, - O.
Using Definition 3.7, we can define the parameterized Breg- man function next:
Definition 3.8. Let A be subset of R. A parameterized Bregman function is R., R" R" such that for all a E A, R. is Bregman function (Definition 3.7). Furthermore, R. is called contracting if / - 0 for a E A.
Theorem 3.6 characterizes the training dynamics of reparam- eterization with regularization. This leads to an additional geometric interpretation, which we use next.
Remark 3.9. If a; 0 for t 2 T (for T > 0), we recover gradient flow with Riemannian metric (22Rer)
3.3. Geometric Interpretation
mirror flow can be interpreted as a gradient flow on a Riemannian manifold (Li et al., 2022: Alvarez et al., 2004).
This implication is useful for proving our next result, which highlights under which conditions we can obtain conver- gence if we switch off regularization at some point during training. We will also verify later in our experiments that this is a promising dynamic regularization strategy. Our next Theorem 3.10 gives the convergence are looking for by using the newly defined contracting property above.
If a Legendre function R induces mirror flow, the iterates Ia follow the dynamies:
dx.- - (V -1 Vx (xx)dd
To g( (Wiitt))
(6)
where the manifold metric is given by RV Accord- ingly, Theorem 3.6 leads to new geometric interpretation of a regularization. Specifically, 2t evolves as follows:
Theorem 3.10. Consider the same setting in Theorem 3.6. Furthermore, assume that o 2 0 and at = O for all t 2 T, where T > O. Moreover, for a E [b, Oj. Ra is a contracting Bregman fannttonnn for some < O. Assume that for all 1. 2 O the integral a; := - -J² a,ds .sss 2 b. For the loss function assume that Vzf is locally Lipschitz and argmin (a:) x E dom is non-empty. Then the following holds: If J is quasi-convex, T2 converges
dr. = - (V2 Ro, (xx+) (V (x+)) + oz v.ttt dl, (7)
with initialization To = g (init) and yo h (( where ye is defined as in Theorem 3.6. Thus, the effect of regular- ization on the training dynamics is described by a changing Riemannian metric, where the metric evolves according to the time-dependent Legendre function Ray
point x. which satisfies V Vz (I (x-x 2 0 for E domBaz. Furthermore, if f is convex, x, converges to a minimizer f in the closed domain domRa..
In practice. we can steer a and thus influence the manifold. Another perspective on this is that the effect of explicit regu- larization is stored in the time-dependent Legendre function Ra.- Therefore, explicit regularization has a lasting effect on the training dynamics, even after it has been turned off, for instance. This creates a novel connection between explicit regularization and implicit bias. Also past regularization influences future implicit bias by shaping the geometry.
Proof, see Theorem B.2 in the appendix. The proof consists of two parts: a) We show that the iterates are bounded up to time T using the contracting property and b) We establish convergence after time 7 using the geomet- ric interpretation of the evolution of xz.
Optimality To show optimality, we need more assump- tions the problem. As common in the context of mirror flows (Li et al., 2022; Jacobs & Burkholz, 2025), we re- cover -eder-ddeerrmined linear regression, as follows. Let {(2,, 9: y;) C Rd x R be a dataset of size n. Given a reparameterizztion g with regularization h, the output of the linear model on the i-th data is 2! g (ww). The goal is to solve
The geometric interpretation not only pro- vides valuable intuition but also helps us to show conver- gence of the mirror flow for time-dependent Bregman func- tions R..- A Bregman function is defined as follows:
Convergence
Definition 3.7. (Bregman function, Definition 4.1 (Alvarez
5

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
the regression for the target vector Y (y1, 9/...... -Wh)
Example 3.14. Consider the reparameterizztionn g: R x Rn - IR. such that g(u, v) = a(u) b(v), where a and b are positive, analytical, increasing functions. Then, the regularization Σi=1 (.a,(u) b.)) can always be em- ployed. By selecting ca 2 0 and co S 0, the optimization problem remains well-posed.
and input vector Z = (21, 22,..., 2n))
Theorem 3.11. Assume the same setting as Theorem 3.6. Furthermore, assume that 04 2 0) and 04 = 0 for all t 2 T, where T > 0. If It converges when t + oo and the limit Jex = Tim Zt satisfies zx = =Y, then the gradient flow minimizes the changed regularizer Ray:
To give concrete examples, this approach encompasses w2k ,2t (Woodworth et al., 2020) and log u - log v.
x argmin, z-= (2)(
Proof. See Theorem C.2.
Quadratic Reparameterizztions Next, we will discuss the class of quadratic reparameterizations, as described in Theorem 4.16 in (Li et al., 2022).
This theorem extends known optimality results on matrix sensing (Gunasekar et al., 2017; Wu & Rebeschini, 2021) and diagonal linear networks (sparse coding) (Pesme et al.,
Theorem 3.15. In the setting of Theorem 3.10, consider the commuting quadratic parametrization G: RR" - Rd and H: TR -> RR where each (ww)) = 1 _u= A/w and H 1 ww Bw with symmetric matrices A ,2...., Aa
2021; Woodworth et al., 2020; Jacobs & Burkholz, 2025).
2
3.4. Characterization of the Explicit Regularization
R DDD and symmetric matrix B E IR. DxD that commute
To make use of the established theoretical results in practice and develop promising regularization strategies, we charac- terize the explicit regularization for two important classes of reparameterizations: separable and quadratic reparame- terizations.
with each other, i.e., A,Aj -A 0 for all i,j E [d] and BA, A, = 0 for all j E [d]. For any Winit E R,,, Aquinis==1 and W.H.(Wiist) Buinit are linearly independent, then the following holds:
if V. (.
1) Qu())) = blll exp(aB + Ei= (u)es)iisst li. is time- dependent Legendre function with domain Rdd
Separable Reparameterizations Our next result encom- passes most previously studied reparameterizations within mirror flow framdworkkWooddorth Woodworth et al., 2020; Pesme et al., 2021; Gunasekar et al., 2017).
2) For all a E IR R. is Bregman function with domR,
range V Qa- Furthermore, if B is positive semi- definie,, then dRa < 0, therefore Theorem 3.10 applies. da
Corollary 3.12. Let g be separable reparameteriza- tion such that ge(ws)
(1=19.)( and h(w) E==EEH (wrs)) where 9o.j : + TR and hig3 : R R. Furthermore, assume that g and h are analytical functions. Then, if and only if h and g satisfy
Proof. The first statement is derived by applying Theorem 4.16 from (Li et al., 2022). The second statement follows from recognizing that exp(aB) acts as a linear transfor- mation of the initialization Winit. Subsequently, applying Theorem 4. 16 of (Li et al., 2022) gives the first part of the last statement. remains to show that R2 is contracting. Since B is positive semi-definite, it follows that da Q. 0. By the reverse ordering property of convex conjugation, we have that do R. < 0. For completeness, let h > 0, then for E R. we have Quth 2 Qa. Applying the reverse ordering property implies Ra S Ra. Rearranging and dividing by h gives 1 concludes the proof.
hagj Ci.j9h.
Vi E [m].] E [mi],
where C.y R is constant, Theorem 3.6 applies.
Proof. The result follows from the commuting relationship between g and h. We use that the Wronskian between two analytical functions is zero if and only if they are linearly dependent (Bôcher, 1901).
R (Ra+h - Ra) < O. Taking the limit + 0
The next two examples highlight the utility of Corollary 3.
Remark 3.16. For the time-dependent Bregman function in Theorem 3.15 to be contracting, B needs to be positive semi-definite.
Example 3.13. The reparameterization g : R x R" R" such that g(u, v) u² - 0² with regularization of the form h(u, v) Em i=1 cuuj - c. Setting e, = and = 1 leads to weight decay regularization on the reparameterization.
Theorem 3.15 encompasses recent works on the reparame- terization g(m, w) = m @ w. where @ denotes pointwise multiplication (Hadamard product). It has been proposed to sparsify neural networks (Jacobs & Burkholz, 2025), and extends work on matrix sensing and transformers (Wu & Rebeschini, 2021; Gunasekar et al., 2017; Sheen et al., 2024). Furthermore, B = I corresponds to weight decay on the reparameterization, which is often used in practice.
Example 3.13 has also been used to study the effect of stochasticity on overparameterized networks (Pesme et al., 2021). We present a more general class of examples that always results in a well-posed optimization problem, i.c., h positive.
6

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Having identified classes where we can determine h, present applications illustrate how time dependence influences the dynamics.
20
4. Analysis: Effects of Explicit Regularization
X
10
We introduce several time-dependent Legendre functions demonstrate the wide applicability of our analysis. We aim gain insights into how explicit regularization affects implicit bias during training. In particular, we focus three distinct effects, as summarized below:
0
a
-2
Type of bias: The shape of Ra changes with a.
0
2
X
Positional bias: The global minimum of Ra changes with a.
(a) Time-dependent hyperbolic entropy.
Range shrinking: The range of VR, can shrink due to specific choice of a.
20
The Reparameterization @ The reparameterization g w) m@w w is an exemplary quadratic reparameteriza- tion, which can also be interpreted as the spectrum of more general quadratic reparameterizations. When the initializa- tion satisfies |wp] < |mal, then Theorem 3.15 holds, We can compute the time-dependent Bregman function Ra (x)):
10
0
-4
-3
2
a
z.aresinh Zi A((a)
V v2 + A.(a)² 2; log
1g.0 11,0
(9)
0
3
X
(b) Time-dependent entropy.
where A:(a) : 2 expp(2 u;,00 and uo = (mo +
wo)/v2 /2 and Up = (mo wp)/v2. This adapts the hyper- bolic entropy (Li et al., 2022: Woodworth et al., 2020; Wu Rebeschini, 2021), which now dependent on a. Note that we used Theorem 3.15 find Ra, can invert the cor- responding function Q.(),, where μ fo Vaf (z, )ds. The regularization thus affects the time-dependent Legendre function by changing a. This allows us to modulate between an implicit 122 and J regularization through explicit reg- ularization (Jacobs Burkholz, 2025). Moreover, a also controls the location of the global minimum, a smaller a corresponds to moving it closer zero. Therefore, we both change the type of bias and the positional bias. Similarly, in case Wg = mg > 0 we recover the entropy (Wu & Rebes- chini, 2021):
Figure 2. Illustrations of the positional bias and type of bias effects of explicit regularization on the time-dependent Legendre function. In both figures - Io aads. Depending the initialization of ma O w the time-dependent Legendre function given by Fig 2a or Fig 2b. Both exhibit type change towards L1 minimization.
Quadratic Reparameterizations Building on the char- acterization of the reparameterizzation m w, we study the more general quadratic reparameterizattons with weight de- cay (B = 1). This covers multiple architectures, including matrix sensing, attention and LoRA, explained in Ta- ble 1. For the dimensions of the parameters, see Table 6 in the Appendix. Theorem 3.15 might not hold. In case of matrix sensing, we are able to apply both Theorem 3.15 and 3.11, where the time-dependent Bregman function for the eigenvalues given by Eq. (10). Thus, weight decay modulates between maximizing and minimizing the nuclear norm of the matrix X = UUT The details are given in Appendix C. For atten- tion, a common building block of Transformer architectures (Sheen et al., 2024; Tarzanagh et al., 2023), additional as- sumptions such as the alignment property are required.
general, however, the assumptions of
log B((a))
-1) 24 aglogr, (10)
where (a) = exp (2a). Here modulates between maximizing and minimizing the -norm. Note that both time-dependent Bregman functions are contracting on a E (-o0, 0]. Moreover, Figure illustrates the effects of type of bias and positional bias for m ④ w.

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
for fixed, where int refers to the interior of the domain (see Lemma 4.8 (Li et al., 2022)). Furthermore, note that I E (-cu a, Cy + a). Since d, is negative, the domain of Qa shrinks over time. Thus, the range of V R. shrinks accordingly. This also lessens the set of acceptable solutions of the original optimization problem, which can make its solution harder. Figure 10 in Appendix illustrates the effect of range shrinking with an approximation of the time- dependent Legendre function.
Table I. Quadratic parametrization.
Matrix sensing
UUT SoftMax(QK"))
Attention
LoRA
Wo + AB
worth noting that attention also has value matrix V and an activation function. Assuming V is not trainable and that the function/ f encompasses the activation function, the gradient flow dynamics of X = QKT is described by Theorem 3.6. This characterizes the implicit bias and can be interpreted as proxy for training full attention. Similarly, for LoRA, finetuning mechanism for LLMs (Hu et al., 2022; Wan et al., 2024), the training dynamics of X = AB is described by Theorem 3.6, assuming the alignment property in addition.
Other Reparameterizationn Appendix E and D present more reparameterizations. In particular, we analyze a repa- rameterization that induces an L1 to L2 change in the type of bias (in contrast the more prevalent flipped change from L2 to L,). In addition, we highlight limitations of the framework by considering deeper reparameterizations
As observed in (Khodak et al., 2021) for quadratic reparame- terizations, weight decay promotes small nuelear norm. This not the full picture, however. According to our results, weight decay changes the manifold geometry according to Eq. (7), which leads to an implicit bias that modulates between the Frobenius and nuclear norm of the matrix X. The eigenvalues in Eq. (9) are therefore subject to a time- dependent Bregman function Ra, This is the most accurate description of the implicit bias among the Bregman func- tions in Eq. (9) and Eq. (10) that we have discussed for the parameterization m @ w, taking the initializations of both attention and LoRA into account. In case of atten- tion, the matrices are randomly initialized, which makes coupling between the spectrum of K and Q unlikely, i.e., mo Wo. In case of LoRA, the initialization is A = 0 and B is random. Regardless of the initialization, weight decay would still regularize towards the nuclear norm, but a coupled initialization would constrain the eigenvalues to be either positive or negative.
5. Experiments
We conduct three experiments to support our theoretical analysis. The first experiment on matrix sensing illustrates the positional bias and type change following Theorem 3.11. Accordingly, we turn off weight decay at some point during training and compare it with linear reparameterizzation with L1 regularization. The second and third experiments are finetuning a pretrained transformer network and an LLM with LoRA, respectively. Both exhibit a gradual change of the implicit bias from Frobenius to nuclear norm, while demonstrating lasting effect of dynamic regularization, which leads to better generalization. Notably, this highlights that our insights extend even to settings where our assump- tions are not strictly met. Moreover, Appendix E discusses the range shrinking effect on sparse coding. Note that the change in positional bias is present in all experiments.
Recovering the Sparse Ground Truth in Matrix Sens- ing We consider a matrix sensing experiment with the setup of (Wu & Rebeschini, 2021). Details can be found in Appendix C. The ground truth is a sparse matrix X* and the reparameterizatioo is X = UUT When initialized with UgU BT, the eigenvalues of X satisfy Theorem 3.11 with the time-dependent Legendre function in Eq. (10), For experiments labeled "furn-off", we turn off the regular- ization at time T = 625. Figure 3 demonstrates that we recover the ground truth after turning off weight decay for the quadratic reparameterization. This can be explained by the fact that the positional bias of the eigenvalues moves closer to zero over time, which causes the type of bias to change to the nuclear norm (see Figure 6) in the appendix. Disabling weight decay in this context reveals the accumu- lated effects of regularization. In contrast, using constant weight decay would prevent us from exploiting this effect. Also note that a linear reparameterization with L regular- ization could achieve high sparsity but at the expense of
The Reparameterization 2² - 2k The reparameteriza- tion 2. 2. serves as proxy for deep neural networks (Woodworth et al., 2020) and provides an example of range shrinking due to explicit regularization. We consider the regularization h(u,v) v) = Ei= ufth as allowed by Corollary 3.12. The current reparameterization also exhibits a change in the implicit bias from I.2 to L,, shown in Theo- rem 3 in (Woodworth et al., 2020). Unfortunately, there is no analytical formula available for the Legendre function in this case. Therefore we only derive the flow and its domain, which is the range of the dependenn mirror flow. The flow Qa. (Px) is given by
1
du
dg Me + ae + Co
(11)
Me + ae + Cn.
24 مi My = - V ff ds
where dy = ((2k- - 2) (2k)
and az = - fo agdss. We have that dom V R. = intdomQ
8

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
reconstruction accuracy, as it cannot recover the ground truth. The geometry of the manifold that defines the implicit bias does not allow for it.
wd 10 0.1
Nuclar wurm Frodeeiis
1.85
wd -.2, turt-oll
wd 0.2
wd - 0.5
wd - 1.0, turt-off
L1 O, knea
wd 1.0
L1 0.02, tum-off, linour
1.75
[1 0.02
10"
wd -
1:70
10'
wd 0.02, tum 0.00
100
200
Iterations
400
500
10
(a) LoRA.
Nuclear norm/ Probennas norm
10
0
250
500
Time
750
1000
1250
U,U22 lurt-off
wd 0.02
wd 0.1
Figure 3. Recovering the sparse ground truth by turning weight decay off for matrix sensing at T = 675. In contrast, linear repa- rameterization with L1 regularization goes towards the minimal L2 norm solution after switch-off.
wd - 0.2, tmm omm-off
we - 0.2
200
300
Egochs
Turning-off Weight Decay for LoRA and Attention Next, we aim to track the effect of the explicit regularization on the quadratic reparameterizations in Table 1. For LoRA, we calculate the nuclear norm and Frobenius norm of the matrix product X = AB, averaging these values across all layers, and then computing their ratio. For each attention layer, we apply the same procedure to the product of the query and key matrices X QK"", The ratio allows us to track the relative sparsity of the matrix. With LoRA, we fine- tune GPT2 (Radford et al., 2019) on the tiny_ "in_shhkespeare (Karpathy, 2015) dataset, training for 500 iterations in two different types of settings. In case of "turn-off", we turn the weight decay off at iteration 200. Furthermore, we fine-tune a pretrained VIT on ImageNet for 300 epochs, turning the weight decay off epoch 150. Figure. 4 shows that increas- ing weight decay reduces the reported norm ratio, indicating change in the type of bias from L2 to T. Moreover, when weight decay is turned off, the ratio intersects with other ratios that are attained by constant weight decay. This cre- ates a "window of opportunity" for unconstrained training with relatively low nuclear norm, leading to improved test accuracy (see Appendix F), in particular, in comparison with constant weight decay trajectories. For a ViT on ImageNet this can lead to more than 1% improved validation accuracy for similar relative sparsity.
(b) Attention.
Figure 4. Ratio between the nuclear norm and Frobenius norm for LoRA and attention. Training with higher weight decay and then turning it off in the shaded region allows for exploring the parame- space at higher sparsity. This creates window opportunity improve performance in a relatively sparse regime.
systematic procedure for identifying suitable regularizations for given reparameterizzatooss and established convergence and optimality within our framework. As the implicit bias can change dynamically during training, it is associated with time dependentd Legendre function, which might be con- ceptually of independent interest. To demonstrate the utility of our theory in applications, have presented experiments sparse coding, matrix sensing, attention in transformers, and LoRA fine-tuning. As we found, switching off weight decay at some point during training could improve general- ization performance by exploiting the accumulated effect of past regularization.
future, our insights could guide the development of more effective regularization techniques that account for implicit bias, such dynamic weight decay strategies tailored to specific model architectures. Moreover, our framework could be used to analyze the impact of early stopping and extended to incorporate other regularization factors like the impact of a large learning rate and stochastic noise.
6. Discussion
We have introduced framework for analyzing the impact of explicit regularization on implicit bias and provided novel geometric interpretation of their interplay. By extending the mirror flow framework, we have outlined method to con- trol dynamic implicit bias through dynamic explicit regular- ization. Our analysis has characterized their joint effects on the training dynamics, including positional bias, type of bias, and range shrinking. Additionally, we have established a
9

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Acknowledgements
Beck, A. and Teboulle, M. Mirror descent and nonlinear projected subgradient methods for con- vex optimization.
The authors thank Celia Rubio-Madrigal for proofreading and designing Figure 1. Moreover, the authors gratefully acknowledge the Gauss Centre for Supercomputing e. V. for funding this project by providing computing time the GCS Supercomputer JUWELS at Jülich Supercomputing Centre (JSC). We also gratefully acknowledge funding from the European Research Council (ERC) under the Horizon Europe Framework Programme (HORIZON) for proposal number 101 16395 SPARSE-MI.
Operations Research Letters, 31(3):167-175, 2003. ISSN 0167-6377. doi:
URL https///ww..ciinnediieeet. /etee/eeaatticlle////////77/////
Belkin, M., Hsu, D., Ma, S., and Mandal, S. Rec- onciling modern machine-learning practice and the classical bias varianar trade-off. Proceedings of the National Academy of Sciences, 116(32):1584-- 15854, 2019. doi: 10.1073/pnas. 1..0733/naa..9900001166 1903070116. URL https: /www. pnas. pnas 1903070116...
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume Springer, 2006.
References
Böcher, M. Certain cases in which the vanishing of the wronskian is a sufficient condition for linear dependence. Transuctions of the American Math- ematical Society, 2:139-149, 1901. URL https: taapp.ssmmnticsctie www.eccchhlaarooooooooopsseeeeee 123122034.
Alvarez, F., Bolte, J., and Brahic, O. Hessian rieman- nian gradient flows in convex programming. SIAM
Journal Control and Optimization, 4372):777-501
January 2004. ISSN 1095-7138. doi: 10.1137/
403630129024199977 URL http://dxx doi.org/ 10. 00.11377803630112244199997
Boyd, S. P. and Vandenherghe, L. Convex optimiza- tion. 2009. URL https: /wtb///wbbbb.eeeee stanford. -boyd/ cvxbook/.
Arora, S., Cohen, N., Hu, W., and Luo, Y. Implicit regu- larization deep matrix factorization. In Wallach, H., Larochelle, H., Beygelzimer, A., d' d'Alché-Bue, F., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL /ttps://prrceeeiings. new.eeeedddings..n.p........... ce paperppppppe htlll/aapfffeeeaeeeaa htwe//fleessppeee////////e c0c78 78777835566m aat5fffcccco00081111444aaaaaaa pdf.
Cubuk, E. D., Zoph, B., Shlens, J., and Le, Q. V. Ran- daugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 702-703, 2020.
Dai, Z.,, Karzand, M., and Srebro, N. Representation costs of linear neural networks: analysis and design. In Proceedings of the 35th International Conference on Neural Information Processing Systems, NIPS '21, Red Hook, NY, USA, 2021. Curran Associates Inc. ISBN 9781713845393.
Arpit, D., Zhou, Y., Ngo, H., and Govindaraju, V. Why regularized auto-encoders learn sparse representation? In International Conference on Machine Learning, pp. 136-144. PMLR, 2016.
Azizan, N., Lale, S., and Hassibi, B. Explicit regular- ization via regularizer mirror descent. arXiv preprint arXiv:2202.10788, 2022.
D' Angelo, F., Varre, A., Andriushchenko, M., and Flam- marion, N. Why do need weight decay for overpa- rameterized deep networks? In NeurIPS 2023 Work- shop on Mathematics of Modern Machine Learning, 2023. URL pttttss:/openreviiiie net. wtt.rieewwee//orumm forum? id=3yBtKsiHEC.
Azulay, S., Moroshko, E., Nacson, M. S., Woodworth, B. E., Srebro, N., Globerson, A., and Soudry, D. On the implicit bias of initialization shape: Beyond infinites- imal mirror descent. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Ma- chine Learning Research, pp. 468 468. 40MS-R7..MM, 18-24 Jul 2021. URL https: htt.lhtttttppoeeeeedeeeeee ngs mlr. .resss/ v1 1.39/azulay211 html.
Daubechies, I., Defrise, M., and De Mol, C. An iterative thresholding algorithm for linear inverse problems with a sparsity constraint. Communications on Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, 57 1111111411145555 2004.
10

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Even, M., Pesme, S., Gunasekar, S., and Flammar- ion, N. (s)gd over diagonal linear networks: Im- plicit bias, large stepsizes and edge of stability. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S. (eds.), Advances in Neural Information Processing Systems, vol- ume 36, pp. 29406-29448, Curran Associates, Inc., 2023. URL https://proceedinggsss neurips. cc cc/paper cc/pppper_flle//pppeeeeeee [aapee__Hlles/pppee//22////1
pp. 448-456, Lille, France, 07-09 Jul 2015. PMLR. URL httttps:/pproceeiimge ml w.w.ppres/vv.. /v3// ioffe15.html
Jacobs, T. and Burkholz, R. Mask the mirror: Implicit sparsification. In The Thirteenth International Conference on Learning Representations, 2025. URL https:// openreview net forum Ud-U47ymIS3utt
Julistiono, A. A. K., Tarzanagh, D. A., and Azizan, N. Comper-Con Oftimiiiin attention with mirror descent: Generalized max-margin token selection. ArXiv, abs/2410.14581. 2024. URL https: /apii. semanticschole httpsss/app...mmmmiicccco.ooo org/ CorpusIDD273482271 273482797
5da olacces0ee77771176011222222222 pdf.
Gonzalez, T., Guzman, C., and Paquette, C. Mirror de- scent algorithms with nearly ianeensionnindependeen rates for differentialll--rriatee stochastic saddle-point problems extended abstract. In Agrawal, S. and Roth, A. (eds.), Proceedings of Thirty Seventh Conference Learning Theory, volume 247 of Proceedings of Machine Learn- ing Research, pp. 1982-1982. PMLR, 30 Jun-03 Jul 2024. URL BRRhttps: :ttptt::::/pppcceeeee http://pooceeeinnnnnccc mww.m.r.... .ress// v247/gonzalez24a.
Karimi, M. R., Hsieh, Y.P., and Krause, Sinkhorn flow
mirror flow: A continuous-time framework for general- izing the sinkhorn algorithm. In International Conference on Artificial Intelligence and Statistics, pp. 4186 PMLR, 2024.
Karpathy, A.
char-rnn. https://github.ce karpathy / charrrnn, 2015.
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y. Deep learning, volume 1. MIT Press, 2016.
Khodak, M., Tenenholtz, N. A., Mackey, L., and Fusi, N. Ini- tialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2021. URL https: /oppnrevieeeeeeeeeeeeeeeeeeeeeee net/ forum? id-KT1JTInof6d
Gunasekar, S., Woodworth, B. E., Bhojanapalli, S., Neyshabur, B., and Srebro, N. Implicit regularization in matrix factorization. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.). Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https:: neurips......................... oc cc/ saaeer__ilesspaaeee pttppee_fflleeesppppp///0/////11 581 (1111cccee1cccceee66777ccededddd Paper. pdf.
Kobayashi, S., Akram, Y., and Oswald, J. Weight decay induces low-rank attention layers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https::: /ppnrevieeeeeeeeeeeeeeeeeeeeeeee .nt// forum? Dorum?dd=oDeqjJNa
Krogh, A. and Hertz, A simple weight decay can im- prove generalization. Advances in neural information processing systems, 4, 1991.
Gunasekar, S., Lee, J., Soudry, D., and Srebro, N. Char- acterizing implicit bias in terms of optimization ge- ometry. In Dy, J. and Krause, A. (eds.), Proceed- ings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learn- ing Research, pp. 1832-1841, PMLR, 10-15 Jul 2018. URL htttttts:/pprceeeeeee gunasekarl8a.cm himl.
Kukačka, J., Golkov, V., and Cremers, D. Regularization for deep learning: A taxonomy, 2017. URL https: //arxiv. wwg/ wpgg/abs//71..11 1ttp://110..18888
mlr wwl..presse press/v80/
Lee, J. Introduction to Smooth Manifolds. Graduate Texts in Mathematics. Springer New York, 2013. ISBN 9780387217529. URL https://books. google. de de/bookssid=w<<bbeae
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.,, and Chen, W. LoRA: Low-rank adap- tation of large language models. In International Confer- ence on Learning Representations, 2022. URL https: /pppnreviewwweecc/ooeeeepooeeeee net forum? ideeneeeeeee..s....... htt.:///oorrmmiimmaaeeeeeee.
Li, Z., Luo, Y., and Lyu, K. Towards resolving the im- plicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Confer- ence on Learning Representations, 2021. URL https: / //openreviww.o net htt///oormm//// wwwoorummii=AAHoo d=AHOs 7m5H7R.
loffe, S. and Szegedy, C. Batch normalization: Accelerat- ing deep network training by reducing internal covariate shift. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, vol- ume 37 of Proceedings of Machine Learning Research,
Li, Z., Wang, T., Lee, J. D., and Arora, S. Implicit bias of gradient descent on reparametrized models: On equiv- alence to mirror descent. In Oh, H., Agarwal, A.,

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Belgrave, D., and Cho, K. (eds.), Advances in Neural
2005013117. URL https: /tt///wwwww//.
http://wwwwpa......oom www.:/passsso...oo.oooo /do/ 10..0...77/pppp/ 2005013117.
Information Processing Systems, 2022. URL https: //openreview hotp://ooommmmdkkK__adad
Radhakrishnan, A., Belkin, M., and Uhler, C. Linear conver-
Liu, Z.,, Michaud, E. J., and Tegmark, M. Omnigrok: Grokking beyond algorithmic data. In The Eleventh International Conference on Learning Representations, 2023. URL https: ht/p//pperrrrreeeeeeeeeeeeeeeeee wtt.:/iiiwwwee/ooomnn Id=zDiHoIWale
gence of generalized mirror descent with time-dependent mirrors, 2021.
Raj, A. and Bach, F. Explicit regularization of stochas- tic gradient methods through duality. In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th In- ternational Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Reseurch, pp. 1882-1890. PMLR, 13-15 Apr 2021. URL https: http:s/://pcccecingsssse /ttp.u/LLtttt::ppreeeeinop v130/raj2la.. html.
Lyu, B. and Zhu. Z. Implicit bias of (stochastic) gradient de- scent for rank-1 linear neural network. In Teirty-seventh Conference on Neurul Information Processing Systems, 2023. URL https::/t///tttt http:///pppeeeeeeeeeeeeeeeeeeeee net forum? id-P -BEUTVEE..
mlr press/
Lyu, K., Jin, J., Li, Z., Du, S. S., Lee, J. D., and Hu, W. Dichotomy of early and late phase implicit biases can provably induce grokking. In ICLR, 2024. URL https: // //openreview. net forum? 9dEG..
Rockafellar, T. R. and Fenchel, W. Conver Analysis. 1970. URL /attps://ap..... semantimani...e semmanticccoolrr org/ Sts/CorppaID::98120333
Marion, P. and Chizat, L. Deep linear networks for gression are implicitly regularized towards flat minima. In The eiirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: //apenreview net wwttfforrnneeee fmr wwwiiddFFFwii. Xm4.
Sheen, H., Chen, S., Wang, T., and Zhou, H. H. Implicit regularization of gradient flow one-layer softmax at- tention, 2024.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1299-1958 2014.
Orvieto, A., Raj. A., Kersting, H., and Bach, F, Ex- plicit regularization in overparametrized models via noise injection. In Ruiz, F., Dy, J., and van de Meent, J.-W. (eds.), Proceedings of The 26th Interna- tional Conference on Artificial Intelligence and Statis- tics, volume 206 of Proceedings of Machine Learn- ing Research, pp. 7265-7287. PMLR, 25-27 Apr 2023. httsstss: pttp///pcceeengs.eeee .lr mll.press// v206/ 2206/orrveeo2233. html.
Sun, H., Ahn, K., Thrampoulidis, C., and Azizan, N. Mir- ror descent maximizes generalized margin and can be implemented efficiently.
Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https: / //openreview net/ www.ffoddddoooooooooo
Pesme, S., Pillaud- Vivien, L.. and Flammarion, N. Implicit bias of SGD for diagonal linear networks: a provable benefit stochasticity, In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https: //openreview net forum? d=vvi d=vvi7KgHQiA.
Tarzanagh, D. A., Li, Y., Thrampoulidis, C., and Oymak, S. Transformers as support vector machines. In NeurlPS 2023 Workshop on Mathematics of Modern Machine Learning, 2023. URL https: http:s/////eeeeeeee...oo .et//
Tian, Y. and Zhang, Y. A comprehensive survey on reg- ularization strategies in machine learning. Information Fusion, 80:146-166, 2022.
Pesme, S., Dragomir, R.-A., and Flammarion, N. Implicit bias of mirror flow on separable data. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: wwwp://veepereeeeeeoomoo forum? wwwa:idwwwwiiasi
Tropp, J. Greed is good: algorithmic results for sparse approximation. JEEE Transactions on Information The- ory, 50(10):2211-2422, 2004. doi: 10. 10.1104//T,.2004. 834793.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019.
Vasudeva, B., Deora, P., and Thrampoulidis, C. Implicit bias and fast convergence rates for self attention, 2024.
Radhakrishnan, A.. Belkin, M., and Uhler, C. Over- parameterized neural networks implement associative memory. Proceedings of the National Academy of Sci- ences, 117744):2222 2020. doi: 10.1073/pnas.
Vaswani, A. Attention all you need. Advances in Neural Information Processing Systems, 2017.
12

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Wan, Z., Wang, X., Liu, C., Alam, S., Zheng, Y., Liu, J., Qu, Z., Yan, S., Zhu, Y., Zhang, Q., Chowdhury, M., and Zhang, M. Efficient large language models: A sur- vey, 2024. URL atttp:://axiii http:/iww.rrgaab///33333 03863.
Woodworth, B., Gunasekar, S., Lee, J. D., Moroshko, E., Savarese, P., Golan, L. Soudry, D., and Srebro, N. Kernel and rich regimes in overparametrized models. In Abernethy, J. and Agarwal, S. (eds.), Proceedings of Thirty Third Conference on Learn- ing Theory, volume 125 of Proceedings of Machine Learning Research, pp. 3635-3673, PMLR, 09-12 Jul 2020. URL.ttpps: /proceedd ngs..ng. ml press/ v12 v125 2125//ooodworth20a html.
Wu, F. and Rebeschini, Implicit regularization in matrix sensing via mirror descent. In Neural Infor- mation Processing Systems, 2021. URL https: //api.semantteeeeeeee/..//a/aaaa www.ssseatiicchhoss.... org/CorpusID: 235248126.
Xie, Z., Sato, I, and Sugiyama, M. On the over- looked pitfalls of weight decay and how mitigate them: A gradient-norm perspective. In Neural In- formation Processing Systems, 2023. URL https: //api. seeannticscholan Crg/CorpsIID: 237581592.
Zhang, Z., Xu, Y., Yang, J., Li, X., and Zhang, D. A survey of sparse representation: algorithms and applications. IEEE access, 3:490-5300 2015.
13

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
A. Implicit bias framework
In this section, for completeness, we present the existing results for the mirror flow framework. Consider the optimization problem in Eq. (1) for loss function f IR" R
min f(x).
FER
We can use the implicit bias framework to study the effect of overparameterization An overparameterization can be accomplished by introducing a function g : M + R", with M a smooth manifold. For particular g. the reparameterization of the loss function f' leads to mirror flow. general framework is given in (Li et al., 2022) to study the implicit bias in terms of mirror flow. Let R R - R be Legendre function (Definition 3.4), then the mirror flow described by
dV R(r,) = - V (
Tintt = g(Waist)
(12)
(Li et al., 2022) provide sufficient condition for the reparameteriization 9 such that it induces a mirror flow Eq. (12). The Legendre function R controls the implicit bias.
Definition A.1. (Legendre function Definition 3.8 ((Li et al., 2022))) Let R: Rd + R U {oo} be diffcrentiable convex function. We say is a Legendre function when the following holds:
R is strictly convex on int(domR))
For any sequence { going to the boundary of domR, lim; [[ = co.
In order to recover the convergence result Theorem 4.14 in (Li et al., 2022) the function It also needs to be a Bregman function, which we define in Definition 3.7.
Definition A.2. (Bregman function Definition 4.1 (Alvarez et al., 2004)) A function R called a Bregman function if it satisfies the following properties:
domR is closed. R is strictly convex and continuous on domR. R is C¹ on int(domR)).
For any dom/ and y IR, {y dom/t)DR(z.1) v) S > is bounded.
For any x E domR and sequence {=:) C (int(domR) such that lim;-roo n, x., it holds that lim;-soo D1a( 0.
For reparameterizzatio to induce a mirror flow with corresponding Legendre function we first have to give two definitions. Furthermore, we define Ög as the Jacobian of the function g. Definition A.3. (Regular Parmeterization Definition 3.4 (Li et al., 2022)) Let M be a smooth submanifold of RP, A regular parameterization g M- - R.Th C*l parameterization such that (G(w) of rank n for all w E M.
For the second definition, we first need define what a Lie bracket is. Definition A.4. (Lie bracket Definition 3.4 (Li al., 2022)) Let M be smooth submanifold of RP. Given two C' vector fields X,Y Y M, we define the Lie Bracket of X and Y as [X, ()(W) = 0Y (w)((w))))))))())))))))(()))(( ax (w) (w))
Definition A.5. (Commuting Parameterization Definition 4.1 (Li et al., 2022)) Let M be a smooth submanifold of R., A C2 parameterization g M + Rd is commuting in a subset S C M iff for any i.j E [n). the Lie bracket [Vi,, Vg,] (w) = 0 for all w S. Moreover, we call g a commuting parameterization if commuting in the entire M.
Besides these two definitions, we need make an additional assumption on the flow of the solution. We define the solution of the gradient (descent) flow of function M - R. initialized at & E M
dr = V (xt
Io = x
(13)
as = (x) which is well defined if the solution exists. Using this we can make the following assumption. Assumption A.6. (Assumption 3.5 (Li et al., 2022)) Let M be a smooth submanifold of R and g A - 1 be a reparameterization. We assume that for any w E M and i E [m], oi, (w) is well-defined for t E (T_, TT,,) such that either lim;-r, |l0 (w)llz2 = oc or T4 = oo and similarly for T_. Also, we assume that for any w E M and i, E [n], it holds that for (t, 8) E R that og oi (w) is dell-defined iff 05 Φ (w)
14

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Using these definitions state the known result for mirror flow.
Theorem A.7. (Theorem 4.9 (Li el al., 2022)) Let M be smooth submanifold of TR and g : M - Re be a commuting and regular parameterization satisfying Assumption A.6. For any initialization Winit M, consider the gradient flow for any time depe-dennndnt loss function L; R R.
dw,-
w. (g((w)))
Wiiit
Define 2, g(we) for all then the dynamics of æt is mirror flow with respect the Legendre function R. given by Lemma 4.8 in (Li al., 2022), ie.,
dV R(x+) - (V
To (Wini))
Moreover, this only depends on the initialization Wint and the reparameterization g. and is independent of the loss function T.,.
We have used Theorem A.7 to show the Theorem 3.6 the main text. Morcover, we recover the convergence result for Bregman functions in Theorem 3. 1.10. The details of these results are presented in Appendix B.
15

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
B. Proofs of Section 3
Theorem B.1. Let g + R and h M R regular and commuting parameterizations satisfying Assumption A.6. Then there exists time-dependent Legendre function R., such that
A. (xe) -Vx[(ziddd,
Z0 = g g(Watt))
where J agds. Moreover, Ba only depends the initialization Wins and the reparameterization and h and is independent of the loss function f.
Proof. Consider the time-dependent loss function (x(x,0) ((x) aty. Applying Theorem A.7 implies there is Legendre function R(x, y) such that
V V2R(Ee ve) S vaf(x,)ds Bg R(ze, = So a,ds.
(14)
We Eq. (14) derive the time-dependent Legendre function. First that ,,,,, v) >> 0 for (Ix, domR since R is strictly convex. This implies that the map + R(x, y) is invertible. Let the inverse be denoted by Qxx, a), where in dynamics 0, = - So asds. Plugging Q into the first part of Eq. (14) gives us
(x), a:)) = f Vaf (xx) ds,
V.R( (FL,
(15)
where va is still the derivative with respect to the first entry. Eq. (15) looks already like a time-dependent mirror flow. We show now that there exists a Legendre function R, with the map V, R (zz, Q (z,aa) as the gradient. This we can do by showing that the Hessian symmetric and positive definite and that the Rs essentially smooth.
By implicitly differentiating, we make the following observation:
dQ
V R. Q).
dr
3, Q)
Next, we compute the Hessian and apply observation B:
V = V & dQ R( Q) Rio Q)
dx Q.
Observe that this matrix is symmetric as it is sum of symmetric matrices. It remains to be shown that the Hessian matrix is positive definite. For this, we use that V R and 82 R are positive definite. This implies that the the Hessian of R inverse is PD. We use the block inversion matrix formula for matrix M,
A
B D
M
C
where A and D are square and invertible. For notation clarity, define the Schur complement of D as:
S S
Then the inverse of M is given by:
8-1 -D-'CS-
-S-IBD-1 D- D'1DD'CS--BDD
M
The first block entry of the matrix 2 R is given by the inverse Shur complement:
v2 R(R,v))) 0,0,R(e.V)
8 N.
16

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
which is also PD. Now this implies the result the inverse of V2 R. is PD. It follows that there exists function Rg such that V Ra Rx,, a)) by Corollary 16.27 in (Lee, 2013), concluding the first part.
Finally, R₂ is essentially smooth by construction, using that R is essentially smooth. The boundary bn( RR)) by construction is the set of points x= that have sequence In E domintv, Q(: a)) such that if In + x² we have |||R|| + oc. Suppose that Ra is not essentially smooth then there exists a sequence { with In - bd( Ra) as n - oo such that lim Rl(nn Q(Tn,,))ll,, l)) < oo. Nevertheless, It is essentially smooth this implies that
lim 11 y, Q(In, a))l =d² ==== 00,
leading to contradiction. Hence, R. is Legendre function with the domain similarly constructed the boundary. D
Theorem B.2. Assume the same settings as Theorem 3.6. Farthermore assume that for ox 2 O there is T > 0 such that for t 2 T. a = 0. Moreover for a E [b, 0], Ra is contracting Bregman fanction for some b < O. Assume that for all t 2 0 the integral a, := So a,ds 2 b. For the loss function assume that zf is locally Lipschitz and argmin{ x E domRa non-empty. Then if f is quasi-convex xt converges to point x. which satisfies V ( (x x.) 2 0 for x E Ra In addition if f is convex I. converges to a minimizer f in Ram
Proof. We can bound the trajectory of zt by using the time-dependent Bregman divergence. The divergence between critical point x² of f and the iterates Iy is given by
Dag (x 2.) Ra, (x*) (14) V (x.) xz) 2 0
Note that the contracting property implies that for 02 I 01 have domRez C domRa,- Thus, a critical point x* in dom RamBa is in domRa, Hence, the divergence is well-defined. Due to that f is quasi convex and R, contracting we have that Do. (x", .r.) is bounded. From the contracting property it follows that Ra (x*) 2 Ra, (2*). By definition of Bregman function, we have that 2t stays bounded for all t 2 0. It follows that æt is in the domain of Ra and bounded. Therefore, we have that Da, (2", 221 I Ra (x*) - Ra, (z,)) - V Ra, (zz)7(x² (x" - xx) =: We Now we show that the evolution of Wi decaying, implying that Da, (x",, v4) is bounded. The evolution given by
d Ra, (xt) )dt - Ra, + V. (2:) dV dV (1s) 12)
dW. at da
< dV ala (zz) Tx= xz)
0
where used that o, 2 0 and the contracting property for the first inequality, the time dependendent mirror flow relationship
in the second and quasi-convexity for the last. Therefore z, stays bounded for t [0, TJ. Now, using the geometerie interpretation Eq. (7) we have that the evolution of ã, 4T is a gradient flow on Riemannian manifold with metric Ra Therefore Theorem 4. in (Li et al., 2022) applies, which concludes the result. D
17

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
C. Optimality characterizing the implicit bias.
In this section, we state a general result for underdetermined linear regression extending Theorem 4.17 (Li et al., 2022). Moreover, we state detailed result for matrix sensing extending Corollary (Wu & Rebeschini, 2021).
For underdetermined linear regression, let {(z,, u+)) C Rd x R be dataset of size n. Given reparameterization g with regularization ht the output of the linear model on the i-th data is = g(w). The goal is to solve the regression for the label vector Y = (y1, 1/2,- .Un) For notational convenience, we define Z = (=1: 22, ................................ in) E Raxn
In order to show optimality for underdetermined linear regression we use the following Lemma:
Lemma C.I. (Lemma B.1 (Li et al., 2022)) For any convex function R. R + R U {oo} and Z E Raad suppose VR(x*) x forr some X E R", then R(x*)=
2: x=Z(z==")== R(x).
min
We now denote the function to be optimized by f(x)= = f(Zx -Y), emphasize the linearity of the optimization problem. Theorem C.2. Assume the same settings as Theorem 3.6. Furthermore, assume that for as 2 there a T > 0 such that for 2 T, ai = O. If de converges as - os and the convergence point Lix. = It. satisfies Z = Y, then
argmin 2: Z==Y Ray (x:).
(16)
Therefore, the gradient flow minimizes the changed regularizer Ray among all potential solutions.
Proof. Since we assume convergence i.e. Zwo = we have to show the KKT condition associated with Eq. (16) is satisfied. We have to show that Vier (x*) E span (z This follows directly from integrating the time-dependent mirror flow for t 2 T:
V Ror (2+) V Rao (x0) = -z" J. vi(zz, - Y) ds E span (z").
Notice by definition of the Legendre function and its convex conjugate we have that V Rao (mr) = 0. Therefore, V Ray (It) span (z²), which further implies that V Ray (Ioo) E span (2²). Applying Lemma C.I concludes the result. O
Theorem C.2 shows optimality for underdetermined linear regression. Moreover, together with Theorem 3.15 we extend results by series of papers on quadratic reparameterizations (Jacobs Burkholz, 2025; Li et al., 2022; Gunasekar et al., 2017; Azulay et al., 2021; Wu & Rebeschini, 2021).
We will focus now focus one particular example: matrix sensing (Wu & Rebeschini, 2021: Gunasckar al., 2017).
The reason for this focus is to show the effect on the spectrum on the matrix. We show we can modulate between the Frobenius norm and nuclear norm similar to the modulation between Lg and L1 regularization as in (Jacobs & Burkholz, 2025). Moreover, we can induce new grokking effect distinct from (Lyu et al., 2024; Liu et al., 2023), which considers large initialization and small weight decay.
Denote by A, E Rax= with i [m] the sensing matrices and consider the loss function f(X) 1 Em ((A, - yi)² Moreover, let S be the class of symmetric positive semi-definite matrices of size
2m
Corollary C.3. Assume that the sensing matrices Ai's 's are symmetric and commute, and that there exists a X* E St
satisfying f(X+) = 0. Moreover, assume that for an 2 0 there is T > such that for t 2 T. 0g = Then, the gradient flow defined by dt - v (U,UUF U,U, and any initialization satisfying UDU = 3T converges matrix U minimizing
de
log AT (
-
Adlog
(17)
among all global minima of f. where (Ar denote the eigenvalues of the matrix X = US and Ar =
exp (2 fo a,ds).
Proof. Convergence follows from Theorem 3.15 and optimality follows from Theorem C.2. left to show that the corresponding time-dependent Bregman function Ra, (X,) is given by Eq. (17). From the gradient flow we can derive the
18

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
time-dependent Bregman function:
-1) XxlogXi
Ray (Xt) = -r X2
log
cxx
-2 Sü agds)
Next, from the eigenvalue decomposition for symmetric matrices and the fact that all the above matrices are simultaneously diagonalizable, we get Eq. (17). D
Experiment To illustrate the implication of Corollary C.3 conduct experiment on matrix sensing. The implication is that when we train with weight decay it is stored in the the time-dependent Bregman function. We can leverage this lasting effect by turning off the weight decay and reach the sparse (optimal) solution as in Eq. (16). This allows us to induce a grokking-like phenomenon.
We use a similar experimental setup as in (Wu & Rebeschini, 2021). Specifically, we generate a sparse groundtruth matrix X by first sampling U R"*r,, where r 5, with entries drawn i.i.d. from NN 1). We then set X U (U*)² and normalize it such that ||X*|| 1. We generate m symmetric sensing matrices A; == (B, + BB,", where the B; entries are drawn L..d. from N (0, 1). We use learning rate 7 = 0.25, initialize Un- 13 with B = 0.1 and train for 5000 steps. We consider 3 scenarios:
Train without weight decay i.e. a
Train with weight decay a = 0.01, 0.02
Train with weight decay a 0.02 for 2500 steps and after that turn weight decay off, i.e. 0.
The second scenario (a = 0.01) and the third scenario are constructed such that the same amount of total regularization is applied at the end of training. Moreover, the same setup is used for the linear paramterization with L, regularization.
In Figure. 5 we present the evolution of the training and reconstruction error ||x Xell3ro We observe that in all scenarios the training error is below 10-2, contrast, only the reconstruction error for the third scenario where we turn off the weight decay goes below 10 Therefore, by accessing the stored weight decay we can reach closer the sparse solution illustrating Corollary C.3. This is also confirmed by the evolution of the nuclear norm in Figure. 6.
Note that keeping the weight decay on in the second scenario (wd 0.1) also leads to a better reconstruction error than the overfitting of scenario one (wd = 0). Nevertheless, due to the explicit trade-off optimality is not possible. This is not in contradiction with the analysis in (Lyu et al., 2024), where very small weight decay is used. In the case of small weight decay, the trade-off is negligible.
The dynamics in Figure. are similar to the dynamics of the grokking phenomenon: generalization happens later after no progress. Nevertheless, a key difference that we induce it by turning off the weight decay in contrast to (Liu et al., 2023; Lyu al., 2024). This leads to relatively frost-grokking".
Ablation with different schedules Consider the family of schedules with constant regularization strength up specific time T; such that a, = og for /. < T; and a, = 0 afterward, for oxo a constant. We choose ox 0.02 and T; = 7/2. In addition, we consider a linear and cosine decay schedule for the regularization with the same total strength (i.c., same integral), but the regularization is switched off after half the training time to ensure convergence. To compare with the effect of turning off (t-o) the regularization, we include the constant schedule with regularization strength ag 0.01.
We observe in Table 2 that all schedules with decay or turn-off (t-o) converge solution with the same nuclear norm of the ground truth, confirming Theorem 3.6, while the constant schedule does not reach the ground truth.
19

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Table 2.
Performance of different regularization schedules.
Schedule
Nuclear norm
Train loss
Rec error
Time to 10-7 train loss
Constant 0.01, no t-o
0.93
7.2 10-4
3.9 x 10
Linear decay
1.00
1.8 x 10-8
2.3 x 10-4
661
Cosine decay
1.00
1.7 x 10-8
2.1 x 10
624
Constant 0.02, t-o
1.00
x x 10-8
1.7 10-4
716
Constant 0.2, t-o
1.00
2.7 x 10
2.7 x 10
209
Constant 2, t-o
1.00
2.1 x 10 10-10 10
2.4 x 10-3
209
Constant 20, t-o
1.00
7.9 X 10
1.4 x 10-6
239
10
wd 0.02. tumn off
wd 0.02, tmm
10
0.02
10a
Thiining ofree
Recomstustin
wd 0.01
wd 0.01
t0
10
10a
10"
250
500
750
10000
1250
10
250
500
750
1000
1250
Time
Time
(a) Training error.
Reconstruction error.
Figure 5. Train and reconstruction error for the matrix sensing experiment for quadratic parameterizations, Observe that both the training and reconstruction error decrease when the weight docay is turned off recovering the sparse ground truth.
0,02,
L1 0.02, mmn-off
1.2
0.02
L1 0.02
1.5
Vostear noem
wd
L1 0
1.0
wd 0.01
L1 0.01
Nuckar
1.0
0.5
250
500
750
1000
1250
250
500
750
1000
1250
(a) Quadratic reparameterization.
(b) Linear reparameteriiztiion
Figure 6. Tracking the nuclear for both quadratic and linear reparamcterization. In the of the quadratic parameterization, the effect of the regularization gets stored resulting in better reconstruction eиror in Figure. 3, whereas this does not happen in the linear case. In the lincar case, the nuclear norm increases to the level of that without explicit L, -errguularizatio.
D. Deeper reparameterization
In this section, we explore several reparameterizations and limitations of the framework. We show that Theorem 3.6 does not apply to linear parametrization. Moreover, Theorem 3.6 does not apply to overparameterizztions with depth larger than 2 and weight decay. Nevertheless, show in experiments that similar effects occur. We illustrate both the type change and range shrinking effect.
Linear parametrization From Corollary 3.12, we derive another corollary for non-overparrmmeterizel parametrization.
Corollary D.I. Let g(x)) be the identity parametrization and h E (RR",R) Then Theorem 3.6 applies if and only if. h is given by h(x)) = En= C,T4 + d where C, E IR are arbitrary coefficients.
Proof. To apply the theorem, h needs to be commuting with implying that 0,0, = 0 Vi E [n]. concluding the result. O
Corollary D.I poses a limitation in the applicability of Theorem 3.6. Since h is not positive for all æ E R", the resulting optimization problem iol-posed. Therefore, standard non-reparameterized loss functions cannot be analyzed in this manner.
20

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Training
0.4
Reconstuction
0.3
Error
0.2
0.1
0.0
250
500
750
1000
1250
Time
Figure 7. Training and reconstruction error for a = 0.02 and turnooff. We recover the sparse ground truth by turning weight decay off for matrix sensing.
We show that the current framework excludes higher-order reparameterization with weight decay. In order to show that
Beyond quadratic reparameteriiation
Theorem D.2. Let g IR + be given by ((w) == = wi a k 2 depth reparamterization. Moreover, let E R and h(w))= = Σ w. Then g and do not commute.
Proof. This follows directly from checking the commuting condition between g and h:
Vwg. V [wl)(w)= = Vug( Vu.(w) - V vuh(w)Viggw))
(4 2k) [E
(4 2k) (4))
In order for this to be equal to zero all products need to be zero. This implics that the gradient flow given by
du,
:
Val(g(w))) aww,dt,
[1 w..e_
becomes dw,- wi and is independent of Hence, g and do commute
Theorem D.2 implies that we can not apply Theorem 3.6. We note that the commuting condition is only sufficient criterion such that pair (g. h) is time-dependent mirror flow.
Experiment Although our theoretical result does not hold for reparametrizations with higher depth, we illustrate that the expected effects do occur as well for higher depth. We consider the reparameterization m @ w @ v for diagonal linear networks and compare with m @ w. both with weight decay. Moreover, we compare with the reparameterization m with L₁ regularization to motivate the importance of the geometry, which is controlled by the time-dependent Legendre function. This is similar to the matrix sensing case explored in the previous section and main paper.
Let d 40 be the amount of data points and Z (0,, I..) for à [d]. We assume sparse ground truth x such that ||="||,, = 5. The training labels are generated 10-3 and we use weight
100 the dimension of the data. We generate independent data
by yk = 2('x*. Moreover, the mean squared error loss function is used. The learning rate decay a E {0.01, 0.1, We run the 100000 steps with weight decay, after that we run the same amount of steps without weight decay. We initialize m = and w = 2 = 1, this ensures that both parametrizations are initialized at zero and have the same scaling. In this setup, we illustrate the type change similarly predicted for the parametrization m w. Moreover,
21

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
illustrate the range shrinking which occurs for higher depth parametrization u2 .22k Note that the ground truth has the following ratio between the L and L2 norm 2.23.
In Figure.8a 8a we observe for m that higher weight decay does not get closer to the ground truth after turning the L regularization off. This is in line with the fact that the regularization is not stored in the geometry as described by Eq. (7), By turning off the regularization we converge to the closest solution in Lz norm. This is best seen in Figure.9a, where the ratio increases above the value of the ground truth.
In Figure.8b 8b we observe for m @ w that higher weight decay gets closer to the ground truth after turning the weight decay off. This is in line with the fact that the regularization is stored in the geometry as described by Eq. (7) and type of bias change from L2 to L1. Furthermore, this is also confirmed in Figure.9b that for large weight decay, the ratio gets close to the ratio of the ground truth only after turning the weight decay off. This also illustrates Theorem 3.10 and 3.11.
In Figure.8c 8c, we observe for the regularization strength le- - 1 a similar effect corresponding to the type of bias change from Lg to L1. In contrast, the higher regularization does not exhibit the same behaviour. We claim this is due to the range-shrinking effect. To motivate this is not due to the dynamics getting stuck at x = we report the final value of the first parameter. The value is equal to 1.58 which is not equal either or the ground truth value 1. To add to this, in 9 unveil that the ratio for large weight decay stays constant.
In conclusion, the type of bias can improve generalization, whereas m ④ even goes to the ground truth with high regularization, m does not. Moreover, when we use higher order reparametrization such as m O w @ z we encounter a different phenomenon: range shrinking. To add to this, higher-order parametrization still exhibits the type of bias change in a certain range of regularization strength. Thus, our theoretical framework leads to verifiable predictions. These can be used to improve the training dynamics of neural networks in general.
16²
10
10
10
10
10
1674
10
10"4
16tt
1014
wi 1o-2
wd Io-2
vdd 1e-2
10'22
10
10:10
ao 100
wil 1dll
wd- 1x0
10.
10'2
10.
100000 Steps
150000
200500
50000
100000
150000
200000
50000
102000 Stops
150000
200000
Stpps
(a) The evolution of m.
(b) The evolution of m ② W.
(c) The evolution of m + w O
Figure& 8. Illustration of the effect of weight decay with higher order reparameterizaations on generalization performance.
sid - 1o-3
wd- Io-2
6'
1p-1
6-10"
wiw Io-1
4%110
4-10"
4x:00
3-10"
3-10"
wd 160
33:10'
2x10"
2-10
2x100
10.
50000
(a) The evolution of ma.
(b) The evolution of O u.
(c) The evolution of m O
Figure 9, The ratio between the L1 and Le for diagonal linear networks.
22

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
E. Sparse coding
extend our study to the traditional sparse coding problem, with our proposed reparameterization substituting the standard sparse coding step. For this experiment, we use the Olivetti faces dataset. We denote the dictionary with D, labels with 2, the code with g(u v) and regularization with h(u, v). The feature dimension of D is n. This is solved as a linear regression problem with mean squared error. We have used learning rate 7 0.001 L))(D) where Lip(D) D) denotes the resulting Lipschitz constant of the optimization problem depending on the dictionary D. In addition, we set the number of features 50 and run for 100 iterations.
The reparameterization a* 22k In this context, we reparameterize the sparse code as g(u, v) = n²* - 22 and set the regularization h(u, v) = Ei= u² + v²th as discussed in the main paper. This parameterization exhibits range shrinking as illustrated by the time-dependent Legendre function Figure 10.
0.002
Ra(x)
0.00
0.000
0.6
0.2
0.0
-0.2
X
Figure 10. The evolution of the approximated R2 associated with a² v., where a = - ands.
The parameters are initialized as Uh = (V +
+x) and un 12 + x) where 3 = 1, NOO, I,). and all operations are pointwise. We set the regularization strength to 0.001 and explore various values of k E [7]. Throughout the training process, we track two key metrics: the reconstruction mean squared error (MSE) and the nuclear norm the sparse code, g(u, v) = 2h 2 The evolution of the nuclear norm is presented in Figure. la. We observe the effect of the range shrinking for 1, for larger & the evolution of the nuclear norm becomes stationary faster. This indicates that the range in which the dime-dependent Legendre function is allowed to move has shrunk. The shrinking also causes the MSE to converge faster for large shown in Figure. lb.
The reparameterization log(u) log(v) We consider novel reparameterization. In the main text, have that the regularization changed the type of bias from La to L1. We now consider reparameterization with explicit regularization that leads to the opposite type of bias change. The reparameterization is g(w) = log(u) log(v). The regularization found in Corollary 3.12 is (w)) = Σn log(u.) + log(v.). Then for > 1 we can apply Theorem 3.6.
now give the resulting time-dependent Legendre function. The time-dependent Legendre function is
Ra (a(x)
4 (n2 2a) log 2c- + + (v8,= 2a) log (62ts + +1) Va < 2 min fug v2.
The global minimum is centered at V R. 0 and is given by log (V - 2a log (V when a changes, illustrating the positional bias. Moreover, to illustrate the type change, consider the balanced initialization Uo = vo = 31, the Legendre function is then given by
2a Thus shift occurs
1 (82-20) 2a) log (2 cosh(xx))
Rx( 4
23

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
1600
k= 1
k ==1
Reconstruction MSE
k= 2
Nuclear norm
1400
k=
10
k= 3
k 3
k= 4
1200
k = 4
k 5
k = 5
k = 6
k = 7
1000
k 7
10 10-
25
50
75
100
0
25
50 Iterations
75
100
Iterations
(a) Nuclear norm of sparse code w.
(b) Reconstruction MSE of i.
Figure Results for sparse coding reparameterisation g(w) 22
which resembles the log-cosh loss function with vertical rescaling. The rescaling changes the type of bias from L1 + L2. The type here L2 close to the origin and L1 further away from zero. Due to the scaling, it becomes closer and closer to Lg. This is illustrated Figure. 12. Furthermore, we will show in experiments that the type change is crucial for generalization.
6
4
2
$
a
0
2
-1 -2
X
Figure 12, From L1 to L2 implicit bias, with = So agds.
In this context, we reparameterize the sparse code as g(w) = log(u) - log(v) E 1Rt and replace the regularization as discussed. We initialize the parameters as u사 = 1/((11 + ))) and 0g = 1/(3(1 + e²)), where 3 = and r = 0.1. Note that the initialization is different for stability reasons. We explore various values for a E (0.0001, 0.001, 0.01, 0.0, 1.0). During the training process, we track two key metrics: the reconstruction Mean Squared Error (MSE) and the nuclear norm of the sparse code, defined g(w) log u log v. The results illustrated in Figure. We observe that higher regularization leads faster increase in the nuclear norm, which confirms the movement L2 regularization. This leads to construction error.
24

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
nuclear norm of sparse code (log scale
reconstruction MSE (lag scale)
1=0.1
A
A=1
0.00
= 0.0001
iterations
iterations
(a) nuclear norm of sparse code w
(b) Reconstruction MSE of x
Figure 13. Results sparse coding reparameterisation g(w) = log u log
F. Hyperparameters and additional figures on quadratic reparameterizations
We present the experimental details and additional figures such as validation the evolution of the time-dependent Legendre functions corresponding to m @ w in Figures. 2a and 2b In the case of attention, we used the optimizer AdamW with learning rate le -4 4 and a constant learning rate. We start training from a pretrained tiny-ViT on ImageNet. Moreover, the CIFAR 10 experiment is run over 3 seeds. Finally, for LoRA we use SGD with momentum (0.9), constant learning rate 2e - 4, LoRA rank 8, alpha 8 and no drop-out. The GPT-2 experiment is run over seeds.
and accuracy. Morcover, present
The validation accuracy is given in Table for the attention experiment. The validation loss is given in Table 5 for the LoRA experiment. In all cases turning off the weight decay leads to improved validation score (accuracy error). In Table 3 we provide an ablation for various turn off points of the weight decay for strength 0.2. Turning off leads to better generalization at the intersection point or even before intersection.
Table 3. Validation accuracy and ratio at different weight decay turn-off points.
WD Off
Intersect
Val Acc (WD)
Val Acc (Off)
Norm Ratio
50
104
70.7
72.4
6.9
100
195
70.2
72.6
6.5
150
270
71.1
73.4
6.4
200
None
70.3 (6.3)
72.5 (6.1)
Furthermore, we provide training of a tiny-ViT from scratch on CIFAR10 with varying weight decay in Figure. 14. The learning rate is le 3 and we use cosine warmup. Observe that higher weight decay is necessary to keep the ratio down at the end of the training. This can improve the validation error significantly as observed for wd = 0.1.
Table Validation accuracy for tiny-ViT experiment (attention).
Dataset
wd = 0.2
wd 0.2, turn-off wd 0.1
udd 0.02
wdd = 0.02, turn-off
wd 0.01
ImageNet
71.08
73.66
71.76
74.576
75.24
75.05
CIFAR10
52.38(+0016)
56. (0..99
54.95(++0.3)
37.05(+0.27)
57.78(+0.32)
57.33(+0.44)
25

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
Table Validation loss for LoRA experiment Shakespeare dataset.
Architecture GPT2-xl
wd. 1.0
wd = 1.0, turn-off
wd = 0.5
wd = 0.2
wd 0.2, turn-off
wd = 0.1
2.99
2.96
2.97
2.96
2.95
2.96 3.42(#0.01)
GPT2
3.44(+:0.00)
3.42(+0.000
3.43(++0.00)
3.43(++.01)
3.41(+0.00)
Table Quadratic reparameterization.
Matrix sensing
UUT
U E Rax
Attention
Softmax( (ofimar(QK'J Wo + AB
Q K, V E R"
LoRA
A E R B R
Wo R where r <<
wd 0.5
Kuclear porm trebenius norm
4.0
2.2
0.05
wd 0.01
3.0
0.001
0005
2.5
1.6
0001
1.5
1.2
20
40
60
80
100
0
20
40
60
80
100
Epochs
Epochs
(a) Validation error.
(b) Ratio of the nuelear norm and the Frobenius norm.
Figure Varying the weight decay parameter on CIFARIO. Higher weight decay leads to lower ratio can also lead lower validation error..
G. Learning rate schedule
We further study the effect of the learning rate scheduler. Specifically, we run pre-trained ViT-tiny on ImageNet classification fine-tuning task. We set the learning rate to le - 4 with AdamW optimisers. We also vary the weight decay in the range [0.001, 0.003, 0.005, 0.007, 0.01]. Moreover, for each of the settings, we train two comparison experiments, one without a learning rate scheduler, and one with the popular Annealing WarmRestarts.
The results are shown in 15g Furthermore, results with SGD optimizer are included in Figure. We observe in both figures that the validation accuracy increases for the decaying schedule in comparison to the constant schedule. Moreover, we again observe decaying ratio, for stronger weight decay the ratío decreases more. Moreover, decaying the learning rate has a similar effect as turning off the weight decay the implicit and explicit regularization. Note that the Adam optimizer can have additional effects that also contribute to changing the ratio. Furthermore, regardless of the learning rate schedule, the ratio is decreasing indicating a modulation from Frobenius norm towards nuclear norm minimization.
26

---

Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?
m 0.01, acteduler
8.5
0.02. schozuer
0.007. scheller
75
Nudeea norm Frobenlus norm
We 0.027. no schodaler
005 schelder
wi 905. schedaler
003 scheluer
8.3
0.001 othediter
803 sthedaler
65
0.001
82
Val.
60
0.01
9.01.
scheduler
scheduler
8.1
0 5.007
0 2070 sr scneddder
0.003
schedieer
9 9.005. scheduler
003
schediler
50
wet 0.000
7.9
wet 0.000.
9.501,
schodller
100
150
200
250
300
50
100
150 Epoch
200
250
300
Epoch
(a) Average ratio |K Q) / K Q) frob
Validation accuracy
Figure 15. Results for ViT-tiny fine-tuning task with Adam W optimiser on ImageNet.
70
Nuclear norm Frobenius norm
60
50
Ja acc
40
wd 0.01, scheduler
wd 0.01, scheduler
0.007,
30
0.007, scheduler
4
0.007, scheduler
wd 0.007. scheduler
0.005, scheduler
scheduler
20
wd 0.005, ne scheduler
wd 0.003, no scheduler
0.003, scheduler
0.003,
10
0.003, scheduler
wd 0.001, no scheduler
0.001,
0.001,
wd 0.001, scheduler
50
100
150 Epoch
200
250
300
0
50
100
150
200
250
300
Epach
(a) Average ratio k Queee/ KT (TQ Frob
(b) Validation accuracy
Figure Results for ViT-tiny fine-tuning task with SGD optimiser on ImageNet.
27


Reviews:
[{"review_id": "JCK0dh27Ry", "reviewer": "Reviewer_Eyxr", "summary": "The paper investigates how external regularization influences the implicit bias of gradient flow. By leveraging the equivalence between parameterized gradient flow and mirror flow, the study provides a detailed analysis of how external regularization alters this mirror flow. Within the framework proposed by Li et al. (2023), when the parameterizing function $g$ and the regularization function $h$ are commuting re-parameterizations, the gradient flow on the regularized loss is shown to be equivalent to a mirror flow with a time-varying Bregman function. This Bregman function depends on regularization through the parameter $a$, which is defined as the integral of the regularization over time. The equivalence is used to study the convergence of regularized gradient flow and across different parameterizations or", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the reviewer’s time and effort in providing valuable comments on our manuscript and appreciate the acknowledgement of the theoretical contributions. Below, we address the raised questions and concerns, but would be happy to extend the discussion on request. \n\n**How the 3 effects are distinct**\n\nWe provide some examples that highlight how the 3 effects are distinct from each other and also can occur independently from each other:\n1. The range shrinking means that the actual range of $\\nabla R_a$ changes during training; this does not occur for quadratic reparameterizations, for example.\n2. The positional bias is distinct. When we initialize a quadratic parameterization at zero, the positional bias stays zero. So it does not change, while the type of bias changes still from L2 to L1.\n3. The type of bias can be different in all these scenarios and even changes dynamically in most of our examples.\n\t\n**Commuting condition and eigenbasis**\n\nWe will restate the commuting condition such that it is clear that indeed $(g,h)$ is commuting in Theorem 3.2. Moreover, we will mention the connection between an appropriate eigenbasis choice and the commuting condition explicitly after the quadratic parameterization result.\n\n**Figure 2,  L1 setting**\n\nThe training dynamics for the linear parameterization are different from that of the quadratic reparameterization. Therefore, it is hard or impossible to match the same performance in the experiment.\nThe main takeaway is that the type of implicit bias of the linear parameterization has not changed. In general, what would occur when the L1 regularization is turned off is that the training dynamics go to the L2 interpolation solution. To alleviate any concern, we have run an additional experiment for the L1 setting with higher regularization ($0.2$) and turned it off at the same time, this also leads to similar performance, i.e. it goes to the L2 interpolator with a reconstruction error of $0.32$. Therefore, the amount of regularization does not change the type of implicit bias. We will include this additional experiment in Appendix C of the revised manuscript.\n\n\n**Figure 3, ratio**\n\nWe do not make claims about the desirability of these results; this would depend on the specific application. A lower rank (smaller ratio) might be desirable in cases where efficiency or interpretability is a priority, for example. For the practitioner it is relevant to understand how regularization impacts the rank and thus the solution. This insight is the purpose of our theory.\nThe main message of this Figure 3 is that we can obtain similar ratios by turning off weight decay. Appendix F furthermore demonstrates that this can achieve better generalization. Specifically, at similar ratios, the validation accuracy of a ViT on ImageNet can be improved by more than 1%.\nThis illustrates the utility of controllable implicit bias in practice, but the main purpose of our experiments is to verify our theory. We still believe our theory could inspire better algorithm design in future, as mentioned in the discussion.\n\n**Different schedules for matrix sensing**\n\nConsider the family of schedules with constant regularization strength $\\alpha_i$ up to specific time T_i such that $\\int_0^{T_i} \\alpha_i ds = T_i \\alpha_i = C$ for $C >0$ a constant. We choose $\\alpha_i = [0.02, 0.2, 2, 20]$.  In addition, we consider a linear and cosine decay schedule for the regularization with the same total strength (i.e. same integral), but the regularization is switched off after half of the training time to ensure convergence. To compare with the effect of turning off (t-o) the regularization, we include the constant schedule with regularization strength $\\alpha=0.01$. Note that this schedule is also reported in Appendix C. We observe that all schedules with decay or turn-off (t-o) converge to a solution with the same nuclear norm of the ground truth, confirming Theorem 3.6, while the constant schedule does not reach the ground truth. We would be happy to include this additional experiment in Appendix C.\n\n| Schedule                     | Nuclear norm | Train loss | Rec error | Time to 1e-7 train loss |\n|------------------------------|-------------|------------|-----------|--------------------------|\n| Constant 0.01, no t-o   | 0.93        | 7.2e-4     | 3.9e-2    | -                        |\n| Linear decay                 | 1.00        | 1.8e-8     | 2.3e-4    | 661                      |\n| Cosine decay                 | 1.00        | 1.7e-8     | 2.1e-4    | 624                      |\n| Constant 0.02, t-o      | 1.00        | 1.1e-8     | 1.7e-4    | 716                      |\n| Constant 0.2, t-o       | 1.00        | 2.7e-10    | 2.7e-5    | 209                      |\n| Constant 2, t-o         | 1.00        | 2.1e-10    | 2.4e-5    | 209                      |\n| Constant 20, t-o        | 1.00        | 7.9e-13    | 1.4e-6    | 239                      |", "claims_and_evidence": "a) The paper claims of different effects of regularization which it names as positional bias, type of bias and shrinking range which they claim as the main contribution of the paper. However, it feels that the the three effects are essentially the same, the range shrinking of $a$ changes the bias of $L_2$ to $L_1$ in quadratic parameterization.", "methods_and_evaluation_criteria": "N/A as the paper is majorly theoretical in nature.", "theoretical_claims": "a) In Theorem 3.2, the authors need to clearly specify that $(g,h) : M \\to R^{n+1}$ should be a commuting parameterizations. In the current form, it reads as $g$, $h$ are commuting parameterization among themselves and not necessarliy $[ \\nabla g_i, \\nabla h]$ = 0", "experimental_designs_or_analyses": "a) In the experiment corresponding to Figure 2, it is unclear how the regularization coefficient for the $ell_1$ is chosen, for appropriate choice I expect it to match the loss with regularization before turning it off. Also it is unclear comparison when both the regularization are turned off at the same iteration. \n\nb) It is hard to interpret the experiment corresponding to Figure 3, if the higher nuclear norm to Frobenious norm ratio is desired, should the ideal method be to just choose a lower weight decay constant ?", "relation_to_broader_scientific_literature": "The equivalence between mirror flow and reparameterized gradient flow is extensively used to argue about the implicit bias of gradient methods with various hyperparameters  Woodworth et al., 2020; Pesme et. al. 2021. The work of Li et.al. 2022 lays the general framework when such a equivalence is possible and the current work extends this framework for regularized loss highlighting the differences.", "essential_references_not_discussed": "To the best of my knowledge, the essential references have been sufficiently discussed.", "other_strengths_and_weaknesses": "Strengths : \n- Theorem 3.2 is a nice addition to the literature of implicit bias and mirror flow showing how the weight decay modifies this revealing that the integral of the coeffecient across time is a key determining factor. \n\nWeakness : \n-  The framework is applied to different problems like matrix factorization and attention framework, however, the analysis holds when the matrices commuting. Hence, under the appropariate choice of eigenbasis, it is equivalent to the quadratic parameterization, it is recommended that the authors explicitly mention this.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "a) Theorem 3.6 uncovers an intriguing property: the integral of the regularization parameter is the sole determining factor. It would be nice to make this a bit concrete with experiments across some architectures, for example with various schedule which keep the integral to a constant value.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t4lL1ZJb8D", "reviewer": "Reviewer_ZErA", "summary": "Motivated by the fact that the inductive bias of a trained neural network depends on implicit biases of the training algorithm and explicit regularization such as weight decay, this paper studies how the two interact. Appealing to previous works, they adapt the mirror flow framework for objectives with explicit regularization and use their new framework to demonstrate how explicit regularization can influence implicit bias. The key insight is that the implicit bias of the optimization algorithm can be adapted and controlled during the optimization procedure via dynamic explicit regularization. The paper demonstrates how the insights are manifest in experiments on sparse coding, matrix sensing, attention and LoRA fine-tuning illustrating potential applications.\n\n## Update after rebuttal\nAfter reading the reviews of others and seeing the rebuttals of the authors I have a better understanding of the work and its significance. I have appropriately moved my score up to a 3 (weak accept).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the reviewer’s time and effort in providing valuable comments on our manuscript and appreciate the acknowledgement of the work's relevance and novelty. We would be happy to discuss any open questions or concerns, if there remain any.\n\n**Background knowledge**\n\nWe appreciate the reviewer’s observation regarding the significant background knowledge needed to fully understand the details. Given the initial page limit, we focused on presenting the main storyline. \n\nIn the current manuscript, we elaborate on the implications with concrete examples in Section 4. \nMoreover, the main theorems are accompanied by detailed descriptions of the key proof steps, along with characterizations of the regularization and a geometric interpretation. These findings and concepts have been formally defined, highlighting important details of the analysis.\nAs we would be allowed to use an additional page for the main manuscript upon acceptance, we would be happy to make adjustments. To this end, we will include more definitions from the appendix into the main manuscript like:\n1. The definitions of a regular and commuting parameterization in Section 3 (Introduction part).\n1. The definition of the Legendre function in Section 3 (Introduction part).\n1. The definition of the Bregman function in Section 3.2.\n\nWith these additions, we believe the main storyline would be accessible to a broader audience, with the key idea being that explicit regularization changes the implicit bias. \n\n\n**The 3 effects**\n\nWe believe that the introduction provides a precise definition of the three effects without requiring the explicit introduction of the parameterized Legendre function. In Section 4, we present concrete theoretical examples of all three effects. However, we are happy to further clarify these effects by including the definition of the parameterized Legendre function at the beginning of Section 4:\n1. Type of bias: The shape of $R_a$ changes with $a$.\n1. Positional bias: The global minima of $R_a$ changes with $a$.\n1. Range shrinking: The range of $\\nabla R_a$ can shrink due to a specific choice of $a$. \n\n**Experimental implications**\n\nThe experiments are there to verify the theory and to show that the principles hold in more practical settings. A practical implication is the potential utility of dynamic regularization schedules. \nFor instance, in the case of quadratic reparameterizations, to get the best generalization accuracy for a desired sparsity ratio, it is beneficial to regularize more during the first half of training and then turn off weight decay in the second half. Moreover, our work provides a stepping stone for further research to derive better regularization schedules that take the implicit bias into account, as mentioned in the discussion.\n\n**Miscellaneous**\n\nWe agree that it is more clear to use $R_{a_t}$ than $R_a$. The intent was to make a more direct reference to the previous definition of the parameterized Legendre function.  \nIn addition, $x_*$ is a solution of the objective $f$ as the regularization is turned off eventually. In other words, after turning off the regularization the iterates move to a solution of the objective $f$.\nWe revise our manuscript accordingly.", "claims_and_evidence": "I am unsure whether the claims are well supported. In particular, the claims about 3 distinct effects of explicit regularization mentioned in the introduction. Moreover, the statement of the claims are unclear and not precise.", "methods_and_evaluation_criteria": "The paper is mainly a theoretical work but the evaluation criteria used in the experiments are appropriate. Although the applications are unclear.", "theoretical_claims": "The paper relies heavily on the setup and works of previous paper connecting the implicit bias of gradient descent to mirror flow. Largely building off of [1]. Without this background it is difficult to understand the setup or the motivation behind it. Moreover, too much of the technical details are left to the Appendix making it hard to read the paper. In light of this while I believe the theoretical claims are correct I am not familiar enough with this line of literature to verify their correctness.\n\n\n[1] https://arxiv.org/abs/2207.04036", "experimental_designs_or_analyses": "The experiments are sound although their implications are not clear.", "relation_to_broader_scientific_literature": "I am familiar with the idea of implicit bias and as far as I know it is typically studied independent of any explicit regularizers. Given that neural networks are often trained with explicit regularization, such as weight decay, I believe this paper brings insight to the important question of how explicit and implicit regularization interact and would be useful to the broader machine learning community.", "essential_references_not_discussed": "I am not familiar with this line of research to suggest any essential references not discussed.", "other_strengths_and_weaknesses": "### Strengths\nThe topic being studied is very interesting, novel, relevant and worthy of study. Extending the previous mirror flow framework to account for explicit regularization could also be of independent interest.\n\n### Weaknesses\nThe biggest issue with the paper is presentation. I am not familiar with past work on mirror flow and this paper requires a lot of background knowledge about previous works connecting implicit bias of gradient descent to mirror flow. This makes it very difficult to understand the results and interpret their significance. The readability of the paper is also effected as many of the core details are left to the appendix and never formally defined. Moreover, the three different biases discussed in the introduction are also stated quite informally (except for the first) making it hard to really understand what they mean.", "other_comments_or_suggestions": "### Minor\n- Line 193, $R_a$ or $R_{a_t}$?", "questions_for_authors": "1. By doing this dynamic explicit regularization, the objective function is changing for each $t$. Therefore, when the algorithm terminates, for which objective is $x_{*}$ a solution?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4gVP6vrqb5", "reviewer": "Reviewer_PG36", "summary": "The authors propose to analyze the effect of combining implicit and explicit regularization on training dynamics using the mirror flow framework. While their framework is more general, they specifically examine the impact of weight decay when it is turned off at a particular training step. To investigate this, the authors conduct several experiments to assess the influence of weight decay on both the sparsity of neural networks and their performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s time and valuable feedback. We’re happy to address any further questions or concerns.\n\n**Turning-off at different points**\n\nWe are happy to provide an ablation on turning off the regularization at different points for the vision transformers:\nWe disable weight decay (WD) at different epochs during fine-tuning of Tiny-ViT on ImageNet. We train for 300 epochs with AdamW (lr=1e-4, WD=0.2) and disable WD at epochs [50, 100, 150, 200]. We report the intersection epoch of nuclear norm/Frobenius norm ratio and validation accuracy with/without WD off. If no intersection occurs, we report final validation accuracy and ratio. Results show better performance for similar ratios, supporting our theory. We will include this additional experiment in the revised manuscript.\n\n| WD Off | Intersect | Val Acc (WD) | Val Acc (Off) | Norm Ratio |\n|--------|----------|--------------|---------------|------------|\n| 50     | 104      | 70.7%        | 72.4%         | 6.9        |\n| 100    | 195      | 70.2%        | 72.6%         | 6.5        |\n| 150    | 270      | 71.1%        | 73.4%         | 6.4        |\n| 200    | None     | 70.3% (6.3)  | 72.5% (6.1)   | -          |\n\n**Theoretical bounds on $T$ and $\\alpha$**\n\nFor underdetermined linear regression (Thm 3.6), we search for the schedule with the fastest convergence given a fixed desirable $a_T = \\int_0^T \\alpha_s ds < \\infty$. This is equivalent to regularizing as much as possible and turning off regularization quickly. Yet, it is known that the saddle point near the origin slows down the  training dynamics. Also, this is impacted by a finite learning rate inducing a discretization error. Table for Reviewer Eyxr identifies a solution to the expected trade-off.\nYet, the optimal $T$ depends on intricacies of the loss landscape and the training dynamics. For example, for the goal of sparsifying a neural network, gradually moving the implicit bias from $L_2$ to $L_1$ works well [1]. Intuitively, we expect to train with regularization until we hit a loss basin to switch it off soon thereafter. A theoretical bound presents a great challenge that is beyond the scope of this paper with general framework character.\n\n[1] Jacobs, T. and Burkholz, R. Mask in the mirror: Implicit sparsification. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=U47ymTS3ut.\n\n\n\n**Details of main proofs**\n\n$R(x,y)$ is unrelated to the loss function $L$. At the end of Theorem A.7, it is defined by the reparameterization, i.e. $(g,h)$, and the initialization. The second derivative is positive. This follows from the definition of a Legendre function (Definition A.1), which is strictly convex. We will refer to this explicitly.\nWe can express the Hessian of $R$ in terms of block matrices that are derivatives with respect to $x$ and $y$ and then apply the block matrix inversion lemma (see  Proposition 2.8.7 in [1]). The first diagonal block matches exactly the inverse of the constructed Hessian for the time-dependent Legendre function. As the inverse of a positive definite matrix is positive definite, it follows from the definition of positive definiteness that also this diagonal block matrix is positive definite. To see this more precisely, denote the inverse of the Hessian matrix with $A$. $A$ is positive definite if and only if for all $z = (x,y) \\in \\mathbb{R}^{n+1}$, $z^T A z > 0$. Now restricting this to particular $z = (x,0)$ with $x \\in \\mathbb{R}^n$ yields that the diagonal block matrix that we are interested in is also positive definite.\nWe suppressed the dependency of $\\nabla R_{a_t}$ on $x_t$. We agree that it would be more clear to use $\\nabla R_{a_t}(x_t)$ instead. In addition, we will separately explain the application of the time-dependent mirror flow relation and the contraction property to highlight how the contraction property is used.\nWe will include these extra explanations in the revised manuscript. Moreover, we will provide the block matrix inversion calculation according to Proposition 2.8.7.\n\n[1] Bernstein, D. S. (2005). Matrix Mathematics: Theory, Facts, and Formulas with Application to Linear Systems Theory. http://engineering.nyu.edu/mechatronics/Control_Lab/ToPrint.pdf\n\n**Experiment interpretation**\n\nWe agree that the improvement for the LORA experiment is lower, although significant. However, for the vision transformers, the increase in validation accuracy is substantial for the same ratio. The additional experiments above support this claim. Furthermore, we would like to emphasize that our experiment is not designed to maximize performance but to confirm the prediction of the theory. The desirability of a particular ratio is problem-dependent. For example, a lower rank (smaller ratio) might be desirable in cases where efficiency or interpretability is a priority. Moreover, the framework could be used to inspire and help design new regularization schedules that are tuned for better performance.", "claims_and_evidence": "The authors suggest that their theoretical framework provides insights for designing appropriate regularization strategies, and they conduct experiments to support this claim. The author take the special case of turn-off weight decay to illustrate the application of their framework and let to future work do design temporal weight decay. In the experimental section, it would be beneficial to include an ablation study examining the impact of the time step at which weight decay is turned off. Additionally, deriving theoretical boundaries for (T,weight decay) and comparing them with the experimental results would strengthen the analysis.", "methods_and_evaluation_criteria": "To validate their theoretical findings, the authors investigate a specific case where weight decay is switched off at a particular training step T. They analyze the impact on accuracy and the ratio of the Nuclear norm to the Frobenius norm across various architectures (e.g., LoRA, attention mechanisms) and datasets, including ImageNet and the Shakespeare dataset.", "theoretical_claims": "I have reviewed the proofs of Theorems B.1 and B.2 and have a few questions:\n\n1. **B.1 – L785:** I do not fully understand the justification for $\\partial_y\\partial_y R(x,y) > 0$. Given that the authors consider the loss function $ L(x,y) = f(x) + \\alpha y $, the second derivative should be $\\partial_y\\partial_y R(x,y) = 0$. Could you clarify where my reasoning is incorrect?  \n\n2. **B.1 – L804:** I did not fully grasp your demonstration regarding why $\\nabla^2_x R$ is positive definite. Could you provide additional explanation?  \n\n3. **B.2 – L843 to L846:** You state that  $d \\nabla_x R^T_{a_t}(x^* - x_t) = - d \\nabla_x f(x_t)^T (x^* - x_t).$  However, in my understanding, it should be  $d \\nabla_x R^T_{a_t}(x^* - x_t) = - \\nabla_x f (x^* - x_t).$ Consequently, I do not see why we can conclude that $\\nabla_x f (x^* - x_t) \\leq \\nabla_x f (x_t)^T (x^* - x_t).$", "experimental_designs_or_analyses": "Regarding the **experimental section**, I appreciate that the authors demonstrate, in the case of linear hyperparameterization, that turning off weight decay improves the error in this particular setting, validating the theoretical results (Figure 2). However, I have some concerns about the experiments on real datasets in terms of accuracy and sparsity metrics:\n\n1. **Tables 2 and 3 in the appendix:** The performance differences between the best turn-off setup and the best weight decay configuration (which is slightly smaller) do not appear to be very significant. In my view, it is difficult to draw a strong experimental conclusion because an additional hyperparameter (the step at which weight decay is turned off) needs to be tuned, and similar results can be achieved by selecting an appropriate weight decay value.  \n\n2. **Figure 3:** While the experiment illustrates the impact of the turn-off mechanism, the advantage of turning off weight decay in terms of the nuclear norm to Frobenius norm ratio is not entirely clear.  \n\nCould you clarify whether I have misinterpreted the analysis of these experiments?", "relation_to_broader_scientific_literature": "I have no strong opinion on this matter, as I am not particularly specialized in this litterature.", "essential_references_not_discussed": "I have no strong opinion on this matter, as I am not particularly specialized in this litterature.", "other_strengths_and_weaknesses": "Strengths: The paper is well-written, clear, and easy to understand.\n\nWeaknesses: The proofs are less accessible at times due to brief explanations between certain steps, making it harder to follow the reasoning.", "other_comments_or_suggestions": "None", "questions_for_authors": "I would appreciate responses to my theoretical and experimental analysis questions outlined in the \"Theoretical Claims\" and \"Experimental Designs Or Analyses\" sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a novel theoretical analysis of how external regularization affects the implicit bias of gradient flow, by extending the mirror flow framework to account for time-varying Bregman functions induced by regularization. The reviewers appreciated the originality and relevance of the topic, as well as the insightful contribution of Theorem 3.2, which reveals the role of WD through integrated regularization coefficient. While the presentation could be improved, particularly for readers unfamiliar with prior mirror flow work, the technical depth and potential impact on our understanding of implicit bias are significant. I recommend acceptance.
Decision: Accept (poster)