Lorenza: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM
Authors: Yehonathan Refael
Iftach Arbel
Ofir Lindenbaum
Tom Tirer
Venue: Accepted by TMLR ()

Abstract: Modern applications often require fine-tuning large language models (LLMs) within strict memory and computational limits, but existing memory-efficient optimizers tend to compromise robustness and generalization. To tackle this, we introduce Lorenza, a low-memory optimizer based on Sharpness-Aware Minimization (SAM). Lorenza employs a stochastic zeroth-order estimator to approximate ascent directions, reducing the computational complexity of SAM while, as we prove, maintaining its convergence guarantees. Additionally, by applying randomized singular value decomposition, Lorenza performs efficient low-rank gradient updates, achieving memory efficiency similar to traditional methods. Our theoretical analysis and experiments demonstrate that Lorenza improves robustness and generalization, particularly in challenging language tasks. Furthermore, we present Lorenza+, which enhances Lorenza by incorporating the discarded orthogonal gradient component, resulting in additional performance gains without requiring extra memory or computational overhead.

Published in Transactions on Machine Learning Research (09/2025)
Lorenza: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM
Yehonathan Refael
refaelhalim@tau.cc..
Faculty of Engineering
Tel Aviv University
Tel Aviv 6997801, Israel.
Iftach Arbel
i.arbel84 @ggaail com
Independent Reseorcher
Ofir Lindenbaum
ofir. dirdeeeeennna @biu.ac.il
Facully of Engineering
Bar-Ilan University
Romat- Gan 5290002, Israel.
Tom Tirer
firen, trer.tomern Sbiu.
Faculty of Engineering
Bar-Ban University
Ramal- Canall-Gan 5290002, Israel.
Reviewed on OpenReview: https: openreview. net/ forum? id= YyA51ekcoo
Abstract
Modern applications often require fine-tuning large language modcls (LLMs) within strict memory and compulational limits, but existing memory-ellicient optimizers tend to com- promise robustness and generalization. To tackle this, we introduce Lorenza, low-memory optimizer based on Sharpness- Minimization (SAM). Lorenza employs a stochastic zeroth-order estimator to approximate ascent directions, reducing the computational com- plexity of SAM while, as we prove, maintaining its convergence guarantees. Additionally, by applying randomized singular value decomposition, Lorenza performs efficient low-rank gra- dient updates, achieving memory efficiency similar to traditional methods. Our theoretical analysis and experiments demonstrate that Lorenza improves robustness and generaliza- tion, particularly in challenging language tasks. Furthermore, we present Lorenza+, which enhances Lorenza by incorporating the discarded orthogonal gradient component, resulting in additional performance gains without requiring extra memory or computational overhead.
1 Introduction
Large language models (LLMs) have garnered considerable attention due to their remarkable ability to per- form various tasks, including engaging in dialogue and generating text. Their performance can be improved through supervisod fine-tuning and additional pre-training across different tasks and domains. However, training these models presents substantial computational power and memory challenges. This difficulty arises because the process of updating gradients requires storing billions of trainable parameters along with the optimizer's state (which includes gradients and moments). For instance, in the Adam optimizer (Kingma & Ba 2017), the storage requirements for gradients and the estimated first and second moments can triple the overall size of the model (Xu et al. 2024 Brown et al. 2022 Kim et. al. 2023
Rescarchers have developed various optimization techniques to reduce memory usage during training, tackling the challenges associated with LLM fino-tuning. One key research topic that has emerged is Parameter- Efficient. Fine-Tuning (PEFT) Han al. 2024h, which enables the adaptation of pre-trained language

---

Published in Transactions on Machine Learning Research (09/2025)
models to different tasks without the need to fine-tune all model parameters. A prominent method for PEFT is the Low-Rank Adaptation (ToRA) algorithm, introduced by (Hu et al. 2021 LoRA reparameterizes a weight matrix W E Rmx= into W = Wo + BA, where Wo is a frozen full-rank matrix, and B E Rmxr and A E R are low-rank adapters. Since r < mminmm, n), the low-rank adapters A and B require fewer trainable parameters, reducing memory usage. Several LoRA variants have been proposed, for example
CChnn et al. 2023 Xu al.
2023: Wang et al. 2023
Table 1: Comparison between Lorenza, GaLore Zhao et al. 2024 LoRA (Hu ct al. 2021 SAM
Forct
et al. 2021 and AdaSAM (Sun el al. 2023). Assume R (n 2 m), constant low-rank r.
Lorenza
GaLore
LoRA (using Adam) nm + Dr + mr
SAM
AdaSAM
Weights Optimizer states
nm
nm
nm
nu
10r
2me
2mr
2nr 2mr
vin
4ru
Computation
O O(mmr- + mnr/T)
mnr
m m/n/T)
(mnr)
(mnr)
O (mnr)
Fine-Tuning
Pre-Training
Multi-Subspace
X
Num. of Backprop per step
1
2
2
Sharpness- Aware
V
Despite the reduction in memory demands, low-rank weight adapters have drawbacks. One notable weakness of LoRA-type methods is their potential to fall short in accuracy for more challenging fine-tuning tasks compared to Full Fine- Tannin (FFT) (Meng et al. 2024). These issues may stem from the fact that optimal weight matricos are not inherently low-rank or from changes in gradient training dynamics introduced by the reparameterization Another limitation, for instance, is that these methods are primarily designed for adapting pretrained LLMs through fine-tuning, rather than for use during the pretraining phase.
In an effort to mitigate these gaps, recently, efficient-memont optimizers have emerged. For example, GaLore (Zhao et al. 2024) can perform both pretraining and fine-tuning LLMs by updating the model weights within an adaptive low-rank subspace. This method significantly reduces the number of fine-tuned parameters, offering several key advantages. First, it eliminates the need for additional adapters alongside the pre-trained model. Second, it removes the requirement to store all gradient parameters during training. Third, it reduces memory usage by bypassing the need to retain the full dimension original optimizer states. Demonstrating its cffectiveness, GaLore succossfully pre-trained an LLM with 7 billion parameters a consumer-level GPU, using just 24 GB of memory. The method attracted considerable atlention, leading to various new oplimizers
aimed at further reducing memory usage (Refael et al. 2025 Liao et al. 2024 Zhang et al. 2024b Das.
2024 Huang et al. 2024).
Still, a notable limitation of menior-effiiient optimizers their reduced generalization in out-of-distributtion, domain shift, and zero-shot settings. This performance disparity tends to emerge in scenarios involving complex target tasks, such as mathematical reasoning coding. This raiscs an open question: Can we design optimizer that simply retains the low memory footprint of gradient descent while achieving the generalization performance of FFT-based methods?
To address this question, we introduce Lorenza, a computationally efficient, sharpness- optimization method that leverages adaptive low-rank gradient updates and memory-efficien zeroth-order sharpness mini- mization to enhance generalization, specified in Table Unlike existing sharpness-awan fine-tuning methods that require costly double backpropagation (Foret et al. 2021 Sun et al. 2023), Lorenza eliminates this over- head through a backpropagatton free perturbation (BPFP) scheme, significantly reducing computational and memory complexity. For example, for the OPT-13B model, the backpropagation consumes approximately x6 more memory during fine-tuning than using the proposed BPFP and approximately x4 more in cal- culation time. Furthermore, Lorenza employs a dynamic low-rank subspace selection mechanism, ensuring optimization updates remain efficient while maintaining the benefits of full-rank tuning. From a theoretical perspective, we establish a convergence guarantce for Lorenza, proving that it efficiently finds flat minima that promote better generalization while maintaining computational efficiency and significantly reducing memory overhead.
2

---

Published in Transactions on Machine Learning Research (09/2025)
Finally, empirically demonstrate that Lorenza outperforms state-of-the-art methods in pre-training and fine-tuning of LLMss. Notably, Lorenza exhibits robust adaptation across diverse datasets and challenging tasks, highlighting its potential as a scalable, efficient alternative for training and fine-tuning LLMs under resource constraints.
2 Related work
The gencralization ability of neural networks has been shown to correlate with the flatness of the minima HHcchreitr & Schmidluber 1997 Keskar et al., 2017 Sun et al. 2023 Si & Yun 2023 Yue al. 2024) In regions around flat minima in the loss landscape, as illustrated in Figure 1 small parameter changes lead to minimal loss variation, reducing the model's sensitivity to noise and perturbations. This robustness has been shown to enhance the model's ability to generalize to unseen data, compared to standard optimization methods that may converge to sharp minima.
W.
W.
W
An illustration showing how the flatness of different minima can impact test loss. Specifically, W1 and Wa, are located in sharp regions that have a high generalization error, while W2, found in a flatter region, exhibits lower generalization error (Dong et al. 2024).
Figure 1:
Sharpness-Award Minimization. Sharpness-Awan Minimization (SAM) Foret al. 2021 aims solve the min-max optimization problem: minw i fs(w +), where fs(w) denotes the empirical loss. Note that the objective value per is equal to the highest value of the loss within neighborhood of defined as ball of radius & centered at Therefore, this problem promotes flat minimizers, where small perturbations in the weights (even the "worst" e) do not increase the empirical loss significantly.
To simplify the problem, SAM approximates the solution to the inner maximization using a first- order Taylor expansion around w. This leads to the following approximation of the perturbation, + 6 arg max (sw PTV (a Substituting this back into the outer minimization reformu-
fwff(()
lates the objective as: minw fs + P V. (w))ll In practice, given mini-batch B, SAM extends the standard stochastic gradient descent (SGD) (Battash et al. 2024) update to the following two-step process: V Vw/a(w.)
(1) for the current weights w,, compute the adversarial perturbation: E; = PIV..f/a(w.illl (2) evaluate the gradient of the perturbed weights W. + G and use it to update W,, namely W++1 W, - GGSAM where GSAM Vw/n(w, +e), and 1j the learning rate. This procedure ensures that SAM balances the trade-off between minimizing the empirical loss and achieving flat minimum, improving generalization performance.
Variants of SAM. AdaSAM (Snn et al. 2023 enhances SAM by integrating adaptive estimates of the first and second moments of the gradients to further improve optimization efficiency and generalization in deep neural networks, similar to the popular Adam optimizer (Kingın & Ba, 2017) and its weight decay regularization variant, AdamW (Loshchilov & Hutter 2019). Specifically, the algorithm alternates between caleulating a perturbation and updating parameters using the Adam optimization rule. Formally, AdaSAM modifies the SAM optimization process by incorporating the notion of AdaM (Tan et al. 2019 introducing momentum term, M, (1-))) G.A. which is weighted by momentum factor B, Additionally, it
tracks a socond-moment estimate using a smoothing parameter 32, namely V, = f3 V.-1+ + (1 -2) [GSAM]² This allows it to dynamically adjust using historical gradient information. achieves a convergence rate of O(IVVTT), where b is the batch size, resulting in linear speedup with increased batch sizes and making
3

---

Published in Transactions on Machine Learning Research (09/2025)
it suitable for large-scale training scenarios. This adaptive variant requires the storage of both M, and V at each time step, resulting in a memory cost of 2mn. Additionally, the perturbation introduces an extra memory usage of mn, bringing the total memory access for AdaSAM optimization to 4mn. It is important to note that this inefficiency leads to high memory requirements and increases computational time. Comparod to gradient- based optimizers, SAM and ils variants involve two gradients, which means two backpropagation procedures are performed during a single update step.
Surrogate Gap Guided Sharpness-Awart Minimization (GSAM) (Zuuang et al. 2022) jointly minimizes the perturbed loss fp(w) := max]]e]|ss f(w + E) and the surogate gap h(w) = fp(w) f(w). It de- composes V.((w) VI((() + Vif(w), parallel and orthogonal to Vfz(w), and updates via w =
Wr - 7/ (V f (w:) - aV (ww)), promoting flatter minima for better generalization.
Memory efficient optimizers. Reducing the memory demands of training large language models (LLMs) has driven extensive rescarch in algorithmic development. One major approach reduces trainable parameters via low-rank adaptation (Hu et al. (021), though such methods often fall short of fully parameterized mod- els, especially during pre-training. Another direction focuses on optimizing training methods to reduce the memory footprint of optimizer states through low-rank gradient projection, with notable examples includ- ing AdaRankGrad, GaLore, Fira. Flora, Adam-mini, GaLore-mini, LDAdam, GoLore, LoQT, Apollo, and
AdamSN, SubTrack [Reael
(Relaell et al.; 2025 Zhao el al. 2024 Chen et al. 2024b Hao et al.] 2024 Robert
et al. 2025 Zhang et al. 2024a Nguyen & Nguyen 2025 Rajabi et al. 2025) integrating low-rank gradient projections in optimization. Elforts to reduce SAM's memory requirements have also been reported. They all appear to focus solely on the overhead caused by the perturbation (ascent step) computation. FSAM [Zhong et al. 2022) and SSAM (Zhao et al. 2023 leverage Fisher information to selectively perturb pa- rameter subset, achieving 50%-90% memory savings from the overhead at the cost of increased computation. Recent work on =-SAM (Anonymous 2024) employs nuclear norm constraints during the ascent step for greater memory efficiency. Similarly, SAM-ON (Mueller et al. 2023) introduces perturbations only to the normalization layers. However, these approaches do not address the high memory demands of the baseline optimizer performing the descent step. Furthermore, they often trade memory savings related to the ascent step for increased computational complexity or struggle to generalize across diverse ane-tuning tasks. Our method bridges this gap by introducing a low-rank gradient optimization framework that is applied in both ascent and descent directions. We also estimate randomized ascent direction (gradient perturbation), leading to both memory elficiency and computational simplicity while enabling robust generalization in pre-training and fine-tuning LLMs.
Low-rank gradient optimization. The phenomenon of low-rank gradients naturally arises during the training of neural networks, a subject that has been extensively examined both theoretically and practically,
e.g.; (Zhao et al. 2022 Cosson al., 2023 Vang et al. 2023). This characteristic low-rank structure gradient has been leveraged to reduce memory usage during training processes (Gooneratne et al. 2020;
Huang et al. 2023 Modoranu et al. 2023), resulting in reduced computational complexity compared to standard gradient descent methods. Recent work in (Refael et al. 2025) theoretically and empirically showed natural phenomenon in which the rank of reversible layer gradients (Tian et al., 2021) monotonically diminishes to one during training and suggested leveraging it to adaptively reduce the rank of the gradients during Adam optimization steps.
Zeroth-Order Optimization Zeroth-order (70) optimization estimates gradients using finite differences and relies only on function value oracles. Despite this, its structure is similar first-order (FO) gradient- based methods. It has gained significant attention due to its effectiveness across various modern machine learning challenges (Liu et al. 2020). Methods for ZO include approaches that leverage historical data to enhance gradient ostimators (Meier et al. 2019 Cheng et al. 2021). These methods utilize gradicnt structural information (Singhal el al.) 2023), exploit sparsity to reduce dinensional dependence (Cai et. al.) 2021 2022; Chen et al. 2024a), and reuse intermediate features (Chen et al.) 2024a) or random perturbations (Malladi ct al.] (023). These strategics have demonstrated significant advancements in addressing large-scale machine learning challenges. In this study, we will further leverage the effectiveness of ZO to reduce unnecessary computational costs.
4

---

Published in Transactions on Machine Learning Research (09/2025)
3 Method
In this section, we propose computationally and memory-efficiet adaptive-SAM optimization method that utilizes efficient zero-order gradient estimation lo compule the ascent direction (perturbation) and analyse its convergence guarantees. Next, we leverage a computationally efficient subspace selection method, which offers lower complexity compared to an exact Singular Value Decomposition (SVD). This subspace selection is later used to determine the optimization subspace (gradient projection). Finally, we present Lorenza, along with its enhancement variant Lorenza+, which both apply memory and computationally efficient low-rank SAM optimization updates.
3.1 Single gradient SAM approach via zeroth-order ascent estimation
In this subsection, we introduce Algorithm [ which we call AdaZo-SAM. This algorithm is designed to reduce computational effort, training time, and memory consumption associated with the adaptive SAM schema. Unlike traditional gradient-based optimizers, SAM and its variants require the calculation of two gradients, effectively performing backpropagation twice in a single update step. Inspired by the randomizod gradient estimator (RGE) (Nesterov & Spokoiny] 2017 Duchi al. 2015), which relies the finite difference of function values along randomly chosen direction vector, propose estimating the perturbation ascent direction instead of calculating the exact SAM perturbation. This approach climinates the need for additional costly gradient calculations, ions, allowing backpropagation to be applied only once during each update step rather than twice.
Given a scalar-valued function S(W) where W E gm the RGE (referred to as VS(W)) expressed using central difference scheme, namely
S(W MMU S(W U
vS(W)
U
(1)
24
where U, E mm is a random direction vector typically drawn from the standard Gaussian distribution I),,I), q the number of function queries, and μ > is a small perturbation stepsize (also known as smoothing parameter). The value of 9 trades off the variance of the RGE gradient estimator against the 1, the finite difference of the function values in overall computation cost. Notice that as f 0 and q vec(V (W)), denoting the directional derivative of vec(U.))
approaches f'W U,) Tr (Vf(W)TU, S(W), along the random direction U, yielding v/(W) + J'W,, U,), thus,
vec (tr(w))] = Eu ve.(U.)) ( U.)
lim #
= -uu vec(U.) vec(U;) vee( (I (W.())))) = B[ vec(V.
(2)
It might be assumed that estimating the ascent direction would break the convergence guarantees of SAM and AdamSAM. However, we show that even using just one randomized matrix, i.e., g 1, is sufficient to maintain the same convergence properties. To analyze the convergence of the AdaZo-SAM algorithm, for the simplicity of writing, we let [(()](W) Eg w ()W; where ~ PD is stochastic input batch, and PP is the sampling distribution over dataset/domalt DD
Theorem 3.1 (AdaZo-SAM convergence rate). Consider 3-smooth, non-cowver function I parametrized by matrir W E mR where m S n without loss of generality. Suppose that I satisfies sup Bells(W: s C
W
for some large constont C' R4. Then, Algoritlom 1 when initialized at Wa and run with slep size BVT achieves the convergence rate
C3 VT
Σ B
50
+ 32
5

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1 Efficient single gradient step Adaptive SAM with zeroth-order ascent estimation (daZo-SAM)
Initial parameters Wo, M-1 V- and momentum parameters 31, 12, small number q N (default 9 1).
0, base learning rale 7, neighborhood size μ4, perturbation size p.
Output: Optimized parameter WT+1-
for iteration t E 20.
1 do
Sample mini-batch B = [1, E2. (1B (1) Random U, ~ 00, 1iiiiee [g]
{ Compute ascent direction (perturbation)
(2) Compute Pert using
E
(WW++U. UU U;
2p
Compute SAM gradient: GSAM
VwS(W, + P (t)
G
M. = M, 1 (1-8) -81) GAAM
{Update momentum) }
V. 32 32V, 1-1 + (1-82) (G) 2
(Update momentum and variance]
M, = (1
V (1- (85
Update parameters:
W We - WM/ (VV.+s)
end for
where f (W)) denotes the randomized gradient estimator (RGE) defined in Equation 11 0, and Po sampled from the data distribulion PD over domain D.
i with q 1,
The proof of Theorem 3. can be found in Appendix B.1 Consistent with all convergance proofs of other SAM variants (e.g., Si & Yun 2023 Sun et al. 2023). Theorem 3.1 demonstrates that SAM converges to stationary points for non-convex smooth functions, provided the perturbation size sufficiently small or decaying. Practically, for example, GSAM proposes scheduling Pt as = Pmin + Pinne- enabling the algorithm to initially explore flatter regions of the loss landscape and then gradually reduce the perturbation size in proportion to the learning rate.
Trmax Iraan
Recently, (Zhao et al. 2024 Refael et al. 2025) analyzed the gradient structure of broad family of nonlinear networks known reversible networky" Tian et al. 2021). This family encompasses various types of layers, including linear layers (MLP and convolutional) and (leaky) ReLU nonlinearities. It was proved that the low- rank structure of their gradients naturally diminishes during training and fine-tuning, a phenomenon observed empirically across various layer lypes. This insight inspired the development of optimization methods that leverage low-rank update steps to reduce memory usage while enhancing accuracy, an approach we build upon and detail the following subsection.
3.2 Low-rank gradient optimization via efficient zeroth-order adaptive SAM
Consider neural network denoted as (=::0), which consists of L layers and is parameterized by 0 A x4f- - W xdf *] Here, W, represents the weights tensor parameters associated with
,..., L-1 the i-th layer, for E [L]. By ((: we denote the loss for a sample & and prediction $({:0). With slight abuse of notation, we write g(W:E) if the context. refers to the weights of certain layer. The proposed algorithm, Lorenza, is stated in Algorithm 2 The mathematical formulation of the weights update rule proposed in this paper is detailed in Appendix A.1 It comprises four main blocks, all contained within an outer loop that terminates when convergence is reached. The role of each block is as follows.
Block 1: We select the subspace along the directions the r largest cigenvectors, but since comput- ing full SVD for large matrices is computationally intensive and memory-demandingg we leverage the Randomized-SVI by (Halko al.) (2010), and presented in Appendix 3 which is an efficient technique for producing "good" proxy for the optimal low-rank approximation. It solves the optimization prob-
These networks are formally defined in Appendix C.1

---

Published in Transactions on Machine Learning Research (09/2025)
lem arg miggeeeo* G QQ QQT
F and approximates the matrix as Gapp.r & QQ G, that requires O(mar + mr?) operations, instead of O(min(mn², m²n)) applied by SVD.
Block 2: We calculate the ww--rank-pertuuraation representing the adversarial (ascent) direction within a low-rank subspace. This is achieved by randomizing a linear combination of directions within the subspace, projecting it back onto the original space, and then using it empirically estimate the projected gradient the selected subspace (i.c., the gradient components that reside in the selected low-rank subspace). Note that throughout the paper, we use y =
The SAM direction is calculated using the low-dimensional and memoor-efficient projected gradi- ents on the selected subspace. It then updates Adam's estimates for the first and second moments.
Block 3:
Block The low-rank Adam step is projected back onto the full dimensions, and the model parameters are updated until the convergence criteria are met. Such criteria could be, for example, the number of epochs gradient norm reaching below predefined threshold, namely ||Grll I =..)
Wo AWa, 155
AW;, 45, aW, +a5
Figure The illustration depicts the training process of Lorenza The process begins by selecting a low-rank subspace using the efficient Randomized-SVD, visualized here as a 2D plane (blue and orange). Next, a low-rank AdaZo-SAM optimization step ( performed. Specifically, the estimated low-rank ascent direction S, is computed using the RGE method, on the 2Dsuubspace. This low-rank ascent direction is being used calculate the adversarial gradient G,, the perturbated weights, W, + ST..
then projected onto
the 2D-subspace, namely as G ?2 Q D2x"Gnxm Gnxm. Following this, a low-rank Adam optimization step is applied. After predetermined number of Lorenza steps, the optimization subspace Q, is updated, and the process is repeated.
Theorem 3.2 (Convergence of Lorenza). Suppose the following conditions hold:
The objective f = L(((.)) is B-smooth, non-conver, and bounded by M TR..
Time intervals Te are determined by the criterion ||G S se for update criteria).
projection
Algorithm utilizes vanilla SGD weight updates with exact SVD projection and a decay perturbation p.

---

Published in Transactions on Machine Learning Research (09/2025)
Then, for any >> there erists constant C R+ and N E N such that for all TN S
N- T-
Σ
Σ
G..SA
Se.
(3)
F
Specifically, Lorenza achieves e-critical point (t.e., | < for some t N and all layers j [Z].
The proof of Theorem 3.1 can be found in Appendix B.1 Several important points should be noted. First, to reduce memory usage, Algorithm updates weights on per-layer basis during backpropagation, following recent approaches (e.g., (Ly et al., 2024)). This differs from standard optimizers, which typically store full gradients in memory before updating all weights, leading to inefficiencies. Second, the Adam update stop in Algorithm 2 can be replaced with any quantized Adain variant (e.g., (Li et al. 2017 Chen al.j 2021
Seok & Kim 2021 enabling fine-tuned quantized models or quantized adapters. Finally, 4-bit projected
Algorithm 2 Lorenza
Lorenza + optimization
Input: A weight matrix W E mR=x" with m S n. Step size scale factor decay rates {81, 82), weight decay X, rank r, subspace update frequency T, small number E N (default = 1), a small interval length
μ.
Initialize: / + and optionally, schedule: Pmin
(Prwy Pooto MMr Irmin]
Imaa Immm
repeat
# Block Calculate low rank gradient projection.
Sample mini-batch B {E1, (.1.,
a Compute G, E aw C (0(2,, 0). if mod K 0 then
Q + Truncated Randomized (aanoomize_SVVD(GGG end if
(Alternatively criteria ||G. s s}
# Block 2: Low-rank rank ascent perturbation Randomize vector u," rx1 (O(0,) 1), j [g]
(Compute low-rank random directions]
Set P,, Qidiag(u,) R,, jE [q]
G,pert -1 -
P,
(Compute ascent direction (perturbation)
2p
# Block 3: Low-rank adaptive SAM
GSAM
Bl LEED ! (, +
1
G ;
G.en
(Compule decent direction]
G Q, GSAM
M, -- M, + -1-1 (1-33) G.
V + B2V, + (1- 32) G?
M. + M,/(1 (1-(1-3 31)
V, V./ 55)
Block 4: Update weight in original space.
W, - W. a (VV.+H +
(Lorenza update step]
GSAM- -GGGGAA GSAM - -Q2 Q.Q. GSAM
W, W. a Vv.to
Q.M. + GSMM
(Lorenza+ update stop}
t 1+ until convorgence criteria met return Wr
fe.g. epoch number, gradient norm] {A flat local minima)
gradient updates, as introduced in Q-GaLore Zhang et al. 2024b), can easily incorporated. Remark 3.3. Our convergence statements (Theorem 3.1 and Theorem 3.2) assume a uniform bound on the loss values, e.g., sup (ellf((We)) < C (and similarly that f is bounded by M, where f = L(²(-) is the
8

---

Published in Transactions on Machine Learning Research (09/2025)
composition of loss L and the network T). This assumption plays technical role in the proofs: since uses weroth-order finiteddiferrence function queries, boundedness of [) controls these evaluations and ensures the resulting estimator is well-behaved in expectation, which is needed for the stated rate. The assumption is reasonable for the regime of interest: training is performed on a finite dataset and the iterates { produced by the algorithm are effectively confined to a bounded (hence compact) region; by continuity, the empirical loss is bounded on this region by some finite constant M, consistent with our assumption. In classification and language modeling with cross-entropy, the loss is nonnegative and remains finite along the sequence (W2) (in particular, the logits do not diverge). Moreover, standard practices such as weight decay and normalization layers (e.g., LayerNorm) further restrict the effective search region, making unbounded loss growth unlikely. Finally, bounded-loss-type conditions also appear in analyses of memory-efficien low-rank optimization frameworks, and we adopt them here to obtain a stronger guaranlee in our zeroth-order/ low-rank SAM setting. We emphasize that, although this assumption is stronger than the smoothness /anesss/buuuu bounded-variance conditions commonly adopted in related work (c.g., [Si & Yun (2023); Sun et al. (2023) ) Theorems 3.1 and 3.2 also provide a correspondingly stronger guarantee. However, this assumption is standard in convergence analyses of low-rank optimization methods; see, for example, the convergence proof of GaLore in Zhao et al. (2024).
Remark 3.4 (Memory usage). Our goal is to reduce end-to-end training memory, i.e, both (i) optimizer- state memory and (ii) SAM-specific overhead. First, standard Adam requires storing gradients in addition to first- and recond-moment buffers, which can be up to ~ 3x the parameter memory. In contrast, Lorenza keeps the same model weights in memory but replaces full-rank optimizer states with low-rank states, so the optimizer-state footppint scales with the rank rather than the matrix dimension mn. This is summarized in Table [ for W E R=xm, Lorenza's optimizer states scale as nr + 2mr, comparable to GaLore and substantially smaller than additional overhead due to two backward passes. Second, to reduce optimizer-state memory, Lorenza is designed to reduce peak GPU memory usage, which is typically dominated by the largest set of tensors that must be simultaneously resident during an update (activations ggaddisnsss,sss, plus any additional SAM buffers). Standard SAM/ AdaSAM incur larger peak footprint because they require two backpropagations per step, effectively repeating peraationggraiieee contrast, our BPFP construction enables one backpropagation per step (Table D. which directly reduces this peak requirement; empirically, for OPT-13B, the backprop-based perturbation computation can consume ~ 6x more memory during fine-tuning than BPFP. Finally, Algorithm 2 further reduces peak usage by updating weights per-layer during backpropagation, avoiding the need to store full gradients for all layers before applying updates, as proposed by recent works, see, c.g., Lv et al. 2024..
which maintain full-rank states and (for SAM variants) incur
materialization ion and adding SAM-specific overhead. By
3.3 Lorenza+: Enhancement of Lorenza
We introduce Lorenza+, an extension of the Lorenza algorithm (Algorithm 2 that better exploits previonsly computed gradient information, thereby improving performance across a wide range of architectures.
In the standard Lorenza update, the optimizer discards the component of the SAM gradient G. that lies outside the low-rank subspace spanned by Q,, i.e., G$AM- = G$AM Q.Q, G$AM, This orthogonal term captures informative directions that are not represented in the low-rank subspace. Importantly, it does not interfere with the moment estimates, meaning it can be incorporated without compromising stability. Yet, existing optimizers such as GaLore, Flora, Adam-mini, GaLore-mini, LDAdam, AdamSN, and + ignore this orthogonal component in their update rules.
Lorenza+ leverages this neglected term. Specifically, it augments the Lorenza update with an additional SGD-like correction that uses GSAM I. whhile retaining the original low-rank update through M, and V..
The resulting update rule is, W. < W. a ( /vite
Q.M. ( +AMM+G,AMM
Here, the first term coincides with the standard Lorenza step, while the second introduces the complementary gradient component. Since G is already computed and stored in each iteration, no extra memory is required. Furthermore, because Q₂ is of low rank (typically rank 4, 8, or 16), the additional computational overhead is negligible.
9

---

Published in Transactions on Machine Learning Research (09/2025)
4 Experiments
Fine-tuning on the GLUE Benchmark. We evaluate our approach on the GLUE benchmark (Wang ct al. 2019 by fine-tuning the pro-trained RoBERTa-base model (Liu et al. 2019). The results, comparod against full fine-tuning, LoRA, and GaLore methods, are summarized in Table 2 For evaluation metrics, we report overall accuracy (matched and mismatched) for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, Fl-score for MRPC, and accuracy for the remaining tasks. Our method demonstrates improved fine-tuning accuracy while maintaining comparable training memory average. We employ p scheduling (as proposed in the GSAM method (Zhuang et al. 2022)), with Pmaz = 0.01, Pmin le 6, and cosine anncaling learning rate scheduler.
Table 2: Evaluating Lorenza and Lorenza+, compared to state-of-the-art memory-effeiient fine-tuning meth- ods on the GLUE benchmark using pre-trained RoBERTa Basse. We used NVIDIA A100 for the experiments.
Model
Memory
CoLA
STS-D
MITPC
ICT
SST2
MILI
QNLI
QQP
Full Fine- Tuming
747M
62:21
50.02
91:30
79.42
94:57
87.18
92:35
M2:25
747M
SAM.ON
89.95
60.58
92.27
LottA (rank=9)
257M
61 68
50.57
91.07
78:70
92.89
86.82
92.18
91.20
GoLore (rank-4)
253M
60.35
50.73
92.25
79.42
94.0
87.0
92:24
91.06
Lorensa (ranke44)
253M
61.51
91.01
92:57
81:26
94.83
K7.2
92.54
91.82
Loremset (rank(4)
253M
61.97
91.14
92.76
81.35
94.68
87. 16
92.78
91.88
LoitA (munkm8)
264M
61.83
90.80
91.90
79.06
93.46
86.94
92.25
91.22
Galom (rank=8)
257M
60.00
90.82
92.0
79.78
94.38
87. 17
02.2
91.11
LDAdam (sank-8)
257M
09.91
85.00
87.81
91.28
Sal/Truck4 (runkm5)
258M
58.28
(Li 2024
253M
62.14
92.61
81.0
94.18
87.46
Lorenza + (rank-8)
237M
0.027
0.023
0.026
0.009
0.013
47,
0.036
91. 0,004
Pre-training LLAMA on C4 Dataset. To evaluate the performance of Lorenza, we follow (Zhao et al. 2024) by comparing it with the state-of-the-at method in terms of perplexity and memory efficiency. For this evaluation, we trained large ALaMA-bascd models on the C4 dataset, a curated and extensive version of the Common Crawl web corpus (Raffel et al., 20200). This dataset is widely used for pre-training language models and developing word representations. To better reflect real-world pre-training scenarios, we conducted training non-repeating, largo-scale dataset and scaled model sizes up to billion paramcters. The results of these experiments are shown in Table 3
Table 3: Comparison of low-rank state-of-the-art algorithms for pre-training LLaMA models of varying sizes on the C4 dataset. The results are reported in terms of validation perplexity. Experiments were conducted using an NVIDIA H200 GPU.
Method
60M
130M
350M
1B
Full-Rank
34.06 (0.36G)
25.08
(0.76G)
18.80
(2.06G)
15.56(7.80G)
GaLore
34.88 (0.24G)
25.36 (0.52G)
18.95
(1.22G)
15.64(4.38G)
Low-Rank
78.18 (0.26G)
45.51 (0.54G)
37.41 (1.08G)
(42.53(3.57G)
LoRA
34.99 (0.36G)
33.92 (0.80G)
25.58
(1.76G)
19.21(6.17G)
ReLoRA
37.04 (0.36G)
29.37 (0.80G)
29.08 (1.76G)
18.33(6.17G)
AdamSN
29.75 (0.36G)
22.90 (0.80G)
17.49 (1.76G)
14.96(6.17G)
Apollo
31.55 (0.26G)
22.94 (0.57G)
16.85
(1.29G)
14.20 (4.43G)
Apollo mini
31.93 (0.23G)
23.53 (0.43G)
17.18 (0.93G)
14.17 (2.98G)
LDAdam
22.65 (0.54G)
17.30 (1.23G)
Lorenza
34.29 (0.24G)
24.92 (0.53G)
18.87
(1.24G)
14.93 (4.38G)
Lorenza+
29.69
(0.24G)
22.38
(0.53G)
16.94
(1.24G),
13.98
(4.38G)
Training
1.1B
2.2B
6.4B
13.1B
Tokens
r /dmmodel
128/256
256/768
256/1024
512/2048
Tew/Zero-shot reasoning and long-context generalization.
To evaluate the performance of our
method on complex reasoning task, we utilize the GSM8K dataset.
(Cobbe et al. 2021) to test. systematic
10

---

Published in Transactions on Machine Learning Research (09/2025)
generalization. For these experiments, used batch size of 32 and 10 epochs for fine-tuning. We present the performance result in Table 4 training Phi-2 (2.7B) model (Javaheripi et al. 2023), and training Lamma (IB) model (Touvron et al. 2023 The results demonstrate that the proposed method significantly improves generalization to out-of-distribution data. The experiments were conducted on an NVIDIA H200.
Table 4: Evaluation GSM8K: Zero-shot (Phi-2) and 8-shot (LLaMA) settings, both with Rank- 64.
Model
Accuracy
Model
Accuracy
Base
15.16%
Base
17.93%
Calore
52.24%
Calore
74.9%
Zero-shot (Phi-2, 2.7B)
LoRA
42.8%
8-shot (LLaMA, IB)
LoRA
68.3%
LDAdam
52.47%
LDAdam
75.29%
Lorenza
53.37%
Lorenza 76.4%
Lorenza+
55.26%
Lorenza+ 77.1%
In addition, Table presents the results of pretraining an LLaMA-350M model, where we used a sequence length 1024 and a total batch size of 512 for 60,000 steps. For the baseline AdamW optimizer, we conducted a learning rate sweep and selected the best checkpoint. Our Lorenza method was evaluated under the same conditions for consistency. We follow the zero-shot evaluation protocol described in Table of Zf et al. (2225), using comprehensive suite of commonsense and mathematical reasoning benchmarks. Further details are specified in Appendix D
Table Zero-shot evaluation of 3.L.MA-500M models across reasoning tasks.
Method
Memory
PPL
BoolQ
RTE
HS
WG
OBQA
ARC-E
ARC-C
PIQA
SeiQ
MathQA
Avg.
AdaanW
1.77G
16.30
0.4917
0.3683
0.3688
0.5233
0.332
0.3729
0.2449
0.6534
0.609
0.2064
0.4272
APOLLO
0.34G
15.64
0.5373
0.4608
0.3850
0.4925
0.322
0.3788
0.2483
0.6681
0.624
0.2127
0.4406
APOLLÓ-Minl
16.12
0.6376
0.4562
0.3707
0.5217
0.324
0.3758
0.2312
0.0638
0.619
0.2224
0.4374
0.31G
15.53
0.5450
0.4702
0.3900
0.3250
0.3840
0.2510
0.6095
0.627
0.2240
0,4413
Lorenza+
0.31G
15.27
0.3481
0.1711
0.3921
0.5318
0.327
0.3846
0.2521
0.6703
0.628
0.2265
0.1432
Table 6: The LLaMA 7B model was pre-trained on the C4 datasct for 120K steps with 8-bit quantization range in optimization. Lorenza and Galore were utilized for low-rank training, specifi- cally with rank of 256. Experiments were conducted using an NVIDIA H200.
Steps/Tokent Tokens
GaLore
Adam
Lorenza
Lorenza
40K 5.2B
17.94
18.09
17.81
18.03
80K /10.5B
15.39
15.47
15.28
15.11
120K /15.7B
14.95
14.83
14.82
14.78
The following hyper parameerrssssss used in the experiments: 1, E = 10-12, Ir 0.5 10-3, Pmaxx fhaxx/Pmin Pmin 101061
WD 0.5 10 4,T
= 180, μ 10-9.
4.1 Ablation Studies
4.1.1 Block 1: Effect of Using Full SVD in Projection
In Block 1 of the Lorenza optimizer, the projection matrix is computed using randomized SVD, which serves practical approximation to reduce computational costs associated with full SVD. To assess the trade-off, we conduct an ablation study comparing the original randomized SVD variant with one that uses the exact full SVD computation.
The results show that replacing the randomized SVD with a full SVD yields only marginal performance improvements, while incurring a substantial training time overhead of approximately 30%. The memory usage difference is negligible and thus not reported.
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: Ablation Study; Randomized SVD vs, Full SVD in Block 1
Model
CoLA
STS-B
MRPC
RTE
SST2
MNLI
QNLI
QQP
Lorenza (Rank=4)
62.1
90.93
92.8
81.17
94.84
87.14
92.72
91.26
Using Full-SVD
62.6
90.98
93.04
81.24
94.91
87.27
92.86
91.32
Added Time Ratio
x1.37
x1.28
x1.34
x1.32
x1.34
x1.31
x1.36
x x1.29
4.1.2 Block Effect of Backpropagation Low-Rank Gradient Estimation
Block 2 approximates the low-rank gradient using zero-order (backpropagationn-tee)) method, thereby avoiding the computational burden of full backward passes. This ablation study examines the effect of replacing this approximation with the exact gradient obtained through backpropagation.
Table 8: Ablation Study: Zero-Order Backpropagation-BBn Based Gradient in Block 2
Model
CoLA
STS-B
MRPC
RTE
SST2
MNLI
QNLI
QQP
Lorenza (Rank=4)
62.1
90.93
92.8
81.17
94.84
87.14
92.72
91.26
Using Backpropagation
62.4
91.07
93.16
81.03
94.92
87.22
92.94
91.23
Added Time Ratio
x1.23
x x.227
x1.19
x1.21
x1.24
x1.26
x1.23
1.221
The results indicate that while using true backpropagation slightly improves performance, the gains are minimal and come at the cost of 20-25% increase in training time. As with Block the memory usage differences are considered negligible.
5 Discussion
In this work, we introduce Lorenza, a novel memory-effiiiett optimization framework that enhances the gen- eralization of low-resource training for large language models (LLMs).. By combining zeroth-order sharpness- aware minimization with low-rank gradient updates, Lorenza reduces computational overhead while retaining the benefits of full-rank fine-tuning. Lorenza and Lorenza+, its enhanced version, achieve state-of- saate-of-the-rt performance across multiple benchmarks, surpassing existing low-rank adaptation methods such as LoRA and GaLore.
Our theoretical analysis further provides convergence guarantees, underscoring the robustness of our method. Lorenza effectively bridges the gap between memory efficiency and generalization, making it strong alter- native for resourco-constraince scenarios. By addressing key limitations in PEFT, Lorenza enables efficient and generalizable fine-tuning, advancing LLM training and deployment.
References
Aida Amini, Saadía Gabriel, Peter Lin, Rik Koncel-Kediiorsii, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms, 2019.
Anonymous, vSAM: Mefior-efficient sharpness-awee aware minimization via muclear norm constraints. Submitted Transactions on Machine Learning Research, 2024. Under review.
Barak Battash, Lior Wolf, and Ofir Lindenbaum. Revisiting the noise model of stochastic gradient descent. In International Conference on Artificial Intelligence and Statistics, pp.
PMLR, 2024.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 34, pp. 7432 7439, 2020.
Tom Brown, Noam Smith, al. Efficient memory usage in large language models. In Proceedings of the 36th Conference Neural Information. Processing Systems (NeurIPS 2022), 2022.
12

---

Published in Transactions on Machine Learning Research (09/2025)
HanQin Cai, Yuchen Lou, Daniel McKenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. orXiv preprint or arXiv:2022 0att210000000,,, 2021.
HanQin Cai, Daniel Mckenzie, Wolao Yin, and Zhenliang Zhang. Zeroth-order regularized optimization (zoro): Approximately sparse gradients and adaptive sampling. 687-714, 2022.
SIAM Journal Optimization, 32(2):
Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order opti- mization for deep model training. ICLR, 2024a.
Congliang Chen, Li Shen, Haozhi Huang, and Wei Liu. Quantized adam with error feedback. ACM Trans- actions on Intelligent Systems and Technology (TIST), 12(5):1-28) 26, 2021.
Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can we achieve full-rank training of Iims under low-rank constraint? arXiv preprint arXiv:2410.01623 2024b.
Y. Chen, Y. Li, and X. Liu. Lora+: Improving low-rank adaptation with parameter-specift learning rates. arXiv preprint 2305. 16045, 2023.
Shuyu Cheng, Guoqiang Wu, and Jun Zhu. On the convergence of prior-guided zeroth-order optimization algorithms. Advances in Neural Information Processing Systems, 14:14620-14631 2021.
C. Clark, K. Lee, M.W. Chang, T. Kwialkowski, M. Collins, and K. Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1004,44400 2019.
Peter Clark and Oren Etzioni. Think you have solved question answering? try are, the ai2 reasoning challenge. In aiXiv preprint arXiv:1803.mm 05457, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Retcchír Nakano, et al. Training verifiers to solve math word problems, 2021. Arzio, 2021.
Romain Cosson, Ali Jadbabaie, Anuran Makur, Armin Reisizadeh, and Devavrat Shah. Low-Rank Gradient Descent. IEEE Open Journal of Control Systems, 2023.
Arijit Das. Natural galore: Accelerating galore for memory-efficient Ilm training and fine-tuning, 2024.
Mingrong Dong, Yixuan Yang, Kai Zeng, Qingwang Wang, and Tao Shen. Implicit sharpness- awarne aware minimiza- tion for domain gencralization. Remote Sensing, 16(16), 2024. ISSN 2072-4292. doi: 10. 10.3330/1s16162877
John C Duchi, Michael I Jordan, Martin J Wainwright, and Andre Wibisono. Optimal rates for zero-order convex optimization: The power of two function evaluations. IEEE Transactions on Information Theory, (5):2778 2800, 2015.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mininization for efficiently improving generalization, 2021.
M. Gooneratne, K. C. Sim, P. Zadrazil, A. Kabel, F. Beaufays, and G. Motta. Low-rank gradient ap- proximation for memory-efficient on-device training of deep neural network. In 2020 IEFF International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Bareelona, Spain, May 4-8, 2020. IEEE, 2020.
Nathan Halko, Per-Gunnar Martinsson, and Joel A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, 2010.
Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-cfficient fino-tuning for large models: A comprehensive survey, 2024.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. Arxiv, abs/2402. 03293, 2024. URL https:: /tttps://api. semanticscholar http://aa......//opoppaaa////a11
13

---

Published in Transactions on Machine Learning Research (09/2025)
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1:1-44, 1997.
Edward J. Hu, Yelong Shon, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Shuang Huang, Brian D. Hoskins, Michael W. Daniels, Matthew D. Stiles, and George C. Adam. Low-Rank Gradient Descent for Memoor-Effiiient Training of Deep In- In-Memory Arrays. ACM Journal on Emerging Technologies in Computing Systems, 2023.
Weihao Huang, Zhenyu Zhang, Yushun Zhang, Zhi-Quan Luo, Ruoyu Sun, and Zhangyang Wang. Galore- mini: Low rank gradient learning with fewer learning rates. In NearIPS 2024 Workshop on Fine- Tuning in Modern Machine Learning: Principles and Scalability, 2024.
Mojan Javaheripi, Sébastien Bubeck, Marah Abdin, Jyoti Aneja, Sebastien Bubeck, Caio César Teodoro Mendes, Weizhu Chen, Allie Del Giorno, Ronen Eldan, Sivakanth Gopi, et al. Phi-2: The surprising power of small language modcls. Microsoft Research Blog, 1(3):3, 2023.
Matt Gardner Johannes Welbl, Nelson F. Liu. Crowdsourcing multiple choice science questions. 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima, 2017.
Beomsu Kim, Huishuai Xu, Jinxian Tang, and Song Han. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXie preprind arXiv:2305. 14152, 2023.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Bingcong Li, Liang Zhang, and Niao He. Implicit regularization of sharpness-aware minimization for scale- invariant problems, 2024. URL https: htttp:://arxivorg//// abs/ 14802
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: A deeper understanding. I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Xutao Liao, Shaohui Li, Yuhui Xu, Zhi Li, Yu Liu, and You He. Galore+: Boosting low-rank adaptation for Iims with heass-head projection, 2024.
Sijia Liu, Pin- Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod Varshney. A primer zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. volume 37, pp. 43 54. IEEE, 2020.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. anXiv preprint arXte:1907.1169) 11692 2019.
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. Adalomo: Low-memory optimization with adaptive learning rate, 2024.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. arXiv preprint arXiv:2305.17333 2023.
Florian Meier, Asier Mujika, Marcelo Matheus Gauy, and Angelika Steger. Improving gradient estimation in cvolutionary strategics with past descent directions. arXiv preprint XrXi::1900. 05268. 2019.
Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang, Poiyi Wang, Qingxiu Dong, Liang Chen, and Zhifang Sui. Periodiclora: Breaking the low-rank bottleneck in lora optimization, 2024.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Todor Mihaylov, Peter Clark, Oyvind Tafford, and Tushar Khot. Can suit of armor conduct electricity? a new dataset, for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381 2391,, 2018.
Tonut-Vlad Modoranu, Alexander Kalinov, Ermin Kurtic, Erwin Frantar, and Dan Alistarh. Error Feedback Can Accuralely Compress Preconditioners. ArXiv preprinl XrXi::2306.06088, 2023, urXiv:2306.0609)
Maximilian Mueleer, Tiffany Vlaar, David Rolnick, and Matthias Hein. Normalization layers are all that sharpness-aware minimization needs, 2023.
Yurii Nesterov and Vladimir Spokoiny, Random gradient-fres minimization of convex functions. Foundations of Computational Mathematics, 17:527-566, 2017.
Thien Hang Nguyen and Huy Nguyen. Lean and mean adaptive optimization via subset-norm and subspace- momentum with convergence guarantees. In Forty-second International Conference on Machine Learning, 2025. URL https:///penrreviet jttp://ww.net/tooommmmddddadaaaa
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1667, 2020.
Sahar Rajabi, Nayeema Nonta, and Sirisha Rambhatla. S Gradient subspace tracking for scalable Ilm training, 2025. URL
01586
Ychonathan Refacl, Jonathan Svirsky, Boris Shustin, Wasim Hulcihel, and Ofir Lindenbaum. Adarankgrad: Adaptive gradient rank and moments for memory-efficiet LLMs training and fine-tuning, In The Thir- teenth International Conference on Learning Representations, 2025.
Thomas Robert, Mher Safaryan, Tonut-Vlad Modoranu, and Dan Alistarh. Ldadam: Adaptive optimization from low-dimensional gradient statistics, 2025. URL https:///rxii wwtp:/o//./aa...oaaeeooo
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge al scale. Communications of the ACM, 64(9):99- (19),99-106, 2021.
Jinwuk Seok and Jeong-Si Kim. Quantized adam with monotonically increasing resolution of quantization. In 2021 International Conference on Information and Communication Technology Convergence (ICTC), pp. 1719-1721 IEEE, 2021.
Dongkuk Si and Chulhee Yun. Practical sharpness-aware minimization cannot converge all the way to optima, 2023.
Utkarsh Singhal, Brian Cheung, Kartik Chandra, Jonathan Regan-Kelley, Joshua B Tenenbaum, Tomaso A Poggio, and Stella X Yu. How to guess a gradient. arXie preprint arXiv:2312.04 709,, 2023.
Hao Sun, Li Shon, Qihuang Zhong, Liang Ding, Shixiang Chen, Jingwei Sun, Jing Li, Guangzhong Sun, and Dacheng Tao. Adasam: Boosting sharpnest-aware minirnization with adaptive learning rale and momentum for training deep neural networks, 2023.
Tao Tan, Shiqun Yin, Kunling Liu, and Man Wan. On the convergence speed of amsgrad and beyond. In 2019 IEEE 31st International Conference Tools with Artificial Intelligence (ICTAI), pp. 464 470. IEEE, 2019.
Yuandong Tian, Lantao Yu, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning with dual deep networks, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bash- lykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Xiv preprint orXiv:2307.0922 2023.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi- task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding sys- tems. In Advances in Neural Information Processing Systems, volume 32, 2019.
Y. Wang, H. Zhang, and J. Li. Adaptive lora: A rank-adaptive method for efficient 1lm fine-tuning. arXiv preprint ar xrv::2306 06188, 2023.
Jiayi Xu, Zhibo Liu, and Cheng Wang. Understanding llms: A comprehensive overview from training to inference. Xiviv preprint arXiv:2401.00 0200008 2024.
Z. Xu, J. Zhang, and X. Wang. Sparse lora: Efficient fine-tuning of large language models with sparse low-rank adaptation. arXiv preprint arxi:2200. 17626, 2023.
Greg Yang, Jacob B. Simon, and Jeremy Bernstein. A spectral condition for feature learning. arXiv preprint WwXfv:2310118815 2023. arXiv:2310118113.
Yun Yue, Jiadi Jiang, Zhiling Yo, Ning Gao, Yongchao Liu, and Ke Zhang. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800, 2019.
Yushum Zhang, Congliang Chen, Ziniu Li, Tian Ding, Chenwei Wu, Yinyu Ye, Zhi-Quan Luo, and Ruoyu Sun. Adam-mini: Use fewer learning rates to gain more. arXiv preprint Xiv: 16793, 2024a.
Zhenyu Zhang, Ajay Jaiswal, Lu Yin, Shiwei Liu, Jiawei Zhao, Yuandong Tian, and Zhangyang Wang. Q-galore: Quantized galore with int4 projection and layer-adaplive low-rank gradients, 2024b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian.
Galore: Memory-effciint Ilm training by gradient low-rank projection, 2024.
Jingzhao Zhao, Frederik T. Schaefer, and Anima Anandkumar. Zero initialization: Initializing neural net- works with only zeros and ones. Transactions Machine Learning Research, 2022.
Yang Zhao, Hao Zhang, and Xiuyuan Hu. Randomized sharpness-aware training for boosting computational efficiency in deep learning, 2023.
Qihuang Zhong, Liang Ding, Li Shen, Peng Mi, Juhna Liu, Bo Du, and Dacheng Tao. Improving sharpness- aware minimization with fisher mask for better generalization on language models. ArXiv, 2022.
Hanqing Zhu, Zhenyu Zhang, Wenyan Cong, Xi Liu, Sem Park, Vikas Chandra, Bo Long, David Z. Pan, Zhangyang Wang, and Jinwon Lee. Apollo: Sgd-like memory, adamw-level performance, 2025. URL
https://arxiv. org/ abs//////// ab//a////../..///////////a/////a 25270
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, sekhar tatikonda, James s Duncan, and Ting Liu. Surrogate gap minimization improves charpness-aware training. International Conference on Learning Representations, 2022.
16

---

Published in Transactions on Machine Learning Research (09/2025)
A Appendix
A.1 Additional algorithm details
Definition A.1. LowwRank Gradient by Efficient Leroth-Ordes Adaptive SAM (Lorenza)] Lorenza defines the following gradient update rules.
Pert Q.Q. (w:((: R.
Gp
GSAM = Tws vww
W.+P
Lorenza
G, Qipe (QT G.AMR.)
R
T-1
Wr Wo Σ Ge.
where Pe an entry-wise stateful gradient regularizer (e.g., Adam), Q E m and R. E R'x" are projection matrices, T E N is the subspace update time, the learning rate, and is stochastic batch.
Note that Definition A.1 describes the Lorenza step in general form. However, the perturbation projection is actually implemented more efficiently (as in Algorithm 2
Following, we present the Algorithm 3 Truncated Randomized SVD, which efficiently approxinates the leading singular vectors and values of a large matrix while reducing computational cost compared to the classical SVD.
Algorithm 3 Truncated Randomized SVD
Input: Matrix A E jmxxn target rank r, oversampling p, power iter. g
Draw Ω (++(p))) set Y = AΩ
for i = 1 to do
Y = =A(AT) Y)
end for
Q = orth Y)
= Q A, then B= ÜΣV
QU
Return: Truncated U,, V
The computational elficiency of this method stems from its key operations: multiplying the matrix A with random projection matrix Ω (complexity O(mer)) and performing a QR decomposition on the resulting matrix (complexity O(mr²)). The overall complexity, O(mrr m simplifies to ((mrr) when r & m, n, making significantly more scalable than the exact SVD, which requires O(min(mn", mmminm"",mm²n)) This approach enables efficient extraction of leading singular vectors large-scale data, making it well-suited for computationally constrained scenarios.
B Proofs
B.1 Proof Theorem 3.
Consider B-smooth, non-convez function parametrized by a matria W Rmxn where S n, without loss of generality. Suppose f salisfying sup (el] C for some large C E R4 then, Algorithm 7 initialized at Wo with step size
W
DVT'
C3 VT
+
50
+ p2
17

---

Published in Transactions on Machine Learning Research (09/2025)
where is the RGE i of function f with q 1, 0, and & PD is stochastic batch, drown by distribulion PP over domain D.
Proof. First, consider the following notation. For the simplicity of writing, we let Vf(W)= = Vwf(W::) where & ~ P is a stochastic input batch, and Pd is the sampling distribution over dataset/ domain DD
Accordingly, we denote E [V](W)] = EEg [Vwf(W;E)) We denote the estimated gradient at W by
Eg Wwf
1
9
E f(W + (U;))) s(W (UU;))) U. R
9
24
where U NO, 1/n) is a randomized matrix. Similarly, for simplicity of writing, we denote Vf(W) = ES (w)(W;)) Notice that as + 0 and q 1, the finite difference of the function values in approaches f',,,, U,) = Tr (V U,), denoting the directional derivative of f(W), along the random direction Us, yielding ((W (W,, U,) thus,
lim E ( ? = Eu IIf (W.) U.ll2 = EU Tr (VS(W.) U.) U. = E f (WJl). P-
(4)
For the simplicity of writing, let X, = W. + ((w)) 9-smoothness of f, we have
br.wo] thus W4I1 = W. - /VV Now, by the
Ef (111) SBf (W.) + vec (V.(W.) vec (W++1- [W:1--:)] + E [ 2+ Well?
8
/n7 2
Ef (We) 6) - TE vec (Vf vec (xx))
+
E / (X.)
(T)
2
=RS (W,) - - [ - FEIVS ( )|
R (/ (X(X)
Bn²
+
E E (I))
2
< Ef (W;) 3 V (W.)
IW, X.II.
(TT)
2
32p2nn
(W,) - 3B V
(5)
(1TT)
2
82p2n
B
44 (W,)
+
(iV)
2
2
where (1) follows by the definition of X., (II) follows from 22
An 2
(11I)
follows by
20
29T
(( P FTW F
X Well
and finally (IV) follows by Equation
(4)
Rearrearage both sides to
bound the gradient Forbimus norm, we obtain
E f < (Bf (B.) Ef (W++1)) + p²ppp²
2 17
Adding up the inequality for E [T 1], and dividing both sides by T, we get
2 (ES (Wo) - Es (WT)) + 32p²
<
20
8²0²
T/T
Now, choosing 17 BV7' we have
2C
203
+ B2p² <
+ p
nT
VT
18

---

Published in Transactions on Machine Learning Research (09/2025)
thus
T Σ E f 1=0
CB 50 (0 + B².
B.2 Proof of Theorem 3.2
Consider B-smooth nonconver composition of = L (T(.)) that is bounded by some M Rt. Let G denote the gradient matriz w.r.t. the j-th reversible layer W
2. at time E N, for all E [L] and t E N, and
T E N times are set by convergence criterion (that is, ||G I1 S st). Consider any decay perturbation p
then, for any & > 0, there exist C E RR+ and N such that for all Tu S, th EN E.
G SAM
T=
S.
Principally, Algoritho 2 with vanilla SGD weight update ? achieves an e-crilical point for some N, and any j E [T].
i.e.,
G
S
Proof. We denote by Tr E the time index at which we update the subspace, at Block of the algorithm, for the b-th time, for E & N. For the simplicity of writing, for the j-th layer W we omit j from W and use instead G. = Vws (0,) = V (We). In addition, for simplicily, we let X, = W. W W. - V (X.). Consider the SVD decomposition the gradient ( - UT, Er. VT,- Accordingly, for [T:, Ti+1 - 1], we define the low rank gradient as G, A QT, for QT: UT, [ r] UT, [s:r] where UT, is obtained by the subspace search, using the exact truncated SDV calculation. Now, let h P
v.(W.) (TV/WW [[]]
thus W
Es (W.) - Eff (WT,+1), and 1j = 1/ denote the learning rate. Then,
h++1 Ef (W++1) Ef (WT++1)
pppnn 2
EF (W,) - [Ef (WT++1)
"E
IVV (W
2
p8ppn 2
h
E 44
(I)) 2
F
where (1), follows Equation 5 and (II) follows by Equation T n,, for all 2 0, we readily obtain that,
4h. Rearranging equation and choosing
Tua-1
2(HT. hTwi) (Tit1 p2
Σ
E
7
Thus, for N N,
N-1T
2
2(hT. hTem)) + (T+1 T2)
Σ Σ
E
TN t=T;
TN
2(HT.
hTx)
(TN - 82)
T M
TN
BVTN + p²
(7)
1 where 17 AVTN"
Now by the definition of QT. for any i N there exists some (0, for which
&f(W)) Qr. v of
<a vl (
(8)
We focus SGD simplicity (as is standard practice related literature, egg. Zhao 3 Also known as e-stationary, see, e.g., (Cosson
2024
2023
19

---

Published in Transactions on Machine Learning Research (09/2025)
Obviously, the following clearly holds for any t N,
QT. v.
12
Q7. Q.. 43 8/ (w.)
Qr. v (W.)
2
( Qr. V/ (W,)
(9)
p.
and thus by plugging equation Qr. v (WT.) Accordingly,
g into equation
at t = Ti, for any i E N, we get, (1 - a)]l (WT,)
Qf ( < 1-a
- QT. VJ (WT.) 2
(10)
Recall from Lemma B.3, Equation 31, in
Zhao et al. 2024 that for the reversible layer,
E
4 (W,)
E (I n/8 (W.-)) ||
(11)
F
S [1(1 - -n/s 18
(12)
max 1 1
(13)
where {Xe]e are the eigenvalue S. Thus, using the fact that S is positive semi-definite matrix, the learning rate 17 satisfies 77 - where Amax is the maximum eigenvalue of S, it follows that max, - s 1.
E (W Therefore, the Frobenius norm of the gradient decreases monotonically as function of time 1. Now, recall that is any positive number such that & < vI- a |& (WT._.) [[.. According to (10), this necessarily implies that in each block i, we will execute (at least once) the low-rank optimization block (indeed, the condition || I (W 1T, || > & is satisfied). This, conjugated with the monotonicity property that || E TT+1 - 1] and E [N], imply that
Consequently, this implies that E [
b (WT,) for any
N-1 T41-1
N
T41-1
T Σ Σ
(W:)
(
(14)
TN
t=
N-ITaI-1
Σ
QT. (/
(15)
(1 TN
T=T
M
82p²
(16)
(1 (1-a)8
1-a
Accordingly, for decaying perturbation p= Pt, without the loss of generality for any small enough 6, 2 there exist 2 + S2, where TN > !- 2 2 and pN < 30,
0,
M²
C
11
(
(s Se 52 Se
min OStSTx
F
and thus, there exists an iteration index t [0, Tı] for which,
[>())) Se,
(17)
which, by definition, implies that Algorithm achieves an e-critical point.
20

---

Published in Transactions on Machine Learning Research (09/2025)
C Additional definitions
Definition C.1. (Reversibility Tian et al. 2021 A neural network ø that maps input to output y - =d(x;;) 0) is reversible, if there exists L(x; 0) so that = L(x; 0)x, and the backpropagated gradient Jæ satisfies 9= = I. (z; f) where By is the backpropagated gradient at the output y- L(x;; 0) depends on the input x and weight in the network
Evaluation on Zero-Shot Reasoning Tasks additional details
Specifically, we evaluate the pretrained models on the following tasks:
Perplexity: Measured on the C4 dataset (Raffel et al.
2020
Commonsense Reasoning: BoolQ (Clark al. 2019 RTE
Wang al. 2018 HcllaSwag (HS)
(Zellers et al. 2019), Winogrande (WG) (Sakaguchi et al.,
2021),
OponBookQA (OBQA) Miihaylov
et al. 2018
ARC-Easy (ARC-E), and ARC-Challenge (ARC-C)
Clark & Etzioni 2018
Physical and Scientific Reasoning: PIQA Bisk et al.
2020 SciQ (Johannes Welbl 2017 and
MathQA (Amini et al. 2019).
To ensure a fair comparison, we repeat the experimental setup used for AdamW, APOLLO, and APOLLO- Mini. The results are summarized in the table below.
21


Reviews:
[{"review_id": "NpKblH1JgP", "reviewer": "Reviewer_EzyA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Lorenza (and its enhancement Lorenza+), a memory-efficient training algorithm that demonstrates good generalization performance, applicable to both pretraining and fine-tuning of large language models (LLMs). The algorithm combines zero-order gradient estimation, low-rank gradient updates, and Sharpness-Aware Minimization (SAM) to achieve memory savings while retaining the advantages of SAM in improving model generalization.", "claims_and_evidence_yn": "No", "claims_explanation": "1. Motivation is unclear: While the paper aims to propose a low-memory fine-tuning algorithm with generalization comparable to full fine-tuning, it does not clearly articulate why the sharpness-aware (flat-minima) principle of SAM is particularly beneficial in the fine-tuning context. Fine-tuning presents distinct challenge, such as domain shift, limited data, and catastrophic forgetting. The paper would benefit from a clearer discussion on how encouraging flat minima relates to these characteristics. Moreover, the pretraining experiments are also included, but pretraining faces its main memory bottleneck in activation values, not optimizer state, so the memory savings claimed by Lorenza might not be as crucial during pretraining.\n\n2. Lack of intuitive understanding of the algorithm: The introduction of the correction term in Lorenza+ seems to be added directly to the gradient update, but the rationale behind this step is not explained intuitively. Is there a simpler or clearer explanation for why this correction term is necessary, and how it contributes to the performance improvement?\n\n3. Insufficient experiments: \n    - The fine-tuning experiments are only conducted on a Roberta-base model (with <1B parameters). It would be beneficial to include experiments on larger models, such as OPT (>1B), to better demonstrate the scalability of the method.\n    - Furthermore, experimental settings and hyperparameters are not clearly stated. For instance, the frequency of performing SVD and other key details need to be included to ensure reproducibility.\n    - From the experimental results, the advantage of Lorenza does not seem significant, particularly for pretraining tasks. The Lorenza+ variant shows more obvious improvements, which raises the question: Is the main contributing factor the correction term introduced in Lorenza+? More experiments and clarifications are needed here.\n\n4. Additional concerns:\n    - Algorithm details: In standard SAM, the normalization is performed using the 2-norm, but in Algorithm 1 and Algorithm 2, the normalization is done using the Frobenius norm. What is the reasoning behind this choice? In fact, the perturbation term , in the default case, is essentially just a scalar multiplication of a random normal matrix . This makes the perturbation direction essentially randomly determined with only the perturbation length differing. Why does this random choice of perturbation direction not lead to a significant degradation in performance, especially with its potentially high variance?\n    - Inaccurate statements and improper citations: There are instances of imprecise statements and improper citation practices. For example, in the Introduction, it is stated that `GaLore reduces the number of fine-tuned parameters’, but GaLore mainly reduces the optimizer state size. Similarly, in Table 3, many of the experimental results appear to be derived from the paper of GaLore or Apollo, but these results should be cited properly and it should be confirmed that the numbers are consistent with the original papers.\n    - Ablation experiments: The ablation studies are insufficient. It is recommended that these studies be presented in the main body of the paper rather than in the appendix. Additionally, ablation experiments should include results on how different components (e.g., correction term, low-rank updates, zero-order gradient) influence performance.", "audience_yn": "Yes", "audience_explanation": "The proposed algorithms are of interest to researchers and practitioners focusing on memory-efficient LLM fine-tuning.", "requested_changes": "Major Changes\n\n- Clarify Motivation and Connection to Fine-Tuning: The authors are encouraged rewrite the motivation to explicitly articulate why the sharpness-aware (flat-minima) principle is specifically beneficial for fine-tuning. The current text does not sufficiently connect SAM's objectives to unique fine-tuning challenges such as domain shift, limited data availability, or catastrophic forgetting.\n\n- Provide Intuition for the \"Correction Term\": The introduction of the correction term in Lorenza+ currently lacks sufficient explanation. Please provide a clear intuitive or theoretical justification for why this term is added directly to the gradient update and how it specifically contributes to performance gains.\n\n- Expand Experiments to Larger Models: The current evaluation on RoBERTa (<1B parameters) is insufficient to prove the method's utility for modern Large Language Models. The authors are requested to conduct fine-tuning experiments on models larger than 1B parameters (e.g., OPT-1.3B, Llama-series, or similar) to demonstrate the scalability and robustness of the proposed method.\n\n- Strengthen Ablation Studies: (1) Placement: Key ablation studies should be moved from the Appendix to the main body of the paper. (2) Content: Expand the ablations to isolate the impact of specific components, particularly the correction term versus the low-rank updates. Given that Lorenza+ significantly outperforms Lorenza (which shows marginal gains), the authors must clarify if the performance improvement is primarily driven by the correction term rather than the core algorithm.\n\n- Clarify Algorithm Mechanics: (1) Please explain the rationale for using the Frobenius norm for normalization in Algorithms 1 and 2, as opposed to the standard 2-norm used in vanilla SAM. (2) Provide a justification for the random perturbation strategy. Since the perturbation is essentially a random normal matrix scaled by a scalar, the direction is random. Please discuss why this high-variance approach does not lead to performance degradation.\n\nMinor Changes.\n\n- Improve Reproducibility: The authors are encouraged to explicitly list all experimental settings and hyperparameters, specifically details such as the frequency of SVD computation. A detailed hyperparameter table should be included to ensure the work is reproducible.\n\n- Correct Inaccuracies and Citations:\n\n    - Correct the statement regarding GaLore in the Introduction. GaLore primarily reduces optimizer state size, not the number of fine-tuned parameters.\n\n    - In Table 3 (and similar tables), properly cite the sources of the baseline results (e.g., GaLore, Apollo). If numbers are taken directly from those papers, please clearly indicate this and verify consistency.", "broader_impact_concerns": "No ethical concerns", "additional_comments": "I recommend a **major revision** of the manuscript.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hp1KS7gxXP", "reviewer": "Reviewer_BZsz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a family of memory-efficient sharpness-aware optimizers for training large neural networks based on zeroth-order perturbation estimation and low-rank gradient projection. \n\nThe first contribution in Section 3.1 is an optimizer AdaZo-SAM  that imitates SAM’s two-step update but avoids the expensive second gradient computation. The key idea is to use a zeroth-order finite-difference estimator with a single random probe to reduce the cost in back propagation. Theorem 3.1 provides  convergence analysis. \n\nThe second contribution is Lorenza, which extends AdaZo-SAM by projecting gradients into a low-rank subspace using randomized SVD. This reduces memory footprint while still attempting to capture high-curvature directions within a subspace. Theorem 3.2 provides  conference analysis.\n\nThe third contribution extends Lorenza with an additional correction term that reintroduces the gradient component outside the low-rank subspace at essentially no additional memory cost. This is intended to reduce the generalization gap caused by low-rank projection.", "claims_and_evidence_yn": "No", "claims_explanation": "Although the paper claims to implement sharpness aware minimization (SAM), I do not see this supported in the actual theoretical results. Theorem 3.1 and 3.2 do not provide any guarantees that is consistent with SAM. They only provide standard convergence results. \n\nAlthough the algorithms are somewhat inspired by SAM as far as I can see the proposed approach does not approximate the SAM inner maximization nor optimize the SAM minimax objective.", "audience_yn": "Yes", "audience_explanation": "I believe the convergence analysis would be of broad interest to researchers interested in fast optimization of LLMs.", "requested_changes": "Please address the following questions:\n\nTheorem 3.1:\n\n1. How do your results hold for realistic $\\mu$ rather than $\\mu \\rightarrow 0$?\n2. Why does the variance term that scales as the dimension $d$ not appear in your bound?\n3. The theorem does not analyze the SAM objective at all. How does it ensure SAM-like behavior?\n4. The bounded-loss assumption is violated for CE-based LLM losses. Can you clarify this discrepancy?\n\nTheorem 3.2 \n1. The theorem assumes reversible layers. Is this consistent with the models used in the experiments?\n2. There is no bound on the R-SVD approximation error. How does this affect convergence?\n3. How does the analysis incorporate the noise introduced by the q=1 zeroth-order ascent?", "broader_impact_concerns": "Ethical implications do not seem applicable.", "additional_comments": "Please address the questions above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "sq3Fc83de8", "reviewer": "Reviewer_De7q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces Lorenza, which is a memory-efficient optimizer that combines Sharpness-Aware Minimization (SAM) with low-rank gradient updates to improve generalization in LLM training. The authors provide theoretical foundations for the method. The method is evaluated in pre-training, pos-training and zero-shot reasoning benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I'm not an expert of this field, but the theoretical part could set a good foundation of the method. However, my concern is regarding the assumption of the method could be too strict and hard to generalize.", "audience_yn": "Yes", "audience_explanation": "This work focus on the main part (optimizer) of language model training.", "requested_changes": "1. The experimental results although beat the baselines, the improvement is marginal. Shows 0.5-2% gains over baselines across GLUE, pre-training, and reasoning tasks - improvements that, while statistically questionable due to missing confidence intervals. Lorenza+ shows better results, but it is essentially Lorenza + standard gradient component. \n\n2. The reversible network assumption is so limited, and could only applies to specific architectures. This is also the foundation of the while 3.2 theorem. \n\n3. I'm also worried about the zeroth-order approximation. Numerical precision limits how small μ can be. And there is no analysis of finite μ effects on convergence. This also make me confuse about why it works so well.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)