Three Forward, One Backward: Memory-Efficient Full-Rank Fine-Tuning of Large Models via Extra Forward Passes
Authors: Jia Zhang
Yu Bai
Hualin Zhang
Tianshuo Chen
Zhaogeng Liu
zhiqiang xu
Yi Chang
Bin Gu
Venue: ICLR 2026 Poster (2026)
Keywords: "LLM tuning"
"LoRA"
"Zeroth order"
Primary area: optimization

Abstract: Fine-tuning large language models (LLMs) has achieved significant success in downstream tasks.
However, as the model size continues to grow, traditional fine-tuning methods have become increasingly impractical due to their high computational and memory costs.
This has motivated researchers to explore parameter-efficient and memory-friendly fine-tuning strategies to enable scalable approaches, with Low-Rank Adaptation (LoRA) standing out as a representative work.
However, the LoRA update is restricted to a low-rank subspace, which results in suboptimal performance compared to the full-parameter update.
Recent research has also explored memory-efficient fine-tuning LLMs using just forward passes while suffer from high variance in gradient estimation and low convergence speed.
To address the issues above, we propose a new alternating optimization framework called LMAO (**L**ow-rank and **M**emory-efficient Zeroth-Order **A**lternating **O**ptimization), which combines the advantages of LoRA and MeZO.
This method alternately updates the low-rank components and zeroth-order directions during training.
By performing three forward propagations and one backward propagation, each update is full-rank, thereby reducing feature loss and enabling efficient fine-tuning under strict memory constraints.
We provide theoretical guarantees on the convergence and convergence rate of this method.
Empirical results demonstrate that, in experiments on multiple models (e.g., OPT, RoBERTa-large), LMAO achieves performance comparable to first-order methods.
This presents a practical and scalable solution for fine-tuning large-scale models.
Our source code is available at [https://github.com/workelaina/LMAO](https://github.com/workelaina/LMAO).

Published as a conference paper at ICLR 2026
THREE FORWARD, ONE BACKWARD: MEMORY- EFFICIENT FULL-RAND FINE-TUNING OF LARGE MOD- ELS VIA EXTRA FORWARD PASSES
Jia Zhang** Zhaogeng Liu
Yu Bai Zhiqiang Xu
Hualin Zhang 2+
Tianshuo Chen Bin Gu
Yi Chang 1.3
1 School of Artificial Intelligence, Jilin University, China
Mohamed bin Zayed University of Artificial Intelligence. Abu Dhabi, UAE
"Ineeraational Center of Future Science, Jilin University, China 4 Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, MOE, China (jiazhang25, baiy23, chents25, zgliu20]emaiiss jlu. edu. on
zhanghhualiin88@@ai.....
zhiqiang xu@mbzuai acc.aai.acaaa.. ac.
fyichang, gubin]e flu. www.luu..u...
ABSTRACT
Fine-tuning large language models (LLMs) has achieved significant success in downstream tasks. However, as the model size continues to grow, traditional fine-tuning methods have become increasingly impractical due their high com- putational and memory costs. This has motivated researchers to explore parameter- efficient and memory-friendly fine-tuning strategies to enable scalable approaches, with Low-Rank Adaptation (LoRA) standing out as representative work. However, the LoRA update is restricted to low-rank subspace, which results suboptimal performance compared to the full-parameter update. Recent research has also explored memory- effficeent fine-tuning LLMs using just forward passes while suffer from high variance in gradient estimation and low convergence speed. To address the issues above, we propose a new alternating optimization framework called LMAO (Low-rank and Memory-efficient Zeroth-Order Alternating Optimization), which combines the advantages of LoRA and MeZO. This method alternately updates the low-rank components and zeroth-order directions during training. By performing three forward propagations and one backward propagation, each up- date full-rank, thereby reducing feature loss and enabling efficient fine-tuning under strict memory constraints. We provide theoretical guarantees on the conver- gence and convergence rate of this method. Empirical results demonstrate that, in experiments multiple models (e.g., OPT, RoBERTa-large), LMAO achieves performance comparable to first-order methods. This presents a practical and scalable solution for fine-tur mne-ttnnng large-scale models. Our source code is available at httts://giihub..oo/// Lir.
INTRODUCTION
Large language models (LLMs) have achieved remarkable performance across wide range of domains (Solaiman et al. 2019; Brown et al.) 2020 Achiam et al., 2023). However, as LLMs continue to scale, gradient computation
demand over 12x the memory required for inference (Malladi et al., 2023). posing significant challenges to model training and fine-tuning particularly exacerbating optimization difficulty in resource-constrained environments. This challenge has spurred the development of memory-efficient alternatives, such as parameter-efficient fine-tuning (PEFT), which aim to reduce resource requirements while preserving task-specific performance.
"Co-first authors with equal contributions.
'Corresponing author.

---

Published as a conference paper at ICLR 2026
Output
Output
Transforme Layer
Transformer Layer
x1
Transfomer Layer
Transformer
Layer
Weight
Transformer
Transformer
Layer
Layer
updated
Pre-update
pertutbasion)
Input
Input
backpropegaiio
LoRA
LMAO
Figure 1: Comparison of standard LoRA and LMAO: LoRA performs single forward- bawkwrd pass, while LMAO alternates it with two zeroth-order forward passes. Note: the forward and backward passes update only the low-rank matrices B and A in LoRA, while the base weight matrix W is updated solely through the forward pass.
PEFT methods allow the adaptation of LLMs to downstream tasks by tuning only a subset of model parameters or by adding task-specific modules, such adapters (Houlsby al. 2201)), pre- fix embeddings (Li & Liang 2021). and prompt tuning (Lester et al. (202)) without altering the base architecture. Studies further demonstrate that fine-tuning pretrained models can be effectively conducted within reduced- ddimensioonl parameter subspaces (Aghajanyan et al.) 2020). Low-Rank Adaptation (LoRA), a prominent PEFT technique, leverages this insight by performing adaptations within optimized low-dimensional parameter spaces. This strategy significantly enhances the effi- ciency of adapting large-scale pretrained models, validating the theoretical foundations regarding low-rank representations in neural networks (Hu al. 2022 Aghajanyan et al. 2020 Li et al. 201 8). Despite the efficiency, LoRA still falls short of matching full-model fine-tuning performance due to its limited representation power of fine-tuned language models (Hu et al. 2022).
Meanwhile, zeroth-order optimization offers an alternative approach to model adaptation, particularly when gradient access is restricted. Zeroth-orde optimizers approximate gradients through forward passes alone, eliminating the need for backpropagation and further reducing memory requirements. MeZO (Malladi et al. 2023). Memory-efficient Zeroth-Orden optimizer, established the feasibility of fine-tuning large models using forward passes only, enabling gradient-free adaptation for various downstream tasks. However, the inherent noise and high variance of zeroth-order gradient estimates pose limitations on convergence speed and task performance.
In this paper, propose memory-efficient full-rank fine-tuning method, the Low-rank and Memory- efficient Zeroth-order Alternating Optimization (LMAO) algorithm. LMA0 combines LoRA and McZO optimize model performance by alternately updating low-rank matrix components and zeroth-order directions. Each iteration involves three forward passes and one backpropagation. ensuring full-rank updates. This approach reduces feature loss and enhances model expressiveness compared to traditional low-rank methods. LMAO also integrates the memory efficiency of both PEFT methods, ensuring tight control over memory usage. It introduces new alternating opti- mization framework with theoretical convergence guarantees, making it memory-efficient solution for fine-tuning large models. The algorithm is well-suited for memory-constrainee environments, maintaining high performance while staying within memory limits, enabling efficient large-scale model optimization.
2

---

Published as a conference paper at ICLR 2026
Accuracy/F3 (96
76
CB
WSC
MultiRC
ReCoRD
Zero-shot
ICL
Mezo
LoRA
LMAO (ours)
Figure 2: OPT-1.3B results under zero-shot, ICL, MeZO (Memory-efficient Zeroth-Orde Poate-rrderrr optimiza- tion), LoRA (Low-Rand Rank Adaptation), and LMAO (ours). LMAO achieves superior performance on most tasks. See Table 2 for details.
2 RELATED WORK
Parameter-Effliient Fine-Tuning. Parameter-Efficien Fine-Tunie Tuning (PEFT) methods reduce the computational and memory costs of fine-tuning large pre-trained language models (LLMs). PEFT adapts LLMs to downstream tasks by adjusting a small subset of the model's ' parameters while keeping most weights fixed. This approach achieves task- improvements without the extensive hardware required for full-model fine-tuning. Prompt tuning (Lester et al. (022) optimizes continuous prompt vectors appended to input embeddings, enabling task-specific conditioning. Prefix tuning Li & Liang, [021] introduces trainable prefix tokens at each transformer layer for lightweight conditioning. Adapters (Houlsby al. 2019) add small neural modules between transformer layers, which are tuned independently. Low-Rank Adaptation (LoRA) (Hu et al. (022) reduces parameter overhead by injecting low-rank matrices into attention and feedforward layers.
Zeroth-Order Fine- Tame-TTnnng of LLMs. Zeroth-order (ZO) optimization methods enable gradient- free fine-tuning, ideal for scenarios where gradient access infeasible or computationally expensive. Me70 MMalladi et al.) 2023), Memory-Efficient Zeroth-Orda optimizer, shows that 70 fine-tuning can achieve memory efficiency similar to model inference with forward-pass paaaad-pass-only updates. MeZ0 uses Simultancous Perturbation Stochastic Approximation (SPSA) (Spall 1992) for gradient estima- tion, allowing updates without backpropagation. However, MeZO laces challenges in convergence speed due to high variance in ZO gradient estimates. Recent improvements to MeZO focus sparsity (Liu et al. 2024] Guo et al. 2024), variance reduction (Gautam et al.) (002)), low-rank structures (Yu
al. 2024b Chen et al. 2024 and Hessian information (Zhao al. 2024 Yu et al. 2024a).
3
METHODOL.OGY
3.1 PRELIMINARIES
LoRA. Low-Rank Adaptation of Large Language Models (Hu al. 2022) a classical fine-tuning method for large language models. By introducing trainable low-rank matrices, LoRA reduces the number of trainable parameters to less than 0.1% of the full model, decreases GPU memory usage by two-thirds, and introduces no additional inference latency. The study by Aghajanyan et al. (2020) demonstrates that pretrained models possess an extremely low intrinsic dimensionality, indicating that fine-tuning small number of parameters in a low-dimensional subspace can achieve performance comparable to hall-parameter fine-tuning. The core idea is that the parameter update AW with respect to the pretrained weight matrix Wo [ can be represented via a low-rank decomposition, i.c.,
Wo AW Wo BA, B B A x and min(m, n).
MeZO. Although traditional ZO methods can estimate gradients using only two forward passes, they suffer from the burden of additional memory consumption induced by random perturbations,
3

---

Published as a conference paper at ICLR 2026
k=16 RoBERTa-large
RoBERTa-large
100
88-
Accuracy (%)
SST-2
SST-5
SST-2
TREC
Zero-shot
MeZO
MeZO (prefix)
FT
FT (prefii
MeZo (LoRA)
Mezo-Adam
(LoRA)
LMAD (ours)
Figure 3: Experiments on RoBERTa-large. We report results for zero-shot, linear probing (LP), MeZO and its variants, our method LMAO, as well as full fine-tuning (FT), LoRA, and prefix-tuning. With k = 16, LMAO significantly outperforms LP and MeZO variants. At k = 512, LMAO continues to achieve strong performance. Detailed results are shown in Table
which have the same shape as the model parameters. To address this limitation, Memory-efficient Zeroth-Orde optimizer (MeZO) (Malladi et al., (023) utilizes the random seed trick to generate random perturbations sequentially on-the-fly, enabling in-place optimization without modifying the model architecture and achieving memory usage equivalent to inference during fine-tuning. Moreover, MeZO exhibits broad compatibility, supporting both full aull-parmetter fine-tuning and parameter-efficient techniques such as LoRA.
3.2 LOW-RANK AND MEMORY-EFFICIENT SEROTH-ORDDER ALTERNATING OPTIMIZATION
Although LoRA reduces trainable parameters by restricting updates to low-dimensional subspace, this constraint limits the representational capacity of fine-tuned models, often yielding suboptimal performance compared to full-parann parammeterr updates (Hu et al. 2022). In contrast, McZO avoids gradient computations but suffers from high- variance gradient estimates, leading slower convergence (Yu et al. 2024a). To leverage the strengths of both, we propose the Low-rank and Memory-efficine -oroth-order Alternating Optimization (LMAO) framework, which strategically alternates between these complementary paradigms enable efficient fine-tuning of LLMs under limited resources while maintaining strong performance.
Each LMAO training iteration consists of two phases: 1) Low-rank adaptation: Low-rank matrices A and are updated using standard gradient- based optimizers (e.g., Adam (Loshchilov Hutter 2017). SGD), involving forward pass and backward propagation compute exact gradients for the low-rank modules. 2) Zeroth-order optimization: Following the low-rank update, a memory- efficient zeroth-order step adjusts the base model parameters W using two perturbed forward passes to estimate gradients with respect to the global weights.
The full LMAO procedure summarized in Algorithm Each iteration performs three forward passes-ont one for the LoRA update and two perturbed passes for zeroth-order gradient estimation and single backward pass for LoRA gradients. This design ensures that global weight updates remain full- rank, substantially mitigating the feature loss inherent to purely low-rank or zeroth-orde methods.
4 THEORY
In this paper, we consider the following optimization problem:
min
C(W:A,
(I)
W.A.B
where W mm represents the model parameters, Rr E mm are low-rank matrices, and L denotes the loss function.
4

---

Published as a conference paper at ICLR 2026
Algorithm LMAO (Low-rank and Memory-efficient Zeroth-Order Alternating Optimization)
Input: Model Weight Parameters W, loss L. step budget T, perturbation scale E, batch size B, learning rate scheduler }
1: for 1, 1.... .T do
2:
Sample batch B C D and random seed s
3:
[A, B] LoRA_Update( W, B)
Forward pass and backpropagation
4:
<- PerturbParameters(WW (W, 8,
5:
L+ L(W, A, B)
Forward pass w/ positive perturbation
6:
W. <- PerturbParameters(M -2),))
7:
L L(W, A, B)
Forward pass w/ negative perturbation
8:
W < PartuurPrramttes (W, s,
9:
projected < (C (2==
10:
Reset random number generator with seed
for W, W do
12:
~ (0, 1)
13: W, - W - The projected_grad :z
14: end for
15: end for
16: function LoRA- A, B)
17: [A, B = [A, B] V ABC A;))
18: return [A.
19: end function
function PerturbParameeters(W....
21: Reset random number generator with seed s
for W, W do
~ (0, 1) 23: W, W; WW+E2
25: end for
26: return W
27:
end function
We briefly introduce the theoretical foundations of the LoRA and MeZO components of our algorithm I followed by theoretical analysis of their alternating optimization.
Firstly, we analyze the theoretical aspects based on the following assumptions.
Assumption 4.1 (Lipschitz smoothness). Let L(W; A, B) be a differentiable function, there exist constants Lw and LBA such that for any W1, W2 E Rmxx, and B, B2 E Rmmxn the following condition holds:
1. WWW V C(W A, B)1I & Lw [ W2ll
2. VVACC(W 41, B1) BAC(W; A2, B2)ll & T.BA [_BAI(BI, A1] [B2, Az]ll
3. VW AI, Bi) VwC(W; A2, B2) & Louaal || B2 1.
Assumption 4.2 (Expected smoothness (Khaled & Richtárik, 2020: Malinovsky et al. 2024 The second moment of the stochastic gradient satisfies
E IVLW; A, B)1B&] < 2a (L(W; A, L*) (WW B)
for some a, B > 0 and all W e Ruxn B E Buxr and A E Brxn
Part of LoRA. We begin by discussing the LoRA component. The update procedure is follows:
[B+1, [B,, т/ BBA A,, Bt).
(2)
where y) is the learning rate, and V CWW: A., B,) represents the gradient as expressed below:
V BBAC(W;; A, B,) [V A, B:), Ar.
First, under Algorithm I and based on Assumption follows.
we present the descent lemma for LMAO as

---

Published as a conference paper at ICLR 2026
Lemma 4.3 (Descent lemma). Let L be function satisfying Assumption 4.
Under the condition
?) < L/axs where Lmax = LBA Lw: Lqua). Then we have
L(W A B(+1) CW ; A., B.) 7 2/ V BAW A, B. VV A4t1, 1l+1)
Part of MeZO. The MeZ0 method is an efficient memory optimization approach that combines zeroth-order methods. Given a labeled dataset 'D and mini-batch M of size [M]. we introduce the classic zeroth-order gradient as follows.
Definition 4,4 (Zeroth Gradient). Given model with parameters W E mm and loss function L., SPSA approximates the gradient on minibatch as
CwC(W;A,B:M. f(W 82; A, M: -L(W ez; A, B;M)
2s
(3)
222 V A, M)),,
where E IR N(00, mxx) and - 0+ is the perturbation scale.
The update procedure of McZO is follows:
W++1 = W. '/ W L(W;; Att1, B++1;M)-
(4)
Definition 4.5. The SGD gradient estimate on a minibatch of size M has covariance Σ as
M
|M R [[VC(W A, B;M) L(W; A, B: VL(W; A, BYV" L(W; A, A,).
W L(W; A, B; M) be a statistic, we need to examine the unbiasedness of its corresponding estima- which leads to the following lemmas.
Lemma 4.6 (Unbiasedness). Based on Definition 4.4 for a mini-batch M, we have
Eg~N(0,1) W CWW; A, B; M)) Vw C(W: A, B)- Lemma 4.7 (Biased Estimate). Based on Definition 4.4 the gradient norm Me70 is
mn N E B)
E A,
where m, n is the size of parameters and N the number of z sampled in SPSA (Definition 4.4).
we optimize by alternately applying LoRA and McZO methods. To analyze the convergence, we require the following lemma:
Alternating Optimization. In Algorithm!] !
Lemma 4.8. Let L(W; A, B) be the fannoottt we have
LBA Linax
[VV B,) < 1
(,, B.)l IV Ar41, B++1) 111+)
Under Assumptions 4.1 and Assumption following Theorem:
4.2
the convergence of Algorithm I is guaranteed by the
Theorem 4.9. The iterates of Algorithm 7 with SGD satisfy
6(1 LBA (C Ao. Bo) L*)
min E [[IvC(W;; Art1. B.+1) [2]]
L
OS
Tmax
where the stepsize Mmma == max(1)a, ()) satisfy
N
1000N
Mmax
< min
(n N3ILBA + 202 Lw (mn + N-1
4
N
LBA' T(Na,LBA + azln (mn +N- --))
and is set according to the experûmental configuration as specified in expression
Timia
Tmax
1000

---

Published as a conference paper at ICLR 2026
Building the finite series construction technique introduced in (Malinvvsy al. 20224 for the theoretical analysis of LoRA, we incorporate SGD-based zeroth-order optimization and tangential scaling inequalities in alternating scheme (Lu et al., 2019). This leads to adaptive convergence analysis, with detailed proofs provided in the appendix.
5 EXPERIMENTS
In this section, we present comprehensive experiments on multiple fine-tuning tasks across models with different parameter scales, aiming to rigorously evaluate the performance of LMAO. The experimental setup is detailed as follows:
Models. The experiments are evaluated across multiple language models, including the medium-sized RoBERTaa-argee(Liu et al. 2019) and the large autoregressive Open Pre-trained Transformers (OPT) series (Zhang et al. 2022) comprising OPT-1.3B, OPT-2.7B, and OPT-6.7B.
Datasets. We conduct experiments across multiple fine-tuning tasks in GLUE (Wang et al. 2018).
SuperGLUE (Wang al., 2019) benchmark and other datasets. For RoBERTa-large,
conduct
experiments on SST-2 (Socher et al.
2013), SST-5 (Socher et al. al.||201 2013)) SNLI (Bowman et al. 2015
MNLI (Williams et al.; 2017), RTE (Dagan al. [205]] Haim al. [2006] Giampiccolo et al. 2007]
Bentivogli et al. 2009). and TREC VVorrhes & Tice|[2000 2000). For OPT-1.3B, we conduct experiments
on SST-2 (Socher et al. 2013), CB (De Marneffe et al. 20T9), BoolQ (Clark et al. 2019). WSC
[eevesqee et al. 2012). WIC (Pilchvar & Camacho-Collaan Collados 2018), MultiRC (Khashabi et al. 2018
COPA (Roemmele al., 2011). l). and ReCoRD (Zhang al. 2018).
Baselines,
The baselines compared in our experiments contain both gradient-free-based and gradient- based methods, including: Zero-shot, In-context learning, 3) Memory-efficieet zeroth-order
optimizer (MeZO), 4) Low-rank adaptation (LoRA), and 5) Fine-tuning (Full-parameter). The
available code for the baselines from https///ghttuh..oomm///ttttttttp
5.1 MEDIUM-SIZED MASKED LANGUAGE MODELS
We first conduct experiments using the masked language model RoBERTa-large. We compare the performance of the LMAO with other methods on sentiment analysis, natural language inference, and topic classification tasks. We adopt the classic approaches from previous studies, including few-shot and many-shot settings. For k 16 and 512, we sample & cases per class. uniform fine-tuning of 1K steps is applied. The results are summarized in Table land Figure
LMAO significantly outperforms Zero-shot, LP, MeZO, and its variants. Across all six tasks, LMAO consistently optimizes the pre-trained model and outperforms Zero-shot, LP, MeZO, and its various extensions. On several tasks, it even surpasses full fine-tuning (FT), with the remaining performance gaps within 0.5%.
LMAO approaches the performance full fine-tuning (FT) as the number samples increases, with at most 2 % gap. For k 16, LMAO consistently outperforms LoRA across all six tasks, and in most cases even surpasses FT. When increasing k, the performance gap between LMAO and FT narrows further, with some tasks showing less than 1% difference.
5.2 LARGE-SCALE AUTOREGRESSIN LANGUAGE MODELS
After achieving promising results with RoBERTa-large we extend the LMAO approach the OPT series models, including three scales with 1.3B, 2.7B, and 6.7B parameters. To systematically evaluate the performance of I.AO,, we select several tasks from the SuperGI.UF (Wang al.. 2019) benchmark, encompassing text classification, multiple-choice, and generation tasks. In the experiments, we compare LMAO with several state-of-the-ar methods, including LoRA, MeZO, ICL (In-context Learning), and Zero-shot learning. To assess the performance differences under identical training conditions, all experiments are conducted with consistent settings, with each model fine-tuned for 1K iterations. This setup allows for more accurate reflection of the strengths and weaknesses of each method. The results for DPT-1.3B are summarized in Table 2 and Figure 2 while additional results for larger models are provided in Appendix
7

---

Published as a conference paper at ICLR 2026
Table Experiments on RoBERTa-large (350M parameters). LP: Linear probing: 20, 20 (LoRA), and ZO (prefix); memory-ctficient ZO-SGD with full full-parameter tuning, LoRA, and prefix-tuning respectively; FT: fine-tuning with Adam. All reported numbers are averaged accuracy (standard deviation). Bold numbers indicate the best performance each task, while underlined numbers denote the second-best.
Task Type
SST-2
SST-5
SNLI
MNLI
RTE
TREC
sentiment
natural language inference
topic
Zero-shot
79.0
35.5
50.2
48.8
51.4
32.0
Gradient-free methods: k
16
LP
76.0 (2.8)
(1.9)
66.0 (2.7)
56.5 (2.5)
59.4 (5.3)
51.3 (5.5)
MeZO
90.5 (1.2)
45.5
68.5
58.7 (2.5)
(3.3)
76.9 (2.7)
MeZO (LoRA)
91.4 (0.9)
(1.6)
69.7 (6.0)
64.0 (2.5)
73.1 (6.5)
MeZO (prefix)
90.8 (1.7)
(2.0)
(2.5)
63.4 (1.8)
65.4
80.8 (3.6)
MeZO-AAam
90.4 (1.4)
(1.5
(2.7)
64.3 (0.8)²
59.2 (11.))
78.3 (1.4)
Gradient-basee methods: k = 16
FT
91.9 (1.8)
(1.9)
77.5 (2.6)
70.0 (2.3)
66.4 (7.2)
85.0 (2.5)
FT (LoRA)
91.4 (1.7)
46.7 (1.1)
74.9 (4.3)
67.7 (1.4)
66.1 (3.5)
82.7 (4.1)
FT (prefix)
91.9 (1.0)
47.7 (1.1)
77.2 (1.3)
66.5 (2.5)
66.6 (2.0)
85.7 (1.3)
ours: 16
LMAO (ours)
92.8 (1.2)
48.4 (2.3)
77.9 (1.7)
69.7 (2.8)
67.3 (4.9)
87.0 (3.4)
Gradient-free methods: k 512
LP
91.3 (0.5)
51.7 (0.5)
(1.0)
71.5 (1.1)
73.1 (1.5)
89.4 (0.5)
MeZO
93.3 (0.7)
53.2 (1.4)
(1.0)
78.3 (0.5)
(2.0)
94.3 (1.3)
MeZO (LoRA)
93.4 (0.4)
52.4 (0.8)
(0.0)
77.9 (0.6)
71.3
95.0 (0.7)
MeZO (prefix)
93.3 (0.1)
53.6 (0.5)
84.8 (1.1)
79.8 (1.2)
(0.8)
94.4 (0.7)
MeZO-Adam
93.3 (0.6)
53.9 (0.8)
85.3 (0.8)
79.6 (0.4)
79.2 (1.2)
95.1 (0.3)
Gradient-based methods:
512
FT
93.9 (0.7)
55.9 (0.9)
88.7 (0.8)
84.4 (0.8)
82.7 (1.4)
97.3 (0.2)
FT (LoRA)
94.2 (0.2)
55.3 (0.7)
88.3 (0.5)
83.9 (0.6)
83.2 (1.3)
97.0 (0.3)
FT (prefix)
93.7 (0.3)
54.6 (0.7)
88.3 (0.7)
83.3 (0.5)
82.5 (0.8)
97.4 (0.2)
ours: = 512
LMAO (ours)
93.7 (0.5)
56.1 (1.8)
87.9 (0.7)
84.3 (0.9)
83.4 (1.5)
97.3 (0.3)
Table 2: Experiments on OPT-1.3B (with 1000 examples). ICL: in-context learning; LP: linear probing: LoRA: Low-Rank Adaptation; LMAO: Low-rank and Memory-efficient Zeroth-Ordes Alternating Optimization. LMAO achieves the best performance on out of tasks and ranks second only to LoRA RTE, with marginal difference. Bold numbers indicate the best performance each task, while underlined numbers denote the second-bese best.
Tasks
Method
SST-2
RTE
CB
BoolQ
WSC
WIC
MultiRC
COPA
ReCoRD
classification
multiple choice
Zero-shol
53.6
53.1
39.3
46.5
43.3
57.5
44.7
75.0
71.3
ICL
74.4
52.9
53.6
54.2
54.5
51.8
45.4
70.3
70.5
MeZO
58.1
53.2
46.4
55.6
49.0
57.9
49.0
74.0
71.4
LoRA
92.2
61.1
67.9
61.4
53.2
58.4
56.3
74.7
72.0
LMAO (ours)
92.4
59.4
69.0
62.0
54.8
58.9
56.7
76.7
72.1
8

---

Published as a conference paper at ICLR 2026
LMAO combines efficiency with strong generalization capabilities. On the OPT-1.3B model, achieves superior performance across a range SuperGLUE tasks, consistently outperform- ing mainstream fine-tuning and zero-shot approaches, positioning it competitive alternative framework.
LMAO (ours) achieves the best performance the majority of tasks. LMA0 ranks first 7 out of 9 benchmarks, including substantial gains on SST-2, CB, BoolQ, COPA, and ReCoRD. On the remaining two tasks, its performance is competitive, highlighting the robustness and consistency of the approach.
LMAO also serves as a strong PEFT approach. While LoRA is widely adopted PEFT baseline with competitive performance, LMAO consistently outperforms it on 8 out of 9 tasks, establishing itself as a perffrrrfnn and promising PEFT paradigm.
LMAO effectively combines the advantages of zeroth-order optimization and low-rank adap- tation. Compared zero-shot, ICL, and the zeroth-order method MeZO, LMAO demonstrates consistently superior performance. While MeZO generally outperforms Zero-shot and ICL, it still lags behind LMAO-for instance, MeZO achieves comparable results on WiC (57.9% vs. 58.9%) and ReCoRD 11.4% vs. 72.1 %)), but exhibits notable gaps on other tasks.
5.3
ABLATION STUDY
We conduct comprehensive ablation studies systematically assess the effectiveness of our method. All ex- periments are performed on multiple benchmark tasks using the OPT-1.3B model. The ablation analysis exam- ines several key components, includ- ing:
OPT-4.3B: Accuracy Memory (per
100
80
Aosuray (%)
50
Memory
70
80
30
LMAO w/o LoRA: Freez- ing the low-rank LoRA blocks optimized by LMAO, and updating only the zero- order optimizer components.
50
40
Copa
MAD
MAD
LRRA
MAO wio Ma20 Mammory (per twwkk]
LMAO w/o MeZO: Freez- ing the zero-order optimizer components optimized by LMAO, and updating only the low-rank LoRA blocks.
Figure 4: Ablation study results.
Figure 4 presents the ablation results across several representative tasks. As shown, the full LMAO method consistently achieves substantially higher accuracy than its individual components, namely the low-rank LoRA module and the zeroth-order MeZO module. This indicates that the integration of these two update mechanisms enables LMAO more effectively capture task-relevant information and enhances the model's 's adaptation capability. The performance gains are observed across all evaluated datasets, further validating the effectiveness and robustness of the alternating optimization strategy adopted in LMAO.
Notably, despite combining two distinct update paradigms within unified optimization framework, LMAO does not incur higher peak memory consumption compared to using LoRA or MeZO alone. This indicates that the integration of low-rank adaptation and zeroth-order updates is designed without introducing redundant intermediate states or additional backward-pid -aas overhead. Consequently, LMAO achieves improved performance without adding memory overhead, maintaining a memory footprint comparable to its component methods. Overall, these findings highlight that LMA0 delivers superior task performance while preserving computational efficiency and resource usage, underscoring its practical value in cenorryconstrained training scenarios.
5.4 PARAMETER SENSITIVITY ANALYSIS

---

Published as a conference paper at ICLR 2026
In method, and are two key hy- perparameters: r determines the rank the trainable low-rank matrices, while scales the contribution of the adapter's out- put during fine-tuning To comprehensively analyze the sensitivity of parameters w.r.t. and a, we evaluate the RoBERTa-large model on the SST-2 dataset with varying values of and a. The results are provided in Figure 5 We can see that our method not sensitive to the settings of r and a. Therefore, in practical applications, we can prioritize smaller values of r to ensure fine- tuning of the model with few parameters as possible.
100
(%) Gammaay
50
0
Figure 5: Parameter sensitivity w.r.t hyperparameters r and in LMAO.
5.5 MEMORY UTILIZATION ANALYSIS
To examine the memory footprint of LMAO, we measure the peak GPU memory consump- tion subset of datasets (CB, SST-2, BoolQ, and COPA) and compare it against two rep- resentative baselines: the low-rank adaptation method LoRA and the zeroth-order optimiza- tion method MeZO. To ensure fair comparison, all methods are evaluated under identi- cal training configurations and hardware settings. The statistics are summarized in Table 3
As shown in Table 3 the peak memory consumption of LMAO never exceeds that of LoRA or MeZO, despite integrating both low-rank updates and zeroth-order opti- mization mechanisms. This indicates that LMAO does not introduce additional mem- ory burden during training and maintains memory footprint comparable to its com- ponent methods. Such efficiency is par- ticularly important in essurce--onssrraiee settings, demonstrating that LMAO can be deployed without increasing hardware re- quirements.
Table 3: Peak GPU Memory (GB) Usage on OPT- 1.3B (1000 Examples). LoRA: Low-Rank Adapta- tion; MeZO: Memory-Efficient Zeroth-Order Opti-
mizer; LMAO: Low-rank and Memory-efficient Zeroth-
Order Alternating Optimization.
Method
SST-2
CB
BoolQ
COPA
MeZO
7.90
10.99
8.75
7.70
LoRA
12.62
65.05
38.48
8.31
LMAO (ours)
12.62
65.05
23.08
8.31
More importantly, LMAO exhibits substantial performance gains across variety of tasks, consistently outperforming both LoRA and MeZO on classification and multiple-choice benchmarks. These results suggest that LMAO effectively leverages the complementary strengths of the two update strategies, enhancing the model's learning capability and adaptability while preserving computational efficiency. Overall, the findings strongly support the design philosophy of LMAO, showing that delivers notable performance improvements while remaining memory-efficient.
6 CONCLUSION
We introduce LMAO, a memory-efficient fine-tuning method for large language models that leverages full-rank updates. Unlike traditional low-rank adaptation (e.g.. LoRA) and zeroth-order optimization (e.g., McZO) (c.g techniques, which face inherent limitations, LMAO combines the strengths of both approaches by alternating low-rank matrix updates and zeroth- forward-par updates. This hybrid design significantly enhances model performance while effectively maintaining striet memory consumption, making it particularly suitable for large language models under resource constraints. Our theoretical analysis establishes rigorous convergence guarantees, ensuring the robustness and reliability of the method. Empirical experiments conducted on widely-used models such as RoBERTa- large and OPT demonstrate that LMAO delivers competitive performance compared to existing baseline methods. A remaining challenge lies in its training inefficiency large datasets long sequences. We propose that this issue could be addressed through optimization techniques such as component training, which remains important avenue for future research and development.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Dr. Zhaogeng Liu was supported by the Young Scientists Fund (C Class) of the National Natural Science Foundation of China under Grant No. 62506142. Dr. Yi Chang was supported by the National Key R&D Program of China under Grant No. 2023YFF0005400, and the National Natural Science Foundation of China through grant No. U2341229.
ETHICS STATEMENT
All participants in this work, well as the paper submission, adhere to the ICLR Code of Ethics httpss/cccccccpbbbcccoooooohsss
REPRODUCIBILITY STATEMENT
We affirm that the results of this work are fully reproducible. Appendix provvides the theoretical proofs. Appendix A details the experimental implementations. The source code is available at http:://gttbbb.coo/wwwolcl//////
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255, 2020.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint /508 05326, 2015.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.07669, 2024.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1905 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance- reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.08080 2024.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
11

---

Published as a conference paper at ICLR 2026
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastami, Christopher De Sa, Xiaodong Yu, Beidi Chen, and Zhaozhuo Xu. Zeroth-order fine- tuning of LLMs with extreme sparsity. In 2nd Workshop Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization (WANT@ICM) 2024), 2024. URL fttt:::://oppnrevieee net. for:/formm? wmm.nnddppp
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pp. 785-794, 2006.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
Ahmed Khaled and Peter Richtárik. Better theory for sgd in the nonconvex world. arXiv preprint arXiv:2002.03329 2020.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 252-262, 2018.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104 004941 2021.
Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. KR, 2012:13th, 2012.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv: 08838, 2018.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00 0000000,,0 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019.
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order Ilm fine-tuning. arXiv preprint 21Xiv22402.15551 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 2017.
Songtao Lu, Mingyi Hong, and Zhengdao Wang. Pa-gd: On the convergence of perturbed alternating gradient descent to second-order stationary points for structured nonconvex optimization. In International Conference on Machine Learning. pp. 4134 4143. PMLR, 2019.
Grigory Malinovsky, Umberto Michieli, Hasan Abed Al Kader Hammoud, Taha Ceritli, Hayder Elesedy, Mete Ozay, and Peter Richtárik. Randomized asymmetric chain lora: The first meaningful theoretical framework for low-rank adaptation. arXiv preprint arXiv:2410.08300 2024.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Dangi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53077 2023.
12

---

Published as a conference paper at ICLR 2026
Mohammad Taher Pilehvar and Jose Camacho-Colladee Wie: the word-in-contex: dataset for evaluating context-sensitive meaning representations. arXiv preprint arXiv: 09121, 2018.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alterna- tives: An evaluation of commonsense causal reasoning. In AAAI spring symposium: logical formalizations of commonsense reasoning, pp. 90-95, 2011.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pp. 1631-1642, 2013.
Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert- Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint urXiv:1908 190800908,,0 2019.
James C Spall. Multivariate stochastic approximation using a simultancous perturbation gradient approximation. IEEE transactions on automatic control, 37(3):332 341, 1992.
Ellen M Voorhees and Dawn Tice. Building question answering test collection. Proceedings of the 23rd annual international ACM SIGIR conference Research and development in information retrieval, pp. 200-207, 2000.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXivv1804.07461, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenn corpus for sentence understanding through inference. arXiv preprint 17X1::1004.0446, 2017.
Zhiyuan Yu, Yifei Cheng, Liang Ding, Xinmei Tian, Li Shen, and Dacheng Tao. Memory-efficient
block coordinate descent for hessian-informed zeroth-order optimizer, 2024a. URL https
//openreview.co wtt hott://oorrmmmi--0ggtttt0...
Ziming Yu, Pan Zhou, Sike Wang, Jia Li, and Hua Huang. Subzero: Random subspace zeroth-order optimization for memory-efficient Ilm fine-tuning. arXiv preprint arXiv:2410.08989, 2024b.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arviv: 1810. 12885, 2018.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint mrXiv:2055000666 2022.
Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-order fine-tuning without pain for 1lms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:240..1517, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXPERIMENT SETUP
A.1 DATASETS
For the RoBERTa- large experiments, we consider a set of standard classification benchmarks, includ-
ing SST-2 (Socher et al. 2013), SST-5 (Socher et al. 2013), MNLI (Williams et al., (017)) SNLI
(Bowman et al. 2015), and RTE (Dgaan al. 2005 Haim et al. 2006 Giampiccolo al. 2007
Bentivogli et al. 2009). To enable efficient evaluation, we follow the setting in (Malladi et al. 2023)
and limit the test set to 1000 examples. For training and validation, we adopt two configurations with k = 16 and k = 512, where l: denotes the number of examples per class for each split.
For the OPT experiments, we evaluate on the SuperGI.UE benchmark suite (Wagg al.] 2019). which comprises BoolQ (Clark et al., 2019), CB (De Marneffe et al. 2019),, COPA (Roemmele et al. 2011 MultiRC (Khashabi et al. 2018), ReCoRD (Zhang et al. 2018), RTE (Dagan al. 2005 Haim et al.
2006, Giampiccolo et al. 2007 Bentivogli al. 2009), WIC (Pilehvar & Camacho-Colladoo 2018
and WSC (Leeesqe et al, (012). In addition, we include SST-2 (Socher et al. 2013). For all tasks,
randomly sample 1000 examples for training, 500 for validation, and 1000 for testing.
A.2 PROMPTS
LMAO adopts the same zeroth-order technique as (Malladi et al. 2023), and therefore we follow their setup for the downstream tasks and prompt templates used to fine-tune RoBERTa-large. The specific templates are provided in Table
Table 4: The prompts used in our RoBERTa-large experiments (see Table I and Figure 2 follow the setup (Malladi et al., 2023), including template and set of label words that fill the [MASK] token. The symbols < S and < S₂ > denote the first and second input sentences, respectively.
Datasets
C
Type
Prompt
Label words
SST-2
2
sentiments cls.
S1 It was [MASK].
(great, terrible)
SST-5
5
sentiments cls.
S1 It was [MASK].
(great, good, okay, bad, terrible) (Description, Expression, Entity, Human, Location, Number] Yes, Maybe, No} Yes,, Maybe, No} {Yes, No}
TREC
6
topic cls.
[MASK]: < S1 >
MNLI
3
NLI
S1 >? [MASK], < <8
SNLI
3
NLI
< S1 >? [MASK], < 82 >
RTE
2
NLI
< S >? [MASK], < <
Table 5 presents the prompt templates used in our OPT experiments. Specifically, the OPT setup includes two types of tasks: classification and multiple-choice. All prompts follow the setup in (ocher et al. 2013
A.3 HYPERPARAMETERS
For experiments with the RoBERTa-large model, we follow the prompt-based fine-tuning paradigm for masked language models described in (Socher et al. 2013). In OPT experiments, adopt a similar training strategy to (Socher et al.) 2013) for classification tasks, where we extract logits corresponding to the label words and apply cross-entropy loss. For multiple-choice and generative tasks (e.g.. QA), we retain only the correct candidate and use teacher forcing on the correct examples. Loss is computed solely the tokens in the candidate portion, excluding the prompt.
The specific hyperparameter settings for RoBERTa-large and OPT are provided in Table 6 and Table respectively.
14

---

Published as a conference paper at ICLR 2026
Table We use prompt templates in our OPT model experiments, covering classification (cls.) and multiple-choice (mch.) tasks. Templates follow (Malladi et al. 2023). For multiple-choice, we score candidates by average log- oggllikelhhod and select the highest. For QA, we apply greedy decoding.
Dataset
Type
Prompt
SST-2 cls.
<text >It was terrible/great
RTE
cls.
preemisse >
Does this mean that "<hypothesis>" 'is true? Yes or No?
Yes/No/Maybe
CB
cls.
Suppose <premise>Can we infer that "chypothesis>"" '? Yes, No, or Maybe?
Yes/No/Maybe
BoolQ
cls.
<pssage>><qestiinn
Yes/No <lext> Does the word "<word> "have the same meaning in these two sentences? Yes, No?
WSC
cls.
Yes/No
WIC
cls.
Does the word '<word> 'have the same meaning in these two sentences? Yes, No?
<sentl>
<sent2>
Yes/No
MultiRC
cls.
<paragraph>
Question: <question>
found this answer <answer>>' Is that correct? Yes or No?
Yes/No
COPA
mch.
soremase <candidate'>
ReCoAD
mch.
<passage>
<query> @eplachholder'.' <candidate>>
Table 6: Main hyperparameter settings for all tasks (SST-2, RTE, CB, BoolQ, WSC, WIC, MultiRC, COPA, ReCoRD) under the RoBERTa-large model.
Method
Batch size
Learning A
Learning rale
Scaling factor
Rank
Perturbation scale
LMAO
24
LoRA
(4, 8. 16)
(lo-4, 30-4, 5o-4)
16
8
(LoRA)
{4. 8. 16)
[10-4 30-4. 50-4)
16
MeZO
64
(1e-7, 106, 1e:5)
le:3
Zero shot
64
B MORE EXPERIMENT RESULTS
B. PERFORMANCE ON LARGER MODELS
We scale up to the OPT-2.7B and OPT-6.7B model evaluate LMAO and compare it with several state-of-the-arr methods, including LoRA, MeZO, ICL (in-context learning), and zero-shot learning. All methods are trained with 1000 iterations under consistent settings. The detailed results are shown in Table and Table
Results show that LMAO performs strongly on OPT-2.7B, achieving superior results on six tasks (CB, BoolQ, WIC, MultiRC, COPA, ReCoRD). We further scale the model to OPT-667B 7B and evaluate algorithm performance against MeZO and LoRA on a subset of tasks. The results show that LMAO continues to perform excellently OPT-6.7B. achieving the best performance on four out of six tasks (CB, BoolQ, WIC, MultiRC, COPA, ReCoRD). Combined with the results on OPT-2.7B, this demonstrates LMAO's robustness and cross-task generalization, validating its effectiveness and adaptability on larger models. Thus, LMAO proves to be an efficient and memory cemmory-frienlly fine-tuning approach for large language models.
15

---

Published as conference paper at ICLR 2026
Table Hyperparameter settings for all datasets and corresponding methods under the OPT-1.3B model.
Tanks
Method
Hypergarameters
SST-2
RTF
CB
BoolQ
WSC
WIC
MulúRC
COPA
ReCoRD
Batch size
2
2
Learning rate 18
le-5
le.5
le.5
le.5
le:5
le:5
le.5
le-5
le-5
Learning rate vW
le-6
le-6
lo-6
le-6
le-6
Te-6
lo-6
lo-6
le-6
Scaling factor
16
16
16
16
16
16
16
16
16
Perturbation scall
le-3
le-3
Io-3
Io-3
1o-3
Io-3
Io-3
1o-3
lo-3
Batch size Learning rate r)
8
2
le-5
le-5
lo-5
lo-5
1o-5
lo-5
lo-5
le-5
lo-5
LoRA
Scaling
rank
8
8
8
8
Batch size Learning rale !)
8
2
2
MeZO
le:7
le:7
le:7
le:7
1e:7
1e:7
1e:7
1e-7
1e-7
Perturbation scale
lo-3
1o-3
1o-3
1o-3
10-3
1o-3
Zeno shot
Batch size
64
64
64
Batch size
16
16
16
16
Learning rate
le-5
le-5
le-5
/
le-5
Batch size Learning rate
le-5
le-5
le-5
le-5
Batch size
16
16
16
16
Learning rate
lc-5
Io-5
le-5
1o-5
Batch size
AdaLoRA
rank
8
Learning rate
Table 8: Results on OPT-2.7B (with 1,000 examples). ICL: in-context learning; MeZO: Memory- efficient Zeroth-Order optimizer; LoRA: Low-Rank Adaptation; LMAO: Low-rank and Memory- efficient zeroth-order Alternating Optimization. LMAO achieves the best performance across all six tasks. Bold values denote the best result per task.
Tasks
Method
CB
BoolQ
WIC
MultiRC
CОРA
ReCoRD
classification
multiple choice
Zero-shot
50.0
50.9
52.5
43.2
71.0
75.0
ICL
55.4
56.9
51.6
52.4
77.0
74.0
MeZO
56.5
59.5
52.4
48.2
73.0
74.9
LoRA
66.1
68.3
55.5
56.4
83.3
75.2
LMAO (ours)
68.5
68.5
57.3
57.5
85.3
75.3
This demonstrates LMAO's robustness and cross-task generalization, validating its effectiveness and adaptability larger models. LMAO thus serves efficient and memory- memory-friendly fine-tuning approach for large language models.
B.2 COMPARISON WITH MORE METHODS
To enable rigorous comparison with recent parameter-efffcient fine-tuning (PEFT) approaches, we augment our evaluation with additional peroth-ordes and LoRA variants. The experimental results summarized in Table 10 Based on Table 10, LMAO attains the best overall performance across
16

---

Published as a conference paper at ICLR 2026
Table 9: Results on OPT-6.7B (with 1,000 examples). MeZO: Memory-efficient Zeroth- optimizer, LoRA: Low-Rank Adaptation; LMAO: Low-rank and Memory-efficte zeroth-orde Alternating Optimization. LMA0 leads on seven of nine tasks. Bold values denote the best result per task.
Tasks
Method
CB
BoolQ
WIC classification
MultiRC
CОРA
ReCoRD multiple choice
McZO
50.0
44.2
52.7
49.9
80.3
78.1
LoRA
64.3
49.4
52.9
57.0
81.0
79.7
LMAO (ours)
66.1
65.1
50.0
57.2
81.3
77.6
Table 10: LMAO PEFT baselines OPT-1.3B (1000 examples). LOZO: Low-rank Zeroth- Order Stochastic Gradient Descent; SubZero: Random Subspace Zeroth-Order Optimization; DoRA: Dynamic Low-Rank Adaptation; AdaLoRA: Adaptive Budget Allocation for Parameter- Efficceett
Fine-Tuning: LMAO: Low-rank and Memory-efficient Zeroth-Order Alternating Optimization.
Method
SST-2
CB
BoolQ
CОРA
LOZO
86.0
72.0
62.4
69.0
SubZero
63.6
43.0
65.8
76.0
AdaLoRA
87.8
71.4
55.7
76.0
DoRA
87.3
69.4
60.6
75.0
LMAO (ours)
92.4
89.0
62.0
76.7
the four benchmark tasks. Except for isolated cases, it outperforms competing methods; even when baseline holds single-task advantage, LMAO's s aggregate superiority remains evident, indicating stronger overall effectiveness and robustness.
C CONVERGENCE ANALYSIS
C.1 SYMBOL SPECIFICATION
The notation used the derivation of the proofs is summarized in Table
C.2 PROOF LEMMA 4
Proof. Using the total differential formula, we have
V L(W; Aqi, B(n)
[W A411, B111),
Ww.(W, A4t1, B(t1)]
17

---

Published as a conference paper at ICLR 2026
Table 11: Table of notations.
Variable
Definition
The loss function.
L
The theoretical optimum of the objective function.
Low-rank matrix in g*x*.
B
Low-rank matrix in Rmxr..
W
The weight parameter matrix of pre-trained model. The learning rate of the low-rank module in Algorithm |
IBA
The learning rate of the zeroth-order module in Algorithm
LBA. Lw Lqua
The Lipschitz constant.
M
A minibatch sample in SPSA-bases sampling.
M
The number of samples M in minibatch.
VL
The update gradient from algorithmic sampling.
m.
The dimensionality of the pre-trate pre-trained model's parameter matrix.
N
The number of samples drawn in SPSA.
a
The smoothness constant in the expected smoothness assumption.
Sym(-)
The symmetrization function for tensors.
Tensor product.
Under Assumption 4.1 we have (descent lemma)
A
A1, Bi) BAACWW; A,, A, - [Be, Ar))
LBA
[ A+til [B, A.llE Vir,(W;; A411, B141) We)
2
Lw
||W141- - Woll3
2
C(W6; A4, B,) MBA BAC ( Bu-1)
M LBA VBAC(W;; A,, B,)
MBALW
A4t1, Bet1)
2
1BA (I) BBA A,, [I V C(W; A4+1, A
L(W;: A,, B,)
where (a) is true because of the update rule of gradient descent in each block and Assumption (b) we used TBA & Lmax and Lmax == max(LBAn Lw).
C.3 PROOF OF LEMMA 4.6
Proof.
Using the equation equation 3 we note that
(Vw. A, B:M)]
BEWN [22 V A, B;M)]
=WwL(W; A, B:
=VwL(W; A, M)
AW
18

---

Published as a conference paper at ICLR 2026
C.4 PROOF OF LEMMA 4.7
Proof. We simplify the notation V C(W;; Ar, Br; Mol, as (wr
vt)), then we
rewrite the expression as follows:
Ez-Nio. [IV L(W, A. B;
= -E, V C(W: A., BG; M)
W. Ar, Mi;
Σ (2 an) M.
Σ
E V V 92))]
[M
Specifically, if u and are two independent vectors, then have
Es,,, 12/2/u0 oj 23] uv²
when i * j and
B2, 2222ur2 2f E. [20 (u, 21)
3mn
Sym 1 )u, v)
+ 2
mn uf vl + mn + 2 uv
mn +
Therefore
E VT CWW, B;;M) WwW, A,, Bc; Mc;
Σ
N-1
2mn
N
Nmm 22)
(MM x [2 Vw V
C(W 12)))²
(( (V
(mmn 2)
Note that when (I1, 91) # (I2, v2). we have
(VV
Ae, AW, B;;M,,
and when (I1, (1) = (I22, V2). we have
(1 (22))
= CWW,; A., B.; M)) A., Bi; Σ
M
then we get
Σ
(22)))
M
(at (MM
Ww A,, B,) V. Ar,
Ar, At, B,)+ + E
Summing these terms yields
E[[VwL(W: Ad, B.; M)) (V: A., B;:
1 N(mn + 2)
(Gw-CWV: B. C(W,, Ar, Be) + IM
(II&CC(W::A,,B. B,)
[M tr( >)
N(mn+2 2)
From above, obtain
+ N - N
E W[I Ar, B;MMi11
W A., A..
19

---

Published as a conference paper at ICLR 2026
C.5 PROOF OF LEMMA 4.8
Proof. By the triangle inequality, we obtain
VW A.. Bill & V L(W;; B(41) VW,
+ Aw
By the Mean inequality, we have
Vw Ar, B.)l < 21 V w L(W; A Aw B.)
+ 2 w L(W;; A4t1, B(+1)l1
Combining the updata procedure 2 with Assumption
have
A,, B.) < 2L [| B+41] - [A,, B.]ll + V A4+1, B
22 qua MBA alllllV BA BB 2 L(W,; A 11+1)ll
For a fixed TBBA <
where Lmax == max(LBA Lw, Liqu)), the above expression reduces
Lmaxx
to:
Liua
A(W B.)l <
BBA A,, Bo)l + Vw L(W;; Ac-1, Be-1)11e)
L
Fixed I2aax > 21 gua-
then the gradient norm of Algorithm satisfies
IVC(W; Ar, B.)l
1/ C(W,; ([[ Ar, B + |V Ar, B.)
1
2
VBAC(W;; A,, B. + Lw A411. B411
L
A,, Bi)
qua
VBBA A, BUl + L(W,,
max
I
(IV + Amn A.
L2
Lequa
1
|
B
Linax
C.6 PROOF OF THEOREM 4.9
Proof. We start from L-smoothness and use equation equation 2 and equation 4
L(W A B1+1) & L(W; A,, B,) + ( C(W; A,11, B,41), We+1 Wt)
+ (VBAL(W;; A,, B.), ( BB,,1, A (B))
Lw (IIB A [B, A.11 + Tww
+ 2
2 || Wall3
L(We; Ar. B₂) 7/ (VwL(W, A441, B++1). V. L(W:; Att1, B+t1))
MAA (V (W, B.), C(W, B,))
LRA 2
+ "BA BBA A,, 13)
Lw 2 + 2 W' Acti, Br+ 11
20

---

Published as a conference paper at ICLR 2026
Taking conditional expectation and combining Lemma 4.6 Lemma 4.7 have
18 [[ Ac+1. B.= Ar, B:)]
C(W; Ar, B,) -
A 1++1) MBA V BAA,WW B,) 2 IBA || V A., 1. Lw T/ Bl A4t1, 1
LBA
L(W; Ar, At, Bi) Mmin (1 w L(W; A4t1, IV (.)
L LBA Mmaxx Bill BAI A,, B.
2
Lw mn N
Flll W W L(W;; A B
Vmax N
Subtracting L from both sides of the inequality and using Assumption 4.2 have
[[[(W++1: A4+1, B++1)((W; A,, B() - L
C(W;; B.) C" Mmiin
(WW;: (WW. B.)
LBA Mmaxx [201 (E(W;; AG, Bi) - L*) + 31 V BAC A, B.
+
2
Lw mn + N
[aa2( A411, B(+1) 2*) + 3 V L(W;; A411, B1+1)ll
Umax
N
Using the Assumpiton
4.1
have
L(W;; A411, B141)
L = L(We; Act1, B(41) - L(W; A,, B₂) + L(W;; A,, B₂) L (VnAC(W;; Ar, B.), (lB++1, A4+1] [Br, Ar)))
LBA
[| A A++1] [B. A. 11 + L(W,; A., B:) L'.
2
Define 8141 E B[C(W[[[W:W]L] A441 B(+1) have
BB, Ae]] L and combining the above equations, then
se+1 & 8 7min
1min ( (l A B(+1) F BAW, A, B,)
= Log 5c + _ B 31 V BA <(W: A, B.)l
2
mn + - (L(W; A2+1, B2+1) - L(W;; A,, B,)) N
+oz Daax
mn + N
+ +agL M
6, + 2
B2
Lw n V A4+1, B(+1)
N
mn + N
+ an Ln
+ az Lw
max
6.
N
Muiin
mn + N - 2 LeLW Tmax
1
|/ W C(W;; Art1, B.+1)l1.
-]aax
7max
N
Qmax
Muin
2 L BA 31 - 012 L
mn + N
(7mmin +
IBBA
Mmax
Tmnax
N
2
x B A., B.)
Combining the above equations, we have
2000 Mmax ((W A B1+1)112 + BIVA Ac, B,)l
mn +
1 + LI + 1 m
6; 8
N
where 77 satisfies
Mmin
B2Lw Trax
mn + N
7max
2
N
2000
Mmni
I -oggTa Tonaxx Mmnx
Omin
LBA
Mmax
1
7max
2
2000
LBA n < 7max
Muin
+
2
21

---

Published as a conference paper at ICLR 2026
L..
1000N
Mhux &
min
100002Lw (mn + N - 1)' NBILBA + Lw (mn + N 1) LBA
Let fix p-1 > 0 and define Pt = P--1 (1+ (1+01 + a2 Lw mm + N -
for all
N
t > 0. Multiplying by Pr
we have
Tmnax
Pe (IVW A B VV Ar, B.)lii) & (pe-1de -vpd011-
Summing up both sides as 0,1,2, , 1, we have
Σ 2
B++1) + WRaaL(W AG, B.)
E (p:- 11)(+1) & p-100
T
Define Pr
Σ
Pt and we have
T-1 Pr
Pt
2
min Po-P1. ".. PPr-- = Tp
1=
t=
Tp-1
mnn N T.
1 LBAn 0z LaL
Dividing both sides by Pr we have
min
Ac+1, D I B.
OS
T-1
Pr - B.112)
P- 100
7max Pr
Using the fact that + x & exp(x), we have
a1 + Lw
mn + - 1
N
< expT (aILBAVimax + az Lww < exx((1) $3,
mn + N
N
where the second inequality holds because 7max & /// WT((Na) + +22 (mn + N- - 1))) by assumption. Combining the above equations, we have
3(L(Wo; An, Bo) L*)
min 061ET-1
V L(W;; A41. B4t1)l BV A,, BOll
Muax T
By the Lemma 4.8 we have
| L(We; A Br 1)
LBA
2
1+
([V BJIL L(W:: B++1)l1
Lmax
22

---

Published as a conference paper at ICLR 2026
Taking expectations and minimizing over t on both sides of the inequality, then we have
min E Al Ae, Boll 061
LBA
[B[IIBA Ar, BO]l - VW Ac+1. Be+t =
min 2 1 + 05 -1
Limax
LBA (C Ao, Bo) L*)
6(1
T.max
Mmax T
D USE OF LLMS
LLMs are used for grammar checking and language polishing.
23


Reviews:
[{"review_id": "fdFz5rrR4p", "reviewer": "Reviewer_sMQJ", "summary": "LMAO introduces a novel alternating optimization framework for fine-tuning large language models, combining LoRA's low-rank efficiency with MeZO's memory-friendly zeroth-order updates. By performing three forward passes and one backward pass per iteration, it enables full-rank adaptations while minimizing memory use and feature loss, achieving convergence guarantees and superior performance over baselines like LoRA and MeZO on tasks such as SuperGLUE with models like OPT and RoBERTa.", "questions": "1) the MeZO baseline results in Table 2 on certain datasets seem unusually low—for instance, on SST-2 and CB—which raises questions about their reliability and might suggest inconsistencies in implementation or evaluation setup[1][2][3].\n\n2) The experiments fall short by missing key baselines; they should include comparisons with advanced zeroth-order optimization algorithms designed to mitigate MeZO's high variance issues, such as SubZero[2] and LoZO[3]. Additionally, contrasts with LoRA variants that emulate full-rank updates—like DoRA[4]—would be valuable. Overall, the evaluations need a more comprehensive analysis that balances fine-tuning performance against memory usage and computational time overhead.\n\n[1] Zhang, Yihua, et al. \"Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark.\" International Conference on Machine Learning. 2024.\n\n[2] Yu, Ziming, et al. \"Zeroth-order fine-tuning of llms in random subspaces.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2025. \n\n[3] Chen, Yiming, et al. \"Enhancing Zeroth-order Fine-tuning for Language Models with Low-rank Structures.\" The Thirteenth International Conference on Learning Representations.\n\n[4] Liu, Shih-Yang, et al. \"Dora: Weight-decomposed low-rank adaptation.\" Forty-first International Conference on Machine Learning. 2024.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This paper features solid writing with a clear, logical structure and concise language, allowing quick understanding of innovations and results. The theory is rigorous, supported by convergence proofs and rate analyses under assumptions like Lipschitz smoothness for reliability. Experiments are thorough, covering models like RoBERTa and OPT series across tasks like SuperGLUE, including few-shot setups, ablations, sensitivity analyses, and comparisons to baselines such as LoRA and MeZO.", "weaknesses": "While the paper is innovative, it has some drawbacks. Though it targets resource-constrained fine-tuning, LMAO doesn't match MeZO's ultra-low, inference-level memory efficiency due to the required backward pass, which adds noticeable memory overhead. It emphasizes full-rank updates for improved expressiveness, but the performance gains over LoRA are often marginal(see Ablation Study), making the benefits feel underwhelming. Plus, the extra forward passes introduce significant computational costs, potentially making it inefficient for long sequences or large datasets, and rendering it somewhat niche or even a bit of a half-measure compared to simpler options like LoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VONLIu5uYg", "reviewer": "Reviewer_6rvB", "summary": "This paper introduces LMAO (Low-rank and Memory-efficient Zeroth-Order Alternating Optimization), a novel fine-tuning method for large language models that combines the strengths of LoRA and MeZO in an alternating optimization framework. LMAO performs gradient-based updates on low-rank LoRA matrices using one forward and one backward pass, followed by memory-efficient zeroth-order updates on full-rank base model weights using two additional forward passes. This \"three forward, one backward\" approach ensures base model parameters receive full-rank updates, overcoming LoRA's primary limitation while maintaining memory efficiency. The paper provides theoretical convergence analysis for this alternating scheme and demonstrates through extensive experiments on RoBERTa-large and OPT models up to 6.7B parameters that LMAO consistently outperforms LoRA and MeZO, achieving performance competitive with and sometimes superior to full fine-tuning.", "questions": "1.  Could the authors provide a quantitative comparison of the wall-clock training time for LMAO against the baselines (especially LoRA and FT) for a representative task? For example, reporting the time to complete 1K training steps for OPT-1.3B on one of the SuperGLUE tasks would be very informative for understanding the practical computational trade-offs.\n2.  To further strengthen the claims, could the authors provide a table or figure detailing the peak GPU memory consumption (e.g., in GB) of LMAO compared to FT, LoRA, and MeZO under an identical experimental setup (e.g., same model, batch size, and task)?\n3.  LMAO uses separate learning rates for the ZO update ($\\eta_W$) and the LoRA update ($\\eta_{BA}$). Could the authors comment on the sensitivity of the model's performance to the ratio of these two learning rates? Are there any heuristics or best practices you discovered for setting them?\n4. In Algorithm 1 Line 9, the step computes a projection and then updates each parameter by that scalar times its corresponding perturbation entry z. That is a rank-1 direction in the vectorized parameter space (one random direction), not a full-rank update. The paper repeatedly asserts “ensuring full-rank updates,” which is incorrect per-iteration; at best, many different random directions across iterations can span the full space in expectation. Please fix the claim and its implications (e.g., “reduces feature loss”) or use multiple independent directions per step.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The main strength is the creative combination of LoRA and MeZO. It uses the precise gradients from backpropagation for the low-rank part while leveraging the memory efficiency of ZO for the full-rank part, achieving the best of both worlds.\n* The method's effectiveness is demonstrated on models of increasing scale, from RoBERTa-large (350M) up to OPT-6.7B. This suggests the approach is a scalable solution for future, even larger models.\n* The authors provide a formal convergence analysis for their algorithm. This adds a layer of theoretical rigor that is often missing in purely empirical papers and builds confidence in the method's stability.", "weaknesses": "* The most significant weakness is the computational cost. The \"three forward, one backward\" approach inherently requires more computation per iteration than LoRA (one forward, one backward) or MeZO (two forward). The paper acknowledges this as \"training inefficiency\" in the conclusion but fails to provide a quantitative analysis of the wall-clock training time. This is a crucial piece of information for practitioners to assess the practical trade-offs.\n*  While the method is motivated by memory efficiency, the paper does not present direct measurements of peak GPU memory usage during training. A table or plot comparing the memory consumption of LMAO against FT, LoRA, and MeZO would make the \"memory-efficient\" claim much more concrete and impactful.\n*  By combining two different optimizers, LMAO introduces more hyperparameters that require tuning (e.g., learning rate for LoRA $\\eta_{BA}$, learning rate for ZO $\\eta_W$, perturbation scale $\\epsilon$, LoRA rank $r$ and alpha). While a sensitivity analysis for $r$ and $\\alpha$ is included, a more in-depth discussion on the tuning strategy and the sensitivity to the relative learning rates would be beneficial for the method's practical adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fCM2XTITuK", "reviewer": "Reviewer_N9p4", "summary": "This paper introduces a novel PEFT method for large language models. The proposed approach integrates the advantages of LoRA and MeZO by alternating between low-rank gradient updates and zeroth-order updates. Specifically, LMAO performs three forward passes and one backward pass per iteration, enabling full-rank parameter updates under stringent memory constraints. Overall, the paper presents a reasonably innovative idea; however, it would benefit from a broader set of experiments to better demonstrate its generalization capability across different model architectures. Moreover, I would like to see its performance on MMMU.", "questions": "[1]. The performance on multimodality settings?\n\n[2]. The performance on DyLoRA or AdaLoRA?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "[1]. The authors provide open-source code and describe the experimental setup thoroughly, which enhances the transparency and reproducibility of the work.\n\n[2]. The proposed method, LMAO, demonstrates strong empirical performance, outperforming or matching full fine-tuning on multiple NLP benchmarks, showing both effectiveness and efficiency.\n\n[3]. The paper is clearly written, well-organized, and easy to follow. The motivation, methodology, and results are presented in a logical and coherent manner.", "weaknesses": "[1]. The method reduces memory usage during the zeroth-order phase, but the backward pass for low-rank updates still incurs considerable memory cost. This may limit scalability when applied to very large models or long-sequence tasks.\n\n[2]. Although the paper claims good scalability, experiments are only conducted on models up to OPT-2.7B. Evaluating larger models such as LLaMA 3 (13B or 70B) or Qwen2.5 would better support the scalability claim.\n\n[3]. The experiments focus only on text classification and inference tasks. It remains unclear whether the method generalizes well to generative or multimodal settings such as MMMU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1KSR6kBvdv", "reviewer": "Reviewer_9oXR", "summary": "Large language models have shown remarkable performance but fine-tuning them is computationally and memory-intensive due to growing model sizes. The motivation is to develop scalable fine-tuning methods that reduce resource demands while maintaining high performance on downstream tasks. Challenges include LoRA's suboptimal performance from low-rank constraints and MeZO's high variance and slow convergence from zeroth-order gradient estimates. The solution, LMAO, alternates low-rank matrix updates using backpropagation with zeroth-order updates on base parameters via forward passes, enabling full-rank fine-tuning with three forwards and one backward per iteration under memory constraints.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. LMAO integrates the low-rank efficiency of LoRA with the memory-saving forward-only updates of MeZO. This hybrid approach achieves full-rank parameter updates without excessive memory use. It effectively reduces feature loss common in low-rank methods while controlling variance better than pure zeroth-order techniques.\n\n2. Experimental results on RoBERTa-large demonstrate that LMAO outperforms baselines in few-shot and many-shot settings. It achieves accuracies close to or surpassing full fine-tuning on tasks like SST-2 and SNLI. This highlights its ability to optimize medium-sized models effectively across sentiment and inference tasks.\n\n3. On larger OPT models up to 6.7B parameters, LMAO consistently ranks highest on most SuperGLUE tasks. It improves over LoRA on classification and multiple-choice problems like BoolQ and COPA. Such performance validates its scalability and generalization to autoregressive architectures.\n\n4. The theoretical framework provides convergence guarantees under Lipschitz and expected smoothness assumptions. Descent lemmas and rate analyses ensure reliable optimization. This mathematical rigor strengthens the method's foundation beyond empirical evidence.", "weaknesses": "1. LMAO requires more forward passes than standard LoRA, increasing computational overhead. This can extend training time significantly on large datasets. Efficiency gains in memory do not fully offset the added flops.\n\n2. Zeroth-order components introduce inherent noise despite alternation. This variance may hinder convergence on noisy or complex tasks. Smoother gradients from full backpropagation are absent in parts of the update.\n\n3. Evaluations are limited to models up to 6.7B parameters. And the utilized evaluation datasets are outdated. Scaling issues could emerge with extreme sizes.\n\n4. Comparisons omit advanced PEFT variants like sparse adapters. Only basic LoRA and MeZO are baselines. Broader benchmarking would clarify relative advantages.\n\n5. Inference implications are undiscussed after training. Merging adapters into base weights could add latency. Deployment considerations are overlooked.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)