Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
Authors: Qitao Tan
Jun Liu
Zheng Zhan
Caiwen Ding
Yanzhi Wang
Xiaolong Ma
Jaewoo Lee
Jin Lu
Geng Yuan
Venue: NeurIPS 2025 poster ()
Keywords: Memory-efficient fine-tuning; Zeroth-order optimization
Primary area: deep_learning

Abstract: Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recently, zeroth-order (ZO) optimization stood out as a promising memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource-constrained scenarios. However, ZO method lags far behind FO method in both convergence speed and accuracy. To bridge the gap, we introduce a novel layer-wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. Aiming to resemble the learning capacity of FO method from the findings, we propose \textbf{Di}vergence-driven \textbf{Z}eroth-\textbf{O}rder (\textbf{DiZO}) optimization. DiZO conducts divergence-driven layer adaptation by incorporating projections to ZO updates, generating diverse-magnitude updates precisely scaled to layer-wise individual optimization needs. Our results demonstrate that DiZO significantly reduces the needed iterations for convergence without sacrificing throughput, cutting training GPU hours by up to 48\% on various datasets. Moreover, DiZO consistently outperforms the representative ZO baselines in fine-tuning RoBERTa-large, OPT-series, and Llama-series on downstream tasks and, in some cases, even surpasses memory-intensive FO fine-tuning. Our code is released at \url{https://github.com/Skilteee/DiZO}.

Harmony in Divergence: Towards Fast, Accurate, and Memory-efficien Zeroth-order LLM Fine-tuning
Qitao Tan'
Jun Liuè Xiaolong Ma* University of Georgia 3 University of Minnesota
Zheng Zhan
Caiwen Ding
Yanzhi Wang²
Jaewoo Leel
Jin Lu
Geng Yuan'
Northeasteenn University
Univeesiiyy of Arizona
Abstract
Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recently, zeroth-order (ZO) optimization stood out as a promis- ing memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource- constrained scenarios. However, ZO method lags far behind FO method in both convergence speed and accuracy. To bridge the gap, we introduce a novel layer- wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. Aiming to resemble the learning capacity of FO method from the findings, we propose Divergence-driven Zeroth-Ordes (DiZO) optimization. DiZO conducts divergence-driver layer adaptation by incorporating projections ZO updates, generating magene-- updates precisely scaled layer- wise individual optimization needs. Our results demonstrate that DiZO signifi- cantly reduces the needed iterations for convergence without sacrificing through- put, cutting training GPU hours by up to 48% on various datasets. Moreover, DiZO consistently outperforms the representative ZO baselines in fine-tuning RoBERTa-large, OPT-series, and Llama-series on downstream tasks and, in some cases, even surpasses memory-intensive FO fine-tuning. Our code is released https::/gii gigghu........ wtt..//ilieeeee/aee..a
Introduction
Fine-tuning large language models (LLMs) via backpropagation achieves strong performance across many NLP tasks 22 4 5. but their large parameter counts create substantial memory burdens, limiting downstream applicability. Following neural scaling laws [6] 77, next-generation LLMs grow rapidly, e.g., model sizes increase 410x every years, far outpacing DRAM bandwidth (1.4x) and capacity (2x) growth. This imbalance leads to the memory 8), a growing challenge especially for deployment on memory- -imited devices [10 [.].
Zeroth-order (ZO) optimization has recently emerged as a memory-efficient approach for LLM 14 15). By relying solely on forward passes for gradient estimation, ZO eliminates the need for backpropagation and significantly reduces memory usage for activations, gradients, and optimizer states. As shown in [14), ZO fine-tuning reduce memory cost by up to 12x. However, ZO still shows a notable gap in convergence speed and accuracy compared to first-order (FO) methods, shown in Table Although ZO benefits from higher throughput due to its simpler computation, it requires over 10x more iterations to converge, ultimately increasing GPU time. Prior work often attributes this gap noisy gradient estimates,
fine-tuning, gaining growing attention [22 13
without exploring other contributing factors [14 16 15
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To further uncover the fundamental causes of this gap, we begin by analyzing the distinct update patterns shown by FO and 70 meth- ods during L.M fine-tuning. Interestingly, our analysis reveals a substantial difference in the magnitude of weight updates between lay- ers. Specifically, FO methods benefit from fine-grained gradient estimation and enable diverse-magnitune magnitude updates precisely scaled to the wise individual optimization needs. In contrast, ZO method tends to behave with uniform-magnitude updates without consider- ing layer-wise individual characteristics. This is attributed to the nature of ZO that relies on high-dimensional random search and leverages random perturbation for gradient estimation. Based this, we conjecture that the compromised performance of ZO stems from its limited capability in achieving magente- updates. This naturally raises the question: if we could enable ZO achieve the desired diverse-magntude updates, could we effectively achieve training acceleration and accuracy improvement?
Table 1: Fine-tuning results on SST-2 datasets. Although ZO method shows advantages in memory saving, left behind FO method in terms of both accuracy and GPU hours.
GPU
Model
Type
Acc.
Memory
Hours
FO
91.9
9.2 GB
12.3% 100.0%
RoBERTa
70
90.5
4,5 GB
FO
94.2
45.4 GB
16.8%
OPT-2.7B
ZO
90.0
6,8 GB
100.0%
To validate conjecture and fill the performance gap, innovatively propose Divergence-drin driven Zeroth- Ordeer optimization (DiZO), which performs divergence- layer adaptation via anchor- based learnable projections, enabling principled adaptive updates that resemble FO methods. Specifi- cally, DiZO guides updates along geometrically constrained directions by learning target distances from an anchor point (e.g., the pre-trained model). We also design ZO-based method for projection learning that ensures the entire training process is memory-efficient We extensively evaluate DiZO on range of tasks, including classification and generation, using models such as RoBERTa-large the OPT series, and the Llama series. Results show that DiZO significantly reduces training iterations and cuts GPU hours by up 48% without sacrificing throughput. DiZO also integrates seamlessly with parameter-efficieet tuning methods like LoRA [17], and consistently outperforms ZO baselines, sometimes even surpassing wemory-intensiv FO fine-tune savettuning. Finally, we comprehensively analyze several potential alternatives and validate the necessity and effectiveness of our approach.
The summary of contributions is follows:
We introduce a novel layer-wise divergence analysis to uncover the fundamental differences in the updating patterns of FO and ZO methods.
We introduce DiZO, novel ZO method using divergence-driven layer adaptation, achieving a learning capacity closely resembling FO while maintaining the throughput benefit.
DiZO consistently exceeds cxisting baselines in both accuracy and GPU hours, and it can be scamlessly integrated with LoRA for additional benefits. These advantages hold across diverse tasks and LLM architectures.
We also provide comprehensive analysis and discussions on overheads, convergence guar- antee, and potential alternatives, which further strengthen the efficiency, feasibility, and necessity of our proposed approach.
2 Preliminaries and Pattern Analysis
2.1 Revisiting Zeroth-order Optimization
Recently, ZO optimization has gained significant attention in machine learning [8 19 20 21). Unlike conventional FO optimization, which calculates gradients via backpropagation, ZO optimization estimates gradients using only objective oracles via finite differences [22 can be leveraged for LLM fine-tuning to alleviate the extensive memory costs. Specifically, as ZO only needs two forward passes obtain the estimated gradients, it avoids computing and storing the most memory-come connumin information needed in the conventional FO training, i.c., activations in the forward process, gradients in the backward process, and the optimizer state.
23
24
This property
The core idea of 70 optimization is to estimate gradients by applying random perturbations to the weights and computing differences in the objective. For mini-bate of data B, sampled from

---

Irer100: 62.0
Iter2000:: 90.0
Tee440: 93.0
Wy
Wh:
Wgg
Ml
We
Distance Gap Idda
TterS0: 68.0
Tter1000: 85.0
Iter20: MX.2
PD pdaat
Leifeere Maguitade
Birase
1/3
Wl
Wg 11000- Wi
We
MG
Iterd: 5&8 (Ase)
Iterd:
W3
My
W6
Wy
Wy
Wo
Wo
MeZO layer-wise update pattern
PO layer-wise update pattern
FO
DiZo
Ditancee Gap (100
0.15
0.1
0.00
09
2556
50%
75%
100%
0%
25.55
500
75%
300%
0%
255
50%5
75%
100%
Epochs 195)
Epochs (96)
Figure 1: Comparison of the training dynamics of ZO and FO methods. For the upper subfigure, WK. Wy, Wo, Wo indicate the corresponding weight matrix in the attention module.
labeled dataset D = { Ya W model with parameters 0 E Rd where d represents the dimension of the parameter space, and the corresponding loss function L(0; B). The gradient is estimated as:
L (0 + eu;; B) f (0 eu;;; B) U
VC q
(1)
where u, ~ (O, is a random perturbation typically drawn from standard Gaussian distribution, the number of queries, and > 0 is small perturbation scalar for smoothing.
Given the learning rate and the mini-batch data B, at t-th iteration, once the estimated gradient VL(0; B,) is obtained, then ZO-SGD updates the parameters with the following rule:
0,+1 0: LV
2.2 Layer-wise Divergence Analysis
70 optimization applies uniform-magnitude updates across layers, with similar 1.2-norms per iteration (see Appendix|B [F). This uniformity may underlie its weaker performance. To explore how update divergence impacts convergence and accuracy, analyze the training dynamics of ZO and FO methods.
Analysis indicator. To quantify the effect of updates, we adapt the layer-wise L2-norm distance gap between the weights of the pre-trained and the weights of fine-tuned model as an indicator. The layer-wise L2-nomn distance gap is defined
| !!0!0) 0
(3)
where is 1-th fine-tuning iteration, is b-th layer of the model, and 0 indicates the weights of E-th layer of pre-trained model.
Analysis result. Figure [ compares the training dynamics of FO and 70 optimization methods, focusing on how each shapes the layer-wise distance gap between model parameters and the pre- trained initialization. Both methods show increasing divergence among layers over time, as illustrated by the twisting lines in the upper sub-figure, indicating that layers benefit from different levels of deviation from the pre-traines model. However, FO and ZO differ in how this divergence accumulates. FO uses fine-gre -reingriidd gradients produce fliverse-magnituee updates, quickly establishing meaningful distance gaps within a few iterations. In contrast, ZO performs random search with uniform-man -aggnitude updates, requiring thousands of iterations to reach similar divergence. Additionally, FO-based methods show fast but converging growth in distance gaps, while 20-based methods exhibit linear, unconstrained growth. This continued expansion under 70 may reflect lack of effective constraint, contributing to accuracy drops in later stages and resulting in suboptimal performance.
3

---

Algorithm 1: Divergence-diven ZO Optimization (DizO)
Require: parameter of f-th iteration and pre-trained model 0., loss function step budget 7, perturbation scalar mini-bore mini-batch data B,, learning rate 1). projection at t-th iteration 7 {1))
projection update interval for to do
V. = GradEst (0,, e, e,
0, 1t-1- VVL
f mod
then
7: arg minly L(o, [_P,]
0. = 4pplyProjection(0, 00. 7)):
end
end
10 Subroutine GradEst (0, 6, BB)
Sample: W1..... 11g (0, I);
Query: 1/1 L( eu;; B) C(0 eu;; B);
13
Estimator: VL
yeul;
14
return VL;
15 return
16 Subroutine ApplyProoection(0, 0o. 7:):
17
for = 1......, I. do
// Project 1-th layer
0000 00 100 A
18
19
end
29
return Oc:
return
3 Methodology
3.1 Design of the Divergence-driven Layer Adaptation
To provide layer-wise adaptive updates for ZO optimization, we propose applying anchor-based learnable projections to the updates of different layers. The pseudocode for the method is shown in Algorithm
Specifically, we treat training iteration as two-step process that iteratively updates the weights and the projection factor. Our approach involves two key steps performed in an alternating manner. First, we perform vanilla ZO optimization as defined in Eq. (2). Second, we identify the ideal projections for the weights and apply them, generating the projected weights. Formally, we define the ideal projection learning as solving the following minimization problem:
Y
min L(O, + 71
A0;
|A0,||
1 where 7. (0) Jz=1 is a projection vector t-th iteration, and the number of layers. While searching for the ideal projection, we freeze the model weights and use the same mini-batch data B, that is employed for the main ZO weight fine-tuning.
After finding the ideal projection for the t-th ZO step, we project the weights as:
0. 0u +
| A0,
Y
where get the new 0, after projection, and then fine-tuning. When the value of 7/8 is larger than || enlarges the distance gap between the fine-tuned model and the pre-trained model, and vice versa.
the projected one for the following
3.2 How to Learn the Projection?
Although promising, finding the ideal projection (defined in Eq. remains challenging due to the high complexity of the objective. A straightforward solution is to also perform backpropagation for

---

gradient computation and optimize the projection accordingly (FO-based method). For example, we use Adam optimizer to directly update Ye- The results are shown Table [ which significantly reduces 67.7% of the iterations and 58.5% of GPU hours, and increases by 3.4% in accuracy.
However, searching projection with the FO method makes DiZO only partially gradient-free. Specifically, while the model weights are updated via ZO, the per-layer projection pa- rameter 7/e is updated via FO, which still requires the backward pass and storing memory-intensive activation. The only memory saving, compared to the vanilla FO fine-tuning, is the optimizer state. As a result, relying on FO to find the ideal projection, though it achieves faster convergence speed and better accuracy in ZO optimization. offers limited overall benefit. It is worth noting that the peak memory usage during training of the FO-based DiZO is similar that of low-rank adaptation (LoRA) 17
Table 2: Fine-tuning OPT-2.7B on SST-2 dataset. partial
gradient-free; DiZO': : learning projection by FO method;
Gradient Free
#Train Iter.
GPU
Task Type
Acc.
Hours
Mc70
/
90.0
100%
100%
DiZO' (w. FO)
93.4
33.3%
41.5%
FT
X
94.2
9.3%
16.8%
3.3 Projection Learning by Zeroth-order Optimization
Our major goal is to find the ideal projection for adaptive updates while avoiding memory-intensive backpropagation. One potential promising solution is to also utilize the ZO method to update the projection. We estimate the gradient and update the projection as:
C(7+ + eu; 0₂) E (7 - cu; 01)
V
(6)
2e
Mo++1 = -Yt - "7V[(( 0,)
where u E R- is a random vector from from.N((. (0, I) and E = L(0o + T.P..l A0;; B.).
However, directly applying vanilla 70 optimization limited improvement and can even cause failure, undermining the main fine-tuning process (see Appendix C.2). This failure stems from two key issues. First, projection values depend not only on Y but also on the distance gap |. |. Ignoring this gap leads to uninformative updates and suboptimal solutions. Second, due to noisy ZO updates over few iterations, projection magnitudes can become too small or too large. A small projection pulls the model too close to the pre-trained state, erasing progress, while large one applies overly aggressive updates that destabilize training.
the sub-task of projection learning yields
To address the above issues, two strategies are devised. Re-initialization. To introduce the distance gap || into the projection learning process, the initial value 7t.o is reset to || | each time the projection is optimized. This means that, initially, the projection magnitude T |ل 1. If not perform projection updates, DiZO reverts to standard 7O. Projection clipping. To prevent drastic weight changes and maintain training stability, we introduce projection clipping. Specifically, given clipping range >> 0, if the projection magnitude T &
[1-7, r, 1+ + r], it is clipped to remain within this interval. This prevents aggressive model adjustments that could destabilize training.
4 Overhead Analysis
We simply analyze the computational overhead of our method here and will elaborate further later.
Memory overhead. Our method requires additional memory as it involves storing the pre-trained model and calculating the weight distance gap with the fine-tuned model, which can become costly when scaling to large LLMs. However, in DiZO, we find that projecting only the weight updates of the Query and Value layers in the attention module, instead of updating all layers, not only reduces memory requirements but also delivers better performance. As a result, we only need to store the weights of these two types of layers from the pre-trained model, accounting for approximately 16.7% of the parameters in OPT-2.7B, which is manageable overhead. Similarly, LoRA (17) also focuses on weight decomposition for Query and Value layers, which echoes observation.
5

---

RoBERTO
K-512 RoBERTa
Rccuroy (N)
40
33
991.5
MNLI
KIE
S5T2
SMU
Zano-shot
Mezo ILAAI
FIZDO (LORA)
DIZD (urtA)
FT (LaRtA)
Mezi
022
Figure 2: Experiments RoBERTa-large. DiZO outperforms the baselines with and without LoRA. Detailed numbers are presented in Table E.I and the loss trajectory is shown Figure E.
Computational overhead. Our method introduces extra computational cost, the projection is learned alongside the main optimization (fine-tuning). However, we observe that performing projection learning intermittently, only once every few training iterations, does not compromise performance and significantly reduces the added overhead. This strategy reduces the computational burden while maintaining efficiency, allowing DiZO to achieve throughput comparable to vanilla ZO fine-tuning. Additionally, the reduced frequency of projection updates ensures that DiZO remains scalable for larger models and datasets.
5 Convergence Analysis
In this section, we give a nonconvex convergence guarantee terms of the expected gradient norm. The bound improves over basic ZO-SGD by replacing the factor d (full dimension) with an effective dimension the order of D where denote by D maxigess d(
We assume the following, which are standard in ZO analyses:
Assumption 5.1. C is Ly-smooth, i.e. there exists L, > 0 such that for all 0,0',
(V VL(00)II s L, |0 0'|
Assumption 5.2. Due to DiZO's layerwise projection step, each 0 remains in a ball (or line segment) around 0 of radius R(), In partieular, define
S = (0 | @ s R(O) veJ.
The algorithm ensures 0, E S for all iterations t. Theorem 5.3. Under Assumptions 5.2 suppose DiZO runs for T iterations with step size
CT for sufficiently small constant > O. Then there exist constants such that
[[I = of USt<T
min
Note that D can often be much smaller than the total parameter count Ele d()) This drastically improves the variance bounds in the zeroth-order gradient estimation, leading faster convergence rate in practice.
1==1
Standard ZO-SGD in Rd often incurs a factor of Vd in its nonconvex stationarity bound, due to estimator variance. By restricting each layer g( to remain near the pre-trained 0(²² DiZO effectively reduces the dimension D improving the rate O(VD/VT) The full proof and detailed analysis are provided in Appendix F2
Experiments
6.1 Experimental Settings
Models and datasets. evaluate DiZO with various models, including medium-sized masked models (2] (RoBERTa-large) and large-sized autoregressive models [25] 26 with different size,

---

including OPT-2.7B, OPT-6.7B 7B. OPT-13B, Llama2-7B, Llama3-3B, and Llama3-8B. The total parameter size is ranging from 355M to 13B. We evaluate on the SuperGLUE [27] benchmark.
MMLU 28 and M'T-Bench
29
benchmarks. More details on datasets are shown in Appendix B.1
Baseline. We mainly compare with two ZO works, memory-efficien 70 optimization (MeZO) 14
and Hessian-informin ZO optimization (HiZO0) [1]] MeZO is a fundamental and representative work in ZO LLM fine-tuning but suffers from slow convergence speed. HiZOO is recently proposed ZO acceleration for LLM fine-tuning, which leverages the estimated second-order information to speed up. In addition, we also incorporate the parameter-efficies fine-tuning (PEFT) technique LoRA [17]
Evaluation. For SuperGLUE, follow prior work 130 [4], evaluating few-shot and many-shot settings on RoBERTa-large using k: = 16 and k = 512 samples per class. For each setting, we randomly sample data for training, validation, and testing. For OPT and LLAMA, we 1000, 500, and 1000 samples respectively. For MMLU and MT-Bench, we adopt the setup in [31 32], fine-tuning on the Alpaca GPT-4 dataset [33]. All experiments are conducted on NVIDIA A 100 and A6000 GPUs, with results averaged over three trials. Full results are provided in Appendix B
6.2 Medium-sized Masked Language Models
We conduct experiments RoBERTa-large across three types of datasets and compare DiZO with two ZO baselimes. We also explore PEFT by integrating LoRA. Figure 2 presents the results, while Figure E shows the trajectory of training loss curves, indicating the convergence speed of DiZO and MeZO. Our key findings are as follows:
DiZO greatly increases the convergence speed over MeZO. By using divergence-driver layer adaptation, the loss curve of DiZO decreases much faster, cutting the required iterations by over 50% on SST-2, MNLI, and RTE. Moreover, DiZO improves accuracy by 1.7%, 3.6%, and 8.5%.
DiZO outperforms MeZO and achieves results on par with full fine-tuning. From Figure DiZO consistently surpasses McZO on all six datasets. Notably, on SST-2 and RTE datasets, DiZO even shows better performance than FO full-parameter fine-tuning, increasing by 0.3% and 1.5%.
DiZO is effective for both full-paran parameeerrter fine-tuning and PEFT. Although DiZO applies projec- tions based on the distance with the pre-trained model, while such prior knowledge does not exist for the decomposed weights of A, it still delivers some gains.
6.3 Large Autoregressive Language Models
Table 3: Experiments results of fine-tuning OPT-2.7B (with 1000 training samples). Better results between Me70, HiZOO, and DiZO are highlighted in bold.
Dataset
SST-2
RTE
CB
BoolQ classification
WSC
WIC
MultiRC
SQuAD
DROP
Task Type
generation
Zero-shot
56.3
$4.2
50.0
47.6
36.5
52.7
44.4
29.8
10.0
FT
94.2
81,2
82.1
72.2
63.8
65.8
71.6
78.4
30.3
LoRA
94.6
80.8
82.7
77.7
59.8
64.0
72.8
77.9
31.1
MeZO
90.0
69.6
67.4
61.5
57.6
58.7
68.7
22.9
HiZOO
90.8
60.6
70.4
68.0
60.2
56.6
54.8
68.3
23.4
DiZO
92.5
68.2
71.4
67.0
63.4
57.9
56.4
69.0
24.3
McZO LoRA
91.4
66.6
71.1
67.6
59.6
57.0
57.0
70.8
22.5
HiZoO LoRA
90.6
65.2
71.4
67.4
52.6
58.8
59.0
71.8
22.7
DiZO LoRA
91.5
68.4
71.8
70.0
61.6
58.4
56.2
74.4
23.3
To assess the generalizability DiZO, we run experiments on the OPT and Llama. The overall
results summarized in Table 3 Table 4 and Figure 3 for OPT-2.7B, OPT-6.7B and Llama series, respectively. We also compare the convergence speeds DiZO and MeZO on OPT-2.7B across datasets in Figure We highlight key observations from experiments as follows.
DiZO significantly reduces training GPU hours over MeZ.. As shown Table 4 DiZO achieves faster convergence with up to 48% less GPU time by quickly establishing effective layer-wise
7

---

Table 4: Experiment results OPT-6.7B (with
8
1000 training samples).
Accorncy
88
Dataset Type
SST-2
RTE
CB
WSC
SQuAD -generation
70
classtlication
80
Mc20
90.2
73.2
71.4
62.2
76.0
HIZOO
90.7
14.2
71.8
62.1
77.3
BST-2
RTE
CB
WSC
9QUAD
DiZ0
91.1
74.8
73.2
61.8
78.6
Datanots
Mc20 LoRA
91.6
71.2
71.4
61.8
76.3
Figure 3: Experiment result on Llama3-3B and Llama3-8B. More results and detailed numbers are shown in Appendix E.4
HiZOO LoRA
91.3
71.3
71.4
62.1
76.1
DIZO LoRA
92.4
70.2
71.8
62.6
77,9
divergence. Unlike HiZOO, which reduces iterations but suffers from slow throughput due to costly Hessian estimates, DiZO maintains MeZO-level efficiency with a lightweight projection update using only two forward passes.
DiZO outperforms baselines in both standard and parameter-efficint settings. As shown Table 3 DiZO consistently outperforms MeZO and HizOO, with or without LoRA, achieving performance close to FO methods. It ranks first on five of seven classification tasks and leads both text generation tasks. These gains extend OPT-6.7B (Table and I.lama models (Figure highlighting the benefit of laser-wise adaptive updates.
Forward
100%
Proportion(%)
29%
80%
91%
65%
41%
41%
98%
60%
40%
WSC
CB
SST-2
WIC
DROP
RTE
SQUAD
MulliCC
BoolQ
# MeeO
DiZO on Datasets
Figure 4: Comparison between MeZO and DiZO on speed.
6.4 Memory and Speed Analysis
Table 5: Memory utilization and speed test on OPT-2.7B on RTE dataset (180 tokens per example on average). partial gradient-free; DiZO" learning projection with Adam. For fair comparison, the speed and memory are measured on the same machine with the same settings.
Gradient
LoRA
Peak
Averaged Memory
#Train
GPU
Task Type
Throughput
Free
Added
Memory
Iter.
Hours
FT
X
62.2 GB
62.2 GB
1.05 it/s
10.0%
16.2%
LoRA
42.5 GB
42.5 GB
2.12 it/s
8.3%
6.6%
DiZO
x
44.7 GB
10.1 GB
1.43 it/s
33.3%
39.6%
DiZO LORA
40.1 GB
9.8 GB
2.40 it/s
26.6%
18.8%
MeZO
V
7.8 GB
7.8 GB
1.70 it/s
100.0%
100.0%
HIZOO
/
13.2
13.2
1.21 it/s
63.3%
88.9%
DiZO
GB
9.5
1.54 it/s
60.0%
62.3%
MeZO LoRA
GB
GB
3.10 it/s
94.2%
51.6%
HiZOO LoRA
13.0 GB
13.0 GB
2.07 it/s
80.0%
65.7%
DiZO LoRA
9.4 GB
9.4 GB
2.87 it/s
66.7%
39.5%
In this section, we examine the memory utilization and convergence speed of DiZO in comparison with both ZO baselines and FO fine-tuning approaches (with and without LoRA). Table presents the results of fine-tum -uning OPT-2.7B on the RTE dataset, more results are shown Appendix E.2
From a memory perspective, DiZO avoids backpropagation and memory-heavy activations, cutting memory use by 90% compared to FO. overhead stems only from storing Query and Value weights (16.7% of total). In contrast, HiZOO stores full-layer Hessians, scaling poorly with model size. In

---

terms of convergence speed, DiZO significantly reduces iteration count while maintaining throughput comparable McZO, leading to much lower training GPU hours. By comparison, HiZOO achieves less iteration reduction and slows throughput of Me7. by about 1.5x due to Hessian estimation, resulting in only modest savings, or even higher training cost in some cases, such as HiZOO+LoRA on RTE
notable byproduct of our method is using FO (e.g., with the Adam optimizer) to learn the projections. While this version has memory consumption comparable to LoRA A and requires additional training GPU hours, offers distinct advantages. Since DiZO does not update projections every iteration, FO-based DiZO exhibits significantly lower average memory usage than "O-based LoRA, with an average memory overhead close to that of the 20-based DiZO. Although average memory usage may seem less critical in sngle-process, single- -P- setup, many real-world on-device training scenarios involve multi-process environments R4 35. In such cases, the FO-based DiZO can stagger memory usage phases across processes, enabling parallel operations that purely FO methods cannot achieve. Furthermore, compared with ZO-based DiZO, the FO version reduces extra training GPU hours and delivers better performance. These qualities make it particularly appealing for specific on-device training cases.
6.5 Discussion on Potential Alternatives and Limitations
Adaptive learning rate methods may appear analogous to DiZO at first glance, as both introduce per-layer adaptive control over parameter updates, but their principles differ. Methods like Adam and RMSProp adjust update magnitude based on gradient history, controlling how fast parameters move. DiZO, by contrast, uses geometric constraints to guide parameters toward a learnable target distance from fixed anchor (the pre-trained model), determining where the parameters move. This projection- based approach enables principled, divergence-aware updates that step-size modulation alone cannot replicate. Additionally, adaptive methods maintain gradient moment estimates, adding memory and computational overhead, particularly in LLMs. We empirically compare DiZO to these methods in Appendix [
Line search could potentially be simpler method replace ZO for optimizing projection scalars. However, line search approaches, e.g., backtracking, generally require tuning each layer's 's scalar independently, leading inefficient and unscalable coordinate-win wise search. These methods also rely on directional derivatives and assume smooth interactions, which do not generalize well to joint tuning across layers. DiZO avoids these issues by using ZO-based strategy that updates all scalars simultaneously, stabilized by projection clipping and re-initialization. As shown in Appendix D replacing ZO with line search under the same forward-pass budget leads to worse performance, confirming its inefficiency in this context.
Limitations. While DiZO demonstrates notable improvements in both accuracy and training ef- ficiency, the theoretical foundations behind its design choices remain incomplete. For example, the choice of using a pre-trained model as the anchor point and the selection of which layer to be projected is primarily supported by empirical observations rather than formal justification ablation study in Appendix ( The absence of solid theoretical framework to explain why such design yields consistent performance gains leaves open questions about the optimality of the approach. Nevertheless, our findings offer valuable insights and point to promising directions for future re- search, particularly in developing anchor-guided, adaptive ZO optimization frameworks with stronger theoretical grounding.
Conclusion
In this paper, we propose novel layer-wise divergence analysis to reveal the distinct update pattern between FO and Z0 methods. Building on these insights, we present DiZO, an enhanced Zo method using divergence-driven layer adaptation resemble the learning capacity of the FO method. DiZO achieves significant training acceleration and superior performance across diverse tasks and architectures. Moreover, our method can be seamlessly integrated with PEFT techniques like LoRA for additional speedup. For future work, we plan to explore DiZO in other fields, particularly for fine-tuning large pre-trained vision models.

---

8 Acknowledgment
This work was supported by the U.S. National Science Foundation, under Grant No. CCF-2553684, CNS-2312158, No. 1943046 and IIS-2223768.
References
[1] Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. End-to-end open-domain question answering with bertserini. arXiv preprint arXiv: 1XXiv:9922001716 2019.
[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeutlemoyer and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/ 1692, 2019.
[3] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge. arXiv preprint arXiv:/81 2018.
[4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113. 2023.
[5] Minghang Zheng, Peng Gao, Renrui Zhang, Kunchang Li, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv preprint arXiv:2011.09315. 2020.
[6] Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. An empirical analysis of compute-optimal large language model training. Advances in Neural Information Processing Systems, 35:30016-30MM 2022.
[7] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint XXX::0001.0836) 2020.
[8] Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W Mahoney, and Kurt Keutzer. Ai and memory wall. IEEE Micro, 2024.
[9] Shulin Zeng, Jun Liu, Guohao Dai, Xinhao Yang, Tianyu Fu, Hongyi Wang, Wenheng Ma. Hanbo Sun, Shiyao Li, Zixiao Huang, al. Flightllm: Efficient large language model inference with complete mapping flow on fpgas. In Proceedings of the 2024 ACM/SIGDA International Symposium on Field Programmable Gate Arrays, pages 223 234, 2024.
[10] Hongzheng Chen, Jiahao Zhang, Yixiao Du, Shaojie Xiang, Zichao Yue, Niansong Zhang, Yaohui Cai, and Zhiru Zhang. Understanding the potential of fpga-bas based spatial acceleration for large language model inference. ACM Transactions on Reconfiguruble Technology and Systems, 2024.
[11] Qitao Tan, Sung-En Chang. Rui Xia, Huidong Ji, Chence Yang, Ci Zhang, Jun Liu, Zheng Zhan, Zhenman Fang, Zhou Zou, et Perturbation-efficcen zeroth-order optimization for hardware-friendly on-device training. arXiv preprint 2rXiv:2504.20314 2025.
[12] Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin-Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient Ilm fine-tuning: benchmark. arXiv preprint arXiv:2402. 1592, 2024.
[13] Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance zeroth-order Um fine-tuning. arXiv preprint 1757, 2024.
10

---

[14] Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 04-53038-53073 2023.
[15] Yanjun Zhao, Sizhe Dang, Haishan Ye, Guang Dai, Yi Qian, and Ivor W Tsang. Second-ord order fine-tuning without pain for Ilms: A hessian informed zeroth-order optimizer. arXiv preprint arXiv:2402.15177 2024.
[16] Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance-reduced zeroth-order methods for fine-tuning language models. arXiv preprint arXiv:2404.0808% 2024.
[17] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.090: 2021.
[18] Astha Verma, Siddhesh Bangar, A Venkata Subramanyam, Naman Lal, Rajiv Ratn Shah, and Shin'ich Satoh. Certified zeroth-order black-box defense with robust unet denoiser. arXiv preprint 23440:3300.064300, 2023.
[19] Amit Dhurandhar, Tejaswini Pedapati, Avinash Balakrishnan, Pin- Yu Chen, Karthikeyan Shan- mugam, and Ruchir Puri. Model agnostic contrastive explanations for structured data. arXiv preprint arXiv:I906 001100011,, 2019.
[20] Xiaoxing Wang, Wenxuan Guo, Jianlin Su, Xiaokang Yang, and Junchi Yan. Zarts: On zero- order optimization for neural architecture search. Advances in Neural Information Processing Systems, 35:12868 12880, 2022.
[21] Jiaqi Gu, Chenghao Feng, Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z Pan. Efficient on-chip learning for optical neural networks through power-aware sparse zeroth-order optimization. In Proceedings of the AAAI conference artificial intelligence, pages 7583-7591, 2021.
[22] Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02005. 2023.
[23] Sijia Liu, Bhavya Kailkhura, Pin- Yu Chen, Paishun Ting, Shiyu Chang, and I Lisa Amini. Zeroth- order stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
[24] Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth-order optimization for black-box adversarial attack. arXiv preprint 182.v:1812.11377. 2018.
[25] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
[26] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 20Xiv:2302.13977,, 2023.
[27] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
(28) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[29] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-4666 2023.
11

---

[30] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. arXiv preprint UrXiv:2012.15723 2020.
[31] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: lay- erwise importance sampling for memory -efooreefffiitttt large language model fine-tuning. Advances in Neural Information Processing Systems, 77:57018-5704, 2024.
[32] Qijun Luo, Hengxu Yu, and Xiao Li. Badam: A memory efficient full parameter optimization method for large language models. Advances in Neural Information Processing Systems, 37:24926-24958, 2024.
[33] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followin llama model, 2023.
[34] Xiangyu Li, Yuanchun Li, Yuanzhe Li, Ting Cao, and Yunxin Liu. Flexnn: adaptive dnn inference memory-constrainne edge devices. In Proceedings of the 30th Anmual International Conference on Mobile Computing and Networking, pages 709-723, 2024.
Efficient and
[35] Shengyuan Ye, Liekang Zeng, Xiaowen Chu, Guoliang Xing, and Xu Chen. Asteroid: Resource- efficient hybrid pipeline parallelism for collaborative dnn training on heterogeneous edge devices. In Proceedings of the 30th Anmual International Conference on Mobile Computing and Networking, pages 312-326, 2024.
[36] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv: 10964, 2020.
[37] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[38] Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18X0.:8810044580 00.000 2018
[39] Jun Chen, Han Guo, Kai Yi, Boyang Li, and Mohamed Elhoseiny. Visualgpt: Data-efficient adaptation of pretrained language models for image captioning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 18030-18040, 2022.
[40] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 7748-8763. PMLR, 2021.
[41] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Pra- teck Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 804-814, 2022.
[42] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint orXiv:2101.00/11 2021.
[43] Shih-Yan Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In forty-first International Conference on Machine Learning, 2024.
[44] Jiawei Zhao, Zhenyu Zhang. Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Ilm training by gradient low-rank projection, arXiv preprint arXiv:2403.03507 2024.
[45] Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O): 8-bit matrix multiplication for transformers at scale, Advances in neural information processing systems, 35:30318-3033 303322, 2022.
12

---

[46] Guangxuan Xiao, Ji Lin. Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[47] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint arXiv:2304. (11 145, 2023.
[48] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu. Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantiza- tion for large language models. arXiv preprint arXiv:2308. 13137, 2023.
[49] Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26, 2017.
[50] Amit Dhurandhar, Pin- Yu Chen, Ronny Luss, Chun-Chen Tu, Paishun Ting, Karthikeyan Shan- mugam, and Payel Das. Explanations based on the missing: Towards contrastive explanations with pertinent negatives. Advances in neural information processing systems, 31, 2018.
[51] Anirudh Vemula, Wen Sun, and J Bagnell. Contrasting exploration in parameter and action space: zeroth-order optimization perspective. The 22nd International Conference on Artificial Intelligence and Statistics, pages 2926-2935. PMLR, 2019.
[52] Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26, 2013.
[53] Yao Shu, Zhongxiang Dai, Weicong Sng, Arun Verma, Patrick Jaillet, and Bryan Kian Hsiang Low. Zeroth-order optimization with trajectory- -njeetory-informmed derivative estimation. In The Eleventh International Conference on Learning Representations, 2023.
[54] Ozan Sener and Vladlen Koltun. Learning to guide random search. arXiv preprint arXiv:2004. 12214, 2020.
[55] Richard Socher, Alex Perelygin. Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language processing, pages 1631-1642, 2013.
[56] Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 1at::1508.05550 05326, 2015.
[57] Ellen M Voorhees and Dawn M Tice, Building a question answering test collection. In Proceed- ings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200-207, 2000.
[58] Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581-590. IEFF, 2020.
[59] Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pages 177-190. Springer, 2005.
[60] Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the second PASCAL. challenges workshop on recognising texuaal entailment, volume 1. Citeseer, 2006.
[61] Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 18)) 2009.
13

---

[62] Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9, 2007.
[63] Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. pages 107-124, 2019.
[64] Christopher Clark, Kenton Lee, Ming-Wee Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprini arXiv; 1905. 10044, 2019.
[65] Mohammad Taher Pilchvar and Jose Camacho-Collados. Wic: the word-in-context dataset for meaning representations. arXiv preprint arXiv: 1808008800 2018.
evaluating
[66] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[67] Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprchension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pages 2018.
[68] Rajpurkar. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1666.000 05250, 2016.
[69] Dheeru Dua Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint
2019.
[70] Joshua Ainslie, James Lee-Thorp Michiel de Jong, Yury Zemlyanskiy, Federico Lebrón, and Sumit Sanghai. Gqa: Training generalized multi-query transformer models from multi-heal checkpoints. arXiv preprint arXiv:2305.13244 2023.
14

---

A Related Work
A.1 Memory--fficient Fine-tuning
Fine-tuning a pre-trained model offers a powerful way reuse learned representations and reduce training costs compared to building models from scratch, often achieving superior performance [36
37.. Initially successful in NLP with models like BERT, RoBERTa, and GPT 38 2 39
fine-tuning
has also shown promise in vision tasks such as CLIP and SWAG [40 41).
Despite the success of fine-tuning, its high cost makes it not feasible. Therefore memory efficient fine-tuning method come up. Recent varameter-efficiee fine-tuning (PEFT), including LoRA [17], and prefix tuning [42], minimize resource needs by updating only small subset of parameters, preserving most of the tre-trained weights and ensuring valuable knowledge is retained. Low-rank recompositioonbasee methods, led by LoRA, have achieved remarkable success. The main idea is to minimize resource needs by updating only a small subset of parameters, preserving most of the pre- trained weights and ensuring valuable knowledge is retained. DoRA 4 decomposes the pre-trained weight into two components, magnitude and direction, to enhance both the learning capacity and training stability of LoRA. GaLore [4 proposed gradient low-rank projection, allows full-parameter learning while retaining the memory advantages of low-rank training. Beside low-rank method, quantization stood out a promising method to reduce resources utilization. GPT3.int8() [45] identified the outlier in activation, and include a new mixed-precisioe decomposition scheme, which isolates the outlier feature dimensions into a 16-bit matrix multiplication while still more than 99. 99.9% of values are multiplied in 8-bit. Despite training with mix- peieppreciioon SmoothQuant [46] smooths the activation outliers by offline by scaling, migrating the quantization difficulty from activations to weights. Moreover, Outlier Suppression+ [47] and OmniQuant [48] apply channel-wise shifting for asymmetry and channel-wise scaling for concentration for inte- quantization.
A.2 Zeroth-order Optimization and Acceleration
ZO optimization emerges as an attractive technique that optimizes the model without backpropaga- tion 22 49 24
18 50 19]. Unlike most frequently used FO optimization, which directly obtains
and leverages the gradient for optimization, the zeroth-order method utilizes the objective function value oracle only, estimating the gradient by finite differences. ZO method has a wide range of
applications in machine learning fields, including adversarial attack and defense [49 24 18] machine
learning explainability [50, 19), reinforcement learning [51,, and on-chip training 21). Recently,
the ZO method has been proposed to be leveraged on LLM fine-tuning to address the significant
memory usage. [14] proposed MeZO, first scaling ZO optimization to fine-tuning -intensive LLMs. greatly reducing memory utilization. On top of MeZO, [15] proposed HiZOO, leveraging the estimated Hessian information for better learning capacity, but reducing the throughput of MeZO to some extent.
ZO optimization, although it significantly saves memory, converges more slowly than FO methods due to higher variance from random search. [23 introduced ZO-SVRG by incorporating variance proposed using a Gaussian process to model objective function queries, thereby reducing query complexity and allowing more frequent queries to lower gradient variance. (54) performed random search on learned low-dimensional manifold, reducing the number of needed objective queries. However, existing ZO accelerators face two main challenges when adapting to ZO fine-tuning for LLMs. First, these approaches were typically designed for smaller- scale tasks involving fewer parameters and less data, and cannot be directly extended to large-scale LLMs. Second, many prior methods focus on improving query efficiency, whereas recent work has shown that single query can suffice for LLM fine-tuning [14]. How to effectively accelerate ZO optimization on large model fine-tuning remains a problem.
reduction techniques 52
53
Moreover, ZO has several properties that make it well-suited for on-device or edge training scenarios.
1) Memory efficiency: Edge devices such as mobile phones and FPGAs typically offer limited memory resources. ZO significantly reduces memory usage by avoiding activation and gradient storage, making it more deployable in such constrained setting. 2) Forward-only optimization: As 70 only relies on forward passes, it is compatible with existing inference accelerators (e.g.. NNAPI on Android, edge TPUs, etc.), which typically lack support for backpropagation. This makes 70 a strong candidate for adapting inference-only hardware for training.
15

---

B Experiment Settings and Analysis
B.1 Datasets and Evaluation
Table B.1:
The hyperparameter for experiments. For DiZO and DiZO LoRA, we only show the setting of extra hyperparameters, and have the same setting in other common hyperparameters with McZO and McZO LoRA respectively.
Experiment
Hyperparameters
Values
Batch size
8
FT
Learning rate
{le-5, 5e-5)
Constant for RoBERTa
Lr schedule
Linear OPT Llama
Batch size
{64, 16)
Learning rate y (I.r)
(1e-6,
MeZO
1e-3 Constant RoBERTa
Lr schedule
Linear for OPT and Llama {64, 16)
Batch size Learning y (Lr)
(le4,
McZO LoRA
le-2
Constant for RoBERTa
Lr schedule
Linear for OPT and Llama {50. 100, 200, 400)
Projection update cycle Smoothing scalar
DiZO (LoRA)
[le-1, 1, 5e-2] {0.1, 0.2, 0.3)
Clip range T
For the RoBERTa-large model, SNLI [56], TREC [57]. MNLI (58), and RTE [59] 60) 61 62). Following previous studies, we cap the test size at 1000 samples. Two training settings considered: i = 16 and k: = 512, where we randomly select 16 or 512 samples per class for both training and validation.
the following classification datasets: SST-2 55
SST-5 55
For the OPT and Llama series models, we use the SuperGL.UF benchmark 27), which includes
RTE [59 60 61 62]. CB (63. BoolQ [64], WIC (65). WSC [66] and MultiRC 67 We also include
SST-2 [55] and two question answering datasets, SQuAD [68 and DROP 669 For each of these
datasets, we randomly sample 1000 instances for training, 500 for validation, and 1000 for testing.
B.2 Hyperparameter Setting
We use the hyperparameters in Table B.1 for experiments on RoBERTa-large, OPT-series, and Llama- series models. Specifically, the choice clip range did not significantly impact the performance. The selection of the projection update cycle and scalar for projection affects the performance somewhat. Generally, for datasets that need larger iterations for convergence, or for these harder datasets, DiZO prefers larger update cycle, while for those less complicated datasets, DiZO benefits from smaller update cycle.
C Ablation study DiZO
C.1 Ablation for Projection Layers Selection
Instead of applying projections to all layers, which would require storing the entire pre-trained model, we focus only on projecting the weights the Query and Value in the attention modules. shown in Table C this strategy achieves the best trade-off between the overall performance and extra storage requirements, does not reduce the performance and only 16.7% of the parameters the pre-trained model are needed to store. A Similar strategy has also been adopted in LoRA [17]
16

---

Table C.I: Ablation study for selecting which layers project. The highlighted line with blue rectangle the setting used in DiZO. Extra memory indicates the extra memory needed due to pre-trained model storing. Attn Q: attention Query layer; Atm V: attention Value layer; Attn K:
attention Key layer, Attn_O attention output projection; Dense: dense fully connected layer.
Attn
Attn
Attn K
Attn O
Dense
Extra memory
SST-2
RTE
SQuAD
/
100%
91.7
68.4
67.3
/
/
/
33.3%
92.2
67.9
69.2
/
/
X
x
25.0%
91.9
67.1
68.1
/
x
X
I6.7%
92.3
68.2
69.0
8
8.4%
90.5
64.9
66.5
C.2 Ablation for Strategies in ZO Projection Learning
As discussed in Section 3.3
introduce two strategies, re-initialization (Re-init) and projection clipping (Clipping), to enhance projection learning and improve the stability of fine-tuning. The ablation results for these strategies, along with the corresponding loss curves, are shown in Figure C.1
Overall (left in Figure C.I omitting either Re-init or Clipping significantly diminishes the benefits of DiZO, with MeZO outperforming DiZO in these cases. Specifically, without Re-init, accuracy drops sharply, falling below MeZO. Similarly, without Clipping, while DiZO slightly outperforms MeZO on simpler datasets like SST-2, it suffers from severe model collapse on more challenging datasets, leading to significant decline accuracy.
From the loss curve trajectory (right Figure C.T without Re-init, DiZO loses its advantage in training acceleration, the loss curve becomes noticeably slower decrease. Without Clipping. the loss curve exhibits significant oscillations during certain training steps. This instability arises when projections are optimized unsuitable values, such as extremely large or small magnitudes. These inappropriate projections cause substantial changes in model weights, leading to pronounced oscillations in the loss.
Type Mc70
Re-init
Clipping
SST-2
SNLI
TREC
MeZO
Wio Re-init
90.5
66.0
76.9
DIZO
W/O Clipping
x
88.6
64.2
73.8
DiZO
/
x
90.9
56.2
61.2
92.2
71.6
77.4
Iterations
Figure C.I: Ablation study for the two strategies: re-initialization and projection clipping, which is conducted on RoBERTa-large (k- = 16). Left: overall results when ablating the strategies. Right: loss curve when ablating the strategies.
C.3 Ablation for the Selection of the Anchor
conduct an experiment to illustrate the effect of other anchor points beyond the pre-trained weights, the results are shown in Table C.2 In conclusion, using 0 or 0,-1 significantly reducing the benefit of our method. Specifically, using 0 as an anchor yields similar results to using 0,-1 in terms of GPU hours. but causes unstable training, the accuracy decreases at the later training stage, and causes the results to be even worse than MeZO. Therefore, the effectiveness of our method is inseparable from the choice of anchors. Selecting more robust anchor could not only improve accuracy but also the convergence speed.
C.4 Ablation on Hyperparameter Setting
We conduct experiments to investigate the effect of different hyperparameter settings, including different clip ranges, smoothing scalars, and projection update frequencies. Results are obtained by fine-tuning RoBERTa- large on SST-2 and SST-5 with 3 different settings for the 3 hyperparameters,
17

---

Table C.2: Comparison on conducting projection on learning rate (LR) or use weight (t- iteration 0z-1 instead of the weight of the pre-trained model do as the base of projection. Results are obtained by fine-tuning OPT-2.7B.
SST-2
RTE
SQUAD
Anchor
GPU
GPU
GPU
Acc.
Acc.
F1.
Hours
Hours
Hours
NA (MeZO)
90.0
100.0%
63.5
100.0%
68.7
100.0%
0
86.9
85.7%
58.4
91.0%
62.2
85.8%
0 1 projection
90.7
87.8%
64.5
90.3%
67.2
88.4%
DiZO
92.5
55.7%
68.2
62.3%
69.0
65.4%
Netti'ss loo 9877
Fulditts bagerea
14
64-
124
41-
41-
1
3
32
Jarget SSTE
EaCCRT SSTE
JarkTi_argeo SOTE
34.
20-
H58 D7 X00
Figure C.2: Hyperparameter sensitive testing, including clip range, smooth scalar, and update frequency. Evaluated by training Roberta-large on SST2 and SST5 datasets.
as shown in Figure C.2 In conclusion, for easier classification tasks like SST-2, the weight of the model changes fast, and so as the convergence speed, therefore, we can apply more aggressive projection strategies, apply a larger clip range, a larger smoothing scalar, and update the projection more frequently. In contrast, for the rather more difficult tasks, a more conservative projection strategy preferred.
C.5 Quantization of Anchor (Pre-trained) Model
Our method introduce a anchor model (i.e., the pre-trained model) for projection, roughly 16% of the pretrained parameters must still reside in memory. To further reduce extra memory cost, we explored anchor compression via quantization of the pretrained Query and Value matrices to 8-bit and 4-bit precision [48]. As shown in Table C.3 DiZO can effectively incorporate with the quantization technique, still preserving advantages in both accuracy and GPU hours. Exploring more advanced quantization or compression methods to further improve anchor efficiency while maintaining performance is an important avenue for future work.
D Do Other Alternative Strategies Work?
As discussed in Section 6.5
we compare with two representative straightforward alternative strategies, Adam and Backtracking linesearch, to highlight the effectiveness of our method.
Adam as representative of learnable learning rate methods, Adam leverages first- and second- moment estimates of historical gradients to adaptively modulate the update magnitude. However, incorporating Adam into ZO optimization poses significant practical challenges. Storing gradient mo-
18

---

Table C.3: DiZO with quantized anchor.
Method
SST-2
RTE
Acc
GPU hours
Acc
GPU hours
MeZ0
90.0
100%
63.5
100%
DiZO (8-bits)
92.2
63%
67.2
71%
DiZO (4-bits)
91.7
67%
65.2
68%
DiZO
92.5
56%
68.4
62%
Table D.I: Results on fine-tuning OPT-2.7B on SST2 when using Adam the optimizer. Adam is either
or introduces a lot of extra computational overhead.
Method
Optimizer
Acc.
Training FLOPs
Memory
Iter.
McZO
SGD
85.2
100%
6.8 GB
2K
McZO
Adam (Recompute)
86.1
431%
6.8 GB
2K
McZO
Adam (Store)
86.1
100.2%
17.6GB
2K
DiZO
SGD
86.3
61% %
7.5GB
IK
DiZO
SGD
89.8
122%
7.5GB
2K
ments leads to over 3x memory overhead compared to ZO-SGD, as shown Table D.I MeZO 14
attempts address this by recomputing moment statistics the fly rather than storing them. How- ever, this strategy is also impractical as the total computational overhead increases quadratically with the training iterations, e.g., x more FL.OPs for only 2K iterations. Given that zeroth-order methods often require tens of thousands of iterations to converge, both strategies render Adam either memory-intensive computationally impractical at scale. In contrast, DiZO with SGD achieves higher accuracy with significantly lower FLOPs and comparable memory usage.
Line Search appears be simpler alternative for tuning the projection scalar. However, it requires freezing all other layers when optimizing the scalar for a particular layer, resulting in significant computational overhead. To empirically validate this limitation, search DiZO with Armijo-style backtracking line search. For fairness, the line search is initialized (1 7) original learning rate, matching the scale used in DiZO. The results in Table D.2 show that, under the same forward pass budget, backtracking performs notably worse than the original MeZO. Furthermore, even when increasing the number of forward passes to 7800, only matches MeZO's performance and still falls short of DiZO. These results confirm that traditional line search is only less effective but also less efficient the context of sigh-iimensional, layer-wise projection tuning.
replace the ZO-based projection
E
More Experimental Results
In this section, we provide comprehensive presentation of our results across various datasets and models to complement the main paper. Specifically, the results include:
Detailed accuracy number and trajectory of training loss on RoBERTa-largg (Table F. and Figure|
Table D.2: Results fine-tuning OPT-2.7B SST2. For fair comparison, Armijo-style backtracking line search to replace ZO in our method.
Searching Strategy
Forward Pass
Accuracy
NA (MeZO)
4000
85.2
Backtracking
4000
81.6
Backtracking
7800
85.1
ZO (DiZO)
4000
89.3
19

---

RoBERTa on 951-2
RoBERTs on RTC
Mc20
2,5
Mez0
Ma20
0.5
D1ZO
1.5
Di2O
0.4
1.0
0.2
1,9
0.5
0.1
0.0
0
1250
2500
3750
5000
1250
2500
3750
5003
1250
2500
3750
5000
Iterations
lleratons
leralions
Figure E.I: Trajectory of training loss curves when using Me70 and DiZO fine-tune Roberta-large.
More memory and speed results fine-tuning OPT-2.7B on SST-2 and SQuAD datasets (Table E.2 and Table E.3
Results on larger model, OPT-13B (Table E.4).
Results on Llama3-3B and Llama-8B (Table E.5 and Table E.6).
Results on more challenging benchmark MMLU and MT-Bench (Table E.7 and Table E.8)
E.1
RoBERTaalarge Experiments
Table
reports the corresponding detailed numbers from Figure 2 and Figure E.1 shows the trajectory of training loss.
Table E.1: Experiment results on RoBERTa-large six datasets.
Datasct
SST-2
SST5
SNLI
MNLI language inference
RTE
TREC -topic-
Task Type
sentiment
Zero-shot
79.0
35.5
50.2
48.8
51.4
32.0
Gradient- free methods:
16
McZO
(1.2)
45.5 (2.0)
(3.9)
56.5 (2.5)
(5.3)
(2.7
MeZO LoRA
(1.9
43.0 (1.6)
(9.7)
64.0
(3.6
73.1 (6.5)
HIZOO
(1.6)
46.1 (1.3)
(3.1)
(3.1)
(7.4
77.2
HiZOO LoRA
91.4
44.8 (1.5)
70.7 (5.2)
64.3 (2.8)
66.9 (3.2)
72.7 (7.3)
DiZO
92.2
47.1 (1.3)
(3.1)
60.1 (3.5)
67.9 (4.7)
77.4 (2.4)
DiZO LoRA
(0.8)
44.6 (1.7)
71.6 (3.8)
65.6 (2.8)
67.3 (3.9)
(4.3)
Gradient methods:
16
(1.9
47.5 (1.9)
(2.6)
(2.3)
(6.4
(5.0
LoRA
91.4 (1.7)
46.7 (1.I)
74.9 (4.3)
67.7 (1.4)
66.1 (3.5)
82.7 (4.1)
Gradient- freee methods: & 512
Me70
(0.7
(1.4)
(1.0
(0.5)
(2.0
(1.3
McZO LORA
93.4
52.4 (0.8)
(0.8)
77.9 (0.6)
(1.3
95.0 (0.7)
HIZOO
(0.4)
53.5 (1.2)
(1.44
71.2
79.1 (1.2)
94.9 (1.7)
HiZOO LORA
94.3 (0.5)
54.1 (0.6)
82.7 (1.8)
(9.0)
(0.6
93.1 (0.5)
DiZO
94.6 (0.1)
53.6 (1.7)
(0.6
79.8 (0.9)
80.3
93.8 (1.5)
DiZO LORA
94.3 (0.3)
54.1 (1.4)
83.7 (1.1)
(7.6)
(1.4)
95.7 (0.9)
Gradient-based methods:
512
FT
(0.7)
55.9 (0.9)
(0.7
(0.4
(2.7
(0.2
FT LoRA
94.2 (0.2)
55.3 (0.7)
88.3
83.9 (0.6)
83.2 (1.3)
97.0 (0.3)
E.2
More Memory and Speed Analysis
We present the memory and speed results for OPT-2.7B on the SST-2 and SQuAD datasets in Table and Table E.3 respectively. DiZO significantly reduces the number of required iterations while maintaining throughput comparable to MeZO, leading to substantially fewer training GPU hours. In contrast, HizOO achieves only modest iteration savings and further reduces the throughput of Me70 by approximately 1.5x due to its reliance on second-order information estimation. result, HiZOO offers only slight improvement over MeZO in terms of training GPU hours. In some
20

---

Table E.2: Memory utilization and speed test on OPT-2.7B on SST-2 dataset. partial gradient-free
v: gradient-free free;; X: gradient-basee Dizo'; searching projection with Adam.
Gradient
LoRA
Peak
Averaged
#Train Iter.
GPU
Task Type
Throughput
Free
Added
Memory
Memory
Hours
FT
X
45.4 GB
45.4 GB
1,81 it's
9.3%
16.8%
LoRA
X
/
18.4 GB
18.4 GB
4,50 it/s
5.6%
4.3%
DiZO' (W. FO)
17.8 GB
GB
2.63 it/s
33.3%
41.5%
DiZO LORAT
/
16.1 GB
14.7 GB
4.16 it/s
22.2%
17.5%
MeZO
/
6.8 GB
6.8 GB
3.28 it/s
100.0%
100.0%
HiZOO
/
11.8
2.22 it/s
59.2%
87.4%
DIZO
/
GB
7B
3,05 it/s
51.8%
55.7%
MeZO LoRA
6.5 GB
6.5
5.56 it/s
74.1%
43.7%
HiZoO LoRA
7
11.5 GB
GB
3.70 it/s
46.3%
41.0%
DiZO LoRA
GB
GB
4.92 it/s
38.9%
25.9%
Table E.3: Memory utilization and speed test on OPT-2.7B on SQuAD dataset (300 tokens per example on average). partial gradient-free; v: gradient-free X: gradient-based. DiZO searching projection with Adam.
Gradient Free
LoRA
Peak
Averaged
#Train Iter.
GPU
Task Type
Throughput
Added
Memory
Memory
Hours
FT
X
73.5 GB
73.5 GB
0,36 it/s
7.5%
27.7%
LoRA
/
58.5 GB
58.5 GB
0.73 it/s
6.3%
11.5%
DiZO'
x
57.8
20.3 GB
it/s
41.7%
45.5%
DiZO LORA'
/
49.4 GB
19.9 GB
2.44 it/s
31.7%
17.3%
MeZO
X
GB
GB
1,33
100.0%
100.0%
HiZOO
X
12.3 GB
GB
0,97 it/s
66.7%
91.5%
DiZO
X
9.7 GB
9.7 GB
1.22 it/s
60.0%
65.4%
MeZO LORA
4
8B4
GB
2,80 it/s
73.3%
34.8%
HIZOO LoRA
V
GB
GB
2.10
56.7%
35.9%
DiZO LoRA
9.6 GB
9B
2,49 it/s
45.0%
24.0%
cases, such HiZOO combined with LoRA SQuAD, even consumes training GPU hours than MeZO with LoRA.
E.3 Larger OPT Models Fine-tuning
To further illustrate the generalizability of our method, we conduct experiments on OPT-13B. The results are shown in Table E.4 DiZO consistently outperforms the baselines both in terms of accuracy and speed.
E.4 Llama Fine-tuning
To demonstrate the generalizability DiZO, conducted experiments on the Llama-series models. The results for Llama3-3B and Llama3-8B are presented in Table E.5 and Table E.6] respectively. DiZO consistently outperforms MeZO across both the 3B and 8B Llama models.
Table E.4: Experiment results on OPT-13B (with 1000 training samples). Better results are highlighted
in bold.
SST-2
RTE
SQuAD
Dataset
Acc
GPU hours
Acc
GPU hours
Acc
GPU hours
McZO
91.4
100%
66.1
100%
84.7
100%
HiZOO
92.1
86%
69.3
82%
82.9
91%
DiZO
92.4
69%
72.6
76%
85.2
73%
21

---

However, we observed that ZO LoRA performs poorly with Llama models (including DiZO, MeZO and HiZ0O). The loss value remains stagnant, and the resulting accuracy comparable to or even worse than zero-shot results. We leave it to future work to investigate why 70 LoRA fails with Llama models. We suspect that this limitation may be related to the Group Query Attention (GQA) (70 mechanism employed in Llama3.
Table E.5: Experimental results on Llama3-3B for seven classification datasets and two text generation datasets (with 1000 training samples). Better results between MeZO and DiZO are highlighted in bold.
Task
SST-2
RTE
CB
BoolQ
WSC
WIC
MultiRC
SQUAD DROP
Task Type
classification
generation
FT
94.2 (0.4) (2.1) (4.7) 72.2 (4.2) 63.8 (1.8) 65.8 (2.3) 78,2 (3.2) (2.9) 40.3 (1.2)
MeZO
88.8 (1.1) 67.4 (1.7) 73.2 (2.4) 78.0 (4.4) 56.6 (3.8) 63.4 (2.3) 64.8 (3.1) 61.9 (2.7) 27.8 (2.0)
HiZOO
89.5 (1.4) 67.1 (1.3) 74.4 (1.9) 78.8 (4.7) 56.3 (3.1) 64.4 (2.4) 64.3 (2.9) 61.7 (2.8) 28.6 (3.0)
DiZO
90.0 (0.9) 68.2 (1.6) 76.7 (3.3) 76.8 (3.8) 57.8 (4.2) (1.7) 64.2 (2.9) 63.2 (2.7) 29.7 (1.3)
Table E.6: Experiments results on Llama3-8B for seven classification datasets and two text generation datasets (with 1000 training samples). Better results between MeZO and DiZO are highlighted in bold.
Task
SST-2
RTE
CB
WSC
SQuAD
Task Type
-lassification
generation
McZO
90.0 (0.7)
67.8 (1.4)
71.4 (2.2)
60.2 (1.6)
67.0 (2.6)
HiZOO
91.1 (0.9)
68.2 (1.3)
71.4 (2.9)
62.2 (1.9)
68.3 (3.1)
DiZO
91.5 (0.8)
69.4 (1.8)
73.2 (3.1)
63.4 (2.9)
67.4 (2.1)
E.5
Fine-tuning on MMLU and MT-Bench
To demonstrate the generalizability DiZO in more realistic and challenging scenarios, we evaluate our method on MMLU and MTT-Bench benchmarks. we follow the setting in [31 32), fine-tune on the Alpaca GPT-4 dataset (33), which consists of 52k conversations, and then evaluate. We conduct experiments based Llama2-7B and Llama3-8B. the results are shown in Table E.7 and Table E.8 respectively.
E.6 Compare with Sparse Technique
We conducted a direct comparison between our DiZO and Sparse MeZO 13 in terms of both accuracy and training efficiency across two datasets and two model sizes. The results are presented in Table R.6 and Table R.7. From the accuracy perspective, DiZO consistently outperforms Sparse MeZO under all evaluated settings, demonstrating the effectiveness our projection- based approach.
From the efficiency perspective, Sparse MeZO requires generating the sparsity mask dynamically during training. Compared DiZO. Sparse MeZO requires longer GPU hours for about 20%. and slows the throughput for more than 30%. Moreover, with the grows of model size, the throughput of Sparse MeZO will further decrease, due to the growing cost of maintaining and updating the sparse
Table E.7: Results of fine-tuning Llama2-7B more challenging benchmarks, better results highlighted in bold.
MT-Bench
MMLU (5 shot)
GPU hours
Zero-shot
3.93
45.87
McZO
4.59
45.22
100%
HiZOO
4.62
45.42
92%
DiZO
4.79
45.91
78%
22

---

Table E.8: Results of fine-tuning Llama3-8B on more challenging benchmarks, better results are highlighted in bold.
MT-Bench
MMLU (5 shot)
GPU hours
Zero-shot
5.46
65.20
MeZO
5.89
65.08
100%
Hizoo
5.93
65.20
95%
DiZO
6.15
65.42
83%
mask. While Sparse Me70 reduces the number of training iterations, its lower throughput results in longer total GPU hours compared to DiZO. These findings demonstrate that DiZO not only delivers better accuracy but also achieves more practical training efficiency compared to Sparse MeZO.
Table E.9: Acc and Speed Comparison OPT-2.7B.
Method
Dataset
Accuracy
Throughput
#Train Iter.
GPU Hours
MeZO
SST2
90.0
3.3it/s
100%
100%
Sparse MeZO
SST2
91.4
2.3it/s
55%
79%
DiZO
SST2
92.3
1.9it/s
52%
56%
MeZO
RTE
63.5
1,7it/s
100%
100%
Sparse Me70
RTE
67.1
1.1it/s
50%
73%
DiZO
RTE
68.4
1.5it/s
60%
62%
Table E.10:
Ace and Speed Comparison OPT-6.7B.
Method
Dataset
Accuracy
Throughput
#Train Iter.
GPU Hours
MeZO
SST2
90.2
1.8it/s
100%
100%
Sparse MeZO
SST2
91.9
1,0it/s
47%
84%
DiZO
SST2
92.4
1.7it/s
62%
69%
MeZO
RTE
73.2
0.6it/s
100%
100%
Sparse McZO
RTE
74.3
0,3it/s
39%
88%
DiZO
RTE
74.8
0.5it/s
65%
81%
23

---

Theoretical Analysis
F.1 Variance Symmetry under Isotropic Perturbations
We consider neural network with layers parameter blocks) and analyze the zeroth-order gradient estimator constructed via two-point finite differences. Let L(0; B) denote the loss evaluated on mini-batch B, with 0 = (o(1) ,.... o(4) the full parameter vector. To estimate the gradient V. without access to derivatives, we apply two-sided estimator along randomly sampled directions uu R
V Fon
L(O + Eu.)
L(O eu:)
(E) Ui
9
2c
where ! the sub-vector of direction u, corresponding to layer l.
We aim to characterize the variance of this estimator, in particular:
E
V 0in) C
Key assumptions:
1. Each u, is drawn independently from an isotropic distribution in RR i.c., Elu,u, = I.
2. The scalar A, is the same across all parameter blocks for given i, it depends only the global perturbation.
3. Each block u has zero mean, unit covariance in its own subspace Rde, and is uncorrelated with other blocks (m)) for l m. (.m")
Variance Expansion: We examine the norm-squared of the estimator:
1 9 -
V e
Au
(8)
Taking expectation over {u.}:
E
((t) L
||]
Σ
E
A
[11
e
Σ
E
q²
q2
Due to independence and zero-mean isotropy, cross terms vanish, and we obtain:
E
oco
E
A² | 1
q
where A and u ( are representative samples from the same distribution.
Conclusion. Since A is shared across layers and (*) has expected squared norm proportional to the layer dimension de, we conclude:
E
de.
In other words, the second-Ne momeet of the gradient estimator depends the layer only through its dimensionality d,, and not through any asymmetry in the distribution of direction vectors. If d, are the same for all f, each block exhibits identical expected variance.
This property justifies using uniform per-layer treatment in analysis and initialization when random direction sampling is isotropic.

---

F.2 The Proof of Convergence Analysis
We now prove Theorem
Proof. By Smoothhesss (Assumption 5.1. for any 0,0':
Li 110' 011.
[(0') & [()) (VL(0), 0' 0) +
2
Set 0 = 0g and 0' = Oc+1- We get:
Li 1| - 0.l12.
[(0++1) z ((0.) (VL())), oc+1 O.) +
2
Taking conditional expectation Ee[-] : E[- 0.] yields
L
Eg [(((()) - C(0.) < E, (V)()))) au+1 - 0,71 +
E, 01+1 -..1l?.. 2
(8)
Denote 0.+1 = 0: - -] Then
0.+1 (roojs((+1),
0 0: (0++1 - -0+11) 19.-
Hence
(VL().), 0c+1 - 0z) (VL())), 0c+1 02+1) n ()(VC(()). 9.)
Because 0:-1 is the nearest point S to Oe+1 (by definition of projection). we have
|| 0141ll $ ||0, - 0242ll = 7 l1
Thus
10 - 0.+1ll s n 119eel-
(9)
Taking conditional expectation: Eg (VVL(()) 0c+1- 0r)] (.(()))(),) 0++1 - 0.+1)] 1EE, 9:)]
Using
with Schwarz:
(V.(0,), 0.+1 0++1)l - |VC()))) |0 00+1ll S /l ll
Moreover, the two-point estimator is unbiased, so
E, (VVC())) ].)] (VL(),), E.gg]) |VC(0.)
Hence
E, (V.().), 0.+1 0.0 s n|[vc(e)] E. [199:]]] ql[V2(0))
(10)
Again from the projection property:
||0+11 0el
||rojs (0++) - 0.1l < |10.1 - 0ell 1 ll
Thus
| 0.112 < 1/2 ||
Taking expectation completes:
E, [110 - 0.113] < 7 E, [[]
Substitute 10 and the above bound into (8)
T. 2
E, [(().)) [(0.)] n [V))))) E.ilgg II] /V
+
n² Eellge
Taking total expectation and summing over = to -
T-1
E [((T)) (((o) <
Σ EEE//VL(0)) [|s.I]] [ollll-E[[ill E(
4 1フ [19
25

---

Assume that the two-point Amite-difference estimator satisfies
[l1 V2(0.)) < (D.9),
where o² (D, q) grows primarily with D (the maximum layer dimension) and the number of queries 9. rather than the full sum of dimensions. Therefore, we have
E [11 S ci (IVCC +
+D,(), +
for some constant C1 > 0. addition, B[IVL(()))I II S (E | <
22 + (for a constant c2). Collecting terms and choosing = c/v with
> 0 sufficiently small ensures that the -n [V) term dominates the positive terms for large
T. A standard telescoping argument shows
T Σ B[IVCCO)) = =
VD
vT
which implies the stationarity measure
[V = of (V) OSTET
min
VT
This completes the proof.
F.3 7-Stability of the Clipping Step
Assumptions.
A1. L: L: is Ly
A2. Zeroth-order updates use step-size Th = 17 and produce an intermediate point of 0, - 7796 with an unbiased estimator 9t.
A3. After every is iterations we apply projection clipping:
044 = 0 + Pi (e) (0
Pt == clip [o; (0) - 1+7]
(6)
(6) where Pt
(8) /1/ 1|2. and (0, 1) is the scalar clipping width.
Let
R max | 1l
Gmax max |19e|22-
Then for any 7, we have the stability bound:
|0++1 12112 S G/maxx + T Bimax
(1I)
Moreover, if 7 satisfies
7/Gmxx S Cr Rmax
0 0<cr 1
(12)
then under the same conditions as Theorem 5.3 the projected DiZO iterates satisfy
VD D
min E B o 0gieT
(13)
That is, the original non-convex convergence rate is preserved. If T violates (12). the bound inflates
linearly with i.c.
DVD/T+T/m
26

---

Proof. For any Ly loss, we have
L(0++1) & L(O,) - (VL(0)),0(+1 01) + 41 10 0.112
(14)
Because 0(+1 is the nearest feasible point to 0f (Euclidean projection), the proof in the paper shows
|0 0 ll2 S nl19
(15)
The additional clipping in Pi imposes another bound:
| 8!0) 112 = lp, 11l| 112 ST RRmax
(16)
hence combining gives
||0+1 Bell2 S n/Guax + - Rmax.
(17)
Plugging into (4) and taking conditional expectation yields
E. (L(0+))) L(O.)) < - 9:)] Ly(( + Rmax G
Using standard ZO analysis, the inner product term satisfies
" 2 VL(00)1 + 7 00²
=2"
and after telescoping the left-hand side for = 0,..., T
1, dividing by nT, we obtain
T
VD VT
T TBmax
T ([ S
+ 0
r satisfies condition
12) the second term is dominated, yielding (13).
Remark. Equation
12 implies that the projection impulse matches single ZO step:
71191112
~
1
During fine-tuning, we typically observe || || E [0.5, 1.5] after warm-up, and learning
rates 10~10-2210 This leads in the range with 0.2 being the empirically stable choice across tasks.
27

---

G Implementation
The following an implementation of our "ZO projection learning" in PyTorch.
def 0o Lccnrii (theta_t, theta_0. Gammas, delta, eta tau, x):
Perform Zeroth-ordes Projection Learning.
Args:
theta_t: Current model parameters to be fine-tuned.
theta_0: Pre-trained model parameters (anchor).
Gammas: Projection parameters need to be optimized.
delta: Smoothing parameter.
eta: Learning rate for projection gradient descent.
tau: Clipping factor for projection bounds.
x: Input data for the forward pass.
...
# Calculate the L2 norm of the distance gap norms =
name: torch norm (param data anchor data)
for (name, param), anchor in zip [theta_t named_parameters () theta_0 parameters ())
# Initialize the projection values for name, gamma in Gammas. paammd__rramttrrs/ () gamma. data norms/name:
for range Lange(max (max _ixx_iters:::
# Step 1: Perturb and apply projection, then compute loss Gammas PerturbGamma (Gammas, delta)
ApplyProjection (theta_t, pre_trained, Gammas) lossl Forward (theta_t, x) ReverseProjection (theta_t) # Reset the parameter before projection
Step 2: Reverse and apply projection then compute loss Gammas PerturbGamma (Gammas. -2 delta) Apply Projectio (theta_t, pre_traing traitrai Gammas) loss2 Forward (theta_t, x) ReverscProjection (theta_t) # Reset the parameter before projection
# Step 3: Reset projection and compute gradient Gammas CeeturrGaammaaGanne delta) # Reset projection grad (lossl loss2)
delta)
# Step 4: Gradient descent with clipping for name, gamma in Gammas, named_ named__arametter(()
torch Beeuual_seeddde (seed) # For resampling perturbation torch normal (mean=0, std=l, size=gamma. data size ())
gamma. data torch. corch
gamma.. data eta grad * z,
(1 tau) norms name).
(1 tau) norms name]
) Conduct descent and apply clipping
return Gammas
28

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] " is generally preferable to "[No]" it is perfectly acceptable to answer " "[No] provided proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often
nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we claimed our contribution is to devise method to improve the performance ZO on LLM.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations and future work are discussed in Section 6.5 and conclusion.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
29

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
We have our proof both in Section 5 and Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Thcorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We present our pseudocode and PyTorch-style implementation in the paper and release our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
30

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We release our code with the anonymized url.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
gewss CCodeSoodiieeeee ssi mollicyy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
31

---

Answer: [Yes]
We present our experiment setting in both the experimental section and the appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the standard deviation of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computing resources we use, and carefully analyze the computa- tional overhead of our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
32

---

9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httppp////pcrrs.....oooooccchhhs
Answer: [Yes]
Justification: Yes, do.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
33

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we have cited all the necessary works.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappessii thcode www.com//a/oo/oc datasat has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, have released all the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
No human- related parts are included this paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not related.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Not related.
Guidelines:
The answer NA means that the method development in this research does involve L.Mss as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tttp:://neuris... cttpiiissscc.///oooo /Cco///rnnnee//22///////// for what should or should not described.
35


Reviews:
[{"review_id": "xdonrJnwlA", "reviewer": "Reviewer_yQqP", "summary": "This paper proposes DiZO, a gradient-free optimization method for efficient fine-tuning of large language models (LLMs). The authors begin with a layer-wise divergence analysis, observing that first-order (FO) optimization produces updates with varying magnitudes across layers, while zeroth-order (ZO) optimization tends to apply uniform-magnitude updates. This discrepancy is identified as a key limitation of ZO methods in terms of convergence and performance. To address this, DiZO introduces a projection mechanism guided by the divergence between the current model and the initialization (used as an anchor), enabling each layer’s updates to be geometrically constrained and adaptively scaled. This allows ZO to mimic FO-like learning behavior while retaining its memory efficiency. Experiments across multiple models (RoBERTa, OPT, LLaMA) and tasks demonstrate that DiZO significantly outperforms prior ZO baselines and, in some cases, even surpasses FO methods. The paper further provides a convergence analysis of the projection update under effective dimension assumptions and shows compatibility with parameter-efficient fine-tuning (PEFT) methods such as LoRA.", "questions": "1.\tIs using the pretrained model as the anchor empirically optimal? Are there theoretical motivations or data-driven alternatives for selecting a more suitable anchor?\n\n2.\tCurrently, DiZO updates the projection parameter γ at fixed intervals k. Could a task-aware or training-state-aware dynamic adjustment mechanism be considered? Such an approach might improve robustness and generalizability across tasks.\n\n3.\tThe current experiments focus on NLP classification and generation tasks, while the conclusion mentions future applications of DiZO to vision models. Have the authors observed any consistency or transferability of the projection parameter γ across tasks or model types?\n\nIf these questions are addressed, my assessment of the paper’s theoretical contribution and generalizability may improve.\nThe method assumes full access to pretrained model parameters, which may be infeasible in privacy-constrained or adapter-only deployment scenarios. In addition, although DiZO is designed for memory efficiency, its real-world performance on edge hardware or production environments has not been evaluated. Further discussion or experiments in such contexts would strengthen the paper’s practical relevance.", "limitations": "The method assumes full access to pretrained model parameters, which may be infeasible in privacy-constrained or adapter-only deployment scenarios. In addition, although DiZO is designed for memory efficiency, its real-world performance on edge hardware or production environments has not been evaluated. Further discussion or experiments in such contexts would strengthen the paper’s practical relevance.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1.\tQuality: The method is technically sound, with a clear structure and practical implementability. The proposed algorithm is empirically well validated across diverse models and tasks, with convincing performance and convergence gains.\n\n2.\tClarity: The paper is clearly written with rigorous derivations, well-designed figures and ablations, and a coherent logical flow that supports both understanding and reproducibility.\n\n3.\tSignificance: ZO optimization is increasingly relevant in resource-constrained deployment scenarios. By improving both convergence speed and accuracy, this work enhances the practicality of ZO-based fine-tuning for LLMs.\n\n4.\tOriginality: The use of layer-wise divergence to construct projection-based updates that emulate FO behavior under a ZO framework is novel and insightful.\n\nWeaknesses\n\n1.\tWhile the method is thoroughly evaluated on standard NLP tasks, it lacks analysis of failure cases or known vulnerabilities of ZO methods.\n\n2.\tThe periodic projection mechanism and its associated hyperparameter settings are not thoroughly analyzed for robustness and may be sensitive to the choice of tasks.", "was_revised": true, "final_justification": "The authors' responses were of high quality, not only giving detailed and sincere answers to each question, but also supplementing new experiments and data on multiple key points, which fully illustrated their confidence and ability in their work, and solved some of my doubts during the initial review - the robustness of hyperparameters, the rationality of anchor point selection, and the applicability and generality in restricted scenarios. \nI improved my rating.", "rebuttal": "**Thank the reviewer for the valuable feedback. We carefully address the questions raised by the reviewer and added results as suggested by the reviewer.**\n\n---\n\n### Q1: While the method is thoroughly evaluated on standard NLP tasks, it lacks analysis of failure cases or known vulnerabilities of ZO methods.\n\nA: Thank you for this insightful question. According to our comprehensive experiment, DiZO works well on a wide range of NLP datasets and tasks, yielding significant improvement in accuracy and GPU hours compared to other ZO baselines, and reduces significant memory overhead compared to FT. Moreover, we also conduct experiments on a vision-language-action (VLA) model, fine-tuning it via ZO on LiBERO datasets, and we observe that the loss decreases from 0.6 to around 0.2. Although it's only preliminary results, it illustrates that ZO works in more complex scenarios and is promising for further exploration. However, it's also worth noting that ZO does not work well in training-from-scratch scenarios, only performs well in fine-tuning tasks (based on a strong pretrained model).\n\n---\n\n### Q2: The periodic projection mechanism and its associated hyperparameter settings are not thoroughly analyzed for robustness and may be sensitive to the choice of tasks.\n\nA: Thank you for your important question. We had ablation on hyperparameters like update interval, clipping range, etc. in **Appendix C.4**. Specifically, we find that setting the clipping range to 0.2 is the best across different tasks. As for the update interval, we set it to [50, 100, 200] and find that the loss curve also does not change too much. Experimental results illustrate that our method is not sensitive to hyperparameter selection and is robust. Moreover, we add more ablation results, including loss curve on 2 more datasets and models, and add results of set update interval as 150. The pattern remains consistent on the extra results, we will plot the additional loss curve in the revision.\n\n---\n\n### Q3: Is using the pretrained model as the anchor empirically optimal? Are there theoretical motivations or data-driven alternatives for selecting a more suitable anchor?\n\nA: Thank you for your valuable feedback. As discussed in **Appendix C.3**, we have empirically evaluated different anchor choices, such as using 0 or the previous iteration parameter $\\theta_{t-1}$. Among them, using the pretrained model as the anchor consistently yields the best results. We also provide theoretical evidence (to be added) showing that using the pretrained model ensures convergence under our update rule. However, we acknowledge that this choice is not guaranteed to be optimal in all scenarios, and our selection is primarily based on empirical findings.\nWhile we have not yet explored data-driven anchor selection strategies, we believe this is a promising future direction. For instance, anchors informed by loss curvature or early-stage performance could offer improved stability in noisy ZO settings. Moreover, recent FO fine-tuning literature [1][2] also suggests that leveraging the pretrained model as a reference improves robustness, so we conjecture that ZO method can also benefit from pretraining model-guided fine-tuning.\n\n[1] How should pre-trained language models be fine-tuned towards adversarial robustness? Neurpis 2021\n\n[2] Pre-trained model guided fine-tuning for zero-shot adversarial robustness CVPR 2024\n\n\n---\n\n### Q4: DiZO updates the projection parameter $\\gamma$ at fixed intervals $k$. Could a task-aware or training-state-aware dynamic adjustment mechanism be considered? Such an approach might improve robustness and generalizability across tasks.\n\nA: Thank you for your insightful suggestion. We explored two dynamic strategies for adjusting the projection interval based on training progress:\n- an update frequency decreasing strategy, where the projection is performed more frequently at the beginning and less frequently later in training;\n- an update frequency increasing strategy, with the opposite schedule.\n\nAs shown in Table R.1, the increasing strategy yields slightly faster convergence in early stages compared to the fixed-interval baseline, though final accuracy is marginally lower. The decreasing strategy performs worse in both convergence and final performance. These results suggest that adapting the projection frequency to training dynamics can be beneficial, especially early in training, but further tuning is required. We plan to explore more advanced task-aware or training-state-aware interval adjustment mechanisms in future work, potentially informed by loss trends, gradient norms, or validation metrics.\n\n**Table R.1** Accuracy of Different Strategies for Adjusting Projection Interval.\n| Intervals Strategy | 500 (Iters) | 1000  |  1500 |  2000 |  2500 |  3000 |\n|--------------------|:-----------:|-------|:-----:|:-----:|:-----:|:-----:|\n| Fixed              |    63.6    | 73.4 | 81.0 | 90.8 | 91.6 | 91.8 |\n| Decreasing Frequency         |    64.0    | 78.0 | 83.6 | 91.6 | 91.0 | 90.6 |\n| Increasing Frequency         |    62.6    | 71.6 | 75.8 | 83.4 | 89.2 | 89.8 |\n\n---\n\n### Q5: The current experiments focus on NLP classification and generation tasks, while the conclusion mentions future applications of DiZO to vision models. Have the authors observed any consistency or transferability of the projection parameter $\\gamma$ across tasks or model types?\n\nThank you for your insightful question. We do observe a consistent pattern in the behavior of the projection parameter $\\gamma$ across our experiments on both classification and generation tasks. Specifically, the projection strength, which is quantified by the ratio $\\frac{\\gamma_t}{|\\Delta \\theta_t|}$, tends to be larger than 1 in the early stages of training, pushing the model away from the pre-trained initialization. In the later stages, this ratio stabilizes around or below 1, and the distance between the fine-tuned model and the pre-trained one oscillates rather than continuing to grow. This behavior contrasts with MeZO, where the distance from the pre-trained model increases steadily throughout training. We provide detailed statistics in the distance gap with the pretrained model in Table R.2, and visualize the gap in Figure 1 in the main paper. Previous studies have shown that staying closer to the pre-trained model can improve robustness, especially in noisy optimization settings, supporting our use of projection to regulate the deviation.\n\nWhile we have not yet evaluated DiZO on vision models, we believe the consistency of $\\gamma$’s dynamics across NLP tasks suggests potential transferability to other domains. This opens up promising directions for future work on cross-modal projection strategies and task-aware projection scheduling.\n\n**Table R.2.** Distance gap with the pre-trained model, using results on Value layer in the first Attention module as an example.\n| Iter. |  400  | 800  | 1200 | 1600 | 2000 | 2400 | 2800 | 3200 |\n|-------|:-----:|------|:----:|:----:|:----:|:----:|------|------|\n| MeZO  | 1.429 | 2.18 | 2.54 | 2.98 | 3.37 | 3.83 | 4.23 | 4.53 |\n| DiZO  | 1.59  | 2.36 | 3.46 | 3.29 | 2.98 | 3.48 | 3.32 | 3.37 |\n\n---\n\n### Q6: The method assumes full access to pretrained model parameters, which may be infeasible in privacy-constrained or adapter-only deployment scenarios.\n\nA: Thank you for your valuable question. We acknowledge that in privacy-constrained or adapter-only settings, full access to pretrained parameters may be unavailable. To address this, we evaluate DiZO with LoRA (Table 3, Table 4, Table E.1), using projection anchors that are either randomly initialized (matrix A) or set to $0$ (matrix B). Although these anchors lack pretrained knowledge, DiZO still improves convergence speed and final accuracy over MeZO with LoRA. This indicates that DiZO’s benefits arise not only from pretrained parameters, but also from its ability to control parameter deviation and stabilize zeroth-order optimization. Additionally, unlike FO methods requiring gradient access, DiZO remains fully applicable in black-box scenarios, showing strong performance even without access to pretrained weights, highlighting its practicality in privacy-aware fine-tuning.\n\n---\n\n### Q7: DiZO's real-world performance on edge hardware or production environments has not been evaluated. Further discussion or experiments in such contexts would strengthen the paper’s practical relevance.\n\nThank you for your valuable suggestion. We agree that evaluating DiZO on real-world edge hardware would further strengthen the practical relevance of our work, and recognize this as an important direction for future work. DiZO has several properties that make it well-suited for on-device or edge training scenarios:\n- **Memory efficiency**: Edge devices such as mobile phones and FPGAs typically offer limited memory resources. DiZO significantly reduces memory usage by avoiding activation and gradient storage, making it more deployable in such constrained setting.\n- **Forward-only optimization**: As DiZO only relies on forward passes, it is compatible with existing inference accelerators (e.g., NNAPI on Android, edge TPUs, etc.), which typically lack support for backpropagation. This makes DiZO a strong candidate for adapting inference-only hardware for training. \n\nOne potential challenge is DiZO’s need for large quantities of random numbers for perturbation, while generating a large amount of random numbers poses a significant system burden for edge devices. We conduct tests on the OnePlus 12 Snapdragon 8 Gen 3, compared to generating a large amount of random numbers on Linux, generating on a mobile phone with an Arm-GPU only produces ignorable overhead, i.e., generating 1e7 random numbers on Linux costs 0.180576 s while it only costs 0.0598 s on a phone.\n\nDeploying our method on an edge device is a promising future research direction, and we will add more discussion regarding it in our revision."}, {"review_id": "2y2Cd7ukVn", "reviewer": "Reviewer_t4pe", "summary": "This paper introduces Divergence driven Zeroth Order optimization method. It begins with an analysis on layer-wise divergence between FO and ZO methods. Motivated by this, it proposes anchor-based, learnable projections to adapt ZO updates layer-wise. Experiments across different models show that DiZO outperforms ZO baselines on both full parameter and PEFT settings with faster convergence and improved accuracy.", "questions": "Minor Typo: Line 265 is missing a period at the end of the sentence.\n\nPlease see weaknesses for questions.", "limitations": "Yes.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The analysis comparing ZO and FO optimization is clearly presented and provides a strong foundation for motivating the proposed method.\n2. The proposed DiZO is memory-efficient and achieves performance on par with FO methods.\n3. The experiments are thorough, covering both autoencoding and autoregressive models, and are complemented by detailed analyses of memory usage and computational efficiency.\n\nWeaknesses:\n1. On larger autoregressive models, the performance gap between DiZO and FT seems bigger than that observed in smaller autoencoding models. It would be helpful if the authors could provide insights into the cause of this difference.\n2. The method introduces additional hyperparameters such as projection update interval and clipping range. The paper provides limited discussion on how sensitive performance is to these design choices.", "was_revised": true, "final_justification": "I have read the authors' responses and found some of them already explained in the appendix. I encourage the authors to integrate these clarifications into the revision. I have no further concerns and maintain my positive recommendation for this paper.", "rebuttal": "**Thank the reviewer for the valuable feedback. We carefully address the questions, and hope our response can help clarify the reviewer's questions.**\n\n---\n\n### Q1: On larger autoregressive models, the performance gap between DiZO and FT seems bigger than that observed in smaller autoencoding models. It would be helpful if the authors could provide insights into the cause of this difference.\n\nA: Thank you for your important question. We agree that the performance gap (in both **convergence speed and accuracy**) between the ZO method and full FT appears larger on large-scale autoregressive models than on smaller autoencoding models, but it's a general problem as all ZO methods present this phenomenon. We believe this is because the variance of the gradient estimated by ZO increases with model size. As shown in [1], the variance of the zeroth-order gradient estimator grows proportionally with the parameter dimension $d$ (i.e., $O(d/\\epsilon^{2})$). Larger autoregressive models like LLaMA-7B/8B and OPT-13B thus suffer from higher gradient noise, making convergence more challenging for ZO methods, which is a natural limitation of ZO itself.\n\nDespite this, DiZO still outperforms other ZO baselines in both convergence speed and final accuracy, efficiently reducing this gap between ZO and FT (in both **convergence speed and accuracy**). Moreover, it preserves its key advantage of low memory usage, making it particularly promising in constrained training scenarios where full FT is infeasible. We hope to explore improved stabilization techniques to further close this gap in future work. \n\n[1] Nesterov Y, Spokoiny V. Random gradient-free minimization of convex functions[J]. Foundations of Computational Mathematics, 2017, 17(2): 527-566.\n\n---\n\n### Q2: The method introduces additional hyperparameters such as projection update interval and clipping range. The paper provides limited discussion on how sensitive performance is to these design choices.\n\nA: Thank you for your important question. We had ablation on hyperparameters like update interval, clipping range, etc. in **Appendix C.4**. Specifically, we find that setting the clipping range to 0.2 is the best across different tasks. As for the update interval, we set it to [50, 100, 200] and find that the loss curve also does not change too much. Experimental results illustrate that our method is not sensitive to hyperparameter selection and is robust. Moreover, we add more ablation results, including loss curve on 2 more datasets and models, and add results of set update interval as 150. The pattern remains consistent on the extra results, we will plot the additional loss curve in the revision.\n\n### Minor Typo: Line 265 is missing a period at the end of the sentence.\n\nA: Thank you for your careful review. We have added the period in the revision!"}, {"review_id": "1IVrjg9lgY", "reviewer": "Reviewer_fcb4", "summary": "This paper identifies that standard ZO fine-tuning methods (eg MeZO) apply fixed-magnitude updates to each layer of the network, whereas first-order methods compute gradients of varying magnitudes for each layer of the network. As such, they propose a way to rescale the updates for each layer in the network. They include a convergence analysis and experiments on medium-sized BERT models as well as larger autoregressive language models.", "questions": "1. Can you explain why the convergence analysis is significant? It actually does not contain a convergence result in the first place and only contains a bound on the variance of the gradient. I would raise my score if the theory can be improved to more rigorously justify the tricks in DiZO -- for example, if the theoretical result provides clear guidance on how to set the clipping hyperparameters.\n\n2. Why did you not compare against any layerwise learning rate methods?\n\n3. Figure 1 is missing a numerical y-axis, which makes it hard to determine if the plotted differences between FO and ZO methods is actually significant in scale. It is also not clear why you only plotted the Q, K, V, O values out of all the parameters in the network.\n\n4. Why do you call this a projection? Projections of gradient estimates are usually something like a preconditioner. This is just a scalar multiplied to the update on each layer. I found the writing extremely confusing for a very simple point.", "limitations": "The limitations only discuss issues with the theoretical section and do not mention the fundamental issues that I raised above. It would be good if the authors acknowledge the limited empirical efficiency of their method, the introduction of an additional hyperparameter, and the overall uncertainty around their hypothesis concerning the ZO-FO gap when fine-tuning models.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- It is worthwhile to investigate the gap between zeroth and first order training methods in order to improve the former to be as efficient as the latter.\n- There are thorough experiments evaluating the efficacy of this method\n\nWeaknesses:\n- **Lacks comparison to layerwise learning rates**: The method amounts to using a layerwise learning rate for each layer of the network. The authors acknowledge this as well, but they do not compare to existing layerwise LR methods (the most popular of which is LARS). These methods are expensive to naively implement on top of ZO methods, but I think there are straightforward ZO approximations to be made. At the very least, the experiments should include ablations and comparisons against the more expensive but standard way to impose layerwise learning rates.\n\n- **Performance benefit is marginal**: The performance benefit in the setting where the benefits of ZO are apparent (ie at the ~7B model scale) is marginal. There is still a substantial gap to the first order methods, suggesting that the original premise of the paper -- that fixed-magnitude updates to each layer are the culprit for the performance gap -- is not actually true. This problem is made worse by the fact that the authors do not do multiple trials of fine-tuning, which is standard in FT literature, given the noise present when optimizing on small datasets.\n\n- **Theoretical analysis is poor**: The theoretical analysis is poor and provides little to no insight. The technique is to assume a bounded update at each timestep and then to say that the variance of the gradient estimate is reduced. A better theoretical justification for this method would be to argue rigorously that the gradient update $R^{(\\ell)}$ will be small enough when using the additional tricks in DiZO -- this is a more complex analysis and is likely not true in general. As it stands, the theory adds nothing to my understanding of the paper. A good theory section would actually advocate for why layerwise LRs are necessary.\n\n- **Requires hacky tricks**: The two tricks used to make DiZO work, re-initializing and clipping, are pretty hacky, and I am not confident that they will work in new + unseen settings, especially because clipping requires setting an additional hyperparameter.", "was_revised": true, "final_justification": "Much of the relevant information was buried in the appendices. Additional experiments run by the authors suggest the method works consistently. The theory clarification, along with the additional corollary makes the section much more interesting. I am not putting a 5 because I still think the validation settings are not very difficult. Many of these models can solve these tasks without fine-tuning in the first place.", "rebuttal": "**Thank the reviewer for the valuable feedback. We carefully addressed the questions raised by the reviewer and added results as suggested by the reviewer.**\n\n---\n\n### Q1: Lacks comparison to layerwise learning rates: The method amounts to using a layerwise learning rate for each layer of the network. The authors acknowledge this as well, but they do not compare to existing layerwise LR methods (the most popular of which is LARS).\n\nA: Thank you for this insightful comment. We clarify two key points:\n\n- First, **our method is fundamentally different from layer-wise learning rate approaches, as discussed in Section 6.5**. DiZO uses geometric constraints to guide parameters toward a target distance from a fixed anchor, deciding where to move, unlike layer-wise LR methods that control how fast to move.\n\n- Second, **we compared DiZO with a representative adaptive LR method in Appendix D**. Modern adaptive methods typically combine momentum and second-order estimation. As shown, momentum incurs high computational overhead in ZO settings, while second-order methods like HiZOO reduce throughput and increase memory usage. DiZO avoids both issues, remaining lightweight and efficient.\n\nTo address your concern directly, we incorporated LARS into ZO and report results in Table R.3. ZO+LARS underperforms MeZO and often fails to converge. This is likely due to ZO gradients having larger and noisier magnitudes, making LARS’s update rule highly unstable. These results suggest naive LARS adoption is ill-suited for ZO. DiZO instead offers a stable, low-overhead alternative that captures layer-specific behavior through geometric constraints.\n\n**Table R.3** Apply LARS to MeZO.\n| Method    | SST-2 |  RTE |\n|-----------|:-----:|:----:|\n| MeZO      | 90.0  | 63.5 |\n| MeZO+LARS | 57.2  | 53.4 |\n| DiZO      | 92.5  | 68.2 |\n\n---\n\n### Q2: Performance benefit is marginal: The performance when at the ~7B model scale is marginal. There is still a substantial gap to the FO methods.\n\nA: Thank you for your important question. We would like to point out that the performance improvement of our method mainly lies in GPU hours (reducing training iterations without sacrificing throughput), instead of accuracy. For example, DiZO achieves 22% and 17% GPU-hour reduction on MT-Bench and MMLU with LLaMA2-7B and LLaMA3-8B, and improves accuracy to some extent. These results reinforce our central claim: fixed-magnitude updates are a critical bottleneck in ZO fine-tuning, and addressing them is an effective way to improve real-world ZO training.\n\nMoreover,  it is common that a gap between ZO and FO methods exists, especially for ~7B model scale. This is in part due to the inherent variance in ZO gradient estimators, which increases with the model dimension $O(d/\\epsilon^2)$. Thus, performance gaps at larger model scales (e.g., 7B+) are to be expected and reflect a known limitation of ZO methods. \n\n--- \n\n### Q3: The authors do not do multiple trials of fine-tuning, which is standard in FT literature, given the noise present when optimizing on small datasets.\n\nA: Thank you for your important question. For multiple trials, we apologize for the omission in the main text: all experiments were conducted with three random seeds, and the reported numbers are mean values. Due to page limitations, the variances are not provided in the main paper. As evidence, please check our appendix (Table E.1, Table E.5, and Table E.6), where we include the variances. We will make this clearer and add corresponding results in the final version of the paper. \n\n---\n\n### Q4: Theoretical analysis is poor: The technique is to assume a bounded update at each timestep and then to say that the variance of the gradient estimate is reduced.\n\nA: Thanks for the reviewer’s suggestion. Thm. 5.3 is not a mere variance‑reduction lemma. It establishes a non‑convex convergence guarantee with the convergence rates of \n\n$$\n\\min_{ t < T} \\mathbb{E}[\\|\\nabla L(\\theta_t)\\|_2^2] = \\mathcal{O}(\\frac{\\sqrt{D}}{\\sqrt{T}}),\n$$\n\nwhere $D:=\\max_{1\\le\\ell\\le L} d^{(\\ell)}$ is the effective dimensionality produced by DiZO’s layer‑wise projection.\n As $T\\to\\infty$, the expected gradient norm converges to zero, i.e. the algorithm reaches a stationary point. This goes well beyond showing that the variance of the gradient estimator decreases. \nIf we suppressed the layer‑wise decomposition, we would revert to the classical ZO‑SGD bound \n\n$$\n\\min_{t<T}\\mathbb{E}\\bigl[\\|\\nabla\\mathcal L(\\theta_t)\\|^2\\bigr]\n  =\\mathcal{O}(\\frac{\\sqrt{d}}{\\sqrt{T}}).\n$$\n\nBecause modern models have $d\\gg D$, the constant $\\sqrt{d/D}$ slowdown is non‑negligible (e.g., for a 24‑layer vision transformer with uniform 1 K‑dim layers, $d/D \\approx 24$).  Thm. 5.3 therefore proves that the layer‑wise schedule is not a heuristic: it yields a measurably faster convergence rate.  This is precisely the kind of justification the reviewer asked for. Per suggestion from reviewers, we will move the rate formula into the statement of Thm. 5.3 and add a one‑sentence corollary emphasising $\\lim_{T\\to\\infty}\\mathbb{E}\\|\\nabla\\mathcal L(\\theta_t)\\|=0$.\n\n---\n\n### Q5: I would raise my score if the theory can be improved to more rigorously justify the tricks in DiZO, for example, how to set the clipping hyperparameters.\n\nA:  Thank you for your insightful suggestion. To address your concern regarding the theoretical justification of the clipping hyperparameter $\\tau$, we have added a formal result, **Corollary 5.4 (τ-stability of the clipping step)**, which rigorously analyzes how the clipping range impacts optimization stability and convergence.\n\nSpecifically, we show that for any $\\tau \\in (0, 1)$, the deviation between the clipped iterate $\\theta_{t+1}^{(\\ell)}$ and the unclipped point $\\theta_t^{(\\ell)}$ is bounded as:\n\n$$\n|\\theta_{t+1}^{(\\ell)} - \\theta_t^{(\\ell)}| \\le \\eta G_{\\max} + \\tau R_{\\max} \\tag{1},\n$$\n\nwhere $G_{\\max}$ is the maximum norm of the gradient estimator and $R_{\\max}$ is the maximum norm of the layerwise update directions.\n\nFurthermore, we prove that if $\\tau$ is chosen such that\n\n$$\n\\tau \\le \\frac{c_\\tau \\eta G_{\\max}}{R_{\\max}}, \\quad 0 < c_\\tau \\le 1 \\tag{2},\n$$\n\nthen the clipping step preserves the original non-convex convergence rate of DiZO:\n\n$$\n\\min_{0 \\le t < T} \\mathbb{E}[\\|\\nabla L(\\theta_t)\\|_2^2] = \\mathcal{O}\\left(\\frac{\\sqrt{D}}{\\sqrt{T}}\\right) \\tag{3}.\n$$\n\nThis shows that the clipping impulse introduces only a bounded perturbation to the ZO step and does not interfere with theoretical guarantees.\n\nIn practice, we approximate the above bound by setting\n\n$$\n\\tau \\approx \\frac{\\|g_t\\|_2}{\\|\\Delta \\theta_t\\|_2} \\tag{4},\n$$\n\nwhich corresponds to the theoretical threshold in Eq. (2). Empirically, we use $\\tau = 0.2$ as a default setting, and this value works well across different tasks, as observed in our experiments (in **Appendix C.4**).\n\nWe believe this theoretical development and practical guidance provide a rigorous and actionable justification for the clipping mechanism in DiZO, and we hope this addresses your concern.\n\n---\n\n### Q6: Requires hacky tricks: re-initializing and clipping, are pretty hacky, and I am not confident that they will work in new + unseen settings, especially clipping requires additional hyperparameter.\n\nA: Thank you for raising this important concern. We have demonstrated consistent improvement across multiple models and datasets, and our code is publicly available.** Appendix C.2** includes detailed ablations (with loss curves) showing that removing either strategy leads to performance drops. For clipping, we observe that a single value (0.2) works robustly across all tasks and models (see **Appendix C.4**). All reported results use this same value. These techniques are also common in practice, FO methods also often rely on clipping or rescaling for stability. We believe the strong empirical performance and robustness validate the practical value of these design choices.\n\n---\n\n### Q7: Figure 1 is missing a numerical y-axis. It is also not clear why you only plotted the Q, K, V, O values.\n\nA: Thank you for the helpful comment. We appreciate the suggestion and will revise Figure 1 accordingly. While the upper and lower subplots share the same y-axis scale, we agree that omitting the numeric y-axis in the upper plot hinders clarity. We will include it in the final version.\nRegarding the focus on Q, K, V, O parameters: we chose these because they are core components of the attention module, which has been shown in prior work to be critical for transferability and model performance. These parameters also tend to exhibit the most noticeable differences between FO and ZO gradients, making them representative for illustrating our point. We will add a more plot in the Appendix showing similar statistics for other layers in the model. We find the trend to be consistent, further supporting our conclusions.\n\n---\n\n### Q8: Why do you call this a projection? Projections of gradient estimates are usually something like a preconditioner. This is just a scalar multiplied to the update on each layer.\n\nA: Thank you for raising this important point. In our method, however, “projection” refers to a geometric constraint applied in parameter space, specifically, we rescale each layer’s update to achieve a target distance from a fixed anchor. This operation is not a projection of the gradient, but rather a form of parameter update rescaling. Moreover, we adopted the term “projection” also by analogy to recent works (e.g., [1, 2]) that apply similar parameter-level constraints and also refer to them as projections. That said, we agree that this terminology might be misleading and will revise the writing to make this distinction clearer in the final version of the paper.\n\n[1] Towards calibrated robust fine-tuning of vision-language models. Neurips, 2024\n\n[2] Fast trainable projection for robust fine-tuning. Neurips, 2023"}, {"review_id": "4Zs3LjSxBQ", "reviewer": "Reviewer_C9cT", "summary": "The paper introduces Divergence-driven Zeroth-Order (DiZO) fine-tuning for large language models.\n\n1. Observation: First-order (FO) fine-tuning naturally produces different update magnitudes across layers, while existing zeroth-order (ZO) methods apply nearly uniform random updates, slowing convergence. \n\n2. Method: DiZO keeps memory-light ZO steps but periodically projects the updates toward the pretrained weights (“anchor”) for selected layers (mainly Query/Value matrices), recreating FO-like layer-specific behavior without any backward pass.\n\n3. Results: On RoBERTa-large, OPT, and Llama, DiZO cuts iterations by up to 50 % and GPU hours by up to 48 %, sometimes even surpassing full FO fine-tuning, while requiring only ~17 % extra stored parameters.", "questions": "Good work, but it requires additional memory overhead. Meanwhile, several sparse ZO approaches have achieved notable success—for instance, “Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning.” Could the present paper demonstrate that it outperforms such methods? With comparable accuracy, users will naturally favor the option that consumes less memory.  If the authors can demonstrate that their method outperforms the sparse ZO approaches, I would be inclined to raise my score.  The authors could demonstrate this by presenting evidence on either accuracy or convergence speed.", "limitations": "1. The theoretical analysis is insufficient: the paper does not clearly explain why the pretrained LLM is chosen as the anchor, nor why only **Q** and **V** are projected instead of also including **K** and **O**.\n2. The performance gains come at the cost of increased memory usage.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper tackles a central challenge in Large Language Model (LLM) fine-tuning: how to improve the convergence speed and accuracy of zeroth-order (ZO) optimization while maintaining its memory efficiency. \n\n2. The paper introduces a novel \"layer-wise divergence analysis\" revealing that FO methods make diverse, fine-grained layer updates, unlike ZO's uniform updates. Building on this, DiZO proposes \"divergence-driven layer adaptation\" using learnable, anchor-based projections to scale ZO updates for each layer, mimicking FO's adaptive capabilities without backpropagation's memory cost. Crucially, these projections are learned via a ZO-based method for end-to-end memory efficiency, with \"Re-initialization\" and \"Projection Clipping\" enhancing stability.\n\nWeaknesses:\n\n1. Why is the checkpoint chosen as the anchor the original pretrained model, rather than a model that has already been partially fine-tuned on the task data?\n\n2. Even when the anchor is limited to the Query and Value matrices, roughly 16 % of the pretrained parameters must still reside in memory. While this footprint is acceptable for cloud servers, it can be prohibitive for devices with extremely limited VRAM or other edge hardware. The authors do not explore weight quantization, pruning, or other anchor-compression techniques to further reduce this overhead.\n\n3. There are now several Sparse MeZO works that achieve competitive—or even higher—accuracy. It would be valuable for the authors to include a direct comparison with these sparse approaches.", "was_revised": true, "final_justification": "Thank you for your comment. Your feedback has been addressed in the discussion thread, and I have no further questions at this time.", "rebuttal": "**We would like to thank the reviewer for the positive feedback and valuable questions. We carefully address all the reviewer’s questions and provide more results, like comparison with Sparse MeZO. We hope our response can help clarify the reviewer's questions.**\n\n---\n\n### Q1: Why is the checkpoint chosen as the anchor the original pretrained model, rather than a model that has already been partially fine-tuned on the task data?\n\nA: Thank you for your valuable question. We have tried to use a partially fine-tuned model $\\theta_{t-1}$ as an anchor, as shown in **Appendix C.3**, but using the original pretrained model yields better performance. To further address your valuable concern, we add two extra settings, using the earlier checkpoint (e.g., $\\theta_{t-500}$), and even the best-performing checkpoint ($\\theta_{t}^{best}$) as the anchor for projection. The results are shown in the following Table. Across all settings, using the original pretrained model consistently yields better performance. \nWe attribute this to the high variance and instability of intermediate checkpoints in zeroth-order (ZO) optimization. Since ZO gradients are noisy by nature, later-stage local parameters $\\theta_{t}$ may drift and become unreliable as stable anchors. In contrast, the pretrained model offers a clean and robust initialization, serving as a more reliable geometric reference for projection. This observation is also supported by recent studies on robust fine-tuning [1][2], which highlight the importance of leveraging pretrained knowledge to stabilize adaptation under noise-prone settings.\n\n**Table R.4.** Results of using different partially pretrained model as anchor.\n| Anchor           | SST-2 |           |  RTE |           |\n|------------------|:-----:|-----------|:----:|:---------:|\n|                  | Acc   | GPU hours | Acc  | GPU hours |\n| NA (MeZO)        | 90.0  | 100%      | 63.5 | 100%      |\n| $\\theta_{t-500}$ | 88.4  | 116.7%    | 61.8 | 113.0%    |\n| $\\theta_{t-1}$   | 90.7  | 87.8%     | 64.5 | 90.3%     |\n| $\\theta_{best}$  | 91.0  | 81.8%     | 66.9 | 80.0%     |\n| DiZO ($\\theta_{0}$)             | 92.5  | 55.7%     | 68.2 | 62.3%     |\n\n[1] Dong X, Luu A T, Lin M, et al. How should pre-trained language models be fine-tuned towards adversarial robustness?[J]. Neurpis 2021, 34: 4356-4369.\n\n[2] Wang S, Zhang J, Yuan Z, et al. Pre-trained model guided fine-tuning for zero-shot adversarial robustness[C]// CVPR 2024: 24502-24511.\n\n---\n\n### Q2: Even when the anchor is limited to the Query and Value matrices, roughly 16 % of the pretrained parameters must still reside in memory. Authors do not explore weight quantization, pruning, or other anchor-compression techniques to further reduce this overhead.\n\nThank you for this valuable suggestion. Our method is orthogonal to these methods and can be combined with these methods. We explored anchor compression via quantization of the pretrained Query and Value matrices to 8-bit and 4-bit precision [1]. As shown in Table R.5, DiZO can effectively incorporate with the quantization technique, still preserving advantages in both accuracy and GPU hours. Exploring more advanced quantization or compression methods to further improve anchor efficiency while maintaining performance is an important avenue for future work.\n\n**Table R.5** DiZO with quantized anchor.\n| Method                | SST-2 |           | RTE  |           |\n|-----------------------|-------|-----------|------|-----------|\n|                       | Acc   | GPU hours | Acc  | GPU hours |\n| MeZO                  | 90.0  | 100%      | 63.5 | 100%      |\n| DiZO (8-bits)   | 92.2  | 63%       | 67.2 | 71%       |\n| DiZO (4-bits)   | 91.7  | 67%       | 65.2 | 68%       |\n| DiZO | 92.5  | 56%       | 68.4 | 62%       |\n\n[1] Shao W, Chen M, Zhang Z, et al. Omniquant: Omnidirectionally calibrated quantization for large language models[J]. ICLR 2023.\n\n---\n\n### Q3: There are now several Sparse MeZO works that achieve competitive—or even higher—accuracy. If the authors can demonstrate that their method outperforms the sparse ZO approaches, I would be inclined to raise my score.\n\nA: Thank you for your insightful suggestion. Since Sparse MeZO has not released its source code, we implemented it ourselves based on the best configuration reported in the paper. Specifically, we set the sparsity ratio to 0.75, which was shown to yield the best performance. Because the original paper does not clearly specify how to determine the pruning threshold, we adopted a percentile-based strategy to achieve the target sparsity.\n\nWe conducted a direct comparison between our DiZO and Sparse MeZO in terms of both accuracy and training efficiency across two datasets and two model sizes. The results are presented in Table R.6 and Table R.7. From the accuracy perspective, DiZO consistently outperforms Sparse MeZO under all evaluated settings, demonstrating the effectiveness of our projection-based approach.\n\nFrom the efficiency perspective, Sparse MeZO requires generating the sparsity mask dynamically during training. Compared to DiZO, Sparse MeZO requires longer GPU hours for about 20%, and slows the throughput for more than 30%. Moreover, with the grows of model size, the throughput of Sparse MeZO will further decrease, due to the growing cost of maintaining and updating the sparse mask. While Sparse MeZO reduces the number of training iterations, its lower throughput results in longer total GPU hours compared to DiZO. These findings demonstrate that DiZO not only delivers better accuracy but also achieves more practical training efficiency compared to Sparse MeZO. \n\n**Table for R.6** Acc and Speed Comparison on OPT-2.7B\n| Method      | Dataset | Accuracy | Throughput | #Train Iter. | GPU Hours |\n|-------------|:-------:|:--------:|:----------:|:------------:|:---------:|\n| MeZO        |   SST2  |   90.0   |   3.3it/s  |     100%     |    100%   |\n| Sparse MeZO |   SST2  |   91.4   |   2.3it/s  |      55%     |    79%    |\n| DiZO        |   SST2  |   92.5   |   3.1it/s  |      52%     |    56%    |\n| MeZO        |   RTE   |   63.5   |   1.7it/s  |     100%     |    100%   |\n| Sparse MeZO |   RTE   |   67.1   |   1.1it/s  |      50%     |    73%    |\n| DiZO        |   RTE   |   68.4   |   1.5it/s  |      60%     |    62%    |\n\n**Table for R.7** Acc and Speed Comparison on OPT-6.7B\n| Method      | Dataset | Accuracy | Throughput | #Train Iter. | GPU Hours |\n|-------------|:-------:|:--------:|:----------:|:------------:|:---------:|\n| MeZO        |   SST2  |   90.2   |   1.8it/s  |     100%     |    100%   |\n| Sparse MeZO |   SST2  |   91.9   |   1.0it/s  |      47%     |    84%    |\n| DiZO        |   SST2  |   92.4   |   1.7it/s  |      62%     |    65%    |\n| MeZO        |   RTE   |   73.2   |   0.6it/s  |     100%     |    100%   |\n| Sparse MeZO |   RTE   |   73.8   |   0.3it/s  |      39%     |    88%    |\n| DiZO        |   RTE   |   74.8   |   0.5it/s  |      65%     |    81%    |\n\n---\n\n### Q4: The theoretical analysis is insufficient: the paper does not clearly explain why the pretrained LLM is chosen as the anchor, nor why only Q and V are projected instead of also including K and O.\n\nThank you for your insightful question. We address the two aspects below:\n- Why the pretrained model is selected as the anchor.\n   - While the anchor choice in DiZO is empirically motivated, it is supported by both theoretical intuition and experimental evidence. We have explored alternatives such as zero vectors and partially fine-tuned checkpoints (**Appendix C.3**), and consistently found that using the pretrained model as the anchor leads to better convergence and final performance. Intuitively, pretrained parameters provide a stable and informative prior in noisy ZO settings, acting as a regularization target to prevent overfitting and instability, an effect also observed in robust FO fine-tuning literature [1,2]. Although we cannot theoretically guarantee optimality, we will provide convergence analysis with this anchor choice in the revised version.\n\n- Why we project only the Q and V matrices. \n   - This decision is also empirically driven and supported by ablation studies in **Appendix C.1** (similar pattern holds across datasets). We find that projecting only Q and V strikes the best balance between performance gain and memory cost. This is consistent with the design choice in LoRA, which also focuses on Q and V as the most influential components in transformer attention for task adaptation. Including K and O introduces additional overhead but yields marginal gains. We will clarify this reasoning more explicitly in the final version.\n\nIn summary, while these design choices are currently based on empirical evidence and practical insights, they offer strong performance across diverse tasks and models. We agree that formal theoretical analysis on optimal anchor and projection selection is an interesting direction for future work.\n\n---\n\n### Q5: The performance gains come at the cost of increased memory usage.\n\nA: Thank you for raising this important concern. DiZO gains accuracy improvement and significant GPU hours reduction (overall more than 35%!) with a modest cost (7% drop in throughput and 16% increased memory overhead, or only 4% with quantization). In contrast, other ZO baselines often incur much higher costs. For example, HiZOO reduces throughput by 32% and, in theory, requires up to 100% more memory to store second-order statistics. Sparse MeZO avoids additional memory usage but sacrifices 50% of throughput. Compared to these baselines, DiZO strikes the best balance between performance, efficiency, and resource usage. Moreover, when compared to resource-intensive FO full-parameter fine-tuning and LoRA, DiZO achieves comparable accuracy while reducing memory usage by nearly 90%, highlighting its practicality for large-scale deployment."}]

Author rebuttal:

Meta-review: Zeroth-order (ZO) optimization is a promising memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource-constrained scenarios. However, ZO method lags far behind the first-order (FO) method (the conventional gradient based back propagation method) in both convergence speed and accuracy. This paper introduces a novel layer-wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. The proposed DiZO is memory-efficient and achieves performance on par with FO methods. The experiments are thorough, covering both autoencoder and autoregression models, and are complemented by detailed analyses of memory usage and computational efficiency. The reviews are overall positive.
Decision: Accept (poster)