Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
Authors: Yong Liu
Zirui Zhu
Chaoyu Gong
Minhao Cheng
Cho-Jui Hsieh
Yang You
Venue: NeurIPS 2025 poster ()
Keywords: Zeroth-Order Optimization
Parameter-Efficient Fine Tuning
Primary area: optimization

Abstract: While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-based training. Memory-efficient Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-friendly. However, compared with exact gradients, ZO-based gradients usually exhibit an estimation error, which can significantly hurt the optimization process, leading to slower convergence and suboptimal solutions. In addition, we find that the estimation error will hurt more when adding to large weights instead of small weights. Based on this observation, this paper introduces Sparse MeZO, a novel memory-efficient zeroth-order optimization approach that applies ZO only to a carefully chosen subset of parameters. We propose a simple yet effective parameter selection scheme that yields significant performance gains with Sparse-MeZO. Additionally, we develop a memory-optimized implementation for sparse masking, ensuring the algorithm requires only inference-level memory consumption, allowing Sparse-MeZO to fine-tune LLaMA-30b on a single A100 GPU. Experimental results illustrate that Sparse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead. For example, it achieves a 9% absolute accuracy improvement and 3.5x speedup over MeZO on the RTE task.

Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
Yong Liu', Zirui Zhu', Chaoyu Gong'; Minhao Cheng", Cho-Jui Hsieh", Yang You!
National University of Singapore PPennsylvania State University
University of California, Los Angeles
(liuyong, youy)@comp.. nus edu. sg, wow_ggyynussed....
Abstract
While fine-tuning large language models (LLMss for specific tasks often yields im- pressive results, comes the cost of memory inefficiency due to pack-propagation in gradient- based training. Memory- Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-ffiendll However, compared with exact gradients, ZO-based gradients usually exhibit an estimation error, which can significantly hurt the optimization process, leading to slower convergence and suboptimal solutions. In addition, we find that the estimation error will hurt more when adding large weights instead of small weights. Based this observation, this paper introduces Sparse Me70, a novel memory-efficient zeroth-order opti- mization approach that applies 70 only ters. We propose a simple yet effective parameter selection scheme that yields significant performance gains with Sparse-MeZO. Additionally, we develop a memory-optimize implementation for sparse masking, ensuring the algorithm requires only inference-level memory consumption, allowing Sparse-MeZO fine-tune LLaMA-300 on single A GPU. Experimental results illustrate that Searse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead. For example, it achieves 9% absolute accuracy improvement and 3.5x speedup over McZO on the RTE task. Code is available com/NUS- NUSS- HPC- AI LabLLaaaaaaaa SSaaaaaeaaa........
carefully chosen subset of parame-
https://gt /ttp://ggtttt.t.// com
1 Introduction
Fine-tuning large language models for specific tasks datasets has become prevalent practice in machine learning. However, a major obstacle in fine-tuning is the substantial memory requirements, which escalate models increase in size and complexity, thereby limiting the scalability and accessibility for those with limited computational resources.
To mitigate the memory constraints, Parameter Efficient Fine- Line-Tuning (PEFT) has been developed, allowing for the modification of only a subset of parameters and achieving comparable results to full model tuning (Hu et al., 2021; Lester al., 2021; Li & Liang. 2021; Zaken al., 2021; Zhang et al., 2023). However, PEFT methods still necessitate the calculation of gradients for backpropagation and caching of numerous activations during training, which introduces additional memory overhead, For instance, Malladi et al. demonstrates that, even with PEFT, training still requires approximately times more memory than the memory cost for inference. This discrepancy raises critical question: Can large language models be fine-tuned solely with the cost of inference?
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

In response to these challenges, zeroth-ories (ZO) optimization presents a promising solution (Spall, 1992). ZO optimization gradient-free method that estimates gradients using only the forward pass of the model, eliminating the need for backpropagation and, consequently, reducing memory usage. Me70 (Malladi al., 2023) a recently proposed zeroth-order method for fine-tuning L.L.s that has demonstrated impressive performance.
However, compared exact gradients, ZO- based gradients usually exhibit an estimation error, which can defined as noise. This noise can significantly hurt the optimization process, leading to slower convergence and suboptimal solutions. Moreover, we find that the estimated 70 gradient difficult to generalize across batches. Specifically, while it can successfully reduce the training loss on the sampled batch with a high probability, it is more likely to in- crease the loss on other batches.
MezO
0.80
S:MeZO
0.75
3.5x Speedup
Accuracy
0.70
0.65
0.60
0.55
To address this challenge, we investigate the im- pact of gradient noise in zeroth- order optimiza- tion for LLM fine-tuning. We measure how the noise affects optimization by evaluating its ef- fect generalization performance across differ- data batches. Interestingly, our experiments reveal that the noise has more significant im- pact when added to large weights compared to small weights. Based this finding, we propose novel sparse memory efficient zeroth-order method (Sparse-MeZO) to selectively optimize small weights, which are more resilient to noise perturbation. By focusing on these noise-resistant weights, we demonstrate that our method enables the use of larger learning rates, leading to improved performance and faster convergence. Our contributions can be summarized as follows:
75 100
125
150
175
200
Step (*100)
Figure 1: Performance McZO and Sparse- MeZO (S-MeZO) RTE task. S-MeZO can achieve 3.5x speedup compared with MeZO.
In this paper, investigate the impact of gradient noise in zeroth-order optimization for LLM fine-tuning. Our evaluations show that the gradient noise can make the estimated ZO gradient difficult generalize across batches and the noise will hurt more when adding large weights instead of small weights.
Based on the above finding, we propose a sparse Memory-Effine Effficiet Zeroth- Order optimization method Sparse-Mede MeZO (S (S-MeZO) for large language model fine-tuning, We also provide theoretical analysis show the convergence of Sparse-MeZO.
Different from the efficient implementation with random seed in MeZO, we propose novel memory-efficient implementation of Sparse-MeZO, which can compute the sparse mask and perturb parameters in the forward pass. The technique enables fine-tuning LLaMA-30b with Sparse-MeZO on a single A 110 GPU.
We conduct empirical studies on LLaMA, OPT, and Mistral. The experimental results demonstrate that Sparse-MeZO can improve fine-tuning performance and yield faster convergence rate compared with vanilla MeZO across wide range of natural language processing tasks. For example, it achieves a 9% absolute accuracy improvement and 3.5x speedup over McZO on the RTE task, as shown in Figure 1.
Preliminaries
2.1 Parameter-Efficient Fine-Tuning
Parameter-Efiice Fine- Tuning (PEFT) is designed to facilitate efficient adaptation by updating only subset of the model's parameters, rather than fine-tuning the entire model (Hu et al., 202 Zaken et al., 2021; Pan et al., 2024; Liu et al., 2025). These PEFT approaches can be categorized into selective and additive methods.
Selective Methods. Selective Methods try to selectively fine-tune a portion of model and these methods have been explored in various studies. For example, Zaken et al.; Cai et al. focused on the

---

model's 's bias terms, finding that fine-tuning these terms alone could rival the results of fine-tuning the entire model. However, the effectiveness of this approach diminishes with larger datasets, as shown in further analysis by 7aken et al.. Beyond static parameter adjustments, there has been an exploration into dynamically modifying parts of the model (Brock et al., 2017). This concept was later applied to language models, with AutoFreeze (Liu et al., 2021b) confirming its viability.
Additive Methods. Additive methods, as an alternative to updating existing parameters, involve incorporating new layers into models, with the fine-tuning process focusing solely on these added layers (Houlsby et al., 2019; Hu et al., 2021; Lin et al.. 2020; Rebuffi et al., 2017). Traditional techniques in this category, such as adapters (Houlsby et al., 2019), implemented layer additions in a sequential manner, which unfortunately led to increased inference latency. LoRA (Hu et al., 2021) has been proposed to mitigate this issue, which freezes the weights of the rre-trained model and introduces trainable matrices based on rank decomposition into each layer. IA3 (Liu et al., 2022) introduced innovative methods for adding parameters, balancing parameter count with accuracy, while LST (Sung et al., 2022) introduced a highway structure that learns only small, auxiliary channels.
2.2 Zeroth-Order Optimization
Unlike traditional tradient-basedd optimization methods that rely on derivatives to guide the search for optimal solutions, Geroth-Order (ZO) optimization techniques do not require derivatives for optimization Spall (1992); Liu et al. (2018, 2019); Shu et al. (2023). These methods utilize only the value of the objective function, denoted as (ax) at any chosen point æ. To estimate the gradient in the direction of vector the objective function is assessed at two points in close proximity, f( (zz) and fax cz), with c being a minimal value. Following this, conventional optimization algorithms, such as gradient descent or coordinate descent, are implemented using these approximated gradient values. Currently, ZO methods have been widely used various applications, such as adversarial attack and defense (Chen et al., 2017: Ilyas et al., 2018; Tu et al., 2019; Ye et al., 201 8), Auto-ML (Ruan et al., 2019; Wang et al., 2022), natural language processing (Sun et al., 2022a.b). reinforcement learning (Vemula et al., 2019), Signal Processing (Liu et al., 2020), and on-chip training (Gu et al., 2021).
2.2.1 MeZO
ZO-SGD employs SPSA (Spall, 1992) to estimate the gradient. In general, conventional ZO-SGD algorithms utilizing SPSA consume twice the inference memory. McZO (Malladi et al., 2023) is memory-efficient variant of ZO-SGD. It circumvents the storage of gradients by saving the random seed and resampling the same random noise z with the seed during forward process. More specifically, to calculate L(O + cz) - C(O cz), MeZO will sample a noise z to perturb 0 to 0 + ez and then calculate L(e + ez). Then it resamples the same noise with the same seed and move the parameter back 0 - ez and calculates the loss. As a result, the zeroth-order gradient estimator can be computed without any memory overhead, leading to numerous variants of MeZO being proposed in the literature Zhang et al. (2024); Yang et al. (2024); Jiang et al. (2024); Chen et al. (2024); Wang et al. (2024); Yu et al. (2025); Tan al. (2025); Sun et al. (2025); Zhang et al. (2025); Zhou et al. (2025).
2.2.2 Sparsity for Zeroth-order Optimization
The hypothesis proposed by Frankle & Carbin, known as the lottery ticket hypothesis, showed that within densely connected neural network that is randomly initialized, there exists subnetwork of sparse yet high-quality connections. Based on the hypothesis, model pruning aims to identify and preserve the crucial 'winning tickets' sparse subnetworks within the larger neural network that can achieve comparable or even superior performance, such as Wanda (Sun et al., 2023) and GPa (Frantar & Alistarh, 2023). In addition, Dynamic Sparse Training (DST) has been proposed to reduce the training and inference cost in first-order optimization (Liu et al., 202 1a; Evei et al., 2020). Recently, several related works have tried to apply the sparsity to zeroth-order optimization (Balasubramanian & Ghadimi, 2018: 8; Cai et al., 2021, 2022; Chen et al., 2023; Gu et al., 2021; Ohta et al., 2020; Wang et al., 2018). For example, DeepZero (Chen et al., 2023) proposes a novel ZO training protocol with model pruning guided sparsity.
3

---

CUETEIT BURET
2.75
0.70
meeahii of Lon loreee
woither bance
0.5
5.65
0.00
5.00
0.25
0.30
6.30
0.45
2.45
100 125
153
10
20
40
52
60
70
-0
3000
122
Step 27000
Skep (*3000)
Step (7100)
(a) RTE-LR
(b) RRE-Baace
(c) RTE -Tg--tgt
Figure 2: (a) Test Accuracy with Different Learning Rates on RTE Task. We find McZO is very sensitive to the selection of learning rate. Even small increase from x 10 to 2 10-5 causes divergence and instability. (b) Probability of Loss Increase on Different Batch. We find the estimated ZO gradient can successfully reduce the loss the same batch but may be difficult decrease the loss on the new held-out batch. (c) Continuing training from the drop point with small and large weights. We find that optimizing only small weights can recover and further improve test accuracy.
3 Proposed Method
3.1 Empirical Observation on MeZO
For large language models, zeroth-order optimization algorithms like MeZO are often necessary when exact gradients are unavailable prohibitively expensive to compute. However, compared with exact gradients, these methods inherently introduce noise in the gradient estimates used for optimization.
where L the loss function. As shown in Figure 2(a), McZO exhibits extreme sensitivity the choice of learning rate. Even a small increase from 1 x 10 to 2 10 causes divergence and instability, while this larger learning rate totally fine when fintuning with first-order methods. This suggests that the gradient noise introduced by the zeroth-order approximation, defined as ō = (f) 9= (0) where g (0) the exact gradient, significantly hinders the optimization process when large step sizes are used. This motivates to analyze the effects of this gradient noise ō and understand how it impacts optimization performance.
Specifically, the zeroth-order gradient 9= (0) is approximated as 9= (0) ((0+ex)-Elt -((e=e)
2r
To quantify how the gradient noise d hurts the optimization process, we evaluate its effect on the generalization performance of the estimated gradients. Specifically, we measure whether the zeroth- order gradient estimate computed on one batch can effectively reduce the loss on other held-out batches. For a batch B, = (B) B,), with 32 data points, we use 16 samples to estimate the zeroth- order gradient 9= (0; B3) on batch B, and evaluate it on the remaining 16 held-out samples B7. The results are shown in Figure 2. Interestingly, we find a stark contrast in performance while the estimated gradient 9= (0; B!) can reliably reduce the loss on the same batch B it was computed on (90% success rate), only manages to decrease the loss on the new held-out batch B² around 50% of the time. This suggests that the zeroth-order gradient estimates suffer from overfitting or noise that makes them less generalizable to unseen data samples. The gradient noise d, while allowing descent on the current batch, appears to introduce errors that prevent reliable descent directions for unseen batches. Therefore, the noise ō can be seen as hurting the optimization process by degrading the generalization performance of the parameter updates.
Next, we aim understand if this effect is uniform across all model parameters or if certain parameter groups are more vulnerable to noise corruption. We notice the nature of vanilla Me7O, where ((0+c=)-E(0-e) is used to estimate the gradient, and all parameters share the same value of
2x
C(0+cs)-C(0-c0 This means not all parameters are optimized in the true gradient direction, which could be a limitation. To analyze this, we divide the parameters into different groups based on their magnitude the top 20% largest weights are considered "large". while the bottom 20% are "small". Interestingly, our experiments reveal that the gradient noise d hurts optimization more when added to large weights compared to small weights. As shown in Figure 2(c), when continuing training from the point where test accuracy drops (due to noise), we find that optimizing only the small weights can recover and further improve test accuracy. Therefore, the experimental illustrate that noise 8 does

---

Finally, this gradient can be not impact all parameters equally it disproportionately disrupts the optimization of larger weights. Selectively optimizing smaller, noise-resilient weights may be a promising direction to mitigate the effects of gradient noise in zeroth-order optimization. In the next section, we will introduce the proposed Spare-MeZO algorithm, which can only select small weights to perturb and update weights.
3.2 Sparse-MeZO
Consider a labelled dataset D = {( wetep and let B)) denotes the loss on a mini- batch B. We can define sparse mask m E {0, 1]d to selectively sample the random noise Rd with 2 ~ (O, Ia) on the sub-net of pre-trained model. A sparsified version of random perturbation can be defined as 2 E Id:
m=
Based on this sparse perturbation 2. can redefine MeZO algorithm on Section 2.2.1 as Sparse- MeZO. The main difference is from the estimated gradient g (0), which can be defined as
ga (0) = C(O + e2; B) C( <2; B)
2
2e
C(0 + em 3 z: B) L(O em 1 Z;
2e
where c represents the perturbation scale. Based on the observations from our motivation, we can create sparse mask, m. determined by parameter magnitudes. Specifically, we only update parameters of smaller magnitude. These targeted parameters are defined as θ m @ 0. It's important to note that we still preserve the complete set of parameters, but we apply sparse perturbations and gradient estimations only to the selected ones. This approach allows to integrate the sparse mask into the standard MeZO method as a straightforward, adaptable tool. Then, we will introduce when and how calculate the mask.
Setting the Mask Before Training. We compare the parameter values threshold for each layer to set the mask before training begins. However, a significant downside of this approach is the extra memory required to store sparse mask, which is as large as the pre-trained model itself. Our goal is for our method to enhance performance without using more GPU memory or causing extra overhead.
Constant Mask:
Dynamic Mask: Determining Mask at Each Iteration. We can establish threshold for each layer before training and then generate the mask by comparing parameter values to this threshold during each iteration. This method avoids the necessity of storing a large mask, m.
In this paper, we'll employ dynamic mask to choose which parameters perturb and update, addressing the issue of memory constraints. In addition, we determine thresholds using a principled sparsity- -ased approach. Specifically, based on a target sparsity level.
a percentile-based method where the threshold is set
The pseudo-code is provided in Algorithm 1. This algorithm outlines that we first establish the threshold he for each layer before beginning training. We then use GetMask (Algorithm 3) to compare each parameter against its threshold h; and create the mask m. Following this, we introduce the function PerturbParameterr (Algorithm 2) to generate a Gaussian noise sample z ~ N/(0, (0, Ia) and apply the mask to produce sparse perturbation & m 3 z. With &, we perturb the current parameters O, to get new parameters 0₂ + E& and 0₂ - eλ. This allows us to compute two distinct loss values: 1 L(O, cε) and ! L(e, cε). From these losses, we calculate the estimated sparse gradient gm (0,) = proj_grad * ε, where proj grad used with learning rate 7): to update Or.
--1_
2c
3.3 Memory-Efficient Implementation of Sparse-MeZ0
In this paper, our primary aim to introduce an efficient method for fine-tuning language models using zeroth-order optimization, enhancing performance on downstream tasks. As outlined in Algorithm 1, our approach involves perturbing the parameters O, twice to generate two distinct sets of parameters, 0g Oz ez and 0 Or We then the estimated gradient to update the
5

---

Algorithm Sparse-MeZ0 (S-MeZO)
Require: 0 represents pre-trained LLM weight, N is the number of layers in model, learning rate Tt, s represents sparsification interval.
Initialize random seed s Determine threshold h = , hN, of each layer with the sparsification interval for / 1 to T do
Sample Minibatch B from X and random seed 8.
m GetMask((O,. h)
0t - PerturbParameters(@e...c e, m)
[ [(,,; B)
0. + PerturbParametters(. -2, m)
1- L(O,: ())
0. + PerturbParameters 0 c, m))
proj_grad grad - (1+ (2cc)
Reset random seed
for 02 E 0 do
~ (0,
0, < 0, 7h proj grad x my *
end for end for
original parameters Oz. This step typically requires storing two separate sets of parameters, leading to increased memory usage during fine-tuning.
Recently proposed MeZO, conserves memory by saving random seeds and using it to resample for calculating f, 0 and reconstructing 0, without needing extra memory. However, applying Me70 poses memory issue. We cannot
sparse mask m calculating sparse perturbation simply reconstruct 2 by saving the random seed because the sparse mask, determined by parameter magnitudes, changes when parameters are altered by the perturbation. To address this, we propose potential solutions for the memory issue.
1-bit Quantization: We can apply I-bit quantization store the mask m, as it consists solely of Os and Is. However, this method still increases memory usage, which isn' 't our goal. As a solution, we introduce a novel, memory-saving approach for zeroth-order optimization that calculates the mask m on the fly during the forward pass.
By computing the mask and perturb parameters in the forward pass, we eliminate the need to store perturbed parameters 0 and 0 This means only have to keep the original parameters 0, throughout training. For vanilla implementation, we first need to calculate the perturbed parameters with mask m: 0g = 0. + em @ z. After that, we can use perturbed parameters 0g calculate the loss value (+ with the forward process. For example, the output of layer can be defined as (+) = 0² (+) + ((),, Noted that we need to the vanilla parameters O₂ and mask m for vanilla implementation. However, for our proposed efficient implementation, we only need to save vanilla parameters Oz. More specially, we can calculate the mask m (+) layer during the forward process and then obtain the output of this layer: (i) (o + em(0t) (z) b(i,, where m(-) represents GetMask to calculate mask (i), Then, can release the memory of mask m and calculate the output and mask of the next layer.
Calculating the Mask During the Forward Pass:
4 Experiments
Following similar setting to MeZO, we evaluate the performance of our proposed method on SuperGLUE Wang et al. (2019).
4.1 Experimental Setting
Datasets. To verify the performance gain of our proposed method, we conduct experiments on various fine-tuning tasks include SST-2 (Socher et al., 2013), RTE (Bentivogli et al., 2009; Dagan et al., 2005;

---

2.42
0.20
0.64
0.75
1.62
1.02
1.35
MM
2.3
0.50
5.54
0.35
1.32
303 :25 Step (7100)
132
200
173
122 125
Step M100
Step P1001
(a) RTB
(b) BoolQ
(c) WIC
Figure 3: Convergence Curves of Fine-Tuning LLaMA-7b with MeZO and Sparse- MeZo (S (S-MeZO)
on (a) RTE, (b) BoolQ, (c) WIC tasks.
Giampiccolo et al., 2007), BoolQ (Clark et al.. 2019), WIC (Pilchvar & Hemacho-Colladeee 2018),
MultiRC (Khashabi et al., 2018)) and multi-class task COPA (Roemmele et al., 2011).
Models. We primarily use pre-trained LLaMA-7b Touvron et al. (2023) to evaluate the performance of our proposed method on downstream tasks. To further demonstrate our method's versatility, we Jiang al. (2023) and OPT-13b Zhang al. (2022). We provide
also test it with Mistral-7B-vol more details about the results in the Appendix E. Additionally, to examine our method's scalability, evaluate it larger models, such as LLaMA-30b.
Baselines. First, we compare our method the vanilla MeZO to demonstrate how sparsification enhances MeZO's convergence speed and overall performance. Additionally, to show that our proposed S-MeZO effectively identifies and modifies crucial parameters, we contrast it with R- MeZO (a version of McZO applying random mask to select parameters for optimization). In addition, we also explore the impact of zero-shot optimization on improving a pre-trained language model'ss capabilities through experiments with zeroth-shot learning and in-context learning techniques (Brown et al., 2020). Lastly, to understand the performance gap between zeroth-ordes and first-order optimization in fine-tuning large language models, present results from conventional full -araameer fine-tuning (FT) using the Adam optimizer, the most widely used method for such tasks. In addition, also compare MeZO and its variants against LoRA, the most widely adopted PEFT method.
Training Procedure. We adopt most of the training hyperparameters from the standard MeZO, including dataset configuration, batch size, training epochs, epsilon value, and task prompts, with the key difference being a higher learning rate for S-MeZO due to updating only a subset of the parameters (Zhou et al., 2023). We perform the experiments using three different seeds and report the average the outcomes.
4.2 Performance on SuperGLUE
To evaluate the performance of our proposed method S-McZO, we initially tested it on the SuperGLUE benchmark using the LLaMA-7b model. The fine-tuning results, presented in Table 12, reveal that S-MeZO method outperforms other zero-order (ZO) techniques like MeZO and R-MeZO. For instance, S-MeZO boosts MeZO's accuracy from 71.7% to 80.7% on RTE (1 9%) and from 75.9% to 80.9% on BoolQ (1 5%). Furthermore, all zeroth-order-be braer-beddd methods surpassed the performance of Zero-shot learning and in-context learning, demonstrating that zeroth-order optimization significantly enhances the pre-trained model's effectiveness downstream tasks. Finally, we can find S-MeZO significantly bridges the performance gap between zero-order and first-order optimization methods.
To further verify the generality of proposed S-McZO, also evaluate it on Mistral-7B--O. The performance is shown Table 11. We can find that S-MeZO can consistently improve the perfor- mance of vanilla MeZO and narrow down the performance gap between zeroth-order optimization and first-order optimization. For example, S-MeZO can improve the accuracy vanilla MeZO from 81.6 to 85.3 BoolQ and then achieve comparable performance with full fine-tuning.
We conducted additional evaluations of S-MeZO on LLaMA2-7h and compared against an expanded set of baseline methods. These include methods proposed by Zhang et al. (Zhang al., 2024) such as 7O-SGD-Cons, ZO-SGD-Sign, and 70-SGD-Adam, as well as other significant baselines including ZO- Ad-AdaUU (Jiang et al., 2024) and AdaZeta (Yang et al., 2024). The results shown in the Table
7

---

Method
BoolQ
RTE
WIC
MultiRC
SST-2
COPA
Average
Zero-Shot
65.1
49.5
50.6
55.8
79.7
59.7
60.1
ICL
67.4
54.5
52.7
58.7
81.2
84.4
66.5
LoRA
84.5
82.3
67.6
78.3
95.0
86.0
82.3
FT
84.5
83.6
68.4
80.2
95.7
85.0
82.9
MeZO
75.9
71.7
61.4
69.8
94.6
86.3
76.6
McZO-LoRA
77.9
71.9
60.8
72.6
95.0
84.3
77.6
R-MeZO
76.9
75.2
62.1
68.1
94.6
84.3
76.9
S-MeZO
80.9 (1:50)
80.7
(1.9.0)
64.9 (1:35)
73.3 (1:35)
95.0
(1.4)
86.7 (1044)
80.3
Table I: Accuracy of Fine-Tuning LLaMA-7b on SuperGLUE (1,000 examples). ICL: In-Context Learning, FT: full-parameter fine-tuning with Adam, R-MeZO: MeZO with Random Mask.
We find that S-MeZO consistently outperforms other zeroth-order methods, improving MeZO's accuracy from 78.8% to 82.2% BoolQ (t 3.4%) and from 70.2% to 77.6% on RTE (f 7.4%). While LoRA achieves the highest average accuracy 83.0%, S-Me70 significantly narrows this gap with an average of 80.0%, surpassing both Me70 (76.3%) and R-Me70 (76.6%).
Model
Method
BoolQ
RTE
WIC
SST-2
Average
LLaMA2-7b
LoRA
84.0
83.2
68.8
96.0
83.0
LLaMA2-bb
MeZO
78.8
70.2
62,2
94.0
76.3
LLaMA2-7b
MeZO-LoRA
80.3
76.5
63.0
94.0
78.5
LLaMA2-7b
ZO-SGD-Cons
77.1
63.0
94.0
LLaMA2-7b
ZO-SGD-Sign
68.2
61.0
55.6
82.8
66.9
LLaMA2-7b
ZO-SGD-Adam
78.9
73.6
62.1
93.8
77.1
LLaMA2-7b
ZO-AdaMU
78.2
76.5
63.0
93.6
77.8
LLaMA2-7b
AdaZeta
79.8
75.8
62.0
94.0
77.9
LLaMA2-7b
R-MeZO
76.7
74.0
61.4
94.2
76.6
LLaMA2-7b
S-MeZO
82.2 3.4)
77.6
7.4)
65.3 (1 3.1)
94.8 (1 0.8)
80.0 (1 3.7)
Table 2: Accuracy of Fine-Tuning LLaMA2-7b SuperGLUE (1,000 examples).
4.3
Performance on Commonsense Reasoning and Mathematics Tasks
To further verify the performance of Sparse MeZ0 on more challenging tasks, we have con- ducted additional experiments commonsense reasoning tasks (PIQA, SIQA, BoolQ) and mathematies task (AQuA (Ling al., 2017)), consistent with the evaluation protocols used in SensZOQ (Guo al.).
Method
BoolQ
PIQA
SIQA
AQuA
MezO S-MeZO
84.3
68.5
24.0
79.2
85.3
70,2
26.6
Table 3: Accuracy of Fine- Fine-Tuning Mistral-7B Challenging Tasks.
The results, presented in the Table 3, demon- strate that Sparse MeZO (S-MeZO) consistently outperforms Me7O across all these more complex tasks. In particular, we observe substantial im- provements the AQuA mathematics task (+2.6%) and the SIQA commonsense reasoning task (+1.7%), further validating the effectiveness of our approach across diverse tasks.
4.4 Convergence Rate
To verify that S-MeZO converges faster than MeZO, carried out multiple experiments for compar- ison. The accuracy over steps plotted Figure 3, which shows that S-MeZO can use fewer steps to achieve a better performance than vanilla MeZO. For example, S-MeZO only needs about 5,000

---

steps to achieve 70% accuracy but vanilla MeZO needs 17,500 steps. Finally, S-MeZO can achieve about 3.5x speedup on RTE and 3x speedup on BoolQ.
4.5 Memory Usage
Table 4 shows the memory consumption for MeZO, S-MeZO, and traditional full-parameter fine- tuning of LLaMA-7b. The data reveal that S-MeZO does not require more memory than MeZO and offers substantial saving of roughly 12 times less GPU memory compared to full- parameter fine- tuning. For instance. S-MeZO with Efficient Implementation (S-MeZO-EI) cuts down the memory needed from 158.6 GB for full tuning to just 14.6 GB MultiRC task. In addition, S-MeZO with efficient implementation can reduce the memory cost from 28.3 GB of vamilla S-MeZO 14.6 GB across all five tasks, which also illustrates the efficiency of our proposed implementation method: Calculating the Mask During the Forward Pass. As result, we can use only inference memory cost to fine-tune large language models.
Method
SST-2
RTE
BoolQ
WIC
MultiRC
CОРA
Average
FT
114.7
123.7
128.7
115.3
158.6
119.1
128.2
LoRA
15.7
25.5
16.1
34.2
23.1
22.4
MeZO
14.6
14.6
14.6
14.6
14.6
14.6
14.6
S-MeZO
28.3
28.3
28.3
28.3
28.3
28.3
28.3
S-MeZO-EI
14.6
14.6
14.6
14.6
14.6
14.6
14.6
Table 4: Memory Usage (batch size 1) of Fine- LLaMA-7b on SuperGLUE (1,000 examples). El represents the Efficient Implementation in section 3.3.
4.6
Sparse Rate
For S-MeZO, we need to define the sparsity of the pre-trained model before starting to fine-tune To analyze the effects of sparsity value the performance, conduct experiments with various sparsity values (from 0.0 to 0.8). Table 10 summarizes these experimental results with different sparsity values. We can find that a significant performance gain can be obtained when use the sparsity value from 0.5 to 0.8. In addition, for most tasks, a sparsity value of 0.8 usually means better performance. For example, S-MeZO improve the accuracy from 71.7% to 82.3% (when r = 0.8). It can also obtain performance gain of 6.6% for BoolQ (from 75.9% to 82.5%).
4.7 Scalability
In Table 12, mainly introduce the perfor- mance of our methods on LLaMA-7b. A direct question is whether scale to larger language models. Therefore, in this section. we further explore our proposed method S-MeZO on LLaMA-30b. As shown in Table can see that the larger model usu- ally can obtain a better fine-tuned performance. For example, the accuracy on RTE with MeZO can be improved from 71.1% LLaMA-7b to 76.9% on L.LaMA-30b Our method S-MeZO can further improve the performance on RTE to 82.1% on I.L.MA300b. In addition, S-MeZO can further improve the accuracy on BoolQ to 85.7% on LLaMA-30b.
Model
Method
BoolQ
RTE
WIC
proposed method can
LLaMA-7b
MeZO
71.7
61.4
LLaMA-7b
S-MeZO
80.9
80.7
64.9
LLaMA-30b
MeZO
83.8
76.9
63.3
LLaMA-30b
S-MeZO
85.7
82.1
67.3
Table 5: Accuracy of Fine- TineTTning LLaMA-7b and LLaMA-30b on SuperGLUE.

---

Conclusion
In this paper, we propose novel memory- efficient zeroth-order fine-tuning method Sparse-MeZO, which can use similar memory cost to the inference process. We evaluate the performance of fine-tuning LLaMA and OPT with Sparse-MeZO on SuperGLUE benchmark and the experimental results illustrate that Sparse-MeZO can achieve higher accuracy and faster convergence. Finally, we can fine-tune LLaMA-300 on a single A100 GPU.
Limitation: There is still a performance gap between our proposed method Sparse-MeZO and first-order fine-tuning methods. We plan address these limitations and enhance Sparse- MeZO's capabilities in our future research and conduct more experiments on state-of-the-ard pre-trained language models.
Acknowledgements
Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professor- ship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant and Alibaba grant. This work is also supported by NSF 2048280, 2325121, 2244760, 2331966 and ONR N0001423-1- 2300:P00001.
References
Krishnakumar Balasubramanian and Saeed Ghadimi. Zeroth-order (non)-convex stochastic optimiza- tion via conditional gradient and gradient updates. Advances in Neural Information Processing Systems, 31, 2018.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1 2009.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Freezeout: Accelerate training by progressively freezing layers. arXiv preprint arXiv: 04993, 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 2020.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce activations, not trainable parameters for efficient on-device learning. arXiv preprint arXiv:2007. 2020.
HanQin Cai, Yuchen Lou, Daniel McKenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-bor box optimization. International Conference on Machine Learming, pp. 193-1203. PMLR, 2021.
HanQin Cai, Daniel Mckenzie, Wotao Yin, and Zhenliang Zhang. Zeroth-order regularized optimiza- tion (zoro): Approximately sparse gradients and adaptive sampling. SIAM Journal on Optimization, 32(2):687-714, 2022.
Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. arXiv preprint arXiv:2310.02025, 2023.
Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsich. Zoo: Zeroth order optimization based black-box attacks deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp- 15-26, 2017.
Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing ooderh-order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.076698, 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova, Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 10044 2019.
10

---

Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer. 2005.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International conference machine learning, pp. 2943-2952. PMLR, 2020.
Jonathan Frankle and Michael Carbin, The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 2018.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models be accurately pruned in one-shot. In International Conference on Machine Learning, pp. 10323-10337. PMLR, 2023.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third paseal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL. workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Jiaqi Gu, Chenghao Feng. Zheng Zhao, Zhoufeng Ying, Ray T Chen, and David Z Pan. Efficient on-chip learning for optical neural networks through power-aware sparse zeroth- optimization. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 7583-7591, 2021.
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, et al. Zeroth-order fine-tuning of ilms with transferable static sparsity. In The Thirleenth International Conference on Learning Representations.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2090-279999 PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 (06.082/06.0968,, 2021.
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pp. 2137-2146.
PMLR, 2018
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Shuoran Jiang, Qingcai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18363-18371, 2024.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 252-262, 2018.
Brian Lester, Rami AI-Rfou, and Noah Constant, The power of scale for parameter-efficient prompt tuning. arXiv preprini 10X. 10400669,, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.0012 2021.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parfeeter--ffficient transfer learning. arXiv preprint arXiv:2004.03829, 2020.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. arXiv preprint I 146, 2017.
11

---

Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin A Raffel. Few-shot parameter-effiient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 15:1950-1965, 2022.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense wver-parameterizationn in-time over-paramettrization in sparse training. In International Conference on Machine Learning, pp. 6989-7000. PMLR, 2021a.
Sijia Liu, Bhavya Kailkhura, Pin- Yu Chen, Paishun Ting, Shiyu Chang, and Lisa Amini. Zeroth- order stochastic variance reduction for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
Sijia Liu, Pin-Yu Yu Chen, Xiangyi Chen, and Mingyi Hong- signsgd via zeroth-order oracle. In International conference on learning representations. International Conference on Learning Repre- sentations, ICLR, 2019.
Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43-54, 2020.
Yong Liu, Di Fu, Shenggan Cheng, Zirui Zhu, Yang Luo, Minhao Cheng, Cho-Jui Hsieh, and Yang You. SeedloRA: A fusion approach to efficient L.LM fine-tuning. In Forty-second International Conference on Machine Learning, 2025.
Yuhan Liu, Saurabh Agarwal, and Shivaram Venkataraman, Autofreeze: Automatically freezing model blocks to accelerate fine-tuning. arXiv preprint arXiv:2102.01386, 2021b.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. fine-tuning language models with just forward passes. arXiv preprint 2rX1v:2305.17333, 2023.
Mayumi Ohta, Nathaniel Berger, Artem Sokolov, and Stefan Riezler. Sparse perturbations for improved convergence in stochastic zeroth-order optimization. In Machine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19-23, 2020, Revised Selected Papers, Part Il 6, pp. 39-64. Springer, 2020.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning. Advances in Neural Information Processing Systems, 111577718774455 2024.
Mohammad Taher Pilehvar and Jose Camacho-Collados. Wie: the word-in-context dataset for
evaluating context-sensitive meaning representations. arXiv preprint arXiv: /808.0888//11// 2018.
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in neural information processing systems, 30, 2017.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
Yangjun Ruan, Yuanhao Xiong, Sashank Reddi, Sanjiv Kumar, and Cho-Jui Hsieh. Learning to learn by zeroth-order oracle. arXiv preprint arXiv: 1eXiv:1910.0900 1910.0946, 2019.
Yao Shu, Zhongxiang Dai, Weicong Sng, Arun Verma, Patrick Jaillet, and Bryan Kian Hsiang Low. Zeroth-order optimization with trajectory-ifformedd derivative estimation. In The Eleventh International Conference on Learning Representations, 2023.
Richard Socher, Alex Perelygin, Jean Wu. Jason Chuang. Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
James Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions automatic control, 37(3):332-341, 1992.
12

---

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695. 2023.
Tianxiang Sun, Zhengfu He, Hong Qian, Yunhua Zhou, Xuan-Jing Jing Huang, and Xipeng Qiu. Bbtv2: towards a gradient-free future with large language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1916-3930 2022a.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model--ee modee-ss-ddll-------eee--------- assae-ervieee. In International Conference on Machine Learning, pp. 20841 20841-20855. PMLR, 2022b.
Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, and Dacheng Tao. Tezo: Empowering the low- rankness on the temporal dimension in the zeroth-order optimization for fine-tuning llms, arXiv preprint arXiv:2501. (9f1:2550/.9057, 2025.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35:12991-13005. 2022.
Qitao Tan, Jun Liu, Zheng Zhan, Caiwei Ding, Yanzhi Wang, Jin Lu, and Geng Yuan. Harmony in divergence: Towards fast, accurate, and memory-efficient zeroth-order IIm fine-tuning. arXiv preprint arXiv:2502.03304 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2 1371,, 2023.
Chun-Chen Tu, Paishun Ting, Pin-Yo Chen, Sijia Liu, Huan Zhang, Jinfeng Yi, Cho-Jui Hsieh, and Shin-Ming Cheng. Autozoom: Autoencoder-bante zeroth order optimization method for attacking black-box neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 742-749, 2019.
Anirudh Vemula, Wen Sun, and Bagnell. Contrasting exploration in parameter and action space: A zeroth-order optimization perspective. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2926-2935. PMLR, 2019.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Fei Wang, Li Shen, Liang Ding, Chao Xue, Ye Liu, and Changxing Ding. Simultaneous computation and memory efficient zeroth-order optimizer for fine-tuning large language models. arXiv preprint arXiv:2410.0982. 2024.
Xiaoxing Wang, Wenxuan Guo, Jianlin Su, Xiaokang Yang, and Junchi Yan. Zarts: On zero-order optimization for neural architecture search. Advances in Neural Information Processing Systems, 1286 2022.
Yining Wang, Simon Du, Sivaraman Balakrishnan, and Aarti Singh. Stochastic zeroth-order opti- mization in high dimensions. In International conference on artificial intelligence and statistics, pp. 1356-1365. PMLR, 2018.
Yifan Yang, Kai Zhen, Ershad Banijamal, Athanasios Mouchtaris, and Zheng Zhang. Adazeta: Adaptive zeroth-order tensor-train adaption for memory-efficient large language models fine- tuning. arXiv preprint arXiv:2406.18060 2024.
Haishan Ye, Zhichao Huang, Cong Fang, Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth- order optimization for black-box adversarial attack. arXiv preprint 1 1377, 2018.
Ziming Yu, Pan Zhou, Sike Wang, Jia Li, Mi Tian, and Hua Huang. Zeroth-order fine-tuning of llms in random subspaces. Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 4475-4485, 2025.
13

---

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-modes. arXiv preprint arXiv:2106.10199 2021.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512. 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trate me-trained transformer language models. arXiv preprint arXiv:2205.0n 01066, 2022.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenging Zheng, Pin-Yo Chen, Jason Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang. Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Proceedings of the 41st International Conference on Machine Learning. volume 235 of Proceedings of Machine Learning Research, pp. 59173-5911999 PMLR, 21-27 Jul 2024.
Zhen Zhang, Yifan Yang, Kai Zhen, Nathan Susanj, Athanasios Mouchtaris, Siegfrried Kunzmann, and Zheng Zhang. Mazo: Masked zeroth-order optimization for multi-task fine-tuning of large language models. arXiv preprint 250.: 2025.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less more for alignment. arXiv preprint arXiv:2305. /2206, 2023.
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, and Zheng Zhang. Quzo: Quantized zeroth-order fine-tuning for large language models. arXiv preprint 2Xiv:2502.12346 2025.
14

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations in the last section.
15

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions and proof are included in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have provided hyperparameters and training details in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will release the code when the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/ Sublic/guides/Cone http://oeeeuiiisiooooooooooooeoo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCo//Soodemiiiceeee Soooeeuumisssoooooo
/ //nips... /tippscc/pputt wttp:cccpppppppcccccceo/ococpcca
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We specify all the training and test details. For example, the choice of hyperparameters is proved in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: We provide the error bars in the main figures and tables. For example, Table 10 provides the error bars of our main results.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have introduced the computer resources in the experimental setting.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https://harrips..ccppbllle ctChideieelllleececcceecc ines?
Answer:
[Yes]
Justification:
We make sure to observe the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper can take positive societal impacts on saving computing cost.
Moreover, it will not take any direct negative societal impacts in the present form.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
functioning correctly, harms that could arise when the
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
19

---

Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Our code is based on the open-sourced repositories, and cite these works in our paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have provided the training details to assist readers in running the experi-
ment.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
The proposed method developed by ourselves.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy (https / /ttp///een..p//ee/ (roonerrenee//2222 /LLM) for what should or should be described.
21

---

A The Prompts in LLaMA and OPT
Dataset
Type
Prompt
SST-2
cls.
(premise)
Does this mean that "(hypothesiss" is true? Yes or No?
Yes/No
RTE
cls.
Suppose "[premise]" Can we infer that "[hypothesis)"? Yes, No, Maybe?
Yes/No/Maybe
BoolQ
cls.
passage [question] Yes/No
WTC
cls.
Does the word "(word)" have the same meaning in these two sentences? Yes, No?
[sent]]
[sent2]
Yes/No
MultiRC
cls.
(paragraph)
Question: (question)
found this answer "(answer)". Is that correct? Yes No?
Yes/No
COРA
mch.
[premise) so/because [candidate]
Table 6: The prompts of the datasets we used in our LLaMA experiments.
B
Experimental Setting
B.1
Hyperparameters
We will introduce the hyperparameters searching grids in Table 7 and 8, which help people reproduce our results.
Experiment
Hyperparameters
Values
McZO
Batch size
16
Learning rate
{5e-7, 10--7, 2e-6}
1e-3
MeZO-Ram Ran0-Random
Batch size
16
Learning rate
{1e-6, 2e-6, 6, 3e-6, 4e - 6, 5e-6)
le-3
S-Mc70
Batch size
16
Learning rate
(1e-6, 6, 3e-3.-..0 6, 4e 6, 5e-6)
le-3
FT with Adam
Batch size
Learning Rates
fle-5, 5 5,
Table 7: The hyperparameter searching grids for LLaMA-70 7b experiments.
22

---

Experiment
Hyperparameters
Values
MeZO
Batch size
16
Learning rate
{1e-8, 20-8, 3e- 8, 5e
8, le-7, 7e-7, 1,-7,,e-6,22 2e-66
1e-3
McZO-Random
Batch size
Learning
(le-6, 2e 6, 3e- 4e 6,-6666-66
le-3
S-MeZO
Batch size
16
Learning rate
{le-6, 2e- 6, 3 4e
4e-6. 6, 5e-6)
le-3
FT with Adam
Batch size
8
Learning Rates
[1e-5, 5e-5, 8e-5
Table 8: The hyperparameter searching grids for Mistral-7b experiments.
B.2 The Setting of Threshold
We determine thresholds using principled sparsity-based approach. Specifically, we use a percentile- based method where the threshold is set based on a target sparsity level. For example, with 80% sparsity, we sort the weight values of each layer and the threshold at the 80th percentile. Impor- tantly, this threshold is determined once before training begins and remains fixed throughout the optimization process. We then introduce the sparsity of each task in SuperGLUE when we fine-tune LLaMA-7b. The setting is shown in the Table 9.
Method
SST-2
RTE
BoolQ
WTC
MultiRC
LLaMA Sparse McZO
0.70
0.75
0.80
0.80
0.80
Mistral Sparse MeZO
0.70
0.60
0.60
0.70
0.60
Table 9: Sparsity in SuperGI.UE when we fine-tune LLaMA-7b and Mistral.
B.3
Sparse Rate
In this section, we provide the sensitivity analysis of sparsity. Table 10 summarizes the experimental results with different sparsity values. We observe that significant performance gains are achieved when using sparsity values from 0.5 to 0.8. Moreover, for most tasks, sparsity value of 0.8 typically yields the best performance. For example, on RTE, S-MeZO improves accuracy from 71.7% to 82.3% when 0.8. Similarly, it achieves performance gain of 6.6% on BoolQ (from 75.9% to 82.5%).
Sparsity
0.5
0.8
RTE
77.1 5.4
79.0
79.5
82.3 10.6
BoolQ
79.1 t 3.2
80.3 t 4.4
81.1 5.2
82.5 6.6
WIC
63.8 f 2.4
63.6 f 2.2
63.4 2.0.
64.3 2.9
Table 10: The effects of Sparsity for Fine-tuning LLaMA-7b with S-MeZO. The performance of vanilla MeZO 71.7 on RTE, 75.9 on BoolQ and 61.4 on WIC.
23

---

C The Performance of Fine-Tuning Mistral-7b on SuperGLUE
In this section, we provide the experimental results of MeZO and Sparse MeZO on Mistral-7b. From the results in Table 11, we observe that S-MeZO can achieve consistent improvement on various sub-tasks.
Model
Method
BoolQ
RTE
WIC
MultiRC
SST-2
COPA
Average
Mistral-7b
Zero-Shot
69.3
55.2
50.0
57.1
55.5
84.0
61.9
Mistral-7b
ICL
76.7
78.0
61.4
71.3
94.6
90.0
78.7
Mistral-7b
LoRA
84.8
87.4
68.2
83.9
95.6
91.0
85.2
Mistral-7b
FT
86.7
87.1
71.2
86.1
95.6
91.2
86.3
Mistral-7b
MeZO
81.6
82.7
93.8
86.7
81.5
Mistral-7b
McZO LoRA
83.5
80.1
60.7
82.6
93.8
86.9
81.3
Mistral-7b
R-MeZO
84.0
78.7
63.2
83.1
92.4
84.1
80.9
Mistral-7b
S-MeZO
85.3
84.5
64.3
84.9
94.2
86.1
83.2
Table 11: Accuracy of Fine-Tuning Mistral-7b SuperGI.UE (1,000 examples). ICL: In-Context Learning, FT: full-parameter fine-tuning with Adam, R-MeZO: McZO with Random Mask.
To further illustrate the performance gain of our proposed Sparse MeZO, we provide error bars for the results on LLaMA2-7b.
Average
LLaMA-bb
Zero-Shot
65.1
49.5
50.6
55.8
79.7
59.7
60.1
LLaMA-7b
FT
84.5
0.0
0.9
68.4 : 1.3
1: 1.4
95.7 1: 0.3
85.0 1: 0.8
82.9 1: 0.8
LLMMA-7b
MiZO
75.9 1.1
71.7 1.5
61.4 = 1.8
69.8 : 0.7
94.6 0.3
86.3 0.9
76.6 + 1.1
LLaMA-Tb
R-MeZO
76.9
76.2 1.7
0.4
76.9
LLaMA-7b
8-MZ0
80.9 1.6
80.7 1.4
64.9 -: 1.5
73.3 =: 1.2
0.3
86.7 = 0.7
80.3 1.2
Table 12: Accuracy of Fine-Tuning LLaMA-7b on SuperGLUE (1,000 examples). ICL: In-Context
Learning, FT: full-parameter fine-tuning with Adam, R-MeZO: MeZO with Random Mask.
D Comparison between MeZO and SGD
In this section, we provide the Probability of Loss Increase with MeZO on Different Batch in Figure 4(a) and Probability of Loss Increase with SGD on Different Batch in Figure 4(b). We calculate the probability of loss increase for each epoch.
24

---

0.7
0.5
Probability of Loss Increase
0.6
another batch
Probabiltty of Lass Increase
0.4
another batch
0.5
0.3
0.4
0.2
1 mmm
0.1
0.0
Step (*100)
Epoch
RTE-MeZO
(b) RTE-SGD
Figure 4: (a) Probability of Loss Increase with McZO on Different Batch. (b) Probability of Loss
Increase with SGD on Different Batch. We calculate the probability of loss increment for each epoch.
E The experimental Results on OPT
We also provide the experimental results on OPT. As shown in the Table 13, Sparse MeZO can consistently improve the performance of vanilla MeZO on the three tasks of SuperGLUE.
Model
Method
BoolQ
RTE
WIC
OPT-13b
Zero Shot
59.0
55.0
OPT-13b
ICL
66.9
62.1
50.5
OPT-13b
MeZO
72.1
75.5
62.2
OPT-13b
R-MeZO
72.3
75.2
61.7
OPT-13b
S-MeZO
73.8
77.6
63.7
Table 13: Accuracy of Fine-Tuning OPT on SuperGLUE (1,000 examples). ICL: In-Context Learning, R-MeZO: MeZO with Random Mask.
Pesudo-Code
this section, we provide the pseudo-code which is mentioned Algorithm 1: GetMask (Algorithm 3) to compare each parameter against its threshold h; and create the mask m. PorturbParameters (Algorithm 2) to generate Gaussian noise sample z ~ (O,, Ia) and apply the mask m to produce sparse perturbation & m z.
Algorithm 2 PerturbParameters
Input: represents pre-trained LLM weight, perturbation scale random seed s, mask m..
Reset random seed
for 0: E 0 do
2, /O 1)
0, - 0, m. EZ4
end for
25

---

Algorithm 3 GetMask
Input: 0 represents pre-trained LLM weight, threshold h (h, represents threshold of each layer).
Output: Mask m for i <- Layer to Layer N do
for f,, E 0 do
if Oz. h; then
O. =
else
0g = 0 end if
end for end for
G Convergence Analysis of Sparse-MeZO
In this section, we will explain why Sparse-Me20 can accelerate the convergence, which is based on the theory from (Ohta et al., 2020). We can define a sub-network in pre-trained large language models, which is determined by the sparse mask m. The main idea of our proof is that if follow the updated role in Sparse-Me7O, the gradient norm on the sub-network can be smaller than o² after o(L)) steps, where d the number of parameters in the sub-network. Therefore, ZO can use fewer steps to converge when we only focus on sub-network. Some related work has illustrated that only tuning the sub-network can achieve comparable performance, which will be empirically verified in our experiments.
Firstly, assume the loss function L(0;z) is Lipschitz Continuous: Assumption (Lipschitz Continuous).
L(I) [))0=)) - VL((0, x)]l < 2 1 0'1
where V (((:z) denotes the true first-order gradient of on and L(I) represents the Lipschitz
constant of ((.). Given f. (0) = Eg [( <2)] and the above Assumption 1, we can obtain the
relationship between sparse gradient (0) and the expectation of estimated sparse ZO gradient (0):
Lemma 1. 70 gradient g= (f) is unbiased estimation of voL=
(0): = m O VoEg[C(e + c2)
@ VoC=(0)
C(O C(0 (2) L(e (t) 2)
(4)
E2
26
(0))
where ge (0) -(0+c2)- We can find that ge (0) is unbiased estimation of (o). the distance
2c
Then, based on the equation VoL, (0) Esig. (0) in Lemma 1, [[voL: VoLm (0)l to analyze the the relationship between the true sparse gradient VeL, m O (o) and sparse gradient & voL:
Lemma 2. Let C be Lipschitz Continuous, we have:
(oLm((0))' 2 2 2I2 2
(d 4)3.
where VoL. = m @ VoL(O), d d m; the number selected parameters in mask m. L(!) represents the Lipschitz constant. Finally, obtain the convergence rate of Sparse-MeZO.
26

---

Theorem 1. Assuming sequence of generated parameters 12e in Sparse-MeZO. We can have:
B B (0r)
o²
(6)
for any T 0
where I(()= v I, for all [(O²). This theorem illustrates that the presence of pronounced sparsity patterns, along with the smoothness of the objective function, can significantly enhance the rate of convergence, potentially achieving linear acceleration.
H
The Proof of Lemma
Let (0) be the expectation of C(0 + em @ O.)):
C=(0) B. + em O 2)]
(7)
-E + cB)]
We can obtain the Lemma:
VoL=(0) = m & VoL=(0)
O E, VoL(( em 2 2)]
E,1 [(O 2 cm ② 2) L(B cm ① 2)
3 2]
(8)
2e
C(0
C(0 + +Z)
C(O Eε
E2l
26
Proof:
(oL: = VoLEs[C(0 + +.)]
Ee / pdf; (2f;(() Ez)d2
m 3 Vo / (dfg(2) + ez)dz
m 3
Vopdf; + e=)d=
1 m
/ Voe +|*+1 [(o + E2)ds
O
k
m ④
Voe 111=00 1² L(y) c
1
dy
k
K m
O
e - 11110-00 011 L(y) et
dy
k m O
-
2
+|==1
L(0 + Ez)dz
pdf (z)((( <z) xz) 2dz
m ?
C(O + +e8)
Eg im
E3 C(0 + EE)
where can define y 0 + ez, + em 2 k / (2 and d' the number of in
Therefore, obtain the gradient (0) is equal to Eg C(o+ra) 2).
27

---

In addition, we will prove E3 [(0++3) 2) is also equal to E ((0++))) - 2]:
C(0 Eτ) (()) -((0)
Eg
[(0 + E2) -[(0)
26 =1|=11 dz
k
(10)
k: J C(O + c2)
1
=11==11 - d2 L( [
=1==11? dz 2e -
20
C(O + E.)
Eg
After that, can get the relationship between Eg [(0) ((10)-200-00 2(0-)))) 2] and Eg ((0+e))) {]:
Eg (
L(O
L(O + c( (-2)) L(0) (-2))
{]
1331
L(O + eB [())) 2]
=E=(
(11)
=E C(O =22) s).
Based on the Equation 10 and Equation 11, we can obtain:
L(O
c3)
C(0
cZ)
Eg
2)
2e
1
L(B + (2) 2 L(()
2 L(0), 2 L(O ci)
[E
2
1
C(0 + eε) - -C(0)
(()) - L( - -
(Ess
E3
2
(12)
1
C(O + c2)
C(0 + E {
(L&g
2) +Eg
2
C(O + Eτ)
E.
(0)
Finally, we can obtain the relationship between Eg ((0+e))) (aftett)-((--)))) c(0--8)g proof.
2) and (o))(0) and finish the
2c
I The Proof of Lemma 2
Vo = 2I2(1
2 4)². 4)
(13)
Proof:
We can first define the distance between (o |Esge (0)] and sparse FO gradient V C. (0) as:
28

---

(0) - [ofm(())
= -1 1
b /. [(0 6z) - -[(0 -22)
(VoLm(0), =)) - dell
2c
7
1
/
C(0 Ez) - -(()
(m o V,L(0), =))=E= +===11 dāll
1
[C(0 + cz) -[()) (V) 201e =|1=11 de
(14)
ke
L(l)
/ 1m1l 3 21le- =bl=ll da
2k
cL(I)
2 Bs [1=11=111]
cL(L) (d + 3)
2
where is the number of selected parameters with mask m. The last inequality holds because the Equation (25) in (Ohta al., 2020). In addition, ||a + bll2 s 2||a||2 2/bb|² we can define a = a - and obtain that ||a|| < 2| - bll + 2|bb|| Let a = VoLm (0) and b = (oL=(0), we can obtain:
(oom((0)
[ + (
(15)
2
2 ( (d 4)3 + (0))
2
J
The Proof of Theorem
Proof:
C=(0) L(0) Es[C(0 e2) [(0)]
E2[C(0 eε) ((0) - E(VC((), 2)]
To [C(0 + Ez) C(() <(V 2)]e +1l=11 dz
1
1o J/2 / 2
1
||1|8e- =11=11 dz
(16)
L [1
2 L(()) d
The first inequality holds because Lipschitz Continuous: |C(0') C()) (VL()), 0' - 5 2
L()) |10' - 0|1 where 6' ez. The second inequality holds because | d, where d is the number of 1 in mask m.
- (())))){(()) -
C(O
< (() - C(O) +[( - C(0 2
CL 2+ eL 2
(17)
2
2
L2)
29

---

The first inequality is due to ||a + bl v 2 2|| + 2/ where - L2(0) - L(0), b - Cz
c2) [(0 + c.). The second inequality is due the Equation 16.
[Ca(0 00000000000000000000000000000100
< + -
(I
||11| 22(
(18)
2
(( ||||| + 122
The first inequality is due to ||a + bl12 < 2||a|| + 2| The second inequality holds because
Lipschitz Continuous: |C((") - (()) (VL(0), 0- - 0) - 400 11 - 0|1?, where 6' + cl.
[C(0 <8) [(0)]
s (0) L())) (L2 + - L(O + <))))² - 20 -c)())
(19)
s 2c + (1)11)) + (2
The first inequality is due to ||a + b112 < 2 a| + b| 17 and Equation 18.
The last inequality holds because Equation
B.. (0)) B2(l [(0 + < L(O Eε)
2113]
2c
= =Eell [(O + E2) (()),
2 ((0) L(O E2)
2|121
2c
2c
C(O + c2) - [(A)
[(0) - L(A - e2)
2 + 211
2|12]
2E
(20)
1
EEs 26
[C(0 + <2) [(0) 1121 + [C(0) - C(B - c ||211
20
< = +20000000""4444+)111:::;0:00 S 3x2 4)3 (0)
The first inequality holds because || + b112 < | + bl1 where ((0+c3)- 2, b 2e
((0)-L(8-e)) 2. The second inequality is due to the Equation 19. The third inequality holds because
2x
Es || d, Eg [| < (d + p) for p 2 2. The last inequality holds because 2d + (d + 6)3 <
3(d + 4)3.
Based on the assumption about Lipschitz Continuous, we can obtain: (c(0++1) L(O.) (VL(0.), 6 0.)1 < L(0)
10+11 -1
Then, we can obtain:
(2(0x=)) - Co - è (C)) - -,) < (00+1)) - Co(O,) - (VC1( - -0,)
L())
(21)
<
2 10+1 - 0211
Based on the equation, we can follow the update rule: Opn1 = 0, "Tiga (0:) and we can find:
L(l)
(2(04))) < Cs((t) + ( (V) 01)
2 |10, 0.41
(22)
= C2(,)) - - ( + L())
(0))l2
30

---

where 7/ represents the learning rate at step t. Then, we can take the expectation of Equation 22 for and imput z;
E (0c+1))
(n))
Bg,= [C:():)] 1r (Z)
2
(Tt))
B. [C:( m. (0))
+ 4)9)
2
(23)
The first inequality is due the Equation 8 and Equation 22. The second inequality holds because Equation 20 provides the result about g, (0,)
Then, we can select learning rate T/t 4(d.+4)E(T) and obtain:
322 L(I) +4) (24)
E (4:::(() Eg, [C=(0)] 8(d, + 4))
B (0.)))
32
Then, taking the sum of Equation 24 over the index from T ++1 to have that
(3 Cg
32²
B [0 < 8(å 4)
+
L(d + +4)
(25)
T ++1
32
where L(I) S L for all L(O,). Thus, based on Lemma 2, we can have:
Eg, mm < LL (d 4) + g.
(1 ((T))(
(26)
< 16(d + 4) (bb) -
2
(á+))(d b/2
T+1
2
The second inequality is due to the Equation 25. To obtain o-accurate solution: E2, x ||| V | (0r) < x², we can define 12(
d3L
16(à + 4)L (00) L
L2 - L2) +
+0(212)))
16(d 4)L
T+1
T+1
(27)
dt d2
T 0
Finally, we can finish the proof of the theorem. This theorem illustrates that the presence of pronounced sparsity patterns, along with the smoothness of the objective function, can significantly enhance the rate of convergence, potentially achieving linear acceleration.
31


Reviews:
[{"review_id": "EB3shDbpA9", "reviewer": "Reviewer_i838", "summary": "The paper proposes to improve the convergence of zero-order optimizers by only applying the perturbation to smaller weights. The paper begins by presenting empirical observations on how the magnitude of weights in MeZO updates influences loss convergence. Based on this, the authors propose a practical solution: masking the largest weights during perturbation application. They demonstrate that this approach significantly enhances the convergence process and downstream performance in MeZO.", "questions": "* “Impressive results” line 1 seems like a superlative statement\n* Line 165, how are the percentages chosen to define small and large values? Why set the threshold to 20%?\n* Lines 222-225, I am not sure how this paragraph helps with the argumentation. According to the author, this method is not aligned with the goal of the paper and is not discussed anywhere else in the paper. I would suggest simply removing this paragraph for clarity.\n* It would be helpful to report the standard variations on top of the mean across the three runs for the results in at least Table 1. This could help capture the stability of the approach.", "limitations": "The authors have discussed the limitations in the last section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n* **s1:** The paper is well-motivated and written, clearly balanced between motivation, methods, and empirical evaluation. The method is simple and does not require a complex procedure on top of existing zero-order optimization processes.\n* **s2:** The experimental results are compelling, showing an average improvement of 3.7 points on the SuperGLUE tasks over vanilla MeZO without impact on the memory consumption or convergence speed.\n\n**Weaknesses**\n\n* **w1:** I am not fully convinced by the analysis presented in lines 147-159. As I understand it, the authors compute gradients on a set of 16 data points and then observe whether applying a gradient step in that direction, but starting from 16 entirely independent points, reduces the loss. The observation that there's roughly a 50% chance it won't be the case doesn't completely surprise me. It seems possible to me that a gradient computed on one set of 16 points aligns with a gradient computed on another, completely independent set of 16 points. I would like to know if similar results would hold for supervised fine-tuning (SFT). In particular, to include a comparison in Figure 2(b) with SFT.\n* **w2:** The results regarding the convergence rate in Section 4.4 and Figure 3 only compare convergence curves for MeZO and Sparse-MeZO, but it would be relevant to add the comparison with SFT here.", "was_revised": true, "final_justification": "Overall, my assessment of the paper remains unchanged. I find the proposed approach relatively straightforward and likely easy to apply in practice. The paper is well-written and generally well-motivated, though I believe there is room for improvement in the experimental justification of the approach (lines 147–159).", "rebuttal": ""}, {"review_id": "z5YXpSJ7Tq", "reviewer": "Reviewer_EoWB", "summary": "This paper introduces Sparse-MeZO, a memory-efficient zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Instead of updating all parameters, Sparse-MeZO selectively optimizes a subset by applying a sparse mask to target low-magnitude, noise-tolerant weights. Built upon the MeZO framework, this approach reduces gradient estimation noise, accelerates convergence, and lowers memory consumption. Experimental results show that Sparse-MeZO not only outperforms MeZO in convergence speed and final performance but also enables the fine-tuning of large models like LLaMA-30B using limited hardware, such as a single A100 GPU.", "questions": "* For the setting of the threshold, set its value related to the sparsity, e.g., 80% sparsity -> 80th percentile. Are these only empirical findings? It's better to have some ablation studies on the selection of the threshold.\n* Some parts of the appendix are not referred to in the main paper and are without a textual description.\n* Is the threshold computed every iteration? As the learning rate for zo is small, and the update per iteration is also small. Therefore, can you compute it intermittently?", "limitations": "Yes.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n* Incorporating sparsity and MeZO is an interesting direction for performance improvement.\n* The paper is structured logically, making it easy to follow the motivation and methodology.\n\n**Weaknesses:**\n* The paper lacks discussion and comparison with some important ZO works in sparsity, like [1]. Moreover, seems the method works due to gradient estimation with less noise, so it’s helpful to compare with some variance-reduction works of ZO, like [2] and [3].  \n* There are no specific numbers regarding the computational overhead introduced by threshold calculation, and dynamic masking and perturbation layer by layer in the forward pass.\n* The overall contribution is incremental as the work adds common magnitude-based sparse steps on top of the MeZO optimization algorithm.\n* It would be better to have some evaluation on more challenging task, like MMLU and MT-Bench, for example, using the settings in [4].\n\n[1] Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity\n\n[2] Zeroth-order stochastic variance reduction for nonconvex optimization\n\n[3] Zeroth-order optimization with trajectory-informed derivative estimation\n\n[4] Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning", "was_revised": true, "final_justification": "After reading the author's response and other reviewers' comments, the author addressed most of my concerns, but I still think the novelty of this paper is not that impressive. I raised my score from 2 to 3.", "rebuttal": ""}, {"review_id": "aKZOcFFxsO", "reviewer": "Reviewer_Waaa", "summary": "The paper present sparse-Mezo, an algorithm based on Mezo with the observation that optimizing small weights is more helpful than optimizing large weights for zeroth order method. Then the algorithm is a simple modification of Mezo: in each forward pass, the algorithm identifies small weights in each layer and only update these weights. The paper show experiments that support the observation and improve Mezo on fine-tuning tasks.", "questions": "- Please add the experiment results I mentioned in the appendix\n- For the completeness and self-containment of the paper, please also for each experiment provide a description of the set up (for example, section 4.3 is unclear)\n- Regarding the new observation: When training on top 20% of the largest weight vs the bottom 20%, how are the step sizes determined for each case and what are the impacts on the stability of the training and the performance of the model?\n- Is the disproportionate impact of the weights also observed when we train the model by first order method instead of zeroth order method?", "limitations": "Adequately addressed", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n- The paper provides a new observation for zeroth order methods that small weights have bigger impacts on training than large weights. This is very interesting and somewhat counter-intuitive. I think this should be studied more in the future.\n- The experimental results are quite strong against vanilla mezo and support the hypothesis.\n\n**Weaknesses**\n- The experiments could be more comprehensive. Some of the datasets are missing from some tables (for example, table 2 and others in the appendix). Please also add other baselines such as ZO-SGD to table 3 and where it’s relevant.\n- The algorithm introduces a sparsity parameter and it’s unclear how to set this parameter other than tuning\n- As acknowledged by the authors, there is still a significant gap between this method and gradient methods\n- The theorem in the appendix only applies for a fixed masking, not the masking by threshold used in the algorithm.", "was_revised": true, "final_justification": "I maintain my score after the discussion.", "rebuttal": ""}, {"review_id": "GA5yMCETn1", "reviewer": "Reviewer_sJB2", "summary": "This paper addresses the memory inefficiency of fine-tuning large language models. It identifies that while Zeroth-Order (ZO) optimizers like MeZO save memory by avoiding back-propagation, they introduce gradient estimation \"noise\" that harms performance, especially when applied to large-magnitude weights.\n\nTo solve this, the paper proposes Sparse MeZO (S-MeZO), a method that selectively applies updates only to the model's smaller-magnitude weights. This approach is more resilient to noise, which allows for larger learning rates, leading to faster convergence and better accuracy.\n\nA key contribution is a memory-efficient implementation that calculates the necessary sparse mask on-the-fly during the forward pass, keeping memory usage at the same level as inference. This innovation enabled fine-tuning a 30-billion parameter model on a single A100 GPU.\n\nExperiments conducted on LLaMA, OPT, and Mistral models show that S-MeZO consistently outperforms the original MeZO. For example, on the RTE task, it achieved a 9% absolute accuracy improvement and a 3.5x speedup.", "questions": "Have the authors explored the upper limits of the sparsity ratio for this method? For instance, what is the highest sparsity tested, and does performance continue to improve or begin to degrade at very high levels of sparsity?", "limitations": "The authors have acknowledged the primary performance-related limitation of their work, which is the remaining performance gap between S-MeZO and first-order fine-tuning methods. This transparency is commendable. However, the discussion of limitations could be expanded to make the paper even stronger: Broader Societal Impact: While the work itself does not have a direct negative impact, it contributes to a line of research that makes powerful LLM fine-tuning more accessible and efficient. The authors could briefly acknowledge the dual-use nature of such technology; lowering the barrier to entry can empower more users for positive applications, but it can also potentially make it easier for malicious actors to create fine-tuned models for generating disinformation or other harmful content. A short discussion of this broader context is standard for LLM research and would demonstrate a more comprehensive consideration of the work's potential impact.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Significant Practical Impact and Accessibility: The paper's most significant contribution is making the fine-tuning of extremely large models more accessible. By developing a memory-efficient implementation that calculates sparse masks on-the-fly, S-MeZO successfully keeps memory usage at the level of inference. This innovation enabled the fine-tuning of a LLaMA-30b model on a single A100 GPU, a task that is typically infeasible with standard methods. This dramatically lowers the hardware barrier for LLM research and development.\n\n2. Novel and Insightful Core Idea: The paper is founded on a highly original empirical observation: the noise inherent in Zeroth-Order (ZO) gradient estimates is more detrimental to large-magnitude weights than to small-magnitude ones. This insight is novel and provides a clear, well-motivated basis for the proposed S-MeZO algorithm, which selectively optimizes these more noise-resilient small weights. This approach offers a new perspective on mitigating noise in ZO optimization.\n\n3. A notable strength of this paper, setting it apart from some other works focused on ZO optimization, is the inclusion of baselines that use back-propagation to calculate gradients. \n\nWeaknesses:\n1. Since the parameter selection is based on the smallest weights, the benchmark could be strengthened by including a comparison with the opposite strategy: selecting and updating only the largest-magnitude weights. This would serve as a valuable ablation study to more directly validate the paper's core hypothesis.\n\n2. A potential area for strengthening the experimental comparison would be to include another recent and relevant method in sparse ZO optimization. For instance, comparing S-MeZO against SensZOQ would provide a more comprehensive picture of its performance within this specific sub-field.", "was_revised": true, "final_justification": "The authors have provided a thorough set of baseline comparisons.", "rebuttal": ""}]

Author rebuttal:

Meta-review: The paper proposes Sparse MeZO, a memory-efficient zeroth-order (ZO) optimization approach for LLM fine-tuning. The core claim is that ZO gradient noise disproportionately harms large-magnitude weights, so selectively applying updates only to small-magnitude weights reduces noise sensitivity, accelerates convergence, and improves performance. The method also includes a memory-optimized masking implementation that enables fine-tuning models as large as LLaMA-30B on a single A100 GPU. Experimental results across SuperGLUE, commonsense reasoning, and math tasks show consistent gains over MeZO, with improvements of up to 9% accuracy and 3.5× faster convergence.

The strengths of the paper include a clear empirical observation motivating the method, a simple yet effective algorithm with negligible overhead, strong empirical results across models and tasks, and an impactful system contribution that lowers hardware barriers for LLM fine-tuning. Reviewers also noted the novelty of focusing on weight magnitudes and the practical significance of enabling large-scale fine-tuning under constrained resources.

Weaknesses raised include limited theoretical guarantees (the provided theorem applies only to fixed masks), incremental contributions relative to MeZO, incomplete baselines in the initial submission, and questions about sparsity threshold choice. Some reviewers also noted the gap between Sparse MeZO and first-order methods remains and suggested broader evaluation on challenging benchmarks and comparison with other sparse ZO methods such as SensZOQ.

The decision to accept is based on the strong empirical evidence, practical utility, and the clear motivation behind the method. While the contribution could be considered incremental, the simplicity, demonstrated effectiveness, and memory savings make it valuable for the community. The paper stands out because it addresses a critical bottleneck in ZO optimization with a principled and practical solution that enables new capabilities, such as fine-tuning 30B-parameter models on commodity hardware.

During the rebuttal, reviewers requested additional comparisons (e.g., with SensZOQ, ZO-SVRG, and ZO-SGD), clarity on sparsity selection, more ablations, reporting of variance, and evaluations on harder tasks. The authors addressed these comprehensively by running new experiments, providing ablations that confirmed the advantage of small-weight updates, clarifying the fixed-threshold scheme, reporting standard deviations, and expanding evaluations (including MMLU and comparisons with variance-reduction ZO methods). These responses resolved most concerns and improved reviewer confidence, with several reviewers raising their scores. Overall, the combination of methodological insight, practical system implementation, and strong rebuttal makes this a solid accept.
Decision: Accept (poster)