Fine-tuning Quantized Neural Networks with Zeroth-order Optimization
Authors: Sifeng SHANG
JIAYI ZHOU
Chenyu Lin
Minxian Li
Kaiyang Zhou
Venue: ICLR 2026 Poster (2026)
Keywords: Memory-efficient Training
Zeroth-order Optimization
Quantization
Primary area: foundation or frontier models, including LLMs
TL;DR: Fine-tune a quantized large language model with zeroth-order optimization to save memory up to 18.4x

Abstract: As the size of large language models grows exponentially, GPU memory has become a bottleneck for adapting these models to downstream tasks. In this paper, we aim to push the limits of memory-efficient training by minimizing memory usage on model weights, gradients, and optimizer states, within a unified framework. Our idea is to eliminate both gradients and optimizer states using zeroth-order optimization, which approximates gradients by perturbing weights during forward passes to identify gradient directions. To minimize memory usage on weights, we employ model quantization, e.g., converting from bfloat16 to int4. However, directly applying zeroth-order optimization to quantized weights is infeasible due to the precision gap between discrete weights and continuous gradients, which would otherwise require de-quantization and re-quantization. To overcome this challenge, we propose Quantized Zeroth-order Optimization (QZO), a simple yet effective approach that perturbs the continuous quantization scale for gradient estimation and uses a directional derivative clipping method to stabilize training. QZO is orthogonal to both scalar-based and codebook-based post-training quantization methods. Compared to full-parameter fine-tuning in 16 bits, QZO can reduce the total memory cost by more than 18$\times$ for 4-bit LLMs, and enables fine-tuning Llama-2-13B within a single 24GB GPU.

Published as a conference paper at ICLR 2026
TININGUNING QUANTIZED NEURAL NETWORKS WITH ZEROTH-ORDER OPTIMIZATION
Sifeng Shang' Jiayi Zhou' Chenyu Lin'
Minxian Li
Kaiyang Zhou
Hong Kong Baptist University
2 Nanjing University of Science and Technology
https//gghhkkoommmaaafoodddtttso
ABSTRACT
As the size of large language models grows exponentially, GPU memory has become bottleneck for adapting these models to downstream tasks, In this paper, we aim to push the limits of memory-efficient training by minimizing memory usage model weights, gradients, and optimizer states, within a unified framework. Our idea is to eliminate both gradients and optimizer states using zeroth- optimization, which approximates gradients by perturbing weights during forward passes to identify gradient directions. To minimize memory usage on weights, we employ model quantization, e.g., converting from 6 to int4. However, directly applying zeroth-order optimization to quantized weights is infcasible due to the precision gap between discrete weights and continuous gradients, which would otherwise require de-quantization and re-quantization. To overcome this challenge, we propose Quantized Zeroth-order Optimization (QZO), simple yet effective approach that perturbs the continuous quantization scale for gradient estimation and uses directional derivative clipping method to stabilize training. QZO is orthogonal to both scalar-based and codebook- -bsee post-training quantization methods. Compared full-parameter aie-ttnnng in 16 bits. QZO can reduce the total memory cost by more than 18x for 4-bit LLMs, and enables fine-tuning 1lamm-2- 13B within a single 24GB GPU.
INTRODUCTION
Pre-trained large language models (LLMs) (Zhang et al. 2022 Touvron et al. 2023a b Grattafioni
et al. 2024) have demonstrated great potential in numerous downstream applications, ranging from sentiment classification and text summarization, to more challenging open-ended question answering and creative writing. However, with the model size growing at exponential rate, adapting LLMs to downstream tasks presents significant challenges to computational resources. For instance, fine- tuning a Llama-7B model stored in 6 typically requires 56GB GPU memory: 14GB for model weights, 14GB for gradients, and another 28GB for optimizer states when adaptive gradient-baat based optimization methods are used (c.g., the first and second moments in AdamW (Loshchilov & Hutter 2017). which cost twice the size of gradients). Such an enormous memory cost makes it infeasible for researchers and practitioners with limited computational resources fine-tune LLMs.
In general, there are four key components that determine memory usage: (1) model weights, (2) gradients (typically the same size as weights), (3) optimizer states (often twice the size as gradients). and (4) activations cached for gradient computation. Since activations are mostly affected by the size of mini-batch, existing memory-efficient training methods mainly target the first three components (Zhao et al. 2024 Malladi et al.) 2023). In this work, aim to push the limits of memory-efficient training by minimizing memory usage on model weights, gradients, and optimizer states, within unified framework.
Our main idea is to eliminate gradients and optimizer states using zeroth- -rder optimization (Spall 1992). which gets rid of backpropagation by approximating gradients solely through forward passes (i.c., perturbing model weights to identify gradient directions). When it comes to model weights, the
Corresponding author

---

Published as a conference paper at ICLR 2026
Memory Profiling SST2
113.7
18.3x
87.6
80
VRAT (GB)
60
26.8
26
20.4
15.2
4.8
0
0PT-6,7 78
Llama 2:78
Liama 3.1-8B
Fine-tune w/ AdamW (16-bit)
Fine-tune w/ SGD (16-bit)
Mezo (16-bit)
Q20 (4-bit)
Figure 1: Memory profiling on SST-2 (Socher et al. 2013) with (per-device) batch size set to 1. Fine-tuning w/ AdamW is done with shaly-sharded data parallel.
optimal approach is to quantize the weights, e.g., converting from bíloat16 to int4 can significantly cut memory cost by x. However, directly applying zeroth-order optimization to quantized weights is non-trivial because (1) quantized weights cannot be perturbed in the continuous space, and (2) the gradients estimated by a zeroth-order optimizer are continuous and therefore cannot be used to update discrete quantized weights (which would otherwise require de de-quantization and re-quantization)
To overcome the aforementioned challenges, we propose a novel approach called Quantized Zeroth- order Optimization (QZO), which enables quantized neural networks to be fine-tuned with zeroth- order optimization, hence achieving maximum reduction in memory consumption comparedd to full-parameter fine-tuning in 16 bits, QZO significantly reduces the total memory cost by 18x for 4-bit I.MMs (see Figure T Specifically, Q7O approximates the gradients of quantized weights by perturbing the continuous quantization scale parameter(s) rather than the discrete weights, which are kept fixed throughout training. To further stabilize training, we propose gradient clipping method and provide a theoretical proof to justify that the clipping method essentially reduces the variance of the gradient estimate.
We evaluate QZO on different families LLMs including OPT (Zhang et al. 2022) and Llama (Tou vron et al. 2023b) Grattafiori et al.) 2024). as well as using a diverse set of quantization methods. The experiments are conducted on five popular NL.P benchmarks including both classification and generation tasks. Using 4-bit L.Mss, Q70 significantly outperforms both quantized and un-quantized zero-shot models while performing on par with McZO (Malladi et al. (202)), which applies zeroth- order optimization to un-quantized models. In the extreme quantization case where the model is quantized to 2-bit, QZO still beats the zero-shot baseline by a large margin, demonstrating the effectiveness of QZO in fine-tuning quantized models. We also provide both theoretical evidence and ablation experiments to demonstrate the effectiveness of directional derivative clipping in stabilizing the training, which functions through reducing the variance of the gradient estimates.
2 RELATED WORK
Memory-Efficient Training Fine-tuning LLMs often requires significant amount of GPU memory, making it challenging for model adaptation on resource-constrrd memory-effiee efficieef training methods mainly focus reducing GPU memory usage for the following components: (1) learnable model weights, (2) gradients, (3) optimizer states storing additional gradient information, and (4) activations cached for gradient computation. To memory cost
hardware. general, current
By quantization, we refer to post-training quantization throughout this work, unless specified otherwise.
2

---

Published as a conference paper at ICLR 2026
for optimizer states, GaLore (Zhao et al. 2024) projects the first and second moments of gradients in AdamW (Loshchilov & Hutter 2017) onto a low-rank subspace. McZO (Malladi et al. 2023) eliminates gradients and optimizer states by using zeroth-order optimizer (Spall) 1992]. which estimates gradients using only forward passes and therefore keeps the memory cost the same as inference. CoL. (Nguyen et al. 2025) uses small mini-batches whose gradients match those of large mini-batches, leading to huge memory reduction in activations. Our approach further pushes the limits of memory--fficient training by fine-tuning quantized LLMs with zeroth-order optimization, which significantly cuts memory usage across all components requiring GPU memory.
LLM Quantization Post-training quantization (PTQ) is a popular paradigm for compressing
LLMs. Most PTQ methods (Dettmers al., 2022 Frantar et al. 2023 Lin al. 2024 Xiao et al. 2023 Ashkboos et al. 2024) reduce the bit width for each model parameter by representing the
numerical range with low-precision integers while using full precision for quantization parameters. These methods can achieve up 4-bit quantization, resulting in up to 4x reduction in memory usage compared to the widely-used BF16 representation. Different from the popular scalar-based
quantization paradigm, recent research (Tseng et al. 2024 Egiazarian et al. [024 Liu et al. 2024 has explored using codebooks for storing full-precision numbers, which are indexed with integers to represent the original model weights. These codebook-based methods can achieve extreme quantization in 2 or 3 bits without observing significant performance drops. Typically, quantized LLMs are not suitable for fine-tuning because continuous gradients cannot be directly applied to updating discrete quantized weights (which would require de-quantization and re-quantization). Our approach seamlessly combines memory- efficient training with quantization to enable fine-tuning on quantized LLMs, achieving maximal reduction on GPU memory usage. More importantly, our approach is orthogonal to most PTQ methods, including both 4-bit and 2-bit quantization methods.
Zeroth-order Fine-tuning for Quantized Models Inspired by foundational approach, ZO- signSGD (Liu et al., 2019). several prior works (Feng et al. [024]] Zhou et al. [025]] Bar & Giryes) 2025) expand on this study to enable the fine-tuning of quantized models, using a shared paradigm that involves quantizing perturbation noises and directly applying sign-based SGD on discrete, quantized weights. Although sharing a similar spirit in minimizing the memory footprint, namely combining zeroth-order optimization with quantization, the proposed QZO approach is inherently more efficient and flexible, as it does not require quantization of perturbation noises or re-quantization of model weights at each optimization iteration. Furthermore, it can be applied to existing scalar-based or codebook-basse PTQ methods, such as GPTQ (Frantar et al. 2023) and AQLM (Egiazarian et al. 2024), in plug-and-play manner.
3 METHODOLOGY
3.1 BACKGROUND: ZEROTH-ORDER OPTIMIZATION
Zeroth-order optimization (ZO) methods are often used in cases where gradients and higher-order derivatives of the objective cannot be directly computed or are unreliable (Conn et al. 2009h. The
pioneering work, Simultaneous Perturbation Stochastic Approximation (SPSA) Spall 1992 is defined as follows,
Definition 3.1 (Simultaneous Perturbation Stochastic Approximation, SPSA Spall 1992 ). Given a model parameterized by 0 E Rd and a loss function C, SPSA estimates the gradients of 0 on a mini-batch B using the following formula:
VoL(0,B) L(e + cz; c); L(0 (2;
z = zz V).
(1)
2e
where z E Bdd is random vector sampled from N(0, Ia), and the perturbation scale.
Built top of SPSA, recent work (Malladi et al. 2023) proposed memory-efficient zeroth-order optimization (MeZO) for LLMs. In particular, MeZO uses random seeds as a trick to eliminate the storage cost of 2, and result. the memory footprint is kept the same level as inference. MeZO also replaces the regular SGD (Robbins & Monro [95I) with zeroth-orde stochastic gradient descent (ZO-SGD), which is defined below:
3

---

Published as a conference paper at ICLR 2026
Definition 3.2 Seroth-Orrde Stochastic Gradient Descent, ZO-SGD MMaladi et al. 2023 Given learning rate n, ZO-SGD updates the parameters 8g at t-th step using gradients estimated by SPSA as follows:
0+11=
= 0
where B, denotes the input mini-batch at step L.
3.2 QZO: QUANTIZED ZEROTH-ORDER OPTIMIZATION
QZO minimizes the memory usage not only on gradients and optimizer states but also on model weights this can save huge memory cost when using large models of more than 10B parameters, e.g., when using bfloat16, 10B model's weights consume 20GB memory, while using int4, the weights only take 5GB of memory. QZO consists of two core modules: Quantized Simultaneous Perturbation Stochastic Approximation (Q-SPSA), and directional derivative clipping. The former extends SPSA to quantized weights while the latter stabilizes training by reducing the variance of gradient estimation.
3.2. FROM SPSA TO Q-SPSA
SPSA (Eq. I cannot be directly applied to quantized weights because (1) quantized weights are discrete and therefore cannot be perturbed the continuous space, and (2) the continuous gradients cannot be used to update discrete weights, which would otherwise require de-quantization and re-quantization. To overcome these challenges, we propose Quantized Simultaneous Perturbation Stochastic Approximation (Q-SPSA), which only applies perturbation to the continuous quantization scale. We begin by introducing quantization and de-quantization, which are two essential steps in model quantization. Concretely, for each single element w in weight set W, these two steps can be formulated as
W
(3)
W
W
where A denotes element-wis wise quantization scale, and i. the quantized counterpart stored using lower bits. The weight set W determined by the choice of quantization group, while the implemen-
tation of A varies among different quantization methods. For example, when A ahsmax(W) (W) Eqs. and refer to the standard salar-based quantization in bibit.
Since the de de-quantization process Eq. Jalign with the normal forward propagation, we decompose the model parameters 0 in Fq. into A ④ O. and perturb the scaling component A while keeping the discrete weights O fixed. Therefore, Q-SPSA can be formulated
Definition 3.3 (Quantized Simultaneous Perturbation Stochastic Approximation, Q-SPSA), Given a quantized model with integer parameters O E R2 and quantization scales A, and a loss function C, Q-SPSA estimates the gradients of A over mini-batch B using the following formula:
C(( + EZ) о B) - C(A ez) 00
VAL A ④
0:
2x
(5)
zz V C O:
where E Rdd is random vector sampled from N'0 Ia), the perturbation scale, and 3 the Hadamard product.
Similar to MeZO, all quantization scales within linear layer are perturbed to save computation. In practice, one may choose to fine-tune the continuous quantization scale only, or combine Q-SPSA with SPSA to jointly update the unquantized counterparts. It is worth noting that Q-SPSA can be applied to both scalar-basee and codebook-base quantization methods: in the experiments we successfully fine-tune both 4-bit LLMs quantized by the scalar-based GPTQ (Frantar et al. 2023) and 2-bit LLMs quantized by the codebook- based AQLM (Egiazarian al. 2024) (in this case both the channel-wise scales and un-quantized weights updated).
show that our approach
4

---

Published as a conference paper at ICLR 2026
3.2.2 DDC: DIRECTIONAL DERIVATIVE CLIPPING
Gradient estimation via ZO is notorious for causing unstable training due large gradient vari- ance (Malladi el al. 2023). This was also observed when combining Q-SPSA with the vanilla ZO-SGD method in our preliminary experiments where training often collapsed. To mitigate this problem, we propose Directional Derivative Clipping (DDC) and apply this method before updating the model with ZO-SGD at each optimization step.
Specifically, the gradient estimate in Eq. 5 can be viewed product of the random vector and the estimated directional derivative of loss funetion along w.r.t. A (which is essentially a scalar). Let denote the estimated directional derivative, Eq. can be re-written as VaC(A 3 0; - d
Then, DDC applies clipping to d by:
C, i d. del-c,C
d'
(6)
-C,,
where C' is a non-negative constant. The gradient estimate then becomes V (' @ 0 B) d'
which is plugged into ZO-SGD. We provide theoretical evidence highlight that DDC can reduce the variance of the gradient estimate and thereby stabilize the training. We first propose the following theorem as a preliminary to our analysis. The proof of Theorem i available Appendix
Theorem Clipped gradient estimate Val' A 3 0;B) is an unbiased estimate of the full gradient
of loss w.r.t quantization sclaes V C( A 0).
Since 2 S d2 by definition of DDC in Eq. 6 the following inequality holds:
A 0; 1
|| <
| B
C(A + 0; B)
(7)
Therefore, the element-wiss variance of the clipped gradient estimate has the following derivation:
L' 0; V (A
0: 3) ('
0]
< BIII A. C(A
0; B)B)| B[ L'
B;B))
Var(Va, C(A
0; EVA, @
0)
C 0)
VarlVA, C(A
0: + VV C(A
0))² E C' A 0: B))
(8)
By Theorem Var VarlVa,, c 0,B) S Var V A, <( 3 0) holds almost surely.
Our experimental results in Section 4.3 also reveal that DDC effectively stabilizes the training through rectifying abnormal loss values, and the ablation study also demonstrates that QZO relatively robust to the magnitude of C.
3.2.3 ALGORITHM
We summarize QZO in Algorithm I Note that although the quantization scales are perturbed per parameter in the pseudo code, practice one may perturb the entire quantization scales of linear layer to save training time (Malladi et al. 2023).
Remarks QZO seamlessly combines ZO with quantization and therefore leads maximum reduction in memory usage: gradients and optimizer states are eliminated while model weights compressed. To further cut memory usage on activations, one can divide the batch size while increasing the total number of optimization steps, or release activations during forward passes since ZO does not need cache activations for gradient computation.
EXPERIMENTS
4.1 EXPERIMENTAL. SETUP
Models and Datasets We evaluate our approach using three 7B-level LLMs, namely OPT-6.7B (Zhang et al. 2022). Llama-2-7B (Touvron al. 2023b), and Llama-3.1-
5

---

Published as a conference paper at ICLR 2026
Algorithm I Quantized Zeroth-order Optimization
Require:
quantization scales A E Rdd quantized weights E Rdd, loss function Rd -> IR learning rate T),, optimization steps T, perturbation scales clipping threshold C.
for t 1...I' do Sample batch of inputs B and random seed 8
A <- PERTURB SCALES(A, 8)
4 + C ④ 0; B)
1st forward pass
A PERTURE SCALES(A, -2e,
L_ + C(A @ B;
>> 2nd forward pass
A < PERTURB SCALES e)
dt --)
d' < (d, CC
D Directional derivative clipping, Eq.
Reset random number generator with seed
for A; E
~ (0,
A - max - 1/1 * d' 2,
> Ensure non-negative scales
end for end for
procedure PERTURB_SS SUUUR__AAASSSA,,. c, 8)
Reset random number generator with seed for A; E A do
2 (0, - A
end for end procedure
8B (Grattafiori et al. 2024), and one large-sized model with 13B parameters, i.e., -lama-2-113B TTou vron et al. 2023b). For QZO, the 7B models are quantized to 4-bit while the 13B model to 2-bit to test QZO's effectiveness under extreme quantization. Following prior work (Malladi et al. 2023),
evaluate our approach on five popular NLP datasets covering both classification and generation tasks. Specifically, for classification,
use SST2 (Socher al.) 2013) and three subsets from SuperGLUE
collection (Wang et al. 2019). i.e., RTE (Dagan et al. 22055 Haim et al., 22066] Giampiccolo et al. 2007; [entivogli al. 2009), CB (De Marneffe al. 2019) and BoolQ (Clark al.) 2019). For generation, we use SQuAD (Rajpurkar et 2016), which is question answering dataset. Following the common practice, we randomly sample 1.000 examples for training. 500 examples for validation. and 1,000 examples for testing. We report accuracy for classification tasks, whereas the metric for generation tasks is score.
Baseline Methods wide range of baseline methods chosen for comparison to justify QZO's effectiveness. Specifically, QZO is compared with: (1) Zero-Shot, and Zero-Shot-Q. the original and quantized zero-shot models, respectively, which are viewed as the lower-bound; (2) Fine-tuning on 16-bit models, which is considered the upper-bound (3) MeZO (Malladi et al. 2023 which applies ZO to un-quantized models.
Implementation Details For 4-bit quantization, we apply GPTQ Frantar et al. 2023) to the 7B-level LLMs (i.e.. OPT-6.7B, Llama-2-7B, and Llama-3.1-8D The quantization group in GPTQ set to 128. For extreme quantization in 2-bit, we apply AQLM (Egiazarian et al. 2024 with codebook of 16 bits to Llama-2-1332 [ We use QZO to fine-tune the channel wise scales in AQLM. Following prior work (Eggiazarian et al. 2024 Tseng et al.) 2024), the un-quantized parts are jointly fine-tuned using the regular SPSA and ZO-SGD. To accelerate QZO fine-tuning in 2-bit, also
Due to limited budget computational resources, fine-tuning experiments are conducted with SGD optimizer unless otherwise specified.

---

Published as a conference paper at ICLR 2026
Table Experiments based on OPT-6.7B, Llama-2-7B, and Llama-3.1-8B. Zero-Shot and Zero-Shot- Q serve as the lower-bound, while Fine-tuning (with SGD) is the upper-bound QZO works well across different model architectures on all datasets, with memory footprint signifieantly lower than MeZO and Fine-tuning.
Model Precision
Memory Profiling
Classficiation
Generation SQuAD
SST-2
RTE
CB
BoolQ
Fine-tuning
16 bits
26.8GB
95.4
79.8
73.2
69.6
77.6
Zero-Shot
16 bits
61.2
55.2
51.8
59.5
36.5
OPT-6.7B
Zero-Shot-Q
4 bits
60.1
53.8
51.8
59.1
35.9
MeZO
16 bits
14.8GB
93.0
67.9
79.6
QZO
4 bits
4,8GB
87.6
61.7
67.9
66.4
78.5
Fine-tuning
16 bits
26.0GB
92.8
63.2
60.7
75.0
83.7
Zero-Shot
16 bits
58.1
61.7
32.1
66.0
55.6
Llama-2-7B
Zero-Shot-Q
bits
58.5
53.4
35.7
64.6
53.6
MeZO
16 bits
14.8GB
83.5
58.1
67.9
69.6
80.7
QZO
5.0GB
90.0
69.6
68.2
85.5
Fine-tuning
16 bits
31.9GB
93.7
62.5
84.9
Zero-Shot
16 bits
59.6
45.8
46.4
66.1
64.8
Llama-3-8B
Zero-Shot-Q
4 bits
58.7
50.2
37.5
65.0
59.2
Mc20
16 bits
20,5GB
92.5
70.0
91.1
83.4
86.9
QZO
4 bits
6.3GB
93.0
66.8
69.6
78.2
88.3
Table 2: Training statistics collected on SST-2. computation-efficien..
Overall, QZO is both memory-efficient and
Trainable Paramters
Total FLOPs
(SST-2)
Fine-tuning
6.65 x 10a
2.17 10'6
OPT-6.7B
MeZO
6.65 x 10
9.91 x 10'7
QZO
5.03 x 107
8.19 10
Fine-tuning
6.74 x 10
2.47 x 1016
Lllmm-2-7B
MeZO
6.74 x 10
1.13 10
QZO
5.06 x 107
2.26 x 1016
Fine-tuning
8.03 x 10
2.48 x 1016
Llama-3.1
MeZO
8.03 10
1.13 10
QZO
5.45 x 107
7.9 x 10
modify AQLM's Triton inference kernel to disentangle matrix reconstruction and matrix-vector multiplication For QZO, set the learning rate to 10 the batch size 16, training steps to 20k, the perturbation scale to 10 3 and the clipping threshold C to 100. For fine-tuning experiments with SGD, the learning rate is initialized as 8 x 10-4 with linearly scheduled decay, and the batch size set to 8. A single Nvidia RTX 4090 GPU (24GB) is used for all experiments (except Fine-tuning, which requires an A100 80GB GPU). For McZO, adopt the official code 6
4.2
MAIN RESULTS
QZO on 4-bit Quantization Table compares QZO with different baselines across three model architectures the five NLP datasets. The detailed training statistics shown in Table Following MeZO, memory profiling measures the peak memory usage during the first 100 optimization steps. The dataset used for memory profiling is SST2 and the (per-device) batch size is to I to test the minimum VRAM requirement. We summarize our main findings below.
"https//gituu..co//rroooooo

---

Published as a conference paper at ICLR 2026
Table 3: Experiments based on Llama-2-13B. QZO demonstrates strong potential under extreme quantization.
Model Precision
Memory Profiling
Classification
Generation
SST-2
RTE
CB
BoolQ
SQuAD
Zero-Shot-Q
2 bits
57.6
53.1
46.4
69.2
55.4
Llama-2-13B
QZO
2 bits
5.78GB
80.5
54.5
55.4
70.2
59.4
QZO w/ DDC
QZO w/o DDC
OZO w/ DDC
QZO w/o DDC
Directional Derivation Eslimetion
1.4
200
NaN observed at 22-th step
1.2
1.20
observed
22-th step
1.0
100
10
0.8
MA
I
0.6
0.4
-200
200
100
600
800
1000
200
400
600
800
1000
Steps
Steps
Figure 2: Directional derivatives (left) and loss values (right) collected during the early 1,000 training steps. Without DDC, the training extremely unstable, often leading to abnormal directional derivatives and eventually NaN values for the loss.
QZO demonstrates effectiveness consistently across all model architectures and NLP tasks. Specifi- cally, QZO achieves significant improvements over Zero-Shho-Q, meaning that QZO successfully fine-tunes these quantized LLMs. On most datasets, QZO performs on par with MeZO, despite using 3x less memory; sometimes QZO even beats MeZO with noticeable margins, e.g., 85.5 vs. 80.7 on SQuAD when using Llama-2-7B. It is worth highlighting that MeZO is based on 16-bit models while QZO is based on 4-bit models with much lower precision. Compared with the upper-bound, i.e., fine-tuning, the gap is still huge some of the tasks. This makes sense because ZO methods rely merely on forward passes for gradient estimation, which would be much less accurate than that of backpropagation.
QZO demonstrates both memory-efficiency and computation-effiieee efficiency to the extreme by eliminating gradients and optimizer states while reducing weights precision. Therefore, the memory usage is minimal compared to the baselines like MeZO and Fine-tuning. Table compares QZ. with MeZO and Fine-tuning on learnable parameter count and FLOPs. is worth noting that Q70 uses only about 1% of the trainable parameters and 1 1% of the FLOPs of MeZO. This because QZO only fine-tunes the continuous quantization scale while leaving most weights (which are quantized) fixed. We expect the difference to be further increased when more powerful quantization methods are used.
QZO pushes memory-
QZO on 2-bit Quantization Table shows that QZO beats the zero-shot model with significant margins. The results strongly justify QZO's effectiveness under extreme quantization. QZO has the potential to be applied to on-device learning scenarios for edge devices.
4.3 ABLATION STUDIES
In this section, we mainly evaluate the DDC component. Recall that DDC (Directional Derivative Clipping. Eq. 6)) clips abnormal directional derivatives estimated via QZO (i.e., d in Eq. 6p. We use QZO to train two Llama-2-7B models, with and without using DDC, and record the directional derivatives and loss values for the first 1,000 steps. Figure 2 shows that without DDC the directional
8

---

Published as a conference paper at ICLR 2026
90
85
Accuracy
75
70
65
55
0
25
50
75
100
125
150
Clipping Threshold
Figure 3: Impact of clipping threshold. small effectively avoids abnormal directional derivatives, but suffers from underfitting due to small optimization step size. A large C' fixes this issue, but may introduce the risk of producing abnormal values. Note the performance at C corresponds with zero-shot accuracy of the quantized model.
derivative often gets abnormal values that go beyond the range of [-C, CJ (C' is the clipping threshold in Eq. (6, leading to NaN value for the loss (which means the training collapses).
We also study how sensitive QZO is to the clipping threshold C. Intuitively, a small C' should effectively avoid abnormal directional derivatives, but may suffer from underfitting due to small optimization step size. A large C' fixes this issue, but also increases the risk of producing abnormal values. For quantitative analysis, we train Llama-2-7B models on SST-2 with different values of C' and record the final accuracies. The results are presented in Figure 3 The trend of the line plot suggests underfitting at C 50, and stable performances can observed when C' > 75. When C is set to a value bigger than 150. the training becomes unstable and sometimes collapse, which algins with the observation F w/ DDC can be seen as setting to an infinitely large value).
5 CONCLUSION, LIMITATIONS, AND FUTURE WORK
QZO enables fine-tuning quantized neural networks via ZO, which greatly reduces memory usage related to model weights, gradients, and optimizer states. We show that QZO works for a wide range of LLMs and is compatible with both scalar-based and codebook-based quantization methods. When using 4-bit LLMs, QZO achieves performance on par with MeZO, while using 3x less GPU memory. In the extreme quantization scenario, QZO successfully fine-tunes 2-bit LLama-2-133 across different NLP datasets. The results indicate that QZO has the potential to be applied to on-device learning for edge devices.
In addition to LLMs, we have also applied QZO to fine-tuning text-to-image generation models, namely Stable Diffusion 3.5 Large (Esser et al. (0024). The results and discussions are presented in Appendix F QZO fine-tunes Stable Diffuusion 3.5 Large using only 12.4GB of memory in a single Nvidia RTX 4090 GPU. The visualization results are also encouraging: the data distribution generated by QZO is visually closer to the ground truth than the zero-shot model.
However, QZO has some limitations. First, QZO's performance depends on how good the quantization method is. Specifically, if the quantization method has a large quantization error, this makes the forward passes in 70 noisy and therefore could make the gradient estimation less accurate. the other hand, QZO could benefit from a better quantization method with higher accuracy. Therefore, practitioners are suggested to choose high-precision quantization methods for QZO to maximize the gains.
Second, the performance on diffusion models lags behind LLMs because there is a noticeable gap between QZO's images and the ground truth. This may be caused by the mismatch in the noise scheduling between ZO and diffusion. One potential solution to redesign the noise scheduling in 7. such that it aligns with diffusion. We leave this as future work.

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We clarify that our research is free from the issues in the code of ethics. Our research focuses on the efficiency of LLM training and does include any human subjects. The datasets used do not include sensitive content that violates data privacy.
7 REPRODUCIBILITY STATEMENT
Our code has been publicly released to ensure reproducibility of experiments. All the datasets involved are also publicly accessible. The proof of Theorem |
provided the Appendix.
ACKNOWLEDGEMENT
This research supported by Hong Kong Research Grants Council Early Career Scheme (No. 22200824).
REFERENCES
Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLMs. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024. URL https: /tttps://operzeee //wpreevieeeeeeeeeeeeoeee /eet/foruuie 2aeee//oorummm.aafaffs33333
Noga Bar and Raja Giryes. Zoqo: Zero-order quantized optimization. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, IEEE, 2025.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1, 2009.
Christopher Clark, Kenton Lee, Ming-Wor Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of NAACL-HLT, pp. 2924 2936, 2019.
Andrew Conn, Katya Scheinberg, and Luis N Vicente. Introduction to derivative-free optimization. SIAM, 2009.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int8() 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: www.revieeeeeeeeeeeemeoe net wtw.ee/wwwee/ooumeidiiee wtt.ummiddd-010apxxxaaaaa
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman. and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 10.10088-10115. 2023.
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Alistarh. Extreme compression of large language models via additive quantization. International Conference on Machine Learning, pp. 12284-1223 12303. PMLR, 2024.
Patrick Esser. Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
10

---

Published as a conference paper at ICLR 2026
Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Ramchalam K Ramakrishnan, Zhaocong Yuan, and Andrew Z Li. Stepping forward on the last mile. Advances in Neural Information Processing Systems, 37:94851-948777 2024.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representations, 2023.
net/ ptn
Alireza Ganjdanesh, Reza Shirkavand, Shangqian Gao, and Heng Huang. Not all prompts are made equal: Prompt-based pruning of text-to-image diffusion models. arXiv preprint arXiv:2406. 12042, 2024.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entuilment und paraphrasing, pp. 1-9, 2007.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint vrXiv:2407.21768 2024.
Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume 7, pp. 785-794, 2006.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
Sijia Liu, Pin-X Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International conference on learning representations, 2019.
Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. In Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp. 8181-8196 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53000 2023.
Dang Nguyen, Wenhan Yang, Rathul Anand, Yu Yang. and Baharan Mirzasoleiman. Mini-batch coresets for memory-efficient language model training on data mixtures. In The Thirteenth International Conference Learning Representations, 2025. URL
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392 2016.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
James C Spall. Multivariate stochastic approximation using a simultancous perturbation gradient approximation. IEEE transactions automatic control, 37(3):332-311, 1992.
11

---

Published as a conference paper at ICLR 2026
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arX1v:2322.1397, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2tti::23000092888. 2023b.
Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip #:
Even better Ilm quantization with hadamard incoherence and lattice codebooks. In International Conference on Machine Learning, pp. 48630-48656. PMLR, 2024.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpon language understanding systems. Advances in neural information processing systems, 32, 2019.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. Proceedings of the 40th International Conference on Machine Learning, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- pre-trained transformer language models. arXiv preprint arXiv:2205.0000 2022.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Ilm training by gradient low-rank projection. In International Conference on Machine Learning, pp. 61121-61143. PMLR, 2024.
Jiajun Zhou, Yifan Yang, Kai Zhen, Ziyue Liu, Yequan Zhao, Ershad Banijamali, Athanasios Mouchtaris, Ngai Wong, and Zheng Zhang. Quzo: Quantized zeroth-order fine-tuning for large language models, arXiv preprint 2rXiv:2502.13346, 2025.
12

---

Published as a conference paper at ICLR 2026
A PROOF OF THEOREM 1
A.1 PRELIMINARY FORMULATIONS
In the SGD algorithm, the stochastic gradient of mini-batch B is given by
VoL(0; =
Vlo(O:m))
B
KEB
In our QZO, following Definition 3.3 the estimated gradient is formulated
VAC(A : B) = dz R zz VAL(A @ B) 2 ~ (a)
where d t((A+=2)0:08) C(( -2))0 z. Note that VoL(0; B) and VaL(A o 0) are un- biased estimate of the full gradient Ve((O) and full gradient of loss w.r. quantization scales V C( 0), respectively.
A.2
PROOF
this section, we detail the proof of Theorem estimate can be reformulated by:
Let C' be the clipping threshold, the clipped gradient
C 0) = clip(d, -c
99)
Proof. Suppose that the mini-batch B is sampled from the dataset 2, and || denotes the number of mini-batches in the dataset.
B C'
0;B)] = Ez
diz
|DD
iED
= =x N
Σ
diz
Σ dz
M
<ED,d<<[C|
iED,d,>IC|
E N
Σ
dz
Σ
C||
(10)
M
<ED,
IED,L
C
EZl
Σ
zW VAL(A
0; + E
Σ
x] (11)
M
IED
iED,d,>
V C(
0:
+0
(12)
EEz/z FR VAL(A 8 0:B)
(13)
VaL(A & 0)
Js represents the sample mean of the N observations, and the transition from Eq 12 to Eq. 13 holds because the sample μ an unbiased estimate of the
Eq. T0leuall Eq. as 6 0. Eq. 12
expectation.
B
CURVES
We include the plots loss-accuracy curves in Figure 4 to visually compare the convergence and training stability of QZO and the zeroth-order baseline, MeZO. Specifically, we use QZO and MeZO fine-tune an OPT-6.7E 7B model on the SST-2 task. We report loss values every 10 steps and test set accuracy every 4,000 steps. The illustrated loss and accuracy curves for QZO show clear convergence pattern similar that of MeZO, indicating its training stability as a zeroth-orde optimization method.
C IMPACT ON TRAINING SET SAMPLING
To explore the impact of training set sampling, we fine-tune OPT-6.7B models with QZO on down- stream NLP tasks with three different seeds, which control the training set partition. The results of
13

---

Published as a conference paper at ICLR 2026
QZO
MeZO
0.8
0.6
80
loss
0.4
65
5000
10000
15000
20000
5000
10000
15000
20000
step
step
Figure 4: Plots for loss lossaacy curves. We fine-tune an OPT-6.7B model using QZO and McZO on SST-2. The loss values are reported every 10 steps, while the test accuracy is recorded every 4,000 steps.
Table Experiments with three different seeds controlling the training set partition. The average results are reported with error bars representing the 95% confidence intervals.
Seed
SST-2
RTE
CB
BoolQ
SQuAD
0
87.6
61.7
67.9
66.4
78.5
88.2
59.2
71.4
65.9
77.2
OPT-6.7B
2
89.2
62.1
69.6
64.8
76.3
Average
88.3 = 0.9
61.0 + 1.8
69.6 + 2.0
65.7 = 0.9
77.3 + 1.3
each rum, together with the averages and corresponding 95% confidence intervals, are presented in the Table The results demonstrate that QZO is robust to training set sampling, as the performance of different seeds does not exhibit significant discrepancy.
D DISCUSSION WITH PEFT METHODS
We first clarify that the proposed QZO orthogonal parameter-efficient fine-tuning (PEFT) methods, as it directly tunes the scales of quantized models without requiring any additional trainable adapters. This also indicates that the QZO and PEFT methods can be combined fine-tune quantized models jointly using the zeroth-order optimizer. For a quantitative study, we conduct a series of experiments to compare the performance of QZO, QLoRA (Dettmers et al., (203), and the combination of both. The results are presented in Table 5 in the table to fine-tune a (quantized) OPT-6.7B model. The low-rank adapters are injected into the weight matrices of the q_oo and v v_proj layers across all experiments conducted with PEFT methods. And the LoRA rank and LoRA alpha are set to 8 and 16 consistently. We also naively combine QZO with zeroth-order QLoRA A create the variant QZO+QLoRA by fine-tuning low-rank adapters during the first half training with zeroth-order QLoRA, while training the remaining quantization scales in layers without LoRA using QZO in the second half. Based on the results, report the following findings:
Specifically, we use the methods listed
(i) First-order methods consistently outperform zeroth-order methods. This is reasonable, since real gradients are used for optimization rather than approximations. We emphasize that although QLoRA fine-tuning shows low memory usage in memory profiling, this efficiency achieved only when gradient checkpointing is enabled, and paged optimizer is used. In comparison, QZO and its variant directly achieve memory efficiency through zeroth-order optimization without bells and whistles.
(ii) possible to combine QZO and QLoRA to improve the performance while keeping the memory efficiency. Based the results, the QZO+QLoRA variant could generally outperform the original QZO while maintaining a similar low memory footprint. We also emphasize that the OZO+QLoRA oRR variant requires only 8, 000 steps, saving 60% of the total fine-tuning steps required
14

---

Published as a conference paper at ICLR 2026
Table Comparison with parameter-efficien fine-tuning methods. The LoRA rank and LoRA alpha are set to 8 and 16 consistently across all experiments. QZO+QLORA is a variant by fine-tuning low-rank adapters during the first half of training with zeroth-order QLoRA, while training the remaining quantization scales in layers without LoRA using QZO in the second half.
Method
Model Precision
Memory Profiling
SST-2
RTE
CB
Fine-tuning
16 bits
26.8 GB
95.4
79.8
73.2
First-order
LoRA
16 bits
14.0
95.6
83.8
71.4
QLoRA
4 bits
5.6 GB
96.1
84.1
67.9
MeZO
16 bits
14.8 GB
93.0
64.6
67.9
Zeroth-order
QZO
4 bits
4.8 GB
87.6
61.7
67.9
QZO+QLoRA
bits
4.9 GB
93.3
61.7
69.9
by the original QZO and McZO (Mlladi et al. 2023). We believe that combining QZO and PEFT methods can be more effectively accomplished than our naive approach, suggesting a promising direction for future research.
CONVERGENCE GUARANTEE AND TRAINING TIME
theoretical study from prior work Malladi et al. 2023 indicates that the zeroth-order optimizer guarantees convergence rate similar to SGD, with slowdown factor proportional to the local effective rank of the Hessian loss. Although QZO perturbs the quantization scales rather than the model weights in full precision, we believe this finding for the zeroth-order optimizer also supports our method. We also note that QZO requires less training time than MeZO (Malladi et al. 2023). inference kernels fine-tuning an OPT-6.7B on SST-2 with a single NVIDIA 4090, MeZO training takes approximately 4 hours and 26 minutes, whereas QZO takes approximately hours and 16 minutes.
accelerate the forward pass for quantized models. For example, when
F RESULTS WITH STABLE DIFFUSION
F.1 EXPERIMENTS ON STABLE DIFFUSION
Model and Dataset We evaluate approach on Stable Diffusion 3.5 Large (Esser et al. (024), the current state-of-the-art text-too-mage model (the largest among the 3.5 series). We choose the Styled Image Dataset (Ganjdanesh al., 2024) for evaluation, which includes the Frosting Lane, PŠ1, Tarot, and Yarn styles, with 10,000 image-caption pairs per style. For each style, the images of 512x512 resolution are split using the 8:2 train-test ratio.
Implementation Details For QZO, NF4 quantization is applied to Stable Diffusion 3.5 Large using BitsAndBytes (Dettmers et al. 2023). The batch size is set to 16. The learning rate is set to le-6. The perturbation scale is set to le-3. The total number of training steps is 20k. Following the common practice, only the DiT part in Stable Diffusion 3.5 Large is fine-tuned.
Memory Usage Stable Diffusion 3.5 Large consists of a VAE, a DiT, and three text encoders VCT/G CLIP-ViT/L, and T5-XXL). For regular training in 6/16/bff6, this model requires 0.37GB for the VAE, 21.26GB for the text encoders, 16.2GB for the DIT, 16.2GB for gradients, and considering other overheads like caches and buffers). In contrast, QZO takes only 12.4GB of memory for fine-tuning, which easily fit into single Nvidia RTX 4090 GPU (24GB). To knowledge, this is the first work showing that fine-tuning Stable Diffusion 3.5 Large can be done on a single consumer-grade GPU.
32.4GB for optimizer states, totaling 86.43GB of memory usage (without
Qualitative Results and Discussion The results are visualized in Figure 5 to Figure 8 Overall, the results are encouraging: the data distribution generated by QZO visually closer to the ground truth than the zero-shot model, which suggests that QZO works some extent for fine-tuning quantized
15

---

Published as a conference paper at ICLR 2026
Five cell phoses, aringed onder from oliod nowee. inann land
covers Zer face siting in clair beside dovenny thrugh which minor ieenn
An elepbant walking down pulloay wilka the clephert, mard tone style
A brnch of kiees that Intend taod style
huilting,
stle
Ground Truth
Zero- Shot (quanitzed)
QZZ
Figure 5: Tarot style image generation results.
text-to-image models. However, there is still noticeable gap between QZO's 's images and the ground truths.
We discuss two reasons that may explain why QZO does not produce the same level of performance as in LLMs. First, unlike discrete probability modeling as in LLMs, Stable Diffusion is essentially a regression model that predicts continuous noise values. This architectural difference leads to sensitivity issue: the deviations of the estimated gradients via ZO are directly manifested differences at the pixel level during image generation, and such errors are propagated through continuous output, resulting in fidelity degradation.
Second, recall that ZO introduces noise perturbations in latent representations. Consider a linear layer without bias, y Wa, the forward call in QZO leads to y (W nd z)a after one optimization step, where 17 denotes the learning rate and the estimated directional derivative discussed in Eq. 6 This update injects additional Gaussian noise rdd z into the activations, which is propagated through the denoising process and thus disrupts the pre-configured noise schedule (it acts as conflicting noise patterns). The diffusion model is unable to simultancously remove the scheduled and ZO-induced noise, thus resulting in incomplete denoising.
G THE USE OF LARGE LANGUAGE MODELS (LLMS)
We restrict our use of LLMs to grammar checking and writing polishing. Content translation is not used throughout the paper, and any significant use that could lead research misconduct is intentionally avoided.
16

---

Published as a conference paper at ICLR 2026
riding beown home yom style
onagg street reuth "olstruciaa" has a directional arron, yam stye
enclosed standinng two wooden tall logs. yor style
baildiing top sonues and bell on tep.. yarn style
Ground Truth
OBSTRICTION
BALAC
Zore-Shor (quantized)
obstruction
obstrucion
Our
Figure Yarn style generation comparison.
are clepbents standing paol weler. style
aamay wolking budeground, slyle
beseball player bokiing baschall during ganc. style
top lange pils. stl:
Ground Truth
Zoro Shot (quantized)
Ous
Figure 7: PS1 style generation comparison.
17

---

Published as a conference paper at ICLR 2026
girl wasning a and white manis deess and has her naaject rasiyy to bii., frvingln style
A hadnoron with nistrstnd, windows, and decoar with a tolesision stop it, fniingla style
large man bolding an unboeelia strnding pest to dag and unother persrn, Eneingh style
Several hanping out ou the sctt pla vidka games laguther for Excingo style
Groand Truth
Zoro Slott (quantized)
Omss
Figure Frosting Lane style generation comparison.
18


Reviews:
[{"review_id": "bToSyVu2cA", "reviewer": "Reviewer_EThz", "summary": "The paper proposes Quantized Zeroth-Order Optimization (QZO) for fine-tuning quantized neural networks—primarily LLMs—without backpropagation. The key idea is to apply SPSA-style zeroth-order perturbations to the continuous quantization scales while keeping discrete quantized weights fixed, thereby eliminating gradients and optimizer states and drastically reducing memory. A simple Directional Derivative Clipping (DDC) further stabilizes training by reducing gradient-estimate variance.", "questions": "1. Can QZO be combined with LoRA/QLoRA to improve accuracy while keeping memory low?\n\n2. How sensitive is QZO to ε and learning rate across bit-widths (2/3/4-bit)? Any principled schedule for C beyond static thresholds?\n\n3. Two forward passes per step suggest similar step-costs to MeZO; what are end-to-end time-to-accuracy comparisons under matched hardware?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Q-SPSA operates on Δ, keeping integer weights fixed; avoids backprop and optimizer states.\n\n2. DDC has theoretical backing and empirical impact (NaNs avoided; improved stability).\n\n3. ~18× memory saving; single-GPU (24 GB) fine-tuning for large models; reduced trainable params/FLOPs vs MeZO.\n\n4. Works across 4-bit (GPTQ) and 2-bit (AQLM) quantization; preliminary coverage of diffusion.\n\nOverall, I think this is a good paper.", "weaknesses": "1. The comparison set omits strong practical baselines such as LoRA/QLoRA/AdaLoRA on the same datasets/models/bit-widths. Since QZO and PEFT are orthogonal (QZO tunes scales; PEFT adds low-rank adapters), including them would contextualize accuracy-vs-memory/latency trade-offs more fairly than only MeZO/SGD. (Tables report MeZO and full FT but not PEFT.)\n\n2. The method’s success is acknowledged to depend on PTQ quality, but there’s no systematic study across quantizers (e.g., AWQ/SmoothQuant vs GPTQ/AQLM) at fixed bit-widths to isolate the effect of QZO when quantization error changes.\n\n3. Supervised experiments use small subsampled splits (1k train per task), which can inflate variance and may not reflect real fine-tuning regimes; larger or diverse instruction-tuning sets would strengthen claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kyrhV89nO2", "reviewer": "Reviewer_wVKC", "summary": "The paper aims to reduce GPU memory usage for LLMs by employing zeroth-order optimization on quantized neural networks, a method referred to as QZO. The key idea behind QZO is to approximate gradients by perturbing the continuous parameters involved in the quantization process while keeping the model weights quantized. The authors demonstrate that QZO achieves substantial reductions in GPU memory consumption while maintaining strong performance compared to prior methods and baselines that require significantly more memory.", "questions": "- In RL, gradient estimates often have high variance since they are computed over trajectories, for instance under the current policy in the on-policy setting. A common approach to reduce variance, which tends to be less brittle than the clipping used in PPO, is to introduce a KL-based regularization term. Could the objective be modified to incorporate such a regularization mechanism instead of relying on clipping?\n- While QZO achieves substantial memory reduction, it would be helpful to see plots showing the full trajectory of the loss and test accuracy. How quickly do these approaches converge compared to the baselines?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is a particularly relevant problem, as researchers without access to high-end GPUs often face significant challenges in conducting LLM research due to the models’ large memory requirements.\n- The paper is well written, the proposed solution is elegant, and the experiments are carefully designed to evaluate different components of QZO, such as clipping.", "weaknesses": "- Using clipping for variance reduction may make the proposed method sensitive to hyperparameter choices. As shown in Figure 3, the clipping parameter has a substantial impact on accuracy, and it is unclear how this parameter should be selected beyond trial and error.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qSbSdIhdw", "reviewer": "Reviewer_g1TF", "summary": "This paper introduces Quantized Zeroth-Order Optimization (QZO), a straightforward yet powerful method that estimates gradients by perturbing the continuous quantization scale and enhances training stability through a directional derivative clipping technique.", "questions": "0. Please first review the items listed under Weakness.\n1. Is there a comparison with the Adam optimizer? The gap between the zero-shot optimizer and Adam remains unclear.\n3. In addition to the final evaluation results, providing training loss curves would help to more intuitively compare training convergence and stability.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work proposes Quantized Zeroth-order Optimization (QZO), which significantly reduces training memory usage.\n2. The optimization of quantized weights requires no de-quantization or re-quantization.\n3. The memory optimization is effective — for example, fine-tuning Llama-2-13B on a single 24GB GPU.", "weaknesses": "1. In the Introduction, the paper lacks a clear distinction between full-parameter fine-tuning and parameter-efficient fine-tuning. The current discussion implicitly assumes full-parameter fine-tuning, which is insufficient.\n2. The experiments are also inadequate. Comparisons with parameter-efficient fine-tuning methods (e.g., LoRA, QLoRA) are missing, both in terms of memory consumption and fine-tuning performance. As a result, the paper does not demonstrate the trade-off between performance and efficiency, nor does it convincingly show the practical utility of the proposed method.\n3. As shown in Table 1, the proposed method suffers from a notable performance drop during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RIwYPcQ18E", "reviewer": "Reviewer_p1QC", "summary": "The paper studies memory-efficient fine-tuning of LLMs using quantization and zeroth-order optimization. As the size of LLMs grows quickly, the memory cost of fine-tuning becomes expensive. Zeroth-order optimization has shown significant improvement in saving memory for LLMs fine-tuning. The paper further studies how to even reduce it by quantization and proposes quantized zeroth-order optimization (QZO). Theoretical analysis is provided to show that the proposed method reduces variance in gradient estimation. Empirical experiments on fine-tuning LLMs on downstream tasks show great savings in fine-tuning memory, while preserving similar performance.", "questions": "Questions are already mentioned in weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is clearly written and has nice structure. The main claims are verified with either theory or experiments. The studied topic is well-motivated, and the problem setting is interesting and relevant to the community. By designing memory-efficent methods for LLMs fine-tuning, more and more people that do not necessarily have large computation resources can benefit from the rapid development and advancements of LLMs.", "weaknesses": "I believe there is a large body of related works that are not properly discussed, especially on fine-tuning with zeroth-order methods and quantization. \n\n1. The study of quantization for zeroth-order fine-tuning of LLMs is not new; see [1,2,3]. The paper should properly discuss what are the major differences and contributions compared to these related works. It is hard to evaluate the novelty and contributions of this work as such discussions are missing. Indeed, the settings and algorithms in this work seem to be very similar to these previous papers. Therefore, I am confused and thus require additional explanations from the authors.\n\n2. There are also other ways to further save memory compared to MeZO, such as sparse MeZO [4,5]. This could also serve as a valid baseline to compare memory consumption and model performance.\n\n3. This is less relevant. The clipped method used in the algorithm is also popular and introduced in the context of private and zeroth-order fine-tuning; see [6,7].\n\nThe experiments are a bit limited.\n\n1. The paper considers OPT and Llama, with model size 7B, 8B, and 13B. It is not clear whether the method also works for other model family, and whether the algorithm scales to even larger models, as fine-tuning 13B model does not really reach the memory bottleneck.\n\n2. Comparisons with LoRA and QLoRA are missing. LoRA and QLoRA with gradient checkpoint and gradient accumulation should also significantly reduce the memory while keeping similar performance. I am not sure how quantized zeroth-order optimization compares to that.\n\n3. The considered downstreams tasks are a bit limited. Only 5 different tasks are used.\n \n[1] ZOQO: Zero-Order Quantized Optimization. ICASSP 2025. (arXiv:2501.06736)\n\n[2] QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models. 2025. (arXiv:2502.12346)\n\n[3] Stepping Forward on the Last Mile. NeurIPS 2024. (arXiv:2411.04036)\n\n[4] Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning. 2024. (arXiv: 2402.15751)\n\n[5] Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity. 2024. (arXiv:2406.02913)\n\n[6] DPZero: Private Fine-Tuning of Language Models without Backpropagation. ICML 2024. (arXiv:2310.09639)\n\n[7] Private Fine-tuning of Large Language Models with Zeroth-order Optimization. TMLR 2024. (arXiv:2401.04343)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y0BDxNRMzT", "reviewer": "Reviewer_mqYe", "summary": "The large model size of LLM poses a challenge to computational resources. This work aims to reduce the memory cost of all the weights, gradients and optimizer states through one algorithm. The authors propose an algorithm QZO. Specifically, it uses the quantized model to reduce the memory of weights, and proposes a novel technique Q-SPSA that only perturbs the continuous quantization scale to compute the zeroth-order gradient with low memory cost. In addition, QZO adopts the clipping on the obtained zeroth-order gradient to increase the stability of training. A theoretical analysis is provided to verify its effectiveness. \n\nIn the experiments on several tasks, benefiting from reducing the memory cost of weights, gradients and optimizer states, QZO significantly reduces the memory usage compared to other baselines. Meanwhile, the gap of performances between it and full fine-tuning is not large, even higher than full fine-tuning in some cases. The ablation study shows the effects of gradient clipping.\n\nThe main contribution of this work is that it proposes an algorithm that could update the quantized weight with the zeroth-order gradient. This idea is verified in the experiments: the memory cost is around 1/3 of that of the SOTA algorithm.", "questions": "The experimental results in Table 1 show that QZO always outperforms baselines in the generalization task, but is comparable to or even obviously worse than baselines in the classification tasks. Could you give some explanations for this, i.e. why QZO performs so well in the generalization task?\n\nTable 2 shows that the trainable parameters of QZO are significantly less than baselines. The authors explain that “This is because QZO only fine-tunes the continuous quantization scale”. I think QZO just reduces the precision of the trainable parameters, further reducing the flops. But the number of trainable parameters should be unchanged, as each element $w$ has a corresponding $\\Delta$ to compute its zo gradient. Could you respond to this issue?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Previous works either reduce the memory of weights by quantization, or reduce the memory in the optimization step by the zo gradient. This work designs the Q-SPSA method to apply the zo gradient on the quantized weights. The originality and novelty are good. The experimental results reveal that this is an effective method for memory-efficient LLM training, indicating that this is a significant work. The motivation of this work is clear, and the description for the algorithm is easy to follow.", "weaknesses": "This work does not provide any convergence guarantee for the proposed algorithm. The experiments focus on fine-tuning the model on downstream tasks and do not include any pretraining tasks. \n\nTypo: the parenthesis in equation (3) seems to be incorrect.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)