Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity
Authors: Wentao Guo
Jikai Long
Yimeng Zeng
Zirui Liu
Xinyu Yang
Yide Ran
Jacob R. Gardner
Osbert Bastani
Christopher De Sa
Xiaodong Yu
Beidi Chen
Zhaozhuo Xu
Venue: ICLR 2025 Poster (2025)
Keywords: Zeroth-order optimization
LLM fine-tuning
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: Transferable static sparse zeroth-order fine-tuning LLMs on memory-constrained devices

Abstract: Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, applying ZO fine-tuning in memory-constrained settings such as mobile phones and laptops remains challenging since these settings often involve weight quantization, while ZO requires full-precision perturbation and update. In this study, we address this limitation by combining static sparse ZO fine-tuning with quantization. Our approach transfers a small, static subset (0.1%) of "sensitive" parameters from pre-training to downstream tasks, focusing fine-tuning on this sparse set of parameters. The remaining untuned parameters are quantized, reducing memory demands. Our proposed workflow enables efficient ZO fine-tuning of an Llama2-7B model on a GPU device with less than 8GB of memory while outperforming full model ZO fine-tuning performance and in-context learning.

Published as a conference paper at ICLR 2025
ZEROTH-ORDER FINE-TUNING OF LLMS WITH TRANS- FERABLE STATIC SPARSITY
Wentao Guo', Jikai Long', Yimeng Zeng', Zirui Liu Xinyu Yang", Yide Ran", Jacob Gardner', Osbert Bastani', Christopher De Sa Xiaodong Yu?. Beidi Chen', Zhaozhuo Xu²
'Princeton University, "University of Pennsylvania, Stevens Institute of Technology, *University of Minnesota, CCamegie Mellon University, "Crnell University ieceeprrinettooo
wg04200pr nice edu, filong,, yyrani,yy38,, xyu38, zxu79 9 @stevens. edu syimenz, jacobrg, obastani @seas upenn edu, zr1 Surrriuuee fu@umn. edu (xinyuya2, beidic; (aaddiicaaaddeer (andrew. cmu. edu, cdesaucs. cor.eel...oreee..eee......
ABSTRACT
Zeroth-order optimization (20) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, applying ZO fine-tuning in eeemory-constraiin settings such as mobile phones and laptops remains chal- lenging since these settings often involve weight quantization, while ZO requires full-precision perturbation and update. In this study, we address this limitation by combining static sparse ZO fine-tuning with quantization. Our approach transfers a small, static subset (0.1%) of "sensitive" parameters from pre-training to down- stream tasks, focusing fine-tuning on this sparse set of parameters. The remaining untuned parameters are quantized, reducing memory demands. Our proposed work- flow enables efficient ZO fine-tuning of an Llama2-7B model on GPU device with less than 8GB of memory while outperforming full model ZO fine-tuning performance and in-context learning. We provide an open-source implementation at https: / /ii hub. com/ /arrruarrroooaoaaa BaeeGGuoo/Sn/gggg Sens200.
INTRODUCTION
Large language models (LLMs) have demonstrated superior performance in general-purpose language generation (Brown et al., 2020; Radford al., 2019; Liu et al., 2019; Shypula al., 2024; Hicke et al., 2023; Liu et al., 2024a; Achiam et al., 2023). Despite their success, fine-tuning LLMs for specific tasks remains necessary to achieve optimal results. However, the fine- fine-tuning process often requires significantly more memory compared to inference. Specifically, there are four main components that occupy the memory during fine-tuning LLMs: (1) the weight parameter; (2) the optimizer state, which contains the information about the past gradient (Kingma & Ba, 2015); (3) the gradient used to update the parameters; (4) the activation cached to calculate the weight gradient (Liu et al., 2024c); Previous work, such QLoRA (Dettmers et al., 2023), has successfully reduced memory usage for both (1) and (2) by combining weight quantization and low-rank adaptation (Hu et al., 2021). which enables fine-tuning huge LLMs under consumer level GPUs. However, on memory-constraine hardware, such as smartphones, the memory required for caching (3) gradient and (4) activations remains significant. Prior approaches address this issue are often system- -based, such CPU offloading. The disparity between the memory demands of LLM fine-tuning and hardware capacity limits the adaptability of LLMs, especially when customizing them for edge devices.
Exploring zeroth-order optimization in LLM fine-tuning, Recently, there has been a resurgence of interest in Zeroth- Caad---rrrrrr (Z0) optimization methods for LLM fine-tuning (Malladi et al., 2023a; Liu et al., 2024c; Chen et al., 2024). ZO optimization method perturbs model parameters in random directions and utilizes the loss value difference to compute the gradient direction for parameter updates. A key advantage of ZO methods in LLM fine-tuning is that they do not require backpropagation procedures, significantly reducing computation and memory requirements. Being backpropagation- free, ZO methods do not need to cache (3) gradients and (4) activations during fine-tuning. In practice, ZO methods have demonstrated the potential to achieve performance comparable to first-order methods in L.M fine-tuning which creates new venues for efficient L.M adaptation strategies.

---

Published as a conference paper at ICLR 2025
Efficient ZO LLM fine-tuning with sparsity. Although ZO methods eliminate the need for backpropagation. a significant drawback of these methods is the slow convergence rate (Zhao et al., 2024b; Liu et al., 2024c). A recent approach addresses this by fine-tuning with sparse mask (Liu et al., 2024c; 7hang et al., 2024b), achieving approximately ~ 75% dynamic sparsity (perturb & tune 25% parameters per step). Nonetheless, this sparsity level barely reduces computational overhead, as the latency during the forward pass with even 90% sparsity is still comparable that of dense matrix operations. This latency increase can greatly impact user experience on applications such as personal assistants, where even a twofold increase in latency is perceptible. In addition, dynamic sparsity leads reduction training iterations but not necessarily wall-clock time determine and apply the sparsity pattern for each training step could be expensive. Moreover, dynamic sparsity inherently assumes the whole model must all in dense weights, and an attempt to combine dynamic sparse training with parameter-size reduction techniques such as quantization is not computationally tractable (otherwise it will involve frequent dequantization and quantization). This raises the questions:
possible to develop an extreme static sparsity method for ZO fine-tuning that is easy to combine with the quantization method? Would the memory efficiency of 20 be pushed even further?
Our proposal: ZO LLM fine-tuning with transferable static sparsity. In this paper, we answer the raised research question by proposing a transferable static sparse ZO LLM fine- tnn-tuning strategy. We observe an extreme sparsity pattern in LLM parameters: a subset, determined by selecting the top k magnitude entries from the diagonal of the empirical Fisher information matrix, effective for ZO fine-tuning. Moreover, we find this sparsity pattern can be obtained through LLM's pre-training process and transferred various downstream tasks without modification (as static selection).
Summary of contributions. Building on these insights, our work proposes a comprehensive framework for ZO fine-tuning, making the following contributions:
We identify that only an extremely small portion (0.1 %) of LLM parameters should be updated during ZO LLM fine-tuning. Moreover, we observe that this sparsity pattern be derived in LLM pre- trainrrinng process and transferred across different downstream tasks while still maintaining good ZO performance without any modification.
Based on this observation, we propose SensZOQ, an on-device LLM personalization workflow via integrating Sensitive ZO optimization with Quantization to further improve the memory- efficiency of 7.0 fine-tuning (Figure 1).
We conduct extensive experiments across various LLMs and demonstrate that our method achieves competitive performance across various downstream tasks.
RTE
WIC
COPA
80
Model waights
CUDA Merooyy (GB)
Optinizer
60
ZO Pertriattio
FO Fardwwadw
GB
GB
8 GB (our target)
SensZOO
MaZO
FO.Adam
FO Adam
SeneZ0Q
MeZO
FO-Adam
FO Ad-Adam
SensZ0Q
MazO
FO Ao-Adam
FO Ad-Adaam
LoRA
FT
LoRA
FT
LoRA
FT
Figure I: CUDA memory profiling of Llama2-7B on 3 fine-tuning tasks (all with batch size 8). "SensZOQ" our method, and Me70 refers the 70 full fine-tuning (Malladi al., 2023a). "Z0" refers to backpropagation-Fin free zeroth-order optimization method, and "FO" refers to the first-order optimization method. "LoRA" stands for Low-Rank adaptation (Hu et al., 2021). We find that SensZOQ can meet the 8 GB memory target without any CPU offloading.
2

---

Published as a conference paper at ICLR 2025
2 SPARSE ZO FINE-TUNING WITH STATIC SENSITIVE PARAMETERS IN LLM
In this section, we first go through the background 20 optimization Section 2.1. We then inspect an extreme sparsity pattern in LLMs in Section 2.2 and its theoretical guarantees in Section 2.3.
2.1 ZEROTH-ORDER OPTIMIZATION
Zo surrogate gradient estimator. ZO optimizers have been studied widely in the machine learning community. Given dataset D (x1,)) y1)..... parameters E Rd ZO optimizer will estimate the gradient at w via ZO surrogate gradient estimator. Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall, 1992) is such an estimator that would first sample a random vector IR and uses the loss value difference to scale the update direction. z is usually sampled from an Gaussian distribution
(x.......n... (Xn.9n.) Un)} and a loss function f with model
Definition (Simultaneous Perturbation Stochastic Approximation (SPSA) (Spall, 1992)). SPSA estimates the gradient w.r.f. w with data example (x, y). small constant e E R, and sampled random vector z E Fd as follows:
g(w, (x, y), z) f(w + ez; (x, u)) f(w ez; (x, u))
(I)
2e
There are other ZO surrogate gradient estimators (Liu et al., 2020; Ohta et al.. 2020), but in practice, SPSA achieves good performance in ZO fine-tuning. Some 70 algorithms such as DeepZero (Chen et al., 2024) would utilize the parameter-wise finite difference of loss values to derive parameter-wise update directions. This would yield (d) query costs per training step even when combining with certain sparse masking methods and is not practical for LLM fine-tuning scenarios. We therefore select SPSA with random Gaussian perturbation as our ZO gradient estimator.
ZO-SGD algorithm. ZO-SGD is optimizer similar to SGD but replaces the FO gradient with ZO surrogate gradient estimate per training step, as defined below:
Definition 2 (ZO-SGD update rule). ZO-SGD is an optimizer that uses ZO surrogate gradient to update parameters we with learning rate πi and a data example (xt, yt) sampled at timestep t:
W++1 = Wt - T):0 (Wes (Xt, Mt), Ze)
MeZO (Malladi al., 2023a) is ZO-SGD algorithm that uses the "random seed trick" to the need of caching ZO surrogate gradient. The choice of optimizer (SGD) is orthogonal to ZO optimization techniques, but in our preliminary experiments we find adaptive optimizers such as Adam (Kingma & Ba, 2015) would not necessarily accelerate ZO convergence in LLM fine-tuning scenarios. There are other 70 optimizers aware of the parameter-wiss heterogeneity of loss curvatures to accelerate the optimization convergence (Zhao et al., 2024b), and we leave how to combine our method with theirs as future works.
2.2 SPARSE ZO OPTIMIZATION WITH STATIC SENSITIVE PARAMETERS.
Given model parameters w, loss function f. a data example (x. y). sensitive parameters are defined as parameters whose corresponding FO coordinate gradient square values are maximized.
Definition 3 (Sensitive parameter mask). A sensitive sparse mask mg {0, 1) with k nonzero entries /E mi) = k) is defined as'
my argmax, m
ff(w)
(3)
In the context of ZO optimization, we will update sensitive parameters only. Denote that ž = z O m We will modify the SPSA gradient estimator from ğ( (w, (x, y), z) to g(w, (x, v), z), and accordingly:
Definition 4 (Sensitive sparse ZO-SGD update rule).
W++1 W. - (Wes (Xes Mt), Ze mg,)
The theoretical support of sensitive parameters can be derived from the lens of SPSA gradient estimator and Fisher information matrix as follows:
When the context is clear, we will abbreviate f(w; (x, v)) as f(w) and Vf(w; (x) as Vf(w). Notice that for full batched gradient we will use V.F(w).
3

---

Published as a conference paper at ICLR 2025
Maximum zeroth-order loss value changes, from the lens of ZO SPSA estimator.
The square (account for negativity) of loss value difference for 9w (w, (x, y), 2) is as follows:
Ez (f + ez; ()) f(w eZ; (x, x))}² ~ {2 Vw [w)( 4e" | I Wwf(w)l2
Since by Definition 3 our sensitive mask would maximize m| @ [w](w)]l for a given sparsity ratio, we would expect our sensitive mask to maximize the magnitude of the loss value difference for any given sparsity ratio. This property is important for ZO as ZO directly leverages the loss-value difference a probe of loss landscape to determine the descent direction.
Maximum coverage of Hessian diagonal, from the lens of Fisher matrix. LLMs are often pre-trained
large text corpus' to reach low perplexity before entering the fine- tuning stage. In this case, we would assume PLIM x) (y(l))) which implies the empirical Fisher F should be close to the (true) Fisher matrix F as follows:
F Exx-po.b~puuttt V log x) V log (LLM(())))
~ F (xx V Pog (L))))
As assume the empirical Fisher matrix approximates Fisher, which also approximates the Hessian, and empirical Fisher's 's diagonal is equal to the coordinate-wise gradient square vector when computing with downstream task-specific loss, our sensitive parameters would cover a large fraction of the largest Hessian diagonal entries.
RTE
WIC
COPA
Cumu. norm. sum of
D [V f(:: [X,, Xx>)
1.00
moat
0.75
0.50
0.25
0.00
10
10
10
10-2
10-1
10P
10-5
10
10-3
10
10
100
10
10
10
10-1
10a
Ratio
Ratio
Ratio
Figure 2: Cumulative normalized sum of wordinate-wise E. [V (w: (x., })) of linear layers during Llama2-7B (Touvron al., 2023) FO-SGD full fine-tuning. For each linear layer, first sort parameters by the decreasing order of their gradient square value >> [V[(;; (x; 1:))]² and we take the cumulative + std of all blue curves. Similar figures for Mistral-7B and OPT-6. 7B are in Figure 7. We observe that roughly 1.1%-1%% % parameters contribute about 50% gradient norm square.
and normalize it draw blue curve, and the red-shaded region is the mean
This idea of sensitive parameters has been studied in the quantization community (Kim et al., 2024; Guo et al., 2023) and FO optimization (Sung et al., 2021). However, we are the first one leverage the extremely sparse sensitive parameters in LLM fine-tuming to accelerate ZO fine- fimettning with LLMs. When we have perturbation and updating the scale of billion parameters, finding which parameters to fine-tune would be important for improving 70 performance. Notice that here we use sensitive masks m. for understanding purposes. In Section 3.2, we will discuss how to transform Definition to parfmeter-efffcient optimization pipeline via transferable static sparsity.
2.3 THRORETICAL CONVERGENCE RATE
We would investigate the theoretical convergence sensitive sparse ZO-SGD on sensitive parameters under the non-convex optimization settings. Our assumptions included in Appendix C.I.
Theorem 1 (Convergence rate of sensitive sparse ZO-SGD (Definition 4)), If we pick n = 1/(L(k+2)) under Assumptions (bounded gradient error), (Lipschitz, smoothness), and 4 (sparse sensitive parameters), we have
V F(w:)l < ( 4 (F (F(o))
Σ Ez (x.)) 1=0
-
(5)
Heee assume data examples (x.y) PD in fine-tuning datasets after verhalization would also appear in the large text corpus during pre-training.
4

---

Published as a conference paper at ICLR 2025
If we pick 77t ()) with an extra P.L.. condition (Assumption 3), we have
( -F) 171=(1-0) (1-0 ""= (F(wo) -F)
o²
(6)
24
The proof for Equation in Appendix C.2 and the proof for Equation is in Appendix C.3. we choose & d and 1, both convergence rates trivially reduce to the standard zeroth-order convergence rates as (d/T)) + (cconstant) and (1 (1/d))² + O(constant). As we assume > k/d, we know d > k/c and therefore both ((k/cc and (1 O(c/k))² are much lower than (ddTT) and (1 O(1/d)² that zeroth-order method will yield.
We want emphasize that our contributions are more on empirical LLM fine-tuning instead of general machine learning tasks, and in Section 4.2 we extensively compare our sparse ZO methods with other sparse ZO methods and we demonstrate its superiority during LLM fine-tuning. We do not use the strict "local r-effective rank" assumption that Malladi et al. (2023a) uses, and our Assumption 4 can be easily observed empirically in Figure 7. Liu et al. (2024c) and Ohta al. (2020) also provide analysis on the convergence of sparse 7.0 optimization but they do not include our sensitive sparse masks in their studies.
SENSZOQ: A SPARSE ON-DEVICE FINE-TUNING RECIPE
In this section, we describe the transferable static sparsity pattern we observed in LLMs and how we utilize it for developing an on-device fine-tuning pipeline of LLMs SensZOQ.
3.1 TRANSFERABILITY OF STATIC SENSITIVE PARAMETER
Transferable sensitive gradient features: from dynamic to static sparsity. Our Theorem focuses on dynamic sparse fine-tuning. However, Panigrahi al. (2023) notices that in real LLM fine-tuning scenarios, the fine-tuning performance could be attributed to sparse subset of weights (~ 0.01%). Malladi et al. (2023b) also finds certain fine-tuning tasks would demonstrate kernel behaviors, which include "fixed (gradient) features": V (Waterr FT; (x, y)) V (Woetore FT: (x. y)).
The similarity of gradient features during fine-tuning would imply that we do not need to re-select our sensitive parameters during fine-tuning i.e. select once before fine-tuning should be sufficient. This hypothesis can be validated by Figure and Figure 6b. In Figure 3, the fact that "task grad, static" does not vanish and still has large ratio over "task grad, dyn." " the end of training demonstrates that we can select parameters before fine-tuning.
Surrogate static sensitive parameter mask. Another observation from Figure 3 is that the sensitive parameters derived from pre-training datasets (C4) would still cover a large fraction of model sensitivity. Specifically, the parameters overlap between top C4 gradient entries and task gradient entries are much (>20x) higher than all weight magnitude baselines. Therefore, we could use it as surrogate sensitive sparse mask when gradients on downstream tasks are unavailable, particularly in scenarios of on on-device personalization.
C4 covers diverse of text corpus across different domains and we believe it will produce generally good transferable static mask. We also note that if we have better knowledge of the exact downstream domain or task our method will be applied to, we can extract sparse masks with better specific task performance. So we include an ablation study on other surrogate sensitive parameter masks Appendix D.3 and we analyze the overlap ratio top gradient features in Appendix E.3.
3.2 SENSZOQ: AN OPPORTUNITY FOR ON-DEVICE LLM PERSONALIZAATOO
Transferable static sparse fine-tuning as a parameter-efficient optimization method. The sparse optimization on fixed parameters can be implemented as parameter-efficiee optimization workflow, which will reduce the perturbation and updating time during ZO optimization. Suppose we have
Obtainiig gradients of LLMs on edge devices is expensive, and we usually cannot transfer data from edge devices to the cloud compute the gradient downstream tasks the cloud. In this case, would need some surrogate gradient information derive sensitive sparse masks onthe cloud. We will discuss this in Section 3.2.
5

---

Published as a conference paper at ICLR 2025
RTE
WIC
COPA
101
Cumu. norm. sum of
22 ( ((w:; (x., Bo))l*
10""1 .80
2004 grad Johy!
wegtte jieppet, siatici weights (a mallesstt siancj rander (eark))
385% grad (stonic)
grad (stanic
Before Epoch
Epoch
End
Before
Epoch
End
Figure 3: Cumulative normalized gradient square values (y-axis) of different sparse masks on Llama2- 7B checkpoints during FO-SGD full FT. "task grad (dyn.)" refers to the sensitive parameters selected at the given timestep (x-axis) on each downstream dataset, and "task grad (static)" refers to the sensitive parameters selected before tine-tunigg. "C4 grad (static)" refers to the sensitive parameters selected once with gradients taken from causal language modeling on C4 dataset (Raffel et al., 2019). "weights (largest/smallest, static)" mean pick the weights with the largest/smaalest magnitude in the pre-trained model, respectively. We provide a longer discussion in Appendix E.2.
derived sensitive sparse mask mg, and we know is fixed during fine-tuning. Instead of applying mg to we would apply it directly to and extract the nonzero parts as below:
O mc, Weense = C (ld m;
Squrse
Denote Zi,t (O I&) as the Gaussian perturbation sampled in timestep L. We will determine Wxparse before fine-tuning and optimize on Wepase only and leave Wdense frozen during fine-tuning. In this case, our sensitive sparse ZO-SGD update rule will become:
W W+parse,r+
W Wsparse,t
T/.6 W sppass,ts (Xt, Vr), Zk.r)
(8)
Equation 8's formulation allows an opportunity for an on-device personalization workflow.
SensZOQ: integrating sensitive sparse ZO fine-tuning with quantization. As LLMs are often pre-trained with user-agnostic public datasets, personalizing LLMs with individual users' preferences and meeting users' specific needs before real-world deployment are vital (Tan et al., 2024a; Mairittha et al., 2020). However, transferring the user-specific data to the upstream cloud before fine-tuning LLMs would raise privacy concerns (Xu al., 2018). On the other hand, personal devices usually have less computational budget and are more memory-constraing than the cloud (Zhu et al., 2023), and performing full fine-tuning would easily exceed the device memory budget.
In response, we propose an on-device personalization workflow SensZOQ illustrated in Figure 4. surrogate gradient information from pre-training datasets
The high-level overview is that V (wPLLM(y/x) to extract sensitive parameters W and keep W Wsparse in 16 bits, while we quantize the remaining dense weights Waense (Step 1-4). We send Wsparse and Q Wdense) personal devices (Step 5), and we perform on-device ZO fine-tuning only on Wsparse (Step 6).
We highlight that SensZOQ's memory consumption nearly minimal: we can fine-tune Llama2-78 model under 8 GB GPU memory without any offloading as illustrated in Figure 1. This would satisfy memory constraint by a wide range of edge or mobile devices as illustrated in Table 13.
In addition, our method does not put strict constraints on specific choices of quantization al- gorithms since any algorithm that aims to minimize the least-square quantization error term Q(w) 1E 2020; Frantar et al., 2022; Lin et al., 2023; Kim et al., 2024).
w)xllž or its variant would suffice (Chee et al., 2024; Nagel al.,
Efficient implementation of sensitive sparse linear layers. In Appendix F, we discuss how to efficiently implement our sensitive sparse ZO ane-ttnnng in forward passes of linear layers with training and inference workflow: use Equation 19 when we have access to efficient uniform integer matmul to compute @(Waense) Q(x) and in other cases, we would usually use Equation 21 for token generation and Equation 23 for 7.0 training.

---

Published as a conference paper at ICLR 2025
Cloud
Edge
1. Get gradients from pre we-traininn datasets
Get surtogate sonsitive sparse parameter masks
V. (ap
stgmax,, Im @ C |
4. Quantiz dense partbs
send models personal devices
dense and
b0 Wine
mady to serve!
w (1, ma)
Wicure 193
Wagura
Wapuw
Waguwm" 27
Predrsined LLM
Apply sumaitive musks
fine-duning
Figure 4: SensZOQ: an on-device LLM personalization workflow via integrating Sensitive ZO optimization with Quantization. The high-level idea is that we first decompose w to w Wsparse and Wensee and then quantize dense (on the cloud), On the edge devices, we will fine-tune sparse only.
EXPERIMENTS
In this section, aim to validate the effectiveness of our SensZOQ, shown in Figure 4, as memory- efficient LLM fine-tuning solution. This naturally leads to comparison with other ZO methods, which we evaluate in Section 4.1. Additionally, we assess the effectiveness of our sensitive parameter mask derived from pre- traininin texts (C4) against other heuristic sparsity methods in Section 4.2. Specifically, we aim address the following research questions:
RQ1: What is the performance of our SensZOQ compared with other ZO methods?
RQ2: optimizing C4-gradient-derivee sensitive parameters more effective than optimizing other subset of parameters during 70 fine-tuning?
We focus on 7B-level LLM models, including Llama2-7B (Touvron et al., 2023), Mistral-7B (Jiang et al., 2023), and OPT-6.7B (Zhang et al., 2022) they would fit with common on-device memory constraints (8 GB) listed on Table 13 after applying quantization. We use SST-2 (Socher al., 2013), RTE (Wang et al., 2018), CB (De Marneffe et al., 2019). BoolQ (Clark et al., 2019), WSC (Levesque et al., 2012), WiC (Pilehvar & Camacho-Collados, 2019), COPA (Roemmele et al., 2011), and WinoGrande (WinoG) (Sakaguchi al., 2020) datasets. We follow standard ZO fine-tuning settings and use the same codebase as in Malladi et al. (2023a). More details of our experiments (hyperparameters, task-specific prompts, etc.) are described in Appendix G.
We include additional results for Llama2-13B and OPT-13B, and harder tasks such as commonsense reasoning, math reasoning, and MMLU in Appendix D. To the best of our knowledge, there are no ZO-LLM research yet evaluated on commonsense reasoning or math tasks. We take pioneering step in this direction and establish these ZO baselines.
4.1 -N-DEVICE PERSONALIZATION
We evaluate the performance of our SensZOQ method in Table 1. We follow the exact recipe as described Figure 4, where we only optimize 0.1% sensitive parameters derived from a small batch of C4 texts top of 4-bit quantized model. SensZOQ's s's results are shown as 1st row for each subtable.
Comparison with ICL & ZO Full FT. The results in-context learning (ICL) and ZO full fine- tuning (ZO Full FT) on FP16 models are shown the 3rd and 5thh row for each substable in Table 1. 7B models are usually not large enough such that ICL would outperform with FT (Liu et al., 2022: Mosbach et al., 2023). addition, ICL induces additional KV-cache memory burden if the number of demonstration examples is too many. Our method SensZOQ does not induce significant inference latency and memory burden as it only needs use % parameters. Sens7OQ also outperform ICL. and 70 Full FT. This is impressive given that quantization would degrade performance of the base model, but SensZOQ still manages to match the fine-tuning performance.
Comparison with ZO LoRA. The primary purpose of quantization is to represent parameters in less bits (therefore reducing model sizes) and improve system-level metrics such as weight loading time and inference latency (Dettmers et al., 2022; Chee al., 2024). In order retain such benefits
7

---

Published as a conference paper at ICLR 2025
Table Fine-tuning performance of different methods. In the first column, "Q" means the full model is quantized with 4-bit quantization method (SqueezeLLMM (Kim et al., 2024)), and "ZO-SGD" means the model is fine-tuned with ZO-SGD optimizer. For each cell, we report the mean and standard deviation of validation set accuracy (1) of 3 random trials in the format of meanad- We finally report the average accuracy in the last 2 columns. Notice that we add the results for FO-SGD & FO-Adam for reference and we do not use it for performance rank computation.
(a) Llama2-7B
Methods
SST-2
RTB
CB
BoolQ
WSC
WiC
COPA
WinoGi
Acct
ZO-SGD
SensZOQ
91.4u.s
76.9..0
70.8s
83,90.4
61.90.9
63,40.8
85.7..9
65. lo.s
75.3
LoRA
92.90.a
58.70.9
67.91.6
77.71.4
62.80.6
61.3p.4
85.00.0
62.91.a
71.2
ZO-SGD
Full FT
94.60.s
73.3
66.7a.a
81.90.s
60.63.4
61.9p.a
82.71.
63.10.4
73.1
7ero-shon ICL
80.0o.a
57.80.0
32.1o.0
69.90,2
50.2p.0
36.5p.o
79.0g.0
64.80.8 8
59.9
94.8p.2
71.54.a
72.6:5.2
77.54.6
53.2
61.14.a
87.02.a
67.3,
73.2
PO-SGD
Full FT Full FT
95-40.
10.
73.2g.0
85.
62.8o.s
72.01.8
85.31.2
71.11.7
78.6
TO Adam
95.50.
85.70.7
92.91
85.30.9
6..11.2
71.3p.4
85.3o.5
78.31.a
82.3
(b) Mistral-7B
Q Z0ZDGGG
SensZOQ LoRA
94. 94.2p.4
76.8s.a
67.34.s
76.31.0
59.91
62.2p.4
87 87.To.s
73.60.7
74.8
91.61.4
69.32.2
64.9p.s
65.22.0
57.71.
62.11.1
88.30.a
71.6p.8
71.3
ZO SGD
Full FT
94.60
74.62.1
68.8p
76.60.
54.86.
62.60
88.30.
72.20.5
74.1
7em-shot ICL
54.86.0
50.5o.o
37.50.0
43.41.8
50.80.0
39.4p.o
78.0o.a
66.
52.6
60.716.7
55.2..7 7
$3.313.1
46.86.5
50.46.
633. 13.88p..
88.70.5
74.0p
59.1
FO-SGD
Full FT
94.9o.e
87.61.2
85.7s.9
86.10.7
62.50.0
70.
88. 88.31.
82.11.1
82.3
FO-Adam
Full FT
15.1p.2
86.4..r
88 15.4
83.11.s
64.77.3
72.72.9
82.7..7
85.9g.3
82.3
(c) OPT-6.7B
Q O0ZD-GG
SensZQQ LoRA
94.6,
74.1 1
81.03.0
70.41.0
57-72.4
62.2g.
81.7o.9
64.8 6
73.3
93.8p.9
71.0g.4
69.01.7
70.2
60.6, .6
57.5p.s
84.3
63.0p.9
ZO-SGD
Full FT
94.4p.a
72.71.2 a
79.8s.o
72.11.2
G7.44,
60.20.9
82.32.6
64.60..
72.9
Zero-shot ICL
61.0p.o
60.7o.0
46.40.0
55.71.0
55.5o.0
36.5p.o
77 0o.0
61.1o.a
56.7
74.014.0
65. 65.811.2
54.8s.9
67.92.1
53.21.7
41.04.5
80.7g.
61.bg.s
62.4
FO-SGD
Full FT
20.2
81.8p
92.35.0
79.2
59.07.7
66.5a.3
85. a
68.8p.6 6
78.6
FO-Adam
Full FT
95.7
81.12.6
83.9s.
81.1p.7
56.17.9
66.5p.a
81.3,.2
66.4g.a
76.5
during fine-tuning stage, our fine-tuning methods should be parameter-efficient. A natural baseline becomes fine-tuning other PEFT methods such as LoRA (Hu et al., 2021) on top of the same quantized LLM weights as SensZOQ. These results are shown as the 2nd row for each substable in Table 1. SensZOQ still outperforms LoRA when applied to the same 4-bit quantized base model.
Empirical convergence speedup over ZO full FT. To investigate the empirical convergence rate of SensZOQ, we plot their training losses w.r.t. 20k optimization steps for OPT-13B fine-tuning tasks, as shown in Figure 5. In this figure, "Sensitive ZO (C4 mask, 4 bits)" is our SensZOQ method, and the Sensitive ZO (C4 mask, FP16) will keep the dense weights in FP16.
In Figure 5a, we find that a direct learning rate transfer from 70 Full FT to SensZOQ already results in similar convergence. In Figure 5h, we identify that SenzZOQ achieves faster convergence under the same hyperparameter searching efforts. In Figure 5b, the best learning rates for ZO Full FT are still 2 while we use 5e-7 for Sensitive ZO and produce much faster convergence results.
Comparison with Adam Full FT of smaller size model. The memory-efficiency of 70 achieved via cheap queries in the loss landscape with random perturbation. The absence of accurate descent direction naturally leads an inferior performance than first-order (FO) full FT on the same model size, as also observed by Malladi et al. (2023a). However, we argue that such performance degradation is still acceptable in terms that ZO FT on larger models outperforms FO FT on smaller models. In Table 2, pick 2 popular LLM sizes (IB and 7B scales) and find that applying SensZOQ 6.7B OPT model generally outperforms FO-Adam full FT on 1.3B OPT model, and the latter already surpasses the 8 GB memory budget (e.g., 12.5 GB RTE compared to SensZOQ's 5.6 GB).
8

---

Published as a conference paper at ICLR 2025
RTE
WIC
COPA
0.70
Sens 20 (C4, 4 bits)
Sens live 20 (CA, FP16)
2.2
Training loss
0.65
20 Full FT (FP16)
0.70
2.0-
0.69
0.60
1.8-
0.68
1.6-
0.55
5
10
20
10
15
20
0
5
10
15
20
Optimization steps (x10")
Optimization steps (x10")
Optimization steps (x10")
(a) Convergence of Sensitive ZO vs. 20 Full FT under the best learning rate for ZO Full FT only. WIC
RTE
COPA
0.7
2.25
0.6
0.70
2.00-
Training
0.5
Sensitive 2D (C4, 4 bits)
0.68
1.75
Sonstive (C4, FP16)
1.50
20 Full FT (FP16)
0.66
0.4
20
5
20
Optimization steps (x10*)
Optimization steps (x10")
Optimization steps (x110)
(b) Convergence of Sensitive Zo vs. ZO Full FT under the best learning rate for each method.
Figure 5: Convergence of SensZOQ for OPT-13B. SensZOQ refers to "Sensitive ZO (C4 mask, 4
bit)", and we also provide the convergence results of fine-tuning FP16 sensitive weights. In Figure 5a, We first search the best learning rate for ZO Full FT that reaches the lowest training loss in [1e-7, 2e-7, 5e-7, 1c-6] grid, and we use such learning rate for sensitive ZO. In Figure 5b, we search for the best learning rate for ZO Full FT and sensitive ZO separately the same grid. The other hyperparameters are kept the same for both subfigures. We plot the mean over 3 random trials for all lines.
Table 2: Fine-tuning performance of SensZOQ versus Adam FT of smaller model in the OPT family. We follow the same experiment procedure as in Table
Params
Methods
SST-2
RTF
CR
RoolQ
WSC
WIC
COPA
WinoG
Acct
bits
6.7B
SensZOQ
94.60.7
74.11.1
81.0s.0
70.41.a
57.72.4
62.2p.6
81.70.9
64.81.
73.3
Z0DSGD
FP16
Z0 SGD
6.7B
Full FT
94.4g.3
72.71
79.83.0
72.11.2
37.41.6
60.2p.9
82.3g.6 6
64.60.3
72,9
BF16
FO-Adam
1.3B
Full FT
93.6g.a
73.9a.4
75.03.
73.8..0
61.9,.2
62.41.5
76.71.2
60.8o.s
72.3
4.2 EFFECTIVENESS OF SPARSE ZO FINE-TUNING ON SENSITIVE PARAMETERS
Comparison with other static sparsity masks. We investigate the performance of optimizing our sensitive parameters versus other subsets of parameters in a static sparsity regime with the FP16 model. We consider standard weight-magnitude baselines as weights with largest or smallest magnitude (Sparse:MeZO's 's sparsity patterns (Liu et al., 2024c)), and random subset of weights baseline. There are some other weight importance metrics the pruning community, such as GraSP (Wang et al., 2020), and we also evaluate their performance in the static transfer setting. We note that these pruning metrics were originally proposed for deciding which parameters to retain instead of being removed during pruning, and it has no direct implications for ZO FT. Given threshold vector T, the definitions of all methods are listed below:
sensitive parameters with C4 gradients: m [ FTi (Xc4, NC4)) 2
random subsets: m = random_diin
vector watrr_wthhhh__t k anz k)
weights with largest magnitude: m Wocfore FT 2
weights with smallest magnitude: m = Whefore FT S T
smallest GraSP scores: m Whefore FT ( E(x.)) -D H (Whefore FT) V ('Whefore FT) $
9

---

Published as a conference paper at ICLR 2025
WIC
COPA
Average
Via
100 sparseç
90-
730
Accuracy
62.--
Si-
70.5
Coeeettt cosoaeo (CA
700+
aer prightu carger (all)) ai alrgati sonatie,, (AB
87-
07.5
Fanddr
68-
Full frekkingg
10'4
10-2
16:01
10'1
Pallo
Ratio
Ratio
Ratio optimize
(a) Static transferability performance of different sparsity masks. "static" means that we will determinc the trainable parameters (sparsity mask) before fine-tuning and other parameters are kept unchanged.
RTE
WIC
COPA
Average
la small
Accuracy
Ph5--
62-
Semethe CC gred, stalo
82-
07.5
Seculioo Jusk qail (ee)
Rondoon
10"2
10-44
18
107
06
10
10
16
18
10
16-
0001
16
10-1
10
Patio optimize
Fallo optinize
Ratio optimine
Ratio to optimbee
(b) The performance of sensitive parameters derived from causal LM loss in C4 datasets and gradients from each fine mne-tuning task. "Dyn." means the sparse masks are updated every 100 training steps (dynamic sparsity).
Figure 6: Performance of different sparsity methods in Llama2-7B ZO fine-tuning.
As illustrated in Figure 6a, we can find that ZO FT would benefit from sparse optimization, as all methods would achieve higher than 70 full FT when optimizing 10% parameters. However, only sensitive parameters would maintain its performance as we move to the extreme sparsity region (<1%). In fact, the performance curve of sensitive parameters w.r.t. different sparsity levels is near flat curve, which indicates the performance loss by moving from 10% to 0.1% is minimal. We also find that optimizing weights with smallest magnitude more effectively than optimizing weights with largest magnitude, which aligns with Liu et al. (2024c)'s findings. However, sensitive parameters is still more effective for optimizing weights with smallest magnitude. This suggests that sensitive parameters are more effective to serve as the sparse parameters in our SensZOQ instead of SparseMeZO's 's weights with smallest magnitude.
Transferability of SensZOQ's sparsity masks. SensZOQ uses C4 gradients to produce transfer- able static mask used downstream tasks. In Figure 6b, we compare the performance of optimizing sensitive parameters with gradients on C4 dataset with its theoretical upper bound: static sensitive parameters derived from gradients on each fine-tuning task as the solid line and its dynamic version as the dash-dotted line. We also include the static and dynamic random subset parameters as a bascline. We can find that the gap of sensitive parameters between deriving from gradients on the C4 dataset and gradients on each finettnning task ratio 1e-3 is small. Together with Figure that we evaluate the top gradient entries' similarity between C4 and downstream tasks, we believe SensZOQ's sensitive masks from C4 gradients would yield satisfactory performance in general.
CONCLUSION
In this work, we identify that only a small portion of LLM parameters needs to be updated during ZO fine-tuning, and these static and sparse subset parameters can be derived during the pre-training phase and transferred across various downstream tasks without requiring any modifications, preserving effi- cient ZO performance. We propose SensZOQ, a workflow that integrates sparse ZO optimization with 4-bit quantization to further enhance the memory efficiency of on-device fine-tuning. SensZOQ lever- ages static sparse fine-tuning to enable the personalization of 7B LLMS on-device, reducing memory consumption to less than 8 GB of CUDA memory. In addition to this efficiency, SensZOQ achieves better performance than both in-context learning and ZO full fine-tuning. Therefore, SensZOQ creates a new venue to facilitate on-device fine-tuning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This research used resources of the Argonne Leadership Computing Facility, U.S. Department of Energy (DOB) Office of Science user facility at Argonne National Laboratory and is based on research supported by the U.S. DOE Office of Science-Advanced Scientific Computing Research 1357. We also appreciate the support from Amazon, Intel, Li Auto, and Moffet AL We also thank Lin Xiao and numerous anonymous reviewers for providing valuable feedback on this paper.
Program, under Contract No. DE DEEAACO--0CCII
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087 303007777774 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Aochuan Chen, Yimeng Zhang, Jinghan Jia, James Diffenderfer, Jiancheng Liu, Konstantinos Parasyris, Yihua Zhang, Zheng Zhang, Bhavya Kailkhura, and Sijia Liu. Deepzero: Scaling up zeroth-order optimization for deep model training. In International Conference on Learning Representations, 2024. doi: 0..48500/rxi..211.000
Christopher Clark, Kenton Lee, Ming-Wos Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, 2019.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. A discourse-aware attention model for abstractive summarization of long documents. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), 2018.
doi: 10. 11853//1/118-200000 URL http: / /xxx doi org/ 1861186633// / 8090
A Feder Cooper, Wentao Guo, Khiem Pham, Tiancheng Yuan, Charlie F Ruan, Yucheng Lu, and Christopher De Sa. Coordinating distributed example orders for provably accelerated training. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Tri Dao. Flashattention- Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.0869) 2023.
Marie-Catherie De Marneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In Proceedings of Sinn und Bedeutung, pp. 107-124, 2019.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3, int8 (): 8-bit matrix multiplication for transformers scale. Advances in Neural Information Processing Systems, 35: 00318-30332, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 10088-10115. Curran Associates, Inc., 2023.
11

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Ming-Wei
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations, 2019. doi: 10.48550/arXiv. 1803.03635.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.11 17323, 2022.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2001.00027, 2020.
Tanmay Gautam, Youngsuk Park, Hao Zhou, Parameswaran Raman, and Wooseok Ha. Variance- reduced zeroth-order methods for fine-tuning language models, 2024.
In Gim and JeongGil Ko. Memory-efficient dnn training on mobile devices. In Proceedings of the 20th Annual International Conference on Mobile Systems, Applications and Services, pp. 464-476, 2022.
Han Guo, Philip Greengard, Eric Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. In The Twelfth International Conference on Learning Representations, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Yann Hicke, Abhishek Masand, Wentao Guo, and Tushaar Gangavarapu. Assessing the efficacy of large language models in generating accurate teacher responses. arXiv preprint Att::2307.04274 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2/06.09668 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. Lim-adapters: An adapter family for parameter-efficien fine-tuning of large language models. In Proceedings of the 2023 Conference on Empiricul Methods in Natural Language Processing, pp. 5254-5276, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Shuoran Jiang, Qingeai Chen, Youcheng Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization. Proceedings of the AAAI Conference on Artificial Intelligence, (8(16):18363-18377 Mar. 2024. doi: a0.1609/ammi.v3888 v38.16.29796.6666 http::
//ojs. aaai. org in//xx.dxxx../// php/ http://tttieeeeiieeeeeeeeeeeeeee
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 200060000000,,,,, 2020.
Schoon Kim, Coleman Richard Charles Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-sparse quantization. In Forty-first International Conference on Machine Learning, 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. doi: 10.48550/arXiv. 1412.6980..
12

---

Published as a conference paper at ICLR 2025
Hector Levesque, Emesst Davis, and Leora Morgenstem, The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
Luchang Li, Sheng Qian, Jie Lu, Lunxi Yuan, Rui Wang. and Qin Xie. Transformer-lite: High-efficiency deployment of large language models on mobile phone gpus. arXiv preprint arXiv:2403.20041, 2024a.
Zeman Li, Xinwei Zhang, and Meisam Razaviyayn. Addax: Memory-efficient fine-tuning of language models with combination of forward-backwane and forward-only passes. In 5th Workshop practical ML for limited/low resource settings, 2024b.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang. and Song Han. Awq: Activation- aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.0000 00978, 2023.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. ACL, 2017.
Zhenqing Ling, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. On the convergence of zeroth-order federated tuning for large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 1827-1838, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704901. doi: 10.1145/3637528. 3671865. URL https:tt///// /doi. orgoorgg///oooo /10.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng. Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412. 19437, 2024a.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficien fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-111 1965, 2022.
Hong Liu, Zhiyuan Li, David Leo Wright Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. In The Twelfth International Conference on Learning Representations, 2024b.
Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod Varshney. A primer zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications. IEEE Signal Processing Magazine, 37(5):43-54, 2020.
Yinhan Liu, Myle Oit, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 11692, 2019.
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order 1lm fine-tuning. arXiv preprint arXiv:2402.1575: 15751, 2024c.
Liu, J Lou, W Bao, Y Hu, Li, Z Qin, and K Ren. Differentially private zeret methods for scalable large language model finetuning. arXiv preprint arXiv:2402.07818. 2024d. URL https: aaxiiiii.. o als//////// abss/242 2402 07818.
Zichang Liu, Jue Wang, Tri Dao, Tianyi Zhou, Binhang Yuan, Zhao Song, Anshumali Shrivastava, Ce Zhang, Yuandong Tian, Christopher Re, et al. Deja vu: Contextual sparsity for efficient 1lms at inference time. In International Conference on Machine Learning, pp. 22137-22176, PMLR, 2023.
Zirui Liu, Guanchu Wang, Shaochen Henry Zhong, Zhaozhuo Xu, Daochen Zha, Ruixiang Ryan Tang, Zhimeng Stephen Jiang, Kaixiong Zhou, Vipin Chaudhary, Shuai Xu, et al. Winner-take-al column row sampling for memory efficient adaptation of language model. Advances in Neural Information Processing Systems, 36, 2024e.
13

---

Published as a conference paper at ICLR 2025
Yucheng Lu. Wentao Guo, and Christopher M De Sa. Grab: Finding provably better data permutations than random reshuffling. Advances in Neural Information Processing Systems, 35:8969-8988 2022.
Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. Adalomo: Low-memory optimization with adaptive learning rate. Findings of the Association for Computational Linguistics ACL 2024, pp. 12486-12502, 2024.
J MacQueen. Some methods for classification and analysis of multivariate observations. In Pro- ceedings of 5-th Berkeley Symposium on Mathematical Statistics and ProbabilityUniivesity of California Press, 1967.
Nattaya Mairittha, Tittaya Mairittha, and Sozo Inoue. Improving activity data collection with device personalization using fine-tuning. In Adjunct Proceedings of the 2020 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2020 ACM International Symposium on Wearable Computers, pp. 255-260, 2020.
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D) Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. Advances in Neural Information Processing Systems, 36:53038-53079 2023a.
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pp. 23610-23641. PMLR, 2023b.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations. 2016.
Marius Mosbach, Tiago Pimentel, Shauli Ravfogel, Dietrich Klakow, and Yanai Elazar. Few-shot fine- tuning vs. in-context learning: A fair comparison and evaluation. arXiv preprint arXiv:2305.16638, 2023.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Mayumi Ohta, Nathaniel Berger, Artem Sokolov, and Stefan Riezler. Sparse perturbations for improved convergence in stochastic zeroth-order optimization. In Machine Learning, Optimization, and Data Science: 6th International Conference, LOD 2020, Siena, Italy, July 19-23, 2020, Revised Selected Papers, Part 11 6, pp. 64. Springer, 2020.
Yijiang Pang and Jiayu Zhou, Stochastic two points method for deep model zeroth-order optimization arXiv preprini u4Xvv:2402.01621, 2024. URL httpt:ttp//// //axxii wogg /2.b2///4222 01 621.
Abhishek Panigrahi, Nikunj Saunshi, Haoyu Zhao, and Sanjeev Arora. Task-specific skill localization in fine-tuned language models. In International Conference on Machine Learning, pp. 27011- 27033. PMLR, 2023.
Gunho Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Jung Kwon, Byeong- wook Kim, Youngjoo Lee, Dongsoo Lee, et al. Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models. The Twelfth International Conference Learning Representations, 2024.
Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba. Openwebmath: An open dataset of high-quality mathematical web text. In The Twelfth International Conference on Learning Representations, 2024.
Zhimin Peng, Ming Yan, and Wotao Yin. Parallel and distributed sparse optimization. In 2013 Asilomar conference on signals, systems and computers, pp. 659-646 IEEE, 2013.
Mohammad Taher Pilehvar and Jose Camacho-Collee
Wie: the word-in pard-in-contex. dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 1267-1273, 2019.
14

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv e-prints, 2019.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 201/ AAAI Spring Symposium Series, 2011 1.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8732-888888 8740, 2020.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604, PMLR, 2018.
Alexander Shypula, Aman Madaan, Yimeng Zeng, Uri Alon, Jacob Gardner, Milad Hashemi, Graham Neubig, Parthasarathy Ranganathan, Osbert Bastani, and Amir Yazdanbakhsh. Learning performance-improvn code edits. In International Conference on Learning Representations, 2024. doi: 2302.07867.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Evan Walsh, Luke Zettlemoyer, Noah Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo. Dolma: an open corpus of three trillion tokens for language model pretraining research. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 37725-15785 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. URL https: /aw.aaaannnnooooooaaaaaaaaaaalat org/ 2024 aal.laclllnnla. 840.
James Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE transactions automatic control, 37(3):332-341, 1992.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193 -42005, 2021.
Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democra- tizing large language models via personalized parameter-efficient fine-tuning. arXiv preprint 24l1:2402.04401. 2024a.
Zhen Tan, Tianlong Chen, Zhenyu Zhang, and Huan Liu. Sparsity-guided holistic explanation for 11ms with interpretable inference-time intervention. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 21619-21627, 2024b.
Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mittal. Private fine-tuning of large language models with zeroth-order optimization. arXiv preprint arXiv:2401.00 04300004433,,,, 2024. URL https::t///......... /aaxiviiiiv..////////aa////aaavv org/ aat/a/ss22240 htt.l/.rggbbbbb///bb//
04343.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307 002888. 2023.
15

---

Published as a conference paper at ICLR 2025
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, 2018.
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference on Learning Representations, 2020.
Haocheng Xi, Changhao Li, Jianfei Chen, and Jun Zhu. Training transformers with 4-bit integers. Advances in Neural Information Processing Systems, (6:49146-4911 2023.
Haojun Xia, Zhen Zheng, Yuchao Li, Donglin Zhuang, Zhongzhu Zhou, Xiafei Qiu, Yong Li, Wei Lin, and Shuaiwen Leon Song. Flash-llm: Enabling cost-effective and highly-efficient large generative model inference with unstructured sparsity. In Proceedings of the VLDB Endowment, Vol. 17, No.
2, 2023. doi: 10114788662699233266600
Mengwei Xu, Feng Qian, Qiaozhu Mei, Kang Huang, and Xuanzhe Liu. Deeptype: On-device deep learning for input personalization service with minimal privacy concern. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2(4):1-26, 2018
Zhaozhuo Xu, Zirui Liu, Beidi Chen, Shaochen Zhong, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, and Anshumali Shrivastava. Soft prompt recovers compressed Ilms, transferably. In Forty-forst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: /pwnr//ieeeeeeeeeeeeeeeeeeeeeeee net de mor:mmid-mmBBBPIII222
Xinyu Yang, Jixuan Leng, Geyang Guo, Jiawei Zhao, Ryumei Nakada, Linjun Zhang, Huaxiu Yao, and Beidi Chen. s'ft: Efficient, scalable and generalizable Ilm fine-tuning by structured sparsity. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020.
Eric Zelikman, Qian Huang, Percy Liang, Nick Haber, and Noah D. Goodman. Just one byte (per gradient): A note on wow-bandwidth decentralized language model finetuning using shared randomness. arXiv preprint arXiv:2306. 10015, 2023. URL https://arxiv/ http//axivvv./// org/ abs/ 10015.
Liang Zhang, Bingcong Li, Kiran Koshy Thekumparampil, Sewoong Oh, and Niao He. Dpzero: Private fine-tuning of language models without backpropagation. arXiv preprint arXiv:2310.06339 2024a. URL https: arxiv v. wwwp/asg////.oooooo///.../...eet /tbs//311 2310 09639.
Linfan Zhang and Arash Amini. Label consistency in overfitted generalized k-means. Advances in Neural Information Processing Systems, 34:7965-7977, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre- mre-trained transformer language models. arXiv preprint arXiv:2205.0 00068,, 2022.
Yihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin- Yu Chen, Jason D Lee, Wotao Yin, Mingyi Hong, et al. Revisiting zeroth-order optimization for memory-efficient Ilm fine-tur mee-ttning: A benchmark. arXiv preprint arXiv:2402.11592 2024b.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient Ilm training by gradient low-rank projection. In International Conference on Machine Learning, pp. 61121-61144 143.. PMLR, 2024a.
Yanjun Zhao, Sizhe Dang, Haishan Yc, Guang Dai, Yi Qian, and Ivor W Tsang. Second-orde fine-tuning without pain for llss: A hessian informed zeroth-order optimizer. arXiv preprint 1511 2024b.
Shaochen Zhong, Guanqun Zhang, Ningjia Huang, and Shuai Xu. Revisit kernel pruning with lottery regulated grouped convolutions. In International Conference Learning Representations, 2021.
16

---

Published as a conference paper at ICLR 2025
Shaochen Henry Zhong. Zaichuan You, Jiamu Zhang, Sebastian Zhao, Zachary LeClaire, Zirui Liu, Daochen Zha, Vipin Chaudhary, Shuai Xu, and Xia Hu. One less reason for filter pruning: Gaining free adversarial robustness with structured grouped kernel pruning. Advances in Neural Information Processing Systems, 36, 2024.
Ligeng Zhu, Lanxiang Hu, Ji Lin, Wei-Ming Chen, Wei-Chen Wang, Chuang Gan, and Song Han. Pockengine: Sparse and efficient fine-tuning in pocket. In Proceedings of the 56th Annual JEEE/ACM International Symposium on Microarchirecture, pp. 1381-1394 2023.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
In Section A, we discuss some related works of our paper. In Section B we describe all notations used in this paper. In Section C, we include the assumptions and exact proof on the convergence rate (Theorem 1). In Section D, we include experiment results on OPT-13B and Llama2-13B. We also take a pioneering move to investigate the effectiveness of SensZOQ and ZO methods in commonsense reasoning, Math, and MMLU tasks. In Section E, we inspect the appearance of sensitive parameters across models and tasks. We also investigate the effects of different data sources that would produce different sensitive masks. In Section F, we provide a high-level recommendation on how to efficiently implement our SensZOQ2s framework in linear layers with existing quantization methods or training/inference workflow. In Section G, we describe miscellancous details (hyperparameters. task templates, hardware config, etc.) in our experiments.
A RELATED WORKS
Zeroth-order fine-tuning of LLMs. Since MeZO (Malladi et al., 2023a) first demonstrates the ef- fectiveness of ZO for LLM fine-tuning, ZO has attracted great research interest from the LLM commu- nity in different aspects. For example, ZO is effective for edge device fine-tuming in communication constrained and federated settings (Ling et al., 2024; Zhang al., 2024a; Tang et al., 2024; Liu et al., 2024d). Notably, Zelikman et al. (2023) illustrates the possibility of exchanging single-byte projected gradients in distributed zeroth-order workloads, yielding both communication and privacy benefits. Numerous research has also focused on enhancing the optimizer aspect of zeroth-order optimization (Jiang et al., 2024; Pang & Zhou, 2024; Gautam et al., 2024). Other researchers are also interested in improving ZO's efficiency convergence rate from cleverer optimizer designs. Li et al. (2024b) explores the middle ground between balance the convergence speed and memory footprints. Liu et al. (2024c) and Zhang et al. (2024b) suggest that sparsity would potentially accelerate ZO optimization convergence. Zhao et al. (2024b) preconditions ZO perturbation with knowledge from parameter-wis wise loss curvature heterogeneity to gain convergence speedup. To the best of our knowledge, Liu et al. (2024c)'s work is the only ZO fine-tuning with sparsity (smallest weight magnitude mask) work this moment, and we have ablated on its static extreme sparsity performance in Figure 6a.
FO-SGD and large-batched batched ZO-SGD to
Sparsity in LLM. Sparsity-driven techniques are widely adopted in improving ML model's effi- ciency (Frankle & Carbin, 2019; Xia et al., 2023; Liu et al., 2023; Peng et al., 2013; Xu et al., 2024; Tan et al., 2024b) and robustness (Zhong et al., 2024; 2021). Frankle & Carbin (2019) shows that within large feed-forward networks, there exists a subnetwork that, when trained in isolation, can achieve test accuracy comparable to that of the original network. In the foundation models era, Liu et al. (2023) demonstrates that transformer-based models, such OPT (Zhang et al., 2022), exhibit great sparsity (z 95%) in activations. Moreover, Panigrahi et al. (2023) discovers that for RoBERTa (Liu et al., 2019), fine-tuning very small subset of parameters (~ 0.01%) can yield performance exceeding 95% of that achieved by full fine-tuning.
First-order gradient optimization methods. Researchers in the optimization community also propose variety of first-order optimization methods to promote faster convergence or reduce the memory footprints. For faster convergence, Liu al. (2024b) designs a second-order optimizer that approximates the diagonal Hessian and enables more stable and faster LLM training. You et al. (2020) proposes a variant of Adam that enables the use of large batch sizes to finish BERT (Devlin et al., 2019) pre-training within 76 minutes. Cooper et al. (2023) and Lu et al. (2022) also attempt to effectively reorder the training dataset online to accelerate convergence. To reduce memory footprints, Zhao et al. (2024a) projects the gradient to low-rank subspace that will simultancously reduce gradients and Adam optimizer states. Both Shazeer & Stern (2018) and Lv et al. (2024) factor the second-moment of adaptive optimizer to reduce the Adam optimizer state memory costs.
18

---

Published as a conference paper at ICLR 2025
B NOTATIONS
We present the notations used in this work as follows.
Table 3: Notations used in this paper
Term/Symbol
Explanation
loss function
optimization timestep
74
number of training data examples (training dataset size)
d
number of model parameters
duyee
number parameters
linear layer. This means the number input channels times the number of oulput channels in each linear layer.
(Nc, 8b.)
data example sampled timestep as pair of input vector and training target
Wr E R
weight/parameter vector at optimization timestep F
(x,,))
training loss of evaluated at single data example (x. y)
F(w)
full-batched training loss of w
(w:: (xx,))
Hessian matrix of evaluated (x,y)
a small perturbation scaling constant (close to 0)
Ze IRd
random Gaussian perturbation vector sampled timestep
0(w, (x, z)
estimated ZO surrogate gradient for w with a data example (x, y) and sampled
Gaussian perturbation vector z (Definition 1)
"/
learning for ZO-SGD optimizer (Definition 2) timestep
my (0, 1)"
sensitive sparse mask with l nonzero entries (Definition 3)
ma,
E 1),
sensitive sparse mask with & nonzero entries derived at optimization timestep /
Ia
Identity matrix with shape pdxd
Identity matrix Ia with the main diagonal masked by my
1a
vector of size d with all entries equal
trace operation
((w)
parameter vector that is quantized by quantizer Q
F
(true) Fisher information matrix
F
empirical Fisher information matrix
PLIM
L.M probabilistic model
po
distribution
L
Lipschitz constant in Assumption
14
P.I. condition number in Assumption 3
stochastic gradient error term in Assumption the query weight matrix in self attention layers the key weight matrix in self attention layers
Wq
Wx
Wv
the value weight matrix in self attention layers
Wo
the output weight matrix self attention layers
Wome
the weight matrix for the gated linear unit in SwiGLU layers
Wip
the weight matrix for the up projection layer in SwiCLU/MLP layers
Woown
the weight matrix for the down projection layer in SwiGLU/MPP layers
19

---

Published as a conference paper at ICLR 2025
C THEORETICAL CONVERGENCE RATE
C.I ASSUMPTIONS
We start with listing standard assumptions in the nonconvex optimization literature:
Assumption 1 (Bounded stochastic gradient errors). For any data example (x. D and for any w E RR, denote the full-batched loss function F(w) = E (xxx)))D] (x, ()) we have [(; (x, y)) w. S o².
(9)
Assumption 2 (Lipschitz smoothness). We assume that f [w., x) L-Lipschitz, smooth (I 0): for any w, w E Rd
f (x, y)) - Vwf(w'; (x. v)) S L[[w-w'll
(10)
Assumption (P.L. inequality). We assume that F(w) fulfills the Polyak- Lojjak--jjsieiiczz (P.L.) condi- lion: there exists some μ > 0, for any w E TR
1/ [IV 2 ((F())-F') JF is the minimum value F = inf F(w).
(11)
w
Inspired by Figure 7, we assume the sensitive parameters of w are sparse.
Assumption 4 (Sensitive parameters are sparse). We assume at timestep 3m, E {0, 13d with the number of nonzero entries k, 3c E [0, 1] such that
|| O V (ws
w))1 Ww 91)1??
Here we assume e >> k/d." 4
C.2 PROOF FOR EQUATION 5, THEOREM
We will start with formulating the expectation of sensitive sparse ZO surrogate gradient norm square in terms of its corresponding stochastic gradient norm square.
Lemma (Sensitive sparse Z0 surrogate gradient norm square).
Eg [ We, (x,, v1). 2, 1 = (2 + k))c V fw.(w, ()))?
Proof for Lemma 1. We know that our Z can be considered being sampled from N/ Ia.me) where ma is the identity matrix La with the main diagonal masked by mp-
We expand the sensitive sparse ZO surrogate gradient covariance matrix as follows:
E2ğ(w, (x, y), Z)g( w, (x, v), T)T
Ez, | (mmg O V (mm @ (w ž,
= 2 ((m: O V fw: (x, y) (ma & Vw (x,) - || o V fw (x; 1))
Then the sensitive sparse 70 surrogate gradient square is the square of the diagonal of its corresponding covariance matrix:
Ex We, xx, 2 diag (w, (x, y), 2) v),
2c| V w o (x)) kell (xe, ve))
= (2 + k)cl Www(w 16)11
Then we are in good shape of deriving the convergence rate under the Lipschitz smoothness condition:
From Figure 7, we know that for 0.5, we only need k/d ~ 0.001 to 0.01. In this case k/c 0.002d 0.02d.
20

---

Published as a conference paper at ICLR 2025
Proof for Equation 5, Theorem I.
(1) S (x, + (V/ ([3;; (x,, Wo+11 -1:--w -w)) +
|| - wwel?
2
Lib ([ Ne. 2i)l
$ f(w,; (xx,, b1)
(
f f(w; (x,, v.)).(0)..w... (,)) +
2
Lije
E2f(W(+1. < Ezf(ww v1)) Bg ma. Vf(we: (xe, 3o 1]12
Eg 11 (w, X4, 2]
Exf(We+1 _ (w,: (x4. - on Ex (w)) (x,, Be))11 + L
c(k + 2 V f(wr: (xc: Bt.)
2
(x.( < (x.,) (F(w)) -nnll V Fw:)ll + Lid 2
Lun ckk 2)0²}
c(k + 2) V wwF((w:)
2
(z.( S (x.v) (F(w)) (on-
Lin
Lリ ck 2)
c(k 2) [[:))l
2
Denote Lo(k 2), we will have
Eg (x,) (x.v) (F(w)) -97e
(c-
2 211)
F(w:))
a
2
1
L(k 2) Suppose we use constant learning rate T/: = a
we have
Eg, (x.g)) S (x.,)) {
(((w,) on |VF(w,)) 02n2
(12)
If we apply our descent rule (Equation 12) recursively for T steps,
a 021
T
Σ Ex (x | F(w:)1
2 aqi
(F(wo) -F))
T
2
21 + 2)
(F(wo) - -F)) + 00
cT
2L + 2)
(F(wo) - -F") +
c
0 T
(F(wo) F*) + O (constant)
C.3 PROOF FOR EQUATION 6, THEOREM
We can derive convergence of sensitive sparse Z0-SGD optimization method under P.I.. inequality and
as follows (this proof resumes from our prior proof with the Lipschitz-smoothness condition alone):
Proof for Equation 6, Theorem 1. Let's resume from Equation 12:
'If we use diminishing learning rate can remove the constant noise term, but this is not our main focus in this paper.
21

---

Published as a conference paper at ICLR 2025
Eg (x,(x.)) S (x,)) (F(w)) o [V((:)))? + a o
2
2
S (x,() (F(w)) - cHm(F(w.) Ft} + 0²1
(2 (F(w++1) - F S E (x,)) ((F(we) FF') (F((F( -F')
2 "
a 2
(x,(x.g) (F(W+11) - 3 s (1 - cuu) Ex((x) - F + a
2 00
1
1 and sum up recursively for T iterations,
Plugging in constant learning rate v/ = min(
T.(K: + 2)' cy
(F (1
qu
(F(((o)) F*) + 0
(1
cpm)
L(k + 2)
2
j=0
< (1
qu
[[((ww)) F²) + a
1
L(k 2)
2
cuu)
I (1
((F(wo) - FF) +
a
L(k + 2)
2
cp
{
cp
(T((wo)) -F*) + 2p
I(k: + 2)
= -11 0 (" 4
T k)) ) (F(wo) F)) + Occostannt)
22

---

Published as a conference paper at ICLR 2025
D MORE EXPERIMENT RESULTS
D.1 13B MODEL RESULTS.
In Table we compare SensZOQ vs. ZO full FT ICL in 2 popular 13B models: Llama2- 13B and OPT-13B. SensZOQ still maintains its superior performance over both ICL and ZO Full FT. In the last row of OPT- 13B (Table 4b), we ablate on the effect of 4-bit Squeezel.I.A quantization (K-means (MacQueen, 1967; Zhang & Amini, 2021) based quantization) and as expected, the average test accuracy is roughly the same after we remove the quantization on the dense weight parts.
Table 4: Fine-tuning performance of different methods. In the first column, "Q" means the full model is quantized with 4-bit quantization method (Squeezel.LMM (Kim et al., 2024)), and "70" means the model is fine-tuned with ZO-SGD optimizer. For each cell, we report the mean and standard deviation of test set accuracy (1) of random trials in the format of meansid- We finally report the average accuracy across tasks in the last column.
(a) Llama2-13B
Methods
SST-2
RTE:
CB
BoolQ
WSC
WIC
COPA
WinoG
20
SensZ0Q
95.80.3
78.81.0
81.54.5
80.6g 3
63.5p.8
59.8p.9
89.70.5
74.6.
78.0
Full
95.0g.
74.41.0
85.7
50.5g.4
54.84.7
60.9
90.7p.5
70.0g.a a
Zero-shox ICL
70.6p.0
G4.3p.o
48.20.
76.10.2
39.44..
50.6g
83.00.0
65.4g.4
61.6
95.4g.
78. 23.62.5
79.85.
75.74.2
57.41.6
57. $7.6g.0
89.30.9
71.3p.6
75.6
(b) OPT-13B
ZZ
SensZOQ
93. 83.6p.a
75.2..1
69.01.7
73.01
61.5u.a
61.0
87.0u.a
65.1g.2
73.2
ZO
Full
93.9g.5
74.0,
67.92.5
72.40.3
61.52.4
58.62.3
87.0,
63.31.3
72.3
Zero-shot
61.0p.o
68.5p.o
48.4p.0
58.80..
36.00
0
60.77.
57.1
ICL
63.0g.5
59.8
72.01.7
71.82.4
38.15 38.12.3
84 0g
63.2p
65.6
(C4, static)
94.1g.6
75.12.1
69.0g.2
72.01.0
59.02.0
60.9p.7
98.7p.6
65.40.7
D.2 COMMONSENSE REASONING, MATH, MMLU DATASET RESULTS.
In Table 5, still compare SensZOQ vs. ZO full FT & ICL in standard commonsense benchmarks (Hu et al., 2023; Yang et al., 2024), math algebraic word problem task AQuA (Ling et al., 2017), and MMLU (Hendrycks al., 2021). To the best of our knowledge, there yet evaluated on harder commonsense reasoning or math tasks. We take a pioneering step in this direction and establish the ZO baselines.
ZO-LLM research
If we compare SensZOQ with ZO full FT on pairs. SensZOQ wins 6/8 for commonsense reasoning, and the math task. However, SensZOQ loses MMLU task, and we speculate it might be due to a data distribution mismatch between C4 and education domain in MMLU. In Table we find that switching the source of extracting sensitive parameters from C4 OpenWebMath will significantly close this gap. Even so, C4 is still generally good choice.
D.3 ALTERNATIVE DATA SOURCES FOR EXTRACTING SENSITIVE PARAMETERS.
C4 (Raffel et al., 2019) is extracted and cleaned from the CommonCrawl and covers a wide range of domains, and this drives to adopt it as the default choice for extracting sensitive parameters. Here, we evaluate the SensZOQ's performance when fine-tuning sensitive parameters extracted from some alternative text choices that have certain domain specialties. As case study, we pick 3 alternative choices that are more domain-specific and also commonly selected for pre- pre-trainin trainininn dataset mixtures (e.g. Dolma (Soldaini al., 2024)) follows:
23

---

Published as a conference paper at ICLR 2025
Table Fine-tuning performance different methods for Mistral-7B commonsense reasoning tasks (cs), math task (math), and MMLU task. For each cell, we report the mean and standard deviation of validation accuracy (1) of random trials in the format of mean We finally report the average accuracy across tasks in the last column.
Methods
Are-E (os)
(re-C(en)
(or)
OBQA (es)
PIQA
SIQA (ox)
(oulQ
WinoG (ex)
AQuA (marh)
MMLU
Avg
Sem200
79.8p.9
8346.5
77.90.7
8496.9
69.41.0
76.41.0
7366.7
26.1g.7
58.44..
71.9
Full
89.60.4
78.31
78.80.3 a
54.30.3
68.5p.4
76.6
72.20.5
59.2o.1
Zero-soot
86. 86.80.0
75.90.0
77.90.8
71.0p.0
82.1
59.90.s
43.
66.20.1
23 23.51.9
57.50.0
64.4
JCL
90.5g 2
50.0g.0
80.31.4
79.80 7
$4.0g.9
69.9
46.86.5
74.06.s
59.20.3
ArXiv (Cohan et al., 2018), pile of scientific papers. We the ArXiv articles subset from this iii___ppeeeeeesssepep
dataset. https: /tgg://ggaggg.c..ee.c.oooo co dataaats//ramaaaaa http///raac////////aa/aaa/aaaaaa
OpenWebMath (Paster et al., 2024), pile of Internet mathematical proofs. https:// muggingfacee coodatasett open-web-math wwwebb-mtthhope-weeb-bbb web--web--atth
Wiki103 (Merity et al., 2016), pile of selected Wikipedia articles. https://huggingtfie wikitext
co
Table 6: Fine-tuning performance of SensZOQ with 0.1% sensitive parameters extracted from C4, OpenWehMath ArXiv, and Wiki103 for Mistral-7B on 8 commonsense reasoning tasks, math task, MMLU task, and SuperGLUE tasks (task category follows this order and is separated by vertical bars). For each cell, we report the mean accuracy (f) over random trials. In the last row, we give ZO Full FT & ICL baselines as reference. We finally report the average accuracy across tasks the last column.
Source
Arc-E
Are-C
HS
OBQA
PIQA
SIQA
BoolQ
WinoG
AQuA
MMLU
RTE
WiC
COPA
Avg
83,6
87.7
ArXiv
89.1
76.0
82.4
76.3
84.3
69.1
76.0
73.6
26.9
58.4
70.0
61.6
88.7
71.7
OpenWehMath
89.5
79.9
82.7
75.7
83.1
67.4
73.8
73.2
26.9
58.8
77.9
61.1
88.0
72.2
Wiki103
77.6
ZO Full FT
89.6
78.3
82.3
76.8
84.3
68.5
76.6
72.2
24.0
59.2
74.6
62.6
88.3
72.1
79.8
55.2
In Table 6, we can find that when fine-tuning 0. 0.1% sensitive parameters with Mistral-7B. C4 achieves the highest average accuracy, with notable performance on commonsense QA tasks like OBQA, PIQA and SIQA. If we want better performance on an expert-level knowledge task such as MMLU, OpenWebMath is better choice than C4.
We believe that C4 is not the only choice but rather generally good choice for downstream tasks, and the latter more desired as we are using the same sel of sparse parameters for different tasks and want it to produce satisfactory performance for as many tasks possible rather than focusing excellent performance on a single task. Otherwise, we would have to create a separate sparse mask and quantized models for each task, and this will make our method impractical.
We also believe that a high-quality and diverse pre-training dataset mixture such as Pile (Gao et al., 2020) and Dolma (Soldaini et al., 2024) should also perform well, and we leave this study future research.
24

---

Published as a conference paper at ICLR 2025
E SENSITIVE PARAMETERS IN LLMS
E.1 GRADIENT SPARSITY DURING LLM FINE-TUNING
SuperGLUE tasks. In Figure 2, we explore the gradient sparsity of Llama2-7B during FO-SGD fine-tuning (at epoch 1 or 10% training steps). Here we plot the FO-SGD gradient sparsity for Llama2- 7B, Mistral-7B, and OPT-6.7B before fine-tuning, epoch (10% steps), and end of fine-tuning.
We observe that the gradient sparsity is exhibited throughout the fine-tuning and increases toward the end. OPT-6.7B, which uses ReLU as the activation function, demonstrates greater sparsity across tasks compared to Llama2-7B and Mistral-7B which both use SiLU. Overall, the gradient sparsity pattern is generally consistent across architectures, tasks, and fine-tuning time.
25

---

Published as a conference paper at ICLR 2025
Cumulativv nornalized sum of f/w; (Ki, s:))
RTE, Before FT
RTE, Epoch
RTE, End of FT
104
0.
WIC, Belore FT
WiC, Epoch
WiC, End of FT
00-
COPA, Before FT
COPA, Epoch
COPA, End of FT
104
Ratio
Ratio
Ratio
(a) Lllmm2-7B
Cumulaaivv normalized sum of V (fri (Ke, s:)
RTE, Before FT
RTE. Epoch
RTE, of FT
104
WIC, Before FT
WIC, Epoch
WiC, End of FT
0.5
COPA, Before FT
COPA, Epoch
COPA, End of FT
104
19-3
10 10m1
-0-5
10-2
10-11
102
10"5
10-2
Reto
Rasio
Retio
(b) Mistral-7B
Cumulaaive normalized sum S..1 V f(w;; (36, 22
RTE, Before FT
RTE, Epoch
RTE, End of FT
1.0-
00-
WIC, Before FT
WIC, Epoch
WIC, End of FT
104
COPA, Before FT
COPA, Epoch
COPA, End of FT
00-
10-5
10-4
10-1
20
10-11
10P
-0
10
10-11
10--2
10-11
109
10-
10
20t
10-2
10-1
10
Ratio
Ratio
Ratio
(c) OPT-6.BB
Figure 7: Cumulative normalized sum of coordinate-wist gradient square
(x,, w:))]²
of linear layers for Llama2-7B (subfigure 7a), Mistral-7B (subfigure 7b), and OPT-6.7B (sub- figure 7c) across RTE, WiC, and COPA tasks during FO-SGD full FT for 10 epochs. For each linear layer, we first sort parameters by the decreasing order of their gradient square value and normalize draw
- [V [V[(;; (Xi, y:) y2) [dhnyer] and we take the cumulative blue curve, and the red-shaded region is the mean + std of all blue curves.
26

---

Published as a conference paper at ICLR 2025
Arc-C, Before FT
Arc-C, Step 100
Arc-C, End of FT
1,0
[V ([/(w (X we)
0.5
600
HellaSwag. Before
HallaSwag. Step 100
HellaSwag, End
1.0
0.-
Cumulattvv normalized sum of
PIQA, Before FT
PIQA, Step 100
PIQA, End of FT
1.0
0.5
AQuA, Before FT
AQuA, Slep 100
AQuA, End of FT
1.0
0.5
0.0
10
100
10
10
10
10
10
Ratio
Ratio
Ratio
Figure& 8: Cumulative normalized sum of coordinate-wise gradient square 4 of linear layers for Mistral-7B across Are-C, HS, PIQA, and AQuA tasks during FO-Adam full FT for 1000 steps.
-2 V f(w; (x,, y:)) ))}²
Reasoning tasks. We further analyze the sparsity of sensitive parameters for hard reasoning tasks such Arc-C, HellaSwag, PIQA, and math reasoning task AQuA. We follow the same method that produced Figure 7b to produce Figure 8. We can see that sensitive parameters are still fairly sparse: we still need 0. %- % sensitive parameters for sufficient coverage of the gradient norm.
E.2 TRANSFERABILITY OF GRADIENT FEATURES FROM TRAINIGGN DATASETS TO DOWNSTREAM TASKS
SuperGLUE tasks. In Figure 3, we explore the transferability of gradient features from pre-training datasets (C4) to downstream tasks, and here will also validate this phenomenon across models, as shown in Figure 9. As there are no solid lines (top-(le-2,1) 2,- (e-3,le-4) parameters with C4 gradient entries prior to fine-tuning) vanish to O, know the transferability of gradient features from C4 datasets to downstream datasets hold across models and tasks. We also include the results of weights with largest magnitude, weights with smallest magnitude, and random subsets weights in Figure 10 at le-3 nnz level (same as lc-3 in Figure 9) for comparison. It is clear that "C4 grad (static)" has much higher similarity with "task grad (dyn.)" than all of these baselines. also note that weights with largest magnitude (weight outliers) are usually NOT gradient outliers as observed in Figure 10.
In this case, sensitive parameters determined from C4 gradients would still be similar to sensitive parameters determined from downstream task-specific gradients across models.
Reasoning tasks.
The transfcrability results for reasoning tasks are shown in Figure 11.
As there are still no solid lines (for all C4 gradient mask still demonstrates great transferability to these commonsense and math task. At step 100, the lowest covered gradient norm for top-1e-3 C4 gradient mask is ~022 while the maximum possible (task grad, dyn.) is ~0.3 across tasks.
parameters with C4 mask) vanish to O,
We still note that for AQuA (math algebraic word problem task), C4's transferability is weaker than the other commonsense reasoning tasks. We speculate that this is due to the need learning more math knowledge during FT the covered task gradient squares by task gradient mask before FT mask (lask grad, static) also declines more during FT.
27

---

Published as a conference paper at ICLR 2025
COPA
-- [P (K,, (N..
Cumu sum
6.8
0.6
6-4-
Bstore
Epoch
End FT
Balord FT
Epesh 1
Eld
BrtreFT
Epoch
Endel
(a) Llama2-7B
RTE
WCC
COPA
> (V ([w (N,, u.
Cumu norm. sum
0.-
0.4-
62-
Balor
Epoch
End
Balon FT
Epazh
End
Baron FT
Epoch
Endiel
(b) Mistral- -B
RTE
WCC
COPA
Cumu nor.. sum
6.9
0.4
62-
grd..
Balow
#poch
End
Telon
Eposh
Fed
Rerow
Fposh
Eidel
(c) OPT-6.7B
Figure 9: Cumulative normalized sum of coordinate-wise 1 E (subfigure 9a), Mistral-7B (subfigure 9b), and OPT-6.7B (subfigure 9c)'s linear layers after applying sparsity masks of each method during FO-SGD full fine-tuning for 10 epochs. For given model and training checkpoint, we report the average value across all linear layers as a line in each subfigure. For each line, the colors represent the fraction of parameters (1e-2,1e-3,) 1,,11e4)))) and the line style represents the category. "task grad, dyn." refers to the sensitive parameters selected at the given timestep (x-axis), and "task grad, static" refers the sensitive parameters selected before fine-tuning. "C4 grad, static" refers to the sensitive parameters selected with gradients taken from causal language modeling on C4 datasets, and we keep it unchanged during fine-tuning.
== V f(w; (x;, (:))) of Llama2-7B
28

---

Published as a conference paper at ICLR 2025
RTE
WIC
COPA
22 1o...V (7((.. [x.. 36.))
Cumu. norm. sum
105% grad idhn;
weght forgest, auil mngite jemulless, statcc randert (aark))
2004 giad [ebeic]
C4 grad (static)
Before
Epoch
End FT
Before
Epoch
of
Before
Epoch
End
Llama2-7B
RTE
WIC
COPA
2 V (V:( (X.. 01.)))
Cumu. norm. sum
10-1
grad
weights Forgest, sistec
2054 grad (static)
weights ja malleett statia sandon (5a0))
C4 grod (Static)
10-7-
10
Before FT
Epoch
End of FT
Before FT
Epoch
En FT
Before FT
Epoch
End of FT
(b) Mistral-7B
RTE
WIC
COPA
107
Cumu norm. sum of
22 (w: [X... 30211
10-1
gradl godd [ebekc]
weight weighte stalic random (earic)
grat (ttaic)
10
Before FT
Epoch
End of FT
Betora FT
Epoch
End
Betore FT
Epoch
End of FT
(c) OPT-6.7B
Figure 10: Cumulative normalized sum of moordinate-wise 1 i=1 VV(w; (x, v.)] of Llama2-7B (subfigure 10a), Mistral-7B (subfigure 10b), and OPT-6.7B (subfigure 10c)'s linear layers after applying 99.9% sparsity masks of each method during FO-SGD full fine-tuning. The results of "C4 grad, static", "task grad, dyn.", and "task grad, static" are the same as their 1e-3 results in Figure 9.
Arc-C
HellaSwag
PIGA
AQuA
100
D0
135% g cyn
back grad. sbaid
OLmu nom. sumot
02-
00
C4 grad, statio
Belon
Step 100
FT
Beture
Step 190
Belore
ttep 199
of
Batore
Stap 100
Figure 11: Cumulative normalized sum of coordinate-wise after applying sparsity masks of each method during FO-Adam full FT for 1000 steps.
Vf(w: (x;, ys )2 of Mistral-7B
29

---

Published as a conference paper at ICLR 2025
E.3 OVERLAP RATIO OF TOP GRADIENT FEATURES
In addition to Table 6, we dive into the overlap ratio of the top gradient square entries from different data sources, shown in Figure 12.
Notice that the order here is mapped to the column/row rank for each subfigure in Figure 12.
1. C4. SensZOQ's ' default choice.
2.
We abbreviate it as "Math".
3. ArXiv
4. Wiki103
We abbreviate it as "Wiki""
5. Task gradient before FT: V E V Whefore FT; (Xi; W:)]]²
We abbreviate it as (...s.)
6. Task gradient after 10% fine-tuning steps: V n2 Σx We abbreviate it as V fwima))
V mid FTi (x,, vi)))
7. Task gradient at the end of FT: V Ei WWnd (x),)))1=
We abbreviate it as V I (Wend).
We consider 7 tasks (3 commonsense reasoning tasks Arc-C, HellaSwag, PIQA, math task AQuA, 3 SuperGLUE tasks RTE, WiC, COPA).
For a quick overview, we include Figure 13 as the average overlap ratio across tasks for the pre-training text. The second row in Figure 13 provides an empirical evidence for the "fixed gradient feature" " during FT as the top entries in f...s.)) (task grad before FT) resemble f.((widd) (task grad during FT) and VV((eei) (task grad at the end FT)
Empirical findings
The top gradient entries from all 4 pre-training text corpus overlap considerably with the task gradient.
C4 generally covers more top gradient entries than OpenWebMath, ArXiv, and Wiki103.
30

---

Published as a conference paper at ICLR 2025
ApG 02'3
ADG. 1S
Arc- The
6+-
CIM
Wa
832
G14
JUM
EXC
600
6777
600
ser
975
Pelideep
Le Top
Man-
2
LOK
2:80
WM-
E33
EU]
540
F90
PMU
6007
K00
X-I
1:4
CG7
0PM
Overlap ratio of top entries of gradient squares
MIA
Mah-
100
CO)
Wi--
2:03
230
030
1211
NQJA Trod
AGHA Top 3%
AQLA
Man
MEN
Amky
3311
CG3
2001
LSFT
241
241
ED0
EO1
ETD
22U
120
C/6
SWI
ATE Top 001%
RTE
Top 6.1%
RTE,
Top 1%
CGD
C/00
CPF
OPF
VIW
JMI
310
COX
COT
0154
VM0
2:43
0766
C11
8.10
2/61
WIG, Tap 031%
IWC Top IN
WCC Top 1%
END
330
COS
C118
6134
2/81
616
6311
2:48
CUS
0/10
0:07
S/N
T/Ku
EX1
EG2
CO3
100
2:50
0:18
5:00
2:20
COIN Thp 4.0%
Tep
9.1%
COFA,
C4-
830
CA
CS
200
C/Y
22V
920
Figure 12: Overlap ratio in top entries of gradient squares of Mistral-7B for C4, OpenWeettath ArXiv, Wikil03, and Vf(wa.), (W(mi)), and across 3 commonsense reasoning, I math, and SuperGLUE tasks.
31

---

Published as a conference paper at ICLR 2025
Top 0.01%
Top 0.1%
Top 1%
Aveeaae overtop ratio lop gradient square entries across 3 commonsenss math, and 3 SuperGLUE tacks
C4
0.384
0:196
0:172
0.304
0,209
0.100
0.361
0.329
0.353
0.202
0.177
0.316
0.302
0.200
0.326
0.248
0.239
AxX0
0.110
0.306
0.270
0.311
0.185
0.162
0.288
0.279
0.187
0.887
0.310
0.250
1.000
0.476
0.177
1,000
0.501
0.285
1.000
0.541
0.378
V//moul
0.476
1,000
0.419
0.501
1.000
0.427
0.641
1,000
0.484
0.419
V/iwe.)
Vi(Wess)
V/(Waad)
V.
V.(ww.))
V/Weea))
W.(Wwad)
Figure 13: Average overlap ratio of top gradient square entries across 7 tasks in Figure 12.
Top 0.01%
Top 0.1%
Top 1%
Overlap ratio top gradient square entries
across 4 dala mixtures
1,000
0.800
0.400
0.685
1,000
0.625
0.363
0.575
1,000
0.880
0.558
0.683
1.000
0.452
0.709
0.625
1.000
0.511
0.657
0.609
1.000
0.655
10
0.556
0.709
0.421
0.575
0.657
0.523
0.597
C4
Math
ArXiv
Wiki
C4
Math
A/CVv
Wid
C4
Math
ArXiv
Wil
Figure 14: Overlap ratio of top gradient square entries among C4, OpenWebMath, ArXiv, and Wiki103 Figure 12.
32

---

Published as a conference paper at ICLR 2025
F IMPLEMENTATION OF SPARSE OPERATIONS IN LINEAR LAYERS
Linear layers in LLMs often contribute most parameters (Kaplan et al., 2020). Since from Equation 7 we know
sparse
m
Wdense
O (1. m
Vsparse
dense
(13)
and since demse would have same shape (and the same computational intensities) as w, we need to improve wall-clock time efficiency of W X to improve the computational efficiency of linear layers after extracting the sparse parameters. In this case, we would have two different methods to implement the forward pass of linear layers (with induced sparse operation colored in red):
WX W dense X W sparse X
(14)
SparsAAdMMMDDessMMM((( (Wdnss X), W Waparse x) faster with token generation
(15)
(Weense W sparse
(16)
DenseMMMSSpareAAlee sparse; Wdense X)
faster with ZO training
(17)
FyTorch's E.linear uses Y W b.
X P for simplicity. 3 class SensitiveZOLinear (Zinear):
Here
w_sparse in PP16, CSR format w_dense # in PP16, dense tenser in FP16, dense tenscr
bias
8
def forward smsllssalllllll batched decodddgsdddddl X):
9
# dense matmul (linear forward) dense_result = dense_lineat (self. w_dense, X, self.bias) sparse addmm
return sparse_addmi (densc_rcsult,
x) self.blas
14
def
baarge_batchedd2_trraee
X)
$ add sparse weights to the dense weights
helf.w_dense self.w_sparse
dense natmul (linear forward) return dense linear (w, X, self.bias)
18
Listing 1: Example PyTorch-like code snippet that implements the forward calls with FP16 sparse and FP16 dense parameters.
Training (A6000)
Training (A100)
Inference (A6000)
Inference (A100)
Sec forward
1.0
SparsofcaMM
Sec/ token
0.2
0.01
0.01%
0.1%
1%
0.01%
0.1%
1%
0.01%
0.1%
1%
0.01%
0.1%
1%
Fraction to optimize
Fraction to optimize
Fraction to optimize
Fraction to optimize
Figure 15: Time of SparseAdd (Equation 17) versus SparseAddMM (Equation 15) in Llama2-7B ZO training forward & inference. In subfigure and 3, we use Nvidia RTX A6000 and Intel Xeno Gold 6342 CPUs, with PyTorch version 2.2, HuggingFace version 4.36, and CUDA 12.2. subfigure and 4, we use Nvidia A100-SXM4 (40 GB) and AMD EPYC 7543P 32-Core CPU with Py Torch version 2.1, HuggingFace version 4.38.2, and CUDA 12.2. We use Flash Attention 2 (Dao, 2023) for all 4 subfigures.
33

---

Published as a conference paper at ICLR 2025
The specific choice of employing Equation 15 or Equation 17 needs benchmarking, and we also provide general guideline based on the size of input vectors x and the resulting arithmetic intensity.
Size of input vectors x and arithmetic intensity. Wapase in Equation 15 would have compu- tational dependency over x. During large-batched ZO training, x would be large enough such that Equation 15 would induce large computational overhead, shown in subfigure 1 of Figure 15. In contrast, the computational costs of Equation 17 is independent of x and when x is large, we would expect Equation 17 is much faster than Equation 15. As an example, we sequence length of 512 and batch size 16 sampled from WikiText-2 dataset (Merity et al., 2016) as a representative forward costs for ZO training subfigures and 2 in Figure 15.
However, during autoregressive token generation, on each step we would only append single token the previously cached embeddings, and in this case x is small and computing Wdense + W Wsparse is generally not worthwhile, especially given that Wauure is already sparse. This is also illustrated in subfigure 3 and in Figure 15. However, we note that the specific implementation choice is hardware and task dependent and requires thorough benchmarking and will leave it future work.
We recommend using Equation 17 during batche ZO training and Equation 15 during small-batcher autoregressive token generation.
Integration with weight quantization method. Weight integer quantization algorithms can be categorized into 2 categories: uniform integer quantization method and non-uniform integer quan- tization method. For uniform integer quantization method, we use integer matrix multiplication with fused scaling to compute FP16 (Xi et al., 2023; Park et al., 2024). However, this is not compatible with our "SparseAdd" approach as we will violate the construint of mniformly-spaced int quantization bins by computing SparseAdd ( (Wensse) + Wsparse. In this case, we also have 3 different implementations:
Qx() efficiently without first dequantizing Q( ((Weense) to
Q(w)x Q( (ednese) (xx)==++)))))) xparse
(18)
SparseAddMM ScaledIntMMM( (Wden(()))))))))) Q(x))) w Wsparse, X uniform int Q
(19)
Q( x sparseX
(20)
SparseAddMM Dequantize(C (WWdensee) X, W spaasse)
similar Equation 15
(21)
(Q(wdense) sparse)
(22)
(SpareM (Weparesy Dequantize( (Weense))
similar to Equation
(23)
without dequantizing Q(Wdense) to FP16. Notice that both Equation 21 and Equation 23 would still dequantize Q @(Weense) first and the choice of implementation would follow into discussion of input vector size in last paragraph. We leave practical implementation and thorough benchmarking into future work.
Equation 19 would compute (CaledIntMMM(((Waaaa)
class Sens0OQ (Lii l
x_sparse
in FP16, CSR format in Int4, dense tensor in FP16, dense tensor
3
l.cense__qunntonn
4 bias
5
def dequantize (self, x_quantized)
7
8
return v_16_bit
9
def ceaordd_calee_inee matmul (self, X):
dense int4 matmul with fused scaling
dense_result scaled htw.://mtaiai..............aaaa. X) sparse addmm return
13
self self.w_sparse, X) self.bias
def forward small __t.acched_eeccddigggge x)
34

---

Published as a conference paper at ICLR 2025
dequantize the dense weights to FP16 __dens_____bit delf.dequantize dense FP16 linear forward
19
dcnse_result dense_linear (v_dense_16_bit, x, self.bias)
# sparse addmm
sparse_addmm (dense_result, self.x_sparse, X) self.bias
24
def
X)
# dequantize the dense weights to FP16
dende 16 bit self self. dequantizeeeeeeeeeeeeeeeee (self___deeeeee few__denee_quaee cense_quantized
add sparse weights w_dense_l6_bie self.bias) bias)
the dense weights
w
+ self.w_sparse
dense FP16 linear forward
return dense linear (w, X, self
Listing 2: Example Py Torch-llie code snippet that implements the forward calls with 16-bit sparse and quantized dense parameters.
We recommend using Equation 19 when integer matmul to compute (Wense (x() and in other cases, using Equation 21 or Equation 23 follows our previous recommendation.
35

---

Published as a conference paper at ICLR 2025
G SUPPLEMENTARY EXPERIMENT DETAILS
G.1 HYPERPARAMETERS IN EXPERIMENTS
For all ZO experiments, we use 20,000 training steps with ZO-SGD optimizer (Definition 2). We evaluate on the validation or test set at the end of the training. Usually, the training/validation set will be sampled from the original training dataset with size 1000/500 respectively and the evaluation set is of size min( 1000, joriginal validation test set).. However, for CB and COPA, we use 100 for the validation set size. We use 2051 1/200 for Arc-E, 919/200 for Are-C, 39705/200 for HellaSwag, for PIQA, 4757/200 for OBQA, 33210/200 for SIQA, 20000/200 for MMLU (training is on an auxiliary training set), and 97267/200 for AQuA.
For all ZO experiments in Table 7, 8, and 9, MMLU experiment in Table we use batch size of 8 for methods.
a batch size of 16 except for the Mistral-7B on
Table 7: The chosen hyperparameters (20 perturbation constant, learning rates, and # ICL examples) for experiments in Table and 4. We repeat each learning rate for 3 random trials and report the average and standard deviation in Table and 4.
(a) Llama2-7B
Methods
SST-2
RTE
CB
BoolQ
WSC
WiC
COPA
WinoG
Q, 2O, SGD
SensZOQ (c =1e33)
5e-7
le-6
le-6
le-6
5e-7
le-6
le-6
2e-6
LoRA =le-3)
le-5
2e-5
le-5
26-5
le-5
2c-5
le-5
Se-5
Z0. SGD
Full FT (E =le-3)
5c-7
5c-7
5c-7
5o-7
2c-7
5e-7
5c-7
1e-7
ICL (#examples)
16
16
16
8
16
8
16
FO, SGD
Full Full
1c-4
1c-4
20-5
5e-5
5c-5
5e-5
lc-4
Adam
le-5
Se-6
1e-5
5e-6
2e-6
2e-5
(b) Mistral-7B
Q 20, SGD
SensZOQ (5 =lc4)
2c-8
5e-8
2c-8
2c-8
1c-8
2c-8
1c-7
LoRA le-4)
2e-6
2e-6
2e-6
2e-6
2e-6
le-5
70, SGD
Full FT (f =1e-4)
20-8
2e-8
le-8
le-8
le-8
lc-8
20-8
5e-8
ICT. (Hexamples)
4
16
8
FO, SGD
Full FT Full FT
2e-6
5e-6
le-5
2c-6
le-5
2e-6
le-5
5e-6
FO, Adam
2e-6
20-6
2c-6
1c-6
2e-6
2c-6
2c-6
2e-6
(c) OPT-667B
Q. ZO
SensZOQ (c (1-3)
2c-7
Se-7
5c-7
Se-7
2c-7
5e-7
2c-7
1c-6
LoRA (E =lc3)
lc-5
20-5
lc-5
20-5
le-5
20-5
2e-5
5c-5
Z0, SGD
(6 lc-3)
2c-7
20-7
2c-7
20-7
20-7
2+-7
5c-7
5c-7
ICL (#examples)
FO, SGD
Full FT Full
5c-5
2c-5
lc-4
20-5
2o-5
lc44
5c-5
FO, Adam
5e-6
5c-6
26-5
le-5
5c-6
le-5
2c-6
2e-6
(d) Llama2-133
Q 70, SGD]
(e (1-33)
le-6
le-6
le-6
le-6
le-7
1c-6
26-6
5e-6
70, SGD
Full FT (c (1e3)
Se-7
Se-7
Se-7
Se-7
le-7
Se-7
Se-7
2e-6
ICL (#xxmmplcs
16
8
16
8
8
8
16
16
(e) OPT-13B
Q. 20. SGD|
SensZOQ (e (1c-))
2c-7
5c-7
2c-7
2c-7
2c-7
5c-7
2c-7
1c-6
20,SGD
Full =ee33)
2e-7
2e-7
2e-7
Sens. (C4, static) =le-3)
2--7
5e-7
2e-7
20-7
20-7
Se-7
20-7
1c-6
ICL (fixxamples)
16
16
16
16
16
16
16
16
For the ZO-LoRA baseline in Table always add it to all linear layers with 8 and 16.
36

---

Published as a conference paper at ICLR 2025
Table 8: The chosen learning rates for experiments in Table and 6. We repeat each learning rate for 3 random trials and report the average and standard deviation. Notice that in Table 6, the learning rates for RTE, WiC, COPA tasks for SensZOQ (Open SpgeWWebMath, ArXiv, and Wiki103) are the same as SensZOQ (C4), which are reported in Table 7.
Methods
Arc-E
Are-C
HS
OBQA
PIQA
SIQA
AQuA
MMLU
Q ZO, SGD
C4 (1c-4)
5c-9
le-8
lc-7
2c-8
2c-7
2c-8
5c-8
1c-8
ArXiv (c =lc-4)
5e-9
le-8
le-7
2c-8
2c-7
2c-8
5e-8
1c-8
OpenWebMath 1)-4)
5e-9
le-8
le-7
2e-8
2e-8
5e-8
1e-8
Wikil03 (c =le44)
5e-9
le-8
le-7
2e-8
2e-7
2e-8
5e-8
1e-8
z0, SGD
Full FT (c =le-4)
5e-9
5e-9
5e-8
5e-9
Se-8
1e-8
1e-8
5e-9
ICL (#examples)
16
16
16
8
16
16
16
16
Table 9: The chosen learning rates for experiments in Figure 6a and Figure 6b. We repeat each learning rate for random trials and report the average to draw line in Figure 6a and Figure 6b, and we use Llama2-7B for experiments. For each subtable, we include the fraction to optimize on its header and report the chosen learning rate each cell.
(a) RTE
Methods
le-5
le-4
lc-3
le-2
lc-1
Sensitive (C4, static) (c =1e-3)
1e-5
le-6
1c-6
le-6
lc-6
Sensitive (task-specific, static) -e-33)
le-5
le-6
le-6
le-6
le-6
Sensitive (task-specific, dynamic) =le-3)
5e-6
le-6
le-6
le-6
5e-7
Random (static) (e 1e-33)
2c-2
5e-3
5c-4
5e-5
5e-5
Random (dynamic) (c =le-3)
2e-2
Se-3
2e-4
5e-5
5e-6
Weights with largest magnitude (static) =le-3)
2e-3
Se-5
le-S
Weights with smallest magnitude (static) (E = lc-3)
20-3
5e-4
1c-4
le-5
1c-6
smallest GraSP scores (static) (e = -le-3)
20-5
5e-6
2e-6
2e-6
le-6
(b) WiC
Methods
1e-5
le-4
le-3
le-2
1e-1
Sensitive (C4, static) (e =1c-3)
le-5
2c-6
1e-6
le-6
1c-6
Sensitive (task mask-specificc static) =le-3)
le-5
le-6
le-6
le-6
Sensitive (task-specific, dynamic) (6 =1e-3)
1e-5
2c-6
1c-6
le-6
1c-6
Random (static) (E =lc-3)
2c-2
5e-3
5e-4
5e-5
5c-6
Random (dynamic) (e =le-3)
2e-2
5e-5
5e-6
Weights with largest magnitude (static) (e =1c-3)
1e-3
5e-4
2c-4
lc-4
2c-5
Weights with smallest magnitude (static) = 1c-3)
1e-3
5e-4
lc-4
le-5
2c-6
smallest GraSP scores (static) (e 1e-3)
2e-5
le-5
5e-6
2e-6
2e-6
(c) COPA
Methods
le-5
le-4
lc-3
lc-2
lc-1
Sensitive (C4, static) lee33)
5e-6
Se-7
le-6
le-6
Sensitive (task-specific, static) (c 1e-3)
le-5
le-6
2e-6
le-6
5e-7
Sensitive (task-specific, dynamic) (f =1e-3)
5e-6
1c-6
1c-6
lc-6
5c-7
Random (static) (c =le-3)
Se-5
5e-6
Random (dynamic) (c =le-3)
2e-3
le-3
2e-4
2e-5
2e-6
Weights with largest magnitude (static) =1c-3)
1e-3
5e-4
5e-4
lc-4
lc-5
Weights with smallest magnitude (static) = le-3)
2e-3
Se-4
2e-5
2e-6
2e-6
smallest GraSP scores (static) (c =le-3)
5e-6
Se-6
le-6
2e-6
le-6
37

---

Published as a conference paper at ICLR 2025
For the smaller FO-Adam experiment in Table 2, report the used learning rates in Table 10. We use a batch size of 8 and train for 1000 steps. We use the Adam optimizer with linear learning rate decay (without warmup) to 0 and weight decay. We evaluate the model's performance at the end of 1000-step training.
Table 10:
The chosen learning rates for experiments in OPT-1.3B FO FT in Table 2. We repeat each learning rate for 3 random trials and report the average and standard deviation in Table 2.
Methods
SST-2
RTE
CB
BoolQ
WSC
WiC
COPA
WinoG
FO-Adam
Full FT
le-5
20-5
1c-5
le-5
2c-5
le-5
1c-5
20-5
38

---

Published as a conference paper at ICLR 2025
G.2 TPSK-SPECIFIC PROMPTS EXPERIMENTS
We describe our task templates in Table 11 and 12.
Table 11: Task templates for all experiments (1/2). On the left column we include the task name and the model name, and on the right column we describe the exact prompt with answer candidates.
Task
Prompts
SST-2
### Sentence: <text> Sentiment: negative/positive
(l.ama2-7B, Llama2-13B)
SST-2
<text> It was terrible/great
(Mistral-7B. OPT-6.7B, OPT-13B)
RTE
Suppose "<premise>" Can we infer that "chypothesis>"" Yes No? Yes/No
(Llama2-7B, Llama2-13B)
RTB
<premise>
(Mistral-7B, OPT-6.7B, OPT-13B)
Does this mean that "chypothesis>" true? Yes or No? Yes/No
CB
Suppose <premise> Can we infer that "chypothesis>"? Yes, No, Mayhe?
(Llama2-7B, Mistral-7B, OPT-6.7B,
Llama2-13B, OPT-13B)
Yes/No/Maybc
BoolQ
<passage> <question>? Yes/No
(Llama2-7B, Llama2-13B)
BoolQ
<passage> <question>?
(Mistral-7B, OPT-6.7B, OPT-13B)
Yes/No
WSC
<lexD>
(Llama2-7BB 2B, Mistral-7B, OPT-6.7B,
In the previous sentence, does the pronoun "<span2>" refer
Llama2-13B, OPT-13B)
to <span>>? Yes or No?
Yes/No
WiC
Does the word "<word>" have the meaning in these two sentences? Yes, No?
(Llama2-7B, Mistral-7B, OPT-6.7B, Llama2-13B, OPT-13B)
<sent<>
<sent2>
Yes/No
CОРA
<premise> so/because <candidate>
(Llama2-7B, Mistral-7B, OPT-6.7B, 1l3ma2-13BB OPT-13B)
WinoGrande
<context> <option>
(Llama2-788 7B, Mistral-7B, OPT-6.7B, Llama2-13B, OPT-13B)
39

---

Published as a conference paper at ICLR 2025
Table 12: Task templates for all experiments (2/2).
Are-E, Are-C, OBQA, MMLU (Mistral-7B)
Question: <question>
(A) <choice A>
(B) <choice B>
(C) <choice C>
(D) <choice D>
Answer: A/B/C/D
HS (HellaSwag)
<context> <candidate>
(Mistral-7B)
PIQA
Goal: <gnal>?
(Mistral-7B)
Answer: <candidate>
SIQA
Question: <question>
(Mistral-7B)
(A) <choice A>
BB) <choce B>
(C) <choice C>
Answer: A/B/C
AQuA
Question: <question>
(Mistral-7B)
(A) <choice A>
B <choice B>
CC) <choice C>
(D) <choice D>
(E) <choice E>
Answer: A/B/C/D/E
40

---

Published as a conference paper at ICLR 2025
G.3 DN-DEVICE MEMORY CONSTRAINTS
As illustrated in Table 13, a wide range of mobile or edge devices impose a memory constraint of GB, which the our target when we develop SensZOQ in Section 3.2.
Table 13: Device memory of some mobile devices or consumer consumer-graded GPUs.
Devices
Memory
NVidia GeForce GTX 1080 Ti
11 GB
NVidia GeForce RTX 3060 Ti
GB
NVidia Jetson TX2
8 GB
OPPO Find X7 Ultra (Li al., 2024a)
12 GB
Samsung Galaxy S10 with Mali-GT, GPU (Gim & Ko, 2022)
GB
G.4 HARDWARE, PLATFORM, LIBRARIES, AND OTHER DETAILS FOR FINE-TUNING AND BENCHMARKING
Figure 15 (subfigure 1 and 3) is trained and evaluated on single GPU node with 1 NVidia RTX A6000 GPU and Intel Xeon Gold 6342 CPU, with PyTorch version 2.2, HuggingFace Transformer version 4.36, and CUDA 12.2. In subfigure 2 and in Figure 15, and AMD EPYC 7543P 32-Core CPU with Poroch version 2.1, HuggingFace version 4.38.2, and CUDA 12.2. We Flash Attention (Dao, 2023) in HuggingFace Transformers library throughout our experiments, and the base model for ZO full fine-tuning and benchmarking is always Llama2-7B with 6 datatype (torch.float1)) We also use the Floatl6 datatype (torch.float.6 for all of our sparse parameters (sensitive sparse, random subsets, etc.) in ZO fine-tuning experiments. In our preliminary experiments we found BrainFloat16 with 7 bits (ULP=2 7 == of mantissa cannot make ZO FT's training loss decreases, but ZO FT still works with Float16 that has 10 bits of mantissa (ULP=2-10 s=e-3)). We leave the investigation between ZO FT and the number of precision bits to future research. We still use 1 for FO-SGD Full FT while using BrainFloat 16 for FO-Adam Full FT.
NVidia A100-SXM4 (40 GB)
In Figure 15, use sequence length of 512 and batch size 16 sampled from WikiText-2 dataset (Mer- ity et al., 2016) as a representative computational intensity for ZO training.
41


Reviews:
[{"review_id": "dksC4km1PE", "reviewer": "Reviewer_eqp9", "summary": "This paper proposes a transferable static sparse ZO LLM fine-tuning strategy. It observes an extreme sparsity pattern in LLM parameters: a subset, determined by selecting the top k magnitude entries from the diagonal of empirical Fisher information matrix, is effective for ZO fine-tuning. Moreover, it finds this sparsity pattern can be obtained through LLM’s pre-training process and transferred to various downstream tasks without modification. Building on these insights,  it proposes SensZOQ, an on-device LLM personalization workflow\nvia integrating Sensitive ZO optimization with Quantization to further improve the memory efficiency of ZO fine-tuning.", "questions": "see the weakness. \n\nIt is better to highlight the contributions and discuss the fairness of memory comparison. It is better to have some results on larger models such as 13B and 30B to demonstrate the general performance on larger models. It may be better to provide some high level discussions for why the inference can be faster.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "It identifies that only an extremely small portion (0.1%) of LLM parameters should be updated during ZO LLM fine-tuning. Moreover, it observes that this sparsity pattern can be derived in LLM pre-training process and transferred across different downstream tasks while still maintaining good ZO performance without any modification.\n\nIt conducts extensive experiments across various LLMs and demonstrate that the method achieves competitive performance across various downstream tasks.", "weaknesses": "The novelty may be limited. The proposed method mainly use masks to select a small set of parameters and update these parameters with ZO method while quantizing the rest parameters. The quantization and ZO methods are very typical, following previous works. The mask construction by selecting the top k magnitude entries from the diagonal of empirical Fisher information matrix has been proposed in (Sung et al., 2021, Training Neural Networks with Fixed Sparse Masks).  The idea of updating the weights with fixed sparse masks for language models has been investigated in (Sung et al., 2021) and SparseMeZO  (Liu et al., 2024a). The technical components are not novel and their combination seems to be straightforward. The technical contribution may be limited. \n\nThe comparison on memory usage may not be fair. The method quantize the model and it compares the memory of quantized model with other unquantized models from other methods. We are not sure whether simply quantizing models can lead to such memory reduction. Furthermore, during the mask construction, it needs to use backpropagation to obtain the gradients of all parameters. This part costs massive memory. But in the memory comparison, it does not count the memory cost for mask construction and only include the memory in ZO finetuning after the mask is obtained. And it also not count the memory cost for quantization part. Although the authors use a on-device flow to count on-device memory,  it may not be fair to compare part of the memory usage of the proposed method with the full memory usage of other methods.  \n\n\nIt only experiments with 7B  or smaller models. It is better to have some results on larger models such as 13B and 30B to demonstrate the general performance on larger models. It claims to be efficient with less memory usage and faster training speed. It should be easy to apply to larger models. \n\nIt only experiments with one single finetuning dataset C4. It may be better to experiment with multiple finetuning datasets to demonstrate the general performance. \n\nIt claims to be faster in inference with higher Token generation speed. But after finetuning, it is still a dense model, and we are not sure how the faster token generation is achieved. Though there are some discussions in appendix for the sparse operations,  there do not seem to be much details and we only see some function calls without detailed implementation for the functions to understand why it is faster.  It may be due to the quantization. But quantization in the work just follows SqueezeLLM (Kim et al., 2024). It may be better to provide some high level discussions for why the inference can be faster.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "up2G6gXBMq", "reviewer": "Reviewer_KZSn", "summary": "This paper proposes SensZOQ, a workflow framework sparse zeroth-order optimization with quantization for on-device fine-tuning of LLM. It identifies a small, static subset of \"sensitive\" parameters in pre-training, transfers them to downstream tasks, and quantizes the rest to reduce memory demands. Experiments on 7B-level LLMs show SensZOQ outperforms other methods in memory-constrained settings, achieving better performance than in-context learning and full ZO Full-FT.", "questions": "What is the rank meaning in Table 1, 2? Average all ranks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The work enables fine-tuning of Llama2-7B model on <8GiB GPU memory by using only 0.1% static sensitive parameters and quantizing the rest, meeting memory constraints of edge/mobile devices and enabling more customization scenarios at the edge.\n\nSensZOQ outperforms ICL and ZOFull-FT, achieving competitive results across various downstream tasks and models like SST-2, RTE, etc.\n\nThe work also conducts details theory analysis and overview of related ZO methods (Sec 2,3) and reveals that 0.1% contributes about 50% gradient norm scales.", "weaknesses": "The paper is well-presented with a clear structure. My main concerns revolve around the experiment setup and comparisons.\n\nHow does the fine-tuning fit within 8GB of memory? Does this include the memory for the model states themselves? For instance, the authors claim that LLaMA-7B fits within 8GB memory constraints. However, even when quantized to 4 bits, storing the model alone requires 3.5GB, not accounting for other intermediate buffers needed during training.\n\nWhat is the transferability of the sparse mask (e.g., sensitive sparse masks), and how is it generated? Is this sparse mask generally applicable across different downstream tasks, or does SensZOQ require a new one for each task? Lines 236-240 reference a fixed work, but the authors don't provide an ablation study to support this claim.\n\nThe convergence curve or training curve is missing. As mentioned in the introduction and related work, zero-order (ZO) methods suffer from slow convergence. However, the authors only discuss the theoretical convergence rate without providing an empirical curve. Additionally, in all experiment tables, the results of full fine-tuning (Adam, BF16) are missing, and no actual memory usage or throughput data is provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yrRsApKsfj", "reviewer": "Reviewer_Agsv", "summary": "This paper proposes a memory-efficient zeroth-order (ZO) optimization approach for fine-tuning large language models (LLMs) in memory-limited environments, such as mobile devices. By tuning only a sparse subset of \"sensitive\" parameters derived from pre-training, the proposed method, SensZOQ, enables efficient on-device fine-tuning without backpropagation. SensZOQ demonstrates competitive performance compared to in-context learning and full-model ZO fine-tuning, even on GPUs with less than 8 GiB of memory. Key contributions include the identification of transferable sparse parameters and the integration of static sparsity for enhanced efficiency. While SensZOQ addresses memory limitations effectively, the ZO optimization’s slower convergence and static sparsity’s limited adaptability across tasks are noted as areas for potential improvement.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Intuitive and Effective Approach Supported by Theory: The paper presents an intuitive yet impactful approach by focusing on a sparse subset of sensitive parameters for zeroth-order (ZO) optimization. This idea is both straightforward and theoretically sound, and the authors support it well with theoretical analysis that adds credibility to their method.\n\nStrong Experimental Validation: Through extensive experiments, the paper demonstrates the superiority of its proposed method. SensZOQ is shown to perform effectively across multiple tasks, validating the approach's potential for on-device fine-tuning in memory-constrained environments.", "weaknesses": "Sensitivity Assessment Limited to C4 Dataset: The authors measure sensitivity based solely on the C4 dataset, which predominantly covers natural language tasks. This raises concerns about the applicability of their sensitivity-based sparse tuning approach to tasks with different characteristics, such as code generation. It remains uncertain whether the proposed method would generalize well to tasks with substantially different data distributions and requirements.\n\nEvaluation on Small Model Sizes Only (7B): The experiments focus on models with a maximum of 7 billion parameters, which is relatively small compared to leading models. This raises questions about whether the proposed approach would remain effective on larger models where more precise fine-tuning might be required. It would be valuable to see an assessment of SensZOQ’s applicability and performance on larger, state-of-the-art models where scaling issues might become more prominent.\n\nLack of Analysis on Slow Convergence Issue: The paper notes that ZO optimization can suffer from slow convergence, which is a well-documented drawback of ZO methods. However, it lacks an in-depth analysis of this issue. A discussion of potential mitigations or how this slow convergence might impact the method’s performance, especially in time-sensitive applications, would have added valuable context.\n\nUnclear Insights from Theoretical Convergence Rate (Section 2.3): While the authors present a theoretical convergence rate, the practical implications or lessons from this analysis are unclear. It would benefit the reader to understand what actionable insights can be drawn from this theoretical result. Without this clarity, the theoretical component feels somewhat disconnected from the rest of the work.\n\nSimilarity to Existing Research on Activation Outliers: Although the paper asserts the effectiveness of using a static mask, this approach bears resemblance to previous work on activation outliers. If the sensitivity mask is closely related to known methods for identifying important model parameters, such as activation outliers, the novelty of this contribution may be limited. Further discussion comparing SensZOQ with prior work in this area could clarify its unique contributions.\n\nInsufficient Details on Quantization Methodology: The paper does not provide adequate details regarding the specific quantization technique used. Given the potential impact of quantization on model performance, a clearer explanation is necessary to assess the reproducibility of the experiments. Without this information, readers may find it challenging to replicate the results, which is a significant drawback for scientific validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xgcy58SBVz", "reviewer": "Reviewer_dmR6", "summary": "The paper proposes a sparsity-aware ZO fine-tuning technique and the authors point out the method can be useful in on-device fine-tuning.", "questions": "I will be much more convinced if the authors could provide answers to the following questions (from high to low priority):\n1. What is the justification of applying SensZOQ to LLaMA2-7B vs. directly fine-tuning decoder-encoder transformers like RoBERTa? It seems those smaller transformers consume less memory (great for edge devices), yet achieve better performance.\n2. Can you provide some experiment results to show how well SensZOQ performs on harder common-sense reasoning, math/code reasoning and domain-knowledge benchmarks like (PIQA, gsm8k/humaneval, MMLU etc.)? I think those are the benchmarks that LLMs can excel in comparison with those smaller models. Even if SensZOQ fine-tuning doesn't perform well on those benchmarks and there is a big gap between SensZOQ and full-parameter fine-tuning, I think it would be fruitful to inform the community about its limitations. \n3. Is the code going to be open-sourced in the future?\n4. Since the loss landscapes of LLMs can be highly non-convex and complex, would assumption 2, 3 (page 17) always hold? If there are literature studies that discuss this issue, could you provide the pointers?\n5. Could the authors elaborate on the concept of \"fixed (gradient) features\" and explain why it holds? Additionally, more evidence supporting the validity of transferable static sparsity across other tasks would be helpful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "I believe the paper has the following strength:\n\n1. ZO can help reduce memory consumption due to gradient and optimizer. When used in combination with quantization, dense weights can also be compressed, further reducing memory consumption.\n\n2. Supporting the \"sparse sensitive parameters\" assumption with cumulative normalized sum of gradient norm square is intuitive, and apply it sparse ZO fine-tuning (SensZOQ) is elegant. \n\n3. I appreciate detailed formulation on convergence rate of SensZOQ, along with proof for Eq.5 and Eq. 6. \n\n4. Solid ablation studies on the efficacy of SensZOQ versus other weight importance metrics. Detailed analysis of gradient sparsity and sparse sensitive parameter dynamics in the appendix. Implementation and experiment details for reproducibility.", "weaknesses": "I have two major concerns about the paper. \n\n1. The first is about its experiment section and its practicality. On device personalization is an important topic. I understand the authors might want to focus on edge applications and on some easier NLP benchmarks. However, all benchmarks used are can be easily handled with decoder-encoder transformers, yet at a much smaller scale. For example, RoBERTa (see page 8 in their paper) can outperform SensZOQ applied to LLMs (numbers shown in in Table 1 and Figure 5). RoBERTa-base only consists of 125M parameters, with RoBERTa-large at 355M scale. On the other hand, even with SOTA W3/W4 quantization, LLaMA2-7B takes > 3GiB of memory, just for forward pass. And this number is significantly higher than using smaller decoder-encoder transformers like RoBERTa, T5 etc. In this case, for , why should I use SensZOQ fine-tuning + LLMs instead of regular fine-tuning + smaller decoder-encoder transformer models? \n\n2. The second concern is about the assumption: the transferable static sparsity argument is a strong argument. In Figure 3 and Figure 5, the authors have shown empirical evidences that the hypothesis holds for some tasks including RTE, WiC, COPA. However, how well this can be generalized to some other harder tasks (math, common sense reasoning, code, MMLU etc.)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yiY9S3QTdH", "reviewer": "Reviewer_upxM", "summary": "This paper introduces SensZOQ, a method that combines static sparse zeroth-order (ZO) fine-tuning with quantization to address the limitations of standard ZO. SensZOQ focuses on fine-tuning a small subset (0.1%) of \"sensitive\" parameters, while keeping the remaining parameters quantized to reduce memory usage and maintain competitive performance on various downstream tasks. This framework enables efficient fine-tuning of an LLaMA2-7B model on a GPU with less than 8GB of memory, outperforming traditional full-model ZO in both memory efficiency and learning performance.", "questions": "For **Table 1**:\n1. Although the proposed method surpasses the listed methods, it would be helpful to also report the fully fine-tuned FO results as a baseline.\n2. Since this paper emphasizes training efficiency in both memory and wall-clock time, providing memory and time comparisons would help to further illustrate its effectiveness.\n3. Could the authors explain why the proposed method achieves better performance compared to ZO Full FT (the fourth row), despite incorporating both sparse funetuning and quantization?\n4. The introduction section discusses various advanced ZQ methods; comparing the proposed method with these would better highlight and validate its effectiveness.\n\nFor **Table 2**:\n1. Similarly, it would be helpful to include both memory and runtime comparisons to better demonstrate the effectiveness of the proposed method.\n2. Could the authors explain why the proposed method achieves better performance than 16-bit ZO-SGD (the second row), despite using both sparsity and 4-bit quantization?\n3. While this paper focuses on training efficiency and notes that the memory consumption of 16-bit FO-Adam on OPT-1.3B is higher than that of 4-bit ZO-SGD on OPT-6.7B, I am curious about their memory and runtime consumption from an inference perspective.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "**Memory Efficiency**: The method significantly reduces memory usage, allowing LLM fine-tuning on low-memory devices (less than 8GB).\n\n**Quantization Integration**: Incorporating quantization further enhances memory efficiency without substantial performance loss.\n\n**Resource-Constrained Applicability**: Designed for mobile and low-power devices, making it suitable for on-device personalization.", "weaknesses": "Some key experiment comparisons and discussions are missing; please refer to the questions for details.\n\nSome abbreviations are not defined before their first use, making the paper somewhat challenging to follow. For example:\n1. “FO” is not defined before it first appears.\n2. Baseline methods listed on the x-axis of Fig. 1 are not explained in either the figure caption or the text referencing the figure, which hinders understanding of this key figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)