Second-Order Fine-Tuning without Pain for LLMs: A Hessian Informed Zeroth-Order Optimizer
Authors: Yanjun Zhao
Sizhe Dang
Haishan Ye
Guang Dai
Yi Qian
Ivor Tsang
Venue: ICLR 2025 Poster (2025)
Keywords: LLM; deep learning; zeroth order optimizer
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose HiZOO to leverage the diagonal Hessian to enhance ZOO for fine-tuning LLMs.

Abstract: Fine-tuning large language models (LLMs) is necessary for specific downstream tasks, but classic first-order optimizer entails prohibitive GPU memory because of the back propagation. Recent works such as MeZO have turned to zeroth-order optimizers for fine-tuning, which reduce substantial memory by using two forward passes. However, heterogeneous curvatures across different parameter dimensions in LLMs often cause model convergence instability or even failure. In this work, we propose HiZOO, a diagonal Hessian informed Zeroth-Order Optimizer , which is the first work to leverage the diagonal Hessian to enhance ZOO for fine-tuning LLMs. We provide theoretical proof for HiZOO and visualize the optimization trajectories on test functions to illustrate how it improves convergence in handling heterogeneous curvatures. Extensive experiments on various models (RoBERTa, OPT, Phi-2 and LLama3, with 350M$\sim$66B parameters) indicate that HiZOO significantly reduces training steps and enhances model accuracy, while keeping the memory advantage of ZOO. For example, on SST2 task HiZOO achieves $8\times$ speedup and better accuracy over MeZO across different models. We also propose HiZOO-L, which reduces the Hessian memory cost to 10\% of the MeZO, while maintaining almost same performance. Compared with ZO-Adam, HiZOO-L achieves a 4.3\% improvement, just using 50\% of the GPU memory. Code is available at https://anonymous.4open.science/r/HiZOO-27F8.

Published as a conference paper at ICLR 2025
SECOND-ORDER FINE-TUNING WITHOUT PAIN FOR LLMs: A HESSIAN INFORMED ZEROTH-ORDER OPTIMIZER
Yanjun Zhao Sizhe Dangh Haishan Yc1.21, Guang Dai Yi Qian Ivor W.Tsang Xi'an Jiaotong University, China, 2SGTT Al Lab, State Grid Corporration of China, China CCAR and IHPC, Agency for Science, Technology and Research, Singapore 'College of Computing and Data Science, Nanyang Technological University, Singapore [yanjun, zhao, darknight1118/ yehaishan@x jtu.tetue..o edu. en, (tt.://sstu..uuuu..... guang. dai@gma@aa @gmall.............mmmm...m....e com,
edu.
ABSTRACT
Fine-tuning large language models (LLMs) is necessary for specific downstream tasks, but the classic adaptive first-order optimizer entails prohibitive GPU memory because of backpropagation. Recent works such as MeZO have turned to zeroth- order optimizers for fine-tuning, which reduce substantial memory by using just two forward passes. However, heterogeneous curvatures across different parameter dimensions in LLMs often cause convergence instability or even failure. In this work, we propose HiZOO, diagonal Hessian informed Zeroth-Orde Ordder Optimizer, which is the first to leverage the diagonal Hessian to enhance 700 for fine-tuning LLMs. We provide the theoretical proof for HiZOO and visualize the optimiza- tion trajectories on the test functions. Extensive experiments on various models (RoBERTa, OPT, Phi-2, and LLama3, with 350M~66B parameters) indicate that HizOO significantly reduces the number of training steps and improves model accuracy. For example, on the SST2 task, HiZOO achieves an 8x speed-up and better accuracy. Even when scaled to 66B-model, HiZOO outperforms MeZO with up to 5.1 % absolute improvement. We also propose HiZOO-L, which re- duces the Hessian memory cost 10 % of the MeZO, while maintaining almost same performance. Compared with ZO-Adam, HiZOO-L achieves a 4.3 % ab- solute improvement, just using 50% of the GPU memory. Code is available
Adaun
Memory
Mez-
Converge slowly
intensive X
updated parameters
HeDo HIDCO
projected
CULKE
forward passes
forward pass
Hiz-00
Memory efficient Converge quickly
fist
Spc00d onder
grdd
updato
Iuackosed pass
forward
C.C. C.
Hessin
Figure 1: (Left) Comparison of HiZOO, MeZO and Adam. (Right) Heterogeneous curvatures example. HiZOO updates along the direction with greater curvature (X) and converges more quickly than MeZO. The corresponding loss curves are shown in Section 3.5.
Equal contribution. This work completed during the internship at SGIT Lab, State Grid Corporation of China. Corresponding author.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Fine-tuning pre-trained LLMs for specific tasks has gained significant attention recently. As the number of model parameters increases, full parameter fine-tuning (FT) becomes markedly memory- intensive. To alleviate GPU memory limitations, parameter-effiieet fine-tuning (PEFT) methods (Hu et al., 2022; Li & Liang, 2021; Dettmers et al., 2023; Zhao et al., 2024b; Pan et al., 2024) have been developed, which only fine-tune a small number of (extra) model parameters. As a result, they significantly reduce the computational and storage cost, while achieving performance comparable to a fully fine-tuned model.
Adaptive first-order optimizers such as Adam (Kingma & Ba, 2015) and Adam W (Loshchilov & Hutter, 2019) are widely used fine-tune LLMs. However, using these optimizers still leads to substantial memory consumption, primarily due to the inherent backpropagation process to calculate the gradient. To address these limitations, MeZ0 (Malladi et al., 2023) proposed to utilize a zeroth- order optimizer (ZOO) to estimate the gradient with just two forward passes per step, no need for backpropagation anymore. This achieves numerous memory reductions and makes it accessible to train and store LLMs on consumer hardware.
However, the parameters of LLMs often exhibit heterogeneous curvatures across different dimensions, as documented in recent studies (Sagun et al., 2017; Ghorbani et al., 2019; Zhang et al., 2020). This significant difference of second derivative makes the MeZ0 converge towards saddle point, slowing down the convergence speed, shown in Figure/ 1 (right). Since the incorporation of Hessian to measure the curvature properties of the loss landscape, second-ord -oderr methods (Liu & Li, 2023; Yao al., 2021; Anil al., 2021) can solve this suboptimal behavior. Unfortunately, in the context of zeroth-order optimization, one cannot directly compute the Hessian atop first-order derivatives.
RoBERTa HargRlbbarge//F] on SST2
OPT-138 on SST2
Ulama3(88) SST2
1.0
Me70: 90.5%
MezO: 91.4%
Me20: 92.2%
10
HiZ00: 93.2%
0.9
HiZ00: 92.1%
10
HiZ00: 93.5%
HiZ0O-L: 93.2%
HIZ00-L: 92.1%
0.3)
HIZOO-L: 94.3%
30-1
0.7
Lod5
0.6
Loss
10-4
0.4
10
250 500 750 1000 1250 1500 1750 2000 Forward Pass Number(X10)
50 100 150
1.00
200
250
350
Forward Pass Numbbe((100)
Forward Pack Number(X100)
Figure 2: Performance of Me70, Hiz0O and HiZ0O-I on SST2 task when fine-tuning RoBERTa- large, OPT-13B, L8B) models. HiZOO can achieve 8 xx speedup and 1.55% absolute accuracy improvement compared with McZO.
In light of above, we propose HiZOO, as shown in Figure (left), which estimates the diagonal Hessian by one more forward pass. HiZOO can act as pre-conditioner, directly adjusting the update size of different parameters according to their curvatures. So that it can improve the model convergence when encountered with heterogeneous curvatures. As shown in Figure 2, HiZOO can significantly reduce number of training steps and improve model accuracy. Here we summarize our key contributions as follows:
1. In this work, we estimate the Hessian in zeroth-order optimizer fine-tune LLMs for the first time. Our HiZOO reduces the total number of forward passes required for model convergence and achieves better accuracy. By utilizing diagonal Hessian, HiZOO reduces the corresponding memory cost from (d to Odd). Furthermore, we propose HiZOO-L, reducing the memory usage of Hessian to 10 % of the MeZO.
2. We provide theoretical analysis to prove that HiZOO provides an unbiased estimation of the Hessian. Also, we illustrate how HiZOO utilizes Hessian to improve the convergence process by visualizing the optimization trajectories on test functions.
3. We conduct extensive experiments across different models (RoBERTa-large OPT, Llama3 and Phi-2) with scales from 350M to 66B, different methods (FT, LoRA, prefix), and different downstream tasks (classification, multiple-choice, and generation) to verify the effect of the HiZOO. For example, SST2 task HiZOO achieves a better accuracy and
2

---

Published as a conference paper at ICLR 2025
8 speedup over MeZO average across different models. Even OPT-66B, HizOO outperforms better than MeZO with up to 5.1% absolute improvement.
4. Further exploration in Section 4.3 showcases that HizOO can achieve better performance in optimizing non-differentiable objectives such as Fl score. Specifically, HiZOO significantly outperforms McZO 's results with 6.5 % absolute average.
2
RELATED WORKS
Here we present a concise overview optimizers used in fine-tuning LLMssdetails in Appendix A).
First-Order adaptive optimizer used in fine-tuning LLMs Optimization methods have consis- tently been a popular research domain. Adaptive first-order optimizer, such as Gradient Descent (GD), Momentum, Adagrad (Duchi et al., 2011), are fundamental in many areas like computer vision, natural languagle processing (NLP). Among them, Adam (Kingma & Ba, 2015) plays dominant role due to its fast convergence and is often chosen for training and fine-tuning LLMs. AdamW (Loshchilov & Hutter, 2019) improves upon Adam by adding the weight decay to alleviate overfitting But both of them requires lots of memory cost due to the backpropagation process. This issue has become increasingly critical as the number of LLM parameters skyrockets.
Enhanced optimizers with Hessian On the other hand, researchers incorporated second-order information (Hessian) provide richer guidance for gradient descent during the training. For example BROYDEN (BROYDEN, 1970). .Nsttrrvv & Polyak (Nesterov & Polyak, 2006) and Conn al. (Conn et al., 2000) utilized curvature information pre-condition the gradient; Magoulas et al. (Magoulas et al., 1999) applied diagonal Hessian as the pre-conditioner; Martens (Martens, 2010) approximated the Hessian with conjugate gradient. Sophia (Liu Li, 2023) used light- weight estimate of the diagonal Hessian for pre-trate pre-training LLMs. Despite their potential, above optimizers require the enormous GPU-memory cost. Additionally, these methods can only be used when first-order gradients are available.
Zeroth-Order Optimizer Zeroth- order optimizers, with just forward passes to estimate the gradient, can greatly reduce the memory consumption. It appears in wide range of applications where either the objective functions is implicit or its gradient impossible or expensive obtain. Methods like SPSA (Spall, 1992) have been shown to perform well in non-convex multi-agent optimization (Tang et al., 2021; Hajinezhad & Zavlanos, 2018) or generating black-box adversarial examples (Chen et al., 2017; Cai et al., 2021; Liu et al., 2019a; Ye al., 2019). Recently, MeZO (Malladi al., 2023) first adapted the classical ZO-SGD method to fine-tune L.M,, achieving comparable performance with significant memory reduction. Then Zhang et al. (2024) proposed wider array of 7O optimization techniques. However, these methods often struggle with heterogeneous curvatures.
METHODS
In the following, we briefly introduce the classical ZO gradient estimator SPSA (Spall, 1992), which is used in MeZO. Then describe how HiZOO estimates diagonal Hessian and cooperates with ZOO. We also provide detailed proof for our method.
3.1 PRELIMINARIES
Definition 3.1. Simultaneous Perturbation Stochastic Approximation or SPSA
Given a model with parameters Idd and loss function L, SPSA estimates the gradient on minibatch B, based on the concepts of sampling and differencing, as shown below:
j (0.) C(O, uw; B) L(O. - pm; B)) souu VL(0;; B),
2/r
where u E &d and is sampled from A0 Iz). μ is the perturbation scale. The SPSPSA gradient estimate averages (f) over randomly sampled
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 HiZOO
Require: parameters 0 E IR loss L : [R + IR, step budget Z", perturbation scale learning rate schedule !/, smooth scale ag, diagonal Hessian Ey
1: for 1,... I do
2: Sample batch B and random seed
3: f f [(: B))
PorturbParameterss(,, 14, E./2 s)
4:
5:
4 < L(0; B)
PerturbParameters// -2p,, E1/2 8)
6:
7:
L_ < L(0; B)
< PerturbParameterno 14, x-1/2 8)
b Reset parameters before descent
9:
M 2nte (e+ + --0---2)))2--1 (2, u 2
1/2
Update diagonal Hessian
10:
D = (1-a,
|diag( EgI
projected grad + (c+ (-) E, /2 /22
11:
Reset random number generator with seed s for 0; E 0 do
For sampling ui
13:
Sample u; ~ (N, Ia) 0; t 0; - 1)* projected_grat *u;
15:
16: end for
17:
end for
18: function PERTURBPARAMETERG. μr, E//² 8)
19:
Reset random number generator with seed
> For sampling u,
20: for 0; E do
21:
Sample ii (0, Ta)
22:
0; + 0g + + /
b Modify parameters in place
end for 24: return /
23:
end function
3.2
HESSIAN INFORMED ZEROTH -ERDER OPTIMIZATION
We will present how to estimate Hessian inverse matrix Σ in detail in Section 3.3. Given E, then can construct the following descent direction:
C(o, 1T) un B)) L(O. 1.Tl! uu; 18) 2/// u.
9,, (..)
(1)
2p
With the above descent direction, we can update 0, as follows:
0,+1 = (0t,
(2)
It's guaranteed that (g,()) can estimate the descent direction by the following equation:
E [C(00:1 B3] L(0,; n. ((V (0,; B3, 9n. (0.) +On))
-(0, Ntt
V VC B),
u. Σ 2/2,,,,,
+ (0(1))+0m
L(0,; nel 2 C(,;; B)); + +
where the first and second equality are both from the Taylor's expansion. Above equation shows that when T/ is properly chosen, 9n. (0) can accurately estimate the direction of gradient descent, which is the key the success of fine-tuning large language models.
3.3 DIAGONAL HESSIAN ESTIMATOR
Given a model with parameters Rd, storing the exact full spectral Hessian (d x d) requires (dd) memory (Yao et al., 2018; Xu et al., 2019; Dembo al., 1982), which is sufficient but never
4

---

Published as a conference paper at ICLR 2025
necessary. In HiZOO, just estimate and retain only the diagonal Hessian which requires O(d) memory. It serves as pre-conditioner to scale the direction and magnitude of the model parameter updates according to their respective curvatures.
Drawing from the lemma presented in MiNES (Ye, 2023):
2 (u T /// -
(E-1/2uu) un E
- - H,
(3)
where is the Hessian V²L(0: B) and E is positive definite matrix.
Thus, we can approximate the diagonal Hessian by the zeroth order oracles. Firstly, we will access to the L( + / B), L(O /2; B) and L(0; B). Through the Taylor's expansion, we yield the following results:
1²2 " "1 -00
L(O u B) C(0; m(C(0;B), 2
Similarly, we also have:
1²2
L(0 10 B) = (( B) - ((((0; E2/2u)
T> 2 a(0, -1E2
2
Then we can calculate the difference AC by:
AC -000000--0 + 100 L n. 12 B) B(
= x
a 12 a(0, /2)
Since a(0, (E. and (0, -n 2 u) of order O(r²), can obtain that:
AC
2 = u / L B) (()).
Upon substituting the above results into the left side the Eq. equation 3, we arrive at:
AC 112
(2-1//2
x-
v2 C(0;B)+ O()).
Therefore, by generalizing above equation to the multi-sampling version, we can approximate the diagonal Hessian 4 ((0) at by:
()
AC
Ex 1/-
u.u, u,2
Ex DT ---11-1-111
(4)
2n
122
where n denotes the number of sampling instances for u, indicating the frequency of estimation per step. larger n diminishes the variance of the diagonal Hessian estimation and simultaneously increases computational overhead. Here we adopt 1 as the default setting and present the pseudo-code of HiZOO in Algorithm Further experimental investigation into the impact of varying n is available in the Section 4.6.
Above equation shows that we can approximate the diagonal entries of L² B) by diag(E((0). requiring just one more forward pass per step compared with MeZO.
Due the presence of noise in the calculation of the Hessian, we utilize exponential moving average (EMA) to denoise the diagonal Hessian estimation.
Ez1 -1-aa) oz)Er + |diag(E)))
(5)
In the above equation, we firstly initial the Σg = Ia and update it every step with O(d) memory cost all the time. We also use [diag(Xx)] to keep all entries of Σ, to non-negative.
To further reduce Hessian memory consumption,
propose HiZOO-L to maintain it in low-rank subspace, motivated by Adafactor (Shazeer & Stern, 2018). For Đ-1 Roxq, will store two low-rank matrices R E R and C E R with k 1. Specifically, get 5 by:
5

---

Published as a conference paper at ICLR 2025
= * C1p * R.),
where 1, (1, E Rº denotes a column vector of ones. Then in each step, we will update the and C' separately:
04) o4) + az diag(())) [ii 14: R -1-o4) TT = (1 - (1-aa) CT + aahp * diag(Ei)
Detailed Algorithm can be seen in Appendix D.
3.4 CONVERGENCE ANALYSIS
In this section, we will analyse the convergence based on the assumption of non-convex optimization (details in Appendix B).
Theorem 3.2. Let the descent direction (0) defined as:
1/2 u4; Be) L(0, -.51 (1:3 (0.) = Σ I(0: + u...
uz; 15e) /:1/2 us.
(6)
2bμ
Based on Assumption B. B.I.B.3, if the update rule for is 0c+1 = 0g -ng for a single step, then it's established that: E ([((4))) l| L(0,)
Tht
(VL()) + 2n (tr(E,) Ba) 0(12).
4
Furthermore, given iteration number T. we choose the step size SVT (r..))+..) Pout fg with uniformly sampled from {1....,T) T). Then, have
and take
32T (max, (trxE)) + Ba) (L(01) L(0.))
E [ [l
T3/23e + +0 (4²).
(8)
VT
where L(0)) minimizes the function L(0:). ). The above equation shows that as T - oC, HiZ00 can converge to the stationary point.
Proof. Detailed proof can be found in Appendix B.
3.5 VISUALIZATION HIZOO ON TEST FUNCTIONS
Despite above theoretical guarantee, we still want to illustrate how HizOO utilizes Hessian to improve the convergence process. But it's impractical for large models to visualize their optimization trajectories. Therefore choose three test functions (see details in Appendix C) with heterogeneous curvatures across different parameters and visualize the optimization trajectories on them.
As illustrated in Figure 3, HiZOO and Adam both achieve better convergence three functions, and HizoO even requires less steps for convergence than Adam. However, MeZO only achieves effective convergence in either the y dimension, but not both, indicating limitation in capturing this curvature difference. Particularly in function (c) curvature of x is extremely bigger than In this case, HiZOO can sense this difference parametric curvature and update the function along x on purpose, achieving quicker convergence. In contrast, MeZO is very hard to converge.
4 EXPERIMENTS
Large language models are generally classified into two types: (1) Encoder-Decoder, also known as masked language models, such as BERT (Devlin et al., 2019) and ALBERT (Lan et al., 2020); (2) Recoder-Only also recognized as generative language models, such as GPT family (Radford et al., 2019: Brown et al., 2020), OPT (Zhang et al., 2022a), LLaMA (Touvron et al., 2023), Phi (Li et al., 2023; Gunasekar al., 2023).

---

Published as a conference paper at ICLR 2025
function (n) Optimiattion Paths
Function (b) Optimisation Paths
Punetion lri Optmmrattin Paths
7.4
1.30
Adam:37
Adam 25
Hezo
H9200:39
6.5
1341
Optinal
1.00
Optimaal
Mdm:111 313
6.0
255
H2200:24
0.30
4.5
Cytimal Point
-0.
4.2
3.5
-12
-0.2
8.00
0.50
6.75
1.00
150
-0.5 0.5
-0.30
0.00
0.25
035
Lie
Figure 3: Optimization trajectories of Adam, Me70 and HiZOO on test functions. We have labeled the number of iterations required for the loss to drop to 0.1.
Table I: Experiments on RoBERTa-large (350M parameters, k=16). PEFT represents using LoRA and prefix and we report the best result of them. All reported numbers are averaged accuracy (standard deviation) across 5 runs.
Task Type
SST-2
SST-5
SNLI
MNLI
RTE
TREC topic
Average
sentiment
natural language inference
Zero-shot
79.0 76.0 (42.8)
35.5
50.2
48.8 56.5 (+2255
51.4
32.0 51.3 (15.55
49.5
LP
40.3 (=11
66.0 (+2.7)
59.4 (155.3
58.3
(+)8))
47.5 77.5 (=:19)
70.0 (+2.3)
66.4 (#7.2)
85.0 (4:2.5)
74.9
PEFT
47.7 (=100)
(+1.3)
67.7 (+1.4)
66.6 (+22.0)
85.7 (+133)
72.8
McZO
45.5 (=1.2)
(13.9)
58.7 (42.5)
64.0 (13.3)
76.9 (12.77
67.4
McZ0 (PEFT)
91.4 (1:0))
45.8 (42.0)
(42.5)
62.1 (42.5)
61.0 (+3.99
80.3 (13.6)
68.7
HiZOO
93.2
46.2
74.6
64.9 (+1.7)
66.8 (+11.2)
79.8 (+1.3)
70.9
HİZOO(PEFT)
(+1.2)
(+1.))
71.1 (+1.1)
62.1 (+1.7)
65.4 (+1.2)
82.0 (+2.00)
70.0
To rigorously assess the universality and robustness of our HiZOO, we have chosen models from
each category for empirical testing. Additionally, investigate FT and PEFT (LoRA (Hu al., 2022) and prefix (Li & Liang, 2021)). Detailed experiment settings are presented in Appendix E.
4.1 MASKED LANGUAGE MODELS
Firstly, we conduct experiments RoBERTa-large 350M (Liu et al., 2019b) on three NLP task paradigms: sentence classification, multiple choice and text generation. We follow the experimental setting (Malladi et al., 2023) in studying the few-shot and many- sampling k examples per class for h = 16 (results in Table 1) and 512 (results in Appendix E) We did not utilize HiZOO-L here due model's smaller parameter count.
FT
LoRA on MNLI
prefix on MNU
MeZO:
MeZO:
McZ0: 59.9%
HIZOO: 64.9%
HIZ00: 62.1%
HiZ00: 61.6%
10
10-2
1004-
10-4
10-4
10'"
10-4
10-4
10
10"
0 250 500 750 1000
1250 1500 1750 2003
500
1000
2000 1500
2500
3000
250 520 750 1500 1250 1500 1750 2000 Farward Pass Number(X100)
Forward Pess Number(X100)
Forward Pess NumberX1100)
Figure 4: Training loss curves when using Adam, McZO and HiZOO to fine-tune Roberta-large on MNLI. The evaluation accuracy curves can be found in Figure 11 in Appendix E.
Hizo0 greatly increases the convergence speed across full-parameter tuning, LoRA and prefix. As shown in Figure 4, HiZOO achieves 4x speedup over MeZ0 on average while getting the same training loss compared with McZO. What's more, HiZOO finally achieves a 2.2 2% absolute accuracy improvement on MNLI better than MeZO.
7

---

Published as a conference paper at ICLR 2025
HizOO achieves better performance compared with MeZO. Table shows that HiZOO out- performs MeZO's results with 3.5% absolute on average on all datasets across different tasks. Specifically, HiZOO outperforms MeZO more than 6% in both the SNLI and MNLI dataset.
Table Experiments on three different models(with 1000 examples). We highlight the best results between Me7O, HiZOO and HiZOO-I. in bold to facilitate comparison.
Model
Method
SST-2
RTE
CB
WSC
WIC
COPA
MultiRC
Average
Phi-2
MeZO
86.6
67.1
75.0
59.6
78.2
72.4
Phi-2
HiZOO
88.9
69.0
75.2
62.5
59.4
86.0
79.2
74.3
Phi-2
HiZ0O-L
88.9
68.9
75.2
62.4
59.2
86.0
79.2
74.2
Llama3
MeZO
92.2
74.4
69.6
63.5
57.8
88.0
77.6
74.7
Llama3
HiZOO
93.5
75.1
69.6
63.5
59.7
89.0
78.2
75.5
Llama3
HiZ00-I
94.3
75.1
69.6
63.5
57.7
89.0
77.9
75.3
OPT-13B
MeZO
91.4
66.1
66.0
63.5
88.0
57.3
70.2
OPT-13B
HiZOO
92.1
69.3
69.6
63.5
59.4
89.0
61.3
72.1
OPT-13B
HiZOO-L
92.1
68.2
67.9
65.4
59.4
89.0
61.1
71.9
4.2 AUTO-REGRESSIVE LANGUAGE MODELS
Then extend experiments with Phi-2(2.7B), Llama3(SB) and OPT family the same NLP task paradigms. The results the experiment in Table 2 show that HiZOO outperforms McZO in most cases. Also, loss curves to show the better convergence process of our HiZOO in Appendix E.2.
that HiZOO-L has only slight decrease in accuracy. We also provide relative
HizOO is capable of scaling to large models with up to 66B parameters, while preserving its exceptional performance. As depicted in Table 3, on OPT-30B HizOO outperforms MeZO with up 2.9% increase and 1.1 % increase on average. Even scaling to OPT-66B, HiZOO(prefix) still outperforms (eeO(prefix) with up to 5.1 % increase and 2.7 increase on average.
4.3 TRAINING WITH NON-DIFFERENTIABLE OBJECTIVES
Our proposed HiZOO employs gradient estimation to update parameters, allowing for the use of non-differentiable objectives for training. Following the setting of MeZO (Malladi et al., 2023), we conduct extensive experiments using Fl as optimization objective. The results presented in Table indicate that our method outperforms McZO by 6.54% absolute on Fl average.
Table 3: Experiments on OPT-30B (we use FT and prefix-tuning, report the best of them) and OPT-66B (we use prefix-tuning).
Table 4:
Experiments on non-differentiable optimization objectives (F1). For classification
(k = 512), we use full-paramett tuning and for
SQuAD (1,000 examples), we use prefix tuning.
Task
SST-2
RTE
WSC
WIC
Average
30B McZO
90.6
66.4
63.5
59.1
69.9
Model
RoBERTa-lage (350M)
OPT-13B
30B HiZOO
91.2
60.2
71.0
Task
SST-2
SST-5
SNLI
TREC
SQuAD
30B HiZOO-L
91.1
70.8
Zero-shot
35.5
50.2
32.0
46.2
66B McZO
93.6
66.4
57.7
58.6
69.0
Mc70
92.7
48.9
82.7
68.6
78.5
66B HiZ0O
93.6
71.5
60.6
61.1
71.7
HiZOO
94.9
52.9
83.1
90
83.21
66B HiZ00-L
93.6
71.0
60.3
60.9
71.4
4.4 MEMORY USAGE AND TIME EFFICIENCY ANALYSIS
Memory Usage As shown in Figure 5, HiZOO increases the memory usage compared to MeZO because the storage of the diagonal Hessian(refer Appendix F for detailed numbers). To further reduce memory consumption, motivated by Adafactor (Shazeer & Stern, 2018). Detailed Algorithm can seen in Appendix D. As result, HiZOO-L increases 10% memory more than MeZO, while maintaining the original performance HiZOO. Specifically, using the same GPUs, HiZOO-L allows for tuning model that 10 times larger than what is feasible with FT average.
propose HiZOO-L, the low-rank implementation of HiZ00,
Time Efficiency We analyse the wall-clock time efficiencies and find that HiZOO and HiZOO-L spend 1.5x time per step compared with MeZO, mainly from the extra forward pass, details in
8

---

Published as a conference paper at ICLR 2025
Appendix G. However, HiZOO reduces total number of forward passes required for convergence. For example, HiZOO achieves a 8x and 4x speedup on SST2 and MNLI tasks.
Zero shot
10x
FT
10x
100
MeZ0
HIZOO
Meroory
7x
7x
Figure 5: GPU memory consumption with different OPT models and tuning methods on MultiRC (400 tokens per example on average). More details can be found in Appendix F.
4.5 COMPARISON WITH OTHER ZO VARIANTS
We also compare our HiZOO with a broader array of ZO optimization techniques Zhang et al. (2024). As shown in Table 5, our HiZOO outperforms all other ZO methods. Compared with ZO-Adam who leverages second-order moment to guide gradient descent, our HiZOO-L achieves a notable 4.3 absolute improvement, while using 50% % of the GPU memory.
Table 5: Performance comparison on SST2 and OPT-1.3B) and COPA(OPT-13B) using different ZO methods. Memory and runtime cost are multiples of ZO-SGD.
Model/Task
Roberts-Large
OPT-1.3B
OPT-13B
Average
Memory
Runtime
FT
prefix
FT
prefix
FT
prefix
ZO-SGD
89.4
90.0
90.8
91.4
90.0
79.0
88.4
1.0x
1.0x
Z0-SGD-MMTT
89.6
89.1
85.2
91.2
87.0
85.0
87.8
1.56x
1.0x
70-SGDD-Cons
89.6
89.1
88.3
88.1
82.0
84.0
86.8
1.0x
2.49x
ZO-SGD-Sign
52.5
53.6
87.2
89.5
80.0
78.0
73.4
1.0x
1.0x
ZO-Adam
89.8
90.2
84.4
91.4
82.0
79.0
86.1
2.47x
1.04x
HiZOO
92.7
90.7
91.4
88.0
87.0
90.5
2.04x
1.37x
HiZOO-L
92.7
90.7
91.4
88.0
87.0
90.4
1.12x
1.39x
4.6
HYPERPARAMETER ANALYSIS
Influence of estimation times step
Influence of Smooth Scale(EMA)
n=1
a.=1e-2
n
d==le44
n=16
1.35
1.50
d=lle-8
a 0.-1e-10
1.25
0.50
0.3
50000
100000
300
Steps
Steps
Figure 6: Influence of EMA a for hessian in Eq. equation We use HiZOO (prefix) to fine- tune Roberta-large on SNLL More results can be found in Appendix H.1.
Figure 7: Loss curves on Function (a) using the variant HiZ0O-multi with different estimation times n per step. Trajectory visualization can be found in Appendix H.1.

---

Published as a conference paper at ICLR 2025
Influence of Smooth Scale a; in EMA To assess the robustness of the optimizer, grid search is conducted to evaluate the sensitivity of the prper-paraeeter a2 on RoBERTa-large (350M). Figure illustrates that a, is incrementally increased from zero, the training loss decreases faster. However, too large og values may impede convergence or even cause training to fail due to gradient explosion.
Influence of Estimation Times Per Step We also propose variant of HizOO in Appendix D.2: HizZOOmmutt, which has n > per step. As shown in Figure 7, different n maybe doesn't affect the final accuracy. However, the larger n. will estimate the diagonal Hessian more accurate per step and accelerate model convergence, reducing the overall training steps. But it will also increase the computation per step. Balancing these factors is crucial for efficient training.
CONCLUSION
In this work, we introduce HiZOO, which is the first ZOO that incorporates diagonal Hessian for fine- tuning L.MMs. By introducing one more forward pass, HiZOO can handle heterogeneous curvatures across different parameter dimensions. We provide theoretical analysis and visualize the optimization trajectories to explore how it works. Further experiments show that HiZOO converges in much fewer steps than MeZ0 and achieves better performance across various LLMs. We also explore memory efficient implementation (HiZOO-L) to reduce the Hessian consumption.
ACKNOWLEDGMENT
This work was supported in part by National Key Research and Development Project of China under Grant 2022YFA1004002 and National Natural Science Foundation of China under Grant 7247 118..
REFERENCES
Alekh Agarwal, Martin Wainwright, Peter Bartlett, and Pradeep Ravikumar. information-theoreticc lower bounds on the oracle complexity of convex optimization. Advances in Neural Information Processing Systems, 22, 2009.
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning, 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners, 2020.
C. G. BROYDEN. The Convergence a Class of Double-rank Minimization Algorithms 1. General Considerations. IMA Journal of Applied Mathematics, 6(1):76-90, 03 1970. ISSN 0272-4960. doi: 10.1099//mamat/6.. 1.76. URL https: /oii org/ /
1 10.100..0 1093/
Hanqin Cai, Yuchen Lou, Daniel Mckenzie, and Wotao Yin. A zeroth-order block coordinate descent algorithm for huge-scale black-box optimization. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 1193-1200 PMLR, 18-24 Jul 2021.
Minping Chen, You-Liang Huang, and Zeyi Wen. Towards efficient low-order hybrid optimizer for language model fine-tuning. 2025a
Pin-V Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. New York, NY, USA, 2017. Association for Computing Machinery. ISBN 450355222 doi: 3140448.
10.1 1144333428572314444444 URL https: /ooi orw...rg/1i. 11..00.000 1411111//222
Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth- order fine-tuning for language models with low-rank structures. arXiv preprint arXiv:2410.07668, 2024.
10

---

Published as a conference paper at ICLR 2025
Yiming Chen, Yuan Zhang, Yin Liu, Kun Yuan, and Zaiwen Wen. A memory efficient randomized sub- space optimization method for training large language models. arXiv preprint arXiv:2502.07222, 2025b.
Andrew R. Conn, Nicholas M. Gould, and Philippe L. Toint. Trust-Region Methods. Society for Industrial and Applied Mathematics, USA, 2000. ISBN 0898714605.
Ron S. Dembo, Stanley C. Eisenstat, and Trond Steihaug, Inexact newton methods. SIAM Journal on Numerical Analysis, 19(2):400-408 1982. doi: 10.1137/0719025. URL https: //oii org/
N0.1137/071900
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms,, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapter of the Associa- fion for Computatioonal Linguistics, 2019. URL https: //tp:/aappp apppp www.nticscholarr......e..aaa.aae org/
Ming-Wei
CorpusID 52967399.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Mach. Learn. Res., (2(null):21112215) jul 2011. ISSN 1532-443 -435.
FairScale authors. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https: / thub. cccmm//acceooooeeeeeradcoo / airrscale, 2021.
Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approximate natural gradient descent in cronecker-factored eigenbasis. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS' 18, pp- 9573-9553, Red Hook, NY, USA, 2018. Curran Associates Inc.
Saced Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23(4):2341-25 -23688 2013.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural net opti- mization via hessian eigenvalue density. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning. volume 97 of Proceedings of Machine Learning Research, pp. 2232-2241. PMLR, 09-15 Jun 2019. URI https: https//poocceddnge www.mrrss..s...mmmmmmmmoomms v97 2t37/ghobbani1. html.
Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, Adil Salim. Shital Shah, Harkirat Singh Behl, Xin Wang, Sébastien Bubeck, Ronen Eldan, Adam Tauman Kalai, Yin Tat Lee, and Yuanzhi Li. Textbooks are all you need, 2023.
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang. Yide Ran, Jacob R Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, et al. Zeroth-order fine-tuning of Ilms with extreme sparsity. arXiv preprint arXiv:2406. 02913, 2024.
Davood Hajinezhad and Michael M. Zavlanos. Gradient-free multi-agent nonconvex nonsmooth optimization. 2018 IEEE Conference on Decision and Control (CDC), pp. 4939-4944, 2018. URL https: /upp:///ap.. www.nticschoo..........m.
org/ Corpus 586664445
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhr Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International
Conference on Learning Representations, 2022. URL https: /oppeeeeeeeeeeeee net. fort//frmm? id=nZevKeeFyYf,
Kevin G Jamieson, Robert Nowak, and Ben Recht. Query complexity of derivative-free optimization. Advances in Neural Information Processing Systems, 25, 2012.
Shuoran Jiang. Qingcai Chen, Youchen Pan, Yang Xiang, Yukang Lin, Xiangping Wu, Chuanyi Liu, and Xiaobao Song. Zo-adamu optimizer: Adapting perturbation by the momentum and uncertainty in zeroth-order optimization, 2023.
11

---

Published as a conference paper at ICLR 2025
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), San Diega, CA, USA, 2015.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference Learning Representations, 2020. URL https: ptattp:://ppeeevetie /ptp///ppeeeeee.eee/eeee net forum? id=Hl e A7.EtvS.
Xiang Lisa Li and Percy Liang. Prefix-tuning Optimizing continuous prompts for generation. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 4582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653//1//0221.aaal aacl.oo.....- org/ 2022 ael-oonnnnnnn.... 353.
URL httph::///// /aaaaccaannnhooo
Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. Textbooks are all you need ii: phi-1.5 technical report, 2023.
Hong Liu and Zhiyuan Li. Sophia: A scalable stochastic second-order opti- mizer for language model pre-training. https:/ https//ttccaccco//aaae. 17aca766-2011 4c7c- a0f4- 5078557daabb,, 4 2023.
Sijia Liu, Pin-Yo Yu Chen, Xiangyi Chen, and Mingyi Hong. signsgd via zeroth-order oracle. In International Conference on Learning Representations, 2019a. URL https://api.
semant comanticschllin wrw.aorpssCCoooCooooo...o
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1907. 1692, 2019b.
Yong Liu, Zirui Zhu, Chaoyu Gong, Minhao Cheng, Cho-Jui Hsieh, and Yang You. Sparse mezo: Less parameters for better performance in zeroth-order I1m fine-tuning. arXiv preprint arXi::2402.1551! 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019, URL https:////// wwwp//pweeeeweee////pe/e net forum?id= Bkg6Ricqy7.
Jan R Magnus et al. The moments of products of quadratic forms in normal variables. Univ., Instituut voor Actuariaat en Econometrie, 1978.
G. D. Magoulas, M. N. Vrahatis, and G. S. Androulakis. Improving the convergence of the backpropagation algorithm using learning rate adaptation methods. Neural Compul., 11(7); 1769-1796, oct 1999. ISSN 0899-7667. doi: 10. 101166298899666933000 62233. URL https: //doi. org. 10. 1tt.//....11229996699338338
Sadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian, Jason D. Lee, Danqi Chen, and Sanjeev Arora. Fine-tuning language models with just forward passes. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:/ opwnreveeneeeeeeeeeeeeeee net/ forum? um?
id-Vota6rFhBC.
James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML' 10, pp. 735-742, Madison, WI, USA, 2010. Omnipress. ISBN 978 1605589077.
Yurii Nesterov and B. T. Polyak. Cubic regularization of newton method and its global performance. Math. Program., 108(1):177-205 aug 2006. ISSN 0025-5610.
Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024.
Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks, 2014.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Maxim Raginsky and Alexander Rakhlin. Information- complexity, feedback and dynamics in convex programming. IEEE Transactions Information Theory, 57(10):7036-7053 2011.
Levent Sagun, Leon Bottou, and Yann LeCun. Eigenvalues of the hessian in deep learning: Singularity and beyond, 2017. URL httpttt/tptppeeeeeeeeette wwwprewwww..eeeeeeoeeeeoe..o.wwe net wwt.teeeeetiieeeeeeoe P
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. In Sanjoy Dasgupta and David McAllester (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 Proceedings of Machine Learning Research, pp. 343-351, Atlanta, Georgia, USA, 17-19 Jun 2013. PMLR. URL httpt:ttt//......t... http:///////eeeeeeee ngs . ml press / v288/ scsauu133 html.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4596-4604. /ttptpp////p/oo/eeeo.ooo
PMLR, 10-15 Jul 2018. URL https: http://ccccceeee..ceeeeee mlr http.pppss///////aeeee1....... shweeeeraaa.eeaa..m html.
James C. Spall. A one-measurement form of simultaneous perturbation stochastic approximation. doi: 1111016680000-100899)00000111110
Automatica, 33(1):109 1997. ISSN 0005-1095,
URL https:://///mm doii http://ooooooog/////// /t.zg/10.100 1016 /00005 1098 (96) 00149-5. 5.
J.C. Spall. Multivariate stochastic approximation using a simultaneous perturbation gradient approxi- mation. IEEE Transactions on Automatic Control, 37(3):332 -311, 1992. doi: 10.1109/9. 199632.
Yan Sun, Tiansheng Huang, Liang Ding, Li Shen, and Dacheng Tao. Tezo: Empowering the low- rankness on the temporal dimension in the zeroth-order optimization for fine-tuning Ilms. arXiv preprint arXiv:2501. 19057, 2025.
Qitao Tan, Jun Liu, Zheng Zhan, Caiwei Ding, Yanzhi Wang, Jin Lu, and Geng Yuan. Harmony in divergence: Towards fast, accurate, and memory-effiiiet zeroth-order Ilm fine-tuning. arXiv preprint ar Xiv:2502.033044 2025.
Xinyu Tang. Effectively learning from data and generating data in differentially private machine learning. PhD thesis, Princeton University, 2024.
Yujie Tang, Junshan Zhang, and Na Li. Distributed zero-order algorithms for nonconvex multiagent optimization. IEEE Transactions on Control of Network Systems, 8(1):269-28)) 2021. doi: 10.11099TCS..222..3022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302. 13971, 2023. URL htttps:/aap..semmnt..c.o semanticscholar. org/ CorpusID:25721944
A. T. Vakhitov, O. N. Granichin, and L. S. Gurevich. Algorithm for stochastic approximation with trial input perturbation in the nonstationary problem of optimization. Autom. Remote Control, 70(11):1827-1835 httpss://ol org 10.
doi: 10.1134/500511119991000000 2009. ISSN 0005-1179.
URL
Yilong Wang, Haishan Ye, Yong Liu, Guang Dai, Ivor Tsang, and Jingdong Wang. The advance- ment in stochastic zeroth-order optimization: Mechanism of accelerated convergence of gaussian direction on objectives with skewed hessian eigenvalues.
Zhongruo Wang, Krishnakumar Balasubramanian, Shiqian Ma, and Meisam Razaviyayn. Zeroth- order algorithms for nonconvex minimax problems with improved complexities. arXiv preprint arXiv:2001.078/9, 2020.
Peng Xu, Fred Roosta, and Michael W. Mahoney. Newton-type methods for non-convex optimization under inexact hessian information, 2019.
13

---

Published as a conference paper at ICLR 2025
Zhewei Yao, Peng Xu, Farbod Roosta-Khorasan Khorasani, and Michael W. Mahoney. Inexact non-convex newton-type methods, 2018.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning. Proceedings of the AAAI
Conference on Artificial Intelligence, 35(12):00655-0067 May 2021. doi: 10.1600/aaai.v33iii 17275. URL https: /tt////j///// aaai. org/ index. AAA/AAAA//////// httpp///aaaaaaaile////ee////////
Haishan Ye. Mirror natural evolution strategies, 2023.
Haishan Ye, Zhichao Huang, Cong Fang. Chris Junchi Li, and Tong Zhang. Hessian-aware zeroth- order optimization for black-box adversarial attack, 2019.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsich. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference on Learning Representations, 2020. URL https::/ /pwpreveeeee...em/////e net forum? ind Syx.d--yyxwtEtttt
Matthew D. Zeiler. Adadelta: An adaptive learning rate method, 2012.
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit, Seungyeon Kim, Sashank Reddi, San- jiv Kumar, and Suvrit Sra. Why are adaptive methods good for attention models? In H. Larochelle, M. Ranzato. R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 15383-15393. Curran Associates, Inc., 2020. URL htt.t:p:ppppooeeiiiiec ngs. neurips cc/paper_teeefeeeeeeeeeeee fww.fills//eeeee pptpp/pppr/222//////// (tte://bbb777aacdl10088848388883 4006447000000033eeea..eee.
Lin Zhang, Shaohuai Shi, and Bo Li. Eva: Practical second-order optimization with kronecker- vectorized approximation. In The Eleventh International Conference on Learning Representations, 2023. URL
Lew. net/ forum? d= Mi cBV96Voy.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models, 2022a.
Yan Zhang, Yi Zhou, Kaiyi Ji, and Michael M. Zavlanos. A new one-point residual-feedbaee oracle for black-box learning and control. Automatica, 136(C), feb 2022b. ISSN 0005- doi: 10.1016/j automaaaattmmmcc22222. 10006. URL https:::: /twi/oi....//////// /0010///000000///////// automatt ca
2021. 110006.
Yihua Zhang, Pingzhi Li, Jumyuan Hong, Jiaxiang Li, Yimeng Zhang, Wenqing Zheng, Pin- Yu Chen, Jason D. Lee, Wotao Yin, Mingyi Hong, Zhangyang Wang, Sijia Liu, and Tianlong Chen. Revisiting zeroth-order optimization for memory-efficient LLM fine-tuning: A benchmark. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 59173-59190. PMLR, 21-27 Jul 2024. URL https:: htt.e:p::pppceeeeeeeeeeeeeeeeeee http:/pocceeeeeee pttpppppcceen...eeee/
www.s.ppess/eeeeee vrressvv35. v235/223///////// 222335//aaaan zhang2 4ad. html.
Huaqin Zhao, Jiaxi Li, Yi Pan, Shizhe Liang, Xiaofeng Yang, Wei Liu, Xiang Li, Fei Dou, Tianming Liu, and Jin Lu. Helene: Hessian layer-wise clipping and gradient annealing for accelerating fine-tuning Il with zeroth-order optimization. arXiv preprint arXiv:2411. 2024a.
Jiawei Zhao, Zhenyu Zhang. Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024b.
14

---

Published as a conference paper at ICLR 2025
A RELATED WORKS
A.1 FIRST-ORDER OPTIMIZER USED IN LLMS
Optimization methods have consistently been popular research domain, encompassing techniques such as Gradient Descent (GD). Momentum, Adagrad (Duchi et al., 201 1). ADADELTA (Zeiler, 2012), and Newton's method, which have been instrumental in advancing fields like computer vision. However, the emergence of large-scale models, characterized by their massive parameter counts and intricate architectures, has challenged the efficacy of conventional optimization methods for training tasks. Amidst this landscape, Adam (Kingma & Ba, 2015) has emerged as the preferred choice for its ability to rapidly converge, making it particularly suitable for the training and fine-tuning large models. Then A (Loshchilov & Hutter, 2019) was proposed to add a weight decay coefficient to alleviate over-fitting Notwithstadding these advancements, a limitation persists with these optimizers: they have implicit batch size ceiling. Exceeding this threshoid can provoke extreme gradient updates, thus impeding the convergence rate of the models. This bottleneck is particularly problematic in the context of large-model training, which typically necessitates substantial batch sizes. To circumvent this constraint, LAMB (You et al., 2020) was devised to apply principled layer-wise adaptation strategy to accelerate the training of large models employing large batches.
A.2 HESSIAN BASED FIRST-ORDER OPTIMIZER
Compared with first-order optimizers, second-order optimizer considers second-order information in the process of gradient calculation. As a result, it has more abundant information to guide gradient descent and is considered to be more promising. Previous studies utilized curvature information to pre-condition the gradient (BROYDEN, 1970; Nesterov Polyak, 2006; Conn et al., 2000). Subsequently, Magoulas et al. (Magoulas et al., 1999) applied diagonal Hessian as the pre-conditioner, which greatly promotes the landing of second-order optimizer in the field of deep learning. Martens (Martens, 2010) approximated the Hessian with conjugate gradient. Schaul et al. (Schaul et al., 2013) utilized diagonal Hessian to automatically adjust the learning rate of SGD during training. Another work (Pascanu & Bengio, 2014) extended natural gradient descent incorporate second order information alongside the manifold information and used a truncated Newton approach for inverting the metric matrix instead of using diagonal approximation of it. EVA (Zhang al., 2023) proposed to use the Kronecker factorization of two small vectors to approximated the Hessian, which significantly reduces memory consumption. AdaHessian (Yao et al., 2021) incorporates an approximate Hessian diagonal, with spatial averaging and momentum to precondition the gradient vector.
Although great progress has been made in the research of second-order optimizer, it has not been widely used because of the extra computation and memory cost when gradient updating, and this situation is extremely serious in the training of large language models. Based on the above dilemma, recent works (Anil al., 2021; George et al., 2018) proposed to offload Hessian computation to CPUs and utilized ResNets and very large batch size to approximate the Fisher information matrix. Sophia (Liu & Li, 2023) was the first to apply second-order optimizer and achieve a speed-up on large language models in total compute successfully.
A.3 ZEROTH-ORDES OPTIMIZER
Zeroth-order optimization, is also known as derivative-free or black-box optimization. There have been many one-point gradient estimators in past works (FairScale authors, 2021; Spall, 1997; Vakhitov et al., 2009; Spall, 1992; Jamieson et al., 2012; Agarwal et al., 2009; Raginsky & Rakhlin, 2011 1; Wang et al., 2020). However, cursory experiments with one such promising estimator (Zhang et al., 2022b) reveal that SPSA outperforms other methods.
In previous works, it appears in a wide range of applications where either the objective function is implicit its gradient is impossible or too expensive to compute. For example, methods (Tang et al., 2021; Hajinezhad & Zavlanos, 2018) consider derivative- free distributed algorithms for non-convex multi-agent optimization. ZO-BCD(Cai et al., 2021), ZOO(Chen et al., 2017). 20- (Liu et al., 2019a) and ZO- Aware (Ye et al., 2019) utilize zeroth-order stochastic optimization to generate black-box adversarial example in deep learning.
15

---

Published as a conference paper at ICLR 2025
Beyond that, MeZO (Malladi et al., 2023) firstly adapted the classical ZO-SGD method to fine-tune LLMs, while achieving comparable performance with extremely great memory reduction and GPU- hour reduction. Recently there have been many subsequent excellent works (Jiang et al., 2023; Zhao et al., 2024a; Liu et al., 2024; Guo et al., 2024; Tang, 2024; Chen et al., 2024; Wang et al.; Chen al., 2025a; Tan et al., 2025; Chen al., 2025b; Sun et al., 2025). All of these optimizers provide researchers with a new and promising technique for fine-tuning large models.
B DETAILED CONVERGENCE ANALYSIS
Firstly, our convergence analysis requires the following assumptions:
Assumption B.1. The objective function L(0) is L-smooth, which means that for any 0,. 02 E Rd. holds that:
L(O2) < ((,)) + (VI()))) 82 01) + + 01112.
(9)
Assumption B.2. The stochastic gradient (() has o² variance, which means:
E ( - VL(()) 0²²
(10)
Assumption B.3. Each entry of Σg lies in the range [Se, Sul with 0 < Be S Bu-
Then we will give the detailed proof for convergence.
Proof. By the update rule of 0g and Assumption B.1, we have
E ([ 0.]
<L(0,) - (VV + Lu (0.)11
<I(0)) - (V))1 + +m (0/////
+ 1 (tr(E,) 8w) (VL(
+ 2n/ L (tr(E,) + 8.) +
<((O)) " (L) + + (tr(E,) + (a) (O)(E
+ +2 L (ur(E)) + +b,) 0² + +0(2)
=L((.) Tt (1 - 4m [(t((E))) + B₂)) (VL(())
+ +n T. (E+().))) + +Bu) + 0((2)
<L())) - 1t VL + 2n L (tr(E,) - 0
where the second inequality is because of Lemma B.4 and the last inequality is because of the value of ?)-
Rearrange above equation and summing up it, can obtain that
T (0.)
T
E
Th 4
S Σ (L(O,) I(0(41))
t=1
+ 2n L (tr(E,) + +B,) + OTT)?)
=L(01) L(0r-1) + 2 (tr(2,) + (.) O
<I(0)) L(0.) (tr(Ea) 02 (TT)))
16

---

Published as a conference paper at ICLR 2025
By taking out 8g with uniformly sampled from {I, and taking expectation, we can obtain that
T (L((6,)
E [ = T
Σ
T3
1=1
1=1
4(L(01) - -((0)) + (tr(E,) + 3u) 0² 0
TBE
The
32L (tr(E,) + (.,) (L(0)) - L(O.))
0²
+ + (1²),
VT3
T3/23
where the first inequality is because of the assumption that the diagonal entries of Σ, is no less than Be.
Eq. equation shows that once we choose the step size 1) properly. I(f++1) will be less than
((,)) in expectation up some noises of order μ²² Specifically, if set 1) 8/TL(max, tr(E))+a.))
Eq. equation 8 implies that we can find an solution out such that E [IVV I e in O( iterations. This rate matches the one of (Ghadimi & Lan, 2013).
Lemma B.4. We assume that Assumption B.2 and Assumption B.3 hold. Then, gp defined in Eq. equation has the following properties:
E 19 (0:) = E, ( + 0(4)
E (0))) - 4 (tr(E.) + B.) (0,)),
+ 43, (tr(E,) o²) + 0(p2).
Proof. By the definition of (0r), we have
9u (0.)
8 L(G 1. (u) L(0: /.D./ Us) =2112 u4
26p
V L(0,) ) |112 U, + 01112)
1///2
26μ
u T Σ 12 ( + O()).
Thus, we can obtain that
E
(0)) (.
(11)
Moreover,
E
(0)))1
1/2 uuu
1/2 +
E
<2E
1
/2 u.u
11 12 V
+0(22)
2
E || | 1/2 nuu
u 2 / 22
0ww3)
-6
-2ir(E.) ((..)
+ + L ((..))VL(()) + 0
<2 (tr(E)) + B.)) V ((0,),, 0(13)
17

---

Published as a conference paper at ICLR 2025
where the last equality is because of Lemma B.5.
Finally, we have
E V [(0))2.VL(00) (I)
2E (8.) + 2B ( - (L((.)) 2 (8,)) + + [ (0,) - VL(O))
V + 23,,0²
where the second inequality is because of Assumption B.3 and the last inequality is because of Assumption B.2.
Therefore.
E (00)l < <4(t((2)) + B.) (ZL + +48, (tr(Ee) + o?
Lemma B.5. (Magnus al., 1978) Let A and B be two symmetric matrices, and obeys the Gaussian distribution, that is, u ~ (O, Ia)- Define 2 = u Au-u Bu. The expectation of z is:
E., | (trA)(trB) + 2tr(AB).
(12)
C TEST FUNCTIONS OF THE OPTIMIZATION TRAJECTORIES
For better illustrating how HiZOO utilizes hessian to improve the convergence process, we choose below three test functions with heterogeneous curvatures across different parameters. Figure we provide the 2D convergence paths of three functions and the variation of their losses with respect to steps.
Function (a): f(z, y) 8(x 1)3 + 2x 1) + 0.5 4)2
Function (b): f(z y) |x| + lyl
Function (c): f(z y) 10000 + n²
Adarc25
Nezc/uilure
HE200.20
#2003
6.0
1,00
2007
M620 Falure
H200 126
mpti PaiK
-1.
1.0
-058--3
25
330
0.5
858
075
1,00
335
150
-1.
-05
0.
E.5
1.
-0.50
00.00-11
50-005 6CE
0.00
6.55
130
Lass Campariion
(e)
Loss Compar son on Function 06)
Loss Comparison on Function (c)
2.30
Adam::7
11300
Adam.25
1.75
Mc2O/falure
H2200
1.5
6300
4300
139
250
20
255
Stegs
Staps
Figure 8: 2D trajectories of Adam, MeZO and HiZOO 3 test functions. The upper figures are the 2D trajectories of gradient descent, and the bottom parts are the corresponding loss curves.
Function (a) from (Liu Li, 2023).
18

---

Published as a conference paper at ICLR 2025
D HIZOO VARIANTS
D.I HIZOO-I
Due to the storage of Hessian, HiZOO introduces extra memory cost, which is equal to the size of the model parameters. To address this limitations, we propose HiZOO-I, the low-rank implementation for the storage of Hessian, motivated by Adafactor (Shazeer & Stern, 2018). Details can be see in Algorithm 2. We also visualize the loss curves of HiZOO and HiZOO-L in Figure 9 and find that on datasets two algorithms perform closely. This also indicates that the estimation of Hessian in HiZOO may be sparse, so we encourage researchers to try other memory efficient algorithms to compress the Hessian.
Algorithm 2 HiZOO-L
Require: parameters 0 E Rd loss R IR, step budget perturbation scale μ. learning rate schedule smooth scale a,, diagonal Hessian Ro. Co
for 1, .... do
2: Sample batch B c 'D and random seed
3: + [(;;
4:
0 < PerturbParaameters((.
R1-1, Ct-1,
5:
4 t L(0; B)
6:
9 PerturbParameters(( -2p, RI-1, C)-1,
7:
L <- L(0;B) B)
8:
PenturbParameterii μ4, 14 Ct-1,
Reset parameters before descent
9:
--1 (B4-1 C1-1)/(1, R2(1)
Σ 2n² (e+ + l- -20)(= -1/2 U.u1
--112
10:
2
11:
R = (1 - (=,))) + ar diag(( (i) mm
12:
(1- az) + arl diag({)) (2))
projected grad + (44 - (-) /1/2/21
13:
14:
Reset random number generator with seed
b For sampling u,
15:
for 0g E 0 do
16:
Sample u; (0, Ta)
17:
0, 1,+0,-- - + projected_ grad *u
end for
19:
end for
20: function PERTURBPARAMETERGS μr, R,, C₁ s)
21: Reset random number generator with seed
D For sampling uy
22: for 0; E 0 do
23:
Sample us ~ No Ia)
24:
Σ = (R * (1) R.)
25:
0, - 0, + 1 1/2
Modify parameters in place
26:
end for
27: return 0
28:
end function
D.2 HIZOO-MULTI
There is rich history of transferring ideas from first order optimization enhance ZO algorithms. Below, we highlight the variant of HiZOO: Hiz00 -ulti which can perform estimation times per step efficiently as shown in Algorithm 3. We conducted experiments to explore the influence of estimation times per step as shown in Figure 16. We can conclude that when is larger, the estimation diagonal Hessian is more accurate. It can decrease the variance of the estimated diagonal Hessian matrix during each step and thus reduce the overall training steps, but will cause much more computation per step meanwhile. So choosing an appropriate value of n is very important during the training.
19

---

Published as a conference paper at ICLR 2025
Liama3 on SST2
Lama3 on CB
Uama3 on Copa
2.4
HIZ00: 93.5% 5%
HiZ00: 64%
H2000: 89%
HIZ0O-L: 94.3%
HIZ0O-L: 69 64%
HIZ00-L: 89%
0.6
1055
1.8
64
0.6
Number(X100)
Number(X100)
Figure
Loss curves on Llama3 between HiZOO and HiZOO-L.
Algorithm 3 HiZOO-multi
Require: parameters 0 E Rd, loss L: Rd - IR, step budget T, perturbation scale batch size B, learning rate schedule n/, smooth scale 0g, estimate times n, Hessian matrix Eg
1: for 1,,,.., do
2: seeds, projected grads
3: for j = I,... do
4:
Sample batch B C and random seed s
5:
- L(A; [3)
PerturbParameters((, μ. x//² s)
6:
7:
4 + L(0; B3)
8:
- PerturhParametern -2,, x-1-1- 8)
9:
L_ + L(0; B)
+ Perturb ParametParameters((, μ4 1//2 8)
b Reset parameters before descent
Σ - (f+ + +4 20, 1 1/2 ueu. E,-1
12:
Σ, (1 a. - 0t |diag Eg)
Update Hessian matrix
projected_greet _rad - (f4 {-) E 1/1 /2μ
13:
poojected_gradsjjj projected_grat
15:
seeds[j] <
end for end for
17:
for j j= 1= 1,..., do
19:
Reset random number generator with seeds[j]
20:
for 0, E do
21:
u; (0,
22:
A: 8, - (ott-m(nnn))xojjeeeeeee (n)))pojecteddgrraba *u;
b Avg grad for u1,... Up
23:
end for
24:
end for
25:
function PERTURBPARAMETERRC μ., 11 .3)
Reset random number generator with seed s
27:
for 0g E 0 do
u, N(, Ia)
29:
0, 0,+ 115.)
Modify parameters in place
30:
end for
31: return 0
32: end function
20

---

Published as a conference paper at ICLR 2025
E EXPERIMENTS ON LLMS
E. DETAILED EXPERIMENTS ROBERTA-LARGE
We use the hyperparameters in Table 6 for HiZOO experiments RoBERTa-large. Regarding learning rate scheduling and early stopping, we use constant learning rate for all HiZO0 experiments.
Table 6: The hyperparameter grids used for RoBERTa-large experiments. HiZOO uses a constant learning rate schedule. All HiZOO experiments 100K steps.
Experiment
Hyperparameters
Values
HizO0
Batch size Learning rate
64
{lc- 7, 1e-1e-00 6, 1e-5)
le-3
Weight Decay
0
(iZ00(prefix)
Batch size
64
Learning rate
(le 2, 2e.333
3, 1e-3)
le-1
Weight Decay # prefix tokens
5
HiZOO(LoRA)
Batch size Learning rate
64
{le- 5, 5e-5, 1e-4)
1e-3
Weight Decay (r,
0.1 (8, 16)
Table 7: Experiments on RoBERTa-large (350M parameters, k=512), For McZO we report the results reproduced.
Task Type
SST-2
SST-5
SNLI
MNLI
RTE
TREC topic
Average
sentiment
natural language inference
Zero-shot
79.0
35.5
50.2
48.8 71.5 (++1.)
51.4
32.0
49.5
(+0.5)
51.7 (+-05)
80.9 (+11.))
73.1 (+:1.)
(++0.))
76.3
FT
(+1.8)
(+11.)
(+2.6)
70.0 (+2.3)
(+772)
(4.0.)
73.1
FT (LoRA)
(+1.7+
(+1.1
(+4.3)
67.7 (+1.4)
66.1 (+-3.5)
(+-.1)
71.6
(prefix)
(1:10)
(11.1)
(11.3)
(1:2.)
66.6 (1:2.0)
85.7 (1:13)
72.6
Mc70
(+0077
(3.244
(3100)
78.3 (++.5)
78.6
94.3
80.1
Me70 (LoRA)
(+-06))
45.4(=1.1)
62.1(+0.9) 64.6(+1.2)
61.((+8))
80.8++1.5)
67.4
McZO (prefix)
(1...)
(10.5)
(21.1)
(111.)
77.2.0..8
88.2 (L0.7)
78.4
HiZOO
(+-2.)) (+004)
(2.67)
77.7 (+0.6)
80.0 (+
94.6 (+I.
80.6
HiZOO (LoRA)
91.7 (1:03)
45.3
76.5 (1:03)
63.1 (1:0.6
70.4 (1:14)
(115))
72.1
HiZOO (prefix)
96.1 (+0.2)
54.2
85.7 (=0.7)
79.7 (=1.0)
77.3 (+0.2)
93.9 (+0.6)
81.2
Rolt:Sie-angee711 ATT
RoBERTS (angeluRR)) RTE
RoBERTo www.seDDoooelpeeehiooooo
3620
Mi20
Mi20
H200
Hz200
H2200
1244
150
1063
1983
3530
3000
1000
1500
2530
330
3200
1500
2003
2500
3005
Figure 10: Loss curves on RoBERTa-large between MeZO and HiZOO.
In Table 7 we show the full experiment results. Additionally, plot more loss curves to compare with MeZO. As shown in Figure 10, we can see that HiZOO can greatly accelerate the training process over MeZO, which verifies the robustness of HiZOO.
21

---

Published as a conference paper at ICLR 2025
N:BeHto Srg.FFlo 5512
Ropesta MageeloorrgePoooooooooooo of SKU
R:BERO Aegg on kiE
2
130
1983
Forwars Sime5X1100)
Forwarg timesiX1001
Forwart (imes(X100)
Figure 11: Accuracy curves on RoBERTa-large between MeZO and HizOO.
E.2 DETAILED RESULTS ON VARIOUS LLMS
We use the hyperparameters in Table 8 for HiZOO experiments on OPT. Full results for OPT-30B and OPT-66B are in Table 9. We also provide the relative loss curves of fine-tuning OPT family in Figure 12. We provide several loss curves of fine-tuning Phi-2(2.7B) and Llama3(8B) in Figure 13 and Figure 14.
Table The hyperparameter grids used for OPT experiments. All weight decay set to 0. HiZOO uses 20K steps and constant learning rates.
Experiment
Hyperparameters
Values
HiZOO
Batch
Learning rate
{le- 6, 5e- 7, Telee-) 7}
1e-3
HiZOO(prefix)
Batch size Learning rate
16
{5e- 2, le- 2, 5e-3)
le-1
prefix tokens
HiZOO(LORA)
Batch size
16
Learning rate
{1e-4, 4, 5e-5, le-5)
/
le-2 (8, 16)
(r, a:)
with Adam
Batch size
Learning Rates
(le 5, 5e-5ee 5, 8e 8e-5) 5)
Table 9: Experiments on OPT-30B and OPT-66B(with 1000 examples). The best results are highlighted in bold for better comparison. We highlight the best results between HiZOO and MeZO in bold to facilitate comparison.
Task
SST-2
RTE
WSC
WIC
30B zero-shot
56.7
52.0
38.5
50.2
30B ICL.
81.9
66.8
56.7
51.3
30B MeZO
90.6
66.4
63.5
48.9
30B MeZO(prefix)
87.5
66.1
55.8
59.1
30B HiZOO
90.3
69.3
63.5
53.4
30B (izOO(prefix)
91.2
68.6
57.7
60.2
66B zero-shot
57.5
67.2
43.3
50.6
66B ICI.
89.3
65.3
52.9
54.9
66B (e7OOpreffxx
93.6
66.4
57.7
58.6
66B (iz(OOpreeixx)
93.6
71.5
60.6
61.1
22

---

Published as a conference paper at ICLR 2025
DPT-138tprefx on CB
OFT-30B(prefiv) on WSC
(PT-668(prefix) WSC
0.75
0.9
MeZ0 HiZ0O
0.70
Mazo HizaO
0.70
MeZ0
0.61
2.5
0.60
105
0.7
5 0055
0.50
2.5
0.00
0.50
3.5
0.40
50
100
150
200
250
100
205
200
420
0,40
Forward Pass Numberr(X10))
Forward Pass NumberrX1101
Number(X100
Figure 12: Loss curves on OPT between Me70 and HiZOO.
Phi-2 on RoCoRD
Phi-2 on SST2
Phi-2 on WIC
3.3
1.0
MezO: 86.58%
1.0
MeZO: 54.4%
HIZOO: 88.87%
HiZOO: 59.4%
3.0
2.9
8
0.6
2.7
2.6
McZO: 71.7%
2.5
HIZ00: 72.1%
100
200
350
400
600
1:0
200
300
400
500
200
300
400
500
Forward Pasa Number0X100)
Forward Pass (umber(X100)
Forward Rogs Number(K100)
Figure 13: Loss curves on Phi-2 between MeZO and Hizoo.
Llama3 an RTE
Liama3 on S5T2
Llama3 on WIC
MczO: 74.36%
McZO: 92.2%
1.4
MczO: 88%
0.8
HiZ00: 75. 09%
1.0
HIZOO: 93.46%
1.2
HIZOO: 89%
0.0
1.0
0.6
0.4
0.3
0.2
0.4
Forward NumberiX100)
Forward Number(x100)
Forward Poss Number(X100)
Figure 14: Loss curves on Llama3 between Me70 and HiZOO.
F DETAILS ABOUT MEMORY USAGE
Here we show the detailed numbers of memory profiling results Table 10. We did not turn on any advance memory-saving options, e.g., gradient checkpointing. We set the per-device batch size as 1 to test the minimum hardware requirement to run the model with specific optimization algorithms. We use Nvidia's nvidia smi command to monitor the GPU memory usage.
Table 10: Memory usage on the MultiRC (average tokens=400) dataset. Results of ICL and full- parameter tuning are from (Mallaa et al., 2023).
Method
zero-shod/MeZO(FT)
HiZOO(FT)
HiZO0-L(FT)
ICL
Adam(FT)
1.3B
IxA100 (4GB)
IxA100 (7GB)
IxA (4GB)
IxA100 (6GB)
IxA (27GB)
2.7B
1xA100 (7GB)
1xA100 (13GB)
1xA100 (8GB)
1xA100 (8GB)
1xA100 (55GB)
6.7B
IxA100 (14GB)
IxA100 (29GB)
IxA100 (15GB)
IxA100 (16GB)
2xA100 (156GB)
13B
IxA100 (26GB)
IxA (53GB)
IxA100 (29GB)
IxA100 (29GB)
4xA100 (316GB)
30B
IxA100 (58GB)
2xA100 (118GB)
IxA100 (64GB)
IxA100 (62GB)
8xA100 (633GB)
66B
2xA (128GB)
3xA100 (246GB)
2xA 100 (140GB)
2xA100 100 (134GB)
16xA100
G DETAILS ABOUT WALLCLOCK TIME EFFICIENCY
In this section, we measure the wallelock time efficiency of HiZOO compared to McZO and full- parameter fine-tuning (FT) with respect different model sizes. Due to the lack of NV-Link
23

---

Published as a conference paper at ICLR 2025
connectivity in our A100 GPUs, selected models that can be fully fine-tuned single A100 GPU for comparison. As shown in Table 11, HiZOO exhibits a longer per-step duration compared to McZO, within 50% margin. This result indicates that the primary overhead in hierarchical optimization methods lies the forward propagation process. Given that HiZOO involves an additional forward pass compared to Me70, the time per step increases by approximately 1.4 to 1.5 times.
In conclusion, the speedup factors derived from the forward pass step used in our comparisons between HiZOO and MeZO reflect the actual wallclock time efficiency improvements accurately.
Table 11: Wallclock time per step between MeZO, HiZOO and HiZOO-L. The increase in wallclock time per step for HiZOO compared to MeZO is less than 1.5 times across different model sizes. To avoid introducing additional overheads such as inter-GPU communication, results are measured on the same dataset (SST-2) and GPUs (80GB A100). with each result averaged over 100 steps. "BS"" refers to batch size. For the relatively smaller RoBERTa-laye model, we used a BS=64, while for models larger than IB parameters, used BS=16.
Model
RoBERTa--argg((50MM
Phi-2(2.7B)
LlBma3(8B)
OPT(13B)
MeZO
0.2092x(BS=64)
0.3011s(BS=16
1.1108<(BS=16) 0.7471s(BS=16)
HiZOO
0.3023<(BS=64)
0,4486s(BS=16)
1.1090s(BS=16)
(.5225)(BS=16)
HiZOO-L
0.3193s(BS=64)
0.4851s(BS=16)
1.1996s(BS=16)
L.6222(BS=16)
H
DETAILS ABOUT ABLATION EXPERIMENTS
H.1 INFLUENCE OF SMOOTH SCALE AND NUMBER OF ESTIMATION PER STEP
We conducted experiments SST-2, SST-5, MNLI datasets when fine-tuning RoBERTa-large to research the influence of smooth scale a; Figure 15 shows that the value of a mainly affects the convergence speed of the model. Additionally, the best value of a will vary between different datasets. Figure 16 shows that the influence of the number of estimation n per steps.
Roberts largeiprefiet on 5572
Roberts Inrpeiprefidi on 5575
Roberts larglppeffii on MNU
0.25
0y-1c-10
0-m1e-B
0--10
0.-1e-2
-116
0;:le12
0::1e-6
8
0.2
8:35
0.55
0.89
23000
4000e
53000
00633
100033
23000
40006
$3000
00000
100000
Steps
Steps
Stops
(b)
(c)
Figure 15: More experiments on influence of the value of Smooth scale a₁ on RoBERTa.
Function(a) Optimication Paths
Influence of estimation times per stop
6.5
30.6
n=4
60.
n616
55
>50
35
Oscmal Point
305
30
10.5--223
0.00
025 0.00
0.75
3.50
1.2
1.30
Figure 16: Influence of number of estimation per step. (left) 2D trajectories of gradient descent; (right) Corresponding loss curves.
24

---

Published as a conference paper at ICLR 2025
H.2 EXPERIMENTS ABOUT OMITTING [-E-] TERM IN EQ. EQUATION 4
We conducted experiments on SST-2 datasets using three methods to fine-tune RoBERTa-large to compare the difference between with [-E-]] term and without this term. Figure 17 shows that this term can make negligible influence.
Foll:Sto-lrreiieeetoo 5312
Nobt Notthifia-langeLLLatt on 5512
with
without
without
without
22-4
2003 (a)
2003
10000 20030
J00CE 40000 55993
0000
(b)
(c)
Figure 17: Experiment about the error generate by omitting the [-E-1] tenm in Eq equation 4. 'with' means holding the term and 'without' means omitting the term.
25


Reviews:
[{"review_id": "fS1SmZbJn0", "reviewer": "Reviewer_RuKr", "summary": "Motivated by the limited optimization effectiveness of zeroth-order optimizers in deep learning, this manuscript leverages a diagonal Hessian to enhance the optimization quality. Though introducing second-order curvature information to aid the optimization is standard, the manuscript is quite comprehensive: it builds methods step-by-step, provides a theoretical justification, and verifies the effectiveness across various model architectures and datasets.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The manuscript studies an interesting problem: efficient and effective zeroth-order fine-tuning. The manuscript has a clear motivation in the introduction part, with a thorough step-by-step explanation in Section 3.3. Extensive empirical studies consider evaluating both encoder-decoder and decoder-only neural architectures on the GLUE benchmark over several baseline methods. The evaluation uses the number of forward passes (which is very good) and discusses memory usage and time efficiency.\n* The manuscript has some preliminary convergence analysis.", "weaknesses": "1. The convergence analysis part can be strengthened. It would be great if the analysis could cover the case of MeZO (and other zeroth-order algorithms) and carefully explain the gain introduced by the diagonal Hessian. See some examples in [1], where the query complexity can be discussed and compared.\n2. The manuscript structure can be improved. E.g., some detailed derivates in Sec 3 can be simplified, and the definition of three test functions in Sec 3.5 can be moved to the main text.\n3. Extending to other LLM SFT datasets: it would be great if the manuscript could verify the effectiveness of HiZOO on some SFT datasets.\n\n### Reference\n[1] Stochastic Two Points Method for Deep Model Zeroth-order Optimization, https://arxiv.org/abs/2402.01621.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ColRxKmP8c", "reviewer": "Reviewer_MAUK", "summary": "This paper proposes the Hessian-informed Zeroth-Order Optimizer (HiZOO), which achieves faster convergence than traditional zeroth-order SGD (specifically, MeZO) in LLM fine-tuning scenarios by leveraging Hessian information through a zeroth-order approach. Although HiZOO requires one additional forward pass compared to MeZO (totaling three forward passes), it demonstrates faster convergence in terms of the number of function calls. Furthermore, HiZOO outperforms MeZO in terms of model generalization for various tasks in LLM fine-tuning and the authors also provide convergence guarantee of their HiZOO method in theory.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The first approach that leverage the second-order information via zeroth-order construction for fine-tuning LLMs\n2. Illustration of HiZOO on some test functions with superior generalization ability of the method\n3. Convergence guarantee of HiZOO with their diagonal inverse Hessian estimator", "weaknesses": "1. The Hessian estimator from Equation (3) appears to estimate the Hessian matrix only if the matrix $\\Sigma$ is close to the inverse Hessian. \nHowever, according to Algorithm 1 in the paper, I saw that the matrix (actually, the vector) $\\Sigma_0$ is initialized simply as the identity matrix, which suggests that this estimator may not accurately estimate the (diagonal) inverse Hessian. Furthermore, since the initial value is merely an identity matrix, it is difficult to consider the subsequent estimated $\\Sigma_t$ values as accurate estimations of the diagonal inverse Hessian. \nInstead, it seems more likely that HiZOO are estimating something positive definite.\n\n---\n\n2. The theoretical result seems a bit unusual. \nTypically, convergence results are derived as bounds on the norm of the true gradient (i.e. the gradient evaluated on the entire training dataset) rather than the norm of the stochastic gradient. \nConsequently, Theorem 3.2, which presents the current convergence result, is very confusing and may need to be revised for clarity.\n\n---\n\n3. In addition to the second concern, the convergence rate in Theorem 3.2 is on par with first-order methods. However, given that second-order information, such as the diagonal Hessian, is utilized—even if constructed solely from function values—there should ideally be an advantage in convergence compared to existing methods like MeZO or other vanilla SGD-based approaches. This expected benefit, however, does not seem apparent. Moreover, $\\max_t \\mathrm{Tr}(\\Sigma_t)$ would also depend on $d$. \nFor example, given that Algorithm 1 initializes $\\Sigma_0$ as the identity matrix, the quantity $\\mathrm{Tr}(\\Sigma_0)$ becomes $d$, so $\\max_t \\mathrm{Tr}(\\Sigma_t)$ is at least on the order of the parameter dimension $d$. \nIf the order of $\\max_t \\mathrm{Tr}(\\Sigma_t)$ exceeds $d$ (e.g., something like $d\\sqrt{d}$), the convergence rate would actually be slower than zeroth-order vanilla SGD (ZO-SGD), which undermines the theoretical contributions of the proposed algorithm.\n(To the best of my knowledge, the convergence of ZO-SGD can be derived faster than $O(d)$). Also, importantly, the advantage of leveraging second-order information should be apparent in terms of theory.\n\n---\n\n4. Looking at the comparison on the test function, it appears to be a comparison of just optimization from scratch rather than fine-tuning. \nI'm curious about the hyperparameter settings, as it seems to perform better than Adam. \nIf this advantage holds, there should also be experiments demonstrating the use of zeroth-order methods not only in fine-tuning but also in pre-training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dVGW7TrLh2", "reviewer": "Reviewer_Wjn7", "summary": "The work presents a novel optimization method called HiZOO, designed for fine-tuning large language models (LLMs). HiZOO addresses this by incorporating a diagonal Hessian estimation through an additional forward pass, allowing it to act as a pre-conditioner that adjusts updates based on parameter curvatures. This approach reduces training steps and improves accuracy while maintaining efficient memory use, scaling well even for models with billions of parameters. The authors also propose HiZOO-L, a low-rank version that significantly cuts down on memory cost while preserving performance.", "questions": "Please see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The introduction of HiZOO, a Hessian-informed zeroth-order optimizer, is original.\n2. The proposed low-rank variant, HiZOO-L, which reduces memory overhead, demonstrates a approach to solving memory constraints while maintaining optimization quality.\n3. The paper is well-organized, with clear sections outlining the motivation.", "weaknesses": "1. The code implementation assumes that  u_i Hadamard product u_i is treated as a diagonal matrix. However, this is incorrect as \\( u_i u_i^T \\) is an outer product in the algorithm description in the paper resulting in a rank-1 matrix. This assumption could introduce inconsistencies or inaccuracies in the Hessian estimation and parameter updates. \n\n2. The value for `Hessian_smooth` in the implementation is set to \\( 1e^{-6} \\), which seems quite small. This could imply that the contribution of the Hessian information is not significant enough to make a substantial impact on convergence or stability. \n\n3. Tables 1 and 3 present results for a range of NLP tasks, but no generation tasks are included. This limits the understanding of HiZOO’s performance in broader language modeling applications where generative capabilities are critical.\n\n4. While Table 5 reports promising results on SST2, Figures 12 to 14 reveal that HiZOO demonstrates poor training performance and even training failure in some cases. This discrepancy raises concerns about the stability and reliability of the optimizer under different conditions.\n\n5. Line 9 in Algorithm 2 is unclear. Calculating \\( R^{-1} \\) and \\( C^{-1} \\) involves significant computational overhead, making this step inefficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)