Title: PaZO: Preconditioned Accelerated Zeroth-Order Optimization for Fine-Tuning LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: zeroth-order optimization

fine-tuning LLMs

preconditioner
Abstract: This paper introduces PaZO, a preconditioned accelerated zeroth-order optimization algorithm for fine-tuning large language models (LLMs).      First, we theoretically demonstrate the necessity of preconditioning in zeroth-order optimization, proving that zeroth-order stochastic gradient descent (ZO-SGD) alone fails to achieve the ideal convergence rate.     Building on this, we propose a Preconditioned Simultaneous Perturbation Stochastic Approximation (PSPSA) and theoretical version of PaZO, and demonstrate that setting the order of preconditioner as $-1/2$ in PSPSA yields the improved convergence rate for PaZO.     Moreover, we design a practical version of PaZO that stabilizes training via diagonal Hessian estimate and moving average technique.  Extensive experiments on diverse downstream tasks with models like RoBERTa-large and OPT show PaZO’s effectiveness.  Compared to other zeroth-order baselines, PaZO achieves better performance across models and tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the role of preconditioning in zeroth-order optimization for memory-efficient fine-tuning of large language models and introduces PaZO. The authors define a general PSPSA estimator that perturbs parameters with preconditioner $H^{-\\alpha}$ and prove that only choosing $\\alpha = 1/2$ yields the optimal convergence rate. The authors then proposes a practical algorithm named PaZO that utilizes diagonal Hessian estimator via three forward-pass perturbations. Empirical evaluation across multiple downstream tasks and models demostrate the effectiveness of the proposed algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The authors give the answer to an important question regarding the necessity of preconditioning in ZO and identify the optimal order in preconditioner.\n\n2. The three-point perturbation scheme combined with gradient-based correction and periodic reset is a novel, lightweight way to incorporate second-order information without prohibitive memory.\n\n3. Extensive empirical evaluation on both masked (RoBERTa-large) and generative (OPT-1.3B) models across a diverse suite of NLP tasks, demonstrating robustness of PaZO.\n\nWeakness:\n1. There are several formulation mistakes in estimating the diagonal Hessian. The RHS of Equation (11) should be $diag(H)$. The LHS of Equation (13) should be $\\ell_{+} + \\ell_{-} - 2\\ell$.\n\n2. Some hyperparameter choices (e.g., reset period $T_0$ schedule) are introduced without intuitive justification.\n\nQUESTIONS:\n1. While estimating the diagonal Hessian matrix, 1) why do we need to incorporate $diag( \\tilde{g} \\circ  \\tilde{g})$ as a correction item to integrate local first-order estimated information into $\\Sigma_{t-1}$ through moving average mechanism? 2) why $\\tilde{\\Sigma}$ is computed through $((1-\\beta_1) \\Sigma_{t-1}^2 + \\beta_1 \\cdot diag( \\tilde{g} \\circ  \\tilde{g}))^{1/2}$, not $(1-\\beta_1) \\Sigma_{t-1} + \\beta_1 diag( \\tilde{g} \\circ  \\tilde{g})$. 3) In theory, such correction is not applied in estimating the term $\\mathcal{I}$ while computing $\\Sigma_t$, which makes Equation. (14) no longer an unbiased estimation to the diagonal Hessian. 4) In practice, the authors conducted an ablation experiment on studying the influence of parameter choice of $\\beta_1$, why too small choice of $\\beta_1$ (including 0) result  in NaN. From my understanding, setting $\\beta_1$ to be zero will make PaZO similar as that in HiZOO. In this ablation study, I can't figure out the real effect of incorporating the estimated gradient as a correction.\n\n2. Apart from the correction term, another difference between PaZO and HiZOO is the periodical reset mechanism for diagonal Hessian estimation. However, there is no ablation study design for this mechanism. Why this this step is necessary?\n\nLIMITATIONS:\nyes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a preconditioned accelerated zeroth-order method, PaZO, for fine-tuning LLMs. Theoretically, the paper justifies the use of preconditioners for ZO methods by showing the convergence rate gap between with and without Hessian preconditioning. Motivated by the theory, the paper proposes a practical realization of PaZO using diagonal Hessian estimate and moving average. Numerical experiments on masked LLMs and generative LLMs demonstrate the memory and timestep efficiency of PaZO compared to first-order methods and ZO methods without preconditioning acceleration.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n\n1. Reducing the memory usage for LLM fine-tuning is an important topic and has significance in practice. \n\n2. The paper is properly framed under questions A, B and C. It provides their method along with the motivating theory. \n\nWeaknesses: \n\nWhile the paper is well structured, its content leaves many puzzles lacking explanations. See Questions.\n\nQUESTIONS:\n1. In Theorem 3.5, can you provide more explanation on the relation between $D_\\alpha$ and $d^2$? In particular, how does the lower bound scale with $\\alpha$? While the paper claims that $\\alpha=0$ leads to strict inequality and is thus suboptimal, it would be better to have more explicit characterizations. \n\n2. In Theorem 3.8, how large is $\\epsilon_0$? It seems to be treated as small so that $Err$ represents high-order errors, but how is it guaranteed? \n\n3. In Appendix C.6, Table 6, why are $\\beta_1$ and $\\beta_2$ so small (around $10^{-8}$)? This would be a particular issue given that many LLMs use low precision (e.g., bfloat16) during training to save memory, and $10^{-8}$ seems to be too small. This makes the results less convincing. \n\n4. How much more memory overhead does PaZO have compared to MeZO? Figure 1 uses a log scale, which makes such a comparison less transparent. \n\n5. Is it possible to plot the validation loss vs steps? It would better illustrate the efficiency of PaZO compared to other methods. \n\n6. What are the standard deviations in Table 6? From the numbers, PaZO seems to have marginal improvement over HiZOO, especially in the LoRA and prefix settings.\n\nLIMITATIONS:\n1. Sensitivity to $\\beta_1$ and $\\beta_2$ should be mentioned in the main text, as they are important hyperparameters and are abnormally small.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces PaZO, a preconditioned zeroth-order optimization algorithm designed for the memory-efficient fine-tuning of LLMs. The authors provide a solid theoretical foundation, demonstrating the necessity of preconditioning in ZO optimization to achieve faster convergence. They derive the optimal preconditioner and develop a practical algorithm that estimates the diagonal Hessian using only forward passes. The experiments show that PaZO outperforms existing ZO methods like MeZO and HiZOO and achieves performance comparable to first-order fine-tuning while using significantly less memory.\n\nSTRENGTHS AND WEAKNESSES:\nPros:\n\n- The practical PaZO algorithm is an application of the theoretical insights. The design choices, such as estimating the diagonal Hessian to maintain O(d) memory and using moving averages and resets for stability, are well-justified and make the algorithm practical for fine-tuning modern LLMs.\n\n- The core theoretical strength is not just that the authors prove preconditioning is necessary, but that they derive the optimal form of the preconditioner from a formal convergence analysis. The technical argument in Theorem 3.5 is particularly elegant. \n\n- The paper's main strength lies in its rigorous theoretical analysis. It methodically answers three key questions about preconditioning in ZO optimization. The convergence analysis in Theorems 3.5 and 3.8 provides a clear, formal justification for why preconditioning is necessary and demonstrates that using the inverse square root of the Hessian (H^{-1/2}) as a preconditioner is optimal for achieving the fastest convergence rate. This is a significant theoretical contribution.\n\nCons:\n\n- The authors are transparent about PaZO's per-step time, noting it is about 1.5x slower than MeZO due to an additional forward pass (Figure 2). They argue this is offset by a better convergence rate. However, the analysis would be much stronger with a direct comparison of the total time (or number of steps) required to reach a target accuracy. A plot of accuracy vs. wall-clock time for PaZO against the baselines would more directly substantiate the claim of overall faster training.\n\n- While the theoretical framework is novel, some components of the practical algorithm in Section 4 are based on existing techniques. For example, estimating the Hessian diagonal via forward passes and using moving averages to stabilize training are known methods. The main novelty is the specific formulation justified by the theory for alpha = 1/2, which is a strong contribution, but the practical implementation itself is a clever combination of established components rather than a completely new mechanism.\n\nQUESTIONS:\nTo strengthen the claims about \"convergence speed\", could you provide practical \"time-to-accuracy\" results? For example, on a representative task like SST-2, how much wall-clock time does it take for PaZO, MeZO, and FT to reach 95% of their respective final accuracies?\n\nLIMITATIONS:\nI did not see potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces PaZO, a preconditioned accelerated zeroth-order optimization algorithm designed for fine-tuning large language models (LLMs) without backpropagation. The authors address three core challenges: (A) whether preconditioning is necessary in zeroth-order optimization, (B) how to optimally apply preconditioning for fastest convergence, and (C) how to practically estimate Hessian information to improve performance. They theoretically prove that zeroth-order stochastic gradient descent (ZO-SGD) alone is suboptimal and that using a preconditioner of order −1/2 yields the best convergence rate. PaZO incorporates this insight into a practical algorithm using diagonal Hessian estimates and moving averages for stability. Extensive experiments on models like RoBERTa-large and OPT-1.3B across various tasks and parameter-efficient fine-tuning (PEFT) methods demonstrate that PaZO outperforms existing zeroth-order methods (e.g., MeZO, HiZOO) in both accuracy and memory efficiency, achieving state-of-the-art results with manageable computational overhead.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well written, and tackles an important problem: efficient LLM fine-tuning\n- The authors rigorously prove that standard zeroth-order stochastic gradient descent (ZO-SGD) cannot achieve the optimal convergence rate, establishing the necessity of preconditioning in zeroth-order optimization. They introduce a generalized framework—Preconditioned Simultaneous Perturbation Stochastic Approximation (PSPSA)—and derive convergence bounds for both quadratic and general smooth functions. A key strength is the identification of the optimal preconditioning order (α = 1/2), which is shown to yield the fastest convergence rate. These results are supported by detailed proofs and assumptions, including extensions to scenarios with approximate Hessians.\n- The authors present a practical algorithm, PaZO, and demonstrate its effectiveness empirically across a wide range of downstream tasks,  model architectures, and fine-tuning strategies.\n\nWeaknesses:\n- The practical version of PaZO is not so practical. It strongly resembles HiZOO, with slight differences in estimating the Hessian, and introduces extra hyperparameters ( $T_0$, $\\beta_1$ and $\\beta_2$) that are not properly studied. The choice of $T_0$ is not properly ablated, and Table 6 in the appendix shows that the PaZO is highly sensetive to the choices of $\\beta_1$ and $\\beta_2$. \n- The authors claim that PaZO maintains the memory efficiency of HiZOO, but Figure 1 does not show the memory overhead of HiZOO.\n- The performance improvement of PaZO over HiZOO is only demonstrated for OPT (and not RoBERTA). For OPT, the improvement does not seem to be significant.\n\nQUESTIONS:\n- How does PaZO compare to MeZO and FO methods on more recent LLMs, such as Llama 3?\n- Can the authors show learning curves of the experiments reported in Tables 1 and 2, it would be curious to see how fast each method converges.\n\nLIMITATIONS:\nThe authors did not properly asses the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-06"}