Title: Harmony in Divergence: Towards Fast, Accurate, and Memory-efficient Zeroth-order LLM Fine-tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Memory-efficient fine-tuning; Zeroth-order optimization
Abstract: Large language models (LLMs) excel across various tasks, but standard first-order (FO) fine-tuning demands considerable memory, significantly limiting real-world deployment. Recently, zeroth-order (ZO) optimization stood out as a promising memory-efficient training paradigm, avoiding backward passes and relying solely on forward passes for gradient estimation, making it attractive for resource-constrained scenarios. However, ZO method lags far behind FO method in both convergence speed and accuracy. To bridge the gap, we introduce a novel layer-wise divergence analysis that uncovers the distinct update pattern of FO and ZO optimization. Aiming to resemble the learning capacity of FO method from the findings, we propose \textbf{Di}vergence-driven \textbf{Z}eroth-\textbf{O}rder (\textbf{DiZO}) optimization. DiZO conducts divergence-driven layer adaptation by incorporating projections to ZO updates, generating diverse-magnitude updates precisely scaled to layer-wise individual optimization needs. Our results demonstrate that DiZO significantly reduces the needed iterations for convergence without sacrificing throughput, cutting training GPU hours by up to 48\% on various datasets. Moreover, DiZO consistently outperforms the representative ZO baselines in fine-tuning RoBERTa-large, OPT-series, and Llama-series on downstream tasks and, in some cases, even surpasses memory-intensive FO fine-tuning. Our code is released at \url{https://github.com/Skilteee/DiZO}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Divergence-driven Zeroth-Order (DiZO) fine-tuning for large language models.\n\n1. Observation: First-order (FO) fine-tuning naturally produces different update magnitudes across layers, while existing zeroth-order (ZO) methods apply nearly uniform random updates, slowing convergence. \n\n2. Method: DiZO keeps memory-light ZO steps but periodically projects the updates toward the pretrained weights (“anchor”) for selected layers (mainly Query/Value matrices), recreating FO-like layer-specific behavior without any backward pass.\n\n3. Results: On RoBERTa-large, OPT, and Llama, DiZO cuts iterations by up to 50 % and GPU hours by up to 48 %, sometimes even surpassing full FO fine-tuning, while requiring only ~17 % extra stored parameters.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper tackles a central challenge in Large Language Model (LLM) fine-tuning: how to improve the convergence speed and accuracy of zeroth-order (ZO) optimization while maintaining its memory efficiency. \n\n2. The paper introduces a novel \"layer-wise divergence analysis\" revealing that FO methods make diverse, fine-grained layer updates, unlike ZO's uniform updates. Building on this, DiZO proposes \"divergence-driven layer adaptation\" using learnable, anchor-based projections to scale ZO updates for each layer, mimicking FO's adaptive capabilities without backpropagation's memory cost. Crucially, these projections are learned via a ZO-based method for end-to-end memory efficiency, with \"Re-initialization\" and \"Projection Clipping\" enhancing stability.\n\nWeaknesses:\n\n1. Why is the checkpoint chosen as the anchor the original pretrained model, rather than a model that has already been partially fine-tuned on the task data?\n\n2. Even when the anchor is limited to the Query and Value matrices, roughly 16 % of the pretrained parameters must still reside in memory. While this footprint is acceptable for cloud servers, it can be prohibitive for devices with extremely limited VRAM or other edge hardware. The authors do not explore weight quantization, pruning, or other anchor-compression techniques to further reduce this overhead.\n\n3. There are now several Sparse MeZO works that achieve competitive—or even higher—accuracy. It would be valuable for the authors to include a direct comparison with these sparse approaches.\n\nQUESTIONS:\nGood work, but it requires additional memory overhead. Meanwhile, several sparse ZO approaches have achieved notable success—for instance, “Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning.” Could the present paper demonstrate that it outperforms such methods? With comparable accuracy, users will naturally favor the option that consumes less memory.  If the authors can demonstrate that their method outperforms the sparse ZO approaches, I would be inclined to raise my score.  The authors could demonstrate this by presenting evidence on either accuracy or convergence speed.\n\nLIMITATIONS:\n1. The theoretical analysis is insufficient: the paper does not clearly explain why the pretrained LLM is chosen as the anchor, nor why only **Q** and **V** are projected instead of also including **K** and **O**.\n2. The performance gains come at the cost of increased memory usage.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies that standard ZO fine-tuning methods (eg MeZO) apply fixed-magnitude updates to each layer of the network, whereas first-order methods compute gradients of varying magnitudes for each layer of the network. As such, they propose a way to rescale the updates for each layer in the network. They include a convergence analysis and experiments on medium-sized BERT models as well as larger autoregressive language models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- It is worthwhile to investigate the gap between zeroth and first order training methods in order to improve the former to be as efficient as the latter.\n- There are thorough experiments evaluating the efficacy of this method\n\nWeaknesses:\n- **Lacks comparison to layerwise learning rates**: The method amounts to using a layerwise learning rate for each layer of the network. The authors acknowledge this as well, but they do not compare to existing layerwise LR methods (the most popular of which is LARS). These methods are expensive to naively implement on top of ZO methods, but I think there are straightforward ZO approximations to be made. At the very least, the experiments should include ablations and comparisons against the more expensive but standard way to impose layerwise learning rates.\n\n- **Performance benefit is marginal**: The performance benefit in the setting where the benefits of ZO are apparent (ie at the ~7B model scale) is marginal. There is still a substantial gap to the first order methods, suggesting that the original premise of the paper -- that fixed-magnitude updates to each layer are the culprit for the performance gap -- is not actually true. This problem is made worse by the fact that the authors do not do multiple trials of fine-tuning, which is standard in FT literature, given the noise present when optimizing on small datasets.\n\n- **Theoretical analysis is poor**: The theoretical analysis is poor and provides little to no insight. The technique is to assume a bounded update at each timestep and then to say that the variance of the gradient estimate is reduced. A better theoretical justification for this method would be to argue rigorously that the gradient update $R^{(\\ell)}$ will be small enough when using the additional tricks in DiZO -- this is a more complex analysis and is likely not true in general. As it stands, the theory adds nothing to my understanding of the paper. A good theory section would actually advocate for why layerwise LRs are necessary.\n\n- **Requires hacky tricks**: The two tricks used to make DiZO work, re-initializing and clipping, are pretty hacky, and I am not confident that they will work in new + unseen settings, especially because clipping requires setting an additional hyperparameter.\n\nQUESTIONS:\n1. Can you explain why the convergence analysis is significant? It actually does not contain a convergence result in the first place and only contains a bound on the variance of the gradient. I would raise my score if the theory can be improved to more rigorously justify the tricks in DiZO -- for example, if the theoretical result provides clear guidance on how to set the clipping hyperparameters.\n\n2. Why did you not compare against any layerwise learning rate methods?\n\n3. Figure 1 is missing a numerical y-axis, which makes it hard to determine if the plotted differences between FO and ZO methods is actually significant in scale. It is also not clear why you only plotted the Q, K, V, O values out of all the parameters in the network.\n\n4. Why do you call this a projection? Projections of gradient estimates are usually something like a preconditioner. This is just a scalar multiplied to the update on each layer. I found the writing extremely confusing for a very simple point.\n\nLIMITATIONS:\nThe limitations only discuss issues with the theoretical section and do not mention the fundamental issues that I raised above. It would be good if the authors acknowledge the limited empirical efficiency of their method, the introduction of an additional hyperparameter, and the overall uncertainty around their hypothesis concerning the ZO-FO gap when fine-tuning models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces Divergence driven Zeroth Order optimization method. It begins with an analysis on layer-wise divergence between FO and ZO methods. Motivated by this, it proposes anchor-based, learnable projections to adapt ZO updates layer-wise. Experiments across different models show that DiZO outperforms ZO baselines on both full parameter and PEFT settings with faster convergence and improved accuracy.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The analysis comparing ZO and FO optimization is clearly presented and provides a strong foundation for motivating the proposed method.
2. The proposed DiZO is memory-efficient and achieves performance on par with FO methods.
3. The experiments are thorough, covering both autoencoding and autoregressive models, and are complemented by detailed analyses of memory usage and computational efficiency.

Weaknesses:
1. On larger autoregressive models, the performance gap between DiZO and FT seems bigger than that observed in smaller autoencoding models. It would be helpful if the authors could provide insights into the cause of this difference.
2. The method introduces additional hyperparameters such as projection update interval and clipping range. The paper provides limited discussion on how sensitive performance is to these design choices.

QUESTIONS:
Minor Typo: Line 265 is missing a period at the end of the sentence.

Please see weaknesses for questions.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes DiZO, a gradient-free optimization method for efficient fine-tuning of large language models (LLMs). The authors begin with a layer-wise divergence analysis, observing that first-order (FO) optimization produces updates with varying magnitudes across layers, while zeroth-order (ZO) optimization tends to apply uniform-magnitude updates. This discrepancy is identified as a key limitation of ZO methods in terms of convergence and performance. To address this, DiZO introduces a projection mechanism guided by the divergence between the current model and the initialization (used as an anchor), enabling each layer’s updates to be geometrically constrained and adaptively scaled. This allows ZO to mimic FO-like learning behavior while retaining its memory efficiency. Experiments across multiple models (RoBERTa, OPT, LLaMA) and tasks demonstrate that DiZO significantly outperforms prior ZO baselines and, in some cases, even surpasses FO methods. The paper further provides a convergence analysis of the projection update under effective dimension assumptions and shows compatibility with parameter-efficient fine-tuning (PEFT) methods such as LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.\tQuality: The method is technically sound, with a clear structure and practical implementability. The proposed algorithm is empirically well validated across diverse models and tasks, with convincing performance and convergence gains.\n\n2.\tClarity: The paper is clearly written with rigorous derivations, well-designed figures and ablations, and a coherent logical flow that supports both understanding and reproducibility.\n\n3.\tSignificance: ZO optimization is increasingly relevant in resource-constrained deployment scenarios. By improving both convergence speed and accuracy, this work enhances the practicality of ZO-based fine-tuning for LLMs.\n\n4.\tOriginality: The use of layer-wise divergence to construct projection-based updates that emulate FO behavior under a ZO framework is novel and insightful.\n\nWeaknesses\n\n1.\tWhile the method is thoroughly evaluated on standard NLP tasks, it lacks analysis of failure cases or known vulnerabilities of ZO methods.\n\n2.\tThe periodic projection mechanism and its associated hyperparameter settings are not thoroughly analyzed for robustness and may be sensitive to the choice of tasks.\n\nQUESTIONS:\n1.\tIs using the pretrained model as the anchor empirically optimal? Are there theoretical motivations or data-driven alternatives for selecting a more suitable anchor?\n\n2.\tCurrently, DiZO updates the projection parameter γ at fixed intervals k. Could a task-aware or training-state-aware dynamic adjustment mechanism be considered? Such an approach might improve robustness and generalizability across tasks.\n\n3.\tThe current experiments focus on NLP classification and generation tasks, while the conclusion mentions future applications of DiZO to vision models. Have the authors observed any consistency or transferability of the projection parameter γ across tasks or model types?\n\nIf these questions are addressed, my assessment of the paper’s theoretical contribution and generalizability may improve.\nThe method assumes full access to pretrained model parameters, which may be infeasible in privacy-constrained or adapter-only deployment scenarios. In addition, although DiZO is designed for memory efficiency, its real-world performance on edge hardware or production environments has not been evaluated. Further discussion or experiments in such contexts would strengthen the paper’s practical relevance.\n\nLIMITATIONS:\nThe method assumes full access to pretrained model parameters, which may be infeasible in privacy-constrained or adapter-only deployment scenarios. In addition, although DiZO is designed for memory efficiency, its real-world performance on edge hardware or production environments has not been evaluated. Further discussion or experiments in such contexts would strengthen the paper’s practical relevance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}