Title: Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Zeroth-Order Optimization

Parameter-Efficient Fine Tuning
Abstract: While fine-tuning large language models (LLMs) for specific tasks often yields impressive results, it comes at the cost of memory inefficiency due to back-propagation in gradient-based training. Memory-efficient Zeroth-order (MeZO) optimizers, recently proposed to address this issue, only require forward passes during training, making them more memory-friendly. However, compared with exact gradients, ZO-based gradients usually exhibit an estimation error, which can significantly hurt the optimization process, leading to slower convergence and suboptimal solutions. In addition, we find that the estimation error will hurt more when adding to large weights instead of small weights. Based on this observation, this paper introduces Sparse MeZO, a novel memory-efficient zeroth-order optimization approach that applies ZO only to a carefully chosen subset of parameters. We propose a simple yet effective parameter selection scheme that yields significant performance gains with Sparse-MeZO. Additionally, we develop a memory-optimized implementation for sparse masking, ensuring the algorithm requires only inference-level memory consumption, allowing Sparse-MeZO to fine-tune LLaMA-30b on a single A100 GPU. Experimental results illustrate that Sparse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead. For example, it achieves a 9% absolute accuracy improvement and 3.5x speedup over MeZO on the RTE task.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the memory inefficiency of fine-tuning large language models. It identifies that while Zeroth-Order (ZO) optimizers like MeZO save memory by avoiding back-propagation, they introduce gradient estimation \"noise\" that harms performance, especially when applied to large-magnitude weights.\n\nTo solve this, the paper proposes Sparse MeZO (S-MeZO), a method that selectively applies updates only to the model's smaller-magnitude weights. This approach is more resilient to noise, which allows for larger learning rates, leading to faster convergence and better accuracy.\n\nA key contribution is a memory-efficient implementation that calculates the necessary sparse mask on-the-fly during the forward pass, keeping memory usage at the same level as inference. This innovation enabled fine-tuning a 30-billion parameter model on a single A100 GPU.\n\nExperiments conducted on LLaMA, OPT, and Mistral models show that S-MeZO consistently outperforms the original MeZO. For example, on the RTE task, it achieved a 9% absolute accuracy improvement and a 3.5x speedup.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Significant Practical Impact and Accessibility: The paper's most significant contribution is making the fine-tuning of extremely large models more accessible. By developing a memory-efficient implementation that calculates sparse masks on-the-fly, S-MeZO successfully keeps memory usage at the level of inference. This innovation enabled the fine-tuning of a LLaMA-30b model on a single A100 GPU, a task that is typically infeasible with standard methods. This dramatically lowers the hardware barrier for LLM research and development.\n\n2. Novel and Insightful Core Idea: The paper is founded on a highly original empirical observation: the noise inherent in Zeroth-Order (ZO) gradient estimates is more detrimental to large-magnitude weights than to small-magnitude ones. This insight is novel and provides a clear, well-motivated basis for the proposed S-MeZO algorithm, which selectively optimizes these more noise-resilient small weights. This approach offers a new perspective on mitigating noise in ZO optimization.\n\n3. A notable strength of this paper, setting it apart from some other works focused on ZO optimization, is the inclusion of baselines that use back-propagation to calculate gradients. \n\nWeaknesses:\n1. Since the parameter selection is based on the smallest weights, the benchmark could be strengthened by including a comparison with the opposite strategy: selecting and updating only the largest-magnitude weights. This would serve as a valuable ablation study to more directly validate the paper's core hypothesis.\n\n2. A potential area for strengthening the experimental comparison would be to include another recent and relevant method in sparse ZO optimization. For instance, comparing S-MeZO against SensZOQ would provide a more comprehensive picture of its performance within this specific sub-field.\n\nQUESTIONS:\nHave the authors explored the upper limits of the sparsity ratio for this method? For instance, what is the highest sparsity tested, and does performance continue to improve or begin to degrade at very high levels of sparsity?\n\nLIMITATIONS:\nThe authors have acknowledged the primary performance-related limitation of their work, which is the remaining performance gap between S-MeZO and first-order fine-tuning methods. This transparency is commendable. However, the discussion of limitations could be expanded to make the paper even stronger: Broader Societal Impact: While the work itself does not have a direct negative impact, it contributes to a line of research that makes powerful LLM fine-tuning more accessible and efficient. The authors could briefly acknowledge the dual-use nature of such technology; lowering the barrier to entry can empower more users for positive applications, but it can also potentially make it easier for malicious actors to create fine-tuned models for generating disinformation or other harmful content. A short discussion of this broader context is standard for LLM research and would demonstrate a more comprehensive consideration of the work's potential impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper present sparse-Mezo, an algorithm based on Mezo with the observation that optimizing small weights is more helpful than optimizing large weights for zeroth order method. Then the algorithm is a simple modification of Mezo: in each forward pass, the algorithm identifies small weights in each layer and only update these weights. The paper show experiments that support the observation and improve Mezo on fine-tuning tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper provides a new observation for zeroth order methods that small weights have bigger impacts on training than large weights. This is very interesting and somewhat counter-intuitive. I think this should be studied more in the future.\n- The experimental results are quite strong against vanilla mezo and support the hypothesis.\n\n**Weaknesses**\n- The experiments could be more comprehensive. Some of the datasets are missing from some tables (for example, table 2 and others in the appendix). Please also add other baselines such as ZO-SGD to table 3 and where it’s relevant.\n- The algorithm introduces a sparsity parameter and it’s unclear how to set this parameter other than tuning\n- As acknowledged by the authors, there is still a significant gap between this method and gradient methods\n- The theorem in the appendix only applies for a fixed masking, not the masking by threshold used in the algorithm.\n\nQUESTIONS:\n- Please add the experiment results I mentioned in the appendix\n- For the completeness and self-containment of the paper, please also for each experiment provide a description of the set up (for example, section 4.3 is unclear)\n- Regarding the new observation: When training on top 20% of the largest weight vs the bottom 20%, how are the step sizes determined for each case and what are the impacts on the stability of the training and the performance of the model?\n- Is the disproportionate impact of the weights also observed when we train the model by first order method instead of zeroth order method?\n\nLIMITATIONS:\nAdequately addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Sparse-MeZO, a memory-efficient zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Instead of updating all parameters, Sparse-MeZO selectively optimizes a subset by applying a sparse mask to target low-magnitude, noise-tolerant weights. Built upon the MeZO framework, this approach reduces gradient estimation noise, accelerates convergence, and lowers memory consumption. Experimental results show that Sparse-MeZO not only outperforms MeZO in convergence speed and final performance but also enables the fine-tuning of large models like LLaMA-30B using limited hardware, such as a single A100 GPU.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n* Incorporating sparsity and MeZO is an interesting direction for performance improvement.\n* The paper is structured logically, making it easy to follow the motivation and methodology.\n\n**Weaknesses:**\n* The paper lacks discussion and comparison with some important ZO works in sparsity, like [1]. Moreover, seems the method works due to gradient estimation with less noise, so it’s helpful to compare with some variance-reduction works of ZO, like [2] and [3].  \n* There are no specific numbers regarding the computational overhead introduced by threshold calculation, and dynamic masking and perturbation layer by layer in the forward pass.\n* The overall contribution is incremental as the work adds common magnitude-based sparse steps on top of the MeZO optimization algorithm.\n* It would be better to have some evaluation on more challenging task, like MMLU and MT-Bench, for example, using the settings in [4].\n\n[1] Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity\n\n[2] Zeroth-order stochastic variance reduction for nonconvex optimization\n\n[3] Zeroth-order optimization with trajectory-informed derivative estimation\n\n[4] Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning\n\nQUESTIONS:\n* For the setting of the threshold, set its value related to the sparsity, e.g., 80% sparsity -> 80th percentile. Are these only empirical findings? It's better to have some ablation studies on the selection of the threshold.\n* Some parts of the appendix are not referred to in the main paper and are without a textual description.\n* Is the threshold computed every iteration? As the learning rate for zo is small, and the update per iteration is also small. Therefore, can you compute it intermittently?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to improve the convergence of zero-order optimizers by only applying the perturbation to smaller weights. The paper begins by presenting empirical observations on how the magnitude of weights in MeZO updates influences loss convergence. Based on this, the authors propose a practical solution: masking the largest weights during perturbation application. They demonstrate that this approach significantly enhances the convergence process and downstream performance in MeZO.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n* **s1:** The paper is well-motivated and written, clearly balanced between motivation, methods, and empirical evaluation. The method is simple and does not require a complex procedure on top of existing zero-order optimization processes.\n* **s2:** The experimental results are compelling, showing an average improvement of 3.7 points on the SuperGLUE tasks over vanilla MeZO without impact on the memory consumption or convergence speed.\n\n**Weaknesses**\n\n* **w1:** I am not fully convinced by the analysis presented in lines 147-159. As I understand it, the authors compute gradients on a set of 16 data points and then observe whether applying a gradient step in that direction, but starting from 16 entirely independent points, reduces the loss. The observation that there's roughly a 50% chance it won't be the case doesn't completely surprise me. It seems possible to me that a gradient computed on one set of 16 points aligns with a gradient computed on another, completely independent set of 16 points. I would like to know if similar results would hold for supervised fine-tuning (SFT). In particular, to include a comparison in Figure 2(b) with SFT.\n* **w2:** The results regarding the convergence rate in Section 4.4 and Figure 3 only compare convergence curves for MeZO and Sparse-MeZO, but it would be relevant to add the comparison with SFT here.\n\nQUESTIONS:\n* “Impressive results” line 1 seems like a superlative statement\n* Line 165, how are the percentages chosen to define small and large values? Why set the threshold to 20%?\n* Lines 222-225, I am not sure how this paragraph helps with the argumentation. According to the author, this method is not aligned with the goal of the paper and is not discussed anywhere else in the paper. I would suggest simply removing this paragraph for clarity.\n* It would be helpful to report the standard variations on top of the mean across the three runs for the results in at least Table 1. This could help capture the stability of the approach.\n\nLIMITATIONS:\nThe authors have discussed the limitations in the last section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}