Title: Bilevel ZOFO: Efficient LLM Fine-Tuning and Meta-Training
Venue: NeurIPS 2025
Accepted: yes
Keywords: arge language model

Multi-Task Learning

Zeroth-order Fine-tuning

Parameter Efficient Fine-tuning
Abstract: Fine-tuning pre-trained Large Language Models (LLMs) for downstream tasks using First-Order (FO) optimizers presents significant computational challenges. Parameter-Efficient Fine-Tuning~(PEFT) methods have been proposed to address these challenges by freezing most model parameters and training only a small subset. While PEFT is efficient, it may not outperform full fine-tuning when high task-specific performance is required.
Zeroth-Order (ZO) methods offer an alternative for fine-tuning the entire pre-trained model by approximating gradients using only the forward pass, thus eliminating the computational burden of back-propagation,
% in first-order methods, 
but they converge painfully slowly and are very sensitive to the choice of task prompts.
We bridge these worlds with Bilevel‑ZOFO, a penalty‑based bilevel formulation that treats adapter parameters as a lower‑level learner coupled to an upper‑level ZO optimizer of the full backbone. This double-loop optimization strategy only requires the gradient of the PEFT model and the forward pass of the base model. We provide theoretical convergence guarantees for Bilevel ZOFO. Empirically, we demonstrate that Bilevel-ZOFO significantly outperforms existing ZO methods, achieves 2–4$\times$ faster training, and reduces sensitivity to prompts. Bilevel-ZOFO also outperforms FO PEFT methods while maintaining similar memory efficiency. Additionally, we show its strong potential for meta learning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to improve parameter-efficient fine-tuning (PEFT) algorithms by incorporating a zero-th order optimization technique (MeZO) to update the full model parameters with little memory overhead, maintaining the **memory efficiency** of PEFT algorithms and improving the downstream task accuracy at the expense of **slightly longer training time**. The proposed bi-level minimax algorithm outperforms the performance of first-order PEFT and parameter-efficient MeZO by achieving the pareto-front optimal tradeoff over the baselines in terms of training duration and peak memory consumption, as illustrated in Figure 1.\n\nSTRENGTHS AND WEAKNESSES:\n**Writing**\n- In line 70, \"converge guarantees\" should be \"convergence guarantees\".\n- In Eq. (6), the second term on the right hand side has $\\theta$ as its first argument, which should be $\\theta^k$.\n- Missing full stop in line 157.\n- I suggest the authors to give a concluding summary in Section 5, re-emphasizing the comparison with the FO/ZO PEFT baselines in terms of **memory efficiency** and **runtime/iteration complexity**. This will expand on the sentence \"We propose a new method that is more **efficient** ...\" in line 329 and let the readers know what kind of **efficiency** is achieved by Bilevel-ZOFO.\n\n-------\n\n**Convergence Analysis in Appendix B**\n\n**Pros**\n- The proposed convergence analysis removed the dimension dependence of zeroth order method using a bounded hessian assumption (Assumption B.5).\n\n**Cons**\n- The convergence theorem depends on the assumption that the lower level objective is strongly convex. I suggest the authors to emphasize this limitation in line 154, because such limitation is not mentioned nor discussed in the main text.\n\n------ \n**Significance**\n- The proposed algorithm achieves state-of-the-art training time and memory consumption tradeoff. This is important for practical training especially in large language model fine-tuning.\n\n\n**Originality**\n- While the techniques of the proposed algorithm are not novel, the combination of first-order PEFT and zeroth-order full parameter tuning is novel to the best of my knowledge.\n\nQUESTIONS:\n**Wall Clock Time Analysis in Table 4**\n- It is not clear to me why ZO methods consumes more time per step than FO methods. Intuitively, FO methods require computing the forward and backward pass and therefore should be more time-consuming than ZO methods which only requires 2 forward passes. I hope the authors can provide explanation to this phenomenon, and include it in the main text to give readers a better understanding on the practical complexity of the baseline algorithms.\n\nLIMITATIONS:\nSee **Strengths And Weaknesses**.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out that Zeroth-Order methods suffer from slow convergence, due to the requirement for full-parameter gradient estimation and the necessity of careful prompt selections. Thus, to address these issues, this paper proposes Bilevel ZOFO, which optimizes the full backbone model in the upper level and optimizes the PEFT in the lower level. Experiments show that Bilevel ZOFO outperforms other FO PEFT methods with faster convergence.\n\nSTRENGTHS AND WEAKNESSES:\n**_Strengths_**:\n1. The paper finds that the FO methods suffer from slow convergence and careful prompt selections, and proposes Bilevel ZOFO to address these issues.\n2. The paper provides theoretical guarantees for the convergence of the proposed Bilevel ZOFO.\n3. The experiments show that the proposed Bilevel ZOFO outperforms other FO methods.\n\n**_Weaknesses_**:\n1. The motivation is weak: In the introduction, the authors use the SST-2 dataset as an example to argue that first-order (FO) methods require careful prompt engineering. However, to my understanding, SST-2 was originally designed for masked language models like BERT. When adapting it for decoder-only models such as GPT, which rely on next-token prediction, prompt engineering becomes necessary. In contrast, for tasks like GSM8K or HumanEval, such manual prompt design is typically not required. Therefore, the paper's motivation seems to be based primarily on datasets tailored for masked language models, rather than those naturally suited for decoder-only architectures. When evaluated on datasets such as GSM8K or HumanEval, the core motivation of this paper — the necessity for careful prompt engineering — no longer hold. \n2. I do not fully understand the authors' reasoning for shifting from the observation that hard prompts significantly affect the performance of first-order (FO) methods to proposing a hybrid approach that combines PEFT-based first-order optimization with full-parameter zeroth-order optimization. In effect, simple soft prompt tuning can achieve significantly better performance on GLUE and SuperGLUE benchmarks compared to BiLevel ZOFO, which makes the proposed BiLevel ZOFO a bit meaningless [1].\n3. The experiments in this paper are quite limited. Most of them are conducted on datasets with only 1,000 samples, and the datasets used are all from before 2020. For models with 1.3B or 7B parameters, these datasets (e.g., GLUE or SuperGLUE) are too simple. In my opinion, such datasets are insufficient to evaluate and compare the performance of different algorithms on models of this scale. I think the paper would be stronger if the method was tested on reasoning-heavy datasets like GSM8K, Math, MBPP, HumanEval, and MTBench. It would also help to study how sensitive these tasks are to changes in hyperparameters\n4.  The majority of experiments in this paper are conducted on datasets with only 1,000 samples. I cannot understand why the authors chose this experimental setup. \n\n[1] DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning. In ICLR 2024.\n\nQUESTIONS:\nI think Algorithm 2 (BiLevel ZOFO) is actually a three-level optimization process, and the hyperparameter tuning could be very complex. This increases my concerns about the feasibility of scaling the algorithm to more complex tasks.\n\nLIMITATIONS:\nThere is no experimental evidence showing that the proposed bilevel ZOFO can scale to complex reasoning tasks such as math, code, and instruction following.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Bilevel ZOFO, a fine-tuning method that combines parameter-efficient fine-tuning with gradient-free (zeroth-order) optimization for full-parameter fine-tuning. Experimental results show that the proposed method has certain advantages over either approach individually.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. This paper presents a bilevel fine-tuning method, where introducing a certain degree of gradient computation increases the stability and convergence speed of zeroth-order optimization algorithms.\n2. The proposed method does not significantly increase memory usage.\n\nWeaknesses\n1. The writing of the paper is not good. The abstract’s explanation of the motivation appears to be more like a direct combination of MeZO-type work and PEFT-type work, which leads to an unclear statement of the core issue in the introduction and affects readers’ understanding of the main contributions. In addition, the presentation in the methods section is not clear, and some symbols lack explicit definitions.\n2. The novelty is limited. The paper simply combines MeZO and gradient-based PEFT without clear improvement or contribution, and this combination lacks reasonable practical significance. The core advantage of MeZO is that it does not require backpropagation, which is applicable to many deployment hardware devices. The core advantage of PEFT is to obtain good model adaptation with limited training resources. I cannot imagine a realistic application scenario for the method proposed in this paper.\n3. The experiments are limited. Since the proposed method allows gradient backpropagation, it is recommended to use larger-scale SFT datasets for tasks such as mathematical reasoning and code generation for validation, rather than only using subsets of classification datasets as in purely gradient-free methods like MeZO.\n\nQUESTIONS:\n1. In the description of Algorithm 1, it is recommended to add explanations for different symbols.\n2. The equations after line 113 do not have equation numbers.\n3. The definitions of symbols $s$, $K$, and $T$ should be explicitly stated.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Bilevel ZOFO, a bi-level fine-tuning algorithm for memory efficient training of LLMs. The method proposed uses improvements from both first-order (FO) PEFT and zeroth-order (ZO) tuning methods to improve overall model quality. The authors identify that ZO take long training durations to converge and have affinity to certain prompt formats. On the other hand, PEFT methods like lora are not able to fully achieve the same model quality as full-finetuning. The authors propose a bilevel optimization framework, where both FO-PEFT + ZO are combined. The base model is updated only using ZO, while the PEFT weights are updated in a low-level pass at some update frequency. By combining these two methods, the authors show better performance over either method on some downstream tasks. They further extend their method to meta-learning and show improvements over conventional methods such as Meta-ICL, while being more compute (memory + throughtput) efficient.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The method has good convergence guarantees and all proposed lemmas have proofs that can be followed. \n2. The method improves over standard PEFT and Zero-Order (ZO) methods, including some limitations of ZO methods such as affinity to prompt structure and longer training intervals.\n3. The paper has ablations for different design choices related to the final algorithm, showcasing the strengths of the designed method. \n4. The method is easily adaptable to meta-learning settings, where it outperforms methods such as Meta-ICL on several tasks, while being more compute efficient and taking lower time per training iteration. \n\n**Weaknesses**\n\n1. For most of the main experimental results, we see comparisons against lora and other peft methods, but there is no full fine-tuning with first order methods to compare as a baseline, so as to understand the gap that ZO or PEFT methods have and how much of that is ZOFO able to cover.\n2. There are some parts of the paper which are tricky to follow. For example, in Section 3, when the original method is being introduced, a lot of variables such as $D_f$ and $s$ are explained after their use in equations. Also, usage of dimensions such as d vs d' etc. are not explained beforehand. It took some time and multiple re-reads of the section to fully understand this part. Would recommend rewording and introducing necessary notations for clarity.\n\nQUESTIONS:\n1. the paper looks at very simple classification tasks for experimental evaluation. Have the authors tried it on more difficult tasks such as code / math tuning and have been able to see any success from the method ? \n2. For methods like lora, there have been many more advancements in the literature, that enable high quality PEFT training. Did the authors look at those and see if any of them are more useful in the setting of ZOFO?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}