Bilevel ZOFO: Bridging Parameter-Efficient and Zeroth-Order Techniques for Efficient LLM Fine-Tuning and Meta-Training

Authors: Reza Shirkavand, Qi He, Peiran Yu, Heng Huang
Primary area: foundation or frontier models, including LLMs
Keywords: Zeroth-order Fine-tuning, Parameter Efficient Fine-tuning, Large Language Models, Bilevel Optimization

Abstract:
Fine-tuning pre-trained Large Language Models (LLMs) for downstream tasks using First-Order (FO) optimizers presents significant computational challenges. Parameter-Efficient Fine-Tuning (PEFT) methods have been proposed to address these challenges by freezing most model parameters and training only a small subset. While PEFT is efficient, it may not outperform full fine-tuning when high task-specific performance is required.
Zeroth-Order (ZO) methods offer an alternative for fine-tuning the entire pre-trained model by approximating gradients using only the forward pass, thus eliminating the computational burden of back-propagation in first-order methods. However, when implementing ZO methods, it is crucial to ensure prompt-based text alignment, and relying on simple, fixed hard prompts may not be optimal. In this paper, we propose a bilevel optimization framework that complements ZO methods with PEFT to mitigate sensitivity to hard prompts while efficiently and effectively fine-tuning LLMs. Our Bilevel ZOFO (Zeroth-Order-First-Order) method employs a double-loop optimization strategy, where only the gradient of the PEFT model and the forward pass of the base model are required. We provide convergence guarantees for Bilevel ZOFO. Empirically, we demonstrate that Bilevel ZOFO outperforms both PEFT and ZO methods in single-task settings. Additionally, we show its strong potential for multitask learning. Compared to current first-order meta-training algorithms for multitask learning, our method has significantly lower computational demands while maintaining or improving performance.

Decision: reject
Reviewer scores: 6, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1730599669873

thread_type:
official_review

<thread_object>
creator: reviewer_mll4

<content>
summary:
The paper introduces a bilevel optimization method called Bilevel to efficiently finetune LLMs by combining PEFT and ZO optimization techniques. The motivation is to address the computational inefficiencies of traditional finetuning, which requires backpropagation and substantial memory. The primary research question centers on whether PEFT can enhance ZO fine-tuning for both single-task and multitask settings without requiring extensive computational resources. Bilevel ZOFO uses a two-level optimization framework that applies ZO for full model fine-tuning at the upper level and PEFT at the lower level to minimize computational cost while maintaining high performance. Experiments demonstrate that Bilevel ZOFO outperforms both individual ZO and PEFT methods on single tasks and achieves competitive results in multitask learning with reduced computational demand.

strengths:
The paper has several major strengths:

1. Bilevel ZOFO is effectively extended to multitask learning scenarios, allowing models to handle multiple tasks simultaneously. This is beneficial in resource-limited environments where large, labeled datasets are scarce, such as in medical domains.

2. The paper provides theoretical guarantees for the convergence of the Bilevel ZOFO method.

3. Empirical results demonstrate that Bilevel ZOFO consistently outperforms both standalone ZO and PEFT approaches in single-task finetuning, and it is competitive with SOTA meta-learning methods for multitask scenarios.

weaknesses:
The weaknesses of this paper are as follows:

1. The mathematical notations can be further improved to follow the convention. For example, when denoting a vector, usually bold font is preferred, such $\boldsymbol{\theta} \in \mathbb{R}^d$ instead of $\theta \in \mathbb{R}^d$. This helps the reader differentiate vectors from scalars and improves readability.

2. Typo at the end of line 269 "Sso" should be 'So'

3. In Eq. 2, I am not fully convinced why we need to train the PEFT module p and the base model parameter $\theta$ at the same time? Wouldn't avoiding training $\theta$ be the motivation for using PEFT modules?

4. The motivation for using ZO method in Algorithm 2 is not very clear to me. You may notice that as long as you need to do backpropagation, even if it is parameter efficient finetuning, the computational graph of all the network (from shallow to deep layers) should all be stored and thus, the memory consumption is still very high. Wouldn't this contradict the motivation for using ZO method? Why wouldn't the authors use ZO method throughout the algorithm, namely, use ZO method to calculate the gradient of the PEFT modules as well. 

5. As a follow-up question of 4, this paper does not report the efficiency comparison of any method. Specifically, the memory consumption, time consumption, and the convergence speed should be carefully measured and reported. I am afraid that the combination of ZO and PEFT in a way presented in this paper would harm the efficiency of both methods.

questions:
Please see my questions in the weakness column.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652941787

thread_type:
official_review

<thread_object>
creator: reviewer_dsjq

<content>
summary:
This paper realised two issues in current LLM tuning research. One is that parameter-efficient fine-tuning (PEFT) cannot sufficiently compete with full-model tuning, because only part of model parameters are tuned, which limits the model capacity. The second is that zeroth-order (ZO) optimization can tune full model parameters while relying on fixed non-optimal prompts. The authors propose to complement ZO with PEFT to mitigate the sensitivity to hard prompts through bi-level optimization. During formulation, they first transform the objective into single-level optimization problem and then use zeroth-order information to approximate gradient.

strengths:
1. This paper presents a bi-level optimization method that is more suitable for tuning full pre-trained large language models, compared with parameter efficient tuning and zeroth-order tuning.
2. The proposed method can be extended to a lightweight meta-training process for multi-task learning.
3. The final results outperform both FO.and recent work MeZO.

weaknesses:
1. I understand the motivation of complementing ZO with PEFT, but bi-level is known for its high computation cost, even after the zeroth-order approximation. In this sense, how is the tuning efficiency of the proposed method?
2.On line 201, you split data into two parts. Can the two-level optimization share the same tuning dataset?
3. From Eq.3 to Eq. 7, this process is following existing works. Any new technical contributions here? And have you demonstrated this transformation is better than others?
4. Efficiency comparision is lacked in experiments.
5. There are some improved LLM tuning works over MeZO. If this is not necessary to compare or even mention them, please provide the reasons.
6. Why not consider tuning soft prompts in this paper? As zeroth-order techniques are also explicitly used to tune prompts. [1]

[1] Black-Box Tuning for Language-Model-as-a-Service. ICML 2022.

questions:
Please refer to Weaknesses.

The theoretical analysis presents the convergence of bi-level optimisation. It might be interesting for some readers but I doubt its significance in LLM tuning. For example, can you explain how will the convergence theory instruct the tuning process in real applications?

It would be better to draw a pipeline to show the optimization procedure, which can be left to appendix if not sufficient space.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730698355787

thread_type:
official_review

<thread_object>
creator: reviewer_21gx

<content>
summary:
This paper proposes a bilevel optimization framework and combines zero-order finetuning with first-order PEFT to achieve higher performance on the downstream task. Experiments show that their proposed method outperformance both zero-order finetuning and PEFT only.

strengths:
1. The proposed method is novel and interesting.
2. The experiment results look pretty strong compared to vanilla finetuning.

weaknesses:
1. The motivation of this paper is unclear. The authors mention that with the bilevel optimization, the sensitivity of ZO to hard prompts can be mitigated. Are there any evidence for this claim? Or is the performance improvement simply comes from more tunable parameters?
2. Similar to point 1, have the authors done experiments with PEFT followed by a ZO finetuning?
3. The experiment section is a bit unclear, the MeZO baseline is not explained clearly.

questions:
Please see the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730885209738

thread_type:
official_review

<thread_object>
creator: reviewer_tcaa

<content>
summary:
The paper propose  Bilevel ZOFO, a framework combining PEFT and Zeroth-Order (ZO) optimization to improve fine-tuning and meta-training efficiency for large language models. Using a bilevel approach, it achieves better performance in single- and multi-task settings. 
It provides convergence guarantees for the optimization process.

strengths:
Bilevel ZOFO effectively combines PEFT and ZO optimization to reduce computational and memory costs, making large model fine-tuning more efficient. 
Its bilevel structure allows for competitive performance in both single- and multi-task settings with theoretical convergence guarantees.

weaknesses:
The experiments are limited to small- to medium-sized models in its experiments, raising concerns about scalability to larger models commonly used in practice and fair comparison.

questions:
Q1 Could you provide quantitative evaluations for convergence, wall-clock time per step, memory profiling, and memory consumption, as seen in the MeZO paper?

Unlike MeZO that only require forward passes,  Bilevel ZOFO relies on First-Order gradient calculations, which can significantly increase computational and memory demands.
Minimax optimization often faces convergence problems due to saddle points, which can cause the training to stall and slow down overall progress. 


Q2 Could you provide the experimental results for larger models such as OPT13B.

ZO methods are particularly beneficial for reducing computational resources in large-scale model training. Thus, existing methods have generally evaluated models of at least moderate scale, such as models larger than OPT13B. The experiments in this paper, however, are limited to small- to medium-sized  models, up to 7B parameters.
To ensure a fair comparison with existing methods, it is essential to test on models of at least comparable size, such as OPT13B.
Effective hyperparameters may vary significantly based on model size.
However, MeZO tested in this paper may not necessarily be optimal for small-scale settings.
as existing methods are not designed with small-scale models.


Q3 Could you provide detailed experimental settings?
The MeZO paper provides detailed information on hyperparameters and computational environment, which is necessary for reproducibility and accurate comparison.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732152010391

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Comment Addressing Shared Concern

comment:
We thank the reviewers for their efforts and valuable feedback.  As there are is common concern among the reviewers about lack of memory profiling and wall-clock-time analysis, we address it here.

In our experimental settings, the memory usage of our method matches that of the most memory-intensive baselines. We have now included **Figure 5** and **Table 6** in the revised version (pages 23 and 24 in the appendix) to address these concerns more comprehensively. Our implementation builds upon MeZO’s codebase, and memory profiling as well as latency calculations are based on their code. We summarize the results below. 

## **Memory Profiling:** 
Figure 5 (now included in the paper) illustrates memory consumption across tasks for Bilevel-ZOFO, MeZO, and FO Prefix-Tuning. Memory usage varies depending on the task, with MeZO and first-order PEFT methods occasionally surpassing each other. The GPU memory requirement of our method is comparable to the memory usage of FO PEFT and MeZO experiments. Despite this, as demonstrated in Table 1 of the paper, our method consistently outperforms both PEFT and MeZO on average. We include the values of Figure 5 in Tab. 1 below for ease of reference. Specifically, the values correspond to maximum GPU memory allocated in terms for GB for  fine-tuning OPT1.3b on each task using a batch size of 8 and on a single A6000ada 48GB GPU. 

### Tab. 1: Memory Consumption Comparison (Values from Figure 5 of the paper)

| Datasets | MeZO  | FO (Prefix) | Bilevel-ZOFO (Prefix)|
|----------|-----------|------------------|----------------------------|
| Copa     | 8.12      | 2.74             | 8.12                       |
| MultiRC  | 10.12     | 22.72            | 22.72                      |
| ReCoRD   | 8.99      | 11.22            | 11.22                      |
| WSC      | 8.13      | 4.24             | 8.13                       |

## **Wall-clock Time Analysis**

We also present a wall-clock time analysis of bilevel-ZOFO compared to the baseline. Tab. 2 below shows the results (See Table 6 in the appendix for a complete version). Similar to MeZO, we observe that each zeroth-order tuning steps takes longer compared to first-order steps. The results indicate that our bilevel-ZOFO achieves comparable step duration to the FO-PEFT method while significantly requiring fewer step duration compared to MeZO. Moreover, as highlighted in Table 1 of our paper, bilevel-ZOFO outperforms both methods on average. 

The values in Tab.2 correspond to the duration of each step of the methods in seconds. The values are measured based on tuning OPT1.3b on a single A6000ada 48GB GPU. The wallclock time is averaged over 3 different runs that produced the values of Table 1 of our paper. We use a batch size of 8 for all experiments.
| 

### Tab. 2: Wall clock time duration (in seconds) of each step of our method and the baselines. (Values from Table 6 of the paper)
Task     | MeZO | FO Prefix-Tuning  | Bilevel-ZOFO (Prefix) |
|----------|----------|----------------------|---------------------------|
| Copa     | 0.299    | 0.127                | 0.135                     |
| MultiRC  | 0.622    | 0.474                | 0.502                     |
| WSC      | 0.278    | 0.120                | 0.164                     |
</content>
</thread_object>

</thread>
<thread>
creation date:
1733287451746

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the discussion period

comment:
We sincerely thank the Area Chair and the reviewers for their efforts. The discussion improved the clarity and overall presentation of our work. During this rebuttal process, the reviewers highlighted the following strengths in our work:

1. **Novelty:** This work introduces a novel bilevel optimization framework specifically designed for fine-tuning large pre-trained language models.

2. **Efficiency and Performance:** This work use bilevel structure that relates  PEFT to zeroth-order (ZO) optimization to significantly reduce computational and memory costs, achieving competitive results in both single-task and multi-task settings. It outperforms First-Order PEFT, and Zeroth-order approaches like MeZO.

3. **Multitask Learning:** This method supports lightweight meta-training processes, making it suitable for efficient multi-task learning in diverse applications.  The proposed method is particularly beneficial in resource-limited settings, such as medical domains with limited labeled data.

4. **Theoretical Soundness:** We provided theoretical guarantees for convergence, enhancing the framework's reliability.

They also raised some concerns which we summarize below and explain how they were addressed. There were minor concerns, such as correcting notation and including a pipeline figure that we fully addressed. However, we have omitted them here for the sake of brevity.

------------
## Questions, Concerns of Reviewer tCaA and How We Addressed Them:

1. **Scalability to models larger than 13B:** We demonstrated consistent performance across 1B–7B models and clarified that the method is expected to scale effectively to larger models, given the strong baseline performance in this range.  


2. **Memory and Time Profiling:** We provided detailed analyses showing bilevel-ZOFO does not increase the memory usage of PEFT and MeZO, while outperforming both.


3. **Convergence guarantee beyond theory:** We added evaluations confirming smooth convergence aligned with theoretical guarantees.  


4. **Experimental Details:** We expanded experimental settings, hyperparameters, and computational environment descriptions for clarity and comparability.  

**Outcome:**  
The reviewer acknowledged that all concerns were fully addressed and increased their score.

------------
## Questions, Concerns of Reviewer 21GX and How We Addressed Them:

1. **Motivation and Sensitivity of ZO to Hard Prompts:**  


   - We added additional experiments to demonstrate that Bilevel-ZOFO effectively mitigates MeZO’s sensitivity to hard prompts, maintaining robust performance with minimal variance.  


   - We validated that the improvements are not due to more tunable parameters through comparisons with a two-stage PEFT-MeZO approach, which showed degraded performance compared to Bilevel-ZOFO.  


     - We conducted rigorous hyper-parameter tuning and presented training loss curves for two-stage PEFT-MeZO experiments as requested by the reviewer, confirming the limitations of a two-stage pipeline and justifying Bilevel-ZOFO’s effectiveness.  

2. **Explanation of MeZO Baseline:**  


   - We provided detailed explanations of MeZO’s methodology and limitations to enhance clarity in the manuscript and offer sufficient background for readers totally unfamiliar with the approach.

3. **Question about Low-resource setting Rationale** 
 
   We addressed the reason for the low-resource experimental setting is to align with benchmarks from prior work (e.g., MeZO) to ensure fairness in comparison.  

**Outcome:**  

The reviewer acknowledged that their concerns were addressed but hesitated to increase their score. They reasoned that they are unfamiliar with related works and that our performance improvement over PEFT was not significant, despite having highlighted it as a strength in their review at the beginning.

We further emphasized the novelty and impact of the work, clarifying inconsistencies in the reviewer’s evaluation and providing additional evidence for the significance of the results. Despite we resolving all concerns, the reviewer did not further engage in discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses two key issues in current research on Large Language Model (LLM) tuning. First, it points out that parameter-efficient fine-tuning (PEFT) cannot fully compete with full-model tuning, as it only tunes a subset of model parameters, limiting the model’s capacity. Second, the paper identifies that zeroth-order (ZO) optimization can tune all model parameters but relies on fixed, non-optimal prompts, which may not be ideal. To address these challenges, the authors propose complementing ZO with PEFT through a bi-level optimization approach to reduce sensitivity to hard prompts. In their formulation, they first transform the problem into a single-level optimization problem and then use zeroth-order information to approximate the gradient.

However, the experimental section lacks clarity and efficiency comparisons. The advantage over the MeZO baseline is not sufficiently explained, and the authors do not address other baseline methods that could improve MeZO. Furthermore, the motivation for combining ZO with PEFT is not fully articulated. While bi-level optimization is known for its high computational cost, even with zeroth-order approximations, the combination of these methods appears relatively simple. Overall, the motivation behind this paper remains unclear.