BiDoRA: Bi-level Optimization-based Weight-Decomposed Low-Rank Adaptation

Authors: Peijia Qin, Ruiyi Zhang, Pengtao Xie
Primary area: foundation or frontier models, including LLMs
Keywords: Parameter-Efficient Fine-Tuning, Bi-level Optimization, Large Language Model

Abstract:
Parameter-efficient fine-tuning (PEFT) of large language models (LLMs) has gained considerable attention as a flexible and efficient way of adapting LLMs to downstream tasks.
Among these methods, weighted decomposed low-rank adaptation (DoRA) has emerged as a promising approach.
DoRA bridges the gap between low-rank adaptation (LoRA) and full fine-tuning (FT) by decomposing the weight matrices into magnitude and direction components, thereby maintaining learning behavior similar to FT.
Although DoRA shows encouraging performance, it introduces additional parameters compared to LoRA, which potentially increases the risk of overfitting.
Moreover, optimizing magnitude and direction simultaneously leads to a coupled gradient updating pattern for both components, limiting its learning capacity.
To overcome these limitations, we propose BiDoRA, a bi-level optimization-based PEFT method.
In BiDoRA, the direction and magnitude components are optimized on two distinct datasets at different optimization levels, mitigating the risk of overfitting.
Additionally, the asynchronous optimization of the two components promotes their decoupling, allowing for more flexible gradient updates suitable for various downstream tasks.
Evaluation of BiDoRA on fourteen datasets spanning natural language understanding, natural language generation, and token classification reveals that it significantly outperforms DoRA and other PEFT methods.
The superior performance of BiDoRA underscores its effectiveness.
The code for BiDoRA is available at https://anonymous.4open.science/r/BiDoRA-5D31.

Decision: reject
Reviewer scores: 3, 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730353578628

thread_type:
official_review

<thread_object>
creator: reviewer_ycca

<content>
summary:
This paper proposes BiDoRA, a novel method for parameter-efficient fine-tuning (PEFT). While based on DoRA's methodology, it introduces a bi-level optimization approach that optimizes parameters separately. The authors argue that this approach reduces overfitting and enables learning patterns more similar to full fine-tuning. Through experiments across various NLP datasets, they demonstrated that BiDoRA consistently outperforms both DoRA and other PEFT methods.

strengths:
This work represents a valuable first attempt at applying bi-level optimization to PEFT, extending DoRA's magnitude/direction decomposition in a natural progression. Experimental validation includes a comprehensive evaluation across diverse NLP tasks, such as NLU, NLG, and token classification, with well-suited comparisons to baselines and detailed ablation studies demonstrating the effectiveness of each component. The implementation is both clear and reproducible, contributing to the overall technical completeness of the work.

weaknesses:
Methodological Limitations: The approach may be perceived as a straightforward combination of existing methods (DoRA + bi-level optimization), which could limit its novelty. Additionally, the method introduces substantial computational complexity, running at 3.92 times the cost of LoRA, and requires extra hyperparameter tuning. To address these issues, improvements in performance relative to computational cost are needed. Developing an automated framework for setting additional hyperparameters could also streamline the process and reduce user intervention.

Experimental Limitations: The study lacks experiments involving recent large language models (LLMs), does not sufficiently analyze performance on very small datasets, and has limited exploration of data split ratio effects. To enhance experimental rigor, including evaluations with larger, contemporary models, examining performance across varying dataset sizes, and conducting sensitivity analyses across different split ratios would offer a more comprehensive assessment of the method’s effectiveness.

Theoretical Limitations: The paper currently provides limited theoretical guarantees for generalization performance and lacks convergence analysis for the bi-level optimization approach. Strengthening the theoretical foundation could be achieved by introducing an upper bound analysis for generalization error and supplying a convergence proof for the optimization algorithm.

Together, addressing these methodological, experimental, and theoretical gaps would significantly enhance the robustness and credibility of the study.

questions:
1) Are there experimental results involving larger, more recent LLMs?
2) Is there an analysis for cases with very small dataset sizes?
3) Are there optimization plans in place to reduce computational complexity?
4) Is there additional analysis on how the choice of data split ratios affects performance?
5) Can theoretical guarantees for generalization error be provided?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730409651737

thread_type:
official_review

<thread_object>
creator: reviewer_p28u

<content>
summary:
This paper proposes BiDoRA, a new fine-tuning method that:
1. Decomposes the model’s weights into two parts—magnitude and direction.
2. Optimizes each part separately: The "direction" part is trained on one dataset split (training set), while the "magnitude" part is optimized using another split (validation set).
3. Iterates between these two optimization levels to decouple the updates, helping the model generalize better.

This bi-level approach allows BiDoRA to perform more like full fine-tuning while avoiding overfitting. In experiments across various NLP tasks, BiDoRA consistently outperforms other parameter-efficient fine-tuning methods like LoRA and DoRA. The authors validate BiDoRA on natural language understanding, generation, and token classification tasks, showing that it reduces overfitting and achieves better overall performance than existing methods.

strengths:
Originality: BiDoRA introduces a unique bi-level optimization approach to fine-tuning large language models (LLMs), addressing a common tradeoff in parameter-efficient fine-tuning (PEFT) between generalization and computational efficiency. By decomposing the model’s weights into magnitude and direction components and optimizing each on different data splits, BiDoRA creatively combines aspects of neural architecture search with PEFT, offering a compelling alternative to current methods like LoRA and DoRA. This methodological innovation could inspire further development in PEFT techniques across domains.

And specifically I think it's interesting that in BiDoRA, both the magnitude and direction components are trainable, but they are optimized separately through a bi-level optimization framework.

Decomposition: The model's weights are decomposed into two parts:

Magnitude (m): Represents the scale of each parameter.
Direction (V): Represents the vector defining the orientation of each weight.
Optimization Approach:
BiDoRA optimizes the direction component at the lower level using a training set split. During this phase, the magnitude remains fixed, allowing the model to focus solely on finding the optimal direction adjustments.
At the upper level, BiDoRA optimizes the magnitude component by minimizing the loss on a validation set split, with the direction component fixed. This stage uses the optimized direction from the lower level to update the magnitude through hyper gradient descent.
Iterative Training:

The magnitude and direction components are iteratively updated until convergence. This iterative decoupling enables each component to learn independently while promoting generalization and reducing overfitting.
This bi-level approach allows each component to be optimized in a way that would ideally yield better performance than traditional PEFT methods, which tend to optimize these components together, leading to coupled gradients and potentially reduced flexibility.

Quality: The paper demonstrates a high level of rigor, with comprehensive experimental evaluations across multiple tasks, including natural language understanding, generation, and token classification. The experiments span diverse datasets, and the authors include robust analyses, such as ablation studies and weight decomposition, that substantiate the effectiveness of the bi-level optimization framework. This thorough evaluation establishes BiDoRA’s consistent performance advantage over existing PEFT methods, solidifying the validity of the approach.

Clarity: The paper is well-organized and clearly articulates both the theoretical underpinnings and practical implications of BiDoRA. Key concepts, such as weight decomposition and bi-level optimization, are introduced methodically, and diagrams aid in visualizing the process. Algorithmic details, hyperparameters, and dataset splits are documented with transparency, making it easier for readers and future researchers to reproduce and build upon this work.

Significance: The introduction of BiDoRA is a valuable contribution to the field of large language model fine-tuning, where computational efficiency is increasingly critical. By reducing overfitting and enhancing generalization, BiDoRA effectively bridges the gap between parameter efficiency and performance, providing a practical solution that could impact a wide range of applications requiring adaptable, high-performing LLMs. Its relevance to real-world LLM deployment in resource-constrained settings adds to the significance of its contributions.

weaknesses:
Computational Cost and Efficiency: Although BiDoRA shows a significant performance improvement, the bi-level optimization approach introduces a high computational cost, as reported with nearly fourfold overhead compared to LoRA. This could limit BiDoRA’s practicality in scenarios where resources are constrained. An in-depth analysis of ways to reduce computational complexity without sacrificing performance—such as approximations, alternative regularization techniques, or a comparative exploration of optimization strategies with fewer computational steps—would enhance the method’s accessibility.

Hyperparameter Sensitivity and Tuning: The paper provides limited insight into the sensitivity of BiDoRA to hyperparameters like data split ratios, learning rates for each level, and the orthogonal regularization coefficient. Given that bi-level optimization frameworks often require precise tuning, a sensitivity analysis would help clarify the stability of BiDoRA’s performance under various configurations. This analysis would also support practitioners in understanding which parameters are most influential, guiding them in practical applications.

Scope of Evaluation on Model Types and Tasks: While BiDoRA’s evaluation spans diverse NLP tasks, it primarily focuses on RoBERTa and GPT-2 models. Expanding the scope to include additional architectures, particularly those with different structural properties (e.g., BERT or T5), would verify BiDoRA’s generalizability across model types. Furthermore, testing BiDoRA in non-NLP tasks, such as vision or multimodal tasks, would strengthen the claim of BiDoRA as a general-purpose PEFT framework and broaden its potential applications.

Limited Exploration of Overfitting Mitigation: The paper emphasizes BiDoRA’s success in mitigating overfitting through the decoupling of magnitude and direction optimizations. However, it does not provide an in-depth comparison with existing regularization techniques or alternative overfitting mitigation strategies within PEFT. A comparison or ablation study exploring how BiDoRA’s approach to overfitting compares with alternative techniques (e.g., dropout or adaptive regularization) would better demonstrate the unique value of the bi-level approach.

questions:
Could the authors provide additional details on the computational efficiency of BiDoRA? Given the reported computational overhead (approximately four times that of LoRA), are there strategies under consideration for reducing this cost? For instance, would approximate or adaptive bi-level optimization approaches be feasible to explore?

Since bi-level optimization can sometimes suffer from convergence issues, could the authors clarify the convergence criteria used in BiDoRA? Additionally, are there any stability challenges encountered during training, and if so, how were these addressed? This information would be valuable for practitioners looking to implement BiDoRA in different contexts.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731056897687

thread_type:
official_review

<thread_object>
creator: reviewer_mkru

<content>
summary:
Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method. Weighted Decomposed Low-Rank Adaptation (DoRA) is a variant of LoRA, where the update matrix is decomposed into two components: magnitude and direction.

This paper argues that the additional parameters in DoRA increase the risk of overfitting, and simultaneously optimizing both magnitude and direction limits its learning capacity. To address these issues, the authors introduce Bi-Level Optimization-Based Weight-Decomposed Low-Rank Adaptation (BiDoRA), which integrates bi-level optimization into DoRA.

The core idea of BiDoRA is to separate the optimization of magnitude and direction. In the first phase, BiDoRA alternately trains magnitude and direction until convergence. Additionally, the training data for these two components is separated to further prevent overfitting: the magnitude is trained on a validation set, while the direction is trained on a training set. In the second phase, the direction is further trained on the combined dataset until convergence.

Experiments are conducted on natural language understanding (NLU), natural language generation (NLG), and token classification tasks. BiDoRA shows marginal improvements over the baselines, with almost all gains being less than 1 percentage point. Ablation studies reveal that the effects of different design components are relatively small. The training time for BiDoRA is 3.92 times that of LoRA, while DoRA requires 1.3 times the training time of LoRA.

strengths:
* The paper is well-written and easy to understand.
* The authors' efforts to evaluate across multiple tasks are commendable. However, there are still many problems with these evaluations (see Weaknesses).

weaknesses:
1. **The motivation of this paper appears to be questionable.** The authors claim that DoRA increases the risk of overfitting, basing this on two pieces of evidence:

   - DoRA introduces additional parameters compared to LoRA.
   - The gap between training and test accuracy curves for DoRA is larger than that of BiDoRA.

   However, these two points do not convincingly support the claim. First, while additional parameters can sometimes contribute to overfitting, they are not a sufficient condition for it. In fact, DoRA adds only a negligible number of parameters (0.01% of the model size, as reported by the authors) beyond LoRA. Moreover, prior work [1] suggests that LoRA learns less than full fine-tuning and may even act as a form of regularization, implying that the risk of overfitting is generally low across these PEFT methods.

   Additionally, the training curves are not necessarily indicative of overfitting, as they can be significantly influenced by factors such as hyperparameters, model architecture, and dataset characteristics. The authors present results from only a single configuration, which limits the generalizability of their findings.

   Finally, the authors’ attribution of an *alleged overfitting problem* to DoRA’s concurrent training lacks a strong foundation.

2. **The proposed BiDoRA method is overly complex and difficult to use.** It requires a two-phase training process, with the first phase itself consisting of two sub-steps. It also introduces two additional hyperparameters: the weight of orthogonality regularization and a ratio for splitting training and validation sets. As a result, BiDoRA takes 3.92 times longer to train than LoRA.

3. **Performance differences between methods are minimal across evaluations**. In nearly all results, the performance differences between the methods are less than 1 percentage point, which may be attributable to random variation. Furthermore, the benchmarks selected are outdated and likely saturated.

[1] [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673)

questions:
The results in Table 3 do not align with those reported in the original LoRA paper. In the original paper, LoRA achieves a BLEU score of 70.4 with 0.35M parameters. However, in your results, LoRA only reaches 63.7 BLEU with 0.39M parameters. This discrepancy suggests there may be errors in your evaluation or setup.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731119334277

thread_type:
official_review

<thread_object>
creator: reviewer_wcvc

<content>
summary:
This paper proposes a method for parameter efficient fine tuning using bi-level optimization that competes with LoRA. Results test it on training RoBERTa and GPT-2 medium for NLU and NLG tasks respectively. The overall results demonstrate consistently better results than LoRA and DoRA.

strengths:
- In experiments, the proposed method demonstrates consistently superior results compared to the others.
- The paper was written reasonably clearly, so I was able to follow the equations.

weaknesses:
- The motivation for the bi-level optimization approach was a bit vague. I did read the explanation in the introduction, but it says "Furthermore, in DoRA, the magnitude and incremental direction components are optimized concurrently, leading to a highly constrained updating pattern that may overlook the diverse learning patterns required for different downstream tasks." It is not clear what this means concretely...
- Experiments were on models that do not represent the state of the art at this point. It would be much more convincing if the methods were demonstrated to work on more powerful models (e.g. at least 7B).
- As noted in 5.7, the proposed method is accompanied with significant additional training time, which may limit its practical usefulness.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1732534090956

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear reviewers,

We have submitted the rebuttal version of the paper with major changes highlighted in purple. We highly value your concerns regarding insufficient experiments, computation efficiency, and hyperparameter sensitivity. We provide the following response in the hope of addressing your concerns.

### Further Experimental Studies

We thank all reviewers for raising concerns about the lack of diverse experimental studies. To address your concerns, we have improved our work by conducting two additional experiments:

- **Larger Models:** We conducted experiments with much larger-scale LLMs, including the DeBERTa-v2-xxlarge model (1.5B parameters) and the Llama2-7b model (7B parameters). The results presented in Section 5.6 indicate that BiDoRA yields better performance when fine-tuning models with a very large number of parameters.

- **Extremely Small Datasets:** We also conducted experiments with extremely small datasets and presented the results in Section 5.5. Specifically, we fine-tuned the protein language model ESM on three prediction tasks with extremely small datasets (3695, 936, and 200 training samples, respectively). In these cases, overfitting is more severe in LoRA and DoRA, and BiDoRA shows improvements with a larger margin.

 The two additional results confirm our previous findings on BiDoRA's effectiveness when applied to various tasks and diverse architectures.

### Computational Efficiency

We thank all reviewers for raising concerns about computational efficiency. We would like to clarify that although our method is computationally costly than DoRA, it converges much faster, as shown by our additional results in Figure 4 (Section 5.9). It helps mitigate the computational overhead to some extent, and allows BiDoRA to achieve comparable performance in the same wall-clock time. Moreover, BiDoRA can outperform the baselines with a longer time budget, underscoring its practicality.

In fact, the most computationally expensive part of our method is the hyper-gradient estimation. Therefore, the efficiency of BiDoRA could be improved by using more accurate hyper-gradient estimators, such as implicit differentiation [2] and SAMA [3]. These can be easily integrated into our framework thanks to support from the Betty library [4]. We leave this potential solution to future work and have added a comment on this at the end of Section 6. Currently, we focus on the finite difference approximation scheme [1] for ease of understanding.

### Hyperparameter Sensitivity

We are aware that BiDoRA introduces additional hyperparameters such as dataset split ratio, upper-level learning rate, and regularization coefficient. Fortunately, we found them relatively robust within certain ranges. While further fine-tuning could potentially improve performance, a straightforward setup already achieves superior performance compared to the baselines. Please refer to Appendix B for all hyperparameter settings. Specifically, we set the additional parameters in the following ways:

- **Train-validation Split Ratio:** We kept the train-validation split ratio the same (train: validation=8: 2) across all experiments, and we found this ratio to be effective, though it could be fine-tuned for better performance (see Appendix F for details).
- **Regularization Coefficient:** We did not perform extensive tuning; a value of 0 or 1e-5 was used for all experiments.
- **Upper-level Learning Rate:** Is was set to the same value as the lower-level learning rate for all experiments without tuning.

In conclusion, our method achieves superior performance without extensive hyperparameter tuning, making BiDoRA practical and easy to use.



We are open to further discussions regarding the rebuttal version. If you have any additional questions, please feel free to share them. Again, thank you all for your valuable reviews of our work.

[1] Liu, Hanxiao, Karen Simonyan, and Yiming Yang. "Darts: Differentiable architecture search." *arXiv preprint arXiv:1806.09055* (2018).

[2] Rajeswaran, Aravind, et al. "Meta-learning with implicit gradients." *Advances in neural information processing systems* 32 (2019).

[3] Choe, Sang, et al. "Making scalable meta learning practical." *Advances in neural information processing systems* 36 (2024).

[4] Choe, Sang Keun, et al. "Betty: An automatic differentiation library for multilevel optimization." *arXiv preprint arXiv:2207.02849* (2022).
</content>
</thread_object>

</thread>


Metareview:
metareview:
BiDoRA is a bi-level optimization method designed for parameter-efficient fine-tuning (PEFT) of large language models (LLMs). It builds on DoRA by decoupling the optimization of magnitude and direction components, offering a novel perspective on fine-tuning. The paper is well-written and accessible.

However, all reviewers noted that the significant additional training time could limit its practical utility, despite its ability to achieve fast convergence. Furthermore, the evaluation results show only minimal improvements on the GLUE benchmark.

I believe the paper would benefit from another round of review, focusing on refining its motivation, presenting more robust experiments, providing stronger theoretical support, and conducting an in-depth comparison with existing regularization techniques or alternative strategies for mitigating overfitting.

Therefore, I recommend rejection.