BiDoRA: Bi-level Optimization-based Weight-Decomposed Low-Rank Adaptation

## Abstract

Abstract Parameter-efficient fine-tuning (PEFT) of large language models (LLMs) has gained considerable attention as a flexible and efficient way of adapting LLMs to downstream tasks.
Among these methods, weighted decomposed low-rank adaptation (DoRA) has emerged as a promising approach.
DoRA bridges the gap between low-rank adaptation (LoRA) and full fine-tuning (FT) by decomposing the weight matrices into magnitude and direction components, thereby maintaining learning behavior similar to FT.
Although DoRA shows encouraging performance, it introduces additional parameters compared to LoRA, which potentially increases the risk of overfitting.
Moreover, optimizing magnitude and direction simultaneously leads to a coupled gradient updating pattern for both components, limiting its learning capacity.
To overcome these limitations, we propose BiDoRA, a bi-level optimization-based PEFT method.
In BiDoRA, the direction and magnitude components are optimized on two distinct datasets at different optimization levels, mitigating the risk of overfitting.
Additionally, the asynchronous optimization of the two components promotes their decoupling, allowing for more flexible gradient updates suitable for various downstream tasks.
Evaluation of BiDoRA on fourteen datasets spanning natural language understanding, natural language generation, and token classification reveals that it significantly outperforms DoRA and other PEFT methods.
The superior performance of BiDoRA underscores its effectiveness.
The code for BiDoRA is available at https://anonymous.4open.science/r/BiDoRA-5D31 .

## 1 Introduction

Large language models (LLMs) (Radford et al., ; Brown et al., ) have achieved state-of-the-art results across a broad range of NLP tasks, from natural language understanding (NLU) (Wang et al., ) to natural language generation (NLG) (Novikova et al., ).
Although LLMs are pre-trained on extensive datasets in the general domain, additional methods are required to adapt them to specific downstream datasets for better performance.
For example, full fine-tuning (FT) involves updating all pre-trained parameters on downstream datasets (Qiu et al., ), which typically yields superior results.
However, as LLMs continue to scale, FT incurs significant computational costs and increases the risk of overfitting during the fine-tuning process (Karimi Mahabadi et al., ).
Alternatively, in-context learning adapts LLMs to new tasks by incorporating a few data examples directly into the prompt, eliminating the need for parameter updates (Brown et al., ).
While this approach substantially reduces computational costs and mitigates the risk of overfitting, it is constrained by the use of only a limited number of examples from the downstream data, often leading to suboptimal performance.

To address the limitations above, parameter-efficient fine-tuning (PEFT) methods (Houlsby et al., ; Hu et al., ) have been introduced as a promising solution.
PEFT approaches update only a subset of the pre-trained parameters, achieving performance comparable to FT while requiring significantly fewer computational resources.
Among these methods, low-rank adaptation (LoRA, Hu et al. ()) has gained popularity due to its simplicity and effectiveness.
LoRA attaches low-rank matrices to the pre-trained model weights, updating only these matrices during fine-tuning without changing the model architecture.
However, LoRA exhibits a parameter updating pattern distinct from FT, which may limit its learning capacity (Liu et al., ).
Specifically, when decomposing the parameter updates of both methods into magnitude and direction components, the correlation between these components tends to be positive in LoRA, whereas it is negative in FT.
To bridge this gap, weight-decomposed low-rank adaptation (DoRA, Liu et al. ()) introduces an explicit reparameterization of the pre-trained weights matrix as the product of magnitude and direction components.
The model can be trained end-to-end in the same way, with direction components parameterized with incremental low-rank matrices.
This approach enables DoRA to share similar learning patterns with FT, thereby outperforming LoRA in multiple tasks.
Nonetheless, DoRA introduces additional parameters compared to LoRA, which can exacerbate overfitting issues, particularly when adapting to smaller downstream datasets (See Figure 2).
Furthermore, in DoRA, the magnitude and incremental direction components are optimized concurrently, leading to a highly constrained updating pattern that may overlook the diverse learning patterns required for different downstream tasks.

Figure: Figure 1: An overview of BiDoRA. BiDoRA performs PEFT using a bi-level optimization framework. At the lower level, BiDoRA learns the incremental direction component $\Delta V$ of the update matrices using the training split of the downstream dataset. At the upper level, BiDoRA optimizes the magnitude component $m$ with optimized $\Delta V$ from the lower level, using the validation split of the dataset. After determining the optimal magnitude, the direction component undergoes further fine-tuning on a combined set of both training and validation splits to maximize overall performance.
Refer to caption: x1.png

To address the challenges above, we propose BiDoRA, a bi-level optimization-based weight-decomposed low-rank adaptation method for parameter-efficient fine-tuning of LLMs.
BiDoRA mitigates overfitting and facilitates a flexible weight updating pattern by separately optimizing the magnitude and incremental direction components on different splits of the downstream datasets using distinct optimization steps.
BiDoRA is based on a bi-level optimization framework:
At the lower level, the low-rank incremental direction component is updated using the training split of the downstream dataset, while the magnitude component remains fixed.
At the upper level, the magnitude component is updated by minimizing the loss on the validation split via hypergradient descent.
These two optimization steps are performed iteratively until convergence.
Figure 1 provides an overview of BiDoRA.

In BiDoRA, the two distinct components are trained on separate splits of the dataset, which effectively reduces the risk of overfitting.
A similar strategy is utilized in the well-established practice of differentiable neural architecture search (DARTS, Liu et al. ()), where architecture and model parameters are learned using different splits of the training dataset.
Since the architecture selection module introduces additional parameters beyond the base model, directly optimizing this overly complex supernetwork can result in severe overfitting.
Similarly, by treating the magnitude component as the architecture and the incremental direction component as the model in neural architecture search, training these components on separate datasets helps reduce overfitting.
As shown in Figure 2, BiDoRA demonstrates better resistance to overfitting compared to DoRA, given the smaller performance gap between the training set and test set.
Furthermore, the asynchronous gradient update steps at the two optimization levels in BiDoRA facilitate better decoupling of the magnitude component from the incremental direction component, leading to a more flexible update pattern that closely resembles FT.
As illustrated in Figure 3, the updates of the magnitude and direction components across different layers using BiDoRA have a correlation value that is closest to that of FT, highlighting its superior learning capability compared to both DoRA and LoRA.

Our work makes the following key contributions:

- •
We propose BiDoRA, a novel parameter-efficient fine-tuning method based on bi-level optimization.
In contrast to DoRA, which trains the magnitude and incremental direction components on a single dataset, BiDoRA optimizes these components on different splits of a downstream dataset through distinct optimization steps.
- •
Our strategy effectively mitigates the risk of overfitting and results in a parameter update pattern that more closely resembles full fine-tuning.
- •
Extensive experiments on fourteen datasets highlight the superior performance of BiDoRA.
BiDoRA consistently surpasses several baseline methods, including LoRA and DoRA, across tasks such as text classification, language generation, and token classification.

Figure: Figure 2: Training and test accuracy versus global training steps on the ModHayes split of the Reuters21578 dataset when fine-tuning a RoBERTa-base model using DoRA and BiDoRA. The training and test curves for DoRA show a larger gap compared to BiDoRA, highlighting the effectiveness of our method in reducing overfitting.
Refer to caption: extracted/5922464/images/overfitting.png

Figure: (a) Full FT($k=-65.816$)
Refer to caption: extracted/5922464/images/experiments/wd/ft-query-variation.png

## 2 Related Work

### 2.1 Parameter Efficient fine-tuning Methods

Parameter-efficient fine-tuning (PEFT) methods aim to reduce the high costs associated with fully fine-tuning large-scale models by updating only a relatively small subset of pre-trained parameters, rather than the entire model, to adapt to downstream tasks.
Existing PEFT methods can be mainly categorized into three types.
The first category, known as adapter-based methods, injects additional trainable modules into the original frozen backbone.
For instance, Houlsby et al. () suggests adding linear modules in sequence to existing layers, while He et al. () proposes integrating these modules in parallel with the original layers to enhance performance.
The second category is prompt tuning methods, which add extra soft tokens (prompts) to the initial input. During the fine-tuning stage, only these trainable soft tokens are updated, as demonstrated in works such as Lester et al. () and Razdaibiedina et al. (). Unfortunately, the first two categories lead to increased inference latency compared to fully fine-tuned models.
The third category is low-rank adaptation methods, pioneered by the foundational work LoRA (Hu et al., ).
These methods attach low-rank matrices to pre-trained weights and use only these matrices for weight updates during fine-tuning.
Since low-rank updates can be merged with pre-trained weights before inference, low-rank adaptation-based PEFT methods do not increase inference time.
Following LoRA, Zhang et al. () applies SVD decomposition to low-rank matrices and prunes less significant singular values for more efficient updates.
Zhang et al. () uses meta-learning to search for the optimal rank of LoRA matrices, further improving its performance on downstream tasks.
Most recently,Liu et al. () uses weight decomposition analysis to reveal that LoRA exhibits a distinct weight updating pattern compared to FT, which may constrain its learning capacity.
Therefore, DoRA (Liu et al., ) was then proposed to bridge the gap between LoRA and FT.
DoRA decomposes the pre-trained weights into two components—magnitude and direction—and fine-tunes both, which results in a more closely aligned updating pattern compared to FT.

### 2.2 Bi-level Optimization

Bi-level optimization (BLO) has been widely applied in various machine learning tasks, including meta-learning (Finn et al., ; Rajeswaran et al., ), neural architecture search (Liu et al., ; Zhang et al., ), and hyperparameter optimization (Lorraine et al., ; Franceschi et al., ).
Despite its wide usage, solving BLO problems can be challenging due to the inherent nature of nested optimization problems.
Several algorithms have been proposed to address this challenge, including zeroth-order methods such as Bayesian optimization (Cui & Bai, ) and first-order algorithms based on hypergradients (Pearlmutter & Siskind, ; Lorraine et al., ).
Among these approaches, gradient-based BLO has received significant attention because it can scale to high-dimensional problems with a large number of trainable parameters.
In this work, we extend the application scenarios of gradient-based BLO to develop a robust and effective parameter-efficient fine-tuning method for pre-trained models.

## 3 Preliminary: Weight-Decomposed Low-Rank Adaptation

LoRA (Hu et al., ) involves attaching the product of two low-rank matrices to the pre-trained weights and fine-tuning these low-rank matrices on downstream datasets with the pre-trained weights frozen.
It is based on the assumption that parameter updates made during fine-tuning exhibit a low intrinsic rank.
Formally, given a pre-trained weight matrix $W_{0}\in\mathbb{R}^{d\times k}$, LoRA attaches a low-rank update matrix $\Delta W\in\mathbb{R}^{d\times k}$ to the pre-trained weight. This update matrix can be further decomposed as $\Delta W=BA$, where $B\in\mathbb{R}^{d\times r}$ and $A\in\mathbb{R}^{r\times k}$ are two low-rank matrices, with $r\ll\min(d,k)$.
Consequently, the weight matrix $W^{\prime}$ after LoRA fine-tuning can be represented as follows:

$$ $W^{\prime}=W_{0}+\Delta W=W_{0}+BA$ (1) $$

In this setup, the pre-trained weight matrix $W_{0}$ remains fixed during the fine-tuning process, while the LoRA matrix $\Delta W$ is updated. However, after performing weight decomposition on fine-tuned weight matrices, it was found that LoRA and full fine-tuning exhibit different learning patterns (Liu et al., ).
To bridge this discrepancy, weight-decomposed low-rank adaptation (DoRA, Liu et al. ()) further reparameterizes the weight matrices by explicitly decomposing them into learnable magnitude and direction components. Formally, DoRA performs adaption as follows:

$$ $W^{\prime}=m\frac{V+\Delta V}{\|V+\Delta V\|_{c}}=m\frac{W_{0}+BA}{\|W_{0}+BA \|_{c}}$ (2) $$

where the incremental direction component $\Delta V$ is parameterized as a product of two learnable low-rank matrices, $B$ and $A$, while the magnitude component $m\in\mathbb{R}^{1\times k}$ is a learnable vector.
Here, $\|\cdot\|_{c}$ represents the vector-wise norm of a matrix computed across each column.
In DoRA, both components are optimized concurrently on a single downstream dataset.

## 4 Methods

### 4.1 Overview of BiDoRA

Our method, BiDoRA, optimizes the trainable parameters in DoRA layers by solving a bi-level optimization problem.
Let $\mathcal{M}=\{m_{1},m_{2},\ldots,m_{n}\}$ denote the set of magnitude components for all $n$ DoRA modules, and $\mathcal{V}=\{\Delta V_{1},\Delta V_{2},\ldots,\Delta V_{n}\}$ denote the set of corresponding incremental direction components.
Specifically, we first learn the incremental direction components $\mathcal{V}^{*}(\mathcal{M})$ on the training split of the downstream dataset $\mathcal{D}_{tr}$ at the lower level.
The magnitude component $\mathcal{M}$ is tentatively fixed at this level, thus the resulting optimal incremental direction component $\mathcal{V}^{*}(\mathcal{M})$ is a function of $\mathcal{M}$. At the upper level, we determine the optimal magnitude component $\mathcal{M}^{*}$ by optimizing the loss on a validation split $\mathcal{D}_{val}$. In practice, $\mathcal{D}_{tr}$ and $\mathcal{D}_{val}$ are typically created by splitting the original training set without using additional data. This bi-level optimization problem is solved using an efficient gradient-based algorithm, where parameters in two levels are optimized iteratively until convergence.

### 4.2 Orthogonal Regularization

The orthogonality of neural network weights has been identified as a beneficial property (Bansal et al., ) and can effectively mitigate the overfitting issue (Balestriero & richard baraniuk, ).
Therefore, we define a Gram regularization loss (Xie et al., ) for the direction component:

$$ $\mathcal{R}(\mathcal{V})=\sum_{k=1}^{n}\left\lVert(V_{k}+\Delta V_{k})^{\top}( V_{k}+\Delta V_{k})-I\right\rVert_{F}^{2}$ (3) $$

where $I$ is the identity matrix and $\|\cdot\|_{F}$ denotes the Frobenius norm.
Intuitively, $\mathcal{R}(\mathcal{V})$ encourages each column of the direction matrix, representing a specific direction, to be orthogonal to one another.
Since each column has already been normalized (equivalent to projected to the unit sphere), this also prompts each column to be far away from the other, thereby reducing the redundancy of parameters.

### 4.3 A Bi-level Optimization Framework

#### Lower Level

At the lower level, we train the low-rank incremental direction component $\mathcal{V}$ by minimizing a loss $\mathcal{L}_{tr}$ defined on the training set $\mathcal{D}_{tr}$.
The overall training objective at this level is $\mathcal{L}_{tr}(\mathcal{V},\mathcal{M})=C(\mathcal{V},\mathcal{M};\mathcal{D
}_{tr})+\gamma\mathcal{R}(\mathcal{V})$. Here, $C$ represents the fine-tuning loss, given the low-rank incremental direction component $\mathcal{V}$, the magnitude component $\mathcal{M}$, and the training split $\mathcal{D}_{tr}$ of the downstream dataset.
$\mathcal{R}(\mathcal{V})$ is the orthogonal regularizer defined in Eq. (3), with $\gamma$ as a trade-off hyperparameter.
In this level, we only update $\mathcal{V}$ while keeping $\mathcal{M}$ fixed, resulting in the following optimization problem:

$$ $\displaystyle\mathcal{V}^{*}(\mathcal{M})=\arg\min_{\mathcal{V}}\>\mathcal{L}_ {tr}(\mathcal{V},\mathcal{M})$ (4) $$

where $\mathcal{V}^{*}(\mathcal{M})$ denotes the optimal solution for $\mathcal{V}$ in this problem, which is a function of $\mathcal{M}$.

#### Upper Level

At the upper level, we validate the previously fixed magnitudes $\mathcal{M}$ on the validation set $\mathcal{D}_{val}$, using the optimal incremental direction component $\mathcal{V}^{*}(\mathcal{M})$ that was learned at the lower level. This results in a validation loss $\mathcal{L}_{val}(\mathcal{V}^{*}(\mathcal{M}),\mathcal{M})=C(\mathcal{V}^{*}(
\mathcal{M}),\mathcal{M};\mathcal{D}_{val})$.
We determine the optimal magnitude component $\mathcal{M}$ by minimizing this validation loss:

$$ $\displaystyle\min_{\mathcal{M}}\mathcal{L}_{val}(\mathcal{V}^{*}(\mathcal{M}), \mathcal{M})$ (5) $$

#### A Bi-level Optimization Framework

Integrating the two levels of optimization problems, we have the following bi-level optimization framework:

$$ $\displaystyle\min_{\mathcal{M}}\>\mathcal{L}_{val}(\mathcal{V}^{*}(\mathcal{M} ),\mathcal{M})$ $\displaystyle s.t.\quad$ $\displaystyle\>\mathcal{V}^{*}(\mathcal{M})=\arg\min_{\mathcal{V}}\>\mathcal{L }_{tr}(\mathcal{V},\mathcal{M})$ (6) $$

Note that these two levels of optimization problems are mutually dependent on each other.
The solution of the optimization problem at the lower level, $\mathcal{V}^{*}(\mathcal{M})$, serves as a parameter for the upper-level problem, while the optimization variable $\mathcal{M}$ at the upper level acts as a parameter for the lower-level problem.
By solving these two interconnected problems jointly, we can learn the optimal magnitude component $\mathcal{M}^{*}$ and incremental direction matrices $\mathcal{V}^{*}$ in an end-to-end manner.

Two reasons exist behind the choice of setting the magnitude component as the upper level instead of the converse one: 1) In literature, the upper level usually has much fewer parameters than the lower level.
In our case, the design of setting the magnitude of complexity $\mathcal{O}(k)$ as the upper level and the direction of complexity $\mathcal{O}(dr+kr)$ as the lower level is consistent with the common practice.
2) BiDoRA resembles the DARTS method (Liu et al., ) in neural architecture search where the subnets are selected by a selection variable.
Specifically, the magnitude vector resembles a selection variable on the direction matrix by softly selecting each direction (subnets) via scaling.

#### Optimization Algorithm

Figure: Algorithm 1 BiDoRA

We use a gradient-based optimization algorithm (Choe et al., ) to solve the bi-level optimization problem presented in Eq. (4.3). A significant challenge in this process is that precisely computing the gradient of the upper-level loss $\mathcal{L}_{val}$ with respect to the magnitude component $\mathcal{M}$ can be computationally prohibitive due to the lack of an analytical solution for $\mathcal{V}^{*}(\mathcal{M})$ at the lower-level optimization problem. To address this issue, we use the following one-step-unrolled approximation of $\mathcal{V}^{*}(\mathcal{M})$ inspired by previous work (Liu et al., ):

$$ $\nabla_{\mathcal{M}}{\mathcal{L}_{val}(\mathcal{V}^{*}(\mathcal{M}),\mathcal{M })}\approx\nabla_{\mathcal{M}}{\mathcal{L}_{val}(\mathcal{V}-\xi\nabla_{ \mathcal{V}}{\mathcal{L}_{tr}(\mathcal{V},\mathcal{M})},\mathcal{M})}$ $$

where $\xi$ is the learning rate at the lower level, and the one-step-unrolled model $\bar{\mathcal{V}}=\mathcal{V}-\xi\nabla_{\mathcal{V}}{\mathcal{L}_{tr}(
\mathcal{V},\mathcal{M})}$ is used as a surrogate for the optimal solution $\mathcal{V}^{*}(\mathcal{M})$.
We then compute the approximated gradient as follows:

$$ $\displaystyle\nabla_{\mathcal{M}}{\mathcal{L}_{val}(\mathcal{V}-\xi\nabla_{ \mathcal{V}}{\mathcal{L}_{tr}(\mathcal{V},\mathcal{M})},\mathcal{M})}$ $\displaystyle=$ $\displaystyle\nabla_{\mathcal{M}}\mathcal{L}_{val}(\bar{\mathcal{V}},\mathcal{ M})-\xi\nabla^{2}_{\mathcal{M},\mathcal{V}}\mathcal{L}_{tr}(\mathcal{V}, \mathcal{M})\nabla_{\bar{\mathcal{V}}}\mathcal{L}_{val}(\bar{\mathcal{V}}, \mathcal{M})$ (7) $\displaystyle\approx$ $\displaystyle\nabla_{\mathcal{M}}\mathcal{L}_{val}(\bar{\mathcal{V}},\mathcal{ M})-\xi\frac{\nabla_{\mathcal{M}}\mathcal{L}_{tr}(\mathcal{V}^{+},\mathcal{M}) -\nabla_{\mathcal{M}}\mathcal{L}_{tr}(\mathcal{V}^{-},\mathcal{M})}{2\epsilon}$ (8) $$

where $\epsilon$ is a small scalar and $\mathcal{V}^{\pm}=\mathcal{V}\pm\epsilon\nabla_{\bar{\mathcal{V}}}{\mathcal{L}
_{val}(\bar{\mathcal{V}},\mathcal{M})}$. Since directly computing the matrix-vector multiplication term in Eq. (7) is computationally expensive, we use finite difference to approximate this product as in Eq. (8), following Liu et al. ().
As detailed in Algorithm 1, the incremental direction component $\mathcal{V}$ and the magnitude component $\mathcal{M}$ are updated using gradient descent iteratively until convergence. After acquiring the optimal magnitudes $\mathcal{M}^{*}$ through the process above, the incremental direction component $\mathcal{V}$ is retrained on the union of training and validation splits to achieve the best performance on downstream tasks, resulting in the final learned $\mathcal{V}^{*}$.

## 5 Experiments

### 5.1 Experimental Setup

We compare BiDoRA with several PEFT methods, including Adapter tuning (Houlsby et al., ), LoRA (Hu et al., ), and DoRA (Liu et al., ).
BiDoRA does not use any additional data compared to other baselines, as we create the validation set for upper-level optimization by splitting the original training set with an $8$:$2$ ratio for all tasks.
All methods in the experiment, including ablation studies, are trained until convergence for a fair comparison.
Detailed descriptions of these baseline methods are provided in Appendix C.

**Table 1: RoBERTabase/large (Rb/l) with different fine-tuning methods on the GLUE benchmark. A higher value is better for all datasets. The best results are shown in bold.**
| Method | Param | MNLI | SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Rb(FT) | $125.0$M | $90.3$ | $94.8$ | $89.3$ | $61.6$ | $86.7$ | $92.8$ | $76.9$ | $91.2$ | $85.5$ |
| Rb(Adapter) | $0.9$ M | $86.5$ | $94.0$ | $88.4$ | $58.8$ | $92.5$ | $89.1$ | $71.2$ | $89.9$ | $83.8$ |
| Rb(LoRA) | $0.15$ M | $86.8$ | $94.3$ | $88.0$ | $60.3$ | 93.0 | $89.6$ | $72.9$ | $90.1$ | $84.4$ |
| Rb(DoRA) | $0.17$ M | $86.8$ | $94.2$ | $89.2$ | $60.5$ | $92.9$ | $89.6$ | $73.2$ | 90.2 | $84.6$ |
| Rb(BiDoRA) | $0.17$ M | 87.1 | 94.4 | 89.4 | 61.3 | $92.7$ | $90.6$ | 76.0 | $90.1$ | 85.2 |
| Rl(FT) | $355.0$M | $90.2$ | $96.4$ | $90.9$ | $68.0$ | $94.7$ | $92.2$ | $86.6$ | $92.4$ | $88.9$ |
| Rl(Adapter) | $0.8$M | $90.3$ | $96.3$ | $87.7$ | $66.3$ | 94.7 | $91.5$ | $72.9$ | $91.5$ | $86.4$ |
| Rl(LoRA) | $0.39$ M | 90.6 | $96.3$ | $90.0$ | $66.9$ | $94.5$ | $91.2$ | $86.3$ | $91.7$ | $88.4$ |
| Rl(DoRA) | $0.39$ M | 90.6 | 96.4 | $89.8$ | $65.8$ | 94.7 | $91.2$ | $86.6$ | 92.0 | $88.4$ |
| Rl(BiDoRA) | $0.39$ M | 90.6 | $96.1$ | 90.1 | 67.0 | $94.6$ | 91.7 | 86.9 | 92.0 | 88.6 |

**Table 2: RoBERTabase/large (Rb/l) with different fine-tuning methods on the Reuters21578 benchmark. A higher value is better for all datasets. The best results are shown in bold.**
| Method | Param | ModApte | ModHayes | ModLewis |
| --- | --- | --- | --- | --- |
| Rb(FT) | $125.0$M | $85.4$ | $77.6$ | $77.1$ |
| Rb(Adapter) | $0.9$ M | 85.3 | $77.5$ | $76.8$ |
| Rb(LoRA) | $0.15$ M | $84.7$ | $74.3$ | $74.7$ |
| Rb(DoRA) | $0.17$ M | $84.8$ | $78.2$ | $76.6$ |
| Rb(BiDoRA) | $0.17$ M | 85.3 | 79.9 | 77.6 |
| Rl(FT) | $355.0$M | $84.8$ | $77.5$ | $76.6$ |
| Rl(Adapter) | $0.44$ M | $84.8$ | $77.9$ | $76.7$ |
| Rl(LoRA) | $0.39$ M | $84.7$ | $77.7$ | $76.7$ |
| Rl(DoRA) | $0.39$ M | $84.8$ | $77.4$ | $76.7$ |
| Rl(BiDoRA) | $0.39$ M | 84.9 | 78.9 | 77.3 |

Our experiments covere a wide range of tasks, including natural language understanding (NLU), natural language generation (NLG), and token classification.
For NLU tasks, we fine-tune the RoBERTa-base and RoBERTa-large models on the GLUE benchmark (Wang et al., ) and the Reuters21578 dataset (Padmanabhan et al., ) using all baseline PEFT methods and BiDoRA. Detailed descriptions of these datasets and pre-trained models are provided in Appendix A.
Following existing practices, the development set is used in GLUE as the test data since the actual test set is not publicly available.
We report the overall (matched and mismatched) accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for the other tasks.
On the Reuters21578 dataset, the F1 score is used as the evaluation metric across all three splits.
For NLG tasks, we fine-tune GPT-2 medium on the E2E (Novikova et al., ) dataset.
We use BLEU (Papineni et al., ), NIST (Lin & Och, ), METEOR (Banerjee & Lavie, ), ROUGE-L (Lin, ), and CIDEr (Vedantam et al., ) as evaluation metrics. For token classification, we fine-tune the RoBERTa-base and RoBERTa-large models on the BioNLP (Collier et al., ) dataset and the CoNLL2003 (Sang & De Meulder, ) dataset.
Accuracy, precision, recall, and F1 score are used as evaluation metrics.

For all experiments, our implementation is based on the Huggingface Transformers library (Wolf et al., ) and the Betty library (Choe et al., ). We use a single NVIDIA A100 GPU for all experiments.
More detailed experimental settings are provided in Appendix B.

### 5.2 Experiments on Natural Language Understanding Tasks

In this section, we evaluate the performance of BiDoRA on NLU tasks, with a particular focus on text classification.
Table 1 presents the results of fine-tuning the RoBERTa-base and RoBERTa-large models on the GLUE benchmark with baseline PEFT methods and BiDoRA. The results show that BiDoRA achieves superior or comparable performance compared to baseline methods across all datasets with the same number of trainable parameters.
Table 2 presents the results of fine-tuning RoBERTa models on the Reuters21578 datasets, where BiDoRA outperforms all baseline methods by an even larger margin. Notably, BiDoRA achieves performance comparable to or even better than full fine-tuning.
The superior performance of BiDoRA on both benchmarks verifies the effectiveness of its bi-level optimization mechanism.
By training the magnitude and incremental direction components on two distinct sub-datasets, BiDoRA enhances the flexibility of the learning process and improves learning capacity compared to DoRA, resulting in a performance boost.

**Table 3: Performance of BiDoRA and baseline methods for fine-tuning GPT2-medium on the E2E dataset. A higher value is better for all metrics. The best results are shown in bold.**
| Method | Param | BLEU | NIST | MET | ROUGE-L | CIDEr |
| --- | --- | --- | --- | --- | --- | --- |
| Full FT | $354.9$M | $68.0$ | $8.61$ | $46.1$ | $69.0$ | $2.38$ |
| Adapter | $11.1$M | $67.0$ | $8.50$ | $45.2$ | $66.9$ | $2.31$ |
| LoRA | $0.39$M | $63.7$ | $8.19$ | $44.1$ | $68.0$ | $2.17$ |
| DoRA | $0.39$M | $67.0$ | $8.48$ | $45.4$ | $70.1$ | $2.33$ |
| BiDoRA | $0.39$M | 69.0 | 8.72 | 46.2 | 70.9 | 2.44 |

### 5.3 Experiments on Natural Language Generation Tasks

In this section, we evaluate BiDoRA’s performance on the NLG task.
Table 3 presents the results of fine-tuning a GPT-2 model on the E2E dataset with baseline PEFT methods and BiDoRA. The results show that BiDoRA achieves the best performance across all five evaluation metrics, demonstrating the superiority of BiDoRA in fine-tuning pre-trained models for NLG tasks.

### 5.4 Experiments on Token Classification

**Table 4: RoBERTabase/large (Rb/l) with different fine-tuning methods on BioNLP data and CoNLL2003 dataset. A higher value is better for all metrics. The best results are shown in bold.**
|  |  | BioNLP | CoNLL2003 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Param | Accuracy | Precision | Recall | F1 | Accuracy | Precision | Recall | F1 |
| Rb(FT) | $125.0$M | $93.9$ | $69.0$ | $78.9$ | $73.6$ | $99.3$ | $95.7$ | $96.3$ | $96.0$ |
| Rb(Adapter) | $0.9$ M | $93.9$ | $69.1$ | $78.8$ | $73.7$ | 99.3 | $95.7$ | $96.4$ | $96.0$ |
| Rb(LoRA) | $0.15$ M | $93.9$ | $69.0$ | $78.8$ | $73.6$ | 99.3 | $95.4$ | $96.3$ | $95.8$ |
| Rb(DoRA) | $0.17$ M | 94.0 | $69.2$ | 79.1 | $73.8$ | 99.3 | $95.3$ | $96.2$ | $95.8$ |
| Rb(BiDoRA) | $0.17$ M | $93.9$ | 71.2 | $78.6$ | 74.7 | 99.3 | 95.9 | 96.5 | 96.2 |
| Rl(FT) | $355.0$M | $94.0$ | $69.4$ | $79.6$ | $74.1$ | $99.4$ | $96.2$ | $97.0$ | $96.6$ |
| Rl(Adapter) | $0.44$ M | 94.0 | $69.4$ | $79.7$ | $74.2$ | 99.4 | $96.1$ | $97.0$ | $96.6$ |
| Rl(LoRA) | $0.39$ M | $93.9$ | $69.2$ | $79.3$ | $73.9$ | 99.4 | $96.2$ | $97.0$ | $96.6$ |
| Rl(DoRA) | $0.39$ M | 94.0 | $69.4$ | 79.7 | $74.2$ | 99.4 | $96.2$ | 97.1 | $96.6$ |
| Rl(BiDoRA) | $0.39$ M | 94.0 | 71.3 | $79.3$ | 75.1 | 99.4 | 96.4 | 97.1 | 96.7 |

Further evidence of the effectiveness of BiDoRA can be observed in Table 4, which reports the results of token classification tasks.
Unlike the NLU tasks discussed in the previous section, which involve classifying entire sentences and focusing on capturing global semantics, token classification requires classifying each token within a sentence, highlighting the importance of capturing local context.
On the BioNLP dataset, BiDoRA consistently outperforms baseline methods by a large margin in terms of F1 score.
On the CoNLL2003 dataset, BiDoRA either outperforms or matches all baseline methods across all metrics.
Consistent with our previous findings, BiDoRA effectively fine-tunes pre-trained models for token classification tasks.

### 5.5 Ablation Studies

In this section, we perform ablation studies to investigate the effectiveness of individual modules or strategies in BiDoRA. We fine-tune a RoBERTa-base model on the GLUE benchmark under different ablation settings, and the results are shown in Table 5.

Retraining
We test the model directly obtained from the search phase to evaluate the effectiveness of further retraining the incremental direction component.
The results show that BiDoRA outperforms BiDoRA (w/o retraining) on average, highlighting the necessity of retraining.

Bi-level Optimization
We set $\xi$ to zero in Algorithm 1 to assess the effectiveness of the bi-level optimization framework.
This ablation setting can be interpreted as an alternative learning method where two optimization steps are carried out alternately on two different splits of the training dataset.
Notably, in the alternative learning method, the updating of each component is unaware of each other, making the training less stable.
In contrast, the hyper-gradient used in bi-level optimization avoids this issue by connecting the two levels in a certain way.
The results show that BiDoRA outperforms BiDoRA ($\xi=0$) on average, demonstrating the efficacy of the bi-level optimization strategy.

Orthogonal Regularization
We examine the effectiveness of the orthogonality constraint in Eq. (3) by setting $\gamma$ to zero.
Results show that BiDoRA outperforms BiDoRA (w/o cst.) on average, indicating the effectiveness of applying the orthogonality regularizer to alleviate overfitting.

**Table 5: Ablation studies. We evaluate the performance of BiDoRA without retraining (w/o retraining), without bi-level optimization ($\xi=0$) and without orthogonal regularization (w/o cst.).**
| Method | MNLI | SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BiDoRA (w/o retraining) | $87.0$ | $94.2$ | $89.0$ | $57.3$ | $92.4$ | $90.6$ | $71.6$ | $90.0$ | $84.0$ |
| BiDoRA ($\xi=0$) | $86.9$ | $94.2$ | $89.0$ | $59.4$ | $90.8$ | 91.2 | $75.9$ | $90.0$ | 84.7 |
| BiDoRA (w/o cst.) | $87.0$ | 94.4 | $88.6$ | 61.3 | 92.7 | $90.2$ | $76.0$ | 90.1 | $85.0$ |
| BiDoRA | 87.1 | 94.4 | 89.4 | 61.3 | 92.7 | $90.6$ | 76.1 | 90.1 | 85.2 |

### 5.6 Weight Decomposition Analysis

One important motivation of DoRA is to bridge the inherent differences between LoRA and FT.
Similar to DoRA, we conduct a weight decomposition analysis on the correlation between the change of magnitudes and that of directions (detailed in Appendix E) for BiDoRA and baseline methods by fine-tuning a GPT2-medium model on the E2E dataset.
As shown in Figure 3, FT, DoRA, and BiDoRA all exhibit negative correlation values, while LoRA shows a positive correlation, consistent with the findings in Liu et al. ().
Notably, BiDoRA achieves a negative correlation of $-8.042$, closer to FT than DoRA’s $-1.784$.
This improvement is attributed to the decoupled training process of the two layers, which allows for a higher learning capacity compared to DoRA.

### 5.7 Computation Costs

**Table 6: Average training time cost on the MNLI, QQP, and SST-2 datasets.**
| Method | LoRA | DoRA | BiDoRA |
| --- | --- | --- | --- |
| Cost | $\times 1$ | $\times 1.30$ | $\times 3.92$ |

Since BiDoRA has the same architecture as DoRA, our method only requires two extra forward and backward passes of the lower level for the hypergradient calculation of the upper level, as shown in Eq. 8.
In theory, this would make our method roughly threefold computationally costly.
A similar analysis holds for the memory consumption analysis.
Empirically, table 6 shows the average training cost of BiDoRA and two baseline methods on the MNLI, QQP, and SST-2 datasets from the GLUE benchmark, being consistent with the theoretical analysis.
We normalize the cost of LoRA to $1$ for reference.
Given BiDoRA’s superior performance across various tasks, the increase in computational costs is acceptable in most cases, underscoring its practicality.

## 6 Conclusion

We propose BiDoRA, a novel bi-level optimization framework for parameter-efficient fine-tuning of large-scale pre-trained models.
By conducting weight decomposition following the DoRA approach, our method trains the two components separately in two interconnected optimization levels using different sub-datasets.
In this way, BiDoRA not only decouples the learning process of the two components, resulting in a learning pattern closer to FT, but also effectively alleviates overfitting. Empirical studies on various NLP tasks demonstrate that BiDoRA outperforms DoRA and other baselines, highlighting the effectiveness of our method.

## Appendix A Datasets and Models

### A.1 Natural Language Understanding

The GLUE Benchmark comprises a diverse array of tasks that are widely employed for evaluation in natural language understanding.
It encompasses two single-sentence classification tasks, three tasks assessing similarity and paraphrasing, and four tasks focusing on natural language inference. Specifically, it includes MNLI (MultiNLI, Williams et al. ()), SST-2 (Stanford Sentiment Treebank, Socher et al. ()), MRPC (Microsoft Research Paraphrase Corpus, Dolan & Brockett ()), CoLA (Corpus of Linguistic Acceptability, Warstadt et al. ()), QNLI (Question NLI, Rajpurkar et al. ()), QQP (Quora Question Pairs), RTE (Recognizing Textual Entailment), and STS-B (Semantic Textual Similarity Benchmark, Cer et al. ()). We summarize the statistical data for all datasets within the GLUE Benchmark in Table 7.

The Reuters-21578 (Padmanabhan et al., ) dataset is one of the most widely used data collections for text categorization research. It was collected from the Reuters financial newswire service in 1987 and is used for text classification and natural language processing tasks. Three splits are available: ModApte, ModHayes, and ModLewis. These documents cover various topics, such as politics, economics, and sports. We summarize the statistical data for all text classification tasks used in our experiments in Table 8.

**Table 7: The statistical data for all datasets within the GLUE Benchmark.**
| Dataset | Metrics | Train | Dev | Test | Label | Task |
| --- | --- | --- | --- | --- | --- | --- |
| MNLI | Accuracy | 393k | 20k | 20k | 3 | NLI |
| SST-2 | Accuracy | 67k | 872 | 1.8k | 2 | Sentiment |
| MRPC | Accuracy | 3.7k | 408 | 1.7k | 2 | Paraphrase |
| CoLA | Matthews Corr | 8.5k | 1k | 1k | 2 | Acceptability |
| QNLI | Accuracy | 108k | 5.7k | 5.7k | 2 | QA/NLI |
| QQP | Accuracy | 364k | 40k | 391k | 2 | Paraphrase |
| RTE | Accuracy | 2.5k | 276 | 3k | 2 | NLI |
| STS-B | Pearson Corr | 7.0k | 1.5k | 1.4k | 1 | Similarity |

**Table 8: The statistical data for the Reuters-21578 dataset.**
| Dataset | Metrics | Train | Test |
| --- | --- | --- | --- |
| ModApte | F1 | 8.8k | 3k |
| ModHayes | F1 | 18k | 0.7k |
| ModLewis | F1 | 12k | 5.5k |

### A.2 Natural Language Generation

In our experiments on natural language generation, we use the E2E (Novikova et al., ) dataset, which was initially introduced as a dataset for training end-to-end, data-driven natural language generation systems.
Multiple references can be associated with each source table used as input. Each sample input $(x,y)$ consists of a series of slot-value pairs accompanied by an associated natural language reference text.
The E2E dataset comprises approximately 42,000 training examples, 4,600 validation examples, and 4,600 test examples from the restaurant domain.

We utilize the following five evaluation metrics: BLEU (Papineni et al., ), NIST (Lin & Och, ), METEOR (Banerjee & Lavie, ), ROUGE-L (Lin, ), and CIDEr (Vedantam et al., ).
We summarize its statistical data in Table LABEL:tab:datasetnlg.

**Table 9: The statistical data for E2E dataset.**
| Dataset | Metrics | Train | Validation |
| --- | --- | --- | --- |
| E2E | BLEU,NIST,MET,ROUGE-L,CIDEr | 42k | 4.6k |

### A.3 Token Classification

BioNLP (Collier et al., ) is a Named Entity Recognition dataset that contains biological entities such as DNA, RNA, and protein.
It is essentially a token classification task where we want to classify each entity in the sequence.
CoNLL-2003 (Sang & De Meulder, ) focuses on language-independent named entity recognition.
It concentrates on four types of named entities: persons, locations, organizations, and miscellaneous entities that do not belong to the previous three groups.
We summarize the statistical data for all used token classification tasks in Table 10.

**Table 10: The statistical data for token classification tasks**
| Dataset | Metrics | Train | Validation | Test |
| --- | --- | --- | --- | --- |
| BioNLP | Accuracy,Precision,Recall,F1 | 17k | 1.9k | 3.9k |
| CoNLL2003 | Accuracy,Precision,Recall,F1 | 14k | 3.3k | 3.5k |

## Appendix B Experimental Settings

In this section, we provide detailed experimental settings.
We maintain consistent configurations across experiments, including LoRA rank, LoRA $\alpha$, batch size, maximum sequence length, and optimizer, to ensure a fair comparison.
The hyperparameter tuning for our method is straightforward and convenient.

### B.1 RoBERTa

We summarize the experimental settings for the GLUE benchmark in Table 11 and for the Reuters21578 dataset and token classification tasks in Table 12.

**Table 11: The hyperparameters we used for RoBERTa on the GLUE benchmark.**
| Method | Settings | MNLI | SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Optimizer | AdamW |  |  |  |  |  |  |  |
|  | Warmup Ratio | $0.06$ |  |  |  |  |  |  |  |
|  | Scheduler | Linear |  |  |  |  |  |  |  |
|  | LoRA rank | ranka = ranku = $4$ |  |  |  |  |  |  |  |
|  | LoRA $\alpha$ | $8$ |  |  |  |  |  |  |  |
| RoBERTa-base | Total batch size | $32$ |  |  |  |  |  |  |  |
|  | Global steps | $20000$ | $12000$ | $25000$ | $20000$ | $15000$ | $20000$ | $15000$ | $12000$ |
|  | Lower learning rate | 5e-5 | 1e-5 | 2e-5 | 5e-5 | 2e-5 | 5e-5 | 1e-5 | 1e-5 |
|  | Upper learning rate | 5e-5 | 1e-5 | 2e-5 | 5e-5 | 2e-5 | 5e-5 | 1e-5 | 1e-5 |
|  | Lower weight decay | $0.1$ |  |  |  |  |  |  |  |
|  | Upper weight decay | $0.1$ | $0.1$ | $0.1$ | $0.1$ | 0 0 | $0.1$ | $0.1$ | $0.01$ |
|  | Max Seq Length | $512$ |  |  |  |  |  |  |  |
|  | Regularization Coefficient | 1e-5 |  |  |  |  |  |  |  |
| RoBERTa-large | Total batch size | $32$ |  |  |  |  |  |  |  |
|  | Global steps | $50000$ | $20000$ | $30000$ | $20000$ | $60000$ | $40000$ | $15000$ | $10000$ |
|  | Lower learning rate | 1e-5 |  |  |  |  |  |  |  |
|  | Upper learning rate | 1e-5 |  |  |  |  |  |  |  |
|  | Lower weight decay | $0.5$ | $0.5$ | 0 0 | $0.2$ | $0.5$ | $0.5$ | $0.5$ | $0.5$ |
|  | Upper weight decay | $0.5$ | $0.05$ | 0 0 | $0.2$ | $0.5$ | $0.5$ | $0.1$ | $0.5$ |
|  | Max Seq Length | $128$ |  |  |  |  |  |  |  |
|  | Regularization Coefficient | 0 | 0 | 1e-5 | 1e-5 | 0 | 1e-5 | 0 | 1e-5 |

**Table 12: The hyperparameters we used for RoBERTa on the Reuters21578 dataset, BioNLP dataset, and CoNLL2003 dataset.**
| Method | Settings | ModApte | ModHayes | ModLewis | BioNLP | CoNLL2003 |
| --- | --- | --- | --- | --- | --- | --- |
|  | Optimizer | AdamW |  |  |  |  |
|  | Warmup Ratio | $0.06$ |  |  |  |  |
|  | Scheduler | Linear |  |  |  |  |
|  | LoRA rank | ranka = ranku = $4$ |  |  |  |  |
|  | LoRA $\alpha$ | $8$ |  |  |  |  |
| RoBERTa-base | Total batch size | $32$ |  |  |  |  |
|  | Global steps | $20000$ | $20000$ | $20000$ | $12000$ | $12000$ |
|  | Lower learning rate | 3e-5 | 3e-5 | 3e-5 | 1e-5 | 2e-5 |
|  | Upper learning rate | 3e-5 | 3e-5 | 3e-5 | 1e-5 | 2e-5 |
|  | Lower weight decay | $0.1$ | $0.1$ | $0.1$ | $0.1$ | $0.2$ |
|  | Upper weight decay | $0.1$ |  |  |  |  |
|  | Max Seq Length | $512$ |  |  |  |  |
|  | Regularization Coefficient | 0 | 1e-5 | 0 | 1e-5 | 0 |
| RoBERTa-large | Total batch size | $32$ |  |  |  |  |
|  | Global steps | $20000$ | $20000$ | $20000$ | $12000$ | $15000$ |
|  | Lower learning rate | 1e-5 | 1e-5 | 1e-5 | 2e-5 | 1e-5 |
|  | Upper learning rate | 1e-5 | 1e-5 | 1e-5 | 2e-5 | 1e-5 |
|  | Lower weight decay | $0.2$ | $0.1$ | $0.2$ | $0.02$ | $0.1$ |
|  | Upper weight decay | $0.1$ | $0.1$ | $0.1$ | $0.02$ | $0.1$ |
|  | Max Seq Length | $128$ |  |  |  |  |
|  | Regularization Coefficient | 0 | 1e-5 | 0 | 0 | 1e-5 |

### B.2 GPT-2

We summarize the experimental settings for the GPT-2 experiments in Table 13.
The experimental configuration, particularly during the inference stage, follows the approach described by Hu et al. ().

**Table 13: The hyperparameters we used for GPT-2 on the E2E NLG benchmark.**
| Settings | Training |
| --- | --- |
| Optimizer | AdamW |
| Warmup Ratio | $0.06$ |
| Scheduler | Linear |
| LoRA rank | ranka = ranku = $4$ |
| LoRA $\alpha$ | $32$ |
| Label Smooth | $0.1$ |
| Lower learning rate | 1e-3 |
| Upper learning rate | 1e-4 |
| Lower weight decay | $1$ |
| Upper weight decay | $1$ |
| Max Seq Length | $512$ |
| Regularization Coefficient | 1e-5 |
| Settings | Inference |
| Beam Size | $10$ |
| Length Penalty | $0.9$ |
| no repeat ngram size | $4$ |

## Appendix C Baselines in Experiments

We compare BiDoRA with Full Fine-Tuning (FT), Adapter tuning (Houlsby et al., ), LoRA (Hu et al., ), and DoRA (Liu et al., ) in all our experiments.
We provide a brief introduction to these methods here.

Full Fine-Tuning (FT) is a commonly used method for adaptation. The model is initialized with pre-trained weights and biases, and all model parameters are updated through gradient descent.

Adapter tuning (Houlsby et al., ) inserts layer adapters between neural modules, such as the MLP module or the self-attention module. It incorporates two fully connected layers within an adapter layer, with a nonlinearity function applied between them.

LoRA (Hu et al., ) adds trainable incremental update matrices to pre-trained weight matrices. Following the experimental settings of LoRA, we applied BiDoRA to $W_{q}$ and $W_{v}$ matrices (the query and value weight matrices in the self-attention module) for a fair comparison.

DoRA (Liu et al., ) proposes weight-decomposed adaptation, which formulates the incremental matrices as a product of magnitude and direction components, thereby accelerating training and aligning the training behavior with full fine-tuning. In contrast, our BiDoRA trains the two components on distinct sub-datasets to alleviate overfitting.

## Appendix D Theoretical Insights

We can gain several insights into how BiDoRA can alleviate the overfitting issue by analyzing both the gradient of the bi-level optimization and theoretical results from existing work.

In terms of gradient analysis, we can demonstrate the robustness of our method by examining the behavior of the loss $\mathcal{L}_{val}^{\prime}(\mathcal{V},\mathcal{M})=\mathcal{L}_{val}(\mathcal
{V}-\xi\nabla_{\mathcal{V}}{\mathcal{L}_{tr}(\mathcal{V},\mathcal{M})},
\mathcal{M})$.
We analyze its gradient with respect to the magnitude parameters $\mathcal{M}$ using the chain rule,

$$ $\displaystyle\frac{d\mathcal{L}_{val}^{\prime}}{d\mathcal{M}}=\frac{\partial \mathcal{L}_{val}^{\prime}}{\partial\mathcal{M}}+\frac{\partial\bar{\mathcal{V }}}{\partial\mathcal{M}}\times\frac{\partial\mathcal{L}_{val}^{\prime}}{ \partial\bar{\mathcal{V}}}$ (9) $$

The second term on the right-hand side, particularly the term $\frac{\partial\bar{\mathcal{V}}}{\partial\mathcal{M}}$, referred to as the best-response Jacobian in the literature, captures how the direction component would react to changes in the magnitude. More specifically, the update of $\mathcal{M}$ must consider not only the current value of the direction $\mathcal{V}$ but also additional information about how the direction responds to changes in the magnitude.
This facilitates finding a globally optimal magnitude value, thereby enhancing stability and robustness. Consequently, the model becomes more resilient to overfitting.
In contrast, if $\xi$ is set to 0 0, Algorithm 1 reduces to a first-order approximation, which performs coordinate descent for $\mathcal{M}$ and $\mathcal{V}$ on different empirical distributions.
In this scenario, the second term in Eq. (9) vanishes, making the approach insufficient to effectively alleviate overfitting.

On the other hand, we can also see the generalization guarantees of BiDoRA using various theoretical works in the literature (Oymak et al., ; Bai et al., ; Bao et al., ; Chen et al., ; Huang et al., ).
For example, Theorem 1 in Oymak et al. () provides strong evidence of the benefits of applying bi-level optimization to alleviate overfitting.
Oymak et al. () discussed the generalization guarantees for neural architecture search with a train-validation split.
They assumed the same scenario as ours: using bi-level optimization where one optimizes the weights over the training data (lower-level problem) and hyperparameters over the validation data (upper-level problem).
The theorem shows that as soon as the size of the validation data exceeds the effective number of hyperparameters (up to logarithmic factors), the test error is close to the validation error (i.e., validation error is indicative of the test error).
In practice, the number of magnitude parameters is generally less than or comparable to the size of the validation set, making this assumption feasible.
We empirically show that validation performance and test performance align, as theoretically supported by Oymak et al. ().
In Figure 4, we plot the performance on the training, validation, and test datasets for the CoLA and SST-2 datasets during the search phase. The first observation is that training performance always overfits until reaching zero error, whereas validation performance closely tracks test performance. This training behavior aligns with the fact that large-capacity networks can perfectly fit (Du et al., ; Ji & Telgarsky, ; Oymak & Soltanolkotabi, ) in the overparameterized setting, where the available data (less than $10$k for CoLA and $67$k for SST-2) is less than the number of learnable parameters (around $170$k).
To truly find a model that achieves good generalization performance, the optimization procedure should evaluate the generalization loss.
Thus, as shown in Figure 4, the validation phase serves as a crucial test proxy in the overparameterized setting where training performance may not be indicative of test performance.

Figure: Figure 4: Train/Validation/Test performance during the search phase of fine-tuning the RoBERTa-base model on the CoLA dataset (left) and SST-2 (right).
Refer to caption: extracted/5922464/images/analysis/ga_cola.png

## Appendix E Weight Decomposition Analysis

We provide a brief review of the weight decomposition analysis proposed in Liu et al. ().
Define the weight decomposition of a weight matrix $W\in\mathbb{R}^{d\times k}$ (e.g., query matrix in an attention layer) as $W=m\frac{V}{\|V\|_{c}}=\|W\|_{c}\frac{W}{\|W\|_{c}}$,
where $m\in\mathbb{R}^{1\times k}$ is the magnitude vector, and $V\in\mathbb{R}^{d\times k}$ is the directional matrix, with $\|\cdot\|_{c}$ representing the vector-wise norm of a matrix across each column. This decomposition ensures that each column of $V/\|V\|_{c}$ remains a unit vector, and the corresponding scalar in $m$ defines the magnitude of each vector.
Liu et al. () examine the magnitude and directional variations between $W_{0}$ and $W_{\mathrm{FT}}$, defined as
$\Delta M_{\mathrm{FT}}^{t}=\frac{\sum_{n=1}^{k}|m_{\mathrm{FT}}^{n,t}-m_{0}^{n
}|}{k}$ and
$\Delta D_{\mathrm{FT}}^{t}=\frac{\sum_{n=1}^{k}(1-\cos(V_{\mathrm{FT}}^{n,t},W
_{0}^{n}))}{k}$.
Here, $\Delta M_{\mathrm{FT}}^{t}$ and $\Delta D_{\mathrm{FT}}^{t}$ represent the magnitude and direction differences between $W_{0}$ and $W_{\mathrm{FT}}$ at the $t$-th training step, respectively, with $\cos(\cdot,\cdot)$ denoting cosine similarity.
$m_{\mathrm{FT}}^{n,t}$ and $m_{0}^{n}$ are the $n^{th}$ scalars in their respective magnitude vectors, while $V_{\mathrm{FT}}^{n,t}$ and $W_{0}^{n}$ are the $n^{th}$ columns in $V_{\mathrm{FT}}^{t}$ and $W_{0}$.
Intuitively, a consistent positive slope trend across all the intermediate steps implies a difficulty in concurrent learning of both magnitude and direction, suggesting that slight directional changes are challenging to execute alongside more significant magnitude alterations.
In contrast, a relatively negative slope signifies a more varied learning pattern, with a more pronounced negative correlation indicating a larger learning capacity.

Complementary to Figure 3 in the main paper on the query matrix, we provide additional results of weight decomposition analysis in Figure 5 on the value matrix to complement the findings in Section 5.6.
We can draw two key observations from Figure 5: 1) Consistent with the results in Liu et al. (), both FT and DoRA exhibit negative correlation values of $-49.279$ and $-5.485$, respectively, while LoRA shows a positive correlation with a value of $2.503$. 2) BiDoRA achieves a negative correlation value of $-10.547$, indicating closer alignment with FT compared to DoRA.
The analysis of how BiDoRA achieves this improvement is similar to that discussed in Section 5.6.

Figure: (a) FT($k=-49.279$)
Refer to caption: extracted/5922464/images/experiments/e2e/ft-value-variation.png

## Appendix F The Role of Hyperparameter

The hyperparameter tuning for BiDoRA is simple, convenient, and straightforward.
We further conducted experiments regarding the dataset partition of $\mathcal{D}_{tr}$ and $\mathcal{D}_{val}$ to provide insights into its role in BiDoRA.

The dataset partition helps maintain the balance of inner/outer optimization by assigning different portions of data.
The direction component has more trainable parameters, so it is reasonable to use more data for training the lower level while using the remaining data for training magnitudes.
As shown in Table 14, We varied the inner-level dataset $\mathcal{D}_{tr}$ partition from $0.6$ to $1.0$ with $0.1$ intervals and experimented with RoBERTa-base on three splits of the Reuters21578 dataset to examine its influence.

The results indicate that both extreme cases are negative to the overall performance.
When the inner partition is too small ($\leq 0.6$), directions are not well-trained, and when the inner partition is $1.0$, magnitudes are not trained at all, leading to a significant performance drop.
These findings demonstrate that bi-level optimization is effective in the sense that both levels are necessary for enhancing performance.
Although tuning the partition ratio may further improve overall performance, we maintain a consistent data partition of $8$:$2$ in all the experiments for simplicity.
A fixed configuration of data partition already consistently yields superior performance of BiDoRA, demonstrating that our method is robust to this hyperparameter within a certain range.

**Table 14: Experiment results on different data partitions of BiDoRA**
| Partition | ModApte | ModHayes | ModLewis |
| --- | --- | --- | --- |
| $0.6$ | $85.32$ | $79.76$ | $77.69$ |
| $0.7$ | $85.32$ | 80.01 | 77.74 |
| $0.8$ | 85.34 | $79.93$ | $77.63$ |
| $0.9$ | $85.27$ | $79.85$ | $77.64$ |
| $1.0$ | $85.23$ | $79.59$ | $77.42$ |

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: None
soundness: 2
strengths: - In experiments, the proposed method demonstrates consistently superior results compared to the others.
- The paper was written reasonably clearly, so I was able to follow the equations.
summary: This paper proposes a method for parameter efficient fine tuning using bi-level optimization that competes with LoRA. Results test it on training RoBERTa and GPT-2 medium for NLU and NLG tasks respectively. The overall results demonstrate consistently better results than LoRA and DoRA.
weaknesses: - The motivation for the bi-level optimization approach was a bit vague. I did read the explanation in the introduction, but it says "Furthermore, in DoRA, the magnitude and incremental direction components are optimized concurrently, leading to a highly constrained updating pattern that may overlook the diverse learning patterns required for different downstream tasks." It is not clear what this means concretely...
- Experiments were on models that do not represent the state of the art at this point. It would be much more convincing if the methods were demonstrated to work on more powerful models (e.g. at least 7B).
- As noted in 5.7, the proposed method is accompanied with significant additional training time, which may limit its practical usefulness.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: The results in Table 3 do not align with those reported in the original LoRA paper. In the original paper, LoRA achieves a BLEU score of 70.4 with 0.35M parameters. However, in your results, LoRA only reaches 63.7 BLEU with 0.39M parameters. This discrepancy suggests there may be errors in your evaluation or setup.
soundness: 1
strengths: * The paper is well-written and easy to understand.
* The authors' efforts to evaluate across multiple tasks are commendable. However, there are still many problems with these evaluations (see Weaknesses).
summary: Low-Rank Adaptation (LoRA) is a parameter-efficient fine-tuning (PEFT) method. Weighted Decomposed Low-Rank Adaptation (DoRA) is a variant of LoRA, where the update matrix is decomposed into two components: magnitude and direction.

This paper argues that the additional parameters in DoRA increase the risk of overfitting, and simultaneously optimizing both magnitude and direction limits its learning capacity. To address these issues, the authors introduce Bi-Level Optimization-Based Weight-Decomposed Low-Rank Adaptation (BiDoRA), which integrates bi-level optimization into DoRA.

The core idea of BiDoRA is to separate the optimization of magnitude and direction. In the first phase, BiDoRA alternately trains magnitude and direction until convergence. Additionally, the training data for these two components is separated to further prevent overfitting: the magnitude is trained on a validation set, while the direction is trained on a training set. In the second phase, the direction is further trained on the combined dataset until convergence.

Experiments are conducted on natural language understanding (NLU), natural language generation (NLG), and token classification tasks. BiDoRA shows marginal improvements over the baselines, with almost all gains being less than 1 percentage point. Ablation studies reveal that the effects of different design components are relatively small. The training time for BiDoRA is 3.92 times that of LoRA, while DoRA requires 1.3 times the training time of LoRA.
weaknesses: 1. **The motivation of this paper appears to be questionable.** The authors claim that DoRA increases the risk of overfitting, basing this on two pieces of evidence:

   - DoRA introduces additional parameters compared to LoRA.
   - The gap between training and test accuracy curves for DoRA is larger than that of BiDoRA.

   However, these two points do not convincingly support the claim. First, while additional parameters can sometimes contribute to overfitting, they are not a sufficient condition for it. In fact, DoRA adds only a negligible number of parameters (0.01% of the model size, as reported by the authors) beyond LoRA. Moreover, prior work [1] suggests that LoRA learns less than full fine-tuning and may even act as a form of regularization, implying that the risk of overfitting is generally low across these PEFT methods.

   Additionally, the training curves are not necessarily indicative of overfitting, as they can be significantly influenced by factors such as hyperparameters, model architecture, and dataset characteristics. The authors present results from only a single configuration, which limits the generalizability of their findings.

   Finally, the authors’ attribution of an *alleged overfitting problem* to DoRA’s concurrent training lacks a strong foundation.

2. **The proposed BiDoRA method is overly complex and difficult to use.** It requires a two-phase training process, with the first phase itself consisting of two sub-steps. It also introduces two additional hyperparameters: the weight of orthogonality regularization and a ratio for splitting training and validation sets. As a result, BiDoRA takes 3.92 times longer to train than LoRA.

3. **Performance differences between methods are minimal across evaluations**. In nearly all results, the performance differences between the methods are less than 1 percentage point, which may be attributable to random variation. Furthermore, the benchmarks selected are outdated and likely saturated.

[1] [LoRA Learns Less and Forgets Less](https://arxiv.org/abs/2405.09673)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the authors provide additional details on the computational efficiency of BiDoRA? Given the reported computational overhead (approximately four times that of LoRA), are there strategies under consideration for reducing this cost? For instance, would approximate or adaptive bi-level optimization approaches be feasible to explore?

Since bi-level optimization can sometimes suffer from convergence issues, could the authors clarify the convergence criteria used in BiDoRA? Additionally, are there any stability challenges encountered during training, and if so, how were these addressed? This information would be valuable for practitioners looking to implement BiDoRA in different contexts.
soundness: 3
strengths: Originality: BiDoRA introduces a unique bi-level optimization approach to fine-tuning large language models (LLMs), addressing a common tradeoff in parameter-efficient fine-tuning (PEFT) between generalization and computational efficiency. By decomposing the model’s weights into magnitude and direction components and optimizing each on different data splits, BiDoRA creatively combines aspects of neural architecture search with PEFT, offering a compelling alternative to current methods like LoRA and DoRA. This methodological innovation could inspire further development in PEFT techniques across domains.

And specifically I think it's interesting that in BiDoRA, both the magnitude and direction components are trainable, but they are optimized separately through a bi-level optimization framework.

Decomposition: The model's weights are decomposed into two parts:

Magnitude (m): Represents the scale of each parameter.
Direction (V): Represents the vector defining the orientation of each weight.
Optimization Approach:
BiDoRA optimizes the direction component at the lower level using a training set split. During this phase, the magnitude remains fixed, allowing the model to focus solely on finding the optimal direction adjustments.
At the upper level, BiDoRA optimizes the magnitude component by minimizing the loss on a validation set split, with the direction component fixed. This stage uses the optimized direction from the lower level to update the magnitude through hyper gradient descent.
Iterative Training:

The magnitude and direction components are iteratively updated until convergence. This iterative decoupling enables each component to learn independently while promoting generalization and reducing overfitting.
This bi-level approach allows each component to be optimized in a way that would ideally yield better performance than traditional PEFT methods, which tend to optimize these components together, leading to coupled gradients and potentially reduced flexibility.

Quality: The paper demonstrates a high level of rigor, with comprehensive experimental evaluations across multiple tasks, including natural language understanding, generation, and token classification. The experiments span diverse datasets, and the authors include robust analyses, such as ablation studies and weight decomposition, that substantiate the effectiveness of the bi-level optimization framework. This thorough evaluation establishes BiDoRA’s consistent performance advantage over existing PEFT methods, solidifying the validity of the approach.

Clarity: The paper is well-organized and clearly articulates both the theoretical underpinnings and practical implications of BiDoRA. Key concepts, such as weight decomposition and bi-level optimization, are introduced methodically, and diagrams aid in visualizing the process. Algorithmic details, hyperparameters, and dataset splits are documented with transparency, making it easier for readers and future researchers to reproduce and build upon this work.

Significance: The introduction of BiDoRA is a valuable contribution to the field of large language model fine-tuning, where computational efficiency is increasingly critical. By reducing overfitting and enhancing generalization, BiDoRA effectively bridges the gap between parameter efficiency and performance, providing a practical solution that could impact a wide range of applications requiring adaptable, high-performing LLMs. Its relevance to real-world LLM deployment in resource-constrained settings adds to the significance of its contributions.
summary: This paper proposes BiDoRA, a new fine-tuning method that:
1. Decomposes the model’s weights into two parts—magnitude and direction.
2. Optimizes each part separately: The "direction" part is trained on one dataset split (training set), while the "magnitude" part is optimized using another split (validation set).
3. Iterates between these two optimization levels to decouple the updates, helping the model generalize better.

This bi-level approach allows BiDoRA to perform more like full fine-tuning while avoiding overfitting. In experiments across various NLP tasks, BiDoRA consistently outperforms other parameter-efficient fine-tuning methods like LoRA and DoRA. The authors validate BiDoRA on natural language understanding, generation, and token classification tasks, showing that it reduces overfitting and achieves better overall performance than existing methods.
weaknesses: Computational Cost and Efficiency: Although BiDoRA shows a significant performance improvement, the bi-level optimization approach introduces a high computational cost, as reported with nearly fourfold overhead compared to LoRA. This could limit BiDoRA’s practicality in scenarios where resources are constrained. An in-depth analysis of ways to reduce computational complexity without sacrificing performance—such as approximations, alternative regularization techniques, or a comparative exploration of optimization strategies with fewer computational steps—would enhance the method’s accessibility.

Hyperparameter Sensitivity and Tuning: The paper provides limited insight into the sensitivity of BiDoRA to hyperparameters like data split ratios, learning rates for each level, and the orthogonal regularization coefficient. Given that bi-level optimization frameworks often require precise tuning, a sensitivity analysis would help clarify the stability of BiDoRA’s performance under various configurations. This analysis would also support practitioners in understanding which parameters are most influential, guiding them in practical applications.

Scope of Evaluation on Model Types and Tasks: While BiDoRA’s evaluation spans diverse NLP tasks, it primarily focuses on RoBERTa and GPT-2 models. Expanding the scope to include additional architectures, particularly those with different structural properties (e.g., BERT or T5), would verify BiDoRA’s generalizability across model types. Furthermore, testing BiDoRA in non-NLP tasks, such as vision or multimodal tasks, would strengthen the claim of BiDoRA as a general-purpose PEFT framework and broaden its potential applications.

Limited Exploration of Overfitting Mitigation: The paper emphasizes BiDoRA’s success in mitigating overfitting through the decoupling of magnitude and direction optimizations. However, it does not provide an in-depth comparison with existing regularization techniques or alternative overfitting mitigation strategies within PEFT. A comparison or ablation study exploring how BiDoRA’s approach to overfitting compares with alternative techniques (e.g., dropout or adaptive regularization) would better demonstrate the unique value of the bi-level approach.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1) Are there experimental results involving larger, more recent LLMs?
2) Is there an analysis for cases with very small dataset sizes?
3) Are there optimization plans in place to reduce computational complexity?
4) Is there additional analysis on how the choice of data split ratios affects performance?
5) Can theoretical guarantees for generalization error be provided?
soundness: 3
strengths: This work represents a valuable first attempt at applying bi-level optimization to PEFT, extending DoRA's magnitude/direction decomposition in a natural progression. Experimental validation includes a comprehensive evaluation across diverse NLP tasks, such as NLU, NLG, and token classification, with well-suited comparisons to baselines and detailed ablation studies demonstrating the effectiveness of each component. The implementation is both clear and reproducible, contributing to the overall technical completeness of the work.
summary: This paper proposes BiDoRA, a novel method for parameter-efficient fine-tuning (PEFT). While based on DoRA's methodology, it introduces a bi-level optimization approach that optimizes parameters separately. The authors argue that this approach reduces overfitting and enables learning patterns more similar to full fine-tuning. Through experiments across various NLP datasets, they demonstrated that BiDoRA consistently outperforms both DoRA and other PEFT methods.
weaknesses: Methodological Limitations: The approach may be perceived as a straightforward combination of existing methods (DoRA + bi-level optimization), which could limit its novelty. Additionally, the method introduces substantial computational complexity, running at 3.92 times the cost of LoRA, and requires extra hyperparameter tuning. To address these issues, improvements in performance relative to computational cost are needed. Developing an automated framework for setting additional hyperparameters could also streamline the process and reduce user intervention.

Experimental Limitations: The study lacks experiments involving recent large language models (LLMs), does not sufficiently analyze performance on very small datasets, and has limited exploration of data split ratio effects. To enhance experimental rigor, including evaluations with larger, contemporary models, examining performance across varying dataset sizes, and conducting sensitivity analyses across different split ratios would offer a more comprehensive assessment of the method’s effectiveness.

Theoretical Limitations: The paper currently provides limited theoretical guarantees for generalization performance and lacks convergence analysis for the bi-level optimization approach. Strengthening the theoretical foundation could be achieved by introducing an upper bound analysis for generalization error and supplying a convergence proof for the optimization algorithm.

Together, addressing these methodological, experimental, and theoretical gaps would significantly enhance the robustness and credibility of the study.

Meta-review: BiDoRA is a bi-level optimization method designed for parameter-efficient fine-tuning (PEFT) of large language models (LLMs). It builds on DoRA by decoupling the optimization of magnitude and direction components, offering a novel perspective on fine-tuning. The paper is well-written and accessible.

However, all reviewers noted that the significant additional training time could limit its practical utility, despite its ability to achieve fast convergence. Furthermore, the evaluation results show only minimal improvements on the GLUE benchmark.

I believe the paper would benefit from another round of review, focusing on refining its motivation, presenting more robust experiments, providing stronger theoretical support, and conducting an in-depth comparison with existing regularization techniques or alternative strategies for mitigating overfitting.

Therefore, I recommend rejection.

Decision: Reject