BiDoRA: Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation
Authors: Peijia Qin
Ruiyi Zhang
Pengtao Xie
Venue: Accepted by TMLR ()

Abstract: Parameter-efficient fine-tuning (PEFT) is a flexible and efficient method for adapting large language models (LLMs) to downstream tasks. Among these methods, weight-decomposed low-rank adaptation (DoRA) is a promising approach that decomposes weight matrices into magnitude and direction components to mimic full fine-tuning (FT) better. However, DoRA's simultaneous optimization of these components makes it over-expressive, increases the risk of overfitting, and creates a coupled updating pattern that limits its learning capacity. To address these issues, we propose Bi-level Optimization-Based Weight-Decomposed Low-Rank Adaptation (BiDoRA), a novel PEFT method based on a bi-level optimization framework. BiDoRA fundamentally differs from DoRA by optimizing the magnitude and direction in two separate, asynchronous loops using distinct training and validation data splits. This decoupled optimization process effectively mitigates overfitting and allows for more flexible updates that align even more closely with FT. For instance, weight decomposition analysis shows BiDoRA achieves a magnitude-direction update correlation of $-8.042$, significantly closer to the FT ideal compared to $-1.784$ for DoRA. Evaluation of BiDoRA on diverse tasks spanning natural language understanding, generation, token classification, and extremely small biomedical datasets reveals that it consistently outperforms DoRA and a wide range of leading PEFT methods. This improvement is statistically significant, as demonstrated on the GLUE benchmark where BiDoRA surpasses DoRA with a p-value of $2.4\times10^{-4}$ in terms of the Wilcoxon signed-rank test. The code for BiDoRA is available at https://github.com/t2ance/BiDoRA.

Published in Transactions on Machine Learning Research (07/2025)
BiDoRA: Bi-level Optimization-Basel Weight-Decomposee
Low-Rank Adaptation
Peijia Qin
presdd edu
University of California, San Diego
Ruiyi Zhang
ruz0488hucsd edu
University of California, San Diego
Pengtao Xie
plaie Grcceerrsdd..oo edu
University of California, San Diego
Reviewed on OpenReview: https: // openreview. forum? id= v220m3VYY4
Abstract
Parameter-effiiien fine-tuning (PEFT) a flexible and efficient method for adapting large language models (LLMs) to downstream tasks. Among these methods, weight-decomposee low-rank adaptation (DoRA) is promising approach that decomposes weight matrices into magnitude and direction components to mimic full fine-tuning (FT) better. However, DoRA's simultaneous optimization of these components makes it over-expressive, increases the risk of overfitting, and creates a coupled updating pattern that limits its learning capac- ity. To address these issues, we propose Bi-level Optimization-Base Based Weight- Dieght-Decomposed Low-Rank Adaptation (BiDoRA), a novel PEFT method based on a bi-level optimization framework. BiDoRA fundamentally differs from DoRA by optimizing the magnitude and direction in two separate, asynchronous loops using distinct training and validation data splits. This decoupled optimization process effectively mitigates overfitting and allows for more flexible updates that align even more closely with FT. For instance, weight decomposi- tion analysis shows BiDoRA achieves a magnitude-direction update correlation of -8.042, significantly closer to the FT ideal compared to 11.784 for DoRA. Evaluation of BiDoRA on diverse tasks spanning natural language understanding, generation, token classification, and extremely small biomodical datasets reveals that it consistently outperforms DoRA and wide range of leading PEFT methods. This improvement is statistically significant, as demonstrated the GLUE benchmark where BiDoRA surpasses DoRA with p-value of 2.4 x 10-4 in terms of the Wilcoxon signed-rank test. The code for BiDoRA is available at com/t2ance/BiDool
1 Introduction
Large language models (LLMss) (Radford et. al., 2019; Brown et al., 2020) have achieved state-of-the-art results across broad range of NLP tasks, from natural language understanding (NLU) (Wang al., 2019) to natural language generation (NLG) (Novikova et al., 2017). Parameter--ffiiien fine-tuning (PEFT) methods (Houlsby et al., 2019; Hu et al., 2022b) have been introduced promising solution for adapting LLMs for downstream data. PEFT approaches update only subset of the pre-trained parameters, achieving performance comparable to full-inetuning (FT) while requiring significantly fewer computational resources.
One popular type of PEFT is low-rank adaptation (LoRA, Hu et al. (2022b)), which attaches low-rank matrices the pro-trained weights and updates only these matrices during fino-tuning. Liu et al. (2024a) shows that when decomposing the weights into magnitude and direction, their correlation (See Appendix D) tends to be positive in LoRA, whereas it is negative in FT. To bridge the training pattern distinction, they

---

Published in Transactions on Machine Learning Research (07 /225)
*
Temporarily Frozen Component
Magritude in R
LoRA A E R
Trainable Component
Direction 2
LoRA R
Merged Weight
Untrainable Component
Upper Lower: Magnitude
X
OONOO
X
X
1/IVV aVII.
1/IIV + aVIl.
1/1IV AVIL.
soft selection
Av
AV
B
B
B
Pretrained Weight
Pretrained Weight
Pretrained Weight A
Lower Upper: Direction
min (V,,3)
layers
min Cver(V" (ve)(((( (I)
layers
layers
V- xE
(e) Search phose
(me
(b) Retraining phose
Figure An overview of BiDoRA. BiDoRA performs PEFT using a BLO framework. At the lower level, BiDoRA learns the direction component AV of the update matrices using the training split of the downstream dataset. At the upper level, BiDoRA optimizes the magnitude component with optimized AV from the lower level, using the validation split of the dataset. After determining the optimal magnitude, the direction component undergoes further fine-tuning on combined set of both training and validation splits to maximize overall performance.
introduce an explicit reparameterization of the pre-trained weights matrix. The method, named DoRA, decomposes the weights into the column-wise product of magnitude and direction, which determines the direction and magnitude of the weight update, respectively. This approach enables DoRA to share similar learning patterns with FT, thereby outperforming LoRA in multiple tasks. Nonethcless, DoRA introduces additional parameters and over-expressive architecture compared to LoRA, which can exacerbate overfitting issues when adapting to small downstream datasets (See Table 3). Furthermore, in DoRAA, the magnitude and direction components are optimized concurrently, leading to a constrained updating pattern due shared optimization setup (e.g., learning rate, optimizer, batch size).
To address the challenges above, we propose BiDoRA, Bi-level Optimization-Basse Low-Rank Adaptation method for PEFT. BiDoRA facilitates an even more flexible updating pattern and mitigates overfitting by separately optimizing the two components on different data splits with distinct optimization levels. BiDoRA is based on a bi-level optimization (BLO) framework: At the lower level, the low-rank diroction component is updated using the training split, while the magnitude component remains fixed. At the upper level, the magnit component is updated by minimizing the loss on the validation split via hypergradient descent. Subsequently, the direction component further fine-tuned with the optimal magnitude frozen to maximize the performance. These two optimization steps are performed itcratively until convergence. Fig. 1 provides overview of BiDoRA.
A similar strategy of combating overfitting based on BLO has been utilized in the well-established practice of differentiable neural architecture search (DARTS, Liu et al. (2019)), where architecture and sub-networks are learned using different dataset splits. Optimizing the selection variables and sub-networks a single loop can result in an over-expressive network since the selection variables tend to select all sub-networks to achieve the best expressiveness, which, however, incurs severe overfitting. In contrast, training the sub- networks with the selection module fixed on the training split while validating the effectiveness of the selection module on the unseen validation split effectively eliminates the risk of overfitting. Similarly, treat the magnitude component as the architecture and the direction component as the sub-networks and train these components on separate datasets. As shown in Table 3, BiDoRA demonstrates better resistance to overfitting compared to DoRA, given the smaller performance gap between the training set and test set. Furthermore, the asynchronous gradient update steps at the two optimization levels in BiDoRA facilitate better decoupling of the two components, leading to more flexible update pattern that closely resembles FT. As illustrated in Fig. 3, the updates across different layers using BiDoRA have correlation
2

---

Published in Transactions on Machine Learning Research (07/2025)
value that is closest to that of FT, highlighting its superior learning capability compared to both DoRA and LoRA.
Our work makes the following key contributions:
We propose BiDoRA, a novel PEFT method based on bi-level optimization. contrast to DoRA, which trains the magnitude and direction components on a single dataset, BiDoRA optimizes these components at different optimization levels.
Our strategy effectively mitigates the risk of overfitting and results in parameter update pattern that more closely resembles full fine-tuning.
Extensive experiments on various downstream tasks highlight the superior performance of BiDoRA. BiDoRA consistently surpasses several bascline methods, including LoRA and DoRA.
2 Related Work
2.1 Parameter Efficient Fine- Tuning Methods
Parameter-efficient fine-tuning (PEFT) methods aim to reduce the high costs associated with full fine-tuning large-scale models by updating only a relatively small subset of pre-trained parameters, rather than the entire model, to adapt to rownstream tasks. Existing PEFT methods can be mainly categorized into three types.
The first category, known as adapter-based methods, injects additional trainable modules into the original frozen backbone. For instance, Houlsby al. (2019) suggests adding linear modules in sequence existing layers, while He et al. (2022) proposes integrating these modules in parallel with the original layers to enhance performance. Recent advances include SAN (Xu et al., 2023), FÁDA (Bi et al., 2024), and SET (Yi et al., 2024). SAN presents a side adapter network attached to frozen CLIP model, which contains two branches for predicting mask proposals and attention biases. FADA introduces a frequency-adaplee learning scheme that uses the Haar wavelet transform to decompose frozen features into low- and high-frequency components, which are processed separately to enhance domain generalization. SET proposes a spectral- decomposed token learning framework that leverages the Fast Fourier Transform to separate frozen features into amplitude and phase components, enhancing them with spectral tokens and attention optimization.
The second category is prompt tuning methods, which add extra soft tokens (prompts) to the initial input. During the fine-tuning stage, only these trainable soft tokens are updated, as demonstrated in works such as Lester et al. (2021) and Razdaibiedina et al. (2023). Unfortunately, the first two categories lead to increased inference latency compared to fully finc-tuned models.
The third prominent category focuses on low-rank adaptation, pioneered by LoRA (Hu et al., 2022a). LoRA injects trainable, low-rank matrices into a model's layers, freezing the original weights. A key advantage is that these low-rank updates can be merged into the original weights before inference, thus incurring no additional latency. Subsequent works have aimed to improve LoRA's 's efficiency and performance. For instance, AdaLoRA (Zhang et al., 2023) dynamically reallocates the parameter budget based on the importance scores of weight matrices. Zhang et al. (2024b) uses meta-learning to search for the optimal rank of LoRA matrices, further improving its performance on downstream tasks. Pushing parameter efficiency further, VeRA (Kopiczko et al., 2024) employs a single pair of shared low-rank matrices across all layers, while AFLoRA (Liu et al., 2024b) freezes a portion of adaptation parameters based on learned score. A distinct sub-direction has emerged that performs adaptation in the frequency domain, including FourierFT (Gao et al., 2024), LaMDA (Azizi et al, 2024), SSH (Shen et al., 2025b), and MaCP (Shen et al., 2025a). These methods learn updates in transformed spectral spaces, such as the Fourier, discrete Hartley, or discrete cosine domains, rather than directly in the weight space. Other research has focused on bridging the performance gap between LoRA and full fine-tuning. Liu et al. (2024a) found that LoRA's update patterns differ significantly from full finc-tuning, potentially constraining its learning capacity. To mitigate this, they proposed DoRA (Liu et al., 2024a), which decomposes pre-trained weights into magnitude and direction components and uses LoRA for efficient directional updates, better mimicking full fine-tuning.
3

---

Published in Transactions on Machine Learning Research (07/ (025))
2.2 Bi-level Optimization
Bi-level optimization (BLO) has been widely applied in various machine learning tasks, including meta- learning (Finn et al., 2017; Rajeswaran et al., 2019), neural architecture search (NAS) (Liu et al., 2019; Zhang et al., 2021), and hyperparameter optimization (Lorraine et al., 2020; Franceschi et al., 2017). Despite its wide usage, solving BLO problems can be challenging due to the inherent nature of nested optimization problems. Several algorithms have been proposed Bayesian optimization (Cui & Bai, 2019) and first-order algorithms based on hypergradients (Pearlmutter & Siskind, 2008; Lorraine al., 2020). Among these approaches, gradient-based BLO has received significant attention because it can scale to high- diel-dimensioaal problems with large number of trainable parameters.
address this challenge, including zeroth-order methods such as
Inspired by NAS, where bi-level approach is used to Iearn an architecture and its sub-network weights on separate data splits to prevent overfitling, we adapt the BLO framework to parameter-eflicint fine-luning (PEFT), specifically for the weight-decooposed adaptation introduced by DoRA. Unlike in NAS, where BLO scarches for a network architocture, BiDoRA repurposes it to decouple the optimization of weight matrix's two components: magnitude and direction. This approach marks a significant departure from previous PEFT methods like LoRA and DoRA, which optimize all trainable parameters simultaneously dataset. In this work, extend the application of gradient- PEFT method for pre-trained models. By assigning the magnitude and direction components to different optimization levels with distinct data splits, BiDoRA creates a decoupled, flexible updating pattern that better mitigates overfitting and more closely resembles the learning behavior of full fine-tuning.
single BLO to develop a robust and effective
3 Preliminary
LoRA (Hu et al., 2022b) involves attaching the product of two low-rank matrices to the pre-trained weights and fine-tuning these low-rank matrices on downstream datasets with the pre-trained weights frozen. It is based on the assumption that parameter updates made during fine-tuning exhibit a low intrinsic rank. Formally, given a pre-trained weight matrix Wo E Raxk LoRA attaches a low-rank update matrix AW e Raxk to the pre-trained weight. This update matrix can be decomposed as AW BA, where B E Rdxr and A E Rgrk are two low-rank matrices, with r < min(d, k). Consequently, the weight matrix W is represented as follows:
W' Wo + AW Wo + BA
(1)
In this sctup, only the LoRA matrix A AWis is updated. Liu al. (2024a) found that LoRA and full fine-tuning exhibit different learning patterns by performing weight decomposition fine-tuned weight matrices (See Appendix D). To bridge this discrepancy, weight-decomppsee low-rank adaptation (DoRA, Liu et al. (2024a)) further reparameterizes the weight matrices by explicitly decomposing them into learnable magnitude and direction components. Formally, DoRA performs adaptation as follows:
V+AV
Wo + BA
w' = m IV AVII = m IWo BAI
(2)
where AV is a product of two learnable low-rank matrices, B and A, while the magnitude component m E IR 1x is a learnable vector. Here, ll represents the vector-wise norm of a matrix computed across each column, using the L2 norm. In DoRA, both components are optimized concurrently on single downstream dataset. In this work, we aim to improve DoRA by further decoupling the training of the two components.
4 Methods
4.1
Overview of BiDoRA
Our method, BiDoRA, optimizes the trainable parameters in DoRA layers by solving a BLO problem. Let M {ii m2. m denote the set of magnitude components for all n DoRA modules, and y (AV1, ....... AV, denote the set of corresponding direction components. Specifically, we first learn the direction components y" (M) the training split of the downstream dataset. Dr, at the lower level. The
4

---

Published in Transactions on Machine Learning Research (07 (07/)025)
magnitude component M is tentatively fixed at this level; thus, the resulting optimal direction component (M(M) is function of M. At the upper level, we determine the optimal magnitude component MM by optimizing the loss on a validation split vett In practice, Der and Deat are typically created by splitting the original training set without using additional data. This BLO problem is solved using an efficient gradient- based algorithm, where parameters al two levels are optimized iteratively until convergence. While this work focuses on the empírical validation of BiDoRA, our choice of optimization strategy is grounded established theoretical research. The convergence properties of similar gradient-based bi-level algorithms have been previously analyzed (Pedregosa, 2016; Rajeswaran et al., 2019), providing confidence in the stability of our training procedure. Furthermore, the ability of such frameworks to improve generalization core objective of BiDoRA has also been formally studied (Bao et al., 2021), supporting the rationale that our approach can mitigale overfitting.
4.2 Orthogonal Regularization
A central goal of BiDoRA to learn the two disentangled components of weight update: magnitude and direction. The direction component, AV, is responsible for finding a low-rank basis for the update directions. To maximize the expressive power of this component and prevent overfitting, its basis vectors (i.e., the columns of the direction matrix) should be as diverse and non-redundant as possible.
The orthogonality of neural network weights has been identified as a beneficial property (Bansal et al., 2018) and can effectively mitigate the overfitting issue (Balestriero & richard baraniuk, 2018). By enforcing orthogonality, the direction vectors are constrained to represent distinct, independent pathways for updates. This ensures that the limited parameter budget of the low-rank matrix is used efficiently to explore the solution space. Therefore, we define Gram regularization loss (Xie el al., 2017) for the direction component:
R(V) = Σ II(V& + AV+) +AV)) I|
where I is the identity matrix and | ||r denotes the Frobenius norm. Intuitively, R(V) encourages each column of the direction matrix, representing a specific direction, to be orthogonal to one another. Since each column has already been normalized (equivalent to projected to the unit sphere), this also prompls each col- umn be far away from the other, thereby reducing the redundancy of parameters. The effectiveness of this constraint is empirically validated in our ablation study (See Table 5), which shows consistent performance improvement resulting from the enhanced generalization ability of the learned direction component.
4.3 A Bi-level Optimization Framework
Lower level. At the lower level, we train the low-rank direction component V by minimizing a loss Per defined on the training set Drr. The overall training objective at this level is M) = LCV, M: + yR().. Here, L represents the fine-tuning loss, given the low-rank direction component V, the magnitude component M, and the training split Der of the downstream dataset. R(V) is the orthogonal regularizer defined in Eq. (3), with y trade-off hyperparameter. In this level, only update V while keeping M fixed, resulting in the following optimization problem:
(*M = arg argmin (V,
(4)
where (M) denotes the optimal solution for V in this problem, which is function of M.
Upper level. At the upper level, we validate the previously fixed magnitudes M on the validation set Deat, using the optimal direction component (*M) that was learned at the lower level. This results in a We determine the optimal magnitude component
validation loss (MM), (M), M by minimizing this validation loss:
min Loal (V* MM
(5)
M
5

---

Published in Transactions on Machine Learning Research (07 (025)
Algorithm 1: BiDoRA
Input: Training dataset Drr and validation dataset Deal
Initialize trainable magnitude components M {m=]k=1 and low-rank direction components
V (AV& [[A+)==1: (Bx
Search Phase
while not converged do
Update magnitude, M by descending (VV
Update direction V by descending VyLin(V,M)
Derive the optimal magnitude M = {m 1k=1
// Retraining Phase
Train V until converge using Drr UD and derive the optimal direction V* Output: V and M
bi-level optimization framework. Integrating the two levels of optimization problems, we have the following BLO framework:
min Lval (M) M
s.l. V*(M) arg min (v,
(6)
Note that these two levels of optimization problems are mutually dependent on each other. The solution of the optimization problem at the lower level, (MM,, serves as a parameter for the upper-level problem, while the optimization variable M at the upper level acts as a parameter for the lower-level problem. By solving these two interconnected problems jointly, we can learn the optimal magnitude component M and incremental direction matrices V in an end-to-end manner.
Two reasons exist. behind the choice of setting the magnitude component the upper level instead of the converse one: 1) In literature, the upper level usually has fewer parameters than the lower level. In our case, the design of sctting the magnitude of complexity O(k) as the upper level and the direction of complexity O(dr + kr) as the lower level is consistent with the common practice. 2) BiDoRA resembles the DARTS method (Liu et al., 2019) in NAS literature, where the subnets are selected by a selection variable. Specifically, the magnitude vector resembles a selection variable on the direction matrix by softly selecting each direction (subnets) via scaling.
Optimization algorithm. We use a gradient- gradient-based optimization algorithm (Choe et al., 2023b) to solve the BLO problem presented in Eq. (6). A significant challenge in this process is the computation of the upper-level loss gradient with respect to the magnitude component M, as this gradient depends on the optimal solution of the lower-level problem, (MM) For deep neural nefworks, the lower-level objective is non-convex, meaning that finding the true optimal solution (MM) would require running its optimization process to full convergence. Performing this complete inner optimization for every single update of the upper-level variable M is computationally intractable.
To address this issue, we use the following onc-stop-unrolled approximation of (M) inspired by previous work (Liu et al., 2019):
Ven (V* MM -VViaa(V-VVVyerte
(7)
where & the learning rate the lower level, and the one-step-unrolled model V = V (V. -
is used surrogate for the optimal solution V (MM).. We then compute the approximated gradient

---

Published in Transactions on Machine Learning Research (07 (025)
follows:
M) 2 MLull (V, - (V+ .M) VM ,
V Mfaal
(8)
(9)
2
where is small scalar and y V = (V, Since directly computing the matrix-vector multiplication term in Eq. (8) is computationally expensive, we use finite difference to approximate this product as in Eq. (9), following Liu et al. (2019). As detailed in Algorithm 1, the direction component V and the magnitude component M are updated using gradient. descent iteratively until convergence. After acquiring the optimal magnitudes M through the process above, the direction component V is retrained on the union of training and validation splits to achieve the best performance on downstream tasks, resulting in the final learned V*. All splits are intentionally used during retraining to maximize data utilization and performance.
In practice, the convergence of the search phase is determined by the evaluation metric at the upper level. For the subsequent retraining phase, we adopt stopping criterion similar to DoRA's, observing performance on a separate, held-out test set that is not used during training.
Table 1: RoBERTahase/ hass//rgg (Rb/)) and DeBERTaxXI. (DxxI.) with different fine-tuning methods the GLUE benchmark (Wang et al., 2019). A higher value is better for all datasets. The best results are shown in bold.
Method
#Params
MNLI SST-2
MRPC CoLA
QNLI
QQP
RTE
STS-B
Avg.
(FT)
125.0M
90.3
94.8
89.3
61.6
86.7
92.8
76.9
91.2
85.5
Rb (sdAapter)
0.9 M
86.5
94.0
88.4
58.8
92.5
89.1
71.2
89.9
83.8
Rs (LoRA)
0.15 M
86.8
94.3
88.0
60.3
93.0
89.6
72.9
90.1
84.4
Rs (DoRA)
0.17 M
86.8
94.2
89.2
60.5
92.9
89.6
73.2
90.2
84.6
Rg (BiDoRA)
0.17 M
87.1
94.4
89.4
61.3
92.7
90.6
76.0
90.1
85.2
(((FT)
355.0M
90.2
96.4
90.9
68.0
94.7
92.2
86.6
92.4
88.9
([Adapter))
0.8M
90.3
96.3
87.7
66.3
94.7
91.5
72.9
91.5
86.4
(L(LA)A)
0.39 M
90.6
96.3
90.0
66.9
94.5
91.2
86.3
91.7
88.4
(D(DR)A)
0.39 M
90.6
96.4
89.8
65.8
94.7
91.2
86.6
92.0
88.4
(BiDoRAA)
0.39 M
90.6
96.1
90.1
67.0
94.6
91.7
86.9
92.0
88.6
DxxL (DoRA)
4.9M
91.2
96.3
92.3
71.1
95.3
91.6
91.8
90.8
90.0
Dxxx(BiDoRl)
4.9M
91.7
96.3
92.6
72.3
95.2
92.0
92.3
90.8
90.4
5
Experiments
5.1 Experimental Setup
We compare BiDoRA with several PEFT methods, including Full Fine-Tuning (FT), Adapter Tuning (Houlsby et al., 2019; Lin et al., 2020; Rücklé et al., 2021; Pfeifler el al., 2021), LoRA (Hu al., 2022a), A AdaLoRA (Zhang et al., 2023), DoRA (Liu et al., 2024a), VeRA (Kopiczko et al., 2024), FourierFT (Gao et al., 2024), AFLoRA (Liu et al., 2024b), LaMDA (Azizi et al., 2024), SSH (Shen et al., 2025b), MaCP (Shen et al., 2025a). BiDoRA does not use any additional data compared to other baselines, as we create the validation set for upper-level optimization by splitting the original training set with an 8:2 ratio for all tasks. Detailed descriptions of these baseline methods are provided in Appendix C.
Our experiments cover a wide range of tasks, including natural language understanding (Section 5.2), ex- tremely small biomedical datasets (Section 5.3), natural language generation (Appendix H.1), and token
7

---

Published in Transactions on Machine Learning Research (07 (07/2025)
classification (Appendix H.2). Please refer to the detailed dataset settings and experimental settings in Appendix A and Appendix B, respectively. Our implementation is based on the Hugginglace Transformers library (Wolf et al., 2019) and the Betty library (Choe et al., 2023b).
5.2 Experiments on Natural Language Understanding Tasks
In this section, evaluate the performance of BiDoRA NLU tasks.
Main results. Table presents the results of fine-tuning the RoBERTa-base RoBERTaalarge, and De- BERTa XXL models on the GLUE benchmark with baseline PEFT methods and BiDoRA. The results show that BiDoRA achieves superior comparable performance compared to baseline methods across all datasets with the same number of trainable parameters. The superior performance of BiDoRA verifies the effectiveness of its BLO mechanism. By training the magnitude and direction components on two distinct sub-datasets, BiDoRA enhances the flexibility of the learning process and improves learning capacity compared to DoRA, resulting in a performance boost.
We also present an experiment on the GLUE benchmark with the RoBERTa-base model, on larger, wide range of baselines, following the settings from Shen al. (2025b) and Shen et al. (2025a) and citing their reported baseline results for reference. The results in Table 2 indicate that BiDoRA consistently outper- forms all baselines, including DoRA, across these diverse NLU tasks, demonstrating its robust generalization capability.
Table 2: Performance of various fine-tuning methods on the GLUE benchmark for the RoBERTa-base model. The best ones are highlighted by bold and the second ones are highlighted by italic.
Model
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
FT
94.8
90.2
63.6
92.8
78.7
91.2
85.22
BitFit
93.7
92.7
62.0
91.8
81.5
90.8
85.42
Adpi
94.7
88.4
62.6
93.0
75.9
90.3
84.15
LoRA
95.1
89.7
63.4
93.3
78.4
91.5
85.23
AdaLoRA
94.5
88.7
62.0
93.1
81.0
90.5
84.97
AFLoRA
94.1
89.3
63.3
91.3
77.2
90.6
84.33
LaMDA
94.6
89.7
64.9
91.7
78.2
90.4
84.92
VeRA
94.6
89.5
65.6
91.8
78.7
90.7
85.15
FourierFT
94.2
90.0
63.8
92.2
79.1
90.8
85.02
SSH
94.1
91.2
63.6
92.4
80.5
90.9
85.46
MaCP
94.2
89.7
64.6
92.4
80.7
90.9
85.42
DoRA (r 8)
94.9
89.9
63.7
93.3
78.9
91.5
85.37
BiDoRA (r
95.7
90.2
65.8
93.4
79.4
90.5
85.83
DoRA (r= 16)
94.8
90.4
65.6
93.1
81.9
90.7
86.08
BiDoRA (r=16)
95.0
90.8
66.7
93.3
82.6
90.9
86.55
Table 3: Quantitative performance gap between training and test sets for DoRA and BiDoRA using the RoBERTa- model. The gap is calculated as the training metric minus the test metric, where a smaller value indicates less overfitting.
Method
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg.
DoRA
2.0
9.5
32.5
6.6
18.0
8.8
12.9
BiDoRA
1.7
7.0
23.3
0.2
14.0
4.7
8.5
8

---

Published in Transactions on Machine Learning Research (07 (025)
Test: DoRA
Train: DoRA
Test: BIDoRA
Train: BIDoRA
0.95
0.9
0.85
0.8
0.75
Step
0.7
5k
10k
15k
20k
Figure/ 2: Training and test accuracy versus global training steps on the ModHayes split of the Reuters21578 dataset (Padmanabhan et al., 2016) when finc-tuning RoBERTa-base model using DoRA and BiDoRA. The training and lest curves for DoRA show larger gap compared to BiDoRA, highlighting the effectiveness of our method in reducing overfitting.
Robustness of BiDoRA towards different rank settings. We explore the impact of different rank configurations on BiDoRA and DoRA, ovaluating them with ranks 8 and 16 addition to the rank of 4 used in Table 1. The average accuracies reported in Table demonstrate that BiDoRA consistently surpasses DoRA across all rank configurations, highlighting its resilience and superior performance regardless of the rank setting.
Performance gap between training and testing set. As visualized in Fig. 2, BiDoRA achieves smaller gap between the training and test curves. Quantitatively, Table 3 presents this performance gap on the RoBERTa-base model. The training set metric is calculated as moving average of the per-batch metric with a decay ratio of 0.99. Since BiDoRA has two training loops, its training metric is a weighted average (0.8 x mnner-loop-metrg + 0.2 x nuter-loop-metricn our case. The results show that the performance gap for BiDoRA consistently lower than that of DoRA across all datasets. This suggests that DoRA more prone to overfitting, issue that BiDoRA effectively addresses.
based on the data split size, inner : outer = 8 2, in
5.3 Experiments on Extremely Small Datasets
Fino-tuning ESM on the thermostability prediction task (Chen et al., 2023) (left), the BBP task (Dai al., 2021) (middle), and the MIC task (Ledesma-Fernaanl Fernandez et al., 2023) (right). A higher value is better for all metrics except for MSE. The best results are highlighted in bold.
Table 4:
Methods # Params| Accuracy Precision Recall F1
#Params Accuracy Precision Recall Fl
#Params
MSE
FT
652.7M
79.8
81.2
79.8
78.4
652.9M
89.4
89.9
89.4
89.4
652.7M
0.2894
LoRA
1.5M
75.9
78.2
75.9
75.5
1.9M
86.8
87.7
86.8
86.7
1.7M
0.3433
DoRA
1.6M
76.9
78.7
76.9
76.2
2.0M
91.3
89.4
89.3
1.8M
0.2918
BiDoRA
1.6M
78.8
79.1
78.8
78.2
2.0M
92.1
93.1
92.1
92.0
1.8M
0.2818
We conduct additional experiments tasks-therma tookmstthrmmttablittyy prediction (Chen et al. (2023), 936 training samples) and blood-brain barrier peptide prediction (BBP, Dai et al. (2021), 200 training samples) and one regression task, minimum inhibitory ratcon prediction (MIC, contain significantly fewer samples than standard NLP tasks.
biomedical datasets, including two classification
et. al. (2023), 3,695 training samples), which
The results are presented in Table 4. Consistent with our previous findings, BiDoRA effectively fine-tunes pre-trained models extremely small datasets. Our method outperforms the baselines by larger margin

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 5: Ablation studies. We evaluate the performance of BiDoRA without, retraining (w/o retraining). without BLO (E = 0), without orthogonal regularization (w/o est.), and with retraining magnitude.
Method
MNLI SST-2
MRPC CoLA QNLI QQP
RTE STS-B Avg.
BiDoRA (retraining magnitude)
87.0
94.3
89.1
60.7
92.7
91.0
73.4
89.9
84.8
BiDoRA (w/o retraining)
87.0
94.2
89.0
57.3
92.4
90.6
71.6
90.0
84.0
BiDoRA (5 = 0)
86.9
94.2
89.0
59.4
90.8
91.2
75.9
90.0
84.7
BiDoRA (w/o cst.)
87.0
94.4
88.6
61.3
92.7
90.2
76.0
90.1
85.0
BiDoRA
87.1
94.4
89.4
61.3
92.7
90.6
76.1
90.1
85.2
as the dataset size decrcases, confirming our previous conclusion that our method effectively combats the overfitting issue on various network archilectures and diverse lasks.
5.4 Ablation Studies
In this section, we perform ablation studies to investigate the effectiveness of individual modules or strategies in BiDoRA. We fine-tune RoBERTa-base model on the GLUE benchmark under different ablation settings, and the results are shown in Table 5.
Retraining. We test the model directly obtained from the search phase to evaluate the effectiveness of further retraining the direction component. The results show that BiDoRA outperforms BiDoRA (w/o retraining) on average, highlighting the necessity of retraining. Table 5 also validates that retraining the direction component leads to superior performance than retraining the magnitude.
Bi-level optimization. We set to zero in Algorithm to assess the effectiveness of the BLO framework. This ablation setting can be interpreted as an alternative learning method where two optimization steps are carried out alternately on two different splits of the training dataset. Notably, in the alternative learning method, the updating of each component is unaware of the others, making the training less stable. In contrast, the syper-gradient used in BLO avoids this issuc by connecting the two levels in a certain way. The results show that BiDoRA outperforms BiDoRA (E 0) on average, demonstrating the efficacy of the BLo strategy.
Orthogonal regularization. We examine the effectiveness of the orthogonality constraint in Eq. (3) by setting to zero. Results show that BiDoRA outperforms BiDoRA (w/o est.) on average, indicating the effectiveness of applying the orthogonality regularizer to alleviate overfitting.
5.5 Weight Decomposition Analysis
One important motivation of DoRA is to bridge the inherent differences between LoRA and FT. Similar to DoRA, we conduct weight decomposition analysis on the correlation between the change of magnitudes and that of directions (detailed in Appendix D) for BiDoRA and baseline methods by fine-tuning GPT2-medium model on the E2E dataset. As shown in Fig. 3, FT, DoRA, and BiDoRA all exhibit negative correlation values, while LoRA shows a positive correlation, consistent with the findings in Liu BiDoRA achieves a negative correlation of -8.042, closer FT than DoRA's -17784. This improvement is attributed to the decoupled training process of the two layers, which allows for a higher learning capacity compared to DoRA.
(2024a). Notably,
5.6 Discussion
The advantage of BiDoRA is supported by both theoretical insights and empirical evidence, as detailed follows.
10

---

Published in Transactions on Machine Learning Research (07/ (225))
<- 00.915
0.506
F- 3.784
Layer
Lager:
0.0-
Lyye
Liyer Liser
Layer 4
029
Layer 5
Layar
022
Layer 0
415
Frier stop
Inter
Ieer stap
slop 1
Iear
616
Heer
(a) Full FT 65.816)
(b) LoRA(k 0.836)
(c) DoRA(K 1.7784
(d) BiDoRA(k -8.042)
Figure 3: Magnitude and direction updates for (a) FT, (b) LORA, (c) DoRA, and (d) BiDoRA of the query matrices across different layers and intermediate steps after fine-tuning the GPT2 model the F2F dataset (Novikova et al., 2017), where & denotes the correlation value. Different markers represent matrices from different training steps, with each color corresponding to a specific layer. AM denotes the average change in weight vector magnitude, and AD denotes the average change in direction, as formally defined in Appendix D.
Motivation. Theoretically, Liu et al. (2024a) showed that LoRA's training pattern tends to be coupled in terms of magnitudc--irection correlation, which degrades learning capacity. Their solution was to introduce a reparanmeterization that decouples these components in the formulation. We build upon DoRA following their theory and further decouple magnitude and direction in terms of training dynamics. Specifically, the two components are trained in separate loops within bilevel optimization framework, which is expected to improve performance in an intuition similar to DoRA.
Empirical evidences. We performed a Wilcoxon signed-rank test to compare the performance of DoRA and BiDoRA. Specifically, we used the results from Table 1. For cach PEFT method, we collected 9 values (8 values from each dataset plus the average performance) from one base model. We concatenated the results from three base models (RoBERTa-base, RoBERTa-large, and DeBERTa-XXI.) to obtain a list of 27 values. A comparison of these 27 values between DoRA and BiDoRA reveals that BiDoRA is significantly better than DoRA, with a p-value of 2.4 x 10-4. This result demonstrates that BiDoRA offers non-marge con-marginal improvement. over DoRA.
Additionally, the weight decomposition analysis, including (Fig. 3 and Fig. 4), indicates that BiDoRA achieves better decoupling of the components compared to DoRA. Evaluation metrics across various tasks demonstrate the superior performance of BiDoRA, confirming that our decoupled optimization loop leads to improved outcomes.
6 Conclusion and Future Works
We propose BiDoRA, a novel bi-level optimization framework for PEFT of large-scale pre-trained models. By conducting weight, decomposition following the DoRA approach, our method trains the two components separately two interconnected optimization levels using different sub-datascts. In this way, BiDoRA not only decouples the learning process of the two components, resulting in a learning pattern closer to FT, but also effectively alleviates overfitting. Empirical studies various NLP tasks demonstrate that BiDoRA outperforms DoRA and other basclines, highlighting the effectiveness of our method.
One potential limitation of BiDoRA is its training efficiency (see Appendix E) in terms of per-step cost, which could be reduced by using more advanced ayper-gradient estimators, such as SAMA (Choe et al., 2023a) or MixFlow-MG (Kemaev et al., 2025). Furthermore, while we have empirically shown that BiDoRA induces better decoupling between the magnitude and direction components (Fig. 3 and Fig. 4), a formal theoretical analysis of this property is currently lacking and serves for future work.
11

---

Published in Transactions on Machine Learning Research (07 (07/2023) 2025)
Acknowledgments
P.X. acknowledges funding support from NSF IIS2405974, NSF IIS2339216, NIH R35GM157217, and NIH R21GM154171.
References
Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram. Lamda: Large model fine-tuning via spectrally decomposed low-dimensionnal adaptation. ArXiv preprint, abs/2406. 12832, 2024. URL https: attpss//arxiv. org/abs/2406 12832.
Randall Balestriero and richard baraniuk. A spline theory of deep learning. In Jemnifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- aeedings of Machine Learning Research, pp. 374-383. PMLR, 2018. URL https://proceedings mlr. html.
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. Jade Goldstein, Alon Lavie, Chin- Yew Lin, and Clare Voss (eds.), Proceedings of the ACT. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65 72, Ann Arbor, Michigan, 2005. Association for Computational Linguistics. org/W05-0909.
URL https::/acllatthlllls
Nitin Bansal, Xiaohan Chen, and Zhangyang Wang. Can we gain more from orthogonality regular-
izations in training deep cnns? In Newral Information Processing Systems, 2018. URL https:
t/appp/saaantaeeeeeeaeeaaaaaaatt semantccscholar org/CorpusID
ortpaa/tccccclla..ooo//ooooaaaaa :t:CCorpsII:55705000000 55704502.
Fan Bao, Guoqiang Wu, Chongxuan Li, Jun Zhu, and Bo Zhang. Stability and generalization bilevel programming in hyperparameter optimization. In Marc' Aurelio Ranzato, Alina Beygekimee, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Anrual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp.
2021. URL https: / pttp///roeeen...e//ee prroeeeiings.neurips...ssss.e hccppaapre//22//aaaaa/ hash// Abstract...ttt............aa...t html.
Qi Bi, Jingjun Yi, Hao Zheng, Haolan Zhan, Yawen Huang, Wei Ji, Yuexiang Li, and Yefeng Zheng. Learning frequency-adapted vision foundation model for domain generalized semantic segmen- tation. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: An- nual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http::/papers: html. Baf50cS)ccStc01886447666623222 1499- -tt.raaaaaaaaa---aarr-------o--o Ceterract--ooffreeee
cttpsscccpppperttiieee seeeerffles/ppeee ftilee/paper/222//aaaa
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodci. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-FForina Balcan, and Hsuan-Tien Lin (eds.), Advances in Newral Information Pro- cessing Systems 39: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpp://ppeet htttoe//proceeeingg.. wwwpeeediinnnnnnuss....m httph/ps.ccpppppp/222/////// hash// Abstract. html.
Daniel Cer, Mona Diab, Encko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Sleven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2011) pp. 1-14, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10.18653/v1/S11-2000 URL attps://aclanthologg org/S17 2tg//17-000.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Tianlong Chen, Chengyue Gong, Daniel Jesus Diaz, Xuxi Chen, Jordan Tyler Wells, Qiang Liu, Zhangyang Wang, Andrew D. Ellington, Alex Dimakis, and Adam R. Klivans. Hotprotein: A novel framework for protein thermostability prediction and editing. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net net, 2023. URL https://
Sang Keun Choe, Sanket Vaibhav Mehta, Hwijeen Ahn, Willie Neiswanger, Pengtao Xie, Emma Strubell, and Eric P. Xing. Making scalable meta learning practical. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Sacnko, Moritz Hardt, and Sergey Lovine (eds.), Advonces in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 16, 2023, 2023a. URL http://papers, httpss.ccpppppp.ppppppppp fil.filll///llee htalle//papr//223/... /ashh/ 1519988CL.6688888877974448888889
Awwtractttooriiiiiooaaeee Confreecct-oofeeeneeeee html.
Sang Keun Choe, Willie Neiswanger, Pengtao Xie, and Eric P. Xing. Betty: An automatic differentiation library for multilevel optimization. In The Eleventh International Conference on Learning Representations, net./
ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview net, 2023b. URL pdf L MeMS38q9.
Nigel Collier, Tomoko Ohta, Yoshimasa Tsuruoka, Yuka Tateisi, and Jin-Dong Kim. Introduction to the bio-entity recognition task at JNLPBA. In Nigel Collier, Patrick Ruch, and Adeline Nazarenko (eds.). Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicciee and its Applications CILPBA//ioNLLL pp. 73-78, Geneva, Switzerland, 2004. COLING. URL https:// aclanthology org/W04-1200 1213.
Hua Cui and Jie Bai. new hyperparameters optimization method for convolutional neural networks. Pattern Recognition Letters, 125:828 834, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Ruyu Dai, Wei Zhang, Wending Tang, Evelien Wynendaele, Qizhi Zhu, Yannan Bin, Bart De Spiegeleer, and Junfeng Xia. Bbppred: sequence-based prediction of blood-brahe barrier peptides with feature repre- sentation learning and logistic regression. Journal of Chemical Information and Modeling, 61 (1))::25 534, 2021.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (IWP2005), 2005. URL https:// aclanthology www./oog//--/... 5000.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017. URL http://proceeddnnss mlr, press/v70/ Finn17a.htm
Luca Franceschi, Michele Donini, Paolo Frasconi, and Massimiliano Pontil. Forward and reverse gradient- based hyperparameter optimization. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1165 1173. PMLR, 2017. URL http:// proceedings. mlr. Stt.r.press///////
htt.://s//o//rrrcccccc177 www.///raaeecchi7/... htmm.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parfmeter-effcieet fine-tuning with discrete fourier transform. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net 2024. URL https://openreviex net/ forum?ide XUHHSSSuu.....
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards uni- fied view of parameten-efficient transfer learning. In The Tenth International Conference Learning
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL https: wwrpeerrvveew.nt/ffrrrere http://///oeeeee/////o//o/aaaaaa dddornDaxx...
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficinnt transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.); Proceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790 2799. PMLR, 2019. URL http://proocedding..oo mlr. wal...presa/v9] houlsby19a 1ttssb999..l.....1..
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representalions, ICLR 2022, Virtual Event, April 25-29, 2022. Open 2022a. URL attps://openreet
net/foorumiiddneeeeeee.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Represenlations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.nel, 2022b. URL
https://openreviii http://rrrrvveeeeeettrrtrrraaaaa
lurii Kemaev, Dan A Calian, Luisa M Zintgraf, Gregory Farquhar, and Hado van Hasselt. Scalable meta- learning via mixed-mode differentiation. ArXiv preprint, abs/2505.00793, 2025. URL
org/abe ht5/abs/2500 aaaaab/...... 00793.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. Vera: Vector-based random matrix adapta- lion. In The Twelfth International Conference on Learning Represenlations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nee net, 2024. URL attps://openreviien http:riiw...ettoooooo/ooiaaaaaaa
Alba Fernandez Susana Velasco-Lozano, Javier Santiago- Arcos, Fernando Jópez-Gallege and Aitz- iber Cortajarena. Engineered repeat proteins as scaffolds to assemble multi-enzyme systems for efficient cell-free biosynthesis. Nature Communications, 14(1):2587, 2023.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Longuage Processing, pp. 3045-300 3059, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.emnlp-ne mainmmmainnn22333 URL https: http:///ccaaatooggo..oo.. 000g////0011oo...//201.1.111 emnlp-main, 243.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Texl Summarization Branches Out, pp. 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics. URL
attps://acllantholog) org/W04-1013.
Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the As- sociation for Computational Linguistics (ACL-04), pp. 605 612, Barcelona, Spain, 2004. doi: 10.3115/ org/P04-1077.
1218955.1219032. URL https://aclanthhne
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parfmeter-efficient transfer learning. In Trevor Cohn, Yulan He, and Yang Liu (eds.), Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 441-459, Online, 2020. Association for Computational Linguistics. doi: org/2020. mondings-emnl....
URL https://aclanttologyy
Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: diffcrentiable architecture scarch. In 7th Inter- national Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. Reveeeview,net, 2019. URL httt:p////ppeeeeree.a.eee////sss
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Con- ference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview. net, 2024a. URL
Zeyu Liu, Souvik Kundu, Anni Li, Junrui Wan, Lianghao Jiang, and Peter Anthony Beerel. Aflora: Adap- tive freezing of low rank adaptation in parameter efficient fine-tuning of large models. ArXiv preprint, //////4433.1112... 2024b. URL https: //arxiv.. org/abs/2403. 13269.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions hyperparameters by implicit differentiation. In Silvia Chiappa and Roberto Calandra (eds.), The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online (Palermo, Sicily, Italy]: volume 108 of Proceedings of Machine Learning Research, pp. 1540-1552. PMLR, 2020. URL http: / //proceedings.nt mml.. mlr. www.mml.....s//eeemeemmmmee press/v108// 101100/1/rree00aaa html.
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Kristiina Jokinen, Manfred Stede, David DeVault, and Annie Louis (eds.), Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, Saarbrücken, Germany, 2017. Association for Computational Linguistics. doi: 10. 11865//11/117/ org/W17-5525.
URL
Divya Padmanabhan, Satyanath Bhat, Shirish Shevade, and Y Narahari. Topic model based multi-label classification. 2016 IEFE 28th International Conference on Tools with Artificial Intelligence (ICTAT), pp. 996-1003. IEEE, 2016.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics. doi: 01155/107088.110111111 URL https:
//aclanthology org/P02-1040.
Barak A Pearlmutter and Jeffrey Mark Siskind. Reverse-mode ad in functional framework: Lambda the ultimate backpropagator. ACM Transactions on Programming Languages and Systems (TOPLAS), 30(2): 1 36, 2008.
Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In Maria-Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 737-746. JMLR.org, 2016. URL http://proceeddngs mww.mlr.p...... wal..press//ee pttp:///oeeeerrer1.... html.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destruetive task composition for transfer learning. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaly (eds.), Proceedings of the 16th Conference of the European Chopter of the Association for Computational Linguistics: Main Volume, pp. 487-503, Online, 2021. Association for Computational Linguistics. doi: 10.18653/ 200033/v//200 22201.cccl-mainn
URL haap::/:///llathhooo// org/2021. wwtg/222.eacl-eaeee.e. macl-mainn.3. 39.
Alec Radford, Jeffrey WU, Rewon Child, David Luan, Dario Amodei, Tlya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Aravind Rajeswaran, Chelsca Finn, Sham M. Kakado, and Sergey Levine. Meta-lcarning with implicit gradients. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alchh-BBue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Con- ference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van- couver, BC, Canado, pp. 124, 2019. URL attps://proceedings wwwp:cceediing..uurss..... www.ine..rrrrerreeeeeee (ooripps.c//pppr/20099 h/aash/ 77220000a118222442224444422b22 -tw.teeeeeeea--A--aaattea.. Abstract html.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Tryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784 789, Mel- bourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P11-2222 URL attps://aclanthoolog org/P18-2124.
Anastasiia Razdaibiedina, Yuning Mao, Madian Khabsa, Mike Lewis, Rui Hou, Jimmy Ba, and Amjad Almahairi. Residual prompt tuming: improving prompt tuning with residual reparameterization. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 6740 6757, Toronto, Canada, 2023, Association for Computational Linguistics. org/2023. findings-ace aww.acl..22
22200///v1222../// http://w////2....nnnnnse// findings- 421. URL cetps://aclantaae doi: 10.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15): e2016239118, 2021.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In FranciFr Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7930 7946, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi: (0.18633/v1/2211..om waw.llmmmllaaaii main.626. URL https://aclaathhology wtt.//2.g22211.1... emnlp-main. 626.
Yixian Shen, Qi Bi, Jia-Hong Huang, Hongyi Zhu, Andy D Pimentel, and Anuj Pathania. Macp: Minimal yet mighty adaptation via hierarchical cosine projection. ArXiv preprinl, abs/2505. 238700, 2025a. URL https://arxi: argg/bs//500/ 23870.
Yixian Shen, Qi Bi, Jia-hong Huang, Hongyi Zhu, Andy D. Pimentel, and Anuj Pathania. SSH: Sparse spectrum adaptation via discrete hartley transformation. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 10400-10415, Albuquerque, New Mexico, 2025b. Association for Computational Linguistics. ISBN 979-8-89176-189-5 doi: 10. http://111222.maccclee
URL https: https://aclanthollla org/ naacl-long. 12221
Richard Socher, Alex Perclygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christo- pher Polts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard (eds.), Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631 1642, Seattle, Washington, USA, 2013. Association for Computational Linguisties. URL https://aclanthologg org/ D13-1170.
Erik F. Tjong Kim Sang. Introduction the CoNLL-2002 shared task: Language-indopenden named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLIL-002). 2002. URL https://aclanthollge how.aanttoooooooooo//0/0-0-- org//02-2022
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. Tn IEEE Conference on Computer Vision and Pattern Recognation, CVPR 2015, Boston, MA, USA, June 7-12, 2015, pp. 4566 4575, IEEE Computer Society, 2015. doi: 10.1009/CVPR..005557203 URL https://doi. http:::///oooor/////// org/10.10 1109/CVPR. 2015. 7299087.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019. URL http:::p/pppaaeeeeeeeaa///aaaaaa
16

---

Published in Transactions on Machine Learning Research (07/2025)
Zhiguo Wang, Wael Hamza, and Radu Florian. Bilateral multi-persppetive matching for natural language sentences. In Carles Sierra (ed.), Proceedings of the Twenty-Sizth International Joind Conference on Artificial Intelligence, LJCAI 2017, Melbourne, Australia, August 19-25, 2017, pp. 4144 4150, ijcai.org, 2017. doi:
URL attps://ddet http://doi..rr.o http::///oo..o//....oo 24963/ijcai 2017/579.
Alex Warstadt, Amanprect Singh, and Samuel R. Bowman. Neural network acceptability judgments. Trans- aclions of the Associalion for Compulational Linguislics, 7:625 641, 2019. doi: 10.1162/ 00290.
URL
org//19-1000
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1122-1122, New Orleans, Louisiana, 2018. Association for Computational Linguisties. doi: (118653/11/118--1/1 org/N18-118 1101
URL https://aclanthool.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing. ArXic preprinl, abs/1910.03771, 2019. URL. https: wwarrxiv..ogg/ae. org/aba/ 03771.
Di Xie, Jiang Xiong, and Shiliang Pu. All you need is beyond good init: Exploring better solution for training extremely deep convolutional neural networks with orthonormality and modulation. In 2017 IEEE Conference on Compuler Vision and Pallern Recognition, CVPR 2017, Honobuu,, HI, USA, July 21-26, 2017, pp. 5084. IEEE Computer Society, 2017. doi: 10. A0.L109/CVPR.2017..55 URL https: oroo/.0 1109/CVPR. 2017.533..1
Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In LEEE/CVL Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pp. 2945 2954. IEEE, 2023. doi: D0.1109/CVPR5275 2023.00288. URL https: : ort./....rg/..... 10. 1109 1119/CVPR52722
00288..
Jingiun Yi, Qi Bi, Hao Zheng, Haolan Zhan, Wei Ji, Yawen Huang, Yuexiang Li, and Yefeng Zheng. Learning spectral-decommositeed tokens for domain generalized semantic segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 8159-8168, 2024.
Li Zhang, Han Guo, Leah V Schaffer, Young Su Ko, Digvijay Singh, Hamid Rahmani, Danielle Grotjahn, Elizabeth Villa, Michael Gilson, Wei Wang, et al. Proteinaligner: A multi-modal pretraining framework for protein foundation models. bioRrie, pp. 2024-10, 10, 2024a.
Miao Zhang, Steven W. Su, Shirui Pan, Xiaojun Chang, M. Ehsan Abbasnejad, and Reza Haffari. idarts: Differentiable architecture search with stochastic implicit gradients. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12557-12566. PMLR, html.
2021. URL http://proceeddings wmw.eeeddinnnm............ee...e www.:.aww.........memmemmmmmmeee
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
Ruiyi Zhang, Rushi Qiang, Sai Ashish Somayajula, and Penglao Xie. AutoLoRA: Automatically luning matrix ranks in low-rank adaptation based meta learning. In Kevin Duh, Helena Gomez, and Steven Bethard (ods.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compulational Linguislics: Human Longuage Technologies ( (Volume 1: Long Pupers), pp. 5048 5060, Mexico City, Mexico, 2024b. Association for Computational Linguistics. URL https://tset htap::///ccaathhloog... org/2022 naacl-1ong.288 282.
17

---

Published in Transactions on Machine Learning Research (07 (025)
A
Datasets and Models
Table 6: Summary of datasets used in the experiments
Task Group
Dataset
Metrics
Train
Dev
Val
Test
MNLI
Accuracy
393k
20k
20k
SST-2
Accuracy
67k
872
1.8k
MRPC
Accuracy
3.7k
408
1.7k
Natural Language
CoLA
Matthews Corr
8.5k
1k
1k
Understanding
QNLI
Accuracy
108k
5.7k
5.7k
QQP
Accuracy
364k
40k
391k
RTE
Accuracy
2.5k
276
3k
STS-B
Pearson Corr
7.0k
1.5k
1.4k
ModApte
F1
8.8k
3k
Text Classification
ModHayes
F1
18k
0.7k
ModLewis
F1
12k
5.5k
Natural Language Generation
BLEU, NIST, MET, ROUGE-L, CIDEr
E2E
42k
4.6k
Accuracy, Precision, Recall, F1
Token Classification
BioNLP
17k
1.9k
3.9k
Accuracy, Precision, Recall, F1
CONI.L2003
14k
3.3k
3.5k
biirmostability prediction
Accuracy, Precision, Recall, F1
3,695
924
Biomedical Experiments
Accuracy, Precision, Recall, F1
BBP
200
38
MIC
MSF
936
104
In this section, present the datasets and models used in experiments, and summarize the statistical data in Table 6.
A.1 Natural Language Understanding
The GLUE Benchmark (Wang et al., 2019) comprises diverse array of tasks that widely employed for evaluation in natural language understanding. It encompasses two single-sentence classification tasks, three tasks assessing similarity and paraphrasing, and four tasks focusing natural language inference. Specif- ically, it includes MNLI (MultiNLI, Williams et al. (2018)), SST-2 (Stanford Sentiment Treebank, Socher et al. (2013)), MRPC (Microsoft Research Paraphrase Corpus, Dolan & Brockett (2005)), CoLA (Corpus of Linguistic Acceptability, Warstadt et al. (2019)), QNL (Question NLI, Rajpurkar et al. (2018)), QQP (Quora Question Pairs, Wang ct al. (2017)), RTE (Recognizing Textual Entailment, Dagan al. (2005)). and STS-B (Semantic Textual Similarity Benchmark, Cer et al. (2017)). We summarize the statistical data for all datasets within the GLUE Benchmark in Table 6. Following existing practices, the development set is used in GLUE the test data since the actual test set is not publicly available. We report the overall (matched and mismatched) accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for the other tasks.
The Reuters-21578 (Padmanabhan al., 2016) dataset one of the most widely used data collections for text categorization research. It was collected from the Reuters financial newswire service in 1987 and is used for text classification and natural language processing tasks. Three splits available: ModApte,
18

---

Published in Transactions on Machine Learning Research (07/2025)
ModHayes, and ModLewis. These documents cover various topics, such politics, economics, and sports. F1 score is used as the evaluation metric across all three splits.
A.2 Natural Language Generation
In our experiments on natural language generation, we use the E2E (Novikova et al., 2017) datasct, which was initially introduced as a dataset for training end-to-end, data-driven natural language generation systems. Multiple references can be associated with each source table used as input. Each sample input (z, y) consists of a series of slot-value pairs accompanied by associated natural language reference text. The E2E dataset comprises approxinately 42k training examples, 600 validation examples, and 4, 600 test examples from the restaurant domain.
We utilize the following five evaluation metrics: BLEU (Papineni et al., 2002), NIST (Lin & Och, 2004), METEOR (Banerjee & Lavie, 2005), ROUGE-L (Lin, 2004), and CIDEr (Vedantam et al., 2015).
A.3 Token Classification
For token classification, we fine-tune the RoBERTa- and RoBERIa-large models on the BioNLP dataset (Collier et al., 2004) and the CoNLL2003 dataset (Tjong Kim Sang, 2002). BioNLP (Collier et al., 2004) is Named Entily Recognition dataset that contains biological entities such as DNA, RNA, and protein. It is essentially a token classification task where we want to classify each entity in the sequence. CoNLL-2003 (Tjong Kim Sang, 2002) focuses on langnage-independent named entity recognition. It concentrates on four types of named entities: persons, locations, organizations, and miscellancous entities that do not belong to the previous three groups. Accuracy, precision, recall, and F1 score are used as evaluation metrics.
A.4 Biomedical Experiments
The ESM (Evolutionary Scale Modeling, Rives al. (2021)) model is transformer-based protein language model designed for protein sequence analysis, leveraging the transformer architecture to capture evolutionary patterns. We fine-tune the ESM model using the Protein Aligner checkpoint (Zhang et al., 2024a) on two classification mosks-thermostabiliie
prediction (Chen et al. (2023), 936 training samples) and blood-brain barrier peptide prediction (BBP, Dai et al. (2021), 200 training samples) and one regression task, minimum inhibitory concentration prediction (MTC,
et al. (2023). 3,695 training samples). Notably, protein analysis datasets are typically much smaller than those in NLP, which case the large pre-trained models prone to overfitting, even when using PEFT methods. The trainable parameters (on the order of millions) are significantly overparameterized compared to the available samples (thousands or even hundreds). highlighting the need for our overfitting-resinne resilient counterpart.
B Experimental Settings
In this section, we provide detailed experimental settings. We maintain consistent configurations across experiments, including LoRA rank, LoRA a, batch size, maximum sequence length, and optimizer, to ensure fair comparison. For results other than Table 2, we do not include the bias term in PEFT linear layers. The hyperparameter tuning for our method is straightforward and convenient.
B.1 RoBERTa
We summarize the experimental settings for the GLUE benchmark (Table 1) and for the Reuters21578 dataset and token classification (Table 13) tasks in Table
B.2 GPT-2
We summarize the experimental settings for the GPT-2 experiments (Table 12) in Table 8. The experimental configuration, particularly during the inference stage, follows the approach described by Hu et al. (2022b).
19

---

Published in Transactions on Machine Learning Research (07 (025)
Table 7: The hyperparameters used for RoBERTa on the GLUE benchmark (Wang al., 2019);
Reuters21578 dataset (Padmanabhan et al., 2016), BioNLP dataset (Collier ct al., 2004), and CoNLL2003 dataset (Tjong Kim Sang, 2002).
Settings
MNLI SST-2 MRPC CoLA QNLI QQP RTE STS-B ModApte ModHayes ModLewis BioNLP CoNLL2003
Optimizer
AdamW
Warmup Ratio
0.06
Scheduler
Linear
LoRA rank
rank
LoRA
Total batch size
RoBERTs basee
Global stepe
20k
12k
15k
20k
15k
12k
20k
20k
12k
12k
Lower learning
Ge-5
le-5
2e-5
5e-5
2e-5
5e-5
le-5
le-5
3e-5
3e-5
3e-5
le-5
2e-5
Upper learning rate
5o-5
1c-5
20-5
5o-5
26-5
5e-5
16-5
1o-5
3e-5
Je-5
Jo-5
1o-5
26-5
Lower weight decay
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.1
0.2
Upper weight decay
0.1
0.1
0.1
0.1
0
0.1
0.1
0.01
0.1
0.1
0.1
0.1
0.1
Max Seq Length
612
Regularization Coefficient
1e-5
1c-5
1e-5
Te-5
16-5
1e-5
le-5
16-5
le-5
16-5
Total batch size
32
RoBERTs Ta-eeg-- large
Global steps
50k
20k
30k
20k
60k
40k
15k
10k
20k
20k
20k
12k
15k
Lower learning rate
le-5
le-5
le-5
le-5
le-5
le-5
le-5
1c-5
1c-5
le-5
1c-5
26-5
lc-5
Upper learuing rate
le-5
le-5
le-5
le-5
le-5
le-5
le-5
lc-5
le-5
lc-5
lc-5
26-5
lc-5
Lower weight decay
0.5
0.5
0
0.2
0.5
0.5
0.5
0.5
0.2
0.1
0.2
0.02
0.1
Upper weight decay
0.5
0.05
0.2
0.5
0.5
0.1
0.5
0.1
0.1
0.1
0.02
0.1
Max Seq Length
128
Regularization Coefficient
0
1e-5
le-5
0
1e-5
le-5
1e-5
0
le-5
Table 8: The hyperparameters we used for GPT-2 the E2E NLG benchmark (Novikova et al., 2017).
Settings
Training
Optimizer
AdamW
Warmup Ratio
0.06
Scheduler
Linear
LoRA rank
rank. rank, =
LoRA
32
Label Smooth
0.1
Lower learning rate
1e-3
Upper learning rate
le-4
Lower weight decay
Upper weight decay
Max Seq Length
512
Regularization Coefficient
1e-5
Settings
Inference
Beam Size
10
Length Penalty
0.9
no repeat ngram size
C
Baselines in Experiments
Here, we provide a brief introduction compare baselines all our experiments.
Full Fine-Tuning (FT): The entire model is fine-tuned, with updates to all parameters.
Adapter Tuning (Houlsby et al., 2019; Lin al., 2020; Riicklé al., 2021; Pfeiffer et al., 2021): Methods that introduce adapter layers between the self-attention and MLP modules for parameter-efficient tuning.
20

---

Published in Transactions on Machine Learning Research (07/2025)
LoRA (Hu et al., 2022a): A method that estimates weight updates via low-rank matrices.
AdaLoRA (Zhang et al., 2023): An extension LoRA that dynamically reallocates the param- eter budget based on importance scores.
DoRA (Liu et al., 2024a): Decomposes pretrained weights into magnitude and direction, using
LoRA for efficient directional updates.
VeRA (Kopiczko et al., 2024): Employs single pair of low-rank matrices across all layers to reduce trainable parameters.
FourierFT (Gao et al., 2024): Fine-tunes models by learning subset of spectral coefficients in the Fourier domain.
AFLoRA (Liu et al., 2024b): Freezes low-rank adaptation parameters using a learned score, reducing trainable parameters while maintaining performance.
LaMDA (Azizi et al., 2024): Fine-tunes large models via spectrally decomposed low-dimensional
adaptation.
SSH (Shen et al., 2025b): Fine-tunes large models after transforming weight matrices with the discrete Hartley transformation (DHT).
MaCP (Shen et al., 2025a): Fine-tuncs large models by projecting the low-rank adaptation weight change into the discrete cosine space.
D Weight Decomposition Analysis
We provide brief review of the weight decomposition analysis proposed in Liu et al. (2024a). Define the weight decomposition of weight matrix W E gaxk (e.g., query matrix in an attention layer) W == == [ || W | where m E Rixk is the magnitude vector, and V E Ra is the directional matrix, with || lle representing the vector-wise norm of a matrix across each column. This decomposition ensures that each column of V/IlVile remains unit vector, and the corresponding scalar in m defines the magnitude of each vector. Liu et al. (2024a) examine the magnitude and directional variations between Wo
and WET, defined as AMPT Σ Imp [m//'mm) and AD (1- Wa")) Here, AMFT and AD represent the magnitude and direction differences between Wo and WPT at the t-th training step, respectively, with cos(-.-) denoting cosine similarity. MFT magnitude vectors, while V and Wo" are the th columns in Vp PT and Wo. Intuitively, a consistent positive slope trend across all the intermediate steps implies difficulty in concurrent learning of both magnitude and direction, suggesting that slight directional changes are challenging to execute alongside more significant magnitude alterations. In contrast, a relatively negative slope signifies a more varied learning pattern, with a more pronounced negative correlation indicating a larger learning capacity.
n.t and my are the th scalars in their respective
Complementary to Fig. 3 in the main paper on the query matrix, we provide additional results of weight decomposition analysis in Fig. 4 on the value matrix to complement the findings in Section 5.5. We can draw two key observations from Fig. 4: 1) Consistent with the results in Liu et al. (2024a), both FT and DoRA exhibit negative correlation values of -49.279 and -5.485, respectively, while LoRA shows positive correlation with a value of 2.503. 2) BiDoRA achioves a negative correlation value of 100.547, indicating closer alignment with FT compared to DoRA. The analysis of how BiDoRA achieves this improvement is similar to that discussed in Section 5.5.
E
Training Cost
Table compares the training efficiency of LoRA, DoRA, and BiDoRA on the GLUE benchmark using the RoBERTa-bee base model. The table details the total training steps required for convergence and the per-step computational cost, which is normalized relative to LoRA for reference. For fair comparison, all methods
21

---

Published in Transactions on Machine Learning Research (07 (025)
+48
t-2 220
-445
Logor
2
Laper
12
61
110p3
2010 100
00
60
017
Alc con
(a) FT = 49.279)
(b) LoRA(k = 2.503)
(c) DoRA(k = 5.485)
(d) BiDoRA(k = 10.5)
Figure 4: Magnitude and direction updates for (a) FT, (b) LORA, (c) DoRA, and (d) BiDoRA of the value matrices across different layers and intermediate steps after finc-tuning the GPT2 model on the E2E dataset (Novikova et al., 2017). Different markers represent matrices from different training sleps, while different colors indicate matrices from each layer. The values of negative correlation are shown at the top, denoted by k.
Table Average training time cost on the GLUE benchmark (Wang et al., 2019).
Method
LoRA
DoRA
BiDoRA
Per-step cost
x1
x1.36
x3.54
Total steps
27.45k
27.45k
17.37k
Total time
x1.36
x2.24
were benchmarked single NVIDIA A100 GPU. The results show that BiDoRA converges in fewer steps than LoRA and DoRA, while the per-step cost for BiDoRA is modestly higher, as its BLO process requires iterative updates between the two levels and the computation of hypergradients. The total training time for BiDoRA is approximately 1.64 times that of DoRA, a training cost that remains comparable to the baselines. Given BiDoRA's superior performance across various lasks, we argue that this slight increase in computational cost is an acceptable trade-off, underscoring our method's practicality.
F The Role of Hyperparameter
The hyperparameter tuning for BiDoRA is simple, convenient, and straightforward. We further conducted experiments regarding the dataset partition of D,, and Dvel provide insights into its role in BiDoRA. The dataset partition helps maintain the balance of inner/outer optimization by assigning different portions of data. The direction component has more trainable parameters, so it is reasonable to use more data for training the lower level while using the remaining data for training magnitudes. As shown in Table 10, we varied the inner-level dataset Drr partition from 0.6 to 1.0 with 0.1 intervals and experimented with RoBERTa-base on three splits of the Reuters21578 datasct examine its influence.
The results indicate that both extreme cases are negative to the overall performance. When the inner partition is too small (S 0.6), directions are not well-trained, and when the inner partition is 1.0, magnitudes are not trained at all, leading to significant performance drop. These findings demonstrate that BLO is ellective in the sense that both levels are necessary for enhancing performance. Although luning the partition ratio may further improve overall performance, we maintain a consistent data partition of 8 2 in all the experiments for simplicity. A fixed configuration of data partition already consistently yields superior performance of BiDoRA, demonstrating that our method is robust this hyperparameter within certain range.
22

---

Published in Transactions on Machine Learning Research (07 (075))
Table 10: Experiment results on different data partitions of BiDoRA.
Partition
ModApte
ModHayes
ModLewis
0.6
85.32
79.76
77.69
0.7
85.32
80.01
77.74
0.8
85.34
79.93
77.63
0.9
85.27
79.85
77.64
1.0
85.23
79.59
77.42
Table 11: Experiment results different weight decay values and different dropout rates of DoRA.
Method
CoLA
MRPC
RTE
DoRA (weight decay 0)
59.3
88.7
72.9
DoRA (weight decay 0.05)
60.1
89.2
73.3
DoRA (weight decay 0.1)
60.5
89.2
73.2
DoRA (weight decay 0.2)
60.3
89.0
73.2
DoRA (dropout rate 0)
59.2
89.2
72.9
DoRA (dropout rate 0.1)
60.2
88.9
71.4
DoRA (dropout rate 0.2)
55.1
87.8
64.2
BiDoRA
61.3
89.4
76.0
G
Comparison with Other General Methods for Addressing Overfitting
There common experimental settings that may be used to reduce overfitting. For DoRA, two promising methods are increasing weight decay and adopting more aggressive dropout rate. We conducted experiments on these two methods separately. We kept hyperparameters that have been well-tuned in DoRA and can achieve optimal results while only tuning the weight decay value. Similarly, we tune the dropout rate of DoRA while keeping the weight decay value to be optimized. We conducted experiments on RoBERTa-base on three datasets. The results are presented in Table 11.
We can draw the observation that neither of these approaches effectively addresses overfitting issues enhances the model's generalization ability. On the other hand, BiDoRA exploits the specific magnitude- direction structure of DoRA and the strategy of training the two distinct components on separate splits of the dataset. An advantage of our methodology is that it can be easily combined with other general-purpose overfitting-alleiiainn strategics since BiDoRA does not modify the original DoRA architecture.
Additional Experiments
H.1 Experiments on Natural Language Generation Tasks
In this section, we evaluate BiDoRA's performance on the NLG task. Table 12 presents the results of fine- tuning a GPT-2 model on the E2E datasct with baseline PEFT methods and BiDoRA. The results show that BiDoRA achieves the best performance across five evaluation metrics, demonstrating the superiority of BiDoRA in fine-tuning pre-trained models for NLG tasks.
H.2 Experiments on Token Classification
The effectiveness BiDoRA can also be observed Table 13, which reports the results of token classification tasks. Unlike the NLU tasks discussed in the previous scction, which involve classifying entire sentences and focusing on capturing global semantics, token classification requires classifying each token within sentence, highlighting the importance of capturing local context. On the BioNLP dataset, BiDoRA consistently
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table 12: Performance of BiDoRA and baseline methods for fine-tuning GPT2-medium on the E2E dataset
(Novikova et al., 2017). A higher value is better for all metrics. The best results shown in bold.
Method
#Params
BLEU
NIST
MET
ROUGE-L
CIDEr
FT
354.9M
68.0
8.61
46.1
69.0
2.38
Adapter
11.1M
67.0
8.50
45.2
66.9
2.31
LoRA
0.39M
67.1
8.54
45.7
68.0
2.33
DoRA
0.39M
67.0
8.48
45.4
70.1
2.33
BiDoRA
0.39M
69.0
8.72
46.2
70.9
2.44
outperforms baseline methods by a large margin in terms of Fl score. On the CoNLL2003 dataset, BiDoRA either outperforms or matches all bascline methods across all metrics. Consistent with our previous findings, BiDoRA effectively fine-Lunes pre-trained models for token classification tasks.
H.3 More Experiments on Natural Language Understanding Tasks
Table presents the results of fine-tuning RoBERTa models on the Reuters21578 datasets, a text classifi- cation task, where BiDoRA outperforms all baseline methods by an even larger margin. Notably, BiDoRA achieves performance comparable to or even better than full fine-tuning, providing further evidence of its superiority.
Table 13: RoBERTa bbase/large (Rb/) with different fine-tuning methods the Reuters21578 (Padmanabhan et al., 2016), BioNLP (Collier al., 2004), and CoNLL2003 (Tjong Kim Sang, 2002) benchmarks. A higher value is better for all metrics. The best results are shown in bold.
Reuters21578
BioNLP
CoNLL2003
Method
# Params ModApte ModHayes ModLewis Accuracy Precision Recall F1
Acccurayy Precision Recall Fl
RFT)
125.0M
85.4
77.6
77.1
93.9
69.0
78.9
73.6
99.3
95.7
96.3
96.0
Ra (Adapter)
0.911
85.3
77.5
76.8
93.9
69.1
78.8
73.7
99.3
95.7
96.4
96.0
Rs, (LoRA)
0.15M
84.7
74.3
74.7
93.9
69.0
78.8
73.6
99.3
95.4
96.3
95.8
Rs R₂ (DoRA)
0.17M
81.8
78.2
76.6
94.0
69.2
79.1
73.8
99.3
95.3
96.2
95.8
Rs (BiDoRA)
0.17M
85.3
79.9
77.6
93.9
71.2
78.6
74.7
99.3
95.9
96.5
96.2
R2(FT)
355.0M
84.8
77.5
76.6
94.0
69.4
79.6
74.1
99.4
96.2
97.0
96.6
(Adapter))
0.44M
84.8
77.9
76.7
94.0
69.4
79.7
74.2
99.4
96.1
97.0
96.6
(I(LoRA)
0.39M
84.7
77.7
76.7
93.9
69.2
79.3
73.9
99.4
96.2
97.0
96.6
(DDRAA)
0.39M
81.8
77.4
76.7
94.0
69.4
79.7
71.2
99.4
96.2
97.1
96.6
R (BiDoRA)
0.39M
84.9
78.9
77.3
94.0
71.3
79.3
75.1
99.4
96.4
97.1
96.7
Evidence on Orthogonality of Incremental Matrix
To verify that the orthogonal regularization (OR) proposed in Section 4.2 encourages the columns of the direction matrix to orthogonal, we visualize the normalized eigenvalues of the matrix in Fig. J. The results show that, compared to methods without OR (i.e., DoRA and BiDoRA w/o cst.), BiDoRA with OR produces eigenvalues that are more closely aligned with those of a purely orthogonal matrix, where all eigenvalues would be one. This effect holds for both the query and value matrices and verifies the effectiveness of the OR constraint.
24

---

Published in Transactions on Machine Learning Research (07/202) 2025)
168
160
Remalnsz aggrnaaa
lpord
(a) DoRA-query matrix
(b) BiDoRA (w/o est.)-query matrix
(c) BiDoRA-query matrix
Bye
kp214
kyc:4
kp2r4
loort
Lijeraaue
Exjeriolae rOrk
Egporatue
(d) DoRA-vahe matrix
BIDoRA (w/o vvauu- matrix
(f) BiDoRA-value matrix
Figure Eigenspectra the direction matrix for query (top) and value (bottom) matricos across different layers. The figure compares three fine-tuning methods: BiDoRA, BiDoRA without orthogonal regularization (w/o cst.), and DoRA. Both axes visualization clarity. Experiments were conducted on the CoLA dataset (Warstadt et al., 2019) with the RoBERTa-bar model.
log scale, and only the 64 largest eigenvalues shown for
25


Reviews:
[{"review_id": "UO4Ww0b6ZW", "reviewer": "Reviewer_oMbr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- PEFT is widely adopted across various large models, including LLMs, making this a highly relevant and important topic for the machine learning community.\n- The use of bi-level optimization to separately estimate magnitude and direction parameters is a novel contribution.\n- The experimental evaluation is thorough, and the ablation studies are well-designed and informative.\n\nWeaknesses\n- The motivation for separately estimating magnitude and direction parameters remains unclear. While the paper identifies two issues with simultaneous estimation—(1) increased expressiveness potentially leading to overfitting, and (2) a coupled update pattern that may limit learning capacity—it does not provide sufficient explanation or evidence to support these claims.\n- Specifically, it is not demonstrated whether overfitting or limited learning capacity actually occur in practice, nor is it clearly shown how bi-level optimization addresses these issues, either logically or through experimental validation. This lack of clarity is a major concern.\n- In Section 2, the relationship between the proposed method and related work is not clearly delineated. While some differences can be inferred, it would strengthen the paper to make these distinctions explicit.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an extension to DoRA by separately and iteratively optimizing the magnitude and direction of parameters in the PEFT setup. This approach is framed as a form of bi-level optimization, a technique commonly used in neural architecture search. The authors identify a limitation in the original DoRA method, namely the difficulty of estimating magnitude and direction simultaneously, and address it with their proposed optimization strategy. The method is evaluated on a range of NLP tasks, with experimental results demonstrating its effectiveness, though improvements are occasionally marginal. The paper also includes ablation studies that support the contribution of bi-level optimization and other proposed techniques.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Major requests\n- Please provide a clearer explanation of the overfitting issue and the limitation in learning capacity associated with the simultaneous estimation of magnitude and direction. Additionally, explain how the proposed bi-level optimization approach addresses these problems, both conceptually and empirically.\n- Clarify the distinctions between the proposed method and related work. While some differences are implied, an explicit comparison would help readers better understand the novelty of the approach.\n\nMinor comments for improving the paper\n- Section 4.1: The connection between the methods mentioned in the last two sentences and the proposed approach is unclear. Please elaborate on how these methods relate to the main contribution.\n- Section 4.2: Orthogonal Regularization is introduced without sufficient context. Although its empirical effectiveness is shown in Table 4, the rationale behind its use is not discussed. Please explain its role and importance in the proposed method.\n- Section 4.3: The explanation of the \"Optimization Algorithm\" section, particularly regarding the analytical solution for $\\mathcal{V}^{\\ast}(\\mathcal{M})$, is confusing. Since neural networks generally do not yield analytical solutions due to their non-linear nature, this claim requires clarification. I may be misunderstanding the point, so a more detailed explanation would be helpful.\n- Section 5: There is no discussion of training cost. Since LoRA, DoRA, and other PEFT methods are typically valued for their efficiency and lightweight training requirements, it would be beneficial to include a comparison or analysis of training costs for the proposed method.", "broader_impact_concerns": "There is no broader impact concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TWsA31XOyL", "reviewer": "Reviewer_t6jb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper provides a principled analysis of the coupled relationship between the direction and magnitude in LoRA and develops a novel algorithm to address the problem.\n2. The proposed BiDoRA shows effective results in experiments, providing empirical validation on a wide range of tasks and model scales.\n\nWeaknesses:\n1. Figure 2 is not professional: axis arrows are missing, and there is an additional distracting symbol in the top right corner.\n2. For updating the magnitude $M$, why is the validation dataset used rather than the training dataset? Besides, for retraining the direction $V$, why are the training and validation datasets used? The rationale should be better justified. \n3. In Algorithm 1, it is unclear how the convergence of BiDoRA is determined. What is the specific stopping criterion for the search phase and retraining phase?\n4. The experiments fix the LoRA rank to 4, but do not report BiDoRA’s performance under varying ranks. It would be informative to understand how the method scales with rank.\n5. While retraining the direction improves BiDoRA’s performance, it incurs additional computational cost. Moreover, since the direction and magnitude are coupled, it is unclear why retraining the magnitude is not performed. If one component is updated, why not adjust the other? Does this mean the convergence condition of the search phase is insufficient or suboptimal?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper analyzes the coupled relationship between the direction and magnitude components in LoRA. Based on this insight, the authors propose BiDoRA, a bi-level optimization framework that decouples the optimization of magnitude and direction to improve performance and mitigate overfitting. The experimental results show that BiDoRA consistently outperforms DoRA and LoRA, similar to full fine-tuning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses. Specifically:\n\n1. Improve the clarity and presentation of Figure 2.\n2. Justify the use of the validation dataset for magnitude optimization and the training and validation datasets for retraining direction.\n3. Specify explicit convergence criteria for both the search and retraining phases.\n4. Include results of BiDoRA across different LoRA ranks to demonstrate robustness.\nClarify whether retraining the magnitude was considered, and discuss whether retraining only the direction leads to sub-optimal coupling. Address whether this choice impacts the validity of the search phase convergence.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nts2OuBqZ7", "reviewer": "Reviewer_nEhv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n+ Overall this work is moderately novel for publication.\n\n+ This paper is generally easy-to-follow with enough clarity.\n\nWeakness: However, there are still multiple issues in terms of experimental validation, recent state-of-the-art works and presentation. Therefore, a major revision is needed before possible publication. Please refer to the specific comments as follows.\n\nMajor concerns:\n \n- Sec.5.6, ablation study. More visual analysis and empirical evidence is needed to support the effectiveness of the orthogonal regularization. This could be, for example, a feature space comparison, or a distance measurement comparison, with and without the use of this operation. \n\n- Sec.5.7, when discussing the weight decomposition, there is a lack of the visualized weigh matrix. It would be much more convincing to compare the weight matrix with and without the use of the proposed method. \n\n- The abstract needs a significant improvement.\n\n(1) The background of this task is too long, occupying 60% of its content. Please make it more concise.\n\n(2) The abstract should mention the numerical improvement.\n\n- The variation of LoRA has been extensively studied in the past three years. This paper lacks a discussion, and if necessary a comparison, with some more recent state-of-the-art LoRA methods, for example:\n\n[1] Shen, Yixian, et al. \"SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation.\" NAACL, 2025.\n\n[2] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"LoRA+: Efficient Low Rank Adaptation of Large Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Shen, Yixian, et al. \"MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection.\" ACL, 2025.\n\nPlease enrich it for discussion and if necessary comparison. \n\n- There are still multiple issues on the mathematical notations and equations. For example:\n\n(1) All the tensors and matrices should be in bold to distinguish them from the scalar. \n\n(2) ‘|| · ||_c represents the vector-wise norm of a matrix compute …’. Which specific norm operation do you use? Please clarify.\n\n(3) For the loss function in Sec.4.3. It may be better to keep consistent to use $\\mathcal{L}$ instead of $\\mathcal{C}$.\n\n(4) An equation in the optimization algorithm does not have the equation number.\n\n- In the related work section, the adapter based PEFT methods are too old. The authors are suggested to discuss more recent PEFT methods in 2023-2024, and give their due credit to the adapter field. For example:\n\n[4] Bi, Qi, et al. \"Learning frequency-adapted vision foundation model for domain generalized semantic segmentation.\" Advances in Neural Information Processing Systems 37 (2024): 94047-94072.\n\n[5] Xu, Mengde, et al. \"Side adapter network for open-vocabulary semantic segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[6] Yi, Jingjun, et al. \"Learning spectral-decomposited tokens for domain generalized semantic segmentation.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n- Caption of Fig.3. Please define $\\Delta D$ and $\\Delta M$ in the caption. \n\n- Can some failure cases and limitation be discussed at the end of this paper?\n\n- In the Section E of the appendix, please explicitly mention which specific type of the hardware the authors use for the experiments. \n\nMinor comments for improvement:\n\n- In all the tables’ caption, when there is a dataset, please cite it.\n\n- The content in the top slide of Fig.1 should be made along the same height.\n\n- The caption of Fig.2. Please cite the dataset. \n\n- There are still multiple grammar issues in this paper. The authors need a throughout check. For example:\n\n(1) ‘…, where architecture and sub-network are learned …’\n\n(2) ‘Furthermore, the asynchronous gradient update steps at the two optimization levels in BiDoRA facilitate better decoupling of two components’", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper focuses on parameter-efficient fine-tuning (PEFT), which is a fundamental problem to harness the foundation model to various down-stream tasks. \n\nThe authors proposes a variation of weighted decomposed low-rank adaptation (DoRA), which leverages the bi-level optimization to mitigate the risk of overfitting.\n\nExperiments on multiple downstream tasks show its effectiveness over the baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to each point of weakness than has been mentioned, which the reviewer has been attached as follows. \n\nMajor concerns:\n \n- Sec.5.6, ablation study. More visual analysis and empirical evidence is needed to support the effectiveness of the orthogonal regularization. This could be, for example, a feature space comparison, or a distance measurement comparison, with and without the use of this operation. \n\n- Sec.5.7, when discussing the weight decomposition, there is a lack of the visualized weigh matrix. It would be much more convincing to compare the weight matrix with and without the use of the proposed method. \n\n- The abstract needs a significant improvement.\n\n(1) The background of this task is too long, occupying 60% of its content. Please make it more concise.\n\n(2) The abstract should mention the numerical improvement.\n\n- The variation of LoRA has been extensively studied in the past three years. This paper lacks a discussion, and if necessary a comparison, with some more recent state-of-the-art LoRA methods, for example:\n\n[1] Shen, Yixian, et al. \"SSH: Sparse Spectrum Adaptation via Discrete Hartley Transformation.\" NAACL, 2025.\n\n[2] Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"LoRA+: Efficient Low Rank Adaptation of Large Models.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Shen, Yixian, et al. \"MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection.\" ACL, 2025.\n\nPlease enrich it for discussion and if necessary comparison. \n\n- There are still multiple issues on the mathematical notations and equations. For example:\n\n(1) All the tensors and matrices should be in bold to distinguish them from the scalar. \n\n(2) ‘|| · ||_c represents the vector-wise norm of a matrix compute …’. Which specific norm operation do you use? Please clarify.\n\n(3) For the loss function in Sec.4.3. It may be better to keep consistent to use $\\mathcal{L}$ instead of $\\mathcal{C}$.\n\n(4) An equation in the optimization algorithm does not have the equation number.\n\n- In the related work section, the adapter based PEFT methods are too old. The authors are suggested to discuss more recent PEFT methods in 2023-2024, and give their due credit to the adapter field. For example:\n\n[4] Bi, Qi, et al. \"Learning frequency-adapted vision foundation model for domain generalized semantic segmentation.\" Advances in Neural Information Processing Systems 37 (2024): 94047-94072.\n\n[5] Xu, Mengde, et al. \"Side adapter network for open-vocabulary semantic segmentation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[6] Yi, Jingjun, et al. \"Learning spectral-decomposited tokens for domain generalized semantic segmentation.\" Proceedings of the 32nd ACM International Conference on Multimedia. 2024.\n\n- Caption of Fig.3. Please define $\\Delta D$ and $\\Delta M$ in the caption. \n\n- Can some failure cases and limitation be discussed at the end of this paper?\n\n- In the Section E of the appendix, please explicitly mention which specific type of the hardware the authors use for the experiments. \n\nMinor comments for improvement:\n\n- In all the tables’ caption, when there is a dataset, please cite it.\n\n- The content in the top slide of Fig.1 should be made along the same height.\n\n- The caption of Fig.2. Please cite the dataset. \n\n- There are still multiple grammar issues in this paper. The authors need a throughout check. For example:\n\n(1) ‘…, where architecture and sub-network are learned …’\n\n(2) ‘Furthermore, the asynchronous gradient update steps at the two optimization levels in BiDoRA facilitate better decoupling of two components’", "broader_impact_concerns": "The reviewer does not envision negative societal or broader negative impact on this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)