DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution (2024)

Abstract: Fine-tuning large-scale pre-trained models is inherently a resource-intensive task. While it can enhance the capabilities of the model, it also incurs substantial computational costs, posing challenges to the practical application of downstream tasks. Existing parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) rely on a bypass framework that ignores the differential parameter budget requirements across weight matrices, which may lead to suboptimal fine-tuning outcomes. To address this issue, we introduce the Dynamic Low-Rank Adaptation (DoRA) method. DoRA decomposes high-rank LoRA layers into structured single-rank components, allowing for dynamic pruning of parameter budget based on their importance to specific tasks during training, which makes the most of the limited parameter budget. Experimental results demonstrate that DoRA can achieve competitive performance compared with LoRA and full model fine-tuning, and outperform various strong baselines with the same storage parameter budget. Our code is available at https: //github.com/MIkumikumi0116/DoRA

heading: 1 Introduction
text: Pre-trained Language Models (PLMs) (Kenton and Toutanova, 2019; Brown et al., 2020; Liu et al., 2019; He et al., 2020, 2021b) play a crucial role in Natural Language Processing (NLP), offering substantial improvements in various downstream tasks (Lee et al., 2020; Mars, 2022; Raffel et al., 2020). Customizing these models for specific tasks typically involves fine-tuning them to adapt pretrained knowledge to particular requirements (Alabi et al., 2022; Uppaal et al., 2023). However, with the increasing scale of PLMs, the cost of fullmodel fine-tuning becomes prohibitive (Qiu et al., 2020). This has highlighted the demand for and
*Equal contributions. †Corresponding author.
increased interest in more parameter-efficient finetuning (PEFT) methods (Zeng et al., 2023; Ding et al., 2023b).
Common PEFT methods introduce extra parameters to adapt downstream tasks and freeze all original parameters (Li and Liang, 2021a; Liu et al., 2022; Lester et al., 2021a). For instance, Low-Rank Adaptation (LoRA) (Hu et al., 2022a) has gained popularity for its streamlined approach by incorporating low-rank trainable matrices into existing fixed weight matrices in a PLM. However, LoRA assigns trainable parameters uniformly across all matrices, and there are studies (Zhang et al., 2023) indicate that not all weights contribute equally to fine-tuning performance. This could result in inefficient parameter usage. Therefore, for optimal fine-tuning, is it possible to evaluate the parameter budget needs of each matrix and strategically allocate the limited parameters?
Fortunately, there are methods like AdaLoRA (Zhang et al., 2023) that can alleviate the limitations of the prior PEFT methods by introducing a more nuanced parameter distribution strategy. Training with AdaLoRA begins with a higher parameter budget and simulates an SVD (Singular value decomposition) decomposition process, progressively pruning smaller singular values and corresponding singular vectors. It opens the door to implementing adaptive allocation of parameter budgets. However, its dependence on orthogonal regularization for the simulated SVD decomposition might restrict further improvements in fine-tuning efficiency. Additionally, the pruning strategy of AdaLoRA focuses solely on singular values and does not fully exploit all the available information in projection matrices, potentially leading to less-than-optimal decisions.
To address existing challenges, this work introduces the Dynamic Low-Rank Adaptation (DoRA) method, as depicted in Figure 1. Different from LoRA approaches, DoRA decomposes high-rank
11662
LoRA layers into sums of single-rank components, evaluates the contribution of each component to overall performance, and prunes components with fewer contributions. This allows for the on-demand allocation of parameter budgets to modules of the PLM, maximizing the use of limited parameter budgets. Compared to existing methods of dynamic parameter allocation (e.g., AdaLoRA), DoRA can allocate parameter budgets more appropriately based on a richer set of information from projection matrices.
To sum up, our contributions are as follows:
• We introduce a novel PEFT method, DoRA, which surpasses the performance of full model fine-tuning with less than 0.3% of the trainable parameters.
• DoRA can efficiently identify the modules in PLMs that play a crucial role in the finetuning task, thereby allocating a larger parameter budget to these key modules.
• DoRA maximizes the use of a limited parameter budget. Experiments demonstrate that DoRA outperforms baseline approaches across multiple downstream tasks under the same parameter budget constraints.

heading: 2 Background
text: The emergence of PLMs such as BERT (Kenton and Toutanova, 2019), GPT (Radford et al., 2019), and Llama (Touvron et al., 2023) has meaningfully advanced the field of NLP. Trained on extensive text datasets, PLMs capture intricate linguistic patterns, enabling superior performance in various NLP tasks such as text categorization, named entity recognition, and machine translation (Zhao et al., 2023). Their flexibility in adapting to specific datasets via fine-tuning renders them exceedingly versatile for addressing various linguistic challenges.
PLMs predominantly leverage the Transformer architecture (Vaswani et al., 2017) which features stacked Transformer blocks. Each block comprises two key components: a Multi-Head Attention (MHA) mechanism and a Feed-Forward Neural (FFN) network. In particular, MHA effectively captures contextual relationships in text and is given by:
MHA(x) = Concatenate(head1(x),
head2(x), . . . , headh(x))Wo (1)
headi(x) = Softmax ( (xWqi)(xWki) T
√ dh
) xWvi
(2)
where x ∈ Rn×d is the input feature, n is the sequence length and d is the hidden dimension. The mechanism consists h self-attention heads, each aiming to capture different aspects of information. For each head headi, there are three projection matrices: query Wqi, key Wki, and value Wvi, each with dimensions Rd×dh , where dh is the dimension of each head, typically set to d/h. The output projection matrix Wo ∈ Rd×d is used to produce the final output.
Attention scores are calculated by normalizing the dot product of queries and keys through the softmax function and are given by:
Softmax(xi) = exi∑n j=1 e xj (3)
These scores determine the attention each sequence position pays to other positions. Subsequently, these scores are multiplied by the value projection result to yield the output of each head. Finally, the outputs of all heads are concatenated and multiplied by the output projection matrix Wo, forming the final MHA output.
Following MHA, the FFN further processes the information:
FFN(x) = ReLU(xWf1 + b1)Wf2 + b2 (4)
This allows for more complex interactions between the features extracted by the self-attention mechanism. Each Transformer block incorporates a residual connection that adds the input of the block directly to its output. This approach helps to alleviate the vanishing gradient problem and ensures a consistent information flow across the layers of the models.

heading: 3 Dynamic Low-Rank Adaptation
text: In this paper, we aim to optimize the use of a limited parameter budget in fine-tuning PLMs with LoRA. We make improvements based on LoRA (Hu et al., 2022a) and AdaLoRA (Zhang et al., 2023), as shown in Table 1. We propose
Algorithm 1 DoRA 1: Input: Dataset D; total steps T ; initial budget
b(0); final budget b(T ); learning rate γ; regularization coefficient η; smoothing factor β, DoRA parameters A, B, and c. 2: for t = 1 to T do 3: Sample a mini-batch d from D and compute the true label loss Ltrue = L(A,B, c, d); 4: Compute the regularization loss Lreg as Equation 10; 5: Combine losses by adding true label loss
and regularization loss Lcombined = Ltrue + ηLreg;
6: Perform backpropagation to compute the gradients of Lcombined, and update the parameters with the learning rate γ; 7: Compute the importance score s as Equation 7, update smoothed importance score s̃(t) as Equation 8; 8: Compute the current parameter budget b(t) as Equation 9; 9: Prune the components with smaller s̃(t) based on b(t), set their c to 0; 10: end for 11: Output: The fine-tuned parameters {A,B, c}.
DoRA that stands out due to its innovative approach, comprising three main strategies: a decomposition strategy that views a high-rank LoRA layer as a combination of multiple single-rank LoRA components, a dynamic rank allocation mechanism that adjusts these components based on their contribution to the overall performance of the model and a regularization penalty to ensure stable pruning throughout the process. The overall algorithm is shown in Algorithm 1.

heading: 3.1 Parameterization
text: DoRA introduces a novel perspective on PEFT for PLMs, building upon and enhancing the foundational LoRA technique. A standard LoRA layer is defined as:
W = W0 +∆W = W0 +AB (5)
where W is the weight matrix after fine-tuning, W0 denotes the original weight matrix, and A, B are the low-rank matrices introduced by LoRA. By contrast, DoRA reinterprets this configuration and is given by:
W = W0 +
r′∑
i=1
∆Wi = W0 +
r′∑
i=1
AiBici (6)
here, r′ represents the number of LoRA components, which will be explained in detail in Section 3.3. A LoRA component is a triplet of Ai, Bi, and ci, where Ai and Bi are single-rank matrices, shaped as d×1 and 1×d respectively. ci is a scalar used for pruning the component, it is set to 0 if the component is to be pruned.

heading: 3.2 Importance Scoring
text: To evaluate the importance of each LoRA component, we employ an importance scoring mechanism that quantifies the contribution of each ∆Wi and is given by:
si = ∥∆Wi∥F /∥ r′∑
j=1
∆Wj∥F
= ∥AiBici∥F /∥ r′∑
j=1
AjBjcj∥F
(7)
here, ∥x∥F denotes the Frobenius norm, a measure that calculates the square root of the sum of the squares of all elements in a matrix.
Employing the Frobenius norm allows us to measure the proportion of each LoRA component contribution to the total update magnitude of its corresponding LoRA layer. This metric facilitates an estimation of the potential impact on the total update of the LoRA layer if a particular component were to be pruned. Components with smaller impacts on the overall update magnitude are prioritized for pruning. This ensures that the pruning process minimally affects the performance, focusing on removing components that contribute the least to the effectiveness of the LoRA layer.
Compared to previous methods (Zhang et al., 2023), we use ∥∆Wi∥F instead of ci to assess the importance of components, thereby incorporating information from Ai and Bi for a more comprehensive evaluation of component importance.
Moreover, to enhance the precision of the importance score, we employ a smoothing method by
applying an exponential moving average to the importance scores. The smoothed importance score for the i-th LoRA component at time t, denoted as s̃i(t), blends the current importance score si with the previous one, adjusted by a factor β:
s̃i(t) = β · s̃i(t− 1) + (1− β) · si (8)

heading: 3.3 Parameter Scheduling and Pruning Strategy
text: Parameter budget refers to the average number of LoRA components in each LoRA layer. It starts with an initial parameter budget, b(0) = r′, which is deliberately set higher than the eventual target budget, b(T ) = r, where r′ and r are hyperparameters. Setting r′ greater than r allows DoRA to explore a wider range of potential parameter allocations, facilitating the search for the optimal distribution.
DoRA adopts a gentle pruning strategy. For the pruned triplets Ai, Bi, and ci, pruning is performed merely by setting ci to 0 while keeping Ai and Bi unchanged. During subsequent training, the pruned triplets can be restored as long as ci is updated to a non-zero value by backpropagation and is not pruned again.
DoRA warms up the training without pruning for the first ti step, i denotes initial steps and then follows a cubic decrement pattern to prune components with lower importance scores until the remaining components reach the budget b(T ). Subsequently, it fixes the component distribution in the last tf steps, f denotes the final steps. The overall budget scheduler is given by:
b(t) =    b(0) if 0 ≤ t < ti, b(0) − (b (0) − b(T )) b(0) · ( t− ti tf − ti )3 if ti ≤ t ≤ T − tf ,
b(T ) if t > T − tf . (9)

heading: 3.4 Dimensional Equilibrium Modulator
text: DoRA utilizes the Frobenius norm of components for pruning, with a preference for clipping those with smaller norms. However, a potential issue arises when a component has most elements near zero and a few with considerably high values, leading to a relatively low Frobenius norm and thus being selected for pruning. This scenario can result
in remarkable alterations in a limited number of dimensions of the total update, ∆W , resembling the effects of gradient explosion and adversely impacting model stability and fine-tuning performance.
To avoid this, we introduce the Dimensional Equilibrium Modulator (DEM) loss, which penalizes the variance of components as:
R = 1
n
n∑
i=1
(Var(Ai) + Var(Bi)) (10)
where Var(Ai) and Var(Bi) represent the variances of components Ai and Bi, with n indicating the number of components. DEM encourages a uniform distribution of elements within components, avoiding disproportionate impacts from isolated or few dimensions, effectively reducing perturbations from model pruning, and enhancing model stability.

heading: 4 Experiments

heading: 4.1 Experimental Setup
text: We compared DoRA with existing baseline methods to evaluate its performance in natural language understanding (NLU), question answering (QA), and text generation (summarization) tasks. We chose RoBERTa (Liu et al., 2019) and Bart (Lewis et al., 2019) as the foundational models, used respectively for NLU and QA tasks, and for summarization tasks.
RoBERTa is an optimized version of the BERT (Kenton and Toutanova, 2019) architecture, which significantly improves performance on a variety of language understanding tasks through extended training, larger datasets, and finer tuning of parameters. Bart is a Transformer-based (Vaswani et al., 2017) sequence-to-sequence pre-trained model specifically designed for text generation tasks, such as summarization. It effectively handles various generation tasks by combining bidirectional and autoregressive Transformer architectures.
We tested the performance on several standard datasets: using the GLUE (General Language Understanding Evaluation) (Wang et al., 2018) dataset to evaluate NLU tasks, SQuAD (Rajpurkar et al., 2016) (Stanford Question Answering Dataset) for QA, and Xsum (Narayan et al., 2018) for text summarization. GLUE is a set of dataset for training and testing NLU systems, including various tasks such as sentiment analysis and textual entailment. SQuAD is a question answering dataset that
consists of questions generated from Wikipedia articles and their corresponding answers. Xsum provides a testing environment for extreme summarization tasks aimed at generating single-sentence summaries, challenging the models under extreme information compression conditions.
We selected several mainstream fine-tuning methods as baselines, including LoRA, AdaLoRA, Adapter Tuning, BitFit, and full model fine-tuning. LoRA fine-tunes the model weights by adding low-rank matrices to the pre-trained matrices; AdaLoRA is an improvement of LoRA, adding an adaptive adjustment mechanism. Adapter Tuning fine-tunes by inserting lightweight network modules into the PLM. BitFit adjusts only the bias parameters in the PLM. Full model fine-tuning is a traditional method that involves comprehensive adjustment of all model weights.
We report the average results based on 5 random seeds, as shown in Table 2, Table 3, and Table 4, The hyperparameter settings for the experiments can be found in Appendix E.

heading: 4.2 Results
text: We investigate the performance of DoRA and baseline methods across subtasks of the GLUE benchmark, conducting experiments under two different parameter budget scenarios.
As shown in Table 2, DoRA and AdaLoRA, employing adaptive parameter allocation strategies, outperform all baseline methods using uniform parameter distribution, demonstrating the remarkable effectiveness of adaptive parameter allocation. Across the GLUE benchmark, DoRA surpasses LoRA by 0.84% and 0.88%, and AdaLoRA by 0.59% and 0.45% under two parameter budgets, further proving the broad applicability and effectiveness of DoRA’s adaptive parameter allocation strategy in multiple tasks.
Especially noteworthy is DoRA’s performance on the CoLA dataset, where it shows the highest improvement, surpassing the highest performing baseline method by 1.48% and 1.73% under two parameter budgets. This highlights DoRA’s advantage in handling the linguistic acceptability task, showcasing its efficiency in dealing with challenging NLP tasks. However, DoRA’s performance on the MNLI task slightly lags behind AdaLoRA, likely due to MNLI being the largest dataset in GLUE with high task complexity, indicating a need for further optimization of the adaptive parameter allocation strategy when dealing with large-scale
complex tasks. It is worth mentioning that DoRA demonstrates exceptional parameter efficiency, surpassing the performance of full model fine-tuning with only 0.34M parameters, less than 0.3% of full model fine-tuning, highlighting DoRA’s capability in effectively utilizing a limited parameter budget.
Similar results are also observed in the experiments on SQuAD and Xsum, where DoRA outperformes all baseline PEFT methods under both parameter settings.

heading: 5 Analysis and Discussion

heading: 5.1 Effectiveness of DEM
text: To verify the effectiveness of DEM, we tested fine-tuning on datasets including STS-B, CoLA, and SST-2, with and without DEM, without DEM means setting hyper-parameter regularization coefficient η to 0, as shown in Table 5.
Enabling DEM imposes penalties on the variance of LoRA components, encouraging a uniform weight distribution, and avoiding extreme variations in overall update ∆W across a few dimensions due to pruning. Fine-tuning with DEM enabled achieved higher results, demonstrating the effectiveness of DEM.

heading: 5.2 Parameter Allocation Preference
text: To validate whether DoRA can identify key modules in PLMs, we set the final budgets b(T ) to 2, 4, 8, and 16, with 1.5 times the final budget as the initial budget b(0), and conducted fine-tuning experiments on SST-2 dataset respectively.
The results are visually presented in Figure 2, which shows that, in the intermediate layers, the query and key matrices are allocated with more parameter budget, while the value matrices are allocated with fewer budget. The initial output matrices receive more budget. In the feed-forward neural network, the lower projection matrices, represented as Wf2 in the figure, at the backend, especially in the last few layers, are allocated with very few budgets.
DoRA exhibited the same parameter allocation tendencies across all four configurations, demonstrating its ability to consistently identify key modules in PLMs and allocate more parameter budgets to them accordingly.

heading: 5.3 Impact of Initial Budget
text: We investigated the impact of the initial budget b(0) across the MRPC, STS-B, and SST-2 datasets. We fine-tuned models starting from various initial budgets and pruned them to a consistent final budget b(T ) of 2. The results are presented in Table 6. The first row indicates that when the initial budget is 2,
it matches the final budget, which means no model pruning was performed.
Intriguingly, our findings suggest that maintaining a constant final parameter budget while starting with a higher initial parameter budget improves model performance. We attribute this improvement to a more generous initial parameter budget offering a wider exploration space for DoRA, thereby increasing the chance of preserving essential parameters during pruning and optimizing the model’s final performance.

heading: 2 76.32 90.97 94.88

heading: 3 89.74 91.28 94.98

heading: 4 89.92 91.40 95.01

heading: 6 90.14 91.62 95.05

heading: 8 90.17 91.65 95.11

heading: 6 Related Work
text: PEFT is crucial for the fine-tuning of PLMs in practical applications. These techniques primarily focus on updating a select subset of the model’s parameters or introducing new parameters on a small scale, enabling more efficient use of resources. These approaches are particularly valuable in scenarios constrained by computational resources. Existing PEFT methods can generally
be divided into three categories: addition-based methods, specification-based methods, and reparameterization-based methods (Ding et al., 2022).
Addition-based methods achieve adjustment by adding extra modules or trainable parameters to PLMs, such as trainable adapters or soft prompts. These methods are scalable and applicable to models of various sizes, with the performance gap between them and full model fine-tuning narrows as model size increases. Examples include adapters (Houlsby et al., 2019; Pfeiffer et al., 2021; He et al., 2021a; Zhu et al., 2021), which insert small neural modules into transformer layers for adjustment, and prompt-based tuning (Li and Liang, 2021b; Gao et al., 2021; Hu et al., 2022b; Tan et al., 2022; Lester et al., 2021b; Vu et al., 2021), which stimulates PLMs by adding additional context around the original input.
Specification-based methods focus on fine-tuning a few inherent parameters within the model without altering its internal structure (Vucetic et al., 2022; Holmes et al., 2021). By directly specifying which parts of the parameters to optimize, these approaches achieve efficient model adaptation, maintaining performance close to full parameter fine-tuning while reducing the number of parameters adjusted. Examples include BitFit (Ben Zaken et al., 2022), which optimizes only the bias terms in the model, and Diff Pruning (Guo et al., 2021), which introduces sparsity by optimizing a difference vector.
Reparameterization-based methods optimize models by transforming adaptive parameters into more efficient forms, often based on the low-rank hypothesis. These methods (Holmes et al., 2021; Karimi Mahabadi et al., 2021; Edalati et al., 2022; Zhang et al., 2023; Lialin et al., 2023; Ding et al., 2023a; Valipour et al., 2023; Su et al., 2024) aim to reduce computational and memory costs by optimizing low-dimensional proxy parameters while maintaining or surpassing the performance of full parameter fine-tuning. They are grounded in the theory that PLM adaptations to downstream tasks are inherently low-rank. Examples include LoRA (Hu et al., 2022a), which optimizes based on the hypothesis of a low intrinsic rank of weight changes.

heading: 7 Conclusion
text: In this paper, we introduce Dynamic Low-Rank Adaptation (DoRA), a novel method aiming at enhancing the efficiency of fine-tuning PLMs by dynamically adjusting parameter distribution. DoRA innovatively allocates parameter budgets based on their importance to specific tasks, demonstrating considerable improvements in NLP applications. Experimental results indicate that DoRA surpasses baseline methods, highlighting its potential for broader adoption in model optimization efforts.
The innovation of DoRA lies in its adoption of an adaptive parameter allocation strategy, which, unlike traditional uniform distribution, dynamically adjusts the distribution of parameter budgets based on their contribution. Additionally, DoRA employs a component-wise decomposition approach for handling LoRA layers, treating high-rank LoRA layers as a combination of single-rank LoRA components. These components are adjusted through a dynamic rank allocation mechanism, pruned according to their contribution to the overall model performance. To ensure stable pruning throughout the process, DoRA incorporates a regularization penalty term focused on reducing component variance.
Limitation
Our study confirms the effectiveness of DoRA in several NLP tasks. However, its evaluation has been limited to these tasks, and its efficacy in handling more complex NLP challenges, such as machine translation or multimodal tasks, has yet to be established. Moreover, the models used in our experiments are somewhat limited in scale, as we have not conducted experiments with large language models (LLMs). Addressing this limitation, future work could explore DoRA’s potential in these sophisticated areas of NLP.

heading: Acknowledgements
text: This work is supported by the National Natural Science Foundation of China (No.62376019, 61976015, 61976016, 61876198 and 61370130) and the Talent Fund of Beijing Jiaotong University (2024JBRC005). We sincerely thank the reviewers for their insightful comments and suggestions to improve the quality of the paper.

heading: A Potential Risks of Our Method
text: Since our proposed method adapts pre-trained models to specific tasks, it has the potential to extend the range of languages supported by these models. However, there is a risk that some malicious users might exploit this new capability to provide services in politically sensitive languages and tasks. For instance, a malicious user could use our method to generate hateful or offensive sentences in some politically sensitive languages.

heading: B Dataset Detail
text: GLUE (Wang et al., 2018) benchmark is a collection of diverse natural language understanding tasks designed to evaluate and analyze the performance of models across a wide range of linguistic challenges. The benchmark encompasses a variety of tasks, including linguistic acceptability (CoLA (Warstadt et al., 2019)), sentiment analysis (SST-2 (Socher et al., 2013)), paraphrase detection (MRPC (Dolan and Brockett, 2005), QQP (Wang et al., 2018)), semantic textual similarity (STSB (Wang et al., 2018)), and natural language inference (MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), RTE (Bentivogli et al., 2009)). The dataset statistics are presented in Table 7.
SQuAD (Rajpurkar et al., 2016) is an extensive reading comprehension dataset aimed at evaluating the ability of models to understand and answer questions based on Wikipedia article contents. The dataset features two major versions: SQuAD 1.1 and SQuAD 2.0. SQuAD 1.1 consists of over 100,000 question-answer pairs on 500+ articles, where questions are posed by crowd workers on a given passage and the answers are segments of text from the passage. SQuAD 2.0 extends the SQuAD 1.1 dataset with over 50,000 additional unanswerable questions that are written adversarially by crowd workers to look similar to answerable ones but do not have answers in the text. This
makes SQuAD 2.0 more challenging and helps models better emulate human reading comprehension abilities by not only retrieving answers but also determining when no valid answer exists within the text.
Xsum (Narayan et al., 2018) is designed specifically for the task of single-sentence news summarization to create a concise abstract of a news article. Xsum consists of approximately 227,000 articles collected from the British Broadcasting Corporation (BBC). Each article comes with a professionally written, single-sentence summary, making it particularly challenging for models due to the extreme brevity and the need for models to abstract rather than simply extract content. Unlike other summarization datasets, which often focus on extracting salient sentences directly from the text, Xsum requires models to generate informative, concise, and grammatically coherent summaries that capture the core essence of the article content, often requiring synthesis and rephrasing skills beyond mere extraction.

heading: C Baseline Detail
text: LoRA (Hu et al., 2022a): LoRA achieves finetuning by integrating low-rank matrices within the weight matrices of the PLM. It maintains efficiency by adjusting a reduced set of parameters, which is particularly advantageous for scaling to large models.
AdaLoRA (Zhang et al., 2023): Building upon LoRA, AdaLoRA optimizes parameter utilization by adaptively adjusting the parameter budget throughout the training process. This adaptive strategy improves fine-tuning efficiency and has demonstrated enhanced performance in a variety of NLP tasks.
Adapter Tuning (Houlsby et al., 2019; Rebuffi et al., 2017; Pfeiffer et al., 2021): Incorporates methods such as the Houlsby Adapter (H-Adapter) and Pfeiffer Adapter (P-Adapter). Adapter Tuning fine-tunes models by inserting small, trainable adapter modules between existing layers of the PLM. This approach does not modify the original weights in the PLM, offering a flexible yet conservative way to adapt the model to new tasks.
BitFit (Ben Zaken et al., 2022): An minimalistic PEFT method that focuses solely on adjusting the bias terms within the model. BitFit has been shown to achieve performance levels comparable to those of full model fine-tuning under specific conditions.
Full Model Fine-tuning: This conventional method updates all the parameters of a PLM to tailor it to specific downstream tasks. Full model fine-tuning demands remarkable computational resources.

heading: D Training Setup
text: Our experiments were conducted using the PyTorch (Paszke et al., 2019) framework, with models and datasets sourced from the Huggingface (Wolf et al., 2020) platform. The computations were performed on NVIDIA GeForce RTX 3090 GPUs.
We refer to LoRA’s initialization method, applying Kaiming initialization (He et al., 2015) to Ai and Bi, and initializing ci with 0. This ensures that the initial update amount ∆Wi of each component is 0, preserving the behavior of the original model at the initial stage of training.

heading: E Hyper Parameter
text: We fix the following hyperparameters across all experiments:
• Initial budget b(0): 1.5 times the final budget b(T )
• Smoothing factor β: 0.9
• Initial steps ti: 15% of the total steps
• Final steps tf : 50% of the total steps
We fixed sentence length and searched remain hyperparameters, including the learning rate, batch size, number of training epochs, regularization coefficient, and pruning step interval, as shown in Table 8

heading: F Computing Efficiency
text: We evaluate the computational efficiency of DoRA compared to baseline methods based on the training time per epoch on QNLI dataset and GPU memory usage. The results are shown in Table 9. DoRA’s computational efficiency is comparable to AdaLoRA, slightly lower than LoRA, and significantly higher than full model fine-tuning.
Corpus length learning rate batch size epochs regularization coefficient pruning step interval

heading: Future Work
text: Addressing this limitation, future work could explore DoRA’s potential in these sophisticated areas of NLP