BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models

## Abstract

Abstract Large language models (LLMs) have demonstrated remarkable proficiency across various natural language processing (NLP) tasks. However, adapting LLMs to downstream applications requires computationally intensive and memory-demanding fine-tuning procedures. To alleviate these burdens, parameter-efficient fine-tuning (PEFT) techniques have emerged as a promising approach to tailor LLMs with minimal computational overhead. While PEFT methods offer substantial advantages, they do not fully address the pervasive issue of bias propagation from pre-training data. This work introduces Bias-Alleviating Low-Rank Adaptation (BA-LoRA), a novel PEFT method designed to counteract bias inheritance. BA-LoRA incorporates three distinct regularization terms: (1) a consistency regularizer, (2) a diversity regularizer, and (3) a singular value decomposition regularizer. These regularizers aim to enhance the models’ consistency, diversity, and generalization capabilities during fine-tuning. We conduct extensive experiments on natural language understanding (NLU) and natural language generation (NLG) tasks using prominent LLMs such as LLaMA, Mistral, and Gemma. The results demonstrate that BA-LoRA outperforms LoRA and its state-of-the-art variants. Moreover, our method effectively mitigates the adverse effects of pre-training bias, leading to more reliable and robust model outputs.
The code is available at https://github.com/cyp-jlu-ai/BA-LoRA .

## 1 Introduction

The emergence of large language models (LLMs) has marked a new era in natural language processing (NLP). Models such as GPT-4 (OpenAI, ), Llama (Touvron et al., ), Mistral (Jiang et al., ), and Gemma (Team et al., ) have demonstrated exceptional performance across a wide array of NLP tasks, including language comprehension, generation, and reasoning (Zhao et al., ; Chang et al., ). The remarkable advancements of LLMs can be largely attributed to their training on vast datasets (Zhao et al., ). As LLMs continue to evolve rapidly, training on extensively scaled web-derived corpora has become standard practice to improve model generalization, thus bypassing the labor-intensive processes of data curation and annotation (Gao et al., ; Penedo et al., ). However, the corresponding increase in data volume has introduced several challenges, such as the presence of imbalanced, duplicated, and corrupted information (Parashar et al., ; Liu & He, ; Chen et al., ; Yang et al., ).

Recent research has shown that various forms of bias in training data can negatively affect LLM behavior (Dong et al., ; Dodge et al., ; Longpre et al., ; Chen et al., ). For example, noise within the training data can degrade model generalization (Chen et al., ), while the long-tailed distribution of concepts in web-scale data can cause LLMs to overemphasize overrepresented topics (Zhu et al., ). Furthermore, biases introduced during pre-training can persist even after fine-tuning, potentially compromising model performance and safety in real-world applications (Qi et al., ; Bommasani et al., ; Mallen et al., ; Carlini et al., ).

This phenomenon, termed "Catastrophic Inheritance" by (Chen et al., ), has spurred investigations into mitigation strategies. While constructing less biased datasets and developing more robust model architectures are prominent approaches (Liu & He, ), this study explores an alternative: innovations in fine-tuning. Fine-tuning LLMs is a powerful method for enhancing task-specific performance (Han et al., ), aligning models with user intent (Ouyang et al., ; Xu et al., ), and eliciting desired behaviors (Bai et al., ; Rafailov et al., ). However, fine-tuning large-scale models’ computational and memory demands are substantial (Hu et al., ). For instance, 16-bit fine-tuning of a Llama-65B model requires over 780 GB of GPU memory (Dettmers et al., ). To address these limitations, parameter-efficient fine-tuning (PEFT) techniques, such as Low-Rank Adaptation (LoRA) (Hu et al., ), have gained prominence.

LoRA posits that parameter updates during fine-tuning can be efficiently represented by low-rank matrices. Therefore, given a pre-trained weight matrix $W\in\mathbb{R}^{m\times n}$, instead of updating all parameters of $W$ directly, LoRA introduces an auxiliary low-rank adapter $\Delta W=AB$, where $A\in\mathbb{R}^{m\times r}$ and $B\in\mathbb{R}^{r\times n}$ with rank $r\ll\min(m,n)$. Here, $A$ and $B$ are learnable matrices initialized as:

- •
$A$ is initialized with a scaled normal distribution: $a_{ij}\sim\mathcal{N}(0,\sigma^{2})$.
- •
$B$ is initialized to zero: $b_{ij}=0$.

For a given input $X$, the output $Y$ is computed as:

$$ $\displaystyle Y$ $\displaystyle=X(W+\Delta W)=X(W+AB).$ (1) $$

Only $A$ and $B$ are updated during fine-tuning while $W$ remains frozen. This initialization ensures that $AB=0$ at the start of training, thus preserving the model’s original output. Since the rank $r$ is significantly smaller than the dimensions of $W$, LoRA substantially reduces training overhead compared to full fine-tuning (Hu et al., ).

To mitigate the detrimental effects of Catastrophic Inheritance, particularly noise and imbalance, we propose Bias-Alleviating Low-Rank Adaptation (BA-LoRA). Building upon Principal Singular Values and Singular Vectors Adaptation (PiSSA) (Meng et al., ), which addresses convergence issues in standard LoRA, our approach incorporates three distinct regularization terms: a consistency regularizer, a diversity regularizer, and a singular value decomposition (SVD) regularizer. The consistency regularizer preserves valuable pre-trained knowledge during fine-tuning, while the diversity regularizer encourages varied model outputs. The SVD regularizer enhances the generalization capabilities of generative models. Recognizing the fundamental differences between Natural Language Understanding (NLU) and Natural Language Generation (NLG), such as determinism in NLU versus diversity in NLG, we tailor our regularization strategies accordingly.

To evaluate the efficacy of BA-LoRA, we conduct comprehensive experiments across diverse benchmarks, including mathematical reasoning (GSM8K (Cobbe et al., ) and MATH (Yu et al., )), coding (HumanEval (Chen et al., ) and MBPP (Austin et al., )), natural language understanding (GLUE (Wang et al., )), and general language evaluation (MT-Bench (Zheng et al., )). Our experiments utilize prominent LLMs such as LLaMA 2-7B (Touvron et al., ), Mistral-7B (Jiang et al., ), and Gemma-7B (Team et al., ), as well as encoder-only architectures like RoBERTa-large (Liu et al., ) and DeBERTa-v3-base (He et al., ). The results unequivocally demonstrate BA-LoRA’s superiority over LoRA and PiSSA. Moreover, our method effectively attenuates noise inherited from pre-training, leading to more robust and generalizable models.

## 2 Related Works

Parameter-efficient fine-tuning (PEFT) techniques (Xu et al., ; Han et al., ) have garnered significant attention as an approach to adapting LLMs for specific tasks under limited hardware resources. Three main categories of PEFT techniques are commonly used. The first category includes adapter-based methods (Houlsby et al., ; Lin et al., ; Lei et al., ; He et al., ), which introduce additional layers into the model and fine-tune these layers (typically with far fewer parameters) to reduce computational costs. The second category comprises soft prompt tuning methods (Hambardzumyan et al., ; Lester et al., ; Li & Liang, ; Liu et al., ), which prepend learnable soft prompts to the model’s input to tailor it to specific tasks. These methods leverage the inherent capabilities of pre-trained models, requiring only the appropriate prompts to adapt to downstream tasks. The third category encompasses low-rank adaptation (LoRA) and its variants (Hu et al., ; Zhang et al., ; Dettmers et al., ). LoRA introduces the product of low-rank matrices within existing layers to approximate weight updates during fine-tuning (Hu et al., ).

Variants of LoRA enhance its efficiency and performance in different ways. AdaLoRA adaptively distributes the parameter budget among weight matrices based on their importance, improving efficiency and performance by pruning unimportant updates and minimizing computational overhead (Zhang et al., ). DoRA increases LoRA’s learning capacity and stability by decomposing pre-trained weights into magnitude and direction components for fine-tuning (Liu et al., ). LoHA enhances LoRA by employing Hamiltonian products (Hyeon-Woo et al., ). DyLoRA addresses the fixed size and rank optimization limitations of LoRA by dynamically training LoRA blocks across varying ranks (Valipour et al., ). DeltaLoRA improves the representational capacity of LoRA by updating the model’s original weights using parameters from adapter layers (Zi et al., ). PiSSA initializes adapter matrices $A$ and $B$ to approximate the original matrix $W$ through singular value decomposition, leading to faster convergence and improved performance (Meng et al., ).
While many LoRA variants focus on accelerating convergence or reducing memory consumption, our BA-LoRA method uniquely addresses the core challenge of Catastrophic Inheritance in LLM fine-tuning.

## 3 Method

### 3.1 Principal Singular Values and Singular Vectors Adaptation (PiSSA)

As a variant of LoRA, PiSSA addresses the convergence speed challenge by retaining the core LoRA architecture while innovating in initialization. Specifically, PiSSA leverages the principal components of the original weight matrix, $W$, to initialize the adapter matrices, $A$ and $B$. The remaining components are encapsulated within a residual matrix, $W^{res}\in\mathbb{R}^{m\times n}$. The SVD of $W\in\mathbb{R}^{m\times n}$ is expressed as $W=USV^{T}$, where $U\in\mathbb{R}^{m\times\min(m,n)}$ and $V\in\mathbb{R}^{n\times\min(m,n)}$ are orthogonal singular vectors, and $S=diag(s)\in\mathbb{R}^{\min(m,n)\times\min(m,n)}$ is a diagonal matrix, where the operation $diag(s)$ transforms $s$ to $S$ and $s\in\mathbb{R}^{\min(m,n)}_{\leq 0}$ represents the singular values arranged in descending order. PiSSA partitions the singular values and vectors into principal and residual components, denoted as $\{U_{[:,:r]},S_{[:r,:r]},V_{[:,:r]}\}$ and $\{U_{[:,r:]},S_{[r:,r:]},V_{[:,r:]}\}$, respectively, where the matrix slicing notations are the same as those in PyTorch, $[:r]$ denotes the first $r$ dimensions, and $r$ signifies the intrinsic rank of $W$. The principal components are then employed to initialize the low-rank adapter with $A\in\mathbb{R}^{m\times r}$ and $B\in\mathbb{R}^{r\times n}$:

$$ $\displaystyle A$ $\displaystyle=U_{[:,:r]}\,S_{[:r,:r]}^{1/2}\in\mathbb{R}^{m\times r},$ (2) $\displaystyle B$ $\displaystyle=S_{[:r,:r]}^{1/2}\,V_{[:,:r]}^{T}\in\mathbb{R}^{r\times n}.$ (3) $$

The residual matrix $W^{res}$ remains frozen during fine-tuning:

$$ $W^{res}=U_{[:,r:]}\,S_{[r:,r:]}\,V_{[:,r:]}^{T}\in\mathbb{R}^{m\times n}.$ (4) $$

PiSSA preserves the pre-trained model’s full capacity at the start of fine-tuning by using $W=W^{res}+AB$. This approach prioritizes training the most influential parameters, thereby accelerating convergence. Inheriting LoRA’s benefits of reduced parameter count and deployment simplicity, PiSSA further leverages efficient SVD computations to expedite the training process.

### 3.2 Bias-Alleviating Low-Rank Adaptation (BA-LoRA)

Catastrophic Inheritance encapsulates the challenges posed by biased large-scale training data, which can manifest in LLMs as vulnerabilities and limitations arising from duplicated, noisy, imbalanced, or unethical samples. These inherited flaws can adversely impact downstream tasks, leading to diminished generalization, degraded performance, security breaches, and biased outputs. To address the specific issues caused by noisy and imbalanced data, we introduce BA-LoRA, a method incorporating three distinct regularization terms: (1) consistency regularizer, (2) diversity regularizer, and (3) SVD regularizer. Recognizing the nuanced differences between NLU and NLG, we have tailored specific variants of each regularizer to optimize performance for respective task domains.

#### 3.2.1 Regularizations for NLU Tasks

Consistency Regularization.
To safeguard valuable pre-trained knowledge during the fine-tuning process, we introduce a regularization term based on the mean squared error (MSE) loss between normalized output logits produced by the pre-trained model, $\mathbf{F}_{P}$, and those generated by the fine-tuned model, $\mathbf{F}_{F}$. This loss function incentivizes the fine-tuned model to retain essential pre-trained information while adapting to downstream task requirements.

$$ $\mathcal{L}_{\mathrm{CR_NLU}}=\left\|\frac{\mathbf{F}_{p}}{\|\mathbf{F}_{p}\| _{2}}-\frac{\mathbf{F}_{f}}{\|\mathbf{F}_{f}\|_{2}}\right\|_{2}^{2}$ (5) $$

This objective facilitates the inheritance of critical pre-trained knowledge in $\mathbf{F}_{f}$ after fine-tuning.

Diversity Regularization.
To address the detrimental effects of imbalanced data, we introduce a diversity regularizer aimed at eliciting more diverse representational structures within LLMs and preventing the encoding of semantically similar samples during fine-tuning. Inspired by (Bardes et al., ), we employ a covariance loss to minimize the off-diagonal elements of the covariance matrix of the fine-tuned outputs $\mathbf{F}_{f}$:

$$ $\mathcal{L}_{\mathrm{DR_NLU}}=\frac{1}{D}\sum_{i\neq j}[C(\mathbf{F}_{f})]_{i ,j}^{2}$ (6) $$

where $D$ represents the dimensionality of $\mathbf{F}_{f}$ and $C(\mathbf{F}_{f})$ is the covariance matrix of $\mathbf{F}_{f}$, which is defined as:

$$ $C(\mathbf{F}_{f})=\frac{1}{M-1}\sum_{i=1}^{M}\left(f_{i}-\bar{f}\right)\left(f _{i}-\bar{f}\right)^{T}$ (7) $$

where $M$ denotes the number of elements involved in $\mathbf{F}_{f}$, $f_{i}$ is the $i$-th element in $\mathbf{F}_{f}$, and $\bar{f}$ is the mean value of $\mathbf{F}_{f}$.

Singular Value Decomposition Regularization.
The SVD regularizer is designed to enhance model generalizability to mitigate the adverse effects of noisy data. Building upon the insight from (Chen et al., ) that eigenvectors corresponding to the largest singular values significantly contribute to model generalizability, we propose an SVD regularizer that maximizes the sum of the top $k$ singular values of a batched fine-tuned output matrix:

$$ $\mathcal{L}_{\mathrm{SVDR_NLU}}=-\frac{\sum_{i=1}^{k}\sigma_{i}}{\sum_{j=1}^{ D}\sigma_{j}}$ (8) $$

where $k$ is a hyperparameter, $\sigma_{i}$ denotes the $i$-th singular value of the top $k$ singular values of the output matrix, and $\sum_{j=1}^{D}\sigma_{j}$ is the sum of all singular values obtained from the SVD of the output matrix. This decomposition represents the matrix as $\mathbf{U}\bm{\Sigma}\mathbf{V}^{\top}$, where $\bm{\Sigma}$ is a diagonal matrix containing singular values $\{\sigma_{1},\ldots,\sigma_{D}\}$. This regularization term emphasizes significant components of the logit matrix, enhancing the model’s generalizability across various downstream tasks.

#### 3.2.2 Overall Objective Function for NLU

The overall objective function for NLU tasks is formulated as follows:

$$ $\mathcal{L}_{\mathrm{NLU}}=\mathcal{L}_{\mathrm{task_NLU}}+\lambda_{1} \mathcal{L}_{\mathrm{CR_NLU}}+\lambda_{2}\mathcal{L}_{\mathrm{DR_NLU}}+ \lambda_{3}\mathcal{L}_{\mathrm{SVDR_NLU}}$ (9) $$

where $\mathcal{L}_{\mathrm{task_NLU}}$ represents the standard cross-entropy loss function for the downstream task, and $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are weighting parameters to balance each regularization term.

#### 3.2.3 Regularizations for NLG Tasks

Consistency Regularization.
To ensure that the fine-tuned model retains knowledge from pre-training, we utilize the Kullback-Leibler Divergence (KLD) to measure the divergence between the output distributions of the fine-tuned and pre-trained models (Dong et al., ). Specifically, we define the consistency regularization loss as:

$$ $\mathcal{L}_{\mathrm{CR_NLG}}=\frac{1}{T}\sum_{t=1}^{T}\mathrm{KL}\left( \mathcal{P}_{\mathrm{pt}}(y_{t}\mid y_{<t},x)\parallel\mathcal{P}_{\mathrm{ft} }(y_{t}\mid y_{<t},x)\right),$ (10) $$

where $\mathcal{P}_{\mathrm{pt}}(y_{t}\mid y_{<t},x)$ and $\mathcal{P}_{\mathrm{ft}}(y_{t}\mid y_{<t},x)$ represent the conditional probability distributions of the pre-trained model and the fine-tuned model, respectively. For the current token $y_{t}$, given the input $x$ and the preceding token sequence $y_{<t}$.
KLD encourages the model to continuously retain useful pre-training information during the fine-tuning process, which is crucial for maintaining the style and coherence of the generation task.

Diversity Regularization.
To enhance the diversity of the generated text, we introduce an entropy-based regularization term, inspired by previous work (Gat et al., ). This regularization term aims to increase the entropy of the predicted token distributions during fine-tuning, thus encouraging more varied and diverse outputs.

$$ $\mathcal{L}_{\mathrm{DR_NLG}}=-\frac{1}{T}\sum_{t=1}^{T}\sum_{i=1}^{N}P_{ \mathrm{ft}}(x_{i}|h_{t})\log P_{\mathrm{ft}}(x_{i}|h_{t})$ (11) $$

where, $P_{\mathrm{ft}}(x_{i}|h_{t})$ represents the probability assigned to token $x_{i}$ at time step $t$, given the model’s hidden state $h_{t}$. Maximizing entropy at each time step minimizes repetitive outputs and encourages more diverse and enriched text generation.

Singular Value Decomposition Regularization
To enhance the generalization capability of generative models, we introduce a regularization technique that accentuates the most significant singular values, enabling the model to capture the principal components of the data and prioritize the most informative patterns.

$$ $\mathcal{L}_{\mathrm{SVDR_NLG}}=-\frac{\sum_{i=1}^{k}\sigma_{i}}{\sum_{j=1}^{ D}\sigma_{j}}$ (12) $$

where $\sigma_{i}$ denotes the $i$-th largest singular value, $\sigma_{j}$ represents the $j$-th singular value, and $D$ is the total number of singular values. This regularization term aims to maximize the relative contribution of the top $k$ singular values, encouraging the model to focus on the most critical aspects of the data. By integrating this regularization, the model is steered towards generating higher-quality and more diverse outputs, ultimately improving its performance and robustness.

#### 3.2.4 Overall Objective Function for NLG

The objective function for downstream NLG tasks is formulated as follows:

$$ $\mathcal{L}_{\mathrm{NLG}}=\mathcal{L}_{\mathrm{task_NLG}}+\lambda_{1} \mathcal{L}_{\mathrm{CR_NLG}}+\lambda_{2}\mathcal{L}_{\mathrm{DR_NLG}}+ \lambda_{3}\mathcal{L}_{\mathrm{SVDR_NLG}}$ (13) $$

where $\mathcal{L}_{\mathrm{task_NLG}}$ denotes the standard loss for the downstream generative task, and $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$ are weighting parameters to balance each regularization term.

## 4 Experiments

This section presents a comprehensive evaluation of our proposed BA-LoRA method across a diverse range of natural language generation (NLG) and natural language understanding (NLU) benchmarks. Our results unequivocally demonstrate the superiority of BA-LoRA over existing LoRA variants. Furthermore, through rigorous experimentation, we elucidate BA-LoRA’s efficacy in mitigating the adverse impacts of noisy data, thereby enhancing model robustness and generalizability.

### 4.1 Models and Datasets

To evaluate the effectiveness of our approach, we conduct experiments using several prominent language models and assess their performance on a diverse array of datasets, covering both Natural Language Generation and Natural Language Understanding tasks. Specifically, for language generation models, we include LLaMA 2-7B (Touvron et al., ), LLaMA 3-8B (AI@Meta, ), Mistral-7B (Jiang et al., ), Gemma-7B (Team et al., ), and GPT-2-XL (Radford et al., ). For language understanding models, we use BERT-Large (BERT-L) (Devlin et al., ), RoBERTa-Large (Liu et al., ), and DeBERTa-v3-base (He et al., ). This selection ensures coverage across various architectures and parameter scales, facilitating a comprehensive evaluation.

For the datasets, we employ a wide range of tasks in both Natural Language Generation (GSM8K (Cobbe et al., ), MATH (Yu et al., ), HumanEval (Chen et al., ), MBPP (Austin et al., ), MT-Bench (Zheng et al., )) and Natural Language Understanding. In the latter, we assess in-domain (ID) performance using the GLUE benchmark (Wang et al., ) and out-of-domain (OOD) generalization using the GLUE-X benchmark (Yang et al., ). These datasets span a broad range of challenges, allowing for a thorough examination of our method’s generalization capabilities.

### 4.2 Baselines

We compare BA-LoRA with several baselines to demonstrate its effectiveness:

Full Fine-Tuning (Full FT): Fine-tuning the model with all parameters, which requires the most resources.

LoRA (Hu et al., ): Fine-tuning the model by inserting a low rank adapter $AB$ into linear layers.

PiSSA (Meng et al., ): Performing SVD on the weight matrix $W$ at the beginning of training and initializing $A$ and $B$ based on the components with singular values.

### 4.3 Implementation Details

In our experiments, we adopt the PiSSA (Meng et al., ) implementation strategy. We compute the loss using only the responses from the instruction-following dataset, ensuring lora_dropout to 0. We utilize the Float32 computation type for both the base model and the adapter in BA-LoRA.
For the NLU tasks, we set the hyperparameters as: $\lambda_{1}=1e-4$, $\lambda_{2}=4e-4$, and $\lambda_{3}=1e-4$. We set lora_r $=$ lora_alpha $=128$ and use AdamW (Loshchilov & Hutter, ) optimizer with a batch size of 128, a learning rate of $2e-5$, cosine annealing schedules, and a warmup ratio of 0.03, without any weight decay.
For the NLG tasks, the hyperparameters are set as: $\lambda_{1}=1e-4$, $\lambda_{2}=3e-4$, and $\lambda_{3}=1e-4$. We set lora_r as 8 and select lora_alpha in 8, 16. We utilize AdamW with a linear learning rate schedule to optimize and tune the learning rate (LR) from $1e-4$, $2e-4$, $3e-4$, $4e-4$, $5e-4$, $6e-4$, $5e-5$, $3e-5$. Batch sizes (BS) are selected from 6, 8, 16, 32.
Appendix Section B presents the detailed hyperparameters we utilized on the GLUE benchmark.
All experiments were conducted using NVIDIA A40 (48G) GPUs.
All presented results are derived from three independent experiments, ensuring the consistency and robustness of our findings. Each experiment was conducted under identical conditions, and the results were averaged to mitigate any variability.

### 4.4 Results and Analysis

**Table 1: Performance Comparison of Various Models and Methods on NLG Tasks. The best outcome is highlighted in bold.**
| Models | Methods | Parameters | GSM8K | MATH | HumanEval | MBPP | MT-Bench | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-2-7B | Full FT | 6738M | 49.05 | 7.22 | 21.34 | 35.59 | 4.91 | 23.62 |
| LoRA | 320M | 42.47 | 5.60 | 17.03 | 31.48 | 4.62 | 20.24 |  |
| PiSSA | 320M | 52.01 | 7.76 | 21.55 | 33.09 | 4.87 | 23.86 |  |
| BA-LoRA | 320M | 53.83 | 9.13 | 23.58 | 36.86 | 5.11 | 25.70 |  |
| Mistral-7B | Full FT | 6738M | 67.02 | 18.60 | 45.12 | 51.38 | 4.95 | 37.41 |
| LoRA | 168M | 67.68 | 19.90 | 42.54 | 55.74 | 4.92 | 38.16 |  |
| PiSSA | 168M | 71.90 | 21.72 | 45.20 | 60.83 | 5.23 | 40.98 |  |
| BA-LoRA | 168M | 73.04 | 22.11 | 46.31 | 62.29 | 5.41 | 41.83 |  |
| Gemma-7B | Full FT | 6738M | 71.34 | 22.74 | 46.95 | 55.64 | 5.40 | 40.41 |
| LoRA | 200M | 74.64 | 31.16 | 51.64 | 62.84 | 5.01 | 45.06 |  |
| PiSSA | 200M | 77.58 | 31.47 | 53.15 | 65.49 | 5.66 | 46.67 |  |
| BA-LoRA | 200M | 78.13 | 32.25 | 54.41 | 66.12 | 5.73 | 47.33 |  |

**Table 2: Performance Comparison of Various Models and Methods on NLU Tasks. The best outcome is highlighted in bold.**
| Models | Methods | Parameters | MNLI | SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RoBERTa-large | Full FT | 355M | 90.20 | 96.40 | 90.90 | 68.00 | 94.70 | 92.20 | 86.60 | 91.50 | 88.81 |
| LoRA | 1.84M | 90.51 | 96.08 | 90.53 | 68.14 | 94.78 | 91.47 | 86.30 | 92.75 | 88.82 |  |
| PiSSA | 1.84M | 90.56 | 96.65 | 91.76 | 69.08 | 95.18 | 91.55 | 91.07 | 92.84 | 89.84 |  |
| BA-LoRA | 1.84M | 91.24 | 96.90 | 92.10 | 70.26 | 95.81 | 92.21 | 91.70 | 93.24 | 90.43 |  |
| DeBERTa-v3-base | Full FT | 184M | 89.90 | 95.61 | 89.50 | 69.23 | 94.09 | 92.44 | 83.85 | 91.71 | 88.29 |
| LoRA | 1.33M | 90.71 | 94.79 | 89.85 | 70.05 | 93.94 | 92.07 | 85.43 | 91.67 | 88.56 |  |
| PiSSA | 1.33M | 90.47 | 95.81 | 91.48 | 72.27 | 94.41 | 92.21 | 87.14 | 91.93 | 89.47 |  |
| BA-LoRA | 1.33M | 90.92 | 96.25 | 91.83 | 72.79 | 94.84 | 92.59 | 87.87 | 92.15 | 89.91 |  |

#### 4.4.1 Analysis of the NLG and NLU Performance of BA-LoRA

To evaluate BA-LoRA’s effectiveness on NLG tasks, we fine-tuned LLaMA-2-7B, Mistral-7B, and Gemma-7B on the MetaMathQA dataset (Yu et al., ) and assessed their mathematical problem-solving capabilities using the GSM8K (Cobbe et al., ) and MATH (Yu et al., ) validation sets, reporting accuracy. Similarly, models were fine-tuned on the CodeFeedback dataset (Zheng et al., ) and evaluated for coding proficiency via HumanEval (Chen et al., ) and MBPP (Austin et al., ), with PASS@1 metrics reported. To assess conversational abilities, models were trained on the WizardLM-Evol-Instruct dataset (Xu et al., ) and evaluated on MT-Bench (Zheng et al., ), with response quality judged by GPT-4 and first turn scores reported. All experiments utilized 100K data points and a single training epoch for efficiency.

Table 1 presents the experimental outcomes, clearly demonstrating BA-LoRA’s superior performance compared to baseline methods. For instance, BA-LoRA enhanced LLaMA 2-7B, Mistral-7B, and Gemma-7B performance on GSM8K by 1.82%, 1.14%, and 0.55%, respectively, compared to PiSSA. HumanEval improvements were 2.03%, 1.11%, and 1.26%, while MT-Bench enhancements reached 0.24%, 0.18%, and 0.07%. Notably, BA-LoRA achieved a remarkable 6.92% performance uplift over full parameter fine-tuning on Gemma, utilizing only 2.3% of trainable parameters across five tasks.

To evaluate BA-LoRA’s efficacy on NLU tasks, we assessed its performance on the GLUE benchmark (Wang et al., ), encompassing two single-sentence classification tasks (CoLA, SST), five pairwise text classification tasks (MNLI, RTE, QQP, MRPC, QNLI), and one text similarity prediction task (STS-B). Evaluation metrics included overall matched and mismatched accuracy for MNLI, Matthew’s correlation for CoLA, Pearson correlation for STS-B, and accuracy for remaining tasks. Two language models, RoBERTa-large (Liu et al., ) and DeBERTa-v3-base (He et al., ), were employed. Following (Meng et al., ), BA-LoRA was applied to $W_{Q}$ and $W_{A}$ for RoBERTa-large, which has $355M$ trainable parameters, resulting in $1.84M$ trainable parameters, while DeBERTa-v3-base, with its $184M$ trainable parameters, had BA-LoRA applied to $W_{Q}$, $W_{A}$, and $W_{V}$, totaling $1.33M$ trainable parameters, ensuring a relatively fair comparison. Table 2 presents results across eight tasks for both models, demonstrating BA-LoRA’s consistent superiority over baselines. On average, BA-LoRA outperformed PiSSA and LoRA by 0.59% and 1.61% for RoBERTa-large, and 0.44% and 1.35% for DeBERTa-v3-base, respectively. These results underscore BA-LoRA’s effectiveness in enhancing NLU model performance.

A comparative analysis of Tables 1 and 2 reveals BA-LoRA’s consistent performance advantages across both NLG and NLU tasks. This indicates BA-LoRA’s proficiency in augmenting both generative and comprehension capabilities for language models. By incorporating consistency, diversity, and SVD regularization, BA-LoRA effectively mitigates the adverse effects of Catastrophic Inheritance, fostering consistent, diverse, and generalized model outputs. Furthermore, BA-LoRA’s modest computational requirements render it suitable for efficient fine-tuning of LLMs with limited resources.

**Table 3: ID Performance Comparison of BERT-L and GPT-2-XL Using LoRA and BA-LoRA Methods on GLUE Benchmark. The best outcome is highlighted in bold.**
| Model | Methods | MNLI | SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BERT-L | LoRA | 87.24 | 93.19 | 90.10 | 64.73 | 93.13 | 90.94 | 73.14 | 90.63 | 85.39 |
| BA-LoRA | 89.72 | 94.85 | 92.23 | 65.49 | 95.48 | 91.72 | 75.77 | 91.71 | 87.12 |  |
| GPT-2-XL | LoRA | 85.28 | 95.38 | 86.17 | 50.63 | 89.42 | 88.56 | 72.29 | 89.27 | 82.13 |
| BA-LoRA | 88.14 | 96.52 | 89.23 | 52.76 | 91.26 | 89.95 | 74.57 | 90.83 | 84.16 |  |

**Table 4: OOD Performance Comparison of BERT-L and GPT-2-XL Using LoRA and BA-LoRA Methods on GLUE-x Benchmark. The best outcome is highlighted in bold.**
| Model | Methods | MNLI | SST-2 | MRPC | CoLA | QNLI | QQP | RTE | STS-B | Avg |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| BERT-L | LoRA | 85.19 | 93.49 | 89.93 | 63.49 | 92.32 | 87.73 | 73.65 | 90.57 | 84.55 |
| BA-LoRA | 87.91 | 94.18 | 90.62 | 65.81 | 93.04 | 89.06 | 75.41 | 91.21 | 85.91 |  |
| GPT-2-XL | LoRA | 87.02 | 95.11 | 86.81 | 60.95 | 91.77 | 87.59 | 78.76 | 89.25 | 84.66 |
| BA-LoRA | 89.58 | 96.40 | 88.18 | 63.11 | 92.68 | 88.62 | 81.21 | 90.37 | 86.27 |  |

#### 4.4.2 Analysis on mitigate noisy data

This study aims to evaluate BA-LoRA’s efficacy in mitigating the detrimental effects of noise inherent in large-scale pre-training data on downstream tasks. Given the ubiquitous presence of noise in human-annotated datasets, its influence on pre-training is unavoidable. To comprehensively assess the impact of noisy pre-training data, we employ both ID and OOD evaluation using the GLUE and GLUE-x benchmarks, respectively. BERT-L (Devlin et al., ), pre-trained on BooksCorpus (Zhu et al., ) and English Wikipedia, and GPT-2-XL (Radford et al., ), pre-trained on the noisy WebText dataset derived from Common Crawl, serve as our models.

As detailed in Tables 3 and 4, BA-LoRA consistently outperforms LoRA across all tasks, underscoring its superior generalization capabilities. Specifically, BA-LoRA achieves average performance improvements of 2.03% and 2.47% for BERT-L and GPT-2-XL, respectively, on the GLUE benchmark. Similarly, on GLUE-x, BA-LoRA surpasses LoRA by 1.61% and 1.90% for BERT-L and GPT-2-XL, respectively. These results substantiate the effectiveness of our proposed regularization terms in mitigating the negative impacts of noise in pre-training and enhancing model robustness.

Figure: (a) BERT-L, LoRA
Refer to caption: x1.png

#### 4.4.3 Analysis on Mitigating Imbalanced Data

This experiment evaluates the effectiveness of BA-LoRA in addressing imbalanced data. Specifically, using the MNLI dataset, LoRA and BA-LoRA are applied to fine-tune the BERT-L and GPT-2-XL models, respectively. The hidden layer features of the last training step are extracted and visualized using t-SNE (Van der Maaten & Hinton, ) technology for comparison.

As shown in Figure 1, the models fine-tuned with standard LoRA in sub-figures (a) and (c) have low discrimination between categories and obvious category mixing. In contrast, the models fine-tuned with BA-LoRA in sub-figures (b) and (d) have clearer category separation, especially the results of BERT-L, which have higher intra-category clustering and clearer boundaries. These analyses show that BA-LoRA can effectively alleviate the impact of imbalanced data in pre-training.

Figure: Figure 2: Ablation study results of BA-LoRA regularizations on the GSM8K and MATH datasets. Here, "$\mathrm{Reg}$" stands for "Regularization" and "$\mathit{w/o}\ \mathrm{Reg}$" means "without regularization". $\mathcal{L}_{\mathrm{CR_NLG}}$, $\mathcal{L}_{\mathrm{DR_NLG}}$, and $\mathcal{L}_{\mathrm{SVDR_NLG}}$ denote the application of only the corresponding regularization, and "BA-LoRA" refers to the baseline using all regularizations.
Refer to caption: x5.png

#### 4.4.4 Ablation Study

This ablation experiment aims to analyze the impact of three regularization terms ($\mathcal{L}_{\mathrm{CR_NLG}}$, $\mathcal{L}_{\mathrm{DR_NLG}}$, $\mathcal{L}_{\mathrm{SVDR_NLG}}$) in BA-LoRA on model performance. The experiment selected three models, LLaMA-2-7B, Mistral-7B, and Gemma-7B, and evaluated them on GSM8K and MATH datasets.
To ensure clear results, we only tested single regularization terms to reveal their independent contributions.

As shown in Figure 2, the model without regularization ("$\mathit{w/o}\ \mathrm{Reg}$") produced the lowest performance across both datasets. In contrast, introducing different regularization terms led to varying degrees of performance improvement. Specifically, $\mathcal{L}_{\mathrm{CR_NLG}}$ contributed to balanced gains across both datasets, while $\mathcal{L}_{\mathrm{DR_NLG}}$ had a notably strong impact on Gemma-7B. Ultimately, the model incorporating all three regularization terms achieved the highest performance on both datasets. These findings validate the regularization strategy and show that combining the terms in BA-LoRA further enhances model generalization.
These findings confirm the effectiveness of the proposed regularization strategy, and further indicate that combining the regularization terms in BA-LoRA enhances the model’s generalization ability.

## 5 Conclusion

This paper introduces Bias-Alleviating Low-rank Adaptation (BA-LoRA), a novel parameter-efficient fine-tuning method designed to mitigate catastrophic inheritance in pre-trained language models.
BA-LoRA incorporates three key components: consistency regularization, diversity regularization, and singular value decomposition regularization. These components work in concert to preserve pre-training knowledge, enhance output diversity, and improve model generalization.
Extensive experiments demonstrate that BA-LoRA consistently outperforms existing baselines on various NLG and NLU tasks while robust to noisy and imbalanced pre-training data.
Furthermore, our ablation studies confirm the effectiveness of the three regularization terms both individually and in combination.
These results highlight the potential of BA-LoRA as a general-purpose fine-tuning method for pre-trained language models and effectively address the key challenges of deploying these models in real applications.

## 6 Ethics Statement

This study aims to develop and evaluate BA-LoRA, a novel parameter-efficient fine-tuning method designed to mitigate bias and enhance the performance of LLMs. Our research utilizes existing open-source public datasets for both fine-tuning and evaluation purposes.
For Natural Language Generation tasks, we employed widely recognized datasets within the research community, including MetaMathQA, CodeFeedback, and WizardLM-Evol-Instruct. These datasets have no known ethical concerns. For Natural Language Understanding tasks, we utilized the GLUE and GLUE-X benchmarks, standard evaluation datasets in machine learning.
We are committed to the responsible development and application of AI technologies. Throughout this research, we will continue to monitor and address any ethical issues that may arise.

## 7 Reproducibility

To ensure the reproducibility of our results, we provide a detailed description of our experimental setup in Section 4.3 and Appendix Section B, including model introduction, dataset introduction, hyperparameter configuration, and evaluation procedures. All models and datasets used are publicly available. In addition, we have refined the implementation scripts and fine-tuning strategies to facilitate independent verification. Our source code and pre-trained model weights will be made public upon acceptance of this paper, ensuring that our results are fully transparent and reproducible.

## Appendix A Background

### A.1 Challenges of Bias and Noise in Pre-training Data

Bias and noise within pre-training datasets present significant hurdles in constructing dependable machine-learning models. Mislabeled data and imbalanced distributions can lead to models that not only underperform on downstream tasks but also reinforce existing biases in the data (Torralba & Efros, ; Barocas & Selbst, ; Mehrabi et al., ). This issue is especially problematic in large-scale datasets, where manual curation is impractical. Consequently, reliance on automated data collection methods may introduce various inaccuracies and biases (Northcutt et al., ; Birhane & Prabhu, ). The challenge becomes even more severe when dealing with real-world, instance-dependent label noise. Models trained on such data may inadvertently learn these inaccuracies, resulting in poor generalization (Frénay & Verleysen, ; Song et al., ; Algan & Ulusoy, ). Addressing these challenges is essential for advancing machine learning and ensuring models are both effective and equitable.

### A.2 Mitigating Bias and Noise through Parameter-Efficient Fine-Tuning Methods

To counteract the adverse effects of bias and noise in pre-training data, parameter-efficient fine-tuning methods have emerged as promising solutions. These approaches aim to adapt pre-trained models to new tasks with minimal parameter updates, thereby reducing the risk of overfitting to noisy or biased data (Houlsby et al., ; Zaken et al., ; Lester et al., ). Techniques such as integrating lightweight adaptation modules (Pfeiffer et al., ; Jang et al., ), utilizing prefix tuning (Li & Liang, ; Liu et al., ), and employing low-rank adaptations (Hu et al., ; Ding et al., ) enable efficient model refinement while preserving the valuable representations acquired during pre-training. Selectively fine-tuning specific model components can enhance performance on downstream tasks, improve generalization, and reduce the influence of noise and bias (Zaken et al., ; Mahabadi et al., ; Guo et al., ). This strategy not only results in more robust models but also contributes to the development of fairer AI systems by directly addressing fundamental data quality issues.

## Appendix B Details of Models and Datasets

### B.1 Details of Models

We use a variety of pre-trained language models, including Meta AI’s LLaMA-2-7B and LLaMA-2-13B (Touvron et al., ) and the latest LLaMA-3-8B and LLaMA-3-70B (AI@Meta, ), which have good performance in natural language generation tasks. In addition, we also use Mistral AI’s Mistral-7B-v0.1 (Jiang et al., ) optimized for medium-sized model efficiency, and Google’s lightweight open-source model Gemma-7B (Jiang et al., ), which performs well in tasks such as question-answering summarization and reasoning. Alibaba Cloud’s Qwen-1.5-7B (Bai et al., ) model also provides strong language understanding and generation capabilities, while the 34B parameter Yi-1.5-34B (Young et al., ) is designed for high-level language tasks. DeepSeek-MoE-16B (Dai et al., ) is a model that uses expert routing to increase capacity without significantly increasing computational costs. Mixtral-8x7B-v0.1 (Jiang et al., ) is a Sparse Mixture of Expert models that efficiently utilizes active parameters to outperform larger models like Llama 2 70B and GPT-3.5 across several benchmarks. We also leveraged mature models such as BERT-Large (Devlin et al., ), RoBERTa-large  (Liu et al., ), DeBERTa-v3-base (He et al., ), and GPT-2-XL (Radford et al., ), which continue to set standards in natural language processing and text generation tasks.

**Table 5: Comparison of Pre-trained Data and Methods for Various Language Models.**
| Model | Pre-trained Data | Pre-training Method |
| --- | --- | --- |
| BERT-L (Devlin et al., ) | BooksCorpus and English Wikipedia | Masked Language Modeling |
| RoBERTa-L (Liu et al., ) | BooksCorpus and English Wikipedia | Masked Language Modeling |
| GPT-2-XL (Radford et al., ) | WebText | Autoregressive Language Modeling |
| DeBERTa-v3-base (He et al., ) | Wikipedia, BooksCorpus, OpenWebText, CC-News, and Stories | Replaced Token Detection with GDES |

Table 5 presents an overview of the pre-trained language models used in our study. BERT-Large (BERT-L) and RoBERTa-Large (RoBERTa-L) are pre-trained on the BooksCorpus and English Wikipedia datasets using a masked language modeling objective. In contrast, GPT-2-XL is pre-trained on WebText with an autoregressive language modeling objective. Additionally, DeBERTa-v3-base is trained on a diverse dataset comprising Wikipedia, BooksCorpus, OpenWebText, CC-News, and Stories, utilizing a replaced token detection objective with Gradient Disentangled Embedding Sharing (GDES). These models span a variety of architectures and pre-training strategies, offering a robust basis for evaluating the performance of our proposed approach.

### B.2 Details of Datasets

Table 6 provides an overview of the GLUE benchmark datasets and their evaluation metrics. The GLUE benchmark comprises a diverse set of natural language understanding tasks, including grammatical acceptability (CoLA), sentiment analysis (SST-2), paraphrase detection (MRPC and QQP), sentence similarity (STS-B), natural language inference (MNLI, QNLI, and RTE), and coreference resolution (WNLI). The number of training examples varies significantly across datasets, from as few as 634 in WNLI to as many as 393,000 in MNLI. Tasks involve binary or multi-class classification, with up to five classes in STS-B. Evaluation metrics are tailored to each task, employing accuracy, F1 score, Matthews correlation coefficient, and Pearson/Spearman correlation coefficients where appropriate. This comprehensive suite serves as a standard benchmark for assessing and comparing the performance of models across a wide array of linguistic challenges.

Table 7 summarizes the GLUE-X out-of-domain tasks employed for evaluating transfer performance. The datasets cover a broad spectrum of natural language understanding tasks, including natural language inference (SNLI, HANs, SciTail, MNLI mismatched), sentiment analysis (IMDB), question answering (NewsQA), semantic relatedness (SICK), and grammatical error detection (Grammar Test). Each task involves binary classification, with test sizes ranging from 9,832 samples (MNLI mismatched) to 570,152 samples (SNLI). Accuracy is the primary evaluation metric across most datasets, except for the Grammar Test, which uses the Matthews correlation coefficient. These diverse tasks provide a comprehensive benchmark for assessing the models’ ability to generalize across different domains and tasks.

**Table 6: GLUE Benchmark Datasets and Evaluation Metrics**
| Dataset | Task Type | Classes | Train Examples | Metric | Description |
| --- | --- | --- | --- | --- | --- |
| CoLA | Acceptability | 2 | 8.5k | Matthews Corr. | Grammatical acceptability |
| SST-2 | Sentiment | 2 | 67k | Accuracy | Sentiment analysis |
| MRPC | Paraphrase | 2 | 3.7k | Accuracy/F1 | Paraphrase detection |
| QQP | Paraphrase | 2 | 364k | Accuracy/F1 | Duplicate question detection |
| STS-B | Similarity | 5 | 7k | Pearson/Spearman Corr. | Sentence similarity |
| MNLI | NLI | 3 | 393k | Accuracy | Multi-genre NLI |
| QNLI | NLI/QA | 2 | 108k | Accuracy | QA/NLI converted from SQuAD |
| RTE | NLI | 2 | 2.5k | Accuracy | Textual entailment |
| WNLI | Coreference | 2 | 634 | Accuracy | Winograd Schema Challenge |

**Table 7: Summary of GLUE-X Out-of-Domain Tasks for Transfer Performance Evaluation**
| Dataset | Task Type | Classes | Train Examples | Metric | Description |
| --- | --- | --- | --- | --- | --- |
| SNLI | NLI | 2 | 570k | Accuracy | Sentence-level inference tasks |
| IMDB | Sentiment | 2 | 50k | Accuracy | Movie review sentiment analysis |
| HANs | NLI | 2 | 60k | Accuracy | Adversarial NLI examples to test models |
| NewsQA | QA | 2 | 119k | Accuracy | QA from news articles |
| SICK | Semantic Relatedness | 2 | 9.8k | Accuracy | Semantic relatedness and entailment |
| Grammar Test | Grammar Detection | 2 | 304k | Matthews Corr. | Grammatical error detection |
| SciTail | NLI | 2 | 26.5k | Accuracy | Science question entailment |
| MNLI mismatched | NLI | 2 | 9.8k | Accuracy | NLI with mismatched genres |

### B.3 Specific Hyperparameter Settings of RoBERTa-large and DeBERTa-v3-base on GLUE

We fine-tuned the RoBERTa-large and DeBERTa-v3-base models on the GLUE benchmark datasets using carefully selected hyperparameters tailored to each task. For RoBERTa-large, we trained on MNLI and SST-2 for 10 epochs with a batch size of 32; MNLI employed a learning rate of $1\times 10^{-4}$, while SST-2 used $2\times 10^{-4}$, both with LoRA_alpha set to 16. Smaller datasets such as MRPC, CoLA, and RTE were trained for 20 epochs with batch sizes of 16, utilizing higher learning rates ranging from $3\times 10^{-4}$ to $6\times 10^{-4}$ and LoRA_alpha values of 8 or 16. For DeBERTa-v3-base, MNLI was trained for 5 epochs with a batch size of 16, a learning rate of $5\times 10^{-5}$, and LoRA_alpha set to 8. Datasets such as SST-2 and MRPC were trained for 20 epochs with batch sizes of 16 or 32, learning rates between $3\times 10^{-5}$ and $2\times 10^{-4}$, and LoRA_alpha of 8. Notably, RTE was trained for 50 epochs with a batch size of 16, a learning rate of $1\times 10^{-4}$, and LoRA_alpha of 8. The LoRA_alpha parameter was set to either 8 or 16, depending on the model and dataset. In all cases, the LoRA_rank was set to 8. These hyperparameters were meticulously chosen to suit the specific requirements of each dataset, ensuring rigorous and optimal training across tasks such as natural language inference, sentiment analysis, paraphrase detection, linguistic acceptability, and semantic textual similarity.

### B.4 Specific Hyperparameter Settings of BERT-L and GPT-2-XL on GLUE and GLUE-x

To ensure consistent and reliable performance, the BERT-Large (BERT-L) and GPT-2-XL models were trained on the GLUE benchmark tasks using three different random seeds per task over 10 epochs. A hyperparameter search was conducted over learning rates $\{2\times 10^{-5},3\times 10^{-5},5\times 10^{-5}\}$, and a batch size of 32 was chosen to balance computational efficiency and memory usage. For fine-tuning, the training schedule was adjusted to 20 epochs for smaller datasets, while larger datasets such as QNLI, MNLI, and QQP were trained for 5 epochs. Learning rates were explored within $\{2\times 10^{-4},3\times 10^{-4},5\times 10^{-4}\}$. The parameters were set with LoRA_rank $=8$ and LoRA_alpha $=16$, with the batch size reduced to 16 due to increased model complexity. All other parameters, including max_length, adhered to Hugging Face Transformers guidelines(^1^11[https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)).

Regarding the GLUE-x tasks, BERT-L and GPT-2-XL models trained on GLUE were evaluated without further fine-tuning. GLUE-x encompasses 13 out-of-distribution (OOD) tasks, introducing domain shifts. For sentiment analysis, models fine-tuned on SST-2 were evaluated on test sets from IMDB (Maas et al., ), Yelp (Zhang et al., ), Amazon (Kaushik et al., ), and Flipkart (Vaghani & Thummar, ), offering a broader assessment of domain variability and testing the robustness beyond SST-2.

For t-SNE visualization, we used the MNLI subset from GLUE due to its diverse linguistic styles and label distributions. Training was limited to one epoch to expedite the process, while still providing insights into how well the models differentiate between classes and sentence structures.

### B.5 Model Evaluation Details

For evaluation, we employed publicly available frameworks. The model’s code generation capabilities were assessed using datasets like HumanEval and MBPP through the BigCode Evaluation Harness(^2^22[https://github.com/bigcode-project/bigcode-evaluation-harness](https://github.com/bigcode-project/bigcode-evaluation-harness)). Instruction-following performance was evaluated using MTBench(^3^33[https://github.com/lm-sys/FastChat](https://github.com/lm-sys/FastChat)).

## Appendix C More Experiments

### C.1 Analysis on Different Sizes and Types of Models

Figure: Figure 3: Performance Comparison of Different Models on GSM8K and HumanEval Benchmarks.
Refer to caption: x6.png

Figure: Figure 4: Performance Comparison of LoRA and BA-LoRA Across Various Ranks.
Refer to caption: x7.png

This experiment compares LoRA, PiSSA, and BA-LoRA across ten models: LLaMA-2-7/13B (Touvron et al., ), LLaMA-3-8B/70B (AI@Meta, ), Mistral-7B-v0.1 (Jiang et al., ), Gemma-7B (Jiang et al., ), Qwen1.5-7B (Bai et al., ), Yi-1.5-34B (Young et al., ) and Mixture-of-Experts (MoE (Shazeer et al., )) models: DeepSeek-MoE-16B (Dai et al., ) and Mixtral-8x7B-v0.1 (Jiang et al., ). These models were fine-tuned on the MetaMathQA-100K and CodeFeedback-100K datasets and evaluated on the GSM8K and HumanEval benchmarks. As depicted in Figure 3, BA-LoRA consistently surpasses LoRA and PiSSA across all models and tasks, underscoring its superior ability to enhance model generalization.

### C.2 Evaluating the Performance of Different Ranks

We compare the performance of BA-LoRA, LoRA, and PiSSA at different ranks using LLaMA-2-7B and Mistral-7B-v0.1 models. Each method is fine-tuned for one epoch on the MetaMathQA-100K dataset with ranks ranging from 1 to 128 and evaluated on the GSM8K and MATH datasets.
As shown in Figure  4, BA-LoRA consistently outperforms LoRA and PiSSA across all rank settings and datasets.
As the rank increases, the performance of BA-LoRA and PiSSA surpasses full parameter fine-tuning. However, BA-LoRA performs better, especially on Mistral-7B-v0.1.

### C.3 t-SNE Visualizations of Feature Evolution during the Fine-tuning with LoRA and BA-LoRA

This section provides more detailed t-SNE visualization results to compare the feature evolution during fine-tuning of LoRA and BA-LoRA.

Figure 5 shows that during LoRA fine-tuning of BERT-L, the feature separation is slow, with the class distributions remaining scattered and overlapping even towards the end. In contrast, Figure 6 demonstrates that with BA-LoRA fine-tuning, class separation begins earlier and is much clearer, ultimately forming a distinct "Y" shape with well-defined class boundaries.

Similarly, Figure 7 shows that during LoRA fine-tuning of GPT-2 XL, the feature clusters remain scattered and overlapping throughout the training, with only minimal separation between classes by the final steps. In contrast, Figure 8 demonstrates that BA-LoRA fine-tuning results in much clearer and more distinct class separation, with well-defined boundaries emerging earlier in the training process and becoming more pronounced over time.

Figure: Figure 5: t-SNE Visualization of Feature Evolution during LoRA Fine-Tuning of BERT-L.
Refer to caption: x8.png

Figure: Figure 6: t-SNE Visualization of Feature Evolution during BA-LoRA Fine-Tuning of BERT-L.
Refer to caption: x9.png

Figure: Figure 7: t-SNE Visualization of Feature Evolution during LoRA Fine-Tuning of GPT-2-XL.
Refer to caption: x10.png

Figure: Figure 8: t-SNE Visualization of Feature Evolution during BA-LoRA Fine-Tuning of GPT-2-XL.
Refer to caption: x11.png

## Appendix D More Discussions

Here, we offer further insights into our work.

### D.1 Limitations

While BA-LoRA demonstrates significant improvements in mitigating catastrophic inheritance and enhancing model performance, several limitations warrant further investigation. Firstly, our evaluations primarily focus on English language tasks, which may limit the generalizability of our findings to other languages and specialized domains. Additionally, the computational overhead introduced by the consistency, diversity, and SVD regularizers adds complexity to the training process, potentially impacting efficiency. Furthermore, the impact of BA-LoRA on other forms of bias, such as fairness and societal stereotypes, remains unexplored. Lastly, the selection and weighting of regularization terms in BA-LoRA are fixed across different tasks, which may not be optimal for all scenarios.

### D.2 Future works

Future research should extend assessments of BA-LoRA to multilingual settings and specialized domains to ensure broader applicability. Exploring optimization techniques could help reduce the computational overhead introduced by the regularizers, balancing performance gains with efficiency. Investigating the impact of BA-LoRA on other forms of bias, including fairness and societal stereotypes, is crucial for developing more equitable models. Additionally, refining the selection and weighting of regularization terms—possibly through automated or dynamic adjustment methods—could enhance adaptability across different tasks and models. Testing the scalability of BA-LoRA on larger models with hundreds of billions of parameters and exploring its integration with other bias mitigation strategies may yield synergistic effects and further improve model robustness.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weaknesses.
soundness: 2
strengths: Three regularization terms are proposed for LoRA to mitigate the detrimental effects of Catastrophic Inheritance from pre-training.

The paper conducts experiments on natural language understanding and natural language generation tasks with serveral benchmarks and LLMs.
summary: The paper introduces Bias-Alleviating Low-Rank Adaptation (BA-LoRA) with three regularization terms: (1) a consistency regularizer, (2) a diversity regularizer, and (3) a singular value decomposition regularizer. They conduct experiments on natural language understanding and natural language generation tasks, and the results demonstrate that BA-LoRA outperforms LoRA and PiSSA.
weaknesses: The experiments are not sufficient, which only compare limited baselines, but ignore many relevant methods, such as DoRA, LoHA, DyLoRA, DeltaLoRA, which are mentioned in related works. The paper does not provide enough evidence to support its claim that BA-LoRA outperforms LoRA and its state-of-the-art variants.

In the experiments, according to Table 1 and 2, why Full FT has worse performance than PEFT method? 

The discussion about why these three regularization terms can mitigate the detrimental effects of Catastrophic Inheritance is not sufficient. 

In my option, these regularization terms should be universally applicable. That is to say, the regularization terms should be added to LoRA’s variants, and improve their performance. More experiments should be conducted for this. 

In section 3, why PISSA is described in such detail, is it highly relevant to BA-LoRA?

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses
soundness: 3
strengths: A regularization formula for LoRA training was proposed for NLU and NLG tasks respectively, and excellent improvement was achieved in model performance.
summary: This paper investigates the challenges associated with fine-tuning large language models (LLMs), particularly focusing on catastrophic forgetting and bias introduction. It introduces BA-LoRA, a method aimed at improving the adaptation of LLMs while minimizing computational costs. The authors highlight the issue of catastrophic inheritance, where models lose previous knowledge when trained on new tasks. BA-LoRA employs a low-rank adaptation technique, allowing for efficient parameter updates during fine-tuning. Experimental results demonstrate that this approach effectively mitigates forgetting and reduces bias in adapted models, outperforming traditional fine-tuning methods. The findings suggest that BA-LoRA represents a significant step forward in making LLM adaptation more efficient and reliable for various natural language processing applications.
weaknesses: 1.Ablation experiments need to supplement the experimental results of  Ltask_NLG and  Ltask_NLU to enrich Figure 2.


2.According to the results in Figure 2, Ldr_NLG improves LlaMA-2-7B more significantly than Gemma-7B. Is the " while LDR_NLG had a notably strong impact on Gemma-7B." in the text inconsistent with the picture information?


3.Why can we conclude that 'BA-LoRA can effectively alleviate the impact of imbalanced data in pre-training' based on Figure 1 and the observation that 'BA-LoRA in sub-figures (b) and (d) shows clearer category separation'? What is the relationship between these two observations?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - As mentioned in the weakness section, this work highlights the catastrophic inheritance issue in the pertaining data, which is not better solved by other LoRAs. However, this work did not provide a clear definition of what kinds of noise and imbalance issues this work will consider. Especially, authors claim that BA-LoRA effectively mitigates the adverse effect of Catastrophic Inheritance, fostering consistent, diverse and generalized model outputs in line 379 section 4.4.1 based on the results of Tables 1 and 2. As the definition is unclear, it is hard to directly conclude that all credits for improvements from Tables 1 and 2 can be given to noise and imbalance mitigation. We need to see more examples that BA-LoRA can actually mitigate the specific noise and imbalance issues.
- In Table1, it is not clear that what kind of metric is used to evaluate the performance in NLG tasks.
- In section 4.3, the values of $\lambda_1$, $\lambda_2$, and $\lambda_3$ for NLG and NLU tasks are so small. Do these values bring enough contributions to the entire loss function?
- In sections 4.4.2 and 4.4.3, there are only results and analyses on NLU datasets. More results on both tasks are needed to better demonstrate the ability of BA-LoRA
soundness: 2
strengths: - BA-LoRA designed 3 different regularizations in terms of consistency, diversity and generalization for NLG and NLU datasets based on the principle singular values and singular vectors adaptation (PISSA), respectively.
- Based on the experiment results, BA-LoRA improves several baselines on NLG and NLU datasets.
summary: In this paper, authors proposed a BA-LoRA, which focuses on mitigating the bias propagation issue from the pertaining data used for different LLMs. BA-LoRA includes three different regularization loss functions regarding NLG and NLU tasks, i.e., a consistency regularizer, a diversity regularizer, and a singular value decomposition regularizer to enhance model's consistency, diversity and generalization during fine-tuning. BA-LoRA was evaluated on different NLG and NLU datasets using different LLMs, such as LLaMA 2 7B, Gemma 7B, and some encoder-only LMs, such as BERT-Large, RoBERTa-Large, which demonstrates the better performance of BA-LoRA.
weaknesses: - The definition of noise and imbalance in this work is not clear regarding different NLG and NLU datasets
- It lacks enough baselines to compare in terms of "Catastrophic Inheritance", especially for the noise and imbalance on the NLG and NLU datasets.
- BA-LoRA needs adapted regularization terms designed for NLG and NLU tasks, which might bring generalization concerns for this BA-LoRA if it is applied to other tasks that are not discussed in this paper, such as classification task.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: There seems to be no clear difference between spectral regularization applied to NLU and NLG tasks. And what is the overall need to distinguish between them? Why do we need to target them separately?
soundness: 1
strengths: The regularization of LLMs is a crucial area of study, and the proposed techniques have a strong foundation with clear descriptions. 

The paper is well-written, and the experiments are notable, including recent benchmarks like MT-Bench.
summary: The paper addresses the issues of bias and catastrophic inheritance, proposing several loss components based on prior studies to regularize LLMs. It includes experiments demonstrating the method's superiority over regular LoRA, PiSSA, and full fine-tuning without regularization.
weaknesses: **The baselines are insufficient.**

While LoRA and PiSSA are included, they lack any regularization. As stated in the paper, dropout is set to 0, meaning there is no regularization within the layers for these baselines, nor is there any weight decay applied. Consequently, the baselines may easily collapse due to the choice of hyperparameters. As it stands, it's unclear whether the gains from your method could simply be achieved by incorporating weight decay and optimized dropout.

The lack of regularization is further evident from the fact that full fine-tuning performs worse than LoRA. While isolated cases of this behavior might occur, it shouldn’t be a consistent trend; otherwise, tuning the entire network would serve little purpose.

**Novelty**

Regarding novelty, regularization has been an extensively studied topic. Although the paper cites works on entropy, KL regularization, and spectral regularization, there is an implication that these are novel contributions, which they are not. If these regularizers have been previously studied, they should be included as baselines. This approach would essentially frame the study as an adaptation of these regularizers to LLMs and a combination of them, without showing each individually. Thus, it appears there is limited novelty beyond incremental adaptations and combinations tailored for LLMs.

**Computational Costs**

The computational costs appear to counteract the main advantage of PEFT. For instance, consistency regularization requires an additional forward pass and doubles the storage for activations. The diversity component is somewhat manageable, but the SVD aspect raises concerns. Based on my knowledge and experience, computing all eigenvalues, particularly for large matrices, can be significantly time-consuming. This is further complicated by being a CPU-bound task, which limits parallelization. For example, the smallest matrix in LLaMA is 4096x4096, requiring roughly 5 seconds to compute, and this is just the smallest case. It raises serious doubts about the practical scalability of this approach.

**Catastrophic Inheritance**

I doubt that bias and catastrophic inheritance were truly assessed. According to the paper, it is simply done by using models that were "probably" trained on noisy data, such as BERT and GPT-2. Moreover, they are outdated and not commonly used in current applications. Improvements in final scores do not necessarily indicate any effective handling of biases.

Regarding MNLI, it’s unclear how it is described as imbalanced. The official dataset on Hugging Face has an equal distribution across classes, so any claims about imbalance need clarification.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: How do you determine the optimal values for the regularization weights (λ1, λ2, λ3)?
What is the computational overhead compared to standard LoRA?
How does the performance fare on non-English datasets or other NLP tasks?
Have you tested the effects on other types of bias beyond those mentioned in the paper?
soundness: 4
strengths: This paper addresses an essential issue for the practical deployment of large language models (LLMs) by proposing BA-LoRA, a parameter-efficient fine-tuning method aimed at mitigating catastrophic inheritance problems. 

The motivation is clear and timely, given the growing demand for efficient fine-tuning techniques in LLMs. The work introduces technical innovations, including a novel combination of regularization strategies tailored to handle task-specific characteristics in natural language understanding (NLU) and natural language generation (NLG). 

The method effectively integrates with existing LoRA frameworks, enhancing its applicability and relevance. Experimental validation is thorough, covering various models and demonstrating substantial performance improvements over baseline approaches. The analysis is supported by detailed component studies and persuasive qualitative analysis, including t-SNE visualizations, which further illustrate the effectiveness of BA-LoRA.
summary: This paper proposes BA-LoRA, a parameter-efficient fine-tuning method to mitigate the catastrophic inheritance issue in LLMs. Key contributions include (1) three regularization terms for bias mitigation, (2) optimized adaptation methods for NLU/NLG tasks, and (3) comprehensive evaluations across various models and benchmarks.
weaknesses: However, the paper has some limitations in terms of theoretical analysis, experimental scope, and comparative analysis. 

Theoretical analysis is lacking, with no clear basis for the selection of regularization weights and limited discussion on computational complexity and convergence. 

The experimental scope is restricted, as the study only tests on English datasets, leaving the method's performance in multilingual contexts unexplored( and also other tasks as well ). Additionally, the analysis of computational overhead is minimal.

Meta-review: This paper proposes BA-LoRA to mitigate catastrophic inheritance in LLMs through three regularization terms (consistency, diversity, SVD regularization). The authors claim BA-LoRA could outperform (vanilla) LoRA and other parameter-efficient fine-tuning methods, and effectively reduce the bias propagation from pretraining data.

Strengths: This paper addresses an important challenge - bias mitigation during LLM fine-tuning. Both quantitative and qualitative (t-SNE visualization) analysis have been conducted.

Weaknesses: The main issue raised by the reviewers is that the experimental results cannot support the main claim. While performance improvements are shown, the causation between these improvements and the reduction in catastrophic inheritance is not clearly established (and defined). The experimental setup also raises some concerns, as baseline methods lack standard regularization methods. Therefore, I recommend to reject this paper.

Decision: Reject