BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
Authors: Yupeng Chang
Yi Chang
Yuan Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Parameter-Efficient Fine-Tuning
Low-Rank Adaptation
Catastrophic Inheritance
Bias Mitigation
Regularization
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We identify that low-rank adaptation can exacerbate "Catastrophic Inheritance" of pre-training biases and propose BA-LoRA, a regularization framework that mitigates this issue, achieving superior robustness and performance.

Abstract: Parameter-efficient fine-tuning (PEFT) has become a *de facto* standard for adapting Large Language Models (LLMs). However, we identify a critical vulnerability within popular low-rank adaptation methods such as LoRA: they can exacerbate ``Catastrophic Inheritance''---the unchecked propagation of biases, noise, and data imbalances from pre-training. This phenomenon can degrade model robustness and fairness, undermining the benefits of efficient adaptation. To address this, we introduce Bias-Alleviating Low-Rank Adaptation (BA-LoRA). Our approach is founded on a principled decomposition of Catastrophic Inheritance into three core challenges: Knowledge Drift, Representation Collapse, and Overfitting to Noise. BA-LoRA systematically mitigates these issues by incorporating a trio of targeted regularizers: consistency, diversity, and an SVD-based term, designed to preserve core knowledge, enforce representational richness, and promote robust, low-rank output representations, respectively. We conduct comprehensive evaluations on a suite of Natural Language Generation (NLG) and Understanding (NLU) tasks using diverse, prominent open-source language models (e.g., LLaMA-2-7B and DeBERTa-v3-base). Our results show that BA-LoRA not only outperforms state-of-the-art LoRA variants in terms of performance and stability, but also demonstrates superior robustness and bias mitigation on targeted evaluations. These results provide evidence that BA-LoRA can counteract the adverse effects of Catastrophic Inheritance. Code is available at https://github.com/llm172/BA-LoRA.

Published as a conference paper at ICLR 2026
BA-LORA: BLAS-ALLEVIATING LOW-RANK ADAPTA- TION TO MITIGATE CATASTROPHIC INHERITANCE IN LARGE LANGUAGE MODELS
Yupeng Chang', Yi Changl.23, Yuan Wu1
Schooo of Artificial Intelligence, Jilin University
Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University IInternational Center of Future Science, Jilin University
yupeng. chang. ai & emmaal com, yichange elu.eddu.c.. yuanwul edu. on
ABSTRACT
Parameter-efficient fine-tuning (PEFT) has become a de facto standard for adapting Large Language Models (LLMs). However, we identify critical vulnerability within popular low-rank adaptation methods such as LoRA: they can exacerbate "Catastrophic Inheritance" the unchecked propagation of biases, noise, and data imbalances from pre-training. This phenomenon can degrade model robustness and fairness, undermining the benefits efficient adaptation. To address this, we introduce Bias-A Aaleviatiii Low-Rank Adaptation (BA-LoRA). Our approach is founded on a principled decomposition of Catastrophic Inheritance into three core challenges: Knowledge Drift, Representation Collapse, and Overfitting to Noise. BA-LoRA systematically mitigates these issues by incorporating a trio targeted regularizers: consistency, diversity, and an SVD-based term, designed to preserve core knowledge, enforce representational richness, and promote robust, low-rank output representations, respectively. We conduct comprehensive evaluations a suite of Natural Language Generation (NLG) and Understanding (NLU) tasks using diverse, prominent open-source language models (e-g., I.aMM-2-7BB and DeBERTa-v3-base). Our results show that BA-LoRA not only outperforms state-of- the-art LoRA variants in terms of performance and stability, but also demonstrates superior robustness and bias mitigation targeted evaluations. These results pro- vide evidence that BA-LoRA can counteract the adverse effects of Catastrophic In- heritance. Code is available at https::ss.... //tt:::////ttbb.... com 1m1172 172 BA LORA
INTRODUCTION
Large language models (LLMs) like GPT-4 (OpenAl
2023) and LLaMA Touvron al., 2023) have redefined the state-of-the-art in natural language processing (NLP). largely due to their training on vast, web-scale corpora (Zhao et al., 2023 Chang et al., (024). This strategy, while enabling unprecedented generalization (Gao al. 2020 Penedo et al. (023), comes at a cost: models inevitably inherit and internalize the biases, noise, and imbalances latent within these unfiltered
datasets (Parashar al. 2024 Liu & He 2024 Chen et al. 2024).
Recent research confirms that these inherited flaws can degrade model performance and persist even after fine-tuning, posing significant risks to fairness and safety (Qi et al. 2023 Bommasani et al.)
2021 Mallen et al. 2022; Carlini et al., 2023 For example, noise within the training data can degrade model generalization (Chen al. 2024), while the long-tailed distribution of concepts
cause LLMs to overemphasize overrepresented topics (Zhu et al. 2024 Dong et al. 2023).
This phenomenon, termed "Catastrophic Inheritance" (Chen al.] 2024), arises when models inherit such biases, noise, and imbalances from pre-training. In this work, we focus on how these inherited artifacts can be further amplified during downstream fine-tuning, challenge that has spurred our investigation into targeted mitigation strategies. While constructing less biased datasets
"Corresponding author

---

Published as a conference paper at ICLR 2026
Training Objective
y
Frash Arc + Lpp
5412 Boat (sr)
Pretrained Matrix
Pretrained Matrix
Residual Matrix
W E R B
W
E R
A-M(0, a)
Exvne
//
(a) Full Fine-tuning
(b) LoRA
(c) BA-LoRA (ours)
Comparison of three fine-tuning frameworks: (a) Full Fine-tuning, updating the entire matrix W; (b) LoRA, training a low-rank adapter for frozen W; and (c) our proposed BA-LoRA. Blue and orange modules denote frozen and trainable parameters, respectively. Our method initializes its adapter and residual matrix (WTess from the SVD of W (following PiSSA (Meng al. 2024 It augments the task loss (Lusk) with three regularizers (purple module) to mitigate Catastrophic Inheritance by preserving knowledge, promoting diversity, and focusing on core data patterns.
Figure 1:
and developing more robust model architectures are prominent approaches (Liu & He 2024). this study explores alternative: innovations in fine-tuning Fine-tuning is powerful method for enhancing task sase-ipe performance and aligning models with user intent (Han et al., 2024 Ouyang al. 2022). However, its computational demands substantial; for instance, 16-bit fine-tuning of
an 1LaMA-655 model requires over 780 GB of GPU memory (Dettmers et al.) 2024). To address these limitations, parameter-efficient fine-tuning (PEFT) techniques, such Low-Rank Adaptation (LoRA) (Hu et al. 2021), have gained prominence.
LoRA enables efficient fine-tuning by approximating parameter updates using low-rank matrices. As illustrated in Figure T ()). Full Fine-tuning directly updates the entire weight matrix W. In contrast, LoRA (Figure [b) introduces a learnable low-rank adapter AW AAB, where A Rm and B E R.xn are trainable matrices with rank < min(m,, n). Only A and B are updated, while the original weights W remain frozen. By initializing with scaled random values and B to zero, LoRA ensures the adapter has no effect at the start of training. The forward pass is then computed as Y ==x0 XW + AB), significantly reducing computational costs (Hu et al. 2021.
While PEFT methods like LoRA offer remarkable efficiency, their constrained, low-rank updates introduce critical vulnerability: they can exacerbate Catastrophic Inheritance when fine-tuning noisy imbalanced data without explicit regularization. By forcing all model adjustments through a low-dimensional bottleneck, these methods may lack the capacity to correct for inherited biases, instead amplifying spurious correlations from pre-training data. To bridge this gap, argue that a more principled approach is needed. We first deconstruct Catastrophic Inheritance into three primary failure modes: Knowledge Drift, where the model unintentionally forgets robust pre-trained knowledge while learning new tasks (Kirkpatrick et al.. 2017); Representation Collapse, where tinettunng imbalanced data causes output diversity plummet (Bardes al. 2021): ); and Overfitting to Noise, where the model learns spurious correlations from the training data that hinder generalization (Chen et al. 2019). This paper introduces Bias-Alleviating Low-Rank Adaptation (BA-LoRA), novel method that systematically mitigates these issues. As depicted in Figure (C)) BA-LoRA builds upon the efficient PiSSA (Meng et al. 2024) initialization and incorporates a trio of regularizers: consistency regularizer to combat Knowledge Drift, diversity regularizer to prevent Representation Collapse, and an SVD-based regularizer to mitigate Overlitting to Noise. Recognizing the differences between NLU and NLG tasks, we tailor these strategies accordingly.
Our comprehensive evaluation establishes BA-LoRA's superior performance and deconstructs the sources of its effectiveness. BA-LoRA consistently outperforms leading LoRA variants across diverse benchmarks, including mathematical reasoning, coding, and conversational AI for NLG, as well as the GLUE benchmark (Wang et al., 2018) for NLUU, using models such I.aMA-2-7BB (Touvron et al.
2

---

Published as a conference paper at ICLR 2026
2023) and DeBERTa-v3 -a---------aee- (He al. 2021). Crucially, move beyond standard leaderboards to test our central hypothesis. An empirical comparison of models pre-trained with high-quality curated (RoBERTa (Liu al., 2019)) versus noisier web-scale (T5 (Raffel et al., 2020)) data shows that BA-LoRA achieves larger gains on the latter, which is consistent with our hypothesis that BA-LoRA is particularly beneficial when mitigating the effects of inherited noise. This primary finding is supported by comprchensive ablation studies and qualitative visualizations that confirm the necessity of our three-pronged strategy. Together, these results not only demonstrate BA-LoRA's superiority but also validate our theoretical framework for understanding and mitigating this phenomenon.
2 METHOD
2.1 PRINCIPAL SINGULAR VALUES AND SINGULAR VECTORS ADAPTATION (PISSA)
As variant of LORA, PiSSA (Meng al. 2024) accelerates convergence by retaining the core LoRA architecture but changing the initialization. It leverages the principal components of the original weight matrix W to initialize the adapter matrices A and B, and stores the remaining components in residual matrix Wies E mR We write the SVD of W E R as W = USVT, where U and V contain the left and right singular vectors and S is a diagonal matrix of singular values sorted in descending order. PiSSA partitions the singular components into principal {V(s.r] Strra)) (irrr and residual {U (U.rr:) Str.rg) Vi.ra J parts, where is the user-specified adapter rank. The principal components are then used to initialize the low-rank adapter with A E Rmxr and B E T
A = Up.oo]
$1/2 E m [[war]
(1)
B S $1/2 V Bx*x [ma)r
[Far]
(2)
The residual matrix Wres remains frozen during fine-tuning:
W = Up.r Strard Vinn E Buxn
PiSSA preserves the pre-trained model's full capacity at the start fine-tuning by using W WTes AB. This approach prioritizes training the most influential parameters, thereby accelerating convergence from the start. Inheriting LoRA's benefits of reduced parameter count and deployment simplicity, PiSSA further leverages efficient SVD computations to expedite the training process. This concentration of updates also motivates the output-space regularizers introduced in BA-LORA.
2.2 BIAS-ALLEVIATING Low-R RANK ADAPTATION (BA-LORA)
Catastrophic Inheritance refers to vulnerabilities from biases inherent in large-scale training data, particularly attribute bias and class imbalance, that degrade downstream performance, introduce unfair biases, and pose security risks. These effects manifest during fine-tuning as three distinct subproblems: Knowledge Drift, where the model unintentionally forgets or distorts robust pre-trained knowledge (Kirkpatrick et al. 2017); Representation Collapse (Bardes et al. [02I];; and Overfitting to Noise (Chen et al. 2019). To address them, we propose BA-L unified framework with three regularizers consistency, diversity, and SVD each aligned to one subproblem. Instead of constraining low-rank adapter weights, BA-LoRA regularizes the output space directly shape functional behavior and mitigate bias, with tailored variants for NLU and NLG tasks.
2.2. REGULARIZATIONS FOR NLU TASKS
Consistency Regularization. To directly combat Knowledge Drift, we adopt a knowledge distil- lation approach based on standard practices (Hinton et al.) 2015), using the Kullback-Leibler (KL) divergence between the temperature-sealed probability distributions. Let Zp, ZF E I2 NDD be the batch output logits from the pre-trained and fine-tune tne-tuned models respectively, where N is the batch size and is the number of classes. The loss is defined as:
CCR 2 (atosfftaaxaP///TT0 softmax(Zr/T))
3

---

Published as a conference paper at ICLR 2026
where T' is temperature parameter that softens the distributions. This objective encourages the fine-tuned model to mimic the nuanced decision decision-making process of the vre-trained model on examples where the teacher signal is reliable, preserving foundational knowledge. The T scaling factor ensures gradient magnitudes are commensurate with standard cross-entropy loss.
Diversity Regularization. To counteract Representation Collapse, partieularly imbalanced datasets, we promote diversity in the model's predictions across a batch. Inspired by (Bardes et al.) 2021), regularize the batch-wise output logits decorrelate the predictions for different classes. Let Zp E R be the logit matrix for batch. We first center the logits and then compute the D covariance matrix C(ZF). The regularizer penalizes the off-diagonal elements of this matrix:
NDU = D
(5)
where the covariance matrix is computed using its matrix form:
1 N Eeentered Ceentered where Zeenered
C(ZF)
Zp - ZF
(6)
Here, ZF the matrix where each row is the mean logit vector computed over the batch. This loss discourages excessive correlation between the model's predictions for any two distinct classes across the batch, thus preventing the model from collapsing towards few dominant classes.
Singular Value Decomposition Regularization. To mitigate Overfitting to Noise and encourage the model learn robust features, we introduce regularizer that encourages the spectral energy of the batch-wise output logit matrix to concentrate in its leading singular components. Inspired by the principle that dominant singular values capture the most salient data patterns (Chen et al. 2019). this regularizer incentivizes the model to form simpler, more coherent decision boundaries for samples within a batch, rather than fitting to high-frequency logit fluctuations that are poorly aligned with the task labels. On the fine-tuned logit matrix Zp E RNDD, we perform SVD and maximize the ratio of spectral energy concentrated in the top-k singular values:
11 Oi
LSVDR
(N.(N.D)
Dj=1
where d; is the i-th largest singular value. The hyperparameter & controls the rank preference, In the NLU experiments, where the number of classes D typically moderate, the computational cost of performing an exact SVD is minimal and poses no challenge to the training efficiency.
Overall Objective Function for NLU. The overall NLU objective is formulated as follows:
INLU = Ltask + A NCu DD + A3CCVVD
(8)
where Ltask_NLU _LU represents the standard cross-entropy loss function for the downstream task, and λ1. A2, and λ are weighting parameters to balance each regularization term's 's contribution.
2.2.2 REGULARIZATIONS FOR NLG TASKS
Consistency Regularization. To combat Knowledge Drift, we employ temperature-controle knowledge distillation (Hinton al. 2015), using the Kullback-Leibler Divergence (KLD) between the output distributions of the fine-tuned (student) model, Pr, and the pre-trained (teacher) model, Pp. A temperature parameter, T, softens these distributions, compelling the student learn the teacher's nuanced output, not just its top prediction, especially on tokens where the teacher distribution provides a meaningful and well-calibrated soft target. The loss is defined as:
just
M
1 M (
LCR.LG =
Σ
[| T))
where for an input sequence X, Zp,; and ZF,; denote the teacher and student logit vectors at position i, respectively. The temperature-scaled conditional probability distributions over the vocabulary are formally defined as Pp((1 T)) = softmax(2p.:/T) and Pr(- x; softmax /TT).. The loss averaged over all M valid (non paddedd tokens in the batch. The critical T scaling factor maintains gradient magnitude consistency with standard distillation.
4

---

Published as a conference paper at ICLR 2026
Diversity Regularization. To counteract Representation Collapse in generation, address fundamental challenge: naively maximizing the entropy of the entire vocabulary distribution conflicts with the task objective of producing coherent text (Gat al. 2020). We resolve this with novel focused entropy regularizer. Inspired by Top- K sampling, our method promotes diversity exclusively within the set of most plausible candidate tokens, denoted Vis K- For each token, minimize the negative entropy (equivalently, maximize entropy) computed solely within this restricted set:
M
DDRNLG =
Σ Σ PEV UppK
Pi h;) log (1 h,)
(10)
M
2=1
DE
where Pbp(1) h,) is the re-normalized probability from the fine-tuned model for candidate token within the set D(9 lop- for the i-th valid token, given the corresponding final hidden state h; It is
computed as:
PF(u h.)
Pp h.)
(11)
Everon Pr(v' h;)
Singular Value Decomposition Regularization. To mitigate Overfitting to Noise, regularize the structure of the batch-wise output logit matrix. Building on the principle that dominant singular values capture salient data patterns (Chen al.] 2019), encourage low-rank structure. For tractability with large vocabularies, we use randomized SVD (Halko et al., [201]) and normalize by the efficient Frobenius norm to avoid expensive full-spectrum computation. We thus define the loss as the negative ratio of the sum of the top-k singular values to the Frobenius norm:
Σ 21=1 Oz
LSVDR_NLC NLG
(12)
[Zraiialzz
Here, o, the i-th largest approximated singular value of the valid logit matrix Zealid |MXM where [V] is the vocabulary size, and ||-|| denotes the Frobenius norm.
Overall Objective Function for NLG. Integrating these components, the final objective function for NLG tasks is weighted sum of the task loss and our three regularization terms:
LNLG NLG + CC DD As As
(13)
where Ltask_Ni NLG is the standard causal language modeling loss. A detailed sensitivity analysis is provided in Appendix C.2
EXPERIMENTS
3.1
IMPLEMENTATION DETAILS
Our experimental setup is broadly aligned with recent PEFT studies (Meng et al. 2024). For NLG tasks on LLaMA-2-7B, we use the AdamW optimizer (Loshchilov & Hutter 2017) with learning rate 2 2 10-5, 50-5__ cosine schedule (0.03 warmup ratio), and no weight decay. We set 1ora_ propout to 0, use B6 precision, LoRA rank (r) and alpha (a) of 128, and effective batch size of 32. The key regularization weights for our method are set to \1 = 0.025, X2 = 0.005, X = 0.005, with SVD rank of k 10. For NLU tasks on the GLUE benchmark, learning rates, batch sizes, and other core hyperparameters are task-specific to strictly align with our baseline, as detailed in the appendix. For our method in the NLU setting, we use no weight decay and regularization hyperparameters to b 0.15, X2 0.03, A3 = 0.03 with an SVD rank k = 5. For each backbone, regularization weights are selected via coarse grid search on held-out validation split and then kept fixed for that setting. Unless otherwise specified, our runs were conducted on NVIDIA A40 GPUs and averaged over three random seeds (42, 1024, 2024). Full and detailed hyperparameter configurations for all models and tasks are available in Appendix B
3.2
RESULTS AND ANALYSIS
3.2.1
PERFORMANCE ON NLG AND NLU TASKS
To evaluate BA-LoRA on NLG tasks, we conduct a fair comparison against strong baselines (Table To ensure matched setup, report baseline scores directly from their original publications (see
5

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison on NLG tasks. We compare our method (BA-LoRA) against
popular fine-tuning baselines, including Full Fine-tuning and various state-of-the-art parameter- efficient techniques. The best results in each column highlighted in bold. Avg the arithmetic mean of the five reported metrics.
Methods
GSM8K
MATH
HumanEval
MBPP
MT-Bench
Avg
Full FT
48.9
40.22
20.52:..229
23. 23.64
10.09
21.08
LoRA
02.58
5.92+0.15
16.80+0.38
21.51 10.43
4.60
18.30
AdaLoRA
41.95
6.24
18. 1040.46
20.19
4.7
18.25
DoRA
41.77+0.74
6.20
16.8640.51
21.60
4.48
18.18
MiLoRA
43.09+11.66
4.33
17.55
20.22
4.50
18.33
LoRA+
47.84
7.21
20.07 #0.38
23.69
5.11
20.78
LoRA-FA
40.2511..4
5.6610.47
15.91 10.41
20.01 10.32
4.67
17.30
LoRA-GA
50.47 10.98
7.13
19.44 10.45
23,05 10.40
5.04
21.03
PiSSA
51.48.40.4
7.60
19.48+0.45
23.8440.46
4.92+0.07
21,46
CorDA
53.90+0.56
8.5240.27
21.03
24.15400.4
5.15
22.55
CorDA+
55.03
8.95+
21.76+00.9
24.74
5.64
23.22
BA-LoRA
55.86+0.35
9.47 +0.52
23.58
36.86
+0.05
26.18
Rank=128
Rank=32
0.8
0.8
BA (t-128)
BA-LARA (t-32)
LoRA
LoRA (r)
1955A (x-12))
0.7-
PESA (r-32)
Training Loss
Training Loss
0.5-
0.4
0.4
0.3-
0.2
0.2
100
200
300
400
500
600
700
100
200
300
400
500
600
700
Training Steps
Training Steps
Figure 2: Training task loss of Full Fine- Tuning (Full FT), LoRA, PiSSA, and BA-LoRA on MetaMathQA: (left) rank 128 and (right) rank 32. All curves are smoothed for visual clarity.
Appendix B.3
and rigorously evaluate BA-LoRA under the identical recipe. These baselines include
Full Fine-tuning. LoRA (Hu et al. 2021). AdaLoRA (Zhang et al. 2023b). DoRA (Liu et al. 2024
MiLoRA (Wang et al. 2024a), LoRA+ (Hayou et al. 2024), LoRA- FA (Zhang et al. 2023a). LoRA-
GA /Wang et 2004b)) PiSSA (Meng et 2024)) CorDA (Yang et al. (0024), and CorDA+ (Yang
ct al. (025). Following the standard literature setting of utilizing 100K data points and a single epoch for efficiency,
fine-tune I.LaMA-2-7B (Touvron et al. 2023) equipped with BA-LoRA
MetaMathQA (Yu et al. (023) and assess mathematical
mroblem-solving capabilities (Luo et al.
2025) using the GSMSK (Cobbe al. [021] and MATH (Yu al. (2023) validation sets, reporting Accuracy. Similarly, for coding tasks, our model is fine-tuned on CodeFeedback (Zheng et al. 2024
and evaluated via HumanEval (Chen al. 2021) and MBPP (Austin et al.) [02T]) reporting PASS@
metrics. To assess conversational abilities, BA-LoRA is trained on MevurdL--Evoo-Inseeeet
(Xu
al. 2024) and evaluated on MT-Bench (Zheng al. 2024). with response quality judged by GPT-4 and first-turn scores reported.
As shown in Table T BA-LORA achieves the best performance among the compared methods on LLaMA-2-7 B. Specifically, compared to the highly competitive CorDA++, BA-LoRA further enhances performance on the reasoning task GSMSK by 0.83 points and the coding task HumanEval by 1.82 points. While CorDA+ maintains an edge on MT-Bench, BA-LORA's substantial gains on other benchmarks lead to a superior average score, achieving a 1-96-point uplift over CorDA++, with the largest margin MBPP, small natural language-te to to-eedeett--oee benchmark with limited test coverage and susceptibility to overfitting and spurious patterns, where BA-LoRA's output-space regularization is helpful under noisy supervision. This performance improvement is further corroborated by the

---

Published as a conference paper at ICLR 2026
Table 2: Performance Comparison on NLU Benchmarks. We compare BA-LoRA with various PEFT baselines on the DeBERTa-v3-base model. The best result in each column is highlighted in bold.
Methods
#Params
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg
Full FT
184M
26.33 00,34_0.38
89.95 2..95
21.43
94,24_.0.00
92. 2
83.
63.75
91 91.04zo.s
88.65
BiFF
0.1M
89.54_0.3
94.68
1.
17.31
92.45_.
58.72
79.12.0.3
91.63 10:37
86,43
HAdapter
1.22M
90.23 u007
95.38 15.38
89.97
68.73
-4.3
91 11.99.s.
84.76gu.2
91.58 91.58
88.37
PAdapter
1.18M
90.42-00.0
95. 10.
89.71 20.55
69.04.0.10
94,38_0.20
92. 15
85,53
91 01.69
88.55
LoRA
1.33M
u.366
94.79 10:10
10.25
70.05 10.31
93.94_
12.07
85.43.u.cm
91.67 1.020
DoRA
1.27M
90.48.00.10
95 25.00
91.0420.15
71.03
94.21
92 02.34.0.1s
86.192n.25
86.19
91. 21.92
89.13
AdaLoRA
1.27M
90.87 0.08
16.188.c. 16.43
90.81
71.6410.11
94.68,
92.37.10.3
87.78
91.97 10:43
89.54
PISSA
1.33M
90.47:00.4
95.8 a0.41
91 21.48.1.40
72.27.a0.2
94,41-.0.41
92.21aa.25
87.14
91 01.93
89.47
BA-LORA
1.33M
91.26_00.0
96.25+c.0>
92.11
75.4660000
95.35-0.14
93.6340.02
88.58
92.71 92.71 40.35
90.67
model's optimization dynamics. As illustrated in 2 BA-LoRA also demonstrates superior training efficiency on the MetaMathQA dataset. Across both high (r 128) and low (r 32) ranks, our method achieves a lower final training task loss than LoRA and PiSSA, reaching levels comparable Full Fine-tuning, which we attribute our principled regularization scheme guiding the optimization toward a more favorable solution space. An extensive comparison across diverse model families and scales, including dense and MoE models from 7B to LLaMA-3-70B, is provided in Appendix C.3 (Figure
To assess BA-LoRA on NLU tasks, we experimented on the GLUE benchmark (Wang et al., 2018). which includes two single single-sentence classification tasks (CoLA, SST), five paired-text classification tasks (MNLI, RTE, QQP, MRPC, QNLD), and one text similarity prediction task (STS-B). The evaluation metrics comprise the accuracy for MNLI (strictly following the evaluation protocol of our baselines), the Matthews correlation coefficient for CoLA, the Pearson correlation coefficient for STS-B, and the accuracy for the remaining tasks. We used the DcBERTa-v3-basz model (He et al.) 202T) and compared BA-LoRA against eight strong baseline methods, including Full Fine-tuning FFul BitFit (Zaken et al. 2021 HAdapter (Houlsby et al. 2019), PAdapter (Preiffer et al. 2020). LoRA (Hu et al. 2021), DoRA (Liu et al. 2024), AdaLoRA (Zhang et al. 2023b). and PISSA
Meeng
al. 2024).
Table 2 presents the results of DeBERTa-v3-base across eight NLU tasks, demonstrating the strong overall performance of BA-LORA. It surpasses all parameter-efficient fine-tuning (PEFT) baselines across all tasks and achieves the highest average score. On average, BA-LoRA outperforms PiSSA and LoRA by 1.20 and 2.11 points, respectively. The consistent, broad-based improvements across this diverse suite of both NLG and NLU tasks provide strong evidence that our principled, three- pronged strategy not only mitigates the fundamental failure modes associated with Catastrophic Inheritance but also offers practical route to strong and competitive parameter-efficient adaptation.
3.2.2 MITIGATING THE EFFECTS OF NOISY PRE-TRAINING DATA
Given that large-scale pre-training corpora from web crawls are inherently noisy (Gao et al. 2020 Dodge et al. 2021), a critical challenge is ensuring that fine-tuning enhances the core signal rather than inherited noise. To investigate BA-LoRA's ability to address this, we conduct a comparative study on models pre-trained on corpora of distinct quality. Our testbeds are RoBERTa- base (Liu eet al. 2019). pre-trained on a curated mixed corpus, and T5-base (Raffel et al. 2020), pre-trained on the large-scale C4 web corpus While these models differ in architecture, their distinct pre-training corpora provide an ideal but not fully controlled testbed for evaluating robustness against inherited noise. We evaluate on a representative subset of the GLUE benchmark.
As detailed in Table 3 baselines. The central finding is that the advantage of BA-LORA is substantially more pronounced on the model pre-traine pre-trained on noisier data. While BA-LoRA establishes a solid 1. 1-pint average improvement over the strongest baseline (PiSSA) on the cleanly trained RoBERTa-base (86.34 vs. 85.23). this performance gain nearly triples to a substantial 3.26 points on the base (87.97 vs. 84.71). The pronounced disparity in improvement margin (ATs = 3.26 vs. AROBERT = 1.11) is
BA-LORA achieves the best average performance against strong PEFT
'Specifically, RoBERTa is trained on ama1160BB mixed corpus including BooksCorpus and English Wikipedia alongside web text. In contrast, the C4 corpus (~750GB) used for T5 is derived from the Common Crawl web scrape via heuristic filtering, making more representative testhed for web-scale noise.
7

---

Published as a conference paper at ICLR 2026
Table 3: Performance comparison of our method (BA-LoRA) against PEFT baselines (LoRA, PISSA) on RoBERTa-bane and T5-base. Models are evaluated on subset of the GLUE benchmark. The best result for each model in bold.
Model
Methods
MNLI
SST-2
CoLA
QNLI
MRPC
Avg
LoRA
85.63
94.03
62.40
91.37
87.98_0.23
84.28
RoBERTa-base
PiSSA
85.72
93.644...1
67.28
91.40
88. 88.11 20.24
85.23
BA-LoRA
86.59 +0.58
94.83.20.45
67.91 +0.21
92.28
90.07 =0.32
86.34
LoRA
85.30
94.0440.11
69.35 4.005
92.96
68.38
82.01
15-base
PISSA
85.75
94.07 1007:00.0 20.06
74.27 +0.39
93.15 20.14
76.3 =0.51
84.71
BA-LORA
86.91 +0.48
95.20+0..2
11.03
94.12 00.32
83.43 0.771
87.97
Spats
SuRaseta Sanrs 3322
Sdairatte
.ONN DiSLAAAAII
HONES HONES Disumnbin
SONEC
Nestrd
Necrel
AS Timetiion
+GNB Dinwasirn
HSNE Timerdon
(a) LoRA, Balanced
(b) PISSA, Balanced
(c) BA-LoRA, Balanced
Stam
SNNS DIIHENAA
12
LONE LONE,
Neatral
Nord
Slmetatenn
66NE Dimwexton
HNNC Dimerttes
(d) LoRA, Imbalanced
(e) PiSSA, Imbalanced
(I) BA-LoRA, Imbalanced
Figure 3: t-SNE visualizations of features from RoBERTa- base fine-tuned with LoRA, PiSSA, and BA-LoRA the MNLI task under balanced (top) and imbalanced (bottom) settings.
consistent with our hypothesis that BA-LoRA is particularly beneficial for models pre-trained on noisier web corpora, though does not isolate architectural factors.
3.2.3 MITIGATING REPRESENTATIONAL BIAS FROM DATA IMBALANCE
This experiment qualitatively investigates BA-LoRA's capacity to counteract the representational degradation caused by data imbalance, which one way Catastrophic Inheritance can manifest during downstream fine-tuning. We visualize the final hidden-layer feature representations from RoBERTa-base fine-tuned the MNLI task using t-SNE (Van der Maaten & Hinton| 2008). As shown in Figure 3 we compare feature manifolds learned on the standard balanced dataset against those from deliberately imbalanced ceeriinn-oosstuuttee by subsampling the training data 100:10:1 ratio for the 'Entailment', 'Neutral', and "Contradiction' classes. This controlled comparison designed to simulate the challenge of learning from highly skewed data distributions on top of pre-trained model, where Catastrophic Inheritance-style failures arise
The visualization contrasts the methods' resilience to data imbalance. While representations from baseline LoRA and PiSSA suffer degradation and class overlap (Figure (dee). BA-LoRA maintains a well-separated manifold (Figure 3 f): see Table 12 in Appendix Co quantitative verification). This visualization is consistent with the effectiveness of our diversity regularizer (CDR) in preventing feature degradation from skewed data distributions. This effect is reinforced by the consistency (LCR) and SVD (LSVdR) regularizers, which together help the representations remain distinct and robust.
8

---

Published as a conference paper at ICLR 2026
Table 4: Ablation study of BA-LoRA regularizations on GSM8K, MATH, and NLU tasks. Results on GSM8K and MATH are from LLaMA-2-7B, while the NLU task refers to the average GLUE score from DeBERTa-v3-ber
"Baseline" (PiSSA) is fine-tuned without our proposed regularizations. +CCR, +LDR. and +LSVDR denote adding only single corresponding regularization to the baseline. "BA-LoRA (Full)" the full model using all regularizations.
Configuration
GSM8K
MATH
Average of GLUE
Baseline (PiSSA)
51.48 + 0.34
7.60 + 0.18
89.47
+ LCR
54.25 + 0.59
9.15 - 0.25
90.18
+ LDR
53.60 + 0.46
8.95 + 0.18
89.85
+ LSVDR
52.95 = 0.55
8,70 0.22
89.71
BA-LORA (Full)
55.86 + 0.35
9.47 + 0.52
90.67
3.2.4
ABLATION STUDY
Our ablation study (Table empirically supports our principled deconstruction of Catastrophic Inheritance. On NLG tasks with the LLaMA-2-7B model, we observe a consistent pattern: each regularizer yields a positive contribution over the baseline ('Baseline'). Specifically, gains from the consistency regularizer (LCR) support its role in combating Knowledge Drift by preserving foundational knowledge. Similarly, improvements from the diversity regularizer (CDR) highlight the importance of preventing Representation Collapse, and the significant contribution from the SVD regularizer (CşvDR) confirms the benefit of mitigating Overfitting to Noise.
This trend is mirrored in NLU tasks, where the DeBERTa-v3-barr model also shows clear uplift with each regularizer over the baseline. The full BA-LoRA model, which synergistically combines all three components, consistently achieves the highest performance across all evaluated settings. In summary, these results provide strong evidence that Knowledge Drift, Representation Collapse, and Overfitting to Noise important and complementary failure modes in fine-tuning. Consequently, our integrated, multi- pronged solution yields strong generalization and robustness across both NLU and NLG domains. The selection of our regularization coefficients (A₁, λg. A3) is validated by detailed sensitivity analysis in Figure 4 and Table [0] further shows that the same regularization framework consistently improves multiple LoRA-style methods (LoRA, DoRA, PiSSA).
3.2.5 COMPUTATIONAL COST ANALYSIS
To quantitatively evaluate the compu- tational efficiency and performance of
Table 5: Computational Cost and Performance Compari- son. Costs are measured on two A40 GPUs for fine-tuning LLaMA-2-7B. Memory cost reports the aggregated peak GPU memory across the two A40 GPUs. Full FT exceeds the available memory under this setup (OOM); thus, we re- port lower bounds (>96GB, 224h).
our method, we conducted a compar- ative experiment two A40 (48GB)
GPUs using DpepSpeed (Rasley et al.. 2020) ZeRO-2 optimization: Full FT is OOM under x A40 ZeRO-2, so we report lower bounds for its mem- ory cost and training time. We fine- tuned the LLaMA-2-7B model on the first 100,000 entries of the Meta- MathQA dataset. This experiment benchmarked four methods: full fine- tuning (Full FT), LORA, PiSSA, and proposed BA-LoRA. For each method, we measured the peak GPU memory consumption and the total training time to assess computational cost. Model performance
Method
Memory Cost
Training Time
GSMSK
Full FT
>96 GB
>24h
48,9 + 0.49
LoRA PISSA
66.32 GB
4h 31min
42.68 0.54
66.59 GB
4h 17min
51.48 0.34
BA-LORA
77.34 GB
4h 48min
55.86 0.35
subsequently evaluated on the GSMSK benchmark.
The results in Table 5 quantify the performance-com trade-offs of various methods. BA-LoRA achieves the highest GSM8K score among the compared methods (55.86), significantly outperforming all baselines. This substantial performance gain is achieved with a modest overhead compared to PiSSA (+10..5 GB memory, +-1 min training), highlighting compelling performance-cost balance.

---

Published as a conference paper at ICLR 2026
4 RELATED WORK
Our work bridges two critical research areas.
In Cerammter-Efficienn Fine-Tuning (PEFT), our method builds upon Low-Rank Adaptation (LoRA) (Hu et al. 2021). Numerous LoRA A variants have focused on enhancing performance and efficiency, such as QLoRA (Dettmers al. 2024) and PiSSA (Meng et al., 2024), or addressing activation memory bottlenecks, such as LoRA FA (Zhang et al., (023a). However, empirical analyses reveal that the constrained low-rank updates can impact the forgetting boggeting-plastictyy trade-off and interfere with pre-trained knowledge (Magistri et al. 2024). Crucially, the systematic mitigation of inherited biases within this restricted parameter space remains a significant gap. Beyond classical PEFT, several recent works explicitly study forgetting and knowledge drift under
adaptation. For instance, Smith et al. introduce C-LoRA for continual customization text-to-image diffusion models, mitigating drift by constraining adapter updates (Smith et al. 2023). Similarly, Chen and Garner propose Bayesian PEFT to reduce catastrophic forgetting via priors on adapters (Chen & Gamer 2024). However, these approaches mainly operate in parameter space and are evaluated on diffusion or continual-learning settings. In contrast, BA-LoRA targets Catastrophic Inheritance in language models (Chen et al. 2023 2025) and uses three output-space regularizers (consistency, diversity, and SVD-based) that are instantiated in a unified way for both NLU and NLG tasks.
In Bias Mitigation, addressing biases from web-scale corpora is foundational concern (Bender et al. 2021). While a rich literature exists on data filtering (Dodge et al., [021] and algorithmic adjustments for full fine-tuning- such representation debiasing (Ravtogel et al. 2020) and decod- ing strategies (Sheng et al., 2019) (see (Gallegos et al. 2024) for survey) these are not directly applicable PEFT. Our design is also related to work that analyzes and mitigates label noise large- scale pre-training. Chen et al. propose feature-space framework combining consistency, covariance, and
regularization to improve robustness to noisy labels (Chen et al. 2023) 2025). Their regularizers act intermediate representations, whereas BA-LoRA applies analogous ideas directly to the output logits of PEFT-adapted models. More broadly, our diversity regularizer is conceptually aligned with redundancy -reeucciinn objectives such Barlow Twins (Zbontar et al.) 2021) and VICReg (Bardes et al.) 2021), which encourage high variance and low cross- covariance to avoid representation collapse. Although recent analyses explore LoRA optimization dynamics, such as mitigating double descent (Chang et al. 2025). and fairness issues within PEFT (Ding et al. (202). BA-LoRA is, our knowledge, the first propose concrete, multi-faceted algorithmic framework. It moves beyond analysis to systematically mitigate the broader problem of Catastrophic Inheritance by integrating a principled regularization scheme directly into the LoRA-based process.
CONCLUSION
This paper introduces BA-LoRA, a novel parameter-officient fine-tuning framework aimed at mitigat- ing Catastrophic Inheritance. Our core contribution is principled approach that decomposes this challenge into three tibb-roolles-- Knowledge Drift, Representation Collapse, and Overfitting to Noise Noise-and addresses them with three targeted regularizers. Extensive experiments across NLG and NLU benchmarks support our integrated strategy, which achieves strong performance and improves robustness to inherited data biases relative to standard LoRA variants. By addressing Catastrophic Inheritance explicitly within the LoRA framework, BA-LoRA offers a more reliable pathway to adapt pre-trained models to downstream tasks where robustness and fairness are important.
ACKNOWLEDGMENTS
This work is supported by the National Key Research and Development Program of China the National Natural Science Foundation of China (No.U2341229) and the Reform Commission Foundation of Jilin Province (No.2024C003).
10

---

Published as a conference paper at ICLR 2026
REFERENCES
@ Meta. Ilama model card. 2024. URt.https:///gttabb..ooo/////eee /
blob/ma ma weraii/Monn L CARD md
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07733 2021.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang. Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang. Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 166099 2023.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-iivariancc--vvaanen reg.aaaaaiainneeuurgririiii regularization for self-supervised learning. arXiv preprint 1.Xiv:2:05.049000 2021.
Solon Barocas and Andrew D Selbst. Big data's disparate impact. Calif. L. Rev., 104:671, 2016.
Emily M Bender, Timnit Gebru, Angelina McMillan-Majje and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pp. 610-623, 2021.
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1536-1546. IEEE, 2021.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.07258 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. arXiv preprint
arXiv:2202.076646, 2022.
Nicholas Carlini, Matthew Jagielski, Christopher A Choquette-Choo, Daniel Paleka, Will Pearce, Hyrum Anderson, Andreas Terzis, Kurt Thomas, and Florian Tramèr. Poisoning web-scale training datasets practical. arXiv preprint 10X112230..00149 2023.
Isaac Caswell, Julia Kreutzer, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orsee Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, et al. Quality at a glance: An audit of web-crawled multilingual datasets. arXiv e-prints, pp. arXiv-2103, 2021.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijic Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM Transactions Intelligent Systems and Technology, 15(3):1 -45 2024.
Yupeng Chang, Chenlu Guo, Yi Chang, and Yuan Wu. Lora-mgpo: Mitigating double descent in low- rank adaptation via momentum-guides perturbation optimization. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 648-659, 2025.
Hao Chen, Jindong Wang, Ankit Shah, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, and Bhiksha Raj. Understanding and mitigating the label noise in pre-training on downstream tasks. arXiv preprint 2rXiv: 17002, 2023.
Hao Chen, Bhiksha Raj, Xing Xie, and Jindong Wang. On catastrophic inheritance of large foundation models. arXiv preprint M0Xi::2402.01900 2024.
Hao Chen. Zihan Wang, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, Bhiksha Raj. and Jindong Wang. Impact of noisy supervision in foundation model learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
11

---

Published as a conference paper at ICLR 2026
Haolin Chen and Philip N Garner. Bayesian parameter-efficien fine-tuning for overcoming catas- trophic forgetting. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Horbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021.
Xinyang Chen, Sinan Wang, Mingsheng Long, and Jianmin Wang. Transferability vs. discriminability: Batch spectral penalization for adversarial domain adaptation. In International conference on machine learming, pp. 1081-1090. PMLR, 2019.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2110.14168, 2021.
Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint arXiv:2401.0660 406666666666..... 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lims. Advances in Neural Information Processing Systems, 36, 2024.
Zhoujie Ding, Ken Ziyu Liu, Pura Peetathawatchai, Berivan Isik, and Sanmi Koyejo. On fairness of low-rank adaptation of large models, 2024. httptttpp//......///////////////
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint (0ti:::2144.0777 2021.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang. Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint wr.iv:2310005492 2023.
Yanai Elazar, Akshita Bhagia, Ian Magnusson, Abhilasha Ravichander, Dustin Schwenk, Alane Suhr, Pete Walsh, Dirk Groeneveld, Luca Soldaini, Sameer Singh, al. What's in my big data? arXiv preprint arXiv:2310.20707 2023.
Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7382-7392, 2024.
Benoît Frénay and Michel Verleysen. Classification in the presence of label noise: survey. IEEE Iransactions on neural networks and learning systems, 25(5):845-869 2013.
Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dermon- court, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3): 1097-117,, 2024.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
12

---

Published as a conference paper at ICLR 2026
Itai Gat, Idan Schwartz, Alexander Schwing, and Tamir Hazan. Removing bias in multi-modal classifiers: Regularization by maximizing functional entropies. Advances in Neural Information Processing Systems, 33:3 197-3299,, 2020.
Nathan Halko, Per-Gunnar Martinsson, and Joel A Tropp. Finding structure with randomness: Probabiliisicc algorithms for constructing approximate matrix decompositions, SIAM review, 53(2): 217-288, 2011.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficien fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprini 22X1::2402./2354, 2024.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with drndrent-disentangled embedding sharing, 2021.
Danny Hernandez, Tom Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dede Dodds, Tom Henighan, Tristan Hume, et al. Scaling laws and interpretability of learning from repeated data. arXiv preprint arXiv:2205. 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.0255: 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning, pp. 2790-2799. PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825. 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.040000 2024.
Jiale Kang and Qingyu Yin. Balancing lora performance and efficiency with simple shard sharing, 2025. URL httlt:tts://artt wttp://..rrr/a//////000
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska-Barr Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-355 --2226211.56, 2017.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efffcient prompt tuning, 2021.
Shih-Yan Yang Liu, Chien-Yo Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint 0033:32402.09333.. 2024.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907. 11692, 2019.
Zhuang Liu and Kaiming He. A decade's 's battle on dataset bias: Are we there yet? arXiv preprint arXiv:2403.08632, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 171.0 2017.
13

---

Published as a conference paper at ICLR 2026
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Haipeng Luo, Huawen Feng, Qingfeng Sun, Can Xu, Kai Zheng, Yufei Wang, Tao Yang, Han Hu, Yansong Tang, and Di Wang. Agentmath: Empowering mathematical reasoning for large language models via tool-augmented agent. arXiv preprint arXi::2512.2074 2025.
Simone Magistri, Joost van de Weijer, Andew D Bagdanov, et al. An empirical analysis of forgetting in pre-trained models with incremental low-rank updates. arXiv preprint arXiv:2405.1800 2024.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. arXiv preprint arXiv:2212.10511, 2022..
2022.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1609. 07843,, 2016.
Curtis G. Northeutt, Lu Jiang, and Isaac L. Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research (JAIR),
2021.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, and Shu Kong. The neglected tails in vision vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12988-12997) 2024.
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon IIm: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2006.0//66 2023.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.0024: 002447 2020.
Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin- Yu Chen, Ruoxi Jia, Prateck Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint 230.2330 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System opti- mizations enable training deep learning models with over 100 billion parameters. In SIGKDD, pp. 3505-3506, 2020.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. Null it out: Guarding protected attributes by iterative nullspace projection. arXiv preprint arXiv:2004.07666 2020.
Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley, Data contamination through the lens of time. arXiv preprint arXiv:2310.1000 2023.
Rylan Schaeffer. Pretraining on the test set is all you need. arXiv preprint arXiv:2309.08633 2023.
Emily Sheng, Kai- Wei Chang, Premkumar Natarajan, and Nanyun Peng. The woman worked as a babysitter: On biases in language generation. arXiv preprint
2019.
14

---

Published as a conference paper at ICLR 2026
James Seale Smith, Yen-Chang Hsu, Lingyu Zhang, Ting Hua, Zsolt Kira, Yilin Shen, and Hongxia Jin. Continual diffusion: Continual customization text-to-ine to-image diffusion with c-lora. arXiv preprint ar arXiv:2304.06022 2023.
Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gi Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE transactions on neural networks and learning systems, 34(1 1) 1):8135 18815-8153, 2022.
Lichao Sun, Yue Huang, Haoran Wang, Siyuan Wu, Qihui Zhang, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li. et al. Trustllm: Trustworthiness in large language models. arXiv preprint ar Xrii:2400.0055 100561, 2024.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. International Conference Learning Representations, 2018.
Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficient Ilm finetuning. arXiv preprint 49X4422406.09044, 2024a.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. Advances in Neural Information Processing Systems, 37:54905-54931, 2024b.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
Hu Xu, Saining Xie, Xiaoqing Ellen Tan, Po- Yao Huang, Russell Howes, Vasu Sharma, Shang- Shang-Wen Wen Li, Gargi Ghosh, Luke Zettlemoyer, and Christoph Feichtenhofer. Demystifying clip data. arXiv preprint arXiv:2309. /6671, 2023.
Yibo Yang. Xiaojie Li, Zhongzhu Zhou, Shuaiwen Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task- aware parfieter-efffcient fine-tuning. Advances in Neural Information Processing Systems, 37: 71768-71791, 2024.
Yibo Yang, Sihao Liu, Chuan Rao, Bang An, Tiancheng Shen, Philip HS Torr, Ming- Hssun Yang, and Bernard Ghanem. Dynamic cortext- decomposition for task-aware low-rank adaptation with less forgetting and faster convergence. arXiv preprint arXiv:2506. 13187, 2025.
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. Yi: Open foundation models by 01, ai. arXiv preprint arXiv:2403. 240320652, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. urXiv preprint arXiv:2309. 12284, 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-baser masked language-models. arXiv preprint 10X.v:2106.00/99 2021.
15

---

Published as a conference paper at ICLR 2026
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International conference on machine learning, pp. 12310- 12320. PMLR, 2021.
Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprini arXiv:2308. 03300, 2023a.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficiet fine- tuning. arXiv preprint urXiv:2303. 10512, 2023b.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 0eet22333..2222 18223, 2023.
Lianmin Zheng, Wii Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et Judging IIm-as aun-ae-a-uudge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Beier Zhu, Kaihua Tang, Qianru Sun, and Hanwang Zhang. Generalized logit adjustment: Calibrating fine-tuned models by removing label bias in foundation models. Advances in Neural Information Processing Systems, 36, 2024.
Andy Zou, Zifan Wang. J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint urXiv2230..10043 2023.
16

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
Background
17
A.1 Challenges of Bias and Noise in Pre-training Data
17
Mitigating Bias through Earameter-Efficiennt Fine- Tuning
18
A.3
Typologies of Noise in Pre-training Data
18
Experimental Setup
18
Models
19
B.2
Tasks, Datasets, and Metrics
B.3
Implementation and Training Details
19
B.4
Hyperparameter Settings
20
More Experiments
21
Generality of the Regularization Framework
21
C.2
Hyperparameter Sensitivity Analysis
22
Analysis Across Diverse Model Architectures and Scales
23
Performance Analysis Across Different Ranks
23
Comparison of Regularization Objectives for NLU
24
C.6
Quantitative Analysis of Cluster Quality and Minority Representation
24
Direct Evaluation of Pre-trained Capabilities)
24
C88
Asymptotic Complexity of Regularizers
25
D
More Discussions
25
D.I
Practical Hyperparameter Guidelines
25
D.2
Additional Discussion on RoBERTa vs. T3
26
D.3
Discussion on the Choice of Regularization Targets
26
D.4 Conceptual Foundations and Synergy of Regularizers
26
D.5
On Applying Representation Learning Principles during Fine- Tuning
27
D.6
Limitations and Future Work
27
D.7 Ethies Statement
27
D.8
Reproducibility
28
A BACKGROUND
A.1 CHALLENGES OF BIAS AND NOISE IN PRE-TRAINING DATA
Bias and noise within pre-training datasets present significant hurdles in constructing dependable machine-learning models. Mislabeled data and imbalanced distributions lead to models that
17

---

Published as a conference paper at ICLR 2026
not only underperform downstream tasks but also reinforce existing biases (Barocas & Selbst 2016; [Gallegos et al.] 2024). This issue especially problematic in large-scale datasets where manual curation impractical. and reliance inaccuracies (Northcutt et al. 2021; Birhane & Prabhu 2021). Consequently, models trained on such data risk not only poor general generalization lzzatin but also the inheritance of these data-induced flaws, which can be amplified during adaptation to downstream tasks (Frénay & Verleysen [013] Song et al.) 2022). A critical goal of fine-tuning therefore learn new capabilities while mitigating the effects of this "Catastrophic Inheritance".
automated data collection may introduce various
A.2 MITIGATING BIAS THROUGH PARAMETER- EFFICIENT FINE-TUNING
Parameter-efficient fine-tuning (PEFT) methods offer promising foundation for mitigating Catas- trophic Inheritance. By design, adapting models with minimal parameter updates can theoretically limit overfitting to inherited noise and help preserve foundational knowledge (Houlsby al. [019]]
Zaken al., 202 | Lester et al.) 2021). However, as we argue in the main paper, this promise is not
fully realized practice. Techniques like low-rank adaptations (LoRA) (Hu et al.) 202T) introduce their own inductive biases, such as the low-rank bottleneck, which can inadvertently exacerbate the very issues they are meant to solve by amplifying spurious correlations. This critical gap motivates the development of more principled, explicit regularization techniques like those proposed in our work- that are tailored to the unique challenges the PEFT paradigm.
A.3 TYPOLOGIES OF NOISE IN PRE-TRAINING DATA
The vast web-scale corpora used train modern language models, such LLaMA-2 (Tuvrro et al. 2023) and GPT-4 (OpenAT] 2023). inevitably contain significant noise and distributional biases. The sheer scale of these datasets makes comprehensive manual curation impractical, meaning models often exposed to duplicated, corrupted, or irrelevant information during pre-training (Elazar et al.) 2023 Birhane & Prabhu 202T). When fine-tuned, these models can struggle to distinguish signal from noise, which in turn degrades downstream performance. Understanding the specific typologies of this data-induced noise is therefore crucial for developing more robust models. We categorize the primary challenges as follows.
Low-Qualit, Data This category stems from the uncurated nature of web data. A key issue is data duplication, where near-identical content can lead model overfitting and privacy leakage risks (Carlini et al., 2022; Hernandez et al. 2022). Another challenge is data corruption, where inconsistent or erroneous inputs degrade model robustness and performance (Fan et al., 2024 Caswell et al. 2021 Furthermore, test set contamination, the leakage of evaluation data into the training corpus, can lead inflated performance metrics and invalidate a model's 's evaluation (Roberts et al. 2023 Schaeffer 2023).
Distributional Skew This form of bias arises from non-uniform data distributions. The most common form is category imbalance, where an underrepresentaaion certain topics or classes causes the model to perform poorly those categories, leading to biased or unreliable outputs (Xu ct al. 2023
Zhu et al. 2024 Parashar et al. 2024).
Unsafe and Unethical Content Finally, web corpora often contain undesirable content. The presence of toxic and harmful text, including offeensive, biased, or malicious content, cause the model to generate inappropriate or harmful outputs, posing significant safety and ethical risks (Zou
al. 2023 Sun al. 2024).
B EXPERIMENTAL SETUP
To rigorously evaluate proposed method, conduct comprehensive set of experiments suite of Natural Language Generation (NLG) and Natural Language Understanding (NLU) tasks. Our experimental design, including models, datasets, and training configurations, is detailed below.
18

---

Published as a conference paper at ICLR 2026
Table 6: Comparison of pre-trained data and methods for various language models.
Model
Pre-trained Data
Pre-training Method
DeB -3-hase (He et 2021
Wikipedia, BooksCorpus OpenWeb OC-News, Stories
Replaced Token Detection with GDES
RoBERTh.bsee (Lid
BooksCorpus, English Wikiipedia, CC News. Wpen Stories Colossal Clean Crwwted Conps (C4)
Masked Language Modeling
15-base Rattel 2010
Text-to- Text Denoising Objective
Table Evaluation metrics for the NLG datasets.
Datasets
GSM8K
MATH
HumanEval
MBPP
MT-Bench
Metric
Accuracy
Accuracy
Pass
Pass
GPT-4 Evaluation
B.1 MODELS
Our evaluation leverages a wide array of pre-trained language models to ensure a comprehensive assessment. For NLG tasks, we primarily utilize large language models renowned for their generative capabilities, including LLaMA-2 (7B, 13B) (Touvron et al. 2023), LLaMA-3 (8B, 70B) (AT@ Metal
2024). Mistral-7B-v0.1 (Jiang et al. 2023), -xxttral-877--v (Jiang al. 2024), Gemma- 7B (Team
et al. 2024). Qwen-1.5-775 7B Bai et al. 2003). Yi-1,5-34B (Young et al. 2024), and the Mixture-of-
Experts model DeepSeek-MoF--Il
(Dai al. 2024).
For NLU tasks, our experiments employ several key models to investigate different aspects of performance. Our main fine-tuning experiments on the GLUE benchmark utilize DeBERTa-v3- base (He et al.) (202I). For the controlled study on pre-training data noise, we specifically select RoBERTa-base (Liu et al., 2019) and T5-base (Raffel al. 2020) due to their distinct corpus characteristics.
detailed overview of the primary NLU models presented Table 6 These models provide robust foundation for study due to their diverse pre-training methodologies. For instance, RoBERTa-base was pre-trained on a high-quality mixed corpus, whereas T5-base was pre-trained on the large-scale and noisier C4 web corpus. DeBERTa-v3-barr utilized another diverse dataset with replaced token detection objective. This architectural and methodological diversity is crucial for a thorough evaluation of our approach.
B.2 TASKS, DATASETS, AND METRICS
Natural Language Generation (NLG) For NLG, we assess model capabilities across mathematical reasoning, code generation, and instruction following. The benchmarks include GSM8K (Cobbe et al. 2021). MATH (Yu et al. (023). HumanEval (Chen et al. 202T), MBPP (Austin et al. (021). and MT-Bench (Zheng et al. 2024). As summarized in Table evaluation metrics are task-specific: Accuracy for GSM8K and MATH, Pass@1 I for HumanEval and MBPP, and GPT--4based evaluation for MT-Bench.
Natural Language Understanding (NLU) For our NLU evaluation, we utilized the GLUE bench- mark (Wang et 2018), which comprises diverse set of tasks. These tasks can be categorized into three groups: two single-sentence classification tasks (CoLA, SST-2), five pairwise text classification tasks (MNLI, RTE, QQP, MRPC, and QNLI), and one text similarity prediction task (STS-B) Fol- lowing the standard evaluation protocol. we report the Matthews correlation coefficient for CoLA, Pearson correlation for STS-B, and accuracy for the remaining tasks. For MNLI specifically, we report both matched and mismatched accuracy.
B.3 IMPLEMENTATION AND TRAINING DETAILS
Baseline Comparison For a fair and direct comparison, all baseline results presented our main experiments are directly obtained from their original publications. Specifically, the NLG baseline results in Table I are sourced from the comprehensive study by Yang et al. (2025). For the NLU benchmarks, the results for DeBERTa-v3-base in Table 2 are taken from Kang & Yin (2025). For our BA-LoRA runs, we follow the same core fine-tuning configuration (model, dataset splits, optimizer, learning-rate schedule, batch size, and LoRA rank/placements) in these works, and only
19

---

Published as a conference paper at ICLR 2026
introduce our additional regularization terms with coefficients chosen described in Section 3.1 and Appendix B B.4, to ensure a controlled and equitable evaluation.
For all GLUE experiments, the consistency and diversity regularizers are computed on the fine-tuning data by passing inputs through the pretrained backbone and the BA-LoRA AA-LoRA--daaped backbone, using the same shared classification head, generate teacher and student logits, respectively.
MiLoRA vs. BA-LORA. MiLoRA proposes a spectral variant of LoRA that explicitly exploits
minor singular components of pretrained weight matrices: instead of parameterizing adapters along the top singular directions (as in PiSSA), MiLoRA allocates capacity to lower-energy directions in weight space, arguing that these under-utilized components can be effective for adaptation while potentially reducing interference with core pretrained knowledge. In contrast, BA-LoRA keeps the underlying LoRA-style parameterization (e.g., standard LoRA, PiSSA, DoRA) and operates entirely in output space: it introduces three regularizers on the logits a consistency term to control knowledge drift, diversity term to avoid representation collapse, and a spectral SVD term suppress noisy high-freyuency components. Thus, MiLoRA primarily changes which spectral directions in weight space are used to represent the adapters, whereas BA-LoRA constrains how the adapted model behaves through output-space regularization. These two perspectives are complementary and could, in principle, be combined.
Data Preprocessing for Visualization To analyze the model's feature space under data imbalance, constructed custom imbalanced version of the MNLI training dataset. This process began by separating the full training set into three subsets based on their labels. We then retained all samples from the entai lment class (100%), while randomly downsampling the neutral class to 10% and the cont radi ct on class to 1% of their original sizes. Finally, these three subsets were concatenated and shuffled to form the training set for the visualization model, thereby simulating scenario with a highly skewed label distribution.
t-SNE Visualization For the t-SNE visualization, we fine-tuned a RoBERTa-base model for 3 epochs on the imbalanced MNLI training set described above. Subsequently, we extracted the [CLS] token representations from the final hidden layer for all samples in the original, balanced MNLI validation set. These high-dimensional features were projected into two dimensions using the t-SNE algorithm with perplexity of 30, 1000 iterations, and fixed random seed (42) for reproducibility. The quality of the resulting clusters was also quantitatively assessed using the silhouette score with a cosine distance metric.
Evaluation Frameworks For evaluation, we employed publicly available frameworks. The model's code generation capabilities were assessed using datasets like HumanEval and MBPP through the BigCode Evaluation Harness Instruction -folloiinn performance was evaluated using MT-Bench
B.4 HYPERPARAMETER SETTINGS
NLG (LLaMA-2-7B) Our Natural Language Generation (NLG) experiments involved fine-tuning the LLaMA-2-78 model on a 100,000-sample subset of the MetaMath dataset. The model was trained for a single epoch using BFloat16 (bf16) precision, maximum sequence length of 512, and an effective batch size of 32, achieved with a per-device batch size of 4 and 4 gradient accumulation steps. For optimization, we employed the AdamW optimizer with learning rate of 10-5, no weight decay, and a cosine learning rate schedule with a 3% warm-up phase. The base LoRA configuration featured a rank (r) of 128, an alpha (a) of 128, and no dropout, with adapters applied comprehensively jrojj, jppjr and down_prroj layers.
to the q p_proj_ j, k proj, v_ v_rroj. o p_proj, For our proposed BA-LoRA method, we set the regularization coefficients to X = 0.025, X 0.005, and A - 0.005. The primary coefficient, \1, also followed cosine schedule, while the lambda focus schedule was set to two_ swo_phase with 0.2 warm-up and 0.05 ramp-up ratio. Additional parameters for the SVD-based components included an SVD rank (svo_k) of 10, entropy top-k of 20, distillation temperature of 2.0, and the use of the Frobenius norm for SVD normalization.
huttas://qitubb.cc//////eeeee// /b' gcode eva luat inntiion-oe conhharness http:::/giithub.com//msssss
20

---

Published as a conference paper at ICLR 2026
Table Fine-tuning hyperparameters for the DeBERTa-v3-base model on each task of the GLUE benchmark. The settings are aligned with the PiSSA baseline.
Dataset
Epochs
Batch Size
Learning Rate
LoRA Alpha
MNLI
5
16
5 x 10
SST-2
20
16
x 10-5
MRPC
20
32
2 x 10
CoLA
20
1 10
QNLI
10
1 x 10
QQP
10
X 10
RTE
X 10-4
STS-B
10
Table 9: Task Task-specific hyperparameters for fine-tuning RoBERTa-base with LoRA on the GLUE benchmark.
Hyperparameter MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Batch Size
16
32
32
16
Epochs
80
25
40
Learning Rate
5 x 10
5 x
x 10-4
4 x 10
4 x 10
5 x 10
4 10-1
x 10-4
NLU (GLUE Benehmark) Our Natural Language Understanding (NLU) experiments on the GLUE benchmark involved three models, each with specific hyperparameter configurations as detailed below.
We fine-tuned DeBERTa-V3-bise on the GLUE tasks using the AdamW optimizer with linear learning rate schedule. To strictly align with the PiSSA baseline, adopted a set task-specific hyperparameters. The LoRA rank (r) was consistently set to 8 across all tasks. Other key hyperparameters, including the number of epochs, batch size, learning rate, and LoRA alpha, were individually configured for each dataset. The precise configurations are detailed in on GLUE), attach a task-specific
DEBERTA-V3-BAR
T For encoder-only NLU models (e.g., DeBERTa-v3-bel linear classification head and compute Pp and FF by passing the same fine-tuning inputs through the pretrained (teacher) encoder and the M--LoRA-adapterr (student) model, respectively, using the shared classification head, matching the notation used in Section 2.2.1
T5-BASE In our experiments with T5-Base we fine-tuned all models for a single epoch using FP32 precision, maximum sequence length of 128, and batch size of 32. Optimization was performed with the A optimizer (Loshchilov Hutter 2019 (B) = 0.9, f32 = 0.999, e = 10-$, and no weight decay), coupled with a learning rate of 10 The learning rate schedule incorporated a 3% warm-up phase followed by a cosine decay. For the LoRA configuration, we set the rank (r) to 8, alpha (a) 16, and applied it to all linear modules except for the embedding, layer normalization, and language model head layers.
ROBERTA-BASH For fine-tuning RoBERTa-base on the GLUE benchmark, our setup aligns with standard practices for LoRA-basee methods. We employed the AdamW optimizer with a linear learning rate schedule, preceded by a warm-up phase over the first 6% of the total training steps. The LoRA configuration was kept consistent across all tasks: the rank (r) was set to 8 for the query (ạ) and value (v) projection matrices, and the alpha (a) was set to 8. The maximum sequence length was fixed at 512 tokens. Other crucial hyperparameters, including the number of epochs, batch size, and the peak learning rate, were individually tuned for each GLUE task to ensure optimal performance. The precise per-task configurations are detailed in Table 9
C MORE EXPERIMENTS
C.I GENERALITY OF THE REGULARIZATION FRAMEWORK
To verify that our regularization framework's benefits extend beyond PiSSA, we integrated it with standard LoRA and DoRA. The results, presented in Table I0 demonstrate the framework' broad applicability and yield a crucial insight. While our regularizers provide substantial performance gains across all tested methods, their effect standard LoRA is particularly noteworthy. Augmenting
21

---

Published as a conference paper at ICLR 2026
Table 10: Impact of the proposed regularization framework on various LoRA-style methods, evaluated on LLaMA-2-7B. "Ree" denotes the application of our three regularization terms. All results are averaged over 3 runs.
Method
GSMSK
MATH
HvmanEval
MBPP
MT-Bench
Avg
LoRA
42.68 0.54
5.92 0.15
16.80 0.38
21.31 0.43
4.60 0.14
18.30
LoRA Reg
51.82 + 0.36
8.69 0.39
21.03 0.58
33.81 0.51
4.73 + 0.24
24.02
DoRA
41.77 + 0.74
6.20
16.86 0.54
21.60 0.49
4.48 0.14
18.18
DoRA Reg
52.71 + 0.42
8.23 0.27
21.05
0.31
34.78 0.28
4.96 0.22
24.35
PISSA
51.48 0.34
7.00 0.18
19.48
0.45
23.84
4.92 0.07
21.46
BA-LORA (PISSA Reg)
55.86 + 0.35
9.47 0.52
23.58
0.25
36.86 + 0.31
5.11 1
+ 0.05
26.18
2.000
4.300
2000
200
Viduo iCoesiraneo
of (Censronny
(a) MATH, A1
(b) GSM8K, À1
(c) MATH, A2
(d) GSMSK, A
Figure 4: Sensitivity analysis of the core BA-LoRA regularization coefficients. Panels (a.b) show the effect of varying the consistency weight λ MATH and GSM8K separately, and panels (c,d) show the effect of changing the balance between 22 and Ag.
standard LoRA with our regularizers is sufficient to match and even surpass the performance of the more advanced PiSSA baseline. This finding underscores that our regularization framework can function powerful, model-agnostic enhancement for wide range of PEFT methods.
Despite the strong standalone performance of the regularizers, the optimal results are consistently achieved by our full BA-LoRA model. This indicates that PiSSA's principled initialization provides a superior foundation upon which our regularization framework can build, leading to the highest overall performance. This validates integrated approach the most effective configuration for mitigating Catastrophic Inheritance and achieving state-of-the-arr results.
C.2 HYPERPARAMETER SENSITIVITY ANALYSIS
To validate the principled selection of our framework's hyperparameters, conducted detailed sensitivity analysis. Centered around our final BA-LoRA configuration on LLaMA-2-7B, this study systematically investigates the influence of the core regularization coefficients (A₁, , A2, Aa) by perturbing them from their default values. The results, visualized in Figure 4 reveal broad region of stable performance, supporting the robustness of our chosen configuration.
Sensitivity to the Consistency Anchor (A₁) As illustrated Figures b),,), we vary across (0.0125, 0.025, 0.0375) while keeping X2 A3- 0.005. Performance on both MATH and GSM8K remains highly stable across this range, with only minor fluctuations, indicating a broad region of insensitivity. Our chosen value of X 0.025, highlighted as the optimal point in the figure, is thus empirically validated as a robust setting that balances preserving pre-trained knowledge with acquiring new task-specific capabilities.
Sensitivity the Symbiotic Balance ()2 and Xg) Next, we investigate the symbiotic balance between the other two regularizers, which govern trade-off between performance on the two reasoning benchmarks (MATH and GSMSK). As shown Figures (c.d), we compare our final configuration's balanced setting (A2 >a) against two asymmetric conditions: "Weak Chaos" setting (A2 & As), where the structural regularizer (Aş) dominates, and a "Weak Order" setting (As < A2). where the diversity regularizer (22) is dominant. The results reveal clear trade-off: disrupting the equilibrium leads to specialized improvements on one benchmark at the cost of the other, while the balanced configuration provides strong performance both.
22

---

Published as a conference paper at ICLR 2026
LoRA
OLoRA OPISSA
LoRA
QLaRA
RISA B4
PISSA BA
QR5SA
6SNaK Accures (80
Humannba Accursc
DiigSedl-HcE-11 Het---
LLaNA
LLMee-2 2-
Goen 5-73
Mistral/Be 78-001
LLaRA-
Genma-7 10
Mictral
1r- 3-3--4
LLMMA- 708
Laa8A2-77
LLMMA
DespStekt Mol- 16B
LLaMe eg
Gemma- 7B
Goml 5-78
Matral-a-77
Mistral
Figure 5: Performance comparison of different models on the GSM8K and HumanEval benchmarks.
LLaMA-2-78
LLaMA-278
Mistral-78-v0.1
55
Lows BA-LARA Falie
72.5
BALLAA
DACarA
45
NATH AcCurMy
Accurncy
20
65.0
35-
18
7554
25
60.0
16
2
4
8
16
32
04
128
2
26
32
64
128
2
16
32
64
120
2
3.
32
64
328
Bark
Park
7ark
Figure 6: Performance comparison of full fine-tuning, LoRA, PISSA, and BA-LoRA across different ranks.
C.3 ANALYSIS ACROSS DIVERSE MODEL ARCHITECTURES AND SCALES
To assess the generalizability and robustness BA-LoRA, we conducted a comparison against LoRA and PiSSA across distinct pre-trained models. This includes models of varying scales (e.g., LLaMA-2-7B up to I.LMMA33770B and architectures, featuring both standard dense models and Mixture-of-Expert (MoE) models such Mixtral-8x7B. All methods were fine-tuned on blend of reasoning and code datasets (MetaMathQA-1O0) and CodeFeedback- (O0kklkkkkkkO and evaluated on GSM8K and HumanEval.
As visualized in Figure 5, BA-LoRA typically achieves the best or near- best performance among LoRA-style methods on both benchmarks, across most model families and scales. The gains over LoRA and PiSSA are especially pronounced on several mid- and large-scale models, suggesting that our regularization framework remains effective beyond the I.I.N. I.L.MA-2-7B setting.
Furthermore, this performance advantage largely carries over to cnpputation-constaanne settings. The figure also plots the performance of 4-bit quantized versions of each method (QLoRA, QPiSSA, and QBA-LoRA), The overall trend is similar: QBA-LoRA generally matches or exceeds the other quantized baselines, indicating that the benefits of our framework are robust to quantization and remain useful for resource-efficient deployment.
C.4 PERFORMANCE ANALYSIS ACROSS DIFFERENT RANKS
We analyzed the performance of BA-LORA, PiSSA, and LoRA across a range of ranks (from 1 to 128) on the LLaMA-2-7B and Mistral-7B-vO.1 models. Each method was fine-tuned for one epoch on the MetaMathQA- -00OK dataset and evaluated on GSM8K and MATH. The results, presented in Figure 6 show that BA-LoRA consistently outperforms both LoRA and PiSSA across all ranks, models, and tasks, demonstrating its stable and universal superiority. Furthermore, both BA-LoRA and PiSSA exhibit the remarkable ability to surpass the performance of full fine-tuning at higher ranks, with BA-LoRA often achieving this milestone relatively low ranks (e.g., rank 16-32). This
23

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of diversity regularization objectives on MNLI (DeBERTTaaa-
Method
MNLI Accuracy
Standard LoRA
90.71
BA-LoRA (w/ Entropy Reg.)
90.41
BA-LoRA (w/ Covariance Reg.)
91.26
Table 12: Quantitative analysis of feature separation and classification performance the minority class (MNLI imbalanced setting). Metrics are computed in the high high-dimensional logit space.
Method
Global Silhouette
Minority-Class Silhouette (Cluster Quality)
Minority- -lasss Recall
(All Classes)
(Accuracy (%)
Standard LoRA
0.207
0.015
5.8
PiSSA BA-LoRA (Ours)
0.247
0.128
26.4
0,351
0.425
61.7
highlights the strong regularization effect of our approach, as standard LoRA consistently lags behind the full fine-tuning baseline. Moreover, the performance advantage of BA-LoRA over its counterparts even more pronounced on the Mistral-7B. 1 model, suggesting its benefits generalize effectively across different foundational model architectures. These results collectively validate BA-LoRA as highly efficient and superior fine-tuning method.
C.5 COMPARISON OF REGULARIZATION OBJECTIVES FOR NLU
To empirically validate our design choice of Covariance Regularization (CDR_ NL.U) over Entropy Regularization (commonly used in NL.G) for discriminative tasks, we conducted a comparative study on the MNLI benchmark. While entropy regularization is often used to encourage diverse generation, we hypothesized that applying sample-wis entropy minimization to noisy or imbalanced NLU data may inadvertently encourage the model to be "confidently wrong" on minority classes, rather than effectively preventing representation collapse. The results presented in Table [1 support this hypothesis. Notably, substituting our covariance- based regularizer with an entropy- based one results in performance (90.41 %) that is not only inferior to our method (91.26%) but also falls below the standard LoRA A baseline (90.71%) %).. This finding confirms that batch-wise feature decorrelation is a structurally superior strategy for mitigating representation collapse in discriminative fine-tuning.
C.6 QUANTITATIVE ANALYSIS OF CLUSTER QUALITY AND MINORITY REPRESENTATION
To complement the visual analysis provided in Figure g (main text), we conduct a quantitative evaluation of the feature space separation under the imbalanced fine-tuning setting (MNLI task). We compute metrics in the original high-dimensional logit space (before t-SNE projection) to avoid dimensionality reduction artifacts. We report the Global Silhouette Score (over all classes), the Minority-Cllass Silhouette Score (measuring the isolation of the minority cluster), and the Minority- Class Recall (measuring classification accuracy the minority class).
Table 12 presents the results. Standard LoRA exhibits a near-zero minority silhouette score (0.015) and trivial minority recall (5.8), quantitatively confirming the representation collapse observed visually. In contrast, BA-LORA achieves a significantly higher minority silhouette score (0.425) and restores the minority recall 61.7. These results demonstrate that BA-LoRA effectively mitigates the Catastrophic Inheritance of pre-trained priors, enabling the model to establish a distinct and semantically meaningful decision boundary for the minority class.
C.7 DIRECT EVALUATION OF PRE-TRAINED CAPABILITIES
To directly assess how different adaptation schemes affect the underlying pre-trained model, we measure perplexity the Wikitexr-2 (Merity al. (016) language modeling benchmark. We evaluate three variants of LLaMA-2-7B the original pre pre-trdined model, a PiSSA-tuned model, and our BA- LoRA A model-using the same backbone as in the main math-reasoning experiments and standard
24

---

Published as a conference paper at ICLR 2026
Table 13: Wikitext-2 test perplexity (lower is better) for LLaMA-2-7B under different adaptation schemes.
Method
Wikitext-2 PPI.
Pre-trained LLaMA-2-7B
5.42
BA-LoRA (ours)
5.68
PISSA
7.12
left-to-right evaluation on the Wikitext-2 achieves a perplexity of 5.42, while BA-LoRA attains 5.68, indicating only a small degradation relative the pre-trained model. In contrast, the PiSSA baseline reaches 7.12, suggesting a more pronounced shift away from the original pre-training distribution. Together with the downstream results, this supports the view that BA-LoRA preserves the pre-trained model's behavior on natural text substantially better than pure PiSSA -ebseed adaptation.
As shown in Table the pre-trained LLaMA-2-7B
C.8 ASYMPTOTIC COMPLEXITY OF REGULARIZERS
To complement the empirical wall-clock measurements in Table summarize the asymptotic cost of each regularizer in BA-LoRA for both NLU and NLG settings. Table[14reports the additional computational complexity top of the standard forward pass and task loss, expressed in terms of the batch size N, number of classes D (for NLU), number of valid tokens M (for NLG), vocabulary size [V). the Top-K candidate size K, and the SVD rank used in the randomized approximation.
Table 14: Additional asymptotic complexity of BA-LoRA regularizers on top of the standard forward pass and task loss. Here N is the NLU batch size, D the number of classes, AM the number of valid tokens in NLG, V the vocabulary size, K the Top- K subset size, and & the rank used in randomized SVD.
Regularizer
NLU (legits Z & R RN
NLG (logits Z R
Consistency
OLND) (batch over D classes)
OLM(V) ) (token-wise over vocabulary)
Diversity
O(XD)) (batch covariance on D D-dimensional logits)
O(M
M. (Top-K selection and entropy over Vap-R K < (V]
SVD-based
Oçmin NV Dj max(N, D)) (exact (exact
O(M (randomized SVD with
D MORE DISCUSSIONS
Here, we offer further insights into our work.
D.1 PRACTICAL HYPERPARAMETER GUIDELINES
For new model-task configuration, BA-LoRA can be tuned with a simple recipe, without fragile fine-grained search. First, choose initial values for Auomsisency- Ativersity and Ava such that, small calibration batch, each regularization term contributes a meaningful fraction to the total loss (ensuring gradient magnitudes are commensurate with the task loss), that no compo- nent is inactive or overwhelmingly dominant. Second, keep the ratios between the three coeffi- cients fixed and tune a single global scale Àgiobal that multiplies all of them, i.e., A + Àgtobul for i E fconsistency. diversity, svd}; in our experience, coarse search over a few values (e.g.. (0.5, 1.0, 2.0}) on held-out validation split sufficient. Third, needed, individual coefficients can be adjusted by coarse factors (such as x0.5 or x2) to emphasize preserving pretrained knowl- edge (larger Acumsistency) avoiding collapse (larger Adiversit), or suppressing noisy nigh-frequency components (larger Asvd).
As shown by the ablation study in Section 3.2.4 and the sensitivity analysis Appendix C.2 BA-LoRA achieves stable performance around these configurations and consistently outperforms vanilla LoRA and PiSSA in this region, suggesting that this lightweight procedure is sufficient in practice. Moreover, for each backbone, fix a single configuration of the auxiliary hyperparameters (s, T. K, k) based on small validation experiment and reuse it across all datasets, avoiding per- benchmark tuning. Given the stability observed in our prelininary experiments, and to maintain
25

---

Published as a conference paper at ICLR 2026
efficiency, we use these reasonable per- barbbac defaults rather than performing exhaustive, computationally expensive sensitivity sweep.
D.2 ADDITIONAL. DISCUSSION ON ROBERTA VS. T5
Section 3.2.2 uses RoBERTa-base and T5-base as natural comparison for probing robustness to inherited noise, since RoBERTa trained on curated mixture of corpora while T5 is pretrained primarily on the C4 web corpus. However, these models also differ in architecture (encoder-onl) vs. decoodr) and pre-training objective (masked LM vs. text-to-text denoising). As result, the larger BA-LoRA gains we observe on T5 should be interpreted as evidence that is compatible with our noisy- eesy-pprtraining hypothesis, rather than fully controlled causal test. more definitive analysis would require models with closely matched architectures and objectives but systematically varied pre-training corpora, which we leave for future work.
D.3 DISCUSSION ON THE CHOICE OF REGULARIZATION TARGETS
key design choice in BA-LoRA is the application of regularization terms in the model's output space (i.e., logits and their derived distributions) rather than directly the trainable adapter parameters (A and B). This section provides further justification for this principled decision.
Regularizing the low-rank adapter weights directly, for instance, by penalizing the norm of A or B, is a viable alternative. However, this approach presents a significant challenge: the mapping from the -ow-dimesiioal parameter space of the adapters to the high-dimensional functional space of the model's 's final output is highly complex and non loneli Consequently, a simple constraint on the adapter weights (e.g.. small norm) does not guarantee the desired functional behavior (e.g.. output diversity or consistency with the pre-trained model). The effect of such parameter-space regularization on the final model output often unpredictable and difficult to control.
In contrast, applying regularization directly in the output space offers a more direct and interpretable path to achieving our goals. By directly penalizing undesirable properties in the output logits or probability distributions-suce as their deviation from the pre-trained model (Knowledge Drift), their lack of diversity (Representation Collapse), or their over-reliance on non-robust features (Overfitting to Noise) we explicitly constraining the model's final behavior. This approach ensures that our optimization objective is perfectly aligned with the ultimate goal of mitigating the functional consequences of Catastrophic Inheritance. The strong and consistent performance of our framework across diverse models, tasks, and ranks, as demonstrated in our experiments, serves as powerful empirical validation for this output-space regularization strategy.
D.4 CONCEPTUAL FOUNDATIONS AND SYNERGY OF REGULARIZERS
The three regularization terms proposed in MALLoRRA-cosistennn diversity, and SVD-based regularization- were not chosen arbitrarily. Each is inspired by well-established principles in the machine learning literature for improving model robustness and generalization, and they are designed to work in synergy.
Origins. The Consistency Regularizer (implemented as a KLD-based distillation loss in our experiments) a form of knowledge distillation (Hinton al. 2015), specifically self-distillation, where the pre-trained model acts as the teacher. The Diversity Regularizer is rooted in principles from representation learning and information theory. The covariance-based term for NLU is directly inspired by methods that combat representation collapse in self-supervised learning (Bardes et al. 2021), while the entropy-based term for NLG a classic technique to prevent mode collapse and improve diversity in generative models (Cover, 1999). Finally, the SVD Regularizer builds upon the principle spectral regularization, where the singular value spectrum of a weight or feature matrix is constrained to improve generalization. The insight that dominant singular values capture the most robust data patterns is a recurring theme in robust machine learning and transfer learning (Chen follow the standard temperature-scales distillation
et al. 2019). Throughout our experiments, convention (Hinton et al. 2015) and multiply the KL-based distillation loss by T2, Since dividing logits by 7 scales the gradients of the KI. divergence approximately 1/T2, this factor keeps the
26

---

Published as a conference paper at ICLR 2026
effective gradient norm of the consistency loss roughly invariant to T, so that changing T primarily controls the softmess of the teacher distribution rather than unintentionally reweighting the regularizer.
Synergy. While each regularizer addresses distinct failure mode, their combination creates synergistic effect. For instance, solely enforcing consistency (CcR) might excessively constrain the model, preventing from fully adapting to the downstream task. However, when combined with the diversity regularizer (CDR), the model is encouraged to explore new, diverse representations within the bounds of the pre-trained knowledge. Similarly, the SVD regularizer (CSVDR) helps ensure that the diverse representations learned are also the most robust and generalizable ones, preventing the model from learning spurious correlations encouraged by a simple diversity objective. Our ablation study (Section 3.2.4) empirically confirms this synergy, showing that the performance of the full BA-LoRA model surpasses the sum of the individual components' contributions.
D.5 ON APPLYING REPRESENTATION LEARNING PRINCIPLES DURING
key consideration for our work is whether incorporating principles from self-supervised learning (SSL), such as our diversity regularizer, during fine-tuning could disrupt the model's pre-trained representations. We contend that our framework effectively mitigates this risk through two primary mechanisms.
First, the PEFT paradigm, specifically LoRA, inherently limits the scope of any changes. With the vast majority of parameters frozen, the model's 's core representational geometry remains anchored. Our regularizers guide only the small perturbations introduced by the low-rank adapters, ensuring these updates refine rather than overwrite the foundational knowledge.
Second, our regularization scheme is synergistie. The consistency regularizer (CcR) acts as a crucial counterweight to the diversity regularizer (LDR). While LD encourages adaptation and prevents representation collapse on the downstream task, LCR ensures this adaptation does not stray from the pre-trained model's 's robust knowledge manifold. It is precisely this calibrated balance-what we term "guided exploration within a trusted neighborhood". -hat allows BA-LoRA to enhance task-specific performance without inducing catastrophic forgetting.
D.6 LIMITATIONS AND FUTURE WORK
While this study validates the effectiveness of BA-LoRA, there are areas for future research. Our empirical evaluation has primarily focused on foundation for BA-LoRA; however, future work should extend this validation to multilingual settings and specialized domains to ensure broader applicability of the method. Moreover, our analysis of noisy pre-training in Section r.2.2 relies on RoBERTa-base and T5-base checkpoints, which differ in both architecture and pre-training objective; results should be interpreted as suggestive rather than controlled evidence about pre-training noise, and more definitive study with fixed architecture pre- trained under systematically varied noise levels is an important direction for future work. addition, while BA-LoRA's regularization components have shown strong promise, tpsk-sp adaptations could further optimize their performance across a wider range of applications, and exploring such adjustments will be valuable for enhancing robustness and adaptability in diverse use cases.
benchmarks, which are robust
D.7 ETHICS STATEMENT
This study aims to develop and evaluate BA-LoRA, a novel parameter-efficient fine-tuning method designed to mitigate bias and enhance the performance of LLMs. By aiming to create more robust and less biased models, primary ethical motivation of this work is to contribute to safer and more reliable AI systems. Our research utilizes existing open-source public datasets for both fine-tuning and evaluation purposes, including MetaMathQA, CodeFeedback, WizardLM-Ewe -not--nstuutt and the GLUE benchmark.
While these datasets are widely recognized and adopted within the research community, we acknowl- edge that large-scale corpora and LLM -..--eeerrtted instructions may contain inherent societal biases or noisy artifacts. Importantly, rather than ignoring these issues, BA-LoRA is explicitly designed to prevent the exacerbation of such underlying biases during model adaptation. Furthermore, we recognize that while parameter-efficient methods democratize the fine-tuning of L.LMs, they also
27

---

Published as a conference paper at ICLR 2026
lower the barrier for potential malicious adaptation (dual-use risk). By constraining the output space to prevent representation collapse and Catastrophic Inheritance, our proposed regularizers serve as step toward ensuring that adapted models remain closely aligned with their robust, pre-trained behaviors. We remain committed to the responsible development and transparent reporting of Al technologies.
D.8 REPRODUCIBILITY
To ensure the reproducibility of our results, provide a detailed description of our experimental setup in Section 3.T and Appendix [ including model introduction, dataset introduction, hyperparameter configuration, and evaluation procedures. All models and datasets used publicly available. In addition, we have refined the implementation scripts and fine-tuning strategies facilitate independent verification. To further facilitate reproducibility, source code, including scripts to replicate all main experiments, is publicly available at wtthtps:: http::///gtubb com/ coommlmmg 11m1 72 BA- LORA
USE OF LARGE LANGUAGE MODELS
In the preparation of this manuscript, large language model (LLM) was utilized a writing assistant. The LLM's role was strictly limited to improving the clarity, conciseness, and grammatical correctness of the text. Specifically, it was used for tasks such as rephrasing sentences, suggesting alternative vocabulary, and checking for stylistic consistency. All core scientific ideas, experimental designs, data analyses, and conclusions were conceived and formulated exclusively by the human authors.
28


Reviews:
[{"review_id": "CmfbTp40xf", "reviewer": "Reviewer_N4aD", "summary": "The paper proposes to add three additional regularizers in LoRA fine-tuning: 1) distillation loss between pretrained and fine-tuned models to reduce output shift; 2) penalty on correlation between output classes or entropy of outputs to promote diversity; 3) penalty on high-frequency components in the output logit matrix.", "questions": "The purpose of the paper is unclear. At the beginning, the paper states that the goal is to evade/eliminate catastrophic inheritance, i.e. the undesired traits of a pretrained model (e.g. imbalances, biases, spurious correlations) being kept/exacerbated during fine-tuning, but the paper first applies a distillation-style regularizer to prevent shift from the pretrained model outputs. This appears to be self-contradictory. If the outputs of the pretrained model are undesired and need to be fixed during fine-tuning/post-training, then knowledge drift is what we desire. The case is similar in the other two subtasks: L90-92 attribute deteriorated performance due to the fine-tuning data quality, instead of the pretraining data. I think further explanation is critical for a consistent paper.\n\nL85-86: It is claimed that LoRA exacerbates catastrophic inheritance; this makes sense, but I don't see any empirical evidence supporting that in the paper.\n\nL155&L201: It is the outputs given pretraining or fine-tuning data? They have completely different implications. Also, how do you obtain the outputs of a un-finetuned encoder (e.g. Deberta) on NLU tasks? I assume that an extra linear projection head needs to be attached.\n\nL159&L206: The reason for using T^2 is not really clear and needs further explanation.\n\nL161: Should use \\citep in this case.\n\nL166: I'm confused by the purpose of this regularizer. Classes in different categories may not be orthogonal to each other, and some classes can be inherently correlated. Also, why is correlation a sign of a lack of diversity? If the goal is to promote diversity/avoid over-confidence, similar entropy-based regularizers like the one used in NLG (L214) should also work.\n\nL177: Why do we expect the logit matrix to be low-rank? Is there any specific reason? How is the high-frequency component in the output logits related to the data noise? Particularly, if each of the sample certainly belongs to a class and D=N, the logit should be full-rank.\n\nL180: What is \"spurious intra-batch variations\"? I assume that samples within each batch should be independently sampled.\n\nL250: The selection of regularization weights appears to be quite arbitrary. Tuning three hyperparameters can be challenging and can be highly varied between tasks. Appendix C.2. actually demonstrates the issue, as the trend on MATH and GSM8K in Fig.4 (a) are different. Also, the change in accuracy in Fig. 4 is downplayed with the large difference in scale of the accuracy on the two tasks. I would suggest putting the lines on different figures. \n\nL1038 states that the regularizers steer the model along the creativity-robustness spectrum, but both MATH and GSM8K are reasoning tasks, and I doubt if MATH can be used as a proxy for creativity.\n\nSec 3.2.2: In addition to differences in training data, there are considerable differences in the model architecture and data size between RoBERTa and T5, hence I doubt if the difference in improvements can be attributed to noise resilience.\n\nTo summarize, I fear that there are too many claims and assumptions in this paper that are not sufficiently supported by concrete evidence. I would recommend the authors to   \n1) Make their goals clear.  \n2) Provide empirical evidence that the issues (as for this paper, forgetting, diversity, and noise) exist by dedicated evaluations, especially with LoRA, e.g., by comparing generation diversity in the original, fully fine-tuned, and LoRA fine-tuned models. Standard benchmarks on reasoning, understanding, or commonsense cannot directly support your claim.  \n3) Show that each of the regularization terms can improve the issue, respectively.  \n4) Demonstrate the synergy of the trio.\n\nThere are other papers discussing alleviating knowledge drift in LoRA fine-tuning, e.g.   \nSmith, James Seale, et al. \"Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRA.\" Transactions on Machine Learning Research (2024).  \nChen, Haolin, and Philip N. Garner. \"Bayesian parameter-efficient fine-tuning for overcoming catastrophic forgetting.\" IEEE/ACM Transactions on Audio, Speech, and Language Processing (2024).", "limitations": "", "rating": null, "confidence": 3, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "The empirical advantage of combining the three regularizers is supported by improved empirical results.", "weaknesses": "The purpose of the proposed methods is not clear to me; see below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b0YktdgBWQ", "reviewer": "Reviewer_SgzW", "summary": "The paper proposes to use a bunch of existing regularization techniques combined with the existing LoRA variation to improve the fine-tuning performance. The paper evaluates the method on Llama2 7B and shows that it performs better than many existing methods.\n\nIt's good to know that combining all these regualrziation techniques together improves the performance but I don't quite understand the novelty of this paper.", "questions": "- Can the author clarify what are the actual contributions of this paper?\n\n- How are the 3 lambdas chosen?\n\n- Why is BA-LoRA better than full model fine-tuning?\n\n- The lines in Fig.2 are confusing in that: why the proposed method improves training performance. Isn't the main question to be addressed about generalization ability?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is well motivated.\n\nThe empirical results seem to be promising.", "weaknesses": "- The base model used for evaluation is extremely out of date, Llama2 is released in 2023, and I am not sure if the conclusion drawn is transferable to newer models.\n\n- The method has 3 hyperparameter to tune, and the paper does not provide any guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dV1FjnbkIk", "reviewer": "Reviewer_nsxF", "summary": "The authors present *Catastrophic Inheritance* in LLMs, a propagation of biases, noise, and data imbalances from pre-training into fine-tuning models, and ways to mitigate it. They identify: **Knowledge Drift**, where the model forgets pre-trained knowledge while learning new tasks; **Representation Collapse**, where fine-tuning on imbalanced data causes a lack of output diversity; and **Overfitting to Noise**, where the model learns spurious correlations that impact generalization. They offer three regularizers to combat these issues in Natural Language Understanding and Generation tasks: consistency regularizer for Knowledge Drift, diversity regularizer for Representation Collapse, and Singular Value Decomposition regularizer for Overfitting to Noise.\n\n1. The consistency regularizer is based on a knowledge distillation approach via KLD between pre-trained and fine-tuned model probability distributions to preserve foundational knowledge.\n2. The diversity regularizer is based on penalizing off-diagonal elements in a covariance matrix or maximizing entropy within the most plausible tokens to promote diversity in the model’s predictions across a batch.\n3. The SVD regularizer is based on encouraging low-rank structure by maximizing the ratio of spectral energy in the top-k singular values to learn robust features.\n\nThey test the method **Bias-Alleviating Low-Rank Adaptation (BA-LoRA)** in mathematical reasoning, coding, and conversational AI for NLG, as well as on the GLUE benchmark for NLU, using models such as LLaMA-2-7B and DeBERTa-v3-base. Additionally, they look at gains in models that were trained with clean or noisy data, as well as t-SNE visualizations of the features to see the impact of data imbalance, and perform ablation studies.", "questions": "1. Page 2 L86, can you provide a reference if this been shown before in LoRA?\n2. How does the model manage to mitigate the Knowledge Drift? And how does PiSSA initialization relate to the Knowledge Drift?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper offers an original perspective on catastrophic inheritance with clear methodology and strong experimental evaluation, making it a significant and well-presented contribution.\n\n---\n\n- The abstract is very clear, well written, and easy to understand.\n- The experimental setup is thoroughly described, with clear reporting of hyperparameters.\n- The results are evaluated over multiple random seeds.\n- The work covers a wide range of setups and datasets, reflecting a comprehensive and up-to-date evaluation.\n- The experiment comparing noisy vs. clean data is particularly valuable and insightful.\n- The appendix offers a lot of information on the setup, different models, sensitivity analyses, and performance across rank. This is very solid experimental work, and it should be highlighted more prominently in the main text, as a substantial amount of effort and insight resides there.", "weaknesses": "### Methodology and Experiments\n* A comparison between NLU and NLG is missing. The methodology section duplicates the description of the regularizers; what is missing is the motivation for changes required to adapt the approach to NLG, followed by a clear presentation of that modified setting. Because of this, the methods section feels unsatisfactory.\n* Despite introducing a regularizer for forgetting of pre-trained knowledge, the authors never directly evaluate forgetting. Overall, the first regularizer remains insufficiently probed.\n\n### Fairness & Comparability of Evaluation\n* Sourcing scores from original publications with “comparable” setups may not be a valid comparison if the experimental configurations are not identical.\n* C4 is much larger and should arguably have been sampled or sliced to make comparisons fairer.\n\n### Computational Considerations\n* The computational costs of the different steps should be described (e.g., randomized SVD).\n\n### Initialization Choices\n* PiSSA seems like a questionable initialization strategy, it has been shown in [1] that PiSSA induces forgetting of pre-trained knowledge, as the adapters are initialized with the “core” knowledge. It seems BA-LoRA mitigates an issue amplified by this initialization.\n\n### Related Work\n* The related work section is too short and lacks explicit comparison to this work.\n\n[1] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning. (2025) Hanqing Wang and Yixia Li and Shuo Wang and Guanhua Chen and Yun Chen", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CA09GIGEyd", "reviewer": "Reviewer_KjTf", "summary": "The authors state that LoRA can lead to Catastrophic Inheritance, which hurts robustness and fairness. \nThey propose Bias-Alleviating LoRA (BA-LoRA), which decomposes the issue into Knowledge Drift, Representation Collapse, and Overfitting to Noise. \nBA-LoRA adds three regularizers to preserve core knowledge, maintain representational richness, and encourage robust low-rank behavior.", "questions": "1. The paper builds on PiSSA initialization. How much of BA-LoRA’s improvement remains if you start from random low-rank adapters or standard LoRA init? Do the three regularizers still deliver comparable gains?\n2. How sensitive are results to the SVD rank, distillation temperature, and the top-k entropy window?\n3. How well does BA-LoRA scale to larger models (e.g., 13B, 70B)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. Important topics\n2. Three well-motivated losses mapped cleanly to three failure modes; easy to plug into existing LoRA workflows.\n3. Broad NLU/NLG coverage with ablations that attribute gains to each component.", "weaknesses": "1. Although the framework names are shared, the diversity and SVD regularizers differ for NLU vs. NLG, so it reads like two papers rather than one universal design.\n2. Some studies are missing in the paper (e.g., impact of different r/T/k, larger models)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pL7kdZJ3el", "reviewer": "Reviewer_4aVv", "summary": "This paper introduces BA-LoRA, a PEFT method designed to mitigate catastrophic inheritance, including the propagation of biases, noise, and data imbalances from pre-trained LLMs. BA-LoRA addresses three failure modes via three regularizers applied in the output space, i.e., consistency regularizer, diversity regularizer, and SVD regularizer.\nExperiments evaluate BA-LoRA on NLG and NLU tasks, achieving state-of-the-art among LoRA variants. A controlled study shows BA-LoRA's gains are amplified on models pre-trained with noisy data. Ablation studies validate each regularizer's contribution.", "questions": "1. In Table 1, the gain of superior performance mainly depends on the MBPP benchmark, where BA-LoRA gains 36.86, whereas the second-best baseline only achieves 25.74. It would be valuable if the author could analyze why BA-LoRA performs particularly well in this dataset.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel decomposition of catastrophic inheritance into three actionable failure modes. Each failure mode is directly linked to a specific regularizer (consistency, diversity, SVD), creating a coherent and interpretable architecture.\n2. BA-LoRA sets a new state-of-the-art performance with strong empirical validation, on both NLG benchmarks and NLU tasks. Moreover, performance gains are even higher on noisier pre-trained models, directly validating its core hypothesis about mitigating inherited noise.", "weaknesses": "1. The comparison between RoBERTa-base and T5-base shows that BA-LoRA's advantage is significantly larger on T5 (3.26 vs 1.11 points). However, the models differ in both architecture (encoder-only vs encoder-decoder), which should be controlled to strengthen the claim. This makes it difficult to attribute the performance difference solely to the 'noisier data' factor.\n2. The SVD regularizer uses different normalization schemes: NLU uses the sum of all singular values, while NLG uses the Frobenius norm. What is the theoretical justification for this difference? Understanding whether this choice is principled (e.g., due to task differences) or empirical is important for the generalization of the method beyond NLU and NLG.\n3. The method introduces too many hyperparameters, including the tradeoff parameters $\\lambda$s, temperature T, number of top-K, and SVD rank. Ablation studies on their sensitivity can enhance the usability of BA-LoRA in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)