KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
Authors: Fan Wang
Juyong Jiang
Chansung Park
Sunghun Kim
Jing Tang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Models
Parameter-efficient Fine-tuning
Singular Value Decomposition
Primary area: foundation or frontier models, including LLMs

Abstract: The increasing sizes of large language models (LLMs) result in significant computational overhead and memory usage when adapting these models to specific tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have been devised to mitigate these challenges by training a small set of parameters for the task-specific updates of the model weights. Among PEFT methods, LoRA stands out for its simplicity and efficiency, inspiring the development of a series of variants. However, LoRA and its successors disregard the knowledge that is noisy or irrelevant to the targeted task, detrimentally impacting model performance and leading to suboptimality. To address this limitation, we introduce Knowledge-aware Singular-value Adaptation (KaSA), a PEFT method that leverages singular value decomposition (SVD) with knowledge-aware singular values to dynamically activate knowledge based on its relevance to the task at hand. We conduct extensive experiments across a range of LLMs on tasks spanning natural language understanding (NLU), generation (NLG), instruction following, and commonsense reasoning. The experimental results demonstrate that KaSA consistently outperforms FFT and 14 popular PEFT baselines across 16 benchmarks and 4 synthetic datasets, underscoring our method's efficacy and adaptability. The source code of our method is available at https://github.com/juyongjiang/KaSA.

Published as a conference paper at ICLR 2025
KASA: KNOWLEDGE-AWARE SINGULAR- VALUE ADAPTATION OF LARGE LANGUAGE MODELS
Fan Wang", , Juyong Jiang", Chansung Park + Sunghun Kim 1* Jing Tang 10
"2
The Hong Kong University of Science and Technology (Guangzhou)
Electronics and Telecommunications Research Institute
The Hong Kong University of Science and Technology
ccsfanwang, cs [hunkim, jingtang ) @ust.t.. hk
d cap)
ABSTRACT
The increasing sizes of large language models (LLMs) result in significant com- putational overhead and memory usage when adapting these models to specific tasks or domains. Various parameter-efficient fine-tuning (PEFT) methods have been devised to mitigate these challenges by training a small set of parameters for the task-specific updates of the model weights, Among PEFT methods, LoRA stands out for its simplicity and efficiency, inspiring the development of a series of variants. However, LoRA and its successors disregard the knowledge that is noisy or irrelevant to the targeted task, detrimentally impacting model performance and leading to suboptimality. To address this limitation, we introduce Knowledge- aware Singular-value Adaptation (KaSA), PEFT method that leverages singular value decomposition (SVD) with knowledge-awee aware singular values to dynamically activate knowledge based on its relevance to the task at hand. We conduct exten- sive experiments across a range of LLMS on tasks spanning natural language un- derstanding (NLU), generation (NLG), instruction following, and commonsense reasoning. The experimental results demonstrate that KaSA consistently outper- forms FFT and 14 popular PEFT baselines across 16 benchmarks and 4 synthetic datasets, underscoring our method's efficacy and adaptability. The source code method is available at https: // //g/ thub. com/ juyong j jiagg/ KaSA.
INTRODUCTION
Large language models (LLMs) pretrained on massive general domain data have shown remarkable generalization ability,, facilitating their application across diverse tasks (Zhao et al., 2023; Touvron et al., 2023b; OpenAI, 2023; Yoo et al., 2024; Jiang et al., 2024). The adaptation of these pretrained language models (PLMs) to specific downstream tasks generally involves full fine-tuning (FFT), where all model parameters are updated and distinct replicas of the parameters are saved for each task (Guo al., 2021; Mao et al.. 2022: Gao al., 2024). However, the increasing size of LLMs significantly raises the computational and memory costs associated with FFT. making FFT imprac- tical in resource-constrained environments (Lester et al., 2021; Cai et al., 2024; Meng et al., 2024). Consequently, surge of parameter-efficient fine-tuning (PEFT) methods (Zaken et al., 2021; Li & Liang, 2021; Hu et al., 2021; Liu al., 2023; Pfeiffer et al., 2021; Houlsby et al., 2019; Liu al., 2024) have emerged, aiming to reduce the computational and memory costs by only updating a small set of parameters while fixing the base model (Mao et al., 2022: Lialin et al., 2023).
Notably, LoRA (Hu et al., 2021) is popular for its simplicity and effectiveness ( (Wang et al., 2024a; Liu et al., 2024; Gao et al., 2024). reparameterizes the task-specific update AW E R= with a couple of low-rank matrices, A and B, while keeping the base model ((() E < unchanged during fine-tuning. Without loss of generality, we suppose > to simplify the notation. The fine-tuning process of LoRA can be formally expressed as (")) + AW = W()) + BA, where
"qqual contributors: Fan Wang. Juyong Jiang, and Chansung Park. Corresponning authors: Sunghun Kim and Jing Tang.

---

Published as a conference paper at ICLR 2025
B E R A E Rrxm, is scaling constant, and the rank < m. A significant advantage of LoRA is its practicality in integrating the low-rank matrices back into the base model, thereby preserving the model architecture and avoiding additional inference latency (Hu et al., 2021; Han et al., 2024; Meng et al., 2024).
Despite LoRA's success, its initialization strategy, which employs random Gaussian noise for A and zeros for B, creates an unguided subspace for the trainable parameters, causing slow convergence and suboptimal performance (Meng et al., 2024; Wang et al., 2024a). To address this problem, PiSSA (Meng et al., 2024) and MiLoRA (Wang et al., 2024a) use singular value decomposition (SVD) for optimizing initialization. SVD can factorize a matrix into three distinct matrices (U, E, V), where U and V are semi- cemi-orthogonal matrices, and Σ is a diagonal matrix containing sin- gular values sorted in descending order. In particular, the magnitude of singular values represents the importance of parametric knowledge encapsulated in their corresponding singular vectors, with large values indicating important world knowledge and small values indicating noisy or long-tail knowledge (Yan et al., 2021; Wang et al., 2024a; Yang et al., 2023; Sharma al., 2023). PiSSA and MiLoRA apply SVD to decompose the base model into two components: the principal components correlated with major singular values, and the residual components associated with minor singular values. Specifically, PiSSA fine-tunes the low-rank matrices, B and A, initialized with principal components, while preserving the residual components frozen, resulting in faster convergence and improved model performance (Meng et al., 2024). In contrast, MiLooRA focuses on fine-tuning B and A initialized with the minor singular value components, while fixing the principal components, aiming to boost performance and alleviate world knowledge forgetting (Wang et al., 2024a).
However, PiSSA and MiLoRA disregard two issues that can detrimentally affect model performance. Firstly, a portion of the task-specific updates targets the weight changes of the noisy knowledge en- coded in the base model, potentially leading to suboptimal performance. Secondly, the low-rank matrices, whether initialized with the principal or residual components, inherit knowledge from the base model. These components may include information that is irrelevant to the specific downstream task, leading to conflicts within the parametric knowledge and degrading the model's representa- tional capability.
To address these problems, we propose PEFT method, named KaSA (Knowledge-awwe Singular- value Adaptation), which leverages SVD with knowledge-aware singular values to dynamically ac- tivate parametric knowledge according to its relevance to downstream tasks. Specifically, KaSA begins by performing snowledge-based SVD truncation to the base model ((() for removing the minor singular components W nosee E R= that contain noisy and long-tail knowledge (Gu al., 2024; Wang et al., 2024b; Meng et al., 2024). This process results in an SVD-truncated model W world E _2 that retains essential world knowledge. To maintain a consistent representa- tional space between W world and its task-specift updates AW, KaSA reparameterizes A WW in the SVD form, AW AUAEAV where AE comprises knowledge-awwid singular values (Ar)...,, Ao,). ). The singular-value adaptation presents dual benefits: 1) reparameterizing the task- specific updates in SVD form ensures that these updates and W world share the same representational space, thereby preserving knowledge consistency; 2) the knowledge-awar activate the parametric knowledge according its pertinence to particular downstream tasks, reduc- ing the intervention of irrelevant knowledge, consequently enhancing model performance.
singular values learn to
We conduct extensive experiments to fine-tune LLMs of varying sizes and architectures across var- ious tasks, including natural language understanding (NLU), natural language generation (NLG), instruction following, and commonsense reasoning. Substantial experimental results demonstrate that KaSA consistently outperforms FFT and 14 existing popular PEFT baselines across different LLMS on 16 benchmarks and 4 synthetic datasets, highlighting its efficacy and adaptability. In summary, the key contributions in our work are follows:
We propose a novel PEFT method, KaSA, which leverages SVD with knowledge-aware singular values to activate parametric knowledge based its relevance to downstream tasks, achieving superior performance over FFT and existing prevailing PEFT techniques across various tasks.
KaSA features a linear framework allowing seamless integration of the singular value adap- tation module with the SVD truncated model, inducing inference latency. Furthermore,
2

---

Published as a conference paper at ICLR 2025
method supports training distinct adaptation modules for different tasks, all sharing single base model, thereby reducing the storage needs for task-switching.
We conduct extensive experiments on NLU, NLG, instruction following, and common- sense reasoning tasks using various LLMs on well-known benchmarks. Our KaSA consis- tently outperforms FFT and 14 PEFT baselines across different benchmarks and synthetic datasets, demonstrating its efficacy and adaptability.
We make all high-quality synthetic instruction foolowwin datasets generated by GPT4o pub- licly available enabling the community enhance the functionality of PEFT and support future research endeavors.
2 RELATED WORK
2.1 PARAMETER-EFTICHEE FINE-TUNING
The increasing LLM scale presents significant challenges efficiently adapting these models to spe- cific tasks (Lialin et al., 2023; Zhao et al., 2023). In response, surge of PEFT methods has emerged, reducing the computation burden by updating a minimal set of parameters during fine-tuning (Mao et al., 2022; Karimi Mahabadi et al., 2021; Han et al., 2024). PEFT methods can be generally categorized into selective, additive, and re- re-parameterized methods (Ding et al., 2022; Lialin et al., 2023: Xu et al., 2023). Selective methods (Zaken et al., 2021; Sung et al., 2021; Guo et al., 2021; He et al., 2023) train a predetermined set of the model's existing parameters while keeping the rest of the model intact. Additive methods (Houlsby al., 2019; He et al., 2022a: Li & Liang, 2021; Liu et al., 2023; Lester et al., 2021) introduce extra modules or parameters to fine-tune and maintain the original base model frozen. Reparametrized methods (Hu al., 2021; Dettmers al., 2023; Zhang et al., 2022; Valipour et al., 2023; Liu et al., 2024) reparameterize the model's 's weight updates into an equivalent low-rank form for fine-tuning. Among reparameterized approaches, LoRA stands out for its simple yet efficient mechanism of employing two low-rank matrices to approximate task specifce updates. The fine-tuned LoRA matrices can be integrated with the base model, ensuring no inference latency. LoRA has inspired a series of variants, each targeting specific improvements. For instance, DyLoRA (Valipour et al., 2023) trains the low-rank matrices across a spectrum of ranks by sorting the representation learned at different ranks during training. shortening the training time. QLoRA (Dettmers et al., 2023) combines 4-bit quantization with LoRA for enhanced resource efficiency. DoRA (Liu al., 2024) decomposes the base model into magnitude and direction components for fine-tuning, reducing the number of trainable parameters and improving performance over LoRA. Our method, KaSA, diverges from these reparametrized methods by employing a knowledge-aware SVD structure, enhancing the fine-tuning efficacy further.
2.2 SINGULAR VALUE DECOMPOSITION IN NATURAL LANGUAGE PROCESSING
SVD plays a crucial role various domains, such as model compression (Yuan et al., 2023; Wang et al., 2024b; Hsu et al., 2021; Chen et al., 2021), dimensionality reduction of word embeddings (Tanwar et al., 2018; Shyamasundar & Rani, 2016), and latent semantic structure analysis (Deer- wester et al., 1990; Kou & Peng, 2015; Horasan et al., 2019). In the rapidly growing realm of LLMs. SVD emerges as a promising, yet relatively underexplored, technique for PEFT. A series of SVD-based PEFT methods exploit the relationship between SVD and matrix rank ascertain optimal ranks for specific downstream tasks. For example, AdaLoRA (Zhang et al., 2022) employs SVD to reparameterize task-specific updates and adaptively determines the suitable rank through importance scoring, thus improving the model performance and parameter efficiency. SARA (Gu al., 2024) conducts SVD at the initialization phase identify the appropriate rank for each layer, thereby maintaining the benefits of LoRA and boosting performance. PíSSA (Meng et al., 2024) and MiLoRA (Wang et al., 2024a), as mentioned in Section 1, utilize SVD to optimize LoRA's ini- tialization. Specifically, PiSSA (Meng et al., 2024) only fine-tunes the low-rank matrices initialized with the principal components associated with a few largest singular values, while preserving the residual frozen. This initialization strategy facilitates faster convergence and enhanced performance. Conversely, MiLoRA (Wang et al., 2024a) fine-tunes the minor components associated with mini- mal singular values, enhancing model performance while preserving the model's 's world knowledge.
https//tuggingffccco/lllaa -olla
3

---

Published as a conference paper at ICLR 2025
Frozen
Learnable
Y1
Yi
V1.
Masked
Truncation
U R
Σ E Rick
VT E R Rkxm
AVT E R
AV AV =
Pre-trained Weights W E m
AE 400
AUT AU
AU E IRXX
m
Figure 1: The architecture of our proposed KaSA encompasses two stages: (Left) knowledge-based SVD truncation remove the noisy knowledge from the base model; (Right) knowledge-awaad singular-van value adaptation to adjust singular values that dynamically activate knowledge across A AW model parameters based on its relevance to downstream tasks.
Unlike these methods, our method emphasizes the adaptive adjustment of singular values, allowing nuanced and dynamic activation of parametric knowledge based on its importance downstream tasks.
METHODOLOGY
In this section, we commence with modeling the general PEFT process and training objective in Section 3.1. We subsequently provide detailed introduction of KaSA in Section 3.2, followed by the description of its training objective in Section 3.3.
3.1 PROBLEM STATEMENT
Before introducing KaSA, it is essential delineate and model the process and objective of PEFT LLMs based on the Transformer architecture (Vaswani, 2017). Fundamentally, PEFT is the process of training pretrained model targeted task using task-specific dataset. It aims to minimize the divergence between the predicted probability distribution of the fine-tuned model and the actual distribution of the training data, while only modifying small set of parameters.
Consider pretrained model ( initially parameterized by Og. To adapt this model to particu- lar task, employ PEFT using dataset D {( y1 ))){ comprising Q input-output instances. The PEFT process utilizes a limited set of parameters, denoted as V, to learn the task-specific up- date A0, ensuring that | < (O.l. This results in fine-tuned model W, parameterized by Oo + (T)).. The objective is to align the predicted probability distribution of W with the ac- tual distribution of training data, thereby enhancing the fine-tuned model's task performance. The primary objective of PEFT is thus centered on the optimization of y:
Li(V)= = Σ
(a.)))
3.2 KNOWLEDGE-AWAR SINGULAR-VAIE VALUE ADAPTATION
As depicted in 1, KaSA encompasses two primary stages: 1) the knowledge- -based SVD trunca- tion, which removes the noisy knowledge from the base model; and 2) knowledge-aware singular- value adaptation, which involves adjustment of singular values that dynamically activate parametric knowledge based on its relevance to the targeted task.
KaSA begins with knowledge-baadd based SVD truncation to the base model ())) E = For sim- plicity of denotation, suppose This process factories (0)) using SVD and subsequently

---

Published as a conference paper at ICLR 2025
truncates the minor singular components W noise R removing noisy and long-tail knowl- edge and resulting in a lower-rank model W worll E Raxm We use this refined model W world to approximate the base model, making the adaptation of W be resembled by that of W world:
m
m
W = w AW UEV + A(UEVT)
Au,o,
(2)
(W world + W noise) (AW
world +
AW (oosse)
(3)
m-
1,0,0 U40 1)
+
(4)
mh-
m-
2 W world + world Σ
+ Σ ((uo,,,))
(5)
R=xm and V is the transpose of V. U= [ui, um] and V [Vi,.... .... Um are the corresponding matrices containing left and right singular vectors, respectively. The diagonal matrix Σ E mm contains positive singular values (0),..., T., sorted from high to
where U E Raxm
low (01 2 02 2 2 Jm 2 0). The hyperparameter represents the number of truncated minor singular values, with r < m. and are semi- orthooothogonal, satisfying that:
UT- V = II
(6)
where the identity matrix In E mm Following the knowledge-hased SVD truncation, we employ the knowledge-awail singular-value adaptation, which reparameterizes the task-specific updates of W Wworld in the SVD form with knowledge-aware singular values. Therefore, the weight of a model fine-tuned with KaSA can be formally expressed as:
W = w()) + AW ~ W World + + AAAEAA Σ =1
uloo)); T + Σ Au, (Ao,
==1
s.t. AU AV
where E Rr 17 is constant scaler, the diagonal matrix AΣ R comprising learnable knowledge-awar singular values (Ar Ao,) The matrices AU and AV are semi-orthogonal, ensuring that the updates retain necessary structural properties.
3.3 TRAINING OBJECTIVE
FFT typically serves as a comparative performance upper bound for PEFT methods (Valipour al., 2023). Consequently, we expect that the performance of the model fine-tuned with KaSA will closely approximate that of FFT. We denote the FFT model Wfft w()) AW. We impose a regularization [Wrft W Worrddl represented by the Frobenius norm, to constrain the task- specific updates. Based on the properties of Frobenius norms, we can further explore the boundary of the task-specific updates:
[W5sell++ | FF Z
| fft W Woorrall |AAAAAV F Σ j=1
(Ao,))))
||
(8) To stabilize the model training and extend the searching space, introduce L2 to minimize the lower boundary of [[Wrre fft W Wworldlls::
(x(EE) |AASEAV 1
According to the Eekart-Youngg Young-Mirsky theorem (Eckart & Young, 1936), C2 reformulated as:
22 = |AAAAA J
Σ
Au,(
T Σ (Ao,)²
(10)
j=1
j=1
5

---

Published as a conference paper at ICLR 2025
Our method proposes knowledge-awee aware singular-value adaptation, which reparameterizes the task- specific update in the SVD form and guides AU and AV to conform to orthogonality. Given this, we introduce L3 to constrain AU and AV adhere to orthogonality, such that:
C3( ) || AU Ille= + |lAV AV -I
(11)
Overall, methods leverage L1, L2, and L3 to serve jointly for optimizing the model's task performance while adhering to SVD structure. For adjusting L2 and L3. we introduce /3 > 0 and y as their corresponding scalers. The overall training objective of KaSA can be expressed as:
L ΑΣ)
min V.Ax
(L ( AΣ) + (C2(AE) (C3(())
(12)
We present the Py PyTorch-style pseudocode for KaSA and its training objective in Appendix A.
4 EXPERIMENTS
In this section, we evaluate KaSA's effectiveness across different downstream tasks, including natu- ral language understanding (NLU),, natural language generation (NI.G) (see Appendix F.2), instruc- tion following, and commonsense reasoning. For NLU tasks, we evaluate KaSA with RoBERTa (Liu et al., 2021) and DeBERTaV3 (He et al., 2022b) on the GLUE (Wang et al., 2018) benchmark. For NLG tasks, we assess our method with GPT-2 (Radford et al., 2019) on the E2E NLG Challenge (Novikova et al., 2017) benchmark. We further assess the instruction following performance using well-known LLMs, including LLaMA3 8B (Meta, 2024), Mistal 7B (Jiang et al., 2023), Gemma 7B (Gemma Team, 2024), and LLaMA2 13B (Touvron al., 2023b). These models are fine-tuned with different PEFT methods using four synthetic datasets generated by GPT4o, each tailored to summa- rization, classification, coding, and closed QA. GPT4o then employed as a judge to evaluate the fine-tuned models' performance, assigning scores on a scale of 10. We also follow (Kopiczko et al., 2023) and (Gao et al., 2024) to fine-tune the four models on the Alpaca dataset (Taori al., 2023b) and report evaluation results on MTT-Bench, with GPT4 serving as the judge, yielding scores within 10. Additionally, we substantiate KaSA's generality by fine-tuning LLaMA2 7B and LLaMA3 8B models on the Commonsense 70K dataset (Hu et al., 2023), which includes training sets from eight commonsense reasoning datasets, and evaluating them on individual test sets of these constituent datasets. Finally, we conduct ablation studies to investigate the impacts of different components, budget parameter scalability, and the distribution of knowledge-awwe awarr singular values across various layers. All experiments conducted NVIDIA A100-SXMO (80GB) GPUs, except for the NLU experiments, which are conducted on NVIDIA GeForce RTX 3090 (24GB) GPUs.
4.1 BASELINES
We compare KaSA with FFT and 14 PEFT baselines substantiate its efficiency and robustness:
Adapter-based methods We consider four representative Adapter tuning methods as baselines:
1) Adapter" (Houlsby et al., 2019); 2) Adapter (Rücklé et al., 2021); 3) Adapter' (Lin et al., 2020);
and 4) Adapter (Pfeiffer et al., 2021).
LoRA-based methods We select LoRA and its variants: 1) LoRA (Hu et al., 2021); ); 2) DyLoRA Valipour et al., 2023): 3) VeRA (Kopiczko et al., 2023); and 4) DoRA (Liu et al., 2024).
SVD-based methods Considering that our method is associated with SVD, we chose SVD-based
PEFT baselines: 1) AdaLoRA (Zhang et al., 2022); 2) PiSSA (Meng et al., 2024); 3) MiLoRA
(Wang et al.. 2024a); SARA (Gu et al., 2024); and 5) CorDA (Yang et al., 2024).
Other methods Apart from the aforementioned baselines, we also consider other important fine- tuning methods: 1) FFT: and BitFit (Zaken et al., 2021).
To ensure a fair comparison with these baselines, we meticulously replicate the experimental con- figurations as described in previous studies (Hu et al., 2021: Zhang et al., 2022: Gu et al., 2024). Introductions of the baselines and comprchensive details of the experimental setup are provided in Appendix B and Appendix E, respectively.
4.2 NATURAL LANGUAGE UNDERSTANDING
Models and Datasets. For NLU tasks, our method involves fine-tuning foundation models such as RoBERTa-base (125M), RoBERTa-large (355M) (Liu et al., 2021), and DeBERTaV3, V3-ERaaV--base (He

---

Published as a conference paper at ICLR 2025
Table 1: Performance of RoBERTa-base (RoBrase) and RoBERTa-large (arg) with different adaptation methods on 6 datasets of the GLUE benchmark. We report the overall (matched and mismatched) aceuracy for MNLI, Matthew's correlation coefficient (Mec.) for CoLA, Pearson cor- relation coefficient (Pec.) for STS-B, B, and accuracy (Acc.) for all the remaining tasks. We report the average result of five runs with different random seeds. The best results for each dataset are shown in bold. Higher is better for all metrics.
# Trainable Parameters
SST-2 (Acc.)
MRPC (Aoc.)
CoLA (Mcc.)
QNLI (Acc.)
RTE (Acc.)
STS-B (Poc.)
All Avg.
Modeo(Method)
(FFT).(FFT)
125.0M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
RoB BitFFit)
0.1M
93.7
92.7
62.0
91.8
81.5
90.8
85.4
RoBese(Adpt (Adpt") )
0.3M
94.2
88.5
60.8
93.1
71.5
89.7
RoBe...(App."
0.9M
94.7
88.4
62.6
93.0
75.9
90.3
84.2
RoB....L.oRA..
0.3M
95.1
78.4
91.5
RoBe...(AdaL.... (AdaLoRA)
0.3M
94.5
88.7
62.0
93.1
81.0
90.5
85.0
RoBe...(D) (DyLoRA)
0.3M
94.3
89.5
61.1
92.2
78.7
91.1
84.5
RoB
(PISSA)
0.3M
95.0
88.2
65.5
92.0
75.1
90.4
84.4
S.Be...MiLoff
0.3M
94.6
88.7
63.1
80.5
91.3
RoB
(KaSA)
0.3M
95.2
90.7
65.8
93.3
81.6
91.1
86.3
RoB
RoBarg (IFFT
(FFT)
355.0M
96.4
90.9
68.0
94.7
86.6
92.4
88.2
RoBargeAAppt
3.0M
96.1
90.2
83.8
92.1
87.6
RoBaags (Addf)))
0.8M
96.6
67.8
80.1
86.8
RoB Bags (Adpt"))
6.0M
96.2
88.7
66.5
94.7
83.4
91.0
86.8
RoB (
0.8M
96.3
87.7
66.3
94.7
72.9
91.5
84.9
RoB orge (LoRA)
0.8M
96.2
90.2
85.2
92.3
87.8
RoB (KaSA)
0.8M
96.9
91.2
69.4
94,9
88.8
92.5
89.0
al., 2022b) using the GLUE (General Language Understanding Evaluation) benchmark (Wang et al., 2018). The GLUE benchmark encompasses a wide array of datasets designed to test various aspects of NLU, including question answering, natural language inference, sentiment analysis, and textual entailment. In this context, our evaluation is conducted across 6 datasets from the GLUE: SST-2, MRPC, CoLA, QNLI, RTE, and STS-B. Detailed statistical information about the GLUE benchmark can be found in Appendix C.I.
Implementation Details. Basically, we follow the experimental setup applied in (Hu et al., 2021; Zhang et al., 2022) to ensure a fair comparison. We randomly initialize the knowledge-aware sin- gular values without bias, which only introduces negligible coefficients evaluated datasets in GLUE, we meticulously tune the hyperparameters. including the learning rates Ir [1E-5, 1E-3), the rank of SVD truncation & E {1,2, 4, 8, 16, 32, 64, 128), and two trade-off loss coefficients 3 [1E-5, and 7 [1E-5, 1]. The results present are the median outcomes from runs, each conducted with a distinct random seed. To maintain fair trainable parameters, we fine-tune the query and va lue weights in each Transformer block and set rank 8 across all datasets. More detailed hyperparameters are presented in Appendix E.
each layer. For
Main Results. Table 1 presents the performance of RoBERTa- base and RoBERTa-large models fine-tuned using our KaSA contrast to PEFT baselines. KaSA achieves the best performance across all datasets except MRPC and STS-B for the RoBERTa-base model. Notably, KaSA registers the highest average performances for both RoBERTa models: 86.3% for RoBERTa-base and 89.0% for RoBERTa-large. This underscores the effectiveness, adaptability, and scalability of our proposed approach. In a significant comparison with FFT, our KaSA, which utilizes merely up to 0.24% (approximately 0.3.//125, .0M) of trainable parameters, outperforms FFT in 13 out of 14 scenarios and matches its performance on the STS-B B dataset for the RoBERTa-bas model. The results from DeBERTaV3-base are presented in Appendix F.I.
4.3 INSTRUCTION FOLLOWING
To validate KaSA's adaptability and versatility, we extend our experiments to include instruction tuning of LLaMA3 8B (Meta, 2024), Mistral 7B (Jiang et al., 2023), Gemma 7B (Gemma Team, 2024), and LLaMA2 13B (Touvron al., 2023b). We fine-tune the models using four synthetic instruction-followiii datasets produced by GPT4o, each containing 128K samples, covering tasks such summarization, classification, coding, and closed QA. Additionally, fine-
Models and Datasets.

---

Published as a conference paper at ICLR 2025
Table 2: Instruction following evaluation results with average scores for the most popular LLMs fine-tuned on the 128k synthetic datasets and the Alpaca dataset, and evaluated by GPT4o and GPT4 with the scores within 10 on test subsets and MT-Bench, respectively.
# Trainable
Model
Method
Classification
Summarization
Coding
Closed QA
MT-BBench
Parameters
FT
2.41
3.07
2.56
FFT LoRA
8,54B
5.58
7.78
7.61
8.88
4.69
3.21M
5.98
7.29
7.75
8.18
4.32
Gemma 7B
PISSA
3.21M
6.23
7.80
8.22
4.66
MiLoRA
3.21M
6.30
7.62
7.71
8.27
4,53
w/o FT
2.31
2.81
2.32
3,02
1.16
FFT
7:25B
6.73
7.18
7,53
8.75
4.22
Mistral 7B
PISSA
3,40M
5.46
5.86
6,41
7.24
4,24
MiLoRA
3.40M
6.52
7.28
4.29
KaSA
3.41M
5.72
6.82
6.74
7.75
4.58
w/o FT
2.04
2.03
2,86
3,33
3.11
8:03B
4.11
LoRA
3.40M
6.12
7.20
7.37
6.02
4.19
LLaMA3 8B
PISSA
3.40M
6.35
7.59
6.18
4.26
MiLoRA
3,40M
6.37
7.61
7.65
6.39
4.32
KaSA
341M
6.55
7,83
7,89
6.81
4,71
FFT
13,02B
5.86
7.93
7.88
8.97
4.37
LoRA
6.55M
6.23
7.38
7.54
6.25
4,43
LLaMA2 13B
PISSA
6.55M
6.47
7.45
7.83
6.54
4.39
MiLoRA
6.55M
6.45
7.63
7.85
6.82
4,51
KaSA
8.09
4.95
tune using the Alpaca dataset (Taori et al., 2023b) and report the evaluation results MT-Bench (Zheng et al., 2023), with GPT4 serving as the judge, yielding scores within 10. The detailed processing and statistical information of the synthetic datasets, Alpaca, and MT-Bennch are presented in Appendix C.3 and C.4, respectively.
the sum- marization, classification, coding, and closed QA subsets from the "No Robots" (Rajani et al., 2023) dataset seeds create distinct synthetic datasets via GPT4o. We fine-tune the mentioned LLMs using these datasets and then prompt each fine-tuned model to generate four responses based on prompts sampled from the test subsets of the seed dataset. To ensure fair comparisons, we maintain a consistent fine-tuning and inference configuration across all fine-tuned models. We subsequently use GPT4o as a judge to apply single-answer grading strategies to evaluate the response quality of the fine-tuned LLMs on scale from to 10. For the Alpaca dataset, we fine-tune the specified mod- els and prompt them to generate responses to questions from MT-Bench, with GPT4 serving as judge, assigning scores within 10. Detailed prompts for data synthesis and performance evaluation, along with hyperparameter settings, are presented Appendix C.3, D, and E.3, respectively.
Implementation Details. Following the experimental setup in (Park al., 2024),
Main Results. In Table 2, the results show that KaSA consistently surpasses LoRA, PiSSA, and MiLoRA across four 128k synthetic datasets, regardless the model used. Notably, Gemma 7B and LLaMA3 8B, fine-tuned with KaSA, even surpass FFT in the classification, summarization, and coding datasets. In the evaluation using MT-BBnch,, KaSA consistently outperforms FFT and PEFT baselines all models, showing remarkable efficacy. With significance tests showing in 9 out of 12 experimental settings on MT-Bench, KaSA demonstrates significant performance improvements over LoRA, PiSSA, and MiLoRA. These results further highlight the effectiveness, robustness, and adaptability of our method.
0.05)
4.4 COMMONSENSE REASONING
Models and Datasets. Following (Wang et al., 2024a), we fine-tune the LLaMA2 7B (Touvron et al., 2023a) and the LLaMA3 8B (Meta, 2024) models using the Commonsense 170K dataset, aiming to conduct a comprehensive evaluation across eight well-known commonsense reasoning tasks: BoolQ (Clark al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap al., 2019), HellaSwar (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021 1), ARC-e, - (Clark et al., 2018), and OBQA (Mihaylov et al., 2018).
8

---

Published as a conference paper at ICLR 2025
Table 3: Performance comparison of LLaMA2 7B and 3.aMA3 8B with different adaptation meth- ods on eight commonsense reasoning datasets. The symbol f indicates that the results are taken from (Wang et al., 2024a). The best results are shown in bold. Higher is better for all tasks. denotes
that the best results do not surpass ChatGPT.
Model
Method
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
Avg-
ChaGGTS
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
LoRAi
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
PISSAt
67.6
78.1
78.4
76.6
78.0
75.8
60.2
75.6
73.8
LLaMA2 7B
MiLoRAt
67.6
83.8
80.1
88.2
82.0
82.8
68.8
80.6
79.2
KaSA
73.6
84,4"
80.2
91.5
84,5
84.7
72.1'
81.2
81.5
LeRAf
70.8
85,2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
PISSAT
67.1
81.1
77.2
83.6
78.9
77.7
63.2
74.6
75
LLaMA3 8B
MiLoRAt
68.8
77.2
92.9
85.6
86.8
75.5
81,8
81.9
KaSA
73.6
88.1
80.4
94.7
85.5
89.7
79.4"
85.6
84.6
Base
SVD
Adaptive Singular-Valuu
Sing ular-Valla Regularization
Orthog onall Regularizatin
MRPC
CoLA
RTE
66.0
Maltheas Coaft
65.5
81
90.0
65.0
89.5
64.5
64.0
88.5
Figure 2: Components ablation study about knowledge-based SVD truncation, knowledge-aware singular value adaptation, singular value regularization L2, and orthogonal regularization L3 on MRPC, CoLA, and RTE datasets.
Implementation Details. To ensure fair comparison, we implement our KaSA within the LLM- Adapters framework (Hu et al., 2023), following MiLoRA (Wang et al., 2024a). We adhere strictly to the hyperparameter configurations for training and evaluation specified by (Wang et al., 2024a) and (Hu et al., 2023), without any tuning, such as tuning the training epochs and learning rate. For detailed hyperparameters utilized, refer to Appendix E.4.
Main Results. As illustrated Table 3, KaSA consistently surpasses all established baselines for both LLaMA2 7B and LLaMA3 8B across all eight benchmarks when using identical hyperparam- eter settings. Notably, KaSA achieves the highest average score, reflecting significant performance improvements across diverse range of reasoning tasks. These results, obtained from rigorously controlled comparisons, align with our observations in NLU, NLG, and instruction following tasks. This consistency further corroborates the robustness and superiority of our method.
4.5 IN-DEPTH ANALYSIS
Components Ablation Study. Our method encompasses four principle components: knowledge- based SVD truncation, knowledge-award singular value adaptation, singular value regularization L2, and orthogonal regularization La- To examine the collective contributions of these compo- nents, we conduct ablation experiments on MRPC, CoLA, and RTE datasets from GLUE using the MOBERTa-base. Specifically, compare KaSA with the following variants: (1) standard LoRA (as the base); (2) SVD truncation + LoRA; (3) SVD truncation + knowledge-aware singular-value adaptation: (4) SVD truncation knowledge-awarc singular-value adaptation + L2; (5) SVD trun- cation + knowledge-awand
singular-value adaptation + L2 + L3. From the results in Figure 2, we observe that the model performances continually increase more components are involved in the fine-tuning. The fifth bar in Figure 2 shows that variant (5), the full implementation of KaSA, achieves significant performance improvements across all three datasets. Conversely, excluding any of these components results in performance declines ranging from 2.05% to 3.25%, underscoring their collective importance enhancing KaSA's effectiveness. Additional results of the components ablation study on SST-2, QNLI, and STS-B datasets are detailed in Appendix F.3.
Phtpps//githh...oo///aa AGI Fad--nagrneennes///II LLMMMlaatersssss

---

Published as a conference paper at ICLR 2025
PISSA
KaSA (Ours)
COLA
8
% Matthows Cor:: Coaft
80
ACCUSS
04.5
Accuracy
02.0
76
8
09.5
74
07.5
8
70
Trainable Parameters (%)
Trainable Parameters (16)
# Tranable Parameters (%)
Figure 3: Budget parameter scalability of fine-tuning RoBERTa-base with LoRA, PISSA, MiLoRA, and KaSA on MRPC, CoLA, and RTE datasets.
MNLI W.
O0P W,
QOP w,
Rution
1011112
1011332
101112 Layer
Layer
Figure 4: The final distribution of knowledge-aware singular values for W and W, upon fine- tuning the RoBERTa-bas model on the MNLI QQP benchmarks. In this context, the r-axis corresponds to the layer index, and the y-axis denotes the position index. Each value signifies the relevance of the associated knowledge.
Budget Parameter Scalability. We compare the performance fine-tuning RoBERTa-base with LoRA. PiSSA. MiLoRA, and KaSA across various scales trainable parameters. Specifically, employ these methods to the query and value weights of the transformer block and use range of ranks {1, 4, 8, 16, 32, 64, 128] to control the parameter scales. Figure shows that KašA consistently outperforms LoRA, as well as the SVD-based baselines, at equivalent parameter scales across various datasets, indicating our method's efficacy and robustness. Moreover, we observe that enlarging trainable parameter scales does not invariably result in performance improvement. Notably, both methods peak performance at r 8, with KaSA enhancing LoRA by 1.96% on MRPC, 2.05% Mec. on CoLA, and 2.53% Acc. RTE.
Knowledge-Aware Singular-Valle Value. The conventional FFT, which updates all parameters indis- criminately, often incorporates irrelevant minimally contributory knowledge to the task at hand, leading to overfitting and a decline model generalization capability (Valipour et al., 2023). To this end, propose novel knowledge-awes task-specific knowledge. To validate our motivation, we visualize the knowledge-aware singular values of W,, and W, when fine-tuning RoBERTa-base the MNLI and QQP benchmarks, as depicted in Figure 4. We can clearly observe that different scales of singular values are allocated across different layers, indicating that dynamically prioritizes knowledge across parameters.
singular value module adaptively activate the relevant
5 CONCLUSION
In this paper, we introduce a PEFT method, KaSA, which incorporates SVD with knowledge-awwe singular values for dynamic activation of parametric knowledge according to its relevance to the given tasks. KaSA commences with knowledge-based SVD truncation of minor singular value components to remove noisy knowledge within the base model. Subsequently, it reparameterizes task-specific updates in the SVD form, leveraging knowledge-aware singular values for dynamic knowledge activation according to relevance. Our extensive experiments on various LLMs across tasks in NLU, NLG, instruction following, and commonsense reasoning reveal that KaSA consis- tently surpasses FFT and a variety of prevailing PEFT baselines across well-known benchmarks and synthetic datasets, highlighting the superiority of method.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Jing Tang's s work was partially supported by National Key R&D Program of China under Grant No. 2023YFF0725100 and No. 2024YFA1012701, by the National Natural Science Foundation of China (NSFC) under Grant No. 62402410 and No. U22B2060, by Guangdong Provincial Project (No. 2023QN10X025), by Guangdong Basic and Applied Basic Research Foundation un- der Grant No. 2023A1515110131, by Guangzhou Municipal Science and Technology Bureau der Grant No. 2023A0310667 and No. 2024A04J4454, by Guangzhou Municipal Education Bu- reau (No. 2024312263), and by Guangzhou Municipality Big Data Intelligence Key Lab (No. 2023A03J0012), Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A0310628) and Guangzhou Municipal Key Laboratory of Financial Technology Cutting-Edge Research (No. 2024A03J0630). This work was also supported by IITP grant funded by the Ko- rea goveramen(MSSIT)RS (01IRS--222-00219995 Development of Access Agnostic wired and wireless integrated optical access technology].
REFERENCES
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about physical com- monsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Weilin Cai, Juyong Jiang. Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts. arXiv preprint 40X14:2407.06204, 2024.
Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. Drone: Data-aware low-rank compression for large nlp models. Advances in neural information processing systems, 34:29321- 29334, 2021.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprini arXiv: arXiv: 1803.05457 2018
Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, (6):391-407, 1990.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llm.. arXiv preprint 13XIv:2305.133144 2023,
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06990 2022.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211-22 11-22,, 1936.
Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter-efficient fine-tuning with discrete fourier transform. arXiv preprint arXiv:2405.03003. 2024.
Gemma Team. Gemma: Open models based gemini research and technology. arXiv preprint arXiv:2403.08225, 2024.
Jihao Gu, Shuai Chen, Zelin Wang. Yibo Zhang, and Ping Gong. Sara: Singular value based adap- tive low-rank adaption. arXiv preprint arXiv:2408. 03290, 2024.

---

Published as a conference paper at ICLR 2025
Demi Guo, Alexander M Rush, and Yoon Kim. cemmeter--effiien transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pp. 4884-4896, 2021.
Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprini arXiv:2403. 14608, 2024.
Haoyu He, Jianfei Cai, Jing Zhang, Dacheng Tao, and Bohan Zhuang. Sensitivity-aware visual parameter-efficient fine-tuning, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11225 18225-18335 11835, 2023.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kink-Kikpattrck, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, 2022a. URL https:///// http::::///peeeeiee/e net. wwwoummdddoonn Ld=ORDcd5Axok.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra- style pre-training with gradient-disentangleed embedding sharing, In The Eleventh International Conference on Learning Representations, 2022b.
Fahrettin Horasan, Hasan Erbay, Fatih Varçı,, and Emre Deniz. Alternate low-rank matrix approxi- mation in latent semantic analysis. Scientific Programmingg 2019(1):009564,, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2021.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Lim-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304. 01933, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682) 2023.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.005/5, 2024.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34: 14:1022- 1035, 2021.
Dawid Jan Kopiezko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310. 1454, 2023.
Gang Kou and Yi Peng. An application of latent semantic analysis for text categorization. Interna- tional Journal of Computers Communications & Control, 10(3):357-369, 2015.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 4592,4597, 2021.
12

---

Published as a conference paper at ICLR 2025
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. Scaling down to scale up: A guide to parameter-efficien efficcin line-tuning arXiv preprint arXiv:2303.15647, 2023.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. In Findings of the Association for Computational Lin- guisties: EMNLP 2020, pp. 441-459, 2020.
Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint wxXiv22402.09353. 2024.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
Zhuang Liu, Wayne Lin, Ya Shi, and Jun Zhao. A robustly optimized bert pre-training approach with post-training. In China National Conference on Chinese Computational Linguistics, pp. 471-484. Springer, 2021.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Wen Tau Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficien language model tuning. In 60th Annual Meeting of the Association for Computational Linguistics, ACL 2022, pp. 6253- 6264. Association for Computational Linguisties (ACL), 2022.
Fanxu Meng. Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02948, 2024.
Meta. Introducing Meta Llama 3: The most capable openly available LLM date. https:
//ai wwwa. www.ooolloooooooo. talog///etae... 1laaa-aaa---mmmm--mm-m- 3/, 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint FrXiv:1809002780 2018.
Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. The e2c dataset: New challenges for end-to- end generation. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 201-206, 2017.
OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. URL UtRlhhtts:///axiv.oom abs/2303.08m
Chansung Park, Juyong Jiang, Fan Wang, Sayak Paul, Jing Tang, and Sunghun Kim. Llamaduo: Limops pipeline for scamless migration from service Ilms to small-scale local 1lms. arXiv preprint arXiv:2408. 13467, 2024.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https: hetps://huggti ngface, co datpsets.asss/aeaee/oo hectttassest//gggiiieeee Hggginggacctt. no s 2023.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7930-7946, 2021.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106. 2021.
13

---

Published as a conference paper at ICLR 2025
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Common- sense reasoning about social interactions. arXiv preprint WrXiv:1904.09722 2019.
Pratyusha Sharma, Jordan T Ash, and Dipendra Misra. The truth is in there: Improving reasoning in language models with layer-selective rank reduction. arXiv preprint arXiv:2312.13558, 2023.
LB Shyamasundar and P Jhansi Rani. Twitter sentiment analysis with different feature extractors and dimensionality reduction using supervised learning algorithms. In 2016 IEEE Annual India Conference (INDICON), pp. 1-6. IEEE, 2016.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
Sudeep Tanwar, Tilak Ramani, and Sudhanshu Tyagi. Dimensionality reduction using pea and svd in big data: A comparative case study. In Future Internet Technologies and Trends: First International Conference, ICFITT 2017, Surat, India, August 31 31-September 2, 2017, Proceedings
1, pp. 116-125. Springer, 2018
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instraction-following llama model. https: /ttggtttubb../////// wwwm/tatus--aae subi wannoodd_appaee 2023a.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-follooinng llama model, 2023b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2r0.v22307.09288, 2023b.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter- efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3266-3279, 2023.
Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parfieter-efffcientt Ilm finetuning. arXiv preprint arXiv:2406.09044, 2024a.
Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. Svd-llm: Truncation-aware singular value decomposition for large language model compression. arXiv preprint arXiv:240..07378, 2024b.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi. and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- genereteeetdd instructions. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguisties (Volume
1: Long Papers), pp. 13484-13508 2023.
Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint 2r..:23312.12111 2023.
Chao Yan, Yankun Zhang, Weiyi Zhong, Can Zhang, and Baogui Xin. A truncated svd- based arima model for multiple qos prediction in mobile edge computing. Tsinghua Science and Technology, 27(2):15-324, 2021.
14

---

Published as a conference paper at ICLR 2025
Miaorui Yang, Yonggang Xu, Kun Zhang, and Xiangfeng Zhang. Singular component decomposi- tion and its application in rolling bearing fault diagnosis. Measurement Science and Technology, 35(1):015120, 2023.
Yibo Yang, Xiaojie Li, Zhongzhu Zhou, Shuaiwen Leon Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models for task-aware parameter-efficient fine-tumie ane-tuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Kang Min Yoo, Jaegeun Han, Sookyo In, Heewon Jeon, Jisu Jeong, Jaewook Kang, Hyunwook Kim, Kyung-Min Kim, Munhyong Kim, Sungju Kim, et al. Hyperclova technical report. arXiv preprint ar S..1v:2244.0195555 2024.
Zhihang Yuan, Yuzhang Shang, Yue Song, Qiang Wu, Yan Yan, and Guangyu Sun. Asvd: activ singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.0582), 2023.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficien fine-tuning for transformer-based masked language-modele arXiv preprint arXiv:2106.10//90 2021.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint 19XX::1905007800 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-effiient fine-tuning. In The Eleventh Inter- national Conference on Learning Representations, 2022.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging aa--as---udge with mt- bench and chatbot arena. Advances in Neural Information Processing Systems, 01444655646666 2023.
15

---

Published as a conference paper at ICLR 2025
A PSEUDOCODE FOR KASA
Algorithm PyTorch-style pseudocode for KaSA.
(nn. Module):
def init (self, rank: int = 8, # kasa rank
alpha: layer: mndule pre-trained layer
alpha
):
definitions self.r rank
self self.alpha =alpha
self scaling alpha rank
11
self features, self. features base mase_layer.. in features,
base layer.out__eaturree features
Step knowledge-based sVD truncation self.sw rank self.in. features self.r
14
S, Vh vorch..inalg... (base layer. weight data, full matrices=Falsee)
wayer.weight.......oo... data U(:, :self, rank] torch. Vhl: avd. rank, :] self base layer base_layer
forch.diag diag (S[: svd rank])
19
Step 2: knovledge-aware singular-value adaptation self delttav Linear (self. in features, feeture, self.r, bias=False) self.delta. sigma nn. Parameter setorrchrannnneete malf.delta. nn. Linear
20
requires graddffuerr
self.out. features, bias=False)
23
forward(self, torch. : Step 3: marge Delta self delta_u torch. diag( (alssellleeeteeeeeeeeeeeeeaaeeee www.deeeledeeeeeeee sigma) self, delta_v result self base Layer_a
Delta (Eq.
26
torch.diag
28
result result torch.einaum einaum ('ijk, k1->ij1', 8, Delta_N) scaling
29
return result
regularization. loss( model: na Module,
32
beta: float, gamma: float
35 )
definitions 0.0
38
13 loss = 0.0
name, param if param. requires grad:
named paremmterraa ():
41
42
$ singular value regularization
delta. _eigma'e in
44
num parampaammmmm += diag norm torch. sum (param ** 2)
45
12_loss diag_norm
norm
47
orthogonsl regularization
48
elif 'deltayv'' in name 'delta_u" u' in name:
if 'delta
matmul result torch matmul (param.T, param)
51
else:
matmul result torch. matmul (param, param.
54
= torch eye (matmul result.ttt... size
devicermatmul result, rew.lldeeeettttiieieeeee device)
diff_I matmul result
matrix loss torch. noxmmdifff p= fxo')
57
13_loss += matrix loss loss
auxi_
(beta 12_loss gamma 13_1oss) num param num_param else 0.
zeturn
B BASELINES
To demonstrate its efficacy and robustness, we evaluate KaSA against FFT and multiple well- regarded PEFT baselines. The descriptions of our selective baselines are as follows:
Full fine-tuning (FFT) initializes the base model with pre-trained weights and biases, up- dating all parameters during fine-tuning, Full fine-tuning typically serves as a comparative performance upper bound for PEFT methods (Valipour et al., 2023).
Bitfit (Zaken et al., 2021) fine-tunes the bias vectors, leaving other model parameters un- changed.
16

---

Published as a conference paper at ICLR 2025
Adapter tuning integrates tunable adapter layers into Transformer blocks, featuring pair of down-projection and up up-projection matrices with a non-linear activation function in be- tween. We compare four Adapter variants: Adapter" (Houlsby ct al., 2019) inserts adapter layers after the attention and the feed-forward block to fine-tune. Adapter" (Rücklé al., 2021) discards non-activated adapters to improve fine-tuning efficiency. Adapter (Lin et al., 2020) employs an efficient design, placing adapter layers after the MLP module and LayerNorm. Adapter (Pfeiffer et al., 2021) applies adapter after the feed-forward layer and employs a two-stage learning strategy to enhance multi-task performance.
LoRA (Hu et al., 2021) only fine-tunes a pair of low-rank matrices to approximate the task- specific knowledge updates, effectively diminishing the number of trainable parameters.
AdaLoRA (Zhang et al., 2022) reparameterizes task-specific knowledge updates in the SVD form and adaptively allocates the parameter budget through pruning the less important singular values.
DyLoRA (Valipour et al., 2023) dynamically trains LoRA for range of ranks, reducing the training time to find a fixed, optimal rank.
VeRA (Kopiczko et al., 2023) employs learnable vectors to adapt shared pair of frozen random matrices across layers to reduce the trainable parameters count.
DoRA (Liu et al., 2024) decomposes the base model weights into magnitude and direction components for fine-tuning, reducing the number trainable parameters.
PiSSA (Meng et al., 2024) performs SVD to portion the base model into principal compo- nents with larger singular values and residual components with smaller ones, fine-tuning the low-rank matrices initialized with the principle components while keeping the residual components unchanged.
MiLoRA (Wang et al., 2024a) also utilizes SVD for parameter initialization but diverges from PiSSA by fine-tuning low-rank matrices initialized with residual components and maintaining the principal ones unchanged.
SARA (Gu et al., 2024) conducts SVD at the initialization stage to adaptively find the appropriate rank for each layer.
CorDA (Yang et al., 2024) performs SVD on the base model, oriented by the covariance matrix that encodes the context the target task. CorDA supports two fine-tuning modes: 1) initializing the tunable low-rank matrices with principal components for enhanced per- formance; and 2) freezing the principle components while using minor components to ini- tialize tunable matrices, thereby preserving world knowledge.
C DETAILS OF BENCHMARK DATASETS
C.1 GLUE BENCHMARK
For natural language understanding (NLU), employ the GLUE benchmark (Wang et al., 2018), which is a widely used benchmark containing a collection of 8 NLU datasets, including CoLA, SST- MRPC, STS-B, QQP, MNLI, QNLI, and RTE. We present the statistical information of the GLUE benchmark in the table below.
C.2 E2E NLG CHALLENGE
For natural language generation (NLG), we utilize the E2E (End-to-End) NLG Challenge dataset (Novikova et al., 2017), which is commonly used for the evaluation of natural language generation models. This dataset includes approximately 42k training samples, 4.6k validation samples, and 4.6k test samples from the restaurant domain. The E2E dataset involves evaluations across five metrics: BLEU, NIST, METEOR, ROUGE-L, and CIDEr. Detailed explanations of these metrics are follows:
BLEU (Bilingual Evaluation Understudy) evaluates the quality of machine -eeeerate text by comparing more numan-generatee reference translations.
17

---

Published as a conference paper at ICLR 2025
Table 4: Overview of task descriptions and dataset statistics within the GLUE benchmark.
Corpus
Task
Train
Val
Test
# Labels
Metrics
Domain
Sentee-Sentence Tasks
CoLA SST-2
Acceptability
8.55k
1.4k
1.06k
Matthews Corr.
misc.
Sentiment
67.3k
872
1.82k
Accuracy
Movie reviews
Similariy and Paraphrase Tasks
MRPC
Paraphrase
3.67k
408
1.73k
Accuracy/F1
News
STS-B
Sentence similarity
5.75k
1.5k
1.38k
Pcarson/Spearman Corr. Accuracy/1
mise.
QQP
Paraphrase
364k
40.4k
391k
Social QA
Inference Tasks
MNLI
NLI
393k
19.65k
19.65k
Accuracy
mise.
QNLI
QANLI
105k
5.46k
5.46k
Accuracy
Wikipedia
RTE
NLI
2.49k
277
3k
2
Accuracy
News & Wikipedia
NIST (National Institute of Standards and Technology) evaluates the quality of machine- generated text by calculating the similarity between machine output and a reference text using weighted average of n-grams precision.
METEOR (Metrie for Evaluation of Translation with Explicit ORdering) measures the alignment between the machine geniirted and reference texts by calculating a score based the harmonic mean of precision and recall.
ROUGE-L (Recall-Oriented Oriented Understudy for Gisting Evaluation) measures the longest common subsequence(LCC)) between the machine output and the reference. It specifically focuses the sequence of words, making sensitive the fluency and order of informa- tion in the generated text.
CIDEr (Consensus-basse Image Description) measures the similarity of the machine- generated text and the seman-generated ground truth by considering both the n-gram over- lap and the consensus among human annotators.
C.3 SYNTHETIC DATASET
For instruction following tasks, we employ synthetic datasets generated using GPT4o, based on the foundational "No Robots" seed dataset (Rajani al., 2023). Task-specific subsets, including summarization, classification, coding, and closed QA, serve as seeds for generating synthetic data through the framework proposed by (Park et al., 2024). Table presents the volume of data samples and token-level statistical information for these task-specific synthetic subsets.
Table 5: Data volume and token-level statistics of the train and test synthetic datasets generated by GPT4o for each instruction-folloiinng task.
Data Volume
Token-level Statistics
Task
Split
Seed
Synthesis
Min
Max
Avg.
Std.
Train
395
128K
10
2,386
95
53
Summarization
Test
25
100
148
1,150
426
245
Train
334
128K
2,159
67
37
Classification
Test
64
520
119
109
Train
334
128K
6,518
151
84
Coding
Test
16
64
49
821
317
189
Train
128K
1,701
135
59
Closed QA
Test
60
126
1,578
378
C.4 ALPACA AND MT-BENCH
Alpaca (Taori et al., 2023a) is well-known instruction dataset that contains 51k instruction- following demonstrations generated by text-davinci-o These data are synthesized using an im- proved self-instruct method (Wang al., 2023). The dataset designed for instruction-tuning LLMMs
18

---

Published as a conference paper at ICLR 2025
to improve their ability to follow instructions. Each sample includes an instruction, input (if ap- plicable), and an output. A specific example is presented below.
"1 "instruction": "Create classification task by clustering the given 1ist of items.",
"input": "Apples, oranges, bananas, strawberries, pineapples", "output": "Class 1: Apples, Oraages 2: Bananas, Strawberries\
3: Pineapples",
"text": "Below is an instructicn that describes a task, paired with an input that provides further context. Write a response that apprepriately compLetes the request, Anlustt.nnnttt AnAnt+ Instruction: AnCreate classi classificatio fication task by clustering the given 1ist of items. An Input: AnApples, oranges, bananas, strawberries, pineapples/m/nttt Response: AnClass 1: Apples, Oranges\nClass 2: Bananas, Strawberries AnCris AnClass 3: Pineapples"
The instruction describes the targeted task to be performed by the model. Each of the 52k instructions is unique. The input can represent the optional input to the task or serve as the additional context the corresponding instruction. The output the response generated by text- davinci-003 to the associated instruction. The Text is the formatted combination of the instruction, input, and output, using the prompt template for fine-tuning models.
MT-bench (Zheng et al., 2023) contains 80 predefined open-ended questions across diverse domains such as writing, roleplay, reasoning, math, coding, extraction, STEM, and humanities. These chal- lenging questions are designed to automatically assess LLM's instruction-followii with advanced service LLMs like GPT-4 acting as judges. Below is an example from MT-bench.
capabilities,
"question_iie id": 101, "category"; "reasoning",
"turns":
"Imagine you are participating in race with group of people. If you have just overtaken the second person, what's your current position? Where is the person you just overtoox?", "If the \"second person)" is changed to \"last person)" in the above question, what would the answer be?"
},
"reference":
"You are in second place.",
10
"Uncertain."
C.5 COMMONSENSE REASONING
The Commonsensel 170K dataset (Hu et al., 2023) contains data samples from eight well-known commonsense reasoning tasks:
BoolQ (Clark et al., 2019) dataset comprises 15,942 naturally occurring yes/no questions, generated in unprompted and unconstrained settings.
PIQA (Bisk et al., 2020) dataset consists of samples structured as multiple-choice ques- tions, each presenting question with two possible solutions that require physical com- monsense to answer.
SIQA (Sap et al., 2019) dataset contains multiple-choice questions regarding the prag- matic implications of social events, which measure LLMss abilities to address social commonsense reasoning.
HellaSwag (Zellers et al., 2019) dataset includes commonsense natural language inference questions, offering context and multiple endings complete it.
19

---

Published as a conference paper at ICLR 2025
WinoGrande (Sakaguchi al., 2021) dataset is structured fill-in-the- two options, designed to test a model's ability to correctly solve the problem using com- monsense reasoning.
tiintth--blann blank task with
ARC-e and ARC-c are the Easy and Challenge Set of the ARC (Clark et al., 2018) dataset, which contains grade-school level, multiple-choice science questions. Notably, the Chal- lenge Set includes questions answered incorrectly by both the retrieval-bas algorithm and word co-occurrence algorithm.
OBQA (Mihaylov et al., 2018) dataset contains multiple-choice elementary-leve science questions requiring multi-step reasoning, use of additional common and provided science facts (open book), and rich text comprchension.
D PROMPT TEMPLATES
Following the typical practices of (Wang et al., 2023) and (Zheng et al., 2023), we leverage two specialized prompt templates: 1) one for generating synthetic datasets and 2) another for evaluating the outputs of fine-tuned LLMs. To be specific, Figure 5 presents the prompt template crafted for generating synthetic data aimed at the summarization task, whereas Figure shows the prompt template for other tasks. We guide GPT4o in generating analogous data samples by using reference and its corresponding response $ gresponse
example pair consisting of prompt $ sinstructt from the training subset of the seed dataset. In addition, the template is designed to request multiple synthetic data samples in a single query, thus maximizing the efficiency of API use. On the other hand, Figure 7 shows the prompt template used for assessing the precision and similarity between the response $ m response and $human response given the same $ nat ruct i on from the test subset of the seed dataset, where the $ symbol indicates placeholder, designed to be substituted with actual data during the runtime. We only report the precision results in our experiments for the sake of brevity. Given the unique features of different downstream tasks, there no optimal prompt template that universally applies. Therefore, the actual content of the prompt template is adjusted to align with the specific requirements of the task for which the synthetic dataset being generated.
Prompt Synthesis Summarization
Generate a series of (instruction, response) pairs that are similar in context and structure to the example provided below. Each pair should consist of concise instruction followed by an appropriate, detailed response. The instruction should pose a clear task or question, while response should provide comprehensive answer solution that could understood someone with basic understanding
subject.
Example pair:
Instruction: Sinstruction
Response: $response
Your task to generate more pairs that maintain this level of clanity and detail, The topic $topic. Write long text of instruction by yourself, then summarize the instruction in Ensure that responses informative and accurate, suitable
educational
Store the generated pairs in JSON format, with each pair as an object within an array. Each object should have two key-velue pairs: "instruction" and "response". For instance:
ccntentts::
"Instruction": "text", "response": "text"), "Instruction": "text", "response": "text"),
Remember to maintain consistency the format and the generated pairs diverse and cover a broad of subjects. You response asked you must add any additional in your response.
Figure 5; Prompt template of data synthesis for summarization tasks by GPT4o.
20

---

Published as a conference paper at ICLR 2025
Prompt of Data Synthesis for Classification, Coding, and Closed QA Tasks
context structure example provided below. Each pair should consist of cancise instruction followed by an appropriate, detailed respanse. The instruction should pase a clear task or question, while the response should provide comprehensive answer solution that could be understood by someone with basic understanding subject.
Generate series (instruction, response) pairs
Example pair:
Instruction: sinstruction
Response: $response
Your task is to generate more pairs that maintain this level clarity and detall. The topic Stopic. Ensure that the responses are informative and accurate, suitable for an educational context.
generated pairs format, 'instructoon' and "response", , For instance:
pair as an object an array. abject key-value pairs:
"contents":
("instruction": "text", "response": "text">, ("Instruction": "bext", "response": "bext"),
Remember to maintain consistency the format and ensure the generated pairs are diverse and cover a broad range of subjects. You must return the response in the asked format and you must not edd any edditional text in your response.
Figure 6: Prompt template of data synthesis for classification, coding, and closed QA tasks by GPT4o.
Generated Text Assessment Prompt
You meticulous evaluator assessing quality response generated a specific instruction. Your task is assign score between 1 and 10 (whole numbers only, no decimals) based on how well the response satisfies the requirements of the instruction, Consider the following onteria:
Completeness:
response address espects instruction?
Relevance: response focused aligned instruction's requirements?
3. Clarity: Is the response clear and easy to understand?
Provide brief justfiictioon
highlighting key strengths weaknesses the Output evailuation in the
following JSON format:
('score': finteger
10], "justification" "[brief explenation score]">
Instruction:
$instruction
Response:
rim_response
Example Output:
"score":
"justiiiation": response complete, relevant, mostly dear, improvement in phrasing."
Figure 7: Prompt template to evaluate the fine-tuned model's response by GPT4o.
TRAINING DETAILS
E.1 NATURAL LANGUAGE UNDERSTANDING
For NLU tasks, we align with the experimental setup detailed in (Hu et al., 2021: Zhang et al., 2022) for a fair comparison. The detailed configurations of KaSA for RoBERTa-base RoBERTa-large, and DeBERTaV3-bann is important to note that our adaptation process for the MRPC, RTE, and STS-B tasks begins with the pre-trained RoBERTa model, rather than model that has already been adapted to MNLI As result, we fine-tune tune the models on all datasets starting from their original pre-trained weights. The results present the median results from 5 runs, each conducted with distinct random seed.
the GLUE benchmark are depicted in Table 6 and Table 7, respectively.
21

---

Published as a conference paper at ICLR 2025
Table 6: The hyperparameters used for RoBERTa-base and RoBERTa-large the GLUE bench-
mark.
Model
Settings
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Optimizer
AdamW
Common
Warmup Ratio
0.06
LR Schedule
Linear
Batch Size
128
32
128
# Epochs
100
100
100
100
10
100
100
40
Learning Rate
SE-04
SE-04
4E-04
4E-04
4E-04
5E-04
4E-04
3E-04
Weight Decay
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
RoBERTapss
KaSA Rank
Tiuery
KaSA
16
KaSA ,
2.4B-3
1E-04
1E-01
IB-04
1E-02
IE-4
2,4E-01
IE404
KaSA
2.4H-4
1K-03
1H-03
IH-03
1H-05
1H-3
2,4H-04
1E-05
KaSA Dropout
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Max Sey. Len.
512
512
512
$12
512
512
512
512
Batch Size
64
32
32
8
32
32
# Epochs
10
10
100
20
100
20
Learning Rate
4E-04
3E-04
3E-04
4E-04
4E-04
3E-04
Weight Decay
0.1
0.1
0.0
0.0
0.0
0.0
RaBeRTatuge
KašA Rank
Equery
Tvaiue
KaSA
16
KaSA 8
IE-04
IE-02
2.4E-01
1E-02
IE-04
1E-03
KaSA 7
1E-04
1E-02
2.4E-04
1E-03
1E-03
1E-02
KaSA Dropout
0.0
0.0
0.0
Max Seq. Len.
512
512
512
512
128
Table 7: The hyperparameters we used DeBERTaV3-base on the GLUE benchmark.
Model
Scttings
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Optimizer
AdamW
Warmup Ratio
0.06
I.R Scheduler
Linear
Batch size
128
32
32
32
32
# Epochs
10
10
100
20
100
20
Learning Rate
5E-4
4E-4
4E-4
4E-4
5E-4
4E-4
Weight Decay
0.0
0.0
0.0
0.0
0.0
0.0
DeBERTaV3-bor
KaSA Rank
Yywery
Pontus 8
KaSA a
16
KaSA /
IE-04
1.0
2.4E-01
IE-01
IE-04
1E-01
KaSA 7
1E-03
1.0
2.4E-04
1E-01
1E-03
1E-01
KaSA Dropout
0.0
0.0
0.0
Max Seq. Len.
512
512
512
512
512
E.2 NATURAL LANGUAGE GENERATION
For NLG tasks, KaSA adheres the experimental setup outlined in (Hu al., 2021; Gu al., 2024) to ensure a fair comparison. The comprehensive configurations of KaSA for GPT-2 Medium and GPT-2 Large models on the E2E NLG Challenge benchmark are depicted in Table 8.
E.3 INSTRUCTION FOLLOWING
For instruction following tasks, adopt the framework proposed by (Park et al., 2024) stream- line the processes of data synthesis, fine-tuning, and evaluation. We fine-tune several of the most popular LLMs, including LLaMA3 8B, Mistal 7B, Gemma 7B, and LLaMA2 13B, using KaSA and different PEFT basclines to facilitate comparative analysis. Detailed hyperparameter configurations provided in Table 9.
22

---

Published as a conference paper at ICLR 2025
Table 8: The hyperparameters for GPT-2 on E2E NLG Challenge.
Stage
Settings
Medium
Large
Optimizer
AdamW
Weight Decay
0.01
Dropout Prob
Batch Size
8
EEpoch
5
Warmup Steps
500
Training
LR Scheduler
Linear
Lahel Smooth
0.1
0.1
Leurning Rate
2E-4
KaSA Rank
Yquery
4
KaSA a
32
KaSA 8
1E-4
KaSA y
IB-3
Beam Size
10
Inference
Length Penalty
0.9
0.8
no repeat ngram size
Table Detailed configurations used for the instruction following task.
Stage
Settings
Classification
Summarization
Coding
Closed QA
MT-Bench
Optimizer
Adam W
Batch Size
Gemma 7B 8, Mitral 7B LLaMA3 8B 16
Epoch
1
Warmup Ratio
0.1
Data Type
Bfloat16
LR Scheduler
Cosine
Training
Learning Rate
2.0E-04
KaSA Rank
"query Tealue
KaSA
16
KaSA /
IE-4
KaSA y
IE.3
KaSA Dropout
0.05
Max Seq. Len.
512
Number of Beams
10
Inference
Length Penalty
0.8
No Repeat N-Gram Size
4
E.4 COMMONSENSE REASONING
We adhere strictly to the hyperparameter configurations for training and evaluation as specified by (Wang al., 2024a) and (Hu al., 2023), without any tuning. The specific hyperparameter configurations used are shown in Table 10.
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 NATURAL LANGUAGE UNDERSTANDING ON DEBERTAV -
As demonstrated in Table 11, the DeBERTaV3-ben results consistently surpass all baseline perfor- mances across the datasets, with the exception of STS-B B, achieving the highest average performance of 88.72%. This further validates the efficacy of our method across different model architectures.
F.2 NATURAL LANGUAGE GENERATION
Models and Datasets. For NLG tasks, employ KaSA and other PEFT baselines fine-tune both GPT-2 Medium (355M) and GPT-2 Large (774M) models (Radford et al., 2019) on the well- established E2E (End-to-Eni EndE NLG Challenge benchmark (Novikova al., 2017), which focuses on restaurant domain information. The statistics of the E2E NLG Challenge benchmark and the evaluation metries applied are detailed in C.2.
23

---

Published as a conference paper at ICLR 2025
Table 10: The hyperparameter configurations for ILaMA2 7B and LLaMA3 8B commonsense reasoning tasks. To ensure fair comparison, these configurations remain consistent across LoRA, PiSSA, and MiLoRA, with the exception of the specific hyperparameters unique KaSA, namely B3 and y, well as PiSSA and MiLoRA, where 32.
Commonsense Reasoning
Hyperparameters
LLaMA2 7B
LLaMA3
Optimizer
W
Batch Size
16
# Epoch
3
Warmup Steps
100
LR Scheduler
Linear
Learning Rate
3E-4
KaSA Rank
32
KaSA a
64
Dropout Proh
0.05
KaSA B
1E-2
IE-4
KaSA 7
IE-3
IE-3
Placement
query, key, value, MLP up, MLP down
Table 11:
Performance of DeBERTaV3-base (DeBes) with different adaptation methods on datasets of the GLUE benchmark. We report the average result of five runs with different dom seeds. The best results for each dataset are shown in bold. Higher is better for all metrics.
Trainable
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
All
Metho(M)
Parameters
(Acc.)
(Acc.)
(Mec.)
(Acc.)
(Acc.)
(Pec.)
Avg.
(eFT)
184.0M
95.63
89.46
69.19
94.03
83.75
91.60
87.28
(Adp)(Adpt")
0.6M
95.30
89.22
67.87
93.76
85.56
91.30
87.17
(eBps(ddpf))
0.6M
95.53
89.22
69.48
93.98
84.12
91.52
87.31
DeB,3(LoRA)
0.3M
94.95
68.71
94.03
85.56
91.68
87.44
(eB.s(AdaLoRA)
0.3M
85,80
90.44
70.04
94.49
87.36
91.63
88.29
(cB.s(PiSSA)
0.3M
95.30
91.42
70.29
93.59
84.84
91.37
87.80
(MBLLo)))
0.3M
95.99
89.71
70.34
94.14
85,92
90.28
87.73
(KUS))
96.22
91.42
70.41
94.55
88.09
91.62
88.72
Implementation Details. We adopt the experimental configurations delineated in (Hu et al., 2021; Gu et al., 2024) for the fine-tuning of query and value weights within each Transformer block, setting rank of r 4. The A optimizer is employed, paired with linear learning rate sched- ule over 5 epochs. The reported results represent the mean outcomes from runs, each initialized with distinct random seed, selecting the performance the last epoch of each run for comparison. For further details on the hyperparameters utilized, refer to E.2.
Main Results. We present the performance comparison in Table 12. As can be seen, our method consistently outshines the baselines in language generation capabilities across various evaluated metrics. More specifically, regarding the GPT-2 Medium model, KaSA outperforms the baselines in out of 5 metrics and achieves comparable performance (72.1 vs. 72.3) in the ROUGE-I. metric with the top-perfore perferronn baseline, SARA. In the GPT-2 Large model, KaSA surpasses the baselines across all metrics, further confirming its superior performance and scalability.
F.3 COMPONENTS ABLATION STUDY ON SST-2, QNLI, AND STS-B
Figure 8 shows the results of ablation studies conducted on the SST-2, QNLI, and STS-B datasets. From the results, we observe that: 1) the model's performance consistently improves with the in- clusion of additional components during fine-tuning: 2) excluding any of these components leads to a decline in performance. These findings align with that observed Section 4.5, emphasizing the effectiveness of each designed principal component of KaSA in enhancing model performance.
24

---

Published as a conference paper at ICLR 2025
Table 12: Performance of GPT-2 Medium and Large models with different adaptation methods the E2E NLG Challenge. For all metrics, higher values indicate better performance. indicates that the results are reported in prior works. Best results are shown bold.
Trainable Parameters
Medho(M)thod))
BLEU
NIST
METEOR
ROUGE-L
CIDEr
GPal-Ma (FFT+)
354.92M
68.2
8.62
46.2
71.0
2.47
-P (Adpt' (
0.37M
66.3
8.41
45.0
69.8
2.40
GPT-2 (Adpt'
11.09M
68.9
8.71
46.1
71.3
2.47
Medium (Adpt"
11.09M
67.3
8,50
46.0
70.7
Medium (LORA*)
0.35M
70.4
8.85
46.8
Medimm (AdaLoRA)
0.38M
68.2
8.58
44.1
70.7
Medium (DyLoRA)
0.39M
69.2
8.75
46.3
2,46
GPT-2Medium VeRA)
0.098M
69.1
8.71
46.3
70.8
2.43
GPT-2 (ARRA)
0.33M
70,4
8.84
46.7
72.3
2.55
GPT-2 Medium (aSA)
0.35M
70,6
8.86
46.9
72.1
2.55
GPT Laree (FFT*)
774.03M
68.5
8.78
46.0
69.9
GPT-21 sego cepp-2 (Adptt*)
0.88M
69.1
8.68
46.3
71.4
2.49
GPT-2; crgo (Adpi=*)
23.00M
68.9
8.70
46.1
71.3
2,45
GPT-2 (LoRA*)
0.77M
70.4
8.89
46.8
72.0
2.47
GPT-2 Large (aSA)
0.77M
70.5
8.90
47.0
72.0
2.50
Base
SVD
Adaptive Singular-Value
+ Singular-Value Regularization
+ DDtthogonal Regularizatin
5ST-2
QNLI
STS-8
95.2
91.2
93.2
Pearson Donn Covet.
95.0
93.0
94.8
%
92.6
30.6
98.6
90.4
94.4
92.2
Figure 8: Components ablation study about knowledge-based SVD truncation, knowledge-awee aware singular value adaptation, singular value regularization L2, and orthogonal regularization L3 on SST-2, QNLI, and STS-B datasets.
F.4 RANK k OF KNOWLEDGE- BASED SVD TRUNCATION
As depicted in Section 1, components of the original base model weight matrix (0)) associated with smaller singular values are identified to contain noise or less relevant information (Sharma et al., 2023; Wang al., 2024a). This presence adversely affect the convergence of model training and its overall efficacy. We propose the truncation of these components to refine the focus of the base model towards more pertinent knowledge domains, thereby mitigating the adverse impacts. Therefore, we delve into the impact of varying the rank (denoted as & E [1, 2, 4, 8, 16, 32, 64, 128)) of SVD truncation the model's performance, using RoBERTa-base on the MRPC, CoLA, and RTE datasets. As illustrated in Figure 9, an enhancement in model performance is observed as increases from 8. Conversely, an escalation i from 8 to 128 results in decrement in performance. This observation highlights the criticality of identifying rank that achieves a delicate balance between incorporating world knowledge with large singular values and excluding disruptive noise information with smaller singular values, thereby optimizing model performance. The adaptive determination of the optimal SVD truncation rank emerges as compelling avenue for future research.
optimal SVD truncation
F.5 RANK / OF KNOWLEDGE-AWARE SINGULAR- VAGULAR-VVAUEE ADAPTATION
We explore the impact of different rank settings on performance across a range of tasks. Specif- ically, our analysis focuses on LoRA, MiLoRA, PiSSA, and KaSA, using ranks ranging from r = 4, 8, 16, 32, 64, 128} on the CoLA, MRPC, and RTE datasets. As presented in Table 13, KaSA consistently surpasses the baselines across various rank settings in 92 out of 96 cases across the four datasets, highlighting the efficacy and robustness of our proposed method. To further our investigation, increase the rank to 128 and compare KaSA with LoRA, DoRA (Liu al., 2024),
25

---

Published as a conference paper at ICLR 2025
MRPC
CoLA
RTE
91.0
% Metheees Cart. Coeft
66.0
81.5
90.5
% Accuracy
68.5
% Accurary
80.5
$5.0
80.0
$4.5
79.5
$4.0
19.0
78.
Rank Truncation
Rank SVD Truncation
of SVD Truncation
Figure 9: The impact of varying the rank of SVD truncation the model's performance across three datasets.
Table 13:
Performance comparison of LoRA and SVD-based baselines on CoLA, MRPC, and RTE datasets across different ranks of knowledge-aware singular-value adaptation.
Dataset
Method
16
64
LoRA
60.08
61.17
63.14
63.77
63.58
63.82
62.70
63.45
MiLoRA
60.84
61.36
63.10
63.07
63.57
64.56
63.60
63.66
CoLA
PiSSA
59.56
62.68
60.57
65.54
61.32
63.31
63.35
63.60
KaSA
63.32
65.58
63.56
65.82
64.39
65.05
64.82
65.06
LoRA
88.73
87.74
88.97
88.73
89
89.95
88.97
88.97
MiL.oA
89.71
89.22
88.48
88.73
88.
90.20
88.73
88.73
MRPC
PiSSA
87.25
87.99
88.24
88.24
89.46 .6
89.71
88.97
89.95
KaSA
89.46
87.99
90,20
90.69
89.95
90.44
90,20
90.44
LoRA
71.84
72.56
75.45
78.70
77.26
77.98
79.78
78.70
MiLoRA
75.09
80.14
79.42
80.51
79.06
79.81
81.59
80.87
RTE
PISSA
68.95
73.29
76.17
75.09
76.90
78.34
76.53
79.42
KaSA
77.62
77.62
78.70
81.59
80.51
81.23
82.67
81.23
CorDA (Yang al., 2024), PiSSA, and MiLoRA. The comparison is conducted by fine-tuning and evaluating the RoBERTa-bee base model on the GLUE benchmark. The results, as illustrated in Table 14, show that KaSA consistently outperforms all baselines across six datasets, with slight excep- tion for the QNLI dataset, where it performs marginally worse than FFT (92.71 vs. 92.8). This is in line with the previous observations, further demonstrating the robustness and scalability of KaSA.
F.6 PARAMETER INITIALIZATION AW- AUAEAVT
In the context of PEFT, the initialization of tunable parameters is pivotal for optimizing model per- formance, evidenced by (Hu et al., 2021; Meng et al., 2024; Wang al., 2024a). As explicated in Section 2.2, PiSSA (Meng et al., 2024) and MiLoRA (Wang et al., 2024a) initialize the low-rank adaptation block by differentiating components based on their singular value magnitudes. under- scores the necessity of exploring the influence of various initialization strategies on the task- knowledge update, represented as AW ΔUAEAV and its consequent impact on model effi- cacy. In this study, adopt default initialization strategy where AU 0 and both A V and AΣ follow a normal distribution (H, o²). We examine three distinct variants initialization strategies: 1) initializing AUAEAVT with W principal; 2) using W minor for initialization; and 3) adopting normal distribution N(u, a²) for both AU and AΣ while setting AV to O. The comparative outcomes of these strategies across three datasets are illustrated in Figure 10. Our analysis reveals that different initialization strategies distinctly affect model performance across various datasets. Notably, our adopted strategy AU = 0. (AV, ΔΣ} ~ (N,,, o²), consistently outperforms the al- ternative variants across all evaluated datasets and metrics. Among the variant strategies examined, initializing with AUAEAVT W Wprineipal demonstrates superior performance on the CoLA and RTE datasets, yet underperforms when utilizing AUAEAV W Wminor on the MRPC datasets. This observation leads to conjecture that the innovative design of our knowledge-aware singular-
26

---

Published as a conference paper at ICLR 2025
Table 14: Performance of RoBERTa-bass with different adaptation methods using large rank of 128 on 6 datasets from the GLUE benchmark. We report the overall (matched and mismatched) accuracy for MNLI, Matthew's correlation coefficient (Mcc.) for CoLA, Pearson correlation coef- ficient (Pec.) for STS-B, and accuracy (Acc.) for all the remaining tasks. The symbols i and indicate that the results are taken from (Gao et al., 2024) and (Yang al., 2024), respectively. We report the average result of five runs with different random seeds. The best results for each dataset are shown bold. Higher better for metrics.
# Trainable
SST-2
MRPC
CoLA (Mcc.)
QNLI (Acc.)
RTE (Acc.)
STS-B (Pec.)
All
Method
Parameters
(Acc.)
(Acc.)
Avg.
FFT;
125.0M
94.8
90.2
63.6
92.8
78.7
91.2
85.2
LoRA*
21M
94.15
82.84
54.24
92.48
64.26
88.58
79.43
DoRA"
21M
93.58
83.58
51.93
92.59
64.98
88.71
79.23
CorDA*
21M
93.12
89.71
59.60
91.49
76.17
90.17
83.38
PISSA
21M
94.61
89.95
63.60
92.90
79.42
90.55
85.17
MiLoRA
21M
94.72
88.73
63.66
92.55
80.87
90.79
85.22
KaSA
21M
95.30
90.44
65.06
92.71
81.23
91.36
86.02
MRPC
CoLA
RTE
90.5
AMAIOV Wover
Wornps
Marthews Carr. Coeft.
65
66
% Accuracy
AU o, (2V Alu.
64
% Accuraty
63
89.5
62
89.0
61
60
Figure 10: The impact parameter initialization on the task-specific knowledge update, denoted as AW = A(USV across three datasets.
value module significantly enhances the model's capacity rapidly identify optimal parameters within a larger parameter search space, thereby optimizing performance.
F.7 SINGULAR- VALUE AND ORTHOGONAL REGULARIZATION
To evaluate the effectiveness of singular- value regularization |
F and orthogonal regularization AA AU I, Ir and A AV I, we adopt the training configuration outlined in Section 4.2. This involves fine-tuming RoBERTabase model on the CoLA dataset using KaSA. We then plot the loss curve of these three regularization terms throughout the training process. As depicted Figure 11, the application of the adapter the query W and value W. matrices results in ini- tial increase followed by decrease in singular-value regularization || This pattern suggests that the model progressively fine-tunes the significance of task-specific knowledge by adjusting the singular values. Intriguingly, the trend observed for orthogonal regularization ||AU AU I, and | AV I, g varies between the query W, and value W, matrices, indicating distinct adaptation behaviors. To elucidate further, within the query matrix W,, the trend of orthogonal regularization | -I mirrors that of the singular-valu value regularization || II ini- tially increasing before decreasing. Conversely, A decreasing and then increasing. In the value matrix W.. the behaviors of ||AUT AU - AU- -r and AV AV I, demonstrate a reversal compared to those observed in the query W This finding diverges from the trends reported AdaLoRA (Zhang et al., 2022). To delve deeper, we examine the overall training loss, depicted in the lower part of Figure 11. It is observed that the overall training loss converges to notably low value (e.g., 0.058) by the end of the training period. Based these observations, or AV matrices may facilitate more efficient search for an optimal representation by narrowing the search space. This premise will be explored in our future research.
TAUTUU AU 1,ll exhibits an opposing pattern,
hypothesize that the imposition of orthogonality
either the AU
27

---

Published as a conference paper at ICLR 2025
|AU AU IF (left y-axis)
I (right y-axis)
|Av AV IF (left y-axis)
Query Wq
Value W.
10.2
7.30
28.4
7.05
10.0
Overall Regularization
7.00
28.2
9.8
6.85
28.0
28.0
6.90
9.6
27.8
6.85
9.4
6.80
27.6
4000
6000
8000
10000
12000
2000
4000
6000
8000
10000
12000
Training Steps
Training Steps
2000
4000
6000
8000
10000
12000
Training Steps
Figure 11: The singular-value and orthogonal regularization (Upper) and overall training loss curve (Lower) on CoLA dataset.
the last layer of RoBERTan Thase
Table Sensitivity of regularization coefficients 33 and y for RoBERTa-base on CoLA, RoBERTa- large on SST-2, and DeBERTa-v3-base on MRPC.
RoBERTa-base CoLA
RoBERTa-large SST-2
DeBERTa-v3-base MRPC
Hyperparameters
= 0.01, = 1.0
0.6581
0.9587
0,9044
B 0.1, 0.0001
0,6334
0,9587
0,8971
0.01, = 0.1
0.6414
0.9622
0.8995
0.0, = 0.0
0.646
0,9599
0.902
0.001, 0.01
0,6358
0,9587
0.9093
0.001, 0.001
0.6553
0.9576
0.9093
0.01, = 0.001
0.6506
0,5092
0.902
8 0.1, 0.01
0.6333
0,9587
0,902
0.0001, 7 0.1
0.6485
0,9622
0,8995
0.01, = 0.0001
0.6347
0,9576
0,9044
8 0.0001, 7 = 0.01
0.658
0,9599
0.9069
1.0, 0.1
0.6241
0.9599
0.8971
1.0, 1.0
0.6291
0.9553
0.9142
0.1,7 = 1.0
0.6436
0,961
0,9093
0.1, 0.1
0.653
0,9587
0.9082
1.0, 7 0,01
0.6397
0.9587
0.8995
0.01, = 0.01
0.6433
0,9576
0.8995
3 0.0001, 7 0.0001
0.6565
0,9687
0.9044
0.0001, 7 0.001
0.6582
0.961
0.9093
0.1,7 0.001
0.6338
0,9599
0.902
5 0.001, 0.0001
0.6504
0.961
0.9093
0.001, 0.1
0.648
0.9679
0.8971
F.8 HYPERPARAMETER SENSITIVITY ANALYSIS
KaSA introduces two key hyperparameters, 3 and 7/ to scale the singular value regularization L2 and orthogonal regularization C3. respectively. To gain a deeper understanding of how these regular- ization coefficients influence performance, meticulously tune the coefficients, / [1E-5, 1]
28

---

Published as a conference paper at ICLR 2025
Table 16: Efficiency and complexity analyses of the NLU task the CoLA benchmark with RoBERTa-base 125M and the NLG task on the MT-Bench benchmark with LLaMA3 8B, using dif- ferent adaptation methods on a single NVIDIA GeForce RTX 3090 (24GB) GPU and an NVIDIA A100-SXM4 (80GB) GPU, respectively.
RoBERTa-buue 125M on Single NVIDIA GeForee RTX 3090 (24GB) GPU
NLU
LoRA
PISSA
MiLoRA
KaSA
# Trainable Parameters
0.23716%
0,23716%
0,23716%
0.23732%
GPU Memory
1638M
1638M
1638M
1650M
Teaining FLOPs (x 2 per sample)
2.0306
1.9270
1.9270
2.1503
Training Latency (per epoch)
9,4868s
9. 9,8825s
9.9267s
11.3679%
Inference Latency (per batch size 32)
0,0173s
0.0108s
0.0165s
0.01 19s
rank W) = m
rank W = m
rank( (W) = m
rank( (W) m
Matrix Rank
rank A W)
rank W)
rank A W)
rank A WW) S
CoLA Performance (Moc.)
63.4%
65.5%
63.1%
65.8%
LLaMA3 SB Single NVIDIA A10-SXXM (80GB) GPU
NLG
LoRA
PISSA
MiLoRA
KaSA
# Trainable Parameters
0.04241% %
0.04241%
0.04241%
0,04242%
# GPU Memory
71023M
71023M
71023M
71095M
# Training FLOPs (x per samplo)
250.
240.2583
240.2583
240.2585
Training Latency (per epoch)
2469 6s
2543.1s
2476.8s
2528.99 9s
Inference Latency (per batch size 16)
0.7898s
0.7687s
0.7705s
0.7771s
rank( W
rank W)
rank(
rank (ink(W)
Matrix Rank
rank A W)
rank (an =
rank A W) =
rank A W) S.
MT Benchehhhh Performance (Scores)
4.1937
4,2625
4.3187
4.7125
and 7 E [1E-5, 1]. and conduct a sensitivity analysis for RoBERTa-bass on CoLA, RoBERTa-large on SST-2, and DeBERTa-v3-bass on MRPC. The results, presented in Table 15, demonstrate that KaSA exhibits robustness to variations the regularization coefficients and
F.9 EFFICIENCY AND COMPLEXITY ANALYSIS
We conduct a comprehensive efficiency and complexity comparison between LoRA and SVD base- lines across different tasks and model scales, as shown in Table 16. The dynamic singular value adaptation introduced in KaSA is a learnable one- miedimmnsioonal vector of size r < m and requires parameter regularizations, incurring negligible training overheads compared to the standard LoRA. In addition, due to the low-rank approximation of the original matrix, we reduce the rank of W from m to m - r. accelerating the inference particularly for small-scale language models like RoBERTa- base 125M (i.e., with small m!).
As can seen, compared to LoRA, KaSA's extra training overhead is less than 20% (resp. 3%) for the NLU (resp. NL.G) tasks, while speeding up the inference by 1.45x (resp. 1.02x) times. When compared to PiSSA and MiLoRA, our method incurs an average of less than 13% extra training overhead for NLU tasks, while maintaining comparable or improved inference latency. For NLG tasks, our method introduces similar training overhead or inference latency.
G INITIALIZATION AND SINGULAR-VALUE ADAPTATION ANALYSIS
In this section, we conduct detailed analysis of initialization dilemmas associated with PiSSA and MiLoRA, and subsequently explore the core advantages of KaSA, aiming to provide comprehen- sive understanding of the foundational principles governing these PEFT methods. Before embark- ing on detailed examination of each method, we summarize the general mechanism underpinning PEFT. Considering a base model characterized by weight matrix ( E Raxm PEFT aims to efficiently fine-tune (0)) by learning a task task-specific update AW with as few trainable parameters as possible, such that the updated weights ( + A W are better aligned with the requirements of downstream tasks. PEFT approaches generally involve keeping the base model ())) frozen during training, while exclusively updating the parameters of AW.
29

---

Published as a conference paper at ICLR 2025
G.1 INITIALIZATION DILEMMAS OF AW IN PISSA AND MILORA
PiSSA employs SVD on the base model weight matrix W E RR* decomposing it as:
W UEVT
(13)
where U Raxm and V E Rm are oeth-onth matrices, and E Rmxx is diagonal
matrix with singular values (f1:..., satisfying (Ø) 2 d2 ... m 2 0). Following the
standard SVD, PiSSA splits the base model into two distinct components: the principle low-rank
matrix W pris which encompasses the largest r singular values, and the residual matrix W res, which contains the remaining singular values:
W W pri + W res
(14)
W pri pri ppr V
pris W res
Urea Cres V
(15)
where Upri Up::: rl. Epri = diag(o o,), V pri = Vl:,: r]. Ures = Ul:,r r :]. Eres = diag(Or+1:.. Tm), and V res Vlrr :]. Subsequently, PiSSA subtracts pri from the base model W(0) to initialize the low-rank matrices for the task-specific update, resulting in:
W Wase = w
W prs
Wres
(16)
|W Winse
W priille
(Ao;)²
(17)
This subtraction of W pri removes the principal components of W(O), which can lead to consider- able information loss and the forgetting of crucial world knowledge. Given that Wpri is the best rank-r approximation of ("), its removal can adversely impact the model's initial representational capacity, potentially resulting in degraded performance. PiSSA subsequently freezes W base and leverages two low-rank matrices, A and B, to learn the task-specific update during fine-tuning, The matrices and are initialized as:
A U pri E pré: B
V Eprt V T pri
(18)
Therefore, in the PiSSA framework, the task specispeiiiic update AW is expressed as:
AW AB Upri Epri
VT V pri: AW
Wpri
(19)
In the initial stage, the value of A WWi is equivalent Wpri- pri- During fine-tuning, the updates to A and B are significantly influenced by their initialization, which is based on U and V prr. As result, the gradient updates predominantly follow the directions of the initial singular vectors associated with the largest singular values. This limits the model's ability to explore the param- eter space and effectively learn new knowledge relevant to the downstream task, as the knowledge presented by the largest singular values in W pri may not be necessary for the downstream task and can negatively impact model performance.
In contrast to PiSSA, MiLoRA subtracts the residual components associated with the smallest singular values from the base model, resulting in:
W base W( W res
W pri
(20)
W pri = U
pri V
pri: W = U'
E
T rea
(21)
res res
where U' pri = U/ r). E pri diag 01 ...... mm V
pri Vl:: -r) U res Ui: :].
E' res diag(am initialize the tunable matrices A' and B' as:
J.). ) and V ree = V(G T :.. MiLoRA subsequently uses U res to
A'= U res E' FEST B'
res T
(22)
During the fine-tuning stage, MiLoRA keeps W task-specific update A W, which given by:
W base frozen and updates A' and B' to learn the
AW
A'B' =
U res E
AW W
(23)
res
res
30

---

Published as a conference paper at ICLR 2025
In the context of SVD, the smallest singular values often correspond to noise long-tail knowledge (Yan et al., 2021; Wang et al., 2024a; Yang et al., 2023; Sharma et al., 2023), which can impede the learning process for downstream tasks. MiLoRA, which initializes A' and B' based on U' U'res and V ress confines the model's learning predominantly the directions of the less significant singular vectors associated with the smallest singular values. This constraint could potentially hinder the model's ability to acquire essential knowledge required for downstream tasks. In addition, the introduction of noise through MiLoRA's initialization can adversely impact the model during the initial stages of training, leading to reduced stability and slower convergence, as observed in Figure 4 of the original MiLoRA paper. The training updates for A' and B' are constrained within the trivial subspace spanned by U' res and V res. which leads to suboptimal performance.
G.2 KNOWLEDGE-AWARE SINGULAR VALUE ADAPTATION OF KASA
In response to the issues of initialization presented by PISSA and MiLoRA, we propose KaSA, which leverages knowledge-awesd aware singular values to activate parametric knowledge based on its relevance to downstream tasks. Our method commences with the knowledge-basee SVD truncation of the minor singular components W noise E RR= that contain the smallest r singular values. This operation effectively filters out the noise from the base mode (0)), resulting in a matrix W world E Raxm that encapsulates essential world knowledge:
W world W
W noise UEV U'res' rea V
(24)
KaSA uses the low-rank matrix W world to approximate W eliminating irrelevant and noisy knowledge while preventing the world knowledge forgetting issue. Following the truncation, KaSA introduces novel parameterization learn AW in the form of SVD:
AW AUAEAV AUTAU- V A. A
(25)
where AU and AV are semi-orthogonal matrices, ensuring the orthogonality condition. The ma- trix AE is trainable diagonal matrix, with knowledge-aware singular values that can be adaptively tuned, allowing the model to emphasize knowledge relevant to the downstream task and providing fine-grained learning pattern. To maintain the orthogonality of AU and AV during training, we add an orthogonal regularization:
Cs( v | AU LllF + |AVTAV AV -,,
(26)
where || more adhere to the SVD's framework, facilitating the seamless integration of AW with W world- Since the A W learned by KaSA is in SVD form, its spectral norm is equal the largest singular value in AΣ, satisfying:
|| ||-||r denotes the Frobenius norm. This regularization can ensure KaSA's learned AW can
| 2 max Ao, |
(27)
3
where Aoy are the adaptive singular values of the diagonal matrix ΛΣ. Therefore, by controlling AE, we can directly control A W's magnitude. This allows for adjustments to the weight updates, enhancing the controllability of the fine-tuning process for downstream tasks. In particular, KaSA's training objective is more comprehensive than that of orthogonal regularization alone. The over- all training objective L includes the task-specific loss L1, the singular value regularization L2, and orthogonal regularization L3. Therefore, the gradients with respect to AU, AV, and AΣ are for- mulated as:
OC QL1
+ AAU AU I,)
(28)
OAU OAU
ØC UL1
+ +4AV AV- --V-I, I,)
(29)
DAV
dAV
OC
OL,
+ 2AA
(30)
AAΣ
OAΣ
The gradients with respect to AU and AV are particularly influenced by the orthogonal regulariza- tion component, which facilitates stable training dynamics. This orthogonal regularization, along with the computed gradients, contributes to maintaining stable parameter updates, thereby mitigat- ing potential issues such as gradient vanishing or explosion.
31

---

Published as a conference paper at ICLR 2025
G.3 SUMMARIZATION
To summarize, our analysis of PiSSA and MiLoRA highlights the dilemmas posed by their initializa- tion strategies while emphasizing the advantages of KaSA for knowledge-aware singular-value adaptation. Specifically,
PiSSA's initialization with principle components associated with the largest singular val- ues potentially lead to world knowledge forgetting and introduce updated knowledge unnecessary for downstream tasks, leading to diminished task performance.
On the other hand, MiL.RA's initialization with minor components associated with the smallest singular values introduces noisy and long-tail knowledge, resulting in reduced training stability, slower convergence, and suboptimal performance.
In contrast, KaSA offers several advantages based on the aforementioned analysis: noise reduction-be by filtering out components with minor singular values, KaSA eliminates their detrimental impacts on task performance; 2) knowledge-awareness through adjusting the knowledge-awwe singular values, KaSA allows the model to adaptively align with the requirements of downstream tasks; and 3) stable
orthogonal regular- ization ensures stable gradient updates and control over the magnitude of weight updates.
H CASE STUDY
We present series of case studies demonstrate the improved performance in instruction-folloiinn of models that have been fine-tuned with our KaSA method. This fine-tuning utilized synthetic datasets generated by GPT4o for each specific task. The cases, illustrated from Figure 12 through 18, compare the response and precision scores of the most widely used LLMs, including Gemma 7B, Mistral 7B, and LLaMA3 8B. These models were fine-tuned with both LoRA and KaSA techniques and evaluated across range of tasks such as summarization, classification, coding, and closed QA. For each case, GPT4o assesses the precision scores of the models' responses. These instances col- lectively showcase not only the adaptability of KaSA A across diverse set of leading LLMs but also its effectiveness in enhancing the performance of pre-trained language models (PLMs) in various downstream applications.
32

---

Published as a conference paper at ICLR 2025
Given that ((x) 4x*3 14, find the value of T(2).
response
To find the value of f(2), substitute 2 into the function f(x) 4xn3 9x 14.
4(2)^3
((2) 4(8)
T(2) 13-118-1
f(2) = 14
((2) 0
f(2)
MT-Bench Score (GPT4o Judge)
Gemma
Using the given function, we can subtstitute 2 Into the equation and solve for f(2):
9
(4(2)^3) (9(2)) (14)
(4(8)) -14
32 32
Therefore, ((2) equal to
((2) (4 2-3) (9 2) (14) 32 18 -14 = -10 Diatego ((2) equel to -10.
3
Gemma
First, need to substitute the value of which 2, Into the function:
f(2)
3
+4(2)33-9(2)-14
-64-18-14
Figure 12: Responses on a math problem from MT-Bench. Each response is generated by Gemma 7B models fine-tuned 51K Alpaca dataset with KaSA, LoRA, and PiSSA methods respectively.
33

---

Published as a conference paper at ICLR 2025
Prompt
In the rield quantum physics, what superposition, and how does it relate to the phenomenon quanium entanglement?
MT-Bench Score (GPT4o Judge)
LLaMA3 88 with KaSA
quantum physics, superposition refers ability quantum system exist in multiple the contrast
physics, where object only example, classical physics, coi can only be either heads or tails, but in quantum physics, the coin can exist in superposition of both states until it observed, at which point it will "collapse" into one state or the other.
same
8
wey the state one partide affects the state the other, even when they are separated by a large distance. other words, the properties of one particle are correlated with the properties of the other particle, regardless of the distance between them.
Quantum entanglement phenomenon
linked
Superposition entanglement entangled, they are their properties are correlated, and they cannot be described independently of eech other. In this way, entanglement can be thought of as a special case of superposition, where the particies are linked together in that allows their properties to correlated.
closely superposition.
concepts quantum physics.
particles state
the particles
Overall, superposition states for single quantum system, while entanglement refers correlation between the properties two or more entangled particles. These two concepts are closely related and play a crucial role understanding the behavior of quantum systems and their potential applications in areas such as quantum computing and quantum communication.
fundamental concept quantum physics
multiple
LLaMA3 88 78 with PISSA
Superposition fundamental principle quantum mechanics, which contrest atorn or photon, can where an object can only exist in one state at time.
system, world,
states same
8
Quantum entanglement phenomenon that occurs when two more particies are linked such that be described independently of the others, when particles measuring the affect the state of the other partide, regardless of the distance between them.
the quantum state each particle separated large
particle instantaneously
Superposition and entanglement are closely related concepts in quantum physics, In fect, superposition is necessary condition for entanglement occur. When two particles are entangled, they said to be superposition states, meaning that states. This
described being in entanglement
being combination allows for the transfer of information between particles that are separated by large distances.
powerful phenomenon,
In summary, superposition refers to the ability of a quantum system to be in multiple states simultancously, that
while entanglement refers described independently when particies invelved state superposition.
them from being between concepts closely related, entanglement can only occur other.
quantum
LLaMA3 BB 7B with LoRA
Superposition fundamental principle quantum mechanics states quantum system, suth atom or a photon, can object can only exist in one state at time.
multiple states simultaneously. This contrast to classical physics, where an
6
Quantum entanglement, the other hand, system dependent linked in systems are entangled, they measured in a particular state, the other system will instantaneously assume the corresponding state, even the two systems are separeted by large distance.
that
other. When systems
way
to in superposition of states, This means that If one system is
Superpasition and entanglement superposition allows to demonstrate the existence of superocsition. Together, these two concepts form the foundation quentum mechanics and have far-reaching implications for our understanding of the nature of reality and the limits of our ability to predict and control the behavior of matter at the atomic and subatomic levels.
physics. principle of related concepts phenomenan entanglement can existence entangled states,
Figure 13: Responses on STEM problem from MT-Bench. Each response generated by LLaMA3 8B models fine-tuned on 51K Alpaca dataset with KaSA, LoRA, and PiSSA methods respectively.
34

---

Published as a conference paper at ICLR 2025
Prompt
compleined principal bullying during aides to be vigiiant, Which situation should the aides report to the principal?
principal to quickly
instructing recess
a) An unengaged girl is sitting alone on a bench, engrossed in book and showing no interaction with her peers.
b) Two boys engaged in one-on-one basketball
are involved heated argument regarding the last scored basket. has surrounded another girl appears boys huddled handheld game, against rules not permitted on school grounds.
group
taken possession backpack.
Target response
<) group girls surrounded another girl appears possession backpackk
MT-Bench Score (GPT4o Judge)
Mistral with KaSA
The situation that should be reported to the principal option C. Bullying is defined as unwented, aggressive behavior among school-aged children that involves a real or perceived power imbalance. The behavior is repested, or has the potential be repeated, over time. this case, a group girls hes surrounded and laken possession another girl's backpack, which dear example of builying behavior. The principal should made aware this situation so that appropriate action can be taken to address the issue and ensure the safety and well-being all students.
10
On the other hand, options A, and D not meet the criteriaa for builying. In option A, the girl is engaged a solitary activity and does not appear to be interacting with her peers a negative way. In option B, while the boys may be engeged in an argument, it is not dear if it is releted to bullying behevior or simply disegreement over the rules of the game. Finelly, in option D. while the use of handheld video games against the school's rules, it does not meet the definition of bullying as it does not involve power imbalance or repeated aggressive benavior.
principal, involved. Option to the principal the situation escalates or becomes serious. Overall, it important for the aides to remein vigilant report instainces of bullying to the principal in order to maintain safe and positive learning environment for students.
conclusion, criteria
described option requires report attention should be manitored by
do recessarily require
The situation thet should be reported to the principal is option C. Bullving is defined as unwented, aggressive behavior among school-aged children that involves real perceived girls important for address the issue and ensure the sefety and well-being all students. Option does not involve bullying, es the girl engaged in solitary activity and does not appear to be interacting with her peers in negative way. Option B involves disagreement between two boys, which is not necessarlly bullying behavlor. Option involves violation of school rules, conclusion, option
imbalance. In this case, example bullying.
possession another girl's backpack, appropriate action can principal
9
report behavior
does not involve appropriate situation to school principal, it involves bullying benavior that needs to addressed immediately prevent further harm to the victim and ensure safe and positive learning enviranment for all students.
is not bullying behavior
imbalance
aggression towards another
Bullying the use of force, threat, coercion to abuse, intimidate, exduuson dominate others, Bullying take form verbal harassment, physical habitual. can occur variety settings, including schaols, workplaces, online, have seriaus negative consequences for both the victim and the perpetrotor.ilkpeddi
A recess aide school staff member who is responsible for supervising students during recess time. Their role to that students safe, engaged appropriate activities, and following the school's rules and reguletions. wikipedia
School princpplss are educational leaders of school. They are responsible for the overall management and operation of the school, indluding the development and implementation of school policies, the supervision of staff and students, and the maintenance of sefe and supportive learning environment. wikipedia
Figure 14: Responses on a reasoning problem from MT-Bench. Each response is generated by
Mistral 7B models fine-tuned on 51K Alpaca dataset with KaSA and MiLoRA methods respectively.
35

---

Published as a conference paper at ICLR 2025
Prompt
What type of story would you dlassify the emojis below as?
Pick from the following categories: comedy, action, animation, romance, horror, westerm, and fantasy. PS A
Target response
cetegory
Score (GPT4o Judge)
LLaMA3 BB with KaSA
classify ematis horror, depict person walking dilapidated feeling warried, then opening suspense, aporoaching zombie.
hallmarks
10
genre. emojis' visuels ceptions convey sense supernaturel, frightening events, such, cetegorize emojis
LLaMA3 8B with LoRA
Horror The emojis suggest story filled with cene encounters and spine-tingling suspense. With the walking old building, sed face, door, zombie, frightened face, running with mask, paints picture unnerving
9
LLaMA3 with PiSSA
The emojis in this case are mestly related to the horrar and fantasy categorics. The S emaji represents fantasy world where the and
zombie, which character has supernstural abilities. emojis é, II and Therefore, this story can be classified as horror and fantasy
harror and the 3 combination
6
Figure 15: Responses on classification problem from the test split of "No Robots" dataset. Each response is generated by LLaMA3 models fine-tuned on 128K synthetic dataset with KaSA, PiSSA, and LoRA methods respectively.
36

---

Published as a conference paper at ICLR 2025
Prompt
Summarize the following story in few sentences.
Use the text following Please summarize this post. Make easier read and much shorter. should be about paragraph govermments. law set it, designating
request. After World created many rallroad scheduling and safety problems, however, Congress passed Uniform nationwide dates of daylight seving time from the last Sunday April until the last Sunday in October. In 2007, Congress amended the act to expand the period in which daylight saving time in effect from the second Sundey in March to the first Sunday in November dates that remain in effect today, The Uniform Time Act allows states territories to out of daylight saving time, however. Arizona Hawali
daylight saving
permanent standard many other states considering permanentt standard time, springing have legislation and resolutions under consideration to support
Virgin Islands, Northern Mariana Islands, stop falling
Samoa.
U.S.
many others have been cansdering permanent daylight saving time. Legisiation and resolutions for permanent standard time have increased from 15% in 2021 to 31% in 2023. in March 2022, the U.S. Senate passed the Sunshine Protection Act bid to make daylight saving time permanent. forward with this legislation. Florida Sen. Marco Rubio reintroduced the bill on March 2023. The spike twice-yearly changes raflects altogether, and choose permonent
But the House did not activity among practice. Now, up legisiators saving time.
people recognizing downsides
whether
daylight.
Target response
After WW11, states were given the right to choose the start and end dates for daylight saving time, but this caused serious problems for rail schedules. To address this, Congress passed the Uniform Time Act in 1966 which standardized the changes, but still allowed states and territories to opt out entirely. Arizona, Hawall, Puerto Rico, and some U.S. temtories dacided to stay an permanent time. Many other states and legislators are considering abolishing paylight saving time, but na bill has passed. People are starting to resagnize the problems caused by changing the time twice a year and want to move to a permanent standard time.
Score Judge)
Gemma 7B with KašA
the Uniform deternined March the first
daylight saving daylight scheduling second Sunday Sunday April the last Sunday October. 2007, daylight time. States cansidering
8
November. States
permanent standard time daylight The Senate the Sunshine Protection March 2022, but House did not take action. reintroduced in March 2023. This shift toward permanent time reflects recognizing the downsides the changeover.
15% 2021
including Florida.
Gemma LoRA
Daylight saving time originally established energy by edjusting the clocks based changing managed daylight saving pertod of daylight saving time.
daylight hours. However, govemments. 2007, Congress passed Energy Policy Act, extending States have the option to opt out, but some, like Arizona and Hawail, have permanent standard time. Many states are also considering eliminating the twice-vearly time shifts, with 31% supporting permanent standard time and 21% supporting permanent daylight saving time. 2022, the Sunshine Protection Act was passed by the make daylight saving permanent, didn't reintroduced
7
further action
Sen.
2023.
Gemma with PISSA
Daylight saving time started in the after World War II, with states setting their dates. Congress passed the Uniform Time Act in 1966 to standardize these dates, extending them from the last Sunday April the first Sunday in November. States can opt out, end Arizona and Hawaii are on permenent standard time. The Sunshine Protection Act, introduced in 2023, aims to make daylight seving time permanent. Cfr. the text.
6
Responses summarization problem from the test split of "No Robots" dataset. Each response is generated by Gemma 7B models fine-tuned on 128K synthetic dataset with KaSA, LoRA, and PiSSA methods respectively.
Figure 16:
37

---

Published as a conference paper at ICLR 2025
Prompt
From what is mentioned below, how long does the quaking aspen live average?
The quaking aspen (Populus tremulaides), name refers how leaves tremble at the slightest breeze, tree willow family. Other common names refer to such features as its fall color (goiden aspen), bark color (white espen), ar favorite habitat (mountain aspen). It's easily identified by its smooth, light-calored hark, interrupted at intervals by darker knots and horizontal scars. An even better-identifying feature is its fiattened leafstalks, which cause the leaves to "quake" when the wind blows. The leaves are rounded to slightly triengular, inches every lendscape, but if you wish grow it, the area, grows about feet a year. average, lives about yeors and up to years the western United States. An Iconic of the Rocky Mountain states, quaking aspen is a maghtmaintenance plant due to its ability to spread unless it is large stand of quaking aspen that you desire over time. Such stand or colony could be suitable for an isclated area of your property. But to grow single quaking espen
and finely toothed. The flower seedhead is catkin. Quaking aspen isn't the best spring,
fest-growing
the frost date your
specimen plant shade tree, must continually remove septic systems, It's not technically invasive is referred
plants from the root system avoid colonizer,
growing unwanted
protect foundations sending many genetically identical trees immediate vidniiy, Grow quaking aspen tree in full sun best results, although does tolerate some shede.
Target response
On average, the quaking aspen lives for about 60 years and 150 years the western United States.
Score (GPT4o Judge)
Mistral 7B with KaSA
The quaking lives average about This best planted spring before the last frost date. grows full but tolerates shade.
8
7B with MiLoRA
The quaking aspen lives on everage for 60 to 150 years. The instructions below explein how to properly set up a new WordPress website 1. Create an account with a web hosting provider. 2. Choose e comein name for your site. 3. Install WordPress on your server using the hosting provider's tools. 4. Configure your WordPress settings, including domain and time zone. 5. Choose theme for your website s apgearance, 6. Customize your website content with WordPress's 5 built-in tools, 7, Ensure your site is secure and becked up regulary. 8. Add any required plugins for additional functionality. 9. Test your site thoroughly before launching it. 10. Market your website to attract visitors, Below are some specific details recarding this process: Creeting en account with a web hosting provider: Choose a reputable web hosting service like SiteGround, Bluehost, or GoDaddy, These services offer competitive pricing and easy-to-use interfaces. 2. Choosing domain name: Select a unique name that reflects your business or the purpose of your website. Avoid numbers and hyphens your domain name. Installing WordPress on your server: Most web hosting providers offer one-dlik installation control panels. 4. Configuring WordPress settings:
7
WordPress directly from
WordPress dashboard premium themes thet can easlly settings. Choosing theme: and customized. content: Use WordPress's security
configure settings like your
utle, tagline, time zone,
WordPress
vast arrey
built-in that aligns your website's style purpose, Customizing manage content, such as posts, pages, backups: Install security plugins Wordrence regularly backups your site using
Mistral 7B PISSA
average, the quaking aspen about years up 150 years. source includes information: The quaking aspen (Populus tremuloides) is tree in the willow family, It's easily identified by its smooth, light-calored bark, internupted at intervals by darker knots and horizontal scars. An even better- identifying feature is its flattened leafstalks, which cause the leaves to quake' when the wind blows. The leaves are rounded slightly triangular, inches across, and finely toothed, The flower or seedhead grows about
5
catkin.
Quaking aspan gast-growing
60 years and western United States. lcanic tree the Racky Mountain states, quaking aspen
year. average,
150 years math-mainteeecce plant due to its ability to spreed unless it a large stand of quaking aspen that you desire over time. It's not technically invasive but is referred to as colonizer, sending out many genetically identical trees in the immediate vicinity. The best time to plant quaking aspen is spring, after the last frost date in your although tolerates quaking aspen. According
Quaking aspen grows best about the physical features and meintenance requirements aspen is fast-growing tree that can grow up to 2 feet per year. On everage, it lives about 60 years and up to 150 years, but can vary significantly depending on the region. It requires full sun to grow optimally and known for its ability to spread, which is why regular maintenance necessary control the grewth. The text
shade. The
includes details quaking
Responses on ClosedQA problem from the test split of "No Robots" dataset. Each response is generated by Mistral 7B models fine-tuned on 128K synthetic dataset with KaSA, MiLoRA, and PiSSA methods respectively.
Figure 17:
38

---

Published as a conference paper at ICLR 2025
Prompt
This lava function below not warking es expected, it's supposed to go through arrey of numbers and if it finds number in the arry that divisible by with remainder D, then adds to new and returns the new array at the end. Can you help debug the function explain why working expected?
public chockforÉvens((nt nums) ArrayList new ArrayLis((); for(int 0; lums.length; I+++ (i(nnms(i) f answer.aid(nums[1
Target response
Certainly! The function 'checkForEvens' iterates through an array and number in the array is divisible by 2, then it gets added to answer which is later returned.
The reason the function get divided looking
returning the expected result because the statement, 'If (nums[T) /2 == The statement array
align
you expect function
Instead, you want to use the modulo operator to check the remainder number that being divided by number is divided by 2 and returns remainder of D, then we can assume that it's an even number. So, the small change of 'if (nums[]] % 2 = 0j should salve your lssue.
rewritten
public static List checkForEvens((ml nums) ( ArrayList anwer new brrayListsInteger for(int = 0; numss.ength i++)] if(nums(i] -- answeradd(nums((
Score (GPT4o Judge)
Gemma 7B with KašA
typo your
9
Here is the corrected version the function:
public static List<Integer>> checkforEvens(int]] nums) List-cIntager>> answer for (int num nums]
(r
If (num % 2 -- 0) answer.dd((uum);
answer;
Gemma 7B with PiSSA
Your function is close to working correctly. There are few simple fixes that need to be made. First, the madulus operator %% needs used 'if" condition check for numbers, Second, the "add' needs retur" statement Here's the comeeted version: "jeva
ArrayList" with element numsfi]" of numsfi] Finally, included in order to return the even numbers.
10
List<enteggr>> checkForévenn((in]] nums) List<Integer> answer ArrayList<>));
for (int 0; x numsslength; [+++)
(nums[i) 0) answeraddsnumss[))
return answer;
This function works correctly and will return an 'ArrayList" of even numbers from the input array. If you need help understanding how this fixes your original function, please let me know.
Figure 18: Responses Coding problem from the test split of "No Robots" dataset. Each response generated by Gemma 7B models fine-tuned on 128K synthetic dataset with KaSA and PiSSA methods respectively.
39


Reviews:
[{"review_id": "FF2abIQoRm", "reviewer": "Reviewer_pwGj", "summary": "This paper presents a novel parameter-efficient learning algorithm for LLMs. It addresses two issues of the prior research: i) a subset of model updates are associated with the noise encoded in the base model; and ii) the low-rank matrices A and B are initialized using the knowledge from the base model that is not relevant to the downstream tasks. The new method tackles the two issues with four components: knowledgebased SVD truncation, knowledge-aware singular value adaptation, singular value regularization L2, and orthogonal regularization. The ablation studies show that all four of them are effective for the selected evaluation tasks.", "questions": "It would be great to justify why only MRPC, CoLA and RTE are chosen for the ablation studies.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The evaluated tasks are comprehensive, including NLU, NLG and instruction-following.\n* The ablation study provides evidence to demonstrate the effectiveness of all four design choices on selected NLP tasks.", "weaknesses": "* The improvement of the proposed method over the baselines is marginal. The variances and the results of the significance tests may help justify the improvement.\n* It is unclear why GPT-2 is used for NLG tasks, while Gemma, Mistral and LlaMA3 are used for instruction following tasks. Hence, the experimental design is not systematic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsxpw4oZc5", "reviewer": "Reviewer_9ooo", "summary": "This study proposes a new technique for low-rank finetuning and inference. “KaSA” reparameterizes finetuned weights in two steps: task-specific SVD truncation, and “knowledge-aware singular- value adaptation” that selects particular singular values based on their relevance to particular tasks.\n\nThis study shows that KaSA is competitive with popular low rank approaches such as LoRA across multiple models and tasks. In particular, they report extensive results for RoBERTa evaluated on the GLUE tasks, GPT2 evaluated on the E2E NLG challenge, and three 7-8B LLMs evaluated on four custom synthetic instruction following datasets.", "questions": "1. What exactly is the reader supposed to take away from the visualizations in Figure 4? It seems somewhat unclear. It might be helpful to have a comparison with and without KaSA\n\n2. Why scale between 0 and 100 when using GPT4o as a judge? It seems a bit arbitrary/unnecessary. Do you have strong motivation for using this scaling?\n\nSome small nits:\n* Typo: line 043 “gets popular”\n* Typo: line 426 “four synthetic datasets using GPT-4”; is it GPT-4 or GPT4o?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "KaSA seems like a promising technique, and is competitive with LoRA and SVD style techniques like PiSSA and MiLoRA. The authors do extensive experiments and benchmarking, and the paper is very well written.\n\nMinor notes:\n* The paper has extensive citations that are very helpful to the reader\n* Figure 2 with “ablations” is very informative - the trends are quite clear\n* The “no robots” dataset is a good choice for seeding synthetic data, since it was generated by human annotators.", "weaknesses": "The experimental results are overall quite strong. The main weakness of the manuscript is that it lacks a straightforward explanation of the method. I’m not sure I fully understand the KaSA method here based on the description, and the diagram is slightly confusing as well. It would be very helpful to have a pseudocode block/section.\n\n1. My current understanding is that the method requires (1) LoRA finetuning followed by (2) SVD truncation followed by (3) “knowledge aware singular-value adaptation.” Is this correct? If so, doesn’t this mean that it will require more training/compute than vanilla LoRA? Can you do a direct comparison of training compute/FLOPs for LoRA vs. KaSA?\n\n2. “KaSA employs a small set of parameters Ψ to reparameterize the task-specific update △Θ” is the task specific update full finetuning or LoRA? I assume the task specific update is vanilla LoRA.\n\n3. How do you calculate $W_{fft}$ for $\\mathcal{L}_2$? Do you have to do full finetuning? I assume not…\n\nThere are a few smaller details that I think are worth mentioning:\n\n1. What is the rank for instruction following? I don’t see it in the manuscript\n\n2. It feels like important baselines are missing from Table 4 - how well do the models do out of the box? Since these datasets are created specifically for this study, it is hard to get a sense of task difficulty or to compare to other papers\n\n3. The authors claim no extra inference latency due to the technique. It would be helpful to show these numbers. If I understand correctly, however, there will be extra _training_ latency. Can you expand on this in the paper? \n\n4. The most similar methods to KaSA are PiSSA and MiLoRA. In Table 4 it could be helpful to do a comparison with PiSSA and/or MiLoRA", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LWbV489muH", "reviewer": "Reviewer_dg7a", "summary": "The paper introduces KaSA (Knowledge-aware Singular-value Adaptation), a parameter-efficient fine-tuning (PEFT) method that adapts large language models (LLMs) using knowledge-aware singular values derived from singular value decomposition (SVD). KaSA aims to selectively activate task-relevant knowledge, discarding irrelevant or noisy information to improve model performance on specific tasks. The method consists of knowledge-based SVD truncation to filter noise and knowledge-aware singular value adaptation to dynamically manage task-specific knowledge activation. Experiments across natural language understanding, generation, and instruction-following tasks demonstrate that KaSA consistently outperforms other PEFT approaches, including LoRA and its variants.", "questions": "The SVD truncation stage may discard knowledge components deemed noisy or irrelevant. However, there’s a risk that some truncated components might contain task-relevant information. Did you experiment with different truncation thresholds, or analyze the potential trade-offs of truncation in various tasks? A discussion here would enhance understanding of KaSA’s robustness.\n\nThe SVD rank and regularization coefficients play an important role in KaSA’s performance. Could you provide practical guidelines or a sensitivity analysis to help practitioners choose these values for new tasks or datasets? This could make the method more accessible to those less familiar with PEFT or SVD.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "KaSA combines SVD truncation with a knowledge-aware adaptation process, enabling it to achieve impressive performance on various NLP tasks. KaSA often outperforms other PEFT methods on benchmarks, showing its effectiveness in balancing performance and efficiency. The experiments are thorough, covering natural language understanding, generation, and instruction-following.\nThe ablation studies break down how each component (like the knowledge-aware adaptation) contributes to the overall performance, giving readers a clearer picture of why KaSA works so well.", "weaknesses": "The method adds some complexity, especially with the dynamic singular value adaptation, which might make it harder to implement compared to simpler PEFT methods. A potential limitation is the risk that SVD truncation might discard useful knowledge, especially if it’s not fully adapted to a task. It would also help to see this method applied to a wider range of LLM architectures to confirm its generalizability. While KaSA does well on the tested models, applying it to newer transformer architectures or other types of large models could demonstrate its versatility even more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X7JfcnDpOB", "reviewer": "Reviewer_uEtH", "summary": "This paper proposes a novel PEFT method called KaSA. The main idea is to leverage singularvalue decomposition with knowledge-aware singular values to dynamically activate knowledge based on its relevance to the task at hand. Experiments on NLU, NLG and instruction tuning tasks demonstrate the effectiveness of the proposed method.", "questions": "- The PEFT methods are sensitive to hyperparameters like the learning rate. How does the paper ensure a fair comparison with the baseline models? For instance, in Section 4.2, a hyperparameter search is conducted for KaSA for each dataset. Is a similar hyperparameter search performed for all the baseline methods? For Section 4.3 and 4.4, do KaSA and baselines share the same hyperparameters?\n- The setup for the instruction-following experiments is questionable. Given the availability of numerous instruction tuning datasets, why not conduct experiments using these existing datasets? Since other researchers may not have access to the synthetic dataset, reproducing the results could be challenging.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well written and easy to follow. \n- The performance is good.", "weaknesses": "- The proposed method has a more complex training objective compared to baselines like LoRA, PiSSA, and MiLoRA. Consequently, the training time for KaSA may be longer than that of the baselines. However, the paper does not provide detailed information on this.\n- The training objective introduces new hyperparameters, including \\beta and \\gamma, but there is no analysis regarding their impact.\n- How does KaSA perform at different ranks? Can it consistently exceed the baselines across various ranks? This aspect also needs to be validated.\n- Some important baselines are missing, like [1]\n\n[1] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J34XfhOrh6", "reviewer": "Reviewer_Xp8u", "summary": "The paper presents Knowledge-aware Singular-Value Adaptation (KaSA), an innovative PEFT method aimed at overcoming the computational and memory challenges of adapting LLMs to specific tasks. KaSA employs singular value decomposition with knowledge-aware singular values to dynamically activate relevant parametric knowledge for downstream tasks. Through extensive experiments, the authors show that KaSA consistently outperforms full fine-tuning and other popular PEFT methods across a range of tasks, including NLU, NLG, and instruction following.", "questions": "How well does KaSA scale to even larger models?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method is novel, integrating SVD with knowledge-aware singular values. This method allows for the dynamic activation of relevant knowledge, which is a significant advancement over existing PEFT techniques.\n2. Extensive experiments (various tasks with different LLMs, a wide range of baselines) consistently validate the efficacy and adaptability of the proposed method.\n3. The paper is well written with good clarity.", "weaknesses": "My main concern -- most experiments, except experiments in Table 4 on instruction following, only use a small model (less than 1B). Need more experiments and fair comparison with the baselines on larger models. And the improvements on instruction following evaluation results are not significant, and there are even drops in performance on LLaMA3 8B.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)