CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
Authors: Yibo Yang
Xiaojie Li
Zhongzhu Zhou
Shuaiwen Leon Song
Jianlong Wu
Liqiang Nie
Bernard Ghanem
Venue: NeurIPS 2024 poster (2024)
Keywords: parameter-efficient fine-tuning
LLM
knowledge forgetting
Primary area: natural_language_processing
TL;DR: A task-aware parameter-efficient fine-tuning method

Abstract: Current parameter-efficient fine-tuning (PEFT) methods build adapters widely agnostic of the context of downstream task to learn, or the context of important knowledge to maintain. As a result, there is often a performance gap compared to full-parameter fine-tuning, and meanwhile the fine-tuned model suffers from catastrophic forgetting of the pre-trained world knowledge. In this paper, we propose **CorDA**, a Context-oriented Decomposition Adaptation method that builds learnable **task-aware adapters** from weight decomposition oriented by the context of downstream task or the world knowledge to maintain. Concretely, we collect a few data samples, and perform singular value decomposition for each linear layer of a pre-trained LLM multiplied by the covariance matrix of the input activation using these samples. The inverse of the covariance matrix is multiplied with the decomposed components to reconstruct the original weights. By doing so, the context of the representative samples is captured through deciding the factorizing orientation. Our method enables two options, the **knowledge-preserved adaptation** and the **instruction-previewed adaptation**. For the former, we use question-answering samples to obtain the covariance matrices, and use the decomposed components with the smallest $r$ singular values to initialize a learnable adapter, with the others frozen such that the world knowledge is better preserved. For the latter, we use the instruction data from the fine-tuning task, such as math or coding, to orientate the decomposition and train the largest $r$ components that most correspond to the task to learn. We conduct extensive experiments on Math, Code, and Instruction Following tasks. Our knowledge-preserved adaptation not only achieves better performance than LoRA on fine-tuning tasks, but also mitigates the forgetting of world knowledge. Our instruction-previewed adaptation is able to further enhance the fine-tuning performance to be comparable with full fine-tuning, surpassing 
the state-of-the-art PEFT methods such as LoRA, DoRA, and PiSSA.

CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Effiient Fine-tuning
Yibo Yang',
Xiaojie Li
Zhongzhu Zhou',
Shuaiwen Leon Song",
Jianlong Wu²
Liqiang Nie
Bernard Ghanem !:t
Kinn Abdullah University of Science and Technology (KAUST)
Harbin Institute Technology (Shenzhen) *Peng Cheng Laboratory *University of Sydney f: corresponding authors
httpa://githin htt.a////c///oii//////....
Abstract
Current parameter-efficient fine-tuning (PEFT) methods build adapters widely agnostic of the context of downstream task to learn, or the context of important knowledge maintain. As result, there is often a performance gap compared to Cull-parameter fine-tuning, and meanwhile the fine-tuned model suffers from catastrophic forgetting the pre- trainrained world knowledge. In this paper, we pro- pose CorDA, Context-oriented Decomposition Adaptation method that builds learnable task-aware adapters from weight decomposition oriented by the context of downstream task or the world knowledge to maintain. Concretely, we collect a few data samples, and perform singular value decomposition for each linear layer pre-trained LLM multiplied by the covariance matrix of the input activation using these samples. The inverse of the covariance matrix multiplied with the decom- posed components to reconstruct the original weights. By doing the context of the representative samples captured through deciding the factorizing orientation. Our method enables two options, the wwwleege-preservee adaptation and the cntruuction-rreviewwr adaptation. For the former, we use samples to obtain the covariance matrices, and use the decomposed components with the smallest r singular values to initialize a learnable adapter, with the others frozen such that the world knowledge is better preserved. For the latter, the instruction data from the fine-tuning task, such as math or coding, to orientate the decomposition and train the largest components that most correspond to the task to learn. We conduct extensive experiments on Math, Code, and Instruction Following tasks. Our Knowledge-preserved adaptation not only achieves better performance than LoRA on fine-tuning tasks, but also mitigates the forgetting of world knowledge. Our instruction instruction-previewedd adaptation able to further enhance the fine-tuning performance to be comparable with full fine-tuning surpassing the state-of-thed ofttthe-rtt PEFT methods such as LoRA, DoRA, and PiSSA.
Introduction
Large language models (LLMs) have shown remarkable abilities in wide range of challenging tasks, including presting-answering [11, 51], common sense reasoning [6], and instruction following [85]. While being powerful, LLMs demand exorbitant computation and memory cost when fine-tuning the whole model on downstream tasks due the huge model capacity. To enable resource- friendlly adaptation on downstream tasks, parameter-efficienn fine-tuning (PEFT) methods are proposed to largely reduce the number of trainable parameters, by only fine-tuning the newly added adapters [20, 21, 17] or tokens [31, 37, 53], with the original pre-trained weights frozen.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Among these PEFT methods, LoRA i2 increasingly attractive because it is able to keep the model architecture unchanged after fine-tuning so does not induce extra burden inference. LoRA suggests that the weight change in fine-tuning presents a low rank structure, and employs low-rank matrices with a low hidden dimension to approximate the adaptation [21]. Following studies introduce adaptive low rank choice among different layers [78, 59, 75], decouple the learning of magnitude and direction [42], combine LoRA with pruning or quantization [10, 65, 38, 77], and further reduce the number of trainable parameters [26, 55J. However, existing studies build learnable adapters without considering any data context. As a result, these initialized adapters are task-agnostic, and may not be the optimal choice for the downstream task to learn. Moreover, even if PEFT methods only train small number of parameters, the fine-tuned model will still suffer from catastrophic forgetting, losing much of the world knowledge contained in the pre-trained LLM [13, 61, 83]. As shown in our visualization (Figures 4 and 5 in Appendix C) for the covariance matrices with input from different datasets, similar outlier patterns
be observed for inputs belonging to the same kind of task, which implies that the responsive parts of the L.LM pre-trained weights are different when different tasks are triggered. Therefore, the adapter should be built upon the components most associated with the task of concern, egg, a new ability to learn or the QA ability to maintain pre-trained world knowledge.
To this end, we propose task- PEFT method named as CorDA, based on Context-oriented Decomposition Adaptation. It adopts the same low-rank design as LoRA [21], namely introducing two low rank matrices for each linear layer as a learnable adapter, but associates the context of world knowledge or fine-tuning task with the process of building these adapters. First, we randomly collect few data samples and assume that they contain representative context of the corresponding task. For example, the question from a question-answeriig preserving the corresponding knowledge, and the query to write code carries the context of the coding task. We feed these samples into a pre-trained LLM, and obtain the covariance matrix of the input activation of each linear layer, i.e., C= XX" E Rdd xdin, where X is the input of this layer. We then perform singular value decomposition (SVD) for the weight W E Rdd X multiplied by the covariance matrix, i.e., SVD(WC) C) = UEV' where U and V are singular vectors and E is the diagonal matrix with the singular values arranged in descending order. In this way, the representative context expressed by these covariance matrices is able to direct the factorizing orientation. Finally, the inverse of these covariance matrices is multiplied with the decomposed components to hold the same inference result with the original model at initialization, i..., W- = UEVTC-! where W is the weight after decomposition and reconstruction.
dataset well indicates the ability of
Our method supports two optional modes for practitioners, noowledge-preserven adaptation and instraction-previeweed adaptation. LLM fine-tuning on downstream tasks is always accompanied by the damage of world knowledge acquired from massive pre-training data [13]. Our knowledge- preserved adaptation enables to learn new tasks effectively while keeping world knowledge as sound dataset, such as TriviaQA [23] and Natural Questions [27, 29], to obtain the covariance matrices whose pattern corresponds to the LLM ability in retrieving knowledge. After SVD with these covariance matrices, we use the components with the smallest r' singular values, i.e., Up.-r.). I[-r.]. and (VT (-))rr)) I to initialize a learnable low-rank adapter, and the other components that are key to preserving knowledge are frozen. Alternatively, when one only aims to achieve performance as high as possible on the fine- tuning task without concern for world knowledge maintenance, our instruction-prett previewed adaptation suggested. In this mode, we use the instruction and response from the fine-tuning task, e.g. query to write code and its answer, to produce the covariance matrices. Similarly, the pre-trained weights will be decomposed in an orientation such that the context of the fine-tuning task dominates in the principal singular values and vectors. Therefore, we use the largest components, i.e., Up.or]. [ and (V' Cr=)) to initialize learnable low-rank adapter, with the other components frozen. The adapter built upon the context of the fine-tuning task well accommodates the new ability, and thus leads to a better fine-tuning performance.
as possible. In this mode, we use questions from question-ans
Our method brings ficxibility in choosing between stronger fine-tuning performance or more preserved world knowledge, and can be adopted according to the practical demand. Both the two modes as efficient as LoRA [21]. After fine-tuning, the adapter can be merged with the frozen part in the
"U..-.jj Ej-r:j and (V C (-rr.) represent the last columns of U, the last r diagonal elements of Σ, and the last r rows of VT respectively. Up En,. and (VTC-l) represent the first r columns of U. the first diagonal elements of 2, and the first rows of C respectivvey.
2

---

same manner as LoRA. In experiments, CorDA in knowledge-presseee preserved adaptation not only enjoys better fine-tuning performance than LoRA on Math [9, 71]. Code [8. 3], and Instruction Following world knowledge benchmarks including TriviaQA [23], NQ open [29], and WehQS [4]. The instruction- pereviewn- adaptation is able to further strengthen the performance on fine-tuning tasks, surpassing the state-of-the-art PEFT
[80]. but also largely mitigates the deterioration of performance
methods including LoRA [21], DoRA [42], and PiSSA [47].
Related Work
ParaaeeterEtttccient Fine-Tuning, Since large language models (LLMs) have tens and even hundreds of billions of parameters [1, 51, parlmmarreeter fine-tuning will cause unbearable computation and memory cost [79, 67]. Parameter-efficien fine-tuning (PEFT) developed to reduce resource consumption by only fine-tuning a small number of learnable parameters [12, 64]. Adapter- methods introduce additional modules into LLMs and only fine-tune them during fine-tuning [20, 17, 30, 45, 49, 24]. Another line of research appends extra soft prompts into the input or hidden layers and only train these learnable vectors [31, 37, 53, 84]. However, most of these methods change the model architecture or increase the inference burden. Based on the insight that the weight change after fine-tuning possesses a low rank structure [32, 2]. low-rank adaptation (LoRA) [21] proposes to use two low-rank small matrices as the learnable adapter, without modifying model architecture or bringing inference cost after fine-tuning. LoRA has inspired a range of variants that employ adaptive low rank in different layers [78, 59, 75]. explore the adapter design [42, 7, 50, 79], combine
LoRA with pruning [77]. quantization [10, 65, 38], and mixture-of expert [41, 13], and introduce alternative way to initialize the adapter [47]. Nevertheless, existing PEFT methods rarely consider data context when building the learnable adapter. Data context has been proved to be instrumental in guiding quantization and compression [40, 73, 28]. In our study, utilize data context for PEFT, building adapters based on context-oriented decomposition to better maintain world knowledge or accommodate new ability.
Knowledge Forgetting. Deep learning models are prone to drastically forgetting the acquired knowledge when adapting new task, known as catastrophic forgetting [14, 54, 25, 52, 44, 70, 36]. series of methods has been proposed to mitigate this issue using knowledge distillation [39, 19, 33], rehearsal [56, 69]. and dynamic architecture [66]. In the era of large models [62], however, world knowledge is acquired by pre-training on massive data, which could be intractable to re-use in fine-tuning [34, 35J. The huge model capacity also hinders the feasibility of knowledge distillation and dynamic architecture, especially in the case of continuous fine-tuning [16, 74, 57, 15, 22]. Some studies introduce extra LLaMA layers [61] mixture of experts [13] with the pre-trained layers frozen to strike a balance between keeping world knowledge and learning new tasks. Alternative approaches adopt merging schemes to enable diverse abilities [76, 72, 83]. Different from these studies, our method enables to achieve world knowledge maintaining in the process of parameter- efficient fine-tuning, without changing model architecture or relying on post-merging step.
3 Method
We review the LoRA method in Sec. 3.1. We develop our context-oriented decomposition in Sec. 3.2, which provides the basis of our knowledge- knowledge-preserved adaptation and instruction previewwededwddddeee adaptation introduced in Sec. 3.3 and See. 3.4, respectively.
3.1 Preliminaries Low-Rank Adaptation
LoRA suggests that the weight change in LLM fine-tuning presents a low rank structure, and thus proposes to use the product of two low rank matrices to learn the weight change with the pre-trate traitre weights frozen during fine-tuning [21]. Given the pre-trained weight W E Rddo Xdin from an LLM, the weight after fine-tuning can be formulated as:
W = W + AW = W + BA,
(I)
where W* is the weight after fine-tuning, AW is the weight change, and BA is the low rank decomposition of AW into two smaller matrices B E Rd and A E X with an intrinsic rank of r < min (dout, ). In this way, the number of learnable parameters can be largely reduced
3

---

Data example for @ knowleegg-ppreereee adapialion
Bxl.
Question:
LLM
When wexe the subways built in new yozk city?
dmp
X
X
C
inverse
dlm
Data example Tr. instruction iinsinnctionprreeewweee previewed adaptation
Quezy: We have that $2a 1 and value $a5?
dan
1.6 What
SVD
Responso: Fron the first equation, we have $2a-05, so Sa-0$. Substituting this into the second
W
have $b-0-1S, so (bx The answer is: 1
weight
U
Σ
dus
dust
17
8 Unni Eipt
W' W BA
W' W BA
Frazen
Frozen
Leamable Adapter
Leamaale Adapter
dee
nhowledgg-preserree adaptation
Instruction-previewed adaptation
An overall illustration our proposed method. We perform singular value decomposition oriented by the covariance matrix to aggregate task context into the principal components (up), which are frozen for maintaining world knowledge (down left) or utilized to initialize the learnable adapter for better with the largest r singular values, while the light one is composed of the smallest r components.
Figure I:
performance (down right). The dark-colored adapter refers to the components
by freezing the pre-trained weight W and only fine-tuning the matrices B and A. LoRA adopts the Kaiming initialization [18] to randomly initialize A, and initialized as an all-O matrix such that AW = 0 at the start of training to circumvent a deviation from the pre-trained model. After fine-tuning, the learnable adapter BA can be merged into pre-trained weight W without changing the original model architecture and introducing extra inference burden.
Despite the success of LoRA-based methods, when building the learnable adapter, existing studies widely ignore the data context from the target ability that users are particularly concerned with.
3.2 Crrtext-Oriented Decomposition
Pre-trained large language models endowed with multi-faced abilities, such answering the question regarding world knowledge, common sense reasoning, and instruction following. When different kinds of input messages are fed into an LLM, e.g., a question in some domain and query to solve a math problem, even though they are processed by the same pre-trained weights, different abilities triggered. The covariance matrix of each layer's activation will exhibit different outlier patterns as they are responsive to the task triggered to highlight different aspects of the pre-trained weight. Therefore, the covariance matrix is able to capture task context. Inspired by this insight, we leverage the covariance matrix inside LLM to build adapters catering to a certain ability.
The process of our context-oriented decomposition is shown in Figure First, we randomly collect some samples from the training data of some task with interest, e.g., question answering or Math, and feed these samples into the LLM used to fine-tune. Denote X E R... x BL, as the input activation of a linear layer where d,,, is the input dimension, B is the number of samples we collect, and represents the sequence length. We have the covariance matrix C = XX xdi., We then perform singular value decomposition for the weight multiplied by the covariance matrix as:
R
SVD(WC) = UEVT
Σ

---

where W E Rdouttden is the weight of this linear layer, U E Rdoutxdout and V E Rdi, xden are orthogonal matrices containing singular vectors u, E Root and V, E R E R Xdin is diagonal matrix with singular values on its diagonal arranged in descending order, and R is the rank (the number of non-zero singular values) of WC, i.e, S min (dout, d
To not change the inference result at the initialization of fine-tuning, we reconstruct W by:
W = SVD(WC)C=+== -XI Σ o.u.
where C-1 denotes the inverse of C, and v is the i-th row vector of VTC-1, In case the covariance matrix C is not invertible, we adopt a strategy to dynamically add positive values on the diagonal elements of C' to ensure invertible. Concretely, we multiply a positive coefficient with the average value of the diagonal elements of C. and add it on the diagonal. Then we calculate the &y distance between CC-1 and an identity matrix. If it is higher than a threshold, we double the coefficient and perform this step again, until the distance reaches below the threshold.
After our context-oriented decomposition, the first several components of u, and V; with the largest singular values o; depict the dominant characteristics of the task associated with C. We can decide either to maintain these key components to not sacrifice the corresponding ability, to adapt them for better performance on the task, which leads to our two implementation modes in the following two subsections, respectively.
3.3 Mode 1: Knowledge-Preserved Adaptation
We introduce knowledge-preserved adaptation that enables to learn new tasks while maintaining world knowledge. In this mode, use the question data from gunseenn-answering training data, such as TriviaQA [23] and Natural Questions [27, 29], to obtain the covariance matrices whose pattern corresponds the knowledge retrieving ability of the LLM. When fine-tuning on new task, as shown in Figure I, we use the last r components with the smallest r singular values in Eq. (3) to build learnable adapters as:
W = W- - BA, B B-U A A=
CV"
(4)
where B E Rdout and A Bexd... are the initialized matrices in the learnable adapter, BA a diagonal matrix
ER Ei=R-r+1 o. corresponds to the last r components Eq. (3), V rrj with the squared root of the smallest singular values on the diagonal, and W corresponding to the first R - components in Eq. (3) is frozen during fine-tuning. We have W' as the difference between W and BA instead of summing the first r components to avoid the numerical error between W and W introduced by the decomposition and inversion operations. After fine-tuning, the learned matrices B* and A" can be merged into W as W = W + B² Ar. This mode is featured by preserving world knowledge as the knowledge retrieving ability captured by the principal R - components is frozen. It is also more effective than zero-initialized adapter in learning new abilities as verified by our experiments.
3.4 Mode 2: Instruction-Previewwe Adaptation
In the circumstance that pursuing a higher performance on the fine-tuning task is the priority, our instruction previewwdd adaptation will be favorable. this mode, we collect instruction and response from the training data used for fine-tuning, e.g. the query to solve math problem and its answer shown in Figure as an example. The prompts are fed into the LLM to produce the covariance matrices whose pattern is associated with the task to learn. We use the first components with the largest r singular values in Eq. (3) to build learnable adapters as:
W' = W BA, B = U [
A= V (VT -
(5)
where B and are the initialized matrices in the learnable adapter, BA = Σi= o. corresponds to the first r components in Eq. (3), V is the squared root of the largest r singular values in a diagonal matrix. Similar to knowledge- preserved adaptation, W' containing the remaining R
prowledge-rreservedd
components is frozen during fine-tuning. This mode enables the initialized adapters to pre-capture
5

---

80
Plain SVD
70
ASVD (with samples)
ASVD (with PTB 256 samples)
CO-SVD (with Wiki 256 samples)
CO.SVD (with samples)
60
100
50
80
40
30
20
10
20
0
16
32
64
128
236
512
1024
0 16
32
64
120
256
512
1024
The number of smallest rank discard
number smallest discard
(a) Perplexity on Wikitext-2
(b) Perplexity on PTB
Figure 2: Perplexity (lower is better) on (a) Wikitext-2 and (b) Penn TreeBank (PTB) after decompos- ing the LLaMA-2-7B weights and reconstruction discarding the smallest singular values and their singular vectors. We compare our context-oriented decomposition (CO-SVD) with plain SVD and ASVD. The perplexity of plain SVD on PTB at 1024 is 763.4, which out of the shown range.
the main characteristics of the fine-tuning task, leading to stronger performance after training. A recently proposed method [47] also performs SVD and uses the first components to initialize an adapter for fine-tuning. However, their decomposition is agnostic with respect to any data context. Our adapter capturing the task context in advance well accommodate the new ability and lead to better fine-tuning performances in our experiments.
Experiments
In experiments, we fine-tune the pre-trained large language model LLaMA-2-76 [58] Math, Code, and Instruction Following tasks, and also apply our method to the General Language Understanding
Evaluation (GLUE) benchmark with RoBERTatse [43]. The world knowledge is evaluated by the
exact match scores (%) on TriviaQA [23]. NQ open [29], and WebQS [4]. Following the settings
in [47], the Math ability is trained MetaMathQA [71] and tested GSM8k [9] and Math [71]
validation sets. Code is trained on CodeFeedback [82] and tested on HumanEval [8] and MBPP [3]. Instruction following is trained WizardL.M-
[63] and tested on MTBench [81]. The
complete implementation details are described Appendix A.
4.1 Analysis of the Ability Capture Context
We conduct an experiment demonstrate the ability of our proposed decomposition method Sec. 3.2 to capture context in its principal components. We use different methods including the Plain SVD, ASVD [73], and our Context-Oriented SVD (CO-SVD), to perform full decomposition of the LLaMA-2-7B pre-trained weights in all layers, and then discard the smallest singular values and their corresponding left and right singular vectors to reconstruct weights for testing.
As shown in Figure 2, as the number of discarded ranks increases, the performances with Plain SVD on both Wikitext-2 [48] and PTB [46] are getting worse steeply. ASVD considers data context using activation absolute mean values, and helps to relieve the deterioration compared to Plain SVD. However, when discarding more than 256 ranks, the Perplexity also diverges sharply. contrast, our method is able to maintain stable performance very close to the original pre- rre-trained weights even when the smallest 1024 components are discarded. The result indicates that our method is proficient with aggregating context from limited samples (256 Wiki or PTB samples in this example) into the principal components. also evidences the potential of our method to maintain important knowledge when freezing the principal components and to learn new abilities when adapting these components. The detailed numbers of the experiment in Figure are listed in Table (Appendix B), where we also test the effect of sample number and dataset choice when collecting the covariance matrices.

---

Table 1:
The experimental results of CorDA in the knowledge- wwwwledd--rreerveddd adaptation mode and comparison with full fine-tuning, LoRA, and PiSSA. LLaMA-2-7B is used to fine-tune on (a) Math, (b) Code, and (c) Instruction Following tasks. The rank r LoRA, PISSA, and CorDA is 128. CorDA is initialized with the NQ open samples to collect the covariance matrices. All methods are implemented by us under the same training and evaluation settings. The row of "ILLMMA-2-7B" shows the world knowledge performance of the original pre-trained model.
(a) Math
Method
#Params
Trivia QA
NQ open
WebQS
GSMSk
Math
Avg
LLaMA-2-7B
52.51
14.99
5.86
Full fine-tuning
6738M
43.64
3.13
6.35
48.90
7.48
21.90
LoRA [21]
320M
44.17
1.91
6.64
42.68
5.92
20.26
PiSSA [47]
320M
39.71
1.02
6.30
51.48
7.60
21.22
CorDA (ours)
320M
44.30
9.36
7.14
44.58
6.92
22.46
(b) Code
Method
#Params
Trivia QA
NQ open
WebQS
HumanEval
MBPP
Avg
LLaMA-2-7B
52.51
14.99
5.86
Full fine-tuning
6738M
29.29
8.53
3.44
25.42
25.64
18.46
LoRA [21]
320M
51.42
9.30
8.46
16.8
21.51
21.50
PiSSA [47]
320M
47.07
9.16
8.14
19.48
23.84
21.54
CorDA (ours)
320M
50.02
11.72
8.56
18.36
20.91
21.91
(c) Instruction Following
Method
#Params
Trivia QA
NQ open
WebQS
MTBench
Avg
LLMMA-2-7B
52.51
14.99
5.86
Full fine-tuning
6738M
26.6
8.45
6.84
4.85
11.69
LoRA [21]
320M
47.46
10.28
7.73
4.60
17.52
PiSSA [47]
320M
36.76
9.67
5.86
4.92
14.30
CorDA (ours)
320M
50.34
14.43
8.17
5.05
19.50
4.2 Knowledge-Preserver Adaptation Results
We fine-tune LLaMA-2-7B with full fine-tuning, LoRA, PiSSA, and our proposed CorDA on Math, Code, and Instruction Following tasks. In the knowledge- knowledge-preserved preserveeeeed adaptation mode, we randomly sample 256 questions from the NQ open training set and collect covariance matrices initialize the adapters by Eq. (4). We report the fine-tuning performance and also the world knowledge performance of the fine-tuned model to manifest the overall ability of both new task learning and world knowledge Math, all the three compared methods, full
maintaining. As shown in Table la, after fine-tuning fine-tuning, LoRA, A, and PiSSA, suffer from drastic performance drop on TriviaQA and NQ open. Especially on NQ open, the ability is almost lost. PiSSA achieves the best accuracies on both GSM8K and Math, but meanwhile has the lowest results on the world knowledge benchmarks among the four method not only enjoys better fine-tuning performances than LoRA,
methods. As comparison, but also achieves the best results on the three world knowledge benchmarks.
A similar pattern can be also observed in the fine-tuning of Code. As shown in Table 1b, full fine-tuning has the best ability on HumanEval and MBPP, but is the lowest on world knowledge performance. It is understandable that CorDA in knowledge-preservedd preserved adaptation is not as advanta- geous as PiSSA for fine-tuning performance, because PiSSA uses the largest singular values and their singular vectors the adapter that dominates the weight update, while we keep them frozen and adapt the smallest components. Nevertheless, our method has the best average score in all the three fine-tuning tasks. A surprising result is achieved by our method in instruction following in Table
7

---

JGSMBk:
LoRA-1128 (6SMBC-42) 42.7, Math: 5.9)
LoRde132
(GSMBk: 36.1. Math:
ASSA+128 [GSMMicc 51.5, Math: 7.6]
PI58A-132
(GSMMk:: 43, Math: 5.7)
0.350
CorDA-1128 [GSMBk: 53.9, Mathi:
0.350
Cor DA-132
(GSMBk: 45.4,
Mati:
Uoin ksse
0.325
tain 600
0.325
0.300
0.300
0.275
0.275
0.225
0.225
0.200
0.200
100
300
500
700
Iteration
Iteration
(a) 128
(b)
32
Figure 3: The training loss curves on MetaMath of full fine-tuning, LoRA, PISSA, and CorDA with (a) rank 128 and (b) rank 32. The corresponding accuracies GSM8K and Math reported on the legends. Smoothing is performed for the loss curves.
Table 2:
The experimental results of CorDA in the instruction-previewed adaptation mode on Math, Code, and Instruction Following tasks using LLaMA-2-7B. CorDA is initialized with samples from each of the fine-tuning datasets (MetaMathQA, CodeFeedback, and three tasks, respectively. The rank LoRA, DoRA, PiSSA, and CorDA is 128. All methods are
for the
implemented by under the same training and evaluation settings.
Method
#Params
GSM8k
Math
HumanEval
MBPP
MTBench
Avg
Full fine-tuning
6738M
48.9
7.48
25.42
25.64
4.85
22.46
LoRA [21]
320M
42.68
5.92
16.80
21.51
4.60
18.30
DoRA [42]
321M
41.77
6.20
16.86
21.60
4.48
18.18
PiSSA [47]
320M
51.48
7.60
19.48
23.84
4.92
21.46
CorDA (ours)
320M
53.90
8.52
21.03
24.15
5.15
22.55
Ic, where CorDA surpasses all the three compared methods in both world knowledge performance and the new ability evaluated by MTBench. The world knowledge on NQ open is almost intact (14.43%) compared the original performance (14.99%) before fine-tuning. These results reveal that our knowledge knowledge-preservee preserved adaptation is an effective way to mitigate world knowledge forgetting and improve the overall ability.
4.3 Prrruction--reviwweed Adaptation Results
When the goal is to learn the target task as much as possible without concerning the loss of world knowledge, CorDA in the instruction prreiiewed adaptation mode satisfies this demand as is able to further strengthen the fine-tuning performance, In this mode, we randomly sample instruction and response from the training data to collect the covariance matrices, and adopt the largest components to initialize the adapters by Eq. (5). We compare our method with full fine-tuning, LoRA, DoRA, and PiSSA the same three tasks, Math, Code, and Instruction Following. Compared with the tnowledge-preservee mode largely improves the fine-tuning performance on the five benchmarks, as shown in Table 2. Concretely, our method achieves the best performance score. Compared with PiSSA that adopts similar adapter design but with no data context, our method has better results all the five benchmarks. The training loss curves Math are shown in Figure 3, where CorDA converges at a lower loss than LoRA and PiSSA in both r = 128 and 32. Only CorDA exhibits obvious lower loss than full fine-tuning when = 128. These results corroborate the benefits of data context to the initialized adapter, i.e., the pre-captured task characteristic able accommodate the new ability and lead better performance.
adaptation in Table 1, CorDA in the instruction- previewwed adaptation
GSM8k, Math, MTBench, and the average
We also apply our method the General I. Language Understanding Evaluation (GLUU) benchmark [60] by fine-tuning the RoBERTabass model [43]. We adopt LoRA, DoRA, and our method with

---

Table 3: The experimental results of CorDA in the instruction preeiiown--eewwweddee adaptation mode on the GLUE benchmark using RoBERTamas CorDA is initialized with samples from each of the fine- tuning datasets. The rank r of LoRA, DoRA, and CorDA is 128. All methods are implemented by us under the same training and evaluation settings. Matthew's correlation and Pearson's 's correlation are the metrics of CoLAA and STS-B, respectively. The metric of the other tasks accuracy.
Method
#Params
SST-2
MRPC
CoLA
QNLI
RTE
STS-B
Avg
Full fine-tuning
125M
93.81
88.48
59.56
92.07
74.01
90.49
83.07
LoRA [21]
21M
94.15
82.84
54.24
92.48
64.26
88.58
79.43
DoRA [42]
21M
93.58
83.58
51.93
92.59
64.98
88.71
79.23
CorDA (ours)
21M
93.12
89.71
59.60
91.49
76.17
90.17
83.38
Table 4: Ablation experiments of the data choice used to collect covariance matrices and the adapter building manner in the knowledge -eeeggevedeeeeedddee adaptation mode. t: corresponds to the result of PiSSA that performs plain SVD and uses the largest components to initialize the adapter.
Method
Context
Adapter
Trivia QA
NQ open
WebQS
GSM8k
Math
Avg
Plain SVD
none
largest
39.71
1. 1.02
6.30
51.48
7.60 16
21.22
Plain SVD
none
smallest
39.94
4.21 40.41
6.25
43.29 #0.37
5.96
19.93
CO-SVD
Wikitext-2
smallest r
42.93
7.20
6.40
42.99
5.80
21.06
CO-SVD
Trivia QA
smallest r
44.59+0.34
8.86
7.53
44.81 +0.22
6.84
22.53
CO-SVD
NQ open
smallest
44.30
9.36
7.14
44.58
6.92
22.46
Table 5: The instruction following performance CorDA using WizardL -- and Alpaca data to collect covariance matrices in the instruction-preeveeedr adaptation mode.
Method
Context
MTBench
CorDA (ours)
Wizardl BePard..M-RE Instrruttttt
5.15
CorDA (ours)
Alpaca
5.06
a rank of 128 for all linear layers in the model except the classification head. For our method, we sample train data from each of the tasks initialize adapters in the instruction-previever mode and fine-tune the corresponding task. As shown in Table 3, our method achieves the best performance on the MRPC, CoLA, and RTE tasks, and the highest average score.
4.4 Discussions
Ablations. We ablate the data choice used to produce covariance matrices and the adapter building manner in Table 4. The first corresponds the implementation of PiSSA that uses the largest singular values and their singular vectors as the initialized adapter by plain SVD. If we use the smallest components by plain SVD build adapters, there is apparent improvement in world knowledge benchmarks. This implies that the plain SVD cannot precisely capture world knowledge related ability into the principal components and thus freezing them does not help mitigate knowledge forgetting. When our context-oriented decomposition (CO-SVD) is adopted with Wikitext-2, which not closely correlated with question answering. the performance improved. When the context collected by the covariance matrix is from question answering data, i.e.. TriviaQA NQ open, the world knowledge performance is further improved by significant margin, and the average score is also enhanced as result. Therefore, data context is important to orientate the decomposition process such that the characteristics of the ability concerned can be better aggregated into the principal components for maintaining or adapting. As shown in Table 5, similar to TriviaQA adaptation, collecting context from different data sources
world knowledge is much
and NQ open the smowledge-presentt belonging to the same category, namely WizardLMM MM---M------seeeeto and Alpaca, also results in close performance in the instruction-pres prrviieee adaptation. It is noteworthy that CorDA with A Alpaca on MTBench (5.06) is still the highest among the compared basclines in Table 2.

---

Limitations. The two adaptation modes developed in this paper highlight different aspects in usage. However, the mnowledge-preservee mode, while being adept at maintaining world knowledge, naturally not advantageous mode. How to develop an initialization strategy [68] for adapters combining the merits of the two modes to maximize both objectives deserves future exploration.
fine-tuning performance compared with the instruction-previcnn
5 Conclusion
In this paper, we propose a new parameter--fficient fine-tuning method, named context-oriented decomposition adaptation (CorDA). It performs singular value decomposition for pre-trained weights oriented by the covariance matrix that captures the context of the task concerned, and aggregates the context into the principal components for maintaining adapting. Accordingly, able to support two implementation modes, the knoweedge-preserved adaptation to mitigate world knowledge forgetting and the instruction-previewesr adaptation for better fine-tuning performance. In experiments, our mnowledge-preserved adaptation not only achieves better fine-tuning performance than LoRA, but also maintains the world knowledge well, leading to the best average scores on three fine-tuning tasks. Our instruction pren-eeeeeedd adaptation is able to further enhance the fine-tuning performance, surpassing the state-of-the-art parameter-efficient fine-tuning methods DoRA and PiSSA.
method is
Acknowledgments and Disclosure of Funding
The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940.
This work was also supported in part by the National Natural Science Foundation of China under Grant 62376069, in part by Young Elite Scientists Sponsorship Program by CAST under Grant 2023QNRC001, in part by Guangdong Basic and Applied Basic Research Foundation under Grant 1524A 2027, and in part by the Shenzhen Science and Technology Program (Grant No. 2DDSYYSS02306220992000000
10

---

References
[1] J. Achiam, S. Adler S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint
27X1::3303.08774 2023.
[2] A. Aghajanyan, I.. 7ettleemoe,r and S. Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint arXiv:2012.13255 2020.
[3] Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry,
Q. Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07773
2021.
[4] J. Berant, A. Chou, R. Frostig, and Liang. Semantic parsing freebase from question-answer pairs. In EMNLP, pages 1533-1544, 2013.
[5] Bie, Y. Yang, Z. Zhou, A. Ghanem, M. Zhang, Z. Yao, X. Wu, C. Holmes, P. Golnari, D. A. Clifton, et al. Renaissance: A survey into text-to-image generation in the era of large model. arXiv preprint arXiv:2309.00S: 000000 2023.
[6] A. Bosselut, H. Rashkin, M. Sap. C. Malaviya, A. Celikyilmaz. and Y. Choi. Comet: Commonsense transformers for automatic knowledge graph construction. arXiv preprint
1.Xiv:/906.0555 05 2019.
[7] A. Chavan, Liu, D. Gupta, E. Xing, and Z. Shen. One-for-all: Generalized lora for parameter- efficient fine-tuning, arXiv preprint arXiv:2306.0788
2023.
[8] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. O. Pinto. J. Kaplan, H. Edwards, Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained code. arXiv
preprint Xiiv:2007003374 2021.
[9] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint
arXiv:2110. 14168, 2021.
[10] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer Qlora: Efficient finetuning of quantized 1lm.. NeurIPS, 36, 2023.
[11] Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint arXiv: 1810.04855, 2018
[12] N. Ding, Y. Qin, G. Yang, F. Wei, Z. Yang, Y. Su, S. Hu, Y. Chen, C.-M. Chan, W. Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235 2023.
[13] S. Dou, E. Zhou, Y. Liu, S. Gao. J. Zhao, W. Shen, Y. Zhou, Z. Xi, X. Wang, X. Fan, et al. Loramoe: Alleviate world knowledge forgetting in largelanguage models via moe-style plugin. arXiv preprint 1rXiv:23/2.099777 2023.
[14] J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint 1xXxv:3312.6211 2013.
[15] K. Gupta, B. Thérien, A. Ibrahim, M. L. Richter, Q. G. Anthony, E. Belilovsky, I. Rish, and
T. Lesort. Continual pre-training of large language models: How to warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023.
[16] He, H. Guo, M. Tang, and J. Wang. Continual instruction tuning for large multimodal models. arXiv preprint arXiv:231 211.2266,, 2023.
[17] J. He, C. Zhou, X. Ma. T. Barg-Kirkpatrick, and G. Neubig. Towards a unified view of parameter-efficient transfer learning. In ICLR, 2022.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015.
[19] Hou, X. Pan, C. Loy, Z. Wang, and D. Lin. Learning a unified classifier incrementally via rebalancing. In CVPR, pages 831-839 2019.
[20] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and Gelly. Parameter-efffiieet transfer learning for nlp. In ICML, pages 2790-2799. PMLR, 2019.
[21] E. J. Hu, yelong shen, P. Wallis, Z. Allen Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[22] A. Ibrahim, B. Thérien, K. Gupta, M. L. Richter, Q. Anthony, T. Lesort, E. Belilovsky, and
I. Rish. Simple and scalable strategies to continually pre-train large language models. arXiv preprint arXiv:2403.088 08766, 2024.

---

[23] M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
[24] R. Karimi Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex
adapter layers. NeurlPS, 14:1022-1000 1035, 2021.
[25] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardinss A. A. Rusu, K. Milan, Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 1 [1<(13):52113522 2017.
J. Quan, T. Ramalho, A.
[26] D. J. Kopiczko, T. Blankevoort, and Y. M. Asano. Vera: Vector-based random matrix adaptation.
In ICLR, 2024.
[27] T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019. [28] C. Lee,, J. Jin, T. Kim, H. Kim, and E. Park. Owq: Outlirr-aaare weight quantization for efficient fine-tuning and inference of large language models. In AAAI, volume 38, pages 13355-13364, 2024.
[29] K. Lee, M.-W. Chang, and K. Toutanova. Latent retrieval for weakly supervised open do- main question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
[30] T. Lei, J. Bai, S. Brahma, J. Ainslie, K. Lee, Y. Zhou, N. Du, V. Y. Zhao, Y. Wu, B. Li, et al. Conditional adapters: Parameter-efficient transfer learning with fast inference. In NeurIPS, 2023.
[31] B. Lester, R. AI-Rfou, and N. Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, pages 3045-3059, 2021.
[32] C. Li, H. Farkhoor, R. Liu, and J. Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR, 2018.
[33] X. Li, S. He, Wu, Y. Yu, L. Nie, and M. Zhang. Mask again: Masked knowledge distillation for masked video modeling. In Proceedings of the 3 31st ACM International Conference on Multimedia, pages 2221-2232, 2023.
[34] X. Li, J. Wu, S. He, S. Kang, Y. Yu, L. Nie, and M. Zhang. Fine- fine-grained key-value mem- ory enhanced predictor for video representation learning. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2264-2274, 2023.
[35] X. Li, Y. Yang. X. Li, J. Wu, Y. Yu, B. Ghanem, and M. Zhang. Genview: Enhancing view quality with pretrained generative model for self-supervised learning. In ECCV, 2024.
[36] X. Li, Y. Yang. J. Wu, B. Ghanem, L. Nie, and M. Zhang. Mamba-fseil: Dynamic adapta- tion with selective state space model for few-shot class- mass-incremental learning. arXiv preprint arXiv:2407.0m 0.600006666,... 2024.
[37] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582 4597, 2021.
[38] Y. Li, Y. Yu, C. Liang, N. Karampatziakiss P. He, W. Chen, and T. Zhao. Loftq: LoRA-fine- luning-aware aware quantization for large language models. In ICLR, 2024.
tuning-
[39] Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, (0(12):2335-294
2017. [40] J. Lin, J. Tang, H. Tang. S. Yang. W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and
S. Han. Awq: Activation -etivation-awwre aware weight quantization for Ilm compression and acceleration. In MLSys, 2024.
[41] Q Liu, X. Wu, X. Zhao, Y. Zhu, D. Xu, F. Tian, and Y. Zheng. Moelora: An moe-based parameter efficient fine-tuning method for multi-task medical applications. arXiv preprint 20X 2023.
[42] S.-Y. Liu, C.-Y. Wang. H. Yin. P. Molchanov, Y.-C. F. Wang, K.-T. Cheng, and M.-H. Chen. Dora: Weight-decomposed low-rank adaptation. In JCML, 2024.
[43] Y. Liu, M. Out, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer,
and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 2019.
[44] D. Lopez-Paz and M. Ranzato. Gradient episodic memory for continual learning. In NewrlPS, volume 30, 2017.
12

---

[45] R. K. Mahabadi, S. Ruder, M. Dehghani, and J. Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Pupers), pages 565-576, 2021.
[46] M. Marcus, B. Santorini, and M. A. Marcinkiewicz. Building a large annotated corpus of english: The penn treebank. Computational linguistics, 19(2):313-330 1993.
[47] F. Meng. Z. Wang, and M. Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint XXiv:2404.0294/ 2024.
[48] S. Merity, C. Xiong, J. Bradbury, and R. Socher. Pointer sentinel mixture models. arXiv preprint
arXiv:1609.077 1609.07843, 2016.
[49] J. Pfeiffer, A. Kamath, A. Rücklé, K. Cho, and I. Gurevych. Adapterfusion: Non-destructive
task composition for transfer learning. In Proceedings of the 16th Conference of the European Chapter of the Association for Computationall Linguistics: Main Volume, pages 487-503, 2021. [50] Z. Qiu, W. Liu, H. Feng, Y. Xue, Y. Feng, Z. Liu, D. Zhang, A. Weller, and B. Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In NeurlPS, volume 36, pages 79320-79362, 2023.
[51] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding
by generative pre-training 2018.
[52] D. Rao, F. Visin, A. Rusu, R. Pascanu, Y. W. Teh, and R. Hadsell. Continual unsupervised representation learning. In NeurlPS, volume 32, 2019.
[53] A. Razdaibiedina, Y. Mao, M. Khabsa, M. Lewis, R. Hou, J. Ba, and A. Almahairi. Residual prompt tuning: improving prompt tuning with residual reparameterization. In ACI., pages 6740-6757, 2023.
[54] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In CVPR, pages 2001-2010, 2017.
[55] A. Renduchintala, T. Konuk, and O. Kuchaiev. Tied-lora: Enhacing parameter efficiency of lora
with weight tying. arXiv preprint arXiv:2311.00 000578,,,,8 2023.
[56] M. Riemer, I. Cases, R. Ajemian, M. Liu, I. Rish, Y. Tu, and G. Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In ICLR, 2019.
[57] T. Scialom, T. Chakrabarty, and S. Muresan. Fine-tuned language models are continual learners.
In EMNLP, 2022.
[58] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv
preprint arXiv:2307.00220 07.0888..8. 2023.
[59] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and A. Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint
20.0:2210.07533 2022.
[60] A. Wang. A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task bench- mark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
[61] C. Wu, Y. Gan, Y. Ge, Z. Lu, J. Wang, Y. Feng. P. Luo, and Y. Shan. Llama pro: Progressive llama with block expansion. arXiv preprint urXiv:2401.022 1,2411,,, 2024.
[62] T. Wu, L. Luo, Y.-F. Li, S. Pan, T.-T. Vu, and G. Haffari. Continual learning for large language models: A survey. arXiv preprint arXiv:2402.0 01.01166, 2024.
[63] C. Xu, Q. Sun, K. Zheng, X. Geng, P. Zhao, J. Feng, C. Tao, Q. Lin, and D. Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In ICLR, 2024. [64] I.. Xu, H. Xie, S. J. Qin, X. Tao, and F. L. Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXi1:2312.12148, 2023.
[65] Y. Xu, L. Xie, X. Gu, X. Chen, H. Chang, H. Zhang, Z. Chen, X. ZHANG, and Q. Tian. QA-IoRA Quantization-aware low-rank adaptation of large language models. In ICLR, 2024. [66] S. Yan, J. Xie, and X. He. Der: Dynamically expandable representation for class incremental learning. In CVPR, pages 3014-30010 44222,00332220 2021.
[67] Y. Yang, X. Li, M. Alfarra, H. A. A. K. Hammoud, A. Bibi, P. Torr, and B. Ghanem. Towards interpretable deep local learning with successive gradient reconciliation. In ICML, 2024. [68] Y. Yang, H. Wang, H. Yuan, and Z. Lin. Towards theoretically inspired neural initialization optimization. In NewrIPS, volume 35, pages 18983-18995, 2022.
13

---

[69] Y. Yang, H. Yuan, X. Li, Z. Lin, P. Torr, and D. Tao. Neural collapse inspired feature-classiteee alignment for few-shot class- Dass-incremental learning. In ICLR. 2023.
[70] Y. Yang, H. Yuan, X. Li, Wu, L. Zhang, Z. Lin, P. Torr, D. Tao, and B. Ghanem. Neural collapse terminus: A unified solution for class incremental learning and its variants. arXiv
preprint arXiv:2308. 01746, 2023.
[71] L. Yu, W. Jiang. H. Shi, J. YU, Z. Liu, Y. Zhang, J. Kwok, Z. Li, A. Weller, and W. Liu. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR, 2024.
[72] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In ICML, 2024.
[73] Z. Yuan, Y. Shang, Y. Song. Q. Wu, Y. Yan, and G. Sun. Asvd: Activation-aware singular value decomposition for compressing large language models. arXiv preprint arXiv:2312.05822 2023. [74] Y. Zhai, S. Tong, X. Li, M. Cai, Q Qu, Y. J. Lee, and Y. Ma. Investigating the catastrophic forgetting in multimodal large language model fine-tuning, In Conference on Parsimony and Learning (Proceedings Track), 2024.
[75] F. Zhang, L. Li, J. Chen, Z. Jiang, B. Wang, and Y. Qian. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning, arXiv preprint arXiv:2308. /2043, 2023.
[76] J. Zhang. J. Liu. J. He, et al. Composing eerameter-efficient modules with arithmetic operation.
NeurlPS, 36: 11--2228861110 2023.
[77] M. Zhang, C. Shen, Z. Yang. L. Ou, X. Yu, B. Zhuang, et al. Pruning meets low-rank parameter- efficient fine-tuning, arXiv preprint 2rX.
2023.
[78] Q Zhang, M. Chen, A. Bukharin, P. He, Y. Cheng. W. Chen, and T. Zhao. Adaptive budget allocation for parameter-efficient fine-tuning, In ICLR, 2023.
[79] J. Zhao, Z. Zhang, B. Chen, Z. Wang, A. Anandkumar, and Y. Tian. Galore: Memory-efficient
Ilm training by gradient low-rank projection. In ICML, 2024.
[80] L. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang, Z. Wu, Zhuang, Z. Lin, Z. Li, D. Li, E. Xing,
et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In NeurIPS, volume 36, 2023. [81] I.. Zheng, W.-L. Chiang, Y. Sheng. S. Zhuang, 7. Wu, Y. Zhuang. 7. Lin, 7. Li, D. Li, E. Xing, H. Zhang, J. Gonzalez, and Stoica. Judging L-M--a--ajuddee with MT-bench and chatbot arena. In shernh-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[82] Zheng, Zhang, Shen, X. Liu, B. Y. Lin, Fu, W. Chen, and X. Yue. Opencodeinterpreeer. Integrating code generation with execution and refinement. arXiv preprint 2rXivv2402.144: 2024.
[83] D. 7hu, Z. Sun, Z. Li, T. Shen, K. Yan, S. Ding, K. Kuang, and C. Wu. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. In ICML, 2024.
[84] W. Zhu and M. Tan. SPT: Learning to selectively insert prompts for better prompt tuning. In EMNLP, 2023.
[85] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. C.risttiano, and G. Irv- ing. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1ete:99900000 08593, 2019.
14

---

Table 6: The detailed numbers and more results of the experiment in Figure
discarded ranks
Test Data
Method
16
32
128
512
1024
Plain SVD
5.47
6.32
7.31
9.89
18.03
18.5
25.42
73.92
ASVD [?3] (with 256 Wiki samples)
5.47
6.08
6.67
7.86
8.71
12.37
20.34
Wikitexi-2
CO-SVD (with 32 Wiki samples)
5.47
5.48
5.48
5.49
5.52
5.58
5.79
6.62
CO-SVD (with 256 Wiki samples)
5,47
5.48
5,48
5,48
5,5
5.54
5.69
6,35
CO-SVD (with 256 PTB sumples)
5.47
5.49
5.5
5.52
5.57
5.74
6.25
8.69
Plain SVD
20.82
35,25
33.42
37.46
55,47
70.25
98.6
763.4
ASVD [73] (with 256 PTB samples)
20.84
33.42
32.05
31.67
35,36
40.23
51.28
93.42
PTB
CO-SVD (with 32 PTB samples)
20.75
20.75
20.76
20.78
20.83
20.91
21.17
22.68
CO-SVD (with 256 PTB samples)
20.88
20.88
20.88
20.89
20,91
20.94
21.14
22.28
CO-SVD (with 256 Wiki samples)
20.34
20.34
20.32
20.41
20.59
21.25
22.94
29.69
A Appendix: Implementation Details
A.1 Fientuning on Math, Code, and Instruction Following
For fine-tuning tasks on Math, Code, and Instruction Following, we adopt the same training setting as PiSSA [47]. Concretely, optimization is performed with the AdamW optimizer, batch size of 128, and learning rate of 2e-5. We employ cosine annealing schedules with a warmup ratio of 0.03 and do apply weight decay. Training is conducted exclusively on the first 100,000 conversations from the dataset for one epoch, with loss computation solely based on the response. Our experiments are executed on the NVIDIA A100-SXMM440/80000
GPUs. Publicly available platforms are utilized for the evaluation of world knowledge (TriviaQA, NQ open, and Web QS) 2, Code (HumanEval and MBPP) 3, and Instruction Following (MTBench)) 4.
A.2 GLUE Benchmark
To ensure fair comparison across Full fine-tuning, LoRA, DoRA, and CorDA in the GLUE benchmark, we implement all methods under the same training and evaluation settings. The Adam optimizer used with batch size of 32 and a learning rate of 4e-5 for 3 epochs, following linear learning rate schedule. The max token length is set as 128. The rank of LoRA, DoRA, and our CorDA 128. For covariance matrix collection of CorDA, sample to form a text sequence. From this sequence, 256 text segments, each containing 256 tokens, randomly sampled. The selected content for each task is follows: MRPC, RTE, and STS-B using "sentencel", CoLA and SST-2 using "sentence", and QNLI using "question". All methods are trained single NVIDIA A100-SXM4(40080GB GPU.
concatenate the representative content of each training
B Appendix: More Results
Table 6 lists the detailed numbers and more results of the experiment in Figure 2. It is shown that the number of sampled data only has a very limited impact. When the smallest 1024 ranks are discarded, using 32 samples is slightly worse than 256 samples in both Wikitext-2 and PTB. It implies that a small number of samples is enough to capture context into the principal components. Besides, collecting samples from the same dataset the one used test is able attain better performance after discarding a large number of ranks. For example, when discarding the smallest 1024 ranks, CO-SVD (with 256 Wiki samples) is better than CO-SVD (with 256 PTB samples) on Wikitext-2 (6.35 v.s. 8. L.69, and CO-SVD (with 256 PTB samples) is better than CO-SVD (with 256 Wiki samples) on PTB (22.28 v.s. 29.69). This also reveals that precisely capturing the data context in our decomposition is crucial for better maintaining the task characteristics into the principal components, and explains why our method superior to the PEFT methods without considering data context.
hhttpa://githbb wwwpb//iedde-eoooeeeetbiiiieeeee wtw.://iigcoeeeeeellaaaiiioo evaluation harness
'nttps://githur com/lo com/Im-sys//asttm.
15

---

layers, wwp.oowo.pm pmm)
15-
20
20-
70-
Jayers 0.m paot__ro)
Hath
NO open
Trivia
25
25-
25
modil _atttsk_rre
Mathh
30
13
10
20
25
32
20
30
model lyyess de aett h.oppro
NO
QA
30
10
25-
Figure 4: Covariance visualization results for "self_attn atttttttttt_tnn autn.""proj",, and "mlp.gate_ "roj"" weights the 0-th layer. Please zoom for a better view.
"self_attn.o seel__attn.o_pro) "mpppoow___pppopr""""""""
16

---

model Japwn...1..... Leef [_tttiicooooi
15
model Zapl altnue (ri)
27-
20
Kath
NO open
Trivss DA
15
25
70
75
12
20
modellayerss 20 wwt.n.o__nnn
QA
20
Figure 5: Covariance visualization results for "self attn.o proj" weights in different depth layers. Please zoom in for a better view.
17

---

C Appendix: Covariance Visualization Results
We provide the visualization results of the covariance matrices collected from three tasks MetaMath. NQ open, and Trivia QA in Figures and 5.
Since the original dimension in 4096 or 11008 will be too large to be informative, down- sample the covariance matrices into 32 x 32 and visualize their heatmaps. We provide the re- sults from the activations before different linear weights including self Latatttnkkpp .tpro. (the same as self attn. q_proj and self hhlp.dow__ppte and m1p. aap.gate_prooo (the same as mlp. aalp.up_ppee up_proj) in the first layer, and the self_attme o-proj weight in later layers. It is shown that the heatmaps from NQopen and TriviaQA (both are QA tasks) share some similar patterns (marked in red circles), which do not appear in the heatmap from the different task MetaMath. Therefore, when the inputs of different tasks are fed into an LLM, the covariance matrix from activations will exhibit different patterns. The visualization result empirically supports that the covariance matrix patterns can be used to characterize the triggered task. We use such patterns to orientate the decomposition LLM pretrained weights, make the resulting adapter initialization task-dependent.
v-proj due to the same input), self __t_t_tt__o wattn.o_pro.... __.roj
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our Introduction section does not claim any contribution out of the scope. All the contributions mentioned are supported in the later sections, see See.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this work in Sec. 4.4.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
No theory and proof are concerned by this study.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The complete implementation details to produce our experimental results are described in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: Code and trained models are publicly available with detailed instructions to implement our method.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https:
ctpps.cc/pulli///gg/ guwdee/iee///ee CCodeSubeesiaceecc sssiissiinn ssssonoliccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and test details in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report the mean and standard deviation values in Table 4 by running each result 3 times with different seeds. For the other results, the repeated experiment of full fine-tuning will consume too much time and resource.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
21

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the type of GPU in Appendix A.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
http://wwwpp..p.pppppeeeeeeeeeee httppp/cc//ccccccccccceeeeeeeeee ines?
Answer: [Yes]
Justification:
We obey the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our study is parameter-efficient fine-tuning method for better fine-tuning performance and mitigating knowledge forgetting. No societal impact is concerned by our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all the papers whose datasets are used in our experiments. We also explicitly mention the code used for evaluation in Appendix A.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No experiment involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No experiment involving crowdsourcing or human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "SI3LgbuKSJ", "reviewer": "Reviewer_527n", "summary": "The paper proposes an initializes algorithm for LoRA based fine-tuning, with the aim to maintain world knowledge and also improve training performance on the fine-tuning task at hand. The authors carefully explain the reasoning behind their initialization scheme, and conduct extensive studies to showcase the efficacy of their framework. Furthermore, they show that they can get surprising improvements on MT bench, simply by maintaining knowledge information. Overall, the paper presents a systematic approach to initializing LoRA parameters, and the general principle can guide the community towards better training of large language models.", "questions": "Please check my questions above.", "limitations": "The authors discuss the limitations of their work in section 4.4, and clearly indicate the future directions that the community can pursue starting from their work.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The strength of the paper lies in its simplistic exposition of its motivation, proposed framework, and the extensive experimental study to show the efficacy of the initialization framework. The authors begin with a clean exposition to SVD, and how one can think of the different eigenvectors in the input weight covariance matrix. With a very intuitive discussion, the authors show the directions along which the weight parameters aren't perturbed during training to maintain world knowledge. The directions are selected by looking at the covariance matrix of  knowledge-test datasets like TriviaQA, and NaturalQA. Furthermore, the authors indicate the directions along which they want to pre-capture features of the fine-tuning task at hand, which can give bigger returns compared to training LoRA parameters from scratch. Overall, the principle of data-dependent initialization is novel and can guide the community towards more systematic training designs.", "weaknesses": "Overall, I don't see much weakness with the work. I have a few questions regarding the experimental framework, which I would like to discuss during the rebuttal period.\n\n\n- How did the authors decide to maintain the last $r$ eigenvectors for world-knowledge? I believe, the authors should cite relevant works or make proper justification on this design choice.\n\n- How scalable is their proposed approach to other datasets where we may want to maintain knowledge? That is, how likely is the model to maintain knowledge on TriviaQA, if it wasn't included during LoRA initialization?\n\n- What is the necessary sample complexity for initializing the LoRA parameters, i.e. how many samples from the knowledge datasets are necessary to get a good estimate of the directions to freeze?\n\n- Is the knowledge preservation necessary at each weight parameters and each layer of the model? How do results change when the model's LoRA parameters are restricted only in the lower layers, while the higher layers are given more freedom during fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work and your valuable comments.\n\n> (1) How did the authors decide to maintain the last $r$ eigenvectors for world-knowledge? Cite relevant works or make proper justification on this design choice. \n\nAs shown in Figure 1 and Eq. (4), in the knowledge preserved adaptation, we use the last $r$ eigenvectors to build adapters, instead of maintaining them. We explain the design choice in more details as follows.\n\nOur context-oriented SVD (CO-SVD) decomposes the pre-trained weights into orthogonal abilities where the largest principle components most correspond to the task context captured by the covariance matrix. In instruction previewed adaptation (IPA), we want to better learn the target ability, so we use the first $r$ eigenvectors to build adapters. In knowledge preserved adaptation (KPA), the first components correspond to the QA ability, which is what we want to maintain. So, we use the last $r$ eigenvectors to build adapters while freezing the first ones. \nTherefore, the design motivations of IPA and KPA are not opposite. The principle components after CO-SVD both represent the ability indicated by the covariance matrix. The difference is a result of the purpose, _i.e._ better adapt these components (used as adapters in IPA) or better maintain these components (frozen in KPA). \n\nWe thank the reviewer for the suggestion of citing relevant works to justify the design choice. Actually, ASVD [65] is just based on a similar design motivation. They discard the last eigenvectors and only maintain the largest several components for model compression. In our KPA, we also maintain the principle components to preserve world knowledge, but adapt the last $r$ eigenvectors to learn new ability instead. Despite a similar motivation, we have shown that our CO-SVD is much better at capturing the characteristics of the target task than ASVD in Figure 2 and Table 6. Previous stuides [A, B, C] also adopt SVD for model compression. We will cite these works and explain the design choice of KPA and IPA more thoroughly in the revised paper.\n\n[A] Language model compression with weighted low-rank factorization, ICLR'22.\n\n[B] Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation, NeurIPS'14.\n\n[C] GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking, NeurIPS'18.\n\n> (2) How scalable is their proposed approach to other datasets where we may want to maintain knowledge? How likely is the model to maintain knowledge on TriviaQA, if it wasn't included during LoRA initialization?\n\nOur method guides the decomposition of pretrained weights by the data-dependent covariance matrices. However, the knowledge to preserve is definitely **NOT** constrained to the data used for adapter initialization. Actually, we only sample 256 questions from a QA dataset to collect the covariance matrices. The ability of our method to preserve world knowledge cannot be derived from these limited samples themselves. That is to say, what we maintain is the QA ability, instead of some specific knowledge from the collected data. \n\nWe use the covariance matrices, whose outlier patterns characterize the target task, to make the decomposition task-specific. Therefore, the same kinds of input/query (e.g. questions from TriviaQA and NQopen) have a similar effect because they both trigger a similar ability. \nWe have conducted analysis in Table 4 (TriviaQA and NQopen) and Table 5 (WizardLM-Evol-Instruct and Alpaca) comparing results with data from similar tasks. \nPlease refer to our response in **the global rebuttal -> common issue -> (2) Type/quality.**\nThese results indicate that randomly collecting context from one dataset has a scalable effect to other datasets of the same task.\n\n> (3) What is the necessary sample complexity for initializing the LoRA parameters, i.e. how many samples from the knowledge datasets are necessary to get a good estimate of the directions to freeze?\n\nAs described in Line 233 of our paper, we sample 256 questions for our experiments of knowledge preserved adaptation. \nWe collect 256 samples in all our experiments for both KPA and IPA modes. \n\nIn Table 6, we analyze the effect of sample number. We compare the results of collecting 32 and 256 samples for Wikitext-2 and PTB, respectively. In order to also investigate the effect of sample number in the KPA experiment, we collect less samples (128 and 32) from NQopen. Please refer to our response and result in **the global rebuttal -> common issue -> (1) Amount.**  \n\n> (4) Is knowledge preservation necessary at each weight parameters and each layer? How do results change when the model's LoRA parameters are restricted only in the lower layers, while the higher layers are given more freedom?\n\nWe follow the standard setting in LoRA, DoRA, and PiSSA, _i.e._, evenly using the same low rank for all linear layers. It also facilitates fair comparison with them. \nSince large deviation of lower layer weights will go through more later layers whose accumulative effect will cause a large shift for the final representation, restricting the adaptation of lower layers and finetuning higher layers with more freedom indeed may be preferable to maintain the pre-trained knowledge. \nThus we can use a small intrinsic rank ($r$) for lower layers to constrain their adaptation, and adopt a large $r$ and even full finetuning for higher layers to learn downstream tasks. \nWe can also adopt an adaptive strategy, _e.g._ based on the eigenvalue distribution after our CO-SVD, to assign ranks. \nThese strategies may be more parameter-efficient than using the same rank for all layers. \nBesides, Transformer different weights and MLP blocks may also play different roles in maintaining pre-trained knowledge. Investigating the importance of different weight parameters in preserving knowledge and assigning adapters accordingly will be a valuable extension of CorDA and deserve our future exploration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AOvd3zzLBq", "reviewer": "Reviewer_Mgdn", "summary": "This paper introduces a new parameter-efficient fine-tuning (peft) method called Context-oriented Decomposition Adaptation (CorDA). Although fundamentally similar to LoRA, it differs in that it initializes two low-rank matrices using the SVD results of pre-trained weights reflecting the data context. To incorporate the data context, few samples specific to the task are required, utilizing the input activations of these samples for the large language model (LLM). CorDA can be applied in two ways: knowledge-preserved adaptation and instruction-previewed adaptation. The former aims to retain specific knowledge during fine-tuning, while the latter is used to adapt the LLM more effectively to the desired instructions.", "questions": "- It would be beneficial to include visualizations of the covariance matrix derived from different data.\n- The paper should incorporate a data selection strategy for $C$ and an extensive analysis on $r$.", "limitations": "The authors adequately addressed the limitations in Section 4.4.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Originality**: The approach of modifying the LoRA method to maintain or enhance the performance of language models on specific data types appears to be a novel attempt. This method has not been explored in existing works dealing with parameter-efficient fine-tuning, thus offering originality.\n- **Quality**: Experimental results demonstrate that CorDA is more effective than existing methods like LoRA or PiSSA in fine-tuning models such as Llama and RoBERTa.\n- **Significance**: The idea of using data to derive effective low-rank matrices could inspire future research.", "weaknesses": "- **Originality**: There are methodological similarities with PiSSA, and comparisons with existing works like AdaLoRA [1] are lacking.\n- **Quality**: The exact role of the covariance matrix $C$ is not clearly explained. The paper states in lines 145-147 that “the covariance matrix of each layer’s activation will exhibit different outlier patterns as they are responsive to the task triggered to highlight **different aspects** of the pre-trained weight.” Providing visualizations to show how data from different tasks trigger different parts of the weight would enhance clarity.\n- **Clarity**: The amount and type of data used to derive the covariance matrix $C$ significantly impact performance, but the paper lacks a clear discussion and analysis on this. Moreover, there is no guidance on how to choose $r$ in Equations 4 and 5, which appears to be a crucial hyperparameter. This omission makes it challenging to practically apply the proposed method based solely on the information provided in the paper.\n- **Significance**: Although the research seems advantageous compared to existing parameter-efficient fine-tuning methods, it is unclear how much and what quality of data is needed to derive $C$ for better performance. The usability of the CorDA heavily depends on the amount and quality of data required. Despite this, the concept of using data to derive better low-rank matrices is novel.\n\nReference\n\n[1] AdaLoRA: Adaptive Budget Allocation For Parameter-efficient Fine-tuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work and your valuable comments. \n\n> (1) Originality: methodological similarities with PiSSA, and comparisons with existing works like AdaLoRA\n\n_Similarity with PiSSA:_\n\nBoth our method and PiSSA adopt SVD for the pre-trained weights, however, our method has fundamental difference and advantage compared with PiSSA as follows:\n\n- Activation awareness has been proven to be important in LLM compression and quantization. But in PEFT, existing studies rarely consider task context. PiSSA provides a better adapter initialization than LoRA using SVD, but still does not consider any task context. **Although our method also adopts SVD, we never claim that the novelty of our method lies in the usage of SVD to build LoRA adapters. The fundamental originality is that we introduce task context, captured by the covariance matrix from activations, to orientate the decomposition of weights, such that the resulting adapter initialization is task-dependent**. \n\n- Existing PEFT methods including PiSSA rarely support the option of finetuning with pre-trained knowledge better preserved. Since our method is task-dependent, we offer such flexibility enabling both knowledge-preserved adaptation and instruction-previewed adaptation, customized for the actual need. \n\n- Even when the maintenance of pre-trained knowledge is not a concern, our method in instruction-previewed adaptation surpasses DoRA and PiSSA in the three downstream tasks of Math, Coding, and instruction following, as shown in Table 2 of our paper. It is noteworthy that both DoRA (ICML24) and PiSSA (arXiv:2404.02948, online in April 2024, within 2 months of our submission) are the latest studies, and PiSSA actually could be considered as contemporaneous. \n\nTherefore, we believe that **our originality and contributions, including the methodology of introducing task context into LoRA adapter initialization, and the experimental performance of both maintaining pre-trained world knowledge and improving downstream task ability, should NOT be unappreciated just because we also adopt SVD in the adapter initialization process**. \n\n_Comparison with AdaLoRA:_\n\nOur method improves the LoRA adapter initialization by introducing task context, and adopts the low intrinsic dimension $r$ following the standard setting in LoRA.\nAdaLoRA aims to dynamically adjust $r$ during finetuning. So, CorDA and AdaLoRA are methods focusing on different aspects (how to better build adapters and how to dynamically adjust the rank). \nIn our experiments, we mainly compare CorDA with methods of the same stream, PiSSA and DoRA. PiSSA also focuses on the LoRA adapter initialization using SVD, but does not consider task context. DoRA builds the adapter with a normalization and a learnable magnitude, and also does not consider task context. It is noteworthy that both DoRA (ICML 24) and PiSSA (released on Arxiv in April 2024) are recent studies and are strong baselines. Besides, full parameter finetuning is the most direct reference because it usually has the best finetuning performance without considering parameter efficiency. For downstream tasks, as shown in Table 2 and Table 3, our method achieves finetuning performances on par with full parameter finetuning, and better performances than the compared PEFT methods LoRA, DoRA, and PiSSA. For comprehensive ability (with knowledge benchmarks included), as shown in Table 1, our method has the best average performance among full parameter finetuning and the PEFT methods. Therefore, our experimental results are already able to demonstrate the effectiveness of our proposed method. \n\n> (2) Quality: The exact role of the covariance matrix $C$ is not clearly explained. The paper states that “the covariance matrix of each layer’s activation will exhibit different outlier patterns as they are responsive to the task triggered to highlight different aspects of the pre-trained weight.” Providing visualizations to show how data from different tasks trigger different parts of the weight would enhance clarity. \n\nWhen the inputs of different tasks are fed into an LLM, the covariance matrix from activations will exhibit different patterns. We use such patterns to orientate the decomposition of LLM pretrained weights, to make the resulting adapter initialization task-dependent. \nWe did not provide the visualization of the covariance matrix because the dimension in 4096 or 11008 is too large to be informative. In the rebuttal, we downsample the covariance matrices into 32 $\\times$ 32 and visualize their heatmaps. \nPlease refer to our response in the **global rebuttal and its PDF attached**.\nIt is shown that the heatmaps from NQopen and TriviaQA share some similar patterns (marked in red circles), which do not appear in the one from the different task MetaMath. \nWe hope this result can further justify that the covariance matrix patterns can be used to characterize the triggered task. \n\nBesides, we can find some supports from the literature. OWQ [27] proposes outlier-aware weight quantization based on covariance matrix. ASVD [65] performs SVD considering activations for compression. But as shown in Figure 2 and Table 6, our CO-SVD based on covariance matrix instead of only the activation itself, is much better at capturing the task context.\n\nWe will add these visualization results and more discussion about the literature support in the revised version of our paper.\n\n> (3) Clarity and Significance: \"The amount and type of data used to derive the covariance matrix $C$ significantly impact performance, but the paper lacks a clear discussion and analysis on this. Moreover, there is no guidance on how to choose $r$ in Equations 4 and 5, which appears to be a crucial hyperparameter.\"\n\n**We address your concern about the impact of data amount and type to derive $C$ on performance and how to choose $r$ in the global rebuttal. Please refer to the global rebuttal-> common issue.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qnLkC5vl04", "reviewer": "Reviewer_HCBt", "summary": "The paper proposes a context-oriented decomposition adaptation method for large language models (LLMs) called CorDA. This method constructs learnable adapters by considering the context of downstream tasks or world knowledge. It can bridge the performance gap between parameter-efficient fine-tuning (PEFT) and full-parameter fine-tuning, while also mitigate the problem of catastrophic forgetting. Specifically, the method decomposes the weights of LLMs using singular value decomposition (SVD) guided by the covariance matrix of input activations. Through this approach, the model can identify important weights related to specific contexts, thereby creating context-aware adapters. These adapters can be customized for different tasks, enabling the model to retain general knowledge while enhancing performance on specific tasks.", "questions": "none", "limitations": "none", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1: The structure of the paper is relatively clear. The introduction progressively leads readers to understand the current state of fine-tuning LLMs and the challenges posed by knowledge forgetting after fine-tuning. This approach allows readers to quickly grasp the current situation of LLMs fine-tuning. In the subsequent method section, related modules are introduced based on these challenges.\nS2:The main challenge addressed in this paper is the catastrophic forgetting of knowledge when fine-tuning LLMs. To tackle this, the authors propose a context-oriented decomposition method and introduce two modules. The first module generates an adapter that retains the model’s general knowledge by activating the covariance matrix using other question-answering datasets. The second module, designed to adapt to task-specific instructions, is called the instruction preview adaptation module, which generates adapters specific to the given tasks. The paper thoroughly explains the implementation process of the method, including the calculation of the covariance matrix and the application of SVD.\nS3:The experimental validation in this paper is comprehensive, verifying the effectiveness of each module. The results of the modules are thoroughly analyzed through extensive experiments, demonstrating the effectiveness of the proposed modules.", "weaknesses": "S1:In the experimental section, it is necessary to add a description of the experimental environment and provide parameters to offer readers more possibilities for replication and ensure authenticity. Additionally, the paper lacks specific quantification of catastrophic forgetting in the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work and your valuable comments. \n\n> (1) It is necessary to add a description of the experimental environment and provide parameters to offer readers more possibilities for replication and ensure authenticity. Lacks specific quantification of catastrophic forgetting in the model.\n\nWe have provided the implementation details in our main paper and Appendix A. Concretely, we describe the data choice and number to collect covariance matrix, the rank choice, and datasets used in the experiment section of our paper. In Appendix A, we provide the training settings (optimizer, batchsize, learning rate, etc), the GPU device we use, and the evaluation tools. Our code and models will be publicly available. Please let us know if the reviewer has any question about our experimental details.  \n\nUsually the average performance and the performance drop compared with the model before training/finetuning are the common quantification metrics to evaluatuate catastrophic forgetting. We have shown the average performance over world knowledge benchmarks and downstream tasks in Table 1. \nIt can reflect the comprehensive ability of the finetuned model in maintaining the pre-trained knowledge while learning the new task. \nBesides, for the world knowledge benchmarks (Trivia QA, NQ open, and WebQS columns), the performance difference between the second row (LLaMA-2-7B) and the rows below shows the performance drops of each finetuning method compared with the original LLaMA-2-7B model. It is shown that our method enjoys the lowest performance drop in most cases. We will mark the performance drop beside these numbers to show the advantage more explicitly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ok0d77gWvf", "reviewer": "Reviewer_kduk", "summary": "This paper proposes CorDA, a context-oriented decomposition adaptation method that initializes LoRA adapter with different components from the weight decomposition to support two different options: knowledge-preserved adaptation and instruction-previewed adaptions. Through extensive experiments on LLaMA-2-7b and RoBERTa, the paper shows that the knowledge-preserved adaptation can better preserve performance on general knowledge task and the instruction-previewed adaptations can yield better target task performance compared to the baseline PEFT methods.", "questions": "1. Does the major difference between CorDA and LoRA lie in the initialization? Have you compared with other PEFT methods that focuses on initialization technique? Currently, the baseline comparison is not very comprehensive.", "limitations": "Yes, it's discussed from Line 293.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The experimental results on the instruction-previewed mode are strong. Table 2 and Table 3 demonstrate that the purposed method yields better performance when fine-tuning on targets compared with other PEFT methods and its performance is nearly on par with full finetuning.\n2. The proposed method is intuitive and simple in concept. If the decoupled components capture context in the principle components decoupled, the two modes are well-motivated.", "weaknesses": "1. One major concern of this work is the lack of theoretical support, and some arguments in the paper are not well justified.\n    - Line 45-47: “The covariance matrix of each layer’s activation … are responsive to the task triggered to highlight different aspects of the pre-train weight.” Is there any support for this or is this a pure intuition?\n    - Why do you perform SVD on $WC$ ? What does $WC$ represent?\n2. The knowledge-preserved mode is not very compelling given this method will hurt the target task performance and cannot achieve the best of both worlds by design. Although the authors have already recognized this when stating the limitations, I don’t know why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode. Regarding the design of method to achieve the best of both worlds, [1] may be a pointer for reference as it also studies how to preserve the forgetting of general knowledge in LMs when continual training them.\n\n[1] Continual Pre-training of Language Models, Ke et al., ICLR 2023\n\n**Update:** Most of points are addressed by the author response, so I update my score from 4 to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contributions of our work and your valuable comments.\n\n> (1) One major concern is the lack of theoretical support, and some arguments are not well justified. Is there any support for Line 45-47 or is this a pure intuition? Why SVD on $WC$? What does $WC$ represent?\n\nWe first highlight the significance of our work even though it is lacking a theoretical support, and then provide some supports from literature and our empirical analysis justifing Line 45-47. Finally, we explain our method of SVD on $WC$. \n\n_1. Significance of our work even with no theory:_\n\nDespite the lack of theory, the contribution of our work is significant for the following reasons\n\n- **It is not necessary to use a theory to show its validity.** Activation awareness has been proven to be important in LLM compression and quantization [27, 35, 65]. But in PEFT, existing studies rarely consider task context. \nSo, a fundamental originality of our method is that we introduce task context such that the resulting adapter initialization is task-dependent. We provide some other supports (in _\"2. Supports for Line 45-47\"_) for why using covariance matrix and its advantage over only using the activation itself. \n\n- Existing _PEFT methods_ rarely support the option of finetuning with the pre-trained knowledge better preserved (**[1] is not a PEFT method**). Since our method is task-dependent, we offer such flexibility enabling both knowledge-preserved adaptation and instruction-previewed adaptation, customized for the actual need. When knowledge maintenance is a need, the former satisfies this demand. When we only want to push the limit of the downstream tasks without concerning about the pre-trained knowledge maintenance, the latter is favorable.\n\n- In experiments, compared with LoRA, our knowledge-preserved adaptation achieves improvements on both worlds (shown in Table 1), enjoying better results than LoRA on both world knowledge benchmarks and downstream tasks, and having the best comprehensive average performance among all the compared methods. \nOur instruction-previewed adaptation is able to further improve the downstream task performance, surpassing LoRA, DoRA, and PiSSA in all the three finetuning tasks (shown in Table 2). In either case, the experimental performance is a progress in the field. \n\nTherefore, we believe that our contributions should not be unappreciated just because our method lacks a theoretical support, which is actually a common issue in existing PEFT methods. Actually, there is no theorem in the original LoRA paper and most of the follow-up studies. \n\n_2. Supports for Line 45-47:_\n\n- **Support from literature**. Activation awareness has been proven to be effective in LLM compression and quantization [27, 35, 65]. OWQ [27] proposes to leverage the outlier awareness from covariance matrix to make weight quantization task-specific. ASVD [65] also considers outliers, scaling the pre-trained weight with the input activation to perform SVD for model compression. The rationality behind these studies is that the outlier in activation or covariance matrix is responsive to the task triggered by the input, which is also our motivation of using covariance matrix to orientate the decomposition.\n\n- **Support from empirical analysis**. As suggested by Reviewer Mgdn, we visualize the covariance matrix collected from samples from the three tasks, MetaMath, NQ open, and Trivia QA. Note that we downsample the covariance matrices from a high dimension (4096 or 11008) into $32 \\times 32$, and visualize their heatmaps. Please see the **PDF file of the global rebuttal** for the visualization results. It is shown that the heatmaps from NQopen and TriviaQA (both are QA tasks) share some similar patterns marked in red circles, which do not appear in the heatmap from the different task MetaMath. The visualization result further supports that covariance matrix can be used to characterize the triggered task. \n\n- **Support from experimental results**. As stated above, ASVD and our method has a similar motivation. ASVD performs SVD on the pre-trained weights scaled by input activation for model compression. We use covariance matrix to orientate the decomposition of weights for building adapters. We compare our context-oriented SVD (CO-SVD) with ASVD in Figure 2 and Table 6 of our paper. It is shown that when discarding the smallest several components, CO-SVD is much better at maintaining the performance of WikiText-2 and PTB than ASVD and Plain SVD. This result indicates that our CO-SVD has a stronger ability in assembling task context into its principle components. Therefore, covariance matrix is better choice for us to build task-dependent adapters.  \n\n_3. Why SVD on $WC$ and what does $WC$ represent:_\n\nAs we respond above, our CO-SVD, using covariance matrix to orientate the decomposition of pre-trained weights, has a strong ability in assembling task context into the principle components. As shown in Figure 2 and Table 6, our CO-SVD, _i.e._ SVD on $WC$, is much better than the plain SVD that does not include task context, and ASVD that performs SVD on the weights scaled by activation. \nSo, we perform SVD on $WC$, where $W$ is the pre-trained weight and $C$ is the covariance matrix collected from a few samples. Its principle components after decomposition are task dependent and better capture task context than only using the activation itself. \n\n\n> (2) The knowledge-preserved mode is not very compelling given this method will hurt the target task performance. Why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode.\n> \n> (3) The other questinos\n\nWe have discussion about the unique value of our knowledge-preserved mode and answer to your remaining questions in the following comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank AC and all reviewers for reviewing our submission and recognizing the contributions of our work. We are grateful for the valuable comments and suggestions. 

### 1. Response to each review

For each review, we address the major question/concern in the rebuttal. We leave discussions and answers to minor questions in a comment appended.

### 2. Covariance matrix visualization

As suggested by Reviewer Mgdn, we provide the visualization results of the covariance matrices, collected from the three tasks MetaMath, NQ open, and Trivia QA, in **the PDF attached.** Please zoom in for a better view. 

Since the original dimension in 4096 or 11008 will be too large to be informative, we downsample the covariance matrices into 32 $\times$ 32 and visualize their heatmaps. We provide the results from the activations before all the weights including ``self_attn.k_proj`` (the same as ``q_proj`` and ``v_proj`` due to the same input), ``self_attn.o_proj``, ``mlp.down_proj``, and ``mlp.gate_proj`` (the same as ``mlp.up_proj``) in the first layer, and the ``self_attn.o_proj`` weight in later layers. 
We use red circles to mark the similar patterns, which the heatmaps from NQopen and TriviaQA share but do not appear in the one from the different task MetaMath. 
The visualization result empirically supports that the covariance matrix patterns can be used to characterize the triggered task.  

### 3. Common issue by Reviewer Mgdn and Reviewer 527n: 

Reviewer Mgdn concerns about the impact of data amount and type to derive $C$ and how to choose $r$. Reviewer 527n has questions about scalability to other datasets and sample complexity. 

- _Amount and type of data_:

**We respectfully disagree with the comment that "The amount and type/quality of data used to derive the covariance matrix $C$ significantly impact performance/usability of CorDA".
Actually, we have conducted an analysis in our paper about the samples used to collect covariance matrix.**

(1) Amount

In Table 6, we analyze the impact of sample number. We compare the results of collecting 32 and 256 samples for Wikitext-2 and PTB, respectively. In both Wikitext-2 and PTB, we observe a slight advantage of using 256 samples only when discarding the smallest 1024 ranks (6.35 v.s. 6.62 for Wikitext-2 and 22.28 v.s. 22.68 for PTB). When discarding the smallest 0-512 ranks, the results of using 32 and 256 samples are very similar and are much better than Plain SVD and ASVD. Therefore, only a few samples are enough to capture the task context by the covariance matrix. In order to also investigate the effect of sample number in the knowledge-preserved adaptation (KPA), we collect less samples (128 and 32) from NQopen and compare the resutls as follows.

|Samples|Trivia QA|NQ open|WebQS|GSM8k|Math|Avg|
|---|---|---|---|---|---|---|
| 256|44.30 |9.36 |7.14 |44.58 |6.92 |22.46|
| 128|44.53 |9.15 |7.16 |44.79 |6.85 |22.50|
| 32|44.11 |9.30 |6.94 |44.70 |6.93 |22.39|

Similarly, the sample number choice within [32, 256] does not cause a huge performance deviation. **Collecting covariance matrices from only a few samples are enough to implement our method.** 

(2) Type/quality

The inputs/queries from a close task (e.g. questions from TriviaQA and NQopen) have a similar effect because they both trigger a similar ability. We have conducted analysis in our experiments. For the KPA mode, in Table 4 of our paper, the last two rows indicate that collecting data from TriviaQA and NQopen has a close performance on the QA benchmarks and the finetuning task. They are both much better than Plain SVD without considering any task context. For the IPA mode, as shown in Table 5, we collect covariance matrices from WizardLM-Evol-Instruct and Alpaca to build adapters, respectively, and finetune them on instruction following. They also lead to similar results (5.15 and 5.06) on MTBench, which are both better than the full finetuning, LoRA, and PiSSA results listed in Table 2. These results indicate that **randomly collecting context from one dataset has a scalable effect to other datasets of the same task.** 

Moreover, in Table 4, we show the standard deviation of the results run with different seeds. It is shown that **randomly sampling data does not cause a large performance deviation**, which implies that it is not necessary to specially check the data quality. 
Indeed introducing a data selection strategy when collecting the covariance matrix may bring further improvement, but we believe that this is an interesting extension of CorDA deserving our future exploration.  

**Therefore, the effectiveness of our method is NOT sensitive to the data selection and does NOT rely on a large amount of data or specific type/quality.** 

- _How to choose $r$:_

**Please note that $r$ is NOT a hyper-parameter introduced by our method.** It is just the low intrinsic dimension of the LoRA adapter. A higher $r$ has a better finetuning performance with more trainable parameters. A lower $r$ is more parameter-efficient but has lower performance. 

The goal of our study is to introduce task context into the process of adapter initialization, and accordingly enable parameter-efficient finetuning with better world knowledge maintenance or stronger finetuning performance. 
Therefore, **we follow the standard setting in LoRA, DoRA, and PiSSA**, _e.g._, usually setting $r=128$ for all linear layers. 
It also facilitates fair comparison with these methods.  In our Figure 3 of our paper, we compare CorDA with full finetuning, LoRA, and PiSSA in both $r=128$ and $r=32$, to show the effectiveness of our method in different scenarios.

Therefore, we do not need to provide any guidance on how to choose $r$. Just the same as LoRA, DoRA, and PiSSA, $r$ is a configuration to control the tradeoff between efficiency and performance, determined by user's preference.

Meta-review: The paper introduces a new PEFT method to better initialize LoRA parameters using a covariance matrix of input samples to preserve the context of the task. The authors claim that, by using input dependent covariance matrix, the model could be better adapted to either better preserving the world knowledge adaptation or achieving better instruction following adaptation. The authors compare the proposed method with various baselines and show that it is superior to the existing methods.

Summary Of Reasons To Publish:
The reviewers commonly appreciate that the paper is well-organized and clearly written. Also, the reviewers generally agree that the proposed method is novel. Reviewers agree that the experiments presented in the paper well supports the claim by comparing the baselines. There were common questions among the reviewers how much the method is sensitive to the amount and quality of the sample data. The authors addressed this issue by providing supporting experiments and the reviewers agreed.

Summary Of Suggested Revisions:
One request from reviewer kduk was to tone down the theoretical claim as it was not fully supported. The authors agreed on the point and promised they would tone it down.

All in all, this paper provides a new PEFT clearly.
Decision: Accept (poster)