CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning

Authors: yibo  yang
xiaojie  li
zhongzhu  zhou
shuaiwen leon song
jianlong  wu
liqiang  nie
bernard  ghanem
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 20:14:07

Abstract: Current parameter-efficient fine-tuning (PEFT) methods build adapters widely agnostic of the context of downstream task to learn, or the context of important knowledge to maintain. As a result, there is often a performance gap compared to full-parameter fine-tuning, and meanwhile the fine-tuned model suffers from catastrophic forgetting of the pre-trained world knowledge. In this paper, we propose **CorDA**, a Context-oriented Decomposition Adaptation method that builds learnable **task-aware adapters** from weight decomposition oriented by the context of downstream task or the world knowledge to maintain. Concretely, we collect a few data samples, and perform singular value decomposition for each linear layer of a pre-trained LLM multiplied by the covariance matrix of the input activation using these samples. The inverse of the covariance matrix is multiplied with the decomposed components to reconstruct the original weights. By doing so, the context of the representative samples is captured through deciding the factorizing orientation. Our method enables two options, the **knowledge-preserved adaptation** and the **instruction-previewed adaptation**. For the former, we use question-answering samples to obtain the covariance matrices, and use the decomposed components with the smallest $r$ singular values to initialize a learnable adapter, with the others frozen such that the world knowledge is better preserved. For the latter, we use the instruction data from the fine-tuning task, such as math or coding, to orientate the decomposition and train the largest $r$ components that most correspond to the task to learn. We conduct extensive experiments on Math, Code, and Instruction Following tasks. Our knowledge-preserved adaptation not only achieves better performance than LoRA on fine-tuning tasks, but also mitigates the forgetting of world knowledge. Our instruction-previewed adaptation is able to further enhance the fine-tuning performance to be comparable with full fine-tuning, surpassing 
the state-of-the-art PEFT methods such as LoRA, DoRA, and PiSSA.

Main paper: 
 introduction: 
 Large language models (LLMs) have shown remarkable abilities in a wide range of challenging tasks, including questing-answering [11, 51] , common sense reasoning [6] , and instruction following [85] . While being powerful, LLMs demand exorbitant computation and memory cost when fine-tuning the whole model on downstream tasks due to the huge model capacity. To enable resource-friendly adaptation on downstream tasks, parameter-efficient fine-tuning (PEFT) methods are proposed to largely reduce the number of trainable parameters, by only fine-tuning the newly added adapters [20, 21, 17] or tokens [31, 37, 53] , with the original pre-trained weights frozen. Among these PEFT methods, LoRA [21] is increasingly attractive because it is able to keep the model architecture unchanged after fine-tuning so does not induce extra burden in inference. LoRA suggests that the weight change in fine-tuning presents a low rank structure, and employs low-rank matrices with a low hidden dimension to approximate the adaptation [21] . Following studies introduce adaptive low rank choice among different layers [78, 59, 75] , decouple the learning of magnitude and direction [42] , combine LoRA with pruning or quantization [10, 65, 38, 77] , and further reduce the number of trainable parameters [26, 55] . However, existing studies build learnable adapters without considering any data context. As a result, these initialized adapters are task-agnostic, and may not be the optimal choice for the downstream task to learn. Moreover, even if PEFT methods only train a small number of parameters, the fine-tuned model will still suffer from catastrophic forgetting, losing much of the world knowledge contained in the pre-trained LLM [13, 61, 83] . As shown in our visualization (Figures 4 and 5 in Appendix C) for the covariance matrices with input from different datasets, similar outlier patterns can be observed for inputs belonging to the same kind of task, which implies that the responsive parts of the LLM pre-trained weights are different when different tasks are triggered. Therefore, the adapter should be built upon the components most associated with the task of concern, e.g., a new ability to learn or the QA ability to maintain pre-trained world knowledge.To this end, we propose a task-aware PEFT method named as CorDA, based on Context-oriented Decomposition Adaptation. It adopts the same low-rank design as LoRA [21] , namely introducing two low rank matrices for each linear layer as a learnable adapter, but associates the context of world knowledge or fine-tuning task with the process of building these adapters. First, we randomly collect a few data samples and assume that they contain representative context of the corresponding task. For example, the question from a question-answering dataset well indicates the ability of preserving the corresponding knowledge, and the query to write a code carries the context of the coding task. We feed these samples into a pre-trained LLM, and obtain the covariance matrix of the input activation of each linear layer, i.e., C = XX T ∈ R din×din , where X is the input of this layer. We then perform singular value decomposition (SVD) for the weight W ∈ R dout×din multiplied by the covariance matrix, i.e., SVD(W C) = U ΣV T , where U and V are singular vectors and Σ is the diagonal matrix with the singular values arranged in descending order. In this way, the representative context expressed by these covariance matrices is able to direct the factorizing orientation. Finally, the inverse of these covariance matrices is multiplied with the decomposed components to hold the same inference result with the original model at initialization, i.e., Ŵ = U ΣV T C -1 , where Ŵ is the weight after decomposition and reconstruction.Our method supports two optional modes for practitioners, knowledge-preserved adaptation and instruction-previewed adaptation. LLM fine-tuning on downstream tasks is always accompanied by the damage of world knowledge acquired from massive pre-training data [13] . Our knowledgepreserved adaptation enables to learn new tasks effectively while keeping world knowledge as sound as possible. In this mode, we use questions from question-answering dataset, such as TriviaQA [23] and Natural Questions [27, 29] , to obtain the covariance matrices whose pattern corresponds to the LLM ability in retrieving knowledge. After SVD with these covariance matrices, we use the components with the smallest r singular values, i.e., U [:,-r:] , Σ [-r:] , and (V T C -1 ) [-r:,:]1 , to initialize a learnable low-rank adapter, and the other components that are key to preserving knowledge are frozen. Alternatively, when one only aims to achieve performance as high as possible on the finetuning task without concern for world knowledge maintenance, our instruction-previewed adaptation is suggested. In this mode, we use the instruction and response from the fine-tuning task, e.g. query to write a code and its answer, to produce the covariance matrices. Similarly, the pre-trained weights will be decomposed in an orientation such that the context of the fine-tuning task dominates in the principle singular values and vectors. Therefore, we use the largest r components, i.e., U [:,:r] , Σ [:r] , and (V T C -1 ) [:r,:] , to initialize a learnable low-rank adapter, with the other components frozen. The adapter built upon the context of the fine-tuning task well accommodates the new ability, and thus leads to a better fine-tuning performance.Our method brings flexibility in choosing between stronger fine-tuning performance or more preserved world knowledge, and can be adopted according to the practical demand. Both the two modes are as efficient as LoRA [21] . After fine-tuning, the adapter can be merged with the frozen part in the same manner as LoRA. In experiments, CorDA in knowledge-preserved adaptation not only enjoys better fine-tuning performance than LoRA on Math [9, 71] , Code [8, 3] , and Instruction Following [80] , but also largely mitigates the deterioration of performance on world knowledge benchmarks including TriviaQA [23] , NQ open [29] , and WebQS [4] . The instruction-previewed adaptation is able to further strengthen the performance on fine-tuning tasks, surpassing the state-of-the-art PEFT methods including LoRA [21] , DoRA [42] , and PiSSA [47] . 
related work: 
 Parameter-Efficient Fine-Tuning. Since large language models (LLMs) have tens and even hundreds of billions of parameters [1, 5] , full-parameter fine-tuning will cause unbearable computation and memory cost [79, 67] . Parameter-efficient fine-tuning (PEFT) is developed to reduce resource consumption by only fine-tuning a small number of learnable parameters [12, 64] . Adapter-based methods introduce additional modules into LLMs and only fine-tune them during fine-tuning [20, 17, 30, 45, 49, 24] . Another line of research appends extra soft prompts into the input or hidden layers and only train these learnable vectors [31, 37, 53, 84] . However, most of these methods change the model architecture or increase the inference burden. Based on the insight that the weight change after fine-tuning possesses a low rank structure [32, 2] , low-rank adaptation (LoRA) [21] proposes to use two low-rank small matrices as the learnable adapter, without modifying model architecture or bringing inference cost after fine-tuning. LoRA has inspired a range of variants that employ adaptive low rank in different layers [78, 59, 75] , explore the adapter design [42, 7, 50, 79] , combine LoRA with pruning [77] , quantization [10, 65, 38] , and mixture-of-expert [41, 13] , and introduce alternative way to initialize the adapter [47] . Nevertheless, existing PEFT methods rarely consider data context when building the learnable adapter. Data context has been proved to be instrumental in guiding quantization and compression [40, 73, 28] . In our study, we utilize data context for PEFT, building adapters based on context-oriented decomposition to better maintain world knowledge or accommodate new ability. Knowledge Forgetting. Deep learning models are prone to drastically forgetting the acquired knowledge when adapting to a new task, known as catastrophic forgetting [14, 54, 25, 52, 44, 70, 36] . A series of methods has been proposed to mitigate this issue using knowledge distillation [39, 19, 33] , rehearsal [56, 69] , and dynamic architecture [66] . In the era of large models [62] , however, world knowledge is acquired by pre-training on massive data, which could be intractable to re-use in fine-tuning [34, 35] . The huge model capacity also hinders the feasibility of knowledge distillation and dynamic architecture, especially in the case of continuous fine-tuning [16, 74, 57, 15, 22] . Some studies introduce extra LLaMA layers [61] or mixture of experts [13] with the pre-trained layers frozen to strike a balance between keeping world knowledge and learning new tasks. Alternative approaches adopt merging schemes to enable diverse abilities [76, 72, 83] . Different from these studies, our method enables to achieve world knowledge maintaining in the process of parameterefficient fine-tuning, without changing model architecture or relying on a post-merging step. 
method: 
 We review the LoRA method in Sec. 3.1. We develop our context-oriented decomposition in Sec. 3.2, which provides the basis of our knowledge-preserved adaptation and instruction-previewed adaptation introduced in Sec. 3.3 and Sec. 3.4, respectively.LoRA suggests that the weight change in LLM fine-tuning presents a low rank structure, and thus proposes to use the product of two low rank matrices to learn the weight change with the pre-trained weights frozen during fine-tuning [21] . Given the pre-trained weight W ∈ R dout×din from an LLM, the weight after fine-tuning can be formulated as:EQUATIONwhere W * is the weight after fine-tuning, ∆W is the weight change, and BA is the low rank decomposition of ∆W into two smaller matrices B ∈ R dout×r and A ∈ R r×din with an intrinsic rank of r ≪ min (d out , d in ). In this way, the number of learnable parameters can be largely reduced Despite the success of LoRA-based methods, when building the learnable adapter, existing studies widely ignore the data context from the target ability that users are particularly concerned with.Pre-trained large language models are endowed with multi-faced abilities, such as answering the question regarding world knowledge, common sense reasoning, and instruction following. When different kinds of input messages are fed into an LLM, e.g., a question in some domain and a query to solve a math problem, even though they are processed by the same pre-trained weights, different abilities are triggered. The covariance matrix of each layer's activation will exhibit different outlier patterns as they are responsive to the task triggered to highlight different aspects of the pre-trained weight. Therefore, the covariance matrix is able to capture task context. Inspired by this insight, we leverage the covariance matrix inside LLM to build adapters catering to a certain ability.The process of our context-oriented decomposition is shown in Figure 1 . First, we randomly collect some samples from the training data of some task with interest, e.g., question answering or Math, and feed these samples into the LLM used to fine-tune. Denote X ∈ R din×BL as the input activation of a linear layer where d in is the input dimension, B is the number of samples we collect, and L represents the sequence length. We have the covariance matrix C = XX T ∈ R din×din . We then perform singular value decomposition for the weight multiplied by the covariance matrix as:EQUATIONwhere W ∈ R dout×din is the weight of this linear layer, U ∈ R dout×dout and V ∈ R din×din are orthogonal matrices containing singular vectors u i ∈ R dout and v i ∈ R din , Σ ∈ R dout×din is a diagonal matrix with singular values σ i on its diagonal arranged in descending order, and R is the rank (the number of non-zero singular values) of W C, i.e., R ≤ min{d out , d in }.To not change the inference result at the initialization of fine-tuning, we reconstruct W by:EQUATIONwhere C -1 denotes the inverse of C, and vT i is the i-th row vector of V T C -1 . In case the covariance matrix C is not invertible, we adopt a strategy to dynamically add positive values on the diagonal elements of C to ensure invertible. Concretely, we multiply a positive coefficient with the average value of the diagonal elements of C, and add it on the diagonal. Then we calculate the ℓ 2 distance between CC -1 and an identity matrix. If it is higher than a threshold, we double the coefficient and perform this step again, until the distance reaches below the threshold.After our context-oriented decomposition, the first several components of u i and vi with the largest singular values σ i depict the dominant characteristics of the task associated with C. We can decide either to maintain these key components to not sacrifice the corresponding ability, or to adapt them for better performance on the task, which leads to our two implementation modes in the following two subsections, respectively.We introduce knowledge-preserved adaptation that enables to learn new tasks while maintaining world knowledge. In this mode, we use the question data from question-answering training data, such as TriviaQA [23] and Natural Questions [27, 29] , to obtain the covariance matrices whose pattern corresponds to the knowledge retrieving ability of the LLM. When fine-tuning on a new task, as shown in Figure 1 , we use the last r components with the smallest r singular values in Eq. ( 3) to build learnable adapters as:EQUATIONwhere B ∈ R dout×r and A ∈ R r×din are the initialized matrices in the learnable adapter, BA = R i=R-r+1 σ i u i vT i corresponds to the last r components in Eq. ( 3), √ Σ [-r:] is a diagonal matrix with the squared root of the smallest r singular values on the diagonal, and W ′ corresponding to the first R -r components in Eq. ( 3) is frozen during fine-tuning. We have W ′ as the difference between W and BA instead of summing the first R -r components to avoid the numerical error between Ŵ and W introduced by the decomposition and inversion operations. After fine-tuning, the learned matrices B * and A * can be merged into W ′ as W * = W ′ + B * A * . This mode is featured by preserving world knowledge as the knowledge retrieving ability captured by the principle R -r components is frozen. It is also more effective than a zero-initialized adapter in learning new abilities as verified by our experiments.In the circumstance that pursuing a higher performance on the fine-tuning task is the priority, our instruction-previewed adaptation will be favorable. In this mode, we collect instruction and response from the training data used for fine-tuning, e.g. the query to solve a math problem and its answer shown in Figure 1 as an example. The prompts are fed into the LLM to produce the covariance matrices whose pattern is associated with the task to learn. We use the first r components with the largest r singular values in Eq. ( 3) to build learnable adapters as:EQUATIONwhere B and A are the initialized matrices in the learnable adapter, BA = r i=1 σ i u i vT i corresponds to the first r components in Eq. ( 3),Σ [:r] is the squared root of the largest r singular values in a diagonal matrix. Similar to knowledge-preserved adaptation, W ′ containing the remaining R -r components is frozen during fine-tuning. This mode enables the initialized adapters to pre-capture the main characteristics of the fine-tuning task, leading to stronger performance after training. A recently proposed method [47] also performs SVD and uses the first r components to initialize an adapter for fine-tuning. However, their decomposition is agnostic with respect to any data context. Our adapter capturing the task context in advance can well accommodate the new ability and lead to better fine-tuning performances in our experiments. 
experiments: 
 In experiments, we fine-tune the pre-trained large language model LLaMA-2-7b [58] on Math, Code, and Instruction Following tasks, and also apply our method to the General Language Understanding Evaluation (GLUE) benchmark with RoBERTa base [43] . The world knowledge is evaluated by the exact match scores (%) on TriviaQA [23] , NQ open [29] , and WebQS [4] . Following the settings in [47] , the Math ability is trained on MetaMathQA [71] and tested on GSM8k [9] and Math [71] validation sets. Code is trained on CodeFeedback [82] and tested on HumanEval [8] and MBPP [3] . Instruction following is trained on WizardLM-Evol-Instruct [63] and tested on MTBench [81] . The complete implementation details are described in Appendix A.We conduct an experiment to demonstrate the ability of our proposed decomposition method in Sec.3.2 to capture context in its principle components. We use different methods including the Plain SVD, ASVD [73] , and our Context-Oriented SVD (CO-SVD), to perform full decomposition of the LLaMA-2-7B pre-trained weights in all layers, and then discard the smallest r singular values and their corresponding left and right singular vectors to reconstruct the weights for testing.As shown in Figure 2 , as the number of discarded ranks increases, the performances with Plain SVD on both Wikitext-2 [48] and PTB [46] are getting worse steeply. ASVD considers data context using activation absolute mean values, and helps to relieve the deterioration compared to Plain SVD. However, when discarding more than 256 ranks, the Perplexity also diverges sharply. In contrast, our method is able to maintain a stable performance very close to the original pre-trained weights even when the smallest 1024 components are discarded. The result indicates that our method is proficient with aggregating context from limited samples (256 Wiki or PTB samples in this example) into the principle components. It also evidences the potential of our method to maintain important knowledge when freezing the principle components and to learn new abilities when adapting these components.The detailed numbers of the experiment in Figure 2 are listed in Table 6 (Appendix B), where we also test the effect of sample number and dataset choice when collecting the covariance matrices. We fine-tune LLaMA-2-7B with full fine-tuning, LoRA, PiSSA, and our proposed CorDA on Math, Code, and Instruction Following tasks. In the knowledge-preserved adaptation mode, we randomly sample 256 questions from the NQ open training set and collect covariance matrices to initialize the adapters by Eq. ( 4). We report the fine-tuning performance and also the world knowledge performance of the fine-tuned model to manifest the overall ability of both new task learning and world knowledge maintaining. As shown in Table 1a , after fine-tuning on Math, all the three compared methods, full fine-tuning, LoRA, and PiSSA, suffer from drastic performance drop on TriviaQA and NQ open.Especially on NQ open, the ability is almost lost. PiSSA achieves the best accuracies on both GSM8k and Math, but meanwhile has the lowest results on the world knowledge benchmarks among the four methods. As a comparison, our method not only enjoys better fine-tuning performances than LoRA, but also achieves the best results on the three world knowledge benchmarks.A similar pattern can be also observed in the fine-tuning of Code. As shown in Table 1b , full fine-tuning has the best ability on HumanEval and MBPP, but is the lowest on world knowledge performance. It is understandable that CorDA in knowledge-preserved adaptation is not as advantageous as PiSSA for fine-tuning performance, because PiSSA uses the largest singular values and their singular vectors as the adapter that dominates the weight update, while we keep them frozen and adapt the smallest components. Nevertheless, our method has the best average score in all the three fine-tuning tasks. A surprising result is achieved by our method in instruction following in 1c, where CorDA surpasses all the three compared methods in both world knowledge performance and the new ability evaluated by MTBench. The world knowledge on NQ open is almost intact (14.43%) compared to the original performance (14.99%) before fine-tuning. These results reveal that our knowledge-preserved adaptation is an effective way to mitigate world knowledge forgetting and improve the overall ability.When the goal is to learn the target task as much as possible without concerning the loss of world knowledge, CorDA in the instruction-previewed adaptation mode satisfies this demand as it is able to further strengthen the fine-tuning performance. In this mode, we randomly sample instruction and response from the training data to collect the covariance matrices, and adopt the largest r components to initialize the adapters by Eq. ( 5). We compare our method with full fine-tuning, LoRA, DoRA, and PiSSA in the same three tasks, Math, Code, and Instruction Following. Compared with the knowledge-preserved adaptation in Table 1 , CorDA in the instruction-previewed adaptation mode largely improves the fine-tuning performance on the five benchmarks, as shown in Table 2 . Concretely, our method achieves the best performance on GSM8k, Math, MTBench, and the average score. Compared with PiSSA that adopts a similar adapter design but with no data context, our method has better results on all the five benchmarks. The training loss curves on Math are shown in Figure 3 , where CorDA converges at a lower loss than LoRA and PiSSA in both r = 128 and r = 32. Only CorDA exhibits an obvious lower loss than full fine-tuning when r = 128. These results corroborate the benefits of data context to the initialized adapter, i.e., the pre-captured task characteristic is able to accommodate the new ability and lead to a better performance.We also apply our method to the General Language Understanding Evaluation (GLUE) benchmark [60] by fine-tuning the RoBERTa base model [43] . We adopt LoRA, DoRA, and our method with a rank of 128 for all linear layers in the model except the classification head. For our method, we sample train data from each of the tasks to initialize adapters in the instruction-previewed mode and fine-tune the corresponding task. As shown in Table 3 , our method achieves the best performance on the MRPC, CoLA, and RTE tasks, and the highest average score.Ablations. We ablate the data choice used to produce covariance matrices and the adapter building manner in Table 4 . The first row corresponds to the implementation of PiSSA that uses the largest r singular values and their singular vectors as the initialized adapter by plain SVD. If we use the smallest r components by plain SVD to build adapters, there is no apparent improvement in world knowledge benchmarks. This implies that the plain SVD cannot precisely capture world knowledge related ability into the principle components and thus freezing them does not help to mitigate knowledge forgetting. When our context-oriented decomposition (CO-SVD) is adopted with Wikitext-2, which is not closely correlated with question answering, the performance on world knowledge is much improved. When the context collected by the covariance matrix is from question answering data, i.e.,TriviaQA or NQ open, the world knowledge performance is further improved by a significant margin, and the average score is also enhanced as a result. Therefore, data context is important to orientate the decomposition process such that the characteristics of the ability concerned can be better aggregated into the principle components for maintaining or adapting. As shown in Table 5 , similar to TriviaQA and NQ open in the knowledge-preserved adaptation, collecting context from different data sources belonging to the same category, namely WizardLM-Evol-Instruct and Alpaca, also results in close performance in the instruction-previewed adaptation. It is noteworthy that CorDA with Alpaca on MTBench (5.06) is still the highest among the compared baselines in Table 2 .Limitations. The two adaptation modes developed in this paper highlight different aspects in usage. However, the knowledge-preserved mode, while being adept at maintaining world knowledge, is naturally not advantageous on fine-tuning performance compared with the instruction-previewed mode. How to develop an initialization strategy [68] for adapters combining the merits of the two modes to maximize both objectives deserves future exploration. 
conclusion: 
 In this paper, we propose a new parameter-efficient fine-tuning method, named context-oriented decomposition adaptation (CorDA). It performs singular value decomposition for pre-trained weights oriented by the covariance matrix that captures the context of the task concerned, and aggregates the context into the principle components for maintaining or adapting. Accordingly, our method is able to support two implementation modes, the knowledge-preserved adaptation to mitigate world knowledge forgetting and the instruction-previewed adaptation for better fine-tuning performance. In experiments, our knowledge-preserved adaptation not only achieves better fine-tuning performance than LoRA, but also maintains the world knowledge well, leading to the best average scores on three fine-tuning tasks. Our instruction-previewed adaptation is able to further enhance the fine-tuning performance, surpassing the state-of-the-art parameter-efficient fine-tuning methods DoRA and PiSSA. 2 , Code (HumanEval and MBPP) 3 , and Instruction Following (MTBench) 4 . 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all the papers whose datasets are used in our experiments. We also explicitly mention the code used for evaluation in Appendix A.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: To ensure fair comparison across Full fine-tuning, LoRA, DoRA, and CorDA in the GLUE benchmark, we implement all methods under the same training and evaluation settings. The AdamW optimizer is used with a batch size of 32 and a learning rate of 4e-5 for 3 epochs, following a linear learning rate schedule. The max token length is set as 128. The rank of LoRA, DoRA, and our CorDA is 128. For covariance matrix collection of CorDA, we concatenate the representative content of each training sample to form a text sequence. From this sequence, 256 text segments, each containing 256 tokens, are randomly sampled. The selected content for each task is as follows: MRPC, RTE, and STS-B using "sentence1", CoLA and SST-2 using "sentence", and QNLI using "question". All methods are trained on a single NVIDIA A100-SXM4(40/80GB) GPU.B Appendix: More Results (22.28 v.s. 29.69) . This also reveals that precisely capturing the data context in our decomposition is crucial for better maintaining the task characteristics into the principle components, and explains why our method is superior to the PEFT methods without considering data context. We provide the visualization results of the covariance matrices collected from three tasks MetaMath, NQ open, and Trivia QA in Figures 4 and 5 .Since the original dimension in 4096 or 11008 will be too large to be informative, we downsample the covariance matrices into 32 × 32 and visualize their heatmaps. We provide the results from the activations before different linear weights including self_attn.k_proj (the same as self_attn.q_proj and self_attn.v_proj due to the same input), self_attn.o_proj, mlp.down_proj, and mlp.gate_proj (the same as mlp.up_proj) in the first layer, and the self_attn.o_proj weight in later layers. It is shown that the heatmaps from NQopen and TriviaQA (both are QA tasks) share some similar patterns (marked in red circles), which do not appear in the heatmap from the different task MetaMath. Therefore, when the inputs of different tasks are fed into an LLM, the covariance matrix from activations will exhibit different patterns. The visualization result empirically supports that the covariance matrix patterns can be used to characterize the triggered task. We use such patterns to orientate the decomposition of LLM pretrained weights, to make the resulting adapter initialization task-dependent.• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).U [:,-r:] , Σ [-r:] , and (V T C -1 ) [-r:,:] represent the last r columns of U , the last r diagonal elements of Σ, and the last r rows of V T C -1 , respectively. U [:,:r] , Σ [:r] , and (V T C -1 ) [:r,:] represent the first r columns of U , the first r diagonal elements of Σ, and the first r rows of V T C -1 , respectively.https://github.com/EleutherAI/lm-evaluation-harnesshttps://github.com/bigcode-project/bigcode-evaluation-harnesshttps://github.com/lm-sys/FastChat 


Decision: The paper introduces a new PEFT method to better initialize LoRA parameters using a covariance matrix of input samples to preserve the context of the task. The authors claim that, by using input dependent covariance matrix, the model could be better adapted to either better preserving the world knowledge adaptation or achieving better instruction following adaptation. The authors compare the proposed method with various baselines and show that it is superior to the existing methods.

Summary Of Reasons To Publish:
The reviewers commonly appreciate that the paper is well-organized and clearly written. Also, the reviewers generally agree that the proposed method is novel. Reviewers agree that the experiments presented in the paper well supports the claim by comparing the baselines. There were common questions among the reviewers how much the method is sensitive to the amount and quality of the sample data. The authors addressed this issue by providing supporting experiments and the reviewers agreed.

Summary Of Suggested Revisions:
One request from reviewer kduk was to tone down the theoretical claim as it was not fully supported. The authors agreed on the point and promised they would tone it down.

All in all, this paper provides a new PEFT clearly.

Reviews: review: paper_summary: paper_summary: The paper proposes an initializes algorithm for LoRA based fine-tuning, with the aim to maintain world knowledge and also improve training performance on the fine-tuning task at hand. The authors carefully explain the reasoning behind their initialization scheme, and conduct extensive studies to showcase the efficacy of their framework. Furthermore, they show that they can get surprising improvements on MT bench, simply by maintaining knowledge information. Overall, the paper presents a systematic approach to initializing LoRA parameters, and the general principle can guide the community towards better training of large language models.
strength_weakness: strength_weakness: Overall, I don't see much weakness with the work. I have a few questions regarding the experimental framework, which I would like to discuss during the rebuttal period.


- How did the authors decide to maintain the last $r$ eigenvectors for world-knowledge? I believe, the authors should cite relevant works or make proper justification on this design choice.

- How scalable is their proposed approach to other datasets where we may want to maintain knowledge? That is, how likely is the model to maintain knowledge on TriviaQA, if it wasn't included during LoRA initialization?

- What is the necessary sample complexity for initializing the LoRA parameters, i.e. how many samples from the knowledge datasets are necessary to get a good estimate of the directions to freeze?

- Is the knowledge preservation necessary at each weight parameters and each layer of the model? How do results change when the model's LoRA parameters are restricted only in the lower layers, while the higher layers are given more freedom during fine-tuning?
questions: questions: Please check my questions above.
limitations: limitations: The authors discuss the limitations of their work in section 4.4, and clearly indicate the future directions that the community can pursue starting from their work.
paper_summary: paper_summary: This paper introduces a new parameter-efficient fine-tuning (peft) method called Context-oriented Decomposition Adaptation (CorDA). Although fundamentally similar to LoRA, it differs in that it initializes two low-rank matrices using the SVD results of pre-trained weights reflecting the data context. To incorporate the data context, few samples specific to the task are required, utilizing the input activations of these samples for the large language model (LLM). CorDA can be applied in two ways: knowledge-preserved adaptation and instruction-previewed adaptation. The former aims to retain specific knowledge during fine-tuning, while the latter is used to adapt the LLM more effectively to the desired instructions.
strength_weakness: strength_weakness: - **Originality**: There are methodological similarities with PiSSA, and comparisons with existing works like AdaLoRA [1] are lacking.
- **Quality**: The exact role of the covariance matrix $C$ is not clearly explained. The paper states in lines 145-147 that “the covariance matrix of each layer’s activation will exhibit different outlier patterns as they are responsive to the task triggered to highlight **different aspects** of the pre-trained weight.” Providing visualizations to show how data from different tasks trigger different parts of the weight would enhance clarity.
- **Clarity**: The amount and type of data used to derive the covariance matrix $C$ significantly impact performance, but the paper lacks a clear discussion and analysis on this. Moreover, there is no guidance on how to choose $r$ in Equations 4 and 5, which appears to be a crucial hyperparameter. This omission makes it challenging to practically apply the proposed method based solely on the information provided in the paper.
- **Significance**: Although the research seems advantageous compared to existing parameter-efficient fine-tuning methods, it is unclear how much and what quality of data is needed to derive $C$ for better performance. The usability of the CorDA heavily depends on the amount and quality of data required. Despite this, the concept of using data to derive better low-rank matrices is novel.

Reference

[1] AdaLoRA: Adaptive Budget Allocation For Parameter-efficient Fine-tuning
questions: questions: - It would be beneficial to include visualizations of the covariance matrix derived from different data.
- The paper should incorporate a data selection strategy for $C$ and an extensive analysis on $r$.
limitations: limitations: The authors adequately addressed the limitations in Section 4.4.
paper_summary: paper_summary: The paper proposes a context-oriented decomposition adaptation method for large language models (LLMs) called CorDA. This method constructs learnable adapters by considering the context of downstream tasks or world knowledge. It can bridge the performance gap between parameter-efficient fine-tuning (PEFT) and full-parameter fine-tuning, while also mitigate the problem of catastrophic forgetting. Specifically, the method decomposes the weights of LLMs using singular value decomposition (SVD) guided by the covariance matrix of input activations. Through this approach, the model can identify important weights related to specific contexts, thereby creating context-aware adapters. These adapters can be customized for different tasks, enabling the model to retain general knowledge while enhancing performance on specific tasks.
strength_weakness: strength_weakness: S1:In the experimental section, it is necessary to add a description of the experimental environment and provide parameters to offer readers more possibilities for replication and ensure authenticity. Additionally, the paper lacks specific quantification of catastrophic forgetting in the model.
questions: questions: none
limitations: limitations: none
paper_summary: paper_summary: This paper proposes CorDA, a context-oriented decomposition adaptation method that initializes LoRA adapter with different components from the weight decomposition to support two different options: knowledge-preserved adaptation and instruction-previewed adaptions. Through extensive experiments on LLaMA-2-7b and RoBERTa, the paper shows that the knowledge-preserved adaptation can better preserve performance on general knowledge task and the instruction-previewed adaptations can yield better target task performance compared to the baseline PEFT methods.
strength_weakness: strength_weakness: 1. One major concern of this work is the lack of theoretical support, and some arguments in the paper are not well justified.
    - Line 45-47: “The covariance matrix of each layer’s activation … are responsive to the task triggered to highlight different aspects of the pre-train weight.” Is there any support for this or is this a pure intuition?
    - Why do you perform SVD on $WC$ ? What does $WC$ represent?
2. The knowledge-preserved mode is not very compelling given this method will hurt the target task performance and cannot achieve the best of both worlds by design. Although the authors have already recognized this when stating the limitations, I don’t know why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode. Regarding the design of method to achieve the best of both worlds, [1] may be a pointer for reference as it also studies how to preserve the forgetting of general knowledge in LMs when continual training them.

[1] Continual Pre-training of Language Models, Ke et al., ICLR 2023

**Update:** Most of points are addressed by the author response, so I update my score from 4 to 6.
questions: questions: 1. Does the major difference between CorDA and LoRA lie in the initialization? Have you compared with other PEFT methods that focuses on initialization technique? Currently, the baseline comparison is not very comprehensive.
limitations: limitations: Yes, it's discussed from Line 293.

Comments: title: Discussion about the value of knowledge-preserved mode and answers to remaining questions
comment: Thanks for your responses. After going through these responses,  I decide to manintain my score.
I thank the authors for their response. After going through the response, I am maintaining my score.
Thank the author for the clarifications. The response effectively addresses my concerns, clearly highlighting the key contributions of this work in comparison to PiSSA, as well as the different patterns of covariance matrices across tasks. Therefore, I have increased the score accordingly.
Dear Reviewer kduk,

Thank you for increasing the score and your suggestions. We will rephrase the argument, and include more discussion about the empirical comparison in the revised paper. 

Authors
I thank the authors for the detailed response.

The first part of the response adequately addresses my Weakness 1. However, it's worth noticing that I am not saying every paper necessarily needs to have a theoretical part. Solid, inspiring empirical results are very important for AI/ML application. The reason why I raised Weakness 1 is the current writing of Section 3.2 gives people an impression that the paper states that the covariance matrix is tightly connected with task context/patterns, where the latter concept itself does not have a formal definition (it's really hard to articulate what task context is). I suggest weakening the argument here and discussing the empirical comparison more, as it's the empirical results that show decomposing $WC$ could lead to a better results.

Overall, the author response addresses my major concern about the method proposed in this paper, so I increase my score accordingly.
> (2) The knowledge-preserved mode is not very compelling given this method will hurt the target task performance. I don’t know why the paper emphasizes this mode in parallel with the instruction-previewed adaptation mode. [1] may be a pointer for reference as it also studies how to preserve the forgetting of general knowledge in LMs when continual training them.

**Please note that the knowledge-preserved adaptation does NOT hurt the target task performance**. As shown in Table 1, compared with LoRA, our method has better performance than LoRA in both worlds (world knowledge benchmarks and downstream tasks) in most cases. We **quote some results of LoRA and CorDA in Table 1 here** for your reference.

|Method|Trivia QA | NQ open | WebQS | GSM8k | Math | Avg. |
|---|---|---|---|---|---|---|
|LoRA|44.17 | 1.91 |6.64| 42.68| 5.92| 20.26|
|CorDA | 44.30 | 9.36 | 7.14 | 44.58 | 6.92 | 22.46 |

|Method|Trivia QA | NQ open | WebQS | MTBench | Avg.|
|---|---|---|---|---|---|
|LoRA|47.46 |10.28| 7.73| 4.60| 17.52|
|CorDA | 50.34 |14.43| 8.17| 5.05| 19.50|

More importantly, as we respond above, existing PEFT methods rarely consider or support finetuning with knowledge better perserved. There are some studies on the continual training of LLMs [1] and [14, 21] (ref. in our paper), but they are not PEFT methods. We will cite [1] in the revised version of our paper.
As shown in Table 1, when using the average performance of the both worlds to measure the comprehensive ability, CorDA in knowledge-preserved adaptation achieves the best results among all the compared methods in all the three tasks. **Therefore, we respectfully do not agree with your comment that "knowledge-preserved adaptation is not compelling" and "it hurts the target task performance".** 

Admittedly, CorDA in knowledge-preserved adaptation is not stronger than the instruction-previewed adaptation when only evaluating the downstream task performance. But please note that maintaining pre-trained knowledge better and pursuing a better finetuning performance is inherently a tradeoff (also mentioned in [14] and [21]). A method may be better than another one in both worlds, just like CorDA v.s. LoRA in the results above, but for the method itself, the two worlds are still a tradeoff. We would like to use an analogy to support. The advanced neural architecture (_e.g._ Transformer) may be better than a traditional MLP/CNN architecture in terms of both accuracy and parameter efficiency. But for the better architecture itself, higher accuracy still brings more parameters (BERT_large is better than BERT_base with more parameters). 

Therefore, when knowledge maintenance is not a concern, we introduce our instruction-previewed adaptation, which puts all its efforts for the downstream task, surpassing the competitive studies DoRA and PiSSA in the three finetuning tasks, Math, Code, and instruction following, as shown in Table 2. 
In conclusion, the knowledge-preserved adaptation and instruction-previewed adaptation highlight the comprehensive performance and the specialized ability of downstream task, respectively. We think this is also a feature of our PEFT method, allowing for customized selectivity based on the actual need. 
That is why we emphasize the two modes in parallel.  

[1] Continual Pre-training of Language Models, Ke et al., ICLR 2023


> (3) Questions: Does the major difference between CorDA and LoRA lie in the initialization? Have you compared with other PEFT methods that focuses on initialization technique? Currently, the baseline comparison is not very comprehensive.

Yes, CorDA brings task context into the LoRA adapter initialization.
Adopting the same LoRA structure not only facilitates fair comparison, but also enables to restore the orignal LLM architecture after finetuning without architectural change or introducing inference burden. 
Yes, we have compared with PiSSA, which also focues on the LoRA adapter initialization but does not consider task context. 
DoRA builds the adapter with a normalization and a learnable magnitude, and also does not consider task context. 
It is noteworthy that both DoRA (ICML 24) and PiSSA (released on Arxiv in April 2024) are recent studies and are strong baselines. Besides, full parameter finetuning is the most direct reference because it usually has the best finetuning performance without considering parameter efficiency. 
For downstream tasks, as shown in Table 2 and Table 3, our method achieves finetuning performances on par with full parameter finetuning, and better performances than the compared PEFT methods LoRA, DoRA, and PiSSA.
For comprehensive ability (with knowledge benchmarks included), as shown in Table 1, our method has the best average performance among full finetuning and the PEFT methods. 
Therefore, our experimental results are already able to demonstrate the effectiveness of our proposed method.