Learn more, but bother less: parameter efficient continual learning

Authors: fuli  qiao
mehrdad  mahdavi
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:33:54

Abstract: Large Language Models (LLMs) have demonstrated profound capabilities due to their extensive pre-training on diverse corpora. However, LLMs often struggle with catastrophic forgetting when engaged in sequential task learning. In this paper, we propose a novel parameter-efficient approach for continual learning in LLMs, which empirically investigates knowledge transfer from previously learned tasks to new tasks through low-rank matrix parameters, enhancing the learning of new tasks without significant interference. Our method employs sensitivity-based analysis of low-rank matrix parameters to identify knowledge-specific parameters between sequential tasks, which are used to initialize the low-rank matrix parameters in new tasks. To maintain orthogonality and minimize forgetting, we further involve the gradient projection technique that keeps the low-rank subspaces of each new task orthogonal to those of previous tasks. Our experimental results on continual learning benchmarks validate the efficacy of our proposed method, which outperforms existing state-of-the-art methods in reducing forgetting, enhancing task performance, and preserving the model's ability to generalize to unseen tasks.

Main paper: 
 introduction: 
 Large Language Models (LLMs) [1, 5, 32, 40] have demonstrated exceptional performance across a broad spectrum of tasks, significantly revolutionizing the landscape in diverse areas driven by artificial intelligence. Full fine-tuning all the parameters of the pre-trained models becomes prohibitive in adapting pre-trained models to a large number of downstream tasks. Thus, following LoRA [11] , multiple variants [24, 50] of low-rank adaptation have been proposed to prompt parameter-efficient learning for LLMs. AdaLoRA [50] enhances the flexibility of low-rank matrices for various device budgets by parameterizing the increments through singular value decomposition. It generalizes the matrices, enabling layer-specific rank adjustments to meet different parameter constraints.Despite the tremendous success of pre-trained models on static tasks, learning multiple sequential tasks, known as continual learning (CL), still poses significant challenges [45] . There are two primary challenges: (i) overcoming catastrophic forgetting, where a model's performance on previous tasks significantly deteriorates upon training with new data [28, 34] , and (ii) facilitating forward transfer, where knowledge from old tasks is harnessed to enhance the learning of new tasks. In the context of LLMs, continual learning extends beyond merely enhancing linguistic and reasoning capabilities and it encompasses a multi-faceted process, including continual pertaining [12] , continual instruction [52] , and continual alignment [49] .While existing parameter-efficient tuning (PET) methods for CL [27, 31, 39, 43] have primarily focused on mitigating the forgetting issue, they often overlook the equally important objective of facilitating forward knowledge transfer. This transfer is crucial as it allows the model to leverage previously acquired knowledge to better generalize to new tasks or domains. For instance, O-LoRA [43] , operating within the PET framework, proposes an orthogonal low-rank adaptation for continual instruction. This method incrementally learns new tasks in an orthogonal subspace, ensuring that the LoRA parameters from previous tasks remain fixed to minimize forgetting. Besides, compared to full fine-tuning, LoRA inherently forgets less of the source domain, serving as a form of regularization [2] . However, O-LoRA does not explicitly address knowledge transfer across different tasks. Focusing on knowledge transfer among tasks, many existing non-PET knowledge transfer methods in CL, such as Progressive Network [37] which tackles forward knowledge transfer, and CUBER [22] which employs backward knowledge transfer, while having distinctive approaches to managing knowledge across tasks, they are not directly applicable to the continual learning setting in the PET framework due to their prohibitive computational costs. Although there have been recent attempts at parameter-efficient fine-tuning of LLMs, these methods have failed when applied to CL. For instance, while the parametric knowledge transfer paradigm [53] in PET harnesses the rich knowledge embedded within a teacher's parameters by extracting task-specific parameters and injecting them into a student model via sensitivity metrics, such methods do not exist in CL for LLMs. T2 52.9 54.9 54. 5 53.9 To investigate the impact of leveraging parametric knowledge in the incremental learning of LoRA parameters on a new task, we conduct an experiment where layers from the previous task's LoRA-based layers are simply replaced at the training initialization of the new task's LoRA layers, without a predefined strategy for selecting key parameters. The results, presented in Tab. 1 and Fig. 1 using the pre-trained model T5large [32] with incremental LoRAs for Amazon Reviews and Yahoo Answers [51] , indicate that this simple layer replacement can enhance performance. Notably, Tab. 1 reveals that the best replacement strategy significantly improves the performance of both the new task T 2 and the old task T 1 compared to scenarios with no replacement.The aforementioned observations motivate that an ideal approach for continual learning of LLMs in the PET framework should take the best of both worlds by simultaneously overcoming catastrophic forgetting and promoting forward transfer for enhanced generalization across a continual stream of tasks. Motivated by these needs, we seek to explore a new dimension in CL for LLMs:How can we effectively inject knowledge from previous tasks into new tasks (for improving generalization) while maintaining the orthogonality of each task's low-rank subspaces (for mitigating forgetting) to facilitate parameter-efficient continual learning?To answer this question, we propose a novel all-for-all parameter-efficient approach for continual learning in LLMs, which empirically investigates knowledge transfer from previously learned tasks to new tasks using low-rank matrix parameters evaluated by sensitivity scores while maintaining orthogonality via gradient projection techniques. We name this method LB-CL (Learn more but bother less Continual Learning). Specifically, LB-CL first calculates sensitivity metrics for SVDbased low-rank parameters of previous tasks to guide the injection of parametric knowledge into new task parameters. Then, it supports incremental learning of new tasks in orthogonal subspaces by preserving low-rank parameters of previous tasks. When transferring knowledge, LB-CL prioritizes low-rank triplets (consisting of a singular value and its corresponding singular vectors) from past tasks based on their sensitivity scores, enabling the new task to learn more from higher-scoring triplets. To preserve the orthogonality of low-rank subspaces, we project the gradients of the new task onto the subspaces formed by previously learned low-rank triplets, encouraging the new task to deviate appropriately from more impactful triplets. Our experimental results validate that LB-CL surpasses previous state-of-the-art methods on standard continual learning benchmarks. Moreover, our analysis highlights the critical role of initialization strategies in promoting generalization through effective parametric knowledge transfer, while also using low-rank orthogonal gradient projection to mitigate catastrophic forgetting.Training Data (Task T t+1 )Seed Samples1 ⃝ Random Sample Pre-trained Model W 0 + V 1 Σ 1 U 1 + • • • + V t Σ t U t + V t+1 Σ t+1 U t+1 2 ⃝ Inference 3 ⃝ Extraction + V t+1 0 Σ t+1 0 U t+1 0 4 ⃝ Injection 5 ⃝ Training 5 ⃝ Training V t + 1 Σ t + 1 U t + 1 V t + 1 0 Σ t + 1 0 U t + 1 0 Student Teacher 6 ⃝ Orthogonal Gradient Update 7 ⃝ Mergefrozen weights trainable weightsFigure 2 : Overview of our LB-CL framework. Starting with the pre-trained model including SVD weights of previous tasks, sensitivity metrics are calculated using a set of seed samples, facilitating the extraction of task-specific knowledge. Subsequently, the extracted layer triplets initialize SVD weights for the new task. Then, the new task is trained in an orthogonal subspace, employing orthogonal gradient projection to minimize forgetting.■ Summary of Contributions. This paper makes three key contributions: (1) A novel parameterefficient continual learning framework for LLMs that balances generalization through parametric knowledge transfer and the mitigation of forgetting through low-rank orthogonal subspace learning for new tasks; (2) Through comprehensive evaluations, our method demonstrates superior performance over existing state-of-the-art approaches on standard continual learning benchmarks; and (3) We provide an in-depth analysis that deepens our understanding of the dynamics of parametric knowledge transfer within continual learning for LLMs, pinpointing critical factors that drive its effectiveness. 
continual learning maestro: learn more but bother less: 
 We consider a continual learning scenario where the learner is presented with a sequence of tasks {T 1 , T 2 , . . . , T T } over time. Each task T k is associated with a data distribution D k and contains a separate target dataset S k = {(x k,i , y k,i )} nt i=1 where x k,i ∈ X k and y k,i ∈ Y k . The goal of continual learning is to find a set of parameters θ ∈ Θ that can effectively solve all tasks up to the current task T k , while minimizing catastrophic forgetting of previously learned tasks. In continual learning of LLMs, we are given a pre-trained model W 0 , and would like to continually fine-tune on a sequence of tasks, utilizing the incremental SVD-based low-rank matrix parameters U k Σ k V k to fine-tune on task T k , where U k ∈ R d1×r and V k ∈ R r×d2 represent the left/right singular vectors and the diagonal matrix Σ k ∈ R r×r contains the singular values {λ i } 1≤i≤r with r ≪ min (d 1 , d 2 ). To enforce the orthogonality of U and V , i.e.U ⊤ U = V V ⊤ = I, we use the regularizer: R(U , V ) = ∥U ⊤ U -I∥ 2 F + ∥V V ⊤ -I∥ 2 F. The continual learning model parameters after fine-tuning on task T k is:θ k = W 0 + k m=1 U m Σ m V m .Our continual learning goal is to optimize the following objective across all tasks:EQUATIONwhereθ = W 0 + T k=1 U k Σ k V k .Our method contains two important stages: (i) Learning from knowledge extraction and injection, which transfers knowledge from previously learned tasks to new tasks by incremental SVD triplet sensitivity metric; (ii) Training in Orthogonal Subspaces, which keeps the low-rank subspaces of new tasks orthogonal to those of old tasks. The detailed description of our proposed method, LB-CL, is shown in Fig. 2 .■ Generalization and Forgetting Tradeoff of Low-rank Finetuning. Before delving into the detailed steps of proposed algorithm, here we motivate these steps by investigating the forgetting and generalization tradeoff of SVD based low-rank parameter-efficient methods in CL. The performance of a CL algorithm is typically evaluated using two key metrics [3, 4, 16, 21, 23, 25] as defined below: (i) Forgetting error: which quantifies the performance degradation on previously learned tasks after acquiring a new task formulated as:EQUATIONwhereθ t = W 0 + t k=1 U k Σ k V k , L t (•)is the generalization error on task T t , and L t (θ T )-L t (θ t ) is the performance degradation (forgetting) on tasks T t between the model after training on task T t and the model after training on the final task T T .(ii) Generalization error: which measures the algorithm's capability to effectively learn a new task while preserving the knowledge gained from prior tasks defined as:EQUATIONwhere L t (θ t ) -L t (θ * t ) measures the generalization gap between the CL model θ t and the optimally fine-tuned modelθ * t = W 0 + U t * Σ t * V t * on task T t .The generalization of final model on all tasks can be decomposed in terms of forgetting generalization errors as follows:EQUATIONIt is crucial to note that the generalization error is based on the final model over all tasks, which may not fully capture each task-specific optimal setting since each task could potentially achieve better performance metrics when fine-tuned individually. Given the complexities in addressing our generalization error due to limited theoretical support, we utilize initialization parameters to further decompose the per-task generalization term on the left-hand side of Eq. 4. This involves measuring the performance difference between the CL model θ t after learning on tasks T t and its initialization θ(0)t , evaluated on task T t as:EQUATION(II): improvement by fine-tuning+ L t (θ (0) t ) -L t (θ * t ) (III): generalization of initial modelFrom this decomposition, it is clear that all three terms must be carefully calibrated to enhance the overall performance of CL algorithm. In the context of SVD-based low-rank fine-tuning, we have:• The term (I) captures the forgetting error. It is evident that models that undergo minimal changes when fine-tuned to a new target domain will exhibit less forgetting of the source domain. This automatically holds for fine-tuning with low-rank updating methods as it acts as a regularization. This can be further amplified by fine-tuning in orthogonal subspace. • The term (II) captures the performance of fine-tuning algorithm itself. Since fine-tuning with low-rank updates is not capable of approximating full fine-tuning accuracy [2] , and in order to balance between stability (retaining old knowledge) and plasticity (acquiring new knowledge) we need to leverage methods such as orthogonal gradient projection to better align with current task. • The term (III) captures the generalization of the initial model. Since the knowledge transfer from previous tasks happens through the initial model, naive averaging of low-rank models from previous tasks might entail a dramatically poor generalization on the current task, in particular when there is a larger domain shift among tasks. Consequently, the contribution of each task should be proportional to its similarity to the current task, measured by leveraging effective discrepancy estimation methods. We formulate this process as the parametric knowledge extraction:EQUATIONwhere β = [β 1 , . . . , β t-1 ] ⊤ extracts optimal coefficients approximating the influence of new task data on each previous task SVD-based low-rank adapters. Then the parametric knowledge injection in the CL model initialization for task T t is constructed as follows:EQUATIONIn summary, while updating with low-rank models in CL can provide an implicit regularization effect to mitigate the forgetting issue, their generalization capability is limited due to the low-rank perturbation of parameters. This necessitates effective initialization to overcome the generalization limitations of low-rank updates. Specifically, an initialization that can effectively capture the shared knowledge across tasks based on their similarities to augment the knowledge captured by the pretrained model is required to facilitate knowledge transfer and improve generalization when performing low-rank updates in a CL setting.■ Sequential Task Low-rank Adapter. In our continual learning framework, for a pre-trained weight matrix W 0 , it parameterizes the incremental updates for task T t by SVD-based low-rank matrices:θ t = θ t-1 + ∆ = W 0 + t-1 k=1 U k Σ k V k + U t Σ t V t .Similar to [50] , we define G t l,i = {U t l, * i , λ t l,i , V t l,i * } as the i-th triplet (i-th singular value and its vectors) of layer l for task T t , which is different from the dependent and not orthogonal doublet in LoRA-based low-rank matrices. With the SVD-based flexible regularizer, the training objective for task T t is represented asL t (U t , Σ t , V t ) = L t (U t , Σ t , V t )+γ n i=1 R(U t i , V t i ),where n is the number of weight matrices and γ > 0 is the regularization coefficient.■ Sensitivity of the Parameters. Parameter sensitivity quantifies the impact on the loss function when a specific parameter is altered, typically set to zero, providing insight into the parameter's importance [14, 19, 26, 30, 53] . Consider a teacher model with parameters θ = [θ 1 , . . . , θ Nt ], where N t denotes the total number of parameters, the i-th parameter is θ i = [0, . . . , θ i , . . . , 0]. The sensitivity S i,j for sample x j from task T is calculated asS i,j = |θ i ∇ θ L(x j )| ≈ |L(θ)-L(θ -θ i )|,where the approximation uses the first-order Taylor expansion of L(•) at θ i . Thus, we formulate the sensitivity of each triplet of low-rank matrices, rather than each parameter of the model. By masking the l-th layer i-th triplet G k l,i = {U k l, * i , λ k l,i , V k l,i * } of low-rank matrices for previous task T k , following [50] , we can obtain the sensitivity score of the l-th layer i-th triplet of low-rank matrices for previous task T k , where k ∈ {1, . . . , t -1}, on new data sample x t from new task T t :EQUATION■ Knowledge Extraction and Injection. For simplicity in analysis and mathematical expression, we assume that each layer within every task shares the same SVD-based low-rank adapter rank r. Our first step is to assess the layers of the previous task T k low-rank matrices where k ∈ {1, . . . , t -1}, by Eq. 7, we calculate triplet-specific sensitivity scores in layer l and this layer sensitivity score is represented as S k l = r i=1 S k l,i . To preserve the inherent structure of the tasks' low-rank adapters, the layers are subsequently mapped to the new task T t adapter maintaining their original sequential order. Having computed the sensitivity scores of all triplets for layer l in previous tasks {T 1 , . . . , T t-1 }, we proceed to arrange them with weighted sensitivity scores to initialize the layer l in new task T t :EQUATIONwhere α k l,i is the weight for the components of the triplet G k l,i , andα k l,i = S k l,i t-1 k=1 S k l,iobtained from the sensitivity founded in Eq. 7. By aggregating weighted triplets across layers, we derive the extracted triplets from previous tasks' adapters and inject these triplets into the new task's adapter.We utilize the SVD-based adapter's low-rank subspace to represent the gradient subspaces of previous tasks, asserting that these parameters capture essential model update directions rather than just numerical adjustments [43] . This hypothesis allows us to minimize interference with previously learned tasks by training within a subspace orthogonal to that of the SVD-based subspace. We approximate the layer l subspace for task T k as the subspace consisting of the triplets G k l . To ensure orthogonal to the previously learned tasks layer subspaces, we first project the gradients of layer triplets of new task T t onto the previously learned tasks layer subspaces spanned by {G k l }, where k ∈ {1, . . . , t -1}, and then make the gradients far away from these subspaces. The gradients of layer l triplets of data x t from new task T t become:EQUATIONwhere proj(u, v) = ⟨u,v⟩ ⟨v,v⟩ v is the projection of u in the direction of v [7] .3 ExperimentsQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [No] Justification: The paper has limitations, but those are not discussed in the paper.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experimental setup: 
 Our experiments employ the encoder-decoder architecture of the T5-large and T5-base models [32] , consistent with previous work in CL for NLP. All experiments are conducted on NVIDIA A6000 GPUs, utilizing the DeepSpeed repository.■ Standard CL benchmark. We evaluate our approach using a CL benchmark specifically designed for language models. This benchmark comprises five text classification datasets: AG News, Amazon Reviews, Yelp Reviews, DBpedia, and Yahoo Answers, as introduced by [51] . We adhere to the CL setup for the T5 model as outlined in LFPT5 [31] and experiment with three different task orders within this benchmark. ■ Large number of tasks. Our method's efficacy is further tested on extended task sequences through a comprehensive CL benchmark encompassing 15 datasets, as detailed in [35] . This benchmark integrates tasks from three sources: five from the standard CL benchmark, four from the GLUE benchmark (MNLI, QQP, RTE, SST-2), five from the SuperGLUE benchmark (WiC, CB, COPA, MultiRC, BoolQ), and the IMDB movie reviews dataset. For each task, we train using 1000 randomly selected samples and validate using 500 samples per class, following the methodology of [35] . ■ Metrics. We define the testing accuracy on the task T i after training on the task T j as a i,j . The main metric for evaluation is Average Accuracy (AA), calculated as the mean accuracy across all tasks after training on the last task: 1 T T i=1 a i,T . ■ Baselines. We compare our method against various baseline approaches:• SeqFT [6] : train all model parameters on a sequence of tasks (without adding any regularization or replaying samples from the previous tasks). • SeqLoRA: fixed-size LoRA parameters are trained on a sequence of tasks (without adding any regularization or replaying samples from the previous tasks). • IncLoRA: incremental learning of new LoRA parameters on a sequence of tasks (without adding any regularization or replaying samples from the previous tasks). • SeqSVD: fixed-size SVD parameters are trained on a sequence of tasks (without adding any regularization or replaying samples from the previous tasks). • Replay: fine-tune the whole model with a memory buffer, and replay samples from old tasks when learning new tasks to avoid forgetting. • EWC [13] : fine-tune the whole model with a regularization loss that prevents updating parameters that could interfere with previously learned tasks.• LwF [18] : constrains the shared representation layer to be similar to its original state before learning the new task. • L2P [44] : uses the input to dynamically select and update prompts from the prompt pool in an instance-wise fashion. • LFPT5 [31] : continuously train a soft prompt that simultaneously learns to solve the tasks and generate training samples, which are subsequently used in experience replay. • L-CL: incremental learning of new SVD parameters on a sequence of tasks with initialization and SVD regularization. • B-CL: incremental learning of new SVD parameters on a sequence of tasks with gradient projection and SVD regularization. • NLNB-CL: incremental learning of new SVD parameters on a sequence of tasks (with adding only SVD regularization and without replaying samples from the previous tasks), also called IncSVD. • ProgPrompt [35] : adopts task-specific soft prompts for each task, training distinct models per task and using task IDs during inference. • O-LoRA [43] : incrementally train new tasks in an orthogonal subspace while fixing the LoRA matrices of previous tasks. • PerTaskFT: train a separate model for each task.• MTL: train a model on all tasks as multi-task learning, serving as the benchmark's upper bound of the performance limit.Tab. 2 presents a performance comparison of LB-CL and baseline continual learning methods across two CL benchmarks. Following LFPT5, we report results from three random runs with different task orders on the CL benchmark. For fairness, we use the same rank of the LoRA-based and SVD-based matrix in each corresponding comparison experiment. To reduce computation time, we reasonably focus on some high-level layers to narrow the searching range of exploring critical parametric knowledge, since previous research, e.g. [43, 50] , has already demonstrated that high-level layers are important for model performance.■ Results on Standard Continual Learning Benchmarks. Across all task orders of the standard CL benchmark, LB-CL consistently surpasses previous methods by a significant margin. Notably, LB-CL achieves performance improvements in all task orders compared to O-LoRA, the prior state-of-the-art.Our approach demonstrates performance on par with multi-task learning and significantly exceeds that of PerTaskFT. This indicates that LB-CL not only effectively prevents catastrophic forgetting but also efficiently utilizes knowledge from prior tasks to enhance the learning of new tasks.■ Performance with Large Number of Tasks. In a more demanding benchmark featuring a large number of tasks, LB-CL surpasses the state-of-the-art, O-LoRA, in terms of average performance across three task orders. Although ProgPrompt shows superior performance in managing longsequence tasks, its limitations are notable. ProgPrompt is strictly dependent on the tasks it has been trained on and relies heavily on task IDs during inference, which restricts its generalization capabilities and adaptability for use in LLMs. However, LB-CL does not use task ID during testing, which keeps its generalization. It is important to note that nearly all existing continual learning methods fall significantly short of the performance achieved by PerTaskFT and MTL, underscoring that continual learning with a large array of tasks remains a formidable challenge.■ How do initialization and gradient projection effectively influence the performance of LB-CL?We systematically evaluated the influence of two components: initialization and gradient projection.The results across various orders and their average performances, as presented in the last rows in Tab. 2, reveal distinct trends. L-CL, which employs only initialization, suggests that while this component does provide a beneficial starting point for task learning, it falls short of maintaining knowledge across tasks in Order 3. B-CL, utilizing only gradient projection, demonstrates slightly higher performance than L-CL in Order 1 and Order 3. However, the absence of initialization limits its effectiveness, particularly in establishing a robust foundation in Order 2. NLNB-CL, which neither initializes nor employs gradient projection, surprisingly performs slightly better than both L-CL and B-CL on average, but it does not excel in any specific order. This outcome suggests that the model may possess inherent adaptive capabilities or rely on other compensatory mechanisms. LB-CL, integrating both initialization and gradient projection, exhibits the highest overall performance with consistent scores across all orders. This robust performance indicates that the synergistic contribution of both components significantly enhances the model's ability to effectively handle CL tasks.■ How do different initialization strategies affect the performance? To elucidate the impact of distinct initialization strategies, we compare two different initialization strategies motivated by [29] to initialize new task T t : (i) with previous tasks low-rank matrix triplets G k , and (ii) without the singular values Σ k but leveraging {U k , V k } where k ∈ {1, . . . , t -1}. These strategies are assessed using T5-large model over three task orders in the standard CL benchmark. Our results, depicted in Fig. 3 , reveal that utilizing only {U , V } from prior tasks' triplets surpass the full triplet configuration in average performance across three task orders. While "with Σ" strategy exhibits peak performance in Order 3, "without Σ" approach demonstrates better consistency and stability. This suggests that excluding Σ may lead to a more robust generalization across diverse tasks, but it cannot fully represent previously learnt tasks' important subspaces, thus we use "with Σ" as the initialization of LB-CL, and "without Σ" can be used as an improvement strategy in implementation. Furthermore, both strategies outperform the performance of the O-LoRA method over these three orders. ■ How does the number of seed samples affect the performance of the model? The number of seed samples significantly influences the reliability and efficiency of the sensitivity score computations derived from the teacher model. We explored the impact of varying the number of seed samples from new tasks on the sensitivity of parameters from previous tasks. As illustrated in Fig. 4 , increasing the number of seed samples improves performance incrementally at a slight rate. Notably, the variance in performance metrics is considerably lower with 4 and 8 seed samples. Based on these findings, we have selected 8 seed samples as the optimal number for our hyperparameter setting.■ How does the training computation cost perform? We compare the training computation cost between LB-CL and O-LoRA in Tab. 3, using Order 1 in the standard CL benchmark with T5-large model. Tab. 3 shows that the GPU memory consumption of both methods is similar, indicating comparable resource efficiency during training. For the number of training parameters, we examine a single layer and denote r as the rank of SVD-based and LoRA-based matrix, m as the input dimension, and n as the output dimension of the layer. Given that r ≪ min(m, n), the number of training parameters for both methods remains close, further highlighting their efficiency in parameter. We analyze the distribution of parametric knowledge across the model's layers to identify those most critical for retaining task-specific information. In Fig. 5 , our analysis of sensitivity and Fisher information on the average of three task orders in standard CL benchmark, reveals that higher-level layers, particularly in decoder layers, exhibit significant sensitivity. Particularly, the top 4 layers of the decoder are notably sensitive, suggesting that focusing sensitivity analyses on these layers could represent the entire decoder effectively, thus reducing computational demands. Validation with Fisher information confirms that these high-level layers are crucial in both the encoder and decoder, especially the top 3 layers of the decoder. This alignment underscores that our sensitivity scores effectively identify the most crucial layers for task-specific knowledge transfer. Given the time-intensive nature of Fisher information calculations, our sensitivity score approach provides a more efficient alternative, enhancing training efficiency by focusing on several critical high-level layers. ■ What's the optimal rank r for LB-CL? To explore the impact of the rank r on the performance of LB-CL, we conduct experiments using the T5-large model on the standard CL benchmark. The results, presented in Tab. 4, examine how varying r affects the accuracy across different task orders. It shows that increasing the rank r does not lead to a significant improvement in model performance. Furthermore, the small standard deviations across different orders for each rank underscore the model's consistent performance, irrespective of rank variations. This suggests that by leveraging more knowledge from previous tasks, our method allows the gradient space of the new task to diverge more significantly from those of prior tasks, thereby enhancing stability across various ranks.■ How do different pre-trained models influence performance? We investigate the impact of model scale on performance by comparing T5-base and T5-large models using a standard CL benchmark. We evaluate both our method and O-LoRA across three task orders. The results, presented in Tab. 5, clearly demonstrate significant performance differences between the two model sizes and the methods employed. For the T5-base model, LB-CL consistently outperforms O-LoRA. While for the T5-large model, LB-CL significantly surpasses O-LoRA's outcomes. Moreover, LB-CL shows exceptional consistency across all task orders in the T5-large model, highlighting its robustness and effectiveness when scaled up. This analysis confirms the influence of model size on the success of different continual learning strategies, with LB-CL proving particularly effective in larger models.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: The paper does not include theoretical results. 
conclusion: 
 In this paper, we investigate the balance between overcoming forgetting and achieving generalization in the continual learning of LLMs, decompose the generalization error with the task low-rank matrix initialization, then propose a novel framework, exploring parametric knowledge transfer between tasks and utilizing the inherent forgetting less ability of low-rank matrix. Instead of storing extra taskspecific auxiliary parameters, we just utilize the low-rank parameters which would be merged into the pre-trained model. Our experiments across standard CL benchmarks validate the effectiveness of this approach. Furthermore, we analyze the critical factors influencing initialization in CL, providing insights for further enhancements in this field.In our analysis, we aim to elucidate the distribution of parametric knowledge intrinsic to different task orders and compare the sensitivity scores with Fisher Information. The insights from Fig. 6 ,7,8 indicate the distributions of parametric knowledge across layers remains consistent among different task orders. It suggests that regardless of the task sequence, the layer-wise distribution of parameters critical for task performance does not significantly vary. Both sensitivity scores and Fisher Information depict similar patterns, underscoring the robustness of our model's learning mechanisms. Our findings also highlight that higher-level decoder layers exhibit increased sensitivity compared to their lowerlevel counterparts. This heightened sensitivity in the decoder suggests that these layers play a more crucial role in refining the outputs, possibly due to their direct involvement in generating end task results. Notably, the sensitivity scores of the encoder layers in task order 3 are higher than those observed in the first two task orders. This variation could be attributed to the specific nature or complexity of the tasks in order 3, which might demand more nuanced feature extraction capabilities from the encoder layers. BoolQA According to the following passage, is the question true or false? Choose one from the option.MultiRC According to the following passage, is the question true or false? Choose one from the option. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The experimental results are not presented as error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We report the details in the appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We focus on the techniques.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Additional Related Works Continual Learning. Continual learning is a field focused on developing algorithms that can accumulate and refine knowledge over time, particularly as they encounter non-stationary data streams. The primary challenge in this domain is overcoming catastrophic forgetting, where the performance of a model on old tasks significantly diminishes when trained with new data. To address this, existing research has generally been divided into three main categories: (i) The rehearsal-based methods utilize a memory buffer to retain data samples from previous tasks, incorporating techniques such as experience replay [36] , or constrained optimization to simultaneously train the model on current and previous tasks [8, 25] . (ii) The regularization-based methods introduce additional terms into the loss function to penalize changes in important model parameters, thereby limiting interference with previously learned tasks [7, 13, 18, 38] . EWC [13] remembers old tasks by selectively slowing down learning on the weights important for those tasks. OGD [7] constrains the parameters to move within the orthogonal space defined by the stored gradients of previous tasks. (iii) The architecture-based methods the aim is to minimize interference between tasks by dynamically expanding model capacity or creating isolated components for each task [17, 33, 35, 37, 46] . Progressive Prompts [35] enhances forward transfer and mitigates catastrophic forgetting by learning a distinct prompt for each new task and sequentially appending these new task-specific soft prompts to the ones previously learned.Parameter-efficient Tuning. Recent works on parameter efficient tuning (PET) [9] have shown that training only a subset of model parameters can yield performance comparable to training the full model, reducing computational demands and annotation efforts [10, 11, 15, 48, 50] . BitFit [48] finds that updating only the bias-items during fine-tuning is very effective. Prompt tuning [15] uses learnable 'soft prompts' through back-propagation to condition frozen language models for specific tasks. LoRA [11] uses low-rank adapters in adapting models to new tasks with minimal additional parameters, and AdaLoRA [50] improves the performance of LoRA by adaptively allocating the parameter budget based on the weight matrices importance score. While the majority of PET has focused on learning one single task, there has been several efforts to apply PET to continual learning. AdapterCL [27] introduces an individual adapter block for each task. LFPT5 [31] utilizes a large soft prompt that is continuously trained on all tasks. O-LoRA [43] incrementally learns new tasks in orthogonal subspaces while keeping the LoRA parameters learned from previous tasks fixed to minimize catastrophic forgetting. InfLoRA [20] mitigates catastrophic forgetting by reparameterizing pre-trained weights with a small set of parameters, enabling fine-tuning within a subspace to maintain previous knowledge. The proposed MoE-Adapters and DDAS collaborate in [47] mitigates long-term forgetting by dynamically expanding a pre-trained CLIP model with Mixture-of-Experts adapters and preserves zero-shot recognition through a Distribution Discriminative Auto-Selector for routing inand out-of-distribution inputs. ConPET [39] adapts existing continual learning strategies, originally developed for relatively smaller models to LLMs by incorporating PET with a dynamic replay approach. While O-LoRA addresses catastrophic forgetting through its incremental learning within orthogonal subspaces, it focuses on LoRA-based architecture rather than more general low-rank matrices and does not explore the knowledge transfer across different tasks.All our experiments involving T5 models were performed on a server outfitted with four NVIDIA A6000 GPUs, utilizing the DeepSpeed repository for implementation. For every sequence of tasks across different orders, we standardized our experimental setup as follows: A constant rate of 1e-3 was maintained throughout the experiments. We used a total batch size of 32, distributed as 8 per GPU to leverage the computational capabilities of all four A6000 GPUs efficiently. We set the dropout rate at 0.1. We applied a regularization rate of 0.1 to the orthogonal matrices derived from the Singular Value Decomposition (SVD). A rate of 0.0 was employed, indicating no additional penalty on the model's weights during training.Tab. 6 provides detailed information on the 15 datasets utilized in our continual learning (CL) experiments, including the evaluation metrics used for assessment. Our selection encompasses datasets from established benchmarks:: the standard CL benchmark [51] , GLUE [42] , and SuperGLUE benchmarks [41] , and added IMDB movie reviews dataset. → cb → wic → copa → qqp → boolqa → rte → imdb → yelp → amazon → sst-2 → dbpedia → ag → multirc → yahoo 5 T5-large multirc → boolqa → wic → mnli → cb → copa → qqp → rte → imdb → sst-2 → dbpedia → ag → yelp → amazon → yahoo 6 T5-large yelp → amazon → mnli → cb → copa → qqp → rte → imdb→ sst-2 → dbpedia → ag → yahoo → multirc → boolqa → wic A.Given a word and two sentences, whether the word is used with the same sense in both sentence? Choose one from the option. Justification: In the abstract and introduction, we list our contributions.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We present datasets, models, and implementation details in the experimental section and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material.Justification: We follow the ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).38th Conference on Neural Information Processing Systems (NeurIPS 2024).(a) Sensitivity Scores (b) Fisher Information 


Decision: The paper proposes a method for continual learning of LLMs to defy forgetting based on LoRA. The approach is novel and considers the knowledge transfer and parameters relevance among different tasks which is an interesting direction for large overparameterized models.
The method shows SOTA performance on various continual learning benchmarks.
The concerns raised by reviewers were mostly addressed by the authors. We encourage the authors to include the clarification and the additional results and computation cost estimation in the final version.

Reviews: review: paper_summary: paper_summary: This paper presents a novel approach to continual learning for large language models (LLMs). The proposed method, LB-CL, incorporates parameter-efficient tuning using low-rank subspace learning and orthogonal subspace projection to mitigate catastrophic forgetting. The study leverages incremental SVD-based low-rank matrix parameters for fine-tuning LLMs across a sequence of tasks. Comprehensive evaluations on benchmark datasets demonstrate the superiority of LB-CL over existing state-of-the-art methods in continual learning.
strength_weakness: strength_weakness: 1. The paper does not provide open access to the code and datasets used, which may hinder reproducibility and wider adoption of the proposed method.
2. The method in this paper is similar to O-LoRA [1] and has some improvements over O-Lora. This paper claims that "O-LoRA does not explicitly address knowledge transfer across different tasks." However, there is no specific evaluation in the experimental part of this paper to prove that the improvement of this paper on O-LoRA can improve "knowledge transfer across different tasks".
3. In the past, there have been some continual learning methods based on orthogonal subspaces in non-LLM fields, such as [2] and [3]. This paper does not point out the core differences between this method and these previous methods. Is this method just an application of past methods in the fields of LLM and LoRA?

[1] Wang, Xiao, et al. "Orthogonal subspace learning for language model continual learning." arXiv preprint arXiv:2310.14152 (2023).
[2] Chaudhry, Arslan, et al. "Continual learning in low-rank orthogonal subspaces." Advances in Neural Information Processing Systems 33 (2020): 9900-9911.
[3] Farajtabar, Mehrdad, et al. "Orthogonal gradient descent for continual learning." International Conference on Artificial Intelligence and Statistics. PMLR, 2020.
questions: questions: see Weaknesses
limitations: limitations: see Weaknesses
paper_summary: paper_summary: This paper proposes the Learn More but Bother Less Continual Learning (LB-CL) algorithm for Continual Learning (CL) of Large Language Models. Unlike previous research, this paper introduces the idea of using SVD-based low-rank matrices to inject knowledge learned from previous tasks into new tasks, thereby enhancing plasticity during the CL process. Additionally, it suggests a method for learning in an orthogonal subspace to prevent forgetting of previous tasks. Experimental results on various text classification datasets demonstrate that the proposed algorithm outperforms existing algorithms. Furthermore, various ablation studies and analyses experimentally show the role and effectiveness of each component of the proposed algorithm.
strength_weakness: strength_weakness: LB-CL is motivated by O-LoRA and proposes a similar idea (Section 2.2: Training in Orthogonal Subspaces). Although the proposed algorithm achieves state-of-the-art performance in various experiments, the performance improvement over the previous SOTA algorithm, O-LoRA, is not very substantial (e.g., an average improvement of 1.3% on standard CL benchmarks and 0.4% on a large number of tasks). While LB-CL introduces additional ideas for knowledge transfer, this might increase the overall cost of the algorithm. Therefore, to verify the superiority of LB-CL over O-LoRA, it is necessary to compare the computation costs of these algorithms (e.g., training time, number of training parameters, or FLOPS).

Additionally, I have identified the following corrections that need to be made while reading the paper:

1. Line 192: It seems that Eq.5 should be corrected to Eq.7.
2. Line 224: It would be beneficial to include a brief explanation of O-LoRA under the 'Baselines' section.
questions: questions: I find LB-CL proposed in this paper is somewhat novel. However, due to the similarity with the existing algorithm in certain aspects and the lack of significant performance improvement compared to the existing algorithm, I believe additional experiments and analysis are needed to properly validate the merits of the proposed idea. Please check the Weakness Section for more details.
limitations: limitations: There is no potential negative societal impact of this paper.
paper_summary: paper_summary: This paper introduces LB-CL, a continual learning algorithm designed to tackle the issues of catastrophic forgetting and forward knowledge transfer. The approach integrates orthogonal low-rank SVD decomposition and sensitivity-based parameter initialization. The orthogonal subspace learning component addresses catastrophic forgetting by ensuring the SVD decompositions of different tasks remain orthogonal. The sensitivity-based initialization enhances forward transfer by optimizing parameter initialization weights. Comprehensive experiments across multiple representative datasets demonstrate that LB-CL outperforms many existing methods.
strength_weakness: strength_weakness: 1. The paper does not sufficiently clarify the advantages and necessity of the proposed SVD decomposition over existing methods like LoRA. 
2. The paper does not provide experiments to demonstrate the sensitivity of the method to the order of tasks.
3. The paper does not address whether the classification head is distinguished by task-ID during inference.
4. The paper's experimental section lacks a thorough comparison with other common initialization methods, which makes it difficult to assess the superiority of the proposed initialization strategy.
5. The computational complexity and scalability of maintaining orthogonal subspaces for a growing number of tasks are not addressed. As the number of tasks increases, the computational overhead may become prohibitive.
questions: questions: Could you provide more details on why SVD decomposition is preferred over LoRA and what specific advantages it offers?
Have you tested the sensitivity of LB-CL to the order in which tasks are presented? How might task order impact the performance of the method?
How does the method perform when tasks are highly similar and distinguishing features overlap? Does the orthogonality constraint still effectively prevent forgetting in such cases?
During inference, does the model require the task-ID to be known? If so, how does this affect the practicality and flexibility of LB-CL?
The choice of rank in the low-rank SVD decomposition significantly impacts performance. How is this rank chosen optimally and how sensitive is the method to this hyperparameter?
limitations: limitations: *
paper_summary: paper_summary: The paper presents a new parameter-efficient method for continual learning in LLMs. The method focuses on two aspects of continual learning: (1) Catastrophic forgetting and (2) Forward transfer.  To address catastrophic forgetting the gradient update of the current task is performed on the orthogonal space of previous tasks. To address forward transfer, they propose to initialize the new SVD parameters of the current task using a combination of previous tasks via learned coefficients.  The method is tested on two benchmarks with three task orders. An ablation is provided for each proposed component. The method is compared to another parameter-efficient method for CL, variations of the proposed method, some basic baselines (MTL, standard finetuning), and few previous CL methods on standard architectures.
strength_weakness: strength_weakness: - Despite that forgetting is one of the main aspects addressed by the paper, it is not evaluated and no forgetting metric is reported. 
- Some very related works are missing [1,2].
- Some design choices/observations are not clear, see the questions section. 
- Minor: Paper needs proofread. Suggestions for a few improvements are below:
    - L149: “since the the” 
    - Section 3.1: A description of O-LoLA is missing in the baselines section
    - L 84 description of triplet could be provided here for clarity. 
    - 3 rows (out of 4) from Table 3 are already presented in Table 2 
    - Since the parameters of previous tasks are kept frozen, I am not sure whether forgetting is the right terminology or one should use interference (since forgetting is avoided by design somehow) 

[1] Yu, Jiazuo, et al. "Boosting continual learning of vision-language models via mixture-of-experts adapters." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Liang, Yan-Shuo, and Wu-Jun Li. "InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: - Is the training performed in two stages? One for determining sensitivity parameters and one for training the task?
- Do you assume that the task identity is not available at the test time? If yes, is the input image at the test time forwarded to the sum of the SVD of all tasks?
- In Table 3, it is not clear to me why NLNB -CL is better than L-CL and B-CL, any thoughts on that?
- Do the same observations in Table 3 hold for the other benchmark?
- Figure 3 shows that excluding Sigma leads to better performance, I am wondering what could be the reasons behind this and why you don’t consider doing this in your proposed approach. 
- How is your method compared with the mentioned works above?
limitations: limitations: Limitations and societal impact are not discussed.

Comments: title: Please check authors' rebuttal
Official comment by Reviewer qwD9
Official Comment by Reviewer aBm6
comment: Dear reviewer, 

Please check authors' rebuttal they have made the effort to respond to your concerns.

AC
Thank you for considering our responses. We are happy some of our answers were satisfying, and we promise that we will gladly incorporate the valuable and constructive comments raised by all reviewers in the revised version. We are also pleased to offer further clarification to make sure your remaining concerns are resolved.
Thank you for checking our responses. We’re glad that most of your concerns have been addressed. We are also more than happy to make more clarifications that could address any remaining concerns and potentially increase the score.
Thank you for reviewing our responses. We appreciate your feedback and are glad that we could address almost all of your concerns. If you have any remaining questions or need further clarification, we are more than willing to provide additional clarification.
Thank you for your response. Some of my concerns have been addressed. I encourage the authors to consider the points raised by all reviewers in the revised version. I am leaning towards keeping my original score.
Thank you for the positive response. Most of my doubts have been addressed. Therefore, I will keep my original rating.
Thank you for providing the author response. Almost all of my concerns have been addressed, so I keep my initial score 'weak accept'.