Title: Turning the Tables: Enabling Backward Transfer via Causal-Aware LoRA in Continual Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Continual Learning; parameter-efficient fine-tuning; Causal Learning
Abstract: Current parameter-efficient fine-tuning (PEFT) methods have shown superior performance in continual learning. However, most existing PEFT-based methods focus on mitigating catastrophic forgetting by limiting modifications to the old task model caused by new tasks. This hinders backward knowledge transfer, as when new tasks have a strong positive correlation with old tasks, appropriately training on new tasks can transfer beneficial knowledge to old tasks. Critically, achieving backward knowledge transfer faces two fundamental challenges: (1) some parameters may be ineffective on task performance, which constrains the task solution space and model capacity; (2) since old task data are inaccessible, modeling task correlation via shared data is infeasible. To address these challenges, we propose CaLoRA, a novel \textbf{c}ausal-\textbf{a}ware \textbf{lo}w-\textbf{r}ank \textbf{a}daptation framework that is the first PEFT-based continual learning work with backward knowledge transfer. Specifically, we first propose \textbf{p}ar\textbf{a}meter-level \textbf{c}ounterfactual \textbf{a}ttribution (PaCA) that estimates the causal effect of LoRA parameters via counterfactual reasoning, identifying effective parameters from a causal view. Second, we propose \textbf{c}ross-t\textbf{a}sk \textbf{g}radient \textbf{a}daptation (CaGA) to quantify task correlation by gradient projection and evaluate task affinity based on gradient similarity. By incorporating causal effect, task correlation, and affinity, CaGA adaptively adjusts task gradients, facilitating backward knowledge transfer without relying on data replay. Extensive experiments across multiple benchmarks and continual learning settings show that CaLoRA outperforms state-of-the-art methods. In particular, CaLoRA better mitigates catastrophic forgetting by enabling positive backward knowledge transfer.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes CaLoRA, which enables positive backward transfer in the context of adapter-based continual learning. The method combines parameter-level causal attribution (PaCA), which identifies LoRA parameters that causally contribute to task performance, and cross-task gradient adaptation (CaGA) with gradient surgery to adjust new-task gradients based on their alignment with prior tasks. This selective update mechanism allows CaLoRA to improve both new and previous tasks without explicit rehearsal. Experiments on NLP and vision benchmarks demonstrate that CaLoRA consistently outperforms recent PEFT baselines in terms of average accuracy and reduced forgetting. The paper also provides an analysis across multiple model scales and presents comprehensive ablation studies that validate the effectiveness of each component.\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths\n> \n- The paper presents a crucial direction in continual learning by focusing on achieving positive backward transfer rather than merely preventing forgetting, using pre-trained models and adapter-based model expansion.\n- The motivation and problem setting are clearly stated, and the flow from the introduction to the method and experiments is easy to follow.\n- The comparisons cover a wide range of adapter-based continual learning methods in both NLP and vision tasks, demonstrating the broad applicability of the approach.\n- The experiments include recent strong baselines, and the performance improvements are clearly demonstrated. The ablation studies are complete and well-organized, and their results support effectiveness of each components(PaCA and CaGA) of CaLoRA.\n- The figures and tables are clear and help readers understand the results without confusion.\n\n---\n\n> Weaknesses\n> \n\n**Overall**\n\n- The paper does not report the explicit computational overhead introduced by PaCA and CaGA, which raises concerns about scalability as the number of tasks or model size increases.\n- As noted in the introduction and previous work on positive backward transfer[1,2], its effectiveness depends heavily on inter-task correlation. However, the paper lacks a detailed analysis of how task correlation influences performance.\n\n**Experiments**\n\n- The experiments are conducted with only three random seeds, which may not be sufficient to assess statistical stability, especially on complex benchmarks.\n\n> [1] Ke, Zixuan, Bing Liu, and Xingchang Huang. \"Continual learning of a mixed sequence of similar and dissimilar tasks.\" Advances in neural information processing systems 33 (2020): 18493-18504.\n> \n> [2] Lin, Sen, et al. \"Beyond not-forgetting: Continual learning with backward knowledge transfer.\" Advances in Neural Information Processing Systems 35 (2022): 16165-16177.\n>\n\nQUESTIONS:\n- Since the framework accumulates adapter parameters over time, has there been any consideration of compressing or storing them more efficiently during the intermediate stages of continual learning?\n- I’m curious about how much additional preprocessing time CaLoRA requires compared to other baselines. Would you consider this overhead negligible relative to the overall training time?\n- Is there any analysis or result that explores in more detail how inter-task correlation affects CaLoRA’s performance?\n- Could the authors elaborate on whether specific characteristics of the data, such as incomplete or partially informative examples, might have contributed to the observed transfer effects? A discussion of such factors could help clarify when and why positive backward transfer occurs in the presented settings.\n- This is a minor point, but would it be possible to include qualitative examples from NLP tasks where samples that initially failed were later corrected through positive backward transfer?\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a learning framework, CaLoRA (Causality-aware Low-Rank Adaptation), aimed at mitigating catastrophic forgetting in sequential cross-task learning. By introducing parameter-level counterfactual attribution (PaCA), the method identifies causally effective parameters. Additionally, a cross-task gradient adaptation (CaGA) mechanism is employed to integrate task correlation and affinity, enabling selective gradient updates that favor previous tasks and facilitate positive backward knowledge transfer. Extensive experiments demonstrate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This work targets the issue of catastrophic forgetting in continual learning for large model fine-tuning. It aims for improving parameter efficiency and enabling backward knowledge transfer are essential for scalable and sustainable deployment of pretrained models.\n2. The proposed approach introduces parameter-level counterfactual attribution (PaCA) to identify causally effective low-rank parameters and cross-task gradient adaptation (CaGA) to incorporate task correlation and affinity for selective gradient updates.\n3. Extensive experiments on both NLP and vision benchmarks, as well as ablation studies, demonstrate the effectiveness of the approach.\n\nWeaknesses:\n1. The paper characterizes inter-task correlation and transfer direction through gradient projection (Task Correlation) and gradient similarity (Task Affinity). However, this approach appears to rely on a linear assumption in the gradient space, which may limit its ability to capture more complex nonlinear relationships between tasks.\n2. The authors introduce a parameter-level counterfactual attribution method (PaCA). However, beyond the ablation studies, the experimental section lacks an in-depth investigation into the significant differences in causally effective parameters across different tasks. This limits the understanding of PaCA’s role in capturing cross-task causal relationships.\n3. In Section 3.3, the authors claim that task order affects performance. However, the experiments in Tables 1 and 2 only present results for two task sequences. There is a lack of deeper analysis on how gradient updates specifically influence the dynamic changes of task correlation and affinity. Moreover, additional experiments with varied task orders are not provided to substantiate this claim.\n\nQUESTIONS:\n1. The authors compute task correlation and affinity at the gradient level but do not include visualizations illustrating the relationships between tasks in the experiments.\n2. Are the baseline results reported in the experiments based on the original papers’ reports or reproduced by the authors?\n3. In the ablation studies, how exactly are the effects of task correlation and task affinity removed? Are these values set to 1 or other?\n4. Please refer to the Weaknesses section for other questions.\n\nLIMITATIONS:\nplease see weakness and questions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a continual learning method built on Parameter-Efficient Fine-Tuning (PEFT), focusing on enabling backward knowledge transfer—i.e., improving the performance on previous tasks when learning new tasks.\nThe method consists of two components:\nParameter-level Counterfactual Attribution (PaCA): Estimates the causal effect of each LoRA parameter via counterfactual reasoning to identify and prioritize parameters that truly improve task performance, avoiding ineffective adaptation.\nCross-task Gradient Adaptation (CaGA): Models task correlation by measuring gradient similarity through projection, enabling affinity-aware parameter updates to promote transfer between related tasks while reducing interference.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nEvaluation on both NLP and image classification tasks demonstrates the adaptability of the proposed method, whereas existing methods typically focus on either vision tasks or NLP tasks exclusively.\n\nThe presentation is excellent. The paper is well-organized, well-motivated, and clearly written.\n\nWeaknesses:\n\nThe effectiveness of backward transfer is not convincingly demonstrated; there is insufficient evidence. This is especially important given that pre-trained models typically contain richer general knowledge than what can be learned from prior tasks.\n\nThe proposed techniques appear to share similarities with existing methods (e.g., PaCA and influence functions, CaGA and EWC), but these connections are not thoroughly discussed.\n\nSee additional comments in the Questions section.\n\nQUESTIONS:\nCould you illustrate a scenario where a new task can transfer beneficial knowledge to old tasks?\n\nCould the authors clarify the relationship between the proposed causal effect estimation and influence functions, which also aim to quantify the impact of individual parameters?\n\nCan the proposed approach generalize to randomly initialized models?\n\nCould the authors provide an ablation study that quantitatively verifies the backward transfer effect—specifically, showing how the with-in performance on an earlier task (e.g., Task 1) changes as additional tasks are learned? \n\nThe paper does not discuss another important class of PEFT-based continual learning methods, namely MoE-based approaches. \nParticularly, this approach could eliminate the catastrophic forgetting, and has flexibility to learn any number of new tasks.\n\nLIMITATIONS:\nSee Questions", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes CaLoRA, a causal-aware low-rank adaptation method for parameter-efficient continual learning that, unlike prior work, explicitly aims to enable backward knowledge transfer from new tasks to old ones. CaLoRA combines parameter-level counterfactual attribution (PaCA) to estimate causal importance of parameters and cross-task gradient adaptation (CaGA) to assess task correlation and affinity. These components adapt gradients during training to selectively update parameters that benefit both new and old tasks. Experiments on NLP and vision benchmarks show that CaLoRA outperforms several strong baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**S1.** This paper tackles the clear motivated problem of backward knowledge transfer in PEFT. \n\n**S2.** The authors use a diverse set of benchmarks from both NLP and Computer Vision, employ modern and large-scale pre-trained models (T5, LLaMA-2, ViT), and compare against a comprehensive list of recent and relevant baselines. The testing across different model scales, task orders, and task lengths demonstrates the method's robustness.\n\n**S3.** CaLoRA consistently outperforms all baselines across all reported metrics. Crucially, it shows substantial improvements in Backward Transfer (BWT), often turning the negative BWT of other methods into a positive value, which directly validates the paper's central hypothesis and contribution.\n\n**W1.** As acknowledged by the authors, CaLoRA requires storing the gradients for all previous tasks, which could lead to significant memory overhead in scenarios with a very large number of sequential tasks. Furthermore, the per-step computation of SVD, gradient projections, and similarities for all previous tasks appears computationally intensive, and a more detailed analysis of this overhead would be beneficial.\n\n**W2.** The method operates in a standard continual learning setting where task boundaries are clearly defined. Its applicability to more complex scenarios like online continual learning, where task boundaries are blurred, is a limitation, as noted by the authors.\n\nQUESTIONS:\n**Q1.** In Algorithm 1, the causal effect $E_t$ is estimated once at the beginning of training for task t. Have you considered the possibility of updating this causal effect estimation periodically during the training of task t? One might hypothesize that the importance of certain parameters could evolve as learning progresses on the new task.\n\n**Q2.** Can you provide quantitative evidence that gradient similarity is a reliable indicator of positive transfer?\n\n**Q3.** How does the method perform if old task gradients are noisy or approximate?\n\nLIMITATIONS:\nThis paper addressed limitations well, but not addressed potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}