Title: Continual Knowledge Adaptation for Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning; Markov Decision Process
Abstract: Reinforcement Learning enables agents to learn optimal behaviors through interactions with environments. However, real-world environments are typically non-stationary, requiring agents to continuously adapt to new tasks and changing conditions. Although Continual Reinforcement Learning facilitates learning across multiple tasks, existing methods often suffer from catastrophic forgetting and inefficient knowledge utilization. To address these challenges, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge. Specifically, we introduce a Continual Knowledge Adaptation strategy, which involves maintaining a task-specific knowledge vector pool and dynamically using historical knowledge to adapt the agent to new tasks. This process mitigates catastrophic forgetting and enables efficient knowledge transfer across tasks by preserving and adapting critical model parameters. Additionally, we propose an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors to address scalability challenges, reducing memory requirements while ensuring the retention of essential knowledge. Experiments on three benchmarks demonstrate that the proposed CKA-RL outperforms state-of-the-art methods, achieving an improvement of 4.20% in overall performance and 8.02% in forward transfer. The source code is available at https://github.com/Fhujinwu/CKA-RL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript proposes a new continual reinforcement learning method, Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which aims to address catastrophic forgetting and inefficient knowledge transfer in non-stationary environments. CKA-RL maintains a pool of task-specific knowledge vectors and dynamically adapts historical knowledge to new tasks. Furthermore, an adaptive knowledge merging mechanism is introduced to enhance scalability by combining similar knowledge vectors, thereby reducing memory requirements and redundant knowledge. Experimental results on three benchmarks demonstrate superior performance compared to other continual reinforcement learning methods, particularly in terms of overall performance and forward transfer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The manuscript is well-structured and clearly written, making the methodology and results easy to understand.\n    \n2. The proposed approach effectively adapts model merging techniques from continual learning to the continual reinforcement learning domain, offering a simple yet impactful solution.\n    \n3. CKA-RL achieves a balanced trade-off among plasticity, stability, and scalability, which is potentially influential for the continual reinforcement learning community.\n    \n4. Extensive experiments are conducted across multiple benchmarks, with comprehensive comparisons to recent state-of-the-art methods, demonstrating the effectiveness of the proposed approach from multiple perspectives.\n    \n5. The supplementary material provides detailed descriptions of environments, tasks, and implementation details, supporting reproducibility and clarity.\n    \n\nWeaknesses\n\n1. The experimental evaluation lacks statistical rigor: error bars or other statistical significance indicators are absent, and experiments appear to be conducted with a single random seed (which is not specified in the main text). Given the inherent instability of reinforcement learning, this significantly undermines the credibility of the reported results. Additionally, the absence of code availability limits reproducibility.\n    \n2. There is insufficient discussion of related work regarding model merging (task vector) in continual learning, despite its close relevance to the proposed method.\n    \n3. The evaluation of model plasticity (in Section 5.2) is inadequate. Compared methods in the literature typically use more tasks and longer experimental horizons, making it difficult to draw meaningful comparisons about plasticity from the current experimental setup.\n    \n4. There is a conceptual inconsistency in the discussion of initial performance (in Section 5.4): the manuscript claims that superior initial performance is due to effective use of historical knowledge vectors, but in the second task, no such historical vectors are available.\n    \n5. The scalability analysis (in Section 5.4) is limited. The experiments only involve up to 10 tasks, and the knowledge vector pool is fixed at 5. In contrast, related work such as CompoNet evaluates scalability with up to 300 tasks. The performance and memory efficiency of CKA-RL under longer task sequences remain unclear.\n\nQUESTIONS:\n1. Can the authors provide experimental results with error bars or other statistical significance measures (e.g., multiple seeds, standard deviations)? This is crucial for assessing the reliability of the reported improvements and the overall evaluation of the work.\n    \n2. How does the proposed method differ from existing model merging approaches in continual learning? Is the main contribution the adaptation of these techniques to continual reinforcement learning, or are there substantive methodological novelties?\n    \n3. How does CKA-RL scale with a substantially larger number of tasks (scalability)? The current experiments do not demonstrate performance or memory efficiency in long task sequences.\n    \n4. How does the size of the knowledge vector pool affect performance, especially with increasing task diversity and quantity? Are there guidelines or heuristics for determining the optimal pool size in such scenarios?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenges of catastrophic forgetting and inefficient knowledge utilization in reinforcement learning. The authors propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), a framework designed to facilitate the accumulation and effective reuse of historical knowledge across tasks. Specifically, CKA-RL maintains a task-specific knowledge vector pool and leverages these historical knowledge representations to better adapt to new tasks. To address scalability concerns, the authors introduce an Adaptive Knowledge Merging mechanism, which combines similar knowledge vectors to reduce redundancy. Experimental results across three benchmarks show that CKA-RL outperforms state-of-the-art methods, achieving a 4.20% improvement in overall performance and an 8.02% gain in forward transfer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper is clearly written and easy to follow, making the proposed method accessible to a broad audience.\n\n2.\tThe experimental results demonstrate the effectiveness of the proposed approach, showing clear performance advantages over existing methods.\n\nWeaknesses:\n\n1.\tDue to the use of a softmax function for computing the weights $\\beta$, it is not possible to enforce $\\alpha_1 = 1$. As a result, parameter conflicts remain unavoidable under the formulation presented in Equation (3).\n\nQUESTIONS:\n1.\tWhy the performance of CompoNet in this paper appear significantly lower than reported in its original publication?\n\n2.\tAccording to Equation (3), the task-specific parameter vectors are expected to be of the same dimensionality as the base model parameters $\\theta_{\\text{base}}$. Given this, is the overall parameters of the model the form $y = \\theta_{\\text{base}} x$ until the size of the knowledge pool in Figure 3?\n\n3.\tSince mitigating catastrophic forgetting is one of the central motivations of this work, it is surprising that the standard forgetting metric is not reported in Table 1.\n\n4.\tThe proposed method uses a simple weighted sum of historical task vectors to mitigate catastrophic forgetting and enable knowledge transfer. However, this raises several concerns: \n\n  a)\tFor catastrophic forgetting, the weighted sum of history vectors may not be robust to conflicting knowledge across tasks. For instance, if a previously learned task vector $v_i$ is incompatible with another vector $v_j$ (due to opposite objectives or gradients), the resulting combined vector could destructively interfere with prior knowledge. Since the weights are computed via a softmax over hyperparameters, it is impossible to assign exclusive weight (e.g., $\\alpha_1 = 1$) to a single vector, making such interference likely. One potential solution is to condition on the task ID and use only the corresponding task vector during adaptation; however, the adaptive knowledge merging mechanism complicates this, as it dynamically modifies and merges task vectors, potentially discarding task-specific identities.\n\n  b)\tFor knowledge transfer, the claim in lines 11–13, stating that the method “preserves and adapts critic model parameters,” seems overly strong. Since the model combines vectors from previous tasks via a weighted sum, it does not strictly preserve previous critic parameters unless the hyperparameter for the current task is exactly zero—which the softmax-based formulation prevents. In practice, this weighted combination is more consistent with model merging, a technique known to work well only when the tasks are highly similar. Therefore, I am inclined to interpret the benefit of this approach primarily as providing a stronger initialization for task $k$, rather than truly preserving and adapting previous critic parameters.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address catastrophic forgetting and inefficient knowledge utilization in continual reinforcement learning (RL). The authors propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), a continual RL algorithm. CKA-RL maintains a pool of task-specific knowledge (policy’s model parameters) from previously learned tasks and leverages this knowledge to adapt an agent to a new task. To reduce the memory accumulated by historical knowledge, CKA-RL also includes an adaptive knowledge merging mechanism that combines similar knowledge vectors in the pool. \n\nCKA-RL is compared with other methods across Meta-World, SpaceInvaders, and Freeway, and outperforms them in terms of average performance and forward transfer. CKA-RL is also more efficient in terms of memory usage and inference time when evaluated on SpaceInvaders.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper is written with a clear flow, and one can easily catch the high-level idea of the proposed method. \n2. The idea of including a knowledge merging mechanism is crucial, in my opinion, for scalable continual RL algorithms. \n3. The experimental results show that CKA-RL consistently outperforms other evaluated methods in terms of average performance and forward transfer. Some ablation studies are also provided to help understand the effectiveness of the two components in CKA-RL. Especially, I found Figure 11 very interesting, as it demonstrates how distinct each knowledge vector is.\n\n# Weaknesses\n1. The motivation and design of CKA-RL are not sufficiently elaborated. \n- Knowledge vectors: In Continual RL, various forms of knowledge can be transferred across tasks, such as policies, feature representations, reward functions, and reasoning models. In this paper, the knowledge vectors are specifically designed to be the “residual model parameters” of a policy. It would be easier to follow if the authors could explain why they chose this specific design.  \n- The normalized adaptation factors, alphas and betas: After reading the paper, I still have a hard time understanding how these parameters are learned. Specifically, how are betas updated?  \n- The similarity between knowledge vectors: The paper proposed that the normalized cosine similarity of two vectors of residual policy parameters can represent the similarity between two policies. However, I believe there is a difference between the representation and functional similarity of neural networks. Thus, even if the cosine similarity is close to 1, it does not guarantee that the two policies perform similarly. \n- Following the previous point, simply merging two knowledge vectors can result in new knowledge with very different behaviors. \n2. Ablation studies can be more comprehensive. \n- Effects of knowledge pool size: This analysis is conducted only on Freeway, not on other environments. In addition, the size evaluated only ranges between 3 and 7, which is relatively small and may not necessarily generalize to different scales of pool size. \n- Reward curve comparison in Figure 4: Which environment is this analysis conducted in? \n- Knowledge Retention of CKA-RL: Without statistical significance results, Table 3 does not necessarily indicate that CKA-RL outperforms other baselines in mitigating catastrophic forgetting, as the performance difference is not substantial. \n- Memory efficiency and inference efficiency: These analyses are only performed on SpaceInvaders, and the results for other environments are not provided. \n3. The statistical significance results are not presented in the paper. Specifically, the number of random seeds used for training is not specified, and most figures and tables do not provide the standard deviation of the results. \n4. In the Introduction, the paper mentions that CKA-RL aims to address cross-task conflict; however, after reading the paper, I do not find that CKA-RL can address conflicts between tasks.\n\nQUESTIONS:\n1. Since the model parameters of the first task will be set as the base parameter, does the choice of this first task matter?  In other words, does the sequence of tasks affect the results? \n2. (Section 4.2) “For flexibility, we define the null knowledge as…”  What does “flexibility” mean here, and why does defining the null knowledge improve the flexibility? \n3. How are betas updated? \n4. (Section 4.2) “By setting alpha_1 = 1, the model can disregard previously learned knowledge when it is not beneficial for learning new tasks.”  Why? \n5. Do you store the betas learned in different tasks? If not, how do you restore the policies for the tasks that have already been learned? \n6. (Table 2) Why does the performance, not only the memory efficiency, improve after adding the adaptive knowledge merging mechanism? \n7. (Section 5.4) Why does CKA-RL lead to superior initial performance and rapid convergence? \n8. How many random seeds did you run for each experiment?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), a new method for continual reinforcement learning (CRL) designed to mitigate catastrophic forgetting and improve knowledge transfer while maintaining scalability. The core idea is to represent task-specific knowledge as \"knowledge vectors.\" To address the challenge of ever-growing memory costs, the authors propose an \"Adaptive Knowledge Merging\" mechanism, which identifies and merges the most similar pairs of knowledge vectors in a fixed-size pool, thus keeping the memory footprint bounded.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Important problem and the algorithm generally makes sense.\n2. The analysis in Figure 5 and Appendix D clearly demonstrates the superior memory and inference efficiency of CKA-RL.\n\nWeaknesses:\n1. Significance of Empirical Results is Unclear: My biggest concern is the lack of statistical rigor in the experimental validation. The performance improvements reported in the main results (Table 1) appear to be quite marginal (e.g., ~1-4% in many cases). The authors state in the checklist that results are from a single run with a fixed seed and do not report standard deviations. For an empirical RL paper, this is a critical omission. Without error bars over multiple runs, it is impossible to determine if these small advantages are statistically significant or simply the result of run-to-run variance. \n\n2. Moderate Methodological Novelty: The core idea of decomposing model updates into task-specific vectors is heavily inspired by prior work in model editing and task arithmetic. While its application to CRL is new, the fundamental mechanism is not\n\nQUESTIONS:\nFor related work, could the authors also discuss more recent crl specific papers like 1. Prediction and Control in Continual Reinforcement Learning. Anand et al. 2. Knowledge Retention for Continual Model-Based Reinforcement Learning. Sun et al. etc. ?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}