Continual Knowledge Adaptation for Reinforcement Learning
Authors: Jinwu Hu
ZiHao Lian
Zhiquan Wen
ChenghaoLi
Guohao Chen
Xutao Wen
Bin Xiao
Mingkui Tan
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning; Markov Decision Process
Primary area: reinforcement_learning
TL;DR: We propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge.

Abstract: Reinforcement Learning enables agents to learn optimal behaviors through interactions with environments. However, real-world environments are typically non-stationary, requiring agents to continuously adapt to new tasks and changing conditions. Although Continual Reinforcement Learning facilitates learning across multiple tasks, existing methods often suffer from catastrophic forgetting and inefficient knowledge utilization. To address these challenges, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge. Specifically, we introduce a Continual Knowledge Adaptation strategy, which involves maintaining a task-specific knowledge vector pool and dynamically using historical knowledge to adapt the agent to new tasks. This process mitigates catastrophic forgetting and enables efficient knowledge transfer across tasks by preserving and adapting critical model parameters. Additionally, we propose an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors to address scalability challenges, reducing memory requirements while ensuring the retention of essential knowledge. Experiments on three benchmarks demonstrate that the proposed CKA-RL outperforms state-of-the-art methods, achieving an improvement of 4.20% in overall performance and 8.02% in forward transfer. The source code is available at https://github.com/Fhujinwu/CKA-RL.

Continual Knowledge Adaptation for Reinforcement Learning
Jinwu Hu Zihao Lian' Zhiquan Wen'* Chenghao Li Guohao Chen Xutao Wen' Bin Xiao Mingkui Tan 14
South China University of Technology. "Pazhou I Laboratory, Chonggiing University of Posts and Telecommunications
Key Laboratory Big Data Intelligent Robot, Ministry Education,
Abstract
Reinforcement Learning enables agents to learn optimal behaviors through in- teractions with environments. However, real-world environments typically non-stationary, requiring agents to continuously adapt to new tasks and changing conditions. Although Continual Reinforcement Learning facilitates learning across multiple tasks, existing methods often suffer from catastrophic forgetting and inef- ficient knowledge utilization. To address these challenges, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge. Specifically, we introduce Continual Knowledge Adaptation strategy, which involves maintaining task-specific knowledge vector pool and dynamically using historical knowledge to adapt the agent to new tasks. This process mitigates catastrophic forgetting and enables efficient knowledge transfer across tasks by preserving and adapting critical model parameters. Additionally, we propose an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors address scalability chal- lenges, reducing memory requirements while ensuring the retention of essential knowledge. Experiments on three benchmarks demonstrate that the proposed CKA- RL outperforms state-off of----f-ri off-the-rrt methods, achieving an improvement of 4.20% in overall performance and 8.02% in forward transfer. The source code available https: //gi thub. com/ Fhuj inwu/ CKA-RL
Introduction
Reinforcement Learning (RL) has emerged as a powerful paradigm in machine learning, enabling agents to learn optimal behaviors through interactions with dynamic environments 24
45 RL
has achieved significant success in fields such as robotic control [23] 335. embodied intelligence 15.. and natural language processing [17
10 [8] However, traditional RL usually assumes static environment where tasks and data distributions remain fixed, intending to solve a single, well-defined problem 332.. In contrast, real-world environments are typically non-stationary, requiring agents to continuously adapt to new tasks and evolving conditions. In light of this, Continual Reinforcement Learning (CRL) (25) has been introduced enable agents adapt and maintain performance across multiple tasks, facilitating more robust decision-makinn in dynamic environments [4] 2 50].
Unfortunately, CRL still faces several challenges, which follows. /) Cross-task conflict: In continual learning, tasks may share certain structures or knowledge while also having incompatible goals or constraints. This interplay complicates the direct reuse previously learned information [27].
"Equal contribution. Email: fhujinwu@ @orruu@@mmaaicoom gmail.com, lianzihaolzh@ caaiioolll@mmmi.cooomm gmail.com. sewenzhiquan gmail.com *Coresponding author. Email: mingkuitan @xcut.edu.cn, xiaobin @cqupt.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2) Catastrophic forgetting: When learning new tasks, the agent may overwrite or distort previously acquired knowledge, leading loss of performance on earlier tasks [32 55]
Recently, several attempts have been proposed to enable agents to continuously learn across multiple tasks while maintaining enhancing their performance on previously acquired tasks (60 19] The
existing methods can be broadly categorized into four main groups [50]. The regularization-bect baasdd
methods (20] 37 introduce regularization terms into the learning objective that penalize large updates
to important parameters for previously learned tasks. The tittarrsl-baseer methods [9] [5) [38]
mitigate forgetting by storing previous experiences in memory and periodically replaying them during the learning of new tasks. By leveraging past experiences, these methods help reduce short- term biases and improve task performance across the sequence. The architecture-bar
methods 49 32 S 39] focus on learning shared structure, such as modularity or composition, to facilitate continual learning. They reuse parts of previous solutions by forming abstract concepts or skills. The meta-learning based methods 40 12 30 8 improve the learning efficiency of agents by utilizing past successes and failures to refine their optimization processes. This creates an inductive bias that enhances sample efficiency and adaptability in acquiring new behaviors.
Although there has been significant progress in existing methods, they still face several limitations as follows. Firstly, existing methods, such as PackNet [33], often fail to effectively address task dependencies and conflicts, which makes it challenging to transfer knowledge across tasks with- out interference or degradation in performance. Secondly, most methods still exhibit substantial performance degradation on previously learned tasks upon acquisition of new ones. This issue is particularly pronounced in methods that rely on regularization, as they may struggle to efficiently retain knowledge from earlier tasks as the task sequence lengthens. Lastly, many methods face scalability issues, especially as the number of tasks grows significantly, leading to increased memory and computational costs, such as CompoNet [32].
To address these limitations, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowl- edge, thereby accelerating learning in new tasks and explicitly reducing performance degradation on previous tasks. Specifically, we assume that the agent acquires a unique knowledge vector for each task during continual learning. Based on this, we propose a Continual Knowledge Adaptation strategy, which involves maintaining a task- speciffi knowledge vector pool and dynamically using historical knowledge to adapt the agent to new task. This method mitigates catastrophic forgetting and facilitates the efficient transfer of knowledge across tasks by preserving and adapting crucial model parameters. Furthermore, we introduce an Adaptive Knowledge Merging mechanism that clusters and consolidates similar knowledge vectors to address scalability issues, reducing memory requirements while ensuring the retention of essential information.
Main novelty and contributions. 1) called CKA-RL. It enables agents to reuse knowledge from previously learned tasks, mitigating catastrophic forgetting, and leveraging this historical knowledge to enhance the learning efficiency. 2) We propose Continual Knowledge Adaptation strategy, which dynamically adapts historical knowledge to a new task. To reduce storage requirements, we introduce an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors, addressing scalability issues. 3) Experiments demonstrate that the proposed method outperforms mate-of-the-eee methods on three benchmarks, achieving an improvement of 4.20% in performance and 8.02% in forward transfer.
We propose a novel continual reinforcement learning method,
Related Work
Continual Reinforcement Learning (CRL) aims to enable agents to continuously learn and optimize strategies in dynamic environments, improving their ability to adapt to environmental changes and achieve goals more efficiently. The existing methods can be broadly categorized into four main types [50]: regularization-ber based methods, rehearsal- -based methods, architecture-baser methods, and meta-learning-bant based methods, each of which is detailed as:
Regularization-basee Methods. These methods [7] add regularization terms to the training process to balance learning new and old tasks without storing models of old tasks. ITER 20 periodically distills the current policy and value function into newly initialized network during training and imitates the teacher network using a linear combination of loss terms to enhance model generalization. TRAC (37] adaptively adjusts the regularization strength based online convex

---

optimization theory to prevent excessive weight drift, thereby reducing plasticity loss and enabling rapid adaptation to new distribution changes. Anand et al. [3] propose a method that decomposes the value function into permanent and transient components to address the stability-plastiitty dilemma in continual reinforcement. learning, enabling efficient adaptation and control in dynamic environments.
Rehearsal-based Methods. These methods [9 59 58 [3] utilize experience replay, generation replay, parameter replay, etc., store past experiences and prevent catastrophic forgetting in agents. RECALL [58] focuses on improving plasticity and stability in continuous reinforcement learning through multi-head neural network training, coupled with adaptive normalization and policy distilla- tion techniques. IQ [59] employs a context partitioning strategy based on online clustering, combined with multi-head networks and knowledge distillation technology. to reduce interference between different state distributions. DRAGO [13] leverages synthetic experience replay and exploration- based memory recovery to retain knowledge across tasks, mitigating catastrophic forgetting without requiring the storage of past task data.
These methods 49 32 5 39 concentrate on learning a policy with set of shared parameters to handle all incremental tasks, including parameter allocation, model reorganization, and modular networks. For example, MaskNet [5] employs learnable modulation mask to isolate the parameters of different tasks on a fixed neural network and accelerates learning new tasks by linearly combining masks from previous tasks. CompoNet 332 uses modular architecture with self- compocon policies to enable efficient knowledge transfer and scalable learning in continual reinforcement learning. COMP [4 decomposes complex tasks into multiple subtasks corresponding to different neural modules. These modules can be combined to form a complete policy. REWIRE 46 redefines the connection methods of the neural network and reorders the neurons in each layer to achieve additional plasticity in unstable environments.
Architecture-baser Methods.
Meta-Learning based Methods. These methods 40
assist agents in rapidly adapting to new tasks by simulating the reasoning phase during training. MB-MPO [8] is a model-based method that meta-learns policies robust to ensemble dynamics, reducing reliance on accurate models. MAML [2] is a model-agnostic meta- meta-learning algorithm that explicitly trains models to achieve strong generalization from a small number of gradient updates. ESCP [30] enhances the robustness and responsiveness of context encoding, significantly accelerating adaptation in reinforcement learning involving sudden environmental changes.
12 30
3 Problem Formulation
The standard Reinforcement Learning (RL) problem [47 is modeled as a Markov Decision Process (MDP), with tuple M AS r. where S is state space, A is action space, p S x S + [0. 1] is the state transition probability function, r : S x A - IR is the reward function, and y E [0, 1] the discount factor. At each time step E N, the agent observes the current state st E S and takes an action at ~ T(- where S x A + [0, 1] is the policy. The environment transitions to a new state $1+1 ~ p(11s,, at), and the agent receives a reward r(st, at). The goal of RL is to find an optimal policy x² that maximizes the expected discounted return as follows:
max E.
( az)],
(I)
where the expectation is over trajectories generated by policy π and state transitions The discount factor y controls the trade-off between immediate and future rewards.
Given N tasks, the agent expected maximize the RL objective for each task in T (TI, T2...; TN). Each task т; is associated with a distinct MDP M, = (S,, Ai, Ps. 7t) An RL problem is considered an instance of Continual Reinforcement Learning (CRL) if agents are required to continuously learn. Following the definition presented by Abel et al. [2) we have the following concept: an RL problem defined by a tuple (e, v, A), where is the environment, is the performance function, and A is the set of all agents, the problem is a CRL problem if VY'xA'* f (never reaches) Ag, where AB A is basis such that Ag + (generates) A (in e) and A"= arg maxxea v(A, e). In other words, the best agents continue to search indefinitely over the basis Ag and do not converge
fixed policy. According to CKA-RL, whenever the number of tasks [T] increases, particularly when TN+1 is added, a new knowledge vector uN+1 will be introduced. Therefore, the new task will ensure that À f Ag, and thus continues to learn
3

---

Learnable
Tk
Frozen
Knowledge
Sequence
1
Oasse
2k
Vi P2
V2-1
P/-1 Pa Banage
61
Merge by Eq.
task Te
Vnsaye
task Tg+1
New Knowledge
Historical Knowledge
Adaptation
(n) Continual Knowledge Adaptation
Adaptive Knowledge Merging
Figure 1: An illustration of the CKA-RL. When learning a new task Tk: the agent Ty adapts by using historical knowledge vectors V = (U,,..., tK-1 with [V] S. Kmax, and learning a new task-specific knowledge vector Uk. while the base parameter Ohase remains fixed. After training, the new knowledge vector UR is added to a knowledge vector pool V. To maintain memory efficiency, we merge the most similar pairs of knowledge vectors (Um: vn)) in V into Umerge when V Kmax, thus ensuring essential knowledge is retained while supporting scalable continual learning across sequential tasks.
Continual Knowledge Adaptation for Reinforcement Learning
In this paper, propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge, thereby accelerating learning in new tasks while mitigating catastrophic forgetting. As shown in Figure D our proposed CKA-RL is composed of three key components: 1) Knowledge Vectors: These vectors capture task-specific knowledge that is used to adapt the model Continual Knowledge Adaptation: This component dynamically uses historical knowledge to adapt task, enabling efficient transfer and retention of knowledge (see in Sec. 4.2))
tasks, detailed in Sec. 4.1 2)
the agent to
3) Adaptive Knowledge Merging: This component merges similar knowledge vectors to address scalability issues, reducing memory requirements while ensuring the retention of essential knowledge (see Sec. 4.3).
The pseudo-code of CKA-RL is summarized in Algorithm [
4.1 Knowledge Vectors
In continual reinforcement learning, leveraging knowledge from previous tasks in dynamic envi- ronments is crucial to enhance agent performance in subsequent tasks and mitigate catastrophic forgetting, Therefore, we introduce the concept of knowledge vectors to facilitate continual learning, inspired by model editing techniques [21]. Specifically, the knowledge vector, denoted as d E R" represents the learned parameters that adapt the pre-trained model to specific task Tk, where d denotes the dimensionality of the model parameter. These vectors can be combined linearly with historical knowledge vectors enable cross-task knowledge transfer.
Formally, given the pre-trained model weights O E Rdd, the knowledge vector ey is optimized during task-specific training to capture incremental adaptations. The final task-specific parameter O2 is generated through a combination of the base parameter Orase. historical knowledge matrix V (V) ......, U]. and the current task vector Ug. which is as follows:
0₂ 0 basc
04 Uk:
A linear combination of multiple knowledge vectors enables the model to reuse knowledge from previous tasks [26 6). This highlights the potential of knowledge vectors to facilitate continual knowledge adaptation, effectively tackling key challenges in continual learning.
4.2 Continual Knowledge Adaptation
The knowledge vector stores task-specific knowledge, and the knowledge from historical tasks facilitates the transfer of policies new tasks. Under the Continual RL setting, assuming that the

---

policy т shares a consistent observation space O and action space A, the policy can be represented using a unified architecture parameterized by 0. To achieve cross-task knowledge transfer, we propose Continual Knowledge Adaptation strategy. Specifically, it is first necessary to construct shared base model parameter Onsee. The policy parameter initialized as 02 and trained on the initial task TI, with the optimization objective being to maximize the expected discounted return. The optimized parameter 0₁ is then used as the base parameter Ohase which encompasses general feature representations and serves the foundation for subsequent knowledge adaptation.
The knowledge from previous tasks stored as knowledge vectors in a knowledge vector pool V = {v1.-.. ), where each Ug represents the knowledge vector for task т;. For flexibility, we define the null knowledge as V1 01 Desee 0 and include it in When learning new task tk- the model parameter is generated by adapting the historical knowledge vectors in V with the new task-specific knowledge 1k. which is as follows:
P₂ = buse
0y + ok, where *
11
The Okk [af..... .i.............................. a represents the normalized adaptation factors for task Tke where
Σ az = 1. These factors are derived from learnable parameter f3 = 18k 18f softmax function:
, - through the
exp 3
(4)
-1 (xp
Here, Gy controls the contribution of each historical knowledge vector vy to the new task Tke and oa is the optimizable knowledge vector for the current task. Notably, we initialize oy to 0 to allow the model to gradually learn the task-specific knowledge vector. This prevents large initial adjustments, ensuring stable learning without disrupting previously learned tasks. By setting a1 = 1, the model can disregard previously learned knowledge when it is not beneficial for learning new tasks. This design enables continual adaptation of historical knowledge while preserving task-specific knowledge. During the task training, the base parameter O is fixed to maintain the foundation for knowledge adaptation, and only the adaptation factors 0g (derived from B²) and the current task knowledge vector Ui are optimized. Upon the completion of the training process, the obtained knowledge vector U2 is added the knowledge vector pool V, thereby facilitating efficient knowledge transfer within dynamic environments.
4.3 Adaptive Knowledge Merging
Although the proposed Continual Knowledge Adaptation strategy can accelerate the agent learning process and mitigate catastrophic forgetting by utilizing knowledge vectors from previous tasks, maintaining all vectors becomes impractical the number of stored knowledge vectors increases. Therefore, we propose Adaptive Knowledge Merging to address scalability issues while preserving essential knowledge. Specifically, we merge similar knowledge vectors into compact representations, ensuring that the memory footprint remains manageable. The pairwise similarity between knowledge vectors measured using normalized cosine similarity as follows:
8; Di
Sij | ||
(5)
where Sig E [-1, measures the directional alignment between vectors v, and vy. A value of Si -1 indicates that the two knowledge vectors are in direct conflict. Conversely, value closer to signifies strong alignment, suggesting consistency in the knowledge they encode. This similarity metric effectively captures the functional similarities in how vectors modify the base policy. When the number of knowledge vectors exceeds the maximum capacity Kmax which serves as a hyper- parameter for controlling memory usage, we compute the pairwise similarity matrix S across all stored knowledge vectors. Then, we identify the most similar pair (Vm, u,)) by finding the pair with the highest similarity score, which as follows:
(Um; vn.) = arg max;, Si.s-
(6)
These vectors Um and Vn are merged by averaging:
D Umerge
2 (vm vn))
(7)
5

---

Algorithm The Pipeline of the Proposed CKA-RL
Input: Task sequence T= [T Initial model parameters 0,, Knowledge pool V= (0)
I: Base Policy Learning: O <- arg max Evg 7 (st,
2: for & <- 2.....N N do
3: Task Initialization: vs Br NO0. 1). ax softmax SK-1 ubo uk
4: Policy Construction: 0' <- Orese + 2
5: Policy Optimization: arg max Ery. [2 (81,
6:
Knowledge Preservation: 0s + 0', V.V. (v
7:
if [[] > Kma then
8:
Compute similarity matrix using Eq 5 (Um; f- arg ,
9:
10:
Merge Um and U,, into Umarge using Eq
11:
Update V: V & (V) {Um, U, (Vmage
Output:
Learned policies { Tex }
where Umerge is added to the knowledge vector pool V. The merged vector Umerge replaces Um and Vn. reducing memory requirement while maintaining the knowledge from both vectors. It ensures that the knowledge vector pool remains compact and efficient while preserving knowledge from previous tasks. Through iterative merging, this strategy dynamically maintains balance between memory efficiency and knowledge retention, addressing scalability issues in dynamic environments.
Remark: For a detailed mathematical analysis supporting the performance and stability of the proposed method, please refer to Appendix A
5 Experiments
5.1 Experimental Settings
Benchmarks. We follow the experimental settings established in prior work [32] and compare
CKA-RL with SOTA CRL methods across three distinct dynamic task sequences, including 1) Meta-World (57), 2) Freeway [11. and 3) SpaceInvaders [31]. These sequences are designed to
evaluate the robustness and generalization capabilities of different methods under varying levels of task complexity and action space characteristics. More details can be seen in Appendix C
Metrics. Following standard evaluation metrics in CRL 55 report key metrics, including average performance forward transfer. The average performance at step f, denoted as P(t):
1
P(t)
Σ Pt(t). N
where (t) E [o, 1] is the success rate on task at step t, and each of the tasks is trained for A steps, where N is the number of tasks, so the total number of steps is T = N A. Its final value P(T), serving as a conventional evaluation metric in CRL 332 (33. effectively captures the model's stable performance across dynamic environments. The forward transfer measures the extent to which a CRL method able to transfer knowledge across tasks. computed as the normalized area between the training curve of the measured and the training curve of reference model trained from scratch. Let De E [0, I] denote the reference performance. The forward transfer on task denoted FT,, is defined follows:
FT, = AUC, - AUC
//
p.(t) dt. AUC, A /.²
1 AUC! AUC; A
pi()) dt.
(9)
(1-)
The average forward transfer for all tasks FT is defined as:
1 N FT
N FT.
Σ
(10)
i=

---

Table I: Experimental results comparing the proposed method with nine SOTA methods across Meta- World, SpaceInvaders, and Freeway environments. We report the results for average performance (PERF.) and forward transfer (FWT.), with the best results highlighted in bold. The proposed CKA-RL achieves superior performance and forward transfer in all three sequences.
Meta- World
SpaceInvaders
Freeway
Average
Method
PERF.
FWT.
PERF
FWT.
PERF.
FWT.
PERF
FWT.
Baseline
0.4191
0.0000
4.6314
a.0000
0.1247=024
0.0000
0.3917
0.0000+a.0
FT-1
0.0313 +0.2142
0.6864
0.1512
0,6935 s.00
0.2079
0,3886+0.7
FT-N
0.3774 +0.2142
0.9785 1oo4
0.6864
0.7532::0.6
0.6935
0.7030
0.3886+1..0
ProgNet
0,4157 -0.037910.1
0.3757
-0.0075
0.3125
0,1938 10.31
0,368010.6 0,0495
PackNet
0.2523300 +o 0.2299
-0.0750=aus
0.2767=0.36
0.1970=aa2
0,2530
-03
MaskNet
0.3263 =0.3695
0.0000
-0.3866=0ss
0.0644=a.7
-0.0503
0.1302+
-0.2688
CReLUs
0.3789 =0.0089+0.2
0.
0.5308
0.7835
0.7303
0.6832
0.4174::0.2
CompoNet
0.4131 +0.0055
0,9828
0.6963
0.7629
0.7115
0.7196+0.3
0.46744
CbpNet
0.4368 -0.0826+0.2
0.83
a.28444aa2
0.7678=
0.7201 +0.07
0.6813+eo.
0.3740
CKA-RI.
0.4642_a.s0 -0.00322a21 0.9928 0.7749
0.7923 0.7429 0.7498 0.5049 2017
Methods. We compare the CKA-RL with nine SOTA methods. 1) Baseline. 2) FT-1 (Fine- Tuning
Single Model) (53). 3) FT-N (Fine-Tuning with Model Preservation) [53]. 4) ProgNet 33 5)
PackNet 33 MaskNet (5). 7) CReLUs [I] 8) CompoNet (32. 9) CbpNet [1].
Implementation Details. We follow the prior work [32), employing SAC [6 for Meta- World and PPO 42 for Freeway and Spacelnvaders. For dighddimesiionl Atari inputs (210x160 RGB), CNN encoder maps images to compact latent features. All tasks are trained for = 1M steps. We use Adam (momentum 0.9, second moment 0.999), with batch sizes 1024/122 and learning rates 2.5 10-4/1 x 10-3 for PPO/SAC. The discount factor 7 0.99. For SAC, the action standard deviation is constrained to je-20, e²). with target smoothing coefficient 5 10 auto-tuned entropy coefficient 0.2, and action noise clipped to 0.5. Learning starts after 5 x 10a steps using 10ah random actions for exploration. Policy and target networks are updated every 2 and steps, respectively, using 3-layer MLPs with 256 hidden units. For PPO, we apply GAE with À = 0.95 across 8 parallel environments, gradient clipping at 0.5, PPO clip of 0.2, entropy coefficient 0.01, and 128 rollout steps. The agent uses a 2-layer MLP with 512 units, and advantage normalization is employed. Following 55 32 55) experiments using 10 different random seeds to ensure the robustness and reliability of the results.
CRL is applied only the actor, while the critic is reinitialized at each task. We conduct
5.2 Comparison Experiments
We compare the proposed CKA-RL with SOTA meth- ods to demonstrate its superior performance and knowledge transfer capability. Experiments ducted on three distinct task sequences, including Meta-World, Freeway, and Spacelnvaders, as summa- rized in Table and Figure
0.8
CKA-RL(OUrS)
CompoNer
2024
0.7-
Chypaa22024
Average Performance
CRcLU-2022
0.6
0.5
0.4
ProgNe12016
Consistent superiority of CKA-RL across diverse environments. The comparative results across three environments, as shown in Table demonstrate the exceptional performance of CKA-RL. Notably, the CKA-RL consistently surpasses SOTA CRL meth- ods in all environments. Specifically, when averaged across three environments, the proposed CKA-RL achieves an improvement of at least 4.20% in per- formance (0.7196 > 0. 0.7498) and 8.02% in forward transfer (0.4674 + 0.5049) compared to CompoNet.
0.3
PaccNNe20088
Better
0.2
2022
0.1
0.6
Average Forward Transfer
Figure 2: Comparison of SOTA methods with the proposed CKA-RL in terms of average forward transfer and average performance.
demonstrating its superior capability in handling diverse task sequences.
Robust performance dynamic environments. As shown in Table D our CKA-RL maintains consistent superiority over existing CRL methods across various dynamic environments. In Meta-
7

---

Performance [Porward Thunster
FINPE
6.78
CorpoNaa
ProgNe
500
400
0.74
Epiandic
100
6.7
Fiewaa Tratafer. (Cuza)
200
Feesoal Tearsfor (PT-N)
Perfommaace. (Ches)
100
0.70
Perbomasss (FTN))
0
15k
50k
45k
Size of Knowledge Vector Pool
Global Step
Figure 3: Effects of different pool size max Adaptive Knowledge Merging in the Freeway sequence. Our proposed CKA-RL achieves ex- cellent performance when Kmax 5.
Figure Reward curve comparison in the SpaceInvaders sequence. CKA-RL attains higher initial reward and converges faster, un- derscoring superior use of historical vectors.
World sequence, CKA-RL achieves a significant performance improvement of 6.27% compared to CbpNet (0.4368 - 0.4642). Similarly, it achieves performance improvement of 1.02% over CompoNet (0.9828 + 0.9928) in SpaceInvaders sequence and a performance improvement 1. 1.12% CReLUs (0.7835 0.7923) in Freeway sequence. These consistent improvements across different environments highlight the robustness and stability of our method in dynamic environments.
Superior knowledge transfer capability. From Table ! CKA-RL consistently achieves superior forward transfer performance across all three environments, demonstrating its exceptional knowledge transfer ability. In Meta-Wort sequence, CKA-RL shows improvement of 41.82% compared to CompoNet. Similarly, it outperforms CompoNet (0.6963 + 0.7749) by 11.29% in SpaceInvaders sequence and surpasses CReLUs by 1.73% in Freeway sequence. These results substantiate that method enables more efficient knowledge transfer compared to existing CRL methods.
Superior model plasticity. As shown in Table | our method outperforms SOTA plasticity enhance- ment techniques [ D. across evaluation metrics, including ChpNet and CReLUs. Notably, in Spacelnvaders sequence, our method achieves a significant improvement of 18.30% in average performance (0.8392 -> 0.9928) and an 59.97% increase forward transfer (0.4844 + 0.7749) compared to CbpNet. These comprehensive experimental results demonstrate that our approach surpasses existing plasticity enhancement methods, by effectively leveraging historical knowledge for adaptation. This highlights the potential of adaptive processing to improve model plasticity.
5.3 Ablation Studies
Effectiveness of Components in CKA- RL. Our CKA-RL enhances the learn- ing process by effectively using historical knowledge vectors and merging redundant vectors. We ablate both components in Ta- ble [2] Compared with the base method (FT-N), simply averaging knowledge tors without adaptive weighting actually performs worse than the baseline (0.7437),
Table 2: Effectiveness of components in CKA-RL in Freeway task sequence. 'Avg' is averaging knowledge vectors, 'Adapt' is the Continual Knowledge Adaptation, and 'Merge' is the Adaptive Knowledge Merging.
Method
Extra Memory PERF FWT.
Base
0
0.7532 0.6935
Base + Avg
0.7437 0.7127
indicating that naive merging without adap- contrast, in-
Base Adapt
0.7821 0.7321
tation harms performance. troducing the Continual Knowledge Adap- tation strategy achieves higher performance
Base Adapt + Merge
0.7923 0.7429
and forward transfer (e.g., 0.7532 (0.6935) vss. 0.7821 (0.7321) on Freeway). This confirms the effectiveness of utilizing historical knowledge vectors for enhancing knowledge transfer across dynamic environments. When further merging redundant knowledge vectors that are similar, our method achieves comparable performance (e.g., 0.7821 + 0.7923 on Freeway), improves the forward transfer (e.g., 0.7321 - 0.7429), and reduces memory requirements (e.g.. N + 5), demonstrat- ing the effectiveness of Adaptive Knowledge Merging strategy in addressing scalability issues and maintaining the stable performance dynamic environments.

---

0.009
Parameler Memory (MB)
100
CompoNet
0.008
CompoNet
ProgNet
ProgNet
@
80
CKA-RL
CKA-RL
0.006
60
0.005
40
0.003
20
0.002
0.001
2
4 Tassk
8
0
2
6
8
(a) Total parameter memory consumption across an increasing number of sequential tasks.
(b) Inference latency across an increasing number of sequential tasks in Spacelnvadors environment.
Figure Analysis of Total Parameter Memory Consumption and Inference Latency: (a) demonstrates that CKA-RL maintains nearly constant parameter and activation size through vector merging, while others grow linearly; (b) shows that CKA-RL maintains nearly constant inference latency, while others suffer from increasing computational overhead. More details can be found in Appendix B
Effects of Pool Size in Adaptive Knowledge Merging, The size of the knowledge vector pool plays a crucial role controlling the preservation of historical knowledge. We conduct experiments with Kmax values set to (3, 4,5, 6, 7j. From Figure our CKA-RL. achieves excellent performance when Kmax equals 5. Fither smaller larger K max hampers the performance. The reasons are as follows. When K is small, CKA-RI. removes too many knowledge vectors during adaptive knowledge merging, thus being unable to utilize enough knowledge. When Kax is too large, redundant or even conflicting knowledge may slow down the knowledge adaptation process, resulting in performance degradation. Since larger Kmux leads to higher memory requirements, we set Kmux to for Freeway and SpaceInvaders experiments, and to 8 for Meta- World, which has larger number of tasks.
5.4 More Discussions
Superior Initial Performance and Faster Convergence. One of the key advantages of our proposed CKA-RL is its superior initial performance and rapid convergence, as shown in Figure The reward curve of our CKA-RL demonstrates higher starting point compared indicating that our method effectively utilizes the historical knowledge vectors. Furthermore, the curve exhibits a steeper ascent, reaching convergence much faster than the competing methods. This not only highlights the efficiency of our CKA-RL but also underscores its effectiveness in utilizing historical knowledge vectors.
existing CRL methods,
Cross-task Performance of CKA-RL. To assess how well the final policy consoli- dates information acquired throughout the learning process, we freeze the model af- ter the last task and evaluate its perfor- mance on all previously seen tasks, porting the average performance across three environments in Table CKA-RI achieves the highest average performance (0.3966), slightly surpassing the closest competitor CbpNet (0.3933) and clearly outperforming FT-N (0.3733) and CReLUs (0.3666). These results indicate that CKA- RL, which combines knowledge adaptation with adaptive merging, results in effective final policy that performs better across multiple tasks.
Table 3: Cross-task performance comparison evaluated with the final policy Tēx
Method
Pub. Year
Average Performance
Baseline
0.2833
FT-N
53
0.3733
ProgNet 39
0,2966
PackNet 33
CVPR'2015
0.1900
MaskNet
TMLR '2023
0.2266
CReLUs 1
CoLLAs' '022
0.3666
CompoNet 32
(CML'202)
0.3900
CbpNet [1
Nature'200 2024
0.3933
CKA-RL (Ours)
0.3966
Memory Efficiency CKA-RL. evaluate the memory efficiency of CKA-RI. by comparing the growth of model parameters across tasks from SpaceInvaders. As shown in Figure 5a the total

---

parameter memory of CKA-RL remains nearly constant beyond the fifth task due to our use of the adaptive knowledge merging, which maintains a bounded knowledge vector pool. This ensures that model complexity does not scale linearly with the number of tasks.
Inference Efficiency of CKA-RL. From Figure 5b] CKA-RL achieves the highest performance and forward transfer with an average inference time of only 0.0012s per input. Notably, CKA-RL latency remains essentially constant regardless of the number of tasks, since consolidates historical knowledge into a fixed-size policy parameter during the parameter construction phase, eliminating the need for complex runtime composition. contrast, existing architectures such as CompoNet and ProgNet exhibit significant increases in inference cost as the number of tasks grows.
Effectiveness of CKA-RL in Miti- gating Catastrophic Forgetting. We use the standard forgetting metric (58). which evaluates performance degradation after training on new lasks. As shown in Table 4 CKA- RL achieves relatively low forget- ting rate (0.45), demonstrating its ef- fective retention of previously learned knowledge while continuing to adapt to new tasks. In contrast, PackNet, which shows the lowest forgetting rate (0.04), sacrifices significant overall per- formance (0.2767). In contrast, CKA- RL achieves the highest performance (0.7923) and forward transfer (0.7429) across all methods, with a balanced forgetting rate, highlighting its robustness preventing catas- trophic forgetting while maintaining strong performance.
Table 4: Forgetting analysis on Freeway. We report overall performance (PERF.), forward transfer (FWT.., and forget- ting, computed pre-post performance on each prior task after learning a new task (lower is better).
Method
Pub. Year
PERF. FWT. Forgetting
ProgNet
39
0.3125 01938
0.36
PackNet
33
CVPR 0.2767 0. 1970
0.04
MaskNet
AMLR'22023 0.0644-0.00 -0.0503
0.04
CReLUs 0
CoLLAs 2023 0.7303
0.52
CompoNet 32
ICML' '024 0.7629 0.7115
0.49
ChpNet 11
Nature'2024 0.7678 0.7201
0.46
CKA-RL (Ours)
0.7923 0.7429
0.45
6 Conclusion
In this paper, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge, thereby accelerating learning in new tasks and explicitly reducing performance degradation on previous tasks. Specifically, we assume that the agent acquires a unique knowledge vector for each task during continual learning. Based on this, we develop a Continual Knowledge Adaptation strategy that enhances knowledge transfer from previously learned tasks. Furthermore, we introduce an Adaptive Knowledge Merging mechanism that combines similar knowledge vectors to address scalability challenges. The CKA-RL outperforms SOTA methods, with 4.20% overall gain and an 8.02% boost in forward transfer.
Acknowledgements
This work was partially supported by the Joint Funds of the National Natural Science Foundation of China (Grant No. U24A22225, No.U23B2013).
References
[1] Z. Abbas, R. Zhao, J. Modayil, A. White, and M. C. Machado. Loss of plasticity in continual deep reinforcement learning. In S. Chandar, R. Pascanu, H. Sedghi, and D. Precup, editors, Conference on Lifelong Learning Agents, 22-25 August 2023, McGill University, Montréal, Québec, Canada, volume 232 of Proceedings of Machine Learning Research, pages 620-636. PMLR, 2023.
[2] D. Abel, A. Barreto, B. V. Roy, D. Precup, H. P. van Hasselt, and S. Singh. A definition of continual reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA USA, December 10- 16, 2023, 2023.
10

---

[3] N. Anand and D. Precup. Prediction and control in continual reinforcement learning. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023.
[4] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. J. Artif. Intell. Res., 47:253-2799 2013.
[5] E. Ben-Iwhiwhu, S. Nath, P. K. Pilly, S. Kolouri, and A. Soltoggio. Lifelong reinforcement learning with modulating masks. Trans. Mach. Learn. Res., 2023, 2023.
[6] G. Chen, S. Niu, D. Chen, S. Zhang, C. Li, Y. Li, and M. Tan. Cross-device collaborative test-time adaptation. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
[7] P. Chen, D. Ji. K. Lin, W. Hu, W. Huang, T. H. Li, M. Tan, and C. Gan. Learning active camera for multi-object navigation. S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November
28 December 9, 2022, 2022.
[8] I. Clavera, J. Rothfuss, J. Schulman, Y. Fujita, T. Asfour, and P. Abbeel. Model-based reinforce- ment learning via meta-policy optimization. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pages 61 617-629. PMLR, 2018.
[9] Z. A. Daniels, A. Raghavan, J. Hostetler, A. Rahman, I. Sur, M. R. Piacentino, A. Divakaran, R. Corizzo, K. Faber, N. Japkowicz, M. Baron, J. S. Smith, S. Joshi, Z. Kira, C. E. Taylor, M. B. Gurbuz, C. Dovrolis, T. L. Hayes, C. Kanan, and J. Gallardo. Model-free free generative replay for lifelong reinforcement learning: Application to starcraft-2. In S. Chandar, R. Pascanu, and D. Precup, editors, Conference on Lifelong Learning Agents, CoLLAs 2022, 22-24 August 2022, McGill University, Montréal, Québec, Canada, volume 199 of Proceedings of Machine Learning Research, pages 1120-1145. PMLR, 2022.
[10] Dee.Seck-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501.12948, 2025.
[11] S. Dohare, J. F. Hernaneez-Gariia, Q. Lan, P. Rahman, A. R. Mahmood, and R. S. Sutton, Loss of plasticity in deep continual learning. Nat., 632(8026):768-774, 2024.
[12] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In D. Precup and Y. W. Teh, editors, Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6- 6- August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1126-1133 1135. PMLR, 2017.
[13] H. Fu, Y. Sun, M. Littman, and G. Konidaris. Knowledge retention in continual model-based reinforcement learning. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net, 2025.
[14] J. Gaya, T. Doan, I. Caccia, L. Soulier, I.. Denoyer, and R. Raileanu. Building subspace of policies for scalable continual learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[15] A. Gupta, S. Savarese, S. Ganguli, and L. Fei-Fei. Embodied intelligence via learning and evolution. CoRR, abs/2102.02202, 2021.
[16] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off Ooflpy maximum entropy deep reinforcement learning with stochastic actor. In J. G. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML, 2018, Stockholmsmüsssa, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR, 2018

---

[17] J. Hu, Y. Wang, S. Zhang, K. Zhou, G. Chen, Y. Hu, B. Xiao, and M. Tan. Efficient dynamic ensembling for multiple LLM experts. In Proceedings of the Thirty-Fourdy International Joint Conference on Artificial Intelligence, IJCAI 2025, Montreal, Canada, August 16-22, 2025, pages 8095-8103. ijcai.org. 2025.
[18) J. Hu, W. Zhang, Y. Wang, Y. Hu, B. Xiao, M. Tan, and Q. Du. Dynamic compressing prompts for efficient inference of large language models. CoRR, abs/2504. 1004, 2025.
[19] S. Hu, Y. Zhou, Z. Fan, J. Hu, L. Shen, Y. Zhang, and D. Tao. Continual task learning through adaptive policy eeff-comoosition. CoRR, abs/241 1.1 1364, 2024.
[20] M. Igl, G. Farquhar, J. Luketina, W. Boehmer, and S. Whiteson. Transient non non-stationarity and generalisation in deep reinforcement learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[21] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing mod- els with task arithmetic. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
[22] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. pages 19599,, 2019.
[23] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll. J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. In International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019, pages 6023 1229.602. IEEE, 2019.
[24] L. P. Kaelbling, M. L. Littman, and A. W. Moore. Reinforcement learning: A survey. J. Artif. Intell. Res., 4:237-285, 1996.
[25] K. Khetarpal, M. Riemer, I. Rish, and D. Precup. Towards continual reinforcement learning: A review and perspectives. J. Artif. Intell. Res., 75:1401-1476, 2022.
[26] B. Kim, C. Ahn, W. J. Baddar, K. Kim, H. Lee, S. Ahn, S. Han, S. Suh, and E. Yang. Test- time ensemble via linear mode connectivity: path better adaptation. The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net 2025.
[27] D. Kumaran, D. Hassabis, and J. L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends cognitive sciences, 20(7):512- 534, 2016.
[28] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
[29] K. Lin, Y. Wang, P. Chen, R. Zeng, Y. Lei, S. Zhou, Q. Du, M. Tan, and C. Gan. When to align: Dynamic behavior consistency for multiagent systems via intrinsic rewards. IEEE Trans. Neural Networks Learn. Syst., 36(12):20105-20) 2011005-2011999 19, 2025.
[30] F. Luo, S. Jiang, Y. Yu, Z. Zhang, and Y. Zhang. Adapt to environment sudden changes by learning a context sensitive policy. In Thirty- SSxth AAAI Conference on Artificial Intelligence, Innovative Applications of Artificial Intelligence, IAAI
AAAI 2022, Thirty-Fourth Conference 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- - March 1, 2022, pages 7637-7646. AAAI Press, 2022.
[31] M. C. Machado, M. G. Bellemare, E. Talvitie, Veness, M. J. Hausknecht, and M. Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Artif. Intell. Res., 61 552-562, 2018
[32] M. Malagón, J. Ceberio, and J. A. Lozano. Self-composing policies for scalable continual reinforcement learning. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024.
12

---

[33] A. Mallya and S. Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 7765-7773. Computer Vision Foundation / IEEE Computer Society, 2018.
[34] J. A. Mendez, H. van Seijen, and E. Eaton. Modular lifelong reinforcement learning via neural composition. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[35] Y. Meng, Z. Bing. X. Yao, K. Chen, K. Huang, Y. Gao, F. Sun, and A. Knoll. Preserving and combining knowledge in robotic lifelong reinforcement learning. Nat. Mac. Intell., 7(2):256- 269, 2025.
[36] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In M. Balcan and K. Q Weinberger, editors, Proceedings of the 33nd International Conference Ma- chine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1933 JMLR.org. 2016.
[37] A. Muppidi, Z. Zhang, and Yang. Fast TRAC: A parameter- free optimizer for lifelong reinforcement learning. In A. Globersons, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. M. Tomezak, and C. Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver. BC, Canada, December 10 15, 2024, 2024.
[38] B. Peng, X. Li. J. Gao, J. Liu, and K. Wong. Deep dyna-q: Integrating planning for task- completion dialogue policy learning. In I. Gurevych and Y. Miyao, editors, Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics. ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume I: Long Papers, pages 2182-2192 Association for Computational Linguistics, 2018
[39] A. A. Rusu, N. C. Rabinowitz. G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. CoRR, abs/1606.04671 2016.
[40] Schmidhuber, J. Zhao, and N. N. Schraudolph. Reinforcement learning with self-modifying policies. In S. Thrun and L. Y. Pratt, editors, Learning to Learn, pages 293-209 Springer, 1998.
[41] Schulman, S. Levine, P. Abbeel, M. I. Jordan, and P. Moritz. Trust region policy optimization. In F. R. Bach and D. M. Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JorR. 2015.
[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. CoRR, abs/1777.003477 2017.
[43] Z. Shao, P. Wang, Q. Zhu, R. Xu, Song, M. Zhang, K. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. CoRR, abs/2402.03300, 2024.
[44] J. Shen, H. Lai, M. Liu, H. Zhao, Y. Yu, and W. Zhang. Adaptation augmented model-based policy optimization. Mach. Learn. Res., 24:218:1-2188 2023.
[45] H. Sun, J. Hu, Z. Zhang, H. Tian, X. Xie, Y. Wang, Z. Yu, X. Xie, and M. Tan. Open-world drone active tracking with goal-centered rewards. In The Thirty-ninth Annual Conference Neural Information Processing Systems, 2025.
[46] 7. Sun and Y. Mu. Rewiring neurons in non-stationary environments. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[47] R. S. Sutton and A. G. Barto. Reinforcement learning an introduction. Adaptive computation and machine learning. MIT Press, 1998.
13

---

[48] R. S. Sutton, D. A. McAllester, S. Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function upproximation. In S. A. Solla, T. K. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems 12, [NIPS Conference, Denver, Colorado, USA, November 29- December 4, 1999], pages 1057-1063. The MIT Press, 1999.
[49] E. Ullah, T. Mai, A. Rao, R. A. Rossi, and R. Arora. Machine unlearning via algorithmic stability. In M. Belkin and S. Kpotufe, editors, Conference Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA, volume 134 of Proceedings of Machine Learning Research, pages 1126-4142 PMLR, 2021.
[50] Z. Wang, E. Yang. L. Shen, and H. Huang. A comprehensive survey of forgetting in deep learning beyond continual learning. IEEE Trans. Pattern Anal. Mach. Intell., 2025.
[51] C. J. H. Watkins et al. Learning from delayed rewards. 1989.
[52] R. J. Williams. Simple statistical
algorithms for connectionist reinforcement
learning, Mach. Learn., 8:229-256, 1992.
[53] M. Wolczyk. M. Zajac, R. Pascanu, L. Kucinski, and P. Milos. Continual world: A robotic benchmark for continual reinforcement learning. In M. Ranzato, A. Beygelkimer, Y. N. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems 34: Anmual Conference on Neural Information Processing Systems 2021, NeurlPS 2021, December 6-14, 2021, virtual, pages 28496-2888 28510, 2021.
[54] M. Wolezyk, M. Zajac, R. Pascanu, L. Kucinski, and P. Milos. Disentangling transfer in continual reinforcement learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
[55] Y. Yang, T. Zhou, Jiang, G. Long, and Y. Shi. Continual task allocation meta-policy network via sparse prompting. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 06623-39638... PMLR, 2023.
[56] Q. Yu, Z. Zhang, R. Zhu, Y. Yuan, X. Zuo, Y. Yue, T. Fan, G. Liu, L. Liu, X. Liu, H. Lin, Z. Lin, B. Ma, G. Sheng, Y. Tong, C. Zhang, M. Zhang, W. Zhang, H. Zhu, J. Zhu, J. Chen, J. Chen,
C. Wang, H. Yu, W. Dai, Y. Song, X. Wei, H. Zhou, J. Liu, W. Ma, Y. Zhang, L. Yan, M. Qiao, Y. Wu, and M. Wang. DAPO: an open-source LLM reinforcement learning system scale.
CoRR, abs/2503. 14476, 2025.
[57] T. Yu, D. Quillen, Z. He. R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. In L. P. Kaelbling. D. Kragic, and K. Sugiura, editors, 3rd Anmual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pages 1094-1100. PMLR, 2019.
[58] T. Zhang, K. Z. Shen, Z. Lin, B. Yuan, X. Wang, X. Li, and D. Ye. Replay-enhanced continual reinforcement learning. Trans. Mach. Learn. Res., 2023, 2023.
[59] T. Zhang, X. Wang, B. Liang, and B. Yuan. Catastrophic interference in reinforcement learning: A solution based on context division and knowledge distillation. IEEE Trans. Neural Networks Learn. Syst. 34(12):9925-9939 2023.
[60] C. Zheng, H. Yin. J. Chen, T. Ng. Y. Ong, and I. W. Tsang. Mastering continual reinforcement learning through fine-grained sparse network allocation and dormant neuron exploration. CoRR, his/2503300246,, 2025.
14

---

Supplementary Materials for "Continual Knowledge Adaptation for Reinforcement Learning"
Contents
Mathematical Analysis
16
More Related Work
17
Environments and Tasks
17
Meta-World
18
Spacelnvaders
18
Freeway
19
Implementation Details
20
E Efficiency of CKA-RL
22
Performance Memory Cost Analysis
22
Performance Inference Cost Analysis
23
Further Experimental Results
23
Discussions and Future Works
24
Limitations and Future Works
24
Broader Impacts
24
15

---

A
Mathematical Analysis
We provide some theoretical analysis to support the improved performance of our proposed method. The analysis includes several key lemmas and corollaries, which demonstrate the stability and performance the knowledge merging mechanism in the context of continual reinforcement learning.
Preliminaries. We parameterize the policy for task Th as:
02 Obsse Σ Cajoz
Uke with
-1
(11)
where the normalized adaptation factors at are produced by softmax over learnable 8 (Eq. When the pool size exceeds K Kmax, we merge the most similar knowledge vectors by averaging (Eq.
Lemma (Drift bound under convex reuse). Let D :- 02 Boase nonnegative and sum
Ejen ufjs + ok- Since a
have:
| S
Σ a jck
S
Σ jek
Latllu 1 I
max |lyl + I|wxll- jek
(12)
Thus, the deviation from the base is controlled by the magnitudes of (a small subset of) vectors actually reused and the current task vector. This uses the normalization of a given by Eq. 3 44..
Corollary 1 (Lipschitz Performance Stability). If the task-k return J& (0) is T-LLipschitz parame- ters, then:
(Ox) - JomOne + oK)l S L 3 oll S 1 1l
(13)
j<k
jck
Thus, reusing historical vectors cannot hurt beyond tunable, data-dependent bound, and the bound tightens as a concentrates on small-norm or well-aligned vectors (see Eq. (3
(4)).
Lemma 2 (Interference Reduces with Near Near-Orthogonaligy Let Sig
": o4
be the cosine
Nvx |
similarity. [Sil] S < 1 for i * j. then:
Σ a = (aa)) jek
Σ aaaa iA3
s Eaa jek
jEh
(14)
Hence, the "cross-task" term is Os)). Our empirical cosine analysis shows knowledge vectors nearly orthogonal, with off-diagonal values in [-0.24, 0.12], while full fine-tuned parameters have strong correlations (0.93 0.95). This matches the design goal of reducing interference.
Corollary (Combining Lemma and Lemma 2). Combining Lemma I and Lemma 2, both the the improved retention
drift and the cross-terms that cause interference are seen in the final-policy evaluation (Table 3
Lemma (Bounded Error of Adaptive Merging). Suppose we must replace (Um Un) by Umerge (u/m + 2,)) when V >> K Kmax- For any convex coefficients X, 2 0, A + 1, we have:
|eem + pmnn
Umerree S 1 ||em U
(15)
their cosine similarity Smn 2 8 with 5 E [0, 2), then:
Ulm v,.l S 2(1 Smn) max (.. [| I) S 20 max([Ium |lw,l)).
(16)
Thus, the parameter perturbation induced by merging is O(vō).
Corollary 3 (Performance Stability under Merging). With L-Lipschitz Jg. we have:
Jk (Bafter-merre) - Ju (e (gbefore-merge) - L 11 - -.,ll < L 2(1 - Smn) max([Iuml)) [[...)). (17)
Hence, merging similar vectors (large Smm) has small, explicitly bounded effect, justifying our "merree-th-moo" tethe-mot--smillll rule in Eq. 6
16

---

B More Related Work
Reinforcement Learning (RL) (2 29 constitutes a paradigm within machine learning wherein an agent learns to optimize its decision-making process through interaction with an environment. This interaction involves performing actions and receiving consequent feedback, typically in the form of rewards or penalties. The principal learning objective in RL is the maximization of a cumulative reward signal. In contrast to supervised learning, which relies on datasets comprising pre-defined input-output pairs for model training, RL entails an agent acquiring knowledge from the repercussions of its actions, mediated by this reward-penaly, mechanism. This iterative, trial-and-error learning process, coupled with its emphasis on sequential decision-making under uncertainty, distinguishes RL from supervised learning methodologies that depend on labeled datasets. Existing reinforcement learning algorithms can be broadly categorized based on whether an explicit model of the environment is learned or utilized, leading to two principal classes: Model-free RL and Model-based RL.
Model-free RL. Model-free RL algorithms enable the agent to learn optimal policies directly from trajectory samples accrued through interaction with the environment, without explicitly construct- ing an environmental model. Within model-free RL, algorithms are further distinguished by the components they learn, leading to three primary sub-categories: actor-only, critic-only, and actor- critic algorithms. Actor-only algorithms directly learn a policy network, denoted as (a(a))), which maps states to actions. This network takes the current state St as input and outputs the action at- Prominent examples of such algorithms include Reinforce [52] and various policy gradient methods [48] Critic-only algorithms, in contrast, focus solely on learning a value function (c.g., state-value or action- value function). Given state 8t, the learned value model is used to evaluate all possible actions a' E A, and the action a yielding the maximum estimated value is selected. This category encompasses methods such as Q-learning [51] Actor-critic algorithms combine these two approaches by concurrently maintaining and learning both a policy network (the actor) for action selection and a value function model (the critic) for evaluating actions or states. This category includes algorithms such as Deep Deterministic Policy Gradient (DDPG) (28). Trust Region Policy Optimization (TRPO) Proximal Policy Optimization (PPO) 42). and Asynchronous Advantage Actor-Critic (A3C) 36 Notably, PPO has gained considerable traction for training large language models. Recent advancements in this area include GRPO 43 which employs group-based based advantage estimates within a KL-regularized loss function to reduce computational overhead and enhance update stability, and DAPO [56), which utilizes distinct clipping mechanisms and adaptive sampling techniques to improve efficiency and reproducibility during the fine-tuning of large-scale models.
Model-based RL. Model- RL algorithms endeavor learn an explicit model of the environment, thereby addressing challenges related to sample efficiency. This is because the agent can leverage the learned model for planning and decision-making, reducing the necessity for extensive direct environmental interaction. The learned representation of the environment is commonly termed a 'world model'. This world model typically predicts the subsequent state 8(+1 and the immediate reward r based on the current state s; and the action ae taken. Exemplary model-based RL algorithms include Dyna-Q (38), Model-Based Policy Optimization (MBPO) [22], and Adaptation Augmented Model-based Policy Optimization (AMPÓ) (4].
C Environments and Tasks
Our continual learning experiments evaluate agents across three complementary task domains de- signed to test different capabilities: robotic manipulation with continuous control (eela)Wold)) [57], and two vision-based Atari environments [3 4 emphasizing dynamic decision -aaiiig (SpaceIn- vaders) and sparse-reward navigation (Freeway). This combination spans key RL challenges including high-dimensional state spaces, delayed rewards, procedural variations, and partial observability.
The World tasks evaluate precise motor control and tool manipulation, while the Atari envi- ronments provide contrasting challenges. SpaceInvaders tests rapid visual processing and threat response under varying enemy behaviors, and Freeway examines strategic planning in sparse-reward conditions with evolving obstacle patterns. Collectively, these environments form a comprehensive benchmark for evaluating continual learning.
17

---

Figure 6: Example frames from each the Meta- World tasks used in the continual learning benchmark. Tasks are repeated twice to evaluate sequential skill acquisition.
C.1 Meta-World
The robotic manipulation sequence utilizes the Meta- World benchmark [57]. featuring 39-dimensional state observations (encoding arm/object positions) and 4-dimensional continuous actions (arm dis- placement and gripper torque in [-1. 1.). To evaluate continual learning capabilities, we adopt 20-task sequence consisting of 10 distinct manipulation tasks repeated twice, following the Continual World (CW200 benchmark [53. This selection covers diverse manipulation skills while maintaining consistent evaluation protocols with prior work. The following lines describe the selected tasks:
hammer-v2. Hammer a screw into a wall with randomized initial positions.
push-wall-v2. Navigate a puck around obstacles to reach target location.
faucet-close-v2. Rotate faucet handle clockwise from variable starting positions.
push-back-v2. Position a mug beneath coffee machine with spatial randomization.
stick-pull-v2. Retrieve a box using stick as a tool.
handle-press-side-v2. Apply lateral force to press down handle.
push-v2. Basic puck pushing to variable target locations.
shelf-place-v2. Precisely place puck onto a shelf.
window-close-v2. Slide window closed from randomized openings.
peg-unplug-side-v2. Remove laterally mounted peg.
Each episode features randomized object and goal positions, testing robustness to environmental variations. The task sequence progresses from basic manipulations (pushing) complex tool- use scenarios (hammering, stick-pulling), providing a comprehensive benchmark for evaluating generalization across skills.
C.2 SpaceInvaders
This arcade-style challenge utilizes the AS environment from the Arcade Learning Environment. In this classic game, the agent controls a laser cannon defend Earth against descending alien invaders. Observations are provided as RGB frames 22101600x] with the action space comprising six discrete actions: NOOP (no operation), FIRE, RIGHT, LEFT, RIGHTFIRE (combined movement and firing), and LEFTFIRE, The agent has three lives, and the game terminates when either all líves are lost invaders reach the ground. Rewards granted for destroying invaders, with higher-value targets located in back rows.
Additional details available hhttps://all http://ll....aaaaaooo/oggg////// http:///w.....aaaooooeeeeaaatttt
18

---

DOOD
ROOO
BOOS DOON
0015 DOOO
BOSD
DOON
2045
2OOD
X A
K
X A X
X
X
A
K
X
A X
A
X X
A
R
R R
R
A
Figure 7: Example gameplay frames from the Spacelnvaders environment.
To systematically evaluate the agent's capability in dynamic threat scenarios, examine ten strategically selected game modes that modify enemy behavior and environmental dynamics:
Mode 0 (Baseline). Standard configuration with static shields and predictable bomb trajectories.
Mode 1 (Mobile Shields). Shields oscillate horizontally, eliminating reliable cover positions.
Mode 2 (Zigzag Bombs). Invader bombs follow non non-linear trajectories, increasing evasion difficulty.
Mode 3 (Composite Challenge). Combines mobile shields (Mode 1) with zigzag bombs (Mode 2).
Mode 4 (High-Speed Bombs). Baseline configuration with accelerated bomb descent rates.
Mode 5 (Mobile Shields + Fast Bombs). Integrates Mode I's dynamic shields with Mode bomb velocity.
Mode 6 (Zigzag+ Fast Bombs), Combines Mode 2's erratic bomb paths with increased speed.
Mode 7 (Full Complexity). Merges all modifiers: mobile shields, zigzag bombs, and high velocity.
Mode 8 (Intermittent Visibility). Invaders periodically become invisible, testing memory and prediction.
Mode 9 (Dynamic Visibility). Mobile shields (Mode 1) coupled with intermittent invader visibility.
C.3
Freeway
The Freeway experiments are conducted using the ALE/Freeway-vs environment from the Arcade Learning Environment. In this environment, the agent controls a chicken attempting to cross multi-lane highway with moving vehicles. Observations are provided as RGB frames (210x160x3) with action space consists of three discrete actions: NOOP (no operation), UP (move forward), and DOWN (move backward). Rewards are exceptionally sparse, the agent only receives +1 upon successfully reaching the top of the screen after crossing all traffic lanes.
Figure 8: Example gameplay frames from the Freeway environment.
To evaluate the agent's adaptability across varying difficulty levels, we select eight distinct game modes from the available configurations, each modifying traffic patterns and vehicle behaviors:
Mode 0 (Default). Default configuration with standard traffic density and vehicle speeds.
Additional details available wttttps://aaee wttpt//////aaaa......oo/aoaaaaaa www.aa..w.rgaav...oooettttteerrr
19

---

Mode 1 (Increased Traffic & Trucks). Increased traffic density with faster vehicles. Introduces trucks in the upper lane closest to the center these longer vehicles require more strategic avoidance.
Mode (High-Speed Trucks). Enhanced difficulty from Mode I with trucks moving at higher speeds and further increased traffic density.
Mode (All-Lane Trucks). Maximum truck presence with trucks appearing in all lanes, maintaining the high speeds established in Mode 2.
Mode (Randomized Speeds). Dynamic speed variation where vehicle velocities change randomly during episodes, while maintaining similar traffic density to previous modes without trucks.
Mode (Clustered Vehicles & Speed Variability). Combines characteristics of Mode with additional stochastic elements: vehicle speeds vary dynamically and some vehicles appear in tightly- spaced clusters (2-3 vehicles).
Mode 6 (Heaviest Traffic with Clusters). Builds upon Mode 5 with the most dense traffic configura- tion, creating the most challenging navigation scenario.
Mode 7 (All-Lane Trucks with Random Speeds), All lanes are filled with trucks, and their speeds vary randomly during the episode.
D Implementation Details
To ensure fairness and reproducibility, we build upon the official implementations released by CompoNet [32], which provide well-tested baselines for standard reinforcement learning algorithms. Our modifications to the original SAC and PPO codebases are kept minimal: we only substitute agent definition with proposed architecture and introduce the corresponding learning mechanisms.
Both Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) follow the standard actor- critic paradigm, where the actor samples actions from a parameterized policy distribution and the critic estimates state values. In all continual reinforcement learning (CRL) methods, the continual adaptation mechanisms are applied only to the actor network, while the critic is reinitialized at the beginning each task, following common practice the literature.
Table 5: Hyperparameters shared by all methods in the algorithm.
task sequence under the SAC
Description
Value Adam (0.9, 0.999)
Optimizer
Adam's 31 and B3
Discount rate (7)
0.99 e²
Common
Max Std.
Min Std.
e 20
Activation Function
ReLU
Hidden Dimension (dmode)
256
Batch size
128
Buffer size
106
Target Smoothing Coef. (r)
0.005
Entropy Regularization Coef. (a)
0.2
Auto. Tuning of a
Yes
SAC Specific
Policy Update Freq.
Target Net. Update Freq.
Noise Clip
0.5
Number of Random Actions
10
Timestep to Start Learning
5 x 1
Target Net. Layers
Critic Net. Layers
Networks
Actor's Learning Rate
10 10-3
Q Networks' Learning Rate
10 10-3
20

---

For Meta- World tasks, we adopt SAC as the underlying optimization algorithm. Both the actor and critic networks are implemented two-layer multilayer perceptrons (MLPs), each followed by separate linear output heads predicting the mean and log standard deviation of the Gaussian policy.
For Atari- tasks including Spacelnvaders and Freeway, PPO is used for training. All methods employ shared encoder network to extract compact feature representations from image observations. Two single-layer output heads are used produce the categorical policy logits (actor) and the scalar value estimates (critic). Unless otherwise noted, all hyperparameters are kept identical across different methods and are consistent with those in the reference implementations.
Table 6: Hyperparameters shared by all methods in the SpaceInvaders and Freeway task sequences under the PPO algorithm.
Description
Value
Optimizer
Adam
AdamW's B1 and B2
(0.9, 0.999)
Max. Gradient Norm
0.5
Common
Discount Rate (?)
0.99
Activation Function
ReLU
Hidden Dimension (dmmste)
512
Learning Rate
2.5 10
PPO Value Function Coef.
0.5
GAE À
0.95
Num. Parallel Environments
Batch Size
1024
Mini Batch Size
256
Num. Mini Batches
PPO Specific
Update Epochs
PPO Clipping Coefficient
0.2
PPO Entropy Coefficient
0.01
Learn. Rate Annealing
Yes
Clip Value Loss
Yes
Normalize Advantage
Yes
Num. Steps Per Rollout
128
Methods, We compare the CKA-RL with nine SOTA methods. 1) Baseline involves training a randomly initialized neural network for each task, providing fundamental and essential reference point for comparison. 2) FT-1 (Fine-Tuning Single Model) [53] continuously fine-tunes a single neural network model across all relevant tasks. 3) FT-N (Fine-Tuning with Model Preservation) (53 follows a similar fine-tuning approach but maintains separate model instances for each task to mitigate forgetting. 4) Net (39] instantiates a new neural network whenever the task changes, freezing the parameters of the previous modules and adding lateral connections between their hidden layers. 5) PackNet [33] stores the parameters to solve every task of the sequence in the same network by building masks to avoid overwriting the ones used to solve previous tasks. 6) MaskNet [5] leverages previous score parameters across tasks learn task-specific score parameters, which are then used to generate masks for the neural network, dynamically adapting the model to new tasks. 7) CReLUs [ enhances the agent capability CRL by concatenating ReLUs, which reduces the incidence of zero activations and addresses the plasticity loss issue in neural networks. 8) CompoNet 32 introduces a scalable neural network architecture that dynamically composes action outputs from previously learned policy modules, rather than relying on shared hidden layer representations across tasks. 9) ChpNet [IT] uses a variation of prok-propagation, continually and randomly reinitializing small fraction of underutilized units to maintain the plasticity of neural networks.
21

---

Parameter Memory (MBB
100
CompoNet
Activate Memory Usaiee (MBB
300
CompoNet
ProgNet
250
ProgNet
80
CKA-RL
CKA-RL
200
150
40
100
20
2
8
Task ID
Task ID
(a) Total parameter memory tasks. CKA-RL plateaus after tasks, while others grow linearly.
(b) Activation related memory usage during train-
ing. CKA-RL remains flat, whereas ProgNet shows quadratic growth.
Figure 9: Memory cost comparison. CKA-RL maintains nearly constant parameter and activation size through vector merging, while baselines show linear or even quadratic memory growth.
E Efficiency of CKA-RL
E.1
Performance vs. Memory Cost Analysis
We evaluate the memory efficiency of CKA-RL by comparing the growth of model parameters and activation-related memory across tasks from Spacelnvaders. As shown in Figure 9a the total parameter memory of CKA-RL remains nearly constant beyond the fifth task due our use of the adaptive knowledge merging, which maintains a bounded knowledge vector pool. This ensures that model complexity does not scale linearly with the number of tasks. In contrast, both ProgNet and CompoNet show linearly increasing memory usage, reaching 64.95 MB and 108.21 MB at the tenth task, respectively.
As shown in Figure 96] the activation-related memory usage CKA-RL stays nearly constant throughout training. After merging, the knowledge vectors and base parameter construct fixed-size policy parameter, which drastically reduces the activation memory overhead. Notably, the memory usage in ProgNet exhibits a quadratic growth trend, whereas CKA-RL remains constant. This suggests severe scalability limitations in ProgNet. This memory overhead in ProgNet primarily stems from its architectural design: during training and inference, relies on the hidden representations of all previously learned policies. As more tasks are added, the number of such dependencies increases, leading to substantial growth in activation memory consumption.
Table 7: Performance and Forward Transfer Comparison. CKA-RL outperforms existing methods across three benchmarks while using significantly less memory.
Meta-World
SpaceInvaders PERF
Freeway PERE
Average PERE
Method
Pub. Year
PERF
FWT.
FWT.
FWT.
FWT.
ProgNet 39
0.4157
-0.0379
0.3757
-0.0075
0.3125
0.1938
0.3680
0.0495
CompoNet 32
ICML'2024
0.4131
-0.0055
0.9828
0.6963
0.7629
0.7115
0.7196
0.4674
CKA-RL (Ours)
0.4642
-0.0032
0.9928
0.7749
0.7923
0.7429
0.7498 0.5049
CKA-RL achieves a forward transfer of 0.7749 with only 11.49MB of memory overhead, highlighting its superior efficiency. Unlike methods that accumulate task-specific modules, approach leverages compact knowledge vectors to consolidate transferable knowledge. This design allows CKA-RL to outperform CompoNet and ProgNet with significantly smaller memory footprint.
As shown in Table 7 CKA-RL not only achieves the best average performance and forward transfer across diverse tasks (Meta-Worke Spacelnvaders, Freeway). but also maintains high efficiency in historical knowledge utilization. These results underscore the practicality of CKA-RI. in continual learning scenarios with constrained memory budgets.
22

---

E.2 Performance vs. Inference Cost Analysis
We further compare the inference efficiency of different methods on the Spacelnvaders bench- mark. shown in FigurefTO, CKA-RI. achieves the highest performance and forward transfer
0.009
CompaNet
ProgNet
0.0.07
CKA-RL
with an average inference time of only 0.0012s
Inference Time
0.006
per imput. Importantly, its inference latency re- mains almost constant regardless of the number of tasks. This is because CKA-RL consolidates historical knowledge into a fixed-size policy pa- rameter during the parameter construction phase, eliminating the need for complex runtime com-
0.005
0.004
0.002
0.001
0
4
6
8
Task
position.
Figure 10: Inference time number of tasks on SpaceInvaders. CKA-RL maintains nearly constant inference latency due to adaptive knowl- edge merging, while CompoNet and ProgNet suf- fer from increasing computational overhead.
In contrast, existing architectures such as Com- PoNet and ProgNet exhibit significant increases in inference cost as the number tasks grows. CompoNet shows linear growth trend in in- ference time, since its action selection relies on aggregating outputs from all previously learned policies. ProgNet, the other hand, demonstrates a quadratic growth pattern due to dependence on all intermediate hidden layers for knowledge integration during inference.
These results underscore the scalability advantage of CKA-RI. By offloading the knowledge fusion process the model-buildin stage, it minimizes computational overhead test time, making well-suited for continual learning in resource-constrained or real-tima environments.
F Further Experimental Results
1.0
0.00
0.00
0.00
0.00
0.00
0.00 0.00 0.00 0.00
1.00 0:95 0.94 0.94
0.94
0.94
0.94
0.94
0.94
-0.01
0.8
1.00
0.93
0.8
0.00
-024 1.00
0.13
00044
0.00
0.06
-0001 0.00
Swosnnd Modal Pirameter
0.94 0:94 1.30 0.84
0.83
0.93
0.93
0.83
0.84
0.83
0.8
Secand Knowledge
0.4
0.00
-0.04
0.04
00.05
1.00
0.05
0.12
0.09
0.06 -0.03
*
0.94 0.83 0.89
0.94 1.00
0.93 0.84
0.83
0.94
0.83
0.00
-0.01 40
-0.08
0.05
1.00
0.12 0.11
0.01 -0.03
0.94 0.98 0.93 0.94 93
0.93
0.98
0.93
0.93
0.08
0.12
0.93
1.00
0.93
0.00 0.07 0.06
0.06 0.09
0.11
0.15
1.00
0.07
+0.06
0.94 0.93 0.93 0.94 0.83 0.93 0.94
1,00
0.94
0.94
-.2
0.01
0:94
0:94
1.00
0.94
0.00 -0.08 0.00 -0.04 0.00 -0.0 0.000.00000000 00.04
0.94 0.84 0.94 8.99 0.93 0.80
0.94
1.00
04
6
Knowlodge
Madel Parameter
(a) Knowledge vector similarity (from CKA-RL)
(b) Full parameter similarity (from FN)
Figure 11: Cosine similarity of task representations: (a) Knowledge vectors from CKA-RL exhibit oethoortl structure across tasks, with off- off-diagonal values ranging from -0.24 to 0.12; the zero vector for the first task reflects the base model initialization. (b) In contrast, full parameters from standard fine-tur meeetuning show strong off-diagona correlations (0.93 ~ 0.95), indicating significant parameter overlap and interference across tasks.
Our CKA-RL decomposes policy parameters into stable base weights dewse and task-specific knowl- edge vectors { To validate this decomposition, we compute pairwise cosine similarities between all parameters of models fine-tuned on different tasks (Figure Tīb). The strong off-diagonal correla- tions (0.93 0.95) reveal that standard sequential fine-tuning causes substantial parameter overlap. Then we analyze cosine similarities between our learned knowledge vectors e; (Figure TTap. The near-zero off-diagonal values (-0.24 0.12) demonstrate that knowledge vectors occupy nearly
23

---

orthogonal directions in parameter space. The first task's 's zero-valued vector reflects the initial base model training phase.
This decomposition brings two key advantages: (1) stability, by preserving the base parameters t tasks, and (2) plasticity, through task-specific knowledge vectors that remain nearly orthogonal. The orthogonality ensures minimal interference between tasks while enabling effective adaptation. As result, CKA-RL achieves forward transfer of 0.7 0.7749 and an average performance of 0.9928 across tasks in continual learning settings, outperforming standard fine-tuning (0.6864 and 0.9785, respectively).
G Discussions and Future Works
G.1 Limitations and Future Works
In this paper, we propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowledge. However, we believe that there are potential studies worth exploring in the future to further capitalize on the advantages of CKA-RL:
Complex-En Environment Evaluation: Our current experiments focus on standard continual reinforcement learning benchmarks, and the scalability and robustness of CKA-RL in complex, real-world settings (e.g., -igee-iinssional visual perception or long-hooe horizon robotic control) remain unverified. In future work, we will extend our evaluation to domains with richer dynamics and observation modalities (e.g.. outdoor vision-language navigation tasks) to rigorously assess the generality and practical utility of CKA-RL.
Large-Seale Architecture Generalization: CKA-RI has been validated only on small-scale neural networks, and its applicability to deeper or novel architectures remains untested, In future work, we will evaluate CKA-RL within large-scale models, such during the RLHF phase of LLMs training, to assess its scalability and effectiveness in deeper networks.
G.2 Broader Impacts
Positive Societal Impacts. The continual knowledge adaptation mechanism introduced by CKA-RL can substantially improve the data- and compute-efficiency autonomous systems that must operate in non-stationary environments. By enabling robots, intelligent assistants, and other agents to rapidly integrate new skills without repeatedly retraining from scratch, our method can reduce energy consumption and carbon footprint associated with large-scale model updates.
Negative Societal Impacts. As with any advanced continual learning technique, there is risk that CKA-RL could be exploited to develop adaptive adversarial agents that continuously learn to evade detection or defenses.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which enables the accumulation and effective utilization of historical knowl- edge.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The details can be seen in the Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The details can be seen in Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The source code available at
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "nipsscc/puullc//uuu hotp:///ggiddeeeeeouuuuumbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The details can be seen in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Table
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
27

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details can be seen in Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The paper meets the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification:
The details can be seen Appenidx
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
28

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We strictly follow the license of the assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
29

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
30

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
31


Reviews:
[{"review_id": "rGYwzgdaiW", "reviewer": "Reviewer_Mgpw", "summary": "This paper introduces Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), a new method for continual reinforcement learning (CRL) designed to mitigate catastrophic forgetting and improve knowledge transfer while maintaining scalability. The core idea is to represent task-specific knowledge as \"knowledge vectors.\" To address the challenge of ever-growing memory costs, the authors propose an \"Adaptive Knowledge Merging\" mechanism, which identifies and merges the most similar pairs of knowledge vectors in a fixed-size pool, thus keeping the memory footprint bounded.", "questions": "For related work, could the authors also discuss more recent crl specific papers like 1. Prediction and Control in Continual Reinforcement Learning. Anand et al. 2. Knowledge Retention for Continual Model-Based Reinforcement Learning. Sun et al. etc. ?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Important problem and the algorithm generally makes sense.\n2. The analysis in Figure 5 and Appendix D clearly demonstrates the superior memory and inference efficiency of CKA-RL.\n\nWeaknesses:\n1. Significance of Empirical Results is Unclear: My biggest concern is the lack of statistical rigor in the experimental validation. The performance improvements reported in the main results (Table 1) appear to be quite marginal (e.g., ~1-4% in many cases). The authors state in the checklist that results are from a single run with a fixed seed and do not report standard deviations. For an empirical RL paper, this is a critical omission. Without error bars over multiple runs, it is impossible to determine if these small advantages are statistically significant or simply the result of run-to-run variance. \n\n2. Moderate Methodological Novelty: The core idea of decomposing model updates into task-specific vectors is heavily inspired by prior work in model editing and task arithmetic. While its application to CRL is new, the fundamental mechanism is not", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful to you for recognizing the strengths of our work, particularly the clear presentation of our methodology and the balanced trade-off CKA-RL achieves among plasticity, stability, and scalability. We sincerely appreciate your acknowledgment of our comprehensive experimental validation and the potential impact of our approach on the continual reinforcement learning community.\n\n>Q1. Significance of Empirical Results is Unclear: My biggest concern is the lack of statistical rigor in the experimental validation. The performance improvements reported in the main results (Table 1) appear to be quite marginal (e.g., ~1-4% in many cases). The authors state in the checklist that results are from a single run with a fixed seed and do not report standard deviations. For an empirical RL paper, this is a critical omission. Without error bars over multiple runs, it is impossible to determine if these small advantages are statistically significant or simply the result of run-to-run variance.\n\n**A1.** Thank you for your valuable comment regarding the statistical rigor of our experimental evaluation. We will address this in the revised version, as detailed below:\n* **Multiple Random Seeds Used.** Our experiments were conducted with 10 different random seeds, consistent with standard practice in the recent CRL method CompoNet. We acknowledge that this detail was not explicitly stated in our initial submission, which was an oversight. \n* **Statistical Significance Evidence.** We have computed and included standard deviations for all reported results. The updated results table (provided in Tab. A) now shows mean performance ± standard deviation across all task sequences and methods. Notably, CKA-RL consistently demonstrates lower variance compared to baselines.\n* **Contextualizing the Magnitude of Improvements.** While the percentage improvements may appear modest numerically, they represent meaningful advances in the context of continual RL. 1) The field of CRL has seen diminishing returns in performance gains, with recent SOTA methods (CompoNet, CbpNet) showing only average 1-3% improvements over prior work (CReLUs); 2) Our method achieves these gains while using significantly less memory (about 5.8MB vs CompoNet's about 108.21MB at the 10th task).\n\n\nTab. A: Summary of results across all task sequences and methods.\n|METHOD|META-WORLD PERF. |META-WORLD FWT.| SPACEINVADERS PERF.|SPACEINVADERS FWT.| FREEWAY PERF. |FREEWAY FWT.|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|ProgNet|0.4157$\\pm$ 0.49|-0.0379$\\pm$ 0.13 |0.3757$\\pm$ 0.27|-0.0075$\\pm$ 0.22|0.3125$\\pm$ 0.27 |0.1938$\\pm$ 0.31|\n|PackNet|0.2523$\\pm$ 0.40|-0.6721$\\pm$ 1.40|0.2299$\\pm$ 0.30|-0.075$\\pm$ 0.13|0.2767$\\pm$ 0.36| 0.1970$\\pm$ 0.32|\n|MaskNet|0.3263$\\pm$ 0.47 |-0.3695$\\pm$ 0.45|0$\\pm$ 0 |-0.3866$\\pm$ 0.53|0.0644$\\pm$ 0.17|-0.0503$\\pm$ 0.12|\n|CReLUs|0.3789$\\pm$ 0.47|-0.0089$\\pm$ 0.24|0.8873$\\pm$ 0.10|0.5308$\\pm$ 0.29|0.7835$\\pm$ 0.13|0.7303$\\pm$ 0.12|\n|CompoNet|0.4131$\\pm$ 0.50|-0.0055$\\pm$ 0.20|0.9828$\\pm$ 0.02|0.6963$\\pm$ 0.32|0.7629$\\pm$ 0.12|0.7115$\\pm$ 0.10|\n|CbpNet|0.4368$\\pm$ 0.50 |-0.0826$\\pm$ 0.22|0.8392$\\pm$ 0.11|0.4844$\\pm$ 0.28|0.7678$\\pm$ 0.10|0.7201$\\pm$ 0.07|\n|CKA-RL|**0.4642 $\\pm$ 0.50** |**-0.0032$\\pm$ 0.21**|**0.9928 $\\pm$ 0.01**|**0.7749$\\pm$ 0.20**|**0.7923$\\pm$ 0.10**|**0.7429$\\pm$ 0.07**|\n\n\n>Q2. Moderate Methodological Novelty: The core idea of decomposing model updates into task-specific vectors is heavily inspired by prior work in model editing and task arithmetic. While its application to CRL is new, the fundamental mechanism is not.\n\n**A2.** Thank you for your thoughtful comment regarding the methodological novelty of our work. We would like to clarify the substantive contributions of CKA-RL beyond its conceptual inspiration from model editing techniques.\n* **Acknowledgment of Inspiration.**  You are correct that our knowledge vector concept is inspired by prior work in model editing and task arithmetic, as explicitly acknowledged in Section 4.1 of our paper. \n* **Fundamental Novelty in CRL Context.**  The core contribution of CKA-RL lies not merely in applying this concept to continual reinforcement learning, but in developing a complete framework specifically designed for the unique challenges of CRL.\n    * **Dynamic Knowledge Adaptation Mechanism** Unlike static model merging approaches in supervised learning that typically combine pre-trained models with fixed weights, our Continual Knowledge Adaptation strategy dynamically determines how to utilize historical knowledge through learnable adaptation factors. This allows the agent to flexibly select and weight relevant knowledge based on the current task, which is critical in non-stationary RL environments where task relationships are complex and evolving.\n    * **Adaptive Knowledge Merging for CRL.** Our merging mechanism (Eqn. (8)) is specifically designed for the sequential nature of continual RL, where tasks arrive sequentially rather than simultaneously. The similarity metric and merging strategy we propose are tailored to the unique characteristics of RL knowledge transfer, considering both parameter similarity and performance impact.\n\n\n>Q3. For related work, could the authors also discuss more recent crl specific papers like 1. Prediction and Control in Continual Reinforcement Learning. Anand et al. 2. Knowledge Retention for Continual Model-Based Reinforcement Learning. Sun et al. etc. ?\n\n**A3.** Thank you for your valuable suggestion regarding the inclusion of more recent CRL-specific papers in our related work section. We appreciate this insightful observation and will incorporate these important works in our revision. Below is our detailed response:\n* **Inclusion of Suggested Works.** We sincerely thank you for bringing \"Prediction and Control in Continual Reinforcement Learning\" by Anand et al. and \"Knowledge Retention for Continual Model-Based Reinforcement Learning\" by Sun et al. to our attention. These represent significant recent contributions to the CRL field that were inadvertently omitted from our initial submission.\n* **Comparison with Anand et al.'s Work.** Anand et al. propose decomposing the value function into permanent and transient components that update at different timescales. While their approach shares conceptual similarities with our work in addressing the stability-plasticity dilemma, there are fundamental differences: Anand et al. focus on value function estimation, while CKA-RL operates directly on policy parameters. Our knowledge vector approach decomposes policy parameters into stable base weights ($θ_{base}$) and task-specific knowledge vectors ($v_i$), rather than manipulating value functions.\n* **Relationship to Sun et al.'s DRAGO.** Sun et al.'s DRAGO framework introduces innovative mechanisms for knowledge retention in model-based CRL. While highly relevant, our approach differs in scope aspects: DRAGO specifically targets model-based RL where tasks share state spaces and dynamics but differ in rewards, while CKA-RL is designed for both model-free and model-based settings with broader applicability across diverse task distributions.\n\n***\nWe sincerely hope our clarifications above have addressed your concerns. We would be grateful if you could kindly reconsider the evaluation of our paper."}, {"review_id": "tmPyLzQB4k", "reviewer": "Reviewer_qLPG", "summary": "This paper aims to address catastrophic forgetting and inefficient knowledge utilization in continual reinforcement learning (RL). The authors propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), a continual RL algorithm. CKA-RL maintains a pool of task-specific knowledge (policy’s model parameters) from previously learned tasks and leverages this knowledge to adapt an agent to a new task. To reduce the memory accumulated by historical knowledge, CKA-RL also includes an adaptive knowledge merging mechanism that combines similar knowledge vectors in the pool. \n\nCKA-RL is compared with other methods across Meta-World, SpaceInvaders, and Freeway, and outperforms them in terms of average performance and forward transfer. CKA-RL is also more efficient in terms of memory usage and inference time when evaluated on SpaceInvaders.", "questions": "1. Since the model parameters of the first task will be set as the base parameter, does the choice of this first task matter?  In other words, does the sequence of tasks affect the results? \n2. (Section 4.2) “For flexibility, we define the null knowledge as…”  What does “flexibility” mean here, and why does defining the null knowledge improve the flexibility? \n3. How are betas updated? \n4. (Section 4.2) “By setting alpha_1 = 1, the model can disregard previously learned knowledge when it is not beneficial for learning new tasks.”  Why? \n5. Do you store the betas learned in different tasks? If not, how do you restore the policies for the tasks that have already been learned? \n6. (Table 2) Why does the performance, not only the memory efficiency, improve after adding the adaptive knowledge merging mechanism? \n7. (Section 5.4) Why does CKA-RL lead to superior initial performance and rapid convergence? \n8. How many random seeds did you run for each experiment?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths\n1. The paper is written with a clear flow, and one can easily catch the high-level idea of the proposed method. \n2. The idea of including a knowledge merging mechanism is crucial, in my opinion, for scalable continual RL algorithms. \n3. The experimental results show that CKA-RL consistently outperforms other evaluated methods in terms of average performance and forward transfer. Some ablation studies are also provided to help understand the effectiveness of the two components in CKA-RL. Especially, I found Figure 11 very interesting, as it demonstrates how distinct each knowledge vector is.\n\n# Weaknesses\n1. The motivation and design of CKA-RL are not sufficiently elaborated. \n- Knowledge vectors: In Continual RL, various forms of knowledge can be transferred across tasks, such as policies, feature representations, reward functions, and reasoning models. In this paper, the knowledge vectors are specifically designed to be the “residual model parameters” of a policy. It would be easier to follow if the authors could explain why they chose this specific design.  \n- The normalized adaptation factors, alphas and betas: After reading the paper, I still have a hard time understanding how these parameters are learned. Specifically, how are betas updated?  \n- The similarity between knowledge vectors: The paper proposed that the normalized cosine similarity of two vectors of residual policy parameters can represent the similarity between two policies. However, I believe there is a difference between the representation and functional similarity of neural networks. Thus, even if the cosine similarity is close to 1, it does not guarantee that the two policies perform similarly. \n- Following the previous point, simply merging two knowledge vectors can result in new knowledge with very different behaviors. \n2. Ablation studies can be more comprehensive. \n- Effects of knowledge pool size: This analysis is conducted only on Freeway, not on other environments. In addition, the size evaluated only ranges between 3 and 7, which is relatively small and may not necessarily generalize to different scales of pool size. \n- Reward curve comparison in Figure 4: Which environment is this analysis conducted in? \n- Knowledge Retention of CKA-RL: Without statistical significance results, Table 3 does not necessarily indicate that CKA-RL outperforms other baselines in mitigating catastrophic forgetting, as the performance difference is not substantial. \n- Memory efficiency and inference efficiency: These analyses are only performed on SpaceInvaders, and the results for other environments are not provided. \n3. The statistical significance results are not presented in the paper. Specifically, the number of random seeds used for training is not specified, and most figures and tables do not provide the standard deviation of the results. \n4. In the Introduction, the paper mentions that CKA-RL aims to address cross-task conflict; however, after reading the paper, I do not find that CKA-RL can address conflicts between tasks.", "was_revised": true, "final_justification": "The authors have provided additional math insights, experiments, and analyses, which help with understanding the underlying mechanisms of their methods. I found the merging mechanism and its results interesting and decided to raise my rating accordingly.", "rebuttal": ">Q1. Why did you choose residual model parameters for knowledge vectors?\n\n**A1.** Thank you for this insightful comment regarding our knowledge vector design. Clarify as follows:\n* **Optimal Stability-Plasticity Balance [18].** Unlike full policy parameters exhibiting high cross-task correlation, residual vectors $v_k$ capture task-specific adaptations in **nearly decoupled directions** (Fig. 11a, similarity: −0.24–0.12). This minimizes interference while enabling effective transfer, with $\\theta_{base}$ providing stability and $v_k$ enabling plasticity. \n* **Comprehensive Policy Adaptation:** The residual vector compactly encodes all necessary adjustments to the base policy’s behavior, as validated by near-perfect performance on complex tasks. \n* **Scalability Through Merging:** Fig. 5a proves our method maintains **constant memory** (11.49MB after 10 tasks) while CompoNet requires 108.21MB.\n\n>Q2. How are betas updated?\n\n**A2.** Thank you for your insightful comment on the learning mechanism of adaptation factors. The $\\beta$ are directly optimized during the training of each new task through standard gradient-based methods. Specifically:\n* For each new task, we initialize a set of learnable parameters $\\beta_k=[\\beta_1^k,\\ldots,\\beta_{k-1}^k]$.\n* These $β^k$ parameters are directly optimized alongside the current task knowledge vector $v_k$ during the reinforcement learning process.\n* The gradients for $β^k$ come from the RL objective function, just like gradients for policy parameters.\n\n>Q3. Design of similarity between knowledge vectors. \n\n\n**A3.** Thanks for your comment. Clarify as follows:\n* **Cosine Similarity vs. Functional Similarity.**  Cosine similarity in parameter space does not directly capture the functional similarity between policies. However,  task (or knowledge) vectors [18] are defined as the difference between fine-tuned and base models, exhibit meaningful structure: near-orthogonal vectors often correspond to disjoint tasks, and similar vectors often lead to shared behaviors. While not a perfect proxy, cosine similarity remains a practical measure of representation similarity, which correlates with functional behavior in many settings [18].\n* **Empirical results.** The high cosine similarity between certain knowledge vectors suggests shared structure, and we confirm this with empirical results:\n    * Our merging mechanism preserves performance across tasks, showing no degradation in previously learned policies.\n    * From Fig. 11, the near-orthogonality of learned vectors helps reduce interference, supporting functional retention.\n\n>Q4. Effects of knowledge pool size.\n\n**A4.** Thanks for your insightful comment. **Please see Reviewer-dAWz, A6**.\n\n>Q5. Which environment in Fig. 4?\n\n**A5.**  Fig. 4  is in the **SpaceInavders environment**.\n\n>Q6. The standard forgetting metric.\n\n**A6.** Thanks for your insightful comment. **Please see Reviewer-33rx, A4**.\n\n>Q7. Memory efficiency and inference efficiency.\n\n**A7.** Thanks. We have conducted thorough memory and inference efficiency evaluations on both Meta-World and Freeway environments. From Tabs A&B, CKA-RL maintains nearly constant parameter size and inference latency through vector merging, while other methods show linear growth and increasing computational overhead.\n\nTab. A:  Analysis in the Meta-World.\n||Task ID|5|10|50|150|200|300|\n|-|-|-|-|-|-|-|-|\n|Parameter Memory (MB)|ProgNet|5.64|17.22|334.79|2878.74|5088.21|11382.16|\n||CompoNet|7.18|14.43|72.48|217.60|290.17|435.29|\n||CKA-RL|5.64|10.34|10.34|10.34|10.34|10.34|\n|Inference latency (s)|ProgNet|0.0010|0.0021|0.0315|0.2558|0.4610|1.0353|\n||CompoNet|0.0023|0.0045|0.0213|0.0636|0.0842|0.1264|\n||CKA-RL|0.0002|0.0001|0.0002|0.0003|0.0002|0.0002|\n\nTab. B: Analysis in the Freeway.\n||Task ID|0|1|3|5|7|\n|-|-|-|-|-|-|-|\n|Parameter Memory (MB)|ProgNet|7.44|13.82|26.61|39.39|52.17|\n||CompoNet|13.86|24.35|45.31|66.28|87.25|\n||CKA-RL|7.44|8.45|10.48|12.51|14.54|\n|Inference latency (s)|ProgNet|0.0018|0.0017|0.0027|0.0041|0.0061|\n||CompoNet|0.0015|0.0019|0.0032|0.0046|0.0035|\n||CKA-RL|0.0015|0.0015|0.0015|0.0015|0.0015|\n\n\n>Q8. The statistical significance results.\n\n**A8.** Thanks for your valuable feedback. **Please see Reviewer-Mgpw, A1**.\n\n\n>Q9. Why can CKA-RL address conflicts between tasks?\n\n**A9.** Thanks. Cross-task conflict in CRL primarily manifests as parameter overlap and interference. CKA-RL mitigates this through the knowledge vector.\n* **Orthogonal knowledge vectors.** From Fig. 11a, CKA-RL’s task-specific knowledge vectors exhibit near-orthogonal structures in parameter space. This minimizes representational overlap between task-specific adaptations, directly reducing the potential for conflict when learning new tasks.\n* **Stable base parameters.** The fixed base parameter preserves general knowledge across tasks, ensuring that foundational abilities are not overwritten by task-specific updates. This separation prevents new task parameters from disrupting core knowledge needed for earlier tasks.\n* **Adaptive knowledge weighting.** $\\alpha_k$ dynamically adjusts the contribution of historical knowledge vectors to new tasks, allowing the model to prioritize relevant historical knowledge while suppressing conflicting information.\n\n>Q10. Task Sequence Impact.\n\n**A10.** Thanks for your comment. Clarify as follows:\n* **Task Sequence Impact.** The choice of the first task as base parameter does matter. Results confirm that different task sequences lead to performance variations (about 3.7% difference in performance). \n* **Robustness Analysis.** Despite this sensitivity, CKA-RL demonstrates greater robustness to task ordering than methods like CompoNet, as shown in Tab. C.\n\nTab. C: Different task orders on the Freeway.\n|Order|METHOD| FREEWAY PERF. |FREEWAY FWT.|\n|-|-|:-:|:-:|\n|Standard|CompoNet|0.7629$\\pm$ 0.12|0.7115$\\pm$ 0.10|\n||CKA-RL|**0.7923$\\pm$ 0.10**|**0.7429$\\pm$ 0.07**|\n|Shuffle|CompoNet|0.7066$\\pm$ 0.19|0.6515$\\pm$ 0.18|\n||CKA-RL|**0.8294$\\pm$ 0.11**|**0.7704$\\pm$ 0.10**|\n\n\n>Q11. Why does defining the null knowledge improve the flexibility?\n\n**A11.** Thanks. The term \"flexibility\" **refers to the unified mathematical formulation** of knowledge adaptation across all tasks, including the first task $\\tau_1$. By defining $v_1 =\\theta_1 - \\theta_{base} = 0$ and including it in the knowledge pool $\\mathcal{V}$. In Eqn.(4), the adaptation factors $\\alpha_j^k$are derived from learnable parameters $\\beta^k$ through softmax. When $v_1 = 0$, the value of $\\alpha_1^k$ becomes irrelevant to the final policy parameter, **allowing the model to \"learn\" to ignore historical knowledge when it's not beneficial.**\n\n\n>Q12. “The model can disregard previously learned knowledge when it is not beneficial for learning new tasks.” Why?\n\n**A12.** Thanks. This statement refers to the role of the null knowledge vector $v_1$ in our framework. \n* $\\alpha_1$ is the adaptation factor corresponding to $v_1$, the null knowledge vector. Since the sum of all $\\alpha^k_j$ (j = 1, ..., k-1) is constrained to 1, setting $\\alpha_1 = 1$ forces all other factors $\\alpha^k_2, ..., \\alpha^k_{k-1}$ to be 0. \n* Because $v_1 = 0, \\alpha_1 = 1$ means the model’s parameter update for the new task $\\tau_k$ depends **only** on the base parameter and the current task’s knowledge vector, with no contribution from other historical knowledge vectors $v_2, ..., v_{k-1}$. In this case, the model effectively disregards previously learned knowledge stored in these vectors. This design ensures the model avoids over-reliance on unhelpful historical knowledge.\n\n>Q13. Do you store learned betas and how are policies restored?\n\n**A13.** Thanks. We do not store the $\\beta^k$ parameters. \n* **Role of $\\beta^k$ Parameters.** The $\\beta^k$ serve as temporary, task-specific variables during the training of a new task. Their role is to dynamically determine the adaptation factors $\\alpha^k$ that weight historical knowledge vectors, facilitating the acquisition of the current task’s knowledge vector. \n* **Restoring Policies for Previously Learned Tasks.** To restore the policy for a previously learned task $\\tau_i$, we rely on the stored knowledge vector $v_i$ and the fixed base parameter $\\theta_{base}$. The policy parameter is defined as: $\\theta_i = \\theta_{base} + v_i$.  Since $v_i$ encapsulates all task-specific adaptations needed for $\\tau_i$, restoring the policy only requires retrieving $v_i$ from the knowledge pool and combining it with $\\theta_{base}$. \n\nQ14. Why does performance improve with the adaptive knowledge merging mechanism?\n\n**A14.** Thanks. Clarify as follows: \n* **Reducing interference from conflicting knowledge.** By merging redundant or conflicting vectors, the model avoids carrying irrelevant or contradictory information into new task learning. This minimizes \"noise\" in knowledge transfer, allowing the model to focus on truly useful historical insights.\n* **Preserving and amplifying critical information.** The merging process retains essential task-specific knowledge while consolidating similar vectors into more generalized representations. This strengthens the signal of transferable knowledge, enabling more effective adaptation to new tasks.\n\n>Q15. Why does CKA-RL lead to superior initial performance and rapid convergence?\n\n**A15.** Thanks. The reason is that CKA-RL ability to leverage high-quality historical knowledge as a useful prior while minimizing interference during new task learning. \n* **Effective reuse of historical knowledge.** CKA-RL initializes the current task’s knowledge vector to 0 and dynamically weights relevant historical vectors via $\\alpha^k$, directly boosting initial performance. \n* **Refined knowledge signals from merging.** The adaptive knowledge merging mechanism consolidates redundant vectors, strengthening the signal of transferable knowledge. This ensures the historical knowledge used to initialize new tasks is more relevant and less noisy, further speeding up learning."}, {"review_id": "bZILnSMIqW", "reviewer": "Reviewer_33rx", "summary": "This paper addresses the challenges of catastrophic forgetting and inefficient knowledge utilization in reinforcement learning. The authors propose Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), a framework designed to facilitate the accumulation and effective reuse of historical knowledge across tasks. Specifically, CKA-RL maintains a task-specific knowledge vector pool and leverages these historical knowledge representations to better adapt to new tasks. To address scalability concerns, the authors introduce an Adaptive Knowledge Merging mechanism, which combines similar knowledge vectors to reduce redundancy. Experimental results across three benchmarks show that CKA-RL outperforms state-of-the-art methods, achieving a 4.20% improvement in overall performance and an 8.02% gain in forward transfer.", "questions": "1.\tWhy the performance of CompoNet in this paper appear significantly lower than reported in its original publication?\n\n2.\tAccording to Equation (3), the task-specific parameter vectors are expected to be of the same dimensionality as the base model parameters $\\theta_{\\text{base}}$. Given this, is the overall parameters of the model the form $y = \\theta_{\\text{base}} x$ until the size of the knowledge pool in Figure 3?\n\n3.\tSince mitigating catastrophic forgetting is one of the central motivations of this work, it is surprising that the standard forgetting metric is not reported in Table 1.\n\n4.\tThe proposed method uses a simple weighted sum of historical task vectors to mitigate catastrophic forgetting and enable knowledge transfer. However, this raises several concerns: \n\n  a)\tFor catastrophic forgetting, the weighted sum of history vectors may not be robust to conflicting knowledge across tasks. For instance, if a previously learned task vector $v_i$ is incompatible with another vector $v_j$ (due to opposite objectives or gradients), the resulting combined vector could destructively interfere with prior knowledge. Since the weights are computed via a softmax over hyperparameters, it is impossible to assign exclusive weight (e.g., $\\alpha_1 = 1$) to a single vector, making such interference likely. One potential solution is to condition on the task ID and use only the corresponding task vector during adaptation; however, the adaptive knowledge merging mechanism complicates this, as it dynamically modifies and merges task vectors, potentially discarding task-specific identities.\n\n  b)\tFor knowledge transfer, the claim in lines 11–13, stating that the method “preserves and adapts critic model parameters,” seems overly strong. Since the model combines vectors from previous tasks via a weighted sum, it does not strictly preserve previous critic parameters unless the hyperparameter for the current task is exactly zero—which the softmax-based formulation prevents. In practice, this weighted combination is more consistent with model merging, a technique known to work well only when the tasks are highly similar. Therefore, I am inclined to interpret the benefit of this approach primarily as providing a stronger initialization for task $k$, rather than truly preserving and adapting previous critic parameters.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is clearly written and easy to follow, making the proposed method accessible to a broad audience.\n\n2.\tThe experimental results demonstrate the effectiveness of the proposed approach, showing clear performance advantages over existing methods.\n\nWeaknesses:\n\n1.\tDue to the use of a softmax function for computing the weights $\\beta$, it is not possible to enforce $\\alpha_1 = 1$. As a result, parameter conflicts remain unavoidable under the formulation presented in Equation (3).", "was_revised": true, "final_justification": "I believe that some points need further clarification, particularly regarding the performance inconsistency and inevitable catastrophic forgetting, which do not show improvement.\n\nNevertheless, in light of the overall contributions, I have decided to raise my score accordingly to \"Borderline accept\".", "rebuttal": ">Q1. Softmax function for computing the weights $\\beta$, it is not possible to enforce $\\alpha_1=1$. Parameter conflicts remain unavoidable.\n\n**A1.** We sincerely thank the reviewer for this insightful observation. Below we address the concern regarding the use of the softmax function for computing the weights $\\beta$ and the potential for parameter conflicts:\n* **Intent and Mechanism Behind the Design Choice.** The statement “by setting $\\alpha_1 = 1$” was meant to convey the **functional behavior** of the model under specific learned conditions, rather than a hard-coded assignment. Specifically:\n    * When learning the very first task ($k = 1$), there are no previous knowledge vectors. Therefore, the summation $\\sum_{j=1}^{k-1} \\alpha_j \\mathbf{v}_j$ is empty, and the policy is constructed solely from the base parameters and the current task’s knowledge vector $\\mathbf{v}_1$ (initialized to zero and then optimized).\n    *  In this case, the model effectively starts learning from scratch without interference from prior tasks, which is what we intended to capture with the idea of “setting $\\alpha_1 = 1$.”\n * **Handling Subsequent Tasks and Emergent Behavior.** For subsequent tasks, while the softmax does not allow any single $\\alpha_j$ to be exactly 1 unless others are 0, our model **learns** to assign very high weight (close to 1) to beneficial knowledge vectors and near-zero weights to irrelevant or conflicting ones. From Fig. 11a, our learned knowledge vectors exhibit near-orthogonal structures, indicating minimal interference. This emergent behavior allows the model to **effectively** disregard harmful prior knowledge, which mirrors the intended behavior of \"setting $\\alpha_1 = 1$\", i.e., preventing negative transfer.\n * **Clarification of Original Phrasing.** We will revise the manuscript to clarify that this is an emergent property of the learned attention weights $\\alpha_k$, not a fixed constraint. This will ensure that readers understand that the model adapts dynamically based on the learned task-specific knowledge.\n\n\n>Q2. Why the performance of CompoNet in this paper appear significantly lower than reported in its original publication?\n\n**A2.** We appreciate your attention to the performance comparison between our implementation of CompoNet and its original publication. Clarify as follows:\n* **Consistent Performance Metrics.** When comparing results rounded to two decimal places, our implementation of CompoNet shows performance metrics that align closely with those reported in the original CompoNet paper. Specifically, in Meta-World (0.41 vs 0.42) and SpaceInvaders (0.98 vs 0.99), the performance values are consistent within reasonable rounding precision.\n* **Code Implementation and Corrections.** We built our evaluation upon the official CompoNet implementation released by the authors [28]. During our implementation process, we identified several inconsistencies in the original codebase, including mismatched MLP layer configurations compared to the baseline implementations. We have carefully corrected these issues while maintaining the core algorithmic principles of CompoNet to ensure fair and accurate comparisons across all methods.\n* **Reproducibility Commitment.** We will release our complete codebase upon paper acceptance. This will allow the research community to verify our implementation and reproduce all experimental results, including the CompoNet baseline evaluations.\n\n>Q3. Is the overall parameters of the model the form $\\theta_{base}x$ until the size of the knowledge pool in Fig. 3?\n\n**A3.** Thanks for this important feedback regarding the parameter growth in our CKA-RL framework. We would like to clarify this point as follows:\n* **Overall Parameter Count in CKA-RL.**  The overall parameter count in CKA-RL remains **bounded**, regardless of the number of tasks. While each knowledge vector $v_j$ shares the same dimensionality as the base parameters $\\theta_{base}$, the **Adaptive Knowledge Merging** mechanism (Sec. 4.3) ensures that the knowledge vector pool $V$ never exceeds a fixed maximum size $K_{max}$.\n* **Adaptive Knowledge Merging Mechanism.** When the knowledge vector pool size $|V| > K_{max}$ (typically set to 5, see Fig. 3), our algorithm identifies the most similar pair of knowledge vectors $(v_m, v_n)$ using Equation (6) and merges them into a single compact representation: $v_{\\text{merge}} = \\frac{1}{2}(v_m + v_n)$. This merging process ensures that the **essential knowledge** is preserved while keeping the pool size bounded.\n* **Parameter Composition in CKA-RL.** The total number of parameters in CKA-RL is composed of:\n    *  **Fixed base parameters** $\\theta_{base}$ (which are never updated after initial training).\n    *  **A bounded knowledge vector pool** with at most $K_{max}$ vectors.\n    *  **Additional lightweight parameters** for the attention mechanism ($\\beta^k$).\n\n>Q4. The standard forgetting metric is not reported in Tab. 1.\n\n**A4.** We sincerely thank the reviewer for this insightful observation regarding the evaluation of catastrophic forgetting. We agree that including the standard forgetting metric would provide a more comprehensive assessment of our method's capabilities. Below is our response addressing this concern:\n* **Supplementing the Evaluation with Standard Forgetting Metric.** We have supplemented our evaluation with the standard forgetting metric: Forgetting=before training on new task - after training on new task [r1], as shown in Tab. A.\n* **Key Insights from the Results.** PackNet achieves the lowest forgetting rate (0.04), but this comes at the cost of significantly lower overall performance (0.2767).In contrast, **CKA-RL** achieves the highest performance (0.7923) and forward transfer (0.7429) among all methods, while maintaining a **relatively low forgetting rate** (0.45).\n\nTab. A: Results on Freeway. \n|METHOD|PERF.|FWT.|Forgetting|\n|-|:-:|:-:|:-:|\n|ProgNet|0.3125|01938|0.36|\n|PackNet|0.2767|0.1970|**0.04**|\n|CReLUs|0.7835|0.7303|0.52|\n|CompoNet|0.7629|0.7115|0.49|\n|CbpNet|0.7678|0.7201|0.46|\n|CKA-RL|**0.7923**|**0.7429**|0.45|\n\n[r1] Replay-enhanced Continual Reinforcement Learning, TMLR 2023.\n\n>Q5. For catastrophic forgetting, the weighted sum of history vectors may not be robust to conflicting knowledge across tasks. \n\n**A5.** Thanks for this insightful feedback regarding potential knowledge conflicts in our Continual Knowledge Adaptation. Clarify as follows:\n* **Emergent Orthogonality of Knowledge Vectors.** Softmax prevents exact assignment of $\\alpha_1$, but our framework naturally learns to approximate this behavior. From Fig. 11a, the learned knowledge vectors exhibit near-orthogonal structure across tasks, with off-diagonal cosine similarities ranging from -0.24 to 0.12. This emergent property enables the model to assign weights extremely close to 1 for beneficial knowledge and near-zero for conflicting knowledge. In practice, when a new task conflicts with previous ones, the weights converge to values where $\\alpha_j\\approx0$ for incompatible vectors and $\\alpha_1\\approx1$ for compatible ones.\n* **Base Parameter Stability.** The base parameters $\\theta_{base}$ remain fixed, providing a stable foundation that prevents complete overwrite of fundamental capabilities. This ensures that even when historical knowledge vectors conflict, the core functionality of the agent remains preserved.\n*  **Conflict Resolution via Adaptive Merging.** Our Adaptive Knowledge Merging mechanism actively identifies and resolves conflicts by only merging highly similar knowledge vectors, while preserving distinct task-specific knowledge. \n*  **Task Identity Preservation.** Contrary to the concern about discarding task-specific identities, our merging mechanism actually preserves task-specific knowledge by only combining highly similar vectors. Additionally, our approach intentionally avoids task-ID information to reflect real-world scenarios where task boundaries are unknown.\n*  **Empirical Validation.** From Tab. A and Tab. 3, CKA-RL effectively balances the stability-plasticity trade-off—preserving sufficient knowledge from previous tasks while remaining adaptable to new tasks. Our method successfully mitigates catastrophic forgetting without sacrificing the ability to learn new tasks effectively.\n \n>Q6. The claim that the method “preserves and adapts critic model parameters,” seems overly strong. \n\n**A6.** We appreciate this opportunity to clarify the mechanism of our approach. \n* **Clarifying \"Preservation\" Terminology.**  We acknowledge that our phrasing could be misinterpreted as suggesting strict preservation of exact parameter values. In reality, CKA-RL achieves functional preservation rather than literal parameter preservation. As demonstrated in Figure 11a, our learned knowledge vectors exhibit near-orthogonal structure, enabling the model to effectively preserve task-specific knowledge by assigning weights extremely close to 1 for beneficial knowledge and near-zero for conflicting knowledge. This emergent orthogonality allows the model to functionally disregard harmful prior knowledge while retaining useful information.\n* **Beyond Simple Initialization.** While a stronger initialization does contribute to performance, our method provides more than just initialization benefits: 1) During training on task k, the model dynamically accesses relevant knowledge from previous tasks. 2) From Fig. 4, CKA-RL demonstrates consistent performance improvements throughout the entire training trajectory, not just at initialization.\n* **Knowledge Transfer Mechanism.** Our adaptive merging mechanism specifically targets vectors with high similarity while preserving distinct task-specific knowledge. This selective merging enables effective transfer even across dissimilar tasks, as evidenced by our strong performance across diverse environments.\n\n***\nWe sincerely hope our clarifications above have addressed your questions."}, {"review_id": "KYuD4Ixnoz", "reviewer": "Reviewer_dAWz", "summary": "The manuscript proposes a new continual reinforcement learning method, Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL), which aims to address catastrophic forgetting and inefficient knowledge transfer in non-stationary environments. CKA-RL maintains a pool of task-specific knowledge vectors and dynamically adapts historical knowledge to new tasks. Furthermore, an adaptive knowledge merging mechanism is introduced to enhance scalability by combining similar knowledge vectors, thereby reducing memory requirements and redundant knowledge. Experimental results on three benchmarks demonstrate superior performance compared to other continual reinforcement learning methods, particularly in terms of overall performance and forward transfer.", "questions": "1. Can the authors provide experimental results with error bars or other statistical significance measures (e.g., multiple seeds, standard deviations)? This is crucial for assessing the reliability of the reported improvements and the overall evaluation of the work.\n    \n2. How does the proposed method differ from existing model merging approaches in continual learning? Is the main contribution the adaptation of these techniques to continual reinforcement learning, or are there substantive methodological novelties?\n    \n3. How does CKA-RL scale with a substantially larger number of tasks (scalability)? The current experiments do not demonstrate performance or memory efficiency in long task sequences.\n    \n4. How does the size of the knowledge vector pool affect performance, especially with increasing task diversity and quantity? Are there guidelines or heuristics for determining the optimal pool size in such scenarios?", "limitations": "Yes", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. The manuscript is well-structured and clearly written, making the methodology and results easy to understand.\n    \n2. The proposed approach effectively adapts model merging techniques from continual learning to the continual reinforcement learning domain, offering a simple yet impactful solution.\n    \n3. CKA-RL achieves a balanced trade-off among plasticity, stability, and scalability, which is potentially influential for the continual reinforcement learning community.\n    \n4. Extensive experiments are conducted across multiple benchmarks, with comprehensive comparisons to recent state-of-the-art methods, demonstrating the effectiveness of the proposed approach from multiple perspectives.\n    \n5. The supplementary material provides detailed descriptions of environments, tasks, and implementation details, supporting reproducibility and clarity.\n    \n\nWeaknesses\n\n1. The experimental evaluation lacks statistical rigor: error bars or other statistical significance indicators are absent, and experiments appear to be conducted with a single random seed (which is not specified in the main text). Given the inherent instability of reinforcement learning, this significantly undermines the credibility of the reported results. Additionally, the absence of code availability limits reproducibility.\n    \n2. There is insufficient discussion of related work regarding model merging (task vector) in continual learning, despite its close relevance to the proposed method.\n    \n3. The evaluation of model plasticity (in Section 5.2) is inadequate. Compared methods in the literature typically use more tasks and longer experimental horizons, making it difficult to draw meaningful comparisons about plasticity from the current experimental setup.\n    \n4. There is a conceptual inconsistency in the discussion of initial performance (in Section 5.4): the manuscript claims that superior initial performance is due to effective use of historical knowledge vectors, but in the second task, no such historical vectors are available.\n    \n5. The scalability analysis (in Section 5.4) is limited. The experiments only involve up to 10 tasks, and the knowledge vector pool is fixed at 5. In contrast, related work such as CompoNet evaluates scalability with up to 300 tasks. The performance and memory efficiency of CKA-RL under longer task sequences remain unclear.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns with additional experiments and clarifications that provide further support for the manuscript’s claims. However, I still find the substantive novelty of the work to be limited, and the original submission suffered from missing key details and unfair comparisons, which continue to affect my confidence in the overall quality of the manuscript. Therefore, I am raising my score to “borderline reject” to reflect that I do not strongly oppose acceptance, but remain unconvinced about a clear recommendation for acceptance.", "rebuttal": "We are deeply grateful to you for recognizing the strengths of our work, particularly the clear presentation of our methodology and the balanced trade-off CKA-RL achieves among plasticity, stability, and scalability. We sincerely appreciate your acknowledgment of our comprehensive experimental validation and the potential impact of our approach on the continual reinforcement learning community.\n\n>Q1. The experimental evaluation lacks statistical rigor (e.g., multiple seeds, standard deviations), and the absence of code availability limits reproducibility.\n\n**A1.** Thank you for your valuable feedback on the statistical rigor of our evaluation. We will address this in the revised version, as detailed below:\n* **Multiple Random Seeds Used.** Our experiments were conducted with **10 different random seeds**, consistent with standard practice in the recent CRL method CompoNet. We acknowledge that this detail was not explicitly stated in our initial submission, which was an oversight. \n* **Statistical Significance Evidence.** We have computed and included standard deviations for all reported results. The updated results table (see Tab. A) now shows mean performance $\\pm$ standard deviation across all task sequences and methods. Notably, CKA-RL demonstrates lower variance compared to most SOTA methods.\n* **Code Availability Commitment.** We confirm that our code will be released upon paper acceptance. \n\nTab. A: Summary of results across all task sequences and methods.\n|METHOD|META-WORLD PERF.|META-WORLD FWT.|SPACEINVADERS PERF.|SPACEINVADERS FWT.|FREEWAY PERF. |FREEWAY FWT.|\n|-|:-:|:-:|:-:|:-:|:-:|:-:|\n|ProgNet|0.4157$\\pm$ 0.49|-0.0379$\\pm$ 0.13 |0.3757$\\pm$ 0.27|-0.0075$\\pm$ 0.22|0.3125$\\pm$ 0.27 |0.1938$\\pm$ 0.31|\n|PackNet|0.2523$\\pm$ 0.40|-0.6721$\\pm$ 1.40|0.2299$\\pm$ 0.30|-0.075$\\pm$ 0.13|0.2767$\\pm$ 0.36| 0.1970$\\pm$ 0.32|\n|MaskNet|0.3263$\\pm$ 0.47|-0.3695$\\pm$ 0.45|0$\\pm$ 0|-0.3866$\\pm$ 0.53|0.0644$\\pm$ 0.17|-0.0503$\\pm$ 0.12|\n|CReLUs|0.3789$\\pm$ 0.47|-0.0089$\\pm$ 0.24|0.8873$\\pm$ 0.10|0.5308$\\pm$ 0.29|0.7835$\\pm$ 0.13|0.7303$\\pm$ 0.12|\n|CompoNet|0.4131$\\pm$ 0.50|-0.0055$\\pm$ 0.20|0.9828$\\pm$ 0.02|0.6963$\\pm$ 0.32|0.7629$\\pm$ 0.12|0.7115$\\pm$ 0.10|\n|CbpNet|0.4368$\\pm$ 0.50|-0.0826$\\pm$ 0.22|0.8392$\\pm$ 0.11|0.4844$\\pm$ 0.28|0.7678$\\pm$ 0.10|0.7201$\\pm$ 0.07|\n|**CKA-RL**|**0.4642 $\\pm$ 0.50** |**-0.0032$\\pm$ 0.21**|**0.9928 $\\pm$ 0.01**|**0.7749$\\pm$ 0.20**|**0.7923$\\pm$ 0.10**|**0.7429$\\pm$ 0.07**|\n\n>Q2. There is insufficient discussion of related work on model merging in CL. Is the main contribution the adaptation to CRL, or are there novel methodological innovations?\n\n**A2.** We appreciate your insightful comments. While we referenced model editing techniques as inspiration for our knowledge vectors, we agree that a deeper analysis of the connection to model merging literature would strengthen our paper. **We will expand the discussion on model merging in the revised version of our manuscript.** While our work is indeed inspired by model merging techniques, CKA-RL introduces several substantive methodological novelties beyond simply adapting these techniques to RL: \n* **Dynamic Knowledge Adaptation Mechanism.** Unlike traditional model merging that uses static pre-trained models, our Continual Knowledge Adaptation strategy dynamically determines how to utilize historical knowledge through learnable adaptation factors ($\\alpha$). This allows the agent to flexibly select and weight relevant knowledge based on the current task, which is critical in non-stationary RL environments where task relationships are complex and evolving.\n* **Task-Specific Knowledge Vector Learning.** In contrast to existing model merging methods that operate on full model parameters, we introduce a specialized knowledge vector structure that isolates task-specific adaptations from the base policy. This design specifically addresses RL challenges like policy interference and catastrophic forgetting by preserving a stable base policy while allowing task-specific adjustments. \n\n>Q3. The methods in literature typically use more tasks and longer horizons, making comparisons on plasticity challenging in the current setup.\n\n**A3.** We appreciate your feedback regarding the evaluation of model plasticity. Specifically:\n* **Standard Benchmarking Practice.** Our main experimental setup, including task count and evaluation protocol, follows standard practice in the continual reinforcement learning (CRL) literature. Like CompoNet [ICML 2024], our core experiments are conducted on 20-task sequences using widely accepted CRL benchmarks. This ensures a fair and consistent comparison across all methods. \n* **Comprehensive Performance Metrics.** Beyond simple task count, we measure plasticity through multiple metrics: 1)Average performance; 2) Forward transfer; 3) Memory efficiency analysis (Fig. 5a), demonstrating our method parameter memory remains nearly constant after the 5th task, while competing methods like ProgNet and CompoNet show linear growth.\n* **CKA-RL maintains superiority on longer task sequences** We conduct additional experiments with extended **22-task sequences on the Meta-World environment**. The results demonstrate that even with longer task sequences, CKA-RL retains its performance advantage **(CKA-RL vs CompoNet: (0.4601, -0.0042) vs (0.4098, -0.0073))**. We will include these results in the revised manuscript to strengthen our plasticity evaluation.\n\n>Q4. The manuscript claims that superior initial performance is due to effective use of historical knowledge vectors, but in the second task, no such historical vectors are available.\n\n**A4.** Thanks for your valuable comments. We will clarify in the revision. Below we address this concern with specific points:\n* **Clarification of Knowledge Vector.** The historical knowledge vector $v_1$ is defined as a zero vector ($v_1 = \\theta_1 - \\theta_{base} = 0$) as stated in Eqn.(3) of our paper. Therefore, the superior initial performance observed in the second task is not solely due to historical knowledge vectors.\n* **Mechanism of Knowledge Transfer from the Third Task Onward** For tasks beyond the second, historical knowledge vectors $(v_1, v_2)$ become non-zero and actively contribute to superior initial performance. From Fig. 4, the reward curve of CKA-RL demonstrates a higher starting point compared to baselines like CbpNet and CReLUs.\n\n>Q5. The performance and memory efficiency of CKA-RL under longer task sequences remain unclear.\n\n**A5.**  We would like to clarify the rationale behind our experimental design, while providing additional evidence for the scalability of CKA-RL.\n* **Standard Benchmarking Practice in CRL.** While CompoNet presents a theoretical scalability analysis involving 300 tasks, by constructing models and measuring parameter growth without actual training, our scalability analysis is conducted on real tasks with full training, enabling a more realistic assessment of memory and performance under continual learning.\n* **Scalability Guarantee.** The core of our scalability argument lies in the Adaptive Knowledge Merging mechanism. Unlike methods that accumulate parameters linearly (e.g., ProgNet, CompoNet), CKA-RL maintains a bounded knowledge vector pool through similarity-based merging.\n* **Additional Long-Sequence Experiments.** Following CompoNet setting, we conduct supplementary experiments with extended task sequences (300 sequences). From Tabs B&C, CKA-RL maintains nearly constant parameter size and inference latency through vector merging, while other methods show linear growth and increasing computational overhead.\n\nTab. B: Total Parameter Memory Consumption (MB) in the Meta-World.\n|Task ID|5|10|50|150|200|300|\n|-|-|-|-|-|-|-|\n|ProgNet|5.64|17.22|334.79|2878.74|5088.21|11382.16|\n|CompoNet|7.18|14.43|72.48|217.60|290.17|435.29|\n|CKA-RL|5.64|10.34|10.34|10.34|10.34|10.34|\n\nTab. C: Inference latency (s) in the Meta-World.\n|Task ID|5|10|50|150|200|300|\n|-|-|-|-|-|-|-|\n|ProgNet|0.0010|0.0021|0.0315|0.2558|0.4610|1.0353|\n|CompoNet|0.0023|0.0045|0.0213|0.0636|0.0842|0.1264|\n|CKA-RL|0.0002|0.0001|0.0002|0.0003|0.0002|0.0002|\n\n\n>Q6. How does the size of the knowledge vector pool affect performance, especially with increasing task diversity and quantity? Are there guidelines or heuristics for determining the optimal pool size in such scenarios?\n\n**A6.** Thank you for your insightful feedback. We appreciate the opportunity to elaborate on this aspect.\n* **Empirical Analysis of Pool Size Impact.** From Fig. 3, we evaluated the effects of different $K_{max}$ values on both performance and forward transfer. When $K_{max}$ is small, CKA-RL removes too many knowledge vectors during adaptive knowledge merging, thus being unable to utilize enough knowledge. When $K_{max}$ is too large, redundant or even conflicting knowledge may slow down the knowledge adaptation process, resulting in performance degradation. \n* **Task Diversity Considerations.** When task diversity increases, the optimal Kmax tends to increase as well. We conduct experiments in Meta-World (which has higher task diversity), we find $K_{max}=8$ to be optimal (**see Tab. D**).\n* **Practical Guidelines for Determining $K_{max}.$** 1) Initial Estimation: Start with $K_{max} = 5$ as a baseline for moderate task diversity; 2) Task Similarity Analysis: If tasks are highly heterogeneous, increase $K_{max}$ proportionally to the task diversity; 3)Memory-Performance Trade-off: For resource-constrained environments, select the smallest Kmax where performance plateaus.\n\nIn future work, we plan to investigate dynamic pool sizing mechanisms that automatically adjust $K_{max}$ based on task similarity metrics and performance feedback.\n\nTab D. Effects of different pool sizes in Meta-World.\n|Pool size|4|8|12|16|18|\n|-|-|-|-|-|-|\n|PERF.|0.342|**0.484**|0.405|0.394|0.400|\n|FWT.|-0.110|**-0.100**|-0.100|-0.100|-0.150|-0.130|\n\n***\nWe sincerely hope our clarifications above have addressed your questions."}]

Author rebuttal:

Meta-review: Summary: This paper addresses the important problem of continual reinforcement learning in non-stationary environments, proposing CKA-RL, which combines dynamic weighting of prior task knowledge, task-specific residual vectors, and an adaptive merging mechanism to maintain bounded memory. 

Strengths: The reviewers agree that the paper is clearly written, empirically well-supported, and tackles a relevant challenge. Strengths include the memory/inference efficiency analysis, consistent performance gains across Meta-World and Atari, and the introduction of a merging mechanism that enables scalability. 

Weaknesses: The main concerns of the reviewers center on the perceived incremental novelty of task-vector based methods, limited improvement on catastrophic forgetting, and questions about the fairness of comparing bounded-pool scalability to baselines with growing memory. 

Rebuttal: The rebuttal successfully clarified questions on experimental results (e.g. adding error bars), provided theoretical analysis (drift bounds, orthogonality, merging error), and added long-horizon experiments, which led three reviewers to raise their scores and the fourth to maintain a positive borderline-accept rating.

Recommendation: Borderline Accept. While this is an incremental contribution, the work represents a step forward in scalable continual RL under bounded resources, with contributions that will interest the community.
Decision: Accept (poster)