Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning
Authors: Ke Sun
Hongming Zhang
Jun Jin
Chao Gao
Xi Chen
Wulong Liu
Linglong Kong
Venue: ICLR 2026 Poster (2026)
Keywords: continual learning
reinforcement learning
meta learning
Primary area: reinforcement learning

Abstract: Inspired by the human learning and memory system, particularly the interplay between the hippocampus and cerebral cortex, this study proposes a dual-learner framework comprising a fast learner and a meta learner to address continual Reinforcement Learning~(RL) problems. These two learners are coupled to perform distinct yet complementary roles: the fast learner focuses on knowledge transfer, while the meta learner ensures knowledge integration. In contrast to traditional multi-task RL approaches that share knowledge through average return maximization, our meta learner incrementally integrates new experiences by explicitly minimizing catastrophic forgetting, thereby supporting efficient cumulative knowledge transfer for the fast learner. To facilitate rapid adaptation in new environments, we introduce an adaptive meta warm-up mechanism that selectively harnesses past knowledge. We conduct experiments in various pixel-based  and continuous control benchmarks, revealing the superior performance of continual learning for our proposed dual-learner approach relative to baseline methods.

Published as a conference paper at ICLR 2026
PRINCIPLED FAST AND META KNOWLEDGE LEARN- ERS FOR CONTINUAL REINFORCEMENT LEARNING
Ke Sun Hongming Zhang Jun Jin', Chao Gao*, Xi Chen', Wulong Liu", Linglong Kong University of Alberta
National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institution of Automation, Chinese Academy of Sciences
Edmonton Research Center, Huawei Canada 'Huawei Noah's Ark Lab "Beaalffinity
{ksuné, jjin5, hongming zhang @ia ac. (chao. gao4, xi chen @ com wulong. 1 liu@gmail. com.
ca
ABSTRACT
Inspired by the human learning and memory system, particularly the interplay between the hippocampus and cerebral cortex, this study proposes dual-learner framework comprising fast learner and meta learner address continual Re- inforcement Learning (RL) problems. These two learners are coupled to perform distinct yet complementary roles: the fast learner focuses on knowledge transfer, while the meta learner ensures knowledge integration. In contrast to traditional multi-task RL approaches that share knowledge through average return maximiza- tion, our meta learner incrementally integrates new experiences by explicitly mini- mizing catastrophic forgetting. thereby supporting efficient cumulative knowledge transfer for the fast learner. To facilitate rapid adaptation in new environments, we introduce an adaptive meta warm-up mechanism that selectively harnesses past knowledge. We conduct experiments in various pixel-based and continuous con- trol benchmarks, revealing the superior performance of continual learning for our proposed dual-learner approach relative to baseline methods. The code is released in https::: htt::////itta com datake
INTRODUCTION
Most deep reinforcement learning (RL) algorithms (Sutton & Barto 2018 Mnih al.) 2013 Schul- man et al. 2015 Haarnoja et al. 2018a Schulman et 2017) are designed for a single task. where environmental dynamics and reward function often remain stationary over time. In contrast, hu- mans continually face diverse and evolving environments, learning to solve new tasks sequentially throughout their lives. Building artificial general agents with such adaptive capabilities requires continual learning, i.e., the ability acquire new knowledge efficiently without forgetting previ- ously learned skills. In this realm, Continual Reinforcement Learning (Khetarpal et al.) 2022: Abel crucial paradigm, aiming balance plasticity (rapid adaptation to new
al. 2023) emerges tasks) and stability (retention of past knowledge). An ideal continual learning agent is capable of transferring useful knowledge forward to accelerate learning in new environments while avoiding catastrophic forgetting (Dohare et al. 2024) across previously encountered tasks.
Recent work in continual RL spans wide range of strategies (Barreto et al. 2020 Kessler al.
2022
Kaplanis et al., 2019 Caccia et al. 2022 Kaplanis et al.] [0118] Gaya el al. 2023 Yang et al.,
2023
Wolczyk al. 2022 Anand & Precup 2023 Wan al.) 2022 Chandak
2020 Sun
et al.
2025
Liu et al., 2025; Tang et al.) (025)) Despite rapid progress, continual RL remains
fragmented: existing algorithms above are often motivated by heuristics developed from distinct
Equal Contribution.
Corresponding Author.

---

Published as a conference paper at ICLR 2026
perspectives (see Appendix for a detailed comparison of related work), but there is no principled way to understand when and why they work. This lack of a theoretical foundation makes difficult to assess when knowledge transfer will be beneficial, how to mitigate forgetting, and how to set explicit optimization objectives, hindering principled algorithm development in continual RL..
To address these challenges, our study contributes to new foundations of continual RL by (1) defin- ing the MDP difference to formally quantify environment similarity, and (2) introducing quantita- tive measure of catastrophic forgetting applicable both value- and policy-based RL. Building on these new foundations and principles, we propose a dual-learner paradigm that mirrors hippocam- pal-cortical interactions in the human learning and memory systems (Kumaran et al.) 2016). decom- posing continual RL into two complementary objectives: knowledge transfer and knowledge inte- gration. Specifically, we maintain two distinct yet complementary components, i.e., a fast learner and meta learner, which are analogous to the functional roles of the hippocampus (the fast learner) and the neocortex (the meta learner) in the human brain. Our approach also elucidates its more pro- found connection to the transfer and multi-task RL problems (Schaul et al. 2015 Rusu et al. 2015
Parisotto et al. 2016 Rajendran et al. 2017 Teh et al. 2017;
Bai et al. 2023).
Knowledge Transfer via Fast Learner. We leverage a fast learner to rapidly acquire knowledge from a new task by adaptively transferring prior knowledge stored in a meta learner. To circumvent the potential negative transfer issue, adaptive meta warm-up strategy is developed by either di- rectly copying parameters as initialization or adding a behavior cloning regularization in the early training phase to guide the exploration. The function of fast learner in knowledge transfer resem- bles the hippocampus. By swiftly encoding the new experiences and discriminating the effectiveness of existing knowledge, the hippocampus, guided by the neocortex, specifically functions quickly assimilate novel scenarios in response to immediate environmental changes or drifts.
Knowledge Integration via Meta Learner. After assimilating the new knowledge by the fast leamer, an incremental knowledge integration incorporates the new experiences into the existing knowledge pool stored in the meta learner. Under the new foundation, the knowledge integration incrementally updated in the principle of catastrophic forgetting minimization under specific diver- gence metrics. After consolidating old and new experiences, the meta learner enhances the adaptive meta warm-up, facilitating the knowledge transfer in the next environment. The knowledge integra- tion process plays role akin to the cerebral cortex, which gradually integrates, incorporates, and consolidates new knowledge into the existing cognitive structure in the human brain to build more generalizable, robust, and stable decision decision-making system.
Contributions. The contributions of our study can be succinctly summarized as follows:
We propose new foundations of continual RL, including the definition of MDP difference and the measure of catastrophic forgetting, underpinning the algorithmic innovations in the future.
We devise a dual-learner system that incorporates distinct yet complementary fast and meta learn- ers perform knowledge transfer and knowledge integration. The interplay between fast and meta learners mimics the hippocampal-cortinn cortical dialogue in the brain's 's memory systems.
We provide comprehensive empirical studies to validate the efficiency of our dual-learner system in discrete and continuous action domains, across both value- and policy- RL algorithms.
PROBLEM SETTING AND NEW CONTINUAL RL FOUNDATIONS
Problem Setting, Let [K] denote 21. K}. We consider a sequence of K tasks, where each task & E [K] is modeled by Markov Decision Process (MDP) M AS, Px, Rs, 7). Sy and A denote the state and action spaces, Pss : S x A - P(Sx) is the transition dynamics, Re : St x A& + R is the reward function, and y is the discounting factor. We define the action- value function Q" a) = E, [2:00 72 R S, = s, A, =a] given a state s, an action a, and policy Following common practice in continual RL (Wolczyk et al. 2022 Khetarpal et al.) 2022 Malagon et al. 2024), we adopt three assumptions: (1) the same state and action spaces, (2) known task boundaries, i.c., semi-continual RL (Anand & Precup 2023), and (3) training budget with a moderate model size and an allowable computation cost. An "optimal" policy can generalize favorably across all tasks by balancing adaptation new tasks with prior knowledge retention.
2

---

Published as a conference paper at ICLR 2026
Foundation 1: MDP Distance. Theoretical analysis in continual RL requires quantitative sim- ilarity measure between environments to determine when knowledge transfer will be beneficial or harmful, and to assess how strongly new tasks may interfere with previously learned ones. A desir- able MDP distance should consider variations from both reward functions and transition dynamics between MDPs. To this end, we utilize the distance between two diDe-detrmieeed optimal Q func- tions or task-specific optimal policies to quantify the MDP distance in Definition
Definition (MDP Distance) For two finite MDPs: MDP: (S.A,Ry, R1,,,,,.1.. P1,7) and MDP₂ (S, A. R2, ?)) we denote their optimal Q functions as Qi and Q2 and the optimal policies as тi and m2- The Q-value-baser and policy-based MDP distances are defined as do(Qi, Q5) and d, (Ti, m2) under certain divergences or distances dq and dxs e.g, the lo loss or the KI. divergence.
Foundation 2: Catastrophic Forgetting. Our definition catastrophic forgetting in continual RL is inspired by distribution drift and calastrophic forgetting quantified in deep learning litera- ture (Doan et al., 202T), which we briefly recap Appendix [ Grounded in the definition of MDP difference in Definition [T] introduce catastrophic forgetting between two MDPs in Definition 2
Denote 2 and pig the state visitation distributions when policy x greedy policy Te over Q defined by n (-Is) = arg max, ((s, a), interacts in the k-th environment in a sequence of K tasks.
Definition 2. (Catastrophic Forgetting across Two Environments) Denote Qk-1, Qu and Tk-1, as functions and policies after training RL algorithms across the (k- 1)-hh and k-th environments sequentially, The catastrophic forgett denoted by CF, is defined as
CF(QA-1,Qk))
Qk Qa Mk-1
(1)
CF(Tk-1:
Th- M (s)dx (Tx (T(1)))
For each s and a, the weights -x-1 (5) (a]s) and (x) characterize the relative impor- tance when measuring discrepancies between Q functions and policies. Crucially, evaluate these weights using the preceding policy π--1 (r 1) rather than the current policy Te (s (TQ2), ), as the past policy better reflects states and actions that mattered most in the old task. In contrast, if use Th Qt) for the weight evaluation, significant changes in the Q-function or policy on previously important date-action pairs might be overlooked, since the new policy may no longer visit them.
3 FAME: PRINCIPLED FAST AND META KNOWLEDGE CONTINUAL RL
The proposed FAME approach is applicable to both value-based and policy-based RI. In value-based RI., we denote Q₁ as the updated fast learner after learning task k, followed by a meta learner Q that integrates knowledge from the preceding meta learner Q and Qu. In policy- based RL, we denote Tı as the fast learner after learning task k and then meta learner T integrates knowl- edge from the preceding meta learner Th M and Tg. The coupled updating of the fast and meta learners in FAME illustrated in Figure [ In princi- ple, the fast learner rapidly learns the new task guided by the meta learner via the proposed adaptive meta warm-up. Meanwhile, the meta learner consolidates the experience from the preceding meta learner and the current fast learner via knowledge integration to minimize catastrophic forgetting.
Fast Learner
T
22
π3
Meta Learner
M 1
2
MM
3
4
M
knowledge transfer
knowledge integration
Illustration of FAME. In value-based continual RL, the fast learner can be denoted by {Qx] 1K accordingly instead of 1K
Figure 1:
3

---

Published as a conference paper at ICLR 2026
3.1 VALUE-BASED CONTINUAL RL WITH DISCRETE ACTION SPACES
3.1.1 KNOWLEDGE INTEGRATION: CATASTROPHIC FORGETTING MINIMIZATION PRINCIPLE
After the fast learner Qs completes training in the k-th environment, the knowledge integration phase begins. In this phase, the meta learner Q is updated to consolidate information by combining the prior knowledge encoded in the preceding meta learner Q and the new knowledge acquired by the fast learner Qt. Unlike classical multi-task RL, which maximizes the average rewards, our meta learner is explicitly designed to minimize the catastrophic forgetting defined in Definition 2
Q- --aeeue-based Catastrophic Forgetting and Limitations. We extend the Q-value-based catas- trophic forgetting defined in Eq. over a sequence of K tasks. In the k-th environment, the optimal meta Q value function M is the minimizer by solving the following objective function:
Q = arg min
ΣΣ (s) - (s.a)) Q
where we recall that 14: is the state visitation distribution when the greedy policy 7 (i.e.. 99 (-Is) = arg max Qi(s, a)) interacts with the i-th environment. Intuitively, the direct minimizer QM by solving Eq. 3 weighted average among {Q,);- However, developing the capability of continual learning by storing all previous Q functions fails to scale the number of tasks, which is one of the crucial requirements in continual RL. Indeed, we rewrite the above objective function as an incremental updating rule between the preceding meta learner Q and the fast learner Qs. which we defer Appendix C.I Nonetheless, the fundamental limitation of Q-value-based catas- trophic forgetting lies in the fact that it is mainly applicable to distinct environments with similar scales Q values, such as those with varying transition dynamics yet the same reward function. As the new arriving environment is agnostic, the scale of the Q-values may be hard to learn because it is not necessarily bounded and can be quite unstable (Rusu et al.. 2015). The previously well-learned tasks with high rewards tend to be more salient in consolidating knowledge than those with small re- wards (Zhang et al., 2023). Therefore, the policy-based definition catastrophic forgetting Eq, is more versatile than the Q-value-basee based one in Eq. [1] serving preferable alternative. In addi- tion. policies may inherently enjoy lower variance than value functions, contributing to improved performance and stability (Greenssithh et al. 2004).
Policy-based Catastrophic Forgetting. Even in value-based continual RI., it is preferable to em- ploy the policy-based definition of catastrophic forgetting based Eq. 2 for incremental update of the meta learner. Akin to the Q-value-basse catastrophic forgetting in Eq. 3 the optimal meta policy n in the k-th environment is the minimizer by solving the following objective function:
M arg min AM
(s) (T.(1s)) (+1s))
Incremental Softmax Meta Learner Update for Value-based Continual RL. Define the weight function w (s, a) ! ( for each E [K]. For any measurable function f(s a) and weight function with Es.a a(s,a) E. [f] E.. w(s, a)),,, a). When equipped with the categorical representation, the Q-values can be converted into Softmax (Boltzmann) policy, allowing the value-based continual RL to minimize the policy-based catastrophic forgetting objective defined Eq. 4 Specifically, given a temperature т, we denote x (als) exp (a((s))/T) /E. exp (Q'(a'(s)/T) T). By employing the KL divergence as dz. we derive a concise meta learner update rule in Proposition
and w(ss 2 0 for each and a, we define
Proposition 1 (Incremental Softmax Q- Value-based Meta Learner Update). Denote TH (als) = exp (@M (als)/T) / /Ea exp (a')))))) After a softmax policy transformation, the Q-value- based meta learner incremental update is written as
Q arg min Q
k-1
10
T log
M K
+
E
log
Q π
an &
arg max Q
k
E
[og 7 M1
(5)
4

---

Published as a conference paper at ICLR 2026
The proof of Proposition I is straightforward and is therefore deferred to Appendix C.2 for com- pleteness. Crucially, minimizing the policy-based catastrophic forgetting in Eq. 5 is simply equiv- alent to the Maximum Likelihood Estimator (MLE) by fitting the meta learner Q to mixture of state-action distributions across encountered environments. We highlight that the final simplified objective in Eq. 5is independent of Q and Qx: however, knowledge integration in principle takes the form of an incremental update rule, which we instantiate in Section 3.2.1
3.1.2 KNOWLEDGE TRANSFER VIA ADAPTIVE META WARM-UP
Challenges. An effective knowledge transfer necessitates rapidly adapting to the new environment by taking advantage of the previous knowledge if accessible. However, the commonly used finetun- ing is effective when tasks similar, but can lead to negative transfer issue that frequently occurs in continual RL (Ahn et al. [025 Wolczyk et al.] |002]. The negative transfer, a crucial factor of the loss of plasticity (Dohare et al. 2024), leads to performance degradation owing the dissimilarity between the two tasks. Training from scratch (i.e.. reset) is easy to implement to circumvent the
negative transfer (Chen et al.) 2024 Ahn et al. 2025). However, this naive warm-up lacks flexibility and fails to make full use of the accumulated knowledge to speed up the adaptation to a new task.
Adaptive Meta Warm-Up via One-vs-all Hypothesis Test. When new task arrives, it com- mon strategy to initialize the fast learner with parameters from the meta learner. Nonetheless, knowl- edge and skills previously acquired in past tasks may become misleading when the new environment differs substantially. For instance, it is particularly evident that humans make incorrect decisions or take suboptimal actions when the new information contradicts earlier experiences. To harmonize the potentially conflicting objectives, we propose an adaptive meta warm-up approach that chooses the most effective warm-up strategy among the preceding meta learner, a random learner (L.e., reset), and the preceding fast learner (i.e., finetune). Formally, the adaptive meta warm-up is framed as a one-vs-all hypothesis test based on policy evaluation during the early interaction with a new envi- ronment. When the k-th task arrives, we have access to three types of warm-up learners, including preceding fast learner Qx random Q function Q" associated with the policy x Evaluating the three warm-up learners yields their value functions defined by Vid = E [R]. V.M = E, [R], and Vig Eg [R]. For each task that arrives, the one-vs-all hypothesis test with composite null is expressed as
1, a meta policy á R with the softmax transformation from Q and a
Ho VEN < max {vL. V
vs. H VE max VV vz)
(6)
When the null hypothesis Ho cannot be rejected, we further compare V and V via a common parametric hypothesis test, e.g., f-test. most scenarios, picking the best warm-up strategy accord- ing to the empirical ranking often performs favorably. However, in safety-critical scenarios, e.g., autonomous driving, a rigorous statistical test is crucial either by bootstrapping or anytime valid inference (Ramdas et al. 2023) on the adaptively collected dataset used for the policy evaluation.
Meta Warm-Up via Behavior Cloning Regularization. Once we reject Ho, we are ready to per- form the meta warm-up. However, directly initializing the fast learner Qs via the meta policy TM is infeasible as the meta learner is now represented as a policy instead of a Q function under the update in Proposition[] [T An easy and effective way address this policy-to-valle value transfer mismatch to impose Behavior Cloning (BC) regularization in the early training phase, when the meta policy Th serves the expert for data collection and early exploration. Concretely, QR is the minimizer (+|ss)] where Lo(Q)) is (-))((( (-Is)][=*
of the BC regularized loss L(Qx) Lo(Q)) + AE, [L M
the original loss to update Qt, such as the MSE or Huber loss in DQN (Mnih et al. 2015).
3.1.3 ALGORITHM: VALUE-BASED FAME
Meta Buffer M in Knowledge Integration. In the last N steps of updating the fast learner in each environment, we additionally store the state-action pairs in meta learner's buffer M, which are used to approximate w for E [k] in Eq. 5 Note that the stored state-action pairs are only a small portion of the training dataset for each task (around 1% or 2% in our experiments), yielding a moderate size of the meta buffer M. The moderate and fixed size of a meta buffer is crucial in reality, are not expected to store too much past data in continual RL..
5

---

Published as a conference paper at ICLR 2026
Algorithm Value-based FAME Update in the k-th Environment
1: Initialize: Fast Buffer F, Meta Buffer M, Q QK Q", Warm-Up Step L, Estimation Step
N.
2: # Knowledge Transfer: Adaptive Meta Warm-Up
3: Initialize Qu in {Qu-1, Q20 via Eq. 6 within L steps
4: for =
do 5: Observe St, take action At. receive R4, observe S1+1
6: Store (S,, A,, R2, 1) in
7: Update Qs
8:
if < T
N then
9: Store As) M # To Estimate w
10: end if
end for
12: Reset F
13: Knowledge Integration: Minimize Catastrophic Forgetting
14: Update Q via Eq. 5 using state-action pairs in M
Algorithm. Denote the buffer of the fast learner as F and Q" the randomly initialized Q function. We denote as the timesteps in each environment. As suggested Algorithm [ when the k-th environment arrives, we warm start the fast learner Q₁ via the adaptive meta warm-up strategy among the preceding meta learner 1 the preceding fast learner Qx-1 and random learner Qo (i.e., reset) within the first L steps. The adaptive meta warm-up makes full use of previous information to perform adaptive knowledge transfer. Once the k-th task ends, the knowledge integration phase starts, when the meta learner Q is updated via Eq. 5 on the data collected in the meta buffer M. The meta learner Q incorporates the acquired knowledge in Qx into Q via an incremental update rule in principle.
3.2
POLICY-BASED CONTINUAL RL WITH CONTINUOUS ACTION SPACES
3.2.1
KNOWLEDGE INTEGRATION: CATASTROPHIC FORGETTING MINIMIZATION PRINCIPL.E
As opposed to the value-based continual RL with incremental softmax meta learner updates in Proposition 1 in policy-based continual RI, we directly minimize the policy-based catastrophic forgetting in Eq. 4 regarding the parameterized policy function. The detailed incremental update rule depends on the choice of d and how we represent the policy in a continuous action space. Next, we will introduce two variants of policy-based continual RL methods when equipped with the forward KL divergence and Wasserstein distance, respectively.
Method 1 (FAME-KL): Policy Distillation under Forward KL Divergence. We show that the posecy-based based knowledge integration reduces to a form of policy distillation. Analogous to Propo- sition in the vasue-based continual RL, the policy-based knowledge integration in Eq. when instantiated with the forward KI. divergence under an accessible probabilistic policy, yields up- date rule as
policy-
Evy [lg xM
T
arg max
where we recall that (s, 1 (a)s) denotes the policy- based steady state-action distribu- tion on the i-th environment. Importantly, the policy-based based knowledge integration objective above coincides with the knowledge distillation update used in policy distillation (Rusu et al. 2015) and with typical multi-task RL formulations (Teh et al. 2017). establishing an intriguing connection.
Method 2 FDD::-WD): Wasserstein Distance bbsed Knowledge Integration. Although the KL divergence is often adopted for its computational simplicity. the Wasserstein distance is preferable when comparing more complex policy distributions. In contrast to the KL divergence, which only measures pointwise differences between distributions, the Wasserstein distance by defi- nition explicitly accounts for the underlying geometry of the data space, such as the probability space of the policy outputs, and therefore provides a more informative notion distributional discrep- ancy (Panaretos & Zemel 2019 Arjovsky et al. 2017). In Proposition 2 we derive the policy-based

---

Published as a conference paper at ICLR 2026
incremental update rule under the Wasserstein distance. Particularly, when the policy function is rep- resented by a (multivariatee Gaussian distribution, as is common in many policy-baser algorithms, closed-fore formm incremental update rule becomes available, enabling efficient knowledge integration for the meta learner. The proof is given in Appendix C.3
Proposition 2 (Incremental Policy-basee Meta Learner Update under Wasserstein Distance). Con- sider dx to be the squared 2- 2-Wasserstein distance denoted by W2 in Eq.. of Definition The policy is represented as an independent (multivariate) Gaussian distribution over the action a. Minimizing policy-based based catastrophic forgetting in Eq. 4 is equivalent to:
A-1 1 W
" = arg min
ΣΣ
(T (16), -
Σ
1 Th (8) (TN (10),
(8)
3.2.2 KNOWLEDGE TRANSFER VIA ADAPTIVE META WARM-UP
For the adaptive meta warm-up in policy-based RL, we perform policy evaluation across the first L steps and conduct the one-vs-all hypothesis test in Eq. when a new task arrives, which is similar to value-based continual RL. Once we determine the best-performing warm-up policy among the fast
M policy тA-1, the meta policy "k-1- and random policy n², we directly initialize the fast policy. Using parameter initialization as the meta warm-up strategy is more convenient for deployment than adding the BC regularization used in the value-based continual RL introduced in Section 3.1.2
Algorithm. Since the description of the policy-based FAME algorithm closely parallels Algo- rithm [ we defer it with discussion of computational cost and practical guidance to Appendix D We also provide a general discussion of FAME framework in Appendix
4 EXPERIMENTS
In this section, we validate our FAME approach across sequence of tasks from multiple environ- ments and domains, including the pixel-based tasks with discrete action space in Section 4.1 and control problems with a continuous action space in Section 4.2 The central hypothesis is that the interplay between knowledge transfer and knowledge integration of the fast and meta learners in FAME benefits both forward transfer (i.e., plasticity) and catastrophic forgetting (i.e., stability).
Evaluation Metrics. We employ the standard metries (Wolczyk et al. 2021| 2022) in continual RL to evaluate average performance, forgetting to measure stability, and forward transfer to quantify plasticity. Let (1) be the success rate or average returns in task by using the policy at time with I E K T). where K is the number of environments, and T is the total timesteps in each task. Pt(() is ssec-spe with pt (t( E R for our pixel-based tasks and (t( E [0, 1] in our control tasks.
Average Performance. The average performance is evaluated the policy at time across all K K tasks by Pk ((( K E Pi((). By default, the average performance is calculated on the final policy when = K x T. For FAME, this metric is calculated on the meta learner.
Forward Transfer (FT): The forward transfer is defined as the normalized area between the training curve of the considered algorithm and the baseline. Namely, FT
R ΣK FITr, with
FTr; AUC, - AUC
A
f (l()). (9)
AUC, A (1-1)-A
(((d)dt
AUC, A ((-1)
1 AUC
To evaluate this metric in pixel-based tasks, we first normalize pt(t) in each task to ensure AUC, e [0, 1] and then we calculate a normalized metric of the forward transfer.
Forgetting (F): Forgetting is the performance difference between the policy at the end of task and after the whole sequence of tasks. Namely, F = k ΣK F, with F. pr(i T) pi(K TT).
Experimental Setup. (1) For the pixel-basce tasks, we perform experiments on MinAtar and Atari games (Young & Tian 2019 Bellemare et al.) 2013). MinAtar is a commonly used continual RL benchmark (Anand & Precup 2023]Tang et al. 2025 with relatively lighter computational require-
ments, allowing to sweep range of hyperparameters, report statistical results averaged over 30
7

---

Published as a conference paper at ICLR 2026
seeds, and probe the mechanism and advantages of our proposal. We employ DQN (Mnih et al. 2013) in breakout, freeway, and spaceinvaders games, and run for 3.5M steps by randomly choosing each of the three games every 500k steps, i.c., 7 tasks in each sequence, We study two sequences of Atari games following (Malagon et al. 2024). with the 10 playing modes of the ALF//pace/nvadeer v5 environment and playing modes of the Al.F/Freeway-s5 environment. We run Proximal Policy Optimization (PPO) (Schulman et al. 2017) and each task is run for 1M timesteps.
(2) For the
robotics arm manipulation tasks, we employ Meta-World (Yu et al., 22020) a standard and estab- lished benchmark commonly employed in continual RL (Malagon et al.] 2024 Chung et al., 2024] Ahn al.) 2025). Note that the traditional Continual World (Wolezyk et al.) 2021) is built on top of Meta- Meta-Worid with specific sequence of tasks (e.g.. CW 10 or CW20). Instead, we evaluate on randomly selected task sequences following (Chung et al., 2024) Meta-World (see Ap- pendix G.T). offering more flexible and robust evaluation. We deploy the Soft Actor-Critic (SAC) algorithm (Zhang et al. 2020: Haarnoja et al. 2018a) with 1M timesteps on each task with 10 tasks in each sequence.
4.1 PIXEL-BASED ENVIRONMENTS WITH DISCRETE ACTION SPACES
Comparison Methods. (1) For MinAtar, we follow (Anand & Precup 2023) and compare
our FAME approach with DQN (Reset), DON-Finetune (Finetune), DQN with large buffer (LargeBuffed) fer), DQN with multi-heads that knows the task identity (HeaddH PI-DQN (Anand & Precup 2023). Both fast and meta learners in our FAME method employ the same DQN architecture. (2) For Atari games, add PackNet (Mallya & Lazebnik 2018) and P Progress i te (Rusu et al. 2016) as baselines. Both fast and meta learners in FAME adopt the same PPO architecture. Except for Finetune, we reset the parameters of all baseline methods when each new environment arrives. By contrast, FAME applies the adaptive meta warm-up among fast, random initialization, and initial learning with behavior cloning regularization in Section 3.1.2. More details of our experimental setup and hyperparameters are given in Appendix F.1
Main results MinAtar on Average Performance (Avg. Perf), Forward Transfer (FT), and Forget- ting. Results (Mean + SE) are averaged over 10 sequences, each with 3 seeds. 1 denotes a positive metric (more is better), while 4 is a negative one (less is better). Reset is the baseline for evaluating FT. Forgetting is nor- malized by the standard deviation in each task.
Table 1:
Ave. Perf f Spaceinvader
Method
FT 1
Forgetting |
Breakout
Freeway
Reset
6.51 + 1.67
3.29 + 3.09
0.74 0.38
0.00 + 0.00
1.31 + 0.23
Finetune
10.62 + 2.75
4.95 + 2.92
0.89 0.49
0.13 + 0.03
1,26 4: 0.32
Multileaad
6.85 1 1.76
3,26 1. 2.99
0.94 0.42
-0.01 1 0.00
1.25 1 0.22
Buffer
10.71 + 2.84
3,24 2.91
1.16 0.59
0.16 + 0.02
1.65 + 0.33
PT-DQN
0.39 0.02
0.00 0.00
0.00 0.00
0.07 0.02
1.64 + 0.02
FAME
14.54 0.58
18.72 1 0.52
1.69 0.17
0.16 0.03
0.72 1 0.13
Main Results: MinAtar. Table D summarizes the metric scores of all methods, demonstrating that FAME consistently outperforms other baselines in improving knowledge transfer and retaining all knowledge to mitigate catastrophic forgetting. Notably, for the average performance. F FAME is most stable with minimal variations among all algorithms except for PT-DQN, for which the permanent value finction (i.e., the counterpart of the meta learner) in (Anand & Precup 2023) has limited capability to retain the knowledge and thus keeps almost zero average performance. Regarding the forward transfer, LargeBuf fer performs similarly to FAME as storing more past knowledge also contributes to adapting to a known environment. A detailed sensitivity analysis about X, Warm-Up step L, and Estimation step N is provided in Appendix F22
Performance of Knowledge Integration. Figure (left) presents the average performance of all methods at the end of cach task, reflecting the tendency of catastrophic forgetting. It turns out that FAME achieves the highest average performance in the whole training process in most cases, vali- dating the effectiveness of the meta learner in retaining information through knowledge integration.
Performance of Adaptive Meta Warm-Up in Knowledge Transfer. Figure 2 (right) exhibits the warm-up selection ratio when the agent encounters different types of arriving environments, reveal- ing the adaptive mechanism. Concretely, if the agent has already stored relevant data previously in M about the arriving environment, the meta warm-up is chosen with 95.1 1% probability. When
8

---

Published as a conference paper at ICLR 2026
new task occurs against the agent's knowledge, the random initialization is more commonly selected in the adaptive meta warm-up. The learning curves of all algorithms are given in Appendix
Main Results: Atari games. We further compare FAME with more baselines two sequences of Atari games: ALE/Spacennvaders-vS and ALE/Freeway-v5. Table 2 showcases that FAME outperforms all baselines in terms of average performance and for- ward transfer. While Finetune also benefits from forward transfer, espe- cially on SpaceInvader, where its for- ward transfer is comparable to FAME, we hypothesize that this advantage arises from the similar underlying envi- ronmental dynamics and objectives in each sequence of tasks, despite the distinct playing modes. We also provide the learning curves of
Table 2:
Main results Atari games Average Perfor- mance (Avg. Perf)) and Forward Transfer (FT). Results (Mean + SE) are averaged over seeds. The Forgetring metric is omitted as P PackNet and progressiveNee store past model param- eters have zero forgetting. Reset the baseline for FT.
Freeway Avg. Perf t
Invaelr Avg. Perf
Method
FT t
FT
Reset
0.16 + 0.18
0.00
0.10
0.22
0.00
Finetune
0.21 0.17
0.53
0.61 0.41
0.65
ProgressiveNet
0.39 0.25
0.21
0.61
0.03
0.06
PackNet
0,41 + 0.24
0.18
0.47 + 0.06
0.17
FAME
0.90 0.12
0.68
0.96 0.02
0.63
all considered algorithms in Appendix F3
4.2 ROBOTIC MANIPULATION TASKS WITH CONTINUOUS ACTION SPACES
Comparison Methods. (1) Reset; (2) neTuee,; (3) Average: we average the Temporal Dif- ference (TD) targets among all past tasks in evaluating the critic loss; (4) PackNet; (5) FAME-KE KL: apply knowl- edge integration under Wasserstein distance Eq. g (Method 2). All methods share the same net- perform the policy evaluation for 10 episodes among a random policy, the preceding fast policy, and the meta policy. Then we initial- ize the fast policy with the best-performing one. The collected data in evaluation is also stored in the fast learner's replay buffer F without incurring additional interaction costs with the environment. More experimental details our FAME methods (5) and (6) are provided Appendix G.I
employ knowledge integration under KL in Eq. 7 (Method 1); (6) FAME-WW
work architecture as standard SAC. In adaptive meta warm-up,
Main Results. As exhibited in Table 3 both FAME- KL and FAME-WD outperform most baselines significantly. PackNet achieves zero forget- ting by storing (masks of) past model parameters and knowing the task identifiers and num- ber in advance, which is less practical in real scenarios. By contrast, the superior forward transfer of FAME indicates that the adaptive meta warm-up fosters the fast learner to adapt to new environment by leveraging prior knowledge from the meta learner Moreover, the highest average performance and almost minimal
Table Main results on Meta-World on Average Performance (Ave. Perf), Forward Transfer (FT), and Forgetting averaged over 3 sequences. Results are presented as averages and standard errors across 10 seeds.
Methods
Avg. Perf 1
FT
Forgetting J
Reset
0,093 + 0.017
0.000 = 0.000
0.710 + 0.030
Finetune
0,037 : 0.011
-0,265 + 0.028
0.427 = 0.033
Average
0,013 + 0.007
-0.530 0.024
0.070 + 0.022
PackNet
0.491 - 0.025
-0.194 0.018
0.000 = 0.000
FAMF-KI.
0.733 + 0.026
0.022 + 0.015
0.073 + 0.019
WD
0.767 + 0.024
-0.003 0.014
0.023 1 0.015
Breakout
Spaceinvaders
Freeway
Warm-Up Selection Ratio (%)
Avenige Performnn Performunce
Av usuge verformncce
Averago Per foctuancc
100
Random 96.1%
Percentage (%%
800
2.2%
Meta
Fast
60
19.3%
Task
0.0%
DON-Rwat
Partee
77:0ON
FAME
Environment
Old
Figure 2: (Left) Average performance the policy across each task across sequences on MinAtar. Results are averaged over seeds. The vertical lines at each point represent the standard errors. (Right) The selection ratio among three warm-up strategies when the arriving environment previously encountered novel.

---

Published as a conference paper at ICLR 2026
forgetting of our FAME approaches highlight that the meta learner consolidates all past knowledge by conducting incremental updates knowledge integration.
Performance of Knowl- edge Transfer. To compre- hensively verify the knowl- edge transfer benefit due to the adaptive meta warm-up in FAME, we present the performance profile (Agar- wal et al.] 2021) that re- flects the overall perfor- mance of the fast learner over the whole sequence of tasks. Figure 3 (left) sug- gests that both FAME meth- ods outperform all base- lines, substantiating that the meta learner effectively consolidates knowledge and enhances knowledge transfer. Learning curves and performance pro- files for each sequence are also given in Appendix G.2
FAME: FAMEWD
PackSet
0.8
0.6
0.6
Sucees Rux
9
Scecou
Figure 3: (Leftt) Performance profile of the fast Icarner across tasks, where the y-axis shows the proportion of tasks that achieve success rate greater than or equal to the x-axis value. (Right) Average performance by evaluating average success rates past tasks across 10 seeds.
Performance of Knowledge Integration. To reflect the tendency of the catastrophic forgetting of FAME, we also illustrate the average performance of the meta learner at the end of each task and then take the average over 3 sequences. As suggested in 3 F FAME-ER KL and FAME-M -MD enjoy the highest average performance over time across all encountered tasks.
DISCUSSIONS AND CONCLUSION
In this paper, contribute to the foundation of continual RL and develop novel dual-leamer algorithm to conduct the knowledge transfer and integration via the coupled update of fast and meta knowledge learners. Two ideas might be worth reemphasizing here. (1) Adaptively selecting practical prior knowledge (e.g., via the hypothesis test) is crucial to overcoming the negative transfer issuc. (2) Deriving an incremental update rule based on existing multi-task learning objectives is necessary to connect continual and multi-task RL.
Limitations and Future Work. In this study, meta learner is utilized retain all knowledge. Alternatively, it is also possible to learn an effective latent representation that can not only distill all knowledge but also perform efficient reasoning to guide the adaptation yond the proposed adaptive meta warm-up. more techniques in knowledge transfer can be explored in the future, such as context embedding. Lastly, extending our algorithm to the full continual RI. context without knowing the task boundary (possibly by developing online one-vs-all hypothesis test) or assuming the same state and action spaces (Hu et al. 2025) is also valuable for practitioners.
new environment. Be-
Ethics Statement. This study focuses theoretical and algorithmic aspects of continual rein- forcement learning, without involving human subjects, personal data, or sensitive applications. We acknowledge that continual learners deployed in real-world systems could, without careful control and monitoring, exhibit unexpected behaviors due to forgetting or transfer misalignment. However, study remains purely theory- and algorithm-oriented, and therefore do not foresee any direct ethical concerns arising from this research.
ACKNOWLEDGEMENNSS
Linglong Kong was partially supported by grants from the Canada CIFAR AI Chairs program, the Alberta Machine Intelligence Institute (AMII), the Natural Sciences and Engineering Council of Canada (NSERC), and the Canada Research Chair program from NSERC. Hongming Zhang was supported by the National Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences (Grant No. ESSPFZ0112). We also thank all the constructive suggestions and comments from the reviewers and area chairs.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
David Abel, Yuu Jinnai, Sophie Yue Guo, George Konidaris, and Michael Littman. Policy and value transfer lifelong reinforcement learning. In International conference on machine learning, pp. 20-29. PMLR, 2018.
David Abel, André Barreto, Benjamin Van Roy, Doina Precup, Hado van Hasselt, and Satinder Singh. A definition continual reinforcement learning. Advances in neural information process- ing systems, 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural informa- fion processing systems, 34:29304-2993 29320, 2021.
Hongjoon Ahn, Jinu Hyeon, Youngmin Oh, Bosun Hwang, and Taesup Moon. Prevalence neg- ative transfer in continual reinforcement learning: Analyses and simple baseline. In The Thir- leenth International Conference on Learning Reprexentations, 2025.
Nishanth Anand and Doina Precup. Prediction and control in continual reinforcement learning. Advances in neural information processing systems, 2023.
Marein Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. Advances neural information processing systems, 30, 2017.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp.
PMLR, 2017.
Fengshuo Bai, Hongming Zhang, Tianyang Tao, Zhiheng Wu, Yanna Wang, and Bo Xu. Picor: Multi-task deep reinforcement learning with policy correction. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 37, pp. 6728 6736, 2023.
André Barreto, Shaobo Hou, Diana Borsa, David Silver, and Doina Precup. Fast reinforcement learning with generalized policy updates. Proceedings of the National Academy of Sciences, 117 (48):30079-30087, 2020.
Mare G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of artificial intelligence research, 47: 253-279, 2013.
Glen Berseth, Zhiwei Zhang, Grace Zhang, Chelsea Finn, and Sergey Levine. Comps: Continual meta policy search. International Conference Learning Representations, 2022.
Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Task- agnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint M.Xi:22005.14495 2022.
Rich Caruana. Multitask learning. Machine learning, 28:41-75, 1997.
Yash Chandak, Georgios Theocharous, Shiv Shankar, Martha White, Sridhar Mahadevan, and Philip Thomas. Optimizing for the future in non-stationary mdps. In International Conference Ma- chine Learning, pp. 1414-1425 PMLR, 2020.
Feng Chen, Fuguang Han, Cong Guan, Lei Yuan, Zhilong Zhang, Yang Yu, and Zongzhang Zhang Stable continual reinforcement learning via diffusion- based trajectory replay. ICLR 2024 Work- shop on Generative Models for Decision Making, 2024.
Wesley Chung, Lynn Cherif, Doina Precup, and David Meger. Parseval regularization for continual reinforcement learning. Advances Neural Information Processing Systems, 37:1179937-127967, 2024.
Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning mod- ular neural network policies for multi-task and multi-robot transfer. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2169-2176 IEFE, 2017.

---

Published as a conference paper at ICLR 2026
Thang Doan, Mehdi Abbana Bennani, Bogdan Mazoure, Guillaume Rabusseau, and Pierre Alquier. A theoretical analysis of catastrophic forgetting through the ntk overlap matrix. In International Conference Artificial Intelligence and Statistics, pp. 1072-1080. PMLR, 2021.
Shibhansh Dohare, Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard S Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
Elizabeth Dolan and Jorge Moré. Benchmarking optimization software with performance pro- files. Mathematical programming, 91:201 223,, 2002.
Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdi- nov. Off-dynamics reinforcement learning: Training for transfer with domain classifiers. Inter- national Conference Learning Representations (ICLR), 2021.
Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in reinforcement learning agent. In Proceedings of the fifth international joint conference on Autonomous agents and mul- tiagent systems, pp. 720-727, 2006.
Norman Ferns and Doina Precup. Bisimulation metries are optimal value functions. In UAI, pp. 210-219, 2014.
Chelsea Finn and Sergey Levine. Deep visual foresight for planning robot motion. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 2786-2793. IEEE, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta- leta-learning for fast adaptation of deep networks. In International conference machine learning, pp. 1126- 1135. PMLR, 2017.
Jean-Baptiste Gaya, Thang Doan, Lucas Caccia, Laure Soulier, Ludovic Denoyer, and Roberta Raileanu. Building a subspace of policies for scalable continual learning. 2023.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471-1550, 2004.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. PMLR, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint arXiv:111..05905. 2018b.
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learn ing Representations, 2018
Jifeng Hu, Sili Huang, Li Shen, Zhejian Yang, Shengchao Hu, Shisong Tang, Hechang Chen, Yi Chang, Dacheng Tao, and Lichao Sun. Tackling continual offline rl through selective weights activation on aligned spaces. Advances in Neural Information Processing Systems, 2025.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in neural information processing systems, 31, 2018.
Christos Kaplanis. Murray Shanahan, and Claudia Clopath. Continual reinforcement learning with complex synapses. In International Conference on Machine Learning, pp. 2497-2506. PMLR, 2018.
Christos Kaplanis, Murray Shanahan, and Claudia Clopath. Policy consolidation for continual rein- forcement learning. International Conference on Machine Learning, 2019.
Samuel Kessler, Jack Parker- Philip Ball, Stefan Zohren, and Stephen Roberts. Same state, different task: Continual reinforcement learning without interference, In Proceedings the Conference Artificial Intelligence, volume 36, pp. 7143-7151, 2022.
the AAAI
12

---

Published as a conference paper at ICLR 2026
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401- 1476, 2022.
Dharshan Kumaran, Demis Hassabis, and James McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated. Trends in cognitive sciences, 20
(7):512-534, 2016.
Zichen Liu, Guoji Fu, Chao Du, Wee Sun Lee, and Min Lin. Continual reinforcement learning by planning with online world models. International conference on machine learning. 2025.
Mikel Malagon, Josu Ceberio, and Jose A Lozano. Self-composing policies for scalable continual reinforcement learning. In Forty-first International Conference on Machine Learning, 2024.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference Computer Vision and Puttern Recognition, pp. 7765-7733, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning, nature, 51 (187550):529-5333 2015.
Aneesh Muppidi, Zhiyu Zhang, and Heng Yang. Fast trac: A parameter-free optimizer for lifelong reinforcement learning. Advances in Neural Information Processing Systems, 37:51 199-5199,,,,,, 2024.
Victor M Panaretos and Yoav Zemel. Statistical aspects of wasserstein distances. Annual review of statistics and its application, (11))::0333111 2019.
Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning, International Conference Learning Representations, 2016.
Janarthanan Rajendran, Aravind Srinivas, Mitesh M Khapra, Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. arXiv preprint ArXiv: 111.v:1510.02879, 2017.
Aaditya Ramdas, Peter Grünwald, Vladimir Vovk, and Glenn Shafer. Ghme-theoretic statistics and safe anytime-valid inference. Statistical Science, (8(())8(())6000 2023.
Andrei Rusu, Sergio Gomez Colmenarejo, Caglar Gulcehre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distil- lation. arXiv preprint arXiv: 2015.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv: (eoe::66604777 2016.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06377 2017.
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context- based representations. In International Conference on Machine Learning, pp. 1767-9779.PMLR. 2021.
Yixiang Sun, Haotian Fu, Michael Littman, and George Konidaris. Knowledge retention for contin- ual bosel-based reinforcement learning. International conference machine learning, 2025.
13

---

Published as a conference paper at ICLR 2026
Richard S Sutton and Andrew G Barto. Reinforcement learning: An Introduction. MIT press, 2018.
Hongyao Tang, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Glen Berseth. Mitigating plasticity loss in continual reinforcement learning by reducing churn. International conference on machine learning, 2025.
Yunzhe Tao, Sahika Gene, Jonathan Chung, Tao Sun, and Sunil Mallya. Repaint: Knowledge transfer in deep reinforcement learning. In International conference on machine learning, Pp- 10141-10152. PMLR, 2021.
Yee Tch, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. Advances in neural information processing systems, 30, 2017.
Andrea Tirinzoni, Andrea Sessa, Matteo Pirotta, and Marcello Restelli. Importance weighted trans- fer of samples in reinforcement learning. In International Conference on Machine Learning, pp- 4936-4945. PMLR, 2018.
Yi Wan, Ali Kahimi-Kalahroudi Janarthanan Rajendran, Ida Momennejad, Sarath Chandar, and Harm H Van Seijen. Towards evaluating adaptivity model-based reinforcement learning meth- ods. In International Conference Machine Learning, pp. 22536-22561 PMLR, 2022.
Maciej Wolezyk, Michal Zajkac, Razvan Pascanu, Lukasz Kuciñski, and Piotr Mitoš. Continual
world: A robotic benchmark for continual reinforcement learning. Advances in Neural Informa- fion Processing Systems, 34:28496-28100, 2021.
Maciej Wolczyk, Michal Zajac, Razvan Pascanu, Lukasz Kucinski, and Piotr Milos. Disentangling transfer in continual reinforcement learning, Advances in Neural Information Processing Systems, 35:6304-6317, 2022.
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences. In Conference Lifelong Learning Agents, pp. 838 855. PMLR, 2022.
Yijun Yang. Tianyi Zhou, Jing Jiang, Guodong Long, and Yuhui Shi. Continual task allocation in meta-policy network via sparse prompting. 2023.
Kenny Young and Tian Tian. Minatar: An atari-inspired testbed for thorough and reproducible reinforcement learning experiments. arXiv preprint arXiv: 03/.00177,.. 176 2019.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pp. 1094-1100. PMLR, 2020.
Hongming Zhang, Tianyang Yu, and Ruitong Huang. Combine deep q-networks with actor-critic. In Deep Reinforcement Learning: Fundamentals, Research and Applications, pp. 213-245. Springer, 2020.
Hongming Zhang, Tongzheng Ren, Chenjun Xiao, Dale Schuurmans, and Bo Dai. Provable repre- sentation with efficient planning for partially observable reinforcement learning. In Proceedings of the 41st International Conference on Machine Learning, pp. 59759-59782, 2024a.
Hongming Zhang, Chenjun Xiao, Chao Gao, Han Wang, Bo Xu, and Martin Müller. Exploiting the replay memory before exploring the environment: enhancing reinforcement learning through empirical mdp iteration. Advances in Neural Information Processing Systems, 17.86688-85697 2024b.
Tiantian Zhang, Kevin Zehua Shen, Zichuan Lin, Bo Yuan, Xueqian Wang, Xiu Li, and Deheng Ye. Replay-enhanee continual reinforcement learning. Transaction of Machine Learning Research, 2023.
14

---

Published as a conference paper at ICLR 2026
Appendix
Table of Contents
Related Work
16
B Preliminaries: Definition of Distribution Drift and Catastrophic Forgetting in Deep
Learning
17
heoretical Results
17
Incremental Q- value-based Meta Learner Update in Proposition
17
Proof Proposition
Proof Proposition
D Policy-based FAME Algorithm
22
E
Discussion of the FAME Framework
23
Experiments: Value-based Continual RL with Discrete Action Space
24
F. Details Experimental Setup and Comparison Methods
F2
Ablation Study MinAtar
25
F3
Learning Curves Analysis for Plasticity Ability
27
G
Experiments: Policy-based Continual RL with Continuous Action Space
29
Details of Experimental Setup and Comparison Methods
29
More Experimental Results and Details
30
The Use Large Language Models (LLMs)
Additional Results
33
I.1
More Baselines the Three Sequences Tasks in Meta- World
33
Additional Sequence of Task: CWIO
36
15

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Measure of MDP Distance and Catastrophic Forgetting. Although in a different perspective, Bisimulation metrics (Ferns & Precup. 2014) provides additional evidence to support the strategy that optimal value functions can be naturally used to measure the state similarity.
Continual RI., Continual RL (Khetarpal et al. 2022 Abel et al. 2023) addresses the challenges in learning a sequence of decision-making tasks, particularly balancing the stability (i.e., mitigating catastrophic forgetting) and plasticity (i.c., rapid adaptation to a new environment). Below, we cat- egorize the existing literature into two groups based on their focus and make a detailed comparison with our contribution in this study.
Catastrophic Forgetting. (1) The replay- -ased approach is commonly applied to mitigate forget- ting. A replay-basee recurrent methodology was initially proposed for task-agnostic agents (Cac-) Cia et al. 2022). RECALL (Zhang et al. 2023) leverages adaptive normalization on approxi- mate targets and policy distillation on old tasks to enhance generality and stability. Generative replay (Chen et al. (2024) was recently proposed using the diffusion model to memorize the high- return trajectory distribution of each encountered task. [Sun et al. (025)) continually learns a generative model for experience replay without storing the past data within a model-based RL framework. Similarly, Liu et al. (222) learns an online world model and acts by planning via model prediction control to construct a unified world dynamics to handle the catastrophic forget- ting issue. (2) The second branch is regularization-basee from distinct perspectives. Earlier, the behavioral cloning was investigated across historical policies in (Woleyyk al. 2022). Inspired by complex synapses, (Kaplanis et al., 2018 2019) developed the policy consolidation strategy by si- multancously remembering the agent's 's policy at a range of timescales and regularizing the current policy by its own historical experience. Sparse prompting (Yang et al., 2023) imposes regular- ization via dictionary learning to produce sparse masks as prompts, extracting a sub-network for each task from a meta-policy network. (3) Model expansion also serves promising direction
to investigate (Mallya & Lazebnik] 2018 [Malgonn et al.) 2024; Gaya et al., 2023). |aay et al.) (2023) builds the subspace of policies to consider the scalable continual RL, while pointing out the trade-off between the agent's 's size and the performance of continual learning. Malagon et al. (2024) uses growing policy neural network and applies the attention mechanism integrate the knowledge from the previous policies and the current state to "self-compose" an internal policy. Despite the effectiveness of model expansion- based approaches, the primary concerns lie in their high memory and inference costs due to the leverage of previous policies with specific aggregated strategies. For instance, the attention module is adopted in (Malagon et al. 2024). where the num- ber of parameters grows linearly and the theoretical computational cost is quadratic with respect to the number of tasks. Comparison: Our FAME approach with a dual-learner strategy can be viewed lying at the intersection of the three predominant paradigms. In the knowledge integra- tion, minimizing catastrophic forgetting requires an experience replay from the meta buffer. In the knowledge transfer, a behavior cloning regularization is possibly adopted to guide the exploration if the meta learner is chosen as the best in the adaptive meta warm-up framework. Although em- ploying dual-learner strategy, FAME retains the fixed model sizes of both fast and meta learners to avoid the stability issue, which is distinct from other model expansion approaches.
Knowledge Transfer and Loss of Plasticity. The effectiveness of knowledge forward transfer determines the loss of plasticity, reduced capability to rapidly adapt to new environment (Abel et al. 2018 Dohare et al. 2024; Wolczyk et al.) 2022 Tao et al., 2021). Within the continual RL literature, large number of studies mainly focus on the knowledge transfer and plasticity capability through distinct perspectives. Xie & Finn (022) improves the forward transfer and mitigates the loss of plasticity by selectively identifying the most relevant samples for the new task using learnable importance weights. The value function decomposing approach (Anand & Precup 2023) is proposed to perform an interplay between fast and slow learning at various levels for value-based continual RL. with a discrete action space to address the loss of plasticity. Through the lens of optimization, Parseval network (Chung et al. 2024) imposes orthogonality constraints to mitigate interference, while Muppidi et al. (2024) employs parameter-free online convex op- timization to retain plasticity. A recent work by (Tang et al.) 2025) establishes the connection between plasticity and the churn via the Neural Tangent Kernel (NTK) matrix. Negative transfer issue (Ahn et al.) 2025) was revealed in the adaptation to a new task due to the task dissimilarity, amplifying the loss of plasticity. As such, Reset and Distill (R&D) (Ahn et al. 2025) explicitly
16

---

Published as a conference paper at ICLR 2026
resets the policy in each new environment to avoid the negative transfer. Comparison: In terms of the knowledge transfer, it is not necessary that the reset is always the best choice, especially when the task similarity often exists. By leveraging the one-vs-all hypothesis test, we propose the adap- tive meta warm-up approach that selectively discriminates the most effective weight initialization and warm-up strategy to facilitate the knowledge transfer and reduce the loss of plasticity.
Transfer, Multi-task, and Meta RL. The knowledge transfer phase in our approach is closely linked with transfer RL, where the accrued knowledge can be transferred through representa- tion (Rusu et al., 2016; Devin et al. 2017), learned models (Finn & Levine [2017] Eysenbach al.
2021), and network weights (Fernández & Veloso 2006) or experience (Andrychowicz et al. 2017
Tirinzoni al. 2018
Xie & Finn 2022). Although the set of tasks should be learned simul tane-
ously, multi-task RL
also integrates the component of knowledge transfer, such as (Caruana 1997]
Sodhani et al.] 2021 Teh al.; 2017] Yang al., 2023 Rusu al. 2013 Hausman et al.] 2018).
As such, transfer RL has been an underpinning building block for both multi-takk and continual RL, e.g., the explicit knowledge transfer in our dual-learner algorithm. As opposed to multi-task RL, the incremental or sequential learning nature of continual RL makes it more challenging to minimize catastrophic forgetting. However, our study shows that minimizing catastrophic forgetting can be, in principle, equivalent to the objectives in multi-task learning. Meta RI. (Finn et al. 2017) can be seen as an extension or generalization of multi-task RL, with explicit mechanisms for fast adaptation and few-shot learning, and is also closely linked with continual RL. Continual Meta-Policy Search (CoMPS) (Berseth et al. 2022) probes the setting of meta-training in an incremental fashion, extend- ing meta-RL to a continual learning scenario. Comparison: The fast and meta learners adopted in our FAME approach intermingle the key techniques of transfer, mulli-task, and meta RL, illuminat- ing their deep connections within our algorithmic framework. The interplay of knowledge transfer and knowledge integration via the coupled updates between fast and meta learners simultaneously tackles the involved challenges, exhibiting promising solutions address continual RL.
B PRELIMINARIES: DEFINITION OF DISTRIBUTION DRIFT AND CATASTROPHIC FORGETTING IN DEEP LEARNING
We first introduce the concept of drift in the process of learning a parameterized function f from the source data distribution Ts with the dataset D,g the target data distribution Ty with the dataset Drr- After learning f on the source dataset Drs, we obtain the estimated function fry. Then we apply the same model architecture on the target dataset Drz with any learning algorithms, and finally we evaluate the drift of the attained Foz via 8 defined as (Doan et al. 2021
8TH (X*))= = (Frn(() 1
(r)( (
(10)
(a.v)eDre
Based on the definition of drift, we define the vanilla catastrophic forgetting A T8- as
Ts Tt (x**)= 188 +r (12 = Σ (x) ((ox))Fr((() (x))
(11)
(e.N)EDT
where the catastrophic forgetting can further simplified as A -8
(X*8) (w
in the Neural Tangent Kernel (NTK) regime (Doan al.) 2021 Jacot al. 2018). allowing the
proposal of new continual learning approaches. In deep learning, minimizing the catastrophic for- getting A T is equivalent to minimizing a weighted drift in terms of the prediction function f with the weights determined by the dataset. The preliminary results in deep learning largely inspire us to explore the foundations in continual RL.
C THEORETICAL RESULTS
C.I INCREMENTAL. Q- Q-VALUB-BASED META LEARNER UPDATE IN PROPOSITION 3
In Proposition 3 we provide an efficient incremental update rule of the meta learner to minimize the principled D-value-basse catastrophic forgetting that we define in Definition
17

---

Published as a conference paper at ICLR 2026
Proposition 3 (Incremental Q- Value-basee Meta Learner Update), Let do be &g loss in Eq. 7 in
Definitioon 2 Minimizing Q-value-bar based catastrophic forgetting in Eq. 3 equivalent to:
R-1
[(0M)
27
(ee- 25 2
Q = arg min @N
Σ 1
+ w
(12)
Proof. Step 1: Optimality Condition. Recap w, (s. a) = 14: (1:(1)) (s) We aim to minimize
the Q-value-basee catastrophic forgetting defined in Eq.
arg min ΣΣ (s. (Q)(s.a)- Q (s.a))
k
ON
For each and o, by taking the derivative of the objective in Eq. regarding Q. the first-order optimality condition is
w (s a) (s,a) (((s,a)) =0
(13)
By rewriting the two optimality conditions regarding Q and 1 attain that
Σ -i-1 w ((, (s)
(s, Σ w (s. a)Q,(s 1j (s,
(),a)
Step 2: Incremental Update Rule. For brevity, we employ the expectation operation E,,, Based on the two optimality conditions above, we can derive the following incremental update rule:
(Qi- )
Q arg min G
27 + 1 @M)
+ [Q6-
arg min QN
(14)
1-1
(Q: - - (0 2M) +2 - (0h-1 - 0xa))
1
E
En
(Q: (Q:-QM1) 2
+ (OM 2M ) )/+
k-1
2
Σ
@ - 2N (s,
Σ
wi (s, a) () (s, a))
9,6
i=1
(Q: - -Q/ + (Q 2N¹) )
E w.
R-1
[QM-- 2M 3
C +
E End
1=1
where (a) holds as Σi=1 wo
( a) (Qi 1
the optimality condition for
Q in Eq. 13 of Step C' Σ _ (Q.-Q.M )²²
a constant in terms of Q Putting
all together into Eq. 14 have
Q arg min
Rng Σ
[ -Qt²²²
-
F (Qx (Qx=)N))
QM
1=1
18

---

Published as a conference paper at ICLR 2026
C.2 PROOF OF PROPOSITION
[Incremental Softmax Q-Value-based Meta Learner Update] Denote Th M (als) exp QQ (als)/r) Ea exp (Q (a))s)/r). After softmax policy transformation, the Q-value- based meta learner incremental update is rewritten as
Proposition
M log
Q arg min QN
E m1
---1
E W
log nt
arg max Q
E S
[lg
T MM
Proof. We rely on the softmax transformation to transfer a meta Q function to meta policy. As such, the policy-based catastrophic forgetting in Eq, 4 when adapted from value-based continual RL and equipped with KI. divergence as d can be expressed as
M = arg min
ΣΣ 2m
M4 (s) KL (11s))))) (1+)]
(15)
ON
where Th. (a|s) = exp (or ( (als)/T) E.. /Ew
(a'ls) (d'ls)/T). By the definition of the KL di- exp (a'sss)
vergence, we can rewrite the objective function in Eq.
15 as an incremental update rule:
(alss
Q arg min
Σ
(als)
p
=M (a|s)
π
arg min
E
log
MM 1
QN
Qi 7 M
Qx
arg min QN
log
k-1
+ 22
log M
M M
k-
M log k- |
7Q=
arg min QN
E
+ E
log
C
T 1--1
π Q
arg min
E
log M 7
+ E
log
(16)
M
QN
arg min QM
log MM 1k
log MM W
arg
E my
m M [lg
(17)
where C = -/-= E
log 2.
is a constant and is independent Q Although it may be
w trivial to keep the form of Eq. 16 it emphasizes an incremental update rule of Q based on Q.M. (xM)) and Qs (x.)). Eventually, this minimization leads to an Maximum Likelihood estimation regarding the meta learner @ in Eq. 17 on a mixture of state-action distribution of all encountered environments up to k.
C.3 PROOF OF PROPOSITION 2
Proposition 2 [Incremental Policy-based Meta Learner Update under Wasserstein Distance] Con- sider d, to be the squared 2- Wasserstein distance in Eq. Bq of Definition 2 and the policy repre- sented as independent (multivariate) Gaussian distribution over the action a. Minimizing policy-
19

---

Published as a conference paper at ICLR 2026
based catastrophic forgetting in Eq. 4 is equivalent to:
k RM
arg min
ΣΣ (s) (Th (-16), ml (+1a)) + E " (s) (ak (1ss), (1ss), ((0)))
RM
Proof. Recap the objective of the policy-based catastrophic forgetting based on Eq. under squared 2- Wasserstein distance:
k
arg min 2
(s)
i=1
where the squared 2- Wasserstaii distance between two Gaussian distributions p and g has closed- form solution:
W = 11p1-1,-,,00 -val22 + tr Ep + +Zg - 2 2//2) 1/2 )
(18)
with the two Gaussian distributions denoted by (up, Ep) and (ug Eg)) In particular, when the policy is represented as independent (multivariate) Gaussian distribution across the action implies that 2p and Mg are diagonal (i.e., variables are independent), then the squared 2- 2-Wasserstain distance in Eq. 18 can be further simplified as
W2 9) 11 Myy vyl22
11op agll2.
(19)
where on and 0g are the diagonal vector of Ep and Eg. respectively. Then, the objective of the policy-based catastrophic forgetting based Eq. can be simplified
M 7
arg min 2
Σ ot
Σ uo (s) (1(((s) D (s) lor(s) (
(20)
where TT(-(s) is represented as (multivariate) Gaussian distribution (V(ss(s), oz (s)), where (s((s) and of (s) are the mean (vector) and (the diagonal vector of) the variance. Similarly, The is repre- sented as (multivariate) Gaussian distribution N (u (s), ( (s))
Step 1: Optimality Condition. For each we take the derivative of Eq. 20 in terms of Di and
-M respectively. Consequently, it arrives at the following optimality condition:
k
M (s) - 1/ - (s)) =
(21)
Σ 1 (s) (a,(s) - (s)) 0.
(22)
Step 2: Incremental Update. We first rewrite Eq. 20 as
arg min
A (s)) - DM (+)l
-
- -M (+) En
- -6M (s)113
20

---

Published as a conference paper at ICLR 2026
k-1
1 =
14 (1)
-1)
- [ [
1=1
k-1
M (s) (1))(())- (3)] + (11l E (s)] +
k-1
ΣΣ>
Σ -0) (5), 25 (s))
(s) (112(()) - _ (3]1 + | (s) 23 (+)]1) +
Σ
(x(0))60))~~1111 - (s) 23 (s))
R-1
(s) (1]
(s) + (1)) - (s)1
where (a) holds due to the optimality condition Σ (s) (p)(s)) - = 0 we derived in
(s)
Eq. of of Step 1. Similarly, we can show this simplification regarding the variance:
of (M,1 + - (s)]
k-1
1 (s)
- (1o (s)1 + 11 M (s) - म. (s) +
2 (s)(o az (
M (s) 7. (s))
Σ 1 (5) (s)ox()) - (s)1l3 + 1 (s) 3x (s) +
2 Σ - (s) - 3% (s))
(s) (loa(s) - ( + 1oo
-o- (s)
where (b) holds due to the optimality condition Σ (x) - OM (s)) we derived in
Eq. 22 of Step 1. Putting all together, we have
MM
MT (s) (11)(5) - M + (s) - (s)1 + > - (s)
Tk
arg min M
aM
i=1
Σ
(
|1اي क5 (s)118) + Σ (s)lo(())) (5)11
21

---

Published as a conference paper at ICLR 2026
By removing the constant terms independent of 2 and o further have
(s) ( Σ ue -
arg min o
(s) T (s)1 + Σ 1e (
(s)1
E (F (10) (10)
arg min
This leads the incremental policy-based meta learner update under the squared 2- Wasserstein distance.
D POLICY-BASED FAME ALGORITHM
Algorithm Description. We first denote the fast buffer as F and 7² as the initialized policy. As suggested in Algorithm ! when the k-th environment arrives, we initialize the fast learner Te via the adaptive meta warm-up among the preceding meta learner MM the preceding fast learner TT-1 and a random learner x (reset strategy) within L steps. The adaptive meta warm-up makes full use of previous information to perform an effective knowledge transfer. Once the k-th task ends, the knowledge integration phase starts, when the meta learner TE is updated via Eq. (FAME- KL) or via Eq. [8 (FAME-MD) the data collected in the meta buffer M. The meta learner incrementally M leading to an updated meta learner MM
incorporates the knowledge from Ty. into x E-11
Algorithm 2 Policy-based FAME Update in the k-th Environment
1: Initialize: Fast Buffer F, Meta Buffer M, - 1 x0, Warm-Up Step I,, Estimation Step N.
2: # Knowledge Transfer: Adaptive Meta Warm-Up
3: Initialize m, in { TT) via Eq, Gafter L steps
4: for = to do
5: Observe S,, take action At, receive RG, observe S
6: Store (S,, A,, R2, 1e+1) 1) in
7: Update ne
8: if > T
N then
9: Method 1 (FAME-KL): Store (S,, At) in M # To Estimate We
10: Method 2 (FAME-WD): Store S; in M # To Estimate Pk Th
end if
12: end for
13: Reset JF
14:
# Knowledge Integration: Minimize Catastrophic Forgetting
Method (FAME-KL)): Update TH via Eq. on state-action pairs in M
Method ('AME-WD) Update o via Eq. on states in M
Computational Cost, Performance Variability, and Practical Guidance. The increased com- putational overhead of FAME-WD WD is negligible compared to FAME-KL KL when using standard Gaus- sian policy parameterizations, which is common RL in a continuous action space. As exhibited in Eq. 19 the simplification of Wasserstein distance leads to the same output of the policy network, ensuring a comparable computational cost to the KL divergence. We also remark that although the Wasserstein distance better captures the data geometry, the Gaussian policy may restrict the Wasser- stein distance's 's advantage. Therefore, FAME WD does not necessarily outperform FAME-KL. In complex environments, the task involves high distributional shift and the change of stochasticity, where capturing distribution geometry becomes critical. We believe that is when FAME-W- WD be- comes potentially superior to FAME-KI.
22

---

Published as a conference paper at ICLR 2026
E DISCUSSION OF THE FAME FRAMEWORK
Scalability of Storing Past Trajectories in the Meta Buffer. As described in the value-based FAME algorithm, at the end of training in each environment, we store a subset of state-action pairs to estimate the weight wi. Importantly the stored pairs constitute only small fraction of the to- tal training data for each task, approximately 1-2% in our experiments. Consequently, our FAME approach incurs negligible scalability overhead on the benchmarks considered. For fair compar- ison, we maintain the meta buffer the same size as the replay buffer used in standard RL, and our FAME approaches still exhibit superior performance across all considered benchmarks. Still, as shown Table 6 in Appendix F.22 enlarging the meta buffer by including more past trajectories generally leads to improved performance. In addition, as the meta learning update is in a supervised learning manner, which is much cheaper than the online RL training, even across a larger training set in the meta buffer. Overall, FAME achieves strong performance with minimal scalability concerns, highlighting its potential to effectively address key challenges in continual reinforcement learning.
Principled Strategy for Determining for the Policy Evaluation. Regarding the choice of the warm-up step L, the optimal L is indeed task-dependent. As increasing L will reduce the number of available samples for the online learning, given fixed total number of interaction steps, the optimal L needs to strike a balance between selecting the most effective prior knowledge and preserving sufficient data for online adaptation. However, the Meta-World experiments, the warm-up evalu- ation was simply fixed at 10 episodes because this choice already provides strong performance and significantly outperforms other baselines. In value- based continual RL, a longer warm-up duration L in the behavior cloning regularization does not necessarily improve overall learning. When the meta learner is selected to warm up the fast learner via the one-vs-all hypothesis test, the conclu- sion is that the prior knowledge from the meta learner provides better initialization than either random learner or the preceding fast learner, and is able to accelerate the training process especially in the early stages of training. However, as learning progresses, the fast learner gradually adapts to the new environment and can surpass the meta learner's s performance. In such cases, prolonged behavior cloning regularization tends to over- over-constrain the fast learner, thereby hindering adaptation and leading to the degradation in final performance (e.g., the non-monotonic effect in Table
Principled Strategy for Determining N. Regarding the number of stored trajectories N, its se- lection primarily depends on computational and memory constraints (i.e., the meta buffer size). As shown in Table of Appendix [22 increasing N generally improves performance but comes with higher costs in both memory (increasing the meta buffer size) and computation (meta learner update on a larger trajectory dataset) in the knowledge integration phase. However, as discussed previously, FAME achieves strong performance with minimal scalability concerns and we can increase N in practice to pursue superior performance as long as the memory budget is allowed.
Space and Computational Complexity of FAME. (1) In terms of space complexity, the dual learner system of FAME requires an additional memory copy as the fast learner (the normal learner in baselines such as Reset and Finetune), and an additional meta replay buffer with the same size as the buffer of the fast learner, which is scalable as the total sample memory is fixed and inde- pendent of the number of future tasks. (2) In terms of the computational cost, we employ the same number of agent's 's interaction steps with the environment for a fair comparison. In other words, the policy evaluation occurs at the cost of reducing the policy optimization update steps in total. The main additional computation cost is the updating of the meta- letanle which is efficiently conducted in a supervised learning way. For instance, we found it only takes around a couple of minutes across 200 epochs to update the meta-learners in MinAtar after one task finishes. This additional computa- tion overhead is negligible to the overall computation cost in the online training of RL algorithms.
23

---

Published as a conference paper at ICLR 2026
F EXPERIMENTS; VALUE-BASED CONTINUAL RL WITH DISCRETE ACTION SPACE
F.1 DETAILS OF EXPERIMENTAL SETUP AND COMPARISON METHODS
Metric Calculation. As average performance is the main metric in continual RL, we report the results for each environment. For the evaluation of forgetting, we first calculate the metric scores for each environment and then normalize them by their standard deviation across all methods in each environment. This standard normalization mitigates the influence of different reward scales of each game and allows us to average them across games to report a more comprehensive forgetting score.
Hyperparameters: MinAtar. Our implementation adapts from the released code (Anand & Precup 2023). For our FAME approach, after doing the line search of the hyperparameter in Sec- dio[F2 we choose the estimation step N 12000, i.e., the number of data be stored in the meta buffer M with size 100000 in each task, the policy evaluation step 600 for one-vs-all hypothe- sis test, and the warm-up step 50000 (10% of the training steps in each task). Note that we use L to represent the warm-up steps in the learning with the behavior cloning regularization, while in- troducing another notation n to denote the policy evaluation steps for adaptive warm-up. The ratio of stored data across the whole training steps in each task is /2000/500K 500k 2.4%. In the knowledge integration phase, we train the meta learner across 200 epochs from a 1 x 10-3 learning rate with decaying strategy. The learning rate for the fast learner kept as 1 x 10-5, the same the other variants of DQN baselines. Every time a new environment arrives, we clear the fast buffer J and reinitialize the parameters of all involved learners, except for DON-Finetune. For FAME, after the adaptive meta warm-up. we can automatically choose between a random initialization and an initial- ization from the preceding fast leamner with or without an additional behavior cloning regularization term for demonstration. We choose = 1 in Proposition I across all tasks.
Sequences of Tasks: MinAtar. We randomly select sequences of environments and then fix them for reproductivity. We run 3 seeds for each sequence of tasks.
1. (breakout', 'spaceinvaders', 'breakout', 'spaceinvaders', nspaceinvaders', 'freeway', 'breakout']
"spaceinvaders', 'breakout', 'breakout', 'spaceinvaders', sspaceinvaders', 'breakout', 'breakout"]
3. [ ('breakout', 'spaceinvaders', 'breakout', 'freeway', 'freeway', 'breakout', 'freeway']
4. ('freeway', 'breakout', 'spaceinvaders', 'breakout', 'breakout', 'breakout', 'spaceinvaders')
5. [ ('freeway', 'freeway', 'spaceinvaders', 'spaceinvaders', 'breakout', 'breakout', 'freeway']
('freeway', 'spaceinvaders', 'freeway', 'freeway', 'breakout', 'spaceinvaders', 'breakout']
('freeway', 'spaceinvaders', 'breakout', 'freeway', 'spaceinvaders', 'freeway', 'breakout']
[ ('breakout', 'spaceinvaders', 'freeway', 'breakout', 'spaceinvaders', 'freeway', 'breakout']
9. [ ('breakout', espaceinvaderss 'spaceinvaders', 'spaceinvaders', 'freeway', 'breakout', 'breakout']
10. ('freeway', 'breakout', 'freeway', 'spaceinvaders' 'freeway', 'breakout', 'freeway']
Remark: Choice of Repeated Tasks. The shuffled order of tasks is particularly useful to illumi-
nate the mechanism of our adaptive meta warm-up strategy. In real applications, tasks are not strictly one-pass; the newly arriving task is agnostic, which can be either totally unknown or re-encountered with a similar structure- aruuttre--especiallly in dynamic, cyclical, or seasonal settings. For instance, the house robot is required to clean the floor again in the following weeks after it finishes this week. Al- though a shuffled order of sequence is slightly less challenging than totally non-repeating scenarios, evaluating mixed types of new tasks is also meaningful for general continual RL scenarios.
24

---

Published as a conference paper at ICLR 2026
Hyperparameters: Atari Games. Our implementation adapts from the released code in (Malagon et al., 2024). For our FAME approach based on PPO, we choose the estimation step N 20000, Le., the number of data be stored in the meta buffer M with size 200000 in each task. the policy evaluation step n = 1200 for one-vs-all hypothesis test, and the warm-up step = 50000 (5% of the training steps in each task). The ratio of stored data over the whole training data in each task is 20000 1MM = 2%. In the knowledge integration phase, we train the meta learner across 200 epochs with 2.5 10 learning rate. The learning rate for the fast learner is also 2.5 x 10-4 We adopt À = 1.0 for the behavior cloning regularization in the meta warm-up. We choose = 1 in
Proposition T across all tasks.
Sequences of Tasks: Atari Games. We follow the setting and adapt the implementation from (Malagon et al., 2024). (1) For the MLE/Spucenvvaderr-555 environment, we have 10 tasks and each task refers one playing mode of the game. All tasks share the same objective: shoot space invaders before they reach the Earth. Observations consist of 210 x 160 RGB images of the frames, and actions are: do nothing, fire, move right, move left, a combination of move right and fire, and combination of move left and fire. The detailed descriptions of the modes are as follows:
Mode O: It is the default setting. The player has three lives, and destroying space invaders is rewarded (hitting the invaders in the back rows gives more reward).
Mode 1: Shields move back and forth on the screen, instead of staying in fixed position. Using
them as protection becomes unreliable.
Mode 2: The laser bombs dropped by the invaders zigzag as they come down the screen, making it more difficult predict the place they are going to land.
Mode 3: This task combines modes and 2.
Mode 4: Same as mode 0 but laser bombs fall considerably faster.
Mode 5: Same as mode but laser bombs fall considerably faster.
Mode 6: Same as mode but laser bombs fall considerably faster.
Mode 7: Same as mode but laser bombs fall considerably faster.
Mode 8: Same mode 0 but invaders become invisible for few frames.
Mode 9: Same as mode but invaders become invisible for few frames
For the ALE/Freeway-v5 environment, the objective is guide chicken cross road with busy traffic. The detailed descriptions of the 7 modes are as follows:
Mode 0: is the default setting.
Mode 1: Traffic is heavier and the speed of the vehicles increases, the upper lane closest to the
center has trucks. Trucks are longer vehicles, and thus, more difficult to avoid.
Mode 2: Trucks move faster than the fastest vehicles of the previous modes, and traffic heavier.
Mode 3: There are trucks all lanes, and trucks move as fast as in mode 2 in some of the lanes.
Mode 4: Similar traffic to previous modes, there are trucks, but the velocity of the vehicles is
randomly increased or decreased.
Mode 5: Same mode with the speed of the vehicles randomly changing and some vehicles
come in groups of two three very close to each other.
Mode 6: Same as the previous mode but with heavier traflic.
F.2 ABLATION STUDY IN MINATAR
Regularization Hyperparameter in Behavior Cloning. Table suggests that an overly large or small À results in inferior performance in FT and Forgetting, although the average performance is still favorable. For example, the metric scores in FT and Forgetting are worst for FAME (A 0.1). In practice, we could choose À 1.0 achieve the highest score in FT, or X 5.0 for the best forgetting score.
Warm-Up Step L under the BC Regularization. Table showw the comprehensive metric scores of FAME in terms of the number of warm-up steps with the BC regularization. is interesting to highlight that increasing the number of warm-up steps boosts the FT and average performance
25

---

Published as a conference paper at ICLR 2026
Table 4: Ablation Study of Regularization Hyperparameter on MinAtar on Average Performance (Avg.
Perf), Forward Transfer (FT), and Forgetting. Results are averaged over 10 sequences, each with 3 seeds.
Ave. Perf Spaceinvader
Method
FT 1
Forgetting +
Breakout
Freeway
FAME (A=0.1)
12.79 + 0.42
19.52 = 0.50
1.71 0.17
0.13 = 0.03
0.77 = 0.08
FAME (A=1.0)
14.54 0.58
18.72 = 0.52
1.69
0.17
0.16 + 0.03
0.72 + 0.13
FAME (A=5.0)
13.90 + 0.55
19.38 = 0.62
1.62
+ 0.16
0.14 + 0.03
0.64 : 0.07
FAME (A=10.0)
13.88 + 0.55
19.52 = 0.57
1.63 + 0.16
0.14 + 0.03
0.67 + 0.08
certain games, such Spaceinvade and Freeway. However, it worsens the general forgetting. To maintain the forgetting capability, it is recommended to keep the warm-up until a specific phase of the fast learning, such as 5 10a 20 10t training steps.
Table 5: Ablation Study of Warm-Up Step on MinAtar Average Performance (Avg. Perf), Forward Transfer (IT), and Forgetting, Results are averaged over 10 sequences, each with 3 seeds.
Ave. Perf Spaceinvader
Method
FT 1
Forgetting
Breakout
Freeway
FAME (L x 10*)
13.34 0.52
19.17 + 0.68
1.64 0.16
0.13 + 0.03
0.74 + 0.08
FAME (L = x 10*)
14.54 0.58
18.72 + 0.52
1.69 0.17
0.16 + 0.03
0.72 + 0.13
FAME (L- 20 109)
13.28 0.50
19.87 1 0.66
1.65 1: 0.16
0.17 1 0.03
0.72 + 0,08
FAME (L 50 X 10")
11.83 0.71
20.00 1 0.96
1.87 0.20
0.17 1 0.03
0.78 1 0.09
Weight Estimation Step N. For fixed size the meta learner buffer M, we vary the weight estimation step N, which determines the collected data in new environment used for knowledge integration the meta learner. saoweess that decreasing the weight estimation step consis- tently worsens the performance across all metric scores. It is worthwhilc to increase N in the future to further enhance our performance, but this would require a larger buffer size of M than the one employed in other baselines. We leave the investigation of the performance of FAME with larger buffer size of M as future work.
Table 6: Ablation Study of Weight Estimation Step N MinAtar Average Performance (Avg. Perf), Forward Transfer (F7), and Forgetting, Results are averaged over 10 sequences, each with seeds.
Ave. Perf t Spaceinvader
Method
FT
Forgetting
Breakout
Freeway
FAME (N 4000)
11.95 = 0.47
14.69 + 0.55
1.54 = 0.17
0.14 + 0.03
0.93+ 0.08
FAME (000)
12.49 1 0.37
17.99 0.58
1.67 - 0.17
0.15 0.03
0.86 0.08
FAME 22=12000)
14.54 + 0.58
18.72 0.52
1.69 + 0.17
0.16 0.03
0.72 0.13
Policy Evaluation Step n for Adaptive Meta Warm-Up before the BC Regularization with L Steps. In Table 7 a proper range of the policy evaluation step does not affect the metric scores significantly. As expected, a small policy evaluation step may not sufficiently select the best warm- up strategy, thus decreasing FT. We found = 600 is sufficient to maintain favorable FT, while managing the forgetting as well.
Table 7: Ablation Study of Policy Evaluation Step MinAtar Average Performance (Arg. Perf), Forward Transfer (FT), and Forgetting. Results are averaged over 10 sequences, each with seeds.
Ave. Perf t Spaceinvader
Method
FT
Forgetting $
Breakout
Freeway
FAME (n=300)
13.02 + 0.49
19.19 +0.65
1,42 + 0.11
0.12 + 0.03
0.69 + 0.07
FAME (n=600)
14.54 0.58
18.72 0.52
1.69 + 0.17
0.16 + 0.03
0.72 + 0.13
FAME (x=1200)
13.46 0.63
19.39 0.42
1.83
- 0.20
0.16 0.03
0.79 0.08
FAME (m=5000)
13.09 0.41
19.32 0.52
1.84 + 0.18
0.14 + 0.03
0.76 0.07
26

---

Published as a conference paper at ICLR 2026
F.3 LEARNING CURVES ANALYSIS FOR PLASTICITY ABILITY
Learning Curves of the Fast Learner: MinAtar. We provide the learning curves of all considered continual RL algorithms in the MinAtar environment across 10 sequences of tasks in Figure + demonstrating the favorable adaptation capability of the fast learner guided by the adaptive meta warm-up in each new environment.
MinAlar
Dol
anpeliate
080
000
273
MirAllrr (Bequence 21
MinAlar (Sequance 31
DO0
PRODMES
Fecioe
DANd
Steps
Steps
MinAtar (Sequence
MixAlar (Sequence
Eppoccle ieetmm
010
0.
2.0
35
000
0.5
10
2.0
33
33
Mi-Alar (Sequerica 6)
MilAAler (Sequerice
204 Protire
55%
Epioodt Rimur
S3pg
Slupe
MinAlar (Sequercc B)
MinAler (Sequercc
50%
10
Figure 4: Learning curves of the fast learner in FAME MinAtar Eavironments sequences of tasks.
Learning Curves of the Fast Learner: Atari Games. We provide the learning curves of all con- sidered continual RL algorithms the two Atari environments in Figures(5Jandã) They demonstrate the favorable adaptation capability of the fast learner guided by the adaptive meta warm-up in each new environment.
27

---

Published as a conference paper at ICLR 2026
1.0
0.5
Baseinn
0.5
0.0
1.0
FT
0.5
Progressvveee
0.0
1.0
PackNet
0.5
0.5
0.0
Task
Baseline
ProgressiveNett
PackNett
FAME
Figure 5: Learning curves of the fast learner in FAME on the Spacelvader environment averaged over 3 seeds.
1.0
0.5
Baselina
0.0
1.0
FT
0.5
PackNet Prograssivoeet
0.0
1.0
0.5
0.0
0.5
0.0
Taik ID
Baseline
ProgressiveNet
PackMett
FAME
Figure 6: Learning of the fast learner in FAME the Freeway environment averaged over 3 seeds.
28

---

Published as a conference paper at ICLR 2026
G EXPERIMENTS: POLICY-BASED CONTINUAL RL WITH CONTINUOUS ACTION SPACE
G.1 DETAILS OF EXPERIMENTAL SETUP AND COMPARISON METHODS
Hyperparameter Details. Our implementation adapts from the released code in (Haarnoja al., 2018b Chung et al. 2024] Zhang et al. 2024a b Malagon et al. 2024). We employ a replay buffer size of 1M for the fast learner and U.1M for the meta learner. The buffer size for the meta learner should not be overly large as we expect the algorithm to develop the continual learning capability without replaying too much data in the past. When the new environment arrives, the fast learner starts the training guided by the adaptive meta warm-up (i.e., knowledge transfer). At the same time, the replay buffer for the fast learner is reset, which aims only contain transitions from the current task. In contrast, the meta learner maintains a buffer that stores the most recent I % of transitions (50 episodes) leveraged to update the fast learner for each task. After finishing task, the meta learner is trained using 5000 x k mini-batches, where k is the number of tasks so far.
Task Sequences. We randomly choose sequences of tasks from (Chung et al. 2024
"button-press 'plate blcde-back--eeeeeeeee
window-close--22' slaat--slid-sidde
'peg-unplug ueeg-uuplg--sde-ee side-v2', "plate-slice-bacckvv",, 'coffee-button-v22, 'window-open-v2', 'door-close-v2"),
"aandee-pu1-ssde-222
"platte-slidebbacckssdd--=-12 'soccer-v'',
'sweep-into-v2',
handde--uull-side--V0
latpa//2/.pppplaalaaaa_ee "plate-sllid-stdde==??
peg--unglug-side--22" 'door-lock-v2', 'reach-v2',
"plate-slldde-back-222 "coffae-butttoow22),,
''coffee-push-v2'' buttonnpress-v22' 'reach-v2',
upeg-unplug-side v2', 'reach-wall-v2" 'door-closex22', "window open- "soccer-v2').
handll-pull-sider
spldte-slide-bart side-
Table 8: Results Meta-Wort World on Average Performance (Ave. Perf), Forward Transfer (FT), and Forgetting for each sequence of tasks. Results are presented as averages and standard crrors across 10 seeds. Each table, from top bottom, represents cach sequence of tasks, respectively. The best results are highlighted in bold, and the second best are underlined. 1 denotes positive metric (more is better). while is negative one (less better). Reset the baseline for evaluating FT.
Methods
Avg. Perf
FT f
Forgetting
Reset
0,090 + 0.029
0.000 = 0.000
0.800 + 0.040
Finetune
0.070 + 0.026
-0.294 0.039
0.480 + 0.050
Average
0.020 + 0.014
-0.584 + 0.035
0.100 + 0.030
PackNet
0.703 + 0.041
-0.111 0.028
0.000 + 0.000
FAME- WD
0.87 0.034
0.004 1 0.022
0.010 1 0.017
FAME-KL
0.86 - 0.035
0.042 1 0.019
0.050 1 0.026
Methods
Avg. Perf 1
FT f
Forgetting
Reset
0.110 + 0.031
0.000 + 0.000
0.680 + 0.047
Finetune
0.040 + 0.020
-0.252 + 0.045
0.440 + 0.052
Average
0,000 0.000
-0.496 1 0.039
0.110 1 0.031
PackNet
0.413 0.041
-0.249 0.031
0.000 = 0.000
WD
0.750 0.044
0.008 1 0.024
0.040 + 0.032
FAME-KL KL
0.680 = 0.047
0.004 = 0.029
0.120 + 0.036
Methods
Avg. Perf f
FT t
Forgetting 1
Reset
0.080 + 0.027
0.000 0.000
0.650 0.052
Finetune
0.000 + 0.000
-0.25 0.036
0.360 0.048
Average
0.020 = 0.014
-0.509 + 0.036
0.000 + 0.002
PackNet
0.358 = 0.043
-0.222 + 0.032
0.000 0.000
FAME WD
0.680 + 0.047
-0.020 1 0.028
0.020 0.032
FAME-KI. KI.
0.660 + 0.048
0.022 0.028
0.050 0.030
29

---

Published as a conference paper at ICLR 2026
Table 9: Results on Meta-Wort World averaged over the three sequences on Average Performance (Ave. Perf), Forward Transfer (FT), and Forgetting. Results are presented as averages and standard errors across 10 seeds. The best results are highlighted in bold, and the second best are underlined. f denotes a positive metric (more better), while &. negative (less is better). Reset is the baseline for evaluating FT.
Methods
Avg. Perf 1
FT 1
Forgetting +
Reset
0,093 0.017
0.000 0.000
0,710 0.030
Finctune
0.037 + 0.011
-0.265 + 0.028
0.427 0.033
Average
0.013 + 0.007
-0.530 + 0.024
0.070 + 0.022
PackNet
0.491 + 0.025
-0.194 1 0.018
0.000 + 0.000
FAME-WI
0.767 1 0.024
-0.003 1 0.014
0.023 0.015
FAME-KL KI.
0.733 0.026
0.022 0.015
0.073 0.019
G.2 MORE EXPERIMENTAL RESULTS AND DETAILS
Metric Scores. Table presents detailed results on Average Performance (Ave. Perf), Forward Transfer (FT), and Forgetting for each sequence of tasks. Table P further shows the average score over the three sequences. summary, compared to the baselines, the FAME variants achieve su- perior performance in Average Performance and Forward Transfer. For the Forgetting metric: as PackNet retains its previous policies, Forgetting score is O. By contrast, Average fails to learn meaningful knowledge and achieves poor overall performance, leaving it with nothing to "for- get". As a result, the best Forgetting score does not always align with the best Average Performance.
Evaluation of Performance Profile and Final Average Performance. The performance pro- file (Agarwal et al. 2021 Dolan & Moré 2002) provides a comprehensive view of the fast learner's overall performance across the entire task sequence. As shown in Figure 7 (left), both FAME vari- ants consistently outperform all baselines, demonstrating the meta-learner's ' ability to effectively consolidate knowledge over time and facilitate transfer learning. Moreover, Figure 7 (right) high- lights the advantage of FAME across all previously seen tasks. While baseline methods typically degrade as more tasks are introduced, FAME-KL and FAME achieve the highest average perfor- mancce-indicanl minimal catastrophic forgetting and robust retention over time. Notably, the last row Figure 7 indicates the average performance profile and average performance over the three sequences of tasks.
Learning Curves of the Fast Learner. Figure g showcases that the fast learner in our FAME methods achieves higher success rates across three sequences of tasks throughout training. The first three rows indicate the learning curves for the three sequences of tasks, while the last row represents their average learning curve. Learning curves are averaged over 10 seeds, and the shade region represents the standard error. This superiority implics that the adaptive meta warm-up enhances the adaptation of the fast learner in each new environment.
H THE USE LARGE LANGUAGE MODELS (LLMs)
In this study, large Language Models (LLMs) are only used for minor language polishing and edit- ing. They do not contribute to the theory, methodology, or content development.
30

---

Published as a conference paper at ICLR 2026
Reset
Finetune
FAME-KI
FAME-WD
Average
PackNet
1.0
1.0
Success Rate
Sisccess
0.6
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.8
1.0
2
10
1.0
1.0
0.8
Success Rale
0.6
0.6
0.2
0.0
0.2
Success Rate
Task Index
0.8
0.8
0.6
Succas Rate
0.6
0.4
0.4
0.2
0.2
0.0
0.0
0.2
0.4
0.5
1.0
2
Success
1.0
0.8
0.6
Success Rale
PSSuceess
0.4
0.2
0.0
0.2
Success Rate
Tack Index
Figure 7: (Left) Performance profile of the fast learner across tasks, where the y-axis shows the proportion of tasks that achieve success rate greater than equal to the x-axis value. (Right) Average performance over time by evaluating the average success rates in the past tasks. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.
31

---

Published as a conference paper at ICLR 2026
Reset
Finetune
FAME K1.
FAME WD
Average
PackNet
1.0
0.8
Sucess Rate Rate
0.6
0.2
0.0
Steps
1.0
Succoss Rate
0.6
0.4
0.0
Steps
x10
1.0
0.8
Suceess Rate Rato
0.6
0.2
Steps
1.0
Sueeess Rate
0.6
0.4
0.0
Steps
x10
Figure 8: Learning of the fast learner the Meta-World benchmark. The x-axis shows the total number of environment interactions, and the y-axis indicates the success rate. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.
32

---

Published as a conference paper at ICLR 2026
1 ADDITIONAL RESULTS
In the camera ready period of this study, we have also added more baselines: ProgressiveNet (Rusu ct al. 2016) and CompoNet (Malagon et al. 2024), and new sequence of task from CW10 in Meta- World environment. The empirical results also suggest the superiority of our FAME algorithms.
I.1 MORE BASELINES ON THE THREE SEQUENCES OF TASKS IN META- WORLD
Table 10: Results on Meta- World on Average Performance (Ave. Perf), Forward Transfer (FT), and Forgetting for each sequence of tasks with baselines. Results are presented averages and standard errors across 10 seeds. Each table, from top to bottom, represents each sequence of tasks, respectively. The best results are highlighted in bold, and the second best underlined. denotes positive metric (more is better), while 4 a negative one (less is better). Reset is the baseline for evaluating FT.
Methods
Avg. Perf
FT f
Forgetting
Reset
0,090 1 0.029
0.000 + 0.000
0.800 + 0.040
Finetune
0.070 + 0.026
-0.294 0.039
0.480 = 0.050
Average
0.020 + 0.014
-0.584 0.035
0.100 = 0.030
PackNet
0.703 0.041
-0.111 0.028
0.000 + 0.000
ProgressiveNet
0.760 + 0.038
-0.063 0.031
0.000 = 0.000
CompoNet
0,803 + 0.036
-0.007 0.022
0.000 + 0.000
FAME-W- WD
0.87 0.034
0.004 = 0.022
0.010 =-2 0.017
FAME-K KL
0.86 0.035
0.042 = 0.019
0.050 = 0.026
Methods
Avg. Perf 1
FT
Forgetting
Reset
0.110 0.031
0.000 + 0.000
0.680 + 0.047
Finetune
0.040 0.020
-0,252 0.045
0.440 + 0.052
Average
0,000 + 0.000
-0.496 + 0.039
0.110 + 0.031
PackNet
0,413 + 0.041
-0.249 + 0.031
0.000 + 0.000
ProgressiveNet
0.604 + 0.042
-0.133 = 0.03
0.000 = 0.000
CompoNet
0.566 + 0.041
-0.125 0.027
0.000 0.000
FAME-WD
0.750 0.044
0.008
0.024
0.040 0.032
FAME-KL
0.680 + 0.047
0.004 + 0.029
0.120 + 0.036
Methods
Avg. P
FT f
Forgetting +
Reset
0.080 + 0.027
0.000 = 0.000
0.650 + 0.052
Finetune
0,000 + 0.000
-0,25 + 0.036
0.360 + 0.048
Average
0,020 + 0.014
-0.509 + 0.036
0.000 + 0.002
PackNet
0.358 + 0.043
-0.222 0.032
0.000 = 0.000
ProgressiveNet
0.494 0.044
-0.137 0.031
0.000 1 0.000
CompoNet
0.468 + 0.045
-0.136 0.037
0.000 + 0.000
FAME-WD
0.680 + 0.047
-0.020 + 0.028
0.020 + 0.032
FAME-KL
0.660 + 0.048
0.022 + 0.028
0.050 + 0.030
Table 11: Results on Meta- World averaged over the three sequences with more baselines on Average Per- formance (Ave. Perf), Forward Transfer (FT), and Forgetting. Results are presented averages and standard errors across 10 seeds. The best results are highlighted in bold, and the second best are underlined. f denotes a positive metric (more is better), while J a negative one (less better). Reser is baseline for evaluating FT.
Methods
Avg, Perf
FT
Forgetting
Reset
0.093 0.017
0.000 = 0.000
0.710 = 0.030
Finetune
0.037 0.011
-0.265 0.028
0,427 = 0.033
Average
0.013 0.007
-0.530 0.024
0.070 + 0.022
PackNet
0.491 + 0.025
-0.194
0.018
0.000 + 0.000
ProgressiveNet
0,619 + 0.025
-0.111 - 0.016
0.000 = 0.000
CompoNet
0.612 + 0.025
-0.089 + 0.016
0.000 + 0.000
FAME WD
0.767 0.024
-0.003
0.014
0.023 1 0.015
FAME-KI.
0.733 + 0.026
0.022 + 0.015
0.073 + 0.019
33

---

Published as a conference paper at ICLR 2026
Boset
Finsuan:
FAMEXL
FAME WD
Aueraee
PaciNet
Progresiieee
CompaNei
1.0
1.0
0.8
0.8
Succuss Rate
0.6
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.5
0.8
Success Rate
Task Index
1.0
0.8
Succoss Rate
SSnceets
0.4
0.2
0.2
0.2
0.8
Success Rate
Task Index
1.0
0.8
0.8
Succust Rtle
0.6
0.4
0.0
0..0
0.2
0.4
0..6
0.8
1.01
10
Success Rate
Task Index
0.8
X]
0.6
Succes Rate Rate
0.4
0.2
0.0
0.2
0.8
Success Rate
Task Index
Figure 9: (Left) Performance profile of the fast learner across tasks with more baselines, where the y-axis shows proportion tasks that achieve success rate greater than or equal to the x-axis value. (Right) Average performance time by evaluating the average success rates in the past tasks. Each Row represents the result for one sequence of tasks. The last row shows the average performance of the three sequences.
34

---

Published as a conference paper at ICLR 2026
Reset
Finetune
RAMB:
TAME WD
PackNet
ProgressiveNet
CompuNet
1.0
0.8
Success Rate
0.6
0.2
0.0
Steps
x10
1.0
0.8
Suocess Rate
0.6
0.4
0.0
Stops
x10
0.8
Succes Rate
0.6
0.2
0.0
Steps
1.0
0.8
Stoosss Rate
0.6
0.4
0.0
Steps
x10
Figure 10: Learning curves of the fast learner the Meta-World benchmark with more baselines. The x- axis shows the total number of environment interactions, and the y-axis indicates the success rate. Each Row represents the result for sequence of tasks. The last shows the average performance of the three sequences.
35

---

Published as a conference paper at ICLR 2026
1.2 ADDITIONAL SEQUENCE OF TASK: CW10
Due to time constraints, the baselines of PackNet, ProgressiveNet and CompoNet on CW10 were with 3 random seeds, while all other baselines were run with 10 random seeds.
Results on the sequence of tasks CW10 (Wolczyk et al. 2021) on Average Performance (Ave. Perf), Forward Transfer (FT), and Forgetting. Results are presented averages and standard errors 10 seeds. The best results are highlighted in bold, and the second best are underlined. denotes a positive metric (more is better), while J is negative (less is better). Reset is the baseline for evaluating FT.
Table 12:
Methods
Avg. Perf f
FT
Forgetting
Reset
0,020 + 0.014
0,000 = 0.000
0.370 + 0.049
Finetune
0.020 + 0.014
-0.123 + 0.028
0.240 + 0.047
Average
0,010 + 0.01
-0.288 0.039
0.010 = 0.017
PackNet
0.292 0.071
-0.086 0.019
0.000 = 0.000
ProgressiveNet
0.325 + 0.074
-0.034
0.016
0.000 + 0.000
CompoNet
0,208 + 0.063
-0.129 + 0.029
0.000 + 0.000
FAME-WI WWD
0.340 + 0.048
-0,003
0.016
0.030 - 0.017
FAME-KL
0.370 + 0.049
0.015 + 0.017
0.060 + 0.024
Progesaveee
0.4
Success RacesssRtte Rato
0.6
0.3
PrSSucess
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0..6
0.8
10
Figure 11: Results on the CW10 /Wolzyyk et al.
2021 sequence tasks. (Left) Performance profile of
fast learner across tasks, where the y-axis shows the proportion of tasks that achieve success rate greater than or equal to x-axis value. (Right) Average performance over time by evaluating average success rates in the past tasks.
Reset
Pinetune
RAMB XL
FAME WD
Average
PackNet
ProgressiveNet
CompoNat
1.0
0.8
Success Rate
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Stops
x10
Figure 12: Learning of the fast learner the CW10 Wolezyk et al. x-axis shows the total number of environment interactions. and the y-axis indicates the success rate. Each Row represents the result for sequence of tasks. The last shows the average performance of the three sequences.
2021 sequence of tasks. The
36


Reviews:
[{"review_id": "PEKh6T0aTk", "reviewer": "Reviewer_gasT", "summary": "This paper proposes new foundations for continual RL by formally defining measures for catastrophic forgetting and MDP distance—addressing a principle way to understand why continual learning algorithms may or may not work. Building on these foundations, the authors develop Principle Fast and Meta Knowledge RL (FAME), a dual-learner framework that decomposes continual RL into two complementary processes: (1) knowledge transfer via adaptive meta warm-up with hypothesis testing, and (2) knowledge integration via incremental updates that explicitly minimize catastrophic forgetting. Extensive experiments across MinAtar, Atari, and Meta-World validate the approach.", "questions": "- Regarding the weakness, do you think there is a strategy to discard or store trajectories dynamically from the meta buffer to achieve the same continual learning objective?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper provides the first formal definitions of MDP distance and network level catastrophic forgetting in continual RL, which is directly optimizable.\n- The method is sound: It is an adaptive approach considering different knowledge transfer scenarios. It is also flexible, tested on both value-based and policy based algorithms grounded theoretically.\n- Experiments are thorough, with diverse domains covering both discrete action spaces (Atari) and continuous action space control (Meta-World). The results shows improvements upon baseline methods.", "weaknesses": "- If I understand it correctly, the method requires storing a small sample of trajectories from every tasks, leading to potential scalability issue. The sampling approach is also very simple, with only 1-2% of the training steps, which might cause poor estimation of $\\mu$ when the environment gets complex.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ur7zFEqMjj", "reviewer": "Reviewer_mRHC", "summary": "This paper introduces FAME, a biologically inspired framework for Continual Reinforcement Learning that mirrors the complementary roles of the hippocampus and neocortex in human cognition. The framework decomposes continual RL into two coupled processes, where a fast learner that performs rapid adaptation and forward knowledge transfer, and a meta learner that incrementally consolidates knowledge to mitigate catastrophic forgetting. The authors formally define two foundational concepts for continual RL: (1) MDP Distance, which quantifies environment similarity, and (2) a Catastrophic Forgetting metric applicable to both value- and policy-based RL.\nOn this foundation, FAME employs adaptive meta warm-up and incremental meta updates based on KL and Wasserstein distances to achieve a balance between plasticity and stability. Extensive experiments on pixel-based and continuous control tasks demonstrate consistent improvements in average performance, forward transfer, and resistance to forgetting compared to baselines such as DQN, PPO, PackNet, and ProgressiveNet.", "questions": "1. The definition of MDP Distance assumes shared state and action spaces between tasks. How would this metric behave when the state-action distributions of two environments are only partially overlapping or completely distinct?\n2. Is there any explicit regularization to prevent the fast learner from overfitting to transient experiences before meta consolidation? If so, how is the trade-off between immediate adaptability and long-term stability controlled?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a principled dual-learner formulation inspired by human memory systems, offering a new perspective on continual RL.\n2. The formalization of MDP Distance and Catastrophic Forgetting metrics fills a long-standing theoretical gap, providing measurable quantities for transfer and forgetting analysis.\n3. The adaptive meta warm-up and incremental knowledge integration elegantly combine statistical hypothesis testing with RL optimization, resulting in a flexible yet stable continual learning pipeline.", "weaknesses": "1. The meta learner’s learned representations are opaque and lack interpretability, and the paper does not analyze the memory or computational scaling of maintaining meta buffers as the number of tasks increases.\n2. While theoretically elegant, the algorithm involves multiple nested components, like dual learners, adaptive warm-up, statistical tests, and two types of meta-updates, which may hinder practical adoption or extension without significant engineering effort.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DMz5ce4r0t", "reviewer": "Reviewer_GXyV", "summary": "The paper proposes **FAME**, a principled framework for continual reinforcement learning (CRL) that couples a fast learner (responsible for rapid adaptation and knowledge transfer) with a meta learner (responsible for long-term knowledge consolidation and minimizing catastrophic forgetting). The method is theoretically motivated by defining two new foundations for CRL:\n1. MDP Distance, a formal measure of environment similarity.\n2. Catastrophic Forgetting, quantified across sequential tasks.\n\n**FAME** integrates these foundations through incremental updates and an adaptive meta warm-up strategy based on hypothesis testing, choosing between finetuning, reset, or meta initialization. Experiments on MinAtar, Atari, and Meta-World benchmarks show strong results across both discrete (DQN-based) and continuous (SAC-based) RL settings.", "questions": "Q: Can you elaborate on the connection between the incremental update rule in Eq. (5) and existing multi-task optimization techniques (e.g., EWC, Distillation, or Distral)?\nQ: How does FAME behave when task similarity (MDP distance) is low? Is there a threshold beyond which adaptive warm-up consistently defaults to reset?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper provides a rigorous definition of MDP similarity and forgetting, which are often treated heuristically in continual RL.\n* The dual-learner analogy (hippocampus–cortex interaction) is conceptually elegant and clearly motivates the architectural decomposition.\n* The formulation of knowledge integration as minimizing policy-based catastrophic forgetting is mathematically clean and bridges multi-task and continual RL.\n* Demonstrates consistent improvement across diverse environments (discrete and continuous), outperforming strong baselines like PackNet, ProgressiveNet, and PT-DQN.", "weaknesses": "* The exposition is mathematically dense, and while the theoretical sections are solid, the algorithmic intuition could be clearer (e.g., Eq. 5).\n* Although some hyperparameter studies are referenced in Appendix they are not discussed anough in the main paper.\n* It’s unclear how the meta buffer scales with very long task sequences or partially observable settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pm0bPAbJ6b", "reviewer": "Reviewer_6F7r", "summary": "The paper introduces FAME (Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning), a dual-learner framework designed to address the challenges of continual Reinforcement Learning (RL). The FAME framework decomposes continual RL into two distinct yet complementary objectives handled by the dual-learner system. Fast Learner and Meta Learner.\n\n**Fast Learner (Knowledge Transfer):** This component focuses on rapidly acquiring knowledge from a new task. It leverages an adaptive meta warm-up mechanism that selectively transfers prior knowledge from the meta learner to facilitate rapid adaptation and circumvent the issue of negative transfer. This transfer is achieved either by directly copying parameters or by adding a Behavior Cloning (BC) regularization during the early training phase.\n\n**Meta Learner (Knowledge Integration):** This component ensures knowledge integration and safeguards against catastrophic forgetting. It incrementally integrates new experiences by minimizing catastrophic forgetting, as formally defined in the paper. This consolidation process enhances the adaptive meta warm-up for subsequent environments.\n\nBesides, the authors provide definitions about MDP Distance and Catastrophic Forgetting and provide theoretical results used in FAME.", "questions": "I have two primary inquiries concerning the sensitivity and selection process for the critical hyperparameters $L$ (Warm-Up Step) and $N$ (Weight Estimation Step), whose impact is detailed in the ablation studies.\n\n1.  **On the Non-Monotonic Effect of the Warm-Up Step ($L$):**\n    The analysis in Table 5 reveals a complex trade-off associated with the warm-up duration ($L$). While a longer $L$ might intuitively suggest improved learning from the meta-learner, the empirical results show a non-monotonic trend. Specifically, extending $L$ to $50 \\times 10^4$ yields marginal gains in Forward Transfer (FT) (0.16 to 0.17) but significantly worsens Forgetting (0.72 to 0.78) and degrades the 'Average Performance' on Breakout. This result is counter-intuitive. Why does a prolonged application of Behavior Cloning (BCI appreciate if you let me know this phenomena.\n\n2.  **On a Principled Strategy for Determining $L$ and $N$:**\n    The ablation studies  confirm that performance is highly sensitive to the choice of $L$ and $N$. These parameters appear inherently task-dependent. $N$ dictates the fidelity of the approximated state-visitation distribution ($w_k$ or $\\mu_k$), while $L$ manages the plasticity-stability trade-off during adaptation. However, the paper does not seem to offer a principled method for their selection beyond empirical tuning within a specific domain. Could the authors share heuristics or a more adaptive strategy for setting these parameters? For instance, in the Meta-World experiments, the warm-up evaluation was set to 10 episodes. What rationale guided this specific choice, and how might such a heuristic generalize to environments with varying task complexities or episode lengths?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Outperforming Performances\n\n2. Clear Motivation \n\n3. Theoretical Foundations", "weaknesses": "1. Hyperparameter settings are required, e.g., $L$ and $N$.\n\n2. Inefficiency to train multiple policies\n\n3. Insufficient Abalation Study for $N$ and $L$", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)