Title: Principled Fast and Meta Knowledge Learners for Continual Reinforcement Learning
Venue: NeurIPS 2025
Accepted: no
Keywords: continual reinforcement learning

meta learning

multi-task learning
Abstract: Inspired by the human learning and memory system, particularly the interplay between the hippocampus and cerebral cortex, this study proposes a dual-learning framework comprising a fast learner and a meta learner to address continual Reinforcement Learning~(RL) problems. These two learners are coupled to perform distinct but complementary roles: the fast learner focuses on knowledge transfer, while the meta learner ensures knowledge integration. Unlike traditional multi-task RL approaches that share knowledge via average return maximization, our meta learner incrementally integrates new experiences by explicitly minimizing catastrophic forgetting, and then transfers accumulated knowledge to a single fast learner. To support rapid adaptation to new environments, we introduce an adaptive meta warm-up mechanism that selectively leverages past knowledge. We perform experiments in the pixel-based benchmark and continuous control problems, revealing the comprehensive performance of continual learning for our proposed dual learning approach relative to baseline methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FAME (Fast and Meta knowledge learners), a novel framework for continual reinforcement learning that draws inspiration from the human brain’s memory systems. It proposes a dual-learner architecture comprising a fast learner, which rapidly adapts to new tasks, and a meta learner, which incrementally integrates knowledge to prevent catastrophic forgetting. To support efficient adaptation and stability, the authors define principled metrics such as MDP distance (for task similarity) and catastrophic forgetting (for performance degradation). A key contribution is the adaptive meta warm-up mechanism, which selects the best initialization strategy for new tasks using statistical hypothesis testing, thereby mitigating negative transfer. Theoretical foundations are coupled with practical algorithms for both value-based and policy-based RL, using divergence-based incremental updates. Experiments on MinAtar and Meta-World benchmarks show that FAME achieves superior average performance, forward transfer, and forgetting resistance compared to strong baselines, highlighting its effectiveness in balancing plasticity and stability in continual learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: (1) The paper demonstrates solid theoretical grounding with clear derivations and well-justified design choices. (2) It introduces quantitative definitions for MDP distance and catastrophic forgetting, advancing the formal treatment of continual RL. (3) Mathematical formulations (e.g., CF definitions, update rules) are intuitive and well-motivated. (4) The principled decomposition into knowledge transfer (via fast learner) and integration (via meta learner) provides a modular foundation for future research and extensions.\n\nWeaknesses: (1) The performance of the Q-value-based update may degrade when Q-values have inconsistent scales across tasks (acknowledged but not deeply addressed in the main paper). (2) No ablation or sensitivity study is presented for the hypothesis test-based warm-up mechanism, which may introduce fragility under noisy returns or limited samples. (3) The exposition could benefit from clearer visual diagrams summarizing the update mechanisms or data flow between the fast and meta learners. (4) No results are shown on longer sequences of tasks (e.g., 10+ tasks), which would better test scalability and forgetting. (5) The neuroscience motivation, while engaging, does not clearly influence the architecture beyond analogy.\n\nQUESTIONS:\n1. The adaptive warm-up mechanism relies on a hypothesis test between fast, meta, and reset strategies using early task returns. However, early rollouts can be noisy or sparse, especially in high-variance environments. How sensitive is the adaptive warm-up mechanism to noise in early evaluations?\n2. The current framework assumes known task boundaries, which simplifies the design but limits applicability in non-stationary environments without such information. How well does the method generalize to task-agnostic continual RL?\n3. How does the method handle tasks with inconsistent Q-value or reward scales?\n4. The framework includes a meta buffer, separate learners, and evaluation steps. While said to be scalable, resource costs are not quantified. Can the authors clarify the memory and computational cost of FAME?\n\nLIMITATIONS:\nThe authors have included a clear discussion of the limitations in Section 5, acknowledging issues such as: (1) The assumption of known task boundaries, (2) the scalability limits of the meta learner, (3) and opportunities for extending to task-agnostic continual RL and representation-level knowledge integration. They do not claim that their approach fully solves continual RL, and they appropriately suggest directions for future improvement.\n\nRegarding societal impact, they mark it as “Not Applicable”, which is reasonable in this case. The work is theoretical and methodological in nature, without direct application to sensitive or high-risk domain. However, if deployed in real-world systems, one could imagine downstream impacts where continual learning could cause unexpected behavior due to forgetting or transfer misalignment. This could be briefly noted in future revisions for completeness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FAME, a dual-learner framework for continual RL. It has two components: (1) a fast learner that rapidly adapts to each new task initialized by the best network among random network, meta learner and the preceding fast learner (2) a meta learner that incrementally consolidates past and current policies or value functions by minimizing a principled measure of catastrophic forgetting. These two components are instantiated as value based methods in discrete action space and instantiated as policy based methods in continuoous action space. Their experiments show that they outperform baselines on MinAtar and Meta-World tasks, improving forward transfer and greatly reducing forgetting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n - This paper gives rigorous deifnition of MDP distance and comprehensive mathematic proofs for meta learner updating rools, which gives the work a solid foundation.\n - This paper takes inspiration from biology that proposes a hippocampus-like faster learner and cortex-like meta learner. Such inspiration brings a new perspective to continual learning research.\n - The authors develop theories for both value-based and policy-based algorithms for FAME, making it applicable to broader usecases.\n\nWeakness:\n - The authors perform the one-vs-all hypothesis to initialize a fast learner. Considering that there are three candidate learners including a randomly initialized one, the time and space complexity to perform the hypothesis could be a concern to apply this mechanism in real-world application. It would be nice to have an ablation study how much computational overhead it takes and how much improvements it gains compared to using one of them only\n - The baselines in this paper are too weak to evaluate. There are lots of works in continual RL [1]. I suggest the authors compare with SOTA continual RL algorithms to make the results more convincing.\n- Additionally, running experiments on three different tasks repeatedly in shuffled order is not enough to show the knowledge transfer and resistance to catastrophic forgetting of this work. Referring to [1], a typical benchmark runs 6 different Atari games to evaluate a continual RL algorithm. I suggest the authors to run more tasks to make the evaluation more solid.\n\n[1] Powers, Sam, et al. \"Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents.\" Conference on Lifelong Learning Agents. PMLR, 2022.\n\nQUESTIONS:\n1. Does the replay buffer size increase linearly with the task? An illustration would be helpful to clarify the contribution of the work. If not, how does the meta learner resist the catastrophic forgetting in far old tasks? If so, ideally, continual learning takes constant extra space (or no extra space) to save replay data. I suggest a generative replay buffer, e.g., [1], to reduce the space complexity.\n2. The authors choose KL divergence and Wasserstein Distance without an explanation. The main results in the experiments section show that FAME-WD is always worse than FAME-KL, so what is the motivation to bring WD in this work? An explanation would be greatly helpful for readers to understand the technical motivation.\n\n[1] Shin, Hanul, et al. \"Continual learning with deep generative replay.\" Advances in neural information processing systems 30 (2017).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is about proposing a new model inspired by the neuroscience, akin to the theory of complementary learning systems with the role of slow learner being replaced by a meta learner. The role of the meta learner is to perform knowledge integration via minimising catastrophic forgetting across previously encountered tasks. In order to ensure minimal negative transfer, the authors proposed an adaptive meta warm-up strategy, which includes using the fast learner, the meta policy and a random Q function. The authors evaluated their approach using Minatar and Metaworld environments, with baseline models including resetting parameters of the deep neural network, fine-tuning parameters of the deep neural network and a permanent and transient model that also attempts to learn using fast and slow timescales. The proposed model in the paper manages to achieve higher forward transfer and reduced forgetting in continual RL settings when compared with the baseline models.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n1. The paper is mostly clear and well-written. Most of the mathematical propositions were well-defined. \n2. The proposed idea in the paper seems to be novel and the results do show that the proposed approach seems to be better at mitigating forgetting.\n\n# Weakness \n1. The experiments are done with only 3 random seeds. I would recommend performing with at least 5 random seeds to reduce the variability in the experimental results.\n2. $\\tilde{Q}^{M}_{k}$ is not defined in Eq 3 and 4.\n3. The proposed approach requires the knowledge of task information and task boundary. How will the approach fare where task boundary  is unclear? While to the credits of the authors, they did indicate this as a limitation in the limitation section.\n4. In line 43-44, the authors mentioned that many existing methods have been proposed that trade off stability and plasticity, without quantifying explicit objectives to optimise. It will be nice if the authors can provide some citations and elaborate what could be some of these explicit objectives to consider.\n5. The Minatar benchmark is somewhat simplistic in terms of representation and game dynamics complexity. How would the model fare in more complex games like Atari? I recommend trying 3-5 Atari games, but with less than 50Million steps (200 Million Frames).\n\nQUESTIONS:\n1. How is the ideas proposed in definition 1 and definition 2 differs or relate to  bisimulation metrics[1]? \n\n2. What about interference? Are there signs of interference for the tasks studied in the experiments? If so, how does the proposed model cope with limiting the effects of catastrophic interference? \n\n\n[1] Ferns, Norman, and Doina Precup. \"Bisimulation Metrics are Optimal Value Functions.\" UAI. 2014.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces FAME, a CRL framework that combines a fast learner for task-specific adaptation with a meta-learner that consolidates knowledge to prevent forgetting. Forgetting is defined as a weighted divergence in behavior or value estimates based on prior task visitation. The meta-learner is updated incrementally using a compact buffer of high-quality transitions. To avoid negative transfer, the method employs an adaptive warm-up that selects between meta-initialization, fine-tuning, or reset based on early performance via a hypothesis test. When the meta-policy is used, behavior cloning regularizes early training. Experiments on MinAtar and Meta-World show improved transfer and reduced forgetting over simple baselines.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. When confronted with a new task in the sequence, the authors propose to select the best warm-up strategy among three options after running short rollouts with each. At the expense of a small overhead at the beginning of each task, FAME can better avoid the negative transfer problem. The most novel strategy is the use of the meta-policy. By estimating task similarity via either policy or Q-value divergence, the method avoids reusing prior knowledge when it's likely to be harmful. If optimal behaviors differ significantly across tasks, reusing past policies can lead to poor performance. Similarly, if the same state-action pairs yield very different Q-values, it signals a mismatch in task dynamics.\n2. The authors introduce a precise, weighted definition of forgetting based on MDP differences, framing it not merely as a drop in performance, as is common in prior work, but as a principled measure of how much the agent's behavior or value estimates diverge from what it previously knew, specifically on the states and actions that mattered most in past tasks. This grounds forgetting in the agent’s own experience, offering a more behaviorally meaningful and task-relevant signal than aggregate performance metrics alone.\n3. To update the meta-learner in the value-based setting, the authors apply a clever softmax transformation to convert Q-functions into probability distributions. This allows them to operate in policy space, where forgetting is easier to interpret, optimize (via KL divergence), and align with actual agent behavior, which ultimately matters more than raw Q-values. Aligning Q-functions directly across tasks can be more brittle. In contrast, aligning policies is cleaner and more robust.\n4. The memory footprint is low. Only ~1% of the most recent task’s training data is stored in the meta buffer, making it well-suited for continual RL where storage is constrained. Importantly, FAME doesn’t store arbitrary rollouts, but captures samples from the final training phase, when the policy is more likely to be competent. This focused distillation ensures that the meta-learner is updated using higher-quality, task-relevant behaviors.\n5. FAME is grounded in strong theoretical foundations, with key components formally derived and supported by proofs. The paper is also clearly written and well-cited, situating the work effectively within the broader literature.\n\n# Weaknesses\n\n1. The evaluation is weak. It is based on seven simplified Atari-style environments and a single 10-task Meta-World sequence, with all results averaged over just three seeds. The authors do not evaluate FAME on established continual RL benchmarks such as Continual World [1], COOM [2], or CoRA [3], despite the fact that these benchmarks align well with FAME's assumptions (shared state/action spaces, known task boundaries, and replayable transitions). This makes it difficult to assess the scalability and generality of the method. In terms of baselines, the paper compares only against simple adaptations of standard RL algorithms, while omitting well-established continual learning methods like EWC, AGEM, MAS, and PackNet. The CRL benchmarks listed above already include many baseline evaluations to directly compare against. For instance, the authors discuss R&D as a motivation, but never include it in the empirical comparisons. As a result, the evaluation does not provide a clear picture of how FAME performs relative to state-of-the-art methods targeting similar objectives.\n2. FAME’s adaptive warm-up mechanism appears to provide the most benefit when tasks are revisited, rather than when encountering entirely new environments. As shown in Figure 2 (Right), the warm-up strategy selects the meta-policy 95% of the time when the incoming task has been seen before, but only 19% for genuinely novel tasks. This suggests that much of the advantage stems from re-encountering previously learned tasks. In their MinAtar setup, the authors construct a 7-task sequence using only 4 environments, meaning that task repeats are explicitly included. Revisiting tasks is a less challenging setting than tackling a continual stream of novel tasks in non-repeating, task-incremental scenarios.\n\n[1] Wołczyk, Maciej, et al. \"Continual world: A robotic benchmark for continual reinforcement learning.\" *Advances in Neural Information Processing Systems* 34 (2021): 28496-28510.\n\n[2] Tomilin, Tristan, et al. \"Coom: A game benchmark for continual reinforcement learning.\" *Advances in Neural Information Processing Systems* 36 (2023): 67794-67832.\n\n[3] Powers, Sam, et al. \"Cora: Benchmarks, baselines, and metrics as a platform for continual reinforcement learning agents.\" *Conference on Lifelong Learning Agents*. PMLR, 2022.\n\nQUESTIONS:\n1. Why not evaluate FAME on established benchmarks for continual reinforcement learning?\n2. Why is FAME not compared against popular or SOTA CL methods?\n3. How sensitive is FAME to the choice of temperature in the softmax transformation and the size of the meta buffer?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}