Continual Reinforcement Learning by Planning with Online World Models
Authors: Zichen Liu
Guoji Fu
Chao Du
Wee Sun Lee
Min Lin
Venue: ICML 2025 spotlightposter (2025)
Keywords: model-based rl
continual rl
online learning
incremental learning
catastrophic forgetting
Primary area: reinforcement_learning
TL;DR: we learn the world model online and plan with the model to achieve continual reinforcement learning

Abstract: Continual reinforcement learning (CRL) refers to a naturalistic setting where an agent needs to endlessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learning a new task, known as catastrophic forgetting. In this paper, we propose to address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leader shallow model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven regret bound of $\mathcal{O}(\sqrt{K^2D\log(T)})$ under mild assumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incrementally. To assess OA, we further design Continual Bench, a dedicated environment for CRL, and compare with several strong baselines under the same model-planning algorithmic framework. The empirical results show that OA learns continuously to solve new tasks while not forgetting old skills, outperforming agents built on deep world models with various continual learning techniques.

Continual Reinforcement Learning by Planning with Online World Models
Zichen Liu 12 Guoji Fu 2 Chao Du
Wee Sun Lee? Min Lin
Abstract
this "multitask" view of CRL because task-specific com- ponents are constructed and used (e.g., separate weights, task boundaries or IDs) (Kirkpatrick et al., 2017; Mallya & Lazebnik, 2018; Chaudhry et al., 2019a; Kessler et al., 2022; Gaya et al., 2023).
Continual reinforcement learning (CRL) refers to naturalistic setting where an agent needs to end- lessly evolve, by trial and error, to solve multiple tasks that are presented sequentially. One of the largest obstacles to CRL is that the agent may forget how to solve previous tasks when learn- ing a new task, known as catastrophic forget- ring. In this paper, we propose address this challenge by planning with online world models. Specifically, we learn a Follow-The-Leadde shal- low model online to capture the world dynamics, in which we plan using model predictive control to solve a set of tasks specified by any reward functions. The online world model is immune to forgetting by construction with a proven re- gret bound of D under mild as- sumptions. The planner searches actions solely based on the latest online model, thus forming a FTL Online Agent (OA) that updates incremen- tally. To assess OA, we further design Contin- ual Bench, a dedicated environment for CRL, and compare with several strong baselines un- der the same model-planning algorithmic frame- work. The empirical results show that OA learns continuously to solve new tasks while not for- getting old skills, outperforming agents built on deep world models with various continual learn- ing techniques.
However, a more naturalistic scenario would involve a learning agent that seamlessly interacts and learns in an en- vironment where there is clear cut tasks. To distinguish with the above, we call such agent an Online Agent (OA). We expect the solution of OA to contain learning compo- nents that are shared through out the agent's lifetime, and can be updated incrementally.
Existing CRL methods are not building towards online agents because of two challenges. First, not all RL compo- nents have their online alternatives. For example, the value function and policy are dependent on the reward function defined by a task, thus it is innately not learning compo- nent that can be shared across tasks. To alleviate this is- sue, prior works often learn them per-task (Chaudhry et al., 2019a; Wolezyk et al., 2021). Second, for components that can be shared, they should be learned incrementally with- out forgetting under severe data distributional shift. Huang et al. (2021) has attempted this with model-based RL, but they still rely on per-task buffer and embedding.
In this paper, we aim to develop an OA to tackle the CRL problem. We first notice that a unified world dynamics is the key component that can be shared across tasks and should be learned, and planning with the learned model gives
RL agent that maximizes long-term rewards (Garcia et al., 1989; Hutter, 2000). Since the planner usu- ally has no trainable parameters (De Boer et al., 2005; Williams et al., 2015), the goal of OA is to learn an online world model with low regret incrementally. To this end, we employ the recent Hello--The-Leader (FTL) shallow mod- els (Liu et al., 2024) which permit efficient online updates for world modelling, and plan with the learned models ing cross-entropy method (CEM) for acting. Under mild assumptions, we theoretically show the sparsely updating models (Liu et al., 2024) are no-regret, providing defini- tive answer for the design of DA. Consequently, OA's ev- ery interaction with the real world brings new information to lear a more accurate world model without forgetting, which immediately benefits the planning at the next step.
Introduction
Continual reinforcement learning (CRL) (Khetarpal et al., 2022; Abel et al., 2024) quests for agents that can continu- ously evolve to solve possibly infinite number of tasks that are revealed sequentially to them. From a task-level defini- tion, a very strong baseline to just train a separate agent for each task, and use the information of the task ID to se- lect the agent. Many existing works more less fall within
'quual contribution 'Sea Al Lab National University of Sin- gapore. Correspondence to: Min Lin <linmin@ sca.com>.
Proceedings of the 1nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Continual Reinforcement Learning by Planning with Online World Models
To evaluate OA in the CRL settings, we further develop Continual Bench, which explicitly focuses the design of a unified world dynamics. It also takes account of for- getting and transfer simultaneously and is computationally lightweight (see motivations in Section 2 part 2). The em- pirical results demonstrate the superior performance of OA over strong CRL baselines under the same model-planning framework, suggesting its great promise in developing fu- ture autonomous artificially agents.
Task
Task
Task
pr = 2,+1
2x+1 24
0.1
3/-1 = 24
pu
F++1 [1,0.1,-1]'
Figure 1: A motivating example comparing task-indexed dynam- ics (P") with unified dynamics (P"). Under the task-indexed viewpoint, the state space of each task is the box's location r E R while the action space is ( ("apply force rightwards", "apply force leftwards"). J. Clearly, for the same action "apply force rightwards" (denoted as red arrows) applied at the same state xr. the dynamics pr is conflicting to each other. Hence, there is no solution for a CRL agent to model the dynamics, unless incorporating task IDs. In contrast, the unified dynamics P" treats each task individually and builds "global" view of the world, where the solutions to all tasks can co-exist without the need of task IDs.
2 Related Work
In this section, we provide an overview of how our work integrates with existing literature. Since this work tries to contribute in both algorithm and benchmark aspects, the discussion is structured into the following two subsections.
Algorithm perspective. The most related work to ours is Liu et al. (2024), where the authors propose a non-linear encoding technique and an analytic update rule to learn world models online. We follow their work to employ sim- ilar online shallow networks to learn the world dynamics, To bridge the gap between practice and theory, further analyze the convergence of the online sparse model learn- ing, and identify that regularization is necessary to guaran- tee it converges to the offline optimal solution. Our work is also methodology-wiin -iise distinct from Liu et al. (2024), which is based on the Dyna architecture (Sutton, 1990) and still learns model-free components such as value and pol- icy functions using mydel-synthetic data. This could fur- ther compound difficulties of applying model-free methods to develop continual agents, such as the need to maintain a replay buffer (Isele& & Cosgun, 2018), the loss of plastic- ity (Abbas et al., 2023), and even the dependency on task- specific models (Garcia & Thomas, 2019; Wolczyk et al., 2021). In the supervised CL settings, (Zhuang et al., 2022; 2023; 2024) also employ an analytic solution solve class incremental learning without forgetting (an FTL approachh, but they rely on pretrained deep features over all tasks, which is impossible for online agents. On the contrary, we propose to use a model-planning framework to tackle the CRL problem, given a unified world dynamics representa- tion and external reward function, making our agent (OA) potential to generalize new tasks zero-shot (Sancaktar al., 2022).
from the need for a consistent shared observation space, which they choose to be the pixel space, The games in- cluded are also visually-distinet with little to no overlap- ping, making the benchmark suitable for studying forget- ting. However, the image-based environments demand pro- hibitive computation resources, and the lack of meaning- ful overlapping prevents us from evaluating the transfer of CRL algorithms. Continual- World (Wolezyk et al., 2021), on the other hand, focuses on continuous control and trans- fer, using sequence of 10 robot manipulation tasks se- lected from Meta-World (Yu et al., 2019). Nevertheless, their state-based inputs encode the positions of different objects at fixed locations, which results in inconsistent slute space (further explained in Section 5), hindering the origi- nal purpose of studying transfer. In this work, we propose a lightweight but realistic benchmark environment, Contin- ual Bench, that has a consistent state space and takes both forgetting and transfer into consideration.
3 Preliminaries
3.1 Continual reinforcement learning
We first review some necessary RL basics and then define the setting of continual RL. RL is usually formulated as a Markov decision process (MDP) (Bellman, 1957) M = AS.A,P, 7/s Po), where 8, E S and a; E A are state and action at time step t, P : S x A + S is the transition dynamics, R : S x A + R is reward function associated with a specific task, E (0, 1) is a discount factor, and Po is the initial state distribution. An RL agent aims to identify an optimal policy ा S + A that maximizes long-term reward (or return): x² arg max, E,,P [E:= 7 R (st, a₁, at, 8141) So A continual RL agent (Abel et al., 2024), on the other hand, needs to conduct indefinite search for policies to control a sequence of MDPs M, = A, P", R 7. (.P)), with a consistent and shared state and action space. We also consider a unified dynamies pa that describes the whole
Benchmark perspective. CRL has typically been eval- uated on a sequence of Atari games (Kirkpatrick et al., 2017; Schwarz et al., 2018; Rolnick et al., 2019). CORA (Powers et al., 2022) recently extends this conventional protocol a set of image-based discrete control environ- ments, additionally including procedurally-generaae envi- ronments (Cobbe et al., 2020; Küttler et al., 2020) and real- istic home simulators (Kolve et al., 2017; Shridhar et al., 2020). Their focus on based environments stems
pu].
2

---

Continual Reinforcement Learning by Planning with Online World Models
world an agent can see, instead of time-indexed pieces Pr (Wolczyk et al., 2021; Khetarpal al., 2022) that can only be distinguished by explicit task IDs, or otherwise becoming POMDPs whose task ID information needs to be inferred (Nagabandi et al., 2018). We depict a motivating example in Figure to give intuitions on the differences between pr and P While our definition offers a more simplified formulation, it does not compromise nonstation- arity, the core problem in CRL, because the nonstationary reward function Rt will drive the CRL agent to perform different tasks, incurring distributional shift in state-action visitation. This gives the objective of a continual agent
cess, followed by our results on the regret of the sparse model learning (Section 4.2). Based on the learned no- regret model, Section 4.3 describes how the policy is con- structed using planning.
4.1 Online world model learning
We present the learning process of OA, where we employ shallow but wide networks that support efficient Follow- The-Leader (FTL.) online learning model the world. In particular, we learn a model of the form y (o(Px), where P is a projection matrix filled with fixed random values drawn from a Gaussian distribution, a is an acti- vation function and W contains learnable weights. Such architecture allows universal approximation (Huang et al., 2006) but requires a very high dimensional hidden layer to have great capacity. Following Liu al. (2024), we learn a linear model online per time step, using high-dimensional sparse feature encoder for o (Px) and solving for W in closed form with FTL strategy.
JT(#)= = Σ BT, P 1 [s: ai, 8141)(80 -8 Po
(1)
Note that T in Eq. (1) is the number of tasks the agent has seen till now, instead of the total number of tasks, which is
an unknown for a continual agent. In other words, a contin- ual agent is expected to gain high return for all experienced tasks. In this paper, we develop such an agent by learning world model online, and form its policy by planning with the learned model.
Modeling. Let S and denote the dimenionality of state and action spaces. When the agent is in state 8, it exe- cutes an action a and observes how the world changes. The observation may incur a loss thus correcting its previ- ous perception of the world dynamics, which is modeled linearly by y: = o(xx)) T W. W E RDxS is the linear layer, xt = [s,, a.l E RS+A is the input for the world model and ((xx) E RP is its sparse high-dimensional projection, and y = (st+1 - 8.) E R the state dif- ferences. At the start of time step t, we have Φ;-1 (o(z)))..,,$<..11) ([)+
3.2 Planning and model predictive control
Planning. Given the world dynamics P and reward func- tion R, planning involves optimizing action sequences Oct++ HI to maximize the H-step finite-horizon return given current state 8t:
a+:+H= = arg max
Σ
R R(#44) a14s) 18441 ~ P(s,, at).
R(11))xD accumulating all the inputs and Y,-1 E ((-11 as well for targets. A FTL world model can be solved by regularized least squares:
(2)
As model-based RL method, planning has the advantage that it can reuse the same world dynamics and optimize for different reward functions without adaptation. This is es- pecially of our interest in developing a continual agent, be- cause when facing a new task the agent can directly utilize prior knowledge about the world attempt to solve it.
Vt, (1 arg min
P W Y. 11 + W W
WERDXS
where || || denotes the Frobenius norm.
Learning the world model simply requires finding so- lution to Eq. (3), which can be obtained analytically as Though this
Model predictive control (MPC) synthesizes closed- loop policy from the planning results. It takes the first ac- tion ar from the optimal action sequence, executes it in the environment and re-plans based on the new environment state. An MPC-based agent collects experiences following such policy, while improving its internal estimation about the world through experiences (Negenborn al., 2005) for better planning. In this paper, we focus on the unified world dynamics learning driven by an external time-varying re- ward function, which aligns with our CRL formulation.
( = ( ( .-1 + computation does not grow with the amount of data, the in- version could still be inefficient since D is usually large for modeling complex environments. We thus utilize the fea- ture sparsity and conduct incremental update the model:
wi( = (A!
(t + 11 -1 (B A (t- 68
where =T D. and ((--1) = T Y 1 contains all the indices on which the latest input feature ((æ;-1) has non-zero values, i.e., the newly coming data point only activates K = |s| < D nodes, and 3 is its complement. Using them in subscript means taking a sub- matrix from the original one. The activation ratio K/D
Online Agent for Continual Reinforcement Learning
In this section we present the proposed Online Agent (OA).
Section 4.1 introduces the online world model learning pro-
3

---

Continual Reinforcement Learning by Planning with Online World Models
fixed and small for any input to guarantee a constant and low update overhead. Note that when covers all coordi- nates (thus the feature is dense), Eq. (4) recovers the ana- lytic solution to Eq. (3). We refer to Appendix A.1 for a more detailed sparse feature construction process.
in spectral norm, or the feature value at a is larger than previously observed data, smaller K suffices ensure model performance. In practice, we choose K such that As- sumption 3 hold.
Theorem 1 (Regret Bounds for Sparse Online Model
Despite the resemblance to the update rule in Algorithm 2 of Liu et al. (2024). Eq. (4) highlights a crucial differ- ence that _ derived from the ridge regularization term is important for the local update to have unique minimizer (Lemma 1), or otherwise it may not converge (Peng & Vi- dal, 2023) when A!--1) is rank-deficient, which is the case during the initial phase of the learning when data points are only few. We provide theoretical analysis in Section 4.2.
Learning). Let W(1), W ,.... be the sequence produced by the sparse online model learning Eq. (4). Let Let 2 1 be a constant defined as in Eq. (18). Suppose that there exist CW, ,W,Cg Cy > 0 and X 2 1 such that Assumptions I to 3 hold. Then, for all T 2 1 and all 5 E
RR Dx5, we have
T
Regret (T) = Σ 51 f((W()) Σ f((
4.2 Regret analysis
$ 7 cw +5X(K2
TM + (logg())))T) + 1). (6)
In this section we show the sparse update rule (Eq. (4)) learns a no-regret (O()g(T)) world model. With the fol- lowing mild assumptions for hyperparameters and inputs,
If = /c 55 + ()Do +1) 2 1 and it
satisfies Assumptions 1 and 3, then we have
have Theorem 1.
Regret(T) S Sowey 20(K2r2 /
+ 1)D(log(TT) + 1). (7)
Assumption 1. For all 1 and 3 , 22,..., Ct. we as- sume that there exists X 2 1 such that
Proof Sketch (for formal proof see Appendix E.3.2). Note that we can decompose the regret into:
sup (()))()))
Σ ((x))((;))
|1, At"
(5)
T Regret (T) = Σ (((Mo))))(())))))
Assumption 2. Assume for all t 2 1, 1 S Cy with Vt.: 2 0. and [ $ cw.
T
T
Assumption 3. For all t 2 1, we set K such that
K 2 min DD.
The first term involves the gap between the solutions of Eq. (3) and the sparse online model learning Eq. (4) at each time step. By Schur's Complement Lemma for block ma- trix inversion (Zhang, 2006) together with the sparse up- date rule Eq. (4), we can upper-bound the difference be- tween ( and W(()) (see Proposition 3), which is fur- ther used to bound (W) f:(((()) (see the proof in Appendix E.3.2). The second term is the regret bound for online model learning Eq. (3). By bounding the difference between w and W (see Lemma 2), we can upper- bound f(W(1) - f(W(+++)) (see Lemma 3). Then, fol- lowing a similar proof (Shalev-Shwarz, et al., 2012) for the regret bounds of FTL models, we obtain the regret bound for online model learning (see Corollary 1). Combining the bounds for the two terms we conclude the proof. We refer to Appendices E.2 and E.3 for more details and other results.
& [ا_ ((a.)ooa)) ((x.) ((x1)
(A (A 11:2 }1} (9) (2)
Remark 1 (Discussions on the assumptions). Assump- tion ensures that as more data points are observed, the feature mappings stabilize. The difference between the outer product of any new input's 's feature mapping and the average outer product of the observed mappings decreases as increases. In essence, as more data is gathered, the em- pirical covariance matrix better approximates the true co- variance matrix. The term X controls this stabilization rate, with X as a scaling factor. It ensures the model's representations become more reliable over time, improv- ing generalization to new data. In practice, Assumption holds we explore new points near to the observed data, i.e., within 1 away from the center in the feature space. Assumption assumes bounded input and output, which is mild condition. Assumption 3 states that the choice of the hyperparameter K depends on two factors: (1) the weights of the features at time over all observed data and (2) the sparsity of the activated elements in the feature ma- trix A (t)) Intuitively, If Ass is more significant than Ags
4.3 Planning with online world models
Based on its internal no-regret model about the world, OA acts in the environment by planning and MPC. For plan- ning we use the cross-entropy method (CEM) (Rubinstein,
4

---

Continual Reinforcement Learning by Planning with Online World Models
1999; De Boer et al., 2005), stochastic drrivative-free optimization technique that has demonstrated effectiveness in different model-based RL scenarios (Chua et al., 2018; Wang & Ba, 2020). It solves Eq. (2) in an iterative manner as described below.
In each iteration, it first generates N action sequence candi- dates C = {Al:++ J for a planning horizon H, in which each a; is independently sampled from N ((,, diag( (²)) with o. E R... Then, individual candidates are evaluated by simulating the action sequences in the learned model to compute the total rewards. Finally, only a fixed number of elite candidates with high total rewards is selected and used to estimate the parameters {H,, o by maximum likelihood for the next iteration. After a few iterations, an approxi- mately optimal action sequence can be found and used by MPC.
Potential physical conflicts between two consecutive tasks in Continual World (Wolczyk et al., 2021). The handles of the door and the drawer follow different trajectory when being opened.
Figure 2:
named
where they select a sequence of tasks from Meta-World (Yu et al., 2019) and run experi- ments using soft actor-critic (SAC) (Haarnoja et al., 2018) with various continual learning techniques. This bench- mark has put explicit focus on low-level transfer, because the same robotic arm is instructed to achieve different goals by interacting with real-world objects. However, the poten- tial underlying physical conftict impedes the original de- sign purpose.
Instead of the plain CEM, we adopt several improvements to make it efficient for model-based RL. The first modifica- tion is iofff-initialization.. At time step (t+ 1), we initialize the candidates C++1 using the solution of the previous de- cision step C; shifted by one step. This is because MPC only takes the first best action and discards the remaining (H - 1) results, which could be utilized to provide bet- ter initialization for the next time step. We also incorpo- rate colored noise and memory proposed by Pinneri et al. (2021) for better sample efficiency. The colored noise in- jects temporal correlation along the planning horizon when generating dtt+H which potentially leads to deeper ex- ploration of the state space. Meanwhile, the memory keeps the elite candidates generated at each CEM iteration and carry fraction of them over to initialize the next iteration. Reusing these high quality samples could speed up the con- vergence of CEM iterations. We refer to Appendices A.2 and A.3 for a detailed algorithm and hyperparameters used in this paper.
We use Figure 2 to illustrate how such conflict would ap- pear and why it is undesirable. The figure shows two sim- ilar tasks: open the door (left) and open the drawer (right). As different tasks all share the same observation space (Wotezyk et al., 2021), the locations of both the door handle and the drawer handle will be regarded as "the first object location", thus their values are placed into the same posi- tions of the observation vector. However, when the gripper pulls the handle, the door will allow a circular movement in the zy plane, but the drawer will constrain the displacement to the axis, immediately resulting in conflicting world dy- namics. This could prevent us from studying forgetting or transfer in CRL, because there is even no common solu- tion about the world, not to mention how the skills learned from one task can transfer to the next one. This issue is not obvious in Wolczyk et al. (2021) because they work with model-free methods, take advantage of task ID infor- mation, and learn separate heads for different tasks.
5 Continual Bench: An Environment for CRL Evaluation
Motivated by above-mentioned issues, we design Continual Bench, a dedicated environment for CRL evaluation, for assessing OA and comparing it with baselines. Our devel- opment is based the task primitives proposed by Meta- World, but instead of directly concatenating several tasks temporally like Continual- World, we redesign the environ- ment to arrange different tasks spatially. This ensures the existence of a unified world dynamics, which corresponds to the multi-task solution, a basic assumption for achiev- ing an ideal continual learner (Peng et al., 2023). Fig- ure 3 gives overview of the environment. Starting from the lower left and anti-clockwise, the 6 selected tasks with diverse difficulty levels are: peg-unplug, faucet-clos pick-place door, window-close, buttonspress. They are placed in a circle with maximal distances to each other, facilitating test on forgetting in the presence of dis-
The field of machine learning in general benefits from properly designed datasets and benchmarks to iterate al- gorithms. However, in CRL, there are not many widely adopted benchmarks, and prior works tend to themselves create a sequence of tasks from an existing suite of envi- ronments. For example, Kirkpatrick et al. (2017) select a sequence of Atari games (Bellemare et al., 2013) to learn sequentially, and Huang et al. (2021) vary the world prop- erties (e.g. the density of the same object) in single envi- ronment. The former type may overlook knowledge trans- fer across tasks due to the lack of meaningful overlapping, and the latter kind could be unrealistic because real world physics does not change.
Recently, Wolezyk et al. (2021) propose a benchmark
5

---

Continual Reinforcement Learning by Planning with Online World Models
Regret
Oracle agent
0.8
Success rate
Online agent
0.6-
0.0
t=1 T T T=4 T=
Continual tasks
Figure 3: The Continual Bench environment consists of tasks with diverse difficulty levels. All tasks share the same unified dynamics.
Figure 4: Regret can be calculated with the area in blue, which integrates the sub-optimality gap of the online agent. tasks to evaluate agents at global time step w:
tributional shift in state-action visitation when switching tasks. Different tasks also share meaningful overlapping. allowing us to study transfer, though it is not the focus of this work. We open source the code of Continual Bench' and hope this realistic but lightweight environment can ac- celerate the progress of CRL research. Please see Ap- pendix B for detailed environment specifications.
1 T
AP(w)
Pr (w),
with
(8)
Pr (w) Ex, Pu
11 [[[| gl12 <8) =8 po
(=0
Te is the number of tasks an agent has experienced at step w, Pr (w) is the success rate of the policy at time w on task т, and Il is the indicator function. Intuitively, this metric reflects the offline performance of π at time step w across all observed tasks, thus accounting for forgetting.
Experiment
We evaluate OA using Continual Bench, and compare its CRL capability with several continual learning baselines under the same agent design framework.
We also define regret to better capture the online perfor- mance of an agent:
6.1 Setup and metrics
1 [11 (1-PI, (I,(T))
We mainly focus the model-based planning framework, where all agents we compare are model-based RL agents that learn world model using experiences and act using CEM planning with MPC (refer to Figure 7). In this setup, OA learns the world model online (Section 4. while the baselines employ a deep world models, default choice of many prior approaches (Chua al., 2018; Wang Ba, 2020; Kessler et al., 2023).
Reg(w) = w
(1-
which effectively calculates the normalized area enclosed by the online agent's performance curve and that of an ora- cle agent (with success rate always being 1), as depicted in Figure 4.
6.2 Baselines
The reward function is provided to the agent and changed upon task switch, but no task boundary information is given to the world model learning (unless the continual learn- ing baseline requires it). We let all agents run in Con- tinual Bench continuously solve a sequence tasks in the order (pick-place, button-press, door-open, peg-unplug, window-close, faucet-close). The order is chosen such that adjacent tasks are as spatially far away as possible to increase the nonstationarity.
Continual learning (CL) methods can be categorized into several paradigms, such regularization-basel methods (Kirkpatrick et al., 2017; Zenke et al., 2017; Aljundi al., 2019b). replay-based methods (Riemer et al., 2018; Aljundi et al., 2019a: Chaudhry et al., 2019b) and architecture- based methods (Mallya & Lazebnik, 2018; Kang et al., 2022). Prior CRL works typically apply CL methods to base RL agents, such as the model-free Soft Actor- Critie (SAC) (Haarnoja et al., 2018; Yang et al., 2023) the model-based Dyna (Sutton, 1990; Liu et al., 2024). We follow the practices to test multiple representative CL methods with both a model-free actor-critic agent and a model-based planning agent. Concretely, we consider EWC (Kirkpatrick et al., 2017) and (Zenke et al., 2017) for regularization-hant based methods, PackNet (Mallya
The performance is measured accordance to the CRL objective defined in Eq. (1). Since all the tasks Continual Bench have a binary success measure depending on current and goal state, define average performance on all seen
hhttps:///ithhe www../o.ccccala.a.e.eo http://hwww...cemalaegg/oonnnaaa
6

---

Continual Reinforcement Learning by Planning with Online World Models
Task pick-place
Task 2: button-press
Task 3: door-open
1.0
0.8
0.5
Sureess rate
0.1
0.4
0.4
0.2
0.0
0.0
0.0
Task 4: peg-unplug
Tosk 5: window-close
Task 6: lanoet-close
1.0
1.0
0.8
Sureess rate
0.6
00.4
0.4
0.2
0.2
0.0
0.0
Episodes
xI
Episodes
x10P
Episodes
x10
Finc-tuning
SI
Coresct
Perfect Memory
OA
Figure Performance comparison our agent OA with different deep model-based planning agents on Continual Bench environment. Results are aggregated from 7 runs with different seeds. The solid lines show the means and shaded areas are the standard errors. The grey regions denote the learning period for the current task r E [1, .. 6].
& Lazebnik, 2018) for architecture-based methods, and Coreset (Vitter, 1985) for replay-based methods. We also include the vanilla baseline of Fine-tuning which fine- tunes the model only on the current task, and upper-bound baselines Perfect Memory that train on all previous data, More details about the baselines can be found in Ap- pendix A.5.
window-close). Maintaining Coreset for experience play performs better than SI, resembling the state-of-the-ee performance of replay-based methods in supervised contin- ual learning (Boschini et al., 2022). However, its perfor- mance diminishes as the proportion of older experiences decreases. This suggests that determining the size of the replay is challenging design choice, especially when the number of tasks and the number of samples in each task are unknown. In comparison, OA stands out by demonstrat- ing non-forgetting capability building on the online FTL world models. For all tasks it has seen, OA maintains a high performance over all remaining time steps, matching the performance of Perfect Memory. However, Perfect Memory ensures non-forgetting by keeping all the data and performing SGD updates until convergence, which is inef- ficient with unboundedly growing computation for lifelong agents. OA, on the other hand, achieves the same perfor- mance by much more efficient online update with con- stant overhead.
6.3 Learning curves of model-based agents
We first present the comparison of learning curves between OA and baseline agents in the model-based planning set- ting on the Continual Bench environment. Figure shows the performance (success rate) of different agents for all 6 tasks, measured along the learning process. Different tasks presented the agents sequentially, with the shaded region in grey denoting the learning period of that par- ticular task. We first observe that during the learning pe- riod of a particular task, all agents effectively learn to per- form the task well, showing the plasticity of model-based agents: can reuse the knowledge of the world dynam- ics and quickly adapt to new tasks by planning with cor- responding reward functions. Focusing the stability (to retain the performance previously seen tasks), notice that Fine-tuning agents immediately cease to perform well on the old task when switching to the new one due to catas- trophic forgetting. Using SI to regularize the weight up- dates can alleviate the forgetting slightly, resulting in a bit higher final performance on two tasks (button-pree and
6.4 Average performance of model-based and model-free agents
We show the average performance (Eq. (8)) curves over all seen tasks in Figure 6(a). We scale the performance by Ta for better visual effect. The dashed lines denote task 6 switching, upon which we can observe performance jump of different scales for agents. Due to the loss of stabil- ity, deep agents (even with various continual learning tech-
7

---

Continual Reinforcement Learning by Planning with Online World Models
Average performance
Sparse model utilization
Firal average performance
Finel average performancce
015
Prefet Menery
0.5
0.4
0.4
012
0.2-
Moueyy
14394 LAREE
28000
03 Utiliation
Episndes
Epiendoo
Builee size
(1)
(b)
(c)
(d)
Figure (a) The average performance curves of different methods. (b) Ratio of the activated weights of the OA sparse world model. (c)
Final agent performance with different buffer budgets. (d) Ablation on the world model sparsity.
Table 1; Comparison on average performance and regret (in %) of regularization--eee based (7). architecture-base (4,52. replay-based and perfect memory (dia methods.
world model. Figure 6(b) shows the change of the model utilization (ratio of activated weights) of OA. Interestingly, can observe the sparse model gradually grows its uti- lization as learning more new tasks. Note that even if OA reaches nearly full model utilization, it can still learn from newly coming data because the model solves for an overall least squares solution based on sufficient statistics (Eq, (3)) and the sparse learning is no-regret (Theorem 1). con- trast, methods based on iterative solution finding from a prior solution subspace (Farajtabar al., 2020; Peng al., 2023) may exhaust the solution space quickly and fail to learn from new data. Figure 6(c) compares the perfor- mance against the buffer size, and shows that OA achieves the best performance under buffer constraints. Finally, in Figure 6(d) we ablate the effect of different sparsity of the model. A denotes the number of bins in Losse ((Liu al., 2024), Appendix A.1); larger A means sparser models. The results show that sparser model contains greater pacity and reaches higher performance with less activated weights. This is a natural result as D increases with (the network is becoming wider), but the attractive property is that both the update and inference computation remains al- most the same since is fixed. This shows the flexibility of choosing suitable sparsity levels to model environments with different complexities.
Base
Methods
AP (t)
Reg (1)
Fine-tuning
0.69
67.
Model-free -irel--reee
EWC
31.45
77.38
(SAC)
PackNet
41.72
78.50
Coresct
37.31
77.97
Perfect Memory
41.51
77.58
Fine-tuning
24.86
37.74
Model based
SI
39,96
33.57
Coreset
61.83
30.83
Perfect Memory
73.09
30.95
OA (ours)
72.93
27.62
niques) struggle to succeed on all sequentially seen tasks. Only OA (and deep agent with perfect memory) can improve the average performance continually. Table gives quanti- tative results of the average performance (AP) and the re- gret (Reg) measured over all tasks at the final step of agent learning. For model-free agents, various CL methods im- prove AP compared to the Fine-tuning baseline, while they all incur high regret. This might be because the learned policy and value are hard to adapt to new tasks quickly. On the other hand, model-based agents generally exhibit lower regret by planning with the learned world model, which is shared across tasks thus being more generalizable. Notably, the proposed method achieves even lower regret than deep agents with perfect memory. This is because our incremen- tal update ensures the optimal solution at each step, while SGD updates on all previous data are not guaranteed and could be less efficient. While model-based results in Fig- ure & Table measure the agent's CRL performance (a joint result of world model and planner), the world model accuracy provides more direct measure. Please see Ap- pendix C for more results.
7 Conclusion
In this paper, we propose to tackle the CRL problem by de- veloping Online Agents, which should learn a shared com- ponent throughout the agent's 's lifetime and update incre- mentally. In environments with a unified world dynamics across tasks, we learn such a shared component by online FTL world modeling and act by planning. Theoretically we prove that the sparse online update learns a no-regret world model. To assess the agent's CRL capability, we fur- ther develop benchmark named Continual Bench. Empir- ical results show that our Online Agents outperform sev- eral strong baselines under a fair setting, demonstrating its effectiveness and promise in building future autonomous agents.
6.5 Ablation analysis
OA perceives the state-action inputs as high- nigh-dimensional sparse features (Section 4.,,, and naturally learns sparse
8

---

Continual Reinforcement Learning by Planning with Online World Models
Impact Statement
De Boer, P.-T., Kroese, D. P., Mannor, S., and Rubinstein, R. Y. A tutorial on the cross-entropy method. Annals of operations research, 2005.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Farajtabar, M., Azizan, N., Mott, A., and Li, A. Orthogo- nal gradient descent for continual learning. Chiappa, S. and Calandra, R. (eds.), The 23rd International Con- ference on Artificial Intelligence and Statistics, 2020.
References
Gareia, C. E., Prett, D. M., and Morari, M. Model predic- tive control: Theory and practice-a survey, Automatica, 1989.
Abhas, 7., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Proceedings of The 2nd Conference Life- long Learning Agents, 2023.
Garcia, F. M. and Thomas, P. A meta-mdp approach to exploration for lifelong reinforcement learning. Neural Information Processing Systems, 2019.
Abel, D., Barreto, A., Van Roy, B., Precup, D., Has- selt, H. P., and Singh, S. A definition of continual re- inforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Gaya, J.-B., Doan, T., Caccia, L., Soulier, L., Denoyer, L., and Raileanu, R. Building subspace of policies for scalable continual learning. International Conference on Learning Representations, 2023.
Aljundi, R., Lin, M., Goujaud, B., and Bengio, Y. Gradi- ent based sample selection for online continual learning. Neural Information Processing Systems, 2019a.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, Soft actor-critic: Off-policy maximum entropy deep rein- forcement learning with stochastic actor. In Interna- tional Conference on Machine Learning, 2018.
Aljundi, R.,
Rohrbach, M., and Tuytelaars, Selfless se- quential learning. International Conference on Learning Representations, 2019b.
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat- form for general agents. Journal of Artificial Intelligence Research, 2013.
Huang. G.-B., Chen, L., Siew, C. K., et Universal ap- proximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans. Neu- ral Networks, 2006.
Bellman, R. A markovian decision process. Journal of mathematics and mechanics, 1957.
Huang, Y., Xie, K., Bharadhwaj. H., and Shkurti, F. Con- tinual model- reinforcement learning with hyper- networks. In IEEE International Conference on Robotics and Automation, 2021.
Boschini, M., Bonicelli, L., Buzzega, P., Porrello, A., and Calderara, S. Class-incremental continual learning into the extended der-verse. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2022.
Hutter, M. A theory of universal artificial intelligence based on algorithmic complexity. arXiv preprint arXiv: cs,004001, 2000.
Chaudhry, A., Ranzato, M., Rohrbach, M., and Elhoseiny, M. Efficient lifelong learning with A-GEM. In Interna- fional Conference on Learning Representations, 2019a.
Isele, D. and Cosgun, A. Selective experience replay for lifelong learning. AAAI Conference on Artificial Intelli- gence, 2018
Chaudhry, A., Rohrbach, M., Elhoseiny, M., Ajanthan, T., Dokania, P., Torr, P., and Ranzato, M. Continual learning with tiny episodic memories. In Workshop on Multi- Task and Lifelong Reinforcement Learning at ICML, 2019b.
Johnson, W. B. and Lindenstrauss, J. Extensions of lip- schitz mappings into hilbert space. Contemporary Mathematics, 1984.
Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning in a handful of trials us- ing probabilistic dynamics models. Advances in Neural Information Processing Systems, 2018.
Kang, H., Yoon, J., Madjid, S. R., Hwang, S. J., and Yoo, C. D. Forget-free continual learning with soft- winning subnetworks. International Conference on Ma- chine Learning, 2022.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Leverag- ing procedural generation to benchmark reinforcement learning. International Conference on Machine Learn- ing, 2020.
Kessler, S., Parker-Holdde J., Ball, P. J., Zohren, S., and Roberts, S. J. Same state, different task: Continual rein- forcement learning without interference. AAAI Confer- ence on Artificial Intelligence, 2022.

---

Continual Reinforcement Learning by Planning with Online World Models
Kessler, S., Ostaszewski, M., Bortkiewicz, M. P., Zarski, M., Wolezyk, M., Parker-Holder, J., Roberts, S. J., and Mis, P. The effectiveness of world models for contin- ual reinforcement learning. In Proceedings of The 2nd Conference on Lifelong Learning Agents, 2023.
Pineda, L.., Amos, B., Zhang, A., Lambert, N. O., and Ca- landra, R. Mbrl-lib: A modular library for model-basee reinforcement learning. Arxiv, 2021.
Pinneri, C., Sawant, S., Blaes, S., Achterhold, J., Stueckler, J., Rolinek, M., and Martius, G. Sample-efficient cross- entropy method for real-time planning. In Conference on Robot Learning, 2021.
Khetarpal, K., Riemer, M., Rish, I., and Precup, D. To- wards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 2022.
Powers, S., Xing, E., Kolve, E., Mottaghi, R., and Gupta, A. Cora: Benchmarks, baselines, and metries as a plat- form for continual reinforcement learning agents. In Proceedings of The 1st Conference on Lifelong Learn- ing Agents, 2022.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In International Conference Learning Representations, 2015.
Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A.. Milan, K., Quan, J., Ra- malho, T., Grabska-Barwar Barwinska, A., al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Riemer, M., Cases, I., Ajemian, R., Liu, M., Rish, I,, Tu, Y., and Tesauro, G. Learning learn without forgetting by maximizing transfer and minimizing interference. Inter- national Conference Learning Representations, 2018.
Rolnick, D., Ahuja, A., Schwarz, J., Lillicrap. T. and Wayne, G. Experience replay for continual learning. Neural Information Processing Systems, 2019.
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 1712.05444 2017.
Rubinstein, R. The cross-entropy method for combinatorial and continuous optimization. Methodology and comput- ing in applied probability. 1999.
Küttler, H., Nardelli, N., Miller, A., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T. The nethack learning environment. Advances in Neural Information Processing Systems, 2020.
Sancaktar, C., Blaes, S., and Martius, G. Curious explo- ration via structured world models yields zero-shot ob- ject manipulation. In Advances in Neural Information Processing Systems, 2022.
Liu, Z., Du C., Lee, W. S., and Lin, M. Locality sensitive sparse encoding for learning world models online. In International Conference on Learning Representations, 2024.
Schwarz, J., Luketina, J., Czarnecki, W. M., Grabska- Barwinska, A., Teh, Y. W., Pascanu, R., and Hadsell, R. Progress compress: A scalable framework for continual learning. International Conference on Machine Learn- ing, 2018.
Mallya, A. and Lazebnik, Packnet: Adding multiple tasks to single network by iterative pruning. IEEF/CV Conference Computer Vision and Pattern Recogni- tion, 2018.
Shalev-Shwartz, S. et al. Online learning and online con- vex optimization. Foundations and Trends in Machine Learning, 2012.
Nagabandi, A., Finn, C., and Levine, S. Deep online learn- ing via meta-learning Continual adaptation for model- based rl. International Conference on Learning Repre- sentations, 2018.
Sharma, A., Xu, K., Sardana, N., Gupta, A., Hausman, K., Levine, S., and Finn, C. Autonomous reinforcement learning: Formalism and benchmarking. International Conference on Learning Representationss 2022.
Negenborn, R. R., De Schutter, B., Wiering, M. A., and Hellendoorn, H. Learning-bast based model predictive con- trol for markov decision processes. IFAC Proceedings Volumes, 2005.
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zeullmoyer, L., and Fox, D. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF confer- ence computer vision and pattern recognition, 2020.
Peng, I.. and Vidal, R. Block coordinate descent on smooth manifolds: Convergence theory and twenty-one exam- ples. Conference on Pursimony and Learning, 2023.
Peng, L., Giampouras, P. V., and Vidal, R. The ideal con- tinual learner: An agent that never forgets. International Conference on Machine Learning, 2023.
Sutton, R. S. Integrated architectures for learning, plan- ning, and reacting based on approximating dynamic pro- gramming. Machine Learning Proceedings. 1990.
10

---

Continual Reinforcement Learning by Planning with Online World Models
Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In IEEE International Conference on Intelligent Robots and Systems, 2012.
Vitter, J. S. Random sampling with reservoir. ACM Trans- actions on Mathematical Software, 1985.
Wang, T. and Ba, J. Exploring model-based planning with policy networks. International Conference on Learning Representations, 2020.
Williams, G., Aldrich, A., and Theodorou, E. Model predictive path integral control using covariance vari- able importance sampling. arXiv preprint arXiv: 1509.01 149, 2015.
Wolezyk, M., Zajac, M. Pascanu, R., Kuciński, L., and Mitoš, P. Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 2021.
Yang, Y., Zhou, T., Jiang, J., Long, G., and Shi, Y. Con- tinual task allocation in meta-policy network via sparse prompting. In International Conference on Machine Learning, pp. 39623-39638. PMLR, 2023.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evalu- ation for multi-task and meta reinforcement learning. In Conference on Robot Learning. 2019.
Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International Confer- ence on Machine Learning, 2017.
Zhang. F. The Schur complement and its applications, vol- ume 4. Springer Science & Business Media, 2006.
Zhuang, H., Weng, Z., Wei, H., Xie, R., Toh, K.-A., and Lin, Z. ACIL: Analytic dlassincremennal learning with absolute memorization and privacy protection. In Ad- vances in Neural Information Processing Systems, 2022.
Zhuang, H., Weng, Z. He, R., Lin, Z., and Zeng, GKEAL: Gaussian kernel embedded analytic learning for few-shot class incremental task. In Proceedings of the IEEE/CVF Conference Computer Vision and Pat- tern Recognition, 2023.
Zhuang, H., He, R., Tong, K., Zeng, Z. Chen, C., and Lin, Z. DS-AL: dual-stream analytic learning for exemplar-free class-incremental learning. Proceedings of the AAAI Conference Artificial Intelligence, 2024.
11

---

Continual Reinforcement Learning by Planning with Online World Models
A
Agent details
In this section we provide more details about OA, including the sparse non-lineat feature encoding, the learning and acting process, the model hyperparameters, and the planing algorithm. We also include baseline and experimental details for reproducibility.
A.1 Sparse feature encoding
We revisit the feature construction process of Losse (Liu et al., 2024), on which we will build our online world models for planning. At time step t, given an input vector 2; E IRS+A, we first randomly project it to obtain bounded random features (a;;) = sigmoid( Pæ:), where E (Rdx()+A) is sampled from multivariate Gaussian A(O, S+A similarity (Johnson Lindenstrauss, 1984), and sigmoid is applied element-wise to ensure each feature value bounded between (0, 1). such that the binning edges can be clearly defined. Next, each element of (xx,) is binned softly by locating its neighboring edges and computing the distances from them. We illustrate using 1-dimensional feature grid with A bins. For the i-th element of a (æ:)) let I; = (A - 1) (x;);; denote its projection location on the grid. Then the "activated" indices are
1 I) to preserve feature
= [[T,,, [T]] + 1],
and their associated values are
0. = - (4, - (4,)) [4]]
Therefore, the i-th slice of the resulting feature vector a A-long zero vector with non-zero values filled indices s; with values Uts and the final feature <(xx)) is the concatenation of d such slices.
A.2 The learning and acting process
Online Agent (OA) Learning and Acting Loop
Require:
zero-initialized agent memories A (0), ( and world model weights W sparse encoder S+A + RR; initial index s [D] planner CEM; sequence of tasks ( initial state 8 pot time step
1: loop
OA runs forever, updates per step
2:
if task changes then
3:
Pe + init values E AxH
4: else
14 +- shifted pt- (fill the last column with init values)
end if'
!! (A (1-1) +1 A ( (((-1))
(1-1)
7:
a,, M1.1111 + CEM(s,, W ,4. RT) RT) 81+1 environment(s,,d;
Appendix A.4 for details
9:
10:
[st. asl: Yi + 81+1 8:
11:
4 nonzero_inneeeeee (onzero_inde(((x))
A
A (1-1) + (x:)
(( + B (1-1) + o. (2.)
13:
14:
t+
15: end loop
Figure 7: OA learning and acting loop.
12

---

Continual Reinforcement Learning by Planning with Online World Models
A.3 OA hyperparameters
To build the sparse world model, we use 300 2-dimensional Losse features with A = We perform a coarse sweeping over different sparsity levels (A (5, 7, 9, 11)) and find A 5 slightly under-fits the environment while all others give robust good performance. See Figure 6(d) for comparison. We find x 0.005 as a good regularization strength without any tuning.
For the planner, we use candidate sampling size of N = 150 with planning horizon H = 15 and K = 3 iterations. The ratio of elite candidates is 0.1.
A.4
CEM planning
We give detailed algorithm of CEM that we use to extract policy from learned model below.
Require:
candidate sampling size N; number of iteration K; planning horizon H; init noise init-
1: function CEM(&, W. nr, R
2: о f init values E IRAXH
3: for 1, K do
1* add colored noise
#/
4: candidates = (a NN < N samples from clip(p + (A, Oa)
/* memory $/
5:
add fraction of elite-set into C kf
6: t- unroll N trajectories to get returns
Σ R(s so =8, 8.41 + +(s(s, a.)) W
7:
elit alite-set <- best K candidates
8: 11, + fit Gaussian distribution to elite-set
9: end for
end function
A.5 Baseline details
Fine-tuning. This serves as minimal baseline for agents that use deep world models. The agent is allowed to keep all the data within the current task, but cannot carry the data over to the next task. The world model is repeatedly fine-tuned using a sequence of datasets.
Coreset. This refers to replay-basee based CL methods where small buffer kept across all tasks. We use reservoir sampling (Vitter, 1985), which updates the buffer such that its distribution approximates the empirical distribution of all observed samples. We use large buffer (10K) in experiments.
Synaptic Intelligence (Zenke et al., 2017). As regularization-be based method, synaptic intelligence (SI) estimates the importance of each parameter based on the gradients and weights deviation, and regularizes the parameter updates based on the recorded importance. Similar to Fine-tuning only the data the current task can be kept, and task boundary information is needed for SI to recompute the importance measure. The two hyper-parameters used in SI, c = 0.5 and { 0.05, are selected using a coarse grid search.
Perfect Memory. It is impractical assume unlimited buffer capacity, but this baseline serves the upper bound for agents using deep world models. It can also be regarded as deep version of the FTL world model (Liu et al., 2024) the current model is optimized using all previous data but in an inefficient way.
In all above baselines, we use a 4-layer MLP with hidden size 200 and ReLU activation for world modeling. We use the Adam optimizer (Kingma& Ba, 2015) with learning rate of x 10-4 and minibatch size of 256. Since updating deep models per time step is computationally impractical, especially for CRL settings where the dataset size grows along inter-
13

---

Continual Reinforcement Learning by Planning with Online World Models
Task 1: pick-place
Task button-press
Task 3: door-open
10-
10"2
Model error
10
10T4
10"4
10-3
10-2
10-6
10-4
10-4
Task 4: peg-unplug
Task 5: window-close
Task 6: faucet-close
10
Model yrrre
10-3
10
10
10-4
10-1
10-1
10-4
3
Epinodes
x10P
Episodes
x10
Episores
X10*
Fine-tuning
SI
Coreset
Perfect Memory
OA
Figure Curves of the world model errors. Same as the performance measure in the main text, results are aggregated from 7 runs with different seeds. The solid lines show the means and shaded areas are the standard errors. The grey regions denote the learning period for the current task [1.2.....6]
actions, we update the models every 250 environment step. On every update, the deep models are trained until convergence over all available data (stopped when the validation loss 5% holdout data does not improve over 5 consecutive epochs). All hyperparameters of the planner are the same OA's for a fair comparison (see Appendix A.3).
A.6 Experimental details
All experiments in the paper are run on the internal cluster, with each job consuming one A100 GPU and 16 CPUs. The training time single experiment ranges from 10 hours to 15 hours, given a total 600 episodes budget. We develop our agent and the baselines using the MBRL Library² (Pineda et al., 2021) by Meta (MIT license, v0.2.0).
B Continual Bench details
We build Continual Bench based on the Mujoco (Todorov et al., 2012) physics engine and task primitives from Meta-World (Yu et al., 2019). We include in total tasks (pick-place, outton-presss door-open, peg-unplug, window-close, faucet-close) with diverse difficulty levels. Incorporating all the information about the world on the table, we have dim(S) 26. The state includes: hand position (3), gripper openness (1), button position (3), door handle position (3), door opening angle (I), window handle position (3), faucet handle position (3), peg position (3), block position (3), gripper velocity (1), gripper pad offsets (2). The action space is the same Meta- Kora-,orld, 2-tuple consisting the change of end- effector position in 3D space (3) and a normalized torque applied to gripper fingers (1). When the environment starts, each episode can be at most 500-step long, with early termination on task success. The reward function /² is changed according to the task on hand upon switching. All reward functions are defined based on their well-crafted subroutines (Section E. of Yu et al. (2019)).
C More empirical results
We present more empirical results in this section. In Figure 8, we show the mean squared error (MSE) of world models when the agents learn sequentially on the Continual World environment, to more directly investigate the forgetting issues. We keep a separate buffer for each task, and measure the MSE the model on the data of every seen task. We can observe that the baseline agents (except the one with perfect memory) all exhibit increasing model errors when leaving the current task, but OA can maintain nearly the same accuracy along the whole learning process.
14

---

Continual Reinforcement Learning by Planning with Online World Models
D Limitations and Future Work
One limitation of the current OA is that the online world model can only deal with moderate-dimensionn. state-based observation and does not capture world uncertainties. We envisage that developing highly capable probabilistic models that permits efficient online learning is future research avenue. Besides, though the current framework is simple yet effective, the model planing does not incorporate explicit exploration. We leave this topic for future research. The proposed benchmark environment, Continual Bench, is limited to episodic settings with explicit task switch across episodes. This is mainly due to the challenge brought by irreversible states (Sharma et al., 2022) since the underlying MDP is non-ergodic (e.g,, block falling outside the table can never be placed back). Developing a reset-free CRI. environment consisting of many tasks a potential future work.
E Theoretical analysis and proofs
E.1 Closed-form solutions for sparse online world model learning
Lemma (Sparse Online Model Learning Minimizer). For every E [1, T), S (D), the following problem at time step I. has unique global minimizer
( == arg min
(t-1)) Wel) 1 + i 1l (t-), W.lli
(10)
W.
The global minimizer W has a closed-form solution as Eq. (4), i.e.:
((t) A 111 ( A (0) w(-
W
where (1)
[W (1-1), 1), (
J,A' (t))
TJ t-1
1, B(t) = - and they can be decomposed into
A (4)
A (8)
B
w
)
A
.B()
w
A
B(d)
!
Proof. Uniqueness. By the convexity of Frobenius norm and both two terms in Eq. (10) are quadratic in W., the objective function Eq. (10) is strictly convex and has a unique solution.
Closed-form solution. Note that
& Y + W 1
- + || 2 Y)F WI
1 1 W
=Tr "T-(WT क ww) + TO (T 2T (WT
=Tr(W AW) + (Y 2Tr ( B) + W
W W.
)
A
Ass
W
)
+ +T:(YTT) 2Tr ((
W3
W, )
B
-T
A.s
Ass
W.
B
W
W
+ Tr
W.
W.
Tr ( A. Wz) + Tr (W..T A,s W.) + 2T (W. A Wz) + Tr (YTT)
2Tr W B₂) - 2Tr (W.1) B,)
1 Tr
(
W.) + ト Tr (W. W.)
1
15

---

Continual Reinforcement Learning by Planning with Online World Models
As result,
min W. x
[Ws;: W,) Y. W
min W.eK
Tr (W: ,. W.) + TTr (W. Aga W.) 2T (W, B.) + X
(W, W.)
(1)
Differentiate the objective function Eq. (11) w.r. W. and set it to zero, obtain
W, AA, + +1+1 -1
(B, A Wz).
E.2 Theoretical analysis for online model Learning
Before derive the regret bounds for sparse online model learning Eq. (10), first derive the regret bound for online model learning Eq. (3). The developed regret bound of online model learning, given in Corollary will be further used to derive the regret bounds for sparse online model learning Eq. (10) in Appendix E.3.2.
E.2.1 ONLINE MODEL LEARNING
Fort 2 1, i = ,,...,t, t, we define
((W) Tr(W o();))((;;)) T W) Tr(y, y1) (T (o(:)))) )
1 I
Lgg (W)
Then, the regularized least squares problem Eq. (3) can be rewritten as
Vt. ( arg min
Σ ((W)+R(W) R(W).
(12)
W WERDXS
Lemma 2. Let (1), W be the sequence produced by Eq. (12) (or equivalently Eq. (3)). Suppose that there exist such that Assumption holds, then for all t. 2 1, we have
(1 (1- w + A. w((1) 3 w(() + A.:
where
Σ ((:)))((z))) + +11) p(x.)Dt
4, :
(13)
Proof.
(f
V1,
(x(x))) W 21b(x:)))
dW
Then we have
d
d
de W
df((W)
w Σ 20(x))o(z;) TW Σ 20(2 20(x,) + w.
2 W.
dW
dW
dW
16

---

Continual Reinforcement Learning by Planning with Online World Models
Therefore, we get
Σ (x;)(((1))) T + )1
w(++1)=
Σ ((z))\ T
Σ
o xc ) T + )
Σ (22;)1? + Σ ((x)))(()))
+ ++11)
o(x))y1
T + )
p(x;)o(a;) T + +,1 1 w + Σ O( (zzz)) T+ 11) o(xi)vl
Σ
0 [ (
Σ ((x)))(z;)) T + 11
)(xi))(E)) w( +1e
= I 1 w(1)+ +..
(Assumption 1)
w + A..
=
On the other hand,
Σ p(x))(();) T + 11
l(++1= =
Σ b(xx))0
Σ )(Z)))(;)) T + +
Σ (()))))))) + 21 -1
Σ p(z;)
+
dxz))
Σ (a.)c((a))) + )1) -1
3 Σ (aa))((a:) + +
Σ (ax)) T
d(x;)y1
(0) + A,
Lemma 3. Let (l), W .... be the sequence produced by Eq. (12) (or equivalently Eq. (3)). Suppose that there exist
A 2 Cy > 0 such that Assumptions I and 2 hold. Then for all 2 1, we have
) - { D t.
17

---

Continual Reinforcement Learning by Planning with Online World Models
Proof. Recall from Lemma that A. 1 W W W 3 A.. Then we have
- - -[T(o(a=)((=) W - ( w
= Ir Ir(0((;))((z)) T w( 2 (T(((()))) w
- (wit+1) w( - ) + 2 +2 ((x))))())) (w(ttt-ww") w((
- -2Tr(6(0)1) )( + (TT((())))) W()T) ((11)
Tr (d(x))((z)) ( ( (v())))))) + 2 [ (
-w
+2/T((((m))) (w((t)) w
- ((((()))(()))))AA)) + (6 A, ((T)
(6(2))())) 4T)
(Lemma 2)
((( ) + 2 Tr ((z)))(Ez))
(z)))((;; + x) p(x))v1 wlOTT)
3 I by Assumption! I
+ ( )((())) (( o(z))((x;) + ((
< [ + { Ir (a)))) + (I(o(a)),TW(rr)) w()T) Ir 2 (1(z))(()))
( [ + t1) "
2 I by Assumption ]
+ Ir (ol + c3D 2
< (T(
For the first term on the right hand side of the above inequality, we have
Tr ( TAA
Σ ((;))((z)))T + 41)
(($ (())))(()))))) + 11
= Th ((x)))()))T T
Σ (a))((a)) + \1 -1
= =1yy:112 Tr ((x)))()))
((x:))(())))
Σ (ax))
11 -1
Σ (ax))())) T 11)
$ 2 Tr y
0 (2)))(())) T
d(2))((a;)
Σ o az (o(x))
T
x
Σ o(z))o(x)) T 111 )(z1)((Z)) Σ ozx:)o(zz)) T+ k1
2 Tr
)(x1)d(Z;)
2/r
(Assumption 1)
< D 1
(2212 for all / t= 1,2....)
18

---

Continual Reinforcement Learning by Planning with Online World Models
For the second term, have
Tr (ó(zz)))
(o(o))).T(oo) 'Ir
d((r)))
Σ ((x))o(a)))
+
Σ (()))(())))) x221)
Tr
(1 vid((a))
1 Tr
Σ (()))
Σ (Zx)) T
+ 2111
< 47
2 Tr
((z))o(z)) + +
Σ )(22)(())))
t
21 T
Σ (p)))((())) mn)) T + 1
Σ (l (1))))))(=)))
a D.
(Assumption 1)
Combining the above results, we obtain
((((W)))))) - f(((())) s Tr (0(2)))((1) AAT)) (AA + 4 Tr (o(z))) w(oT)
s &A 1 4e D
t s SA _ + 222
(A 1121)
s t
Lemma 4. Let W()), W(2).... be the sequence produced by Eg. (12) (or equivalently Eq. (3)). For all t 2 and all Ç E Dx8 we have
T Eawwtt
Proof. We prove this lemma by induction. For 1, follows directly from the definition of ((+1), i.e. W arg mixe fiW.. that
fw S (E)...
Assume that the inequality holds for T - 1, then for all & E DxS we have
((W(+))) < ME)))
Adding fr (T(W))))) to both sides we obtain
Ei(wittt) s ST(W((+1)) + T-1
T
Σ
Σ fi(S)
19

---

Continual Reinforcement Learning by Planning with Online World Models
The above holds for all & and in particular for W(++1), Thus,
T
f((((tt)) $ f((((++))) = min
Σ
A(W)
WEROXS
Lemma 5. Let W(1), ( .... the sequence of vectors produced by Eq. (12) for equivalently Eq. (3)). Then, for all t 1 and all & E RDx we have
T
r
Σ (n(Wi)) - fe(E)) & (5g(5) - Log WW))
) -
+
Proof.
Let fo(W) = Leg(() (W). Using Lemma we obtain
Σ f(W(+)) s Σ f(())
(14)
Adding El=o (o) to both sides and rearrange, get
T T E En(w(()) - = -
1=0
E.2.2 REGRET BOUND FOR ONLINE MODEL LEARNING
Now, we are ready to prove the regret bound for the online model learning problem Eq. (3):
Corollary (Regret Bound for Online Model Learning). Let ( w..).... be the sequence produced by Eq. (3) (or equivalently Eq. (12)). Suppose that there exist À 2 L, Cy 0 such that Assumptions I and hold. Then, for all T and all & E IR D have
T
1 Σ f((E) < Σ (((()))
7 Jci (llg(()) + 1).
Proof.
T
Σ ((() Σ fe(E) Log(E) Log (W(1) Σ (((W())-f((Wl))) f((((tt)))
(Lemma 5)
T
1
cw + Σ
5c
DA
(Lemma 3)
1=1
1
civ
(llg(T)) + 1).
T tm
< log(T)) + 1)
E.3
Theoretical analysis for sparse online model learning
E.3.1
USEFUL. L.EMMAS
Lemma (Schur Complement for Block Matrix Inversion (Zhang, 2006)), Given a matrix M that can be partitioned into four blocks as below,
M
M 12
M =
M
My
20

---

Continual Reinforcement Learning by Planning with Online World Models
it be inverted blockwise as:
Mi + Mii M125 Mi -S- M21 Mi
-M
M125-1) )
M
S-1
where S = -M M21 M12 is the Schur complement.
Proposition 1. Given A, B, w as defined by A 11-1 B = 1i-1 - and W = (A + x1)=²
and they
can be decomposed into
,w(0- (
w
Asa
A
A
A,3
Asa
B
B:
w!!!!
then, we have
(AAT+
( 111
+ +D B² (Ax (Ast A B.,
w = =S- BB- - -S-AAss ss
Ass + 111)
B3,
where S == Ass + A&z (Az + b1) -1 A D = (A + +
A.5(A A +
Proof. Consider the inverse of A + I, which can be written as:
A +
Ags
A+
Ass
Ass + $1
Applying Lemma 6 to our matrix A + I, we get:
A+ A+11 1) 11 =( (Asr - 1 + (Agr + A AsT 1)
+ +Axx+ -
Asa5-1)) )
-51 A A + 11
S-1
where S = Ags + 1 A, AAx + 1| -ss-- Ags-
Now, we can write the expression for W as:
w(t= = ( (Ass IT- + (A,s + I) A..S Ass (Ass+ 1-- -(Ass
(L - SS1 (A55 + +
+ 11) Sgg ) (
B. B.
S-1
Therefore, we get
w ( + 1 1) (Azz + 1 I)
x
Ass (Ax S-
11 BN B Ba-(Ass
+ x 1) S B.,
W(O.- - S-11 A Agg
+
|
1|
B + S
X
Let D := (A [ - S
A.z (A -1 we obtain:
(A
x 1
+ D +D) B (A + \ 1 A B.,
w = 5-1 B, - -5-1 1 A ( -1+t11) Br.
21

---

Continual Reinforcement Learning by Planning with Online World Models
Proposition 2. Suppose that Assumption 3 holds, then have
0 3 K
A +10+7 11)
((z))
AA 20 +
1
B
Proof. Note that
Asa
Azo
B3 B,
Asx
Ass
We have
A+ + 1 (
AAe + 21)= - -1
Az +
Ass
Ass I )
(0
0 0
(Ass + (As
A
0
A 8s
(Ais + x111) -1
)
0
I
Then, we have
((A \1 B.)
0
b.)
0
[Ags 211) 11)11 )
(B:)
B:
B.
Azs (Ass + 211
A + x
B.
(15)
Note that by the properties of the block matrix's spectral norm, we get
-1
(( o Az. (Aso + 11) 0 I
Azs
Age
+
x
+ ||
1
-1
Ass
Aso
+
1
+ 1
(16)
Additionally, if Assumption 1 holds for ey 2 1, we have for all
By,o(at) ((x))1 =
Σ ((c:)u7 vro(ar) ((.:)))T..((..)
Σ ((c.)o(Z)) plawve
Σ ((z))((1:)) + x1) o(az)))
(by Assumption 1)
y
Moreover, by
(gi((1)) ( ((x))))))~ = (6(x))((())))) y1) v1) yev.
22

---

Continual Reinforcement Learning by Planning with Online World Models
obtain
Σ ((z)) T + " d())) (uo(()) ((or))))(
B 3 2 y
2
Σ ((a)))())))
+ 1 ((rr) v.v. (o(x)) ( a(ad)e
Σ Eo(a))(a)) + 11) olos)a
(xx) 6(x1)
3 K" (1)))))
(17)
where K" 2 سد o(z,) T m(x,) + 1).
((z,) ((z+)
Combining Eq. (15) (17) we obtain,
0
1 B.) RT.)
A x1=
-1
(
0
A (A.s + +1) ) B
(A.s + 2T)
0
Azs Ass + 1 x1)
+1 (A (A+11) )1 B
AZA Ass +
2
+1 A ++++² -1
K'
((x))v
2
+ )1
K
:(x))y1
where we choose
-1 2
+ 1
111
K = K"
Ass
Asss
x
+1
1
-1112 2
E oaxs) T ((xi))
)
2 2
cy
((x1) (o)))T((2)) (((x)
+
\
Aza
A
+
- x
+1.
Proposition 3. Let W(1), W .... be the sequence produced by the online model learning Eq. (3) and (1). (2))
be the sequence produced by the sparse model learning Eq. (4). For all t 2 1, let (() ;= A A + }11
A and
TM = sup
[1 + X
1 I M() (A + 1 I - -11- 11
(18)
L.E((e......55))
Suppose that there exist X such that Assumption 3 holds. Then, for all 1. 1, we have KrMA..
w(++1) - W(+1)
where A, is defined by Eq. (13) in Lemma 2.
Proof. Note that A is symmetric and A,= AT then, M Ass (A + +1) -1 Ass
0, and have S ==
A.s + 1 Asx (A + 11) Azs Ass + 111 M. Moreover,
1
1 I-
A. Ass (Asx +
1--A A,s (Ass +
X
x
1 1) 2 0.
MS 1 (A +
x
23

---

Continual Reinforcement Learning by Planning with Online World Models
Then, with Proposition I, we get
/ - Ass + X
(A
11)
B, + AAsst + 1 -1
x1
A,
x
= -S B, -5--S-AA -s3 Ass + I1) -1
Ass + 1 x1)
-1
Bs
B.
Aa + 111
-1
((157 + x11 -1
A A3
+ D
Bg - (A3s + x 1) A B.
BB.
5-1 1 + Aax + 111 -1
Ass + 1 -1
A (0st-(Ass
(Ans + -
-1
Ass
B.
Ass + x 1 -1
+Asss
Asss + 111 -1
A.5 ( + 11) -1 + D) )) BN
5- As5
+
5 s-1 + Ass + 111 -1
Asa Ass + 11) -1
S- 1----(1-
(Ass + x B,
-1)
A
Ass 1as++} ++22) -1
1
-1
1 + 1
-1
s-I+ +
As3 Ags
+
1
x
+
Ass
A D
Bs
x
20
20
-1
Ass + 11) -1
Ass +la+/ 21) -1)
5 8-1 1 + Ass + 111)
Ass
Ass
(Ase
B.
Ans +
MS- (05-1-(An))
(Anx (Am+ B.,
= S-1+ +
set [A!! + t + ( 10 + - -11
Let rw == supr, (0 s.......
Frr we have
Ass + (A
=111 B,+ + Asss + x1 Ass (:
W(O) -
Ass + 1 )
-1
- -A + 11) B.
3 5 +
MS
1
1
M) ( AAs +
1 I - (M)
-1 -
+ (Ax+ B.
I +
A
+x x
x
( Asss + 11
Ass + - I+ + M) ( Asso + x
- - M)
-1
"-(Asc AAs + 1
21)
B
X
IPMI
Ass + 1 -1 (
I
Aes
+ 1
I M)
Aos + I (M M)
-1
-1 B
x
1(M
AAss + (1)² -1
3 (TM - 1).
B,
24

---

Continual Reinforcement Learning by Planning with Online World Models
Recall from Assumption and Proposition 2 that
(
0 (A + B =
A + 11)
which gives that
W(++1
-
W W(++1)
w((+1)
w (++1)
W + A W W((+1)
w A!l + 111 -1
w( + A
B A w
0; (
Al s5
+ 1 B +
A
x1)
w((t)
+ +
88
A(0) 1 1
-1
(rM - 1)
+ A
K(rM 1)4, + A.
(Assumption 3 and Proposition 2)
KrMA..
E.3.2 REGRET BOUNDS FOR SPARSE ONLINE MODEL LEARNING
Theorem (Regret Bounds for Sparse Online Model Learning). Let (L),, W .... be the sequence produced by the sparse online model learning Eq. (4). Let rM be constant defined as in Eq. (18). Suppose that there exist ow Cy and X such that Assumptions I to 3 hold. Then, for all T 2 1 and all E DDS we have
T Regret (T) = Σ ((WW)) - Σ f((5) <
1 f
+ 1)c + 1),
X
1=1
tm1
Furthermore, If cwwc///b /5(K3 + +)D((og(TT + +1) 2 and it satisfies Assumptions and 3, then we have
Regret(T) S cw Swrcy/20/00 20(K2r3 + 1)oglo)(T) + 1).
Proof. For t = 1, we have fe (()) (((W())) and f((W()) - f((W(1)) = 0.
For 2 2, denote by T'M = KrM, we have
25

---

Continual Reinforcement Learning by Planning with Online World Models
((( - ((((()))
(I(WiTd(as)daay) ( T 9 - - (x))))
-Tr(W (())))())))
- Tro v.) + + TT )(x)))))
Tr ( W( 2TH (0(z))y) W
- (w()))()))"((00)))))))) ( 2Tr (0(0)))) W()T)
Ir (())))(2))) (w (wr(o) (w()-w(.)) - ) + Ir ((()))))
S TM Tr (6(2)))((z) AA-AAT))) + 27' Tr (o(x)))))
(Proposition 3)
For the first term in the right hand side of the above inequality, we have
Tr (A.
Σ ((as))(aas)) + +²1)
Σ (ax))((()))) TT
+
Tr
((x)) T
Σ ((az))(ars) + x1)
(a)))(()))) T 1
112-1112 Tr ((x)))
(()-11)0(2111)
Σ ((zz))(Z;) + 21)
T + 1
$ 2 Tr
((x,)d(x))
0(2)-))0(21-1) T
Σ
Σ (z.))(ZZ)) + 71 ((E-1))(0)-1)
Σ )(x;))(2;) 111
2 Tr
)(z))((Z))
2GD (1 (t-1)² 1)22
1
(Assumption 1)
1
S DD t-1
(A212 1 for all .....)
For the last term, we have
[ (1)))))))) + 1
(r(((1)))TTT)) = Tr ((()))) yo- (f 11-10(0-1)
+ +
2 Tr 0(2))6(()) T - cy
Σ (Zz))((Z)) T
Σ ((z))o((;)) + +111 ((21)((2)111)
2 Tr y
< D t-1
26

---

Continual Reinforcement Learning by Planning with Online World Models
Therefore, the regret bound is upper-bounded as below:
T Σ (W Σ f(W(0)) + Σ (w(()) - T Σ f((E)
Regret (T)
$ (((W)))(((((()))
1
+
c +
+ 1)
(Corollary 1)
A
1
1
2 (rM + 2
+
c + 5 (TTog(T) + +1)
x
S e (PMA+ 2r (log(T' 1) + 1) + ス A + D + +1)
1
1 2 $ J +5 + 1)c, + +),
(22 1, 1)
which holds for all that satisfies Assumptions 1 and 3.
Furthermore, If cw 2 c5V5(K33 + L)D(log(T) + 1), and À = V - 1) satisfies Assumptions and 3, then we have
Regret(T) S Sweyy / 220(22r2 + ()D(lo)(T) + 1).
27


Reviews:
[{"review_id": "Xo2z6r3cQ0", "reviewer": "Reviewer_mVEi", "summary": "Update after rebuttal:\n\nI have read the author responses and reviews. I would like to maintain my score recommending acceptance.\n\n\nSummary: \n\nThe paper proposes an approach to Continual Reinforcement Learning (CRL) through the development of an Online Agent (OA) that leverages online world models. The central idea is to address catastrophic forgetting—a major challenge in CRL—by employing Follow-The-Leader (FTL) shallow models to capture world dynamics, paired with model predictive control (MPC) for planning. The authors introduce Continual Bench, a new benchmark designed to test both forgetting and transfer in CRL environments. Empirical results show that OA outperforms strong baselines built on deep world models with traditional continual learning techniques.\n\n\nOverall impression: \n\nI really like the way the paper is written. It is quite modular and clear. Sufficient literature has been cited throughout the text. The core idea of learning a unified world dynamics that can be shared among different tasks is appealing for transfer. I think the community will benefit from this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review and questions. Below we respond to the comments and raised questions.\n\n---\n\n***Q1: What is meaningful overlapping referred to?***\n\nWe intended to mean the overlapping of task attributes. For example, two Atari games may lack meaningful overlapping because they have distinct appearance and different underlying game logic. The lack of task attribute overlapping makes the study of transfer difficult. In our proposed environment (Continual Bench), we carefully designed a sequence of tasks that share certain attributes (e.g., moving the robotic arm, gripping objects), ensuring meaningful overlapping of task attributes.\n\nThank you again for your question, and we will make the term clearer in our next revision, as well as fixing the typo you mentioned.", "claims_and_evidence": "This paper claims that the OA agent can solve CRL tasks incrementally without catastrophic forgetting by planning with online world models. OA surpasses deep world models combined with various continual learning methods. Regret bounds for the sparse online model learning process are formally derived and justified with key assumptions (like feature mappings stabilization and bounded inputs/outputs).", "methods_and_evaluation_criteria": "The evaluation metrics used (average performance, regret, learning curves) seem appropriate. I really like the way the authors evaluate on all previously seen tasks even when learning on a particular new task to track catastrophic forgetting.", "theoretical_claims": "The paper provides a regret bound for the sparse online model learning process, ensuring that OA's world model updates incrementally without forgetting past tasks.", "experimental_designs_or_analyses": "The setup allows studying both forgetting (by testing how agents retain old skills) and transfer (by evaluating knowledge reuse across similar tasks). Tasks are sequenced to maximize distributional shifts, testing how well models generalize without task-specific identifiers.", "relation_to_broader_scientific_literature": "The work builds on model-based RL foundations (Sutton, 1990) and extends them into CRL.\nIt directly engages with recent FTL-based approaches (Liu et al., 2024) and compares against standard CL methods like EWC, SI, and PackNet. The benchmark Continual Bench critiques and refines prior environments like Continual-World (Wołczyk et al., 2021), addressing their limitations by ensuring consistent state spaces for testing transfer and forgetting.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Covered already.", "other_comments_or_suggestions": "Typo:\nLine 292: stat -> state", "questions_for_authors": "What is meaningful overlapping referred to? overlapping of what?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "265TyHLryK", "reviewer": "Reviewer_7VEQ", "summary": "In this paper, the authors focus on the problem of Continual Reinforcement Learning, solving multiple tasks that are presented in sequence. In practice, this is a difficult problem because conventional methods often lead to Catastrophic Forgetting. To this end, the authors propose a model-based agent that learns via a Follow-The-Leader approach. The authors justify their approach through mathematical reasoning and proof. They then evaluate their approach on a multi-task setup (Continual Bench) based on Meta-World, demonstrating promising results over the state of the art.\n\n## update after rebuttal\nIn light of the other reviewer's comments, I keep my assessment of accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive review, as well as the suggestions that we should evaluate the method on additional datasets/benchmarks. The main challenge of doing so is that there is no appropriate continual RL test suite to our knowledge. This is also the motivation for us to discuss the importance of unified world dynamics and to propose a new benchmark (Continual Bench).\n\nNevertheless, we will continue working on this area and test on newly developed benchmarks if any. Thank you again for your feedback!", "claims_and_evidence": "Yes. The basic claims made in the paper seem to be justified empirically in the evaluations, at least in the specific dataset used (Continual Bench). The quantitative results are strong and show promise.", "methods_and_evaluation_criteria": "Yes, albeit only one dataset was used (Continual Bench), and this dataset was created by the authors. If there were a way to try other Continual Learning benchmarks, that would strengthen the case of the paper.", "theoretical_claims": "The theoretical claims made in the paper appear to be correct, although it's possible I may have missed details.", "experimental_designs_or_analyses": "Yes. The experimental design is sound.", "relation_to_broader_scientific_literature": "Compared to prior work, the authors propose a new method for Continual Reinforcement Learning in addition to a new dataset (Continual Bench)", "essential_references_not_discussed": "No. Not to my knowledge.", "other_strengths_and_weaknesses": "The authors focus on an important problem in Reinforcement Learning (Multi-task/Continuous Learning) and propose an approach with strong theoretical justification. Furthermore, they present promising results and a new dataset for future research. I believe this paper would be a good contribution to the conference. The paper's argument could be strengthened with evaluation on more datasets/domains; although it may be difficult to find appropriate continual learning domains as the authors note.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1FjZsdvYNa", "reviewer": "Reviewer_ULR9", "summary": "The paper presents a Follow-The-Leader-based online world model, implemented as a composition of a learnable linear layer and a fixed-weight non-linear layer, that is used to solve the continual reinforcement learning problem as a part of model predictive control. The world model has a regret bound of $\\mathcal{O}(\\sqrt{K^2D\\log(T)})$ under certain assumptions. The paper also introduces the $\\textit{Continual Bench}$ benchmark and demonstrates the superiority of their online world model.\n\n## update after rebuttal\n\nI will maintain my score, given that the rebuttal has not significantly changed my opinion of the paper and my rebuttal comment is unaddressed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and questions. Below we respond to the comments and raised questions.\n\n***Claims And Evidence*** \n> The claim that \"image-based environments...\n\nWe agree that the Atari games were solved even back in 2015. However, it typically takes millions to billions of simulated time steps for learning a single task, which amounts to hours to days of wall-clock time even in a distributed training setup [1,2]. Besides, in the online continual RL setting, we could not parallelize the data collection since the online agent only has a single lifetime. This constraint, together with the fact that we need to learn multiple tasks sequentially in continual RL, could make the computation required by image-based experiments quite expensive, and therefore we opt for the state-based environments.\n\n> Line 128 states that continual agent...\n\nAdmittedly, the presented algorithm still requires the task boundary information to reset the planner state. We save the planner state (memory, please see line 245) to mainly reuse the previous planning for better initialization, which is an implementation choice. We could also discard the memory and re-plan every time step, and therefore remove this condition.\n\n\n***Methods And Evaluation Criteria***\nWe agree that 6 tasks are limited. However, we hope to clarify that Continual World's CW20 is actually CW10 repeated twice. Although we only include 6 sequential tasks, we cover different objects like window, peg, button, door and faucet, which have similar difficulty level to CW10's tasks. As a future work, we plan to extend the Continual Bench to include more tasks, making it more useful for evaluating life-long learning online agents.\n\n***Experimental Designs Or Analyses***\n\nWe agree with your points. We set the maximum number of episodes per task as 100 episodes due to the limited computational budget and time constraint. We'd love to conduct experiments for longer episodes once the computational resources are available.\n\n***Other Strengths And Weaknesses***\n\nThanks for raising this important question. We admit that the existing algorithm has difficulty scaling up to high-dimensional problems, as we have discussed in Appendix D. Nevertheless, we hope the online world model learning + planning framework can be useful for developing online agents for more complex problems. We hope to continue this line of research to develop more capable model classes for learning world models online, and design more efficient approximate planning algorithms.\n\n***Other Comments Or Suggestions***\n\nSorry for missing this. $g$ is the goal state and $\\delta$ is a small number to determine whether the current state reaches the goal (typically 0.005 in our experiments). We will add the explanation in our revision.\n\n***Questions For Authors***\n> How applicable is their assumption 1 for theorem 1?...\n\nWe'd like to clarify that the condition $\\\\sup\\_{\\\\mathbf{x}}\\\\|\\\\phi(\\\\mathbf{x})\\\\phi(\\\\mathbf{x})\\^{\\\\top} - \\\\frac{1}{t}\\\\sum\\_{i=1}\\^t\\\\phi(\\\\mathbf{x}\\_i)\\\\phi(\\\\mathbf{x}\\_i)\\^{\\\\top}\\\\|\\_2 \\\\leq \\\\frac{1}{\\\\lambda t}$ does not require that new observations must become more and more similar to features that have already been seen. Instead, it ensures that any single feature vector has negligible impact on the overall covariance once enough data is gathered, enabling robust and consistent estimates. \n\nWe acknowledge that this analysis may be less suited to highly nonstationary environments. However, if the environment evolves smoothly enough that each new “epoch” of data has a bounded impact on the overall covariance after sufficient samples, the same theoretical guarantees should apply.\n\n> How effective is the sparse online world model learning update for discrete...\n\nWe use L2 loss primarily for analytical simplicity. Similar guarantees hold with other losses (e.g., cross-entropy for discrete state space cases) by maintaining the same assumptions. Our L2-based derivation does not rely on the state continuity and is not a fundamental constraint. Even with discrete states, one can also embed them in a continuous feature space (e.g., one-hot or learned embeddings) and apply the same incremental updates.\n\n> How does continual bench change reward exactly?\n\nWhen the environment switches to a new task, the observation and the reward signal of the **new** task will be provided to the agent, and the agent will **not** receive reward for previous tasks. So the agent only has a single stream of experience, and it is expected to learn from it without forgetting how to solve old tasks.\n\n---\n\n[1] Espeholt, Lasse, et al. \"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.\" International conference on machine learning. PMLR, 2018.\n\n[2] Kapturowski, Steven, et al. \"Recurrent experience replay in distributed reinforcement learning.\" International conference on learning representations. 2018.", "claims_and_evidence": "The paper's main claims are supported as its method outperforms agents built on deep world models on their $\\textit{Continual Bench}$ benchmark, and their regret proof is sound given their assumptions.\n\nSection 2 claims that \"image-based environments demand prohibitive computation resources, and the lack of meaningful overlapping prevents us from evaluating the transfer of resources\". This claim isn't necessarily true since Atari environments have been successfully tackled, for offline learning, even back in 2015 with much less compute [1]. It's unclear why online learning should be more challenging w.r.t. computational resources.\n\nA minor criticism is that the line 128 states that continual agent may not know the number of tasks, but the paper's algorithm in the appendix makes use of task boundaries to reset $\\mu$.\n\n[1] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.", "methods_and_evaluation_criteria": "Yes. However, the Continual Bench's number of tasks is only 6, which is considerably smaller than that of Continual World [1], which is 20. It would be ideal to test on a longer number of tasks, especially since the paper's learning algorithm OA can run out of capacity due to being a fixed-size linear model.\n\n[1] Wołczyk, M., Zając, M., Pascanu, R., Kuciński, Ł., & Miłoś, P. (2021). Continual world: A robotic benchmark for continual reinforcement learning. Advances in Neural Information Processing Systems, 34, 28496-28510.", "theoretical_claims": "I have read the proof of Theorem 1 present in the appendix, but not extremely carefully.", "experimental_designs_or_analyses": "Yes. The evaluation of OA and baselines on the Continual Bench benchmark was insightful.\n\nI have one concern, which is that the Continual Bench experiments only train the learning algorithms on 100 episodes per task. To my knowledge, 100 is not a lot of gradient steps for RL algorithms to maximally learn from an environment, and the success rate plot in Figure 5 shows that when task learning is over many models' performance curves don't plateau. While this is not a critical issue for continual learning since the setup does not assume the models can be trained to convergence, repeating the experiment for a larger episode count would yield valuable insights into how different world model performances change when given more chance to learn.", "relation_to_broader_scientific_literature": "The paper adds to the array of recent work that investigates continual reinforcement learning through the lens of model stability and plasticity. To my knowledge, this is the first work that investigates online world model learning applied to continual reinforcement learning.", "essential_references_not_discussed": "To my knowledge, the paper covers all essential references.", "other_strengths_and_weaknesses": "- It is unclear how OA can scale to more complex problems, as it seems both OA and CEM are suited for simple, low-dimensional state and action spaces.", "other_comments_or_suggestions": "- Specify what $g, \\delta$ are in Eq 8.", "questions_for_authors": "- How applicable is their assumption 1 for theorem 1? Doesn't it assume the observations become more and more similar to past observations? In this case, wouldn't the constant regret assumption not apply to highly nonstationary environments?\n\n- How effective is the sparse online world model learning update is for discrete state space? It seems to hinge on L2 loss for derivations to hold.\n\n- How does continual bench change reward exactly? When learning a new task does the model not receive reward for previous tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PuMrwhqfzg", "reviewer": "Reviewer_WQD5", "summary": "In this work the authors propose a new task-unknown continual reinforcement learning setting, in which the agent needs to learn a sequence of tasks without exactly boundary or id. To deal with this new setting, the authors propose a new continual RL method, OA, which introduce a sparse world model by FTL models to overcome the catastrophic forgetting. Besides, the authors propose a new CRL benchmark, the Continual Bench, which is helpful for following works", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review and questions. Below we respond to the comments and raised questions.\n\n***Methods And Evaluation Criteria***\n\n> What does \"task changes\" in Algorithm A.2 mean and how does the agent know this?\n\nAdmittedly, the presented algorithm still requires the task boundary information to reset the planner state. We save the planner state (memory, please see line 245) to mainly reuse the previous planning for better initialization, which is an implementation choice. We could also discard the memory and re-plan every time step, and therefore remove this condition.\n\n> Will the sparsity change after line 12-13 of Algorithm A.2?\n\nYes, the sparsity of the matrix will change after the update. However, each update is guaranteed to be sparse because the sparsity of $\\phi_s(x_t)$ is pre-determined, which also bounds the computation per update.\n\n***Theoretical Claims***\n>whether the chosen $\\lambda$ in the experiments is adequately configured to ensure that Assumptions 1 and 3. \n\nBoth Assumption 1 and 3 depend on selecting a sufficiently large $\\\\lambda$ to control the growth of the feature-space covariance and the norms of sub-block inverses. Specifically, Assumption 1 ensures that the empirical average of feature outer products stays close to those from new data, a condition more easily met when $\\\\lambda$ is larger. Meanwhile, Assumption 3 employs $\\\\lambda$ in the regularized inverse $\\\\bigl(\\\\mathbf{A}\\_{ss}\\^{(t)} + \\\\tfrac{1}{\\\\lambda}\\\\mathbf{I}\\\\bigr)\\^{-1}$ to keep $K$ within reasonable bounds. In practice, an appropriate $\\\\lambda$ helps maintain both assumptions throughout experiments by mitigating outlier effects in the feature space and ensuring that inverse sub-blocks remain well-conditioned. However, because the ideal value of $\\\\lambda$ can vary by environment or dataset, some tuning is typically required.\n\n\n***Experimental Designs Or Analyses***\n> why the horizon $H$ is set to 15?\n\nThis hyperparameter would trade off the planning accuracy and the computational requirement. Setting $H$ too small will get efficient planning but the planned action can be short-sighted; a longer $H$ could incur larger planning cost, but the policy can better optimize long-term rewards. However, since we plan with a learned world model, the model error can compound along the planning horizon, so there is no monotonic benefit from using longer horizons. $H=15$ is an empirical value which is obtained from our pilot trials.\n\n\n***Questions For Authors***\n\nThank you for pointing out the mistake. We missed an extra term in our proof. Below is the corrected derivation:\n\\\\begin{align}\n    {} &\n    \\\\mathrm{Tr}(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\mathbf{W}\\^{(t+1)}\\\\mathbf{W}\\^{(t+1)\\\\top}) \n    -\n    \\\\mathrm{Tr}(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\mathbf{W}\\^{(t)}\\\\mathbf{W}\\^{(t)\\\\top}) \n    \\\\\\\\\n    = {} &\n    \\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\left(\\\\mathbf{W}\\^{(t+1)} - \\\\mathbf{W}\\^{(t)}\\\\right)\\\\left(\\\\mathbf{W}\\^{(t+1)} - \\\\mathbf{W}\\^{(t)}\\\\right)\\^\\\\top\\\\right) \\\\\\\\\n    & +\n    2\\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\left(\\\\mathbf{W}\\^{(t+1)} - \\\\mathbf{W}\\^{(t)}\\\\right)\\\\mathbf{W}\\^{(t)\\\\top}\\\\right)\n\\\\end{align}\n\nNotice that by the definition of $\\\\Delta\\_t$, we have\n\\\\begin{align}\n    {} &\n    2\\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\left(\\\\mathbf{W}\\^{(t+1)} - \\\\mathbf{W}\\^{(t)}\\\\right)\\\\mathbf{W}\\^{(t)\\\\top}\\\\right)\n    \\\\\\\\\n    \\\\leq {} &\n    2\\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\Delta\\_t\\\\mathbf{W}\\^{(t)\\\\top}\\\\right)\n    \\\\\\\\\n    = {} &\n    2\\\\mathrm{Tr}\\\\left(\\\\underbrace{\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\left(\\\\sum\\_{i=1}\\^{t}\\\\phi(\\\\mathbf{x}\\_i)\\\\phi(\\\\mathbf{x}\\_i)\\^\\\\top + \\\\frac{1}{\\\\lambda}\\\\mathbf{I}\\\\right)\\^{-1}}\\_{\\\\preceq \\\\mathbf{I}}\\\\phi(\\\\mathbf{x}\\_t)\\\\mathbf{y}\\_t\\^\\\\top\\\\mathbf{W}\\^{(t)\\\\top}\\\\right)\n    \\\\\\\\\n    \\\\leq {} &\n    2\\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\mathbf{y}\\_t\\^\\\\top\\\\mathbf{W}\\^{(t)\\\\top}\\\\right). \n\\\\end{align}\n\nSo the correct bound should be $f\\_t(\\\\mathbf{W}\\^{(t+1)}) - f\\_t(\\\\mathbf{W}\\^{(t)}) \\\\leq \\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\phi(\\\\mathbf{x}\\_t)\\^\\\\top\\\\Delta\\_t\\\\Delta\\_t\\^\\\\top\\\\right) + \\\\frac{4}{t}\\\\mathrm{Tr}\\\\left(\\\\phi(\\\\mathbf{x}\\_t)\\\\mathbf{y}\\_t\\^\\\\top\\\\mathbf{W}\\^{(t)\\\\top}\\\\right)$, which differs from the original bound where the coefficient of the second term is $\\\\frac{2}{t}$. \nWe will revise the proof in our manuscript.", "claims_and_evidence": "I think the claims are clear and convincing evidence.", "methods_and_evaluation_criteria": "I have two questions about the algorithm in Section A.2.\n\nFirst, in the line 2, the loop start with a condition: task changes. How does the agent know when the task changes? What exactly this condition sentence means?\n\nSecond, in line 12 to line 13, the $A_{ss}^{\\left(t\\right)}$ is calculated with $A_{ss}^{\\left(t+1\\right)}$ and $\\phi_s$. Will the sparsity of $A_{ss}$ and $B_{s}$ change after these lines? How to maintain the sparsity?", "theoretical_claims": "I am interested in understanding the extent to which Assumption 1 and Assumption 3 are sustained throughout the experiments. Specifically, I would like to inquire whether the configuration of $\\Lambda$ in the experimental setup is sufficient to satisfy the conditions stipulated in Assumption 3.", "experimental_designs_or_analyses": "I would like to know that why the horizon $H$ is set to 15? As I know, this is a large number for the model-based RL, but not an enough number when you want to generate a whole trajectories.", "relation_to_broader_scientific_literature": "After Wołczyk, this work propose another kind of vision of the CRL, which will helpful for following works. I think this work tells us that the CRL is a very different question with the continual classification problem.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "I’m sorry but I do not understand the proof in Lemma 3 Proof line 2 to 3. Why\n\n$\\text{Tr}\\left(\\phi\\left(\\mathbf{x}_t\\right)\\phi\\left(\\mathbf{x}_t\\right)^\\top\\mathbf{W}^{\\left(t+1\\right)}\\mathbf{W}^{\\left(t+1\\right)\\top}\\right)-\\text{Tr}\\left(\\phi\\left(\\mathbf{x}_t\\right)\\phi\\left(\\mathbf{x}_t\\right)^\\top\\mathbf{W}^{\\left(t\\right)}\\mathbf{W}^{\\left(t\\right)\\top}\\right)$\n\nless or equal than\n\n$\\text{Tr}\\left(\\phi\\left(\\mathbf{x}_t\\right)\\phi\\left(\\mathbf{x}_t\\right)^\\top\\left(\\mathbf{W}^{\\left(t+1\\right)}-\\mathbf{W}^{\\left(t\\right)}\\right)\\left(\\mathbf{W}^{\\left(t+1\\right)}-\\mathbf{W}^{\\left(t\\right)}\\right)^\\top\\right)$ ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an Online Agent (OA) for Continual Reinforcement Learning (CRL), leveraging Follow-The-Leader (FTL) world models and Model Predictive Control (MPC) to mitigate catastrophic forgetting. It also introduces Continual Bench, a new benchmark designed to evaluate both forgetting and transfer across tasks.

All reviewers agree that the paper tackles an important problem in CRL and is grounded in strong theoretical foundations. The OA approach is clearly articulated, with a regret bound derived under assumptions generally considered reasonable. The proposed Continual Bench is a valuable contribution with improved consistency over prior environments by maintaining a fixed state space. Empirical results are promising and support the proposed methodology.

There are some concerns. The benchmark includes only six tasks, all defined by the authors, which is relatively small compared to prior work such as *Continual World* (Reviewers 7VEQ and ULR9). This raises questions about the scalability of the approach. Additionally, OA’s reliance on shallow models may limit performance in more complex or high-dimensional settings (Reviewer ULR9). Finally, some assumptions made in the theoretical analysis could be better reflected or motivated in the experimental setup (Reviewer WQD5).

Overall, this paper presents a novel, theoretically sound approach to CRL, supported by a meaningful empirical evaluation. Despite some concerns about scalability and generalization, the contribution is considered valuable and relevant to the ICML community.
Decision: Accept (spotlight poster)