Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning
Authors: Adrian Remonda
Cole Corbitt Terrell
Eduardo E. Veas
Marc Masana
Venue: Accepted by TMLR ()

Abstract: Model-based reinforcement learning uses a learned dynamics model to imagine actions and select those with the best expected outcomes. An experience replay buffer collects the outcomes of all actions executed in the environment, which is then used to iteratively train the dynamics model. However, as the complexity and scale of tasks increase, training times and memory requirements can grow drastically without necessarily retaining useful experiences. Continual learning proposes a more realistic scenario where tasks are learned in sequence, and the replay buffer can help mitigate catastrophic forgetting. However, it is not realistic to expect the buffer to infinitely grow as the sequence advances. Furthermore, storing every single experience executed in the environment does not necessarily provide a more accurate model. We argue that the replay buffer needs to have the minimal necessary size to retain relevant experiences that cover both common and rare states. Therefore, we propose using an uncertainty-based replay buffer filtering to enable an effective implementation of continual learning agents using model-based reinforcement learning. We show that the combination of the proposed strategies leads to reduced training times, smaller replay buffer size, and less catastrophic forgetting, all while maintaining performance.

Published in Transactions on Machine Learning Research (02/2025)
Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning
Adrian Remonda
oremonda@@nou--cntee..
Graz University of Technology and Know-Center GmbH
Cole Terrell
clervel@@hnoo--eeterr.dt
Graz University of Technology and Know-Center GmbH
Eduardo Veas
eweas@know-center.tt
Graz University of Technology and CRow-CCetee GmbH
Marc Masana
marc.masana@ieg tugroz.at at
Graz University of Technology and SAL Dependable Embedded Systems
Reviewed on OpenReview: http::///oooaeeeeeeeeeeeeertetta
Abstract
Model-based reinforcement learning learned dynamics model to imagine actions and select those with the best expected outcomes. An experience replay buffer collects the out- comes of all actions executed in the environment, which is then used to iteratively train the dynamics model. However, as the complexity and scale of tasks increase, training times and memory requirements can grow drastically without necessarily retaining useful experiences. Continual learning proposes a more realistic scenario where tasks are learned in sequence, and the replay buffer can help mitigate catastrophic forgetting. However, it is not realis- tic to expect the buffer to infinitely grow as the sequence advances. Furthermore, storing every single experience executed in the environment. does not necessarily provide more accurate model. We argue that the replay buffer needs to have the minimal necessary size to retain relevant experiences that cover both common and rare states. Therefore, we pro- pose using an uncertainty-baat based replay buffer filtering to enable an effective implementation of continual learning agents using model-basod reinforcement learning. We show that the combination of the proposed strategies leads reduced training times, smaller replay buffer size, and less capastrophic forgetting, all while maintaining performance.
1 Introduction
Model-Based Reinforcement Learning (MBRL) has gained popularity due to its tendency to have lower sample complexity compared to model-free algorithms (Lillicrap et al., 2015; Haarnoja et al., 2018). MBRL agents function by building a model of the environment in order to predict trajectories of future states based on imagined actions (Hafner et al., 2019). An MBRL agent maintains an extensive history of its experiences, actions response them, and their resulting reward in an experience replay buffer. This stored information is used to train a dynamics model that ileratively predicts the outcomes of the imagined actions into trajectory of future states. At each time step, the agent executes only the first action in the trajectory and then the model re-imagines new trajectory given that result (Nagabandi et al., 2018; Rao, 2010; Chua et al., 2018)
By leveraging internal models to anticipate and plan, MBRL fosters dynamic adaptation to changing en- vironments (Nagabandi al., 2019). This adaptability makes MBRL a promising approach for continual learning scenarios. For many real-world applications, tasks are presented in sequence of arbitrary length, accruing repetitive experiences which need to be learned efficiently. In most cases when the sequence is long,

---

Published in Transactions on Machine Learning Research (02/2025)
Task
Task 2
Task
MERL
FIFO
Train f given
RS
MPC
UBER
D DU (an
TRAJECTORY PLANNER
f:SxA-5
(1 Ois Phy (1-1:
METHODS
ENVIRONMENT
Figure 1: Workflow. The blue boxes showcase single-task MBRL setting (Sec. 3, 4), and the task sequence describes continual reinforcement learning setting (Sec. 6, 7). Highlighted in red, our method (UBER, Sec. 5) filters data added to the buffer.
it becomes unfeasible inefficient to store all experiences and train the model on all the multiple tasks at once. Therefore, continual learning (CL) proposes to learn the tasks one at a time they are presented to the agent (Mnih et al., 2013; Normandin et al., 2021). However, when learning single task without any further help, performance in previous tasks docreases, which is known as catastrophic forgetting (McCloskey & Cohen, 1989; Kirkpatrick et al., 2017). A way to mitigate catastrophic lorgetting is the usage of an experience replay buffer, which retains experiences from previous tasks used to enhance the training of the model and avoid forgetting previously learned tasks (Masana et al., 2022; Normandin al., 2021).
Furthermore, extending training becomes concern when accumulating similar repetitive experiences in unlimited replay buffer. Schaul et al. (2016) showed that transitions in replay buffer can be more or less surprising, redundant, or task-relevant, potentially leading to buffer inundated with redundant information which consequently under-represents other important states. Conversely, a buffer that is too small will be unlikely retain enough relevant experiences, leading minimal or no impact on the training of the model (Iscle Cosgun, 2018). Idcally, the size of the buffer should be the minimal amount required to capture sufficient detail for all relevant states to be learned (Zhang & Sutton, 2017). Note that, in the continual learning setting, knowing a priori all relevant states is unfeasible without extensive exploration having access to future knowledge.
We argue that these problems can be subverted by employing strategy that avoids relaining experiences in the buffer which the model has already sufficiently mastered. Therefore, in this work we propose a method, depicted in Figure 1, that determines what gets admitted into the replay buffer the first place, as mcasured by the uncertainty of the model when learning them. Only those experiences which provide value to the model added, thus also limiting the size of the buffer to the minimal amount. needed to avoid forgetting previously learned tasks. We further evaluate our method in proposed continual learning setting that processes series of tasks sequentially and aims to mitigate forgetting. Therefore, our contributions are:
Development of a method for estimating model uncortainty when predicting imagined trajectorics.
A novel approach to retain relevant experiences in the replay buffer.
Introduction of two racing-related environments with multiple circuits for sequential task learning.
Evaluation of generalization and catastrophic forgetting in continual learning setting.
2

---

Published in Transactions on Machine Learning Research (02/2025)
The adoption of these strategies enables the MBRL model to self-manage the buffer size, including under the CL. setting. We consider these strategies to be critical as a starting point toward the implementation of effective and stable continual learning agents.
The remaining of this mamuscript is divided as follows. Details about related work are presented in Section 2. A After, this manuscript is divided into two main parts: single task MBRL and our proposed method to filter the experiences retained in the replay buffer, and the expansion towards continual learning settings. First, Section 3 provides background on MBRL methods, followed by a motivation experiment which highlights the importance of the replay buffer size in Section 4, and our proposed uncertainty-basee Second, we expand MBRL adaptation towards CL settings in Section 6, and provide an experimental analysis in Section 7. Finally, we provide discussion of the limitations and conclusions in Section 8.
filtering in Section 5,
2 Related Work
Model-based Reinforcement Learning. MBRL has been applied in real-world control tasks, such as robotics (Zhu et al., 2020). Compared to model-free approaches, MBRL tends to be more sample- efficient (Deisenroth et al., 2013). MBRL can be grouped into four main categories (Zhu et al., 2020). Dyma-style algorithms optimize policies using samples from a learned world model (Sutton, 1990). Model- augmented value exponsion methods, such as MVE (Oh et al., 2017), use model- based rollouts to enhance targets for modcl-free temporal difference updates. Analytic value gradients can be used when a differentiable world model is available, which adjust the policy through gradients (e.g. using Gaussian processes for the dynamics model (Deisenroth & Rasmussen, 2011)). Model predictive control (MPC) and shooting methods use planning to select actions, but have the drawback of being computationally intensive (Rao, 2010; Chua et al., 2018). The present work belongs to the last group.
Neural networks efficiently reduce sample complexity for problems with high- digensimmns non-linear dynam- ics (Nagabandi et al., 2018). MBRL approaches need to induce potential actions which will be evaluated with dynamics model to choose those with best reward. Random shooting MPC methods artificially generate large number of actions and can be used to select optimal actions (Camacho et al., 2004). Neu- ral networks are a suitable alternative to families of equations used to model the environment dynamics in MBRL (Williams et al., 2017), although they tend to make overconfident incorrect predictions. Thus, quantifying predictive uncertainty becomes crucial. Ensembles of probabilistic networks proved a good alter- native to their Bayesian counterparts in determining predictive uncertainty (Lakshminarayanan et al., 2016). Furthermore, an extensive analysis about the types of model that better estimate uncertainty in the MBRL setting favored ensembles of probabilistic networks (Chua et al., 2018). The authors identified two types of uncertainty: aleatoric (inherent to the process) and epistemic (resulting from datasets with too few data points). Combining uncertainty aware probabilistic ensembles in the trajectory sampling of the MPC with a cross enpy controller demonstrated asymplotic performance comparable to SAC but with sample elficient convergence. The MPC, however, is still computationally expensive (Chua et al., 2018; Zhu et al., 2020). Quantifying predictive uncertainty provides a mcasure of confidence in an imagined trajectory. Remonda et al. (2021) utilized this concept to avoid unnecessary replanning by relying on sequences of actions the model is confident in, thereby reducing computations. Similarly, our approach aims to determine reliable predictions from the dynamics model in relation to the imagined actions, but as a foundation for managing the growth of the experience replay buffer.
Use of Experience Replay in MBRL. While an uncertainty-aware dynamics model helps to mitigate the risks of prediction overconfidence, other challenges remain, such as the shifting of the state distribution as the model trains. Experience replay was introduced by Lin (1992), and has been further improved upon. One variation of this is prioritized experience replay (Schaul et al,, 2016), which aims to make learning more efficient by prioritizing transitions that are more relevant for lcarning, rather than sampling transitions uniformly from the replay buffer, as typically done in reinforcement learning. This method improves how the model samples experiences from the already- filled replay buffer, but does not address how the replay buffer is filled in the first place. Additionally, neither work addresses the importance of the size of the replay buffer as a hyperparameter (Zhang & Sutton, 2017). Our approach limits the replay buffer by only adding experiences that should improve future prediction capacity and keeps training time bounded to a minimum.
3

---

Published in Transactions on Machine Learning Research (02/2025)
Task- Agnosti Continual Learning. We assume that agents can train a sequence of relatively similar tasks of arbitrary length. Specifically, we aim for continuous task-agnostic reinforcement learning (Nor- mandin et al., 2021), where task boundaries are not observed and transitions may occur gradually (Zeno et al., 2021). However, we specifically focus on the incremental case where the contimual learning system learns each lask from a sequence without access to previous lasks before receiving a new one. Xie & Finn (2021) develop a method that exploits data collected from previous tasks using importance sampling, al- though with the requirement of the agent knowing when tasks change. In our case, the task identifier never available during training and the model has no explicit information about task transitions. In such context, an incremental learner can be seen as an autonomous agent learning over an endless stream of tasks, where the agent has to: i) continually adapt in a non-stationary environment, ii) retain memories which are useful, iii) manage compute and memory resources over a long period of time (Khetarpal et al., 2020; Thrun, 1994). Our proposed approach satisfies these three requirements. Ammar et al. (2015) focus on agents that acquire knowledge incrementally by learning multiple tasks consccutively over their lifetime. Their approach rapidly learns high performance safe control policies based on previously learned knowledge and safety constraints on each task, accumulating knowledge over multiple consecutive tasks to optimize overall performance. Knowl- edge is shared via latent basis that captures reusable components of the learned policies. The latent basis is then updated with newly acquired knowledge. This results in an accelerated learning of new task and an improvement in the performance of existing models without retraining on their respective tasks.
Several works propose task-agnostic methods that do not require task information to perform continual reinforcement learning. They propose strategies to address the challenge of deciding which experiences to discard from a filled experience replay, departure from the conventional First-In-First-Out (FIFO) approach commonly adopted. Isole & Cosgun (2018) augmented the standard FIFO buffer by selectively storing experiences over four retaining strategies: favoring surprise, prioritizing high rewards, aligning with the global training distribution (reservoir sampling (Vitter, 1985)), and ensuring broad coverage of the state space. They found that reservoir sampling effectively prevents catastrophic forgetting. Rolnick et al. (2019) proposed the CLEAR method. CLEAR, utilizes experience replay buffers to prevent forgetting. The hod involves actor-critic training, integrating both new and replayed experiences. This approach adopts distributed training based on the Importance Weighted Actor-Learner Architecture (Espeholt et al., 2018). A single learning network receives both new and replayed experiences from multiple acting networks, which have their weights updated asynchronously to align with the learning network. Training is done using the V-Trace off-policy learning algorithm, applying truncated importance weights to adjust for off- off-policy distribution shifts. Kessler et al. (2023) demonstrated that MBRL is appropriate for the continual learning setting, evaluating strategies both for adding to and sampling from the buffer. Their findings indicate that reservoir sampling facilitates a balanced representation of experiences helps to mitigate forgetting. However, these works address the problem of sampling and discarding data from replay buffer. In contrast, our focus is determining what gets admitted into the replay buffer in the first place, which does not require setting replay buffer size in advance.
the replay buffer, which in turn
3
Model Based Reinforcement Learning
3.1 Preliminaries
We consider unknown stochastic dynamical systems which we formulate as a finite-horizon Markov Decision Process (Bellman, 1957). At each time t, the agent is at state 8, S, executes action a; E A, and receives from the environment both a reward r, = r(se, 0t) and new state 81+1 according to a transition function f 5:5x S A A + S. When training reinforcement learning policy, the goal is to maximize the accumulated reward obtained from the environment. This can be adapted to the infinite horizon setting by maximizing the sum of discounted rewards R = (i-))) az), where 7 E 10.
Instead, the random shooting MPC family of MBRL algorithms artificially generates a huge amount of potential future actions, given current state 8t, to select the optimal action. MBRL attempls to learn a discrete time dynamics model f (s,, a:) to predict the future state ŠttA, of executing action o, state st. To reach a state into the future, the dynamics model iteratively evaluates sequences of actions, au+H (at, (at.-.., H-1) over a longer horizon H, to maximize their discounted reward E Ei+H-1 ( (0000)(8), az). These
4

---

Published in Transactions on Machine Learning Research (02/2025)
sequences of actions with predicted outcomes are called imagined trajectories. The dynamics model f is inaccurate representation of the transition function f and the future is only partially observable. So, the controller executes only a single action in the trajectory before solving the optimization again with the updated state 81+1- The process is formalized in Algorithm 1. The dynamics model fo is learned with data Denv: collected on the fl. With Se, the simulator starts and the controller is called to plan the best trajectory resulting H- Only the first action of the trajectory af executed in the environment and the rest is discarded. This is repeated for Th number of steps. The data collected from the environment is added to Dene and fo is trained further. The process repeats for N iterations. Note that generating imagined trajectories requires subsequent inference of the dynamics model to chain predicted future states 8t4n with future actions up to the task horizon, making it only partially parallelizable.
Dynamics model. We use a probabilistic model to model a probability distribution of next state given current state and an action. To be specific, we use regression model realized using a neural net similar to Lakshminarayanan et al. (2016) and Chua et al. (2018). The last layer of the model outputs parameters of Gaussian distribution that models the aleatoric uncertainty (the uncertainty due to the randomness of the environment). Its parameters are learned together with the parameters of the neural network. To model the epistemic uncertainty (the uncertainty of the dynamics model due to gencralization errors), we ensembles with bagging where the members of the ensemble are identical and only differ in the initial weight values. Each element of the ensemble has as input the current state S; and action d and is trained to predict the difference between and 8t41, instead of directly predicting the next step. Thus the learning objective for the dynamics model becomes, As = 81+1 - 8t. fo outputs the probability distribution of the future state Ps(++1) from which we sample the future step and its confidence 3,3, fo(s, [a]). Where the confidence s, captures both, epistemic and aleatoric uncertainly.
Algorithm 1 MBRL
1: Tnit D with one iteration of random controller
2:
for Iteration i = 1 To N do
3: Train f given D
4: for t= 0 To Th do
5:
gommppttOppiimalTnneettttrte f'
6:
Execute aj from optimal actions oL.4
7:
+ DU (st, (1.+1) // Record autcome
8: end for
9: end for
Trajectory Generation. Each ensemble element outputs the parameters of a normal distribution. To generale trajectories, P particles are created from the current state, sf St: which are then propa- gated by: 1441 ~ fo(s),,)) a;), using a particular boot- strap element b E (I,..,,,, B). Chua et al. (2018) ex- perimented with diverse methods to propagate par- ticles through the ensemble. The 7'S, method de- livered the best results. It refers to particles never changing the initial bootstrap element. Doing so, re- sults in having both uncertainties separated at the end of the trajectory. Specifically, aleatoric state variance is the average variance of particles of same boot- strap, whilst epistemic state variance is the variance of the average of particles of same bootstrap indexes. We use also TS.
A Algorithm 2 Get Optimal Trajectory Planning
1: Input: current state Sintey dynamics model f
2: Initialize P particles, of, with the initial state, Ainit
3: for 0,44 ~ C.M(), 1 To CEMSamples do
4: Propagate state particles s. using TS and fKDD,antH)) at
1+11
5:
Evaluate actions as
Σ Σp= 15
T=t 6: Update CFM(() distribution
7: end for
8: return d/14
Planning. To select the best course of action leading to 8H, MBRL generates a large number of trajectories K and evaluates them in terms of reward. To find the actions that maximize reward, we used the cross entropy method (CEM) (Botev et al., 2013), an algorithm for solving optimization problems based on cross- entropy minimization. CEM gradually changes the sampling distribution of the random search so that the rare-event is more likely to occur and estimates sequence of sampling distributions that converges to distribution with probability mass concentrated in region of near-optimal solutions. Algorithm 2 shows the use of CEM to get the optimal sequence of actions d
Reliable imagination. Random shooting MBRL methods are computationally intensive, as they require evaluating a sequence of actions with the dynamics model for each trajectory generated. BICHŌ (Remonda et al., 2021) is method that improves the runtime and training time of MBRL by reducing the computational
5

---

Published in Transactions on Machine Learning Research (02/2025)
cost. It does this by continuing to act an imagined trajectory until it can no longer be trusted, and then replans. The BICHO method employs a probabilistic approach to determining the reliability of a trajectory and decide when to replan. This allows for efficient exploitation of trusted trajectories while reducing computational costs. Our approach exploits the BICHO mochanism to evaluate the reliability of trajectories in order filter out unnecessary additions to the replay buffer.
3.2 Baseline methods
We describe and compare the relevant baseline methods used in this work, highlighting key properties and their respective implementations (see also Table 1).
PETs: Standard shooting MBRL which uses a replay buffer of infinite size. We use the imple- mentation from the authors of PETs (Chua et al., 2018). It represents an upper bound to assess performance when retaining everything in the buffer.
Scratch: We also include baseline in which the model is trained from scratch for each individual lask of the training lasks set. This means that once the model is trained on particular task, we evaluate its performance on separate test task set. Prior to transitioning to new task, we discard both the model parameters and the experiences present in the replay buffer associated with that task. This baseline serves the purpose of demonstrating the model's performance without the benefit of additional experiences obtained from solving other tasks. For this baseline, we utilize an infinite buffer allocated for each task. As a result, we anticipate that this approach will yield the lowest performance scores in terms of overall performance, generalization capability, and forgetting.
To manage the continuously growing experience replay buffer, we evaluate straight- forward first-in, first-oul (FIFO) strategy. Once the buffer size is fixed, new experiences are added until the buffer is full. Subsequently, for each new experience added, the oldest experience in the buffer removed to accommodate the new one. We also evaluate the impact of different buffer sizes.
VanillaFIFO:
Adaptative FIFO (A-FIFO): Since VanillaFIFO retains only the latest experiences, it has a bias towards more recent tasks. Following inspiration from other online and offline incremental learning settings (Prabhu et al., 2020; Masana et al., 2022; Mai et al., 2022), we propose to reduce adaptive version of the FIFO (adaptive-FIFO) which always retains the latest experiences from previous tasks equally distributed among all the lasks seen. Given a reply bulfer of maximum experiences, at task we store a maximum of E/T experiences for each task, following the FIFO strategy. When a new task is addod, the necessary oldest memorics from each task are removed to make room for the new lask experiences. We consider that these baselines are more representative stralegy for an continual learning setting, although it does not achieve all the same properties as our proposed approach. While Adaptive-F1FO offers a more balanced strategy for continual learning setting, it has two limitations when comparing to our method. Task number awareness: i. The method requires prior knowledge of the total number of tasks, which is not always feasible in real-world, dynamic scenarios. ii Task Transition Awareness: Adaptive-FIP requires an indication or marker of when a task transition occurs. This awareness is often unrealistic, as transitions in CL can be subtle or even seamless.
Reservoir sampling (RS) (Isele & Cosgun, 2018; Vitter, 1985) augments VanillFIFO by selec- tively storing experiences aligning with the global training distribution to ensure coverage of the state space. Isele & Cosgun (2018) found that RS effectively prevents catastrophic forgetting.
3.3 Environments
We evaluate the methods in the CartPole and Reacher environment provided by the MuJoCo (Todorov et al., 2012) physics engine. Additionally, we introduce our own proposed environments related racing, including Masspoint and Non-linear Bicycle model. The choice of CartPole and Reacher environments is based on their established use in RL research, allowing for meaningful comparisons with existing approaches. By
6

---

Published in Transactions on Machine Learning Research (02/2025)
introducing racing-related environments, we further assess the methods' effectiveness in more complex scenarios, which are well-suited for continual RL. (see details in the Appendix).
CartPole (CP). Inverted pendulum problem, which involves balancing a pole on cart. It has movable cart that travels along a frictionless track. On top of the car a pole with one end attached to the cart, is standing upright. The goal to keep the pole upright for as long as possible by moving the cart left or right. The agent receives reward of for every time step the pole remains upright.
Reacher (RE). A robotic arm, with Degrees of Freedom, aiming to reach target position in space. Given the multiple joints and their rotations. The reward function aims to minimize the distance between the end effector and the target position.
Masspoint (MP). An extended version of the Masspoint environment proposed by Thananjeyan et al. (2020). It navigation task in which a point mass moves toward specified goal. We modified the agent's goal so that it must move as quickly as possible without deviating from a given path. The complexity of each lask is determined by the geometry of the path. The reward is set to maximize speed while minimizing deviation from the path.
Non-linear Bicycle Model (Bike). This model captures vehicle dynamics with greater fidelity and features higher action and observation dimensions compared to MassPoint. It considers aerodynamics, tire dynamics, and rolling resistance. Additionally, we have integrated track boundaries to enhance the realism and challenge the simulation. The control variables include steering and combined throttle and brake system. The reward system is designed maximize speed, and the episode ends if the car goes off track.
4 Motivation experiment
Eslocd Rewar
20c-01
Stee
1.5x-1
100
FPOOO US2
1.0e0)1
500000
30
0.0 000
Episode
Eptscce
Episddo
Epiodd Rovord
Run Wall Time
FETP
LSGR
Epeazda
Epieade
Epassde
Eprodd Rovord
1982
Run Venl Time
1003
112
2.00000
13
25
10
15
15
Eplacda
Episoda
Episode
Figure 2: Performance of algorithms in (top to bottom) Cartpole, Reacher, and Masspoint. From left to right column: episode reward, time per episode (s), cumulative number of observations stored in the replay buffer. The x-axis is the number of training episodes.
In this section, we present motivational experiment designed to analyze the growth of the replay buffer and training time when utilizing typical MBRL method with different buffer sizes. For this experiment, we specifically employ PETs (Chua et al., 2018) on single task. This setup intentionally mimics controlled environment with abundant and redundant data, demonstrating how standard MBRL algorithms handle such redundancies. We aim to highlight why standard MBRL technique falls short for continual learning. We compare an unbounded buffer size and a FIFO with different buffer sizes. We evaluate performance based on per-episode reward, per-episode wall time, and replay buffer size.
Results the CP. RE, and Masspoint environments are depicted in Figure top, middle, and bottom, re- spectively. Figure 2-1eft shows comparable reward per episode, while FIFO variants exhibit a cyclic reward, which we attribute to forgetting and regaining experiences. Results for reward in CP and RE are consistent.
7

---

Published in Transactions on Machine Learning Research (02/2025)
FIFO rewards for Masspoint are more stable. The training time per episode (Figure 2-middle) for Vanil- IaFIFO variants stabilizes after reaching the maximum buffer size. The wall time for PETs exhibits linear growth and takes substantially longer than FIFO to complete an episode (i.e., it takes longer to update the model as the replay buffer grows linearly).
These results shed light on the stability issues from an under-tuned replay buffer within a MBRL context, as well as the increasing training time when employing an excessively large replay buffer. These findings underscore the challenges that these issues pose to the practical applicability of MBRL in CL scenarios. Namely, the lack of memory management consequently hinders compute management and impairs learning as the agent cannot determine which experiences are uscful.
Adding experiences to the replay buffer. We posit that it would be preferable to retain only those experiences that could not be adequately anticipated by the model during each episode in the environment. Esscntially, we would like to only add to the replay buffer obscrvations for which the model issued poor prediction, as retaining redundant experiences will fill the buffer with unnecessary data and reduce its capacity to store meaningful, hard-to-predic observations. On the contrary, we would like to avoid filling the replay buffer or updating the model on observations that the model is good at predicting. We contend that these two elements will lead eventually to a balanced replay buffer, which will contain only relevant observations and will contribute to the objective of incremental learning. In the following section, we further describe our proposed approach.
5 UBER: Uncertainty-Baser Experience Replay
Continual reinforcement learning requires the MBRL agent to adapt in a non-stationary environment, re- taining memorics that are useful whilst avoiding catastrophic forgetting, and effectively manage compute and memory resources over long period of time (Khetarpal et al., 2020). We address these issues with our proposed method, UBER
We propose Algorithm 3 for selecting which experiences to include in the replay buffer. In the event of an unreliable future, the algorithm replans the future trajectory and adds the current observation to the buffer. By doing so, unnecessary replanning and additions to the buffer are avoided, reducing computation time and the size of the buffer. The optimal actions C are calculated using the GetOptimalT rajectory function (as outlined in Algorithm 2) based on the current state of the environment and the model f. The future trajectory and its uncertainty, P are then determined by using o s and 8, with f. The flag unreliable. odel is set to true when the algorithm determines that the imagined trajectory is not trustworthy. Depending on its value, further calculations and additions to the replay buffer may be avoided, roducing computation time and the size of the buffer. If unreliable is False, the next predicted action is execuled in the environment. Subsequent. actions from 4e4t are executed until the model is unreliable or the environment reaches the maximum number of steps, Th. The process is repeated for the maximum number of iterations allowed per task. Hereby, the buffer stores only experiences for which the model could not predict (imagine) its outcome.
Uncertainty estimation. To determine when the predictions given by the dynamics model are still trust- worthy, we utilize the BICHO method introduced by Remonda et al. (2021). BICHO evaluates the validity of the current trajectory by making probability estimates of the projected future reward, ensuring that it does not deviate significantly from the imagined future reward P and that the uncertainty in the model remains low. BICHO is built under the assumption that if certain parts of the trajectory do not vary, their projected reward will align with the model's imagination with certain level of confidence. To achieve this, after calculating trajectory, the distribution of rewards pž is calculated for H steps in the future. At each step of the environment, whether replanning step was skipped or not, a new trajectory pt, of H steps is projected, starting from the state 8, provided by the environment and using actions d'+, from the imagined trajectory. We use a distance metric to find how much these two distributions change after each time step in the environment. If the change is > 8 then unreliableModel is Truc. We can control how many steps ahead we would like to compare the two distributions. The comparison done for LA steps (< H), which is a hyper parameter to tune. If the two distributions differ significantly, then the trajectory is unreliable.
8

---

Published in Transactions on Machine Learning Research (02/2025)
Algorithm 3 UBER
Initialize dynamics model f parameters; Initialize replay buffer D with an iteration of random controller
2: wndeliableModel True and trainModel- = Folse
to N do 4: if Mrdel then Train f given D
3: for Iteration
5: for t to Tи do
if unrelkable.Mode then
7:
Get d from ComputeOpienalTraaeeta (st:
8:
Get Pr(++1:+4) given (s.. 0la48) // Use f to predict rewards ahead ==
9:
10:
else
4+ end if
12:
Get first action ar from available optimal actions ai4t
14:
Execute in the enviromment first action di from remaining optimal actions d to obtain s041 and re-1 Discard first action and keep rest dj = 444144 // Uncertainty Estimation Scoe = Compute umcertaintySoome
16:
17:
18:
if uncertaintyScore:> 3 then unreliableModel = TRUE else unreliableModel False
19:
if unreliableM odel then
20:
Record outcome: D t D (s,,01,
21:
// Updates on novel information
if new data in D new dato threshold (ength(D) then trainModel. = True
23:
end if 25: end if
end for
27: end for
Algorithm 4 Get Uncertainty Score
Input: i, P7(441248) and LA
1: Get P from ComputeTrajcctoryyPror (st, f.
L == min(H, LA i, MPD) // Calculate number steps ahead to consider
3: UncertaintySeove= W Wasserstein aagpee/eeeaaaaaoeoeeeeeeaaaaeeee
4: return UncertaintyScore
That is, if the projected reward differs from the imagined one the outcome the actions uncertain. We empirically evaluate different metrics and found that the Wasserstein distance (Bellemare al., 2017) is more stable and performs better than KT.-Divergence.
Maximum prediction distance. Even for model that has converged, accurately predicting trajectories of great length is infeasible. Recalculations at the end of trajectories are inevitable and do not necessarily indicate the presence of new information, but rather the limitations of the successful model in a complex environment. Therefore, we exclude such recalculations from the buffer. The mazimum prediction distance (MPD) sets a cutoff point for a trajectory and regulates the strictness of the filtering mechanism.
Updates on novel information. Over-training the dynamics model leads to instabilities due overfitting, This problem is exacerbated when the replay buffer contains just the minimum essential data. If we only filter the replay bulfer, continuously updating the parameters of the dynamics model will eventually lead to overfitting. Instead, our method updates the parameters of the dynamics model only when there is sufficient new information in the replay buffer.
5.1 Motivation experiment including UBER
The primary purpose of the proposed algorithm for the resulting replay buffer retain only relevant, non-redundant, experiences that will be useful for learning single task. This experiment is intended to

---

Published in Transactions on Machine Learning Research (02/2025)
show that even when learning single task throughout long training sessions, our method retains sufficient experiences to solve the lask while curlailing buffer growth and unnecessory model updates.
Figure 2 top shows the results obtained in CP. Figure 2-mid-righh shows the size of the replay buffer during training. We observe that while the replay buffer grows in the case of PETs, the size the buffer derived from UBER is comparably flat: the buffer resulting from UBER is 10x smaller. The training time per episode (Figure 2 mid-left) remains nearly constant and lower for UBER. FIFO variants achieve a stable wall-time after reaching the max buffer size. The wall time for PETs exhibits linear growth and takes substantially longer than both FIFO and UBER to complete an episode (i.e., it takes longer to update the model as the replay buffer grows linearly). Figure 2-1eft shows comparable reward per episode for PETs and UBER, whilst FIFO variants exhibit a cyclic reward, which we attribute to forgetting and regaining experiences. Results for reward in Figure 2 CP and RE (row 2) are consistent. FIFO rewards for Masspoint are more stable.
Discarding Model Paramcters. To further demonstrate that our algorithm retains relevant samples while maintaining minimal bulfer size, we evaluate the methods by retaining and discarding model pa- rameters. First, train each method to convergence and save the replay buffer. then, we discard the network parameters, retrain the model using only the replay buffer from the previous step, and evaluate its performance. For each method, we report both upper and lower performance bounds. The starting point for buffer size selection is the size obtained from UBER. From there, we explore larger and smaller sizes in fixed increments. The upper bound corresponds to a buffer size beyond which further increases do not improve performance. Further, we gradually reduce the buffer size until the algorithm can no longer solve the task. This experiment aims to highlight the sensitivity of traditional MBRL methods to buffer size selection. If the experiences in the buffer are truly essential, performance should remain unchanged after discarding the model parameters.
Figure 3 illustrates the efficacy of each method when either retaining or discarding network parameters post- training. Across all environments, UBER maintains a relatively small buffer while sustaining performance, even after discarding the model parameters, retraining, and testing. The minimal performance drop after discarding weights demonstrates that UBER effectively covers the state space while maintaining a compact replay buffer with only relevant experiences. Notably, UBER generally performs on par with RS while offering the key advantage of not requiring a pro-set buffer size. This highlights an important distinction: RS and FIFO exhibit significant sensilivity to buller size, whereas UBER mitigates this issue.
Experiences added. Figure 4 shows the buffer growth in Masspoint when ex- periences are added to the replay buffer by UBER. When the model untrained, many experiences are added throughout the episode. However, after the model is trained, UBER stops adding experi- ences to the buffer as the model can pre- dict them. As a result, new experiences are considered redundant and unneces- sary for the model. The results sup- port our claim that UBER achieves a much smaller, intelligently populated re- play butfer containing only relevant infor- mation, without needing to know task boundaries or set a fixed buffer size, while still maintaining perfor- mance.
Sectort: Episode 1
Sectort: Episode 4
Sectort: Episode 99
300
Reward
200
100
200
200
200
Slep
Reward
Cumulative Steps to Buffer
Figure 4: Steps Added to the Replay Buffer. Per-step re- ward and cumulative steps added to the replay buffer for an un- trained (left), partially trained (middle), and fully trained model (right) in Masspoint. The plots show that UBER reduces redun- dant experiences as training progresses.
6 Towards continual learning
Applying MBRL to a continual learning setting a promising avenue for research since the dynamics model could constantly improve and adapt dynamically to changes in the environment. Many real-world applications can be broken into sequences of tasks of arbitrary length, which, in some cases, support a more realistic learning regime for the agent. Although capturing the complete dynamics requires exposure to
10

---

Published in Transactions on Machine Learning Research (02/2025)
CP
Reacher
100
100
Relative perf. (%)
Perforance Discard
Relative perf. (%)
Perfomance Keep
Relaive Boxter But S22
75
75
50
25
25
0
PETs
FIFOS00
FIFO1000
RS500
RS1000
UBER
PETs
FIFO1000
FIFO5000
RS1000
RS2000
UBER
MassPoint
NonLinearBikeModel
100
100
Relative perf. (%)
Relative perf. (%)
75
75
50
50
25
PETs FIFO250
FIFO1000
RS250
RS1000
RS1500
UBER
PETs FIFO1250 FIFO5000
RS1000
RS2000
RS8000
UBER
Figure Effect of Model Parameters Retention and Discard. The left columns show results from training convergence and subsequent testing. The middle columns show outcomes of discarding network parameters post-training while preserving experiences in the replay buffer and testing. Both left and middle columns show performance percentage relative to PETs. The right columns compare the relative size of the replay buffer to the PETs. Minimal performance drop after discarding weights shows UBER effectively covers the state space, maintaining & minimal replay buffer size with relevant experiences.
longer sessions of CL, we argue that as the model learns, the experience buffer and training time should diminish when the tasks share some similarities with previously learned ones. Arbitrarily long repetitive tasks lead to increasing redundancy in the experience replay, which constantly grows indefinitely with all collected experiences. However, we pose that for scalable CL setting, the experience replay buffer should be limitod those experiences that properly represent the common and rare events, excluding experiences that do not provide new information to the buffer. This serves two purposes: first, to avoid infinitely growing buffer the sequence grows longer, and second, to provide better support and less redundancy from the experience replay.
The Cartpole and Reacher environments are typical benchmarks in RL that emphasize short-term, episodic adjustments to either maintain or reach a specific state. However, real-world settings often require continual learning with long-term adaptations, evolving contexts, and the integration of past knowledge into new tasks. We propose using the racing environment, which is diverse and versatile. It can accommodate various track geometries and shapes, and incorporates overtaking, cornering, and fuel management. This diversity provides richer learning landscape compared to more constrained tasks like Cartpole Reacher. By focusing on racing environments, we can gain deeper insights into the strengths and weaknesses of our algorithms, pushing the boundaries of continual learning.
6.1 Scenarios and task implementation
To further Lest our method we show that our approach helps to mitigale catastrophic forgetting. When using a fixed replay buffer size, it is important to ensure that the appropriate maximum buffer size is chosen (Zhang & Sutton, 2017). If this value undortuned, important experiences can be jettisoned, and catastrophic forgetting can occur. The continual learning setting exacerbates this effect because, without knowing the number of tasks, the buffer size cannot be determined in advance. Thus, we experiment in
11

---

Published in Transactions on Machine Learning Research (02/2025)
Task Agnostic Continual Reinforcement Learning where the model is not aware of tasks or task transitions. In addition, the tasks in the environments overlap, in the sense that some experiences are redundant and may occur similarly in many tasks while other experiences appear rarely or even exclusively in some tasks. This setting requires algorithms to retain releevant collection of experiences to succeed and to achieve strong performance with less data. To lest the existence of these characteristics, after training on each train task, the methods are each tested on the test tasks. The model must remember what it learned by training on each sub-task and apply this knowledge to navigate more complex, unseen task. The scenarios offer such characteristics by building on tasks aiming to traverse path according to some imposed policy (be fast, reduce energy use).
MassPoint tasks. Tasks are defined as segments or as entire tracks. Tasks are defined by the geometry of the path to be followed. EASY Tasks (T01-T08) consist of straightforward and brief segments that make up the Barcelona circuit. MID tasks (T9-T11), these represent three segments of the Barcelona circuit. Note that these MID tasks partially comprise one or more EASY Task. Refer to Figures 8, 9 and 10 in the appendix. HARD Tasks (T12-T14), these tasks involve complete tracks. They are RedBullRing (T12), Barcelona (T13), and simpler Oval (T14). In all tasks, except for T02 and T04, the objective is to achieve maximum speed. However, in tasks TO2 and TO4, specific target speed is set. This demonstrates that our model is not only capable of driving at its fastest but can also maintain a designated speed.
Bike tasks. Similar to MassPoint, we defined a set of tasks to evaluate performance and generalization to unseen complex tasks. EASY tasks: The sub-tasks (T01, TO2, TO3, TO4, TO5, TOG, TO7, TOS, T09) that form parts of the Barcelona (T10) circuit. HARD tasks: The three complete tracks. Barcelona (T10), Red Bull Ring (T11), and Oval (T12). Refer to Figures 11 and 12 in the appendix.
6.2 Evaluation metrics
When evaluating in continual learning scenarios, we need to adapt the metrics to reflect the performance basod on the current performance of the model, the forgetting of previous tasks, and the gencralization towards future tasks. Let M be the task performance mairix where each element M, represents the perfor- mance of the algorithm on test task T, after being trained on train task T. The diagonal elements of the matrix represent the immediate performance of the algorithm on task right after training on that same task. This is denoted by Ma, such that the algorithm is tested on task T; immediately after being trained on task T;.
After the algorithm finishes training on the last training task Tm, its performance is evaluated across all test tasks Tj, for 2.......... which corresponds to the last row in the matrix. This provides the per- formance of the algorithm on all tasks after training on the whole sequence, by averaging over the row: Mn. = A Mmj- This is one of the main metrics used to report performance over sequence of tasks (Nor- mandin et al., 2021; Masana et al., 2022).
For task Tj, foryetting is reported as the difference between the performance of the algorithm when it was Iearned and after training on all tasks, F5 = Mjj - Mmj- This measures how much the performance on a task has deteriorated after the algorithm has been trained on all subsequent tasks (Chaudhry al., 2018). We report the average forgetting across all tasks: 1 (M; Mmy).
Finally, for measuring the gencralization capabilities of the algorithm, we can evaluate the model on unseen tasks. These would correspond to the values above the diagonal of M. However, also measure the performance on unseen tasks U, (tasks not included in the training sequence) after the algorithm learns all training tasks. This measures how well the algorithm can apply the knowledge learned during training to new, unseen tasks.
7 Experimental analysis/resulte
7.1 Experimental Protocol
We conduct three runs for each hyperparameter configuration: varying the 8 parameter for UBER, and the buffer size for the other methods. All other hyperparameters are kept consistent. with the settings from
12

---

Published in Transactions on Machine Learning Research (02/2025)
EASY HARD
EASY+MID HARD
ALL -> ALL
Relative Perf. (%)
Relative Perf. (%)
Relative Perf. (%)
0.8
0.8
0.8
PETs
0.7
0.7
Scratch
0.7
FIFO
0.6
0.6
UBER
0.6
102
10
10
10"
10
10
Buffer size (log scale)
Buffer size (log scale)
Buffer size (log scale)
Name
Easy
Hard
Easy+Mid +
Hard
All + All
Buff Size
Perf.
Buff Size
Perf.
Buff' Size
Perf.
PETs
37150
100.0
37150
100.0
97150
100.0
Scratch
6300
11.0
6300
97.7
21000
33.6
FIFO 2000
2000
10.4
2000
96.6
2000
48.9
FIFO 20k
20000
100.0
20000
99.9
20000
96.5
FIFO 50k
37150
100.0
37150
100.0
50000
98.8
A-FIFO250
250
1.3
250
80.3
250
89.5
A-FIFO500
500
2.6
500
94.6
500
99.8
A-FIFO1000
1000
5.2
1000
98.5
1000
100.1
RS125
125
0.7
125
89.3
125
98.8
RS250
250
1.3
250
95.2
250
99.1
RS500
500
2.6
500
98.0
500
99.4
RS1000
1000
5.2
1000
99.4
1000
99.4
UBER(3=1.4)
966
5.0
966
93.4
1558
99.4
(BER()=1.1)
1390
6.8
1390
99.6
1698
99.4
UBER(3=0.9)
1942
9.4
1942
100.0
2303
99.3
UBER(3 -0.7)
2712
12.7
2712
100.0
3225
99.5
Figure Generalization and Performance in Masspoint. The top row presents performance plots showing generalization trends: (left) training on EASY tasks and testing on HARD tasks; (center) training EASY and MID tasks and testing on HARD tasks; (right) training and testing all difficulty levels. The x-axis (log scale) represents the number of experiences in the replay buffer, while the y-axis represents normalized rewards. The table below summarizes numerical results. Black cross is PETs.
the motivation experiment. Each run uses different random seeds and initial conditions. PETs is treated special case of VanillaFIFO, where the buffer size is effectively infinite, allowing all experiences to be retained indefinitely. For VanillaFIFO, A-FIFO, and RS, we evaluate different buffer sizes to assess the adaptive algorithm's 's ability to dynamically adjust and optimize memory usage based on the task at hand. For UBER, we build on our single-task experiments, keeping the same hyperparameters to validate their generalizability in more complex scenarios. However, we also explore a range of 3 values to analyze its effect on performance. This experiment is set in a Task Agnostic Continual RL setting, where the model is not aware of tasks or task transitions. We utilize both the Masspoint racing environment and the more complex NonLinear Bicycle model, defining different tasks with varying levels of complexity.
13

---

Published in Transactions on Machine Learning Research (02/2025)
7.2 Masspoint
We evaluate PETs, VanillaFIFO, Scratch, A-FIFO, RS, and UBER with various buffer sizes. PETs and Scratch use virtually infinite buffers. VanillaFIFO and A-FIFO are tested with different buffer sizes, and UBER with various values of B. For VanillaFIFO, buller sizes are BS 2000, 20000, 50000. For A-FIFO, buffer sizes are BS E 250, 350, 500, 1000, 2000. For UBER, 3 values close to 1.1 are tested 3 E 0.7, 0.9, 1.1, 1.4.
We assess performance in terms of per-episode reward, and replay buffer size. Each task is trained for 30 episodes each lask and then lested in the test tasks for a single episode. We report, generalization to unseen tasks, final performance seen tasks, and forgetting. Each test result is normalized by divid- ing it by the maximum theoretical value, calculated as the top speed multiplied by the task's time steps. Episode Reward
Normalized Value Top Speedx Time Steps"
Generalization to unseen tasks. Figure 5-left shows the generalization from EASY to unseen HARD tasks. PETs showed excellent generalization to hard tasks. VanillFFIFO with sufficient buffer size achieved performance similar to PETs, but with a smaller buffer, it failed due to filtering out important examples. Scratch failed to generalize. A-FIFO's performance varied with changes buffer size. This indicates that A-FIFO is sensitive to the choice of replay buffer size. RS shows similar trend while UBER achieved same performance PETs and A-FIFO with significantly smaller buffer.
Figure 5-middle shows the generalization from MID to unseen HARD tasks. Similar to the previous case, PETs generalized well. Vanilla with 50k and 20k samples achieved comparable performance to PETs, while Vanilla with 2k samples underperformed. A-FIFO's behavior was consistent with the previous case. UBER achieved optimal performance with 1700 examples, which is 23% less examples than A-FIFO and just a 4% of the total examples collected by PETs.
Same-Distribution Testing. Figure 5 (right) shows the test performance on the same tasks used during training, including EASY, MID, and HARD tasks. VanillaFIFO's performance declines after the buffer size reaches 20k samples, indicating an increasing need for buffer size as more tasks are introduced. A-FIFO and RS achieve optimal performance with only 1 steps, demonstrating the value of information ion from all tasks. UBER performs consistently well across different tested hyperparameters. With a B of 1.1, UBER's final buffer size is 37% smaller than A-FIFO's and only 1.5% of PETs' buffer size.
Forgetting. Figure 6 illustrates the forgetting metrics for each method and different hyperparameters. The models are trained on tasks T1 to T14. After completing each training task, the model is tested across all tasks encountered up to that point. The y-axis in each figure represents the forgetting metric, while the x-axis indicates the last task completed before the respective test. A negative mumber indicates forgetting, while a positive number indicates an improvement in performance.
Note that the forgetting met- ric assesses the extent of for- getting but does not account for the model's absolute per- formance on the test. tasks. As a result, a model may exhibit poor overall performance yet show minimal forgetting. This explains why certain methods experience reduction in for- getting after encountering new tasks, leading to a positive for- getting trend. This trend oc- curs because the model's per- formance improves after en- countering a new task, con- tributing to an overall im- provement.
0.04
Method
Forgetting
J
Forgetting
0.02
PETs
0.00
Scratch
2.40
0.00
FIFO1768
2.51
-0.02-
A-FIFO350
0.20
-0.04
A-FTFO500
0.00
T0S T07 TO8 T11
RS250
0.30
Scratch
PETs
UBER 0.9
RS1000
0.02
FIFO 1768
A-FIFO 250
RS 500
UBER,30.9
0.00
Figure 6:
Forgetting in Masspoint. Most methods exhibit forgetting
when the buffer size is reduced. UBER maintains performance while keeping the buffer size to a minimum. The table shows the sum average forgetting for cach method (4 lower better).
14

---

Published in Transactions on Machine Learning Research (02/2025)
As expected, PETs has zero forgetting due to its infinitely large buffer. Conversely, Scratch has the highest forgetting rate, as it does not retain any information from previous tasks. With VanillaFIFO, reducing the buffer size limit leads to observable forgetting. In the case of A-FIFO, there is virtually no forgetting for buffer sizes larger than 500. However, reducing the buffer size limit results in an increase in forgetting. RS follows a similar trend. In the case of UBER, forgelting is virlually zero for all the lested hyperparameters except for / 1.4. This is expected, as UBER retains experiences only when the model is not confident enough.
Summary, Our results in the Masspoint environment reveal that undertuned fixed buffer size leads to poor performance and that the non-fitterngg algorithms hit the buffer size cap and throwing away valuable experiences, resulting in the model forgetting how to properly solve tasks that were trained carly on. This is detrimental in the three aspects investigated in this experiment: generalization, forgetting and same- distribution testing. While PETs successfully excels in the evaluated aspects, it does so at the cost of having an unbounded buffer and increasing training time, making unsuitable for the CL setting. Seratch training failed due to lack of retaining important information when switching tasks. A-FIFO and RS have a good performance, but al the cost. of having to lune the replay buller size, making it dependent on the number of tasks in hand, which is not feasible in the CL setting, UBER keeps the number of examples collected to a minimum without having to tune the replay buffer size, keeping the training time at minimum and effectively addresses catastrophic forgetting. It is the only method presented in this work suitable for the Continual Learning setting as it is the only method whose RB size does not need to be tuned depending on the number of tasks.
7.3 Non linear Bicycle model
We evaluate PETs, VanillaFIFO, A-FIFO, RS, and UBER, omitting Scratch due to its poor performance
in this environment. For VanillaFIFO, we evaluated buffer sizes of BS E 50000,50000, For A-FIFO, we evaluated buffer sizes of BS E 5000, 10000, 25000, 50000. For UBER, we evaluated different values of B close to 1.5 x 10 44 which gave the best results in the preliminary single-task study: 3 [2.5 x 10 10-4,5.0 x 10-1, 1.5 x 10-1, 1.0 x 10-"). For RS, we used buffer sizes of 5000 and 10000. We assess performance in terms of the number of steps (simulator time) to finish a task and replay buffer size. Each method is trained for 30 episodes per task and then tested on the test tasks for a single episode.
Unlike the Masspoint enviromment, the number of steps per episode in this environment is variable, lacking theoretical upper bound. Therefore, we normalized the results for cach task by calculating the percentage of performance drop relative to PETs (the best-performing method). We subtracted the PETs task time from the resulting task time to get the delta (dt = task time - PETIs task time), and then normalized it using p = 1- max delta Modols were tested and trained on EASY, HARD and ALL tasks.
di
Generalization to Unseen Complex Tasks, Figure 7 (left) shows the generalization from EASY to unseen HARD tasks for each method with different hyperparameters. No method generalizes well to hard tasks, with each showing a significant drop in performance compared to PETs at convergence. However, increased experience leads to slightly better generalization. In the very low sample regime, RS performs worse than UBER, but at 10k samples, RS outperforms UBER. UBER's 's core design focuses on retaining the experiences most useful for the current task, making it less robust when the target task is out-of-distribution. This selective retention works well for nificantly different tasks. In contrast, RS performs better in this specific setting due to its diverse experience buffer, which is beneficial when dcaling with a small number of tasks. However, RS does not scale as well to scenarios involving more tasks, since the buffer size needs to be carefully tuned. These results suggest that achieving strong generalization may require training on broader set of diverse tasks.
tasks but may struggle when transitioning to sig-
Samc-Distribution Testing. For EASY to EASY tasks (Figure 7 center left), methods with larger buffers perform better. UBER maintains reasonable performance with a very low buffer. When training on all tasks and testing on hard tasks (Figure center right), and training on ALL tasks and testing on ALL tasks (Figure 7 right), VanillaFIFO performs well but requires a hand-tuned buffer. Plots show a clear increase in performance with larger buffer sizes. UBER performs similarly to RS and better than A-FIFO. PETS shows slight performance drop when training on ALL tasks and testing on ALL tasks, indicating that redundant
15

---

Published in Transactions on Machine Learning Research (02/2025)
EASY HARD
EASY -> EASY
ALL HARD
1.0
1.0
1.0
x
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
10
10
10
10
Name
Easy
Hard
Easy
+ Easy
All + Hard
Buff Size
Perf.
Buff Size
Perf.
Buff Size
Perf.
PET:
26905
58%
26905
91%
80752
100%
FIFO25k
25000
42%
25000
90%
25000
58%
FIFO50k
28362
78%
28362
89%
50000
100%
A-FIFO5k
5000
27%
5000
78%
5000
59%
A-F1FO10k
10000
24%
10000
83%
10000
95%
A-FIFO25k
25000
35%
25000
80%
25000
99%
A-FIFO50k
27351
51%
27351
82%
50000
99%
RS5000
5000
18%
5000
58%
5000
59%
RS10000
10000
91%
10000
90%
10000
100%
UBER(8=0.00050)
4819
39%
4819
77%
5767
95%
UBER(8-000005)
9156
40%
9156
80%
11486
97%
UBER(3=0000015)
9940
25%
9940
82%
14412
99%
UBER(3=0.0000))
18544
40%
18544
77%
27773
100%
Figure 7: Generalization and Performance in the Nonlinear Bicycle Environment. The top row presents performance plots: (left) train on EASY tasks and test on HARD tasks (generalization); (center) train on EASY and test EASY; (right) train on ALL and test on HARD. The x-axis (log scale) represents the number of experiences in the replay buffer, while the y-axis represents the performance drop relative to PETs. Below, the table summarizes numerical results for different buffer sizes and methods.
data detrimental in this case. Other methods do not exhibit this issue and require smaller buffer sizes. UBER's advantage lies in not needing to set the buffer size, making it suitable for larger number of tasks.
Summary. The results follow a similar trend to the MassPoint experiment in the same-distribution setting, but all methods perform poorly in the generalization setting, highlighting the complexity of the tasks. Our experiments show that UBER performs well independently of the number of tasks.
8 Discussion and Conclusion
The replay buffer cnhances the stability of deep neural networks in RL and is an essential component of several algorithms. However, analyses of replay bullers are relatively scarce. Recently, research has begun to focus on analyzing the contents and strategies for managing the replay buffer in RL agents (Fedus et al., 2020), as well as in supervisod learning (Aljundi et al., 2019). In this work, we contribute to this body of research by analyzing and proposing strategies to manage the growth of the replay buffer in model-based RL. In this setting, our studies show that. UBER maintains leaner and more relevant collection of experiences
16

---

Published in Transactions on Machine Learning Research (02/2025)
Method
Retention Strategy
Buffer Size
Buffer Specification
Task Transition Aware
PETs
Greody
Fixed
Automatic (All and Keep)
No
PETs Scratch
Greedy
Fixed
Automatic (All and Discard)
Yes
FIFO
FIFO
Fixed
Manual
No
A-FIFO
Adaptive
Fixed
Manual Alyper-parameter
Yes
RS
Replacement Adaplive
Fixed
Manual Slyper-parameten
No
UBER
Flexible
Automatic
No
Table 1: Comparison of continual reinforeement learning properties for MBRL algorithms.
in the replay buller than do baseline algorithms. These characteristics of the proposed algorithm, we posit, result in strong test performance with less data and greater stability.
Comparing UBER with Existing Replay Buffer Strategies. Table provides a comparative overview of the replay buffer management strategies across various methods. Most methods employ a greedy retention strategy with fixed buffer size. Methods like FIFO and A-FIFO utilize first-in-fist-out or strategess with manual syper-parameter tuning. In contrast, UBER distinguishes itself by using adaptive retention strategy and a flexible buffer size with automatic management, which eliminates the need for extensive pre-tuning.
It is important to note that RS and UBER are inherently different, and each has their own strengths. UBER addresses a problem that cannot. be directly addressed with RS, namely the continuous interaction and learning in the environment when we have no information about the cap size of the experience replay buffer. UBER's key advantage lies in not requiring prodefined buffer size. This ficxibility is particularly beneficial in dynamic, never-ending or multi-task settings, where pre-determining the optimal buller size is challenging. Our core contribution is a method that automatically optimizes the replay buffer without requiring preset sizes.
Future Work. Having managed growth, there are several aspects we would like to turn to in the future: i) identifying task boundary from the novelty of experiences, ii) managing what to forget for limited size buffers, iii) managing what to remember / refresh when a change in task is evident. This would allow to run agents for arbitrary time without having to deal with size of the buffer and would ofler promising opportunities for deploying MBRL in a CL setting.
UBER could be used to prioritize entries in the replay buffer where the model was uncertain. Indeed, prioritized buffer strategies support the usage of experience once it is in the buffer, but as stated by Schaul et al. (2016), strategies for what to add and when (our work) are important open avenues for rescarch. We did not explore our methods in environments where the tasks have interfering dynamics. But, if the dynamics change, poor predictions by the model will result in adding experiences to the replay buffer. What happens interfering tasks occur permanently an interesting follow up.
Expanding the experiments high-dimensional or additional multi-task settings, such Meta World, is out- side the scope of the current study. UBER is implemented top of PETs, which rely explicitly modeling the environment dynamics and incorporsting domain knowledge into the dynamics model. Implementing UBER into state-of-the-art MBRL algorithms would facilitate expansion to high-dimensional visual inputs and represent a promising research direction for future work.
Conclusion, We proposed strategics that comply with requirements for continual learning. Our approach retains only memories which are useful: it obtains lean and diverse replay bullers capturing both common and sporadic experiences with sufficient detail for prediction in longer learning sessions. Our approach manages compute and memory resources over longer periods: it deals with the unbounded growth of the replay buffer, its training time and instability due to catastrophic forgetting. These results offer promising opportunities for deploying MBRL in continual learning setting.
17

---

Published in Transactions on Machine Learning Research (02/2025)
Acknowledgments
Marc Masana acknowledges the support by the "University SAL Labs" initiative of Silicon Austria Labs. The Know-Center is funded within the Austrian COMET Program Competence Centers for Excellent Technologies under the auspices of the Austrian Federal Ministry of Transport, Innovation and Technology, the Austrian Federal Ministry of Economy, Family and Youth and by the State of Styria.
References
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Informalion Processing Systems, pp. 11816-18111 11825, 2019.
Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret, 2015.
Mare G Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449 458. PMLR, 2017.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, pp. 679-684, 1957.
Zdravko I. Botev, Dirk P. Kroese, Reuven Y. Rubinstein, and Pierre L L'Ecuyer. Chapter 3- the cross-entropy method for optimization. In C.R. Rao and Venu Govindaraju (eds.), Handbook of Statistics, volume 31 of Handbook of Stalislics, pp. 35 - 59. Elsevier, 2013. doi: 00003-5. URL http:/ http://wwww.ee htt.e//ciie....ooo.ccc..eeeo/eea
E.F. Camacho, C. Bordons, and C.B. Alba. Model Predictive Control. Advanced Textbooks in Control and Signal Processing. Springer London, 2004. ISBN 9781852336943. URL https://books google. at/books? id=Sc1H3f3E8CQC..
Arslan Chaudhry, Puncet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the Eurooeaan conference on computer vision (ECCV), pp. 532 547, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models, 2018.
M. P. Deisenroth, G. Neumann, and J. Peters. 2013. doi: 10.1561/2300000021.
Marc Peter Deisenroth and Carl Edward Rasmussen. Pilco: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning, 2011.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Robert Dumning, Shane Legg, and Koray Kavukcnoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. 2018. URL https://arxiv.orr abs/1802. 01561.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay, 2020.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination, 2019. URL https: http:///aaaiiv.or///////////////
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the Thörty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Arti- ficial Intelligence Conference and Eighth AAAI Symposium on Educational Advances Artificial Intelli- gence, AAAT18/TAATIN/FAATT AAAT Press, 2018. ISBN 978-1-57733--80--8
18

---

Published in Transactions on Machine Learning Research (02/2025)
Samuel Kessler, Mateusz Ostaszewski, Michal Bortkiewicz, Mateusz Zarski, Maciej Wolezyk, Jack Parker- Holder, Stephen J. Roberts, and Piotr Milos. The effectiveness of world models for continual reinforcement learning, 2023.
Khimya Khetarpal, Matthew Riemer, Trina Rish, and Doina Precup. Towards continual reinforcement learn- ing: A review and perspectives, 2020.
James Kirkpatrick, Razvan Pascamu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521 -3526, 2017.
Jason Kong, Mark Pfeiffer, Georg Schildbach, and Francesco Borrelli. Kinematic and dynamic vehicle models for autonomous driving control design. 2015 IEEE Intelligent Vehicles Symposium (IV), pp.
2015.
Balaji Lakshminrrayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncer- tainty estimation using deep ensembles, 2016.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning, 2015. URL https: wwrrxiv.org/ab//155 02971..
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, pp. 293-30 321, 1992.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey, Neurocomputing, 469:28 51, 2022.
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D. Bagdanov, and Joost van de Weijer. Class-ineremental learning: Survey and performance evaluation on image classification. pp. 1-20, 2022. doi:
M. McCloskey and N. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Molivation, 24:109-14 165, 1989.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Toannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. In Advances in Neural Information Processing Systems, Deep Learning Workshop, 2013.
Anusha Nagabandi, G. Kahn, Ronald S. Fearing, and S. Levine. Neural network dynamics for model-basod deep reinforcement learning with model-free fine-tuning, 2018 IEEE International Conference on Robolics and Automation (ICRA), pp. 7559-7566, 2018
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning: Continual adaptation for model-based rl, 2019. URL https://arxxt wttpr/wwwworg/b////...rrr 07671
Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez, Matthew D Riemer, Julio Hur- tado, Khimya Khetarpal, Dominic Zhao, Ryan Lindeborg, Timothée Lesort, et al. Sequoia: A software framework to unify continual learning research. arXiv e-prints, pp. arXiv-2108, 2021.
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In NIPS, 2017.
Ameya Prabhu, Philip HS Torr, and Puncet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In Proc. European Conference on Computer Vision, pp. 524-540. Springer, 2020.
Rajesh Rajamani. Vehicle dynamics and control. Springer Science & Business Media, 2011.
Anvil V. Rao. A survey of numerical methods for optimal control. Advances in the Astronauticol Science, 135:497-528, 2010.
19

---

Published in Transactions on Machine Learning Research (02/2025)
Adrian Remonda, Eduardo E. Veas, and Granit Luzhnica. Acting upon imagination: when to trust imagined trajectories in model based reinforcement learning. CoRR, abs/2105.05716, 2021. URL https: //axxvv. org/abs/2105.0577 05716.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experi- ence replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Newral Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https://proceedingg neurssnneriipip httpscccppperrfeleeeeeepp http//perr..lllssppee/////////// www.c-Paae......omeae
Tom Schaul, John Quan, Toannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR
(Poster), 2016.
Richard Sutton. Integrated architoctures for learning, planning, and reacting based on approximating dynamic programming. In ML Workshop, 1990.
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E. Gonzalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safety augmented value estimation from demon- strations (saved): Safe deep model-based for sparse cost. robotic tasks, 2020.
S. Thrun. A lifelong learning perspective for mobile robot control. In Proceedings of IEEE/RSI RRS International Conference on Intelligent Robots and Systems (IROS'94), volume 1, pp. 23 30 vol.1, 1994. doi: 10.1109/ IROS.1994.400 1994.40411.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033. IEEE, 2012. ISBN 978- 178-1-4673-173755
Jeffrey Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Softwore (TOMS), 11:37-57, 03 1985. doi: 10.1145/3147.3165
Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. Information theoretic model predictive control: Theory and applications to autonomous driving, 2017.
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences. arXiv preprinl arXiv:2109.091 2021.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task-Agnostic Continual Learning Using Online Variational Bayes With Print-Point Updates. Neural Computation, 33(11):3139-3111 -177,, 10 2021. ISSN 0899-7667. doi: 10.1162/neco
01430. URL https://doi. 1ttg:/001112//eccaa_011111
Shangtong Zhang and Richard S. Sutton. A Deeper Look at Experience Replay. arXiv e-prints, art. arXiv:1712.01275, December 2017.
Guangxiang Zhu, Minghao Zhang, Honglak Lee, and Chongjie Zhang. Bridging imagination and reality for model- based deep reinforcement learning. NewrIPS, 2020.
20

---

Published in Transactions on Machine Learning Research (02/2025)
Appendix
9 Mass point tasks Task definition
Tasks are defined as segments the Barcelona track (T13) or as entire tracks. Tasks are defined by the geometry of the path be followed. In all tasks, except for T02 and T04, the objective is to achieve maximum speed. However, in tasks TO2 and TO4, specific target speed is set. This demonstrates that model is not only capable of driving at its fastest but also maintain designated speed. The specific definitions of the lasks are as follows: HARD Tasks (T12-T14): Displayed in Figure 8, these tasks involve complete tracks. From left to right, they are RedBullRing (T12), Barcelona (T13), and a simpler Oval (T14). MID Tasks (T9-T11): As seen in Figure 9, these represent three segments the Barcelona circuit. EASY Tasks (T01-T08): Figure depicts these lasks. They consist of straightforward and brief segments that make up the Barcelona circuit (T13).
Figure HARD tasks (T12-T14) for the Masspoint environment. In each figure, the x-axis and y-axis represent the x,y coordinates of the path the mass point bot should follow. The red dot denotes the starting position. Top left-to-right: RedBullRing (T12), Barcelona (T13), and simpler Oval (T14), respectively,
Figure 9: MID tasks (T9-T11) for the Masspoint environment. In each figure, the x-axis and y-axis represent the x,y coordinates of the path the mass point bot should follow. The red dot denotes the starting position. Left-to-right: Sectorl (TO9), Sector2 (T10), and Sector3 (T11) of the Barcelona circuit (T13).
Figure 10: EASY tasks (T01-T08) for the Masspoint environment. In each figure, the x-axis and y-axis represent the x.y coordinates of the path the mass point bot should follow. The red dot denotes the starting position. These are sub-tasks taken from the Barcelona circuit (T13)
10 Non-linear Bicycle Model Task definition
In the Non-linear Bicycle Model environment, we have defined set of tasks. Figure 11 presents the three complete tracks: Barcelona (T10), RedBullRing (T11) and Oval (T12), displayed in the left, middle, and right images, respectively. Figure 12 illustrates the sub-tasks (T01, TO2, TO3, TO4, TO5, T06, TO7, TO8,
21

---

Published in Transactions on Machine Learning Research (02/2025)
T09) that form parts of the Barcelona circuit. The x-axis and y-axis in each figure denote the x coordinates of the track path's center. The model receives data about the track borders, the objective is to ensure the car remains within the track.
Figure 11: Tasks for the Non-linear bike model environment. The x-axis and the y-axis of each figure represents the x, y coordinates of the path to be followed by the car, Left-to-right: Barcelona (T10), RedBullRing (T11) and Oval (T12)
Figure 12: Sub-tasks for the Non-linear bike model environment derived from the Barcelona circuit (T10). In each figure, the x-axis and y-axis denote the and y coordinates of the path the should follow. Top row, left-to-right: (T01, TO2, TO3, T04, TO5). Bottom row, left-to-right: (T06, TO7, TO8, TO9).
11 Maximum Prediction Distance
A parameter of interest when using UBER is the marimum prediction distance (MPD). This parameter is based the idea that even if a model has reached convergence, in certain environments, prodicting very long necessary at the end of such extended trajectories.
trajectories is not feasible. Therefore, recalculations These recalculations do not necessarily indicate the arrival of new unseen information, but instead reflect the limitations of a successful model in a complex environment. As result, we would prefer not to add these experiences to the buffer.
The cutoff for what considered a great length trajectory can be adjusted, allowing us to fine-tune the strictness of UBER's filtering mechanism. For example, in Ex.1 and Ex.2, we set the maximum prediction distance to 1 to apply the strictest filtering of the replay buffer.
In Figure 13, we evaluate the effect of MPD on UBER's performance in the cartpole environment, focusing on its impact on recalculation rates and replay buffer size. As shown in Figure 13, all models converge successfully, though they display slight differences recalculation rales and buller filtering. The strictest setting, MPD=1, results in the smallest buffer but slightly higher recalculation rates compared to models with MPD=2 and MPD=4.
These results indicate that MPD is useful tool for adjusting the strictness of UBER's buffer filtering.
22

---

Published in Transactions on Machine Learning Research (02/2025)
Ephice Rohard
9.24
200
100
100
0.22
100
20
50
300
500
Figure 13: Performance of the examined algorithms in Cartpole using different maximum prediction distances (MPD). The blue line represents UBER with an MPD=1. The red line is UBER with an MPD=2. The green line is UBER with an MPD=4. From left to right column: episode reward, time per episode (s), cumulative number of observations stored in the replay buffer, new experiences added to the buffer per episode.
12 Hyperparameters
Table 2 shows the hyper paramcters used to train UBER. Look-ahead refers to the number of steps ahcad UBER uses to asses the quality of the imagined trajectories. B controls the sensitivity of UBER to inform whether a trajectory is still valid or not. New Data Train Threshold refers to the amount of fresh data that must be added to the replay buffer before the UBER algorithm triggers the training of the dynamics model.
Cartpolc
Reacher
Masspoint
Kook-Ahead
1
8
0.005
0.004
1.5
Training episodes
100
100
30/task
CEM population
400
400
400
CEM # elites
40
40
40
CEM # iterations
5
CEM
0.1
0.1
0.1
MPD)
10
Table 2: Hyperparameters used for UBER implementation.
13 Environments
We evaluate the methods on agents in the CartPole and Reacher environment provided by the Mu- JoCo (Todorov et al., 2012) physics engine. Additionally, we introduce our own proposed environments related to racing, including Masspoint and a Non-linear Bicycle model. We utilize these environments to assess the performance of our methods. The choice of CartPole and Reacher environments is based on their established use in reinforcement learning research, allowing for meaningful comparisons with existing approaches. Introducing our own racing-related environments, such as Masspoint and Non-linear Bicycle, enables us to evaluate the methods' adaptability and effectiveness in scenarios beyond the conventional ones.
13.1 CartPole (CP)
An inverted pendulum problem, which involves balancing pole on a cart. It has movable cart that travels along frictionless track. On top of the car pole with one end attached to the cart, is standing upright. The objective is to prevent the pole from falling over. The state variables, represented by S e R& include: the position of the cart along the track, the velocity of the cart, angle of the pole rolative to vertical, the angular velocity of the pole. The action space (R+) is continuous, allowing the agent to apply a force F to the cart, where F' can vary within a predefined range: -Fmax < F < Fmax- The goal to keep the pole upright for as long as possible by moving the cart left or right. The episode ends after a time limit (TaskH) of 200 steps has been exceeded. The Trajectory horizon set for the controller (H), is 25. The agent receives reward of for every time step the pole remains upright.
23

---

Published in Transactions on Machine Learning Research (02/2025)
13.2 Reacher (RE)
The Reacher task involves a robotic arm with 6 Degrees of Freedom (6-DoF) aiming to reach a target position in space, given the multiple joints and their rotations. State Variables: joint angles for each of the 6 joints, angular velocities for each joint, current position (x, y; w) of the end effector, larget position (x, y, z). Actions: the action space is continuous, allowing the agent to apply torques to each of the joints. The torque applied each joint can vary within a predefined range: -T maxxx S T; S Tmax, for i= --.,..... 1,...: 6. The state space in this case is S E R!7 and the actions space is E R". We set TaskH to 150 and H to 25. The reward function aims to minimize the distance between the end effector and the target position. The agent receives a reward based on the negative Euclidean distance between the current end effector position and the target.
13.3 Masspoint (MP)
We also included an extended version of the Masspoint environment proposed by Thananjeyan et al. (2020). Masspoint is a navigation task which a point mass navigates to a given goal. It is a 5-dimensional (z. y, Uz; Vy: p) state domain (S IRo). Where (2, v) is the position of the agent, (Va, fly) its speed, and p is the distance between the agent and the closest point a given path. The agent can exert force in cardinal directions (A IR?) and experiences drag coefficient v. We use e 0.6 and included noise in the starting position. We modified the goal of the agent so that it must move as fast as possible without deviating from a given path. Each task and its complexity is then determined by the geometry of the path to be followed. The reward is calculated as r = V(1 - (p)). Where V is the speed of the agent and p the distance to the task's path. We set H to 25, and TaskH depends on the task.
13.4 Non-linear Bicycle Model (Bike)
We introduce a new environment based on a Non-linear Bicycle Model, capturing vehicle dynamics with greater fidelity and featuring higher action and observation dimensions than MassPoint. The bicycle model simplifies fourewheeled vehicle to a two-wheeled bicycle. This model considers aerodynamics, tire dynamics, and rolling resistance. Additionally, we have integrated track boundaries to enhance the realism and challenge of the simulation. The control variables are steering and combined throttle and brake. The state variables are the position (x. y. v)), velocity (i: ú, v)), and acceleration (æt, ū. ()), throttle, steering angle 5, out-of- track, going backwards, and the distance to the reference path. Our objective is to learn the underlying dynamics model. To achieve this, we train the model to predict accelerations based on current velocities and accelerations. By integrating twice, we obtain the position and can predict trajectories. The complexity of each task depends on the geometry of the designated path. To ensure the model can generalize across different tracks, we deliberately omit the x; y, from the observations. Note that the model still needs to infer the 2, y, 1b positions, but it must do so based on the current state and the predicted accelerations. The reward aims to maximize the speed, and the episode is terminated if the car goes off track.
Vehicle dynamics. The state includes the position variables æ, M, v. their first derivatives I,, ú, v. their second derivatives é. ÿ. Throttle, 6, out-of-track, boing-backwards, and pissest-distanceto-ppee represent. the coordinates of the center of mass in an inertial frame (X, Y). 1/ is the inertial heading. and ÿ/ are the longitudinal and lateral speeds in the body frame, respectively. & denotes the yaw rate. 8 represents the steering wheel angle. Throttle is the combined signal of the throttle and brake pedal, representing the external longitudinal force. oft-of-track and going- backwards are boolean signals, indicating when the model is off the track or moving in reverse, respectively. Closest-distance-to- the designated path. Our objective is to learn the underlying dynamics model. To achieve this, we train the model to predict accelerations based on current. velocities and accelerations. By integrating twice, we obtain the position and can predict trajectories. We introduce noise into the initial position and velocity. The complexity of each task hinges on the geometry of the designated path.
x and y
measures the nearest distance to
The nonlinear continuous time equations that describe dynamics bicycle model in inertial frame are:
24

---

Published in Transactions on Machine Learning Research (02/2025)
á v/ + az
(1)
ÿ +i + ay
(2)
à Ffy Ls cos(ō) - Fry Lr
(3)
1=
X cos 1b - i (s))n()))
(4)
Y I sin v + cos(
Where by is F which are the sum of lateral forces include simplified tire model, acrodynamics and rolling resistance (see the pseudo code). Fcz and Fc, denote the lateral tire forces at the front and rear wheels, respectively, in coordinate frames aligned with the wheels.
Learned dynamics model. The dynamics model aims predict the next state given the current state (e.g. speed and accelerations) and controls (steer, throttle and brake). With good estimate of the future state, we expect to reliably predict trajectories on a finite horizon. We train fa to predict the dynamics of the vehicle, hence the accelerations of the car which can be then integrated twice to recover position.
Applying Newton's second law yields Σs = dy where dy is the vehicle's inertial acceleration at the gravity m center in the direction of the y axis, m the total mass of the car, and Σ F, is the summation the lateral forces. Two terms contribute dy: acceleration due to the motion in the y axis tyy, from which the position is recovered by integrating and the centripetal acceleration V. (Rajamani, 2011):
Er by vath 06 = 0y -Uza) m
4y
(6)
Similarly, equation equation defines az, where vi is the acceleration in the x axis and ny is the centripetal acceleration:
a₂ = v Dypb. di = =a + rygi
(7)
Which can then be integrated to recover the position of the vehicle. We split the state vector in kinematic state variables sı and dynamics state variables Sd, 8s
The kinematic part of the state is St
(Pas Py, T and the dynamic part is Sya (Uz, By. di R)T and the actions given by et al., 2015):
(Kong
st + 1 = (t) + k(s(t))
(8)
The coordinate transformation ion matrix k defined by equation 9 maps the position vector from the vehicle to the inertial frame of reference.
(cos(v)), sin()) sin(v)u, + co)()))
k(s)=
(9)
The full state space equation which we define as the dynamics model (F) becomes:
(5k
k(s)
F(s,, ar) Sc+1
+
(o(Ss))))
At
(10)
(sa)
Where the dynamic part is:
+a(++1) sa(() fo(sa(().
(11)
25

---

Published in Transactions on Machine Learning Research (02/2025)
We get the learning objective for the neural networks
salt + 1) - su(()
(U eig 2b)
(12)
At
Thus, the neural network model
(U 2g. () = fo(ss(t),
(13)
13.5 Pseudo code
epsilon 1e-6
c_a 1.36
Aerodynamics coefficient
c_rl = 0.01
Road rolling resistance
max_steer radians (30.0) [rad] max steering angle
L = 2.9
[m] Wheel base of vehicle
dt_physics 0.1
[s] physics sampling time [m]
Lr = L 2.0
Lr
Cf = 1600.0 0 2.0
N/rad
Front tires coefficient
Cr 1700.0 0 2.0
N/rad kg m2 kg
Rear tires coefficient
Iz 2250.0
Inertia
m = 1500.0
mass of the vehicle
def next_ (del_saa throttle): scale from 1 to to delta = delta * max_steer delta clip(delta, -max_steer. max_steer)
steer
Position the non-intertial frame reference + x² (osi) * dt
sin(psi) dt y = y x² sin(psi) dt y' cos(psi)
psi psi + psi" * dt
psi normalize_angleepssi)
forces
Ffy -Cf arctan2(((y' Lf psi') (x' epsilon) delta), 1.0)
Fry -Cr * (rctan2((y'' - Lr * psi') / (x' epsilon), 1.0)
Aerodynamics
F_aero = c_a * x' **
Road rolling resistance
R_x c_rl x'
F_load F_aero R_x
Dynamics -> to be learned by the NN
x (throttle Ffy sin(delta)
F_load / + y' psi')
y" (Fry / m Ffy * cosldal / m - x" psi')
"s ((Ffy * LF cos(dalta) Fry Lr) Iz)
Velocities relative to the body frame x' = xx' x* * dt y' y' +
psi' psi" " dt dt
26


Reviews:
[{"review_id": "ltKQAcLhRI", "reviewer": "Reviewer_beJY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Overall\nStrengths:\n* Extensive experimentation, and the experiments are motivated.\n\nWeaknesses:\n* The results lack sufficient evidence to conclusively demonstrate UBER’s superiority over Reservoir Sampling (RS).\n* All experiments are conducted in relatively simple state spaces where replay buffer size is not as critical. Testing UBER in visually rich settings would better highlight its effectiveness, as large replay buffers become significantly costlier in those environments.\n* Figure legends are not comprehensive.\n* Results are not well explained in the text.\n\n## Details\n\n### Figure 2\n* Color Accessibility: The color choices are not color-blind friendly, which hinder accessibility.\n* Comparison with RS: It is unclear why RS was not included as a baseline in this figure. Comparing UBER to RS here would provide a more comprehensive view of UBER’s effectiveness.\n\n### Figure 3\n\n* Baseline Hyperparameters: Could you provide additional context on the baseline hyperparameter selection in Figure 3? Detailing the rationale behind these choices would strengthen the comparison.\n* Baseline Competitiveness: The baselines perform quite comparably to UBER across most environments, with CartPole as a notable exception. This figure deserves a more extensive discussion, as it is currently addressed with only a single sentence at the bottom of page 10.\n\n### Figure 5\n* Legend Clarity: The symbols used are not explained in the legend, leaving the black cross unexplained. While this becomes somewhat clearer from Figure 6, each figure should ideally be interpretable independently.\n* Comparative Performance: The figure suggests that RS might achieve better performance than UBER with smaller buffer sizes. Could you clarify if this interpretation is correct?\n\n### Figure 6\n* Comparison with RS: While the legend is clear, the benefits of UBER over RS are not apparent in this figure. Additional context on UBER’s comparative advantages would strengthen the analysis.\n\n### Figure 7\n* RS vs. UBER Performance: Why does RS outperform UBER in the EASY -> HARD setting? Furthermore, why was RS’s buffer not scaled up to match UBER’s buffer size for a direct comparison?\n\n## Minor comments\n* Punctuation: The following sentence on page 10 is missing punctuation: “UBER achieved optimal performance with 1700 examples which is 23% less examples than A-FIFO and just a 4% of the total examples collected by PETs.” A comma after “examples” would improve readability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the challenge of managing replay buffer growth in continual model-based reinforcement learning (RL). The authors introduce \"Uncertainty-Based Experience Replay\" (UBER), a method that leverages the BICHO algorithm to measure model accuracy. UBER monitors the discrepancy between predicted and actual rewards in transitions to filter stored experiences effectively. By admitting only uncertain samples, UBER reduces memory use and mitigates catastrophic forgetting, enhancing efficiency in long task sequences while maintaining performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Improve Figure Legends and Color Scheme:\n\n* Update all figures with clear legends and consistent symbols to ensure each plot is independently interpretable. For example, symbols such as the black cross in Figure 5 should be explicitly defined in the legend.\n* Use a color scheme that is color-blind accessible for better readability and accessibility. Also enhance consistency across figures.\n\n2. Experiment Settings Demonstrating Clear Superiority of UBER over RS:\n* Currently, the experimental evidence supporting UBER’s superiority over RS is limited to the CartPole (CP) results in Figure 3 and the very low data regime in Figure 7.\n* I would like to see at least one additional experiment where UBER consistently outperforms RS, particularly in settings relevant to the paper’s contributions. For instance, it would be valuable to test UBER’s effectiveness in scenarios where RS might be limited by buffer size constraints or task complexity.\n\n3. Pixel-Based Experiment:\n* If feasible, add an experiment in a pixel-based setting to illustrate UBER’s advantage in managing replay buffer size. In rich visual environments, buffer constraints become more impactful, and UBER’s performance in this context would highlight its relevance to more complex, real-world applications where retaining all observations is prohibitively costly.", "broader_impact_concerns": "I do not have broader impact concerns. This work is theoretical in nature.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EG6F9ELpHh", "reviewer": "Reviewer_krR9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Simple and Intuitive: The core idea behind UBER is simple and intuitive: prioritise experiences that the model finds surprising. This simplicity makes it easy to understand and potentially integrate into existing MBRL algorithms.\n- UBER addresses a relevant issue in the continual learning domain. It isn't feasible or desirable to store all experience when training an agent on a sequence of tasks, and research in filtering out redundant data is of practical importance.\n- The empirical results demonstrate that UBER effectively manages the buffer size, keeping it smaller than baseline methods while maintaining performance. This efficiency is crucial for long-running MBRL agents and continual learning scenarios.\n- The discard experience experiment is a good demonstration of the relevance of the their setting.\nMitigation of Forgetting: The experiments suggest that UBER can help mitigate catastrophic forgetting in continual learning, although further investigation in more complex continual learning (CL) settings is needed. \n\n**Weaknesses**\n- Limited Scope of CL Evaluation: The continual learning experiments are limited to only two environments. Evaluating UBER on more complex and diverse CL benchmarks would significantly strengthen the validity of the claims \n- In some environments, RS seems to be better than UBER at achieving higher/equivalent performance with a smaller buffer size (Fig 3 & 5). What would the benefits of UBER be over RS? \n- Some of implementation choices are unclear, please see the requested changes section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Uncertainty-Based Experience Replay (UBER), for managing the replay buffer in model based RL. UBER leverages the model's uncertainty in its predictions to selectively add experiences to the buffer. The intuition is that experiences about which the model is uncertain are more informative and therefore more valuable for learning and avoiding forgetting. If the model can confidently predict the outcome of an action, the corresponding experience is considered redundant and not added to the buffer. This filtering mechanism allows UBER to maintain a smaller, more relevant buffer, potentially leading to faster training, reduced memory usage, and less forgetting. The authors evaluate UBER in both single-task and task-agnostic continual learning scenarios across several environments, including racing-inspired domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be useful to evaluate UBER on more multi task environments such as MetaWorld. This is the most important change that I think is needed for acceptance. \n- Provide a comparison of UBER vs RS and why a practitioner might choose one over the other.\n- Section 4 aims to highlight why MBRL fails in CL. But 2/3 environments (CP and RE) used are single task environments, how are they relevant when motivating UBER for continual learning?\n- It might be easier to read the results in Fig 5 and 6 if they are presented as a table. It would also address the issue of values being outside the range of the y axis in those figures.\n\n**Changes for clarity**\n- In Algorithm 3 and 4: \n  - Given that $a_t^*$ is discarded from the action sequence in L15, I don't think it is correct to index it by $t+i$ in L1 of Algorithm 4. $s_t$ would be the current state and you want to evaluate the log probs for the remaining actions in the sequence. Otherwise L15 should be removed from Alg 3\n  - Furthermore, the projected reward $p_r^'$ in Alg 4 should be from $t$ since it is from the current time step and is recalculated every step. Only $p_r^*$ should be from $t+i+1$\n  - The maximum prediction distance (MPD) is never mentioned in the algorithm. Shouldn't the prediction of actions and $p_r^*$ be only for MPD and not H? This is especially relevant since MPD=1 for 3/4 environments tested.\n- The Appendix lists the value of LA for all environments as 1. This means that the uncertainty is only measured for the next step. Isn't it too short sighted for measuring the uncertainty of the model? Could the authors elaborate on why this value was chosen? I would like the value of LA be mentioned in the main text itself along with an explanation of the rationale.", "broader_impact_concerns": "I don't think a Broader Impact Statement is necessary.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Cf5cyElWUP", "reviewer": "Reviewer_d2yw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* Efficient Memory Management: UBER focuses on storing only the most informative experiences by filtering based on predictive uncertainty, which can significantly reduce memory requirements. This is especially advantageous in continual learning, where memory constraints are often a bottleneck.\n\nMitigation of Catastrophic Forgetting: By prioritizing uncertain experiences, UBER helps to reinforce memories for tasks that might otherwise be forgotten over time, potentially improving long-term performance in sequential tasks. Although, the novelty surrounding might require few attributions - eg. PER [1], IER. [2], and few others.\n\n[1] https://arxiv.org/abs/1511.05952\n[2] https://arxiv.org/abs/2206.03171\n\n* Task-Agnostic Design: UBER’s filtering mechanism does not rely on explicit task boundaries, which makes it adaptable to real-world scenarios where task shifts are gradual and not clearly defined. This aligns well with the objective of task-agnostic continual learning.\n\n**Weaknesses:**\n\n* Reliance on Uncertainty Estimation: Calculating predictive uncertainty accurately in complex environments can be computationally intensive. If the uncertainty estimation is unreliable, UBER could either store redundant experiences or discard valuable ones, impacting performance. Furthermore, the costs of these uncertainty estimations are unclear. For instance, it might be the case that this measure might be more simpler environments, and the costs in atari environments might hinder the speedup. Further exploration here would help better understand when this would be useful.\n\n* Potential Overfitting: Since UBER filters experiences aggressively, there is a risk of underfitting if critical, less uncertain experiences are consistently filtered out. This could lead to over-specialization in certain areas of the task space. Could this not be a problem, when we are attempting a multi-task RL problem through CL?\n\n* Generalizability and Scalability Concerns: UBER’s performance has yet to be demonstrated in high-dimensional or vision-based reinforcement learning tasks, where feature space complexity could challenge the efficacy of uncertainty-based filtering. I would suggest Atari or gridworld, MT50, or other larger benchmarks to help prove the scalability aspect of UBER.\n\n* Limited Exploration: By focusing mainly on uncertain experiences, UBER might reduce exploration over time, potentially neglecting more stable experiences that could reinforce the policy and benefit long-term learning. Although such works of prioritized sampling have been presented earlier, and the exact difference other than the selective thresholding aspect is not clear. Please note, that this by itself is a useful contribution by itself, but how one manages the earlier weaknesses mentioned above is not exactly clear to me and your response would be helpful.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper, \"Uncertainty-Based Experience Replay for Task-Agnostic Continual Reinforcement Learning,\" introduces Uncertainty-Based Experience Replay (UBER) to manage experience retention in model-based reinforcement learning (MBRL) for continual learning (CL). UBER selectively stores experiences based on predictive uncertainty, aiming to minimize buffer size while retaining essential information for sequential task learning. This method targets catastrophic forgetting by dynamically filtering experiences before adding them to the replay buffer, which contrasts with traditional uniform sampling.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is an exploratory step in confronting one of many challenges raised in the continual learning setting. There is certainly an audience for these findings and I think the central claim that UBER can selectively retain relevant data when the environment is non-stationary and the length of interaction is indeterminate is sufficiently supported. Many possible clarity revisions were suggested during the review process and I encourage the authors to take them into account in their revision. The area that I would particularly like to look over once more before publication is the discussion of the comparison between UBER and RS; as-is the plots raise a number of questions in an attentive reader and it's important that the paper take a measured, well-supported approach to situating UBER in the context of these existing methods.
Decision: Accept (poster)