Memory of Unimaginable Outcomes in Experience Replay
Authors: Adrian Remonda
Cole Corbitt Terrell
Eduardo E. Veas
Venue: Submitted to ICLR 2023 (2023)
Keywords: Transfer Multitask and Meta-learning
Robotics
Model-Based Reinforcement Learning
Batch/Offline RL
Deep RL
Continuous Action RL

Abstract: Model-based reinforcement learning (MBRL) applies a single-shot dynamics model to imagined actions to select those with best expected outcome. The dynamics model is an unfaithful representation of the environment physics, and its capacity to predict the outcome of a future action varies as it is trained iteratively. An experience replay buffer collects the outcomes of all actions executed in the environment and is used to iteratively train the dynamics model. With growing experience, it is expected that the model becomes more accurate at predicting the outcome and expected reward of imagined actions. However, training times and memory requirements drastically increase with the growing collection of experiences.  
Indeed, it would be preferable to retain only those experiences that could not be anticipated by the model while interacting with the environment.  
We argue that doing so results in a lean replay buffer with diverse experiences that correspond directly to the model's predictive weaknesses at a given point in time. 
We propose strategies for: i) determining reliable predictions of the dynamics model with respect to the imagined actions, ii) retaining only the unimaginable experiences in the replay buffer, and iii) training further only when sufficient novel experience has been acquired. 
We show that these contributions lead to lower training times, drastic reduction of the replay buffer size, fewer updates to the dynamics model and reduction of catastrophic forgetting. All of which enable the effective implementation of continual-learning agents using MBRL.

Under review as a conference paper at ICLR 2023
MEMORY OF UNIMAGINABLE OUTCOMES IN EXPERI- ENCE REPLAY
Anonymous authors Paper under double-blind review
ABSTRACT
Model-based reinforcement learning (MBRL) applies single-shot dynamics model imagined actions to select those with best expected outcome. The dy- namics model is an unfaithful representation of the environment physics, and its capacity to predict the outcome of future action varies as it trained iteratively. An experience replay buffer collects the outcomes of all actions executed in the environment and is used iteratively train the dynamics model. With growing experience, it is expected that the model becomes more accurate at predicting the outcome and expected reward of imagined actions. However, training times and memory requirements drastically increase with the growing collection of experi- ences. Indeed, it would be preferable to retain only those experiences that could not be anticipated by the model while interacting with the environment. We argue that doing so results in a lean replay buffer with diverse experiences that corre- spond directly the model's predictive weaknesses at given point in time.
2
6
8
10
11
12
13
14
We propose strategies for: i) determining reliable predictions of the dynamics model with respect to the imagined actions, ii) retaining only the unimaginable experiences in the replay buffer, and iii) training further only when sufficient novel experience has been acquired. We show that these contributions lead to lower training times, drastic reduction of the replay buffer size, fewer updates to the dynamics model and reduction of catastrophic forgetting. All of which enable the effective implementation of continual-learning agents using MBRL.
16
18
20
21
INTRODUCTION
22
Model-Based Reinforcement Learning (MBRL) is attractive because it tends have lower sample complexity compared to model-free algorithms like Soft Actor Critic (SAC) (Haamoja et al. (2118) MBRL agents function by building a model of the environment order to predict trajectories fu- ture states based off of imagined actions. An MBRL agent maintains observations, its actions in response to observations, the resulting reward, and new observation in an experience replay buffer. The information stored in the replay buffer is used to train a single-shot dynamics model that iteratively predicts the outcomes of imagined actions into a trajectory of future states. At each time step, the agent exccutes only the first action in the trajectory. and then the model re- re-imagines new trajectory given the result of this action (Nagabandi et al. (2018). Yet, many real-world tasks consist in sequences of subtasks of arbitrary length accruing repetitive experiences, for example driving over a long straight and then taking a corner. Capturing the complete dynamics here requires longer sessions of continual learning. (Xie & Finn )202T)
23
24
25
extensive history of its
26
27
28
30
32
33
34
Optimization of the experience replay methodology is an open problem. Choice of size and mainte- nance strategy for the replay buffer both have considerable impact on asymptotic performance and training stability (Zhang & Sutton (2017)). From a resource perspective, the size and maintenance strategy of the replay buffer pose major concerns for longer learning sessions.
35
38
37
38
The issue of overfitting is also a concern when accumulating similar or repetitive states. The buffer
39 can become inundated with redundant information while consequently under-representing other im- portant states. Indefinite training remember, less common states. Conversely, too small a buffer will be unlikely to retain sufficient relevant experience into the future. Ideally, a buffer's size would be the exact size needed cap-
redundant data can result in an inability to generalize to, or
41
43 ture sufficient detail for all relevant states (Zhang & Sutton (2017). Note that knowing priori all relevant states unfeasible without extensive exploration.

---

72 ness in standard NN, becomes crucial. Ensembles of probabilistic NNs proved a good alternative to Bayesian NNs in determining predictive uncertainty (Lakshminarayanan et al. (2016)). Further- Under review as a conference paper at ICLR 2023
45 We argue that these problems can be subverted by employing strategy that avoids retaining expe- riences that the model already has sufficiently mastered. Humans seem to perform known actions almost unconsciously (e.g.. walking) but they reflect actions that lead to unanticipated events (e.g. walking over seemingly solid ice and falling through). Such is our inspiration to attempt to curate the replay buffer based on whether the experiences are predictable for the model.
46
48
49
50
Through this work, we propose techniques to capture both common and sporadic experiences with sufficient detail for prediction in longer learning sessions. The approach comprises strategies for: i) determining reliable predictions of the dynamics model with respect to the imagined actions, ii) the replay buffer, iii) training further only when
51
52
53
retaining only the unimaginable experiences sufficient novel experience has been acquired, and iv) reducing the effects of catastrophic forget- ting. These strategies enable model to self-manage both its buffer size and its decisions to train, drastically reducing the wall-time needed to converge. These are critical improvements toward the implementation of effective and stable continual-learninn agents.
54
55
56
57
58
Our contributions can be summarized as follows: i) contributions towards the applicability of MBRL in continual learning settings, ii) a method to keep the replay buffer size to a minimum without sacrificing performance, iii) method that reduces the training time. These contributions result in keeping only useful information in a balanced replay buffer even during longer learning sessions.
59
60
81
RELATED WORK
82
Compared to MFRL, MBRL tends to be more sample-efficient (Deisenroth et al. (2013) at a cost of reduced performance. Recent work by Nagabandi et al. (2018) showed that neural networks effi ciently reduce sample complexity for problems with high- nigh-dimensional non non-linear dynamics. MBRL approaches need to induce potential actions which will be evaluated with a dynamics model to choose those with best reward. Random shooting methods artificially generate large number of ac- tions (Rao (2010)) and model predictive control (MPC) can be used select actions (Camacho et al.) 2004 Neural networks (NNs) are suitable alternative to families of equations used to model the environment dynamics in MBRL (Williams et al. (2017)) But, overconfident incorrect predictions, 71 which are common in DNNs, can be harmful. Thus, quantifying predictive uncertainty. weak-
68
64
85
66
67
68
70
74 more, an extensive analysis about the types of model that better estimate uncertainty in the MBRL. setting favored ensembles of probabilistic NNs (Chua et al.] (2018)). The authors identified two types of uncertainty: aleatoric (inherent to the process) and epistemic (resulting from datasets with 77 too few data points). Combining uncertainty aware probabilistic ensembles in the trajectory sam- pling of the MPC with a cross entropy controller the authors demonstrated asymptotic performance 79 comparable SAC but with sample efficient convergence. The MPC, however, is still computation- ally expensive (Chua et al. (2018) 8); Zhu et al. (2020). Quantification of predictive uncertainty serves 81 as a notion of confidence an imagined trajectory. Remonda et al. (202T). used this concept to pre- vent unnecessary recalculation, effectively using sequences of actions the model is confident in and reducing computations. Our approach also seeks to determine reliable predictions of the dynamics model with respect to the imagined actions, but basis to manage growth of the experience replay. 85 Use of Experience Replay in MBRL: While an uncertainty aware dynamics model helps to mit- igate the risks of prediction overconfidence, other challenges remain. Another considerable issue when training an MBRL agent is the shifting of the state distribution as the model trains. Experi- ence replay was introduced by Lin (1992). and has been further improved upon. Typically in RL, transitions are sampled uniformly from the replay buffer at each step. Prioritized experience replay (PER) (Schaul et al. (2016)) attempts to make learning more efficient by sampling more frequently transitions that are more relevant for learning. PER improves how the model samples experiences from the already-fille filled replay buffer, but it does not address how the replay buffer is filled in the first place. In addition, neither work addresses the importance of the size of the replay buffer as a hyperparameter (Zhang& & Sutton (2017)). Our method attempts to balance the replay buffer by only adding experiences that should improve the future prediction capacity and keeps the training time bounded to a minimum.
75
78
78
83
84
87
88
89
90
91
92
93
94
96
Task Agnostic Continual Learning: The context of our work originates in tasks consisting in com- binations of possibly repetitive subtasks of arbitrary length. In the terminology Normandin et al. we aim for continuous task-agnostic continual reinforcement learning. Meaning that the
98
88
2021),
2

---

Under review as a conference paper at ICLR 2023
task boundaries are not observed and transitions may occur gradually (Zeno et al. (2021) l). In our case, the task latent variable is not observed and the model has no explicit information about task transitions. In such context, continual learner can be seen
101
102
autonomous agent learning over an endless stream of tasks, where the agent has to: i) continually adapt in a non-stationary environment, ii) retain memories which are useful, iii) manage compute and memory resources over long period of time ( [Khetarpal et al. (2020). Thrun (1994)). Our proposed strategies satisfy these requirements. Matüisen et al. (2020) address the issue of retaining useful memories in curriculum learning set- ting by training a "teacher" function that mandates a learning and re-learning schedule for the agent assuming that the agent will not frequently revisit old experiences/states and will eventually forget them. Ammar et al. (2015) focus on agents that acquire knowledge incrementally by learning mul- tiple tasks consecutively over their lifetime. Their approach rapidly learns high performance safe control policies based on previously learned knowledge and safety constraints on each task, accu- mulating knowledge over multiple consecutive tasks to optimize overall performance. [ou Ammar & Taylor (2014) developed lifelong learner for policy gradient RL. Instead of learning control policy for a task from scratch, they leverage on the agent's 's previously learned knowledge. Knowl- edge is shared via latent basis that captures reusable components of the learned policies. The latent basis is then updated with newly acquired knowledge. This resulted in an accelerated learning of new task and an improvement in the performance of existing models without retraining on their re- spective tasks. With our method, we imbue the RL agent with the ability to self-er eefleevaluate and decide in real- time if it has sufficiently learned the current state. Unlike the method presented by Matiisen| our method requires no additional networks to be trained in parallel.
103
104
105
108
107
108
109
110
112
114
115
116
117
118
119
120
et al. 20200)
Xie & Finn 202 identified two core challenges in the lifelong learning setting: enabling forward 121
122
transfer, 1.e. reusing knowledge from previous tasks to improve learning new tasks, and to improve backward transfer which can be seen as avoiding catastrophic forgetting (Kirkpatrick et al. (2017). They developed method that exploits data collected from previous tasks to cumulatively grow the agent's skill-set using importance sampling. Their method requires the agent to know when the task changes whereas method does not have this constrain. Additionally, they focus in forward transfer only. Our method addresses both forward and backward transfer.
123
125
126
127
128
PRELIMINARIES
129
At each time t, the agent is at a state St E S. executes an action a E A and receives from the environment reward r. r(se; ae) and state 81+1 according some environment transi- tion function f S x A + S. RI consists in training a policy towards maximizing the accu- mulated reward obtained from the environment. The goal is to maximize the sum of discounted rewards Ee (6t)))((ss a:), where y E J0, 1]. Instead, given a current state St, MBRL artifi- cially generates a huge amount of potential future actions, for instance using random shooting ( (Rao) 2010) cross entropy( [Chaa al. (2018)). Clarification of these methods is beyond the scope of this paper; we defer the interested reader to the bibliography. MBRL attempts to learn a dis- crete time dynamics model ff = (st, ax) to predict the future state $Lt., of executing action aa at state St. To reach state into the future, the dynamics model iteratively evaluates sequences of actions, OLa+H = (or...., Gx+H -1) over longer horizon H, to maximize their discounted reward
130
131
132
139
134
135
136
137
138
139
H (1-0) az). These sequences of actions with predicted outcomes are called imagined trajectories. The dynamics model f is an inaccurate representation of the transition function f and the future is only partially observable. So, the controller executes only single action a in the tra- jectory before solving the optimization again with the updated state 8t+1- The process is formalized in Algorithm| I The dynamics model fo is learned with data Dene. collected on the fly. With for the simulator starts and the controller is called to plan the best trajectory resulting in o Only the first action of the trajectory ot is executed in the environment and the rest is discarded. This is repeated for Task Hoizznn number of steps. The data collected from the environment is added to Dene and fe is trained further. The process repeats for NN Ittertions. Note that generating imag- ined trajectories requires subsequent calls to the dynamics model to chain predicted future states Stnn with future actions up to the task horizon, and so it is only partially parallelizable.
140
141
142
143
144
145
146
147
148
149
150
Dynamies model. We use probabilistic model to model probability distribution of next state given current state and an action. To be specific, we use a regression model realized using a neural network similar to Lakshminarayanan al. (016)) and Chua et al. (018).. The last layer of the
152
153
3

---

Under review as a conference paper at ICLR 2023
model outputs parameters of Gaussian distribution that models the aleatoric uncertainty (the un- certainty due to the randomness of the environment). Its parameters are learned together with the parameters of the neural network. To model the epistemic uncertainty (the uncertainty of the dy- namics model due to generalization errors), we use ensembles with bagging where the members of the ensemble are identical and only differ in the initial weight values. Each element of the ensemble has as input the current state 9t and action de and is trained to predict the difference between and 11 instead of directly predicting the next step. Thus the learning objective for the dynamics model becomes, As = 81+1 81- Lo outputs the probability distribution of the future state Ps(t+1) from which we can sample the future step and its confidence 8, 8, [a(s, Where the confidence 8. captures both, epistemic and aleatoric uncertainty.
155
157
158
159
180
161
162
163
Trajectory Generation. Each ensemble element outputs the parameters of a normal distribution. To generate trajectories, P particles are created from the current state, sy = 8t, which are then propagated by: s st+1 fu(s), at). using particular bootstrap ele- ment {I..... ..., B). Chua et al experimented with diverse methods to propagate particles through the ensemble. The TS. method delivered the best re- sults. It refers particles never changing the initial bootstrap element. Doing so, results in having both uncertainties separated at the end of the trajectory- Specifically, aleatoric state variance is the average variance of particles of same bootstrap, whilst epis- temic state variance is the variance of the average of particles of same bootstrap indexes. We use also TS
165
Algorithm 1 MBRL
167
Init D with one iteration of a random controller for Iteration i 1 to N NIlerations do Train f given D for Time t = to ToskHorizon do from ComputeOptimalT rajjmmmorrrjecoyy((... f)
168
189
170
171
Get
diat
172
173
Execute o; from optimal actions d H D
Record outcome:
175
(Se. 8141
176
177
178
179
To select the best course of action leading to 8H. MBRL generates a large number of trajectories K and evaluates them in terms of reward. To find the actions that maximize reward, we used the cross entropy method (CEM) Botev et al. (2013), an algorithm for solving optimization problems based on cross- entropy minimization. CEM gradually changes the sampling distribution of the random search so that the rare-event is more likely to occur and estimates sequence of sampling distributions that converges to a distribution with probability mass concentrated in a region of near- optimal solutions. Appendix A details the use of CEM to get the optimal sequence of actions ".:4
Control.
180
181
182
183
184
4 TOWARDS CONTINUAL LEARNING
186
187
Applying MBRL. to a continual learning setting is a promising venue for research. The dynam- ies model could be constantly improving and adapting dynamically to changes in the environment. Many real-world tasks can be broken in sequences of subtasks of arbitrary length. Capturing the complete dynamics then requires exposure to longer sessions of continual learning. Arbitrarily long repetitive tasks lead to increasing redundancy in the experience replay constantly increasing of the amount of experience collected. These issues hinder the use of MRBL in continual learning settings.
188
189
190
192
What to add to the replay buffer: We posit that it would be preferable to retain only those experi- ences that could not be adequately anticipated by the model during each episode in the environment. Essentially, we would only like to add to the replay buffer observations for which the model issued a poor prediction. On the contrary, we would like to avoid filling the replay buffer or updating the model observations that the model is good at predicting. We contend that these two elements will lead eventually to a balanced replay buffer, which will contain only relevant observations. This will contribute the objective of continual learning.
194
196
198
199
5
UARF: UNCERTAINTY AWARE REPLAY FILTERING
200
Continual learning requires the MBRL agent to adapt in a non-stationary environment, retaining memories that are useful whilst avoiding catastrophic forgetting, and it can manage compute and memory resources over long period of time KKetarpal al. (020)). The proposed method, UARF, addresses these issues with a variety of strategies. Algorithm 2 is the main algorithm used to select which observations to append in the replay buffer. The optimal actions d+++ are computed
202
204
205
4

---

Under review as a conference paper at ICLR 2023
function (See Appendix A) given the current state of the environment St and f. The future trajectory and its uncertainty, Pr(++ is then obtained by
by the ComputeOptiiallIqqae
207
using Le4+H and 8 with f. The variable unreliable.M Model is set to true when the algorithm believes the imagined trajectory not to be trustworthy. Depending on its value, calculation of new trajectories and additions to the replay buffer will be avoided and therefore computation time and size of the replay buffer will be reduced. If wnreliable odel is False, the next predicted action is executed in the environment. Subsequent actions from dit++ Hr are executed until the unr unreliable. odel flag is set to False or the environment reaches T faskHorizon number of steps. The process is repeated for the maximum iterations allowed for the task. After the first action, every time an action a ++11++H
208
209
210
211
212
213
214
215
is executed trajectory computation is avoided and this new observation is not added to the replay buffer on the basis that the model can already predict its outcome. If unreliable. odel is True, the algorithm calculates a new trajectory and adds the current observation to the replay buffer. Hereby, the buffer stores only observations for which the model could not predict (imagine) the outcome.
216
218
Trustworthy imagination (Algorithm 2 L:18-21). The algorithm that assigns value to uar odel is named BICHO. BICHO will essentially return True as long as the reward projected in the future does not differ significantly with respect to the imagined future reward pt and the confidence of the model remains high. BICHO is built assuming that if parts of the trajec- tory do not vary, their projected reward will be as imagined by the model with some confidence. After calculating a trajectory, the distribution of rewards p, is calculated for H steps in the future. Whereas, at each step of the environment, independent if the recalculation was skipped or not, a new trajectory p', of H steps is projected, starting from state 9, which is given by the environment and using actions a in the imagined trajectory. We use the Wasserstein distance (Bellemare et al. 2017)) to find how much these two distributions change after each time step in the environment. the change > 3 (which is a hyper parameter to tune) then mnreliable odel True. We can control how many steps ahead we would like to compare the two distributions. The comparison is done for just c steps (< H). which is hyper parameter to tune. If they differ significantly, then the trajectory is unreliable. That is, if the projected reward differs from the imagined one the outcome
220
222
223
224
225
226
227
228
229
231
232
233
of the actions is uncertain and the trajectory should be recalculated.
Even for model that has converged, predicting trajectories of great length is impossible. Recaleu- lations inevitably occur at the end of trajectories. Such recalculations do not necessarily represent the appearance of unseen information, but rather a limitation of the successful model in a complex environment. Hence, we would not want to add them to the buffer. The maximum prediction dis- tance (MPD) defines a cutoff for a trajectory, and adjusts the strictness of the filtering mechanism. Refer to Appendix E for an extensive analysis.
235
236
237
238
239
Updates on novel information (Algorithm (::24225)) over-training the dynamics model leads to instabilities due to overfitting. This problem is exacerbated when the replay buffer contains just the minimum essential data. If we only filter the replay buffer, continuously updating the parameters of the dynamics model will eventually lead to overfitting. Instead, our method updates the parameters of the dynamics model only when there is sufficient new information in the replay buffer. We train the dynamics model only when new data exceeds the new waatl_laaaeeelleeeee
240
241
242
244
hyper parameter. For 246 our experiments we set this variable to 0.01 training only when 1% of the experiences in the replay buffer are new since the last update of the parameters of the dynamics model.
6 EXPERIMENTS
249
The primary purpose of the proposed algorithm is for the resulting replay buffer to retain only relevant, non-redundant, experiences that will be useful for learning the task. We envision applying this method to tasks that require longer training sessions and in continual learning settings.
250
251
252
We designed three experimental procedures. The first experiment seeks to establish that our method indeed retains a reduced buffer sufficient for achieving expected rewards when learning a single task throughout long training sessions. To this end, we evaluate the proposed method in benchmark environments for higher number of episodes than in Chua et al. (2018). The second experiment seeks to prove that UARF retains a small number of complementary experiences compared to non- filtering baseline algorithms when training on sequence of different but related tasks in a continual
253
254
256
258
leamnigg setting. We evaluate our method in a combined task including unseen subtasks. The third experiment seeks to show how UARF addresses the effects of catastrophic forgetting.
259
5

---

Under review as a conference paper at ICLR 2023
Algorithm 2 UARF
1: Initialize dynamics model f parameters; Initialize replay buffer 'D with an iteration of random controller
2: unreliable odel True and trainModee False
for Iteration
N I'eerationss do tree then Train given D for Time = to TaskHorizznn do if MareliableMddel then
5:
6:
7:
Get 07.4. from Compute@ppimallIaajjee Get (-(1+1+++) given (st, f. G444H) Use f to predict H rewards ahead 1=
(st,f)
8:
10:
else i
11:
Get first action de from available optimal actions d
12:
Execute in the environment a, to obtain 8++1 and Discard first action and keep the rest 0++1:+H
13:
14:
Get Pr(+++1:1++ HI) given (st, f, a(+1++))
15:
Trustworthy imagination
16:
L= min(H, c i) Calculate the number steps ahead to consider
17:
r_error W aassessseel stein Distance (Prtss+11: 1:+i+ 22 (e:. if rerrrrr > B then unreliableModel TRUE else unreliableModel = False
18:
19:
if unreelable Model then
20:
Record outcome: < (st, az, 9
21:
I/ Updates on novel information
22:
if new dala_in nu D >> new new.dalaa_aee Ahresholl length((D) then
23:
trainModk odel True
2.2
220
EShosde EdlotteBooed
3.50
Expereenco
000
Replly
1.00
Ephoce
Epl
Fyseode
better doe [165)
Experiences 4dceed
130
-200
0.10
EASELAE
100
Extodd
-200
0.00
Epioods
Eplocce
Fyisode
Thm par epaecie D4
feee Det3
Added
1.0
BACOL AC
0.30
500
0.00
335
0.00
Eplooda
Es0oce
Eptocdo
Epoost Revond (40)
LACS
hewee
Mes
20
42
68
82
123
60
80
100
Eplscde
Es
Figure 1: Performance of algorithms (BL: green, BICHO: red, UARF: blue) in (top to bottom) Cartpole, Pusher, Reacher, and Masspoint sectorl. From left to right column: episode reward, time episode (s), cumulative number of observations stored in the replay buffer, new experiences added to the buffer per episode. The rightmost plots illustrate with dashed vertical lines episodes that resulted in UARF updating its model parameters.
280
6.1
EI- CONTINUING TO LEARN A TASK AFTER CONVERGENCE
281 This experiment intended to show that our method retains sufficient experience to solve the task while curtailing buffer growth and unnecessary model updates. We intend to prove that this results in

---

Under review as a conference paper at ICLR 2023
dramatic reduction in the replay buffer size (which is free of any artifcialll-imppoee limits) while retaining strong performance (per-episode reward) and reducing per- per-episode wall clock run-time.
264
We use the MuJoCo (Todorov et al. (0112)
286
physics engine and environments Cartpole (CP), Pusher (PU) and Reacher (RE) with task length H)) and trajectory horizon (H)
Socart:: Eiiodd
Sootrtt Epicdd
Sectart Epioode 66
268
Rowand
12-
269
chosen for valid comparison with [Chua et al.
270
2018). With similar training scenarios, Re-
200
230
271
monda et al. (202T) trained CP for 30 episodes, PU and RE for 150. Instead, we trained each for 100 episodes. We also included a modified version of the Masspoint environment (Thaan-- jeyan et al. (2020) (also used in E2). Mass- point is navigation task in which point mass navigates to given goal. It a 5-dimensional (r, y: Ug. Uy. p) state domain. Where (æ, y) is the position of the agent, Uy)
Step
272
Rowort
Camaioaiin Swg.uts Faller
273
Figure 2: Per-step reward and cumulative steps added to the replay buffer for episodes 1 (left), 4 (middle), and 99 (right) in the Masspoint Sector I maneuver. These plots show that UARF adds fewer redundant experiences to the replay buffer as the model converges.
274
275
277
279
) its speed, and P is the distance between the agent and the clos-
280
281
est point to a given path. The agent can exert force in cardinal directions and experiences drag coef- ficient di. We use & 0.6 and included noise in the starting position. We modified the goal of the agent so that it must move as fast as possible without deviating from a given path. Each task and its complexity is then determined by the geometry of the path to followed. The reward is calculated as r = V(1 - pl). Where V is the speed of the agent and و the distance to the task's path. This ex experiment used sectorl (Figure in Appendix B) and Hyperparameters shown in Appendix F.
282
283
284
285
286
287 We assess performance in terms of per-episode reward, per-episode wall time, and replay buffer size. We evaluate three algorithms: baseline (BI.) is conventional MBRL (PETS Chua et al. (2018). BICHO uses functionality to avoid unnecessary computation, and UARF. BICHO and UARF used the same values of 3 and look-ahead, estimated empirically to produce a reasonable balance in terms of per-episode reward and percentage of recalculation. All experiments use random seeds and randomized initial conditions for each run, and ran in workstations with Nvidia 3080TI GPUs.
286
290
292
Fig | top shows the results obtained in CP. Fig D mid-right shows the size of the replay buffer during training. We observe that while the replay buffer keeps grows in the case of BL and BICHO, the size of the buffer derived from UARF is comparably flat: the buffer resulting from
Results:
294
285
296
UARF is 10x smaller. The training time per episode (Fig mid-left) remains nearly constant and
lower for UARE. BL. takes substantially longer than both BICHO and UARF to complete episode. The wall time of both the BL and BICHO exhibit linear growth. It takes longer to update the model as the replay buffer grows linearly. Fig left shows comparable reward per episode for all methods. Results in Fig for PU (row 2), RE (row 3) and Masspoint (bottom) are consistent with those of CP. illlstrates the management of buffer growth in Masspoint by showing exactly at which steps experiences are added to the replay buffer during untrained (EI) and trained (E399) episodes. These plots reveal that when the model untrained, many experiences are added to the buffer throughout the episode. After the model is trained (E99), UARF stops adding experiences to the buffer as the model is able to predict them. Hence, new experiences are deemed redundant and not useful to the model. Results support our claim that UARF obtains drastically smaller replay buffer that is intelligently populated with only relevant information. This is achieved while maintaining strong performance in the environment compared to BL. Note that while the curves are plotted per episode, it is misleading to assume that all methods converge roughly at the same time. The time per episode in the case of UARF is at least half that of BL for approximately in every environment and it remains stable, while it increases linearly for BL and BICHO with increasing buffer size. The total wall time in average for CP was BL=1 1.hh,, BICHO=0.59h and UARF=0.57h. For PU, it was BL=0.97h, BICHO=0.73h and UARF=O.CC -0.6... For RE, it was BL=1.99h BICHO=1.55h and UARF=1.455 and for MP was BI. 15h, BICHO= 1.94h and UARF=1.68h.
298
299
300
301
Fig
2
303
304
305
307
308
309
310
311
312
313
314
315
6.2
Ex 2. CONTINUAL LEARNING EXPERIMENT
316
This experiment is set in Task Agnostic Continual Reinforcement Learning (the model is not aware of tasks or task transitions). In this setting, we sought prove that UARF maintains leaner and
317
7

---

Under review as a conference paper at ICLR 2023
Ediodd Roward
15
BID
Buteer (1e))
DACEUNDOION
Unde Toins
10
50
RIC-D GASELINE
0.0
Ppiende
Eplende
Eplonde
Echodd
Figure 3: Training performance of Baseline, BICHO and UARFin Task Agnostic Continual Learn- ing. Models trained on seven maneuvers: cornerl, cornerl inverse, chicane, chicane inverse, cor- ner14, corner14 inverse, and straight. Vertical lines indicate a task switch. In the middle-right plot, cyan vertical lines also indicate when UARF triggers a model update. Full-track evaluation in the far-right plot; cumulative reward achieved each step on the full-track without further training.
318
more relevant collection of experiences in the replay buffer than do baseline algorithms. These char- acteristics of the proposed algorithm, posit, result in strong test performance with less data and greater stability. The existence of these characteristics can be verified by observing (after training) the size of the buffer, the number of experiences from each maneuver present in the buffer, and the performance of the models on the test task. We used the Masspoint racing environment, defining different simple tasks that can be composed to solve a complex, unseen one.
319
320
322
323
324
Each algorithm trains a model on a sequence of seven separate sub-tasks: two corners and their in- verses, a chicane and its inverse, and a straight (de- tails illustrated in Appendix B). The models retain their parameters and replay buffers between training on each task individually. After training on the last task, the methods are each tested on the full track, which contains some of the sub-tasks seen during training (colored in the full-track image, Appendix B) and tasks unseen during training (shown in black in the full-track image). The model must remember what it learned by training on each sub-task and ap- ply this knowledge to navigate more complex, un- seen task. All of the algorithms had virtually un- limited replay buffer size. Each model was trained for 30 episodes on each sub-task and then tested on the test task.
325
Diversity of Final Trained Buffer
328
Buffer Size [1e4]
327
328
1.0
329
0.5
331
0.0
BASELINE
BICHO
UARF
333
334
335
Figure 4: Distribution of experiences from each sub-task in the replay buffers of each al- gorithm immediately following training. De- tail shows a zoomed-in version for UARF.
336
337
338
339
340
Results Figure 3 shows episode reward, wall-time, buffer size during training, and new experiences added to the buffer per episode. Vertical lines illustrate task divisions. High episode reward indicates that each model adequately learns each subtask. UARF maintains almost constant wall-time, while BL and BICHO increase as experience accumulates. Buffer growth for BL and BICHO is linear, but UARF evidences asymptotic growth (13x smaller) adding new experiences at the end of training. Figure 4 shows the buffer growth of UARE. A larger amount of additions to the replay buffer occur while training the first tasks. Growth slows near halt during the last tasks. This is the case for example with the fourth task (chicane inverted). The previous task (chicane) is similar, and the information solve the previous task is enough that the algorithm does not require significant ssows the distribution of experiences from each sub-task present in each algorithm's 's replay buffer immediately following training. BL and BICHO employ a naive approach, resulting in replay buffers with distributions of experience determined exclusively by the length of the various maneuvers. The filtering mechanism of UARF results in a distribution of experience with some maneuvers having limited representation (e.g., the inverse maneuvers) This is because the UARF algorithm intelligently decides to omit redundant experiences from the buffer and leaves only the relevant ones. Figure 3 right shows that all three algorithms result in a model that adequately solves the test task. UARF continues to manage buffer growth while achieving high performance. The results support our initial hypothesis by illustrating clearly the proposed algorithm's 's propensity to maintain a smaller and more relevant replay buffer while achieving the performance of the baseline in a continual learning setting.
342
344
346
348
349
350
amount of new experience to solve chicane inverted.
351
352
353
354
355
357
359
360
8

---

Under review as a conference paper at ICLR 2023
6.3
Ex 3. CATASTROPHIC FORGETTING
382
Our approach helps to mitigate catastrophic forgetting. When using a fixed replay buffer size, it is important to ensure that the appropriate maximum buffer size is chosen (Zhang & Sutton] (2017).
363
364
If this value is undertuned, important experiences can be jettisoned, and catastrophic forgetting can occur. To illustrate how UARF helps to alleviate this risk, we ran the same experiment shown in section Ex.2 but with replay buffer of fixed size (5000 samples; roughly 4x the replay buffer size used by UARF in the unlimited size setting). comparess rewards achieved by each algorithm with both unlimited and fixed buffers. The models were validated on the full track and also on maneuver that was trained early on in the training process (cl inverse). Results reveal that with an undertuned fixed buffer size, BL loses about 10% performance both on the full track and on cl inverse. This is indicative of the fact that the non- non-filtrring algorithms are hitting the buffer size cap, throwing away valuable experiences, and forgetting how to properly solve maneuvers that were trained early on. This impacts performance on the full track as well.
385
386
367
368
369
371
Unlimited Buffer
Fixed Buffer Full Track
Fixed Buffer
Fixed Buffer cl inverse Post- Dossi-rainnng
Full Track
cl inverse First Pass
BASELINE UARF
22172
20235
1787
1561 1795
21975
22102
1781
Fixed Buffer Experiment. Results demonstrate susceptibility to catastrophic forgetting when not using UARF The BL forgets previous maneuvers after the FIFO mechanism of the fixed- size replay buffer eliminates experiences from them with an impact of about 10% in reward.
Table 1:
374
7
DISCUSSION AND CONCLUSION
375
The results in E1 reveal that continuing to run our algorithm in a repetitive environment with re- dundant or monotonous actions leads to, in some tasks, no increase in buffer and reduced dynamics model updates. This has the consequence of reduced running and training times, while reducing the effects of catastrophic forgetting and keeping the replay buffer size a bare minimum. In E2, a con- tinual learning setting, we demonstrated that using our approach leads outcomes with 1/25th of the experiences without performance degradation. UARF effectively deals with an unbounded growth of the replay buffer, which again reduces training time and instabilities. This effect is accentuated when training on a continual learning setting. UARF uses a buffer 43x smaller than the baseline.
378
377
378
379
380
382
383
The replay buffer is an instrument that makes the use of deep neural networks in RL more stable and it is an essential part in algorithms such as PETs. Such analyses of replay buffer are searce. But recently, research has turned to analyze the contents and strategies to manage the replay buffer of
384
386 RL agents Fedus et al. (220)), and also in supervised learning (Aljundi et al. (2019). We contribute to such body of work analyzing and offering strategies to manage growth of replay buffer in model
based RL. Having managed growth, there are several aspects we would like to turn to in the future: identifying task boundary from the novelty of experiences, ii) managing what forget for limited size buffers. iii) managing what to remember refresh when a change in task is evident. All this would allow to run agents for arbitrary time without having to deal with size of the buffer and would offer promising opportunities for deploying MBRL in a continual learning setting.
388
389
390
391
392
393
BICHO could be used to prioritize entries in the RB where the model was uncertain. Indeed, prior- itized buffer strategies support the usage of experience once it is in the buffer, but as the authors of the PER paper state, strategies for what to add and when (our work) are important open avenues for research. We did not explore our methods in environments where the tasks have interfering dynam- ics. But, if the dynamics change, poor predictions by the model will result in adding experiences to the replay buffer. What happens if interfering tasks occur permanently is an interesting follow up.
394
395
397
399 In summary, we proposed strategies that comply with requirements for continual learning. Our ap- proach retains only memories which are useful: it obtains lean and diverse replay buffers capturing both common and sporadic experiences with sufficient detail for prediction in longer learning ses- sions. Our approach manages compute and memory resources over longer periods: it deals with the unbounded growth of the replay buffer, its training time and instability due to catastrophic forgetting.
401
403
These results offer promising opportunities for deploying MBRL in continual learning setting.

---

Under review as a conference paper at ICLR 2023
8
REPRODUCIBILITY STATEMENT
406
To make our experiments reproducible, we provide the source code in the supplementary material. We include instructions describing how run all the experiments and to create the images. We in-
407
408
clude the source code of the proposed algorithms, the MassPoint environment and clear instructions showing how install extra packages and dependencies needed to reproduce our experiments.
409
410
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning, 2019.
412
Haitham Bou Ammar, Rasul Tutunov, and Eric Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret, 2015.
414
415
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International Conference Machine Learning, pp. 449 458. PMLR, 2017.
Zdravko Botev, Dirk P. Kroese, Reuven Y. Rubinstein, and Pierre I L''Ecyyr. Chapter 3 the cross-entropy method for optimization. In C.R. Rao and Venu Govindaraju (eds.), Handbook of Statistics, volume 31 of Handbook of Statistics, pp. 35 59. Elsevier, 2013, doi: https://doi. org/10. 011110010001998---4444---330 534-59944----99--9-0909939999999 8 000003-5. URL http:/ www www.sciieeee... rect. com/
418
419
420
421
422
Haitham Bou Ammar and Matthew Taylor. Online multi-task learning for policy gradient methods.
01 2014.
E.F. Camacho, C. Bordons, and C.B. Alba. Model Predictive Control. Advanced Textbooks in Control and Signal Processing. Springer London, 2004. ISBN 9781 7781353336943. URL https:
425
//books. goog atl.ee....boook.. d=Sc1 H3f
427
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models, 2018.
428
429
M. P. Deisenroth, G. Neumann, and J. Peters. 2013. doi: 10.1561/2300000021.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio. Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay, 2020.
431
432
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
434
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforce- ment learning: A review and perspectives, 2020.
435
436
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwingg et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-322 2017.
437
439
440
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles, 2016.
442
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine Learning, pp. 293-321, 1992.
443
Tambet Matisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learn- ing. IEEE Transactions on Neurul Networks and Learning Systems, 3109)3332-33740, 2020. doi: (00.9//NNNS..019.99999
445
Anusha Nagabandi, G. Kahn, Ronald S. Fearing, and S. Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning. 2018 IEEE International Conference Robotics and Automation (ICRA), pp. 7559-7566 2018.
448
10

---

Under review as a conference paper at ICLR 2023
Fabrice Normandin, Florian Golemo, Oleksiy Ostapenko, Pau Rodriguez, Matthew D Riemer, Julio 451
Hurtado, Khimya Khetarpal, Dominic Zhao, Ryan Lindeborg, Timothée Lesort, et al. Sequoia: A software framework unify continual learning research. arXiv e-prints, pp. arXiv-2108, 2021.
Anvil V. Rao. A survey of numerical methods for optimal control. Advances in the Astronautical Science, 135:497-528, 2010.
454
455
Adrian Remonda, Eduardo Veas, and Granit Luzhnica. Acting upon imagination: when trust imagined trajectories in model based reinforcement learning, CoRR, abs/2105.05716, 2021. URL
457
abs/ 2105.0116
458
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR (Poster), 2016.
480
Brijen Thananjeyan, Ashwin Balakrishna, Ugo Rosolia, Felix Li, Rowan McAllister, Joseph E. Gon- zalez, Sergey Levine, Francesco Borrelli, and Ken Goldberg. Safety augmented value estimation from demonstrations (saved): Safe deep model-based rl for sparse cost robotic tasks, 2020.
463
S. Thrun. A lifelong learning perspective for mobile robot control. In Proceedings of IEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS'94), 994), volume pp. 23-30 vol.1, 1994. doi: 10. 10.1109/ 1911000SS.99..... 1994.407411
465
466
Emauuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In IROS, pp. 5026-5033. IEEE, 2012. ISBN 978-1-4673-1737-5.
Grady Williams, Paul Drews, Brian Goldfain, James M. Rehg, and Evangelos A. Theodorou. In- formation theoretic model predictive control: Theory and applications to autonomous driving, 2017.
469
Annie Xie and Chelsea Finn. Lifelong robotic reinforcement learning by retaining experiences. arXiv preprint MXXiv:2109.09//80 2021.
472
473
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates. Neural Computation, 33(11):113-33177, 10 2021. ISSN 0869-7667. doi: (166///co.a.01440 URL httpts::///...t..a http///t.oorrr//./oo 1162/
475
neco 01430
Shangtong Zhang and Richard S. Sutton. A Deeper Look at Experience Replay. arXiv e-prints, art. arXiv:1712.01275. December 2017.
478
479
Guangxiang Zhu, Minghao Zhang, Honglak Lee, and Chongjie Zhang. Bridging imagination and reality for model-based deep reinforcement learning. NeurIPS, 2020.
481
A
OPTIMAL TRAJECTORY GENERATION
Algorithm
shows the use of CEM to compute the optimal sequence of actions O
Algorithm 3 Compute Optimal Trajectory
Input: current state of the environment, dynamics model
1: Initialize P particles, sf, with the initial state, Sinit
2: for Actions sampled autt C.M((-), to C.MSless do
3: Propagate state particles s using TS and f (D, amct 1+#
4: Evaluate actions
Σ
1/4 Σ r(st,a+)
p=1
5: Update CEM() distribution
6: return 4/44
482
11

---

Under review as a conference paper at ICLR 2023
B
MASS POINT TASKS
484
Each algorithm trains a model on a sequence of seven separate sub-tasks: two corners and their inverses, a chicane and its inverse, and a straight ( The full track contains some of the sub-
485
tasks seen during training (Shown with different colors in the full-track image (Appendix Figure
486
487
addition tasks unseen during training (shown in black in the full-track image).
Figure 5: Tasks for the Masspoint environment. The x-axis and the y-axis of each figure represents the x,y coordinates of the path to be followed by the mass point bot. The red dot represents starting point. Top left-to-righ: cl, cl inverted, chicane, chicane inverted and c14. Bottom left-to- right: c14 inverted, straight, full track (comprising sub-tasks. chicane, c14. straight, cl), sectorl and sectorl inverted
C ENVIRONMENTS
488
489
We evaluate the methods on agents the MuJoCo Todorov Modoooooodorov al. (2012) physics engine. To establish valid comparison with Chua et al. (2018) we use four environments with corresponding task length
491
(T (askHH) and trajectory horizon (H)).
Cartpole (CP): S RR A R. Taskl 200, H 25
492
Reacher (RE): R!7 A RR, TaskH 150, H 25
Pusher (PU): S _0 A R TaskH 150, H 25
Masspoint: E R A E ?R? Task H 290, H 25
496
This means that each iteration will run for Task H. task horizon, steps, and that imagined trajectories include H trajectory horizon steps. S E R$ A R refers the dimensions of the environment state consisting in a vector of components and the action consisting in a vector of components.
498
D Ex 2. CONTINUAL LEARNING EXPERIMENT. ADDITIONAL RESULTS
500
6 shows additional results with the wall-time during the training process for the continual learning experiment.
Figure
501
502
E
MAXIMUM PREDICTION DISTANCE
503
An additional parameter of interest when using UARF what we call the "maximum prediction distance" or MPD. This parameter operates on the assumption that even for model that has reached convergence, some environments, predicting trajectories of great length is impossible. As such, recalculations must inevitably occur at the end of such long trajectories. These recalculations do not necessarily represent the appearance of new, unseen information, but rather limitation of the successful model in complex environment. Hence, would not want to add these experiences to the buffer.
505
507
509
Where define the cutoff for trajectory of "reatt length" can be changed, and it serves adjust the strictness of UARF's filtering mechanism. For Ex.1 and Ex.2, we chose to set the maximum prediction distance to ensure the strictest filtering of the replay buffer.
511
12

---

Under review as a conference paper at ICLR 2023
In 7 evaluate the effect of the MPD the performance of UARF in the cartpole environ- ment. We were particularly interested in the effect on the rate of recalculation and on the size of the replay buffer. In one can that
Per Episode Wall Time
514
UARF
40
BICHO
516
BASELINE
518
the models converge with no issue, but they do
20
519
differ slighty the rates of recalculation and buffer filtering. The strictest MPD, MPD=1,
www.w
520
results in the leanest buffer, but its recalcula-
0
tion rate is slightly higher than the models with MPD=2 and MPD4.
50
100
150
200
525
Episode
524
These results show that the MPD serves as way to tune the strictness of UARF's buffer fil- tering mechanism. It would be area of future research to find the optimal way to tune this pa- rameter automatically throughout training such 629
525
Figure 6: Per Per episode wall time for the three meth- ods during the training process Ex.2. Vertical lines indicate task switch points.
527
as to best balance recalculation rate and replay
530
buffer filtering.
Figure 7: Performance of the examined algorithms in Cartpole using different maximum prediction distances (MPD). The blue line represents UARF with an MPD=1. The red line is UARF with an MPD=2. The green line is UARF with an MPD-4. From left to right column: episode reward, time per episode (s), cumulative number of observations stored in the replay buffer, new experiences added to the buffer per episode.
F HYPERPARAMETERS
532
Table 2 shows the hyper parameters used to train UARF Look-ahead refers to the number of steps ahead BICHO and UARF using asses the quality of the imagined trajectories. 3 controls the sensitivity of BICHO and UARF to inform whether a trajectory is still valid or not. "New Data
534
Train Threshold" refers the amount of fresh data that must be added the replay buffer before the UARF algorithm triggers the training of the dynamics model.
536
Cartpole 10
Pusher
Reacher
Masspoint
Look-Ahead
10
10
10
B
0.005
0.005
0.005
0.5
New Data Threshold
1%
1%
1%
1%
Training episodes
100
100
10
30/task
CEM population
400
500
400
400
CEM # elites
40
50
40
40
CEM iterations
5
5
5
CEM
0.1
0.1
0.1
0.1
MPD
10
10
10
Table Hyperparameters used for UARF implementation.
13


Reviews:
[{"review_id": "cqhExaSZg2r", "reviewer": "Reviewer_LPd9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes approaches for determining when to add samples to replay buffer (in context of MBRL). It considers experiments on simple environments to show that the proposed method can perform at par with the baselines while keeping the size of the replay buffer small.", "summary_of_the_review": "At this point, I think the paper needs several updates to both the writing and the experiments. I encourage the authors to point out mistakes / oversights in my review and I am open to changing my views and scores.", "strength_and_weaknesses": "## Strength\n\n1. The paper studies an important problem - choosing which examples to save to the replay buffer (in the context of MBRL).\n\n## Areas for improvement\n\n1. Writing issues:\n\n    1.1 The paper needs to flow better. It starts with MBRL (which is an important problem to study), then jumps to Replay Buffer (RB) and Continual Learning (CL). The transition from MBRL to RB and CL needs to be clarified. Are RB-related issues the only challenges in MBRL, or why are those issues more important? Similarly, why should one care about CL in the context of MBRL? The authors need to motivate these questions better. \n   \n    1.2 Related work needs references to approaches that select what examples are retained/sampled from the replay buffer in the context of CL methods. Given that the paper's primary focus is on reducing the size of the replay buffer and determining what examples/transitions to store, referencing such works is helpful.\n\n    1.3  Typos (like MRBL in line 192)\n    \n    1.4 Need to include details like the number of seeds/trials for experiments.\n\n    1.5 Several phrases are not explained, e.g., \"complementary experiences\" in line 256, \n\n2. Experiments\n\n2.1 Several details are missing or glossed over. E.g., in line 254, \"we evaluate the proposed method in benchmarks environments for higher number of episodes than in Chua et al. (2018)\" - why more episodes? Or line 270, \"With similar training scenarios, Remonda et al. (2021) trained CP for 30 episodes, PU and RE for 150. Instead, we trained each for 100 episodes.\" - > why change the number of episodes?\n\n2.2 Several hyperparameters are introduced, but the corresponding ablations/analysis are missing.\n\n2.3 Reporting wall-clock time can be misleading as the wall-clock time depends on the load on the system when the time was recorded. A better metric to report is the number of floating-point operations\n\n2.4 It is not obvious to me what the difference is between BICHO and UARF\n\n2.5 In line 298, the paper notes, \"It takes longer to update the model as the replay buffer grows linearly.\" Is this because the system is trained on the entire replay buffer at every update step? Did the authors consider the variant where a fixed number of data points (respective of the size of the replay buffer) is used for an update? \n\n2.6 In line 263, the paper says, \"replay buffer size (which is free of any artificially-imposed limits)\" - it is unclear if the baseline approaches need the unlimited buffer size. Maybe a small buffer size (equal to the buffer size used by the proposed method) is sufficient for solving the task, and the extra examples are unnecessary. My suspicion grows stronger when I look at the first column of figure 1, where the models often reach quite close to the convergence performance in a few episodes. Still, the training seems to have continued to inflate the size of the replay buffer for the baseline methods. I would also like to see an ablation where the size of the replay buffer (for the baselines) is fixed, and older entries are thrown away in FIFO order as new entries become available.\n\n2.7 Regarding \"time per episode,\" I understand that the baselines would take longer because the replay buffers are larger (read 2.5 for a query on that), but this effect should not kick in when the number of episodes is very small. e.g., in Figure 1, for 0 episodes, I expect all the methods to take the same amount of time (maybe the proposed method takes a bit longer due to the computation of some variables). Still, the PETS baseline is taking much longer, even with 0 episodes. Why is that the case? Isn't the proposed method using the PETS model as the underlying MBRL method?\n\n2.8 In Section 6.2, the tasks considered are basically the same line-following task. Given that the agent has access to the distance from the closest point on the path, there is not much distinction between the tasks. It would not be surprising that an agent that (relatively) easily generalize across different task instances (presented as different tasks).\n\n2.9 The previous question (2.6) about the replay buffer size also applies here. It must be clarified if the baseline needs large buffers to solve the task.\n\n2.10 Generally, the paper needs to consider more complex tasks (with longer episode lengths) if the claims are about improving performance in the lifelong learning setup.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "c5EU7NeJG5", "reviewer": "Reviewer_qNFh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addresses replay buffer management. The authors propose a method that uses model prediction uncertainty to decide the value of each transition. A transition is added to the replay buffer only when its value is above a certain threshold. To combat overfitting and improve computational efficiency, the authors further propose to update the model only when there are enough fresh transitions added to the replay buffer. Experiments in the single-task setting and the task-agnostic continual learning setting demonstrate the efficacy of the proposed methods on reducing the replay buffer size and reducing computational cost. The authors also conduced an experiment to show the potential of the proposed method in mitigating catastrophic forgetting.", "summary_of_the_review": "I have difficulty understanding this paper thoroughly. I don't think this paper is ready to publish in its current form.", "strength_and_weaknesses": "# Strength\n* This work addresses an important open problem in how to decide if a transition should be added to the replay buffer. An effective method for maintaining replay buffers can potentially make a big impact as experience replay is a standard building block in many modern deep reinforcement learning agents.\n* The experiments show that the proposed method can indeed reduce the memory footprint of the replay buffer and the computational cost while maintaining the agent's performance on the task.\n\n# Weakness\n* The paper is challenging to follow. It is hard to capture the main question of interest of this paper at the first pass. The connection to continual learning feels unnatural. Other suggestions on writing improvements are discussed below.\n* Section 5 and Algorithm 2 are hard to understand. The authors introduce a lot of notations on-the-fly. I would suggest the authors to define all these notations in clearly all at once. I would also suggest the authors to consider replacing the incomprehensible symbols like $^{*}$ and $'$ with easier-to-interpret ones.\n* It will be good to provide a clear description of BICHO. In line 289, it says \"BICHO uses functionality to avoid unnecessary computation\". This is rather confusing. What functionality? How does it avoid unnecessary computation? Given that BICHO is one of the three methods evaluated in the experiments, I think it is worth spending some space on a clear description of BICHO.\n* The proposed method only demonstrates weak evidence in improving the agent's performance on the task in the experiment in Section 6.3. This paper would benefit from a more in-depth evaluation of the proposed method on task performance.\n\n# Questions and minor issues\n* On page 1: what does \"a single-shot dynamics model\" mean?\n* The first line of Section 2: The abbreviation \"MFRL\" shows up without explanation.\n* Line 213: \"...until the _unreliableModel_ flag is set to False...\" I think it should be True but I'm not 100% sure...\n* Line 220: What is BICHO precisely? What does \"BICHO returns True\" mean?\n* Line 240: \"...(Algorithm 2 L: 24-25)...\" There are only 23 lines in Algorithm 2\n* Line 300: \"Results in Fig 1 for PU, RE, and Masspoint are consistent with those of CP.\" Not really - BICHO clearly performs worse than BL in Masspoint in the Episode Reward panel.\n* Line 386: References to Fedus et al and Aljundi et al should use \\citep{}.\n* Line 393: What does \"RB\" stand for?\n* Regarding the format: other submissions do not have line number. Could the authors double check if they are using the correct template?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RTLSdAebvB0", "reviewer": "Reviewer_3gsD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to restrict what is stored in a replay buffer used to train a model in order to avoid filling the buffer with things that are already well-learned.  The proposed method is applied with open-loop control using an ensemble of probabilistic models for planning. The method is based on a notion of model uncertainty. Intuitively, if the model predictions are already reliable, in a certain sense, for a given transition, the transition is not added to the buffer. Model uncertainty is simultaneously used to determine when to replan a trajectory during execution, replanning only occurs when model uncertainty becomes high, and the preplanned trajectory is otherwise followed. The particular notion of model uncertainty used appears to be the Wasserstein distance between the predicted distribution of future rewards over some horizon as computed the last time the model predictions were considered \"unreliable\", and predictions beginning from the current state. Experiments demonstrate that the proposed method reduces buffer size and computation in a number of tasks without compromising performance.", "summary_of_the_review": "While the general idea is interesting, I feel the particular implementation is not very well motivated and the empirical evaluation is not very extensive or convincing. For these reasons I don't think the paper is ready for publication at ICLR in its current form.", "strength_and_weaknesses": "I really like the general idea of this work. Training a model excessively on, and storing, things that are already well-learned is indeed wasteful of computation and memory. Devising approaches to mitigate this is an interesting area for research. I hope the authors continue to work on this topic.\n\nOn the other hand, I feel the particular implementation is not very well motivated and the empirical evaluation is not very extensive. For these reasons I don't think the paper is ready for publication at ICLR in its current form. \n\nThe notion of model uncertainty used is a little odd to me and doesn't come across as well-motivated, particularly for deciding whether to include transitions in the replay buffer. If the predicted future rewards gradually drift away from their earlier predicted values until the error passes a certain threshold, why does it make sense to assume the particular step at which they passed the threshold is to blame and thus should be used for further model training? Overall, I feel this needs to be better motivated.\n\nCertain details of the algorithm are not clear, for example, \"maximum prediction distance\" is mentioned as a hyperparameter in line 238, but I can't see it in Algorithm 2 so I had difficulty understanding precisely how it is used. I am also unclear on whether the reward is learned along with the dynamics or provided to the planner explicitly as I couldn't find any mention of reward learning in the paper. If it is provided to the planner, this would have significant implications in particular for the multitask experiments, where I believe essentially all that changes between tasks is the reward. This would also make it disingenuous in my mind to say \"the model is not aware of tasks or task transitions\" as on line 317 so this point should definitely be clarified.\n\nThe empirical results are not very convincing. The experiments in Figure 1 show that by restricting what is added to the replay buffer in the way suggested, replay size remains smaller than if one does not restrict additions, while performance remains similar. However, the obvious baseline of using a fixed-size replay buffer and dropping the oldest item is not included so it's not clearly demonstrated that the particular method used is beneficial. Similarly, it is shown that only replanning when the model is found to be \"unreliable\" will save computation without sacrificing performance, but this is not compared to simply replanning after a fixed number of steps or any other simple strategy so one cannot say much about the effectiveness of the particular method suggested.\n\nI didn't find the continual learning (task switching) experiments really added much that wasn't already demonstrated in the single-task experiments. Once again we see that the proposed method reduces buffer usage and maintains similar performance, but no simple alternatives for limiting buffer size are evaluated.  One exception is the \"catastrophic forgetting\" experiment where the effect of limiting buffer size to 5000 is evaluated and shown (with a single random seed?) to improve final performance on past tasks in the continual learning setting. This experiment is a step in the right direction, but I think more needs to be done. \n\nAnother issue is that I believe all experiments are run with a single random seed (please correct me if I'm wrong). For the most part, I don't see this affecting the outcome as most claims don't deal with performance. However, it does make it unclear whether the performance benefit compared to a fixed-size buffer in Table 1 is a real effect or just noise. At least this experiment should be done with more random seeds and with confidence bounds.\n\nAnother, more minor, issue is the exposition of model-based RL which is currently unclear and misleading. This should be improved to give a more complete picture of the present paper's place in the broader literature, as well as the limitations of the approach. In my mind model-based, RL refers to any reinforcement learning algorithm that learns some model of its environment from experience as an intermediate step to learning, or computing, a good policy. However, this paper makes a number of statements about MBRL that are really about a specific subset of approaches including:\n* \"An MBRL agent maintains an extensive history of its observations, its actions in response to observations, the resulting reward, and new observation in an experience replay buffer.\" which refers to algorithms which use experience replay along with a model.\n* \"At each time step, the agent executes only the first action in the trajectory, and then the model re-imagines a new trajectory given the result of this action.\" Which seems to refer only to decision time-planning approaches and omits algorithms which use a model in the background for policy learning.\n* \"the simulator starts and the controller is called to plan the best trajectory resulting in $a^*_{t:t+H}$\" seems to refer to specifically open-loop planning.\nNote that I don't mean to suggest focusing on a subset of techniques is inherently problematic, only that these things should not be said to characterize MBRL more generally.\n\n\n\nMinor Comments and Corrections\n==============================\n* In the abstract and introduction: what is a \"single-shot\" dynamics model?\n* line 63: MFRL acronym is never actually defined.\n* Line 137: $\\hat{f}=(s_t,a_t)$ should probably just be $\\hat{f}(s_t,a_t)$\n* Line 220: Is \"BICHO\" an acronym? I can't see that it is ever defined.\n* Line 220: \"BICHO will essentially return True as long as the reward projected in the future does not differ significantly with respect to the imagined future reward...\", I think \"True\" here should be False since BICHO returns true when the model is unreliable.\n* Line 107: Backwards quotation marks on \"teacher\", also elsewhere.\n* Line 341: \"Figure 3 shows episode reward, wall-time, buffer size...\", I don't think wall time is actually included in Figure 3.\n* Algorithm 2 (line 17): I think there may be a typo here, as it appears to be comparing rewards with different time indices.\n\n\nSuggestions\n===========\nOne suggestion for future work in this direction is to work on motivating the problem better, which may in turn lead to a more principled approach. One direction to look at is the use of importance sampling for variance reduction. One can show that lower variance gradient estimates can be achieved by sampling examples with large gradients more often and correcting the bias with importance sampling. This ties in well with the proposed approach of excluding samples which are already well-learned. Perhaps a similar analysis could be done to better justify the proposed approach or derive a more principled way to achieve the same basic goal.\n\nIt would be good to have a baseline with restricted buffer size in the experiments in Figure 1 and elsewhere to check whether the particular method of restricted items in the buffer is better than arbitrarily limiting it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-Mxiuyto73", "reviewer": "Reviewer_SwfR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tries to address the memory scalability issue of MBRL in a very long-time horizon or even infinitely time horizon (say continual learning) settings. An observation is that the model would become more accurate as training proceeds, and some experiences can be discarded. The paper tries to select those experiences which cannot be predicted well by the model to store in the replay buffer. To do this, the paper proposes a method to determine the certainty of the model’s prediction. The method is to measure the Wasserstein distance between the reward distributions of predicted and true trajectories. If the distance is larger some threshold the trajectory would be stored. \n\nTo further reduce the computation cost, the authors propose only training the model when sufficient novel experiences are acquired, which is done by simply introducing an additional parameter adjusting how many experiences should be considered as “sufficient”. Experiments are conducted on both standard RL tasks and continual learning tasks. ", "summary_of_the_review": "The paper is most empirical. But it obviously lacks justifications for several important design choices. It also lacks some literature review, and hence some intuitive baselines are not included in the comparison. ", "strength_and_weaknesses": "Strength: \n\nThe paper studies an important and interesting topic — I agree that the memory and computation scalability issue is important in an extremely long-time horizon learning setting.\n\nWeaknesses: \n\n----------\nNovelty. The paper is mostly combing some intuitive designs together to form an algorithm. I did not see the exactly same algorithm before. But the ideas behind the algorithm are already studied. \n\n----------\nSoundness. \nThe paper is mostly empirical, and the algorithmic design is mostly based on intuition. Though it is fine not to have theoretical support, each design choice should be well-justified. However, many design choices in the paper could be problematically intuitively. Here are some examples. \n\nFirst, using reward distribution to judge whether a trajectory is novel or not is not well-justified. The reward distribution match may not imply anything about the underlying state-action-state transitions. I cannot say that such a reward-based measure does not work at all, but the paper does not provide any insight into why it works. \n\nSecond, it is hard to believe why the method can avoid catastrophic forgetting. Intuitively, if you discard some experiences which have been predicted accurately, the model can still forget those experiences as the sampling distribution shifts (because the policy is changing). \n\nThird, the authors use trajectory optimization and CEM in the MBRL method. There are large amounts of MBRL algorithms. It is better to provide some reasons why the particular method is chosen. \n\nI strongly suggest the authors carefully justify each design choice they used. \n\n----------\nMissing a large body of related works. \n1. Some simple and theoretically sound memory-saving methods, such as reservoir sampling (I think this should be a simple baseline and it is easy to implement) \n\n2. A body of work saving memory by using theoretically justified approaches in MBRL (e.g. organizing experiences: a deeper look at replay mechanisms for sample-based planning … by Pan et al.)\n\n3. Decision/value-aware robust model-based RL by Amir-massoud et al.\n\n4. There are also papers studying/using techniques to avoid/reduce compounding errors in long-time horizon predictions. \n\n5. It is also good to survey papers about avoiding catastrophic forgetting in a supervised learning setting. \n\n----------\nThe empirical results should at least justify the key claims of the paper:\n1. why the particular uncertainty measure is needed/better than the existing uncertainty measure\n2. why the proposed method can avoid catastrophic forgetting ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject