Model-based Lifelong Reinforcement Learning with Bayesian Exploration
Authors: Haotian Fu
Shangqun Yu
Michael Littman
George Konidaris
Venue: NeurIPS 2022 Accept (2022)
Keywords: Deep Reinforcement Learning
Lifelong Reinforcement Learning

Abstract: We propose a model-based lifelong reinforcement-learning approach that estimates a hierarchical Bayesian posterior distilling the common structure shared across different tasks. The learned posterior combined with a sample-based Bayesian exploration procedure increases the sample efficiency of learning across a family of related tasks. We first derive an analysis of the relationship between the sample complexity and the initialization quality of the posterior in the finite MDP setting. We next scale the approach to continuous-state domains by introducing a Variational Bayesian Lifelong Reinforcement Learning algorithm that can be combined with recent model-based deep RL methods, and that exhibits backward transfer. Experimental results on several challenging domains show that our algorithms achieve both better forward and backward transfer performance than state-of-the-art lifelong RL methods.

Model-based Lifelong Reinforcement Learning with Bayesian Exploration
Haotian Fu, Shangqun Yu, Michael Littman, George Konidaris Department of Computer Science, Brown University (hfu7, ;hfu7,syu633 syu68, mllitman, gdk) brown.
Abstract
We propose model-based lifelong reinforcement-learing approach that estimates a hierarchical Bayesian posterior distilling the common structure shared across different tasks. The learned posterior combined with sample-based Bayesian exploration procedure increases the sample efficiency of learning across a family of related tasks. We first derive analysis of the relationship between the sample complexity and the initialization quality of the posterior in the finite MDP setting. We next scale the approach to continuous-stan tional Bayesian Lifelong Reinforcement Learning algorithm that can be combined with recent model-based deep RL methods, and that exhibits backward transfer. Experimental results on several challenging domains show that our algorithms achieve both better forward and backward transfer performance than state-of-the-art lifelong RL methods
domains by introducing Varia-
1 Introduction
Reinforcement learning (RL) 12 has been successfully applied to solve challenging individual tasks such as learning robotic control [I] and playing Go [38]. However, the typical RL setting assumes that the agent solves exactly one task, which it has the opportunity to interact with repeatedly. In many real-world settings, agent instead experiences collection of distinct tasks that arrive sequentially throughout its operational lifetime; learning each new task from scratch is inefficient, but treating them all as single task will fail. Therefore, recent research has focused algorithms that enable agents to learn across multiple, sequentially posed tasks, leveraging knowledge from previous tasks to accelerate the learning of new tasks. This problem setting is known lifelong reinforcement learning 77 [50, 25). The key questions in lifelong RL research are: How can an algorithm exploit knowledge gained from past tasks to quickly adapt to tasks (forward transfer), and how can data from new tasks help the agent perform better on previously learned tasks (backward transfer)?
We propose to address these problems by extracting the common structure existing in previously encountered tasks so that the agent can quickly learn the dynamics specific to the new tasks. We consider lifelong RL problems that can be modeled as miiden-pprammee where variations among the true task dynamics can be described by set of hidden parameters. Our algorithm goes further than previous work in both lifelong learning and HiP-MDPs by 1) Separately modeling epistemic and aleatory uncertainty over different levels abstraction across the collection of tasks: the uncertainty captured by world-model distribution describing the probability distribution over tasks, and the uncertainty captured by task-specific model the (stochastic) dynamics within a single task. To enable for these two quantities and maintain a hierarchical Bayesian posterior that approximates them. 2) Performing Bayesian exploration enabled by the hierarchical posterior: The method lets the agent act optimistically according to models sampled from the posterior, and thus increases sample efficiency.
MDPs or HiP-MDPs 10 27].
accurate sequential knowledge transfer, we separate the learning process
Cooee repository available at bastps://gitaan www..b//////ammmaaaaaaaaa /BLLLL
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Specifically, we propose model-based lifelong RL approach with Bayesian exploration that estimates a Bayesian world-model posterior that distills the common structure previous tasks, and then uses this between-task posterior as within-task prior to learn a task- model in each subsequent task. The learned hierarchical posterior model combined with sample-based Bayesian exploration procedures can increase the sample efficiency of learning. We first derive an explicit performance bound that shows that the task-specific model requires fewer samples to become accurate as the world-model posterior approaches the true underlying world-model distribution for the discrete case. We further develop Variational Bayesian exploration for Lifelong RL (VBLRL), more scalable version that uses variational inference approximate the distribution and leverages Bayesian Neural Networks (BNNs) [L G] to build the hierarchical Bayesian posterior. VBLRL provides a novel way to separately estimate different kinds of uncertainties in the HiP-MDP setting. Based on the same framework, we also propose a backward transfer version of VBLRL that is able to provide improvements for previously encountered tasks. Our experimental results on set of challenging domains show that our algorithms achieve better both forward and backward transfer performance than sfate-of-the-art lifelong RL algorithms when given only limited interactions with each task.
2 Background
RL is the problem of maximizing the long-term expected reward of an agent interacting with an environment [2). We usually model the environment described by a five tuple: (S, R, T, 7), where S is finite set of states; A is a finite set of actions; R: SS A + [0, 1] is a reward function, with lower and upper bound of 0 and 1; T: S A + Pr(S) is transition function, with T(s' (s, a) denoting the probability of arriving in state s' E S after executing action
Markov Decision Process MDP (3)
A in state s; and y [0. 1) is discount factor, expressing the agent's preference for delayed over immediate rewards.
An MDP is a suitable model for the task facing a single agent. In the lifelong RL. setting, the agent instead faces a series of tasks m.,,.., mn, each of which can be modeled as an MDP: (S()) A R(())
For lifelong RL problems, the performance of a specific algorithm usually evaluated based on both forward transfer and backward transfer results 60 :
Forward transfer: the influence that learning task & has on the performance future task k > t.
Backward transfer: the influence that learning task has on the performance in earlier tasks k << t.
A key question in the lifelong setting is how the series of task MDPs are related; we model the collection of tasks as HiP-MDP, where a family of tasks is generated by varying a latent task parameter w drawn for each task according to the world-model distribution Pq. Each setting of w specifies a unique MDP, but the agent neither observes w nor has access to the function that generates the task family. The dynamics T(s')s a; wi) and reward function R(rls, a; wi) for task i then depend on Wi E 22 which is fixed for the duration of the task. The tasks are i.ddd sampled from a fixed distribution and arrive one at a time.
3 Related work
The first category of lifelong RL algorithms learns single model that encourages transfer across tasks by modifying objective functions. EWC (X] imposes a quadratic penalty that pulls each weight back towards its old values by an amount proportional to its importance for performance on previously-leamed tasks to avoid forgetting. There are several extensions of this work based on the
core idea of modifying the form of the penalty 22 53 B3) Another category of lifelong RL methods uses multiple models with shared parameters and task-specific parameters to avoid alleviate the catastrophic problem [5)
24 B. The drawback of this method is that it is hard to incorporate the knowledge learned from previous tasks during initial training on new task [T] Nagabandi et al. B2] introduce a modell-asse continual learning framework based on MAML, but they focus on discovering when new tasks were encountered without access to task indicators.
Published HiP-MDP methods use Gaussian Processes I0 or Bayesian neural networks 27 to find a single model that works for all tasks, which may trigger catastrophic forgetting 5
24 Meta-
RI. 47 12 and multi-task RI.
35 4xj settings also attempt to accelerate learning by transferring
knowledge from different tasks. Some work employs the MAML framework with Bayesian methods

---

to learn a stochastic distribution over initial parameters 16 13. trajectories to infer the hidden parameter, which is taken as an additional input when computing the policy 37 56 14] Our method, however, focuses on problems where the tasks arrive sequentially instead of having a large number of tasks available at the beginning of training. This sequential setting makes it hard to accurately infer the hidden parameters, but opens the door for algorithms that support backward transfer.
Other work uses the collected
Some prior work uses Bayesian methods in RL to quantify uncertainty over initial MDP models [S
Several algorithms start from the idea of sampling from a posterior over MDPs for Bayesian RL, maintaining Bayesian posteriors and sampling one complete MDP [1] 49 or multiple MDPs [2] Instead of focusing on single-task RL, our algorithm aims to find posterior over the common structure among multiple tasks. Wilson et al. 49 uses hierarchical Bayesian infinite mixture model to learn a strong prior that allows the agent to rapidly infer the characteristics of a new environment based on previous tasks. However, it only infers the category label of a new MDP and only works in discrete settings.
Model-based Lifelong Reinforcement Learning
Our approach built upon two main intuitions: First, transferring the transition model instead of policy/value function leads to more efficient usage of the data when "finetuning" on new task. As we show empirically in Section 5.1, although some model-free lifelong RL algorithms perform better than the proposed model-based method in single-task cases, in lifelong RL setting of the same task type the model-based method is still able to achieve comparable/better performance with only half the amount of data. Secondly, with model that is able to capture different levels of the uncertainty within HiP-MDPs, an agent can employ sample-based Bayesian exploration further improve sample-fficiency.
The model underlying our approach is a hierarchical Bayesian posterior over task MDPs controlled by the hidden parameter w. Intuitively, we maintain probability distributions that separately capture two categories of uncertainty within lifelong learning tasks: The world-model posterior P(w) captures the epistemic uncertainty of the world-model distribution over all future and past tasks m1," . M, controlled by the hidden parameter W1, Wwn ~ Pa- As the learner is exposed more and more tasks, this posterior should converge to the world-model distribution Pq. The task- specifce posterior P(w:) captures the epistemic uncertainty of the current task m. (Throughout the paper we will often write for simplicity.). As the learner is exposed to more and more transitions within the task, this posterior should approach the true distribution corresponding to ws, i.e. peaking at the true wi for this specific task i, leaving only the aleatoric uncertainty of transitions within the task, which is independent of other tasks, Each time the agent encounters a new task, we initialize the task-specific model using the world-model posterior and further train it with data collected only from the new task. One of our key insights here is that the sample complexity of learning a new task will decrease as the initial prior of task- sask-specific model approaches the true underlying distribution of the transition function. Thus, the agent can learn new tasks faster by exploiting knowledge common to previous tasks, thereby exhibiting positive forward transfer.
Specifically, we model the task-specific posterior via the transition dynamics using p(s5+1, r''s", a ww)) The task-specific posterior, given new state acteonction pair from task i, can be rewritten via Bayes' rule:
P(s (i),otttr''(),a'sa P(s++1 P(stt1,r'|D rr((((
P(uy) a', 1+1 ,
(1)
where D = s is the agent's history of task i until time step t. The world-model posterior, given the new data from task i, can be rewritten as:
P(w)D P(w|Drai11) ((D.))P(D/DDaam
(2)
P(D.(Dra-1)
where DIs denotes the agent's 's history with all the experienced tasks ~ i until current task In particular, each time when the agent faces a new task and has not started updating its task specific posterior yet (that is, Dj = Ø), we first use the world-model posterior to initialize the task-specific prior: P(w, P(wl Dr.). The world-model distribution aims to approximate the underlying
3

---

Po.. The task- specific distribution aims to approximate the distribution that peaks at the true wa for this specific task
In Section we derive a sample complexity bound in the finite MDP case which explicitly show how the distance between the distribution of task's true transition model and our task-specific model's prior initialized by the parameters of the world-model posterior will affect the learning efficiency of new task. Then, Section 4.2 and we extend our high-level idea scalable version that can be combined with recent model-based RL approaches and exhibits positive forward backward transfer.
4.1 Sample Complexity Analysis
In this subsection, we consider the finite MDP setting and use a standard Bayesian exploration algorithm BOSS 2 as single-task baseline. Note that BOSS creates optimism in the face of uncertainty as the agent can choose actions based on the highest performing transition of the K models sampled, which drives exploration. We included explanations of the finite MDP version of our algorithm BLRL (Bayesian Lifelong RL) based on BOSS in appendix C and simple experiments on Gridworlds in appendix
BLRL uses the world-model posterior P(w] D1:5-1 learned from previous - 1 tasks to initialize (copy distribution) the task-specific prior of P(w)) of new task i, aiming to decrease the number of samples needed to Iearn an accurate task-specific posterior. Our analysis focuses on how the properties of the Bayesian prior affect the sample complexity of learning each specific task.
Let π (w;) denote the prior distribution on the parameter space r. We consider set of transition- probability densities p( (w::)) = =((s*1 "rr++*""*" s' a", w:) indexed by wi, and the true underlying density g. We also denote the model family {p(: w.) We E TT) by the same symbol I.
Lemma 4.1. The task-specific posterior in Equation Ccan be regarded as the probability density g(w:) with respect to w. that attains the infimum of:
((stt1 D' a') p(sttl, r DrrDD, (',a))
Rg) = Ew,(w)
Σ
In
DKr(gdT] (TT)
(3)
inf R, controls the complexity of the density estimation process for g(w:). Intuitively, Lemma converts the Bayesian posterior into information theoretical minimizer that allows us to further investigate the relationship between the properties of the Bayesian prior and the risk/complexity of attaining the posterior.
Proposition 4.2. Define the prior-mass radius of the transition densities as:
d= = iin=fff inf(d: : d - In ((pp T: DRI(gll)) <d)))-
(4)
Intuitively, this quantity measures the distance between the Bayesian prior of the task- model and the true underlying task-specific distribution. Then, we adopt the same settings in Zhang (55): dp E (0, 1) and n 2 let
En (1 (1+ [ + (1 D)Epprrp ((1 1)/(1-1)),
(5)
where Eupper,n is the critical upper-bracketing radius
(55). The decay rate of E upper, controls the consistency of the Bayesian posterior distribution [2]. Let = 3, we have for all states and actions, h 2 0 and E (0, 1). with probability least 1
Th </ (p g113/2 2E, + (4n 2)h
|x)
(6)
8/4
Ten
I
en
where nn (w:) denotes the posterior distribution over w after collecting samples. X denotes the n collected samples. It functions the same as D. in Equation
Similar to BOSS, for new MDP mn* ~ M with hidden parameters wm. we can define the Bayesian concentration sample complexity for the task-specific posterior: f(s, a, co. do. Po). as the minimum number u such that, if u IID transitions from (s, a) are observed, then, with probability at least 1 - dg.
Pr Pimosposterion (1))(:) wm) T(:1s, Wm )|/ < <0) 2 1 - Pa-
(7)

---

Intuitively, the inequality means that for model with the hidden parameter sampled from the learned posterior distribution, the probability that that it within co far away from the true model is larger than
PO-
Lemma 4.3. Assume the posterior of each task is consistent (that is, Eupper,n = o(1) and set 1 then the Bayesian concentration sample complexity for the lask-specific posterior [(, 0, p) -
d,+In A5-dz
0
Proof (sketch). This bound can be derived by directly combining Lemma 4.2 and Equation
The above lemma suggests an upper bound of the Bayesian concentration sample complexity using the prior-mass radius. We can further combine this result with PAC-MDP theory 30 and derive the sample complexity of the algorithm for each new task.
Proposition 4.4. For each new task, set the sample size K- 0 In St)) and the parameters
8 c c -0 5 PD = 3-A'K then, with probability at least 1 1 48, V*(st) V*( 4co
in all bui Ö 12 01559122 steps, where O(-) suppresses logarithmic dependence.
Proof (sketch). The central part of the proof is Proposition 4.2 (detailed proof in appendix), and the re- maining parts are exactly the same as those for BOSS. In general, the proof is based on the PAC-MDP theorem |ОО combined with the new bound for the Bayesian concentration sample complexity we derived in Lemma 2. For each new task, the main difference between BLRL and BOSS is that we the world-model posterior to initialize the task-specifie posterior, which results in new sample complexity bound with dz
The result formalizes how the sample complexity of the lifelong RL algorithm will change with respect to the initialization quality of the posterior: if we put a larger prior mass at a density close to the true q such that d₂ is small, the sample efficiency of the algorithm will increase accordingly. In other words, the sample complexity of our algorithm drops proportionally to dz, which is the distance between the Bayesian prior of the task-specific model initialized by the parameters of the world-model posterior and the true underlying task-specific distribution. We provide a illustrative example in the appendix O
4.2 Variational Bayesian Lifelong RL
The intuition from the last section is that, if we initialize the task-specific distribution with a prior that is close to the true distribution, sample complexity will decrease accordingly. To scale our approach, we must find a efficient way to explicitly approximate these distributions. We propose practical approximate algorithm, VBLRL, that uses neural networks and variational inference [21].
We choose Bayesian neural networks (BNN) to approximate the posterior. The intuition is that, in the context of stochastic outputs, BNNs naturally approximate the hierarchical Bayesian model since they also maintain a learnable distribution over their weights and biases [17 23
We use the uncertainty embedded in the weights and biases of networks to capture the epistemic uncertainty introduced by hidden parameters of different tasks, while we also set the outputs of the neural networks to be stochastic to capture the aleatory uncertainty within each specific task. In our case, the BNN weights and biases distribution g(w; Φ) (a distribution over w but parameterized by Φ) can be modeled as fully factorized Gaussian distributions 3
[
q(w;
II (W)ll,,,)))
(8)
where e = a), and μ the Gaussian's mean vector while the covariance matrix diagonal.
We maintain world-mode BNN across all the tasks and a task-specific BNN for each task. The input for all the BNNs a state-action pair, and the output are the mean and variance of the prediction for reward and next state. Then, the posterior distribution over the model parameters can be computed leveraging variational lower bounds
22 23
Φ arg min DKill(w; (e [[[([[]] Ban loogpss st1 D w]]
(9)
5

---

where p(W) represents the fixed prior distribution of w, also recall that D' {s', "e',aa,, ,,','',_,,___ rr...... , In VBLRL. p(w) for the task-specific distribution is initialized by the world-model distribution, while p(w) of the world-model distribution is simply set to Gaussian. (That could be improved with more informed task family knowledge.) The second term on the right hand side can be approximated through 1
N' Σi= log p(s r²1 D", w;) with N samples from Wi ~ q()). This optimization can be performed in parallel for each s, keeping Φ1-1 fixed. Details about our BNN structure can be found in appendix B
Once we have the world-model as well as the task-specific posterior model, we can do pos- terior sampling to drive Bayesian exploration. Osband et al. M and Rakelly et al. K3 already show the benefit of posterior sampling for improving sample efficiency in single- task RL and meta-RL settings. Here, instead of sampling from the posterior of value func- tions or latent context representations, we directly sample from the posterior of the transition model and choose the optimal action based on the transition samples. Then we use the col- lected samples to update the posterior and do sampling again. The agent thus continually do Bayesian exploration and acts more and more optimally as the posterior distribution narrows. We provide the framework of VBLRL in Fig- ure [ The outer loop (yellow rectangle) stands for the loop of training world-model BNN across tasks (noted at the bottom-right corner) the in- ner loop (cyan rectangle) stands for the training of the task-specific BNN within each specific task. We employ our posterior knowledge mod- els in the context of model- RL method. When encountering new task, VBLRL first uses the model parameters (that is, {ur o of weights and biases of BNN) from the general knowledge model to initialize the task-specific posterior network (directly copy parameters). We sample transitions from the task- pos- terior and select actions based on the generated transitions. The detailed algorithm is summa- rized in Algorithm|] | Specifically, for planning, at each step we begin by creating P particles from the current state s- = stvp. = Then, we sample N candidate action sequences ou:+T from a learnable distribution. These two steps are the same as PETS g Then we propagate the fate-action pairs using the learned task-specific model Pm, (-1s, a) (BNN) and use the cross entropy method [4] to update the sampling distribution to make the sampled action sequences close to previous action sequences that achieved high reward. We further calculate the cumulative reward estimated (via the learned model) for previously sampled sequences and select the current action based on the mean of that distribution. By sampling from the task-specific posterior each step, the agent explores in a temporally extended and diverse manner.
Tran
Ropky lufter
WorlS-mode
Copy
callturan
Tran
Task
Transtors
Sample weight
Frotecement
a
Somple
Paarer
Steps
Tasko
Figure 1: Variational Bayesian Lifelong RL (VBLRL). "NN" is the network using fixed pa- rameters sampled from the weight distribution of the task-specific BNN. When the agent does CEM planning, we let it propagate the state particles using several NNs that use different network pa- rameters, all sampled from the BNN to achieve randomness in transitions.
For each specific task, we the task- BNN to plan during forward learning. They updated with only the data from the current task thus avoid catastrophic forgetting in the forward transfer process. The world model is updated using the data from all the visited tasks. The intuition is to guide the two posteriors to separately learn two categories of uncertainty within lifelong learning tasks. The world-model posterior captures the epistemic uncertainty of the general knowledge distribution (shared across all tasks controlled by the hidden parameters) via the internal variance of world-model BNN. As the learner is exposed to more and more tasks, the posterior should converge to Pg. The task-specific posterior captures the epistemic uncertainty of the current task m, which comes from the aleatory uncertainty of the world model when generating wi for a new task, via the internal variance of task-specific BNN. We provide additional illustration in Figure 9 In general, we used the world-model BNN to encode Pn over a family of tasks, and the task-specific BNN to encode P(w,), which should peak at the specific true Wi sampled for the current task.
Note that in single-task cases, other CEM-based algorithms like PETS usually maintain set of neural networks using the same training data, and sample action sequences from each of the neural nets to achieve randomness in transitions. However, in lifelong RL settings, it is unrealistic to maintain

---

(2 30) models for each task encountered. Our usage of BNNs avoids such problems as we only have train one neural network using the same data for each task, and we can sample an unlimited number of different action sequences to cover more possibilities as needed. In PETS, the epistemic uncertainty is estimated via the variance of the output mean of different neural networks, while, in VBLRI. it is estimated via the variance of the weights and biases distribution of the BNN.
4.3 Backward Transfer of Variational Bayesian Lifelong RL.
In our lifelong RI setting, the agent interacts with each task for only a limited number of episodes and the task-specific model stops learning when the next task is initiated. As result, there may exist portions of the transition dynamics in which model uncertainty remains high. However, as the world-model posterior continues to train on new tasks, it gathers more experience in the whole state space and can provide improved guesses concerning the "unknown" transition dynamics, even for previously encountered tasks.
Intuitively, the performance of an agent on one task has the potential to be further improved (positive backward transfer) if there exists sufficiently large set of task-specific model's predictions are not confident due to lack of data. In our algorithm, the aleatory uncertainty (irreducible chance in the outcome) is measured by the output variance of the prediction {a,z: ox²) ), and the epistemic uncertainty (due to lack of experience) corresponds to the uncertainty of the output mean and variance (see Definition I below). Thus, a straightforward method to improve a previously learned task-specific model is to find the predictions it needs to make that have high epistemic uncertainty, and replace them with the predictions from the world-model posterior, which has lower epistemic uncertainty. If we only consider reward prediction, the quantity for measuring whether a task- musk-specific/worlke model is sufficiently confident is as follows.
transition pairs of which the
Definition 4.5. For given state-action pair (s, a), we define the confidence level c of the predictions (reward) r (s, a) from a task-specific/world model as:
Ep=r (Fr - =,,) P-1
(o, - 0,
(10)
a+
P-1
where P is the number of particles and a is hyperparameter controlling the scale of the second term. A similar definition applies the task-specific/world model's next-state prediction. Intuitively, c measures the uncertainty of the output mean and variance for each dynamic prediction. During CEM planning, we propagate each pair of P state particles with different network parameters thanks to the usage of BNN, resulting in / different { J, }. Thus we can further calculate the confidence level of current predictions based on these P pairs of output mean and variance.
We show the detailed backward transfer algorithm in Algorithm 2 Compared with the forward training version, the agent will also calculate the confidence level of the task-specific and general- knowledge model for each particular transition and compare the value of them when predicting the state particles. If the task-specific model is not confident enough for this state-action pair (i.e. Cm. Cwm), we will use the world-model to do the predictions instead.
5
Experiments
5.1 OpenAl Gym MuJoCo Domains
We evaluated the performance of VBLRI. on HiP-MDP versions of several continuous control tasks from the Mujoco physics simulator 45). SaalfCheetah-graviny gravity, Hopper-bodyparts. falkerrggraving marks used in prior work (3 For each of six different domains, the task-specific hidden parameters correspond to different gravity values or different sizes and masses of the simulated body parts. Details can be found in the appendix. Compared with prior work, we substantially reduced the number of iterations that the agent can sample and train on: 100 iterations for each task and horizon of 100 (Halfcheetah) or 400 (Hopper& Walker) for each iteration. We used such settings to increase the difficulty of lifelong learning and test the sample efficiency of lifelong RL algorithms, given that is hard for a single-task training algorithm to obtain good policy within such limited number of interactions with the environments. However, we show in appendix H and that, as the agent is
SaalsCheetah-bdyyarre Hopper- Walker-bodypary all of which are lifelong-RL. bench-
7

---

exposed to more and more tasks, our lifelong learning agent is able to achieve similar performance to that of more fully trained single-task agents while requiring far fewer per-task samples.
VBLRI
T-HIP-MPP
LPG.FTW (2x samples)
E2C samples)
Single-tack MBRI
CG Sun
160.68
+ 46 80
126.95
95
31.41
-61.59 + 9.18
-3426.76 4 627 99
-83, 96 1 60.10
CG Itain
226.72
26.53
170. 20
39.92
-29.49 + 11.03
-3440.66
1007.00
-40.47 + 10..88
Back
231.79
23 49
97.84
22.04
-29.95
11.64
-6672.33
3748.63
CB-Strt
110.74
96
78.95
18.43
-263.91
40.80
-5016.93 = 1708.10
-101.02 02 39.19
CW-Train
173.97
78 26
57 57.20
9.42
227.86
+ 42.82
-5454.52
21.45.82
-58.93 4: 33.24
CP Back
181.60
67 50
116.03
17.35
-116.41
1 65.64
-18889,31
6851.05
268..11
+ 33.29
230
30.75
305.63
306. 79 29.14
18.62 1.51
BG Train
332.89
1: 23.68
255.88
41.48
352.10
25.25
345.47
40.26
20.46
1.77
HG Back
360. 94
7.71
312.
2 45
347
1 44.09
301.06 1 114.11
HR HR-Stmrt
12.97
256.13
69.68
133.95
27.
19.12 1.70
Train
296.34
11. 15
227
41.78
285.62
139. 48.24
21:25
2.67
18 Back
316. 74 :: 20. 72
213.93
75.95
281.99
74.96
384.27
199.27
WG-Sturt
248.
19.95
217
39.99
140.75 + 87 64
163.61
86.30
4.32 + 0.23
TGairain
333.65
26.73
268
168.00
45.04
181.76
97.42
210.41 39.34
364.18
49.
173.33
58.43
108.39
94.19
216.09
62.67
WB Start
222 88
34. 49
220 U7
20.31
164.79
10.31
164.00
344.11
4.64 + 0.23
WB Trann
311
:: 16. 30
266
97
26. 72
165. 75
4: 2.39
206.97 4: 41.40
196.42 22.04
WB.Back
152.94
70.39
Table 1: Results OpenAI Gym Mujoco domains. CG denotes Cheetah-Gravity, CB denotes Cheetah-Bodyparts HG denotes Hopper-Gravin HB denotes Hopper-Bodyparts, WG denotes Walker- WB denotes Walker-Bodyparts.
We compared VBLRL against: 1. state-of-the-an art lifelong RL method LPG-FTW BII 2. EWC 28
which is single-model lifelong RL algorithm that achieves comparable performance with LPG-FTW RL (using the same BNN structure and
as shown in the latter paper, 3. singletassk Model- planning procedures), which stands for training the agent from scratch for each new task and do not use the world model to initialize the task- task-specific model, 4. T-HiP-MDP 177. which is a model-based lifelong RL baseline. For a fair comparison, we further replaced the DDQN algorithm [4 used in T-HiP-MDP with CEM planning. and let the transition model also predict the reward for each state-action pair. This modified baseline is similar to the single-model version of VBLRL (i.e., only using the world-model posterior across all the tasks). Moreover, LPG-FTW and EWC are built upon a relatively simple policy gradient baseline, which does not perform as well as our model-based baseline in the single-task setting within the same amount of interactions. Thus, we let them collect twice the amount of samples each iteration compared to VBLRL. We show in Appendix that, with such modifications, in single-task settings the baselines achieve better or at least similar performance compared to the single-task version of VBLRL.
The results are shown in Table D and Figure 7 in Appendix 0 For all six domains, we report the average performance of all the tasks at the beginning of training (Start) and after all training for each new task (Train), as well as the average performance for all previous tasks after training for a given number of tasks, which our backward transfer test (Back). As shown in the results, our all three test stages of the HalfCheetah domain and Walker
VBLRL shows better performance domain, as well as better backward transfer performance on Hopper-gravity and Hopper-bodypart. than the other three algorithms. Comparing Figure 6 and Figure 7 we find that, in the Hopper domains, even though the single-task baseline used by LPG-FTW and EWC performed much better than VBLRL after we let them collect 2x samples each iteration, VBLRL still achieves comparable performance with LPG-FTW and EWC in lifelong RL experiments, suggesting that VBLRL is capable of making better use of the data in the lifelong learning setting. In the walker domains where the single-task baselines achieved similar performance, VBLRL shows significantly better performance than LPG-FTW/EWWC in lifelong RL experiments. These comparisons also suggest that VBLRI's lifelong learning performance may be further improved when combined with better model-based deep RL algorithms, like Dreamer 20) EWC fails in some of the tasks it hard to directly learn a single shared policy that achieves good performance when the tasks are highly diverse. T-HiP-MDP shows good results on some of the tasks because it is more sample-efficient in learning a shared model across all the tasks and more easily captures the world-model uncertainty. However, it cannot achieve as good performance as VBLRL as it hard to model the task task-specific uncertainty using only one model across all tasks, which also leads to negative backward transfer performance on tasks like Cheetah-Gravity. Comparing VBLRL's performance on the Train stage and Back stage, we also find that it shows positive backward transfer results on most tasks, without showing patterns of catastrophic forgetting. Overall, VBLRL's world-model posterior contributes to better forward transfer performance (Start), the learning of task-specific posterior contributes to
8

---

better forward transfer training for each new task (Train), and the combination of these two posteriors guides the agent to achieve better backward transfer performance (Back). Comparing our method to LPG-FTW and EWC highlights the advantage of model- techniques for sample efficiency in lifelong RI. setting, while comparing to T-HiP-MDP and single-task MBRI. demonstrates the importance of estimating both global and local uncertainty.
5.2 Meta-world Domains
contains suite of challenging evaluation benchmarks on a robotic SAWYER We evaluated the performance of VBLRL as well as the baselines on two Meta-World task sets: Reach and Reach-Wall following the settings given by [54]. We used the vI version of Meta- World, with state observations and dense versions of the reward functions. The hidden parameters in these cases are the goals we want the robot reach, which controls the reward functions and are not included the agent's observation space, We measured performance in terms of average cumulative rewards across tasks. shown Figure 2 VBLRL achieves significantly higher performance than the other baselines.
Meta-World
5
114
VBLRL Backward
LPS FTW
EWC
VOLIL
THP MCP
Single-takk MERL
Figure Average performance during training across all tasks. Left: Reach; Right: Reach-Waa
19LRL-Backward denotes VBLRL's average backward transfer performance on all the tasks and is higher than the final performance of VBLRL's forward training in both tasks, suggesting that our approach can achieve positive backward transfer.
aucrnnsiiyy sled teaks (NT1))
aaccesdilly comslaned made: perser
tack
UPG 110 2ing
Figure 3: Performance comparison of VBLRL with the other baselines on MTIO and MT50. We also compare the maximum number of successfully completed tasks of different methods. The performance of VBLRL similar LPG-FTW on MT10 but significantly outperforms LPG-FTW on MT50 as the world-model posterior becomes more accurate when the number of tasks is larger.
We include additional experiments MT10 and MT50 domains of MetaWorld (v2) in Figure MT10 and MT50 consist and 50 different categories of manipulation tasks, including reach, push, drawer close, button press etc., and is highly difficult setting for multi-task RL. We modified these two domains to be lifelong learning settings: instead of getting all 10/50 tasks at the same let them arrive sequentially. For instance, the first task is "reach" and
time the beginning, we let the agent interact with it for 150 iterations, and then switch to "push" and the agent cannot collect data from "reach" anymore, after another 150 iterations the task changes to "pick-place", etc. We compare the results with LPG-FTW, T-HiP-MDP as well as Single-task MBRL (train the agent from seratch for each new task without using the world-model posterior to initialize the task specific model). With the limited number of interactions for each task, our proposed algorithm is still able to achieve reasonably good performance in such settings. In MTI0, the total number of tasks (10) is relatively small so the world-model distribution may be far from accurate, but VBLRL still performs better than Single-task MBRL which does not use the world model to initialize the task-specific model. In MT50, the number of tasks is large enough (50) and we can see that VBLRL significantly outperforms all the other baselines. This is consistent with our intuition that sample efficiency improves in the task specific regime when the world model approaches the environment distribution. As the single-task model-based RL baseline we used here relatively simple (CEM planning) to be comparable with the single-task baseline used by LPG-FTW, it is reasonable to expect the lifelong learning performance to even better after combining our framework with model-basee methods like Dreamer [20] we leave such evaluations for future work.

---

5.3 Ablation Study
This section evaluates the essentiality of VBLRL's components. As shown in Figure 4 left, we tested different backward transfer strat- egy for VBLRI. Bask-specifie-Backeee Backward de- notes the performance if directly using the learned task-specific model to do all the pre- dictions without using world model. World- model-Backward denotes the performance if us- ing world-model to do all the predictions with- out using each task-specific model. Compared to these baselines, VBLRL's backward transfer strategy achieves the best performance by com- bining these two kinds of models according to their confidence level. We also explored the in- fluence of using Bayesian neural network to model the task-specific model. As shown in the right side of the figure, using BNN enables faster adaptation to new tasks compared with using regular neural networks VYBLRL-Deterministte). We also include ablation studies on the number of particles in CEM planning in Appendix N
Coshare
Figure 4: Ablation Study on Reach domain. Left: Backward transfer strategy; Right: Using Bayesian neural networks to model the task-specific poste- rior.
Conclusion and Discussion
To improve sample efficiency in lifelong RL, our work proposed a model-based lifelong RL approach that distills shared knowledge from similar MDPs and maintains Bayesian posterior to approximate the distribution derived from that knowledge. We gave a sample-complexity analysis of the algorithm in the finite MDP setting. Then, we extended our method to use variational inference, which scales better and supports both backward and forward transfer. Our experimental results show that the proposed algorithms enable faster training on new tasks through collecting and transferring the knowledge learned from preceding tasks.
One of the core questions in lifelong learning setting is how tasks should be related for transfer to be effective, and in this paper we are making the HiP-MDP assumption, that is, family of tasks is generated by varying latent task parameter vector drawn for each task according to the world-model distribution. The relationship between different tasks can be modeled in different ways in lifelong RL and this problem setting will affect the algorithm's 's performance. However, we do believe our method can help the agent transfer knowledge for tasks that are in other ways related, as we show in the results on MT10 and MT50, which are not typical HiP-MDP settings.
The overall lifelong learning performance of proposed algorithm is largely limited by the performance of the single-task model- baseline. We show the advantages of model-based lifelong learning methods over model-free methods when the single-task baselines' performance close. In practice, it's possible that the model-free lifelong RL methods still outperforms model-based methods simply because the model-free single-task baseline is much stronger. Bayesian Neural Networks are more computationally expensive than regular neural networks. The longest forward running time among all our experiments hopper which took around 96 hours to finish training and evaluation on all the tasks. This is in general acceptable time cost and can be potentially improved with more recent BNN techniques.
Acknowledgement
The authors thank the members of Brown bigAl for discussions and helpful feedback, and the anonymous reviewers for valuable feedback that improved the paper substantially. This research was supported by the NSF under grant #1955361 and CAREER award #1 8449600 and the DARPA Lifelong Learning Machines program under grant -FA87750-88-2-011 18-2-0117. The U.S. Govemmentt is authorised to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NSF or DARPA. This research was conducted using computational resources and services at the Center for Computation and Visualization, Brown University.
10

---

References
[1] Asmuth, J. and Littman, M. Learning is planning: Near Bayes-optimal reinforcement learning via Monte-Carlo tree search. In UAI, 2011.
[2] Asmuth, J., Li, L., Littmann M. L., Nouri, A., and Wingate, D. A Bayesian sampling approach to exploration in reinforcement learning. In UAI 2009, Proceedings of the Twenty-Fift Conference on Uncertainty in Artificial Intelligence, 2009, pp. 19-26. AUAI Press, 2009.
[3] Blundell, C., Cornebise, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural networks. CoRR, abs/1 505.05424, 2015.
[4] Botev, Z., Kroese, D. P.. Rubinstein, R., and L'Ecuyer, P. Chapter 3 the cross-entropy method for optimization. Handbook of Statistics, 31:35-59, 2013.
[5] Bou-Ammar, H., Faton, E., Ruvolo, P., and Taylor, M. E. Online multi-task learning for policy gradient methods. In ICML., 2014.
[6] Brafman, R. I. and Tennenholtz, M. R-max a general polynomial time algorithm for near- optimal reinforcement learning. (mull)), 2003. ISSN 1532-4435.
[7] Brunskill, E. and Li, Pac-inspired option discovery lifelong reinforcement learning. In ICML,, 2014.
[8] Chua, K., Calandra, R., McAllister, R., and Levine, S. Deep reinforcement learning handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 4770, 2018
[9] des Combes, R. T., Bachman, P., and van Seijen, H. Learning invariances for policy generaliza- tion. In 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net, 2018.
[10] Doshi- F. and Konidaris, G. D. Hidden parameter markov decision processes: A semi- parametric regression approach for discovering latent task parametrizations. In Proceedings of the Twenty-Fifh International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 1432- 1440. IICAI/AAA Press, 2016.
[11] Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. Benchmarking deep reinforce- ment learning for continuous control. In ICML, 2016.
[12] Finn, C., Abbeel, P., and Levine, S. Model-agnostic meta-learning for fast adaptation of deep networks. In 2017.
[13] Finn, C., Xu, K., and Levine, S. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 9537-9548, 2018
[14] Fu, H., Tang, H., Hao, J., Chen, C., Feng, X., Li, D., and Liu, W. Towards effective context for meta-reinforcement learning: approach based contrastive learning. In Thirty-Fifhh AAAI Conference on Artificial Intelligence, AAAI 2021, pp. 7457-7465. AAAI Press, 2021.
[15] Ghavamzadeh, M., Mannor, S., Pineau, J., and Tamar, A. Convex optimization: Algorithms and complexity. Foundations and Trends in Machine Learning, 865-6)335 2015. ISSN 1935-8245.
[16] Grant, E., Finn, C., Levine, S., Darrell, T., and Griffiths, T. L. Recasting gradient- based meta- learning hierarchical bayes. In 6th International Conference on Learning Representations, ICLR 2018. OpenReview.net, 2018.
[17] Graves, A. Practical variational inference for neural networks. In Advances in Neural Infor- mation Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011, pp. 2348-22481 3348-235,, 2011.

---

[18] Guez, A., Silver, D., and Dayan, P. Efficient adaativee reinforcement learning using sample-based search. In NIPS, 2012.
[19] Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, JCML 2018, pp. 1856-1865. PMLR, 2018.
[20] Hafner, D., Lillicrap. T. P., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. ArXiv,
2020.
[21] Hinton, G. E. and van Camp, D. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual ACM Conference Computational Learning Theory, COLT 1993, pp. 5-13. ACM, 1993.
[22] Hinton, G. E. and van Camp, D. Keeping the neural networks simple by minimizing the description length of the weights. In COLT '93, 1993.
[23] Houthooft, R., Chen, X., Duan, Y.. Schulman, J., Turck, D., and Abbeel. P. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems Annual Conference on Neural Information Processing Systems 2016, pp. 1109-1117, 2016.
[24] Isele, D., Rostami, M., and Eaton, Using task features for zero-shot knowledge transfer in lifelong learning. In Proceedings of the Twenty-Fifth International Joint Conference Artificial Intelligence, IJCAI 2016, pp. 1620-1626. UCAI/AAAI Press, 2016.
[25] Isele, D., Rostami, M., and Eaton, E. Using task features for zero-shot knowledge transfer in lifelong learning. In IJCAI, 2016.
[26] Kaelbling, L. P., Littman, M. L., and Moore, A. W. Reinforcement learning: A survey, J. Artif. Intell. Res., 4:237-285, 1996.
[27] Killian, T. W., Daulton, S., Delei-Velez, F., and Konidaris, G. D. Robust and efficient transfer learning with hidden parameter markov decision processes. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, pp. 6250-6261, 2017.
[28) Kirkpatrick, J., Pascanu, R., Rabinowitz, N. C., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska- Barbinkk,, A., Hassabis, D., Clopath, C., Kumaran, D., and Hadsell, R. Overcoming catastrophic forgetting in neural networks. CoRR, abs/ abs/1612.00796, 2016.
[29] Li, Z. and Hoiem, D. Learning without forgetting, 2017.
[30] Lopez-Paz, D. and Ranzato, M. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 6467-6476 2017.
[31] Mendez, A. M., Wang, B., and Eaton, E. Lifelong policy gradient learning of factored policies for faster training without forgetting. ArXiv, 0obs/2007.0701.. 2020.
[32] Nagabandi, A., Finn, C., and Levine, S. Deep online learning via meta-learning: Continual adaptation for model-based rl. ArXiv, abs//181.00761,, 2019.
[33] Nguyen, C. V., Li, Y., Bui, D., and Turner, E. Variational continual learning, 2018.
[34] Osband, I., Blundell, C., Pritzel, A., and Roy, B. V. Deep exploration via bootstrapped dqn. In NIPS, 2016.
[35] Parisotto, E.. Ba, L. J., and Salakhutdinov, R. Actor-mimic: Deep multitask and transfer reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016, 2016.
[36] Puterman, M. I.. Markov Decision Processes-Discee Discrete Stochastic Dynamic Programning. John Wiley & Sons, Inc., 1994.
12

---

[37] Rakelly, K., Zhou, A., Finn, C., Levine, S., and Quillen, D. Efficient off off-policy meta- reinforcement learning via probabilisticc context variables. Proceedings of the 36th Interna- lional Conference on Machine Learning, ICMI. 2019, volume 97 of Proceedings of Machine Learning Research, pp- 1331- PMIR, 2019.
[38] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T.. Hui, F., Sifre, L., Driessche, G. V. D., Graepel, T., and Hassabis, D. Mastering the game of go without human knowledge. Nature, 550:354-359 2017.
[39] Strehl, A., Li, L.,, and Littman, M. Incremental model-based learners with formal learning- time guarantees. ArXiv, abss/100..em. 6.700, 2006.
[40] Strehl, A. L., Li, L., and Littman, M. L. Reinforcement learning in finite mdps: Pac analysis. J. Mach. Learn. Res., 10:2413-2444, 2009,
[41] Strens, M. J. A. A Bayesian framework for reinforcement learning. In Proceedings of the Seventeenth International Conference on Machine Learning (ICMI. 2000), pp. 943-950, 2000.
[42] Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, 1998.
[43] Tch, Y. W. Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems 30: Anmual Conference Neural Information Processing Systems 2017, pp. 4496-4506, 2017.
[44] Thompson, W. R. On the likelihood that one unknown probability exceeds another view of the evidence of two samples. Biometrika, 25:285-290 294, 1933.
[45] Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RS/ International Conference on Intelligent Robots and Systems, IROS 2012, pp. 5026-5033. IEEE, 2012.
[46] van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q-learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence,2016 2016, pp. 2094 2100.
AAAI Press, 2016.
[47] Wang, J. X., Kurth Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M. Learning reinforcement learn. CoRR, abs/1611.05763, 2016.
[48] Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., Zhang, S., Zhang, G., Abbeel, P. and Ba, J. Benchmarking model-based reinforcement learning. CoRR, abs/1907 0205002 2019.
[49] Wilson, A., Ferm, A., Ray, S., and Tadepalli, P. Multi-task reinforcement learning: A hicrarchical Bayesian approach. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), 2007, volume 227 of ACM International Conference Proceeding Series, pp. 1015-1022, ACM, 2007.
[50] Wilson, A., Fern, A., Ray, S., and Tadepalli, Multi-task reinforcement learning: a hierarchical bayesian approach. In ICML '07, 2007.
[51] Yoon, J., Kim, T., Dia, O., Kim, S., Bengio, Y., and Ahn, S. Bayesian model-agnostic meta- learning. In Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NewrIPS 2018, pp. 7343-7353, 2018
[52] Yu, T., Quillen, D., He, Z., Julian, R. C., Hausman, K., Finn, C., and Levine, S. Meta- world: A benchmark and evaluation for multi-tass and meta reinforcement learning. ArXiv, abs/1910. 10897, 2019.
[53] Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence, 2017.
[54] Zhang, J., Wang, J., Hu, H., Chen, T., Chen, Y., Fan, C., and Zhang, C. Metacure: Meta reinforcement learning with empowerment--rivee exploration. In ICML, 2021.
13

---

[55] Zhang, T. From e-entropy to kl-entropy: Analysis of minimum information complexity density estimation. Annals of Statistics, 34:2180-2211 2006.
[56] Zintgraf, M., Shiarlis, K., Igl. M., Schulze, S., Gal, Y., Hofmann, K., and Whiteson, Varibad: A very good method for bayes-adaptive deep RL via meta-learning. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "SUPDPsX1U8G", "reviewer": "Reviewer_thFd", "summary": "This paper proposes that to more effectively handle life-long reinforcement learning settings, where tasks are presented sequentially, one can use model-based techniques with model uncertainty measured globally across tasks and locally within tasks. By tracking model uncertainty and using planning over stochastic models, the method can better tackle both forward transfer (better performance on future tasks after experiencing previous tasks) and backward transfer (better performance on previous tasks after observing new information from future tasks). The paper demonstrates these results in two continuous, life-long reinforcement learning settings: a suite of Mujoco tasks, and Meta-World, comparing to other model-free techniques as well as approaches that do not model both global and local uncertainty.", "questions": "* Can the authors elaborate on how the approach compares to other model-based techniques for RL from meta-learning? Are the technical differences important for getting the approach to work in the life-long setting?\n* Can the authors clarify what they believe the main message of the paper is? Is it specifically about the presented method, more generally about the importance of multiple types of uncertainty, or about model-based vs. model-free approaches to life-long reinforcement learning?\n* Can the authors discuss how the method would fare in settings that do not have obvious, simple ways in which the tasks differ for life-long RL? What would happen if the presented tasks actually share nothing in common? Would the method successfully avoid catastrophic forgetting?No ethical limitations to consider here. The authors could go into more detail on the technical limitations of the approach.\n", "limitations": "No ethical limitations to consider here. The authors could go into more detail on the technical limitations of the approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n* (Significance) life-long RL is an increasingly critical setting, so paper is addressing an important topic. Furthermore, doing so in a model-based way as this paper demonstrates clearly has advantages over model-free methods, even when the planner used is exceptionally simple, which should encourage people to consider whether model-based methods may be more sensible for their setting.\n* (Originality) The method is somewhat novel in estimating both global and local uncertainty within models of transitions using Bayesian Neural Networks. While the overall concept is similar to Nagabandi et al (2018) which models the global and local uncertainty using a MAML-like approach, this paper shows that such a technique is a better choice for the life-long learning setting than alternative life-long RL methods.\n* (Clarity) The paper is clearly written. It was straightforward to follow the method and the figures were mostly well explained. As I mention below, I think some of the proofs could be moved to the appendix to make more space for results in the main body of the text.\n\nWeaknesses\n* (Quality / Clarity) The choice of baselines seemed somewhat odd, and made the overall message of the paper a bit muddled. In my reading, the most important aspect of the method is to model two layers of uncertainty in the MDPs: globally across tasks, and locally within tasks. The most relevant baselines for this question seem like those that also apply model-based methods to the problem of multi-task learning – for example Nagabandi et al, 2018. I understand their focus was different, but the technique seems very similar and it would help to have better contextualization for why it is different technically, and whether those technical differences matter. As it stands, it was not clear to me if the paper’s main message was an argument for the specific method introduced, or about using model-based methods with planning as a more stable approach to life-long reinforcement learning in general (and the multiple levels of uncertainty just being critical to get this to work). In either case, presenting baselines from both model-based and model-free techniques (and carefully signposting these) would help clarify the message and interpretations of the results.\n* (Quality / Clarity) The number of experiments was reasonably small – only one page of results relative to 6 pages of method description. I think some of the proofs/lemmas could be safely moved to the supplement to make more space for results that better show advantages of multi-level uncertainty model-based vs. model-free for lifelong settings and leave more room for discussion around why multiple types of uncertainty are important empirically.\n* (Originality) While I am not aware of anyone applying exactly this kind of approach to the continual RL setting, the method feels similar conceptually to several approaches from meta-learning that could also be applied to a life-long setting (but are not included as baselines in this paper). \n* (Significance) I am not sure if this will impact how people do lifelong reinforcement learning, as it’s unclear how computationally expensive BNNs are, and whether the approach would work for settings in which the tasks cannot be clearly mapped onto a single parameter which is varying\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "M12SSivMFNh", "reviewer": "Reviewer_osbM", "summary": "This paper proposes an approach for lifelong learning for a set of parametrically varying tasks, by maintaining a joint world model across all tasks encountered, as well as task-specific models. In the absence of task-specific data, the prior of the task-specific models is set to the world model posterior. The paper includes theoretical analysis that shows that this leads to increased sample efficiency on the task, and also includes empirical evaluation on a set of gym tasks with parametric variation (eg: gravity, mass of body parts), as well as meta-world reaching. \n", "questions": "-> How much sample efficiency benefit does using the joint world model yield, as opposed to re-training from scratch with only the task-specific model?\n\n-> Does the approach scale to settings with more meaningful variation across tasks (eg- reach, push, pickplace) in metaworld? ", "limitations": "Limitation are not adequately discussed. As described before, the primary limitation is the quality of experimental domains considered. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Update after Author Response**\n\nI thank the authors for their detailed response. The authors have sufficiently addressed all of my concerns (described in detail below). In light of these clarifications, the  new experimental evidence included and given that the approach does seem original and intuitive, and the problem considered is of importance to the community (as noted in my original review), I am updating my score. With the new results, I think the paper will be of great relevance to the lifelong RL community.\n\n\n-> How much does the world model prior help? The authors clarify that this is actually included in the paper, though not clearly described in the writing (which can be easily amended for the camera ready version). Further, the authors add new experiments (appendix L) to directly compare to a version of their method using just a fixed prior, to highlight the benefit of using the world model prior.\n\n->  Can the method handle semantically varying tasks? Lifelong learning approaches will need to handle this setting when they're deployed for problems of interest in robotics. The initial experiments were run on parametrically varying tasks (different gravity, body mass etc) or transferring from reaching to reaching with a wall. I appreciate the effort of the authors in testing their approach on the much more challenging MT-10 and MT-50 benchmarks (Figure 5, Appendix H), which have actually different tasks (eg: opening a boxes/closing drawers/opening windows etc). These results make the claims of the paper \\emph{significantly} strengthen the claims of the paper, and demonstrate the effectiveness of the approach in comparison to current methods.  \n\n-> How does the method compare to robust policy ? The authors clarify that this considered in the paper that originally introduced the tasks considered in the initial experiments. Further, this concern is sufficiently addressed with the MT-10/MT-50 experiments. The authors also add discussion related to this in appendix Q. \n\n** Original Review ** \n\nSignificance  - \n \n-> Lifelong learning is an important challenge for RL, as learning from scratch for each new task is infeasible, especially for robotics. Proposed solutions are more likely to be adopted by the community at large if they show signs of progress on tasks closer to the actual problem setting, i.e a stream of semantically different tasks that are encountered over an agent’s lifetime. The main results in the paper are presented on a set of gym tasks (cheetah, walker) with very simple parametric variations (changing gravity, changing body mass etc). For such domains, since access to all previous data is assumed, one could train a single robust model-free policy that can perform quite well, using algorithms like SAC. Furthermore, the metaworld suite includes semantically different tasks (reaching, pushing, pick-place etc) and a better evaluation of the algorithm would include evaluating the approach on these variations. Without this it is difficult to understand whether the approach is actually applicable to the problems faced by RL for deployment. \n \n-> More broadly, part of the issue stems from considering the set of tasks to only have parametric variation. While this restriction might provide added structure to help in theoretical analysis, it is worth considering if the same general idea (joint world model which is used to initialize task-specific models) can also be used for more general sets of tasks, that do not vary in terms of an interpretable parameter. This would open up the approach for evaluation on semantically varying tasks, as even in this case there are similarities to be shared across tasks, such as physics. \n\nQuality - \n\n->  The empirical results don’t include sample efficiency comparison of the approach without using the world model to initialize the task-specific models. This seems important as it’s one of the central claims of the paper. Furthermore, as described before the experimental domains are too far removed from settings of interest for the deployment of RL for lifelong learning. \n\n-> The probabilistic formulation of the individual task models and the joint world model and the corresponding update schemes are sound  and intuitive to follow. The theoretical result that initializing the task specific prior to the joint world model posterior leads to sample efficiency gains also seems intuitive, given that the joint model is trained across tasks. \n \nOriginality -\n \n->  The proposed approach using world models for continual learning does seem new, and can potentially be very impactful since transition dynamics should be preserved across tasks, and this is exactly what the models learn. Further, the authors provide a probabilistic framework for learning on a particular task while leveraging prior knowledge from a joint model, and provide theoretical analysis for the advantage in doing so. The related work section is adequate and includes most prior work. \n \nClarity - \n \n-> The paper is clearly written and not too difficult to follow, and includes detailed algorithms that are sufficient for reproducing the method.\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-XYirE-RlS5", "reviewer": "Reviewer_E1cW", "summary": "* The authors tackle the problem of lifelong learning in RL and propose to use hierarchical Bayesian modeling to achieve transfer between tasks over the course of learning.  \n* In particular a world model will be used alongside task specific modeling to extract common and specific structure in existing tasks to model new tasks. \n* This facilitates modeling new tasks by using the between-task posterior as a within-task prior for newly sampled tasks. \n* Tasks are related and sampled sequentially throughout learning and generally depend on a task latent to define specific task features.  Tasks are modeled with HiP-MDPs.\n* The authors define forward transfer and backward transfer across tasks by using prior and posterior data to improve the performance across the set of sampled tasks.\n* They next derive a performance bound that shows that fewer samples are required to train specific task models as the world model converges on the underlying distribution of environment dynamics.\n* The authors further develop Variational Baysian exploration for Lifelong RL (VBLRL) for better scale by using variational inference to approximate a world model using Bayesian Neural Networks.  Both forward and backward (previously encountered tasks) task transfer is implemented to boost performance.\n* The authors experiment with this approach on MuJoCo and Meta-world tasks against baselines in lifelong learning and demonstrate state of the art results.\n\n**Claims**:\n\n 1. Demonstrate better forward and backward transfer performance against state of the art lifelong RL algorithms.\n 2. Bayesian modeling across tasks that improves sample efficiency.\n\n", "questions": "\n* Does forward and backward transfer always result in monotonically increasing performance?  Are there cases where the transfer is counter-productive to performance improvement? Finally, if so, under what conditions does this typically occur?\n* You show that sample efficiency improves in the task specific regime when the world model approaches the environment distribution, however is it always the case that the world model improves in this sense?  Is there reason to believe that this will be very limited for some environments?\n* Was any study undertaken of the number of particles and what effect this has on posterior sampling and learning convergence?  What impact does this have on the computational complexity?\n", "limitations": "I couldn't see much discussion about limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* Lifelong RL is an active area of research which presents a number of challenges that will require scalable approaches to overcome.  The authors target a task modelling approach that aims to address this through learning of the underlying dynamics with hierarchical Bayesian methods.\n* The authors rigorously demonstrate that task models can be trained with fewer samples as the world model posterior approaches the true underlying dynamics of the tasks environment.  \n* The authors cover prior work in using Bayesian methods in RL, lifelong RL, and Hip-MDP processes in their related work and provide some background to help contextualise their work.\n* Ablation shows BNNs and backward transfer improved performance.\n* Results look overall good and VBLRL improves over comparable baselines.  Overall the authors seemed to have satisfied the claims made, demonstrating that their VBLRL approach improves sample efficiency while attaining state of the art performance over a number of baselines.\n\n\n**Weaknesses**\n\n* There seem to be some missing references in sectionThere is no reference for VBLRL in section 1 and I could not find a reference for \"Bayesian Exploration for Lifelong Reinforcement Learning\" Fu et al. 2021.  There should be a reference here to prior work.  In the same section BNNs are referenced but no work is cited nor are they really explained.  In section 2, some explanation for both of these approaches would be helpful.\n* While the authors demonstrate that the approach is effective the novelty of it seems somewhat limited.  However, the authors provide some theoretical justification.\n* Section 4.2 seems like it could use more detail.  For example, figure 1 is fairly high level and doesn't seem to capture in a straightforward way the dynamics of the model learning across tasks.  The algorithms which explain the VBLRL in detail are in the appendix. I feel this section (and 4.3) are meant to really define the unified approach yet I didn't come away with a clear understanding of how back transfer and VBLRL are working here.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept