Bayesian Exploration for Lifelong Reinforcement Learning
Authors: Haotian Fu
Shangqun Yu
Michael Littman
George Konidaris
Venue: ICLR 2022 Submitted (2022)

Abstract: A central question in reinforcement learning (RL) is how to leverage prior knowledge to accelerate learning in new tasks.  We propose a Bayesian exploration method for lifelong reinforcement learning (BLRL) that aims to learn a Bayesian posterior that distills the common structure shared across different tasks. We further derive a sample complexity analysis of BLRL in the finite MDP setting. To scale our approach, we propose a variational Bayesian Lifelong Learning (VBLRL) algorithm that is based on Bayesian neural networks, can be combined with recent model-based RL methods, and exhibits backward transfer. Experimental results on three challenging domains show that our algorithms adapt to new tasks faster than state-of-the-art lifelong RL methods.

Under review as a conference paper at ICLR 2022
BAYESIAN EXPLORATION FOR LIFELONG REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
A central question in reinforcement learning (RL) is how to leverage prior knowl- edge to accelerate learning in new tasks. We propose a Bayesian exploration method for lifelong reinforcement learning (BLRL) that aims to learn Bayesian posterior that distills the common structure shared across different tasks. We further derive sample complexity analysis of BLRL in the finite MDP setting. To scale our approach, we propose a variational Bayesian Lifelong Learning (VBLRL) algorithm that is based on Bayesian neural networks, can be combined with recent model-based RL methods, and exhibits backward transfer. Experimental results on three challenging domains show that our algorithms adapt to new tasks faster than state-of- satte-of-theassd art lifelong RL methods.
INTRODUCTION
Reinforcementtleammigg (RL) methods (Sutton & Barto 1998 Kaelbling et al., 1996) have been successfully applied to solve challenging individual tasks such as learning robotic control (Duan et al. 2016) and playing expert-level Go (Silver et al. 2017). However, in the real world, a robot usually experiences collection of distinct tasks that arrive sequentially throughout its operational lifetime; learning each new task from scratch is infeasible, but treating them all as a single task will fail. Therefore, recent research has focused on algorithms that enable agents learn across multiple, sequentially posed tasks, leveraging past knowledge from previous tasks to accelerate the learning of new tasks. This problem setting known as lifelong reinforcement learning (Brunskill & Li 2014 Wilson et al., 2007b Isele et al. 2016b). The key questions in lifelong RL research are: How can an algorithm exploit knowledge gained from past tasks to improve performance new tasks (forward transfer), and how can data from new tasks help the agent to perform better on previously learned tasks (backward transfer)?
To answer these two questions, first consider simple problem, which is to find different items in different houses. Here, a single task corresponds to finding items in a specific house. Although items may be stored in different locations in different houses, there still exists some shared information that connects all houses. For instance, a toothbrush is more likely to be found in a bathroom than kitchen, and room without window is more likely to be a bathroom than a living room. Such information can significantly accelerate the search for items in newly encountered houses. We propose that extracting the common structure existing in previously encountered tasks can help the agent quickly learn the dynamics of the new tasks. Specifically, this paper considers lifelong RL problems parrame- MDPs or IiP--MDPs (Doshi-Vedd) & Konidaris 2016 Killian et al. 2017), where variations among the true task dynamics can be described by set of hidden parameters. We model two main categories of learning across multiple tasks: the woded-model distribution, which describes the probability distribution over tasks, and the task-specific model, that defines the (stochastic) dynamics within a single task. To enable more accurate sequential knowledge transfer, separate the learning process for these two quantities and maintain a hierarchical Bayesian posterior to approximate them. The world-model posterior is designed to manage the uncertainty in the world-model distribution, while the task-specific posterior handles the uncertainty from the data collected from only the current task.
that can be modeled
We propose Bayesian exploration method for lifelong RL (BLRL) that learns Bayesian world- model posterior that distills the common structure of previous tasks, and then uses it as a prior to learn task-specific model in each subsequent task. For the discrete case, derive an explicit

---

Under review as a conference paper at ICLR 2022
performance bound that shows that the task-specific model requires fewer samples become accurate as the world-mode posterior approaches the true underlying world-model distribution. We further develop VBLRL, more scalable version of BLRL that uses variational inference to approximate the world-model distribution and leverages Bayesian Neural Networks (BNNs) to build the hierarchical Bayesian posterior. Our experimental results set of challenging domains show that our algorithms achieve better forward and backward transfer performance than state-of-the-ar lifelong RL algorithms within limited samples for each task.
2 BACKGROUND
RL is the problem of maximizing the long-term expected reward of an agent interacting with an environment (Sutton & Barto, 1998). We usually model the environment
Markov Decision Process or MDP (Puterman 1994). described by five tuple: (S, A, R, T,T)), where S is a finite set reward function, with lower and upper bound O and 1; T S x A - Pr(S) is transition function, with T(s' |s, a) denoting the probability of arriving in state s' E S after executing action
of states; A is finite set of actions; R: S x A [0,
A in state s; and y E [0, 1) is discount factor, expressing the agent's preference for delayed over immediate rewards.
An MDP a suitable model for the task facing a single agent. In the lifelong RL setting, the agent instead faces a series of tasks Ti,..., Tu. each of which can be modeled as an MDP: (S A (2()) T(.) 7(0)) A key question is how these task MDPs are related; we model the collection of tasks HiP-MDP (Doshi-Velez & Konidaris 2016 Killian al. 2017), where family of tasks is generated by varying a latent task parameter drawn for each task according to the world-model distribution Pq. Each setting of specifies unique MDP, but the agent neither observes w nor has access to the function that generates the task family. Formally, then, the dynamics T(s'|s, a; w;) and reward function R(r s, a:; w:) for task i depend on W; E Q2 which fixed for the duration of the task. For lifelong RL problems, the performance of a specific algorithm is usually
evaluated based on both forward transfer and backward transfer results (Lopez-Paz & Ranzato 2017):
Forward transfer: the influence that learning task t has on the performance in future task k > t.
Backward transfer: the influence that learning task has on the performance in earlier tasks l: < t.
BAYESIAN EXPLORATION FOR LIFELONG REINFOR LEARNING
The key component of our approach is a hierarchical Bayesian posterior over task MDPs controlled by the hidden parameter w. Intuitively, we maintain probability distributions that separately capture two categories of uncertainty within lifelong learning tasks: the world-model posterior captures the epistemic uncertainty of the world-model distribution over different tasks controlled by the hidden parameter. As the learner is exposed tasks, this posterior should converge to the world-model distribution Pq. The task-specific posterior captures the epistemic uncertainty of the current task m. As the learner is exposed to more and more transitions within the task, this posterior should approach m, leaving only the aleatoric uncertainty of transitions within the task, which is independent of other tasks. By sampling from the world- model posterior, an agent can learn new tasks faster by exploiting knowledge common to previous tasks, thus exhibiting positive forward transfer.
@i
my
and more
R
Figure 1: Plate representation for the BLRL approach. denotes trajectory {s, a, s s') There are K different tasks and the agent samples R trajecto- ries from each task.
We first consider the finite MDP setting. Concretely, we use a hi- erarchical Bayesian model to represent the distribution over MDPs. Figure shows our generative model in plate notation. V the parameter set that represents dis- tribution Pq. It functions the world-model posterior that aims to capture the common structure across different tasks. The resulting MDP m; is created based on W;, which is one hidden parameter sampled from V. We sample from our approximation of V create and solve possible MDPs.
2

---

Under review as a conference paper at ICLR 2022
The proposed BLRL approach is formalized in Algorithm 3 in the appendix. Initially, before any MDPs are experienced, the world-model posterior ge (-|st, at) is initialized to an uniformed prior. For each new task m,, we first initialize the task-specific posterior 90 (-|s,, a,) with the parameter values from the current world-model posterior, and then, for each timestep, select actions using a Bayesian exploration algorithm based sampling from this posterior (Thompson 1933 Asmuth et al. 2009). A set of sampled MDPs drawn from qg mh is concrete representation of the uncertainty within the current task. BLRL samples K models from the task-specific posterior whenever the number of transitions from a state action pair has reached threshold B. Analogously to RMAX (Brafman & Tennenholtz. 2003) and BOSS (Asmuth et al., 2009). we call a state state-action pair known whenever it has been observed as.... B times. For each state-action pair, if it is known, we use the task- specific posterior to sample the model. If unknown, we instead sample from the world-model posterior. These models are combined into a merged MDP m. # and BLRL solves m. to get a policy n." " This approach is adopted from BOSS (best of sampled set) to create optimism in the
face of uncertainty, and thereby drive exploration. The new policy nt" " will be used to interact
with the environment until a new state-action pair reaches the sampling threshold. The collected transitions from this task will be used to update the task-specific posterior immediately, while the world-mode posterior will be updated using transitions from all the previous tasks at slower pace. For simple finite MDP problems practice, we use the Dirichlet distribution (the conjugate for the multinomial) to represent the Bayesian posterior. Thus, the updating process for the posterior is straightforward to compute. Intuitively, BLRL able to rapidly adapt to new tasks as long as the prior of the task-specific model (that is, the world-model posterior) is close to the true underlying model and captures the uncertainty of the common structure of a set of tasks.
3.1 SAMPLE COMPLEXITY ANALYSIS
We now provide simple theoretical analysis of BLRL. First, we use the setting and results of Zhang (2006) to describe the properties of the Bayesian prior and how it relates the sample complexity for the concentration of the Bayesian posterior.
Lemma 1. Let т (w) denote the prior distribution on the parameter space I.. We consider set of transition transition-robbabiltty densities p(-(w) w) indexed by w. and the true underlying density q. Define the prior-mass radius of the transition probabbiity densities as:
dg inf d - In ((p I'; DKz(gllp) d})}.
(I)
Intuitively, this quantity measures the distance between the Bayesian prior to initialize the posterior and the true underlying distribution. Then, vp E (0, 1) and " 2 1, let
1 nd, + (n- (o0-p))upprr.ddeee ((1 - 1)/(1 - p)), En = (1 +
(2)
n
where Supper.n is the critical upper-bracketing radius (Zhang 2006). The decay rate of e uupeer,n controls the consistency of the Bayesian posterior distribution AAsmut et al. 2009). Let p we have for 2 0 and 8 E (0, 1), with probability at least 0,
(f ( ppe:: 1|p 11/l//222 2cn + (4n - 2) {x) S 1 en
(3)
8/4
Proof (sketch). The proof is similar to that of Corollary 5.2 of Zhang 2006) (see Appendix A.5). Instead of using the critical priss-mass radius εn., describe certain characteristics of the Bayesian prior, we define and use the prior-mass radius dz, which is independent of the sample size n and measures the distance between the prior and true distribution.
Similar to BOSS, for a new MDP m M with hidden parameters Wm. we can define the Bayesian concentration sample complexity for the task-specific posterior: f(s, a, cg, do, Po). as the minimum number such that, if IID transitions from (s, a) are observed, then, with probability at least 1 - do.
Pr Prm-sposterior (I a, Wm) Ten Em-(8,, a, (mwmm)))) 60) PD-
(4)
3

---

Under review as a conference paper at ICLR 2022
Lemma 2. Assume the posterior is consistent (that is, Eupper,n o(1) and set 17 2, then the Bayesian concentration sample complexity for the task-specific posterior f(s, a, e, 6) = da d0
O
Proof (sketch). This bound can be derived by directly combining Lemma and Equation
The above lemma suggests upper bound of the Bayesian concentration sample complexity using the prior-mass radius. We can further combine this result with PAC-MDP theory (Strehl et al. 2006 and derive the sample complexity of the algorithm for each new task.
Theorem 1. For each new task, set the sample size K = @( A In SA) and the parameters (1 --)) 000 S0:Po = 3-ARK. then, with probability least 1 46, VA 2 V*(s() 4eo in all but Ó S'A'd= steps, where Ö(-) suppresses logarithmic dependence.
5
8
(1-)))))
Proof (sketch). The proof based on the PAC-MDP theorem (Strehl et al. 2009) combined with the new bound for the Bayesian concentration sample complexity we derived in Lemma 2. In general, we use the same process in BOSS to verify the three required properties of PAC-MDP: optimism, accuracy and learning complexity. For each new task, the main difference between BLRI and BOSS is that we use the world-model posterior to initialize the task-specific posterior, which results in new sample complexity bound based on the prior-mass radius.
The result formalizes the intuition that, if we put a larger prior mass at a density that is close to the true q such that d, is small, the sample complexity of our algorithm will be lower. In the meantime, the sample complexity is bounded by polynomial functions of the relevant quantities, showing that our training strategy preserves the properties required by PAC-MDP algorithms (Strehl et al. 2009).
SCALING UP: VARIATIONAL BAYESIAN LIFELONG RL
Directly computing the exact posterior typically not possible for large scale problems. Instead, we propose a practical approximate algorithm, VBLRL, that uses neural networks and variational inference (Hinton & van Camp 1993a). We model the posterior via the transition dynamics using p(81+1, Te Te|3,, a,; 0), 00,00 E O The posterior, given new state-action pair, can be rewritten via Bayes' rule:
p(0) ae 8441, p(0) (i)D a.;
((5441.re re,D,,,,)
where D; is the agent's history with all the experienced tasks up until time step t. As representing the posterior p(0) D) is intractable, we approximate it through an alternative distribution q(f: Φ) by minimizing D (0; ()]lp(() D)), leveraging variational lower bounds (Hinton & van Camp. 1993b Houthooft et al. 2016).
We choose Bayesian neural networks (BNN) to approximate the posterior. The intuition is that, the context of stochastic outputs, BNNs naturally approximate the hierarchical Bayesian model since they also maintain a learnable distribution over their weights and biases (Graves) 201T Houthooft al., 2016). We expect the uncertainty embedded in the weights and biases of networks can capture the epistemic uncertainty introduced by hidden parameters of different tasks, while we also set the outputs of the neural networks be stochastic capture the aleatoric uncertainty within each specific task. In our case, the BNN weights and biases distribution q(); Φ) can be modeled as fully factorized Gaussian distributions (Blundell et al. 2015):
(0: Φ) = II N ((O0)l4.0))
(6)
where o = (u, and μ is the Gaussian's 's mean vector while a is the covariance matrix diagonal. Then, the posterior distribution over the model parameters can be computed through:
d arg min [DKZI9(0: p()) (oog ( Dr. 0)]
(7)
4

---

Under review as a conference paper at ICLR 2022
where p(0) represents the fixed prior distribution of A. The second term the right hand side can be approximated through & Σ Ei=1 log p(5 Dz ae; 0,) with N samples from 0g g This optimization can be performed in parallel for each 8, keeping Φ:-1 fixed.
World-model posterior
Epistemic uncertainty
Pa
Aleatory uncertainty
Fd
1
A
:
dd
Episternic uncertainty
Task-specific posterior
Wm
Pd
P(s' rls,, az wm)
Aleatory
uncertainty
Figure 2: How VBLRL estimates different kinds of uncertainties in HiP-MDP. The world-model posterior captures the epistemic uncertainty of the general knowledge distribution (shared across all tasks controlled by the hidden parameters) via the internal variance of world-model BNN. As the learner is exposed to more and more tasks, the posterior should converge to Pa. The task-specific posterior captures the epistemic uncertainty of the current task m, which comes from the alleatory uncertainty of the world model when generating w/m. for a new task, via the internal variance of task-specific BNN. The posterior should output the highest probability for w near the true wm as the agent collects enough data from the task. The alcatory uncertainty of the final prediction is measured by the output variance of the prediction.
We provide the intuition of how our design capture the uncertainties of lifelong RL in Figure 2 and summarize the method in Algorithm|] T. The left side of the figure shows the process of how transitions are generated from the environment's true distribution, while the other parts show how our models generate transition predictions and how they separately estimate different uncertainties generating from approximating the true underlying distribution. We employ our posterior knowledge models in the context of model-based RI. method. When encountering a new task, VBLRL first uses the model parameters (that is, {u4 o of weights and biases of BNN) from the general knowledge model to initialize the task-specific posterior network. The task-specific model outputs the predicted next state and reward given a state-action pair. Then, we use model preddccive control Garriia et al. 1989) to select actions based the generated transitions.
For planning, at each step, we begin by creating P particles from the current state s = sp Then sample N candidate action sequences a+++T from learnable distribution. We propagate the state-action pairs using the learned task-specific model Pm, (-1s, a) (BNN) and use the cross entropy p-m,(-1s, method (Botey et al. (203)) to update the sampling distribution make the sampled action sequences close to previous action sequences that achieved high reward. We further calculate the cumulative reward estimated (via the learned model) for previously sampled sequences and use the mean of that distribution to select the current action.
The spec-ip posterior is updated using the data collected from only the current task. The world-model posterior is updated after a few more steps with the collected transitions from all the visited tasks. The intuition to guide the two posteriors to separately learn two categories of uncertainty within lifelong learning tasks. Note that other CEM-based based model-based RL algorithms like PETS (Chua et al. (2018) usually maintain a set of neural networks using the same training data, and sample action sequences from each of the neural nets to achieve randomness in transitions. Besides the problem of requiring special training tricks, it is unrealistic to maintain (2 30) models for each task in lifelong RL settings. Our usage of BNNs avoids such problems one neural network using the same data for each task, and we can sample an unlimited number of different action sequences to more possibilities needed. In PETS, the epistemic uncertainty
only have train
5

---

Under review as a conference paper at ICLR 2022
is estimated via the variance of the output mean of different neural networks, while in VBLRL, it is estimated via the variance of the weights and biases distribution of BNN during training. This is implied the objective function we use to update Φ, is from minimizing D (q(;; ()]
Algorithm 1: Variational Bayesian Lifelong RL
Initialize general knowledge model Purm (-1s, (,5,;; Owm), replay buffer Dmps'" for each task m; from i = 1, 2, ... .M do
Dm
Initialize task-specific model Pm, (+|,
J,s a; 0m.) with general knowledge model pum
for each episode do TaskHorizon do Sample Actions Ou:TT ~ CEM(.)
for Time t
Propagate state particles sy with Pin: (5'1s,a) Evaluate actions as Σ p5 2p=1 Pmn (r|8, a)
Update CEM()) distribution.
Execute optimal actions 422+T
end
Add transitions to replay buffer D.,.
Update task-specific model according to Equation 7 given Dmn
Update general knowledge model according to Equation given (Dmys'.. D
end
end
4.1 BACKWARD TRANSFER OF VARIATIONAL BAYESIAN LIFELONG RL
In our lifelong RL setting, the agent interacts with each task for only limited number of episodes and the task-specific model stops learning when the next task is initiated. As result, there may exist portions of the transition dynamics in which model uncertainty remains high. However, as the world-model posterior continues to train new tasks, it gathers more experience in the whole state space and can provide improvements its guesses concerning the "unknown" transition dynamies, even for previously encountered tasks.
Intuitively, the performance of an agent on one task has the potential to be further improved (positive backward transfer) if there exists sufficiently large set of state-action transition pairs of which the task- task-specific model's predictions are not confident due to lack of data. This type of model uncertainty is sometimes called epistemic uncertainty (Kiureghian & Ditlevsen, 2009 Ciosek al., (2220). In our algorithm, the aleatory variability (irreducible chance in the outcome) is measured by the output variance of the prediction {a,e, oze ), and the epistemic uncertainty (due lack of experience) corresponds to the uncertainty of the output mean and variance (see Definition 1 below). Thus, a straightforward method improve previously learned task-specific model is to find the predictions it needs to make that have high epistemic uncertainty, and replace them with the predictions from the world-mode posterior, which has lower epistemic uncertainty. we only consider reward prediction, the conditions for measuring whether a task task-specifie model is sufficiently confident are as follows.
Definition 1. Assume there exist known constants 5p,, &z For given itte-action pair (s, a). the task-specific model (reward) is proclaimed confident when the following conditions are satisfied:
Ep (u. -1,g)² P
Ep- o,.) P-1
<Snm,,
So..
where P is the number of particles. Similar definition applies to the task-specific model's next-state prediction. Intuitively, ε,., and da. function as the threshold to judge whether the uncertainty of the output mean or variance for each dynamic prediction is too high to be called confident prediction.
The detailed backward transfer testing algorithm can be found in the appendix. In practice, it is often hard to find specific confidence thresholds (6 6,,, 50,, 8_.) that are effective. Instead, we implement simpler approach: During planning, for each prediction, the output mean and variance of the world model and the task-specific model, and then choose the one with lower values, which indicates higher confidence level.
compare the uncertainty of

---

Under review as a conference paper at ICLR 2022
5 EXPERIMENTS
5.1 GRID- WORLD ITEM SEARCHING
We first evaluate BLRL in simple World domain. Our testbed consists of collection of houses, each of which has four rooms. The goal of each task is to find specific object (blue, green or purple) in the current house. The type of each room is sampled based on an underlying distribution given by the environment. Each room type has corresponding probability distribution of which kind of objects can be found in rooms of this type. Different tasks/houses vary in terms of which rooms are which types and precisely where objects are located in the room (the task's 's hidden parameters).
To simplify the problem, instead of modeling the whole MDP distribution, we use BLRI to model the object distribution the Bayesian posterior and sample MDPs from the distribution. We use BOSS with fixed prior (no intertask transfer) as our baseline. The average training performance of all 300 tasks are shown in Figure top right. Each task consists of 10 epochs, with 21 sample steps for each epoch. Within the limited steps allotted for each task, BLRL able to discover and transfer the common knowledge and helps the agent quickly adapt comparison, running BOSS with a fixed prior is able to find the optimal policy eventually but needs more sample steps and learns more slowly than BLRL.
tasks the training goes In
BOSS
5R
Tpoch
10L5.
UK P1W
-50
10
25
Figure 3: Top-left: Grid-World Item Searching; Top-right: Foid-World Item Searching evaluation
results; Bottom-left: Box-jumping Task; Bothom-right: Box-jumping Task evaluation results.
5.2 BOX-JUMPING TASK
We use simplified version of the jumping task (des Combes 2018) as testbed for the proposed algorithm VBLRL. As shown in the screen by jumping over the obstacle. The agent only choose from two actions: jump and right. It will hit the obstacle unless the jump action is chosen at precisely the right time. We set different obstacle positions as different tasks, constituting the HiP-MDP hidden parameters. The 4-element state vector describes the (z, y) coordinates of the agent's current position, and its velocity in the and y directions.
left, the goal of the agent is to reach the right side of
Figure 3 bottom right presents the average performance during training across all 300 tasks. Each task run for 30 episodes. We compare VBLRL against state-of-the-ar lifelong RL methods LPG- FTW (Mendez et al. 2020) from the multi-model category, EWC (Kirkpatrick al. 2016) from the single-model category. singel-task model-base RL baseline (using the same BNN structure and well as HiP-MDP baseline (Killian et al. 2017). For fair comparison, further replace the DDQN algorithm (van Hasselt et al. 2016) used in Killian et al.' 's paper with CEM planning, and let the transition model also predict the reward for each state-action pair. This modified
planning procedures)
7

---

Under review as a conference paper at ICLR 2022
baseline is similar to the single-model version of VBLRL (i.e., only using the world-model posterior). VBLRL clearly learns faster than the HiP-MDP baseline and reaches better final performance. Thus, in the lifelong RL setting, separating the updating processes of the world-model posterior and the task-specific posterior can lead to better learning efficiency.
5.3 OPENAI GYM MUJoCo DOMAINS
We evaluate the performance of VBLRL on HiP-MDP versions of several continuous control tasks from the Mujoco physics simulator (Todrrov et al., 2012). HalfCheetah--raaye HalfCheetah- bodyparts, Hopper-gravity, Hopper-bodyparts, Walker-gravit Walker-budyparts, all of which are lifelong-RL. benchmarks used in prior work (Mendez et al. 2020). For each of six different domains, the task-specific hidden parameters correspond to different gravity values different sizes and masses of the simulated body parts. More details can be found in the appendix. Compared with prior work, substantially reduced the number of iterations that the agent sample and train on (100 iterations for each task and a horizon of 1 100/200 for each iteration). We used such settings to increase the difficulty of lifelong learning and ensure that fearning-from-seraacch algorithm could obtain good policy in the available training time.
VBLKL
HiP-MDP baseline
LPG-FTW
EWC
Single-task MBRL
CG-Start
160.68 + 48,
126.95 + 31.41
-81.39 + 9.18
-3426.70 827
-83.96 00.10 400.47 10.68
CG-Train
226.72
26.53
170.20
39.92
-29.49
11.03
-3440.66 = 1007.50
CG Back
231.79
23.49
97.84
22.04
299.95
11.64
-6672.33 1. 3748.63
Start
110.74 + 41.96
78.95
18.43
263.994 40.80
5016.93 = 1708.
-101.00 4: 39.11
CB-Train
173.97
78.26
87.2
217.86 42.88
5544..2 2145.82
33.24
CB Back
181.60 :: 67.50
67.50
116.03 +: 17.35
-116.41 65.64
13889.31 + 6651105
HG-Start
-149.79 28.7
-130.54
14.86
16.15 22.83
-614.80 = 600014
-408.46- + 36.10
HG-Train
-125.40
-110.19 22.03
22.98
31.34
-999.49 63-1.01
-386.77 + 65.77
HG-Back
55.95 1:5:::441 74.32
-142.82 16.30
2222336
106.91
5816.74 = 4103.66
Start
119.29
-33.41 36.99
15.56
48.63
4701.72 +: 1527.74
-00200 1. 30.03
HB-Train
.99.00
96.668 - 38.725
41.29 + 12.19
7384.54 = 3232.86
-394.77 77 4: 25.77
BtckBack
76.47 13.27
92.45
7921.96 1: 1147.94
WG-Start
- -19.53
20.86 5.337
200.35 + 70.95
-467.54 = 249.19
-440.89 + 59.41
WG-Train
6.38
1.57
-91.31 61.36
-361 = 260.76
3359 82 + 45.11
WG-Back
18.56 7.42
9.79 6.79:6633
-90.41 2 61.66
T34.90 = 413.40
WB Start
43.01
10.82
64.62 1: 28.02
315.60 1: 31.66
1144.62 180.21
-437.9 1 26.24
WR-Train
2.66 3.67
31.29
30.07
187.96 + 72.03
1131.03 451.38
-367.79 1: 74.37
WB-Back
4.04 2.95
33.75 # 40.00
-66.97 + 74.71
-- 2-222.700 7063.70-2 = 692.70
Table I: Results on OpenAl Gym Mujoco domains. CG denotes Cheetah-Gravity, CB denotes Bheetab-Bodyparts, HG denotes Hopper-Gravity, HB denotes Hopper-Bodyparrs, WG denotes Wraker- WB denotes Walker-Bodyparts.
The results are shown in T We compare our algorithm against the three algorithms described in Section 5.2] For all six domains, we report the average performance of all the tasks at the beginning of training (Start) and after all training for each new task (Train), as well as the average performance for all previous tasks after training for a given number of tasks, which is the backward transfer test (Back). As shown in the results, our method VBLRL shows better performance of the HalfCheetah domain and Walker domain, as well as better backward transfer performance on Hopper-gravity and Hopper-bodyparts than the other three algorithms. LPG-FTW exhibits better forward training performance in the Hopper domain, but still shows some signs of catastrophic forgetting as there a huge gap between its training performance and backward performance. EWC fails in most of the tasks as the tasks are diverse and we set very limited sample steps for each task, which means it is hard directly learn single shared policy that achieves good performance. The HiP-MDP baseline shows good results some of the tasks because it is more sample-efficient to learn a shared model across all the tasks and easier to capture the world-model uncertainty. However, it cannot achieve as good performance using only one model across all tasks, which also leads to the negative backward transfer performance on Cheetah-Gravity Comparing VBLRL's performance the Train stage and Back stage, also find that it shows positive backward transfer results on most of the tasks, without showing patterns of catastrophic forgetting. Overall, VBLRL's world-model posterior contributes better forward
all three test stages
VBLRL it is hard to model the task-specific uncertainty
We changed the environment settings for Hopper and Walker to make them amenable to model-based RI. following Wang et al. (019))
8

---

Under review as a conference paper at ICLR 2022
transfer performance (Start), the learning of task-specific posterior contributes to better forward transfer training for each new task (Train), and the combination of these two posteriors guides the agent to achieve better backward transfer performance (Back).
6 RELATED WORK
HiP-MDPs (Doshi Velez & Konidaris 2016 provide framework for studying lifelong RL. Published HiP-MDP methods use Gaussian Processes (Doshi-Vele) & Konidaris 2016) or Bayesian neural networks (Killian et al. 2017) to find a single model that works for all tasks, which may trigger catastrophic forgetting. Other single-model lifelong RL algorithms encourage transfer across tasks by modifying objective functions. EWC (Kirkpatrick et al.) (016) imposes quadratic penalty that pulls each weight back towards its old values by an amount proportional to its importance for performance on previously-Iearned tasks avoid forgetting. There are several extensions of this work based on the core idea of modifying the form of the penalty (Li & Hoiem 2017 Zenke et al. 2017] Nguyen et al.) 2018). Another category of lifelong RL methods uses multiple models with shared parameters and task-specific parameters to avoid or alleviate the catastrophic problem (Bou-Amman jet al. 2014 Isele et al., 2016a Mendez et al. 2020). The drawback of this method is that it is hard to incorporate the knowledge learned from previous tasks during initial training on new task (Mendez) cct al. 2020) Nagabandi et al. (2019) introduce a model-based continual learning framework based on MAML, but they tocus on discovering when new tasks were encountered without access to task indicators.
Research in Meta-RL (Wang et al. 2016; Finn al. 2017) and multi RL PParisotto et al. 2016 Teh et al. 2017) settings also attempts to find ways to accelerate learning by transferring knowledge from different tasks. Some work employs the MAML framework with Bayesian methods to learn a stochastic distribution over initial parameter (Yoon et al., 2018 Grant et al.] 2018: Finn et al. 2018). Other work uses the collected trajectories to infer the hidden parameter. which is taken as an additional input when computing the policy (Rakelly et al.) 2019: Zintgraf et al. 2020 Fu et al. 2021). Our method, however, focuses on problems where the tasks arrive sequentially instead of having a large number of tasks available at the beginning of training. This sequential setting makes hard to accurately infer the hidden parameters, but opens the door for algorithms that support backward transfer. Further, our method approximates the true HiP-MDP model by learning the Bayesian posterior over past tasks and uses it to initialize a model for each new task, encouraging the agent to explore places where the epistemic uncertainty of the world model is high.
Some prior work uses Bayesian methods in RL to quantify uncertainty over initial MDP mod- els (Ghavamzadeh et al. 2015 Asmuth & Littman 2011 Guez et al., 2012). Several algorithms start from the idea of sampling from a posterior over MDPs for Bayesian RL, maintaining Bayesian posteriors and sampling one complete MDP (Strens 2000 Wilson et al., (007a) multiple MDPs
from this distribution (Asmuth et al. 2209). Instead of focusing on single-task RL, our algorithm aims find a posterior over the common structure among multiple tasks. Wilson al. (2007a) uses hierarchical Bayesian infinite mixture model to learn a strong prior that allows the agent to rapidly infer the characteristics of new environment based on previous tasks. However, only infers the category label of a new MDP and uses that information to find parameter values. Moreover, their method only works in discrete settings and cannot be applied to the kind of continuous problems we included in our evaluation. Lifelong learning has also been widely studied within the supervised learning domain with explicit performance bounds [Baxter] 2000, Pentina & Lampert 2014]. Our work is one of the first papers to give explicit sample complexity bounds for lifelong reinforcement learning algorithm, where data efficiency is essentially important.
CONCLUSION
To address the lifelong RL problems, our work proposed to distill the shared knowledge from similar MDPs and maintain a Bayesian posterior to approximate the distribution derived from that knowledge. We gave a sample complexity analysis of the algorithm in the finite MDP setting. Then, we extended method to variational inference, which scales better and supports both backward and forward transfer. Our experimental results show that the proposed algorithms enables faster training on new tasks through collecting and transferring the knowledge learned from preceding tasks.

---

Under review as a conference paper at ICLR 2022
REFERENCES
J. Asmuth and M. Littman. Learning is planning: Near Bayes-optimal reinforcement learning via Monte-Carlo search. In UAI, 2011.
John Asmuth, Lihong Li, Michael L. Littman, Ali Nouri, and David Wingate. A Bayesian sampling approach to exploration in reinforcement learning. In UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty Artificial Intelligence, 2009, pp. 19-26. AUAI Press, 2009.
Jonathan Baxter. A model of inductive bias learning. J. Artif. Intell. Res., 12:149-198, 2000.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. CoRR, abs/ 2015.
Z. Botev, Dirk P. Kroese, R. Rubinstein, and P. L'Ecuyer. Chapter 3 the cross-entropy method for optimization. Handbook of Statistics, 31:35-59, 2013.
Haitham Bou-Ammar, Eric Eaton, P. Ruvolo, and Matthew E. Taylor. Online multi-task learning for policy gradient methods. In ICML, 2014.
Ronen I. Brafman and Moshe Tennenholtz. R-max a general polynomial time algorithm for near-optimal reinforcement learning. 3(null), 2003. ISSN 1532-4435.
Emma Brunskill and Lihong Li. Pac-inspired option discovery in lifelong reinforcement learning. In ICML,, 2014.
G. Carpenter and S. Grossberg. The art of adaptive pattern recognition by a self-organizing neural network. Computer, 21:77-88, 1988.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, pp. 4759 4770, 2018.
Kamil Ciosek, Vincent Fortuin, Ryota Tomioka, Katja Hofmann, and Richard E. Turner. Conservative uncertainty estimation by fitting prior networks. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
Remi Tachet des Combes, Philip Bachman, and Harm van Seijen. Learning invariances for pol- icy generalization. In 6th International Conference on Learning Representations, ICLR 2018.
OpenReview.net, 2018.
Finale Doshi Velez and George Dimitri Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 1432-1440. IICAL/AAII Press, 2016.
Yan Duan, Xi Chen, Rein Houthooft, John Schulman, and Abbeel. Benchmarking deep reinforce- ment learning for continuous control. In ICML, 2016.
Chelsea Finn, P. Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ICML, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic madel-agnostic meta-learning. In Advances in Neural Information Processing Systems 31: Annual Conference Neural Information Processing Systems 2018, NeurIPS 2018, pp. 9537-9548, 2018
Robert M. French. Catastrophic interference connectionist networks: Can it be predicted, can be prevented? In Advances in Neural Information Processing Systems 6, [7th NIPS Conference, 1993), pp. 1176-1177 Morgan Kaufmann, 1993.
Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng. Dong Li, and Wulong Liu. Towards effective context for reinforeenfr learning: approach based contrastive learning. In Thirty-Fiftt AAAI Conference on Artificial Intelligence, AAAI 2021, pp. 7457-7465. AAAI Press, 2021.
10

---

Under review as a conference paper at ICLR 2022
C. Garcia, D. M. Prett, and M. Morari. Model predictive control: Theory and practice survey. Autom., 25:335-345 348, 1989.
Mohammed Ghavamzadeh, Shie Mannor, Joelle Pineau, and Aviv Tamar. Convex optimization: Algorithms and complexity. Foundations and Trends@ in Machine Learning, 8(5-6):359 483, 2015. ISSN 1935-8245.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas L. Griffiths. Recasting gradient-based meta-learning hierarchical bayes. In 6th International Conference Learning Representations, ICIR 2018. OpenReview.net, 2018
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 201 I, pp. 2-56,, 2011.
A. Guez, D. Silver, and Dayan. Efficient bayes-adaptive reinforcement learning using sample-based search. In NIPS, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pp. 1856-1865, PMLR, 2018.
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual ACM Conference on Computational Learning Theory, COLT 1993, pp. 5-13. ACM, 1993a.
Geoffrey E. Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In COLT '93, 1993b.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. VIME: variational information maximizing exploration. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, pp. 1109-1117, 2016.
David Isele, Mohammad Rostami, and Eric Eaton. Using task features for zero-shot knowledge transfer in lifelong learning. In Proceedings of the Twenty-Fifh International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 1620-1626. ICCAVAAA Press, 2016a.
David Isele, Mohammad Rostami, and Eric Eaton. Using task features for zero-shot knowledge transfer in lifelong learning. In IJCAI, 2016b.
Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore. Reinforcement learning: A survey, J. Artif. Intell. Res., 4:237-285, 1996.
Taylor W. Killian, Samuel Daulton, Finale Doshi- Doshi-Velez, and George Dimitri Konidaris. Robust and efficient transfer learning with hidden parameter markov decision processes. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp.
2017.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. CoRR, abs/1612.00796, 2016.
Demis
Armen Der Kiureghian and O. Ditlevsen. Alcatory or epistemic? does it matter? Structural Safety, 31:105-112, 2009.
Zhizhong Li and Derek Hoiem. Learning without forgetting, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 6467-6476, 2017.

---

Under review as a conference paper at ICLR 2022
Jorge Armando Mendez Mendez, Boyu Wang, and Eric Eaton. Lifelong policy gradient learning of factored policies for faster training without forgetting. ArXiv, abs/207..0001. 2020.
Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta-learning Continual adaptation for model-based ArXiv, abs/1812.07671, 2019.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning, 2018.
Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In 4th International Conference Learning Representations, ICLR 2016, 2016.
Anastasia Pentina and Christoph H. Lampert. A pac-bayesian bound for lifelong learning. ArXiv, abs/1311.2838, 2014.
Martin L. Puterman. Markov Decision Processes-Discrees Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off- policy meta-reinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 5331-5340. PMLR, 2019.
Anthony Robins. Catastrophic forgetting, rehearsal and pscudorchearsal. Connect. Sci., 7:123-146, 1995.
D. Silver, Julian Schrittwieser, K. Simonyan, Ioannis Antonoglou, Aja Huang, Guez, Thomas Hubert, Lucas baker, Matthew Lai, A. Bolton, Yutian Chen, T. Lillicrap. Fan Hui, L. Sifre, G. V. D. Driessche, Graepel, and D. Hassabis. Mastering the game of go without human knowledge. Nature, 550:354-359, 2017.
A. Strehl, Lihong Li, and M. Littman. Incremental model-based learners with formal learning- time guarantees, ArXiv, abs/1206.6870, 2006.
Alexander L. Strehl, Lihong Li, and Michael L. Lituman. Reinforcement learning in finite mdps: Pac analysis. J. Mach. Learn. Res., 10:2413-2444 2009.
Malcolm J. A. Strens. A Bayesian framework for reinforcement learning. In Proceedings of the
Seventeenth International Conference on Machine Learning (ICMI. 2000), pp. 943-950, 2000.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 1998.
Yee whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 4406-4506, 2017.
William R. Thompson. On the likelihood that one unknown probability exceeds another view of the evidence of two samples. Biometrika, 25:285-200 2944,, 1933.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model--asee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, pp. 5026-5033 IEEE, 2012.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 2016, pp. 2094-2100. AAAI Press, 2016.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Rémi Munos, Charles Blundell, Dharshan Kumaran, and Matthew Botvinick. Learning to reinforcement learn. CoRR, abs/161 0,0563,,, 2016.
12

---

Under review as a conference paper at ICLR 2022
Tingwu Wang, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. Benchmarking model-based reinforcement learning. CoRR, abs/1997.02007, 2019.
Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: A hierarchical Bayesian approach. In Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), 2007, volume 227 of ACM International Conference Proceeding Series, pp. 1015-1022. ACM, 2007a.
Aaron Wilson, Alan Fern Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In ICML '07, 2007b.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian model-agnostic meta-learning. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, pp. 7343-7353 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence, 2017.
Tong Zhang. From -entropy to kl-entropy: Analysis of minimum information complexity density estimation. Annals of Statistics, 34:2180-2210, 2006.
Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL. via meta-learning In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
13

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 BACKWARD TRANSFER
We take backward transfer to be the influence of subsequent learning on the agent's performance of previous task. Negative backward transfer, which is also known as catastrophic forgetting, describes the scenario in which the agent's progress on task is erased after learning new tasks (French) [993] Carpenter & Grossberg, 1988 Robins 1995). Our algorithm avoids catastrophic forgetting as it maintains separate task specifceiific model for each task. In the mean time, there are circumstances in which VBLRL exhibits positive backwards transfer as introduced in the main text.
Algorithm 2: Variational Bayesian Lifelong RI (Backward transfer)
input: Test task m,, model uncertainty thresholds 5p,, 5p.. da,, S0, for Time 1 = 0 to TaskHorizon do to do Sample Actions 01:t+ ~ CEM())
for Trial k
for each action do
Propagate state particles si and reward r with Tm, (s', rl,, rrss
if T (s', a) is not confident (Definition 1) then Repropagate state particles sf. and reward r.. with Te(s', r)),,))
end end
Evaluate actions as M+t F Ep' (rls,a)))
Update CEM(-) distribution. end
Execute optimal actions G74T
end
The difference between the backward transfer version and the forward training version is that, when we do predictions for the state particles, each time we will also calculate the confidence level (Definition 1) of the task-specific model for this particular transition. If the task-specific model is not confident enough for this state-action pair, we will use the world-model instead and recalculate the prediction value.
In practice, it is often hard to find specific confidence thresholds (6 6,,- 6G., 5g.) that are effective. Instead, implement a simpler approach: During planning, for each prediction, we compare the uncertainty of the output mean and variance of the world model and the task-specific model, and then choose the one with lower values, which indicates higher confidence level. Here is link the The video shows the agent trained by our algorithm learns some meaningful behaviors the current setting within limited steps. and performs better compared with the behaviors learned by other baseline algorithms.
learned behaviors of VBLRL "BLRL/atts: //yutu/yuuuuuuu/uu/ be PRRAA699vva
A.2 BNN MODEL
As we model the transition models as Gaussian distributions:
T(-ss, a) = NJ (s, a), (s,
The function fe represented as Bayesian neural network parameterized by 8, which is further modeled as the posterior distribution parameterized by o, predicts the mean Mas Hr and variance a. Fr given current state and action 8, a. To make it clearer, we can view the BNN model in VBLRL as infinite neural network ensemble by integrating out its parameters:
/ T(s', r|s, ())(O; o)d0
T(s)r)),,a)
(10)
Compared to previous model-based algorithms that use finite number of neural network ensembles (e.g. PETS), our choice of BNN is more suitable for lifelong RL as we only need to maintain one neural network for each task, and can sample unlimited number of predictions from it which
better estimates the uncertainty and is essential in our setting where both dynamic function and reward function given unlike prior model-based RL. methods.
14

---

Under review as a conference paper at ICLR 2022
A.3 BLRL ALGORITHM
Algorithm 3: Lifelong Bayesian Sampling Approach Algorithm
input:K, B
initialize MDP set, the world-model posterior ge (s Te/st az);
for each MDP m, do
Naa - 0, U,V.
do sample TRUE:
initialize the task-specific posterior 0g (8t+1, 82 ax) < go (0 re ar);: for all timesteps 2. do
if do sossmmlle then
Sample K models m4, Mig from the task-specific posterior
9e 1i Merge the models into the mixed MDP m
($t+1,1 Te a
Solve m. to obtain πt"
do samalee + FALSE
end
Use x- for action selection: a, x m (st) and observe reward ry and next state 814 m. Na,a, +- as, + 1;
Update the task-specific posterior distribution o0 (s1+1, Te|se, ar) for the current MDP; IfN., No..a B then
Update the world-model posterior distribution (e($41,77oss, ae) with the collected
transitions;
damsamele TRUE
end
end
end
A.4 EXPERIMENTAL SETTING
A.4.1 GRID-WOO WORDD ITEM SEARCHING
Our testhed consists of a collection of houses, each of which has four rooms. The goal of each task is to find specific object (blue, green purple) in the current house. The type of each room sampled based on an underlying distribution given by the environment. Each room type has corresponding probability distribution of which kind of objects can be found in rooms of this type. Different tasks/houses vary in terms of which rooms are which types and precisely where objects are located in the room (the task's hidden parameters). Room types sampled from joint distribution.
Room type probability
Room
Room 2
Room 3
Room
Top-left
0.4
0.4
0.2
Bottom-left
0
0.8
0
0.2
Top-right
0.1
0
0
0.9
Bottom-right
0.8
0.2
Table 2: Room type probability distribution
A.4.2 BOX-JUMPING TASK
We simplified version of jumping task (des Combes al., 2018) simple testbed for the proposed algorithm VBLRL. We select a random position of obstacle between 15 ~ 33 for each task. The 4-element state vector describes the (z, y) coordinates of the agent's current position, and its velocity in the r and directions. The agent can choose from two actions: jump and right. The reward function for this box box-jumping task
R₂ T(8 reach the right wall) T{Se+1 hit the obstacle) + ir I(8441 not hit the obstacle) (11)
15

---

Under review as a conference paper at ICLR 2022
Object type probability
Blue ball
Green box
Purple box
Room 1
0.3
Room 2
Room 3
0.6
Room 4
0
Table 3: Object type probability distribution
A.4.3 OPENAI GYM MUJOCO DOMAINS
Figure 4: Left: Walker-2D; Middle: Halfcheetah; Right: Hopper
Similar to (Mendez al., 2020), evaluated on the HalfCheetah, Hopper, and Walker-2D environ- ments. For the gravity domain, we select a random gravity value between 0.5g and 1.5g for each task. For the body-parts domain, thigh, and leg) to a random value between 0.5x and 1.5x its nominal value. As shown in Appendix of (Mendez al. (2020), these changes lead to highly diverse tasks for lifelong RL. Further, as required by model based deep RL methods, we change the environment settings for Hopper and Walker following (Wang al. 2019). When implementing VBLRL, we found that in the first few
the size and mass of each of the four parts of the body (head, torso,
Name of Environment
Reward function
HalfCheetah
à: 0.1 ||.]
Hopper Walker2D
0.1 |a 3.0 x (z) 1.3)²
de 0.1 aa:12 3.0 x (ze 1.3)2
Table 4: Reward functions for OpenAl Gym domains
episodes of each new tasks, the agent hasn't 't collected enough samples of the new task, which results in overfitting problems when training the task-specific. Thus, we use the world-model posterior instead to do the first few rounds of predictions and let the task-specific model begin training after collecting enough samples. The world-model has lower possibility overfitting as its training data comes from all the previous tasks and has much larger quantity. We list the other implementation details below. The planning horizons are selected from values suggested by previous model- RL papers (Chua al. 2018 Wang al. 2019).
For LPG-FTW and EWC, we use the original source code with parameters and model architectures suggested in the original paper. Specifically, we select step size from (0.005, 0.05, 0.5). For LPG- FTW, e use A = le 5,44 le 5 and select k from 3,5.10. For EWC, we select À from {le 6, le le 4). For HiP-MDP baseline, we modify the original algorithm for a fair comparison. We replace the DDQN algorithm used in Killian et al.'s paper with the exact same CEM planning method we used in VBLRI as well as the same parameters. And we use the same model architecture of Bayesian Neural network by modifying the baseline algorithm to also predict reward for each state-action pair (the original method only considers next-state prediction).
16

---

Under review as a conference paper at ICLR 2022
hyyper-parmeeten
CG
CB
HG
НB
WG
WB
# iterations
100
100
100
100
100
100
Steps (each iteration)
100
100
200
200
200
200
Ieamning rate (world model)
0.001
0.001
0.0005
0.0005
0.0005
0.0005
learning rate (task-specifer model)
0.0005
0.0005
0.0002
0.0002
0.0002
0.0002
planning horizon
20
20
30
30
kl-divergence weight
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
particles (CEM) batch size (world-model) batch size (task- cask-sspeiffic)
50
50
50
x 64
8 x 64
8 x 64
8 x 64
8 x 64
8 x 64
256
256
256
256
256
256
# tasks search population size clites (CEM)
40
40
30
30
40
40
500
500
500
500
500
50
50
50
50
50
Table 5: Room type probability distribution
For BOSS and BLRL, we set the number of sampled models K 5, and 7 0.95, value iteration.
A 0.01 for
A.5 DETAILED PROOF FOR LEMMA 1
Following the same settings introduced in section 5 of (Zhang 2006), we define the resolvability of standard Bayesian posterior
1 n
(g) inf [E (0) ( + =
(12)
In E D (xll(((((
We refer the readers to Zhang's paper for further explanation of the denotations. Intuitively, the Bayesian resolvability controls the complexity of the density estimation process. Based on this definition and our previous definitions of dz,, we can derive a simple and intuitive estimate of the standard Bayesian resolvability.
Lemma 3. The resolvability of standard Bayesian posterior defined in 12 can be bounded as
n
(n(g)= S
dx
proof. For all d > 0, we have
1 Tn(g)
In E DDDx < < Infe nd ndd((p= ElpeD l' DKL(gllp) d)]
n1 n dx
= d xI<hTTEE : s d)) <
This bound links the Bayesian resolvability the number of samples n and prior-mass radius d, which is fixed property of the density given a specific prior and the true underlying density. Intuitively, the Bayesian posterior is better behaved when the Bayesian prior is closer to the true distribution (d, is smaller) and more samples are used (n is larger).
Now prove the main theorem of Lemma 1. Let p }. 6, 25,, + (4n-2)s define P1 {p E r : D < a and l2 {p E: DRe (gllp) > ee). We let e-nt and define T') C when 0 E I and x z'(0): when 0 E l2, where the normalization constant C (a=(Ti) ((2))) [1, Firstly,
8/4
1 Exr((Tz|X)e & E,E. (0)x) ,11x all < Ex (xX))
17

---

Under review as a conference paper at ICLR 2022
according to the Markov inequality (with probability at least 8) and Pinsker's inequality. Then according to Theorem 5.2 and Proposition 5.2 in Zhang's paper,
17 in E (Dxc(ullp(()))
Ex E. x x' (8IX) _
p - 1)m
+ 7 p inf In Σ ( (1000-111000000 Txbb(1))))
p(1 - p)n
{
11 - (n//n) In E D
7- p (1 - 1)t
11 7-1
+ Eupper,n
p(1 p)
p( p) -
1) p
(2n 1)t
(n//n) In (alllllllllll + (17 uppupeer, - 11 p))
p(1 p)
p(1 Pl-p)
Then, using the definitions of d,, we further obtain
Exz (T2|X)et
(2n - 1)t +
17 iinfaoold - 1 II : (xl()) & d]}] + (n - p)eupper. (1)-))/nnnn/(((((((( - p))
p(1 p)
p(1 - p)
(2n- 1)t
11(1 + H)da + (11 p)eupper,n((1) 11
+
p(1 p)
p(1 - p)
(2n- 1)t En
p(1-p)
We use instead of ry which is used in the original paper to avoid confusion with the discount factor. Then we further divide both sides by 6 and obtain x X s 0.5. Then by definition,
π (T2/X) = an
-1 (1 (T)))
a
a
I+ end
Thus, we get the desired bound.
A.6 DETAILED PROOF FOR THEOREM
Theorem 2. (Full version of the bound Theorem 1) For each new task, set the sample size
K = O A 8 In $A) and the parameters = e(1 -- 60
5 FΛR then, with
ST.
82 A +In
probability at least 46, (**(s)) 2 (s:) - 4e all but O steps.
In 1 In
dc 10
5 (1->))
First, we would like to introduce two lemma from BOSS Asmuth et al.
2009 :
Lemma 4. The sample size K O A In SA) suffices to guarantee V Vi( 2 V*(s) for all during the entire learning process with probability at least 8. 8.
Lemma 5. If the knownness parameter = mor (s, a, 6, SA g= then the transition function of all the sampled models are e-close (in the l sense) the transition function for all the known state-action pairs during the entire learning process with probability at least -
proof. Given discrete state and action spaces, the proof that BLRL on each new task is PAC-MDP depends three main assumptions, following general PAC-MDP theorem from Strehl al. 2006):
1. Learning complexity condition. In our settings, state-action pair is claimed to be known after being visited for B times. In the meantime, there are S.A unknown state-action pairs total at the beginning of each task, the bounded discoveries condition is guaranteed.
18

---

Under review as a conference paper at ICLR 2022
2. Optimism. This is guaranteed by Lemma 3. event occurs. The values for each unknown state in every hyper-mon model are optimistic.
We construct new hyper-model each time discovery
3. Accuracy. For each new task, since the prior has bounded sample complexity of B for du and co. known state-action pair will be locally eg-accurate with probability at least ög. Given the definition of Bayesian concentration sample complexity and Lemma co c(1 y)² translates into an c error bound the value function (Asmuth et al. 2009).
de da 1 c000-a de
Finally, given Lemma 2, we know [(8 a, c0; 0o: po) O
let co <(1 - -
SA(d, +In =²(0K) (1-(1-
SA Po S we can get [(s, a, en, do, po) -
We can get the final form
of the bound by replacing B with these quantities.
A.7 ADDITIONAL EXPLANATION OF THE ALGORITHM
Here we first provide an example to help the readers better understand our plate notation. In our Gridworld Item Searching case, V/ represents the parameters of PR. which is the room-type and object distribution. For each task, the environment samples a hidden parameter w, which is the actual room and object layout of this house, from this distribution Pn- The sampled w then will result in an MDP m and let the agent interact with it.
A.7.1 PLANNING ALGORITIM
With the transition dynamics and reward functions, a planning algorithm like CEM is not the only way to solve the MDP to get an optimal policy. Another option would be using other Deep RL algorithms like Soft actor-critic (Haarnoja et al., 2018) with data generated from the model. However, in this case, incorporating a deep RL algorithm means that we need to introduce additional neural networks (that is, policy/value networks) for each task. The update signal from the RL loss is usually stochastic and weak, which is even worse in this case when our model is still far from accurate. So, here we assume applying planning algorithm is a better way to get the policy.
A.8 COIN EXAMPLE
Consider coin- coin-flipping environment. We want to find the sample complexity of the unbiased coin (i.e. How many times we need to flip this coin such that our posterior samples are accurate.). Consider a Dirichlet prior, 0g- = (M1, n2)) and fg (2. () We want to find sample complexity such that the posterior likelihood for coin with heads likelihood in [0.5 c, 0.5 + c] is at least - 5.
Note that the Dirichlet distribution on the -roddinnnsional simplex is the Beta distribution. The Multinomial distribution with two outcomes is the Binomial distribution. That is, given the process
H ~ Hin(H)p 0.5, B), p~ Beta(pla = H + nI, B n2),
(13)
(14)
choose a value B such that
2
(15)
B Σ
0.5+c
B),
Beta = H + nı, B = B - + 2 1 8.
(16)
f-0.5-
#=D
Here, ni and n2 capture the prior. The smallest B that satisfies Equation T6 can be found numerically.
We set == 0.1 and 8 = 0.3. Here are the results of sample complexity given different values of
71, ng:
We fix the sum of (n, R2) as 10. As shown in the results, the value of sample complexity B becomes lower we more accurate prior (from (10, 0) to 5)).
19

---

Under review as a conference paper at ICLR 2022
(71 12
lowest B
(0.10)
78
(1.9)
68
(2,8)
58
(3,7)
49
(4,6)
42
(5,5)
40
(6,4)
42
(7.3)
48
(8,2)
58
(9,1)
68
(10,0)
78
In general, for the task-specific posterior, we can relate B, and ō with the following equation:
[ Dir(Po PT
Σ
Mult(N]Po. B) [
- D
NjP dPo 2 1-5
Po
N
(17)
For the world model posterior:
Σ
Muli(NN|Pmo Bw)
UPs Dir(PanlPPaa + N)dP 2 21-8 (18)
N: B
For each task, first pick true model Po according to the true distribution and initialize the spec-fpe prior Pold = p.. Then, we make some observations from the world. Once we have the true model and the observations, weighted according to their posterior likelihood.
calculate how many models are e-close to the model,
20


Reviews:
[{"review_id": "Y-1lqEi3QHb", "reviewer": "Reviewer_7Ubo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main theoretical contribution suggested in the paper is a PAC-MDP of Theorem 1 for a single task. This theorem is based on Lemma 2, for which a full proof is not provided in the main text or appendix, so its veracity cannot be verified. Moreover, it is based on the assumption that the posterior is consistent, which I believe is what needs to be shown in a meta-learning setup, and cannot be assumed. The form of the bound is also strange as it depends on \\delta rather than on \\ln(1/\\delta) as in previous bounds (e.g., Strehl 2006 and Asmuth 2009), and its dependence on \\gamma and \\epsilon is also worse than previous bounds.  As far as I understand this is a bound for the single instance setting rather than for meta-learning. \n\nFollowing the theoretical part the authors develop a variational approach based on probabilistic networks that model the posterior distribution. As far as I am aware their variational approach is rather standard, although the authors do not refer to previous work. Also, their use of model-predictive control is common in current ML applications, but, again, this is not mentioned or discussed (e.g., the length of the future horizon and how is it selected). \n\nThe authors conclude by presenting numerical simulations for a grid-world and some MuJoco problems for continuous control. While the method compares well to simple baselines it is hard to assess performance relative to more recent work such as Liu et al., \"Taming MAML: Efficient Unbiased Meta-Reinforcement Learning\" and the other baselines measured in it. \n", "summary_of_the_paper": "The paper deals with the problem of lifelong RL, also referred to as meat-RL, where an agent attempts to solve a sequence of tasks in order to facilitate the solution of a novel task. The framework follows that of Baxter 2000 (albeit that paper deals with supervised learning), and has been widely studied in recent years.  The basic assumption is that the tasks are drawn from an underlying task-distribution, and each task (an MDP) is stochastically selected from  a task-specific distribution. The authors work with a Bayesian framework, assuming a hierarchical distribution of the two levels, and learn the two levels separately. This framework has the advantage of providing both estimates and uncertainty estimates. For the discrete case they present a sample complexity analysis, and suggest a variational approach to practical learning. Finally, experiments are provided supporting the utility of the approach. \n\nThe formal framework is that of hidden-parameter MDPs (HiP-MDPs) from Doshi-Velez 2016, and each MDP is modeled based on a transition and a reward model based on a hidden parameter. As more tasks are encountered the posterior over world models sharpens, and, being used to learn new tasks, is expected to facilitate learning. The learning of each new task is as in BOSS, and takes place by sampling from the learned MDP distribution, creating a mixed MDP, and using standard model-based approaches to solve these. ", "summary_of_the_review": "The paper is phrased within a sequential approach to meta-learning that has been widely studied within the supervised learning community (e.g., Baxter 2000, Pentina and Lambert, \"A PAC-Bayesian Bound for Lifelong Learning\" 2014 and much later work), with explicit performance bounds. It would be nice to acknowledge these roots. The present approach is plausible, and combines previous work, such as HiP-MDPs, BOSS, variational Bayes, in a sensible manner. However, I do not find that the level on innovation in this combination of approaches suffices for publication at ICLR, nor did I find the theoretical or experimental results of sufficient  interest (see comments above). \n\nFollowing rebuttal: Following the authors' response and my response to their rebuttal I have lowered my assigned grade due to my dissatisfaction with their replies, which served to enhance my existing concerns about the paper.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8ucuDKS4CKH", "reviewer": "Reviewer_FNYu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "############## Strengths ##############\n\n1. The overall idea of model-based lifelong RL is very relevant, particularly since lifelong RL precisely seeks to reduce sample complexity\n2. The high-level idea of replacing the task model with the world model whenever the task model is uncertain is intuitively appealing\n3. The use of the task model permits deriving reduced sample complexity bounds thanks to the Bayesian formulation\n\n############## Weaknesses ##############\n\n1. The low performance of the agent in the more complex MuJoCo evaluations makes the results unconvincing\n2. The technical approach is very closely tied to existing works, in particular HiP-MDPs and variants and BOSS\n3. There are no comparisons to existing model-based lifelong RL methods\n\n############## Arguments ##############\n\nThe primary contribution of this work is the introduction of one of the first approaches for model-based lifelong RL. Since one of the key desiderata of lifelong RL is to decrease the amount of experience needed to learn new tasks (forward transfer), using model-based techniques, which are inherently more data efficient than model-free techniques, seems to be a promising direction. \n\nHowever, the proposed method itself is fairly incremental. In particular, it heavily hinges on BOSS (as an exploration technique) and HiP-MDPs (as a multi-task model-based model). While the use of these models permits relatively simple adaptations of the proofs of BOSS to demonstrate decreased sample complexity (which is a nice plus), it does make the work contain very little technical insights. In particular, when adapting a multi-task method (HiP-MDPs) into a lifelong method, I would have expected to see considerable effort in designing a technique for updating the shared model with knowledge from new tasks. However, the authors simply train this shared model in a multi-task fashion with data from all tasks. This is not only costly in terms of memory footprint (requiring to store vast amounts of data for each task), but is also computationally inefficient, since the multi-task training step is executed iteratively _for each task_ (O(n^2) cost). The one piece of technical insight offered in the submission that I found interesting was the idea of replacing the task model with the world model whenever the task model is not yet well trained. Unfortunately, the approach itself is very heuristic (relying on thresholds on the uncertainty across sampled models) and the heuristic choices are not discussed in detail or validated empirically (e.g., via ablative tests or sensitivity analysis on the thresholds). I intuit that the method is quite sensitive to these thresholds: if they are too high, the task model will almost always be used, even if it's uncertain; if they are too low, the world model will be used too frequently and task performance will likely degrade by reverting to the \"average\" world model. Would it not be possible to do some sort of soft combination (of the task and world model) weighted by the uncertainty, instead of a hard selection of one or the other? \n\nHowever, my main concerns with the submission lie on the empirical evaluation. The first major concern is that all the agents evaluated on MuJoCo domains achieve very low performance, as compared to the results in the papers cited in the submission for the experimental design of these tasks (Mendez et al. and Wang et al.). While it is clear that the reduced performance stems from the choice of using fewer environment interactions (which itself is a nice choice, given the use of model-based techniques), this does raise questions about the conclusions drawn from these results. In particular, how useful or informative are forward and backward transfer if the agent hasn't really learned any meaningful behaviors? It would be useful to include videos of the learned behaviors to assess whether the transfer results are in any way significant from a behavioral perspective, or if they're simply minor reward increases across poor behaviors. \n\nMy second major concern is the choice of baselines for these evaluations. On one hand, the authors chose to only compare against model-free lifelong RL techniques. In this setting, it should certainly be expected that the model-based approach outperforms those baselines simply by the nature of the underlying techniques. This is not novel insight. Yet the authors claim these as general improvements over existing lifelong RL methods, which seems like a stretch. Instead, the authors should have considered existing model-based lifelong RL methods (e.g., [1]) for a more apples-to-apples comparison. Note that [1] is a task-agnostic method, so this would require some adaptation to handle the setting where the agent is given access to task indicators. On the other hand, the authors make claims about forward transfer, but in the MuJoCo domains there is no comparison to a single-task or no-transfer baseline, like used in the box-jumping task. Such a baseline is critical for assessing whether the approach is actually achieving transfer across tasks, since improvements w.r.t. single-task training are precisely what demonstrate transfer. As one additional comment regarding the evaluations, there is no information about the implementation details of any of the baselines, including their model architectures and hyper-parameters. How were these chosen to guarantee a fair comparison?\n\n\n############## Additional feedback ##############\n\nThe following points are provided as feedback to hopefully help better shape the submitted manuscript, but did not impact my recommendation in a major way.\n\nIntro\n- I wonder if the example of different houses and toothbrushes matches the HiP-MDP formulation introduced immediately after\n- The intro is fairly clear and describes the solution approach well. \n- I'd suggest including an example that more closely matches the HiP-MDP formulation, since this is the formulation adopted throughout the paper.\n\nSec 4\n- The ideas seem to be very closely related to the original HiP-MDP papers, especially the BNN extension of Killian et al. (2017)\n- The notation for the BNN needs a fair bit of work. The authors never explain what the \"particles\" are. Are these the (s,a,r,s') tuples sampled from the sequences given by the combination of CEM and the BNN? This becomes increasingly relevant in 4.1 where the authors define their approach to backward transfer. My understanding is that \"aleatory variability\" is modeled as the BNN's internal variance, whereas the epistemic uncertainty is measured as the variance in the (mu, sigma) output by the BNN across sampled particles. Is this understanding correct?\n\nSec 5\n- The gird-world evaluation shows nothing about forgetting/backward transfer. \n- Box-jumping: Why no comparison to a single-task variant of the solution? This is required to assess forward transfer. Also no backward transfer measure.\n- MuJoCo: again, no single-task learner, so it's unclear if there's forward transfer. Plus, the rewards are very low, so it seems that even VBLRL is not solving the tasks. How useful are these results then? Even though VBLRL is the best, it's not really achieving meaningful behaviors.\n- I disagree with the claim that the model \"cannot\" suffer from forgetting, since certainly the wrong choice of threshold for backward transfer could lead to forgetting.\n- How was this hyper-parameter chosen?\n\nTypos\n- Sec 2, second paragraph: the task facing a single agent -> the agent facing a single task?\n\n[1] Nagabandi et al. Deep online learning via meta-learning: Continual adaptation for model-based RL. ICLR 2019.\n", "summary_of_the_paper": "This submission presents an approach for Bayesian model-based exploration in a lifelong RL setting, building upon existing approaches for Bayesian exploration (BOSS) and Bayesian multi-task modeling (HiP-MDPs). The approach keeps separate models for sampling transitions and rewards for each task, and each task model is drawn from a shared prior that models the distribution over tasks. The method continually updates the shared model with data from all observed tasks and the task-specific model with data from the current task. To achieve backward transfer, the approach replaces the task model with the shared model whenever the task model has not been sufficiently trained on a particular state-action pair. \n", "summary_of_the_review": "\nUnfortunately, I recommend the rejection of this work. While I agree with the premise of the submission that model-based lifelong RL is a relevant area of research, with potential implications on real-world applications of lifelong RL, the submission as it stands appears to not be ready for publication. On the technical side, the approach seems to add just a few incremental changes to multi-task HiP-MDPs to adapt them to the lifelong setting. This on its own is perhaps relatively minor, since the novelty comes from adapting it to a new problem setting. However, such technically incremental contributions should generally be accompanied by strong empirical evaluations, which is not the case in this work. In particular, the low overall performance of all agents on MuJoCo domains suggests that none of the agents are learning to achieve meaningful behaviors, which raises questions about the conclusions reached by the authors. Moreover, the authors should have compared (at least qualitatively, but ideally also empirically) to existing work in lifelong model-based RL. On the flip side, the submission does include an interesting insight of replacing the task-specific model with the shared model whenever the task model is uncertain.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7RZI2iv7_8x", "reviewer": "Reviewer_eZuT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n- Clarity of the paper should be improved significantly. The current version makes understanding hard. \n     - The paper does not describe the notion of many notations (e.g., the indices j and k in Figure 1.) and denotes some distributions in its short-form, e.g., q_e or q_\\theta^{m_i}, without introducing its full form. These are not just a few, but observed broadly across the entire paper, perhaps Section 3 requires the most significant improvement. \n     - The paper also explains the proposed method by directly explaining lines of the pseudo-code without discussing the big picture or the rationale behind the design.\n     - It also skipped describing the full joint distribution of the model which is essential in describing such probabilistic models.\n- The key idea of the proposed method is quite simple. It's like applying Variational Continual Learning to the world model learning in the RL setting. It can also be understood as Bayesian meta-learning but with sequential task exposure. \n- The experiment is also quite simple. The first two toy tasks are quite too toyish, very low-dimension and tow task complexity. The last MuJoCo experiments show the superiority of the proposed model. \n- The algorithms (e.g., variational continual learning, Bayesian meta learning, and mean-field variational inference) on which the proposed method is based are known to be not scalable in more complex settings like high-dimensional data (like image), many sequential tasks (e.g., 1000 or more tasks), and the mean-field approximation has significant limitation in its expressiveness. So, I'm somewhat doubtful if this method can be an important milestone toward more realistic and complex settings. ", "summary_of_the_paper": "The authors proposes a Hierarchical Bayesian approach for lifelong RL. The global world-model posterior models the world model shared across tasks and the task-specific model learns the dynamics within a specific task. The task-specific model achieves forward transfer by initializing from the global world model. The authors use mean-field variational approximation to scale the proposed model. Also, the authors introduce sample complexity analysis. The method is evaluated on two toy tasks (grid-world and box jumping) and one on MuJoCo simulator and showed superior performance to the previous works. ", "summary_of_the_review": "The writing clarity requires a significant improvement. It's currently a major drawback hindering the understanding of the proposed model (I understood at the later part of the paper but it was hard until reaching there). The evaluation is quite simple and uses toy tasks. I'm doubtful about the potential of the proposed model to extend to more realistic and complex settings such as image inputs. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The topic of this paper is timely and important.  However, ultimately the reviewers remained unconvinced that this paper provides a sufficiently clear and sufficiently significant advance to lifelong RL.

As an additional note, the setting under investigation here is not the full lifelong learning setting.  E.g., several of the challenges outlined by Schaul et al. [1] are not treated, and this work is, instead, situated in a somewhat typical multi-task setting with substantlal structure.  That is not bad, but it would be good if this is reflected clearly in all the statements, and, e.g., in the title of the work.

The authors are encouraged to carefully take the provided feedback and see how they can use it to improve their work.  This is an important research direction.  It was just felt the current submission was not quite ready for publication yet.

[1] https://arxiv.org/abs/1811.07004
Decision: Reject