Offline Meta Reinforcement Learning -- Identifiability Challenges and Effective Data Collection Strategies
Authors: Ron Dorfman
Idan Shenfeld
Aviv Tamar
Venue: NeurIPS 2021 Poster (2021)
Keywords: Meta-RL
Offline RL
Bayesian RL

Abstract: Consider the following instance of the Offline Meta Reinforcement Learning (OMRL) problem: given the complete training logs of $N$ conventional RL agents, trained on $N$ different tasks, design a meta-agent that can quickly maximize reward in a new, unseen task from the same task distribution. In particular, while each conventional RL agent explored and exploited its own different task, the meta-agent must identify regularities in the data that lead to effective exploration/exploitation in the unseen task. Here, we take a Bayesian RL (BRL) view, and seek to learn a Bayes-optimal policy from the offline data. Building on the recent VariBAD BRL approach, we develop an off-policy BRL method that learns to plan an  exploration strategy based on an adaptive neural belief estimate. However, learning to infer such a belief from offline data brings a new identifiability issue we term MDP ambiguity. We characterize the problem, and suggest resolutions via data collection and modification procedures.
Finally, we evaluate our framework on a diverse set of domains, including difficult sparse reward tasks, and demonstrate learning of effective exploration behavior that is qualitatively different from the exploration used by any RL agent in the data. Our code is available online at \url{https://github.com/Rondorf/BOReL}.

Offline Meta Reinforcement Learning - Identifiability Challenges and Effective Data Collection Strategies
Ron Dorfman
Idan Shenfeld
Technion
Technion
rdorfman@campus. wwchnion.ac.......
idanshen@campus. technion.
Aviv Tamar
Technion
avivt@techmion.ac.c
Abstract
Consider the following instance of the Offline Meta Reinforcement Learning (OMRL) problem: given the complete training logs of N conventional RL agents, trained on N different tasks, design a meta-agent that can quickly maximize reward in new, unseen task from the same task distribution. In particular, while each conventional RI. agent explored and exploited its own different task, the meta-agent must identify regularities in the data that lead to effective exploration/exploitatio in the unseen task. Here, we take Bayesian RL (BRL) view, and seek to learn a Bayes-optimal policy from the offline data. Building on the recent VariBAD BRL approach, we develop an off-policy BRL method that learns to plan an exploration strategy based an adaptive neural belief estimate. However, learning to infer such a belief from offline data brings new identifiability issue we term MDP ambiguity. We characterize the problem, and suggest resolutions via data collection and modification procedures. Finally, we evaluate our framework on a diverse set of domains, including difficult sparse reward tasks, and demonstrate learning of effective exploration behavior that is qualitatively different from the exploration used by any RL agent in the data. Our code is available online https://github www.c/o//oooaa Betom/Rondorf/BBaa.
Introduction
central question reinforcement learning (RL) is how to learn quickly (i.c., with few samples) in environment, Meta-RL addresses this issue by training agent large set of training environments (5. 9]. Intuitively, the meta-RL agent can learn regularities the environments,
which allow quick learning in any environment that shares similar structure. Indeed, recent work
demonstrated this by training memory-based controllers that 'identify' the domain [5] 28
[8 or by
leamning parameter initialization that leads to good performance within few gradient steps 9
Another formulation quick RL is Bayesian RL [BRL., 11] In BRL, the environment parameters are treated as unobserved variables, with known prior distribution. Consequentially, the standard problem of maximizing expected returns (taken with respect to the posterior distribution) explicitly accounts for the environment uncertainty, and its solution is Bayes-optimal policy, wherein actions optimally balance exploration and exploitation. Recently, Zintgraf al. [36] showed that meta-RL RL in fact an instance of BRL, where the meta-RL environment distribution is simply the BRL prior. Furthermore, a Bayes-optimal policy can be trained using standard RL methods, by adding to the state the belief over the environment parameters. The VariBAD algorithm (36) implements this idea using a variational autoencoder (VAE) for belief estimation and deep neural network policies.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Offline Data
Meta RL Agent
Agent
Ippeade Epesode 2
Agent 2
Agent
Agent
Figure 1: Offline meta-RI. on the Semi-Circle domain: the task is to navigate to goal position that can be anywhere on the semi-circle. The reward is sparse (light-blue), and the offline data (left) contains training logs of conventional RL agents trained to find individual goals. The meta-RL agent (right) needs to find a policy that quickly finds the unknown goal, here, by searching across the semi-circle in the first episode, and directly reaching it the second completely different strategy from the dominant behaviors in the data.
Most meta-RL studies, including VariBAD, have focused on the online setting, where, during training. the meta-RL policy is continually updated using data collected from running it in the training environments. domains where data collection is expensive, such as robotics and healthcare to name few, online training is a limiting factor. For standard RL, offline (a.k.a. batch) RL mitigates this problem by learning from data collected beforehand by an arbitrary policy 21. this work we investigate the offline approach to meta-RL. (OMRL).
Any offline RL approach is heavily influenced by the data collection policy. To ground our inves- tigation, we focus on the following practical setting[ []] we assume that data is collected by running standard RL agents on a set of environments from the environment distribution. While the data was not specifically collected for the meta-RL RL task, we hypothesize that regularities between the training domains can still be learned, to provide faster learning in new environments. Figure [ iillustaates our problem: in this navigation task, each RL agent in the data learned to find its own goal, and converged to behavior that quickly navigates toward it. The meta-RL. agent, on the other hand, needs to learn a completely different behavior that effectively searches for the unknown goal position.
Our method for solving OMRL is an off-policy variant of the VariBAD algorithm, based on replacing the on-policy policy gradient optimization in VariBAD with an off-policy Q-learning based approach. This, however, requires some care, as Q-learning applies states of fully observed systems. We show that the VariBAD approach of augmenting states with the belief in the data applies to the off-policy setting as well, leading an effective and practical algorithm. The offline setting, however. brings about another challenge- when the agent visits different parts of the state space in different environments, learning to identify the correct environment and obtain an accurate belief estimate becomes challenging, a problem we term MDP ambiguity. We formalize this problem, and discuss how it manifests common scenarios such as sparse rewards or sparse differences in transitions. Based on our formalization, we propose a general data collection strategy that can mitigate the problem. Furthermore, when ambiguity is only due to reward differences, we show that a simple reward relabelling trick suffices, without changing data collection. We collectively term our data collectiin/reeabelling and off-policy algorithm as Bayesian Offline Reinforcement Learning (BOReL).
In our experiments, we show that BOReL learns effective exploration policies from offline data on both discrete and continuous control problems. We demonstrate significantly better exploration than meta-RL methods based on Thompson sampling such as PEARL (28). even when these methods are allowed to train online. Furthermore, we explore the issue of MDP ambiguity practice, and demonstrate that, when applicable, our proposed solutions successfully mitigate it.
An important implication of our study is that without suitably collected data, MDP ambiguity can make learning an effective offline BRL solution impossible. This stands in contrast to recent studies in offline (non-Bayesian) RL
30 T0 19 where the focus is on designing learning algorithms for arbitrarily collected data, typically by avoiding actions that were not explored enough. Outside offline RL, however, it is common develop data collection methods based on formal limitations of the problem. For example, the use of randomized controlled trials in medical treatments is well motivated by the theory of causal inference 227 The data collection methods we propose here are simple and practical to implement, and as we demonstrate, effectively handle MDP ambiguity.
'The theory and algorithms develop, however, not limited any particular data collection protocol.

---

Our main contributions are therefore as follows: to our knowledge, this is the first study of meta learning exploration in the offline setting: we provide the necessary theory to extend VariBAD to off-policy RI; we formulate MDP ambiguity, which characterizes which problems are solvable under the offline BRL setting, and based on this formulation we propose several principled data collection strategies; we show non-trivial empirical results that demonstrate significantly better exploration than meta-RL RL methods based on Thompson sampling; finally, and of independent interest, our off-policy algorithm significantly improves the sample efficiency of conventional VariBAD in the online setting.
2 Background
We recapitulate meta-RL, BRL and the VariBAD algorithm.
Meta-RL: In meta-RL, a distribution over tasks is assumed. A task T; is described by Markov Decision Process IMDP, 2 M, = (S, A, R, P.), where the state space S and the action space A are shared across tasks, and R2 and P. are task specific reward and transition functions. Thus, we write the task distribution as p(R, P). For simplicity, we assume throughout that the initial state distribution Ponét(5)) the same for all MDPs. The goal in meta-RL is to train an agent that can quickly maximize reward in new, unseen tasks, drawn from PR,,P)
Bayesian Reinforcement Learning: The goal in BRL is to find the optimal policy in MDP, when the transitions and rewards are not known in advance. Similar to meta-RL, we assume a prior over the MDP parameters p(R., P). and seek to maximize the expected discounted return.
E
Σ
En'r
( oz)
where the expectation is taken with respect to both the uncertainty in state-action transitions St+1 ~ PP(-ss: a.), a, ~ IT, and the uncertainty in the MDP parameters RR ~ p(R, P) Key here is that this formulation naturally accounts for the exploration/exploitation tradeoff an optimal agent must plan its actions to reduce uncertainty in the MDP parameters, if such leads to higher rewards.
One way to approach the BRL problem is to model R, P as unobserved state variables a partially observed MDP [POMDP, B1. reducing the problem to solving a particular POMDP instance where the unobserved variables do not change time. The belief at time bt, denotes the posterior probability over R, P given the history of state transitions and rewards observed until this time 6, P(R, Plh.)). where h. {su, ag. 71, 81 ..... Tt, } (note that we denote the reward after observing the state and ac- tion at time f as r = r(s,, at).. The belief can be updated iteratively according to Bayes rule, where (R, P) = pRR P). and: b (R, P) = PRR, P h2t-1) x (( Fe.....e+hhh.. R, P)b P).
Similar to the idea of solving POMDP by representing it as an MDP over belief states, the state in BRL can be augmented with the belief to result in the Bayes-Adaptive MDP model [BAMDP, 6.. Denote the augmented state 5² = (81, by) and the augmented state space S+ = S x B, where
B denotes the belief space. The transitions in the BAMDP are given by: p* 1sT 04) = Es, [P(* [s,, at)] 5 (b++1 = P(R, P 1lh )), and the reward in the BAMDP is the expected re- ward with respect to the belief: (+ ae) = Es, [R(8t; 0t)] The Bayes-optimal agent seeks to maximize the expected discounted return in the BAMDP, and the optimal solution of the BAMDP
gives the optimal BRL policy. As in standard MDPs, the optimal action-value function in the BAMDP
satisfies the Bellman equation: Vs+ E St E A we have that
Q(s = R (st ,a) +7B max Q(st' .a)]-
(2)
a'
Computing a Bayes-optimal agent amounts to solving the BAMDP, where the optimal policy is function of the augmented state. For most problems this intractable, as the augmented state space is continuous and high-dimensional, and the posterior update is also intractable in general.
The VariBAD Algorithm: VariBAD (6) approximates the Bayes-optimal solution by combining a model for the MDP parameter uncertainty, and an optimization method for the corresponding BAMDP. The MDP parameters are represented by a vector m E d corresponding to the latent variables in a
For ease of presentation, we consider the infinite horizon discounted return. Our formulation easily extends to the episodic and finite horizon settings, considered in our experiments.
3

---

parametric generative model for the state-reward trajectory distribution conditioned on the actions P1. 81 .....
.....A TH Str (a....., an- = Po (m))po(s)) (sp. 1,, $1 ..... :T,, m, Gp. ............. an1-1111111110001 a dmm The model parameters 0 are learned by a variational approximation to the maximum likelihood objec- tive, where the variational approximation to the posterior P(m] so. 71, 81 ..., 811 $1,00,-.. .0100).....0.111)) is chosen to have the structure go go(m) So: 00."", 81 ..........0..................... "",,, 8t) = (m(m)h.)) That is, the ap- proximate posterior is conditioned on the history up to time t. The evidence lower bound (ELBO) in this case is ELBO,, = (-n-gg(-1h..)) [log Pa (So, Ti, 81 "p.....77 "H, SH (m, ................................ ......
D (g (malaa))llee (pw)))))))))) The main claim of Zintgraf et al. [36] is that qo (.((ml..))) can be taken as an approximation of the belief by In practice, 9o (m/())
represented as a Gaussian distribution h,,) = (h)h))), E(ha)), where and > are learned recurrent neural networks. While other neural belief representations could be used [15], we chose to focus on VariBAD for concreteness.
To approximately solve the BAMDP, 36 exploit the fact that an optimal BAMDP policy is function of the state and belief, and therefore consider neural network policies that take the augmented BAMDP state as input T St. go (mm()a))) where the posterior is practically represented by the distribution parameters (h,,)), Σ (h.,)) The policies are trained using policy gradients, optimizing
H
(T) = ER, Ex
7'r(st, at)
(3)
The expectation over MDP parameters in (3) is approximated by averaging over training environments, and the RL agent is trained online, alongside the VAE.
3 OMRL and Off-Policy VariBAD
In this section, we derive an off-policy variant of the VariBAD algorithm, and apply it to the OMRL problem. We first describe OMRL, and then present our algorithm.
3.1 OMRL Problem Definition
We follow the Meta-RL and BRL formulation described above, with a prior distribution over MDP parameters p(R, P). We are provided training data of agent interacting with N different MDPs, N (R, p:) sampled from the prior. Each interaction is organized as M trajectories of length 1.j 3. 34 3.
Lg i.) i E N.j E 1,..., M, where the rewards satisfy std r++1 (s) at the transitions satisfy 8,41 P.(.)s," af")), and the actions are chosen from an arbitrary data collection policy. To ground our work a specific context, we sometimes assume that the trajectories obtained from running a conventional RL agent in each one of the MDPs (i.e., the complete RL training logs), which implicitly specifies the data collection policy. We will later investigate implications of this assumption, but emphasize that this is merely an illustration, and our approach does not place any such constrains - the trajectories can also be collected differently. Our goal is to use the data for learning Bayes-optimal policy, i.e., policy that maximizes Eq. (T)
H, Th = 80 90
,81 .e....,, HH
3.2 Off-Policy VariBAD
The on-policy VariBAD algorithm cannot be applied to our offline setting. Our first step is to modify VariBAD work off-policy. We start with observation about the use of the BAMDP formulation in VariBAD, which will motivate our subsequent development.
Does VariBAD really optimize the BAMDP? Recall that BAMDP is in fact reduction of POMDP to an MDP over augmented states s = (s. b), and with the rewards and transitions given by R+ and p+. Thus, optimal Markov policy for the BAMDP exists in the form of T(s+), ). The VariBAD policy, as described above, similarly takes as input the augmented state, and is thus capable of representing an optimal BAMIP policy. However, VariBAD's policy optimization in Eq. does not make use of the BAMDP parameters R+ and p+! While at first this may seem counterintuitive, Eq. (3) in fact a sound objective for the BAMDP, as we now show
Proposition I. Let So, ao r1, .1..., TH,8H 8u denote a random trajectory from fixed history dependent policy म, generated according to the following process. First, MDP parameters R, P
This result is closely related to the discussion in 14 26 here applied particular setting.

---

R1
R2
Figure 2: Reward ambiguity: from the two trajectories, it is impossible to know if there are two MDPs with different rewards (blue and yellow circles), or one MDP with rewards both locations.
are drawn from the prior P(R,P). Then, the state trajectory generated according to so Pintt ~ <-|ss,, ao, .i,...,.;) St), s1+1 ~ P(-|s;, az) and r++1 ~ R(s;,ar) at). Let by denote the posterior belief at time I, by = P(R, PI P| 0o, rl, r1,...,,81) 8t). Then, P(s (S0, an r1, .1..... Ft, St, a+) = ER.P. P(S (8844118,000 at), and P(r++1 (s, Co, ......,, St, 01) = ER,P.bb, R(r++1|st, at).
For on-policy VariBAD, Proposition [ shows that the rewards and transitions in each trajectory can be seen as sampled from a distribution that in expectation is equal to R+ and Pt, and therefore maximizing Eq.
3 is valid However, off-policy RL does not take as input trajectories, but tuples of the form (s, a, 7, = (state, action, reward, next state), where states and actions can be sampled from any distribution. For an arbitrary distribution of augmented states, we must replace the rewards and transitions in our data with R* and P+, which can be difficult to compute. Fortunately, Proposition I shows that when collecting data by sampling complete trajectories and corresponding beliefs, this is not necessary. as in expectation, the rewards and transitions are correctly sampled from the BAMDP. In the following, we therefore focus on settings where data can be collected that way, for example, by collecting logs of RI. agents trained on the different training tasks.
Based on Proposition 1 we can use state augmentation method similar to VariBAD, which refer to as state relabelling. Consider each trajectory in our data ا 5. 0o ;ri 8H as defined above, Recall that the VariBAD VAE encoder provides an estimate of the belief given the state history qm hha) (
(hh (bba)))) Thus, we can run the encoder on every partial t-length history Tit to obtain the belief at each time step. Following the BAMDP formulation, we define the augmented state s st.h = (sf",b("") b,.)), where 65.5 = p((T E(T)"). We next replace each state in our data s² with st effectively transforming the data to as coming from a BAMDP. After applying state relabelling, any off-policy RL algorithm can be applied the modified data, for learning a aayesalpttmmal policy. In our experiments we used deep Q-learning [DQN, 25 for discrete action domains, and soft actor critic [SAC. [17 for continuous control.
4 Identifiability Problems in OMRL
We take a closer look at the OMRL problem. While in principle, it is possible to simply run off- policy VariBAD on the offline data, we claim that in many problems this may not work well. The reason is that the VariBAD belief update should reason about the uncertainty in the MDP parameters, which requires effectively distinguish between the different possible MDPs. Training the VAE to distinguish between MDPs, however, depends on the offline data, and might not always be possible. This problem, which we term MDP ambiguity, is illustrated in Figure 2 consider two MDPs, one with rewards in the blue circle, and the other with rewards in the yellow circle. If the data contains trajectories similar to the ones the figure, it is impossible to distinguish between having two different MDPs with the indicated rewards, or single MDP with rewards at both the blue and yellow circles. Accordingly, we cannot expect to learn a meaningful belief update. In the following, we formalize MDP ambiguity, and how it can be avoided.
For an MDP defined by {R. P}. denote by Pr. (s, s') and (s, a). respectively, induced by policy π.
(s, a, s') and (p. o) the distribution over
Definition 1 (MDP Ambiguity). Consider data coming from set of different MDPs M (R, P.V.V C M, where M is an hypothesis set of possible MDPs, and corresponding data col- lection policies { N resulting in N different data distributions D = P... (s, a, s') N')
'To further clarify, if we could calculate Rt, replacing all rewards in the trajectories with R will result in a lower variance policy update, similar to expected SARSA (34.
5

---

We say that the data is ambiguous if there is an MDP {R, P) E MM and two policies gt and x' such that P PR.P (s, a, s') (R. o, T, s') and PR, Pz (s,..r,s') PR.P. (8, a, r, s'), for some: + j. Otherwise, the data is termed identifiable
The essence of identifiability, as expressed in Definition I is that there is no single MDP in the hypothesis set that can explain data from multiple MDPs in the data, as in this case it will be impossible to learn an inference model that accurately distinguishes between the different MDPs, even with infinite data 0 Let us now define the notion of identifying state-actions and overlapping state-actions, which will enable us to clearly state a simple sufficient condition for identifiability.
Definition 2 (Identifying Actionn, For a pair of MDPs i and j. we say that (s, a) is an identifying state-action pair (2,s,,) a) * Ry(s, a) and/or P.(s' (s, a) # PP)s'|s,a).
Definition 3 (Overlapping State-Action), Consider the setting in Definition 1. For a pair of MDPs i and j. we say that a state-action pair (8,a) a) overlaps it has positive probability under both i and j. Le., Print (8, Po Paing (8, a) > O.
Identifiability strongly depends on the hypothesis set M. However, for learning deep neural network inference models, we do not want to impose any structure on M. Thus, in the following we provide a sufficient identifiability condition that holds for any M.
Proposition 2. Consider the setting in Definition D If for every i + j there exists an identifying state-action pair that overlaps, then the data is identifiable.
Thus, if the agent has data on identifying state-actions obtained from different MDPs, it has the capability to identify which data samples belong to which MDP. We next categorize several common types of meta-RL problems according to identifiability, as per Proposition 2 we will later revisit this dichotomy in our experiments. For our illustration, we assume that in each training MDP, the data collecting policy is approximately optimal (this would be the case when training standard RL. agents on each MDP). Let us first consider problems that only differ in the reward. Here, when identifying state-actions (i.e.. state-actions with different rewards) in different MDPs do not overlap, we will have an identifiability problem. The sparse reward tasks in Figures D and a are examples of this case- each agent will visit only its own reward area, resulting in ambiguity. When the rewards are dense, however, it is much more likely that the data is identifiable; common tasks like Half-Cheetall Aeel (cf. For MDPs that differ in their transitions, similar argument can
Sec. 6 are examples of this setting | be made about whether the identifying state-actions overlap or not. Most studies on online and offline meta-RL to date considered problems with overlapping identifying state-actions, where ambiguity is not an issue. For example, in the Walker environment of Zintgraf et al. [36], the shape of the agent is varied, which manifests in almost every transition, and successful agent must walk forward, thus many overlapping state-actions are visited; the Wind domain (cf. Sec. 6 is another example. Examples of problems with non-overlapping identifying transitions are, for example, peg-in-hole insertion where the hole position varies between tasks, or the Escape-Room domain in Sec. 6 in such domains we expect ambiguity to be concern. One can of course imagine combinations and variations of the categories above our aim is not to be exhaustive, but to illustrate which OMRL problems are difficult due ambiguity, and which are not.
Note that MDP ambiguity is special to offline meta-RL; in online meta-RL, the agent may be driven by the online adapting policy (or guided explicitly) to explore states that reduce its ambiguity. We also emphasize that this problem is not encountered in standard (non- men-meea) offline RL, as the problem here concerns the identification of the MDP, which in standard RL is unique.
How can one collect data to mitigate MDP ambiguity? We present a simple, general modification to the data collection scheme we term policy replaying, which, under mild conditions on the original data collection policies, guarantees that the resulting data will be identifiable. We importantly note that changing the data collection method in-hindsight is not suitable for the offline setting. Therefore, the proposed scheme should be viewed as a guideline for effective OMRL data collection. For each
$ PP()) = P'()) means equality almost everywhere; P(.) # P'(-) means that equality almost everywhere does not hold.
'For simplicity, Definition I considers discrete set of MDPs, and infinite data. In our experiments, we validate that our insights also hold for finite data and continuous models.
'The ambiguity of sparse reward tasks is very different from the well- exploration difficulty in RL with sparse rewards (e.g.. (I]: ambiguity is not related to the RL algorithum but to the learned belief estimate.

---

MDP, we propose collecting data in the following manner: randomly draw a data collection policy from { f= collect a trajectory following that policy, and repeat. After this procedure, the new data distributions are all associated with the same data collection policy, which we denote Ty-
Proposition 3. For every + j. denote the set of identifying state-action pairs by Iz.-- for every i and every j exists (5t.5. ($t.), E Ii, such that Pp...t (Sig: Oi..) > 0, then replacing x with Ty for all i results in identifable data.
Note that the requirement on identifying states in Proposition is minimal - without it, the original data collecting policies r are useless, as they do not visit any identifying states (e.g.. consider the example in Figure 2 with policies that do not visit the reward at all).
When the tasks only differ in their reward function, and the reward functions for the training environments are known, policy replaying can be implemented in hindsight, without changing the data collection process. This technique, which we term Reward Relabelling (RR), is applicable under the offline setting, and described next. In RR, we replace the rewards in trajectory from some MDP i in the data with rewards from another randomly chosen MDP j # i. That is, for each i E
EE,, I..... N, we add to the data K trajectories pi.k, l E l..... K, where ft. = (s) d. şi.k $1 ...,, pi.k (.A)). 2i.k
where the relabelled rewards i satisfy 'tt1 Rs 1.k ar ". Thus, our relabelling effectively runs Tg MDP j. which is equivalent to performing policy replaying (in hindsight). We remark that the assumption on known reward (during training) is mild, as the reward is the practitioner's 's method of specifying the task goal, which typically known [3 32 31: this assumption is also satisfied in all meta-RL studies to date.
BOReL: we refer to the BOReL algorithm as the combination of the policy replaying/RR techniques and off-policy RL applied to state-relabelled trajectories. In Appendix [B] we provide pseudo-code, and detail how to apply the insights of Proposition 3 to practical episodic RL setting.
Related Work
We focus on meta-RL - quickly learning to solve RL problems. Gradient-based approaches to meta-RL seek policy parameters that can be updated to the current task with a few gradient steps [g] These are essentially online methods. and several studies investigated learning of structured exploration strategies in this setting [6 29 33 Memory-base based meta-RL, on the other hand, map the observed history in a task h. to an action 35 These methods effectively treat the problem as a POMDP, and learn memory based controller for it.
12 12
The connection between meta-learning and Bayesian methods, and between meta-RL. RL and Bayesian
RL in particular, has been investigated in a series of recent papers 20 18 26 36). and our work closely follows these ideas. In particular, these works elucidate the difference between Thompson- sampling based strategies, such as PEARL. (28), and Bayes-optimal policies, such as VariBAD, and suggest to estimate the BAMDP belief using the latent state of deep generative models. Our contribution is an extension of these ideas to the offline RL setting, which to the best of our knowledge novel. Technically, the VariBAD algorithm in (36) is limited to on-policy RL, and the off-policy method in [18] requires specific task descriptors during learning, while VariBAD, which our work based on, does not. One can also learn neural belief models using contrastive learning 15] our methods and identifiability discussion apply to this case as well.
Concurrently and independently with our work, Li (22] proposed MBML, an offline meta- RL algorithm that combines BCQ 10] with a task inference module. Interestingly, Li et al. [22] describe technique similar reward relabelling for discouraging task inference to ignore rewards. Here, we provide a formal and general characterization of identifiability problems in OMRL. Addi- tionally, MBML does not take into account task uncertainty, and cannot plan actions that actively explore to reduce this uncertainty - this is a form of Thompson sampling, where a task-conditional policy reacts the task inference (see Figure I in (22)). Our work is the first to tackle offline meta-learning of Bayes-optimal exploration. In addition, we demonstrate the first offline results on sparse reward tasks, which, compared to the dense reward tasks in [2], require a significantly more complicated solution than Thompson sampling (see experiments section). We achieve this by building on BRL theory, which both optimizes for Bayes-optimaliis and results in a much simpler algorithm. Recent work on meta Q-learning S also does not incorporate task uncertainty, and thus cannot Bayes-optimal. The very recent work of Mitchell et al. 24 considers a different offline meta-RL RL setting, where an offline dataset from the test environment is available.
7

---

Classical works on BRL are comprehensively surveyed by Ghavamzadeh et al. [. Our work, in comparison, allows training scalable deep BRLppllcics Finally, there is growing interest in offline deep RI. [30, 21]. In our experiments, a state-of-ther thtt--r method of this flavor (CQL., [19 led to minor improvements, though future offline RI. developments may possibly benefit OMRL too.
Experiments
In our experiments, aim demonstrate: (1) Learning approximately Bayes-optimal policies in the offline setting; and (2) The severity of MDP ambiguity, and the effectiveness of our proposed resolutions. In the supplementary material, we also report that our off-policy method improves meta-RL performance in the online setting,
Answering (I) is difficult because the Bayes-optimal policy is generally intractable, and because our results crucially depend on the available data. However, in deterministic domains with a single sparse reward, the optimal solution amounts to 'search all possible goal locations efficiently as possible, and stay at goal once found: in subsequent episodes, move directly to goal'. We therefore chose domains where this behavior can be identified qualitatively. Quantitatively, we compare BOReL with two Thompson sampling based methods: MBML. [22) using the same offline data, and PEARL. (28) using online data, and aim to show that the performance improvement due being approximately Bayes-optimal gives an advantage, even under the offline data restriction.
Domains and evaluation metric: we evaluate learning to explore efficiently in a diverse set of domains: (1) A discrete 5 5 Gridworld [36]: (2) Semi-circle a continuous point robot where a sparse reward is located somewhere on a minilcircle (see Figure D (3) Ant-Semi-cirele challenging modification semi-circh task above (see Figure 5): (4) Half-Cheetah- Vel [9] popular figh-dimensional control domain with dense rewards; (5) Reacher-Image 2-link robot reaching an unseen target located somewhere on a quarter circle, with image observations and dense rewards (see Appendix D); (6)
the popular Ant-Goal task X to a sparse reward setting similar to the
Wind a point robot navigating to a fixed goal the presence of varying wind; and (7) EEscapeRRoo point robot that needs to escape circular room where the only opening is somewhere on the semi-cirele (full details Appendix (D). These domains portray both discrete (1) and continuous (2-7)
dynamics, and environments that differ either in the rewards (1-5) transitions (6-7). Domains (3), (4) and (5) are high-dimensional, and the navigation problems (1-3, 7) require non-trivial exploration behavior to quickly identify the task. Importantly, relating the MDP ambiguity discussion in Sec. 4 optimal policies for domains (1-3, 7) have non-overlapping identifying states; here we expect MDP ambiguity to be problem. On the other hand, in domains (4-6) the identifying states are expected to overlap, as the rewards/transition differences are dense. To evaluate performance, we measure average reward in the first 2 episodes on unseen tasks- this is where efficient exploration makes critical difference In the supplementary, we report results for more evaluation episodes.
Data collection and organization: For data collection, we used off-the-shelf DQN (Grid- world) and SAC (continuous domains) implementations. To study the effect of data di- versity, we diversified the offline dataset by modifying the initial state distribution Ponit
*For Gridworld, we measure average reward in the first 1 episades, and for Wind, only in the first episode.
Gridworld
Semi-circle MEAI RR Online PEARL
Ant-Seemi-cirlee
Half-Cheetah-Vel
Reacher-Image
NIBAL Oneine PEARL
Average retur
80Rcs RM
Thompace Sampling
PEARL
PEARL
Train Steps
Train Steps
20
Train Steps
Train Steps
64
12
0z to Us 130 10
Train Steps
Figure 3: Offline performance on domains with varying rewards. We compare BOReL with and without reward relabeling (blue and red, respectively) with Thompson sampling baselines calculated exactly in Gridworld, and using online PEARL and offline MBML for the other domains. Full training curves for baselines appear in the supplementary; here we plot only the best performance.
8

---

to either (1) uniform over large region, (2) uniform over a restricted region, or (3) fixed to a single position. At meta-test time, only the single fixed position is used. The
Wind
Escape-Room
PLAIS.
Awerage return
tasks are episodic, but we want the agent to maintain its
belief between episodes, so that it can continually improve
performance (see Figure[]). We follow Zintgraf et al. [36),
and aggregate & consecutive episodes of length I/ to long trajectory of length k x H, and we do not reset the hidden state in the VAE recurrent neural network after episode termination. For reward relabelling, we replace either the first or last //2 trajectories with trajectories from a randomly chosen MDP, and relabel their rewards. For policy replay we replace trajectories by sampling a new trajectory using the trained RI. policy of another MDP. Technically, network architectures and hyperparameters were chosen similarly to [36), as detailed in the supplementary.
50Ril Pi
sores s/o mn
VBVI w/o PM
PEARL
Train Steps
Train Steps
Figure 4: Offline performance on do- mains with varying transitions. We com- pare BOReL with and without policy re- playing (blue and red, respectively) with online PEARL and offline MBML.
compare our offline algorithm with Thompson sampling based methods, and also with an ablation the reward relabelling method. For Gridworld, the Thomp- son sampling method is computed exactly, while for the continuous environments, we use online PEARL (88] a strong baseline that is not affected by our offline data limitation, and MBML [22] For these results the uniform initial state distribution was used to collect data. Note that we signif- icantly outperform Thompson sampling based methods, demonstrating our claim of learning non-trivial exploration from offline data. These results are further explained qualitatively by ob- serving the exploration behavior of our learned agents. In Figure [ and in Figure 5 we visualize the trajectories of the trained agents in the Semi-circle and Ant-Semi-cirele domains, respectively An approximately Bayes-optimal behavior is evident: in the first episode, the agents search for the goal along the semi-circle, and in the second episode, the agents maximize reward by mov- ing directly towards the already found goal. Similar behaviors for Gridworld and Escape-Room are reported in Appendix F. In contrast, a Thompson sampling based agent will never display such search behavior, as it does not plan to proactively reduce uncertainty. Instead, such an agent will randomly choose an un-visited possible goal at each episode and directly navigate towards it (cf. Figure 1 in )22). We further emphasize that the approximately Bayes-optimal search be- havior is very different from the training data, in which the agents learned to reach specific goals.
Main Results: In Figure
Episode 1
Episode 2
Our results also signify the severity of MDP ambiguity, and align with the theory in Sec. 4 In domains with non-overlapping identify- ing states (1-3, 7), as expected, perfor- mance without policy replaying (RR) is poor, while in domains with overlapping identify- ing states policy replaying has little effect. In Figure 7 in the supplementary, we provide further insight into these results, by plotting the belief update during the episode rollout for Semi-circle: the belief starts as uniform on the semi-circle and narrows in on the target as the agent explores the semi-cirele. With reward rela- belling ablated, however, show that the belief does not update correctly, and the agent believes the reward is at the point it first visited the semi-circle.
Figure 5: Ant-Semi-cirele: trajectories from trained policy on a new goal. In the first episode the ant searches for the goal, and in the second one it directly moves toward the goal it has previously found. This search behavior is different from the goal-reaching behaviors that dominate the data.
"Since the official implementation of MBML does not support discrete domains nor image observations. we omit this baseline for Gridworld and Reacher-Image. We could not get the BCQ component of MBML to produce reasonable results for Wind with policy replaying, nor for Escape- Room with or without policy replaying.
10 Video is provided: https://youtar wew.eeebebbeeee be/Sswg552200

---

Data Quality Ablative Study: To evaluate the dependency of our method on the offline data quality, we report results for the 3 different data collection strategies described above (see sup- plementary for more details), summarized in Ta- ble[] As expected, data diversity is instrumental to offline training. However, we qualitatively show Figure 10 in the supplementary, even on the low-diversity datasets, agents learned non-trivial exploration strategies that search for the goal. This is especially remarkable for the dixe-diistibutton dataset, where is unlikely that any training trajectory traveled along the semi-circle (see supplementary Figure 11
Table Average return in Ant-Semi-cirele for of- fline data with different initial state distributions: Uniform distribution, uniform distribution exclud- ing states on the semi-cirele (Excluding s.c.), and fixed initial position (Fixed).
BOReL
BOReL+CQL
Uniform
171.8 + 7.0
176.0 + 10.2
Excluding
102.8 = 32.7
116.6 + 19.9
Fixed
99.2 = 27.4
112.4 = 31.3
One may ask whether OMRL presents the same challenge as standard offline RL, and whether recent offline RL advances can mitigate the dependency on data diversity. To investigate this, we also compare our method with a variant that uses CQL [9] state-of-the-art offline RL method to train the critic network of the meta-RL agent. Interestingly, while CQL improved results (Table the effect of data diversity is much more significant. Together with results MDP ambiguity, our investigation highlights the particular challenges of the OMRL problem.
7 Conclusion and Future Work
We presented the first approximately Bayes-optimal offline meta-RL algorithm, allowing solve problems where efficient exploration is crucial. The connection between Bayesian RL. and meta learning allows to reduce the problem to offline RL belief-augmented states. However, learning neural belief update from offline data is prone to MDP ambiguity. We formalized the problem, and proposed a simple data collection protocol that guarantees identifiability. In the particular case of tasks that differ in their rewards, our protocol can be implemented in hindsight, for arbitrarily offline data. Finally, we demonstrated state-of-the- fft-hh--rt art results on several challenging domains.
An important investigation that we leave to future work is to formalize the connection between task diversity and task generalization. Additionally, it is intriguing whether other techniques can mitigate MDP ambiguity, for example, by designing exploration policies that induce identifiability, or by using domain knowledge to restrict the hypothesis set of possible MDPs in the belief.
Acknowledgments
This work is partly funded by the Israel Science Foundation (ISF-759/19) and the Open Philanthropy Project Fund, an advised fund of Silicon Valley Community Foundation. The authors would like to thank Luisa Zintgraf for sharing the VariBAD code.
References
[1] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGirew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv: (70.:0/77000149909 2017.
[2] D. P. Bertsekas. Dynamic programming and optimal control, volume 1. Athena Scientific, 1995.
[3] A. R. Cassandra, L. P. Kaelbling, and M. L. Littman. Acting optimally in partially observable stochastic domains. In AAAI, volume 94, pages 1023-1028, 1994.
[4] Clavera, J. Rothfuss, J. Schulman, Y. Fujita, Asfour, and Abbeel. Model-based reinforce- ment learning via meta-policy optimization. In Conference on Rohot Learning, pages 617-629, 2018.
[5] Y. Duan, Schulman, X. Chen, P. I.. Bartlett, Sutskever, and P. Abbeel. RI,3: Fast reinforce- ment learning via slow reinforcement learning, arXiv preprint arXiv:1611.022. 2016.
10

---

[6] M. O. Duff. Optimal Learning: Computational procedures for Bayes-adaptive Markov decision processes. PhD thesis, University of Massachusetts Amherst, 2002.
[7] D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6(Apr):503-55) 556, 2005.
[8] R. Fakoor, P. Chaudhari, S. Soatto, and A. Smola. Meta-q-leaming. arXiv preprint arXiv: 190.00290. 2019.
[9] C. Finn, P. Abbeel, and S. Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1 1126-11355 JMLR. org, 2017.
[10] S. Fujimoto, D. Meger, and D. Precup. Off-policy deep reinforcement learning without explo- ration. arXiv preprint arXiv: /8/ 118..2229000 2018.
[11] M. Ghavamzadeh, S. Mannor, J. Pineau, and A. Tamar. Bayesian reinforcement learning: A 2016.
survey, arXiv preprint 1rXXiv:609.0444
[12] E. Grant, C. Finn, S. Levine, T. Darrell, and T. Griffiths. Recasting gradient- gradient-based based meta-learning as hierarchical bayes. arXiv preprint arXiv: 08930, 2018.
[13] Gu, E. Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipula- tion with asynchronous off-policy updates. In 2017 IEEE international conference roboties and automation (ICRA), pages 3389-3396. IEEE, 2017.
[14] A. Guez, D. Silver, and P. Dayan. Efficient bayes-adaptive reinforcement learning using sample- based search. In Advances in neural information processing systems, pages 1025-1033, 2012.
[15] Z. D. Guo, M. G. Azar, B. Piot, B. A. Pires, and R. Munos. Neural predictive belief representa- tions. arXiv preprint arXiv:18/I.06400 2018.
[16] A. Gupta, R. Mendonca, Y. Liu, P. Abbeel, and S. Levine. Meta-reinforcemmnt learning of structured exploration strategies. In Advances in Neural Information Processing Systems, pages 3302-5311, 2018.
[17] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 202900 2018.
[18] J. Humplik, A. Galashov, L. Hasenelever, P. A. Ortega, Y. W. Teh, and N. Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv:1905.06443 2019.
[19] A. Kumar, A. Zhou, G. Tucker, and S. Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.044777 2020.
[20] G. Lee, B. Hou, A. Mandalika, J. Lee, S. Choudhury, and S. S. Srinivasa. Bayesian policy optimization for model uncertainty. arXiv preprint arXiv:1810.01014, 2018.
[21] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.016643 2020.
[22] J. Li, Q. Vuong, S. Liu, M. Liu, K. Ciosek, H. I. Christensen, and H. Su. Multi-task batch reinforcement learning with metric learning, arXiv preprint arxiv: arxiv: 1909.1133, 2020.
[23] T. P. Lillicrap, J. J. Hunt, A. Pritzel, Heess, T. Erez, Y. Tassa, D. Silver, and Wierstra. Continuous control with deep reinforcement learning. arXiv preprint 150.009,,000
[24] E. Mitchell, R. Rafailov, X. B. Peng, S. Levine, and C. Finn. Offline meta-reiiforcemmnt learning with advantage weighting. arXiv preprint arXiv:2008.06043, 2020.
[25] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, Ostrovski, et al. Human-level control through deep rein- forcement learning. Nature, 51897540) 2015.

---

[26] P. A. Ortega, J. X. Wang, M. Rowland, T. Genewein, Z. Kurth- R. Pascanu, N. Heess, J. Veness, A. Pritzel, P. Sprechmann, al. Meta-learning of sequential strategies. arXiv preprint arXiv: 1905.03030 2019.
[27] J. Peters, D. Janzing, and B. Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
[28] K. Rakelly, A. Zhou, D. Quillen, C. Finn, and S. Levine. Efficient off-policy meta-reinforcement learning via probabilistic context variables. arXiv preprint arXiv:1903.08254 2019.
[29] Rothfuss, D. Lee, Clavera, T. Asfour, and P. Abbeel. Promp: Proximal meta-policy search. arXiv preprint
2018.
[30] E. Sarafian, A. Tamar, and S. Kraus. Safe policy learning from observations. arXiv preprint arXiv: 1805008805 2018.
[31] G. Schoetteer, A. Nair, A. Ojea, S. Levine, and E. Solowjow. Meta-reinforcement learning for robotic industrial insertion tasks. arXiv preprint arXiv:2004. 144044 2020.
[32] D. Schwab, T. Springenberg, M. F. Martins, T. Lampe, M. Neunert, A. Abdolmaleki, T. Her- tweck, R. Hafner, Nori, and M. Riedmiller. Simultaneously learning vision and feature-based control policies for real-world ball-in-a-cup. arXiv preprint arXiv: 10Kiv:9902.044: 04706, 2019.
[33] B. C. Stadie, G. Yang, Houthooft, X. Chen, Y. Duan, Y. Wu, Abbeel, and Sutskever. Some considerations on learning to explore via meta-reinforremmnt learning. arXiv preprint 1rX3v: 118, 2018.
[34] H. Van Seijen, Van Hasselt, S. Whiteson, and M. Wiering. A theoretical and empirical analysis of expected sarsa. In 2009 ieee symposium on adaptive dynamic programming and reinforcement learning, pages 177-184. IEEE, 2009.
[35] X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Ku- maran, and M. Botvinick. Learning reinforcement learn. arXiv preprint arXiv: 161..066, 2016.
[36] L. Zintgraf, K. Shiarlis, M. Igl. S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: A very good method for bdyssvadaptive deep rl via meta-learning In International Conference Learning Representation (ICLR), 2020.
12


Reviews:
[{"review_id": "p3NJTGJ8Dd", "reviewer": "Reviewer_cUbW", "summary": "The paper studies the problem of Offline Meta-Reinforcement Learning (OMRL). In this setting, one observes M trajectories coming from N different environments sampled from the same underlying distribution. The goal of the agent is to perform well in expectation on a new unseen MDP drawn from the same distribution. To do so, the authors propose relabelling strategies to mitigate the problem of MDP ambiguity they introduce/formalize and to enable off-policy learning for the VariBad algorithm. The resulting algorithm was tested and compared with PEARL (online) and MBML (offline) on a series of environments with discrete or continuous action spaces, and sparse or dense rewards.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros**\n\n- The problem investigated in this work is novel and will certainly gain importance in the future. The paper poses the problem well and qualitatively discusses its challenges, lying the groundwork (MDP ambiguity) for future contributions.\n- The relabelling strategies seem simple to implement and efficient in mitigating the MDP ambiguity introduced in the paper.\n- I liked the different environments used, with both sparse and dense rewards highlighting the difference in performance when using the relabelling strategies.\n\n**Cons/Questions**\n\nMy main concern regards the positioning of the paper as doing offline RL. The setup certainly is but the solution isn't.\n\n- The policy relabelling assumes that one can access the learned policies and that additional data can be collected. The offline RL setup would be greatly simplified if one could make such an assumption (in practice). I acknowledge that the authors clearly stated this issue -  L235-237: \"We importantly note that changing the data collection method in hindsight is not suitable for the offline setting. Therefore, the proposed scheme should be viewed as *a guideline for effective OMRL data collection*.\" Nonetheless, the paper is still advertised as doing offline RL.\n- The reward relabelling suffers the same limitation. The authors assume that the reward of transitions can be relabelled. This requires access to the reward signal. I agree this is a softer assumption as opposed to having access to the full environment dynamic. The authors mentioned this is not a problem - L254-257: \"We remark that the assumption on known reward (during training) is mild, as the reward is the practitioner’s method of specifying the task goal, which is typically known; this assumption is also satisfied in all meta-RL studies to date.\" For meta-RL it might be true, but this is a hard assumption when working in an offline setting.\n\n**Reason For Rating**\n\nI consider this paper as a good stepping stone on which the research community can build. The framing of the problem, as well as the discussion around the ambiguity in identifying the MDP, is well conducted. The experimentation is fair and adequate to highlight the benefit of their approach. However, I would encourage the authors to re-position the paper as the offline setting is violated on multiple occasions in their solution. Despite these, I consider the work to be a valuable contribution to the research community and I could be swayed if my concerns are answered.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no social impact to consider in this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "dn6-JrmWl2M", "reviewer": "Reviewer_YKZ2", "summary": "The paper addresses the problem of learning Bayes-Optimal Exploration strategies from offline data.\nThe authors discuss MDP ambiguity problems that may arise in this context and propose BoRel an adaptation\nof the online, on-policy VariBad algorithm that addresses these issues via policy replaying and reward relabeling.\nFinally, the algorithm is empirically assessed on several baselines and compared to a number of existing (online) methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper studies a highly relevant problem setting and is technically sound. It is very clearly structured and well-written. The discussion of MDP ambiguity is novel and highlights an important avenue for future research. However, several remaining weaknesses are listed in the following:\n\n1. The adaptation of VariBad for off-policy learning seems to be of limited novelty. The authors claim that VariBad makes no use of the BAMDP parameters. However, the VAE component (approximately) learns to compute a sufficient statistic of the interaction history which the policy network is conditioned on. This can be interpreted as a posterior over the BAMDP parameters and thus the VariBad policy network acts on hyperstates (which is known to be enough to enable computation of the optimal policy).\nSwitching out the on-policy component with an off-policy one seems trivial as does the re-computation of belief posteriors once the inference VAE is ``  `  fully'' trained.\n\n2. On the topic of training the VAE in the offline setting. The trajectory distribution induced by the final policy is likely to differ significantly from the one induced during data collection. Unless I am mistaken, this would be equivalent to changing the data distribution between training and test time in a ``standard'' supervised context - something that can lead to rather arbitrary behaviour. In light of this, I would be curious as to how a reasonably robust belief computation can be ensured?\n\n3. I find the proposed policy replay strategy somewhat problematic in the context of the offline meta-RL setting. The authors even note that data generally might not be collected specifically for the meta-RL task. Consequently, I do not belief that the agent should be able to prescribe data collection policies.\n\n4. The proposed reward relabeling assumes access to the reward functions of all example tasks which is somewhat limiting. Additionally, it seems to be strongly related to other data augmentation techniques e.g. Hindsight Experience Replay which should at least be mentioned, if not discussed in more detail. \n\n5. While the issue of MDP ambiguity is clearly described and well-illustrated through examples, its consequences in practice remain somewhat unclear. Are there any results regarding the best possible performance one can hope to achieve in its presence? What happens when we cannot make the additional assumptions required for policy replay and reward relabeling? \n\n6. The authors compare to online PEARL claiming it to be a strong baseline, however, other meta-RL algorithm have achieved higher performances (e.g. VariBad itself). Additionally, since PEARL is an online algorithm it needs to both curate its own data (which in BoRel is already done by the expert policies, policy replay and reward relabeling) as well as extract the optimal meta-policy from that data. This implies that PEARL is solving a harder problem and the comparison is somewhat unfair. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"ethical_concerns": "-", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "YZRsBA7qqBg", "reviewer": "Reviewer_Gv9V", "summary": "The paper is doing quite a few things at once:\n\n1. Proposing an algorithm for learning effective exploration policies for meta-RL from only offline data\n2. Defining the offline meta-RL problem (not technically new but concurrent with other recent work)\n3. Defining the MDP ambiguity problem, which proves troublesome for inference-based meta-RL algorithms, and providing two baseline approaches to solving it\n\nThe authors accomplish 1 by converting the VariBAD to work with off-policy data and 3 with intuitive strategies for additional data collection or augmentation, which provably address the problem. The experiments suggest that the proposed solutions are necessary to solve problems for which identifying the MDP is difficult.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is generally well written and focuses on an interesting, important problem. The solutions presented are intuitive and well-motivated, and the experimental results match the theoretical claims. My primary concerns are (a) the impracticality of the data collection/augmentation solution to MDP ambiguity and (b) the explanation of the ambiguity problem and how identifiability is connected to overlapping identifying states.\n\n### Writing\n\nThe writing is clear. A concise and informative recapitulation of necessary technical background information needed to understand the problem is given; the Background section has pedagogical value.\n\nOne confusion I had re: L149: since R^+ and P^+ are unknown (the original data comes from regular MDPs), isn't it somewhat unsurprising that the VariBAD objective doesn't refer to these explicitly?\n\n### Novelty & Significance\n\nThe paper describes an off-policy variant of an existing algorithm, VariBAD, as well as two data collection/augmentation approaches to solving the MDP ambiguity problem. While the solutions are not particularly surprising, but they are well-motivated and explained clearly. Overall, the contributions are sufficiently novel and their effectiveness suggests they are significant as well.\n\n### Experimental Evaluations\n\nThe experimental evaluations are generally intended to assess whether the proposed algorithm learns useful exploration policies as well as whether or not MDP ambiguity presents a problem in practice. The experiments are reasonably convincing relative to other papers in this area, although several environments are very simple.\n\n### Remaining Questions\n\nOverall, I think this is a solid paper that probably deserves acceptance on the basis of clear motivation, well-motivated solutions, and reasonably convincing experiments. Again, my biggest point of confusion is regarding the connection of identifiability of data (used in the theory) to the (not clearly defined as far as I can tell) notion of \"overlapping states.\" I also have some concerns that the proposed data relabeling/augmentation approaches are not viable in practice. In order to raise my score, the authors could convincingly answer a few questions:\n\n- Can the authors more clearly define what \"overlap\" (e.g. L214) means and how it is connected to identifiability? Right now, there is a gap in my understanding of how the identifiability is linked to the specific experimental domains presented, because the identifying states are defined in terms of *pairs* of MDPs, not single MDPs.\n- Although the authors mention that constraining the hypothesis space of MDPs is not desirable, this approach would be attractive because it does not require additional data collection/augmentation. Can the authors comment on the viability of such an approach to solving MDP ambiguity problem, or conjecture what such an approach might look like?\n- Is extrapolation error not a problem when applying off-policy VariBAD to fully offline data, as is a problem in offline RL?\n\n*Update after discussion period*: in light of the authors' responses to my questions as well as the other reviewers, I have decided to raise my score to 7. I think the characterization of the MDP ambiguity problem is a meaningful contribution that will be of interest to the NeurIPS community, even if the proposed solutions to it are relatively straightforward. The authors can improve their paper (as discussed previously in this forum) by further explaining the significance of the problem they describe and the practical limitations of the solutions that they propose.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some discussion of safety during exploration for RL/meta-RL systems would be welcome. In addition, the authors do not describe the technical limitations of their work. In particular, describing settings for which the proposed data collection/augmentation strategies are not practical would be worthwhile.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces an interesting problem, where the meta-agent is required to quickly learn unseen tasks given some offline data for training tasks. In addition, the paper also formally introduces the MDP ambiguity problem induced by the new problem setup. Although the proposed solution (off-policy version of VariBAD + reward relabelling) is a little bit straightforward, it is a reasonable solution to the problem considered in this paper, and the empirical results are also solid. During the post-rebuttal discussion period, the majority of the reviewers agreed that the formal description of the MDP ambiguity problem and the Bayesian RL perspective is interesting enough to be presented at NeurIPS. Therefore, I recommend accepting this paper.
Decision: Accept (Poster)