Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
Authors: Evan Zheran Liu
Aditi Raghunathan
Percy Liang
Chelsea Finn
Venue:  (2021)
Keywords: meta-reinforcement learning
reinforcement learning
exploration

Abstract: The goal of meta-reinforcement learning (meta-RL) is to build agents that can quickly learn new tasks by leveraging prior experience on related tasks. Learning a new task often requires both exploring to gather task-relevant information and exploiting this information to solve the task. In principle, optimal exploration and exploitation can be learned end-to-end by simply maximizing task performance. However, such meta-RL approaches struggle with local optima due to a chicken-and-egg problem: learning to explore requires good exploitation to gauge the exploration's utility, but learning to exploit requires information gathered via exploration. Optimizing separate objectives for exploration and exploitation can avoid this problem, but prior meta-RL exploration objectives yield suboptimal policies that gather information irrelevant to the task. We alleviate both concerns by constructing an exploitation objective that automatically identifies task-relevant information and an exploration objective to recover only this information. This avoids local optima in end-to-end training, without sacrificing optimal exploration. Empirically, DREAM substantially outperforms existing approaches on complex meta-RL problems, such as sparse-reward 3D visual navigation.

Under review as a conference paper at ICLR 2021
DECOUPLING EXPLORATION AND EXPLOITATION FOR META- REINFORCEMENT LEARNING WITHOUT SACRI- FICES
Anonymous authors Paper under double-blind review
ABSTRACT
The goal of meta-reinffrreement learning (meta-RL) is to build agents that can quickly learn tasks by leveraging prior experience on related tasks. Learning a new task often requires both exploring to gather task-relevant information and exploiting this information solve the task. In principle, optimal exploration and exploitation can be learned end-to-end by simply maximizing task performance. However, such meta-RL approaches struggle with local optima due to chicken- and-egg problem: learning to explore requires good exploitation to gauge the exploration's utility, but learning to exploit requires information gathered via exploration. Optimizing separate objectives for exploration and exploitation can avoid this problem, but prior meta-RL. exploration objectives yield suboptimal policies that gather information irrelevant to the task. We alleviate both concerns by constructing an exploitation objective that automatically identifies task-relevant information and exploration objective to recover only this information. This avoids local optima in end-to-end training, without sacrificing optimal exploration. Empirically, DREAM substantially outperforms existing approaches complex meta-RL, problems, such as sparse-reward 3D visual navigation [
INTRODUCTION
A general-purpose agent should be able perform multiple related tasks across multiple related environments. Our goal is to develop agents that can perform a variety of tasks in novel environments, based on previous experience and only a small amount of experience example, we may want a robot to cook a meal (a new task) in a new kitchen (the environment) after it has learned cook other meals in other kitchens. To adapt to a new kitchen, the robot must both explore to find the ingredients, and use this information to cook. Existing meta-reinforcemment learning (meta-RL. methods can adapt to new tasks and environments, but, as we identify in this work, struggle when adaptation requires complex exploration.
the new environment. For
In the meta-RL setting, the agent presented with a set of meta-training problems, each in an environment (e.g., a kitchen) with some task (e.g., make pizza); at meta-test time, the agent is given new, but related environment and task. It is allowed to gather information in a few initial (exploration) episodes, and its goal is to then maximize returns on all subsequent (exploitation) episodes, using this information. A common meta-RL. approach is to learn to explore and exploit end-ro-end by training policy and updating exploration behavior based how well the policy later exploits using the information discovered from exploration (Duan et al. 2016, Wang et al.) [0166] Stadie et al. 2018) Zintgraf al. 2019 Humplik et al. 2019). With enough model capacity, such approaches can express optimal exploration and exploitation, but they create a chicken-and-egg problem that leads to bad local optima and poor sample efficiency: Learning explore requires good exploitation to gauge the exploration's 's utility, but learning to exploit requires information gathered via exploration; therefore, with only final performance as signal, one cannot be learned without already having learned the other. For example, a robot chef is only incentivized to explore and find the ingredients if it already knows how to cook, but the robot can only learn to cook if can already find the ingredients by exploration.
To avoid the -addeen-anddegg problem, we propose to optimize separate objectives for exploration and exploitation by leveraging the problem an easy-to-provide unique one-hot for each training meta-
Project weh page: httpss///anooryrroospppgrrruuubb s8 thub. 1

---

Under review as a conference paper at ICLR 2021
training task and environment. Such problem ID can be realistically available in real-world meta- RL tasks: c.g., in a robot chef factory, each training kitchen (problem) can be easily assigned a unique ID, and in recommendation system that provides tailored recommendations to each user, each user (problem) is typically identified by a unique username. Some prior works (Humplik et al. 2019
Kamienny et al. 2020) also use these problem IDs, but not in way that avoids the chicken-and-en
problem. Others (Rakelly et al., 2019 Zhou et al. [011b] Gupta et al. 2018 Gurumurthy et al. 2019)
Zhang al., 2020) also optimize separate objectives, but their exploration objectives learn suboptimal policies that gather mask-irrelevany information (e.g., the color of the walls). Instead, we propose an exploitation objective that automatically identifies task-relevant information, and an exploration objective to recover only this information. We learn an exploitation policy without the need for exploration, by conditioning on a learned representation of the problem ID, which provides task- relevant information (e.g., by memorizing the locations of the ingredients for each ID) / kitchen). We also apply an information bottleneck to this representation to encourage discarding of any information not required by the exploitation policy (i.e.. task-irrelevant information). Then, we learn an exploration policy to only discover task-relevant information by training it to produce trajectories containing the same information as the learned ID representation (Section 4.. Crucially, unlike prior work, prove that our separate objectives are consistent: optimizing them yields optimal exploration and exploitation, assuming expressive-enough policy classes and enough meta-training data (Section (.1).
Overall, we present two core contributions: (i) we articulate and formalize a chicken-and-egg coupling problem between optimizing exploration and exploitation in meta-RL (Section [.T]; and (ii) we overcome this with a consistent decoupled approach, called DREAM: Decoupling exploRation and ExploitAtion in Meta-RL (Section 4.2). Theoretically, in a simple tabular example, we show that addressing the coupling problem with DREAM provably improves sample complexity over existing end-to-end approaches by a factor exponential in the horizon (Section 55). Empirically, we stress test DREAM's is ability to learn sophisticated exploration strategies on 3 challenging, didactic benchmarks and a sparse-reward 3D visual navigation benchmark. On these, DREAM learns to optimally explore and exploit, achieving 90% higher returns than existing state-of- tht-hhhrtt approaches (PEARL, E-RL?, IMPORT, VARIBAD), which struggle to learn an effective exploration strategy (Section (6).
2 RELATED WORK
We draw on long line of work on learning to adapt to related tasks (Schmidhuber) 1987 7 Thrun &
Pratt 2012. Naik& & Mammone 1992 Bengio et al. 1991 1992 Hochreiter et al. [00l||nndyycoov
icz et al. [016] Santoro al. 2016). Many meta-R RL works focus on adapting efficiently to new task from few samples without optimizing the sample collection process, via updating the policy parameters (Finn et al. 2017 Agarwal et al. 2019: Yang et al., 2019 Houthooft et al. 2018 Men- donca et al. 2019), learning a model (Nagabandi et al.. 2018 [Semunssoon et al.) [0188] Hiraoka et al. 2020), multi-task learning (Fakoor et al. 2019), or leveraging demonstrations (Zhou et al., 2019a). In
contrast, we focus on problems where targeted exploration is critical for few-shot adaptation.
Approaches that specifically explore to obtain the most informative samples fall into two main categories: end-to-ent and decoupled approaches. End-to-end approaches optimize exploration and exploitation end-to-end by updating exploration behavior from returns achieved by exploitation (Duan al. 2016 Wang et al. [00 6al Mishra et al. 2017 Rothfuss al.) 2018 Stadie et al. 2018 Zintgraf
et al. 2019 Humplik et al. 2019 Kamienny et al. 2020 & Tamar (020). These approaches
can represent the optimal policy (Kaelbling et al. 1998). but they struggle to escape local optima
due to a chicken-and-egg problem between learning to explore and learning to exploit (Section 4.1). Several of these approaches (Humplik et al. 2019 Kamienny et al. 2020) also leverage the problem ID during meta-training, but they still learn end-to-end, so the chicken-ande -aee--nnd-egg problem remains.
Decoupled approaches instead optimize separate exploration and exploitation objectives. via, e.g.. coompson-sampling (TS) [Thompson] [933; Rakelly et al. 2219)) obtaining exploration trajectories predictive of dynamics or rewards (Zhou et al. | [009b]] | Gurumurthy et al. 2019 /Zhang et al. 2020). or exploration noise (Gupta et al.) 2018). While these works do not identify the chicken-and-egee
problem, decoupled approaches coincidentally avoid it. However, existing decoupled approaches, including those (Rakelly et al. 2019: Zhang et al. 2020) that leverage the problem ID, do not learn optimal exploration: TS (Rakelly et al. 2019) explores by guessing the task and executing a policy for that task, and hence cannot represent exploration behaviors that are different from exploitation (Russo) et al. 2017). Predicting the dynamics (Zhou et al. 2019b; Gurumurthy et al., 2019: Zhang et al. 2020) is inefficient when only a small subset of the dynamics are relevant solving the task. In
2

---

Under review as a conference paper at ICLR 2021
F
Cook
exploration episode (gather information)
exploitation episode 1
exploitation episode N Isalve the task and maximize returna)
Cook pizza [S, R,, T,,)
exploration episode leather information)
(S.A, R., TT.)
Trial
Trial
Meta-RL setting: Given new environment and task, the agent is allowed to first explore and gather information, and then must use this information to solve the task in subsequent exploitation episodes.
Figure I:
contrast, we propose separate mutual information objective for exploration, which both avoids the
bhicken-and-eed problem and yields optimal exploration when optimized (Section [5. Past work
(Gregor et al.] 2016 Houthooft et al. [2016] Eysenbach et al. 2018 Warde-Farley et al. 2018) also
optimize mutual information objectives, but not for meta-RI..
Exploration in general RL. The general RL setting (i.e., learning from scratch) also requires targeted exploration to gather informative samples that enables learning a policy to solve the problem. In contrast to exploration algorithms for general RL (Bellemare et al. 2016 Pathak et al. [017] Burda et al. 2018 Leibfried et al., 2019), which must visit many novel states to find regions with high reward, exploration in meta- RL can be even more targeted by leveraging prior experience from different problems during meta-traine metaa-raininng As a result, DREAM can learn new tasks in just two episodes (Section G while learning from scratch can require thousands even millions of episodes.
3 PRELIMINARIES
Meta-reinforcemenn learning. The meta-RI. setting considers family of Markov decision processes (MDPs) (S, A, R,,, Tu) with states S, actions A, rewards Ry, and dynamics T,. indexed by a one- hot problem ID j M, drawn from distribution p(u). Colloquially, we refer to the dynamies the environment, the rewards as the task, and the entire MDP as the problem. Borrowing terminology from Duan al. 2016), meta-training and meta-testing both consist of repeatedly running trials. Each trial consists of sampling a problem ID μ ~ p(1)) and running N + 1 episodes on the corresponding problem. Following prior evaluation settings (Finn et al., 2017 Rakelly al. 2019 Rothfuss et al.) 2018 Fakoor et al. 2019), we designate the first episode in a trial as an exploration episode consisting of steps for gathering information, and define the goal as maximizing the returns the subsequent N exploitation episodes [Figure[]. Following Rakelly et al. (201)); (Humplik et al. (2019); Kamienny et al. 2202)) the pasy-io-provide problem is available for meta-training. but not meta-testing trials.
We formally express the goal in terms of an exploration policy T used in the exploration episode and an exploitation policy task used in exploitation episodes, but these policies may be the same or share parameters. Rolling out TP in the exploration episode produces an exploration trajectory -ox = (sg, ao: .o. 8T), which contains information discovered via exploration. The exploitation policy x z-lnsk may then condition op and optionally, its history across all exploitation episodes in trial, to maximize exploitation episode returns. The goal is therefore to maximize:
J (m uskk
Buo-pl)) [ (70*P; u)]
where (7 vt) is the expected returns of tusk conditioned on oxp, summed over the N exploita- tion episodes in a trial with problem ID ja.
End-to-end meta-RL. A common meta-RL approach WWag et al. 2016a Duan et al. 2016 Rothfuss
al. 2018 Zintgraf et al. 2019 Kamienny et al.) 2020; Humplik et 2019) is to learn explore and exploit end-lo--nd by directly optimizing J in (L) updating both from rewards achieved during exploitation. These approaches typically learn a single recurrent policy I T(a | St, T.t) for both exploration and exploitation (i.c., x tusk x = π), which takes action a given state S; and history of experiences spanning all episodes in a trial T: = (So, dg, To..... , (t-1. Intuitively, this policy is learned by rolling out a trial, producing an exploration trajectory exp and, conditioned on rexp and the exploitation experiences so far, yielding some exploitation episode returns. Then, credit is assigned to both exploration (producing 7dP)) and exploitation by backpropagating the exploitation returns through the recurrent policy. Critically, estimates of the expected exploitation returns in I (e.g., from single roll-out value-function approximation) form the learning signal for exploration. Directly optimizing the objective J this way can learn optimal exploration and exploitation strategies, but optimization is challenging, which show in Section 4.1
3

---

Under review as a conference paper at ICLR 2021
(a)
Coupled Exploration and Explontation (Endi to and Optimization
(b)
DREAM: Excloupleddxxoloooiooo and ExploitAtion Meta-RL Problam
Leamed
IiF (
6
Imeta-training)
Tink
max,
(meta-lesting)
V
Figure 2: (a) Coupling between the exploration policy Th and exploitation policy π nink These policies are illustrated separately for clarity, but may be single policy. Since the two policies depend on each other (for gradient signal and the 70P distribution), is challenging to learn when the other policy has not learned. (b) DREAM: th and n=* are learned from decoupled objectives by leveraging a simple one-hot problem ID during meta-training. At meta-test time, the exploitation policy conditions on the exploration trajectory as before.
4
DECOUPLING EXPLORATION AND EXPLOITATION
4.1 THE PROBLEM WITH COUPLING EXPLORATION AND EXPLOITATION
We begin by showing that end-to-end optimization struggle with local optima due to chicken-and- egg problem. Figure 2 illustrates this. Learning nh* relies gradients passed through ~= If r cannot effectively solve the task, then these gradients will be uninformative. However, to learn to efficiently solve the task, rask needs good exploration data (trajectories 70P) from good exploration policy TP This results in bad local optima as follows: if our current (suboptimal) x task obtains low rewards with a good informative trajectory zep the low reward would cause ~0'P to learn to not generate exp "good" This causes "xP to instead generate trajectories Tadd that lack information required to obtain high reward, further preventing the exploitation policy T from learning. Typically, early in training. both "P and t are suboptimal and hence will likely reach this local optimum. In Section 5.2 illustrate how this local optimum can cause sample inefficiency in simple example.
exp
4.2 DREAM: DECOUPLING EXPLORATION AND EXPLOITATION IN META-LEARNING
While we can sidestep the local optima of end-to-end training by optimizing separate objectives for exploration and exploitation, the challenge is to construct objectives that yield the same optimal solution as the end-to-end approach. We now discuss how we can use the casy-to-provine problem IDs during meta-training trainttaannng to do so. good exploration objective should encourage discovering task-relevant distinguishing attributes of the problem (e.g., ingredient locations), and ignoring task- irrelevant attributes (c.g., wall color). To create this objective, the key idea behind DREAM to learn to extract only the task-relevant information from the problem ID, which encodes all information about the problem. Then, DREAM'S 's exploration objective seeks to recover this relk-relevant information.
Concretely, DREAM extracts only the tesk-relevant information from the problem ID f. via stochastic encoder (x μ.). To learn this encoder, we train an exploitation policy task to maximize rewards, conditioned on samples from Fa μ1), while simultaneously applying an information bottleneck to to discard information not needed by glnnk (i.e., task-irrelevant information). Then, DREAM learns an exploration policy TP to produce trajectories with high mutual information with z. In this approach, the exploitation policy task longer relies effective exploration from "x to learn, and once (z | jt) is learned, the exploration policy also learns independently from r decoupling the two optimization processes. During meta-testing, when 1s unavailable, the two policies easily combine, since the trajectories generated by "d are optimized to contain the same information as the encodings: 2 (2 p)) that the exploitation policy jlukk trained on (overview in Figure 2
Learning the problem ID encodings and exploitation policy, We begin with learning stochastic encoder F (a (4) parametrized by t and exploitation policy nh parametrized by 6, which conditions on 2. We learn F& jointly with task by optimizing the following objective:
maximize FF [v (2:11) -X I(2;)): v.0
(2)
Roward
Information bottlencek
where V (2) is the expected return of xt on problem μ given and encoding The information bottleneck term encourages discarding any (task-irrelevant) information from & that does not help maximize reward. Importantly, both terms independent of the exploration policy ,T,, This objective is derived from forming the Lagrangian of constrained optimization problem, where A- is the dual variable, detailed in Appendix F
4

---

Under review as a conference paper at ICLR 2021
We minimize the mutual information (2:; by minimizing variational upper bound on it, Ey (Fo(2 p)][r())),, where r is any prior and 2 is distributed as po(z) = J Folz H)p(p)du.
Learning an exploration policy given problem ID encodings. Once we've ve obtained an encoder Fø ju) to extract only the necessary task-relevant information required to optimally solve each task, we can optimize the exploration policy πh to produce trajectories that contain this same information by maximizing their mutual information [(Te0p; z). We slightly abuse notation use g to denote the probability distribution over the trajectories 7rx, Then, the mutual information I [TT 2) can be
efficiently maximized by maximizing variational lower bound (Barber Agakov 2003 as follows:
I( [(T 2)) (2) H(2 Tap)) 2 H(z) + Eu Fo moumee [log 9u 70P)]
-2(2) + Bu [lo (z)] + PP gra
log ( (=1 log qu (2 7
where qu is any distribution parametrized by w. We maximize the above expression over to learn qu that approximates the true conditional distribution p(z Toxp), which makes this bound tight. In addition, we do not have access to the problem je at test time and hence cannot sample from Po μt). Therefore, 9u serves as a decoder that generates the encoding 2 from the exploration trajectory 7SP,
Recall, our goal is to maximize (3) w.r.t,, trajectories 7St from the exploration policy T#P,, Only the third term depends on TTP,, so we train Th on rewards set to be this third term (information gain):
CxP (at. St+1, :) = (g [log (z [$ ar: ( 1= - log (u [=--1]] - c. (4)
Intuitivelyy, the exploration reward for taking action a, and transitioning state 8t+1 is high if this transition encodes more information about the problem (and hence the encoding 2 ~ Fg p)) than was already present in the trajectory T = (So: do, Tu,-.., 8t-2; Qt- rt-2). We also include a small penalty c encourage exploring efficiently in few timesteps as possible. This reward is attractive because (i) it is independent from the exploitation policy and hence avoids the local optima described in Section 4.1 and (ii) it is dense, so it helps with credit assignment. It is also non-Markov, since it depends on 7 exp so we maximize it with a recurrent (a, st, TT) parametrized by 6.
4.3 A PRACTICAL IMPLEMENTATION OF DREAM
Altogether, DREAM learns four separate neural network components, which we detail below.
1. Encoder (= pa): For simplicity, we parametrize the stochastic encoder by learning determin- istic encoding fo (f)) and apply Gaussian noise, i.e., Fz 14) = (1), We choose a convenient prior r=z) ) to be a unit Gaussian with same variance p² I, which makes the information bottleneck take the form of simple 4 (erreullarizatonn (x)
2. Decoder (z p):): Similarly, we parametrize the decoder q. Txxp) as a Gaussian cen- tered around a deterministic encoding (T*P) with variance I. Then, qu maximizes ||2] w.r.t., w (Equation 3) and the exploration rewards take the form
Fe || pex (a s', oxP; (u) (M) 9- ([>S0) a; a;; sT)112 (1) g- ([7*))) e (Equation 4
|- (Txxp)))
3. Exploitation policy x/. and 4. Exploration policy T We learn both policies with double deep Q-learning (van Hasselt et al. 2016), treating (s, as the state for mraak.
For convenience, we jointly learn all components in an EM-like fashion, where in the exploration episode, we assume f and T.sa are fixed. Appendix A includes all details and a summary (Algo- rithm
Overall, DREAM avoids the chicken-and-egg problem in Section 4.1 DREAM learns the exploitation policy and encoder completely independently from exploration. This enables the encoder to learn quickly, and once it has learned, it (together with the decoder) forms a learning signal for exploration separate from the expected exploitation returns, which improves sample efficiency (Section 5.2)
ANALYSIS OF DREAM
5.1 THEORETICAL CONSISTENCY OF THE DREAM OBJECTIVE
key property of DREAM is that is consistent: maximizing our decoupled objective also maximizes expected returns (Equation) l). This contrasts prior decoupled approaches (Zhou et al. 2019b: Rakelly
5

---

Under review as a conference paper at ICLR 2021
(4)
Sample Complexity
(b)
Exploration Q volues
[4)
Expliitation Q valure
(4)
Returss
Somple (10), ant Cntrett
DREAM
Sub optimal
Optinal o.
RL
LAP (ox)-4))
Number Actions (1AI]
Number Samples
Nuubber Samples
Number Samplex
Figure 3: (a) Sample complexity of learning the optimal exploration policy as the action space |A|| grows (1000
seeds). (b) Exploration Q-values (re The policy arg max, (a)((a is optimal after the dot. (c) Exploitation
values given optimal trajectory (TEP). (d) Returns achieved on a tabular MDP with |.A| = 8 (3 seeds).
ct al. 2019; Gupta et al. 2018; Gurumurthy et al. 2019 Zhang et al. 2020). which also decouple exploration from exploitation, but do not recover the optimal policy even with infinite data. Formally,
Proposition 1. Assume A. Ta) is ergodic for all problems μ M. Let (*) be the maximum expected returns achievable by any exploitation policy with access to the problem ID p. i.e., with complete information. Let meask ne F., and q.(2 (ev) be the optimizers of the DREAM objective. Then, if the function classes DREAM optimizes over are well-specified, there exists a finite T such that if the length of the exploration episode is at least T.
V= (2: = Ep-p(n) [V'(.).
E Eyropla), sempl mom sr (e/pem)
Optimizing DREAM'S objective achieves the maximal returns (*) even without access to μ during meta-testing (proof in Appendix (.TT) We can remove the ergodicity assumption by increasing the number of exploration episodes, and DREAM performs well on non-ergodic MDPs in our experiments.
5.2 AN EXAMPLE ILLUSTRATING THE IMPACT OF COUPLING ON SAMPLE COMPLEXITY
With enough capacity, end-to-end approaches can also learn the optimal policy, but can be highly sample inefficient due to the coupling problem in Section 4.1 We highlight this in simple tabular example to remove the effects of function approximation: Each episode is a one-step bandit problem with action space A. Taking action a, in the exploration episode leads to trajectory 7 that reveals the problem ID μæ; all other actions a reveal no information and lead to T.P. The ID ş identifies a unique action that receives reward 1 during exploitation; all other actions get reward O. Therefore, taking a, during exploration is necessary and sufficient to obtain optimal reward 1. We now study the number of samples required for RI.² (the canonical end-to-end approach) and DREAM to learn the optimal exploration policy with e-greedy tabular Q-learning. We precisely describe a more general setup in Appendix C.2 and prove that DREAM learns the optimal exploration policy in ((IA [M) times fewer samples than RL² in this simple setting with horizon H. Figure 3a empirically validates this result and we provide intuition below.
In the tabular analog of RL2, the exploitation Q-values form targets for the exploration Q-values: (STP( <- frrusk exp) == max Quask (Top) .a)). We drop the fixed initial state from notation. This creates the local optimum in Section 4.1 Initially frrnst (T)))) is low, the exploitation policy has not learned to achieve reward, even when given T, This causes (aPP(a.) to be small and therefore arg max, (a))(a) 7 a, (Figure 3b), which then prevents Plask (7447) from learning (Figure 3 as T. is roughly sampled only once per LAI episodes. This effect is mitigated only when ((sp((.)) becomes higher than (a) for the other uninformative a's (the dot in Figure 3p-d). Then, learning both the exploitation and exploration Q-values accelcrates, but getting there takes many samples.
In DREAM, the exploration Q-values regress toward the decoder q: (a))(a) + ( | (a)) This decoder learns much faster than Quaa, since it does not depend the exploitation actions. Consequently, DREAM'S exploration policy quickly becomes optimal (dot in Figure (3)), which enables quickly learning the exploitation Q- and achieving high reward (Figures 3 and 3
In general, DREAM learns in far fewer samples than end-to-end approaches, since in end approaches like RL?, exploration is learned from quantity requiring many samples to accurately estimate (i.e., the exploitation Q- Q-values in this case). Initially, this quantity is estimated poorly, so the signal for exploration can erroneously "down weight" good exploration behavior, which causes the

---

Under review as a conference paper at ICLR 2021
chicken-and-ege problem. In contrast, in DREAM, the exploration policy learns from the decoder, which requires far fewer samples to accurately estimate, avoiding the chicken-and-en -engeegg-n problem.
6 EXPERIMENTS
Many real-world problem distributions (e.g., cooking) require exploration (e.g., locating ingredients) that is distinct from exploitation (e.g., cooking these ingredients). Therefore, we desire benchmarks that require distinct exploration and exploitation to stress test aspects of exploration in meta-RL, RL, such as if methods can: (i) efficiently explore, even in the presence of distractions; (ii) leverage informative objects (e.g.. map) to aid exploration; (iii) learn exploration and exploitation strategies that general- ize to unseen problems: (iv) scale to challenging exploration problems with high-dimensional visual observations. Existing benchmarks (e.g.. Meta World (Yu et al. (211)) or MuJoCo tasks like HalfChee- tah tah Velocity (Finn et al. 2017 Rothfuss et al. 2018) were not designed to test exploration and are un- suitable for answering these questions. These benchmarks mainly vary the rewards (e.g., the speed to run at) across problems, so naively exploring by exhaustively trying different exploitation behaviors (e.g., running at different speeds) is optimal. They further don't include visual states, distractors, in- formative objects, which test exploration is efficient. We therefore design new benchmarks meeting the above criteria, testing (i-ii)) with didactic benchmarks, and (iv) with a sparse-reward 3D visual nav- igation benchmark, based on Kamienny et al. (2020), that combines complex exploration with high- dimensional visual inputs. To further deepen the exploration challenge, we make our benchmarks goal- con-coondiioned. This requires exploring to discover information relevant to any potential goal, rather than just a single task (e.g., locating all ingredients for any meal vs. just the ingredients for pasta).
Comparisons. We compare DREAM with statc-of-the-art end-to-end (E-RL² (Stadic et al. 2018). VARIBAD (Zintgraf et al. 2019), and IMPORT (Kamienny et al., 2020)) and decoupled approaches (PEARL-UB, upper bound on the final performance of PEARL (Rakelly et al., 2019). For PEARL- UB, we analytically compute the expected rewards achieved by optimal Thompson sampling (TS) exploration, assuming access to the optimal problem-specific policy and true posterior problem distribution. Like DREAM, IMPORT and PEARL also use the one-hot problem ID, during meta- training. We also report the optimal returns achievable with exploration "No exploration." Where applicable, all methods use the same architecture. The full architecture and approach details in Appendix B.3
We report the average returns achieved by each approach in trials with one exploration and one exploitation episode, averaged over 3 seeds with I-standard deviation error bars (full details in Appendix B b. We periodically evaluate each approach on 100 meta-testing trials, after 2000 meta- training trills. In all plots, the training timesteps includes all timesteps from both exploitation and exploration episodes in meta meta-training trials.
6.1 DIDACTIC EXPERIMENTS
(al We first evaluate the grid worlds shown in Fig- ures 44 and 4p. The state consists of the agent's (x, v)- position, one-hot indicator of the object at the agent's ss po- sition (none, bus, map, pot, or fridge), a one-hot indicator of the agent's inventory (none goal. The actions are move up, down, left, or right; ride bus, which, at bus, teleports the agent another bus of the same color; pick up, which, at a fridge, fills the agent's ss inventory with the fridge's ingredients; and drop, which, the pot, empties the agent's inventory into the pot. Episodes consist of 20 timesteps and the agent receives -0.1 reward at each timestep until the goal, described below, is met (de- tails in Appendix B.T qualitative results in Appendix B.2).
(bl
ingredient), and the
Figure 4: Didactic grid worlds to stress test. exploration. (a) Navigation. (b) Cooking.
Targeted exploration. We first test if these methods can efficiently explore in the presence of distractions in two versions of the benchmark in Figure distracting bus and map. In both, there are 4 possible goals (the 4 green locations). During each episode, goal is randomly sampled. Reaching the goal yields +1 reward and ends the episode. The 4 colored buses each lead to near a different potential green goal location when ridden and in different problems μu, their destinations are set to be I of the 4! different permutations. The distracting bus version tests if the agent can ignore distractions by including unhelpful gray buses, which never needed to optimally reach any goal. In different
7

---

Under review as a conference paper at ICLR 2021
Cooking (Training Problems)
Cooking (Unseen Problems)
Visual Navigation
Average Retuurns
100
0.46
4x
-1.0
-10
0:12
62
54
2008
1890
240
2001
940
2000
1993
2900
2001
UCB
1940
200
2000 3000
3940
Timesteps (1e3)
Timesteps (1e3)
Timesteps
(163)
DREAM
E-RL.
IMPORT
VARBAD
PEARI-UB
Optimal
No exploration
Figure 6: Cooking results: only DREAM achieves optimal reward on training problems (left) logged every 10 meta-training trials, and on generalizing to unseen problems (middle). 3D visual navigation results: only DREAM reads the sign and solves the task (right).
problems, the gray buses lead to different permutations of the gray locations. The map version tests if the agent can leverage objects for exploration by including a map that reveals the destinations of the colored buses when touched.
Figure 5 shows the results after 1M steps. DREAM learns to optimally explore and thus receives optimal reward in both versions: In distracting bus, it ignores the unhelpful gray buses and learns the destinations of all helpful buses by rid- ing them. In map, it learns to leverage informative objects, by visiting the map and ending the exploration episode. Dur- ing exploitation, DREAM immediately reaches the goal by riding the correct colored bus. In contrast, IMPORT and E- RL² get stuck in a local optimum, indicative of the coupling problem (Section[4.) 4.1). which achieves the same returns as no exploration at all. They do not explore the helpful buses or map and consequently sub-optimally exploit by just walking to the goal. VARIBAD learns slower, likely because it learns a dynamics model, but eventually matches the sub-optimal returns of IMPORT and RL in ~3M steps (not shown).
Ditracting Bus / Mao
36-
3.1
-16
Detracting bus (IM stepe)
Map (IM steps)
VamBAD
DREASI (no bottlonoc))
PRASI-UB
E-REZ
Optimal
bavor
Figure 5: Navigation results. Only DREAM
optimally explores all buses and the map.
PEARL achieves sub-optimal returns, even with infinite meta-training (see line for PEARL. UBB) as follows. TS explores by sampling problem ID from its posterior and executing its policy conditioned on this ID. Since for any given problem (bus configuration) and goal, the optimal policy rides the one bus leading to the goal, TS does not explore optimally (i.e., explore all the buses read the map), even with the optimal problem-specific policy and true posterior problem distribution.
coblem--ppccffi
Recall that DREAM tries remove extraneous information from the problem ID with an information bottleneck that minimizes the mutual information [(2; (u) between problem IDs and the encoder (a 14). In distracting bus, test the importance of the information bottleneck by ablating it from DREAM. As seen in ((eft)))(eft)) this ablation (DREAM (no bottleneck)) wastes its exploration on the gray unhelpful buses, since they are part of the problem, and consequently achieves low returns.
Generalization to new problems. We test generalization to unseen problems in a cooking bench- mark (Figure 4D). The fridges on the right each contain I of 4 different (color-coded) ingredients, determined by the problem ID. The fridges' contents are unobserved until the agent uses the "pickup" action at the fridge. Goals (recipes) specify placing correct ingredients in the pot in the right order. The agent receives positive reward for picking up and placing the correct ingredients, and negative reward for using the wrong ingredients. We hold out of the 43 64 problems from meta-training.
Figure 6 shows the results on training (left) and held-out (middle) problems. Only DREAM achieves near-optimal returns on both. During exploration, investigates each fridge with the "pick up" action, and then directly retrieves the correct ingredients during exploitation. E-RL" gets stuck in local optimum, only sometimes exploring the fridges. This achieves 3.8x lower returns, only slightly higher than exploration at all. Here, leveraging the problem ID actually hurts IMPORT compared to E- RL2. IMPORT successfully solves the task, given access to the problem ID, but fails without it. As before, VARIBAD learns slowly and TS (PFARI--BB) cannot learn optimal exploration.
8

---

Under review as a conference paper at ICLR 2021
6.2 REWARD 3D VISUAL NAVIGATION
We conclude with challenging benchmark testing both sophisticated exploration and scalability to pixel inputs. We modify a benchmark from Kamienny al. (2020) to increase both the exploration and scalability challenge by including more objects and visual sign, illustrated in Figure 7 In the 3 different problems, the sign on the right says "blue", "red" or "green." The goals specify whether the agent should collect the key or block. The agent receives +1 reward for collecting the correct object (color specified by the sign, shape specified by the goal), 11 reward for the wrong object, and 0 reward otherwise. The agent begins the episode on the far side of the barrier and must walk around the barrier to visually "read" the sign. The agent's observations are 80 x 60 RGB images and its actions are to rotate left or right, move forward, or end the episode.
Figure 7: 3D Visual Navigation. The agent must read the sign to determine what col- ored object go
DREAM is the only method that learns to read the sign and achieve reward (Figure rightt. All end- to-end approaches get stuck in local optima due to the chicken-ande -nd-eeg coupling problem, where they do not learn to read the sign and hence stay away from all the objects, in fear receiving negative reward. This achieves close to 0 returns, consistent with the results in Kamienny et al. (2020). As before, PEARL- UB cannot learn optimal exploration.
CONCLUSION
In summary, this work identifies a chicken-and-egg problem that end-to-end meta-RL approaches suffer from, where learning good exploitation requires already having learned good exploration and vice-versa. This creates challenging local optima, since typically neither exploration nor exploitation is good at the beginning of meta-training. We show that appropriately leveraging simple one-hot problem IDs allows us to break this cyclic dependency with DREAM. Consequently, DREAM has strong empirical performance on meta-RL problems requiring complex exploration, well as substantial theoretical sample complexity improvements in the tabular setting. Though prior works also leverage the problem ID and decoupled objectives that avoid the chicken-and-egg problem, no other existing approaches can recover optimal exploration empirically and theoretically like DREAM.
REFERENCES
R. Agarwal, C. Liang, D. Schuurmans, and M. Norouzi. Learning to generalize from sparse and underspecified rewards. arXiv preprint arXiv:1902.0711 19219999............ 2019.
M. Andrychowicz, M. Denil, S. Gomez, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. D. Freitas. Learning learn by gradient descent by gradient descent. In Advances in neural information processing systems, pp. 3981-3989 2016.
D. Barber and F. V. Agakov. The IM algorithm: variational approach to information maximization. In Advances in neural information processing systems, 2003.
M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count- based exploration and intrinsic motivation, In Advances in Neural Information Processing Systems (NeurIPS), pp. 1471-1479, 2016.
S. Bengio, Y. Bengio, J. Cloutier, and J. Gecsei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, volume 2, 1992.
Y. Bengio, S. Bengio, and J. Cloutier. Learning a synaptic learning rule. In IJCNN-91-Sene International Joint Conference Neural Networks, volume 2, pp. 969.969, 1991.
Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint 1810. 12894, 2018
M. Chevalier-Boisvert. maxi maaximech//gmmmmiiiiiwew 2018
environment for openai gym. https://github. com/

---

Under review as a conference paper at ICLR 2021
R. Dorfman and A. Tamar. Offline meta reinforcement learning. arXiv preprint arXiv:2008.02598. 2020.
Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RL Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 1.02779, 2016.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06070 2018.
R. Fakoor, P. Chaudhari, S. Soatto, and A. Smola. Meta-Q-learning. arXiv preprint 1rX1v:1910.00125, 2019.
C. Finn, P. Abbeel, and S. Levine. Model-agnostie meta-learning for fast adaptation of deep networks In International Conference on Machine Learning (ICML). 2017.
K. Gregor, D. J. Rezende, and D. Wierstra. 1rXiv:1611.07577 2016.
Variational intrinsic control. arXiv preprint
A. Gupta, R. Mendonca, Liu, P. Abbeel, and S. Levine. Meta-reinforcement learning of structured exploration strategies. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5302- 5311, 2018.
S. Gurumurthy, S. Kumar, and K. Sycara. Mame: Model-agnostic meta-exploration. arXiv preprint arXiv:1911.04024, 2019.
Hiraoka, T. Imagawa, Tangkaratt, T. Osa, T. Onishi, and Y. Tsuruoka. Meta-model-based meta- policy optimization. arXiv preprint arXiv:2006.0206 2020.
S. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In International Conference on Artificial Newral Networks (ICANN), pp. 87-94, 2001.
R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. D. Turek, and P. Abbeel. Vime: Variational information maximizing exploration. In Advances Neural Information Processing Systems (NeurIPS), pp. 1109-1117, 2016.
R. Houthooft, Y. Chen, P. Isola, B. Stadie, F. Wolski, O. Ho, and P. Abbeel. Evolved policy gradients. In Advances in Neural Information Processing Systems (NeurIPS), pp. 5400-5409, 2018.
J. Humplik, A. Galashov, L. Hasenclever, P. A. Ortega, Y. W. Teh, and N. Heess. Meta reinforcement learning as task inference. arXiv preprint arXiv: / 1905.0645444 2019.
L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1):99-134, 1998.
Kamienny, M. Pirotta, A. Lazaric, T. Lavril, N. Usunier, and L. Denoyer. Learning adaptive exploration strategies in dynamic environments through informed policy regularization. arXiv preprint arXiv:2005.00 X.2924005002334 2020.
S. Kapturowski, G. Ostrovski, J. Quan, R. Munos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International Conference Learning Representations (ICLR), 2019.
F. Leibfried, S. Pascual-Dias and J. Grau-Moya. A unified bellman optimality principle combining reward maximization and empowerment. In Advances Neural Information Processing Systems (NeurIPS), pp. 7869-7880, 2019.
E. Z. Liu, M. Hashemi, K. Swersky, P. Ranganathan, and J. Ahn. An imitation learning approach for cache replacement. arXiv preprint arXiv:2006.1623 16239, 2020a.
E. 7. I Liu, R. Keramati, S. Seshadri, K. Guu, P. Pasupat, F. Brunskill, and P. Liing. Learning abstract models for strategic exploration and fast reward transfer. arXiv preprint urXiv:2007.05886, 2020b.
R. Mendonca, A. Gupta, R. Kralev, P. Abbeel, Levine, and C. Finn. Guided meta-policy search. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9653-9664, 2019.
10

---

Under review as a conference paper at ICLR 2021
N. Mishra, M. Rohaninejad, X. Chen, and P. Abbeel. A simple neural attentive meta-leamer. arXiv preprint arXiv:1707.0314) 70110,1,,,,,, 2017.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Ried- miller, A. K. Fidjeland, Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 5 518(7540))529-553 2015.
A. Nagabandi, Clavera, S. Liu, R. S. Fearing, Abbeel, S. Levine, and C. Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. arXiv preprint arXiv:180. / 1347, 2018.
D. K. Naik and R. J. Mammone. Meta-neural networks that learn by learning. In [Proceedding 1992] IJCNN International Joint Conference on Neural Networks, volume 1, pp. 437-442, 1992.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and Lerer. Automatic differentiation pytorch, 2017.
D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self-supervised prediction. In Computer Vision and Pattern Recognition (CVPR), pp. 16-17, 2017.
K. Rakelly, A. Zhou, D. Quillen, C. Finn, and S. Levine. Efficient aff-policy renforeenfor learning via probabilistic context variables. arXiv preprini urXiv:/903.000 2019.
Rothfuss, Lee, I. Clavera, T. Asfour, and P. Abbeel. Promp: Proximal meta-policy search. arXiv preprint arXiv: AKK1:0810006788 2018.
D. Russo, B. V. Roy, A. Kazerouni, I. Osband, and Z. Wen. A tutorial on thompson sampling. arXiv preprint arXiv: T.x.vv1177..20035 2017.
Sæmundsson, Hofmann, and M. P. Deisenroth. Meta reinforcement learning with latent variable gaussian processes. arXiv preprint arXiv: /0t3:/880307755 0,055,, 2018.
A. Santoro, S. Bartunov, M. Botvinick, D. Wierstra, and T. Lillicrap. One-shot learning with memory- augmented neural networks. arXiv preprint 1605:
1e05:66056000 1605.06065, 2016.
J. Schmidhuber. Evolutionary principles in self-referential learning, meta-meta-.. hook. PhD thesis, Technische Universität München, 1987.
learning how learn: the
B. Stadie, G. Yang, R. Houthooft, P. Chen, Y. Duan, Y. Wu, P. Abbeel, and Sutskever. The importance of sampling inmeta-reinforceemen learning. In Advances in Neural Information Processing Systems (NeurIPS), pp. 9280-9290, 2018.
W. R. Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3):285 1933.
Thrun and L. Pratt. Learning to learn. Springer Science Business Media Springer Science & Business Media, 2012.
van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of machine learning research, (0)):579-2660)) 2008.
H. van Hasselt, A. Guez, and Silver. Deep reinforcement learning with double Q-learning. In Association for the Advancement of Artificial Intelligence (AAAI), volume 16, pp. 2094-2100, 2016.
X. Wang, Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. 7. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. Learning to reinforcement learn. arXiv preprint 1rXiv:1611.05766 2016a.
Z. Wang, T. Schaul, M. Hessel, H. V. Hasselt, M. Lanctot, and N. D. Freitas. Dueling network architectures for deep reinforcement learning. In International Conference on Machine Learning (ICML), 2016b.
D. Farley T. V. de Wiele, Kulkarni, C. Ionescu, S. Hansen, and V. Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint 1. 2018.
11

---

Under review as a conference paper at ICLR 2021
Y. Yang, K. Caluwaerts. A. Iscen, J. Tan, and C. Finn. Norml: No-reward meta learning. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pp. 313-3233,,,, 2019.
T. Yu, Quillen, Z. He, R. Julian, K. Hausman, C. Finn, and S. Levine. Meta-worid: A benchmark and evaluation for multi-task and meta reinforcement learning. arXiv preprint arXiv:1910. 10897, 2019.
Zhang, Wang, H. Hu, Y. Chen, C. Fan, and C. Zhang. Learn effectively explore in context- based meta-RL, arXiv preprint 2001::2006.887000 2020.
A. Zhou, E. Jang, D. Kappler, A. Herzog, M. Khansari, P. Wohlhart, Y. Bai, M. Kalakrishnan, S. Levine, and C. Finn. Watch, try, learn: Meta-learning from demonstrations and reward. arXiv preprint arXiv: 03.5535,,, 2019a.
W. Zhou, L. Pinto, and A. Gupta. Environment probing interaction policies. arXiv preprint 2019b.
L. Zintgraf, K. Shiarlis, M. Igl, S. Schulze, Y. Gal, K. Hofmann, and S. Whiteson. Varibad: A very good method for bayes-adaptive deep RL. via meta-learning. arXiv preprint arXiv: 19 19.008348 2019.
12

---

Under review as a conference paper at ICLR 2021
A DREAM TRAINING DETAILS
L summarizes a practical algorithm for training DREAM. We parametrize both the exploration and exploitation policies recurrent deep dueling double-Q networks (Wang al. 2016b van Hasselt et al., 2016), with exploration Q-values Q (s, rep, az @) parametrized by (and target network parameters o') and exploitation Q-values Qtask (s, 2, a; 0) parametrized by 0 (and target network parameters 0¹). We train on trials with one exploration and one exploitation episode, but can test on arbitrarily many exploitation episodes, as the exploitation policy acts on each episode independently (i.e. it does not maintain hidden state across episodes). Using the choices for F6 and au in Section 4.3 training proceeds as follows.
Algorithm
We first sample new problem for the trial and roll-out the exploration policy, adding the roll-out to replay buffer (lines 7-9). Then, we roll-out the exploitation policy, adding the roll-out to a separate replay buffer (lines 10-12). We train the exploitation policy on both stochastic encodings of the problem ID (ff(()), and on encodings the exploration trajectory (w
Next, sample from the replay buffers and update the parameters. First, sample (st, at, $1+1, fe, 14,7%% -ttples from the exploration replay buffer and perform a normal DDQN update on the exploration Q-value parameters á using rewards computed from the decoder (lines 13-15). Concretely, we minimize the following standard DDQN loss function w.r.t., the parameters é, where the rewards are computed according to Equation
Lo)p(0) = E 100 T ac Φ) (r + 7 1 ii sc), aDDON; d')
where r = [1v( - gw (T: 112 11
(u) (w exp 112
and #DDON = arg max Qosp (8++1, 1 a4; se):
We perform similar update with the exploitation Q-value parameters (lines 16-17). We sample (s, s', - from the exploitation replay buffer and perform a DDQN update from the encoding the problem ID by minimizing the following loss:
(0, 1b) E ||@thh ( fo()), a; - (r Quakk for (1)), aprob: 8') [1]]
where Guraj arg max Quskk (s', Iw (T a; 0) and Oprob arg max Q Quusk (1
a
Finally, from the same exploitation replay buffer samples, also update the problem ID embedder to enforce the information bottleneck (line 18) and the decoder to approximate the true conditional distribution (line 19) by minimizing the following losses respectively:
Lboutenock (1b) E, min (I)f K)}
and Ldecoder (w) Esyep
Σ -
9m
Since the magnitude (r)) partially determines the scale of the reward, we add a hyperparameter K and only minimize the magnitude when it is larger than K. Altogether, minimize the following loss:
0,,,0,,,)) Le) + Lrask-ia (0, (0, Chonteneck (16) Ldecoder (w).
As is standard with deep Q-learning (Mnih et al. 2013), instead of sampling from the replay buffers and updating after each episode, we sample and perform all of these updates every 4 timesteps. We periodically update the target networks (lines 20-22).
B EXPERIMENT DETAILS
B. PROBLEM DETAILS
Distracting bus / map. Riding each the four colored buses teleports the agent to near one of the green goal locations in the corners. In different problems, the destinations of the colored buses
13

---

Under review as a conference paper at ICLR 2021
Algorithm DREAM DDQN
Initialize exploitation replay buffer Busk = 0 and exploration replay buffer Bex
0
2: Initialize exploitation Q-value Q!ae parameters 9 and target network parameters
3: Initialize exploration Q-value Q parameters di and target network parameters di'
4: Initialize problem ID embedder Ju parameters v and target parameters v
Initialize trajectory embedder du parameters w and larget parameters
6: for trial to max trials do
7: Sample problem ~ p(μ), defining MDP (S,A, Ru;
8: Roll-out e-greedy exploration policy Q (st, Tit "itt",a aas; 6). producing trajectory 70 = (su, av, ..88).))7)))
9: Add tuples to the exploration replay buffer Beep Bap U ((5t, ae S041, 14, 7 701))).
10:
Compute embedding ((fe(u), I). Roll-out e-greedy exploitation policy Quuh 2, ax; 0), producing trajectory (80, do, Yus with
11:
Ft (8++1) Add tuples to the exploitation replay buffer Bunk = Bua {(e,, ae, Ft, 8e+1, H4 )")
12:
13:
Sample batches of (st, 86+1, P) Bag from exploration replay buffer.
14:
Compute reward r. = (1) - 9- (T,"")) [[ go (T 11 (Equation
15:
Optimize é with DDQN update with tuple (st, as r. st+1) with Leup())
16:
Sample batches of (s, a. (4, 70)) Buasia from exploitation replay buffer. Optimize and with DDQN update with tuple ((s, p):: a, (s', ,4) with Laskaa v))
18:
Optimize vb Looukneck (1b) V min(ll (1) K)
19:
Optimize Lancoder (w) V. E. (lf 9w (T " (Equation
20:
trial = (mod target freq) then
21:
Update target parameters o'
@b 0r = 0, v/' = sb.
22:
end if
23:
end for
(a)
(b)
(123
5
24 (8, 0)
3)
(3) goal (8, 8)
goal (0,0)
goal
map
unhelpfull
Figure 8: Examples of different distracting bus and map problems. (a) An example distracting bus problem. Though all unhelpful distracting buses are drawn in the same color (gray), the destinations of the gray buses are fixed within problem. (b) Another example distracting bus problem. The destinations of the helpful colored are a different permutation (the orange and green buses 3 = l (mod 1!). instead of 1. The unhelpful gray buses are also different permutation (not shown), taking on permutation 5 _ (c) An example map problem. Touching the map reveals the destinations of the colored buses, by adding to the state observation.
swapped locations). This takes on permutation
14

---

Under review as a conference paper at ICLR 2021
change, but the bus positions and their destinations are fixed within each problem. Additionally, in the distracting bus domain, the problem ID also encodes the destinations of the gray buses, which are permutations of the four gray locations on the midpoints of the sides. More precisely, the problem ID E {0, 1,....4!! 4! x 4! = 576} encodes both the permutation of the colored helpful bus destinations, indexed as μ (mod 4!) and the permutation of the gray unhelpful bus destinations as # We hold out most of the problem IDs during meta-training 22
21 x 576 = 552 are held-out for meta-training).
In the map domain, the problem ju is an integer representing which of the 4! permutations of the four green goal locations the colored buses map to. The states include an extra dimension, which is set to 0 when the agent is not at the map, and is set to this integer value μ when the agent is the map. Figure 8 displays three such examples.
Cooking. In different problems, the (color- coded) fridges contain of 4 different ingre- dients. The ingredients in each fridge are un- known until the goes to the fridge and uses the pickup action. Figure 9 displays three example problems. The problem ID μ an integer be- tween O and 43, where μ = 4 + 46 + e indi- cates that the top right fridge has ingredient a, the middle fridge has ingredient b and the bot- tom right fridge has ingredient c.
(a)
(b)
1 4
3
0
3 x 4
apent
fridge (ingradient))
pot
Figure 9: Three example cooking problems. The con- tents of the fridges (color-coded) are different in differ- ent problems.
The goals correspond to recipe of placing the two correct ingredients in the pot in the right order. Goals are tuples (a, b), which indicate placing ingredient a in the pot first, followed by ingredient b. In a given problem, we only sample goals involving the recipes actually present in that problem. During meta-training, we hold out a randomly selected problem 1 = 11.
We use the following reward function Ry,. The agent receives per timestep penalty of -0.1 reward and receives +0.25 reward for completing each of the four steps: (i) picking up the first ingredient specified by the goal; (ii) placing the first ingredient in the pot; (iii) picking up the second ingredient specified by the goal; and (iv) placing the second ingredient in the pot. To prevent the agent from gaming the reward function, e.g., by repeatedly picking up the first ingredient, dropping the first ingredient anywhere but in the pot yields a penalty -0.25 reward, and similarly for all steps. To encourage efficient exploration, the agent also receives penalty of -0.25 reward for picking up the wrong ingredient.
Cooking without goals. While we evaluate on goal-conditioned benchmarks to deepen the explo- ration challenge, forcing the agent to discover all the relevant information for any potential goal, many standard benchmarks (Finn et al.) 20177: Yu et al.] (219)) don't involve goals. We therefore in- clude variant of the cooking task, where there are no goals. We simply concatenate the goal (recipe) to the problem ID μ.. Additionally, we modify the rewards so that picking up the second ingredient yields +0.25 and dropping it yields -..25 reward, that it is possible to infer the recipe from the rewards. Finally, to make the problem harder, the agent cannot pick up new ingredients unless its inventory is empty (by using the drop action), and also increase the number of ingredients The results are in Section B.2
Sparse-reward 3D visual navigation. We implement this domain in Gym MiniWorld (Chevalier Boisvert 2018), where the agent's observations 80 x 60 3 RGB arrays. There are three problems = (the sign says "blue"), f = (the sign says "red"), and f = 2 (the sign says "green"). There goals, represented as 0 and 1, corresponding to picking up the key and the box, respectively. The reward function (,, i) is +1 for picking up the correct colored object (according to μ)) and the correct type of object (according the goal) and -1 for picking up an object of the incorrect color or type. Otherwise, the reward is O. On each episode, the agent begins at random location on the other side of the barrier from the sign.
B.2 ADDITIONAL RESULTS
15

---

Under review as a conference paper at ICLR 2021
information bottleneck)
agent
unhelpful bus stop
position after ride bus attion
bus
potential gool
ride action
end caisode action
Figure 10: Examples of DREAM'S learned exploration behavior. (a) DREAM learns the optimal exploration behavior on the distraction variant: riding 3 of 4 helpful colored buses, which allows to infer destinations all colored buses and efficiently reach any goal during exploitation episodes. (a) Without the information bottleneck, DREAM also explores the unhelpful gray buses, since they are part of the problem. This wastes exploration steps, and leads to lower returns during exploitation episodes. (c) DREAM Icarns the optimal exploration the map variant: goes read the map revealing all the buses' destinations, and then ends the episode, though unnecessarily rides one buses.
Analysis of the learned policies. Please see github. io/ii//ree github.ic/dream. wtt/deaa//////oo/ee for videos and analysis of the exploration and exploitation behavior learned by DREAM and other approaches, which is described in text below.
Distracting bus map. Figure 10 shows the exploration policy DREAM learns on the distracting bus and map domains. With the information bottleneck, DREAM optimally explores by riding 3 of the colored buses and inferring the destination of the last colored bus (Figure Ba). Without the information bottleneck, DREAM explores the unhelpful gray buses and out of time explore all of the colored buses, leading to lower reward (Figure (b). In the map domain, DREAM optimally explores by visiting the map and terminating the exploration episode. In contrast, the other methods (RL IMPORT, VARIBAD) rarely visit the colored buses map during exploration and consequently walk to their destination during exploitation, which requires more timesteps and therefore receives lower returns.
we additionally visualize the exploration trajectory encodings 9w (T*P) and problem ID encodings fo (u) that DREAM learns in the distracting bus domain by applying t-SNE (van der Maaten & Hinton 2008). We visualize the encodings of all possible problem IDs dots. They naturally cluster into
In Figure 11
24 clusters, where the problems within each cluster differ only in destinations of the gray distracting buses, and not the colored buses. Problems in the support of the true posterior p( T**) are drawn in green, while problems outside the support (c.g., a problem that specifies that riding the green bus goes to location (0, 1) when it has already been observed in exp that riding the orange bus goes to location (0, 1)) are drawn in red. We also plot the encoding of the exploration trajectory zap so far as blue cross and the mean of the green clusters We find that the encoding of the exploration trajectory (T<P)) tracks the mean of the green clusters until the end of the exploration episode, when only one cluster remains, and the destinations all the colored buses has been discovered. Intuitively, this captures uncertainty in what the potential problem ID may be. More precisely, when the decoder is Gaussian, placing 9u (a=((" (xex) at the center of the encodings of problems in the support exactly minimizes Equation
black square.
16

---

Under review as a conference paper at ICLR 2021
ride blue bus
ride orange bus
ride green bus
Problem encoding (p) in the support of p(u T(m))
Trajectory encoding g. (7
Problem encoding fo(1) outside the support of p( T(P))
Mean the green clusters
Figure 11: DREAM's learned encodings of the exploration trajectory and problems visualized with t-SNE (van der
Maaten & Hinton 2008).
Cooking Goals)
5
3
14
2
agent
fridge (ingredients)
Timestops (1c3)
pot
pickup action
DREAM
VARIAA
Optimal
E.R13
PEABL-UB
No exploration
IMPORT
Figure DREAM learns optimal explo- ration policy, which learns the fridges' con- tents by going to each fridge and using the pickup action.
Figure 13: Cooking without goals results. Only
DREAM learns the optimal policy, achieving ~2x reward than the best approach.
Robustness to imperfections in the problem ID. Recall that the problem ID is a simple and easy- to-provide unique one-hot for each problem. We test of DREAM robust to imperfections in the problem ID by assigning each problem in the map benchmark 3 different problem IDs. When problem is sampled during meta-training, is randomly labeled with I of these 3 different problem IDs. We find that this imperfection in the problem ID does not impact DREAM's 's final performance at all: still achieves optimal exploitation returns 0.8 after 1M time steps of training.
Cooking. Figure 12 shows the exploration policy DREAM learns on the cooking domain, which visits each of the irigges and investigates the contents with the "pickup" action. In contrast, the other methods rarely visit the fridges during exploration, and instead determine the locations of the ingredients during exploitation, which requires more timesteps and therefore receives lower returns.
Cooking without goals. We provide additional results in the case where the cooking domain is modified to not include goals (see Section B.T. The results are summarized in Figure 1 and show the same trends as the results in original cooking benchmark. DREAM learns to optimally explore by investigating the fridges, and then also optimally exploits, by directly collecting the relevant ingredients. The next best approach E-RL?, only sometimes explores the fridges, again getting stuck local optimum, yielding only slightly higher reward than no exploration at all.
Sparse-reward 3D visual navigation. DREAM optimally explores by walking around the barrier and reading the sign. See https:: :etttss:/aanoiii anonymousppper submis ioooisssiin sson. gi github. io/ dream/ for videos. The other methods do not read the sign all and therefore cannot solve the problem.
17

---

Under review as a conference paper at ICLR 2021
B.3 OTHER APPROACHES AND ARCHITECTURE DETAILS
In this section, we detail the loss functions that E-RL², IMPORT, and VARIBAD optimize, as well as the model architectures used in our experiments. Where possible, we use the same model architecture for all methods: DREAM, E-RL?,, IMPORT, and VARIBAD. All approaches are implemented in PyTorch (Paszke et al. 2017), using DQN implementation adapted from Liu et al. (2020b) and code adapted from Liu et al. (202aa).
State and problem ID embeddings. All approaches use the same method to embed the state and problem ID. For these embeddings, we embed each dimension independently with an embedding matrix of output dimension 32. Then, concatenate the per-dimension embeddings and apply two linear layers with output dimensions 256 and 64 respectively, with ReLU activations.
In the 3D visual navigation task, use different embedding scheme for the states, they are images. We apply 3 CNN layers, each with 32 output layers and stride length 2, and with kernel sizes of 5, and 4 respectively. We apply ReLU activations between the CNN layers and apply a final linear layer to the flattened output of the CNN layers, with an output dimension of 128.
All state and problem ID embeddings below use this scheme.
Experience embeddings. E-RL??, IMPORT, VARIBAD and the exploration policy in DREAM also learn an embedding of the history of prior experiences TTP (so, do, ro, .1....) and current state ST. To do this, first separately embed each (st+1, az Tr, de) )ttule, where de is an episode termination flag (true if the episode ends on this experience, and false otherwise), as follows:
Embed the state as e(st), using the state embedding scheme described above.
Embed the action at as e(az) with an embedding matrix of output dimension 16. We set a_ to be O.
For the standard meta-RL setting and during exploitation episodes, embed the rewards with a linear layer of output dimension 16. With reward-free exploration in IMRL, the rewards are not embedded in the exploration policy of DREAM, and are embedded O reward for the other approaches, since the same policy is used during both exploration and exploitation. We set 1 to be O.
Embed the episode termination d, as e(de) with embedding matrix of output dimension 16. Note that is true during all episode terminations within a trial for RL?, IMPORT, and VARIBAD.
Then, we apply a final linear layer with output dimension 64 to the concatenation of the above (e(s:); e(ax); experience separately as above, and then pass an LSTM with hidden dimension 64 over the experience embeddings, where the initial hidden and cell states are set to be O-vectors.
e(r:); e(x)) Finally, obtain embedding of the entire history Tes, we embed each
DREAM. For the decoder (r = (sp- ag ro, -1...., (1))),,)))),, we embed each transition (St, az, Tt, 8141) of the exploration trajectory x using the embedding scheme as above, ex- cept we also embed the next state $:41. We do not embed the rewards in the reward-free adaptation version of IMRL. Then, given embeddings for each transition, we embed the entire trajectory by passing an LSTM with output dimension 128 on top of the transition embeddings, followed by two linear layers of output dimension 128 and 64 with ReLU activations.
For the exploitation policy Q-values Qjak (a 8, 2). we either choose 2 to be the decoder embedding of the exploration trajectory gu (xxP)) or to be an embedding of the problem ID co where we always use the exploration trajectory embedding 9w (7*3)) at meta-test time. We embed the state with a learned embedding functions e(s). Then apply linear layer of output dimension 64 to the concatenation of (e(s); 2] with a ReLU activation. Finally, we apply two linear layer heads of output dimension and |A| respectively dueling Q-network parametrization. To obtain Q-values we add the value function to the advantage function, subtracting the mean of the advantages.
form estimates of the value and advantage functions, using the
18

---

Under review as a conference paper at ICLR 2021
For the exploration policy Q-values 2 (ag St, " we embed the 8 and T. according to the embedding scheme above. Then, we apply two linear layer heads to obtain value and advantage estimates as above.
E-RL². E-RL² learns a policy π (a: st, T&a) producing actions a, given the state 8; and history T. Like with all approaches, we parametrize this with dueling double Q-networks, learning Q-values (+::: Tu, at). We embed the current state 8, and history Tr using the embedding scheme described above (with episode termination embeddings). Then, we apply two final linear layer heads to obtain value and advantage estimates.
IMPORT IMPORT also learns recurrent policy (a, | st; z), but conditions on the embedding which either an embedding of the problem ja or the history Tu-- We also parametrize this policy with dueling double Q-networks, learning Q-values Q(s,, 2, at). We embed the state 8; as e(sr), the problem μ as (p(μ)) and the history Tиt Ca(T:) using the previously described embedding schemes. Then we alternate meta- eetanntaining trials between choosing = cΦ(μ)) and 2 = (Tx)) We apply a linear layer of output dimension 64 to the concatenation [e(st); 2] with ReLU activations and then apply two linear layer heads to obtain value and advantage estimates.
Additionally, IMPORT uses the following auxiliary loss function to encourage the history embedding ea(T,, (T,²) to be close to the problem embedding es (1) (optimized only w.r.t., 0):
LIMPORT (0) (T
Σ (Ie)(Ta) =
where r is a trajectory from rolling out the policy on problem μu. Following Kamienny et al. (2020). we use = 1 in our final experiments, and found that performance changed very little when experimented with other values of B.
VARIBAD. VARIBAD also learns recurrent policy <aa 2). but over belief state capturing the history Tu and current state st. We also parametrize this dueling double Q-networks, learning Q-values Q(,,, 2, a.).
VARIBAD encodes the belief state with an encoder enc(2 7: t). Our implementation of this encoder embeds 8, and Ty using the same experience embedding approach as above, and use the output the mean m for distribution. Then, set enc(= 8t, T: t) _(m,, u² I), where -²= 0,00001. We also tried learning the variance instead of fixing it to 2²1 by applying a linear head the output of the experience embeddings, but found change in performance, so stuck with the simpler fixed variance approach. Finally, after sampling from the encoder, we also embed the current state 8t as e(st) and apply linear layer of output dimension 64 to the concatenation [e(st); 2]. Then, we apply two linear layer heads to obtain value and advantage estimates.
VARIBAD does not update the encoder via gradients through the policy. Instead, VARIBAD jointly trains the encoder with state decoder T(s' 8, 2) and reward decoder R(s' a, 8, z). where is sampled from the encoder, as follows. Both decoders embed the action as e(a) with an embedding matrix of output dimension 32 and embed the state & as e(s). Then we apply two linear layers with output dimension 128 to the concatenation ee(s);e(a) 2]. Finally, apply two linear heads, one for the state decoder and one for the reward decoder and take the mean- squan-s error with the true next state s' and the true rewards respectively. In the 3D visual navigation domain, we remove the state decoder, because the state is too high-dimensional to predict. Note that Zintgraf et al. (2019) found better results when removing the state decoder in all experiments. We also tried to remove the state decoder in the grid world experiments, but found better performance when keeping the state decoder. We also found that VARIBAD performed better without the KL loss term, so we excluded that for our final experiments.
B.4 HYPERPARAMETERS
In this section, we detail the hyperparameters used in our experiments. Where possible, we used the default DQN hyperparameter values from MMih et al. (2015). and shared the same hyperparameter values across all methods for faimess. We optimize all methods with the Adam optimizer (?). T the shared hyperparameters used by all methods and we detail any differences in hyperparameters between the methods below.
19

---

Under review as a conference paper at ICLR 2021
Hyperparameter
Value
Discount Factor
0.99
Test-time
0
Learning Rate
0.0001
Replay buffer batch size
32
Target parameters syncing frequency
5000 updates steps
Update frequency
Grad norm clipping
10
Table 1: Hyperparameters shared across all methods: DREAM, RL3, IMPORT, and VARIBAD.
All methods use a linear decaying schedule for e-greedy exploration. For RL?, IMPORT, and VARIBAD, we decay e from 1 0.01 over 500000 steps. For DREAM, we split the decaying between the exploration and exploitation policies. We decay each policy's s from 1 to 0.01 over 250000 steps.
We train the recurrent policies (DREAM's ' exploration policy, RL. IMPORT, and VARIBAD) with simplified version of the methods in Kapturowski et al. (2019) by storing replay buffer with to 16000 sequences of 50 consecutive timesteps. We decrease the maximum size from 16000 10000 for the 3D visual navigation experiments in order to fit inside a single NVIDIA GeForce RTX 2080 GPU, For DREAM'S 's exploitation policy, the replay buffer stores up to 100K experiences (60K for 3D visual navigation).
For DREAM, we additionally use per timestep exploration reward penalty c = 0.01, decoder and stochastic encoder variance p² = 0.1, and information bottleneck weight À 1. Note that this information bottleneck weight À could be adapted via dual gradient descent to solve the constrained optimization problem in Appendix but find that dynamically adjusting is not necessary for good performance. For the MiniWorld experiments, we use
C ANALYSIS
C.I CONSISTENCY
We restate the consistency result of DREAM (Section 5.1) and prove it below.
Proposition 1. Assume (S,A, Ry, Tn) ergodic for all problems μ M. Let V* be the maximum expected returns achievable by any exploitation policy with access the problem ID iee, with complete information. Let TT ne F. and q, (z (TT) ) be the optimizers of the DREAM objective. Then, if the function classes DREAM optimizes over well-specified, there exists finite such that if the length of the exploration episode at least T,
E Eyropla), tommeaa sara (k/p2w)
V=* (2: = Ep-p(u) [V*(
Proof. Recall that musk and (z (4) are optimized with an information bottleneck according to Equation in order to solve the constrained optimization problem in Appendix E Note that if tusk is optimized over an expressive enough function class and À approaches O. which is necessary to solve the constrained optimization problem associated with the information bottleneck (Appendix then task achieves the desired expected returns conditioned on the stochastic encoding of the problem F. 11) (i.e., has complete information):
Eu [ (2: (z: Eupp(u) [V*(u)),
where V= (25 (1) is the expected returns of taakk on problem y given embedding 2. Therefore, suffices to show that the distribution over z from the decoder 4=(2 xxP) is equal to the distribution from the encoder (2(2) μ1) for all exploration trajectories the support of ox (7 problem μ. Then,
for each
Ep--plu) rewm t z0g (#/700) [v=t" (z:n) Eunpl())s
[v (2:1))
Earp(p) [V)(
We slightly abuse notation to use () [p denote the distribution of exploration trajectories 7PP from rolling out "h problem μ.
20

---

Under review as a conference paper at ICLR 2021
as desired. We show that this occurs follows.
Given stochastic encoder (2 p)) exploration policy π. maximizes 1()=P;)) = H()) - H(z
TES) (Equation 3) by assumption. Since only [ TPP)) depends on esp the exploration policy outputs trajectories that minimize
H(z PP)) = Ex-p((x) B -
(Z log |
E pwmp()) [ (1 (F (1) 11) ->])]]]]
where p(= 7P)) is the true conditional distribution and H(F. (2 1)) T))) is the cross- entropy. The cross-entropy is minimized when p(z TT))) F. p), which be achieved with long enough exploration trajectories T' (S, A, Rus Tn)) is ergodic (by visiting each transition sufficiently many times). Optimized over expressive enough function class, 9-(2 70P) equals the true conditional distribution p(> TTP)) at the optimum of Equation which equals F, 14) as desired.
C.2 TABULAR EXAMPLE
We first formally detail a more general form of the simple tabular example Section 5.2 where episodes are horizon H rather than 1-step bandit problems. Then we prove sample complexity bounds for RL2 and DREAM, with e-greedy tabular Q-learning with e = 1, ie., uniform random exploration.
Setting. We construct this horizon H setting so that taking a sequence of actions a, (instead of single action as before) in the exploration episode leads to trajectory T. that reveals the problem μu; all other action sequences a lead to a trajectory -h that reveals no information. Similarly, the problem μ identifies unique sequence of H actions ay that receives reward during exploitation, while all other action sequences receive reward 0. Again, taking the action sequence a, during exploration is therefore necessary and sufficient to obtain optimal reward 1 during exploitation.
We formally construct this setting by considering family of episodic MDPs (S, A, Ry, Tp) parametrized by the problem ID μ M, where:
Each exploitation and exploration episode is horizon H.
The action space A consists of A discrete actions {1.2,....44 A}.
The space of problems M = (1.
and the distribution p(u) is uniform.
To reveal the problem via the optimal action sequence a, and to allow ag to uniquely receive reward, construct the state space and deterministic dynamics follows.
States E S are (If + 2)-dimensional and the deterministic dynamics are constructed so the first index represents the current timestep t, the middle H indices represent the history of actions taken, and the last index reveals the problem ID if a, is taken. The initial state is the zero vector so 0 and we denote the state at the 1-th timestep 8t as 0,,0)
(l, ag; "1,..-,,--11....
The optimal exploration action sequence a, is set to be taking action for timesteps. In problem μ taking action 0æ-1 1 at state 8 (H 1, ag 1,..., aH-2 1,0,0) (i.e., taking the entire action sequence a.) transitions to the state sH = (H, ag = 1, 1,...,ay -2 1, = 1, revealing the problem (s.
The action sequence a identified by the problem μ is set as the problem μ base [A]: i.e., ap is a sequence of H actions (0a, a1.-.., a1.-..,ay-1 a with Σ at|A| = /4. In prob- lem f with a, = (ao, dj..... aH-1), taking action ax-1 timestep H/ at state (H - -1, ag 01, -H2 8H-1 ( -1 s and actions a.
2, O, 0) (i.c., taking the entire action sequence a,,) yields
Reward everywhere else: i.e., (s. = for all other states
With these dynamics, the exploration trajectory T (so, ap, Pu. ..,.... 8H) is uniquely identified by the action sequence a and the problem j if revealed in sn- We therefore write eap (a) for when a, reveals the problem μı, and oxp (a, 0), otherwise.
21

---

Under review as a conference paper at ICLR 2021
Uniform random exploration. In this general setting, we analyze the number of samples required to learn the optimal exploration policy with RL2 and DREAM via é-greedy tabular Q-learning. We formally analyze the simpler case where 1, i... uniform random exploration. but empirically find that DREAM only learns faster with smaller e, and RI.² only learns slower.
In this particular tabular example with deterministic dynamics that encode the entire action history and rewards, learning a per timestep Q-value is equivalent to learning a Q-value for the entire trajectory. Hence, we denote exploration Q-values (ee((a) estimating the returns from taking the entire sequence of H actions a at the initial state and exeuction Q-values (r a) estimating the returns from taking the entire sequence of H actions a at the initial state so given the exploration trajectory TXP, We drop 8o from notation, since it is fixed.
Recall that RL learns exploration Q-values QUev by regressing toward the exploitation Q-values Quick We estimate the exploitation Q-values Quisk(7 a) the sample mean of returns from taking actions a given the exploration trajectory 7TP and estimate the exploration Q-values QeSs (a) as the sample mean of the targets. More precisely, for action sequences # a.. the resulting exploration trajectory exp is deterministically (a, 0), so we set (a) = Prank (Ta maxa Q (Ta exp a'). _exp For a., the resulting exploration trajectory Ta, may be any of (a., p)) for μ M, so we set Q as the empirical mean of (rask (7a.)) of observed Ta,
Recall that DREAM learns exploration Q-values Queup by regressing toward the learned decoder log q(H= Ta P). ). We estimate the decoder q( | Ta=)) as the empirical counts of (j, =axP) divided by the empirical counts of 2 and similarly estimate the Q-values as the empirical mean of log q(μ ra P) We denote the exploration Q-values learned after timesteps as Q and similarly denote the estimated decoder after i timesteps qt-
Given this setup, we are ready to state the formal sample complexity results. Intuitively, learning the exploitation Q-values for RL2 is slow, because, in problem ju, it involves observing the optimal exploration trajectory from taking actions a, and then observing the corresponding exploitation actions ay. which only jointly happens roughly once per |4|a samples. Since RL regresses the exploration Q-values toward the exploitation Q-values, the exploration Q-values are also slow to learn. In contrast, learning the decoder q(v | TP much faster, it independent of the exploitation actions, and in particular, already learns the correct value from single sample of a.. We formalize this intuition in the following proposition, which shows that DREAM learns in a factor of at least | fewer samples than RI.2.
Proposition 2. Lei T be the number of samples from uniform random exploration such that the greedy- policy is guaranteed to be optimal (i.e., arg max, ②. (a) = for all t 2 T. If Qee is learned with DREAM, the expected value Tf is O(IA) log [A)"). ). If Q is learned with RI.2, the expected value Tf is 22(1A|2 | (A
Proof. For DREAM, Q
(a.) (
> 27 (a) for all a + if log ( | (a., > log âr( I (a, 0))
for all μ and a # a. For all > T. Q,Thh is guaranteed to be optimal, since no sequence of samples will cause log (e( (a.,
0 < log (e (a, 0)) for any * a. This occurs once we 'e observed (H, (a, 0)) for two distinct μu for each a o a, and we've ve observed (J4, (a). for least one μ. We can compute an upperbound on the expected number of samples required to observe (n, T for two distinct j for each action sequence a by casting this as a coupon collector problem, where each pair (n, T is a coupon. There are 2 2| total coupons to collect. This yields that the expected number of samples is O(IA)" " log (A! ).
For 2L2, the exploration policy is optimal for all timesteps 2 T for some T only if the exploitation values ( (zepp (a). 1 for all μ M. Otherwise, there is small, but non-zero probability that (rusk (a, 0)) will be greater at some t > T. For the exploitation values to be optimal at all optimal exploration trajectories 177a = (a., 11) = for all E M, we must jointly observe exploration trajectory rP = (a../) and corresponding action sequence a, for each problem a E M. We can lower bound the expected number of samples required to observe this by casting this coupon collector problem, where each pair (r (a., 11), a,) is a coupon. There
22

---

Under review as a conference paper at ICLR 2021
[M] |AI unique coupons collect and collecting any coupon only with probability |
in each episode. This yields that the expected number of samples is S?(1A)2 [M] log LA A 0
D EXTENDED RELATED WORK
Detailed comparison with IMPORT and Humplik et al. (2019) We compare DREAM with IM- PORT (Kamienny et al., 2020) and and Humplik et al. (2019) greater detail, since they also leverage the problem ID. The key difference between DREAM and these works that these works still learn exploration from returns achieved during exploitation, which leads to the chicken-and
problem. In contrast, DREAM learns exploration by maximizing the mutual information between exploration trajectories and learned 2's containing only besk-releantt information (Section 4.2
More specifically, like DREAM, IMPORT also conditions on the problem ID to learn exploitation with- out the need for exploration, which can accelerate learning exploration, but unlike DREAM, IMPORT still learns exploration from end-to-end signal, which can make learning exploration challenging as follows. Suppose that optimal exploitation conditioned the problem II) is already learned, and exploration is learned from the exploitation returns. Since initially, the exploitation has not learned to interpret the trajectories produced by the exploration policies, may still achieve low returns given good exploration trajectory, which would erroneously "down weight" the good exploration trajectory (i.e., the fericken-and-eeee problem). [Humplik et (2.19) attempts learn better representations by predicting the problem ID as an auxiliary task. This can help learn a better policy, but does not address the chicken-and-ed coupling problem.
E INFORMATION BOTTLENECK
The objective (Equation 2 for learning the exploitation policy and encoder derived from minimizing the mutual information between problem IDs and the encoder's stochastic outputs, under the constraint that the exploitation policy achieves maximal returns, i.c.:
minimize [(2;p)
subject to Ep--p()),s=P(eee)) [V []] is maximal.
We form the Lagrangian with A as the dual variable, which yields Equation 2
This constrained optimization problem is only satisfied with the Langrangian maximized, the dual variable tends to infinity: i.e., À approaches O.
23


Reviews:
[{"review_id": "mJlMuaXfxux", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces an approach to improve meta-learning in RL. Specifically, the approach aims to improve the agent’s exploration during the training phase, so that the agent can better exploit during the test phase where samples collected from the training phase provides useful task-relevant information to the agent.\n\nPros:\nThe paper is well-written.\n\nThe idea is clearly presented.\n\nThe experiments are clear to understand.\n\nThe results presented demonstrate that the approach can either match or improve learning when compared to relevant baselines.\n\nCons: \nDetails about the architecture choices used for the baselines are not provided in the main text, which makes it difficult to understand the experiment setup and its results.\n\nDrawing out the differences between the baselines and their approach would be useful to help understand the contribution of the work.\n\nQuestions:\n1. It seems like the use of problem ID is specific to the approach introduced in the paper. If so, could the authors describe how this was used in the baseline agents? If the problem IDs were not used in the baselines, then it feels like the baselines considered here are unfair.\n2. For the domains considered here, it seems like the problem IDs are sufficient enough to provide any task-relevant information to the meta-learning agent. In this case, what sort of z’s can the encoder produce? The domains considered seem less interesting for the introduced approach, as it seems like the encoder could simply learn an identity mapping of the problem ID (which is its input).\n3. What would be the reason for RL^2 to fail in the 3D navigation task considered?\n4. As a baseline, it would be interesting to see the learning performance of a simple RL^2 agent on the domains, provided they take in the problem ID as input? This would inform whether the encoder-decoder architecture that is introduced in the paper learns something that is beyond the problem ID.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XTz5T9hX9V5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper introduces DREAM, a meta-RL approach that decouples exploration from exploitation. An exploitation policy learns to maximize rewards that are conditioned on an encoder that learns task relevant information. Then an exploration policy learns to collect data that maximizes the mutual information between the encoder and explored states. The work is compared against multiple baselines in simple tasks.\n\n\nOverall, I lean towards accepting the paper, though I am not as familiar with the meta-RL literature to have much of an informed opinion about what relevant benchmarks or approaches are. The paper was well-written and well-motivated, and while the experiments were simple, seemed to highlight the problems that the paper was addressing. It makes sense to separate out exploration and exploitation and I appreciated the inclusion of tasks that helped motivate this point. Furthermore, the paper provides a theoretical analysis of DREAM showing that the decoupled policy maximizes returns. Code and hyperparameters are provided and the paper seems to be reproducible. \n\n\nI do think that the paper should have more discussion and evaluation over approaches that aim to explicitly address the exploration exploitation problem. The paper only considers exploration in the context of meta-learning but of course exploration is a central problem in RL and several approaches have studied it outside of Meta-RL. The paper would be improved by discussing such approaches (for example intrinsic rewards such as empowerment [1] or surprise [2]) and/or evaluating how well these approaches compare to DREAM when trained alone and combined with vanilla algorithms. \n\n\nI also would have liked to see more empirical analysis over the exploration policy being learned by $\\pi^{exp}$. \n\n\nQuestions:\n\n\n1) How was the decay rate for epsilon chosen in Figure 3? How would a policy with a fixed decay rate perform? \n\n\n2) I do not quite understand how trajectories from the exploration policy can be used interchangeably with the encodings $z$ when plugged into $\\pi^{task}$. Could the authors provide more insights into this?\n\n\n[1] A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment. Leibfried et al. \n\n\n[2] Curiosity-driven Exploration by Self-supervised Prediction. Pathak et al.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "469EI6Rp6cN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the authors claim\\\nThe authors claim end-to-end learning of exploration and exploitation policies can result in a poor local minima. To avoid this problem, the authors propose to separate the objectives for exploration and exploitation in meta-RL. Theoretical analyses and the experiments on several artificial tasks show the effectiveness of the proposed method and superiority over existing methods.\n\nOverview of the proposed method\\\nAs is common in the literature, the exploitation policy depends not only on observations, but also on a task-embedding $z$ to adapt the environment across the tasks. $z$ is generated from a stochastic encoder which is conditioned on the problem ID $\\mu$. By minimizing the mutual information between the problem ID $\\mu$ and the task-embedding $z$ which is called the information bottleneck term, the exploitation policy tends to be independent from the task irrelevant information and can reduce the risk of being trapped at a poor local minima. On one hand, the lower bound of the mutual information between the exploration experiences $\\tau^{exp}$ and the task-embedding $z$ is maximized for the learning of the exploitation policy. During this optimization, decoder q(z|\\tau^{exp}) is trained and this decoder is used to estimate the task-embedding without knowing the problem ID. Because the exploitation policy depends on the task embedding that is generated from problem ID and not on the exploration experiences $\\tau^{exp}$, it make the training of the exploitation policy more robust and  successful.\n\nComments on the paper\\\nI appreciate the good experimental results.\\\nBut I think the paper contains inaccurate explanations and the proposed method is not properly characterized.\n- Although the authors emphasize that the decoupling of the exploration and exploitation is important in Section 1, Section 4.1 and also in the title, the actual proposed algorithm has coupled loss functions.\nIn Section 4.1, the authors wrote “To solve the task, $\\pi^{task}$ needs good exploration data from a good exploration policy $\\pi^{exp}$.” Rigorously speaking, this is not true. Even if the exploration policy is just uniform random distribution, we know table Q-learning algorithms can solve the task when we have an infinite number of samples. If the statement means “To **efficiently** solve the task, $\\pi^{task}$ needs good exploration data from a good exploration policy $\\pi^{exp}$.” It will be true, but this is also true for the proposed algorithm when $z$ is computed by not only using the encoder $F(z|\\mu)$, but also by using $g(\\tau^{exp})$. This makes the training of $\\pi^{task}$ more coupled training.  I wonder how much this mixed training heuristics is important. It would be clear if the authors provide the experimental results with and without this heuristics.\nAnother statement, “Learning $\\pi^{exp}$ relies on gradients passed through $\\pi^{task}$.” is true. If the embedding $z$ is not informative for solving the task, that is, $\\pi^{task}$ does not make use of $z$, it is not meaningful to maximize the (lower bound of) mutual information between $z$ and the experience by the exploration policy $\\pi^{exp}$, $\\tau^{exp}$.  Thus it is also true for the proposed method. Therefore, it seems for me the statement not only explains the reason why the existing methods sometimes does not work, but also explain the reason why the reinforcement learning is difficult. I think the stress should be put on the training by using the problem ID which does not depend on the progress of the training rather than the decoupling.\n- The condition of Proposition 1 is not clearly stated. In the proof, the authors assume “$\\lambda$ approaches 0”, but this is not clearly stated in the condition. Also in the experiment, this is not performed, but $\\lambda$ is set to be 1.\n- Usefulness of the bottleneck (as shown in Fig.5) is not supported by the theoretical analyses. It seems for me theoretical analysis does not support the actual proposed method which uses the bottleneck in the optimization. Instead, as I wrote above, the authors emphasize the importance of the decoupling too much. However, based on the theoretical analysis and the experimental results, I think the authors should emphasize more the usefulness of the bottleneck.   \n- To use the proposed algorithm, Problem ID is indispensable. This condition should be stressed. I also wonder how much the performance of the proposed method is affected by the mapping of the problem ID. Is it robust to the random permutation of the problem ID? It would be nice if the authors can test the robustness on the permutation of the problem ID. \n- As written in the introduction, the meta-reinforcement learning is often inspired by the humans' quick adaptation ability to new task which shares some properties with the previously experienced tasks. However, the necessity of the problem ID limits the applicability of the method to such situations. It would be nice if the authors can exemplify what kind of practical tasks the proposed method can be applied.\n- As for the experiments, it is good to test on newly designed environments and tasks to highlight the properties of the proposed method. However, I believe the proposed method should be tested on one of the common benchmarks to make the experimental results more reliable. From these experiments, the readers will understand that the existing methods are implemented appropriately and how much the proposed method works well on the common benchmark.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ViZxEWPsw9", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper investigates the exploration-exploitation problem in meta-learning. The authors explain the problem of coupled exploration and validate it through a toy example. To overcome this issue, the paper introduces DREAM, a meta-algorithm decoupling exploration and exploitation. In the first step, DREAM learns an exploitation policy and a task embedding by maximizing the cumulative reward of the given task (task identifier is known at train). In the second step, DREAM learns an exploration policy that is \"compatible\" with the embeddings generated by the exploitation policy. DREAM outperformed the state-of-the-art algorithms in several experiments.\n\nThe paper is well written and easy to follow. The idea is clearly explained and justified. I have only a few comments.\n\nConcerning the problem of coupled exploration, I'm wondering if you could provide a more formal justification. The current justification (sec 4.1) is easy to follow but quite abstract. Despite being supported by your example (Sec 5.2), it is not clear to me whether this is a general problem of coupled exploration. Eg, is it due to gradient updates?\n\nComparison with the literature: could you provide a detailed comparison with (Humplik et al., 2019; Kamienny et al., 2020)? \nIn the first step, your algorithm learns an encoding of the task $f_\\psi(\\mu)$ by maximizing the task reward (ie learning the exploitation policy). In the second step, it uses this embedding to train an exploration policy to generate trajectories mapping to the relevant information constructed by the embedding (ie maximizing the mutual information between $z$ and the trajectories). To the best of my understanding, this seems very similar to what done in the mentioned approaches. In particular, (Kamienny et al., 2020) also have exploration and exploitation policies. The exploitation policy is trained to maximize the reward while generating an embedding of $\\mu$, thus similarly to your exploitation step. The exploration policy is also trained to maximize the reward while enforcing the RNN state to be similar to $f(\\mu)$. More mildly, even their approach aims to learn to generate trajectories providing information about $z$. The main difference between the two approaches resides in the fact that their exploration policy is trained to also maximize the reward, am I correct? Will their algorithms suffer from the coupling problem mentioned in section 4.1?\n\n\nMinor comments\nProposition 1: could you explain in more details the need for ergodicity? My understanding from the appendix is that you should be able to generate all possible trajectories to have that $p(z|\\tau^{exp}) = F_\\star(z|\\mu)$. However, you would need to potentially enforce exploration at the level of actions. This can be obtained by a random policy, am I correct?\n\nI think $\\mathbb{E}_{\\mu \\sim p(\\mu)}$ is missing in the first and second equations of the proof of Proposition 1.\n\nYou mentioned that you could remove the \"ergodicity assumption by increasing the number of exploration episodes\". Could you clarify this sentence? The number of exploration episodes is a term that does not appear in your current analysis since the reasoning seems to be \"in expectation\".\n\nI didn't check the Appendix C.2\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors clarified many of R1 and R4's concerns, but there were important remaining concerns regarding the presentation.

On the bright side, the approach is novel and the experimental results are solid. 

However, the main point raised by R1 is the mismatch between the narrative and the theory and the actual algorithm and results. Some exemples of this mismatch include:
- Proposition 1 is proved when lambda goes to 0, which is never mentioned in the main paper. One has to look into the appendices to have a discussion of lambda and of the algorithm, while the authors could clearly explain that when discussing the theoretical result. The added discussion on tuning of lambda based on Appendix E does not help because the optimization problem is written as "under the constraint that [...] is maximized", which is not particularly clear. 
- More generally, the theoretical result (including e.g., the assumption of ergodicity) could be discussed more precisely in terms of what is actually done in the algorithm and the experiments.
- as stated by R3, many important aspects of the methods and the experimental setup are only available in appendices, which makes it difficult to understand the similarities and differences in experimental protocol between the curent paper and RL^2, PEARL and IMPORT.
- it is unclear what part of DREAM is critical for performance. There is no thorough ablation study nor discussion of the importance of the information bottleneck term, and the only signal given in Figure 5 is that it is critical. The authors could clarify the two aspects (decoupling and information bottleneck). 


There was some discussion about this paper, but even under the assumption that the authors answered most R3's concerns (R3 didn't engage in discussions), the paper is still borderline. In the end there was little support for acceptance because of the presentation issues above.
Decision: Reject