Learning General World Models in a Handful of Reward-Free Deployments
Authors: Yingchen Xu
Jack Parker-Holder
Aldo Pacchiano
Philip J. Ball
Oleh Rybkin
Stephen J. Roberts
Tim Rocktäschel
Edward Grefenstette
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reward-Free Reinforcement Learning
Deployment Efficiency
World Models
Exploration
Model-based Reinforcement Learning

Abstract: Building generally capable agents is a grand challenge for deep reinforcement learning (RL). To approach this challenge practically, we outline two key desiderata: 1) to facilitate generalization, exploration should be task agnostic; 2) to facilitate scalability, exploration policies should collect large quantities of data without costly centralized retraining. Combining these two properties, we introduce the reward-free deployment efficiency setting, a new paradigm for RL research. We then present CASCADE, a novel approach for self-supervised exploration in this new setting. CASCADE seeks to learn a world model by collecting data with a population of agents, using an information theoretic objective inspired by Bayesian Active Learning. CASCADE achieves this by specifically maximizing the diversity of trajectories sampled by the population through a novel cascading objective. We provide theoretical intuition for CASCADE which we show in a tabular setting improves upon naïve approaches that do not account for population diversity. We then demonstrate that CASCADE collects diverse task-agnostic datasets and learns agents that generalize zero-shot to novel, unseen downstream tasks on Atari, MiniGrid, Crafter and the DM Control Suite. Code and videos are available at https://ycxuyingchen.github.io/cascade/

Learning General World Models in a Handful of Reward-Free Deployments
Yingchen Xu' UCL, FAIR
Jack Parker-Holder' University of Oxford
Aldo Pacchiano" Microsoft Research
Philip J. Ball" University of Oxford
Oleh Rybkin UPenn
Stephen J. Roberts University of Oxford
Tim Rocktäschel UCL
Edward Grefenstette UCL, Cohere
Abstract
Building generally capable agents is grand challenge for deep reinforcement learning (RL). To approach this challenge practically, we outline two key desiderata: 1) facilitate generalization, exploration should be task agnostic; 2) facilitate scalability, exploration policies should collect large quantities of data without costly centralized retraining. Combining these properties, we introduce the reward-free deployment efficiency setting, a new paradigm for RI. research. We then present CASCADE, novel approach for self-supervised exploration in this new setting. CASCADE seeks to Iearn a world model by collecting data with a population of agents, using an information theoretic objective inspired by Bayesian Active Learning. CASCADE achieves this by specifically maximizing the diversity of trajectories sampled by the population through novel cascading objective. We provide theoretical intuition for CASCADE which we show in a tabular setting improves upon naïve approaches that do not account for population diversity. We then demonstrate that CASCADE collects diverse task-agnostic datasets and learns agents that generalize zero-shot novel, unseen downstream tasks on Atari, MiniGrid, Crafter and the DM Control Suite. Code and videos are available at https: h/yc:///gccynncennn.mmmmm wwwpggchhnngigtuu.... io/ii///eeee cascade/
Introduction
Reinforcement learning (RL, [105]) has achieved a number of impressive feats over the past decade, with successes in games [69, 13, 100], robotics [48, 77], and the emergence of real world applications [11, 25]. Indeed, has begun to focus on the grand challenge of building generally capable agents [90, 109, 68, 5].
that RL has successfully mastered host of individual tasks, the community
In this work, we take steps towards building generalist agents at scale, where outline two key desiderata. First, for agents to become generalists that can adapt to novel tasks, we eschew the notion of restricting agent learning task-specific reward functions and focus the reward-free problem setting instead [83, 28], whereby agents must discover novel skills and behaviors without supervision." Consider the problem of learning to control robotic arms, where may already have some expert offline data to learn from. In many cases this data will cover only subset of the entire range of possible behaviors. Therefore, learn additional general skills, it is imperative collect additional novel and diverse data, and to do so without a pre-specified reward function.
Second, to ensure scalability, should have access to a large flect of robots that to gather this data for large number of timesteps [60], without costly and lengthy centralized
deploy
"Equal contribution. Correspondence yexu@ meta.com.
Indeed, designing reward functions to learn behaviors can have unintended consequences [74, 6].
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Diverse Explorers
Train Explorers
Imagine Latent Trajectories
Deploy Explorers
Population Diversity
?
Collect Data
Lasent State
S+
Ver(S)
Encoder
New Tasks
Generalization
Image
O
LD
LD
Lateni Disagreement
Train World Model
Per Info
Figure I: Overview. Left: CASCADE trains population of diverse explorers and uses them to collect large batches of reward-free trajectories for learning general world model that facilitates gen-geenerlizatioon to novel tasks. Right: To train B exploration agents parallel, at each training step CASCADE first infers latent state s, from image observation os. It then rolls out latent trajectories T² ..... T in imagination using the current exploration policies ,....., " The training objective for each policy x² is to optimize 1) the population diversity estimated by the disagreement of the final states of imagined trajectories sampled from policies n² ..... n²; 2) the expected per state information gain over all future timesteps + I,...: T, computed as the disagreement of ensemble of dynamics models.
retraining during this crucial phase [67]. This has recently been referred to as deployment efficiency. falling between the typical online/offline RL dichotomy. Limiting deployments not only reduces the overhead in retraining exploration policies, but also limits the potential costs and risks present when deploying new policies [108, 53], an important consideration in many real world settings, such as robotics [48], education [65] and healthcare [29].
Combining these two desiderata, we introduce the reward-free deployment efficiency setting, a new paradigm for deep RL research. To tackle this new problem we train a population of exploration policies to collect large quantities of useful data via world models [20, 34, 39j. World models allow agents plan and/or train policies without interacting with the true environment. They have already been shown to be highly effective for deployment efficiency [67]. offline RL [118, 49, 8, 88] and self-supervised exploration [99, 97]. Furthermore, world models offer the potential for increasing agent generalization capabilitiess [7, 37, 68, 16, 8, 21, 10, 58]. one of the frontiers of RL research [80, 51]. However, since existing self-supervised methods for learning world models are designed to collect only few transitions with a single exploration policy, they likely produce homogenous dataset when deployed at scale, which does not optimally improve the model.
Instead, drawing analogies from Bayesian Active Learning [42, 52], introduce a new information theoretic objec- tive that maximizes the information gain from an entire dataset collected by a population of exploration agents (see Figure 2). We call our method Coordinated Active Sample Collection via Diverse Explorers or CASCADE (Figure 1). We provide theoretical justification for CASCADE, which emphasizes the importance of collecting data with diverse agents. In addition, we provide rigorous empirical evalua- tion across four challenging domains that shows CASCADE can discover a rich dataset from a handful of deployments. We see that CASCADE produces general exploration strate- gies that are equally adept at both "deep" exploration prob- lems and diverse behavior discovery. This makes it possible to train agents capable of zero-shot transfer when rewards are provided at test time in a variety of different settings.
1
2
ooop
1000
Figure 2: Motivation for CASCADE: Green areas represent high expected information gain. If we train population of agents in- dependently, deployment time they will all follow the trajectory to #1, producing a homogenous dataset. However, we con- sider the diversity of the data then we will produce agents that reach both #1 and #2.
To summarize, our contributions are as follows: 1) We introduce a novel problem setting, Reward- Free Deployment Efficiency, designed to train generalist agents in a scalable fashion; 2) We propose CASCADE, a theoretically motivated model- RL agent designed to gather diverse, highly informative data, inspired by Bayesian Active Learning; 3) We provide analysis that shows CASCADE theoretically improves sample efficiency over other naïve methods that do not ensure sample diversity, and demonstrate that CASCADE is capable of improved zero-shot transfer in four distinct settings, ranging from procedurally generated worlds to continuous control from pixels.

---

2 Problem Statement
Reinforcement learning (RL) considers training an agent to solve Markov Decision Process (MDP), represented as a tuple M = {S,A, P. R. p. 7), where $ E S and a E A are the set of states and actions respectively, P(s [s. a) is probability distribution over next states given previous state and action, R(s, a, s') + r is a reward function mapping a transition to a scalar reward, p is an discount factor. A policy я acting in the environment produces initial state distribution and y 8,, a.r) for an episode with horizon H. Since actions in the trajectory trajectory T = {81, sampled from policy, we can then define the RL problem as finding policy x that maximizes expected returns in the environment, i.c. n² = arg max, E,~ [R(T)]-
We seek to learn policies that can transfer to any MDP within family of MDPs. This can be formalized as a Contextual MDP [51], where observations, dynamics and rewards can vary given a context. In this paper we consider settings where only the reward varies, thus, if the test-time context is unknown at training time we must collect data that sufficiently covers the space of possible reward functions. Finally, to facilitate scalability, we operate in the deployment efficient paradigm [67], whereby policy learning and exploration are completely separate, and during a given deployment, we gather large quantity of data without further policy retraining (c.f. online approaches like DER [112], which take multiple gradient steps per exploration timestep in the real environment). Taken together, we consider the reward-free deployment efficiency problem. This differs from previous work as follows: 1) unlike previous deployment efficiency work, our exploration is task agnostic; 2) unlike previous reward-freee RL work, we cannot update our exploration policy TEXP during deployment. Thus, the focus of our work is on how to train TEXP offfine such that it gathers heterogeneous and informative data which facilitate zero-shot transfer to unknown tasks.
In this paper we make use of model- based RL (MBRL), where the goal is to learn a model of the environment (or world model [96]) and then use it to subsequently train policies solve downstream tasks. To do this, the world model needs to approximate both / and R. Typically, the model will be neural network, parameterized by v/. hence we denote the approximate dynamics and reward functions as Pe and R.... which produces a new "imaginary" MDP, M = (S,A, A, Pa., Rg: p). We focus on Dyna-style MBRL [104], whereby we train policy (Te parameterized by 0) with model-free RL solely using "imagined" transitions inside My. Furthermore, we can train the policy on a single GPU with parallelized rollouts since the simulator is neural network [54]. The general form of all methods in this paper is shown in Algorithm I, with the key difference being step 5: We aim to update TEXP in the new imaginary MDP M& such that continues to collect a large, diverse quantity of reward-free data. Note that TEXP need not be a single policy, but could also refer to collection of policies that we can deploy (either in parallel or in series), such that πT E TEXP-
Algorithm 1 Reward-Free Deployment Efficiency via World Models
1: Input: Initial exploration policy TExP
2: for each deployment do
3: Deploy TEXP to collect a large quantity of reward-free data.
4: Train world model on all existing data.
5: Update TEXP in new imaginary MDP M..
6: end for
We focus learning world models from high dimensional sensory inputs such pixels [34, 76, 47], where at each timestep we are given access to an observation o, rather than a state 8t- A series compact latent
of recent works have shown tremendous success by mapping the observation state 2, [39, 38, 40J. In this paper we will make use of the model from V2 [40], which has been shown to produce highly effective policies in variety of high dimensional environments. The primary component of DreamerV2 is a Recurrent State Space Model (RSSM) that uses a learned latent state to predict the image reconstruction, reward re and discount factor Te- Aside from the reward head, all components of the model are trained jointly, in similar fashion to variational encoders (VAEs, [50, 92]). For zero-shot evaluation, we follow [97] and only train the reward head at test time when provided with labels for our pre-collected data, which is then used to train behavior policy offine. Thus, it is critical that our dataset sufficiently diverse to enable learning novel, unseen behaviors.
3

---

3 Coordinated Active Sample Collection
The aim of this work is to train population of B exploration policies {TEXP)== such that they collectively acquire data which maximally improves the accuracy a world model. To achieve this, we take inspiration from the information theoretic approach in Plan2Explore [97], but crucially focus on maximizing information gain over entire trajectories rather than per state-action, and hence drop the conditional dependence on state and action (see App. C.I for why this distinction is important):
(
TEXP = arg max (M)))MM)) = H(ddm.)) - (M
(I)
where dMg is the distribution of states visited by the policy π in the imaginary MDP MM. This ob- jective produces TEXP, a policy whose visitation distribution has a high entropy when computed over model samples, but has low entropy for each individual MDP model (i.e., high epéstemic/reducable uncertainty). We think of each model Ma as sampled from a posterior distribution over models given the data. A good exploration policy has low entropy on individual models but large entropy across models, i.e. it is intent in visiting regions of the space where there is large uncertainty about the model transitions. To make this objective more general, we represent the trajectory data collected by a policy with a "summary" embedding space [79. 72J. Let 4 ; I - l be a summary function mapping trajectories into this embedding space. 1 [Me] denotes the embedding distribution generated by policy π in imaginary MDP My. We can now write the objective from Eq. 1 as follows:
TEXP = arg max (P*( M&) = H(P* - H(P [M
(2)
This more general framework allows us to consider multiple representations for trajectories, in a similar fashion behavioral characterizations in Quality Diversity algorithms [85]. For the rest of this discussion, we will use the final state embedding as our summary representation. whereby (T) = hg, since in the case of the RSSM, the final latent is compact representation of the entire trajectory collected by the policy, analogous to the final hidden state in an RNN [94, 103].
3.1 A Cascading Objective with Diverse Explorers
We now
consider a population-bassed version of Equation.
2,
using
B agents:
IBB III 1 [Mo)
B
{TEXP 1
arg max I # Bn
p+ MMliMMs)
76
II
[M My
where π ((1, n (B) and IT Ti= #t0 [My] is the product measure of the policies' embedding distributions MM. By definition, the conditional entropy factorizes as:
P
B
[I
Ma = = (Mp)
(4)
It is now possible to show that maximum information gain is achieved with a diverse set of agents: Lemma 1. When all models Ma in the support of the model posterior are deterministic and tabular, and the space of policies consists only of deterministic policies, there always exists a solution
(3) vi j. Moreover, there exists a family of tabular MDP models, (2) {x B satisfying EXX + Vi
(8)
such that the maximum cannot be achieved by setting (i) = π for a fixed π.
The proof of Lemma is in Appendix C.I.I. Since the mutual information objective 3 is submodular, a greedy algorithm yields (1 - 2) approximation of the optimum (where e is Euler's number) [73].
Leveraging this insight, let us assume that we already have set of policies (1), , (i-1), we then select the next policy (+) based on the following greedy objective:
II Th MM 20) vis+-1)
200 arg max I atee
[m 200 Vj visi-1 H [I 2 [Mpd) Vj visi-1)
Which can be factorized in similar fashion to Equation 4 (See Appendix C).

---

3.2 A Tractable Objective for Deep RL
Inspired by [97], we make a couple of approximations to derive a tractable objective for {TTXP 1 in the deep RL setting. First, we assume that the final state embedding distributions Gaussian with means that depend on the policies and sampled worlds, and variances that depend on the worlds, Le. T
w| NN((ww IT) I(w)) In this case, H(PP My w) p(w)) and Eq. 5 reduces solving +()) arg ma en H (TT== [M #0) = vj for
a policy that maximizes the resulting joint entropy of the embedding distribution when added to the policy population. This produces the following surrogate objective, maximizing quadratic cascading disagreement:
PopDiv"(rIax)) 1) [M
Σ #E
(
where D(f-1) is dataset of imagined trajectories sampled from policies (1)),
(f-1) in the
model, and PopDiv is short for Population Diversity. Finally, following [97, 9], we also add a per state information gain component each policy's 's reward encourage richer landscape for data acquisition: InfoGain(T) E, (M,) E(snjer of ((8) a)) where ((;.) is the variance across the ensemble latent state predictions (for details see App. B).
Taken together, these objectives form our approach, which we call Coordinated Active Sample Collection via Diverse Explorers or CASCADE. CASCADE trains agents to optimize: 1) diversity term (PvpDiv) that takes into account the behaviors of the other agents in the population; 2) an information gain term (InfoGain) that encourages an individual agent to sample states that maximally improve the model:
arg max APopDiv " (T| (TI(TEXP)EE) Hil) + (1 A)InfoGain(zt)
(6)
#EII
where À is weighting hyperparameter that trades off whether favor individual model information gain or population diversity. Finally, we train the agents in parallel using (policy) gradient descent over f which makes it possible to achieve the same wall clock time training single agent [23].
3.3 Theoretical Motivation
We now seek to provide a tabular analogue to CASCADE which provides a theoretical grounding for approach. In App. D we outline the pscudo-code of CASCADE-TS, a greedy Thompson Sampling algorithm [4] that produces the i-th exploration policy in a tabular enviornment using "imaginary" data gathered by rumning policies m(1), ((i-1) the model. We can then show the following:
Lemma 2. For the class of Binary Tree MDPs, the CASCADE-TS algorithm satisfies.
Tec, Sequential) < Tee CASCADE-TS) I T(c, SinglePolicyBatele
where TeE,-) are the expected number of rounds of deploying population of B policies necessary to learn the true model up to accuracy; SinglePolicy Batch plays a fixed policy B times in each round; Sequential ial does not have population, and instead interleaves updates and executions of single policy B times within each round.
The proof is in App. D. Indeed, we see that CASCADE-TS achieves provable efficiency gains over a naïve sampling approach that does not ensure diversity in its deployed agents. This provable gain achieved by discouraging the i-th policy away from imaginary state-action pairs sampled by the previous policies by using imaginary counts.
Now returning to CASCADE, we can see the importance leveraging the imaginary data gathered in the model by the previous - policies when training the i-th exploration policy. Concretely, encour- aging policy (() to induce high disagreement with the embeddings produced by (I((),, ... (i.e. the PopDiv term) is analogous the imaginary count bonus term of CASCADE-TS in Line 10 of Alg. 2, which avoids redundant data collection during deployment.
(1-1)) (
5

---

Experiments
In our experiments
whether CASCADE facilitates the learning of generalist agents by evaluating their zero-shot transfer to unseen tasks, given a limited number of reward deployments. In all experiments, agents cannot train online the environment, and instead execute a fixed exploration policy to gather new data during each deployment. We test against three baselines. The first is random exploration, which is used in the majority of RL publications. Our next baseline Plan2Explore [97] (P2E), which trains a single exploration policy that optimizes for expected information gain inside world model. We then include population-basl Plan2Explore (PP2F). PP2E trains a set of randomly initialized agents that independently seek to maximize expected information gain. methods make use of a DreamerV2 world model [40] and use the same hyperparameters for model and agent training (more details in App. B).
version of P2E, which call Population
We test all methods in separate settings: Firstly Exploring Worlds (Sec 4.1), where we seek to collect data in challenging ex exploration environments. We test how well the explorers cover the state space and discover sparse rewards (also known as "deep exploration" [78, 75)). Secondly, we consider Exploring Behaviors (Sec 4.2) where we consider collecting data in continuous control environments. In both settings the model: 2) we train a separate reward head; 3) we train task specific behavior policy and test it in the environment. This tests whether our model is general enough to facilitate learning downstream policies for previously unknown tasks [97, 55, 1 116].
zero-shot transfer follows: we provide reward labels to
FourRaom:
MultiRaom-N4 N4:55
5t000
P2E
13
650
2000
PP2C
1.0
10
Z0r0 Shot Made
CABCADE
5.5
Deployment
Figure 3: MiniGrid Results. Left: Performance statistics for both FourRooms and MultiRoom. For reward- free exploration we show both "state coverage", which corresponds to the the percentage of states visited on fixed set of levels, and "rewarding episodes" which is the count of total solved episodes found after 1M steps. Finally zero-shot transfer we show success rate a task specific behavior policy trained with labels provided only at test time. All plots show the mean of ten seeds with SEM shaded. (Random achieves zero-shot success rate). Right: we show the state coverage on a fixed level in FourRooms, using the exploration policy TEXP for each deployment (Test time goal state indicated by the red dot).
4.1 Exploring Worlds
In many cases we may not know priori which states an environment are highly rewarding. Thus, if we wish to use our world model to subsequently train agents to solve tasks, it is crucial that we accurately model as much of state space as possible. Here we consider exploring worlds three different environments: MiniGrid [19], Atari [12] and Crafter [37]. In all three settings, start with a random deployment before beginning reward-frie exploration.
We begin with MiniGrid, set of rears,-reward, partially observable navigation environments commonly used test state-of-the-art exploration methods [89, 17, 31, 121]. MiniGrid environments procedurally generated, which provides additional generalization challenge [46, 93]. We

---

consider the canonical FourRooms and the more challenging MultiRoom. In each case we are limited deployments of 200k transitions, for total of 1M environment interactions. In order gauge the effectiveness of the reward-free exploration, we use two metrics: 1) state coverage, where we evaluate the percentage the states visited by the exploration policies after each deployment on the same set of held out test levels; and 2) rewarding episodes, where we show the number of rewarding episodes collected during the training process. Note that these two objectives are not useful in isolation, solved levels may only include narrow set of observations, while state coverage alone could be maximized without performing deep, goal-seeking exploration in the environment. However, in combination they provide proxy for exploration.
We show the results in Figure 3, where we see the CASCADE exploration policies cover the state space far more effectively than all baselines. In FourRooms, both CASCADE and PP2E find the goal an equal number of times, while CASCADE finds almost double the number of rewarding episodes in the Mul tooo environment. We also test zero-shot performance, where CASCADE achieves almost 100% success rate. Interestingly, in both settings the Random policy does cover a large proportion of the state space, but finds fewer rewarding episodes and subsequently does not have sufficient quantity to train policy to solve the task (and accordingly achieves 0% success rate).
Next we consider four Atari games: Montezuma's Revenge, Frostbite, Hero and Freeway, often used as barometers for exploration capabilities [27]. We use 15 reward-free deployments, each consisting of 200k steps, and use the final model to train a behavior policy with task reward. We test these agents in a zero-shot fashion, with results shown in the first three columns of Figure 4. We see that CASCADE gets statistically significantly stronger zero-shot returns compared to the baselines. Furthermore, the baseline methods display no clear trend, with random exploration sometimes offering the strongest performance. On the other hand, we see that CASCADE performs consistently well, providing further evidence of being a strong general exploration strategy. See App. A.2 for more detailed analysis.
Montezuma
Frostbite
Hero
Freeway
CASCADE
IQM
PP2E
P2E
Random
PICASCADE > Y)
10
20
30
240
246
252
3300
3400
16 24
PP2E
P2E
Random
0.60 0.75 0.90
0.60
0.75
0.90
0.60
0.75 0.90 CASCADE
0.60 0.75 0.90
Random
P2E
PP2E
Figure 4: Atari Zero-Shot Transfer: Plots show RLiable metric performance aggregated across all seeds. Note that the 'Probability of Improvement lower CIs all exceed 0.5 for Atari, indicating CASCADE provides statistically significant improvement (under the Mann- five seeds of zero-shot test performance.
test) over all baseline methods [3]. We show
Finally, consider the Crafter environment [37], a procedurally generated grid world based the game of Minecraft. This more complex setting requires the agent to master a variety of compositional skills fully explore all possible behaviors. Online P2E represents the state of the art reward-frea frree agent, achieving a "Crafter Score" of 2. Making the problem more challenging, we deploy only 20 distinct exploration policies, each collecting 50k timesteps, which leads weaker performance for P2E vs. the reported result in [40]. However, as we see in Figure 5, adding a population of diverse explorers recovers the majority of this performance. We find this result encouraging as the behaviors required to achieve a higher Crafter score are non-trivial. Moreover, as shown in Figure 6, CASCADE achieves the highest average success rate and unlocks the most number of unique Crafter skills among all baselines, including Making a Stone Pickaxe. We believe this opens the door to further gains with more active tuning of the behavioral representation used in CASCADE.
'Zero-shot success rates of P2E, PP2E and CASCADE increase with more training steps, and CASCADE achieves 100% success rate at 1M steps in both FourRooms and MultiRoom
7

---

IQM
PICASCADE Y)
Training Score
2.1
CASCADE
PP2E
PP2E
P2B
P2E
Random
Random
1.6
1.8
2.0
0.4
0.6
0.8
1.0
0.0
1.0
Random
P2E
PP2E
CASCADE
P2E (Online)
Figure 5: Crafter Score: Plots show final RLiable metric performance and training curve over 1M steps aggregated across all sceds. We show ten seeds of Crafter Score, the geomean of skills discovered in purely reward-free fashion. Again, see that the 'Probability of Improvement' lower Cls all exceed 0.5, indicating CASCADE provides a statistically significant improvement (under the Mann- Monn-Whitncyy test) over bascline methods [3].
Random
P2E
PPZE
CASCADE
1.0
0.5
Collect Coal
Collect Drink
Collect Sapling
Collect Stone
Collect Wood
Defeat Skeleton
Defeat Zombie
Eat Cow
Eat Plant
Make Stone Pickaxe
Make Stone Sword
Make Wood Pickaxe
Make Wood Sword
Place Furnace
Place Plant
Place Stone
Place Table
Wake Up
Figure 6: Crafter Skills: Plots show normalized success rate per skill. Note that CASCADE not only achieves
the highest average success rate but also unlocks the most (16 out 22) unique skills among all baselines. Skills that are not unlocked by any the baselines are excluded from the plot.
4.2 Exploring Behaviors
Next wish to test is whether our agents sufficiently explore range of behaviors in continuous control setting, using the walker environment from the DM Control Suite [111] (DMC). In these experiments, operate from pixels and assume access to initial dataset of 200k transitions. Then, we conduct 14 further reward-free free deployments, each collecting 200k transitions. We consider the possibility of improving generality from arbitrary offline data, using common datasets from the offline RL literature: random, medium and expert. Using random data represents learning from scratch. In Figure 7 we show the behaviors deployed by CASCADE and PP2E in single deployment. The CASCADE explorers all display diverse, useful behaviors, while PP2E agents are homogenous.
Figure 7: DMC Qualitative Results: Each row visualizes behaviors of population of explorers trained after 15 total deployments by the same world model, deployed at the same time in environments with the same initial state. We can see that CASCADE explorers (top) collect data with diversity of behaviors while PP2E agents (bottom) only show less interesting homogenous behaviors.
We then test the generality of the world model after each deployment by training a separate agent from scratch for each of the four individual tasks: stand, walk, run, flip proposed in URLB [56]. This demonstrates whether the world model is capable of producing imaginary MDP that facilitates learning specific behaviors. To compare the performance of CASCADE and the baselines, in Figure 8 we show the aggregated statistics from the RLiable library [3] for given number of deployments. To do this, we combine the results across all datasets (random, medium, expert) and all downstream tasks (stand, walk, run, flip) and compute the following: 1) the Inter-Quartile Mean (IQM) the normalized scores (the robust statistic recommended in [3]); 2) the Probability of Improvement, the likelihood one method outperforming another on new, unseen task. As we see, CASCADE shows clear and consistent gains over the next best baseline (PP2E) throughout 15 deployments, showing

---

Deployment
Deployment 5
Depleyment
Deployment 15
CASCADE
NOM
PP2E
Random
0.21
0.30
PICASCADE > Y)
PP2E
P2E
Random
0.60
0.75
0.90
0.60
0.75
0.90
0.60
0.75
0.90
0.60
0.75
0.90
Random
P2E
PPZE
CASCADE
Figure 8: DMC Zero-Shot Aggregate Results: Plots show RLiable metric performance aggregated across all three initial datasets (random, medium, expert) for total of 30 seeds. Note that the "Probability of Improvement' lower Cls all exceed 0.5, indicating CASCADE provides statistically sigmificant improvement (under Mann- Hann-Whitney test) over all baseline methods [3], even after single deployment.
statistical significance under the Mann- WhinnWy U test [66]. Note that here the single agent P2E performs poorly, likely due to only collecting data with single mode of exploratory behavior. See Figure in the Appendix for a full breakdown of the performance.
4.3 Discussion and Limitations
We have shown that CASCADE the most effective method in training generally capable agents across a wide range of environments. Indeed, CASCADE provides the best of both worlds: deep exploration by seeking the frontier of the current knowledge (via InfoGain), while covering the space of possible behaviors (via PopDiv). Each of our baselines lacked in one of these two key properties. Random exploration fared poorly when exploring worlds, which requires deep exploration. However, it can be highly effective method for exploring behaviors. This is likely why boosting entropy in the action space effective for robotics tasks [36], but less so in deep exploration tasks [75]. Furthermore, the information theoretic baselines were effective in Exploring Worlds, but the disappointing performance for P2E in DMC likely caused by lack of data diversity, while the PP2E populations were not sufficiently heterogeneous match the coverage of CASCADE.
In terms of limitations, the main weakness in current methods is the absolute level of performance. Despite outperforming baselines in DMC, the policies learned by CASCADE are far from optimal. This likely due the challenge of the setting (reward-frr free deployment efficiency) and we believe that future work in the community will be crucial to further improve these methods. Finally, CASCADE does require training additional agents that may provide additional computational expense. this setting we assume the time between deployments is arbitrary, but this may not be the case in all settings and future work could consider ways to make this process more efficient and scalable.
Related Work
At the core of our paper is the notion of training generalist agents, which have recently been of increased interest in RL [90, 68, 109, 1 110]. In particular, focus on the reward-free paradigm, since many cases may not possible to know a priori all possible tasks we may wish our agent to solve [45]. With this in mind, there has been a surge of interest in methods that can learn in self-supervised fashion, to build representations that lead to fast (or even zero-shot) adaptation to future tasks. These methods range from competence based (i.c. learning "skills") [1, 33, 28, 59, 62, 101] data-based that aim to maximize data diversity [63, 117].
In this work we take inspiration from knowledge based approaches, which typically seek to learn world model [9, 83, 84, 30, 107, 15, 98]. particular we build on Plan2Explore [97] which trains an exploration agent by maximizing an objective resembling information gain inside the world model. Plan2Explore was the first work that showed it is possible to transfer a model learned without rewards in zero-shot fashion. However, it differed from our work in that Plan2Explore solely considers the online RI. setting, deploying a new exploration policy every timestep. Other recent work examined the effectiveness of reward-fire free RL for collecting datasets for subsequent offline RL [116, 55]. Once

---

again however, this work trains the exploration agent online. By contrast, we only deploy a handful of exploration agents and collect a larger quantity of data with each one.
Our setring resembles a reward-free version of the deployment efficiency paradigm, first introduced in [67]. By contrast, their method focuses on policy optimization in the supervised RL paradigm, whereas we focus on exploration and zero-shot generalization in the unsupervised RL setting. Follow up work has remained focused on supervised policy optimization [102]. whereas we are focused on training generalist agents in the reward-frre -free paradigm. Recent work has also shown the importance of frequent policy updates in ensuring diverse exploration [95], further supporting the need for a diverse population of explorers when operating in the deployment-efficiene setting.
To address the challenges of this novel problem setting, draw inspiration from variety of fields. In particular, several works have considered exploration with a population of agents [106, 26, 64), but these have only considered task-specific online RI. Our choice of diverse population of agents takes inspiration from the field of Quality Diversity (QD, [85, 24]), where it has been shown that diversity can boost exploration in RL [22]. Recently, [61] showed it is possible to train diverse populations inside a world model to acquire new skills. As far as we are aware, we are the first to consider a cascading objective for training diverse populations, which is typically done either synchronously [82] or sequentially [122, 119].
Also related are works that specifically train policies that are able to maximally cover the state space of given MDP [2. 70J. However, these works focus on the more theoretically amenable (and less general) low-rank MDP setting, and hence do not demonstrate any experiments on larger scale problems. Furthermore, we aim to achieve state-space coverage by encouraging policy diversity, which is distinct from the approaches taken in these works. We make use of an information theoretic objective, which has been shown to be highly effective for exploration in RL [71, 43, 41J. Most similar to our work [120] consider the reward free RL setting, and show that maximizing entropy can lead to improved exploration that can facilitate transfer to arbitrary reward functions. By contrast we consider the limited deployment setting and use model-based RL. Finally, our work relates to collective intelligence [35, 115], since each member of the population contributes a small piece to the puzzle that produces general world model.
Conclusion
We introduced the reward-free deployment efficiency paradigm-an important problem setting for learning generally capable agents in a scalable fashion. To address the challenges of this setting, we proposed CASCADE, a theoretically motivated approach that leverages diverse population of self- supervised exploration agents. We showed a variety of experiments that CASCADE produces general exploration strategies, adept both deep exploration and gathering sufficiently hetercogencous datasets that facilitate the learning of diverse downstream behaviors. We believe this work provides a new approach for collecting large, diverse datasets in self-supervise fashion, inducing potentially open-ended process for training general world models at scale.
For future work, we could consider alternative means for computing diversity, for example using optimization landscapes [81] or explicit behavioral diversity [28, 82]. These could also combined with representations that directly incorporate the behaviors of the policy [14] to further improve the
design of the "summary" embedding space. Furthermore, we would be interested in considering transformer models [1 [1]3] for trajectory modelling [18, 44] that ought to scale more gracefully with larger batches of data [87]. Recent work has shown that this class of models can compose unseen complex multi-task behaviors at test time from fixed offline datasets [32], as well as leverage significantly large and diverse offline datasets to improve behavioral modeling [91], highly relevant to both aforementioned desiderata for the reward-free deployment efficiency paradigm.
Acknowledgments
The authors would like to thank Cong Lu for help with Offline DreamerV2 code, and Minqi Jiang and Robert Kirk for useful discussions. This work was funded by Meta AL.
10

---

References
[1] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel. Variational option discovery algorithms. CoRR, abs/1 807.10299, 2018.
[2] A. Agarwal, S. Kakade, A. Krishnamurthy, and W. Sun. Flambe: Structural complexity and representation learning of low rank mdps. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 20095-20107. Curran Associates, Inc., 2020.
[3] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Deep rein- forcement learning at the edge of the statistical precipice, Advances in Neural Information Processing Systems, 34, 2021.
[4] S. Agrawal and N. Goyal. Analysis of thompson sampling for the multi-armed bandit problem. In Conference learning theory, pages 39-1. JMLR Workshop and Conference Proceedings, 2012.
[5] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, K. Gopalakrishnan, Hausman, A. Herzog, Ho, J. Hsu, J. Ibarz, B. Ichter, Irpan, E. Jang, R. J. Ruano, K. Jeffrey, S. Jesmonth, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, K.-H. Lee, S. Levine, Y. Lu, Luu, C. Parada, Pastor, Quiambao, K. Rao, Rettinghouse, D. Reyes, Sermanet, N. Sievers, C. Tan, A. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, S. Xu, and M. Yan. Do as can and not as say: Grounding language in robotic affordances. In arXiv preprint ar arXiv:2204 Xiiv:2204.01691,, 2022.
[6] D. Amodei and Clark. Faulty Reward Functions in the Wild. https://blog. openai. com/faull reward-wwwdddddteeeeeeeeeee
2016.
[7] A. Anand, J. C. Walker, Y. Li, E. Vértes, J. Schrittwieser. S. Ozair, T. Weber, and J. B. Hamrick. Procedural generalization by planning with self-supervised world models. In International Conference on Learning Representations, 2022.
[8] A. Argenson and G. Dulac-Amold. Model-based offline planning. In International Conference on Learning Representations, 2021.
[9] P. Ball, J. Holder, A. Pacchiano, K. Choromanski, and S. Roberts. Ready policy one: World building through active learning. In Proceedings of the 37th International Conference Machine Learning, ICML.,000)
[10] P. J. Ball, C. Lu, J. Barker-Holder and S. Roberts. Augmented world models facilitate zero- shot dynamics generalization from a single offline environment. In Proceedings of the 38th International Conference on Machine Learning, pages 619-629, 2021.
[11] M. Bellemare, S. Candido, P. Castro, J. Gong, M. Machado, S. Moitra, S. Ponda, and Z. Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588:77-82, December 2020.
[12] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The Arcade Learning Environment: An Evaluation Platform for General Agents. CoRR, 2211120077770 2012.
[13] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, R. Józefowicz, Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. de Oliveira Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and S. Zhang. Dota 2 with large scale deep reinforcement learning. CORR, abs/1912.06680, 2019.
[14] H. Bharadhwaj, M. Bahaeizadeh, D. Erhan, and S. Levine. Information prioritization through empowerment in visual model-based RL. In International Conference on Learning Represen- tations, 2022.
[15] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2019.
11

---

[16] A. Byravan, J. T. Springenberg, A. Abdolmaleki, R. Hafner, M. Neunert, T. Lampe, N. Siegel, N. M. O. Heess, and M. A. Riedmiller. Imagined value gradients: Model-basee policy optimization with transferable latent dynamics models. In CoRI., 2019.
[17] A. Campero. R. Raileanu, H. Kuttler, B. Tenenbaum, T. Rocktäschel, and E. Grefenstette. Learning with AMIGo: Adversarially motivated intrinsic goals. In International Conference on Learning Representations, 2021.
[18] L. Chen, K. Lu, A. Rajeswaran, K. Lee. A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[19] M. Chevalier-Boisvert, L. Willems, and S. Pal. Minimalistic gridworld environment for OpenAI Gym. https: /ttp//ggtttt.tt/// com/maximecb..b...maaoooommmoooe beslo//ggmmmmiii gym-minigrid, 2018.
[20] K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Advances in Neural Information Processing Systems 31, pages 4754-4765, 2018.
[21] I. Clavera, A. Nagabandi, S. Liu, R. S. Fearing, P. Abbeel, S. Levine, and C. Finn. Learn- ing to adapt in dynamic, real-world environments through meta-reinffrcemeet learning. In International Conference on Learning Representaatinss 2019.
[22] E. Conti, V. Madhavan, F. P. Such, J. Lehman, K. O. Stanley, and J. Clune. Improving exploration in Evolution Strategies for deep reinforcement learning via population of novelty- seeking agents. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, pages 5032-5043, USA, 2018. Curran Associates Inc.
[23] B. Cui, H. Hu, L. Pineda, and J. N. Foerster. K-level reasoning for zero-shot coordination in hanabi. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[24] A. Cully, J. Clune, D. Tarapore, and J.-B. Mouret. Robots that can adapt like animals. Nature, 221:503-507,,000 2015.
[25] J. Degrave, F. Felici, J. Buchli, M. Neunert, B. Tracey, F. Carpanese, T. Ewalds, R. Hafner, A. Abdolmalcki, D. de las Casas, C. Donner, L. Fritz, C. Galperti, A. Huber, J. Keeling, M. Tsimpoukelli, J. Kay, A. Merle, J.-M. Moret, S. Noury, F. Pesamosca, D. Pfau, O. Sauter, C. Sommariva, S. Coda, B. Duval, A. Fasoli, P. Kohli, K. Kavukcuoglu, D. Hassabis, and
M. Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning.
Nature, 602:414-419, February 2022.
[26] M. Dimakopoulou, I. Osband, and B. Van Roy. Scalable coordinated exploration in concurrent reinforcement learning. S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018
[27] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune. First return then explore. Nature, 590 7847:580-586, 2021.
[28] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2019.
[29] M. Fatemi, T. W. Killian, J. Subramanian, and M. Ghassemi. Medical dead-ends and learning to identify high-risk states and treatments. M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 4856-4877
Curran Associates, Inc., 2021.
[30] A. Filos, E. Vértes, Z. Marinho, G. Farquhar, D. Borsa, A. Friesen, F. Behbahani, T. Schaul, Barreto, and S. Osindero. Model-value inconsistency as a signal for epistemic uncertainty. arXiv preprint arXiv:2112.0415. 2021.
12

---

[31] Y. Flet-Berliac, J. Ferret, O. Pietquin, P. Preux, and M. Geist. Adversarially guided actor-critic In International Conference on Learning Representations, 2021.
[32] H. Furuta, Y. Matsuo, and S. S. Gu. Generalized decision transformer for offline hindsight information matching. In International Conference on Learning Representations, 2022.
[33] K. Gregor, D. Rezende, and D. Wierstra. Variational intrinsic control. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.
[34] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NeurlPS' 18, pages 2455-2467, 2018.
[35] D. Ha and Y. Tang. Collective intelligence for deep learning: A survey of recent developments. CoRR, abs/2111.14377 2021.
[36] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft actor-critic algorithms and applications. CoRR, abs//812.00 182.0599,, 2018.
[37] D. Hafner. Benchmarking the spectrum of agent capabilities. In International Conference Learning Representations, 2022.
[38] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Represenlations, 2020.
[39] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson. Learning latent dynamics for planning from pixels. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, pages 2019.
[40] D. Hafner, P. Lillicrap, M. Norouzi, and Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021.
[41] E. Hazan, S. Kakade, K. Singh, and A. Van Soest. Provably efficient maximum entropy explo- ration. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 268 1-291. PMLR. 09-15 Jun 2019.
[42] N. Houlsby, F. Huszar, Z. Ghahramani, and M. Lengyel. Bayesian active learning for classifi- cation and preference learning. CoRR, abs/ 1 112.5745, 2011.
[43] R. Houthooft, X. Chen, Y. Duan, J. Schulman, F. De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[44] M. Janner, Q. Li, and S. Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[45] Jin, A. Krishnamurthy, M. Simchowitz, and T. Yu. Reward-free exploration for reinforce- ment learning. In International Conference on Machine Learning, pages 4870-4879. PMLR, 2020.
[46] N. Justesen, R. R. Torrado, P. Bontrager, A. Khalifa, J. Togelius, and S. Risi. Procedural level generation improves generality of deep reinforcement learning. CoRR, abs/1806. 10729, 2018.
[47] L.. Kaiser, M. Babaeizadeh, Milos, B. Osiński, R. H. Camphell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, and H. Michalewski. Model based reinforcement learning for Atari. In International Conference on Learning Representations, 2020.
[48] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference Robot Learning, pages 651-673, PMLR, 2018.
13

---

[49] R. Kidambi, A. Rajeswaran, P. Netrapalli, and T. Joachims. Morel Model-based offline
reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
[50] D. P. Kingma and M. Welling. Auto-encoding variational bayes, 2014.
[51] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111.09794, 2021.
[52] A. Kirsch, J. Van Amersfoort, and Y. Gal. Batchbald: Efficient and diverse batch acquisition for deep bayesian active learning. Advances in neural information processing systems, 32, 2019.
[53] J. Kober, J. A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32( 32(11):1238-1274, 2013.
[54] T. Kurutach, I. Clavera, Y. Duan, A. Tamar, and P. Abbeel. Model-ensemble trust-region policy optimization. In International Conference on Learning Representations, 2018.
[55] N. Lambert, M. Wulfmeier, W. F. Whitney, A. Byravan, M. Bloesch, V. Dasagi, T. Hertweck, and M. A. Riedmiller. The challenges of exploration for offline reinforeement learning. CoRR, 2022.
[56] M. Laskin, D. Yarats. H. Liu, Lee, A. Zhan. Lu, C. Cang, L. Pinto, and Abbeel. URLB: Unsupervised reinforcement learning benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[57] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L., D. Jackel. Backpropagation applied handwritten zip code recognition, 1989.
[58] K. Lee, Y. Sco, S. Lee, H. Lee, and J. Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In International Conference on Machine Learning. 2020.
[59] L. Lee, B. Rysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. 2019.
[60] S. Levine, P. Pastor, A. Krizhevsky, J. Ibarz, and D. Quillen. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int. J. Robotics Res., 37(4-5):421-4333 2018.
[61] B. Lim, L. Grillotti, L. Bernasconi, and A. Cully. Dynamics-awarr quality-diversity for efficient learning of skill repertoires. In ICRA, 2022.
[62] H. Liu and P. Abbeel. Aps: Active pretraining with successor features. In Proceedings of the 38th International Conference on Machine Learning, pages 6736-6747, 2021.
[63] H. Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. In Advances in Neural Information Processing Systems. 2021.
[64] I.J. Liu, U. Jain, R. A. Yeh, and A. Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6826-6836 6836. PMLR, 18-24 Jul 2021.
[65] T. Mandel, Y.-E. Liu, S. Levine, E. Brunskill, and Z. Popovic. Offline policy evaluation across representations with applications to educational games. In Proceedings of the 2014 International Conference on Autonomous Agents and Multi-Agent Systems, AAMAS '14, page 1077-1084. Richland, SC, 2014. International Foundation for Autonomous Agents and Multiagent Systems.
[66] H. B. Mann and D. R. Whitney. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1):50 - 60, 1947.
14

---

[67] T. Matsushima, H. Furuta, Y. Matsuo, O. Nachum, and S. Gu. eegloyment-efficieen reinforce- ment learning via model-based offline optimization. International Conference on Learning Representations, 2021.
[68] Mendonca, O. Rybkin, K. Damiilidis. D. Hafner, and D. Pathak. Discovering and achieving goals via world models. In Advances in Neural Information Processing Systems. 2021.
[69] V. Mnih, K. Kavukeuoglu, D. Silver, A. Rusu, Veness, M. Bellemare. A. Graves, M. Ried- miller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonogiou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-33, 23, 02 2015.
[70] Modi, J. Chen, A, Krishnamurthy, N. Jiang, and Agarwal. Model-free representation learning and exploration in low-rank mdps, 2021.
[71] S. Mohamed and D. J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In Proceedings of the 28th International Conference on Neural Information Processing Systems Volume 2, N1PS'15, page 2125-2133, Cambridge, MA, USA, 2015. MIT Press.
[72] T. Moskovitz, M. Arbel, F. Huszar, and Gretton. Efficient wasserstein natural gradients for reinforcement learning. In International Conference on Learning Representations, 2021.
[73] L. Nemhauser, L. Wolsey, and M. Fisher. An analysis of approximations for maximizing submodular set functions-2 -...-ii. Mathematical programming, 14(1):265-294 1978.
[74] Y. Ng. Harada, and Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In In Proceedings of the Sixteenth International Conference on Machine Learning pages 278-2288880 287. Morgan Kaufmann, 1999.
[75] B. O'Donoghue, I. Osband, and C. lonescu. Making sense of reinforcement learning and probabilistic inference. In International Conference on Learning Representations, 2020.
[76] J. Oh, X. Guo, H. Lee, R. Lewis, and S. Singh. Action-conditional video prediction using deep networks atari games. Proceedings of the 28th International Conference on Neural Information Processing Systems- Volume 2, NIPS' 15, page 2871, Cambridge, MA, USA, 2015. MIT Press.
[77] OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. W. Pachocki, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray. J. Schneider, Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. CoRR, abs/1808. 188.00117. 2018.
[78] I. Osband, B. V. Roy, D. J. Russo, and Z. Wen. Deep exploration via randomized value functions. Journal of Machine Learning Research, 20(124):1-62 2019.
[79] A. Pacchiano, J. Parker-Holder, Y. Tang, A. Choromanska, K. Choromanski, and M. I. Jordan. Learning to score behaviors for guided policy optimization. In Proceedings of the 37th International Conference on Machine Leurning, 2020.
[80] C. Packer, K. Gao, Kos, P. Krähenbühl, V. Koltun, and D. Song. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv: 12282, 2018.
[81] J. Parker- Holkdrr L. Metz, C. Resnick, H. Hu, A. Lerer, A. Letcher, Peysakhovich, A. Pac- chiano, and J. Foerster. Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. In Larochelle, M. Ranzato, Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 753-765. Curran Associates, Inc., 2020.
[82] J. Parker-Holder, A. Pacchiano, K. Choromanski, and S. Roberts. Effective diversity in population-based reinforcement learning. In Advances Neural Information Processing Systems 33, 2020.
15

---

[83] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, Curiosity driven exploration by self- supervised prediction. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, page 2778-2787. JMLR.org. 2017.
[84] D. Pathak, D. Gandhi, and A. Gupta. Self-supervised exploration via disagreement. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5062-5071. PMLR, 09-15 Jun 2019.
[85] J. K. Pugh, L. B. Soros, and K. O. Stanley. Quality diversity; A new frontier for evolutionary computation. Frontiers in Robotics and AI, 3:40, 2016.
[86] S. Qiu, Ye, Z. Wang, and Z. Yang. On reward-free r1 with kernel and neural function approximations: Single-agent mdp and markov game. In International Conference on Machine Learning, pages 8737-8741 PMLR, 2021.
[87] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and Sutskever. Language models are unsupervised multitask learners. 2019.
[88] R. Rafailov, T. Yu, A. Rajeswaran, and C. Finn. Offline reinforcement learning from images with latent space models. In Proceedings of the 3rd Conference on Learning for Dynamics and Control, volume 144 of Proceedings of Machine Learning Research, pages 1164-1668. PMLR, 2021.
[89] R. Raileanu and T. Rocktäschel. Ride: Rewarding impact-driven exploration for procedurally- generated environments. In International Conference on Learning Representations, 2020.
[90] S. Reed, K. Zolna, E. Parisotto, G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, J. Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A generalist agent, 2022.
[91] M. Reid, Y. Yamada, and S. S. Gu. Can wikipedia help offline reinforcement learning?, 2022.
[92] D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286. PMLR, 2014.
[93] S. Risi and J. Togelius. Increasing generality in machine learning through procedural content generation. Nature Machine Intelligence, 2, 08 2020.
[94] D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning representations by back- propagating errors. 323(6088):533-536 Oct. 1986.
[95] T. Schaul, A. Barreto, J. Quan, and G. Ostrovski. The phenomenon of policy churn, 2022.
[96] J. Schmidhuber. Making the world differentiable: On using self-supervised fully recurrent neu- ral networks for dynamic reinforcement learning and planning in non-stationary environments. Technical report, 1990.
[97] Sekar, O. Rybkin, K. Daniilidis, P. Abbeel, D. Hafner, and D. Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583-8592. PMLR, 2020.
[98] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
[99] P. Shyam, W. Jaškowski, and F. Gomez. Model-based active exploration. In International conference on machine learning, pages 5779-5788. PMLR, 2019.
[100] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche. J. Schrittwieser,
I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, Nham, Kalch- brenner, I. Sutskever, T. P. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484 2016.
16

---

[101] D. Strouse, K. Baumli. D. Warde-Farley, V. Mnih, and S. S. Hansen. Learning more skills through optimistic exploration. In International Conference on Learning Representations. 2022.
[102] D. Su, J. D. Lee, J. Mulvey, and H. V. Poor. MURO: Deployment constrained reinforcement learning with model-based uncertainty regularized batch optimization, 2022.
[103] I. Sutskever, O. Vinyals, and Q. V. Le. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
[104] R. S. Sutton. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 2(4):160-163, July 1991.
[105] R. S. Sutton and A. G. Barto. Introduction Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998.
[106] R. S. Sutton, J. Modayil, M. Delp. T. Degris, P. M. Pilarski, A. White, and D. Precup. Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. In The 10th International Conference on Autonomous Agents and Multiagent Systems Volume AAMAS '1, page 761-768, Richland, SC, 2011. International Foundation for Autonomous Agents and Multiagent Systems.
[107] O. Svidchenko and A. Shpilman. Maximum entropy model-based reinforcement learning. CoRR, abs/21 195, 2021.
[108] M. Tan. Cost-sensitive learning of classification knowledge and its applications in robotics. Mach. Learn., 13(1):7-33, oct 1993.
[109] O. E. L. Team, A. Stooke, A. Mahajan, C. Barros, C. Deck, J. Bauer, J. Sygnowski, M. Tre- bacz, M. Jaderberg, M. Mathieu, N. McAleese, N. Bradley-Schmieg.. N. Wong, N. Porcel, R. Raileanu, S. Hughes-Fitt, V. Dalibard, and W. M. Czamecki. Open-ended learning leads to generally capable agents. CoRR, 0bbs/207..1208, 2021.
[110] A. Touati and Y. Ollivier. Learning one representation to optimize all rewards. In Advances in Neural Information Processing Systems, volume 34, pages 13-23, 2021.
[111] S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, Bohez, Merel, T. Erez, Lillicrap. N. Heess, and Y. Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020.
[112] van Hasselt, M. Hessel, and J. Aslanides. When to use parametric models reinforcement learning? In Advances in Neural Information Processing Systems, pages 14322-14333, 2019.
[113] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Guyon, U, V. Luxburg, Bengio, H. Wal- lach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[114] R. Wang, S. S. Du, L. Yang, and R. R. Salakhutdinov. On reward-free reinforcement learning with linear function approximation. Advances in neural information processing systems, 33:17816-17826 2020.
[115] A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine. Collective robot reinforcement learning with distributed asynchronous guided policy search. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 79-86, 2017.
[116] D. Yarats, D. Brandffonreener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don't t change the algorithm, change the data: Exploratory data for offline reinforcement learning. CoRR, abs/2201. 13425, 2022.
[117] D. Yarats, R. Fergus, A. Lazaric and L. Pinto. Reinforcement learning with prototypical representations. In Proceedings of the 38th International Conference on Machine Learning, pages 1 1120--1931, 2021.
17

---

[118] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine. C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 2020.
[119] T. Zahavy, Schroecker, Behbahani, K. Baumli, S. Flennerhag, S. Hou, and S. Singh. Discovering policies with domino: Diversity optimization maintaining near optimality. arXiv preprint arXiv:2205.1333 13521, 2022.
[120] C. Zhang, Y. Cai, L. Huang, and J. Li. Exploration by maximizing renyi entropy for reward- free RL framework. In Thirty-Fifthl AAAI Conference Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 10859-10867. AAAI Press, 2021.
[121] T. Zhang. H. Xu, X. Wang, Y. Wu, K. Keutzer, J. E. Gonzalez, and Y. Tian. Noveld: A simple yet effective exploration criterion. In Beygelzimer, Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[122] Z. Zhou, W. Fu, B. Zhang, and Y. Wu. Continuously discovering novel strategies via reward- switching policy optimization. International Conference on Learning Representations, 2022.
18

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.3
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
19


Reviews:
[{"review_id": "jikxKFtNzu", "reviewer": "Reviewer_zCLK", "summary": "This work introduces a new problem setting, Reward Free Deployment Efficiency focusing on two incentives: 1. Task agnostic exploration facilitates generalization. 2. Exploration policies that can collect large quantities of data without centralized retraining facilitate scalability.\n\nIn addition, this work introduces the Coordinated Active Sample Collection via Diverse Explorers (CASCADE), which can gather a diverse set of data and is inspired by Bayesian Active Learning.\n\nIn this work, B exploratory agents are trained in parallel. At each deployment, a loss containing two parts is maximized; the first part is a diversity term between the agents' behaviors. And the second part is the so-called information gain. \n\nThe authors have used the DreamerV2 agent as the base agent for their work and their baselines. Furthermore, they used the well-known Plan2Explore (P2E) and a modified version (Population Plan2Explore, PP2E) as their baseline. Finally, they have produced experiments to show how well their proposed method can explore the state space and also perform with a zero-shot manner to the task-specific problem.", "questions": "1. Why is rewarding-episodes being considered in a reward-free setting? (This is a bit confusing)\n2. Would it be possible to have the final performance of the P2E (not in the deployment efficiency setting) reported as a dashed line? This is informative in case one wants to see how difficult the proposed problem setting is.\n3. Could you elaborate more on this? \n\n>to facilitate scalability, exploration policies should collect large quantities of data without costly centralized retraining.\n\n I can easily imagine real-world scenarios where access to about 20 different agents is not possible.\n", "limitations": "The authors have addressed the limitation of their work in section 4.3.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\nThis work introduces a new setup that is called reward-free deployment efficiency. Furthermore, inspired by the P2E agent, the authors propose a new method called CASCADE that can perform decently compared to the previous baseline in this setup.\n\nQuality:\nThe authors have done an excellent job explaining their setup, proving their claims, and discussing their experiments.\n\nClarity:\nThe paper is well-written which prevents a familiar reader from doing additional passes through different paragraphs.\n\nSignificance:\nDespite the fact that the proposed method, CASCADE, is well-motivated, I do think that there are few arguments about why the reward-free deployment efficiency matters. I can assume real-world scenarios where access to about 20 different agents is not possible.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Sj1UKx7Mx9y", "reviewer": "Reviewer_Yjhj", "summary": "# New setting for reinforcement learning\n\nThe paper first introduces **reward-free deployment efficiency**, a new setting for reinforcement learning.\n\nThe *deployment efficiency* part means that we limit the number of agent versions that we deploy in the environment to collect new experience (for scalability reasons). \n\nThe *reward-free* part means that the collection procedure is task-agnostic. It aims at exploring the state-action space as much as possible without reward consideration (for generalization reasons).   \n\n# New algorithm within this setting\n\nWithin this setting, they propose **CASCADE**, an algorithm to deploy a fleet of *coordinated* agents that gather task-agnostic experience in the environment. \n\nSimilarly to [Plan2Explore](https://arxiv.org/pdf/2005.05960.pdf), the exploring behavior is trained with an intrinsic reward that estimates the uncertainty of a world model (they use DreamerV2 world model). This objective has an information-theoretic interpretation.  In the same way as Plan2Explore, they provide reward labels at test time, after data collection, to train a reward predictor for the world model. They can then train a task-specific agent in the imaginary MDP defined by the world model. \n\nThe major difference is to have different exploration policies and to *coordinate* them, so that they can organize together to explore different parts of the world.\n\nThey provide a theoretical motivation for this diversity of behaviors on a tabular version of their method (CASCADE-TS). \n\nTheir experiments demonstrate the superiority of CASCADE over random exploration and Plan2Explore, on MiniGrid, Montezuma's Revenge, and 4 tasks in DeepMind Control (stand, walk, run, flip). \n\n", "questions": "1) The `while True` in Algorithm 1 is slightly confusing since the deployment efficiency is precisely to limit the number of iterations of this loop, which here is infinite. Why couldn't you use a `for` loop with the number of deployments instead? That would allow to explicitly mention the number of deployments and precise that it is supposed to be small.\n\n2) L97, could you explain why updating $\\pi_{EXP}$ using the uncertainty of the imaginary MDP is different from Plan2Explore? \n\n3) Figure 3, in FourRooms, how is it that P2E, with a lower state coverage and less rewarding episodes found, seems to beat PP2E on zero-shot success rate? \n\n4) Still in Figure 3, could you explain more why CASCADE consistently reaches 100\\% zero-shot success rate, while its number of rewarding episodes and state coverage are very similar to PP2E?\n\n5) Figure 5, I would expect the 4 methods to be ranked randomly at deployment 0 (they all have access to the same data to start with), but across 30 seeds and 4 tasks, it looks like CASCADE is above the other ones even at deployment 0. Is there any reason for that?\n\n6) Would it be possible to address the weaknesses pointed out in the \"Strength And Weaknesses\" sections? In particular, experiments on more Atari games / Crafter, comparison to Plan2Explore with their number of deployments and environments, and clarification of the formalism.\n", "limitations": "Section 4.3 clearly mentions some limitations. In particular, the authors make it clear that the learnt zero-shot policies are far from being optimal, which is understandable considering the difficulty of the reward-agnostic experience collection.  \n\nThey do not discuss the potential negative impacts of their method, but clearly explain why the \"deployment efficiency\" constraint is useful in terms of security and cost.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\n- Clear demonstration of the ability to explore the world in MiniGrid toy environments and Atari Montezuma's Revenge, with clear visualizations.\n\n- Interesting improvements over an initial dataset of demonstrations with very few deployments in DeepMind Control.\n\n- They provide a theoretical motivation behind the intuition that it is better to have diverse exploration policies.\n\n--- \n\n# Weaknesses\n\n## Lack of experimental evidence\n\nThe experiments about world exploration (4.1) are in toy environments of MiniGrid (FourRooms and MultiRoom) and in only one Atari game (Montezuma's Revenge). Running all Atari might be too much, but some Atari games present interesting world exploration problems (Hero, Freeway, Frostbite for instance). \n\nAlso, [Crafter](https://github.com/danijar/crafter) would be a much less trivial environment to benchmark the capabilities of CASCADE: procedurally generated world with many assets and interesting behaviors to discover.\n\n---\n\n## Deployment-efficiency limits comparison to prior work\n\nThe deployment-efficiency setting seems a bit artifical here. Indeed, it is justified in the text by scalability reasons, that are important when it comes to real world problems, but the environments considered in this work are classical toy environments. \n\nIntroducing a restriction on the number of deployments prevents proper comparison to prior work. In particular, it would have been really interesting to compare CASCADE with Plan2Explore (also reward-free / task-agnostic) by using their number of deployments and all of their DMC environments (a superset of the ones considered here). \n\nIntroducing this new setting (which is not justified by any real world environments) is risky since the comparison / novelty / interest are much harder to assess with a change in the experimental setup.\n\n---\n\n## The formalism is sometimes very unclear\n\nIn particular, L111 to L125, the information theoretic objective taken from Plan2Explore (Equation 1) is confusing. \n\nFirst, looking at [Plan2Explore](https://arxiv.org/pdf/2005.05960.pdf):\n- The intrinsic reward is to collect experience that maximizes the world model uncertainty, and they estimate this uncertainty via the empirical variance of an ensemble of next WM-latent predictors. \n- They clearly state that this objective only *approximates* an information gain, and they frame it as a nice information theoretic interpretation of their objective. \n\nHere, equation 1 states that the exploration policy is the argmax of the mutual information term, which is confusing since it is not directly optimized in Plan2Explore but approximated. Also, L116 mentions several MDPs to explain the mutual information term, who are they? In fact, in Plan2Explore, it is clear that there is a *single* imaginary MDP (defined by a single world model), and that they consider the disagreement in the ensemble of predictors, all trained to predict the dynamics of the same MDP dynamics. The explanation with several MDPs does not seem to make sense. This ensemble of predictors seems to be the central piece of the \"information theoretic\" objective, but is only mentioned in Appendix B as an implementation detail.\n\nAlso, in Plan2Explore, the mutual information is taken with a random variable $w$ that represent the optimal dynamics parameters, in a Bayesian way. Here, in equation 1, what is the mutual information between a distribution of states and an imaginary markov decision process? \n\n---\n\n## Theory is more a motivation than a guarantee\n\nFrom what I understand, they prove that is is better to have diverse explorers (Lemma 1) and that CASCADE-TS (tabular version of CASCADE) learns faster or similarly to a naive fleet of identical explorers. However, there is no guarantee that CASCADE-TS actually converges to a solution of the optimization problem. \n\nThe name of section 3.3 is clear on that point (\"Theoretical Motivation\") but the abstract claim is not (\"theoretical guarantees\").\n\n---\n\n## The writing could be more polished\n\nSome typos, and the notations could be more rigorous (e.g. L92 $\\rho_0$ not defined, L137 $e$ is not defined).\n\n---\n\n# Originality\n\nOriginality is not clearly a strength or a weakness here. The work relies on the formalism of Plan2Explore and the code of DreamerV2, but the added term to maximize diversity among explorers in order to achieve better exploration / zero-shot generalization is interesting. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "96JD7H-SwU1", "reviewer": "Reviewer_bwKR", "summary": "The paper introduces a setting called reward-free deployment efficient setting, where one can interact with the environment with limited number of times/limited number of updates (but could with multiple agents in parallel), in the reward-free fashion. The paper proposes to solve this problem by learning a world model with a network ensemble, and with a certain number of exploration policies. The paper presents to train the world model in a similar fashion as DreamerV2, and train the exploration policies with a composition of heuristic objectives. Finally, the paper evaluates the effectiveness of the algorithm on both game and locomotion benchmarks. ", "questions": "## Major Questions\n\n1. In Figure 1, the top two rows seem identical. Thus there are a lot of redundancies in the figure and in fact seems a little bit strange. Also, the definition of deployment seems to be the number of interactions that the algorithm itself will have with the environment? Thus the number of deployments of offline RL should be 0? Otherwise, if the offline dataset is collected from a heterogeneous source, the number of deployments of offline RL could also be large?\n\n2. The definition of contextual mdp (CMDP): according to [3], the definition of CMDP seems to be that we have a set of mdps, where each mdp, in addition to have different reward, but also have different dynamics conditioned on the context?\n\n3. The derivation of the final objective is very confusing. In the main text, the ImagDiv term seems to be a heuristic surrogate of the first entropy term in line 140. Under the assumption that the conditional latent distribution is a gaussian with variance unrelated to the policy, which seems to be the assumption made for section 3.2, the ImagDiv objective is actually estimating the whole objective? However, in the appendix, the paper also presents the derivation for the InfoGain term, under the same assumption. This is confusing because the paper tries to use two different objectives to estimate the same thing?\n\n4. In the definition of ImageDiv term, is the expectation also over the model ensembles? Also $\\Phi$ is not defined in the ImageDiv term but is defined in the appendix for the InfoGain term. Is $\\Phi$ in ImagDiv the same as $\\mu(m_{\\psi}\\pi)$?\n\n5. In Fig.3, why is PP2E10's zero-shot success rate higher in a more challenging environment?\n\n6. In the experiment for locomotion, why do we need to assume that we have access to the offline dataset? It is known that offline RL needs good coverage from the offline dataset to perform well, thus it looks like in this setting the algorithm already gets a good warm-start? How would the method perform without the offline dataset?\n\n### references\n\n[3] Hallak, Assaf, Dotan Di Castro, and Shie Mannor. \"Contextual markov decision processes.\" arXiv preprint arXiv:1502.02259 (2015).\n\n## Minor issues\n\n- In algorithm 1, it seems like the algorithm will never terminate?\n\n- Many hyperlinks to the sections in the appendix are wrong. For example, line 136.\n\n- Line 140, on the right hand side of the last equality, in the second entropy term, the distribution of latents should be conditioned on $\\tilde \\pi^{(j)}$'s but not $\\pi^{(j)}$'s (the subscripts).\n\n- The x-axis in Fig.5 is different from the text description. ", "limitations": "1. The result in Lemma 1 assumes deterministic MDP and deterministic policy. Would it be obvious that one policy may not be able to explore the whole environment?\n\n2. Section 3.3 seems to be in an awkward position. After a very heuristic description of the deep RL approach, section 3.3 returns to a theoretical analysis of a variant of the algorithm, and the setup is also simpler (tabular MDP). Maybe it could instead serve as a motivating section? ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n1. The idea of using a set of exploration policies is a crucial contribution, which also ensembles the idea of \"policy cover\" used for exploration in multiple theory papers. The paper provides some good intuition on why using beyond one fixed policy is important for exploration, such as Figure.2. However, the theory component does not seem very surprising, which will be more detailed explained in the latter part of the review. \n\n2. The new reward-free deployment efficient setting indeed has some practical significance, especially in the robot learning scenario as the paper mentioned. The combination of reward-free exploration is also well justified. It would be more interesting to see if the proposed algorithm can indeed be adapted to the robot learning scenario, if possible. \n\n3. The performance of the algorithm, especially on the atari game benchmark, is competitive. Although the setting of games may not be a very good fit for the deployment efficient setting, it also shows that the proposed algorithm can indeed be a good algorithm for just performing reward-free exploration. It would be interesting to see any comparison to algorithms with explicit exploration bonuses. \n\n## Weakness\n\n1. The paper is not very well-origanized. Some of the results are not crucial, and there are many unclear parts and unjustified components of the algorithm/derivation. Details will be listed in the questions and limitations section. \n\n2.  The idea of learning a global accurate dynamics model, with a number of exploration policies based on the current dynamics models, in the reward-free exploration section, is not new (c.r. the idea of policy cover mentioned above). There are many theory papers that considered the similar scenario and proposed algorithms with similar intuition, for example, [1,2], which in fact undermines the novelty of the paper. \n\n### references\n\n[1] Agarwal, Alekh, et al. \"Flambe: Structural complexity and representation learning of low rank mdps.\" Advances in neural information processing systems 33 (2020): 20095-20107.\n\n[2] Modi, Aditya, et al. \"Model-free representation learning and exploration in low-rank mdps.\" arXiv preprint arXiv:2102.07035 (2021).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8EaGV2Z9uk", "reviewer": "Reviewer_zB57", "summary": "This paper discusses a common problem in DRL, w.r.t, low efficient exploration in sparse reward tasks, and poor generalization of the trained agent. The authors propose CASCADE that learns to build a world model following a self-supervised exploration strategy. The naive motivation of CASCADE is to theoretically improve the learning objective of Plan2Explore to be more generalized. The results of the empirical evaluation show the effectiveness of the proposed method. ", "questions": "**Concerns and Questions**\n\n1. The pseudo-code leads to misunderstanding. There is only one exploration policy in algorithm 1, which does not match the main contribution of this paper. The input of algorithm 1 should be multiple policies, which can be more explicit and reinforce the proposed method's main difference from the others.\n\n2. As mentioned in Line.123, the embedding of the representation is the final state. I wonder what the advantages of doing this are. It seems that there are multiple MDPs existing. Why not embed the representation in terms of similarity or uncertainty.\n\n3. There is a typo in Line 136. The proof is in Appendix C.1.1.  \n\n4. There lacks an explanation of why it is necessary to extend eq.2 to the population-based version (eq.3).\n\n5. It would be great to include the evaluation results of CASCADE in more complicated environments (as mentioned above). It does not matter even if the trained policy's performance is not very good.\n", "limitations": "**Limitations and potential negative societal impact**\n\nN/A\n\n**Reference**\n\n[1] Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., and Funkhouser, T. (2017). Semantic scene completion from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1746–1754.\n\n##upload after the rebuttal##\n\nConsidering that the authors have conducted a lot of work during the rebuttal, the added experiments reinforce the empirical evaluation parts of the paper in a way. I raise my score by 1. ", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. This paper is well structured and easy to read. The motivation is easy to understand, and I agree with it. It seems that the authors are not trying to over-sell the contributions, which is good.\n \n2. The theoretical proof is clear, and there is no mistakes in term of notations.\n\n3. The representation of the experiment results is clear, and it is clear to find how CASCADE works.\n\n4. This paper includes a strong background introduction and related research.\n\n**Weakness**\n\n1. I encourage the authors to improve the part of the abstract. I cannot tell precisely the novelty and necessity of CASCADE in the first reading.\n\n2. The experiments are not strong enough to demonstrate the advantages of the proposed method. I was expected to see CASCADE to work in more complicated environments, e.g., the SUNCG dataset (refer to [1]).\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept