Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work introduces a new problem setting, Reward Free Deployment Efficiency focusing on two incentives: 1. Task agnostic exploration facilitates generalization. 2. Exploration policies that can collect large quantities of data without centralized retraining facilitate scalability.

In addition, this work introduces the Coordinated Active Sample Collection via Diverse Explorers (CASCADE), which can gather a diverse set of data and is inspired by Bayesian Active Learning.

In this work, B exploratory agents are trained in parallel. At each deployment, a loss containing two parts is maximized; the first part is a diversity term between the agents' behaviors. And the second part is the so-called information gain. 

The authors have used the DreamerV2 agent as the base agent for their work and their baselines. Furthermore, they used the well-known Plan2Explore (P2E) and a modified version (Population Plan2Explore, PP2E) as their baseline. Finally, they have produced experiments to show how well their proposed method can explore the state space and also perform with a zero-shot manner to the task-specific problem.
strengths_and_weaknesses: Originality:
This work introduces a new setup that is called reward-free deployment efficiency. Furthermore, inspired by the P2E agent, the authors propose a new method called CASCADE that can perform decently compared to the previous baseline in this setup.

Quality:
The authors have done an excellent job explaining their setup, proving their claims, and discussing their experiments.

Clarity:
The paper is well-written which prevents a familiar reader from doing additional passes through different paragraphs.

Significance:
Despite the fact that the proposed method, CASCADE, is well-motivated, I do think that there are few arguments about why the reward-free deployment efficiency matters. I can assume real-world scenarios where access to about 20 different agents is not possible.

questions: 1. Why is rewarding-episodes being considered in a reward-free setting? (This is a bit confusing)
2. Would it be possible to have the final performance of the P2E (not in the deployment efficiency setting) reported as a dashed line? This is informative in case one wants to see how difficult the proposed problem setting is.
3. Could you elaborate more on this? 

>to facilitate scalability, exploration policies should collect large quantities of data without costly centralized retraining.

 I can easily imagine real-world scenarios where access to about 20 different agents is not possible.

limitations: The authors have addressed the limitation of their work in section 4.3.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper discusses a common problem in DRL, w.r.t, low efficient exploration in sparse reward tasks, and poor generalization of the trained agent. The authors propose CASCADE that learns to build a world model following a self-supervised exploration strategy. The naive motivation of CASCADE is to theoretically improve the learning objective of Plan2Explore to be more generalized. The results of the empirical evaluation show the effectiveness of the proposed method. 
strengths_and_weaknesses: **Strengths:**

1. This paper is well structured and easy to read. The motivation is easy to understand, and I agree with it. It seems that the authors are not trying to over-sell the contributions, which is good.
 
2. The theoretical proof is clear, and there is no mistakes in term of notations.

3. The representation of the experiment results is clear, and it is clear to find how CASCADE works.

4. This paper includes a strong background introduction and related research.

**Weakness**

1. I encourage the authors to improve the part of the abstract. I cannot tell precisely the novelty and necessity of CASCADE in the first reading.

2. The experiments are not strong enough to demonstrate the advantages of the proposed method. I was expected to see CASCADE to work in more complicated environments, e.g., the SUNCG dataset (refer to [1]).




questions: **Concerns and Questions**

1. The pseudo-code leads to misunderstanding. There is only one exploration policy in algorithm 1, which does not match the main contribution of this paper. The input of algorithm 1 should be multiple policies, which can be more explicit and reinforce the proposed method's main difference from the others.

2. As mentioned in Line.123, the embedding of the representation is the final state. I wonder what the advantages of doing this are. It seems that there are multiple MDPs existing. Why not embed the representation in terms of similarity or uncertainty.

3. There is a typo in Line 136. The proof is in Appendix C.1.1.  

4. There lacks an explanation of why it is necessary to extend eq.2 to the population-based version (eq.3).

5. It would be great to include the evaluation results of CASCADE in more complicated environments (as mentioned above). It does not matter even if the trained policy's performance is not very good.

limitations: **Limitations and potential negative societal impact**

N/A

**Reference**

[1] Song, S., Yu, F., Zeng, A., Chang, A. X., Savva, M., and Funkhouser, T. (2017). Semantic scene completion from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1746–1754.

##upload after the rebuttal##

Considering that the authors have conducted a lot of work during the rebuttal, the added experiments reinforce the empirical evaluation parts of the paper in a way. I raise my score by 1.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: # New setting for reinforcement learning

The paper first introduces **reward-free deployment efficiency**, a new setting for reinforcement learning.

The *deployment efficiency* part means that we limit the number of agent versions that we deploy in the environment to collect new experience (for scalability reasons). 

The *reward-free* part means that the collection procedure is task-agnostic. It aims at exploring the state-action space as much as possible without reward consideration (for generalization reasons).   

# New algorithm within this setting

Within this setting, they propose **CASCADE**, an algorithm to deploy a fleet of *coordinated* agents that gather task-agnostic experience in the environment. 

Similarly to [Plan2Explore](https://arxiv.org/pdf/2005.05960.pdf), the exploring behavior is trained with an intrinsic reward that estimates the uncertainty of a world model (they use DreamerV2 world model). This objective has an information-theoretic interpretation.  In the same way as Plan2Explore, they provide reward labels at test time, after data collection, to train a reward predictor for the world model. They can then train a task-specific agent in the imaginary MDP defined by the world model. 

The major difference is to have different exploration policies and to *coordinate* them, so that they can organize together to explore different parts of the world.

They provide a theoretical motivation for this diversity of behaviors on a tabular version of their method (CASCADE-TS). 

Their experiments demonstrate the superiority of CASCADE over random exploration and Plan2Explore, on MiniGrid, Montezuma's Revenge, and 4 tasks in DeepMind Control (stand, walk, run, flip). 


strengths_and_weaknesses: # Strengths

- Clear demonstration of the ability to explore the world in MiniGrid toy environments and Atari Montezuma's Revenge, with clear visualizations.

- Interesting improvements over an initial dataset of demonstrations with very few deployments in DeepMind Control.

- They provide a theoretical motivation behind the intuition that it is better to have diverse exploration policies.

--- 

# Weaknesses

## Lack of experimental evidence

The experiments about world exploration (4.1) are in toy environments of MiniGrid (FourRooms and MultiRoom) and in only one Atari game (Montezuma's Revenge). Running all Atari might be too much, but some Atari games present interesting world exploration problems (Hero, Freeway, Frostbite for instance). 

Also, [Crafter](https://github.com/danijar/crafter) would be a much less trivial environment to benchmark the capabilities of CASCADE: procedurally generated world with many assets and interesting behaviors to discover.

---

## Deployment-efficiency limits comparison to prior work

The deployment-efficiency setting seems a bit artifical here. Indeed, it is justified in the text by scalability reasons, that are important when it comes to real world problems, but the environments considered in this work are classical toy environments. 

Introducing a restriction on the number of deployments prevents proper comparison to prior work. In particular, it would have been really interesting to compare CASCADE with Plan2Explore (also reward-free / task-agnostic) by using their number of deployments and all of their DMC environments (a superset of the ones considered here). 

Introducing this new setting (which is not justified by any real world environments) is risky since the comparison / novelty / interest are much harder to assess with a change in the experimental setup.

---

## The formalism is sometimes very unclear

In particular, L111 to L125, the information theoretic objective taken from Plan2Explore (Equation 1) is confusing. 

First, looking at [Plan2Explore](https://arxiv.org/pdf/2005.05960.pdf):
- The intrinsic reward is to collect experience that maximizes the world model uncertainty, and they estimate this uncertainty via the empirical variance of an ensemble of next WM-latent predictors. 
- They clearly state that this objective only *approximates* an information gain, and they frame it as a nice information theoretic interpretation of their objective. 

Here, equation 1 states that the exploration policy is the argmax of the mutual information term, which is confusing since it is not directly optimized in Plan2Explore but approximated. Also, L116 mentions several MDPs to explain the mutual information term, who are they? In fact, in Plan2Explore, it is clear that there is a *single* imaginary MDP (defined by a single world model), and that they consider the disagreement in the ensemble of predictors, all trained to predict the dynamics of the same MDP dynamics. The explanation with several MDPs does not seem to make sense. This ensemble of predictors seems to be the central piece of the "information theoretic" objective, but is only mentioned in Appendix B as an implementation detail.

Also, in Plan2Explore, the mutual information is taken with a random variable $w$ that represent the optimal dynamics parameters, in a Bayesian way. Here, in equation 1, what is the mutual information between a distribution of states and an imaginary markov decision process? 

---

## Theory is more a motivation than a guarantee

From what I understand, they prove that is is better to have diverse explorers (Lemma 1) and that CASCADE-TS (tabular version of CASCADE) learns faster or similarly to a naive fleet of identical explorers. However, there is no guarantee that CASCADE-TS actually converges to a solution of the optimization problem. 

The name of section 3.3 is clear on that point ("Theoretical Motivation") but the abstract claim is not ("theoretical guarantees").

---

## The writing could be more polished

Some typos, and the notations could be more rigorous (e.g. L92 $\rho_0$ not defined, L137 $e$ is not defined).

---

# Originality

Originality is not clearly a strength or a weakness here. The work relies on the formalism of Plan2Explore and the code of DreamerV2, but the added term to maximize diversity among explorers in order to achieve better exploration / zero-shot generalization is interesting. 

questions: 1) The `while True` in Algorithm 1 is slightly confusing since the deployment efficiency is precisely to limit the number of iterations of this loop, which here is infinite. Why couldn't you use a `for` loop with the number of deployments instead? That would allow to explicitly mention the number of deployments and precise that it is supposed to be small.

2) L97, could you explain why updating $\pi_{EXP}$ using the uncertainty of the imaginary MDP is different from Plan2Explore? 

3) Figure 3, in FourRooms, how is it that P2E, with a lower state coverage and less rewarding episodes found, seems to beat PP2E on zero-shot success rate? 

4) Still in Figure 3, could you explain more why CASCADE consistently reaches 100\% zero-shot success rate, while its number of rewarding episodes and state coverage are very similar to PP2E?

5) Figure 5, I would expect the 4 methods to be ranked randomly at deployment 0 (they all have access to the same data to start with), but across 30 seeds and 4 tasks, it looks like CASCADE is above the other ones even at deployment 0. Is there any reason for that?

6) Would it be possible to address the weaknesses pointed out in the "Strength And Weaknesses" sections? In particular, experiments on more Atari games / Crafter, comparison to Plan2Explore with their number of deployments and environments, and clarification of the formalism.

limitations: Section 4.3 clearly mentions some limitations. In particular, the authors make it clear that the learnt zero-shot policies are far from being optimal, which is understandable considering the difficulty of the reward-agnostic experience collection.  

They do not discuss the potential negative impacts of their method, but clearly explain why the "deployment efficiency" constraint is useful in terms of security and cost.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a setting called reward-free deployment efficient setting, where one can interact with the environment with limited number of times/limited number of updates (but could with multiple agents in parallel), in the reward-free fashion. The paper proposes to solve this problem by learning a world model with a network ensemble, and with a certain number of exploration policies. The paper presents to train the world model in a similar fashion as DreamerV2, and train the exploration policies with a composition of heuristic objectives. Finally, the paper evaluates the effectiveness of the algorithm on both game and locomotion benchmarks. 
strengths_and_weaknesses: ## Strength

1. The idea of using a set of exploration policies is a crucial contribution, which also ensembles the idea of "policy cover" used for exploration in multiple theory papers. The paper provides some good intuition on why using beyond one fixed policy is important for exploration, such as Figure.2. However, the theory component does not seem very surprising, which will be more detailed explained in the latter part of the review. 

2. The new reward-free deployment efficient setting indeed has some practical significance, especially in the robot learning scenario as the paper mentioned. The combination of reward-free exploration is also well justified. It would be more interesting to see if the proposed algorithm can indeed be adapted to the robot learning scenario, if possible. 

3. The performance of the algorithm, especially on the atari game benchmark, is competitive. Although the setting of games may not be a very good fit for the deployment efficient setting, it also shows that the proposed algorithm can indeed be a good algorithm for just performing reward-free exploration. It would be interesting to see any comparison to algorithms with explicit exploration bonuses. 

## Weakness

1. The paper is not very well-origanized. Some of the results are not crucial, and there are many unclear parts and unjustified components of the algorithm/derivation. Details will be listed in the questions and limitations section. 

2.  The idea of learning a global accurate dynamics model, with a number of exploration policies based on the current dynamics models, in the reward-free exploration section, is not new (c.r. the idea of policy cover mentioned above). There are many theory papers that considered the similar scenario and proposed algorithms with similar intuition, for example, [1,2], which in fact undermines the novelty of the paper. 

### references

[1] Agarwal, Alekh, et al. "Flambe: Structural complexity and representation learning of low rank mdps." Advances in neural information processing systems 33 (2020): 20095-20107.

[2] Modi, Aditya, et al. "Model-free representation learning and exploration in low-rank mdps." arXiv preprint arXiv:2102.07035 (2021).

questions: ## Major Questions

1. In Figure 1, the top two rows seem identical. Thus there are a lot of redundancies in the figure and in fact seems a little bit strange. Also, the definition of deployment seems to be the number of interactions that the algorithm itself will have with the environment? Thus the number of deployments of offline RL should be 0? Otherwise, if the offline dataset is collected from a heterogeneous source, the number of deployments of offline RL could also be large?

2. The definition of contextual mdp (CMDP): according to [3], the definition of CMDP seems to be that we have a set of mdps, where each mdp, in addition to have different reward, but also have different dynamics conditioned on the context?

3. The derivation of the final objective is very confusing. In the main text, the ImagDiv term seems to be a heuristic surrogate of the first entropy term in line 140. Under the assumption that the conditional latent distribution is a gaussian with variance unrelated to the policy, which seems to be the assumption made for section 3.2, the ImagDiv objective is actually estimating the whole objective? However, in the appendix, the paper also presents the derivation for the InfoGain term, under the same assumption. This is confusing because the paper tries to use two different objectives to estimate the same thing?

4. In the definition of ImageDiv term, is the expectation also over the model ensembles? Also $\Phi$ is not defined in the ImageDiv term but is defined in the appendix for the InfoGain term. Is $\Phi$ in ImagDiv the same as $\mu(m_{\psi}\pi)$?

5. In Fig.3, why is PP2E10's zero-shot success rate higher in a more challenging environment?

6. In the experiment for locomotion, why do we need to assume that we have access to the offline dataset? It is known that offline RL needs good coverage from the offline dataset to perform well, thus it looks like in this setting the algorithm already gets a good warm-start? How would the method perform without the offline dataset?

### references

[3] Hallak, Assaf, Dotan Di Castro, and Shie Mannor. "Contextual markov decision processes." arXiv preprint arXiv:1502.02259 (2015).

## Minor issues

- In algorithm 1, it seems like the algorithm will never terminate?

- Many hyperlinks to the sections in the appendix are wrong. For example, line 136.

- Line 140, on the right hand side of the last equality, in the second entropy term, the distribution of latents should be conditioned on $\tilde \pi^{(j)}$'s but not $\pi^{(j)}$'s (the subscripts).

- The x-axis in Fig.5 is different from the text description. 
limitations: 1. The result in Lemma 1 assumes deterministic MDP and deterministic policy. Would it be obvious that one policy may not be able to explore the whole environment?

2. Section 3.3 seems to be in an awkward position. After a very heuristic description of the deep RL approach, section 3.3 returns to a theoretical analysis of a variant of the algorithm, and the setup is also simpler (tabular MDP). Maybe it could instead serve as a motivating section?
review_title: null

Meta-review: metareview: This paper proposes a method to learn world models without rewards, using a collection of agents that explore an environment. The key idea is to maximize diversity between the trajectories collected by the agents to obtain a good world model, with an emphasis on being as efficient as possible. The authors present some theoretical justification for using a population of agents and their empirical results on several datasets provide a good demonstration of the method. The reviewers all agree this is an interesting and important setting and the author response significantly improves the paper on aspects of clarity and empirical results, based on the reviewer concerns. Overall, I believe this work provides interesting ideas and will encourage more work in this direction in the future. I encourage the authors to revise their paper taking the reviewer suggestions into account and add in the new experiments to make it stronger.
confidence: Less certain
Decision: Accept