Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a method for unsupervised skill discovery using learned reward functions. Pairs of reward function and policy pairs are iteratively learned. The reward function evolves by predicting negative rewards for states the agent has visited and positive rewards for states just outside the agent's reach. The policy is then transferred to the new reward using three forward skill transfer methods.

This new method is the first unsupervised skill discovery approach to work in high-dimensional environments without expert knowledge. The paper first analyzes a 2d maze task. Next, the paper shows results in continuous control tasks and compares to 3 baselines. Finally, the paper shows results on Montezuma's revenge from pixel inputs.
strengths_and_weaknesses: Strengths: 
* To the best of my knowledge, learning a reward function for unsupervised skill discovery is novel. I believe this is an important step forward from prior work that relies on more hand-crafted reward objectives. 
* Unlike other skills discovery methods like DIAYN, the proposed method does not require engineering only part of the observation space as input for good performance.
* Method demonstrates superior performance over relevant unsupervised skill discovery algorithms both in terms of the MI metric.
* Qualitative analysis demonstrates the behavior of skills learned in all three benchmarks.
* The algorithm does not need a specified number of skills ahead of time. New skills can continuously be added.

Weaknesses:
* While this weakness was also acknowledged in the limitations section, the proposed method learns skills in a depth first rather than breadth first way. This narrow search of skills could result in lacking skill diversity. Imagine a version of the 2d maze environment where there are multiple possible paths. I think the proposed method would find skills that only go along one path, but never discover skills down the other path because the skills progressively build off each other.
* Several design decisions are not quantitatively evaluated. The impact of adaptive entropy regularization and the three forward transfer methods are not ablated. Furthermore, there are several arbitrary hyperparameters related to the guiding phase like the duration of the guiding phase and the time spent collecting negative actions.
* Only the continuous control results include a comparison to baselines. It is unclear if the method helps in other tasks like maze navigation or Montezuma's revenge. 

questions: * Why use a regression loss for training the reward function? Isn't this a classification problem for distinguishing the already visited negative states from the new positive states?
* I do not understand the justification for clipping the rewards to [0,a]. Could the negative rewards be important penalties for avoiding exploring parts of the state space that were already visited? 
* How does the training compare to the baselines from Gu et al. 2021? Do these baselines also do such extensive hyperparameter tuning to make this comparison fair? 
* What is the strategy used to adapt the number of training steps to ensure each skill has been learned in Montezuma's revenge?
limitations: The authors mention limitations of the work with the potential for better forward transfer methods and using a population of skills rather than the narrow deep search strategy.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a method for unsupervised skill discovery via reinforcement learning. The method involves training a sequence of neural network-based reward functions, and using RL to learn policies that optimize these reward functions.  The reward functions are trained using positive and negative examples, where negative examples are the states reached by current policy, and positive examples are states reached by executing random actions. This encourages learning skills that cannot be visited by the current policy (and all previous policies). The authors apply this method for learning dozens of skills in simulated physics environments (half-cheetah, humanoid etc.), and hundreds of skills in Montezuma’s revenge. The skills often (but not always) recover interesting behaviors, including a humanoid hopping on one leg and the agent in Montezuma’s revenge being able to explore new rooms (beyond the first room). 
strengths_and_weaknesses: Strengths
- Simple, straightforward method. 
- Impressive results in a few different domains. 
- Clear writing and presentation. 

Weaknesses
- The authors only show results on zero-shot transfer (i.e. how well a learned skill performs on some standard reward function), and there are no experiments on using the learned skill(s) for quickly learning a new task (i.e. improving beyond the pre-trained skill).
- The method is prone to learning uninteresting skills / getting stuck in local minimas. For example, in the experiments for Montezuma’s revenge, the authors noted that one of the first skills learned by the agent was to get the life counter to zero, which made it very difficult to do any further exploration (since the episode can terminate very easily once the life counter is at 0). This led the authors to manually intervene in the learning process and hide the part of the screen that shows remaining lives.

questions: Questions
- “In expectation, 7 · 10^27 episodes are needed to do so. This shows that this maze is difficult to navigate.” How did the authors arrive at - this number? 
- The ablations were only performed for the 2D navigation domain. Is there a reason why they were not done for the robotics domain? Since the two domains are fairly different, it is not necessary that all conclusions from the 2D navigation experiments will hold. 
limitations: The paper lacks section on limitations (there is maybe only one or two lines in the conclusion that mention limitations, and they only do so in the context of future work that could be done). I would encourage the author to come up with a more comprehensive discussion of limitations (I mention some in the weaknesses above, but I would guess the authors are probably aware of more limitations).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies the problem of unsupervised reinforcement learning, where an agent aims to explore its environment without a reward signal and learn diverse skills. The proposed approach amounts to learning a reward function which encourages exploration, and a policy with A2C to maximize this reward. The exploration reward function itself is essentially a classifier trained to predict visited states (negatives) from states near the boundary of the visited states (positives). The policy is trained to maximize the score (go near the boundary). 

To recover "skills", the policy checkpoints at various stages of training are used. At each phase, the reward function then policy are trained using the newly collected data. To stabilize learning, the policy/value function are reused from the previous skill in learning the new skill. 

Experiments show this can explore a maze environment as well as ant/humanoid environments. The proposed approach outperforms DIAYN in terms of mutual information between skill and state-distribution. 
strengths_and_weaknesses: *Strengths*
- The paper studies an important problem in unsupervised RL and skill discovery. 
- Expanding the policy to cover the state space is a well motivated objective for unsupervised RL.

*Weaknesses*
- First, the way the proposed method decomposes "skills" is a bit unclear. From what I can tell, the method is training a reward to explore the boundary of states, and a policy to maximize that reward, both being updated online. Then "skills" as denoted in this work are just the policy/reward at different checkpoints in training. So it is less that the method learns a multi-skill conditioned policy, but rather it is a collection of checkpoints (which get manually selected during downstream 0-shot adaptation). 

If this is the case, then the claim in the related work that intrinsic motivation papers are not comparable is not quite accurate.
"Another related line of research to our approach is intrinsic motivation (Stadie et al., 2015; Bellemare et al., 2016; Pathak et al., 2017; Burda et al., 2018b,a; Raileanu & Rocktaschel, 2020). These approaches have managed great success in hard-exploration Atari games. However, these approaches do not learn discrete skills that can be composed or fine-tuned for fast learning of new tasks."
Any of these works could also produce skills via the same checkpointing procedure used here and thus are valid comparisons. 

- Moreover, the core exploration objective proposed in this work is not particularly new compared to prior work. Theres a large literature of works that use state-space coverage as their exploration+skill learning objective, including model error/disagreement [1,2,3], pseudo counts [5,6], state distribution matching [7], and exploring the boundary of states [4,8] to name a few. While certainly not all are necessary, the paper should at least compare to some of these baselines, and currently none are. And the paper should do a better job of explaining how exactly their instantiation is different from what is being proposed in these works.

1. Pathak et al. Curiosity driven Exploration by Self-Supervised Prediction. 
2. Pathak et al. Self-Supervised Exploration via Disagreement. 
3. Sekar et al. Plan to explore vis Self-Supervised World Models
4. Mendonca et al. Discovering and Achieving Goals via World Models
5. Tang et al. # exploration: A study of count-based exploration for deep reinforcement learning.
6. Burda et al. Exploration by Random Network Distillation.
7. Lee et al. Efficient exploration by state marginal matching.
8. Bharadhwaj et al. LEAF: Latent Exploration Along the Frontier. 

- Finally, there are a number of design choices in the proposed approach that are unclear. For example, why does random exploration from the current policys states produce a good set of positive examples for exploring beyond it. I can think of cases (like going through a narrow gap), where such exploration will not hit the states relevant for pushing the boundary. Approaches like those used in [4,7,8] seem like they could be more effective are identifying the boundary to explore.

questions: - If the policy at different stages of training are taken as "skills", why are all of the listed intrinsic motivation papers not valid comparisons?
- How does the proposed method for state coverage differ/compare to the relevant work? There should be comparisons to at least some recent work which also does exploration+skill discovery by optimizing for state coverage.
limitations: Limitations are not really discussed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a method for unsupervised skill discovery in reinforcement learning environments. The authors propose a method which iteratively learns policies for new skills by defining new reward functions for each new skill. New reward functions are constructed by taking the visited states by the previously learned skill, as well as all previously learned skills, and setting a low reward at those states. Rewards are higher for states which are close to being reached by the previous policy. The method takes advantage of some policy and value function transfer techniques to make learning more efficient. Experiments are conducted using a 2D navigation environment, robotics environments simulated using BRAX, and Montezuma’s Revenge.

strengths_and_weaknesses: Strengths of the paper:
- The method is straightforward and appears to make fewer assumptions than other methods like DIAYN do in terms of the x-y position prior, which is a very nice property. 
- Evaluations are performed on sensible tasks and can obtain favorable results compared to DIAYN and GCRL, which is quite impressive given that the method is conceptually simple. 
- Generally the paper is clearly written; I note some suggested small changes below, but there are some other minor grammatical errors which do not impact the clarity of the text but could benefit from another round of proofreading.

Weaknesses/clarifications:
- It seems that this work falls into the category of exploration with intrinsic curiosity. The reason is that while this method does generate pairs of discrete skills and policies whereas most exploration algorithms do not, the skills learned in this work aren’t necessarily disentangled. And while the authors note that “However, these approaches do not learn discrete skills that can be composed or fine-tuned for fast learning of new tasks”, it’s not clear from this work how the discrete skills learned here can be composed or fine-tuned for completing a new task. The zero-shot transfer task investigated in the experiments requires that a *single* learned skill can solve the entire task. 
- As a follow-up to the previous point, it would be helpful to see more comparisons in the experimental section not only to additional unsupervised skill learning methods like DADS, but also intrinsic exploration methods like those mentioned in the related work section. 

questions: - Is the memory consumption of the method limiting? The reward function fitting seems to require keeping track of *all* of the previously seen negative states for all skills, which would grow as training continues. Additionally, as the number of negative states grows, does the objective in the supervised loss for the reward function need to be rebalanced for positive and negative classes?
- For Figure 2, I don’t quite understand what the meaning of the colored circles is. My understanding is that for each skill, a policy is learned which brings the agent from the top left further and further through the maze. But “The circles represent the average position of the most visited locations for each skill.” Does this mean that for a particular skill, the agent actually goes further into the maze than its circle is, and the circle just represents the average position for states visited during that trajectory?

Nitpicks/typos:
- 268: Unsupervised -> unsupervised
- Left quotes in several places in the manuscript are flipped
- Can the number of samples used to learn for Montezuma’s revenge/wall clock time be reported?

limitations: The limitations are mentioned in the conclusion, but it could be nice if the authors addressed the limitations of the current method more explicitly.
review_title: null

Meta-review: metareview: After a strong rebuttal from the authors and an extensive discussion among the reviewers, I believe the paper's pros outweigh its cons and this paper will be a valuable contribution to NeurIPS. I recommend it for acceptance and encourage the authors to address the reviewers comments for the camera-ready version of the paper, especially regarding the newly added baselines and other comparisons to SOTA approaches listed by the reviewers.

confidence: Certain
Decision: Accept