Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel unsupervised skill discovery algorithm. Beginning with the family of such methods (DIAYN, VIC, etc) that learn a discriminator to distinguish skills given some observations of the trajectory and a policy that executes a skill conditioned on the (discrete) skill random variable Z, the premise of the paper is that such methods on their own will fail when new states are encountered during the skill learning process as the discriminator would not have had sufficient data to learn to distinguish novel states. The paper proposes a novel reward bonus that works in addition to a base method such as DIAYN (DIversity is All You Need), such that this bonus “reimburses” the policy for visiting states where the discriminator uncertainty is high (measured using a form of disagreement across an ensemble of discriminators). Experiments on the pedagogical 4-room environment and the Atari suite of environments demonstrates the benefit of the proposed reward bonus in not only learning more skills (or “empowerment” in the VIC nomenclature), but the learnt skills are also superior for downstream tasks (external reward) and lifetime state coverage.
main_review: ## Strengths

1. The paper is well motivated. The shortcomings of existing unsupervised skill discovery algorithms are clearly explained along with how the proposed method overcomes these shortcomings.

1. The derivation of the proposed objective is intuitively appealing and theoretically sound, the final form of the reward bonus is a simple one (a desirable trait) -- entropy of the mean discriminator minus the mean entropy of the ensemble of discriminators.

1. The paper does a very good job in explaining all the hyperparameter and modeling choices made in their experiments and reproduction of baselines.

1. The paper does a good job at thoroughly analysing the pedagogical 4-room setting as well as all 57 Atari game environments.

1. The results in the 4-room and Atari environments are both positive in terms of the three metrics evaluated -- number of skills learned, transfer of skills to downstream tasks and lifetime state space coverage. Appropriate ablations are compared to in all cases, specifically, 

## Weaknesses

1. Unlinke prior work (DIAYN), the paper restricts the choice of environments to the 4-room and Atari environments. Given that DIAYN and subsequent works (e.g. DADS [1]) have shown skill learning in continuous control tasks such as in the MuJoCo locomotion environments, it would be good to see results of the presented experiments in such domains.

## Other comments and feedback

1. Figure 2 does not do a good job in conveying meaningful information about the methodology. For 2(a), the notation is dense and hard to parse, the different colored boxes for each component don’t help. I don’t see the need for Figure 2(b) just to explain the disagreement objective from the ensemble, this was better conveyed in text. Dropping this figure may be a good idea.

### References:

[1] Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019). Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657.


================
## Post-rebuttal review update

The authors have addressed the concerns raised by all reviewers and I will maintain my score of 8 (Accept).

summary_of_the_review: The paper is well motivated, provides a simple and theoretically sound novel reward bonus for overcoming the stated “pessimistic exploration” problem and demonstrates positive results across the board for the 4-room and the 57 Atari game environments. No experiments are presented on continuous control tasks such as MuJoCo locomotion environments, which would have strengthened the paper further.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper is concerned with unsupervised RL where an extrinsic reward signal is not available. The objective is for the agent to master the environment by exploring it while learning a diverse set of skills. This is done by simultaneously training a policy conditioned on a latent variable and a discriminator that tries to infer the latent variable from trajectories. The authors identify that the intrinsic rewards used in skill discovery result in the agent being pessimistic towards exploring novel parts of the environment. To alleviate this, the authors propose a new auxiliary objective, which results in a bonus based on the disagreement of an ensemble of discriminators. Empirical results on the grid world and Atari show improvements in skill discovery and solving downstream tasks compared to baselines.
main_review: **Strengths**

The paper is well-written and I enjoyed reading it. The authors clearly explain the discriminator-based skill discovery via intrinsic rewards given in (3) and why it results in a pessimistic agent. In skill discovery, a low reward is given when the discriminator is poor which happens in the face of unseen states. This is contradictory to the UFO principle for exploration and thus motivates the authors to provide an auxiliary bonus. The discussion on page 4 regarding maximizing the lower bound and ensuring its tightness is interesting. The new exploration method is simple and practical and is shown empirically to improve skill discovery.

**Limitations/Comments**

The problem considered in this paper is closely related to the reward-free pure exploration setting. For that setting, approaches based on maximum entropy exploration have been proposed, such as

Zhang, Chuheng, Yuanying Cai, and Longbo Huang Jian Li. "Exploration by Maximizing R\'enyi Entropy for Reward-Free RL Framework." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

Comparison with these methods is not provided in the paper.

As the authors acknowledge, it is not clear whether the number of learned skills in (4) is a good performance metric. A better metric can be similar to the reward-free paradigm, which is being able to solve any task if the reward function is provided.

In empirical evaluations, the authors compare with RND which is a common pseudocount-based method. However, several other exploration methods are proposed that significantly improve over RND:

Seo, Younggyo, et al. "State entropy maximization with random encoders for efficient exploration." arXiv preprint arXiv:2102.09430 (2021).

Raileanu, Roberta, and Tim Rocktäschel. "RIDE: Rewarding impact-driven exploration for procedurally-generated environments." arXiv preprint arXiv:2002.12292 (2020).

Zhang, Tianjun, et al. "MADE: Exploration via Maximizing Deviation from Explored Regions." arXiv preprint arXiv:2106.10268 (2021).
summary_of_the_review: The motivation behind the method is well-justified and the approach is interesting and practical. The discussion on pessimism in skill discovery is insightful. The paper can be improved by further comparison with methods that are not focused on skill discovery.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper identifies a source of pessimism in DIAYN-style methods for exploring new parts of the state space. They argue that this issue is due to using a single point estimator as a discriminator and that capturing the epistemic uncertainty of the discriminator could serve as an additional signal to guide exploration. They achieve this by using an ensemble of discriminators and incorporating the epistemic uncertainty across the ensemble into an additional intrinsic reward to the diversity of skills reward (through a mixing parameter $\lambda$). They examine this method against DIAYN-style methods and count-based methods and show that this new approach broadly outperforms both these classes of methods.    
main_review: ## Strengths:
- An important problem in unsupervised skill discovery through info-max methods (such as VIC and DIAYN) is identified. 
- A simple and intuitive approach is proposed to fix the issue. 
- The problem and the solution method are both well-motivated.
- Experiments (featuring a standard toy problem --- Four Rooms domain --- as well as pixel-based Atari domains) show strong support for the effectiveness of the solution method. 
- Justifiable sets of baselines are used in each environment. 
- Presentation is clear and provides a good summary of prior work. 

## Weaknesses:
- Discussion of limitations is only included in the supplementary material and should be brought to the main text and discussed earlier on.
- Given the higher variability per seed, why do none of the plots shows variance across seeds?
- Also, not sure why only 3 seeds are run in Atari? Considering this higher variability, shouldn't a larger number of seeds be used? For DQN, the standard is 5 seeds and DQN already shows very low variability across seeds.  
- It is unclear how the authors have realized the sufficiency of not using bootstrapping for training the ensemble (i.e. they use the same mini-batches across the ensemble) --- see my Question 1. Also, not clear how much this choice is responsible for higher variability across seeds?  
- It would have been nice to see a qualitative analysis of learned skills in Atari. Is that not something that can be done quite easily?
 

## Additional questions: 
1. On page 4, it is stated that the same mini-batches are used to train the discriminators within the ensemble and that this was found to be "*both sufficient and simpler*". Regarding sufficiency: (i) how did you experiment to find out bootstrapping vs. same mini-batches yield similar performances? (ii) wouldn't this, in the long run, be problematic/limiting (in a general sense)?

2. In Sec. 5 it is mentioned that this work is the first to apply ensemble learning and epistemic uncertainty capturing to unsupervised skill discovery. However, curiosity has been combined with ensemble models before (Pathak et al., 2019). Also, curiosity has been used to acquire skills by snapshotting (Ref. [1]). While I agree that in some senses the statement in the paper is reasonable, I think this general statement confuses more than it informs. Wouldn't it be more clear if this was stated in the specific context of *diversity-seeking* unsupervised skill discovery?   

## Minor:
- Fig. 4: Legend seems out of place (it's included in part (b), while it's least useful in the context of part (b)).
- Colors in Fig. 4 are too close (RND and Ensemble are difficult to distinguish in part (a)). 

### References: 
[1] Groth, O. et al. (2021). Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration. *arXiv 2109.08603*.
summary_of_the_review: The paper introduces and solves an important problem, proposes a technically sound method, has a very clear presentation of related works, ideas, and results, and offers good experimental evidence for the claims.

Nevertheless, I believe certain clarifications/additions could improve the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper adopts the framework of unsupervised skill learning. To solve the problem that the discriminator will have low confidence in the unseen data thus providing a low intrinsic reward, the paper derives an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement. The paper conducts extensive experiments on tabular grid world and 57 games of the Atari Suite. 
main_review: Strength: 
The strengths are summarized below:

1. The writing flow of the paper is very nice, including comprehensive background, easy-to-follow methodology. It would be better to pair with some nice figures.
3. The experiments are pretty solid including various settings environments including tabular form and image-based tasks (Atari). Experiments in Atari are very solid.

Weakness:
I would like to raise several questions to the paper: 

1. Can the author provide some experimental supports for the argument that the reward provided by the discriminator is pessimistic in face of uncertain states? I am curious how the reward looks like in Eq. 3 when using functional approximation. 

2. As also noted in the paper, the idea of using the ensemble method to encourage exploration has been widely employed in the general RL setting. The proposed intrinsic reward is very similar to those adopted previously, despite the setting being unsupervised skill learning. Thus, I think the algorithm itself is not very novel.

3. In terms of experiments, I think the author should add more comparison between the proposed method and more baselines: for example, a direct comparison would be RND + ensemble. 

4. I didn't see the learning curves for downstream tasks (maximize reward) of Atari in the appendix. The author can provide the learning curve for (d) in figure 4. 

Clarity: The paper is well-written and clear in the flow.

Feedbacks & Questions: Please see details in the weakness.
summary_of_the_review: Overall, I think the paper is well written and has extensive experiments. My main concern is about the novelty of the method. These concerns are pretty important and thus I encourage the author to engage in the discussion period and clarify these if there is any misunderstanding. I am happy to re-evaluate if the author convinces me.
review_title: null

Meta-review: This paper tackles the problem of exploration using intrinsic rewards in RL in states that have never been encountered before. The authors derive an information gain auxiliary objective that involves training an ensemble of discriminators and rewarding the policy for their disagreement, which estimates the epistemic uncertainty that comes from the discriminator not having seen enough training examples. The intrinsic reward resulting from the so-called DISDAIN (discriminator disagreement intrinsic reward) exploration bonus is more tailored to the true objective compared to pseudocount-based methods.

Reviewers agree that the paper is well-motivated and well-written, that the proposed DISDAIN exploration method is simple and practical, and that experiments are convincing. Experiments on continuous control tasks such as MuJoCo locomotion environments could have strengthened the paper further.
Decision: Accept (Spotlight)