Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an approach for unsupervised learning of diverse skills by maximizing an intrinsic reward function. In contrast to prior works (eg DIAYN, DADS) the proposed approach learns skills incrementally using a separate neural network for each skills. In experimental evaluations on OpenAI gym environments this leads to (a) improved diversity in discovered skills and (b) the ability to learn skills in continually changing environments, where prior approaches struggle.

main_review: ## Strengths

- The paper combines two interesting and relevant problems: unsupervised learning of skills and learning in continually changing environments.

- The writing of the paper is clear and the method easy to understand. While a few details are missing (see questions below), the overall approach is simple and can be easily grasped from the explanations. It is also put well into context of prior works.

- The figures visualize the diverse learned skills in an intuitive and easily understandable way.



## Weaknesses

### Conceptual Weaknesses
In comparison to prior works in unsupervised skill discovery the idea of incrementally learning skills with separate networks has a few drawbacks:

- First off, the problem and goal of learning skills in continually changing environment does not seem clearly defined to me: what do we want to final set of learned skills to look like? Should they all work on the final training environment? Should they all work on all environments seen during training? Or should some work on a subset of environments while others work on another subset? What environment would we evaluate the downstream learning in? The paper does not clearly articulate these goals.

-  The requirement to learn a full new network per skill, while feasible in the simple tested environments, seems not scalable to large skill sets. At the same time it seems wasteful to train each new skill from scratch. Some form of parameter sharing between the skills seems required but the very concept of *not* sharing parameters seems to be at the heart of the proposed method and thus in conflict with these considerations.

- The iterative nature of the proposed approach requires us to define a schedule at which we start training new skills. This schedule seems to be fundamentally environment dependent and linked to the rate of change in the environment and thus requires additional human design effort which prior works that learn skills jointly do not require.

- In the setting with environment variations the formerly learned skills are not updated to work on new environments later in training. While this is not a problem in an *expanding* environment (since all skills will still be useful) it becomes an issue in environments that change in different ways, eg by disabling different legs of the ant agent. Now for a downstream task I would need to know which of the skills are still executable in this unseen downstream environment (eg which leg of the ant is currently disabled) as all other skills will not function properly. Priors works will continuously try to adapt all skills to the current environment and thus do not need this additional skill filtering supervision.


### Execution Weaknesses
The paper's execution has a few weak points, in particular in terms of the experimental evaluation:

- The main quantitative evaluation metric used is some distance between skill's end states in a projected state space. I would argue that this does not necessarily captures the diversity of skill learned to reach these states and a more objective metric is the performance on downstream tasks (since this is what we ultimately care about). While the paper does evaluate downstream learning performance, it is not evaluated on all tasks and in general the experiment seems rushed: it is only presented in the appendix and seemingly only evaluated on a single seed, which is bad practice for RL experiments. I would suggest to instead evaluate all experiments with RL on (multiple) downstream tasks and report those result as the main quantitative evaluation.

- It is unclear to me why the baselines perform that badly on the block environments in Figure 2, particularly the Off-DADS approach. I would expect it to learn still somewhat diverse trajectories in the 2D plane it is optimized to diversify (like it does in Fig 4), but instead it learns skills that always walk in the same direction. The text does not explain this phenomenon clearly. Maybe I am not fully understanding the Off-DADS objective?

- The paper argues that the baselines work better than the proposed method in the cheetah environment since they share parameters, which improves learning efficiency. If this is the case (A) shouldn't we be able to achieve the same performance by training DISk longer? (B) shouldn't parameter sharing also be beneficial for learning efficiency in the harder tasks?

- The method section of the paper does not clearly explain *when* new skills are added. The conclusion alludes to some "convergence criterion", but it would be good to add this to the main section (please correct me if I missed it in the text).


### Further Comments
- The paper follows prior works and uses a hand-defined projection of the state space into a lower-dimensional space in which it measures diversity (eg here 2D velocities for the ant agent). To me this seems like a significant assumption that can not easily be translated to more complicated tasks (what would a simple space for measuring "interestingness" of driving skills look like?). Since this is an assumption that many papers in this line of work make, I won't count it as a weakness for this particular submission.

- The paper's contribution of learning skills sequentially instead of jointly seems more largely applicable than just in the context of continual learning (although continual learning is an obvious and intuitive application). If there is a good explanation for why the incremental learning works better than the joint learning even on static environments, it might strengthen the paper's scope and clarity to present it as a full (improved) alternative to DIAYN / DADS and show the continual learning results only as one additional application that was impossible to solve with prior methods.


## Questions

- Do the baseline approaches that learn skills jointly with weight sharing get the same overall (1) number of environment interactions and (2) number of trainable network parameters?

- I wonder whether the "greedy" approach to skill learning in the proposed method (ie learning one skill after the other as opposed to all jointly) can lead to optimization issues where the first discovered skill is somehow suboptimal (eg by combines two primitives into one skills that are frequently used independently of each other in different parts of the skill space and should thus be learned as separate skills) and this suboptimal skill leads to all subsequently found skills also being suboptimal. To put it another way, I am wondering whether the problem of discovering the most optimal skill set can be solved greedily or whether it requires to be solved as a whole in order to optimally divide the behavior space into skills. I am uncertain about this question, but would be interested to hear the authors thoughts on it.

- The idea of learning separate neural networks in a sequential fashion seems applicable to continual learning problems outside RL -- are there other works that apply this idea beyond RL? It might be useful to add more discussion of approaches that train separate neural networks for continual learning in the related work?

- The paper proposes to use a non-parametric estimate for the state entropy. How does this compare to priors works who also needed to measure entropy? Could we instantiate the proposed method with the techniques used in prior works or in turn instantiate prior works with the used non-parametric estimate to understand the relative contribution of this design choice?
summary_of_the_review: In summary, I think the paper proposes an interesting problem along with a simple solution that is clearly explained. As listed above I do however have some major concerns, both conceptually and in terms of the experimental evaluation, and thus cannot recommend acceptance of the submission in its current form.

Post Rebuttal
============
After considering the authors' rebuttal I have increased my score to recommend (weak) acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a new unsupervised skill learning algorithm that combines three features: 1) incremental skill learning (i.e. learns one skill after another), 2) independent networks per skill, and 3) a nearest neighbor entropy estimator. They compare their approach to DIAYN and off-DADS in both static and dynamic MuJoCo environments in OpenAI gym. Comparisons involve skill visualizations, state coverage metrics, and reusing the skills for HRL on a goal-seeking task.
main_review: STRENGTHS

The authors present three features that each might be useful for the unsupervised skill learning literature. Their skill visualizations are well-presented.

WEAKNESSES

The current version of this paper suffers from unclear motivation, missing baselines, and missing ablations. I do believe there are good ideas here, however the paper is not ready for publication.

**Unclear motivation**: The authors motivate their approach by describing skill learning in a slowly evolving environment, where it is important to remember skills for every version of the environment along the way. This strikes me as a quite artificial setting. If the world has evolved, why wouldn't we care about the agent's performance just in the most recent "current" world? If instead the focus is indeed on performing well in many different environments, why not randomly sample one each episode? Depending on the answer to this question, it would e.g. be interesting to rerun the experiment in Figure 3 but with the broken leg resampled every episode, rather than every 1M steps.

Incremental skill learning is a good idea; so good in fact that it's been done before.

**Missing baseline #1: VALOR**: Achiam et al 2018 (https://arxiv.org/abs/1807.10299) introduced a curriculum for learning skills which introduced new skills based on discriminability performance on current skills, rather than at preset intervals. It is an important baseline as previous work, but also to allow the authors to compare performance-based and fixed-interval skill expansion.

**Missing baseline #2: DSP**: Zahavy et al 2021 (https://arxiv.org/abs/2106.00669) introduced Diverse Successive Policies, which incrementally learns new skills that maximize diversity in terms of successor features. This alternative take on fixed-interval skill expansion is also an important baseline.

The approach the authors propose simultaneously introduces three features to unsupervised skill learning, but the current experiments don't tease apart which matter. Ideally, I would like to see *each* combination of these features tried with each baseline, however that be a lot to ask and for example how exactly to adapt DIAYN to learn skills incrementally involves a lot of non-trivial choices. However, one of these features is very easy to try with each baseline.

**Missing baseline #3: all other baselines with one network per skill**: It is an unfair advantage to DISk to get to use a separate network per skill, while other algorithms do not. Each other baseline should be run as two versions - one with a shared network, and one with a network per skill. As the authors rightly identify, it may be that one or the other approach is preferred in specific environments. Additionally, the authors should emphasize more clearly that despite repeatedly using the motivation of life-long learning, an algorithm whose parameter use grows linearly in time seems problematic for that setting.

I would also like to clarify one existing baseline and one feature of DISk.

**Question: why is DIAYN so bad?** DIAYN seems to perform implausibly bad, especially in the Ant environments. For example, if DIAYN is performed on velocity-based features, wouldn't the policies in Figure 2 move? My suspicion is that the authors use velocity-based features *in addition* to the environment-provided features, rather than *instead of* them. If true, this is now an unfair comparison, since of course the DISk skills that are *only* based on velocity diversity will learn to move, while the DIAYN skills can instead manifest as different joint configurations. Thus DIAYN should be rerun on velocity-based features alone. If my suspicious is wrong, however, hopefully the authors can provide an alternative intuition for what is going on.

**Question: how is the skill expansion interval chosen for DISk?** It seems like the interval at which new skills were introduced was hand-crafted to match the interval at which the environment evolves, or in the static environment case, was carefully tuned to a complicated variable-interval schedule (Appendix D.4) to get interesting skills. This again seems quite artificial and unfair to the baselines. How sensitive is DISk to this schedule? Seems like an important ablation. Additionally, how do the authors imagine DISk being used in practice in environments where the user doesn't control the environment dynamics?

In addition to the ablation suggested in the previous question, there is another one missing.

**Missing ablation: shared network**: in section 4.5, the authors rightly state "Compared to prior work, DISk not only learns skills incrementally, but also learns independent neural network policies for each skill." However, the authors then only ablate the first feature, and not the second. The authors should test a version of DISk that uses a single shared network of similar architecture to the baselines.

Additional more minor comments and suggestions:
1. Another interesting approach to compare to is that of Groth et al 2021 (https://arxiv.org/abs/2109.08603), which uses a curiosity bonus alone to drive exploration and then periodically freezes copies of the policy as skills. This approach shares with DISk a gradually evolving continuous development of skills that is targeted to the life-long learning setting. The work seems to have been published after the ICLR deadline, however, so it would be unfair to ask for it as a baseline here. That said, it is obviously a baseline of interest for future versions of this work, and would make for good discussion in the present related work section.
2. The notation of section 3.2 seems to confuse random variables with the probability distributions they are drawn from. Additionally, the union operation in equation 6 doesn't seem well-defined to me. I think what the authors actually mean is: S_m ~ p(s|pi_m), S~(1/M)sum_m p(s|pi_m), and the first entropy term in equation 6 should then be simply H(S).
3. The authors motivate two of DISk's features (incremental skill learning, independent skill policies), but not the third - why do the authors use a seemingly complicated new entropy estimator, rather than for example using the variational (discriminator-based) approach of DIAYN (i.e. breaking down I(S;Z) in the other direction)?
4. Why isn't DIAYN included in Figure 3?
summary_of_the_review: Adapting skill learning approach to dynamic worlds and learning skills incrementally are good ideas, however the current set of experiments do not sufficiently distinguish the author's approach from existing literature, nor pinpoint what is important about their setup. With the baselines, ablations, and clarifications above included, I think this will be a strong paper that I look forward to rereading.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors have presented DISk, an unsupervised skill discovery method that incrementally learns a sequence of skills. This allows DISK to adapt to changes in the environment or agent dynamics during training. The authors demonstrate experimentally that in both evolving as well as static environments, DISk outperforms existing skill discovery methods on both skill quality and the ability to solve downstream tasks.
main_review: ### Pros:
1. The paper addresses an important problem in RL: unsupervised skill discovery in stochastic or evolving environments. Dropping this simplifying assumption brings it closer to real-world environments. As far as I'm concerned, this is the first work in this space. To me, the problem itself is real and practical.
2. The proposed framework of incremental skill discovery (DISk) is novel for separating the learning of individual skills.
3. This paper provides comprehensive experiments, including both qualitative analysis and quantitative results, to show the effectiveness of the proposed framework.

### Cons:

1. One thing I think was missing is the time-wise comparison between the results of DISk vs baselines in the main body of text. How long did the training take for DISk, DIAYN and Off-DADS in terms of both environments steps and wallclock time on the results presented in Section 4.2 - 4.4? DISk needs to train many neural networks (instead of one like in baselines). How slow does this make them compared to other methods (i.e., in terms of time when trained on the same number of environment interactions)? This is especially relevant for Section 4.4: transferring skills to the downstream task. I hope the authors can elaborate on this.
2. Also, I'm not particularly convinced by the ablations performed in the paper. The authors try to examine whether "the performance gains in the static environment are primarily due to the independent policy part", isn't the natural test for this training shared policy in parallel, rather than individual ones?
3. Have the authors considered running a version of baselines where the network weights aren't shared among the skills? Authors themselves wonder whether "the performance gains in the static environment are primarily due to the independent policy part", after all. This additional experiment might shed light on the question.

Some typos:
- Page 4: paragraph 3: "used" → "using"

## Post-rebuttal discusion

I thank the authors for the detailed response. Based on the clarifications that the authors provided, I've decided to increase my score by 1.
summary_of_the_review: To me, the problem is real and practical. The methods that the authors propose is novel. Authors provide comprehensive experiments to supports the proposed method's superiority compared to existing baselines. Nonetheless, I have several concerns regarding fair comparison of the DISk vs existing methods. Specifically, I'd learn more about its computational efficiency and see additional experiments and ablations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces DISk, an information-theoretic skill discovery method that discovers skills in a sequential fashion instead of all at once. This provides better adaptation to changes in environment dynamics. Evaluation is performed on standard continuous control benchmarks under changing and constant dynamics, and DISk compares favorably to baselines like DIAYN and Off-DADS.
main_review: **Positives**

- The paper considers a setting that is both novel and relevant. Providing agents with the ability to continuouslly acquire new skills in evolving environments is an important research direction.
- It is well written and easy to follow. The introduction and related work sections properly put the work in context.
- Results are strong and show that DISk can discover more diverse locomotion behaviors than DIAYN and Off-DADS, both under fixed and changing environment dynamics.


**Concerns**

- My main concern is related to the significance of the domains used to evaluate the different methods. I understand that these locomotion environments where agents need to discover skills that walk/run in different directions, so that a hierarchical architecture can then leverage these skills to solve more complex navigation tasks, have been a standard benchmark in the literature for at least 5 or 6 years. However, if we want to truly develop agents that can discover skills autonomously, we need to start considering more complex environments. I cannot agree more with the first sentence of the paper (*Reward-free, unsupervised discovery of skills is an attractive alternative to the bottleneck of hand-designing rewards in environments where task supervision is scarce or expensive.*). Unfortunately, the paper then considers environments where it is very simple to design reward functions that produce the desired behaviors. Note that previous works have considered unsupervised pre-training on Atari (e.g. VISR, APT). Another interesting domain that should be easier to integrate in the current codebase is the Fetch environments in Mujoco, potentially modified to include more objects that could lead to the emergence of manipulation skills.
- It is not surprising that the baselines fail in the presence of changing dynamics (e.g. overfitting to the last final leg). I wonder if authors could introduce slightly stronger baselines. For instance, a version of DADS where the policy is checkpointed before every change to the dynamics. The discovered set of skills would then be the combination of all the checkpoints (e.g. if there are N checkpoints with M skills each, this version of DADS would return $N \times M$ skills). This would show whether the problem with DADS is lack of adaptation or catastrophic forgetting.


**Other comments and questions**

- Please cite [SNN4HRL (ICLR 2017)](https://openreview.net/forum?id=B1oK8aoxe). This is a very relevant work that combines an intrinsic reward proportional to the magnitude of the speed of the robot with a mutual information regularizer that makes skills distinguishable from each other.
- How does DISk behave when the the environment dynamics are changed by placing new obstacles instead of removing them? For instance, by making a wall similar to the one in Figure 1 [here](https://arxiv.org/pdf/1712.06560.pdf) spawn after a certain number of iterations?
- Figure 9 (right) shows the MHD after a certain number of steps. It would be more insightful to see a plot showing how the MHD evolves as a function of the number of steps.
- I was surprised to see that Skill 2 in Figure 10 does not move at all. Could authors please provide their intuition for why this happens?
- Please provide a reference for the following sentence in the introduction:
> Not only that, but the agents also generalize poorly to any changes in the environment

summary_of_the_review: Overall, I lean towards accepting the paper but I have serious concerns about the adequacy of the evaluation protocol. While they clearly prove the advantages of DISk, I'm not convinced that by making advances in these toy environments we are truly getting closer to developing agents that can acquire useful skills autonomously. I acknowledge that the paper mostly follows standard practice in the community, which is why I still recommend acceptance, but the submission would be much stronger if authors could provide results in other settings (e.g. Fetch, Atari).
review_title: null

Meta-review: This paper is about an unsupervised method to learn new skills in non-stationary environments by maximizing an intrinsic reward function. Experimental evaluations on OpenAI gym environments show that the proposed approach improves the diversity of the learned skills and is able to adapt to continuously changing environments.

This paper is borderline. After reading each other's reviews and the authors' feedback, the reviewers discussed the pros and cons of this work. Even if the reviewers have pointed out that the paper has some limitations, they agree that the paper represents a valuable contribution and have appreciated the improvements implemented by the authors during the rebuttal, thus reaching a consensus towards acceptance. 
The authors need to update their paper according to what they have proposed in their response and they have to take into serious considerations all the reviewers' suggestions while they will prepare the camera-ready version of their paper.
Decision: Accept (Poster)