Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Parallel training of skill discovery methods results in failure of learning some skills. This happens because certain states are never visited by the learned skills. Authors propose recurrent skill training, which involves training a particular skill for some epochs followed by the next skill for the next few epochs. 
They further shape the reward according to the state coverage. This prevents multiple skills from visiting the same states. Further experiments on various 2D navigation tasks and robot locomotion tasks show the new method achieves better state coverage and divergence compared to baseline methods. 

strengths_and_weaknesses: Strengths:
The paper finds a clear and important weakness in the current skill discovery methods. And solves it with a "somewhat" novel solution. 

The paper is well written and clearly explained. The experiments are also well done. 

Weakness: 
The paper does come up with a solution to a problem. But the proposed solution creates more problems of its own. 
1) Proposed method has worse sample complexity compared to other baselines (while other methods have less state coverage)
2) Does not work with a continuous latent
3) Does not address catastrophic forgetting of skills as number of skills grow
4) Needs at least 2*N networks, where N is the number of skills


Post-Rebuttal:
When we compare ReST against current methods in unsupervised skill learning, ReST falls short in multiple accounts.

Has higher sample and computational complexity compared to other methods
Cannot handle continuous latent (while other comparative methods can)
Too many moving pieces compared to other methods (Multiple networks for skills, for novelty detection)

Thus, it is difficult for me to recommend acceptance of the paper. I thank you for your efforts and response.


questions: How does one decide how many epochs to train a particular skill?

Given the current weakness it is difficult to accept the paper. If the authors provide good reasons why this work is important despite the weakness, I am willing to change my score. 

limitations: Please address catastrophic forgetting when number of skills gets large.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: An ideal scenario of unsupervised skill discovery is to acquire a set of skills that has large state coverage as well as diversity. To this end, this paper suggests optimizing these two criteria by (1) using the state visitation-based reward for exploration and (2) iteratively training skills in that each skill is different from other skills. 

The paper identifies that the prior unsupervised skill discovery work discourages skills visiting the bottleneck states due to low intrinsic reward on the bottleneck states $r \sim p(z_i \vert s)$, resulting in the bottleneck state never visited by any skill eventually, which is called the "exploration degradation" problem. This paper claims that this problem comes from simultaneous skill training, and thus proposes to train skills one by one, not in parallel.

The empirical results demonstrate that the proposed method acquires skills that are diverse and identifiable as well as cover wider states, whereas prior works, DIAYN and DADS, only show limited state coverage.
strengths_and_weaknesses: ### Strengths

- The proposed iterative skill learning scheme is novel under the unsupervised skill discovery setting.

- Experiments on the 2D navigation environments are intuitive and well demonstrate the advantage of the proposed method in terms of diversity and coverage of the learned skills, compared to the baseline methods (in Figure 1 and Figure 4).

- The paper clearly explains its motivation, approach, and experiments. The paper is easy to read.

- Both qualitative and quantitative results are well designed and explained.

### Weaknesses

- There is another direction of work for acquiring a goal-conditioned skill policy in an unsupervised manner [1, 2], which demonstrates its wide state coverage and applicability to complex locomotion as well as manipulation environments. Comparisons to these baselines would be necessary as these works have the same goal of covering a wider range of states but are not limited to a fixed number of discrete skills.

- The proposed method mainly involves two changes from prior work: (1) iterative training for the diversity of skills and (2) the state visitation-based reward for exploration. To verify the effect of each component, it would be helpful to have results about the parallel training version of the proposed reward.

- The proposed method is designed only for discrete skills, and scaling to continuous latent skills seems not trivial given the nature of iterative skill training.

[1] Sekar et al. Planning to Explore via Self-Supervised World Models, ICML 2020

[2] Mendonca et al. Discovering and Achieving Goals via World Models, NeurIPS 2021
questions: - Given that RND's prediction networks will become more accurate over time, the reward signal may become less informative about the state visitation statistics. Have the authors experienced this issue for experiments? Is there any way to prevent this?

- Is there any stochasticity in the environment or skills?

### Minor suggestions

- Iteratively sounds more intuitive than recurrently.

- In the abstract, "parallel training procedure inherently discourages exploration" is not intuitive. It would be great to briefly provide an intuition.

- Figure 1 clearly explains the motivation of the paper. However, it does not have enough information to get the intuition without reading until Section 3. A bit more explanation about "two skills (blue and red) visiting the same state (a) discourages both skills to visit that state later (b)" and "colors of trajectories represent skills" would help understanding it.
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an algorithm, ReST for unsupervised skill discovery. ReST trains different skills in sequential, instead of training all of them in parallel. ReST can address the exploration degradation problem proposed by the authors. The efficiency of the proposed approach is demonstrated in various tasks.
strengths_and_weaknesses: I tend to give an accept at this moment, and I might change my opinions after reading the rebuttals.
# Strengths:
1. This paper proposes a novel problem in unsupervised skill discovery: the exploration degradation problem. I think this is an interesting problem and is worth an oral presentation for all the researchers in this field to study and understand.
2. The ideas are well introduced and the motivations are quite convincing. I'm convinced by the toy examples provided by the authors. Figure 1 and Figure 2 are very convincing and the ideas are simple and clear.
3. I believe in the effectiveness of sequential skill learning because it can reduce collisions of the learned skills as introduced by the authors.
4. The provided videos of the discovered skills are convincing. They can show that the ReST can learn different skills efficiently.

# Weakness:
1. The proposed videos https://sites.google.com/view/neurips22-rest are not compared to the videos of any baselines. So I cannot evaluate the advantages of the ReST by seeing the video. I wish to see more comparison videos in the rebuttal phase.
2. The proposed recurrent algorithm may consume much longer time than the baselines. So the comparison might be unfair to the baselines. I notice this in the limitation part, so I do not penalize this too much.
3. The quantitative results are not enough. I only see some comparisons on the right of Figure 4. And the comparison cases are quite toy (e.g. in Maze scenarios). The authors should compare to more realistic cases (robotics) and list the numbers for ReST and the baselines. This could benefit future approaches.
4. The writing quality can be improved. Refer to the following section.
5. The code is not provided so it might affect the reproducibility of this work. Although code should not be a must, in the field of reinforcement learning, reproducibility is worth concern without code. 
6. The broader impact is not included in this paper. In the standard of Neurips submission now, (expecially this paper is concerned with robotics), this paper needs a broader impact section. 

# Writing Suggestions:
1. FIgure 3's text and others are too small for publication quality.
2. Figure 3 does not illustrate the algorithm well. It is too complicated and the authors should walk through Figure 3 (a) in the caption. In general, the ReST algorithm is simple and the authors should provide a simple diagram with clear examples as Figure 3. 
3. Eq. 3 to Eq. 7 are a waste of space. You can only leave one form of the information bound and remove other equations.
4. Eq.1 is also a waste of space because it's too trivial.
5. What are N and M in algorithm1? It should be self-contained. You can replace them via "the number of epochs", etc.
6. Some parts are hard to follow. For example, in Ln 131->152, the authors are trying to adopt RND to learn novel skills. However, the descriptions are not organized well so it is hard to understand here if the readers do not know RND before. One suggestion is to discuss the function of the networks here, then dive into details. Another question is, why two networks are used? This question is briefly answered in Ln 147. But I'm not quite convinced by the function.
7. Eq 10 is not illustrated well and there is no ablation study to support this equation. 

-------------------------------------
To colleague kKQH,
I agree with you on the strengths parts. For the weakness part, you mainly point out some engineering drawbacks to implementing this paper. I'd like to share my opinions on your concerns:
1. The sample complexity is not that important in this sort of problem. This should be a pure research project and we do not need to deploy the training on a real robot/ industrial application at this moment. The longer training epochs would not cause a longer inference time.
By the way, I think somehow you need that long complexity if you wish the skills are "orthogonal" or "different". Recall the algorithm of Standard Orthogonal in Linear Algebra, we need to find each orthogonal vector at each time to ensure that they are fundamentally different.
2. The continuous part is another long-standing challenge. I don't think they can be solved easily in one framework and many so-called general algorithms can only work well for discrete or continuous problems. This should not be a reason for rejection.
3. I don't quite understand the "catastrophic forgetting of skills", do you mean that the skills can hardly be diverse if you increase the number of skills? This should hold true for all algorithms in this domain, not necessarily a limitation of the proposed approach.
4. The number of networks is fine and one may compress or prune them into one network. This should be an engineering problem. 

questions: 1. I think the baselines already encourage the diversity of learned skills. Can we develop better approaches to avoid exploration degradation without sacrificing sample efficiency?
2. If the new skill is forced to visit the novel states, it might not able to visit some existing good states, which might be harmful to the exploration. What's your opinion on this?
3. Can the learned skills help hierarchical reinforcement learning?
4. Will the code be released?
limitations: The authors give a good explanation of the limitations part. However, the broader impact section is not provided.
review_title: null

Meta-review: metareview: The paper identifies a problem in prior works on skill-learning: some states that are visited in training by diverse skills may not be visited after skills are learned (dubbed as exploration degradation problem). The problem is well-motivated and is an important one in unsupervised skill learning. Authors then propose a method to overcome the identified issue. Comparisons are made to state-of-the-art methods such as DADS and LEXA. 

The reviewers are split in their opinion: UvQx recommends an accept, nWfC recommends borderline accept, while kkQh recommends rejecting the paper. Authors addressed the primary concern of nWfC on comparison with LEXA. kkQh's main concerns are about the solution not being elegant. While I agree that more elegant solutions can be found, I think that identifying a bottleneck in learning diverse skills is of good value to the community. Furthermore, the proposed solution works across a range of environments. Therefore, I am lean towards a positive recommendation for this paper. I encourage the authors to clearly address the reviewers suggestions and comments in the camera ready version. 
confidence: Less certain
Decision: Accept