Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a novel skill discovery method that combines standard mutual information maximization with notable changes. The first is that skill now enter a diffusion stage (random actions) before being predicted by a reverse predictor. The second is that environmental resets are exploited in order to construct a skill tree, wherein new skills are executed only after their parent skills. The final change is a pruning/growth strategy similar to VALOR, which is critical for constructing the skill tree.

The authors evaluate their method by comparing it to traditional skill discovery methods in terms of state coverage and performance on downstream tasks. 
main_review: Overall, I really like this idea as well as its execution. The tree structure and separate diffusion process tie together nicely (i.e. no diffusion until leaf node), and the qualitative results are very strong. But there are a few things stopping me from a stronger acceptance recommendation.

There are two critical ablations that are missing. The first is to run UPSIDE without the diffusion stage (likely requiring retuning the action entropy bonus) to see how necessary it truly is. The second is to run UPSIDE without the tree structuring. My hypothesis is that the diffusion stage is only necessary with the tree structure, as it prevents stochasticity from hurting the ability to reach leaf nodes, though perhaps just executing the deterministic policy with otherwise high action entropy bonus would be sufficient. In particular, I don't buy the claim on line 121 that direct-then-diffuse is qualitatively different from direct-with-action-entropy, since e.g. the policy could just be super stochastic after becoming predictable -- effectively replicating the direct-then-diffuse behavior while maintaining the ability to utilize other strategies (e.g. don't randomly take an action that kills you).

In addition, your baselines would be a lot more impactful if you swept over skill duration. UPSIDE can chain together skills, which allows for its impressive performance on e.g. unknown far goal, but perhaps DIAYN could also achieve this with sufficient skill duration.

One final point is that the claim on line 198 that VIC/DIAYN don't have an unsupervised model selection criteria is simply untrue. Its just the value of the lower-bound e.g. E[ log p(z) - log q(z | s) ]. Indeed exponentiating this even gives a value for the effective number of skills, which is more readily interpretable than UPSIDE's actual number of skills, since those won't all be perfectly interpretable. Of course, this criteria could easily be used by UPSIDE as well, and UPSIDE has enough other advantages to offset the loss of this contribution.

 
limitations_and_societal_impact: Environmental resets are mentioned, but should also be addressed as a limitation. Other skill discovery methods can easily be extended non-episodic environments (e.g. compare DIAYN to VISR), but UPSIDE relies heavily on this ability in order to establish a root node for its tree.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The idea of this work is to maximize coverage while ensuring learned skills are distinguishable. The learning framework is based on maximizing mutual information (MI) between latent random variables and states. 
This paper proposes a decoupled objective for local coverage and directness, on top of it, a tree structure based exploration method is proposed to incrementally compose learned skills for maximum coverage.
main_review: While the problem studied in this work is important, there are several issues with clarity and novelty.

The authors need to define what is the length of “skill” before referring it.

The motivation of using the constrained optimization formulation for H(z) - H(z|s) (eq 3 and eq 4) should be elaborated more.

In introduction the objective seems to maximize H(s) - H(s|z) which aligns with its maximizing coverage motivation, but then in the method section, the objective becomes H(z) - H(z|s) which does not encourage explicit exploration, see EDL[Campos et al] and APS [Liu & Abbeel].

Also, EDL is a very related baseline, and should be discussed and compared in terms of methodology and experiments. EDL firstly optimize H(s) then use the data to optimize -H(s|z), which is closely related to this work. In fact, this work conducted experiments in 2D maze with the same topology as in EDL. I suggest the authors to compare their method against EDL.

------------------------ 

After reading the response from the authors, I am rising my score from 4 to 5.
As mentioned in the reply to the author response, while I tend to reject it, my score merely reflects the current status of the paper, with additional experiments in more convincing environments and downstream task fine-tuning, I think this work can be a strong paper.
limitations_and_societal_impact: The authors addressed the limitations and potential negative societal impact
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This method learns a space of intrinsic goals to maximize coverage of the state space. This work utilizes mutual information between skills and states, where the entropy of states and negative entropy of states given options describe the diffusing and directing objectives respectively. This is done by learning skills with two separate parts, a long scale directed component and a short scale diffusing component. These skills have state distributions that differ by at least a specified factor. It then separates these skills along a hierarchical tree structure, where temporal abstraction occurs by calling lower level options, which have a limited range, controlling the shape of the tree by pruning skills with low difference, and adding new skills when all the skills satisfy the condition. 
main_review: Abstract: the abstract is somewhat confusing in that it is not clear whether the skills learns are diffusing or directed, or how that separation is made, since it is difficult for them to be both at once. 

Introduction:
25 Unsupervised RL seems like an unusual phrase. Intrinsic rewards, exploration or unsupervised policy learning seem more descriptive. Reinforcement learning is defined by a reward function, distinct from supervised learning.

Figure 1 is fairly useful, but the description is somewhat verbose and between pictures it is not clear whether skills are being pruned or added, especially between image B and C / D and E.

42 Maximizing 1 does not encourage exploration in RL, though learning options which maximize 1 does tend to produce options which cover the space more fully. 

58 From the introduction, it is still not clear how the tree structure is used and how it differs from the skill structure. 

Setting:
It is not clear that a MDP is still an MDP without reward. This would just be an environment that exhibits Markovian dynamics. This is just a semantic issue, though.

Equation 1: While both forward and backward MI are mathematically equivalent, it is not clear from this section which one is more relevant for an intuition of this task. It seems like the first would be, however, since it better describes the diffuse and directed properties.

Algorithm Structure:
96,97,99 Note that the use of H(S), -H(S|Z) describe the forward form of mutual information, however p(z|s) is then used to describe the discriminability of a skill, but is from the reverse description of mutual information. This switching can be confusing. 

Equation 2: While it makes mathematical sense that the minimum skill is less than the average, it is not entirely clear what a minimum level of discriminability is. In this case, it seems like intention of this lower bounding is to say that the loss is based on how discriminable the least distinct skills is.

Equation 3: It is also not clear how Equation 2 leads to Equation 3. While it makes sense to optimize over the number of skills, and complete the mutual information term, this is a consequence only after combining with Equation 1. It would be useful then to specify this. 

135 This use of "convenient formulation" is unclear. This is used because maximizing over a cardinality of skills is an integer program, and thus using a greedy algorithm is the classic solution to such a problem.

In the final case, the skills are simply learned such that they exhibit better than \log \eta entropy over the states that they cover. It seems appropriate to relate this to clustering because this seems to have parallels to a kind of minimum distance coverage metric that might be used in that case. 

3.3 It seems like the tree structure and how the composition works is integral to this work, but is only discussed in a limited manner. How nodes are chosen to be expanded upon and how an agent would navigate the tree seems like immediate questions.

3.4 While prior work has described how the discriminator q_\phi(z|s) is optimized, it is not entirely clear when reading this section what feedback is used to train the discriminator. This makes the description of the different steps confusing, since they simply assume that the discriminator reward performs well. 

The pruning and growth method for skills in the tree is mentioned only briefly, and seems as though it requires a good number of hyperparameters to manage. 

Related work: 
From a formatting perspective, it would be best to separate the related work by category, especially since this work is primarily only interested in comparison with the line of work doing skill discovery based on mutual information. This is because this work tests only on the navigation domain used to visualize coverage, and uses the same language as these related methods.

Experiments:
While this method appears to perform substantially better than the existing methods, one weakness is that is does not compare with other methods which tend to escape some of the bottlenecks. In particular, though Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills is mentioned, and shown to get much better coverage by greater directedness, it is not compared against.

While the illustrations in Figure 3 are promising, it is worth noting that the performance of such exploration on high dimensional tasks has a much different characteristic. In particular, high dimensional states will take much longer to perform the dense coverage exhibited by the UPSIDE algorithm. This seems to limit the scope of results given (though most methods in this domain appear to struggle with this as well).

The diffusing component is not always tested in this method. For example, in the case where there would be a narrow bottleneck, it seems like the diffusing criteria could cause a skill that enters the bottleneck to fail because it cannot diffuse due to constraints in the environment. None of the experimental cases appear to test this possibility, but in many practical applications there are situations where this property is at play.

One ablative that seems important based on the results is to observe the level of effect of composing options. Since it seems like many of the other methods fail to escape the bottlenecks near the start, it seems like this difference is actually the primary advantage by comparison. 

Overall, this work proposes a new and interesting way to define a set of skills that maximizes coverage over a space while keeping the skills distinct. The work build a novel framework of option composition and provides an alternative way to choose options---based on the worst differentiable option, which is interesting. Though it compares against methods which do not perform as well as other in the same domain, it does demonstrate encouraging results. However, the domain limits the scope of these results from many practical applications. Some missing components in the description and the equations does obfuscate exactly what the algorithm is doing.

Originality: decent

Quality: good

Clarity: marginal

Significance: marginal
limitations_and_societal_impact: The work does not discuss limitations or societal impact. A discussion of how exploration could affect modern navigation methods, whether in maps, robot movement, etc, could have occurred.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a framework for an unsupervised learning of robotic skills that balances directness of exploration and coverage of state space. The theory of the method builds on the notion of mutual information between states and skills, where the conditional entropy between states and skill definitions corresponds to the directed part of the learned policy, and entropy of the states to the diffusing part of the policy. The directed part is optimized to increase discriminability of the skills given the states using a discriminator network. The diffusing part performs a random walk policy with uniform distribution over actions.The paper further proposes to optimize the number of skills by requiring the minimum discriminability of skills and discarding skills that fall below the discriminability threshold. Finally, in order to be able to reach states that are farther away, the paper proposes to combine skills in a tree structure and gradually expand the tree by adding new skill nodes to the tree. The approach is shown to outperform other unsupervised skill discovery methods on such tasks as continuous maze, CartPole and Reacher in OpenAI Gym during both, unsupervised skill discovery phase and training of downstream tasks.
main_review: Strengths
- Exploration and skill discovery is crucial for scaling up robot learning. Optimizing for both directness and exploration through mutual information between states and skills is an elegant way to acquire a wide variety of skills.
- The paper is well-written, and straightforward to understand and follow.
- The method is shown to discover a wider variety of skills and provide a better coverage of the state space than previous methods.

Criticism
- As discussed in the paper, and also visible in Fig. 3, the proposed method pushes the trajectories of different skills apart to increase their discriminability. Using a pre-defined minimum discriminability threshold might result in gaps in the state coverage, which can lead to low performance on tasks that require varying granularity of control, e.g. some manipulation tasks in robotics. It would be interesting to see a discussion how the method could deal with such tasks and ensure an appropriate state coverage.
- It would be useful to see more details about the skill latent variable z, in particular how it is constructed and its dimensionality.
- Experiments on downstream tasks assume that there is already a skill that is close to the new task and can be fine-tuned. It would be interesting to see a discussion how the method would deal with problems where combination of multiple skills is required.
- As also indicated in the paper, it would be interesting to see experiments on higher-dimensional problems.
limitations_and_societal_impact: The authors discussed limitations of their work. The proposed approach does not lead to any obvious negative societal impacts.
review_title: null

Meta-review: Reviewers found the proposed method interesting and early results promising, but ultimately the current set of experiments feel unfinished and fall short of acceptance. First, only a few relatively simple environments are currently considered. Reviewers felt that it is important to see UPSIDE succeed in more challenging environments that have been used for skill learning, e.g. Hopper, Ant, or Walker in Mujoco. Second, two important ablations are missing - the removal of the diffusion stage, and the remove of the tree structuring of skills. While the authors claim both are important for their method, it is important to demonstrate this to readers. With these two (relatively simple) issues addressed, I expect UPSIDE will be a worthy contribution to the community.
Decision: Reject