Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work proposes a skill discovery method given a set of symbolic states (state abstractions). The method consists of two parts. In the first part, a low-level controller, a = pi(s_0, k), which can be regarded as an option, outputs continuous actions to move the robot to the terminal state, s_T, given the initial state s_0, and the option index k. Here, s_0 and s_T correspond to symbolic states z_0 and z_T, respectively. In the second part, there is a symbolic forward model, z_T = q(z_0, k), which outputs the next symbolic state z_T given the current symbol z_0 and the option index k. By maximizing the mutual information between z_T and k, the robot learns a set of options that are both diverse and predictable given the initial symbol z_0. The low-level controller is trained by an RL algorithm (SAC), and the symbolic forward model is trained in a supervised manner.
strengths_and_weaknesses: The proposed method is a novel contribution for discovering skills given state abstractions. Although the assumption of state abstractions is strong, these abstractions can be indeed learned via symbol learning methods also mentioned in the text [24, 25, 27, 28, 29]. The idea of concatenating an option index k to the state, and maximizing the mutual information between k and z_T is very interesting. The paper is nicely written, explaining each definition thoroughly. Experiments are well-designed and there is an ablation study for each part of the method.

One slight downside might be that the method is only tested in 2d environments; it would have been good to see the performance of the method in a scaled environment, say, a 3d version of the lights out, since the exploration of the environment might not be as easy as in 2d. Of course, this is more related to exploration, but it would still be interesting to see the limits of the method.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe the contribution of the paper is very novel and significant. Experiments show the applicability of the method both in simulated and real-world environments. The paper is well-written, and the accompanying video also helps assessing experiments.
issues: Although the related work section is quite extensive and good read, the  authors missed a very relevant literature. A number of recent work (Silver et al., 2021&2022; Chitnis et al., 2021) also works on skill learning where skills are represented with both symbolic and low-level components and hierarchical learning is performed. These studies use demonstrations rather than Reinforcement Learning, but otherwise have very similar assumptions, approaches and goals. The authors need to include one or more of these studies in their paper.

Silver, Tom, et al. "Learning Neuro-Symbolic Skills for Bilevel Planning." arXiv preprint arXiv:2206.10680 (2022).
Chitnis, Rohan, et al. "Learning neuro-symbolic relational transition models for bilevel planning." arXiv preprint arXiv:2105.14074 (2021).

- line 132: please define k before its use.
- line 152: d and are is mixed up.
- According to line 184, a change in state might not be observed in an episode. Then, how sound is it to include such an episode for FM learning (line 192)?
- BFS does not give optimal solution if the actions costs are not same. Because the skills are discovered by the robot, each skill might have different length or cost. Therefore, the solutions might not be optimal. Please comment.
- line 260: In 2D simulated environment, is the robot really torque controlled?
- Although the notation is consistent and sound in Section 3, the use of numbers/labels/brackets in the subsecripts is a bit confusing. Is it possible to improve?
- In task performance evaluation in Section 4, do environment steps refer to the number of optimization steps or the number of data points (i.e., environment interactions)? This is not clear because 10m steps are needed for experiments in the simulation while it is only 5000 interactions in the real world.
- It would be nice to show what kind of skills are discovered after training (i.e., a scatter-plot of executions from an initial point for option k), and the average number of low-level actions for each skill. Regarding the statement, "The board coordinates are x, y ∈ [0, 1], the maximum displacement per timestep is ∆x, ∆y = 0.2.", if each displacements after each action become very high, then it would be enough to output only 3 or 4 low-level actions.
- Moreover, for the reward improvement, I think having multiple high-level actions that moves to the same terminal state from the same initial state (i.e., not unique) might be a feature in other settings in which the robot learns two modes of operations and selects one at different conditions.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper presents a method that jointly learns a predictive model in the symbolic level and a skill-conditioned policy.  After training, the learned predictive model allows the authors to do long-horizon planning in the high level, and execute those plans using the learned policy. The method is both evaluated in simulation and on a real robot.  
strengths_and_weaknesses: ## Strengths
Automatically learning a discrete set of skills as well as a transition function in high level is extremely interesting. The paper shows interesting results that in many cases, the method can recover all the discrete skills needed to solve the task. 

## Weaknesses and questions
-  about the assumption

The authors have made it clear that there is one assumption of this work, which is that the symbol definition and the mapping from state to symbol are given. While I consider this assumption reasonable, it dramatically simplified the problem, thus posing an issue of applying this method to other tasks. 

-  about the efficiency

From the experiments, we can see as the task becomes more complex, recovering all skills becomes harder. It seems to me that the method in this paper is still not efficient enough to be scaled up to more interesting robotic tasks (with a magnitude of more symbolic states and skills). I'm curious about the authors' view on learning efficiency. What bottlenecks, which prevent the system to discover skills more efficiently, does this algorithm have? One reason, as mentioned in the relabelling section, is that the coupled nature of the forward model and the policy make the learning in early stage hard. Another reason I can think of is the exploration issue. Does the method have trouble in encouraging exploration? 

-  about the learned skills

The training objective encourages the skill to reach diverse successor states, and be consistent with the forward model. Are there any constraints on the skills being consistent over different starting states?  For example, in two different starting states where different lights are on, if one skill is learned in the first starting state to be pressing the up-left button, will this skill also be pressing the same button if the initial lights and robot state are different? Figure S1 suggests that the skills are consistent. But I wonder how this is achieved. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper bridges the gap between the neural network based reinforcement learning and symbolic planning. This could potentially have a large impact on the direction of learning more robust and interpretable models. 
issues: No issues.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes ‘SEADS’, a novel hierarchical approach to learning diverse skills as well as a forward model over these skills that will be useful for planning to solve long-horizon tasks. The proposed method assumes that a discrete, symbolic abstraction of the low-level continuous environment is provided as input. Given this, it utilizes an unsupervised exploration reward to collect data via environment interaction, and then learns skills and the corresponding forward model based on these data. Experiments show that the proposed approach is capable of solving two different physically-embedded games (LightsOut and TileSwap) under different settings more efficiently than other baseline hierarchical RL approaches.
strengths_and_weaknesses: **Strengths**

- The proposed method seems novel and is intuitive. I thought the overall idea to learn skills and the forward model jointly was interesting. Moreover, the derivation of the exploration reward approximation, as well as the relabelling were both clever and creative ideas.
- The paper is generally well-written, clear and easy-to-follow.

**Weaknesses**

- The proposed approach applies only to a rather limited setting. To me, it seems that a key limitation of this work is the set of assumptions that (1) the abstract symbolic state space is small, and (2) getting skills to reach the different possible symbolic state changes is desirable. These are not practical in most robotic domains, where even the symbolic state space might be extremely large and only a small subset of symbolic state changes might be important for useful skills. Thus, it seems difficult if not impossible to apply this work to continuous, long-horizon problems beyond the relatively simple setting (physically-embedded games with small and simple symbolic state spaces) that the authors have chosen.
- The task performance evaluation is limited to tasks that only require up to 5 moves. This seems like an unjustified choice that simplifies the task and significantly reduces the horizon. Given that the approach was trained to generalize to settings regardless of horizon, why was the horizon limited?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall, I think this is an interesting, well-written paper that provides an interesting new direction for hierarchical RL. However, the limited applicability and setting make it seem rather narrow in scope compared to existing hierarchical RL approaches. I think it is valuable and interesting to the community, but could be made significantly better if the authors clearly address and point out the applicability of their method in the writing of the paper or via additional environments/domains in the experimental evaluation.
issues: - As mentioned in the “weaknesses” section, I find the approach’s applicability to be rather limited to physical games such as LightsOut and TileSwap. Do the authors agree with this assessment? If so, I think it is important to mention this in the Limitations section. Additionally, I think it is important to highlight the assumptions regarding the size of the state space and desirability of reaching all of it in Section 3.
- As mentioned in the “weaknesses” section, I don’t understand why the evaluation was restricted to tasks that require only up to 5 moves. I think it is important to justify this choice in the text.
- When reading through the paper, it took me a while to understand that the number *K* is being provided as input to the approach. I think it’s important to mention this explicitly at the start of Section 3.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work proposes to use a given symbolic state abstraction to learn and plan with temporally extended skills. The skills are learned by reinforcement learning, with an objective that combines diversity and predictability in terms of reachable symbolic states. Additional tricks are proposed to help avoid local minima. Experiments are in physically embodied grid-based games.

strengths_and_weaknesses: ### Strengths

* The writing is overall very clear. I did not find many grammatical issues and the organization is good.
* It is clear how the proposed approach improves on less structured and more model-free approaches like SAC, VIC, and HAC, and it is good to see those baselines in the experiments.
* The ablations in Figure 3 and the appendix are very helpful for understanding the relative contributions of the proposed approach.
* It’s always impressive to see results with reinforcement learning on a real robot, and the video does a good job visualizing that process.
* The discussion of limitations is helpful and thorough.


### Weaknesses

* The environments used in the experiments are a bit contrived, and it is unclear to what extent the approach proposed in this paper would work beyond these environments.
   * It is unclear whether the robot is really adding anything in these environments. In other words, one could argue that the robot is added just to make the demos more flashy, when in reality, all of the technical difficulty resides in the original board games.
   * If these environments were actually ones that we wish to solve in a real application, there would be much easier ways to solve them, and those ways would probably not involve machine learning. For example, use a symbolic planner to solve the game boards, and use inverse kinematics and a hardcoded press-down motion for the temporally extended actions. (This would be a simple version of task and motion planning, but where the task and motion planning aspects are completely decoupled.)
   * Given that the symbolic state space is highly structured, and it is known what the ideal abstract actions would be (pressing each grid cell), a much more direct RL approach would (1) create one skill for each grid cell; and (2) directly associate skills to cells, and give rewards accordingly.
* The fact that the number of skills needs to be manually specified is a big limitation, and in experiments, the direct correspondence between the number of skills and the symbolic action space is a big injection of knowledge that is not emphasized.
* There are several related works in the AI planning community (see Issues below) that call into question the novelty of this work, but this paper is the first that I have seen with (real) robotics results.
* If the objective of this work is to learn temporally extended skills, the skills learned in the experiments are all essentially the same at the policy level: move and press. It would be more interesting to see a variety of skills learned.
* I’m not completely convinced that the “second-best normalization” idea makes sense. Intuitively, it seems redundant with the diversity objective, but I can understand how optimizing that objective directly could lead to local minima. Moreover, the empirical results do not show a significant improvement vs. the “no sec-best norm” ablation.
* It is not clear to what extent generalization is evaluated in the experiments. L250 states that board configurations are disjoint in training and testing, which is a good start, but I would be curious to know: in the course of a testing episode, how many of the symbolic states encountered are novel with respect to the training distribution? In addition to measuring this directly, it would be informative to see an ablation where the learned forward model is replaced with a tabular model.
* As the paper mentions, planning with breadth first search is very limiting. Given the symbolic states and the existence of powerful AI planners, it would be more impressive if the overall approach were able to use those planners. For example, the works by Asai et al. and Konidaris et al. use PDDL planners and can scale to much longer horizons than the ones in this work.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: *Post-rebuttal:* Thanks to the authors for addressing my questions and concerns. I am raising my recommendation to Weak Accept.

*Pre-rebuttal:*
The work is overall performed with a high level of technical sophistication, and the paper is well-written, but I am concerned about novelty and about the limitations of the experimental environments.

issues: * A “more skills” variation of the main method is briefly mentioned in S.5, with results in Figure S.4, but the variation is never explained (e.g., how many more skills are there?) I was going to ask to see results for a variation like this, so I am interested to know the details.
* It’s not clear why that “more skills” variation and the VIC results are in the appendix, rather than in the main paper. (Maybe they were generated in between the paper deadline and the supplementary deadline?)
* Other references:
   * “Leveraging Approximate Symbolic Models for Reinforcement Learning via Skill Diversity” Guan et al. 2022
   * “Reprel: Integrating relational planning and reinforcement learning for effective abstraction” Kokel et al. 2021
   * “Symbolic Plans as High-Level Instructions for Reinforcement Learning” Illanes et al. 2020
   * “SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning” Lyu et al. 2019
   * “Using Abstract Models of Behaviours to Automatically Generate Reinforcement Learning Hierarchies” Ryan 2002
   * “Integrated Task and Motion Planning” Garrett et al. 2021

### Minor
* $\mathcal{S}_0$ is not defined on L139, and it seems like just $\mathcal{S}$ is used instead starting on L143.
* It’s not immediately obvious why $I$ and $\beta$ are defined over tuples of states rather than individual states. In the original options paper, they’re just defined over individual states. My understanding is that they are defined in this way because the termination function needs to “remember” the original abstract state so that it can terminate when the abstract state changes. It would be helpful to spell this out. 
* L141 grammar: “for the option policy holds”
* L218: remove period before “which”
* L246 grammar: “As step limit for skill execution”
* There are a number of places where the open quotation marks are backwards
review_title: null

Meta-review: metareview: Reviewers like the novelty of the formulation and the good writing. Shared concerns include the limited experiments and the contrived setup. Other concerns include those on generalization and related work.  Please consider addressing these points. 

The authors did a good job during the rebuttal period, after which all reviewers were in favor of acceptance.  The AC agrees.  Please revise the paper accordingly based on the reviews.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)