Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper addresses the problem of learning to execute and adapt existing skills to more complex tasks that are different from the tasks used to build the atomic skills. In particular,  (i) the paper collects state-action pairs of handcrafted controllers, (ii) learns a skill embedding and a state-conditioned skill prior based on this data and (iii) learns an RL policy that acts within this skill space and adapts each skill to task variations. 
strengths_and_weaknesses: # Strengths
- The paper is clearly and well written 
- The idea to condition the skill prior on the current state seems clever, since it rules out state spaces for which the skill is unlikely to be helpful while the skill prior captures the fact that the same state might be part of multiple skills by the multimodal distribution. 

# Weaknesses
- Evaluations only in simulation: data generated in reality might suffer from distributional shift and other problematic influences. It is still to be shown if the techniques presented in this paper can handle such difficulties. Can you discuss, for instance, how the skill embedding space in Figure 3 would change? 
- It appears that the authors composed a new method out of several existing approaches. For instance, the autoencoder used for the skill embedding comes from [1,21]. Please make it very explicit what you're adding. Is it the multimodal structure of the skill prior? A contribution could also be the clever composition of these existing methods to reach higher empirical performance.
- It would be interesting to also compare to hierarchical RL approaches such as HAC [1]. It seems obvious that neither PPO nor SAC perform well in such tasks. 
- Only three random seeds have been used to test the data. Since the progress in RL is generally very noisy – especially in more complex tasks – it is important to evaluate more thoroughly.

[1] Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko. 2019. Learning multi-level hierarchies with hindsight. In Proceedings of the 7th International Conference on Learning Representations
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although the paper presents an interesting approach and addresses an imoortant research question, it also has multiple weaknesses. I encourage the authors to clarify and address these weaknesses. I would be happy to increase the score then.
issues: Please address the weaknesses in the "Strengths And Weaknesses" section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work provides two main improvements to robot skill learning methods: First is a latent skill space, which can be used as a prior for accelerating the learning on both the high-level policy $\pi_{HL}$ and the skill decoder. The second is a residual skill mechanism, which can further adjust the action generated by the skill decoder on different task settings.

The comparison of simulated environments shows the effectiveness of the proposed method and illustrated the advantage of learning with skill abstractions and pre-captured data.
strengths_and_weaknesses: 
This work introduces a latent skill prior, which can be used for accelerating both high-level policy learning and skill decoder optimization. When learning the high-level policy, the proposed latent skill embedding can reduce the variance when doing explorations and can be estimated with variational inferences as proposed by this work. Moreover, when learning the low-level policy (the skill model), a residual policy model is proposed to better fit the task requirements. These two ways of using pre-captured data are intuitive and appropriate.

However, this work has obvious shortcomings. Firstly, the paper is not so well organized and a little bit hard to read, some of the details are missed like how is the high-level policy defined and trained (why not directly use $\pi_{HL}=q(z|s)$ and fine-tuning it on tasks), and how will the skill horizon affect the learning performance and efficiency. Then, the survey on hierarchical RL is not sufficient, there exists some work that uses VAE/cVAE for skill discovery and policy learning, like [1,2] and so on, although they come from option-based hierarchical imitation learning, a certain comparative survey is necessary. Further, the comparison is somehow not sufficient, the comparison among HRL methods like Option AC[3], H-DDPG, and so on are welcome.

[1] Sharma M, Sharma A, Rhinehart N, et al. Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information[C]//International Conference on Learning Representations. 2018.
[2] Hausman K, Chebotar Y, Schaal S, et al. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets[J]. Advances in neural information processing systems, 2017, 30.
[3] Bacon P L, Harb J, Precup D. The option-critic architecture[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2017, 31(1).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This work enhances the skill-based hierarchical RL method from two aspects: a latent prior that can be used for faster exploration and the residual low-level policy, where the technical innovation is limited. Also, the comparisons with HRL methods are insufficient and affect persuasiveness the of the results. Finally, the organization of this work is a little bit unclear, and the experiment and theory do not demonstrate the advantage of using latent prior (see Q.2, Q.3 below).
issues: 1. For experiments, why is the average reward of ReSkill (No Residual) different from ReSkill within the first 20k steps? As shown in the line.400, the residual policy is inactive so ReSkill (No Residual) and ReSkill will be exactly the same.
2. The analysis of ReSkill (No Skill Prior) is missing on Sec.6, and the experiment setting that uses samples from Z is ambiguous, does $z~N(0, I)$ or $z~\pi(z|s)&, maybe the latter would be better?
3. How is the high-level policy trained? can it be fine-tuned from prior $p(z|s)$? Is there any quantitative analysis on how much could the proposed prior narrows the exploration range?
4. More comparisons on Hierarchical RL methods are welcomed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work introduces ReSkill, a method for skill-based RL that enables efficient exploration and adaptation to tasks beyond the initial set of skills. The method is composed of three main parts that are trained sequentially: skill extraction from scripted policy demonstrations, learning a skill embedding and state-conditioned skill prior, and learning a hierarchical policy that combines an action decoder for sampled skills with a learned residual policy. The two main competitive baselines that the paper compares against are PaRRot (non-hierarchical version of ReSkill that uses a similar flow-based state-conditioned generative model for direct action prediction instead of skill prediction) and SPiRL (this can be viewed as ReSkill without the state-conditioned skill prior). ReSkill outperforms all baselines, and ablations show that the skill prior is useful for exploration and the residual policy is useful for adapting to changed environments.
strengths_and_weaknesses: Strengths:
- The method’s state-conditioned skill prior is a powerful abstraction and a very clear improvement over the prior work SPiRL in terms of exploration benefit and skill space expressivity.
- The paper is well presented with great clarity. Each of the method’s three components are explained concisely. The system design and motivations are visualized clearly in Figures 1, 2, and 3. 

Weaknesses:
- The method is only evaluated on simple simulated domains; it is unclear how the method scales with higher dimensional tasks (with complex, high-dimensional skills) where state-conditioning may be intractable

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is presented extremely well and clearly describes the three main components of their method with ample motivation and analysis. The comparisons to relevant baselines and ablations showcase the effectiveness for exploration of the main contribution of the paper, the state-conditioned skill prior via density estimation. While the method is very compelling and technically sound, the main reservation I have is that the method is only evaluated on relatively simple domains, so it is unclear how it extends to more complex domains and especially the real world.
issues: - The test environments differ quite a bit from the training environment used to generate the skills (push and pick and place). As tasks increase in complexity and the test skills are increasingly different from the training skills, the residual policy becomes very important. Therefore, how does ReSkill scale to high-dimensional environments or very diverse tasks, especially as there is an increasing distribution shift between train and test skills?
- Why is “ReSkill (No Skill Prior)” in Figure 6(a) so much better than SPiRL? Shouldn’t they be very similar?
- Is the most novel / important contribution the state-conditioned skill prior or the residual policy, given that they both seem important for performance?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper extends skill-prior RL (SPiRL) by (1) replacing a uni-modal gaussian skill prior with a multi-modal skill prior and (2) adapting low-level skills using a residual skill policy. The proposed multi-modal skill prior not only handles multi-modal skill distributions but also fully constrains exploration within the skill prior distribution. In addition, this paper addresses the problem of the imperfect skill repertoire by allowing low-level skill adaptation. The experiments demonstrate its faster convergence and higher performance over state-of-the-art RL and skill-based RL approaches.
strengths_and_weaknesses: ### Strenghts

* This paper well combines the existing techniques of multi-modal prior and residual policy learning to address the challenges in skill-based RL.

* The paper is clearly written.

### Weaknesses

* The tasks used for evaluation are too simple compared to the ones used in prior works. Testing the proposed method with more diverse skill sets would be necessary to prove the strength of the proposed skill prior.

* By utilizing normalizing flows to implement the multi-modal skill prior, the action space of the high-level policy becomes strictly limited to its skill prior distribution, which may not align well with the downstream task and is not flexible to acquire a new skill distribution. This method alleviates this issue by fine-tuning the low-level controller with the residual skill policy; however, this results in non-stationary training of the high-level policy, which is one of the major challenges in HRL.

* The limitation of the proposed method in the use of on-policy RL algorithms may hurt the sample efficiency of skill-based RL.

* The experiments are conducted in newly designed environments and tasks, which are tailored to the proposed method. Moreover, the tasks are rather simple and short-horizon compared to tasks where skill-based RL is good at. To demonstrate the broad applicability of the proposed method and better comparisons to prior works, it is recommended to perform experiments on commonly used benchmarks for skill-based RL, e.g., FrankaKitchen and PointMaze.

* Many implementation details, such as skill length H, task episode length, and task reward, are missing, which makes it hard to judge the correctness of the results.

### Questions and suggestions

* In Section 4.2.1, the proposed way of embedding skills to the latent skill space has been already proposed in Pertsch et al. [1] and demonstrated its superior performance.

[1] Pertsch et al., Demonstration-Guided Reinforcement Learning with Learned Skills, CoRL 2021

* In Figures 5 and 6, using more intuitive measures, such as success rate, would help understand the results. Or, at least, it should contain an explanation about the returns (rewards) for each task.

* In the references, many conference proceedings appear as arXiv preprints. Please use the proper citation.

* The appendix should be submitted separately. This paper violates the official guideline.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper proposes an intuitive and meaningful extension of SPiRL. However, the experiments are done in a new set of tasks, and many details are missing, which makes the results less convincing. Therefore, I stand for weak rejection.


**After rebuttal**

I appreciate detailed responses and additional experiments in the revised paper. The responses and additional experiments make this paper more solid, so I would raise my score to "Weak accept". But, I think this paper needs significant improvement to make it at the level that I can confidently accept this paper. 

Here are additional comments for the revised paper:
* In Figure 8, ReSkill and ReSkill (No residual) should be reversed.
* L526-527, I do not agree with the claim that "substantially outperform SPiRL in both sample efficiency and convergence
527 to a higher final policy performance". Although ReSkill learns faster at the beginning, it could be due to better pre-training of skills and skill priors, and ReSkill and SPiRL seem to converge at not significantly different final performances.
* In addition, the last subtask is the only task that requires adaptation as the first three subtasks are present in the dataset. However, the proposed method seems to fail at adapting to this new subtask, which weakens the major claim of the paper.
issues: See the weaknesses listed above.
review_title: null

Meta-review: metareview: AE summarizes the strength and weakness of the paper raised by reviewers as follows:
Strengths

- The paper is well-written 

- The learning of the state-conditioned skill prior is interesting, and the implementation based on the normalizing flow appears reasonable
	
Weakness

- The experimental results could be improved by using more tasks and more random seeds

- The comparison with HRL methods is missing

The paper is well-written, and the proposed method appears to be reasonable. Meanwhile, the weakness of the experimental results seems the bottleneck. In the initial submission, results are reported based on new tasks, which seem tailored to the proposed method, and the scores are computed using only three random seeds. 

=== post-rebuttal comments ===

Although reviewers raised several concerns in the initial review, most of concerns were resolved by additional experimental results provided by the authors. Thus, AE recommends the acceptance of the paper. However, as the amout of the revised texts are large, AE stroungly encourage the authors to go through the paper once again and carefully check the writing and presentation. Also, as Reviewer HjV5 indicated, it is necessary to discuss how the proposed method would be applied to a real robot. AE would like the authors to make sure to address these points before submitting the final version.
 

confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)