Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper explores an approach to using existing (learned) skills in a new task in the same domain. The approach keeps the existing skills intact (i.e., does not modify them); learns a policy over them (called the "scheduler") that maximises current task reward; and, at the same time, learns another policy (called the "new skill") that also maximises current task reward but by only using the primitive actions in the domain. Action selection in the domain is done by the scheduler. The approach is tested in two simulated robotics domains. 
strength_and_weaknesses: The primary strength of the paper is that it explores the important problem of knowledge transfer between tasks. In addition, the approach taken is sensible. However, I cannot claim that it is fundamentally innovative. The primary weakness is that the utility of the approach will depend on the particular task and the particular skills that are being used. While the paper gives examples of its successful use, it is not difficult to think about many cases in which the approach will not be successful. The paper does not present useful theory or analysis on the conditions under which the approach is likely to succeed. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is generally clear. It is well organised.

Originality: To the best of my knowledge, the paper is novel in the particular approach being proposed. That said, I would not rate the paper as particularly innovative. Using pre-defined skills without modification is a standard use of the options framework. 

Quality: The analysis in the paper is informative. However, it is essentially aiming to empirically show that the approach **can** be useful. But, in many cases, the approach will not be particularly useful, and can even be harmful. The results in the paper are not particularly informative towards understanding how useful the approach will be generally and the conditions under which it is likely to be useful. 


summary_of_the_review: The paper presents an approach to knowledge transfer between tasks. While the approach is sensible, and will produce good results in some cases, the results in the paper are not particularly informative towards understanding how useful the approach will be generally and the conditions under which the approach is likely to be useful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes SkillS, a method for reusing skills in new reinforcement learning tasks that employs a learned high-level scheduler to execute skills, and a learned new skill, which is included in the set the scheduler selects. In this way, the benefit of temporally extended exploration is retained by executing previously learned skills for numerous steps at a time, catastrophic forgetting is avoided by learning an entirely new set of parameters separate from the frozen skills, and the potential suboptimal convergence of HRL is avoided by allowing the new skill to be unconstrained by previously learned behaviors. They suitably liken their approach to related works that perform “transfer via data” (i.e. the frozen skills are not used in final evaluation, but rather to generate useful training data for the new unconstrained skill). The scheduler is trained via a discretized version of MPO, and the new skill is learned using CRR. In a number of robotic manipulation and locomotion domains, the authors empirically demonstrate the performance gains achieved by SkillS relative to relevant approaches in the literature. The authors also include additional results, including a demonstration of SkillS’ ability to learn with different quantities/quality of skills, analysis of the selection trends of the scheduler, analysis of the utility of flexible temporal abstraction versus fixed, and a study of the importance of having separate collect/infer mechanisms. 
strength_and_weaknesses: Strengths: 

* The exposition is detailed and clear.

* The problem is well defined and compelling and the resulting method is well-motivated. The discussion of prior work is comprehensive, and is presented in a way that effectively feeds into the motivation, and even makes the new method seem inevitable.

* The design decisions are generally well-supported, are described clearly and concisely in the main body (and in even more detail in the Appendix). 

* The environments chosen for evaluation are challenging and diverse. 

* The experimental results are generally compelling and comprehensive. The expected ablations and comparisons are performed.

* Useful avenues for future work are discussed in the conclusion.

Weaknesses/questions: 

* None major.

* How is the dependence on the frozen skills reduced over the course of training, if the scheduler is encouraged to maximize reward? Does this simply occur naturally? This is how I currently understand it. 

* Related to the previous question: did the authors try to regularize learning in any way to encourage the scheduler to choose the new skill more often? Or did this always naturally emerge? I would imagine issues with this not encouraging this; e.g. the scheduler rarely choosing the new skill in certain situations (relying on frozen skills in these), and then come evaluation, the new skill will perform poorly in these cases. Is there any intuition for why this does not (seem to) occur? 

* The results seem to demonstrate that variable temporal abstraction is necessary for best performance, which is an interesting finding. However, the authors appear to compare against only one constant for the fixed temporal abstraction results. How was 200 steps chosen? Can we see results with multiple other values to be convinced that the variable choice matters, and not that the fixed value is merely a poor choice? It could be helpful to compare the variable SkillS to the fixed temporal abstraction SkillS with the constant set to be the average skill length discovered by variable SkillS. This would more convincingly demonstrate the benefit of variable length skills. With my understanding of the current results, I’m not convinced.

* Figure 7a: A little confused by this graph. Are these curves generated using different manually specified schedulers that were otherwise trained with the same learning procedure (i.e. “New Skill” is the performance of the scheduler only choosing the new skill after being trained the same way as the other too)? This seems to be the case, but either way, readers may benefit from a clearer explanation of this. 
clarity,_quality,_novelty_and_reproducibility: * This paper is generally clear and of high quality, except for a few minor points raised in the above section.

* The method and analysis appear to be novel. 

* There is no reproducibility statement or promise to publish code, but the experimental details in the appendix appear to be comprehensive, and the main ideas in the work are simple enough that they should be able to be applied by future work even without the precise implementation details. 
summary_of_the_review: * The paper is well-written and detailed, the method is well-motivated and grounded in relevant prior work, and the results are generally compelling and comprehensive. There are a few minor concerns/questions I have for the authors, but otherwise I am in favor of recommending this paper for acceptance, as I believe it will benefit the research community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the problem of transferring learned policies to a new target task sequentially. A new method SkillS is proposed for the problem which separates the data collection from solution inference. During data collection phase, a high-level schedule chooses the best learned skill to maximize the task reward obtained from the environment. The scheduler is trained with a proposed algorithm, Hierarchical Categorical MPO. The task solution is learned via an off-policy learning algorithm that optimists a new skill for the given task. The scheduler can learn to sequence temporally-extended skills and bias exploration towards useful regions of trajectory space, and the new skill can then learn an optimal task solution off-policy. Through experiments, the authors show the proposed method outperforms related method across all sequential tasks. 
strength_and_weaknesses: Strength: The idea of learning new policy with data generated by following sequencing policies makes sense to me. Especially for complex tasks that can be devide into subtasks, the proposed method will lead to an efficient learning strategy which gradually learn better about the entire task. Also the proposed method is shown to be significantly better than others on sparse-reward manipulation tasks. The sparsity of the reward can be handled by gradually learning subtasks. 

Weakness: There is no convergence analysis for the proposed HCMPO algorithm. Without any assumption on the continuity of the subtasks, it is not clear that the proposed algorithm is guaranteed  to converge to the optimal policy. It will be better if the authors can provide certain analysis on this. 
clarity,_quality,_novelty_and_reproducibility: This paper is well written. And the proposed work is novel.
summary_of_the_review: The idea in this paper makes sense to me. The proposed algorithm is shown to perform significantly better than other on certain tasks.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper tackle the problem of efficient transfer by leveraging both hierarchy and KL-regularisation. The authors claim that fine-tuning, hierarchical methods and imitation based approaches can all fail and therefore propose a method that combines elements from all of them. The authors perform experiments on robotic object-stacking tasks and gait learning and see consistent improvements.
strength_and_weaknesses: # Strengths
- The work is well situated with respect to previous work
- The environments on which the methods are evaluated are challenging
- The authors provide a set of ablations

# Weaknesses
- The experimental details are not clear. For example, how do you leverage previous skills across seeds? How are the skills transfered in the Lift task? Why the need for adding CRC to the mix?
- The authors make multiple hand-wavy claims about how existing methods "can" fail. Any method can fail on a specific task, and saying that something "can" fail is not a great argument, as likely the new proposed approach "can" also fail.
- The experiments are done with only 5 seeds, which to me is a deal breaker. This brings a lot of questions as to the reproducibility of the results. I also couldn't find any indication of what the shaded regions represent.
clarity,_quality,_novelty_and_reproducibility: The clarity of the paper is good in terms of situating the method with respect to previous literature (although the related works section is a bit of an overload of citations. For example there are 10 citations to mention the idea of latent continuous space). The clarity and quality of the method is not great, as there are arbitrary choices that are not explained. Is the skill length random? It seems to be so looking later at the experiments. The section on data augmentation is completely disregarding work on intra-option learning that leverages more data.
In terms of reproducibility I have made this concern obvious, but I can't see how 5 seeds are enough for these hard exploration, difficult tasks.
summary_of_the_review: The authors tackle an important problem, but the proposed solution is not clearly motivated and presented. The experimental details bring major concerns in terms of reproducibility.

======================================================

After discussion with the authors, I have updated my score. The main factor is the acknowledgement that reproducibility is crucial and that the authors have attempted to add more seeds to their experiments. In general the paper still seems to add many elements to the algorithm, yet some of these choices are not well motivated. For example, the authors mention that CRR significantly helps for the off-policy problem, yet looking at Figure 14 there is no significant statistical difference. This same conclusion can also be drawn by Figure 15 concerning data augmentation. Overall, this also points towards the importance of having more seeds such that the significance of these kind of choices are clearly seen.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method for reusing temporally-extended skills in new reinforcement learning tasks.
The proposed approach addresses some of the issues affecting related works.
The effectiveness of the proposed solution is demonstrated by empirical validation using robotic manipulation and locomotion domains.
After reading each others' reviews and the authors' feedback, the reviewers discussed their opinions about this paper, without reaching a consensus.
Although the authors' answers effectively solved some of the issues raised by the reviewers, they still complain that the proposed approach contains many choices that are not well-motivated and whose utility (in some cases) does not seem to be significant.
As the approach is not supported by a theoretical analysis, it is very important to have relevant empirical results that help in advance our comprehension of the transfer of knowledge among reinforcement learning tasks.
We encourage the authors to consider reviewers' suggestions while preparing a new version of their paper.
justification_for_why_not_higher_score: The contribution of this paper is empirical, but the lack of motivation for some choices and the low significance of some experiments put this paper slightly below the bar.
Decision: Reject