Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents an algorithm for skill based model based RL, where the model is not learned as a mapping from state and action to next state but over outcomes of performing skills. The model is learned with a VEA style loss where a skill embedding is learned and a dynamics model in this embedding space is learned together with the skill from offline data. This setup is then used to learn a downstream skill for a new task using the learned dynamics model in a MBRL way. The paper presents experiments on a maze task and a franka kitchen environment where the presented method outperforms the baseline methods. The paper also presents ablation studies.
strengths_and_weaknesses: Strengths:

I think the paper is well written and presents clearly the problem setting, the algorithm and the contribution. I also think the authors are making clear how their approach differs from the related work. The topic and the direction of the paper is very relevant of learning models over higher abstractions instead of low-level sensor measurements and the paper presents convincing results that this might be a good avenue to investigate further. I think this paper would make a clear contribution to the field.

Weaknesses:
- it would be very good and important to know more about the offline data that was used, how close is it to the downstream tasks? How much data was used? I think these details are important to understand the generality of this approach

- robot experiments are missing, how would the transfer look like? still your method requires at least 250k samples, even after all of the pre-training that happened - could you maybe elaborate on this and comment on the real world applicability of the approach?
minor:
- an overview image of the algorithm would be very helpful to understand quicker how the learning setup looks like.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I am willing to adjust my assessment during the rebuttal after reading the authors response on the above mentioned points.
issues: see above under weaknesses.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes an approach for hierarchical model-based reinforcement learning, where the agent first learns a repertoire of skills, and a skill dynamics model from offline dataset, and then trains a hierarchical policy conditioned on the skill space. 

The key contributions of the paper are jointly learning a skill prior and a skill dynamics model from offline data, and demonstrating that the a policy conditioned on skills sampled from the skill prior can be used to guide online exploration in downstream tasks. The paper demonstrates results with the proposed algorithm on simulated robot manipulation (Frank Kitchen) and point agent navigation (maze) environments involving long-horizon tasks. 
strengths_and_weaknesses: Strengths:

1. The key insight of learning re-usable skills from offline data and using the learned skill distribution to condition policies for downstream tasks is very well motivated, and relevant to a lot of robot learning settings. 

2. The paper is well-written. The algorithm design is well motivated, and easy to follow. The related work positions the paper well in the context of prior model based hierarchical RL works. Sufficient details are provided in the experiments to understand the setting and results well. 

3. The modifications of the method over prior work (e.g. Dreamer) are explained well, and logically positioned. 

Weaknesses:

1. The paper's insights regarding learning a skill space to condition a policy in model-based RL, overlaps with several prior works as mentioned in the related works section [28, 5, 29, 30]. However, The point of distinction mentioned "To fully unleash the potential of temporally abstracted 84 skills, we devise a skill-level dynamics model to provide accurate long-term prediction, which is
85 essential for solving long-horizon tasks." is not convincing to me - I don't see any reason for the skill-space to be enable long-term prediction because although the skills are low-dimensional actions (as learned by a VAE latent embedding), there is no temporal chaining of skills - skills are sampled IID from the same Gaussian distribution at each time-step, and this is infact exactly similar to the prior works cited previously. 

2. All the loss terms in equation (5) are weighted equally - it is unclear why this modeling choice makes sense. Since all the terms in equation (5) are ad-hoc and do not correspond to a principled variational lower bound to the overall RL objective, I believe it is necessary to motivate their relative importance, and ablate this choice through experiments. 

3. It is unclear how it is ensured that the policy conditioned on the skill actually makes use of the skill and doesn't simply ignore it? There is no loss term in the objective to actually incentivize making the skills chosen predictable from the observed rollouts. In addition, this is not evaluated through the experiments. 

4. The experiment details and results are thorough in the settings considered, but several relevant baselines are not evaluated. The baseline Dreamer is simply a flat model-based RL approach (no hierarchy) and all the other baselines are ablations of the approach. I think it is very important to compare with several external hierarchical RL approaches proposed in the literature over the years, for example [A] HIRO and [B] DADS, the latter of which is very relevant in learning a skill-level dynamics model. Also I think [A] should be cited in the related works. 

[A] Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in neural information processing systems 31 (2018).
[B] Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019). Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657.



robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I believe the paper targets an important problem in robotics, that of learning re-usable skills from prior data. However, due to the limitations I pointed out in the weaknesses section, I believe the paper is not ready for publication in its current form. 
issues: Kindly explain the issues I listed (1-4) in the Weaknesses section.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The work proposes to combine option (skill)-based learning with model-based RL, to achieve a more sample efficient and robust (higher probability of success) approach. The proposed approach relies on demonstrations to pre-train several model components: the state representation module (E_\psi) and the skill dynamics module (D_\psi), as well as the state decoder (O_\theta) in order to avoid trivial solutions. These pre-trained components are then used in the second phase of training which involves fine tuning them on a rewards-prediction/latent-state prediction, and finally an RL based objective that performs model-predictive control using a recently proposed method, TD-MPC.  Experiments on the maze and FrankaKitchen suites from D4RL demonstrates the method's superior performance. Ablation studies demonstrate the utility of jointly training the skill-representation component & skill-dynamics model. 
strengths_and_weaknesses: Strengths: 
1-The paper's language is pretty clear and it is well written 
2-The performance gains seem impressive on the tasks presented (although I am not familiar with the performance of related works)

Weaknesses: 
1-The method seems *extremely* complicated and would be difficult to reproduce in my opinion. For example, the total number of terms to minimize in all objective functions is 10, and the total number of "model components" (i.e. networks) is 9 (!!!). 
2-The method omits the "Androit" and Gym suite of tasks from D4RL although they seem to be well suited (i.e. they all include demonstrations). Why not include these tasks as well? 
3-Many implementation details are missing, not least the hpram details. 
4-Despite the paper being generally well written, some technical details require major clarifications.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Despite the paper presenting good performance on 2 our of the 4 D4RL suites, I recommend rejection of the paper in the current form. 
The main reasons being: 1-the complexity of the proposed approach, and lack of technical details, makes it unreproducible in my opinion -- and therefore of limited value to the robotics/learning community. 2-The proposed model was only demonstrated on two tasks. Neither of which included real robots. This makes me question the robustness and generality of the proposed approach. 
issues: 1-Do E & D share parameters? If not, then subscripting both with \psi is very confusing.
2-Reconstructing the observation space is known to be problematic, i.e. there are simple solutions that do not capture the details relevant to the problem. Can you give more details on this in the context of your proposed approach? 
3-In Equation 2 is z in the first term output by q_\theta (of the second term)? 
4-Also Equation 2. Why do you apply tanh to the Gaussian samples, i.e. why is it important to clamp their values to be in [-1, 1]? 
5-Equation 3: is E_{\psi^-} a typo (last term)? Or is it some sort of out-dated parameter vector used in the loss? 
6-How is the horizon length (skill vector temporal length) found? 
7-What is "sg" in Equation 4? 
8-Are there no multipliers on the various loss terms in Equation 6? i.e. are they all 1? If so, why? If not, how are the hparams tuned (in all loss terms)?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents Skill-based Model-based RL framework to efficiently learn downstream policies by jointly training a skill dynamics model and skill encoding networks from task-agnostic data. The skill dynamics model learns temporal consistency by predicting the final state after performing a skill (a sequence of actions) from a given state. Skills are encoded in a latent skill space, the skill dynamics model is used to plan in this latent space using cross-entropy methods and skills sampled from a task-level policy. Trajectories from the dynamics model rollouts are evaluated with learned reward and Q functions, with the best trajectories decoded into action sequences that are executed in the environment. Additional fine-tuning of the skill dynamics model is performed during downstream task learning. The framework is demonstrated in simulation with a maze task and a manipulation task, performing significantly better than separate model-based and skill-based methods.


strengths_and_weaknesses: This paper presents a combination of techniques that improve on model-based and skill-based methods. The ideas and motivations are expressed clearly and the paper is well written. The comparisons appear relevant and fair, the results show a clear benefit of the presented method. The ablation study highlighted the benefit of the skill dynamics model and of joint training. The extensive appendix was welcomed and the additional experiments that didn’t make it into the paper shone further light on the method and the improvements over existing approaches. 

The primary weakness (apart from the lack of real robot experiments) was confusion through the method section, particularly Section 3.2.1 where components were introduced, but not defined until the following section (3.2.2). For example, there is no mention of “h” before equation 1. It might be better to move some definitions into 3.2.1, or re-work these two sections. Also consider re-ordering the list in Section 3.2.1 to follow the flow of learning: first train on task-agnostic data, then fine-tune / train on downstream tasks. 

Questions:

Was there a benefit using the state representation and observation modules? Were these modules used in the baselines? Is there utility in training these with target networks?

Was the maze navigation task different in this work compared with the SPiRL paper? The performance of SPiRL seems much worse than the original paper, is this due to the original paper using more goal–reaching trajectories in their training data?

What is the reason success on all four sequential tasks in the Kitchen environment is difficult? Do the authors have an intuition about how to more robustly solve longer task sequences?

What is the effect of fine-tuning after training? Is this expected to be a significant component for real-world experiments? 

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The recommendation is to improve the clarity of the method section.
issues: Is the negative expectation incorrect in equation 4 (negative expectation), minimising this loss will increase the KL divergence? This seems to be the opposite of equation 2. 

Target networks were used through the Algorithms in the Appendix, but were not introduced in the paper. This is not really an issue if this detail was omitted for clarity, except for the superscript “-” on the model parameters for Q and E that was not explained anywhere. 

Modules were given different names throughout, for example, O is introduced as the reconstruction module, though later referred to as the observation, and as the state decoder. Similar for E. Also, what is “sg” (equation 4 and 7)?

Figure 2: “Prior” should be changed to “Skill Prior” to differentiate this from the fixed-prior used to regularise the skill encoder / decoder. q(z|s0,a) takes a vector of actions? Maybe make “a” bold to highlight this. No model actually takes “s” as input except the skill encoder, is that correct? Instead, “h” is used from the state encoder. It makes sense this is omitted from the diagram for clarity, but should probably be mentioned in the caption or where the figure is introduced in the text.

Equation 7, should be pθ, not p_phi to regularise the skill prior pθ? Also, should be π_phi not πθ to regularise the policy, not the skill decoder?

Minor:
Figure 1: ”Skill dynamics model with skills” maybe just ”Skill dynamics model” is sufficient. 

Minor:
There were two mentions of furniture assembly as a complex long horizon task. I was expecting this as an experiment with improvements on the millions to billions of interactions. Perhaps include that maze navigation and kitchen tasks are also very complex long horizon tasks.

Spelling / Grammar / Typos

“first work that jointly learning”

“for more sample efficiency policy learning”

“Conventionally model-based approaches” -> missing comma, or change to “Conventional”

“SkiMo is the first work that jointly learning skills” 

“high level policy” or “high-level policy”, both used throughout.

“crucial in the real world robot learning”
review_title: null

Decision: Accept (Poster)