Skill-based Model-based Reinforcement Learning
Authors: Lucy Xiaoyang Shi
Joseph J Lim
Youngwoon Lee
Venue: CoRL 2022 Poster (2022)
Keywords: Model-Based Reinforcement Learning
Skill Dynamics Model

Abstract: Model-based reinforcement learning (RL) is a sample-efficient way of learning complex behaviors by leveraging a learned single-step dynamics model to plan actions in imagination. However, planning every action for long-horizon tasks is not practical, akin to a human planning out every muscle movement. Instead, humans efficiently plan with high-level skills to solve complex tasks. From this intuition, we propose a Skill-based Model-based RL framework (SkiMo) that enables planning in the skill space using a skill dynamics model, which directly predicts the skill outcomes, rather than predicting all small details in the intermediate states, step by step. For accurate and efficient long-term planning, we jointly learn the skill dynamics model and a skill repertoire from prior experience. We then harness the learned skill dynamics model to accurately simulate and plan over long horizons in the skill space, which enables efficient downstream learning of long-horizon, sparse reward tasks. Experimental results in navigation and manipulation domains show that SkiMo extends the temporal horizon of model-based approaches and improves the sample efficiency for both model-based RL and skill-based RL. Code and videos are available at https://clvrai.com/skimo

Skill-based Model-based Reinforcement Learning
Lucy Xiaoyang Shi'
Joseph J. Lim 2.3+
Youngwoon Lee" NAVER AI Lab
University of Southern California
KAIST
Abstract: Model-based reinforcement learning (RL. is a sample-efficient way of learning complex behaviors by leveraging learned single-step dynamics model to plan actions in imagination. However, planning every action for long-horizon tasks is not practical, akin to a human planning out every muscle movement. Instead, humans efficiently plan with high-level skills to solve complex tasks. From this in- tuition, we propose Skill-based Model-based RL framework (SkiMo) that enables planning in the skill space using a skill dynamics model, which directly predicts the skill outcomes, rather than predicting all small details in the intermediate states, step by step. For accurate and efficient long-term planning, we jointly learn the skill dynamics model and skill repertoire from prior experience. We then harness the learned skill dynamics model to accurately simulate and plan over long horizons in the skill space, which enables efficient downstream learning of long-horizon, sparse reward tasks. Experimental results in navigation and manipulation domains show that SkiMo extends the temporal horizon of model-based approaches and improves the sample efficiency for both model-based RL and skill-based RL. Code and videos are available at https stttps///clvve //llvllll.a..////////////a/...l. com/skimo.
Keywords: Model-Based Reinforcement Learning, Skill Dynamics Model
Introduction
A key trait of human intelligence is the ability to plan abstractly for solving complex tasks [1]. For instance, we perform cooking by imagining outcomes of high-level skills like washing and cutting vegetables, instead of planning every musele movement involved [2]. This ability plan with temporally-extendee skills helps to scale our internal model to long-horizon tasks by reducing the search space of behaviors. To apply this insight to artificial intelligence agents, propose novel skill-based and model- reinforcement learning (RL.) method, which learns a model and a policy in high-level skill space, enabling accurate long-term prediction and efficient long-term planning.
Typically, model-based RL involves learning a flat single-step dynamics model, which predicts the next state from the current state and action. This model can then be used to simulate "imaginary" trajectories, which significantly improves sample efficiency over their model-free alternatives [3, 4]. However, such model-based RL methods have shown only limited success in long-horizon tasks due to inaccurate long-term prediction [5] and computationally expensive search [6, 7, 8).
Skill-based RL enables agents to solve long-horizon tasks by acting with multi-action subroutines (skills) [9, 10, 11, 12, 13, 14] instead of primitive actions. This temporal abstraction of actions enables systematic long-range exploration and allows RI agents to plan farther into the future, while requiring a shorter horizon for policy optimization, which makes long-horizon downstream tasks more tractable. Yet, on complex long-horizg horizon tasks, skill-based RL still requires a few million to billion environment interactions to learn [13], which is impractical for real-world applications.
To combine the best of both model-based RL and skill-based RL, we propose Skill-based Model- based RL (SkiMo), which enables effective planning in the skill space using a skill dynamics model. Given state and a skill to execute, the skill dynamics model directly predicts the resultant state after skill execution, without needing model every intermediate step and low-level action (Figure 1), whereas the flat dynamics model predicts the immediate next state after one action execution. Thus,
AI Advisor at NAVER Al Lab
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Open microwave
Move kettle
(a) Flat dynamies model without skills
(b) dynamies model skills
(e) Skill dynamics model with skills
Figure 1: Intelligent agents can use their internal models imagine potential futures for planning.
Instead of planning out every primitive action (black arrows in a), they aggregate action sequences into skills (red and blue arrows in b). Further, they can leap directly to the predicted outcomes of executing skills in sequence (red and blue arrows in c), which leads to better long-term prediction and planning compared to predicting step-by-step (blurriness of images represents the level of error accumulation in prediction).
planning with skill dynamics requires fewer predictions than flat dynamics, resulting more reliable long-term future predictions and plans.
Concretely, first jointly learn the skill dynamics model and skill repertoire from large offline datasets collected across diverse tasks [15, 12, 16]. This joint training shapes the skill embedding space for easy skill dynamics prediction and skill execution. Then, to solve a complex downstream task, we train high-level task policy that acts in the learned skill space. For more efficient policy learning and better planning, we leverage the skill dynamics model to simulate skill trajectories.
The main contribution of this work is to propose Skill-based Model-based RL (SkiMo), a novel sample-efficient model-based hierarchical RL algorithm that leverages task-agnostic data extract not only a reusable skill set but also a skill dynamics model. The skill dynamics model enables efficient and accurate long-term planning for sample-efficient RL. Our experiments show that method outperforms the state-of-the-as skill-based and model-based RL algorithms on long-horizon navigation and robotic manipulation tasks with sparse rewards.
2 Related Work
Model-based RL leverages (learned) dynamics model of the environment plan sequence of actions that leads to the desired behavior. The dynamics model predicts the future state of the environment after taking specific action, which enables simulating candidate behaviors in imagination instead of in the physical environment. Then, these imaginary rollouts can be used for planning [8, 4] through, e.g., [17[ and MPPI [18], well as for policy optimization [19, 3, 20, 4] to improve sample efficiency. Yet, due to the accumulation of prediction error at each step and the increasing search space, finding an optimal, long-horizon plan is inaccurate and computationally expensive (6. 7, 8).
To facilitate learning of long-horizon behaviors, skill-based RI. lets the agent act over temporally- extended skills (i.c. options [21] or motion primitives [22]). which can be represented as sub- policies or coordinated sequence of low-level actions. Temporal abstraction effectively reduces the task horizon for the agent and enables directed exploration [23]. The reusable skills can be manually defined [22, 24, 10, 11, 14], extracted from large offline datasets [15, 25, 26, 27, 28], discovered online in an unsupervised manner [29, 30], or acquired in the form of goal-reaching policies [31, 32, 33, 34, 20). However, skill-based RL is still impractical for real-world applications, requiring a few million to billion environment interactions [13]. In this paper, we use model- based RL guide the planning of skills to improve the sample efficiency of skill-based approaches.
There have been attempts to plan over skills in model-based RL. [29, 35, 5, 36, 37J. However, most of these approaches [29, 5, 36] still utilize the conventional flat (single-step) dynamics model, which

---

Skills & Skill Dynamics Model
2
Downstream RL
Qis, 2)
Environment
Reward
Rix, 1,,,1,,00
Task Policy
Sidll Encoder
Skill Dynamics
qicl ss a)
Dis Sp.
Skill Prior
(a, (x
p(z sp)
x x'(a,1% 1 2)
Skill Policy
GEM
Policy
Our approach, SkiMo, combines model-based RL and skill-based RL for sample efficient learning of long-horizon tasks, SkiMo consists of two phases: (1) learn a skill dynamics model and a skill repertoire from offline task-agnostic data, and (2) learn high-level policy for the downstream task by leveraging the learned model and skills. We omit the encoded latent state h in the figure and directly write the observation for clarity, but most modules take the latent state as input.
Figure 2:
struggles at handling long-horizon planning due to error accumulation. Wu et al. [35] proposes to learn capporally-extended dynamics model; however, conditions on low-level actions rather than skills and is only used for low-level planning. A concurrent work, Shah et al. [37], is most similar to our work in that learns skill dynamics model, but with limited set of discrete,
camuulll-deffnne
skills. To fully unleash the potential of temporally abstracted skills, we extract the skill space from data and devise a skill-level dynamics model to provide accurate long-term prediction, which essential for solving long-horizon tasks. To the best of our knowledge, SkiMo is the first work that jointly learns skills and a skill dynamics model from data for model-basee RL.
Method
To enable accurate long-term prediction and efficient long-horizon planning for RI, we introduce SkiMo, a novel skill-based and model- RL algorithm that shares synergistic benefits from both frameworks. A key change prior model-based approaches is the of a skill dynamics model that directly predicts the outcome of a chosen skill, which enables efficient and accurate long-term planning. As illustrated in Figure 2, approach consists of two phases: (1) learning the skill dynamics model and skills from an offline dataset (Section 3.3) and (2) downstream task learning with the skill dynamics model (Section 3.4).
3.1 Preliminaries
RL. We formulate problem as a Markov decision process [38], which defined by a tuple (S, R. Po.o.) 7) of the state space S, action space A, reward R(s, a), transition probability P(s' is, a), initial state distribution Po. and discounting factor 7. A policy =(als) maps from state to action a. RL aims to find the optimal policy that maximizes the expected discounted
return, E Psorpoo (ss......... (T, )r
ET 7a R(s., a.) where T; is the variable episode length.
Unlabeled Offfine Data We assume access to reward-free task-agnostic dataset [15, 12], which is set of N state-action trajectories, D = [TI,..., J. Since it is task-agnostic, this data can be collected from human teleoperation, unsupervised exploration, training data for other tasks. We do not assume this dataset contains solutions for the downstream task; therefore, tackling the downstream task requires re-composition of skills learned from diverse trajectories.
Skill-based RL
We define skills as sequence of actions (ao...., a with a fixed horizon² H and parameterize skills skill latent and skill policy, π (a]s, z). that maps skill latent and state to the corresponding action sequence. The skill latent and skill policy can be trained using variational
It is worth noting that our method is compatible with variable-length skills [26, 39, 27] and goal-conditioned skills [20] with minimal change; however, for simplicity, adopt fixed-length skills of H 10 in this paper.

---

auto-encoder (VAE [40]), where a skill encoder ( a)o:H-1) embeds a sequence of transitions into a skill latent z, and the skill policy decodes back the original action sequence. Following SPiRI [12], we also learn skill prior p(z]s), which is the skill distribution in the offline data, to guide the downstream task policy to explore promising skills over the large skill space.
3.2 SkiMo Model Components
SkiMo consists of three major model components: the skill policy (TT), skill dynamics model (Dy). and task policy (T along with auxiliary components for representation learning and value estimation. A state encoder Eg first encodes an observation s into the latent state h. Then, given skill z, the skill dynamics D predicts the skill effect the latent space. The task policy Tg. reward function Rg, and value function Qu predict a skill, reward, and value on the (imagined) latent state, respectively. The following a summary of the notations of our model components:
State encoder:
he Es
Skill dynamies:
h 11 DΦs (he, 2x)
Observation decoder:
S (o(h)
Task policy:
2, ~ T (hr)
Skill prior:
á, ~ po (s:)
(1)
Reward:
P. Rd (he, Zi)
Skill encoder:
ZE (o, a)1++H-1)
Value:
G2 = (be, 2r)
Skill policy:
Fh (8t, 24)
For convenience, we label the trainable parameters v) 0, & of each component according to which phase they are trained on:
1. Learned from offfine data and finetuned in downstream RL (e+ = { d' The state
encoder (Eg) and the skill dynamics model (Dx) are first trained the offline task-agnostic data and then finetuned in downstream RL to account for unseen states and transitions.
2. Learned only from offline data (0 {0o, Op 0z²² }):
: The observation decoder (Og), skill encoder (go), skill prior (po), and skill policy (TrE) are learned from the offline data.
3. Learned in downstream RL (Ø {Øo. R,, )): The value (Q&) and reward (Rş) functions, and the task policy (T&) are trained for the downstream task using environment interactions.
3.3 Pre-Training Skill Dynamies Model and Skills from Task-agnostic Data
SkiMo consists of pre-training and downstream RL phases. In pre-training, SkiMo leverages offline data to extract (1) skills for temporal abstraction of actions, (2) skill dynamics for skill-level planning on a latent state space, and (3) skill prior [12] to guide exploration. Specifically, we jointly learn skill policy and skill dynamics model, instead of learning them separately [35, 5, 36], in self- supervised manner. The key insight is that this joint training could shape the latent skill space Z and state embedding that the skill dynamics model can easily predict the future.
In contrast to prior works that learn models completely online [3, 41, 4], we leverage existing offline task-agnostic datasets to pre-train skill dynamics model and skill policy, This offers the benefit that the model and skills are agnostic to specific tasks so that they may be used in multiple tasks. Afterwards in the downstream RL phase, the agent continues to finetune the skill dynamics model to accommodate task trajectories.
To learn -iwddmensional skill latent space Z that encodes action sequences, train conditional VAE [40, 42] on the offline dataset that reconstructs the action sequence through a skill embedding given state-action sequence in SPiRI. [12, 16). Specifically, given H consecutive states and actions (s, a)0.H-1, a skill encoder qo predicts a skill embedding z and a skill decoder my (i.e. the low-level skill policy) reconstructs the original action sequence from z:
LVVE -B(g,))ow--IVD
ABC H
TT (s,, z) 2,)²+B. KL( a)u: 11) p(z))
(2)
Behavioral cloning
Embedding regularization
where z is sampled from go and Apc- 3 are weighting factors for regularizing the skill latent z distribution a prior of tanh--ransformed unit Gaussian distribution, Z tanh(A((0, 1)).
To ensure the latent skill space suited for long-term prediction, we jointly train a skill dynamics model with the VAE above. The skill dynamics model Iearns to predict h H. the latent state H -teps

---

ahead conditioned on skill for N sequential skill transitions using the latent state consistency loss [4]. To prevent a trivial solution and encode rich information from observations, we additionally train an observation decoder Og using the observation reconstruction loss. Altogether, the skill dynamics Das, state encoder Fej, and observation decoder Og are trained on the following objective:
LREC = ((E..)..NN
Aolls,n - Oo Eig (s. 1 A (ban, zar) - F (8
Observation reconstruction
Latent consistency
(3)
where λo, År, are weighting factors and ho Egg (So) and h(4+1)H Do (har, 2;Hr) such that gradients are back-propagated through time. For stable training, we use a target network whose parameter s is slowly soft-copied from gb.
Furthermore, to guide the exploration for downstream RL, we also extract a skill prior [12] from offline data that predicts the skill distribution for any state. The skill prior is trained by minimizing the KI. divergence between output distributions of the skill encoder qo and the skill prior po:
Ssp E(ssa)o.n---22 Asp KL sg go (z) (Z/S0: ao:H |I po (o(2)))))
(4)
where Asp is weighting factor and sg denotes the stop gradient operator.
Combining the objectives above, we jointly train the policy, model, and prior, which leads to well-shaped skill latent space that is optimized for both skill reconstruction and long-term prediction:
L - + LRCC + Lsp
(5)
3.4 Downstream Task Learning with Learned Skill Dynamics Model
To accelerate downstream RL with the learned skill repertoire, SkiMo learns a high-level task policy T (z h1)) that outputs latent skill embedding z₁, which is then translated into a sequence of H actions using the pre-trained skill policy Th to act in the environment [12, 16].
To further improve the sample efficiency, we propose to use model-based RL in the skill space by leveraging the skill dynamics model. The skill dynamics model and task policy can generate imaginary rollouts the skill space by repeating (1) sampling a skill, Z, ~ (hh)). and (2) predicting H-step future after executing the skill, h++ D& (h,, z,). Our skill dynamics model requires only 1/H dynamics predictions and action selections the flat model-based RL approaches [3, 4], resulting in more efficient and accurate long-horizon imaginary rollouts (see Appendix, Figure 10).
Following TD-MPC [4), we leverage these imaginary rollouts both for planning (Algorithm 2) and policy optimization (Equation (7)), significantly reducing the number of necessary environment interactions. During rollout, we perform Model Predictive Control (MPC), which re-plans every step using CEM and executes the first skill of the skill plan (see Appendix, Section C for more details).
To evaluate imaginary rollouts, we train a reward function R (h,, zı) that predicts the sum of H H-step rewards Ft, and Q-value function Qs (h,, Zz). We also finetune the skill dynamics model D& and state encoder Egg on the downstream task to improve the model prediction:
CREE = Ey, M
D & (b,,, z.) Es (st+ + AuIlre Rsf (h, 2
Latent consiscency
Reward prediction
(6)
+ Avllrx + 7Q0 (be+ H: To (h. H)) Q (he, 2:
Value prediction
Finally, train high-level task policy Te to maximize the estimated Q-value while regularizing it to the pre-trained skill prior po [12], which helps the policy output plausible skills:
LR -Eg,>D[- Q (h, To (g)h))))
KL TT (z: ((())))) )) po pz (s))].
The models (v), ②) are trained to minimize Equation (6) and Equation (7) using fack-propagation through time over N consecutive skill-level transitions, similar to Equation (3).
For clarity, re to abbreviate the sum of H-step environment rewards Σ#11 R[ssti, acti).
5

---

(a) Maze
(b) Kitchen
(c) Mis-aligned Kitchen
(d) CALVIN
Figure 3: We evaluate our method on four long-horizon, sparse reward tasks. (a) The green point mass navigates the maze reach the goal (red). (b, c) The robot arm in the kitchen must complete four tasks in the correct order, Microwave Kettle Bottom Burner Light and Microwave Light
Slide Cabinet Hinge Cabinet. (d) The robot arm needs to complete four tasks in the correct order, Open Drawer Turn on Lightbulb Move Slider Left Turn on LED.
Experiments
In this paper, propose SkiMo, model-based RL approach that can efficiently and accurately plan long-horizon trajectories by leveraging skills and skill dynamics model. our experiments, we aim to answer the following questions: (I) Can the skill dynamics model improve the efficiency of RL for long-horizon tasks? and (2) Is the joint training of skills and the skill dynamics model essential for efficient model-based RL?
compare SkiMo with prior model-based RL and skill-based RL methods on four long-horizon tasks with sparse rewards: maze navigation, two kitchen manipulation, and tabletop manipulation tasks, as illustrated in Figure More experimental details can be found in Appendix, Section C.
4.1 Tasks
Maze We use the maze navigation task from Pertsch et al. [16], where point mass agent is randomly initialized near the green region and needs reach the fixed goal region in red (Figure 3a). The agent observes its 2D position and 2D velocity, and controls its (x, y)-velocity. The agent receives sparse reward of 100 only when it reaches the goal. The task-agnostic dataset [16] consists of 3,046 trajectories between randomly sampled initial and goal positions.
Kitchen We the FrankaKitchen tasks and 603 trajectories from D4RL [43]. The 7-DoF Franka Emika Panda arm needs to perform four sequential sub-tasks, Microwave Kettle Bottom Burner Light. In Mis-aligned Kitchen, also test another task sequence, Microwave Light Slide Cabinet Hinge Cabinet, which has a low sub-task transition probability in the offline data distribution [16]. The agent observes 1ID robot state and 19D object state, and uses 9D joint velocity control. The agent receives a reward of for every sub-task completion in order.
We adapt CALVIN [44] to have the target task, Open Drawer Turn Lightbulb Move Slider Left- Turn on LED, and 21D robot and object states. It uses the Franka Panda arm with end-effector pose control. The offline data is from play data [44] consisting of 1,239 trajectories. The agent receives a reward of for every sub-task completion the correct order.
CALVIN
4.2 Baselines
Dreamer [3] and TD-MPC [4] learn flat (single-step) dynamics and train a policy using latent imagination achieve high sample efficiency.
DADS [29] discovers skills and learns dynamics model through unsupervised learning.
LSP [36] plans the skill space, but using a single-step dynamics model from Dreamer [3].
SPiRL [12] learns skills and a skill prior, and guides high-level policy using the learned prior.
SPiRL Dreamer and SPiRL + TD-MPC pre-train the skills using SPiRI and learn policy and model in the skill space, instead of the low-level action space, using Dreamer and TD-MPC, respectively. In contrast to SkiMo, these basclines do not jointly train the model and skills.

---

Dreamer
SPIRI
SPIRL Dreamer
TD-MPC
ADMPL
DADS
LSP
SkiMo (Ours)
Oeeegge Succeso
Roveaae Stiteeds
02
10
15
623
9.20
835
3.0
624
0.2
0.55
130
2.
staps
(a) Maze
(b) Kitchen
(c) Mis- als-aleg Kitchen
(d) CALVIN
Figure 4: Learning curves of our method and baselines. All averaged over 5 random seeds.
4.3 Results
Maze Maze navigation poses hard exploration problem due to the sparsity of the reward: the agent only receives reward after taking 1,000+ steps to reach the goal. Figure 4a shows that only SkiMo is able consistently reach the goal, whereas baselines struggle learn a policy or accurate model due to the challenges in sparse feedhack and long-term planning.
We qualitatively analyze the behavior of each agent in Appendix, Figure 9. Dreamer and TD-MPC have a small coverage of the maze since is challenging to coherently explore for 1,000+ steps to reach the goal from taking primitive actions. Similarly, DADS and LSP could not learn meaningful skills and never find the goal. SPiRL able to explore a large fraction of the maze, but does not learn to consistently find the goal due to difficult policy optimization in long-horizon tasks. SPiRL + Dreamer and SPiRL TD-MPC fail to learn an accurate model and often collide with walls.
Kitchen Figure 4b demonstrates that SkiMo reaches the same performance (above sub-tasks) with less environment interactions than SPiRI. In contrast, Dreamer, TD-MPC, DADS, and L.SP rarely succeed on the first sub-task due to the sparse reward. SPiRI + Dreamer and SPiRL TDD-MC perform better than flat model-based RI. by leveraging skills, yet the independently trained model and policy are not accurate enough to consistently achieve more than two sub-tasks.
Mis-aligned Kitchen The mis-aligned target task makes the downstream learning harder because the skill prior, which reflects offline data distribution, offers less meaningful regularization to the policy. However, Figure 4c shows that SkiMo still performs well. This demonstrates that the skill dynamics model is able to adapt to the new distribution of behaviors, which might significantly deviate from the distribution in the offline dataset.
CALVIN One the major challenges in CALVIN is that the offline data is very task-agnostic: any particular sub-task transition has probability lower than 0. 11 average, resulting in large number of plausible skills from any state. Figure 4d shows that SkiMo can learn faster than the model-free baseline, SPiRL, which supports the benefit of using our skill dynamics model. Meanwhile, Dreamer performs better in CALVIN than in Kitchen because objects in CALVIN are more compactly located and easier manipulate with the end-effector control; thus, becomes viable to accomplish initial sub-tasks through random exploration. However, it falls short in composing coherent action sequences achieve a longer task sequence due to the lack of semporally-extendee reasoning.
In summary, we show the synergistic benefit of temporal abstraction in both the policy and dynamics model. SkiMo is the only method that consistently solves the long-horizon tasks. Our results also demonstrate the importance of algorithmic design choices (e.g. skill-level planning, joint training of model and skills) as naive combinations (SPiRL+ + Dreamer, SPiRL+ TD-MPC) fail to learn.
4.4 Ablation Studies
Model-based vs. Model-free In Figure 5, SkiMo achieves better asymptotic performance and higher sample efficiency across all tasks than SkiMo + SAC, which uses model-free RL (SAC [45]) to train the high-level policy to select skills. The comparison suggests that the task policy can make more informative decisions by leveraging accurate long-term predictions of the skill dynamics model.
Joint training skills and skill dynamics model SkiMo w/o joint training learns the latent skill space using only the VAE loss in Equation (2). Figure 5 shows that the joint training is crucial in
7

---

SkiMo [Ours]
SKIMO-SAC
SkiMo w/o joint training
SKiMo CRM
SkiMo single-step dynamics
beerge Sucoess
Acenape Stattades
Envirommnnt
1o
Envioomrent
8.%
123
6
0.7
Eistronment
13
(a) Maze
(b) Kitchen
(c) Mis- ali-aedg Kitchen
(d) CALVIN
Figure 5: Learning curves of our method and ablated methods. All averaged over random seeds.
more challenging scenarios, where the agent needs to generate accurate long-term plans (for Maze) the skills very diverse (in CALVIN).
CEM planning As shown in Figure 5, SkiMo learns significantly better and faster in Kitchen, Mis- aligned Kitchen, and CALVIN than SkiMo w/o CEM, indicating that CEM planning can effectively find a better plan. On the other hand, in Maze, SkiMo w/o CEM learns twice as fast. We find that action noise for exploration in CEM leads the agent to get stuck at walls and corners. We believe that with a careful tuning of action noise, SkiMo can solve Maze much more efficiently.
Skill dynamics model SkiMo w/ single-step dynamics replaces the skill dynamics model with conventional single-step dynamics model. Similar to LSP, learns and plans on skills, but additionally pre-trains the skills and the model on offline datasets for fair comparison. As shown in Figure 5, the single-step step model struggles at handling long-horing oon-hhorzoon planning on all tasks, akin the baseline results on DADS and LSP in Figure 4. In contrast, the skill dynamics model can make accurate long-horizon predictions for planning due to significantly less compounding errors.
For further ablations and discussion on skill horizon and planning horizon, see Appendix, Section A.
4.5 Long-horizon Prediction with Skill Dynamics Model
To assess the accuracy of long-term prediction of our proposed skill dynamics over flat dynamics, we visualize imagined trajectories in Appendix, Figure 10, where the ground truth initial state and sequence of 500 actions (50 skills for SkiMo) are given. Dreamer struggles make accurate long- horizon predictions due to eπгоr accumulation. In contrast, SkiMo is able to reproduce the ground truth trajectory with little prediction error even when traversing through hallways and doorways. This confirms that SkiMo allows temporal abstraction in the dynamics model, thereby enabling cenporally-extended prediction and reducing step-by-step prediction error.
Conclusion
We propose SkiMo, intuitive instantiation of saltatory model-basee hierarchical RL. [2], which combines skill-based and model-based RL approaches. Our experiments demonstrate that (1) a skill dynamics model reduces the long-term future prediction error via temporal abstraction in the dynamics model; (2) without needing to plan step-by-step. downstream RL. over the skill space allows for efficient and accurate temporally-exe -xxtende reasoning, improving the performance of prior model-based RL and skill-based RL; and (3) joint training of the skill dynamics and skills further improves the sample efficiency by learning skills conducive predict their consequences. We believe that the ability to learn and utilize skill-level model holds the key to unlocking the sample efficiency and widespread use of RL agents for long-horizon tasks, and our method takes a step toward this direction.
Limitations and future work While our method extracts fixed-length skills from offline data, the lengths of semantic skills may vary based on the contexts and goals. Future work can learn variable-length semantic skills to improve long-term prediction and planning. Further, although we only experimented on state-based inputs, SkiMo is general framework that can be extended to RGB, depth, and tactile observations. Thus, extending our approach real robots with high-dimensional observations would be interesting future work.

---

Acknowledgments
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grants (N0.2019.0-00075, Artificial Intelligence Graduate School Program, KAIST; No. No.2002-0-00077 AI Technology Development for Commonsense Extraction, Reasoning. and Inference from Heterogeneous Data) and National Research Foundation of Korea (NRF) grant (NRF-202IH 200
funded by the Korea government (MSIT). This work was also partly supported by the Annenberg Fellowship from USC. We would like to thank Ayush Jain and Grace Zhang for help writing, Karl Pertsch for assistance in setting up SPiRL and CALVIN, Kevin Xie for providing code of LSP, and all members of the USC CLVR lab for constructive feedback.
References
[1] Legg and M. Hutter. Universal intelligence: definition of machine intelligence. Minds and machines, 17(4):391-444 2007.
[2] M. Botvinick and A. Weinstein. Model-based hierarchical reinforcement learning and human action control. Philosophical Transactions of the Royal Society Biological Sciences, 369 (1655), 2014.
[3] D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: imagination. In International Conference on Learning Representations, 2019.
Learning behaviors by latent
[4] N. Hansen, X. Wang, and H. Su. Temporal difference learning for model predictive control. In International Conference Machine Learning. 2022.
[5] K. Lu, A. Grover, P. Abbeel, and I. Mordatch. Reset-free lifelong learning with skill-space planning. In International Conference on Learning Representations, 2021.
[6] K. Lowrey, A. Rajeswaran, S. Kakade, E. Todorov, and I. Mordatch, Plan online, learn offline: Efficient learning and exploration via model-based control. In International Conference Learning Representations, 2019. URL https: /pppnreveereeeeeeo wwt. id-Byey7n05FQ.
[7] M. Janner, J. Fu, M. Zhang, and S. Levine. When to trust your model: Model-based policy optimization. In Neural Information Processing Systems, 2019.
[8] A. Argenson and G. Dulac-Amold. Model-based offline planning. In International Con- ference on Learning Representations, 2021. URL https: //tp::///ep/eeeeee/e wwt.:neetnttiiiiee 0MNB1G5x2d4.
[9] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211 1999.
[10] Y. Lee, S.-H. Sun, S. Somasundaram, E. S. Hu, and J. J. Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2019. URL https:
[11] Y. Lee, J. Yang, and J. J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. In International Conference on Learning Representations, 2020.
[12] K. Pertsch, Y. Lee, and J. J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning. 2020.
[13] Y. Lee, J. J. Lim, A. Anandkumar, and Y. Zhu. Adversarial skill chaining for long-horizon robot manipulation via terminal state regularization. In Conference on Robot Learning, 2021.
[14] M. Dalal, D. Pathak, and Salakhutdinov. Accelerating robotic reinforcement learning via parameterized action primitives. In Neural Information Processing Systems, 2021.
[15] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning, pages 1113-1132. PMLR, 2020.
[16] K. Pertsch, Y. Lee, Y. Wu, and J. J. Lim. Demonstration-guider reinforcement learning with learned skills. In Conference on Robot Learning, 2021.

---

[17] R. Y. Rubinstein. Optimization of computer simulation models with rare events. European Journal of Operational Research, 9911)899-112, 1997.
[18] G. Williams, A. Aldrich, and E. Theodorou. Model predictive path integral control using covariance variable importance sampling. arXiv preprini arXiv:1509.01 149, 2015.
[19] D. Ha and J. Schmidhuber. World models. arXiv preprint arXiv: 1803. 10/22, 2018.
[20] R. Mendonca, O. Rybkin, K. Daniilidis, D. Hafner, and D. Pathak. Discovering and achieving goals via world models. In Neural Information Processing Systems, 2021.
[21] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 1 11((1-2):181-2 1999.
[22] P. Pastor, H. Hoffmman,, T. Asfour, and S. Schaal. Learning and generalization of motor skills by learning from demonstration. In IEEE International Conference on Robotics and Automation, pages 763-768, 2009.
[23] O. Nachum, H. Tang, X. Lu, S. Gu, H. Lee, and S. Levine. Why does hierarchy (sometimes) work so well in reinforcement learning? arXiv preprint arXiv: 1909. 10618, 2019.
[24] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3):
263-279, 2013.
[25] K. Shiarlis, M. Wulfmeeer, S. Salter, S. Whiteson, and I. Posner. Taco: Learning task decompo- sition via temporal alignment for control. In International Conference on Machine Learning. pages 4654 4663. PMLR, 2018.
[26] T. Kipf, Y. Li, H. Dai, V. Zambaldi, A. Sanchez-Gonzalez, E. Grefenstette, P. Kohli, and Battaglia. Compile: Compositional imitation learning and execution. In International Conference on Machine Learning, 2019.
[27] T. Shankar and A. Gupta. Learning robot skills with temporal variational inference. In International Conference on Machine Learning, 2020.
[28] Y. Lu, Shen, S. Zhou, A. Courville, B. Tenenbaum, and C. Gan. Learning task de- composition with ordered memory policy network. In International Conference on Learning Representations, 2021.
[29] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman. Dynamics-awarr unsupervised discovery of skills. In International Conference on Learning Representations, 2020.
[30] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
[31] O. Nachum, S. S. Gu, H. Lee, and S. Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pages 3303-3313 2018
[32] A. Gupta, V. Kumar, C. Lynch, S. Levine, and K. Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning, 2019.
[33] A. Mandlekar, D. Xu, R. Martín-Martin S. Savarese, and L. Fei-Fei. Gti: Learning generalize across long-horizon tasks from human demonstrations. In Robotics: Science and Systems, 2020.
[34] A. Mandlekar, F. Ramos, B. Boots, L. Fei-Fei, A. Garg, and D. Fox. Iris: without interaction at scale for learning control from offline robot manipulation data. In IEEE International Conference on Robotics and Automation, 2020.
Implicit reinforcement
[35] B. Wu, S. Nair, L. Fei-Fei, and C. Finn. Example-driven model-based reinforcement learning for solving long-horizen visuomotor tasks. In Conference on Robot Learning. 2021.
[36] K. Xie, H. Bharadhwaj, D. Hafner, A. Garg, and F. Shkurti. Latent skill planning for exploration and transfer. In International Conference on Learning Representations, 2020.
10

---

[37] D. Shah, A. T. Toshev, S. Levine, and brian ichter. Value function spaces: Skill-centric state ab- stractions for long-horizon reasoning. In International Conference on Learning Representations. wewtt/oorumm.ddvggsttteeae.
2022. URL
[38] R. S. Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 1984.
[39] T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference Learning Representations, 2020.
[40] D. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
[41] R. Sekar, O. Rybkin, K. Danilidiss P. Abbeel, D. Hafner, and D. Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, 2020.
[42] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerchner. VAt: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations, 2017. URL https: / //openreview net/forum'iaaSSy222uu
[43] Fu, A. Kumar, O. Nachum, G. Tucker, and Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.0722 07219, 2020.
[44] O. Mees, L. Hermann, E. Rosete-Beas, and W. Burgard. Calvin: A benchmark for language- conditioned policy learning for long-horizz robot manipulation tasks. IEEE Robotics and Automation Letters, 2022.
[45] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1856-1865, 2018.
[46] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. Viioo,, Z. Lin, A. Desmaison. L. Antiga, and A. Lerer. Automatic differentiation PyTorch. NIPS Autodiff Workshop. 2017.
[47] Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y. Li, D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. P. Lillicrap, and M. A. Riedmiller. Deepmind control suite. arXiv preprint 1801.
2018
[48] Schaul, J. Quan, I. Antonoglou, and D. Silver. Prioritized experience replay. International Conference on Learning Representations, 2016.


Reviews:
[{"review_id": "evgIzeTYOBY", "reviewer": "Reviewer_gSBE", "summary": "This paper presents Skill-based Model-based RL framework to efficiently learn downstream policies by jointly training a skill dynamics model and skill encoding networks from task-agnostic data. The skill dynamics model learns temporal consistency by predicting the final state after performing a skill (a sequence of actions) from a given state. Skills are encoded in a latent skill space, the skill dynamics model is used to plan in this latent space using cross-entropy methods and skills sampled from a task-level policy. Trajectories from the dynamics model rollouts are evaluated with learned reward and Q functions, with the best trajectories decoded into action sequences that are executed in the environment. Additional fine-tuning of the skill dynamics model is performed during downstream task learning. The framework is demonstrated in simulation with a maze task and a manipulation task, performing significantly better than separate model-based and skill-based methods.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents a combination of techniques that improve on model-based and skill-based methods. The ideas and motivations are expressed clearly and the paper is well written. The comparisons appear relevant and fair, the results show a clear benefit of the presented method. The ablation study highlighted the benefit of the skill dynamics model and of joint training. The extensive appendix was welcomed and the additional experiments that didn’t make it into the paper shone further light on the method and the improvements over existing approaches. \n\nThe primary weakness (apart from the lack of real robot experiments) was confusion through the method section, particularly Section 3.2.1 where components were introduced, but not defined until the following section (3.2.2). For example, there is no mention of “h” before equation 1. It might be better to move some definitions into 3.2.1, or re-work these two sections. Also consider re-ordering the list in Section 3.2.1 to follow the flow of learning: first train on task-agnostic data, then fine-tune / train on downstream tasks. \n\nQuestions:\n\nWas there a benefit using the state representation and observation modules? Were these modules used in the baselines? Is there utility in training these with target networks?\n\nWas the maze navigation task different in this work compared with the SPiRL paper? The performance of SPiRL seems much worse than the original paper, is this due to the original paper using more goal–reaching trajectories in their training data?\n\nWhat is the reason success on all four sequential tasks in the Kitchen environment is difficult? Do the authors have an intuition about how to more robustly solve longer task sequences?\n\nWhat is the effect of fine-tuning after training? Is this expected to be a significant component for real-world experiments? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The recommendation is to improve the clarity of the method section.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Is the negative expectation incorrect in equation 4 (negative expectation), minimising this loss will increase the KL divergence? This seems to be the opposite of equation 2. \n\nTarget networks were used through the Algorithms in the Appendix, but were not introduced in the paper. This is not really an issue if this detail was omitted for clarity, except for the superscript “-” on the model parameters for Q and E that was not explained anywhere. \n\nModules were given different names throughout, for example, O is introduced as the reconstruction module, though later referred to as the observation, and as the state decoder. Similar for E. Also, what is “sg” (equation 4 and 7)?\n\nFigure 2: “Prior” should be changed to “Skill Prior” to differentiate this from the fixed-prior used to regularise the skill encoder / decoder. q(z|s0,a) takes a vector of actions? Maybe make “a” bold to highlight this. No model actually takes “s” as input except the skill encoder, is that correct? Instead, “h” is used from the state encoder. It makes sense this is omitted from the diagram for clarity, but should probably be mentioned in the caption or where the figure is introduced in the text.\n\nEquation 7, should be pθ, not p_phi to regularise the skill prior pθ? Also, should be π_phi not πθ to regularise the policy, not the skill decoder?\n\nMinor:\nFigure 1: ”Skill dynamics model with skills” maybe just ”Skill dynamics model” is sufficient. \n\nMinor:\nThere were two mentions of furniture assembly as a complex long horizon task. I was expecting this as an experiment with improvements on the millions to billions of interactions. Perhaps include that maze navigation and kitchen tasks are also very complex long horizon tasks.\n\nSpelling / Grammar / Typos\n\n“first work that jointly learning”\n\n“for more sample efficiency policy learning”\n\n“Conventionally model-based approaches” -> missing comma, or change to “Conventional”\n\n“SkiMo is the first work that jointly learning skills” \n\n“high level policy” or “high-level policy”, both used throughout.\n\n“crucial in the real world robot learning”\n"}}, {"review_id": "dbKqMcI4C5w", "reviewer": "Reviewer_43gq", "summary": "The work proposes to combine option (skill)-based learning with model-based RL, to achieve a more sample efficient and robust (higher probability of success) approach. The proposed approach relies on demonstrations to pre-train several model components: the state representation module (E_\\psi) and the skill dynamics module (D_\\psi), as well as the state decoder (O_\\theta) in order to avoid trivial solutions. These pre-trained components are then used in the second phase of training which involves fine tuning them on a rewards-prediction/latent-state prediction, and finally an RL based objective that performs model-predictive control using a recently proposed method, TD-MPC.  Experiments on the maze and FrankaKitchen suites from D4RL demonstrates the method's superior performance. Ablation studies demonstrate the utility of jointly training the skill-representation component & skill-dynamics model. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1-The paper's language is pretty clear and it is well written \n2-The performance gains seem impressive on the tasks presented (although I am not familiar with the performance of related works)\n\nWeaknesses: \n1-The method seems *extremely* complicated and would be difficult to reproduce in my opinion. For example, the total number of terms to minimize in all objective functions is 10, and the total number of \"model components\" (i.e. networks) is 9 (!!!). \n2-The method omits the \"Androit\" and Gym suite of tasks from D4RL although they seem to be well suited (i.e. they all include demonstrations). Why not include these tasks as well? \n3-Many implementation details are missing, not least the hpram details. \n4-Despite the paper being generally well written, some technical details require major clarifications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Despite the paper presenting good performance on 2 our of the 4 D4RL suites, I recommend rejection of the paper in the current form. \nThe main reasons being: 1-the complexity of the proposed approach, and lack of technical details, makes it unreproducible in my opinion -- and therefore of limited value to the robotics/learning community. 2-The proposed model was only demonstrated on two tasks. Neither of which included real robots. This makes me question the robustness and generality of the proposed approach. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1-Do E & D share parameters? If not, then subscripting both with \\psi is very confusing.\n2-Reconstructing the observation space is known to be problematic, i.e. there are simple solutions that do not capture the details relevant to the problem. Can you give more details on this in the context of your proposed approach? \n3-In Equation 2 is z in the first term output by q_\\theta (of the second term)? \n4-Also Equation 2. Why do you apply tanh to the Gaussian samples, i.e. why is it important to clamp their values to be in [-1, 1]? \n5-Equation 3: is E_{\\psi^-} a typo (last term)? Or is it some sort of out-dated parameter vector used in the loss? \n6-How is the horizon length (skill vector temporal length) found? \n7-What is \"sg\" in Equation 4? \n8-Are there no multipliers on the various loss terms in Equation 6? i.e. are they all 1? If so, why? If not, how are the hparams tuned (in all loss terms)? "}}, {"review_id": "MvVpAyw5LW7", "reviewer": "Reviewer_ctWV", "summary": "This paper proposes an approach for hierarchical model-based reinforcement learning, where the agent first learns a repertoire of skills, and a skill dynamics model from offline dataset, and then trains a hierarchical policy conditioned on the skill space. \n\nThe key contributions of the paper are jointly learning a skill prior and a skill dynamics model from offline data, and demonstrating that the a policy conditioned on skills sampled from the skill prior can be used to guide online exploration in downstream tasks. The paper demonstrates results with the proposed algorithm on simulated robot manipulation (Frank Kitchen) and point agent navigation (maze) environments involving long-horizon tasks. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The key insight of learning re-usable skills from offline data and using the learned skill distribution to condition policies for downstream tasks is very well motivated, and relevant to a lot of robot learning settings. \n\n2. The paper is well-written. The algorithm design is well motivated, and easy to follow. The related work positions the paper well in the context of prior model based hierarchical RL works. Sufficient details are provided in the experiments to understand the setting and results well. \n\n3. The modifications of the method over prior work (e.g. Dreamer) are explained well, and logically positioned. \n\nWeaknesses:\n\n1. The paper's insights regarding learning a skill space to condition a policy in model-based RL, overlaps with several prior works as mentioned in the related works section [28, 5, 29, 30]. However, The point of distinction mentioned \"To fully unleash the potential of temporally abstracted 84 skills, we devise a skill-level dynamics model to provide accurate long-term prediction, which is\n85 essential for solving long-horizon tasks.\" is not convincing to me - I don't see any reason for the skill-space to be enable long-term prediction because although the skills are low-dimensional actions (as learned by a VAE latent embedding), there is no temporal chaining of skills - skills are sampled IID from the same Gaussian distribution at each time-step, and this is infact exactly similar to the prior works cited previously. \n\n2. All the loss terms in equation (5) are weighted equally - it is unclear why this modeling choice makes sense. Since all the terms in equation (5) are ad-hoc and do not correspond to a principled variational lower bound to the overall RL objective, I believe it is necessary to motivate their relative importance, and ablate this choice through experiments. \n\n3. It is unclear how it is ensured that the policy conditioned on the skill actually makes use of the skill and doesn't simply ignore it? There is no loss term in the objective to actually incentivize making the skills chosen predictable from the observed rollouts. In addition, this is not evaluated through the experiments. \n\n4. The experiment details and results are thorough in the settings considered, but several relevant baselines are not evaluated. The baseline Dreamer is simply a flat model-based RL approach (no hierarchy) and all the other baselines are ablations of the approach. I think it is very important to compare with several external hierarchical RL approaches proposed in the literature over the years, for example [A] HIRO and [B] DADS, the latter of which is very relevant in learning a skill-level dynamics model. Also I think [A] should be cited in the related works. \n\n[A] Nachum, Ofir, et al. \"Data-efficient hierarchical reinforcement learning.\" Advances in neural information processing systems 31 (2018).\n[B] Sharma, A., Gu, S., Levine, S., Kumar, V., & Hausman, K. (2019). Dynamics-aware unsupervised discovery of skills. arXiv preprint arXiv:1907.01657.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I believe the paper targets an important problem in robotics, that of learning re-usable skills from prior data. However, due to the limitations I pointed out in the weaknesses section, I believe the paper is not ready for publication in its current form. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Kindly explain the issues I listed (1-4) in the Weaknesses section. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "KLPtEWDtZQQ", "reviewer": "Reviewer_6Ygv", "summary": "This paper presents an algorithm for skill based model based RL, where the model is not learned as a mapping from state and action to next state but over outcomes of performing skills. The model is learned with a VEA style loss where a skill embedding is learned and a dynamics model in this embedding space is learned together with the skill from offline data. This setup is then used to learn a downstream skill for a new task using the learned dynamics model in a MBRL way. The paper presents experiments on a maze task and a franka kitchen environment where the presented method outperforms the baseline methods. The paper also presents ablation studies.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI think the paper is well written and presents clearly the problem setting, the algorithm and the contribution. I also think the authors are making clear how their approach differs from the related work. The topic and the direction of the paper is very relevant of learning models over higher abstractions instead of low-level sensor measurements and the paper presents convincing results that this might be a good avenue to investigate further. I think this paper would make a clear contribution to the field.\n\nWeaknesses:\n- it would be very good and important to know more about the offline data that was used, how close is it to the downstream tasks? How much data was used? I think these details are important to understand the generality of this approach\n\n- robot experiments are missing, how would the transfer look like? still your method requires at least 250k samples, even after all of the pre-training that happened - could you maybe elaborate on this and comment on the real world applicability of the approach?\nminor:\n- an overview image of the algorithm would be very helpful to understand quicker how the learning setup looks like.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am willing to adjust my assessment during the rebuttal after reading the authors response on the above mentioned points.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "see above under weaknesses. ", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)