Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics
Authors: Krishan Rana
Ming Xu
Brendan Tidd
Michael Milford
Niko Suenderhauf
Venue: CoRL 2022 Poster (2022)
Keywords: Reinforcement Learning
Skill Learning
Transfer Learning
Residual Reinforcement Learning

Abstract: Skill-based reinforcement learning (RL) has emerged as a promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically extracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. However, this \textit{skill space} is expansive, and not all skills are relevant for a given robot state, making exploration difficult. Furthermore, the downstream RL agent is limited to learning structurally similar tasks to those used to construct the skill space. We firstly propose accelerating exploration in the skill space using state-conditioned generative models to directly bias the high-level agent towards only \textit{sampling} skills relevant to a given state based on prior experience. Next, we propose a low-level residual policy for fine-grained \textit{skill adaptation} enabling downstream RL agents to adapt to unseen task variations. Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while significantly accelerating exploration, outperforming prior works.

Residual Skill Policies: Learning an Adaptable Skill-based Action Space for Reinforcement Learning for Robotics
Krishan Rana', Ming Xu', Brendan Tidd', Michael Milford', Niko Sünderhauf
QUT Centre for Robotics, Queensland University of Technology Dataó1 Robotics and Autonomous Systems Group, CSIRO ranakōqut edu.au
Abstract:
Skill-based reinforcement learning (RL) has emerged as promising strategy to leverage prior knowledge for accelerated robot learning. Skills are typically ex- tracted from expert demonstrations and are embedded into a latent space from which they can be sampled as actions by a high-level RL agent. However, this skill space is expansive, and not all skills are relevant for a given robot state, making exploration difficult. Furthermore, the downstream RI. agent limited to learn- ing structurally similar tasks to those used to construct the skill space. We firstly propose accelerating exploration in the skill space using state-conditioned gen- erative models to directly bias the high-level agent towards only sampling skills relevant to a given state based on prior experience. Next, we propose a low-level residual policy for fine-grained skill adaptation enabling downstream RL agents to adapt to unseen task variations. Finally, we validate our approach across four challenging manipulation tasks that differ from those used to build the skill space, demonstrating our ability to learn across task variations while significantly accel- erating exploration, outperforming prior works. Code and videos available our project website: https:/ //s::///rrr / shanrana g thub. i o reskil i1..
Keywords: Reinforcement Learning, Skill Learning, Transfer Learning
Introduction
Humans are remarkably efficient at learning new behaviours. We can attribute this to their ability to constantly draw relevant prior experiences and adapt this knowledge to facilitate learning. Building on this observation, recent works have proposed various methods to incorporate the use of prior experience in deep reinforcement learning (RL) to accelerate robot learning.
One approach that has gained attention recently is the use of skills which are short sequences of single-step actions representing useful and tank-igi behaviours extracted from datasets of ex- pert demonstrations [1, 2, 3). the manipulation domain, for example, such skills could include move-left, grasp-object and lift-object. These skills are typically embedded into a latent space, forming the action space for a high-level RL policy. While yielding significant improvements over learning from scratch, there are still outstanding challenges for training RL. agents using this latent skill space.
Firstly, naive exploration by sampling randomly over all skills can be extremely inefficient [2, 3, 4]. Typical skill spaces encode an expansive set of generic skills, with only a small fraction of skills relevant for execution in given robot state. Furthermore, the set of skills relevant to a given state do not typically cluster the same neighbourhood in the skill space, i.e. the conditional density is multi-modal, making targeted sampling challenging. We address this issue of inefficient exploration by learning a state-conditioned skill prior over the skill space, which directly biases the agent's exploration towards sampling only those skills that are deemed relevant in its current state. Unlike previous works [3], our skill-prior captures the complex multi-modality of the skill space allowing us to sample directly from this density during exploration.
6th Conference on Robot I Learning (CoRI. 2022), Auckland, New Zealand.

---

Skill f (g.
Policy TEELL (g)
1 Hz
Rosidual To(da|s,
Skil Decoder p(al1z,,)
10 Hz
öx
Figure I: Residual Skill Policies (ReSkill). skill-based learning framework for roboties. The skill prior transforms the agent's ' action space to cutte-conditooeed skill space using normalising flows, where only the relevant skills for the current state are explored. The residual policy allows for fine- grained adaptation of the skills to environment variations and unseen tasks. The symbol signifies that the pre-trained weights for these skill modules are frozen during downstream task learning.
Secondly, current approaches to skill-based RL assume that the skills are optimal and that down- stream tasks are drawn from the same distribution used to create the skill space. Consequently, skill-based learning methods have limited generality and adaptability to task variations. For exam- ple, if the available skills were extracted solely from demonstrations of block manipulation empty table, learning a downstream task with obstacles, object variations, or different friction coef- ficients may not be possible. In this case, using non-exhaustive skill space cripple an RL agent's 's learning ability. We address this by introducing a low-level residual policy, which enables fine-graine skill adaptation of available skills variations in the task not solvable by simply composing the available skills. This relaxes the need for exhaustive and expert demon- stration datasets one that can be extracted from existing classical controllers, which are cheap and readily available source of demonstration data for robotics.
action space
We summarise the main contributions of our approach denoted as Residual Skill Policies (ReSkill) as follows: (1) we propose a novel stte-conditioned skill prior that enables direct sampling of relevant skills for guided exploration, (2) introduce a low-level residual policy that adapt the skills to variations in the downstream task, and (3) we demonstrate how our two contributions both accelerate skill-based learning and enable the agent to attain higher final performance in novel environments across four manipulation tasks, outperforming all baseline methods.
Related Work
Skill-Based RL This has emerged promising approach to effectively leverage prior knowledge within the RL framework by embedding reusable skills into a continuous skill space via stochastic latent variable models 2. 6]. Skill-based learning provides temporal abstraction that allows for effective exploration of the search space, with the ability to draw on previously used behaviours accelerate learning. However as described previously, the skill space can be difficult to explore. Pertsch et al. [3] proposed the use of prior over skills to help better guide exploration. They learn a unimodal Gaussian prior that approximates the density of relevant skills for given state and regularise the downstream RL agent towards the prior during training to indirectly bias exploration. Singh et al. [4] proposed a more expressive strategy for learning these priors conditioned generative models. While this was conducted at the single-step action level, they show that they can directly bias exploration by sampling relevant actions from this density. In this work, we extend this idea to based action spaces.
use of state-

---

Combining RL with Classical Control Our work is closely related to methods that seek to com- bine the strengths of classical control and RL for accelerated learning. In contrast to expert datasets or human demonstrators, classical control systems are cheap and readily available for most robotics tasks. Johannink et al. [7] proposed the residual reinforcement learning framework which an RI. agent learns an additive residual policy that modifies the behaviours of predefined handcrafted controller to simplify exploration. Other works in this area have shown that combining the two modes of control allows for sample efficient initialisation [8. 9, 10]. temporal abstraction for ac- celerated learning [11], and safe exploration [12]. While strongly motivating the re-use of existing handcrafted controllers, these methods are restricted to solving only one particular task for which the controller was designed, requiring a new controller for each new task. This limits the generality of these approaches to novel tasks. In this work, decompose these controllers into task-agnostic skills and explore how they can be repurposed for learning a wide range of tasks.
Hierarchical RL (HRL) These methods enable autonomous decomposition of challenging long- horizon decision-making tasks for efficient learning. Typical HRI. approaches utilise a high-level policy that selects from set of low-level temporally extended actions or options [13] that provide broader coverage of the exploration space when executed [14]. Prior work in hierarchical rein- forcement learning can be divided into two main categories: learning low-level and high-level poli- cies through active interaction with an environment [15, 16, 17, 18, 19], and learning options from demonstrations before re-composing them to perform long-horizon tasks through RL [20, 21, 22]. Our work shares similarities with both strategies. We follow the latter approach by pre- training a low-level skill module from demonstration data on which high-level policy operates, while addi- tionally learning a low-level residual policy online for fine-graing graiirr skill adaptation.
3 Problem Formulation
We focus on skill-based RL which leverages skills as the actions for a high-level policy MHL, where a skill is defined sequence of atomic actions {a,,..., H++H-1) over fixed horizon H. Skills are typically embedded into a continuous latent space Z from which THE can sample during exploration [1, 2, 3]. Latent skill spaces expressively capture vast amounts of task-agnostic prior experience for re-use in RL. However, they pose several challenges that can negatively impact downstream learning. Firstly, current methods rely on expert and exhaustive demonstration datasets that capture skills for a wide range of potential task variations. Such datasets can be arduous and expensive to obtain. Secondly, as the skill space abstracts away the RL agent's atomic action space, the generality of skill-based RL is greatly limited to tasks that closely match the distribution of those captured by the dataset. This poses a trade-off between generality and sample efficiency in downstream learning [23). Furthermore, latent skill spaces are expansive and can be arbitrarily structured, making random exploration difficult. Learning structure within this space can better inform exploration to accelerate downstream learning.
The downstream task is formulated as Markov Decision Process (MDP) defined by tuple (S, A. T. R, consisting of states, actions, transition probabilities, rewards and a discount fac- tor. We learn the high-level policy (z|s) to select the appropriate latent skills 2 in a given state s to maximise the discounted sum of H H-step rewards
E -1 7'r), where T is the episode horizon, f is the H-step reward defined as f = Em ir and is the single-step reward.
4 Approach
We aim to re-purpose existing classical controllers as skills to facilitate RL in solving new tasks. We decompose demonstration trajectories produced by these controllers into task-agnostic skills and embed them into a continuous space 2 using stochastic latent variable models. To facilitate explo- ration for downstream RL in this skill space, we propose skills that directly biases the agent towards only sampling skills relevant to given state based on prior experience. Finally, to enable truly general-purpose learning with the skill space, we propose a low-level residual policy T (x(oal.) -) that can adapt the embedded skills to task variations not captured by the demonstration data. This gives the RL. agent access to its single-step action space and relaxes the need for both expert and exhaustive datasets.
conditiood skill prior p(2]sp) over
3

---

-== (g. p(=
Skill Prior p(a 8n)
p(z)
Z
Lembed
Skill 0(=|a, a)
Skill
1 Ho)"
Sprion
Decoder
plalle, sc)
foles
fi(ef,oo)
fale" (0))
%
Figure 2: Schematic for learning the state-conditioned skill space. We train a VAE embedding module which encodes skills into latent embedding space 2. This module is comprised of an encoder and a closed-loop skills decoder, where the decoder recovers atomic actions from latent skill embedding z. The skill prior module learns the state-conditioned density of useful skills based on the dataset. This conditional density is multi-modal in the skill space and we estimate it using normalising flows. Both modules are trained jointly, with coloured arrows illustrating the gradient flow between them.
Our approach can be decomposed into three sub-problems: (1) skill extraction from existing con- trollers, (2) learning a skill embedding and skill prior, and (3) training a hierarchical RL policy within this skill space with low-level, residual skill adaptation policy. We discuss each of these steps in more detail below.
4.1 Data Collection
We re-purpose existing handcrafted controllers as sources of demonstration data for basic manipu- lation tasks involving pushing and grasping object are relatively simple, the trajectories produced possess a vast range skills that can recomposed to solve more complex tasks. We note here that these trajectories do not need to be optimal given our adaptable downstream RL formulation which we will describe more detail in the following sections. The trajectories consist of state-action pairs and perform unsupervised skill extraction by randomly slicing a H length segment from them. We utilise both the extracted sequence of ac- tions as well the corresponding states learn the state-conditioned skill space described in the next section. The state vector comprises of joint angles, joint velocities, gripper position and object positions in the scene, while the actions gripper velocities in Cartesian space. More details about data collection and the specific handcrafted controllers used can found in Appendix ??
empty table. While these controllers
continuous 4D vectors representing end-effector and
4.2 Learning a State-Conditioned Skill Space for RL
Learning state-conditioned skill space consists of key steps: (1) embedding the extracted skills into a latent space: (2) learning state-conditioned prior over the skills which we can sample from during exploration. Figure 2 provides summary of our approach and describe each component in detail below.
4.2.1 Embedding the skills
We follow the approach proposed in prior works [1, 24], where skills are embedded into latent space using variational autoencoder (VAE) [25]. The VAE comprises a probabilistic encoder de and decoder with network weights and 0 respectively, which maps a skill action sequence to

---

a latent embedding space and vice versa. Our encoder g0(2 s.a) jointly processes the full state- action sequence, while our decoder Pn (a, 2, s,) reconstructs individual atomic actions conditioned on current state and skill embedding z. The loss function for our VAE for single training example is given by
H Σ
Lembed = E (xlra))
log po (as 2, st) (2x (s,, || p(z)),
(1)
where p(z) ~ (O 1) and 8 E [0, 1] is the weighting for the regularisation term [25]. The closed- loop nature of the decoder considers the current robot state 8, as well as sampled skill z when recovering actions, which was found to significantly improve performance in downstream learning [26), particularly in dynamic environments.
4.2.2 Learning the state-conditioned skill prior
Skil Space
The nature of the VAE reconstruction loss described in Equation (I) allows us to sample skills from the latent space by using the prior p(z) ~ N(0, I). However, sampling directly from the prior can be in- efficient, since the samples will range across the full, expansive set of available skills as well regions that may not constitute any embeddings as shown in Figure 3. In reality, for particular robot state, only a small subset of these skills are relevant for ex- propose
Embodded
Sampled Skills using
ploration. To address this inefficiency, to learn a state-conditioned skill prior over the la- tent embedding space, which allows us to more fre- quently sample the relevant skills likely to be useful in given state, thus accelerating exploration.
Figure 3: t-SNE of the skill embedding space. Blue denotes samples drawn from the VAE prior p(=). Yellow denotes embedded skills drawn from the dataset. Green denotes samples using our trained skill prior. Note the multi-modality across the latent space. The skill prior samples from regions contain- ing embedded and relevant skills, reducing the chances of sampling irrelevant regions (blue) not containing any meaningful skills.
Mathematically, we are looking to learn a condi- tional probability density over the latent skill space p(2(s)) that we can draw samples from when train- ing THL. This conditional density is typically highly multi-modal, i.e. for a given robot state, many skills can be relevant that may be far apart in the topology
of the latent skill space. Prior works [3] propose a simple unimodal density (e.g. Gaussian) which does not sufficiently capture this multi-modality and hence is unsuitable to directly sample skills from.
One of our contributions is around gracefully handling this multi-modalin modality by using the real-valued non-volume preserving transformations (real NVP) method by Dinh et al. [27] based on normalising flows [28]. Conditional real NVP [29] learns mapping f Z - G which we use to generate samples from p(2[s). Spaces 2 and 9 are identical, i.e. 2 = g = IRd and furthermore f is bijective between 2 and G for fixed state s. Sampling is achieved as follows: first sample g po(g) from a simple distribution (e.g. (g) ~ AO, I)) and then transform into skill space Z using f-1, i.c. (g. s) ~ p(z[s). Since f fully detines p(z]s), we henceforth refer
the skill prior.
We refer the reader to Dinh et al. [27] for more information about real NVP. We train our state- conditioned skill prior jointly with our VAE embedding module. Figure 3 illustrates the multi- modality present in the latent skill space and how our skill prior captures this. Implementation details are provided in Appendix ??.
4.3 Reinforcement Learning in State-Conditioned Skill Space
Once trained, the decoder and skill prior weights are frozen and incorporated within the RL frame- work. Our high-level RL policy denoted गH. TH. (g|s) ) a neural network that maps a state to a vector g E G in the domain of the skill prior transformation f-1 defined in Section 4.2.2. The skill prior f-1 is then used to transform g to latent skill embedding z. Our use of normalising flows for our skills prior is motivated by [4], who hypothesised that the bijective nature of f allows the RL agent
5

---

Algorithm Action Selection
Given: high-level policy THL, residual policy na. skill decoder p. skill prior
Inputs: state 5t
1: g ~ THI (g(ss)
sample a vector 9 from the high-level policy map y to state-relevant skill using the skill prior
2: 2 <- (-) 8.)
3: for skill horizon H do
4: a' + p(a'aat, 2)
D decode the skill to a single-step action using the skill decoder
5: da ~ (6a St, 2,
D sample corrective action from the residual policy sum the decoded and residual action execute the combined action in the environment and continue
6: as a 60
7:
return action di
to retain full control over the broader skill space 2: for any given z, there exists that generates (g) in the original MDP. This means that all skills from can be sampled by the skill prior with non-zero probability.
Once latent skill is attained by the high-level agent, the closed-loop decoder reconstruct each action a' sequentially conditioned on the current state for the skill horizon H. To increase the diversity of downstream tasks that be learned using the fixed skill space, we introduce low- level residual policy wy that can adapt the decoded skills at the executable action level, providing the hierarchical agent with full control over the MDP. We condition this residual policy the current state 8, selected skill 2, and the action proposal generated by the skill decoder a'. The produced action da ~ Ta (ōa|s, z, a') is added
the decoded action a' before being executed on the robot. This low-level system is operated at a frequency of H Hz based on the skill horizon, while the outer Hz. Figure/ 1 summarises our Residual Skill Policy (ReSkill) architecture and pseudo-code for action selection is provided in Algorithm 1. Given the non-stationarity imposed by the low-level residual policy being trained jointly with the high-level policy, we utilise an on on-policy RL algorithm for training the hierarchical agent. Implementation details are provided in Appendix ??.
loop operates
Experiments
We evaluate our approach in robotic manipulation domain simulated in MuJoCO [30], involving 7-DoF Fetch robotic arm interacting with objects placed in front of it as shown in Figure 4. Evalua- tion is performed over four downstream tasks, each one not solvable purely with the prior controllers used for data collection as described in Appendix ??. Each task is adapted from the residual rein- forcement learning set environments proposed by Silver et al. [31]. We describe each of the downstream tasks in detail in Appendix ??.
We compare our approach against several state-of-the-aar RL methods; these methods either operate directly on the single-step action space or train agents using skills-based approach. The specific methods we compared against are as follows:
1. Scripted Controller:
The average performance of the handcrafted controllers used for data collection across the 4 environments.
2. Behavioural Cloning (BC) + Fine-Tuning: Trains an agent by first using supervised learn- ing on state-action pairs from the training set, followed by RL fine-tuning.
3. SAC and PPO: Trains an agent using Soft Actor Critic [32] and Proximal Policy Optimi- sation [33] respectively with random weight initialisation.
4. HAC:
Trains an agent using Hierarchical Actor-Critic [19] using a hierarchy of 2 for all
tasks.
5. PaRRot: Leverages flow-based density estimation to map the agent's action space to single-step actions that are likely to be useful in a given state [4].
6. SPiRL: A hierarchical skill-based approach utilising KL regularisation of the RL policy towards Gaussian skill prior [3]
7. ReSkill (No Skill Prior): Ablation study where the high-level policy samples skills directly
using (2]() with no skill prior guidance.

---

Slippery Push
Table Cleanup
(c) Pyramid
Complex Hook
Figure 4: Downstream Tasks. Robotic manipulation tasks involving a 7 DoF manipulator arm from Fetch Robotics. Each task exhibits a physical or dynamical variation to the environment that was not present the skill extraction environment.
43
Fesh Tablo Charup
Fetth ppery Push
Fetth Palaridd Block
Fetth Compiies Hook
22
3.0
0.2
0.6
0.5
0.3
1.0
0.03
0.25
0.50
0.75
1.00
1.25
1.50
0.00
0.25
0.50
0.75
1.00
1.25
1.50
3.0
02
0.4
05
0.8
1.
Skeps
Steps
Steps
Figure 5: Training Performance. Average training performance across random seeds for the different tasks. ReSkill outperforms all the baselines in both sample efficiency and convergence to the highest-performet final policy.
8. ReSkill (No Residual): Ablation study where we remove the low-level residual policy for adapting skills.
6 Results
The results across each task are summarised in Figure 5. Our approach, ReSkill, outperforms all the baselines, attaining the highest performing policy given its ability to adapt task variations while additionally converging with superior sample efficiency given the guidance of the state-conditioned skill prior. Note, the baseline scripted controllers are sub-optimal across all four tasks as indicated by the dashed black line. SAC, PPO and HAC fail to learn altogether, given the inability of ran- dom exploration alone yield any meaningful behaviours in sparse reward settings. Results incrementally better using BC initialisation and fine-tuning; however, the agent still attains sub- optimal performance with high variance across the trials. A critical difference between PaRRot and ReSkill is the use of skills in ReSkill instead of single-step actions. The temporal abstraction pro- vided by skills allows ReSkill significantly outperform PaRRot in sample efficiency across all tasks. Further, note that PaRRot struggles to learn a high-performing policy in the Slippery-Pushh and Complex-Hook domain which we could attribute to its inability to deal with the variations in the downstream task given the constrained action space. SPiRL accelerates learning, yielding bet- ter sample efficiency than all atomic-action-boe based methods on Table Sible-Cleanup and Complex- Hook. However, it still fails attain high final policy performance as it only composes high-level skills without the ability to adapt them task variations, as shown by ReSkill.
We additionally provide an ablation study in Figure 6 to better understand the impact of the skill prior and residual agent on the overall based learning architecture. ReSkill (No Skill Prior) demon- strates the importance of our sampling- sampling-based based prior in accelerating learning and enabling our agent to learn altogether the more complex stacking task. The ReSkill (No Residual) plots demonstrate importance of the residual in enabling the agent to attain high final performance by re-introducing access to the single-step action space for fine- fine-grained control. This allows the agent to appropri- ately balance the trade-off between generality and sample efficiency when leveraging skills in RL.
7

---

Re51i
SFIRL
ReStil Residal
ReSkil INo PrRY
Farch Takle Chevnss
Ferch Sltpery Puch
23
Fetta Pyrarid Stack
Farch Comples Hock
64
15
40
10
24
20
0.5
22
644
0.5
0.5
100
0.02
0.23
0:00
2.13
1.00
1.25
1.50
5.00
022
0.50
0.75
1.00
125
1.50
0.0
0.2
0.4
16
68
14
Steps
Stepe
Sleps
Steps
Figure 6: Ablation Study. Analysing the impact of the residual and skill prior on the average training performance across the tasks. The skill prior plays an important role in accelerating learning with the more difficult tasks, while the residual is important for attaining higher rewards by adapting the skills to the variations in the training environment.
Furthermore, comparing ReSkill (No Residual) SPiRL, we that our approach attains faster convergence to the final policy. We can attribute this to the nature of our skill prior which enables the agent to directly sample and execute state-relevant skills from the early stages of training. In contrast, SPiRL relies on an indirect bias to the agent's exploratory actions via KL-regularisation towards unimodal skill prior. This requires an initial "burn-in" phase before the policy is appro- priately regularised towards the skill prior in order for the agent to start sampling relevant skills. We provide a more detailed ablation study of the impact of our proposed skill-prior Appendix ??
6.1 Limitations
While we demonstrate significantly faster learning and better asymptotic performance than all the baseline methods, ReSkill still requires thousands of environment samples before converging high success rate. This limited our ability to demonstrate our method on a physical robot. An inter- esting future direction explore would be integrate our approach with off-policy RL algorithm such SAC in order to make better use of exploratory experience for improved sample efficiency. This, however, would require careful consideration of the non staaiooatii imposed by the low-level residual component. Secondly, while we note that the skill prior used in this work is bijective and should theoretically allow the agent to access the entire range of skills in the larger skill space, the likelihood of selecting useful skills not seen in the training dataset becomes very low. This would particularly be an issue when there downstream task. We note here that the residual policy could play major in addressing this, how- ever, we leave a thorough evaluation of this to future work. Finally, learning the state-conditioned skill space requires two modules, specifically the VAE embedding module and flows-based skill prior module. Future work will explore using a single generative model such as a conditional VAE [34] to jointly handle the tasks of both modules.
significant mismatch between the training dataset and the
7 Conclusion
This work proposes Residual Skill Policies (ReSkill), a general-purpose skill-based RL framework that enables effective skill re-use for adaptable and efficient learning. We learn a high-level skill agent within a state-conditioned skill space for accelerated skill composition, as well as low- level residual agent that allows for fine mme-grained skill adaptation variations in the task and training environment. Additionally, we show that our framework allows us to re- re-purpose a handful of scripted controllers to solve a wide range unseen tasks. We evaluate ReSkill across a range manipulation tasks and demonstrate its ability to learn faster and achieve substantially higher final returns across all task variations than the baseline methods. We see this as a promising step towards the effective re- use of prior knowledge for efficient learning without limiting the generality of the agent in learning a wide range of tasks.
8

---

Acknowledgments
K.R., M.X., M.M. and N.S. acknowledge the continued support from the Queensland University of Technology (QUT) through the Centre for Robotics. This research was additionally supported by the QUT Postgraduate Research Award. The authors would like to thank Robert Lee, Jad Abou-Chakra, Fangyi Zhang, Vibhavari Dasagi, Tom Coppin and Jordan Erskine for their valuable discussions towards this contribution. We would also like to thank the reviewers for their insightful and con- structive comments towards the significant improvement of this paper.
References
[1] Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, Wayne, Y. W. Teh, and N. M. O. Heess. Neural probabilistic motor primitives for humanoid control. International Conference on Learning Representations (ICI.)), 2019.
[2] C. Lynch, M. Khansari. T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet. Learning latent plans from play. In Conference on Robot Learning (CoRL), 2019.
[3] K. Pertsch, Y. Lee, and J. J. Lim, Accelerating reinforcement learning with learned skill priors. In Conference on Robot Learning (CoRL), 2020.
[4] A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2021.
[5] K. Hausman, J. T. Springenberg, Z. Wang, N. Heess, and M. Riedmiller. Learning embed- ding space for transfcrable robot skills. In International Conference on Learning Representa- tions (ICLR), 2018.
[6] Merel, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Brez, G. Wayne, and N. Heess. Catch and carry: Reusable neural controllers for vision-guided whole-body tasks. ACM Trans. Graph., 39(4), jul 2020. ISSN 0730-0301.
[7] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine. Residual reinforcement learning for robot control. International Conference on Robotics and Automation (ICRA), 2018
[8] K. Rana, Dasagi, B. Talbot, M. Milford, and N. Sünderhauf. Multiplicative controller fu- sion: Leveraging algorithmic priors for sample-efficient reinforcement learning and safe sim- to-real transfer. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6069 60769 2020.
[9] I. Uchendu, T. Xiao, Y. Lu, B. Zhu, M. Yan, J. Simon, M. Bennice, C. Fu, C. Ma, Jiao, al. Jump-start reinforcement learning. arXiv preprint arXiv:2204.02377. 2022.
[10] L. Xie, S. Wang, S. Rosa, A. Markham, and N. Trigoni. Learning with training wheels: speed- ing up training with a simple controller for deep reinforcement learning. In 2018 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 6276 6283. IEEE, 2018.
[11] F. Xia, C. Li, R. Martin-Martin, O. Litany, A. Toshev, and S. Savarese. Relmogen: Integrating motion generation in reinforcement learning for mobile manipulation. 202/ IEEE Interna- tional Conference on Robotics and Automation (ICRA), pages 4583-4590, 2021.
[12] K. Rana, V. Dasagi, J. Haviland, B. Talbot, M. Milford, and N. Sünderhauf. Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics. arXiv preprint a7Xiv:2107.09822, 2021.
[13] R. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for tem- poral abstraction in reinforcement learning. Artificial intelligence, (1((1-2):111-111 1999.
[14] I.. Steccanella, S. Totaro, D. Allonsius, and A. Jonsson. Hierarchical reinforcement learning for efficent exploration and transfer. In 4th Lifelong Machine Learning Workshop at ICMI. 2020, 2020.

---

[15] P.-L. Bacon, J. Harb. and D. Precup. The option-critic architecture. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), Feb. 2017.
[16] A. G. Kupesik, M. P. Deisenroth, J. Peters, and G. Neumann. Data-efficient generalization of robot skills with contextual policy search. In AAAI, 2013.
[17] N. M. O. Heess, G. Wayne, Y. Tassa, T. P. Lillicrap, M. A. Riedmiller, and D. Silver. Learning and transfer of modulated locomotor controllers. ArXiv, abs/1610.05182, 2016.
[18] T. Haarnoja, K. Hartikainen, P. Abbeel, and S. Levine. Latent space policies for hierarchical reinforcement learning. In International Conference on Machine Learning (ICML), 2018.
[19] A. Levy, G. D. Konidaris, R. P. Jr., and K. Saenko. Learning multi-level hierarchies with hindsight. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
[20] T. Shankar, S. Tulsiami, L. Pinto. and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations (ICLR), 2019.
[21] S. Krishnan, R. Fox, Stoica, and K. Goldberg. Ddco: Discovery of deep continuous options for robot learning from demonstrations. In Conference on Robot Learning (CoRL., 2017.
[22] M. Sharma, A. Sharma, N. Rhinehart, and M. Kitani. Directed-info GAIL: Learning hierar- chical policies from unsegmented demonstrations using directed information. In International Conference on Learning Representations, 2019.
[23] J. Gehring, G. Synnaeve, A. Krause, and N. Usunier. Hierarchical skills for efficient explo- ration. Advances in Neural Information Processing Systems, 34:1 1553-11564, 2021.
[24] T. Pearce, M. Zaki, A. Brintrup, and A. Neel. Uncertainty in neural networks: Approximately bayesian ensembling. International Conference on Artificial Intelligence and Statisties (AIS- TATS), 2020.
[25] D. Kingma and M. Welling. Auto-Encoding Variational Bayes. In 2nd International Con- ference on Learning Reprexentations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Con- ference Track Proceedings, 2014.
[26] K. Pertsch, Y. Lee, Y. Wu, and J. J. Lim. Demonstration-guinn reinforcement learning with learned skills. 5th Conference on Robot Learning. 2021.
[27] L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real nvp. International Conference on Learning Representations (ICLR), 2017.
[28) E. G. Tabak and C. V. Turner. A family of nonparametric density estimation algorithms. Com- munications on Pure and Applied Mathematics, 66(2):145-164, 2013.
[29] L. Ardizzone, J. Kruse, S. Wirkert, D. Rahner, E. W. Pellegrini, R. S. Klessen, L. Maier- Hein, C. Rother, and U. Köthe. Analyzing inverse problems with invertible neural networks. International Conference on Learning Representations (ICLR), 2019.
[30] E. Todorov, T. Erez, and Tassa. Mujoco: physies engine for model-based control. In 20/2 JEFF/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
[31] T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling, Residual policy learning. arXiv preprint 1ew:1882.0629. 2018
[32] T. Haarnoja A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning (ICMI), 2018.
[33] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint urXivv1707.006 1707.06377 2017.
[34] K. Sohn, H. Lee, and X. Yan. Learning structured output representation using deep conditional generative models. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
10


Reviews:
[{"review_id": "nAQKT55aAH5", "reviewer": "Reviewer_fBFD", "summary": "This paper extends skill-prior RL (SPiRL) by (1) replacing a uni-modal gaussian skill prior with a multi-modal skill prior and (2) adapting low-level skills using a residual skill policy. The proposed multi-modal skill prior not only handles multi-modal skill distributions but also fully constrains exploration within the skill prior distribution. In addition, this paper addresses the problem of the imperfect skill repertoire by allowing low-level skill adaptation. The experiments demonstrate its faster convergence and higher performance over state-of-the-art RL and skill-based RL approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strenghts\n\n* This paper well combines the existing techniques of multi-modal prior and residual policy learning to address the challenges in skill-based RL.\n\n* The paper is clearly written.\n\n### Weaknesses\n\n* The tasks used for evaluation are too simple compared to the ones used in prior works. Testing the proposed method with more diverse skill sets would be necessary to prove the strength of the proposed skill prior.\n\n* By utilizing normalizing flows to implement the multi-modal skill prior, the action space of the high-level policy becomes strictly limited to its skill prior distribution, which may not align well with the downstream task and is not flexible to acquire a new skill distribution. This method alleviates this issue by fine-tuning the low-level controller with the residual skill policy; however, this results in non-stationary training of the high-level policy, which is one of the major challenges in HRL.\n\n* The limitation of the proposed method in the use of on-policy RL algorithms may hurt the sample efficiency of skill-based RL.\n\n* The experiments are conducted in newly designed environments and tasks, which are tailored to the proposed method. Moreover, the tasks are rather simple and short-horizon compared to tasks where skill-based RL is good at. To demonstrate the broad applicability of the proposed method and better comparisons to prior works, it is recommended to perform experiments on commonly used benchmarks for skill-based RL, e.g., FrankaKitchen and PointMaze.\n\n* Many implementation details, such as skill length H, task episode length, and task reward, are missing, which makes it hard to judge the correctness of the results.\n\n### Questions and suggestions\n\n* In Section 4.2.1, the proposed way of embedding skills to the latent skill space has been already proposed in Pertsch et al. [1] and demonstrated its superior performance.\n\n[1] Pertsch et al., Demonstration-Guided Reinforcement Learning with Learned Skills, CoRL 2021\n\n* In Figures 5 and 6, using more intuitive measures, such as success rate, would help understand the results. Or, at least, it should contain an explanation about the returns (rewards) for each task.\n\n* In the references, many conference proceedings appear as arXiv preprints. Please use the proper citation.\n\n* The appendix should be submitted separately. This paper violates the official guideline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an intuitive and meaningful extension of SPiRL. However, the experiments are done in a new set of tasks, and many details are missing, which makes the results less convincing. Therefore, I stand for weak rejection.\n\n\n**After rebuttal**\n\nI appreciate detailed responses and additional experiments in the revised paper. The responses and additional experiments make this paper more solid, so I would raise my score to \"Weak accept\". But, I think this paper needs significant improvement to make it at the level that I can confidently accept this paper. \n\nHere are additional comments for the revised paper:\n* In Figure 8, ReSkill and ReSkill (No residual) should be reversed.\n* L526-527, I do not agree with the claim that \"substantially outperform SPiRL in both sample efficiency and convergence\n527 to a higher final policy performance\". Although ReSkill learns faster at the beginning, it could be due to better pre-training of skills and skill priors, and ReSkill and SPiRL seem to converge at not significantly different final performances.\n* In addition, the last subtask is the only task that requires adaptation as the first three subtasks are present in the dataset. However, the proposed method seems to fail at adapting to this new subtask, which weakens the major claim of the paper.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the weaknesses listed above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "mUSj5PjuVjU", "reviewer": "Reviewer_HjV5", "summary": "The paper addresses the problem of learning to execute and adapt existing skills to more complex tasks that are different from the tasks used to build the atomic skills. In particular,  (i) the paper collects state-action pairs of handcrafted controllers, (ii) learns a skill embedding and a state-conditioned skill prior based on this data and (iii) learns an RL policy that acts within this skill space and adapts each skill to task variations. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The paper is clearly and well written \n- The idea to condition the skill prior on the current state seems clever, since it rules out state spaces for which the skill is unlikely to be helpful while the skill prior captures the fact that the same state might be part of multiple skills by the multimodal distribution. \n\n# Weaknesses\n- Evaluations only in simulation: data generated in reality might suffer from distributional shift and other problematic influences. It is still to be shown if the techniques presented in this paper can handle such difficulties. Can you discuss, for instance, how the skill embedding space in Figure 3 would change? \n- It appears that the authors composed a new method out of several existing approaches. For instance, the autoencoder used for the skill embedding comes from [1,21]. Please make it very explicit what you're adding. Is it the multimodal structure of the skill prior? A contribution could also be the clever composition of these existing methods to reach higher empirical performance.\n- It would be interesting to also compare to hierarchical RL approaches such as HAC [1]. It seems obvious that neither PPO nor SAC perform well in such tasks. \n- Only three random seeds have been used to test the data. Since the progress in RL is generally very noisy – especially in more complex tasks – it is important to evaluate more thoroughly.\n\n[1] Andrew Levy, George Dimitri Konidaris, Robert Platt Jr., and Kate Saenko. 2019. Learning multi-level hierarchies with hindsight. In Proceedings of the 7th International Conference on Learning Representations", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper presents an interesting approach and addresses an imoortant research question, it also has multiple weaknesses. I encourage the authors to clarify and address these weaknesses. I would be happy to increase the score then.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please address the weaknesses in the \"Strengths And Weaknesses\" section.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "eSMjGZUO6s", "reviewer": "Reviewer_eiyG", "summary": "This work provides two main improvements to robot skill learning methods: First is a latent skill space, which can be used as a prior for accelerating the learning on both the high-level policy $\\pi_{HL}$ and the skill decoder. The second is a residual skill mechanism, which can further adjust the action generated by the skill decoder on different task settings.\n\nThe comparison of simulated environments shows the effectiveness of the proposed method and illustrated the advantage of learning with skill abstractions and pre-captured data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThis work introduces a latent skill prior, which can be used for accelerating both high-level policy learning and skill decoder optimization. When learning the high-level policy, the proposed latent skill embedding can reduce the variance when doing explorations and can be estimated with variational inferences as proposed by this work. Moreover, when learning the low-level policy (the skill model), a residual policy model is proposed to better fit the task requirements. These two ways of using pre-captured data are intuitive and appropriate.\n\nHowever, this work has obvious shortcomings. Firstly, the paper is not so well organized and a little bit hard to read, some of the details are missed like how is the high-level policy defined and trained (why not directly use $\\pi_{HL}=q(z|s)$ and fine-tuning it on tasks), and how will the skill horizon affect the learning performance and efficiency. Then, the survey on hierarchical RL is not sufficient, there exists some work that uses VAE/cVAE for skill discovery and policy learning, like [1,2] and so on, although they come from option-based hierarchical imitation learning, a certain comparative survey is necessary. Further, the comparison is somehow not sufficient, the comparison among HRL methods like Option AC[3], H-DDPG, and so on are welcome.\n\n[1] Sharma M, Sharma A, Rhinehart N, et al. Directed-Info GAIL: Learning Hierarchical Policies from Unsegmented Demonstrations using Directed Information[C]//International Conference on Learning Representations. 2018.\n[2] Hausman K, Chebotar Y, Schaal S, et al. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets[J]. Advances in neural information processing systems, 2017, 30.\n[3] Bacon P L, Harb J, Precup D. The option-critic architecture[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2017, 31(1).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work enhances the skill-based hierarchical RL method from two aspects: a latent prior that can be used for faster exploration and the residual low-level policy, where the technical innovation is limited. Also, the comparisons with HRL methods are insufficient and affect persuasiveness the of the results. Finally, the organization of this work is a little bit unclear, and the experiment and theory do not demonstrate the advantage of using latent prior (see Q.2, Q.3 below).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. For experiments, why is the average reward of ReSkill (No Residual) different from ReSkill within the first 20k steps? As shown in the line.400, the residual policy is inactive so ReSkill (No Residual) and ReSkill will be exactly the same.\n2. The analysis of ReSkill (No Skill Prior) is missing on Sec.6, and the experiment setting that uses samples from Z is ambiguous, does $z~N(0, I)$ or $z~\\pi(z|s)&, maybe the latter would be better?\n3. How is the high-level policy trained? can it be fine-tuned from prior $p(z|s)$? Is there any quantitative analysis on how much could the proposed prior narrows the exploration range?\n4. More comparisons on Hierarchical RL methods are welcomed."}}, {"review_id": "ST439vznph", "reviewer": "Reviewer_daKY", "summary": "This work introduces ReSkill, a method for skill-based RL that enables efficient exploration and adaptation to tasks beyond the initial set of skills. The method is composed of three main parts that are trained sequentially: skill extraction from scripted policy demonstrations, learning a skill embedding and state-conditioned skill prior, and learning a hierarchical policy that combines an action decoder for sampled skills with a learned residual policy. The two main competitive baselines that the paper compares against are PaRRot (non-hierarchical version of ReSkill that uses a similar flow-based state-conditioned generative model for direct action prediction instead of skill prediction) and SPiRL (this can be viewed as ReSkill without the state-conditioned skill prior). ReSkill outperforms all baselines, and ablations show that the skill prior is useful for exploration and the residual policy is useful for adapting to changed environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method’s state-conditioned skill prior is a powerful abstraction and a very clear improvement over the prior work SPiRL in terms of exploration benefit and skill space expressivity.\n- The paper is well presented with great clarity. Each of the method’s three components are explained concisely. The system design and motivations are visualized clearly in Figures 1, 2, and 3. \n\nWeaknesses:\n- The method is only evaluated on simple simulated domains; it is unclear how the method scales with higher dimensional tasks (with complex, high-dimensional skills) where state-conditioning may be intractable\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is presented extremely well and clearly describes the three main components of their method with ample motivation and analysis. The comparisons to relevant baselines and ablations showcase the effectiveness for exploration of the main contribution of the paper, the state-conditioned skill prior via density estimation. While the method is very compelling and technically sound, the main reservation I have is that the method is only evaluated on relatively simple domains, so it is unclear how it extends to more complex domains and especially the real world.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- The test environments differ quite a bit from the training environment used to generate the skills (push and pick and place). As tasks increase in complexity and the test skills are increasingly different from the training skills, the residual policy becomes very important. Therefore, how does ReSkill scale to high-dimensional environments or very diverse tasks, especially as there is an increasing distribution shift between train and test skills?\n- Why is “ReSkill (No Skill Prior)” in Figure 6(a) so much better than SPiRL? Shouldn’t they be very similar?\n- Is the most novel / important contribution the state-conditioned skill prior or the residual policy, given that they both seem important for performance?\n"}}]

Author rebuttal:

Decision: Accept (poster)