ASPiRe: Adaptive Skill Priors for  Reinforcement Learning
Authors: Mengda Xu
Manuela Veloso
Shuran Song
Venue: NeurIPS 2022 Accept (2022)
Keywords: Reinforcement Learning
Skill Learning
Composing Skill

Abstract: We introduce ASPiRe (Adaptive Skill Prior for RL), a new approach that leverages prior experience to accelerate reinforcement learning. Unlike existing methods that learn a single skill prior from a large and diverse dataset, our framework learns a library of different distinction skill priors (i.e., behavior priors) from a collection of specialized datasets, and learns how to combine them to solve a new task. This formulation allows the algorithm to acquire a set of specialized skill priors that are more reusable for downstream tasks; however, it also brings up additional challenges of how to effectively combine these unstructured sets of skill priors to form a new prior for new tasks. Specifically, it requires the agent not only to identify which skill prior(s) to use but also how to combine them (either sequentially or concurrently) to form a new prior. To achieve this goal, ASPiRe includes Adaptive Weight Module (AWM) that learns to infer an adaptive weight assignment between different skill priors and uses them to guide policy learning for downstream tasks via weighted Kullback-Leibler divergences. Our experiments demonstrate that ASPiRe can significantly accelerate the learning of new downstream tasks in the presence of multiple priors and show improvement on competitive baselines.    

ASPiRe: Adaptive Skill Priors for Reinforcement Learning
Mengda Xu 1:2, Manuela Veloso 2.3 Shuran Song
Department of Computer Science, Columbia University
2 J.P. Morgan Al Research School of Computer Science, Carnegie Mellon University (emeritus) attps://aspire.. cs. columbia.eed eduu
Abstract
We introduce ASPiRe (Adaptive Skill Prior for RL), new approach that leverages prior experience accelerate reinforcement learning. Unlike existing methods that learn a single skill prior from large and diverse dataset, our framework learns a library of different distinction skill priors (i.e., behavior priors) from collection specialized datasets, and learns how to combine them to solve a new task. This formulation allows the algorithm to acquire set of specialized skill priors that are more reusable for downstream tasks; however, also brings up additional challenges of how effectively combine these unstructured sets of skill priors form new prior for new tasks. Specifically, it requires the agent not only to identify which skill prior(s) to use but also how to combine them (either sequentially concurrently) to form a new prior. To achieve this goal, ASPiRe includes Adaptive Weight Module (AWM) that learns to infer adaptive weight assignment between different skill priors and uses them to guide policy learning for downstream tasks via weighted Kullback-Leibler divergences. Our experiments demonstrate that ASPiRe can significantly accelerate the learning of new downstream tasks in the presence of multiple priors and show improvement on competitive baselines.
Introduction
Transferring prior experience to new tasks is central to an agent's adaptability. In this work, we aim to accelerate online reinforcement learning by leveraging prior experience from large offline data. Previous works 2 have proposed to extract temporally extended actions into skills from offline datasets and learn single behavior prior over the skill space, which we refer to as skill prior. The skill prior is then used to guide the learning for new downstream task by constraining the learned policy to stay close to the prior distribution. However, regularizing with a single skill prior might be insufficient for more complex downstream tasks. We conclude two major limitations for using a single skill prior: (1) The datasets could be generated by multiple policies for diverse tasks. As consequence, the action distribution from behavior trajectories might be multi-modal, and the resulting learned prior might learn an "average" behavior 4 (2) Complex tasks not only require reusing previous skills but also composing them. Consider task that requires the agent to traverse a maze with obstacles inside. The agent needs to compose navigation skills and avoid skills concurrently when the obstacle is observed (See Fig. TT and only activate navigation skills when is not. Guided with only navigation skill prior or avoid skill prior might lead to inefficient learning.
This observation suggests guiding the learning with multiple skill priors, with each prior specialized in one task. During the learning of new downstream tasks, the most relevant priors should be selected regularize the policy based the task and specific states. Specifically, it requires the agent to: (1) Identify which skill prior(s) to use. Since not all skill priors are relevant to the new task at hand, imposing irrelevant skill priors to regularize can mislead the learning process. (2) Learn how
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

to combine multiple skill priors. Many complex tasks require the agent to identify and combine multiple priors to solve them effectively. The integration of skill priors can be either sequential (active one prior at a time) or concurrent (active multiple priors at the same time), which need to be determined by the agent automatically.
To address the above challenges, we propose ASPiRe, a new method to accelerate the learning of new downstream tasks by guiding the policy with an adaptive skill prior. To implement this idea, we first extract the multiple primitive skill priors from the separated and labeled offline data. Each primitive skill prior is specialized in solving one specific task (e.g., navigation prior and avoid prior). For a new downstream task, the policy is trained with a new task reward and regularized by the adaptive skill prior. The adaptive skill prior can be one of the primitive skill priors or constructed by composing primitive skill priors.
Our method includes Adaptive Weight Module (AWM) that infers an adaptive weight assignment over learned primitive skill priors and uses these inferred weights to guide policy learning through weighted Kullback-Leibler (KL) divergence. The weight applied to the KL divergence between the learned policy and a given primitive skill prior determines its impact on the learned policy. This implicitly composes multiple primitive skill priors into a composite skill prior and guides the policy learning. Our formulation allows the algorithm to combine the primitive skill priors both sequentially and concurrently as well as adaptively adjust them during the online learning phase.
We evaluate our method in challenging tasks with sparse rewards. We demonstrate that our method can accelerate the learning of new downstream tasks in the presence of multiple priors and show improvement over competitive baselines on both learning efficiency and task success rate.
2 Related work
Offline reinforcement learning. Offline reinforcement learning 4 exploits an existing offiin dataset and learns the best policy from it without interacting with environments. This is valuable when interacting with the environment for a task is expensive. Many algorithms have been developed,
both in model-free (5 6 7
9 8
1O,
II 12 [3]
and model-based [1] 15
16
17
18]
settings.
Offline RL has also demonstrated its ability to accelerate online learning 2
3
19]
While most of the
previous works only consider a behavior prior for a single task, the policy is guided by the behavior mixture latent variable model to compose multiple behavior priors, but this framework only activates one behavior prior at a time. How to leverage multiple behavior priors during online learning both sequentially and concurrently remains an open question. Our method provides a solution to adaptively select behavior prior (i.e., skill prior) or compose new prior guide the policy learning.
prior to achieve efficient exploration. Rao et al. 2 proposes
Reusing and Composing skills. It is crucial for intelligent agents to reuse and compose learned skills to solve new tasks. A wide range of works have been proposed solve this longstanding One way to compose and reuse skills is using value functions
problem [1 22
or Numerous works also explore the hierarchical structure in which
23
24
25
cumulants 26 27
28
29
high-level policy integrates a collection of low-level controllers (primitives) [30 31
and each of low-level controllers contains distinct skill. However, most of the hierarchical methods only sequentially activate the primitives. The other line of work for reusing and composing skills is
32
33
34]
to operate on a latent space, which can be mapped to action space [35 36 37 38
39
40
Among
those works, several approaches employ the idea of mixture-of-experts
by explicitly modeling a
library of primitive skills and learning a weight gate function compose the skills 42
43
42
44
These works show that this additional structure is beneficial for transferring skills to complex tasks. Our work can also be viewed through the mixture-of-experts lens, i.c., solve downstream tasks by composing a library of primitives. However, our work treats the composition as regularization to guide the policy learning, instead of executing the composition as a policy directly.
3 Approach
The goal for ASPiRe is to accelerate downstream task learning using library of primitive skill priors. There are two main challenges for this task: 1) Decide which primitive skill prior should be activated given a new downstream task, where using unrelated primitive skill prior could mislead the learning
2

---

Offline
Online
Avoid
state
Navigate
AWM
Wo
O Agent
Obstacle
a
a22
al H
Skill Priors
W
Skill Encoder ((2 a
(z
Adaptive Skill Prior
Skill Embedding 2
Avoid skill
Skill Decoder
P² (2(21s) (2|ss
P P(al 2
N(O.I)
Nav. skill
a1, 02
CH
Skill Embedding Z
Figure I: ASPiRe overview. In offline, we learn shared skill embedding space and a collection of distinct primitive skill priors, e.g. avoid and navigation primitives are learned. The skill encoder embeds an action sequence a, which is sampled from aggregated dataset, into latent vector in embedding Z and the action sequence get reconstructed by the skill decoder. The primitive skill prior (z]s) imitates the state-skill (in embedding space) pairs in dataset DD In online stage, ASPiRe composes multiple primitive skill priors via weighted KI. divergence, where weighs are inferred by Adaptive Weight Module (AWM).
process. 2) How to best combine multiple primitive skill priors, either sequentially concurrently. In particular, to the best of our knowledge, how to simultaneously constrain the policy learning with multiple skill priors is open challenge.
To address these challenges, we propose method that regularizes the policy learning using an "adaptive" skill prior, which can be either one of the primitive skill priors or a newly created skill prior by the composition of multiple primitive skill priors (i.e., the composite skill prior).
Our approach consists of two learning stages: offline and online (See Fig. D the system learns multiple primitive skill priors from set of labeled datasets. Each dataset contains expert demonstrations of one specific primitive task. In the online stage (for downstream tasks), the algorithm creates an adaptive skill prior by assigning weight over primitive skill priors. The weights are generated by the adaptive weight module (AWM). Then they are used to regularize the policy training. In the running example in Fig. T the policy is regularized by avoid and navigation primitives concurrently when the obstacle is observed.
In the offline stage,
Problem Formulation: In the offline learning stage, the algorithm has access datasets (Di,..,,ppy with task labelling (To..., T..Tkk) Each dataset D, contains action-state trajectories 7 [(80, ao).... (sT,ar)) } for task T, (i.e. push). The datasets generated by expert behavior policy f1.... PK. We further assume that none of the single behaviors can individually solve the typically more complex downstream learning tasks.
The online learning stage for the downstream task formulated as a reinforcement learning problem, modelled an infinite-horizz horizon Markov Decision Process (MDP) MM (S,A. A. P, R, p, 7)). where S the continuous state space, .A is the continuous action space, P is the dynamic p($ Se, ag). R is the scalar reward rt, p the initial state distribution and y is the discount factor. The goal is to learn policy n (a, (s)) maximize the expected discounted sum of rewards Et En (nt(ss, ot)].
3.1 Extracting multiple primitive skill Priors
We denote an action sequence {ae,..., --1 with fixed horizon H as skill a,. the offline learning phase, we first learn shared low-dimensional skill embedding space Z from aggregated
3

---

datasets (D:)--n- We then construct primitive skill priors on top of the skill embedding space for each dataset D. Distinct primitive skill priors can generate different types of skills based on the task associated with D;
To learn the skill embedding. following the previous works 12
45
fit variational autoencoder
and maximize the following evidence lower bound (ELBO):
logp(a;) 2 E logp(a,)2) 3(
log (( a.) - p(z)
(I)
roconstruction
rogularization
Here, ,3 is a parameter that controls the regularization term. The skill encoder g(=la,) embeds the skill a, into latent space Z and the skill decoder p(as|z) reconstructs the latent vectors back to skills. unit Gaussian prior to regularize the encoder learning. Note that action sequences
We set p(z) a are randomly sampled from all offline dataset {D:),- such that diverse action sequences can be embedded in shared skill embedding space with the same encoder and decoder.
We freeze the skill encoder and decoder parameters after sufficient training. Next, we extract multiple primitive skill priors on top of the shared skill embedding. For each offline dataset D, we extract primitive skill prior Pr (=|(8)), which is parameterized as a Gaussian distribution. To train primitives, we sample state and skill tuples (8t, ar) from D; and minimize the KL divergence between the skill posterior and the predicted prior (s,,a))) DKL (q(z [a),, Pá (z]s:)), similar to previous work (2].
3.2 Regularized reinforcement learning via adaptive skill prior
For downstream task learning, instead of learning policy on action space A, we learn policy TT(2)st) which acts on high-level skill embedding space Z. Previous work [2] has demonstrated that learning on the skill embedding space is beneficial for complex long-horizon tasks. The skill embedding z can be decoded into an action sequence via the skill decoder p(alz) for H steps action execution. We further replace the dynamic in our problem formulation with H-step dynamic p' p'(4 fr (rt, 2+) and rewards with H-step rewards ry- = - Tt.
We guide the policy learning via adaptive skill prior over the course of online learning for downstream tasks. Ideally, the policy should be regularized by all relevant primitive skill priors at each state. We consider two scenarios: 1) only one primitive prior is relevant, 2) multiple primitive priors are relevant. Previous work (2) considers the first scenario and regularizes the learning policy by providing an additional negated KI. divergence between the policy and the offline skill prior. Similarly. SVPG [46] regularizes the policy parameters by a prior distribution and optimize the object via SVGD
In states where multiple primitive skills are involved, the agent needs to achieve several goals simultaneously, which are implicitly encoded in distinct primitive skill priors. It is therefore natural to guide the policy learning with multiple primitive skill priors. To implement this idea, we can augment the standard RL objective with negated weighted KL divergence:
T Σ t=
K
(
Ex
r(st, 2)
wi (((s)))a(((())))))d
(2)
weighted KL divergence
Here, we refer to w(s) as the weighting function, which specifies the weights (W1, w2 ..., wx) to apply on KL divergence between the policy and primitive skill priors. We consider weights are bounded (w; Ju E [0, 1] and are normalized ΣX W;- = 1. Assigning weights to the primitive skill priors implicitly forms a composite skill prior. This can be interpreted as the learned policy will be constrained by skill distribution (2(2,) that can minimize the weighted KI. divergence given weights w at state st:
K
Pz(2)) = arg min w (.)
(3)
Po

---

We refer to it as composite skill prior generated by weight w at state Weights at a given state can be viewed as a measure of how much one primitive skill prior should contribute the composite skill prior. The state-dependent weights can be learned simultaneously with the policy, and we discuss it in section 3.3 For now; we assume the weighting function w(s)) is given, and the goal to maximize the Equation 2
We follow the policy constraints formulation 5
10 optimize the objective. We consider
Q-function Qe and policy parameterized by the network with parameters á and 0, respectively, The policy parameters can be optimized by minimizing the weighted KL divergence penalty:
K
(0) = EagD BE [ Q0(s,, ze) + wi(s:) ((x))))xr P
(4)
Here a 2 is a temperature parameter to constrain the weighted KL-divergence, which can be tuned automatically by providing a target divergence parameter [2] (See appendix A.3 for details).
Q-function parameters be optimized by minimizing the Bellman residual:
(Ø) = E, 2 (Qo(s), 24) - (r(st, 2)) + (0(241s44))))) (5)
3.3 Adaptive Weight Module
In this section, we introduce Adaptive Weight Module (AWM), component to infer the optimal weights adaptively over the course of tasks. Our goal is to learn parameterized weighting function Wa to compose the primitive skill priors via weighted KL divergence, and the resulting composite skill prior can guide the policy learning. We evaluate the expected critic value of skills under the composite skill prior distribution generated by the weights w. The weights leading to the highest critic value is selected to assign among primitives. Both the weighting function Wa and the skill prior generator G. are learned online along with the policy Ty.
Learning skill prior generator. To efficiently evaluate the skills under composite skill priors, we implement skill prior generator (zle,, w), generative model which outputs the composite skill prior generated by the weights w given the state 8t. This can be optimized by minimizing the weighted KL divergence between output distribution G-,(-1s,, w) and primitive skill priors:
K
Jo(n) = ELNND Σ
w. DKL(G, w). P.(c(ss)))
(6)
We parameterize the skill prior generator a Gaussian distribution by the network with parameters 77. During the learning, we randomly sample weights w from the unit simplex as the weight input. This helps the skill prior generator to learn all possible compositions across states.
Learning adaptive skill prior. Our goal to construct an adaptive skill prior that biased towards the high reward trajectories, similar to Siegel et al. [10]. The key difference is that our approach aims to construct such adaptive prior for novel downstream task by composing multiple primitive skill priors. In effect, we would like to find weights assignment such that the corresponding composite skill prior can contain the skills that are more likely to lead to task success. We formulate this as:
arg max
EE.DD (t. (Q.
where Q6 is the policy critic defined in section 3.2 The Q-value in the objective measures the usefulness of skills for the task at hand. The inner expectation can he viewed as a measurement the usefulness of the composite skill prior generated by weight w.. The objective prefers composite skill prior, in which skills have higher Q-value and are more promising to be explored. We approximate the inner expectation with M samples from G,,. i.e, Eg~G,, (=))) (+:)) (s =)] 1 (st,s,))) G-,(118, Wa (s:))) We find M 20 is sufficient to provide stable and accurate estimation.
We parameterize the weighting function wi by the network with parameter o, and use softmax as the output layer. To learn the optimal weight, set the Equation 7 as the loss function. We add an
5

---

Algorithm ASPiRe Algorithm
1: Inputs: K primitive skill priors { (Pa st
2: Initialize: replay buffer D, policy To(22 st), critie Qø(se, 26). target critic Q5(st, 2)). weighting
function wa (st), skill prior generator Gy(2 St, wx)
3: for each iteration do
4: for every H environment steps do
5:
21 ~ (8) 8: p(s+ St, 21) + D--u (st, Ze, Tr, s/}
D. Sample skill from policy Execute skill
6:
7:
Store experience in replay buffer
8:
end for
9:
for each gradient step do
Sample batch of transition tuples (8ts Rts Tay {1} from D
11:
Wt = Wo (st)
>> Get weights from weighting function
0 Ag V
Update policy parameters via weighted KL divergence
13:
Φ - A A.VoJoo (
> Update critic parameters
π - 1 X, V.J(m)
Update generator parameters
15:
- 0 [J)(O)
> Update weighting parameters
- TØ + T)Φ
b Update target critic
17:
end for
18:
end for
additional regularization term to penalize weights for deviating from uniform weight K in the early stage training. This encourages the learned policy to explore more skills diverse primitive skill priors and prevents weights from collapsing to a singleton due to the noise in Q-value in the early phase. Here, the annealing coefficient, and gradually decreases. The weighting function parameters be trained minimize the following:
Jw( Es,DD BI-GGt (w))) [ Qo(s),)) + Bllwo (s:)
1
For each gradient step, optimize both the policy and the AWM. The full Adaptive Skill Prior algorithm is summarized Algorithm T
For implementation details, see appendix, A.4.
4 Experiments
We would like to answer the following questions through our experiments: a) Can leveraging multiple primitive skill priors accelerate downstream task learning? b) What's the advantage of using the composite skills as regularization compared to directly using them policy? c) Can AWM optimally assign proper weights over different primitive skill priors based on the downstream tasks, even when irrelevant primitive skill priors presented?
4.1 Experiment setup
We evaluate our method in three modified environments from D4RL 48 and one modified envi- ronment from robosuite environment [49]. ASPiRe given two primitive skill priors in the first two experiments. For the last experiment, we test ASPiRe's ability to compose three primitive skill priors. For each environment, we first collect datasets for distinct skills from expert demonstrations, which allows to extract distinct primitive skill priors. We then evaluate our method with extracted primitives downstream tasks.
Point Maze. We modified the Point Maze environment by adding multiple obstacles in the maze. The task navigate point mass agent in the maze and reach the goal. During this process, the agent must avoid the obstacles randomly placed in the maze. The original point maze environment is already challenging because the sparse reward feedback. Randomly placing obstacles in the maze makes this problem even more difficult. For each episode, the maze layout is fixed. However, the agent start, goal, and obstacles locations are randomly specified. Our method will be equipped with two primitive skill priors for this task: navigation and avoid.

---

Ant Push. The ant needs to push the box to the goal while avoiding an obstacle in this task. ASPiRe will carry two primitive skill priors: push and avoid. The ant, box, obstacle, and goal locations are randomly generated at the heginning of every episode. The agent will receive a sparse reward upon the box reaching the goal location.
Ant Maze. We modified the Ant Maze by adding an obstacle and box in the maze. The agent's goal is to traverse the maze while pushing the box to the goal position without hitting the obstacle. The agent always starts in the lower right corner of the maze. However, the box, goal, and obstacle positions are randomly selected from a pre-defined set. Ant Maze itself poses a complex exploration problem due to sparse reward feedback. We make this problem even more complex by changing the success criteria. In this case, ASPiRe will carry three primitives: push, navigation, and avoid.
Robotic Manipulation. We further test ASPiRe's ability in complex robotic manipulation task in robosuite environment 49]. The task is control Panda robot to grasp the box (in red) without colliding with the barrier (in green) placed in the middle the desk (See sub-figure 1 in Fig. The location of the box and barrier are randomly generated the beginning of the episode. ASPiRe will carry two primitive skill priors: grasp and avoid.
See appendix A.4 for details environment, data collection process and training.
Point Maze
Ant Push
Ant Maze
Robotic Manipulation
2.
Success Rate
2.4
04
0.2
1.2
0.0
3.0
06
06
06
0.5
0.30
375
135
130
885 alo
ais
025 G2
Environment Steps (1c7)
Environment Steps (1e7)
Environment Steps (1c7)
Environment Steps (167)
ASPiRe
BC+SAC
MCP
Scratch
SPiRL
Figure 2: Comparative Experiment. Learning curves of our method and comparative methods for downstream tasks. ASPiRe significantly improves the learning efficiency and performance on all challenging tasks. The shaded area is the standard deviation over eight seeds.
Alternative approaches. We compare ASPiRe with several prior works for downstream task learning performance. To fairly evaluate the performance, let all methods (besides SPiRL, as the method learns its own skill embedding) act on skill embedding space Z learned by the ASPiRe.
Scratch. Train a Soft Actor-Critic (SAC, (50) agent from scratch. This comparison tests the benefit of using the learned skill priors to guide downstream task learning.
Behavioral Cloning with finetuning. behavior cloning policy first trained with offline data and then finetuned using SAC for the downstream tasks.
MCP(51). The method composes primitive policies through multiplicative composition of Gaussians with a learnable weighting function. MCP then samples the actions from the resulting multiplicative composite distribution. For a fair evaluation, MCP is directly given the same set of primitives that ASPiRe is equipped with. This comparison tests the benefit of treating composite skills as prior, instead of directly executing them.
SPiRL 22 Guide the policy learning with a single skill prior. We train the skill prior by combining all primitive datasets into one and learning single prior from it. This comparison tests the benefit of guiding the learning policy with multiple primitive skill priors.
4.2 Experiment results
As shown in Fig. 2 our experiments demonstrate that ASPiRe able to learn efficiently and converge to high success rates in all challenging environments.
7

---

Benefits of multiple primitive skill priors. We first compare ASPiRe with methods without skill prior guiding, namely training from scratch and behavior cloning with finetuning. They fail solve all three tasks (Fig. 2 This confirms the finding in Pertsch et al. [ that skill prior is beneficial for learning complex and long-horizon tasks. Next, we compare our method with SPiRI. [2). which regularizes the learned policy with single skill prior extracted from the aggregated dataset. By leveraging the skill prior, it can achieve more efficient exploration but still relatively low learning speed and final success rates compared to ASPiRe (Fig. 2 We hypothesize that the performance gap between ASPiRe and SPiRL comes from the prior they deploy to guide the policy learning. We find that SPiRL learns "average" behavior over all primitives. It prevents SPiRL from guiding states where the specific primitive skills are required. In the case of ASPiRe, it can construct an adaptive skill prior from all primitive skills priors. Therefore, the learned policy can receive sufficient guidance to explore and learn the tasks.
Point Maze
Weights
0.4-
nevigation
avold
Steps
O Agent
Goal
Obstacle
Figure Weights inference over one episode (Point Maze). ASPiRe is able compose the navigation prior and avoid prior concurrently (grey phases) when the obstacle is observed and only activate navigation prior when it is not (white phases).
Ant Maze
2.4
Weights
Steps
push
nevigation
avoid
Box
Goal
Obstacle
Figure 4: Weights inference over one episode (Ant Maze) In the grey phase, ASPiRe able to
compose the navigation prior and push prior concurrently traverse the maze and reach the box. In the white phase, the model composes the push and avoid prior to push the box to the goal while avoiding the obstacle along the way.
Benefits of using composite skills regularization. We compare ASPiRe with MCP, which directly executes the composite skills as a policy with learnable weight. As shown Figure 2 MCP struggles to learn meaningful policy for all three tasks. This finding demonstrates that using composite skills as regularization provides the necessary flexibility for ASPiRe to learn the policies that specifically solve the downstream task. We find the final learned policy has lower entropy than the skill priors in most cases. In fact, skill priors provide guidance on the valid skills to explore the environment, but the learned policy needs to further search in this valid skill space solve the specific problem. We hypothesize that the resulting policy from MCP might be overly broad as the multiplicative composition of Gaussian inherent the relative high entropy from the primitives.

---

Robotic Manipulation
Weights
avcid
Figure 5: Weights inference over one episode (Robotic Manipulation) the first grey phase (sub-figure 1, 2, 3). ASPiRe is able to compose the grasp prior and avoid prior concurrently to lift the robot arm to avoid the barrier while approach the box. In the white phase, notice that the robot arm is initially very close to the barrier (sub-figure 4). To avoid any potential collision between the robot arm and the barrier, ASPiRe activates the avoid prior to move the robot arm away from the barrier (sub-figure5). In the end, ASPiRe activates the grasp prior to grasp the box.
Learning optimal weights. To demonstrate AS- PiRe's ability to learn optimal weights, we visual- ize the weights generated by the weighting func- tion for the primitive skills priors at different states for point maze, ant maze and robotic manipula- tion environments. From Fig. 3 4 and []] we find that ASPiRe can compose the primitive skill priors both sequentially and concurrently through- out the tasks. More importantly, ASPiRe regular- izes the learned policy with relevant priors, e.g., activates the avoid primitive when obstacles are near the agent. See appendix for more details.
Succes Bate
Weigh
(1c7)
Enviccenent
Push Asard Nav.
Figure 6: Training with unrelated primi- tive. Left: Learning curve. Right: Average weights of skill priors, where we can observe that ASPiRe assigns lower weight for unre- lated skill prior (e.g., push).
Training with irrelevant primitive. To test AS- PiRe's ability in handling irrelevant primitive skill prior, modify the task of the Ant Maze environ- ment, where the ant needs to reach the goal while avoiding the obstacle along the way without pushing the box. We place the box in the lower right corner of the maze, which is not on the path the goal. In this case, the push primitive is irrelevant, and reg- ularizing with it will yield low learning efficiency. From Fig. 6 we find that ASPiRe still achieves good performance even with irrelevant primitive. We plot average weights over the states during the training. ASPiRe weights the navigation primitive heavily and assigns a low weight for the irrelevant primitive push. The weight for avoid also relatively low since there are fewer states where avoiding is needed (i.c., only when the obstacle is in view).
0.8
Swccess Rate
02
0.0
0.0
0.2
0.6
0.5
0.0
1.0
1.2
1.4
Envirenment Steps (1e7)
ASPiie
Navigation
Avoid
Guiding with composite or single primitive. In this experiment, we test the advantage of using composite skill prior to guiding policy learning compared to guiding with a single primitive skill prior. To answer the question, we guide the policy learning with one of the primitive skill priors (i.e., push, navigation and avoid respectively) in ant maze environment. From Fig. 7 we find that the learned policy guided with only the avoid primitive or navigation primitive fails to learn the task. Guided with the push primitive achieves relatively good performance but fails to complete the setting requiring traverse the maze. This experiment suggests that composing multiple skill priors is beneficial for the complex multi-task downstream tasks.
Figure 7: Guiding with composite single primitive. Learning curves in Ant Maze.

---

Limitation and future work. When extracting primitive skill priors, our algorithm requires separated and labeled datasets, each containing specialized skill. However, such dataset may not always be easy to generate, and manually separating or defining different primitive skill priors can be challenging and sometimes ambiguous. Therefore, an interesting future direction will allowing the system to process unlabeled dataset and automatically extract a set of distinct primitive skill priors.
5 Conclusion
We present ASPiRe, an approach to transfer a spectrum of skills from offline data to accelerate the learning of unseen downstream tasks. We propose to guide the policy learning with multiple primitive skill priors and transfer the diverse skills to RL via weighed KL divergence. We model an Adaptive Weight Module to learn optimal weights assignment and construct an adaptive skill prior. We evaluate ASPiRe in challenging, sparse reward environments and demonstrate the benefits of guiding the learning with multiple skill priors. We also show that ASPiRe is able to select the most relevant skill priors even when unrelated skill priors are presented.
Acknowledgement
The Authors would like to thank Nelson Vadori, Sumitra Ganesh, Huy Ha, Cheng Chi, Samir Gadre, Zhenjia Xu, and Zhanpeng He on valuable feedback and support. Mengda Xu's work is supported by JPMorgan Chase & Co. This paper was prepared for information purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co and its affiliates ("JP Morgan"), and is not product of the Research Department of JP Morgan. JP Morgan makes no representation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute a solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction such person would be unlawful.
References
[1] Anurag Ajay, Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Offline primitive discovery for accelerating offline reinforcement learning. In International Conference on Learning Representations, 2020.
[2] Karl Pertsch, Youngwoon Lee, and Joseph J. Lim. Accelerating reinforcement learning with learned skill priors. In Conference Robot Learning (CoRL), 2020.
[3] Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine Parrot: Data-driven behavioral priors for reinforcement learning. In International Conference on Learning Representations, 2020.
[4] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, CoRR, abs/2005.01643. 2020. URL https://arxiv.orr wtt.://r//aab//...0000000
[5] Aviral Kumar, Justin Fu, G. Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In NeurlPS, 2019.
[6] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. CoRR, abs/1911 1361, 2019. URL htttp://arxivv.rg./ http:/pp///axv.oorrr///////.1... wttp://///a////..11a1311
[7] Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Off-policy policy gradient with state distribution correction. CoRR, abs/1904.08473, 2019. URL httt:://arxiv.. org/ abss/900 0tt//1904008773.
[8] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In ICML, 2019.
10

---

[9] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances Neural Information Processing Systems, 33:
11 1179-1191, 2020.
[10] Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference Learning Representations, 2019.
[11] Ziyu Wang, Alexander Novikov, Konrad Zomna, Jost Tobias Springenherg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, al. Critic regular- ized regression. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 7768-7778, 2020.
[12] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. Striving for simplicity in off-policy deep reinforcement learning. CoRR, htss/997.005544 2019. URL httpp:/arraxi org/abs/ 1eabba/1907.0440
[13] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Ägata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. CoRR, 0bss/190..00455 2019. URL //arxiv. org/ www./og/////////eeeeeeeeeeeegeee ab9//90////1////////// 00456
[14] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 13:21810-2188 21823, 2020.
[15] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems,
14142,, 2020.
[16] Arthur Argenson and Gabriel Dulac-Amold. Model-based offline planning. In International Conference Learning Representations, 2020.
[17] Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment-efficceen reinforcement learning via model-based offline optimization. In In- ternational Conference on Learning Representations, 2020.
[18) Catherine Cang, Aravind Rajeswaran, Pieter Abbeel, and Michael Laskin. Behavioral priors and dynamics models: Improving performance and domain transfer in offline rl. In Deep RL Workshop NeurlPS 2021, 2021.
[19] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online re- inforcement learning with offline datasets. CoRR, 00n/2006.09359, 2020. URL httpps
wwggabs//// // //arxiv. orgoorggggoo abs 2006. 0036609359999.
[20] Dushyant Rao, Fereshtch Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, al. Learning transferable motor skills with hicrarchical latent mixture policies. In International Conference on Learning Representations, 2021.
[21] Fernando Fernández and Manuela Veloso. Probabilistic policy reuse in a reinforcement learning agent. In Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS '06, page 720-727, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933034, doi: 10.1 160633.0631111 166766. 160762. URL https
aidoi.org/10 1145/1160633 1160762
[22] Brett Browning, James Bruce, Michael Bowling, and Manuela Veloso. Stp: Skills, tactics, and plays for multi-robot control in adversarial environments. Proceedings of The Institution of Mechanical Engineers Part I-journal of Systems and Control Engineering- PROC INST MECH ENG I-J SYST C, 219, 12 2004, doi: 00.1243/95995115509770

---

[23] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel, and Sergey Levine. Composable deep reinforcement learning for robotic manipulation. CoRR, abs// 1oss//003.00773. 2018. URL
wttp://.rrggaab//8..... 06773
[24] Jonathan J. Hunt, André Barreto, Timothy P. Lillicrap, and Nicolas Manfred Otto Heess. Composing entropic policies using divergence correction. In ICML, 2019.
[25] Benjamin Van Nickerk, Steven James, Adam Earle, and Benjamin Rosman. Composing value functions in reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning. volume 97 of Proceedings of Machine Learning Research, pages 6401-6409. PMLR, 09-15 Jun 2019. URL nttps://proceediing mlr.
wttp:///aaanneeeeeaaaa.. html
[26] André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
[27] Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Matteo Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pages 501-510. PMLR, 2018.
[28] Diana Borsa, Andre Barreto, John Quan, Daniel J Mankowitz, Hado van Hasselt, Remi Munos, David Silver, and Tom Schaul. Universal successor features approximators. In International Conference on Learning Representations, 2018.
[29] André Barreto, Diana Borsa, Shaobo Hou, Gheorghe Comanici, Eser Aygün, Philippe Hamel, Daniel Toyama, Jonathan J. Hunt, Shibl Mourad, David Silver, and Doina Precup. The option keyboard: Combining skills in reinforcement learning. In NeurIPS, 2019.
[30] Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture, 2016.
[31] Matthew Hausknecht and Peter Stone. Deep reinforcement learning in parameterized action space, 2016.
[32] Kevin Frans, Jonathan Ho, Xi Chen, Pieter Abbeel, and John Schulman. Meta learning shared hierarchies. In International Conference on Learning Representations, 2018.
[33] Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqj Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. In International Conference on Learning Representations, 2018
[34] Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Ne- unert, Tim Hertweck, Thomas Lampe, Noah Y. Siegel, Nicolas Heess, and Martin A. Riedmiller. Regularized hierarchical policies for compositional transfer in robotics. CoRR, abs/1900 1228,
2019. URL http::/ wwwp:/ii.rrg//baa.. org/ http:/bs//19000 11228
[35] Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers, 2016.
[36] Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. CoRR, abs/ 1Ss1700403012, 2017. httptttp://///// http:://axxi..ottt/ org/ abs/ 1704 03012
[37] Josh Merel, Leonard Hasenclever, Alexandre Galashov, Arun Ahuja, Vu Pham, Greg Wayne, Yee Whye Teh, and Nicolas Heess. Neural probabilistic motor primitives for humanoid control. In International Conference on Learning Representations, 2018
[38] Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. In International Conference on Learning Representations, 2018
[39] Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2018.
12

---

[40] Tuomas Haarnoja, Kristian Hartikainen, P. Abbeel, and Sergey Levine. Latent space policies for hierarchical reinforcement learning. ICML, 2018.
[41] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-77, 1991. doi: 10. 10.1162/neco.199 1991.33.1 79.
[42] A. H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C. Yip. Composing agnkstgn policies with deep reinforcement learning. In ICLR, 2020.
[43] Wei-Cheng Tseng, Jin-Siang Lin, Yao-Min Feng, and Min Sun. Toward robust long range policy transfer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9958-9966, 2021.
[44] Jie Ren, Yewen Li, Zihan Ding, Wei Pan, and Hao Dong. Probabilistic mixture- mixture-of-expert experetts--eeeeeeeeeeeeeeee for efficient deep reinforcement learning, 2021.
[45] Xiaofei Wang. Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback. In Conference on Robot Learning, pages 1259-1268. PMLR, 2022.
[46] Yang Liu, Prajit Ramachandran, Qiang Liu, and Jian Peng. Stein variational policy gradient. In 33rd Conference on Uncertainty in Artificial Intelligence, UAI 2017, 2017.
[47] Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances in neural information processing systems, 29, 2016.
[48] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[49] Yuke Zhu, Josiah Wong, Ajay Mandlekar, and Roberto Martin-Martin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009. 12293, 2020.
[50] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
[51] Xue Bin Peng. Michael Chang, Grace H. Zhang, P. Abbeel, and Sergey Levine. Mcp: Learning composable hierarchical control with multiplicative compositional policies. In NeurlPS, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] The code in the supplement material. We provide the codes to regenerate the datasets and downstream task learning and all instructions inside readme.md
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See appendix A.4
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] The shaded area in Fig. represents the standard deviation across 8 seeds
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix A.4
4. If you using existing assets (e.g., code, data, models) curating/helesiing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use the D4RL.
benchmark and cite in the section 4
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "idQrqcfDRSI", "reviewer": "Reviewer_iPXJ", "summary": "The authors present a method for 1) extracting skills from offline demonstration datasets and 2) using these learned skills for accelerated learning of downstream tasks, where the new task might often require composing two or more skills (either concurrently, or sequentially). For (1), the authors first train a VAE on action sequences from the offline dataset to learn a latent space of skills. They then extract primitive skills priors P^i_a(z|s_t) (one for each dataset D_i) by minimizing a KL divergence term (line 140 in the paper). Finally, when learning a new task, the policy that is being learned is constrained to stay close to a mixture of skill priors through a KL divergence term. The weights of this mixture (called the adaptive weight module) is learned so as to maximize the Q-values of the downstream task. \n\nThe authors evaluate their method on three simulated navigation tasks. In two of the tasks, two skill priors need to be combined for learning a new task, while there are three skill priors in one of the tasks. The authors provide comparisons against SPiRL (the method on which they closely build), learning from scratch, and naively combining BC with RL, and show that their method significantly outperforms these other techniques. ", "questions": "- Since I have concerns regarding the overall training stability of the method (due to the large number of moving parts), can the authors share in detail what kind of hyperparameter sweeps had to be done? Generally speaking, a hyperparameter sensitivity analysis would be very insightful for this work. \n- Did you try running the method on domains other than navigation? What did you find? Even negative results can be useful to look at here. \n- As mentioned in the introduction, one of the relevant related work is PARROT, but there is no comparison provided to this method. Would it be possible for the authors to run this comparison? \n\nI am open to updating my rating if the authors can provide answers to these questions / make edits to the paper that address these concerns. \n", "limitations": "I think the limitations section in the paper is very limited. The only limitation that the authors mention is the difficulty of collecting useful prior dataset, but I think the work has other limitation as well (mentioned in the weaknesses and questions section above) that would be worthwhile to discuss in more detail. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The paper tackles an important problem – how can we compose a diverse set of previously learned skills to solve new tasks quickly? \n- The paper shows strong performance when compared to the mostly closely related work - SPiRL. \n\nWeaknesses\n\n- The proposed method has a lot of moving parts: skill encoder / decoders, skill priors (one per skill), downstream policy, Q-function, skill prior generator, adaptive weight module, and so on. All of these various components are closely dependent on each other, and makes me a little concerned about the overall training stability of the method and sensitivity of hyperparameters. \n- The experiments are fairly small-scale. The main point of the paper is that it allows combining multiple skill priors to solve the tasks, but two of the experiments involve combining only two skill priors, and one set of experiments involves combining three skill priors. The experiments are also all in navigation domains, so it is unclear if the method will work on other domains, say robot manipulation or Atari games. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bdp2GEZgkXZ", "reviewer": "Reviewer_pJKn", "summary": "The paper tackles the problem of composing skills from multiple offline datasets. Aspire learns multiple priors from different offline datasets and then learns a set of weights over the priors and a composite prior predictor during online training on new tasks that require application of multiple skills. The method outperforms BC, Multiplicative Compositional Policies (MCP), and SPiRL on a set of 3 Maze tasks that require the agent to compose pushing, avoiding, and navigating actions. ", "questions": "- Is $G_\\mu$ learned offline and $\\omega_\\sigma$ learned online?\n- What $\\omega$ is $G_\\mu$ conditioned on in equation 6? Should it be $\\omega_i$?\n- Are all of the baselines trained with the same number of parameters?\n- Lines 305-307 state that the push agent fails to complete the setting requiring traversing the maze. But in figure 4 the weight of the push prior is highest and sometimes 1 while traversing the maze. What accounts for this counterintuitive behavior? Why is the push prior so much stronger than navigation in figure 6?", "limitations": "The authors describe that their algorithm relies on the presence of labeled skill data for offline training and depends on a set of specified skills in advance.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n  - Clarity: The abstract and introduction state the motivation and setup clearly. I found the method section up to 3.3 to be straightforward and concise.\n  - Significance: The authors demonstrate that aspire can effectively learn composite policies from offline data of diverse and distinct skills. These results hold even in the presence of distractor skills. Work in this direction enables datasets to be leveraged for a broader set of target control tasks.\n  - Quality: The evaluation is thorough: the authors test multiple relevant baselines (MCP, spirl) and show the effectiveness of the method in the presence of irrelevant skills.\n  - Originality: To my knowledge, the algorithm and associated losses for a policy regularized by composed primitives is novel.\n- Weaknesses\n  - Clarity: The explanation of the approach could be made more straightforward.\n    - It would be useful to clarify what is learned offline vs online in section 3, especially 3.3. \n    - The motivation behind the skill prior generator is difficult to understand because there is no reference to the skill prior generator in the introduction text of section 3 or in figure 2. Based on the previous sections, I expected the adaptive weight module to be a convex combination of the individually learned priors. \n  - Quality:\n    - A baseline that composes the learned priors with a mixture of experts model (i.e., $G = \\sum_{i=1}^Kp_a^i(z|s)$) would clarify the need for the skill prior generator.\n    - Figure 6 shows that the push prior accounts for much of the success of the model and Figure 4 shows that, even for maze traversal, the push prior is contributes the most weight to the policy. This counterintuitive result is not explained.\n  - Significance:\n    - The baselines and prior work present results on more challenging control tasks. E.g., MCP studies tasks on biped, humanoid, and t-rex in addition to ant. SPiRL studies FrankaKitchen.\n  - Originality: Section 3.2 could make reference to SVGD [1] and SVPG [2]. SVPG has a similar prior regularization, but for policy learning directly as opposed to skill primitives.\n\n[1] https://arxiv.org/abs/1608.04471\n[2] https://arxiv.org/abs/1704.02399", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N1VDdVwgmbv", "reviewer": "Reviewer_1oiG", "summary": "This paper extends the prior work on skill prior-regularized RL by learning a skill prior for each skill. When K skill labels are available from the offline data, we can learn K skill polices using VAE as well as K skill priors, which tell an agent whether each skill is plausible to execute given a state. To intermingle K different skill priors into one composite skill prior for downstream RL, a weighting function of K skill priors is learned to maximize the estimated Q-value when following the composite skill prior. One benefit of the composite skill prior is that it enables simultaneous skill composition, e.g. navigating towards a goal while avoiding an obstacle. The experiments demonstrate that the proposed method can effectively combine up to three skill priors, and tackle downstream tasks by not only exectuing one skill at a time but also executing composite skills.\n", "questions": "- Most papers are cited as arXiv papers, which are conference or journal papers. Please use appropriate citation.\n\n- There are many typos (including notations) and grammatic errors. The writing generally needs some improvement.\n", "limitations": "The limitations and potential negative societal impact are well addressed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n- Figure 3 and 4 are very intuitive and well explain that how the proposed adaptive weight module learns to regulate the downstream task policy.\n\n- The proposed method of weighting different skill priors is novel and the empirical results show the effectiveness of using multiple skill priors quantitatively and qualitatively.\n\n### Weaknesses\n\n- In L28-32, the claim about the single skill prior not being able to handle composition is not very intuitive. Also, Figure 1 does not seem relevant to this context. If this composition is about a mixture of two skills (simultaneous execution of two skills), it should be more explicilty described.\n\n- \"the learned policy\" sounds weird in some contexts, e.g., L43, L46.\n\n- The assumption of having task labels for trajectories can limit the applicability of the proposed method. The major benefit of the prior work, SPiRL, was in being able to leverage unstructured data.\n\n- It might be good to show the performace of the uniform weights for multiple skill priors to see the effect of the proposed adaptive weight module. Given that only push skill prior can achieve comparable results with ASPiRe in Figure 6, this simple baseline may work as well.\n\n- Is the MCP baseline identical to using the proposed composite skill prior as a policy? Otherwise, it would be important to include this baseline.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "2hnY116bYeo", "reviewer": "Reviewer_pGpR", "summary": "This paper presents a hierarchical Reinforcement Learning method, where the high-level controller chooses skills that maximize the reward while staying close to a weighted combination of all skill priors. The skills and skill priors are obtained by fitting one skill prior to each labeled dataset using a VAE. A key contribution of this work is the adaptive skill prior, which learns to decide the weighting among skill priors that leads to the highest critic value. The method is evaluated in 3 simulated environments and is shown to be advantageous in terms of sample efficiency and final model performance.", "questions": "- As talked about in appendix A.2.1, the skill encoder and decoder are essentially learned from a single primitive and expected to generalize to other primitives. Would this make the skill latent space z more tailored towards the first task?", "limitations": "A major limitation of hierarchical learning with fixed primitives is that the task could be impossible to complete if the primitives are not comprehensive enough. Also as the authors have discussed in the paper, separate datasets of different primitives are required. It's unclear how this method will perform in harder downstream tasks that require more primitives.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The components in the proposed approach are mostly well explained, and the whole idea is sound.\n- The Adaptive Weight Module is a novel method to assign optimal weightings among priors to generate skills with high values.\n- The visualization of weights along a rollout trajectory is very intuitive.\n\nWeaknesses:\n- The experiment environments are from a similar domain. In particular, the point mass and ant differ mainly in terms of low-level dynamics, which mainly challenges the skill learning stage (not the main contribution of this work). Thus, the conclusion drawn seems a bit weak.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept