Leveraging Demonstrations with Latent Space Priors
Authors: Jonas Gehring
Deepak Gopinath
Jungdam Won
Andreas Krause
Gabriel Synnaeve
Nicolas Usunier
Venue: Accepted by TMLR ()

Abstract: Demonstrations provide insight into relevant state or action space regions, bearing great potential to boost the efficiency and practicality of reinforcement learning agents. In this work, we propose to leverage demonstration datasets by combining skill learning and sequence modeling. Starting with a learned joint latent space, we separately train a generative model of demonstration sequences and an accompanying low-level policy. The sequence model forms a latent space prior over plausible demonstration behaviors to accelerate learning of high-level policies. We show how to acquire such priors from state-only motion capture demonstrations and explore several methods for integrating them into policy learning on transfer tasks. Our experimental results confirm that latent space priors provide significant gains in learning speed and final performance. We benchmark our approach on a set of challenging sparse-reward environments with a complex, simulated humanoid, and on offline RL benchmarks for navigation and object manipulation.

Published in Transactions on Machine Learning Research (03/2023)
Leveraging Demonstrations with Latent Space Priors
Jonas Gchring
jgchring@@mela.cmmm com
Meta Al (FAIR), ETH Zürich
Deepak Gopinath
dgopinath @metta jungdam@mett com
Jungdam Won
Meta AI (FAIR), Pittsburgh, PA
Andreas Krause
krausea@ethz.ch
ETH Zürich
Gabriel Synnaeve
gab@meta. com wswnier@meta.coo com
Nicolas Usunier
Meta AI (FAIR), France
Reviewed on OpenReview: https: // openreview. net/ forum? id- O=CIn/T4Cx
Abstract
Demonstrations provide insight into relevant state action space regions, bearing great potential ial to boost. the elficiency and practicality of reinforcement learning agents. In this work, we propose to leverage demonstration datasets by combining skill learning and sequence modeling. Starting with learned joint latent space, we scparately train a generative model of demonstration sequences and accompanying low-level policy. The sequence model forms latent space prior over plausible demonstration behaviors to accelerate learning of high-level policies. We show how to acquire such priors from state-only motion capture demonstrations and explore several methods for integrating them into policy learning on transfer tasks. Our experimental results confirm that latent space priors provide significant gains in learning speed and final performance. We benchmark our approach on a set of challenging rparse environments with a complex, simulated humanoid, and on oflline RL benchmarks for navigation and object manipulation
1
Introduction
Recent results have demonstrated that reinforcement learning (RL) agents capable of achieving high performance in variety of well-defined environments (Silver et al. 2016 Vinyals et al. 2019 OpenAT et al,,
2019 Badia et al.) 2020). A key requirement of these systems is the collection of billions of interactions via simulation, which presents a major hurdle for their application to real-world domains. Therefore, it is of high interest to develop methods that can harness pre-trained behaviors or demonstrations to speed up learning on new tasks via effective generalization and exploration.
In this work, set out to leverage demonstration data. by combining the acquisition of low-level motor skills and the construction of generative sequence models of state trajectories. In our framework, low-level skill policies reduce the complexity of downstream control problems via action space transformations, while state-level sequence models capture the temporal structure of the provided demonstrations to generate distributions over plausible future states. We ground both components in a single latent space so that sequence model outputs can be translated to environment actions by the low-level policy. Thus, our sequence models constitute useful priors when learning new high-level policies.
Videos, source code and pre-trained models available at https://aachookkeeacchhhhihhh htgp://eceecccccct.iiiieeeeeeeee

---

Published in Transactions on Machine Learning Research (03/2023)
Demonstrations
Representation
Latent Space Prior
Transfer Tasks
Thi (2/
No (2t #1, 1e-1)
p(2/2r)
To To (a/ss,,
x
Explore
2 ~~ TT(2)
q(= 12)
Th (aaaa
Regularize
Tm((2|s) (z) |(T(())
Options
c~ Tc((ss) z~ x~
Low-Level Policy
Our approach consists of a pre-training phase (left), followed by high-level policy learning on transfer tasks. For pre-training, we embed demonstration trajectories E X into latent representations with an auto-encoder. We separately learn a prior To that models latent space trajectorics, as well as a low-level policy Rio trained to reenact demonstrations from proprioceptive observations & a near-term targets z. On transfer tasks, we train high-level policy Tm(2]ss and utilize the latent space prior To to accelerate learning.
Figure 1:
Our latent space priors empower RL agents to address several key challenges for efficient learning. Directed exploration helps to discover state regions with high rewards and is achieved with augmented exploration policy that samples latent state sequences from the prior. Similarly, they offer additional learning signals by regularizing policies, with the prior distribution encoding behavior that is correspondence with the provided demonstrations. Finally, latent state priors generate temporally extended behaviors, providing flexible temporal abstractions in the form of length options, without relying on fixed temporal partitioning of the training data.
Figure summarizes our approach, which we showcase on continuous control tasks with complex humanoid robot. Our demonstrations are publicly available motion capture sequences that comprise observations of general locomotion skills. First, we learn a latent space demonstration states by auto-encoding trajectorics. Latent space priors and low-level policies are then acquired separately. The prior models demonstrations as sequences of latent states and implemented as an auto-regressive Transformer (Vaswani el. al. 2017). The low-level policy trained reenact the provided demonstration clips in a simulated environment (Hasenclever et al. 2020 Won et al. 2021) where latent states constitute short-term goals for conditioning. On transfer lasks, we train new hevel policies from scratch utilizing pre-trained components, i.e., latent state priors and low-level skill policies. We investigate several methods of integration: (1) augmenting the exploration policy with sequences sampled from the prior; (2) regularizing the policy towards distributions predicted by the prior; (3) conditionally generating sequences from high-level actions actions to provide temporal abstraction.
We evaluate our approach on a set of sparso-reward tasks that pose varying challenges with respect exploration, long-term reasoning and skill transfer. We find that latent space priors provide benefits across all these axes and lead improved learning speed and final performance. Depending on the concrete task at hand, the optimal way of utilizing latent space priors differs: temporal abstraction boosts exploration capabilitios, while augmenting exploration and regularizing the high-level policy towards demonstration behavior accelerates learning and yields high final performance on tasks demanding quick reactions.
2
Latent Space Priors
2.1
Setting
We consider a reinforcement learning agent that seeks to maximize its cumulative reward by acting in a given task modelled as a Markov docision process AS (Sutton and Barto 2018). Our approach is situated in a hierarchical reinforcement learning framework, where high-level policy Tui S + Z steers a fixed, pre-trained low-level policy Tio : x Z A via actions from latent space E Z (Dayan and Hinton 1992.
2

---

Published in Transactions on Machine Learning Research (03/2023)
Vezhnevets et al. 2017). We now define latent space prior generative, auto-regressive model in the high-level action space 2, Lee, mo(=a|s,...., (t-)) Intuitively, a latent space prior models plausible high-level behavior, and we aim to show-case how it can be employed to improve high-level policy learning on new tasks,
A crucial insight is that both the low-level policy Tho as well as the latent space prior To can be acquired from demonstrations. Accordingly, our exposition targets a setting in which we are provided with a dataset X consisting of observation trajectories æ = (I1,,..,,Tn). The dataset is assumed to be unlabeled (i.e., not annotated with rewards) and unstructured (i.e., trajectories may include several tasks skills, which may further be unrelated to our main tasks of interest). We structure our training setup such that latent states encode successive demonstration states. In effect, the latent space prior serves as a generative model of demonstration trajectories, while the low-level policy is conditioned on encoded near-term demonstration states.
We implement our method for continuous control of a simulated humanoid robot and leverage motion capture demonstrations. While we do not require an exact match between the paramcterization of demonstrations and (ask states, we assume a correspondence between demonstration state and proprioceptive states s" = In our experimental setup, proprioceptive states s correspond to local sensor readings of a simulated humanoid robot, and transfer tasks of interest may provide additional state information, e.g., the global position of the robot, goal states extra sensor readings. We provide a mapping of X to S which we describe in Appendix A In the remainder of this section, we will detail the acquisition of a prior To and a corresponding low-levelpoolicy To from demonstrations.
2.2 Modeling Demonstration Trajectories
We learn a generative model for demonstration trajectorics in a two-step process, inspired by recent work in image and audio generation (vvn den Oord el al. 2017 Dhariwal et al. 2020). First, we learn a latent space Z of demonstration observations x via auto-encoding. We then learn a separate, auto-regressive model (the latent space prior) of latent state sequences.
Demonstration trajectories are embedded with a variational aulo-encoder (VAE) (Kingma and Welling 2014h, optimizing for both reconstruction accuracy and latent space disentanglement. Both encoder and decoder consist of one-dinensional convolutional layers such that each latent state covers a short window of neighboring states. Latent states thus capture short-range dynamics of the demonstration data. This eases the sequence modeling task, but is also beneficial to endow the low-level policy mo(a|s, 2) with accurate near-term dynamics (2.3). The training loss of a demonstration sub-trajectory x (I:- Ii+k) of length 2k + 1 is the evidence lower bound (ELBO),
CAE(E, o. 0) =- E [o (x) + 3Dkl. (go()|z) || p(z)),
(=(=))
where qe is the encoder (parameterized by Φ), (x(x) the decoder and p(z) a prior, which we fix to an isotropic Gaussian distribution with unit variance.
VAËs are typically trained with a L2 reconstruction loss, corresponding to the ELBO when assuming Pa to be Gaussian. Domain-specific reconstruction losses may however significantly enhance sample quality (Dhariwal et al. 2020 Li et al. 2021 For our application, we employ forward kinematic layer (Villegas et. al.) 2018) to compute global joint locations from local joint angles in z. Global joint locations, position and orientation of reconstruction and reference are then scored with a custom loss function (Appendix C.I
A After training the VAE on the demonstration corpus, we map each trajectory x = (I1,..., .x)) to a latent space trajectory z with the encoder qo- We then model trajectorics in latent space with an auto-rogressive prior, i.c.,
p(2) II
no 2x , zt-1)
We implement latent space priors as Transformers (Vaswani 2017), with the output layer predicting
mixture distribution of Gaussians (Bishop) 1994 Graves 2013). As an alternative to employing dropout
2Qur low-level policy operates on proproceptive states SP S Marino al. 2019
3

---

Published in Transactions on Machine Learning Research (03/2023)
in embedding layers for regularization, we found that the variability and quality of the generated trajectories, transformed into poses with the VAE decoder, is significantly increased by sampling past actions from (o(*(x) while predicting the mean of the latent distribution via teacher forcing. Training is further stabilized with an entropy term. The full loss for training the prior with neural network weights sb is
( (2,0)=
log To (2) + eHd (TO,w 21
(2)
2.3 Low-Level Policy Learning
We train low-level skill policies to re-enact trajectorics from the demonstrations, presented as latent state sequences {=}. If the dataset includes actions, behavior cloning or offline RL techniques may be employed to reduce computational requirements (Lange et al., 2012 Merel et al. 2019b Levine et al. 2020; Wagener
ct al. 2022). We apply our method to such a setting Section 4.3 In our main experiments, demonstrations consist of observations only; we therefore employ example-guided learning, relying on an informative reward
function for scoring encountered states against their respective demonst counterparts (Peng al. 2018
Won et al. 2020 Hasenclever et al. 2020).
Our training setup is closely related to the pre-training procedure in CoMic Hasenclever et al. 2020 A semantically rich per-step reward measures the similarity between the current state 8t and the respective reference state It, which we ground in the simulation environment as detailed in Appendix A Previous works on skill learning from motion capture data condition the policy on several next demonstration frames, featurized relative the current stale (Hasenclever et al., 2020 Won et al.] 2020 In contrast, our skill policy is conditioned a single latent state, which models near-term dynamics around given time step in the demonstration trajectory (2.2). Recalling that each VAE latent state is computed from window of 2k + neighboring states, we provide the latent state 2 to the skill policy at time step t. We use V-MPO (Song
et al. 2019), an on-policy RL. algorithm, to learn our low-level policy.
3
Accelerated Learning on Transfer Tasks
On transfer tasks, our main objective is to learn a high-level policy Thi efficiently, i.e., without requiring exceedingly many environment interactions and achieving high final returns. We employ Soft Actor-Critic (SAC), a sample-efficient off-policy RL algorithm (Haarnoja et al., 2018). While our pre-trained low-level policy capable of re-enacting trajectories from the demonstration data, the high-level policy still has to explore the latent space in order to determine suitable action sequences for given task. Latent space priors assist high-level policy learning by providing flexible access to distributions over trajectories in the policy's action space Z. We will now detail several mcans to integrate a prior No to(2)|z1,.. 21-1) into the training process of (=)()))))) (Figure 2 which we evaluate on several sparse-reward transfer tasks in Section
3.1 Exploration
As our prior generates latent state sequences that, together with the the low-level policy, result in behavior corresponding to the provided demonstration, it can be directly utilized as an exploration policy. This augments standard exploration schemes continuous control, where actions are obtained via additive random noise or by sampling from a probabilistic policy. While acting according to mo exhibits temporal consistency, the resulting behavior is independent of the specific RL task at hand as it does not take ground-truth simulation states or rewards into account. Consequently, Aa is most useful in the carly stages of training to increasing coverage of the state space. We thus augment exploration in an e-greedy manner and anneal towards 0 during training.
We further correlate exploration by following the actions of either Thi or mo for several steps. A low c naturally results in extended periods of acting with the high-level policy; when exploring with the latent space prior, sample the number of steps i to follow the generated sequence from Poisson distribution, where k's
4

---

Published in Transactions on Machine Learning Research (03/2023)
Flat Policy
Hierarchical Policy
Environment
Policy
Tho
Fixed Policy
a(),a)
Latent Space Prior
Exploration
Regularization
Options
Thi
D
Thi
ma
Ma
a(,,a)
Figure 2: Top: graphical models of learning policy (blue) without and with a pre-trained low-level policy (yellow). For clarity, we omit the low-level policy's dependency on the environment state. Bottom: comparison of methods to accelerate learning high-level policy Thi (blue) with latent state prior mo (green): augmenting exploration with actions from the prior (left); regularizing Thi towards behavior encapsulated by the prior (center); performing temporally extended actions with zo (right).
expectation is defined by lyper-parameter Formally, the augmented exploration policy is defined
( To(22 (2ice) with probability for k ~ Pois(A) steps
with probability -
CNT hi (2(18))
3.2 Regularization
Reinforcement lcarning algorithms commonly employ inductive biases in the form of regularization terms in policy or value function oplimization, with entropy maximization as a popular example (Ziebart et al. 2008 Ahmed et al.) 2019). Tf prior knowledge available the form of a policy, it can form an additional bias, with a modified high-level policy learning objective given as
Σ (e [r(st, az) - 8D (Tmi(-) (8)), no =o(-(zi)))]]
max WhE
Here, is provided by the deterministic low-level policy E. (als, =)], D is suitable divergence function (e.g., the KL divergence), and the scalar 8 trades off reward maximization and prior regularization (Tirumala) et al. 2020 Pertsch et al.) 2020). We extend the SAC loss for policy optimization (Haarnoja et al. 2018 Eq. 7) as follows for network parameters Ps with B denoting the replay buffer and the temperature coefficient. for entropy regularization:
Jana (p)
lo (2el)))) + 5log (=]((=)) - Qua(se, +
s
(-lec)
Our formulation follows from several considerations. As T0(2(124<1) is a mixture distribution plausible future latent states, we aim for matching its support rather than its mode and hence opt for the forward KL divergence. We further leave out the normalization term log (o(2 Rice), which we found to not yield significant benefits. In line with Tirumala et al. (2020) but in contrast. to Pertsch et al. (2020), we retain SAC's entropy regularization objective as we found that it provides significant benefits for exploration.
Similar to using latent space priors as an exploration policy in 3.1 their independence of task-specific rewards renders them most useful in the early stages of training when significant task reward signals lacking.
5

---

Published in Transactions on Machine Learning Research (03/2023)
Hence, likewise amneal å towards 0 training progresses. Depending on the concrete setting, it may however be worthwhile to restrict behavior to demonstrations over the whole course of training, e-g., to avoid potentially destructive actions or to obtain behavior that matches the demonstration dataset closely.
3.3 Options
Hicrarchical decomposition of scarch problems is a powerful concept in planning algorithms (Sacerdoti 1974 and has found numerous applications in reinforcement learning @Dayan and Hinton 1992 Sutton et al, 1999.. The utility of such decomposition rests on problem-specific abstractions over some or all of the agent's 's state and action space, and on the temporal horizon of decision-making In our framework, the joint latent space of Ty and Too constitutes an action space abstraction. Latent space priors further offer temporally extended actions, or options (Sutton et. al.) 1999), via conditional aulo-regressive generation of latent state sequences. Consequently, the high-level policy now provides the conditional input to the prior (Fig. 2 right). We investigate a basic form of conditioning by predicting initial latent state &n from which the the prior then predicts the following 2(+1...., 2++k-1 high-level actions for an option length of k steps.
For learning Thi with temporally extended actions, we modify the SAC objectives with the stop-conditioned critic proposed by Whitney et al. (2020). This yields the following objectives, with 0 S < denoting the number of environment steps from the previous high-level action:
27
h-7-1
(Q)( (t.s t)wB
Quip(se, 20-,, - Σ
(Pr(s(+s,ats))) 1 (V((666)))))
2
3=0
(or log (z),0(2)(0)) - Qhi(se: 24, 20,
(p)
E
E Th
B
3.4 Planning
If demonstrations are closely related to the task of interest, latent space priors may be used for planning in order to act without learning high-level policy. Concretely, the learned auto-encoder can provide a latent representation of the trajectory to initialize new high-level action sequences. Sampled candidate sequences from To can be decoded again into states, which are then ranked by a given reward function. We demonstrate successful planning for a simple navigation task in Appendix F However, the main focus our study are transfer seltings where task dynamics may differ from demonsttrations, e.g., due to additional objects, and demonstrations are obtained from real-world data.
We note that the described methods for integrating latent space priors into downstream policy learning are complementary so that, e.g., To can assist in exploration while also providing regularization ion objective, or sampled latent space sequences can be filtered based on their utility if the reward function is known. In our experiments, we evaluate cach strategy in isolation to highlight its effects on learning performance.
4 Experiments
We evaluate the efficacy our proposed latent space priors on a set of transfer tasks, tailored to a simulated humanoid robot offering a 56-dimensional action space (Hasenclever et al. 2020). Our demonstration dataset. contains 20 minutes of motion capture data of various locomotion behaviors such as walking, turning, running and jumping (Appendix Aj. We consider four sparse-reward tasks that pose individual challenges with respect.
to locomotion, exploration and planning abilities (Fig. 3 additional details in Appendices B and Gl. We investigate the effect of latent space priors by comparing against bascline which also uses our lowelevel policy but learns a high-level policy without extra support. We further compare against low-level policy acquired with CoMic (Hasenclever et al. 2020), in which the high-level action space is learned jointly with the low-level policy.
The utilization of latent space priors for high-level policy learning introduces several hyper- exper-parameters. For exploration (3.1) we set À to 5, and is annealed from 0.1 to over w=10M samples; for regularization

---

Published in Transactions on Machine Learning Research (03/2023)
Figure 3: Trained agents acting in transfer tasks. From left right: GoTo Targetss concerns locomotion along randomly sampled waypoints; in Gaps, the character is not allowed to touch the area between platforms; in Butterflies, floating spheres have to be caught with a light net; Stairs provides series of ascending and descending stair-cases.
we initialize d with 0.01 and annealing zero is performed over w=1M samples. The context (c) of previous high-level actions ži<t provided to ( kiet) is selected from {1, 4, 8} for both exploration and regularization. When integrating priors options (..) evaluate option lengths of 2, 4 and 6 steps. Full training details are provided in Appendix
3.2
4.1 Pre- Pre-Training
Our pre-training stage yields several components: latent space encoder and decoder, the latent space prior as well as low-level policy. We first evaluate our pre-trained skill policy against the low-level policy described in CoMic (Hasenclever et al. 2020).
Training Reward
0.8
Figure compares the training reward achieved by both CoMic and our policy (z-prior). Learning proceeds slower in our setup, although both policies reach comparable performance at the end of training. We note that while we keep V-MPO iyper-parameters fixed across the z-prior and CoMic pre-training runs, the respective policy designs differ siguificantly. In CoMic, the high-level action space encodes both proprioceptive state and reference poses relative to the agent's 's state, while our policy provided with VAE latent states and has learn any relations to simulation states and rewards from experience.
0.4
0.2
0.6
Samples
1e10
Low-Level
2 z-priar
Figure 4: Training Reward during example-guided pre-training for our ap- proach (z-prior) and CoMic. Both poli- cies reach comparable final rewards.
Next, investigate the interplay between the latent space prior To and the low-level policy, which were trained independently but are grounded in joint latent space. In Figure 6 we visualize different continuations sampled from the prior (colored) starting from a reference demonstration trajectory (silver). Qualitatively, we observe varied trajectories that frequently but not always resemble biologically plausible behavior; however, our VAE not explicitly tuned towards realistic kinematic control. In many but not all cases, reenactment by the policy succeeds. For quantitative analysis, we sample 100 latent state sequences of 10 seconds each from the prior. We then measure the average per-frame reward between reenacted and decoded states. The average per-frame reward the original demonstration dataset is 0.89; sampled trajectories, we observe drop to 0.78.
4.2 Transfer Tasks
We plot learning curves of training runs on transfer tasks in Figure 5 Our results demonstrate faster lcarning and increased final performance when integrating latent space priors. In terms of general performance, exploration with the prior (z-prior Explore) provides consistent benefits; however, the nature of the task dictates the most useful integration method. For regularization, we find accelerated learning on Gaps and Butterflies, On Targees, where observations allow the agent to plan ahcad, utilizing the prior as an option policy results in fast learning. On Stairs, the prior helps a larger fraction of agents discover the second
7

---

Published in Transactions on Machine Learning Research (03/2023)
GoToTargets
Gaps
Butterflies
Stairs
10
40
15
Return
30
10
20
10
1
2
3
1
2
3
1
2
3
4
5
2
3
4
5
Samples
1e7
Samples
1e7
Samples
1e7
Samples
1e7
z-prior Explore
z-prior Options
No Prior
z-prior Regularize
CoMic Low-level Policy
Figure Learning curves on transfer tasks. Latent space priors (x-prior) accelerate learning and increase final performance, with the best method of integration depending on task characteristics. On GoToTargets, temporally extended options provide quick exploration; lead to faster exploration, with v-prior Explore and Regularize achieving the highest urts. On Stairs, utilizing the prior as an option policy leads to best mean performance. The comparison to CoMic validates our low-level policy (red). Results averaged over 4 runs, except for Stairs (10 runs).
Gaps and Butterflics, all variants with priors
staircase leading down from the platform, which results in a return above 10. Generally, forming options with To restricts the actions that the high-level policy can perform, and, while useful for exploration, stifles final performance the Gaps and Butterflies lasks.
Compared to CoMic, our low-level policy achieves superior performance on all tasks, even though CoMic often allows for faster learning carly in training, c.g., GoToTargets. We perform further comparisons of both low-level policies in Appendix [ To highlight the dilficulty of our tasks, also trained non- hierarchical policies from scratch with SAC and found that they did not make meaningful learning progress (Appendix H. Recently, Wagener et al. (0022) released an offline dataset for the CMU motion capture database. In experiments detailed in Appendix H.3 we observe that, with a comparable offlnne-earmedd policy, their high-level PPO agent fails on our benchmark tasks.
Figure 6: Sequences sampled from the latent space prior (colored), starting from a demonstration trajectory (silver) and decoded with the VAE. Top: the short jumping motion completed, and sampled trajectories continue walking and rumning with changes in dircction (orange and green) or include jumps from the training data (mauve). Bottom: the sampled trajectories include continuing the running motion (yellow) changing walking speed after turn (mauve).

---

Published in Transactions on Machine Learning Research (03/2023)
1.5
z-prior Explore
Figure Variations of hyper- parameters when integrating priors. We plot the mean return achieved over the full course of training relative to learning without prior, i.e. values larger above 1 signify improvements from latent space priors. Best configurations depend on the nature of the task at hand, with ==1 and k=2 providing good overall performance.
1.0
c-1
c=
0.5
c
0.0
1.5
z-prior Regularize c=1
Return
1.0
c
z-prior Options
1.0
k-2
k-4
0.5
k-6
0.0
Targess
Caps
Butterfiles
Stairs
Task
Overall, we find that optimal typer-parameter such as option length or the number of past actions provided to the prior also depend on the concrete task at hand. For Figure 5 we select fixed hyper-parameters for all methods (explore c (conlext) =1; regularize e=1; options k=2) and report performance for other values in Figure 7 The results indicate that providing larger context of actions for exploration or regularization is helpful in specific scenarios only (e-g., 2-prior Regularize with c=4 on Stairs). Increasing option lengths (z-prior Options) is also not effective whereas initial learning can be accelerated on GoToTTarger and Butterflies. We refer to Appendix E for full learning curves of these ablations as well as further discussion. In Appendix we estimate that good policies on our tasks act markedly different compared to the demonstration data, which supports the finding that providing larger context to our prior generally not helpful.
downstream tasks when focusing achieving high eventual performance,
C
4.3 Offline Datasets
We evaluate latent space priors on two offline RL datasets: maze navigation with a quadruped "Ant" robot and object manipulation with a kitchen robot (Fu et al., 2021). We train low-lcvel policics and latent space priors on offline data, and then learn a high-level policy online in the downstream task. We obtained best results by training a state encoder jointly with the low-level policy, rather than learning a VAE isolation (C.6). For AntMaze, we follow the experimental setup from OPAL (Ajay et al.) 2021) and use their online RL variant baseline; for the kitchen robot, we compare to SPiRL (Pertsch et al. 2020). Both OPAL and SPiRL acquire latent space conditioned, temporally extended low-level primitives with a statc-conditioned prior To(2))) to regularize high-level policy learning.
given offline dataset, along
Results in Figure X show that latent space priors are effective in both domains. In the maze environment (Fig. S left), all forms of prior integration solve the task quickly, with options (we use k=10) yielding best results. Learning high-level policy without a prior (no prior) requires about 3M samples to start reaching
AntMaze Medium Dense
AntMaze Large Dense
Kitchen Mixed
1.0
z-prior Explore
Completed Subgoais
z-prior Regularize
0.8
0.8
2-prior Options
0.6
0.6
No prior
2
SPIRL
0.4
0.4
OPAL
0.2
0.2
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0
2.5
0.5 0.0
1.0 1.5 Samples
2.0
2.5 3,0
Samples
1e6
Samples
1e6
le6
Figure 8: Results for pre-training lowelevel policies and latent space priors on offline datasets. OPAL results are provided by Ajay et al. (2021) at 2.5M samples for the medium and 17.5 samples for the large maze, respectively.
9

---

Published in Transactions on Machine Learning Research (03/2023)
the goal position (not shown). We were not able to tackle the sparse-reward variant of this task successfully (OPAT, achieves a success rate of 81.6% for the medium maze but fails on the large maze). We hypothesize that this is due to our priors not being conditioned on the current environment state; on the other hand, this allows us to casily transfer the pro-trained components to the large maze (Fig. g center). In the kitchen task, we obtain a higher average return than SPiRL with the option integration (k=10). Both exploration and regularization with latent space priors fail to significantly accelerate learning in this environment.
5 Related Work
Our work rosides in long line of research concerned with utilizing demonstration data to assist reinforcement learning agents, which, in their basic setting, attempt to solve each new task by training policy from scratch via. trial and error (an overview is provided by Ravichandar et al. (2020). among others). While demonstration data can be used to acquire policies, models, value or reward functions directly (Pomerleau
1988 Atkeson and Schaal 1997 Ng and Russell 2000 Hester et al. 2018). we seek to acquire structures that accelerate learning in unseen transfer lasks. Further, an important distinction is the type of ratoon that informed the design of given method. In the batch or offline RL setting in particular, the dataset. contains state-action trajectories and a corresponding reward function is assumed to be available (Lange et al. 2012 Ajay et al., 2021 Shiarlis et al., 2018 Levine et al. 2020). As demonstrated in Section 4.3 our approach is compatible with this setting, but our main focus is on demonstrations consisting of observations only, with the assumption that simulated environments are available for low-level policy learning.
Skill Learning from Demonstrations A paradigm that lends itself particularly well to transfer settings is the discovery of low-level behavior, or skills, to provide reusable abstractions over state or action spaces Thrun and Schwartz 1995 Sutton et al.
sequences (Dayan and Hinton 1992
1999 Eysenbach et al. 2019
Hasenclever et al.J 2020: Gehring et al.
2021
Discovering skills from demonst
has been the focus of 2019). Among recent works 2012 Paul et al.
large body of work (Konidaris et al.
2012
Niekum et al.
utilizing deep neural networks as policies, Pertsch et al. (2020) and Ajay et al.
(021)) propose to segment
oflline data into fixed-length sub-trajectories and obtain skills by auto-encoding action and state-action
sequences, respectively, to latent states and back to actions. Singh et al. (222)) propose to learn an invertible action space transformation from action sequences which then forms a low-level policy, We refrain from introducing further inductive biases regarding the segmentation of demonstrations; instead, we model whole trajectories with a generative sequence model and train single-step skill policy. Outside of the context of demonstrations, our method bears similarity with Co-Reyes et al. (2018) who also consider latent sequence generation models and subsequent execution via low-level policies. In their work, learning is performed directly on downstream tasks, and the high-level policy is implemented as model-based planner. Evaluation is however limited to environments with comparably simple dynamics, and lcarning policies from scratch poses severe challenges in our lasks. Recently, Tiiumala el al. (2020) formalized the notion of behavior priors, i.e., policies that capture behavior by regularization with a divergence term, and Siegel et al. (2020) addresses learning them from demonstrations presented offline data.
Generative Sequence Models Our latent space priors are inspired by recent successes of generative sequence models, in particular transformers, in domains such as natural language, music or speech (Brown
ctt al. 2020 Dhariwal et al., 2020 Tan et al. 2022). ). Investigations into utilizing transformers for docision-
making have largely focused on the offline setting, where actions are available as ground-trut tarth targets for
generation CChen et al.] 2021 Janner et al. 2021 Reed et al. 2022). Recent studies also investigate online
fine-tuning Zheng et al. 2022) and model-baser RL (Micheli et al. 2022). With respect to improving exploration, Zhang et al. (2022) employ recurrent networks generative planners integrated into model-fres policy learning. Their planning models take states and task rewards into account and are hence tailored to specific downstream tasks, whercas we develop gencral-purpose priors for a specific agent modality.
Humanoid Control from Motion Capture Data We evaluate our method on a simulated humanoid and motion capture data demonstrations, a setting that is also subject of a separate stream of rescarch concerned with synthesizing natural motions of avatars for kinematic and physics- based control (Peng et al., 2018 Ling et al. 2020). While our latent. space prior provides opportunities for kinematic control, we focus on
10

---

Published in Transactions on Machine Learning Research (03/2023)
transfer performance in physics-based, sparse-reward tasks. Deep RL techniques have been shown to perform successful imitation learning of motion capture demonstrations via example-guided learning (Peng et al., 2018; Won et al.) 2020 Wang et al. 2020), and several works investigate low-level skill learning and transfer in this context (Hasenclever et al., 2020; Won et al., 2020: Liu et al.) 2021). Our pre-training method closely resembles CoMic (Hasenclever et al. 20200, which learns a latent high-level action space and a low-level policy an end-to-end fashion; crucially, we separate these steps and further leverage the temporal structure in demonstrations with latent space priors. Won et al. (2021) target multi-agent downstream tasks, as do Liu et al. (2021), where skills learned from motion capture data serve priors for policies trained from scratch in the enviromment's s native action spaces. We regard these works as complementary to our approach and focus on transfer settings to single-agent downstream tasks by training individual high-level policies. With regards to priors for high-level learning in Humanoid control, Peng et. al. (2021) employ learned discriminator to achieve behavior of a specific style. [ohez et al. (2022) consider transfer of policies to real robots and extend CoMic with an auto-regressive prior on downstream tasks. They show that correctly scalod
auto-regressive regularization improves energy use and smoothness during locomotion.
6 Conclusion
We introduced a new framework for leveraging demonstration data in reinforcement learning that separates low-level skill acquisition and trajectory modeling. Our proposed latent space priors provide various means to accelerate learning in downstream tasks, and we verified their efficacy in challenging sparse-reward environments. Our experimental results also indicate that, among the methods for utilizing latent space priors that we investigated, the nature of downstream tasks determines their ultimate utility, such as quick exploration or increased final performance. Such tradc-offs appear naturally in RL applications, and we regard the interplay of prior utilization and high-level policy learning as a promising avenue for future work.
Our approach is aimed at settings that benefit from the availability of proprioceptive motor skills, which is a common case in robotics applications. The flexibility of latent space priors makes extensions towards classes of problems in which broader repertoires of skills are required, e.g., object manipulation, attractive subjects of further study.
Acknowledgements:
We thank Josh Merel and Alessandro Lazaric for insightful and helpful discussions.
11

---

Published in Transactions on Machine Learning Research (03/2023)
References
z. Ahmed, N. L. Roux, M. Norouzi, and D. Schuurmans. Understanding the Impact of Entropy Policy Optimization. In International Conference on Machine Learning, pages 151-160. PMLR, 2019.
A. Ajay, A. Kumar, P. Agrawal, S. Levine, and O. Nachum. OPAL: Offiine Primitive Discovery for Accelcrating Offiin Reinforcement Learning. In International Conference on Learning Representations, 2021.
C. G. Atkeson and S. Schaal. Robot Learning From Demonstration. In International Conference on Machine Learning, pages 12-20 20, 1997.
J. I.. Ba, J. R. Kiros, and G. E. Hinton. Layer Normalization. arXiv:1607006400 fes, slatj, Jul 2016.
A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, D. Guo, and C. Blundell. Agent57: Outperforming the Atari Human Benchmark. In International Conference on Machine Learning, pages 507 517, 2020.
C. M. Bishop. Mixture Density Networks. Technical Report NCRG/4288, Aston University, Birmingham, UK, 1994.
S. Bohez, Tunyasuvunakool, P. Brakel, F. Sadeghi, L. Hasenclever, Y. Tassa, E. Parisotto, J. Humplik, T. Haarnoja, R. Hafner, M. Wulfmeeer, M. Neunert, B. Moran, N. Siegel, A. Huber, F. Romano, N. Batchelor, F. Casarini, J. Merel, R. Hadsell, and N. Heess. Imitate and Repurpose: Learning Reusable Robot Movement. Skills From Human and Animal Behaviors. arXiv:2203.117 17138 [cs], Mar 2022.
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Hossert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, 1. Sutskever, and D. Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abboel, A. Srinivas, and Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. arXin:2106 006:01066 [es], Jun 2021.
J. Co-Reyes, Y. Liu, A. Gupta, B. Eysenbach, P. Abbeel, and S. Levine. Self-Consistent Trajectory Autoen- coder: Hicrarchical Reinforcement Learning with Trajectory Embeddings. In International Conference Machine Learning, pages 1009 1018, 2018.
P. Dayan and G. E. Hinton. Feudal Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 5, pages 271 278. Morgan-Kaufman, 1992.
P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever. Jukebox: A Generative Model for Music. (rXx::2005.0032 fcs, eess, statf, Apr 2020.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is All You Need: Learning Skills without Reward Function. In International Conference Learning Representations, 2019.
J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4RL: Datasets for Deep Data- Driven Reinforcement Learning. arXiv:2004.072 072211 fes, stat], Feb 2021.
J. Gehring, G. Synnaeve, A. Krauso, and N. Usunier. Hicrarchical Skills for Efficient Exploration. In Advances in Neural Information Processing Systems, volume 34, pages 11553-11564. 2021.
D. Gopinath and Won. Fairmotion Tools to load, process and visualize motion capture data. Github, 2020. URL
A. Graves. Generaling Sequences With Recurrent Neural Networks. ar arXiv:1308.055 [es], Aug 2013.
T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine. Soft Actor- Critii Algorithms and Applications. 18X2: [es], Dec 2018.
12

---

Published in Transactions on Machine Learning Research (03/2023)
L. Hasenclever, F. Pardo, R. Hadsell, N. Heess, and J. Merel. CoMic: Complementary Task Learning & Mimicry for Reusable Skills. In International Conference on Machine Learning, 2020.
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, Dumoc-Armold, Osband, J. Agapiou, J. Z. Leibo, and A. Gruslys. Deep Q-learning from Demonstrations. In AAAI Conference on Artificial Intelligence, pages 2223-3230, 2018.
S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 98) 1997.
M. Janner, Qu Li, and S. Levine. Reinforcement Learning as One Big Sequence Modeling Problem. In Advances in Neural Information Processing Systems, volume 34, pages 2273-1286. 2021.
D. P. Kingma and Ba. Adam: Method for Stochastic Optimization. In International Conference on Learning Representations, 2015.
D. P. Kingma and M. Welling, Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014.
G. Konidaris, S. Kuindersma, R. Grupen, and A. Barto. Robot learning from demonstration by constructing skill trees. The International Journal of Robolics Research, (3):3660 375, 2012.
S. Lange, T. Gabel, and M. Riedmiller. Batch Reinforcement Learning. In Reinforcement Learning, volume 12, pages 45 73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. (tfe:2005.01644 fcs, stat], Nov 2020.
J. Li, R. Villegas, D. Ceylan, J. Yang, Z. Kuang, H. Li, and Y. Zhao. Task- Generie Hierarchical Human Motion Prior using VAEs. In International Conference on 3D Vision, pages 771-781, 2021.
H. Y. Ling, F. Zinno, G. Cheng, and M. Van De Panne. Character controllers using motion VARs. ACM Transactions on Graphics, 39(4), 2020.
S. Liu, G. Lever, Z. Wang, J. Merel, S. M. A. Eslami, D. Hennes, W. M. Czarnecki, Y. Tassa, S. Omidshafiei, Abdolmaleki, N. Y. Siegel, L. Hasenclever, L. Marris, S. Tunyasuvunakoo,, H. F. Song, M. Wulfmeier, P. Muller, T. Haarnoja, B. D. Tracey, K. Tuyls, T. Graepel, and N. Heess. From Motor Control to Team Play in Simulated Humanoid Football. ar arXin:2105.12100 fes], May 2021.
M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi-person lincar model. ACM Trunsactions on Graphics, 34(6):248:1 248:16, 2015.
Mahmood, Ghorbani, F. Troje, G. Pons-Moll, and M. Black. AMASS: Archive of Motion Capture Surface Shapes. In International Conference on Computer Vision, pages 44400 2019.
K. Marino, A. Gupta, R. Fergus, and A. Szlam. Hierarchical RL Using an Ensemble of Proprioceptive Periodic Policies. In International Conference on Learning Representations, 2019.
J. Merel, A. Ahuja, V. Pham, S. Tunyasuvuakkool, S. Liu, D. Tirumala, N. Heess, and G. Wayne. Hicrarchical Visuomotor Control Humanoids. In International Conference on Learning Represenlations, 2019a.
J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V. Pham, G. Wayne, Y. W. Teh, and N. Heess. Neural Prob- abilistic Motor Primitives for Humanoid Control. In International Conference Learning Representations, 2019b.
V. Micheli, E. Alonso, and F. Fleuret. Transformers are Sample Efficient World Models. arXiv:2209.0058. [es], Sep 2022.
A. Y. Ng and Russell. Algorithms for Inverse Reinforcement Learning. In International Conference on Machine Learning, 2000.
13

---

Published in Transactions on Machine Learning Research (03/2023)
S. Nickum, S. Osentoski, G. Konidaris, and A. G. Barto. Learning and generalization of complex tasks from unstructured demonstrations. In 2012 SEE//RJ International Conference on Intelligent Robols and Systems, pages 5239 5246, Vilamoura, Algarve, Portugal, 2012.
OpenAT, C. Berner, G. Brockman, B. Chan, V. Cheung, P. Debiak, C. Dennison, D. Farhi, Q Fischer, S. Hashme, C. Hesse, R. Józefowicz, S. Gray, C. Olsson, J. Pachocki, M. Petrov, H. P. d. O. Pinto, J. Raiman, T. Salimans, J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, F. Wolski, and S. Zhang. Dota 2 with Large Scale Deep Reinforcement Learning. orXiv: 1etit991200000 fes, statj, Dec 2019.
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. PyTTorch: An Imperative Style, High-PPrformane Deep Learning Library. In A Advances in Neural Information Processing Systems, volume 32. 2019.
S. Paul, J. van Baar, and A. K. Roy-Chowdhury. Learning from Trajectories via Subgoal Discovery, In Advances in Newral Information Processing Systems, volume 32. 2019.
X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. DeepMimic: Example-Guided Deep Reinforcement Learning of Physics-Based Character Skills. ACM Transactions on Graphics, 37(4), 2018.
X. B. Peng, Z. Ma, Abbeel, S. Levine, and A. Kanazawa. AMP: Adversarial Motion Priors for Stylized Physics-Basse Based Character Control. ACM Transactions on Graphcs, 40(4), 2021.
K. Pertsch, Y. Lee, and J. J. Lim. Accelerating Reinforcement Learning with Learned Skill Priors. In Conference on Robot Learning, 2020.
D. A. Pomerleau. AIVINN: An Autonomous Land Vehicle in Neural Network. In Aduances in Newral Information Processing Systems, volume 1. Morgan-Kaufmann 1988.
H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard. Recent Advances in Robot Learning from Demonstration. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):297 330, 2020.
S. Reed, K. Zolna, E. Parisotto, G. Colmenarejo, A. Novikov, G. Barth-Maron, M. Gimenez, Y. Sulsky, J. Kay, J. T. Springenberg, T. Eccles, Bruce, A. Razavi, A. Edwards, N. Heess, Y. Chen, R. Hadsell, O. Vinyals, M. Bordbar, and N. de Freitas. A Generalist Agent. arXiv:2205.065 061155 [cs), May 2022.
E. D. Sacerdoti. Planning in a hierarchy of abstraction spaces. Artificial Intelligence, 5(2):115 135, 1974.
K. Shiarlis, M. Wulfmeier, S. Salter, S. Whiteson, and I. Posner. TACO: Learning Task Decomposition via Temporal Aligument. for Control. In International Conference on Machine Learning, 2018
N. Y. Siegel, J. T. Springenberg, F. Berkenkamp, A. Abdolmaleki, M. Neunert, T. Lampe, R. Hafner, N. Heess, and M. Riedmiller. Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning. wrXiw:2002.88396 [cs, statf, Jun 2020.
D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, T. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, Sutskever, T. Lillicrap, M. Leach, K. Kavukeuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484 489,,,4444410 2016.
A. Singh, H. Liu, G. Zhou, A. Yu, N. Rhinehart, and S. Levine. Parrot: Data-Driven Behavioral Priors for Reinforcement Learning. In International Conference Learning Representations, 2021.
S. Sinha, H. Bharadhwaj, A. Srinivas, and A. Garg. D2RL: Deep Dense Architectures in Reinforcement Learning. In NeurIPS 2020 Deep RL Workshop, 2020.
H. F. Song, A. Abdolmaleki, J. T. Springenberg, A. Clark, H. Soyer, J. W. Rae, Noury, A. Ahuja, S. Liu, D. Tirumala, N. Heess, D. Belov, M. Riedmiller, and M. M. Botvinick. V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control. In International Conference on Learning Representations, 2019.
14

---

Published in Transactions on Machine Learning Research (03/2023)
R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. Adaptive Computation and Machine Learning Series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018. ISBN 978-0-262-03922-6.
R. S. Sutton, D. Precup, and S. Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, (12(11)) 211, 1999.
X. Tan, J. Chen, H. Liu, J. Cong, C. Zhang, Y. Liu, X. Wang, Y. Leng, Y. Yi, L. He, F. Soong, T. Qin, S. Zhao, and T.Y. Liu. NaturalSpeech: End-to-End Text to Speech Synthesis with Homan--evve Quality. ar 2rXiv:2205.044 04421 fes, eess), May 2022.
Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, P. Trochim, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, and N. Heess. Dm iin_contooll Software and Tasks for Contimuous Control. arXi::2006.190 12983, 6, Nov 2020.
S. Thrun and A. Schwartz. Finding Structure in Reinforcement Learning. In Advances in Neural Information Processing Systems, volume 7, pages 385-392 MIT Press, 1995.
D. Tirumala, A. Galashov, H. Noh, L. Hasenclever, R. Pascanu, J. Schwarz, G. Desjardins, W. M. Czarnecki, A. Ahuja, Y. W. Teh, and N. Heess. Behavior Priors for Efficient Reinforcement Learning, orXiv:2010.1423 [es], Oct 2020.
E. Todorov, T. Frez, and Y. Tassa. MuJoCo: A physics engine for model-based control. In 2012 TEFE/RS. RRSJ International Conference on Intelligent Robots and Systems, pages 5026 5033, 2012.
A. den Oord, O. Vinyals, and K. Kavukeuoglu. Neural Diserete Representation Learning. In Advances in Neural Information Processing Systems, volume 30, pages 6306 6315. 2017.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems, volume 30, pages 5998 6008. 2017.
A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In International Conference on Machine Learning, pages 3540-3549, 2017.
R. Villegas, J. Yang, D. Ceylan, and H. Lee. Neural Kinematic Networks for Unsupervised Motion Retargetting. Proceedings of the IEEE Conference on Computer Vision and Pottern Recognétion, pages 8639-88 8648, 2018.
O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukeuoglu, D. Hassabis, C. Apps, and D. Silver. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 375(7782):350-8555 2019.
N. Wagener, A. Kolobov, F. V. Frujeri, R. Loynd, C.-A. Cheng, and M. Hausknecht. MoCapAct: A Multi-Tash Datasel for Simulated Humanoid Control. arXio:2208. 07363 fes, eess], Aug 2022.
T. Wang, Y. Guo, M. Shugrina, and S. Fidler. UniCon: Universal Neural Controller For Physics- based Character Motion. 2rXi::2011.15119 fes), Nov 2020.
W. Whitney, R. Agarwal, K. Cho, and A. Gupta. Dynamics-aware Embeddings. In International Conference on Learning Representations, 2020.
J. Won, D. Gopinath, and J. Hodgins. A scalable approach to control diverse behaviors for physically simulated characters. A CM Transactions on Graphics, (4):333333 -33:333:1, 2020.
15

---

Published in Transactions on Machine Learning Research (03/2023)
J. Won, D. Gopinath, and J. Hodgins. Control strategies for physically simulated characters performing two-player competitive sports. ACM Transaclions on Graphics, (004):146:1-144:11 2021.
H. Zhang, S. Starke, Komura, and Saito. Mode-adaptive neural networks for quadruped motion control. ACM Transactions Graphics, 37(4):155:11145:1 2018.
H. Zhang, W. XU, and H. Yu. Generative Planning for Temporally Coordinated Exploration in Reinforcement Learning, In International Conference on Learning Representations, 2022.
Q Zheng, A. Zhang, and A. Grover. Online Decision Transformer. In International Conference on Machine Learning, volume 162, pages 27042 27059. PMLR, 2022.
B. D. Ziebart, A. Maas, J. A. Bagnell, and A. K. Dey. Maximum Entropy Inverse Reinforcement Learning.
In AAAI Conference Artificial Intelligence, pages 1483-1438, 2008.
16

---

Published in Transactions on Machine Learning Research (03/2023)
A Demonstration Data
Our demonstration dataset originates from the CMU Graphics Lab Motion Capture Database f which we access through the AMASS corpus (Mahmood et al. 2019). We select subset of the "locomotion" corpus from Hasenclever al. (2020), , containing about 20 minutes of various walking, running, jumping and turning motions. Each AMASS motion consists of 3D joint rotations and global position for each frame, well as parameters that define the body shape (Loper et al., 2015). We pre-process all motions with fairmotion (Gopinath and Won 2020) as follows. First, we resample cach motion so that its framerate matches the control frequency of the imitation and downstream task environments, i.e., 33.3Hz. Second, we determine custom SMPL shape parameters that approximate our simulated humanoid. The resulting skeleton is employed to compute forward kinematics for the representation learning loss, and to adjust the vertical position of the motion.
For representation learning 2.2 we auto-encode local joint angles as well body position and orientation. The body's X and Y positions are featurized as velocities in X and Y direction to maintain location invariance. Body orientation and local joint rotations are featurized as forward and upward vectors (Zhang et al. 2018 We do not model joint velocities.
For low-level policy training via example-guided learning, we use the MuJoCo physics simulator Tooorov et al., 2012) and the "V2020" humanoid robot from dm hon_contool (Tassa al.) 2020). We modify the robot definition slightly to ease conversion from the SMPL skeleton by adjusting the default femur positions and left shoulder orientation. SMPL poses are mapped to V2020 joint rotations with manually defined mapping: notably, the MuJoCo model uses one-dimensional hinge joints only that, e.g., for the elbow joints, two degrees of freedom are lost. We exclude head and finger joints from the conversion. Global and local joint velocitics are computed with finite differences following Merel et al. (2019a).
B Transfer Tasks
The GoToTargets lask is similar to the "go-to-larget" task in Hasenclever el al. 20200 however, the version we consider requires a larger variety of locomotion skills (fast turns and movements). Instead of selecting the first goal randomly and the following goals in the vicinity the first, we sample all goals randomly within an area of 8x8 meters. For the minimum distance obtain a reward of 1, we use 0.5m instead of 1.0m. A new goal is provided as soon the robot achieves the minimum distance, rather then switching to new goal after 10 steps as in Hasenclever et al. (2020). Hence, agents naturally achieve lower returns in our task. The agent observes the relative positions of the current and next goal.
In the Butterflies task, a light "net" is fixed to the left hand of the robot. Spheres are randomly placed within a 4x4m area, at heights ranging from 1.43m to 2.73m. The agent receives a reward of 1 if the net touches sphere. We limit the number of spheres to 10, which denotes the maximum return. Untouched spheres are encoded as observations by projecting their location onto a unit sphere placed at the robot's head location. The unit sphere is rasterized with longitudes and latitudes spanning 10 degrees cach, and cach entry is set to the exponential of the negative distance to the respective closest sphere, or to zero if no sphere is projected onto that. location.
The Gaps and Stairs tasks have been introduced by Gehring et al. 2021 ; we scale stairs, gap and platform lengths by 130% to better fit the humanoid robot we use in this study, and do not end the episode or provide negative rewards the robot falls over. For Gaps, the agent receives a reward of when the robot reaches platform; if the robot touches the lower area between two platforms, the episode ends immediately with reward of -1. In the Stairs task, reward of is obtained for each new stair that the humanoid's 's body reaches.
hhttp///moccp.cc.co...ch//
17

---

Published in Transactions on Machine Learning Research (03/2023)
C
Training Details
C.1
Auto-Encoding Demonstration States
Our convolutional VAE consists of encoder and decoder residual networks (Dhariwal et al. 2020 All convolutional layers are one-dimensional and kernel size of 3 with 64 output channels. For the encoder, single convolution layer is followed by 4 residual blocks and another final convolution with 32 output channels. From the 32-dimensional encoder output, we prodict the means and standard variation of an isotropic normal distribution which is KL-regularized towards (00, 1). The KL term is weighted with factor of 0.2. The decoder mirrors the encoder's network structure, with the last layer consisting of a transposed convolution. final output layer performs convolution and restores the input dimensionality.
VAE inputs and outputs are the body's X and velocities, position, global orientation and all local joint rotations. X and Y velocities
integrated to positions relative to the start of the mini-batch, and global joint locations and rotations are computed by propagating local joint rotations through the skeleton (App. A Villegas et al. (2018)). The reconstruction loss resembles the reward function proposed by Hasenclever et al. (020), albeit without scoring velocities. In the following, P denotes joint locations, R denotes 6D rotations (Zhang et al. 2018) and Q denotes quaternions computed from R excluding the root joint:
1
1
Crecons
(djpos dirot) + (0.1 + 0.15rapp 0.65rbg)
2
0.6
dppos
p
PIl1
drot
||
1Rtef
Rl/1
exp (-10 | P Prooo|13)
Troot
exp (-40 I
P l'apol?)
Tapp
Tbq = exp Q-2 a @Q13))
Qrof QQ refers the quaternion difference. Appendages (app) consist of the head, left and right wrist and left and right toe joints of the SMPL skeleton.
We train the VAE mini-batches of 128 trajectories with window length of 256. Parameters optimized with Adam (Kngmma and Ba 2015) using learning rate of 0.0003 with linear warmup over 100 updates. We perform 100000 gradient steps, shufiling the training data between epochs.
C.2 Modeling Trajectories
We use the Transformer implementation from Dhariwal et al. (2020) to train auto-regressive generative model encoded demonst.ration trajectories. Inputs are mapped to 256-dimensional embeddings, and the Transformer consists of 4 blocks, each computing single-head, dense, causal attention followed by layer normalization (Ba et al. 2016), a two-layer MLP with GeLU activation and another layer normalization. From the Transformer output, we predict mean and standard deviation for each the 8 mixture components as well as logits for mixture weights. Entropy regularization performed by sampling from the output distribution and subtracting the resulting pobbbiibas the loss term with factor of 0.2.
Training is performed over 1M gradient steps, with tini-batches of 32 windowed trajectories of length 64. Optimization settings match VAE training, i.c., Adam with a learning rate 0.0003 and warmup over 100 updates.
We apply tanh activations to encoded states and sequence model outputs to improve compatibility with the Soft Actor-Critic algorithm used on downstream tasks. Preliminary experiments regarding the application of SAC unsquashed action spaces demanded additional regularization of policy output layers, which inhibitod learning progress.
18

---

Published in Transactions on Machine Learning Research (03/2023)
C.3 Low-Level Policy
We perform example-guided training of low-level policies described in Hasenclever et al. (2020). We implement V-MPO in a distributed, synchronous setting with 320 actors, and perform 100 gradient steps, each on 160 trajoctorics of longth 32, after collocting the roquired number of samples. For paramoter optimization, we use Adam with a learning rate of 0.0001. Policy and value networks use 4 hidden layers with skip connections to input features (Sinha et al.) 2020), each containing 1024 units, and the encoder architecture of our CoMic baseline matches the policy network. The value network predicts each reward term with a separate output unit. All networks process their inputs with layer normalization and We train each low-level policy with 10B samples (about 2M gradient steps).
We found that when training our low-level policy on static references, i.e. future input frames that are not featurized relative to the robot's current pose, it failed to correctly imitate longer clips. This was due to missing guidance regarding the and Y positions, as our latent states encode and Y velocities only without taking the current simulation state into account. We countcract this in two ways: first, we supply policy and value networks with two additional inpuls specilying the X and Y offset to the current reference pose (while latent states encode immediate future reference poses). These two inputs set to zero when employing the policy on downstream tasks. Second, we provide the value function with relative root body positions of future poses.
For CoMic low-level policy training, we follow Hasenclever et al. 2020 in providing 5 future reference poses to the encoder, and regularizing its output distribution towards N (0, 1) with a factor of 0.0005, In preliminary experiments, we were not able to obtain well-performing CoMic policies on static reference features.
C.4 High-level Policies
On transfer tasks, we freeze the low-level policy and train high-level policies operating in the learned latent space. High-level policy networks consist of 4 hidden layers of 256 units each, with skip connections to input features. We employ SAC with the following hyper-parameters:
Parameter
Value
Learning rate
0.0001
Batch size
256
Replay buffer size
5M
Samples between gradient steps
300
Gradient steps
50
Warmup samples
10000
Initial temperature cocfficient
1
Number of actors
5
During warmup, we perform exploration by randomily sampling actions within the high-level action space and do not perform parameter updates. Other chosen to match Haarnoja et al. 20188).
with the exception of network architectures, are
We select above hyper-parameters by training high-level policies for our low-level policy as well as CoMic for 20M frames on the GoToTargets and a separate hurdles jumping task; sweeps were performed over batch sizes in {256, 512], samples between gradient steps in {50, 100, 200, 300, 500), replay buffer size in {2M, 5M}, learning rates in {0.0001, 0.0003) and initial temperature coefficients in [0.1,1}.
Evaluations are performed every 100k samples on 50 task instances using fixed, per-instance seeds.
C.5 Compute Infrastructure
We implement models and training algorithms PyTorch Passke al., 2019). For pre-training, parallel environment execution and feature computation are implemented in C++. We train demonstration auto- encoders 2 GPUs and sequence models single GPU. For low-level policy training, 16 NVidia
19

---

Published in Transactions on Machine Learning Research (03/2023)
V100 GPUs and 160 CPUs; training 10B samples takes approximately 5 days. High-level policies are trained on a single GPU and require 2-3 days to reach 40M samples.
C.6 Offline Datasets
Experiments in Section 4.3 were performed on offline datasets from DART For AntMaze experiments, we pretrain low-level policies and latent space priors on the dataset from wewemaazeemmiiu--diieee learn high-lovel policies in the hattmaz--mddummdiveeeee.
and environments, and nantmaze-largo-divrseee.
providing a dense reward of -11g - -antzyll For pre-training, we exclude the agent's X and Y coordinates in accordance with (Ajay et al. 2021) (personal communication). The kitchen experiments are performed on düitchen-mixed-vol which is only partially observable (joint velocities are not included in the observation). We combat this by presenting the current as well the last three observations to all policies.
We found that best results are obtained by jointly training a latent space encoder and low-level policy, i.c., we do not train a full VAE to perform state reconstruction. Instead, an encoder (parameterized as the low-level policy) computes the mean and standard deviation of an isotropic normal distribution, subject to KL regularization, from a concatenation of the next five observations. Latent states are sampled from the resulting distribution and provided to the low-level policy, which is trained with behavior cloning with the loss
L(o, v))
E || 24) acll
(eD 21- S116)
For both domains, policies consist of 4 hidden layers of 256 units each, with skip connections to network inputs. Latent space dimensions are adopted from Ajay et al. (2021) and Pertsch et al. (2020) with 8 and 10 for AntMaze and Kitchen environments, respectively. We train low-level polices with 0.2M update steps, using a batch size of 256, Adam with learning rate 3e -4 and a KT. regularization coefficient of 1e 1e-3, Latent space priors consist are implemented as transformers with layers of 256 units each, and output a Gaussian mixture distribution with 4 components. Training data is presented sequences of length 64, and we perform 1M gradient steps with Adam and a learning rate of 3e-1 well as entropy regularization with factor of 0.01.
For high-level policy learning, we use SAC as in Appendix C.4 but with a learning rate of 3e a replay buffer size 1M, 50 samples between gradient steps, 5000 warmup samples and a single actor only. Latent space prior integration done with the following hyper-parameters:
Method
Parameter
AntMaze
Kitchen
2-prior Explore
Prior steps À
16
5
Prior context
4
4
Prior temperature
1.0
0.0
Prior rate
0.01
0.01
Annealing steps
00
1M
z-prior Regularize
Regularization factor
0.5
0.01
Prior context
1
4
Annealing steps
2M
0.5M
z-prior Options
Option length k
10
10
D Low-Level Policy Comparison
high-level policies utilizing a CoMic low-level policy (Hasenclever et al. 2020) exhibit faster initial learning but achieve lower final performance compared to our low-level policy. We investigate the behavior of both low-level policies in additional experiments.
In Section 4.2
First, we plot random walks with both low-level policies in the GoTToTrrees task in Figure For 10 episodes, we uniformly sample random high-level actions. With the CoMic low-level policy, this results in
4We the version from SPIRI httppe////ttaaeecooo/rrrraaaaatt
20

---

Published in Transactions on Machine Learning Research (03/2023)
Z-prior Low-level Policy
CoMic Low-level Policy
15
15
10
5
Y Position
0
-5
-5
-10
-10
-15
-15
-15
-10
-5
0
5
10
15
-15
-10
Position
X Position
Figure 9: Random walks with our low-level policy and CoMic in the GoToTargets task.
Run angle=0
Run angle=90
Run angle= 180
Run angle=270
150
150
150
100
Return
100
75
100
100
50
50
50
25
0.0
0.5
1.5
0.5
1.0 Samples
1.5
2.0
0.0
0.5
1.0 Samples
1.5
2.0
0.0
1.0
1.5
2.0
Samples
1e7
1e7
1e7
Samples
1e7
Low-Level Policies:
z-prior
CoMic
Figure 10: Low-level policy evaluation simple locomotion task, without using latent space priors. A dense reward is provided for moving the robot's body at specific angle. The demonstration data contains clips running forward (angle=0) only; for other directions, our low-level policy (z-prior) achieves substantially higher rewards.
basic locomotion behavior, i.e, taking single steps different directions, while our low-level restricts the random walk to a comparably small region. This matches the observation by Hasenclever al. (2020), which they suggest leads to faster learning.
To better understand the gap in final performance between the two policies, locomotion task. Here, the reward is provided by the length of the current hip translation projected onto a unit vector at a specified angle. For example, an angle of 180 degrees requires the humanoid to walk backwards, and an angle of 90 degrees requires a sideways movement. The results in Figure 10 indicate that our low-level policy (z-prior) outperforms CoMic in all variants but forward rumning (angle=0). Tn the demonstration data, running motions are generally forward-directed, and only single clip contains a backwards walking motion. The findings suggest that the CoMic low-level policy provides behavior close to the demonstration data at the expense of restricting high-level policy control.
them in directed
To further support the hypothesis on restricted high-level controllability with CoMic, we select two states encountered during downstream lask training and compare how low-level actions are influenced by high-level actions. In Figure we plot the cosine similarity for pairs of 256 random high-level actions (X-axis) and their resulting native actions (Y axis). Low-level actions produced with CoMic have a higher overall similarity, particular in the stranding pose (left), which implies reduced controllability compared to the z-prior low-level policy.
21

---

Published in Transactions on Machine Learning Research (03/2023)
Standing Pose
Running Pose
Low-leve leveleeelll cosine similarity
Low-lev lev-leeellll cosine similarity
0.9
0.9
0.8
0.8
0.7
0.7
0.5
0.6
0.5
0.5
-0.5
0.0
0.5
-0.5
0.0
0.5
High-level cosine similarity
High-level cosine similarity
Low-Level Policies:
z-prior
CoMic
Comparison of cosine similarity between random high-level actions and the similarity of the resulting lew-level actions in two different states. Our low-level policy (blue) provides a larger range of low-level actions, while CoMic (orange) produce more similar low-level actions, indicating reduced controllability.
Figure 11:
E
Ablation Studies
In Figure 12
we plot learning curves for several ablations regarding hyper-parameters for integrating latent space priors in high-level policy learning.
When using the prior for exploration (top row), providing longer context the prior does not have significant effect in GoToTargees, which matches the demonstration data well, but hurts learning speed (Gaps, Butterflies) and final performance (Stairs) in other tasks. We hypothesize that a longer context of high-level policy actions which are ont-of-distribution for the prior renders its generated sequences less helpful for exploration.
If the prior is used as a regularization target (middle rows), anncaling is performed over a relatively short window at the start of training. The ablation in the third row shows that regularization conflicts with learning the task at hand: in GoToeargets and Stairs, where the learning starts early, annealing over 5M frames clearly delays learning progress even though later performance is slightly increased. Thus, while regularizing towards the latent space prior yields good initializations, the prior's ignorance of ground-truth simulation states and rewards makes it less useful for long-term guidance.
In the bottom row of Figure 12 we vary the option length when generating temporally extended actions with the prior. Longer options result in loss of performance for Gaps and Stairs; in GaroTargess, for which demonstration clips provide the necessary basic skills of accelerating and turning, faster initial learning with option length of 4.
F Planning with Latent Space Priors
We perform brief investigation into planning with latent space priors simple navigation task with dense, distance-based based reward function. A goal is randomly placed circle with radius of 4m, and the trial is considered successful if the robot was moved within distance of 0.5m to the goal position. Rather than learning high-level policy, we sample candidate sequences from the learned prior in order to steer the low-level policy. Planning is performed in a model predictive control fashion: we first generate number of latent space sequences, decode them into robot poscs with the VAE (Section 2.2 and select
the perfoperforming sequence condition the low-level policy for the next steps (Algorithm I]. We score sequences by determining the maximum reward reached among any of decoded states.
22

---

Published in Transactions on Machine Learning Research (03/2023)
GoToTargets
Gaps
Butterflies
Slairs
2-prior Explore
c=1
Return
6
30
c=4
10
4
10
Samples
Samples
Samples
Samples
GoToTargets
Gaps
Butterflies
10
z-prior Regularize
15
c-1
Return
10
cm8
4
2
2
0
2
2
3
2
3
0
Samples
1e7
Samples
Samples
1c7
Samples
1e7
GoToTargets
Gaps
Butterflies
Slairs
z-prior Regularize
w-1M WM
Retur
30
10
4
10
2
3
0
1
2
3
0
1
2
0
1
2 Samples
Samples
1e7
Samples
1e7
Samples
1e7
1e7
GoToTargets
Gaps
Butterfiles
Stairs
10
p-irior Options
k-2
Return
6
k=4
10
km6
10
0
2
5
3
0
3
0
4
Samples
1c7
Samples
1c7
Samples
1c7
Samples
1s7
Figure 12: Learning curves for hyper-parameter ablations when integrating latent space priors in high-level policy learning.
Algorithm 1 Model Predictive Control with Latent Space Priors
Require: Latent space prior To, low-level policy Mo. VAE encoder q and decoder p
Require: Horizon h, Planning interval /
Require:
Reward function r: S + R
1: states < [initial state 8]
while not done do q(states)
z² <- arg max 1...... m max (p(x/ r(s') r(s')
for = 1 1,....f do
+- E. (als, 21)
+ (TEP(a) 8: states + |states, s]
7:
9: end for
10: end while
23

---

Published in Transactions on Machine Learning Research (03/2023)
(b)
Goals Reached
0.8
0.6
0.0
Samples
MPC
Highilevel colicy
XPPattion PPO
co Or1
XPeettin 391
3cPottion (7Q
Visualization of rollouts when planning with latent space priors, projected onto the X/Y plane. The robot (black dot) initially faces the positive X direction and is tasked to navigate to a goal marked in blue. subset of candidate plans is plotted with grey lines; the selected plan is highlighted orange line depicts the actual future trajectory achieved. (b) Comparison to learning progress of a high-level policy on this lask.
Figure 13: (a)
red. The
We re-plan every 4 environment steps by sampling 1024 candidate sequences of length 64. The variance of trajectories is increased with a high temperature for sampling (we scale the logits of the mixture distribution by a factor of 9 and the standard deviation of mixture components by 3). We evaluate on 50 randomly sampled goals, and compare the MPC implementaiinn against the learning curve of a high-level policy that is trained on randomly sampled goals (Figure (3(b)) With MPC, we reach 90% of the sampled goals within the time limit of 1000 steps (30s), and 68% within 500 steps. A high-level policy with default settings requires about 2.5 million samples to successfully reach the same fraction of goals. The learned policy reaches goals much faster, but often reaches them stumbling and falling (the episode ends when the goal is reached).
We visualize the planning process for several trials in Figure 13 a). While the sampled and decoded rollouts provide sufficient variety and quality to obtain useful plans (red lines), the low-level policy is not able to reenact them accurately all cases. In particular, likely due the demonstration data we considered, following turns from sampled trajectories challengins This mismatch can be alleviated by frequent re-planning. The bottom in Figure 13 a) depicts failed trial (later steps not shown): the character misses the goal, indicated by the orange line showing the future trajectory, and the low-level policy is not able to follow future plans to turn around.
G Coverage of Demonstration Data
In best-case scenario, the demonstration dataset would cover all behavior required on transfer tasks. In our setup, the final policies that were found to perform well on transfer tasks are still markedly different.
AA learned high-level policy can however explore full latent action space and discover suitable actions perform fast. in the GoToearees task.
24

---

Published in Transactions on Machine Learning Research (03/2023)
from the original demonstration data. We quantify this by evaluating the likelihood of evaluation rollouts under our latent space prior. The following data was obtained with 50 rollouts each from the best-performing z-prior Explore policy on each task and results in lower log proobabbbiiiiiis (averaged over sequence elements) under the prior. Note that rather than evaluating high-level action sequences, we encode the full motion produced during each episode with the VAE and score the resulting latent state sequence. We observe lower log- Dog-probabilities of final behaviors compared to the demonstration dataset.
Sequences
Prior log-probability
Demonstrations
-134.8 + 50.5
Go
-995.8 = 68.9
Gaps
-375.8 + 46.1
Butterflies
-15..0 + 32.7
Stairs
-238.9 = 102.8
H Additional Baselines
H.1 SAC and Action Repeat
we provide learning curves for further baselines: Soft Actor-Critic without a pre-trained low-level policy, and a high-level policy that repeats every action twice. SAC does not exhibit any learning progress, which highlights the utility of pre-trained skills for tackling our sparse-reward tasks. The action repeat baseline uses the same Q-function update as for integrating latent space priors via options (3.3.. Tt achieves good results on Targeess and Butterflics which are only slightly lower compared to using the prior generate options. In the Gaps task, however, no meaningful learning progress is made within the first 20M samples.
In Figure 14
H.2 LSTM and Non Nor-paraitteric Prior
In our main experiments, we utilized Transformers implement latent space priors. In Figure 15 the z-prior Explore setting that LSTMs (Hochreiter and Schmidhuber 1997) (in a comparable configuration to our Transformer model) could also be employed, albeit with a reduction in performance on Gaps and Stairs. Selecting random sequences from the demonstration data rather than sampling from a sequence model can also work, matching the Transformer prior in out of tasks. This result matches well with our finding that providing a larger amount of context to the Transformer model is generally not helpful (Fig. 7). On the other hand, directly sampling from demonsttations requires the full (encoded) dataset to be available and cannot be utilized for 'z-prior Options' and 'z-prior Regularize", which achieve superior performance in Go Targets and Stairs (Fig. 5).
we show for
H.3 Low-level Policy from Offline Data
Recently, Wagener et al. (0022) released a large corpus (MoCapAct) for offline RL for the CMU motion capture database, with actions from expert policies trained to imitale individual clips. We select the same subset of clips that use in our main experiments and train multi-clip policy following the "locomotion-small" setup, i.c., with reference encoder output dimensionality of 20. We then train a high-level policy as described in Wagener et al. (0022) on the transfer tasks we consider. Our robot model differs slightly from Wagener et al. 2022)) and we hence adapt our environments for compatibility. In Figure 16 we plot learning curves for setups and observe that for MoCapAct, no learning progress is made. Doubling the sample budget for training for MoCapAct has no effect on our tasks (not shown). We note that different from Wagener et al. (2022), our transfer tasks are not initialized from an demonstration pose and do not employ early episode termination if re-implement the VelocityControl task with dense rewards and early termination and obtain learning progress with MoCapAct comparable to the reported results in Wagener et al. 2022 (Fig. 16 right).
the robot loses balance. To control for custom environments,
25

---

Published in Transactions on Machine Learning Research (03/2023)
GoToTargets
Gaps
Butterflies
Stairs
10
40
15
Return
30
10
20
10
1
2
3
1
2
3
1
2
3
4
5
2
3
4
5
Samples
1e7
Samples
1e7
Samples
1e7
Samples
1e7
z-prior Options
Action Repeat
SAC
Figure 14: Further baselines: repeating high-level actions twice and plain Soft Actor-Critic
GoToTargets
Gaps
Butterflies
Stairs
10
40
15
30
Return
10
20
10
2
2
2
3
3
Samples
1e7
Samples
1e7
Samples
1e7
Samples
1e7
Transformer
LSTM
Dataset
Figure 15: z-prior Explore with Transformer and a LSTM for the latent space prior. We also compare to sampling random latent state sequences from the demonstration dataset for exploration (Dataset).
GoToTargets
Gaps
Butterflies
Stairs
VelocityControl
10
8
15
1500
6
Return
100
1000
4
5
500
2
2
Samples
1e7
Samples
1e7
Samples
1e7
Samples
1e7
Samples
1e7
ours
MoCapAct
Figure 16: Comparing our training setup (without prior) to MoCapAct. The same demonstration dataset is used in both cases.
26


Reviews:
[{"review_id": "QsBIoq2yvI", "reviewer": "Reviewer_HF7j", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of pre-training observation-only demonstrations is neat and the resulting three ways of leveraging the pre-trained latent priors and low-levels, i.e. exploration, regularization and options are reasonable and nicely formulated.\n\n2. The experiment results clearly show that the method can outperform no prior and prior method CoMic low-level policy,\n\nWeaknesses:\n\n1. I think the comparison in the paper is not thorough. One very related direction is leveraging human videos in robot learning, which follows exactly the same motivation of the work. I think the authors should consider discussing and comparing to these works such as [1,2]. \n\n2. The domains shown in the paper are limited to simulated locomotion. I think the method should really shine in areas where actions of the demonstrations are hard to obtain, e.g. human videos as discussed above.\n\n3. The idea of the paper is not particularly novel. The autoregressive latent priors [3], hierarchical skill discovery for down-stream task learning [4] and observation-only demonstration learning [1,2] have all been explored before. The authors seem to piece them together to tackle a less explored task (simulated locomotion), which makes the work less exciting.\n\n[1] Chen, Annie S., Suraj Nair, and Chelsea Finn. \"Learning generalizable robotic reward functions from\" in-the-wild\" human videos.\" arXiv preprint arXiv:2103.16817 (2021).\n\n[2] Yamada, Jun, et al. \"Task-Induced Representation Learning.\" arXiv preprint arXiv:2204.11827 (2022).\n\n[3] Sontakke, Sumedh A., et al. \"Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms.\" arXiv preprint arXiv:2109.03813 (2021).\n\n[4] Ajay, Anurag, et al. \"Opal: Offline primitive discovery for accelerating offline reinforcement learning.\" arXiv preprint arXiv:2010.13611 (2020).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to leverage demonstrations with only sequences of observations for learning a latent space prior modeled with transformers, train a low-level policy on top of the latent representation, and then performs down-stream task learning via obtaining a high-level policy guided by the low-level policy. The authors show that such a method can perform well on various down-stream locomotion tasks leveraging publicly available motion capture data for pre-training.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Add more comparisons such as [1, 2].\n2. Add more domains for evaluation, e.g. robotic manipulation, using human videos as the pre-training demonstrations.\n3. Clarify the novelty.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VmuIMWhcUZ", "reviewer": "Reviewer_E714", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths.\n\n1. The paper is well written and quite clear. I appreciate the care that is taken to put it in context with respect to prior works.\n2. Given that a latent space prior is learned, I think that the four ways of using the prior (exploration, regularization, options, and planning) make a lot of sense.\n3. The performance of the method seems strong.\n\n## Weaknesses.\n\n1. I think that the paper is a bit unclear about whether the latent space $z$ corresponds to latent states or actions. In some parts they are referred to as states (which makes sense, since they are learned by auto-encoded states), and in others refer to them as actions.\n2. The above distinction matters since state transitions in an MDP can be controlled by actions as well as by environmental stochasticity. Does this approach work in general in stochastic environments? I suspect that at least with the planning method, high level “actions” that correspond to lucky environment transitions that give high reward may be selected, leading to issues.\n3. The relationship with CoMic should be made very clear, since it is the main comparison.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work explores how a “latent space prior” can be learned from observation data to accelerate reinforcement learning. First, a representation of latent actions $z$ is learned with an auto-encoder on observation trajectories. Next, a sequence model is learned with an auto-regressive model (implemented with a Transformer). A separate “low-level policy” is learned to translate high level actions $z$ into low level actions.\n\nThe paper explores a few ways which such a setup can be used to accelerate policy learning on different transfer tasks: \n\n1. First, it proposes to use the latent space prior together with the low level policy as an exploration policy.\n2. Second, it proposes to use the latent space prior as regularization while learning a high level policy by biasing the learned policy towards the prior\n3. Third, it proposes to use the prior as temporally extended options by learning a policy that sets the initial latent state for the prior, from which $k$ time-steps of high level actions are sampled.\n4. Finally, the latent space priors can be used for planning, by sampling sequences of high level actions and decoding them into states using the auto-encoder and evaluating the rewards of various sampled sequences.\n\nThe strength of these approaches are evaluated on a simulated humanoid robot. The latent space prior is trained on motion capture data, and each of the above methods is tested across four tasks. The method works well and compares strongly to the baselines on all of the tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [Critical] Please add some discussion about how the latent states can be used as high level actions.\n2. [Critical] I would like to see a discussion about the applicability of different approaches in stochastic environments.\n3. [Critical] I would like to see a discussion about the baselines, justifying the choice of baseline and the relationship between the baselines and your method.\n4. I think the related works section could be improved by adding subsection or paragraph headers.", "broader_impact_concerns": "I do not believe there are any broader impact concerns with this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gM63TxDpx3", "reviewer": "Reviewer_TeiQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths and weaknesses:\n\nOverall I enjoyed reading the paper — the motivation is stated clearly and the three strategies to make use of the learned latent space prior are easy to understand. I also like the spirit of presenting a variety of ways that one may approach hierarchical policy integration. Finally, the tasks, although all simulated, seem to be decently challenging.\n\nMy main criticisms of the paper focus on the following aspect (1) technical novelty and (2) insufficient baselines.\n\nLike mentioned in the “strengths” part of the comment, I like the spirit of a unified view of hierarchical policy integration. However, I struggled to differentiate the proposed method from many other works published in the last few years. For example, as stated in the related works section, Co-Reyes et al. 2018, learns latent-space policy but uses only for planning. Pertsch et al. 2020, learns latent-space policy but assumes actions are available. Some other examples include Lynch et al. 2018 (Learning from Play Data), which learns latent-space policy but only uses it for one-shot imitation. And Mandlekar et al., 2019 (IRIS) which learns latent-space for RL but only offline. One can enumerate many more such examples of “X did this, but differs from this paper in this specific aspect”. But why is this new / important? What does this method do that makes it stand out / can learn tasks so difficult that no other methods can do? In my opinion, just presenting an alternative (but not too different from existing works) to a problem without clearly stating and demonstrating the “why” is not sufficient.\n\nRelatedly, the experimental result is lacking. Just comparing with a single baseline on such a densely studied topic is clearly not enoughs. One may argue that none of the closely related works discussed makes exactly the same assumption as the presented work, but many of them can be adapted and used for a fair comparison. The experiment result is also a good place to show why the presented method matters. It’d be great to show that the method can outperform other methods on real robot platforms (e.g., table-top manipulation). \n\nMinor:\n- Figure 2 is somewhat confusing. I only understood what it means after going through section 3 a few times. The colors and arrows don’t really help explain the figure.\n- What is the “no-prior” baseline?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a hierarchical decision making framework that helps make online reinforcement learning faster by using offline demonstration datasets. The high-level idea is to learn a probabilistic latent space of short-horizon skills from demonstrations and use the skill latent space in lieu of low-level action space to learn hard-exploration problems using online RL. Specifically, the paper proposes to train a latent-variable model (VAEs) of short-horizon behaviors from offline demonstration. The latent-space priors are then used to condition control policies to learn the desired behavior through example-guided RL without assuming access to ground-truth actions. The paper proposes to use the behavior priors in three different ways: (1) train an autoregressive model over the priors to generate meaningful exploration behaviors (2) regularize an RL learner to generate “prior-like” behaviors and (3) use the prior-conditional policies as options to learn a hierarchical RL policy.  The paper also shows that the latent-space priors can be used to perform planning. The method is evaluated on four simulated domains with simulated humanoid robot figures. The main result is that the prior-conditional low-level policies can indeed be used to aid task learning in the three different ways discussed above. And that all strategy discussed above can outperform a flat policy baseline (Hasenclever et al., 2020)\n", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall I believe the authors need to spend more effort on showing why the presented method is important. There are many ways to show this, including but not limited to:\n\n1. Present a clear, unified theoretical picture of the hierarchical decision making framework based on latent-space behavior prior. And show the empirical findings agrees with the theory.\n2. Show that the proposed method outperforms a number of state-of-art methods on challenging simulated domain.\n3. Show that the method works on an application of broad interest, e.g., significantly speed up the policy learning on a physical robot. ", "broader_impact_concerns": "The method presents no ethical implication as far as I can tell. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies leveraging offline datasets for learning “latent space prior” using transformers and using that to train a low-level policy. The paper explores several methods of integrating the latent space priors for the downstream tasks, for example, by augmenting the exploration policy with sequences sampled from the prior, regularizing the policy towards distributions predicted by the prior or planning with HRL/options framework. 

The paper is very well-written and easy to read. The novelty of the method is limited, but the experimental results are interesting. Nevertheless, this paper was borderline, with reviewers having confusion about the novelty and experimental protocol presented in this paper. However, the authors address most of the concerns raised by the reviewers during the rebuttal phase. The authors have addressed some of the important concerns the reviewers raised regarding baselines and experiments on other domains. They have clarified some of the concerns related to novelty as well. However, there are still outstanding comments raised by the Reviewer TeiQ about the experiments on real-world tasks. However, running experiments on real robots and including them in the paper would be beyond the scope of this rebuttal. As it stands, the experiments on the simulation environments used in the latest version of the paper should be sufficient for publication. 
Decision: Accept (poster)