Choreographer: Learning and Adapting Skills in Imagination
Authors: Pietro Mazzaglia
Tim Verbelen
Bart Dhoedt
Alexandre Lacoste
Sai Rajeswar
Venue: ICLR 2023 notable top 25% (2023)
Keywords: unsupervised reinforcement learning
skill learning
world models

Abstract: Unsupervised skill learning aims to learn a rich repertoire of behaviors without external supervision, providing artificial agents with the ability to control and influence the environment. However, without appropriate knowledge and exploration, skills may provide control only over a restricted area of the environment, limiting their applicability. Furthermore, it is unclear how to leverage the learned skill behaviors for adapting to downstream tasks in a data-efficient manner. We present Choreographer, a model-based agent that exploits its world model to learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space of the model. During adaptation, the agent uses a meta-controller to evaluate and adapt the learned skills efficiently by deploying them in parallel in imagination. Choreographer is able to learn skills both from offline data, and by collecting data simultaneously with an exploration policy. The skills can be used to effectively adapt to downstream tasks, as we show in the URL benchmark, where we outperform previous approaches from both pixels and states inputs. The skills also explore the environment thoroughly, finding sparse rewards more frequently, as shown in goal-reaching tasks from the DMC Suite and Meta-World. 
Project website: https://skillchoreographer.github.io/

Published as a conference paper at ICLR 2023
CHOREOGRAPHER: LEARNING AND ADAPTING SKILLS IN IMAGINATION
Pietro Mazzaglia"
Tim Verbelen
Bart Dhoedt
Alexandre Lacoste
Sai Rajeswar*
Ghent University imec ServiceNow Research
ABSTRACT
Unsupervised skill learning aims learn rich repertoire of behaviors without external supervision, providing artificial agents with the ability to control and in- fluence the environment. However, without appropriate knowledge and explo- ration, skills may provide control only over restricted area of the environment, limiting their applicability. Furthermore, is unclear how to leverage the learned skill behaviors for adapting to downstream tasks in a data-efficient manner. We present Choreographer, model-based agent that exploits its world model learn and adapt skills in imagination. Our method decouples the exploration and skill learning processes, being able to discover skills in the latent state space the model. During adaptation, the agent uses a meta-controller to evaluate and adapt the learned skills efficiently by deploying them in parallel imagination. Chore- ographer is able to learn skills both from offline data and by collecting data simul- taneously with an exploration policy. The skills can be used to effectively adapt downstream tasks, we show in the URL benchmark, where outperform previous approaches from both pixels and states inputs. The learned skills also explore the environment thoroughly, finding sparse rewards more frequently, shown in goal-reaching tasks from the DMC Suite and Gorld.
Project website: https///si1cchhororrraahhh waaeeerrggttttu github..tb..... 10/
INTRODUCTION
Deep Reinforcement Learning (RL) has yielded remarkable success in wide variety of tasks rang- ing from game playing (Mnih et al. 2013 Silver et al. 2016) to complex robot control (Smith et al.]
2022 OpenAl al. 2019). However, most of these accomplishments are specific to mastering single task relying on millions of interactions to learn the desired behavior. Solving a new task generally requires start over, collecting task-specific data, and learning a new agent from scratch.
Instead, natural agents, such as humans, can quickly adapt to novel situations or tasks. Since their infancy, these agents are intrinsically motivated try different movement patterns, continuously acquiring greater perceptual capabilities and sensorimotor experiences that are essential for the for- mation of future directed behaviors (Corbettal 2021). For instance, child who understands how object relations work, c.g. has autonomously learned to stack one block on top of another, can quickly master how to create structures comprising multiple objects (Marcinowski et al. 2019).
With the same goal, unsupervised RL (URL) methods aim to leverage intrinsic motivation signals, used drive the agent's interaction with the environment, to acquire generalizable knowledge and behaviors. While some URL approaches focus exploring the environment (Schmidhuber) 1991 Mutti et al., 2020. Bellemare et al.) 2016), competence-baxed (Laskin al.] (202)) methods aim to learn a set of options skills that provide the agent with the ability to control the environment Gregor et al. 2016 Eysenbach et al., 2019), a.k.a. empowerment (Salge et al.) 2014).
Learning set of options provide an optimal set of behaviors quickly adapt and generalize to new tasks (Eysenbach et al. 202T). However, current methods still exhibit several limitations. Some of these due to the nature of the skill discovery objective (Achiam et al.. 2018). struggling to capture behaviors that are natural and meaningful for humans. Another major issue with currett
"Correspondence pietro.mazzagliace @uggettt.h

---

Published as a conference paper at ICLR 2023
methods is the limited exploration brought by competence-based methods, which tend to commit to behaviors that are easily discriminable but that guarantee limited control over a small area of the environment. This difficulty has been both analyzed theoretically (Campos et al. 2020 and demonstrated empirically (Laskin et al. 2021 Rajeswar et al. 2022).
A final important question with competence-based methods arises when adapting the skills learned without supervision to downstream tasks: how to exploit the skills in an efficient way, i.e. using the least number of environment interactions? While one could exhaustively test all the options in the environment, this can be expensive when learning a large number of skills (Bysenbach et al. 2019) or intractable, for continuous skill spaces (Kim et al., 202T; Liu & Abbeel 2021a).
In this work, we propose Choreographer, an agent able to discover, learn and adapt unsupervised skills efficiently by leveraging a generative model of the environment dynamics, a.k.a. world model (Ha & Schmidhuber] 2018). Choreographer discovers and learns skills in imagination, thanks to the model, detaching the exploration and options discovery processes. During adaptation, Choreogra- pher can predict the outcomes of the learned skills' actions, and so evaluate multiple skills in parallel in imagination, allowing to combine them efficiently for solving downstream tasks.
Contributions. Our contributions can be summarized as follow:
We describe a general algorithm for discovering. learning, and adapting unsupervised skills and data-efficient. (Section 3).
that is
We propose a code resampling technique to prevent the issue of index collapse when learn- ing high-dimensional codes with vector quantized autoencoders (Kaiser et al. 2018), which we employ in the skill discovery process (Section 3.2):
We show that Choreographer can learn skills both from offline data in parallel with exploration, and from both states and pixels inputs. The skills are adaptable for multiple tasks, shown in the URL benchmark, where outperform all baselines (Section 4.1):
We show the skills learned by Choreographer are effective for exploration, discovering sparse rewards in the environment more likely than other methods (Section 4.2). and we further visualize and analyze them to provide additional insights (Section 4.3
2 PRELIMINARIES AND BACKGROUND
Reinforcement Learning (RL). The RL setting can be formalized as a Markov Decision Process, where we denote observations from the environment with Tt, actions with ar, rewards with r., and the discount factor with 7. The objective of an RL agent is to maximize the expected discounted sum of rewards over time for given task, a.k.a. the return: G, Ekm++ (k-(-1)) 2 We focus on continuous-action: settings, where a common strategy is to learn a parameterized function that outputs the best action given certain state, referred to as the actor To (a(z)), and parameterized model that estimates the expected returns from a state for the actor, referred to as the critic (z). The models for actor-critic algorithms can be instantiated as deep neural networks to solve complex continuous control tasks (Haarnoja et al. 2018: Lillicrap et al., 2016 Schulman et al. (017).
Unsupervised RL (URL). In method, we distinguish three stages of the URL process (Figure []. During stage (i) data collection, the agent can interact with the environment without rewards, driven by intrinsic motivation. During stage (ii) pre-training, the agent uses the data collected to
collection
Pre-training
Fine-tuning
Replay buffer
Envirorment
Agent
PT model
Unlabelled data
model
behavior
FT behavior
Task
Unsupervised phase
Supervised phase
Figure 1: Unsupervised Reinforcement Learning. The agent should effectively leverage the unsu- pervised phase, consisting of the data collection and the pre-training (PT) stages, to efficiently adapt during the supervised phase, where the agent is fine-tuned (FT) for downstream task.
2

---

Published as a conference paper at ICLR 2023
learn useful modules (e.g. world model, skills, etc.) adapt faster to new scenarios. Crucially, stage (i) and (ii) need to happen in parallel for most of the previous competence- basee methods (Eysenbach al.] 2019 Liu & Abbeel 202 Gregor et al. 2016; Achiam al. 2018), skills discriminability is tied to the data collection process. Finally, in stage (iii) fine-tuning, the agent can interact with the environment receiving task reward feedback and should leverage the pre-trained modules to adapt faster to the given task.
Mutual Information Skill Learning. Option discovery can be formalized learning a skill rep- resentation z to condition skill policies π T[aaz, z z). The objective of these policies is to maximize the mutual information between skills and trajectories of environment observations T {Et, T++1, :
[ 2) (Ta)) H(Tx(²) = (=) (z(m))
Il forward
// reverse
The skill policy T(alæ, 2) results in family of diversified policies with predictable behavior. Some previous work uses the forward form, either considering the entropy H (Tz) fixed (Campos et al, 2020) estimating it through sampling-based techniques (Sharma al., 2020 Liu & Abbeel, 2021a Laskin et al., 2022). and learning a function mapping skills to states to estimate conditional entropy. Other works use the reverse form, considering a fixed distribution over skills, with maximal entropy (z),, and learning a skill discriminator that maps states to skills, to estimate conditional
entropy (Achiam et al. 2018
Gregor et al. 2016 Eysenbach et 2019 Hansen al. 2020
3 CHOREOGRAPHER
Choreographer is a model-based RL agent designed for URL., leveraging a world model for discover- ing, learning, and adapting skills in imagination. In addition world model, the agent learns: a) a codebook of skill vectors, clustering the model state space, b) skill-conditioned policies, maximizing the mutual information between latent trajectories of model states and skill codes, and c) a meta- controller, to coordinate and adapt skills for downstream tasks, during fine-tune fine-tuning. An overview of the approach is illustrated in Figure 2 and a detailed algorithm is presented in Appendix G
3.1 WORLD MODEL
Choreographer learns a task-agnostic world model composed of the following components:
Posterior: (s $t-1 (t-,,
Prior: Pe (s (1-5)))-1,,4-11,
Reconstruction:
During the mee-tuning stage, the agent also learns reward predictor (r( (s.))ss.. The model states St have both a deterministic component, modeled using the recurrent state of GRU (Cho et al. 2014)) and discrete stochastic component (Hafner al. 2021 The remaining modules are multi-layer perceptrons (MLPss) or CNNs for pixel inputs (LeCun Cun et al. 1999). The model is trained end-to-end by optimizing an evidence lower bound on the likeliihl of data:
Lum = DKL. (yoo (8t ( a: Fz.) Ip (8t 8t (, - (e (log po (xx(s)));
where sequences of observations x, and actions o are sampled from a replay buffer.
3.2 SKILL DISCOVERY
To discover skills that are exploration-agnnee agnostic, i.e. do not require the agent to deploy them in order evaluate their discriminability, we turn the skill discovery problem into a representation learning one. We learn a skill discrete representation & on top of the model states using a VQ-VAE (yan den Oord et al. 2017). This representation aims encode model states into the N codes of codebook, learning p( =) as well as the inverse relation, decoding codes to model states, via p(s[=).
In the skill auto-encoder, model states are first processed into embeddings using a skill encoder E(s). Then, each embedding is assigned to the closest code in the codebook =q
-(6) i E 1...N, through:
quantize( FF(s)) =({)
k = arg min [{(())
x 1/2
(4)
3

---

Published as a conference paper at ICLR 2023
Codebook
sun max [ 2)
entrrpp
code
quant
Z
skill policy
skill policy
$ C+2
dec
(b) Skill learning.
ctrl
ctrl
neta
task
t-1
a2
8
t+2
(a) Skill discovery.
(c) Skill adaptation.
Figure 2: Choreographer. (a) The agent leverages representation learning to learn a codebook of skill vectors, summarizing the model state space. (b) Skill policies are learned in imagination, maximizing the mutual information between latent trajectories of model states and skill codes. (c) The meta-controller selects the skills to apply for downstream tasks, evaluating and adapting skill policies in imagination by hallucinating the trajectories of states and rewards they would obtain.
The indices are mapped back to their corresponding vectors in the codebook, from which the decoder D(z)) reconstructs the corresponding model states. The training objective is:
C. = |s D(= 112 + B(|sg(= E(s)
(5)
where sg(+) indicates stop gradients. The first loss term the reconstruction error of the decoder, while the second term encourages the embedded vectors to stay close to the codes in the codebook.
Crucially, as opposed to most previous work (van den Oord al. 2017 Razavi et al. 2019). we choose to learn single-code representation, where each model state gets assigned only one vector the codebook. Our skill codes can thus be seen as centroids partitioning the model state space into N clusters (Roy et al. 2018), so that intuitively each skill occupies one area of the state space.
Code resampling. When training single-code representations with VQ- VAE, the codebook may suffer from index collupse (Kaiser et al. 2018), where only few of the code vectors get trained due to preferential attachment process. One way to mitigate this issue is to use sliced latent vectors, which assign multiple smaller codes the same embedding (Kaiser al. 2018 Razavi et al. 2019), but this would imply losing the one-to-one association between codes and skills.
In order to mitigate the issue we propose code resampling technique which consists of two steps:
keeping track of the inactive codes during training, which is achieved by looking at codes to which embedding is assigned for M consecutive training batches;
2. re-initializing the inactive codes, with values embeddings E(s) from the latest training
dE()(s)) batch, selected with probability E. ()))( where d,(.) is the euclidean distance from
the closest code the codebook, i.e. (E(s)) min.e1...N [(s) 2 1
The second step ensures that in every M batches all the codes become active again. In Section 4.3 we show that our technique necessary for training large discrete skill codebooks, as it keeps codes active, and leads to lower training loss. We provide additional details in Appendix C
3.3 SKILL LEARNING
Choreographer learns a generative world model, which allows the agent to imagine trajectories of future model states T, (st, st.1,..... A major novelty of our approach is that we leam skill rep- resentation that is linked to model states rather than to environment observations. As a consequence, for skill learning, define the objective of Choreographer maximize the mutual information
4

---

Published as a conference paper at ICLR 2023
between trajectories of model states and skills [TT,, 2). In visual environments, this is particularly useful as the model states s, can have a much more compact representation than observations z₁. We rewrite the mutual information objective in forward form as:
I(Ts 2) (T,)) H[(T) (Ts)) + Rand, (t) p()2))
(6)
where the second term uses (s) to express the stationary distribution of model states visited by the skill policy conditioned on the skill code 2, and codes are sampled from uniform distri- bution. In order to maximize this objective with RL, we estimate the two terms to be part of an overarching reward function. The entropy term is estimated using a K-NN particle-based estimator (Singh et al. 2003), obtaining Tent X Σx= log ||s . ||2. For the second term, we can rollout canlttcond policies in imagination, obtaining sampled trajectories of model states, and use the distance between the states visited and the decoded state corresponding to the conditioning skill
code: z to estimate the likelihood (Goodfellow et al. 2016). obtaining Tcode Xx -I) - =112.
Choreographer learns N code-conditioned skill policies using actor-critic algorithm:
Takill = Tent + Teode
Skill actor: ( (a.]sss,))))
Skill critic: Usa 2),
(7)
where the policies are encouraged to reach states corresponding to their skill- skill-conditioning code 2, by the Toode term, but at the same time are encouraged to be dynamic and visit different states in one trajectory, by the rent term. Given that the reward terms can be computed using the agent's s model, the skill actor-critics are trained in the agent's imagination, backpropagating through the model dynamics (Hafner et al.j (021). By doing so, we are able to disentangle exploration and skill learning, as the two processes can happen in parallel or at different times. Visualizations of the skills learned by Chorcographer are available in Figure and and in the videos on the project website
3.4 SKILL ADAPTATION
Choreographer is able to learn several skill-conditioned policies in completely unsupervised fash- ion, leveraging representation and reinforcement learning. This feature is particularly useful during the pre-training stage of unsupervised RL, where the agent learns both a task-agnostic world model of the environment and a of reusable behaviors. During fine-tuning, the agent needs leverage such pre-trained components in order to adapt to downstream task in an efficient way.
Exhaustively deploying the skill policies, a.k.a. the agent's options, in the environment, to under- stand which ones correlate best with the downstream task, is expensive and often unnecessary. Choreographer can again exploit its world model to deploy the skill policies in imagination.
In order to coordinate the multiple options available, Choreographer learns a meta-controller, policy over options (Precup) 2000) deciding which skill codes use to condition the skill policy:
Tmeta
Flusky
Meta-ctrl actor:
Meta-ctrl critic: Pmeea(St)..
The meta-controller is updated with rollouts in imagination using the task rewards from reward predictor, trained only at fine-tuning time. The outputs of the meta-controller's actor network are discrete (skill) vectors and we use policy gradients update the actor (Williams) 1992). In order to fine-tune the pre-trained skill policies, we can backpropagate the gradients to the skill actors as well (Kingma & Welling 2013). As we discuss in Section 4.3] this improves performance further as the pre-trained skills are not supposed to perfectly fit the downstream task beforehand.
4 EXPERIMENTS
We empirically evaluate Choreographer to answer the following questions:
Does Choreographer learn and adapt skills effectively for unsupervised RL? (Section We use the URL benchmark (Laskin et al.) (222T) to show that, after pre-training on exploration data, our agent can adapt to several tasks data-efficient manner. We show this holds for both training on states and pixels inputs and in both offline and online settings.
Do the skills learned by Choreographer provide a more exploratory initialization? (Section 4.2 We zoom in the Jaco sparse tasks from URLB and use sparse goal-reaching
5

---

Published as a conference paper at ICLR 2023
tasks from Seta-World (Yu et al. 2019) evaluate the ability of our agent to find sparse rewards in the environment, by leveraging its skills.
Which skills are learned by Choreographer? (Section 4.3) We visualize the diverse skills learned, highlighting the effectiveness of the proposed resampling technique. Finally, we diseuss the "zero-shot' adaptation performance achievable when fine-tuning only the meta- controller, with no gradients flowing to the skill policies.
Implementation details and hyperparameters are provided in Appendix and the open-source code available on the project website
Baselines. For state-based experiments, we adopt the baselines from the CIC paper (Laskin et al, 2022), which can be grouped in three categories: knowledge-basee (blue in plots). which ICM
Pathak et al., 2017), RND (Burda et al.] (201)). Disagreement (Pathak et al. (2119); data-based
(green in plots), which are APT (Liu Abbeel 2021b) and ProtoRL (Yarats al., 2021): and
competence-based (red/orange in plots) APS (Liu Abbeel, (021a). SMM (Lee et al. (0199) and
DIAYN (Eysenbach al. 2019). For pixel- exxel-baed experiments, we compare to the strong baselines
from Rajeswar et al. 2022 combining some of the above approaches with the Dreamer algorithm.
4.1 URL BENCHMARK
Offiine data. Choreographer decouples the exploration and skill learning processes, and can thus learn from offlin pre-collected data. To show this, we use the pre-collected datasets of exploratory data released in the ExORL paper (Yarats et al. (022a). We take the first 2M steps from the datasets corresponding to the state-based URLB domains and use these to pre-train Chorcographer offline, learning the world model and the skill-related components. In Figure 3 we compare the fine-tuning performance of our method, leveraging the meta-controller at fine-tuning time, other methods. In particular, we show the performance when pre-training Choreographer using the data collected by the RND approach (Burda al. 2019). Appendix D we show results with different datasets.
Following (Agarwal et al. 2021), ,, we use the interquartile mean (IQM) and optimality gap metrics, aggregated with stratified bootstrap sampling, as our main evaluation metries. We find that Chore- ographer overall performs best on both metrics, particularly outperforming the other approaches on the Walker and Quadruped tasks and performing comparably to CIC and ProtoRL in the Jaco tasks. We highlight that Choreographer performs far better than the other competence- based approaches, SMM, APS, and DIAYN, and also strongly outperforms RND, which is the strategy used collect its data, showing it's stronger method to exploit the same exploratory data.
Parallel exploration. With no pre-collected data available, Choreographer can learn an exploration policy in parallel to the skills (Mendonca et al.) 202Tb). to collect data and pre-train its compo- nents. This idea detailed in Appendix A We use the pixel-based setup of URLB to see how the performance evolves over time the size of the pre-training data buffer grows.
In Figure 4 we show the performance of Choreographer, collecting data with an exploration policy based the LBS exploration approach (Mazzaglia et al. 2021 and learning skills simultaneously. The agents are being tested at different times during learning, taking snapshots the agent at 100k,
Choreographer
Pre-tmining on steps
GIC
90 B0
SHM
Hommlted Setur (S)
70
APT
ProteRL
ICM
Disagrasment
28
RND
10
0.2
1.0
0.2
Bookstrapped
Optimality Gap
Walker
Quadriped
Jaco
Figure 3: Offline state-based URLB. On the left, Chorcographer (ours), pre-trained with offline exploratory data, performs best against baselines, both in terms IQM and Optimality Gap. On the right, mean and standard deviations across the different domains of URLB are detailed (10 seeds).

---

Published as a conference paper at ICLR 2023
Choreographer
DIAYN
APS
APT
ICM
LBS
Normalized Return (9)
100
90
30
80
60
60
60
40
50
70-
40
20-
100k
500k 1M
100k
500k
2M
100k
500k
1M
100k
500k
2M
Walker
Quadruped
Jaco
Overall
Figure 4: Pixel-based URLB. Performance of Choreographer (ours) function of pre-training steps. Scores are asymptotically normalized and averaged across tasks for each method. (10 seeds)
500k, 1M, and 2M steps and fine-tuning them for 100k steps. Additional baselines from Rajeswar et al. 2022). are presented in Appendix D
Overall, Choreographer's performance steadily on top, with similar performance APT, LBS, and ICM until 500k steps, and becoming the one best-performing algorithm from 1M steps on. As also noticed in other works (Laskin al., (22TT. other competence- struggle with the benchmark. Furthermore, both LBS and APT's performance in Jaco starts decaying after 500k steps. As also discussed in (Rajeswar et al., 2022), this due to the bad initialization of the actor provided by the exploration approach. Our approach overcomes this limitation by learning a set diverse policies, consistently providing a more general initialization that is easier to adapt.
approaches APS and DIAYN
4.2 SPARSE REWARDS
Finding sparse rewards in the environment is challenging, though pre-trained exploratory behaviors may help (Parisi et al. 2021). Choreographer can exploit the skill policies learned during pre- training reach different model states, efficiently skimming the environment find rewards. We test this hypothesis using sparse goal-reaching tasks in two pixel-based environments: Jaco from the DMC Suite, and Meta World (Yu et al.) 2019). order solve these tasks, the agent needs to reach within certain radius of an unknown target goal located in the environment with no feedback.
Jaco. We zoom in on the performance on the four reaching tasks from URLB, Table [ we show the fraction of runs reaching near the target, without any update of the pre-trained policies, over 100 evaluation episodes. Only Choreographer and random actions found the goals all runs, the other methods have to rely entropic exploration to find the goals" The other skill-based methods, APS and DIAYN, also struggled to find the targets consistently.
Meta- Meta-World. World. We sampled three of 50 goals, in the Sawyer environment from MetaWorka For each goal in the sets, we evaluate the policies pre-trained by different methods, testing them for 100 evaluation episodes, without any fine-tuning. The performance in Table that Choreographer finds the most goals in the environment. All the other approaches struggled in the task, with random actions being the best alternative for higher chances of finding sparse rewards for these goals.
For the results Figure all the baselines (except Choreographer) had an action entropy maximization term in the fine-tuning objective Haarnoja el al. 2018), helping them to find the goals.
Domain
Tack
Chorco
Random
APS
DIAYN
API
ICM
LBS
PIE
RND
0.67
Jacn
Bottom right
Top left
1.0
1.0
1.0
1.0
0.67
0.67
0.67
0.67
0.33
0.0
0.67
0.67
0.33
0.0
0.33
1.0
0.33
0.0
right
Goal sci
0.78
0.47
0.31
0.08
0.24
0.33
0.33
0.51
0.35
MetdWolld
Goal set 0.80
0.19
0.18
0.02
0.16
0.37
0.18
0.43
Table I: Sparse rewards. Finding goals in the environment with sparse rewards using the pre- trained behaviors. The table shows the fraction of goals found by each method per task. (3 seeds)

---

Published as a conference paper at ICLR 2023
Code
Tw0
T-10
T-20
Tw30
Tw40
Tw50
T-60
Tw70
T-80
T-90
M
4
Figure 5: Skills visualization. Highlighted in green, reconstructions of the skill codes conditioning skill behavior on their right, sampling frames over time. Animated results on the project website
4.3 SKILLS ANALYSIS
Skill visualization. In Figure visualize some of the skills learned by Choreographer, in the pixel-based Walker and Quadruped environments. The codes conditioning the skill policies are shown by first reconstructing the corresponding model state, and the image observation using the reconstruction model. We note that Choreographer' 's learned behaviors accurately reflect the skill learning objective (Eq. 7 they tend reach the states corresponding the conditioning skill codes, but they also dynamically visit different states. For the Walker, this leads to walking-like behaviors for standing-up skill codes (Ist row), or behaviors that move around, by hopping or flipping on the torso/legs (2nd row). For the Quadruped, skills tend to learn tipping-over behaviors, where the agent reaches the skill code state alternating rotating and overturning moves (3rd/4th rows). Further visualizations are in Appendix Eld on the project website
Code resampling. Learning diversified set skill codes important to learn many useful skill policies for adaptation. Because of index collapse (Kaiser et al., 2018), VQ-VAEs tend to ignore some the skill codes, choosing not to use them during training. As Choreographer samples skills uniformly for skill learning (Eq. 6). it would be inefficient train policies that conditioned on inactive codes. As we show Figure 7 in the pixel-based URLB environments, code resampling cir- cumvents this issue, making nearly all the codes active during training, reducing the reconstruction loss, and avoiding skill code collapse (see examples of collapsed codes in the Figure).
Zero-shot skill control. Pre-training skill policies is general approach provide better initializa- tion of the agent's 's behavior for fine-tuning to downstream tasks (Eysenbach et al. 2021). As shown in (Eysenbach et al., 202I). is possible learn set skill policies able to solve any task, without adaptation. We empirically reinforce this hypothesis with an ablation on the pixel-based URLB, where we learn the meta-controller, which interpolates the skill policies to solve the task, but we do not fine-tune the skill policies. For space constraints, these results, showing a consistent gap in performance versus the skill-adapting counterpart, are presented in Appendix D Combining multiple skills to solve complex long-term tasks with little to adaptation, similarly to hierarchical RL (Gehring et al. 2021
Hafner et al. 2022), a challenge we aim to address in future work.
100
12 codes 128 +R
Reconstruction loss
0115
72 coded CR - 128 conee
Unused cades (%6)
54 codka
522 co0ea
cotee
312 cader
0.1
60
0.08
20
(a) Reconstruction of 64 codes, no code resampling. Inactive, collapsed codes highlighted in red,
1.0
1.F
Sieas (4)
10'
101
10
Steps
Figure 7: Code resampling (CR). When using more than 32 codes, CR reduces the number of
unused codes (~0% after 10k steps) and leads lower reconstruction loss. (3 seeds x 3 domains)
8

---

Published as a conference paper at ICLR 2023
5 RELATED WORK
Unsupervised RL. Among URL methods, we can distinguish (Srinivas & Abbeel 2021 :
knowledge-based methods, which aim visit states of the environment that maximize the agent's
model error//ncertainnyy (Schmidhuber, 1991 Pathak et al., 2017 Burda et al. 2019). data-based
methods, which aim maximize the entropy of the data collected (Hazan al. 2019 Yarats al.
2021 Liu & Abbeel 2021b), competence-basee methods, which learn a set of options or skills that
allow the agent to control the environment (Greeorr et al. 2016 Eysenbachh et al., 2019 Hansen et al.) [222]] Knowledge and data-based methods focus on expanding the available information, while competence-based methods, such as Choreographer, aim to provide empowerment.
Exploration and skills. Exploration is major challenge in skill learning approaches. Previous work has attempted to solve this issue by using state maximum entropy (Laskin et al.; [2022] Liu & Abbeel 202 or introducing variations of the skill discovery objective (Kim et al. 2022 Park
ct al. 2022). J. The idea of decoupling exploration and skill discovery has been less explored and
mostly applied in simpler environments (Campos et al. 2020 Nieto et al. (02I). To the best of our knowledge, Choreographer represents the first approach that enables scaling the idea to complex en-
vironments and tasks successfully from unsupervised data. Additional comparisons in Appendix [
Behavior learning in imagination. The Dreamer algorithm (Hafner et al. 2019 2021) introduced
behavior learning in imagination, establishing a modern implementation of the Dyna framework (Sutton| 199T)_ The idea has been applied for exploration (Sekar et al. 2200 Rajeswar et al. [002]]
Mazzaglia et 2021), goal-directed behavior (Hafner al. [022]] Mendonea et al. 202 and
planning amortization Xie et al. (2020). In particular, RRajeswar et al.
(2022) showed the effec-
tiveness of fine-tuning behavior in imagination on the URL benchmark from pixels, combining unsupervised RL for exploration with Dreamer. Choreographer not only yields better performance
but also decouples exploration and behavior pre-training as the skill policies can be learned offline.
6 DISCUSSION
Competence-based approaches aim to learn set of skills that allow adapting faster to downstream tasks. Previous approaches have struggled to accomplish this goal, mainly due to the impaired exploration capabilities of these algorithms, and the impossibility of efficiently leveraging all the learned skills during adaptation time. In this work, we introduced Choreographer, a model-based agent for unsupervised RL, which is able to discover, learn and adapt skills in imagination, without the need to deploy them in the environment. The approach is thus exploration-agnosti and allows learning from offiine data exploiting strong exploration strategies.
In order to decouple exploration and skill discovery, we learn a codebook of skills, intuitively clus- tering the agent's model state space into different areas that should be pursued by the skill policies. While this proved to be successful in several domains, it might not be the most effective skill repre- sentation, and alternative ways for learning the skills could be explored, such identifying crucial states in the environment (Faccio et al., (022).
Choreographer can evaluate skill policies in imagination during the tine-tuning stage and use meta- controller to decide which policies should be employed and adapted to the downstream task. While our method exploits a world model for data-efficiens adaptation, alternative strategies such as suc- cessor representations (Hansen et al. (020) or meta RL (Zintgraf et al.) (019) could be considered.
The controller allows evaluating all the skill policies in imagination and fine-tuning them for downstream tasks efficiently, but currently doesn't allow combining the skills for longer-horizon complex tasks. We aim to further investigate this in the future, leveraging hierarchical RI. (Hafner ct al. 2022) and model learning, established in the options framework (Precup 2000).
Choreographer a data-driven approach to skill discovery and learning. The skill representation depends on the data provided or collected from the environment and on the world model learned. As a consequence, the skills learned by Choreographer tend to reach multiple areas of the environment and assume different behaviors, but struggle to capture the naturalness that humans develop in their skills. We leave the idea of incorporating human bias in the skill learning process for future work.
Limitations.

---

Published as a conference paper at ICLR 2023
Reproducibility statement We reported in appendix the detailed pseudo-code for both Choreog- rapher (Algorithm{2 and code-resampling (Algorithm [). All the instructions and training hyperpa- rameters implement and reproduce the results in Appendix B along with a summary of other experiments we tried that could be useful for future work. The code is publicly available through the project website
ACKNOWLEDGEMENTS
This research received funding from the Flemish Government (AI Research Program). Pietro Maz- zaglia is funded by Ph.D. grant of the Flanders Research Foundation (FWO).
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. ArXiv, abs/1807.10299, 2018.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning at the edge of the statistical precipice. Advances in Neural Informa- tion Processing Systems, 34, 2021.
Anurag Ajay. Aviral Kumar, Pulkit Agrawal, Sergey Levine, and Ofir Nachum. Opal: Of- fline primitive discovery for accelerating offline reinforcement learning, 2020. URL https:
13611
David Arthur and Sergei Vassilvitskii. k-means+4: The advantages of careful seeding. Technical report, Stanford, 2006.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. D. Lee, M. Sugiyama, U. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, 2016.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation, 2013. URL abs/ abs/1308 1005.138... 3432
Yuri Burda, Harrison Edwards, Amos J. Storkey, and Oleg Klimov. Exploration by random network distillation. ICLR, 2019.
Víctor Campos, Alexander Trout, Caiming Xiong, Richard Socher, Xavier Giro Nieto, and Jordi Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In ICML, 2020.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In SSST EMMLP. 2014.
Daniela Corbetta. Perception, action, and intrinsic motivation in infants" motor-skill develop- ment. Current Directions in Psychological Science, 30(5):418443) 2021. doi: 10.1
09637214211031139,
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In /CLR, 2019.
Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. The information geometry of un- supervised reinforcement learning, 2021. URL
Francesco Faccio, Aditya Ramesh, Vincent Herrmann, Jean Harb, and Jürgen Schmidhuber. General policy evaluation and improvement by learning to identify few but crucial states, 2022. URL
https://arxiv. org/
01566
Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier. Hierarchical skills for efficient exploration, 2021. URL
10809
10

---

Published as a conference paper at ICLR 2023
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control, 2016.
URL
07507
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay pol- icy learning: Solving long-horizon tasks via imitation and reinforcement learning, 2019. URL
https://arxat
org/ /tt.////11000 11956
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems, Curran Associates, Inc., 2018.
Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. Soft actor-critic: Off Off-policy maxi- mum entropy deep reinforcement learning with a stochastic actor. In ICMI, 2018
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In ICLR, 2019.
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In ICLR, 2021.
Danijar Hafner, Kuang-Huei Lee, lan Fischer, and Pieter Abbeel. Deep hierarchical planning from pixels, 2022. URL /ttpttt::///// http:///axxivv.rr.. abs/ 2206.04164
Steven Hansen, Will Dabney, Andre Barreto, David Warde-Farley, Tom Van de Wiele, and Volodymyr Mnih. Fast task inference with variational intrinsic successor features. In Interna- tional Conference on Learning Representations, 2020.
Steven Stenberg Hansen, Guillaume Desjardins, Kate Baumli, David Warde-Farley, Nicolas Heess, Simon Osindero, and Volodymyr Mnih. Entropic desired dynamics for intrinsic control. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, 2021.
Elad Hazan, Sham M. Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. ArXiv, abs/1812. 2019.
Lukasz Kaiser, Aurko Roy, Ashish Vaswani, Niki Parmar, Samy Bengio, Jakob Uszkoreit, and Noam Shazeer. Fast decoding in sequence models using discrete latent variables, 2018. URL https:
//aaxii..rr//as///e80 03382
Jackyeom Kim, Seohong Park, and Gunhee Kim. Unsupervised skill discovery with bottleneck option learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp.
PMLR, 18-24 24 Jul 2021. URL https: https://proceedingg. www.innnssmmm......mmm.mmm..mmmm pttpssss./..//
kim21 11.htmllllm.
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes, 2013. URL https:
7/arxiv. orq/ abs 13121611166 6114
Thomas Kipf, Yujia Li, Hanjun Dai, Vinicius Zambaldi, Alvaro Sanchez-Gonzalez, Edward Grefen- stette, Pushmeet Kohli, and Peter Battaglia. Compile: Compositional imitation learning and execution, 2018. URL httthttss://aa// http:sss//aax....o//aab///////// www./oogg////aoooooo 01483)
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbecl. Urlb: Unsupervised reinforcement learning benchmark, 2021. URL
https: http:///aaxxi..rrgaa//////0 wwtgg//b////a.
Michael Laskin, Hao Liu, Xue Bin Peng, Denis Yarats, Aravind Rajeswaran, and Pieter Abbeel. Cie: Contrastive intrinsic control for unsupervised skill discovery, 2022. URL https://arxiv.
org/abs/2208 00161
Yann LeCun, Patrick Haffner, Léon Bottou, and Yoshua Bengio. Object recognition with gradient- based learning. In Shape, contour and grouping in computer vision, pp. 319-345. Springer, 1999.
11

---

Published as a conference paper at ICLR 2023
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching, 2019. URL https:://axx....otttt
abs/ abs/1906 05274
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), ICLR, 2016.
Hao Liu and P. Abbeel. Aps: Active pretraining with successor features. In ICML, 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 18459-18473. Curran Associates, Inc., 2021b.
Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, and Pierre Sermanet. Learning latent plans from play, 2019. URL https:://axii....rgg//e/oe/////e org/ab///
19C3.01973
E. C. Marcinowski, E. Nelson, J. M. Campbell, and G. F. Michel. The Development of Object Construction from Infancy through Toddlerhood. Infancy, (4(3):368-391) 2019.
Pietro Mazzaglia, Ozan Çatal, Tim Verbelen, and B. Dhoedt. Curiosity-driven exploration via latent bayesian surprise. ArXiv, abs/2104.07495, 2021.
Russell Mendonca, Oleh Rybkin, Kostas Daniiidiss, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models, 2021a. URL
09514
Russell Mendonca, Oleh Rybkin, Kostas Daniilidis, Danijar Hafner, and Deepak Pathak. Discover- ing and achieving goals via world models. In Advances in Neural Information Processing Systems, 2021b.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, 2013.
Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli. Task-agnostic exploration via policy gra- dient of a parammmaettiic state entropy estimate, 2020. URL /Rllittps.///axxi.ccrr////t
2007 04640)
Juan José Nieto, Roger Creus, and Xavier Giro-i Nieto. Unsupervised skill-discovery and skill- learning in minecraft, 2021. URL https: wtgg/bs//////////// 2ttp://110.008888888
OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojeiech Zaremba, and Lei Zhang. Solving rubik's cube with a robot hand, 2019. /tLLtttss////xii......
abs/19100 07113
Simone Parisi, Victoria Dean, Deepak Pathak, and Abhinav Gupta. Interesting object, curious agent: Learning task-agnostic exploration, 2021. URL https: / www.aaxiiv.... wwtp://..ggaab........ea/ 1111191119
Seohong Park, Jongwook Choi, Jackycom Kim, Honglak Lee, and Gunhee Kim. Lipschitz-
constrained unsupervised skill discovery, 2022. URL htttp:://arxiv..rgg// http:/wwwwrg//b///222..... 00914
Deepak Pathak, Pulkit Agrawal, Alexei Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. ICML, 2017.
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In ICML, 2019.
Karl Pertsch, Youngwoon Lee, and Joseph Lim. Accelerating reinforcement learning with learned skill priors, 2020. URL /ttp://s///aaa...or////////01000
12

---

Published as a conference paper at ICLR 2023
Karl Pertsch, Youngwoon Lee, Yue Wu, and Joseph J. Lim. Demonstration guiddd reinforcement learning with learned skills, 2021.
Doina Precup. Temporal abstraction in reinforcement learning. Amherst, 2000.
University of Massachusetts
Sai Rajeswar, Pietro Mazzaglia, Tim Verbelen, Alexandre Piché, Bart Dhoedt, Aaron Courville, and Alexandre Lacoste. Unsupervised model-based pre-training for data-efficient control from pixels, 2022. /tt.ttsss//axx...oooom
12016
12016
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, 2019. Utt/attss: http://////a......o/ooob///b////
00446
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autoencoders, 2018. URL httttts::/// httpa/////aaaiio.oo//////////// 11063
Christoph Salge, Cornelius Glackin, and Daniel Polani. Changing the enviromment based on empow-
erment as intrinsic motivation. Entropy, 1665):778--28) 19, may 2014. doi: 00.3390/e60022780
URL httpss//ool.orgg000330000000 339002000330902e11111151 60527777
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1312-1320, Lille, France, 07-09 Jul 2015. PMLR.
J. Schmidhuber. Curious model-buildngg systems. In /Proceedings) 1991 IEEE International Joint Conference on Neural Networks, pp. 1458-1463 vol.2, 1991. doi: 10.1109/UCNN.1991. 170605.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1777..0344,
Ramanan Sekar, Olch Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, 2020.
Tanmay Shankar, Shubham Tulsiani, Lerrel Pinto, and Abhinav Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2020. URL htkLLhttps:::/peereeeeee
/ forum? HorrkggYYNNWW
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awan unsupervised discovery of skills. In International Conference Learning Representations, 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiisser, Ioannis Antonoglou, Veda Panneershelvam, and Mare Lanctot. Mastering the game of go with deep neural networks and tree search. nature, 2016.
Avi Singh, Huihan Liu, Gaoyue Zhou, Albert Yu, Nicholas Rhinehart, and Sergey Levine. Parrot: Data-driven behavioral priors for reinforcement learning. 2020.
00s/2011.10022
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):301322) 2003.
Laura Smith, Ilya Kostrikov, and Sergey Levine. walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning, 2022. URL
wrwpiiiorrrgae abs/
7228807885
Jost Tobias Springenberg, Karol Hausman, Martin Riedmiller, Nicolas Heess, and Ziyu Wang. Learning an embedding space for transferable robot skills. 2018.
Aravind Srinivas and Pieter Abbeel. Unsupervised learning for reinforcement learning. 2021. URL https://icm wwt.mmccccccaaaaammmmmmm h /1043 _
13

---

Published as a conference paper at ICLR 2023
Richard S. Sutton. Dyna, integrated architecture for learning, planning, and reacting. SIGART Bull., 2(4):160-163, jul 1991. ISSN 0163-5719. doi: 10.1145/1223441122577 URL https:
(/60..00g110.115/1223333 122377
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. Timothy P. Lillicrap, and Martin A. Riedmiller. Deepmind control suite. CoRR, abs/1801.00690, 2018.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL 0tt.:/ee0..0000eeee0000000020222 72480177666000002222332332222222 http:///ps....pppr//////////
http://ppcceeeees...ee. pdf
Xiaofei Wang, Kimin Lee, Kourosh Hakhamaneshi, Pieter Abbeel, and Michael Laskin. Skill preferences: Learning to extract and execute robotic skills from human feedback, 2021. URL
05382
Ronald J. Williams. Simple statistical soadient-following algorithms for connectionist reinforce-
ment learning. Machine Learning, 8(3):22--56, May 1992. ISSN 1573-0565. doi: 10.1007/
BF00992696. BRLLhtts::// /ttp://ooi.oooo orgg 10
Kevin Xie, Homanga Bharadhwaj, Danijar Hafner, Animesh Garg, and Florian Shkurti. Latent skill planning for exploration and transfer, 2020. 13897
Denis Yarats, Rob Fergus, Alessandro Lazarie, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, P. Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. ArXiv, abs/2201.13423 2022a.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous trol: Improved data-augmented reinforcement learning. In ICLR, 2022b.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-worid: A benchmark and evaluation for multi-tasi and meta reinforrememttteeeeeee learning. In Conference Robot Learning (CoRI.,, 2019. URL httptpp://axxiv..rrggoo
10897
Luisa Zintgraf, Kyriacos Shiarlis, Maximilian Igl. Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep rl via meta- learning, 2019. http:t/s: wrg 08348
14

---

Published as a conference paper at ICLR 2023
APPENDIX
A DATA COLLECTION
In previous skill discovery methods (Eysenbach et al. 2019 Liu & Abbeel 2021a Achiam et al,, 2018), skills are primarily used means to discover the environment by maximizing the diversity of skills. In contrast, Choreographer's ' skill discovery is exploration-annen
in that discovers and learns skills using its model. For this reason, Choreographer can be adopted in ways:
using pre-collected dataset used to learn the world model and skill-related components; using exploration policy, which aims collect diverse data to pre-train.
When not using pre-collected data, we learn an exploration actor-critic in imagination that maxi- mizes intrinsic rewards (Mendonca al., 2021b Sekar et al. 2020). In particular, we shape rewards following LBS (Mazzaglia la et al. 2021) I), which has proven to be a reliable exploration method when using world models (Rajeswar et al. 2022). Exploration rewards approximate the information gain between observations and model states, computed by measuring the KL divergence between the posterior and the prior of the model. The exploration actor-critic is defined as follows:
Expl. actor: (ee))())(8)),
Expl. critic: "esp((*));
(9)
Texpl = DKL DKi.lg
a)-1)..
B IMPLEMENTATION
B.1 AGENT
World Model. We follow the architecture of the DreamerV2 agent's world model (Hafner et al, 2021I. Model states have both a deterministic and stochastic component: the deterministic com- ponent is the 200-dimensional output of a GRU (Cho et al. 2014). with 200-dimensional hidden layer; the stochastic component consists of 32 categorical distributions with 32 classes each. For states-input experiments, the encoder and decoder are 4-layers MLP with dimensionality of 400. For pixels-based experiments, the encoder and decoder follow the architecture of DreamerV2, taking 64 x 64 RGB images as inputs.
Skill auto-encoder. The encoder and decoder networks are 4-layer MLP with a dimensionality of 400. The input and target of the auto-encoder are the deterministic part of the world model states.
The codebook consists of 64 codes of dimensionality 16. The gradients of the skill auto-encoder objective are back-propage propppadee through the decoder and the encoder using straight-throug! gradients
(engiio et al. al.] (013)). The codes of the codebooks are updated using an exponential moving average of their assigned embeddings (Razavi et al. 2019). Code resampling happens every M 200
training batches.
Both the world model and the skill-autoenoderr networks are updated sampling batches of 50 quences of 50 timesteps, using Adam with learning rate 3 10-4 for the updates, and clipping gradients norm to 100.
Skill actor-eritic. Skill actor-critic takes as input both model states and skill vectors (Schaul et al, 2015). We use the Dreamer algorithm to update them in latent imagination (Hafner et al. 2019)) imagining sequences of 15 steps with the model, starting from the world model training batch states. The actor and critic networks are instantiated as 4-layer MLP with dimensionality of 400 and updated using Adam with a learning rate of 8- 10-5 and gradients clipped to 100. The actor's output a truncated normal distribution and its gradients are backpropagated using the reparameterization trick Kingma& & Welling (2013).
Rewards are computed in latent imagination using:
K
Takin (8, 2)
K-NN - D()) - 8
K
i=
Tznimpy
15

---

Published as a conference paper at ICLR 2023
For the entropy term, where we approximate particle-basee based entropy estimator (Singh al. 2003
we follow Liu & Abbeel (2021b) in averaging the distance from the K K-nearest neighbors (K = 30) and remove the log. For the code term, we take the L2-norm directly instead of the squared L2-norm.
With these minor modifications, the two terms are on the same scale and we found it unnecessary to add any scaling coefficient to balance the two reward terms.
Meta-controller actor-critic. The meta-controller actor-critic is instantiated and trained in the same way as the skill actor-critic, using the environment rewards provided by the model's reward predictor. While the meta-controller is updated using policy gradients (Williams 1992). we can backpropagate the meta-controller loss to the skill actor-critic to fine-tune them, with the reparameterizatioon trick.
One issue we found is that when the rewards are too small (or zero) to predict, the predictions of the reward predictor and the critic are noisy around zero, and tend collapse the skill behaviors during fine-tuning, In order to avoid that, we introduce reward smoothing mechanism, which makes sure the critic and reward predictions are exactly zero, until a reward 2 1-10-* 4 is found. By doing so, we ensure that, until an actual reward is encountered, the skill policies remain the ones learned during pre-training and can be used to explore the environment, looking for rewards. To provide consistent exploration, skills are sampled and followed for an entire trajectory, until rewards are encountered.
B.2 EXPERIMENTS
URLB. The URL benchmark (Laskin et al., 22221) is designed to compare different URL strategies. The experimental setup consists of two phases: a longer data collection/pre-trainii phase, where the agent has up to 2M environment steps to interact with the environment without rewards, and shorter fine-tuning phase, where the agent has 100k steps to interact with a task-specific version of the environment, where it should both discover rewards and solve a downstream task. In the URLB, there are three domains from the DM Control Suite (Tassa et al. 2018), Walker, Quadruped, and Jaco, with four downstream tasks solve per each domain.
For state- based inputs, we used an action repeat of 1. For pixel-based inputs, we used an action repeat of 2. For the based experiments, we note that the Jaco environment in ExORL slightly differs from the one in URLB, as the position of the target is not provided to the agent in EXORL. Other than that, the Jaco tasks remain the same and able to find the targets without information about their position, performing comparably to the top- performing baselines.
that, nonetheless, Choreographer is
In both URLB settings, all the agent's ' components are updated once every 10 actions (5 steps, when considering action repeat of 2). This is in line with the baselines from (Rajeswar al. (022). For pre- treiirrinnng this amounts to a total of 200k update steps in both settings, which are performed using a fixed dataset in the offline URLB from states experiments, and with a replay buffer growing over time, in the parallel exploration URL. from pixels experiments. The same update frequency is used during the fine-tuning stage.
For the sparse-rewards Jaco, we consider the target reached when the agent finds a reward 2 1-10-4.
Meta World. In MetaWorld, the task can be adapted to be sparse, by setting rewards equal to the success metrics in the environment. For the reach tasks, success is when the agent reaches the target. In order to build the three goal sets, we adapted the ccod to sample 50 goals the intervals: goal = (-0.45, 0.3, 0.1), goal-high
(0.45, 0.9, 0.5). with seeds 21,
For both sparse experiments, the agents are pre-trained on 2M frames collected using their explo- ration policy (Choreographer uses LBS detailed in Appendix AA. Then, the pre-trained behavior is deployed for evaluation, without any information about the goals to find. For the skill-based approaches, which are DIAYN, APS and Choreographer, skills randomly re-sampled every 50 steps.
B.3 OTHER THINGS WE TRIED
Skill Discovery. We tried alternative representations other than the skill auto auto-encoder, for skill discovery. In particular, we tried contrastive learning and soft clustering using the Kankhorn-Knopp algorithm (Yarats et al. (021). While both of them worked, we found the skill vectors found by the autoencoder to be more diverse and explainable (thanks to the reconstruction). We also tried: (i)
16

---

Published as a conference paper at ICLR 2023
using standard discrete VAE, using trriightthhough gradients (Bengio et al. 2013), compared to a VQ-VAE, but the training was more unstable; (ii) using a VAE with a gaussian parameterization, discovering skills in a continuous latent space, but we found this configuration to perform worse adaptation time, as exploring the environment using the large skill space and adapting a potentially infinite number of skills with the meta-controller used to require more data/training.
Skill learning. We attempted to learn different numbers of skills, ranging from 16 up to 512. We found that the best performance was obtained when using a number of skills comprised between 64 and 128, and we stick to 64 for our experiments. Too few skills were sometimes unable to find useful behaviors in the Quadruped or to find sparse targets the Jaco tasks. Too many skills, under the same computational budget, generally led to skills that didn't tt fully converge or were harder to fine-tune. Though, likely, with higher computational budget, more skills might perform better.
Skill adaptation. The meta-controller generally samples skills at a time-frequency of 1. Other works have focused on learning skills that last several timesteps (Sharma et al. 2020), using them for hierarchical control (Gehring al.] 2021). Choreographer, instead, tends to evaluate all skills in imagination and narrow down its choice to a restricted set of skills that are adapted for the down- stream tasks. We found that sampling skills with lower frequency (we tried every 3,5,15,50 steps) does not lead to better adaptation performance, but often does not harm performance either. We be- lieve that leveraging the skills for hierarchical problems is interesting direction for future work.
B.4 CONSIDERATIONS ABOUT EFFICIENCY
Data efficiency. Choreographer is method that allows for greater data-efficiency, after pre-training a model and skills from unsupervised data. We show that the agent achieves state-of-the-are perfor- mance when fine-tuned for 100k steps the URLB settings, both from pixels and states input.
Parameter efficiency. Choreographer is a model-based agent, training a world model of 20M parameters for pixel-based environments and ~8M parameters for state-based environments. For the skill components, a skill autoencoder of ~ IM parameters and skill policies requiring 22 parameters for their actor-critic are trained. During fine-tuning, the meta-controller requires an additional 2M parameters for its actor-critic.
Other skill learning approaches tend to use similar magnitude of parameters for their skill com- ponents, e.g. APS and DIAYN require ~1M parameters each for their networks, CIC requires 4M parameters. When these techniques are used in a model-free fashion, as the state-based URLB base- lines (Laskin et al. 2021 (022). Choreographer number of parameters can be slightly higher because of the world model and the m When these techniques are used in model-based fash- ion, the pixel-based URLB baselines (Rajeswar et al. 2022). the only additional parameters for the meta-controller.
Computation efficiency. In the URLB paper, they claim training time of about 12 hours for 2M frames pre-training using V100 GPUs for their model-free agents. Using the same GPUs, we can pre-train a Choreographer agent in about 44 hours. The main slowdowns GRU network in the world model, which needs to process sequences of inputs sequentially, both during model training and in the imagination process, and (ii) the imagination process, which is executed in parallel with the model training.
due to: (i) the
C CODE RESAMPLING
In RL, mostly face datasets that grow arbitrarily over time, the agent collects new data. These datasets may become large (in our case, up to 2- 200 environment transitions) and thus require a long time to run clustering algorithms, such k-means. Furthermore, as the data increases, the algorithm may need to be run again, several times, to comply with the new data.
By using VQ-VAE, we are able perform clustering of the model state space, as well as decode the cluster centers into their corresponding model states (which is necessary for Choreographer skill learning objective). However, VQ-VAE suffers from index collapse.
2 As quick overview, the model-free DrQ-v2 agent Yaarats et al. 2022b) requires about 7M parameters for pixel-based inputs and DDPG (Lillicrap al. 2016) requires 3M parameters for state-based inputs.
17

---

Published as a conference paper at ICLR 2023
We propose code resampling technique that is inspired by the k-means++ initialization (Arthur & Vassilvitskii 2006). This technique is generally applied with fixed datasets to reduce the k-means convergence time, by providing better initialization. Instead, use the code re-initialization procedure to "resample" codes every training batches, making sure that any collapsed index/code reactivated.
Our technique scales well for the RL setting, as it uses only the data in the previous batch to resample the code. Of course, this may lead to suboptimal code choice, which can then be adjusted following code resampling iterations, if necessary. We showed in Figure that in practice all the codes tend to stay active over training.
Algorithm 1: Code Resampling
if train_step mod M -- 0:
for code in inactive_codes:
distance_matrll
aenorm_distane (eurrent_batch, curzent_codes)
probabilities = min (distance catinnnce_arriiiiiiii axis=code_axis,
sum (min (chstance__mttriiee.e cxis=code_axxis)
code current [eaatchhsaappa [sample index (probs--robbbbitttee)
investigate the robustness of our code resampling technique the hyperparameter M, i.e. "resample every M train steps". In the experiment, we use codebook dimensionality of 512 and train the agent for 2M steps. We found that the reconstruction loss eventually converges to nearly the same quantity for different values of M. However, using value of M of 10 or 100 leads to lower loss earlier and lower percentage unused codes, compared to using larger values (1000 10000). We also experimented with value of M 1 but we find instabilities during training, because of the excessive resampling. Overall, the technique is robust large range of hyperparameters, but we suggest setting M to lower values if earlier convergence can be useful. For Choreographer, we used 200 as a default hyperparameter.
In Figure 8
Reconstruction loss
0.15
Resemple every M staps
1000
Unused codes (%)
80
40
0.06
70
10"
10a Steps
Steps (k)
Figure 8: Code resampling hyperparameter robustness. The plots show the reconstruction loss
and the percentage of unused codes, whenvarying the hyperparameter M seeds domains).
D ADDITIONAL RESULTS
URLB from states. Table 2, we present detailed results and baselines on the state-based URLB.
Choreographer performs best all the Walker and Quadruped tasks and comparably to CIC in the Jaco tasks. The normalization scores for the results in the paper are the Expert results.
In Figure 3 we showed the results of Choreographer pre-trained with 2M steps of exploratory data from the RND approach. In Figure we compare to the results when pre-training with data collected approaches, i.e. ICM, Disagreement,
by other knowledge-basee the quality of the model and the skill learned, and so the fine-tuning performance depends on the data used. Choreographer is able to learn skills that widely explore the model state space. However, the state space explored is little, then the skills will be also limited to a restricted area. This is what generally happens for the case of the random actions in the Figure, but also in the Quadruped
random actions data. We see that
18

---

Published as a conference paper at ICLR 2023
Pre Tn-iminiinn for X UP civinemnnet viggs
Doussin
Tol
Exper
DCP
Desgrrement
RND
AFT
PasoRL
SMM
DLAYS
Clurortaple
Fap
3338222
41:2216
345213
4:622.7
5642214
036.11
430224
318
4055220
631
=
721
Quairopal
7662243
Walk
230
136-221
7939-37
$16
770147
54
8016
576-84
416
723
Reach Tettom
191
72522
11111
13557
110E5
10105
120
4551
19
76:88
155
137
Jaco
Readh buttom
Keodt lom
203
191
187=18
116:22
12:114
07:09
115:10
106:12
177-7
99366
73232
100
12529
13248
46:11
3623
38-5
19
88:111
08:16
145
155-2.7
147 =
164:7
Keach top right
223
94211
105:11
19977
100
90030
19927
4725
78
76:10
165
309 1.
Table 2: Mean and standard error performance of Choreographer on state-based URLB after pre-
training for 2M steps and fine-tuning 100k steps. Baseline scores from (Laskin et 2022)
for ICM (where we visually inspected the data from ExORL verifying that ICM explores poorly in the Quadruped environment).
(handom))
(freeen
steps
steps
102
Normalized Return (55
Normalized Iteurn
50-
50-
42
30-
30-
22
Quachupedd
Jeca
Dueral
Quacruped
Jeco
Figure 9: Different data. Training Choreogra- pher's world model and skills using data from different exploration strategies.
Figure 10: Zero-shot skill control. We attempt to solve the task using the pre-trained skill poli- cies, without any fine-tuning.
In Figure 11 we present an ablation analysis of our approach when removing the meta-controller, and instead adapting only one of the skills. The skill adapted is selected to be the one with highest expected rewards, considering the states and rewards obtained in the initial 4k steps of the fine-tuning stage. We see the performance remains unchanged the Walker and Quadruped tasks, where the agent mostly requires good initialization master the downstream task quickly. In the Jaco tasks, instead, we see that having a meta-contie controllerl that considers all the skills is still beneficial. We believe this is because the task is sparser and maintaining more skills available helps explore the environment better during fine-tuning. We also highlight that the performance of Choreographer without the meta-controller overall higher than the second-best approach (CIC).
Choreographer
Choneogrupher the meta-ctri
Pre-treining on steps
Nomalized
50
Wi bar
Quadruped
Jeco
Dveral
Figure 11: Offline URLB ablation. Ablations on the meta-controller for the state based URLB experiments.
19

---

Published as a conference paper at ICLR 2023
URLB from pixels. In Figure 4 showed the performance of Choreographer in the pixel-based setting over time. This performance was achieved by learning the meta neta-controller and adapting the skill policies during fine-tuning. In Figure TO show the impact of fine-tuning the skill policies learned during pre-training, after 2M steps. This can be seen as restricting the agent's action space set of N state-dependent actions (=64 skills in this case), which the agent can solve the downstream task.
The performance on the Walker and Quadruped tasks is weaker, but the agent still performs compa- rably to some weaker baselines in Figure 4 such as APS and DIAYN. Instead, the performance on Jaco is much worse than the one with skill adaptation, that's because the task requires precision, to reach the target block, and adaptability. in case the block moves.
In Figure 12 we present two ablation analyses of our approach: removing code resampling, so that some of the skills trained are meaningless because of index/code collapse, and removing the meta-controller, adapting instead only one of the skills, selected after the initial 4k frames of the fine-tuning stage. We find that Choreographer always improves performance with respect to its ablations. particular, both using the meta-controller and making sure that all the skills are active and properly learned, seem to be useful in the Walker tasks, especially in the first 500k steps, and in the Jaco tasks, at all fine-tuning times.
Chorep
Chotoo muta-cil)
Chures (ru codi resamppll
Choteo
Chotcs
Charoo (ro coca resample)
Pre-training on 100k steps
Pre fre-training on 500k steps
Norallze fteuin PR)
Nomalzed Rtetrn 1%
70-
70-
62
62
50-
50
42
10-
Walcer
Quachuped
J6ca
Oviral
Waicer
quacruped
|60a
Overal
Chorao
Choree
Chorma (re
Choreo
Choree
Cherma (re resample)
Pre training 1M
training
100
105
92
Normelzed Return (N))
Normelzed Return (N2
52
60
32
32
20-
28
12
J8c0
Quachiped
Figure 12: Pixel-based URLB ablations. Ablations on the meta-controller and code resampling for the pixel-based URLB experiments.
we show additional results, including more baselines from (Raeewarr et al. the pixel-basee based URLB. Overall, Choreographer elearly outperforms all methods, while LBS still competitive on the Quadruped.
In Figure| 13
2022 on
Comparison with SPIRL. As we discuss in Appendix F previous work on offline skill learning also showed promising performance. However, these methods generally employ demonstrations or play data collected by humans. In contrast, Choreographer is able to leverage vast amounts of unsupervised data. To strengthen this thesis we compare with the SPIRL approach (Pertsch et al.) 2020) on the state-based URLB. SPIRL learns different skill policies from offline data autoencoding trajectories in the given dataset, i.e. learning to imitate trajectories in the data through the skills. Then, during adaptation, the skills are combined using a high-level policy that decides which skill should be applied for fixed time horizon. This policy is also regularized by high-level skill prior
20

---

Published as a conference paper at ICLR 2023
Pre-training.om steps
Pre-training steps
Nerm alteeleeed Return 156)
Norallizd ieturn
50-
50
00
40
-0
Welcer
Quatraped
Jaca
Oueral
Quadruped
Jeen
Charecgrapter
APS
KN
RND
Chonesgrapher
APS
KCM
AND
DANI
APT
PandExptore
LBS
DWNN
AFT
PanDTupptrre
LBS
Pre training
1M stops
Pro training 2M stops
100
92
9C
Normel bed Peetum (%)
88-
Normal bed Return (%6)
70
30
20-
20
10
14
Waleer
Figure 13: Pixel-based URLB. Additional baselines for the pixel-based URLB experiments.
that is learned the offline dataset. In Figure 14 paper hyperparameters. SPIRL struggles to solve all the tasks in URLB leveraging its learned skill action space. We believe the main reason is that skills learned by imitating unsupervised ex exploration trajectories are generally less useful for solving downstream tasks. We experimented with different hyperparameters, such varying the skill horizon, reducing the constraint w.r.t. the skill prior, and increasing the capacity of the neural networks used. While these choices led to some improvements in performance, proving that both skill priors and long-term skills are less successful when em- ploying unstructured data, the performance of SPIRL remained underwhelming in this unsupervised offline setting.
report results obtained using the original
Comparison with model-based EDL. As we detail in Appendix F the Explore Discover and Learn (EDL; (Campos al. 2020)) shares some similarities with Choreographer. In order to show the relevance of the differences between Choreographer and combination of EDL + Dreamer, we run additional experiments on URLB, In these experiments, EDL + Dreamer can be considered an ablation of Choreographer where we: (i) use EDL's objective instead of Chorcographer's 's to learn the skills in imagination, (ii) remove code resampling, (iii) remove the meta-controller. Noting that EDL uses the mutual information between skills and environment states, compared Choreographer which uses the mutual information between skills and latent model states, we have also experimented EDL's objective but with latent states rather than
with Latent EDL configuration, which environment states. Compared to Choreographer, Latent EDL still uses no entropy term in the skill reward, no meta-controller and no code resampling. Both the state-based (Figure 15) and in the mixel-based (Figure 16) the performance of EDL is inferior to Latent EDL, showing that using latent codes is beneficial and Latent EDL inferior Choreographer, showing the usefulness of code resampling and of the meta-controller.
21

---

Published as a conference paper at ICLR 2023
Charengrapper
SPALL
Choneggrapher
Pro-tratring staps
traininnggg steps
100
Normalzed haturn PSI
80
70
50
40
20
20
10
Walker
Quadruped
Figure 14: State-based comparison with SPIRL. Comparison with SPIRI (Pertsch
Figure 15: State-based comparison with
EDL. Comparison with different combinations of EDL (Campos al. 2020) with Dreamer.
al. 2020
EDL
Choreo
EDL
Pre-training 100k
Pre-training
100
Nor moiteeddd Retur (%1
Normalized Retur [%4
52
42
42
30
30
20
Quachuped
1600
Quacruped
Cheros
Laiant SDL
EDL
Cherce
Lacant EDL
EOL
Pre-training on 1M steps
Pre-training on 2M steps
00-
90
Normllized Ieebunn P%)
B2
Normlltzdd Rteturn 1%5)
70-
70
28
10
16-
Walcer
Quadhuped
Overal
Walcer
Quadruped
Overal
Figure 16: Pixel-based comparison with EDL. Comparison with different combinations of EDL Campos et al. 2020) with the Dreamer algorithm.
22

---

Published as a conference paper at ICLR 2023
E
ADDITIONAL VISUALIZATIONSS
Code
T=0
T=10
T=20
T=30
T=40
T=50
T=60
T=70
T=80
T=90
4
Figure 17: Walker. Sets of three skill codes and respective behaviors visualized for Walker.
Code
T-0
T-10
T-20
T-30
T-40
T-50
T-60 60
T-70
T-80
T-90
Figure 18: Quadruped. Sets of three skill codes and respective behaviors visualized for Quadruped.
T=50
Figure 19: Jaco. Sets of three skill codes and respective behaviors visualized for Jaco.
Code
T=0
T=10
T=20
T=30
T=40
T=50
T=60
T=70
T=80
T=90
Figure 20: Meta World. Sets three skill codes and respective behaviors visualized for Meta World.
23

---

Published as a conference paper at ICLR 2023
F ADDITIONAL RELATED WORK
Offiine Skill Learning. Choreographer, by decoupling exploration and skill learning, allows learn- ing skills from offline data that collected in completely unsupervised manner. Previous work has also shown that skill learning from offline data is possible and can be especially useful to quickly adapt to complex long-term tasks (Springenberg et al. 2018; Shankar et al., 2020). The skill repre- sentation can be used to compress sequences of meaningful actions (Pertsch et al. | [Singh et al.)
2020 Ajay al. 2020) or reach designed goals (Gupta al. [019]
10199Lynchh> Lynch et al. (019)) One prob-
lem with these approaches is that they generally require the training dataset to contain sequences of coherent actions, e.g. including demonstrations (Pertsch et al., 2021 Kipf al., 2018), human play
data (Gupta et al. 2019 Lynch al. 2019), incorporating human feedback in the learning process
[Wang et al. 202 Chorcographer, in contrast, makes no assumption about the training dataset: extract set of skills in completely unsupervised manner and learn meaningful behaviors to accomplish them in imagination, using the model.
Detailed comparison with EDL. Explore Discover and Learn (EDL; (Campos et al. 2020 is an unsupervised skill approach that is related to Choreographer, as both methods decouple exploration and skill learning by using a VQ-VAR discover skills from data. However, several differences present between the two methods: (i) Choreographer is a model-based agent, where the model is used discover, learn and adapt skills, while EDL does not include model, (ii) Choreographer's skill learning objective is derived from the mutual information between latent model states and skills, while EDL's objective derived from the mutual information between environment states and skills, (iii) Choreographer's 's skill learning reward is made of acch-acci component and of an entropy term, that is maximized in the imagined trajectories, while EDL has no entropy term as the entropy is considered fixed in the offline dataset trajectories, (iv) Choreographer adopts a meta-controller during line-tuning which allows the agent to evaluate and fine-tune multiple skills at once.
Detailed comparison with LEXA. Latent Explorer Achiever (Mendonca al. 2021a) is ap- proach that combines an exploration policy and a goal achieving policy, both learned in imagination, to reach complex goals in the environment. There are some similarities between Choreographer and policies in imagina-
LEXA, as they are both model-based agents that learn ceillgooal-condttonn tion, without interfering with the exploration process. There also several differences that can be summarized as follows: (i) Choreographer is designed to work in a setting where, after an un- supervised pre-training phase, the agent is given short fine-tuning phase adapt for a given task, by maximizing rewards, while LEXA is designed to accomplish visual goals in zero-shot setting, after interacting with the environment, (ii) Choreographer learns discrete latent representation and sample codes internally to learn skill policies, while LEXA samples goal images from external replay buffer and encodes them as targets for the achiever policies, (iii) Choreographer's objective is derived from mutual information term and is made of an entropy term and a code-achieving term, while LEXA uses either cosine distance in image embedding space or temporal distance between encoded goals and imagined states, (iv) Choreographer adopts a meta-controller during fine-tuning, which allows the agent to evaluate and tne-tune multiple skills at once.
24

---

Published as a conference paper at ICLR 2023
G ALGORITHM
Algorithm 2: Choreographer
Initialize agent modules: world model, skill autoencoder, skill policies no replay buffer available then Initialize replay buffer.
Initialize exploration policy.
should explore True.
end
while is unsupervised phase do
// Data collection
should should.explore explore then
Update model state with the posterior 8, Sample exploratory action 0e (e)p((e)(8)) Collect new observation from the environment 1241
Add (Ie, ae, 2(+1) to the replay buffer.
end
// Pre-training
if mod Trraineeeer.nn_teeee 0 then
Draw a batch of past trajectories {x,a] from the replay buffer.
Update world model on batch (Eq. 3 and get states {s}.
if shouldo dexpllore then
Imagine exploration trajectories {s, a, sampling actions from Texpl (4g 82).
Compute exploration rewards Texpl (Eq. 9p.
Update exploration actor-critic.
end
// Skill di scovery
Update skill autoencoder using the model states {s} (Eq. 5)
// Skill learning Sample skill codes from uniform distribution 2 p(2).
Imagine skill trajectories (s, a, 2 sampling actions from Taan (a. [t, 2).
Compute skill rewards rsai (Eq. 7)
Update skill actor-critic.
end
end
Fine-tuning
Reset replay buffer and initialize meta-controller.
while supervised phase do
Update model state with the posterior 8, ((11/9111, (,1,
Sample skill 2 (2|ss) and action a; ~ T (o,
Collect new task observation from the environment 2141,
Add (It, at, 2 I the replay buffer.
if f mod train every n sleps = ( then
Draw batch of past trajectories {ar, a. r) from the replay buffer.
Update world model on batch (Eq, 3) and get states {s}.
// Skill adaptation Imagine traj. (s, a, 2). sampling skills from Tmeta Tmeua(2|ss) and actions from Taiil (ar 1st, z). Predict task rewards Tusk
Update meta-controller actor-critic and propagate gradients to skill actors.
end end
25


Reviews:
[{"review_id": "kTC-Z9luSe", "reviewer": "Reviewer_LVu4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Use unsupervised skill learning with mutual information latent space skills, but derive the skill latent space from the latent space of a world model. The world model is represented with a recurrent neural network (GRU), where the internal state of the world model is the recurrent state of the GRU, and trained to match the input data. The skills are encoded and then quantized into a fixed codebook, and unused quantizations (based on the frequency that a code is assigned) are reassigned according to an embedding from the training batch based on the distance from the closest code. The skills are trained with a reward function that combines an entropy term with a likelihood code, and the meta critic and actor chooses skill-latents as actions to optimize task reward. ", "summary_of_the_review": "I recommend acceptance as long as this method of using latent codes from aw world model has not been done in prior work, which appears to be the case. ", "strength_and_weaknesses": "Latent codes based on a world model could be especially sensitive to domains where the world model struggles to generate a consistent state. Is there a way of adding information back for skill learning if it is not captured in the world model? \n\nThe experiments are somewhat weak because the experimental domains have relatively achievable modeling, and the baselines do not really make use of model-based learning. Since Choreographer makes use of imagination in the model, it will have an advantage in sample efficiency without utilizing the latent space innovation. Furthermore, the goal-reaching tasks probably should have utilized some form of hindsight, which would have much improved the baseline.\n\nOne important check would be to ensure that the latent space learned by the choreographer is giving the benefit since it appears to be the key innovation of this work. As it is, it seems like the latent code could be learned from the base state instead of the latent space of the model to get the same results, as long as the additional code resampling and learning in imagined states were implemented.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eHWu6lZKVT", "reviewer": "Reviewer_2p4W", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an unsupervised skill discovery algorithm based on state clustering in a latent space. Specifically, assuming that exploratory data are given (either by an offline dataset or by a separate exploratory policy), their proposed method learns a latent world model and clusters the latent states into $N$ discrete components using VQ-VAE. It then learns skill policies to reach the clustered states by minimizing the $\\ell_2$ distance in the latent space. The paper demonstrates that their method combined with a hierarchical controller outperforms existing skill discovery methods on DM Control and MetaWorld environments.", "summary_of_the_review": "The paper demonstrates that Choreographer can learn skills from data in pixel domains and they can be effectively adapted to downstream tasks. However, given its limited novelty and insufficient empirical comparisons with other methods, I cannot recommend acceptance at this point.\n\n**[Post-rebuttal update]** I thank the authors for the detailed response. As my initial concerns regarding the experimental results are mostly addressed, I raised my score to 6.", "strength_and_weaknesses": "Strengths\n- The paper successfully demonstrates that the idea of decoupling exploration and skill discovery can be scaled to pixel-based environments.\n- The proposed code resampling technique for VQ-VAE seems sensible and effective for preventing code collapse.\n- The paper is well-written and easy to follow.\n- The authors include some failed experiments with helpful insights in the Appendix.\n\nWeaknesses\n- One of the main weaknesses of this paper is its novelty. Their proposed method is essentially running EDL (Campos et al., 2020) on top of a discrete latent world model (Hafner et al., 2021). The notion of decoupling exploration and skill discovery is also previously suggested by Campos et al. (with the same clustering method of VQ-VAE). While their code resampling technique for preventing code collapse in VQ-VAE seems to some degree novel and sensible, it does not appear to be the primary contribution of the paper, and the degree to which it affects the final performance on downstream tasks is not discussed.\n- The comparisons in the main results (Fig. 3, Table 2, and (possibly) Fig. 4 as well) seem to be not fair. They directly take the reported results from the CIC paper for baselines. However, the other skill discovery methods in the CIC paper are fine-tuned without a hierarchical controller, unlike Choreographer. How well does Choreographer perform with the same fine-tuning configuration without a controller (or the other way around)?\n- In the pixel-based URLB result (Fig. 4), some of the baselines (e.g., CIC) are missing compared to Fig. 3. Also, if the numerical results are directly taken from Rajeswar et al., these comparisons could also be unfair due to the absence of hierarchical controllers for the other methods.\n- It would be better if the authors can provide additional comparisons between Choreographer and LEXA (Mendonca et al., 2021), a similar skill discovery method that also uses a world model with separated exploration.\n- The code is not publicly available.\n\nMinor questions and suggestions\n- How many training epochs (and environment steps) are used for the pre-training/fine-tuning of Choreographer, respectively?\n- Appendix: $3e^{-4} \\to 3 \\cdot 10^{-4}$, etc.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "dD6yVsPwjwk", "reviewer": "Reviewer_g6c7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a model-based unsupervised skill discovery method that discovers and learns skills in the latent space using a trained world model rather than in the environment using raw observations. During the training, they keep a skill codebook equipped with a VQ-VAE for the skill encoding and propose the code resampling technique to prevent the index from collapsing. They learn skills using the mutual information maximization objective where the entropy is estimated using the particle-based estimator. They show the empirical results that the proposed method performs competitively with existing skill discovery methods.", "summary_of_the_review": "The proposed method is somewhat limited in terms of the algorithmic novelty, but it is generally applicable, which is also supported by their empirical results. The overall quality of the work, including the experimental results and the presentation, is good. There are some weaknesses, including the lack of analysis of the method's robustness to the new hyperparameter and discussion regarding the training cost compared to prior skill discovery methods.", "strength_and_weaknesses": "Strengths\n\n- The proposed method is well-defined and quite general (e.g., the applicability to both state and visual observation environments, the VQ-VAE with the code resampling, etc.).\n- The empirical evaluation and analyses, including the performance plots and tables, and qualitative results are extensive and informative.\n- The high-quality presentation, including the writing and the intuitive figures (e.g., Fig.1 and Fig.2).\n\nWeaknesses\n\n- The work is novel to some degree, but algorithmic novelty of the work is not very strong.\n- The proposed code resampling introduces an additional hyperparameter $M$, and the robustness of the method to the choice of $M$ is not examined.\n- The increased burden in the size of the models (and thus the memory) and the computation in comparison with prior skill discovery methods is not discussed.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XTo2fhWK-d", "reviewer": "Reviewer_eUdn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents Choreographer, a method for learning skills in the imagination of a world model.  The method uses mutual-information maximization between skill codes and latent states in a Dreamer-V2 world model.  Since the skills are learned in imagination, the data collected from the environment needs only be used for training the world model, and so Choreographer can be applied to offline RL datasets.  Once the world model and skills have been learned, Choreographer next learns a controller that uses the skills as an action space to adapt to downstream tasks.  Empirical results show that Choreographer compares favorably to other unsupervised RL methods on the URL Benchmark.  ", "summary_of_the_review": "This is a strong paper that presents an interesting extension of skill learning and model-based approaches.  Although the contribution is mostly a combination of other existing work, the combination is interesting and non-trivial, and the results are strong.", "strength_and_weaknesses": "Choreographer is an interesting extension and combination of skill-learning algorithms and world modeling imagination approaches.  Although the combination of two existing approaches is not completely surprising, it is a novel combination as far as I am aware.  And, the method performs well against some strong baselines.  Additionally, there were some non-obvious contributions regarding code resampling in the VQ-VAE component of the world model.  Overall the paper is good.  My main concern with the paper is comparison to other methods for offline skill discovery.  For example, [Ajay et al. OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning, ICLR 2021] uses a VAE to encode offline trajectories and learns skills from that data as well.  Although there are methodological differences, it would be nice to see a comparison to similar offline skill learning approaches.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%