Dynamics-Aware Unsupervised Discovery of Skills
Authors: Archit Sharma
Shixiang Gu
Sergey Levine
Vikash Kumar
Karol Hausman
Venue:  (2020)
Keywords: reinforcement learning
unsupervised learning
model-based learning
deep learning
hierarchical reinforcement learning

Abstract: Conventionally, model-based reinforcement learning (MBRL) aims to learn a global model for the dynamics of the environment. A good model can potentially enable planning algorithms to generate a large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynamical systems is difficult, and even then, the model might not generalize well outside the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model-based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsupervised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demonstrate that zero-shot planning in the learned latent space significantly outperforms standard MBRL and model-free goal-conditioned RL, can handle sparse-reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery.

Published as a conference paper at ICLR 2020
DYNAMICS-AWARE UNSUPERVISED DISCOVERY OF SKILLS
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, Karol Hausman
Google Brain
(architsh, shanegu, s slevine, wwweeiiiissseee
wgwpa//sssmmmmnnggleeeoooo.ooe.e com
ABSTRACT
Conventionally, model-based reinforcement learning (MBRL) aims learn a global model for the dynamics of the environment. A good model can poten- tially enable planning algorithms to generate large variety of behaviors and solve diverse tasks. However, learning an accurate model for complex dynami- cal systems is difficult, and even then, the model might not generalize well out- side the distribution of states on which it was trained. In this work, we combine model-based learning with model-free learning of primitives that make model- based planning easy. To that end, we aim to answer the question: how can we discover skills whose outcomes are easy to predict? We propose an unsuper- vised learning algorithm, Dynamics-Aware Discovery of Skills (DADS), which simultaneously discovers predictable behaviors and learns their dynamics. Our method can leverage continuous skill spaces, theoretically, allowing us to learn infinitely many behaviors even for high-dimensional state-spaces. We demon- strate that zero-shot planning in the learned latent space significantly outper- forms standard MBRL and model-free goal-conditineed RL, can handle sparse- reward tasks, and substantially improves over prior hierarchical RL methods for unsupervised skill discovery. We have open spen-sourced our implementation at: https::/githi: com cww.coooc/eeeeoeoooooooooooooooe rew.glleeeseercccaada...
Figure 1: A humanoid agent discovers diverse locomotion primitives without any reward using DADS. We show zero-shot generalization downstream tasks by composing the learned primitives using model predic- control, enabling the agent to follow an online sequence of goals (green markers) without any additional training.
INTRODUCTION
Deep reinforcement learning (RL) enables autonomous learning of diverse and complex tasks with rich sensory inputs, temporally extended goals, and challenging dynamics, such as discrete game- playing domains (Mnih et al. 20133] Silver et al. (016). and continuous control domains including
locomotion (Schulman et al.] 2015 Heess et al. 2017) and manipulation (Rajeswaran al. 2017 Kalashnikov et al. 2018 Gu et al. (017). Most of the deep RL approaches learn a Q-function or a policy that are directly optimized for the training task, which limits their generalization to new scenarios. In contrast, MBRL methods (Li & Todorov] 2004. Deisenroth & Rasmussen 2011 Watter et al.] (013) can acquire dynamics models that may be utilized to perform unseen tasks at test time. While this capability has been demonstrated in some of the recent works (Levine et al. 2016 Nagabandi et al. 2018 Chua et al. 2018b Kurutach et al. 2018 Ha & Schmidhuber)
"Work done part of the Google AT Residency program.

---

Published as a conference paper at ICLR 2020
2018). learning an accurate global model that works for all state-action pairs can be exceedingly challenging. especially for high -ighddimensional system with complex and discontinuous dynamics. The problem is further exacerbated as the learned global model has limited generalization outside of the state distribution it was trained on and exploring the whole state space is generally infeasible. Can retain the flexibility of model-based RI., while using model-free RL to acquire proficient low-level behaviors under complex dynamics?
While learning a global dynamics model that captures all the different behaviors for the entire state- space can be extremely challenging, learning a model for a specific behavior that acts only in a small part of the state-space can be much easier. For example, consider learning a model for dynamics of all gaits of a quadruped versus a model which only works for a specific gait. If we can learn many such behaviors and their corresponding dynamics, we can leverage model-predictive control to plan in the behavior space, as opposed to planning in the action space. The question then becomes: how do we acquire such behaviors, considering that behaviors could be random and unpredictable? To this end, we propose Dynamics-Aware Discovery of Skills (DADS), unsupervised RL framework for learning low-level skills using model-free RL with the explicit aim of making model-based con- trol easy. Skills obtained using DADS are directly optimized for predictability, providing better representation on top of which predictive models can be learned. Crucially, the skills do not require any supervision learn, and are acquired entirely through autonomous exploration. This means that the repertoire of skills and their predictive model are learned before the agent has been tasked with any goal or reward function. When a task provided at test-time, the agent utilizes the previously learned skills and model to immediately perform the task without any further training.
The key contribution of our work is an unsupervised reinforcement learning algorithm, DADS, grounded in mutual-information--oe baasdd exploration. We demonstrate that our objective can em- bed learned primitives in continuous spaces, which allows us to learn a large, diverse set of skills. Crucially, our algorithm also learns to model the dynamics of the skills, which enables the use of model-based planning algorithms for downstream tasks. We adapt the conventional model predic- tive control algorithms to plan in the space of primitives, and demonstrate that we can compose the learned primitives to solve downstream tasks without any additional training.
2 PRELIMINARIES
Mutual information can been used as an objective to encourage exploration in reinforcement learning (ouuhhoftt et al. 2016 Mohamed & Rezende, 2015). According to its definition, I(X; maximizing mutual information I with respect to Y amounts to maximizing the entropy H of X while minimizing the conditional entropy H(X YY). In the context of RI. X is usually a function of the state and Y a function of actions. Maximizing this objective encourages the state entropy to be high, making the underlying policy to be exploratory. Recently, multiple works (Eysenbach et al. 2018 Gregor et al. 2016 AAchiam et al. 2018) apply this idea to learn diverse skills which maximally cover the state space.
H(() -H(X
To leverage planning- bassdd control, MBRL estimates the true dynamics of the environment by learn- ing model j0s' 8, a). This allows it predict trajectory of states tn 8 & resulting from a sequence of actions without any additional interaction with the environment. While model-based RL methods have been demonstrated to be sample efficient compared their model- free counterparts, learning an effective model for the whole state-space is challenging. An open- problem in model-based RL is to incorporate temporal abstraction in model-based control, to enable high-level planning and move-away away from planning at the granular level of actions.
These seemingly unrelated ideas can be combined into a single optimization scheme, where we first discover skills (and their models) without any extrinsic reward and then compose these skills to optimize for the task defined at test time using model-based planning. At train time, we assume a Markov Decision Process (MDP) Mi = AS...p). The state space S and action space A are assumed to be continuous, and the A bounded. We assume the transition dynamics p to be stochastic, such that p: S x A x S + [0, oo). We learn a skill-conditioned policy T(a 8, z). where the skills 2 belongs to the space Z, detailed in Section 3 We assume that the skills are sampled from a prior p(z) over Z. We simultancously learn skill-conditioned transition function g(s' s, s,2), coined as skill-dynamics, which predicts the transition to the next state from the current state s for the skill under the given dynamics p. At test time, assume an MDP Ma (S,A, r). where A,A.p
2

---

Published as a conference paper at ICLR 2020
match those defined in M1. and the reward function r: : A (o)0, We plan in 2 using q(ss 8, 2) to compose the learned skills & for optimizing r in M2, which we detail in Section
DYNAMICS-AWARD DISCOVERY OF SKILLS (DADS)
Algorithm 1: Dynamics-Aware Dis- covery of Skills (DADS)
skill
m Gompprattoo
Initialize gs:
skill dynamics qua' 4, 2)
while not converged do
Sample a skill z ~ p(2) every
episode;
q 2)
Collect new M on-policy samples; Update 4e using Ki steps of gradient descent on M transitions;
Fal)
Compute (= a, s') for M
transitions;
agent n(al8,))
envirormment p(s' a,
Update т using any RL algorithm;
end
wieodeie the
Figure 2: The agent x interacts with the environment to produce a transition 8 + s'. Intrinsic reward is computed by computing the transition probability under for the current skill z, compared to random samples from the prior p(=). The agent maximizes the intrinsic reward computed for a batch of episodes, while g maximizes the log-probability of the actual transitions of (s, sss
We use the information theoretic paradigm of mutual information obtain our unsupervised skill discovery algorithm. In particular, we propose to maximize the mutual information between the next state s' and current skill 2 conditioned on the current state s.
I(s': 8) = H(= 8) ((= 88,8)
(1)
Hl(s {8) H(s 8,,
(2)
Mutual information in Equation [ quantifies how much can be known about s' given and s, or symmetrically, given the transition from 8 + s'. From Equation P maximizing this objective corresponds to maximizing the diversity of transitions produced in the environment, that is denoted by the entropy H(s' s), while making informative about the next state s' by minimizing the entropy H(' 8, z). Intuitively, skills z can be interpreted as abstracted action sequences which identifiable by the transitions generated in the environment (and not just by the current state). Thus, optimizing this mutual information can be understood as encoding a diverse set of skills in the latent space 2, while making the transitions for a given E Z predictable. We use the entropy- decomposition in Equation 2 to connect this objective with model-based control.
We want to optimize the our skill-conditioned controller T[a 8, 2) such that the latent space ~ ((2) is maximally informative about the transitions s s'. Using the definition of conditional mutual information, we can rewrite Equation
as:
p(2,8,,8 log p(8' 8,2) dsdd
I(s':
(3)
p(s' ())
We assume the following generative model: p(2, 8') = p(z)p(s 22p(s' s, where p(z) is user specified prior over 2, p(((2) denotes the stationary dite-distribution induced by <a | 8, for a skill z and p(s' s,2) denotes the transition distribution under skill 2. Note, p(s' 8, s,
ppss I s, a)) s, 2) is intractable compute because the underlying dynamics are unknown. However, we can variationally lower bound the objective as follows:
I(s': 8) Eg log p(s' 2)
p(s' |s)
Eg log (s's' 8,2)
Bg DRI(U(S' (8,2) go(s' 8, 8,2))]
p(s' 8)
E Ez log 20 8)
p(s' s)
3

---

Published as a conference paper at ICLR 2020
where we have used the non-negativity of KL-divergence, that is DEL 2 0. Note, skill-dynamies do represents the variational approximation for the transition function p( 8, 8,2), which enables
model-based control as described in Section 4 Equation 4 suggests alternating optimization
between and π,, summarized in Algorithm 1 In every iteration:
(Tighten variational lower bound) We minimize Dx (p(s' 8,=) || gs(s' 8, =)) with respect to
the parameters & on 8,8 p to tighten the lower bound. For general function approximators like neural networks, we can write the gradient for follows:
V log P(s' s.2)
Vo (Pec(P(' 8,2) go(8' |8,=))]
(s |8,2).
- s.
V log q (s' 8,22
(5)
which corresponds to maximizing the likelihood of the samples from p under qø.
(Maximize approximate lower bound) After fitting 96. we can optimize π to maximize Es |log (s(s' s,z) ) - log p(s' s)]. Note, this is a reinforcement-leariigg style optimiza-
tion with reward funetion log (s' ( s, log p(s') s). However, log p(s' intractable to
compute, so we approximate the reward function for n::
90 s,)
re(s, (,, log E (s(s' s.24)
+ log L,
24 ~ p(z).
(6)
The approximation is motivated as follows: p(s' |s) = Sp(s' s, s,2)p(z|s)dz ~ s I
2 2)p(2)dz = Σ/= (s' 5,) for 2, ~ p(2), where L denotes the number of samples from the prior. We are using the marginal of variational approximation qo over the prior p(z) to approx- imate the marginal distribution of transitions. We discuss this approximation in Appendix C Note, the final reward function r= encourages the policy æ to produce transitions that are (a) predictable under go (predictability) and (b) different from the transitions produced under p(z) (diversity).
To generate samples from p(z, 8, 8'), we use the rollouts from the current policy for multiple samples p(2) in an episodic setting for fixed horizon T. We also introduce entropy regu- larization for (a 8, 2). which encourages the policy to discover settunnsequenees with similar state-transe tate-ttrassiiions and to be clustered under the same skill making the policy robust besides en- couraging exploration (Haarnoja et al.] 2018a). The use of entropy regularization can be justified from an information bottleneck perspective as discussed for Information Maximization algorithm in (Mohamed & Rezende (015). This is even more extensively discussed from the graphical model perspective in Appendix which connects unsupervised skill discovery and information bottleneck literature, while also revealing the temporal nature of skills 2. Details corresponding to implemen- tation and hyperparameters are discussed in Appendix A
PLANNING USING SKILL DYNAMICS
Given the learned skills (a 8, and their respective skill-transition dynamics go(5' 8,22,, we can perform model-based planning in the latent space Z to optimize for reward that is given to the agent test time. Note, that this essentially allows us to perform zero-shot planning given the unsupervised pre- pre-training procedure described in Section 3
In order perform planning, we employ the model-ppreiieive--oonro) (MPC) paradigm Garcia al. [188), which in a standard setting generates a set of action plans Ps = (ak,1.... . ~ P for planning horizon H. The MPC plans can be generated due the fact that the planner is able to simulate the trajectory îx = (81,1, ... 8&,H+1) assuming access to the transition dynamics p(s' | a). addition, each plan computes the reward r(fx) for its trajectory according to the reward function that is provided for the test-time task. Following the MPC principle, the planner selects the best plan according to the reward function r and executes its first action 4₁- The planning algorithm repeats this procedure for the next state iteratively until it achieves its goal.
We use a similar strategy to design an MPC planner to exploit previously- learned skill- tillstrannition dynamics go(s' s, 2). Note that unlike conventional model-base RL, we generate plan Pa (Zk,),... 2x. ) in the latent space Z as opposed to the action space AA that would be used by a standard planner. Since the primitives are temporally meaningful, it is beneficial to hold primitive
4

---

Published as a conference paper at ICLR 2020
Algorithm 2: Latent Space Planner
Model Predictive Control Ty
8 spi
dynamics uls
Initialize parameters ...... for <- to Hg/ / H2 do
5,555
for to do
planmer
rocstand
singuetie Deward
Leeas Lei
2,, 2.
N N.... N.....NN
Computc Tenv for {Zz 24+ 2 11
Environment
Update pr..... .it Hp
end
enivtromment p(s' ss
agent m(als,2)
Sample 2, from Me:
Execute [[||s, 2:) Hz steps:
Initialize Hp
end
Figure 3: At test time, the planner executes simulates the transitions environment using skill-dynamics q, and updates the distribution of plans according to the computed reward on the simulated trajectories. After few updates to the plan, the first primitive is executed in the environment using the learned agent æ.
for horizon Hz > 1, unlike actions which are usually held for single step. Thus, effectively, the planning horizon for our latent space planner is H = Hp x Hz, enabling longer- horizon planning using fewer primitives. Similar to the standard MPC setting, the latent space planner simulates the trajectory 4x = ($ 2,,1, ag.1 $4,2, 23.2, 2g, $g, and computes the reward r(fe). After small number of trajectory samples, the planner selects the first latent action 21 of the best plan, executes it for Hz steps in the environment, and the repeats the process until goal completion.
The latent planner P maintains distribution of latent plans, each of length Hr. Each element in the sequence represents the distribution of the primitive to be executed at that time step. For continuous spaces, each element of the sequence can be modelled using normal distribution, (H, Σ). .......(H.... Σ). We refine the planning distributions for R steps, using K samples of latent plans Pk. and compute the r3 for the simulated trajectory îx. The update for the parameters follows that in Model Predictive Path Integral (MPPI) controller Williams et al. (2016):
exp(1Tk) Ep=1 exp(1r,)
[1s
K
Vi 1,...
While we keep the covariance matrix of the distributions fixed, it is possible to update that as well as shown in Williams et al. (2016). We show an overview of the planning algorithm in Figure and provide more implementation details in Appendix
5 RELATED WORK
Central to our method is the concept of skill discovery via mutual information maximization. This principle, proposed in prior work that utilized purely model-free unsupervised RL methods (Daniel et al. 2012 Florensa et al. 2017 Eysenbach et al., 2018 Gregor et al. 2016 Warde-Farley et al.) 2018 Thomas al. 2018). aims learn diverse skills via discriminability objective: a good set of skills is one where is easy to distinguish the skills from each other, which means they perform distinct tasks and cover the space of possible behaviors. Building on this prior work, we distinguish our skills based on how they modify the original uncontrolled dynamics of the system. This simultaneously encourages the skills be both diverse and predictable. We also demonstrate that constraining the skills to be predictable makes them more amenable for hicrarchical composition and thus, more useful on downstream tasks.
Another line of work that is conceptually close to our method copes with intrinsic motiva- tion (Oudeyer & Kaplan 2009; Oudeyer et al. 2007] Schmidhuber 2010) which is used to drive
the agent's exploration. Examples of such works include empowerment Klyubin al. (2005); Mo-
hamed & Rezende (2013), count-based exploration Bellemare et al. (2016); Oh et al. (2015) Tang
al. (2017] Fu al. (2017), information gain about agent's dynamics Stadie al. 2015 and
5

---

Published as a conference paper at ICLR 2020
-nward-inverse dynamics models Pathak et al. 2017 While our method uses information- theoretic objective that is similar to these approaches, is used to learn a variety of skills that can be directly used for model-base planning, which is in contrast to learning a better exploration policy for a single skill.
The skills discovered using our approach also provide extended actions and temporal abstrac- tion, which enable more efficient exploration for the agent to solve various tasks, reminiscent of hierarchical RL (HRL) approaches. This ranges from the classic option-critic architecture (Sutton
et al.
1999 Stolle & Precup, 2002 Perkins et al., 1999) to some of the more recent work (Bacon
Hausman et al.] 2018). However, in contrast to end-to-end HRL approaches (Heess et al., 2016 Peng et al. 2017), we can leverage
al.
2017 Vezhnevets al., 2017 Nachum et al. 2018
stable, two-phase learning setup. The primitives learned through our method provide action and temporal abstraction, while planning with skill-dynamics enables hierarchical composition of these primitives, bypassing many problems of end-to-end HRL.
In the second phase of our approach, we use the learned skill-transition dynamics models to perform model-based planning- an idea that has been explored numerous times in the literature. Model-based reinforcement learning has been traditionally approached with methods that are well-suited for low- data regimes such as Gaussian Processes (Rasmussen, 2003) showing significant data-efficiency gains over model-free approaches (Deisenroth et al. et al. 2004 Ko et al. 2007). More recently, due to the challenges of applying these methods to high- dimensional state spaces, MBRL approaches employs Bayesian deep neural networks (Nagabandi iet al. 2018 Chua et al. 20188) Gal et al. 2016 Fu et al. 2016 Lenz et al. 2015) to learn dynamics models. In our approach, we take advantage of the deep dynamics models that are conditioned on the skill being executed, simplifying the modelling problem. In addition, the skills themselves are being learned with the objective of being predictable, further assists with the learning of the dynamics model. There also have been multiple approaches addressing the planning component of MBRL. including linear controllers for local models (Levine et al.] 2016, Kumar al. 2016
Kamthh & Deisenroth] 2017: Kocijan
Chebotar et al. (0177). uncertainty-aware (Chua et al., 2018b Gal et al.) 2016) or deterministic planners (Nagabandi et al. 2018)) and stochastic optimization methods (Williams et al. 2016). The main contribution of our work lies in discovering model-based skill primitives that can be further combined by a standard model- based planner, therefore we take advantage of an existing planning approach- Model Predictive Path Integral (Williams et al. 2016) that can leverage our pre- pre-trained setting.
EXPERIMENTS
Through our experiments, we aim to demonstrate that: (a) DADS as general purpose skill dis- covery algorithm can scale to high-dimensional problems; (b) discovered skills are amenable to hierarchical composition and; (c) not only planning in the learned latent space feasible, but is competitive to strong baselines. In Section 6.1 we provide visualizations and qualitative analysis of the skills learned using DADS. We demonstrate in Section 6.2 and Section 6.4 that optimizing the primitives for predictability renders skills more amenable to temporal composition that can be used for Hierarchical RL. We benchmark against state-of-the model- based RL baseline in Section 6.3 and against goal-conditioned RL in Section 6.5
6.1 QUALITATIVE ANALYSIS
Figure 4: Skills learned different MuJoCo environments in the OpenAl gym. DADS can discover diverse skills without any extrinsic rewards, even for problems with dimennsioii state and action spaces.

---

Published as a conference paper at ICLR 2020
In this section, provide qualitative discussion of the unsupervised skills learned using DADS. We use the MuJoCo environments (Todorov et al. 2012) from the OpenAl gym as our test-bed learn diverse skills without any
(Brockman et al. 2016). We find that our proposed algorithm reward, even in problems with high-dimensional state and actuation, as illustrated Figure DADS discover primitives for Half-Cheetah to forward and backward with multiple different gaits, for Ant to navigate the environment using diverse locomotion primitives and for Humanoid to walk using stable locomotion primitives with diverse gaits and direction. The videos of the discovered primitives are available at: https: htt.//sitesss. google com/
com/view/ http/viiw////eeee hte//dadssssss
Qualitatively, we find the skills discovered by DADS to be predictable and stable, in line with im- plicit constraints of the proposed objective. While the Half-Cheetah will learn to run in both back- ward and forward directions, DADS will disincentivize skills which make Half-Cheetah flip owing to the reduced predictability on landing. Similarly, skills discovered for Ant rarely flip over, and tend to provide stable navigation primitives in the environment. This also incentivizes the Humanoid, which is characteristically prone to collapsing and extremely unstable by design, to discover gaits which are stable for sustainable locomotion.
One of the significant advantages of the proposed objective is that it is compatible with continuous skill spaces, which has not been shown in prior work on skill discovery (Eysenbach et al. 2018). Not only does this allow us embed large and diverse set of skills into compact latent space, but also the smoothness of the learned space allows us to interpolate between behaviors generated in the envi- ronment. We demonstrate this the Ant environment (Figure|5), where learn two-dimensional continuous skill space with a uniform prior over (-1, 1) in each dimension, and compare it to a dis- crete skill space with uniform prior over 20 skills. Similar [Eysenbach al. (2018), we restrict the observation space of the skill-dynamics to the cartesian coordinates (x, V). We hereby call this
the x-y prior, and discuss its role Section 6.2
Trajectories in Discrete Skill Spuce
Trajectories in Continuous Skill Space
Orientation Ant Trajectory
Latome Direnttin
60
-20
Latent Dimension
Figure 5: (Left, Centre) X-Y traces of Ant skills and (Right) Heatmap visualize the learned continuous skill space. Traces demonstrate that the continuous space offers far greater diversity skills, while the heatmap demonstrates that the learned space is smooth, as the orientation of the X- trace varies smoothly as function of the skill.
In Figure 5 we project the trajectories of the learned Ant skills from both discrete and continuous spaces onto the Cartesian plane. From the traces of the skills, it is clear that the continuous latent space can generate more diverse trajectories. We demonstrate in Section 6.3, that continuous prim- itives are tasks. More importantly, we observe that the learned skill space is semantically meaningful. The heatmap in Figure 5 shows the orientation of the trajectory (with respect to the r-axis) func- tion of the skill 2 E 2, which varies smoothly as z is varied, with explicit interpolations shown in Appendix D
amenable to hierarchical composition and generally perform better downstream
6.2 SKILL VARIANCE ANALYSIS
In an unsupervised skill learning setup, is important to optimize the primitives to be diverse. How- ever, we argue that diversity is not sufficient for the learned primitives to be useful for downstream tasks. Primitives must exhibit low-variance behavior, which enables long-horizon composition of the learned skills in a hierarchical setup. We analyze the variance of the xz trajectories in the en- vironment, where we also benchmark the variance of the primitives learned by DIAYN (Eysenbach) jet al., 2018). For DIAYN, we use the x-y prior for the dkill-disrimminator, which biases the dis- covered skills to diversify in the x-y space. This step was necessary for that baseline obtain
7

---

Published as a conference paper at ICLR 2020
Standard Deviation of Trajectories
DIAYN with x-y prior
DIAYN with x-y prior
DADS with
DADS without xy prior
Standard Deviation
Steps in the Environment
DADS without x-y prior
DADS with x-y prior
Figure 6: (Top-Lefft Standard deviation of Ant's position as function of steps the environment, averaged over multiple skills and normalized by the norm of the position. (Top-Right to Bottom-Left Clockwise) X-Y traces of skills learned using DIAYN with x-y prior, DADS with z-y prior and DADS without x-y prior, where the same color represents trajectories resulting from the execution of the same skill in the environment. High variance skills from DIAYN offer limited utility hierarchical control.
competitive set of navigation skills. Figure o (Top-Left) demonstrates that DADS, which optimizes the primitives for predictability and diversity, yields significantly lower-varianee primitives when compared to DIAYN, which only optimizes for diversity. This is starkly demonstrated in the plots of X-Y traces of skills learned different setups. Skills learned by DADS show significant control over the trajectories generated in the environment, while skills from DIAYN exhibit high variance in the environment, which limits their utility for hierarchical control. This is further demonstrated quantitatively in Section 6.4
While optimizing for predictability already significantly reduces the variance of the trajectories gen- erated by primitive, we find that using the z-y prior with DADS brings down the skill variance even further. For quantitative benchmarks in the next sections. we assume that the Ant skills are learned using an z-y prior on the observation space, for both DADS and DIAYN.
6.3 MODEL-BASSE REINFORCEMENT LEARNING
The key utility of learning a parametric model (s'lss 2) is to take advantage of planning algorithms for downstream tasks, which can be extremely sample-efficient In setup, time tasks in zero-shot, that is without any learning on the downstream task. We compare with the state-of-the-art model-based RL method (Chua et al.) 2018a), which learns dynamics model parameterized as p(s"| (slss, a), on the task of the Ant navigating to specified goal with a dense reward. Given goal g. reward any position u given by r(u) -llg u u||2. We benchmark our method against the following variants:
solve test-
Random- MBRLL (rMBRL): We train the model p(s |s, a) on randomly collected trajecto- ries, and test the zero-shot generalization of the model on distribution of goals.
8

---

Published as a conference paper at ICLR 2020
Weak-oracle MBRL (WO-MBRL): We train the model ((s'|ss, a) on trajectories generated by the planner to navigate to a goal, randomly sampled in every episode. The distribution of goals during training matches the distribution at test time.
Strong-oracle MBRL (SO-MBRL); We train the model p(s' a) on trajectories generated by the planner navigate specific goal, which is fixed for both training and test time.
Amongst the variants, only the rMBRL matches our assumptions of having an unsupervised task- agnostic training. Both WO-MBRL and SO-MBRI. benefit from goal-directed exploration dur- ing training, significant advantage over DADS, which only uses inaiuaaiinoomatio----- explo- ration.
Hill as the metric, which represents the distance to the goal averaged over episode (with the same fixed horizon H for all models and experiments), normalized by the initial distance to the goal g- Therefore, lower A indicates better performance and 0 A (assuming the agent goes closer to the goal). The test set of goals is fixed for all the methods, sampled from [-15, 15
We use - -r(u)
Figure 7 demonstrates that the zero-shot planning significantly outperforms all model-based RL baselines, despite the advantage of the baselines being trained on the test goal(s). For the experi- ment depicted in Figure (Right), DADS has an unsupervised pre-training phase, unlike SO-MBRL which is training directly for the task. comparison with Random-MBRL shows the significance of mutual- caael--ifoomatonbb-sseeeee experiment also demonstrates the advantage of learning a continuous space of primitives, which outperforms planning discrete primitives.
(Right)
exploration, especially with the right parameterization and priors. This
Model-based RL
SO-MBRI, DADŞ
Normalized Mean Distance to Goal (4)
SO-MURL
Normalized aelleddd Mean Distance Goel
DADS-c
1.0
0.8
0.3
0.4
0.2
0.1
0.2
0.0
500000
1000000
1500000
2000000
(MBRL
WO-MBAL S0-MBAL
DADS-d
DADS-c
Number of Environment Samples
Figure 7: (Left) The results of the MPPI controller on skills learned using DADS-c (continuous primitives) and DADS-d (discrete primitives) significantly outperforms satc-off-hhe-r model-based RL. (Right) Planning for a new task does not require any additional training and outperforms model-based RL being trained for the specific task.
6.4 HIERARCHICAL CONTROL WITH UNSUPERVISED PRIMITIVES
We benchmark hierarchical control for primitives learned without supervision, against our proposed scheme using an MPPI based planner on top of DADS-learned skills. We persist with the task of Ant-navigation as described in 6.3 We benchmark against Hierarchical DIAYN (Eysenbach 2018), which learns the skills using the DIAYN objective, freezes the low-level policy and learns meta-controller that outputs the skill to be executed for the next Hz steps. We provide the r-y prior to the DIAYN's disciminator while learning the skills for the Ant agent. The performance of the meta-controller is constrained by the low-level policy, however, this hierarchical scheme agnostic the algorithm used to learn the low-level policy. To contrast the quality of primitives learned by the DADS and DIAYN, we also benchmark against Hicrarchical DADS, which learns a meta-controller the same way Hierarchical DIAYN, but learns the skills using DADS.
al.
From Figure (Left) We find that the meta-controller is unable to compose the skills learned by leam compose skills by DADS to navigate the Ant
DIAYN. while the meta-controller to different goals. This result seems to confirm our intuition described in Section 6.2 that the high variance of the DIAYN skills limits their temporal compositionality. Interestingly, learning RI.

---

Published as a conference paper at ICLR 2020
Normelizeed Mean Distance to Coal (5)
HRL on Unsupervised Skills
Normaizzd Mean Distance Goal (3)
2.0
Performance v/s Goal Distance
Herarchical
GCAL-dense
DADS HPPI
GcRRssppase
1.2
Hierarchical DADS
1.5
DADS dense
DADS
1.0
1.0
0.5-
0.2
0.0
0.0
10
20
25
Number of environment samples
Initial Distance to Goal
Figure 8: (Left) A RL-trained meta-controller is unable to compose primitive learned by DIAYN navigate Ant to goal, while it succeeds to so using the primitives learned by DADS. (Right) Goal-Conditioned RL (GCRL-densc/sparse) does not generalize outside its training distribution, while MPPI controller on Icarned skills (DADS- dense//prse// experiences significantly smaller degrade in performance.
meta-controller reaches similar performance to the MPPI controller, taking an additional 200, 000 samples per goal.
6.5 GOAL-CONDITIONED RL
To demonstrate the benefits of our approach conditioned RL on two versions of Ant-navigation: (a) with a dense reward r(u) and (b) with a sparse reward r(u) = 1 if ||u g/22 I c, else 0. We train the condiconnd RL agent using soft actor-critic, where the state variable of the agent is augmented with
model-free RL, benchmark against goal-
the position delta to the goal. The agent gets a randomly sampled goal from [-10, 10ja the beginning of the episode.
In Figure g (Right), we measure the average performance of the all the methods as a function of the initial distance of the goal, ranging from 5 to 30 metres. For dense reward navigation, we ob- serve that while model-based planning on DADS-leamed skills degrades smoothly as the initial distance to goal increases, goal-conditioned RL experiences sudden deterioration outside the goal distribution it was trained on. Even within the goal distribution observed during training of goal-conditioned RL model, skill-space planning performs competitively to it. With sparse reward navigation, goal-conditioned RL is unable to navigate, while MPPI demonstrates comparable perfor- mance to the dense reward up to about 20 metres. This highlights the utility of learning task aank-agnottic skills, which makes them more general while showing that latent space planning can be leveraged for tasks requiring long-howe Long-horizon planning.
CONCLUSION
We have proposed novel unsupervised skill learning algorithm that is amenable to model-based planning for hierarchical control on downstream tasks. We show that our skill learning method can scale to high-dimensional state-spaces, while discovering diverse set of low-variance skills. In ad- dition, we demonstrated that, without any training on the specified task, we can compose the learned skills to outperform competitive model-based baselines that were trained with the knowledge of the test tasks. We plan to extend the algorithm to work with off off-policy data, potentially using relabelling tricks (Andrychowicz rithms. We plan to apply the
al., 2017 Nachum et al., 2018) and explore more nuanced planning algo- method to different domains, such as manipulation and enable skill/model discovery directly from images.
ACKNOWLEDGEMENT
We would like to thank Evan Liu, Ben Eysenbach, Anusha Nagabandi for their help in reproducing the baselines for this work. We are thankful to Ben Eysenbach for their comments and discussion on the initial drafts. We would also like to acknowledge Ofir Nachum, Alex Alemi, Daniel Free- man, Yiding Jiang, Allan Zhou and other colleagues at Google Brain for their helpful feedback and discussions at various stages of this work. We are also thankful to Michael Ahn and others Adept team for their support, especially with the infrastructure setup and scaling up the experiments.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangging Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. tensorflow.org.
Software available from
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. arXiv preprint arXiv: 180. 10299, 2018.
David Barber Felix Agakov. The im algorithm: a variational approach to information maximization. Advances in Neural Information Processing Systems, 16:201, 2004.
Alexander A Alemi and lan Fischer. Therml: Thermodynamics of machine learning. arXiv preprint 1rXivv/807.044 2018
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abheel, and Wojeiech Zaremba. Hindsight experience replay. CoRR, abs/1707.01495, 2017. URL BRLp:pt://axxii http::///aawiiv..o//o.ao/s////// wttp////////////oo.////// ab5/1100.011155 01495
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence, 2017.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. In Advances in Neural Information Processing Systems, pp. 147 1-1499, 2016.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, has/1606.01540, 2016. URL htttttttp//xxxi>
abs/1606. 01540
Yevgen Chebotar, Karol Hausman, Marvin Zhang, Gaurav Sukhatme, Stefan Schaal, and Sergey Levine. Combining model-basee and model-free updates for trajectory-eettrcc reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 703-711. JMLR. org, 2017.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing in handful of trials using probabilistic dynamics models. CoRR, abs/1805. 12114, 8a. URL httt:://axiiiii org/ abs/18b5..1 12114
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learn- ing handful of trials using probabilistic dynamics models. In Advances Neural Information Processing Systems, pp. 4759-47700 2018b.
Imre Csiszár and Frantisek Matus. Information projections revisited. IEEE Transactions on Infor- 2003.
mation Theory,
Christian Daniel, Gerhard Neumann, and Jan Peters. Hierarchical relative entropy policy search. In Artificial Intelligence and Statistics, pp. 273-281, 2012.
Mare Deisenroth and Carl E Rasmussen. Pilco: A model- based and data-efficient approach to policy search. In Proceedings of the 28th International Conference on machine learning (ICML-11), pp- 465-472 2011.
Mare Peter Deisenroth, Dieter Fox, and Carl Edward Rasmussen. Gaussian processes for data- efficient learning in robotics and control. IEEE transactions on pattern analysis and machine intelligence, 37(2):408 423, 2013.
11

---

Published as a conference paper at ICLR 2020
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. arXiv preprint arXiv:1802.06770, 2018
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint arXiv:1704.03011 2017.
Nir Friedman, Ori Mosenzon, Noam Slonim, and Naftali Tishby. Multivariate information bottle- neck. Proceedings of the Seventeenth conference on Uncertainty artificial intelligence, pp- 151. Morgan Kaufmann Publishers Inc., 2001.
Justin Fu, Sergey Levine, and Pieter Abbeel. One-shot learning of manipulation skills with online dynamics adaptation and neural network priors. In 2016 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 4019-4026. IEEE, 2016.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. In Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2577-2587. Curran Associates, Inc., 2017. URL httppppp/ppperrennpsscccccppeeee
685 hh heg.paa--noooo----------eeeeeeee pdf
learning.
Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Eificient Machine Learning workshop, ICML., volume 4, 2016.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: theory and prac- tice- survey. Automatica, 25(3):335-345 1989.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint 161 1.07507, 2016.
Shixiang Gu, Ethan Holly. Timothy Lillicrap. and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 3389-3396. IEEE, 2017.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, pp. 2455 1267,,267,-1 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv:1801.01299, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. CoRR, abs/1812.05905, 2018b. URL httt:://araat wrw.voorgaaaa abs/
1812.05900
Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning embedding space for transferable robot skills. In International Conference Learn- ing Representations, 2018. URL/https://opeeneeeii wttpr//wwweeeeeeoooooooooo/////
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint arXiv: arX1v:1610.05112,, 2016.
Nicolas Heess, Srinivasan Sriram, Jay Lemmon. Josh Merel, Greg Wayne, Yuval Tassa, Tom Erez, Ziyu Wang, SM Eslami, Martin Riedmiller, et al. Emergence of locomotion behaviours in rich environments. arXiv preprint arXiv: /007.2/777.22220 02786,02266... 2017.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Curiosity-dii driven exploration in deep reinforcement learning via bayesian neural networks. CoRR, abs/1605.09674, 2016. URL http:://aa// wwwpsiiiiooorvrroooooo abs 1005 09674
12

---

Published as a conference paper at ICLR 2020
Robert A Jacobs, Michael Jordan, Steven Nowlan, Geoffrey E Hinton, et al. Adaptive mixtures of local experts. Neural computation, 3(1):79-88) 87, 1991.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
Sanket Kamthe and Mare Peter Deisenroth. Data-efficient reinforcement learning with probabilistic model predictive control. arXiv preprint arXiv: 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 128000 2014.
Alexander S Klyubin, Daniel Polani, and Chrystopher! Nehaniv. Empowerment: A universal agent- centric measure of control. In 2005 IEEE Congress Evolutionary Computation, volume pp- 128-135. IEEE, 2005.
Jonathan Ko, Daniel J Klein, Dieter Fox, and Dirk Haehnel. Gaussian processes and reinforce- ment learning for identification and control of an autonomous blimp. Proceedings 2007 ieee international conference on robotics and automation, pp. 742-747. IEEE, 2007.
Juš Kocijan, Roderick Murray-Smith Carl Edward Rasmussen, and Agathe Girard. Gaussian pro- cess model based predictive control. In Proceedings of the 2004 American Control Conference, volume 3. pp. 2212-221221 IEEE, 2004.
Vikash Kumar, Emanuel Todorov, and Sergey Levine. Optimal control with leamed local models: Application to dexterous manipulation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 378-383. IEEE, 2016.
Thanard Kurutach, Ignasi Clavera, Yan Duan, Aviv Tamar, and Pieter Abbeel. Model-ensemble trust-region policy optimization. arXiv preprint arXiv: 10592, 2018.
Ian Lenz, Ross A Knepper, and Ashutosh Saxena. Deepmpe: Learning deep latent features for model predictive control. Robotics: Science and Systems. Rome, Italy, 2015.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1373 2016.
Weiwei Li and Emanuel Todorov. Iterative linear quadratic regulator design for nonlinear biological movement systems. In ICINCO (1), pp. 222-229, 2004.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
Shakir Mohamed and Danilo Jimenez Rezende. Variational information maximisation for intrinsi- cally motivated reinforcement learning. In Advances in neural information processing systems, pp. 2125-122222 2133,,13,,22 2015.
Ofrr Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3307-3317, 2018.
Anusha Nagabandi, Gregory Kahn, Ronald S Fearing. and Sergey Levine. Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning, In 2018 IEEE Interna- tional Conference on Robotics and Automation (ICRA), pp. 7559-7566. IEEE, 2018
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard I. Lewis, and Satinder Singh. Action-condition video prediction using deep networks in atari games. In Advances in neural information process- ing systems, pp. 2863-2871, 2015.
Pierre- Yves Oudeyer and Frederie Kaplan. What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
13

---

Published as a conference paper at ICLR 2020
Pierre- Yves Oudeyer, Frdric Kaplan, and Verena V Hafner. Intrinsic motivation systems for tonomous mental development. IEEE transactions on evolutionary computation, 11(2):265-286, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity- driven exploration by self-supervised prediction. In 2017.
Xue Bin Peng, Glen Berseth, KangKang Yin. and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG), 36(4):41, 2017.
Theodore Perkins, Doina Precup, et al. Using options for knowledge transfer in reinforcement learning. University of Massachusetts, Amhersi, MA, USA, Tech. Rep, 1999.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint /709) 10087, 2017.
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer School on Machine Learning. pp. 63-71. Springer, 2003.
Jürgen Schmidhuber: Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Transactions on Autonomous Mental Development, 2(3):230-247 2010.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In Ieml, volume 37, pp. 1889-1897,201 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, Eugene Brevdo. TF-Agents: A library for reinforcement learning in tensorflow. httppppp://ggi.......o//e/eeeeee aggents) 2018. URL hatt:ss:/ggttabb..mm/teeesssssss 20-1N0.mberr20188.
[Online; accessed
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Noam Slonim, Gurinder S Atwal, Gasper Tkacik, and William Bialek. Estimating mutual informa- tion and multi-information large networks. arXiv preprint cx/0502017, 2005.
Bradly C. Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration in reinforcement learning with deep predictive models. CoRR, abs/1557.00814 2015. URL http:://axxiv.
htg/aba/1555 1507. 00814
Martin Stolle and Doina Precup. Learning options reinforcement learning. In International Symposium abstraction, reformulation, and approximation, pp. 212-223, Springer, 2002.
Richard Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: frame- work for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181- 211, 1999.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schul- man, Filip DeTurck, and Pieter Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. In Advances in neural information processing systems, pp. 2753- 2762, 2017.
Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin. Hugo Larochelle, Joelle Pineau, Doina Precup, and Yoshua Bengio. Disentangling the independently controllable factors of variation by interacting with the world, 2018.
14

---

Published as a conference paper at ICLR 2020
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033 IEEE, 2012.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg. David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning- Learning-Volume Volume 70, pp. 3540- 3549. JMLR. org, 2017.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non-parametric discriminative rewards. arXiv preprint arXiv: 18/ 1118//1111159 2018.
Manuel Watter, Jost Springenberg, Joschka Boedecker, and Martin Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. In Advances neural information processing systems, pp. 2746-2754, 2015.
Grady Williams, Paul Drews, Brian Goldfain, James M Rehg, and Evangelos A Theodorou. Aggres- sive driving with model predictive path integral control. In 2016 IEEE International Conference Robotics and Automation (ICRA), pp. 1433-1440. IEEE, 2016.
A IMPLEMENTATION DETAILS
All of our models are written the open source Tensorflow-Agent (Sergio Guadarrama, Anoop Korattikara, Oscar Ramirez, Pablo Castro, Ethan Holly, Sam Fishman, Ke Wang, Ekaterina Gonina, Chris Harris, Vincent Vanhoucke, Eugene Brevdo 2018), based on Tensorflow (Abadi al. 2015).
A.1 SKILL SPACES
When using discrete spaces, we parameterize as one-hot vectors. These one-hot vectors are ran- domly sampled from the uniform prior p(2) t. where D is the number of skills. We experiment with D < 128. For discrete skills learnt for MuloCo Ant in Section 6.3 we use D = 20. For continuous spaces, we sample
We experiment with D for Ant learnt with x-y prior. D) = 3 for Ant learnt without x-y prior (that is full observation space), to D = 5 for Humanoid on full observation spaces. The skills are sampled once in the beginning of the episode and fixed for the rest of the episode. However, it is possible to resample the skill from the prior within the episode, which allows for every skill to experience a different distribution than the ini- tialization distribution. This also encourages discovery of skills which can be composed temporally. However, this increases the hardness of problem, especially if the skills are re-sampled from the prior frequently.
(miform(-1,11))
A.2 AGENT
We use SAC as the optimizer for our agent 7aa 8, 2,, in particular, EC-SAC (Haarnoja et al. 2018b). The a input to the policy generally excludes global co-ordinates (x, y) of the centre-of- mass, available for lot enviroments OpenAl gym, which helps produce skills agnostic the location of the agent. We restrict to two hidden layers for our policy and critic networks. However, to improve the expressivity of skills, is beneficial to increase the capacity of the networks. The hidden layer sizes can vary from (128, 128) for Half-Cheetale to (512, 512) for Ant and (1024, 1024) for Humanoid. The critic (s,, a, 2) is similarly parameterized. The target function for critic Q is updated every iteration using a soft updates with co-efficient of 0.005. We use Adam (Kingma & Ba 2014) optimizer with a fixed learning rate of 3e and a fixed initial entropy co-efficient normal distribution N Nu(s,, 2). E(s, 2)) where Σ is a diagonal covariance matrix, it undergoes through tanh transformation, to transform the output to the range
B B= 0.1. While the policy is parameterized
(-1, 1) and constrain to the action bounds.
15

---

Published as a conference paper at ICLR 2020
A.3 SKILL-DYNAMICS
Skill-dynamies, denoted by g(s' s, 2, is parameterizee by a deep neural network. A common trick in model-based RI is to predict the As = 8, rather than the full state s'. Hence, the prediction network is ((As 8, 8, Note, both parameterizations can represent the same set of functions. However, the latter will be easy to learn as As will be centred around O. We exclude the global co- ordinates from from the state input to q. However, can (and we still do) predict Ag, Ay. because reward functions for goal-based navigation generally rely on the position prediction from the model. This represents another benefit of predicting state-deltas, as we can still predict changes in position without explicitly knowing the global position.
The output distribution is modelled of experts to be 4. We model each expert as a Gaussian distribution. The input (s, =) goes through two hidden layers (the same capacity as that of policy and critic networks, for example (512, 512) which
coature-of-Expertt (Jacobs et al., [1991). We fix the number
for Ant). The output of the two hidden layers is used as an input to the mixture-of-expere linearly transformed output the parameters of the Gaussian distribution, and discrete distribu- tion over the experts using a softmax distribution. In practice, we fix the covariance matrix of the Gaussian experts be an identity matrix, so we only need to output the means for the experts. We use normmaaaomiiiiio for both input and the hidden layers. We normalize the output targets using their batch-average and batch standadd deviation, similar to natch-normalizatioon
A.4 OTHER HYPERPARAMETER
The episode horizon is generally kept shorter for stable agents like Ant (200), while longer for unstable agents like Humanoid (1000). For Ant, longer episodes do not add value, but Humanoid can benefit from longer episodes as it helps it filter skills which are unstable. The optimization scheme is on-policy, and we collect 2000 steps for Ant and 4000 steps for Humanoid in one iteration. The intuition is to experience trajectories generated by multiple skills (approximately 10) in a batch. Re-sampling skills can enable experiencing larger number of skills. Once a batch of episodes is collected, the skill-dynamies is updated using Adam optimizer with fixed learning rate of 3e 4. The batch size is 128, and we carry out 32 steps of gradient descent. To compute the intrinsic reward, we need to resample the prior for computing the denominator. For continuous spaces, we set T, = 500. For discrete spaces, we can marginalize over all skills. After the intrinsic reward is computed, the policy and critic networks are updated for 128 steps with batch size of 128. The intuition is to ensure that every sample in the batch is seen for policy and critic updates about 3- - times in expectation.
A.5 PLANNING AND EVALUATION SETUPS
For evaluation, we fix the episode horizon to 200 for all models in all evaluation setups. Depending upon the size of the latent space and planning horizon, the number of samples from the planning distribution P is varied between 10- - 200. For Hp = 1, Hz = 10 and a 2D latent space, we use 50 samples from the planning distribution P. The co-efficient 7 for MPPI is fixed to 10. We use a setting of Hp = and Hz = 10 for dense-reward navigation, in which case we set the number of refine steps R. 10. However, for sparse reward navigation it is important to have longer horizon planning. in which case we set Hp 4, H = 25 with a higher number of samples from the planning distribution (200 from P). Also, when using longer planning horizons, we found that smoothing the sampled plans help. Thus, if the sampled plan is 21, 22, 23, 24 .... we smooth the plan to make 22 = 321 + (1 =) and so on, with 3 0.9.
For hierarchical controllers being learnt on top of low-level unsupervised primitives, we use PPO (Schulman et al. 2017) for discrete action skills, while we use SAC for continuous skills. We keep the number of steps after which the meta-action is decided as 10 (that is Hz 10). The hidden layer sizes of the meta-controller are (128, 128). We use a learning rate of Le - for PPO and 3e- -4 for SAC.
For our model-based RL baseline PETS, we use an ensemble size of 3, with a fixed planning hori- zon of 20. For the model, we use neural network with two hidden layers of size 400. In our experiments, we found that MPPI outperforms CEM, so we report the results using the MPPI as our controller.
16

---

Published as a conference paper at ICLR 2020
B GRAPHICAL MODELS, INFORMATION BOTTLENECK AND UNSUPERVISED SKILL LEARNING
We now present novel perspective unsupervised skill learning, motivated from the literature on information bottleneck. This section takes inspiration from (Alemi & Fischer 2018). which helps us provide rigorous justification for our objective proposed earlier. To obtain our unsupervised RL objective, we setup a graphical model /' as shown in Figure 9 which represents the distribution of trajectories generated by given policy æ. The joint distribution given by:
T-1 x St, 2) 2)p(st+11s;)))))) az).
................. a( ... -T- ST, 2) p())p(()) II
2
12
AT
d1
Q2
QT
81
82
ST
82
ST
Figure 9: Graphical model for the world P in which the trajectories are generated while interacting with the environment. Shaded nodes represent the distri- butions optimize.
Figure 10: Graphical model for world N which the desired representation of the world.
We setup another graphical model which represents the desired model of the world. In particular, interested in approximating p( s' z), which represents the transition function for particular primitive. This abstraction helps us get away from knowing the exact actions, enabling model-based planning in behavior space (as discussed in the main paper). The joint distribution for N shown in Figure is given by:
T-1
(71,,, 01, ST,
or, (T, 2) = (=))((81) II ((84))7(8441188222
The goal of our approach is to optimize the distribution (a]ss, z) in the graphical model P to min- imize the distance between the two distributions, when transforming to the representation of the graphical model Z. In particular, we are interested minimizing the KL divergence between p and y, that is ( would be 0 for any valid P. Interestingly, we can exploit the following result from in Friedman et al. 200T to setup the objective for ग, without explicitly knowing 1f:
Note, if N had the same structure P. the information lost in projection
min DxL (pl = Ip IN.
(10)
where Zp and In represents the multi-information for distribution P the respective graphical models. Note, mingeN Dxz(pllm) which is the reverse information projection (Csiszár & Matus 2003). The multi multi-inffrrmtion (Slonim al. 2005) for graphical model G with nodes gi is defined as:
Ign Ia Ign Po(g:)),
(11)
where Pa(g.) denotes the nodes upon which Si has direct conditional dependence G. Using this definition, we compute the multi-information terms:
T I(st (8(-1.2)).
Ip
I(ar: {st, 2)) + Σ I(se: (81-1, a--1)) and In Σ 1=
(12)
17

---

Published as a conference paper at ICLR 2020
Following the Optimal Frontier argument in (Alemi & Fischer 2018 introduce Lagrange mul-
tipliers 3t 2 0, 62 2 for the information terms in Zp to setup an objective R(x)) to be maximized with respect to
R(T) Σ Mssuniss 2)) Be (st, (s
(13)
(14)
As the underlying dynamics are fixed and unknown, simplify the optimization by setting & 0 which intuitively corresponds to us neglecting the unchangeable information of the underlying dynamics. This gives us
R(()) =
E 181411 {8,, - 3.1 {5t,
(15)
Σ
[(s4412 st) - -aci(aci ()))
(16)
t=1
Here, we have used the chain rule of mutual information:
[(8++1; (8t, I(8 8c) +
I5 8t) 2 I(+++1;= 8t), resulting from non-negativity mutual information. This yield us an information bottleneck style objective where we maximize the mutual information moti- vated in Section while minimizing I(a; {st, +))) We can show that the minimization of the latter mutual information corresponds to entropy regularization of raa st, z), as follows:
π (W: 82, 2)
I(a:; (8t, =)) (ea
log
(17)
TT(e))
T ((at 8t,, 2)
log
DKL (T(()) p(a:))
(18)
p(at)
π ((t St, 2)
< (a
log
(19)
p(o+)
for some arbitrary distribution p(a,) (for example uniform). Again, we have used the non- negativity of DEL to get the inequality. We use Equation 19 in Equation 16
get:
T-1 R(T) Σ
- B - (aalaaa log ( (s,22)
(20)
where we have ignored p(a,) as it is constant with respect to optimization for This motivates the use of entropy regularization. We can follow the arguments Section 3 to obtain an approxi- mate lower bound for I(st+1i2 st). The above discussion shows how DADS be motivated from a graphical modelling perspective, while justifying the use of entropy regularization from an information bottleneck perspective. This objective also explicates the temporally extended nature of z, and how it corresponds to a sequence of actions producing a predictable sequence of transitions in the environment.
2
Figure 11: Graphical model for the world P rep- resenting the stationary state, action distribution. Shaded nodes represent the distributions we opti- mize.
Figure 12: Graphical model for the world N using which is the representation are interested in.
18

---

Published as a conference paper at ICLR 2020
We can carry out the exercise for the reward function Eysenbaacch et al. (2018) (DIAYN) to provide
a graphical model interpretation of the objective used in the paper. To conform with objective in the paper, we assume to be sampling be aate-actionn pairs from skill-conditioned stationary distributions in the world P, rather than trajectories. The objective to be maximized is given by:
R(a)) = -pp + IQ
(21)
- [(; {s, 2)) + (2;
(22)
E, llog p(z[s)
log
(23)
p(z)
T(a)
2 E, [log (z(2)s) - log p(z) log ( 2)] = R(T, q6)
(24)
where we have used the variational inequalities to replace p=21s) with (s(=|s) and T(a) with a uniform prior over bounded actions p(a) (which is ignored as a constant).
C APPROXIMATING THE REWARD FUNCTION
We revisit Equation 4 and the resulting approximate reward function constructed in Equation 6 The maximization objective for policy was:
R(T qo) = Ez,s, 1o |8,2) - logg((s' s)]
(25)
The computational problem arises from the intractability of p(s' 8) J p(s' | s, =)z))(z s)dz, where both p(s' 8, 2) and p(2 s) x p(s 2))((z) are intractable. Unfortunately, any variational approximation results in an improper lower bound for the objective. To see that:
R(x (6)= = E.,,,o' log (s | s,2)) go' s)] Dxu(p(s' (s) g(s' s))
(26)
< Be.a. loog (s 8) log a(s' |)]
(27)
where the inequality goes the wrong way for any variational approximation q(s' 8). Our ap- proximation can be seen as a special instantiation of g(s' s) Js((s s, 2)2 This approximation is simple to compute as generating samples from the prior p(2) inexpensive and effectively requires only a forward pass through Gs- Reusing 2, to approximate p(s' s) makes intuitive sense because we want de to reasonably approximate p(s' 8, 2) (which is why we collect large batches of data and take multiple steps of gradient descent for fitting qo). While sampling from the prior p(z) crude, sampling p(z 8) can be computationally prohibitive. For a certain class of problems, especially locomotion, sampling from p(=) is reasonable approximation want our primitives/skills to be usable from any state, which is especially the case with locomotion. Empirically, we have found our current approximation provides satisfactory results. We also discuss some other potential solutions (and their limitations):
well. We
(a) One could potentially use another network qa(= s) approximate p(z s) by minimizing Eg,s<pl [DxL lower bound for R(x g.). However, sampling from this gs might result in better approximation than sampling from the prior p(z) for some problems.
s) |l go(2 s)]]. Note, the resulting approximation would still be an improper
(b) We can bypass the computational intractability of p(s' s) by exploiting the variational lower bounds from Agakov| (2004). We use the following inequality, used in Hausman et al. (2018):
H(x)) / p(ar, x) log p(x, z)
((22z) drdz
(28)
where q is a variational approximation to the posterior p(z(ar).
((s322|3) = -H((s'
(00(s'|s,2, [s, 2) + *ssssss)
(29)
2 R.,,, log go (8 11, =)] + E. - log go (z)6', + HH(6'
(30)
= E -' log (s [s, + log (2]s' {)] + H(ss'
(31)
where we have used the inequality for H(s'(s) s) to introduce the variational posterior for skill infer- ence (= s'ss) besides the conventional variational lower bound to introduce q(s' s, Further decomposing the leftover entropy:
H(ss', 2|s) H((=s) + H((s)(s,))
19

---

Published as a conference paper at ICLR 2020
Reusing the variational lower bound for marginal entropy from Agakov 2004),
get:
H(s'|s, 2, 2 Bs, / p(s', als, 2) log o(als',s,2) ds'da da p(s',als,2)
(32)
- =ooo c + H(s', "ass'als,,>) als,z)
(33)
loge + H(s' s, 2) + H())s,2)
(34)
Since, the choice posterior is upon us, we can choose q(als', s, 1/c to induce a uniform distribution for the bounded action space. For H(s'|s, a, 2), notice that the underlying dynamics p(s'|s, a) are independent of z, but the actions do depend upon 2. Therefore, this corresponds to entropy-regularized RL when the dynamics of the system are deterministic. Even for stochastic dynamics, the analogy might be a good approximation, assuming the underlying dynamics are not very entropic. The final objective (making this low-entropy dynamics assumption) can be written as:
I [[d;)(ss 2 E.Ep( .21)) [log (s')s,, + log qa (z)ss's)) s) logp(> + H(a)s,2)
(35)
While this does bypass the intractability of p(s' 8). it runs into the intractable p(= | 8). despite deploying significant mathematical machinery and additional assumptions. Any variational approx- imation for p(z s) would again result in an improper lower bound for = |s).
(c) One way make our approximation q(s s) to more closely resemble p(s' | is to change our generative model p(z, 8, s'). In particular, if we resample z ~ p(z) for every timestep of the rollout from x, we can indeed write p(= s) = p(z). Note, p(s' s) is still intractable compute, but marginalizing (s 8, 2) over p(z) becomes a better approximation of p(s' s). However, this severely dampens the interpretation of our latent space as temporally extended actions (or skills). It becomes better to interpret the latent space E as dimensional reduction of action space. Empirically, we found that this significantly throttles the learning, not yielding useful or interpretable skills.
|)
D INTERPOLATION IN CONTINUOUS LATENT SPACE
Figure 13: Interpolation continuous primitive space learned using DADS on Ant environment cor-
responds to interpolation the trajectory space. (Left) Interpolation from 2 = [1.0, 1.0] (solid blue) to = [-10, 1.0) (dotted cyan); (Middle) Interpolation from z [1.0, 1.0j (solid blue) to = [-1.0, - 1.0] (dotted cyan); (Right) Interpolation from = [1.0, 1.0] (solid blue) to [1.0, -1.0) (dotted cyan).
E MODEL PREDICTION
we observe that skill-dynamics can provide robust state- patte-predittions over long plan- ning horizons. When learning skill-dynamics with x prior, we observe that the error in prediction rises slower with horizon compared to the norm of the actual position. This provides strong ev- idence of cooperation between the primitives and skill-dynamies learned using DADS with prior. As the error- grooth for skill-dynamies learned on full-observation space is sub-exponential, similar argument can be made for DADS without y prior as well (albeit to a weaker extent).
From Figure 14
20

---

Published as a conference paper at ICLR 2020
20
Prediction error in position vis Hqrizon
with x-y prior
15
without x-y
Prediction
-5
0.2
0.0
-15
Steps in Future
-20
-20
-15
-10
10
Figure 14: (Left) Prediction error in the Ant's co-ordinates (normalized by the norm of the actual position) for skill-dynamics. (Right) X-Y traces of actual trajectories (colored) compared trajectories predicted by skill-dynamiee (dotted-black) for different skills.
21


Reviews:
[{"review_id": "SJg5q-L5Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel approach to learn a continuous set of skills (where a skill is associated with a latent vector and the skill policy network takes that vector as an extra input) by pure unsupervised exploration using as intrinsic reward a proxy for the mutual information  between next states and the skill (given the previous state). These skills can be used in a model-based planning (model-predictive control) with zero 'supervised' training data (for which the rewards are given), but using calls to the reward function to evaluate candidate sequences of skills and actions. The proposed approach is convincingly compared in several ways to both previous model-based approaches and model-free approaches.\n\nThis is a very interesting approach, and although I can already think of ways to improve, it seems like an exciting step in the right direction to develop more autonomous and sample efficient learning systems. I suggest to rate this submission as 'accept'.\n\nRegarding the comparison to model-free RL: although it is true that no task-specific training is needed, a possibly large number of calls to the reward functions are needed during planning. It would be good to compare those numbers with the number of rewarded trajectories needed for the model-free approaches.\n\nMy main concern with the proposed method is how it would scale when the state-space becomes substantially larger (than the 2 dimensions x and y used in the experiments). The reason I am concerned is that the proposed method uses brutal sampling to search for good trajectories in z-space and action-space. It looks like the curse of dimensionality will quickly make this approach unfeasible. Also, it would be nice to have the learning system discover the important dimensions in which to plan (the x and y in the experiments), rather than having to provide them by hand.\n\nA minor concern is the following: is it possible that the optimization could end up discovering a large number of highly predictable (and diverse) but useless skills?\n\nIn the related work section, 1st paragraph, in the list of citations, it might be good to also include the work on maximizing mutual information between representation of the next state and representation of the skill (Thomas et al, arXiv:1802.09484).\n\nThe definition of Delta (page 9) is strange: it is said that Delta should be minimized but Delta is defined as proportional to the rewards (which should be maximized). Maybe a sign is missing. Also, why not simply define the rewards as being normalized in the first place, so that the metric IS the accumulated reward rather than this unusual normalized version of it.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "ByxhdaGN5r", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces an unsupervised learning algorithm Dynamics-Aware Discovery of Skills (DADS) for learning low-level “skills” that can be leveraged for model-predictive control. The skills are learned by maximizing the mutual information between the next state s’ and the current skill z conditioned on the current state s. Maximizing this objective corresponds to maximizing the diversity of transitions produced in the environment, while making the skill z be informative about the next state s’. The idea is that using this objective leads to learning a diverse set of skills that are predictive of the environment. The skills z correspond to a set of action sequences, which are represented by a distribution \\pi(a|s,z). Because the above objective is intractable to compute (because it relies on the true dynamics p(s’|s,a)), it is variationally lower bounded using the approximate dynamics q_{\\phi}(s’|s,z), which represents the transition dynamics when using a certain skill and this variational lower bound is optimized to produce the optimal q_{\\phi}(s’|s,z) and \\pi(a|s,z).\n\nIn the second phase, model predictive control is used to do planning for a new test environment where we have access to the reward function. This corresponds to simulating multiple trajectories using the learned transition dynamics and skill function, computing the reward of each trajectory according to the reward function, executing the first action of the most optimal trajectory and repeating. It is mentioned that planning is done in the latent skill space, which enables easier longer-horizon planning.\n\nExperiments are performed to show that: (1) the learned skills exhibit low-variance behavior (which means that the skills have predictable behavior when used for model predictive control); (2) Model predictive control performs favorably compared to other relevant baselines.\n\nOverall, I feel this is a very well-motivated and interesting submission with very thorough experiments.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylOrNEmqH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors try to incorporate intermediate-level skills into model-based RL, which is an essential problem in the RL field. The algorithm works well even in the case of high-dimensional state and action spaces. Their contributions are in four aspects:\n(1)they propose an unsupervised RL framework for learning intermediate-level representations, i.e. skills, based on maximizing the mutual information between the future state and current skill given the current state. This procedure is well-motivated and the mathematics is easy to follow.\n(2)they reformulate model predictive control (MPC) in the latent skill space.\n(3)their method is compatible with the idea of continuous skill spaces, which seems to give rise to more diverse trajectories and hence offers greater utility.\n(4)their method yields low-variance behavior while maintaining enough diversity.\n\nIt’s an accept for me. On one hand, using model-free unsupervised RL methods to learn intermediate-level skills is not a new idea. on the other, approaching this problem via mutual information is, as far as I know, new to this field. Although, the novelty of this approach remains undecided, this method seems to work well enough compared to model-based, model-free and hierarchical RL methods. Their analysis from the perspectives of continuous skill space and skill variance also seem to hold.\n\nNevertheless, the study would benefit from more comparison with other methods using intermediate-level primitives (apart from DIAYN). Moreover it would be interesting to show this method works in scenarios apart from locomotion. I wonder how well the approximation p(z|s) \\approx p(z) works in non-locomotion tasks. Otherwise, the authors should mention this method is somewhat limited to locomotion tasks in the main text. \n\nOthers:\nTypo:\n1.Page 3: “maximally informative about about …” remove the redundant “about”;\n2.Page 8, first line in section 6.3 “is to be enable use of planning algorithms…” may be changed to “is to take advantage of planning algorithms”.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a very interesting paper on unsupervised skill learning based on the predictability of skill effects, with the incorporation of these ideas into model-based RL.

This is a clear accept, based on the clarity of the ideas presented and the writing, as well as the thorough and convincing experiments.
Decision: Accept (Talk)