Lipschitz-constrained Unsupervised Skill Discovery
Authors: Seohong Park
Jongwook Choi
Jaekyeom Kim
Honglak Lee
Gunhee Kim
Venue: ICLR 2022 Poster (2022)
Keywords: Reinforcement learning

Abstract: We study the problem of unsupervised skill discovery, whose goal is to learn a set of diverse and useful skills with no external reward. There have been a number of skill discovery methods based on maximizing the mutual information (MI) between skills and states. However, we point out that their MI objectives usually prefer static skills to dynamic ones, which may hinder the application for downstream tasks. To address this issue, we propose Lipschitz-constrained Skill Discovery (LSD), which encourages the agent to discover more diverse, dynamic, and far-reaching skills. Another benefit of LSD is that its learned representation function can be utilized for solving goal-following downstream tasks even in a zero-shot manner — i.e., without further training or complex planning. Through experiments on various MuJoCo robotic locomotion and manipulation environments, we demonstrate that LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance on seven downstream tasks including the challenging task of following multiple goals on Humanoid. Our code and videos are available at https://shpark.me/projects/lsd/.

Published as a conference paper at ICLR 2022
LIPSCHITZ-CONSTRRIINED UNSUPERVISED SKILL DISCOVERY
Seohong Park
Jongwook Choi
Jackyeom Kim Honglak Leea.3 Gunhee Kim!
Seouul National University
fartberryx, jaekyeom, gunhee) e @snu. ac. {jwook, honglak ak @umi chiiche.e.. edu
University of Michigan LL AI Research
ABSTRACT
We study the problem of unsupervised skill discovery, whose goal is to learn a set of diverse and useful skills with external reward. There have been num- ber of skill discovery methods based on maximizing the mutual information (MI) between skills and states. However, we point out that their MI objectives usu- ally prefer static skills to dynamic ones, which may hinder the application for downstream tasks. To address this issue, we propose Lipschits-constrrinned Skill Discovery (LSD), which encourages the agent to discover more diverse. dynamic, and far-reaching skills. Another benefit of LSD that its learned representation function can be utilized for solving goal-following downstream tasks even in a zero-shot manner - i.e. without further training or complex planning. Through experiments various MuJoCo robotic locomotion and manipulation environ- ments, we demonstrate that LSD outperforms previous approaches in terms of skill diversity, state space coverage, and performance seven downstream tasks including the challenging task of following multiple goals on Humanoid, Our code and videos available at https: // wwwpark.eepppoeeeeeee... ww/projects///////oooooss //sss/ts//////
INTRODUCTION
Reinforcement learning (RL) aims at learning optimal actions that maximize accumulated reward signals (Sutton Barto, 2005). Recently, RL with deep neural networks has demonstrated re- markable achievements in variety of tasks, such complex robotics control (Gu et al., 2017; Andrychowicz et al., 2020) and games (Schrittwieser et al., 2020; Badia et al., 2020). However, one limitation of the RL framework is that practitioner has to manually define and tune reward function for desired behaviors, which is often tinecconuuming and hardly scalable especially when there multiple tasks learn (Hadfield- Menell et al., 2017; Dulac-Arnold et al., 2019).
Therefore, several methods have been proposed to discover skills without external task rewards (Gre- gor et al., 2016; Eysenbach et al., 2019; Sharma et al., 2020), which is often referred to as the imsu- pervised skill discovery problem. Unsupervised discovery of skills helps not only relieve the burden of manually specifying reward for each behavior, but also provide useful primitives to initialize with or combine hierarchically for solving downstream tasks (Eysenbach et al., 2019; Lee et al., 2020). Moreover, learned skills can effectively demonstrate the agent's capability in the environ- ment, allowing a better understanding of both the agent and the environment.
One of the most common approaches to the unsupervised skill discovery problem is to maximize the mutual information (MI) between skill latent variables and states (Gregor et al., 2016; Achiam al., 2018; Eysenbach et al., 2019; Hansen et al., 2020; Sharma et al., 2020; Choi et al., 2021; Zhang al., 2021). Intuitively, these methods encourage skill latent be maximally informative of states or trajectories obtained from a skill policy (a]ss, 2). As a result, optimizing the MI objective leads to the discovery of diverse and distinguishable behaviors.
However, existing MI-based skill discovery methods share a limitation that they do not necessarily prefer learning 'dynamic' skills (i.e., making large state variations) as diverse locomotion primitives. Since MI is invariant to scaling or any invertible transformation of the input variables, there exist infinitely many optima for the MI objective. As such, they will
task-relevant behaviors such
"Equal contribution, listed in alphabetical order.

---

Published as a conference paper at ICLR 2022
L.D (ours)
DIAYN
(a) Ant environment. (b) Visualization of discovered 2-D skills on the z-y plane. (c) Zero-shot goal following.
Figure 1: Skills discovered by LSD can be used follow goals with no further training planning.
converge to the maximum that is most easily optimizable, which would usually be just learning simple and static skills. For instance, Figure 1b and Figure 17 demonstrate that DIAYN (Eysenbach et al., 2019) simply learns to take various postures in place rather than learning locomotion skills in the Ant environment (Schulman et al., 2016). While these works often employ some feature engineering or prior domain knowledge to discover more dynamic skills (e.g., discriminating skills based on z-y coordinates only (Eysenbach et al., 2019; Sharma et al., 2020)), it brings about other drawbacks: (i) practitioners need to manually specify the dimensions of interest and (ii) the diversity of skills may be limited to specific type (e.g., the z-y prior results in neglecting non-locomotion behaviors).
In order address this limitation of MI-based approaches, we propose unsupervised skill discov- ery method based on a Lipschitz continuity constraint, named Lipschirz Skill Discovery (LSD). Specifically, we argue one reason behind the aforementioned problem is that the MI-based skill discovery methods can easily maximize the MI objective with only slight differences in the state space. To resolve this, we propose novel objective based Lipschitz-constrained state rep- resentation function so that the objective maximization in the latent space always entails an increase in traveled distances (or variations) in the state space (Equation (6)).
LSD has several advantages. First, unlike previous MI-based skill discovery objectives, LSD en- courages learned skills to have larger traveled distances, which leads to more diverse, dynamic, and far-reaching skills. Second, LSD produces a useful state representation function that can be used to solve goll-following downstream tasks (i.e., reaching multiple goals in order) in fully zero-shot manner (Figure 1c); Le., with no further training or complex planning. Finally, LSD is easy to implement in contrast many existing approaches and introduces no additional hyperparameters.
Our contributions can be summarized as follows:
We propose a novel skill discovery objective based on a Lipschitz constraint named LSD, which maximizes traveled distances in the state space unlike existing MI-based approaches, and thus learns more diverse and dynamic behaviors with no need for feature engineering.
Since LSD aligns the directions of skills and differences in latent state representations, it can efficiently solve goal-following tasks with a wider range of goals a zero-shot fashion compared to previous methods, with no burden of additional training or complex planning. Table 1 highlights other distinctive properties of LSD in comparison to existing approaches.
LSD exhibits the best performance in terms of the state space coverage on five MuJoCo environ- ments and final rewards on seven downstream tasks, including AntMultiGoals (Kim et al., 2021), HumanoidMultiGoals and Getah, compared to previous skill discovery methods such as DIAYN (Eysenbach et al., 2019), DADS (Sharma et al., 2020) and IBOL. (Kim et al., 2021).
PRELIMINARIES AND RELATED WORK
2.1 PROBLEM SETTING
We consider Markov decision process (MDP) M = (S, without external rewards, where is a (continuous) state space, AA is an action space, and p(st+1 is a stochastic transition dynamics function. We represent skill with a latent variable Z and latent-conditioned policy π (a)s,,) 2). The skill latent space Z can be either discrete or continuous; we use N to denote the number of skills in the discrete case, and the continuous case. Given skill and a skill policy (T) 2), a trajectory 7 = (So. ao..... sr) is sampled with the following generative process: p² (7|=) p(5)) I (aelse, = =)p($4+1 at).
denote the dimensionality of the skill latent space in
2

---

Published as a conference paper at ICLR 2022
Table Comparison of unsupervised skill discovery methods. Refer to Section 2.2 for citations.
Property
VIC
DIAYN
DADS
VISR
EDL
APS
IBOL
LSD (ours)
Prefer "dynamic' skills
X
X
/
Provide dense reward
X
/
/
/
X
Discover continuous skills
/
/
/
/
Discover discrete skills
x
X
shot goal goal-following
x
Properties "Prefer 'dynamic' skills": whether the algorithm prefers skills beyond simple and static ones. "Zero-shot goal-following whether learned skills can used for following multiple goals (from an arbitrary state) without additional training, where denotes that it needs planning and * denotes that its skill dis- criminator may not cope well with unseen goals or initial states.
We use uppercase letters to denote random variables, and h(-) and [(:+)) to represent the differential entropy and the mutual information, respectively.
2.2 PRIOR WORK ON UNSUPERVISED SKILL DISCOVERY
A number of previous methods maximize (( to learn diverse skills. One line of research em- ploys the identity (2; h(2) - h(Z(S) 2 Egr=(()), (sl=) (sl=)]log [ - ----(=)) [log p(=)] or its variants, where the skill discriminator q(=[s) is a variational approximation of the posterior p(z]s) (Barber & Agakov, 2003). VIC (Gregor et al., 2016) maximizes the MI between the last states and skills given the initial state. DIAYN (Eysenbach et al., 2019) optimizes the MI between individ- ual states and skills. VALOR (Achiam et al., 2018) also takes a similar approach, but considers the whole trajectories instead of states. VISR (Hansen et al., 2020) models the variational posterior as the von-Mises Fisher distribution, which results in an inner- mmer-product reward form and hence enables combining with successor features (Barreto et al., 2017). HIDIO (Zhang et al., 2021) examines mul- tiple variants of the MI identity, and jointly learns skills with a hierarchical controller that maximizes the task reward. Choi et al. (2021) point out the equivalency between the MI-based objective and goal-conditioned RL, and show that Spectral Normalization (Miyato et al., 2018) improves the qual- ity of learned skills. DADS (Sharma et al., 2020) maximizes the opposite direction of the mutual information identity SZ: = h(S) Z(S)Z) with the skill dynamics model g( 1 | 2), which allows zero-shot planning on downstream tasks. However, these methods share a limitation: they do not always prefer to reach distant states or to learn dynamic skills, we can maximize 7(Z; S) even with the smallest state variations. One possible way to address this issue is to use heuristics such as feature engineering; for example, the z-y prior (Sharma et al., 2020) enforces skills to be discriminated only by their z-y coordinates so that the agent can discover locomotion skills.
On the other hand, a couple of methods overcome this limitation by integrating with exploration techniques. EDL (Campos Camúñez al., 2020) first maximizes the state entropy h(S)) with SMM exploration (Lee et al., 2019), and then encodes the discovered states into skills via VAE (Kingma & Welling, 2014). APS (Liu & Abbeel, 2021a) combines VISR (Hansen al., 2020) with APT (Liu & Abbeel, 1b), an exploration method based on k-nearest neighbors. Yet, we empirically confirm that such a pure exploration signal is insufficient to make large and consistent transitions in states. IBOL (Kim et al., 2021) takes a hierarchical approach where it first pre-trains low-level policy to make reaching remote states easier, and subsequently learns a high-level skill policy based on the information bottleneck framework (Tishby et al., 2000). While IBOL can discover skills reaching distant states in continuous control environments without locomotion priors (e.g., z-y prior). it still has some limitations in that (1) IBOL cannot discover discrete skills, (2) it still capitalizes input feature engineering in that they exclude the locomotion coordinates from the low-level policy. and (3) it consists of two-level hierarchy with several additional hyperparameters, which make the implementation difficult. On the contrary, our proposed LSD can discover diverse skills in both discrete and continuous settings without using any feature engineering, and is easy to implement as it requires no additional hyperparameters or hierarchy. Table 1 overviews the comparison of properties between different skill discovery methods.
3 LIPSCHITZ- SKILL DISCOVERY (LSD)
We first analyze limitations of existing MI-based methods for unsupervised skill discovery (Sec- tion 3.1), and then derive our approach for learning continuous skills, cipsthiit Skill
3

---

Published as a conference paper at ICLR 2022
Discovery (LSD), which encourages the agent to have large traveled distances in the state space (Section 3.2). We also show how learned skills can be used to solve goal-following tasks in a zero- shot fashion (Section 3.3), and extend LSD to discovery of discrete skills (Section 3.4).
3.1 LIMITATION OF MI-BASED METHODS
Before deriving our objective for discovery of continuous skills, we review variational MI-based skill discovery algorithms and discuss why such methods might end up learning only simple and static skills. The MI objective (() with continuous skills can be written with the variational lower bound as follows (Eysenbach et al., 2019; Choi et al., 2021):
I(Z = h(()s) + h(Z) = B=-p(=), (s|=) [log p(z|s) log [()]
(1)
1
2 Ez, [(])))===()))))))) + (const) 2 Ez,s [|| - (()) + (const),
(2)
where we assume that skill E Rd is sampled from fixed prior distribution p(z), and g(=[s) variational approximation of p(z[s) (Barber & Agakov, 2003; Mohamed & Rezende, 2015),
parameterized as normal distribution with unit variance, (V(=ss), I) (Choi et al., 2021). l). Some other methods are based on a conditional form of mutual information (Gregor et al., 2016; Sharma et al., 2020); for instance, the objective of VIC (Gregor al., 2016) be written
1(;; ST So)= 2 Ez.r [log [=] sT]] + (const) = - Ex.r [[]= - μu(so, ST ] + (const), (3)
where we assume that p(= (80) = p(2) is a fixed prior distribution, and the posterior is chosen as (22|50, ST) = N (u(so, ST), 1) in a continuous skill setting.
One issue with Equations (2) and (3) that these objectives can be fully maximized even with small differences in states as long different 2's correspond necessarily encouraging more 'interesting' skills. This is especially problematic because discovering skills with such slight less dynamic state variations is usually fower-hanging fruit' than making dynamic and large differences in the state space (e.g., l can simply map the angles of Ant's joints 2). As result, continuous DIAYN and DADS discover only posing skills on Ant (Figures 2a and 17) in the absence of any feature engineering or tricks to elicit more diverse behaviors. We refer to Appendix for quantitative demonstrations of this phenomenon on MuloCo environments.
marginally different sy 's, not
Further decomposition of the MI objective. Before we address this limitation, we decompose the objective of VIC (Equation (3)) to get further insights that will inspire new objective in Section 3.2. Here, we model (Su,, sT) with ((s)) Φ(so) to focus on the relative differences between the initial and final states, where Φ S - is learnable state representation function that maps a state observation into latent space, which will be the core component of our method. This choice makes the latent skill represent direction displacement in the latent space induced by ø. Then, we can rewrite Equation (3) as follows:
Eg.r [log (2 sT) + (const)
E.,r (112-(6(s)) - + + (const)
(4)
2
1
Eg,r ((((T) ((s))) Tz
E.,r [[(Ø(sT) - (100) (const),
(5)
2
alignment of the direction
implicit rogularizer
where we use the fact that [= 2] is constant as p(2) is fixed distribution. This decomposition of the MI lower-bound objective provides an intuitive interpretation: the first inner-product term ii in Equation (5) encourages the direction vector (d(T) ((5)) be aligned with z, while the second term 2 regularizes the norm of the vector ((sT) o(so).
3.2 CONTINUOUS LSD
The LSD objective. We now propose our new objective that is based on neither skill discriminator (Equation (3)) nor mutual information but a Lipschitz-constrained state representation function, in order to address the limitation that 7(Z; S) can be fully optimized with small state differences. Specifically, inspired by the decomposition in Equation (5), we suggest using the directional term ① as objective. However, since this term alone could be trivially optimized by just increasing the value of σ(sr) to the infinity regardless of the final state sy, we apply the 1 1-Lipschitz constraint on the state representation function that maximizing our objective in the latent space can result
4

---

Published as a conference paper at ICLR 2022
in an increase in state differences. This leads to a constrained maximization objective follows:
JLSD E., [(ő(ST) ((s))) =]
s.t. Vx,y E [l((() ا(v)) S | - yll=
(6)
where JLSD is the objective of our proposed Lipschitt-constrained Skill Discovery (LSSD).
The LSD objective encourages the agent to prefer skills with larger traveled distances, unlike previ- ous MI-based methods, as follows. First, in order to maximize the inner product in Equation (6), the length of ((ST) - of so) should be increased. It then makes its upper bound || - soll increase as well due to the I-Lipschitz constraint (i.e.. [|ó)(T) - ((+0)] || S ||sT - sol)) As a result, leads to learning more dynamic skills in terms of state differences.
Note that LSD's objective differs from VIC's in an important way. Equation (3) tries to equate the value of and ja(80, ST) (i.e., it tries to recover = from its skill discriminator), while the objective of LSD (Equation (6)) only requires the directions of and ((sT) - ((80) to be aligned.
We also note that our purpose of enforcing the Lipschitz constraint is very different from its common usages in machine learning. Many works have adopted the Lipschitz continuity to regularize func- tions for better generalization (Neyshabur et al., 2018; Sokolic et al., 2017), interpretability (Tsipras et al., 2018) stability (Choi et al., 2021). On the other hand, we employ it to ensure that maximiza- tion of the reward entails increased state variations. The Lipschitz constant I is chosen empirically as it can be easily implemented using Spectral Normalization (Miyato et al., 2018).
Per-step transition reward. By eliminating the second term in Equation (5), we can further de- compose the objective using telescoping sum as
JLSD = Eg,r [( - (0)))) =1,r (6(())1)-
(7)
This formulation enables the optimization of JLSD with respect to the policy (i.e., reinforcement learning steps) with a per-step transition reward given as:
I.SD (5t, at, $1+1) = (Ø(s)+1) - (st))
(8)
Compared to the per-trajectory reward in Equation (3) (Gregor et al., 2016), this can be optimized more easily and stably with off-the-shel RL algorithms such as SAC (Haarnoja et al., 201 8a).
Connections to previous methods. The per-step reward function of Equation (8) is closely related to continuous DIAYN (Eysenbach et al., 2019; Choi et al., 2021) and VISR (Hansen et al., 2020):
P2 DIAYN (st, Ct, 8t-1) = log o DLAYN (= (8+1)) x -1((((++1) - =||2 + (const)
(9)
VISR 5 (sr, ae 8t-1) = log VISR (ž st) Ø(s.) 2 + (const)
(10)
L. (st, Ce 8t-1) = (d($)41) - ((s,)) T2,
(11)
where 2 and ā(s:) in VISR are the normalized vectors of unit length (i.e,, ||||| and ((ss))/(((())))
respectively. We assume that DIAYN is parameterized as a normal distribution with unit variance, and q VISK von-Mises Fisher (vMF) distribution with scale parameter of (Hansen al., 2020).
While it appears that there are some similarities among Equation (9)- LSD's reward function is fundamentally different from the others in that it optimizes neither a pos-oabbbilly nor mutual information, and thus only L.SD seeks for distant states. For instance, VISR (Fquation (10)) has the most similar form as it also uses an inner product, but ((x-1) ((;)) in P.SD does not need to be a unit vector unlike VISR optimizing the vMF distribution. Instead, LSD increases the difference of á to encourage the agent to reach more distant states. In addition, while Choi et al. (2021) also use Spectral Normalization, our objective differs from theirs as we do not optimize I S): We present further diseussion and an empirical comparison of reward functions in Appendix D.
Implementation. In order to maximize the LSD objective (Equation (8)), we alternately train x with SAC (Haamoja et al., 2018a) and é with stochastic gradient descent. We provide the full procedure for LSD in Appendix C. Note that LSD has the same components and hyperparameters as DIAYN, and is thus easy to implement, especially as opposed to IBOL (Kim et al., 2021), a state-of-the- art skill discovery method on MuloCo environments that requires a two-level hierarchy with many moving components and hyperparameters.
5

---

Published as a conference paper at ICLR 2022
3.3 ZERO- SHRT SKILL SELECTION
Another advantage of the LSD objective JLSD (Equation (6)) is that the learned state representation o(s) allows solving goal-following downstream tasks in a zero-shot manner (i.e.. without any fur- ther training or complex planning). as aligned with the direction in the representation space. Although it is also possible for DIAYN-like methods to reach a single goal from the initial state in a zero-shot manner (Choi et al., 2021), LSD is able to reach goal from an arbitrary state or follow multiple goals thanks to the directional alignment without any additional modifications to the method. Specifically, if we want to make a transition from the current state s E to a target state g E S, we can simply repeat selecting the following 2 until reaching the goal:
= =<((()) - o(s)) [l((g) -(s)ll,
(12)
and executing the latent-conditionel policy T [[a]s,2) 2) to choose an action. Here, a is a hyperpa- rameter that controls the norm of z, and we find that s Ex>p(=>]]l=]] empirically works the best (e.g., a 2 ((1/2) x 1.25 for d - 2). As will be shown in Section 4.3, this zero-shot scheme provides convenient way immediately achieve strong performance on many goal-following downstream tasks. Note that, in such tasks, this scheme is more efficient than the zero-shot planning of DADS with its learned skill dynamics model (Sharma et al., 2020); L.SD does not need to learn models or require any planning steps in the representation space.
3.4 DISCRETE LSD
Continuous LSD can be extended to discovery of discrete skills. One might be tempted to use the one-hot encoding for z with the same objective as continuous LSD (Equation (6)), as in prior methods (Eysenbach et a 2019; Sharma et al., 2020). For discrete LSD, however, we cannot simply use the standard one-hot encoding. This is because an encoding that has a non-zero mean could make all the skills collapse into single behavior in LSD. For example, without loss of generality, suppose the first dimension of the mean of the N encoding vectors is > 0. Then, if the agent finds a final state ST that makes ||St soll fairly large, it simply learn the skill policy reach 8T regardless of z so that the agent can always receive a reward of c ||ST - so |l on average, by setting, e.g., ((TT) = [| soll. 0, ......... 0] and ((s)) [0, 00,0.... ... .). In other words, the agent can easily exploit the reward function without learning any diverse set of skills.
To resolve this issue, we propose using zero-centered one-hot vectors as follows:
if for i,j E (I,...,N,, (13)
Z ~ unif 22..... .Nn] where [=il] = N-1
otherwise
where N is the number of skills, [. denotes the i-th element of the vector, and unif(...) denotes the uniform distribution over a set. Plugging into Equation (8), the reward for the k-th skill becomes
LSD (st, a2 s(+)) (0(s(+1) d(s))k N :E(1.2....N11((4 r
Σ
(0(s++1) - d(s))e.
(14)
This formulation provides intuitive interpretation: it enforces the k-th element of dos) to be the only indicator for the k-th skill in a contrastive manner. We note that Equation (14) consistently pushes the difference in to be as large possible, unlike prior approaches using the softmax function (Eysenbach et al., 2019; Sharma et al., 2020). Thanks to the Lipschitz constraint on Φ(s), this makes the agent learn diverse and dynamic behaviors in the continuous case.
EXPERIMENTS
We compare LSD with multiple previous skill discovery methods on various MuloCo robotic loco- motion and manipulation environments (Todorov al., 2012; Schulman et al., 2016; Plappert al., 2018) from OpenAl Gym (Brockman et al., 2016). We aim to answer the following questions: (i) How well does LSD discover skills on high-dimensional continuous control problems, compared to previous approaches? (ii) Can the discovered skills be used for solving goal-following tasks in a zero-shot fashion? In Appendix, we present an ablation study for demonstrating the importance of LSD's components (Appendix D) and analyses of learned skills (Appendix E).
Experimental setup. We make evaluations three MuJoCo robotic locomotion environments (Ant, Humanoid, HalfCheetab) and three robotic manipulation environments (FetchPush, FetchSlide and FetchPick A AndPlace). Following the practice in previous works (Sharma et al., 2020; Kim al.,

---

Published as a conference paper at ICLR 2022
LSD (ours)
IBOL
DIAYN
DIAYN W/SN
DADS
IBOL--
DADS-XYO
VISR
APS
DIAYN (d-5)
DADS (d-5)
APS (d-5)
DIAYN-XYO ergioerring)
(a) Ant
LSD (ours)
IBOL
DADS
DIAYN (d-5)
IBOL-O (wl ins. cuneang)
DADS-XYO DIAYN-XYO (Wl loa. ergineniig
(W Teal, CRESSOUUL))
(b) Humanoid
Visualization of 2-D (or 5-D) continuous skills discovered by various methods. We plot the z-y trajectories of the agent. Each color represents the direction of the skill latent variable z. See Figure 13 for zoomed-in version.
Figure 2:
2021), we mainly compare skill discovery methods on Ant, but we additionally adopt Humamoid for quantitative comparisons with competitive baselines, since it is often considered the most chal- lenging environment the MuloCo benchmark. On the manipulation environments, we compare LSD baselines combined with MUSIC-u (Zhao et al., 2021), an intrinsic reward that facilitates the agent to have control on target objects. For continuous skills, unless otherwise mentioned, we use two-dimensional skill latents (d sampled from the standard normal distribution, following Kim et al. (2021). On the locomotion environments, we normalize the state dimensions to ensure that the different scales of the dimensions have less effect on skill discovery. We repeat all the experiments eight times and denote their 95% confidence intervals with shaded areas or error bars. We refer to Appendix for the full experimental details.
Baseline methods. We make comparisons with six skill discovery algorithms: DIAYN (Eysenbach al., 2019), DADS (Sharma al., 2020), VISR (Hansen al., 2020), EDL. (Campos Camúñez et al., 2020), APS (Liu & Abbeel, 2021a) and IBOL (Kim et 2021).
Additionally, we consider two variants of skill discovery methods: the z-y prior (denoted with the suffix "XY') and z- omission (with '-O') (Sharma al., 2020; Kim et al., 2021). The z-y prior variant restricts the input to the skill discriminator or the dynamics model only to the positional information, enforcing the agent to discover locomotion skills. The x-y omission variant excludes the locomotion coordinates from the input to policies or dynamics models (but not to discriminators) to impose an inductive bias that the agent can choose actions regardless of its location. We denote the variants that have both modifications with the suffix 'XYO'. While previous methods mostly require such feature engineering or tricks to discover skills that move consistently or have large variations in the state space, we will demonstrate that LSD discover diverse skills MuJoCo locomotion environments without using hand-engineered features.
4.1 SKILLS LEARNED WITH CONTINUOUS LSD
Visualization of skills. We train LSD and baselines the Ant environment learn two- dimensional continuous skills (d = 2). Figure 2a visualizes the learned skills as trajectories of the Ant agent on the z-y plane. LSD discovers skills that move far from the initial location al- most all possible directions, while the other methods except IBOL fail to discover such locomotion primitives without feature engineering (i.e., xyy prior) even with an increased skill dimensionality (d 5). Instead, they simply learn to take static postures rather than to move; such a phenomenon also reported in Gu et al. (2021). This is because their MI objectives do not particularly induce the agent to increase state variations. On the other hand, LSD discovers skills reaching even farther than those of the baselines using feature engineering (IBOL-O, DADS-XYO and DIAYN-XYO).
Figure 2b demonstrates the results on the Humanoid environment. As in Ant, LSD learns diverse skills walking running consistently in various directions, while skills discovered by other methods
7

---

Published as a conference paper at ICLR 2022
space coverade
7500
State coverage
4000
5000
2000
2500
100
100
Lsp
IBoL-ox
DADS xro*
DIAYN--TO
EDL rE: sage)
DADS (ded)
18ct
DIAIN (dwg)
(res)
DADS
DRAYN
APg
ViSA
LsD
IaC2
BADS YO*
EDL saape)
DADS (d-5)
DIAYS (dw5)
Ant
(b) Humanoid
Figure Plots of state space coverage. Asterisks (*) denote the methods with feature engineering.
SkILL
HalfCheetah (N=8)
Ant (N 16)
Humanoid (N 16)
Figure 4: Qualitative results of discrete LSD (Section 4.2). We visualize each skill's trajectories on
the x axis (HalfCheetah) the z-y plane (Ant and Humanoid), See Appendix for more results.
iZero shot
Dwward
(wr orginnsrinnt DADS enginice flsts ong neefing
0
50K
100K
150K
50K
100K
150%
50K
100K
156K
50K eppsodes
300K
150K
eplsodes
esisodes
eppsodes
(a) AntGoal
(b) AntMultiGoals
HumanoidGoal (d) HumanoidMulliGaals
Figure 5: Performance on downstream tasks after skill discovery (Section 4.3).
are limited in terms of the state space coverage or the variety of directions. We provide the videos of skills discovered by LSD at https : // shpark. me meeittme/ppae /ttp//poetttseeeee /1sd/.
Quantitative evaluation. For numerical comparison, we measure the state space coverage (Kim al., 2021) of each skill discovery method on Ant and Humanoid. The state space coverage is measured by the number of occupied 1 x bins on the x-y plane from 200 randomly sampled trajectories, averaged over eight runs. For EDL, we report the state space coverage of its SMM exploration phase (Lee et al., 2019). Figure 3 shows that on both environments, LSD outperforms all the baselines even including those with feature engineering.
4.2 SKILLS LEARNED WITH DISCRETE L.SD
We train discrete LSD on Ant, HalfCheetah and Humanoid with N 6. 16, where N is the number of skills. While continuous LSD mainly discovers locomotion skills, we observe that dis- crete LSD learns more diverse skills thanks to its contrastive scheme (Figure 4, Appendix J). On Ant, discrete LSD discovers skill set consisting of five locomotion skills, six rotation skills, three posing skills and two flipping skills. On HalfCheetah, the agent learns to run forward and backward in multiple postures, roll forward and backward, and take different poses. Finally, Humanoid learns to run or move in multiple directions and speeds with unique gaits. We highly recommend the reader check the videos available on our project page. We refer Appendix G. for guantitative evaluations. To the best of our knowledge, LSD is the only method that can discover such diverse and dynamic behaviors (i.e.. having large traveled distances in many of the state dimensions) within single set of skills in the absence of feature engineering.
4.3 COMPARISON ON DOWNSTREAM TASKS
As done in Eysenbach al. (2019); Sharma al. (2020); Kim al. (2021), make comparisons on downstream goal-foll foll-following tasks to assess how well skills learned by LSD can be employed for solving tasks in hierarchical manner, where evaluate approach not only AntGoal and AntMultiGoals (Eysenbach et al., 2019; Sharma et al., 2020; Kim et al., 2021) but also on the more challenging tasks: HumanoidGoal and HumanoidMultiGoaal In the G-Gaa"" tasks, the agent should
8

---

Published as a conference paper at ICLR 2022
Lea
Dany
FetchPush
FeschSlide
AendPPickAnnPPacc
(b) State space coverage.
PatckSlide
MUSICU
HU/DCC
LSD
DAT5
LSD (ours)
DADS-MURCC
DADS
DIAYN-WEIE
DIAYN
spkandes
Godl
(a) Visualization of trajectories of the target object the z-y plane (i.e., the table) in three Fetch manipulation environments.
(c) Average reward on downstream tasks.
Each color represents the direction of the skill latent variable z.
Figure Comparisons on the Fetch robotic manipulation environments (Section 4.4).
reach a uniformly sampled random goal on the z-y plane, while in the '-MultiGoals" tasks, the agent should navigate through multiple randomly sampled goals in order. The agent is rewarded only when it reaches a goal. We refer to Appendix for the full details of the tasks.
We first train each skill discovery method with continuous skills (without rewards), and then train hierarchical meta-controller on top of the learned skill policy (kept frozen) with the task reward. The meta-controller observes the target goal concatenated to the state observation. At every K-th step, the controller selects a skill [-2, 2| to be performed for the next K steps, and the chosen skill is executed by the skill policy for the K steps. We also examine zero-shot skill selection of LSD, denoted 'LSD (Zero-shot),, where the agent chooses & at every step according to Equation (12).
Results. Figure 5 shows the performance of each algorithm evaluated on the four downstream tasks. LSD demonstrates the highest reward in all of the environments, outperforming even the baselines with feature engincering. On top of that, in some environments such as AntMultiGoals, LSD's zero- shot skill selection performs the best, while still exhibiting strong performance on the other tasks. From these results, we show that LSD is capable of solving downstream goal-following tasks very efficiently with no further training or complex planning procedures.
4.4 EXPERIMENTS ON ROBOTIC MANIPULATION ENVIRONMENTS
In order to demonstrate that LSD can also discover useful skills in environments other than locomo- tion tasks, we make another evaluation on three Fetch robotic manipulation environments (Plappert et al., 2018). We compare LSD with other skill discovery methods combined with MUSIC-u (Zhao et al., 2021), an intrinsic reward that maximizes the mutual information 1(5"; S*) between the agent state 5 and the surrounding state S For fair comparison with MUSIC, make use of the same prior knowledge for skill discovery methods including LSD to make them focus only on the sur- rounding states (Zhao al., 2021), which correspond the target object in experiments.
Results. Figure 6a visualizes the target object's trajectories of 2-D continuous skills learned by each algorithm. They suggest that LSD can control the target object in the most diverse directions. No- tably, FetchPickAndPlar task reward or intrinsic motivation like MUSIC-u. Figures 6b and 6c present quantitative evaluations on the Fetch environments. As in Ant and Humanoid, LSD exhibits the best state space coverage (measured with 0.1 0.1-sized bins) in all the environments. Also, LSD and LSD (Zero-shot) outperform the baselines by large margins on the three downstream tasks.
LSD learns to pick the target object in multiple directions without any
CONCLUSION
We presented LSD as an unsupervised skill discovery method based on a Lipschitz constraint. We first pointed out the limitation of previous MI-based skill discovery objectives that they are likely to prefer static skills with limited state variations, and resolved this by proposing a new objective based on a Lipschitz constraint. It resulted learning more dynamic skills and a state represen- tation function that enables zero-shot skill selection. Through multiple quantitative and qualitative experiments on robotic manipulation and complex locomotion environments, we showed that LSD outperforms previous skill discovery methods in terms of the diversity of skills, state space coverage and performance on downstream tasks. Finally, we refer the readers to Appendix A for discussion of limitations and future directions.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their helpful comments. This work was supported by Sam- sung Advanced Institute of Technology, Brain Research Program by National Research Foundation of Korea (NRF) (2017M3C7A 1047860), Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. 2019-0-01082,
SW StarLab), Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-01344 Artificial Intelligence Graduate School Program (Seoul National University)) and NSF CAREER IIS- 1453651.. J.C. was partly sup- ported by Korea Foundation for Advanced Studies. J.K. was supported by Google PhD Fellowship. Gunhee Kim is the corresponding author.
REPRODUCIBILITY STATEMENT
We make our implementation publicly available the repository at https://visionn snu. fe.kk/prooectts/// /Isd/ and provide the full implementation details in Appendix I.
REFERENCES
Joshua Achiam, Harrison Edwards, Dario Amodei, and Pieter Abbeel. Variational option discovery algorithms. ArXiv, abs/1807. 10299, 2018.
Marcin Andrychowiez. Bowen Baker, Maciek Chociej. R. Józefowicz, Bob McGrew, Jakub W. Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, S. Sidor, Joshua Tobin, P. Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39:20 - 3, 2020.
Adrià Puigdomenech Badia, Bilal Piot, Steven Kapturowski, Sprechmann, Alex Vitvitskyi, Daniel Guo, and C. Blundell. Agent57: Outperforming the atari human benchmark. In ICML, 2020.
D. Barber and F. Agakov. The IM algorithm: a variational approach to information maximization. In NeurlPS, 2003.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado Van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. In NeurlPS, 2017.
G. Brockman, Vicki Cheung, Ludwig Pettersson, Schneider, John Schulman, Jie Tang, and W. Zaremba. OpenAl Gym. ArXiv, abs/1606.01 2016.
Víctor Campos Camúñez, Alex Trott, Caiming Xiong, Richard Socher, Xavier Giró Nieto, and Jordi Torres Viñals. Explore, discover and learn: unsupervised discovery of state-covering skills. In ICML, 2020.
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Gu. Variational empow- erment as representation learning for goal-conditioned reinforcement learning. In ICML, 2021.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural Computation, 5:613-624 1993.
Gabriel Dulac-Arold,, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. ArXiv, abs/1904.1 2901, 2019.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without reward function. In ICLR, 2019.
The garage contributors. Garage: A toolkit for reproducible reinforcement learning research. 2019.
https: /ttp::////hhhli httpaa/.mmmmmwwwwgoooo//////// 2019.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. ArXiv, abs/1611 1,005777, 2016.
10

---

Published as a conference paper at ICLR 2022
Shixiang Gu, Ethan Holly, T. Lillicrap, and S. Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- off-policy updates. In ICRA, 2017.
Shixiang Shane Gu, Manfred Diaz, Daniel C. Freeman, Hiroki Furuta, Seyed Kamyar Seyed Ghasemipour, Anton Raichuk, Byron David, Erik Frey, Erwin Coumans, and Olivier Bachem. Braxlines: Fast and interactive toolkit for rl-driven behavior engineering beyond reward maxi- mization. ArXiv, abs/21 10.04686, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018a.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, G. Tucker, Schoon Ha, Jie Tan, Vikash Ku- mar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algorithms and applications. ArXiv, abs/ 812.05905, 2018b.
Dylan Hadeield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In NeurlPS, 2017.
S. Hansen, Will Dabney, André Barreto, Wiele, David Warde-Farley, and V. Mnih. Fast task inference with variational intrinsic successor features. In ICLR, 2020.
Jackyeom Kim, Scohong Park, and Gunhee Kim. option learning. ICML, 2021.
Unsupervised skill discovery with bottleneck
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhut- dinov. Efficient exploration via state marginal matching. ArXiv, abs/1906.05274, 2019.
Youngwoon Lee, Jingyun Yang, and Joseph J. Lim. Learning to coordinate manipulation skills via skill behavior diversification. ICLR, 2020.
Hao Liu and Pieter Abbeel. APS: Active pretraining with successor features. In ICML., 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. ArXiv, abs/2103.04551, 2021b.
Marlos C. Machado, Mare G. Bellemare, and Michael Bowling. A laplacian framework for option discovery in reinforcement learning. ArXiv, abs/1703.00956, 2017.
Marlos C. Machado, C. Rosenbaum, Xiaoxiao Guo, Miao Liu, G. Tesauro, and Murray Campbell. Eigenoption discovery through the deep successor representation. ArXiv, abs/1710.1 1089, 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization
for generative adversarial networks. In ICLR, 2018.
Shakir Mohamed and Danilo J. Rezende. Variational information maximisation for intrinsically motivated reinforcement learning. In NeurlPS, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David A. McAllester, and Nathan Srebro. A PAC- Bayesian approach to spectrally-normalizee normalized margin bounds for neural networks. In ICLR, 2018.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Macick Chociej. Peter Welinder, Vikash Kumar, and Wojciech Zaremba. Multi-goal reinforcement learning: Challenging robotics environments and request for research. ArXiv, abs/1802.09964, 2018
A. H. Qureshi, Jacob J. Johnson, Yuzhe Qin, Taylor Henderson, Byron Boots, and Michael C. Yip. Composing task- aggossic policies with deep reinforcement learning. In ICLR, 2020.
Julian Schrittwieser, Ioannis Antonoglou, Hubert, K. Simonyan, L. Sifre, Simon Schmitt, A. Guez, Edward Lockhart, Demis Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Master- ing Atari, Go, Chess and Shogi by planning with a learned model. Nature, 588 7839:604-609, 2020.

---

Published as a conference paper at ICLR 2022
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In JCLR, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.0653 2017.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman, Dynamics-awand unsupervised discovery of skills. In ICLR, 2020.
Jure Sokolic, Raja Giryes, Guillermo Sapiro, and Miguel R. D. Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65:4265-4280, 2017.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transac- fions Neural Networks, 16:285-286, 2005.
Naftali Tishby, Fernando C. Pereira, and William Bialek. The information bottleneck method. ArXiv, physics/0004057, 2000.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physies engine for model-based control. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, IEEE, 2012.
D. Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turnrr, and A. Madry. There is no free lunch in adversarial robustness (but there are unexpected benefits). ArXiv, abs/1805, 1180,11152,, 2018.
Jesse Zhang, Haonan Yu, and Wei Xu. Hierarchical reinforcement learning by discovering intrinsic options. ICLR, 2021.
Rui Zhao, Yang Gao, Pieter Abbeel, Volker Tresp, and Wei Xu. Mutual information state intrinsic control. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2022
APPENDIX: LIPSCHITZ--CNSSTRAANN UNSUPERVISED SKILL DISCOVERY
A LIMITATIONS AND FUTURE DIRECTIONS
LSD may not encourage dynamic behaviors in some environments where Lipschitz constraints are not semantically meaningful, such control from pixel observations. This issue is addressable by incorporating representation learning, which we leave for future work. Also, in contrast to discrete LSD that learns more diverse skills, continuous LSD mainly discovers locomotion skills that move as far as possible, since its objective is independent of the magnitude of 2. While this has an advantage in that we can later choose skills just by their directions (enabling zero-shot skill selection), making LSD respect the magnitude as well can be another interesting research direction. Finally, while LSD does not use any explicit feature engineering, we note that the skills LSD learns are still affected by the shape of maximally reachable regions in the (normalized) state space.
B EXTENDED RELATED WORK
The LSD reward (Equation (8) might look very similar to the objective of eigenoptions (Machado et al., 2017; 2018) 8): (Ø(s++1) - ((s))) e, where & is fixed (or pre rre-trained) representation function of states and e is an eigenvector of the successor representation matrix (Dayan, 1993) computed from fixed random policy. They define eigenoptions as the options (or skills) that maximize this reward for each the eigenvectors e of the N largest eigenvalues. However, our interest differs from their setting, since we learn both the policy and the representation function in order to seek diverse and dynamic skills, and our approach applicable to continuous skill settings as well as discrete skill learning.
C TRAINING PROCEDURE FOR LSD
Algorithm 1: Lipschitz-consstraneee Skill Discovery (LSD)
Initialize skill policy and representation function o; while not converged do for i l...., ... episodes per epoch) do
Sample skill z from p(=); Sample trajectory (episode) with 2) and 2;
Compute reward (1,SD 42 8141) (Ø(s=-)) T(s())=z (Equation (8));
Update 6 using SGD to maximize Equation (8) under Spectral Normalization;
Update using SAC:
Algorithm overviews the training procedure for LSD. There are two learnable components in LSD: the skill policy π (a]s, 2) and the representation function Φ(s). In order to impose the 1-Lipschitz constraint on 6, we employ Spectral Normalization (SN) (Miyato et al., 2018). At every epoch, we alternately train π with SAC (Haarnoja et al., 2018a) and ái with stochastic gradient descent (SGD) to maximize Equation (8). When collecting trajectories т, we fix within single episode as in previous works (Eysenbach et al., 2019; Sharma al., 2020; Kim et al., 2021).
13

---

Published as a conference paper at ICLR 2022
w/o 1- L-Lischhitz constraint on +
w/ Lipsschitz constraint on
Normal Dist.
DIAYN
DIAYN
<MF Dist.
VISR
Inner
LSD
((x)
((6)
018
6(8)
((x)
(6
6(8 8) 6(60) ((6)
Figure 7: Ablation study on Ant. See Appendix D.
D ABLATION STUDY
The LSD reward (Equations (6) and (8)) consists of three components: the inner-product form, the of d's difference and the I-Lipschitz constraint on Φ. In order to examine the role and importance of each component, we ablate them with several variants. We consider all possible combinations of the following variations:
(i) The form of the reward function specifying how and (6() are compared:"
(1) the Normal distribution form (as DIAYN, Equation (9)), i.e., the squared distance,
(2) the von-Mises Fisher (vMF) distribution form (as in VISR, Equation (10)), ie., the inner-
product form with normalizing the norms of (0-) and z, or
(3) the inner-product form (as in LSD, Equations (8) and (11)) without normalizing the norms.
(ii The use of the current and/or next states: Φ(s). ((s') or ó(s' - in place of c(s') o(s) (wheres s' denotes the next state).
(iii) The of Spectral Normalization: with without the 1-Lipschitz constraint on d(.)-
Figure/ 7 shows the result the ablation study on the Ant environment. We observe the state space coverage drastically decreases if any of these components constituting the LSD reward (Fqua- tion (8)) is missing: ie., all the three components are crucial for LSD. Especially, just adding Spectral Normalization (SN) the DIAYN objective ('DIAYN w/ SN', Choi al. (2021)) does not induce large state variations, since its objective does not necessarily encourage the scale of o(s) to increase. We also note that the purposes of using SN in our work and in Choi al. (2021) are very different. While Choi et al. (2021) employ SN to regularize the discriminator for better stability, we use SN to lower-bound the state differences (Equation (6)) so that maximizing the LSD objective always guarantees an increase in state differences.
For DIAYN (Eysenbach et al., 2019), omit the - logp(()) term from the original objective as its ex- pectation can he treated as constant assuming that p(2) is a fixed prior distribution and that episndes have the length. For VISR (Hansen al., 2020), only consider its unsupervised skill discovery objective.
14

---

Published as a conference paper at ICLR 2022
0.5
Ant (2-D continuous skills)
0.0
State dim.
0.5
0.5
Humanoid
(2-D continuous skills)
To State dim.
0.0
-0.5
-00
Visualization of skills
((8T) ((80)
Correlation matrix
Figure 8: Analysis 2-D continuous skills discovered by LSD for Ant and Humanoid.
1.0
0.5
Ant (3-D continuous skills)
State dim.
0.5
1.0
1.0
0.5
Ant (4-D continuous skills)
-05
1.0
Ant (5-D continuous skills)
Latent
State dim.
-05
1.0
Visualization of skills
( ((so)
Correlation matrix
Figure 9: Analysis of 3-D, 4-D and 5-D continuous skills discovered by LSD for Ant. We plot only the first two latent dimensions for the ((ST) ((50) figures, and each color also represents only the first two dimensions.
E VISUALIZATION AND ANALYSES OF SKILLS LEARNED
We provide more visual examples and analyses of skills discovered by LSD (Figures 8 to 12, also videos available at https:/ www.rkaaaak.kkk me/ pttp//oooecttee lsd/).
Figure visualizes 2-D continuous skills for Ant and Humanoid, and the learned state representation function é, and demonstrates the correlation coefficient matrices between the state dimensions and skill latent dimensions. We observe that continuous LSD focuses on the x-y coordinates (the first and second state dimensions) both the environments, which is attributable the fact that those two
15

---

Published as a conference paper at ICLR 2022
HalfCheetah (6 discrete skills)
Latent
0.0
State dim.
-0.5
0.5
HalfCheetah
(8 discrete skills)
State dim.
-1.0
Figure 10: Analysis of discrete skills discovered by LSD for HalfCheetah.
1.0
Latent clm..
0.5
Ant (6 discrete skills)
State dim.
0.5
1.0
1.0
Latent clim.
Ant (8 discrete skills)
0.0
State
-05
1.0
1.0
Latent dim.
0.5
Ant (16 discrete skills)
0.0
State dim.
Visualization of skills
Correlation matrix
Figure 11: Analysis of discrete skills discovered by LSD for Ant.
dimensions are the most suitable ones for increasing ((sT) - ((sg) under the Lipschitz constraint. Figure also shows that à Ant has almost no correlation with the 8th to 15th state dimensions, which correspond to the angles of the four leg joints. This because Ant should repeatedly move consistently in direction.
its legs back and forth to
We also experiment continuous LSD with d 3, 4, 5 on Ant. Figure demonstrates that LSD still mainly learns locomotion skills as in d = 2. However, in this case, some skills represent the same direction since there exist more skill dimensions than x and y dimensions. To resolve this, we believe combining continuous LSD with contrastive learning (as in discrete LSD) can be possible solution, which we leave for future work.
Figures 10 to 12 visualize discrete skills learned for HalfCheetah, Ant and Humanoid. We confirm that discrete LSD focuses not only the x-y coordinates but of more diverse dimensions.
16

---

Published as a conference paper at ICLR 2022
0.5
Humanoid discrete skills)
Latent
0.0
State dim.
-0.5
110
0.5
Humanoid (16 discrete skills)
Latent
0.0
State dim.
-0.5
1.0
Visualization of skills
Correlation matrix
Figure 12: Analysis of discrete skills discovered by LSD for Humanoid.
For instance, the example 'Ant (16 discrete skills)' in Figure 11 shows that some skills such as the second and third ones have large correlations with the orientation dimensions (the 4th to 7th state dimensions) Ant. These skills correspond to rotation skills, as shown in the video on our project page.
F ENLARGED VISUALIZATION OF SKILL DISCOVERY METHODS
Figure 13 (a zoomed-in version of Figure visualizes the learned skills on the z-y plane. Note that each plot has different axes.
50
20
-25
-50
-50
-20
20
0.550.0 0.5
-2.5
0.0
-50
50
-25 0
25
LSD (ours)
IBOL
DIAYN
DIAYN w/ SN
DADS
IBOL-O
MADS-XYO for enginoring)
feal claisecii))
2.5
-2
-2.5
-2
0.0
2.5 APS
-2.5
0.0
0.0 2.5
2.5 DIAYN-XYO (wl Ice anginenn
-2. 0.0
VISR
DIAYN (d-5)
DADS
APS (d-5)
(a) Ant
10
-50
-10
50
0
20
50
50
-10
LSD (ours)
IBOL
(d-5)
DIAYN (d-5)
IBOL-O
DAOS-XYO
DIO
(b) Humanoid
Figure 13: Zoomed-in version of Figure 2. We visualize 2-D (or 5-D) continuous skills discovered by various methods by plotting the z-y trajectories of the agent. Each color represents the direction of the skill latent variable 2. Note that each plot has different axes. This result shows that most of the existing approaches (DIAYN, VISR, APS, DIAYN-XYO, etc.) cannot learn far-reaching locomotion skills, shown in Figure 17.
17

---

Published as a conference paper at ICLR 2022
100
LSD
7.5
DIAYN
Hegoard
DADS
5.0
500K episodes)
DIAYN (at S00K episodes)
DADS 500K episodes)
64K episodes
128K
646
128K
64K
126K
apisodes
(a) CheetahHurdle
(b) AntRotation
(c) AntQuaternion
Figure 14: Training curves of LSD, DIAYN and DADS on three non-locomotion downstream tasks after skill discovery.
(Zero-shot)
4
DIYI (d=3) (tereshoti
DAII (di-1)
50K
100K
150K
0
50K
1OOK
150K
0
50K
10CK
150K
50k
100K
150K
<pilodes
# apicodes
# opicodes
(a) AntGoal
(b) AntMultiGaals
(c)
HumanoidGoal (d) HumanoidMulliioosls
Figure 15: Training curves of LSD and DIAYN on four goll-wollowing downstream tasks after skill discovery.
G ADDITIONAL DOWNSTREAM TASKS
G.1 NON-LOCOMOTION DOWNSTREAM TASKS
In order to examine the performance of LSD on more diverse downstream tasks, we make quantita- tive evaluations of discrete skill discovery methods on non-locomotion environments. Specifically, we test discrete DIAYN, DADS and LSD on three different tasks: CheetahHurdle (with N 8) AntRotation and AntQuaternion (with N 16).
In Chrela the HalfCheetah agent should move forward while jumping over evenly spaced hurdles, where employ the same hurdle configuration used in Qureshi al. (2020). The agent given a reward of every time successfully jumps over a hurdle. In AntRotation, the Ant agent should rotate in place to reach randomly sampled angle on the x-y plane. AntQuaternion is the 3-D version of AntRotation, where the agent should rotate or flip to match a randomly sampled rotation quaternion. In both environments, the agent receives reward of when the angle between the target orientation and the current orientation becomes smaller than threshold. Specifically, in AntRotation, we first project both the target angle and the z-axis rotation angle of the agent onto the unit circle and compute the Fuclidean distance between them. If the distance becomes less than 0.05, the agent gets a reward and the episode ends. In AntQuaternion, we compute the distance between two quaternion using d(g1, 92) between the quaternions. When d(g1.92) becomes smaller than 0.3, the agent receives a reward and the episode ends.
(41,92) where (g1. 92) denotes the inner product
Results. Figure 14 shows the results the three downstream tasks, where report both the per- formances of skill discovery methods trained with 500K episodes (= 25K epochs) and 2M episodes of the envi-
(= 100K epochs). Figure 14 demonstrates that LSD exhibits the best performance ronments, suggesting that LSD is capable of learning more diverse behaviors other than locomotion skills well. Also, we observe that the performances of DIAYN and DADS decrease as training of skill discovery progresses in CheetahHuddle. This is mainly because the MI objective (7(Z:S) they use usually prefers more predictable and thus static skills, making the agent incline to posing skills rather than moving or jumping. On the other hand, since LSD's objective always encourages larger state differences, its performance increases with training epochs.
G.2 ADDITIONAL ZERO-SHOT EVALUATION
In this section, we additionally test the zero-shot scheme for DIAYN-like methods (Choi et al., 2021) goal-following downstream tasks. DIAYN-like methods also select skill to reach
18

---

Published as a conference paper at ICLR 2022
Table 2: Comparison of zero-shot performances on PointGoal.
Method
PointGoal (g. = 10)
PointGoal (g. = 20)
PointGoal (g. = 40)
PointGoal (g. = 80)
LSD
1.00 + 0.00
1.00 + 0.00
1.00 + 0.00
0.92 + 0.03
DIAYN
0.41 + 0.05
0.20 + 0.03
0.12 + 0.03
0.05 + 0.02
DIAYN-XYO
1.00 + 0.00
0.80 + 0.01
0.35 + 0.01
0.15 + 0.01
Table 3: Comparison of zero-shot performances on PointMultiGoals.
Method
PointMultiGoals (gm. 10)
PointMultiGoals (gm 20)
PointMultiGoals (ym. 40)
LSD
4.00 0.00
4.00 + 0.00
3,85 1 0.05
DIAYN
1.54 0.18
0.82 + 0.11
0.43 0.08
DIAYN-XYO
2.09 + 0.03
1.27 + 0.02
0.57 + 0.02
target goal 2 in a zero-shot manner by setting 2 = ===lg'=')))],, where denotes its skill discrim- inator. However, in contrast LSD,, which follow multiple goals from arbitrary state with its direction-aligned representation function (Section 3.3), DIAYN-like methods could usually only reach single goal from the initial state because they trained reach the absolute position of the goal with the discriminator ((-1g). Although it may be possible for such methods to deal with unseen goals initial states if we train them with broad range of initial states, this modification could harm the performance as it requires more training data. On the other hand, LSD's 's directional goal scheme enables reaching unseen goals in a zero-shot fashion even if it is trained with fixed initial state. Also, while they require the skill = Eg [u(='[g) [[g]] to be in the vicinity of the prior p(z), LSD free from this constraint as can normalize the skill obtained from Equation (12).
G.2.1 RESULTS ON MuJoCo LOCOMOTION ENVIRONMENTS
Figure 15 demonstrates the performances of LSD, DIAYN and its zero-shot schemes on AntGoal, AntMultiGoals, HumanoidGoal and HumanoidMuliGGals. It suggests that the performance of DIAYN's s zero-shot scheme falls behind DIAYN in the HumanoidMuliGGoals environment, while LSD's s zero-shot performance is mostly comparable or better than LSD, outperforming DIAYN in all the environments.
G.2.2 RESULTS ON POINTENV
In order fairly compare only the zero-shot scheme of each method, make additional com- parisons on a toy environment named PointEnv (Kim et al., 2021). PointEnv is a minimalistic environment in which the state of the agent is defined its z-y position and action denotes the direction in which the agent moves. If the agent performs action = (az, agy) E []1, on state (Sz, sy) E R2, its next state becomes (s,, + ag + ay). Unless otherwise mentioned, the initial state given as (0, 0). We train each skill discovery method with episode length of 10 in this environment. We sample z from the 2-D standard normal distribution.
We prepare two goal-following downstream tasks: PointGoal and PointMultiGoals, which are simi- lar to the '-Gaal' or '-MuliGooals' environments in Section 4.3. In PointGoal, the agent should reach a goal g uniformly sampled from 1-9.. 9.12 within 100 environment steps. In PointMultiGoals, the agent should follow four goals within 400 environment steps (we refer to Appendix 1.1.5 for the full details of -MultiGoals' environments). The agent receives reward of I when it reaches goal.
In these environments, initial states sampled from [-10, 10)2, as well as DIAYN-XYO with a fixed initial state. When training each skill discovery method, we employ two-layered MLP with 128 units for modelling trainable components, and train the models for 250K episodes (= 5K epochs) with four minibatches consisting of 500 transitions from 50 trajectories at every epoch. For downstream tasks, we set 9. {10, 20, 40, 80} and 9m E {10, 20, 40).
test the zero-shot schemes of LSD and DIAYN trained with random
Tables 2 and 3 demonstrate the final average zero-shot performance of each skill discovery method, averaged over eight independent runs with the standard error. The result shows that although both LSD and DIAYN-XYO have the same 'base' performance, achieving the maximum reward with g. = 10 (i.e., when goals are sampled only from the states encountered during training). DIAYN- XYO's performance degrades g, increases (i.e., given previously unseen goals). We speculate one
19

---

Published as a conference paper at ICLR 2022
Auarage disance
Avarage distance
Average distance
Averoge disaacce
400
150
200-
20
200
50-
LSD
DADS
DIAIH
Randen
Lsp
DADS
DIAIN
Randam
DADS (d-5)
LSD
DlAYs (dw5)
DADS
DIAIN Random
LSD
dAoS (d=S)
DAAN (des)
Random
HalfChectah
Ant
Ant
Humanoid (continuous skills)
(8 discrete skills)
(16 discrete skills)
(continuous skills)
Figure 16: Average normalized state difference ||ST
soll of skill discovery methods.
reason behind this is that the probability of some chosen skill 2 ~ p(z) in DIAYN becomes smaller when it encounters a previously unseen goal, which could lead to an unexpected behavior (for exam- ple, notice that the average norm of DIAYN's selected skills is approximately 3.84 when g. 40, which is unlikely to be sampled from the standard normal distribution). Also, the result suggests that training with broad range of the initial state distribution indeed harms the performance of DIAYN. Finally, LSD's zero-shot scheme outperforms DIAYN's on the three PointMultiGoals settings by large margins, indicating that DIAYN's zero-shot scheme (at least empirically) could not cope well with goal following settings (i.e., reaching multiple goals in order).
H QUANTITATIVE EVALUATIONS OF ||St -soll
In order to empirically demonstrate that previous methods using the MI objective T(Z; S) might end up learning only static or simple skills, measure the average difference between the (normalized) initial and final states ||ST so on MuJoCo environments. Figure 16 shows that existing MI-based methods indeed prefer less dynamic skills, in contrast LSD. Notably, in HalfCheetah and Ant, the state differences of DIAYN and DADS after skill discovery become even smaller than a random policy. This natural because it is better for MI-based methods have more predictable, static tra- jectories so that they can accurately restore the skill from the corresponding states (Equation (3)).
I IMPLEMENTATION DETAILS
1.1 MuJoCo LOCOMOTION ENVIRONMENTS
111.1 SETTINGS
We implement LSD and most of the baselines top of the garage framework (garage contributors, 2019), while we train the E stage (Lee et al., 2019) of EDL (Campos Camúñez et al., 2020) using their released codebase2. We provide the implementation for LSD the repository https: //vision. snu ac /rrkkr//a
For the environments used in our experiments, we use the same configurations adopted in Sharma et al. (2020); Kim et al. (2021), with a maximum episode length of 200 for Ant and HalfCheetah, and 1000 for Humanoid. We normalize each state dimension of the environments ahead of skill discovery with the statistics used Kim et al. (2021), where they compute the mean and standard deviation from 10000 rollouts of random policy.
1.1.2 VARIANTS OF SKILL DISCOVERY METHODS
DIAYN-XYO (Eysenbach et al., 2019) limits the input to the skill discriminator g(>ls) to the x- y dimensions and omits them from the input to its skill policy. For DADS-XYO (Sharma et al., 2020), we make its skill dynamics model q(s' 2) only consider the x-y coordinates and exclude them from the inputs both the skill policy and the skill dynamics model, as done in Sharma et al. (2020). IBOL-O (Kim et al., 2021) denotes the exact setting used in their work, where they omit the x-y coordinates from the input to low-level policy.
Phtpps: :/g thub. www.vicccvvii..oeeeooo wtt.:///vvtoroocpccc7.. /ed1
20

---

Published as a conference paper at ICLR 2022
1.1.3 TRAINING OF SKILL DISCOVERY METHODS
We model each trainable component as a two-layered MLP with 1024 units, and train them with SAC (Haarnoja et al., 2018a). At every epoch, we sample 20 (Ant and HalfCheetah) or 5 (Humanoid) rollouts and train the networks with 4 gradient steps computed from 2048-sized mini-batches. For quantitative evaluation of learned skills, we train the models for 2M episodes (= 100K epochs, Ant and HalfCheetah) 1M episodes (= 200K epochs, Humanoid).
For each method, we search the discount factor y from {0.99, 0.995) and the SAC entropy coeffi- cient from (0.003, 0.01, ,0.03, 0.1, 0.3, 1.0, auto-adjust (Haarnoja et al., 2018b)} For continuous skills on Ant, we use 7 = 0.995 for the low-level policy of IBOL.-) and 0.99 for the others, and use an auto-adjusted a for DADS, DIAYN, VISR and APS, (d = 5), and a = 0.3 for DIAYN (d- = 5). For discrete skills, we set a to 0.003 (Ant with N = 16) or 0.01 (HalfCheetah with N 8) for LSD and use auto-adjusted for DADS and DIAYN. We set the default learning rate to 1e- - 4, but 3--5 5 for DADS's s q. DIAYN's and LSD's ' O. and 3e -4 for IBOL's low-level policy. On Ant and HalfCheetah, we train the models with on-policy samples without using the replay buffer (following Sharma et al. (2020); Kim et al. (2021)), while we use the replay buffer for sampling the k(= 5)-nerestt neighbors in APS. For the low-level policy of IBOL-O, we additionally normalize rewards and use full-sized batches, following Kim et al. (2021). In the case of IBOL, we used setting performs better. We use the original hyperparameter choices for their high-level policies.
0.01 for LSD, a 0.03 for DADS
of size 1024 and do not normalize rewards we find this
On Humanoid, we set the discount factor to 7 0.99 and the learning rate to 3e 4, but 1e 4 for DADS's q, DIAYN'S q and LSD's ø. Also, we use the replay buffer and additionally give an alive bonus b at each step (following Kim et al. (2021)) searched from f0, 0.03, 0.3), while we find that b does not significantly affect the performance. We use a = 0.03 for LSD and an auto-adjusted a for the others, and b for DADS and 0.03 for the others.
I.1.4 TRAINING OF SMM
We train the SMM exploration stage (Lee et al., 2019) of EDL (Campos Camúñez et al., 2020) using the official implementation with hyperparameters tuned. Specifically, we set the discount factor y to 0.99 (Ant and Humanoid), the SAC entropy coefficient a to 1 (Ant) or 3 (Humanoid), the 3- coefficient to 1, the alive bonus to 30 (only for Humanoid), and the density coefficient v to 0.5 (Ant) or 0.05 (Humanoid).
1.1.5 DOWNSTREAM TASKS
In AntGoal and HumanoidGoal, a random goal g E [-9s.9.]² is given at the beginning of each episode, and the episode ends if the agent reaches the goal (i.e., the distance between the agent and the goal becomes less than or equal to é). In AntMultiGoal and HumanoidMuliiGoall, (up to X) new goals are sampled within the relative range of [-9m: 9m. from the current 2-D coordinates when the episode begins, the agent reaches the current goal, or the agent does not reach the goal within Y steps. In all of the environments, the agent receives a reward of r when it reaches the goal (no reward otherwise). We use g.- = 50, 9m 15, 3. X = 4 and Y = 50 for AntGoal and AntMultiGoals, and gs- = 20, 9m = 7.5, = 3, X = 4 and Y = 250 for HumanoidGoal and HumanoidMuliiGoals. Also, we set to 10 ("-Goals" environments) or 2.5 ('-MultiGoals' environments).
1.1.6 TRAINING OF HIERARCHICAL POLICIES FOR DOWNSTREAM TASKS
For downstream tasks, we train a high-level meta-controller on top of a pre-trained skill policy with SAC (Haarnoja et al., 2018a) for continuous skills or PPO (Schulman et al., 2017) for discrete skills. The meta-controller is modeled as an MLP with two hidden layers of 512 dimensions. We set K to 25 (Ant and HalfCheetah) or 125 (Humanoid), the learning rate to 3e 4, the discount factor to 0.995, and use an auto-adjusted entropy coefficient (SAC) or an entropy coefficient of ().1 (PPO). For SAC, we sample ten trajectories and train the networks with four SAC gradients computed from full-sized batches at every epoch. For PPO, we use 64 trajectories, ten gradients and 256-sized mini- batches. We normalize the state dimensions with an exponential moving average. For zero-shot learning, set g in Equation (12) to the current state with its locomotion dimensions replaced with the goal's coordinates. Additionally, only for Figure 1c, we set the locomotion dimensions of the input to the pre-trained low-level policy to for better visualization.
21

---

Published as a conference paper at ICLR 2022
(-1.86, 1.06)
1.06, 1.00)
(100, 1.3)
(..)
1.06)
1.06)
2- (- 0.0)
(-1-5, 1.5,
(3.0,
(0.0,0.)
= 0.0)
== (1.5,00))
(-1.06 -1.0)
(-106 -1005)
2- (0.0)
(0.0, 1.5)
=- (1.06. 1.06)
(1.06 1.06)
(a) DIAYN
(b) LSD
Figure 17: 2-D continuous skills for Ant. DIAYN discovers posing skills, as its mutual information objective does not necessarily prefer large state variations. On the other hand, LSD encourages the agent to have more variations in the state space, resulting in learning more dynamic behaviors such as locomotion skills. Videos are available on our project page.
I.2 MuJoCo MANIPULATION ENVIRONMENTS
I.2.1 TRAINING OF SKILL DISCOVERY METHODS
For MuJoCo manipulation environments (FetchPush, FetchSlide, FetchPickAndPlace) we imple- ment skill discovery methods based on the official implementation" of MUSIC (Zhao et al., 2021). We train each method for 8K episodes (= 4K epochs) with SAC and set the model dimensionality to (1024, 1024), the entropy coefficient to 0.02, the discount factor to 0.95 and the learning rate to 0.001. At every epoch, we sample two trajectories and train the models with 40 gradient steps com- puted from 256-sized mini-batches. For methods equipped with the MUSIC intrinsic reward, we set the MUSIC reward coefficient to 5000 with reward clipping, following Zhao al. (2021). For the skill reward coefficient, we perform hyperparameter search over (5. 15, 50, 150, 500, 1500, 5000), where we choose 500 (LSD), 150 (DADS), or 1500 (DIAYN), not clipping the skill reward.
I.2.2 TRAINING OF DOWNSTREAM POLICIES
In FetchPushGoal, FetchStideGoal and FetchPiikAAnPPlaeCoall a random goal is sampled at the beginning of each episode and the episode ends with a reward of if the agent reaches the goal, where we use the same sampling range and reach radius as the original Fetch environments. For training meta-controllers, we use the same hyperparameters as in the skill discovery phase, except that we sample 16 trajectories for each epoch.
J RENDERED SCENES OF LEARNED SKILLS
Figures 17 21 visualize the rendered scenes of skills discovered the MuJoCo locomotion envi- ronments. Each figure demonstrates a single set of skills. We refer the reader to our project page for the videos.
hhtpp:://gt thub. com zhaogi t //music
22

---

Published as a conference paper at ICLR 2022
Skill #1
Skill #9
Skill #2
Skill #110
Skill #3
Skill #11
Skill #41
Skill #12
Skill #5
Skill #13
Still #AG
Skill #14
Skill #7
Skill #15
Skill #8
Skill #16
Figure 18: 16 discrete skills discovered by LSD for Ant. Discrete LSD learns a skill set consisting of locomotion skills (#1, #6, #7, #12, #16), rotation skills (#2, #3, #4, #8, #10, #15), posing skills (#5, #9, #14) and flipping skills (#1 1, #13). Videos are available on our project page.
(-1.06, 1.06)
(0.0, 1.5)
(1.06, 1.06)
(-1.5.0.0)
(0.0,0.0)
0.0)
(-1.06, -1006
(0.0,- -1.5)
(1.06. -1.06)
Figure 19: 2-D continuous skills discovered by LSD for Humanoid. The Humanoid robot can walk various directions specified by 2. Videos are available on our project page.
23

---

Published as a conference paper at ICLR 2022
Skill #1
Skill #9
Skill #2
Skill #10
Skill #3
Skill #11
#4
Skill #12
Skill #5
Skill #13
Skill #6
Skill #14
Skill #7
Skill #15
Skill #8
Skill #16
Figure 20: 16 discrete skills discovered by LSD for Humanoid. Videos are available on our project page.
Skill #1
Skill #2
Skill #3
Skill #4
Skill #5
Skill #6
Skill #7
Skill #8
Figure 21: discrete skills discovered by LSD for HalfCheetah. Videos are available on our project page.
24


Reviews:
[{"review_id": "hUD-_8XZIpp", "reviewer": "Reviewer_QUmk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n\nThe paper is well written and motivated. The discussion in section 3.2 is intuitive and makes the decisions behind the LSD objective quite clear. \n\n\nThe experimental results are quite impressive, in Ant and Humanoid particularly, and it seems that LSD is quite effective at discovering more dynamic skills compared to prior work, without specific domain knowledge. \n\n\nThe downstream task performance of LSD (using a hierarchical controller which selects actions in the space of skills) is a significant improvement compared to a number of sensible prior works.\n\n\nThe method is relatively simple to implement (as an additional reward to a model-free RL method), and an implementation is provided for reproducibility.\n\nWeaknesses:\n\nThe related work by Choi et al also uses spectral normalization to improve learned skill quality, so it needs to be better clarified how this work stands with respect to that.\n\nAs the authors mention in the conclusion, LSD assumes that the Lipschitz constraint is meaningful ($ell_2$ norms in the state space are meaningful) which is not always true, for example when learning from pixels, or perhaps in some more complex manipulation settings. But I agree that representation learning would likely help to address this issue when learning from pixels.\n\nThe scale of figure 2 seems like it could be slightly misleading: while it makes sense that LSD would generate skills which cover the state space well (especially in the state magnitude), it’s unclear if the other discovered skills are also able to perform the same action (maybe the ant can still walk, but just not as far). For example, DIAYN-XYO on the Ant environment may learn some locomotion skills but its trajectories are almost invisible on the figure, which makes it seem like it doesn’t learn any skills.\n", "summary_of_the_paper": "The paper introduces Lipschitz-constraint skill discovery (LSD), a method for unsupervised skill discovery. The method addresses one limitation of the line of unsupervised skill discovery works which use mutual information between states produced by a skill and latent skill representations as the training objective, which is that they often discover overly simple and static skills, and require some additional domain information to be provided to learn more dynamic skills. LSD addresses this problem by reformulating a Lipschitz-constrained objective to learn continuous or discrete skills, and then demonstrates the effectiveness of the learned skills for downstream tasks. \n", "summary_of_the_review": "I think the paper can be accepted with some modifications. The method is well-motivated, tackling an important issue with mutual-information based skill discovery methods. It demonstrates impressive qualitative and quantitative results compared to prior works. The paper reads very well and is easy to follow. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EdjUH8P62kI", "reviewer": "Reviewer_8wK8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper proposes a sensible method with empirically very significant results. Further, interesting theoretical connections to existing work are provided and the paper is generally well written with high-quality evaluation. \n\nHowever, tzhe paper also suffers from several issues\n1. The proposed method is ad-hoc. See Sec 3.2, “Specifically, we suggest replacing the implicit regularizer of VIC (the second term in Equation (5)) with a 1-Lipschitz constraint on φ”, which is done without justification. The connection to mutual information skills is intriguing, but the proposed method is not shown to maximize mutual information. \n2. The “discrete” version of the method is even more ad-hoc. Indeed, it is impossible since a discrete variable cannot be Lipschitz-continuous. Instead, the continuous method is used while restricting the prior skill distribution to a finite number of skills. It is unclear what are the theoretical properties of this method since it is different from the standard methods that use discrete skill variables.\n3. The introduction is very confusing and does not state the actual contribution of the paper. Instead, the claimed contributions 1 and 2 in the intro are not novel. Both constraining the skill encoder to be Lipschitz continuous and using skill discovery for zero-shot evaluation was proposed by Choi’21 (or arguably by even earlier work). Why does the contribution paragraph not focus on differences with prior work?\n4. The paper claims that none of the prior work can perform zero-shot evaluation, while also citing Choi'21 that explains the opposite. Please provide zero-shot evaluation for the prior work that the paper compares to.\n\n\n---- Update -----\n\nThe updated paper resolves most of my initial concerns, therefore I increase my score to accept. However, I am not confident that this method will scale to visual environments as the method crucially relies on Eucledian distance in the observation space to be meaningful (via the Lipschitz constraint), as the authors themselves note in the limitations section. Further, it appears from Fig 9 that the method ignores some of the state dimensions entirely and therefore might not be able to learn diverse skills that involve all state dimensions such as reaching a desired joint pose. Therefore I am not able to give a strong accept.\n\n", "summary_of_the_paper": "\nThe paper proposes a new skill discovery objective based on lipschitz-constrained dynamics skills. Specifically, for a state encoding \\phi, <\\phi_{t+1} - \\phi_t, z> is maximized with a DIAYN-style training, and spectral normalization on \\phi. The connection to DIAYN with Gaussian skills is shown, which is an unconstrained version of the proposed objective. A version of the method that defines a prior distribution of skills with finite support is also presented, which can be used similarly to discrete skill learning methods. Empirically, the method achieves significant improvements over prior work, being able to tackle ant and humanoid domains, where all prior methods fail. \n", "summary_of_the_review": "The paper proposes a novel method with significant performance improvement. However, the paper suffers from several issues with accuracy of the claims. Further, a version of the method dubbed \"discrete Lipschitz-constrained skill discovery\" is proposed, which is clearly a contradiction (discrete variables cannot be Lipschitz constrained). The paper needs to provide a clear explanation of what the discrete version of the method actually is. Without fixing these issues, specifically points 2-4 above, I believe the paper cannot be accepted.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "5UOPo76kz2U", "reviewer": "Reviewer_an2L", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Clear write-up with good motivation that correctly points out a common failure mode of previous MI-based objectives.\n- Good performance on basic and challenging robots, including Humanoid. Overall, the empirical evaluation is quite exhaustive, and multiple relevant baselines are included\n- Alignment of representation and skill enables zero-shot goal reaching, which is nicely demonstrated.\n\nWeaknesses:\n- A lot of focus is put on skills resulting in directed locomotion. While this is a useful result and allows to easily measure coverage, for example, it's also a limited evaluation -- for directed locomotion, it's pretty straightforward to simply learn goal-directed policies and use them as skills. It would be nice to consider other tasks as well, e.g. letting the Cheetah jump over hurdles (as in DIAYN, for example). It's good that the authors also evaluate on robot manipulation tasks (Fetch* environments), but if I understood correctly, in these setting they consider the location of the target object as the only state features for reward and state representation?\n- The authors acknowledge that with continuous skills, their method mostly discovers locomotion skills. I'm left wondering to what degree this is caused by the skill dimension being set to 2. What happens if 3 or more dimensions are used?\n- The notion of \"dynamic skill\", which is used quite a bit in the abstract and motivation, is not really clear. It's finally explained in section 3.1., but I don't find it very intuitive.", "summary_of_the_paper": "This paper proposes a new objective (LSD) for unsupervised skill discovery, building on prior skill discovery work using mutual information criteria. As in previous works, skills are parameterized by a latent variable. The main ingredients are changes to the MI-based reward that (1) maximize alignment of learned state representation and latent variable, and (2) ensuring that state space differences are larger in the actual state space than in the learned representation. These modifications act as a prior so that the skill variable contains semantics (as it encodes a direction in representation space) and skills are encouraged to significantly change the agent's state.", "summary_of_the_review": "Overall I think this is a good paper, but also that the focus on the navigation tasks is rather narrow and that it would be nice to a few non-navigation downstream tasks with the standard MuJoCo agents.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2v2euxg7hMV", "reviewer": "Reviewer_wpgt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Figure 5 seems to show that LSD gets fundamentally better skills as competing methods look like they are asympoting to a lower level in at least AntMultiGoal, HumanoidGoal and HumanoidMultiGoal. The evaluation on classic AI Gym as well as robot pick and place tasks shows good scalability to a variety of interesting tasks. \n\nThe objective looks simpler to evaluate than a number of the competing approaches promising a more efficient algorithm.\n\nThe idea of aligning the direction of latent and observed states is nice for downstream planning but will this ultimately limit the flexibility of the latent states to express important properties of the space that may not be one-to-one? \n\nIn equation 6, is the <= sign backwards? Shouldn’t the projected states be larger || phi(x)-phi(y) || >= ||x-y|| according to the argument of the paper? \n\n“LSD” and “learning” including “machine learning” has a big literature already that might make it hard to find the “LSD” paper. Maybe something like Lipschitz-constrained Unsupervised Skill Acquisition (LUSA)? \n\nWhile LSD technically doesn’t make use of hand engineered features, the fact that the algorithm enforces a directionality consistent with the observed states pace kind of builds in a similar concept to maximizing travel in space … it is a bit subtler but not completely different. The case here seems overstated. ", "summary_of_the_paper": "The paper builds upon the DIAYN idea (Eysenback 2018) that an agent could develop skills in an unsupervised environment by finding a set of skills that collectively visits the whole state space but encourages each skill to cover a different subspace and later use one of these skills to simplify the learning of a downstream task. The paper points out that prior work maximizes the distribution of latent states, but does not necessarily maximize world state differences, because latent states are the result of a learned projection from world states. The paper introduces an explicit Lipschitz constraint that forces latent state differences to be larger than observed state differences and relaxes a prior constraint of equality so that the latent states only need to move in the same direction as observed states but not in a fixed ratio. The paper shows this can be implemented as an efficient per step intrinsic reward for unsupervised learning.  Since the latent state aligns with observed state, one can use simple planner on downstream tasks to choose the right z to achieve a goal in observed space without any additional learning or fine tuning (zero shot skill exploitation). The paper evaluates LSD on 2D skills in the widely used AI GYM benchmark. Qualitatively, the LSD model discovers challenging dynamic behaviors in the difficult humanoid environment that competing unsupervised models fail to discover. Quantitatively, expected reward is higher for LSD and LSD zero shot than competing models. Direct evaluation of state space coverage shows LSD has significantly larger coverage than competing models supporting the paper hypothesis. LSD is also demonstrated on robotic tasks and is able to learn to pick objects in using unsupervised learning. ", "summary_of_the_review": "The paper proposed a new practically implementable objective for unsupervised learning that gives qualitatively and quantitatively better performance on classic AI Gym benchmarks and challenging robotic pick and place.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces unsupervised skill discovery using Lipschitz-constrained skills. It is well-written and demonstrates the advantages in a solid experimental section.
Decision: Accept (Poster)