Lipschitz-constrained Unsupervised Skill Discovery

1 INTRODUCTION . Reinforcement learning ( RL ) aims at learning optimal actions that maximize accumulated reward signals ( Sutton & Barto , 2005 ) . Recently , RL with deep neural networks has demonstrated remarkable achievements in a variety of tasks , such as complex robotics control ( Gu et al. , 2017 ; Andrychowicz et al. , 2020 ) and games ( Schrittwieser et al. , 2020 ; Badia et al. , 2020 ) . However , one limitation of the RL framework is that a practitioner has to manually define and tune a reward function for desired behaviors , which is often time-consuming and hardly scalable especially when there are multiple tasks to learn ( Hadfield-Menell et al. , 2017 ; Dulac-Arnold et al. , 2019 ) . Therefore , several methods have been proposed to discover skills without external task rewards ( Gregor et al. , 2016 ; Eysenbach et al. , 2019 ; Sharma et al. , 2020 ) , which is often referred to as the unsupervised skill discovery problem . Unsupervised discovery of skills helps not only relieve the burden of manually specifying a reward for each behavior , but also provide useful primitives to initialize with or combine hierarchically for solving downstream tasks ( Eysenbach et al. , 2019 ; Lee et al. , 2020 ) . Moreover , learned skills can effectively demonstrate the agent ’ s capability in the environment , allowing a better understanding of both the agent and the environment . One of the most common approaches to the unsupervised skill discovery problem is to maximize the mutual information ( MI ) between skill latent variables and states ( Gregor et al. , 2016 ; Achiam et al. , 2018 ; Eysenbach et al. , 2019 ; Hansen et al. , 2020 ; Sharma et al. , 2020 ; Choi et al. , 2021 ; Zhang et al. , 2021 ) . Intuitively , these methods encourage a skill latent z to be maximally informative of states or trajectories obtained from a skill policy π ( a|s , z ) . As a result , optimizing the MI objective leads to the discovery of diverse and distinguishable behaviors . However , existing MI-based skill discovery methods share a limitation that they do not necessarily prefer learning ‘ dynamic ’ skills ( i.e. , making large state variations ) or task-relevant behaviors such as diverse locomotion primitives . Since MI is invariant to scaling or any invertible transformation of the input variables , there exist infinitely many optima for the MI objective . As such , they will ∗Equal contribution , listed in alphabetical order . converge to the maximum that is most easily optimizable , which would usually be just learning simple and static skills . For instance , Figure 1b and Figure 17 demonstrate that DIAYN ( Eysenbach et al. , 2019 ) simply learns to take various postures in place rather than learning locomotion skills in the Ant environment ( Schulman et al. , 2016 ) . While these works often employ some feature engineering or prior domain knowledge to discover more dynamic skills ( e.g. , discriminating skills based on x-y coordinates only ( Eysenbach et al. , 2019 ; Sharma et al. , 2020 ) ) , it brings about other drawbacks : ( i ) practitioners need to manually specify the dimensions of interest and ( ii ) the diversity of skills may be limited to a specific type ( e.g. , the x-y prior results in neglecting non-locomotion behaviors ) . In order to address this limitation of MI-based approaches , we propose an unsupervised skill discovery method based on a Lipschitz continuity constraint , named Lipschitz-constrained Skill Discovery ( LSD ) . Specifically , we argue one reason behind the aforementioned problem is that the MI-based skill discovery methods can easily maximize the MI objective with only slight differences in the state space . To resolve this , we propose a novel objective based on a Lipschitz-constrained state representation function so that the objective maximization in the latent space always entails an increase in traveled distances ( or variations ) in the state space ( Equation ( 6 ) ) . LSD has several advantages . First , unlike previous MI-based skill discovery objectives , LSD encourages learned skills to have larger traveled distances , which leads to more diverse , dynamic , and far-reaching skills . Second , LSD produces a useful state representation function that can be used to solve goal-following downstream tasks ( i.e. , reaching multiple goals in order ) in a fully zero-shot manner ( Figure 1c ) ; i.e. , with no further training or complex planning . Finally , LSD is easy to implement in contrast to many existing approaches and introduces no additional hyperparameters . Our contributions can be summarized as follows : • We propose a novel skill discovery objective based on a Lipschitz constraint named LSD , which maximizes traveled distances in the state space unlike existing MI-based approaches , and thus learns more diverse and dynamic behaviors with no need for feature engineering . • Since LSD aligns the directions of skills and differences in latent state representations , it can efficiently solve goal-following tasks with a wider range of goals in a zero-shot fashion compared to previous methods , with no burden of additional training or complex planning . Table 1 highlights other distinctive properties of LSD in comparison to existing approaches . • LSD exhibits the best performance in terms of the state space coverage on five MuJoCo environments and final rewards on seven downstream tasks , including AntMultiGoals ( Kim et al. , 2021 ) , HumanoidMultiGoals and FetchPushGoal , compared to previous skill discovery methods such as DIAYN ( Eysenbach et al. , 2019 ) , DADS ( Sharma et al. , 2020 ) and IBOL ( Kim et al. , 2021 ) . 2 PRELIMINARIES AND RELATED WORK . 2.1 PROBLEM SETTING . We consider a Markov decision process ( MDP ) M = ( S , A , p ) without external rewards , where S is a ( continuous ) state space , A is an action space , and p ( st+1|st , at ) is a stochastic transition dynamics function . We represent a skill with a latent variable z ∈ Z and a latent-conditioned policy π ( a|s , z ) . The skill latent space Z can be either discrete or continuous ; we use N to denote the number of skills in the discrete case , and d to denote the dimensionality of the skill latent space in the continuous case . Given a skill z and a skill policy π ( a|s , z ) , a trajectory τ = ( s0 , a0 , . . . , sT ) is sampled with the following generative process : pπ ( τ |z ) = p ( s0 ) ∏T−1 t=0 π ( at|st , z ) p ( st+1|st , at ) . Properties — “ Prefer ‘ dynamic ’ skills ” : whether the algorithm prefers skills beyond simple and static ones . “ Zero-shot goal-following ” : whether learned skills can be used for following multiple goals ( from an arbitrary state ) without additional training , where † denotes that it still needs planning and * denotes that its skill discriminator may not cope well with unseen goals or initial states . We use uppercase letters to denote random variables , and h ( · ) and I ( · ; · ) to represent the differential entropy and the mutual information , respectively . 2.2 PRIOR WORK ON UNSUPERVISED SKILL DISCOVERY . A number of previous methods maximize I ( Z ; S ) to learn diverse skills . One line of research employs the identity I ( Z ; S ) = h ( Z ) − h ( Z|S ) ≥ Ez∼p ( z ) , s∼pπ ( s|z ) [ log q ( z|s ) ] − Ez∼p ( z ) [ log p ( z ) ] or its variants , where the skill discriminator q ( z|s ) is a variational approximation of the posterior p ( z|s ) ( Barber & Agakov , 2003 ) . VIC ( Gregor et al. , 2016 ) maximizes the MI between the last states and skills given the initial state . DIAYN ( Eysenbach et al. , 2019 ) optimizes the MI between individual states and skills . VALOR ( Achiam et al. , 2018 ) also takes a similar approach , but considers the whole trajectories instead of states . VISR ( Hansen et al. , 2020 ) models the variational posterior as the von-Mises Fisher distribution , which results in an inner-product reward form and hence enables combining with successor features ( Barreto et al. , 2017 ) . HIDIO ( Zhang et al. , 2021 ) examines multiple variants of the MI identity , and jointly learns skills with a hierarchical controller that maximizes the task reward . Choi et al . ( 2021 ) point out the equivalency between the MI-based objective and goal-conditioned RL , and show that Spectral Normalization ( Miyato et al. , 2018 ) improves the quality of learned skills . DADS ( Sharma et al. , 2020 ) maximizes the opposite direction of the mutual information identity I ( Z ; S ) = h ( S ) − h ( S|Z ) with the skill dynamics model q ( st+1|st , z ) , which allows zero-shot planning on downstream tasks . However , these methods share a limitation : they do not always prefer to reach distant states or to learn dynamic skills , as we can maximize I ( Z ; S ) even with the smallest state variations . One possible way to address this issue is to use heuristics such as feature engineering ; for example , the x-y prior ( Sharma et al. , 2020 ) enforces skills to be discriminated only by their x-y coordinates so that the agent can discover locomotion skills . On the other hand , a couple of methods overcome this limitation by integrating with exploration techniques . EDL ( Campos Camúñez et al. , 2020 ) first maximizes the state entropy h ( S ) with SMM exploration ( Lee et al. , 2019 ) , and then encodes the discovered states into skills via VAE ( Kingma & Welling , 2014 ) . APS ( Liu & Abbeel , 2021a ) combines VISR ( Hansen et al. , 2020 ) with APT ( Liu & Abbeel , 2021b ) , an exploration method based on k-nearest neighbors . Yet , we empirically confirm that such a pure exploration signal is insufficient to make large and consistent transitions in states . IBOL ( Kim et al. , 2021 ) takes a hierarchical approach where it first pre-trains a low-level policy to make reaching remote states easier , and subsequently learns a high-level skill policy based on the information bottleneck framework ( Tishby et al. , 2000 ) . While IBOL can discover skills reaching distant states in continuous control environments without locomotion priors ( e.g. , x-y prior ) , it still has some limitations in that ( 1 ) IBOL can not discover discrete skills , ( 2 ) it still capitalizes on input feature engineering in that they exclude the locomotion coordinates from the low-level policy , and ( 3 ) it consists of a two-level hierarchy with several additional hyperparameters , which make the implementation difficult . On the contrary , our proposed LSD can discover diverse skills in both discrete and continuous settings without using any feature engineering , and is easy to implement as it requires no additional hyperparameters or hierarchy . Table 1 overviews the comparison of properties between different skill discovery methods .

Summary: The paper builds upon the DIAYN idea (Eysenback 2018) that an agent could develop skills in an unsupervised environment by finding a set of skills that collectively visits the whole state space but encourages each skill to cover a different subspace and later use one of these skills to simplify the learning of a downstream task. The paper points out that prior work maximizes the distribution of latent states, but does not necessarily maximize world state differences, because latent states are the result of a learned projection from world states. The paper introduces an explicit Lipschitz constraint that forces latent state differences to be larger than observed state differences and relaxes a prior constraint of equality so that the latent states only need to move in the same direction as observed states but not in a fixed ratio. The paper shows this can be implemented as an efficient per step intrinsic reward for unsupervised learning.  Since the latent state aligns with observed state, one can use simple planner on downstream tasks to choose the right z to achieve a goal in observed space without any additional learning or fine tuning (zero shot skill exploitation). The paper evaluates LSD on 2D skills in the widely used AI GYM benchmark. Qualitatively, the LSD model discovers challenging dynamic behaviors in the difficult humanoid environment that competing unsupervised models fail to discover. Quantitatively, expected reward is higher for LSD and LSD zero shot than competing models. Direct evaluation of state space coverage shows LSD has significantly larger coverage than competing models supporting the paper hypothesis. LSD is also demonstrated on robotic tasks and is able to learn to pick objects in using unsupervised learning.