Diversity is All You Need: Learning Skills without a Reward Function

1 INTRODUCTION . Deep reinforcement learning ( RL ) has been demonstrated to effectively learn a wide range of rewarddriven skills , including playing games ( Mnih et al. , 2013 ; Silver et al. , 2016 ) , controlling robots ( Gu et al. , 2017 ; Schulman et al. , 2015b ) , and navigating complex environments ( Zhu et al. , 2017 ; Mirowski et al. , 2016 ) . However , intelligent creatures can explore their environments and learn useful skills even without supervision , so that when they are later faced with specific goals , they can use those skills to satisfy the new goals quickly and efficiently . Learning skills without reward has several practical applications . Environments with sparse rewards effectively have no reward until the agent randomly reaches a goal state . Learning useful skills without supervision may help address challenges in exploration in these environments . For long horizon tasks , skills discovered without reward can serve as primitives for hierarchical RL , effectively shortening the episode length . In many practical settings , interacting with the environment is essentially free , but evaluating the reward requires human feedback ( Christiano et al. , 2017 ) . Unsupervised learning of skills may reduce the amount of supervision necessary to learn a task . While we can take the human out of the loop by designing a reward function , it is challenging to design a reward function that elicits the desired behaviors from the agent ( Hadfield-Menell et al. , 2017 ) . Finally , when given an unfamiliar environment , it is challenging to determine what tasks an agent should be able to learn . Unsupervised skill discovery partially answers this question.1 Autonomous acquisition of useful skills without any reward signal is an exceedingly challenging problem . A skill is a latent-conditioned policy that alters the state of the environment in a consistent way . We consider the setting where the reward function is unknown , so we want to learn a set of skills by maximizing the utility of this set . Making progress on this problem requires specifying a ∗Work done as a member of the Google AI Residency Program ( g.co/airesidency ) . 1See videos here : https : //sites.google.com/view/diayn/ learning objective that ensures that each skill individually is distinct and that the skills collectively explore large parts of the state space . In this paper , we show how a simple objective based on mutual information can enable RL agents to autonomously discover such skills . These skills are useful for a number of applications , including hierarchical reinforcement learning and imitation learning . We propose a method for learning diverse skills with deep RL in the absence of any rewards . We hypothesize that in order to acquire skills that are useful , we must train the skills so that they maximize coverage over the set of possible behaviors . While one skill might perform a useless behavior like random dithering , other skills should perform behaviors that are distinguishable from random dithering , and therefore more useful . A key idea in our work is to use discriminability between skills as an objective . Further , skills that are distinguishable are not necessarily maximally diverse – a slight difference in states makes two skills distinguishable , but not necessarily diverse in a semantically meaningful way . To combat this problem , we want to learn skills that not only are distinguishable , but also are as diverse as possible . By learning distinguishable skills that are as random as possible , we can “ push ” the skills away from each other , making each skill robust to perturbations and effectively exploring the environment . By maximizing this objective , we can learn skills that run forward , do backflips , skip backwards , and perform face flops ( see Figure 3 ) . Our paper makes five contributions . First , we propose a method for learning useful skills without any rewards . We formalize our discriminability goal as maximizing an information theoretic objective with a maximum entropy policy . Second , we show that this simple exploration objective results in the unsupervised emergence of diverse skills , such as running and jumping , on several simulated robotic tasks . In a number of RL benchmark environments , our method is able to solve the benchmark task despite never receiving the true task reward . In these environments , some of the learned skills correspond to solving the task , and each skill that solves the task does so in a distinct manner . Third , we propose a simple method for using learned skills for hierarchical RL and find this methods solves challenging tasks . Four , we demonstrate how skills discovered can be quickly adapted to solve a new task . Finally , we show how skills discovered can be used for imitation learning . 2 RELATED WORK . Previous work on hierarchical RL has learned skills to maximize a single , known , reward function by jointly learning a set of skills and a meta-controller ( e.g. , ( Bacon et al. , 2017 ; Heess et al. , 2016 ; Dayan & Hinton , 1993 ; Frans et al. , 2017 ; Krishnan et al. , 2017 ; Florensa et al. , 2017 ) ) . One problem with joint training ( also noted by Shazeer et al . ( 2017 ) ) is that the meta-policy does not select “ bad ” options , so these options do not receive any reward signal to improve . Our work prevents this degeneracy by using a random meta-policy during unsupervised skill-learning , such that neither the skills nor the meta-policy are aiming to solve any single task . A second importance difference is that our approach learns skills with no reward . Eschewing a reward function not only avoids the difficult problem of reward design , but also allows our method to learn task-agnostic . Related work has also examined connections between RL and information theory ( Ziebart et al. , 2008 ; Schulman et al. , 2017 ; Nachum et al. , 2017 ; Haarnoja et al. , 2017 ) and developed maximum entropy algorithms with these ideas Haarnoja et al . ( 2018 ; 2017 ) . Recent work has also applied tools from information theory to skill discovery . Mohamed & Rezende ( 2015 ) and Jung et al . ( 2011 ) use the mutual information between states and actions as a notion of empowerment for an intrinsically motivated agent . Our method maximizes the mutual information between states and skills , which can be interpreted as maximizing the empowerment of a hierarchical agent whose action space is the set of skills . Hausman et al . ( 2018 ) , Florensa et al . ( 2017 ) , and Gregor et al . ( 2016 ) showed that a discriminability objective is equivalent to maximizing the mutual information between the latent skill z and some aspect of the corresponding trajectory . Hausman et al . ( 2018 ) considered the setting with many tasks and reward functions and Florensa et al . ( 2017 ) considered the setting with a single task reward . Three important distinctions allow us to apply our method to tasks significantly more complex than the gridworlds in Gregor et al . ( 2016 ) . First , we use maximum entropy policies to force our skills to be diverse . Our theoretical analysis shows that including entropy maximization in the RL objective results in the mixture of skills being maximum entropy in aggregate . Second , we fix the prior distribution over skills , rather than learning it . Doing so prevents our method from collapsing to sampling only a handful of skills . Third , while the discriminator in Gregor et al . ( 2016 ) only looks at the final state , our discriminator looks at every state , which provides additional reward signal . These three crucial differences help explain how our method learns useful skills in complex environments . Prior work in neuroevolution and evolutionary algorithms has studied how complex behaviors can be learned by directly maximizing diversity ( Lehman & Stanley , 2011a ; b ; Woolley & Stanley , 2011 ; Stanley & Miikkulainen , 2002 ; Such et al. , 2017 ; Pugh et al. , 2016 ; Mouret & Doncieux , 2009 ) . While this prior work uses diversity maximization to obtain better solutions , we aim to acquire complex skills with minimal supervision to improve efficiency ( i.e. , reduce the number of objective function queries ) and as a stepping stone for imitation learning and hierarchical RL . We focus on deriving a general , information-theoretic objective that does not require manual design of distance metrics and can be applied to any RL task without additional engineering . Previous work has studied intrinsic motivation in humans and learned agents . Ryan & Deci ( 2000 ) ; Bellemare et al . ( 2016 ) ; Fu et al . ( 2017 ) ; Schmidhuber ( 2010 ) ; Oudeyer et al . ( 2007 ) ; Pathak et al . ( 2017 ) ; Baranes & Oudeyer ( 2013 ) . While these previous works use an intrinsic motivation objective to learn a single policy , we propose an objective for learning many , diverse policies . Concurrent work Achiam et al . ( 2017 ) draws ties between learning discriminable skills and variational autoencoders . We show that our method scales to more complex tasks , likely because of algorithmic design choices , such as our use of an off-policy RL algorithm and conditioning the discriminator on individual states . 3 DIVERSITY IS ALL YOU NEED . We consider an unsupervised RL paradigm in this work , where the agent is allowed an unsupervised “ exploration ” stage followed by a supervised stage . In our work , the aim of the unsupervised stage is to learn skills that eventually will make it easier to maximize the task reward in the supervised stage . Conveniently , because skills are learned without a priori knowledge of the task , the learned skills can be used for many different tasks . 3.1 HOW IT WORKS . Our method for unsupervised skill discovery , DIAYN ( “ Diversity is All You Need ” ) , builds off of three ideas . First , for skills to be useful , we want the skill to dictate the states that the agent visits . Different skills should visit different states , and hence be distinguishable . Second , we want to use states , not actions , to distinguish skills , because actions that do not affect the environment are not visible to an outside observer . For example , an outside observer can not tell how much force a robotic arm applies when grasping a cup if the cup does not move . Finally , we encourage exploration and incentivize the skills to be as diverse as possible by learning skills that act as randomly as possible . Skills with high entropy that remain discriminable must explore a part of the state space far away from other skills , lest the randomness in its actions lead it to states where it can not be distinguished . We construct our objective using notation from information theory : S and A are random variables for states and actions , respectively ; Z ∼ p ( z ) is a latent variable , on which we condition our policy ; we refer to a the policy conditioned on a fixed Z as a “ skill ” ; I ( · ; · ) andH [ · ] refer to mutual information and Shannon entropy , both computed with base e. In our objective , we maximize the mutual information between skills and states , I ( S ; Z ) , to encode the idea that the skill should control which states the agent visits . Conveniently , this mutual information dictates that we can infer the skill from the states visited . To ensure that states , not actions , are used to distinguish skills , we minimize the mutual information between skills and actions given the state , I ( A ; Z | S ) . Viewing all skills together with p ( z ) as a mixture of policies , we maximize the entropyH [ A | S ] of this mixture policy . In summary , we maximize the following objective with respect to our policy parameters , θ : F ( θ ) , I ( S ; Z ) +H [ A | S ] − I ( A ; Z | S ) ( 1 ) = ( H [ Z ] −H [ Z | S ] ) +H [ A | S ] − ( H [ A | S ] −H [ A | S , Z ] ) = H [ Z ] −H [ Z | S ] +H [ A | S , Z ] ( 2 ) We rearranged our objective in Equation 2 to give intuition on how we optimize it.2 The first term encourages our prior distribution over p ( z ) to have high entropy . We fix p ( z ) to be uniform in our approach , guaranteeing that it has maximum entropy . The second term suggests that it should be easy to infer the skill z from the current state . The third term suggests that each skill should act as randomly as possible , which we achieve by using a maximum entropy policy to represent each skill . As we can not integrate over all states and skills to compute p ( z | s ) exactly , we approximate this posterior with a learned discriminator qφ ( z | s ) . Jensen ’ s Inequality tells us that replacing p ( z | s ) with qφ ( z | s ) gives us a variational lower bound G ( θ , φ ) on our objective F ( θ ) ( see ( Agakov , 2004 ) for a detailed derivation ) : F ( θ ) = H [ A | S , Z ] −H [ Z | S ] +H [ Z ] = H [ A | S , Z ] + Ez∼p ( z ) , s∼π ( z ) [ log p ( z | s ) ] − Ez∼p ( z ) [ log p ( z ) ] ≥ H [ A | S , Z ] + Ez∼p ( z ) , s∼π ( z ) [ log qφ ( z | s ) − log p ( z ) ] , G ( θ , φ )

Summary: This paper proposes a method for learning skills in absence of a reward function. These skills are learned so that the diversity of the trajectories produced by each skill is maximised. This is achieved by having a discriminator attempting to tell these skills apart. The agent is rewarded for visiting states that are easy to distinguish and the discriminator is trained to better infer the skills from states visited by the agent. Furthermore, a maximum entropy policy is used to force the skills to be diverse. The proposed method is general and any RL algorithm with entropy maximisation in the objective can be used, the implementation in the paper uses the Soft Actor Critic method.