Dynamics-Aware Unsupervised Discovery of Skills

1 INTRODUCTION . Deep reinforcement learning ( RL ) enables autonomous learning of diverse and complex tasks with rich sensory inputs , temporally extended goals , and challenging dynamics , such as discrete gameplaying domains ( Mnih et al. , 2013 ; Silver et al. , 2016 ) , and continuous control domains including locomotion ( Schulman et al. , 2015 ; Heess et al. , 2017 ) and manipulation ( Rajeswaran et al. , 2017 ; Kalashnikov et al. , 2018 ; Gu et al. , 2017 ) . Most of the deep RL approaches learn a Q-function or a policy that are directly optimized for the training task , which limits their generalization to new scenarios . In contrast , MBRL methods ( Li & Todorov , 2004 ; Deisenroth & Rasmussen , 2011 ; Watter et al. , 2015 ) can acquire dynamics models that may be utilized to perform unseen tasks at test time . While this capability has been demonstrated in some of the recent works ( Levine et al. , 2016 ; Nagabandi et al. , 2018 ; Chua et al. , 2018b ; Kurutach et al. , 2018 ; Ha & Schmidhuber , ∗Work done a part of the Google AI Residency program . 2018 ) , learning an accurate global model that works for all state-action pairs can be exceedingly challenging , especially for high-dimensional system with complex and discontinuous dynamics . The problem is further exacerbated as the learned global model has limited generalization outside of the state distribution it was trained on and exploring the whole state space is generally infeasible . Can we retain the flexibility of model-based RL , while using model-free RL to acquire proficient low-level behaviors under complex dynamics ? While learning a global dynamics model that captures all the different behaviors for the entire statespace can be extremely challenging , learning a model for a specific behavior that acts only in a small part of the state-space can be much easier . For example , consider learning a model for dynamics of all gaits of a quadruped versus a model which only works for a specific gait . If we can learn many such behaviors and their corresponding dynamics , we can leverage model-predictive control to plan in the behavior space , as opposed to planning in the action space . The question then becomes : how do we acquire such behaviors , considering that behaviors could be random and unpredictable ? To this end , we propose Dynamics-Aware Discovery of Skills ( DADS ) , an unsupervised RL framework for learning low-level skills using model-free RL with the explicit aim of making model-based control easy . Skills obtained using DADS are directly optimized for predictability , providing a better representation on top of which predictive models can be learned . Crucially , the skills do not require any supervision to learn , and are acquired entirely through autonomous exploration . This means that the repertoire of skills and their predictive model are learned before the agent has been tasked with any goal or reward function . When a task is provided at test-time , the agent utilizes the previously learned skills and model to immediately perform the task without any further training . The key contribution of our work is an unsupervised reinforcement learning algorithm , DADS , grounded in mutual-information-based exploration . We demonstrate that our objective can embed learned primitives in continuous spaces , which allows us to learn a large , diverse set of skills . Crucially , our algorithm also learns to model the dynamics of the skills , which enables the use of model-based planning algorithms for downstream tasks . We adapt the conventional model predictive control algorithms to plan in the space of primitives , and demonstrate that we can compose the learned primitives to solve downstream tasks without any additional training . 2 PRELIMINARIES . Mutual information can been used as an objective to encourage exploration in reinforcement learning ( Houthooft et al. , 2016 ; Mohamed & Rezende , 2015 ) . According to its definition , I ( X ; Y ) = H ( X ) − H ( X | Y ) , maximizing mutual information I with respect to Y amounts to maximizing the entropyH of X while minimizing the conditional entropyH ( X | Y ) . In the context of RL , X is usually a function of the state and Y a function of actions . Maximizing this objective encourages the state entropy to be high , making the underlying policy to be exploratory . Recently , multiple works ( Eysenbach et al. , 2018 ; Gregor et al. , 2016 ; Achiam et al. , 2018 ) apply this idea to learn diverse skills which maximally cover the state space . To leverage planning-based control , MBRL estimates the true dynamics of the environment by learning a model p̂ ( s′ | s , a ) . This allows it to predict a trajectory of states τ̂H = ( st , ŝt+1 , . . . ŝt+H ) resulting from a sequence of actions without any additional interaction with the environment . While model-based RL methods have been demonstrated to be sample efficient compared to their modelfree counterparts , learning an effective model for the whole state-space is challenging . An openproblem in model-based RL is to incorporate temporal abstraction in model-based control , to enable high-level planning and move-away from planning at the granular level of actions . These seemingly unrelated ideas can be combined into a single optimization scheme , where we first discover skills ( and their models ) without any extrinsic reward and then compose these skills to optimize for the task defined at test time using model-based planning . At train time , we assume a Markov Decision Process ( MDP ) M1 ≡ ( S , A , p ) . The state space S and action space A are assumed to be continuous , and theA bounded . We assume the transition dynamics p to be stochastic , such that p : S × A× S 7→ [ 0 , ∞ ) . We learn a skill-conditioned policy π ( a | s , z ) , where the skills z belongs to the space Z , detailed in Section 3 . We assume that the skills are sampled from a prior p ( z ) over Z . We simultaneously learn a skill-conditioned transition function q ( s′ | s , z ) , coined as skill-dynamics , which predicts the transition to the next state s′ from the current state s for the skill z under the given dynamics p. At test time , we assume an MDPM2 ≡ ( S , A , p , r ) , where S , A , p match those defined inM1 , and the reward function r : S × A 7→ ( −∞ , ∞ ) . We plan in Z using q ( s′ | s , z ) to compose the learned skills z for optimizing r inM2 , which we detail in Section 4 . 3 DYNAMICS-AWARE DISCOVERY OF SKILLS ( DADS ) . We use the information theoretic paradigm of mutual information to obtain our unsupervised skill discovery algorithm . In particular , we propose to maximize the mutual information between the next state s′ and current skill z conditioned on the current state s. I ( s′ ; z | s ) = H ( z | s ) −H ( z | s′ , s ) ( 1 ) = H ( s′ | s ) −H ( s′ | s , z ) ( 2 ) Mutual information in Equation 1 quantifies how much can be known about s′ given z and s , or symmetrically , z given the transition from s → s′ . From Equation 2 , maximizing this objective corresponds to maximizing the diversity of transitions produced in the environment , that is denoted by the entropy H ( s′ | s ) , while making z informative about the next state s′ by minimizing the entropy H ( s′ | s , z ) . Intuitively , skills z can be interpreted as abstracted action sequences which are identifiable by the transitions generated in the environment ( and not just by the current state ) . Thus , optimizing this mutual information can be understood as encoding a diverse set of skills in the latent space Z , while making the transitions for a given z ∈ Z predictable . We use the entropydecomposition in Equation 2 to connect this objective with model-based control . We want to optimize the our skill-conditioned controller π ( a | s , z ) such that the latent space z ∼ p ( z ) is maximally informative about the transitions s→ s′ . Using the definition of conditional mutual information , we can rewrite Equation 2 as : I ( s′ ; z | s ) = ∫ p ( z , s , s′ ) log p ( s′ | s , z ) p ( s′ | s ) ds′dsdz ( 3 ) We assume the following generative model : p ( z , s , s′ ) = p ( z ) p ( s | z ) p ( s′ | s , z ) , where p ( z ) is user specified prior over Z , p ( s|z ) denotes the stationary state-distribution induced by π ( a | s , z ) for a skill z and p ( s′ | s , z ) denotes the transition distribution under skill z . Note , p ( s′ | s , z ) =∫ p ( s′ | s , a ) π ( a | s , z ) da is intractable to compute because the underlying dynamics are unknown . However , we can variationally lower bound the objective as follows : I ( s′ ; z | s ) = Ez , s , s′∼p [ log p ( s′ | s , z ) p ( s′ | s ) ] = Ez , s , s′∼p [ log qφ ( s ′ | s , z ) p ( s′ | s ) ] + Es , z∼p [ DKL ( p ( s′ | s , z ) || qφ ( s′ | s , z ) ) ] ≥ Ez , s , s′∼p [ log qφ ( s ′ | s , z ) p ( s′ | s ) ] ( 4 ) where we have used the non-negativity of KL-divergence , that is DKL ≥ 0 . Note , skill-dynamics qφ represents the variational approximation for the transition function p ( s′ | s , z ) , which enables model-based control as described in Section 4 . Equation 4 suggests an alternating optimization between qφ and π , summarized in Algorithm 1 . In every iteration : ( Tighten variational lower bound ) We minimize DKL ( p ( s′ | s , z ) || qφ ( s′ | s , z ) ) with respect to the parameters φ on z , s ∼ p to tighten the lower bound . For general function approximators like neural networks , we can write the gradient for φ as follows : ∇φEs , z [ DKL ( p ( s′ | s , z ) || qφ ( s′ | s , z ) ) ] = ∇φEz , s , s′ [ log p ( s′ | s , z ) qφ ( s′ | s , z ) ] = −Ez , s , s′ [ ∇φ log qφ ( s′ | s , z ) ] ( 5 ) which corresponds to maximizing the likelihood of the samples from p under qφ . ( Maximize approximate lower bound ) After fitting qφ , we can optimize π to maximize Ez , s , s′ [ log qφ ( s′ | s , z ) − log p ( s′ | s ) ] . Note , this is a reinforcement-learning style optimization with a reward function log qφ ( s′ | s , z ) − log p ( s′ | s ) . However , log p ( s′ | s ) is intractable to compute , so we approximate the reward function for π : rz ( s , a , s ′ ) = log qφ ( s ′ | s , z ) ∑L i=1 qφ ( s ′ | s , zi ) + logL , zi ∼ p ( z ) . ( 6 ) The approximation is motivated as follows : p ( s′ | s ) = ∫ p ( s′ | s , z ) p ( z|s ) dz ≈ ∫ qφ ( s ′ | s , z ) p ( z ) dz ≈ 1L ∑L i=1 qφ ( s ′ | s , zi ) for zi ∼ p ( z ) , where L denotes the number of samples from the prior . We are using the marginal of variational approximation qφ over the prior p ( z ) to approximate the marginal distribution of transitions . We discuss this approximation in Appendix C. Note , the final reward function rz encourages the policy π to produce transitions that are ( a ) predictable under qφ ( predictability ) and ( b ) different from the transitions produced under zi ∼ p ( z ) ( diversity ) . To generate samples from p ( z , s , s′ ) , we use the rollouts from the current policy π for multiple samples z ∼ p ( z ) in an episodic setting for a fixed horizon T . We also introduce entropy regularization for π ( a | s , z ) , which encourages the policy to discover action-sequences with similar state-transitions and to be clustered under the same skill z , making the policy robust besides encouraging exploration ( Haarnoja et al. , 2018a ) . The use of entropy regularization can be justified from an information bottleneck perspective as discussed for Information Maximization algorithm in ( Mohamed & Rezende , 2015 ) . This is even more extensively discussed from the graphical model perspective in Appendix B , which connects unsupervised skill discovery and information bottleneck literature , while also revealing the temporal nature of skills z . Details corresponding to implementation and hyperparameters are discussed in Appendix A .

Summary: This paper proposes a novel approach to learn a continuous set of skills (where a skill is associated with a latent vector and the skill policy network takes that vector as an extra input) by pure unsupervised exploration using as intrinsic reward a proxy for the mutual information  between next states and the skill (given the previous state). These skills can be used in a model-based planning (model-predictive control) with zero 'supervised' training data (for which the rewards are given), but using calls to the reward function to evaluate candidate sequences of skills and actions. The proposed approach is convincingly compared in several ways to both previous model-based approaches and model-free approaches.