Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learning

1 INTRODUCTION . Reinforcement learning ( RL ) makes it possible to drive agents to achieve sophisticated goals in complex and uncertain environments , from computer games ( Badia et al. , 2020 ; Berner et al. , 2019 ) to real robot control ( Lee et al. , 2018 ; Lowrey et al. , 2018 ; Vecerik et al. , 2019 ; Popov et al. , 2017 ) , which usually involves learning a specific policy for individual task relying on task-specific reward . However , autonomous agents are expected to exist persistently in the world and have the ability to solve diverse tasks . To achieve this , one needs to build a universal reward function and design a mechanism to automatically generate diverse goals for training . Raw sensory inputs such as images have been considered as common goals for agents to practice on and achieve ( Watter et al. , 2015 ; Florensa et al. , 2019 ; Nair et al. , 2018 ; 2019 ) , which further exacerbates the challenge for designing autonomous RL agents that can deal with such perceptually-specific inputs . Previous works make full use of a goal-achievement reward function as available prior knowledge ( Pong et al. , 2018 ) , such as Euclidean distance . Unfortunately , this kind of measurement in original space is not very effective for visual tasks since the distance between images does not correspond to meaningful distance between states ( Zhang et al. , 2018 ) . Further , the measure function is applied in the embedding space , where the representations of raw sensory inputs are learned by means of using a latent variable model like VAE ( Higgins et al. , 2017b ; Nair et al. , 2018 ) or using the contrastive loss ( Sermanet et al. , 2018 ; Warde-Farley et al. , 2018 ) . We argue that these approaches taking prior non-parametric reward function in original or embedding space as above may limit the repertoires of behaviors and impose manual engineering burdens ( Pong et al. , 2019 ) . In the absence of any prior knowledge about the measure function , standard unsupervised RL methods learn a latent-conditioned policy through the lens of empowerment Salge et al . ( 2014 ) ; Eysenbach et al . ( 2018 ) ; Sharma et al . ( 2019 ) or the self-consistent trajectory autoencoder ( Co-Reyes et al. , 2018 ; Hausman et al. , 2018 ) . However , the learned policy is conditioned on the latent variables rather than perceptually-specific goals . Applying these procedures to goal-reaching tasks , similar to parameter initialization or the hierarchical RL , needs an external reward function for the new tasks ; otherwise the learned latent-conditioned policy can not be applied directly to user-specified goals . Different from previous works , a novel unsupervised RL scheme is proposed in this paper to learn goal-conditioned policy by jointly learning an extra abstract-level policy conditioned on latent variables . The abstract-level policy is trained to generate diverse abstract skills while the goalconditioned policy is trained to efficiently achieve perceptually-specific goals that are rendered from the states induced by the corresponding abstract skills . Specifically , we optimize a discriminator in an unsupervised manner for the purpose of reliable exploration ( Salge et al. , 2014 ) to provide the intrinsic reward for the abstract-level policy . Then the learned discriminator serves as an intrinsic reward function for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy . In essence , the abstract-level policy can reproducibly influence the environment , and the goal-conditioned policy perceptibly imitates these influences . To improve the generalization ability of goal-conditioned policy in dealing with perceptually-specific inputs , a latent variable model is further considered in the goal-conditioned policy to disentangle goals into latent generative factors . The main contribution of our work is an unsupervised RL method that can learn perceptually-specific goal-conditioned policy without the prior reward function for autonomous agents . We propose a novel training procedure for this model , which provides an universal and effective reward function for various perceptual goals , e.g . images and text descriptions . Furthermore , we introduce a latent variable model for learning the representations of high-dimensional goals , and demonstrate the potential of our model to generalize behaviors across new tasks . Extensive experiments and detailed analysis demonstrate the effectiveness and efficiency of our proposed method . 2 PRELIMINARIES . RL : An agent interacts with an environment and selects actions in RL so as to maximize the expected amount of reward received in the long run ( Sutton & Barto , 2018 ) , which can be modeled as a Markov decision process ( MDP ) ( Puterman , 2014 ) . An MDP is defined as a tuple M = ( S , A , p , R , γ ) , where S and A are state and action spaces , p ( ·|s , a ) gives the next-state distribution upon taking action a in state s , R ( s , a , s′ ) is a random variable representing the reward received at transition s a→ s′ , and γ ∈ [ 0 , 1 ) is a discount factor . Intrinsic Motivation : RL with intrinsic motivation obtains the intrinsic reward by maximizing the mutual information between latent variables ω and agent ’ s behaviors τ : I ( ω ; τ ) , where the specific manifestation of τ can be an entire trajectory ( Achiam et al. , 2018 ) , an individual state ( Eysenbach et al. , 2018 ) or a final state ( Gregor et al. , 2016 ) ; and the specific implementation includes reverse and forward forms ( Campos et al. , 2020 ) . Please refer to Aubret et al . ( 2019 ) for more details . Disentanglement : Given an observation x with a dimension of N , VAE ( Kingma & Welling , 2013 ; Higgins et al. , 2017a ) is a latent model that pairs a top-down encoder q ( z|x ) with bottom-up decoder network p ( x|z ) by introducing a latent factor z , where dim ( z ) < N . To encourage the inferred latent factor z to capture the generative factor of x in a disentangled manner , an isotropic unit Gaussian prior p ( z ) ∼ N ( 0 ; I ) is commonly used to control the capacity of the information bottleneck ( Burgess et al. , 2018 ) by minimizing the KL divergence between q ( z|x ) and p ( z ) . 3 THE METHOD . In this section , we firstly formalize the problem and introduce the framework . Secondly , we elaborate on the process of how to jointly learn the goal-conditioned policy and abstract-level policy . Thirdly , disentanglement is applied in our setting to improve the generalization ability . 3.1 OVERVIEW . Given perceptually-specific goal g , our objective is to learn a goal-conditioned policy πθ ( a|s̃ , g ) that inputs state s̃ and g and outputs action a as shown in Fig . 1 . The abstract-level policy πµ ( a|s , ω ) takes as input state s and a latent variable ω and outputs action a , where ω corresponds to diverse latent skills . The discriminator qφ is firstly trained at the abstract-level for reliable exploration , then it provides the reward signal for the goal-conditioned policy to imitate the trajectory induced by the abstract-level policy . The abstract-level policy is able to generate diverse states s that are further rendered as diverse perceptually-specific goal g = Render ( s ) . On this basis , πθ ( a|s̃ , g ) conditioned on the rendered goal g interacts with the reset environment under the instruction of the reward function qφ . We use the non-tilde s and the tilde s̃ to distinguish between the states of two policies respectively . Actually , s̃ and s come from the same distribution . 3.2 PROPOSED GPIM METHOD In order to jointly learn the abstract-level policy πµ ( a|s , ω ) and goal-conditioned policy πθ ( a|s̃ , g ) , we maximize the mutual information between the state s and latent variable w for πµ , and simultaneously maximize the mutual information between the state s̃ and goal g for πθ . Consequently , the overall objective function to be maximized can be expressed as follows 1 F ( µ , θ ) = I ( s ; ω ) + I ( s̃ ; g ) . ( 1 ) For clarification , Fig . 2 depicts the graphical model for the latent variable ω , state s induced by πµ , goal g rendered from s , and state s̃ induced by πθ . As seen , the latent variable ω ∼ p ( ω ) 2 is firstly used to generate state s via the policy πµ interacting with dynamic environment . Then , we use the state s to generate perceptually-specific goal g. After that , the goal-conditioned policy πθ outputs action a to interact with the environment to obtain the state s̃ at the next time step . In particular , ω is expected to generate diverse behavior modes through πµ while πθ behaving like s̃ is to imitate this behavior by taking as input the rendered goal g. Based on the context , the correlation between s̃ and g is no less than that between s̃ and w , so that I ( s̃ ; g ) ≥ I ( s̃ ; ω ) ( Beaudry & Renner , 2011 ) . Thus , we can obtain the lower bound of F ( µ , θ ) : F ( µ , θ ) ≥ I ( s ; ω ) + I ( s̃ ; ω ) = 2H ( ω ) −H ( ω|s ) −H ( ω|s̃ ) = Eω , s , s̃ [ log p ( ω|s ) + log p ( ω|s̃ ) − 2 log p ( ω ) ] . ( 2 ) Since it is difficult to exactly compute the posterior distributions p ( ω|s ) and p ( ω|s̃ ) , Jensen ’ s Inequality ( Barber & Agakov , 2003 ) is further applied for approximation by using a learned discrimi- 1To further clarify the motivation , we conduct the ablation study to compare our method with that just maximizing I ( s ; ω ) and that just maximizing I ( s ; g ) in Appendix A.1 . 2p ( ω ) denotes the prior distribution . Fixing p ( ω ) instead of learning it is to prevent πµ from collapsing to sampling only a handful of skills . In experiment , we use a categorical distribution for p ( ω ) following DIAYN . Algorithm 1 Learning process of our proposed GPIM 1 : while not converged do 2 : Sample skill ω ∼ p ( ω ) . 3 : # Step I 4 : Sample initial state s0 ∼ p0 ( s ) . 5 : for t = 0 , 1 , ... , T steps do 6 : Sample action at ∼ πµ ( at|st , ω ) . 7 : Step environment : st+1 ∼ p ( st+1|st , at ) . 8 : Render goal : gt = Render ( st+1 ) . 9 : Compute reward rt for policy πµ using ( 5 ) . 10 : Update policy πµ to maximize rt with SAC . 11 : Update discriminator ( qφ ) to maximize log qφ ( ω|st+1 ) with SGD . 12 : end for 13 : # Step II 14 : Sample initial state s̃0 ∼ p0 ( s̃ ) . 15 : for t = 0 , 1 , ... , T − 1 steps do 16 : Sample action at ∼ πθ ( at|s̃t , gt ) . 17 : Step environment : s̃t+1 ∼ p ( s̃t+1|s̃t , at ) . 18 : Compute reward r̃t for policy πθ using ( 7 ) . 19 : Update policy πθ to maximize r̃t with SAC ( θ = { ϑE , ϑG } when considering the disentanglement ) . 20 : Update ϑE and ϑD by maximizing Dis loss in ( 8 ) using SGD . 21 : end for 22 : end while nator network qφ ( ω|· ) ( see Appendix B.1 for the derivation ) . Thus , F ( µ , θ ) ≥ Eω , s , s̃ [ log qφ ( ω|s ) + log qφ ( ω|s̃ ) − 2 log p ( ω ) ] , J ( µ , φ , θ ) , ( 3 ) where it is worth noting that the identical discriminator qφ with the parameter φ is used for the variational approximation of p ( ω|s ) and p ( ω|s̃ ) . For the state s induced by skill w and s̃ originating from g , the shared discriminator qφ assigns a similarly high probability on w for both states s and s̃ associated with the same ω . Intuitively , we factorize the abstract-level policy and learn it purely in the space of the agent ’ s embodiment ( i.e. , the latent variable ω ) — separate from the perceptually-specified goals ( e.g. , images and text descriptions ) . The space of goals in these two spaces has different characteristics due to the underlying manifold spaces . Therefore , qφ can be regarded as a reward network shared by the abstract-level policy πµ ( a|s , ω ) and goal-conditioned policy πθ ( a|s̃ , g ) . According to the objective function J ( µ , φ , θ ) in ( 3 ) , we propose an alternating optimization between πµ and πθ as follows : Step I : Fix πθ and update πµ and qφ . In this case , θ is not a variable to update and J ( µ , φ , θ ) becomes J ( µ , φ ) = Eω , s [ log qφ ( ω|s ) ] + Eω , s̃ [ log qφ ( ω|s̃ ) − 2 log p ( ω ) ] ︸ ︷︷ ︸ Variable independent term . ( 4 ) According to ( 4 ) , J ( µ , φ ) can be thus optimized by setting the reward at time step t for πµ as rt = log qφ ( ω|st+1 ) − log p ( ω ) , ( 5 ) where the term − log p ( ω ) is added for agents to avoid artificial termination and reward-hacking issues ( Amodei et al. , 2016 ) . We implement this optimization with soft actor-critic ( SAC ) . On the other hand , the reward network qφ can be updated with SGD by maximizing Eω , s [ log qφ ( ω|s ) ] . Step II : Fix πµ and qφ to update πθ . In this case , µ and φ are not variables to update any more and J ( µ , φ , θ ) can be simplified as J ( θ ) = Eω , s̃ [ log qφ ( ω|s̃ ) ] + Eω , s [ log qφ ( ω|s ) − 2 log p ( ω ) ] ︸ ︷︷ ︸ Variable independent term . ( 6 ) According to ( 6 ) , J ( θ ) can thus be optimized by setting the reward at time step t for πθ as r̃t = log qφ ( ω|s̃t+1 ) − log p ( ω ) , ( 7 ) where the term − log p ( ω ) is added for the same reason as above and we also implement this optimization with SAC . These two steps are performed alternately until convergence . The overall GPIM is summarized in Algorithm 1 , which includes the disentanglement ( red ) in the next subsection .

Summary: This paper proposes a method for combining intrinsic motivation on a state space with goal-conditioned reinforcement learning (GCRL), where goals are defined in some “perceptual space,” such as text or images, which describe the current state. The authors assume access to a renderer that maps states to perceptual goals, but do not assume that the renderer is differentiable. The authors propose to train an intrinsically motivated latent-conditioned policy, using similar techniques as past work in which a policy maximizes the mutual information between a latent variable and the current state. The goal-conditioned policy is then trained to effectively imitate the latent-conditioned policy by maximizing the same reward as the latent-conditioned policy, conditioned on only the rendered version of the final state reached by the latent-conditioned policy. The authors demonstrate that the overall method outperforms past GCRL methods on a variety of tasks (Atari, MuJoCo manipulation and locomotion, and toy tasks).