Learning transitional skills with intrinsic motivation

1 INTRODUCTION . Deep reinforcement learning ( DRL ) has shown its great effectiveness in learning various rewarddriven skills in wide domains , such as performing robotic manipulation tasks ( Levine et al . ( 2016 ) ) , playing video games ( Mnih et al . ( 2015 ) ) , playing adversarial board games ( Silver et al . ( 2016 ) ) and implementing robot navigation in complex environments ( Wang et al . ( 2018 ) ) . Nevertheless , for the majority of real applications , there is no reward in a long term until the agent reaches a goal state ( Wu & Chen ( 2007 ) ) , especially in unseen environments . In such cases , DRL has difficulty in carrying out the tasks . By observing the human intelligence that can explore their surroundings and learn valuable skills without reward , a couple of prior works have been recently proposed to generate skills without supervision by incorporating intrinsic motivation into DRL methods ( Barto ( 2013 ) , Ryan & Deci ( 2000 ) ) . Diverse skills can be autonomously acquired without extrinsic reward by maximizing an information theoretic objective using a maximum entropy policy ( DIAYN ( Eysenbach et al . ( 2018 ) ) ; VIC ( Gregor et al . ( 2016 ) ) ; DAS ( Sharma et al . ( 2019 ) ) ) . Discovered skills may help the exploration in complex environments , and can also serve as primitive skills for hierarchical DRL . Particularly , a high-level meta-policy could be adopted in the hierarchical framework to choose corresponding low-level primitive skills to complete tasks in order . Although discovered skills are both distinguishable and diverse , it is still exceedingly difficult to integrate such skills for a complex task that requires smooth transitions between skills ( Lee et al . ( 2018 ) ) . Take the basketball as an example : learning the passing , catching and shooting skills in an isolated way can not guarantee to score in the court due to the possible failure in the process of transitions between skills . To address this problem , we propose to further learn transitional skills ( LTS ) , where discovered primitive skills , same as prior works ( Eysenbach et al . ( 2018 ) ) , are distinguishable and as diverse as possible . More concretely , our LTS method learns both primitive and transitional skills by optimizing an information theoretic objective , where extra transitional skills are generated to fill in the gap between diverse primitive skills . For such purpose , aside from using the latent variable on which we condition primitive skills , an extra latent variable is introduced on which we condition our transitional policy . Furthermore , a compensation term has to be considered in the objective function because the distinct between primitive skills and transitional skills will lead to the decline of multual information between the latent variable corresponding to primitive skills and states generated by transitional skills . This compensation considers the divergence of the latent variable corresponding to primitive skills and that for transitional skills . Different from learning primitive skills , our learning process considers two arbitrary primitive skills and multiple transitional skills between them , where both primitive and transitional skillls are unknown and to be learned . By maximizing the multual information with compensation , both primitive skills and transitional skills are discovered , which can be used to effectively learn downstream tasks . On four simulated robotic tasks , experimental results show that our LTS can discover both primitive skills and transitional skills , successfully perform the transition between primitive skills that are distinguishable , and achieve a better peformance in comparison to the state-of-the-art baseline DIAYN . The main contributions of our work can be summarized as follows . Our proposed LTS can learn both primitive and transitional skills without extrinsic reward , where the primitive skills are distinguishable and diverse , and the transitional skills can accomplish smooth transitions between primitive skills . And extensive experiments are conducted , which demonstrates the effectiveness of our LTS method in solving downstream tasks , performing the transition between primitive skills as well as the weighted way to compose skills . 2 PRELIMINARIES . RL : In the standard RL setup , an agent interacts with an environment over discrete time . At time step t , the agent observes the current state st and selects an action at according to a policy π ( at|st ) . Then , the agent receives a reward rt and comes to the next state st+1 . The objective of learning is to maximize the discounted return R = ∑∞ t=0 γ trt of the policy π , where γ ∈ [ 0 , 1 ] is a discount factor . Learn Skills with RL : Using the notation from information theory : we introduce two random variables S and A for states and actions , respectively . To discover diverse skills , a latent variable ωn ∼ p ( ω ) is defined on which we condition the policy π ( at|st , ωn ) . Such policy is defined as a skill ( Eysenbach et al . ( 2018 ) ) . Prior works verify that maximizing the mutual information between the states S and the skills ωn will successfully generate lots of distinguishable , diverse and useful skills . By primarily maximizing the mutual information between the final state Sf and the skills ωn given the initial state S0 , I ( Sf ; ωn|S0 ) 1 , ( 1 ) the variational intrinsic control ( VIC ) ( Gregor et al . ( 2016 ) ) shows the success of acquiring distinguishable skills from the final state . Furthermore , in order to enhance the diversity of skills as much as possible , DIAYN primarily maximizes the mutual information between the state S at all time stamps and the skills ωn ( Eysenbach et al . ( 2018 ) ) , I ( St ; ωn ) + H [ A|S , ωn ] 2 , ( 2 ) which indicates that different skills generate different trajectories that traverse different states , and such diverse skills can be identified distinguishably . Both VIC and DIAYN successfully discover primitive skills by maximizing the mutual information betweens state and skills . However , prior works can not accomplish a variety of robot tasks that require smooth transition between skills . To address such problem , we propose the LTS scheme in this paper to learn both primitive and transitional skills by using an information theoretic objective function . 1The mutual information is denoted as the formation of conditional probability and contains the initial observation s0 : I ( Sf ; ω|S0 ) = − ∑ sf p ( sf |s0 ) log p ( sf |s0 ) + ∑ ω , sf pJ ( sf |s0 , ω ) pC ( ω|s0 ) log pJ ( sf |s0 , ω ) . The controllability distribution pC ( ω|s0 ) maximizes the behavior diversity . 2The second term suggests that each skill should act as randomly as possible , aiming improving the exploration . 3 METHODOLOGY . In this section , we elaborate our proposed LTS method to discover both primitive and transitional skills without extrinsic reward . Define by S and A states and actions , respectively . Define by Ωn ∼ p ( ω ) , 1 ≤ n ≤ N , N latent variables on which we condition the primitive skill π ( at|st , ω ) . Given the starting and ending primitive skill that are conditioned on ωi and ωj , respectively , we define zi , j , k , 1 ≤ k ≤ K − 1 , as K latent variables on which we condition the transitional skill π ( at|st , zi , j , k ) . Figure 1 shows the block diagram of primitive and transitional skills as well as the correponding states . Our objective is to discover primitive skills π ( at|st , ωi ) , π ( at|st , ωj ) , and transitional skills π ( at|st , zi , j , k ) . Specifically , we discover skills by maximizing the following mutual information L ( θ ) , I ( Sz ; Ωi ) + I ( Sz ; Ωj ) = Eωi∼p ( ω ) , sz∼π ( at|st , zi , j , k ) [ log p ( ωi|s z ) − log p ( ωi ) ] + Eωj∼p ( ω ) , sz∼π ( at|st , zi , j , k ) [ log p ( ωj |s z ) − log p ( ωj ) ] , ( 3 ) where Sz represents one of states Sti , j , k at time stamp t , 1 ≤ k ≤ K−1 , in Figure 1 . It is worth being noted that when the starting primitive skill is same with the ending primitive skill , no transitional skill needs to be discovered . In such case , maximizing L ( θ ) in ( 3 ) is equivalent to maximizing I ( Sz ; Ωi ) , which is identical to learning only primitive skills Gregor et al . ( 2016 ) ; Eysenbach et al . ( 2018 ) . The challege is how to solve such problem associated with two primitive skills and acquire the intrinsic reward . Our solution is to use a compensation for individual mutual information so that two components become separable . It is observed that , when the transitional skill π ( at|st , zi , j , k ) transits from the starting primitive skill π ( at|st , ωi ) to the ending primitive skill π ( at|st , ωj ) , the mutual information I ( Sz ; Ωi ) decreases progressively while the mutual information I ( Sz ; Ωj ) increases progressively . In order to compensate the mutual information , we define fdωi , zi , j , k as the divergence of the latent variables ωi and zi , j , k , which indicates the deviation of zi , j , k from ωi . Similarly , we define fdzi , j , k , ωj as the divergence of the latent variables zi , j , k and ωj , which indicates the deviation of ωj from zi , j , k . The divergence defined here can be measured in a couple of ways 3 . Therefore , the objective function to be maximized becomes L ( θ ) ' I ( Sz ; Ωi ) + I ( Sz ; Ωj ) + fdωi , zi , j , k + f d zi , j , k , ωj = Eωi∼p ( ω ) , sz∼π ( at|st , zi , j , k ) [ log p ( ωi|s z ) − log p ( ωi ) ] + fdωi , zi , j , k︸ ︷︷ ︸ L1 ( θ ) +Eωj∼p ( ω ) , sz∼π ( at|st , zi , j , k ) [ log p ( ωj |s z ) − log p ( ωj ) ] + fdzi , j , k , ωj︸ ︷︷ ︸ L2 ( θ ) . ( 4 ) The sufficient condition of maximizing the objecive function in ( 4 ) is to simultaneously maximize L1 ( θ ) and L2 ( θ ) . It is further observed that , when the transitional skill π ( at|st , zi , j , k ) transits from 3For details , please see it in Section 4 : Implementation . π ( at|st , ωi ) to π ( at|st , ωj ) , the divergence fdωi , zi , j , k increases progressively while the divergence fdzi , j , k , ωj decreases progressively . Thus , the first component L1 ( θ ) that has a decreasing mutual information and an increasing divergence , is symmetric to the second component L2 ( θ ) that has an increasing mutual information and a decreasing divergence . The sole difference lies in the transition from π ( at|st , ωi ) to π ( at|st , ωj ) or from π ( at|st , ωj ) to π ( at|st , ωi ) . So maximizing L1 ( θ ) is equivalent to maximizing L2 ( θ ) . Consequently , maximizing the overall objective function L ( θ ) is equivalent to maximizing L1 ( θ ) or L2 ( θ ) . In such case , we have arg max L ( θ ) ≈ arg max L1 ( θ ) = arg maxEωi∼p ( ω ) , sz∼π ( at|st , zi , j , k ) [ log p ( ωi|sz ) − log p ( ωi ) + fdωi , zi , j , k ] . ( 5 ) Because it is difficult to find p ( ωi|sz ) , it is common to approximate p ( ωi|sz ) with a learned discriminator qφ ( ωi|sz ) . According to Jensen ’ s Inequality , we know that replacing p ( ωi|sz ) with qφ ( ωi|sz ) gives us a variational lower bound G ( θ , φ ) of L ( θ ) . So we have arg max L ( θ ) ⇐⇒ arg maxEωi∼p ( ω ) , sz∼π ( at|st , zi , j , k ) [ log qφ ( ωi|sz ) − log p ( ωi ) + fdωi , zi , j , k ] ︸ ︷︷ ︸ , G ( θ , φ ) . ( 6 )

Summary: The paper aims to learn transitional skills in addition to creating diverse primitive skills without a reward function by using an information theoretic objective, borrowing ideas from instrinsic motivation. The paper addresses the challenge of learning diverse primitives and also learning to transition between any pair of them, which is valuable for real-world complex problem solving with sparse rewards. The proposed LTS approach is compared againt a state-of-the-art DIAYN approach on several benchmark control tasks.