Title: LEARNING TRANSITIONAL SKILLS WITH INTRINSIC MOTIVATION

Abstract: By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward. However, when considering to use multiple consecutive skills to complete a specific task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills. In this paper, we propose to learn transitional skills (LTS) in addition to creating diverse primitive skills. By introducing an extra latent variable for transitional skills and a compensation term for intrinsic reward, our LTS method discovers both primitive and transitional skills by maximizing the mutual information with compensation. By considering various robotic tasks, our results demonstrate the effectiveness of LTS on learning both diverse primitive skills and transitional skills, and show its superiority in smooth transition of skills over the state-of-the-art baseline DIAYN. In addition, we further show the significance of transitional skills for learning downstream tasks.

1. INTRODUCTION:

Deep reinforcement learning (DRL) has shown its great effectiveness in learning various rewarddriven skills in wide domains, such as performing robotic manipulation tasks (Levine et al. (2016) ), playing video games (Mnih et al. (2015) ), playing adversarial board games (Silver et al. (2016) ) and implementing robot navigation in complex environments (Wang et al. (2018) ). Nevertheless, for the majority of real applications, there is no reward in a long term until the agent reaches a goal state (Wu & Chen (2007) ), especially in unseen environments. In such cases, DRL has difficulty in carrying out the tasks.

By observing the human intelligence that can explore their surroundings and learn valuable skills without reward, a couple of prior works have been recently proposed to generate skills without supervision by incorporating intrinsic motivation into DRL methods (Barto (2013) , Ryan & Deci (2000) ). Diverse skills can be autonomously acquired without extrinsic reward by maximizing an information theoretic objective using a maximum entropy policy (DIAYN (Eysenbach et al. (2018) ); VIC (Gregor et al. (2016) ); DAS (Sharma et al. (2019) )). Discovered skills may help the exploration in complex environments, and can also serve as primitive skills for hierarchical DRL. Particularly, a high-level meta-policy could be adopted in the hierarchical framework to choose corresponding low-level primitive skills to complete tasks in order.

Although discovered skills are both distinguishable and diverse, it is still exceedingly difficult to integrate such skills for a complex task that requires smooth transitions between skills (Lee et al. (2018) ). Take the basketball as an example: learning the passing, catching and shooting skills in an isolated way cannot guarantee to score in the court due to the possible failure in the process of transitions between skills. To address this problem, we propose to further learn transitional skills (LTS), where discovered primitive skills, same as prior works (Eysenbach et al. (2018) ), are distinguishable and as diverse as possible.

More concretely, our LTS method learns both primitive and transitional skills by optimizing an information theoretic objective, where extra transitional skills are generated to fill in the gap between diverse primitive skills. For such purpose, aside from using the latent variable on which we condition primitive skills, an extra latent variable is introduced on which we condition our transitional policy. Furthermore, a compensation term has to be considered in the objective function because the distinct between primitive skills and transitional skills will lead to the decline of multual information between the latent variable corresponding to primitive skills and states generated by transitional skills. This compensation considers the divergence of the latent variable corresponding to primitive skills and that for transitional skills.

Different from learning primitive skills, our learning process considers two arbitrary primitive skills and multiple transitional skills between them, where both primitive and transitional skillls are unknown and to be learned. By maximizing the multual information with compensation, both primitive skills and transitional skills are discovered, which can be used to effectively learn downstream tasks. On four simulated robotic tasks, experimental results show that our LTS can discover both primitive skills and transitional skills, successfully perform the transition between primitive skills that are distinguishable, and achieve a better peformance in comparison to the state-of-the-art baseline DIAYN.

The main contributions of our work can be summarized as follows. Our proposed LTS can learn both primitive and transitional skills without extrinsic reward, where the primitive skills are distinguishable and diverse, and the transitional skills can accomplish smooth transitions between primitive skills. And extensive experiments are conducted, which demonstrates the effectiveness of our LTS method in solving downstream tasks, performing the transition between primitive skills as well as the weighted way to compose skills.

2. PRELIMINARIES:

RL: In the standard RL setup, an agent interacts with an environment over discrete time. At time step t, the agent observes the current state s t and selects an action a t according to a policy π(a t |s t ). Then, the agent receives a reward r t and comes to the next state s t+1 . The objective of learning is to maximize the discounted return R = ∞ t=0 γ t r t of the policy π, where γ ∈ [0, 1] is a discount factor.

Learn Skills with RL: Using the notation from information theory: we introduce two random variables S and A for states and actions, respectively. To discover diverse skills, a latent variable ω n ∼ p(ω) is defined on which we condition the policy π(a t |s t , ω n ). Such policy is defined as a skill (Eysenbach et al. (2018) ). Prior works verify that maximizing the mutual information between the states S and the skills ω n will successfully generate lots of distinguishable, diverse and useful skills.

By primarily maximizing the mutual information between the final state S f and the skills ω n given the initial state S 0 , I(S f ;

EQUATION

the variational intrinsic control (VIC) (Gregor et al. (2016)) shows the success of acquiring distinguishable skills from the final state. Furthermore, in order to enhance the diversity of skills as much as possible, DIAYN primarily maximizes the mutual information between the state S at all time stamps and the skills ω n (Eysenbach et al. (2018) ),

EQUATION

which indicates that different skills generate different trajectories that traverse different states, and such diverse skills can be identified distinguishably.

Both VIC and DIAYN successfully discover primitive skills by maximizing the mutual information betweens state and skills. However, prior works cannot accomplish a variety of robot tasks that require smooth transition between skills. To address such problem, we propose the LTS scheme in this paper to learn both primitive and transitional skills by using an information theoretic objective function.

Primitive States:

Primitive States Transitional States ... ... 

3. METHODOLOGY:

In this section, we elaborate our proposed LTS method to discover both primitive and transitional skills without extrinsic reward. Define by S and A states and actions, respectively. Define by Ω n ∼ p(ω), 1 ≤ n ≤ N , N latent variables on which we condition the primitive skill π(a t |s t , ω).

Given the starting and ending primitive skill that are conditioned on ω i and ω j , respectively, we define z i,j,k , 1 ≤ k ≤ K -1, as K latent variables on which we condition the transitional skill π(a t |s t , z i,j,k ).

Figure 1 shows the block diagram of primitive and transitional skills as well as the correponding states. Our objective is to discover primitive skills π(a t |s t , ω i ), π(a t |s t , ω j ), and transitional skills π(a t |s t , z i,j,k ). Specifically, we discover skills by maximizing the following mutual information

EQUATION

where S z represents one of states S t i,j,k at time stamp t, 1 ≤ k ≤ K -1, in Figure 1 . It is worth being noted that when the starting primitive skill is same with the ending primitive skill, no transitional skill needs to be discovered. In such case, maximizing L(θ) in (3) is equivalent to maximizing I(S z ; Ω i ), which is identical to learning only primitive skills Gregor et al. (2016) ; Eysenbach et al. (2018) .

The challege is how to solve such problem associated with two primitive skills and acquire the intrinsic reward. Our solution is to use a compensation for individual mutual information so that two components become separable.

It is observed that, when the transitional skill π(a t |s t , z i,j,k ) transits from the starting primitive skill π(a t |s t , ω i ) to the ending primitive skill π(a t |s t , ω j ), the mutual information I(S z ; Ω i ) decreases progressively while the mutual information I(S z ; Ω j ) increases progressively.

In order to compensate the mutual information, we define f d ωi,z i,j,k as the divergence of the latent variables ω i and z i,j,k , which indicates the deviation of z i,j,k from ω i . Similarly, we define f d z i,j,k ,ωj as the divergence of the latent variables z i,j,k and ω j , which indicates the deviation of ω j from z i,j,k . The divergence defined here can be measured in a couple of ways3 . Therefore, the objective function to be maximized becomes L(θ)

EQUATION

The sufficient condition of maximizing the objecive function in (4) is to simultaneously maximize L 1 (θ) and L 2 (θ). It is further observed that, when the transitional skill π(a t |s t , z i,j,k ) transits from π(a t |s t , ω i ) to π(a t |s t , ω j ), the divergence f d ωi,z i,j,k increases progressively while the divergence f d z i,j,k ,ωj decreases progressively. Thus, the first component L 1 (θ) that has a decreasing mutual information and an increasing divergence, is symmetric to the second component L 2 (θ) that has an increasing mutual information and a decreasing divergence. The sole difference lies in the transition from π(a t |s t , ω i ) to π(a t |s t , ω j ) or from π(a t |s t , ω j ) to π(a t |s t , ω i ). So maximizing L 1 (θ) is equivalent to maximizing L 2 (θ).

Consequently, maximizing the overall objective function L(θ) is equivalent to maximizing L 1 (θ) or L 2 (θ). In such case, we have

EQUATION

)

Because it is difficult to find p(ω i |s z ), it is common to approximate p(ω i |s z ) with a learned discriminator q φ (ω i |s z ). According to Jensen's Inequality, we know that replacing p(ω i |s z ) with q φ (ω i |s z ) gives us a variational lower bound G(θ, φ) of L(θ). So we have

arg max L(θ) ⇐⇒ arg max E ωi∼p(ω),s z ∼π(at|st,z i,j,k ) log q φ (ω i |s z ) -log p(ω i ) + f d ωi,z i,j,k G(θ,φ)

.

(6)

4. IMPLEMENTATION:



4.1. HINDSIGHT AND ONE-HOT ENCODING:

In this section, we discuss about the implementation of learning transitional skills, where there are a couple of issues with the usage of the latent variable corresponding to transitional skills.

Problem 1: Along with the growth of the number of primitive skills N and transitional skills K -1, we have a high training complexity. In our approach, we calculate the conditional probability q φ (ω i |s z ) with a low efficiency because q φ (ω i |s z ) can just keep the consistency of z i,j,k with ω i but ignore the relation with ω j .

To enhance the efficiency, we utilize the hindsight experience reply mechanism to allow samplebased learning from the sparse reward.

Problem 2: The categorical distribution of the latent variable ω i , 1 ≤ i ≤ N , suffers from a dilemma when the discriminator q φ (ω i |s z ) judges the states.

Our solution is to use one-hot encoding for the latent variable ω i , 1 ≤ i ≤ N , corresponding to primitive skills. In such case, the latent variable z i,j,k corresponding to transitional skills has a different expression and further analysis is given in Appendix B.

In the implementation, we change q φ (ω i |s z ) in ( 6) to the following conditional probability distribution:

q φ ( Ω|s z ) = [q φ (ω 0 |s z ), q φ (ω 1 |s z ), ..., q φ (ω N -1 |s z )] T . (7)

Correspondingly, the divergence of f d i,j,k is organized as the following

EQUATION

where k = i. Consequently, the objective is to maximize the following G(θ, φ),

EQUATION

where

z i,k ∼ p z (z|ω i ) = {z i,k |z i,k = z i,j,k , 0 ≤ j ≤ N -1, z i,j,k ∈ p(z i,j |ω i , ω j )}, α is a

hyperparameter, and we ignore the term p(ω i ) by taking a uniform distribution for ω i .

Algorithm 1 Learning Transitional Skills (LTS) 1: while NOT converged do 2:

Sample ω i ∼ p(ω)

3:

Sample a skill z ∼ p z (z|ω i ) and an initial state s 0 ∼ p 0 (s)

4:

for t ← 1 to steps per episode do 5:

Sample an action a t ∼ π θ (a t |s t , z);

6:

Interact with the environment: s t+1 ∼ p(s t+1 |s t , a t );

7:

Compute D t = 1 N q φ ( Ω|s t+1 ) + αf d i,k 2 with the discriminator (φ); 8:

Set the reward for current skill: r t = D t . 9:

By using SAC, update the policy (θ) to maximize the discounted return R = ∞ t=0 γ t r t ;

10:

Update the discriminator (φ) to maximize D t with SGD.

11:

end for 12: end while

4.2. ALGORITHM:

We summarize our LTS method in Algorithm 1. At each roll-out, given the latent variable of primitive skill ω i , we sample a transitional skill z from a fixed distribution p z (z|ω i ). After the agent interacts with the environment at time step t, the discriminator finds the discriminability by

EQUATION

)

where s z t+1 denotes the states at time step t. As mentioned above, we encode primitive skills using one-hot encoding. We constrain j z i,j,k = 1, 0 ≤ z i,j,k ≤ 1, so we simplify the divergence as

f d i,k = Array(z i,j,k )

, where Array(•) means convert one-hot encoding into an array. (See example in Appendix B.1 for more details (16, 17)).

In addition, we adopt soft actor-critic (SAC) algorithm to train our policy, adding the regularization Ei,j[ H[A|S, Z i,j ] to maximize the policy's entropy over actions given states and skills.

5. RELATED WORK:

Real-world tasks often require diverse behaviors. Wang et al. (2017) notes that building versatile embodied agents capable of performing a wide and diverse set of behaviors is one of the long-standing challenges of AI. And learning continuous control of diverse behaviors in locomotion (Merel et al. (2017) ; Heess et al. (2017) ; Peng et al. (2017) ) and robotic manipulation (Ghosh et al. (2018) ; Gu et al. (2017) ) is an active research area. In this scenario, although some complex tasks can be solved through extensive reward engineering, undesired behaviors often emerge because of the sparse nature of reward (Riedmiller et al. (2018) ). Moreover, training complex skills from scratch is not computationally practical. These issues can be addressed by use of intrinsic motivation (Barto (2013); Chentanez et al. (2005) ; Singh et al. ( 2010)), which is a reward-free learning method. Historically, the intrinsic motivation comes from the tendency of organisms to play and explore their environment without any reward (Ryan & Deci (2000) , White (1959) ).

Another line of work that is conceptually close to our method copes with information theories that are used to drive the agent's exploration. The information gain is a reward based on the reduction of uncertainty on environment's dynamics (Little & Sommer (2013) ; Oudeyer & Kaplan (2007)), which can also be assimilated to learning progress (Frank et al. (2013); Oudeyer & Kaplan (2007) ). This can push agents into unknown areas on the one hand, and prevent them from being attracted to random areas on the other.

Recent work has also applied information theory for skill discovery. VIC (Gregor et al. (2016) ) is an optional discovery technique by optimizing a variational lower bound on the mutual information between the context and the final state in a trajectory, conditioned on the initial state. Furthermore, DIAYN (Eysenbach et al. (2018) ) maximizes the mutual information between states and skill to achieve diversity and shows the interest as a pre-training for hierarchical reinforcement learning or as an initialization for learning a task. While discriminative embedding reward networks (DIS-CERN) (Warde-Farley et al. (2018) aim to simultaneously learn a goal-conditioned policy and a goal achievement reward function by maximizing the mutual information between the goal state and the achieved state. Let us notice that the skill space here is discrete, with just one or multiple policies. However, we considered the relationship between different skills during the training process and finally formed a continuous skill space, likely because of inducing a novel latent variable for transitional skills.

In addition, it is important to point out that our skills are transitional with an intrinsically driven approach, which is very different from numerous previous works. While Sharma et al. (2019) discovers predictable behaviors to let the single skill more predictable, it need an external model-predictivecontrol (MPC) paradigm (Garcia et al. (1989) ) to connect skills. Peng et al. (2019) learns reusable motor primitives that can be composed to produce a continuous spectrum of skills. To bridge the gap between skills, Lee et al. (2018) propose a transition policy to get a new smooth skill. In comparison, our method captures intrinsic transition, which is independent from external tasks, and could eliminate the extra fine-tuning process.

6. EXPERIMENTS:

In our experiments, we aim to demonstrate the effectiveness of our approach for learning primitive and transitional skills. We evaluate LTS and compare it to prior works. In this section, we provide visualizations and quantitative analysis for our LTS method. The tasks of CartPole-v0, MountainCar-v0, Pendulum-v0 and HalfCheetah-v3 are based on OpenAI gym4 .

6.1. DIVERSITY AND TRANSITIONAL SKILLS:

To evaluate different skills in different environment, we extract features from the observation of the agent, e.g. in the MountainCar environment, where we use the variance of the car's altitude as the feature. For more details on the features in other environments, please refer to Appendix C. It is observed that, these four skills have different movement patterns so that these skills are easy to be distinguished, so as the skills shown in Figure 2 (c). Moreover, we consider a different number of primitive skills and different environments as in Appendix E, from which it is observed that all primitive skills have an evident difference in feature statistics and are easy to be distinguished.

Transition. Furthermore, we use the transitional skill z 0,1,k to show the performance of transition from one primitive skill ω 0 = [1, 0, 0, 0]) to another ω 1 = [0, 1, 0, 0]), where the number of transitional skills is set as 9. In Figure 3 (a) and 3(b), it is observed that the primitive skill ω 0 smoothly changes to ω 1 via 9 transitional skills. More specifically, there exists a slight increment on the amplitude of features in the first three skills, which is followed by consecutive declines until the ending primitive skill ω 1 is discovered. In Figure 3 (c), we can see the clear transition process between two primitive skills. More experiments on skill transition are given in Appendix F. This demonstrates the effectiveness of our LTS method on discovering transitional skills and accomplishing the successful transition between two primitive skills.

6.2. COMPARISON WITH DIAYN:

In this subsection, we compare our LTS method with the state-of-the-art DIAYN (Eysenbach et al. (2018) ) in terms of learning diverse primitive skills and transitional skills. Experimental results show that LTS achieves an approximate performance on diversity of primitive skills, and a much better performance on skill transition.

Diversity: Different skills have different means or variances from their own trajectories. In this subsection, we use (1) variance of means and (2) mean of variances of the features as metrics to evaluate the diversity of learned primitive skills using LTS and DIAYN. The lower variance of means does not degrade the diversity of learned primitive skills, which can be observed from Figure 4 (b). Although there is a large difference between experiments with random seeds, LTS has generally a similar mean of variance as DIAYN, indicating that our method performs similar with the baseline in terms of learning diverse primitive skills. Transition: We also conduct experiments to evaluate the transitional skills of LTS and DI-AYN, where we use identical encoding scheme for both and the number of transitional skills is set as 9.

Figure 5 shows the all statistical characteristics of 11 skills that including two primitive skills, where the value denotes the mean of features like the middle bar in Figure 3 (b). It is observed that LTS experiences a smooth transition state from one primitive skill to another while the transitional states in DIAYN suffer from one sharp-rising phase and two steady phases indicating a rigid changes of transition states. The steady phase indicates that changing z will not change states very much, and the sharp change indicates that a little fluctuate of z will cause a huge change of states, which is not reasonable in practice. Because of the huge gap, controlling the skills learned by DIAYN changing from ω 0 to ω 1 will not push the agent standing in the state space of ω 2 . On the contrary, we control the transitional skills in LTS changing from ω 0 to ω 1 as in Figure 2 (b), and the agent reaches the target states.

More studies including real statistical characteristics of different transitional skills are conducted and the results are reported in Appendix G. Our LTS method suffers from the high training complexity to learn transitional skills. A reasonable approach to tackle this problem is to use a fixed number of transitional skills. In this experiment, we set this number as 3, indicating that the nonzero element of z i,j,k is from the set {0.25, 0.50, 0.75} in the training phase.

6.3. GENERALIZATION:

Figure 6 shows the transitional skills, where the training phase considers 3 transitional skills and adopting 3 and 50 transitional skills to evaluate the generalization of our LTS method. It is observed that the transitional skills in blue (traning and testing both with 3 transitional skills) suffers from severe declines, possibly leading to the failure of the transition process in practice. Fortunately, the transitional skills in red (training with 3 transitional skills while testing with 50) go through a steady and smooth process from one Table 1 : Success rate(%) for practical tasks, where our method is compared with the baselines with or without hierarchical framework. Our transitional skills outperforms the baselines in all cases, demonstrating the effectiveness of the learning transitional skills for down-stream tasks. Here we adopt spectral clustering to cluster the trajectories. When the label of ending segment of the trajectory keeps the same label with that generated the target skills solely, the transition is regarded as a success; otherwise a failure transition. The target skills are regard as the goal of our tasks, which are generated by sample primitive skills or weighting the actions of different primitives. primitive skill to another. This experiment demonstrates that the strong generalization of LTS can improve the efficiency of the training. These means that even we use a small number of transitional skills in training, we could sample the transitional skills with a higher density and achieve the success of the skill transition, by which we could further improve the efficiency of training model. 

Policy:



6.4. EQUIP THE AGENT WITH MORE TRANSITIONAL SKILLS:

Figure 7 shows much more skills and corresponding transitions between them, where each small square denotes a different skill, the number of primitive skills is 2 and the number of transitional skills is 98. Two primitive skills [1, 0, 0, 0] and [0, 1, 0, 0] locate at the lower left and upper right squares.

It is observed that our LTS method is able to accomplish a smooth transition between two arbitrary skills, whatever primitive skills or transitional skills. However, DIAYN suffers from a rigid transition between two skills, resulting in a possible failure in practical transition. Furthermore, these results provide a deep insight that LTS has the ability to learn a larger continuous skill space. The agent equipped with such numerous skills is expected to become much more powerful.

6.5. TRANSITION WITH HIERARCHICAL FRAMEWORK:

In this subsection, we conduct experiments to answer the question: how do transitional skills yield benefit to the downstream tasks? To evaluate the effect of the transition policies, we take as benchmark hierarchical control for primitives learned by DIAYN. Moreover, we conduct hierarchical framework to weight (or choose) the action modeled by different primitive skills. For primitive skills modeled by DIAYN, we consider the following approaches:1) DIAYN-random:

Transition policy with random weights to affect actions; 2) DIAYN-smooth: Transition policy with smoothly changed weights (artificially designed) to affect actions; 3) DIAYN-master: Transition policy with optimal weights learned by meta-policy.

For transitional skills modeled by LTS, we consider the following approaches: 1) LTS-random: Randomly choose transitional skills between two primitive skills; 2) LTS-smooth: Choosing continously changed transitional skills z i,j,k ( 1 ≤ k ≤ K -1); 3) LTS-master: Transfer policy with optimal weights learned by meta-policy.

Table 1 shows the performance. It is observed that we choose skills randomly or weighted actions modeled by primitive skills, the transition process is more likely to fail. Even with an extensively forecast through the hierarchical framework, DIAYN-master is possible to fail. We believe this is because the primitive skills have a large gap and therefore, one skill are unable to associate with another. Even with meta-policy, it is also hard to transit between primitive skills. In contrast, LTSsmooth has a better performance than DIAYN-master without any meta-policy because of abundant transitional skills. Moreover, LTS-master could select most appropriate transitional skills to fill in the gap between two arbitrary skills.

6.6. VISUALIZING TRANSITION TRAJECTORIES:

Figure 8 shows the two-dimensional t-SNE embedding of the agent's states generated by different skills. Different colors represent different skills (including primitive skills and transitional skills).

The red lines represent the agents' transition trajectories. In Figure 8 (a) and 8(b), we can see a clear boundary between two kinds of skills, which is consistent with the boundary of Figure 7 .

Starting from the initiatial state, the agent tries to transit to another state. But both of agents with DIAYN-random-policy and DIAYN-smooth-policy fail for transiton. The state space of different skills in Figure 8 (c) seems to be more diverse, while it is a pseudomorphism. Because the space is a two-dimensional mapping, the difference between Figure 8 (a) and 8(b) only discloses that different skills do exist a conclusive gap between two kinds of skills, but this uncovers that the learned transitional skills are also distinguishable. We can find in Figure 8 (c) also exits many intersections which are the crucial transition states between two different skills. In the t-SNE space, distant dots with same color are also connective, so there exits many long line. And the intersections existing two endpoints must go through a transition between two skills. Particularly, the trajectory of the agent in Figure 8 (c) shows the transition between different 11 skills.

7. CONCLUSION:

In this paper, we introduce a novel LTS method to learn transitional skills without extrinsic reward by using two kinds of latent varibles to depict different skills. As a result, LTS can discover both primitive skills and transitional skills. Also, LTS achieves a great success in the smooth transition from one primitive skill to another and exhibits its potential in learning a large continuous skill space. Extensive experiments demonstrate the effectiveness of our LTS in the discovery of diverse skills and the smooth transition between skills even for complex down-stream tasks.

Appendices:



A METHODOLOGY:

A.1 DERIVATION OF THE VARIATION BOUND ON MUTUAL INFORMATION

Here we derive the variational bound:

EQUATION

B IMPLEMENTATION:

B.1 HINDSIGHT AND ONE-HOT ENCODING

In our approach, we model the conditional probability given by the discriminator q φ (ω i |s t ) w.r.t. the divergence (or distance) between transitional skill z i,j,k and specific primitive skill ω i . Although (4) constrains two mutual information terms, but (??) is adopted to train our model, which constrains only one term. These efficiency is relatively low. For example, if we want to simultaneously constrain the probability q φ (ω i1 |s t ) and q φ (ω i2 |s t ) in terms of the same transition state s t generated by the same skill z (z = z i1,j1,k1 = z i2,j2,k2 ) but two different primitive skills (ω i1 and ω i2 ) in the training stage, we must wait for the next time step to sample the same z and a different ω in the experience. The efficiency is relatively low because that q φ (ω i |s t ) just constrain the consistency of z i,j,k and ω i which ignores the consistency to other primitive skills. So we utilize the hindsight experience reply mechanism to allow sample-efficient learning form sparse rewards. We calculate a distribution of the conditional probability given by the discriminator instead of a single value. So we change q φ (ω i |s t ) to the conditional probability distribution:

q φ (Ω, s t ) = [q φ (ω 0 |s t ), q φ (ω 1 |s t ), ..., q φ (ω N -1 |s t )] T .

(

EQUATION

By doing this, we could simultaneously constrain the similarity probability distribution given by the discriminator with related to all primitive skills. 

EQUATION

)

On the other hand, for categorical encoded primitive skills, when ω i -ω j = ±1 (i = j), the discriminator q φ will face a dilemma: the intersection of transition states and primitive states are not empty, i.e. S T i,j ∩ S P = , leading to a conflict between diversity and transition. Here we give an example: considering transferring two skills from ω i to ω j (ω i = ω j ), the former categorical encoding will cause extra consumption: if ω i -ω j = ±1, primitive states with related to primitive skills {ω|ω ∈ [ω i , ω j ] or [ω j , ω i ]} will occur in the transition states. For three primitive skills ω i1 , Under review as a conference paper at ICLR 2020 ω i2 and ω i3 , given the optimal discriminator q * φ , there should be q * φ (ω i3 |s t ) = 0 (s t ∼ π(ω i1 ) and ω i1 = ω i3 ) and q * φ (ω i3 |s t ) = 0 (s t ∼ π(ω i2 ) and ω i2 = ω i3 ). While if ω i1 < ω i3 < ω i2 , q * φ (ω 3 |s t ) = log p(ω i1 ) -f d (ω i1 ) + c and q * φ (ω 3 |s t ) = log p(ω i2 ) -f d (ω i2 ) + c5 , which is in contrast to the conditional probability of 1 (ω i3 is also a primitive skill). So, we encode ω ∼ p(ω) with one-hot way: ω 0 = [1, 0, 0, ..., 0]; ω 1 = [0, 1, 0, ..., 0];

... ω N -1 = [0, 0, 0, ..., 1].

(14):

And we denote primitive skills and transitional skills as a set Z + i,j :

EQUATION

where the value of i-th and j-th column keeps decreasing and increasing respectively. Other column always keep 0, which could constrain the incoherence between transition states and other primitive skills. Without causing any misunderstanding, following z i,j,k all comes from Z + i,j . For transition, we assure that the change of z i,j,k only happens on the corresponding dimension, which overcomes the conflict caused by categorical encoding. As show in Fig. 9 , all transition skills in Z 3,1 and primitive skills ω 3 are orthogonal. The transition only reflects on the plane defined by the corresponding primitive skills. Although these will induce a problem of the generation issue, but this could also learning continuous transitional skills as shown in Figure 6 and 7. In fact, there is more than one transitional path, which can be a directed line or any directed curve. As in Figure 7 (b), we can find more than one transitional paths.

An example Z + i,j shows below:

EQUATION

where

N = 4, K -1 = 3. A transition path could be ω 3 → z 3,1,1 → z 3,1,2 → z 3,1,3 → ω 1 .

And an alternative way could be [0, 0, 0, 1

, 0] → [0, 0.2, 0, 0.8, 0] → [0.1, 0.3, 0.2, 0.3, 0.1] → [0, 0.8, 0, 0.2, 0] → [0, 1, 0, 1, 0]. For the similairity f d i,k

, examples are shown below (generated by Z + 3,1 in Formula( 16) ):

EQUATION

where The value of the y-axis denotes the mean of feature values (which is also the statistical characteristic). Here we do not test the transition processes between skills and we just run different transitional skills independently. There should be many isolated dots instead a transition line. Skills with two distant features will means a larger probability of failure of transition. For DIAYN's primitive skills, there exists a large gap which notes an harder transition between the two skills, comparing the smooth line of LTS's skills. Please note that there exits no faster transition instead a large gap. 

f s i,j = 1 -f d i,j .

:

The mutual information is denoted as the formation of conditional probability and contains the initial observation s0:I(S f ; ω|S0) =s f p(s f |s0) log p(s f |s0) + ω,s f p J (s f |s0, ω)p C (ω|s0) log p J (s f |s0, ω).The controllability distribution p C (ω|s0) maximizes the behavior diversity.

2 The second term suggests that each skill should act as randomly as possible, aiming improving the exploration.

For details, please see it in Section

4: Implementation.

http://gym.openai.com/

c is a constant.

https://gym.openai.com/envs/CartPole-v0/

https://gym.openai.com/envs/MountainCar-v0/

https://gym.openai.com/envs/Pendulum-v0/

https://gym.openai.com/envs/HalfCheetah-v2/

Figures and Tables:

Figure 1: Block diagram of primitive and transitional skills.

Trajectories of 4 skills.

Box-plots of 4 skills. (c) Frames of 3 skills.

Figure 2: Diverse primitive skills (ω 0 , ω 1 , ω 2 , ω 3 ) in MountainCar (a, b) and HalfCheetah (c). (a) shows the changes of the feature values of 4 diverse skills. (b) shows the corresponding statistical characteristics of 4 skills in (a), where a single box-plot denotes a skill. (c) shows 3 diverse skills of the half-cheetah.

Figure 2 illustrates the discovered primitive skills in the MountainCar and Halfcheetah environment. As shown in Figure 2(a), all four skills move in an periodic manner. Corresponding to all 4 skills in Figure 2(a), the statistical values of features are shown in Figure 2(b) using Box-plot.It is observed that, these four skills have different movement patterns so that these skills are easy to be distinguished, so as the skills shown in Figure2(c). Moreover, we consider a different number of primitive skills and different environments as in Appendix E, from which it is observed that all primitive skills have an evident difference in feature statistics and are easy to be distinguished.

Box-plots of transition. (c) Illustration of 4 transitional skills.

Figure 3: Transition in Mountain-Car (a,b) and HalfCheetah (c). (a) shows the transition of the trajectory, where 9 transitional skills are uniformly distributed along the horizontal axis from Step 0 to 5500. (b) shows the transition of feature statistics, where the features of two primitive skills and 9 transitional skills are included. (c) shows the sampled frames of the transition trajectory, where K -1 = 9.

Figure 4(a) shows the variance of means while Figure 4(b) shows the mean of variances, where the black perpendicular line represents the range of values from a single experiment and we collect all trajectories of various primitive skills. In Figure 4(a), the height of this line depicts the range of variance of means by considering all trajectories of primitive skills. As shown, LTS obtains a lower variance of means in compare to DIAYN because the learned transitional skills from LTS affect the variance of learned primitive skills.

Figure 4: Comparison of diversity of primitive skills between LTS and DIAYN.

Figure 6: Transitional skills with related to 2 different number of them.

Figure 7: Skill space of DIAYN and LTS.

Figure 8: Visualization of transition trajectories of (a) DIAYN-random, (b) DIAYN-smooth and (c) LTS-smooth. Top row contains the states generated by different skills noted by the dots and the red lines represent the agent's sampled trajectories, both of which are mapped to a two-dimensional space by t-SNE. Bottom contains the frames of transition trajectories.

Figure 9: The space of primitive skills and transitional skills, where we set N = 3, K -1 = 3.

Figure 10: Cartpole primitive skills. (a) -(j) stand for 10 random trials with 4 primitive skills for each, and different skills are distinguished by Boxplot. (k) shows the skills in time domain for one trial.

Figure 15: Cartpole transition process. (a) (e), (b) (f), (c) (g), and (d) (g) are from different 4 trials respectively. Each transitional skill holds 500 steps and then transfers the final state to the next skill.

Figure 16: Transitional skills comparison on MountainCar. The subgraphs (a),(b),(c),(d) represent the mean of features in terms of the transitional skills, and subgraphs (e),(f),(g),(h) represent the variance of features.



Table content: <table><tr><td/><td/><td>: Parameter setting</td></tr><tr><td>Parameters</td><td/><td>Description</td><td>Value</td></tr><tr><td/><td/><td/><td>32 for CartPole,</td></tr><tr><td>H</td><td>hidden state size</td><td/><td>MountainCar and Pendulum;</td></tr><tr><td/><td/><td/><td>64 for HalfCheetah</td></tr><tr><td>layer</td><td>layer count</td><td/><td>3</td></tr><tr><td>epoch</td><td>eposide size</td><td/><td>2*12(cpus)</td></tr><tr><td>vf lr</td><td colspan="2">value network learning rate</td><td>1e-5</td></tr><tr><td>dc lr</td><td colspan="2">discriminator network learning rate</td><td>5e-4</td></tr><tr><td>pi lr</td><td colspan="2">policy network learning rate</td><td>3e-4</td></tr><tr><td colspan="3">max episodes the maximal length of episode</td><td>250</td></tr><tr><td colspan="4">train dc iterv epoch number of training discriminator network</td><td>5</td></tr><tr><td colspan="3">train ac iterv epoch number of training actor network</td><td>1</td></tr><tr><td>train v iterv</td><td colspan="2">epoch number of training critic network</td><td>1</td></tr><tr><td>train dc iters</td><td colspan="3">iteration number of updating discriminator network 80</td></tr><tr><td>train ac iters</td><td colspan="2">iteration number of updating actor network</td><td>50</td></tr><tr><td>train v iters</td><td colspan="2">iteration number of updating critic network</td><td>1</td></tr><tr><td/><td colspan="2">Table 3: Selection of skill features.</td></tr><tr><td colspan="2">RL enviroment</td><td>Observations</td><td>Selected as skill feature</td></tr><tr><td>CartPole</td><td colspan="2">0: Cart Position;</td><td>2: Pole Angle</td></tr><tr><td/><td colspan="2">1: Cart Velocity</td></tr><tr><td/><td colspan="2">2: Pole Angle</td></tr><tr><td/><td colspan="2">3: Pole Velocity at Tip</td></tr><tr><td>MountainCar</td><td>0: Position</td><td/><td>0: Position</td></tr><tr><td/><td>1: Velocity</td><td/></tr><tr><td>Pendulum</td><td colspan="2">0: cos(Angle)</td><td>1: sin(Angle)</td></tr><tr><td/><td>1: sin(Angle)</td><td/></tr><tr><td/><td>2: speed</td><td/></tr><tr><td>HalfCheetah</td><td colspan="2">17-dim values (including the positions</td><td>the frames</td></tr><tr><td/><td colspan="2">and velocities)</td></tr></table>

Peer review (iclr2020):
General description The paper tackles the problem of transition between different skills in hierarchical reinforcement learning. In particular, they follow the work of VIC and DIAYN and define an information-theoretic objective function that maximizes the mutual information between the future states and options, given the initial state, while minimizing the distance between the options and the so-called transitional states. The algorithm, LTS, is compared to DIAYN on three environments, namely CartPole, MontainCar, and Pendulum. General remarks, The problem tackled and the proposed idea are interesting; however, I am not fully convinced by the derivation and the experiments of the paper. The writing is in general not particularly clear and the notations are hard to follow, the symbols are often bloated with superscripts that are not clearly defined, and mixing capital and small letters for random variables and their realization. On the derivation, Eq 3. How is it possible to replace p(s^p | p^t) with p(omega|s_t)? I understand the connection between the two but what guarantees the mutual information is still maximized? (the whole derivation depends on that) The experiments and the plots are interesting, showing a smoother transition between skills than DIAYN, however, it is still not clear how that can help solve the task at hand. Could the author elaborate on that please? Some details: - Page 2 in should be sim. In general, the notation does not clearly distinguish a random variable (in capital) from a realization (in small letters). For instance, page 3, big Omega (the random variable I presume) is written with a subscript i to indicate the ith skill. - Footnote 1 page 2, a log is missing in the MI definition. - Page 3, what does t refer to in S^t_{i ,j, 1}? - Page 3, muture -> mutual. - the paper mentions the experiments are conducted on MuJoCo but the appendix mentions the classical OpenAI Gym experiments.