Year: 2019

Abstract:
By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward. However, when considering to use multiple consecutive skills to complete a specific task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills. In this paper, we propose to learn transitional skills (LTS) in addition to creating diverse primitive skills. By introducing an extra latent variable for transitional skills and a compensation term for intrinsic reward, our LTS method discovers both primitive and transitional skills by maximizing the mutual information with compensation. By considering various robotic tasks, our results demonstrate the effectiveness of LTS on learning both diverse primitive skills and transitional skills, and show its superiority in smooth transition of skills over the state-of-the-art baseline DIAYN. In addition, we further show the significance of transitional skills for learning downstream tasks.

heading: 1 INTRODUCTION
text: Deep reinforcement learning (DRL) has shown its great effectiveness in learning various rewarddriven skills in wide domains, such as performing robotic manipulation tasks (Levine et al. (2016)), playing video games (Mnih et al. (2015)), playing adversarial board games (Silver et al. (2016)) and implementing robot navigation in complex environments (Wang et al. (2018)). Nevertheless, for the majority of real applications, there is no reward in a long term until the agent reaches a goal state (Wu & Chen (2007)), especially in unseen environments. In such cases, DRL has difficulty in carrying out the tasks.
By observing the human intelligence that can explore their surroundings and learn valuable skills without reward, a couple of prior works have been recently proposed to generate skills without supervision by incorporating intrinsic motivation into DRL methods (Barto (2013),Ryan & Deci (2000)). Diverse skills can be autonomously acquired without extrinsic reward by maximizing an information theoretic objective using a maximum entropy policy (DIAYN (Eysenbach et al. (2018)); VIC (Gregor et al. (2016)); DAS (Sharma et al. (2019))). Discovered skills may help the exploration in complex environments, and can also serve as primitive skills for hierarchical DRL. Particularly, a high-level meta-policy could be adopted in the hierarchical framework to choose corresponding low-level primitive skills to complete tasks in order.
Although discovered skills are both distinguishable and diverse, it is still exceedingly difficult to integrate such skills for a complex task that requires smooth transitions between skills (Lee et al. (2018)). Take the basketball as an example: learning the passing, catching and shooting skills in an isolated way cannot guarantee to score in the court due to the possible failure in the process of transitions between skills. To address this problem, we propose to further learn transitional skills (LTS), where discovered primitive skills, same as prior works (Eysenbach et al. (2018)), are distinguishable and as diverse as possible.
More concretely, our LTS method learns both primitive and transitional skills by optimizing an information theoretic objective, where extra transitional skills are generated to fill in the gap between diverse primitive skills. For such purpose, aside from using the latent variable on which we condition primitive skills, an extra latent variable is introduced on which we condition our transitional policy. Furthermore, a compensation term has to be considered in the objective function because the
distinct between primitive skills and transitional skills will lead to the decline of multual information between the latent variable corresponding to primitive skills and states generated by transitional skills. This compensation considers the divergence of the latent variable corresponding to primitive skills and that for transitional skills.
Different from learning primitive skills, our learning process considers two arbitrary primitive skills and multiple transitional skills between them, where both primitive and transitional skillls are unknown and to be learned. By maximizing the multual information with compensation, both primitive skills and transitional skills are discovered, which can be used to effectively learn downstream tasks. On four simulated robotic tasks, experimental results show that our LTS can discover both primitive skills and transitional skills, successfully perform the transition between primitive skills that are distinguishable, and achieve a better peformance in comparison to the state-of-the-art baseline DIAYN.
The main contributions of our work can be summarized as follows. Our proposed LTS can learn both primitive and transitional skills without extrinsic reward, where the primitive skills are distinguishable and diverse, and the transitional skills can accomplish smooth transitions between primitive skills. And extensive experiments are conducted, which demonstrates the effectiveness of our LTS method in solving downstream tasks, performing the transition between primitive skills as well as the weighted way to compose skills.

heading: 2 PRELIMINARIES
text: RL: In the standard RL setup, an agent interacts with an environment over discrete time. At time step t, the agent observes the current state st and selects an action at according to a policy π(at|st). Then, the agent receives a reward rt and comes to the next state st+1. The objective of learning is to maximize the discounted return R = ∑∞ t=0 γ
trt of the policy π, where γ ∈ [0, 1] is a discount factor.
Learn Skills with RL: Using the notation from information theory: we introduce two random variables S and A for states and actions, respectively. To discover diverse skills, a latent variable ωn ∼ p(ω) is defined on which we condition the policy π(at|st, ωn). Such policy is defined as a skill (Eysenbach et al. (2018)). Prior works verify that maximizing the mutual information between the states S and the skills ωn will successfully generate lots of distinguishable, diverse and useful skills.
By primarily maximizing the mutual information between the final state Sf and the skills ωn given the initial state S0,
I(Sf ;ωn|S0)1, (1)
the variational intrinsic control (VIC) (Gregor et al. (2016)) shows the success of acquiring distinguishable skills from the final state. Furthermore, in order to enhance the diversity of skills as much as possible, DIAYN primarily maximizes the mutual information between the state S at all time stamps and the skills ωn (Eysenbach et al. (2018)),
I(St;ωn) + H[A|S, ωn]2, (2)
which indicates that different skills generate different trajectories that traverse different states, and such diverse skills can be identified distinguishably.
Both VIC and DIAYN successfully discover primitive skills by maximizing the mutual information betweens state and skills. However, prior works cannot accomplish a variety of robot tasks that require smooth transition between skills. To address such problem, we propose the LTS scheme in this paper to learn both primitive and transitional skills by using an information theoretic objective function.
1The mutual information is denoted as the formation of conditional probability and contains the initial observation s0: I(Sf ;ω|S0) = − ∑ sf p(sf |s0) log p(sf |s0) + ∑ ω,sf pJ(sf |s0, ω)pC(ω|s0) log pJ(sf |s0, ω). The controllability distribution pC(ω|s0) maximizes the behavior diversity. 2The second term suggests that each skill should act as randomly as possible, aiming improving the exploration.

heading: 3 METHODOLOGY
text: In this section, we elaborate our proposed LTS method to discover both primitive and transitional skills without extrinsic reward. Define by S and A states and actions, respectively. Define by Ωn ∼ p(ω), 1 ≤ n ≤ N , N latent variables on which we condition the primitive skill π(at|st, ω). Given the starting and ending primitive skill that are conditioned on ωi and ωj , respectively, we define zi,j,k, 1 ≤ k ≤ K − 1, as K latent variables on which we condition the transitional skill π(at|st, zi,j,k). Figure 1 shows the block diagram of primitive and transitional skills as well as the correponding states. Our objective is to discover primitive skills π(at|st, ωi), π(at|st, ωj), and transitional skills π(at|st, zi,j,k). Specifically, we discover skills by maximizing the following mutual information
L(θ) , I(Sz; Ωi) + I(Sz; Ωj) = Eωi∼p(ω),sz∼π(at|st,zi,j,k) [ log p(ωi|s
z)− log p(ωi) ] + Eωj∼p(ω),sz∼π(at|st,zi,j,k) [ log p(ωj |s
z)− log p(ωj) ] , (3) where Sz represents one of states Sti,j,k at time stamp t, 1 ≤ k ≤ K−1, in Figure 1. It is worth being noted that when the starting primitive skill is same with the ending primitive skill, no transitional skill needs to be discovered. In such case, maximizing L(θ) in (3) is equivalent to maximizing I(Sz; Ωi), which is identical to learning only primitive skills Gregor et al. (2016); Eysenbach et al. (2018).
The challege is how to solve such problem associated with two primitive skills and acquire the intrinsic reward. Our solution is to use a compensation for individual mutual information so that two components become separable.
It is observed that, when the transitional skill π(at|st, zi,j,k) transits from the starting primitive skill π(at|st, ωi) to the ending primitive skill π(at|st, ωj), the mutual information I(Sz; Ωi) decreases progressively while the mutual information I(Sz; Ωj) increases progressively.
In order to compensate the mutual information, we define fdωi,zi,j,k as the divergence of the latent variables ωi and zi,j,k, which indicates the deviation of zi,j,k from ωi. Similarly, we define fdzi,j,k,ωj as the divergence of the latent variables zi,j,k and ωj , which indicates the deviation of ωj from zi,j,k. The divergence defined here can be measured in a couple of ways 3. Therefore, the objective function to be maximized becomes
L(θ) ' I(Sz; Ωi) + I(Sz; Ωj) + fdωi,zi,j,k + f d zi,j,k,ωj
= Eωi∼p(ω),sz∼π(at|st,zi,j,k) [ log p(ωi|s z)− log p(ωi) ] + fdωi,zi,j,k︸ ︷︷ ︸
L1(θ)
+Eωj∼p(ω),sz∼π(at|st,zi,j,k) [ log p(ωj |s z)− log p(ωj) ] + fdzi,j,k,ωj︸ ︷︷ ︸
L2(θ)
. (4)
The sufficient condition of maximizing the objecive function in (4) is to simultaneously maximize L1(θ) and L2(θ). It is further observed that, when the transitional skill π(at|st, zi,j,k) transits from
3For details, please see it in Section 4: Implementation.
π(at|st, ωi) to π(at|st, ωj), the divergence fdωi,zi,j,k increases progressively while the divergence fdzi,j,k,ωj decreases progressively. Thus, the first component L1(θ) that has a decreasing mutual information and an increasing divergence, is symmetric to the second component L2(θ) that has an increasing mutual information and a decreasing divergence. The sole difference lies in the transition from π(at|st, ωi) to π(at|st, ωj) or from π(at|st, ωj) to π(at|st, ωi). So maximizing L1(θ) is equivalent to maximizing L2(θ).
Consequently, maximizing the overall objective function L(θ) is equivalent to maximizing L1(θ) or L2(θ). In such case, we have
arg max L(θ) ≈ arg max L1(θ) = arg maxEωi∼p(ω),sz∼π(at|st,zi,j,k) [ log p(ωi|sz)− log p(ωi) + fdωi,zi,j,k ] .(5)
Because it is difficult to find p(ωi|sz), it is common to approximate p(ωi|sz) with a learned discriminator qφ(ωi|sz). According to Jensen’s Inequality, we know that replacing p(ωi|sz) with qφ(ωi|sz) gives us a variational lower bound G(θ, φ) of L(θ). So we have
arg max L(θ)⇐⇒ arg maxEωi∼p(ω),sz∼π(at|st,zi,j,k) [ log qφ(ωi|sz)− log p(ωi) + fdωi,zi,j,k ] ︸ ︷︷ ︸
,G(θ,φ)
.
(6)

heading: 4 IMPLEMENTATION

heading: 4.1 HINDSIGHT AND ONE-HOT ENCODING
text: In this section, we discuss about the implementation of learning transitional skills, where there are a couple of issues with the usage of the latent variable corresponding to transitional skills.
Problem 1: Along with the growth of the number of primitive skills N and transitional skills K − 1, we have a high training complexity. In our approach, we calculate the conditional probability qφ(ωi|sz) with a low efficiency because qφ(ωi|sz) can just keep the consistency of zi,j,k with ωi but ignore the relation with ωj .
To enhance the efficiency, we utilize the hindsight experience reply mechanism to allow samplebased learning from the sparse reward.
Problem 2: The categorical distribution of the latent variable ωi, 1 ≤ i ≤ N , suffers from a dilemma when the discriminator qφ(ωi|sz) judges the states. Our solution is to use one-hot encoding for the latent variable ωi, 1 ≤ i ≤ N , corresponding to primitive skills. In such case, the latent variable zi,j,k corresponding to transitional skills has a different expression and further analysis is given in Appendix B.
In the implementation, we change qφ(ωi|sz) in (6) to the following conditional probability distribution:
qφ(Ω̃|sz) = [qφ(ω0|sz), qφ(ω1|sz), ..., qφ(ωN−1|sz)]T. (7)
Correspondingly, the divergence of fdi,j,k is organized as the following f d i,k,
fdi,k = [f d i,1,k, f d i,2,k, ..., f d i,N,k] T, (8)
where k 6= i. Consequently, the objective is to maximize the following G(θ, φ),
G(θ, φ) = 1
N · Eωi∼p(ω),sz∼π(zi,k)(‖ log qφ(Ω̃|s z) + αfdi,k‖2), (9)
where zi,k ∼ pz(z|ωi) = {zi,k|zi,k = zi,j,k, 0 ≤ j ≤ N − 1, zi,j,k ∈ p(zi,j |ωi, ωj)}, α is a hyperparameter, and we ignore the term p(ωi) by taking a uniform distribution for ωi.
Algorithm 1 Learning Transitional Skills (LTS) 1: while NOT converged do 2: Sample ωi ∼ p(ω) 3: Sample a skill z ∼ pz(z|ωi) and an initial state s0 ∼ p0(s) 4: for t← 1 to steps per episode do 5: Sample an action at ∼ πθ(at|st, z); 6: Interact with the environment: st+1 ∼ p(st+1|st, at); 7: Compute Dt = 1N ‖qφ(Ω̃|st+1) + αf d i,k‖2 with the discriminator (φ);
8: Set the reward for current skill: rt = Dt. 9: By using SAC, update the policy (θ) to maximize the discounted returnR = ∑∞ t=0 γ
trt; 10: Update the discriminator (φ) to maximize Dt with SGD. 11: end for 12: end while

heading: 4.2 ALGORITHM
text: We summarize our LTS method in Algorithm 1. At each roll-out, given the latent variable of primitive skill ωi, we sample a transitional skill z from a fixed distribution pz(z|ωi). After the agent interacts with the environment at time step t, the discriminator finds the discriminability by
Dt = 1
N ‖qφ(Ω̃|szt+1) + αfdi,k‖2, (10)
where szt+1 denotes the states at time step t. As mentioned above, we encode primitive skills using one-hot encoding. We constrain ∑ j zi,j,k = 1, 0 ≤ zi,j,k ≤ 1, so we simplify the divergence as fdi,k = Array(zi,j,k), where Array(·) means convert one-hot encoding into an array. (See example in Appendix B.1 for more details (16, 17)).
In addition, we adopt soft actor-critic (SAC) algorithm to train our policy, adding the regularization Ei,j [ H[A|S,Zi,j ] to maximize the policy’s entropy over actions given states and skills.

heading: 5 RELATED WORK
text: Real-world tasks often require diverse behaviors. Wang et al. (2017) notes that building versatile embodied agents capable of performing a wide and diverse set of behaviors is one of the long-standing challenges of AI. And learning continuous control of diverse behaviors in locomotion (Merel et al. (2017); Heess et al. (2017); Peng et al. (2017)) and robotic manipulation (Ghosh et al. (2018); Gu et al. (2017)) is an active research area. In this scenario, although some complex tasks can be solved through extensive reward engineering, undesired behaviors often emerge because of the sparse nature of reward (Riedmiller et al. (2018)). Moreover, training complex skills from scratch is not computationally practical. These issues can be addressed by use of intrinsic motivation (Barto (2013); Chentanez et al. (2005); Singh et al. (2010)), which is a reward-free learning method. Historically, the intrinsic motivation comes from the tendency of organisms to play and explore their environment without any reward (Ryan & Deci (2000), White (1959)).
Another line of work that is conceptually close to our method copes with information theories that are used to drive the agent’s exploration. The information gain is a reward based on the reduction of uncertainty on environment’s dynamics (Little & Sommer (2013); Oudeyer & Kaplan (2007)), which can also be assimilated to learning progress (Frank et al. (2013); Oudeyer & Kaplan (2007)). This can push agents into unknown areas on the one hand, and prevent them from being attracted to random areas on the other.
Recent work has also applied information theory for skill discovery. VIC (Gregor et al. (2016)) is an optional discovery technique by optimizing a variational lower bound on the mutual information between the context and the final state in a trajectory, conditioned on the initial state. Furthermore, DIAYN (Eysenbach et al. (2018)) maximizes the mutual information between states and skill to achieve diversity and shows the interest as a pre-training for hierarchical reinforcement learning or as an initialization for learning a task. While discriminative embedding reward networks (DISCERN) (Warde-Farley et al. (2018) aim to simultaneously learn a goal-conditioned policy and a
goal achievement reward function by maximizing the mutual information between the goal state and the achieved state. Let us notice that the skill space here is discrete, with just one or multiple policies. However, we considered the relationship between different skills during the training process and finally formed a continuous skill space, likely because of inducing a novel latent variable for transitional skills.
In addition, it is important to point out that our skills are transitional with an intrinsically driven approach, which is very different from numerous previous works. While Sharma et al. (2019) discovers predictable behaviors to let the single skill more predictable, it need an external model-predictivecontrol (MPC) paradigm (Garcia et al. (1989)) to connect skills. Peng et al. (2019) learns reusable motor primitives that can be composed to produce a continuous spectrum of skills. To bridge the gap between skills, Lee et al. (2018) propose a transition policy to get a new smooth skill. In comparison, our method captures intrinsic transition, which is independent from external tasks, and could eliminate the extra fine-tuning process.

heading: 6 EXPERIMENTS
text: In our experiments, we aim to demonstrate the effectiveness of our approach for learning primitive and transitional skills. We evaluate LTS and compare it to prior works.

heading: 6.1 DIVERSITY AND TRANSITIONAL SKILLS
text: In this section, we provide visualizations and quantitative analysis for our LTS method. The tasks of CartPole-v0, MountainCar-v0, Pendulum-v0 and HalfCheetah-v3 are based on OpenAI gym 4.
To evaluate different skills in different environment, we extract features from the observation of the agent, e.g. in the MountainCar environment, where we use the variance of the car’s altitude as the feature. For more details on the features in other environments, please refer to Appendix C.
Figure 2 illustrates the discovered primitive skills in the MountainCar and Halfcheetah environment. As shown in Figure 2(a), all four skills move in an periodic manner. Corresponding to all 4 skills in Figure 2(a), the statistical values of features are shown in Figure 2(b) using Box-plot. It is observed that, these four skills have different movement patterns so that these skills are easy to be distinguished, so as the skills shown in Figure 2(c). Moreover, we consider a different number of primitive skills and different environments as in Appendix E, from which it is observed that all primitive skills have an evident difference in feature statistics and are easy to be distinguished.
Transition. Furthermore, we use the transitional skill z0,1,k to show the performance of transition from one primitive skill ω0 = [1, 0, 0, 0]) to another ω1 = [0, 1, 0, 0]), where the number of transitional skills is set as 9.
4http://gym.openai.com/
In Figure 3(a) and 3(b), it is observed that the primitive skill ω0 smoothly changes to ω1 via 9 transitional skills. More specifically, there exists a slight increment on the amplitude of features in the first three skills, which is followed by consecutive declines until the ending primitive skill ω1 is discovered. In Figure 3(c), we can see the clear transition process between two primitive skills. More experiments on skill transition are given in Appendix F. This demonstrates the effectiveness of our LTS method on discovering transitional skills and accomplishing the successful transition between two primitive skills.

heading: 6.2 COMPARISON WITH DIAYN
text: In this subsection, we compare our LTS method with the state-of-the-art DIAYN (Eysenbach et al. (2018)) in terms of learning diverse primitive skills and transitional skills. Experimental results show that LTS achieves an approximate performance on diversity of primitive skills, and a much better performance on skill transition.
Diversity: Different skills have different means or variances from their own trajectories. In this subsection, we use (1) variance of means and (2) mean of variances of the features as metrics to evaluate the diversity of learned primitive skills using LTS and DIAYN.
Figure 4(a) shows the variance of means while Figure 4(b) shows the mean of variances, where the black perpendicular line represents the range of values from a single experiment and we collect all trajectories of various primitive skills. In Figure 4(a), the height of this line depicts the range of variance of means by considering all trajectories of primitive skills. As shown, LTS obtains a lower variance of means in compare to DIAYN because the learned transitional skills from LTS affect the variance of learned primitive skills.
The lower variance of means does not degrade the diversity of learned primitive skills, which can be observed from Figure 4(b). Although there is a large difference between experiments with random seeds, LTS has generally a similar mean of variance as DIAYN, indicating that our method performs similar with the baseline in terms of learning diverse primitive skills.
Transition: We also conduct experiments to evaluate the transitional skills of LTS and DIAYN, where we use identical encoding scheme for both and the number of transitional skills is set as 9.
Figure 5 shows the all statistical characteristics of 11 skills that including two primitive skills, where the value denotes the mean of features like the middle bar in Figure 3(b). It is observed that LTS experiences a smooth transition state from one primitive skill to another while the transitional states in DIAYN suffer from one
sharp-rising phase and two steady phases indicating a rigid changes of transition states. The steady phase indicates that changing z will not change states very much, and the sharp change indicates that a little fluctuate of z will cause a huge change of states, which is not reasonable in practice. Because of the huge gap, controlling the skills learned by DIAYN changing from ω0 to ω1 will not push the agent standing in the state space of ω2. On the contrary, we control the transitional skills in LTS changing from ω0 to ω1 as in Figure 2(b), and the agent reaches the target states.
More studies including real statistical characteristics of different transitional skills are conducted and the results are reported in Appendix G.
6.3 GENERALIZATION
Our LTS method suffers from the high training complexity to learn transitional skills. A reasonable approach to tackle this problem is to use a fixed number of transitional skills. In this experiment, we set this number as 3, indicating that the nonzero element of zi,j,k is from the set {0.25, 0.50, 0.75} in the training phase. Figure 6 shows the transitional skills, where the training phase considers 3 transitional skills and adopting 3 and 50 transitional skills to evaluate the generalization of our LTS method. It is observed that the transitional skills in blue (traning and testing both with 3 transitional skills) suffers from severe declines, possibly leading
to the failure of the transition process in practice. Fortunately, the transitional skills in red (training with 3 transitional skills while testing with 50) go through a steady and smooth process from one
primitive skill to another. This experiment demonstrates that the strong generalization of LTS can improve the efficiency of the training. These means that even we use a small number of transitional skills in training, we could sample the transitional skills with a higher density and achieve the success of the skill transition, by which we could further improve the efficiency of training model.

heading: 6.4 EQUIP THE AGENT WITH MORE TRANSITIONAL SKILLS
text: Figure 7 shows much more skills and corresponding transitions between them, where each small square denotes a different skill, the number of primitive skills is 2 and the number of transitional skills is 98. Two primitive skills [1, 0, 0, 0] and [0, 1, 0, 0] locate at the lower left and upper right squares.
It is observed that our LTS method is able to accomplish a smooth transition between two arbitrary skills, whatever primitive skills or transitional skills. However, DIAYN suffers from a rigid transition between two skills, resulting in a possible failure in practical transition. Furthermore, these results provide a deep insight that LTS has the ability to learn a larger continuous skill space. The agent equipped with such numerous skills is expected to become much more powerful.

heading: 6.5 TRANSITION WITH HIERARCHICAL FRAMEWORK
text: In this subsection, we conduct experiments to answer the question: how do transitional skills yield benefit to the downstream tasks? To evaluate the effect of the transition policies, we take as benchmark hierarchical control for primitives learned by DIAYN. Moreover, we conduct hierarchical framework to weight (or choose) the action modeled by different primitive skills.
For primitive skills modeled by DIAYN, we consider the following approaches:1) DIAYN-random: Transition policy with random weights to affect actions; 2) DIAYN-smooth: Transition policy with smoothly changed weights (artificially designed) to affect actions; 3) DIAYN-master: Transition policy with optimal weights learned by meta-policy.
For transitional skills modeled by LTS, we consider the following approaches: 1) LTS-random: Randomly choose transitional skills between two primitive skills; 2) LTS-smooth: Choosing continously changed transitional skills zi,j,k ( 1 ≤ k ≤ K − 1); 3) LTS-master: Transfer policy with optimal weights learned by meta-policy.
Table 1 shows the performance. It is observed that we choose skills randomly or weighted actions modeled by primitive skills, the transition process is more likely to fail. Even with an extensively forecast through the hierarchical framework, DIAYN-master is possible to fail. We believe this is because the primitive skills have a large gap and therefore, one skill are unable to associate with another. Even with meta-policy, it is also hard to transit between primitive skills. In contrast, LTSsmooth has a better performance than DIAYN-master without any meta-policy because of abundant transitional skills. Moreover, LTS-master could select most appropriate transitional skills to fill in the gap between two arbitrary skills.

heading: 6.6 VISUALIZING TRANSITION TRAJECTORIES
text: Figure 8 shows the two-dimensional t-SNE embedding of the agent’s states generated by different skills. Different colors represent different skills (including primitive skills and transitional skills). The red lines represent the agents’ transition trajectories. In Figure 8(a) and 8(b), we can see a clear boundary between two kinds of skills, which is consistent with the boundary of Figure 7.
Starting from the initiatial state, the agent tries to transit to another state. But both of agents with DIAYN-random-policy and DIAYN-smooth-policy fail for transiton. The state space of different skills in Figure 8(c) seems to be more diverse, while it is a pseudomorphism. Because the space is a two-dimensional mapping, the difference between Figure 8(a) and 8(b) only discloses that different skills do exist a conclusive gap between two kinds of skills, but this uncovers that the learned transitional skills are also distinguishable. We can find in Figure 8(c) also exits many intersections which are the crucial transition states between two different skills. In the t-SNE space, distant dots with same color are also connective, so there exits many long line. And the intersections existing two endpoints must go through a transition between two skills. Particularly, the trajectory of the agent in Figure 8(c) shows the transition between different 11 skills.

heading: 7 CONCLUSION
text: In this paper, we introduce a novel LTS method to learn transitional skills without extrinsic reward by using two kinds of latent varibles to depict different skills. As a result, LTS can discover both
primitive skills and transitional skills. Also, LTS achieves a great success in the smooth transition from one primitive skill to another and exhibits its potential in learning a large continuous skill space. Extensive experiments demonstrate the effectiveness of our LTS in the discovery of diverse skills and the smooth transition between skills even for complex down-stream tasks.

heading: A METHODOLOGY
text: A.1 DERIVATION OF THE VARIATION BOUND ON MUTUAL INFORMATION
Here we derive the variational bound:
Eωi∼p(ω) [log p(ωi|st)] = Eωi∼p(ω) [log qφ(ωi|st)) + αKL(p(ωi|st)|qφ(ωi|st)] ≥ Eωi∼p(ω) [log qφ(ωi|st)]
(11)
B IMPLEMENTATION
B.1 HINDSIGHT AND ONE-HOT ENCODING
In our approach, we model the conditional probability given by the discriminator qφ(ωi|st) w.r.t. the divergence (or distance) between transitional skill zi,j,k and specific primitive skill ωi. Although (4) constrains two mutual information terms, but (??) is adopted to train our model, which constrains only one term. These efficiency is relatively low. For example, if we want to simultaneously constrain the probability qφ(ωi1 |st) and qφ(ωi2 |st) in terms of the same transition state st generated by the same skill z (z = zi1,j1,k1 = zi2,j2,k2) but two different primitive skills (ωi1 and ωi2) in the training stage, we must wait for the next time step to sample the same z and a different ω in the experience. The efficiency is relatively low because that qφ(ωi|st) just constrain the consistency of zi,j,k and ωi which ignores the consistency to other primitive skills. So we utilize the hindsight experience reply mechanism to allow sample-efficient learning form sparse rewards. We calculate a distribution of the conditional probability given by the discriminator instead of a single value. So we change qφ(ωi|st) to the conditional probability distribution:
qφ(Ω, st) = [qφ(ω0|st), qφ(ω1|st), ..., qφ(ωN−1|st)]T. (12) By doing this, we could simultaneously constrain the similarity probability distribution given by the discriminator with related to all primitive skills.
Moreover, we change the criterion of fdi,j,k into f d i,k:
fdi,k = [f d i,1,k, f d i,2,k, ..., f d i,N−1,k] T. (13)
On the other hand, for categorical encoded primitive skills, when ωi − ωj 6= ±1 (i 6= j), the discriminator qφ will face a dilemma: the intersection of transition states and primitive states are not empty, i.e. STi,j ∩ SP 6= , leading to a conflict between diversity and transition. Here we give an example: considering transferring two skills from ωi to ωj (ωi 6= ωj), the former categorical encoding will cause extra consumption: if ωi − ωj 6= ±1, primitive states with related to primitive skills {ω|ω ∈ [ωi, ωj ] or [ωj , ωi]} will occur in the transition states. For three primitive skills ωi1 ,
ωi2 and ωi3 , given the optimal discriminator q ∗ φ, there should be q ∗ φ(ωi3 |s′t) = 0 (s′t ∼ π(ωi1) and ωi1 6= ωi3 ) and q∗φ(ωi3 |s′′t ) = 0 (s′′t ∼ π(ωi2) and ωi2 6= ωi3 ). While if ωi1 < ωi3 < ωi2 , q∗φ(ω3|s′t) = log p(ωi1) − fd(ωi1) + c and q∗φ(ω3|s′′t ) = log p(ωi2) − fd(ωi2) + c 5, which is in contrast to the conditional probability of 1 (ωi3 is also a primitive skill). So, we encode ω ∼ p(ω) with one-hot way:
ω0 = [1, 0, 0, ..., 0];
ω1 = [0, 1, 0, ..., 0];
...
ωN−1 = [0, 0, 0, ..., 1].
(14)
And we denote primitive skills and transitional skills as a set Z+i,j :
Z+i,j = [ωi, zi,j,1, ..., zi,j,k, ..., zi,j,K−1, ωj ] T
=  0 ... 1 ... 0 ... 0 0 ... 1− 1K ... 1 K ... 0 .... 0 ... 1− kK ... k K ... 0
... 0 ... 0 ... 1 ... 0  (K+1)×N , (15)
where the value of i-th and j-th column keeps decreasing and increasing respectively. Other column always keep 0, which could constrain the incoherence between transition states and other primitive skills. Without causing any misunderstanding, following zi,j,k all comes from Z+i,j . For transition, we assure that the change of zi,j,k only happens on the corresponding dimension, which overcomes the conflict caused by categorical encoding. As show in Fig.9, all transition skills in Z3,1 and primitive skills ω3 are orthogonal. The transition only reflects on the plane defined by the corresponding primitive skills. Although these will induce a problem of the generation issue, but this could also learning continuous transitional skills as shown in Figure 6 and 7. In fact, there is more than one transitional path, which can be a directed line or any directed curve. As in Figure 7(b), we can find more than one transitional paths.
An example Z+i,j shows below:
Z+3,1 = [ω3, z3,1,1, z3,1,2, z3,1,3, ω1] T
=  0 0 0 1 0 0 0.25 0 0.75 0 0 0.5 0 0.5 0 0 0.75 0 0.25 0 0 1 0 0 0  5×4 , (16)
where N = 4, K − 1 = 3. A transition path could be ω3 → z3,1,1 → z3,1,2 → z3,1,3 → ω1. And an alternative way could be [0, 0, 0, 1, 0] → [0, 0.2, 0, 0.8, 0] → [0.1, 0.3, 0.2, 0.3, 0.1] → [0, 0.8, 0, 0.2, 0] → [0, 1, 0, 1, 0]. For the similairity fdi,k, examples are shown below (generated by Z+3,1 in Formula(16) ):
fs3,0 = [0, 0, 0, 1, 0] T;
fs3,1 = [0, 0.25, 0, 0.75, 0] T;
fs3,2 = [0, 0.5, 0, 0.5, 0] T;
fs3,3 = [0, 0.75, 0, 0.25, 0] T;
fs3,4 = [0, 1, 0, 0, 0] T,
(17)
where fsi,j = 1− fdi,j .
5c is a constant.

heading: C EXPERIMENTAL ENVIRONMENT
text: The experiments were carried out over four opened reinforcement learning environments (CartPole6, MountainCar7, Pendulum8 and HalfCheetah9.
C.1 CARTPOLE
In this environment, a pole is attached by an un-actuated joint to a cart, which moves along a frictionless track. The system is controlled by applying a force of +1 or -1 to the cart. The pendulum starts upright, and the goal is to prevent it from falling over by increaseing and reducing the cart’s velocity. The episode ends when the pole is more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.
C.2 MOUNTAINCAR
A car is on a one-dimensional track, positioned between two ”mountains”. The goal is to drive up the mountain on the right; however, the car’s engine is not strong enough to scale the mountain in a single pass. Therefore, the only way to succeed is to drive back and forth to build up momentum.
C.3 PENDULUM
The inverted pendulum swingup problem is a classic problem in the control literature. The problem of the pendulum starts in a random position, and the goal is to swing it up so it stays upright.
C.4 HALFCHEETAH
In the HalfCheetah environment, there is a two-legged robot, restricted to a vertical plane, meaning it can only run forward or backward. The agent has 17 dimensions of state space and 6 dimensions of action space.

heading: D HYPERPARAMETERS
text: For all RL algorithm in our experiments, we use the SAC (Haarnoja et al. (2018)) as implementation framework. The hyperparameters are summed up in the Table 2 and we use ADAM (Kingma & Ba (2014)) optimizer.
E VISUALIZING PRIMITIVE SKILLS
In order to better visualize the distinction between skills, we did various experiments and finally determined some optimal observations as feature vector for each skill (see Table 3). Please note that the feature values adopted are aimed at visualizing the skills, and what we put into the discriminator is the all state of the agent instead the single feature value. The following experiments show that it makes sense to calculate the statistical characteristics of skills’ characteristics to represent a skill. Three experiments’ performance was shown in Figure 10, Figure 11 and Figure 12.
F VISUALIZING TRANSITION PROCESS
For Cartpole, MountainCar, Pendulum and HalfCheetah, we get 4 primitive skills, and control variant z only takes 3 fixed values (0.25 0.5 0.75) during training. However, at the test phase of transition, z is added every 0.1. This ways reduce the training complexity as analyzed in Section 6.3. So we can obtain 9 transition skills between any two primitive skills( z = [0, 1]) (See Figure 13, Figure 14, and Figure 15).
6https://gym.openai.com/envs/CartPole-v0/ 7https://gym.openai.com/envs/MountainCar-v0/ 8https://gym.openai.com/envs/Pendulum-v0/ 9https://gym.openai.com/envs/HalfCheetah-v2/

heading: G TRANSITIONAL SKILLS COMPARISON
text: Studies including motion trail and statistical characteristics on MountainCar was reported in Figure 16. The value of the y-axis denotes the mean of feature values (which is also the statistical characteristic). Here we do not test the transition processes between skills and we just run different transitional skills independently. There should be many isolated dots instead a transition line. Skills with two distant features will means a larger probability of failure of transition. For DIAYN’s primitive skills, there exists a large gap which notes an harder transition between the two skills, comparing the smooth line of LTS’s skills. Please note that there exits no faster transition instead a large gap.

Review summaries:
The paper aims to learn transitional skills in addition to creating diverse primitive skills without a reward function by using an information theoretic objective, borrowing ideas from instrinsic motivation. The paper addresses the challenge of learning diverse primitives and also learning to transition between any pair of them, which is valuable for real-world complex problem solving with sparse rewards. The proposed LTS approach is compared againt a state-of-the-art DIAYN approach on several benchmark control tasks.