Learning transitional skills with intrinsic motivation
Authors: Qiangxing Tian
Jinxin Liu
Donglin Wang
Venue:  (2020)

Abstract: By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn useful skills without supervision. However, when considering to use multiple consecutive skills to complete a specific task, the transition from one to another cannot guarantee the success of the process due to the evident gap between skills. In this paper, we propose to learn transitional skills (LTS) in addition to creating diverse primitive skills without a reward function. By introducing an extra latent variable for transitional skills, our LTS method discovers both primitive and transitional skills by minimizing the difference of mutual information and the similarity of skills. By considering various simulated robotic tasks, our results demonstrate the effectiveness of LTS on learning both diverse primitive skills and transitional skills, and show its superiority in smooth transition of skills over the state-of-the-art baseline DIAYN.

Under review as a conference paper at ICLR 2020
LEARNING TRANSITIONAL SKILLS WITH INTRINSIC MOTIVATION
Anonymous authors Paper under double-bling review
ABSTRACT
By maximizing an information theoretic objective, a few recent methods empower the agent to explore the environment and learn skills without extrinsic reward. However, when considering use multiple consecutive skills to complete spe- cific task, the transition from one to another cannot guarantee the success of the process due the evident gap between skills. In this paper, we propose to learn transitional skills (LTS) in addition to creating diverse primitive skills. By intro- ducing an extra latent variable for transitional skills and a compensation term for intrinsic reward, our LTS method discovers both primitive and transitional skills by maximizing the mutual information with compensation. By considering var- ious robotic tasks, our results demonstrate the effectiivneess of LTS learning both diverse primitive skills and transitional skills, and show its superiority smooth transition of skills over the ffte-of-the-aa baseline DIAYN. In addition, we further show the significance of transitional skills for learning downstream tasks.
INTRODUCTION
Deep reinforcement learning (DRL) has shown its great effectiveness in learning various reward- driven skills in wide domains, such as performing robotic manipulation tasks (Levine et al. (2016).
playing video games (Mnih et al.] (2015)), playing adversarial board games (Silver et al. (2016)) and implementing robot navigation complex environments (Wang al. (2018)). Nevertheless, for the majority of real applications, there is no reward in a long term until the agent reaches goal state (Wu & Chen (2007)), especially in unseen environments. such cases, DRL has difficulty in carrying out the tasks.
By observing the human intelligence that can explore their surroundings and learn valuable skills without reward, couple of prior works have been recently proposed to generate skills without supervision by incorporating intrinsic motivation into DRL methods (Barto) Ry & Deci (2000)). Diverse skills be autonomously acquired without extrinsic reward by maximizing information theoretic objective using a maximum entropy policy (DIAYN (Eysenbach et al. (2018)); VIC (Gregor et al. (2016);; DAS (Sharma al. (2019)). Discovered skills may help the exploration in complex environments, and can also serve as primitive skills for hierarchical DRL. Particularly, high-level meta-policy could be adopted in the hierarchical framework choose corresponding low-level primitive skills to complete tasks in order.
Although discovered skills both distinguishable and diverse, it is still exceedingly difficult to integrate such skills for a complex task that requires smooth transitions between skills (Lee et al. (2018). ) Take the basketball as an example: learning the passing, catching and shooting skills in an isolated way cannot guarantee to score in the court due to the possible failure in the process of tran- sitions between skills. To address this problem, we propose further learn transitional skills (LTS), where discovered primitive skills, same as prior works (Eysenbach et al. (2018)), are distinguishable and as diverse as possible.
More concretely, our LTS method learns both primitive and transitional skills by optimizing in- formation theoretic objective, where extra transitional skills are generated to fill in the gap between diverse primitive skills. For such purpose, aside from using the latent variable dition primitive skills, an extra latent variable introduced on which we condition our transitional policy. Furthermore, compensation term has be considered in the objective function because the
which

---

Under review as a conference paper at ICLR 2020
distinct between primitive skills and transitional skills will lead to the decline of multual informa- tion between the latent variable corresponding to primitive skills and states generated by transitional skills. This compensation considers the divergence of the latent variable corresponding to primitive skills and that for transitional skills.
Different from learning primitive skills, our learning process considers two arbitrary primitive skills and multiple transitional skills between them, where both primitive and transitional skillls are un- known and be learned. By maximizing the multual information with compensation, both primitive skills and transitional skills are discovered, which can be used to effectively learn downstream tasks. On four simulated robotic tasks, experimental results show that our LTS can discover both prim- itive skills and transitional skills, successfully perform the transition between primitive skills that are distinguishable, and achieve a better peformance in comparison to the state-of-the-art baseline DIAYN.
The main contributions of our work can be summarized as follows. Our proposed LTS can learn both primitive and transitional skills without extrinsic reward, where the primitive skills are distinguish- able and diverse, and the transitional skills can accomplish smooth transitions between primitive skills. And extensive experiments are conducted, which demonstrates the effectiveness of our LTS method in solving downstream tasks, performing the transition between primitive skills as well as the weighted way to compose skills.
2
PRELIMINARIES
RL: In the standard RL setup, agent interacts with an environment over discrete time. At time step I, the agent observes the current state 8, and selects an action az according to policy x x(a (x). Then, the agent receives reward r, and comes the next state 8t+1- The objective of learning is to maximize the discounted return = Er=o y'r, of the policy æ, where y E [0, 1] is a discount factor.
Learn Skills with RL: Using the notation from information theory: we introduce two random variables S and A for states and actions, respectively. To discover diverse skills, a latent variable Wp ~ p(w) is defined on which we condition the policy I [(oe w..). Such policy is defined as skill (Eysenbach al. (2018)). Prior works verify that maximizing the mutual information between the states S and the skills wn will successfully generate lots of distinguishable, diverse and useful skills.
By primarily maximizing the mutual information between the final state Sy and the skills Wn given the initial state So,
I( SS
(1)
the variational intrinsic control (VIC) Gregor et al. (016)) shows the success of acquiring distin- guishable skills from the final state. Furthermore, in order to enhance the diversity of skills as much as possible, DIAYN primarily maximizes the mutual information between the state S at all time
stamps and the skills wn (Bysenbach al. (2018)).
I(S;,,,) + H[A[S, W.,
which indicates that different skills generate different trajectories that traverse different states, and such diverse skills can identified distinguishably.
Both VIC and DIAYN successfully discover primitive skills by maximizing the mutual information betweens state and skills. However, prior works cannot accomplish variety of robot tasks that require smooth transition between skills. To address such problem, propose the LTS scheme in this paper to learn both primitive and transitional skills by using an information theoretic objective function.
'The mutual information is denoted as the formation of conditional probability and contains the initial ob- servation (So) = E., p(s; 18)) log p(*)(=0) Em.ay p ( log (sy (so, The controllability distribution p ((]]s)) maximizes the behavior diversity. 2The second term suggests that each skill should act as randomly as possible, aiming improving the explo- ration.
2

---

Under review as a conference paper at ICLR 2020
Primitive States
Transitional States
Primitive States
S
St s..1
St k.j.k
ist
W/
24.1.1
21,j,k
2i.j,K-1
Wj
Zi.j
Figure 1: Block diagram of primitive and transitional skills.
METHODOLOGY
In this section, elaborate proposed LTS method to discover both primitive and transitional skills without extrinsic reward. Define by and A states and actions, respectively. Define by On ~ p(w), 1 I N, N latent variables which condition the primitive skill (aalss, w) Given the starting and ending primitive skill that are conditioned on W; and Wj, respectively, we define Zi.jkk, S < K 1, K latent variables which we condition the transitional skill π (arl Ri.,)).
Figure [ shows the block diagram of primitive and transitional skills well the correponding states. Our objective is to discover primitive skills [(agss,, Te,lss, wy). and transitional skills (agls;, A.).)). Specifically, we discover skills by maximizing the following mutual information
L(0) - 1(55;22) + +(();
Buixp(u)) (a
[logp(ws) (8) log p(w:)
+ Buy- (a
log p(w))ss) log p(w;))
where S represents one of states 5 at time stamp f, S S K K, Figure It worth being noted that when the starting primitive skill is same with the ending primitive skill, no transitional skill needs to be discovered. In such case, maximizing L(0) in (3) is equivalent to maximizing
I(S*; Q2). which is identical learning only primitive skills Gregor et al. (2016) Eysenbach et al. 2018).
The challege how to solve such problem associated with two primitive skills and acquire the intrinsic reward. Our solution is to use a compensation for individual mutual information so that two components become separable.
It is observed that, when the transitional skill T (aylst, Zi.j,k) )) transits from the starting primitive skill ı (ap (2, wi) to the ending primitive skill <a, (8,, w;), ), the mutual information I(S* Q.) decreases progressively while the mutual information I(S*; Q2) increases progressively.
In order to compensate the mutual information, we define fum.ei.. the divergence of the latent variables w; and Zij,ks which indicates the deviation of 2i.j.k from wi. Similarly, we define a as the divergence of the latent variables Zi.j and Wj, which indicates the deviation of w/ from Zij.k- The divergence defined here can be measured in couple of ways Therefore, the objective function to be maximized becomes
L(0) ~ I(5*;00) + 1(5);00) + fur.e fd s.
log pog)(()))=) log logp(ws) + fur.ea
(0)
Bus hoappppppp logg p(ws)s*)) log pows( S
L2
The sufficient condition of maximizing the objecive function in 4 is to simultaneously maximize (0) and (2). further observed that, when the transitional skill (aa zi.j.) transits from
For details, please in Section Implementation.

---

Under review as a conference paper at ICLR 2020
a(a))s,,u)) to ((y)))s,wy) the divergence f Jun.s... increases progressively while the divergence figg,ww, decreases progressively. Thus, the first component (0) that has decreasing mutual information and an increasing divergence, is symmetric to the second component L2(0) that has an increasing mutual information and a decreasing divergence. The sole difference lies the transition from πI (ag St, wi) to [ St. wiy) or from (T, w)) to π (4)) So maximizing L1 is equivalent to maximizing (f)).
Consequently, maximizing the overall objective function L(()) is equivalent to maximizing L (0) or (2) In such case, we have
arg max L(0) x arg max (0)
= arg max (wrrp log ((ws((s)) - log p(ws) + Suree (5)
Because it difficult to find p(w,(s"), common to approximate p(w,(s*) with a learned discrim- inator (w(s)s=)) According to Jensen's 's Inequality, know that replacing p(w,)s*) with go (w) gives us a variational lower bound G(0, 6) of L()). So we have
arg max L(0) - arg max
log (w<l)))))))))))) logr(ws) + sel
AG(0,))
(6)
4
IMPLEMENTATION
4.1 HINDSIGHT AND ONE-HOT ENCODING
In this section, we discuss about the implementation of learning transitional skills, where there are couple of issues with the usage of the latent variable corresponding to transitional skills.
Problem 1: Along with the growth of the number of primitive skills N and transitional skills K 1, we have high training complexity. In our approach, we calculate the conditional probability (s()) with a low efficiency because (s) can just keep the consistency of zi.jk with wi but ignore the relation with wy.
To enhance the efficiency, we utilize the hindsight experience reply mechanism allow sample- based learning from the sparse reward.
Problem 2: The categorical distribution the latent variable ws, 1 sii S N, suffers from a dilemma when the discriminator 4o (w,[s=)) judges the states.
Our solution is use one-hot encoding for the latent variable wi, 1 S S N, corresponding to primitive skills. In such case, the latent variable Zij.A corresponding to transitional skills has different expression and further analysis is given in Appendix B.
In the implementation, we change yo (ww in (6) to the following conditional probability distribu- tion:
qo ( = (4o (wo s²) (w]l[(l)))))) ",,,,,,, go(wN-1|s*))
Correspondingly, the divergence of fiij is organized as the following fúl
fe If fle. .... M...., f...
(8)
where ls # i. Consequently, the objective maximize the following G(O, ø),
G(0, の) = N (t
log (+) + a af
(9)
where 2g,0 ~ (=(w) = [2i,125, = 2.j.k. O < j < N 1, E p(ay/ww w)), a is hyperparameter, and ignore the term p(wi) by taking uniform distribution for wi-
4

---

Under review as a conference paper at ICLR 2020
Algorithm Learning Transitional Skills (LTS)
while NOT converged do
2: Sample wi p(w)
3: Sample a skill
(2((u)) and an initial state so ~ po(s) for & <- 1 to steps per episode do
4:
5:
Sample an action a, ~ To(aa [t, z);
6:
Interact with the environment: 81+-1 p(88++ az);
7:
Compute D. x 11 (92)s(1+1
aa [/ ll2 with the discriminator (ø);
8:
Set the reward for current skill: T₁ Dz.
By using SAC, update the policy (0) to maximize the discounted return R Ei= 7 10: Update the discriminator (Ø) to maximize Dr with SGD.
9:
11: end for
12:
end while
4.2 ALGORITHM
We summarize our LTS method in Algorithm [ itive skill Wi, we sample transitional skill from fixed distribution (=(2ww)) After the agent interacts with the environment at time step t. the discriminator finds the discriminability by
At each roll-out, given the latent variable of prim-
1 (000(00))+11 + aftxlla.
D.
(10)
where s741 denotes the states at time step t. As mentioned above, encode primitive skills using one-hot encoding. We constrain E, 2i.j. = 1, I 2ij,k S 1, so we simplify the divergence as
fi Amaa(Ss.,.)), where A-ray(-) means convert one-hot encoding into array. (See example
in Appendix B.1 for more details (16 17)).
In addition, we adopt soft actor-critic (SAC) algorithm to train our policy, adding the regularization Bi.i H[A[S, Zi.] to maximize the policy's 's entropy over actions given states and skills.
5 RELATED WORK
Real-world tasks often require diverse behaviors. Wang et al. (0177) notes that building versatile em- bodied agents capable of performing a wide and diverse set behaviors is one of the long-standing challenges of AI. And learning continuous control of diverse behaviors in locomotion (Merel et al. (0177; Heess et al. (2017); Peng et al. (2017)) and robotic manipulation (Ghosh et al. (2018); Gu et al. (017)) is an active research area. In this scenario, although some complex tasks can be solved through extensive reward engineering, undesired behaviors often emerge because of the sparse nature of reward (Riedmiller et al. (2018). Moreover, training complex skills from scratch is not computationally practical. These issues can be addressed by use of intrinsic motivation (Barto) 2013); Chentanez al. (2005); (Singh al.] (2010)) which is reward-free learning method. His- torically, the intrinsic motivation comes from the tendency of organisms to play and explore their environment without any reward (Ryan & Deci (200). White (1959)).
Another line of work that is conceptually close to our method copes with information theories that
are used drive the agent's exploration. The information gain reward based on the reduction
of uncertainty on environment's dynamics (Little & Sommer (2013): Oudeyer & Kaplan (2077) )
which can also be assimilated to learning progress (Frank al, (2013); Oudeyer & Kaplan (2077) )
This can push agents into unknown areas on the one hand, and prevent them from being attracted to random areas on the other.
Recent work has also applied information theory for skill discovery. VIC (Gregor et al. (016)) is an optional discovery technique by optimizing a variational lower bound on the mutual information between the context and the final state in a trajectory, conditioned on the initial state. Furthermore, DIAYN (Eysenbach et al. (2018)) maximizes the mutual information between states and skill to achieve diversity and shows the interest a pre-training for hierarchical reinforcement learning or as an initialization for learning a task. While discriminative embedding reward networks (DIS- CERN) (Warde-Farlyy et al. (2018) aim simultaneously learn goal-conditioned policy and
5

---

Under review as a conference paper at ICLR 2020
goal achievement reward function by maximizing the mutual information between the goal state and the achieved state. Let us notice that the skill space here is discrete, with just one or multiple poli- cies. However, we considered the relationship between different skills during the training process and finally formed continuous skill space, likely because of inducing a novel latent variable for transitional skills.
In addition, it is important to point out that our skills are transitional with an intrinsically driven ap- proach, which is very different from numerous previous works. While While|Sharma Sharma al. (2019) discovers predictable behaviors to let the single skill more predictable, it need an external model-predictive- control (MPC) paradigm (Gareia et al. (989)) to connect skills. Peng al. (2019) learns reusable motor primitives that can be composed to produce continuous spectrum of skills. To bridge the gap between skills, Lee al. (2018) propose a transition policy to get a new smooth skill. compar- ison, our method captures intrinsic transition, which is independent from external tasks, and could eliminate the extra fine-tuning process.
EXPERIMENTS
In our experiments, aim demonstrate the effectiveness of our approach for learning primitive and transitional skills. We evaluate L.TS and compare it to prior works.
6.1 DIVERSITY AND TRANSITIONAL SKILLS
-02
-033
-04
-07
-00
(a) Trajectories of skills.
(b) Box-plots of 4 skills.
(c) Frames 3 skills.
Figure 2: Diverse primitive skills (Wo. W1, w2, wa) in MountainCar (a, b) and HalfCheetah (c). (a) shows the changes of the feature values of diverse skills. (b) shows the corresponding statistical characteristics of 4 skills in (a), where a single box-plot denotes skill. (c) shows 3 diverse skills of the half-cheetah.
In this section, we provide visualizations and quantitative analysis for our LTS method. The tasks of CartPole-vO, MountainCar-vOß, Pendulum-v0 and HalfCheetah-v3 are based on OpenAl gym [
To evaluate different skills in different environment, we extract features from the observation of the agent, e.g. in the MountainCar environment, where we use the variance of the car's altitude as the feature. For more details on the features in other environments, please refer Appendix
Figure 2 illustrates the discovered primitive skills in the MountainCar and Halfcheetah environ- ment. As shown in Figure [(a) all four skills move in an periodic manner. Corresponding all skills in Figure 2(a] the statistical values of features are shown in Figure 2 using Box-plot. It observed that, these four skills have different movement patterns so that these skills are easy to be distinguished, so the skills shown in Figure 2(c) Moreover, consider different number of primitive skills and different environments as in Appendix E, from which it is observed that all primitive skills have evident difference in feature statistics and easy to be distinguished.
Transition. Furthermore, we use the transitional skill 20,1. to show the performance of transi- tion from one primitive skill wo [1, O, O, 0)) another wh [0, 1, 0, 0)), where the number of transitional skills is set as 9.
hhtt://gym.openaa.com

---

Under review as a conference paper at ICLR 2020
-44
-38
Transhoon
Trajectory of transition.
(h) Box-plots of transition.
(c) Illustration 4 transitional skills.
Figure 3: Transition in Mountain-Car (a.b) and HalfCheetah (c). (a) shows the transition of the trajectory, where transitional skills are uniformly distributed along the horizontal axis from Step 5500. (b) shows the transition of feature statistics, where the features of two primitive skills and 9 transitional skills are included. (c) shows the sampled frames of the transition trajectory, where K 1
In Figure ((a) and ((b) it is observed that the primitive skill Wo smoothly changes via 9 transitional skills. More specifically, there exists a slight increment on the amplitude features in the first three skills, which is followed by consecutive declines until the ending primitive skill discovered. In Figure 3(c) we can see the clear transition process between two primitive skills. More experiments skill transition given in Appendix F. This demonstrates the effectiveness of our LTS method on discovering transitional skills and accomplishing the successful transition between primitive skills.
6.2 COMPARISON WITH DIAYN
In this subsection, compare LTS method with the state-of-the-art DIAYN (Eysenbach et al. (2018)) terms of learning diverse primitive skills and transitional skills. Experimental results show that LTS achieves approximate performance diversity of primitive skills, and much better performance on skill transition.
Diversity: Different skills have different means variances from their own trajectories. In this subsection, we use (1) variance of means and (2) mean of variances of the features as metrics to evaluate the diversity of learned primitive skills using LTS and DIAYN.
Figure shows the variance means while Figure 4 shows the mean of variances, where the black perpendicular line represents the range of values from a single experiment and we collect all trajectories of various primitive skills. In Figure 4(a) the height of this line depicts the range of variance means by considering all trajectories of primitive skills. As shown, LTS obtains lower variance of means in compare to DIAYN because the learned transitional skills from LTS affect the variance learned primitive skills.
The lower variance of means does not degrade the diversity of learned primitive skills, which can be observed from Figure 4bb) Although there is a large difference between experiments with random seeds, LTS has generally similar mean of variance as DIAYN, indicating that our method performs similar with the baseline in terms learning diverse primitive skills.

---

Under review as a conference paper at ICLR 2020
Model DIAIN LTS
Model
0.006
6.0175
DIAYN
LTS
0.905
0,0150
of mimatt
0.004
0.003
0.0075
0.0050
6,0025
Seeds
Seeds
(a) Variance of means.
(b) Mcan Variances.
Figure 4: Comparison of diversity of primitive skills between LTS and DIAYN.
Transition: We also conduct experiments to evaluate the transitional skills of LTS and DI- AYN, where we use identical encoding scheme for both and the number of transitional skills is set as
-0.35
-040
ues
DAYS
-035
showw the all statistical characteristics of 11 skills that including two primitive skills, where the value denotes the mean of features like the middle bar in Figure 3(b) It ob- served that LTS experiences smooth transition state from one primitive skill to another while the transitional states in DIAYN suffer from one sharp-rising phase and two steady phases indicating a rigid changes of transition states. The steady phase indicates that changing z will not change states very much, and the sharp change indicates that a little fluctuate of will cause huge change of states, which is not reasonable in practice. Because the huge gap, controlling the skills learned by DIAYN changing from wi to Wi will not push the agent standing in the state space of W2. On the contrary, we control the transitional skills in LTS changing from w/ to wi as in Figure 2(b) and the agent reaches the target states.
-0.00
-065
-0.00
10
Trans sidls
Figure 5: Comparison of transitional skills.
More studies including real statistical characteristics of different transitional skills are conducted and the results are reported in Appendix G.
6.3 GENERALIZATION
Our LTS method suffers from the high training complexity to learn transitional skills. A rea- sonable approach to tackle this problem is to use a fixed number of transitional skills. In this experiment, set this number as 3, indicating that the nonzero element of ji.j.k is from the set (0.25, 0.50, 0.75) in the training phase.
Number of transition
50
A3.
Fasuure
6.2
Figure 6 shows the transitional skills, where the training phase considers transitional skills and adopting 3 and 50 transitional skills to evaluate the generalization of our LTS method. It is ob- served that the transitional skills in blue (tran- ing and testing both with 3 transitional skills) suffers from severe declines, possibly leading to the failure of the transition process in practice. Fortunately, the transitional skills in red (training with 3 transitional skills while testing with 50) go through steady and smooth process from one
D.1
Figure 6: Transitional skills with related to 2 dif- ferent number of them.
8

---

Under review as a conference paper at ICLR 2020
Table 1: Success rate(%) for practical tasks, where method is compared with the baselines with or without hierarchical framework. Our transitional skills outperforms the baselines in all cases, demonstrating the effectiveness of the learning transitional skills for down-stream tasks. Here we adopt spectral clustering to cluster the trajectories. When the label of ending segment of the trajectory keeps the same label with that generated the target skills solely, the transition is regarded as success; otherwise a failure transition. The target skills regard as the goal of our tasks, which are generated by sample primitive skills or weighting the actions of different primitives.
DIAYN
LTS
Policy
random
smooth
master
random
smooth
master
MountainCar
0
0
20.8
0.2
50.2
100
Pendulum
0
14.2
100
100
100
CartPole
1.6
10
36.8
0
25.8
48
HalfCheetah
29.2
44.7
43.0
78.6
primitive skill to another. This experiment demonstrates that the strong generalization of LTS can improve the efficiency the training. These means that even we use a small number of transi- tional skills in training, could sample the transitional skills with higher density and achieve the success of the skill transition, by which we could further improve the efficiency of training model.
-0.3
-0.40
0.30
-056
0.18
-0.54
0.12
10
(a) Skill space of DIAYN.
(b) Skill space of LTS
Figure 7: Skill space of DIAYN and LTS.
6.4 EQUIP THE AGENT WITH MORE TRANSITIONAL SKILLS
Figure shows much more skills and corresponding transitions between them, where each small square denotes different skill, the number of primitive skills is 2 and the number of transitional skills 98. Two primitive skills [1, O, 0) and [0, 1, 0) locate the lower left and upper right squares.
is observed that our LTS method able to accomplish a smooth transition between two arbitrary skills, whatever primitive skills or transitional skills. However, DIAYN suffers from rigid transition between two skills, resulting
possible failure in practical transition. Furthermore, these results provide a deep insight that LTS has the ability to learn a larger continuous skill space. The agent equipped with such numerous skills expected to become much more powerful.
6.5 TRANSITION WITH HIERARCHICAL FRAMEWORK
In this subsection, we conduct experiments to answer the question: how do transitional skills yield benefit to the downstream tasks? To evaluate the effect of the transition policies, take bench- mark hierarchical control for primitives learned by DIAYN. Moreover, we conduct hierarchical framework weight (or choose) the action modeled by different primitive skills.

---

Under review as a conference paper at ICLR 2020
(a) DIAYN-random
(b) N-Smoot
(c) LTS-
Figure 8: Visualization of transition trajectories of (a) DIAYN-random, (b) DIAYN-smooth and (c) LTS-smooth. Top row contains the states generated by different skills noted by the dots and the red lines represent the agent's 's sampled trajectories, both of which are mapped to two-dimensional space by t-SNE. Bottom contains the frames of transition trajectories.
For primitive skills modeled by DIAYN, we consider the following approaches:1) DIAYN-random: Transition policy with random weights to affect actions; 2) DIAYN-smooth:: Transition policy with smoothly changed weights (artificially designed) to affect actions; 3) DIAYN-master: Transition policy with optimal weights learned by meta-policy.
For transitional skills modeled by LTS, consider the following approaches: 1) LTS-random: Ran- domly choose transitional skills between two primitive skills; 2) LTS-smooth: Choosing continously changed transitional skills Zi.j. weights learned by meta-policy.
x K 1); LTS-master:: Transfer policy with optimal
Table shows the performance. It is observed that choose skills randomly weighted actions modeled by primitive skills, the transition process is more likely to fail. Even with an extensively forecast through the hierarchical framework. DIAYN-master is possible to fail. We believe this is because the primitive skills have a large gap and therefore, one skill are unable to associate with another. Even with meta-policy, it also hard transit between primitive skills. In contrast, L7S- smooth has better performance than DIAYN-master without any meta-policy because of abundant transitional skills. Moreover, LTS-master could select most appropriate transitional skills to fill in the gap between two arbitrary skills.
6.6 VISUALIZING TRANSITION TRAJECTORIES
Figure shows the two-dimensional t-SNE embedding of the agent's states generated by different skills. Different colors represent different skills (including primitive skills and transitional skills). The red lines represent the agents' transition trajectories. In Figure and [8(b] can clear boundary between two kinds of skills, which is consistent with the boundary of Figure 7
Starting from the initiatial state, the agent tries to transit another state. But both of agents with DIAYN-random-polie and DIAYN-smoothhpolicg fail for transiton. The state space of different skills in Figure seems be more diverse, while it is pseudomorphism. Because the space is two-dimensional mapping, the difference between Figure ((a) and [ discloses that different skills exist conclusive gap between two kinds of skills, but this uncovers that the learned transi- tional skills are also distinguishable. We can find in Figure 8 also exits many intersections which are the crucial transition states between two different skills. the t-SNE space, distant dots with same color are also connective, there exits many long line. And the intersections existing two endpoints must go through a transition between two skills. Particularly, the trajectory of the agent in Figure ((c) shows the transition between different 11 skills.
CONCLUSION
In this paper, we introduce a novel LTS method to learn transitional skills without extrinsic reward by using two kinds of latent varibles depict different skills. As result, LTS discover both
10

---

Under review as a conference paper at ICLR 2020
primitive skills and transitional skills. Also, LTS achieves great success in the smooth transition from one primitive skill to another and exhibits its potential in learning a large continuous skill space. Extensive experiments demonstrate the effectiveness of our LTS in the discovery of diverse skills and the smooth transition between skills even for complex down-stream tasks.
REFERENCES
Andrew G Barto. Intrinsic motivation and reinforcement learning. In Intrinsically motivated learn- ing in natural and artificial systems, pp. 17-47. Springer, 2013.
Nuttapong Chentanez, Andrew G Barto, and Satinder P Singh. Intrinsically motivated reinforcement learning. In Advances in neural information processing systems, pp. 1281-1288, 2005.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. arXiv preprint arXiv: 1XXx::800.00070, 2018.
Mikhail Frank, Jürgen Leitner, Marijn Stollenga, Alexander Förster, and Jürgen Schmidhuber. Cu- riosity driven reinforcement learning for motion planning humanoids. Frontiers in Neuro- robotics, 7:25, 2013.
Carlos E Garcia, David M Prett, and Manfred Morari. Model predictive control: theory and prac- tice survey. Automatica, 25(3):335-22 -48, 1989.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and- conquer reinforcement learning. 2018.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:1611.075077 2016.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In IEEE International Conference on Robotics Automation, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. 2018.
Nicolas Heess, T. B. Dhruva, Srinivasan Sriram, Jay Lemmon, and David Silver. Emergence of locomotion behaviours in rich environments. 2017.
Diederík Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Computer Science, 2014.
Youngwoon Lee, Shao-Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph Lim. Com- posing complex skills by learning transition policies. 2018.
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. The Journal of Machine Learning Research, 17(1):1334-1333, 2016.
D. Y. Little and F. T. Sommer. Learning and exploration in action-perception loops. Frontiers in Neural Circuits, 7(7):37, 2013.
Josh Merel, Yuval Tassa, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nicolas Heess. Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02200 0200000 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 518(7540)5529, 2015.
Pierre Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computa- tional approaches. Frontiers in Neurorobotics, 1(6):6, 2007.

---

Under review as a conference paper at ICLR 2020
Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. ACM Transactions on Graphics (TOG), 36(4):41, 2017.
Xue Bin Peng, Michael Chang, Grace Zhang, Pieter Abbeel, and Sergey Levine. Mep: Learn- ing composable hierarchical control with multiplicative compositional policies. arXiv preprint arXiv:1905.09800 2019.
Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing- solving sparse reward tasks from scratch. arXiv preprint arXiv: 10567, 2018.
Richard M Ryan and Edward L Deci. Intrinsic and extrinsic motivations: Classic definitions and new directions. Contemporary educational psychology, 25(1):54 67, 2000.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint 0ef1::1907.006555 2019.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2(2):70-82, 2010.
Xin Wang, Qiuyuan Huang. Asli Çelikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross- matching and self-supervised im- itation learning for vision-language navigation. CoRR, abs/1811.10099 2018. URL. http:
//axxiv.org/abs//// 11t1//111110012222211111111
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, pp. 5320-5329, 2017.
David Warde-Farley, Tom Van de Wiele, Tejas Kulkarni, Catalin Ionescu, Steven Hansen, and Volodymyr Mnih. Unsupervised control through non- non-parametric discriminative rewards. arXiv preprint 11X1111811.11359, 2018
Robert White. Motivation reconsidered: The concept of competence. Psychological review, 66 (5):297, 1959.
Feng Wu and Xiaoping Chen. Solving large-scale and sparse-reward dec-pomdps with correlation- mdps. In Robot Soccer World Cup, pp. 208-219. Springer, 2007.
12

---

Under review as a conference paper at ICLR 2020
Appendices
A METHODOLOGY
A.1 DERIVATION OF THE VARIATION BOUND ON MUTUAL INFORMATION
Here we derive the variational bound:
Busmp(w) (lggp(y)ss) = Ew:-p(u [log to (wilss)) + (p (11)
2 Buw))(()) [log to (w)
B IMPLEMENTATION
B.1 HINDSIGHT AND ONE-HOT ENCODING
In our approach, we model the conditional probability given by the discriminator qa (wr(ss) w.r.t. the divergence (or distance) between transitional skill 2i.j.: and specific primitive skill w; Although { constrains two mutual information terms, but (??) is adopted to train our model, which constrains only one term. These efficiency is relatively low. For example, if we want to simultaneously con- strain the probability go(wi, (&x) and qo (a (sx) in terms of the same transition state generated two different primitive skills (wi and w;2) in the training stage, we must wait for the next time step sample the same and different in the experience. The efficiency is relatively low because that 4o (s) just constrain the consistency of 2ijj. and w; which ignores the consistency other primitive skills. So utilize the hindsight experience reply mechanism to allow sample-efficient learning form sparse rewards. We calculate distribution of the conditional probability given by the discriminator instead of a single value. So we change ya (wi(()) to the conditional probability distribution:
by the same skill z (= 2,1,j1,11 = 222,2,422)
(9, St) = (wo (sts, (o(w]|s; [st),
(12)
By doing this, we could simultaneously constrain the similarity probability distribution given by the discriminator with related to all primitive skills.
W3
Zu,1
Zz.
oo0
80%
W1
W2
080
Z1.3
Figure 9: The space of primitive skills and transitional skills, where we =, K =
Moreover, change the criterion of fkjj into fi
f4 = 1 fd fi,, ft -.
(13)
On the other hand, for categorical encoded primitive skills, when wi wy * #1 (i * j). the discriminator js will face a dilemma: the intersection of transition states and primitive states are not empty, i.e. ST, n SP * @, leading to a conflict between diversity and transition. Here we give example: considering transferring two skills from W; to Wj (w; # wy). the former categorical encoding will cause extra consumption: if w; W/ # +1, primitive states with related to primitive skills (w|w E wyy] or (wj wi]} will occur in the transition states. For three primitive skills wiy
13

---

Under review as a conference paper at ICLR 2020
Wig and Wast given the optimal discriminator 9z. there should be o (Wig (s{) 0 (s} T(wh) and Wi, # wig) and dh ( (8)) 0 (s" ((i;)) and Wig # wig). While if win < Wig wig. 9 (Wwa)ss ({) = log p(w,, (w;) + c and g (wxls(()) = log p(a,,) (wrr)) + c which is in contrast to the conditional probability of (wig is also primitive skill). So, we encode w ~ p(w) with one-hot way:
wi [1, 0........... 0]:
W1 = 1,,1,0..... 0]:
(14)
WN-1 10,0,0.... 1].
And we denote primitive skills and transitional skills as a set Z
Wi Ft.j.1, -.y Fi.j,ks Jij K w,]
0
0
0
1
1
...
(15)
0
k
K
K
0
0_ (K+1)xN
where the value of i-th and j-th column keeps decreasing and increasing respectively. Other column always keep O, which could constrain the incoherence between transition states and other primitive skills. Without causing any misunderstanding, following 2 all comes from Z For transition, we assure that the change of 2i.j.k only happens on the corresponding dimension, which overcomes the conflict caused by categorical encoding. As show in Fig all transition skills in Z3,1 and prim- itive skills wiy are orthogonal. The transition only reflects on the plane defined by the corresponding primitive skills. Although these will induce problem of the generation issue, but this could also learning continuous transitional skills as shown in Figure 6 and In fact, there is more than one transitional path, which can directed line or any directed curve. As in Figure 7(b) we can find more than transitional paths.
An example " shows below:
(w3, 23,1, 23,1,2 23,1,3, (31,3,401 W
0
0
0.25
0
0.75
0
(16)
0.5
0.5
0
0
0.75
0
0.25
0
5x4
where N = 4, K 1 = 3. A transition path could be W3 + 23.1.1 - 2 - 2. w|.
And an alternative way could be [o,
+ 00, 0.8, 0J + [0.1, 02,
[0, 0,0.8, 0, 0.2, 0] + [0, 1, 0, 1, 0j. For the similairity f examples are shown below (generated by zt, in Formula 16
fā.0 00,0,0,
fā,1 = [0, 0.,0.25,00 0, 0.75, 0):;
fā = [0, 0.5, 0...0000..00f
(17)
f5.3 [0, 0..5..5 0. 0.025, 01:
fi = [0, 0,
where fūz = 1 fig.
ii constant.
14

---

Under review as a conference paper at ICLR 2020
C EXPERIMENTAL ENVIRONMENT
The experiments were carried out over four opened reinforcement learning environments (CartPold MountainCafe Pendulum and HalfCheetal
C.I CARTPOLE
In this environment, a pole is attached by an un-actuated joint to a cart, which moves along a fric- tionless track. The system is controlled by applying force of +1 or the cart. The pendulum starts upright, and the goal is to prevent it from falling over by increaseing and reducing the cart's velocity. The episode ends when the pole more than 15 degrees from vertical, or the cart moves more than 2.4 units from the center.
C.2 MOUNTAINCAR
car on one-dimensional track, positioned between two "mountains". The goal is drive up the mountain on the right; however, the car's ' engine is not strong enough to scale the mountain in single pass. Therefore, the only way to succeed is drive back and forth build up momentum.
C.3 PENDULUM
The inverted pendulum swingup problem is a classic problem in the control literature. The problem of the pendulum starts in a random position, and the goal is to swing up stays upright.
C.4 HALFCHBETAH
In the HalfCheetah environment, there is two-legged robot, restricted to vertical plane, meaning can only run forward or backward. The agent has 17 dimensions of state space and 6 dimensions of action space.
D HYPERPARAMETERS
For all RI. algorithm in our experiments, we use the SAC (Haarnoja et al. (2018) as implementation framework. The hyperparameters are summed up in the Table and we use ADAM (Kingma & Ba 20144) optimizer.
E VISUALIZING PRIMITIVE SKILLS
In order better visualize the distinction between skills, we did various experiments and finally determined some optimal observations as feature vector for each skill (see Table Please note that the feature values adopted are aimed at visualizing the skills, and what we put into the discriminator is the all state of the agent instead the single feature value. The following experiments show that it makes sense to calculate the statistical characteristics of skills' characteristics to represent a skill.
Three experiments' performance was shown in Figure [0] Figure II and Figure [2
F VISUALIZING TRANSITION PROCESS
For Cartpole, MountainCar, Pendulum and HalfCheetah, we get primitive skills, and control vari- ant z only takes fixed values (0.25 0.5 0.75) during training. However, at the test phase of transi- tion, z is added every 0.1. This ways reduce the training complexity as analyzed in Section 6.3. So we can obtain transition skills between any two primitive skills( = [0, 1)) (See Figure 13 Figure
14
and Figure 15).
"https://gym..o www.://wpppciiiicmmmmm http:///wwooppaa.cco/////// http://////Caaaaaole/000/
15

---

Under review as a conference paper at ICLR 2020
Table 2: Parameter setting Description
Parameters
Value
32 for CartPole, MountainCar and Pendulum;
H
hidden state size
64 for HalfCheetah
layer
layer count eposide size
epoch
2 12(cpus)
lf
value network learning rate discriminator network learning rate policy network learning rate the maximal length of episode
le-5
de
5e-4
pi Ir
3e-4
max episodes
250
train_de_iin jiterv
epoch number of training discriminator network epoch number of training actor network
train_v v_iterv
epoch number of training critic network iteration number of updating discriminator network iteration number of updating actor network
train rrii.dee.tm Liters
train ac iters
50
train v itters
iteration number of updating critic network
Table 3: Selection of skill features.
RL enviroment
Observations
Selected skill feature
CartPole
0: Cart Position;
2: Pole Angle
1: Cart Velocity
2: Pole Angle
3: Pole Velocity at Tip
MountainCar
0: Position
O: Position
1: Velocity
Pendulum
0: cos( (nglee)
1: sin(Angle)
1: sin(Angle)
2: speed
HalfCheetah
17-dim values (including the positions the frames
and velocities)
16

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(d)
ff)
(h)
(k)
Figure 10: Cartpole primitive skills. (a) ( stand for 10 random trials with primitive skills for each, and different skills are distinguished by Boxplot. (k) shows the skills in time domain for one trial.
G TRANSITIONAL SKILLS COMPARISON
Studies including motion trail and statistical characteristics on MountainCar was reported in Fig- ure [6] The value of the y-axis denotes the mean of feature values (which also the statistical characteristic). Here we do not test the transition processes between skills and we just run differ- ent transitional skills independently. There should be many isolated dots instead a transition line. Skills with two distant features will means larger probability of failure of transition. For DIAYN's primitive skills, there exists a large gap which notes an harder transition between the two skills, comparing the smooth line of LTS's skills. Please note that there exits faster transition instead large gap.
17

---

Under review as a conference paper at ICLR 2020
(b)
(d)
(f)
(k)
Figure 11: Mountain Car primitive skills. (a) - (i) stand for 10 random trials with 4 primitive skills for each, and different skills are distinguished by Boxplot. (k) shows the skills in time domain for one trial.
(a)
(b)
(d)
2
()
(k)
Figure 12: Pendulum primitive skills. (a) (i) stand for 10 random trials with 4 primitive skills for each, and different skills are distinguished by Boxplot. (k) shows the skills in time domain for one trial.
18

---

Under review as a conference paper at ICLR 2020
(a)
(b)
(c)
(d)
(f)
(h)
Figure 13: Mountain Car transition process. (a) (e), (b) (f), (c) (g), and (d) (g) from different
trials respectively. Each transitional skill holds 500 steps and then transfers the final state to the next skill.
(a)
(b)
(c)
(d)
(
(h)
Figure 14: Pendulum transition process. (a) (c), (b) (f), (c) (g), and (d) (g) from different
trials respectively. Each transitional skill holds 500 steps and then transfers the final state to the next skill.
19

---

Under review as a conference paper at ICLR 2020
-00
(b)
(c)
(f)
(h)
Figure 15: Cartpole transition process. (a) (c), (b) (f), (c) (g), and (d) (g) are from different 4 trials respectively, Each transitional skill holds 500 steps and then transfers the final state the next skill.
(a)
(b)
(c)
(d)
(e)
Figure 16: Transitional skills comparison MountainCar. The subgraphs (c). repre- sent the mean of features in terms of the transitional skills, and subgraphs (c). represent the variance of features.
20


Reviews:
[{"review_id": "SJeB8gyy5r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "General description\nThe paper tackles the problem of transition between different skills in hierarchical reinforcement learning. \nIn particular, they follow the work of VIC and DIAYN and define an information-theoretic objective function that maximizes the mutual information between the future states and options, given the initial state, while minimizing the distance between the options and the so-called transitional states. The algorithm, LTS, is compared to DIAYN on three environments, namely CartPole, MontainCar, and Pendulum.\n\nGeneral remarks,\nThe problem tackled and the proposed idea are interesting; however, I am not fully convinced by the derivation and the experiments of the paper. The writing is in general not particularly clear and the notations are hard to follow, the symbols are often bloated with superscripts that are not clearly defined, and mixing capital and small letters for random variables and their realization.\n\nOn the derivation, Eq 3. How is it possible to replace p(s^p | p^t) with p(omega|s_t)? I understand the connection between the two but what guarantees the mutual information is still maximized? (the whole derivation depends on that)\n\nThe experiments and the plots are interesting, showing a smoother transition between skills than DIAYN, however, it is still not clear how that can help solve the task at hand. Could the author elaborate on that please?\n\nSome details:\n- Page 2 \\in should be \\sim. In general, the notation does not clearly distinguish a random variable (in capital) from a realization (in small letters). For instance, page 3, big \\Omega (the random variable I presume) is written with a subscript i to indicate the ith skill. \n- Footnote 1 page 2, a log is missing in the MI definition. \n- Page 3, what does t refer to in S^t_{i ,j, 1}?\n- Page 3, muture -> mutual. \n- the paper mentions the experiments are conducted on MuJoCo but the appendix mentions the classical OpenAI Gym experiments. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyx7VMW19B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to learn transitional skills in addition to creating diverse primitive skills without a reward function by using an information theoretic objective, borrowing ideas from instrinsic motivation. The paper addresses the challenge of learning diverse primitives and also learning to transition between any pair of them, which is valuable for real-world complex problem solving with sparse rewards. The proposed LTS approach is compared againt a state-of-the-art DIAYN approach on several benchmark control tasks.\n\nI have several comments/concerns regarding the experimental results.\n\n1. How would the feature engineering used (Appendix E) affect the performance of tasks solved using the primitive skils learned using LTS and how `would policies learned with this framework perform compared to standard RL policies learned individually with external rewards? This is to establish whether the learned skills and transitions between them are useful for any down-stream tasks. In the DIAYN paper, experiments on down-stream tasks were provided.\n\n2. What are these learned primitive tasks for each benchmark task? A visualization (or some interpretation of the meaning of the skills learned) would prove useful in understanding the primitive skills learned. Again, the DIAYN paper visualizes the primitive skills learned in their tasks.\n\n3. I don't understand how the results in 6.3 demonstrate the generalization ability of the method. The authors clearly state that using 3 transtional skills fails when evaluated on 50 transitional skills. Can this be elaborated on? \n\n4. Related to 1 above, I found it difficult to relate the features used for the tasks with the actual tasks themselves. More background information and motivation for the features is required (as opposed to saying it makes distinguishing between skills better -- what does better mean here?).\n\n5. Lastly, the paper is full of silly typos, such as the incorrect usage of singular and plural nouns. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1ghdRtJcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "What is the specific question/problem tackled by the paper?\nThis paper addresses the problem of learning and using useful skills through unsupervised RL. While past work (VIC, DIAYN) found skills by maximizing the mutual information between skills and states, this paper (LTS) aims to find transitional skills to move between the primitive skills.\n\nThe proposed method is, if the primitive skills found by DIAYN are labeled w_i, to find skills z_[i,j,k] which corresponds to the kth state while transitioning from w_i to w_j. These transition skills are learned jointly with the primitive skills using the mutual information objective.\n\nThe results in the paper are on mountain car, cart pole, and pendulum. The plots show that distinct primitive skills can be learned in these environments, and that the transition skill transitions between them. The results do not show the actual performance of using these skills for any specific task, which was the motivation presented in the introduction. The \n\nIs the approach well motivated?\nThe approach is not well motivated. The paper claims that transitional skills are necessary, but does not show any evidence of an agents behavior improving by using transitional skills. \n\nI argue to reject the paper in its current state; Sections 3 and 4 need to be rewritten to be more easily understandable, and the need for transitional skills should be more empirically motivated.\n\n\nClarity Issues:\n- The method is presented in a very confusing way. I am not confident that I understand how the transitional skills are defined. \n- Figure 2: What is the “statistical characteristic” and “feature value”? The vague terms are not defined. Which value? What statistic?\n- Figure 5: I see that LTS transitions more smoothly between skills, but why is DIAYN’s fast transition bad? Clearly it still succeeds at reaching the other state.\n\n--------------------------------\nUpdate\nDue to the additional experiments that show the benefit of LTS, I have increased my score to weak reject.\nSection 6.5: Thank you for adding this additional experiment. This section lacks details about how the experiment was run. How was the meta-policy trained? In “3) LTS-master: Transfer policy with optimal weights learned by meta-policy”, why is there transfer? Transfer from what to what? \n“Moreover, we conduct hierarchical framework to weight (or choose) the action modeled by different primitive skills.” What does this sentence mean?\n\nReading over the paper again, the paper needs several editing passes to fix the grammar, as a number of sections (like above) are not understandable. While the paper has improved and is on a good track, I do not think it is yet ready for publication. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission has two issues, identified by the reviewers; (1) the description of the proposed method was found to be confusing at times and could be improved, and (2) the proposed transitional skills were not well motivated/justified as a solution to the problem the authors propose to solve.
Decision: Reject