Title: COMPOSING COMPLEX SKILLS BY LEARNING TRANSITION POLICIES

Abstract: Humans acquire complex skills by exploiting previously learned skills and making transitions between them. To empower machines with this ability, we propose a method that can learn transition policies which effectively connect primitive skills to perform sequential tasks without handcrafted rewards. To efficiently train our transition policies, we introduce proximity predictors which induce rewards gauging proximity to suitable initial states for the next skill. The proposed method is evaluated on a set of complex continuous control tasks in bipedal locomotion and robotic arm manipulation which traditional policy gradient methods struggle at. We demonstrate that transition policies enable us to effectively compose complex skills with existing primitive skills. The proposed induced rewards computed using the proximity predictor further improve training efficiency by providing more dense information than the sparse rewards from the environments. We make our environments, primitive skills, and code public for further research at https://youngwoon.github.io/transition.

1. INTRODUCTION:

While humans are capable of learning complex tasks by reusing previously learned skills, composing and mastering complex skills are not as trivial as sequentially executing those acquired skills. Instead, it requires a smooth transition between skills since the final pose of one skill may not be appropriate to initiate the following one. For example, scoring in basketball with a quick shot after receiving a ball can be decomposed into catching and shooting. However, it is still difficult for beginners who have learned to catch passes and statically shoot. To master this skill, players must practice adjusting their footwork and body into a comfortable shooting pose after catching a pass.

Can machines similarly learn new and complex tasks by reusing acquired skills and learning transitions between them? Learning to perform composite and long-term tasks from scratch requires extensive exploration and sophisticated reward design, which can introduce undesired behaviors (Riedmiller et al., 2018) . Thus, instead of employing intricate reward functions and learning from scratch, modular methods sequentially execute acquired skills with a rule-based meta-policy, enabling machines to solve complicated tasks (Pastor et al., 2009; Mülling et al., 2013; Andreas et al., 2017) . These modular approaches assume that a task can be clearly decomposed into several subtasks which are smoothly connected to each other. In other words, an ending state of one subtask falls within the set of starting states, initiation set, of the next subtask (Sutton et al., 1999) . However, this assumption does not hold in many continuous control problems where a given skill may be executed in starting states not considered during training or designing and thus, fail to achieve its goal.

To bridge the gap between skills, we propose a transition policy which learns to smoothly navigate from an ending state of a skill to suitable initial states of the following skill, as illustrated in Figure 1 . However, learning a transition policy between skills without reward shaping is difficult as the only available learning signal is the sparse reward for the successful execution of the next skill. Sparse success/failure reward is challenging to learn from due to the temporal credit assignment problem (Sutton, 1984) and the lack of information from failing trajectories. To alleviate these problems, we propose a proximity predictor which outputs the proximity to the initiation set of the next skill and acts as a dense reward function for the transition policy.

FINAL:

Good initial states for 𝑝 "#$% Bad initial states for 𝑝 "#$% 𝑝 &'() 𝑝 &'() 𝑝 "#$% execution success 𝑝 "#$% execution fail 𝑝 "#$% 𝑝 "#$% 𝑝 &'() execution

Transition policy execution:

Figure 1 : Concept of a transition policy. Composing complex skills using primitive skills requires smooth transition between primitive skills since a following primitive skill might not be robust to ending states of the previous one. In this example, the ending states (red circles) of the primitive policy p jump are not good initial states to execute the following policy p walk . Therefore, executing p walk from these states will fail (red arrow). To smoothly connect the two primitive policies, we propose a transition policy which navigates an agent to suitable initial states for p walk (dashed arrow), leading to a successful execution of p walk (green arrow).

The main contributions of this paper include (1) the concept of learning transition policies to smoothly connect primitive skills;

(2) a novel modular framework with transition policies that is able to compose complex skills by reusing existing skills; and (3) a joint training algorithm with the proximity predictor specifically designed for efficiently training transition policies. This framework is suited for learning complex skills that require sequential execution of acquired primitive skills, which are common for humans yet relatively unexplored in robot learning. Our experiments on simulated environments demonstrate that employing transition policies solves complex continuous control tasks which traditional policy gradient methods struggle at.

2. RELATED WORK:

Learning continuous control of diverse behaviors in locomotion (Merel et al., 2017; Heess et al., 2017; Peng et al., 2017) and robotic manipulation (Ghosh et al., 2018) is an active research area in reinforcement learning (RL). While some complex tasks can be solved through extensive reward engineering (Ng et al., 1999) , undesired behaviors often emerge (Riedmiller et al., 2018) when tasks require several different primitive skills. Moreover, training complex skills from scratch is not computationally practical.

Real-world tasks often require diverse behaviors and longer temporal dependencies. In hierarchical reinforcement learning, the option framework (Sutton et al., 1999) learns meta actions (options), a series of primitive actions over a period of time. Typically, a hierarchical reinforcement learning framework consists of two components: a high-level meta-controller and low-level controllers. A meta-controller determines the order of subtasks to achieve the final goal and chooses corresponding low-level controllers that generate a sequence of primitive actions. Unsupervised approaches to discover meta actions have been proposed (Schmidhuber, 1990; Daniel et al., 2016; Bacon et al., 2017; Vezhnevets et al., 2017; Dilokthanakul et al., 2017; Levy et al., 2017; Frans et al., 2018; Co-Reyes et al., 2018; Mao et al., 2018) . However, to deal with more complex tasks, additional supervision signals (Andreas et al., 2017; Merel et al., 2017; Shu et al., 2018) or pre-defined lowlevel controllers (Kulkarni et al., 2016; Oh et al., 2017) are required.

To exploit pre-trained modules as low-level controllers, neural module networks (Andreas et al., 2016) have been proposed, which construct a new network dedicated to a given query using a collection of reusable modules. In the RL domain, a meta-controller is trained to follow instructions (Oh et al., 2017) and demonstrations (Xu et al., 2017) , and support multi-level hierarchies (Gudimella et al., 2017) . In the robotics domain, Pastor et al. (2009) ; Kober et al. (2010) ; Mülling et al. (2013) have proposed a modular approach that learns table tennis by selecting appropriate low-level controllers. On the other hand, Andreas et al. (2017) ; Frans et al. (2018) learn abstract skills while experiencing a distribution of tasks and then solve a new task with the learned primitive skills. However, these modular approaches result in undefined behavior when two skills are not smoothly connected. Our proposed framework aims to bridge this gap by training transition policies in a model-free manner to navigate the agent from unseen states for following skills to suitable initial states.

Figure 2 : Our modular network augmented with transition policies. To perform a complex task, our model repeats the following steps: (1) The meta-policy chooses a primitive policy of index c;

(2) The corresponding transition policy helps initiate the chosen primitive policy;

(3) The primitive policy executes the skill; and (4) A success or failure signal for the primitive skill is produced.

Deep RL techniques for continuous control demand dense reward signals; otherwise, they suffer from long training time. Instead of manual reward shaping for denser reward, adversarial reinforcement learning (Ho & Ermon, 2016; Merel et al., 2017; Wang et al., 2017; Bahdanau et al., 2019) employs a discriminator which learns to judge the state or the policy, and the policy takes as rewards the output of the discriminator. While those methods assume ground truth trajectories or goal states are given, our method collects both success and failure trajectories online to train proximity predictors which provide rewards for transition policies.

3. APPROACH:

In this paper, we address the problem of solving a complex task that requires sequential composition of primitive skills given only sparse and binary rewards (i.e. subtask completion reward). The sequential execution of primitive skills fails when two consecutive skills are not smoothly connected.

We propose a modular framework with transition policies that learn to make transition between one policy to the subsequent policy, and therefore, can exploit the given primitive skills to compose complex skills. To accelerate training of transition policies, additional networks, proximity predictors, are jointly trained to provide proximity rewards as intermediate feedback to transition policies. In Section 3.2, we describe our framework in details. Next, in Section 3.3, we elaborate how transition policies are efficiently trained with induced proximity reward.

3.1. PRELIMINARIES:

We formulate our problem as a Markov decision process defined by a tuple {S, A, T , R, ρ, γ} of states, actions, transition probability, reward, initial state distribution, and discount factor. An action distribution of an agent is represented as a policy π θ (a t |s t ), where s t ∈ S is a state, a t ∈ A is an action at time t, and θ are the parameters of the policy. An initial state s 0 is randomly sampled from ρ, and then, an agent iteratively takes an action a t sampled from a policy π θ (a t |s t ) and receives a reward r t until the episode ends. The performance of the agent is evaluated based on a discounted return R =

T -1 t=0 γ t r t , where T is the episode horizon.

3.2. MODULAR FRAMEWORK WITH TRANSITION POLICIES:

To learn a new task given primitive skills {p 1 , p 2 , . . . , p n }, we design a modular framework that consists of the following components: a meta-policy, primitive policies, and transition policies. The meta-policy chooses a primitive skill p c to execute at the beginning and whenever the primitive skill is terminated. Prior to running p c , the transition policy for p c is executed to bring the current state to a plausible initial state for p c , and therefore, p c can be successfully performed. This procedure is repeated to compose complex skills as illustrated in Figure 2 and Algorithm 2. 

A i q X M m n m B Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l K o R 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s Y + J U M 9 5 m s Y x 1 L 6 C G S 6 F 4 G w V K 3 k s 0 p 1 E g e T e Y 3 i 7 8 7 h P X R s T q E W c J 9 y M 6 V i I U j K K V H p J h b V i u u F V 3 C b J J v J x U I E d r W P 4 a j G K W R l w h k 9 S Y v u c m 6 G d U o 2 C S z 0 u D 1 P C E s i k d 8 7 6 l i k b c + N n y 1 D m 5 s s q I h L G 2 p Z A s 1 d 8 T G Y 2 M m U W B 7 Y w o T s y 6 t x D / 8 / o p h j d + J l S S I l d s t S h M J c G Y L P 4 m I 6 E 5 Q z m z h D I t 7 K 2 E T a i m D G 0 6 J R u C t /

7 y J u n U q p 5 b 9 e 7 r l W Y 9 j 6 M I F 3 A J 1 + B B A 5 p w B y 1 o A 4 M x P M M r v D n S e X H e n Y 9 V a 8 H J Z 8 7 h D 5 z P H / y / j Y s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 u b T l A h 8 r 0 0 l 6 s 7 9

w A i q X M m n m B Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l K o R 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s Y + J U M 9 5 m s Y x 1 L 6 C G S 6 F 4 G w V K 3 k s 0 p 1 E g e T e Y 3 i 7 8 7 h P X R s T q E W c J 9 y M 6 V i I U j K K V H p J h b V i u u F V 3 C b J J v J x U I E d r W P 4 a j G K W R l w h k 9 S Y v u c m 6 G d U o 2 C S z 0 u D 1 P C E s i k d 8 7 6 l i k b c + N n y 1 D m 5 s s q I h L G 2 p Z A s 1 d 8 T G Y 2 M m U W B 7 Y w

o T s y 6 t x D / 8 / o p h j d + J l S S I l d s t S h M J c G Y L P 4 m I 6 E 5 Q z m z h D I t 7 K 2 E T a i m D G 0 6 J R u C t / 7 y J u n U q p 5 b 9 e 7 r l W Y 9 j 6 M I F 3 A J 1 + B B A 5 p w B y 1 o A 4 M x P M M r v D n S e X H e n Y 9 V a 8 H J Z 8 7 h D 5 z P H / y / j Y s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 u b T l A h 8 r 0 0 l 6 s 7 9 w

A i q X M m n m B Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l K o R 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s Y + J U M 9 5 m s Y x 1 L 6 C G S 6 F 4 G w V K 3 k s 0 p 1 E g e T e Y 3 i 7 8 7 h P X R s T q E W c J 9 y M 6 V i I U j K K V H p J h b V i u u F V 3 C b J J v J x U I E d r W P 4 a j G K W R l w h k 9 S Y v u c m 6 G d U o 2 C S z 0 u D 1 P C E s i k d 8 7 6 l i k b c + N n y 1 D m 5 s s q I h L G 2 p Z A s 1 d 8 T G Y 2 M m U W B 7 Y w

o T s y 6 t x D / 8 / o p h j d + J l S S I l d s t S h M J c G Y L P 4 m I 6 E 5 Q z m z h D I t 7 K 2 E T a i m D G 0 6 J R u C t / 7 y J u n U q p 5 b 9 e 7 r l W Y 9 j 6 M I F 3 A J 1 + B B A 5 p w B y 1 o A 4 M x P M M r v D n S e X H e n Y 9 V a 8 H J Z 8 7 h D 5 z P H / y / j Y s = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 u b T l A h 8 r 0 0 l 6 s 7 9 w

A i q X M m n m B Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 l K o R 4 L X j x W t B / Q h r L Z b t q l m 0 3 Y n Q g l 9 C d 4 8 a C I V 3 + R N / + N 2 z Y H b X 0 w 8 H h v h p l 5 Q S K F Q d f 9 d g p b 2 z u 7 e 8 X 9 0 s H h 0 f F J + f S s Y + J U M 9 5 m s Y x 1 L 6 C G S 6 F 4 G w V K 3 k s 0 p 1 E g e T e Y 3 i 7 8 7 h P X R s T q E W c J 9 y M 6 V i I U j K K V H p J h b V i u u F V 3 C b J J v J x U I E d r W P 4 a j G K W R l w h k 9 S Y v u c m 6 G d U o 2 C S z 0 u D 1 P C E s i k d 8 7 6 l i k b c + N n y 1 D m 5 s s q I h L G 2 p Z A s 1 d 8 T G Y 2 M m U W B 7 Y w

o T s y 6 t x D / 8 / o p h j d + J l S S I l d s t S h M J c G Y L P 4 m I 6 E 5 Q z m z h D I t 7 K 2 E T a i m D G 0 6 J R u C t / 7 y J u n U q p 5 b 9 e 7 r l W Y 9 j 6 M I F 3 A J 1 + B B A 5 p w B y 1 o A 4 M x P M M r v D n S e X H e n Y 9 V a 8 H J Z 8 7 h D 5 z P H / y / j Y s = < / l a t e x i t > p 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " V 2 9 p v U Q G 3 M K Q l V R z t a q q I J O t h f 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m 0 U I 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H

C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 8 n g e l C u u F V 3 A b J O v J x U I E d z U P 7 q D 2 O W R l w h k 9 S Y n u c m 6 G d U o 2 C S z 0 r 9 1 P C E s g k d 8 Z 6 l i k b c + N n i 1 B m 5 s M q Q h L G 2 p Z A s 1 N 8 T G Y 2 M m U a B 7 Y w o j s 2 q N x f / 8 3 o p h j d + J l S S I l d s u S h M J c G Y z P 8 m Q 6 E 5 Q z m 1 h D I t 7 K 2 E j a m m D G 0 6 J R u C t / r y O m l f V T 2 3 6 t 3 X K o 1 a H k c R z u A c L s G D O j T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 o K T z 5 z C H z i f P / 5 D j Y w = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V 2 9 p v U Q G 3 M K Q l V R z t a q q I J O t h f 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m 0 U I 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H

C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 8 n g e l C u u F V 3 A b J O v J x U I E d z U P 7 q D 2 O W R l w h k 9 S Y n u c m 6 G d U o 2 C S z 0 r 9 1 P C E s g k d 8 Z 6 l i k b c + N n i 1 B m 5 s M q Q h L G 2 p Z A s 1 N 8 T G Y 2 M m U a B 7 Y w o j s 2 q N x f / 8 3 o p h j d + J l S S I l d s u S h M J c G Y z P 8 m Q 6 E 5 Q z m 1 h D I t 7 K 2 E j a m m D G 0 6 J R u C t / r y O m l f V T 2 3 6 t 3 X K o 1 a H k c R z u A c L s G D O j T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 o K T z 5 z C H z i f P / 5 D j Y w = < /

l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V 2 9 p v U Q G 3 M K Q l V R z t a q q I J O t h f 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m 0 U I 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H After executing a primitive policy, a previously performed transition trajectory is labeled and added to a replay buffer based on the execution success. A proximity predictor is trained on states sampled from the two buffers to output the proximity to the initiation set. The predicted proximity serves as a reward to encourage the transition policy to move toward good initial states for the corresponding primitive policy.

C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 8 n g e l C u u F V 3 A b J O v J x U I E d z U P 7 q D 2 O W R l w h k 9 S Y n u c m 6 G d U o 2 C S z 0 r 9 1 P C E s g k d 8 Z 6 l i k b c + N n i 1 B m 5 s M q Q h L G 2 p Z A s 1 N 8 T G Y 2 M m U a B 7 Y w o j s 2 q N x f / 8 3 o p h j d + J l S S I l d s u S h M J c G Y z P 8 m Q 6 E 5 Q z m 1 h D I t 7 K 2 E j a m m D G 0 6 J R u C t / r y O m l f V T 2 3 6 t 3 X K o 1 a H k c R z u A c L s G D O j T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 o K T z 5 z C H z i f P / 5 D j Y w = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " V 2 9 p v U Q G 3 M K Q l V R z t a q q I J O t h f 8 = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B b B U 0 m 0 U I 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o Q S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 I J H C o O t + O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S N n G q G W + x W M a 6 G 1 D D p V C 8 h Q I l 7 y a a 0 y i Q v B N M b u d + 5 4 l r I 2 L 1 i N O E + x E d K R E K R t F K D 8 n g e l C u u F V 3 A b J O v J x U I E d z U P 7 q D 2 O W R l w h k 9 S Y n u c m 6 G d U o 2 C S z 0 r 9 1 P C E s g k d 8 Z 6 l i k b c + N n i 1 B m 5 s M q Q h L G 2 p Z A s 1 N 8 T G Y 2 M m U a B 7 Y w o j s 2 q N x f / 8 3 o p h j d + J l S S I l d s u S h M J c G Y z P 8 m Q 6 E 5 Q z m 1 h D I t 7 K 2 E j a m m D G 0 6 J R u C t / r y O m l f V T 2 3 6 t 3 X K o 1 a H k c R z u A c L s G D O j T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 o K T z 5 z C H z i f P / 5 D j Y w = < / l a t e x i t > s < l a t e x i t s h a 1 _ b a s e 6 4 = " / I i G R H E S G 3 M + N p 9 + O E P l e O z F c j U = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 C A l k Y I e C 1 4 8 t m A / o A 1 l s 5 2 0 a z e b s L s R S u g v 8 O J B E a / + J G / + G 7 d t D t r 6 w s L D O z P s z B s k g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 1 n C q G L R a L W H U D q l F w i S 3 D j c B u o p B G g c B O M L m b 1 z t P q D S P 5 Y O Z J u h H d C R 5 y B k 1 1 m r q Q b n i V t 2 F y D p 4 O V Q g V 2 N Q / u o P Y 5 Z G K A 0 T V O u e 5 y b G z 6 g y n A m c l f q p x o S y C R 1 h z 6 K k E W o / W y w 6 I x f W G Z I w V v Z J Q x b u 7 4 m M R l p P o 8 B 2 R t S M 9 W p t b v 5 X 6 6 U m v P U z L p P U o G T L j 8 J U E B O T + d V k y B U y I 6 Y W K F P c 7 k r Y m C r K j M 2 m Z E P w V k 9 e h / Z 1 1 b P c r F X q V 3 k c R T i D c 7 g E D 2 6 g D v f Q g B Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Y W n H z m F P 7 I + f w B 2 A m M 4 Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / I i G R H E S G 3 M + N p 9 + O E P l e O z F c j U = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 C A l k Y I e C 1 4 8 t m A / o A 1 l s 5 2 0 a z e b s L s R S u g v 8 O J B E a / + J G / + G 7 d t D t r 6 w s L D O z P s z B s k g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 1 n C q G L R a L W H U D q l F w i S 3 D j c B u o p B G g c B O M L m b 1 z t P q D S P 5 Y O Z J u h H d C R 5 y B k 1 1 m r q Q b n i V t 2 F y D p 4 O V Q g V 2 N Q / u o P Y 5 Z G K A 0 T V O u e 5 y b G z 6 g y n A m c l f q p x o S y C R 1 h z 6 K k E W o / W y w 6 I x f W G Z I w V v Z J Q x b u 7 4 m M R l p P o 8 B 2 R t S M 9 W p t b v 5 X 6 6 U m v P U z L p P U o G T L j 8 J U E B O T + d V k y B U y I 6 Y W K F P c 7 k r Y m C r K j M 2 m Z E P w V k 9 e h / Z 1 1 b P c r F X q V 3 k c R T i D c 7 g E D 2 6 g D v f Q g B Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Y W n H z m F P 7 I + f w B 2 A m M 4 Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / I i G R H E S G 3 M + N p 9 + O E P l e O z F c j U = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 C A l k Y I e C 1 4 8 t m A / o A 1 l s 5 2 0 a z e b s L s R S u g v 8 O J B E a / + J G / + G 7 d t D t r 6 w s L D O z P s z B s k g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 1 n C q G L R a L W H U D q l F w i S 3 D j c B u o p B G g c B O M L m b 1 z t P q D S P 5 Y O Z J u h H d C R 5 y B k 1 1 m r q Q b n i V t 2 F y D p 4 O V Q g V 2 N Q / u o P Y 5 Z G K A 0 T V O u e 5 y b G z 6 g y n A m c l f q p x o S y C R 1 h z 6 K k E W o / W y w 6 I x f W G Z I w V v Z J Q x b u 7 4 m M R l p P o 8 B 2 R t S M 9 W p t b v 5 X 6 6 U m v P U z L p P U o G T L j 8 J U E B O T + d V k y B U y I 6 Y W K F P c 7 k r Y m C r K j M 2 m Z E P w V k 9 e h / Z 1 1 b P c r F X q V 3 k c R T i D c 7 g E D 2 6 g D v f Q g B Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Y W n H z m F P 7 I + f w B 2 A m M 4 Q = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / I i G R H E S G 3 M + N p 9 + O E P l e O z F c j U = " > A A A B 6 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F 8 C A l k Y I e C 1 4 8 t m A / o A 1 l s 5 2 0 a z e b s L s R S u g v 8 O J B E a / + J G / + G 7 d t D t r 6 w s L D O z P s z B s k g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 1 n C q G L R a L W H U D q l F w i S 3 D j c B u o p B G g c B O M L m b 1 z t P q D S P 5 Y O Z J u h H d C R 5 y B k 1 1 m r q Q b n i V t 2 F y D p 4 O V Q g V 2 N Q / u o P Y 5 Z G K A 0 T V O u e 5 y b G z 6 g y n A m c l f q p x o S y C R 1 h z 6 K k E W o / W y w 6 I x f W G Z I w V v Z J Q x b u 7 4 m M R l p P o 8 B 2 R t S M 9 W p t b v 5 X 6 6 U m v P U z L p P U o G T L j 8 J U E B O T + d V k y B U y I 6 Y W K F P c 7 k r Y m C r K j M 2 m Z E P w V k 9 e h / Z 1 1 b P c r F X q V 3 k c R T i D c 7 g E D 2 6 g D v f Q g B Y w Q H i G V 3 h z H p 0 X 5 9 3 5 W L Y W n H z m F P 7 I + f w B 2 A m M 4 Q = = < / l a t e x i t > (s, v) < l a t e x i t s h a 1 _ b a s e 6 4 = " R R a 6 u 0 / Z J 3 k N M g H e d j m 9 Z A H a Z s 4 = " > A A A B 7 H i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q F D K r g h 6 L H j x W M G t h X Y p 2 X S 2 D c 1 m l y R b K K W / w Y s H R b z 6 g 7 z 5 b 0 z b P W j r C 4 G H d 2 b I z B u m g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 0 k i m G P k t E o t o h 1 S i 4 R N 9 w I 7 C d K q R x K P A p H N 3 N 6 0 9 j V J o n 8 t F M U g x i O p A 8 4 o w a a / l V X R t f 9 s o V t + 4 u R N b B y 6 E C u Z q 9 8 l e 3 n 7 A s R m m Y o F p 3 P D c 1 w Z Q q w 5 n A W a m b a U w p G 9 E B d i x K G q M O p o t l Z + T C O n 0 S J c o + a c j C / T 0 x p b H W k z i 0 n T E 1 Q 7 1 a m 5 v / 1 T q Z i W 6 D K Z d p Z l C y 5 U d R J o h J y P x y 0 u c K m R E T C 5 Q p b n c l b E g V Z

= " > A A A B 7 H i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q F D K r g h 6 L H j x W M G t h X Y p 2 X S 2 D c 1 m l y R b K K W / w Y s H R b z 6 g 7 z 5 b 0 z b P W j r C 4 G H d 2 b I z B u m g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 0 k i m G P k t E o t o h 1 S i 4 R N 9 w I 7 C d K q R x K P A p H N 3 N 6 0 9 j V J o n 8 t F M U g x i O p A 8 4 o w a a / l V X R t f 9 s o V t + 4 u R N b B y 6 E C u Z q 9 8 l e 3 n 7 A s R m m Y o F p 3 P D c 1 w Z Q q w 5 n A W a m b a U w p G 9 E B d i x K G q M O p o t l Z + T C O n 0 S J c o + a c j C / T 0 x p b H W k z i 0 n T E 1 Q 7 1 a m 5 v / 1 T q Z i W 6 D K Z d p Z l C y 5 U d R J o h J y P x y 0 u c K m R E T C 5 Q p b n c l b E g V Z

= " > A A A B 7 H i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q F D K r g h 6 L H j x W M G t h X Y p 2 X S 2 D c 1 m l y R b K K W / w Y s H R b z 6 g 7 z 5 b 0 z b P W j r C 4 G H d 2 b I z B u m g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 0 k i m G P k t E o t o h 1 S i 4 R N 9 w I 7 C d K q R x K P A p H N 3 N 6 0 9 j V J o n 8 t F M U g x i O p A 8 4 o w a a / l V X R t f 9 s o V t + 4 u R N b B y 6 E C u Z q 9 8 l e 3 n 7 A s R m m Y o F p 3 P D c 1 w Z Q q w 5 n A W a m b a U w p G 9 E B d i x K G q M O p o t l Z + T C O n 0 S J c o + a c j C / T 0 x p b H W k z i 0 n T E 1 Q 7 1 a m 5 v / 1 T q Z i W 6 D K Z d p Z l C y 5 U d R J o h J y P x y 0 u c K m R E T C 5 Q p b n c l b E g V Z

= " > A A A B 7 H i c b Z B N S w M x E I Z n 6 1 e t X 1 W P X o J F q F D K r g h 6 L H j x W M G t h X Y p 2 X S 2 D c 1 m l y R b K K W / w Y s H R b z 6 g 7 z 5 b 0 z b P W j r C 4 G H d 2 b I z B u m g m v j u t 9 O Y W N z a 3 u n u F v a 2 z 8 4 P C o f n 7 R 0 k i m G P k t E o t o h 1 S i 4 R N 9 w I 7 C d K q R x K P A p H N 3 N 6 0 9 j V J o n 8 t F M U g x i O p A 8 4 o w a a / l V X R t f 9 s o V t + 4 u R N b B y 6 E C u Z q 9 8 l e 3 n 7 A s R m m Y o F p 3 P D c 1 w Z Q q w 5 n A W a m b a U w p G 9 E B d i x K G q M O p o t l Z + T C O n 0 S J c o + a c j C / T 0 x p b H W k z i 0 n T E 1 Q 7 1 a m 5 v / 1 T q Z i W 6 D K Z d p Z l C y 5 U d R J o h J y P x y 0 u c K m R E T C 5 Q p b n c l b E g V Z

We denote the meta-policy as π meta (p c |s), where c ∈ [1, n] is a primitive policy index. The observation of the meta-policy contains the low-level information of primitives and task specifications indicating high-level goals (e.g. moving direction and target object position). For example, a walking primitive only takes joint information as observation while the meta-policy additionally takes target direction. In this paper, we use a rule-based meta-policy and focus on transitioning between consecutive primitive policies.

Once a primitive skill p c is chosen to be executed, the agent generates an action a t ∼ π pc (a|s t ) based on the current state s t . Note that we did not differentiate state spaces for primitive polices because of the simplicity of notations (e.g. the observation of the jumping primitive contains a distance to a curb while that of the walking primitive only has joint pose and velocities). Every primitive policy is required to generate termination signals τ pc ∈ {continue, success, fail} to indicate policy completion and whether it believes the execution is successful or not. While our method is agnostic to the form of primitive policies (e.g. rule-based, inverse kinematics), we consider the case of a pre-trained neural network in this paper.

For smooth transitions between primitive policies, we add a transition policy π φc (a|s) before executing primitive skill p c , which guides an agent to p c 's initiation set, where φ c is the parameters of the transition policy for p c . Note that the transition policy for p c is shared across different preceding primitive policies since a successful transition is defined by the success of the following primitive skill p c . For brevity of notation, we omit the primitive policy index c in the following equations where unambiguous. The transition policy's state and action space are the same as the primitive policy's. The transition policy also learns a termination signal τ trans which indicates transition termination to successfully initiate p c . Our framework contains one transition policy for each primitive skill, in total n transition policies {π φ1 , π φ2 , . . . , π φn }.

3.3. TRAINING TRANSITION POLICIES:

In our framework, transition policies are trained to make the execution of the corresponding following primitive policies successful. During rollouts, transition trajectories are collected and each trajectory can be naively labeled by the success execution of its corresponding primitive policy. Then, transition policies are trained to maximize the average success of the respective primitive policy. In this scenario, by definition, the only available learning signal for the transition policies is the sparse and binary rewards for the completion of the next task.

To alleviate the sparsity of rewards and maximize the objective of moving to viable initial states for the next primitive, we propose a proximity predictor that learns and provides a dense reward, dubbed proximity reward, of how close transition states are to the initiation set of the corresponding primitive p c as shown in Figure 3 . We denote a proximity predictor as P ωc which is parameterized by ω c . We define the proximity of a state as the future discounted proximity, v = δ step , where step is the number of steps required to reach an initiation set of the following primitive policy. The proximity of a state can also be a linearly discounted function such as v = 1 -δ • step. We refer the readers to the supplementary for comparison of two proximity functions.

The proximity predictor is trained to minimize a mean squared error of proximity prediction:

EQUATION

where B S and B F are collections of states from success and failure transition trajectories, respectively. To estimate the proximity to an initiation set, B S contains not only the state that directly leads to the success of the following primitive policy, but also the intermediate states of the successful trajectories with its proximity. By minimizing this objective, given a state, the proximity predictor is learned to predict 1 if the state is in the initiation set, a value that is between 0 and 1 if the state leads the agent to end up with a desired initial states, and 0 when the state leads to a failure.

The goal of a transition policy is to get close to an initiation set which can be formulated as seeking a state s predicted to be in the initiation set by the proximity predictor (i.e. P ω (s) is close to 1).

To achieve this goal, the transition policy learns to maximize proximity prediction at the ending state of the transition trajectory P ω (s T ). In addition to providing reward at the end, we also use the increase of predicted proximity to the initiation set, P ω (s t+1 ) -P ω (s t ), at every timestep as a reward, dubbed proximity reward, to create a denser reward. The transition policy is trained to maximize the expected discounted return:

EQUATION

However, in general skill learning scenarios, ground truth states (B S and B F ) for training proximity predictors are not available. Hence, the training data for a proximity predictor is obtained online during training its corresponding transition policy. Specifically, we label the states in a transition trajectory as success or failure based on whether the following primitive is successfully executed or not, and add them into the corresponding buffers B S or B F , respectively. As stated in Algorithm 1, we train transition policies and proximity predictors by alternating between an Adam (Kingma & Ba, 2015) gradient step on ω to minimize Equation (1) with respect to P ω and a PPO (Schulman et al., 2017) step on φ to maximize Equation (2) with respect to π φ . We refer readers to the supplementary for further details on training.

In summary, we propose to compose complex skills with transition policies that enable smooth transition between previously acquired primitive policies. Specifically, we propose to reward transition policies based on how close the current state is to suitable initial states of the subsequent policy (i.e. initiation set). To provide the proximity of a state, we collect failing and successful trajectories on the fly and train a proximity predictor to predict the proximity.

Utilizing the learned proximity predictors and proximity rewards for training transition policies is beneficial in the following perspectives: (1) the dense rewards speed up transition policy training by differentiating failing states from states in a successful trajectory; and (2) the joint training mechanism prevents a transition policy from getting stuck in local optima. Whenever a transition policy gets into a local optimum (i.e. fails the following skill with a high proximity reward), the proximity predictor learns to lower the proximity for the failing transition as those states are added to its failure buffer, escaping the local optimum.

4. EXPERIMENTS:

We conducted experiments on two classes of continuous control tasks: robotic manipulation and locomotion. To illustrate the potential of the proposed framework, modular framework with Transition Policies (TP), we designed a set of complex tasks that require agents to utilize diverse primitive skills which are not optimized for smooth composition. All of our environments are simulated in the MuJoCo physics engine (Todorov et al., 2012) . 

4.1. BASELINES:

We evaluate our method to answer how transition policies benefit complex task learning and how joint training with proximity predictors boosts training of transition policies. To investigate the impact of the transition policy, we compared policies learned from dense rewards with our modular framework that only learns from sparse and binary rewards (i.e. subtask completion rewards). Moreover, we conducted ablation studies to dissect each component in the training method of transition polices. To answer these questions, we compare the following methods:

• Trust Region Policy Optimization with dense reward (TRPO) represents a state-of-the-art policy gradient method (Schulman et al., 2015) , which we use for the standard RL comparison. • Proximal Policy Optimization with dense reward (PPO) is another state-of-the-art policy gradient method (Schulman et al., 2017) , which is more stable than TRPO with smaller batch sizes. • Without transition policies (Without-TP) sequentially executes primitive policies without transition policies and has no learnable components. • Transition policies trained on task rewards (TP-Task) represents a modular network augmented with transition policies learned from the sparse and binary reward (i.e. subtask completion reward), whereas our model learns from the dense proximity reward. • Transition policies trained on sparse proximity rewards (TP-Sparse) is a variant of our model which has the proximity reward only at the end of the transition trajectory. In contrast, our model learns from dense proximity rewards generated every timestep. • Transition policies trained on dense proximity rewards (TP-Dense, Ours) is our final model where transition policies learn from dense proximity rewards.

Initially, we tried comparing baseline methods with our method using only sparse and binary rewards. However, the baselines could not solve any of the tasks due to the complexity and sparse reward of the environments. To provide more competitive comparisons, we engineer dense rewards for baselines (TRPO and PPO) to boost their performance and give baselines 5 times longer training times. We show that transitions with sparse rewards can compete with and even outperform baselines learning from dense rewards. As the performance of TRPO and PPO varies significantly between runs, we train each task with 3 different random seeds and report mean and standard deviation in Figure 4 .

4.2. ROBOTIC MANIPULATION:

For robotic manipulation, we simulate a Kinova Jaco, a 9 DoF robotic arm with 3 fingers. The agent receives full state information, including the absolute location of external objects. The agent uses joint torque control to perform actions. The results are shown in Figure 4 and Table 1 .

Pre-trained primitives. There are four pre-trained primitives available: Picking up, Catching, Tossing, and Hitting. Picking up requires the robotic arm to pick up a small block, which is randomly placed on the table. If the box is not picked up after a certain amount of time, the agent fails.

Catching learns to catch a block that is thrown towards the arm with random initial position and velocity. The agent fails if it does not catch and stably hold the box for a certain amount of time.

Tossing requires the robot to pick up a box, toss it vertically in the air, and land the box at a specified position. Hitting requires the robot to hit a box dropped overhead at a target ball.

Repetitive picking up. The Repetitive picking up task requires the agent to complete the Picking up task 5 times. After each successful pick, the box disappears and a new box will be placed randomly on the table again. Our model achieves the best performance and converges the fastest by learning from the proposed proximity reward. With our dense proximity reward at every transition step, we alleviate credit assignment when compared to providing a sparse proximity reward (TP-Sparse) or using a sparse task reward (TP-Task). Conversely, TRPO and PPO with dense rewards take significantly longer to learn and is unable to pick up the second box as the ending pose after the first picking up is too unstable to initialize the next picking up.

Repetitive catching. Similar to Repetitive picking up, the Repetitive catching task requires the agent to catch boxes consecutively up to 5 times. In this task, other than the modular network without a transition policy, all baselines are able to eventually learn while our model still learns the fastest. We believe this is because the Catching primitive policy has a larger initiation set and therefore, the sparse reward problem is less severe since random exploration is able to succeed with a higher chance.

Serve. Inspired by tennis, Serve requires the robot to toss the ball and hit it at a target. Even with an extensively engineered reward, TRPO and PPO baselines fail to learn because Hitting is not able to learn to cover all terminal states of Tossing (i.e. a set of initial states for Hitting is large which demands longer training time). In contrast, learning to recover from Tossing's ending states to Hitting's initiation set is easier for exploration (11% of Tossing's ending states are covered by Hitting's initiation set as can be seen in Table 1 ), which reduces the complexity of the task. Thus, our method and the sparse proximity reward baseline are both able to solve it. However, the ablated variant trained on task reward shows high success rates at the beginning of training and collapses after 100 iterations. The performance drops because the transition policy tries to solve failure cases by increasing the transition length and it reaches to a point that it hardly gets reward. This result shows that once the policy falls into local optima, it is not able to escape because the policy will never get a sparse task reward. On the other hand, our method is robust to local optima since the jointly learned dense proximity reward provides a learning signal to an agent even though it cannot get a task reward.

4.3. LOCOMOTION:

For locomotion, we simulate a 9 DoF planar (2D) bipedal walker. The observation of the agent includes joint position, rotation, and velocity. When the agent needs to interact with objects in the environment, we provide additional input such as distance to the curb and ceiling in front of the agent. The agent uses joint torque control to perform actions. The results are shown in Figure 4 and Table 2 .

Pre-trained primitives. Forward and Backward require the walker to walk forward and backward with a certain velocity, respectively. Balancing requires the walker to robustly stand still under the random external forces. Jumping requires the walker jump over a randomly located curb and land safely. Crawling requires the walker to crawl under a ceiling. In all the aforementioned scenarios, the walker fails when the height of the walker is lower than a threshold.

Patrol (Forward and backward). The Patrol task involves walking forward and backward toward goal points on either side and balancing in between to smoothly change its direction. As illustrated in Figure 4 , our method consistently outperforms TRPO, PPO, and ablated baselines in stably walking forward and transitioning to walk backward. The agent trained with dense rewards is not able to consistently switch directions, whereas our model can utilize previously learned primitives including Balancing to stabilize a reversal in velocity.

Hurdle (Walking forward and jumping). The Hurdle task requires the agent to walk forward and jump across curbs, which requires a transition between walking and jumping as well as landing the jump to walking forward. As shown in Figure 4 , our method outperforms the sparse reward baselines, showing the efficiency our proposed proximity reward. While TRPO with dense rewards can learn this task as well, it requires dense rewards consisting of eight different components to collectively enable TRPO to learn the task. It can be considered as learning both primitive skills and transition between skills from dense rewards. However, the main focus of this paper is to learn a complex task by reusing acquired skills, avoiding an extensive reward design.

Obstacle Course (Walking forward, jumping, and crawling). Obstacle Course is the most difficult among the locomotion tasks, where the walker must walk forward, jump across curbs, and crawl underneath ceilings. It requires three different behaviors and transitions between two very different primitive skills: crawling and jumping. Since the task requires significantly different behaviors that are hard to transition between, TRPO fails to learn the task and only tries to crawl toward the curb

Serve < l a t e x i t s h a 1 _ b a s e 6 4 = " d z B 3 7 9 C J X v T d q Y n M h n t r f j P m B Z g = " > A A A B + H i c b Z D L S s N A F I Y n 9 V b r p V W X b o J F c F U S K e i y 4 M Z l R X u B N p T J 9 K Q d O p m E m Z N i D X 0 S N y 4 U c e u j u P N t n L R Z a O u B g Y / / P 2 f m z O / H g m t 0 n G + r s L G 5 t b 1 T 3 C 3 t 7 R 8 c l i t H x 2 0 d J Y p B i 0 U i U l 2 f a h B c Q g s 5 C u j G C m j o C + j 4 k 5 v M 7 0 x B a R 7 J B 5 z F 4 I V 0 J H n A G U U j D S r l P s I j a p b e g 5 r C v D S o V J 2 a s y h 7 H d w c q i S v 5 q D y 1 R 9 G L A l B I h N U 6 5 7 r x O i l V C F n w l z Y T z T E l E 3 o C H o G J Q 1 B e + l i 8 b l 9 b p S h H U T K H I n 2 Q v 0 9 k d J Q 6 1 n o m 8 6 Q 4 l i v e p n 4 n 9 d L M L j 2 U i 7 j B E G y 5 U N B I m y M 7 C w F e 8 g V M B Q z A 5 Q p b n a 1 2 Z g q y t B k l Y X g r n 5 5 H d q X N d f w X b 3 a q O d x F M k p O S M X x C V X p E F u S Z O 0 C C M J

e S a v 5 M 1 6 s l 6 s d + t j 2 V q w 8 p k T 8 q e s z x / q 2 Z M x < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

d z B 3 7 9 C J X v T d q Y n M h n t r f j P m B Z g = " > A A A B + H i c b Z D L S s N A F I Y n 9 V b r p V W X b o J F c F U S K e i y 4 M Z l R X u B N p T J 9 K Q d O p m E m Z N i D X 0 S N y 4 U c e u j u P N t n L R Z a O u B g Y / / P 2 f m z O / H g m t 0 n G + r s L G 5 t b 1 T 3 C 3 t 7 R 8 c l i t H x 2 0 d J Y p B i 0 U i U l 2 f a h B c Q g s 5 C u j G C m j o C + j 4 k 5 v M 7 0 x B a R 7 J B 5 z F 4 I V 0 J H n A G U U j D S r l P s I j a p b e g 5 r C v D S o V J 2 a s y h 7 H d w c q i S v 5 q D y 1 R 9 G L A l B I h N U 6 5 7 r x O i l V C F n w l z Y T z T E l E 3 o C H o G J Q 1 B e + l i 8 b l 9 b p S h H U T K H I n 2 Q v 0 9 k d J Q 6 1 n o m 8 6 Q 4 l i v e p n 4 n 9 d L M L j 2 U i 7 j B E G y 5 U N B I m y M 7 C w F e 8 g V M B Q z A 5 Q p b n a 1 2 Z g q y t B k l Y X g r n 5 5 H d q X N d f w X b 3 a q O d x F M k p O S M X x C V X p E F u S Z O 0 C C M J

e S a v 5 M 1 6 s l 6 s d + t j 2 V q w 8 p k T 8 q e s z x / q 2 Z M x < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

d z B 3 7 9 C J X v T d q Y n M h n t r f j P m B Z g = " > A A A B + H i c b Z D L S s N A F I Y n 9 V b r p V W X b o J F c F U S K e i y 4 M Z l R X u B N p T J 9 K Q d O p m E m Z N i D X 0 S N y 4 U c e u j u P N t n L R Z a O u B g Y / / P 2 f m z O / H g m t 0 n G + r s L G 5 t b 1 T 3 C 3 t 7 R 8 c l i t H x 2 0 d J Y p B i 0 U i U l 2 f a h B c Q g s 5 C u j G C m j o C + j 4 k 5 v M 7 0 x B a R 7 J B 5 z F 4 I V 0 J H n A G U U j D S r l P s I j a p b e g 5 r C v D S o V J 2 a s y h 7 H d w c q i S v 5 q D y 1 R 9 G L A l B I h N U 6 5 7 r x O i l V C F n w l z Y T z T E l E 3 o C H o G J Q 1 B e + l i 8 b l 9 b p S h H U T K H I n 2 Q v 0 9 k d J Q 6 1 n o m 8 6 Q 4 l i v e p n 4 n 9 d L M L j 2 U i 7 j B E G y 5 U N B I m y M 7 C w F e 8 g V M B Q z A 5 Q p b n a 1 2 Z g q y t B k l Y X g r n 5 5 H d q X N d f w X b 3 a q O d x F M k p O S M X x C V X p E F u S Z O 0 C C M J

e S a v 5 M 1 6 s l 6 s d + t j 2 V q w 8 p k T 8 q e s z x / q 2 Z M x < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = "

d z B 3 7 9 C J X v T d q Y n M h n t r f j P m B Z g = " > A A A B + H i c b Z D L S s N A F I Y n 9 V b r p V W X b o J F c F U S K e i y 4 M Z l R X u B N p T J 9 K Q d O p m E m Z N i D X 0 S N y 4 U c e u j u P N t n L R Z a O u B g Y / / P 2 f m z O / H g m t 0 n G + r s L G 5 t b 1 T 3 C 3 t 7 R 8 c l i t H x 2 0 d J Y p B i 0 U i U l 2 f a h B c Q g s 5 C u j G C m j o C + j 4 k 5 v M 7 0 x B a R 7 J B 5 z F 4 I V 0 J H n A G U U j D S r l P s I j a p b e g 5 r C v D S o V J 2 a s y h 7 H d w c q i S v 5 q D y 1 R 9 G L A l B I h N U 6 5 7 r x O i l V C F n w l z Y T z T E l E 3 o C H o G J Q 1 B e + l i 8 b l 9 b p S h H U T K H I n 2 Q v 0 9 k d J Q 6 1 n o m 8 6 Q 4 l i v e p n 4 n 9 d L M L j 2 U i 7 j B E G y 5 U N B I m y M 7 C w F e 8 g V M B Q z A 5 Q p b n a 1 2 Z g q y t B k l Y X g r n 5 5 H d q X N d f w X b 3 a q O d x F M k p O S M X x C V X p E F u S Z O 0 C C M J

e S a v 5 M 1 6 s l 6 s d + t j 2 V q w 8 p k T 8 q e s z x / q 2 Z M x < / l a t e x i t > FINAL (a) without attempting to jump. In contrast, our method learns to transition between all pairs of primitive skills and often succeeds in crossing multiple obstacles.

Patrol < l a t e x i t s h a 1 _ b a s e 6 4 = " d 1 O y u n v e 2 J K 9 E y N e d / H V D Z S n R o Q = " > A A A B + X i c b Z B N S 8 N A E I Y n 9 a v W r 6 h H L 8 E i e C q J C H o s e P F Y w X 5 A G 8 p m u 2 2 X b n b D 7 q R Y Q v + J F w + K e P W f e P P f u G l z 0 N Y X F h 7 e m W F m 3 y g R 3 K D v f z u l j c 2 t 7 Z 3 y b m V v / + D w y D 0 + a R m V a s q a V A m l O x E x T H D J m s h R s E 6 i G Y k j w d r R 5 C 6 v t 6 d M G 6 7 k I 8 4 S F s Z k J P m Q U 4 L W 6 r t u D 9 k T G p o 1 C G o l 5 p W + W / V r / k L e O g Q F V K F Q o + 9 + 9 Q a K p j G T S A U x p h v 4 C Y Y Z 0 c i p Y P N K L z U s I X R C R q x r U Z K Y m T B b X D 7 3 L q w z 8 I Z K 2 y f R W 7 i / J z I S G z O L I 9 s Z E x y b 1 V p u / l f r p j i 8 D T M u k x S Z p M t F w 1 R 4 q L w 8 B m / A N a M o Z h Y I 1 d z e 6 t E x 0 Y S i D S s P I V j 9 8 j q 0 r m q B 5 Y f r a v 2 6 i K M M Z 3 A O l x D A D d T h H h r Q B A p T e I Z X e H M y 5 8 V 5 d z 6 W r S W n m D m F P 3 I + f w C 7 d Z O o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 1 O y u n v e 2 J K 9 E y N e d / H V D Z S n R o Q = " > A A A B + X i c b Z B N S 8 N A E I Y n 9 a v W r 6 h H L 8 E i e C q J C H o s e P F Y w X 5 A G 8 p m u 2 2 X b n b D 7 q R Y Q v + J F w + K e P W f e P P f u G l z 0 N Y X F h 7 e m W F m 3 y g R 3 K D v f z u l j c 2 t 7 Z 3 y b m V v / + D w y D 0 + a R m V a s q a V A m l O x E x T H D J m s h R s E 6 i G Y k j w d r R 5 C 6 v t 6 d M G 6 7 k I 8 4 S F s Z k J P m Q U 4 L W 6 r t u D 9 k T G p o 1 C G o l 5 p W + W / V r / k L e O g Q F V K F Q o + 9 + 9 Q a K p j G T S A U x p h v 4 C Y Y Z 0 c i p Y P N K L z U s I X R C R q x r U Z K Y m T B b X D 7 3 L q w z 8 I Z K 2 y f R W 7 i / J z I S G z O L I 9 s Z E x y b 1 V p u / l f r p j i 8 D T M u k x S Z p M t F w 1 R 4 q L w 8 B m / A N a M o Z h Y I 1 d z e 6 t E x 0 Y S i D S s P I V j 9 8 j q 0 r m q B 5 Y f r a v 2 6 i K M M Z 3 A O l x D A D d T h H h r Q B A p T e I Z X e H M y 5 8 V 5 d z 6 W r S W n m D m F P 3 I + f w C 7 d Z O o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 1 O y u n v e 2 J K 9 E y N e d / H V D Z S n R o Q = " > A A A B + X i c b Z B N S 8 N A E I Y n 9 a v W r 6 h H L 8 E i e C q J C H o s e P F Y w X 5 A G 8 p m u 2 2 X b n b D 7 q R Y Q v + J F w + K e P W f e P P f u G l z 0 N Y X F h 7 e m W F m 3 y g R 3 K D v f z u l j c 2 t 7 Z 3 y b m V v / + D w y D 0 + a R m V a s q a V A m l O x E x T H D J m s h R s E 6 i G Y k j w d r R 5 C 6 v t 6 d M G 6 7 k I 8 4 S F s Z k J P m Q U 4 L W 6 r t u D 9 k T G p o 1 C G o l 5 p W + W / V r / k L e O g Q F V K F Q o + 9 + 9 Q a K p j G T S A U x p h v 4 C Y Y Z 0 c i p Y P N K L z U s I X R C R q x r U Z K Y m T B b X D 7 3 L q w z 8 I Z K 2 y f R W 7 i / J z I S G z O L I 9 s Z E x y b 1 V p u / l f r p j i 8 D T M u k x S Z p M t F w 1 R 4 q L w 8 B m / A N a M o Z h Y I 1 d z e 6 t E x 0 Y S i D S s P I V j 9 8 j q 0 r m q B 5 Y f r a v 2 6 i K M M Z 3 A O l x D A D d T h H h r Q B A p T e I Z X

m V v / + D w y D 0 + a R m V a s q a V A m l O x E x T H D J m s h R s E 6 i G Y k j w d r R 5 C 6 v t 6 d M G 6 7 k I 8 4 S F s Z k J P m Q U 4 L W 6 r t u D 9 k T G p o 1 C G o l 5 p W + W / V r / k L e O g Q F V K F Q o + 9 + 9 Q a K p j G T S A U x p h v 4 C Y Y Z 0 c i p Y P N K L z U s I X R C R q x r U Z K Y m T B b X D 7 3 L q w z 8 I Z K 2 y f R W 7 i / J z I S G z O L I 9 s Z E x y b 1 V p u / l f r p j i 8 D T M u k x S Z p M t F w 1 R 4 q L w 8 B m / A N a M o Z h Y I 1 d z e 6 t E x 0 Y S i D S s P I V j 9 8 j q 0 r m q B 5 Y f r a v 2 6 i K M M Z 3 A O l x D A D d T h H h r Q B A p T e I Z X

4.4. ABLATION STUDY:

We conducted additional experiments to understand the contribution of transition policies, proximity predictors, and dense proximity rewards. The modular framework without transition policies (Without-TP) tends to fail the execution of the second skill since the second skill is not trained to cover ending states of the first skill. Especially, in continuous control making a primitive skill that can cover all possible states is very challenging. Transition policies trained from task completion reward (TP-Task) and sparse proximity reward (TP-Sparse) learn to connect consecutive primitives slower because sparse reward is hard to learn from due to the credit assignment problem. On the other hand, our model alleviates the credit assignment problem and learns quickly by giving predicted proximity reward for every transition state-action pair.

4.5. TRAINING OF TRANSITION AND PROXIMITY PREDICTOR:

To investigate how transition polices learn to solve the tasks, we present the lengths of transition trajectories and the obtained proximity rewards during training in Figure 5 . For manipulation, we show the results of Repetitive picking up and Repetitive catching. For locomotion, we show Patrol with three different transition policies.

The transition policy quickly learns to maximize the proximity reward regardless of the accuracy of the proximity predictor. All the transition policies increase the length while exploring in the beginning, especially for picking up (55 steps) and balance (45 steps). This is because a randomly initialized proximity predictor outputs high proximity for unseen states and a transition policy tries to get a high reward by visiting these states. However, as these failing initial states with high proximity are collected in the failure buffers, the proximity predictor lowers their proximity and the transition policy learns to avoid them. In other words, the transition policy will end up seeking successful states. As transition policies learn to transition to the following skills, the length decreases to get higher proximity rewards earlier.

4.6. VISUALIZING TRANSITION TRAJECTORY:

Figure 6a shows two transition trajectories (from s 0 to t 0 and s 1 to t 1 ) and two-dimensional PCA embedding of the ending states (blue) and initiation states (red) of the Picking up primitive. A transition policy starts from states s 0 and s 1 where the previous Picking up primitive is terminated.

As can be seen in Figure 6a , the proximity predictor outputs small values for s 0 and s 1 since they are far from the initiation set of Picking up primitive. Trajectories in the figure show that as the transition policy moves toward states with higher proximity, and finally ends up with states t 0 and t 1 which are in the initiation set of the primitive policy.

Figure 6b illustrates PCA embeddings of initiation sets of three primitive skills, Forward (green), Backward (orange), and Balancing (blue). A transition from Forward to Balancing has very long trajectory, but predicted proximity helps the transition policy to reach to an initiation state t 0 . On the other hand, transitioning between Balancing and Backward only requires 7 steps. 

5. CONCLUSION:

In this work, we propose a modular framework with transition policies to empower reinforcement learning agents to learn complex tasks with sparse reward by utilizing prior knowledge. Specifically, we formulate the problem as executing existing primitive skills while smoothly transitioning between primitive skills. To learn transition polices in a sparse reward setting, we propose a proximity predictor which generates dense reward signals and jointly train transition policies and proximity predictors. Our experimental results on robotic manipulation and locomotion tasks demonstrate the effectiveness of employing transition policies. The proposed framework solves complex tasks without reward shaping and outperforms baseline RL algorithms and other ablated baselines.

There are many future directions to investigate. Our method is designed to focus on acquiring transition policies that connect a given set of primitive policies under the predefined meta-policy. We believe that joint learning of a meta-policy and transition policies on a new task would make our framework more flexible. Moreover, we made an assumption that successful transition between two consecutive policies should be achievable by random exploration. To alleviate the exploration problem with sparse rewards, our transition policy training can incorporate exploration methods such as count-based exploration bonuses (Bellemare et al., 2016; Martin et al., 2017) and curiositydriven intrinsic reward (Pathak et al., 2017) . We also assume our primitive policies return a signal that indicates whether the execution should be terminated or not, similar to Kulkarni et al. ( 2016); Oh et al. (2017) ; Le et al. (2018) . Learning to assess the successful termination of primitive policies together with learning transition policies is a promising future direction. Using a linearly discounted proximity function, 1 -δ • step, is also a valid choice. We compare the two proximity functions on a manipulation task (Repetitive catching) and a locomotion task (Obstacle course), as shown in Figure 7 , where δ for exponential decay and linear decay are 0.95 and 0.01, respectively. The results demonstrate that our model is able to learn well with both proximity functions and they perform similarly.

Originally, we opted for the exponential proximity function with the intuition that the faster initial decay near the initiation set would help the policy discriminate successful states from failing states near the initiation set. Also, in our experiments, as we use 0.95 as a decaying factor, the proximity is still reasonably large (e.g., 0.35 for 20 time-steps and 0.07 for 50 time-steps). In this paper, we use the exponential proximity function for all experiments.

B.4 PROXIMITY PREDICTOR:

A proximity predictor takes a state as input which includes joint state information, joint acceleration, and any task specification, such as ceiling and curb information. A proximity predictor consists of 2 fully connected layers of 96 hidden units with ReLU nonlinearities and predicts the proximity to the initiation set based on the states sampled from the success and failure buffers. Each training iteration consists of 10 epochs over a batch size of 64 and use a learning rate of 10 -4 . The predictor optimizes the loss in Equation ( 1), similar to the LSGAN loss (Mao et al., 2017) .

B.5 TRANSITION POLICIES:

An observation space of a transition policy consists of joint state information and joint acceleration. A transition policy consists of 2 fully connected layers of 32 hidden units with tanh nonlinearities and predicts the mean and standard deviation of a Gaussian distribution over an action space. A 2-way softmax layer is followed by the last fully connected layer to predict whether to terminate the current transition or not. We train all transition policies using PPO (Schulman et al., 2017) since PPO is robust on smaller batch sizes and the transition states collected for each update is much smaller than the size of a rollout. Each training iteration consists of 5 epochs over a batch. Compute the discounted proximity v of each state s in B.

22::

Add pairs of (s, v) to B S c or B F c according to τ pc .

23::

Add B to the rollout buffer R c . 24: end while B.6 SCALABILITY Each sub-policy requires its corresponding transition policy, proximity predictor, and two buffers. Hence, both the time and memory complexities of our method are linearly dependent on the number of sub-policies. The memory overhead is affordable since a transition policy (2 layers of 32 hidden units), a proximity predictor (2 layers of 96 hidden units), and replay buffers (1M states) are small.

C ENVIRONMENT DESCRIPTIONS:

For every task, we add a control penalty, -0.001 * a 2 , to regularize the magnitude of actions where a is a torque action performed by an agent. Note that all measures are in meters, and we omit the measures here for clarity of the presentation.

C.1 ROBOTIC MANIPULATION:

In object manipulation tasks, a 9-DOF Jaco robotic arm1 is used as an agent and a cube with the side length 0.06 m is used as a target object. We follow the tasks and environment settings proposed in Ghosh et al. (2018) . The observation consists of the position of the base of the Jaco arm, joint angles, angular velocities as well as the position, rotation, velocity, and angular velocity of the cube. The action space is a torque control on 9 joints.

C.1.1 REWARD DESIGN AND TERMINATION CONDITION:

Picking up: In the Picking up task, the position of the box is randomly initialized within a square region of size 0.1 m × 0.1 m with a center (0.5, 0.2). There is an initial guide reward to guide the arm to the box. There is also an over reward to guide the hand directly over the box. When the arm is not picking up the box, there is a pick reward to incentivize the arm to pick the box up. There is an additional hold reward that makes the arm hold the box in place after picking up. Finally, there is a success reward given after the arm has held the box for 50 frames. The success reward is scaled with number of timesteps to encourage the arm to succeed as quickly as possible. Tossing: The box is randomly initialized on the ground at (0.4, 0.3, 0.05) within a 0.005 × 0.005 square region. A guide reward is given to guide the arm to the top of the box. A pick reward is then given to lift the box up to a specified release height. A release reward is given if the box is no longer in the hand. A stable reward is given to minimize variation in the box's x and y direction. An up reward is given while the ball is traveling upwards in air, up until the box hits a specified z height. Finally, a success reward +100 is given based on the landing position of the box and the specified landing position.

Hitting: The box is randomly initialized overhead the arm at (0.4, 0.3, 1.2) within a 0.005 × 0.005 m square region. The box falls and the arm is given a hit reward +10 for hitting the box. Once the box has been hit, a target reward is given based on how close the box is to the target.

Repetitive picking up: The Repetitive picking up task has two reward variants. The sparse version gives a reward +1 for every successful pick. The dense reward version gives a guide reward to the box after each successful pick following the reward for the Picking up task.

Repetitive catching: The Repetitive catching task gives a reward +1 for every successful catch. For dense reward, it uses the same reward function with that of the Catching task.

Serve: The Serve task gives a toss reward +1 for a successful toss and a target reward +1 for successfully hitting the target. The dense reward setting provides the Tossing and Hitting reward according to box position.

C.2 LOCOMOTION:

A 9-DOF bipedal planar walker is used for simulating locomotion tasks. The observation consists of the position and velocity of the torso, joint angles, and angular velocities. The action space is torque control on the 6 joints. tasks. In comparison, our proposed method learns from sparse reward by re-using prior knowledge and doesn't require reward shaping.

Obstacle Course: The Obstacle Course environment replaces two curbs in Hurdle with a ceiling of height 1.0 and length 3. The sparse reward +1 is given when the agent jumps over a curb or passes through a ceiling (i.e. pass a point 1.5 after a curb or a ceiling). The dense reward is alternating between Jumping before the curb and Crawling before the ceiling.

C.2.2 TERMINATION SIGNAL:

Locomotion tasks except Crawling fail if h < 0.8 and Crawling fails if h < 0.3. Forward and Backward tasks are considered as success when the walker reaches to the target or 5 in front of obstacles. Balancing task is considered successful when the agent does not fail for 50 timesteps. The agent succeeds on Jumping and Crawling if the agent passes the obstacles by a distance of 1.5.

:

http://www.mujoco.org/forum/index.php?resources/kinova-arms.12/

Figures and Tables:

t e x i t s h a 1 _ b a s e 6 4 = " 3 u b T l A h 8 r 0 0 l 6 s 7 9 w

c b m U 7 I h e K s n r 0 P r q u 5 Z f r i u N G p 5 H E U 4 g 3 O o g g c 3 0 I B 7 a I I P D D g 8 w y u 8 O d J 5 c d 6 d j 2 V r w c l n T u G P n M 8 f 3 t a N / A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R R a 6 u 0 / Z J 3 k N M g H e d j m 9 Z A H a Z s 4

c b m U 7 I h e K s n r 0 P r q u 5 Z f r i u N G p 5 H E U 4 g 3 O o g g c 3 0 I B 7 a I I P D D g 8 w y u 8 O d J 5 c d 6 d j 2 V r w c l n T u G P n M 8 f 3 t a N / A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R R a 6 u 0 / Z J 3 k N M g H e d j m 9 Z A H a Z s 4

c b m U 7 I h e K s n r 0 P r q u 5 Z f r i u N G p 5 H E U 4 g 3 O o g g c 3 0 I B 7 a I I P D D g 8 w y u 8 O d J 5 c d 6 d j 2 V r w c l n T u G P n M 8 f 3 t a N / A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " R R a 6 u 0 / Z J 3 k N M g H e d j m 9 Z A H a Z s 4

Figure3: Training of transition policies and proximity predictors. After executing a primitive policy, a previously performed transition trajectory is labeled and added to a replay buffer based on the execution success. A proximity predictor is trained on states sampled from the two buffers to output the proximity to the initiation set. The predicted proximity serves as a reward to encourage the transition policy to move toward good initial states for the corresponding primitive policy.

Figure 4: Tasks and success count curves of our model (blue), TRPO (purple), PPO (magenta), and transition policies (TP) trained on task reward (green) and sparse proximity reward (yellow). Our model achieves the best performance and convergence time. Note that TRPO and PPO are trained 5 times longer than ours with dense rewards since TRPO and PPO do not have primitive skills and learn from scratch. In the success count curves, different temporal scales are used for TRPO and PPO (bottom x-axis) and ours (top x-axis).

e H M y 5 8 V 5 d z 6 W r S W n m D m F P 3I + f w C 7 d Z O o < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " d 1 O y u n v e 2 J K 9 E y N e d / H V D Z S n R o Q = " > A A A B + X i c b Z B N S 8 N A E I Y n 9 a v W r 6 h H L 8 E i e C q J C H o s e P F Y w X 5 A G 8 p m u 2 2 X b n b D 7 q R Y Q v + J F w + K e P W f e P P f u G l z 0 N Y X F h 7 e m W F m 3 y g R 3 K D v f z u l j c 2 t 7 Z 3 y b

Figure 5: Average transition length and average proximity reward of transition trajectories over training on Manipulation (left) and Patrol (right).

Figure 6: Visualization of transition trajectories of (a) Repetitive picking up and (b) Patrol. TOP AND BOTTOM ROWS: contain rendered frames of transition trajectories. MIDDLE ROW: contains states extracted from each primitive skill execution projected onto PCA space. The dots connected with lines are extracted from the same transition trajectory, where the marker color indicates the proximity prediction P (s). A higher P (s) value indicates proximity to states suitable for initializing the next primitive skill. LEFT: two picking up transition trajectories demonstrate that the transition policy learns to navigate from terminate states s 0 and s 1 to t 0 and t 1 . RIGHT: the forward to balance transition moves between the forward and balance state distributions and the balance to backward transition moves from the balancing states close to the backward states.

Figure 7: Success count curves of our model with exponentially discounted proximity function and linearly discounted proximity function over training on course (left) and Repetitive catching (right).

R(s) = λ guide •1 Box not picked and Box on ground +λ pick •1 Box in hand and not picked +λ hold •1 Box picked and near hold point λ guide = 2, λ pick = 100, λ hold = 0.1Catching: The position of the box is initialized at (0, 2.0, 1.5) and the directional force of size 110 is applied to throw the box toward the agent with randomness (0.1 m × 0.1 m).

(s) = 1 Box in air and Box within 0.06 of Jaco end-effector

Success count for robotic manipulation, comparing our method against baselines with or without transition policies (TP). Our method achieves the best performance over both RL baselines and the ablated variants. Each entry in the table represents average success count and standard deviation over 50 runs with 3 random seeds.

Table content: <table><tr><td/><td colspan="3">Reward Repetitive picking up Repetitive catching</td><td>Serve</td></tr><tr><td>TRPO</td><td>dense</td><td>0.69 ± 0.46</td><td>4.54 ± 1.21</td><td>0.32 ± 0.47</td></tr><tr><td>PPO</td><td>dense</td><td>0.95 ± 0.53</td><td>4.26 ± 1.63</td><td>0.00 ± 0.00</td></tr><tr><td>Without TP</td><td>sparse</td><td>0.99 ± 0.08</td><td>1.00 ± 0.00</td><td>0.11 ± 0.32</td></tr><tr><td>TP-Task</td><td>sparse</td><td>0.99 ± 0.08</td><td>4.87 ± 0.58</td><td>0.05 ± 0.21</td></tr><tr><td>TP-Sparse</td><td>sparse</td><td>1.52 ± 1.12</td><td>4.88 ± 0.59</td><td>0.92 ± 0.27</td></tr><tr><td colspan="2">TP-Dense (ours) sparse</td><td>4.84 ± 0.63</td><td>4.97 ± 0.33</td><td>0.92 ± 0.27</td></tr></table>

Success count for locomotion, comparing our method against baselines with or without transition policies (TP). Our method outperforms all baselines in Patrol and Obstacle course. In Hurdle, the reward function for TRPO was extensively engineered, which is not directly comparable to our method. Our method outperforms baselines learning from sparse reward, showing the effectiveness of the proposed proximity predictor. Each entry in the table represents average success count and standard deviation over 50 runs with 3 random seeds.

Table content: <table><tr><td/><td>Reward</td><td>Patrol</td><td>Hurdle</td><td>Obstacle course</td></tr><tr><td>TRPO</td><td>dense</td><td colspan="2">1.37 ± 0.52 4.13 ± 1.54</td><td>0.98 ± 1.09</td></tr><tr><td>PPO</td><td>dense</td><td colspan="2">1.53 ± 0.53 2.87 ± 1.92</td><td>0.85 ± 1.07</td></tr><tr><td>Without TP</td><td colspan="3">sparse 1.02 ± 0.14 0.49 ± 0.75</td><td>0.72 ± 0.72</td></tr><tr><td>TP-Task</td><td colspan="3">sparse 1.69 ± 0.63 1.73 ± 1.28</td><td>1.08 ± 0.78</td></tr><tr><td>TP-Sparse</td><td colspan="3">sparse 2.51 ± 1.26 1.47 ± 1.53</td><td>1.32 ± 0.99</td></tr><tr><td colspan="4">TP-Dense (Ours) sparse 3.33 ± 1.38 3.14 ± 1.69*</td><td>1.90 ± 1.45</td></tr></table>

Input: Primitive polices {π p1 , ..., π pn }. 2: Initialize success buffers {B S 1 , ..., B S n } with successful trajectories of primitive policies. 3: Initialize failure buffers {B F 1 , ..., B F n }. 4: Randomly initialize parameters of transition policies {φ 1 , ..., φ n } and proximity predictors {ω 1 , ..., ω n }. Initialize rollout buffers {R 1 , ..., R n }. Meta policy π meta , primitive policies {π p1 , ..., π pn }, transition policies {π φ1 , ..., π φn }, and proximity predictors {P ω1 , ..., P ωn }. 2: Initialize an episode and receive initial state s 0 . 3: t ← 0 4: while episode is not terminated do t , τ trans ∼ π φc (s t ) Terminate the transition policy if τ trans = terminate. t+1 , τ env ← ENV(s t , a t ) t ← P ωc (s t+1 ) -P ωc (s t ) Store (s t , a t , r t , τ env , s t+1 ) in B t , τ pc ∼ π pc (s t ) Terminate the primitive policy if τ pc = continue. t+1 , τ env ← ENV(s t , a t )

Table content: <table><tr><td colspan="2">Algorithm 1 TRAIN</td></tr><tr><td colspan="2">15: repeat</td></tr><tr><td>6:</td><td/></tr><tr><td>7:</td><td>Collect trajectories using ROLLOUT.</td></tr><tr><td>8:</td><td>for i = 1 to n do</td></tr><tr><td>9: 10:</td><td>Update P ωi to minimize Equation (1) using B S i and B F i . Update π φi to maximize Equation (2) using R i .</td></tr><tr><td>11:</td><td>end for</td></tr><tr><td colspan="2">12: until convergence</td></tr><tr><td colspan="2">Algorithm 2 ROLLOUT</td></tr><tr><td>5:</td><td>c ∼ π meta (s t )</td></tr><tr><td>6:</td><td>Initialize a rollout buffer B.</td></tr><tr><td>9:</td><td/></tr><tr><td>11:</td><td/></tr><tr><td>12:</td><td/></tr><tr><td>13:</td><td>t ← t + 1</td></tr><tr><td>14:</td><td>end while</td></tr><tr><td>15:</td><td>while episode is not terminated do</td></tr><tr><td>16:</td><td/></tr><tr><td>17:</td><td/></tr><tr><td>19:</td><td>t ← t + 1</td></tr><tr><td>20:</td><td>end while</td></tr><tr><td>21:</td><td/></tr></table>

Peer review (iclr2019):
 ** Summary ** The authors propose a new training scheme with a learned auxiliary reward function to optimise transition policies, i.e. policies that connect the ending state of a previous macro action/option with good initiation states of the following macro action/option. ** Quality & Clarity ** The paper is well written and features an extensive set of experiments. ** Originality ** I am not aware of similar work and believe the idea is novel. ** Significance ** Several recent papers have proposed to approach the topic of learning hierarchical policies not by training the hierarchy end-to-end, but by first learning useful individual behavioural patterns (e.g. skills) which then later can be used and sequentially chained together by higher-level policies. I believe the here presented work can be quite helpful to do so as the individual skills are not optimised for smooth composition and are therefore likely to fail when naively used sequentially.