Composing Complex Skills by Learning Transition Policies

1 INTRODUCTION . While humans are capable of learning complex tasks by reusing previously learned skills , composing and mastering complex skills are not as trivial as sequentially executing those acquired skills . Instead , it requires a smooth transition between skills since the final pose of one skill may not be appropriate to initiate the following one . For example , scoring in basketball with a quick shot after receiving a ball can be decomposed into catching and shooting . However , it is still difficult for beginners who have learned to catch passes and statically shoot . To master this skill , players must practice adjusting their footwork and body into a comfortable shooting pose after catching a pass . Can machines similarly learn new and complex tasks by reusing acquired skills and learning transitions between them ? Learning to perform composite and long-term tasks from scratch requires extensive exploration and sophisticated reward design , which can introduce undesired behaviors ( Riedmiller et al. , 2018 ) . Thus , instead of employing intricate reward functions and learning from scratch , modular methods sequentially execute acquired skills with a rule-based meta-policy , enabling machines to solve complicated tasks ( Pastor et al. , 2009 ; Mülling et al. , 2013 ; Andreas et al. , 2017 ) . These modular approaches assume that a task can be clearly decomposed into several subtasks which are smoothly connected to each other . In other words , an ending state of one subtask falls within the set of starting states , initiation set , of the next subtask ( Sutton et al. , 1999 ) . However , this assumption does not hold in many continuous control problems where a given skill may be executed in starting states not considered during training or designing and thus , fail to achieve its goal . To bridge the gap between skills , we propose a transition policy which learns to smoothly navigate from an ending state of a skill to suitable initial states of the following skill , as illustrated in Figure 1 . However , learning a transition policy between skills without reward shaping is difficult as the only available learning signal is the sparse reward for the successful execution of the next skill . Sparse success/failure reward is challenging to learn from due to the temporal credit assignment problem ( Sutton , 1984 ) and the lack of information from failing trajectories . To alleviate these problems , we propose a proximity predictor which outputs the proximity to the initiation set of the next skill and acts as a dense reward function for the transition policy . ∗Equal contribution FINAL The main contributions of this paper include ( 1 ) the concept of learning transition policies to smoothly connect primitive skills ; ( 2 ) a novel modular framework with transition policies that is able to compose complex skills by reusing existing skills ; and ( 3 ) a joint training algorithm with the proximity predictor specifically designed for efficiently training transition policies . This framework is suited for learning complex skills that require sequential execution of acquired primitive skills , which are common for humans yet relatively unexplored in robot learning . Our experiments on simulated environments demonstrate that employing transition policies solves complex continuous control tasks which traditional policy gradient methods struggle at . 2 RELATED WORK . Learning continuous control of diverse behaviors in locomotion ( Merel et al. , 2017 ; Heess et al. , 2017 ; Peng et al. , 2017 ) and robotic manipulation ( Ghosh et al. , 2018 ) is an active research area in reinforcement learning ( RL ) . While some complex tasks can be solved through extensive reward engineering ( Ng et al. , 1999 ) , undesired behaviors often emerge ( Riedmiller et al. , 2018 ) when tasks require several different primitive skills . Moreover , training complex skills from scratch is not computationally practical . Real-world tasks often require diverse behaviors and longer temporal dependencies . In hierarchical reinforcement learning , the option framework ( Sutton et al. , 1999 ) learns meta actions ( options ) , a series of primitive actions over a period of time . Typically , a hierarchical reinforcement learning framework consists of two components : a high-level meta-controller and low-level controllers . A meta-controller determines the order of subtasks to achieve the final goal and chooses corresponding low-level controllers that generate a sequence of primitive actions . Unsupervised approaches to discover meta actions have been proposed ( Schmidhuber , 1990 ; Daniel et al. , 2016 ; Bacon et al. , 2017 ; Vezhnevets et al. , 2017 ; Dilokthanakul et al. , 2017 ; Levy et al. , 2017 ; Frans et al. , 2018 ; Co-Reyes et al. , 2018 ; Mao et al. , 2018 ) . However , to deal with more complex tasks , additional supervision signals ( Andreas et al. , 2017 ; Merel et al. , 2017 ; Shu et al. , 2018 ) or pre-defined lowlevel controllers ( Kulkarni et al. , 2016 ; Oh et al. , 2017 ) are required . To exploit pre-trained modules as low-level controllers , neural module networks ( Andreas et al. , 2016 ) have been proposed , which construct a new network dedicated to a given query using a collection of reusable modules . In the RL domain , a meta-controller is trained to follow instructions ( Oh et al. , 2017 ) and demonstrations ( Xu et al. , 2017 ) , and support multi-level hierarchies ( Gudimella et al. , 2017 ) . In the robotics domain , Pastor et al . ( 2009 ) ; Kober et al . ( 2010 ) ; Mülling et al . ( 2013 ) have proposed a modular approach that learns table tennis by selecting appropriate low-level controllers . On the other hand , Andreas et al . ( 2017 ) ; Frans et al . ( 2018 ) learn abstract skills while experiencing a distribution of tasks and then solve a new task with the learned primitive skills . However , these modular approaches result in undefined behavior when two skills are not smoothly connected . Our proposed framework aims to bridge this gap by training transition policies in a model-free manner to navigate the agent from unseen states for following skills to suitable initial states . Published as a conference paper at ICLR 2019 FINAL Deep RL techniques for continuous control demand dense reward signals ; otherwise , they suffer from long training time . Instead of manual reward shaping for denser reward , adversarial reinforcement learning ( Ho & Ermon , 2016 ; Merel et al. , 2017 ; Wang et al. , 2017 ; Bahdanau et al. , 2019 ) employs a discriminator which learns to judge the state or the policy , and the policy takes as rewards the output of the discriminator . While those methods assume ground truth trajectories or goal states are given , our method collects both success and failure trajectories online to train proximity predictors which provide rewards for transition policies . 3 APPROACH . In this paper , we address the problem of solving a complex task that requires sequential composition of primitive skills given only sparse and binary rewards ( i.e . subtask completion reward ) . The sequential execution of primitive skills fails when two consecutive skills are not smoothly connected . We propose a modular framework with transition policies that learn to make transition between one policy to the subsequent policy , and therefore , can exploit the given primitive skills to compose complex skills . To accelerate training of transition policies , additional networks , proximity predictors , are jointly trained to provide proximity rewards as intermediate feedback to transition policies . In Section 3.2 , we describe our framework in details . Next , in Section 3.3 , we elaborate how transition policies are efficiently trained with induced proximity reward . 3.1 PRELIMINARIES . We formulate our problem as a Markov decision process defined by a tuple { S , A , T , R , ρ , γ } of states , actions , transition probability , reward , initial state distribution , and discount factor . An action distribution of an agent is represented as a policy πθ ( at|st ) , where st ∈ S is a state , at ∈ A is an action at time t , and θ are the parameters of the policy . An initial state s0 is randomly sampled from ρ , and then , an agent iteratively takes an action at sampled from a policy πθ ( at|st ) and receives a reward rt until the episode ends . The performance of the agent is evaluated based on a discounted return R = ∑T−1 t=0 γ trt , where T is the episode horizon . 3.2 MODULAR FRAMEWORK WITH TRANSITION POLICIES . To learn a new task given primitive skills { p1 , p2 , . . . , pn } , we design a modular framework that consists of the following components : a meta-policy , primitive policies , and transition policies . The meta-policy chooses a primitive skill pc to execute at the beginning and whenever the primitive skill is terminated . Prior to running pc , the transition policy for pc is executed to bring the current state to a plausible initial state for pc , and therefore , pc can be successfully performed . This procedure is repeated to compose complex skills as illustrated in Figure 2 and Algorithm 2 . FINAL We denote the meta-policy as πmeta ( pc|s ) , where c ∈ [ 1 , n ] is a primitive policy index . The observation of the meta-policy contains the low-level information of primitives and task specifications indicating high-level goals ( e.g . moving direction and target object position ) . For example , a walking primitive only takes joint information as observation while the meta-policy additionally takes target direction . In this paper , we use a rule-based meta-policy and focus on transitioning between consecutive primitive policies . Once a primitive skill pc is chosen to be executed , the agent generates an action at ∼ πpc ( a|st ) based on the current state st . Note that we did not differentiate state spaces for primitive polices because of the simplicity of notations ( e.g . the observation of the jumping primitive contains a distance to a curb while that of the walking primitive only has joint pose and velocities ) . Every primitive policy is required to generate termination signals τpc ∈ { continue , success , fail } to indicate policy completion and whether it believes the execution is successful or not . While our method is agnostic to the form of primitive policies ( e.g . rule-based , inverse kinematics ) , we consider the case of a pre-trained neural network in this paper . For smooth transitions between primitive policies , we add a transition policy πφc ( a|s ) before executing primitive skill pc , which guides an agent to pc ’ s initiation set , where φc is the parameters of the transition policy for pc . Note that the transition policy for pc is shared across different preceding primitive policies since a successful transition is defined by the success of the following primitive skill pc . For brevity of notation , we omit the primitive policy index c in the following equations where unambiguous . The transition policy ’ s state and action space are the same as the primitive policy ’ s . The transition policy also learns a termination signal τtrans which indicates transition termination to successfully initiate pc . Our framework contains one transition policy for each primitive skill , in total n transition policies { πφ1 , πφ2 , . . . , πφn } .

Summary: The paper presents a method for learning policies for transitioning from one task to another with the goal of completing complex tasks. In the heart of the method is state proximity estimator, which measures the distance between states in the originator and destination tasks. This estimator is used in the reward for the transition policy. The method is evaluated on number of MojoCo tasks, including locomotion and manipulation.