Inducing Reusable Skills From Demonstrations with Option-Controller Network

1 INTRODUCTION . Acquiring primitive skills from demonstrations and reusing them to solve a novel long-horizon task is a hallmark in human intelligence . For example , after learning the necessary skills ( e.g. , steering wheel , changing lanes ) at a driving school , one could be capable of driving across the country by recombining the learned skills . In hierarchical reinforcement learning , different temporal abstractions ( Sutton et al. , 1999 ; Dietterich , 2000 ; Parr & Russell , 1998 ; Nakanishi et al. , 2004 ) are proposed to achieve structured exploration and transfer to a long-horizon task . However , when learning from scratch , these pure HRL methods share an exploration problem : it takes a significant amount of samples for random walk to induce a good temporal abstraction that leads to positive rewards at the beginning of training . To circumvent this issue , recent works ( Le et al. , 2018 ; Levy et al. , 2018 ; Gupta et al. , 2019 ; Jiang et al. , 2019 ) have focused on learning useful skills in a pretraining phase first , and then reusing these skills when finetuning with HRL in the new environment . However , these methods either assume the existence of goal-conditioned policies or require intensive interaction with environments , which limits the practical values of these approaches . One general approach is to leverage the additional unstructured demonstrations during pretraining , e.g. , compILE ( Kipf et al. , 2019 ) pretrains a VAE ( Kingma & Welling , 2013 ) on the demonstrations and uses an action decoder for finetuning . Our work is in this line of research . In this paper , we propose Option-Control Network ( OCN ) . Inspired by the option framework ( Sutton et al. , 1999 ) , OCN is a two-level recurrent policy network including a high-level controller and a pool of low-level options . At each time step , a selected low-level option outputs an action and a termination probability , and the high-level controller selects a new option whenever the old option is terminated . Inspired by Lu et al . ( 2021 ) , we enforce a special recurrent hidden state updating rule to enforce the hierarchical constraint between the controller and the options so that the high-level controller is updated less frequently than the low-level options while keeping the model end-to-end differentiable . As shown in Figure 1 , OCN can jointly learn options and controllers with multitask behavior cloning from unstructured demonstrations . When given a new task , one could perform HRL finetuning by re-initializing the controller and freezing the options . This enables our model to generalize combinatorially to unforeseen conjunctions ( Denil et al. , 2017 ) . Unlike previous works , our method does not require generative models ( Eysenbach et al. , 2018 ) , goal-conditioned policies ( Gupta et al. , 2019 ) , pre-specified policy sketch ( Shiarlis et al. , 2018 ) or constraints on the number of segments ( Kipf et al. , 2019 ) , making our approach conceptually simple and general . We perform experiments in Craft ( Andreas et al. , 2017 ) , a grid-world environment focusing on navigation and collecting objects , and Dial ( Shiarlis et al. , 2018 ) , a robotic setting where a JACO 6DoF manipulator interact with a large number pad . Our results show that with unstructured demonstrations , OCN can jointly learn to segment the trajectories into meaningful skills as well as model this rich set of skills with our pool of low-level options . During HRL finetuning , we show that OCN achieves better performance in more complex long-horizon tasks with either sparse or dense reward compared with existing baselines . We also provide further visualization to show the discovered options are reused during finetuning . Our contributions can be concluded as : 1 . We propose Option-Controller Network , a bi-level recurrent policy network that uses a special recurrent hidden state updating rule to enforce the hierarchical constraints . 2 . We show that the OCN can discover the optimum options decomposition from unstructured demonstration via multitasking behavior cloning . 3 . We also demonstrate that the learned options can be coupled with a new controller to solve an unseen long-horizon task via interaction with the environment . 2 RELATED WORK . Learning to solve temporally extended tasks is an important question for Hierarchical Reinforcement Learning ( HRL ) , including option frameworks ( Sutton et al. , 1999 ) , HAM ( Parr & Russell , 1998 ) and max-Q ( Dietterich , 2000 ) . With the popularity of neural nets , recent works propose to use a bi-level neural network such as option critics ( Bacon et al. , 2017 ) , feudal networks ( Vezhnevets et al. , 2017 ) , generative models with latents ( Nachum et al. , 2018 ) , and modulated networks ( Pashevich et al. , 2018 ) . These models can be furthered combined with hindsight memory ( Levy et al. , 2018 ) to increase the sample efficiency . Our work can also be viewed as designing a specific neural architecture for HRL . However , as discussed in Section 1 , a pure HRL method suffers from serious exploration challenges when learning from scratch ( Gupta et al. , 2019 ) . A general approach to tackle this problem is to introduce a pretraining phase to “ warm up ” the policy . Recent works propose to pretrain the policy with an intrinsic diversity reward ( Eysenbach et al. , 2018 ) or language abstraction ( Jiang et al. , 2019 ) , which is shown to be useful in the HRL . However , assuming access to an environment in the pretrain phase might be infeasible in many tasks . A principal approach is to leverage the additional unstructured expert demonstrations and performs imitation learning during pretraining . Our work can be viewed as solving the “ cold start ” problem for the option framework ( Sutton et al. , 1999 ) . Recent works build upon this “ imitation - finetune ” paradigm . With the prevalence of goal-conditioned policies ( Schaul et al. , 2015 ; Kaelbling , 1993 ; Levy et al. , 2018 ) in robotics , these methods leverage demonstrations with relabelling technique to pretrain the low-level policy ( Gupta et al. , 2019 ) or a generative model ( Lynch et al. , 2020 ) . However , they exploit the fact that the ending state of a trajectory segment can be described as a point in the goal space . Hence it is difficult to apply them beyond goal-conditioned policies . CompILE ( Kipf et al. , 2019 ) treats the segment boundaries as latent variables , and their model can be trained end-to-end with soft trajectory masking . However , CompILE requires specifying the number of segments , while OCN only requires that the number of options to be large enough ( Appendix A.5 . Following a similar VAE setting , SPiRL ( Pertsch et al. , 2020 ) includes a skill prior and a skill autoencoder . They encoder skills into latent variables and use a skill prior to generate high-level instructions – a sequence of latent variables . However , SPiRL define skills as a sequence of actions with a fix horizon H , which could prevent the model from learning skills with clear semantics . Modular policy networks ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ) are also used in this paradigm , where each subtask corresponds to a single modular policy . However , in this setting , the demonstration needs to be segmented beforehand , which requires additional human labor . On the contrary , our work focused on using unstructured demonstrations . OptionGAN ( Henderson et al. , 2018 ) proposes a Mixture-of-Expert ( MoE ) formulation and performs IRL on the demonstration . However , without an explicit termination function , the learnt expert networks do not provide time-extended actions for the high-level controller . As a result , this method still suffers from problems of exploration with sparse rewards ( as also seen in our experimental comparison with an MoE baseline ) . Extracting meaningful trajectory segments from the unstructured demonstration is the focus of Hierarchical Imitation Learning ( HIL ) . These works can be summarized as finding the optimal behavior hierarchy so that the behavior can be better predicted ( Solway et al. , 2014 ) . DDO ( Fox et al. , 2017 ) proposes an iterative EM-like algorithm to discover multiple levels of options , and it is applied in the continuous action space ( Krishnan et al. , 2017 ) and program modelling ( Fox et al. , 2018 ) . VALOR ( Achiam et al. , 2018 ) extends this idea by incorporating powerful inference methods like VAE ( Kingma & Welling , 2013 ) . Directed-Info GAIL ( Sharma et al. , 2018 ) extracts meaning segments by maximizing the mutual information between the subtask latent variables and the generated trajectory . Ordered Memory Policy Network ( OMPN ) ( Lu et al. , 2021 ) proposes a hierarchical inductive bias to infer the skill boundaries . The above works mainly focus on skill extraction , so it is unclear how to use the segmented skills for RL finetuning . Although OCN shares a similar inductive bias with OMPN , OCN replaces the continuous hidden states communication with a softmax distribution over multiple low-level modules ( options ) . This enables OCN to model different subtasks with different options and to effectively reuse them in a new task . 3 METHODOLOGY . An Option-Controller Network ( OCN ) includes a set of N options { o1 , ... , oN } and a controller c. As shown in figure 1 , the OCN starts by using the controller to choose an option to execute the first subtask . Once the subtask is done , the controller will choose another option to execute the second subtask , and so on , until the goal of the task is achieved . OCN shares a similar inductive bias with Ordered Memory Policy Network ( OMPN ) ( Lu et al. , 2021 ) , that the lower-level components ( options ) execute more frequently than the higher-level components ( controllers ) . The inductive bias enables OCN to induce the temporal hierarchical structure of unstructured demonstrations . 3.1 OPTION AND CONTROLLER . Option As shown in the middle of Figure 2 , an option oi models a skill that can solve one specific subtask , for example get wood , get iron or make at workbench . It can be described as : poi , t , h o i , t , ei , t = oi ( xt , h o i , t−1 ) ( 1 ) where xt is the observation at time step t , and hoi , t−1 is the hidden state of the respective option at time step t− 1 ; hoi , t is the hidden state of oi at time step t ; ei , t is a scalar between 0 and 1 , represents the probability that the current option is done ; poi , t is a distribution of actions , including move up , move left and use . These actions are the smallest elementary operations that an agent can execute . During the execution of an option , if probability ei , t is 0 , the option will keep executing the current subtask ; if ei , t is 1 , the option will stop the execution and return to the controller for the next subtask . In our work , each option maintains a separate set of parameters . Controller As shown at the top of Figure 2 , a controller c models a higher level task , like make bed , make axe , or make bridge . Each of these tasks can be decompose to a sequence of subtasks . For example , make bridge can be decompose to 3 steps : 1 ) get iron , 2 ) get wood , 3 ) make at factory . Thus a controller can also be represented as : pct , h c t , e c t = c ( xt , h c t−1 ) ( 2 ) where pct is a distribution over the set of options { oi } , hct is the hidden state for controller , ect is the probability that the current task is done . In this OCN architecture , we don ’ t need the ect , since the environment will provide signal ( reward ) once the task is done . However , OCN can be easily expanded to a multi-level model . In this multi-levels model , a set of multiple controllers become options for a higher-level controller , and their respective tasks become subtasks for a more complicated task . Cell Network In OCN , options and controllers share the same format for input and output . Thus , we parameterize them with the same neural network architecture . To model the policy of controllers and options , we proposed the following cell network : ĥt = MLP ( [ xt , ht−1 ] ) ( 3 ) pt = softmax ( Wactĥt + bact ) ( 4 ) ht = tanh ( Whidĥt + bhid ) ( 5 ) et = sigmoid ( wendĥt + bend ) ( 6 ) xt is the raw observation , the shape of the vector depends on the environment . ht−1 is the recurrent hidden state of size dhid , it allows the model to remember important information from previous time steps . MLP is a multi-layer neural network of Depth lMLP and hidden size dMLP . We use tanh as activation function for MLP . ĥt is a vector of size dMLP . Wact is a matrix of size nact × dMLP , where nact is number of actions . Whid is a matrix of size dhid × dMLP . wend is a vector of size dMLP . Following the fast and slow learning idea proposed in Madan et al . ( 2021 ) , we introduce a temperature term T to controller ’ s softmax function : pct = softmax ( Wactĥt + bact T ) ( 7 ) A large temperature T allows the option to output smoother distribution at the beginning of training . It also reduces the scale of gradient backpropagated into the controller . This results in the controller changes and updates slower than options . We found T makes OCN become more stable in imitation learning and converge to a better hierarchical structure .

Summary:  This work targets the issues in learning from demonstration, in order to achieve the ability of skill reuse/ transferability and improve the sample efficiency. To address the issues, the authors propose an option-control network (OCN), including a high-level controller and a pool of low-level options, which is an "imitation-finetune" paradigm as well. Finally, the proposed method is evaluated on two domains, Craft (discrete action space) and Dial (continuous action space), with results demonstrating its effectiveness.