Inducing Reusable Skills From Demonstrations with Option-Controller Network

1 INTRODUCTION . Acquiring primitive skills from demonstrations and reusing them to solve a novel long-horizon task is a hallmark in human intelligence . For example , after learning the necessary skills ( e.g. , steering wheel , changing lanes ) at a driving school , one could be capable of driving across the country by recombining the learned skills . In hierarchical reinforcement learning , different temporal abstractions ( Sutton et al. , 1999 ; Dietterich , 2000 ; Parr & Russell , 1998 ; Nakanishi et al. , 2004 ) are proposed to achieve structured exploration and transfer to a long-horizon task . However , when learning from scratch , these pure HRL methods share an exploration problem : it takes a significant amount of samples for random walk to induce a good temporal abstraction that leads to positive rewards at the beginning of training . To circumvent this issue , recent works ( Le et al. , 2018 ; Levy et al. , 2018 ; Gupta et al. , 2019 ; Jiang et al. , 2019 ) have focused on learning useful skills in a pretraining phase first , and then reusing these skills when finetuning with HRL in the new environment . However , these methods either assume the existence of goal-conditioned policies or require intensive interaction with environments , which limits the practical values of these approaches . One general approach is to leverage the additional unstructured demonstrations during pretraining , e.g. , compILE ( Kipf et al. , 2019 ) pretrains a VAE ( Kingma & Welling , 2013 ) on the demonstrations and uses an action decoder for finetuning . Our work is in this line of research . In this paper , we propose Option-Control Network ( OCN ) . Inspired by the option framework ( Sutton et al. , 1999 ) , OCN is a two-level recurrent policy network including a high-level controller and a pool of low-level options . At each time step , a selected low-level option outputs an action and a termination probability , and the high-level controller selects a new option whenever the old option is terminated . Inspired by Lu et al . ( 2021 ) , we enforce a special recurrent hidden state updating rule to enforce the hierarchical constraint between the controller and the options so that the high-level controller is updated less frequently than the low-level options while keeping the model end-to-end differentiable . As shown in Figure 1 , OCN can jointly learn options and controllers with multitask behavior cloning from unstructured demonstrations . When given a new task , one could perform HRL finetuning by re-initializing the controller and freezing the options . This enables our model to generalize combinatorially to unforeseen conjunctions ( Denil et al. , 2017 ) . Unlike previous works , our method does not require generative models ( Eysenbach et al. , 2018 ) , goal-conditioned policies ( Gupta et al. , 2019 ) , pre-specified policy sketch ( Shiarlis et al. , 2018 ) or constraints on the number of segments ( Kipf et al. , 2019 ) , making our approach conceptually simple and general . We perform experiments in Craft ( Andreas et al. , 2017 ) , a grid-world environment focusing on navigation and collecting objects , and Dial ( Shiarlis et al. , 2018 ) , a robotic setting where a JACO 6DoF manipulator interact with a large number pad . Our results show that with unstructured demonstrations , OCN can jointly learn to segment the trajectories into meaningful skills as well as model this rich set of skills with our pool of low-level options . During HRL finetuning , we show that OCN achieves better performance in more complex long-horizon tasks with either sparse or dense reward compared with existing baselines . We also provide further visualization to show the discovered options are reused during finetuning . Our contributions can be concluded as : 1 . We propose Option-Controller Network , a bi-level recurrent policy network that uses a special recurrent hidden state updating rule to enforce the hierarchical constraints . 2 . We show that the OCN can discover the optimum options decomposition from unstructured demonstration via multitasking behavior cloning . 3 . We also demonstrate that the learned options can be coupled with a new controller to solve an unseen long-horizon task via interaction with the environment . 2 RELATED WORK . Learning to solve temporally extended tasks is an important question for Hierarchical Reinforcement Learning ( HRL ) , including option frameworks ( Sutton et al. , 1999 ) , HAM ( Parr & Russell , 1998 ) and max-Q ( Dietterich , 2000 ) . With the popularity of neural nets , recent works propose to use a bi-level neural network such as option critics ( Bacon et al. , 2017 ) , feudal networks ( Vezhnevets et al. , 2017 ) , generative models with latents ( Nachum et al. , 2018 ) , and modulated networks ( Pashevich et al. , 2018 ) . These models can be furthered combined with hindsight memory ( Levy et al. , 2018 ) to increase the sample efficiency . Our work can also be viewed as designing a specific neural architecture for HRL . However , as discussed in Section 1 , a pure HRL method suffers from serious exploration challenges when learning from scratch ( Gupta et al. , 2019 ) . A general approach to tackle this problem is to introduce a pretraining phase to “ warm up ” the policy . Recent works propose to pretrain the policy with an intrinsic diversity reward ( Eysenbach et al. , 2018 ) or language abstraction ( Jiang et al. , 2019 ) , which is shown to be useful in the HRL . However , assuming access to an environment in the pretrain phase might be infeasible in many tasks . A principal approach is to leverage the additional unstructured expert demonstrations and performs imitation learning during pretraining . Our work can be viewed as solving the “ cold start ” problem for the option framework ( Sutton et al. , 1999 ) . Recent works build upon this “ imitation - finetune ” paradigm . With the prevalence of goal-conditioned policies ( Schaul et al. , 2015 ; Kaelbling , 1993 ; Levy et al. , 2018 ) in robotics , these methods leverage demonstrations with relabelling technique to pretrain the low-level policy ( Gupta et al. , 2019 ) or a generative model ( Lynch et al. , 2020 ) . However , they exploit the fact that the ending state of a trajectory segment can be described as a point in the goal space . Hence it is difficult to apply them beyond goal-conditioned policies . CompILE ( Kipf et al. , 2019 ) treats the segment boundaries as latent variables , and their model can be trained end-to-end with soft trajectory masking . However , CompILE requires specifying the number of segments , while OCN only requires that the number of options to be large enough ( Appendix A.5 . Following a similar VAE setting , SPiRL ( Pertsch et al. , 2020 ) includes a skill prior and a skill autoencoder . They encoder skills into latent variables and use a skill prior to generate high-level instructions – a sequence of latent variables . However , SPiRL define skills as a sequence of actions with a fix horizon H , which could prevent the model from learning skills with clear semantics . Modular policy networks ( Andreas et al. , 2017 ; Shiarlis et al. , 2018 ) are also used in this paradigm , where each subtask corresponds to a single modular policy . However , in this setting , the demonstration needs to be segmented beforehand , which requires additional human labor . On the contrary , our work focused on using unstructured demonstrations . OptionGAN ( Henderson et al. , 2018 ) proposes a Mixture-of-Expert ( MoE ) formulation and performs IRL on the demonstration . However , without an explicit termination function , the learnt expert networks do not provide time-extended actions for the high-level controller . As a result , this method still suffers from problems of exploration with sparse rewards ( as also seen in our experimental comparison with an MoE baseline ) . Extracting meaningful trajectory segments from the unstructured demonstration is the focus of Hierarchical Imitation Learning ( HIL ) . These works can be summarized as finding the optimal behavior hierarchy so that the behavior can be better predicted ( Solway et al. , 2014 ) . DDO ( Fox et al. , 2017 ) proposes an iterative EM-like algorithm to discover multiple levels of options , and it is applied in the continuous action space ( Krishnan et al. , 2017 ) and program modelling ( Fox et al. , 2018 ) . VALOR ( Achiam et al. , 2018 ) extends this idea by incorporating powerful inference methods like VAE ( Kingma & Welling , 2013 ) . Directed-Info GAIL ( Sharma et al. , 2018 ) extracts meaning segments by maximizing the mutual information between the subtask latent variables and the generated trajectory . Ordered Memory Policy Network ( OMPN ) ( Lu et al. , 2021 ) proposes a hierarchical inductive bias to infer the skill boundaries . The above works mainly focus on skill extraction , so it is unclear how to use the segmented skills for RL finetuning . Although OCN shares a similar inductive bias with OMPN , OCN replaces the continuous hidden states communication with a softmax distribution over multiple low-level modules ( options ) . This enables OCN to model different subtasks with different options and to effectively reuse them in a new task . 3 METHODOLOGY . An Option-Controller Network ( OCN ) includes a set of N options { o1 , ... , oN } and a controller c. As shown in figure 1 , the OCN starts by using the controller to choose an option to execute the first subtask . Once the subtask is done , the controller will choose another option to execute the second subtask , and so on , until the goal of the task is achieved . OCN shares a similar inductive bias with Ordered Memory Policy Network ( OMPN ) ( Lu et al. , 2021 ) , that the lower-level components ( options ) execute more frequently than the higher-level components ( controllers ) . The inductive bias enables OCN to induce the temporal hierarchical structure of unstructured demonstrations . 3.1 OPTION AND CONTROLLER . Option As shown in the middle of Figure 2 , an option oi models a skill that can solve one specific subtask , for example get wood , get iron or make at workbench . It can be described as : poi , t , h o i , t , ei , t = oi ( xt , h o i , t−1 ) ( 1 ) where xt is the observation at time step t , and hoi , t−1 is the hidden state of the respective option at time step t− 1 ; hoi , t is the hidden state of oi at time step t ; ei , t is a scalar between 0 and 1 , represents the probability that the current option is done ; poi , t is a distribution of actions , including move up , move left and use . These actions are the smallest elementary operations that an agent can execute . During the execution of an option , if probability ei , t is 0 , the option will keep executing the current subtask ; if ei , t is 1 , the option will stop the execution and return to the controller for the next subtask . In our work , each option maintains a separate set of parameters . Controller As shown at the top of Figure 2 , a controller c models a higher level task , like make bed , make axe , or make bridge . Each of these tasks can be decompose to a sequence of subtasks . For example , make bridge can be decompose to 3 steps : 1 ) get iron , 2 ) get wood , 3 ) make at factory . Thus a controller can also be represented as : pct , h c t , e c t = c ( xt , h c t−1 ) ( 2 ) where pct is a distribution over the set of options { oi } , hct is the hidden state for controller , ect is the probability that the current task is done . In this OCN architecture , we don ’ t need the ect , since the environment will provide signal ( reward ) once the task is done . However , OCN can be easily expanded to a multi-level model . In this multi-levels model , a set of multiple controllers become options for a higher-level controller , and their respective tasks become subtasks for a more complicated task . Cell Network In OCN , options and controllers share the same format for input and output . Thus , we parameterize them with the same neural network architecture . To model the policy of controllers and options , we proposed the following cell network : ĥt = MLP ( [ xt , ht−1 ] ) ( 3 ) pt = softmax ( Wactĥt + bact ) ( 4 ) ht = tanh ( Whidĥt + bhid ) ( 5 ) et = sigmoid ( wendĥt + bend ) ( 6 ) xt is the raw observation , the shape of the vector depends on the environment . ht−1 is the recurrent hidden state of size dhid , it allows the model to remember important information from previous time steps . MLP is a multi-layer neural network of Depth lMLP and hidden size dMLP . We use tanh as activation function for MLP . ĥt is a vector of size dMLP . Wact is a matrix of size nact × dMLP , where nact is number of actions . Whid is a matrix of size dhid × dMLP . wend is a vector of size dMLP . Following the fast and slow learning idea proposed in Madan et al . ( 2021 ) , we introduce a temperature term T to controller ’ s softmax function : pct = softmax ( Wactĥt + bact T ) ( 7 ) A large temperature T allows the option to output smoother distribution at the beginning of training . It also reduces the scale of gradient backpropagated into the controller . This results in the controller changes and updates slower than options . We found T makes OCN become more stable in imitation learning and converge to a better hierarchical structure .

Summary: This paper proposes to extract options from a dataset using an options framework with a recurrent controller and multiple recurrent option networks. During pre-training (i.e. skill extraction), the actions from the framework are computed by the weighted sum over options, which can be end-to-end differentiable, and trained using behavioral cloning. Once all options are extracted from the dataset, a new controller can be trained to solve a new task with the learned and frozen options via RL. The experimental results on the discrete CRAFT environment are impressive, outperforming all baselines. The results on the continuous DIAL environment show improved results over the baseline methods but the difference is marginal.