Discovery of Options via Meta-Learned Subgoals

Reinforcement learning ( RL ) problems involve learning about temporally-extended actions at multiple time scales . In RL , the options framework ( Sutton et al. , 1999 ) provides a well-defined formalisation for the notion of temporally-extended actions . Options that achieve specific subgoals can be useful in reinforcement learning ( RL ) in at least two ways : in model-based RL , they provide faster rates of convergence through longer-backups of value functions within planning updates ( Silver & Ciosek , 2012 ; Mann & Mannor , 2014 ; Brunskill & Li , 2014 ) , while in model-free RL , temporallyextended actions commit agents to intentional multi-step behaviours , which can translate into better exploration ( Machado & Bowling , 2016 ; Nachum et al. , 2019 ; Osband et al. , 2019 ) . We consider a scenario where an agent learns to solve a distribution over tasks . In such cases having carefully designed temporal abstractions can greatly reduce the overall sample complexity of learning for an RL agent that is trying to master those tasks . The agent can produce faster learning mainly by reusing those abstractions across multiple tasks ( Sutton et al. , 1999 ; Solway et al. , 2014 ) . Many recent approaches have empirically validated this by demonstrating that hand-designed temporal abstractions can often lead to improved learning performance on a variety of challenging multitask RL domains ( Imazeki & Maeno , 2003 ; Kulkarni et al. , 2016 ; Nachum et al. , 2018 ; Riedmiller et al. , 2018 ) . However , if the agent has abstractions that are not useful to a downstream task , it can significantly hurt the performance by making exploration harder ( Jong et al. , 2008 ) , emphasising the challenge involved in carefully hand-designing abstractions that are useful in general , across many domains and tasks . Thus , the automated discovery of temporal abstractions from experience without extensive domain-specific knowledge remains an important open problem for reducing sample complexity in RL . ∗Corresponding Author . Email : vveeriah @ umich.edu †Now at Sea AI Lab 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The main contribution of this work is our hierarchical agent architecture and an associated metagradient algorithm to discover temporally-extended actions in the form of options that can be reused across many tasks . Previously , meta-gradients have been successfully used for learning hyperparameters ( Xu et al. , 2018 ; Zahavy et al. , 2020 ) , intrinsic rewards ( Zheng et al. , 2018 , 2019 ; Rajendran et al. , 2019 ) , and auxiliary tasks ( Veeriah et al. , 2019 ) . Our work is the first to demonstrate that they can successfully learn rich parameterisations of temporal abstractions . Our starting point is the following hypothesis : If we could discover temporal abstractions useful across many training tasks , they would capture regularities across those task environments and have a higher likelihood of being useful and reusable in new , previously unseen tasks . To discover temporal abstractions useful across tasks , our system flexibly defines task-independent subgoals for options through separate discovered rewards and terminations , different for each option . We employ meta-gradients to discover the parameters of such option-rewards and terminations based on their utility across the many training tasks , so that the set of induced options is useful to a hierarchical agent trying to master all training tasks . The meta-gradient approach operates by evaluating a change in the options , caused via changes to the option-rewards and terminations , w.r.to the hierarchical agent ’ s performance on samples drawn from many tasks ; then computes and uses the gradients from this evaluation to discover the option-rewards and terminations . This differs significantly from the previous multi-task option discovery approaches ( Bacon et al. , 2017 ; Frans et al. , 2017 ) , where all options directly optimise the same ( main task ) reward , which may be insufficient to discover reusable , task-independent options . We evaluate the proposed approach empirically in two multi-task RL settings based on an illustrative gridworld and on 3-dimensional first-person task suites from DeepMind Lab ( Beattie et al. , 2016 ) . For each of these , we perform three types of analysis : ( 1 ) we qualitatively demonstrate that our approach indeed produces meaningful temporally-extended options ; ( 2 ) we quantitatively show that our discovered options are extensively used by the agent while learning to solve training tasks ; ( 3 ) we show that our discovered options support faster learning , i.e. , transfer better , in test tasks , compared to options discovered by two strong hierarchical baselines ( i.e. , MLSH ( Frans et al. , 2017 ) & Option-Critic ( Harb et al. , 2018 ) ) . 1 MODAC : Meta-gradients for Option Discovery using Actor-Critic . Why Meta-gradients ? As described in the previous section , the motivation behind our work is to discover temporal abstractions in the form of options that are generally useful across many training tasks ; and to leverage them to allow effective transfer of acquired skills to new tasks . This is based on our hypothesis that options , if useful across many tasks , capture intrinsic properties about those tasks that could lead to better transfer to unseen tasks . For useful options to be discovered , an option-based agent needs to evaluate whether a change in a given option is useful . Essentially , the agent needs to compute the gradient of future performance with respect to the parameterisation of each option , while the option-based behaviour is itself adapted by the conventional RL gradients ; this requires the computation of a gradient through a gradient , and thus meta-gradients is used as the mechanism for driving option discovery . We make this idea concrete in our learning agent MODAC ( which stands for Meta-gradients for Option Discovery using Actor-Critic ) . Background on Options : Options are closed-loop behaviours over extended periods of time . An option is formally defined by specifying an initiation set ( states where the option may be invoked ) , an option-policy ( maps states to actions ) , and a termination function ( maps states to probability of terminating execution of the option ) . The option-policy may be defined implicitly as the policy that maximises an option-reward function describing the subgoal ( or intention ) for that option . 1.1 Agent Architecture . We base our agent ’ s architecture on the standard hierarchical agent from Sutton et al . ( 1999 ) , where a manager chooses among both primitive actions3 and temporally-extended options ; and extend it to the multi-task setup . The agent ( shown in Fig . 1 ) follows a call-and-return option execution-model and consists of the following four modules : 3Primitive actions are a special case of options that terminate after one step . See Sec . 3.1 , pg . 194 of Sutton et al . ( 1999 ) . A Manager network parameterised by θM implements task-conditional policy πM . It maps a sequence of observations ( henceforth the state st ∈ S ) and a task encoding g ∈ G to the union of the set of option-policies O and the set of primitive actions A , i.e. , πM : S × G → { O , A } . We denote a sample from πM as o ∼ πM ( s , g ) . The Manager is trained to maximise the task reward . The Option-policy network represents policies for each discovered option in our hierarchical agent . It takes state as input and has multiple output head that each produce a distribution over primitive actions : the ith head ’ s output represents the option-policy πoi : S → A ) . θoi denotes the parameters of πoi . Each option-policy learns to maximise the corresponding option-rewards produced from the option-reward network described next . The Option-reward network defines the subgoals/intentions for the option-policies . It takes state as input and outputs a scalar reward , one per option-policy , for each primitive action a ∈ A ( the ith option-reward head ’ s output is roi : S → R|A| ) . ηroi denotes the parameters of roi . Option-termination network defines the termination function for the option-policies and has an identical structure to the option-reward network : it also takes state as input and outputs a scalar value βoi : S → { 0 , 1 } for each option-policy , interpreted as the probability of termination for the associated option . ηβ oi denotes the parameters of βoi . Each module in the agent closely follows the options framework : the manager corresponds to the policy-over-options from the options framework as it learns a mapping from states to options . Each option is defined by its associated option-policy and termination defined by the respective networks ( initiation set for each option is softly induced by the manager ’ s policy ) . Option-reward along with their termination defines an option ’ s subgoal . Note , among the four modules , only the manager gets the task goal as input while those corresponding to options ( i.e. , option-policy , reward and termination ) do not . This architecturally enforces our objective of discovering task-independent options that are useful across multiple training tasks , and also supports their transfer to new , previously unseen tasks . We emphasise that it is possible for MODAC to solve each task optimally as the manager gets task as input and can select primitive actions if need be . Also , this form of hiding task goals from the temporal abstractions has been shown to be effective for transfer in several multi-task hierarchical RL approaches ( Dayan & Hinton , 1993 ; Heess et al. , 2016 ; Jaderberg et al. , 2016 ; Nachum et al. , 2018 ) . Separately , note that our assumption of access to task encodings is standard in many multi-task RL ( Beattie et al. , 2016 ; Plappert et al. , 2018 ) and robotics domains ( Kolve et al. , 2017 ; Deitke et al. , 2020 ) . We motivate the choice of allowing the manager to pick between option-policies and primitive actions as opposed to option-policies alone with the following example : consider the case where options take you to the doorways in a building with rooms . Suppose the goal was to go to the middle of some room . Now there is no policy that maps states to options that can achieve that middle-of-the-room goal . But if we allow the manager to choose both options and primitive actions , then it could get to the doorway of the room by traveling from doorway to doorway and then pick primitive actions to get to the middle of the target room . This is a far more flexible use of options . Indeed , this is the way options were presented originally Sutton et al . ( 1999 ) . More recent works of Mann et al . ( 2015 ) and Jong et al . ( 2008 ) also argue for using primitive actions along with options to improve learning performance . Mann et al . ( 2015 ) theoretically study the role of adding primitive actions and proved that using options along with primitive actions allows a planning agent to achieve faster convergence . Jong et al . ( 2008 ) empirically studies this design choice of including primitive actions to a HRL agent and clearly show that the agent that can select between options and primitive actions can obtain optimal performance while the agent that only selects options fails . Furthermore , the work by Baumli et al . ( 2021 ) also explores HRL approaches that learn by selecting among options and primitive actions . In the next section , we present a discovery algorithm based on meta-gradients that is used to train each network .

Summary: Authors propose a meta-learning algorithm for learning temporarily extended actions within the options framework. In addition to the commonly used option modules defined in the original options framework (high-level policy, terminations, sub-policies) the algorithm also uses an aditional learned option-rewards module ($\{r^{o^i}\}$) to reward sub-policies when they perform actions. The algorithm is designed to extract useful temporarily extended actions from multiple environments by using meta-gradients (backpropagating through gradient). Meta-gradients are taken wrt. parameters of termination functions and option-rewards whereas sub-policies and high-level policy are updated in the inner update. The intuition behind using option-rewards and terminations as meta-parameters is that they should define task-agnostic sub-goals.   Authors evaluate the performance of this new algorithm on one simple and one complex navigation tasks. The main performance metric considered is performance on a new unseen task (after pre-training on many tasks), although authors do show additional visualizations of learned options. Comparison is made with two additional algorithms for learning options  MLSH (designed for multi-task learning) and Option Critic (originally designed for single-task) and a single-task Actor-Critic baseline. In experiments, authors show that proposed algorithm outperforms all three baselines when options are transferred to previously unseen tasks. This suggests that proposed algorithm discovers options with better temporal structure.