Discovery of Options via Meta-Learned Subgoals

Reinforcement learning ( RL ) problems involve learning about temporally-extended actions at multiple time scales . In RL , the options framework ( Sutton et al. , 1999 ) provides a well-defined formalisation for the notion of temporally-extended actions . Options that achieve specific subgoals can be useful in reinforcement learning ( RL ) in at least two ways : in model-based RL , they provide faster rates of convergence through longer-backups of value functions within planning updates ( Silver & Ciosek , 2012 ; Mann & Mannor , 2014 ; Brunskill & Li , 2014 ) , while in model-free RL , temporallyextended actions commit agents to intentional multi-step behaviours , which can translate into better exploration ( Machado & Bowling , 2016 ; Nachum et al. , 2019 ; Osband et al. , 2019 ) . We consider a scenario where an agent learns to solve a distribution over tasks . In such cases having carefully designed temporal abstractions can greatly reduce the overall sample complexity of learning for an RL agent that is trying to master those tasks . The agent can produce faster learning mainly by reusing those abstractions across multiple tasks ( Sutton et al. , 1999 ; Solway et al. , 2014 ) . Many recent approaches have empirically validated this by demonstrating that hand-designed temporal abstractions can often lead to improved learning performance on a variety of challenging multitask RL domains ( Imazeki & Maeno , 2003 ; Kulkarni et al. , 2016 ; Nachum et al. , 2018 ; Riedmiller et al. , 2018 ) . However , if the agent has abstractions that are not useful to a downstream task , it can significantly hurt the performance by making exploration harder ( Jong et al. , 2008 ) , emphasising the challenge involved in carefully hand-designing abstractions that are useful in general , across many domains and tasks . Thus , the automated discovery of temporal abstractions from experience without extensive domain-specific knowledge remains an important open problem for reducing sample complexity in RL . ∗Corresponding Author . Email : vveeriah @ umich.edu †Now at Sea AI Lab 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . The main contribution of this work is our hierarchical agent architecture and an associated metagradient algorithm to discover temporally-extended actions in the form of options that can be reused across many tasks . Previously , meta-gradients have been successfully used for learning hyperparameters ( Xu et al. , 2018 ; Zahavy et al. , 2020 ) , intrinsic rewards ( Zheng et al. , 2018 , 2019 ; Rajendran et al. , 2019 ) , and auxiliary tasks ( Veeriah et al. , 2019 ) . Our work is the first to demonstrate that they can successfully learn rich parameterisations of temporal abstractions . Our starting point is the following hypothesis : If we could discover temporal abstractions useful across many training tasks , they would capture regularities across those task environments and have a higher likelihood of being useful and reusable in new , previously unseen tasks . To discover temporal abstractions useful across tasks , our system flexibly defines task-independent subgoals for options through separate discovered rewards and terminations , different for each option . We employ meta-gradients to discover the parameters of such option-rewards and terminations based on their utility across the many training tasks , so that the set of induced options is useful to a hierarchical agent trying to master all training tasks . The meta-gradient approach operates by evaluating a change in the options , caused via changes to the option-rewards and terminations , w.r.to the hierarchical agent ’ s performance on samples drawn from many tasks ; then computes and uses the gradients from this evaluation to discover the option-rewards and terminations . This differs significantly from the previous multi-task option discovery approaches ( Bacon et al. , 2017 ; Frans et al. , 2017 ) , where all options directly optimise the same ( main task ) reward , which may be insufficient to discover reusable , task-independent options . We evaluate the proposed approach empirically in two multi-task RL settings based on an illustrative gridworld and on 3-dimensional first-person task suites from DeepMind Lab ( Beattie et al. , 2016 ) . For each of these , we perform three types of analysis : ( 1 ) we qualitatively demonstrate that our approach indeed produces meaningful temporally-extended options ; ( 2 ) we quantitatively show that our discovered options are extensively used by the agent while learning to solve training tasks ; ( 3 ) we show that our discovered options support faster learning , i.e. , transfer better , in test tasks , compared to options discovered by two strong hierarchical baselines ( i.e. , MLSH ( Frans et al. , 2017 ) & Option-Critic ( Harb et al. , 2018 ) ) . 1 MODAC : Meta-gradients for Option Discovery using Actor-Critic . Why Meta-gradients ? As described in the previous section , the motivation behind our work is to discover temporal abstractions in the form of options that are generally useful across many training tasks ; and to leverage them to allow effective transfer of acquired skills to new tasks . This is based on our hypothesis that options , if useful across many tasks , capture intrinsic properties about those tasks that could lead to better transfer to unseen tasks . For useful options to be discovered , an option-based agent needs to evaluate whether a change in a given option is useful . Essentially , the agent needs to compute the gradient of future performance with respect to the parameterisation of each option , while the option-based behaviour is itself adapted by the conventional RL gradients ; this requires the computation of a gradient through a gradient , and thus meta-gradients is used as the mechanism for driving option discovery . We make this idea concrete in our learning agent MODAC ( which stands for Meta-gradients for Option Discovery using Actor-Critic ) . Background on Options : Options are closed-loop behaviours over extended periods of time . An option is formally defined by specifying an initiation set ( states where the option may be invoked ) , an option-policy ( maps states to actions ) , and a termination function ( maps states to probability of terminating execution of the option ) . The option-policy may be defined implicitly as the policy that maximises an option-reward function describing the subgoal ( or intention ) for that option . 1.1 Agent Architecture . We base our agent ’ s architecture on the standard hierarchical agent from Sutton et al . ( 1999 ) , where a manager chooses among both primitive actions3 and temporally-extended options ; and extend it to the multi-task setup . The agent ( shown in Fig . 1 ) follows a call-and-return option execution-model and consists of the following four modules : 3Primitive actions are a special case of options that terminate after one step . See Sec . 3.1 , pg . 194 of Sutton et al . ( 1999 ) . A Manager network parameterised by θM implements task-conditional policy πM . It maps a sequence of observations ( henceforth the state st ∈ S ) and a task encoding g ∈ G to the union of the set of option-policies O and the set of primitive actions A , i.e. , πM : S × G → { O , A } . We denote a sample from πM as o ∼ πM ( s , g ) . The Manager is trained to maximise the task reward . The Option-policy network represents policies for each discovered option in our hierarchical agent . It takes state as input and has multiple output head that each produce a distribution over primitive actions : the ith head ’ s output represents the option-policy πoi : S → A ) . θoi denotes the parameters of πoi . Each option-policy learns to maximise the corresponding option-rewards produced from the option-reward network described next . The Option-reward network defines the subgoals/intentions for the option-policies . It takes state as input and outputs a scalar reward , one per option-policy , for each primitive action a ∈ A ( the ith option-reward head ’ s output is roi : S → R|A| ) . ηroi denotes the parameters of roi . Option-termination network defines the termination function for the option-policies and has an identical structure to the option-reward network : it also takes state as input and outputs a scalar value βoi : S → { 0 , 1 } for each option-policy , interpreted as the probability of termination for the associated option . ηβ oi denotes the parameters of βoi . Each module in the agent closely follows the options framework : the manager corresponds to the policy-over-options from the options framework as it learns a mapping from states to options . Each option is defined by its associated option-policy and termination defined by the respective networks ( initiation set for each option is softly induced by the manager ’ s policy ) . Option-reward along with their termination defines an option ’ s subgoal . Note , among the four modules , only the manager gets the task goal as input while those corresponding to options ( i.e. , option-policy , reward and termination ) do not . This architecturally enforces our objective of discovering task-independent options that are useful across multiple training tasks , and also supports their transfer to new , previously unseen tasks . We emphasise that it is possible for MODAC to solve each task optimally as the manager gets task as input and can select primitive actions if need be . Also , this form of hiding task goals from the temporal abstractions has been shown to be effective for transfer in several multi-task hierarchical RL approaches ( Dayan & Hinton , 1993 ; Heess et al. , 2016 ; Jaderberg et al. , 2016 ; Nachum et al. , 2018 ) . Separately , note that our assumption of access to task encodings is standard in many multi-task RL ( Beattie et al. , 2016 ; Plappert et al. , 2018 ) and robotics domains ( Kolve et al. , 2017 ; Deitke et al. , 2020 ) . We motivate the choice of allowing the manager to pick between option-policies and primitive actions as opposed to option-policies alone with the following example : consider the case where options take you to the doorways in a building with rooms . Suppose the goal was to go to the middle of some room . Now there is no policy that maps states to options that can achieve that middle-of-the-room goal . But if we allow the manager to choose both options and primitive actions , then it could get to the doorway of the room by traveling from doorway to doorway and then pick primitive actions to get to the middle of the target room . This is a far more flexible use of options . Indeed , this is the way options were presented originally Sutton et al . ( 1999 ) . More recent works of Mann et al . ( 2015 ) and Jong et al . ( 2008 ) also argue for using primitive actions along with options to improve learning performance . Mann et al . ( 2015 ) theoretically study the role of adding primitive actions and proved that using options along with primitive actions allows a planning agent to achieve faster convergence . Jong et al . ( 2008 ) empirically studies this design choice of including primitive actions to a HRL agent and clearly show that the agent that can select between options and primitive actions can obtain optimal performance while the agent that only selects options fails . Furthermore , the work by Baumli et al . ( 2021 ) also explores HRL approaches that learn by selecting among options and primitive actions . In the next section , we present a discovery algorithm based on meta-gradients that is used to train each network .

Summary: This paper develops an approach for option discovery using meta-gradient descent to optimize a set of option networks. These option networks are then used to define the reward, and termination functions. The option policies are learned through normal RL in a separate option-network which is used to define options for a network which learns the behaviour policy using options and the base actions. They restrict their setting to that of multi-task reinforcement learning, where they sample from distributions of tasks with shared underlying regularities. They test their approach on several environments.