Growing Action Spaces

In complex tasks , such as those with large combinatorial action spaces , random exploration may be too inefficient to achieve meaningful learning progress . In this work , we use a curriculum of progressively growing action spaces to accelerate learning . We assume the environment is out of our control , but that the agent may set an internal curriculum by initially restricting its action space . Our approach uses off-policy reinforcement learning to estimate optimal value functions for multiple action spaces simultaneously and efficiently transfers data , value estimates , and state representations from restricted action spaces to the full task . We show the efficacy of our approach in proof-of-concept control tasks and on challenging large-scale StarCraft micromanagement tasks with large , multi-agent action spaces . 1 INTRODUCTION . The value of curricula has been well established in machine learning , reinforcement learning , and in biological systems . When a desired behaviour is sufficiently complex , or the environment too unforgiving , it can be intractable to learn the behaviour from scratch through random exploration . Instead , by “ starting small ” ( Elman , 1993 ) , an agent can build skills , representations , and a dataset of meaningful experiences that allow it to accelerate its learning . Such curricula can drastically improve sample efficiency ( Bengio et al. , 2009 ) . Typically , curriculum learning uses a progression of tasks or environments . Simple tasks that provide meaningful feedback to random agents are used first , and some schedule is used to introduce more challenging tasks later during training ( Graves et al. , 2017 ) . However , in many contexts neither the agent nor experimenter has such unimpeded control over the environment . In this work , we instead make use of curricula that are internal to the agent , simplifying the exploration problem without changing the environment . In particular , we grow the size of the action space of reinforcement learning agents over the course of training . At the beginning of training , our agents use a severely restricted action space . This helps exploration by guiding the agent towards rewards and meaningful experiences , and provides low variance updates during learning . The action space is then grown progressively . Eventually , using the most unrestricted action space , the agents are able to find superior policies . Each action space is a strict superset of the more restricted ones . This paradigm requires some domain knowledge to identify a suitable hierarchy of action spaces . However , such a hierarchy is often easy to find . Continuous action spaces can be discretised with increasing resolution . Similarly , curricula for coping with the large combinatorial action spaces induced by many agents can be obtained from the prior that nearby agents are more likely to need to coordinate . For example , in routing or traffic flow problems nearby agents or nodes may wish to adopt similar local policies to alleviate global congestion . Our method will be valuable when it is possible to identify a restricted action space in which random exploration leads to significantly more meaningful experiences than random exploration in the full action space . We propose an approach that uses off-policy reinforcement learning to improve sample efficiency in this type of curriculum learning . Since data from exploration using a restricted action space is still valid in the Markov Decision Processes ( MDPs ) corresponding to the less restricted action spaces , we can learn value functions in the less restricted action space with ‘ off-action-space ’ data collected by exploring in the restricted action space . In our approach , we learn value functions corresponding to each level of restriction simultaneously . We can use the relationships of these value functions to each other to accelerate learning further , by using value estimates themselves as initialisations or as bootstrap targets for the less restricted action spaces , as well as sharing learned state representations . Empirically , we first demonstrate the efficacy of our approach in two simple control tasks , in which the resolution of discretised actions is progressively increased . We then tackle a more challenging set of problems with combinatorial action spaces , in the context of StarCraft micromanagement with large numbers of agents ( 50-100 ) . Given the heuristic prior that nearby agents in a multiagent setting are likely to need to coordinate , we use hierarchical clustering to impose a restricted action space on the agents . Agents in a cluster are restricted to take the same action , but we progressively increase the number of groups that can act independently of one another over the course of training . Our method substantially improves sample efficiency on a number of tasks , outperforming learning any particular action space from scratch , a number of ablations , and an actor-critic baseline that learns a single value function for the behaviour policy , as in the work of Czarnecki et al . ( 2018 ) . Code is available , but redacted here for anonymity . 2 RELATED WORK . Curriculum learning has a long history , appearing at least as early as the work of Selfridge et al . ( 1985 ) in reinforcement learning , and for the training of neural networks since Elman ( 1993 ) . In supervised learning , one typically has control of the order in which data is presented to the learning algorithm . For learning with deep neural networks , Bengio et al . ( 2009 ) explored the use of curricula in computer vision and natural language processing . Many approaches use handcrafted schedules for task curricula , but others ( Zaremba & Sutskever , 2014 ; Pentina et al. , 2015 ; Graves et al. , 2017 ) study diagnostics that can be used to automate the choice of task mixtures throughout training . In a self-supervised control setting , Murali et al . ( 2018 ) use sensitivity analysis to automatically define a curriculum over action dimensions and prioritise their search space . In some reinforcement learning settings , it may also be possible to control the environment so as to induce a curriculum . With a resettable simulator , it is possible to use a sequence of progressively more challenging initial states ( Asada et al. , 1996 ; Florensa et al. , 2017 ) . With a procedurally generated task , it is often possible to automatically tune the difficulty of the environments ( Tamar et al. , 2016 ) . Similar curricula also appear often in hierarchical reinforcement learning , where skills can be learned in comparatively easy settings and then composed in more complex ways later ( Singh , 1992 ) . Taylor et al . ( 2007 ) use more general inter-task mappings to transfer Q-values between tasks that do not share state and action spaces . In adversarial settings , one may also induce a curriculum through self-play ( Tesauro , 1995 ; Sukhbaatar et al. , 2017 ; Silver et al. , 2017 ) . In this case , the learning agents themselves define the changing part of the environment . A less invasive manipulation of the environment involves altering the reward function . Such reward shaping allows learning policies in an easier MDP , which can then be transferred to the more difficult sparse-reward task ( Colombetti & Dorigo , 1992 ; Ng et al. , 1999 ) . It is also possible to learn reward shaping on simple tasks and transfer it to harder tasks in a curriculum ( Konidaris & Barto , 2006 ) . In contrast , learning with increasingly complex function approximators does not require any control of the environment . In reinforcement learning , this has often taken the form of adaptively growing the resolution of the state space considered by a piecewise constant discretised approximation ( Moore , 1994 ; Munos & Moore , 2002 ; Whiteson et al. , 2007 ) . Stanley & Miikkulainen ( 2004 ) study continual complexification in the context of coevolution , growing the complexity of neural network architectures through the course of training . These works progressively increase the capabilities of the agent , but not with respect to its available actions . In the context of planning on-line with a model , there are a number of approaches that use progressive widening to consider increasing large action spaces over the course of search ( Chaslot et al. , 2008 ) , including in planning for continuous action spaces ( Couëtoux et al. , 2011 ) . However , these methods can not directly be applied to grow the action space in the model-free setting . A recent related work tackling our domain is that of Czarnecki et al . ( 2018 ) , who train mixtures of two policies with an actor-critic approach , learning a single value function for the current mixture of policies . The mixture contains a policy that may be harder to learn but has a higher performance ceiling , such as a policy with a larger action space as we consider in this work . The mixing coefficient is initialised to only support the simpler policy , and adapted via population based training ( Jaderberg et al. , 2017 ) . In contrast , we simultaneously learn a different value function for each policy , and exploit the properties of the optimal value functions to induce additional structure on our models . We further use these properties to construct a scheme for off-action-space learning which means our approach may be used in an off-policy setting . Empirically , in our settings , we find our approach to perform better and more consistently than an actor-critic algorithm modeled after Czarnecki et al . ( 2018 ) , although we do not take on the significant additional computational requirements of population based training in any of our experiments . A number of other works address the problem of generalisation and representation for value functions with large discrete action spaces , without explicitly addressing the resulting exploration problem ( Dulac-Arnold et al. , 2015 ; Pan et al. , 2018 ) . These approaches typically rely on action representations from prior knowledge . Such representations could be used in combination with our method to construct a hierarchy of action spaces with which to shape exploration . 3 BACKGROUND . We formalise our problem as a MDP , specified by a tuple < S , A , P , r , γ > . The set of possible states and actions are given by S and A , P is the transition function that specifies the environment dynamics , and γ is a discount factor used to specify the discounted return R = ∑T t=0 γ trt for an episode of length T . We wish our agent to maximise this return in expectation by learning a policy π that maps states to actions . The state-action value function ( Q-function ) is given by Qπ = Eπ [ R|s , a ] . The optimal Q-function Q∗ satisfies the Bellman optimality equation : Q∗ ( s , a ) = T Q∗ ( s , a ) = E [ r ( s , a ) + γmax a′ Q∗ ( s′ , a′ ) ] . ( 1 ) Q-learning ( Watkins & Dayan , 1992 ) uses a sample-based approximation of the Bellman optimality operator T to iteratively improve an estimate of Q∗ . Q-learning is an off-policy method , meaning that samples from any policy may be used to improve the value function estimate . We use this property to engage Q-learning for off-action-space learning , as described in the next section . We also introduce some notation for restricted action spaces . In particular , for an MDP with unrestricted action space A we define a set of N action spaces A ` , ` ∈ { 0 , . . . , N − 1 } . Each action space is a subset of the next : A0 ⊂ A1 ⊂ . . . ⊂ AN−1 ⊆ A . A policy restricted to actions A ` is denoted π ` ( a|s ) . The optimal policy in this restricted policy class is π∗ ` ( a|s ) , and its corresponding action-value and value functions are Q∗ ` ( s , a ) and V ∗ ` ( s ) = maxaQ ∗ ` ( s , a ) . Additionally , we define a hierarchy of actions by identifying for every action a ∈ A ` , ` > 0 a parent action parent ` ( a ) in the space of A ` −1 . Since action spaces are subsets of larger action spaces , for all a ∈ A ` −1 , parent ` ( a ) = a , i.e. , one child of each action is itself . Simple pieces of domain knowledge are often sufficient to define these hierarchies . For example , a discretised continuous action can identify its nearest neighbour in A ` −1 as a parent . In Section 5 we describe a possible hierarchy for multi-agent action spaces . One could also imagine using action-embeddings ( Tennenholtz & Mannor , 2019 ) to learn such a hierarchy from data .

Summary: The paper presents a method of scaling up towards action spaces, that exhibit natural hierarchies (such as a controllable resolution of actions), throughout joint training of Q-functions over these. Authors notice, and exploit a few interesting properties, such as inequalities that emerge when action spaces form strict subsets that lead to nice parametrisation of policies in a differential way. The evaluation is performed in simple toy-ish tasks, and in micro-management problem in 5 scenarios in the game of SC2.