Multi-Agent Hierarchical Reinforcement Learning for Humanoid Navigation

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has been successful in solving complex planning tasks . Given a sizeable computational budget DRL has displayed superhuman performance on many games ( Mnih et al. , 2015 ; Silver et al. , 2017 ) . However , less progress has been made on the Multi-Agent Reinforcement Learning ( MARL ) problem space , possibly due to the non-stationary optimization of multiple changing policies , which is not easily overcome by collecting more data ( OpenAI , 2018 ) . If the goal is to create agents that can operate in a dynamic multi-agent world , more stable methods with novel forms of communication are needed . The trend to make progress on MARL has been to simplify the optimization problem . For example , converting the multi-agent problem into a single agent centralized model results in large gains in performance but can increase the number of network parameters significantly and imposes a constraint on the generalization to the number of agents ( Lowe et al. , 2017 ) . By using recurrent policies , significant computational power , and constraints on the amount the policy is allowed to change between updates , it is possible to beat the best humans at the multi-agent game of Dota ( OpenAI , 2018 ) . While these methods have shown promise , they require significant amounts of compute and have not yet displayed success in complex and dynamic multi-agent environments . While recent work has been successful in producing competitive behaviour using asymmetric self-play , this work is limited to two agents ( Bansal et al. , 2017 ) . In this work , we propose the integration of MARL with Hierarchical Reinforcement Learning ( HRL ) to produce heterogeneous humanoid agents that can both navigate and interact in dynamic simulation . Specifically , we propose a method to reduce the complexity in the MARL policy optimization problem using a type of parameter sharing . While previous methods have focused on re-framing the problem as a type of single-agent RL problem , we show that this does not scale . Our method 1https : //sites.google.com/view/mahrl preserves the essential features of heterogeneous agent behaviour without adding more network parameters ( Lowe et al. , 2017 ) . This use of HRL with a goal conditioned lower layer ( Kaelbling , 1993 ) , has significant advantages over current methods . We use shared parameter methods for task-agnostic portions of the policy similar to multi-task learning ( Caruana , 1997 ; Argyriou et al. , 2007 ; Lu et al. , 2017 ) where each high-level agent is a different task . Given the shared sub-policy , the optimization is simplified and allows learning complex multi-agent policies with significantly less data . This method represents , to the best of our knowledge , the first method for heterogeneous multi-agent physical character control for locomotion , navigation , and behaviour . 2 RELATED WORK . Simulated robot and physical character control is a rich area of research with many solutions and approaches . Neural models focused on training neural networks by receiving joint or body sensor feedback as input and producing appropriate joint angles as output ( Geng et al. , 2006 ; Kun & Miller III , 1996 ; Miller III , 1994 ) . A biped character ’ s movement controller set can also be manually composed using simple control strategies and feedback learning ( Yin et al. , 2007 ; Faloutsos et al. , 2001 ) . HRL has been proposed as a solution to handling many of the issues with current RL techniques that have trouble with long horizons and weak signal . Many frameworks have been proposed for HRL , but none seem to be the obvious choice for any particular problem ( Sutton et al. , 1999 ; Dayan & Hinton , 1993 ; Dietterich , 1999 ) . One difficulty in HRL design is finding a reasonable communication representation to condition the lower level . Some methods pretrain the lower level on a random distribution ( Heess et al. , 2016 ; Peng et al. , 2017 ; Merel et al. , 2018 ) and others learn a more constructive latent encoding ( Nair et al. , 2018 ; Eysenbach et al. , 2019 ; Gupta et al. , 2018 ) . There is also the present challenge of learning multiple levels of the hierarchy concurrently ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ; Levy et al. , 2017a ) . Here a goal-based approach that uses a footstep space representation that is pretrained to learn a task-agnostic lower level of control . This provides the high-level policies with a single shared model that is conditioned to produce diverse behaviour based on a latent goal input . 2.1 MULTI-AGENT DEEP REINFORCEMENT LEARNING . There are many types of multi-agent learning problems , including cooperative-competitive and withwithout communication ( Bu et al. , 2008 ; Tan , 1993 ; Panait & Luke , 2005 ) . Recent work converts the MARL problem to a single agent setting by using a single Q-function across all agents ( Lowe et al. , 2017 ) . Additional work focuses on the problem of learning communication methods between agents ( Foerster et al. , 2016 ) . While progress is being made , MARL is notoriously tricky due to the non-stationary optimization issue , even in the cooperative case ( Claus & Boutilier , 1998 ) . In this work , we apply a partial parameter sharing method assuming all agents share task-agnostic locomotion and optimize similar goals ( Gupta et al. , 2017 ) . There exists few environments specifically created for MARL evaluation ( Zheng et al. , 2018 ; Suarez et al. , 2019 ) . The focus of these environments is often a type of strategy learning and cooperation ( Tian et al. , 2017 ; Vinyals et al. , 2017 ) . MARL is a growing area of research and , as such , will need increasingly complex environments to evaluate algorithms . In this work , we are interested in the overlapping problems of control and perception . To this end , we have created the first simulation of its type that affords multiple physics-based control tasks with variable numbers of agents . Recent work has begun to combine MARL and HRL but is limited to simple environments , uses additional methods to stabilize the optimization , and includes communication ( Tang et al. , 2018 ; Han et al. , 2019 ) . This work focuses on a decentralized approach for interacting models that generalize across scenarios , agent numbers , and tasks . Our work represents a new paradigm in learning within the multi-agent navigation domain that goes beyond simple navigation control strategies . We show compelling AI that learns navigation and gameplay strategies with fully articulated physical characters . This is achieved through a novel learning strategy that produces high-value policies for a complicated control problem . 3 POLICY REPRESENTATION AND LEARNING . In this section , we outline key details of the general Reinforcement Learning ( RL ) framework . RL is formulated on the Markov Dynamic Process ( MDP ) framework : at every time step t , the world ( including the agent ) exists in a state st ∈ S , wherein the agent is able to perform actions at ∈ A , sampled from a policy π ( st , at ) which results in a new state st+1 ∈ S according to the transition probability function T ( st+1|at , st ) . Performing action at in state st produces a reward rt from the environment ; the expected future discounted reward from executing a policy π is : J ( π ) = Er0 , ... , rT [ T∑ t=0 γtrt ] ( 1 ) where T is the maximum time horizon , and γ is the discount factor , indicating the planning horizon length . The agent ’ s goal is to optimize its policy , π ( ·|θπ ) , by maximizing J ( π ) . The policies in the work are trained using the Proximal Policy Optimization ( PPO ) algorithm ( Schulman et al. , 2017 ) . The value function is trained using TD ( λ ) . 3.1 HIERARCHICAL REINFORCEMENT LEARNING . In HRL the original MDP is separated into different MDPs that should be easier to solve separately . In practice , this is accomplished by learning two different policies in two different layers . The lower level policy is trained first and is often conditioned on a latent variable or goal g. The lower level policy πlo ( a|s , g ) is constructed in a way to give it temporally correlated behaviour depending on the g. After the lower level policy is trained , it is used to help solve the original MDP using a separate policy πhi ( g|s ) . This policy learns to provide goals to the lower policy to maximize rewards . 4 MULTI-AGENT REINFORCEMENT LEARNING . The extension to the MDP framework for MARL is a partially observable Markov game ( Littman , 1994 ) . A Markov game has a collection ofN agents , each with its own set of actionsA0 , . . . , AN and partial observationsO0 , . . . , ON of the full state space S. Each agent i has its own policy π ( a|oi , θi ) that models the probability of selecting an action . The environment transition function is a function of the full state and each agent ’ s actions T ( S′|S , A0 , . . . , AN ) . Each agent i receives a reward ri for taking a particular action ai given a partial observation oi and its objective is to maximize this reward over time ∑T t=0 γ trti , where γ is the discount factor and T is the time horizon . The policy gradient can be computed for each agent as ∇θiJ ( π ( ·|θi ) ) = ∫ Oi dθi ( oi ) ∫ Ai ∇θi log ( π ( ai|oi , θi ) ) Aπ ( oi , ai ) dai doi ( 2 ) where dθ = ∫ S ∑T t=0 γ tp0 ( o0 ) ( o0 → o | t , π0 ) do0 is the discounted state distribution , p0 ( o ) represents the initial state distribution , and p0 ( o0 ) ( o0 → o | t , π0 ) models the likelihood of reaching state s by starting at state o0 and following the policy π ( a , o|θπ ) for T steps ( Silver et al. , 2014 ) . Here Aπ ( o , a ) represents the advantage function estimator GAE ( λ ) ( Schulman et al. , 2016 ) . The challenge in MARL is that each agent learns separately , and this often results in a non-stationary learning problem . Each agent is learning how to estimate the dynamics and advantage of an environment that is affected by other actively learning agents . Data collected and used for learning becomes inaccurate after any learning update ; thus , the problem is non-stationary . MADDPG reformulates the above problem into a fully observable problem with a centralized Q-function using the equation : ∇θiJ ( θ ) = Eoi∼pπ ( · ) , ai∼π ( ·|oi ) [ ∇θi log π ( ai|oi , θi ) Q π i ( o , a ) ] ( 3 ) Here Qπi ( o , a ) is a centralized action-valued function that gives the Q value for agent i . Where o = { o0t , . . . , oNt } is the observations of all agents and a = { a0t , . . . , aNt } all agent actions at time t and θ = { θ0 , . . . , θN } is the collection of policy parameters for each agent . This framework will allow the method to train the policies together , but it treats the agents as disjoint distributions to optimize . Instead , we propose a decentralized method that does not need to learn a complex Qfunction and reduces the optimization challenges by using HRL .

Summary: The submission proposes a method for hierarchical RL in multiagent settings. In particular it proposes to explicitly decouple training of a high-level and low-level controller with grounded the controller interface as goals in the environment to reach for the low-level controller. The model is trained via PPO with GAE and evaluated on a small set of multi agent locomotion tasks.