Multi-Agent Hierarchical Reinforcement Learning for Humanoid Navigation

1 INTRODUCTION . Deep Reinforcement Learning ( DRL ) has been successful in solving complex planning tasks . Given a sizeable computational budget DRL has displayed superhuman performance on many games ( Mnih et al. , 2015 ; Silver et al. , 2017 ) . However , less progress has been made on the Multi-Agent Reinforcement Learning ( MARL ) problem space , possibly due to the non-stationary optimization of multiple changing policies , which is not easily overcome by collecting more data ( OpenAI , 2018 ) . If the goal is to create agents that can operate in a dynamic multi-agent world , more stable methods with novel forms of communication are needed . The trend to make progress on MARL has been to simplify the optimization problem . For example , converting the multi-agent problem into a single agent centralized model results in large gains in performance but can increase the number of network parameters significantly and imposes a constraint on the generalization to the number of agents ( Lowe et al. , 2017 ) . By using recurrent policies , significant computational power , and constraints on the amount the policy is allowed to change between updates , it is possible to beat the best humans at the multi-agent game of Dota ( OpenAI , 2018 ) . While these methods have shown promise , they require significant amounts of compute and have not yet displayed success in complex and dynamic multi-agent environments . While recent work has been successful in producing competitive behaviour using asymmetric self-play , this work is limited to two agents ( Bansal et al. , 2017 ) . In this work , we propose the integration of MARL with Hierarchical Reinforcement Learning ( HRL ) to produce heterogeneous humanoid agents that can both navigate and interact in dynamic simulation . Specifically , we propose a method to reduce the complexity in the MARL policy optimization problem using a type of parameter sharing . While previous methods have focused on re-framing the problem as a type of single-agent RL problem , we show that this does not scale . Our method 1https : //sites.google.com/view/mahrl preserves the essential features of heterogeneous agent behaviour without adding more network parameters ( Lowe et al. , 2017 ) . This use of HRL with a goal conditioned lower layer ( Kaelbling , 1993 ) , has significant advantages over current methods . We use shared parameter methods for task-agnostic portions of the policy similar to multi-task learning ( Caruana , 1997 ; Argyriou et al. , 2007 ; Lu et al. , 2017 ) where each high-level agent is a different task . Given the shared sub-policy , the optimization is simplified and allows learning complex multi-agent policies with significantly less data . This method represents , to the best of our knowledge , the first method for heterogeneous multi-agent physical character control for locomotion , navigation , and behaviour . 2 RELATED WORK . Simulated robot and physical character control is a rich area of research with many solutions and approaches . Neural models focused on training neural networks by receiving joint or body sensor feedback as input and producing appropriate joint angles as output ( Geng et al. , 2006 ; Kun & Miller III , 1996 ; Miller III , 1994 ) . A biped character ’ s movement controller set can also be manually composed using simple control strategies and feedback learning ( Yin et al. , 2007 ; Faloutsos et al. , 2001 ) . HRL has been proposed as a solution to handling many of the issues with current RL techniques that have trouble with long horizons and weak signal . Many frameworks have been proposed for HRL , but none seem to be the obvious choice for any particular problem ( Sutton et al. , 1999 ; Dayan & Hinton , 1993 ; Dietterich , 1999 ) . One difficulty in HRL design is finding a reasonable communication representation to condition the lower level . Some methods pretrain the lower level on a random distribution ( Heess et al. , 2016 ; Peng et al. , 2017 ; Merel et al. , 2018 ) and others learn a more constructive latent encoding ( Nair et al. , 2018 ; Eysenbach et al. , 2019 ; Gupta et al. , 2018 ) . There is also the present challenge of learning multiple levels of the hierarchy concurrently ( Vezhnevets et al. , 2017 ; Nachum et al. , 2018 ; Levy et al. , 2017a ) . Here a goal-based approach that uses a footstep space representation that is pretrained to learn a task-agnostic lower level of control . This provides the high-level policies with a single shared model that is conditioned to produce diverse behaviour based on a latent goal input . 2.1 MULTI-AGENT DEEP REINFORCEMENT LEARNING . There are many types of multi-agent learning problems , including cooperative-competitive and withwithout communication ( Bu et al. , 2008 ; Tan , 1993 ; Panait & Luke , 2005 ) . Recent work converts the MARL problem to a single agent setting by using a single Q-function across all agents ( Lowe et al. , 2017 ) . Additional work focuses on the problem of learning communication methods between agents ( Foerster et al. , 2016 ) . While progress is being made , MARL is notoriously tricky due to the non-stationary optimization issue , even in the cooperative case ( Claus & Boutilier , 1998 ) . In this work , we apply a partial parameter sharing method assuming all agents share task-agnostic locomotion and optimize similar goals ( Gupta et al. , 2017 ) . There exists few environments specifically created for MARL evaluation ( Zheng et al. , 2018 ; Suarez et al. , 2019 ) . The focus of these environments is often a type of strategy learning and cooperation ( Tian et al. , 2017 ; Vinyals et al. , 2017 ) . MARL is a growing area of research and , as such , will need increasingly complex environments to evaluate algorithms . In this work , we are interested in the overlapping problems of control and perception . To this end , we have created the first simulation of its type that affords multiple physics-based control tasks with variable numbers of agents . Recent work has begun to combine MARL and HRL but is limited to simple environments , uses additional methods to stabilize the optimization , and includes communication ( Tang et al. , 2018 ; Han et al. , 2019 ) . This work focuses on a decentralized approach for interacting models that generalize across scenarios , agent numbers , and tasks . Our work represents a new paradigm in learning within the multi-agent navigation domain that goes beyond simple navigation control strategies . We show compelling AI that learns navigation and gameplay strategies with fully articulated physical characters . This is achieved through a novel learning strategy that produces high-value policies for a complicated control problem . 3 POLICY REPRESENTATION AND LEARNING . In this section , we outline key details of the general Reinforcement Learning ( RL ) framework . RL is formulated on the Markov Dynamic Process ( MDP ) framework : at every time step t , the world ( including the agent ) exists in a state st ∈ S , wherein the agent is able to perform actions at ∈ A , sampled from a policy π ( st , at ) which results in a new state st+1 ∈ S according to the transition probability function T ( st+1|at , st ) . Performing action at in state st produces a reward rt from the environment ; the expected future discounted reward from executing a policy π is : J ( π ) = Er0 , ... , rT [ T∑ t=0 γtrt ] ( 1 ) where T is the maximum time horizon , and γ is the discount factor , indicating the planning horizon length . The agent ’ s goal is to optimize its policy , π ( ·|θπ ) , by maximizing J ( π ) . The policies in the work are trained using the Proximal Policy Optimization ( PPO ) algorithm ( Schulman et al. , 2017 ) . The value function is trained using TD ( λ ) . 3.1 HIERARCHICAL REINFORCEMENT LEARNING . In HRL the original MDP is separated into different MDPs that should be easier to solve separately . In practice , this is accomplished by learning two different policies in two different layers . The lower level policy is trained first and is often conditioned on a latent variable or goal g. The lower level policy πlo ( a|s , g ) is constructed in a way to give it temporally correlated behaviour depending on the g. After the lower level policy is trained , it is used to help solve the original MDP using a separate policy πhi ( g|s ) . This policy learns to provide goals to the lower policy to maximize rewards . 4 MULTI-AGENT REINFORCEMENT LEARNING . The extension to the MDP framework for MARL is a partially observable Markov game ( Littman , 1994 ) . A Markov game has a collection ofN agents , each with its own set of actionsA0 , . . . , AN and partial observationsO0 , . . . , ON of the full state space S. Each agent i has its own policy π ( a|oi , θi ) that models the probability of selecting an action . The environment transition function is a function of the full state and each agent ’ s actions T ( S′|S , A0 , . . . , AN ) . Each agent i receives a reward ri for taking a particular action ai given a partial observation oi and its objective is to maximize this reward over time ∑T t=0 γ trti , where γ is the discount factor and T is the time horizon . The policy gradient can be computed for each agent as ∇θiJ ( π ( ·|θi ) ) = ∫ Oi dθi ( oi ) ∫ Ai ∇θi log ( π ( ai|oi , θi ) ) Aπ ( oi , ai ) dai doi ( 2 ) where dθ = ∫ S ∑T t=0 γ tp0 ( o0 ) ( o0 → o | t , π0 ) do0 is the discounted state distribution , p0 ( o ) represents the initial state distribution , and p0 ( o0 ) ( o0 → o | t , π0 ) models the likelihood of reaching state s by starting at state o0 and following the policy π ( a , o|θπ ) for T steps ( Silver et al. , 2014 ) . Here Aπ ( o , a ) represents the advantage function estimator GAE ( λ ) ( Schulman et al. , 2016 ) . The challenge in MARL is that each agent learns separately , and this often results in a non-stationary learning problem . Each agent is learning how to estimate the dynamics and advantage of an environment that is affected by other actively learning agents . Data collected and used for learning becomes inaccurate after any learning update ; thus , the problem is non-stationary . MADDPG reformulates the above problem into a fully observable problem with a centralized Q-function using the equation : ∇θiJ ( θ ) = Eoi∼pπ ( · ) , ai∼π ( ·|oi ) [ ∇θi log π ( ai|oi , θi ) Q π i ( o , a ) ] ( 3 ) Here Qπi ( o , a ) is a centralized action-valued function that gives the Q value for agent i . Where o = { o0t , . . . , oNt } is the observations of all agents and a = { a0t , . . . , aNt } all agent actions at time t and θ = { θ0 , . . . , θN } is the collection of policy parameters for each agent . This framework will allow the method to train the policies together , but it treats the agents as disjoint distributions to optimize . Instead , we propose a decentralized method that does not need to learn a complex Qfunction and reduces the optimization challenges by using HRL .

Summary: This paper proposes a multi-agent hierarchical reinforcement learning algorithm so that multiple humanoid robots can navigate in multi-agent settings (e.g. avoid collisions, collaboration, chase and escape) in a physically simulated environment. The key difference of this paper with the prior work on MARL is that it used an accurate physics simulation of humanoid robots. This is the main reason of using the hierarchical RL.