Multi-Agent Hierarchical Reinforcement Learning for Humanoid Navigation
Authors: Glen Berseth
Brandon haworth
Seonghyeon Moon
Mubbasir Kapadia
Petros Faloutsos
Venue:  (2020)
Keywords: Multi-Agent Reinforcement Learning
Reinforcement Learning
Hierarchical Reinforcement Learning

Abstract: Multi-agent reinforcement learning is a particularly challenging problem. Current
methods have made progress on cooperative and competitive environments with
particle-based agents. Little progress has been made on solutions that could op-
erate in the real world with interaction, dynamics, and humanoid robots. In this
work, we make a significant step in multi-agent models on simulated humanoid
robot  navigation  by  combining  Multi-Agent  Reinforcement  Learning  (MARL)
with  Hierarchical  Reinforcement  Learning  (HRL).  We  build  on  top  of  founda-
tional prior  work in  learning low-level physical  controllers for locomotion  and
add a layer to learn decentralized policies for multi-agent goal-directed collision
avoidance systems. A video of our results on a multi-agent pursuit environment
can be seen here


Under review as a conference paper at ICLR 2020
MULNI-AGENT HIERARCHICAL LEARNING FOR HUMANOID NAVIGATION
REINFORCEMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-agent reinforcement learning is a particularly challenging problem. Cur- rent methods have made progress on cooperative and competitive environments with particle-based agents using communication and centralized training. Little progress has been made on solutions that could operate in the real world with in- teraction, dynamics, and humanoid navigation strategies. In this work, we make a significant step in multi-agent models on simulated humanoid navigation by com- bining Multi-Agent Reinforcement Learning with Hierarchical Reinforcement Learning. Specifically, we develop partial parameter sharing approach wherein the lower level of the hierarchy is shared enabling learning using decentralized methods. This drastically reduces the overall parameter space in the multi-agent problem and introduces structure in the optimization problem.
We build on top of prior foundational work in learning coal-conditioned policies to learn low-level physical controllers for balance and walking. These lower-level controllers are task-agnostic and can be shared by higher-level policies. Overtop of these goal conditioned policies, we can train decentralized heterogeneous policies for multi-agent goal-directed collision avoidance. Surprisingly, our results show that with this combination of methods, RL techniques can be used for finding strong policies. A video of our results on a multi-agent pursuit environment can be seen here]
INTRODUCTION
Deep Reinforcement Learning (DRL) has been successful in solving complex planning tasks. Given a sizeable computational budget DRL has displayed superhuman performance on many games (Mnih) eet al. 2015| [Silver et al.] 2017). However, less progress has been made on the Multi-Agent Rein- forcement Learning (MARL) problem space, possibly due to the non-stationary optimization of multiple changing policies, which is not casily overcome by collecting more data (OpenAll 2018). If the goal is to create agents that can operate in a dynamic multi-agent world, more stable methods with novel forms of communication are needed. The trend to make progress MARL has been to simplify the optimization problem. For example, converting the multi-agent problem into a sin- gle agent centralized model results in large gains in performance but can increase the number of network parameters significantly and imposes a constraint on the generalization to the number of agents (Lowe et al. 2017). By using recurrent policies, significant computational power, and con- straints on the amount the policy is allowed to change between updates, it is possible to beat the best humans at the multi-agent game of Dota (OpenAl 2018). While these methods have shown promise, they require significant amounts of compute and have not yet displayed success in complex and dy- namic multi-agent environments. While recent work has been successful in producing competitive behaviour using asymmetric self-play, this work is limited to two agents (Bansal et al. 2017
In this work, we propose the integration of MARL with Hierarchical Reinforcement Learning (HRL) to produce heterogeneous humanoid agents that can both navigate and interact in dynamic simula- tion. Specifically, we propose a method to reduce the complexity in the MARL policy optimization problem using type of parameter sharing. While previous methods have focused on re- framimn the problem as a type of single-agent RI. problem, we show that this does not scale. Our method

---

Under review as a conference paper at ICLR 2020
preserves the essential features of heterogeneous agent behaviour without adding more network pa- rameters Lowe et al. 2017). This use of HRL with goal conditioned lower layer (aalbling 1993), has significant advantages over current methods. We use shared parameter methods for task agnostic portions of the policy similar to multi-task learning (Caruana 1997 Argyriou et al. 2007 Lu et al, 2017] where each high-level agent is different task. Given the shared sub-policy, the optimiza- tion is simplified and allows learning complex multi-agent policies with significantly less data. This method represents, the best of our knowledge, the first method for heterogeneous multi-agent physical character control for locomotion, navigation, and behaviour.
2 RELATED WORK
Simulated robot and physical character control is rich area of research with many solutions and approaches. Neural models focused on training neural networks by receiving joint or body sen- sor feedback as input and producing appropriate joint angles as output (Geng et al. 2006 Kun & Miller III 1996; Miller III 1994). A biped character's movement controller set can also be manually composed using simple control strategies and feedback learning (Yin et al. 2007; Faloutsos et al.) p.01 HRI. has been proposed as solution to handling many of the issues with current RI. tech- niques that have trouble with long horizons and weak signal. Many frameworks have been proposed for HRL, but none seem to be the obvious choice for any particular problem (Sutton et al. 1999]
Dayan & Hinton 1993 Dietterich| [1999). One difficulty in HRL design is finding a reasonable com- munication representation to condition the lower level. Some methods pretrain the lower level on a random distribution (Heess et al., 2016 Peng et al. 2017: et al. 2018) and others learn more constructive latent encoding (Nair et al. [0188] Eysenbach et al., 2019 9 Gupta et al., 2018). There is also the present challenge of learning multiple levels of the hierarchy concurrently (Vezhnevets) eet al. 2017 Nachum et al. 2018 Levy et al. (017a). Here a goal-based approach that uses a footstep space representation that is pretrained learn task-agnostic lower level of control. This provides the high-level policies with single shared model that is conditioned to produce diverse behaviour based on latent goal input.
2.1 MULTI-AGENT DEEP REINFORCEMENT LEARNING
There are many types of multi-agent learning problems, including cooperative and with- without communication (Bu et al., 2008 Tan 1993 Panait & Luke| (005). Recent work converts the MARL problem to single agent setting by using single Q-function across all agents (Lowe) jet al. 2017). Additional work focuses on the problem of learning communication methods between agents (Foerster et al., 2016). While progress is being made, MARL is notoriously tricky due to the non-stationary optimization issue, even in the cooperative case (Claus & Boutilier) 1998). In this work, we apply partial parameter sharing method assuming all agents share task-agnostic locomotion and optimize similar goals (Gupta et al. 2017).
There exists few environments specifically created for MARL evaluation (Zheng et al. 2018 Suarez et al. 2019). The focus of these environments is often type of strategy learning and coopera- tion (Tian et al. 2017: Vinyals et al. 2017). MARL is growing area of research and, as such, will need increasingly complex environments to evaluate algorithms. In this work, we are interested in the overlapping problems of control and perception. To this end, we have created the first simulation of its type that affords multiple physics-based control tasks with variable numbers of agents. Recent work has begun to combine MARI. and HRI. but limited to simple environments, uses additional methods to stabilize the optimization, and includes communication (Tang et al. 2018 Han et al,, 2019). This work focuses on a decentralized approach for interacting models that generalize across scenarios, agent numbers, and tasks.
Our work represents a new paradigm in learning within the multi-agent navigation domain that goes beyond simple navigation control strategies. We show compelling AI that learns navigation and gameplay strategies with fully articulated physical characters. This is achieved through a novel learning strategy that produces high-value policies for complicated control problem.
2

---

Under review as a conference paper at ICLR 2020
3 POLICY REPRESENTATION AND LEARNING
In this section, we outline key details of the general Reinforcement Learning (RL) framework. RL is formulated on the Markov Dynamic Process (MDP) framework: at every time step t, the world (including the agent) exists in a state & E S. wherein the agent is able to perform actions dt E A, sampled from a policy (5t; 04) which results in new state 8(41 S according to the transition probability function T( [a,, $t). Performing action a in state 8, produces a reward r from the environment; the expected future discounted reward from executing a policy is:
((x)
E
Σ T
where T' is the maximum time horizon, and y is the discount factor, indicating the planning horizon length. The agent's goal is to optimize its policy, ((-10x). by maximizing J(T). The policies in the work are trained using the Proximal Policy Optimization (PPO) algorithm (Schulman et al. 2017). The value function is trained using TD(A).
3.1 HIERARCHICAL REINFORCEMENT LEARNING
In HRL the original MDP separated into different MDPs that should be easier to solve separately. In practice, this is accomplished by learning two different policies in two different layers. The lower level policy is trained first and is often conditioned on a latent variable or goal g. The lower level policy (al(a g) is constructed in way to give it temporally correlated behaviour depending on the g. After the lower level policy trained, it is used to help solve the original MDP using separate policy x (gls). This policy learns to provide goals to the lower policy to maximize rewards.
4 MULTI-AGENT REINFORCEMENT LEARNING
The extension to the MDP framework for MARL is a partially observable Markov game (Littman 1994). A Markov game has collection of N agents, each with its own set of actions .o,....AA and partial observations Og...., ON of the full state space S. Each agent has its own policy (a o, 0,) that models the probability of selecting an action. The environment transition function is function of the full state and each agent's actions T(S' Ao...., Av). Each agent receives a reward r; for taking a particular action a given a partial observation o, and its objective to maximize this reward over time Ei=o 7'r.. where y the discount factor and T the time horizon. The policy gradient can be computed for each agent as
Vo. Vo.J(T((10.)) = ( do. / Ve, log(TTaaloo (.,)) a.) da, do,
(2)
where dg - S sents the initial state distribution, and (o(oo)(op + o t, ma) models the likelihood of reaching state by starting at state and following the policy x(a, ol0²) for T steps (Silver et al. 2014). Here
+ o 1, mD) dop is the discounted state distribution, Po()) repre-
(o, a) represents the advantage function estimator GAE(A) (Schulman et al. 2016).
The challenge in MARL is that each agent learns separately, and this often results in non-stationary learning problem. Each agent is learning how to estimate the dynamics and advantage of an environ- ment that is affected by other actively learning agents. Data collected and used for learning becomes inaccurate after any learning update; thus, the problem is non- -tatstaiionary. MADDPG reformulates the above problem into fully observable problem with a centralized Q-function using the equation:
(3) Here (o, a) is a centralized action-valued function that gives the Q value for agent i. Where o {of....... of is the observations of all agents and a = {af...., a; } all agent actions at time t and 0 {fo...., ONv) } is the collection of policy parameters for each agent. This framework will allow the method to train the policies together, but it treats the agents as disjoint distributions to optimize. Instead, we propose a decentralized method that does not need to lear complex Q- function and reduces the optimization challenges by using HRL.
Vo, [(0) = Eo, (a)opp))))aaaa)))) Vo, log i ( los Q)) (o,a))
3

---

Under review as a conference paper at ICLR 2020
4.1 TASK-AGNOSTIC LOCOMOTION CONTROLLER (LC)
The LC, the lower-level policy in our design, is designed to learn a robust and diverse policy T (azgor, 9t. 02 based on a latent goal Sr. varaible. The footstep goals 9L = {po. PI. foott } in Fig- ure [ consist of the agent root relative distances of the next two footsteps on the ground plane and the desired facing direction at the first step's i end. This goal description is motivated by work that shows people may plan steering decisions two foot placements ahead (aytssy al. 2015).
The LC learns to place its feet as accurately as it can to match these step goals using the reward
rig = (epppp02||sg00 2 - 9r. The better the LCC learns this behaviour, the more responsive the controller will be to provided goals.
4.2 MULTI-AGENT HIBRARCHICAL REINFORCEMEN LEARNING
We construct a multi-agent learning structure that takes advantage of hierarchical design or Multi- Agent Hierarchical Reinforcement Learning (MAHRL). Each agent has its own higher level policy (Multi-Agent Navigation Controller (NC)) (gg) o, Bhi) and a shared task agnositic lower level policy (LC) x(alo, g. Olo). This method allows us to introduce more structure into the difficlut multi-agent optimization problem. This change alters the underlying MDP, such that the policy is queried for new action every & timesteps. This also changes the MDP method by reducing the dimensoinality of the action space to specifiying goals g while using the low-level policy to produce more temporally consistent behaviour in the original action space. The low-level policy is used in a deterministic man- ner futher reduce variance introduced into the problem. We will use the notation ा (aa lo,, 0h4, Olo) to denote the policy induced by the pair of policies. While current research shows that it is challeng- ing but possible train a two-level hierarchy concurrently (Nachum et al. 2018 Levy et al.. 2017b), we instead pretrain the lower level policy and leave it fixed and shared across all agents. This re- duces the MARL problem from learning the details of locomotion via joint torques for each agent to learning goal-based footstep plans for each agent.
The use of HRL is key to the method. When the challenge in MARL is dealing with what can be large changes in the distribution of states visited by the agent, the use of temporally correlated structure given by the goal-conditioned I.C significantly reduces the non-stationarity. Not only is each agent sharing its network parameters with each other agent, but this portion has also been carefully constructed to provide structured exploration for the task, thus greatly reducing the number of network parameters that need to be learned. This is in contrast to centralized methods that take a step away from the goal of solving the heterogeneous problem in a scalable way. We extend the analysis of this work by combining the PPO algorithm in a MARL - parameter shariing setting. Details of the algorithm can be found in the supplementary material. The use of the LC controls the way de, (o.) can change for each agent making casier for each agent to estimate other agents potential changes in behaviour because the LC is already trained to produce a specific type of useful behaviour that a subset of the full space. We also create a version of MADDPG that uses the HRL structure and show that it does not perform as well.
5 LEARNING HIERARCHICAL MULTI- NAVIGATION CONTROL
To solve the hierarchical learning problem, we train in a bottom-up fashion sharing the LC policy among heterogenous NC policies. The levels of the hierarchy communicate through shared actions and state in feedback loop that is meant to reflect human locomotion. The NC's objective is to provide footstep placement goals aи IL for the LC as seen in the right hand side of Figure I These footstep goals are produced as two-step plans. Each step is parameterized with its root- placement, angle on the ground, centre of mass heading vector, and a time signature. The NCs are queried for a new footstep action every 0.5 The NCs decides what action to take based on the egocentric velocity field F. in front of the agent, its pose Scher and the NC goal gn, which together form the NC state C' = {E, Scher, 9H }. seen in the left box of Figure I
5.1 STATE SPACE
In several studies, it has been shown that optic flow, an inherently egocentric phenomenon of map- ping velocities to regions of vision, is key to sighted locomotion (Bruggeman et al. 2007 Warren Jr
4

---

Under review as a conference paper at ICLR 2020
Higher Level Policy
Lower Level Policy
NC States
LC Actions
Navigation Controller (NC)
NC Actions LC Goele
s(a/s)
Conwhet saayrr
Donse
ocommttio Controller (LC)
2-step
An overview the MAHRL approach. From left to right: the NC state includes the relative goal position and distance, an egocentric velocity field capturing the relative velocity of obstacles and other agents, and the physical character link positions and linear velocities; for each agent this state is input to multi-layer CNN, including high-level actions are footstep plans as g. which produces the angle-axis axis targets for joint PD controllers.
Figure I:
dense hidden layers, and outputs actions-the value function
similar network. These form of two-step plans dictating future foot placements; the LC consumes these
al. 2001). Additionally, the visual field directly impacts walking ability and locomotion stabil- ity (Jansen et al. 2011). Taken together, vision's 's role in locomotion forms an egocentric velocity field where perceived distance and movement play different roles locomotion control (TTurnno) et al. (2005). This evidence has been used previously to define multi-agent navigation models, both by constructing a discretized egocentric field (Kapadia et al. 2009) and by learning the discretization of egocentric field (Long al. 2017
The NC uses as input an egocentric velocity field relative to the agent's location and rotation. This egocentric velocity field E is 32 32 samples over space of 5x5 m, starting 0.5 behind the agent and extending 4.5 in front, shown in the left hand side of Figure [ The velocity field consists of two image channels in the and y directions of square area directly in front of the agent, where each sample is calculated as the velocity relative to the agent (Bruggeman et al.) 2007 Warren Jr al. (000)) The current pose the agent is included next, followed by the NC goal. The NC goal consists of two values, the agent relative direction and the distance to the current spatial goal location.
5.2 SIMULATION ENVIRONMENT & TRAINING
We construct collection of physics- based simulation tasks to evaluate the method. At initialization, each agent is randomly rotated, and the initial velocities of the agent's 's links are determined by selected motion capture clip using finite differences and rotated to align with agent's reference frame. Goal locations are randomly generated in locations that are at least m away from any obsta- cle. Each agent randomly placed in the scene such that it does not intersect with any other agent or obstacle. The number and density of agents in the simulation vary depending on the task. For train- ing, we found that starting with 3 agents in the environment is good trade-off between computation cost and the generalization ability of the resulting learned policy. Environment specifies are given in Table We consider the simulation and training environment to be, to the best of our knowledge, another novel contribution. While some simulators exist that support physics-based simulation for robots (Brockman et al., 2016 Tassa et al., 2018), few support more than one agent, with at most 2 which interact beyond physical simulation and actually solve their tasks with respect to each other. Other libraries focus on supporting different kinds of MARL configurations for particle-based agents (Lowe et al. 2017). Our proposed approach represents the first physics- based simulation of its kind that supports MARL.
The NC uses convolutional layers followed by dense layers. The particular network used is as fol- lows: 16 convolutional filters of size 6 x 6 and stride 2 x 2, 16 convolutional filters of size 3 x 3 and stride 1 x 1, the structure flattened and the character and goal features Schars 9H are concate- nated, a dense layer of 256 units and dense layer of 128 units are used at the end. The network uses Rectified Linear Unit (ReLU) activations throughout except for after the last layer which uses a tanh activation that outputs values between [-1, All network inputs are standardized with respect
5

---

Under review as a conference paper at ICLR 2020
Name
agent count obstacle count
size
agent direction
obs location
Pursuit
3
0, 10
10 x 10
random
random
Soccer
4
x towards centre +N
none
Mall
3,5
[0, 10]
random 10 x 20 right (O
10
random
Bottleneck
3,5
around
Table 1: Scenarios and their main parameters.
-20
-25
Eme Sidza
Ene Staps
(a) 5 Agent Mall
(h) agent Bottleneck
Figure 2: Comparative study of the learning curves of MAHRL, MADDPG, and PPO in the Mall scenario.
to all states observed so far. The rewards are scaled by running variance. That is, the variance is computed from running computation during training that updated after every data collection step. The batch size used for PPO is 256, with a smaller batch size of 64 for the value function. The policy learning-rate and value function learning-rate are 0.0001 and 0.001, respectively. The value function is updated four times for every policy update. The NC also uses the Adam stochastic gradient optimization method (Kingma Ba 2014) to train the Artificial Neural Network (ANN) parameters.
RESULTS
In this section, we demonstrate the efficacy of MAHRL. We introduce new of challenging multi-agent simulations and show that the previous method can not solve these tasks. However, using MAHRL many of these tasks can be solved with modest amount of data and computational power. We separate our evaluation into four sections. We examine the performance of MAHRL in terms of training, guantitative metrics, and learned policies. Then we examine the performance in terms of computation cost and generalizability over the number of agents in the environment.
Fean Roandd
-15
Emu Stape
Leb
Simulated Actios x327680, Training Updates x10240.0
(a) 3 Agent Pursuit baseline comparison
(b) Pursuit indiviual agent learning curves
Figure Comparative study of the learning curves MAHRL, MADDPG, and PPO the Pursuit scenario.

---

Under review as a conference paper at ICLR 2020
BLLINCC
A
Strpe
(a) Soccer learning comparison
(b) Soccer
Figure 4: Comparative study of the learning curves of MAHRL based on PPO (blue) and MAHRL using TD3 (red) in the Soccer scenario. The yellow and green agents the same team, while the blue and red on the other team.
6.1 LEARNING
We evaluate MAHRL in complex multi-agent environments by first examining the performance of the reward function over training episodes. We show that MAHRL. performs much better than the basic PPO algorithm without any hierarchical structure in the 5-agent Mall in Figure After lengthy training session, the basic PPO is not able to even produce a standing behaviour. We find that MAHRL performs well and is able to learn navigation behaviour after short amount of training. We find that the hierarchical design provides a significant improvement in this case. We also compare to MADDPG in Figure structure, it struggles to produce good coordinated behaviour. We believe this is related to the large Q-network that needs to be learned for MADDPG. Qualitatively, throughout training, the increased control complexity, our method able to learn successful navigation strategies shown in Figure 8 Each agent in the scenario quickly develops strong navigation behaviours that become more conservative over time as agents value avoiding collisions. This can result in agents taking longer paths in the environment.
and find that, while it does better than PPO as it also uses the HRL
with
MAHRL, is also applied to a Pursuit environment. In this environment, there is one agent (agent O) with the same navigation goal as in previous environments. As well, two additional agents (pursuer 0,1) that have the goal of reaching agent O. This is accomplished by setting high-level goal Iu for the pursuers to the location agent In Figure Bal we show comparative analysis of the learning curves of MAHRL, MADDPG, and PPO. We note that qualitatively, the three agents all begin to increase their average reward via their navigation objective, as learning progresses the pursuing agents outperform agent O, shown in Figure 3b and as they get better agent 0 has an increasingly difficult time reaching its own navigation targets. An example of this behaviour is shown in Figure
Last MAHRL is applied to 2-on-2 soccer simulation, Soccer shown in Figure 4b In this simulation, there are two teams
bipedal agents. The goal of the members of each team to kick the ball into the other team's 's goal area (shown in green). The task is particularly challenging to learn each agent now has to learn with one cooperative agent and two adversarial agents. We find that MAHRL works well at learning policies for agents to locate and kick the ball into the opposing team's 's goal. The learning curves for two different methods that use MAHRL are shown in Figure 4a on based on PPO and another based on TD3, Our results often produce imbalanced teams where one team becomes much better at the game than the other. This can be seen in the accompanying video on the website.
By extracting the gradients on the input features for the value function in the learned model, we can examine some artifacts of what is learned. Recall that for input, we include in the state of the NC a simple model of an egocentric perceptual field- a rectangular region in front of the agent. We show that our mod- els learns two important aspects navigation with respect to this field. The magnitude of the velocity field gradients in our learned model reveals that the learning process has developed
(a) ego field x egocentric velocity field attenuated with distance. This be in Figure 5a Interestingly,
(b) ego fieldy
7

---

Under review as a conference paper at ICLR 2020
000
VTARS Hanggenss
PPO Hancgeas
MANRL
MADOPa
(a) Mall
(b) Pusuit
Figure 6: Comparative analysis of collisions counts across all baselines, MADDPG, MAHRI with and without heterogencous agents, and PPO in the pursuit/tag scenario. MAHRL outperforms both MADDPG PPO. In this game, the collisions are indicators of poor policies leading to negative collisions during pursuit and evasion.
2
Figure 8: Rasterized images from mall environment with humanoid agents navigating and avoiding each other while seeking goals. video for this example can be found here
this field is biased toward the rightward direction (down the figures). This bias supports reciprocal collision avoidance in counter flows.
6.2 QUANTITATIVE COMPARISON
To evaluate learned navigation strategies quantitatively, we capture the mean number of collision events overall agents for each episode in several instantiations of the Mall environment. We perform this study over the state-of-the -fte-of-theeart MADDPG method, MAHRL with and without Heterogeneous agents, and PPO. The results over several random seeds. The basic PPO algorithm does not appear to learn anything. A Kruskal- Wallis rank-sum test and post-hoc Conover's test with both FDR and Holm corrections for ties show the MAHRL methods significantly outperform others (p 0.01).
shown in Figure ba For each model,
perform 155 policy rollouts
To evaluate the pursuit and evade strategies quantitatively, we capture the mean number of collision events overall agents for each episode in several instantiations of the game, We perform this study over the MADDPG method, MAHRL, and PPO. The results can be seen in Figure For each model, we run 155 simulations over several random seeds. A Kruskal-Wall Conover's 's test with both FDR and Holm corrections for ties show the MAHRL and MADDPG methods significantly outperform PPO (p < 0.01). However, MAHRL is not significantly different from MADDPG (p = 0.45), but the distribution is skewed lower than MADDPG with more zero collision samples.
rank-sum test and post-hoc
6.3 QUALITATIVE RESULTS
To evaluate the NC policies qualitatively, we show that agents learn to navigate arbitrary scenarios while avoiding collisions with both obstacles and other agents. First, we show a rasterized version of example episode from the Pursuit environment in Figure 7 where agents learn to corner and tackle. Then, we show that agents can successfully and continuously navigate complicated environ- ments of forced interactions,
in Figure 9a Kapadia et al., 2011). Finally, robust clogging and trampling are shown in both low and high-density bottleneck egress scenarios, respectively Fig-
ure 9h
8

---

Under review as a conference paper at ICLR 2020
(a) Agent Mall
(b) Bottleneck
Figure 9: (a) Agents reaching series of targets in arbitrary environments (images in raster order). (b) Egress scenarios with group of (left) and (right) 21 agents, The density of the second group results in physies- based bumping, pushing, and trampling.
bottlenedk
mell
bottleneck muult-tokk Bectleneeck_mutitttee
bottleneck
40
Time Minutes
30
20
Average
-6
10
8
12
Agent (a)
Agent Count
(b)
Figure 10: The performance the method from two quantitative perspectives, (a) the computational perfor- mance with respect agent count and (b) the generalization performance with respect average reward value.
6.4 COMPUTATION AND GENERALIZATION
The deep integration of physical character control and distributed multi-agent navigation comes with cost di- rectly dependent on the number of active agents. In this section, we show two results the same experiment, the computational costs of increasing the number of agents and the model generalization to more difficult scenar- ios. For two scenarios, mall and bottleneck, the num- ber agents is increased, and we record the average reward and computation time from the simulation. The agent-computation curve in Figure T0(a) indicates a lin- car trend computational cost. While at agents counts in the 20s the simulation is not real-time, the most com- putationally expensive part is not the learning system but the physics simulation. Computationally efficient articu- lated body simulation is an active area of research (Erez et al. 2015). For accurate and stable simulation, we use physics time-step of 0.0005
Figure 7: Rasterized overlays from the pur- suit environment, where the pursuer agents (red) learn to work together to corner and tackle the navigating agent (blue). A video for this example can be found here
To evaluate the learned policy's 's ability to generalize with group size, we vary the number agents for select tasks after a homogeneous policy has been trained. The longer the policy can maintain high average reward, the better the generalization. In addition, the average reward for two different types of policy training styles is compared. The first method trains single task at time; the second method uses multi-task learning in hopes that more generalizable task-independent structure acquired. The multi-tsk method, often preferring optimize easier tasks, does not appear to learn more robust

---

Under review as a conference paper at ICLR 2020
policies compared to the scenario-space based method (Kapadia et al. 2011). All generalization results can be seen in Figure b). However, generalization remains a known and open issue of DRL methods (Zhang et al. 2018)
7 CONCLUSION
In this paper, we present novel method where, for the first time, multi-agent navigation and physi- cal character control integrated. Multiple heterogeneous interacting agents can experience phys- ical interactions, handle physical perturbations, and produce robust physical phenomena like falling and slipping. To achieve this, we developed an integrated model of MARL and HRL with partial parameter sharing- The evaluation of this approach shows how valuable it is for addressing the non-stationary learning problem of MARL in complex multi-agent scenarios. particular, the het- erogencous Pursuit and Soccer scenarios learn complex routing and tackling behaviours akin to state-of-the-art competitive self-play approaches.
While method produces promising results, the work is limited by the fixed LC partial parame- ter sharing. There room for research the area training the LC and NC concurrently. For the NC, introduced a set of reward functions to encourage human-like behaviour while navigating with other agents. The literature motivates these rewards, but balancing them is its own challenge. In the future, it may be beneficial to use additional data-driven imitation terms encourage human- like paths. Finally, considerable effort was made to create combined locomotion, navigation, and behaviour controller that robust the number of agents in the simulation. However, robust gen- cralization remains an open problem.
REFERENCES
Andreas Argyriou, Theodoros Evgeniou, and Massimiliano Pontil. Multi-task feature learning. In Advances in neural information processing systems, pp. 41-48, 2007.
Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, and Igor Mordatch. complexity via multi-agent competition. arXiv preprint 1rXiv:1710.03748, 2017. URL http:
Emergent
/amxiw.org/ab//7110.03744
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.015400, 2016.
Hugo Bruggeman, Wendy Zosh, and William H Warren. Optic flow drives human visuo-locomotor adaptation. Current biology, 17(23)2033-2000 2007.
Lucian Bu, Robert Babu, Bart De Schutter, et al. A comprehensive survey of multiagent reinforce- ment learning. IEEE Transactions on Systems, Man, and Cybernetics, Parl C (Applications and Reviews), (8)2):156-17,, 2008
Bullet. Bullet physics library, 2015. http://bullephyssis.oorg
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/IAAI, 1998:746-752, 1998.
Peter Dayan and Geoffrey Hinton. Feudal Reinforcement Learning, pp. 271278. Morgan- Kaufmann, 1993. http:///ppppessss..cppp///////ee hww.r/forrremmet---rainn.nnooo
Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. arXiv:cs/900001, May 1999. URL httt://aariivorrgabbs////0000010 arXiv: cs/9905014
Tom Erez, Yuval Tassa, and Emanuel Todorov. Simulation tools for model-based robotics: Compar- ison of bullet, havok, mujoco, ode and physx. 2015 IEEE international conference on robotics and automation (ICRA), pp. 4397-4404. IEEE, 2015.
10

---

Under review as a conference paper at ICLR 2020
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. In International Conference on Learning Representa- tions, 2019. URL http:://pppmaaooo....tttto/////s
Petros Faloutsos, Michiel Van de Panne, and Demetri Terzopoulos. Composable controllers for physics- character animation. In Proceedings of the 28th Anmual Conference on Computer Graphics and Interactive Techniques, pp. 251-260. ACM, 2001.
Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pp. 2137-2145, 2016.
Tao Geng, Bernd Porr, and Florentin Wörgötter. A reflexive neural network for dynamic biped walking control. Neural Computation, (655):156-999 2006.
Abhishek Gupta, Russell Mendonca, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Meta- reinforcement learning of structured exploration strategies. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Desa-Biaachh, and R. Garnett (eds.), Advances in Neural Infor-
mation Processing Systems 31, pp. 5302-5311. Curran Associates, Inc., 2018. URL http://p:// hwppsccppaee7777mmmmttermmmoormo wew.eeeeeeeeeee-eeennneeeeee of-ling-oo-ooae off-stucturred wwpploortionnsttateeeeaee
Jayesh K. Gupta, Maxim Egorov, and Mykel Kochenderfer. Cooperative multi-agent control us- ing deep reinforcement learning. In Gita Sukthankar and Juan A. Rodriguez-Aguii (eds.), Au- tonomous Agents and Multiagent Systems, pp. 66-83, 83, Cham, 2017. Springer International Pub- lishing. ISBN 978-3-319-716822
Dongge Han, Wendelin Boehmer, Michael Wooldridge, and Alex Rogers. Multi-agent hierarchical reinforcement learning with dynamic termination. In Proceedings of the 18th International Con- ference on Autonomous Agents and MulliAgent Systems, pp. 2006-2008. International Foundation for Autonomous Agents and Multiagent Systems, 2019.
Nicolas Heess, Greg Wayne, Yuval Tassa, Timothy Lillicrap, Martin Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. arXiv preprint urXiv:1610.055 2016.
Sander EM Jansen, Alexander Toet, and Peter J Werkhoven. Human locomotion through multiple obstacle enviromment: strategy changes as a result of visual field limitation. Experimental brain research, 212(3):449-456, 2011.
I. Kaelbling Learning to achieve goals. In International Joint Conference on Artificial Intelligence (ICAII) volume vol.2, pp. 1094 - 8, 1993.
Mubbasir Kapadia, Shawn Singh, William Hewlett, and Petros Faloutsos. Egocentric affordance fields in pedestrian steering. In Proceedings of the 2009 symposium Interactive 3D graphics and games, pp. 215-223, ACM, 2009.
Mubbasir Kapadia, Matt Wang. Shawn Singh, Glenn Reinman, and Petros Faloutsos. Scenario space: characterizing coverage. quality, and failure of steering algorithms. In Proceedings of the 2011 ACM SICCRAPHHEungrapphics Symposium on Computer Animation, pp. 53-62. ACM, 2011.
Ioannis Karamouzas, Brian Skinner, and Stephen J Guy. Universal power law governing pedestrian interactions. Physical review letters, 113(2))233700, 2014.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Andrew Kun and W. Thomas Miller III. Adaptive dynamic balance of biped robot using neural networks. In Proceedings of the IEEE International Conference on Robotics and Automation, volume pages, pp. 240-245. IEEE, 1996.
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv:1712.00%. 00948 [cs], Dec 2017a. URL htthh://aariiooorgaaa//////20000 arXiv: 1712.00948.
11

---

Under review as a conference paper at ICLR 2020
Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko. Learning multi-level hierarchies with hindsight. arXiv preprint arXiv:1712.00948, Dec 2017b. URL http:///arxxv.or//as///// 17112 00948 arXiv: 1712.00948.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Pinxin Long, Wenxi Liu, and Jia Pan. Deep-learned collision avoidance policy for distributed mul-
tiagent navigation. IEEE Robotics and Automation Letters, 2(2):656-663) 663, 2017.
Ryan Lowe, YI WU, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- agent actor-critic for mixed cooperative-competiine environments. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 6379-63999 6390. Curran Associates, Inc., 2017.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5334- 5343, 2017.
Josh Merel, Arun Ahuja, Vu Pham, Saran Tunyasuvunakool, Siqi Liu, Dhruva Tirumala, Nicolas Heess, and Greg Wayne. Hierarchical visuomotor control of humanoids. CoRR, abs/1811.09655 2018. URL http:///xaxxww.or//bb///// 1.09656
W. Thomas Miller III. Real-time neural network control of biped walking robot. Control Systems, IEEE, 14(1):41-44 1994.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529, 2015.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforce- ment learning. (rxiv:/805.08200 [cs, stat], May 2018. URL http://arxiivorr//ab///8855 arXiv: 1805.08296.
Ashvin Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine. Visual reinforcement learning with imagined goals. arXiv:1807.04742 [cs, stat], Jul 2018. URL http:
/arxiw.orrgabb//880.007774 04742
arXiv: 1807.04742.
OpenAl. Openai five. https///bbgg.oppaaa.commmppenerr 2018.
Liviu Panait and Sean Luke. Cooperative multi-agent learning: The state of the art. Autonomous agents and multi-agent systems, 11(3):387-444, 2005.
Xue Bin Peng, Glen Berseth, KangKang Yin, and Michiel Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning, ACM Transactions on Graphics (TOG), 36(4):41, 2017.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal Policy Optimization Algorithms. ArXiv e-prints, July 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. International Con- ference Learning Representations (ICLR 2016), 2016.
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. In Proc. ICML, 2014.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, and et al. Mastering the game of go without human knowledge. Nature, 550(7676):35435,, Oct 2017. ISSN 1476-4687. doi: 10. 00.10388/auur222700..
12

---

Under review as a conference paper at ICLR 2020
Joseph Suarez, Yilun Du, Phillip Isola, and Igor Mordatch. Neural MMO: A massively multiagent game environment for training and evaluating intelligent agents. CoRR, abs/1903.00784, 2019.
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 1 112(():18121, Aug 1999. ISSN 0004-3702. doi: (00016/00004437722990000000339
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
Hongyao Tang, Jianye Hao, Tangje Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Changjie Fan, and Li Wang. Hierarchical deep multiagent reinforcement leamn ing. arXiv preprint FrXiv:1809.0933.0 2018.
Y. Tassa, Y. Doron, A. Muldal, T. Erez, Y., D. de Las Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq, T. Lillicrap, and M. Riedmiller. DeepMind Control Suite. ArXiv e-prints, January 2018.
Yuandong Tian, Qucheng Gong, Wenling Shang, Yuxin Wu, and C. Lawrence Zitnick. Elf: An ex- tensive, lightweight and flexible research platform for real-time strategy games. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2659-2669. Curran Associates, Inc., 2017.
Kathleen A Turano, Dylan Yu, Lei Hao, and John C Hicks. Optic-flow and egocentric-dirrecioon strategies in walking: Central vs peripheral visual field. Vision research, 45(25-26):3117-311222 2005.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. arXiv: [es], Mar 2017. URL http://aarii..oggga//////....111
arXiv: 1703.01161.
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul. Hado van Hasselt, David Silver, Timothy P. Lillicrap. Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Stareraft II: new challenge for reinforcement learning. CoRR, abs/1 (47822, 2017.
William H Warren Jr, Bruce A Kay, Wendy D Zosh, Andrew Duchon, and Stephanie Sahuc. Optic flow is used to control human walking. Nature neuroscience, 4(2):213,200) 2001.
KangKang Yin, Kevin Loken, and Michiel van de Panne. Simbicon: Simple biped locomotion control. ACM Transactions on Graphics, 26t3):Article 105, 2007.
Petr Zaytsev, Javad Hasaneini, and Andy Ruina. Two steps is enough: no need to plan far ahead for walking balance. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 6295-6300. IEEE, 2015.
Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937, 2018.
Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weinan Zhang, Jun Wang, and Yong Yu. Magent: many-agent reinforcement learning platform for artificial collective intelligence. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018
8 SUPPLEMENTARY MATERIAL
8.1 LOCOMOTION CONTROLLER
8.1.1 REWARD
We found that a combination of imitation and end effector rewards leads to a robust and reactive motion. The goal is to reward joint angle targets which result in mimicking source motions. The
13

---

Under review as a conference paper at ICLR 2020
combination of these competing rewards leads to complex balancing problem of many objectives- trading off imitation behaviour for footstep matching.
8.1.2 IMITATION
The imitation method is predicated on time indexed 0.5 long reference motions (t) created from motion capture data and augmented with footstep parameterized labels. The difference between the agent and the desired pose to imitate is computed as the weighted difference between the time indexed pose and the agent rLp = (x. Σ, | shar - q' (t) l²). The motion database contains 7 of segmented motion that coarsely covers the space of possible footstep positions and the motions necessary to realize those footsteps. The accuracy and diversity of the Low-I Controller (LL.C)'s
motion can be increased or made more robust with additional motion if desired, including stylized, disordered, running, and more.
However, if the agent matches these footstep goals too well, the smoothness of the motion will degrade. A balance is struck between learning robust footstep placement strategies and perfect mimicry the reference motions and foot placements guide naturalness and should not be over-fitted.
8.1.3 END EFFECTOR
An additional reward is given to encourage the policy to match the end-effector positions in the imi- tation motion F'e exp(-0.15 >e [| - q (t) Reward is also given for how well the agent matches the imitated motion's Centre of Mass (COM) TLcom cxp(- 002 Σ coom
q nmam (3)
8.1.4 TORQUE PENALTY
To smooth the actions generated from the control policy, a reward for maximizing the above rewards with minimal torque cost is used ftr exp(-0.2 >: T( These torques are normalized by specified torque limits that keep the agent from displaying unrealistic strength.
8.2 TRAINING & SIMULATION ENVIRONMENT
The LLC state includes the proprioceptive- like joint information. In particular, the components of Schor consist of the COM relative distances and velocities of each links as well as the rotation and angular velocities of each link as shown in Figure d). The trained policy actions are in the form of angle-axis targets for per-joint Proportional Derivative (PD) controllers.
this way, the action is used to set the desired position of each joint of the agent. This can be seen in Figure [e). The method is also able to learn policies for agents of different types. In Figure S we additionally show the method using a full humanoid LLC character.
The agent is simulated with the physics-based rigid body simulation environment Bullet Bullet (2015). The policy is trained in an on-policy fashion. At the beginning of each episode a goal ΦL is sampled from the simulation and updated every 0.5 s. Episodes end when either a time limit T is reached the agent falls. Between training rounds 4096 transition tuples are collected by simulating episodes in parallel. After enough tuples have been collected 16 minibatch updates are performed with batch size of 256. The value function is updated 64 with a minibatch size of 64. For the policy and value function. The policy and value function are both modeled using a ANN with 2 hidden lay- ers of size 512 and 256. The network uses ReLU activations through except for after the last layer where a tanh activation that outputs values between [-1, 1] is used. The Adam stochastic gradient optimization method is used to train the ANN parameters Kinggmaannmaaaaaa & Ba (2014). The learning rates for the policy and value function are 0.0001 and 0.001 respectively. Details related to the learning algorithm can found the supplementary material.
Many previous methods have created robust controllers via imitation. These methods were not in- tended to be used in crowds or dynamic environments with potentially random disturbances from other interacting agents in the simulation. Given that we intend to this agent in physical multi- agent simulation where the agent may bump into other agents or obstacles when trained a crowded setting, we apply additional methods simulate pushes that may be encountered. That is, random pushes between 50- 150 n are applied every 3- - 5 s for a duration of 0.1 0.3 to increase the robustness of the controller. Similarly, the motions in the motion database start from the same facing
14

---

Under review as a conference paper at ICLR 2020
direction. To make the agent more robust to distributed crowd scenarios, the agent's initial facing direction is randomized during training.
8.2.1 NC REWARD
Navigation combines the desire to move towards goals, while avoiding collisions with other objects, in an energy efficient way. The methods purpose is to implicitly learn low effort local turning be- haviour while avoiding collisions without the need to explicitly describe its operation. To elicit this behaviour, combination of reward signals are used. Primarily, a dense reward is used to encourage the agent to walk the direction of its current goal.
THd = ()p(-(min(0, (Near * //oom) - )com))²)
where Ucom the agent's ' velocity and ufar is a normalized vector in the direction of the goal. For this work, a desired speed of Unom 1.0m/ is used.
directional reward is not enough to encourage the agent to proactively reach its goal. To reinforce the importance of goal reaching behaviour, large reward ggool for reaching the goal is aadeddbeing within V2 m. In this work, the value ggool 20 is used and comes from horizon 1/(1 - ?) and the maximum reward the agent can otherwise receive is maz(ra) = 1, a reward that will be more important than travelling the goal should be at least horizon maz((a). This reward greatly increases the goal reaching behaviour. However, it can have the unintended effect of making the agent seek its goal aggressively, by trampling other agents. To reduce this behaviour, a repulsive cost was added when the agent is within v3 m other agents
(a) Σ
-(rs + ((-ist((a,a))
d'E(A-a}
where A is the of agents in the simulation, and dist(a, a') computes the Euclidean distance between the COM of agents a and a'. We empirically found r. = 2.5 and I = 3.0 work well for defining distance dependent penalty in this case. This is intended roughly approximate the power law of pedestrian interactions (Karamouzas et al. 2014). A similar repulsive cost is applied between the agent a and obstacles OB in the scene (Ho(a) LoBEOB -2.5 for each obstacle within 1m of a. The high reward for reaching the goal makes the agent very single-minded. The obstacle penalty introduced to prevent the agents from using obstacles as affordances to regain balance.
Many RL simulation environments use flag to indicate the episode or simulation end. This indicates that either the agent has reach its time limit or the agent has entered into an unrecoverable area of the state space, such as fall. With multiple agents, the likelihood of a fall is high. It is not clear what to do when there are multiple agents being simulated. Terminating early, when one agent has fallen, is sub-optimal for other agents doing well, and waiting for every agent to fall wastes compute resources while most agents are collecting unhelpful data. We chose to reset the simulation when more than half the agents have fallen. However, fallen agents continue to act and need to be heavily penalized, so they receive a fixed reward of -5 which represents the largest penalty in our system.
The NC's final reward function is combination of the task rewards and behavioural costs: if fallen (rna + (aa + Tm)(a) otherwise.
(a (-5
(6)
8.3 LEARNING DETAILS
The modification to this algorithm for the MARL method is to change line 7 7- 8 to execute action for each agent in the simulation in parallel.
The Inputs are standardized wrt to all states seen so far. The rewards are divided by the variance. The variance is computed from a running computation during training that is updated after every data collection step. The advantage is batch normed.
Hyper Parameter Exploration Parameter exploration is a key process in acheiving the best re- sults for DRL methods. In this work explored different network architectures, policy variances, annealing the variance, learning rates and activation types. The results with the best performance are reported in the paper.
15

---

Under review as a conference paper at ICLR 2020
Algorithm Goal-Based Learning Algorithm
1: Randomly initialize model parameters 0z and O..
2: while not done do
3: for E {0............. N do
4: Ti + [
5: for E {O...., T) do
6:
0t f- TT 0x)
7:
9,-1, Ft < execute a, in environment
8:
x < 8t: Oz, Tt
9:
8t 6 8c+1
10:
end for
end for
12:
Update value function V-(() parameters 6, using TN.
Update policy parameters 0z using (-o, TN)
14:
end while
0.8
Mean Keward
0.4
0.2
anneal exploration; True
exploration rate: 0.2
threshold: 0.25
50
100
150
200
250
Simulated Actions x81920, Training Updates x1280.0
Figure 11: Hyperparameter exploration learning curves for the LC training.
8.4 RESULTS DETAILS
8.4.1 LC HYPERPARAMETER EXPLORATION
The LC hyperparameter exploration learning curves in Figure Iillusttaae the process of finding op- timal training parameters. found that annealing the policy variance over time did not increase the learning efficiency. As well, the best settings for the policy variance and [I_hhreholdd are 0.2 and 0.25 respectively.
8.4.2 MULTI-AGENT NAVIGATION UNIT TASKS
In this section, we show several multi-agent navigation tasks and their qualitative performance us- ing our method. These are rendered as birds-eye views of scenarios with each agents trajectory over time rendered. Each agents goal is rendered as red point within red circle and the shortest linear path to that from the agent's 's current position rendered as a red line. We show that agent's 's learn to navigate towards arbitrary goals repeatedly hitting its mark with each new goal in Figure 12 Predic- tive reciprocal collision avoidance is important in multi-agent navigation and lends naturalness to qualitative results. We show method learns high value reipcrocal collision avoidane strategies, and we overlay the sample points for the velocity state space to illustrate how this is acheived in Figure 13 We show that this generalizes the introduction of obstacles in both Figure 14 & 15
16

---

Under review as a conference paper at ICLR 2020
Figure 12: The agent moves towards and reaches goals repeatedly (top-left to bottom-right). The learned model produces smooth goal seeking trajectories show in in blue.
(a)
(b)
Figure 13: Three examples demonstrating reciprocal collision avoidance. The size and position of the egocen- tric state sampling field relative of the agent is shown in (a).
Figure 14: Reciprocal collision avoidance with obstacles. Each agent's initial position is the target location of the other agent.
17

---

Under review as a conference paper at ICLR 2020
Figure 15: An agent reaching goals in arbitrary complex environments. The red circle indicates the final goal.
8.5 ADDITIONAL DISCUSSION
8.5.1 LC AGILITY
While often in the crowd simulation literature it is common have agents that turn point produce holonomic motions, this unrealistic for articulated agents. Interestingly, the agent learned stopping and in-place turning behaviour which not contained in the imitation data the LC was trained on, indicating the system can generate behaviours beyond its design. However, the agents in this work do not make unrealistic sharp turns. This in part related the NC being able to avoid falls, however there are many other factors: The LC motion capture data, IL selection during LC training and that the LC was not trained with other agents. Progress in any of these areas can improve the responsiveness of the LCC and is left for future work.
While the L.C is goal driven, this goal is only based on foot placement, fully interactive agent can have many other types of short term goals, including where to put one's 's hands to manipulate items the environment, like doors.
18


Reviews:
[{"review_id": "rJgHSAbAFH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multi-agent hierarchical reinforcement learning algorithm so that multiple humanoid robots can navigate in multi-agent settings (e.g. avoid collisions, collaboration, chase and escape) in a physically simulated environment. The key difference of this paper with the prior work on MARL is that it used an accurate physics simulation of humanoid robots. This is the main reason of using the hierarchical RL. \n\nIn general, I like this paper. It is an important step towards multi-agent learning in complex physical environments. The results look appealing, too. However, I voted for \"Weak Reject\" for two reasons. First, the technical contribution is lean. Neither the multi-agent learning or the hierarchical learning of the algorithm is novel. The combination of these two methods seems straightforward. Once a low-level walking controller is trained, the high-level multi-agent navigation control is not much different from simple environments, e.g. point mass control, used in the previous works. I do not understand the \"deep integration of MARL and HRL\" that is claimed in the Introduction. I also do not agree with another claim that \"We consider the simulation and training environment to be another novel contribution... few simulator support more than one agent, at most 2\". In most of the simulators that I am familiar with, such as Mujoco, Bullet, DART, it is straightforward to add multiple simulated robots.\n\nSecond, the writing can be greatly improved. Almost half of the technical details are buried in \"8. Supplementary material\". Since it is not fair to use \"Supplementary material\" as a way to extend the page limit, I will make my judgement of the paper solely based on the contents up to Section 7. In the main text (up to Section 7), there is no mentioning of how the low-level controllers are learned, and how to combine PPO in a MARL partial parameter sharing setting. I think that these are important details and may also be the contributions of this paper. Most of these should be moved to the main text. \n\nHere are some more suggestions on writing:\n1) Certain paragraphs in the main text can be significantly shortened, such as the reward shaping in Section 5.2. \n2) It would be great if the paper can clearly define the experiments: \"waypoint\", \"oncoming\", \"mall\", and \"bottleneck\".\n3) The paper needs a thorough proof-reading. There are many grammar mistakes, typos, missing citations. For example,\npromiss->promise\nweek signal->weak signal\nmissing citation [?] in page 3\nreuse the same symbol v_{com} for agent's velocity and desired speed in eq(3)\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SyeMmx63FS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The submission proposes a method for hierarchical RL in multiagent settings. In particular it proposes to explicitly decouple training of a high-level and low-level controller with grounded the controller interface as goals in the environment to reach for the low-level controller. The model is trained via PPO with GAE and evaluated on a small set of multi agent locomotion tasks.\n\nThe paper is overall well written and intuitive but limited in evaluation and novelty (see e.g. [1,2] ) with only limited modifications (sharing low-level controller) for the multi agent case. Furthermore, the experimental section does not compare to other forms of hierarchical approaches for MARL, and generally only provides a single comparison to PPO & MADDPG. To evaluate the impact of the proposed changes in this paper, one would have to perform extended evaluations and ablations for the submission. \n\nA large part of making the MA system work well is based on reward shaping which nearly fills all of page 5. This is clearly interested in as far as solving this particular task but does not provide any general insights for the design of (MA)RL algorithms. \n\nThe experimental section includes various mistakes (see under minor) and misses to describe figures, leading to the assumption that additional time is required for a more detailed evaluation of the algorithm (including more domains and in particular baselines).\nRegarding the challenges (and focus on learning simple tasks), reference [3] might be of interest to the authors.\n\nMinor\n- Direct duplication of text between parts of section 5.3 and 8.3 leading to the duplication of the error of describing the value function learning rate as 0.000.\n- Self-referential sentences in the supplementary materials (i.e. referral to itself)\n- Missing references on page 3\n- The egocentric velocity field is not described (section 5)\n- Section 3.1: maximize\n- The wording new paradigm in MARL might be unsuited given existing work on complex domains.\n‘Our proposed approach represents the first physics-based simulation of its kind that supports MARL.’ This sentence remains unclear as the authors do not propose a simulation engine.\n- Text on experiment figures is much too small.\n\n[1] Andrew Levy, Robert Platt, and Kate Saenko. Learning Multi-Level Hierarchies with Hindsight. In International Conference on Learning Representations, 2019.\n\n[2] Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient Hierarchical Reinforcement Learning. In Advances in Neural Information Processing Systems, pp. 3303–3313, 2018.\n\n[3] Ray Interference: a Source of Plateaus in Deep Reinforcement Learning Tom Schaul, Diana Borsa, Joseph Modayil and Razvan Pascanu\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bkl6DFDQFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper looks at the MARL problem in high-dimensional continuous control settings. To improve learning in this multi-agent setting, they propose to pre-train a lower-level policy that takes as input foot-step goals and is executed for a fixed number of timestep, thereby simplifying both the learning and exploration. \n\nI'm a bit unsure of how to evaluate this paper. On the one hand, I believe it has several contributions:\n- Proposing a new MARL - continuous control environment\n- Proposing a new lower-level policy for high-demensional continuous control environments, including how to learn it\n- Using it to perform MARL in this environment\n\nOn the other hand, it is hard to say what the _main_ contribution is, which in turn makes it difficult to evaluate whether the experimental evaluation is sufficient:\nClearly, a main part of the paper is the work done to construct the hierarchical setup, including goal space, observation space and reward functions. However, this work, as far as I can tell, is separate from the MARL problem. Furthermore, there are several similar ideas already published, so comparison against those (for example by J. Peng, N. Heess or J. Mere) either as argument or even better as experiment, would be helpful to evaluate the quality of the proposed hierarchy.\n\nOn the other hand, there is the application of the hierarchical setup to the MARL problem. However, as far as I can tell, there is no difference between applying such a hierarchy to the MARL case and to the single agent problem. Especially if the lower-level component of the hierarchy is pre-trained in a non-MARL setup, it can just be seen as part of the environment from the point of view of the MARL training, offerring limited new insight into MARL. \nI believe in the second paragraph of 4.1 the authors provide some insight into this matter, however, I have to admit I do not understand this paragraph: \n- Why does temporal correlation reduce the non-stationarity of the MARL problem?\n- Why does structured exploration reduce the number of network parameters that need to be learned?\n- Why does partial parameter sharing make it easier for each agent to estimate other agents potential changes in behavior?\n\n\nIn summary, I think this is interesting work, but a clearer explanation of the relationship between HRL and MARL, as well as a clearer main argument, supported by experimental evidence, would greatly improve this paper.\n\nEdit:\nThank you for your response.\n\nUnfortunately, I don't feel like it sufficiently addresses my questions and concerns. \nI do apologize if my original comment wasn't clear regarding the contribution part of the paper. What I was trying to say is not that I didn't see the individual contributions of the paper, but instead that the paper does multiple things simultaneously, without comparing against the relevant baselines for any of the individual contributions. \n\nRegarding my questions: I understand where the temporal correlation is coming from in an HRL setting. However, what was not clear to me is how this reduces the non-stationarity of MARL. \nI also understand that HRL can reduce the number of parameters, but I don't see how structured exploration reduces the number of parameters.\nAnd lastly, I also can see how parameter sharing can simplify the learning, but I still don't see how it would allow agents to estimate the behaviour change of other agents easier. \nI feel like in the paragraph in questions, a lot of causes and effects are mixed up and more careful descriptions of the benefits of the algorithm would help. \n\nI want to re-iterate that I think that the submitted work by the authors is impressive and can provide valuable insights, but I believe it requires more work and more relevant baselines.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an approach combining multi-agent with hierarchical RL in a custom-made simulated humanoid robotics setting.

Although it is an interesting premise and has a compelling motivation (multi-agent, real-world interaction, humanoid robotics), the reviewers had some trouble pinpointing what the significant contributions are. Partly this is due to lack of clarity in the presentation, such as with overlong sections (eg 5.2), unclear descriptions, mistakes in the text, etc. Reviewers also remarked that this paper might be trying to do too much, without performing the necessary experiments/comparisons and analyses needed to interpret the contributions of each component. 

This work is definitely promising and has the potential to make a nice contribution, given some additional care (experiments, analyses) and rewriting/polishing. As it is, it’s probably a bit premature for publication at ICLR.

Decision: Reject