Learning Control Admissibility Models with Graph Neural Networks for Multi-Agent Navigation
Authors: Chenning Yu
Hongzhan Yu
Sicun Gao
Venue: CoRL 2022 Poster (2022)
Keywords: Graph Neural Networks
Multi-Agent Navigation

Abstract: Deep reinforcement learning in continuous domains focuses on learning control policies that map states to distributions over actions that ideally concentrate on the optimal choices in each step. In multi-agent navigation problems, the optimal actions depend heavily on the agents' density. Their interaction patterns grow exponentially with respect to such density, making it hard for learning-based methods to generalize. We propose to switch the learning objectives from predicting the optimal actions to predicting sets of admissible actions, which we call control admissibility models (CAMs), such that they can be easily composed and used for online inference for an arbitrary number of agents. We design CAMs using graph neural networks and develop training methods that optimize the CAMs in the standard model-free setting, with the additional benefit of eliminating the need for reward engineering typically required to balance collision avoidance and goal-reaching requirements. We evaluate the proposed approach in multi-agent navigation environments. We show that the CAM models can be trained in environments with only a few agents and be easily composed for deployment in dense environments with hundreds of agents, achieving better performance than state-of-the-art methods. 

Learning Control Admissibility Models with Graph Neural Networks for Multi-Agent Navigation
Chenning Yu UCSD chy010@ @uu.d.edu
Hongzhan Yu UCSD hoy021 @ucsd.edu
Sicun Gao UCSD sicung @ ucsd.edu
Abstract: Deep reinforcement learning in continuous domains focuses on learning control policies that map states to distributions over actions that idcally concentrate on the optimal choices in each step. In multi-agent navigation problems, the optimal actions depend heavily on the agents' density. Their interaction patterns grow exponentially with respect to such density, making it hard for learning- bearning-based methods generalize. We propose to switch the learning objectives from predicting the optimal actions to predicting sets of admissible actions, which we call control admissibility models (CAMs), such that they can be easily composed and used for online inference for an arbitrary number agents. We design CAMs using graph neural networks and develop training methods that optimize the CAMs the standard model-free setting, with the additional benefit of eliminating the need for reward engincering typically required to balance collision avoidance and goal-reaching requirements. We evaluate the proposed approach in multi- agent navigation environments. We show that the CAM models can be trained in environments with only few agents and be easily composed for deployment in dense environments with hundreds of agents, achieving better performance than state-of-the-art methods.
Introduction
Multi aalt-- navigation a longstanding problem with wide range of practical applications such as in manufacturing [1. 2], transportation [3], and surveillance [4]. The goal is to control many agents to all achieve their goals while avoiding collisions and deadlocks. Such problems are multi-objective in nature, and the control methods depend heavily on the density of the agents, with the computational complexity growing exponentially in such density [5, 6, 7, 8].
Recently, reinforcement learning approaches have shown promising results on multi-agent navigation problems [9. 10, 11, 12, 13], but there are still two well-known sources of the difficulty. First, balancing safety and goal-reaching requirements often lead ad hoc reward engineering that is highly dependent on the enviromments and the density of the agents [14, 15]. Second, it is shown that such neural control policies leaned on RI. approaches density in the environment changes, which leads to distribution drifts. The core difficulty for generalizability is that the RL approaches designed to capture the optimal actions specific the training environments and rewards. Nevertheless, such optimal actions may change rapidly when the density and interaction patterns change during deployment. The unnecessary optimization of control policies in the training environments makes it fundamentally hard to transfer the learned results to new environments and achieve compositionality in the deployment to varying numbers of agents.
hard generalize when the agent
We propose
learning- based approach to multi-agent navigation that shifts the focus from learning the optimal control policy to a set-theoretic representation of admissible control policies to achieve compositional inference. By decoupling the multi-agent navigation tasks, we avoid the ad-hoc reward engineering using a simple goal-reaching preference function and a learnable set-theoretic component for collision avoidance. We use graph neural networks (GNNs) to represent the set of admissible control actions at each state as Control Admissibility Models (CAM), which are trained with small number of agents during training with sparse rewards. In online inference, we compose the CAMs and apply goal-reaching preference functions to infer the specific action take for each agent. We show that CAMs can be learned purely from the data collected online in the standard RI.
6th Conference on Robot Learning (CoRL. 2022), Auckland, New Zcaland.

---

6(x, a)
0.9
0.6
0.9
0,87
0.00
000
-0.0
-0.4
L.
400
1.0
05
di -406
.05
-10-10
-45
(a) URS
(b) Car
(c) Drone
(d) Landscape of CAM
Figure 1: (a-c) Illustrations of the multi-agent environments. We show videos in the supplemental materials. (d) An example landscape of the proposed CAM ó in the Drone environment. The c-axis and B-axis, are two dimensions in the action space, corresponding the rotation rates in pitch and roll. We project the landscape into a 2D plane at the bottom. The blue line on the plane denotes the zero level set, which divides the action space into the admissible and inadmissible sets.
setting, with no expert or human guidance. We propose relabelling through backpropagation (Section 2.3) for the CAMs to learn rich information from the transitions even given sparse rewards. As shown in Figure ((d), the learned models of the admissible actions are typically complex and suitable for GNN representations. Moreover, in online inference, we decompose the state graph of the agent into much smaller subgraphs and aggregate the CAMs from them. Such compositionality allows to train CAMs with only a small number of agents, directly deploy them in much denser environments, and achieve generalizable learning and inference.
We evaluate the proposed methods in various challenging multi-agent environments, including robot arms, cars, and quadcopters. Experiments show that the CAM generalizes very well. While the training environments only have at most agents, the learned CAM models can be directly deployed to hundreds of agents in comparable environments and achieve a very high success rate and low collision rate. We analyze the benefits of the proposed methods compared state-of-the-ar approaches. Furthermore, we show a zero-shot transfer to a multi-agent chasing game, demonstrating that the trained CAM generalizes to other tasks that not limited navigation.
Related Work. Multi-Agent Navigation. Classical Multi-Agent Path Finding one of the most popular definitions of the multi-agent navigation task, where the state space lies in a pre-constructed graph shared among agents [16). Under this setting, heuristic-based methods have been used to generate high-quality solutions, such as Priority-Based Search and Conflict-Based Search [17, 18). Learning- -based methods have also been proposed to accelerate the planning time, using CNN and GNN [19, 20, 21]. In this work, we focus more on the continuous state space with reactive planning. Traditional methods, including Optimal Reciprocal Collision Avoidance [22], Buffered Voronoi Cells [23], and Artificial Potential Functions [24, 25, 26, 27], are designed manually to address specific structural properties. On the other hand, learning- based multi-agent methods using reinforcement learning and imitation learning [9, 10, 11, 12, 13, 28, 291, may or may not generalize when there is covariant shift for test tasks. Recently, MACBF [30] incorporates multi-agent safe certificates [31, 32, 33] into learning- based framework, which shows the generalization capability while preserving the safety properties. Nevertheless, in [30] the learning of the neural controller requires reference controller, which may constrain the potential of the neural controller, when the actions dissimilar to the reference actions could yield better rewards and safer results. Though learned in an online fashion, our work only assumes sparse reward and avoids the reward hacking problem.
Graph Neural Networks. Graph Neural Networks (GNN), including Graph Convolutional Net- works [34], Message Passing Neural Networks [35], Interaction Networks [36], are deep neural net- works that operate on graph-structurrd data [37]. Graph neural networks are permutation-equivarre to the orders of nodes on graph [38, 39], which is important for homogeneous multi-agent problems since the order among agents should not matter. In robotics, the effectiveness of GNN has been shown on tasks such as path planning [19, 20]. motion planning [40, 41], coverage and exploration [42, 43], and SLAM [44]. In this work, we use GNN as architecture implement CAM due to the multi- agent navigation problem can be modeled as a graph- based problem in nature. However, we believe that the proposed CAM could generalize to other inputs that not graphs.

---

Graph Decomposition
Ø Control Admissibiiity Model
(Giaa)
Pre-Defined
Goal State
min ((
aG2,a)
Ga
32
Egocentric Graph
333
aGs,a
Admissible Actions GG,
130
122'
Most Preferred
Admissible Action
(G
Go
A
Preferences
Figure An overview of the proposed CAM approach. each time step, the state of each agent an egocentric graph. The graph is further decomposed at inference time into set of subgraphs to follow the training data distribution. The CAM takes these subgraphs and outputs the admissibility scores along with a set of sampled candidate actions. We filter out the inadmissible actions by checking whether the minimum of the scores is below O. Given predefined preference over actions, the model outputs the most preferred action in the admissible set for each agent.
2 Control Admissibility Models
We present Control Admissibility Models (CAMs), general representation of the feasible control CAM is task-compositional, robust to reward design, and can be trained with online pipeline similar to model-free RL. With graph decomposition, CAM can plan high-quality solution for a large number of robots, which could be significantly out of the distribution of the training tasks.
2.1 Control Admissibility Models (CAMs)
We define CAMs as a general representation of the feasible actions. In general, CAMs are function approximators that behave as the characteristic functions of such sets, By evaluating the CAM values we can efficiently determine whether an action is feasible. Formally.
Definition 1 (Control Admissibility Models and Admissible Set). Given CAM 6. state action spuce A, the admissible set defined as ((,,)) {a E A. (( 2 0}.
Compositionality of CAMs. An important feature of CAMs is that they can be composed together. Given two CAMs ei and Φ2, we want to find the admissible actions that satisfy both these two CAMs:
{a Φi ó1 a) 2 0} {a Φ2(2, 0}
Proposition 1 (Composition). &h and Φ are CAMs, then A(da, x) defines A(d), n A(d2, given d3- = min(@1, (22))
2.2 Graph Neural Networks for CAMs
Though CAM can be applied to various single-agent problems (see Section 3.1 as an example), we focus multi-agent problems in this paper. At each timestep. the state for each agent is an egocentric graph G (V, E), where V are the vertices and E are the edges. Vertices V is composed of agent vertices V, and static obstacle vertices V.. Edges E connecting all the neighbor vertices the current agent, i.e. E [(w] , v la; E E(a,) (10) v vertices. The one-hot vector is used the features of V. denoting the types of vertices. For the features of edges E, we use relative positions and the states of the two connected vertices, along with a one-hot vector to indicate the edge types. Zero paddings are adopted if necessary [45]. We provide more details on the representation of graphs in the Appendix.
" lo E E(a)), where denotes neighbor
The CAM o has two components: The first component is a GNN layer, which transforms the graph G to hidden state vector h for each agent. The second component is fully-connecter layer f. which

---

takes the hidden state h and an action and predicts the admissibility score aG f(h, a). Such design enables fast computation of the admissibility scores for all the possible state-action pairs. We can merge the compulation of the hidden state for all state-action pairs which share the same input graph into one forward passing in the GNN layer.
We describe the GNN layer with the following details. The vertices and the edges are first embedded
into latent space m( (m), nz In (e), given fully-connected layers Im 9n- Taking m, n, the GNN aggregates the local information for each vertex from the neighbors through K GNN layers.
For the k-tth layer, it performs the message passing with 2 fully-connected layers (k) and ((k)
m (k-1) m; + max{Sm (ml (1e) (k-1) (k)
et (1z, 2x) E Vo, EV ni ni + ((*) (m (k+1), (()) Vey (.) E
We assign the hidden state with the final node embedding m; (K),, where corresponds to the current agent. We use max as the aggregation operator to gather the local geometric information due to its empirical robustness to achieve the order invariance [30, 39]. We use the residual connection to update the vertex embedding and edge embedding due to its simplicity and robust performance for deep layers [46, 47J. Moreover, each agent's 's hidden state is only updated based on its egocentric graph, which is entirely invariant to the other agents' hidden states. This architecture decentralizes the saccision-making of the whole swarm.
2.3 Training CAMs in Reinforcement Learning
We train CAMs in the same online setting as model-free reinforcement learning procedures. At each time step, the agent perceives the observation from the environment and takes action. The state-action pairs for every transition are labeled safe unsafe and then appended the replay buffer. The CAM is updated every time a certain number transitions are collected. The main differences between of using an actor-network, the agent chooses the action among the sampled actions based on the admissibility score generated by CAM. (ii) the label is binary for each state-action pair instead of calculating the cumulative future rewards. (iii) the training objective avoids the overestimation issue and is more stable. We design these three improvements around the underlying structure of the admissibility problem. We provide further explanations as follows.
training approach and the standard RL methods are three-folde fold: (i) instead
non-bootstrapped, which
Algorithm CAM Algorithm for Training
1: Input: Preference function w, exploration noise N
2: Initialize CAM &
3: Initialize replay buffer R
4: for episode = 1, M do
5: for 1, do
6:
Sample candidate actions {a} S A
7:
[G), - all agents' current states and goal states Select action for each agent according to oGG, a), w(G, a)), and N
8:
9:
Execute the selected actions and observe the next states Se+11
10:
Label the transition (St, ax, 8t+1) with ye by checking whether 8,41 is in danger set Store transition (st, ag Yt: St+1) in R
11:
Relabel R through backpropagation Update o using Equation? 2
13:
Sparse Reward Setting, We model the problem the standard Markov Decision Process (MDP) M = A, 7:7) [48]. S is the state space represented as graphs G. The action space A is continuous. The reward is sparse: it only returns non-zero values when the agent enters the region to avoid and the region to reach. These certain regions can also dynamic, i.e., danger regions centering around other agents.
Preference Function. Given current state G and goal state T, assume a preference function over actions, i.c., wGG, T..), is given. The preference function solely focuses on the goal-reaching. For instance, it can simply be the L2 distance from the next state to the goal, since CAM will achieve the obstacle avoidance. In the experiments, we try to choose the preference functions as simple as possible. We refer readers to the Appendix for more details.

---

Action Selection via Sampling, At every timestep, the agent chooses an action among a batch of actions sampled from the action space. Given a state x, the admissibility value every action a is computed with 6zz, a). If there exist one or multiple actions that satisfy {a Φ(z, a) 2 0}, given the goal state T and preference (ar, T.+) over actions, we choose the action that has the highest preference score in the admissible set. Namely, arg max, {w(r, T, a))((1, a) 2 0}. If there does not exist any admissible action, in order fulfill the admissibility property maximally, the agent chooses the action that has the highest admissibility score, arg max (0(z, a}} + N, where N is an exploration noise. We model the exploration noise under the uniform distribution in our experiment.
Training Loss of CAM. Given admissible transitions Ro C 8 x A and inadmissible transitions Ra - S x A, we train the CAM with the following objective:
LB =
Σ
ReLU( - o02 a)) +
Σ (a,
ReLU(12 + +((x)))
Ro
[Ral
(x,a)Efo
+
ReLU(na ò(I, a) \ō(o, (2)
|Ral (z
where 71, 72,72 73 2 O are the coefficients to enlarge the margin of the learned boundary. Instead of simply classifying the transitions with the first and the second loss terms, we encourage the forward invariance property by adding the third term, similar to Control Lyapunov Functions and Control o(r, a). where (a', a²) is
Barrier Functions [31]. In practice, we approximate ő(2, a) by ø(x', the next state-action pair on the trajectory. Intuitively, once all pairs of consecutive transitions satisfy this condition, the transitions would form a forward invariant set, and any trajectory starting from inside the invariant set will never cross the admissible boundary. The XØ(Iz, a) term encourages the trajectory to be asymptotically admissible, even if the agent enters the inadmissible region sometimes.
Relabeling Through Backpropagation. We first label the collected transitions as admissible and inadmissible based on whether the next state is in the danger region. Once the whole trajectory temmnnatss we relabel these transitions through backpropagation. Intuitively, suppose the agent encounters the collision along the trajectory. In that case, it means that starting from some timestep. it enters the inadmissible region and fails to get back to the admissible set (similar to Intermediate Value Theorem [49]). Without relabeling, the labels of the transitions from this step to the collision event will be admissible, which is incorrect. To solve this issue, we relabel these critical transitions.
A state-action pair (xr a) on the trajectory is relabelled as inadmissible, if (i) the next state-action pair (a', a') is inadmissible, and (ii) admissible action is found at state x' namely, max.' 6(x', a') O. Intuitively, the relabelling propagates the inadmissibility from where the agent enters the danger region back to state, where the agent can make decision leading admissible regions. The first condition ensures the optimistic behavior of the CAM. We only relabel those transitions which stay in the inadmissible region and encounter danger in the future. Such optimism is important when no feasible trajectory exists at the early training stage. It encourages the agent to explore instead of predicting almost every transition to be inadmissible. The second condition ensures that there exist no admissible actions approximately by inspecting the admissibility score for all the actions. Furthermore, for the transitions that have potential admissible actions, they will not be relabelled as inadmissible easily at the early training stage since max,' o(x', ,a') < 0 is hard meet for moderately initialized CAM.
2.4 Online Inference with Graph Decomposition
At inference time, the agent behaves in the same manner as the training stage, except that there is no exploration noise during inference. When the CAM is applied directly to large swarm of agents, however, we find that the performance degrades significantly due to the distribution shift. Such degradation occurs considerably when the inference task has much higher density of agents. To solve this issue, we leverage the compositional property of CAMS and propose graph decomposition.
Decomposition on Large Graphs. We can break down the whole graph G into subgraphs (G&). where these subgraphs satisfy the training distribution, e.g., the number of agent vertices does not exceed a certain value. We construct the subgraphs repeatedly until all the edges in G appear at least once in the subgraphs. For each subgraph GR. and action (GGr, a). Though each subgraph defines a less strict subtask for the agent to perform, the admissible set for the original task should lie in the intersection of all these admissible sets of the subtasks. As
calculate the admissibility score as
5

---

result, these values on the subgraphs can be composed to represent the admissibility score on the entire graph o a), which could be totally out of distribution. To compose the admissibility values, we follow Proposition and use (GG, a) = ming.ga(o(Gr.ee
Algorithm 2 CAM Algorithm for Inference
1: Input: CAM Φ, preference function w, distribution P of training data
2: for [= T,
3:
for each agent's current state G, goal state T do
4:
Sample candidate actions {a} S A 5: Initialize (GG a) as 2o for all sampled candidates
6:
repeat
7:
Sample G, S G. where G₁ P
8:
((G, a) = min(o(G,a) a), aG) for all candidates a until all edges G are sampled at least Select action for the agent according to @GG, a) and W
9:
10:
11:
Execute the selected actions and observe the next states $1+1
3 Experiments
3.1 Proof of Concept: CAM for Single Agent Environment
max (x a). æ
min ((x a). for all æ
p(o(x, 2 0). for x
0.8
0.0
1.0
2
0.0
0.6
2
2
0.8
0.4
0.2
0
0.2
0
0.0
0.4
0.0
0.2
-2
-2
-2
0.2
-0.4
-3 -2 -1
2
0.4
-3
-2
2
-2
-1
Figure 3: proof of concept for the proposed CAM in a single-agent environment. The agent aims to reach the goal while avoiding obstacles. The black point denotes the starting state, and the red star denotes the goal state. The shaded blue regions denote the obstacles. Given the learned CAM Φ, action space A (-0.1, 0.1) and an arbitrary state E & { (-3. 3)2 we illustrate m 6(x, a), min.e ((x a). and p(ó(z x a) 0) respectively. We show that the learned CAM never leaves the admissible set, i.c. {x E X max Φ(x, a) 2 0}. along the trajectory.
We begin by analyzing CAM agent single-agent environment. We perform such analysis using a 2D minimal example environment, where we place three danger regions to form narrow passages, as shown Figure 3. We illustrate maX ((x, a), min ø (2x, a), and p(ő(z, a) > 0) respectively. Intuitively, max Φ(z, a) indicates whether there exists any admissible actions that drive the agent the admissible set. minae A (bI, a) indicates the possibility to enter the inadmissible sets if actions are picked randomly. As what we show the figure, the trajectory (blue dashed line) does not intersect with the admissibility boundary, i.e. {or X' maX.e Φ(x, consistent with the norward-invariant objective for learning the CAM: if the agent executes an admissible action A: 6(I, a) 2 0 at every time step, then it will never leave the admissible set.
0). This example stays
3.2 Multi-agent Navigation Experiments
Experiment Setup. We evaluate methods under 4 types of environments: UR5, Car, Dynamic Dubins, and Drone. We briefly describe these environments as follows: (i) The UR5 environment is implemented with PyBullet [50] and contains 4 decentralized 5D robot arms with cubic obstacles. We randomly select solely 2 robot arms from these four arms during training. We fix the configurations of the static obstacles and the goal states of the arms. In order to make the test task solvable, the initial

---

UR5
Dynamic Dubins
Drone
1.00
0.25
CAM
60R6
2.50
2.50
cAM
zero
Saficy 1222
9.00
CAY MAcre
8,20 s09
0.92
Linea
37002
S7016
855
1101
22021
azote
0.50
14009
222000
Numbbrr of Agents
Ratio of Agents over Obstacles
Radio of Agents over Distacies
Rabio of Agents over Dbstacles
URS
Car
Dyramic Dubins
CAM DoPG
Rewsod
CAM DI2ING
Roward
e
GAM
Roward
Reward
MACAF
CAN DORG
-10
Exresa
1 enee
150 ved
312.12
2
J
zote
2042200
Number Agents
200
Rotte
Agents over Obntacies
Ratto
Agonts over Obstades
Agents
Obstac
Figure 4: We show the performances of CAM and prior methods in the UR5, Car, Dynamic Dubins, and Drone environment, where each method is trained with 3 agents during training but tested up to 512 agents. CAM substantially outperforms prior approaches in all the environments.
states are randomized during training and fixed during testing. (ii) For the Car environment, each agent follows the 2D Dubins' car model I, and controls a ID action. The static obstacles are circles with a fixed radius. (iii) The Dynamic Dubins agent controls 2D action and follows a modified Dubins' Car dynamic which enables the control of acceleration. (iv) For the Drone environment, each agent follows 3D quadrotor model adopted from [52] and controls 4D action. Each obstacle is a cylinder with infinite height. For both Car and Drone environments, we only train on environments with 3 agents, randomized initial states and goal states, and a set of random obstacles. The map size is fixed as 3x3 during training while varying during testing. We provide full details of system dynamics and environments in the Appendix.
To evaluate our CAM approach thoroughly, we design diverse scenarios for all four environments. For the UR5 environment, we make the number of agents vary from to For scenarios with 2 or arms, the arms are selected randomly from the four decentralized arms. For the other environments, we fix the map size as 32x32 and vary the ratio of agents over obstacles from 1:1023 to 512:512. For each setting, we average the performance over 100 randomly generated test cases.
The baseline approaches we compare with include DDPG [53] and MACBF [30]. DDPG standard RL approach to learning safe behavior through maximizing rewards. MACBF is state-of-the-art approach for multi-agent safe control problems, which learns a policy jointly with neural safe certificates. We reimplement all the baselines with GNNs for fair comparison. Note that our GNN implementation for DDPG is a multi-agent algorithm, since each agent is aware of the other agents by taking the egocentric graph as the imput. We refer reader to Appendix for more details.
Baselines.
Evaluation Metrics. We evaluate our methods based on two metrics: the safe rate and the average reward. The safe rate counts the number of collisions for each agent at every time step and averages over all the agents through the whole trial [30]. At each time step, the reward is -1 if the agent collides with another agent or the static obstacle. A +10 reward will be given at the end of the trajectory, if the agent reaches its goal any time step along the trajectory. A small negative constant of -0.1 is added to the reward to encourage shorter paths for goal-reaching. The reward is accumulated throughout the trial and averaged over all the agents.
Overall Performance. In Figure 4, we demonstrate the overall performances for the URS, Car, Dynamic Dubins, and Drone environments, Our method shows the generalization across different environments and densities of agents and obstacles. Though trained with only 3 agents, the safety rate of CAM remains nearly 100% for all the environments, up to 512 agents. Even if the density of agents and obstacles deviates from the training environment, our CAM approach can avoid the distribution shift, using decomposition on large graphs and the compositionality of CAMs.
On the other side, though our implementation for MACBF and DDPG can take an arbitrary number of agents using GNNs, the performance degrades significantly when the distribution of input graphs shifts in test environments. We also inspect that the averaged reward for all the methods decreases in most cases when the number of agents increases. It is due more agents failing to reach the goal when the timeout happens since it takes more time for agents to avoid each other. We encourage the reader to view the video demonstrations in the supplementary material for all 4 environments.
7

---

3.3 Zero-Shot Transfer to Chasing Game
Method
CAM
CAM w/o decomposition
Safety Rate (Car)
0.99-10.00
0.90++0.03
Reward (Car)
3.20
-10098
Safety Rate (Drone)
0.989
0.005
0.988+00.008
Reward (Drone)
6.84++1.12
6.54++2.11
Table 1: Left: A snapshot of a trajectory in the chasing game with 16 Drone agents from bird's-eye s-eee view. The agents are the colorful points, and the obstacles are the blue circles of a larger size. Each agent aims to chase another agent without collision. Right: Performances of CAMs with and without graph decomposition on 64 agents, which shows the effectiveness of the graph decomposition.
In this section, we study the generalization of our CAM model for zero-shot transfer to other multi- agent tasks, e.g.. the chasing game. In the chasing game, each agent chases another agent pursue. The target agent is assigned randomly at the beginning of every episode. The agent's 's goal is to maintain the distance to the assigned agent, with no collision with agents and obstacles. We fix the number of agents to 64 in the experiment. This new task is substantially different from the task for training. where the assigned goal is fixed. We provide more details of the setting in the Appendix.
In Table 1, we show that our CAM method can adapt to the new task effectively due its safety and goal-reaching decoupling property. preserving a relatively high safety rate and reward. In addition, we compare our method to the CAM without the graph decomposition. We observe that the graph decomposition works effectively in both environments. The performance of the Car environment benefits more from the graph decomposition. This advancement is because the density of Car agents in 2D is higher than that of the Drone agent in 3D. The higher density makes the test task more unlike the training task, which explains why the graph decomposition is more beneficial for Car.
4 Limitations and Failure Modes
While the framework is generalizable in terms of scalability, it has limitations. Since the proposed GNN architecture is entirely decentralized, it disables the agents from communicating and coordi- nating with each other in the swarm. As result, it could be challenging for agents identify and avoid critical situations requiring interactions among agents, e.g., dead-locks and states with very few feasible actions. As shown in Figure there is slight drop in the safety rate of the Drone environment when the number of agents increases to 512. To understand such a failure mode, we inspect these trajectories and find that the corresponding CAM values are all negative at several consecutive preceding states right before the collision. Such behavior indicates a very low probability of finding feasible actions in these states, which validates the necessity for inter-agent communication or global coordination when the density of agents is very high in local regions. We believe that a preference function considering the global coordination can be helpful for this issue.
5 Conclusion
We proposed new baaming-based control methods for multi-agent navigation problems by shifting the focus from training optimal control policies to training CAMs that implicitly represent a set of feasible actions. We avoid the reward engineering by decoupling the multi-agen navigation tasks using a simple goal-reaching preference function and a learnable CAM for collision avoidance. Our methods use GNNs to represent the CAM which takes egocentric graphs inputs and proposes the relabelling with backtracing to learn from rich information even given sparse reward. In online inference, we propose the graph decomposition to deal with the covariate shift, which achieves a notably high success rate and low collision rate. While unconventional compared the types of models routinely used in multi- useful properties and is particularly effective in multi aggett settings that require compositionality and generalization for scaling at inference time. In future work, we will study how to incorporate multi-agent communication and test its effectiveness in real-world scenarios.
reinforcement learning, we demonstrate that CAM has several
8

---

Acknowledgments
We thank the anonymous reviewers for their helpful comments in revising the paper. This material is based on work supported by the United States Air Force and DARPA under Contract No. FA8750- 18- B3-C-092,, AFOSR YIP PA9550-19-1-0044 NSF Career CCF 2047034, and Amazon Research Award.
References
[1] José Barbosa, Paulo Leitão, Emmanuel Adam, and Damien Trentesaux, Dynamic self- organization in holonic multi-agent manufacturing systems: The adacor evolution. Computers in industry, 66:99-111, 2015.
[2] Ray Zhong, Xun Xu, Eberhard Klotz, and Stephen T Newman. Intelligent manufacturing in the context of industry 4.0: a review. Engineering, 3(5):616 6-00,, 2017.
[3] Ekim Yurtsever, Jacob Lambert, Alexander Carballo, and Kazuya Takeda. A survey of au- tonomous driving: Common practices and emerging technologies. IEEE access, 8:58443-58469, 2020.
[4] Jörg Müller and Klaus Fischer. Application impact of multi-agent systems and technologies:
A survey, In Agent-oriented software engineering, pages 27-53. Springer, 2014.
[5] Reza Olfati-Saber, J Alex Fax, and Richard M Murray. Consensus and cooperation in networked multi-agent systems. Proceedings of the IEEE, 95(1):215-233 2007.
[6] Yongcan Cao, Wenwu Yu, Wei Ren, and Guanrong Chen. An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions Industrial informatics, 9(1):427 438, 2012.
[7] Liviu Panait and Sean Luke. Cooperative multi multi-agent learning: The state of the art. Autonomous agents and multi-agent systems, 133):3877433, 2005.
[8] Manuele Brambilla, Eliseo Ferrante, Mauro Birattari, and Marco Dorigo. Swarm robotics: a review from the swarm engineering perspective. Swarm Intelligence, 7(1):1-41) 2013.
[9] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
[10] Arbaaz Khan, Ekaterina Tolstaya, Alejandro Ribeiro, and Vijay Kumar. Graph policy gradients for large scale robot control. In Conference on robot learning, pages 823-834. PMLR, 2020.
[11] lou-Jen Liu, Raymond Yeh, and Alexander G Schwing. Pic: permutation invariant critic for multi-agent deep reinforcement learning. In Conference Robot Learning, pages 590-602 602. PMLR, 2020.
[12] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-compeiinn Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6379-6390, 2017.
environments. In Isabelle Guyon, Ulrike von
[13] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295 4304. PMLR, 2018.
[14] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint 11Xx:16666.06655,, 2016.

---

[15] Dylan Hadfield- Menell, Smitha Milli, Pieter Abbeel, Stuart J. Russell, and Anca D. Dragan. Inverse reward design. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Gamett, editors, Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6765-6774, 2017.
[16] Roni Sterm, Nathan Sturtevant, Ariel Felner, Sven Koenig, Hang Ma, Thayne Walker, Jiaoyang Li, Dor Atzmon, Liron Cohen, TK Satish Kumar, et al. Multi-agent pathfinding: Definitions, variants, and benchmarks. Twelfth Annual Symposium on Combinatorial Search, 2019.
[17] Hang Ma, Daniel Harabor, Peter J Stuckey, Jiaoyang Li, and Sven Koenig, Searching with consistent prioritization for multi-agent path finding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7643-7650, 2019.
[18] Guni Sharon, Roni Stern, Ariel Felner, and Nathan R Sturtevant. Conflict-based search for optimal multi-agent pathfinding. Artificial Intelligence, 219:40-66, 2015.
[19] Qingbiao Li, Fernando Gama, Alejandro Ribeiro, and Amanda Prorok. Graph neural networks for decentralized multi-robot path planning. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11785-11792, 2020.
[20] Qingbiao Li, Weizhe Lin, Zhe Liu, and Amanda Prorok. Message-awee awaege graph attention networks for lcage-scale multi-robot path planning. IEEE Robotics and Automation Letters, 6(3):5533-5540, 2021.
[21] Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, TK Satish Kumar, Sven Koenig, and Howie Choset. Primal: Pathfinding via reinforcement and imitation multi-agent learning. IEEE Robotics and Automation Letters, 4(3):2378 20::22338338822222 2019.
[22] Jur van den Berg, Stephen J Guy, Ming Lin, and Dinesh Manocha. Reciprocal n-body collision avoidance. In Robotics research, pages 3-19. Springer, 2011.
[23] Saptarshi Bandyopadhyay, Soon-Jo Chung, and Fred Y Hadaegh. Probabilistic and dis- tributed control of a large-scale swarm of autonomous agents. IEEE Transactions Robotics, 33(5): 103-10212 1123, 2017.
[24] Oussama Khatib, Real-time obstacle avoidance for manipulators and mobile robots. In Au- tonomous robot vehicles, pages 396-404. Springer, 1986.
[25] Jerome Barraquand, Bruno Langlois, and J-C Latombe. Numerical potential field techniques for robot path planning. IEEE transactions systems, man, and cybernetics, 2222):224-241, 1992.
[26] Shuzhi Sam Ge and Yun J Cui. Dynamic motion planning for mobile robots using potential field method. Autonomous robots, 13(3):207-222, 2002.
[27] Eduardo G Germández-Martinee Eduardo Franda-Bricaire and F Alkhateeb. Convergence and collision avoidance in formation control: A survey of the artificial potential functions finnttioon approach. INTECH Open Access Publisher Rijeka, Croatia, 2011.
[28] Benjamin Riviere, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung. Glas: safe autonomy synthesis for multi-robot motion planning with end-to-end learning. lEEE Robotics and Automation Letters, 5(3):4249-4256 2020.
Global-to-local
[29] Guanya Shi, Wolfgang Hönig, Yisong Yue, and Soon-Jo Chung. Neural-swarm: Decentralized close-proximity multirotor control using learned interactions. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3241-3247. IEEF, 2020.
[30] Zengyi Qin, Kaiqing Zhang, Yuxiao Chen, Jingkai Chen, and Chuchu Fan. Learning Safe Multi-Agent Control with Decentralized Neural Barrier Certificates. In Conference on Learning Representations. Conference on Learning Representations, jan 2021.
[31] Aaron D Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 2019 18th European Control Conference (ECC), pages 3420-3431, Naples, Italy, June 2019.
10

---

[32] Dongkun Han, Lixing Huang, and Dimitra Panagou. Approximating the region of multi-task coordination via the optimal lyapunov like barrier function. In 2018 Annual American Control Conference, ACC 2018, Milwaukee, WI, USA, June 27-29, 2018, pages 5070-5075. IEEE, 2018.
[33] Shuo Wan, Jiaxun Lu, and Pingyi Fan. Semi-centralized control for multi robot formation. In 2017 2nd International Conference Robotics and Automation Engineering (ICRAE), pages 31-36. IEEE, 2017.
[34] Thomas N. Kipf and Max Welling. -upi-suuereieed classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
[35] Justin Gilmer, Samuel S Schoenholz, Patrick Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1263-1272. PMLR, 2017.
[36] Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neu- ral Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 4502-4510, 2016.
[37] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1);61-80, 2008.
[38] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
[39] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 77-85. IEEE Computer Society, 2017.
[40] Arbaaz Khan, Alejandro Ribeiro, Vijay Kumar, and Anthony G. Francis. Graph neural networks for motion planning. CoRR, abs/2006.062244 2020.
[41] Chenning Yu and Sieun Gao. Reducing collision checking for sampling-based motion planning using graph neural networks. In Proceedings of the 35rd International Conference on Neural Information Processing Systems, 2021.
[42] Lifeng Zhou, Vishnu D Sharma, Qingbiao Li, Amanda Prorok, Alejandro Ribeiro, and Vijay Kumar. Graph neural networks for decentralized multi-robot submodular action selection. arXiv preprint
2021.
[43] Ekaterina Tolstaya, James Paulos, Vijay Kumar, and Alejandro Ribeiro. Multi-robot coverage and exploration using spatial graph neural networks. In 2021 TEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 8944-8950. IEEE, 2020.
[44] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. SuperGlue: Learning feature matching with graph neural networks. In CVPR, 2020.
[45] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Peter L. Bartlett, Fernando C. N. Pereira, Christopher J. C. Burges, Léon Bottou, and Kilian Q. Weinberger, editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106-1114, 14, 2012.
[46] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512 112.03385 2015.
[47] Binxuan Huang and Kathleen M Carley. Residual or gate? towards deeper graph neural networks for inductive graph representation learning. arXiv preprint arXiv:1904.08035, 2019.
11

---

[48] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[49] Walter Rudin et al. Principles of mathematical analysis, volume 3. McGraw-hill New York, 1976.
[50] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. 2016.
[51] Lester Dubins. On curves of minimal length with constraint on average curvature, and with prescribed initial and terminal positions and tangents. American Journal of mathematics, 79(3):497-516, 1957.
[52] Charles Dawson, Zengyi Qin, Sicun Gao, and Chuchu Fan. Safe nonlinear control using robust neural lyapunov-barrier functions. In Conference on Robot Learning, pages 1724-1735 1735. PMLR, 2022.
[53] Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: (59.00290900971 2015.
12


Reviews:
[{"review_id": "rdzEq95DMep", "reviewer": "Reviewer_XUE1", "summary": "The authors proposed a strategy to compute admissible sets for multi-agent systems, i.e., a set of actions that agents can take without compromising safety. These sets are computed using GNN which naturally takes into account interaction between different agents. For training, each state-action pair is labeled based on the state of the system and then a relabeling state is applied to make sure that wrong actions that led to unsafe regions are not labeled as safe. Finally, the authors showed that the proposed approach trained in scenarios with only a few agents generalizes to settings where several agents are interacting. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) Using a GNN for multi-agent navigation makes sense and enables scalability\n2) The re-labeling strategy allows the authors to catch failure early on\n\nWeaknesses\n1) The authors are basically trying to compute invariant sets. It would be nice to introduce a formal definition, clearly state the objective, and provide an analysis (if possible). \n2) From the first example, it is unclear if the method is actually able to approximate forward invariant sets, even in the very simple example shown in the experiment section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think that the paper is well-written and that the method is well-designed. The objective of this work is pretty ambitious as the authors are approximating invariant sets. Clearly, no guarantees can be provided for this problem, so the authors performed a set of experiments to empirically show that the method is working. I think that further experiments are needed. In particular, I would like to see the method applied to high inertia systems where the control action is saturated, see the Issues section for further details.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The method is well presented, but it has no theoretical guarantees (this is totally fine as computing invariant sets is very hard already for a single agent!). So I focused on the examples and I would like to see a few changes to confirm that the method is working:\n1) What is the model used for the single agent environment? I would like to see an agent that has a lot of inertia, e.g., a car where you control the acceleration rate and the maximum acceleration rate is saturated. This would allow us to understand if the re-labeling strategy is working and if you are able to catch failure early on.\n2) Figure 3 should have some information about the velocity. If you are on the boundary of the set with an outside pointing velocity vector the set is not invariant. It would be nice to see if the set that you are computing is actually an invariant (or a good approximation)\n3) Why the set in Figure 3 is not symmetric? As we know that subsets to invariant sets are not invariant, it is unclear if the set shown in Figure 3 is forward-invariant.\n4) In the Dubins car examples, are you controlling the velocity? It would be more convincing to control the acceleration and to have a saturation on the acceleration. For the dynamics, you can simply assume that the velocity is the integral of the acceleration. If you are currently controlling the velocity, could you repeat the experiments when controlling the acceleration? I expect to see more collisions."}}, {"review_id": "SIJwF9_gkWg", "reviewer": "Reviewer_JipF", "summary": "In this paper, the authors study the multi-agent navigation problem and propose a control admissibility model (CAM) to output and compose the action instead of directly approximating the optimal policy. To this end, the navigation problem is represented by a graph, and graph neural network (GNN) is employed to compute the admissibility score. The CAM is validated on three navigation benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is well-written and easy to follow. The authors propose a simple but potentially effective framework, which breaks down the difficult overall optimal policy learning to multiple admissibility policies. However, the main contributions are not clear. First, the authors mainly introduce two multi-agent navigation difficulties, i.e. ad hoc reward engineering and poor generalization capacity. But why the GNN and decomposition-based CAM can deal with the above challenges is unclear. Second, the proposed method seems independent of the multi-agent system. In the multi-agent navigation problem, what are the key problems can CAM tackle? Additionally, the experiments are less convincing since only one multi-agent navigation baseline is compared. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the proposed method is clear and seems potentially effective, the key multi-agent navigation problem to be solved and the empirical results are not clear. Therefore, I suggest rejection of the current version.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "1. In the graph decomposition stage, the sub-graph will be different. How can a single GNN-based CAM deal with graphs with different vertices and edges?\n\n2. In algorithm 1, it shows that CAM does not use reward to guide learning, which can also be seen from Eq. 2. How can CAM learn to navigate if no reward, i.e. goal information provided?\n\n3. What is the influence of three key hyper-parameters in Eq. 2?\n\n4. In Eq. 2, the authors claim the third term enforces the continuity of the CAM values on two consecutive transitions. How to understand it?\n\n5. In the experiments, why the single agent algorithm DDPG outperforms the multi-agent algorithm baseline MACBF?"}}, {"review_id": "18u6ocrftVG", "reviewer": "Reviewer_759H", "summary": "This paper proposes learning a model of admissible controls that would avoid violating specified constraints in an MDP.  The paper calls these models Control Admissibility Models (CAMs), and demonstrate how these can be easily composed to provide a set of admissible actions at any state.  The CAMs are built using a Graph Neural Network, and trained using a time-based back-propoagation, where action that bring into a state that has no admissible actions, even if the state itself is admissible, are also included in the non-admissible state set.\n\nExperiments are performed on a UR5 multi-arm task, a multi-agent driving task, a point-mass chasing task and a drone flight task.  Experiments show significant ability to maintain safety conditions as the number of agents increases, especially in the Car setting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents an interesting approach to dealing with dense multi-agent systems, and seems to have promissing results although I'm not an expert in this area and so don't have much experience in what is achievable.\n\nOne big weakness however is that I couldn't figure out how the paper uses reward from the environment.  It's clearly taken into account otherwise nothing would work, but the paper seems to skip over this important point or I somehow missed it after multiple re-reads, in which case this should definitely be fixed. \n\nI would also appreciate a small mention of the inference time of this setup, to understand to what types of systems it could be deployed (i.e. up to what control frequency could it likely function).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this method is easily applicable to real systems, and provides a believable story on how to deal with a large number of interacting agents, which is an important problem in robotics.  ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "I have mentioned above one blocking issue for me is the discussion of the use of reward, although I recommend an acceptance it is absolutely contingent on addressing this point.  I would also like a brief mention of inference time for the method, as this can drastically impact the types of systems on which it can run."}}]

Author rebuttal:

Decision: Accept (poster)