Towards Scalable Imitation Learning for Multi-Agent Systems with Graph Neural Networks
Authors: Siyu Zhou
Chaitanya Rajasekhar
Mariano J. Phielipp
Heni Ben Amor
Venue:  (2020)
Keywords: Graph Neural Networks
Scalability
Swarms
Imitation

Abstract: We propose an implementation of GNN that predicts and imitates the motion be- haviors from observed swarm trajectory data. The network’s ability to capture interaction dynamics in swarms is demonstrated through transfer learning. We finally discuss the inherent availability and challenges in the scalability of GNN, and proposed a method to improve it with layer-wise tuning and mixing of data enabled by padding.

Under review as a conference paper at ICLR 2020
TOWARDS SCALABLE IMITATION LEARNING FOR MULTI-AGENT SYSTEMS WITH GRAPH NEURAL NET- WORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose implementation of GNN that predicts and imitates the motion be- haviors from observed swarm trajectory data. The network's ' ability to capture interaction dynamics in swarms is demonstrated through transfer learning. We finally discuss the inherent availability and challenges in the scalability of GNN, and proposed method to improve it with layer-wise tuning and mixing of data enabled by padding.
INTRODUCTION
In multi-agent systems (MAS) (Arai et al., 2002), group of agents has to effectively collaborate in order to solve a complex task in a parallel manner. Individual agents have to synchronize their actions and behaviors so as to produce whole that is more than the sum of its parts. This syn- chronization a core characteristics of MAS and of vital importance to a wide range of modern application domains such smart agriculture, military demining, warchouse robotics. However, developing control strategies that allows group of agents to perform a task jointly is still consid- ered to be complex challenge. This statement is particularly true for mobile agents that need to dynamically act in a physical space. To overcome this challenge, a variety of formalisms have been proposed support the modeling and design of multi-agent and multi-robot systems. A promi- nent approach, called Boids or bird-oid, was proposed in |Reynodss (1877). In the Boids model, the system behavior emerges from the interplay of a few simple building- blocks. While simple to implement, it can be challenging to control the overall behavior of the swarm due to the emergent nature of the system dynamics. Other important formalisms for the specification of MAS are based on graph theory (Mesbahi & Egerstedt 2010), game theory, and formal languages. However, the latter approaches still require substantial modeling, coding, and verification efforts in order to yield practical controllers that can scale with the size of the system, i.e., the number of agents. Another obvious choice to reduce or eliminate modeling effort is to use machine learning. Reinforcement learning (Sartoretti et al. 2019, in particular, has previously gained attention for synthesizing col- lective behavior an MAS. However, due to the curse of dimensionality, reinforcement learning approaches often struggles to scale systems with large number of agents.
In this paper, we present an approach for learning multi-agent coordination through imita- tion (Schaal 1999). Rather than specifying the system dynamics, the designer only has provide demonstrations of successful coordination behavior. In turn, this data set is used to learn a neural network representation of the underlying dynamics and rules of interaction. More specifically, we leverage recent insights regarding graph neural networks (GNN) (Battaglia et al. (0088) to learn a structured model of the dynamics. In contrast traditional neural networks which take vector, ma- trix or tensor as input, our graph neural network processes a graph representation of the multi-agent system. An implementation example of GNN can be seen in Kipf al. (2018), where the ability of GNN to better predict the motion of simple physically interacting particles is highlighted. We show that the specific GNN implemented this paper can accurately capture the dynamics of more complex multi-agent motion behaviors given a set of demonstrations. We also discuss and analyze difficulties in scaling learned models up to an MAS with a larger set of agents. Based on this dis- cussion, then propose refinement training procedure to address these issues. The refinement procedure helps tuning a learned model to a system with a different number of agents, i.c., scalability along the size of the MAS.

---

Under review as a conference paper at ICLR 2020
2 MODEL
2.1 OVERVIEW
Demonstration
Imitation
Scaling and Augmentation
Figure 1: A depiction of the training and inference. For training the trajectories of a set of agents is recorded as demonstrations. After we train our GNN model, it can be used to create a copy of the swarm. In addition, it is possible to create larger and smaller swarms by changing the number of agents.
In this section, we describe our methodology for imitation learning of MAS behavior. Fig. D depicts an overview of both the training and inference process for M The input to the learning process set of demonstration trajectories, or execution traces, of the agents in the swarm. Execution traces are discretely sampled trajectories specifying the position and velocity of each agent at time step These trajectories are used to train a graph neural network observed in the swarm. Our network takes the time series as input and generates the predicted next steps in the time series. The prediction error is measured by the mean squared error between output states and expected states from the true time series. The task of our network is to model the motion dynamics of the MAS with minimum prediction error. Without the assumption that the dynamics first-order Markovian, states earlier than the present state carry predictive power for future states and must be accounted for. Historical states and their higher order relations with the current states, for example, velocity and acceleration, necessary for making accurate predictions. Therefore, we apply a series of mee-iimnsional convolutional layers to process a history states for the agents. The ID convolution acts along the time steps of each agent and abstracts representation of its past as the starting point for prediction. This representation will then be taken as the input of the GNN.
model the group behavior
The interaction of the swarm system being embedded in graph, the GNN directly models the information propagation within the graph by explicitly computing the interaction between nodes and account for the influence from the interactions each node update the graph's state. The procedures can in general be identified as 3 steps. Firstly, interactions between all pairs of nodes connected through an edge are computed by function universal across all edges. The interaction, often called edge state, is determined only by its current state and the two nodes at both ends. Secondly, centered on each individual node, all interactions directed to are aggregated by another function universal to all nodes to form the local influence to the node. This step is sometimes called edge aggregation. Lastly, a third function universal to all nodes updates each node state based on the node's 's current state and aggregated influence, thus completing the state update of the entire network. Evolution graph dynamics is fully described by applying the procedures repeatedly. Details of structure and training procedures will be explained in the following subsections.
2.2 DATA STRUCTURE
The input time series of motion states from an instance of swarm simulation are arranged to have the shape T' x N x D, where T is the number of time steps, N is the number of agents, and D is the dimension of the state vector st) of agent at step t, where i, N, N, < T. The state vector contains both the position and velocity, i.e., sy(t) [x (t), X; (t)], where the square brackets stand for concatenation of the position (t) and velocity (t). In all subsequent experiments presented in this paper, all agents live in two-dimensional space, which lets the dimension of state vectors D 4. sequence with window length Tn, of history states (; Tun + 1), 8, - T., +
We will henceforth the term "swarm" and MAS interchangeably.
2

---

Under review as a conference paper at ICLR 2020
2),....s: s;(t 1), s,) are processed to output the next future state s;(t + is realized by repeatedly appending the predicted new step to the sequence of history state, dropping the earlier states and feeding the new sequence of length Tw as input.
1). Long term prediction
2.3 ID CONVOLUTIONS
The goal of one-dimensional convolutions is to abstract a concise representation of the historical states. The 1D convolutions with no padding condense the time sequences for each agent, until the output after the last ID convolution layer is reduced to 1 x N x C' shaped tensor, with C' being the number of channels of the last layer. For history window length Tw and a universal kernel size of K, this would require least [(T 1)/(K' 1)] layers since each layer condenses the length by -1 1. We typically choose the number of ID convolution layers and Tw such that they satisfy T, = L(K 1) 1. K 3 is what we have used throughout the experiments. As the 1D convolutional filters slide through the entire time series of trajectories of length T, T. T ++1 1-1 1 Tu, aub-sequences are condensed, each serving as the basis for prediction, and are sent to the following Graph Neural Network module to produce the predicted states for their corresponding next steps.
2.4 GRAPH NEURAL NETWORK
MAS along with their interactions are embedded in a directed graph 9 = (N,E), where N is the set of nodes which represent the agents, and & {(i,j) .N, * j} is the set of edges whose element (i. j) represents the influence from agent i to agent j through interaction. The state of the node i N is a vector of dimension d,, as v. E TRd.,, while the state of an edge (i. i, Wi of j is a vector of dimension de, that is, eij E Rdde In swarm motion, such interactions include pulling, pushing and stecring, and the edge states may be interpreted as an abstraction of the physical forces.
For each of the T, slices from the output of ID convolution module, the GNN module takes the N C' tensor as states of the N nodes. The node state of the next step inevitably is influenced by the interactions the node experiences. The interactions depend on the current and previous states of the involved nodes. As introduced earlier, this information passing process realized by GNN, formulated by 3 functions below:
Eij = 6 Uz),
(1)
é; = w e
(2)
JEN,
o/ b/ (v;,,,))
(3)
where Nr is the set of source nodes from the edges directed to node i. Function Φ', computes the influence from node i to node j. Function 16 aggregates the influences from all sources for node as the total influence e, And finally, function ø" transforms the total influence and the node's historical states into the prediction of the next step. Definition similar to these equations and thorough discussion of graph neural network may be found in Battaglia et al. (018)). All three functions are each approximated by a multi-layer perceptron (MLP).
Note that in Eq. 2 summation on all edge state vectors is taken before of is applied. It can be argued that influences from different sources may need to be weighted. However, the distinction between edges relies on the distinction of the nodes attached, which is already taken care of by function d". This also brings forth the point, that if the distinction between edges known before hand, a set of individual functions dV;, v2) can be created to cover the different interactions, where k E TE is the label of edge ij among the set of known types TF. The interaction influence between nodes is computed by of Vz) with respect to the edge labels & as part of input.
In principle, the group of GNN operations may be stacked one after another just like layers of multi- layer perceptron, where the output vector of of intermediate layers would serve as hidden states and only the output of the last layer is considered the update for the next step. In that case, interaction horizon gets expanded from the nearest neighbors, since information propagates further down the edges with each GNN layer added. Nevertheless, in this paper, we only employ one GNN layer in our model, is appropriate for the physical settings of swarm system.
is worth pointing out that vi and v/. despite both being the "node state" before and after GNN operations, may carry different meanings semantically, in the practice of this paper where we treat
3

---

Under review as a conference paper at ICLR 2020
the output v as the predicted change between the current state vector s, and future state vector st + 1) of agent i, s, + 1) = v' + (t)., while we assign the representation of history states to Vg. The algorithm of our GNN is presented in Algorithm I
Parallelization in the actual
implementation not included.
Data: History states (I:T.. I.N,E)), Next state (I:N. Edge type one-hot labels
E(::N.1:N.1TTe)
Result: ConvlD parameters w, Function &², 2/ o" parameters 81. 02, 0g
Random initialize w, 01, O2 03 (1:N.I:) f- -TT,,.:NN.:D
11 Current state
(1:N,1:D) Conv1D(S (1:T (:N. for in 1 N do
v + (()1::) ē x 0
for in 1 N do
+ V (..DD) for k Tg do
e + (v, ((j.))
+ et
end
é (e)
end
v' t (v. é)
S S.LDD) + S (.1:D) +
end
+ MS.(S(I.N,..D)) S (INN
(w, 01 O2 03) + Adam L, (w, 01, 02 03))
Algorithm A training step on swarm dataset
2.5 LOSS FUNCTION AND TRAINING
Prediction error is measured as the mean squared error (MSE) between the predicted states s; (L and the ground truth (L + 1) from motion data. Supervised training performed to adjust the parameters in the functions by minimizing the MSE between s; + 1) and st (t over all agents and all time steps.
Given that the simulated motion data is discretely sampled from trajectories, how fine grained the sample rate is affects the manifest of the error. Because this, the error is normalized by the average "natural skip" in the ground truth data, i.e. the MSE of state vectors between two consecutive steps in the ground truth trajectories, L,
T-1 N EE(0((+))) (s((1))-
7. 2DN(T - 1)
Normalized error Lnorm = rectifies the dependence of prediction error on the scale the intrinsic spacing in the ground-truth data.
Multistep prediction is enabled by appending the predicted state (1(+1) back to the sequence Tu, states and taking the new last Tw states the history ground for future prediction. Multi-step pre- diction builds up predictions in the far future on the predictions in the near future, and thus imposes higher demand despite harder to train. We gradually increase the difficulty of the prediction task by increasing the required number of prediction steps. This curriculum learning (Bengio et al. (2009) starting with short term prediction helps to guide the model faster in the earlier stage while still granting good long term prediction power later.
the accuracy and encourages the model leam the true mechanisms better,
SCALABILITY
Keen observation GNN's update rules Eq. 3 reveals that all 3 equations operate locally the neighborhood nodes. In other words, global attributes such as the number nodes are not involved. So long as swarm systems share the same set of mechanisms, manifested by the same set of
4

---

Under review as a conference paper at ICLR 2020
functions of, v5 and 6" in their underlying interaction graphs, the same GNN should be applicable to all the systems without the need for modification to the parameters, Although we argue later that Eq. 2 and aare not directly transferable, the statement still stands true that no change the structure of the GNN is required.
Indeed, closer inspection of Eq. raises suspicion that the number of edges connected node i may be implicitly correlated with the number of nodes in the graph. In reality, it is reasonable to expect agents to interact with more neighbors in larger swarm. Despite that the swarms share the same set of functions, when transferring to a swarm of a different size, function of may be exposed to an unseen range simply due to more edges to be aggregated. Adding to the fact that MLPs are poor at extrapolation (Hettiarachchi et al. 2005), function rh may fail to respond correctly, leading to drop in accuracy the GNN.
The scalability issue caused by poor extrapolation ability by MLP of Eq. and Eq. 3 would become prominent in more realistic swarm models. In realistic swarms, not only does the size of agent neighborhood change with the size of the swarm, but the neighborhood may change dynamically as well. When long interaction range is assumed, agents are often effectively fully connected, extend- ing the local neighborhood to the entire connectivity graph. The strong correlation between local connectivity and swarm size under this assumption makes swams-swarm application of a trained GNN almost impossible.
is plausible that the simple summation in Eq. is a bad choice. However, one may argue that popular numerical models of swarms such Boid (Reynolds) 1999) and Viesek (Helbing et al. 2000) that achieve high level of authenticity are partly or strictly physics based and are built on superposition of accelerations caused by pair wise interactions, realized by simple summation. In the scope of this paper, without deviating away from the proposed update rules of GNN, we propose a layer-wise tuning method realized by data padding as an attempt to reduce GNN's difficulty to scale.
An intuitive answer treat the lack of responsiveness is to adjust the functions for edge aggregation and node update through transfer learning. To avoid catastrophic forgetting (Kirkpatrick et al. 2017). however, the model has to constantly revisit previous data set. Although GNN in principle can freely adapt to different numbers of nodes, having a mixed dimension of input data poses problems to tensor based training frameworks. We note that a node with no interaction with any other node in graph exerts no influence on or be affected by the rest of the graph, and in reverse, adding an isolated node to a graph does not change the dynamics of the members of the original graph. This immediately implies that the input data of a certain swarm can be padded with an arbitrary number "ghost" agents with O edges in the connectivity matrix. Through state update of the graph, the edge message between "ghostt" and any other node is never accounted for, so the GNN works equally well on the real agents. In this way, input data may have a uniform shape in the dimension of N when mixed with swarms of different sizes. We choose initialize the states of "ghost" with zeros. Note that history states S (1N:D.. in Algorithm I is padded to the maximum N in dataset with mixed N's. During transfer learning, w, 01, 0g are locked, and only fg is to be updated in the last line.
EXPERIMENTS
Before exploring the solutions to scalability, we first show that our model performs better than other models in motion prediction on swarms of the size it is trained on (Table D. We test the accuracy of models' prediction on simulated Boid data and Vicsek data. In a Boid system, agents converge towards a goal while trying match neighbors' position and velocity without collision. In a Viesek system, agents only avoid each other via repulsive force. We artificially inject goals to Viesek systems similar to those in Boid systems to guide the agents. All three models are trained on both the Boid and the Vicsek data, with 10 agents only for each model, but are tested on respective models of different sizes. Table shows the MSE on test set for long term prediction upto 40 steps. The table also illustrate how non-GNN base model such Seq2Seq (Sutskever et al. 2014) is not able to be applied to swarms of different sizes. Kipf's GNN (Kipf et al. 2018) being tasked with inferring the latent edge types, comes with the burden of a more complex structure, and is hard to train. Our GNN variant, however, has the advantage of knowing the type labels of edges and a cleaner structure, can more easily capture the dynamics of the interactions, and produce accurate long term predictions.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Model comparison for prediction error
MODEL
LOSS
5 boids
10 boids 20 boids
5 vicseks
10 vicseks 20 vicseks
Seq2Seq
1.45++0.14
1.50++0.13
Kipf"s GNN
111=16 0.93=0.09
11.7=
~ 104
104
10.6#0.66
~ 106
Our GNN
0.41
1.69
0.24
0.05=
0.32:+0.04
Not only that, it already shows better scalability on data of unseen sizes, possibly due to its slimmer structure. Samples of predicted long term trajectories shown in the left pane of Fig. 2
Trained Tested on
Trained Tested
Tuned 10, Tested 10
Figure 2: Predicted trajectories (colored arrows) overlaid on ground truth (gray dash lines). Only To initial states are supplied, and the prediction is done repeatedly upto 40 steps in future. Left: Trajectories of 5 boids by network trained with data of boids. Middle: Trajectories of 10 boids by the same network used in the left pane. Right: Trajectories of 10 boids by the network after tuning on padded data of 5 and 10 boids.
Secondly, following earlier discussion in Section B prove the hypothesis that GNN naturally scales with N, we designed simplistic swarm model called chaser run experiments on as follows. A chaser is an agent that chases another particle with acceleration proportional to its displacement from its target. In chaser swarm, each particle is only influenced by one other particle its target, which means the interaction graph has only one edge attached towards each agent. The summation in Eq. becomes trivial as it equals the single edge state associated with agent and function of can be absorbed into function of when chained together. The second input to Eq. which is the aggregated edge message e, directly inherits the single edge message. So, just like each chaser is oblivious to the rest of the group except for its target, the update rules are independent of N.
We trained GNN on the simulated motion data from chaser swarm of only minimal number of agent, N = 3. Without any modification to the trained GNN, the test results on chaser swarms of various sizes
3, 5, 10. 20, 100 show perfect replication of the ground truth trajectory (Fig. 3 given only one window length of Tu initial steps as starting points. This simplistic case also seems indicate that a learned function df that governs the universal pair-wise interactions transferable across swarms of the same type.
To investigate how scalability is challenged by the change size the neighborhoods through summation in Eq. 2 we modify the dynamic rules of the chaser model, such that each agent may now chase more than target. We again train our model only on chaser swarm with N = 3 and various
M M being the number of targets each chaser has. The trained model is tested combinations of N and M. It is immediate to notice the accuracy degradation in the first row of To illustrate the shift in the input range function we and subsequently function d²,
Table 2

---

Under review as a conference paper at ICLR 2020
N N-
N- N- 20
Figure 3: Predicted trajectories for chaser swarms of different sizes N. Colored arrows mark the predicted velocities and positions, against the grey dashed lines as the ground truth trajectories. Every agent in these systems chases only other agent. Training was done only on systems with N 3.
plot the distribution of the norm of output vectors of the functions' preceding components (Fig. With the output of first function sharing similar distributions across systems, it is unsurprising that the ones with higher M's have the distribution of summation shifted to the right. Less overlapping is shared by the distributions when the distinction in M is larger, though same M still guarantees almost identical distribution. The strong resemblance by the distributions of function o output favors the presumption that the learned function df in GNN from one swarm readily transferable to other swarms with shared interaction rules.
d
sum of
AS 4
Trained on (3, 1)
0.4
(3.1)
0.3
(3.3)
0.4
13.1]
(5.2)
15.2)
(10,2)
(10,2)
(10.2)
0.2
(10.5)
0.2
(10.5)
(10.8)
0.1
(10.8)
(10.0)
0.0
0.0
0.0
10
12
20
40
60
10
15
Tured (10,
dersity
[3.1]
(5.2)
0.2
(3,3)
(5,2)
0.4
(3,31
(5,21
(10.5)
(10.5)
0.2-
(10.5)
(10.8)
(10.81
(10.8)
0.0
0.0
oulput norm
output norm
output norm
Figure 4: Distribution of output vector norms of critical components in our GNN, before and after tuning, on data sets with various M).. Left: Output of function &² Middle: Output of edge message summation before entering function v. Right: Final output of the network from function 6", as changes to agent states.
Next show in Table 2 how tuming only the Eq. 2 and Eq with padded data extend the perfor- mance on swarms of larger neighborhoods. For the chaser model, after training only on a dataset with 3 agent, each with target (N 3, M 1), only the validation set with the same size of neighborhood sees the same level of performance (e.g. N = 3, M = and N = 5, M = 1). We 10, and M varying between
tune the model with Eq. Eq [ocked on mixed of chasers with and 9. Note at this moment, padding is not necessary, because the nature of chaser model also allows a variety of neighborhood sizes without changing the swarm size. After tuning, performance on M less than 9 achieves similar levels. and is also greatly improved on larger swarms with larger neigh- borhoods. We see in Fig. that tuning has corrected functions' response to unseen ranges caused by aggregation on higher number of edge messages. The apparent resemblance of the corrected distributions is due to the design of the chaser model, which a particle chases the average position of all its targets, thus resulting in similar changes regardless of M. For other partly physics based models, the bias is not applicable, since having a higher number of interactions is often coupled with stronger changes. Fig. 5 shows the quality of the prediction even for larger unseen neighborhoods.
The ground for tuning only Eq. and Eq. B is based on the premises that training on one dataset, the network has already learned Eq. as interaction rules universal to all sizes of the same swarms. To show the importance of a learned function 6f and rule out the suspicion that the performance is solely determined by subsequent functions, lock function de right after random initialization and
7

---

Under review as a conference paper at ICLR 2020
Table 2: Chaser evaluation before and after tuning
TRAIN (N,
VALIDATION SET
(3.1)
(5,1)
(5,2)
(10,2)
(10,5)
(10,8)
(20,9)
(20,12)
(20,17)
(3, 1) (10,x)
0.051
0.036
17.02 0.0939
14.27 0.0799
104.93
275.3
350.7 0.291
701.4
1932
0.381
0.223
0.142
0.278
0.608
1.424
== 20, 33
= 20, M=
= 20, M=
20, M
= 20, =
Figure 5: Predicted trajectories after tuning against the ground truth for chaser model with various swarm sizes N and neighborhood sizes M. Only the trajectories of agents are shown for legibility.
train the other functions from scratch, the training error is never able be brought down below even for step prediction, let alone long term prediction.
For Boid, assume the agents fully connected. The number of interaction edges, i.e. the size of neighborhood, is equivalent to the size of the swarms. We initially train the network from scratch on 5 boids and tune with parameters of Eq. locked on mix of padded data of 5 and 10 boids (Table Compared to networks that is trained only on 5 boids or 10 boids from scratch, this tuning and padding method preserves good performance on both data sizes and ones between, while also avoiding catastrophic forgetting that appears in other transfer learning processes without mixed padded data. Trajectories in Fig. 2 show the improved performance from untuned network to tuned one. As comparison, catastrophic forget occurs when revisiting of old data is not available. For instance in Table 3 the network that is trained first with 5 boids and tuned only with 10 boids, performance peak has quickly moved from 5 boids to 10 boids. After tuning on 20 boids, the peak moved to 20 boids as well.
Table 3: Training method comparison
METHOD
LOSS
3
5
7
10
20
5
0.58+0.00
0.14++0.03
0.48=0008
1.66++0.28
9.6++2.0
10
1.96110:4
0.93
0.50-0.11
0.4110.14
1.69-0.15
20
3.01:30.403
1.78
1.31=0.06
0.8610003
0.6410001
5-+10
9.04+8.46
1.50=
0.42-0.04
0.2910.02
1.68=0.31
5-+5-10
0.6911.113
0.13:10.02
0.33-0.08
0.2810.02
1.74=0.27
10-+20
13.1+13.7
6.21+6.116
3.41-2.90
1.36+0.71
0.62+0003
CONCLUSION
In this paper, we proposed an implementation of GNN that predicts and imitates the motion behav- iors from observed swarm trajectory data. The network is combined with curriculum learning to achieve high accuracy prediction for arbitrary time steps. We demonstrated the network's 's ability to capture interaction dynamics in swarms through transfer learning. We discussed the availability and challenges the scalability of GNN, and proposed a method to improve it, by using layer-wist wise tuning and mixing of data enabled by padding.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Tamio Arai, Enrico Pagello, and Lynne E. Parker. Editorial: Advances in Multi-Robot Systems (584 cites). Ieee Transactions on Robotics and Automation, 18(5):655-66,, 2002.
Peter Battaglia, Jessica B. Hamrick, Vietor Bapst, Alvaro Sanchez-Gonzaler, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. pp. 1 1-40, 2018. doi: 10.1017/S0031 82005008516..
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. Pro- ceedings of the 26th annual international conference on machine learning, pp. 4 41-48, 2009.
Dirk Helbing, IlIés Farkas, and Tamás Vicsek. Simulating dynamical features of escape panic. Nature, 407(6803) 2000.
Hettiarachchi, M. Hall, and A. W. Minns. The extrapolation of artificial neural networks for the modelling of rainfallrunoff relationships. Journal Hydroinformatiis, 7(4):291-29) 2005.
Thomas Kipf, Ethan Fetaya, Kuan-chieh Wang Max, and Welling Richard. Neural Relational Infer- ence for Interacting Systems. 2018.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Gtrabsk--Barwinse Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13)33521-3526 2017.
Mehran Mesbahi and Magnus Egerstedt. Graph theoretic methods in multiagent networks. 2010.
ISBN 9780691 140612.
C. W Reynolds. Steering behaviors for autonomous characters. Game Developers Conference, pp- 763-782, 1999.
Craig W. Reynolds. Flocks, herds and schools: A distributed behavioral model. ACM SIGGRAPH Computer Graphics, 21:25-34, 1987.
Guillaume Sartoretti, Justin Kerr, Yunfei Shi, Glenn Wagner, K. Satish Kumar, Sven Koenig, and
Howie Choset. PRIMAL: Pathfinding via Reinforcement and Imitation Multi-Agent Learning. IEEE Robotics and Automation Letters, 4(3):237--2333 2385, 2019.
Stefan Schaal, Is imitation learning the route to humanoid robots? Trends in cognitive sciences, (6):233-242, 1999.
Ilya Sutskever, Oriol Vinyals, and Quoe V. Le. Sequence to Sequence Learning with Neural Net- works. Nips, pp. 9, 2014.


Reviews:
[{"review_id": "HJeeFJLRYB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors have proposed a GNN implementation that predicts and imitates the motion behaviors from observed swarm trajectory data. I have the following major comments on the paper:\n1. I think the authors should discuss more on the related works. They should clearly mention their contributions and difference compared to the prior works.\n2. What do the authors mean by 'functions universal across all nodes and edges'? Are these same functions for all nodes and edges?\n3. I feel that the section 2.3 explaining 1D convolutions is slightly separated from the previous and following sections.\n4. Why does the proposed GNN model performs better than Kipf's GNN for predicting and imitating the motion behaviors? What is the component that makes the difference?\n5. How does the model depend on the history window length T_w? Also I am curious how does the model depend on the number of GNN layers used?\n6. I am not quite satisfied with provided experimental evaluation of the paper. It is not clear to me why Kipf's GNN does not perform better than the proposed model (which simply aggregates the edge states). Also it would be interesting to try some other GNN models, such as ClusterGCN, GCI etc.\n7. Also I want to see a fair comparison in the paper. If the main contribution is a GNN model, then the model should be compared with the recent GNN models, and if the main contribution is predicting the motion behavior from observed swarm trajectory data, it should be compared with MAS based approaches. While reading the paper, it appeared to me that the authors proposed a graph-based method for solving MAS task and they compared with an old GNN method, which is not justified.\nI also have some minor comments as follows:\n1. In page 1, \"re-finement\" should be \"refinement\"\n2. In page 3, \"... my be found in Battaglia ...\" my should be may.\n3. In page 4, \"... sample rate is affects ...\" should be \"... sample rate affects ...\"\n4. In page 6, \"... a agent ...\" should be \"... an agent ...\"\n5. In page 7, \"... swarms of with larger ...\" either of or with\n6. In page 8, Table 3 ending parenthesis is missing. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJlkO_dstB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Review for \"Towards Scalable Imitation Learning For Multi-Agent Systems with Graph Neural Networks\".\n\nThe paper proposes a new time series model for learning a sequence of graphs. \n\nI vote to reject the paper for three reasons.\n\n1. Lack of significance. Algorithm 1 is essentially supervised learning of an autoregressive model with a GNN. The GNN definition is expanded out in the pseudocode, but seems to be completely standard. Moreover, Section 2.5 is also completely standard - you are using a sum of standard MSE losses. It should be much more concise. Also, there is typically no point scaling the loss by D if you use Adam (because the scaling they will eventually cancel out).\n\n2. Poor awareness of prior work. What you call \"scalability issue caused by poor extrapolation\" is normally called poor generalization. Improving generalization is an established problem in ML. You should cite some of that work. See [1] and follow the references.\n\n3. Poor experimental evaluation. Your experiments show that the competing GNN method (what you call Kipf's GNN) didn't converge in certain settings. Robust GNN implementations exist that converge for a wide range of reasonable graphs. You should have used one of them.\n\nAs things stand, because the criticisms concern the core of the submission, I have doubts the quality of the paper can be improved enough within the ICLR revision time to obtain an \"accept\" score. However, I wanted to encourage you not to give up. The building blocks that you have in the paper are very relevant and can be a basis for impactful work. You also write well (despite some minor issues). I hope these skills will help you make great submissions in the future!\n\nMinor points:\n1. I would not call what the paper is doing \"imitation learning\". Imitation learning normally means you have a controllable system and want to learn a policy from expert demonstrations. What this paper does is commonly referred to as learning an autoregressive time series model. \n2. Avoid colorful language. For example, the sentence: \"the apparent resemblance of the corrected distributions in the final output may inspire an inductive bias as part of training objective\" is unclear. \n\n[1] https://papers.nips.cc/paper/7176-exploring-generalization-in-deep-learning.pdf", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "B1x2QZWRFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work considers sequence prediction problems in a multi-agent system (MAS), which I think is different from imitation learning problems where agents try to mimic experts’ behavior given histories or states. In that sense, I think the title of this work should be changed so that readers are not confused. \n\nThe main idea of this work is to use (1) graph neural networks (GNNs) to learn the abstract information among multiple agents, (2) use 1D convolution to extract historical features of each agent, (3) and minimize the MSE loss function between true and predicted states to make expected successor states of multiple agents more accurate. The experiments show training in a small number of agents can be generalized and transferred to the setting in which there is a large number of agents. \n\nAlthough the proposed algorithm is practically useful, I believe the submission is premature to be accepted at a conference due to (1) the lack of comparison with existing works on multi-agent (reinforcement and imitation) learning and (2) the lack of novelty (It seems that the proposed method simply combines existing neural networks and applies it to multi-agent behavior prediction.). There’s a huge recent development on multi-agent RL and IL regarding the scalability of MARL (MF-MARL, https://arxiv.org/pdf/1802.05438.pdf), coordinated multi-agent imitation learning (https://arxiv.org/abs/1703.03121), multi-agent GAIL (https://arxiv.org/abs/1807.09936), etc, which should be considered as related literature. In addition to them, there’s a paper on arXiv that uses GNN for MARL (https://arxiv.org/abs/1810.09202), which may be deeply related to this work as well. \n\nI’ll definitely increase my score if I underestimate the quality of this work. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a graph neural network based approach for scaling up imitation learning (e.g., of swarm behaviors). Reviewers noted key limitations in the discussion of related work, size of the proposed contribution in terms of model novelty, and evaluation / comparison to strong baselines. Reviewers appreciated the author replies which resolved some concerns but agree that the paper is overall not ready for publication.
Decision: Reject