Multi-Agent Imitation Learning with Copulas
Authors: Hongwei Wang
Lantao Yu
Zhangjie Cao
Stefano Ermon
Venue:  (2021)
Keywords: multi-agent imitation learning
dependence modeling
copula

Abstract: Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions, which is essential for understanding physical, social, and team-play systems. However, most existing works on modeling multi-agent interactions typically assume that agents make independent decisions based on their observations, ignoring the complex dependence among agents. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents. Extensive experiments on synthetic and real-world datasets show that our model outperforms state-of-the-art baselines across various scenarios in the action prediction task, and is able to generate new trajectories close to expert demonstrations.

Under review as a conference paper at ICLR 2021
MULTI-AGENT IMITATION LEARNING WITH COPULAS
Anonymous authors Paper under double-blind review
ABSTRACT
Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions, which is essential for understanding physical, social, and team-play -lay systems. However. most existing works on modeling multi-agent interactions typically assume that agents make independent decisions based on their observations, ignoring the com- plex dependence among agents. In this paper, we propose to use copula, pow- erful statistical tool for capturing dependence among random variables, to explic- itly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral pat- terns of each individual agent, as well copula function that solely and fully captures the dependence structure among agents. Extensive experiments on syn- thetic and real-world datasets show that model outperforms state-of-the-art baselines across various scenarios in the action prediction task, and is able to gen- erate new trajectories close to expert demonstrations.
INTRODUCTION
Recent years have witnessed great success of reinforcement learning (RL) for single-agent sequential decision making tasks. As many real-world applications (e.g., multi-player games (Silver et al., 2017; Brown & Sandholm, 2019) and traffic light control (Chu et al., 2019)) involve the participation of multiple agents, multi-agent reinforcement learning (MARL) has gained more and more attention. However, a key limitation of RL and MARL is the difficulty of designing suitable reward functions for complex tasks with implicit goals (e.g., dialogue systems) (Russell, 1998; Ng al., 2000; Fu et al., 2017; Song et al., 2018). Indeed, hand-tuning reward functions to induce desired behaviors becomes especially challenging in multi-agent systems, since different agents may have completely different goals and vate-action representations (Yu et al., 2019).
Imitation learning provides an alternative approach to directly programming agents by taking ad- vantage of expert demonstrations on how task should be solved. Although appealing, most prior works on multi-agent imitation learning typically assume agents make independent decisions after observing state (i.e., mean-field factorization of the joint policy) (Zhan et al., 2018; Le et al., 2017; Song et al., 2018; Yu et al., 2019), ignoring the potentially complex dependencies that exist among agents. Recently, Tian al. (2019) and Liu et al. (2020) proposed to implement correlated poli- cies with opponent modeling, which incurs unnecessary modeling cost and redundancy, while still lacking coordination during execution.
Compared to the single-agent setting, one major and fundamental challenge in multi-agent learning how to model the dependence among multiple agents in an effective and scalable way. Inspired by probability theory and statistical dependence modeling, in this work, we propose to use copulas (Sklar, 1959b; Nelsen, 2007: Joe, 2014) to model multi-agent behavioral patterns. Copulas are pow- erful statistical tools to describe the dependence among random variables, which have been widely used in quantitative finance for risk measurement and portfolio optimization (Bouyé et al., 2000). Using a copulas-based multi-agent policy enables to separately learn marginals that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents. Such factorization is capable of modeling arbitrarily complex joint policy and leads to interpretable, efficient and scalable multi- agent imitation learning. As motivating example (see Figure 1), suppose there two agents, each with one-dimensi action space. In Figure la, although two joint policies are quite different, they actually share the same copula (dependence structure) and one marginal. Our proposed copula-based policy is capable

---

Under review as a conference paper at ICLR 2021
GaumtanjC,
Uniform 10
Gaueslon0, 1)
Unifomm (o,
Sumbel) (.7)
GasoslantD, 1)
Same copula but different marginals
(b) Same marginals different copulas
Figure 1: In each subfigure, the left part visualizes the joint policy T(11, 02|s) on the joint ac- tion space [-3,3] 3]² and the right part shows the corresponding marginal policies (e.g., I [[]ss) Sag (a1,, a2 )daz) well as the copula cumulative distribution function of the marginal (aslss)) and u, Fi(aa)s) is the uniformly dis- tributed random variable obtained by probability integral transform with F,. More details and defi- nitions can be found in Section 3.2.
(ails), (2(a2(s) on the unit cube. Here Fü is the
of capturing such information and more importantly, we may leverage such information to develop efficient algorithms for such transfer learning scenarios. For example, when we want to model team- play in a soccer game and one player is replaced by his/her substitute while the dependence among different roles are basically the same regardless of players, we can immediately obtain a new joint policy by switching in the new player's marginal while keeping the copula and other marginals un- changed. On the other hand, as shown in Figure I two different joint policies may share the same marginals while having different copulas, which implies that the mean-field policy in previous works (only modeling marginal policies and making independent decisions) cannot differentiate these two scenarios achieve coordination correctly.
Towards this end, in this paper, we propose a copula-basee based multi-agent imitation learning algorithm, which is interpretable, efficient and scalable for modeling complex multi-agent interactions. Ex- tensive experimental results on synthetic and real-world datasets show that our proposed method outperforms state-of-the-art multi-agent imitation learning methods various scenarios and gener- ates multi-agent trajectories close to expert demonstrations.
2 PRELIMINARIES
In this work, we consider the problem of multi-agent imitation learning under the framework of Markov games (Littman, 1994), which generalize Markov Decision Processes to multi-agent set- tings, where N agents are interacting with each other. Specifically, in Markov game, S is the common state space, A, is the action space for agent E (I...., NJ. 7) E P(S) is the ini- tial state distribution and P S A1 ... AN - P(S) is the state transition distribution of the environment that the agents are interacting with. Here P(S) denotes the set of probabil- ity distributions over state space S. Suppose at time I, agents observe s[1] S and take actions a(4] = (a, [1).....ay aN (4) E Ai x x AN. , the agents will observe state s + 1] E S at time t with probability Ptsit 1]]s[t], a, [t]t)............)) ... ay [t]). In this process, the agents select the joint action att] by sampling from stochastic joint policy π S + P(A1 X x Av). In the following, we will use subscript i denote all agents except for agent For example, (a;, a-;) represents the actions of all agents; (s) and (;(a,,s, a.;) represent the marginal and conditional policy of agent induced by the joint policy (a]s) (through marginalization and Bayes's 's rule respectively).
We consider the following lif-line imitation learning problem. Suppose we have access to set of demonstrations D {73 MM provided by some expert policy - (a]s), where each expert trajec-
tory 75 = {($ [t], a [t) is collected by the following sampling process:
5² 71ss), B (a))(((), 8/t 1] ((sls(t),a(()) for 1 (l,..., T).
2

---

Under review as a conference paper at ICLR 2021
The goal is to learn parametric joint policy л" to approximate the expert policy π such that can do downstream inferences (c.g., action prediction and trajectory generation). The learning prob- lem is off-line during training, and the reward is also unknown.
cannot ask for additional interactions with the expert policy the environment
MODELING MULTI-AGENT INTERACTIONS WITH COPULAS
3.1 MOTIVATION
Many modeling methods for multi-agent learning tasks employ simplifying mean-field assumption that the agents make independent action choices after observing state (Albrecht Stone, 2018; Song et al., 2018; Yu et al., 2019), which means the joint policy can be factorized as follows:
N
π Ta ON 8)
II n. (a:ls)
(1)
Such mecan-field assumption essentially allows for independent construction of each agent's 's policy. For example, multi-agent behavior cloning by maximum likelihood estimation now equivalent to performing N single-agent behavior cloning tasks:
max R (a.a)><ppg log π [a|s)] = Σ max TE (s.a:)))pp (log (a:lss)
where the occupancy measure Pr S x A1 X .... x AN + denotes the state action distribution encountered when navigating the environment using the joint policy π (Syed et al., 2008; Puterman, 2014) and Pr is the corresponding marginal occupancy measure.
However, when the expert agents are making correlated action choices (e.g., due to joint plan and communication in a soccer game), such simplifying modeling choice not able to capture the rich dependency structure and coordination among agent actions. To address this issue, recent works (Tian et al., 2019; Liu et al., 2020) propose to use a different factorization the joint policy such that the dependency among N agents be preserved:
π (a; as)) = Ta(as[s, --=-)=1((a-111
for f (1,..,,,
(3)
Although such a factorization is general and captures the dependency among multi-agent inter- actions, several issues still remain. First, the modeling cost is increased significantly, because now we need to learn N different and complicated opponent policies (=i(a_i(s) as well as N different marginal conditional policies n; (a, am;), (.))), each with a deep neural network. should be noted that there are many redundancies in such modeling choice. Specifically, sup- pose there are N agents and N 3, for agent and N, we need to learn opponent policies (a...a....... aN s) and wax(aa..... ......aa-lll avs= respectively. These are potentially high dimensional and might require flexible function approximations. However, the dependency structure among agent 2 to agent N - 1 are modeled in both __ and π N, which incurs unnecessary modeling cost. Second, when executing the policy, each agent makes decisions through its marginal pol- icy (a,sss)) _i._ e) (ayls, a_x) by first sampling a.; from its opponent policy m_ then sampling its action a; from (-(s a-)). Since each agent is performing such decision process inde- pendently, coordination among agents are still impossible due to sampling randomness. Moreover, a set of independently learned conditional distributions are not necessarily consistent with each other (i.e., induced by the same joint policy) (Yu et al., 2019).
In this work, to address above challenges, we draw inspiration from probability theory and propose to use copulas, a statistical tool for describing the dependency structure between random variables, model the complicated multi-agent interactions scalable and efficient way.
3.2 COPULAS
When the components of a multivariate random variable x = (x1, (N) are jointly independent, the density of æ can be written as:
N
p(x) p(x))
(4)
3

---

Under review as a conference paper at ICLR 2021
When the components not independent, this equality does not hold any more the dependen- cies among #1, .i..... N can not be captured by the marginals p(z.). However, the differences can be corrected by multiplying the right hand side of Equation (4) with function that only and fully de- scribes the dependency. Such a function is called a copula (Nelsen, 2007), a multivariate distribution function the unit hyper-cube with uniform marginals.
Intuitively, let us consider a random variable 2; with continuous cumulative distribution function Fü- Applying probability integral transform gives us random variable u, - (r,), which has standard uniform distribution. Thus one can use this property to separate the information in marginals from the dependency structures among 21....: n by first projecting each marginal onto one axis of the hyper-cube and then capture the pure dependency with distribution on the unit hyper-cube.
:...........
Formally. a copula is the joint distribution of random variables Wi..... UN. each of which is marginally uniformly distributed on the interval [0, 1]. Furthermore, we introduce the following theorem that provides the theoretical foundations of copulas:
Theorem ((Sklar, 1959a)), Suppose the multivariate random variable (II,..., has marginal cumalative distribution functions Fi, Fi...... ... FN and joint cumulative distribution function F, then there exists unique copula C : [0, 1]xh - [0, such that:
F(z IN) =
Fx(xx))
(5)
When the multivariate distribution has a joint density f and marginal densities f1...., , SN. we have:
N
f(11,... .......,NNNNN = II f((r)) - ((Fi(1)))......... FN((N))
(6)
where c is the probability density function of the copula. The converse is also true. Given a copula C' and marginals F. then C(Fi(z)). (xF((.).......(.)) (x(x)) F(N is N N-dimensional cumulative distribution function with marginal distributions F.(x;).
Theorem 1 states that every multivariate cumulative distribution function ([11....,NN) can be expressed in terms of its marginals (z;) and copula C (Z)() Fx Comparing Eq. (4) with Eq. (6), we can see that a copula function encoding correlations between random variables can be used to correct the mean-field approximation for arbitrarily complex distribution.
3.3 MULTI-AGENT IMITATION LEARNING WITH COPULA-BASIR POLICIES
central question multi-agent imitation learning how model the dependency structure among agent decisions properly. As discussed above, the framework of copulas provides mechanism to decouple the marginal policies (individual behavioral patterns) from the dependency left in the joint policy after removing the information in marginals. In this work, we advocate copula-based policy for multi-agent learning because copulas offer some unique and desirable properties in multi-agent scenarios. For example, suppose we want to model the interactions among players in sports game. Using copula-based policy, we will obtain marginal policies for each individual player as well as dependencies among different roles (e.g., forwards and midfielders in soccer). Such multi-agent learning framework has the following advantages:
Interpretable.
The learned copula density can be casily visualized to intuitively analyze the correlation among agent actions.
Scalable. When the marginal policy of agents changes but the dependency among different agents remain the same (e.g., in a soccer game, one player is replaced by his/her substitute, but the dependence among different roles are basically the same regardless of players), can obtain new joint policy efficiently by switching in the new agent's marginal while keeping the copula and other marginals unchanged.
The copula-based based factorization of the joint policy avoids the redundancy in pre- vious opponent modeling approaches (Tian et al., 2019; Liu et al., 2020) (as discussed in Section 3.1) by separately learning marginals and copula.
Succinet.
4

---

Under review as a conference paper at ICLR 2021
Algorithm 1: Training procedure
Input: The number of trajectories M. the length of trajectory T. the number of agents N. demonstratioons where each trajectory = [(>"[4) ('))))
{T Jr') ii=
Output: Marginal action distribution MLP MLPmarginat, state-dependent copula MLP M copula or
cate-indeepnndnt copula density c(-)
// Learning marginals
while M Marrginall not converge do
for each state-action pair (s, (a1,... aN) )) do
Calculate the conditional marginal action distributions for all agent:
(f,(-(s)))=11 + M MLLPmargnna (8)
for agent = 1,
do
Calculate the likelihood of the observed action aj: {aşl()
Maximize {a{ي)])) by optimizing M L. Marrgiina using gradient descent
/ Learning copula
while LP or c(-) not converge do for each state-action pair (s, (a1, ,aN)) do
{-)((1s))) - M. (arginuaa) for agent j 1, .N do (-1s)) + the CDF of fs(+ss Transform the agent action to uniformly distributed value + Fj
10
11
12
Obtain a = (M....... ,uy) in unit hyper-cuhe [0, 1|
13
14
if copula is set as state-dependent then
Calculate the detendeenndnnt copula density a(-ls) +- MLP L.Popuutal (s) Calculate the likelihood of c(uls)
17
Optimize MLP Leopoii by maximizing log c(uls) using gradient descent
18
else
Calculate the likelihood of c(u)
20
Optimize parameters c(-) using maximum likelihood or non-parametric methods
MLP MLP marginal, MLP copula or c(-)
Learning. In this section, we diseuss how learn copula-based policy from of expert demonstrations. Under the framework of Markov games and copulas, we factorize the parametric joint policy as:
N II Te ( (xt
ss π aN 0)
(7)
where Ty (a,]s;;; 0:) is the marginal policy of agent i with parameters 0, and is the corresponding cumulative distribution function; the function e (parameterized by 0.) is the density of the copula on the transformed actions u; FF (a,]s; 0,) obtained by processing original actions with probability integral transform.
The training algorithm of our proposed method is presented as Algorithm Given set of expert demonstrations D, our goal is to learn marginal actions of agents and their copula function. Our approach consists of two steps.! We first learn marginal action distributions of each agent given their current state (lines 1-6). This is achieved by training MLP MLPmarrina marginal that takes input state and output the parameters of marginal action distributions of N agents given the input state (line 3). our implementation, we use mixture of Gaussians to realize each marginal policy n, (a.|s; 0.) such that we can model complex multi-modal marginals while having a tractable form of the marginal cumulative distribution functions. Therefore, the output of MLPmmarinnal consists of the means, covariance, and weights of components for the N agents' Gaussian mixtures. We then calculate the likelihood of each observed action aj based on agent j's marginal action distribution (line 5), and maximize the likelihood by optimizing the parameters of M MLP marriinal (line 6).
An alternate approach is combine the steps together and end-to-end training, but this does perform well practice because the copula term is unlikely to converge before marginals are well-trained. Here assume that each agent aware of the whole system state. But model can be easily generalized the case where agents only aware of partial system by feeding the corresponding to their MI.Ps.
5

---

Under review as a conference paper at ICLR 2021
After learning marginals, we fix the parameters of marginal MLPs and start learning the copula (lines 7-20). We first process the original demonstrations using probability integral transform and obtain a set of new demonstrations with uniform marginals (lines 8-13). Then we learn the density of copula (lines 14-20). Notice that the copula can be implemented as either state-dependent (lines 14-17) or state-independent (lines 18-20): For state-dependent copula, we use ML. MT.Poopula
take as input the current state s and outputs the parameters of copula density c(-[s) (line 15). Then we calculate the likelihood of copula value u (line 16) and maximize the likelihood by updating M LP copouoi (line 17). For nate-indeeendent copula, we directly calculate the likelihood of copula value u under c(-) (line 19) and learn parameters of c(-) by maximizing the likelihood (line 20).
The copula density (c(-) or e(-ls) can be implemented using parametric methods such as Gaussian or mixture of Gaussians. It is worth noticing that if copula is state-independent, it can also be implemented using non-parametric methods such as kernel density estimation (Parzen, 1962; Davis et al., 2011), In this way, we no longer learn parameters of copula by maximizing likelihood as in lines 19-20, but simply store all copula values u for density estimation and sampling in inference stage. We will visualize the learned copula in experiments.
Inference and Generation. In inference stage, the goal is to predict the joint actions of all agents given their current state 8. The inference algorithm is presented as Algorithm 2 in Appendix A, where we first sample a copula value u (wI,"- : (N)) from the learned copula, either state- dependent or state-independent (lines 1-5), then apply inverse probability transform to transform them to the original action space: âj F (uyls) (lines 7-10). Note that an analytical form of the inverse cumulative distribution function may not always be available. In our implementation, we use binary search to approximately solve this problem since F, is strictly increasing function, which is shown to be highly efficient in practice. In addition, we can also sample multiple i.i.d. copula values from c(-Is) or c(-) (line 5), transform them into the original action space, and take their average as the predicted action. This strategy is shown to be able to improve the accuracy of action prediction (in terms of MSE loss), but requires more running time as a trade-off.
The generation algorithm is presented as Algorithm 3 in Appendix A. To generate new trajectories, we repeatedly predict agent actions given the current state (line 2), then execute the generated action and obtain an updated state from the environment (line 3).
The computational complexity of the training and the inference algorithms is analyzed as follows. The complexity of each round in Algorithm 1 is O(MTN), where M is the number of trajectories in the training set, the length of each trajectory, and N is the number of agents. The complexity of Algorithm 2 is O(N). The training and the inference algorithms scales linearly with the size of input dataset.
RELATED WORK
The key problem in multi-agent imitation learning is how to model the dependence structure among multiple interactive agents. Le et al. (2017) learn latent coordination model for players in coop- crative game, where different players occupy different roles. However, there are many other multi- agent scenarios where agents do not cooperate for same goal or they do not have specific roles (e.g., self-driving). Bhattacharyya et al. (2018) adopt parameter sharing trick to extend generative adversarial imitation learning to handle multi-agent problems, but does not model the interaction of agents. Interaction Network (Battaglia et al., 2016) learns a physical simulation of objects with binary relations, and CommNet (Sukhbaatar et al., 2016) learns dynamic communication among agents. But they fail to characterize the dependence among agent actions explicitly.
Researchers also propose to infer multi-agent relationship using graph techniques or attention mech- anism. For example, Kipf et al. (2018) propose to graph neural networks (GNN) infer the type of relationship among agents. Hoshen (2017) introduces attention mechanism into multi-agent pre- dictive modeling. Li et al. (2020) combine generative models and attention mechanism to capture behavior generating process of multi-agent systems. These works address the problem of reasoning relationship among agents rather than capturing their dependence when agents are making decisions.
Another line of related work is deep generative models in multi-agent systems. For example, Zhan et al. (2018) propose hierarchical framework with programmatically produced weak labels to gen-

---

Under review as a conference paper at ICLR 2021
28
Figure 2: Experimental environments (left to right): PhySim, Driving, and RoboCup.
erate realistic multi-agent trajectories of basketball game. Yeh et al. (2019) use GNN and variational recurrent neural networks (VRNN) to design permutation equivariant multi-agent trajectory gen- eration model for sports games. Ivanovic et al. (2018) combine conditional variational autoencoder (CVAE) and long-short term memory networks (LSTM) to generate behavior of basketball play- ers. Most of the existing works focus on agent behavior forecasting but provide limited information regarding the dependence among agent behaviors.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. We evaluate our method in three settings. PhySim is synthetic physical environment where 5 particles are connected by springs. Driving is synthetic driving environment where one vehicle follows another along a single lane. RoboCup is collected from an international scientific robot competition where two robot teams (including 22 robots) compete against each other. The detailed dataset description is provided in Appendix B. Experimental environments are shown in Figure 2.
Baselines. We compare our method with the following baselines: LR logistic regression model that predicts actions of agents using all of their states. SocialLSTM (Alahi et al., 2016) pre- dicts agent trajectory using RNNs with a social pooling layer in the hidden state nearby agents. IN (Battaglia et al., 2016) predicts agent states and their interactions using deep neural networks. CommNet (Sukhbaatar et al., 2016) simulates the inter-agent communication by broadcasting the hidden states of all agents and then predicts their actions. VAIN (Hoshen, 2017) works with attention mechanism for multi-agents modeling. NRI (Kipf et al., 2018) designs graph neural network based model to learn the interaction type among multiple agents. Since most of the baselines are used for predicting future states given historical state, we change the implementation of their objective functions and use them predict the current action agents given historical states. Each experiment is repeated 3 times, and we report the mean and standard deviation.
neural net-
5.2 RESULTS
We compare our method with baselines in the task of action prediction. The re- sults of root mean squared error (RMSE) between pre- dicted actions and ground- truth actions are presented in Table The number of Gaussian mixture com- ponents our method is set to for all datasets. The results demonstrate that all methods performs the best on Phy dataset, since agents in PhySim follow simple physical rules and the relation- ships among them are linear thus easy to infer. However, the interactions of agents in Driving and complicated, which causes LR and NRI to underperform other base-
STcialL.TM
IN
CommNel
VAIN
NRI
Copula
PhySim
0.061
0.002
0.186
0.032
1 0.013
0.087
1. 0.007
0.089
1. 0.010
0.082
1 0.011
0.055
4 0.005
0.037
Driving
0.335
0.007
0.283
0.024
4 0.033
0,247
4 0.028
0.258
4 0.031
0.242
0.296
0.018
0.158
0.019
RoboCup
0.478
0.009
0.051
0.024
0.042
+ 0.028
0.315
0.042
Table 1: Root mean squared error (RMSE) between predicted actions and ground -tuth actions for our method and baselines.
RoboCup datasets are lines. The performance of IN, CommNet, and VAIN are similar, which in accordance with the result reported in Hoshen (2017). Our method is shown outperform all baselines significantly
7

---

Under review as a conference paper at ICLR 2021
on all three datasets, which demonstrates that explicitly characterizing dependence of agent actions could greatly improve the performance of multi-agent behavior modeling.
To investigate the efficacy of copula, we implement three types of copula function: Uniform copula means we do not model dependence among agent actions. KDE copula uses kernel density estimation model the copula function, which is state- independent. Gaussian mixtures copula uses Gaussian mixture model to character- ize the copula function, of which the pa- rameters are output by an MLP taking input the current state. We train the three models on training trajectories, then calcu- late negative log-likelihooo (NLL) of test trajectories using the three trained models. A lower NLL score means that the model assigns high likelihood to given trajectories, showing that it is better at characterizing the dataset. The NLL scores of the three models on the three datasets are reported in Table 2. The performance of KDE copula and Gaussian copula both surpasses uniform copula, which demonstrates that modeling de- pendence among agent actions is essential for improving model expressiveness. However, Gaussian copula performs worse than KDE copula, because Gaussian copula is state-dependent thus increases the risk of overfitting. Notice that the performance gap between KDE and Gaussian copula is less on PhySim, since PhySim dataset is much larger so the Gaussian copula can be trained more effectively.
Uniform copula
KDE copula
Gaussian mix. copula
Phm
8,994 : 0.001
1.256 E 0.006
2,893 1 0.012
Driving
-0.571 = 0.024
-0.916 = 0.017
-0.621 = 0.028
RoboCup
3.243 + 0.049 0.068 + 0.052
3.124 +: 0.061
Table 2:
Negative likeliiel (NLL) of test trajecto- ries evaluated by different copula. Uniform copula as- sumes dependence among agent actions. KDE cop- ula uses kernel density estimation to model the copula, which is hate-indeepeddet. Gaussian mixtures copula uses Gaussian mixture model to characterize the cop- ula, which is state-dependent.
5.3
GENERALIZATION CAPABILITY OF COPULA
One benefit of copulas is that copula captures the pure dependence among agents, regardless of their own marginal action distri- butions. To demonstrate the generalization capabilities of copulas, we design the following experiment. We first train model on the original dataset, and learn marginal action distributions and copula function (which is called original marginals and original copula). Then we substitute one of the agents with new agent and use the simulator to generate a new set of trajectories. Specifically, this is achieved by doubling the action value of one agent (for example, this can be seen as substituting an existing particle with a lighter one in PhySim). We retrain our model on new trajectories and learn new marginals and new copula. We evaluate the likelihood of new trajectories based on whether to use the original marginals or original copula, which, accordingly, results in four combinations. The NLL scores of four combinations are presented in Table 3. clear, by comparing the first and the last column, that "new marginals + new copula" significantly outperform "original marginals + original copula". since new marginals and new copula are trained on new trajectories and therefore characterize the new joint distribution exactly. To see the influence of marginals and copula more clearly, we further compare the results in column 2 and 3, where we use new copula or new marginals separately. is clear that the model performance does not drop significantly if we use the original copula and new marginals (by comparing column 3 and 4), which demonstrates that the copula function basically stays the same even if marginals are changed. The result supports our claim that the learned copula is generalizable in the case where marginal action distributions of agents change but the internal inter-agent relationship stays the same.
Original marginals Original marginals
New marginals
New marginals
original copola
new copula
+oiiinal copula
new copula
PhySim
10.231 0.562
8.775
0,497
1.301 = 0.016
1,259 + 0.065
Driving
13.184 + 1.627
13.602 = 0.945
0.447 = 0.085
-0.953 + 0.024
RoboCup
4.278 1 0.452
4.121 0.658
0.114 +: 0.020
0.077 1 0.044
Negative log-likelihood (NLL) of new test trajectories in which the action distribution of one agent is changed. We evaluate the new test trajectories based on whether to use the original marginal action distributions or copula, which results in four combinations.
Table 3:
5.4 COPULA VISUALIZATION
Another benefit of copulas is that it is able to intuitively demonstrate the correlation among agent actions. We choose the RoboCup dataset to visualize the learned copula. As shown in Figure 4a
8

---

Under review as a conference paper at ICLR 2021
in Appendix D, we first randomly select game (the 6th game) between cyrus2017 and helios2017 and draw trajectories of 10 players in the left team (L2 ~ LI1, except the goalkeeper). It is clear that the 10 players fulfill specific roles: L2 L4 are defenders, L5 ~ L8 are midfielders, and L9 L11 are forwards. Then we plot the copula density between the x-axis (the horizontal direction) of 1.2 and the x-axis of L.3 L11, respectively, illustrate linear correlation between their moving direction along x-axis, that is, when L2 moves forward other players are also likely move forward. However, the correlation strength differs with respect to different players according to the visualized result: L2 exhibits high correlation with L3 and L4, but low correlation with L9 L11. This is because L2 L4 are all defenders so they collaborate more closely with each other, but L9 ~ L11 are forwards thus far from L2 in the field.
shown in Figure 4b in Appendix D. These figures
5.5 TRAJECTORY GENERATION
The learned copula can also be used to gen- erate new trajectories. We visualize the result of trajectory generation on RobuCup dataset. As shown in Figure 3, the dotted lines denote the ground-truth trajectories of the 10 player in an attack from midfield to the penalty area. The trajectories generated by our copula model (Figure 3b) are quite similar to the demonstra- tion as they exhibit high consistency. It is clear that midfielders and forwards (No. ~ No. 11) are basically moving to the same direction, and they all make left on their way to penalty area. However, the generated trajecto- ries by independent modeling show little corre- lation since the players are all making independent decisions. We also present the result of trajectory generation on Driving dataset Appendix E.
(a) Independent
(b) Copula
Figure 3: Generated trajectories on RoboCup
dataset using independent modeling or copula. Dotted lines are ground-truth trajectories and solid lines generated trajectories.
CONCLUSION AND FUTURE WORK
In this paper, propose copula-based multi-agent imitation learning algorithm that is inter- pretable, efficient and scalable to model complex multi-agent interactions. Sklar's 's theorem allows us separately learn marginal policies that capture the local behavioral patterns of each individual agent and a copula function that only and fully captures the dependence structure among the agents. Compared to previous multi-agent imitation learning methods based on independent policies (mean- field factorization of the joint policy) or opponent modeling, our method is capable of modeling complex dependence among agents and achieving coordination without any modeling redundancy. Experimental results on physical simulation, driving and robot soccer datasets demonstrate the ef- fectiveness of our method compared with state-of-the-ar baselines.
We point out two directions of future work. First, the copula function is generalizable only if the dependence structure of agents (i.e., their role assignment) is unchanged. Therefore, it is interest- ing to study how to efficiently apply the learned copula to the scenario with evolving dependence structure. Another practical question is that whether our proposed method can be extended to the setting of decentralized execution, since the step of copula sampling (line 3 or 5 in Algorithm 2) is shared by all agents. straightforward way to solve this problem is to set a fixed sequence of random seeds for all agents in advance, that the copula sample obtained by all agents is the same at each timestamp. Designing a more robust and elegant mechanism for decentralized execution is also a promising direction.
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 961-971, 2016.

---

Under review as a conference paper at ICLR 2021
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-999 95, 2018.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems. pp. 4502-4510, 2016.
Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer. Multi-agent imitation learning for driving simulation. In 2018 JEEE/RSJ Interna- fional Conference on Intelligent Robots and Systems (IROS), pp. 1534 1539. IEEE, 2018.
Eric Bouyé, Valdo Durrleman, Ashkan Nikeghbali, Gael Riboulet, and Thierry Ronealli. Copulas for finance-a reading guide and some applications. Available at SSRN 1032533, 2000.
Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456): 885-890, 2019.
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions Intelligent Transportation Systems, 21(3): 1086-1095, 2019.
Richard A Davis, Keh-Shin Lii, and Dimitris N Politis. Remarks of a density function. Selected Works of Murray Rosenblatt, pp. 95-100. Springer, 2011.
nonparametric estimates
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In Proceedings of 1st Annual Conference on Robot Learning, pp. 1-16, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning. arXiv preprint arXiv:1710.11248, 2017.
Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. In Advances in Neural Informa- tion Processing Systems, pp. 2701-2711 22111, 2017.
Boris Ivanovic, Edward Schmerling, Karen Leung, and Marco Pavone. Generative modeling of multimodal multi-human behavior. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp. 3088-3095 IEEE, 2018
Harry Joe. Dependence modeling with copulas. Chapman and Hal//CRC, 2014.
Thomas N Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard $ 7emel. Neural relational inference for interacting systems. In International Conference on Machine Learning, 2018.
Hoang M Lee Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proceedings of the 34th International Conference on Machine Learning, pp. 1995-2003, 2017.
Max Guangyu Li, Bo Jiang, Hao Zhu, Zhengping Che, and Yan Liu. Generative attention networks for multi-agent behavioral modeling. In AAAI, pp. 7195-7202, 2020.
Michael L Littman. Markov games as a framework for multi-agent reinforcement learning. In Machine learning proceedings 1994, pp. 157-163. Elsevier, 1994.
Minghuan Liu, Ming Zhou, Weinan Zhang, Yuzheng Zhuang, Jun Wang, Wulong Liu, and Yong Yu. Multi-agent interactions modeling with correlated policies, arXiv preprint arXiv:2001.03415. 2020.
Olivia Michael, Oliver Obst, Falk Schmidsberger, and Frieder Stolzenburg. Robocupsimdata: A robocup soccer research dataset. arXiv preprint 1rX1v:1711.01733, 2017.
Roger B Nelsen. An introduction to copulas. Springer Science & Business Media, 2007.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2, 2000.
10

---

Under review as a conference paper at ICLR 2021
Emanuel Parzen. On estimation of probability density function and mode. The annals of mathe- matical statistics, 33(3):1065-107,, 1962.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programmingg John Wiley & Sons, 2014.
Stuart Russell. Learning agents for uncertain environments. Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 2017.
A. Sklar. Fonetions de Répartition à Dimensions et Leurs Marges. Publications de I. L'Institut de Statistique de L' L'Univesitéé de Paris, 8:229-231, 1959a.
M Sklar. Fonctions de repartition an dimensions et leurs marges. Publ. inst. statist. univ. Paris, 8: 229-231, 1959b.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-agent generative adversarial imitation learning. In Advances in neural information processing systems, pp. 7461-7472, 2018.
Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropaga- tion. In Advances in neural information processing systems, pp. 2244-2252, 2016.
Umar Syed, Michael Bowling, and Robert E Schapire. Apprenticeship learning using linear pro- gramming. Proceedings of the 25th international conference on Machine learning, pp. 1032- 1039, 2008.
Zheng Tian, Ying Wen, Zhichen Gong. Faiz Punakkath, Shihao Zou, and Jun Wang. A regularized opponent model with maximum entropy objective. arXiv preprint arXiv:1905. 08087, 2019.
Raymond A Yeh, Alexander G Schwing, Jonathan Huang, and Kevin Murphy. Diverse generation for multi-agent sports games. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4610-4619, 2019.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-agent adversarial inverse reinforcement learn- ing. In International Conference on Machine Learning, 2019.
Eric Zhan, Stephan Zheng, Yisong Yue, Long Sha, and Patrick Lucey. Generating multi-agent trajectories using programmatic weak supervision. arXiv preprint 18X3v:/803.076/2 2018.

---

Under review as a conference paper at ICLR 2021
A PSEUDO CODE FOR TRAINING, INFERENCE, AND GENERATION PROCEDURE
The pseudo code for inference and generation procedure are presented in Algorithm 2 and 3, respec- tively.
Algorithm 2: Inference procedure
Input: Marginal action distribution MI.P MT. M.I.Parginna: marginals state-dependent copula ML. MTP copuia
state-independent copula density c(-), current
Output: Predicted action
// Sample from copula
if copula is set state-dependent then
Calculate (parameters of) state-dependent copula density c(-[s) +- MLP coopuii
Sample copula value (M1,-.. ux) from c(-Is)
else
Sample copula value (M uN) from c(-)
// Trans form from copula space to action space Calculate (parameters of) the conditional marginal action distributions for all agents:
{-f(-1s))}=1 + LL marginaii
for agent = 1.--- ,N do
F)(1(s) < CDF of f+(ss)
0 + F (w,
10
(a1,-..,0) a5)
11 return à
Algorithm 3: Generation procedure
Input: Inference module (Algorithm initial state =(0]. required length L, environment Output: Generated trajectory
for 0.... :L do
Feed state s(t] to the inference module and get the predicted action ä[l]
Exceute ä({] in environment & and get new state s(1
f = {(s(() (
return f
B DATASET DETAILS
PhySim is collected from a physical simulation environment where 5 particles move in a unit 2D box. The state is locations of all particles and the action is their acceleration (there is need to in- clude their velocities in state because accelerations are completely determined by particle locations). We add Gaussian noise to the observed values of actions. Partieles may be pairwise connected by springs, which can be represented as a binary adjacency matrix A {0, 1 The elasticity be- tween two particles scales linearly with their distance, At each timestamp, we randomly sample an adjacency matrix from {A1, A2} to connect all particles, where A₁ and A₂ are set complimen- tary (i.e. A A2 I 1) to ensure that they are as different possible. Therefore, the marginal action distribution of each particle given system state is Gaussian mixtures with two components. Here the coordination signal for particles can seen as the hidden variable determining which set of springs (A, or Ag) is used at current time. We generate 10, 000 training trajectories, 2, 000 val- idation trajectories, and 2, 000 test trajectories for experiments, where the length of each trajectory is 500.
Driving is generated by CARLA (Dasovitskiy et al., 2017). an open-source simulator for au- tonomous driving research that provides realistie urban environments for training and validation of autonomous driving systems. To generate the driving data, we design a car following scenario,
hhttps:/ccalaa.ogg)
12

---

Under review as a conference paper at ICLR 2021
where a leader and follower car drive in the same lane. We make the leader car alternatively accelerate to speed upper bound and slow down to stopping. The leader car does not care about the follower and drives following its own policy. The follower car tries to follow closely the leader car while keeping a safe distance. Here the state is the locations and velocities of the two cars, and the action is their accelerations. We generate 000 trajectories in total, and split the whole data into training, validation, and test set with ratio of 6 2 2. The average length of trajectories is 85.5 in Driving dataset.
RuboCup (Michael et al., 2017) is collected from an international scientific robot football compe- tition in which teams of multiple robots compete against each other. The original dataset contains all pairings of 10 teams with 25 repetitions of each game (1, 125 games in total). The state of a game (locations and velocities of 22 robots) recorded every 100 ms, resulting a trajectory of length 6, 000 for each game (10 min). We select the 25 games between two teams, cyrus2017 and helios2017. 7, as the data used in this paper. The state is locations of 10 robots (except the goalkeeper) in the left team, and the action is their velocities. The dataset is split into training, validation, and test set with ratio of 6 : 2.
For the three datasets, to learn the marginal action distribution of each agent (i.e. Gaussian mixtures), we use an MLP with one hidden layer to take as input a state and output the centers of their Gaussian mixtures. To prevent overfitting, the variance of these Gaussian mixtures is parameterized by free variable for each particle, and the weights of mixtures are set as uniform. Each dimension of states and actions in the original datasets particles are set to 5. Learning rate is set to 0.01, and the weight of L2 regularizer is set to 10-3 For Driving, learning rate is 0.005 and L2 regularizer weight is 10-s For RoboCup, learning rate 0.001 and L2 regularizer weight is 10-6
normalized range [-1, 1]. For PhySim, the number of
C BASELINE IMPLEMENTATION DETAILS
For LR, we use the default implementation in Python sklearn package. For SocialLSTM (Alahi al.. 2016), the dimension of imput is set as the dimension of states in each dataset. The spatial pooling size is 32, and we use an 8 x 8 sum pooling window size without overlaps. The hidden state dimension in L.ST is 128. The learning rate is 0.001. For IN (Battaglia et al., 2016), all MLPs are with one hidden layer of 32 units. The learning rate is 0.005. For CommNet (Sukhbaatar et al., 2016), all MLPs are with one hidden layer of 32 units. The dimension of hidden states set to 64, and the number of communication round is set to 2. The learning rate is 0.001. For VAIN (Hoshen, 2017), the encoder and decoder functions are implemented as a fully connected neural network with one hidden layer of 32 units. The dimension of hidden states is 64, and the dimension of attention vectors 10. The learning rate is 0.0005. For NRI (Kipf et al., 2018), we use an MLP encoder and an MI. decoder, with one hidden layer of 32 units. The learning rate is 0.001.
D VISUALIZED COPULA ON ROBOCUP DATASET
The trajectories of players in one game well the visualized pairwise copula are presented in Figure 4.
E GENERATED TRAJECTORIES ON DRIVING DATASET
For the Driving dataset, we randomly select 10 original trajectories and 10 trajectories generated by our method, and show the visualization results in Figure 5. The x-axis timestamp and y-axis is the location (coordinate) of two cars. Our learned policy is shown to be able to maintain the distance between two cars.
13

---

Under review as a conference paper at ICLR 2021
LS
L2X
0.8
0.9
0.
0.25
0.50
0.25
6.50
0.75
0.25
0.75
L2X
L2x
L2X
0.5
0.8
0.6
0.6
0.6
110
111
0.4
0.4
0.
0.2
0.2
02
0.25
0.75
0.23
0,50
0.72
5.30
0.75
L2X
L2x
L2X
(a)
(b)
Figure 4: (a) Trajectories of 10 players (except the goalkeeper) of the left team in one RoboCup game; (b) Copula density between x-axis the L2 player and x-axis of another player (L3 ~ L11).
(a) Original trajectories
(b) Generated trajectories
Figure 5: Original and generated trajectories Driving dataset. The x-axis is timestamp and y-axis the location (1D coordinate) of two cars.
14


Reviews:
[{"review_id": "xCMTUyYOl3R", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper discusses the use of copulas for the multi-agent imitation learning scenario. The paper addresses an interesting application of copulas and is generally well written. Efficient imitation learning for MAS is a challenging problem, hence the paper is well motivated. While I like the overall idea of the paper, there are some concerns which the authors should address:\n\n1. The authors claim that learning a copula is more efficient and scalable, however I don't feel this claim is well justified in the paper. In general, a copula captures arbitrary dependencies between the action distributions of $n$ agents, how exactly is then learning a copula any different from learning a joint distribution $\\pi(a_1..a_n)$ in terms of learning complexity?\n\n2. The authors should mention the approximate order complexity of the Algorithms 1, 2 and in general need to compare those with the competing methods mentioned in the paper to give a better understanding of the advantage using copula based policy learning.\n\n3.  In general, maximising over a copula seems like a very hard problem, which is one of the reasons why mean-field methods are proposed, how exactly is the copula based factorisation avoiding this problem?\n\n4. The authors should provide the architecture details of baseline methods to better interpret the results.\n\n5. Root mean squared error (RMSE) between predicted actions and ground-truth actions for Table 1 exps is not a very convincing metric, specially from the standpoint of qualitative nature of policies imitated. It would be better if the authors can report reward scores for some suitable domains (ex. RoboCup).", "extra_scores": {}, "extra_text": {}}, {"review_id": "f5Er0VB2TzZ", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a multi-agent imitation learning method that learns a joint policy for all agents from offline demonstrations. The key idea is to first learn a marginal policy for each agent using behavioral cloning, then fit a copula function that captures dependencies between the agents' policies. Experiments with particle simulations, simulated driving, and simulated RoboCup suggest that this approach could potentially outperform prior methods.\n\nTheorem 1 seems like a powerful tool for multi-agent imitation learning, and I think this paper is a promising initial step. That being said, there are several issues:\n\nIn Tables 2 and 3, there are negative values for the negative log-likelihood. This is not possible, since log-likelihoods cannot be positive. I initially assumed that this was a typo in Table 2, but then I saw that Table 3 included both negative and positive values for the NLL, which makes me unsure about the overall correctness of the experimental evaluations in this paper.\n\nAlgorithm 1 uses behavioral cloning to fit the marginal policies. Doesn't this lead to compounding errors due to state distribution mismatch? Would more recent methods for offline imitation learning that overcome this issue, like ValueDICE [1], be a more suitable choice than behavioral cloning?\n\nDoesn't learning the marginals first, then the copula second, lead to a model misspecification when fitting the marginals? Learning the marginals separately (without a copula) effectively assumes the factorization in Equation 1, which we know cannot represent the ground-truth joint policies in the experiments, so the learned marginals from step 1 are not necessarily the same marginals that you would learn if you were to fit the marginals and the copula simultaneously.\n\nFrom Equation 7, we can see that different combinations of marginals and copulas could potentially induce the same joint policy. If you only fit the model by maximizing the log-likelihood of demonstrations for a single task, how can you distinguish between different pairs of (marginals, copulas) that fit the demonstrations equally well? Wouldn't you need to train on multiple tasks that share the same underlying copula but have different marginals, or on multiple tasks that share the same underlying marginals but have different copulas?\n\nIt would be nice to discuss the weaknesses of the proposed method, as well as directions for future work. For example, could this approach to learning a joint policy be extended to settings that require decentralized execution?\n\nThe dotted lines in Figure 3 are very difficult to see because they are small and because they are dark gray against a dark green background.\n\nIt would be nice to include the pseudocode in the main paper, since the main contribution of the paper is the algorithm.\n\n[1] https://openreview.net/pdf?id=Hyg-JC4FDr \n\nUpdate after rebuttal\n-----------------------------\nThanks to the authors for addressing my concerns. I have updated my score.", "extra_scores": {}, "extra_text": {}}, {"review_id": "MAD9HmUe1VF", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper proposes the use of \"Copulas\" to capture dependencies among agents in multi-player environments. The authors argue that prior work(eg GNN, VAE, LSTM etc based) do not leverage the common structure among agent behaviour. They explain how the copula can more efficiently encode the dependency among policies, as compared to simply factorizing. The authors provide empirical results on real world and synthetic datasets, showcasing superior performance against baseline approaches.\n\nOverall, this seems like a novel approach with promising results on a few experimental set ups. The authors describe how their proposed model builds upon the limitations of prior work. The work seems moderately significant.\n\nComments:\n\n* The writing is easy to follow. The paper motivate the problem well and describe the experimentation clearly.\n\n* Nit: In Figure 1b) Can the authors clarify how the copula's differ in the upper figure? It seems to be a uniform distribution, I assume this applies for only the lower figure.\n\n* For equation (3), can the notation be clarified: this refers to learning opponent policies rather than agents co-operating in a given task?\n\n* Nit: Algorithm 2 can perhaps be described in the main section, being the central idea to this paper.\n\n* For the datasets, can the number of agents in each be briefly described in Section 5.1 or Table 1? Is there any dependence on performance with respect to this parameter? The authors have mentioned heir intuition behind some of the results in Section 5.2.\n\n* In Section 5.3, how were the replacement agents sampled? For eg, in the RoboCup dataset used, would substituting a defender for a forward influence the performance(as hinted in Section 5.4)?\n\n\nGiven the above points, novelty and significance  marking this as accept.\n\nUpdate:\nAfter seeing the author response below, no change to my score.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies an existing tool (copulas) to MARL to represent dependencies between variables.

The reviewers appreciate the use of the copulas for this problem. The experimental section shows promising results on several problems. I appreciate that the authors have answered and addressed many points of concerns of the reviewers. The paper is well written

The reviewers seem to see this paper as a first step only, showing promising results but of moderate significance in itself. In particular, reviewer 3 would like to see more justifications for the use of the copulas, and a more experimental settings would make a stronger paper. 

Decision: Reject