Multi-Agent Interactions Modeling with Correlated Policies
Authors: Minghuan Liu
Ming Zhou
Weinan Zhang
Yuzheng Zhuang
Jun Wang
Wulong Liu
Yong Yu
Venue:  (2020)
Keywords: Multi-agent reinforcement learning
Imitation learning

Abstract: In multi-agent systems, complex interacting behaviors arise due to the high correlations among agents. However, previous work on modeling multi-agent interactions from demonstrations is primarily constrained by assuming the independence among policies and their reward structures. 
In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with explicit modeling of correlated policies by approximating opponents’ policies, which can recover agents' policies that can regenerate similar interactions. Consequently, we develop a Decentralized Adversarial Imitation Learning algorithm with Correlated policies (CoDAIL), which allows for decentralized training and execution. Various experiments demonstrate that CoDAIL can better regenerate complex interactions close to the demonstrators and outperforms state-of-the-art multi-agent imitation learning methods. Our code is available at \url{https://github.com/apexrl/CoDAIL}.

Published as a conference paper at ICLR 2020
MULTI-AGENT INTERACTIONS MODELING WITH COR- RELATED POLICIES
Minghuan Liu', Ming Zhou', Weinan Zhang / Yuzheng Zhuang,, Jun Wang', Wulong Liu", Yong Yu' 1 Shanghai Jiaotong University, 2 HHaawii Noah's Ark Lab
{ (minghuanliu, mingak, wnzhang, yyu) @ @jjtu.edu.cn, (ertuueduucn,,uuuaeee {zhuangyuzheng, w.j. liuwulong ) @ @huawei.com
ABSTRACT
In multi-agent systems, complex interacting behaviors arise due to the high cor- relations among agents. However, previous work on modeling multi-agent inter- actions from demonstrations is primarily constrained by assuming the inddpen- dence among policies and their reward structures, In this paper, we cast the multi- agent interactions modeling problem into a multi-agent imitation learning frame- work with explicit modeling of correlated policies by approximating opponents' policies, which can recover agents' policies that can regenerate similar interac- tions. Consequently, we develop a Decentralized Adversarial Imitation Learn- ing algorithm with Correlated policies (CoDAIL), which allows for decentralized training and execution. Various experiments demonstrate that CoDAIL. can bet- ter regenerate complex interactions close the demonstrators and outperforms state-of-the-art multi-agent imitation learning methods. Our code is available at https://githee com/ /ttp//ape//cDDD......
INTRODUCTION
Modeling complex interactions among intelligent agents from the real world is essential for under- standing and creating intelligent multi-agent behaviors, which is typically formulated multi- agent learning (MAL) problem in multi-agent systems. When the system dynamics are agnostic and non-stationary due to the adaptive agents with implicit goals, multi-agent reinforcement learning (MARL.) is the most commonly used technique for MAL. MARL has recently drawn much atten- tion and achieved impressive progress on various non-trivial tasks, such as multi-player strategy games (OpenAI, 2018; Jaderberg et al., 2018), traffic light control (Chu et al., 2019), taxi-order dispatching (Li et al., 2019) etc.
A central challenge in MARL is to specify good learning goal, as the agents' rewards are cor- related and thus cannot be maximized independently (Bu et al., 2008). Without explicit access to the reward signals, imitation learning could be the most intuitive solution for learning good policies directly from demonstrations. Conventional solutions such as behavior cloning (BC) (Pomerleau, 1991) learn the policy in a supervised manner by requiring numerous data while suffering from com- pounding error (Ross& & Bagnell, 2010; Ross et al., 2011). Inverse reinforcement learning (IRL) (Ng al., 2000; Russell, 1998) alleviates these shortcomings by recovering a reward function but is always expensive to obtain the optimal policy due to the forward reinforcement learning procedure in an inner loop. Generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) leaves better candidate for its model-free structure without compounding error, which is highly effective and scalable. However, cause of the strong correlations among adaptive agents' policies and rewards. Consider if a football coach wants to win the league, he must make targeted tacties against various opponents, in addition to the situation of his team. Moreover, the multi-agent environment tends to give rise to more severe compounding errors with more expensive running costs.
multi-agent interactions could be much challenging imitate be-
Motivated by these challenges, investigate the problem of modeling complicated multi-agent interactions from a pile of off-line demonstrations and recover their on-line policies, which can re- generate analogous multi-agent behaviors. Prior studies for multi-agent imitation learning typically limit the complexity in demonstrated interactions by assuming isolated reward structures (Barrett et al., 2017; Le et al., 2017; Lin al., 2014; Waugh al., 2013) and independence in per-agent

---

Published as a conference paper at ICLR 2020
policies that overlook the high correlations among agents (Song et al., 2018; Yu et al., 2019). In this paper, we cast the multi-agent interactions modeling problem into a multi-agent imitation learning framework with correlated policies by approximating opponents' policies, in order to reach inacces- sible opponents' actions due to concurrently execution of actions among agents when making deci- sions. Consequently, with approximated opponents model, we develop Decentralized Adversarial Imitation Learning algorithm with Correlated policies (CoDAIL) suitable for learning correlated policies under our proposed framework, which allows for decentralized training and execution. We prove that our framework treats the demonstrator interactions as one of c-Nash Equilibrium (c-NE) solutions under the recovered reward.
In experiments, we conduct multi ditiesiimenii comparisons for both the reward gap between learned agents and demonstrators, along with the distribution divergence between demonstrations and regen- erated interacted trajectories from learned policies. Furthermore, the results reveal that CoDAIL can better recover correlated multi-agent policy interactions than other state-of-the-an multi-agent im- itation learning methods in several multi-agent scenarios. We further illustrate the distributions of regenerated interactions, which indicates that CoDAIL yields the closest interaction behaviors to the demonstrators.
2 PRELIMINARIES
2.1 MARKOV GAME AND e-NASII EQUILIBRIUM
Markov game (MG), or stochastic game (Littman, 1994), can be regarded as an extension of Markov Decision Process (MDP). Formally, we define an MG with N agents as tuple SN S,) ((t),,..,,A!!. A(N), (,,(1), ,,(4),...,, ,,(),,, Po; 7), where S is the set of states, A represents the action space of agent i, where i E (1, N),, P S x (A(1) x Aa2)) x... x A x S 4 RR is the distribution of the initial state s and y [0, 1] is the discounted factor. Each agent holds its policy x (a(1)) S A + [0, to make decisions and receive rewards defined (() S A()) (2) x...x..xx X R. We use -i to represent the set of agents except i, and variables without superscript to denote the concatenation of all variables for all agents (e.g., represents the joint policy and denotes actions of all agents).
the state transition probability distribution, Po S
For arbitrary function ʃ (s, a) + IR, there is fact that E,, [f( a)] P [s(s a)] A E [E 7 (st, ac)], where s Po. ae 81+1 P(s az, s,). The objective of agent is to maximize its own total expected return R800) A Er(r(((s,, a)] E
i>> (0)(1)) (sc, dt)].
In Markov games, however, the reward function for each agent depends on the joint agent actions. Such fact implies that one's optimal policy must also depend on others' policies. For the solution to the Markov games, e-Nash equilibrium (e-NE) is commonly used concept that extends Nash equilibrium (NE) (Nash, 1951).
Definition 1. An c-NE is a strategy profile (IT Tt (") such that 3e > 0:
v Tx (9,x((1))) ((-))
2 ( (s, (1), Te ((-1)) - e, ( (1)
(1)
where (1) x (1), xl (11) = -,,0, )ap== (10, a. a. "11
x(-0)
the value function of agent i
under state and (1)) is the set of policies available to agent i.
e-NE is weaker than NE, which can be seen as sub-optimal NE. Every NE is equivalent to an e-NE where e O.
2.2 GENERATIVE ADVERSARIAL IMITATION LEARNING
Imitation learning aims to learn the policy directly from expert demonstrations without any access to the reward signals. In single-agent settings, such demonstrations come from behavior trajectories sampled with the expert policy, denoted as TÆ {(s,, a - However, in multi-agent settings, demonstrations are often interrelated trajectories, that is, which are sampled from the interactions of policies among all agents, denoted as Ωg {(5,, of") ...., a )))) For simplicity, will use the term interactions directly as the concept of interrelated trajectories, and we refer to trajectories for single agent.
2

---

Published as a conference paper at ICLR 2020
Typically, behavior cloning (BC) and inverse reinforcement learning (IRL) two main approaches for imitation learning. Although IRL theoretically alleviates compounding error and outperforms to BC, it is less efficient since it requires resolving RL problem inside the learning loop. Re- cent proposed work aims to learn the policy without estimating the reward function directly, no- tably, GAIL (Ho & Ermon, 2016), which takes advantage of Generative Adversarial Networks (GAN (Goodfellow et al., 2014)), showing that IRL is the dual problem of occupancy measure matching. GAIL regards the environment as black-box, which is non-differentiable but can be leveraged through Monte-Carlo estimation of policy gradients. Formally, its objective can be ex- pressed as
min max E [log (s,, a)] + Ex [log (1 - D(s, a)) - (T)
(2)
D
where D is a discriminator that identifies the expert trajectories with agents' sampled from policy T, which tries to maximize its evaluation from D; the causal entropy for the policy and À the hyperparameter.
2.3 CORRELATED POLICY
In multi-agent learning tasks, each agent makes decisions independently while the resulting reward (5)
p(() (st, di ag ((- depends others' actions, which makes its cumulative return subjected to the joint policy æ. One common joint policy modeling method is to decouple the π with assuming conditional independence of actions from different agents (Albrecht & Stone, 2018):
T Ta a (s)) π
(3)
However, such non-correlated factorization the joint policy is vulnerable simplification which ignores the influence of opponents (Wen et al., 2019). And the learning process of agent i lacks sta- bility since the environment dynamics depends on not only the current state but also the joint actions of all agents (Tian et al., 2019). To solve this, recent work has taken opponents into consideration by decoupling the joint policy as a correlated policy conditioned on state and a- as
ग. ( a (1)) = n. (6) (a (s, al (- (a)
(4)
where (a((aa))))))))))))))) s, ا(=ا) is the conditional policy, with which agent regards all potential ac- (s), and makes decisions through the marginal policy n (1) (a(0 Jaco n (a |s, (())))) (a)) da(-i) E.c-a (0)( (a a(())
tions from its opponent policies (a(0(a())))))
3 METHODOLOGY
3.1 GENERALIZE CORRELATED POLICIES TO MULTI-AGENT IMITATION LEARNING
In multi-agent settings, for agent i with policy (9), it seeks maximize its cumulative reward against demonstrator opponents who equip with demonstrated policies 7g (-i) via reinforcement learning:
RL (r arg max (0) + E R,,, (0 (-9 (1,00 (-))) z(0)
(5)
where H (T) is the y-discounted entropy (Bloem & Bambos, 2014; Haarnoja et al., 2017) of policy (0) and À is the hyperparameter. By coupling with Eq. (5), we define an IRL procedure to find a reward function p(0 such that the demonstrated joint policy outperforms all other policies, with the
regularizer 2b SxAA ((M)) (N)
:::
IRL (T arg max - 15( (0)
max (a + 1119, (s. a ((-1))) m(
(4)
(6)
+ Ex [r (1) (s,a())))))))))))) (1) (a,
3

---

Published as a conference paper at ICLR 2020
It is worth noting that we cannot obtain the demonstrated policies from the demonstrations directly. To solve this problem, we first introduce the occupancy measure, namely, the unnormalized distri- bution of (s, pairs correspond to the agent interactions navigated by joint policy æ.
p=(s,a)=T(als
=
=()))
(=0
With the definition in Eq. (7), can further formulate pr from agent i's perspective
(a( = xaa (aaa),a(-1))) (-(0))))
P((s = s|/T() x,(
Th (a (a) () 2t 0 7² P(st = x((00)
(8)
non correlated form
(a |s D (a Ei=o 7²t P(st =s x1,10)))))
correlated form
where (+)) NNT 7() and (-1) (-1)) Furthermore, with the support of Eq. (8), we have
E Bx 9)
E. Palt Bar Aax( [-]]
Pxt0 - (s, a aa I-]
a,a()),(-1)
In analogy to the definition of occupancy measure of that in single- environment, we follow derivation from Ho & Ermon (2016) and state the conclusion directly'.
Proposition The IRL regarding demonstrator opponents is a dual form of a occupancy measure matching problem with regularizer th, and the induced optimal policy is the primal optimum, specif- ically, the policy learned by on the reward recovered by IRI.cam be characterize by the following equation:
RL o (R arg min (x) + w/² (P 50 (-1) - pmg)
(10)
x(
With setting the regularizer 4b MGGA similar to Ho & Ermon (2016), we can obtain a GAIL-like (1)
imitation algorithm learn ag from Th given demonstrator counterparts π (-1) by introducing the
adversarial training procedures GANs which lead saddle point (I (())::
min max - (z) + Eng [og (i.) (s (1) d +
f [og (1 - D (s, ())
x0 D
(11)
where ( denotes the discriminator for agent i, which plays a role of surrogate cost function and guides the policy learning.
However, such an algorithm is not practical, since we are unable to access the policies of demonstra-
tor opponents T (-i) because the demonstrated policies are always given through sets of interactions necessary to deal with accessible counterparts. Thereby we
data. To alleviate this deficiency,
propose Proposition? 2.
Proposition 2. Let y be an arbitrary function such that μ holds a similar form as ((-1) then (P(0) (s,a
- = Exw,y se P=(t) (s,a
Proof. Substituting (-i) with μ in Eq. (9) by importance sampling.
Note that Ho & Ermon (2016) proved the conclusion under the goal to minimize the cost instead of maxi- mizing the reward of agent.
4

---

Published as a conference paper at ICLR 2020
Proposition 2 raises important point that a term of importance weight quantify the demon- strator opponents. By replacing "E (-t) with x(-1) , Eq. (11) is equivalent with
min max (T( + E#g loog D ( (s. d) + E 1 aa log (1 (s, a )) x00 p90
(12)
-000 (o, (
where a
PL (66)) (,,01)), ay is the importance sampling weight. In practice, is challenging to
estimate the densities and the learning methods might suffer from large variance. Thus, we fix a =
in our implementation, and as the experimental results have shown, it has significant influences on performance. Besides, a similar approach can be found in Kostrikov et al. (2018).
So far, we've built multi-agent imitation learning framework, which be easily generalized to correlated or non-correlated policy settings. No prior has to be considered advance since the discriminator able to learn the implicit goal for each agent.
3.2 LEARN WITH THE OPPONENTS MODEL
With the objective shown in Eq. (11), demonstrated interactions can be imitated by updating dis- criminators offer surrogate rewards and learning their policies alternately. Formally, the update of discriminator for each agent can be expressed as:
U/
(w)
(1)) (a ss a V log (1-11-144 D a ((, ()) da
V log D (s, al) a0 ad-1)]
+ E (s,a() at -D
(13)
and the update of policy is:
(0) E P. D Vo( Jac
10 (a a (1) ( al (0)a)))) (-))da())) AV gco H (T,)),
(14)
where discriminator D is parametrized by w, and the policy (1) is parametrized by 0. It is worth noting that the agent i considers opponents' action a(-9) while updating its policy and discriminator, with integrating all its possible decisions to find the optimal response. However, it is unrealistic to
have the access to opponent joint policy x<a{-i)ss) s) for agent i. Thus, it is possible to estimate using opponent modeling. To that end, we construct function a0 (a(-))) S (()) X (4-1)) x A(++1) .... A(N) - [0, 1p
opponents' actions via approximating -f ((
as the approximation of opponents for each agent i. Then we rewrite Eq. (13) and Eq. (14) as:
(wJ E. a.a a~att 100 Vota log(1 - D(1) (s, (+) (,) a(-
(15)
+ (, al Myeee
V log D (0) (0)aaa()))))) 00)
and V0J= E B- wl00
gois log Te (a al ) (
al a(-))) -V (T
(16)
respectively. Therefore, each agent must infer the opponents model ( to approximate the un- observable policies -(-1), which can be achieved via supervised learning. Specifically, we learn in discrete action space by minimizing a cross-entropy (CE) loss, and a mean-square-eree (MSE) loss in continuous action space:
2 Egsp [| (s(-))s) <<-) |s) [s]
continuous action space
L
(17)
Ea-p (r)))) log o (+)) (a(-))s)
discrete action space.
With opponents modeling, agents are able be trained in fully decentralized manner. We name our algorithm as Decentralized Adversarial Imitation Learning with Correlated policies (Correlated DAIL, a.k.a. CoDAIL) and present the training procedure in Appendix Algo. 1, which can be easily scaled to a distributed algorithm. As a comparison, we also present a non-correlated DAIL algorithm with non-correlated policy assumption in Appendix Algo. 2.
5

---

Published as a conference paper at ICLR 2020
3.3 THEORETICAL ANALYSIS
In this section, we prove that the reinforcement learning objective against demonstrator counterparts shown in the last section is essentially equivalent reaching an e-NE.
Since we fix the policies of agents -i as TE the RL procedure mentioned in Eq. (5) can be regarded as single-agent RL problem. Similarly, with fixed मø x the IRL process of Eq. (6) is cast to single-agent IRL. problem, which recovers an optimal reward function r! which achieves the best performance following the joint action Tg. Thus we have
RL (6) (r(2)) arg max AH (T
x00, (-4 (r(0 (s)" aa(
x(
(18)
π (1)
We can also rewrite Eq. (18) as (H (8) - 10 i v
a {l((T()) + B a (s, aa -11) (19)
a
for all x E (T, which is equivalent to
20
r2 (8t: Of ar (0))
E (9 (1) de No a N 0 =0==
2
(20)
E 20 ~ as W poms
Ar (8t, Of
of of-0))
- H(T VT E (T()
Given the value function defined in Eq. (1) for each agent i, for (= (1 < 0, (+ E 110 have
( (8,
T T E
)) 2 (8, T T
T 10) ((H(TT)) H(
(21)
For H (T()) (1TTT 2 0. ViT(1)
E (1) we have
(s, n
1 B
(2)
1-0)
(
TE
(-0) (
(22)
(s) =f (0) X(H(+(()) H ((T
Let c = A Amax [(()()) H(T
(0) | (1) IIO) }. then we finally obtain
(8,
5 : ( 10 (8, ( ((-)) e, (1 (T))
(23)
which is exactly the e-NE defined in Definition 1. We can always prove that is bounded in small values such that the e-NE solution concept is meaningful. Generally, random policies that keep vast entropy not always considered as sub-optimal solutions demonstrated policies TE in most reinforcement learning environments. As we do not require those random policies, we can remove them from the candidate policy set II))), which indicates that ())()) is bounded in small values, so as e. Empirically, we adopt small X, and attain the demonstrator policy mg with an efficient learning algorithm to become a close-to-optimal solution.
(4)
Thus, we conclude that the objective of our CoDAIL assumes that demonstrated policies institute an e-NE solution concept (but not necessarily unique) that can be controlled the hyperparameter under some specific reward function, from which the agent learns policy. It is worth noting that Yu et al. (2019) claimed that NE is incompatible with maximum entropy inverse reinforcement learning (MaxEnt IRL) because NE assumes that the agent never takes sub-optimal actions. Nevertheless, we prove that given demonstrator opponents, the multi-agent MaxEnt IRL defined in Eq. (6) is equivalent to finding e-NE.

---

Published as a conference paper at ICLR 2020
4 RELATED WORK
Albeit non-correlated policy learning guided by a centralized critic has shown excellent properties in couple of methods, including MADDPG (Lowe al., 2017). COMA (Foerster et al., 2018), MA Soft-Q (Wei et al., 2018), it lacks in modeling complex interactions because its decisions making relies on the independent policy assumption which only considers private observations while ignores the impact of opponent behaviors. To behave more rational, agents must take other agents into con- sideration, which leads to the studies of opponent modeling (Albrecht Stone, 2018) where an agent models how its opponents behave based on the interaction history when making decisions (Claus & Boutilier, 1998; Greenwald et al., 2003; Wen et al., 2019; Tian et al., 2019).
For multi-agent imitation learning, however, prior works fail to learn from complicated demon- strations, and many of them are bounded with particular reward assumptions. For instance, Bhat- tacharyya et al. (2018) proposed Parameter Sharing Generative Adversarial Imitation Learning (PS- GAIL) that adopts parameter sharing trick to extend GAIL to handle multi-agent problems directly, but it does not utilize the properties of Markov games with strong constraints the action space and the reward function. Besides, there are many works built-in Markov games that are restricted un- der tabular representation and known dynamics but with specific prior of reward structures, as fully cooperative games (Barrett al., 2017: Le et al., 2017; Sošic al., 2016; Bogert & Doshi, 2014), two-player zero-sum games (Lin et al., 2014), two-player general-sum games (Lin et al., 201 8), and linear combinations of specific features (Reddy al., 2012: Waugh et al., 2013).
Recently, some researchers take advantage of GAIL. to solve Markov games. Inspired by a spe- cific choice of Lagrange multipliers for a constraint optimization problem (Yu et al., 2019). Song et al. (2018) derived a performance gap for multi-agent from NE. It proposed multi-agent GAIL (MA-GAIL). where they formulated the reward funetion for each agent using private actions and observations. As an improvement, Yu et al. (2019) presented multi-agent adversarial inverse rein- forcement learning (MA-AIRL) based on logistic stochastic best response equilibrium and MaxEnt IRL. However, both of them are inadequate to model agent interactions with correlated policies with independent discriminators. By contrast, our approach can generalize correlated policies to model the interactions from demonstrations and employ a fully decentralized training procedure without to get access to know the specific opponent policies.
Except for the way of modeling multi-agent interactions recovering agents' policies from demon- strations, which can regenerate similar interacted data, some other works consider different effects of interactions. Grover et al. (2018) proposed to learn a policy representation function of the agents based on their interactions and sets of generalization tasks using the learned policy embeddings. They regarded interactions as the episodes that contain only k (in the paper they used agents), which constructs an ggent-interaction graph. Different from us. they focused on the potential re- lationships among agents to help characterize agent behaviors. Besides, Kuhnt al. (2016) and Gindele et al. (2015) proposed to use the Dynamic Bayesian Model that describes physical rela- tionships among vehicles and driving behaviors to model segractioo-dependee behaviors in au- tonomous driving scenario.
Correlated policy structures that can help agents consider the influence of other agents usually need opponents modeling (Albrecht & Stone, 2018) to infer others' actions. Opponent modeling has rich history in MAL (Billings et al., 1998; Ganzfried & Sandholm, 2011), and lots researches have recently worked out various useful approaches for different settings in deep MARL, c.g., DRON (He et al., 2016) and ROMMEO (Tian et al., 2019). In this paper, we focus on imitation learning with correlated policies, and we choose a natural and straightforward idea of opponent modeling that learning opponents' policies in the way of supervised learning with historical trajectories. Opponent models are used both in the training and the execution stages.
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
Environment Description We test our method the Particle World Environments (Lowe al., 2017), which is popular benchmark for evaluating multi-agent algorithms, including several co- operative and competitive tasks. Specifically, we consider two cooperative scenarios and two com-
7

---

Published as a conference paper at ICLR 2020
Table I: Average reward gaps between demonstrators and learned agents in cooperative tasks. Means and standard deviations are taken across different random seeds.
Algorithm
Coop. -Comm.
Coop.-Navi.
Demonstrators
0 + 0
0 + 0
MA-AIRL
0.780 = 0.917
6.696 = 3.646
MA-GAIL
0.638 = 0.624
7.596 + 3.088
NC-DAIL
0.692 = 0.597
6,912 = 3.971
CoDAIL
0.632 = 0.685
6.249 + 2.779
Random
186.001 + 16.710
322. 122 + 15.358
Table Average reward gaps between demonstrators and learned agents in 2 competitive tasks, where 'agent+' and 'agent-' represent 2 teams of agents and 'total' is their sum. Mcans and standard deviations are taken across different random seeds.
Keep-away
Pred-Preyy
Algorithm
Total
Agent+
Agent-
Total
Agent+
Agent-
Demonstrators
0 0:00
0
0: 0
0
0.1:0
0:1: 0
MA-AIRL
12:273
1.817
4.149 = 1,912
8.998 = 4.345
279.535 + 77.903
35,100 1.891
174.235 = 73.168
MA-GAIL
1.963 + 1,689
1.104 1.212
1.303 = 0.798
15.788 + 10.887
4,800 + 2.718
8.826
3.810
NC-DAIL
1,805 +: 1.695
1.193 1: 0.883
1.539 = 1,188
27.611 1: 14.645
8,260 1: 7,087
6.975 i: 5.130
CoDAIL
0.269 + 0.078
0.064
0.041
0.219
0.084
10.456 +: 6.762
4,500 + 3.273
4.359 2,734
Random
28:272
2,968
25.183
2,150
53.455 +: 2.409
100.736 6.870
37.980
2.396
13,204 8.444
with 2 agents and 3 landmarks, where an unmovable speaker knowing the goal, cooperates with listener to reach particular landmarks who achieves the goal only through the message from the speaker; 2) Cooperative-navigatio 3 agents and 3 landmarks, where agents must cooperate via physical actions and it requires each agent to reach one landmark while avoiding collisions; 3) Keep-away, with I agent, I adversary and landmark, where the agent has to get close to the landmark, while the adversary is rewarded by pushing away the agent from the landmark without knowing the target; 4) Predator-pree with I prey agent with 3 adversary predators, where the slower predactor agents must cooperate to chase the away from the adversaries.
petitive ones as follows: 1) cnoperative-commuiicc
with
prey agent that moves faster and try
Experimental Details We aim to compare the quality of interactions modeling in different aspects. To obtain the interacted demonstrations sampled from correlated policies, we train the demonstra- tor agent via MARL learning algorithm with opponents modeling to regard others' policies into one's decision making, since the ground-truth reward in those simulated environments is accessi- ble. Specifically, modify the multi-agent version ACKTR (Wu et al., 2017; Song et al., 2018), an efficient model-free policy gradient algorithm, by keeping an auxiliary opponents model and a conditioned policy for each agent, which can transform the original centralized on-policy learning algorithm to be decentralized. Note that we do not necessarily need experts that can do well in our designated environments. Instead, any demonstrator will be treated as it is from an e-NE strategy concept under some unknown reward functions, which will be recovered by the discriminator. In our training procedure, we first obtain demonstrator policies induced by the ground-truth rewards and then generate demonstrations, i.e., the interactions data for imitation training. Then we train the agents through the surrogate rewards from discriminators. We compare CoDAIL. with MA- AIRL, MA-GAII, non-correlated DAIL (NC-DAIL) (the only difference between MA-GAIL and NC-DAIL is whether the reward function depends on joint actions or individual action) and a random agent. We do not apply any prior to the reward structure for all tasks to let the discriminator learn the implicit goals. All training procedures are pre- pre-trained via behavior cloning to reduce the sample complexity, and we 200 episodes of demonstrations, each with maximum of 50 timesteps.
5.2 REWARD GAP
Tab. and Tab. 2 show the averaged absolute differences of reward for learned agents compared to the demonstrators in cooperative and competitive tasks, respectively. The learned interactions are considered superior if there smaller reward gaps. Since cooperative tasks are reward-sharing, we show only a group reward for each task in Tab. 1. Compared to the baselines, CoDAIL achieves smaller gaps in both cooperative and competitive tasks, which suggests that algorithm has
8

---

Published as a conference paper at ICLR 2020
Table 3: KI. divergence of learned agents position distribution and demonstrators position distribution from an individual perspective in different scenarios. 'Total' is the KI. divergence for state-action pairs of all agents, and 'Per' is the averaged KI. divergence of each agent. Experiments are conducted under the same random seed. Note that unmovable agents not recorded since they never move from the start point, and there only one movable agent in Cooperativ--oommmicntte
Coop.-Come
Coop-NNavi.
Keep-away
Pred.-Prey
Algorithm
Total/Per
Total
Per
Total
Per
Total
Per
Demonstrators
0
MA-AIRL
35.525
18,071
47.241
69.146
98.248
71.568
118.511
MA-GAIL
34.681
15.034
45,550
51.721
69.820
9.998
27.116
NC-DAIL
38.002
16.202
46.040
46.563
61.780
16.698
33.307
CoDAIL
6.427
9.033
23.100
3.113
5.735
8.621
22.600
Random
217.456
174.892
221.209
191.344
234.829
37.555
82.361
robust imitation learning capability of modeling the demonstrated interactions. It is also worth noting that CoDAIL achieves higher performance gaps in competitive tasks than cooperative ones, for which we think that conflict goals motivate Besides, MA-GAIL and NC-DAIL are about the same, indicating that less important is the surrogate reward structure on these multi-agent scenarios. To our surprise, MA-AIRL does not perform well in some environmente and even fails in Predator-pres We list the raw obtained rewards in Appendix C, and we provide more hyperparameter sensitivity results in Appendix D.
complicated interactions than shared goal.
5.3 DIVERGENCE OVER INTERACTIONS
Since we aim to recover the interactions of agents generated by the learned policies, it is proper evaluate the relevance between distributions of regenerated interactions and demonstration data. Specifically, we collect positions of agents over hundreds of state-action tuples, which can be re- garded the low-dimension projection of the state-action interactions. We start each episode from a different initial state but the same for each algorithm in one episode. We run all the experiments under the same random seed, and collect positions of each agent the total 100 episodes, each with a maximum of 50 timesteps.
We first estimate the distribution of position (x. y) via Kernel Density Estimation (KDE) (Rosen- blatt, 1956) with Gaussian kernel to compute the Kullback-Leein Leibler (KI.) divergence between the generated interactions with the demonstrated ones, shown Tab. the KL divergence between regenerated interactions with demonstrator interactions, CoDAIL gen- erates the interaction data that obtains the minimum gap with the demonstration interaction, and highly outperforms other baseline methods. Besides, MA-GAIL and NC-DAIL reflect about-the- same performance to model complex interactions, while MA-AIRL behaves the worst, even worse than random agents on Predator-prey.
evident that in terms of
5.4 VISUALIZATIONS INTERACTIONS
To further understand the interactions generated by learned policies compared with the demonstra- tors, we visualize the interactions for demonstrator policies and all learned ones. We plot the density distribution of positions, (1., y) and marginal distributions of z-position and y-position. We illus- trate the results conducted on Keep-away in Fig. 1, other scenarios can be found in the Appendix E. Higher frequency positions in collected data are colored darker the plane, and higher the value with respect to its marginal distributions.
As shown in Fig. 1, the interaction densities demonstrators and CoDAIL agents are highly similar (and with the smallest KL divergence), which tend walk in the right-down side. In contrast, other learned agents fail to recover the demonstrator interactions. policies interact to earn similar rewards, but still keep vast differences among their generated interactions. Furthermore, such a result reminds us that the real reward is not the best metric to evaluate the quality of modeling the demonstrated interactions imitation learning (Li et al., 2017).
worth noting that even different

---

Published as a conference paper at ICLR 2020
2
0
-1
-1
0
1
2
2
-1
0
1
2
2
0
1
2
(a) Demonstrators (KL = 0)
(b) MA-GAIL (KL 51.721)
(c) MA-AIRL (KL 69.146)
-1
-1
-1
-2
-2
-2
(d) CoDAIL (KL = 3.113)
(c) NC-DAIL (KI. 46.563)
(l) Random (KL 191.344)
The density and marginal distribution of agent positions, in 100 repeated episodes with different initialized states, generated from different learned policies upon Keep-away. The top row of each sub-figure drawn from state-action pairs of all agents. Mcanwhile, the bottom row explains for each individual (KL means the KL divergence between generated interactions shown in the top row and the demonstrators).
Figure 1:
CONCLUSION
In this paper, focus modeling complex multi-agent interactions via imitation learning demonstration data. We develop a decentralized adversarial imitation learning algorithm with corre- lated policies (CoDAIL) with approximated opponents modeling. CoDAIL allows for decentralized training and execution and is more capable of modeling correlated interactions from demonstra- tions shown by multi-dimensional comparisons against other saat-off-the-ar art multi-agent imitation learning methods on several experiment scenarios. In the future, we will consider covering more im- itation learning tasks and modeling the latent variables of policies for diverse multi-agent imitation learning.
ACKNOWLEDDEMEEN
We sincerely thank Yaodong Yang for helpful discussion. The corresponding author Weinan Zhang supported by NSFC (61702327, 61772333, 61632017). The author Minghuan Liu is supported by Wu Wen Jun Honorary Doctoral Scholarship. Al Institute, Shanghai Jiao Tong University.
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-99 95, 2018.
Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone. Making friends on the fly: Cooperating with new leammates. Artificial Intelligence, 242:132-171, 2017.
Raunak P Bhattacharyya, Derek J Phillips, Blake Wulfe, Jeremy Morton, Alex Kuefler, and Mykel J Kochenderfer. Agent imitation learning for driving simulation. In 2018 JEEE/RSJ Interna- fional Conference on Intelligent Robots and Systems (IROS), pp. 1534-1539. IEEE, 2018
Darse Billings, Denis Papp. Jonathan Schaeffer, and Duane Szafron. Opponent modeling in poker. Aaai/iaai, 493:499, 1998.
Michael Bloem and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse rein- forcement learning. In 53rd IEEE Conference Decision and Control, pp. 4911-4916 IEEE, 2014.
Kenneth Bogert and Prashant Doshi. Multi-robot inverse reinforcement learning under occlusion with interactions. In Proceedings of the 2014 international conference on Autonomous agents and Multi-Agent Systems, pp. 173-180. International Foundation for Autonomous Agents and Multiagent Systems, 2014.
Lucian Bu, Robert Babu, Bart De Schutter, et al. A comprehensive survey of multiagent reinforce- ment learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 38(2):
2008
Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li. Multi-Agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 2019.
Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multia- gent systems. AAAI/AAI,, 1998(746-752):2, 1998.
Jakob N Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-Agent policy gradients. In Proceedings of the 32th Conference Association for the Advancement of Artificial Intelligence, 2018.
Sam Ganzfried and Tuomas Sandholm. Game theory-basee opponent modeling in large imperfect- information games. In The 10th International Conference on Autonomous Agents and Multiagent Systems- Volume pp. 533-540. International Foundation for Autonomous Agents and Multia- gent Systems, 2011.
Tobias Gindele. Sebastian Brechtel, and Rudiger Dillmann. Learning driver behavior models from traffic observations for decision making and planning. IEEE Intelligent Transportation Systems Magazine, (11):69-79.20) 2015.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceedings of the 28th Conference on Advances in Neurul Information Processiny Systems, pp. 2672-2680, 2014.
Amy Greenwald, Keith Hall, and Roberto Serrano. Correlated q-Learning In ICML, volume 3, pp. 242-249, 2003.
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations in multiagent systems. In International Conference Machine Learning, pp. 1797-1806, 2018.
Tuomas Haarojj,, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 1352-1361. JMLR. org, 2017.

---

Published as a conference paper at ICLR 2020
He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III. Opponent modeling in deep rein- forcement learning. In International Conference on Machine Learning, pp. 1804-1813, 2016.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Proceedings of the 30th Conference on Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Max Jaderberg, Wojeiech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari Morcos, Avraham Ruderman, et al. Human- level performance in first-person multiplayer games with population- bassdd deep reinforcement learning. arXiv preprint wvxiv:1807.0128/ 2018.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-Actor Criti- Addressing sample inefficiency and reward bias in adversarial imitation learning. 2018.
Florian Kuhnt, Jens Schulz, Thomas Schamm, and J Marius Zöllner. Understanding interactions between traffic participants based on learned behaviors. In 2016 IEEE Intelligent Vehiclee Sym- posium (IV). pp. 1271-1278, IEEE, 2016.
Hoang M Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-Agent imitation learn- ing. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp- 1995-2003. JMLR. org, 2017.
Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-Agent reinforcement learning. In Proceedings of the 30th conference on International World Wide Web Conferences, pp. 983-
994. ACM, 2019.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from vi- sual demonstrations. In Proceedings the 31st Conference on Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Xiaomin Lin, Peter A Beling, and Randy Cogill. Multi-Agent inverse reinforcement learning for zero-Sum games. arXiv preprint arXiv: 1403.6508, 2014.
Xiaomin Lin, Stephen C Adams, and Peter A Beling. Multi-Agen inverse reinforcement learning for general-sum stochastic games. arXiv preprint (af::180609799 2018.
Michael L Littman. Markov games as a framework for multi-Agent reinforcement learning. In Proceedings the the s1 Machine Learning International Conference, pp. 157-163. Elsevier, 1994.
Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi- Agent Actor-Critie for mixed cooperative- cooperativ-Coompttiivee environments. In Proceedings of the 31st Conference on Advances in Neural Information Processing Systems, pp. 1379-6390, 2017.
James Martens and Roger Grosse. Optimizing neural networks with frotecker-factored approxi- mate curvature. In Proceedings of the 32nd conference on International Conference on Machine Learning, pp. 2408-2417, 2015.
John Nash. Non-Cooperative games. Annals of Mathematics, pp. 286-295, 1951.
Andrew Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. Proceedings of the 17th International Conference Machine Learning, volume 1, pp. 2, 2000.
OpenAl. Openai five. http::/blog. opwgga..pen...... wwt.c///o//oa.iiiiiceemo. ve,,, 2018.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neu- ral Computation, 3(1):88-97 97, 1991.
Tummalapalli Sudhamsh Reddy, Vamsikrishna Gopikrishna, Gergely Zaruba, and Manfred Huber. Inverse reinforeement learning for decentralized con-Cooperative multiagent systems. In 2012 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 1930-1935. IEEE, 2012.
12

---

Published as a conference paper at ICLR 2020
Murray Rosenblatt. Remarks some nonparametric estimates of density function. The Annals of Mathematicaa Statistics, pp. 832-837, 1956.
Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 661-668, 2010.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction to no- no-Regret online learning. In Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, pp. 627-635, 2011.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the 11st Annual Conference Computational Learning Theory, volume 98, pp. 101-103, 1998.
Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon. Multi-Agent generative adversarial imitation learning. In Proceedings of the 32ed Conference on Advances in Neural Information Processing Systems, pp. 7461-7472, 2018.
Adrian Šošic. Wasiur R KhudaBukhsh, Abdelhak M Zoubir, and Heinz Koeppl. Inverse reinforce- ment learning swarm systems. stat, 1050: 2016.
Zheng Tian, Ying Wen, Zhichen Gong, Faiz Punakkath, Shihao Zou, and Jun Wang. A regularized opponent model with maximum entropy objective. arXiv preprint arXiv:/905.0000 2019.
Kevin Waugh, Brian D Ziebart, and J Andrew Bagnell. Computational rationalization: The inverse equilibrium problem. arXiv preprint
2013.
Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-Learning, In 2018 AAAI Spring Symposium Series, 2018.
Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. Probabilistic recursive reasoning for multi-Agent reinforcement learning. arXiv preprint arXiv: /91.09900 2019.
Yuhuai Wu, Elman Mansimov, Roger B Grosse. Shun Liao. and Jimmy Ba. Scalable trust-Region method for deep reinforcement learning using Lromecker-Factoree approximation. In Proceedings of the 31 Advances in Neural Information Processing Systems, pp. 5279-5288, 2017.
Lantao Yu, Jiaming Song, and Stefano Ermon. Multi-Agent adversarial inverse reinforcement learn- ing. arXiv preprint 1rXiv:1907.132200 2019.
13

---

Published as a conference paper at ICLR 2020
Appendices
A ALGORITHM OUTLINES
A.1 CODAIL ALGORITHM
Algo. demonstrates the outline for our CoDAIL algorithm with non-correlated policy structure defined in Eq. (4), where we approximate the opponents model (a(-i) (s). improve the discrimi- nator D and the policy (()) iteratively.
Algorithm CoDAIL Algorithm
1: Input: Expert interactive demonstrations Ωg ~ we, policy parameters ((1), ((N), N value parameters (b3), ,.., ((N), N opponents models parameters v)(2), vj(N) and discrim- inator parameters (.()) (N);
2: for k == 1, do
3: Sample interactions among N agents Qg n, where in correlated policy structure and each agent utilizes its opponents model (+) help making decisions.
4: for agent = .., .N do
5: Use state-action pairs (s, ((=))) E Og to update v5((0) to minimize the objective as shown in
Eq. (17).
6: For every state-action pair (s, ( Sg sample estimated opponent policies from oppo-
nents model: ( (()) (a (s), and update w(() with the gradient shown in Eq. (15).
7: Compute advantage estimation A for each tuple (ss a1) ,, ;((-))) with surrogate reward function (s, (1), a(-))) = log(D with (s, (aa"""",a000)) a(-o))) - logg(1 D (8, ( a(-)))
T-1
1 a oz ,0 ((1)) Σ
(7 (S448) affer a( + ((((()) (0o) (s, a-)
(24)
k=0
-v(0
-V0 (s, (()
(25)
8: Update (60) to minimize the objective:
T Σ
2
T(
(s, (0, a00) 1(0) (s,, a()))
(26)
9: Update g(t) following the gradient shown in Eq. (16):
Voca log To (8) (a (a |s, ā ((=))))))))))))))) A (8, (s) (a((()))]]]] VV H(T (3))
)
s~ pl p a0000
(27)
end for 11: end for
14

---

Published as a conference paper at ICLR 2020
A.1.1 NC-DAIL ALGORITHM
We outline the step by step NC-DAIL algorithm with non-correlated decomposition of joint policy defined in Eq. (3) in Algo. 2.
Algorithm NC-DAIL Algorithm
1: Input: Expert interactive demonstrations fg Tg. N policy parameters @(1, ,..., B(N), N value parameters (!11,, ...., g(N) and N discriminator parameters (1)), wl
@()),..,
2: for 0, 1, 1.2,... do
3: Sample interactions between N agents Ok n, where is in non-correlated policy structure.
4: for agent .,,,
N do 5: Use (s, (t)) al-i)) Satoo update (0 with the gradient:
En, V log(D wit (s, a 1)]] + En Vuro log(
log(D Wo (s, (a(-))))))
(28)
6: Compute advantage estimation A for (s, ( aaaaaiaaaaa0 a((1)) E S2 with surrogate reward func- tion ((0(s, (1, ((1)) = -log( (1)
with (() a(()))) - log(1 D witl (8, alt) a(-0))
T-1
A ( (0) . (1)) = Σ (3) ($ aft - + 7 v((( ((i) (s, a
(29)
k=0
-v(1
(ii) (8, (-1)
(30)
7: Update (00) to minimize the objective:
T Σ
2 (s, a aa(-)))) (s,,
(31)
8: Update 8 by taking gradient step with:
Eo, Vaca log n (a ssA AV gvn H(T))))
(32)
9: end for
10: end for
B MODEL ARCHITECTURES
During our experiments, we use two layer MLPs with 128 cells in each layer, for policy networks, value networks, discriminator networks and opponents model networks on all scenarios. Specifically for opponents models, we utilize a mallti-had-stteen opponent's 's action separately, and we get the overall opponents joint action of (-i) by concatenating all actions. The batch size & Grosse, 2015) with learning rate of 0.1 and with small À of 0.05. All other parameters for K- FAC optimizer the same in (Wu al., 2017). We train each algorithm for 55000 epochs with random seeds to gain its average performance on all environments.
network, where each head predicts each
1000. The policy is trained using K-FAC optimizer (Martens
15

---

Published as a conference paper at ICLR 2020
C Raw RESULTS
We list the raw obtained rewards of all algorithms each scenarios.
Table 4: Raw average total rewards in 2 comparative tasks. Means and standard deviations are taken across different random seeds.
Algorithm
Coop.-Comm.
Coop,-Navi. -178.597 6.383
Demonstrators
-24.560 1 1.213
MA-AIRI
-25.366 + 1.492
-172.733 + 5.595
MA-GAIL.
-25.081 + 1,421
-172.169 + 4.105
NC-DAIL
-25.177 + 1.371
-171.685 = 4.591
CoDAIL
-25.107 + 1,486
-183.846 + 5.728
Random
-247.606 + 17.842
-1139,569 + 19.192
Table 5: Raw average rewards of each agent in 2 competitive tasks, where agent+ and agent- represent 2 teams of agents and total their sum. Means and standard deviations taken across different random seeds.
Keep-away Agent+
Algorithm
Total
Agent-
Demonstrators
-18.815
0.909
-12.092 + 0.617
-6.723 + 0.430
MA-AIRL
-31.088 2.371
-15.367 = 3.732
-15.721 = 4.448
MA-GAIL
-20.778 0.994
-12.818 1.105
-7.959 + 0.796
NC-DAIL
-20.619 + 0.957
-12.357 + 1.424
-8.262 + 1.310
CoDAIL
-19.084 0.882
-12.142 + 0.578
-6.942 + 0.433
Random
47.086 + 2.485
13.091 = 2.032
-60.177 = 2.225
Pred.-Prey
Algorithm
Total
Agent+
Agent-
Demonstrators
65.202
18.661
44.820
4.663
-69.258 5.361
MA-AIRL
-210.546 + 80.333
8.040 + 3.626
-234.666 = 71.165
MA-GAIL
65.202 18.661
44.820 + 4.663
-69.258 + 5,361
NC-DAIL
59.553 30.684
42.320
10.323
-67.407 3.700
CoDAIL
79.445 + 5.913
47.480
+ 4.067
-61.909
+ 6.367
Random
-31.747 7.865
5.160
1.170
47.227
7.830
16

---

Published as a conference paper at ICLR 2020
D HYPERPARAMETER SENSITIVITY
Table 6: Results of different training frequency (1:4, 1:2, 1:1, 2:1, 4:1) of and G on Communication- navigation. Means and standard deviations are taken across different random seeds.
Training Frequency
Total Reward Difference
2541.144 487.711 12.004 + 5.496
1:2
1:1
6.249 + 2.779
2:1
1136.255 + 1502.604
2948.878 + 1114.528
Toa
60
0.2
24
66 0A
14
Entmay Contioent A
Figure 2: Results of different entropy coefficient A.
We evaluate how the stability of our algorithm when the hyperparameters change during our ex- periments on cawtumiicationnaiiattiiin agents and demonstrators when we modify the training frequency of D and G (i.e., the policy), which indicates that the frequencies of D and G are more stable when D is trained slower than G, and the result reaches relative better performance when the frequency is 1:2 or 1:1. Fig. 2 illus- trates that the choice of À has little effect on the total performance. The reason may be derived from the discrete action space in this environment, where the policy entropy changes gently.
Tab. 6 shows the total reward difference between learned
17

---

Published as a conference paper at ICLR 2020
E INTERACTION VISUALIZATIONS UPON OTHER SCENARIOS
We show the density of interactions for different methods along with demonstrator policies con- ducted upon Copperativ-coommuiiationn in Fig. 3.
-2
2
0
0
(a) Demonstrators (KI. 0)
(b) MA-GAIL (KL 34.681)
(c) MA-AIRL (KL = 35.525)
-2 -2
-2
-2
Q
1
-1
Q
1
2
0
1
(d) CoDAIL (KL = 6.427)
(e) NC-DAIL (KL 38.002)
(f) Random (KI. = 217.456)
Figure The density and marginal distribution agents' positions, (x, y), 100 repeated episodes with differ- ent initialized states, generated from different learned policies upon Cooperative communication.. Experiments are done under the same random seed, and we only consider one movable agent. KI. the KI. divergence between generated interactions (top figure) with the demonstrators.
18

---

Published as a conference paper at ICLR 2020
We show the density of interactions for different methods along with demonstrator policies ducted upon peaperaaive-navigatinn in Fig. 4.
-1
-1
-1
-2
-2
-2
(a) Demonstrators (KL 0)
(b) MA-GAIL (KL= 15.034)
(c) MA-AIRL (KL 18.071)
2
0
0
-1
-1
-1
-2
-2
-2
0
1
2
0
1
2
0
1
2
(d) CoDAIL (KL = 9.033)
(e) NC-DAIL (KL = 16,202)
(l) Random (KL. = 174.892)
Figure 4: The density and marginal distribution of agents' positions, y). in 100 repeated episodes with different initialized states, generated from different learned policies upon Cooperative-naviiation Experiments are done under the same random seed. The top of each sub-figure is drawn from state-action pairs of all agents while the below explain for each one. KL is the KL divergence between generated interactions (top figure) with the demonstrators.
19

---

Published as a conference paper at ICLR 2020
We show the density of interactions for different methods along with demonstrator policies ducted upon Predator- Predator-prey prey in Fig. 5.
-1
-1
-1
-2
-2
-2
(a) Demonstrators (KL 0)
(b) MA-GAIL (KL 9.998)
(c) MA-AIRL (KL 71.568)
2
0
0
-1
-1
-1
-2
-2
-2
0
1
2
-2
0
1
2
-2
0
1
2
(d) CoDAIL (KL= = 8.621)
(e) NC-DAIL (KL = 16.698
(1) Random (KL 37.555)
Figure S: The density and marginal distributions of agents' positions, (x, y), in 100 repeated episodes with different initialized states, generated from different learned policies upon Predator-prey. Experiments are con- ducted under the same random seed. The top of each sub-figure is drawn from state-action pairs of all agents while the below explains for each one. The KL term means the KL divergence between generated interactions (top figure) with the demonstrators.
20


Reviews:
[{"review_id": "rJezMXKS5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to model interactions in a multi-agent system by considering correlated policies. In order to do so, the work modifies the GAIL framework to derive a learning objective. Similar to GAIL, the discriminator distinguishes between state, action, next state sequences but crucially the actions here are considered for all agents.\n\nThe paper is a natural extension of GAIL/MA-GAIL. I have two major points that need to be addressed.\n\n1. The exposition and significance of some of the theoretical results is unclear.\n- The non-correlated and correlated eqns in 2nd and 3rd line in eq. 8 are not equivalent in general, yet connected via an equality.\n In particular, Proposition 2 considers an importance weighting procedure to reweight state, action, next state triplets. It is unclear how this resolves the shortcomings of pi_E^{-1} being inaccessible. Prop 2 shifts from pi_E^{-1} to pi^{-1} and hence, the expectations in Prop 2 and Eq. 11 are not equivalent. \n- More importantly, how are the importance weights estimated in Eq. 12? The numerator requires pi_E^{-1}, which is not accessible. If the numerator and denominator are estimated separately, it becomes a chicken-and-egg problem since the denominator is itself intended to be an imitating the expert policy appearing in the numerator?\n\n2. Missing related work\nThere is a huge body of missing work in multi-agent interactions modeling and generative modeling. [1, 2] consider modeling of agent interactions via imitation learning and a principled evaluation framework of generalization in the Markov games setting. By sharing parameters, they are also able to model correlations across agent policies and have strong results on generalization to cooperation/competition with unseen agents with similar policies (which wouldn't have been possible if correlations were not modeled). Similarly, [3, 4] are other similar works which consider modeling of other agent interactions/diverse behaviors via imitation style approaches. Finally, the idea of correcting for the mismatch in state, action, next state triplets in Proposition 2 has been considered for model-based off-policy evaluation in [5]. They proposed a likelihood-free method to estimate importance weights, which seems might be necessary for this task as well (re: qs. on how are importance weights estimated?).\n\nRe:experiments. Results look good and convincing for most parts. I don't see much value of the qualitative evaluation in Figure 1. If the KL divergence is low, we can expect the marginals to be better estimated. Trying out various levels of generalization as proposed in [2] would significantly strengthen the paper.\n\nTypos\nsec 2.1 Transition dynamics should have range in R+\nProof of Prop 2. \\mu instead of u\n\nReferences:\n[1] Learning Policy Representations in Multiagent Systems. ICML 2018.\n[2] Evaluating Generalization in Multiagent Systems using Agent-Interaction Graphs. AAMAS 2018.\n[3] Machine Theory of Mind. ICML 2018.\n[4] Robust imitation of diverse behaviors. NeurIPS 2017.\n[5] Bias Correction of Learned Generative Models using Likelihood-free Importance Weighting. NeurIPS 2019.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Hyeeki6nKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, a multi-agent imitation learning algorithm with opponent modeling is proposed, where each agent considers other agents’ expected actions in advance and uses them to generate their own actions. Assuming each agent can observe other agents’ actions, which is a reasonable assumption in MARL problems, a decentralized algorithm called CoDAIL is proposed. For each iteration of CoDAIL, (1) each agent trains opponent models (other agents’ policies) by minimizing either MSE loss (continuous actions) or CE loss (discrete actions), (2) samples actions from those opponent models, (3) updates individual rewards (discriminators) and critics and (4) updates policies with multi-agent extention of ACKTR (which is used in MA-GAIL and MA-AIRL as well).\n\nThe experiments in the submission show that there is a significant gain relative to baselines (MA-GAIL and MA-AIRL) in OpenAI Multiagent Particle Environments (MPE) in terms of (true) reward differences and KL divergence between agents’ and experts’ state distributions.\n\nI think the empirical contribution of this work is clear to be accepted, but I give Weak Accept due to the following comments:\n\n- I think there’s a similarity between Theorem 6 in MA-GAIL paper and Proposition 1 in the submission. I hope the difference between Proposition 1 and Theorem 6 to be clarified. \n\n- Proposition 2 seems to me redundant because it’s neither important for theoretical analysis in 3.3 nor for the experiments. I believe a few sentences are enough to describe why authors choose \\alpha=1 (or equivalent explanations).\n\n- The authors suppose fully observable Markov Games in the paper, but it makes me confused when I consider the experiments in the submission. For example in Cooperative Navigation, each agent’s observation includes (1) position vector relative to agents and landmarks and (2) their own velocities (which cannot be observed by other agents directly). Since authors argue CoDAIL is a decentralized algorithm, I think agents are not allowed to use others’ observation for opponent modeling, but it seems that agents fully utilize others’ observations. I hope it to be clarified and if that’s the case, I wonder if we can regard CoDAIL as a decentralized method. \n\nI’m willing to increase my score if my questions are clearly answered. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "B1gXtGEb5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a decentralized adversarial imitation learning algorithm with correlated policies, which recovers each agent’s policy through approximating opponents action using opponent modeling. Extensive experimental results showed that the proposed framework, CoDAIL, better fits scenarios with correlated multi-agent policies.\n\nGenerally, the paper follows the idea of GAIL and MAGAIL. Differing from the previous works, the paper introduces \\epsilon-Nash equilibrium as the solution to multi-agent imitation learning in Markov games. It shows that using the concept of \\epsilon-Nash equilibrium as constraints is consistent and equivalent to adding the difference of the causal entropy of the expert policy and the causal entropy of a possible policy in RL procedure. It makes sense. \n\nBelow, I have a few concerns to the current status of the paper.\n\n1.\tThe authors propose \\epsilon-Nash equilibrium to model the convergent state in multi-agent scenarios, however, in section 3.1 the objective function of MA-RL (Equation 5) is still the discounted causal entropy of policy, the same as that of MA-GAIL paper. It is unclear how the \\epsilon-NE is considered in modeling MA-RL problem.\n\n2.\tRather than assuming conditional independence of actions from different agents, the authors considered that the joint policy as a correlated policy conditioned on state and all opponents’ actions. With the new assumption, the paper re-defines the occupancy measure and introduces an approach to approximate the unobservable opponents’ policies, in order to access opponents’ actions. However, in the section 3.2 when discussing the opponents modeling, the paper did not clearly explain how the joint opponent function \\sigma^{(i)} is designed. The description \\sigma^{(i)} is confusing.\n\n3.\tTypos: in equation 14 “i” or “-i”; appendix algorithm 1 line 3 “pi” or “\\pi”. \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an extension to the popular Generative Adversarial Imitation Learning framework that considers multi-agent settings with "correlated policies", i.e., where agents' actions influence each other. The proposed approach learns opponent models to consider possible opponent actions during learning. Several questions were raised during the review phase, including clarifying questions about key components of the proposed approach and theoretical contributions, as well as concerns about related work. These were addressed by the authors and the reviewers are satisfied that the resulting paper provides a valuable contribution. I encourage the authors to continue to use the reviewers' feedback to improve the clarity of their manuscript in time for the camera ready submission.
Decision: Accept (Poster)