Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes to model interactions in a multi-agent system by considering correlated policies. In order to do so, the work modifies the GAIL framework to derive a learning objective. Similar to GAIL, the discriminator distinguishes between state, action, next state sequences but crucially the actions here are considered for all agents.

The paper is a natural extension of GAIL/MA-GAIL. I have two major points that need to be addressed.

1. The exposition and significance of some of the theoretical results is unclear.
- The non-correlated and correlated eqns in 2nd and 3rd line in eq. 8 are not equivalent in general, yet connected via an equality.
 In particular, Proposition 2 considers an importance weighting procedure to reweight state, action, next state triplets. It is unclear how this resolves the shortcomings of pi_E^{-1} being inaccessible. Prop 2 shifts from pi_E^{-1} to pi^{-1} and hence, the expectations in Prop 2 and Eq. 11 are not equivalent. 
- More importantly, how are the importance weights estimated in Eq. 12? The numerator requires pi_E^{-1}, which is not accessible. If the numerator and denominator are estimated separately, it becomes a chicken-and-egg problem since the denominator is itself intended to be an imitating the expert policy appearing in the numerator?

2. Missing related work
There is a huge body of missing work in multi-agent interactions modeling and generative modeling. [1, 2] consider modeling of agent interactions via imitation learning and a principled evaluation framework of generalization in the Markov games setting. By sharing parameters, they are also able to model correlations across agent policies and have strong results on generalization to cooperation/competition with unseen agents with similar policies (which wouldn't have been possible if correlations were not modeled). Similarly, [3, 4] are other similar works which consider modeling of other agent interactions/diverse behaviors via imitation style approaches. Finally, the idea of correcting for the mismatch in state, action, next state triplets in Proposition 2 has been considered for model-based off-policy evaluation in [5]. They proposed a likelihood-free method to estimate importance weights, which seems might be necessary for this task as well (re: qs. on how are importance weights estimated?).

Re:experiments. Results look good and convincing for most parts. I don't see much value of the qualitative evaluation in Figure 1. If the KL divergence is low, we can expect the marginals to be better estimated. Trying out various levels of generalization as proposed in [2] would significantly strengthen the paper.

Typos
sec 2.1 Transition dynamics should have range in R+
Proof of Prop 2. \mu instead of u

References:
[1] Learning Policy Representations in Multiagent Systems. ICML 2018.
[2] Evaluating Generalization in Multiagent Systems using Agent-Interaction Graphs. AAMAS 2018.
[3] Machine Theory of Mind. ICML 2018.
[4] Robust imitation of diverse behaviors. NeurIPS 2017.
[5] Bias Correction of Learned Generative Models using Likelihood-free Importance Weighting. NeurIPS 2019.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The authors propose a decentralized adversarial imitation learning algorithm with correlated policies, which recovers each agent’s policy through approximating opponents action using opponent modeling. Extensive experimental results showed that the proposed framework, CoDAIL, better fits scenarios with correlated multi-agent policies.

Generally, the paper follows the idea of GAIL and MAGAIL. Differing from the previous works, the paper introduces \epsilon-Nash equilibrium as the solution to multi-agent imitation learning in Markov games. It shows that using the concept of \epsilon-Nash equilibrium as constraints is consistent and equivalent to adding the difference of the causal entropy of the expert policy and the causal entropy of a possible policy in RL procedure. It makes sense. 

Below, I have a few concerns to the current status of the paper.

1.	The authors propose \epsilon-Nash equilibrium to model the convergent state in multi-agent scenarios, however, in section 3.1 the objective function of MA-RL (Equation 5) is still the discounted causal entropy of policy, the same as that of MA-GAIL paper. It is unclear how the \epsilon-NE is considered in modeling MA-RL problem.

2.	Rather than assuming conditional independence of actions from different agents, the authors considered that the joint policy as a correlated policy conditioned on state and all opponents’ actions. With the new assumption, the paper re-defines the occupancy measure and introduces an approach to approximate the unobservable opponents’ policies, in order to access opponents’ actions. However, in the section 3.2 when discussing the opponents modeling, the paper did not clearly explain how the joint opponent function \sigma^{(i)} is designed. The description \sigma^{(i)} is confusing.

3.	Typos: in equation 14 “i” or “-i”; appendix algorithm 1 line 3 “pi” or “\pi”. 

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: In this work, a multi-agent imitation learning algorithm with opponent modeling is proposed, where each agent considers other agents’ expected actions in advance and uses them to generate their own actions. Assuming each agent can observe other agents’ actions, which is a reasonable assumption in MARL problems, a decentralized algorithm called CoDAIL is proposed. For each iteration of CoDAIL, (1) each agent trains opponent models (other agents’ policies) by minimizing either MSE loss (continuous actions) or CE loss (discrete actions), (2) samples actions from those opponent models, (3) updates individual rewards (discriminators) and critics and (4) updates policies with multi-agent extention of ACKTR (which is used in MA-GAIL and MA-AIRL as well).

The experiments in the submission show that there is a significant gain relative to baselines (MA-GAIL and MA-AIRL) in OpenAI Multiagent Particle Environments (MPE) in terms of (true) reward differences and KL divergence between agents’ and experts’ state distributions.

I think the empirical contribution of this work is clear to be accepted, but I give Weak Accept due to the following comments:

- I think there’s a similarity between Theorem 6 in MA-GAIL paper and Proposition 1 in the submission. I hope the difference between Proposition 1 and Theorem 6 to be clarified. 

- Proposition 2 seems to me redundant because it’s neither important for theoretical analysis in 3.3 nor for the experiments. I believe a few sentences are enough to describe why authors choose \alpha=1 (or equivalent explanations).

- The authors suppose fully observable Markov Games in the paper, but it makes me confused when I consider the experiments in the submission. For example in Cooperative Navigation, each agent’s observation includes (1) position vector relative to agents and landmarks and (2) their own velocities (which cannot be observed by other agents directly). Since authors argue CoDAIL is a decentralized algorithm, I think agents are not allowed to use others’ observation for opponent modeling, but it seems that agents fully utilize others’ observations. I hope it to be clarified and if that’s the case, I wonder if we can regard CoDAIL as a decentralized method. 

I’m willing to increase my score if my questions are clearly answered. 
review_title: Official Blind Review #2

Meta-review: The paper proposes an extension to the popular Generative Adversarial Imitation Learning framework that considers multi-agent settings with "correlated policies", i.e., where agents' actions influence each other. The proposed approach learns opponent models to consider possible opponent actions during learning. Several questions were raised during the review phase, including clarifying questions about key components of the proposed approach and theoretical contributions, as well as concerns about related work. These were addressed by the authors and the reviewers are satisfied that the resulting paper provides a valuable contribution. I encourage the authors to continue to use the reviewers' feedback to improve the clarity of their manuscript in time for the camera ready submission.
Decision: Accept (Poster)