Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:

The paper proposes the use of "Copulas" to capture dependencies among agents in multi-player environments. The authors argue that prior work(eg GNN, VAE, LSTM etc based) do not leverage the common structure among agent behaviour. They explain how the copula can more efficiently encode the dependency among policies, as compared to simply factorizing. The authors provide empirical results on real world and synthetic datasets, showcasing superior performance against baseline approaches.

Overall, this seems like a novel approach with promising results on a few experimental set ups. The authors describe how their proposed model builds upon the limitations of prior work. The work seems moderately significant.

Comments:

* The writing is easy to follow. The paper motivate the problem well and describe the experimentation clearly.

* Nit: In Figure 1b) Can the authors clarify how the copula's differ in the upper figure? It seems to be a uniform distribution, I assume this applies for only the lower figure.

* For equation (3), can the notation be clarified: this refers to learning opponent policies rather than agents co-operating in a given task?

* Nit: Algorithm 2 can perhaps be described in the main section, being the central idea to this paper.

* For the datasets, can the number of agents in each be briefly described in Section 5.1 or Table 1? Is there any dependence on performance with respect to this parameter? The authors have mentioned heir intuition behind some of the results in Section 5.2.

* In Section 5.3, how were the replacement agents sampled? For eg, in the RoboCup dataset used, would substituting a defender for a forward influence the performance(as hinted in Section 5.4)?


Given the above points, novelty and significance  marking this as accept.

Update:
After seeing the author response below, no change to my score.
review_title: Learning a joint representation of agent policies in multi agent games

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a multi-agent imitation learning method that learns a joint policy for all agents from offline demonstrations. The key idea is to first learn a marginal policy for each agent using behavioral cloning, then fit a copula function that captures dependencies between the agents' policies. Experiments with particle simulations, simulated driving, and simulated RoboCup suggest that this approach could potentially outperform prior methods.

Theorem 1 seems like a powerful tool for multi-agent imitation learning, and I think this paper is a promising initial step. That being said, there are several issues:

In Tables 2 and 3, there are negative values for the negative log-likelihood. This is not possible, since log-likelihoods cannot be positive. I initially assumed that this was a typo in Table 2, but then I saw that Table 3 included both negative and positive values for the NLL, which makes me unsure about the overall correctness of the experimental evaluations in this paper.

Algorithm 1 uses behavioral cloning to fit the marginal policies. Doesn't this lead to compounding errors due to state distribution mismatch? Would more recent methods for offline imitation learning that overcome this issue, like ValueDICE [1], be a more suitable choice than behavioral cloning?

Doesn't learning the marginals first, then the copula second, lead to a model misspecification when fitting the marginals? Learning the marginals separately (without a copula) effectively assumes the factorization in Equation 1, which we know cannot represent the ground-truth joint policies in the experiments, so the learned marginals from step 1 are not necessarily the same marginals that you would learn if you were to fit the marginals and the copula simultaneously.

From Equation 7, we can see that different combinations of marginals and copulas could potentially induce the same joint policy. If you only fit the model by maximizing the log-likelihood of demonstrations for a single task, how can you distinguish between different pairs of (marginals, copulas) that fit the demonstrations equally well? Wouldn't you need to train on multiple tasks that share the same underlying copula but have different marginals, or on multiple tasks that share the same underlying marginals but have different copulas?

It would be nice to discuss the weaknesses of the proposed method, as well as directions for future work. For example, could this approach to learning a joint policy be extended to settings that require decentralized execution?

The dotted lines in Figure 3 are very difficult to see because they are small and because they are dark gray against a dark green background.

It would be nice to include the pseudocode in the main paper, since the main contribution of the paper is the algorithm.

[1] https://openreview.net/pdf?id=Hyg-JC4FDr 

Update after rebuttal
-----------------------------
Thanks to the authors for addressing my concerns. I have updated my score.
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper discusses the use of copulas for the multi-agent imitation learning scenario. The paper addresses an interesting application of copulas and is generally well written. Efficient imitation learning for MAS is a challenging problem, hence the paper is well motivated. While I like the overall idea of the paper, there are some concerns which the authors should address:

1. The authors claim that learning a copula is more efficient and scalable, however I don't feel this claim is well justified in the paper. In general, a copula captures arbitrary dependencies between the action distributions of $n$ agents, how exactly is then learning a copula any different from learning a joint distribution $\pi(a_1..a_n)$ in terms of learning complexity?

2. The authors should mention the approximate order complexity of the Algorithms 1, 2 and in general need to compare those with the competing methods mentioned in the paper to give a better understanding of the advantage using copula based policy learning.

3.  In general, maximising over a copula seems like a very hard problem, which is one of the reasons why mean-field methods are proposed, how exactly is the copula based factorisation avoiding this problem?

4. The authors should provide the architecture details of baseline methods to better interpret the results.

5. Root mean squared error (RMSE) between predicted actions and ground-truth actions for Table 1 exps is not a very convincing metric, specially from the standpoint of qualitative nature of policies imitated. It would be better if the authors can report reward scores for some suitable domains (ex. RoboCup).
review_title: Interesting problem but does the proposed solution really help?

Meta-review: This paper applies an existing tool (copulas) to MARL to represent dependencies between variables.

The reviewers appreciate the use of the copulas for this problem. The experimental section shows promising results on several problems. I appreciate that the authors have answered and addressed many points of concerns of the reviewers. The paper is well written

The reviewers seem to see this paper as a first step only, showing promising results but of moderate significance in itself. In particular, reviewer 3 would like to see more justifications for the use of the copulas, and a more experimental settings would make a stronger paper. 

Decision: Reject