Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the important problem of being able to deal with heterogeneous teams of agents (that might change over time) in cooperative multiagent reinforcement learning. In the strictly cooperative setting with identical problem representations among agents, this essentially boils down to a problem of decentralized learning and control, where, effectively, the improvements demonstrated by the paper over existing papers boil down to effectively reducing the ways in which non-local information needs to be communicated between a global observer (coach) and the individual learning (agents). The specific novelty of the approach is to introduce a variational objective to stablise training, and introduce a heuristic for reducing the communication frequency. 

As is the case with many similar papers in deep reinforcement learning, much of the paper's approach is premised on pre-training high-dimensional function approximators with an enormous amount of data to learn an optimal strategy offline that can then be applied in sequential games, in this case a reasonably complex (yet toy) scenario. Only a small part of the paper focuses on the technical innovations while the bulk of it describes the application of DNN techniques to the specific coach-player mechanism in a specific game. In my view, this does not provide enough evidence to be able to assess the general applicability of the approach, in particular because it is only evaluated in a single domain and much of the reported performance could be attributed to fine-tuning of domain-specific parameters. 

The paper is generally well-written and relatively easy to follow, though many of the design decisions are not very well explained, and there is no explicit problem formulation regarding what optimal communication strategies would look like, which would help evaluate the advances reported in the paper with respect to the overall fundamental problem. As is the case with many similar papers, while the existing work is surveyed, I find comparisons to other approaches somewhat unfair, as many of them propose actual algorithms that deal with the problem of coordination at runtime, rather than creating than making (as is proposed here) certain small modifications to immensely data-hungry function approximation methods that learn an effective coordination policy from data, and offer little insight on what it actually looks like that advances our understanding of the problem to build better future AI systems.
 
The language is not perfect in some places, but could be easily polished with additional proofreading.
review_title: An approach to improving the effectiveness of decentralised cooperative learning with limited novelty and validation

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
The paper proposes to extend the centralized training, decentralized execution paradigm for cooperative multi-agent RL with a coach-player framework. The proposed framework would allow the coach to have access to the full observation, but only allow for limited communication of a continuous strategy vector to the other agents / players. It is argued, that the framework helps with dynamic composition of multi-agent teams, which encompasses variable numbers of agents, as well as heterogeneous agents.
A variational objective is introduced, to force the player trajectories to be coherent with the strategy assigned by the coach. A concrete algorithm is proposed, that builds strongly upon [1]. Experimental evidence shows, that the coach-player framework with variational objective outperforms a baseline from this paper (referred to as A-QMIX) even if the latter is granted full observability for its agents.
 
Strong:
Considering the addition of a coach agent with constraint communication makes intuitively sense and there very well might be scenarios, where this addition is useful to solve actual tasks. The paper provides experimental evidence, that structured communication about the coordination of decentralized agents is beneficial, even if the communication frequency is enforced to be sparse. The variational objective seems to be especially helpful. A proof is included, which shows that sparse communication results in only small loss compared to more frequent communication if the representation of the team strategy is relatively stable (wrt. changes in its L2 norm). The paper includes results which suggest good generalization performance of the proposed algorithm wrt. unseen team composition.
 
Weak:
Mainly, I consider the papers experimental evaluation to be not very convincing. This might very well be due to the paper proposing a specific type of communication (with global knowledge), while disregarding to argue about different types of communication in MARL and only comparing to an approach that does not explicitly involve communication. This seems especially strange, since the multi-agent particle environment upon which the experiments build [2], was explicitly contrived to analyze communication (grounded in RL scenarios). The comparison of the proposed algorithm (referred to as COPA) to the baseline (A-QMIX) alone does not seem very insightful. Both show about the same performance if A-QMIX agents are provided with full observations and COPA is ablated wrt. its variational objective. That COPA’s performance for this task seems dependent on the variational objective (which is not further discussed), to me is an indication that important issues are not addressed in the paper / the experiments. Given its focus, I further argue that the paper lacks related literature with respect to learned communication in MARL [e.g. compare with listed literature in a current survey, 3].

Conclusion:
In its current form, I find the paper’s line of argumentation, as well as the experiments, to detached from relevant prior work wrt. learned communication within/for MARL. Further, I believe, the paper would benefit from disentangling the communication idea with the variational objective, which is more in line with hierarchical approaches (which is accounted for in the related work section).
 
 Other comments:
The AI-QMIX reference [1] changed significantly after the paper submission. An update of the reviewed paper with respect to this should be considered. Also, it is easy to get lost in the very involved notation. (Wrt. this: Are the subscripts to z consistent in Section 3.4; especially in the paragraph leading to Theorem 1?)
 
[1] Iqbal, Shariq; Witt, Christian A. Schroeder de; Peng, Bei; Böhmer, Wendelin; Whiteson, Shimon; Sha, Fei (2020): Randomized Entity-wise Factorization for Multi-Agent Reinforcement Learning. In: arXiv preprint arXiv:2006.04222.
[2] Mordatch, Igor; Abbeel, Pieter (2018): Emergence of Grounded Compositional Language in Multi-Agent Populations. In: AAAI Conference on Artificial Intelligence.
[3] Hernandez-Leal, Pablo; Kartal, Bilal; Taylor, Matthew E. (2019): A survey and critique of multiagent deep reinforcement learning. In: Autonomous Agents and Multi-Agent Systems 33 (6), S. 750–797


review_title: In its current form, I find the paper’s line of argumentation, as well as the experiments, to detached from relevant prior work wrt. learned communication within/for MARL. Further, I believe, the paper would benefit from disentangling the communication idea with the variational objective, which is more in line with hierarchical approaches (which is accounted for in the related work section).

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: 
The authors propose a coach-player framework for dynamic team composition of dynamic and heterogeneous agents based on deep Q learning with an attention mechanism and a variational objective to regularize the learning. The authors design an adaptive communication strategy to minimize communication from the coach to the agents. Using a resource-collection task in multi-agent particle environments, the authors evaluated zero-shot generalization for new team compositions at test time. Results show comparable or even better performance against methods where players have full observation but no coach. Interestingly, there is almost no performance degradation even when the coach communicates as little as 13% of the time with the players. 

Reasons for score: 
Although the motivation, novelty compared with the related work, and a superiority of the proposed method in an experimental result were clear, there were unclear points in the background and methods (without sharing codes) and the authors performed only a simple experiment. I think the idea is interesting and contributed to this community, but for the above reasons it is difficult to provide a higher rating. 


Pros:
1. The motivation and novelty compared with the related work were clear. 
2. The contribution in this paper is to propose a coach-player framework for dynamic team composition of dynamic and heterogeneous agents based on deep Q learning with an attention mechanism and a variational objective to regularize the learning. 
3. In a resource-collection task in multi-agent particle environments, the proposed method clearly demonstrated the effectiveness of having a coach in dynamic teams.

Cons:
1. There were many unclear points in Sections 2 and 3 (methodology, see below)
2. The authors performed only one experiment in a resource-collection task.  It seems to be a relatively simple setting for me, but the investigation in another experiment will help us understand the effectiveness of the proposed method from more general perspectives.

Other comments:

Regarding deep recurrent Q-learning (Zhu et al., 2017), was u_0^a in "tau^a = (o_0^a, u_0^a,... o_t^a) correct?  If correct, more formal expression and additional explanation may be required. 

Did u’ in eqs. (1) and (2) mean u_{t+1}? It may be confusing.

Was 2.2 VALUE FUNCTION action-value function or Q function? 

“Speciﬁcally, the input o is represented by two matrices…” Was the input tau rather than o in 2.2? What is the input?

I did not understand the first equation of eq. (4). It seems to be different from the definition of the mutual information that I know, and the idea of the referenced papers (Rakelly et al., 2019; Wang et al., 2020a). A careful introduction is required.

======after rebuttal=========

Thank you for answering my questions.  I understood these points. The authors added a new simple experiment and a code, whereas the manuscript at the current stage can improve the clarity. All things considered, I increased the rating. 

review_title: Marginally above acceptance

final_score: rating: 7: Good paper, accept
review_content: Summary:

This paper studies the dynamic multi-agent team coordination problem, in which the optimal team strategy may change over time as the environment and the team members vary. The authors propose a coach-player framework in which only the coach has full information about the game while the players only have their own local information, separately. In this framework, the coach computes the optimal game plan and sends the plan to the players periodically, and the players play according to the received plan from the coach and their local information. Empirical studies demonstrate the effectiveness of the new framework.

Comments:

This paper is generally well-written and clear. The idea of having a coach to compute the optimal game plan to coordinate the players seems interesting and non-trivial. In the empirical study, COPA with variational objective outperforms the state-of-art benchmark. The authors also have an interesting observation that always communicating the optimal global game plan to the players is not always the optimal strategy.

1. The reviewer is not familiar with A-QMIX. Is it correct that the main difference between A-QMIX (full) and COPA is that in A-QMIX, even though the players have access to the full information, they still compute for their own game plans instead of coming up with a joint optimal game plan for the team?

2. As the authors claim that the coach is more useful when it can make the agents behavior smooth/consistent over time, the reviewer is wondering whether it would be helpful to communicate a smoothed strategy update to the players. More precisely, the current way is to set a threshold on the distance between z_t and z_old to limit the communication frequency. For example, if the coach always communicates, whether it would be helpful if the coach sends a convex combination between z_t and z_old (similar to ideas of online learning) to smooth the strategy update?
review_title: ICLR 2021 Conference Paper2657 AnonReviewer3

Meta-review: This paper proposes an approach for coordinating teams with dynamic composition consisting of an attention mechanism, regularization and communication. The clarity of the paper is currently low seemingly due to the conflated message of the multiple parts of the framework. Improvements to the text via the suggested edits of all reviewers should be a relatively quick fix, but the clearer placement of this piece within the wider literature may require additional experiments to compare against so would be a larger change. 

The reviewers did continue to discuss the paper after the end of the open discussion period with the authors and appreciated the additional experiments performed. In the absence of supporting theory, empirical results in a second domain significantly improve the evidence that the method may be more generally applicable. However, the new experiments raised new questions (included in the reviewers later replies) indicating more experiments in the second domain are needed which would require further peer review.

I hope the authors will take the constructive feedback provided here as intended; to improve the paper, submit the work again at a later stage when the second experimental domain is sufficiently explored to support the proposed framework and in doing so maximise its potential for impact. 
Decision: Reject