Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper approaches the problem of allow members in a team to group together by allowing “conductor agents” to temporally construct groups and then allowing messages to be passed from agents to conductors (with a variational message summariser) in a unified scheduling approach. They show that in the case of having a dynamic number of agents, and also having partial observability, this method has good zero-shot generalisation performance on standard MARL benchmark tasks like predator-prey.

strengths_and_weaknesses: 1. This paper is clearly written and the motivation of this work is very sound
2. The authors spend some amount of time first outlining the downsides of centralised training and decentralised execution and why communication between agents is important and can help handle partial observability.
3. The motivation for self-organised grouping in order to build communication mechanisms that satisfy properties of being lightweight and robust is also clearly laid out.

questions: 1. Are there any insights on the differences between the message summariser using an entropy regulariser? It intuitively seems like it should work but from the results stated it is unclear why it does not
2. Are there results on increasing the message dimension upwards of 3? Do you have thoughts on why the different message dimensions (and ability to store more bits of information) do not impact performance?

limitations: I think the conductor selection might benefit from some prior information/thought put into selecting the conductors rather than randomly selecting.

Nit: some typos throughout the paper, particularly occurrences of “aims->aim” e.g., line 196, 204
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a novel group-based communication scheme for MARL which is trained under the CTDE paradigm. Agents are periodically grouped together with each group having a single conductor which receives messages from each agent and sends back a summary of all its received messages. Messages are received through a variational encoder-decoder architecture and aim to encode information about the local trajectory of agents. Three approaches to select conductors within groups based on random selection, a policy gradient RL agent and determinal point process are proposed. The resulting Self-Organized Group (SOG) algorithm is evaluated in varying environments and demonstrates generalisation ability over varying numbers of agents and varying partial observability across training and testing tasks without further fine-tuning.
strengths_and_weaknesses: # Strengths
1. Generalisation and communication for cooperative MARL are very relevant and difficult problems.
2. The idea to group agents and efficiently communicate among agents based on agent proximity to improve coordination is intuitive and well motivated. The method is clearly outlined and well depicted as three intertwined components.
3. Generalisation to tasks with a different number of agents and varying degrees of partial observability are considered in the conducted evaluation.
4. Experiments are conducted in three different environments and comparison is made to several MARL algorithms with similar architecture and communication. Message learning components are ablated in Table 1 and further experiments investigating generalisation to varying degrees of partial observability at constant number of agents are presented in the appendix.

# Weaknesses
**Major:**
1. I have concerns regarding the novelty and significance of this work. The core components of SOG appear to be (1) identifying groups of agents to communicate through conductor election, (2) encoding messages through a variational encoder optimized with a mutual information objective, (3) summarising/ aggregating messages, and (4) being invariant to number of agents due to architecture design. Out of these, it appears to me that only (1) and (3) are novel with (4) following from the proposed architecture of REFIL [12], and (2) being proposed similarly for NDQ [A]. Also, it is worth noting that prior work already considered the problem of identifying which agents should communicate with each other for most effective communication [A, B, C, D]. The aggregation of messages in (3) is simply a sum and the conductor election process (1) was shown to be comparable to just randomly selecting an agent as conductor in experiments of this work.

2. In l. 325ff the authors state that the RL-based selection of conductors for communication improves generalisation performance compared to the random selection of conductors based on their experimental results, but the results are not conclusive and in some cases even indicate better performance for the random selection process.

**Minor:**

3. For results in the predator prey environment, it is stated that “SOG performs better when transferred to complicated evaluation scenarios” (l. 364) but generalisation returns are very similar to baselines with large variance indicating no significant difference across algorithms.
4. The downsides to CTDE described is mostly a downside of a lack of information (no specific failure/ downside of CTDE). Also the described failure case should be investigated with a simple small experiment of the described task to verify the statements, QMIX and similar algorithms are not designed for these scenarios but agents might implicitly follow a pattern to still effectively coordinate.

[A] Wang, Tonghan, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. "Learning nearly decomposable value functions via communication minimization." International conference on learning representations, 2020.

[B] Du, Yali, Bo Liu, Vincent Moens, Ziqi Liu, Zhicheng Ren, Jun Wang, Xu Chen, and Haifeng Zhang. "Learning correlated communication topology in multi-agent reinforcement learning." In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 456-464. 2021.

[C] Jiang, Jiechuan, and Zongqing Lu. "Learning attentional communication for multi-agent cooperation." Advances in neural information processing systems, 2018.
questions: 1. Is the evaluated SOG algorithm based on REFIL as the MARL foundation with the additional communication paradigm? This does not quite come across clearly in Section 4.3.
2. Do all the evaluated baselines also apply parameter sharing across all agents? I would expect that parameter sharing helps significantly, in particular when communication is applied as agents follow the same communication and therefore already have a “common language” without further coordination. If not, all baselines should be (re-)evaluated with parameter sharing.
3. In Figure 4a, b, it appears that training performance in the Resource Collection task with larger sight range (1.0) is worse than in the task with shorter sight range (0.5)? This appears strange given that inputs are of the same size and in the shorter sight range case more values would simply be masked to 0s. Why do you think this appears?
4. What does shading in all training learning curves represent (standard deviation, variance, confidence intervals, …)? Same for tables 1 and 2.
5. In line 321 regarding the SMAC experiments, it is stated that a “imaginary objective” is applied as in prior work - what does this objective contain and incentivize?
6. In lines 167ff you state - “For those who receive no group invitation or leave the sight of the constructor during in-group communication, they form a group of themselves.” - it is unclear to me whether this means each of these agents is in a group alone or all these agents are together in a group? Latter would allow agents to communicate beyond boundaries of visibility.
limitations: The authors have sufficiently discussed limitations of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper develops a multi-agent reinforcement learning approach with attention to enable zero-shot generalization across scales of homogeneous agents. The approach, self-organized groups (SOG), develops mechanisms for a decentralized team to form ad hoc groups. Within each group, a conductor is elected. The conductor then provides messages to each of the other members of the group. A message summary procedure is added to maximize the mutual information between personal messages and future agent trajectories. Results across StarCraft and Predator-Prey show promising results. Ablation studies are provided to show the contribution of various conductor election schemes. Zero-shot results are provided for scaling the number of agents in Starcraft.
strengths_and_weaknesses: Strengths:
+ Section 3 provides a helpful background to formally define the problem being addressed.
+ Figure 3 provides a helpful illustration describing the model architecture.
+ Figure 4 shows promising results. It might be helpful to choose a color scheme that more clearly reflects that SOG and its ablations are a group, where as COPA, EMP, etc. are baselines.
+ The approach (and ablations) are evaluated in computational studies in StarCraft and Predator-Prey. 
+The evaluations show positive results in answering three key questions: (1) does the approach improve coordination vs. fully-connect communication models, (2) can the approach enable zero-shot generalization, and (3) how does performance vary with the CE selection mechanism?

Weaknesses:
- The color scheme from Figure 4 to Figure 5 is inconsistent. In one, COPA is red; in the other SOG is red.
- The paper seems to have accidentally missed some prior work that have addressed related concepts of scalability and message efficiency.
1) Wang et al. (ICML) develop a MARL approach, called IMAC, that uses variational information reasoning to learn communication protocols, similar to the message summary mechanism in SOG. As such, it would have been helpful to benchmark against IMAC – particularly to compare the ability to learn efficient, information-rich messages.
2) Niu et al., (AAMAS’21) developed a MARL approach, called MAGIC, with a centralized scheduler and message processor that performs message aggregation. The Niu et al. approach is scalable (using attention), but it has the weakness that it brings some centralization as it relies on an agent to act as a message aggregator and scheduler. It would be helpful if this paper would benchmark against MAGIC, as the problem setup is analogous (or provide sufficient justification for why HetNet is not a suitable baseline).
3) Seraj et al., (AAMAS’22) developed a CTDE MARL approach (with communication) for heterogeneous agent teams, called HetNet. The approach scales (again, using attention) with the number of agents of various types and compresses messages to binary. It would be helpful if this paper would benchmark against HetNet (or provide sufficient justification for why HetNet is not a suitable baseline).

Wang, R., He, X., Yu, R., Qiu, W., An, B. and Rabinovich, Z., 2020, November. Learning efficient multi-agent communication: An information bottleneck approach. In International Conference on Machine Learning (pp. 9908-9918). PMLR.

Niu, Y., Paleja, R.R. and Gombolay, M.C., 2021, May. Multi-Agent Graph-Attention Communication and Teaming. In AAMAS (pp. 964-973).

Seraj, E., Wang, Z., Paleja, R., Martin, D., Sklar, M., Patel, A. and Gombolay, M., 2022, May. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems (pp. 1173-1182).

Writing:
-“ which is featured with conductor election (CE) and message summary (MS)” is not grammatically quite correct. Perhaps “a message summary (MS) mechanism” would be better.
-Line 16 should be “have” instead of “has”
-Line 21, no comma before “while”
-Line 49 should have an endash before “based”
-Line 135 “like QMIX [25]” should be offset with commas
-Line 350 should have a comma after “First”
-Line 354 should not have a period after “Table”
-Line 372, it is helpful not to start a sentence with “but”
-Line 328 should have a comma after “Then”
-Line 344 – it is best not to have contractions

====== Post-rebuttal ======

I have raised my score to reflect the improvements the authors have made in their paper.

questions: - In Lines 137-139, the paper seems to suggest that agents that communicate are not part of the CTDE paradigm; however, communication (at least communication that is range-limitted) can be to be compliant with the CTDE paradigm. For example, see Ding et al., (NeurIPS’20). Does the paper mean to suggest otherwise?
-How well does the approach perform as a function of the communication range (and associated ability of the conductor to communicate with just one or all of the agents? 

Ding, Z., Huang, T. and Lu, Z., 2020. Learning individually inferred communication for multi-agent cooperation. Advances in Neural Information Processing Systems, 33, pp.22069-22079. 
limitations: The paper says that limitations were addressed and to see Section 4.2. However, word “limit” and “limitation” doe not appear anywhere in the paper except in regards to limitations of prior work or the ranges of agents tested (Line 319). As such, it is hard to see where limitations are sufficiently addressed as the checklist seems to intend. Note: The paper does state some assumptions in Line 148 (“assuming that all the messages can be delivered accurately without delay and each agent can handle all the received messages at the same time”) and Line 218 (“we assume a perfect communication channel”). However, the reproducibility checklists seems to differentiate between limitation and assumptions (though, they are semantically related). It would have been nice if the paper had proactively addressed the shortcomings of the work, when it would fail, when it might be disadvantageous to use the approach, when societal harm might occur (e.g., military applications), etc.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a Self-Organized Group mechanism for Cooperative MARL, which is claimed to have zero-shot generalization ability with dynamic team composition and varying partial observability. The mechanism first selects a condutor for each group; then the condutor summarizes the messages (by conditional entropy bottleneck objective) from all agents within this group, and sends the summarized message to all agents; finally, the agents generate actions based on the observation and the summarized message. For the condutor selection, the authors proposed Random, DPP-based and Policy Gradient-based methods. The authors conducted experiments on three environments to show the effectiveness of the proposed methods.
strengths_and_weaknesses: Strengths  
* Clear paper writting. In general, this paper is easy to read and follow.   
* Interesting methods for Conductor Election.





Weaknesses  
* The methods and the claims are not related. I do not see the direct relation between the proposed Self-Organized Group mechanism (i.e., both Conductor Election and Message Summary) and the so claimed zero-shot generalization ability with dynamic team composition and varying partial observability. In my opinion,  the zero-shot generalization ability is mostly because of the mean message mixer (i.e., Eq 1) and the MHA-based model structure (i.e., Fig 3).  
* Missing a lot of related works. For example, the Conductor is similar to the centralizer/coordinator used in many MARL communication methods [1,2]. The entropy bottleneck based Message Summary is also related with entropy bottleneck based communication message generation [3, 4]. The group-based multi-agent Message Summary is also similar to the Neighborhood Cognition Consistent proposed in [5], where summarized message is equal to the consistent cognition. But the authors missed these works.  
* Due to the above reason, I found the baselines used in the experiments are not strong. For example, the authors mentioned "Since many MARL methods with communication are not suitable for dynamic team composition", but as far as I know, [1, 4, 5] and many other MARL methods can be applied for dynamic team composition.


[1] Learning agent communication under limited bandwidth by message pruning. AAAI 2020.  
[2] Learning multi-agent communication with double attentional deep reinforcement learning. AAMAS 2020.  
[3] Deep multi agent reinforcement learning for autonomous driving. 2020.  
[4] Multi-agent Communication with Graph Information Bottleneck under Limited Bandwidth. 2021.  
[5] Neighborhood cognition consistent multi-agent reinforcement learning. 2020.
questions: * What is the relation between the proposed Self-Organized Group mechanism (i.e., both Conductor Election and Message Summary) and the so claimed zero-shot generalization ability? 
* Could the authors give some results by comparing with [1, 5] or other more strong baselines? Because [1] has a message prunning mechanism, which is like the T-timestep communication in the current paper. Similarly, the group-based multi-agent Message Summary is similar to the Neighborhood Cognition Consistent proposed in [5], and [5] can also be applied to dynamic team composition.  
limitations: Yes
review_title: null

Meta-review: metareview: The reviewers carefully analyzed this work and agreed that the topics investigated in this paper are important and relevant to the field. They generally expressed positive views on the proposed method but also pointed out a few possible limitations of this paper. One reviewer argued that the authors properly outlined the downsides of alternative approaches and the importance of communication as a way of dealing with partial observability. This reviewer, however, brought up one limitation: that the conductor selection may benefit from/require prior information. After reading the rebuttal, however, this reviewer said that the authors satisfactorily answered most of their questions, and further argued that the insights from this paper will most likely be interesting to the emergent communication community. Another reviewer claimed that this paper introduced a novel group-based communication scheme for MARL. They argued that the ideas explored here are intuitive and well-motivated. This reviewer initially believed that some of the experimental results were inconclusive (e.g., regarding the claims that RL-based selection of conductors improves performance w.r.t. to random selection). The reviewer also commented, in their original review, on the possible lack of novelty: out of four components that compose this method, two are novel, one may follow from [4], and one may follow from [A]". After carefully analyzing the authors' rebuttal, however, this reviewer increased their score: they believe that the authors' detailed rebuttal helped clarify minor concerns and that the reviewer's initially-voiced major doubts (e.g., regarding the significance of this work) were mostly rectified. Overall, this reviewer believes (post-rebuttal) that this is indeed a novel and interesting paper introducing news ideas toward communication in MARL—all of which were well executed and appropriately studied. Another reviewer expressed concerns that the paper did not discuss important prior work [1-3], but was satisfied with the authors' responses and thanked them for adding more baselines as part of the experiments. Finally, one reviewer argued that even though this is an interesting method, they still had (pre-rebuttal) three main points of concern. After reading the authors' rebuttal, this reviewer said that "the authors gave detailed responses to address my concerns and I appreciate the additional experiments". Overall, thus, it is clear that all reviewers were positively impressed with the quality of this work and look forward to an updated version of the paper that addresses the suggestions mentioned in their reviews and during the discussion phase.

confidence: Certain
Decision: Accept