Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper considers multi-agent RL (MARL), where agents share information at each step using communications.  The authors present a centralized training scheme where all the agents observations (sensing) are collected and processed / aggregated, and a focus of attention mechanism is also learned to weight the shared observations for each agent to collect only what is useful of the global information for each agent.  These can be added on to existing MARL methods (section 4.4).  The paper gives an overview the ideas, and then presents several numerical experiments and comparisons.  
strengths_and_weaknesses: The idea of learning a focus of attention in the centralized learning phase is interesting.  Basically, what agents should be communicating. The focus of attention is apparently changing with global state (but not well explained).

The presentation is confusing and hard to read, and the algorithm(s) are never clearly laid out. There is undefined notation, and cases where variables are introduced and (perhaps) defined later.  For example, even in the basic model eqn (1), Q, K, and V aren’t clearly defined.

The presentation is weak on mathematical presentation, and many results are described in an intuitive but often confusing way.  There is no theoretical analysis.

The overall scalability is likely to be poor.  Collecting all observations and then reducing them is not scalable.  

It’s not quite clear if this work is intended to be model-based or model-free.  When a map is known, and the agent’s states are known, then for example it is straightforward in the global training to set up a local neighborhood for communications.  The algorithm seems to come back to this in the examples, which is not surprising.  In section 2 it is stated that “agents only use local information o_i as message to share within the team” so apparently this is an underlying assumption (but it isn’t clear if agents who are far away from each other but have useful information will be enabled to communicate).

There are a lot of parameters and hyper-parameters, and their selection in general is not considered. 

The examples are useful, but also difficult to interpret without significant effort by the reader to examine all the cited works.  For example, it isn’t clear how each algorithm uses communications (or not), and why these are good for some problems and not others.  The conclusions about (local) communications are not surprising in each scenario given.  It seems that when the previously existing algorithm is well matched to the scenario, then there is little be gained with the proposed method.


questions: Why not just start with a notion of neighbors, and proceed from there?

The comparisons are very difficult to interpret.  For example, “QMIX performs well in LBF, which is attributed to the fact that agents can observe the grids near them, and the mixing network of QMIX can help improve the coordination ability of the fixed group of agents in the training phase.”  So, QMIX knows the state of the neighbors (and communication with them is unnecessary)?  

Perhaps some kind of table could be used to compare the different algorithms, including how they communicate or not, the observation model, and what kind of scenarios they fit (or not).  

Please clarify “updating of the focusing network is coupled to the RL objective, making the weights produced by the focusing network could be task-sensitive, which could also facilitate policy learning.”

“other communication methods such as TarMAC, NDQ, and TMC achieve low performance or even fail in this environment.”  Why?  Please be specific here, beyond some intuition.

And please clarify “why FullComm succeeds is that there is hardly any redundancy in agents’ observations in Hallway”.

“TarMAC and NDQ have high variance due to the instability of their messages, while MASIA gains high sample efficiency and can generate steady messages since it aims to reconstruct the state.”  What is message instability?

limitations: No negative social impact is expected.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors proposed MASIA which aims to encourage efficient communication among agents within cooperative MARL settings. MASIA uses information aggregator and extractor and learns the message representations that are compact and sufficient for agents’ communication. Internally, self-attention mechanism and predictive representation are used for such representations, where additional model network is used for predictive representation. MASIA is shown to mostly outperform existing MARL baselines on the tasks where agents’ communication is crucial to the overall performance. There are further ablation studies that shows MASIA’s learned features and its compatibility with existing MARL algorithms. 
strengths_and_weaknesses: Strengths: The authors’ intuitions and approaches are described well. Experiments are done in various perspectives to show the effectiveness of algorithms. Generalizability with existing algorithms is also a strong point.

Weaknesses: The scalability for a large number of agents (e.g., tens or hundreds of agents) is questionable as most of CTDE frameworks has such restriction. I could not find major weaknesses of this work. 

questions: 1. DEC-POMDP is considered to be a theoretical framework for this work, which implies the state s is not directly observable. However, s should be used for the reconstruction loss for information aggregation, which makes me a bit confused. I believe s here seems a set of all agents’ observations under the CTDE, but I hope that the authors clarify this. 

2. How each loss affects the performance of MASIA is unclear. From the appendix, I could see that the loss is balanced through the weights $\lambda_1$ and $\lambda_2$. What would be the performance when (1) $\lambda_1=0$ (2) $\lambda_2=0$ (3) $\lambda_1=\lambda_2=0$?

limitations: There seems no negative societal impact of this research.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose MASIA, a method for improving convergence characteristics and communication in cooperative MARL settings.

The key motivation of their work is to produce compact and informative communication by aggregating information from different agents into a single representation, which individual agents can get relevant information from. To incentivize "good" overall representations (depicted as $z_t$ in the paper, the authors introduce a decoding loss (to predict the full state) and a predictive loss (to predict future states via rollout).

In experiments in several standard MARL domains, the authors show that MASIA integrates well with existing MARL training frameworks and typically trains agents faster and more stably than existing methods. Qualitative analysis of emergent representations indicates somewhat interpretable clustering and similarity of communication vectors learned by MASIA agents.
strengths_and_weaknesses: ## Strengths ##

The authors address an important problem: stably training emergent communication. Their proposed solution integrates several distinct ideas (in my interpretation, the autoencoding and rollout objectives represent different concepts) seemingly successfully to train agents that, in experiments, outperform existing methods.

The evaluation is particularly impressive, using a wide variety of settings from existing literature, and reporting median (great job using robust statistics!) values over 5 random seeds. 


## Weaknesses ##

Literature review: The authors appear to have missed some relevant related works. The paper could benefit from better framing by citing works on the challenges of emergent communication (e.g., Eccles et al on "Biases for Emergent Communication in Multi-agent Reinforcement Learning"). Most importantly, though, the authors do not appear to know of Lin et al.'s "Learning to Ground Multi-Agent Communication with Autoencoders" from NeurIPS last year. Although Lin's technique is certainly different from this paper's, there are deep underlying similarities, especially regarding the use of an autoencoding loss to promote informative communication. Their method would be an extremely relevant baseline.

Claims about compactness: I am concerned about the authors' claims about compactness. The authors repeatedly note that they seek to induce "compact and sufficient" representations. I understand that an autoencoding loss incentivizes informative, or sufficient, representations by reward information content in representations. However, my understanding of compactness corresponds to notions of low information content, or compression. For example, information-bottleneck methods limit communication information, or using fewer discrete messages creates more "compact" representations. I do not see how the proposed work aligns with this notion. The authors write near line 173 that "By forcing the aggregated representation to be predictive on its future states, the aggregated representation could be more correlated with the information required for its decision-making, which meets the compactness standard." I don't think I agree. Could $z_t$ not simply encode all information about observations and states? If so, that is certainly informative of future states. How does that imply compactness?

Ablation studies: MASIA involves multiple new training objectives over baselines, so I wish I had seen a more complete ablation study to reveal the importance of different components. The studies included in the appendices are a good start: in particular A.3 answers a question I had when reading the paper (although I do actually wonder about the significance of the results, as there seems to be a lot of overlap in error bars and it's only conducted for the very early timesteps). However, there is no such ablation study (as far as I can tell) for the autoencoding loss.  To me, studying the autoencoding loss contribution would be more important than the different pooling mechanisms.

Some lack of clarity: Overall, I felt like I understood most, but not all, of the authors' proposed work. I elaborate on my questions in the next section. I do not mean to imply that the work was entirely unclear, only that there were parts that I did not understand from the main text. Depending on the answers to those questions, I will be better able to assess the technical contributions of the work. Because I am currently unsure about what information agents have access to, I'm not sure of how truly decentralized the agent setup is.

Although certainly not disqualifying on their own, this paper contains some small typos that would have to be fixed for publication. Examples include the caption for Figure 3 "ours" vs. "our", line 221 "injury" should be a verb of some sort.

## Summary ##

Overall, I like the paper and think it could be an important contribution, but I need to make sure I understand aspects of the design better before I recommend acceptance. In other words, I am very willing to "change my mind" because my mind isn't currently made up. I look forward to engaging in followup discussion based on my questions.
questions: 1) How do the agents behave different during decentralized execution vs. centralized training? I understand the paradigm and have written code using such frameworks. I just want to understand, given the MASIA architectures, what information agents have access to during test time. In particular, I want to understand what feeds into the information aggregation encoder at test time, and if $z_t$ is used at test time as well. I really need to understand this part to assess the technical merits of this work.

2) What are the inputs to the Information Aggregation Encoder generally? The authors use $o_i$ notation, which to me indicates different observations (including over time, with the $t$ superscript). So, for the traffic junction setting, for example, I assume that means the standard observations of things like the location, route id, and surrounding cells? Do the observations include communication, or is communication simply generated afresh as $z_t$ at every timestep based on the history of state observations?
limitations: The authors could dedicate more time to discussing limitations and societal impacts of their work. I understand that space within the 9-page limit is at a premium; at the very least, including a short discussion in an Appendix would be worthwhile. I simply think that it is an overstatement to say "Besides, we do not see it has obvious negative societal impacts." I can easily list multiple hypothetical scenarios in which, by pooling communication, you allow for adversarial attacks that cover up particular agents' communication, or you regress to mean behaviors instead of allowing for individualized responses. I do not wish to imply that the authors have to address all these concerns, but dismissing them without any discussion in the main paper seems like an overstep.
review_title: null

Meta-review: metareview: The reviewers agree that the main strengths are the generality of the approach, as well as the experimental results (especially after the rebuttal which answered most questions). The overall approach of choosing the attention focus is also well-motivated by specific experiments and makes sense.

A weakness of the paper that has been discussed is the assumption of full broadcast despite considering a decentralized execution scenario, even though in typical applications there are constraints on communication. The high computation cost of the method has similarly been mentioned (and acknowledged by the authors). While these are limitations of the paper, the discussion still ended up in favor of acceptance for the reasons above.


confidence: Less certain
Decision: Accept